2023-10-30T12:54:01.901 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T12:54:01.902 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041 --owner scheduled_leonidus@teuthology --name leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --verbose -- /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/orig.config.yaml 2023-10-30T12:54:01.954 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T12:54:02.035 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041 branch: wip-lusov-balance-after-stop description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7441041' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lusov-balance-after-stop ceph: cephfs: max_mds: 3 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-balance-after-stop sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 owner: scheduled_leonidus@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 8397 sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 sleep_before_teardown: 0 subset: 464/1024 suite: fs suite_branch: wip-lusov-balance-after-stop suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKeRIfRxzODgVbK6MlfsIgDCiYV1EZSosY4UyltWr7mHCPO7pEuEDj8D6YWgeqTEN/fv5kOWCQIbkChn6/v4dvg= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPEFw8CnH+x/8AQAfMpXjhYNhZQb3piNcsHdDTMDzCoqz04lQJ3zW4OeCWF+KuNDgMaIbs2PAfuzKUqbzbZ66m0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_06:34:16 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T12:54:02.035 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa; will attempt to use it 2023-10-30T12:54:02.036 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks 2023-10-30T12:54:02.036 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T12:54:02.038 INFO:teuthology.task.internal:Checking packages... 2023-10-30T12:54:02.059 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '00c1c13cea21a686b046584596fbeac5315a7d71' 2023-10-30T12:54:02.059 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T12:54:02.060 INFO:teuthology.packaging:ref: None 2023-10-30T12:54:02.060 INFO:teuthology.packaging:tag: None 2023-10-30T12:54:02.060 INFO:teuthology.packaging:branch: wip-lusov-balance-after-stop 2023-10-30T12:54:02.060 INFO:teuthology.packaging:sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:54:02.061 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lusov-balance-after-stop 2023-10-30T12:54:02.216 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:54:02.216 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T12:54:02.222 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T12:54:02.222 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T12:54:02.227 INFO:teuthology.task.internal:Saving configuration 2023-10-30T12:54:02.244 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T12:54:02.250 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:54:02.275 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:48:11.395517', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHMtbMh/TQpQMqzPUklmyXXXUjTLHRogWFHByfk3/GsU'} 2023-10-30T12:54:02.311 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:48:11.393968', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHg0q0GsC40GODq9IAmujnCtponcYuOOQU6C7V7oWClK'} 2023-10-30T12:54:02.311 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T12:54:02.318 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-30T12:54:02.318 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-10-30T12:54:02.318 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T12:54:02.386 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb10599bc10>, signals=[15]) 2023-10-30T12:54:02.387 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T12:54:02.393 INFO:teuthology.task.internal:Opening connections... 2023-10-30T12:54:02.393 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T12:54:02.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:54:02.474 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T12:54:02.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:54:02.548 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T12:54:02.554 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2023-10-30T12:54:02.560 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2023-10-30T12:54:02.560 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2023-10-30T12:54:02.609 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T12:54:02.609 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="Ubuntu" 2023-10-30T12:54:02.609 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="22.04" 2023-10-30T12:54:02.609 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T12:54:02.610 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_CODENAME=jammy 2023-10-30T12:54:02.610 INFO:teuthology.orchestra.run.smithi148.stdout:ID=ubuntu 2023-10-30T12:54:02.610 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE=debian 2023-10-30T12:54:02.610 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T12:54:02.610 INFO:teuthology.orchestra.run.smithi148.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T12:54:02.611 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T12:54:02.611 INFO:teuthology.orchestra.run.smithi148.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T12:54:02.611 INFO:teuthology.orchestra.run.smithi148.stdout:UBUNTU_CODENAME=jammy 2023-10-30T12:54:02.612 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2023-10-30T12:54:02.634 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2023-10-30T12:54:02.639 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-10-30T12:54:02.639 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2023-10-30T12:54:02.686 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T12:54:02.686 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Ubuntu" 2023-10-30T12:54:02.686 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="22.04" 2023-10-30T12:54:02.686 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T12:54:02.687 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_CODENAME=jammy 2023-10-30T12:54:02.687 INFO:teuthology.orchestra.run.smithi150.stdout:ID=ubuntu 2023-10-30T12:54:02.687 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE=debian 2023-10-30T12:54:02.687 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T12:54:02.687 INFO:teuthology.orchestra.run.smithi150.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T12:54:02.687 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T12:54:02.688 INFO:teuthology.orchestra.run.smithi150.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T12:54:02.688 INFO:teuthology.orchestra.run.smithi150.stdout:UBUNTU_CODENAME=jammy 2023-10-30T12:54:02.688 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-10-30T12:54:02.708 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T12:54:02.717 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T12:54:02.723 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T12:54:02.723 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2023-10-30T12:54:02.726 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2023-10-30T12:54:02.733 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T12:54:02.739 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T12:54:02.740 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2023-10-30T12:54:02.779 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2023-10-30T12:54:02.799 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T12:54:02.854 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T12:54:02.869 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-30T12:54:02.869 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-10-30T12:54:02.869 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-10-30T12:54:02.870 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T12:54:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:54:02.876 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2023-10-30T12:54:03.042 DEBUG:teuthology.task.kernel:sha1 for client.0 is eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T12:54:03.042 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "eacf00cc51e204f401b4d5d4f84825a68f373c74"... 2023-10-30T12:54:03.042 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-10-30T12:54:03.048 INFO:teuthology.orchestra.run.smithi148.stdout:5.15.0-56-generic 2023-10-30T12:54:03.048 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T12:54:03.049 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-30T12:54:03.049 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2023-10-30T12:54:03.049 DEBUG:teuthology.orchestra.run.smithi148:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-10-30T12:54:03.104 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-10-30T12:54:03.255 INFO:teuthology.orchestra.run.smithi148.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-10-30T12:54:03.256 INFO:teuthology.orchestra.run.smithi148.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-10-30T12:54:03.256 INFO:teuthology.orchestra.run.smithi148.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-10-30T12:54:03.256 INFO:teuthology.orchestra.run.smithi148.stderr:hint: 2023-10-30T12:54:03.256 INFO:teuthology.orchestra.run.smithi148.stderr:hint: git config --global init.defaultBranch 2023-10-30T12:54:03.257 INFO:teuthology.orchestra.run.smithi148.stderr:hint: 2023-10-30T12:54:03.257 INFO:teuthology.orchestra.run.smithi148.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-10-30T12:54:03.257 INFO:teuthology.orchestra.run.smithi148.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-10-30T12:54:03.257 INFO:teuthology.orchestra.run.smithi148.stderr:hint: 2023-10-30T12:54:03.257 INFO:teuthology.orchestra.run.smithi148.stderr:hint: git branch -m 2023-10-30T12:54:03.258 INFO:teuthology.orchestra.run.smithi148.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-10-30T12:54:03.260 DEBUG:teuthology.orchestra.run.smithi148:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-10-30T12:54:03.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-10-30T12:54:03.391 DEBUG:teuthology.orchestra.run.smithi148:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-10-30T12:54:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-10-30T12:54:43.738 INFO:teuthology.orchestra.run.smithi148.stderr: * [new branch] main -> origin/main 2023-10-30T12:54:43.761 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20190312 -> 20190312 2023-10-30T12:54:43.761 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20190416 -> 20190416 2023-10-30T12:54:43.761 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20190514 -> 20190514 2023-10-30T12:54:43.761 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20190618 -> 20190618 2023-10-30T12:54:43.762 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20190717 -> 20190717 2023-10-30T12:54:43.764 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20190815 -> 20190815 2023-10-30T12:54:43.764 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20190923 -> 20190923 2023-10-30T12:54:43.764 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20191022 -> 20191022 2023-10-30T12:54:43.764 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20191215 -> 20191215 2023-10-30T12:54:43.765 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20200122 -> 20200122 2023-10-30T12:54:43.765 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20200316 -> 20200316 2023-10-30T12:54:43.765 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20200421 -> 20200421 2023-10-30T12:54:43.765 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20200519 -> 20200519 2023-10-30T12:54:43.765 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20200619 -> 20200619 2023-10-30T12:54:43.766 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20200721 -> 20200721 2023-10-30T12:54:43.766 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20200817 -> 20200817 2023-10-30T12:54:43.766 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20200918 -> 20200918 2023-10-30T12:54:43.766 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20201022 -> 20201022 2023-10-30T12:54:43.766 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20201118 -> 20201118 2023-10-30T12:54:43.767 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20201218 -> 20201218 2023-10-30T12:54:43.767 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20210208 -> 20210208 2023-10-30T12:54:43.767 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20210315 -> 20210315 2023-10-30T12:54:43.767 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20210511 -> 20210511 2023-10-30T12:54:43.767 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20210716 -> 20210716 2023-10-30T12:54:43.768 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20210818 -> 20210818 2023-10-30T12:54:43.770 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20210919 -> 20210919 2023-10-30T12:54:43.770 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20211027 -> 20211027 2023-10-30T12:54:43.770 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20211216 -> 20211216 2023-10-30T12:54:43.770 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20220209 -> 20220209 2023-10-30T12:54:43.771 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20220310 -> 20220310 2023-10-30T12:54:43.771 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20220411 -> 20220411 2023-10-30T12:54:43.771 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20220509 -> 20220509 2023-10-30T12:54:43.771 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20220610 -> 20220610 2023-10-30T12:54:43.771 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20220708 -> 20220708 2023-10-30T12:54:43.771 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20220815 -> 20220815 2023-10-30T12:54:43.772 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20220913 -> 20220913 2023-10-30T12:54:43.772 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20221012 -> 20221012 2023-10-30T12:54:43.772 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20221109 -> 20221109 2023-10-30T12:54:43.772 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20221214 -> 20221214 2023-10-30T12:54:43.772 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20230117 -> 20230117 2023-10-30T12:54:43.773 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20230210 -> 20230210 2023-10-30T12:54:43.773 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20230310 -> 20230310 2023-10-30T12:54:43.773 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20230404 -> 20230404 2023-10-30T12:54:43.773 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20230515 -> 20230515 2023-10-30T12:54:43.773 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20230625 -> 20230625 2023-10-30T12:54:43.773 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20230804 -> 20230804 2023-10-30T12:54:43.774 INFO:teuthology.orchestra.run.smithi148.stderr: * [new tag] 20230919 -> 20230919 2023-10-30T12:54:47.454 INFO:teuthology.orchestra.run.smithi148.stderr:Updating files: 52% (1649/3168) Updating files: 53% (1680/3168) Updating files: 54% (1711/3168) Updating files: 55% (1743/3168) Updating files: 56% (1775/3168) Updating files: 57% (1806/3168) Updating files: 58% (1838/3168) Updating files: 59% (1870/3168) Updating files: 60% (1901/3168) Updating files: 61% (1933/3168) Updating files: 62% (1965/3168) Updating files: 63% (1996/3168) Updating files: 64% (2028/3168) Updating files: 65% (2060/3168) Updating files: 66% (2091/3168) Updating files: 66% (2113/3168) Updating files: 67% (2123/3168) Updating files: 68% (2155/3168) Updating files: 69% (2186/3168) Updating files: 70% (2218/3168) Updating files: 71% (2250/3168) Updating files: 72% (2281/3168) Updating files: 73% (2313/3168) Updating files: 74% (2345/3168) Updating files: 75% (2376/3168) Updating files: 76% (2408/3168) Updating files: 77% (2440/3168) Updating files: 78% (2472/3168) Updating files: 79% (2503/3168) Updating files: 80% (2535/3168) Updating files: 81% (2567/3168) Updating files: 81% (2596/3168) Updating files: 82% (2598/3168) Updating files: 83% (2630/3168) Updating files: 84% (2662/3168) Updating files: 85% (2693/3168) Updating files: 86% (2725/3168) Updating files: 87% (2757/3168) Updating files: 88% (2788/3168) Updating files: 89% (2820/3168) Updating files: 90% (2852/3168) Updating files: 91% (2883/3168) Updating files: 92% (2915/3168) Updating files: 93% (2947/3168) Updating files: 94% (2978/3168) Updating files: 95% (3010/3168) Updating files: 96% (3042/3168) Updating files: 97% (3073/3168) Updating files: 98% (3105/3168) Updating files: 99% (3137/3168) Updating files: 100% (3168/3168) Updating files: 100% (3168/3168), done. 2023-10-30T12:54:47.458 INFO:teuthology.orchestra.run.smithi148.stdout:HEAD is now at 2b304bfe Merge branch 'main' into 'main' 2023-10-30T12:54:47.478 INFO:teuthology.task.kernel:Downloading kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T12:54:47.478 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T12:54:47.645 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T12:54:47.645 DEBUG:teuthology.orchestra.run.smithi148:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ --input-file=- 2023-10-30T12:54:47.648 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi148.front.sepia.ceph.com to complete... 2023-10-30T12:54:49.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30 12:54:49 URL:https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb [38909658/38909658] -> "/tmp/linux-image.deb" [1] 2023-10-30T12:54:49.167 INFO:teuthology.orchestra.run.smithi148.stderr:FINISHED --2023-10-30 12:54:49-- 2023-10-30T12:54:49.167 INFO:teuthology.orchestra.run.smithi148.stderr:Total wall clock time: 1.5s 2023-10-30T12:54:49.167 INFO:teuthology.orchestra.run.smithi148.stderr:Downloaded: 1 files, 37M in 1.4s (27.4 MB/s) 2023-10-30T12:54:49.169 INFO:teuthology.task.kernel:Installing kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T12:54:49.170 DEBUG:teuthology.orchestra.run.smithi148:> sudo dpkg -i /tmp/linux-image.deb 2023-10-30T12:54:49.450 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-image-6.6.0-rc5-geacf00cc51e2. 2023-10-30T12:54:51.435 INFO:teuthology.orchestra.run.smithi148.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-10-30T12:54:51.439 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-10-30T12:54:51.541 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T12:54:55.044 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T12:54:55.146 INFO:teuthology.orchestra.run.smithi148.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:15.045 INFO:teuthology.orchestra.run.smithi148.stderr:Sourcing file `/etc/default/grub' 2023-10-30T12:55:15.058 INFO:teuthology.orchestra.run.smithi148.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T12:55:15.088 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2023-10-30T12:55:15.409 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:15.424 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:15.715 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T12:55:15.726 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T12:55:15.792 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T12:55:15.797 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T12:55:15.933 INFO:teuthology.orchestra.run.smithi148.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T12:55:15.934 INFO:teuthology.orchestra.run.smithi148.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T12:55:15.934 INFO:teuthology.orchestra.run.smithi148.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T12:55:15.980 INFO:teuthology.orchestra.run.smithi148.stderr:done 2023-10-30T12:55:16.174 DEBUG:teuthology.orchestra.run.smithi148:> dpkg-deb -c /tmp/linux-image.deb 2023-10-30T12:55:16.181 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./ 2023-10-30T12:55:16.181 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./boot/ 2023-10-30T12:55:16.182 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 4415097 2023-10-30 03:46 ./boot/System.map-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:16.193 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 169233 2023-10-30 03:46 ./boot/config-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:16.193 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11352928 2023-10-30 03:46 ./boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:16.212 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/ 2023-10-30T12:55:16.212 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/ 2023-10-30T12:55:16.212 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postinst.d/ 2023-10-30T12:55:16.213 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postrm.d/ 2023-10-30T12:55:16.213 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/preinst.d/ 2023-10-30T12:55:16.213 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/prerm.d/ 2023-10-30T12:55:16.213 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/ 2023-10-30T12:55:16.213 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/ 2023-10-30T12:55:16.214 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/ 2023-10-30T12:55:16.214 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/ 2023-10-30T12:55:16.214 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/ 2023-10-30T12:55:16.214 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/ 2023-10-30T12:55:16.214 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ 2023-10-30T12:55:16.215 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 490233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/aesni-intel.ko 2023-10-30T12:55:16.217 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-30T12:55:16.217 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-30T12:55:16.217 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-30T12:55:16.218 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 93113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-30T12:55:16.218 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 106729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-30T12:55:16.218 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 152241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-30T12:55:16.218 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-30T12:55:16.218 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-30T12:55:16.219 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 83889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-30T12:55:16.219 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-30T12:55:16.219 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-30T12:55:16.219 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-30T12:55:16.220 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-30T12:55:16.220 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-30T12:55:16.220 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-30T12:55:16.220 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 58585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-30T12:55:16.220 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-30T12:55:16.221 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-30T12:55:16.221 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-30T12:55:16.221 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/ 2023-10-30T12:55:16.221 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/ 2023-10-30T12:55:16.222 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/ 2023-10-30T12:55:16.222 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-30T12:55:16.222 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpuid.ko 2023-10-30T12:55:16.222 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/msr.ko 2023-10-30T12:55:16.222 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/ 2023-10-30T12:55:16.223 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 353761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-30T12:55:16.223 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 881713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-30T12:55:16.223 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 2276817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm.ko 2023-10-30T12:55:16.294 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ 2023-10-30T12:55:16.295 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/af_alg.ko 2023-10-30T12:55:16.295 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_hash.ko 2023-10-30T12:55:16.295 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_skcipher.ko 2023-10-30T12:55:16.295 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ansi_cprng.ko 2023-10-30T12:55:16.296 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/anubis.ko 2023-10-30T12:55:16.296 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/arc4.ko 2023-10-30T12:55:16.296 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/ 2023-10-30T12:55:16.296 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-10-30T12:55:16.296 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/ 2023-10-30T12:55:16.297 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_memcpy.ko 2023-10-30T12:55:16.297 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_pq.ko 2023-10-30T12:55:16.297 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-30T12:55:16.297 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_tx.ko 2023-10-30T12:55:16.298 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_xor.ko 2023-10-30T12:55:16.298 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/raid6test.ko 2023-10-30T12:55:16.298 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authenc.ko 2023-10-30T12:55:16.298 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authencesn.ko 2023-10-30T12:55:16.298 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blake2b_generic.ko 2023-10-30T12:55:16.299 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_common.ko 2023-10-30T12:55:16.299 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_generic.ko 2023-10-30T12:55:16.299 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/camellia_generic.ko 2023-10-30T12:55:16.299 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast5_generic.ko 2023-10-30T12:55:16.299 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast6_generic.ko 2023-10-30T12:55:16.300 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast_common.ko 2023-10-30T12:55:16.300 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ccm.ko 2023-10-30T12:55:16.300 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cmac.ko 2023-10-30T12:55:16.300 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crc32_generic.ko 2023-10-30T12:55:16.300 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cryptd.ko 2023-10-30T12:55:16.301 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_engine.ko 2023-10-30T12:55:16.301 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_simd.ko 2023-10-30T12:55:16.301 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_user.ko 2023-10-30T12:55:16.301 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cts.ko 2023-10-30T12:55:16.302 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/des_generic.ko 2023-10-30T12:55:16.302 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/dh_generic.ko 2023-10-30T12:55:16.302 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/echainiv.ko 2023-10-30T12:55:16.302 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/essiv.ko 2023-10-30T12:55:16.302 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/fcrypt.ko 2023-10-30T12:55:16.303 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/khazad.ko 2023-10-30T12:55:16.303 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lrw.ko 2023-10-30T12:55:16.303 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4.ko 2023-10-30T12:55:16.303 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4hc.ko 2023-10-30T12:55:16.303 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/md4.ko 2023-10-30T12:55:16.304 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/michael_mic.ko 2023-10-30T12:55:16.304 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcbc.ko 2023-10-30T12:55:16.304 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcrypt.ko 2023-10-30T12:55:16.304 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/rmd160.ko 2023-10-30T12:55:16.304 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/seed.ko 2023-10-30T12:55:16.304 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/serpent_generic.ko 2023-10-30T12:55:16.305 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 149193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tcrypt.ko 2023-10-30T12:55:16.305 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tea.ko 2023-10-30T12:55:16.305 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_common.ko 2023-10-30T12:55:16.305 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_generic.ko 2023-10-30T12:55:16.305 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/vmac.ko 2023-10-30T12:55:16.306 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/wp512.ko 2023-10-30T12:55:16.306 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xcbc.ko 2023-10-30T12:55:16.306 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xor.ko 2023-10-30T12:55:16.306 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xts.ko 2023-10-30T12:55:16.306 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xxhash_generic.ko 2023-10-30T12:55:16.307 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ 2023-10-30T12:55:16.307 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ 2023-10-30T12:55:16.307 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_extlog.ko 2023-10-30T12:55:16.307 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-30T12:55:16.307 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_pad.ko 2023-10-30T12:55:16.308 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/ 2023-10-30T12:55:16.308 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/einj.ko 2023-10-30T12:55:16.308 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ec_sys.ko 2023-10-30T12:55:16.308 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/platform_profile.ko 2023-10-30T12:55:16.309 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbs.ko 2023-10-30T12:55:16.309 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbshc.ko 2023-10-30T12:55:16.309 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 103025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/video.ko 2023-10-30T12:55:16.309 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ 2023-10-30T12:55:16.309 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/acard-ahci.ko 2023-10-30T12:55:16.310 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 88865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci.ko 2023-10-30T12:55:16.310 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci_platform.ko 2023-10-30T12:55:16.310 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 88289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci.ko 2023-10-30T12:55:16.310 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci_platform.ko 2023-10-30T12:55:16.310 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_acpi.ko 2023-10-30T12:55:16.311 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ali.ko 2023-10-30T12:55:16.311 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_amd.ko 2023-10-30T12:55:16.311 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_artop.ko 2023-10-30T12:55:16.311 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atiixp.ko 2023-10-30T12:55:16.311 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atp867x.ko 2023-10-30T12:55:16.312 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd640.ko 2023-10-30T12:55:16.312 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd64x.ko 2023-10-30T12:55:16.312 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cypress.ko 2023-10-30T12:55:16.312 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_efar.ko 2023-10-30T12:55:16.312 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt366.ko 2023-10-30T12:55:16.313 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt37x.ko 2023-10-30T12:55:16.313 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-30T12:55:16.313 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-30T12:55:16.313 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it8213.ko 2023-10-30T12:55:16.313 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it821x.ko 2023-10-30T12:55:16.314 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_jmicron.ko 2023-10-30T12:55:16.314 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_legacy.ko 2023-10-30T12:55:16.314 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_marvell.ko 2023-10-30T12:55:16.314 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_mpiix.ko 2023-10-30T12:55:16.314 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_netcell.ko 2023-10-30T12:55:16.315 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ninja32.ko 2023-10-30T12:55:16.315 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87410.ko 2023-10-30T12:55:16.315 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87415.ko 2023-10-30T12:55:16.315 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_oldpiix.ko 2023-10-30T12:55:16.315 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_opti.ko 2023-10-30T12:55:16.316 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_optidma.ko 2023-10-30T12:55:16.316 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pcmcia.ko 2023-10-30T12:55:16.316 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-30T12:55:16.316 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-30T12:55:16.317 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_piccolo.ko 2023-10-30T12:55:16.317 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_radisys.ko 2023-10-30T12:55:16.317 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rdc.ko 2023-10-30T12:55:16.317 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rz1000.ko 2023-10-30T12:55:16.317 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sch.ko 2023-10-30T12:55:16.318 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_serverworks.ko 2023-10-30T12:55:16.318 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sil680.ko 2023-10-30T12:55:16.318 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sl82c105.ko 2023-10-30T12:55:16.318 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_triflex.ko 2023-10-30T12:55:16.318 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_via.ko 2023-10-30T12:55:16.319 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pdc_adma.ko 2023-10-30T12:55:16.319 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_inic162x.ko 2023-10-30T12:55:16.319 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 83345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_mv.ko 2023-10-30T12:55:16.319 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 70393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_nv.ko 2023-10-30T12:55:16.320 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_promise.ko 2023-10-30T12:55:16.320 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_qstor.ko 2023-10-30T12:55:16.320 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil.ko 2023-10-30T12:55:16.320 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil24.ko 2023-10-30T12:55:16.320 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sis.ko 2023-10-30T12:55:16.321 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_svw.ko 2023-10-30T12:55:16.321 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sx4.ko 2023-10-30T12:55:16.321 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_uli.ko 2023-10-30T12:55:16.321 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_via.ko 2023-10-30T12:55:16.321 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_vsc.ko 2023-10-30T12:55:16.322 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ 2023-10-30T12:55:16.322 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ks0108.ko 2023-10-30T12:55:16.322 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/ 2023-10-30T12:55:16.322 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 124473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/bcma.ko 2023-10-30T12:55:16.322 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/ 2023-10-30T12:55:16.322 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/ 2023-10-30T12:55:16.323 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 113665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/aoe.ko 2023-10-30T12:55:16.323 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/ 2023-10-30T12:55:16.323 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 965585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/drbd.ko 2023-10-30T12:55:16.323 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 173225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/floppy.ko 2023-10-30T12:55:16.324 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/ 2023-10-30T12:55:16.324 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 94297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-30T12:55:16.324 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 115793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/nbd.ko 2023-10-30T12:55:16.324 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/ 2023-10-30T12:55:16.324 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 110449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/null_blk.ko 2023-10-30T12:55:16.324 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 98193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/pktcdvd.ko 2023-10-30T12:55:16.325 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 263769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/rbd.ko 2023-10-30T12:55:16.325 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/ 2023-10-30T12:55:16.325 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 98145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-30T12:55:16.325 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/ 2023-10-30T12:55:16.325 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/zram.ko 2023-10-30T12:55:16.326 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ 2023-10-30T12:55:16.326 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/applicom.ko 2023-10-30T12:55:16.326 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hangcheck-timer.ko 2023-10-30T12:55:16.326 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/ 2023-10-30T12:55:16.326 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-30T12:55:16.326 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-30T12:55:16.327 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-30T12:55:16.327 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/via-rng.ko 2023-10-30T12:55:16.327 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-30T12:55:16.327 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ 2023-10-30T12:55:16.327 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-30T12:55:16.328 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 148601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-30T12:55:16.328 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-30T12:55:16.328 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 139313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-30T12:55:16.328 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 70921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-30T12:55:16.328 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-30T12:55:16.329 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/lp.ko 2023-10-30T12:55:16.329 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/ 2023-10-30T12:55:16.329 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 84097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/mwave.ko 2023-10-30T12:55:16.329 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/nvram.ko 2023-10-30T12:55:16.329 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ppdev.ko 2023-10-30T12:55:16.330 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tlclk.ko 2023-10-30T12:55:16.330 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/ 2023-10-30T12:55:16.330 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-30T12:55:16.330 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-10-30T12:55:16.330 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-10-30T12:55:16.331 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-10-30T12:55:16.331 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-30T12:55:16.331 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-30T12:55:16.331 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-10-30T12:55:16.331 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/ 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_class.ko 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_core.ko 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/ 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-palmas.ko 2023-10-30T12:55:16.333 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-si5351.ko 2023-10-30T12:55:16.333 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-twl6040.ko 2023-10-30T12:55:16.333 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-wm831x.ko 2023-10-30T12:55:16.333 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/ 2023-10-30T12:55:16.333 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-30T12:55:16.333 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-30T12:55:16.334 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-30T12:55:16.334 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ 2023-10-30T12:55:16.334 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ 2023-10-30T12:55:16.334 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 70801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-30T12:55:16.335 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 243025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp.ko 2023-10-30T12:55:16.335 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/ 2023-10-30T12:55:16.335 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/ 2023-10-30T12:55:16.335 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/ 2023-10-30T12:55:16.335 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 438761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-30T12:55:16.336 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-10-30T12:55:16.336 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-30T12:55:16.336 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-aes.ko 2023-10-30T12:55:16.336 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-sha.ko 2023-10-30T12:55:16.336 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/ 2023-10-30T12:55:16.337 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 68857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-30T12:55:16.337 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/ 2023-10-30T12:55:16.337 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/dca.ko 2023-10-30T12:55:16.338 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ 2023-10-30T12:55:16.338 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/ 2023-10-30T12:55:16.338 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-30T12:55:16.338 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ 2023-10-30T12:55:16.338 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 122569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-30T12:55:16.339 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ 2023-10-30T12:55:16.339 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 79953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/amd64_edac.ko 2023-10-30T12:55:16.339 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/e752x_edac.ko 2023-10-30T12:55:16.339 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 63617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/edac_mce_amd.ko 2023-10-30T12:55:16.339 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3000_edac.ko 2023-10-30T12:55:16.340 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3200_edac.ko 2023-10-30T12:55:16.340 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5100_edac.ko 2023-10-30T12:55:16.340 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5400_edac.ko 2023-10-30T12:55:16.340 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7300_edac.ko 2023-10-30T12:55:16.341 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 52041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7core_edac.ko 2023-10-30T12:55:16.341 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i82975x_edac.ko 2023-10-30T12:55:16.341 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ie31200_edac.ko 2023-10-30T12:55:16.341 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 56825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/sb_edac.ko 2023-10-30T12:55:16.341 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/x38_edac.ko 2023-10-30T12:55:16.341 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/ 2023-10-30T12:55:16.342 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-gpio.ko 2023-10-30T12:55:16.342 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max14577.ko 2023-10-30T12:55:16.342 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max77693.ko 2023-10-30T12:55:16.342 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max8997.ko 2023-10-30T12:55:16.343 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-palmas.ko 2023-10-30T12:55:16.343 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-rt8973a.ko 2023-10-30T12:55:16.343 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-sm5502.ko 2023-10-30T12:55:16.343 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/ 2023-10-30T12:55:16.343 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/dmi-sysfs.ko 2023-10-30T12:55:16.344 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/ 2023-10-30T12:55:16.344 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/efi-pstore.ko 2023-10-30T12:55:16.344 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-30T12:55:16.344 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/ 2023-10-30T12:55:16.345 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-adp5520.ko 2023-10-30T12:55:16.345 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-amd8111.ko 2023-10-30T12:55:16.345 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-arizona.ko 2023-10-30T12:55:16.345 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9052.ko 2023-10-30T12:55:16.345 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9055.ko 2023-10-30T12:55:16.346 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-dln2.ko 2023-10-30T12:55:16.346 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 77849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-f7188x.ko 2023-10-30T12:55:16.346 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-generic.ko 2023-10-30T12:55:16.346 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ich.ko 2023-10-30T12:55:16.346 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-10-30T12:55:16.346 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-kempld.ko 2023-10-30T12:55:16.347 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-lp3943.ko 2023-10-30T12:55:16.347 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max7300.ko 2023-10-30T12:55:16.347 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max730x.ko 2023-10-30T12:55:16.347 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max732x.ko 2023-10-30T12:55:16.347 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-10-30T12:55:16.348 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pca953x.ko 2023-10-30T12:55:16.348 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pcf857x.ko 2023-10-30T12:55:16.348 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-rdc321x.ko 2023-10-30T12:55:16.348 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch.ko 2023-10-30T12:55:16.349 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch311x.ko 2023-10-30T12:55:16.349 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-tps65912.ko 2023-10-30T12:55:16.349 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl4030.ko 2023-10-30T12:55:16.349 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl6040.ko 2023-10-30T12:55:16.349 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-30T12:55:16.349 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-vx855.ko 2023-10-30T12:55:16.350 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm831x.ko 2023-10-30T12:55:16.350 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8350.ko 2023-10-30T12:55:16.350 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8994.ko 2023-10-30T12:55:16.350 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/ 2023-10-30T12:55:16.351 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-a4tech.ko 2023-10-30T12:55:16.351 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-apple.ko 2023-10-30T12:55:16.351 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-appleir.ko 2023-10-30T12:55:16.351 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-aureal.ko 2023-10-30T12:55:16.352 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-axff.ko 2023-10-30T12:55:16.352 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-belkin.ko 2023-10-30T12:55:16.352 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cherry.ko 2023-10-30T12:55:16.352 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-chicony.ko 2023-10-30T12:55:16.352 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cp2112.ko 2023-10-30T12:55:16.353 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cypress.ko 2023-10-30T12:55:16.353 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-dr.ko 2023-10-30T12:55:16.353 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elecom.ko 2023-10-30T12:55:16.353 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elo.ko 2023-10-30T12:55:16.353 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-emsff.ko 2023-10-30T12:55:16.354 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ezkey.ko 2023-10-30T12:55:16.354 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gaff.ko 2023-10-30T12:55:16.354 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-generic.ko 2023-10-30T12:55:16.354 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gt683r.ko 2023-10-30T12:55:16.354 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gyration.ko 2023-10-30T12:55:16.355 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-30T12:55:16.355 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-30T12:55:16.355 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtekff.ko 2023-10-30T12:55:16.355 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-hyperv.ko 2023-10-30T12:55:16.355 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-icade.ko 2023-10-30T12:55:16.356 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kensington.ko 2023-10-30T12:55:16.356 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-keytouch.ko 2023-10-30T12:55:16.356 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kye.ko 2023-10-30T12:55:16.356 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lcpower.ko 2023-10-30T12:55:16.356 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-led.ko 2023-10-30T12:55:16.357 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lenovo.ko 2023-10-30T12:55:16.357 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lg-g15.ko 2023-10-30T12:55:16.357 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 58185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-30T12:55:16.357 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 101369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-30T12:55:16.357 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 80993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech.ko 2023-10-30T12:55:16.358 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-magicmouse.ko 2023-10-30T12:55:16.358 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-microsoft.ko 2023-10-30T12:55:16.358 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-monterey.ko 2023-10-30T12:55:16.359 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-multitouch.ko 2023-10-30T12:55:16.359 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ntrig.ko 2023-10-30T12:55:16.359 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ortek.ko 2023-10-30T12:55:16.359 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-penmount.ko 2023-10-30T12:55:16.359 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-petalynx.ko 2023-10-30T12:55:16.360 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 72937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-picolcd.ko 2023-10-30T12:55:16.360 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-pl.ko 2023-10-30T12:55:16.360 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-plantronics.ko 2023-10-30T12:55:16.360 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-primax.ko 2023-10-30T12:55:16.360 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-rmi.ko 2023-10-30T12:55:16.361 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-30T12:55:16.361 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-common.ko 2023-10-30T12:55:16.361 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-30T12:55:16.361 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-30T12:55:16.361 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-30T12:55:16.362 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-30T12:55:16.362 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-30T12:55:16.362 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-30T12:55:16.362 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-30T12:55:16.362 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-30T12:55:16.363 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-30T12:55:16.363 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat.ko 2023-10-30T12:55:16.363 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-saitek.ko 2023-10-30T12:55:16.364 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-samsung.ko 2023-10-30T12:55:16.364 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-30T12:55:16.364 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sjoy.ko 2023-10-30T12:55:16.364 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sony.ko 2023-10-30T12:55:16.364 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-speedlink.ko 2023-10-30T12:55:16.365 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-steelseries.ko 2023-10-30T12:55:16.365 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sunplus.ko 2023-10-30T12:55:16.365 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-30T12:55:16.365 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tivo.ko 2023-10-30T12:55:16.365 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tmff.ko 2023-10-30T12:55:16.366 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-topseed.ko 2023-10-30T12:55:16.366 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-twinhan.ko 2023-10-30T12:55:16.366 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 71073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-uclogic.ko 2023-10-30T12:55:16.366 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-waltop.ko 2023-10-30T12:55:16.366 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 143473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-wiimote.ko 2023-10-30T12:55:16.367 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-xinmo.ko 2023-10-30T12:55:16.367 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zpff.ko 2023-10-30T12:55:16.367 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zydacron.ko 2023-10-30T12:55:16.367 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 267825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid.ko 2023-10-30T12:55:16.368 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/uhid.ko 2023-10-30T12:55:16.368 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/ 2023-10-30T12:55:16.368 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 125721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbhid.ko 2023-10-30T12:55:16.368 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbkbd.ko 2023-10-30T12:55:16.368 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbmouse.ko 2023-10-30T12:55:16.369 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 217873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/wacom.ko 2023-10-30T12:55:16.369 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/ 2023-10-30T12:55:16.369 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 64201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_balloon.ko 2023-10-30T12:55:16.369 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 89625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_utils.ko 2023-10-30T12:55:16.369 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 305465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_vmbus.ko 2023-10-30T12:55:16.370 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ 2023-10-30T12:55:16.370 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru.ko 2023-10-30T12:55:16.370 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 61393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru3.ko 2023-10-30T12:55:16.370 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-30T12:55:16.370 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7414.ko 2023-10-30T12:55:16.371 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7418.ko 2023-10-30T12:55:16.371 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adc128d818.ko 2023-10-30T12:55:16.371 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1025.ko 2023-10-30T12:55:16.371 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 97977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1026.ko 2023-10-30T12:55:16.371 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1029.ko 2023-10-30T12:55:16.372 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 45553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1031.ko 2023-10-30T12:55:16.372 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm9240.ko 2023-10-30T12:55:16.372 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ads7828.ko 2023-10-30T12:55:16.372 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7410.ko 2023-10-30T12:55:16.372 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7411.ko 2023-10-30T12:55:16.373 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 74433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7462.ko 2023-10-30T12:55:16.373 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7470.ko 2023-10-30T12:55:16.373 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7475.ko 2023-10-30T12:55:16.373 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7x10.ko 2023-10-30T12:55:16.373 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/amc6821.ko 2023-10-30T12:55:16.374 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 55265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/applesmc.ko 2023-10-30T12:55:16.374 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asb100.ko 2023-10-30T12:55:16.374 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 58617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asc7621.ko 2023-10-30T12:55:16.374 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-30T12:55:16.374 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/atxp1.ko 2023-10-30T12:55:16.375 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/coretemp.ko 2023-10-30T12:55:16.375 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9052-hwmon.ko 2023-10-30T12:55:16.375 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9055-hwmon.ko 2023-10-30T12:55:16.375 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-30T12:55:16.375 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 99601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dme1737.ko 2023-10-30T12:55:16.376 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds1621.ko 2023-10-30T12:55:16.376 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds620.ko 2023-10-30T12:55:16.376 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc1403.ko 2023-10-30T12:55:16.376 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc2103.ko 2023-10-30T12:55:16.376 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc6w201.ko 2023-10-30T12:55:16.377 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71805f.ko 2023-10-30T12:55:16.377 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 94457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71882fg.ko 2023-10-30T12:55:16.377 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f75375s.ko 2023-10-30T12:55:16.377 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fam15h_power.ko 2023-10-30T12:55:16.377 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 55449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fschmd.ko 2023-10-30T12:55:16.378 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g760a.ko 2023-10-30T12:55:16.378 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g762.ko 2023-10-30T12:55:16.378 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 46801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl518sm.ko 2023-10-30T12:55:16.378 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 45529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl520sm.ko 2023-10-30T12:55:16.378 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hih6130.ko 2023-10-30T12:55:16.379 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-30T12:55:16.379 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5500_temp.ko 2023-10-30T12:55:16.379 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5k_amb.ko 2023-10-30T12:55:16.379 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmaem.ko 2023-10-30T12:55:16.379 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmpex.ko 2023-10-30T12:55:16.380 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina209.ko 2023-10-30T12:55:16.380 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina2xx.ko 2023-10-30T12:55:16.380 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 130449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/it87.ko 2023-10-30T12:55:16.380 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/jc42.ko 2023-10-30T12:55:16.380 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k10temp.ko 2023-10-30T12:55:16.380 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k8temp.ko 2023-10-30T12:55:16.381 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lineage-pem.ko 2023-10-30T12:55:16.381 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 48601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm63.ko 2023-10-30T12:55:16.381 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm73.ko 2023-10-30T12:55:16.381 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm75.ko 2023-10-30T12:55:16.381 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm77.ko 2023-10-30T12:55:16.382 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm78.ko 2023-10-30T12:55:16.382 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm80.ko 2023-10-30T12:55:16.382 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm83.ko 2023-10-30T12:55:16.382 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 72833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm85.ko 2023-10-30T12:55:16.382 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 46153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm87.ko 2023-10-30T12:55:16.383 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm90.ko 2023-10-30T12:55:16.383 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm92.ko 2023-10-30T12:55:16.385 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 108841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm93.ko 2023-10-30T12:55:16.385 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95234.ko 2023-10-30T12:55:16.385 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95241.ko 2023-10-30T12:55:16.386 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95245.ko 2023-10-30T12:55:16.386 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc2945.ko 2023-10-30T12:55:16.386 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4151.ko 2023-10-30T12:55:16.386 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4215.ko 2023-10-30T12:55:16.386 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4222.ko 2023-10-30T12:55:16.387 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4245.ko 2023-10-30T12:55:16.387 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4260.ko 2023-10-30T12:55:16.387 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4261.ko 2023-10-30T12:55:16.387 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max16065.ko 2023-10-30T12:55:16.387 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1619.ko 2023-10-30T12:55:16.387 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1668.ko 2023-10-30T12:55:16.388 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max197.ko 2023-10-30T12:55:16.388 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6639.ko 2023-10-30T12:55:16.388 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6650.ko 2023-10-30T12:55:16.388 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6697.ko 2023-10-30T12:55:16.389 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mc13783-adc.ko 2023-10-30T12:55:16.389 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mcp3021.ko 2023-10-30T12:55:16.389 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-10-30T12:55:16.389 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6683.ko 2023-10-30T12:55:16.389 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 119817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775-core.ko 2023-10-30T12:55:16.390 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775.ko 2023-10-30T12:55:16.390 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 59737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct7802.ko 2023-10-30T12:55:16.390 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 82337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87360.ko 2023-10-30T12:55:16.390 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87427.ko 2023-10-30T12:55:16.390 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pcf8591.ko 2023-10-30T12:55:16.390 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ 2023-10-30T12:55:16.391 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-30T12:55:16.391 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-30T12:55:16.391 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-30T12:55:16.391 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-30T12:55:16.391 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-30T12:55:16.392 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-30T12:55:16.392 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-30T12:55:16.392 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 82577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-30T12:55:16.392 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-30T12:55:16.392 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-30T12:55:16.392 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-30T12:55:16.393 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-30T12:55:16.393 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/powr1220.ko 2023-10-30T12:55:16.393 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5627.ko 2023-10-30T12:55:16.393 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5636.ko 2023-10-30T12:55:16.393 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-30T12:55:16.394 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht15.ko 2023-10-30T12:55:16.394 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht21.ko 2023-10-30T12:55:16.394 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/shtc1.ko 2023-10-30T12:55:16.394 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sis5595.ko 2023-10-30T12:55:16.394 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47b397.ko 2023-10-30T12:55:16.395 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m1.ko 2023-10-30T12:55:16.395 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m192.ko 2023-10-30T12:55:16.395 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/thmc50.ko 2023-10-30T12:55:16.395 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp102.ko 2023-10-30T12:55:16.395 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp103.ko 2023-10-30T12:55:16.396 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp401.ko 2023-10-30T12:55:16.396 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp421.ko 2023-10-30T12:55:16.396 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via-cputemp.ko 2023-10-30T12:55:16.396 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via686a.ko 2023-10-30T12:55:16.397 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 52457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt1211.ko 2023-10-30T12:55:16.397 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt8231.ko 2023-10-30T12:55:16.397 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 57513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627ehf.ko 2023-10-30T12:55:16.397 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627hf.ko 2023-10-30T12:55:16.397 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 71345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83781d.ko 2023-10-30T12:55:16.398 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 63289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83791d.ko 2023-10-30T12:55:16.398 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 71321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83792d.ko 2023-10-30T12:55:16.398 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 101473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83793.ko 2023-10-30T12:55:16.398 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 119505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83795.ko 2023-10-30T12:55:16.398 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l785ts.ko 2023-10-30T12:55:16.399 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l786ng.ko 2023-10-30T12:55:16.399 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-10-30T12:55:16.399 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-10-30T12:55:16.399 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/ 2023-10-30T12:55:16.399 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/ 2023-10-30T12:55:16.400 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-30T12:55:16.400 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-30T12:55:16.400 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/ 2023-10-30T12:55:16.400 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-10-30T12:55:16.400 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-10-30T12:55:16.401 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-10-30T12:55:16.401 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-30T12:55:16.401 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-30T12:55:16.401 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-30T12:55:16.401 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-10-30T12:55:16.402 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 6929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-30T12:55:16.402 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-10-30T12:55:16.402 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-30T12:55:16.402 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-30T12:55:16.402 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-30T12:55:16.403 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-30T12:55:16.403 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-10-30T12:55:16.403 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-10-30T12:55:16.403 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-30T12:55:16.403 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-30T12:55:16.404 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-30T12:55:16.404 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-10-30T12:55:16.404 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-30T12:55:16.404 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-30T12:55:16.404 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-10-30T12:55:16.404 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-30T12:55:16.405 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-30T12:55:16.405 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-30T12:55:16.405 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-10-30T12:55:16.405 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-30T12:55:16.405 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-30T12:55:16.406 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-10-30T12:55:16.406 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-10-30T12:55:16.406 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-30T12:55:16.406 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-10-30T12:55:16.406 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-30T12:55:16.407 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-30T12:55:16.407 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-30T12:55:16.407 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-30T12:55:16.407 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-10-30T12:55:16.407 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-mux.ko 2023-10-30T12:55:16.408 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-smbus.ko 2023-10-30T12:55:16.408 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-stub.ko 2023-10-30T12:55:16.408 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/ 2023-10-30T12:55:16.408 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-10-30T12:55:16.409 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-10-30T12:55:16.409 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-10-30T12:55:16.409 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ 2023-10-30T12:55:16.409 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ 2023-10-30T12:55:16.409 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 273609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-30T12:55:16.409 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 879969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_core.ko 2023-10-30T12:55:16.410 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 69337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-30T12:55:16.410 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 324217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-30T12:55:16.410 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 97985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-30T12:55:16.410 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 282273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-30T12:55:16.411 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-30T12:55:16.411 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ 2023-10-30T12:55:16.411 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/ 2023-10-30T12:55:16.411 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 493073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-30T12:55:16.411 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/ 2023-10-30T12:55:16.411 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 419737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-30T12:55:16.412 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/ 2023-10-30T12:55:16.412 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 739025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-30T12:55:16.412 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ 2023-10-30T12:55:16.412 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 296625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-30T12:55:16.412 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ 2023-10-30T12:55:16.413 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 153697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-10-30T12:55:16.413 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/ 2023-10-30T12:55:16.413 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 168969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-10-30T12:55:16.413 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ 2023-10-30T12:55:16.413 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ 2023-10-30T12:55:16.414 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 263081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-30T12:55:16.414 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ 2023-10-30T12:55:16.414 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 100241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-30T12:55:16.414 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ 2023-10-30T12:55:16.414 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 111425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-30T12:55:16.415 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ 2023-10-30T12:55:16.415 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 128753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-30T12:55:16.415 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ 2023-10-30T12:55:16.415 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 129529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-30T12:55:16.415 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ 2023-10-30T12:55:16.415 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/evbug.ko 2023-10-30T12:55:16.416 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ff-memless.ko 2023-10-30T12:55:16.416 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/ 2023-10-30T12:55:16.416 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-30T12:55:16.416 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-30T12:55:16.416 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/gameport.ko 2023-10-30T12:55:16.417 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/joydev.ko 2023-10-30T12:55:16.417 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/ 2023-10-30T12:55:16.417 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-10-30T12:55:16.417 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-10-30T12:55:16.417 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-10-30T12:55:16.418 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-10-30T12:55:16.418 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-30T12:55:16.418 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-10-30T12:55:16.418 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lkkbd.ko 2023-10-30T12:55:16.418 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8323.ko 2023-10-30T12:55:16.419 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8333.ko 2023-10-30T12:55:16.419 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-10-30T12:55:16.419 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-10-30T12:55:16.419 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-10-30T12:55:16.419 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-10-30T12:55:16.420 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/newtonkbd.ko 2023-10-30T12:55:16.420 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-10-30T12:55:16.420 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt1070.ko 2023-10-30T12:55:16.420 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt2160.ko 2023-10-30T12:55:16.420 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-10-30T12:55:16.421 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/stowaway.ko 2023-10-30T12:55:16.421 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/sunkbd.ko 2023-10-30T12:55:16.421 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-10-30T12:55:16.421 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-10-30T12:55:16.421 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-10-30T12:55:16.422 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/xtkbd.ko 2023-10-30T12:55:16.422 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/matrix-keymap.ko 2023-10-30T12:55:16.422 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/ 2023-10-30T12:55:16.422 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/appletouch.ko 2023-10-30T12:55:16.422 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/bcm5974.ko 2023-10-30T12:55:16.423 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 123681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/cyapatp.ko 2023-10-30T12:55:16.423 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 89449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-30T12:55:16.423 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/gpio_mouse.ko 2023-10-30T12:55:16.423 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 335977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/psmouse.ko 2023-10-30T12:55:16.423 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/sermouse.ko 2023-10-30T12:55:16.423 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-30T12:55:16.424 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-30T12:55:16.424 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-30T12:55:16.424 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/ 2023-10-30T12:55:16.424 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 109873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-30T12:55:16.424 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ 2023-10-30T12:55:16.424 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/altera_ps2.ko 2023-10-30T12:55:16.425 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/arc_ps2.ko 2023-10-30T12:55:16.425 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ct82c710.ko 2023-10-30T12:55:16.425 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-30T12:55:16.425 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/parkbd.ko 2023-10-30T12:55:16.425 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/pcips2.ko 2023-10-30T12:55:16.426 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ps2mult.ko 2023-10-30T12:55:16.426 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serio_raw.ko 2023-10-30T12:55:16.426 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serport.ko 2023-10-30T12:55:16.426 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/sparse-keymap.ko 2023-10-30T12:55:16.426 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/ 2023-10-30T12:55:16.427 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/ 2023-10-30T12:55:16.427 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-30T12:55:16.427 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/ 2023-10-30T12:55:16.427 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-88pm860x.ko 2023-10-30T12:55:16.427 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-adp5520.ko 2023-10-30T12:55:16.427 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 59753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-bd2802.ko 2023-10-30T12:55:16.428 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-blinkm.ko 2023-10-30T12:55:16.428 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da903x.ko 2023-10-30T12:55:16.428 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da9052.ko 2023-10-30T12:55:16.428 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-gpio.ko 2023-10-30T12:55:16.428 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3530.ko 2023-10-30T12:55:16.429 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3533.ko 2023-10-30T12:55:16.429 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm355x.ko 2023-10-30T12:55:16.429 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3642.ko 2023-10-30T12:55:16.429 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp3944.ko 2023-10-30T12:55:16.429 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp8788.ko 2023-10-30T12:55:16.430 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-max8997.ko 2023-10-30T12:55:16.430 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-mc13783.ko 2023-10-30T12:55:16.430 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-menf21bmc.ko 2023-10-30T12:55:16.430 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca9532.ko 2023-10-30T12:55:16.430 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca955x.ko 2023-10-30T12:55:16.431 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca963x.ko 2023-10-30T12:55:16.431 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-ss4200.ko 2023-10-30T12:55:16.431 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-tca6507.ko 2023-10-30T12:55:16.431 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm831x-status.ko 2023-10-30T12:55:16.431 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm8350.ko 2023-10-30T12:55:16.431 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ 2023-10-30T12:55:16.432 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-30T12:55:16.432 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-30T12:55:16.432 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-30T12:55:16.432 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-30T12:55:16.432 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-30T12:55:16.433 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-30T12:55:16.433 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-30T12:55:16.433 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-30T12:55:16.433 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/ 2023-10-30T12:55:16.433 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/ 2023-10-30T12:55:16.433 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 654969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/bcache.ko 2023-10-30T12:55:16.434 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bio-prison.ko 2023-10-30T12:55:16.434 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 89601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bufio.ko 2023-10-30T12:55:16.434 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache-smq.ko 2023-10-30T12:55:16.434 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 156009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache.ko 2023-10-30T12:55:16.434 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 106897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-crypt.ko 2023-10-30T12:55:16.435 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-delay.ko 2023-10-30T12:55:16.435 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-era.ko 2023-10-30T12:55:16.435 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-flakey.ko 2023-10-30T12:55:16.435 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log-userspace.ko 2023-10-30T12:55:16.435 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log.ko 2023-10-30T12:55:16.435 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-mirror.ko 2023-10-30T12:55:16.436 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 78033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-multipath.ko 2023-10-30T12:55:16.436 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-queue-length.ko 2023-10-30T12:55:16.436 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 70825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-raid.ko 2023-10-30T12:55:16.436 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-region-hash.ko 2023-10-30T12:55:16.436 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-round-robin.ko 2023-10-30T12:55:16.437 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-service-time.ko 2023-10-30T12:55:16.437 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-snapshot.ko 2023-10-30T12:55:16.437 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-switch.ko 2023-10-30T12:55:16.437 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 181865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-thin-pool.ko 2023-10-30T12:55:16.437 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-verity.ko 2023-10-30T12:55:16.438 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-zero.ko 2023-10-30T12:55:16.438 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/faulty.ko 2023-10-30T12:55:16.438 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/linear.ko 2023-10-30T12:55:16.438 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/multipath.ko 2023-10-30T12:55:16.438 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/ 2023-10-30T12:55:16.439 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 199937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-30T12:55:16.439 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid0.ko 2023-10-30T12:55:16.439 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 113073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid1.ko 2023-10-30T12:55:16.439 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 158681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid10.ko 2023-10-30T12:55:16.439 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 395721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid456.ko 2023-10-30T12:55:16.440 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/ 2023-10-30T12:55:16.440 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/ 2023-10-30T12:55:16.440 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 178137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptbase.ko 2023-10-30T12:55:16.440 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 63385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptctl.ko 2023-10-30T12:55:16.440 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptfc.ko 2023-10-30T12:55:16.440 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptlan.ko 2023-10-30T12:55:16.441 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 123321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptsas.ko 2023-10-30T12:55:16.441 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 79041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptscsih.ko 2023-10-30T12:55:16.441 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptspi.ko 2023-10-30T12:55:16.441 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ 2023-10-30T12:55:16.441 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm800.ko 2023-10-30T12:55:16.442 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm805.ko 2023-10-30T12:55:16.442 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm80x.ko 2023-10-30T12:55:16.442 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona-i2c.ko 2023-10-30T12:55:16.442 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 123721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona.ko 2023-10-30T12:55:16.442 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/bcm590xx.ko 2023-10-30T12:55:16.442 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/cros_ec_dev.ko 2023-10-30T12:55:16.443 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/dln2.ko 2023-10-30T12:55:16.443 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/janz-cmodio.ko 2023-10-30T12:55:16.443 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/kempld-core.ko 2023-10-30T12:55:16.443 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-core.ko 2023-10-30T12:55:16.443 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-10-30T12:55:16.444 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lp3943.ko 2023-10-30T12:55:16.444 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_ich.ko 2023-10-30T12:55:16.444 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_sch.ko 2023-10-30T12:55:16.444 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/max8907.ko 2023-10-30T12:55:16.444 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-core.ko 2023-10-30T12:55:16.444 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-10-30T12:55:16.445 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/menf21bmc.ko 2023-10-30T12:55:16.445 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-adc.ko 2023-10-30T12:55:16.445 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-gpio.ko 2023-10-30T12:55:16.445 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633.ko 2023-10-30T12:55:16.445 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-10-30T12:55:16.446 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/retu-mfd.ko 2023-10-30T12:55:16.446 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 80425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/si476x-core.ko 2023-10-30T12:55:16.446 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 48457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/sm501.ko 2023-10-30T12:55:16.446 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-10-30T12:55:16.447 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps65010.ko 2023-10-30T12:55:16.447 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps6507x.ko 2023-10-30T12:55:16.447 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/viperboard.ko 2023-10-30T12:55:16.447 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/vx855.ko 2023-10-30T12:55:16.447 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wl1273-core.ko 2023-10-30T12:55:16.447 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wm8994.ko 2023-10-30T12:55:16.448 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ 2023-10-30T12:55:16.448 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-10-30T12:55:16.448 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 48489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot.ko 2023-10-30T12:55:16.448 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/ 2023-10-30T12:55:16.448 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 46585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-30T12:55:16.449 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds9802als.ko 2023-10-30T12:55:16.449 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds990x.ko 2023-10-30T12:55:16.449 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 45593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/bh1770glc.ko 2023-10-30T12:55:16.449 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/ 2023-10-30T12:55:16.449 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-10-30T12:55:16.449 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/core.ko 2023-10-30T12:55:16.450 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/ 2023-10-30T12:55:16.450 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/cb710.ko 2023-10-30T12:55:16.450 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ds1682.ko 2023-10-30T12:55:16.450 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/dummy-irq.ko 2023-10-30T12:55:16.450 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/ 2023-10-30T12:55:16.451 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/echo.ko 2023-10-30T12:55:16.451 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/ 2023-10-30T12:55:16.451 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/at24.ko 2023-10-30T12:55:16.451 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-30T12:55:16.451 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-30T12:55:16.451 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/max6875.ko 2023-10-30T12:55:16.452 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/enclosure.ko 2023-10-30T12:55:16.452 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/ 2023-10-30T12:55:16.452 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 152201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-10-30T12:55:16.452 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hmc6352.ko 2023-10-30T12:55:16.452 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hpilo.ko 2023-10-30T12:55:16.452 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ 2023-10-30T12:55:16.453 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 72129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-10-30T12:55:16.453 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ics932s401.ko 2023-10-30T12:55:16.453 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29003.ko 2023-10-30T12:55:16.453 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29020.ko 2023-10-30T12:55:16.453 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/ 2023-10-30T12:55:16.454 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-30T12:55:16.454 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-30T12:55:16.454 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/ 2023-10-30T12:55:16.454 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 125209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-me.ko 2023-10-30T12:55:16.454 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 56521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-txe.ko 2023-10-30T12:55:16.455 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 310113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei.ko 2023-10-30T12:55:16.455 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/phantom.ko 2023-10-30T12:55:16.455 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/ 2023-10-30T12:55:16.455 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 74265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/st_drv.ko 2023-10-30T12:55:16.455 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_7xx1.ko 2023-10-30T12:55:16.456 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_core.ko 2023-10-30T12:55:16.456 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tsl2550.ko 2023-10-30T12:55:16.456 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 48697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_balloon.ko 2023-10-30T12:55:16.456 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/ 2023-10-30T12:55:16.456 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 183609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-30T12:55:16.456 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ 2023-10-30T12:55:16.457 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/ 2023-10-30T12:55:16.457 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 490129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/bonding.ko 2023-10-30T12:55:16.457 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/ 2023-10-30T12:55:16.457 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/ 2023-10-30T12:55:16.458 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 106033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-30T12:55:16.458 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-30T12:55:16.458 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-30T12:55:16.458 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/ 2023-10-30T12:55:16.458 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 453737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-30T12:55:16.459 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dummy.ko 2023-10-30T12:55:16.459 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/eql.ko 2023-10-30T12:55:16.459 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ 2023-10-30T12:55:16.459 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/ 2023-10-30T12:55:16.459 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 45785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-30T12:55:16.460 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-30T12:55:16.460 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 130553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-30T12:55:16.460 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 68849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-30T12:55:16.460 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ 2023-10-30T12:55:16.460 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-30T12:55:16.461 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 59689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-30T12:55:16.461 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-30T12:55:16.461 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 99113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-30T12:55:16.461 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/ 2023-10-30T12:55:16.461 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-30T12:55:16.462 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/ 2023-10-30T12:55:16.462 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 69201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-30T12:55:16.462 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/ 2023-10-30T12:55:16.462 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 59585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-30T12:55:16.462 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/ 2023-10-30T12:55:16.463 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-30T12:55:16.463 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/ 2023-10-30T12:55:16.463 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-30T12:55:16.463 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-30T12:55:16.463 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 87297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-30T12:55:16.463 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/ 2023-10-30T12:55:16.464 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/ 2023-10-30T12:55:16.464 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 97337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-30T12:55:16.464 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-10-30T12:55:16.464 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 104673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-30T12:55:16.464 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-10-30T12:55:16.465 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 90257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-30T12:55:16.465 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/ 2023-10-30T12:55:16.465 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 81569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-30T12:55:16.465 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 69729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-30T12:55:16.465 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/ 2023-10-30T12:55:16.465 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 78401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-30T12:55:16.466 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 176761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-30T12:55:16.466 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-10-30T12:55:16.466 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1421497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-30T12:55:16.466 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 128569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-30T12:55:16.466 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/ 2023-10-30T12:55:16.467 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 132089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-30T12:55:16.467 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 312081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-30T12:55:16.467 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/ 2023-10-30T12:55:16.467 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/ 2023-10-30T12:55:16.467 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 323513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-30T12:55:16.467 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/ 2023-10-30T12:55:16.468 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-10-30T12:55:16.468 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 193497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-30T12:55:16.468 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-10-30T12:55:16.468 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 347417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-30T12:55:16.468 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-10-30T12:55:16.469 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1044433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-30T12:55:16.469 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-10-30T12:55:16.469 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 166193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-30T12:55:16.469 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-10-30T12:55:16.469 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-30T12:55:16.470 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/ 2023-10-30T12:55:16.470 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/ 2023-10-30T12:55:16.470 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 204345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-30T12:55:16.470 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/ 2023-10-30T12:55:16.470 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/ 2023-10-30T12:55:16.471 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 69449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-30T12:55:16.471 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-30T12:55:16.471 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 141225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-30T12:55:16.471 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-30T12:55:16.471 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-30T12:55:16.471 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-30T12:55:16.472 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/ 2023-10-30T12:55:16.472 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-30T12:55:16.472 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 62833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-30T12:55:16.472 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 45721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dnet.ko 2023-10-30T12:55:16.472 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ec_bhf.ko 2023-10-30T12:55:16.473 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/ 2023-10-30T12:55:16.473 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/ 2023-10-30T12:55:16.473 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 281137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-30T12:55:16.577 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ethoc.ko 2023-10-30T12:55:16.577 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fealnx.ko 2023-10-30T12:55:16.578 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/ 2023-10-30T12:55:16.578 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 45345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-10-30T12:55:16.578 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ 2023-10-30T12:55:16.578 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 90465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-30T12:55:16.578 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/ 2023-10-30T12:55:16.579 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 295577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-30T12:55:16.579 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/ 2023-10-30T12:55:16.579 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 584497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-30T12:55:16.579 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/ 2023-10-30T12:55:16.579 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 246209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-30T12:55:16.580 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/ 2023-10-30T12:55:16.580 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 925697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-30T12:55:16.580 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/ 2023-10-30T12:55:16.580 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 302849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-30T12:55:16.580 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/ 2023-10-30T12:55:16.580 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 514681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-30T12:55:16.581 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/ 2023-10-30T12:55:16.581 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 93593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-30T12:55:16.581 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-10-30T12:55:16.581 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 734305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-30T12:55:16.581 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-10-30T12:55:16.581 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 151769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-30T12:55:16.582 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 86921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/jme.ko 2023-10-30T12:55:16.582 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/ 2023-10-30T12:55:16.582 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-30T12:55:16.582 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 102289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-30T12:55:16.582 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 122649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-30T12:55:16.583 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/ 2023-10-30T12:55:16.583 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-10-30T12:55:16.583 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 727601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-30T12:55:16.583 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 278937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-30T12:55:16.583 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-10-30T12:55:16.584 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-10-30T12:55:16.584 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1113993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-30T12:55:16.584 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ 2023-10-30T12:55:16.584 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-10-30T12:55:16.584 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-10-30T12:55:16.584 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-10-30T12:55:16.585 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 95665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-30T12:55:16.585 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/ 2023-10-30T12:55:16.585 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-10-30T12:55:16.585 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 108649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-30T12:55:16.585 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ 2023-10-30T12:55:16.585 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 73441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-30T12:55:16.586 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-30T12:55:16.586 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/ 2023-10-30T12:55:16.586 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 150905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-30T12:55:16.586 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/ 2023-10-30T12:55:16.586 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 136721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-30T12:55:16.587 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/ 2023-10-30T12:55:16.587 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-30T12:55:16.587 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 43145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-30T12:55:16.587 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/ 2023-10-30T12:55:16.587 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-10-30T12:55:16.587 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 227897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-30T12:55:16.589 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 87081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-30T12:55:16.589 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-10-30T12:55:16.589 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 672049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-30T12:55:16.589 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/ 2023-10-30T12:55:16.590 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 43017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-30T12:55:16.590 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/ 2023-10-30T12:55:16.590 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 67385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-30T12:55:16.590 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 83985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-30T12:55:16.591 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-30T12:55:16.591 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 189801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-30T12:55:16.591 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/ 2023-10-30T12:55:16.591 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-10-30T12:55:16.591 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 104241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-10-30T12:55:16.591 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/ 2023-10-30T12:55:16.592 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1053921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-30T12:55:16.592 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/ 2023-10-30T12:55:16.592 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-30T12:55:16.592 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/ 2023-10-30T12:55:16.592 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-30T12:55:16.593 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 63337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-30T12:55:16.593 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/ 2023-10-30T12:55:16.593 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 61601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-30T12:55:16.593 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 67193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-30T12:55:16.593 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 73409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-30T12:55:16.593 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 68177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-30T12:55:16.594 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/ 2023-10-30T12:55:16.594 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-10-30T12:55:16.594 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-10-30T12:55:16.594 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-30T12:55:16.594 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-10-30T12:55:16.595 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 486961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-30T12:55:16.595 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/ 2023-10-30T12:55:16.595 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 103337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-30T12:55:16.595 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 166161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-30T12:55:16.595 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-30T12:55:16.595 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 78801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-30T12:55:16.596 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/ 2023-10-30T12:55:16.596 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-30T12:55:16.596 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/ 2023-10-30T12:55:16.596 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 86529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-30T12:55:16.596 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/ 2023-10-30T12:55:16.597 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 80273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-30T12:55:16.597 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 83633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-30T12:55:16.597 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/ 2023-10-30T12:55:16.597 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 46241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-30T12:55:16.597 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-30T12:55:16.598 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/ 2023-10-30T12:55:16.598 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-30T12:55:16.598 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/ 2023-10-30T12:55:16.598 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/defxx.ko 2023-10-30T12:55:16.598 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/ 2023-10-30T12:55:16.599 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 216985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/skfp.ko 2023-10-30T12:55:16.599 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 77345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/geneve.ko 2023-10-30T12:55:16.599 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/ 2023-10-30T12:55:16.599 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 235521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-30T12:55:16.599 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/ 2023-10-30T12:55:16.599 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-30T12:55:16.600 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ifb.ko 2023-10-30T12:55:16.600 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ 2023-10-30T12:55:16.600 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 83769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-30T12:55:16.600 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 62209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvlan.ko 2023-10-30T12:55:16.600 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvtap.ko 2023-10-30T12:55:16.601 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/ 2023-10-30T12:55:16.601 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-30T12:55:16.601 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-30T12:55:16.601 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-gpio.ko 2023-10-30T12:55:16.601 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-30T12:55:16.602 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio.ko 2023-10-30T12:55:16.602 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mii.ko 2023-10-30T12:55:16.602 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 43953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/netconsole.ko 2023-10-30T12:55:16.602 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/nlmon.ko 2023-10-30T12:55:16.602 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/ 2023-10-30T12:55:16.674 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-30T12:55:16.675 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-30T12:55:16.675 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ 2023-10-30T12:55:16.675 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/amd.ko 2023-10-30T12:55:16.675 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-30T12:55:16.675 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-30T12:55:16.676 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm87xx.ko 2023-10-30T12:55:16.676 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/broadcom.ko 2023-10-30T12:55:16.676 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/cicada.ko 2023-10-30T12:55:16.676 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/davicom.ko 2023-10-30T12:55:16.676 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/et1011c.ko 2023-10-30T12:55:16.676 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/icplus.ko 2023-10-30T12:55:16.677 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/lxt.ko 2023-10-30T12:55:16.677 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 82049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/marvell.ko 2023-10-30T12:55:16.677 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 76177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/micrel.ko 2023-10-30T12:55:16.677 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/national.ko 2023-10-30T12:55:16.677 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 97857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/phylink.ko 2023-10-30T12:55:16.678 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/qsemi.ko 2023-10-30T12:55:16.678 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/realtek.ko 2023-10-30T12:55:16.678 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/smsc.ko 2023-10-30T12:55:16.678 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ste10Xp.ko 2023-10-30T12:55:16.678 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/vitesse.ko 2023-10-30T12:55:16.679 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/ 2023-10-30T12:55:16.679 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 43569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/plip.ko 2023-10-30T12:55:16.679 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ 2023-10-30T12:55:16.679 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-30T12:55:16.679 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_async.ko 2023-10-30T12:55:16.680 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-30T12:55:16.680 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-30T12:55:16.680 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-30T12:55:16.680 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppoe.ko 2023-10-30T12:55:16.680 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppox.ko 2023-10-30T12:55:16.681 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pptp.ko 2023-10-30T12:55:16.681 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sb1000.ko 2023-10-30T12:55:16.681 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/ 2023-10-30T12:55:16.681 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/slip.ko 2023-10-30T12:55:16.681 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sungem_phy.ko 2023-10-30T12:55:16.681 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 64385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/tap.ko 2023-10-30T12:55:16.682 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/ 2023-10-30T12:55:16.682 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 106169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team.ko 2023-10-30T12:55:16.682 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-30T12:55:16.682 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-30T12:55:16.683 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-30T12:55:16.683 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_random.ko 2023-10-30T12:55:16.683 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-30T12:55:16.683 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 87017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/veth.ko 2023-10-30T12:55:16.683 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/ 2023-10-30T12:55:16.684 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 144297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-30T12:55:16.684 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/ 2023-10-30T12:55:16.684 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 297329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/vxlan.ko 2023-10-30T12:55:16.684 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/ 2023-10-30T12:55:16.685 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 144649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-30T12:55:16.685 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ 2023-10-30T12:55:16.685 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ntb.ko 2023-10-30T12:55:16.685 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/ 2023-10-30T12:55:16.685 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/ 2023-10-30T12:55:16.686 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 264937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme-core.ko 2023-10-30T12:55:16.686 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 102209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme.ko 2023-10-30T12:55:16.686 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/ 2023-10-30T12:55:16.686 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 133841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport.ko 2023-10-30T12:55:16.686 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_cs.ko 2023-10-30T12:55:16.687 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 84465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_pc.ko 2023-10-30T12:55:16.687 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_serial.ko 2023-10-30T12:55:16.687 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/ 2023-10-30T12:55:16.687 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/ 2023-10-30T12:55:16.688 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-30T12:55:16.688 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-10-30T12:55:16.688 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-10-30T12:55:16.688 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/pci-stub.ko 2023-10-30T12:55:16.689 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 46529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/xen-pcifront.ko 2023-10-30T12:55:16.689 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/ 2023-10-30T12:55:16.689 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/i82092.ko 2023-10-30T12:55:16.690 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 145809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia.ko 2023-10-30T12:55:16.690 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 62137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_core.ko 2023-10-30T12:55:16.690 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-10-30T12:55:16.690 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pd6729.ko 2023-10-30T12:55:16.690 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 112889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-30T12:55:16.691 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/ 2023-10-30T12:55:16.691 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/ 2023-10-30T12:55:16.691 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-10-30T12:55:16.691 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/ 2023-10-30T12:55:16.692 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-10-30T12:55:16.692 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/ 2023-10-30T12:55:16.692 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/ 2023-10-30T12:55:16.692 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 70153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-30T12:55:16.692 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/ 2023-10-30T12:55:16.693 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/ 2023-10-30T12:55:16.693 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-10-30T12:55:16.693 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec.ko 2023-10-30T12:55:16.693 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-10-30T12:55:16.693 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-10-30T12:55:16.694 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-10-30T12:55:16.694 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-10-30T12:55:16.694 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-10-30T12:55:16.694 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ 2023-10-30T12:55:16.695 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 67097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-30T12:55:16.695 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acerhdf.ko 2023-10-30T12:55:16.695 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-30T12:55:16.695 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 59665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-30T12:55:16.695 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-30T12:55:16.696 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 127569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-30T12:55:16.696 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-30T12:55:16.696 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-30T12:55:16.696 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-30T12:55:16.696 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-30T12:55:16.697 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-30T12:55:16.697 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/hdaps.ko 2023-10-30T12:55:16.697 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ibm_rtl.ko 2023-10-30T12:55:16.697 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/ 2023-10-30T12:55:16.698 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-30T12:55:16.698 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-10-30T12:55:16.698 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 43497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel_ips.ko 2023-10-30T12:55:16.698 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-30T12:55:16.698 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-30T12:55:16.699 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-30T12:55:16.699 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-30T12:55:16.699 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-30T12:55:16.699 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-30T12:55:16.699 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-30T12:55:16.700 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-30T12:55:16.700 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-30T12:55:16.700 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 57569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi.ko 2023-10-30T12:55:16.700 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/ 2023-10-30T12:55:16.701 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ 2023-10-30T12:55:16.701 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_battery.ko 2023-10-30T12:55:16.701 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_charger.ko 2023-10-30T12:55:16.701 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 55057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq2415x_charger.ko 2023-10-30T12:55:16.701 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24190_charger.ko 2023-10-30T12:55:16.702 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24735-charger.ko 2023-10-30T12:55:16.702 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-10-30T12:55:16.702 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9030_battery.ko 2023-10-30T12:55:16.702 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9052-battery.ko 2023-10-30T12:55:16.703 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2760_battery.ko 2023-10-30T12:55:16.703 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2780_battery.ko 2023-10-30T12:55:16.703 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2781_battery.ko 2023-10-30T12:55:16.703 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2782_battery.ko 2023-10-30T12:55:16.704 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/gpio-charger.ko 2023-10-30T12:55:16.704 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/isp1704_charger.ko 2023-10-30T12:55:16.704 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/lp8727_charger.ko 2023-10-30T12:55:16.704 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max14577_charger.ko 2023-10-30T12:55:16.704 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17040_battery.ko 2023-10-30T12:55:16.704 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17042_battery.ko 2023-10-30T12:55:16.705 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8903_charger.ko 2023-10-30T12:55:16.705 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8925_power.ko 2023-10-30T12:55:16.705 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/pcf50633-charger.ko 2023-10-30T12:55:16.705 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/sbs-battery.ko 2023-10-30T12:55:16.706 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/test_power.ko 2023-10-30T12:55:16.706 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/tps65090-charger.ko 2023-10-30T12:55:16.706 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_backup.ko 2023-10-30T12:55:16.706 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_power.ko 2023-10-30T12:55:16.706 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm8350_power.ko 2023-10-30T12:55:16.707 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/ 2023-10-30T12:55:16.707 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-30T12:55:16.707 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-30T12:55:16.707 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/ 2023-10-30T12:55:16.707 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/ 2023-10-30T12:55:16.708 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-30T12:55:16.708 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-30T12:55:16.708 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps_parport.ko 2023-10-30T12:55:16.708 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/pps_core.ko 2023-10-30T12:55:16.708 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ 2023-10-30T12:55:16.709 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp.ko 2023-10-30T12:55:16.709 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp_kvm.ko 2023-10-30T12:55:16.709 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/ 2023-10-30T12:55:16.709 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm80x.ko 2023-10-30T12:55:16.709 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm860x.ko 2023-10-30T12:55:16.709 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-30T12:55:16.710 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9052.ko 2023-10-30T12:55:16.710 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9055.ko 2023-10-30T12:55:16.710 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9063.ko 2023-10-30T12:55:16.710 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-30T12:55:16.710 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 68577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-30T12:55:16.711 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-30T12:55:16.711 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-30T12:55:16.711 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-30T12:55:16.711 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-30T12:55:16.712 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-30T12:55:16.712 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-30T12:55:16.712 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-30T12:55:16.712 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-em3027.ko 2023-10-30T12:55:16.712 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-30T12:55:16.713 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-30T12:55:16.713 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-30T12:55:16.713 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-lp8788.ko 2023-10-30T12:55:16.713 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-30T12:55:16.713 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-30T12:55:16.713 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-30T12:55:16.714 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t86.ko 2023-10-30T12:55:16.714 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max6900.ko 2023-10-30T12:55:16.714 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8907.ko 2023-10-30T12:55:16.714 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8925.ko 2023-10-30T12:55:16.714 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8997.ko 2023-10-30T12:55:16.715 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8998.ko 2023-10-30T12:55:16.715 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-10-30T12:55:16.715 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-30T12:55:16.715 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-palmas.ko 2023-10-30T12:55:16.715 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 46633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-30T12:55:16.716 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf50633.ko 2023-10-30T12:55:16.716 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-30T12:55:16.716 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-30T12:55:16.716 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-30T12:55:16.716 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-30T12:55:16.717 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rc5t583.ko 2023-10-30T12:55:16.717 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-30T12:55:16.717 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-30T12:55:16.717 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-30T12:55:16.718 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-30T12:55:16.718 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-30T12:55:16.718 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-s35390a.ko 2023-10-30T12:55:16.718 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-30T12:55:16.718 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps6586x.ko 2023-10-30T12:55:16.719 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps65910.ko 2023-10-30T12:55:16.719 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm831x.ko 2023-10-30T12:55:16.719 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm8350.ko 2023-10-30T12:55:16.719 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-x1205.ko 2023-10-30T12:55:16.719 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ 2023-10-30T12:55:16.720 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 77617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-9xxx.ko 2023-10-30T12:55:16.720 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 57665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-sas.ko 2023-10-30T12:55:16.720 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 55169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-xxxx.ko 2023-10-30T12:55:16.720 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 135185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/BusLogic.ko 2023-10-30T12:55:16.720 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/a100u2w.ko 2023-10-30T12:55:16.720 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/ 2023-10-30T12:55:16.721 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 272305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-30T12:55:16.721 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 100617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/advansys.ko 2023-10-30T12:55:16.721 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/ 2023-10-30T12:55:16.721 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 356265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-30T12:55:16.721 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 295801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-30T12:55:16.722 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/ 2023-10-30T12:55:16.722 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 189185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-10-30T12:55:16.722 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/am53c974.ko 2023-10-30T12:55:16.722 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/ 2023-10-30T12:55:16.722 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 106881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-30T12:55:16.723 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/atp870u.ko 2023-10-30T12:55:16.723 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/ 2023-10-30T12:55:16.723 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 246985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-30T12:55:16.723 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/ 2023-10-30T12:55:16.723 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1171945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/bfa.ko 2023-10-30T12:55:16.724 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/ 2023-10-30T12:55:16.724 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 237273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-30T12:55:16.724 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/ 2023-10-30T12:55:16.724 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 115369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-30T12:55:16.724 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ch.ko 2023-10-30T12:55:16.725 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/ 2023-10-30T12:55:16.725 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 308921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-30T12:55:16.725 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/ 2023-10-30T12:55:16.725 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-10-30T12:55:16.726 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 76145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-30T12:55:16.726 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-10-30T12:55:16.726 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 101449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-30T12:55:16.726 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 146689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-30T12:55:16.726 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 75553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dc395x.ko 2023-10-30T12:55:16.727 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dmx3191d.ko 2023-10-30T12:55:16.727 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/ 2023-10-30T12:55:16.727 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 212425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-30T12:55:16.727 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esp_scsi.ko 2023-10-30T12:55:16.727 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/ 2023-10-30T12:55:16.727 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 78209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-30T12:55:16.728 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 158505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-30T12:55:16.728 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fdomain.ko 2023-10-30T12:55:16.728 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/ 2023-10-30T12:55:16.728 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 263489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/fnic.ko 2023-10-30T12:55:16.729 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 201817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hpsa.ko 2023-10-30T12:55:16.729 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hptiop.ko 2023-10-30T12:55:16.729 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hv_storvsc.ko 2023-10-30T12:55:16.729 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/imm.ko 2023-10-30T12:55:16.729 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/initio.ko 2023-10-30T12:55:16.729 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 266537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ipr.ko 2023-10-30T12:55:16.730 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 101881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ips.ko 2023-10-30T12:55:16.730 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/ 2023-10-30T12:55:16.730 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 317153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/isci.ko 2023-10-30T12:55:16.730 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-30T12:55:16.731 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 52961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-30T12:55:16.731 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/ 2023-10-30T12:55:16.731 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 314865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/libfc.ko 2023-10-30T12:55:16.731 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 145585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi.ko 2023-10-30T12:55:16.731 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-30T12:55:16.732 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/ 2023-10-30T12:55:16.732 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 216665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/libsas.ko 2023-10-30T12:55:16.732 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/ 2023-10-30T12:55:16.732 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 2197249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-30T12:55:16.732 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/ 2023-10-30T12:55:16.733 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 73569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-30T12:55:16.733 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-30T12:55:16.733 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 310617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-30T12:55:16.733 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 84721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid.ko 2023-10-30T12:55:16.733 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/ 2023-10-30T12:55:16.734 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 659777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-30T12:55:16.734 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/ 2023-10-30T12:55:16.734 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 119561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-30T12:55:16.734 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 78129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvumi.ko 2023-10-30T12:55:16.734 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/ 2023-10-30T12:55:16.735 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 81625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-10-30T12:55:16.735 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-10-30T12:55:16.735 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-10-30T12:55:16.735 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-10-30T12:55:16.735 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/ 2023-10-30T12:55:16.736 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 364105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-30T12:55:16.736 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 123617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pmcraid.ko 2023-10-30T12:55:16.736 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ppa.ko 2023-10-30T12:55:16.736 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 55361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla1280.ko 2023-10-30T12:55:16.736 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/ 2023-10-30T12:55:16.737 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1876625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-30T12:55:16.737 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 85289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-30T12:55:16.737 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/ 2023-10-30T12:55:16.737 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 559033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-30T12:55:16.737 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qlogicfas408.ko 2023-10-30T12:55:16.738 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/raid_class.ko 2023-10-30T12:55:16.738 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 238825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_debug.ko 2023-10-30T12:55:16.738 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 173937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-30T12:55:16.738 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 279313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-30T12:55:16.738 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 97729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-30T12:55:16.739 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 73881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-30T12:55:16.739 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-30T12:55:16.739 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ses.ko 2023-10-30T12:55:16.739 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 115889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/st.ko 2023-10-30T12:55:16.739 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 46825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/stex.ko 2023-10-30T12:55:16.739 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/ 2023-10-30T12:55:16.740 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 162121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-30T12:55:16.740 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/virtio_scsi.ko 2023-10-30T12:55:16.740 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-30T12:55:16.740 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/wd719x.ko 2023-10-30T12:55:16.740 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/xen-scsifront.ko 2023-10-30T12:55:16.740 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ 2023-10-30T12:55:16.741 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 129441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ssb.ko 2023-10-30T12:55:16.741 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/ 2023-10-30T12:55:16.741 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/ 2023-10-30T12:55:16.741 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 736601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-30T12:55:16.741 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/ 2023-10-30T12:55:16.742 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-30T12:55:16.742 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_file.ko 2023-10-30T12:55:16.742 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 43025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_iblock.ko 2023-10-30T12:55:16.742 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 930321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_mod.ko 2023-10-30T12:55:16.742 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_pscsi.ko 2023-10-30T12:55:16.743 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 122521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_user.ko 2023-10-30T12:55:16.743 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/ 2023-10-30T12:55:16.743 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 75593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-30T12:55:16.743 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/ 2023-10-30T12:55:16.744 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/ 2023-10-30T12:55:16.744 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/ 2023-10-30T12:55:16.744 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-30T12:55:16.744 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-30T12:55:16.744 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-30T12:55:16.745 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-30T12:55:16.745 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-30T12:55:16.745 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-30T12:55:16.745 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-30T12:55:16.745 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-30T12:55:16.746 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-30T12:55:16.746 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-30T12:55:16.746 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-30T12:55:16.746 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-30T12:55:16.746 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-10-30T12:55:16.746 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-30T12:55:16.747 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-30T12:55:16.747 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-30T12:55:16.747 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ 2023-10-30T12:55:16.747 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ 2023-10-30T12:55:16.747 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 78425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-30T12:55:16.748 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 57865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/moxa.ko 2023-10-30T12:55:16.748 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 50953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/mxser.ko 2023-10-30T12:55:16.748 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 108601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_gsm.ko 2023-10-30T12:55:16.748 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_hdlc.ko 2023-10-30T12:55:16.749 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/nozomi.ko 2023-10-30T12:55:16.749 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/ 2023-10-30T12:55:16.749 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/ 2023-10-30T12:55:16.749 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-10-30T12:55:16.749 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 59497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-30T12:55:16.750 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-10-30T12:55:16.750 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_uart.ko 2023-10-30T12:55:16.750 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/arc_uart.ko 2023-10-30T12:55:16.750 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 72425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-10-30T12:55:16.750 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/ 2023-10-30T12:55:16.751 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 103073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-30T12:55:16.751 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/rp2.ko 2023-10-30T12:55:16.751 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 52817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/sc16is7xx.ko 2023-10-30T12:55:16.751 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/ 2023-10-30T12:55:16.751 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ 2023-10-30T12:55:16.752 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 531265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-30T12:55:16.752 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ 2023-10-30T12:55:16.752 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-30T12:55:16.752 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-10-30T12:55:16.752 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/ 2023-10-30T12:55:16.753 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio.ko 2023-10-30T12:55:16.753 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_aec.ko 2023-10-30T12:55:16.753 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_cif.ko 2023-10-30T12:55:16.753 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_dmem_genirq.ko 2023-10-30T12:55:16.754 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_mf624.ko 2023-10-30T12:55:16.754 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_netx.ko 2023-10-30T12:55:16.754 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pci_generic.ko 2023-10-30T12:55:16.754 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-10-30T12:55:16.754 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_sercos3.ko 2023-10-30T12:55:16.755 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/ 2023-10-30T12:55:16.755 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/ 2023-10-30T12:55:16.755 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 73297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/c67x00.ko 2023-10-30T12:55:16.755 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ 2023-10-30T12:55:16.756 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 144377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-10-30T12:55:16.756 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-10-30T12:55:16.756 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-10-30T12:55:16.756 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-10-30T12:55:16.756 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/ 2023-10-30T12:55:16.757 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 87089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-acm.ko 2023-10-30T12:55:16.757 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-30T12:55:16.757 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usblp.ko 2023-10-30T12:55:16.757 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 89857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usbtmc.ko 2023-10-30T12:55:16.758 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ 2023-10-30T12:55:16.758 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ulpi.ko 2023-10-30T12:55:16.758 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/ 2023-10-30T12:55:16.758 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-10-30T12:55:16.759 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/ 2023-10-30T12:55:16.759 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-10-30T12:55:16.759 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-10-30T12:55:16.759 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 444553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3.ko 2023-10-30T12:55:16.759 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/ 2023-10-30T12:55:16.760 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/ 2023-10-30T12:55:16.760 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_ether.ko 2023-10-30T12:55:16.760 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_serial.ko 2023-10-30T12:55:16.760 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-10-30T12:55:16.760 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-10-30T12:55:16.760 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-10-30T12:55:16.761 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-10-30T12:55:16.761 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 92489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-10-30T12:55:16.761 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-10-30T12:55:16.761 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 122081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-10-30T12:55:16.761 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 57617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-10-30T12:55:16.762 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-10-30T12:55:16.762 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-10-30T12:55:16.762 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-10-30T12:55:16.762 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 85609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-10-30T12:55:16.762 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-10-30T12:55:16.763 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 58113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-10-30T12:55:16.763 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 67337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-10-30T12:55:16.763 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/ 2023-10-30T12:55:16.763 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-10-30T12:55:16.763 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-10-30T12:55:16.763 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-10-30T12:55:16.764 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-10-30T12:55:16.764 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-10-30T12:55:16.764 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-10-30T12:55:16.764 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-10-30T12:55:16.764 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-10-30T12:55:16.765 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-10-30T12:55:16.765 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-10-30T12:55:16.765 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-10-30T12:55:16.765 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-10-30T12:55:16.765 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 62185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-10-30T12:55:16.765 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-10-30T12:55:16.766 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 161225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/libcomposite.ko 2023-10-30T12:55:16.766 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/ 2023-10-30T12:55:16.766 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-10-30T12:55:16.766 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/ 2023-10-30T12:55:16.767 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 157681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-10-30T12:55:16.767 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-10-30T12:55:16.767 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 52721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-10-30T12:55:16.767 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 69705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-10-30T12:55:16.767 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 56033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-10-30T12:55:16.767 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 76057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2272.ko 2023-10-30T12:55:16.768 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 75073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2280.ko 2023-10-30T12:55:16.768 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 73169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-10-30T12:55:16.768 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 65809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-10-30T12:55:16.768 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-10-30T12:55:16.768 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 81505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-10-30T12:55:16.769 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 156073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-10-30T12:55:16.769 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ 2023-10-30T12:55:16.769 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/bcma-hcd.ko 2023-10-30T12:55:16.769 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 80113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/isp116x-hcd.ko 2023-10-30T12:55:16.769 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 87353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-10-30T12:55:16.770 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-10-30T12:55:16.770 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-30T12:55:16.770 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811_cs.ko 2023-10-30T12:55:16.770 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ssb-hcd.ko 2023-10-30T12:55:16.770 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-10-30T12:55:16.771 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/ 2023-10-30T12:55:16.771 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/mdc800.ko 2023-10-30T12:55:16.771 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/microtek.ko 2023-10-30T12:55:16.771 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ 2023-10-30T12:55:16.772 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/adutux.ko 2023-10-30T12:55:16.772 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/appledisplay.ko 2023-10-30T12:55:16.772 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-10-30T12:55:16.772 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cytherm.ko 2023-10-30T12:55:16.772 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ehset.ko 2023-10-30T12:55:16.773 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi26.ko 2023-10-30T12:55:16.773 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi62.ko 2023-10-30T12:55:16.773 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ezusb.ko 2023-10-30T12:55:16.773 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/idmouse.ko 2023-10-30T12:55:16.773 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/iowarrior.ko 2023-10-30T12:55:16.774 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-30T12:55:16.774 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ldusb.ko 2023-10-30T12:55:16.774 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/legousbtower.ko 2023-10-30T12:55:16.774 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/lvstest.ko 2023-10-30T12:55:16.774 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/ 2023-10-30T12:55:16.775 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 64841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-30T12:55:16.775 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-30T12:55:16.775 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usb3503.ko 2023-10-30T12:55:16.775 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usblcd.ko 2023-10-30T12:55:16.775 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-30T12:55:16.776 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 70873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbtest.ko 2023-10-30T12:55:16.776 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/uss720.ko 2023-10-30T12:55:16.776 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/yurex.ko 2023-10-30T12:55:16.776 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/ 2023-10-30T12:55:16.776 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 72953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/usbmon.ko 2023-10-30T12:55:16.776 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/ 2023-10-30T12:55:16.777 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 301545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/musb_hdrc.ko 2023-10-30T12:55:16.777 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/ 2023-10-30T12:55:16.777 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-generic.ko 2023-10-30T12:55:16.777 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-10-30T12:55:16.777 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-isp1301.ko 2023-10-30T12:55:16.778 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-tahvo.ko 2023-10-30T12:55:16.778 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ 2023-10-30T12:55:16.778 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/aircable.ko 2023-10-30T12:55:16.778 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ark3116.ko 2023-10-30T12:55:16.778 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-30T12:55:16.779 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ch341.ko 2023-10-30T12:55:16.779 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 70257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cp210x.ko 2023-10-30T12:55:16.779 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cyberjack.ko 2023-10-30T12:55:16.779 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 58217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-30T12:55:16.779 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-30T12:55:16.779 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/empeg.ko 2023-10-30T12:55:16.780 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/f81232.ko 2023-10-30T12:55:16.780 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 175137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-30T12:55:16.780 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-30T12:55:16.780 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 137081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-30T12:55:16.780 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 124833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_ti.ko 2023-10-30T12:55:16.781 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 67769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipaq.ko 2023-10-30T12:55:16.781 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipw.ko 2023-10-30T12:55:16.781 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ir-usb.ko 2023-10-30T12:55:16.781 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 58233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-30T12:55:16.781 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 107521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan.ko 2023-10-30T12:55:16.781 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-30T12:55:16.782 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-30T12:55:16.782 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-30T12:55:16.782 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mct_u232.ko 2023-10-30T12:55:16.782 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/metro-usb.ko 2023-10-30T12:55:16.782 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7720.ko 2023-10-30T12:55:16.783 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 78217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7840.ko 2023-10-30T12:55:16.783 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 56337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mxuport.ko 2023-10-30T12:55:16.783 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/navman.ko 2023-10-30T12:55:16.783 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/omninet.ko 2023-10-30T12:55:16.783 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/opticon.ko 2023-10-30T12:55:16.784 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 190633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/option.ko 2023-10-30T12:55:16.784 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/oti6858.ko 2023-10-30T12:55:16.784 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/pl2303.ko 2023-10-30T12:55:16.784 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcaux.ko 2023-10-30T12:55:16.784 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcserial.ko 2023-10-30T12:55:16.784 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/quatech2.ko 2023-10-30T12:55:16.785 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/safe_serial.ko 2023-10-30T12:55:16.785 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 51625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/sierra.ko 2023-10-30T12:55:16.785 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-30T12:55:16.785 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ssu100.ko 2023-10-30T12:55:16.785 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/symbolserial.ko 2023-10-30T12:55:16.786 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-30T12:55:16.786 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-30T12:55:16.786 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_debug.ko 2023-10-30T12:55:16.786 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-30T12:55:16.786 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 125609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usbserial.ko 2023-10-30T12:55:16.787 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/visor.ko 2023-10-30T12:55:16.787 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/whiteheat.ko 2023-10-30T12:55:16.787 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/wishbone-serial.ko 2023-10-30T12:55:16.787 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-30T12:55:16.788 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ 2023-10-30T12:55:16.788 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 52913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/uas.ko 2023-10-30T12:55:16.788 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-30T12:55:16.788 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-30T12:55:16.788 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-30T12:55:16.789 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-30T12:55:16.789 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-30T12:55:16.789 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-30T12:55:16.789 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-30T12:55:16.789 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-karma.ko 2023-10-30T12:55:16.789 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-30T12:55:16.790 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-30T12:55:16.790 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-30T12:55:16.790 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-30T12:55:16.790 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-30T12:55:16.791 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 179969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/usb-storage.ko 2023-10-30T12:55:16.791 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/ 2023-10-30T12:55:16.791 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 67481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-30T12:55:16.791 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 72177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-30T12:55:16.791 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 107001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-30T12:55:16.791 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/ 2023-10-30T12:55:16.793 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/ 2023-10-30T12:55:16.793 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 154297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-30T12:55:16.793 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-30T12:55:16.794 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 108737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio.ko 2023-10-30T12:55:16.794 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 77609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-30T12:55:16.794 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/ 2023-10-30T12:55:16.794 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 113873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost.ko 2023-10-30T12:55:16.794 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-30T12:55:16.794 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 58089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_net.ko 2023-10-30T12:55:16.795 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 89249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_scsi.ko 2023-10-30T12:55:16.795 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/ 2023-10-30T12:55:16.795 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/ 2023-10-30T12:55:16.795 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-10-30T12:55:16.795 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/aat2870_bl.ko 2023-10-30T12:55:16.796 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp5520_bl.ko 2023-10-30T12:55:16.796 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8860_bl.ko 2023-10-30T12:55:16.796 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8870_bl.ko 2023-10-30T12:55:16.796 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/apple_bl.ko 2023-10-30T12:55:16.796 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/as3711_bl.ko 2023-10-30T12:55:16.797 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/bd6107.ko 2023-10-30T12:55:16.797 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da903x_bl.ko 2023-10-30T12:55:16.797 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da9052_bl.ko 2023-10-30T12:55:16.797 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/gpio_backlight.ko 2023-10-30T12:55:16.797 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/kb3886_bl.ko 2023-10-30T12:55:16.797 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lcd.ko 2023-10-30T12:55:16.798 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3533_bl.ko 2023-10-30T12:55:16.798 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3639_bl.ko 2023-10-30T12:55:16.798 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lv5207lp.ko 2023-10-30T12:55:16.798 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/max8925_bl.ko 2023-10-30T12:55:16.798 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pandora_bl.ko 2023-10-30T12:55:16.799 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-10-30T12:55:16.799 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-30T12:55:16.799 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/wm831x_bl.ko 2023-10-30T12:55:16.799 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/ 2023-10-30T12:55:16.799 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ 2023-10-30T12:55:16.800 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2482.ko 2023-10-30T12:55:16.800 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2490.ko 2023-10-30T12:55:16.800 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/matrox_w1.ko 2023-10-30T12:55:16.800 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/w1-gpio.ko 2023-10-30T12:55:16.800 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/ 2023-10-30T12:55:16.801 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-30T12:55:16.801 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-30T12:55:16.801 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-30T12:55:16.801 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-30T12:55:16.801 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-30T12:55:16.802 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-30T12:55:16.802 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-30T12:55:16.802 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-30T12:55:16.802 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-30T12:55:16.802 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-30T12:55:16.803 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 56425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-30T12:55:16.803 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 102673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/wire.ko 2023-10-30T12:55:16.803 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ 2023-10-30T12:55:16.803 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/acquirewdt.ko 2023-10-30T12:55:16.803 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/advantechwdt.ko 2023-10-30T12:55:16.803 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-30T12:55:16.804 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-30T12:55:16.804 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/cpu5wdt.ko 2023-10-30T12:55:16.804 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9052_wdt.ko 2023-10-30T12:55:16.804 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9055_wdt.ko 2023-10-30T12:55:16.804 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9063_wdt.ko 2023-10-30T12:55:16.805 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/dw_wdt.ko 2023-10-30T12:55:16.805 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/eurotechwdt.ko 2023-10-30T12:55:16.805 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-30T12:55:16.805 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/hpwdt.ko 2023-10-30T12:55:16.805 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/i6300esb.ko 2023-10-30T12:55:16.806 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-30T12:55:16.806 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-30T12:55:16.806 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ib700wdt.ko 2023-10-30T12:55:16.806 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ibmasr.ko 2023-10-30T12:55:16.806 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-30T12:55:16.807 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-30T12:55:16.807 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it87_wdt.ko 2023-10-30T12:55:16.807 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/kempld_wdt.ko 2023-10-30T12:55:16.807 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/machzwd.ko 2023-10-30T12:55:16.807 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/mena21_wdt.ko 2023-10-30T12:55:16.807 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-10-30T12:55:16.808 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/nv_tco.ko 2023-10-30T12:55:16.808 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-10-30T12:55:16.808 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pc87413_wdt.ko 2023-10-30T12:55:16.808 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-30T12:55:16.808 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-30T12:55:16.809 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/retu_wdt.ko 2023-10-30T12:55:16.809 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-10-30T12:55:16.809 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-10-30T12:55:16.809 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-30T12:55:16.809 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sc1200wdt.ko 2023-10-30T12:55:16.810 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-30T12:55:16.810 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-10-30T12:55:16.810 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/softdog.ko 2023-10-30T12:55:16.810 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-30T12:55:16.810 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/twl4030_wdt.ko 2023-10-30T12:55:16.811 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/via_wdt.ko 2023-10-30T12:55:16.811 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-30T12:55:16.811 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-30T12:55:16.811 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-30T12:55:16.811 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wafer5823wdt.ko 2023-10-30T12:55:16.811 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wdt_pci.ko 2023-10-30T12:55:16.812 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm831x_wdt.ko 2023-10-30T12:55:16.812 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm8350_wdt.ko 2023-10-30T12:55:16.812 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/xen_wdt.ko 2023-10-30T12:55:16.812 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/ 2023-10-30T12:55:16.812 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-evtchn.ko 2023-10-30T12:55:16.813 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntalloc.ko 2023-10-30T12:55:16.813 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntdev.ko 2023-10-30T12:55:16.813 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/ 2023-10-30T12:55:16.813 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 176441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-30T12:55:16.813 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-privcmd.ko 2023-10-30T12:55:16.814 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 71249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-scsiback.ko 2023-10-30T12:55:16.814 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/ 2023-10-30T12:55:16.814 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-30T12:55:16.814 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ 2023-10-30T12:55:16.814 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/ 2023-10-30T12:55:16.815 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 166425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/9p.ko 2023-10-30T12:55:16.815 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/ 2023-10-30T12:55:16.815 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 61073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/adfs.ko 2023-10-30T12:55:16.815 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/ 2023-10-30T12:55:16.815 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 163593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/affs.ko 2023-10-30T12:55:16.816 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/ 2023-10-30T12:55:16.816 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 932817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/kafs.ko 2023-10-30T12:55:16.816 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/ 2023-10-30T12:55:16.816 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 101057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/autofs4.ko 2023-10-30T12:55:16.816 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/ 2023-10-30T12:55:16.816 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 64545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/befs.ko 2023-10-30T12:55:16.817 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/ 2023-10-30T12:55:16.817 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/bfs.ko 2023-10-30T12:55:16.817 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/binfmt_misc.ko 2023-10-30T12:55:16.817 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/ 2023-10-30T12:55:16.817 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 3402585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/btrfs.ko 2023-10-30T12:55:16.818 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/ 2023-10-30T12:55:16.818 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 306073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/cachefiles.ko 2023-10-30T12:55:16.818 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ 2023-10-30T12:55:16.818 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1323945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ceph.ko 2023-10-30T12:55:16.818 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/ 2023-10-30T12:55:16.819 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 100801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/coda.ko 2023-10-30T12:55:16.819 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/ 2023-10-30T12:55:16.819 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 108393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/configfs.ko 2023-10-30T12:55:16.819 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/ 2023-10-30T12:55:16.819 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 29625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/cramfs.ko 2023-10-30T12:55:16.819 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/ 2023-10-30T12:55:16.820 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 642577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/dlm.ko 2023-10-30T12:55:16.820 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/ 2023-10-30T12:55:16.820 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/efs.ko 2023-10-30T12:55:16.820 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/ 2023-10-30T12:55:16.820 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1505025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/f2fs.ko 2023-10-30T12:55:16.821 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/ 2023-10-30T12:55:16.821 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/msdos.ko 2023-10-30T12:55:16.821 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/ 2023-10-30T12:55:16.821 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/freevxfs.ko 2023-10-30T12:55:16.821 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/ 2023-10-30T12:55:16.821 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 226481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/fscache.ko 2023-10-30T12:55:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/ 2023-10-30T12:55:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/cuse.ko 2023-10-30T12:55:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/ 2023-10-30T12:55:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1231529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/gfs2.ko 2023-10-30T12:55:16.822 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/ 2023-10-30T12:55:16.823 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 141209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/hfs.ko 2023-10-30T12:55:16.823 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/ 2023-10-30T12:55:16.823 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 222817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/hfsplus.ko 2023-10-30T12:55:16.823 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/ 2023-10-30T12:55:16.823 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 192329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/hpfs.ko 2023-10-30T12:55:16.824 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/ 2023-10-30T12:55:16.824 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 74065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/isofs.ko 2023-10-30T12:55:16.824 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/ 2023-10-30T12:55:16.824 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 419497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/jfs.ko 2023-10-30T12:55:16.824 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/ 2023-10-30T12:55:16.824 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 280313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/lockd.ko 2023-10-30T12:55:16.825 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/ 2023-10-30T12:55:16.825 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 89193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/minix.ko 2023-10-30T12:55:16.825 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/ 2023-10-30T12:55:16.825 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 124649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/netfs.ko 2023-10-30T12:55:16.825 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/ 2023-10-30T12:55:16.825 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/ 2023-10-30T12:55:16.826 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 81449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-30T12:55:16.826 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/ 2023-10-30T12:55:16.826 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 59449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-30T12:55:16.826 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/ 2023-10-30T12:55:16.827 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 117513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-30T12:55:16.827 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1055649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfs.ko 2023-10-30T12:55:16.827 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv2.ko 2023-10-30T12:55:16.827 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 120177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv3.ko 2023-10-30T12:55:16.827 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 2041689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv4.ko 2023-10-30T12:55:16.827 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/ 2023-10-30T12:55:16.828 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/grace.ko 2023-10-30T12:55:16.828 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/nfs_acl.ko 2023-10-30T12:55:16.828 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/ 2023-10-30T12:55:16.828 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1516809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/nfsd.ko 2023-10-30T12:55:16.828 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/ 2023-10-30T12:55:16.829 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 592833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/nilfs2.ko 2023-10-30T12:55:16.829 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/ 2023-10-30T12:55:16.829 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-celtic.ko 2023-10-30T12:55:16.829 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-centeuro.ko 2023-10-30T12:55:16.829 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-croatian.ko 2023-10-30T12:55:16.829 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-cyrillic.ko 2023-10-30T12:55:16.830 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-gaelic.ko 2023-10-30T12:55:16.830 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-greek.ko 2023-10-30T12:55:16.830 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-iceland.ko 2023-10-30T12:55:16.830 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-inuit.ko 2023-10-30T12:55:16.830 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-roman.ko 2023-10-30T12:55:16.831 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-romanian.ko 2023-10-30T12:55:16.831 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-turkish.ko 2023-10-30T12:55:16.831 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ascii.ko 2023-10-30T12:55:16.831 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1250.ko 2023-10-30T12:55:16.831 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1251.ko 2023-10-30T12:55:16.831 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1255.ko 2023-10-30T12:55:16.832 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp737.ko 2023-10-30T12:55:16.832 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp775.ko 2023-10-30T12:55:16.832 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp850.ko 2023-10-30T12:55:16.832 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp852.ko 2023-10-30T12:55:16.832 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp855.ko 2023-10-30T12:55:16.833 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp857.ko 2023-10-30T12:55:16.833 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp860.ko 2023-10-30T12:55:16.833 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp861.ko 2023-10-30T12:55:16.833 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp862.ko 2023-10-30T12:55:16.833 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp863.ko 2023-10-30T12:55:16.834 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp864.ko 2023-10-30T12:55:16.834 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp865.ko 2023-10-30T12:55:16.834 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp866.ko 2023-10-30T12:55:16.834 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp869.ko 2023-10-30T12:55:16.834 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp874.ko 2023-10-30T12:55:16.835 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 94137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp932.ko 2023-10-30T12:55:16.835 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 142673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp936.ko 2023-10-30T12:55:16.835 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 165745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp949.ko 2023-10-30T12:55:16.835 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 118121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp950.ko 2023-10-30T12:55:16.835 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_euc-jp.ko 2023-10-30T12:55:16.836 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-1.ko 2023-10-30T12:55:16.836 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-13.ko 2023-10-30T12:55:16.836 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-14.ko 2023-10-30T12:55:16.836 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-15.ko 2023-10-30T12:55:16.836 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-2.ko 2023-10-30T12:55:16.837 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-3.ko 2023-10-30T12:55:16.837 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-4.ko 2023-10-30T12:55:16.837 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-5.ko 2023-10-30T12:55:16.837 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-6.ko 2023-10-30T12:55:16.837 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-7.ko 2023-10-30T12:55:16.837 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-9.ko 2023-10-30T12:55:16.838 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-r.ko 2023-10-30T12:55:16.838 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-ru.ko 2023-10-30T12:55:16.838 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-u.ko 2023-10-30T12:55:16.838 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-30T12:55:16.838 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_utf8.ko 2023-10-30T12:55:16.839 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ 2023-10-30T12:55:16.839 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 219417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ntfs.ko 2023-10-30T12:55:16.839 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ 2023-10-30T12:55:16.839 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ 2023-10-30T12:55:16.839 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 244409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-30T12:55:16.840 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ 2023-10-30T12:55:16.840 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 440737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-30T12:55:16.840 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ 2023-10-30T12:55:16.840 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-30T12:55:16.840 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 2592433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2.ko 2023-10-30T12:55:16.840 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-30T12:55:16.841 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-30T12:55:16.841 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-30T12:55:16.841 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/ 2023-10-30T12:55:16.841 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/omfs.ko 2023-10-30T12:55:16.841 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/ 2023-10-30T12:55:16.842 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 352153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/overlay.ko 2023-10-30T12:55:16.842 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ 2023-10-30T12:55:16.842 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ramoops.ko 2023-10-30T12:55:16.842 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/ 2023-10-30T12:55:16.842 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/qnx4.ko 2023-10-30T12:55:16.843 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/ 2023-10-30T12:55:16.843 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 46689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/qnx6.ko 2023-10-30T12:55:16.843 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/ 2023-10-30T12:55:16.843 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_tree.ko 2023-10-30T12:55:16.843 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v1.ko 2023-10-30T12:55:16.843 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v2.ko 2023-10-30T12:55:16.844 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/ 2023-10-30T12:55:16.844 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 532673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/reiserfs.ko 2023-10-30T12:55:16.844 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/ 2023-10-30T12:55:16.844 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/romfs.ko 2023-10-30T12:55:16.845 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/ 2023-10-30T12:55:16.845 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/ 2023-10-30T12:55:16.845 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 2668121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/cifs.ko 2023-10-30T12:55:16.845 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/ 2023-10-30T12:55:16.845 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_arc4.ko 2023-10-30T12:55:16.845 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_md4.ko 2023-10-30T12:55:16.846 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/ 2023-10-30T12:55:16.846 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 134641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/squashfs.ko 2023-10-30T12:55:16.846 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/ 2023-10-30T12:55:16.846 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 90721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/sysv.ko 2023-10-30T12:55:16.846 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/ 2023-10-30T12:55:16.847 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 247737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/udf.ko 2023-10-30T12:55:16.847 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ 2023-10-30T12:55:16.847 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 162497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ufs.ko 2023-10-30T12:55:16.847 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/ 2023-10-30T12:55:16.847 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 4010729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/xfs.ko 2023-10-30T12:55:16.848 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/ 2023-10-30T12:55:16.848 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/ 2023-10-30T12:55:16.848 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 75489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/locktorture.ko 2023-10-30T12:55:16.848 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/ 2023-10-30T12:55:16.848 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/test_udelay.ko 2023-10-30T12:55:16.849 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/torture.ko 2023-10-30T12:55:16.849 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ 2023-10-30T12:55:16.849 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 6833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc-itu-t.ko 2023-10-30T12:55:16.849 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 6529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc7.ko 2023-10-30T12:55:16.849 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc8.ko 2023-10-30T12:55:16.850 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/ 2023-10-30T12:55:16.850 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libarc4.ko 2023-10-30T12:55:16.850 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libdes.ko 2023-10-30T12:55:16.850 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/interval_tree_test.ko 2023-10-30T12:55:16.850 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/libcrc32c.ko 2023-10-30T12:55:16.851 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lru_cache.ko 2023-10-30T12:55:16.851 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/ 2023-10-30T12:55:16.851 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4_compress.ko 2023-10-30T12:55:16.851 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4hc_compress.ko 2023-10-30T12:55:16.852 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/ 2023-10-30T12:55:16.852 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 6617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/cordic.ko 2023-10-30T12:55:16.852 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/memory-notifier-error-inject.ko 2023-10-30T12:55:16.852 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/notifier-error-inject.ko 2023-10-30T12:55:16.852 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/percpu_test.ko 2023-10-30T12:55:16.853 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/pm-notifier-error-inject.ko 2023-10-30T12:55:16.853 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/ 2023-10-30T12:55:16.853 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 143433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/raid6_pq.ko 2023-10-30T12:55:16.853 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/rbtree_test.ko 2023-10-30T12:55:16.853 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/ 2023-10-30T12:55:16.854 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-30T12:55:16.854 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-kstrtox.ko 2023-10-30T12:55:16.854 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-string_helpers.ko 2023-10-30T12:55:16.854 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 4760425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_bpf.ko 2023-10-30T12:55:16.854 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_firmware.ko 2023-10-30T12:55:16.855 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 6369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_module.ko 2023-10-30T12:55:16.855 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_user_copy.ko 2023-10-30T12:55:16.855 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_bm.ko 2023-10-30T12:55:16.855 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_fsm.ko 2023-10-30T12:55:16.855 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_kmp.ko 2023-10-30T12:55:16.855 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/ 2023-10-30T12:55:16.856 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/xz_dec_test.ko 2023-10-30T12:55:16.856 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/ 2023-10-30T12:55:16.856 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/hwpoison-inject.ko 2023-10-30T12:55:16.856 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ 2023-10-30T12:55:16.856 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/ 2023-10-30T12:55:16.856 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 55737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/6lowpan.ko 2023-10-30T12:55:16.857 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_dest.ko 2023-10-30T12:55:16.857 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_fragment.ko 2023-10-30T12:55:16.857 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_hop.ko 2023-10-30T12:55:16.857 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 6985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-30T12:55:16.857 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_mobility.ko 2023-10-30T12:55:16.858 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_routing.ko 2023-10-30T12:55:16.858 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_udp.ko 2023-10-30T12:55:16.858 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/ 2023-10-30T12:55:16.858 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/garp.ko 2023-10-30T12:55:16.858 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/mrp.ko 2023-10-30T12:55:16.859 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/p8022.ko 2023-10-30T12:55:16.859 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/psnap.ko 2023-10-30T12:55:16.859 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/stp.ko 2023-10-30T12:55:16.859 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/ 2023-10-30T12:55:16.859 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 81737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/8021q.ko 2023-10-30T12:55:16.860 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/ 2023-10-30T12:55:16.860 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 198337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet.ko 2023-10-30T12:55:16.860 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_fd.ko 2023-10-30T12:55:16.860 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_rdma.ko 2023-10-30T12:55:16.860 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_virtio.ko 2023-10-30T12:55:16.861 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/ 2023-10-30T12:55:16.861 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 649297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/batman-adv.ko 2023-10-30T12:55:16.861 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/ 2023-10-30T12:55:16.861 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 63809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/br_netfilter.ko 2023-10-30T12:55:16.861 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 777689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/bridge.ko 2023-10-30T12:55:16.862 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ 2023-10-30T12:55:16.862 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-30T12:55:16.862 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-30T12:55:16.862 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-30T12:55:16.862 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-30T12:55:16.863 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-30T12:55:16.863 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-30T12:55:16.863 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-30T12:55:16.863 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-30T12:55:16.863 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-30T12:55:16.864 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-30T12:55:16.864 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-30T12:55:16.864 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-30T12:55:16.864 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-30T12:55:16.864 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-30T12:55:16.864 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-30T12:55:16.865 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-30T12:55:16.865 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-30T12:55:16.865 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-30T12:55:16.865 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-30T12:55:16.865 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-30T12:55:16.866 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 55929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtables.ko 2023-10-30T12:55:16.866 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-30T12:55:16.866 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-30T12:55:16.866 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/ 2023-10-30T12:55:16.866 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1047497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/libceph.ko 2023-10-30T12:55:16.867 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/ 2023-10-30T12:55:16.867 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 125697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/pktgen.ko 2023-10-30T12:55:16.867 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/ 2023-10-30T12:55:16.867 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 194209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp.ko 2023-10-30T12:55:16.867 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_diag.ko 2023-10-30T12:55:16.867 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 57881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv4.ko 2023-10-30T12:55:16.868 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv6.ko 2023-10-30T12:55:16.868 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/ 2023-10-30T12:55:16.868 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 390473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/dsa_core.ko 2023-10-30T12:55:16.868 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_brcm.ko 2023-10-30T12:55:16.868 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_dsa.ko 2023-10-30T12:55:16.868 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_none.ko 2023-10-30T12:55:16.869 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_trailer.ko 2023-10-30T12:55:16.869 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/ 2023-10-30T12:55:16.869 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 99729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/hsr.ko 2023-10-30T12:55:16.869 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ 2023-10-30T12:55:16.869 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ 2023-10-30T12:55:16.870 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 45233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-30T12:55:16.870 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 254625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154.ko 2023-10-30T12:55:16.870 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-30T12:55:16.870 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ 2023-10-30T12:55:16.870 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ah4.ko 2023-10-30T12:55:16.871 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/esp4.ko 2023-10-30T12:55:16.871 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 48401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/fou.ko 2023-10-30T12:55:16.871 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/gre.ko 2023-10-30T12:55:16.871 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 48657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/inet_diag.ko 2023-10-30T12:55:16.871 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_gre.ko 2023-10-30T12:55:16.872 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 61225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_tunnel.ko 2023-10-30T12:55:16.872 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_vti.ko 2023-10-30T12:55:16.872 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipcomp.ko 2023-10-30T12:55:16.872 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipip.ko 2023-10-30T12:55:16.872 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ 2023-10-30T12:55:16.873 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-30T12:55:16.873 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-30T12:55:16.873 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-30T12:55:16.873 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ip_tables.ko 2023-10-30T12:55:16.873 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-30T12:55:16.873 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-10-30T12:55:16.874 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-30T12:55:16.874 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-30T12:55:16.874 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-30T12:55:16.874 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-10-30T12:55:16.874 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-30T12:55:16.875 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-30T12:55:16.875 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-30T12:55:16.875 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-30T12:55:16.875 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-30T12:55:16.875 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-30T12:55:16.876 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-30T12:55:16.876 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-30T12:55:16.876 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-30T12:55:16.876 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-10-30T12:55:16.876 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-30T12:55:16.876 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-30T12:55:16.877 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-30T12:55:16.877 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-30T12:55:16.877 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-30T12:55:16.877 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_bic.ko 2023-10-30T12:55:16.877 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_dctcp.ko 2023-10-30T12:55:16.878 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_diag.ko 2023-10-30T12:55:16.878 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_highspeed.ko 2023-10-30T12:55:16.878 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_htcp.ko 2023-10-30T12:55:16.878 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_hybla.ko 2023-10-30T12:55:16.878 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_illinois.ko 2023-10-30T12:55:16.879 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_lp.ko 2023-10-30T12:55:16.879 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_scalable.ko 2023-10-30T12:55:16.879 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_vegas.ko 2023-10-30T12:55:16.879 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_veno.ko 2023-10-30T12:55:16.879 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_westwood.ko 2023-10-30T12:55:16.880 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_yeah.ko 2023-10-30T12:55:16.880 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tunnel4.ko 2023-10-30T12:55:16.880 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_diag.ko 2023-10-30T12:55:16.880 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 41665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_tunnel.ko 2023-10-30T12:55:16.880 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-30T12:55:16.881 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ 2023-10-30T12:55:16.881 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ah6.ko 2023-10-30T12:55:16.881 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/esp6.ko 2023-10-30T12:55:16.881 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/fou6.ko 2023-10-30T12:55:16.881 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 80897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_gre.ko 2023-10-30T12:55:16.882 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 93577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_tunnel.ko 2023-10-30T12:55:16.882 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-30T12:55:16.882 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_vti.ko 2023-10-30T12:55:16.882 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ipcomp6.ko 2023-10-30T12:55:16.882 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/mip6.ko 2023-10-30T12:55:16.883 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ 2023-10-30T12:55:16.883 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-30T12:55:16.883 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-10-30T12:55:16.883 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-30T12:55:16.883 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-30T12:55:16.883 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-30T12:55:16.884 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-30T12:55:16.884 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-30T12:55:16.884 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-30T12:55:16.884 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-30T12:55:16.884 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-30T12:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-30T12:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-30T12:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-30T12:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-30T12:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-30T12:55:16.885 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-30T12:55:16.886 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-30T12:55:16.886 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-30T12:55:16.886 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-30T12:55:16.886 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-30T12:55:16.886 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-30T12:55:16.887 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-30T12:55:16.887 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-30T12:55:16.887 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-30T12:55:16.887 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-30T12:55:16.887 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 74073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/sit.ko 2023-10-30T12:55:16.887 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/tunnel6.ko 2023-10-30T12:55:16.888 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-30T12:55:16.888 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/ 2023-10-30T12:55:16.888 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 82833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/af_key.ko 2023-10-30T12:55:16.888 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/ 2023-10-30T12:55:16.888 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 125281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_core.ko 2023-10-30T12:55:16.889 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-30T12:55:16.889 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 25953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_eth.ko 2023-10-30T12:55:16.889 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip.ko 2023-10-30T12:55:16.889 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip6.ko 2023-10-30T12:55:16.889 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_netlink.ko 2023-10-30T12:55:16.889 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 53241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ppp.ko 2023-10-30T12:55:16.890 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/ 2023-10-30T12:55:16.890 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 47977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/lapb.ko 2023-10-30T12:55:16.890 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/ 2023-10-30T12:55:16.890 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc.ko 2023-10-30T12:55:16.891 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 267777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc2.ko 2023-10-30T12:55:16.891 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/ 2023-10-30T12:55:16.891 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 290417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/mac802154.ko 2023-10-30T12:55:16.891 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/ 2023-10-30T12:55:16.891 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/mpls_gso.ko 2023-10-30T12:55:16.892 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ 2023-10-30T12:55:16.892 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ 2023-10-30T12:55:16.892 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 119585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set.ko 2023-10-30T12:55:16.892 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-30T12:55:16.892 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-30T12:55:16.893 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-30T12:55:16.893 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 100297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-30T12:55:16.893 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 101441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-30T12:55:16.893 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 102817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-30T12:55:16.893 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 103481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-30T12:55:16.893 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 112849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-30T12:55:16.894 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 60329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-30T12:55:16.894 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-30T12:55:16.894 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 113129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-30T12:55:16.894 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 112585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-30T12:55:16.895 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 111585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-30T12:55:16.895 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 114465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-30T12:55:16.895 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-30T12:55:16.895 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ 2023-10-30T12:55:16.895 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 411729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-30T12:55:16.895 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-30T12:55:16.896 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-30T12:55:16.896 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-30T12:55:16.896 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-30T12:55:16.896 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-30T12:55:16.896 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-30T12:55:16.897 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-30T12:55:16.897 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-30T12:55:16.897 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-30T12:55:16.897 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-30T12:55:16.897 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-30T12:55:16.897 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-30T12:55:16.898 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-30T12:55:16.898 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conncount.ko 2023-10-30T12:55:16.898 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 408257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack.ko 2023-10-30T12:55:16.898 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-30T12:55:16.898 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-30T12:55:16.899 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-30T12:55:16.899 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 118145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-30T12:55:16.899 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-30T12:55:16.899 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-30T12:55:16.899 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 107297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-30T12:55:16.899 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-30T12:55:16.900 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-30T12:55:16.900 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-30T12:55:16.900 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-30T12:55:16.900 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-30T12:55:16.901 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-30T12:55:16.901 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 89473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table.ko 2023-10-30T12:55:16.901 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-30T12:55:16.901 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 37129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_log_syslog.ko 2023-10-30T12:55:16.901 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 95417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat.ko 2023-10-30T12:55:16.901 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-30T12:55:16.902 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-30T12:55:16.902 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_irc.ko 2023-10-30T12:55:16.902 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_sip.ko 2023-10-30T12:55:16.902 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-30T12:55:16.902 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 44065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-30T12:55:16.903 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 615313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_tables.ko 2023-10-30T12:55:16.903 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 42561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink.ko 2023-10-30T12:55:16.903 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-30T12:55:16.903 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-10-30T12:55:16.903 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 39785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_log.ko 2023-10-30T12:55:16.904 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-30T12:55:16.904 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-30T12:55:16.904 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_chain_nat.ko 2023-10-30T12:55:16.904 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_compat.ko 2023-10-30T12:55:16.904 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 40721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_ct.ko 2023-10-30T12:55:16.904 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-30T12:55:16.905 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fib.ko 2023-10-30T12:55:16.905 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-30T12:55:16.905 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_hash.ko 2023-10-30T12:55:16.905 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_limit.ko 2023-10-30T12:55:16.905 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_log.ko 2023-10-30T12:55:16.906 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_masq.ko 2023-10-30T12:55:16.906 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_nat.ko 2023-10-30T12:55:16.906 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_numgen.ko 2023-10-30T12:55:16.906 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_osf.ko 2023-10-30T12:55:16.906 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_queue.ko 2023-10-30T12:55:16.906 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_redir.ko 2023-10-30T12:55:16.907 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject.ko 2023-10-30T12:55:16.907 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject_inet.ko 2023-10-30T12:55:16.907 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_socket.ko 2023-10-30T12:55:16.907 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_synproxy.ko 2023-10-30T12:55:16.907 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_tproxy.ko 2023-10-30T12:55:16.908 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 89313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/x_tables.ko 2023-10-30T12:55:16.908 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_AUDIT.ko 2023-10-30T12:55:16.908 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-30T12:55:16.908 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-30T12:55:16.908 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-30T12:55:16.909 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 18169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CT.ko 2023-10-30T12:55:16.909 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_DSCP.ko 2023-10-30T12:55:16.909 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HL.ko 2023-10-30T12:55:16.909 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HMARK.ko 2023-10-30T12:55:16.909 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-30T12:55:16.909 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LED.ko 2023-10-30T12:55:16.910 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LOG.ko 2023-10-30T12:55:16.910 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-30T12:55:16.910 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NETMAP.ko 2023-10-30T12:55:16.910 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFLOG.ko 2023-10-30T12:55:16.911 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-30T12:55:16.911 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_RATEEST.ko 2023-10-30T12:55:16.911 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-30T12:55:16.911 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_SECMARK.ko 2023-10-30T12:55:16.911 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-30T12:55:16.912 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-30T12:55:16.912 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TEE.ko 2023-10-30T12:55:16.912 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TPROXY.ko 2023-10-30T12:55:16.912 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TRACE.ko 2023-10-30T12:55:16.913 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_addrtype.ko 2023-10-30T12:55:16.913 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_bpf.ko 2023-10-30T12:55:16.913 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cgroup.ko 2023-10-30T12:55:16.913 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cluster.ko 2023-10-30T12:55:16.913 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_comment.ko 2023-10-30T12:55:16.914 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connbytes.ko 2023-10-30T12:55:16.914 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlabel.ko 2023-10-30T12:55:16.914 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlimit.ko 2023-10-30T12:55:16.914 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connmark.ko 2023-10-30T12:55:16.915 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_conntrack.ko 2023-10-30T12:55:16.915 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cpu.ko 2023-10-30T12:55:16.915 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dccp.ko 2023-10-30T12:55:16.915 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_devgroup.ko 2023-10-30T12:55:16.915 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dscp.ko 2023-10-30T12:55:16.915 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ecn.ko 2023-10-30T12:55:16.916 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_esp.ko 2023-10-30T12:55:16.916 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 43289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hashlimit.ko 2023-10-30T12:55:16.916 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_helper.ko 2023-10-30T12:55:16.916 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hl.ko 2023-10-30T12:55:16.916 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipcomp.ko 2023-10-30T12:55:16.917 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_iprange.ko 2023-10-30T12:55:16.917 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipvs.ko 2023-10-30T12:55:16.917 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_l2tp.ko 2023-10-30T12:55:16.917 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_length.ko 2023-10-30T12:55:16.917 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_limit.ko 2023-10-30T12:55:16.918 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mac.ko 2023-10-30T12:55:16.918 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mark.ko 2023-10-30T12:55:16.918 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_multiport.ko 2023-10-30T12:55:16.918 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nat.ko 2023-10-30T12:55:16.918 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nfacct.ko 2023-10-30T12:55:16.919 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_osf.ko 2023-10-30T12:55:16.919 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_owner.ko 2023-10-30T12:55:16.919 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_physdev.ko 2023-10-30T12:55:16.919 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_pkttype.ko 2023-10-30T12:55:16.919 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_policy.ko 2023-10-30T12:55:16.920 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_quota.ko 2023-10-30T12:55:16.920 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_rateest.ko 2023-10-30T12:55:16.920 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_realm.ko 2023-10-30T12:55:16.920 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_recent.ko 2023-10-30T12:55:16.920 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_sctp.ko 2023-10-30T12:55:16.920 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_set.ko 2023-10-30T12:55:16.921 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 16089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_socket.ko 2023-10-30T12:55:16.921 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_state.ko 2023-10-30T12:55:16.921 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_statistic.ko 2023-10-30T12:55:16.921 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_string.ko 2023-10-30T12:55:16.921 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 8553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpmss.ko 2023-10-30T12:55:16.922 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpudp.ko 2023-10-30T12:55:16.922 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_time.ko 2023-10-30T12:55:16.922 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_u32.ko 2023-10-30T12:55:16.922 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/ 2023-10-30T12:55:16.922 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/netlink_diag.ko 2023-10-30T12:55:16.923 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/ 2023-10-30T12:55:16.923 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 12425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/nsh.ko 2023-10-30T12:55:16.923 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/ 2023-10-30T12:55:16.923 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 380033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/openvswitch.ko 2023-10-30T12:55:16.923 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-geneve.ko 2023-10-30T12:55:16.923 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-gre.ko 2023-10-30T12:55:16.924 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 11793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-vxlan.ko 2023-10-30T12:55:16.924 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/ 2023-10-30T12:55:16.924 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/af_packet_diag.ko 2023-10-30T12:55:16.924 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/ 2023-10-30T12:55:16.925 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 110201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/phonet.ko 2023-10-30T12:55:16.925 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/pn_pep.ko 2023-10-30T12:55:16.925 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/ 2023-10-30T12:55:16.925 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 251481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds.ko 2023-10-30T12:55:16.925 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 164937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_rdma.ko 2023-10-30T12:55:16.925 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 67377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_tcp.ko 2023-10-30T12:55:16.926 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/ 2023-10-30T12:55:16.926 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 819657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/rxrpc.ko 2023-10-30T12:55:16.926 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/ 2023-10-30T12:55:16.926 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_csum.ko 2023-10-30T12:55:16.927 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_gact.ko 2023-10-30T12:55:16.927 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_ipt.ko 2023-10-30T12:55:16.927 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_mirred.ko 2023-10-30T12:55:16.927 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_nat.ko 2023-10-30T12:55:16.927 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 43449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_pedit.ko 2023-10-30T12:55:16.928 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_police.ko 2023-10-30T12:55:16.928 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_simple.ko 2023-10-30T12:55:16.928 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_skbedit.ko 2023-10-30T12:55:16.928 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_vlan.ko 2023-10-30T12:55:16.928 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_basic.ko 2023-10-30T12:55:16.928 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_bpf.ko 2023-10-30T12:55:16.929 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_cgroup.ko 2023-10-30T12:55:16.929 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_flow.ko 2023-10-30T12:55:16.929 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_fw.ko 2023-10-30T12:55:16.929 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 38489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_route.ko 2023-10-30T12:55:16.929 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 61409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_u32.ko 2023-10-30T12:55:16.930 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_cmp.ko 2023-10-30T12:55:16.930 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 13409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_ipset.ko 2023-10-30T12:55:16.930 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_meta.ko 2023-10-30T12:55:16.930 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_nbyte.ko 2023-10-30T12:55:16.930 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_text.ko 2023-10-30T12:55:16.931 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_u32.ko 2023-10-30T12:55:16.931 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 21649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_choke.ko 2023-10-30T12:55:16.931 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_codel.ko 2023-10-30T12:55:16.931 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 34633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_drr.ko 2023-10-30T12:55:16.931 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 33609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq.ko 2023-10-30T12:55:16.932 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq_codel.ko 2023-10-30T12:55:16.932 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 35433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_gred.ko 2023-10-30T12:55:16.932 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 49601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hfsc.ko 2023-10-30T12:55:16.932 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hhf.ko 2023-10-30T12:55:16.932 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 66393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_htb.ko 2023-10-30T12:55:16.933 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_ingress.ko 2023-10-30T12:55:16.933 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 30273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio.ko 2023-10-30T12:55:16.933 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio_lib.ko 2023-10-30T12:55:16.933 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_multiq.ko 2023-10-30T12:55:16.933 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 36137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_netem.ko 2023-10-30T12:55:16.934 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 27497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_pie.ko 2023-10-30T12:55:16.934 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_plug.ko 2023-10-30T12:55:16.934 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_prio.ko 2023-10-30T12:55:16.934 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 48793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_qfq.ko 2023-10-30T12:55:16.934 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 32905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_red.ko 2023-10-30T12:55:16.935 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 26881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfb.ko 2023-10-30T12:55:16.935 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 31353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfq.ko 2023-10-30T12:55:16.935 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_tbf.ko 2023-10-30T12:55:16.935 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 28105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_teql.ko 2023-10-30T12:55:16.936 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/ 2023-10-30T12:55:16.936 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 929745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp.ko 2023-10-30T12:55:16.936 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 19033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp_diag.ko 2023-10-30T12:55:16.936 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/ 2023-10-30T12:55:16.936 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/ 2023-10-30T12:55:16.937 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 336529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-30T12:55:16.937 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 54737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-30T12:55:16.937 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 1520161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/sunrpc.ko 2023-10-30T12:55:16.937 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/ 2023-10-30T12:55:16.937 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 707721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-30T12:55:16.937 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/ 2023-10-30T12:55:16.938 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/diag.ko 2023-10-30T12:55:16.938 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 862641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/tipc.ko 2023-10-30T12:55:16.938 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/ 2023-10-30T12:55:16.938 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/unix_diag.ko 2023-10-30T12:55:16.938 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/ 2023-10-30T12:55:16.939 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 85337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-30T12:55:16.939 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 68049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-30T12:55:16.939 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 96193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock.ko 2023-10-30T12:55:16.939 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 10705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-30T12:55:16.940 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 17521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-30T12:55:16.940 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/ 2023-10-30T12:55:16.940 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 125177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/x25.ko 2023-10-30T12:55:16.940 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/ 2023-10-30T12:55:16.940 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 22657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_algo.ko 2023-10-30T12:55:16.940 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 20649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-30T12:55:16.941 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 102257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_user.ko 2023-10-30T12:55:16.941 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/ 2023-10-30T12:55:16.941 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/ 2023-10-30T12:55:16.941 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 15537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/irqbypass.ko 2023-10-30T12:55:16.941 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 639863 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias 2023-10-30T12:55:16.942 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 623919 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias.bin 2023-10-30T12:55:16.942 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 7663 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin 2023-10-30T12:55:16.942 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 23428 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.alias.bin 2023-10-30T12:55:16.942 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 9821 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.bin 2023-10-30T12:55:16.942 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 70004 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.modinfo 2023-10-30T12:55:16.943 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 146313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep 2023-10-30T12:55:16.943 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 232548 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep.bin 2023-10-30T12:55:16.943 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 210 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.devname 2023-10-30T12:55:16.943 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 72606 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.order 2023-10-30T12:55:16.943 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 879 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.softdep 2023-10-30T12:55:16.944 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 254438 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols 2023-10-30T12:55:16.944 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 309363 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols.bin 2023-10-30T12:55:16.944 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/ 2023-10-30T12:55:16.944 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/ 2023-10-30T12:55:16.944 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/ 2023-10-30T12:55:16.944 INFO:teuthology.orchestra.run.smithi148.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T12:55:16.945 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 173 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/changelog.Debian.gz 2023-10-30T12:55:16.945 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r-- root/root 691 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/copyright 2023-10-30T12:55:16.946 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:16.946 INFO:teuthology.task.kernel:searching for kernel 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:16.946 DEBUG:teuthology.orchestra.run.smithi148:> egrep '(submenu|menuentry.*6.6.0-rc5-geacf00cc51e2).*{' /boot/grub/grub.cfg 2023-10-30T12:55:16.954 INFO:teuthology.orchestra.run.smithi148.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T12:55:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T12:55:16.954 INFO:teuthology.orchestra.run.smithi148.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T12:55:16.955 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-10-30T12:55:16.955 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:16.956 DEBUG:teuthology.orchestra.run.smithi148:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-10-30T12:55:16.998 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi148.front.sepia.ceph.com to complete... 2023-10-30T12:55:17.406 INFO:teuthology.orchestra.run.smithi148.stderr:Sourcing file `/etc/default/grub' 2023-10-30T12:55:17.407 INFO:teuthology.orchestra.run.smithi148.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T12:55:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2023-10-30T12:55:17.620 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:17.626 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:55:17.869 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T12:55:17.875 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T12:55:17.921 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T12:55:17.927 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T12:55:18.078 INFO:teuthology.orchestra.run.smithi148.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T12:55:18.079 INFO:teuthology.orchestra.run.smithi148.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T12:55:18.079 INFO:teuthology.orchestra.run.smithi148.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T12:55:18.092 INFO:teuthology.orchestra.run.smithi148.stderr:done 2023-10-30T12:55:49.236 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-30T12:55:49.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T12:55:49.237 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:56:07.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-10-30T12:56:16.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T12:56:16.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:56:19.866 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-10-30T12:56:31.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T12:56:31.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:57:31.870 DEBUG:teuthology.orchestra.remote:timed out 2023-10-30T12:57:46.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T12:57:46.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:57:47.168 DEBUG:teuthology.orchestra.run.smithi148:> true 2023-10-30T12:57:47.820 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T12:57:47.821 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-geacf00cc51e2"... 2023-10-30T12:57:47.821 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-10-30T12:57:47.870 INFO:teuthology.orchestra.run.smithi148.stdout:6.6.0-rc5-geacf00cc51e2 2023-10-30T12:57:47.871 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-geacf00cc51e2 vs 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:57:47.871 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T12:57:47.871 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-30T12:57:48.872 DEBUG:teuthology.parallel:result is None 2023-10-30T12:57:48.873 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T12:57:48.881 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T12:57:48.882 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T12:57:48.885 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T12:57:48.892 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T12:57:48.898 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T12:57:48.907 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T12:57:48.907 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T12:57:48.943 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T12:57:49.161 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T12:57:49.168 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T12:57:49.169 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T12:57:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:57:49.174 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T12:57:49.178 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T12:57:49.239 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:57:49.251 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:57:49.274 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:57:49.298 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:57:49.301 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T12:57:49.308 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T12:57:49.308 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T12:57:49.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T12:57:49.331 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T12:57:49.339 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T12:57:49.340 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T12:57:49.378 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T12:57:49.384 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T12:57:49.435 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T12:57:49.487 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T12:57:49.488 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T12:57:49.563 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T12:57:49.570 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T12:57:49.616 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T12:57:49.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T12:57:49.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2023-10-30T12:57:49.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2023-10-30T12:57:49.792 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T12:57:49.799 INFO:teuthology.task.internal:Starting timer... 2023-10-30T12:57:49.800 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T12:57:49.808 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T12:57:49.817 DEBUG:teuthology.task.selinux:Excluding smithi148: OS 'ubuntu' does not support SELinux 2023-10-30T12:57:49.818 DEBUG:teuthology.task.selinux:Excluding smithi150: OS 'ubuntu' does not support SELinux 2023-10-30T12:57:49.818 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T12:57:49.818 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-30T12:57:49.819 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T12:57:49.819 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T12:57:49.827 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-30T12:57:49.832 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-30T12:57:49.946 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T12:57:49.964 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T12:57:49.966 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi148.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T13:06:17.058 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi148.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2023-10-30T13:06:17.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T13:06:17.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:06:17.140 DEBUG:teuthology.orchestra.run.smithi148:> true 2023-10-30T13:06:17.253 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-30T13:06:17.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-30T13:06:17.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T13:06:17.332 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-10-30T13:06:17.442 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-30T13:06:17.443 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T13:06:17.453 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T13:06:17.453 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T13:06:17.454 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:06:17.457 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T13:06:17.457 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:06:17.493 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T13:06:17.493 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: Command line: ntpd -gq 2023-10-30T13:06:17.493 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: ---------------------------------------------------- 2023-10-30T13:06:17.493 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T13:06:17.494 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T13:06:17.494 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: corporation. Support and training for ntp-4 are 2023-10-30T13:06:17.494 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: available at https://www.nwtime.org/support 2023-10-30T13:06:17.494 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: ---------------------------------------------------- 2023-10-30T13:06:17.495 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: proto: precision = 0.043 usec (-24) 2023-10-30T13:06:17.495 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: basedate set to 2022-02-04 2023-10-30T13:06:17.495 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: gps base set to 2022-02-06 (week 2196) 2023-10-30T13:06:17.496 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:06:17.496 INFO:teuthology.orchestra.run.smithi148.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:06:17.496 INFO:teuthology.orchestra.run.smithi148.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:06:17.497 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:06:17.497 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T13:06:17.498 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T13:06:17.498 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T13:06:17.498 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: Listen normally on 3 enp3s0f1 172.21.15.148:123 2023-10-30T13:06:17.498 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: Listen normally on 4 lo [::1]:123 2023-10-30T13:06:17.498 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7257%5]:123 2023-10-30T13:06:17.499 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:17 ntpd[27959]: Listening on routing socket on fd #22 for interface updates 2023-10-30T13:06:17.511 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T13:06:17.512 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: Command line: ntpd -gq 2023-10-30T13:06:17.512 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: ---------------------------------------------------- 2023-10-30T13:06:17.512 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T13:06:17.512 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T13:06:17.512 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: corporation. Support and training for ntp-4 are 2023-10-30T13:06:17.513 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: available at https://www.nwtime.org/support 2023-10-30T13:06:17.513 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: ---------------------------------------------------- 2023-10-30T13:06:17.514 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: proto: precision = 0.060 usec (-24) 2023-10-30T13:06:17.514 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: basedate set to 2022-02-04 2023-10-30T13:06:17.514 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: gps base set to 2022-02-06 (week 2196) 2023-10-30T13:06:17.514 INFO:teuthology.orchestra.run.smithi150.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:06:17.515 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T13:06:17.515 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:06:17.516 INFO:teuthology.orchestra.run.smithi150.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T13:06:17.517 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T13:06:17.517 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T13:06:17.517 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T13:06:17.517 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: Listen normally on 3 enp3s0f1 172.21.15.150:123 2023-10-30T13:06:17.517 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: Listen normally on 4 lo [::1]:123 2023-10-30T13:06:17.518 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e35%5]:123 2023-10-30T13:06:17.518 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:17 ntpd[28408]: Listening on routing socket on fd #22 for interface updates 2023-10-30T13:06:18.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:18 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:18.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:18 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:18.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:18 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:18.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:18 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:18.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:18 ntpd[27959]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:18.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:18 ntpd[27959]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:18.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:18 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:18.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:18 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:18.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:18 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:18.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:18 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:18.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:18 ntpd[28408]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:18.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:18 ntpd[28408]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:19.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:19 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:19.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:19 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:19.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:19 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:19.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:19 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:19.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:19 ntpd[28408]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:19.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:19 ntpd[28408]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:20.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:20.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:20.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:20.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:20.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:20.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:20.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:20.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:20.497 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:20.497 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:20.497 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:20.497 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:20.497 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:20.498 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:20 ntpd[27959]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:20.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:20 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:20.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:20 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:20.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:20 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:20.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:20 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:20.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:20 ntpd[28408]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:20.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:20 ntpd[28408]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:21.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:21 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:21.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:21 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:21.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:21 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:21.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:21 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:21.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:21 ntpd[27959]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:21.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:21 ntpd[27959]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:21.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:21 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:21.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:21 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:21.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:21 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:21.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:21 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:21.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:21 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:21.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:21 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:21.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:21 ntpd[28408]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:21.514 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:21 ntpd[28408]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:22.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:22.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:22.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:22.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:22.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:22.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:22.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:22.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:22.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:22.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:22.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:22.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:22 ntpd[27959]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:22.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:22.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:22.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:22.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:22.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:22.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:22.514 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:22.514 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:22.514 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:22.514 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:22.514 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:22 ntpd[28408]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:23 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:23 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:23 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:23 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:23 ntpd[27959]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:23 ntpd[27959]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:23.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:23 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:23.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:23 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:23.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:23 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:23.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:23 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:23.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:23 ntpd[28408]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:23.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:23 ntpd[28408]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:24.494 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:24.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:24.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:24.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:24.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:24.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:24.496 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:24.497 INFO:teuthology.orchestra.run.smithi148.stdout:30 Oct 13:06:24 ntpd[27959]: ntpd: time slew -0.001754 s 2023-10-30T13:06:24.497 INFO:teuthology.orchestra.run.smithi148.stdout:ntpd: time slew -0.001754s 2023-10-30T13:06:24.498 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T13:06:24.498 INFO:teuthology.orchestra.run.smithi148.stderr:30 Oct 13:06:24 ntpd[27959]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T13:06:24.512 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:24.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:24.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:24.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:24.513 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T13:06:24.514 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T13:06:24.514 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T13:06:24.514 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T13:06:24.515 INFO:teuthology.orchestra.run.smithi150.stdout:30 Oct 13:06:24 ntpd[28408]: ntpd: time slew -0.001668 s 2023-10-30T13:06:24.515 INFO:teuthology.orchestra.run.smithi150.stdout:ntpd: time slew -0.001668s 2023-10-30T13:06:24.515 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T13:06:24.515 INFO:teuthology.orchestra.run.smithi150.stderr:30 Oct 13:06:24 ntpd[28408]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T13:06:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:06:24.590 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================== 2023-10-30T13:06:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:06:24.590 INFO:teuthology.orchestra.run.smithi148.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:06:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:06:24.591 INFO:teuthology.orchestra.run.smithi148.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:06:24.611 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:06:24.611 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2023-10-30T13:06:24.612 INFO:teuthology.orchestra.run.smithi150.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:06:24.612 INFO:teuthology.orchestra.run.smithi150.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:06:24.612 INFO:teuthology.orchestra.run.smithi150.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:06:24.612 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T13:06:24.614 INFO:teuthology.run_tasks:Running task install... 2023-10-30T13:06:24.623 DEBUG:teuthology.task.install:project ceph 2023-10-30T13:06:24.623 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}} 2023-10-30T13:06:24.624 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'} 2023-10-30T13:06:24.624 INFO:teuthology.task.install:Using flavor: default 2023-10-30T13:06:24.630 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-30T13:06:24.630 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-30T13:06:24.631 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-key list | grep Ceph 2023-10-30T13:06:24.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-key list | grep Ceph 2023-10-30T13:06:24.711 INFO:teuthology.orchestra.run.smithi150.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T13:06:24.737 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T13:06:24.737 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T13:06:24.738 INFO:teuthology.orchestra.run.smithi148.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T13:06:24.739 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T13:06:24.739 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T13:06:24.741 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:06:24.790 INFO:teuthology.orchestra.run.smithi148.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T13:06:24.791 INFO:teuthology.orchestra.run.smithi148.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T13:06:24.792 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T13:06:24.792 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T13:06:24.792 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:06:24.902 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T13:06:24.903 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:06:24.948 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T13:06:24.948 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:06:25.068 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:06:25.069 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T13:06:25.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2023-10-30T13:06:25.119 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:06:25.119 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T13:06:25.139 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get update 2023-10-30T13:06:25.400 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T13:06:25.400 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T13:06:25.401 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T13:06:25.457 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T13:06:25.459 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T13:06:25.459 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T13:06:25.471 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T13:06:25.505 INFO:teuthology.orchestra.run.smithi148.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T13:06:25.518 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T13:06:25.529 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T13:06:25.553 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T13:06:25.577 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T13:06:25.578 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T13:06:25.610 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T13:06:25.635 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T13:06:25.635 INFO:teuthology.orchestra.run.smithi148.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T13:06:25.693 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T13:06:25.755 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 28.5 kB in 0s (57.3 kB/s) 2023-10-30T13:06:25.764 INFO:teuthology.orchestra.run.smithi148.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T13:06:26.064 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 28.5 kB in 1s (40.8 kB/s) 2023-10-30T13:06:26.590 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-30T13:06:26.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:06:26.668 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-30T13:06:26.854 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2023-10-30T13:06:26.855 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2023-10-30T13:06:26.984 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2023-10-30T13:06:26.999 DEBUG:teuthology.orchestra.run.smithi148:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:06:27.022 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:06:27.022 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:06:27.022 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:06:27.023 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:06:27.023 INFO:teuthology.orchestra.run.smithi150.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T13:06:27.023 INFO:teuthology.orchestra.run.smithi150.stdout: python2.7-minimal 2023-10-30T13:06:27.024 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:06:27.025 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2023-10-30T13:06:27.025 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T13:06:27.025 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T13:06:27.025 INFO:teuthology.orchestra.run.smithi150.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:06:27.026 INFO:teuthology.orchestra.run.smithi150.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T13:06:27.026 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T13:06:27.026 INFO:teuthology.orchestra.run.smithi150.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T13:06:27.026 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T13:06:27.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T13:06:27.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T13:06:27.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T13:06:27.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T13:06:27.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T13:06:27.027 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T13:06:27.028 INFO:teuthology.orchestra.run.smithi150.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T13:06:27.028 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T13:06:27.028 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T13:06:27.028 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T13:06:27.028 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T13:06:27.029 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T13:06:27.029 INFO:teuthology.orchestra.run.smithi150.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T13:06:27.029 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T13:06:27.029 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T13:06:27.029 INFO:teuthology.orchestra.run.smithi150.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T13:06:27.030 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2023-10-30T13:06:27.030 INFO:teuthology.orchestra.run.smithi150.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T13:06:27.030 INFO:teuthology.orchestra.run.smithi150.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T13:06:27.030 INFO:teuthology.orchestra.run.smithi150.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T13:06:27.031 INFO:teuthology.orchestra.run.smithi150.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T13:06:27.031 INFO:teuthology.orchestra.run.smithi150.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T13:06:27.031 INFO:teuthology.orchestra.run.smithi150.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T13:06:27.031 INFO:teuthology.orchestra.run.smithi150.stdout: python3-watchdog 2023-10-30T13:06:27.031 INFO:teuthology.orchestra.run.smithi150.stdout:Recommended packages: 2023-10-30T13:06:27.031 INFO:teuthology.orchestra.run.smithi150.stdout: btrfs-tools 2023-10-30T13:06:27.074 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2023-10-30T13:06:27.130 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2023-10-30T13:06:27.130 INFO:teuthology.orchestra.run.smithi150.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T13:06:27.130 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T13:06:27.131 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T13:06:27.131 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T13:06:27.131 INFO:teuthology.orchestra.run.smithi150.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:06:27.131 INFO:teuthology.orchestra.run.smithi150.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T13:06:27.131 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T13:06:27.131 INFO:teuthology.orchestra.run.smithi150.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T13:06:27.132 INFO:teuthology.orchestra.run.smithi150.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T13:06:27.132 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T13:06:27.132 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T13:06:27.132 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T13:06:27.132 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T13:06:27.132 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T13:06:27.133 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T13:06:27.133 INFO:teuthology.orchestra.run.smithi150.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T13:06:27.133 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T13:06:27.133 INFO:teuthology.orchestra.run.smithi150.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T13:06:27.133 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T13:06:27.134 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T13:06:27.134 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T13:06:27.134 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T13:06:27.134 INFO:teuthology.orchestra.run.smithi150.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T13:06:27.134 INFO:teuthology.orchestra.run.smithi150.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T13:06:27.134 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T13:06:27.135 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T13:06:27.135 INFO:teuthology.orchestra.run.smithi150.stdout: socat xmlstarlet 2023-10-30T13:06:27.193 INFO:teuthology.orchestra.run.smithi150.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T13:06:27.193 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 184 MB of archives. 2023-10-30T13:06:27.193 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T13:06:27.194 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T13:06:27.214 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T13:06:27.225 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T13:06:27.245 INFO:teuthology.orchestra.run.smithi148.stdout:Building dependency tree... 2023-10-30T13:06:27.246 INFO:teuthology.orchestra.run.smithi148.stdout:Reading state information... 2023-10-30T13:06:27.274 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T13:06:27.286 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T13:06:27.307 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T13:06:27.310 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T13:06:27.321 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T13:06:27.381 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T13:06:27.386 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T13:06:27.398 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T13:06:27.400 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:06:27.400 INFO:teuthology.orchestra.run.smithi148.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:06:27.401 INFO:teuthology.orchestra.run.smithi148.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:06:27.401 INFO:teuthology.orchestra.run.smithi148.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:06:27.401 INFO:teuthology.orchestra.run.smithi148.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T13:06:27.402 INFO:teuthology.orchestra.run.smithi148.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T13:06:27.402 INFO:teuthology.orchestra.run.smithi148.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:06:27.402 INFO:teuthology.orchestra.run.smithi148.stdout:The following additional packages will be installed: 2023-10-30T13:06:27.403 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T13:06:27.403 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T13:06:27.403 INFO:teuthology.orchestra.run.smithi148.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:06:27.403 INFO:teuthology.orchestra.run.smithi148.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T13:06:27.404 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T13:06:27.404 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T13:06:27.405 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T13:06:27.405 INFO:teuthology.orchestra.run.smithi148.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T13:06:27.405 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T13:06:27.406 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T13:06:27.406 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T13:06:27.406 INFO:teuthology.orchestra.run.smithi148.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T13:06:27.406 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T13:06:27.406 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T13:06:27.407 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T13:06:27.407 INFO:teuthology.orchestra.run.smithi148.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T13:06:27.407 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T13:06:27.407 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T13:06:27.407 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T13:06:27.408 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T13:06:27.408 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T13:06:27.408 INFO:teuthology.orchestra.run.smithi148.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T13:06:27.408 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T13:06:27.408 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T13:06:27.408 INFO:teuthology.orchestra.run.smithi148.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T13:06:27.409 INFO:teuthology.orchestra.run.smithi148.stdout:Suggested packages: 2023-10-30T13:06:27.409 INFO:teuthology.orchestra.run.smithi148.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T13:06:27.409 INFO:teuthology.orchestra.run.smithi148.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T13:06:27.409 INFO:teuthology.orchestra.run.smithi148.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T13:06:27.409 INFO:teuthology.orchestra.run.smithi148.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T13:06:27.410 INFO:teuthology.orchestra.run.smithi148.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T13:06:27.410 INFO:teuthology.orchestra.run.smithi148.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T13:06:27.410 INFO:teuthology.orchestra.run.smithi148.stdout: python3-watchdog 2023-10-30T13:06:27.410 INFO:teuthology.orchestra.run.smithi148.stdout:Recommended packages: 2023-10-30T13:06:27.410 INFO:teuthology.orchestra.run.smithi148.stdout: btrfs-tools 2023-10-30T13:06:27.412 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T13:06:27.412 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T13:06:27.413 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T13:06:27.414 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T13:06:27.415 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T13:06:27.436 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T13:06:27.437 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T13:06:27.437 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T13:06:27.438 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T13:06:27.438 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T13:06:27.441 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T13:06:27.457 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T13:06:27.457 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T13:06:27.458 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T13:06:27.459 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T13:06:27.477 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T13:06:27.477 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T13:06:27.478 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T13:06:27.479 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T13:06:27.479 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T13:06:27.485 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T13:06:27.497 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T13:06:27.497 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T13:06:27.497 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T13:06:27.499 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T13:06:27.516 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T13:06:27.517 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T13:06:27.518 INFO:teuthology.orchestra.run.smithi150.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T13:06:27.519 INFO:teuthology.orchestra.run.smithi150.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T13:06:27.522 INFO:teuthology.orchestra.run.smithi150.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T13:06:27.524 INFO:teuthology.orchestra.run.smithi150.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T13:06:27.541 INFO:teuthology.orchestra.run.smithi150.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T13:06:27.546 INFO:teuthology.orchestra.run.smithi148.stdout:The following NEW packages will be installed: 2023-10-30T13:06:27.546 INFO:teuthology.orchestra.run.smithi148.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T13:06:27.546 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T13:06:27.547 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T13:06:27.548 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T13:06:27.548 INFO:teuthology.orchestra.run.smithi148.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T13:06:27.548 INFO:teuthology.orchestra.run.smithi148.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T13:06:27.549 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T13:06:27.549 INFO:teuthology.orchestra.run.smithi148.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T13:06:27.549 INFO:teuthology.orchestra.run.smithi148.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T13:06:27.549 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T13:06:27.549 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T13:06:27.549 INFO:teuthology.orchestra.run.smithi148.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T13:06:27.550 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T13:06:27.550 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T13:06:27.550 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T13:06:27.550 INFO:teuthology.orchestra.run.smithi148.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T13:06:27.550 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T13:06:27.550 INFO:teuthology.orchestra.run.smithi148.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T13:06:27.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T13:06:27.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T13:06:27.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T13:06:27.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T13:06:27.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T13:06:27.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T13:06:27.552 INFO:teuthology.orchestra.run.smithi148.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T13:06:27.552 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T13:06:27.552 INFO:teuthology.orchestra.run.smithi148.stdout: socat xmlstarlet 2023-10-30T13:06:27.556 INFO:teuthology.orchestra.run.smithi150.stdout:Get:46 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T13:06:27.615 INFO:teuthology.orchestra.run.smithi150.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T13:06:27.616 INFO:teuthology.orchestra.run.smithi150.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T13:06:27.618 INFO:teuthology.orchestra.run.smithi150.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T13:06:27.618 INFO:teuthology.orchestra.run.smithi150.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T13:06:27.619 INFO:teuthology.orchestra.run.smithi150.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T13:06:27.620 INFO:teuthology.orchestra.run.smithi150.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T13:06:27.649 INFO:teuthology.orchestra.run.smithi150.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T13:06:27.653 INFO:teuthology.orchestra.run.smithi150.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T13:06:27.653 INFO:teuthology.orchestra.run.smithi150.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T13:06:27.677 INFO:teuthology.orchestra.run.smithi148.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T13:06:27.677 INFO:teuthology.orchestra.run.smithi148.stdout:Need to get 184 MB of archives. 2023-10-30T13:06:27.677 INFO:teuthology.orchestra.run.smithi148.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T13:06:27.678 INFO:teuthology.orchestra.run.smithi148.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T13:06:27.682 INFO:teuthology.orchestra.run.smithi150.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T13:06:27.689 INFO:teuthology.orchestra.run.smithi150.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T13:06:27.690 INFO:teuthology.orchestra.run.smithi150.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T13:06:27.690 INFO:teuthology.orchestra.run.smithi150.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T13:06:27.691 INFO:teuthology.orchestra.run.smithi150.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T13:06:27.691 INFO:teuthology.orchestra.run.smithi150.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T13:06:27.692 INFO:teuthology.orchestra.run.smithi150.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T13:06:27.692 INFO:teuthology.orchestra.run.smithi150.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T13:06:27.700 INFO:teuthology.orchestra.run.smithi150.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T13:06:27.713 INFO:teuthology.orchestra.run.smithi148.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T13:06:27.722 INFO:teuthology.orchestra.run.smithi150.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T13:06:27.723 INFO:teuthology.orchestra.run.smithi150.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T13:06:27.724 INFO:teuthology.orchestra.run.smithi150.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T13:06:27.729 INFO:teuthology.orchestra.run.smithi150.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T13:06:27.732 INFO:teuthology.orchestra.run.smithi150.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T13:06:27.733 INFO:teuthology.orchestra.run.smithi150.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T13:06:27.739 INFO:teuthology.orchestra.run.smithi150.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T13:06:27.743 INFO:teuthology.orchestra.run.smithi150.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T13:06:27.743 INFO:teuthology.orchestra.run.smithi150.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T13:06:27.744 INFO:teuthology.orchestra.run.smithi148.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T13:06:27.759 INFO:teuthology.orchestra.run.smithi150.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T13:06:27.759 INFO:teuthology.orchestra.run.smithi150.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T13:06:27.761 INFO:teuthology.orchestra.run.smithi150.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T13:06:27.762 INFO:teuthology.orchestra.run.smithi150.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T13:06:27.772 INFO:teuthology.orchestra.run.smithi150.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T13:06:27.772 INFO:teuthology.orchestra.run.smithi150.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T13:06:27.783 INFO:teuthology.orchestra.run.smithi148.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T13:06:27.786 INFO:teuthology.orchestra.run.smithi150.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T13:06:27.789 INFO:teuthology.orchestra.run.smithi150.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T13:06:27.789 INFO:teuthology.orchestra.run.smithi150.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T13:06:27.819 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T13:06:27.838 INFO:teuthology.orchestra.run.smithi148.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T13:06:27.864 INFO:teuthology.orchestra.run.smithi148.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T13:06:27.867 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T13:06:27.889 INFO:teuthology.orchestra.run.smithi148.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T13:06:27.956 INFO:teuthology.orchestra.run.smithi148.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T13:06:27.959 INFO:teuthology.orchestra.run.smithi148.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T13:06:27.973 INFO:teuthology.orchestra.run.smithi148.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T13:06:27.978 INFO:teuthology.orchestra.run.smithi148.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T13:06:27.979 INFO:teuthology.orchestra.run.smithi148.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T13:06:27.988 INFO:teuthology.orchestra.run.smithi148.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T13:06:27.989 INFO:teuthology.orchestra.run.smithi148.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T13:06:27.989 INFO:teuthology.orchestra.run.smithi148.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T13:06:27.990 INFO:teuthology.orchestra.run.smithi148.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T13:06:27.991 INFO:teuthology.orchestra.run.smithi148.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T13:06:28.016 INFO:teuthology.orchestra.run.smithi148.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T13:06:28.016 INFO:teuthology.orchestra.run.smithi148.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T13:06:28.017 INFO:teuthology.orchestra.run.smithi148.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T13:06:28.018 INFO:teuthology.orchestra.run.smithi148.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T13:06:28.018 INFO:teuthology.orchestra.run.smithi148.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T13:06:28.020 INFO:teuthology.orchestra.run.smithi148.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T13:06:28.020 INFO:teuthology.orchestra.run.smithi148.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T13:06:28.021 INFO:teuthology.orchestra.run.smithi148.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T13:06:28.040 INFO:teuthology.orchestra.run.smithi148.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T13:06:28.041 INFO:teuthology.orchestra.run.smithi148.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T13:06:28.060 INFO:teuthology.orchestra.run.smithi148.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T13:06:28.061 INFO:teuthology.orchestra.run.smithi148.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T13:06:28.062 INFO:teuthology.orchestra.run.smithi148.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T13:06:28.063 INFO:teuthology.orchestra.run.smithi148.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T13:06:28.063 INFO:teuthology.orchestra.run.smithi148.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T13:06:28.070 INFO:teuthology.orchestra.run.smithi148.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T13:06:28.070 INFO:teuthology.orchestra.run.smithi148.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T13:06:28.071 INFO:teuthology.orchestra.run.smithi148.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T13:06:28.089 INFO:teuthology.orchestra.run.smithi148.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T13:06:28.090 INFO:teuthology.orchestra.run.smithi148.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T13:06:28.102 INFO:teuthology.orchestra.run.smithi148.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T13:06:28.103 INFO:teuthology.orchestra.run.smithi148.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T13:06:28.104 INFO:teuthology.orchestra.run.smithi148.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T13:06:28.105 INFO:teuthology.orchestra.run.smithi148.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T13:06:28.107 INFO:teuthology.orchestra.run.smithi148.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T13:06:28.108 INFO:teuthology.orchestra.run.smithi148.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T13:06:28.110 INFO:teuthology.orchestra.run.smithi148.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T13:06:28.252 INFO:teuthology.orchestra.run.smithi148.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T13:06:28.253 INFO:teuthology.orchestra.run.smithi148.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T13:06:28.254 INFO:teuthology.orchestra.run.smithi148.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T13:06:28.255 INFO:teuthology.orchestra.run.smithi148.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T13:06:28.255 INFO:teuthology.orchestra.run.smithi148.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T13:06:28.257 INFO:teuthology.orchestra.run.smithi148.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T13:06:28.275 INFO:teuthology.orchestra.run.smithi148.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T13:06:28.277 INFO:teuthology.orchestra.run.smithi148.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T13:06:28.278 INFO:teuthology.orchestra.run.smithi148.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T13:06:28.300 INFO:teuthology.orchestra.run.smithi148.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T13:06:28.307 INFO:teuthology.orchestra.run.smithi148.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T13:06:28.308 INFO:teuthology.orchestra.run.smithi148.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T13:06:28.309 INFO:teuthology.orchestra.run.smithi148.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T13:06:28.309 INFO:teuthology.orchestra.run.smithi148.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T13:06:28.310 INFO:teuthology.orchestra.run.smithi148.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T13:06:28.311 INFO:teuthology.orchestra.run.smithi148.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T13:06:28.311 INFO:teuthology.orchestra.run.smithi148.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T13:06:28.315 INFO:teuthology.orchestra.run.smithi148.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T13:06:28.335 INFO:teuthology.orchestra.run.smithi148.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T13:06:28.337 INFO:teuthology.orchestra.run.smithi148.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T13:06:28.338 INFO:teuthology.orchestra.run.smithi148.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T13:06:28.343 INFO:teuthology.orchestra.run.smithi148.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T13:06:28.346 INFO:teuthology.orchestra.run.smithi148.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T13:06:28.346 INFO:teuthology.orchestra.run.smithi148.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T13:06:28.351 INFO:teuthology.orchestra.run.smithi148.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T13:06:28.363 INFO:teuthology.orchestra.run.smithi148.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T13:06:28.363 INFO:teuthology.orchestra.run.smithi148.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T13:06:28.364 INFO:teuthology.orchestra.run.smithi148.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T13:06:28.386 INFO:teuthology.orchestra.run.smithi148.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T13:06:28.388 INFO:teuthology.orchestra.run.smithi148.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T13:06:28.389 INFO:teuthology.orchestra.run.smithi148.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T13:06:28.399 INFO:teuthology.orchestra.run.smithi148.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T13:06:28.399 INFO:teuthology.orchestra.run.smithi148.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T13:06:28.400 INFO:teuthology.orchestra.run.smithi148.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T13:06:28.402 INFO:teuthology.orchestra.run.smithi148.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T13:06:28.410 INFO:teuthology.orchestra.run.smithi148.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T13:06:28.621 INFO:teuthology.orchestra.run.smithi150.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T13:06:28.796 INFO:teuthology.orchestra.run.smithi148.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T13:06:29.108 INFO:teuthology.orchestra.run.smithi148.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T13:06:29.142 INFO:teuthology.orchestra.run.smithi148.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T13:06:29.166 INFO:teuthology.orchestra.run.smithi148.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T13:06:29.168 INFO:teuthology.orchestra.run.smithi148.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T13:06:29.176 INFO:teuthology.orchestra.run.smithi148.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T13:06:29.178 INFO:teuthology.orchestra.run.smithi148.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T13:06:29.201 INFO:teuthology.orchestra.run.smithi148.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T13:06:29.389 INFO:teuthology.orchestra.run.smithi150.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T13:06:29.553 INFO:teuthology.orchestra.run.smithi150.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T13:06:29.565 INFO:teuthology.orchestra.run.smithi148.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T13:06:29.569 INFO:teuthology.orchestra.run.smithi148.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T13:06:29.590 INFO:teuthology.orchestra.run.smithi148.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T13:06:29.610 INFO:teuthology.orchestra.run.smithi150.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T13:06:29.611 INFO:teuthology.orchestra.run.smithi150.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T13:06:29.615 INFO:teuthology.orchestra.run.smithi150.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T13:06:29.682 INFO:teuthology.orchestra.run.smithi150.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T13:06:29.736 INFO:teuthology.orchestra.run.smithi150.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T13:06:30.678 INFO:teuthology.orchestra.run.smithi148.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T13:06:31.001 INFO:teuthology.orchestra.run.smithi148.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T13:06:31.003 INFO:teuthology.orchestra.run.smithi148.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T13:06:31.004 INFO:teuthology.orchestra.run.smithi148.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T13:06:31.016 INFO:teuthology.orchestra.run.smithi148.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T13:06:31.114 INFO:teuthology.orchestra.run.smithi150.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T13:06:31.175 INFO:teuthology.orchestra.run.smithi150.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T13:06:31.268 INFO:teuthology.orchestra.run.smithi150.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T13:06:31.383 INFO:teuthology.orchestra.run.smithi148.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T13:06:32.585 INFO:teuthology.orchestra.run.smithi148.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T13:06:32.586 INFO:teuthology.orchestra.run.smithi148.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T13:06:32.605 INFO:teuthology.orchestra.run.smithi148.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T13:06:32.686 INFO:teuthology.orchestra.run.smithi148.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T13:06:32.695 INFO:teuthology.orchestra.run.smithi148.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T13:06:32.701 INFO:teuthology.orchestra.run.smithi148.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T13:06:32.773 INFO:teuthology.orchestra.run.smithi148.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T13:06:33.098 INFO:teuthology.orchestra.run.smithi148.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T13:06:33.098 INFO:teuthology.orchestra.run.smithi148.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T13:06:34.892 INFO:teuthology.orchestra.run.smithi148.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T13:06:34.893 INFO:teuthology.orchestra.run.smithi148.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T13:06:34.895 INFO:teuthology.orchestra.run.smithi148.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T13:06:34.895 INFO:teuthology.orchestra.run.smithi148.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T13:06:34.896 INFO:teuthology.orchestra.run.smithi148.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T13:06:34.896 INFO:teuthology.orchestra.run.smithi148.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T13:06:35.354 INFO:teuthology.orchestra.run.smithi148.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T13:06:35.936 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 184 MB in 8s (23.6 MB/s) 2023-10-30T13:06:36.079 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T13:06:36.148 INFO:teuthology.orchestra.run.smithi148.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132932 files and directories currently installed.) 2023-10-30T13:06:36.152 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T13:06:36.194 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T13:06:36.514 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T13:06:36.527 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T13:06:36.572 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:06:36.842 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T13:06:36.856 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T13:06:36.900 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T13:06:37.194 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T13:06:37.209 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T13:06:37.295 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:06:37.510 INFO:teuthology.orchestra.run.smithi150.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T13:06:37.681 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T13:06:37.696 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T13:06:37.739 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:06:38.076 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T13:06:38.090 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T13:06:38.134 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:06:38.412 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T13:06:38.426 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T13:06:38.470 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:06:38.949 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T13:06:38.964 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:06:39.016 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:06:39.128 INFO:teuthology.orchestra.run.smithi150.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T13:06:39.210 INFO:teuthology.orchestra.run.smithi150.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T13:06:39.218 INFO:teuthology.orchestra.run.smithi150.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T13:06:39.410 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T13:06:39.425 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:06:39.468 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:06:39.713 INFO:teuthology.orchestra.run.smithi150.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T13:06:39.797 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T13:06:39.811 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:06:39.863 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:06:40.208 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T13:06:40.223 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T13:06:40.266 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:06:40.886 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librados2. 2023-10-30T13:06:40.901 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:40.944 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:41.338 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librbd1. 2023-10-30T13:06:41.352 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:41.397 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:41.732 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T13:06:41.746 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:41.791 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:42.093 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rados. 2023-10-30T13:06:42.106 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:42.143 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:42.471 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T13:06:42.484 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:06:42.521 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:42.617 INFO:teuthology.orchestra.run.smithi150.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T13:06:42.765 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T13:06:42.778 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:42.823 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:43.076 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T13:06:43.089 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:06:43.134 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:43.437 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T13:06:43.450 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T13:06:43.495 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T13:06:43.740 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T13:06:43.754 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:43.798 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:44.126 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T13:06:44.140 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T13:06:44.175 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:06:44.486 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librgw2. 2023-10-30T13:06:44.501 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:44.545 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:45.022 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T13:06:45.036 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:45.080 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:45.434 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T13:06:45.448 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T13:06:45.491 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:06:45.903 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T13:06:45.917 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:45.953 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:46.373 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-common. 2023-10-30T13:06:46.387 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:46.431 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:46.596 INFO:teuthology.orchestra.run.smithi150.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T13:06:46.596 INFO:teuthology.orchestra.run.smithi150.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T13:06:46.613 INFO:teuthology.orchestra.run.smithi150.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T13:06:46.836 INFO:teuthology.orchestra.run.smithi150.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T13:06:46.869 INFO:teuthology.orchestra.run.smithi150.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T13:06:46.881 INFO:teuthology.orchestra.run.smithi150.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T13:06:47.094 INFO:teuthology.orchestra.run.smithi150.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T13:06:47.511 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-base. 2023-10-30T13:06:47.526 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:47.579 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:48.090 INFO:teuthology.orchestra.run.smithi150.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T13:06:48.091 INFO:teuthology.orchestra.run.smithi150.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T13:06:48.217 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T13:06:48.231 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T13:06:48.276 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:06:48.578 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T13:06:48.592 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T13:06:48.637 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:06:48.998 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T13:06:49.013 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T13:06:49.057 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:06:49.359 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T13:06:49.423 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T13:06:49.423 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:06:49.720 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T13:06:49.735 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T13:06:49.778 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:06:50.065 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T13:06:50.079 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T13:06:50.123 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T13:06:50.426 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-portend. 2023-10-30T13:06:50.440 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T13:06:50.485 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T13:06:50.770 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T13:06:50.785 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T13:06:50.828 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T13:06:51.148 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T13:06:51.162 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T13:06:51.206 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:06:51.526 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T13:06:51.540 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T13:06:51.584 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T13:06:51.887 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T13:06:51.902 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T13:06:51.945 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T13:06:52.298 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T13:06:52.313 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T13:06:52.357 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:06:52.626 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-mako. 2023-10-30T13:06:52.641 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T13:06:52.676 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:06:53.004 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T13:06:53.019 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T13:06:53.062 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:06:53.390 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T13:06:53.406 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T13:06:53.440 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:06:53.743 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-webob. 2023-10-30T13:06:53.758 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T13:06:53.801 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:06:53.849 INFO:teuthology.orchestra.run.smithi150.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T13:06:53.851 INFO:teuthology.orchestra.run.smithi150.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T13:06:53.856 INFO:teuthology.orchestra.run.smithi150.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T13:06:53.857 INFO:teuthology.orchestra.run.smithi150.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T13:06:53.857 INFO:teuthology.orchestra.run.smithi150.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T13:06:53.858 INFO:teuthology.orchestra.run.smithi150.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T13:06:54.145 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T13:06:54.161 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T13:06:54.207 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:06:54.531 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T13:06:54.546 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T13:06:54.590 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:06:54.868 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-paste. 2023-10-30T13:06:54.881 INFO:teuthology.orchestra.run.smithi150.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T13:06:54.883 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T13:06:54.918 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:06:55.212 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T13:06:55.226 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T13:06:55.270 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:06:55.443 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 184 MB in 28s (6,637 kB/s) 2023-10-30T13:06:55.598 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T13:06:55.613 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T13:06:55.620 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T13:06:55.656 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:06:55.705 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 130506 files and directories currently installed.) 2023-10-30T13:06:55.709 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T13:06:55.752 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T13:06:55.988 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T13:06:56.008 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T13:06:56.063 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:06:56.135 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T13:06:56.150 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T13:06:56.193 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T13:06:56.391 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T13:06:56.411 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T13:06:56.449 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T13:06:56.504 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T13:06:56.519 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T13:06:56.562 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:06:56.794 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T13:06:56.814 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T13:06:56.894 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:06:56.934 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T13:06:56.949 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T13:06:56.992 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T13:06:57.272 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T13:06:57.292 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T13:06:57.311 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-idna. 2023-10-30T13:06:57.326 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T13:06:57.330 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:06:57.369 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T13:06:57.681 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T13:06:57.696 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T13:06:57.739 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:06:57.741 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T13:06:57.762 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T13:06:57.808 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:06:58.067 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-requests. 2023-10-30T13:06:58.082 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T13:06:58.117 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:06:58.228 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T13:06:58.249 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T13:06:58.286 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:06:58.395 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T13:06:58.410 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T13:06:58.445 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:06:58.714 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T13:06:58.729 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:06:58.764 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:58.865 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T13:06:58.886 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:06:58.939 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:06:59.152 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T13:06:59.166 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:59.210 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:06:59.452 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T13:06:59.472 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:06:59.518 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:06:59.521 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T13:06:59.535 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T13:06:59.580 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T13:06:59.921 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T13:06:59.933 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T13:06:59.942 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T13:06:59.948 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:06:59.980 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:06:59.991 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:00.386 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T13:07:00.400 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:00.435 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:00.876 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T13:07:00.896 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T13:07:01.425 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:07:01.787 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librados2. 2023-10-30T13:07:01.789 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T13:07:01.804 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:01.808 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:01.845 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:01.847 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:02.364 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librbd1. 2023-10-30T13:07:02.385 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:02.422 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:02.834 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T13:07:02.855 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:02.883 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:03.051 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph. 2023-10-30T13:07:03.066 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:03.117 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:03.153 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rados. 2023-10-30T13:07:03.165 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:03.203 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:03.445 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T13:07:03.460 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:03.503 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:03.564 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T13:07:03.585 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:03.622 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:03.881 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T13:07:03.896 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:03.899 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T13:07:03.912 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:03.939 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:03.949 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:04.219 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T13:07:04.236 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:04.269 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:04.301 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package cephadm. 2023-10-30T13:07:04.316 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:04.359 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:04.606 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T13:07:04.626 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T13:07:04.654 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T13:07:04.664 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T13:07:04.669 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T13:07:04.712 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T13:07:04.966 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T13:07:04.987 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:05.025 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:05.039 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T13:07:05.054 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:07:05.098 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:07:05.346 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T13:07:05.367 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T13:07:05.413 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:07:05.733 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librgw2. 2023-10-30T13:07:05.753 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:05.790 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:06.128 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-babel. 2023-10-30T13:07:06.142 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:07:06.195 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:07:06.276 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T13:07:06.288 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:06.326 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:06.681 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T13:07:06.695 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T13:07:06.704 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T13:07:06.725 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T13:07:06.739 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T13:07:06.762 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:07:07.107 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T13:07:07.109 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T13:07:07.125 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:07.127 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:07.165 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:07.167 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:07.496 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T13:07:07.511 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T13:07:07.554 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:07:07.560 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-common. 2023-10-30T13:07:07.581 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:07.610 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:07.857 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T13:07:07.872 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T13:07:07.915 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T13:07:08.235 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-routes. 2023-10-30T13:07:08.250 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T13:07:08.284 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:07:08.629 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T13:07:08.645 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:08.687 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:09.016 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-base. 2023-10-30T13:07:09.036 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:09.083 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:09.099 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T13:07:09.114 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T13:07:09.158 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:07:09.653 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T13:07:09.667 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T13:07:09.703 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:07:09.911 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T13:07:09.932 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T13:07:09.978 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:07:10.081 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T13:07:10.095 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T13:07:10.141 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:07:10.381 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T13:07:10.401 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T13:07:10.434 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T13:07:10.439 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:07:10.448 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T13:07:10.483 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:07:10.809 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T13:07:10.829 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T13:07:10.900 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:07:11.109 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T13:07:11.123 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:11.167 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:11.412 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T13:07:11.432 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T13:07:11.470 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:07:11.773 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T13:07:11.794 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T13:07:11.823 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:07:12.087 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T13:07:12.100 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T13:07:12.102 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T13:07:12.121 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T13:07:12.136 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T13:07:12.192 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T13:07:12.398 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T13:07:12.414 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T13:07:12.447 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:07:12.495 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-portend. 2023-10-30T13:07:12.516 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T13:07:12.553 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T13:07:12.768 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T13:07:12.783 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T13:07:12.834 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:07:12.881 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T13:07:12.902 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T13:07:12.939 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T13:07:13.137 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T13:07:13.152 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T13:07:13.195 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T13:07:13.234 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T13:07:13.255 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T13:07:13.284 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:07:13.600 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T13:07:13.616 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T13:07:13.637 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T13:07:13.658 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T13:07:13.659 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T13:07:13.720 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T13:07:14.011 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T13:07:14.027 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T13:07:14.032 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T13:07:14.053 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T13:07:14.061 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:07:14.090 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T13:07:14.322 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T13:07:14.338 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T13:07:14.376 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T13:07:14.380 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T13:07:14.388 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T13:07:14.426 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:07:14.709 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T13:07:14.725 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T13:07:14.762 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-mako. 2023-10-30T13:07:14.767 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:07:14.783 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T13:07:14.820 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:07:15.157 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T13:07:15.178 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T13:07:15.233 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:07:15.464 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T13:07:15.478 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:15.522 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:15.535 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T13:07:15.556 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T13:07:15.593 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:07:15.800 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T13:07:15.814 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T13:07:15.858 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:07:15.870 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-webob. 2023-10-30T13:07:15.892 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T13:07:15.920 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:07:16.161 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T13:07:16.176 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:07:16.211 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:07:16.290 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T13:07:16.311 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T13:07:16.351 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:07:16.497 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package jq. 2023-10-30T13:07:16.512 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:07:16.555 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T13:07:16.660 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T13:07:16.681 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T13:07:16.718 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:07:16.850 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package socat. 2023-10-30T13:07:16.866 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T13:07:16.908 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:07:16.996 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-paste. 2023-10-30T13:07:17.008 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T13:07:17.037 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:07:17.195 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T13:07:17.210 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T13:07:17.253 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:07:17.407 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T13:07:17.419 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T13:07:17.457 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:07:17.615 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-test. 2023-10-30T13:07:17.629 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:17.673 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:17.753 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T13:07:17.772 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T13:07:17.830 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:07:18.138 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T13:07:18.158 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T13:07:18.196 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T13:07:18.499 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T13:07:18.520 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T13:07:18.549 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:07:18.912 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T13:07:18.933 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T13:07:18.962 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T13:07:19.307 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-idna. 2023-10-30T13:07:19.327 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T13:07:19.406 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T13:07:19.410 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T13:07:19.426 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:19.469 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:19.726 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T13:07:19.746 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T13:07:19.784 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:07:19.856 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T13:07:19.872 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:19.914 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:20.129 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-requests. 2023-10-30T13:07:20.150 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T13:07:20.187 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:07:20.235 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T13:07:20.250 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T13:07:20.292 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T13:07:20.523 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T13:07:20.537 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T13:07:20.544 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T13:07:20.552 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:20.581 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:07:20.586 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:20.823 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T13:07:20.838 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:20.881 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:20.918 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T13:07:20.939 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:20.976 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:21.175 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T13:07:21.191 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:21.234 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:21.373 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T13:07:21.394 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:21.456 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:21.728 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T13:07:21.744 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T13:07:21.767 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T13:07:21.788 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T13:07:21.826 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T13:07:22.006 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:07:22.366 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T13:07:22.370 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T13:07:22.381 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T13:07:22.390 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:22.424 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:07:22.428 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:22.815 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T13:07:22.835 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:22.852 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T13:07:22.867 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T13:07:22.873 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:22.910 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T13:07:23.239 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T13:07:23.254 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T13:07:23.296 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T13:07:23.625 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T13:07:23.641 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T13:07:23.683 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:07:23.810 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T13:07:23.830 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:23.868 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:23.944 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T13:07:23.959 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T13:07:23.994 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T13:07:24.313 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-py. 2023-10-30T13:07:24.329 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T13:07:24.388 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T13:07:24.716 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T13:07:24.732 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T13:07:24.770 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:07:25.231 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T13:07:25.245 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T13:07:25.289 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:07:25.608 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-toml. 2023-10-30T13:07:25.622 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T13:07:25.666 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T13:07:25.804 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph. 2023-10-30T13:07:25.825 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:25.862 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:25.969 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T13:07:25.983 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T13:07:26.027 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:07:26.190 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T13:07:26.211 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:26.364 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T13:07:26.378 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T13:07:26.422 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:07:26.457 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:26.733 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T13:07:26.748 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T13:07:26.791 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:07:26.852 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T13:07:26.873 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:26.910 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:27.347 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephadm. 2023-10-30T13:07:27.367 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:27.404 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:27.654 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package radosgw. 2023-10-30T13:07:27.668 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:27.728 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:27.741 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T13:07:27.762 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T13:07:27.799 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T13:07:28.160 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T13:07:28.181 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:07:28.218 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:07:28.306 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T13:07:28.321 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:28.356 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:28.615 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T13:07:28.895 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:07:29.020 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:07:29.286 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T13:07:29.332 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-babel. 2023-10-30T13:07:29.353 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T13:07:29.390 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:07:29.692 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T13:07:29.743 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T13:07:29.764 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T13:07:29.801 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T13:07:30.146 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T13:07:30.167 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:30.221 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:30.316 INFO:teuthology.orchestra.run.smithi148.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T13:07:30.330 INFO:teuthology.orchestra.run.smithi148.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-10-30T13:07:30.364 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:30.567 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T13:07:30.587 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T13:07:30.625 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:07:30.842 INFO:teuthology.orchestra.run.smithi148.stdout:Adding system user cephadm....done 2023-10-30T13:07:30.936 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T13:07:30.956 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T13:07:30.985 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T13:07:31.003 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:07:31.280 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:07:31.305 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-routes. 2023-10-30T13:07:31.326 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T13:07:31.372 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:07:31.522 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:07:31.648 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:07:31.725 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T13:07:31.746 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:31.792 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:31.891 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T13:07:32.150 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:07:32.245 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T13:07:32.266 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T13:07:32.301 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T13:07:32.303 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:07:32.612 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:07:32.883 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T13:07:32.904 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T13:07:32.949 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:07:32.980 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T13:07:33.240 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T13:07:33.344 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T13:07:33.366 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T13:07:33.410 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:07:33.499 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T13:07:33.705 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T13:07:33.727 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T13:07:33.767 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:07:33.796 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:07:34.026 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:07:34.294 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:07:34.455 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T13:07:34.477 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:34.522 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:34.704 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:34.971 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:07:35.081 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T13:07:35.357 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:07:35.583 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T13:07:35.604 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T13:07:35.616 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:07:35.641 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T13:07:35.759 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T13:07:35.994 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T13:07:36.016 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T13:07:36.052 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:07:36.127 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:07:36.253 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T13:07:36.448 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T13:07:36.469 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T13:07:36.506 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:07:36.529 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:07:36.805 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:07:36.840 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T13:07:36.861 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T13:07:36.908 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T13:07:37.123 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:07:37.314 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T13:07:37.335 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T13:07:37.388 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T13:07:37.709 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T13:07:37.730 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T13:07:37.767 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:07:37.817 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:07:38.101 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:07:38.103 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T13:07:38.124 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T13:07:38.161 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T13:07:38.236 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:07:38.361 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:07:38.506 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T13:07:38.528 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T13:07:38.564 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:07:38.671 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T13:07:39.072 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T13:07:39.320 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T13:07:39.331 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:07:39.333 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:39.370 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:39.723 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T13:07:39.725 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:07:39.744 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T13:07:39.798 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:07:40.026 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T13:07:40.168 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T13:07:40.189 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:07:40.234 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:07:40.377 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:07:40.478 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package jq. 2023-10-30T13:07:40.499 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T13:07:40.561 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T13:07:40.753 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:07:40.814 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package socat. 2023-10-30T13:07:40.826 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T13:07:40.863 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:07:41.013 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:07:41.139 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:41.217 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T13:07:41.239 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T13:07:41.284 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:07:41.465 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T13:07:41.663 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-test. 2023-10-30T13:07:41.675 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:41.712 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:41.816 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:07:43.039 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:07:43.173 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T13:07:43.441 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:07:43.592 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:07:43.718 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:07:44.027 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:07:44.150 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T13:07:44.172 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:44.216 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:44.287 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T13:07:44.546 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:07:44.638 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T13:07:44.659 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:44.697 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:07:44.704 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:44.823 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:07:45.024 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T13:07:45.045 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T13:07:45.082 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T13:07:45.141 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:07:45.369 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T13:07:45.380 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:45.401 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T13:07:45.418 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:45.677 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T13:07:45.704 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T13:07:45.726 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T13:07:45.771 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:45.970 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:07:46.032 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T13:07:46.044 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:46.082 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:46.229 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T13:07:46.435 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T13:07:46.457 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T13:07:46.497 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:07:46.501 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:07:46.623 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T13:07:46.890 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T13:07:46.902 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T13:07:46.907 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:07:46.941 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:07:47.025 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:07:47.309 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:07:47.395 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T13:07:47.416 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T13:07:47.461 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T13:07:47.627 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:07:47.797 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T13:07:47.819 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T13:07:47.864 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T13:07:47.887 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:07:48.159 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T13:07:48.180 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T13:07:48.233 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:07:48.344 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T13:07:48.495 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T13:07:48.507 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T13:07:48.536 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T13:07:48.598 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:07:48.724 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:07:48.906 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-py. 2023-10-30T13:07:48.927 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T13:07:48.972 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T13:07:49.117 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:07:49.326 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T13:07:49.348 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T13:07:49.385 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T13:07:49.392 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:07:49.653 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:07:49.762 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T13:07:49.880 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T13:07:49.901 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T13:07:49.946 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:07:50.071 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:07:50.283 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-toml. 2023-10-30T13:07:50.305 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T13:07:50.349 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T13:07:50.381 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:07:50.652 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T13:07:50.665 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T13:07:50.674 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T13:07:50.718 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:07:50.791 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T13:07:51.068 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T13:07:51.089 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T13:07:51.110 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T13:07:51.155 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:07:51.352 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:07:51.483 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T13:07:51.505 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T13:07:51.549 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:07:51.637 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:07:52.022 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T13:07:52.323 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:07:52.345 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package radosgw. 2023-10-30T13:07:52.368 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:52.403 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:52.675 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:07:52.801 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:52.935 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:53.138 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:53.198 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T13:07:53.210 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T13:07:53.247 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:53.264 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:53.390 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:53.516 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T13:07:53.599 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T13:07:53.809 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:53.853 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T13:07:53.936 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:54.004 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T13:07:54.061 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:54.265 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T13:07:54.289 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:07:54.289 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:07:54.610 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T13:07:54.913 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:55.039 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:55.060 INFO:teuthology.orchestra.run.smithi150.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T13:07:55.165 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:55.172 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:55.291 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:07:55.765 INFO:teuthology.orchestra.run.smithi150.stdout:Adding system user cephadm....done 2023-10-30T13:07:55.920 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T13:07:56.222 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T13:07:56.464 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T13:07:56.562 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:56.607 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T13:07:56.663 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:56.789 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:56.867 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T13:07:56.915 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:57.066 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:57.126 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T13:07:57.192 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:57.253 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T13:07:57.587 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T13:07:57.746 INFO:teuthology.orchestra.run.smithi148.stdout:Adding group ceph....done 2023-10-30T13:07:57.939 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T13:07:58.088 INFO:teuthology.orchestra.run.smithi148.stdout:Adding system user ceph....done 2023-10-30T13:07:58.174 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T13:07:58.283 INFO:teuthology.orchestra.run.smithi148.stdout:Setting system user ceph properties....done 2023-10-30T13:07:58.298 INFO:teuthology.orchestra.run.smithi148.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T13:07:58.425 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T13:07:58.437 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T13:07:58.710 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T13:07:58.823 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T13:07:58.961 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T13:07:59.220 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T13:07:59.488 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:59.544 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:07:59.751 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:07:59.752 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:07:59.872 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T13:07:59.973 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T13:08:00.233 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T13:08:00.378 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:00.493 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T13:08:00.529 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:00.652 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T13:08:01.012 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T13:08:01.043 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:08:01.043 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:08:01.162 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T13:08:01.430 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T13:08:01.664 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:01.690 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T13:08:01.983 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T13:08:02.025 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T13:08:02.626 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T13:08:02.742 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:02.919 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T13:08:02.976 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:08:02.976 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:08:03.045 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T13:08:03.197 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T13:08:03.472 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T13:08:03.713 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:03.823 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T13:08:03.925 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:08:03.926 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:08:04.075 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T13:08:04.444 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T13:08:04.664 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:04.803 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T13:08:04.975 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:08:04.976 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:08:05.113 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T13:08:05.439 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T13:08:05.582 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:05.665 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T13:08:05.708 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:05.808 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:05.851 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:06.061 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:08:06.061 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:08:06.101 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T13:08:06.418 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T13:08:06.686 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:06.812 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:06.930 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:07.072 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:07.417 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:08:07.549 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:08:07.692 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T13:08:07.985 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T13:08:08.111 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T13:08:08.254 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T13:08:08.580 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T13:08:08.754 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:08:08.839 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T13:08:09.090 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:08:09.216 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T13:08:09.223 INFO:teuthology.orchestra.run.smithi148.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:08:09.227 DEBUG:teuthology.orchestra.run.smithi148:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T13:08:09.302 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2023-10-30T13:08:09.376 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T13:08:09.469 INFO:teuthology.orchestra.run.smithi148.stdout:Building dependency tree... 2023-10-30T13:08:09.469 INFO:teuthology.orchestra.run.smithi148.stdout:Reading state information... 2023-10-30T13:08:09.620 INFO:teuthology.orchestra.run.smithi148.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T13:08:09.620 INFO:teuthology.orchestra.run.smithi148.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T13:08:09.620 INFO:teuthology.orchestra.run.smithi148.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:08:09.620 INFO:teuthology.orchestra.run.smithi148.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:08:09.621 INFO:teuthology.orchestra.run.smithi148.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T13:08:09.621 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:08:09.621 INFO:teuthology.orchestra.run.smithi148.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:08:09.621 INFO:teuthology.orchestra.run.smithi148.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:08:09.621 INFO:teuthology.orchestra.run.smithi148.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:08:09.621 INFO:teuthology.orchestra.run.smithi148.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T13:08:09.622 INFO:teuthology.orchestra.run.smithi148.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T13:08:09.623 INFO:teuthology.orchestra.run.smithi148.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:08:09.623 INFO:teuthology.orchestra.run.smithi148.stdout:The following additional packages will be installed: 2023-10-30T13:08:09.623 INFO:teuthology.orchestra.run.smithi148.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T13:08:09.623 INFO:teuthology.orchestra.run.smithi148.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T13:08:09.623 INFO:teuthology.orchestra.run.smithi148.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T13:08:09.624 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T13:08:09.624 INFO:teuthology.orchestra.run.smithi148.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:08:09.624 INFO:teuthology.orchestra.run.smithi148.stdout:Suggested packages: 2023-10-30T13:08:09.624 INFO:teuthology.orchestra.run.smithi148.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T13:08:09.625 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T13:08:09.625 INFO:teuthology.orchestra.run.smithi148.stdout: libengine-pkcs11-openssl 2023-10-30T13:08:09.671 INFO:teuthology.orchestra.run.smithi148.stdout:The following NEW packages will be installed: 2023-10-30T13:08:09.672 INFO:teuthology.orchestra.run.smithi148.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T13:08:09.672 INFO:teuthology.orchestra.run.smithi148.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T13:08:09.673 INFO:teuthology.orchestra.run.smithi148.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T13:08:09.673 INFO:teuthology.orchestra.run.smithi148.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T13:08:09.673 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T13:08:09.673 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T13:08:09.673 INFO:teuthology.orchestra.run.smithi148.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:08:09.674 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages will be upgraded: 2023-10-30T13:08:09.674 INFO:teuthology.orchestra.run.smithi148.stdout: libssl3 2023-10-30T13:08:09.677 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T13:08:09.755 INFO:teuthology.orchestra.run.smithi148.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T13:08:09.755 INFO:teuthology.orchestra.run.smithi148.stdout:Need to get 54.6 MB of archives. 2023-10-30T13:08:09.755 INFO:teuthology.orchestra.run.smithi148.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T13:08:09.755 INFO:teuthology.orchestra.run.smithi148.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T13:08:09.937 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T13:08:09.963 INFO:teuthology.orchestra.run.smithi148.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T13:08:10.077 INFO:teuthology.orchestra.run.smithi148.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T13:08:10.095 INFO:teuthology.orchestra.run.smithi148.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T13:08:10.095 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T13:08:10.099 INFO:teuthology.orchestra.run.smithi148.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T13:08:10.100 INFO:teuthology.orchestra.run.smithi148.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T13:08:10.101 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T13:08:10.104 INFO:teuthology.orchestra.run.smithi148.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T13:08:10.105 INFO:teuthology.orchestra.run.smithi148.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T13:08:10.105 INFO:teuthology.orchestra.run.smithi148.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T13:08:10.116 INFO:teuthology.orchestra.run.smithi148.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T13:08:10.142 INFO:teuthology.orchestra.run.smithi148.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T13:08:10.205 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T13:08:10.439 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T13:08:10.591 INFO:teuthology.orchestra.run.smithi148.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T13:08:10.593 INFO:teuthology.orchestra.run.smithi148.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T13:08:10.593 INFO:teuthology.orchestra.run.smithi148.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T13:08:10.596 INFO:teuthology.orchestra.run.smithi148.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T13:08:10.596 INFO:teuthology.orchestra.run.smithi148.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T13:08:10.601 INFO:teuthology.orchestra.run.smithi148.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T13:08:10.603 INFO:teuthology.orchestra.run.smithi148.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T13:08:10.610 INFO:teuthology.orchestra.run.smithi148.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T13:08:10.610 INFO:teuthology.orchestra.run.smithi148.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T13:08:10.634 INFO:teuthology.orchestra.run.smithi148.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T13:08:10.639 INFO:teuthology.orchestra.run.smithi148.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T13:08:10.649 INFO:teuthology.orchestra.run.smithi148.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T13:08:10.665 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T13:08:10.676 INFO:teuthology.orchestra.run.smithi148.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T13:08:10.680 INFO:teuthology.orchestra.run.smithi148.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T13:08:10.680 INFO:teuthology.orchestra.run.smithi148.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T13:08:10.681 INFO:teuthology.orchestra.run.smithi148.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T13:08:10.683 INFO:teuthology.orchestra.run.smithi148.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T13:08:10.694 INFO:teuthology.orchestra.run.smithi148.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T13:08:10.703 INFO:teuthology.orchestra.run.smithi148.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T13:08:10.899 INFO:teuthology.orchestra.run.smithi148.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T13:08:10.899 INFO:teuthology.orchestra.run.smithi148.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T13:08:10.900 INFO:teuthology.orchestra.run.smithi148.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T13:08:10.900 INFO:teuthology.orchestra.run.smithi148.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T13:08:10.901 INFO:teuthology.orchestra.run.smithi148.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T13:08:10.901 INFO:teuthology.orchestra.run.smithi148.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T13:08:10.914 INFO:teuthology.orchestra.run.smithi148.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T13:08:10.914 INFO:teuthology.orchestra.run.smithi148.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T13:08:10.924 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T13:08:11.051 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T13:08:11.292 INFO:teuthology.orchestra.run.smithi148.stdout:Preconfiguring packages ... 2023-10-30T13:08:11.335 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T13:08:11.434 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 54.6 MB in 1s (44.3 MB/s) 2023-10-30T13:08:11.486 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T13:08:11.535 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package flex. 2023-10-30T13:08:11.608 INFO:teuthology.orchestra.run.smithi148.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139851 files and directories currently installed.) 2023-10-30T13:08:11.612 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T13:08:11.659 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T13:08:11.787 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T13:08:12.130 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T13:08:12.220 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:08:12.398 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T13:08:12.398 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T13:08:12.795 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:08:12.800 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T13:08:13.034 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T13:08:13.160 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T13:08:13.340 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package bison. 2023-10-30T13:08:13.410 INFO:teuthology.orchestra.run.smithi148.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139924 files and directories currently installed.) 2023-10-30T13:08:13.414 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T13:08:13.459 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:08:13.537 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T13:08:13.788 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T13:08:13.902 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T13:08:13.916 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T13:08:13.982 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:08:14.030 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T13:08:14.140 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T13:08:14.304 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package dump. 2023-10-30T13:08:14.319 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T13:08:14.354 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T13:08:14.475 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T13:08:14.608 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package indent. 2023-10-30T13:08:14.623 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:14.666 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:08:14.910 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T13:08:14.926 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T13:08:14.968 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:08:15.043 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T13:08:15.305 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T13:08:15.319 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T13:08:15.321 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T13:08:15.363 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:08:15.445 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T13:08:15.616 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T13:08:15.631 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T13:08:15.666 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:08:15.738 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T13:08:15.969 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T13:08:15.985 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T13:08:15.988 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T13:08:16.027 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:08:16.265 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T13:08:16.313 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T13:08:16.329 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T13:08:16.371 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:08:16.624 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T13:08:16.639 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T13:08:16.641 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T13:08:16.682 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T13:08:16.935 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T13:08:16.960 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T13:08:16.976 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T13:08:17.010 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:08:17.262 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T13:08:17.388 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:17.497 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:17.725 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:17.860 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:17.986 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:18.137 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T13:08:18.321 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T13:08:18.337 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:08:18.379 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:08:18.413 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:18.514 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:18.657 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T13:08:18.673 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:08:18.674 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:18.715 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:08:18.910 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:08:18.910 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T13:08:19.043 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T13:08:19.058 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:08:19.101 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:08:19.431 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T13:08:19.446 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T13:08:19.459 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:19.489 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:08:19.586 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:19.762 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:19.825 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T13:08:19.841 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:08:19.883 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:08:20.188 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T13:08:20.236 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T13:08:20.251 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:20.294 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:08:20.630 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T13:08:20.646 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:20.688 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:08:21.016 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T13:08:21.031 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:21.074 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:08:21.326 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T13:08:21.342 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:08:21.384 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:08:21.477 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:21.603 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:21.766 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:21.872 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T13:08:21.887 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T13:08:21.896 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:21.955 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:08:21.997 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:22.099 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:22.291 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package modemmanager. 2023-10-30T13:08:22.306 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:08:22.349 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:08:22.667 INFO:teuthology.orchestra.run.smithi150.stdout:Adding group ceph....done 2023-10-30T13:08:22.947 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package network-manager. 2023-10-30T13:08:22.963 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:08:23.013 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:08:23.077 INFO:teuthology.orchestra.run.smithi150.stdout:Adding system user ceph....done 2023-10-30T13:08:23.253 INFO:teuthology.orchestra.run.smithi150.stdout:Setting system user ceph properties....done 2023-10-30T13:08:23.271 INFO:teuthology.orchestra.run.smithi150.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T13:08:23.419 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T13:08:23.628 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ppp. 2023-10-30T13:08:23.644 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T13:08:23.735 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T13:08:23.964 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:08:24.308 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T13:08:24.321 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:24.322 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T13:08:24.366 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T13:08:24.538 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:08:24.539 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T13:08:24.610 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T13:08:24.625 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T13:08:24.660 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T13:08:24.963 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T13:08:24.978 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T13:08:25.038 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T13:08:25.043 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:25.279 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:25.350 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T13:08:25.366 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:25.409 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:08:25.812 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:08:25.812 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T13:08:25.964 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T13:08:25.979 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T13:08:26.040 INFO:teuthology.orchestra.run.smithi148.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T13:08:26.070 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking postgresql-common (238) ... 2023-10-30T13:08:26.322 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:26.487 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T13:08:26.503 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:26.548 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:08:26.680 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T13:08:27.157 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:27.384 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:08:27.385 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T13:08:27.690 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package postgresql. 2023-10-30T13:08:27.704 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T13:08:27.748 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking postgresql (14+238) ... 2023-10-30T13:08:27.926 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:27.975 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T13:08:27.990 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T13:08:28.034 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T13:08:28.146 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:08:28.147 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T13:08:28.277 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T13:08:28.292 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T13:08:28.335 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T13:08:28.655 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T13:08:28.669 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:28.671 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T13:08:28.730 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:08:28.990 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package tcl. 2023-10-30T13:08:29.004 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:08:29.004 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T13:08:29.006 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T13:08:29.053 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T13:08:29.410 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T13:08:29.426 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T13:08:29.468 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:08:29.546 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:29.646 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:29.781 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:29.881 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T13:08:29.897 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T13:08:29.931 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T13:08:29.977 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:08:29.977 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T13:08:30.259 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T13:08:30.275 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T13:08:30.326 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:08:30.609 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T13:08:30.814 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T13:08:30.886 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:30.966 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:08:31.083 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:08:31.180 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:31.193 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:08:31.306 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:31.319 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:08:31.444 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:08:31.457 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T13:08:31.827 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:08:32.454 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T13:08:32.559 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:08:32.902 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:08:32.903 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:08:32.911 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:08:33.425 INFO:teuthology.orchestra.run.smithi150.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:08:33.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T13:08:33.488 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2023-10-30T13:08:33.562 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:08:33.657 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2023-10-30T13:08:33.658 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2023-10-30T13:08:33.688 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:08:33.803 INFO:teuthology.orchestra.run.smithi150.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T13:08:33.804 INFO:teuthology.orchestra.run.smithi150.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T13:08:33.804 INFO:teuthology.orchestra.run.smithi150.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:08:33.804 INFO:teuthology.orchestra.run.smithi150.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T13:08:33.804 INFO:teuthology.orchestra.run.smithi150.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T13:08:33.805 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T13:08:33.805 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T13:08:33.805 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T13:08:33.806 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T13:08:33.806 INFO:teuthology.orchestra.run.smithi150.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T13:08:33.807 INFO:teuthology.orchestra.run.smithi150.stdout: python2.7-minimal 2023-10-30T13:08:33.807 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T13:08:33.807 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2023-10-30T13:08:33.807 INFO:teuthology.orchestra.run.smithi150.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T13:08:33.807 INFO:teuthology.orchestra.run.smithi150.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T13:08:33.808 INFO:teuthology.orchestra.run.smithi150.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T13:08:33.808 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T13:08:33.808 INFO:teuthology.orchestra.run.smithi150.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:08:33.808 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2023-10-30T13:08:33.809 INFO:teuthology.orchestra.run.smithi150.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T13:08:33.809 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T13:08:33.809 INFO:teuthology.orchestra.run.smithi150.stdout: libengine-pkcs11-openssl 2023-10-30T13:08:33.814 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T13:08:33.842 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2023-10-30T13:08:33.842 INFO:teuthology.orchestra.run.smithi150.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T13:08:33.842 INFO:teuthology.orchestra.run.smithi150.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T13:08:33.843 INFO:teuthology.orchestra.run.smithi150.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T13:08:33.843 INFO:teuthology.orchestra.run.smithi150.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T13:08:33.844 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T13:08:33.844 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T13:08:33.844 INFO:teuthology.orchestra.run.smithi150.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T13:08:33.844 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages will be upgraded: 2023-10-30T13:08:33.845 INFO:teuthology.orchestra.run.smithi150.stdout: libssl3 2023-10-30T13:08:33.939 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:08:34.065 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:08:34.192 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:08:34.342 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:08:35.481 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:08:35.607 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T13:08:35.702 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T13:08:35.759 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:08:35.876 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:08:35.994 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T13:08:36.095 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T13:08:36.229 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:08:36.319 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T13:08:36.381 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:08:36.498 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:08:36.667 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:08:36.793 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:08:36.804 INFO:teuthology.orchestra.run.smithi150.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T13:08:36.805 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 54.6 MB of archives. 2023-10-30T13:08:36.805 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T13:08:36.805 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T13:08:36.919 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T13:08:36.920 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T13:08:36.992 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T13:08:37.007 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T13:08:37.008 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T13:08:37.010 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T13:08:37.012 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T13:08:37.012 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T13:08:37.014 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T13:08:37.015 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T13:08:37.016 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T13:08:37.016 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T13:08:37.036 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T13:08:37.087 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T13:08:37.213 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:08:37.420 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T13:08:37.422 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T13:08:37.422 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T13:08:37.425 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T13:08:37.470 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T13:08:37.475 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T13:08:37.477 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T13:08:37.485 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T13:08:37.485 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T13:08:37.510 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T13:08:37.511 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T13:08:37.522 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T13:08:37.543 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T13:08:37.551 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T13:08:37.551 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T13:08:37.552 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T13:08:37.552 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T13:08:37.569 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T13:08:37.572 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T13:08:37.821 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:08:37.821 INFO:teuthology.orchestra.run.smithi148.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T13:08:37.821 INFO:teuthology.orchestra.run.smithi148.stdout:which means they are currently configured by ifupdown: 2023-10-30T13:08:37.822 INFO:teuthology.orchestra.run.smithi148.stdout:- enp3s0f1 2023-10-30T13:08:37.822 INFO:teuthology.orchestra.run.smithi148.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T13:08:37.822 INFO:teuthology.orchestra.run.smithi148.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T13:08:37.822 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:08:37.897 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T13:08:37.898 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T13:08:37.898 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T13:08:37.898 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T13:08:37.899 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T13:08:37.899 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T13:08:37.912 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T13:08:37.912 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T13:08:37.954 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T13:08:38.314 INFO:teuthology.orchestra.run.smithi150.stdout:Preconfiguring packages ... 2023-10-30T13:08:38.348 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T13:08:38.481 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 54.6 MB in 4s (13.4 MB/s) 2023-10-30T13:08:38.618 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package flex. 2023-10-30T13:08:38.694 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137425 files and directories currently installed.) 2023-10-30T13:08:38.699 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T13:08:38.741 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T13:08:38.783 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T13:08:39.344 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:08:39.458 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:08:39.522 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T13:08:39.936 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:08:40.004 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:08:40.086 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T13:08:40.220 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up postgresql-common (238) ... 2023-10-30T13:08:40.565 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package bison. 2023-10-30T13:08:40.651 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137498 files and directories currently installed.) 2023-10-30T13:08:40.656 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T13:08:40.718 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:08:41.151 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T13:08:41.173 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T13:08:41.210 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:08:41.331 INFO:teuthology.orchestra.run.smithi148.stdout:Adding user postgres to group ssl-cert 2023-10-30T13:08:41.579 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package dump. 2023-10-30T13:08:41.601 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T13:08:41.637 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T13:08:41.747 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:08:41.748 INFO:teuthology.orchestra.run.smithi148.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T13:08:41.966 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package indent. 2023-10-30T13:08:41.978 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:42.016 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:08:42.130 INFO:teuthology.orchestra.run.smithi148.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T13:08:42.130 INFO:teuthology.orchestra.run.smithi148.stdout:Removing obsolete dictionary files: 2023-10-30T13:08:42.243 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T13:08:42.265 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T13:08:42.335 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:08:42.647 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T13:08:42.668 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T13:08:42.713 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:08:42.827 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T13:08:43.016 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T13:08:43.028 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T13:08:43.074 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:08:43.282 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:08:43.438 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T13:08:43.460 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T13:08:43.504 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:08:43.645 INFO:teuthology.orchestra.run.smithi148.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T13:08:43.666 INFO:teuthology.orchestra.run.smithi148.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T13:08:43.716 INFO:teuthology.orchestra.run.smithi148.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T13:08:43.716 INFO:teuthology.orchestra.run.smithi148.stdout:This user must also own the server process. 2023-10-30T13:08:43.716 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:08:43.717 INFO:teuthology.orchestra.run.smithi148.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T13:08:43.717 INFO:teuthology.orchestra.run.smithi148.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T13:08:43.717 INFO:teuthology.orchestra.run.smithi148.stdout:The default text search configuration will be set to "english". 2023-10-30T13:08:43.717 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:08:43.717 INFO:teuthology.orchestra.run.smithi148.stdout:Data page checksums are disabled. 2023-10-30T13:08:43.717 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:08:43.718 INFO:teuthology.orchestra.run.smithi148.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T13:08:43.718 INFO:teuthology.orchestra.run.smithi148.stdout:creating subdirectories ... ok 2023-10-30T13:08:43.718 INFO:teuthology.orchestra.run.smithi148.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T13:08:43.732 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T13:08:43.741 INFO:teuthology.orchestra.run.smithi148.stdout:selecting default max_connections ... 100 2023-10-30T13:08:43.753 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T13:08:43.776 INFO:teuthology.orchestra.run.smithi148.stdout:selecting default shared_buffers ... 128MB 2023-10-30T13:08:43.790 INFO:teuthology.orchestra.run.smithi148.stdout:selecting default time zone ... Etc/UTC 2023-10-30T13:08:43.790 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:08:43.791 INFO:teuthology.orchestra.run.smithi148.stdout:creating configuration files ... ok 2023-10-30T13:08:43.960 INFO:teuthology.orchestra.run.smithi148.stdout:running bootstrap script ... ok 2023-10-30T13:08:44.076 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T13:08:44.098 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T13:08:44.142 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T13:08:44.454 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T13:08:44.476 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T13:08:44.512 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:08:44.589 INFO:teuthology.orchestra.run.smithi148.stdout:performing post-bootstrap initialization ... ok 2023-10-30T13:08:45.116 INFO:teuthology.orchestra.run.smithi148.stdout:syncing data to disk ... ok 2023-10-30T13:08:45.640 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T13:08:46.249 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T13:08:46.262 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:08:46.290 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:08:46.576 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T13:08:46.598 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T13:08:46.643 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:08:47.004 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T13:08:47.026 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:08:47.071 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:08:47.415 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T13:08:47.437 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T13:08:47.515 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:08:47.843 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T13:08:47.865 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:08:47.909 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:08:48.296 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T13:08:48.318 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:48.362 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:08:48.723 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:08:48.723 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T13:08:48.745 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:48.790 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:08:48.849 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T13:08:48.988 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up postgresql (14+238) ... 2023-10-30T13:08:49.101 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:08:49.134 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T13:08:49.156 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:49.201 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:08:49.227 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:08:49.462 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T13:08:49.483 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T13:08:49.487 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:08:49.487 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:08:49.553 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:08:50.115 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T13:08:50.137 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T13:08:50.182 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:08:50.271 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T13:08:50.535 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package modemmanager. 2023-10-30T13:08:50.556 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T13:08:50.590 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:08:50.609 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:08:50.741 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:08:51.300 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package network-manager. 2023-10-30T13:08:51.321 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T13:08:51.374 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:08:51.809 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T13:08:51.909 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T13:08:52.023 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ppp. 2023-10-30T13:08:52.045 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T13:08:52.372 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:08:52.627 INFO:teuthology.orchestra.run.smithi148.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:08:52.632 DEBUG:teuthology.parallel:result is None 2023-10-30T13:08:52.720 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T13:08:52.732 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T13:08:52.778 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T13:08:53.022 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T13:08:53.031 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T13:08:53.072 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T13:08:53.565 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T13:08:53.587 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T13:08:53.677 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T13:08:54.275 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T13:08:54.297 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:54.366 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:08:55.285 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T13:08:55.307 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T13:08:55.388 INFO:teuthology.orchestra.run.smithi150.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T13:08:55.416 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-common (238) ... 2023-10-30T13:08:56.034 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T13:08:56.056 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T13:08:56.104 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:08:57.621 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql. 2023-10-30T13:08:57.634 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T13:08:57.671 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql (14+238) ... 2023-10-30T13:08:57.948 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T13:08:57.961 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T13:08:57.998 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T13:08:58.242 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T13:08:58.254 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T13:08:58.316 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T13:08:58.653 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T13:08:58.674 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T13:08:58.719 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:08:59.022 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package tcl. 2023-10-30T13:08:59.045 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T13:08:59.084 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T13:08:59.458 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T13:08:59.481 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T13:08:59.525 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:08:59.954 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T13:08:59.967 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T13:09:00.021 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T13:09:00.424 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T13:09:00.446 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T13:09:00.499 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:09:00.807 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T13:09:01.031 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T13:09:01.140 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T13:09:01.257 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:09:01.417 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T13:09:01.542 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T13:09:01.669 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:09:02.378 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T13:09:02.516 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T13:09:02.896 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:09:02.896 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T13:09:03.486 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T13:09:03.603 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T13:09:03.738 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T13:09:03.864 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T13:09:04.014 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T13:09:04.140 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T13:09:04.267 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T13:09:05.081 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T13:09:05.206 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T13:09:05.300 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T13:09:05.366 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:09:05.526 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T13:09:05.652 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T13:09:05.769 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T13:09:05.870 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T13:09:05.969 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T13:09:06.097 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T13:09:06.223 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T13:09:06.400 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T13:09:06.550 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:09:06.676 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T13:09:06.844 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T13:09:06.945 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T13:09:07.474 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:07.475 INFO:teuthology.orchestra.run.smithi150.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T13:09:07.475 INFO:teuthology.orchestra.run.smithi150.stdout:which means they are currently configured by ifupdown: 2023-10-30T13:09:07.475 INFO:teuthology.orchestra.run.smithi150.stdout:- enp3s0f1 2023-10-30T13:09:07.475 INFO:teuthology.orchestra.run.smithi150.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T13:09:07.476 INFO:teuthology.orchestra.run.smithi150.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T13:09:07.476 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:07.600 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T13:09:07.927 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T13:09:08.285 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T13:09:08.856 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T13:09:09.008 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T13:09:09.159 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T13:09:09.285 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-common (238) ... 2023-10-30T13:09:10.710 INFO:teuthology.orchestra.run.smithi150.stdout:Adding user postgres to group ssl-cert 2023-10-30T13:09:10.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:10.901 INFO:teuthology.orchestra.run.smithi150.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T13:09:11.086 INFO:teuthology.orchestra.run.smithi150.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T13:09:11.086 INFO:teuthology.orchestra.run.smithi150.stdout:Removing obsolete dictionary files: 2023-10-30T13:09:11.677 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T13:09:12.112 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T13:09:12.446 INFO:teuthology.orchestra.run.smithi150.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T13:09:12.464 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T13:09:12.504 INFO:teuthology.orchestra.run.smithi150.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T13:09:12.504 INFO:teuthology.orchestra.run.smithi150.stdout:This user must also own the server process. 2023-10-30T13:09:12.504 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:12.504 INFO:teuthology.orchestra.run.smithi150.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T13:09:12.505 INFO:teuthology.orchestra.run.smithi150.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T13:09:12.505 INFO:teuthology.orchestra.run.smithi150.stdout:The default text search configuration will be set to "english". 2023-10-30T13:09:12.505 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:12.505 INFO:teuthology.orchestra.run.smithi150.stdout:Data page checksums are disabled. 2023-10-30T13:09:12.505 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:12.505 INFO:teuthology.orchestra.run.smithi150.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T13:09:12.506 INFO:teuthology.orchestra.run.smithi150.stdout:creating subdirectories ... ok 2023-10-30T13:09:12.506 INFO:teuthology.orchestra.run.smithi150.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T13:09:12.517 INFO:teuthology.orchestra.run.smithi150.stdout:selecting default max_connections ... 100 2023-10-30T13:09:12.538 INFO:teuthology.orchestra.run.smithi150.stdout:selecting default shared_buffers ... 128MB 2023-10-30T13:09:12.548 INFO:teuthology.orchestra.run.smithi150.stdout:selecting default time zone ... Etc/UTC 2023-10-30T13:09:12.549 INFO:teuthology.orchestra.run.smithi150.stdout:creating configuration files ... ok 2023-10-30T13:09:12.658 INFO:teuthology.orchestra.run.smithi150.stdout:running bootstrap script ... ok 2023-10-30T13:09:13.293 INFO:teuthology.orchestra.run.smithi150.stdout:performing post-bootstrap initialization ... ok 2023-10-30T13:09:14.174 INFO:teuthology.orchestra.run.smithi150.stdout:syncing data to disk ... ok 2023-10-30T13:09:14.567 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T13:09:17.394 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:09:17.520 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T13:09:17.646 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql (14+238) ... 2023-10-30T13:09:17.747 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T13:09:17.864 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T13:09:18.143 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:09:18.143 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T13:09:18.717 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T13:09:19.061 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T13:09:19.229 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T13:09:20.466 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T13:09:20.565 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T13:09:21.219 INFO:teuthology.orchestra.run.smithi150.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T13:09:21.222 DEBUG:teuthology.parallel:result is None 2023-10-30T13:09:21.223 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:09:21.367 DEBUG:teuthology.orchestra.run.smithi148:> dpkg-query -W -f '${Version}' ceph 2023-10-30T13:09:21.391 INFO:teuthology.orchestra.run.smithi148.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:09:21.392 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:09:21.393 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T13:09:21.393 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:09:21.539 DEBUG:teuthology.orchestra.run.smithi150:> dpkg-query -W -f '${Version}' ceph 2023-10-30T13:09:21.558 INFO:teuthology.orchestra.run.smithi150.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:09:21.558 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T13:09:21.559 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T13:09:21.559 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T13:09:21.560 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:21.560 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T13:09:21.583 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:21.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T13:09:21.613 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T13:09:21.614 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:21.614 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T13:09:21.646 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T13:09:21.709 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:21.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T13:09:21.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T13:09:21.776 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T13:09:21.777 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:21.777 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T13:09:21.797 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T13:09:21.857 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:21.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T13:09:21.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T13:09:21.924 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T13:09:21.924 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:21.925 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T13:09:21.945 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T13:09:22.005 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:22.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T13:09:22.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T13:09:22.072 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T13:09:22.132 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T13:09:22.133 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /var/log/ceph 2023-10-30T13:09:22.137 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2023-10-30T13:09:22.153 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T13:09:22.153 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T13:09:22.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T13:09:22.213 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T13:09:22.213 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T13:09:22.256 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T13:09:22.280 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-30T13:09:22.280 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-30T13:09:22.280 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-30T13:09:22.281 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:22.281 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T13:09:22.326 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T13:09:22.421 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:22.421 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T13:09:22.427 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T13:09:22.506 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T13:09:22.507 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T13:09:22.507 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041', 'branch': 'wip-lusov-balance-after-stop', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7441041', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-lusov-balance-after-stop'}, 'ceph': {'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-balance-after-stop', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 8397, 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'sleep_before_teardown': 0, 'subset': '464/1024', 'suite': 'fs', 'suite_branch': 'wip-lusov-balance-after-stop', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi148.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKeRIfRxzODgVbK6MlfsIgDCiYV1EZSosY4UyltWr7mHCPO7pEuEDj8D6YWgeqTEN/fv5kOWCQIbkChn6/v4dvg=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPEFw8CnH+x/8AQAfMpXjhYNhZQb3piNcsHdDTMDzCoqz04lQJ3zW4OeCWF+KuNDgMaIbs2PAfuzKUqbzbZ66m0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'cluster': 'ceph'}}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_06:34:16', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T13:09:22.508 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T13:09:22.512 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T13:09:22.550 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T13:09:22.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T13:09:22.610 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:22.610 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T13:09:22.630 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T13:09:22.630 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2023-10-30T13:09:22.686 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T13:09:22.686 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:09:22.686 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 902 Links: 1 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-10-30 13:02:46.879203222 +0000 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-10-30 13:02:46.495206652 +0000 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-10-30 13:02:46.495206652 +0000 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2023-10-30 13:02:46.495206652 +0000 2023-10-30T13:09:22.688 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T13:09:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-10-30T13:09:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-10-30T13:09:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000370276 s, 1.4 MB/s 2023-10-30T13:09:22.749 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T13:09:22.799 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2023-10-30T13:09:22.854 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T13:09:22.854 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:09:22.854 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 921 Links: 1 2023-10-30T13:09:22.854 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:09:22.855 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-10-30 13:02:47.519197505 +0000 2023-10-30T13:09:22.855 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-10-30 13:02:47.147200828 +0000 2023-10-30T13:09:22.855 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-10-30 13:02:47.147200828 +0000 2023-10-30T13:09:22.855 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2023-10-30 13:02:47.147200828 +0000 2023-10-30T13:09:22.855 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T13:09:22.915 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-10-30T13:09:22.915 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-10-30T13:09:22.916 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.00036547 s, 1.4 MB/s 2023-10-30T13:09:22.917 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T13:09:22.972 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2023-10-30T13:09:23.026 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T13:09:23.026 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:09:23.026 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 936 Links: 1 2023-10-30T13:09:23.026 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:09:23.027 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-10-30 13:02:48.163191753 +0000 2023-10-30T13:09:23.027 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-10-30 13:02:47.791195075 +0000 2023-10-30T13:09:23.027 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-10-30 13:02:47.791195075 +0000 2023-10-30T13:09:23.027 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2023-10-30 13:02:47.791195075 +0000 2023-10-30T13:09:23.028 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T13:09:23.091 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-10-30T13:09:23.091 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-10-30T13:09:23.091 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000364148 s, 1.4 MB/s 2023-10-30T13:09:23.093 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T13:09:23.148 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2023-10-30T13:09:23.202 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T13:09:23.202 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:09:23.202 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 954 Links: 1 2023-10-30T13:09:23.202 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:09:23.202 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2023-10-30 13:02:48.823185858 +0000 2023-10-30T13:09:23.203 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-10-30 13:02:48.443189251 +0000 2023-10-30T13:09:23.203 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-10-30 13:02:48.443189251 +0000 2023-10-30T13:09:23.203 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2023-10-30 13:02:48.443189251 +0000 2023-10-30T13:09:23.203 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T13:09:23.268 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2023-10-30T13:09:23.268 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2023-10-30T13:09:23.268 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000349574 s, 1.5 MB/s 2023-10-30T13:09:23.269 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T13:09:23.319 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:23.320 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:23.320 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T13:09:23.326 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T13:09:23.326 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2023-10-30T13:09:23.374 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T13:09:23.374 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:09:23.374 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 888 Links: 1 2023-10-30T13:09:23.374 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:09:23.375 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-30 13:02:13.798531022 +0000 2023-10-30T13:09:23.375 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-30 13:02:13.478533881 +0000 2023-10-30T13:09:23.375 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-30 13:02:13.478533881 +0000 2023-10-30T13:09:23.375 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-30T13:09:23.375 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T13:09:23.428 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-30T13:09:23.428 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-30T13:09:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000533695 s, 959 kB/s 2023-10-30T13:09:23.429 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T13:09:23.479 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2023-10-30T13:09:23.526 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T13:09:23.526 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:09:23.526 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 907 Links: 1 2023-10-30T13:09:23.526 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:09:23.527 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-30 13:02:14.398525662 +0000 2023-10-30T13:09:23.527 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-30 13:02:14.050528771 +0000 2023-10-30T13:09:23.527 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-30 13:02:14.050528771 +0000 2023-10-30T13:09:23.527 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-30T13:09:23.527 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T13:09:23.580 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-30T13:09:23.580 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-30T13:09:23.580 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000496131 s, 1.0 MB/s 2023-10-30T13:09:23.581 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T13:09:23.631 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2023-10-30T13:09:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T13:09:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:09:23.678 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 921 Links: 1 2023-10-30T13:09:23.679 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:09:23.679 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-30 13:02:14.966520588 +0000 2023-10-30T13:09:23.679 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-30 13:02:14.650523412 +0000 2023-10-30T13:09:23.679 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-30 13:02:14.650523412 +0000 2023-10-30T13:09:23.679 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-30T13:09:23.680 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T13:09:23.733 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-30T13:09:23.733 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-30T13:09:23.734 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000585589 s, 874 kB/s 2023-10-30T13:09:23.735 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T13:09:23.783 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2023-10-30T13:09:23.829 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T13:09:23.830 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T13:09:23.830 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 941 Links: 1 2023-10-30T13:09:23.830 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:09:23.830 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-30 13:02:15.562515266 +0000 2023-10-30T13:09:23.831 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-30 13:02:15.218518338 +0000 2023-10-30T13:09:23.831 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-30 13:02:15.218518338 +0000 2023-10-30T13:09:23.831 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-30T13:09:23.831 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T13:09:23.884 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-30T13:09:23.884 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-30T13:09:23.884 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000446513 s, 1.1 MB/s 2023-10-30T13:09:23.885 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T13:09:23.935 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:23.935 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T13:09:23.935 INFO:tasks.ceph:Generating config... 2023-10-30T13:09:23.936 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-30T13:09:23.937 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-10-30T13:09:23.937 INFO:tasks.ceph:[client] debug client = 20 2023-10-30T13:09:23.937 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T13:09:23.937 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-30T13:09:23.938 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-30T13:09:23.938 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-30T13:09:23.938 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-30T13:09:23.938 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-30T13:09:23.939 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-30T13:09:23.939 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-30T13:09:23.939 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-30T13:09:23.939 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-30T13:09:23.940 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-30T13:09:23.940 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-30T13:09:23.940 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-30T13:09:23.940 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-10-30T13:09:23.941 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-30T13:09:23.941 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-30T13:09:23.941 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-10-30T13:09:23.941 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-10-30T13:09:23.941 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-10-30T13:09:23.942 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-10-30T13:09:23.942 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-10-30T13:09:23.942 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-10-30T13:09:23.942 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-10-30T13:09:23.943 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-30T13:09:23.943 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-30T13:09:23.943 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-30T13:09:23.943 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T13:09:23.944 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T13:09:23.944 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T13:09:23.944 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T13:09:23.944 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T13:09:23.945 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-30T13:09:23.945 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-30T13:09:23.945 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-30T13:09:23.945 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-30T13:09:23.945 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T13:09:23.946 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T13:09:23.946 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-30T13:09:23.946 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-30T13:09:23.946 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T13:09:23.947 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T13:09:23.947 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-30T13:09:23.947 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-10-30T13:09:23.947 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T13:09:23.948 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T13:09:23.948 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T13:09:23.948 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-10-30T13:09:23.948 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-10-30T13:09:23.948 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-10-30T13:09:23.949 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-10-30T13:09:23.949 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-10-30T13:09:23.949 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T13:09:23.949 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-30T13:09:23.950 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T13:09:23.950 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-30T13:09:23.950 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T13:09:23.951 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T13:09:23.995 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T13:09:24.013 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T13:09:24.104 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T13:09:24.125 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.148'), ('mon.b', '172.21.15.150'), ('mon.c', '[v2:172.21.15.150:3301,v1:172.21.15.150:6790]')] 2023-10-30T13:09:24.126 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.148,172.21.15.150,[v2:172.21.15.150:3301,v1:172.21.15.150:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-30T13:09:24.127 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:24.127 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T13:09:24.178 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.148 --add b 172.21.15.150 --addv c '[v2:172.21.15.150:3301,v1:172.21.15.150:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:24.268 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:24.268 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: generated fsid 23a3d51e-f2ab-4ffa-93fe-42135bef502c 2023-10-30T13:09:24.268 INFO:teuthology.orchestra.run.smithi148.stdout:setting min_mon_release = pacific 2023-10-30T13:09:24.269 INFO:teuthology.orchestra.run.smithi148.stdout:epoch 0 2023-10-30T13:09:24.269 INFO:teuthology.orchestra.run.smithi148.stdout:fsid 23a3d51e-f2ab-4ffa-93fe-42135bef502c 2023-10-30T13:09:24.269 INFO:teuthology.orchestra.run.smithi148.stdout:last_changed 2023-10-30T13:09:24.268142+0000 2023-10-30T13:09:24.269 INFO:teuthology.orchestra.run.smithi148.stdout:created 2023-10-30T13:09:24.268142+0000 2023-10-30T13:09:24.269 INFO:teuthology.orchestra.run.smithi148.stdout:min_mon_release 16 (pacific) 2023-10-30T13:09:24.269 INFO:teuthology.orchestra.run.smithi148.stdout:election_strategy: 1 2023-10-30T13:09:24.270 INFO:teuthology.orchestra.run.smithi148.stdout:0: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.a 2023-10-30T13:09:24.270 INFO:teuthology.orchestra.run.smithi148.stdout:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2023-10-30T13:09:24.270 INFO:teuthology.orchestra.run.smithi148.stdout:2: [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] mon.c 2023-10-30T13:09:24.270 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T13:09:24.271 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T13:09:24.281 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 23a3d51e-f2ab-4ffa-93fe-42135bef502c... 2023-10-30T13:09:24.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T13:09:24.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T13:09:24.356 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2023-10-30T13:09:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2023-10-30T13:09:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T13:09:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2023-10-30T13:09:24.356 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2023-10-30T13:09:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2023-10-30T13:09:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.357 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2023-10-30T13:09:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2023-10-30T13:09:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2023-10-30T13:09:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2023-10-30T13:09:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.358 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T13:09:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2023-10-30T13:09:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T13:09:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T13:09:24.359 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2023-10-30T13:09:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2023-10-30T13:09:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2023-10-30T13:09:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2023-10-30T13:09:24.360 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2023-10-30T13:09:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2023-10-30T13:09:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2023-10-30T13:09:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2023-10-30T13:09:24.361 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T13:09:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T13:09:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2023-10-30T13:09:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T13:09:24.362 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2023-10-30T13:09:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2023-10-30T13:09:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.363 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2023-10-30T13:09:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2023-10-30T13:09:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2023-10-30T13:09:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.148,172.21.15.150,[v2:172.21.15.150:3301,v1:172.21.15.150:6790]" 2023-10-30T13:09:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on skipped message = False 2023-10-30T13:09:24.364 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 23a3d51e-f2ab-4ffa-93fe-42135bef502c 2023-10-30T13:09:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.365 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2023-10-30T13:09:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2023-10-30T13:09:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.365 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2023-10-30T13:09:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2023-10-30T13:09:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2023-10-30T13:09:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2023-10-30T13:09:24.366 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2023-10-30T13:09:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2023-10-30T13:09:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2023-10-30T13:09:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2023-10-30T13:09:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.367 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2023-10-30T13:09:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2023-10-30T13:09:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2023-10-30T13:09:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.368 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2023-10-30T13:09:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2023-10-30T13:09:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2023-10-30T13:09:24.369 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2023-10-30T13:09:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2023-10-30T13:09:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2023-10-30T13:09:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: bdev async discard = True 2023-10-30T13:09:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: bdev enable discard = True 2023-10-30T13:09:24.370 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore allocator = bitmap 2023-10-30T13:09:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2023-10-30T13:09:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2023-10-30T13:09:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 1/20 2023-10-30T13:09:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 1/20 2023-10-30T13:09:24.371 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-30T13:09:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2023-10-30T13:09:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 4/10 2023-10-30T13:09:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: mon client directed command retry = 5 2023-10-30T13:09:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T13:09:24.372 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2023-10-30T13:09:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2023-10-30T13:09:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject delay max = 1 2023-10-30T13:09:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject delay probability = 0.005 2023-10-30T13:09:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject delay type = client mds 2023-10-30T13:09:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject socket failures = 2500 2023-10-30T13:09:24.373 INFO:teuthology.orchestra.run.smithi150.stdout: osd blocked scrub grace period = 3600 2023-10-30T13:09:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2023-10-30T13:09:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: osd max markdown count = 1000 2023-10-30T13:09:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2023-10-30T13:09:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2023-10-30T13:09:24.374 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.375 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2023-10-30T13:09:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-30T13:09:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2023-10-30T13:09:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2023-10-30T13:09:24.375 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2023-10-30T13:09:24.376 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:09:24.376 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:09:24.376 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2023-10-30T13:09:24.376 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.376 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2023-10-30T13:09:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-30T13:09:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2023-10-30T13:09:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2023-10-30T13:09:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2023-10-30T13:09:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2023-10-30T13:09:24.377 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2023-10-30T13:09:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:09:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2023-10-30T13:09:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2023-10-30T13:09:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:09:24.378 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T13:09:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T13:09:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T13:09:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.379 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T13:09:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T13:09:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T13:09:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2023-10-30T13:09:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2023-10-30T13:09:24.380 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2023-10-30T13:09:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: mon op complaint time = 120 2023-10-30T13:09:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-30T13:09:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2023-10-30T13:09:24.381 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2023-10-30T13:09:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2023-10-30T13:09:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2023-10-30T13:09:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T13:09:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T13:09:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: client mount timeout = 600 2023-10-30T13:09:24.382 INFO:teuthology.orchestra.run.smithi150.stdout: client_shutdown_timeout = 120 2023-10-30T13:09:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2023-10-30T13:09:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-30T13:09:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2023-10-30T13:09:24.383 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2023-10-30T13:09:24.383 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2023-10-30T13:09:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2023-10-30T13:09:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2023-10-30T13:09:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2023-10-30T13:09:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2023-10-30T13:09:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.e] 2023-10-30T13:09:24.384 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2023-10-30T13:09:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2023-10-30T13:09:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.f] 2023-10-30T13:09:24.385 INFO:teuthology.orchestra.run.smithi150.stdout:[mds] 2023-10-30T13:09:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds = 20 2023-10-30T13:09:24.385 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds balancer = 20 2023-10-30T13:09:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2023-10-30T13:09:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: mds bal fragment size max = 10000 2023-10-30T13:09:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: mds bal merge size = 5 2023-10-30T13:09:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: mds bal split bits = 3 2023-10-30T13:09:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: mds bal split size = 100 2023-10-30T13:09:24.386 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug frag = True 2023-10-30T13:09:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug scatterstat = True 2023-10-30T13:09:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: mds oft prefetch dirfrags = True 2023-10-30T13:09:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: mds op complaint time = 180 2023-10-30T13:09:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: mds verify scatter = True 2023-10-30T13:09:24.387 INFO:teuthology.orchestra.run.smithi150.stdout: mds_dir_prefetch = False 2023-10-30T13:09:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T13:09:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: mon client directed command retry = 5 2023-10-30T13:09:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject delay max = 1 2023-10-30T13:09:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject delay probability = 0.005 2023-10-30T13:09:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject delay type = client mds osd 2023-10-30T13:09:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject socket failures = 2500 2023-10-30T13:09:24.388 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2023-10-30T13:09:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2023-10-30T13:09:24.389 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2023-10-30T13:09:24.390 INFO:teuthology.orchestra.run.smithi148.stdout:[global] 2023-10-30T13:09:24.390 INFO:teuthology.orchestra.run.smithi148.stdout: chdir = "" 2023-10-30T13:09:24.391 INFO:teuthology.orchestra.run.smithi148.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T13:09:24.391 INFO:teuthology.orchestra.run.smithi148.stdout: auth supported = cephx 2023-10-30T13:09:24.391 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.391 INFO:teuthology.orchestra.run.smithi148.stdout: filestore xattr use omap = true 2023-10-30T13:09:24.391 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: mon clock drift allowed = 1.000 2023-10-30T13:09:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: osd crush chooseleaf type = 0 2023-10-30T13:09:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: auth debug = true 2023-10-30T13:09:24.392 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on old message = true 2023-10-30T13:09:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on bug = true 2023-10-30T13:09:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T13:09:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: mon pg warn max object skew = 0 2023-10-30T13:09:24.393 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T13:09:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T13:09:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default size = 2 2023-10-30T13:09:24.394 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow primary affinity = true 2023-10-30T13:09:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow pg remap = true 2023-10-30T13:09:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on legacy crush tunables = false 2023-10-30T13:09:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on crush straw calc version zero = false 2023-10-30T13:09:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on no sortbitwise = false 2023-10-30T13:09:24.395 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on osd down out interval zero = false 2023-10-30T13:09:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on too few osds = false 2023-10-30T13:09:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T13:09:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T13:09:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: mon_allow_pool_size_one = true 2023-10-30T13:09:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.396 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T13:09:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: osd default data pool replay window = 5 2023-10-30T13:09:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: mon allow pool delete = true 2023-10-30T13:09:24.397 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: mon cluster log file level = debug 2023-10-30T13:09:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: debug asserts on shutdown = true 2023-10-30T13:09:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: mon health detail to clog = false 2023-10-30T13:09:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: mon host = "172.21.15.148,172.21.15.150,[v2:172.21.15.150:3301,v1:172.21.15.150:6790]" 2023-10-30T13:09:24.398 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on skipped message = False 2023-10-30T13:09:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: fsid = 23a3d51e-f2ab-4ffa-93fe-42135bef502c 2023-10-30T13:09:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.399 INFO:teuthology.orchestra.run.smithi148.stdout:[osd] 2023-10-30T13:09:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: osd journal size = 100 2023-10-30T13:09:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.399 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub load threshold = 5.0 2023-10-30T13:09:24.400 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub max interval = 600 2023-10-30T13:09:24.400 INFO:teuthology.orchestra.run.smithi148.stdout: osd mclock profile = high_recovery_ops 2023-10-30T13:09:24.400 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.400 INFO:teuthology.orchestra.run.smithi148.stdout: osd recover clone overlap = true 2023-10-30T13:09:24.400 INFO:teuthology.orchestra.run.smithi148.stdout: osd recovery max chunk = 1048576 2023-10-30T13:09:24.400 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.401 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug shutdown = true 2023-10-30T13:09:24.401 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug op order = true 2023-10-30T13:09:24.401 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify stray on activate = true 2023-10-30T13:09:24.401 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.401 INFO:teuthology.orchestra.run.smithi148.stdout: osd open classes on start = true 2023-10-30T13:09:24.401 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug pg log writeout = true 2023-10-30T13:09:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: osd deep scrub update digest min age = 30 2023-10-30T13:09:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: osd map max advance = 10 2023-10-30T13:09:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: journal zero on create = true 2023-10-30T13:09:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: filestore ondisk finisher threads = 3 2023-10-30T13:09:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: filestore apply finisher threads = 3 2023-10-30T13:09:24.403 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: bdev debug aio = true 2023-10-30T13:09:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug misdirected ops = true 2023-10-30T13:09:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: bdev async discard = True 2023-10-30T13:09:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: bdev enable discard = True 2023-10-30T13:09:24.404 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore allocator = bitmap 2023-10-30T13:09:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore block size = 96636764160 2023-10-30T13:09:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore fsck on mount = True 2023-10-30T13:09:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluefs = 1/20 2023-10-30T13:09:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluestore = 1/20 2023-10-30T13:09:24.405 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2023-10-30T13:09:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: debug osd = 20 2023-10-30T13:09:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: debug rocksdb = 4/10 2023-10-30T13:09:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: mon client directed command retry = 5 2023-10-30T13:09:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T13:09:24.406 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd full ratio = 0.9 2023-10-30T13:09:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd nearfull ratio = 0.8 2023-10-30T13:09:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject delay max = 1 2023-10-30T13:09:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject delay probability = 0.005 2023-10-30T13:09:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject delay type = client mds 2023-10-30T13:09:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject socket failures = 2500 2023-10-30T13:09:24.407 INFO:teuthology.orchestra.run.smithi148.stdout: osd blocked scrub grace period = 3600 2023-10-30T13:09:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: osd failsafe full ratio = 0.95 2023-10-30T13:09:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: osd max markdown count = 1000 2023-10-30T13:09:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: osd objectstore = bluestore 2023-10-30T13:09:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: osd op complaint time = 180 2023-10-30T13:09:24.408 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.408 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr] 2023-10-30T13:09:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2023-10-30T13:09:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: debug mgr = 20 2023-10-30T13:09:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2023-10-30T13:09:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2023-10-30T13:09:24.409 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:09:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:09:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: mgr/telemetry/nag = false 2023-10-30T13:09:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.410 INFO:teuthology.orchestra.run.smithi148.stdout:[mon] 2023-10-30T13:09:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2023-10-30T13:09:24.410 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2023-10-30T13:09:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: debug paxos = 20 2023-10-30T13:09:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2023-10-30T13:09:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon data avail warn = 5 2023-10-30T13:09:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon mgr mkfs grace = 240 2023-10-30T13:09:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2023-10-30T13:09:24.411 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd reporter subtree level = osd 2023-10-30T13:09:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd prime pg temp = true 2023-10-30T13:09:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2023-10-30T13:09:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T13:09:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T13:09:24.412 INFO:teuthology.orchestra.run.smithi148.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T13:09:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T13:09:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T13:09:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T13:09:24.413 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: # 1m isn't quite enough 2023-10-30T13:09:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: mon_down_mkfs_grace = 2m 2023-10-30T13:09:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_filestore_osds = false 2023-10-30T13:09:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: mon op complaint time = 120 2023-10-30T13:09:24.414 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:09:24.415 INFO:teuthology.orchestra.run.smithi148.stdout:[client] 2023-10-30T13:09:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: rgw cache enabled = true 2023-10-30T13:09:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable ops log = true 2023-10-30T13:09:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable usage log = true 2023-10-30T13:09:24.415 INFO:teuthology.orchestra.run.smithi148.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T13:09:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T13:09:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: client mount timeout = 600 2023-10-30T13:09:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: client_shutdown_timeout = 120 2023-10-30T13:09:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: debug client = 20 2023-10-30T13:09:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2023-10-30T13:09:24.416 INFO:teuthology.orchestra.run.smithi148.stdout: rados mon op timeout = 900 2023-10-30T13:09:24.417 INFO:teuthology.orchestra.run.smithi148.stdout: rados osd op timeout = 900 2023-10-30T13:09:24.417 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.a] 2023-10-30T13:09:24.417 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.b] 2023-10-30T13:09:24.417 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.c] 2023-10-30T13:09:24.417 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.a] 2023-10-30T13:09:24.418 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.c] 2023-10-30T13:09:24.418 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.e] 2023-10-30T13:09:24.418 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.b] 2023-10-30T13:09:24.418 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.d] 2023-10-30T13:09:24.418 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.f] 2023-10-30T13:09:24.419 INFO:teuthology.orchestra.run.smithi148.stdout:[mds] 2023-10-30T13:09:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: debug mds = 20 2023-10-30T13:09:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: debug mds balancer = 20 2023-10-30T13:09:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2023-10-30T13:09:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: mds bal fragment size max = 10000 2023-10-30T13:09:24.419 INFO:teuthology.orchestra.run.smithi148.stdout: mds bal merge size = 5 2023-10-30T13:09:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: mds bal split bits = 3 2023-10-30T13:09:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: mds bal split size = 100 2023-10-30T13:09:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: mds debug frag = True 2023-10-30T13:09:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: mds debug scatterstat = True 2023-10-30T13:09:24.420 INFO:teuthology.orchestra.run.smithi148.stdout: mds oft prefetch dirfrags = True 2023-10-30T13:09:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: mds op complaint time = 180 2023-10-30T13:09:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: mds verify scatter = True 2023-10-30T13:09:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: mds_dir_prefetch = False 2023-10-30T13:09:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T13:09:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: mon client directed command retry = 5 2023-10-30T13:09:24.421 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject delay max = 1 2023-10-30T13:09:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject delay probability = 0.005 2023-10-30T13:09:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject delay type = client mds osd 2023-10-30T13:09:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject socket failures = 2500 2023-10-30T13:09:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: osd op complaint time = 180 2023-10-30T13:09:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: rados mon op timeout = 900 2023-10-30T13:09:24.422 INFO:teuthology.orchestra.run.smithi148.stdout: rados osd op timeout = 900 2023-10-30T13:09:24.423 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T13:09:24.423 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T13:09:24.492 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T13:09:24.492 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:24.492 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T13:09:24.501 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:24.501 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T13:09:24.550 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:09:24.550 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:24.550 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T13:09:24.551 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T13:09:24.625 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:24.626 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:24.674 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:09:24.674 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:24.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T13:09:24.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T13:09:24.697 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:24.698 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:24.745 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T13:09:24.746 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T13:09:24.746 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T13:09:24.801 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T13:09:24.807 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T13:09:24.849 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T13:09:24.865 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T13:09:24.865 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T13:09:24.920 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T13:09:24.924 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-30T13:09:24.945 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T13:09:25.041 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T13:09:25.046 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-30T13:09:25.066 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T13:09:25.164 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T13:09:25.168 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-10-30T13:09:25.188 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T13:09:25.234 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T13:09:25.237 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-30T13:09:25.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T13:09:25.336 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T13:09:25.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-30T13:09:25.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T13:09:25.476 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T13:09:25.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-10-30T13:09:25.533 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T13:09:25.533 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T13:09:25.576 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T13:09:25.594 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T13:09:25.594 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T13:09:25.594 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T13:09:25.661 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:25.661 INFO:tasks.ceph:role: osd.0 2023-10-30T13:09:25.661 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:09:25.662 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T13:09:25.728 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:09:25.729 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:09:25.729 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:09:25.729 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:09:25.729 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:09:25.730 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:09:25.730 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:09:25.730 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:09:25.730 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:09:25.730 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:09:26.199 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2023-10-30T13:09:26.202 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2023-10-30T13:09:26.202 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T13:09:26.312 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T13:09:26.329 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:09:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:09:26.330 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T13:09:26.393 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:26.394 INFO:tasks.ceph:role: osd.1 2023-10-30T13:09:26.394 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:09:26.394 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T13:09:26.465 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:09:26.466 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:09:26.466 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:09:26.466 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:09:26.466 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:09:26.467 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:09:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:09:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:09:26.467 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:09:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:09:26.965 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2023-10-30T13:09:26.968 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2023-10-30T13:09:26.968 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T13:09:27.122 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T13:09:27.140 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:09:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:09:27.141 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T13:09:27.203 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:27.204 INFO:tasks.ceph:role: osd.2 2023-10-30T13:09:27.204 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:09:27.204 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T13:09:27.275 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:09:27.275 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:09:27.276 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:09:27.276 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:09:27.276 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:09:27.276 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:09:27.276 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:09:27.276 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:09:27.277 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:09:27.277 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:09:27.756 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2023-10-30T13:09:27.758 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2023-10-30T13:09:27.759 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T13:09:27.861 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T13:09:27.879 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:09:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:09:27.880 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T13:09:27.943 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:27.943 INFO:tasks.ceph:role: osd.3 2023-10-30T13:09:27.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:09:27.943 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T13:09:28.014 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:09:28.014 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:09:28.014 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:09:28.014 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:09:28.015 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:09:28.015 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:09:28.015 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:09:28.015 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:09:28.015 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:09:28.015 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:09:28.472 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2023-10-30T13:09:28.476 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2023-10-30T13:09:28.476 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T13:09:28.580 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T13:09:28.598 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:09:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:09:28.599 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:28.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:28.726+0000 7f54f782b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T13:09:28.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:28.730+0000 7f54f782b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T13:09:28.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:28.730+0000 7f54f782b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T13:09:28.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:28.730+0000 7f54f782b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T13:09:28.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:28.730+0000 7f54f782b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T13:09:28.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:28.734+0000 7f54f782b640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T13:09:31.738 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T13:09:31.759 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:31.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:31.850+0000 7f4c11c44640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T13:09:31.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:31.850+0000 7f4c11c44640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T13:09:31.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:31.850+0000 7f4c11c44640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T13:09:31.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:31.850+0000 7f4c11c44640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T13:09:31.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:31.850+0000 7f4c11c44640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T13:09:31.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:31.854+0000 7f4c11c44640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T13:09:34.838 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T13:09:34.861 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:34.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:34.954+0000 7fd9ffa2f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T13:09:34.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:34.958+0000 7fd9ffa2f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T13:09:34.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:34.958+0000 7fd9ffa2f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T13:09:34.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:34.958+0000 7fd9ffa2f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T13:09:34.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:34.958+0000 7fd9ffa2f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T13:09:34.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:34.958+0000 7fd9ffa2f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T13:09:37.949 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T13:09:37.970 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:38.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:38.066+0000 7f606b82c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T13:09:38.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:38.066+0000 7f606b82c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T13:09:38.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:38.066+0000 7f606b82c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T13:09:38.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:38.066+0000 7f606b82c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T13:09:38.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:38.066+0000 7f606b82c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T13:09:38.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:38.070+0000 7f606b82c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T13:09:41.064 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T13:09:41.085 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T13:09:41.099 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:41.099 INFO:tasks.ceph:role: osd.4 2023-10-30T13:09:41.099 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:09:41.100 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T13:09:41.161 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:09:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:09:41.161 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:09:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:09:41.162 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:09:41.162 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:09:41.162 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:09:41.162 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:09:41.163 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:09:41.163 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:09:41.634 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-30T13:09:41.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-30T13:09:41.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T13:09:41.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T13:09:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:09:41.787 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:09:41.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T13:09:41.845 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:41.845 INFO:tasks.ceph:role: osd.5 2023-10-30T13:09:41.845 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:09:41.846 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T13:09:41.910 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:09:41.910 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:09:41.911 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:09:41.911 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:09:41.911 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:09:41.911 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:09:41.911 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:09:41.912 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:09:41.912 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:09:41.912 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:09:42.342 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-30T13:09:42.344 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-30T13:09:42.345 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T13:09:42.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T13:09:42.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:09:42.463 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:09:42.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T13:09:42.517 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:42.517 INFO:tasks.ceph:role: osd.6 2023-10-30T13:09:42.517 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:09:42.517 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T13:09:42.581 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:09:42.581 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:09:42.582 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:09:42.582 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:09:42.582 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:09:42.582 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:09:42.582 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:09:42.583 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:09:42.583 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:09:42.583 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:09:43.047 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-30T13:09:43.049 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-30T13:09:43.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T13:09:43.142 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T13:09:43.151 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:09:43.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:09:43.152 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T13:09:43.208 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T13:09:43.209 INFO:tasks.ceph:role: osd.7 2023-10-30T13:09:43.209 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:09:43.209 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T13:09:43.274 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T13:09:43.274 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T13:09:43.274 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T13:09:43.274 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T13:09:43.275 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T13:09:43.275 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2023-10-30T13:09:43.275 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T13:09:43.275 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T13:09:43.275 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T13:09:43.275 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T13:09:43.732 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2023-10-30T13:09:43.734 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2023-10-30T13:09:43.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T13:09:43.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T13:09:43.886 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T13:09:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:09:43.887 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:44.004 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:44.000+0000 7f6b8eccd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T13:09:44.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:44.004+0000 7f6b8eccd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T13:09:44.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:44.004+0000 7f6b8eccd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T13:09:44.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:44.004+0000 7f6b8eccd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T13:09:44.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:44.004+0000 7f6b8eccd640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T13:09:44.014 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:44.012+0000 7f6b8eccd640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T13:09:47.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T13:09:47.028 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:47.112 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:47.108+0000 7f5f44de7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T13:09:47.112 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:47.108+0000 7f5f44de7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T13:09:47.113 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:47.108+0000 7f5f44de7640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T13:09:47.113 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:47.108+0000 7f5f44de7640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T13:09:47.113 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:47.108+0000 7f5f44de7640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T13:09:47.114 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:47.112+0000 7f5f44de7640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T13:09:50.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T13:09:50.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:50.272 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:50.268+0000 7fe862e32640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T13:09:50.273 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:50.272+0000 7fe862e32640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T13:09:50.273 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:50.272+0000 7fe862e32640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T13:09:50.273 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:50.272+0000 7fe862e32640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T13:09:50.274 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:50.272+0000 7fe862e32640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T13:09:50.274 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:50.272+0000 7fe862e32640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T13:09:52.510 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:09:52.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:09:53.273 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T13:09:53.286 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:53.372 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:53.368+0000 7fda7302b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T13:09:53.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:53.372+0000 7fda7302b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T13:09:53.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:53.372+0000 7fda7302b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T13:09:53.373 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:53.372+0000 7fda7302b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T13:09:53.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:53.372+0000 7fda7302b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T13:09:53.374 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:09:53.372+0000 7fda7302b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T13:09:56.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T13:09:56.429 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T13:09:56.429 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:56.429 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T13:09:56.452 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:56.452 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-30T13:09:56.515 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:56.515 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-30T13:09:56.579 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:56.579 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-10-30T13:09:56.642 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:56.642 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T13:09:56.706 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:56.706 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T13:09:56.770 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:56.770 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T13:09:56.834 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:56.834 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T13:09:56.897 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:56.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T13:09:56.912 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:56.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-30T13:09:56.964 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:56.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-30T13:09:57.020 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:57.020 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-10-30T13:09:57.072 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:57.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T13:09:57.124 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:57.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T13:09:57.175 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:57.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T13:09:57.228 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:09:57.228 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T13:09:57.280 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:09:57.281 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T13:09:57.290 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T13:09:57.290 DEBUG:teuthology.orchestra.run.smithi148:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T13:09:57.337 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T13:09:57.348 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.y] 2023-10-30T13:09:57.348 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAEqz9l95XHLxAAfvhnohNwzyqBJ3mQ6O/Z/A== 2023-10-30T13:09:57.349 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2023-10-30T13:09:57.349 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAEqz9l9B7aNhAAkhvrj2FnVbU/vRzobQf5XQ== 2023-10-30T13:09:57.349 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2023-10-30T13:09:57.349 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAFqz9lA7J0AhAAMLcGSEWg9CmnHAN+UerWzQ== 2023-10-30T13:09:57.349 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.e] 2023-10-30T13:09:57.350 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAFqz9l+XTECRAAhxqdgj9JXyEiU6GAWfdB+Q== 2023-10-30T13:09:57.350 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2023-10-30T13:09:57.350 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAIqz9loJ2LKxAA9jggNHaKfjL9VqvkRCoZpg== 2023-10-30T13:09:57.350 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2023-10-30T13:09:57.350 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQALqz9lqXoFMxAAFfh2BjDCiXxlczlQCgXuLg== 2023-10-30T13:09:57.350 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2023-10-30T13:09:57.351 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAOqz9lxj1SORAAEBm5GJfH2GeueIGc0uqLRg== 2023-10-30T13:09:57.351 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.3] 2023-10-30T13:09:57.351 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQASqz9lle5WBBAAIBg6EDDVFLlslFS4QNYv+A== 2023-10-30T13:09:57.351 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2023-10-30T13:09:57.351 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAEqz9lLb+uMhAAj2OThrB5SeK8ymPY83Xwmg== 2023-10-30T13:09:57.351 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2023-10-30T13:09:57.352 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAFqz9lIhT+DRAAvepZnCdxLoD72hRVNVvIkg== 2023-10-30T13:09:57.352 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2023-10-30T13:09:57.352 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAFqz9lgkocFBAAheM8jTvFesaL6I7hLtWN0A== 2023-10-30T13:09:57.352 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.f] 2023-10-30T13:09:57.352 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAFqz9lmYFrHBAAjnse6idyiJJJN0wlsEKWhQ== 2023-10-30T13:09:57.353 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.4] 2023-10-30T13:09:57.353 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAYqz9lGAVPABAAKdjizR/hcfEtgc9hlxPcNw== 2023-10-30T13:09:57.353 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.5] 2023-10-30T13:09:57.353 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAbqz9lODu9BhAATaDO9l5SU2X3GvgqaDZ8Yg== 2023-10-30T13:09:57.353 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.6] 2023-10-30T13:09:57.353 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAeqz9laoBPEBAA5tjitEoEiQMdYCMQA6JFJQ== 2023-10-30T13:09:57.354 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.7] 2023-10-30T13:09:57.354 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAhqz9lGVlDFhAAkwzD4NP1xsNAsUIfdLp/LQ== 2023-10-30T13:09:57.354 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2023-10-30T13:09:57.354 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAFqz9lGQxfIhAA+LXeIvKBMT+EbcTgyjT93Q== 2023-10-30T13:09:57.355 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.y] 2023-10-30T13:09:57.356 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAEqz9l95XHLxAAfvhnohNwzyqBJ3mQ6O/Z/A== 2023-10-30T13:09:57.356 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.a] 2023-10-30T13:09:57.356 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAEqz9l9B7aNhAAkhvrj2FnVbU/vRzobQf5XQ== 2023-10-30T13:09:57.356 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.c] 2023-10-30T13:09:57.356 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAFqz9lA7J0AhAAMLcGSEWg9CmnHAN+UerWzQ== 2023-10-30T13:09:57.356 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.e] 2023-10-30T13:09:57.357 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAFqz9l+XTECRAAhxqdgj9JXyEiU6GAWfdB+Q== 2023-10-30T13:09:57.357 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.0] 2023-10-30T13:09:57.357 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAIqz9loJ2LKxAA9jggNHaKfjL9VqvkRCoZpg== 2023-10-30T13:09:57.357 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.1] 2023-10-30T13:09:57.357 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQALqz9lqXoFMxAAFfh2BjDCiXxlczlQCgXuLg== 2023-10-30T13:09:57.357 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.2] 2023-10-30T13:09:57.358 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAOqz9lxj1SORAAEBm5GJfH2GeueIGc0uqLRg== 2023-10-30T13:09:57.358 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.3] 2023-10-30T13:09:57.358 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQASqz9lle5WBBAAIBg6EDDVFLlslFS4QNYv+A== 2023-10-30T13:09:57.358 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.x] 2023-10-30T13:09:57.358 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAEqz9lLb+uMhAAj2OThrB5SeK8ymPY83Xwmg== 2023-10-30T13:09:57.358 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.b] 2023-10-30T13:09:57.359 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAFqz9lIhT+DRAAvepZnCdxLoD72hRVNVvIkg== 2023-10-30T13:09:57.359 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.d] 2023-10-30T13:09:57.359 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAFqz9lgkocFBAAheM8jTvFesaL6I7hLtWN0A== 2023-10-30T13:09:57.359 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.f] 2023-10-30T13:09:57.359 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAFqz9lmYFrHBAAjnse6idyiJJJN0wlsEKWhQ== 2023-10-30T13:09:57.359 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.4] 2023-10-30T13:09:57.360 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAYqz9lGAVPABAAKdjizR/hcfEtgc9hlxPcNw== 2023-10-30T13:09:57.360 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.5] 2023-10-30T13:09:57.360 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAbqz9lODu9BhAATaDO9l5SU2X3GvgqaDZ8Yg== 2023-10-30T13:09:57.360 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.6] 2023-10-30T13:09:57.360 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAeqz9laoBPEBAA5tjitEoEiQMdYCMQA6JFJQ== 2023-10-30T13:09:57.360 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.7] 2023-10-30T13:09:57.361 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAhqz9lGVlDFhAAkwzD4NP1xsNAsUIfdLp/LQ== 2023-10-30T13:09:57.361 INFO:teuthology.orchestra.run.smithi148.stdout:[client.0] 2023-10-30T13:09:57.361 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQAFqz9lGQxfIhAA+LXeIvKBMT+EbcTgyjT93Q== 2023-10-30T13:09:57.361 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:09:57.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:09:57.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:57.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:57.515 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:57.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:57.586 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:57.590 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:57.676 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:57.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:57.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:57.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:57.800 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:57.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:57.880 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:57.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:57.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:09:57.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T13:09:58.039 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:58.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:58.123 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:58.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:58.210 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:58.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T13:09:58.267 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:58.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:58.339 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:58.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:58.428 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:58.431 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:58.512 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:58.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T13:09:58.600 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T13:09:58.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T13:09:58.688 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T13:09:58.688 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T13:09:58.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T13:09:58.830 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T13:09:58.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T13:09:58.865 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T13:09:58.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T13:09:58.976 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T13:09:59.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T13:09:59.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T13:09:59.138 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:59.141 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T13:09:59.184 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T13:09:59.185 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T13:09:59.185 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T13:09:59.193 INFO:tasks.ceph.mon.a:Started 2023-10-30T13:09:59.193 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T13:09:59.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T13:09:59.230 INFO:tasks.ceph.mon.b:Started 2023-10-30T13:09:59.230 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T13:09:59.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T13:09:59.232 INFO:tasks.ceph.mon.c:Started 2023-10-30T13:09:59.232 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T13:09:59.233 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T13:09:59.233 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-30T13:09:59.237 INFO:tasks.ceph.mgr.y:Started 2023-10-30T13:09:59.237 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T13:09:59.237 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-30T13:09:59.239 INFO:tasks.ceph.mgr.x:Started 2023-10-30T13:09:59.240 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T13:09:59.240 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T13:09:59.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.377+0000 7f9f700ea640 1 Processor -- start 2023-10-30T13:09:59.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.377+0000 7f9f700ea640 1 -- start start 2023-10-30T13:09:59.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f700ea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:59.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f700ea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f680c0a30 con 0x7f9f680c6e60 2023-10-30T13:09:59.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f700ea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6806f650 con 0x7f9f680722a0 2023-10-30T13:09:59.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f700ea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f680c85b0 con 0x7f9f681508c0 2023-10-30T13:09:59.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f6de86640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:59.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f6e687640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9f680c6e60 0x7f9f68156930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46972/0 (socket says 172.21.15.148:46972) 2023-10-30T13:09:59.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 learned_addr learned my addr 172.21.15.148:0/140015570 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:09:59.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f6de86640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37224/0 (socket says 172.21.15.148:37224) 2023-10-30T13:09:59.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 msgr2=0x7f9f680c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:09:59.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 msgr2=0x7f9f680c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:09:59.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:09:59.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.381+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:09:59.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.585+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:59.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.585+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 msgr2=0x7f9f680c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:09:59.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.585+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 msgr2=0x7f9f680c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:09:59.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.585+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:09:59.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.585+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:09:59.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.989+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:59.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.989+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 msgr2=0x7f9f680c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:09:59.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.989+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 msgr2=0x7f9f680c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:09:59.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.989+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:09:59.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:09:59.989+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:10:00.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:00.793+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:00.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:00.793+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 msgr2=0x7f9f680c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:10:00.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:00.793+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 msgr2=0x7f9f680c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:00.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:00.793+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:00.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:00.793+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:10:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 msgr2=0x7f9f680c6920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:10:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f681508c0 0x7f9f680c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 >> v1:172.21.15.150:6789/0 conn(0x7f9f680722a0 legacy=0x7f9f680bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 >> v1:172.21.15.148:6789/0 conn(0x7f9f680c6e60 legacy=0x7f9f68156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f680c85b0 con 0x7f9f4c006200 2023-10-30T13:10:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c008d60 con 0x7f9f4c003720 2023-10-30T13:10:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 msgr2=0x7f9f4c0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:10:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 msgr2=0x7f9f4c0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:10:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 msgr2=0x7f9f4c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:10:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 msgr2=0x7f9f4c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.381+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:10:02.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:02.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:02.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 msgr2=0x7f9f4c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:10:02.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 msgr2=0x7f9f4c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:02.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:02.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:10:02.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 msgr2=0x7f9f4c0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:10:02.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 msgr2=0x7f9f4c0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:02.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:02.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.585+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:10:02.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:02.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:02.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 msgr2=0x7f9f4c0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:10:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 msgr2=0x7f9f4c0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 msgr2=0x7f9f4c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:10:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 msgr2=0x7f9f4c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:02.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:10:02.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:02.989+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:10:03.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 msgr2=0x7f9f4c0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:10:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 msgr2=0x7f9f4c0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 msgr2=0x7f9f4c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:10:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 msgr2=0x7f9f4c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:10:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:03.793+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:10:05.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 msgr2=0x7f9f4c005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:10:05.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003720 0x7f9f4c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:05.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 msgr2=0x7f9f4c0086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:10:05.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c006200 0x7f9f4c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:05.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 reap_dead start 2023-10-30T13:10:05.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:05.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:05.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6806f930 con 0x7f9f680c6e60 2023-10-30T13:10:05.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f680c85b0 con 0x7f9f4c006b70 2023-10-30T13:10:05.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c008d60 con 0x7f9f4c003b60 2023-10-30T13:10:05.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:05.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:05.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 msgr2=0x7f9f4c006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:10:05.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 msgr2=0x7f9f4c006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:05.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:05.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 msgr2=0x7f9f4c006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:10:05.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 msgr2=0x7f9f4c006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:05.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:05.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:10:05.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.381+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:10:05.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:05.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:05.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 msgr2=0x7f9f4c006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:10:05.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 msgr2=0x7f9f4c006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:05.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:05.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:10:05.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 msgr2=0x7f9f4c006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:10:05.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 msgr2=0x7f9f4c006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:05.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:05.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.585+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:10:05.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:05.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:05.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 msgr2=0x7f9f4c006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:10:05.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 msgr2=0x7f9f4c006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:05.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:05.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:10:05.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 msgr2=0x7f9f4c006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:10:05.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 msgr2=0x7f9f4c006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:05.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:05.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:05.989+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:10:06.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:06.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 msgr2=0x7f9f4c006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T13:10:06.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 msgr2=0x7f9f4c006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:06.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:06.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:06.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:10:06.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 msgr2=0x7f9f4c006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T13:10:06.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 msgr2=0x7f9f4c006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:06.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:06.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:06.793+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T13:10:08.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 msgr2=0x7f9f4c006630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:10:08.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f4c003b60 0x7f9f4c006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:08.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 msgr2=0x7f9f4c006ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:10:08.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.148:3300/0 conn(0x7f9f4c006b70 0x7f9f4c006ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:08.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 >> v1:172.21.15.148:6789/0 conn(0x7f9f680c6e60 legacy=0x7f9f4c00b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:08.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f5f7fe640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:08.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c0087c0 con 0x7f9f4c008220 2023-10-30T13:10:08.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f5f7fe640 1 -- 172.21.15.148:0/140015570 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f680c85b0 con 0x7f9f4c0076e0 2023-10-30T13:10:08.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:08.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 msgr2=0x7f9f4c007b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:10:08.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 msgr2=0x7f9f4c007b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:08.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:08.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.381+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:10:08.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.585+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:08.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.585+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 msgr2=0x7f9f4c007b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:10:08.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.585+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 msgr2=0x7f9f4c007b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:08.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.585+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:08.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.585+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T13:10:08.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.989+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:08.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.989+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 msgr2=0x7f9f4c007b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T13:10:08.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.989+0000 7f9f6de86640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 msgr2=0x7f9f4c007b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:08.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.989+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:08.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:08.989+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T13:10:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.609+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862351042 0 0) 0x7f9f4c0087c0 con 0x7f9f4c008220 2023-10-30T13:10:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.609+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f54003530 con 0x7f9f4c008220 2023-10-30T13:10:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652968476 0 0) 0x7f9f54003530 con 0x7f9f4c008220 2023-10-30T13:10:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 msgr2=0x7f9f4c007b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:10:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 --2- 172.21.15.148:0/140015570 >> v2:172.21.15.150:3300/0 conn(0x7f9f4c0076e0 0x7f9f4c007b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c015030 con 0x7f9f4c008220 2023-10-30T13:10:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f9f58007d80 con 0x7f9f4c008220 2023-10-30T13:10:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 >> v1:172.21.15.150:6789/0 conn(0x7f9f4c008220 legacy=0x7f9f4c0128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6e687640 1 -- 172.21.15.148:0/140015570 reap_dead start 2023-10-30T13:10:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 0x7f9f54003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 0x7f9f54004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f5400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680c85b0 con 0x7f9f540052b0 2023-10-30T13:10:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5400c160 con 0x7f9f54003980 2023-10-30T13:10:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5400c320 con 0x7f9f540047d0 2023-10-30T13:10:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6e687640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f5400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.617+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 0x7f9f54004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.621+0000 7f9f6de86640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 0x7f9f54003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.621+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 msgr2=0x7f9f54003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.621+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 0x7f9f54003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.621+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 msgr2=0x7f9f5400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.621+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f5400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.621+0000 7f9f6d685640 1 -- 172.21.15.148:0/140015570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f54003530 con 0x7f9f540047d0 2023-10-30T13:10:09.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.621+0000 7f9f6d685640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 0x7f9f54004c50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9f58008fd0 tx=0x7f9f580207a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3e5b6d74cfa8aa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:09.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f580052d0 con 0x7f9f540047d0 2023-10-30T13:10:09.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f580ac050 con 0x7f9f540047d0 2023-10-30T13:10:09.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f6ce84640 1 -- 172.21.15.148:0/140015570 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58003840 con 0x7f9f540047d0 2023-10-30T13:10:09.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f700ea640 1 -- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 msgr2=0x7f9f54004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:09.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f700ea640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 0x7f9f54004c50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9f58008fd0 tx=0x7f9f580207a0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f700ea640 1 -- 172.21.15.148:0/140015570 shutdown_connections 2023-10-30T13:10:09.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f700ea640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f5400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f700ea640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 0x7f9f54004c50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f700ea640 1 --2- 172.21.15.148:0/140015570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 0x7f9f54003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f700ea640 1 -- 172.21.15.148:0/140015570 >> 172.21.15.148:0/140015570 conn(0x7f9f680c7910 msgr2=0x7f9f680c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:09.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f700ea640 1 -- 172.21.15.148:0/140015570 shutdown_connections 2023-10-30T13:10:09.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.625+0000 7f9f700ea640 1 -- 172.21.15.148:0/140015570 wait complete. 2023-10-30T13:10:09.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 Processor -- start 2023-10-30T13:10:09.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 -- start start 2023-10-30T13:10:09.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 0x7f9f6814e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:09.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 0x7f9f6814fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:09.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f680ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:09.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f68152cb0 con 0x7f9f540052b0 2023-10-30T13:10:09.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6814ea10 con 0x7f9f54003980 2023-10-30T13:10:09.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6814e730 con 0x7f9f540047d0 2023-10-30T13:10:09.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6e687640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f680ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:09.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6e687640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f680ca020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38564/0 (socket says 172.21.15.148:38564) 2023-10-30T13:10:09.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6de86640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 0x7f9f6814e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:09.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6d685640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 0x7f9f6814fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:09.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6e687640 1 -- 172.21.15.148:0/1631559318 learned_addr learned my addr 172.21.15.148:0/1631559318 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:09.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6e687640 1 -- 172.21.15.148:0/1631559318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 msgr2=0x7f9f6814fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:09.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6e687640 1 --2- 172.21.15.148:0/1631559318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 0x7f9f6814fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6e687640 1 -- 172.21.15.148:0/1631559318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 msgr2=0x7f9f6814e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:09.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6e687640 1 --2- 172.21.15.148:0/1631559318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 0x7f9f6814e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6e687640 1 -- 172.21.15.148:0/1631559318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f68152380 con 0x7f9f540052b0 2023-10-30T13:10:09.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6de86640 1 --2- 172.21.15.148:0/1631559318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 0x7f9f6814e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:09.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f6e687640 1 --2- 172.21.15.148:0/1631559318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f680ca020 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9f68072e50 tx=0x7f9f60007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:09.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f5e7fc640 1 -- 172.21.15.148:0/1631559318 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60015620 con 0x7f9f540052b0 2023-10-30T13:10:09.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f5e7fc640 1 -- 172.21.15.148:0/1631559318 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f60019070 con 0x7f9f540052b0 2023-10-30T13:10:09.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 -- 172.21.15.148:0/1631559318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f68152660 con 0x7f9f540052b0 2023-10-30T13:10:09.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f5e7fc640 1 -- 172.21.15.148:0/1631559318 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f600159a0 con 0x7f9f540052b0 2023-10-30T13:10:09.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f700ea640 1 -- 172.21.15.148:0/1631559318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f68150270 con 0x7f9f540052b0 2023-10-30T13:10:09.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f5e7fc640 1 -- 172.21.15.148:0/1631559318 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9f600273a0 con 0x7f9f540052b0 2023-10-30T13:10:09.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.629+0000 7f9f5e7fc640 1 -- 172.21.15.148:0/1631559318 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9f600146e0 con 0x7f9f540052b0 2023-10-30T13:10:09.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.633+0000 7f9f700ea640 1 -- 172.21.15.148:0/1631559318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f68152e60 con 0x7f9f540052b0 2023-10-30T13:10:09.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.661+0000 7f9f5e7fc640 1 -- 172.21.15.148:0/1631559318 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f9f68152e60 con 0x7f9f540052b0 2023-10-30T13:10:09.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.725+0000 7f9f700ea640 1 -- 172.21.15.148:0/1631559318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9f6804e170 con 0x7f9f540052b0 2023-10-30T13:10:09.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.725+0000 7f9f5e7fc640 1 -- 172.21.15.148:0/1631559318 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9f60023060 con 0x7f9f540052b0 2023-10-30T13:10:09.729 INFO:teuthology.orchestra.run.smithi148.stderr:adjusted tunables profile to default 2023-10-30T13:10:09.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.729+0000 7f9f700ea640 1 -- 172.21.15.148:0/1631559318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 msgr2=0x7f9f680ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:09.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.729+0000 7f9f700ea640 1 --2- 172.21.15.148:0/1631559318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f680ca020 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9f68072e50 tx=0x7f9f60007a50 comp rx=0 tx=0).stop 2023-10-30T13:10:09.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.729+0000 7f9f700ea640 1 -- 172.21.15.148:0/1631559318 shutdown_connections 2023-10-30T13:10:09.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.729+0000 7f9f700ea640 1 --2- 172.21.15.148:0/1631559318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f540052b0 0x7f9f680ca020 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.729+0000 7f9f700ea640 1 --2- 172.21.15.148:0/1631559318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f540047d0 0x7f9f6814fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.729+0000 7f9f700ea640 1 --2- 172.21.15.148:0/1631559318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f54003980 0x7f9f6814e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:09.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.729+0000 7f9f700ea640 1 -- 172.21.15.148:0/1631559318 >> 172.21.15.148:0/1631559318 conn(0x7f9f680c7910 msgr2=0x7f9f680bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:09.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.729+0000 7f9f700ea640 1 -- 172.21.15.148:0/1631559318 shutdown_connections 2023-10-30T13:10:09.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:09.729+0000 7f9f700ea640 1 -- 172.21.15.148:0/1631559318 wait complete. 2023-10-30T13:10:09.751 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T13:10:09.751 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T13:10:09.751 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:10:09.752 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T13:10:09.773 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:10:09.773 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T13:10:09.834 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:10:09.834 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T13:10:09.859 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:09.857+0000 7f0eda53a180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T13:10:09.879 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:10:09.879 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T13:10:09.883 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:09.880+0000 7ff1d1a2c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T13:10:09.942 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:10:09.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T13:10:09.956 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:10:09.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T13:10:10.012 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:10:10.012 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T13:10:10.068 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-30T13:10:10.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T13:10:10.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new dff19510-6825-4999-b159-238bfb5ff95f 0 2023-10-30T13:10:10.287 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.284+0000 7fcdc2036640 1 Processor -- start 2023-10-30T13:10:10.288 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.284+0000 7fcdc2036640 1 -- start start 2023-10-30T13:10:10.289 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.284+0000 7fcdc2036640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdbc151900 0x7fcdbc0c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.289 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.284+0000 7fcdc2036640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc06e570 0x7fcdbc06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.289 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.284+0000 7fcdc2036640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc06e360 con 0x7fcdbc0c79a0 2023-10-30T13:10:10.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.284+0000 7fcdc2036640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdbc06eab0 con 0x7fcdbc151900 2023-10-30T13:10:10.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.284+0000 7fcdc2036640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc0c7480 con 0x7fcdbc06e570 2023-10-30T13:10:10.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbb7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdbc151900 0x7fcdbc0c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbaffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc06e570 0x7fcdbc06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.290 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbb7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdbc151900 0x7fcdbc0c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42346/0 (socket says 172.21.15.150:42346) 2023-10-30T13:10:10.291 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbb7fe640 1 -- 172.21.15.150:0/2333149223 learned_addr learned my addr 172.21.15.150:0/2333149223 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:10.291 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbaffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc06e570 0x7fcdbc06d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:51844/0 (socket says 172.21.15.150:51844) 2023-10-30T13:10:10.291 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbaffd640 1 -- 172.21.15.150:0/2333149223 >> v2:172.21.15.150:3300/0 conn(0x7fcdbc151900 msgr2=0x7fcdbc0c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:10.291 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbaffd640 1 --2- 172.21.15.150:0/2333149223 >> v2:172.21.15.150:3300/0 conn(0x7fcdbc151900 0x7fcdbc0c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.291 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbaffd640 1 -- 172.21.15.150:0/2333149223 >> v1:172.21.15.150:6789/0 conn(0x7fcdbc0c79a0 legacy=0x7fcdbc06f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbaffd640 1 -- 172.21.15.150:0/2333149223 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc06d6f0 con 0x7fcdbc06e570 2023-10-30T13:10:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbb7fe640 1 --2- 172.21.15.150:0/2333149223 >> v2:172.21.15.150:3300/0 conn(0x7fcdbc151900 0x7fcdbc0c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdbaffd640 1 --2- 172.21.15.150:0/2333149223 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc06e570 0x7fcdbc06d1b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000f370 tx=0x7fcdb0020d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba1d05e4a66aaf4b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdba7fc640 1 -- 172.21.15.150:0/2333149223 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb00a9ac0 con 0x7fcdbc06e570 2023-10-30T13:10:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdba7fc640 1 -- 172.21.15.150:0/2333149223 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb00a9c20 con 0x7fcdbc06e570 2023-10-30T13:10:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdba7fc640 1 -- 172.21.15.150:0/2333149223 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000d0d0 con 0x7fcdbc06e570 2023-10-30T13:10:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdc2036640 1 -- 172.21.15.150:0/2333149223 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc06e570 msgr2=0x7fcdbc06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdc2036640 1 --2- 172.21.15.150:0/2333149223 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc06e570 0x7fcdbc06d1b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000f370 tx=0x7fcdb0020d20 comp rx=0 tx=0).stop 2023-10-30T13:10:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdc2036640 1 -- 172.21.15.150:0/2333149223 shutdown_connections 2023-10-30T13:10:10.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdc2036640 1 --2- 172.21.15.150:0/2333149223 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc06e570 0x7fcdbc06d1b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdc2036640 1 --2- 172.21.15.150:0/2333149223 >> v2:172.21.15.150:3300/0 conn(0x7fcdbc151900 0x7fcdbc0c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdc2036640 1 -- 172.21.15.150:0/2333149223 >> 172.21.15.150:0/2333149223 conn(0x7fcdbc0ccfe0 msgr2=0x7fcdbc0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:10.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdc2036640 1 -- 172.21.15.150:0/2333149223 shutdown_connections 2023-10-30T13:10:10.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.288+0000 7fcdc2036640 1 -- 172.21.15.150:0/2333149223 wait complete. 2023-10-30T13:10:10.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 Processor -- start 2023-10-30T13:10:10.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 -- start start 2023-10-30T13:10:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcdbc06e570 0x7fcdbc14e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc0c79a0 0x7fcdbc1eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc151900 0x7fcdbc15ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc1edab0 con 0x7fcdbc06e570 2023-10-30T13:10:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc1eded0 con 0x7fcdbc151900 2023-10-30T13:10:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc1eced0 con 0x7fcdbc0c79a0 2023-10-30T13:10:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc151900 0x7fcdbc15ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc151900 0x7fcdbc15ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42356/0 (socket says 172.21.15.150:42356) 2023-10-30T13:10:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbaffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc0c79a0 0x7fcdbc1eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbbfff640 1 -- 172.21.15.150:0/2281374716 learned_addr learned my addr 172.21.15.150:0/2281374716 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:10.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbb7fe640 1 --2- 172.21.15.150:0/2281374716 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcdbc06e570 0x7fcdbc14e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbbfff640 1 -- 172.21.15.150:0/2281374716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc0c79a0 msgr2=0x7fcdbc1eaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbbfff640 1 --2- 172.21.15.150:0/2281374716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc0c79a0 0x7fcdbc1eaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbbfff640 1 -- 172.21.15.150:0/2281374716 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcdbc06e570 msgr2=0x7fcdbc14e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.300 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbbfff640 1 --2- 172.21.15.150:0/2281374716 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcdbc06e570 0x7fcdbc14e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbbfff640 1 -- 172.21.15.150:0/2281374716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac0094d0 con 0x7fcdbc151900 2023-10-30T13:10:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdbbfff640 1 --2- 172.21.15.150:0/2281374716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc151900 0x7fcdbc15ad30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c430 tx=0x7fcdb400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcda3fff640 1 -- 172.21.15.150:0/2281374716 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4005220 con 0x7fcdbc151900 2023-10-30T13:10:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcda3fff640 1 -- 172.21.15.150:0/2281374716 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb4005380 con 0x7fcdbc151900 2023-10-30T13:10:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 -- 172.21.15.150:0/2281374716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb000f050 con 0x7fcdbc151900 2023-10-30T13:10:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcda3fff640 1 -- 172.21.15.150:0/2281374716 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4007270 con 0x7fcdbc151900 2023-10-30T13:10:10.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 -- 172.21.15.150:0/2281374716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc14a220 con 0x7fcdbc151900 2023-10-30T13:10:10.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcda3fff640 1 -- 172.21.15.150:0/2281374716 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcdb40067e0 con 0x7fcdbc151900 2023-10-30T13:10:10.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcda3fff640 1 -- 172.21.15.150:0/2281374716 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcdb40039a0 con 0x7fcdbc151900 2023-10-30T13:10:10.302 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.292+0000 7fcdc2036640 1 -- 172.21.15.150:0/2281374716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdbc1edbf0 con 0x7fcdbc151900 2023-10-30T13:10:10.303 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.300+0000 7fcda3fff640 1 -- 172.21.15.150:0/2281374716 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fcdbc1edbf0 con 0x7fcdbc151900 2023-10-30T13:10:10.369 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.368+0000 7fcdc2036640 1 -- 172.21.15.150:0/2281374716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dff19510-6825-4999-b159-238bfb5ff95f", "id": 0} v 0) v1 -- 0x7fcdbc04e170 con 0x7fcdbc151900 2023-10-30T13:10:10.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcda3fff640 1 -- 172.21.15.150:0/2281374716 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dff19510-6825-4999-b159-238bfb5ff95f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcdb4006cc0 con 0x7fcdbc151900 2023-10-30T13:10:10.377 INFO:teuthology.orchestra.run.smithi150.stdout:0 2023-10-30T13:10:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcdc2036640 1 -- 172.21.15.150:0/2281374716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc151900 msgr2=0x7fcdbc15ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcdc2036640 1 --2- 172.21.15.150:0/2281374716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc151900 0x7fcdbc15ad30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c430 tx=0x7fcdb400c8f0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcdc2036640 1 -- 172.21.15.150:0/2281374716 shutdown_connections 2023-10-30T13:10:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcdc2036640 1 --2- 172.21.15.150:0/2281374716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc151900 0x7fcdbc15ad30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcdc2036640 1 --2- 172.21.15.150:0/2281374716 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcdbc0c79a0 0x7fcdbc1eaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcdc2036640 1 --2- 172.21.15.150:0/2281374716 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcdbc06e570 0x7fcdbc14e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcdc2036640 1 -- 172.21.15.150:0/2281374716 >> 172.21.15.150:0/2281374716 conn(0x7fcdbc0ccfe0 msgr2=0x7fcdbc150bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:10.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcdc2036640 1 -- 172.21.15.150:0/2281374716 shutdown_connections 2023-10-30T13:10:10.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.376+0000 7fcdc2036640 1 -- 172.21.15.150:0/2281374716 wait complete. 2023-10-30T13:10:10.391 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:10.389+0000 7f0eda53a180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T13:10:10.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 4fc28859-6374-44ef-96b4-9f7c0c4ddcb0 1 2023-10-30T13:10:10.464 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:10.460+0000 7ff1d1a2c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T13:10:10.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.508+0000 7f5b71eac640 1 Processor -- start 2023-10-30T13:10:10.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.508+0000 7f5b71eac640 1 -- start start 2023-10-30T13:10:10.513 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.508+0000 7f5b71eac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c151900 0x7f5b6c0c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.514 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.508+0000 7f5b71eac640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b6c06e570 0x7f5b6c06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.514 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b6c0c79a0 0x7f5b6c06f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.514 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c06eab0 con 0x7f5b6c06e570 2023-10-30T13:10:10.514 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c06e360 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.514 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0c7480 con 0x7f5b6c151900 2023-10-30T13:10:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c151900 0x7f5b6c0c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b6c0c79a0 0x7f5b6c06f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c151900 0x7f5b6c0c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:51864/0 (socket says 172.21.15.150:51864) 2023-10-30T13:10:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b6c0c79a0 0x7f5b6c06f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42368/0 (socket says 172.21.15.150:42368) 2023-10-30T13:10:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6b7fe640 1 -- 172.21.15.150:0/3800602915 learned_addr learned my addr 172.21.15.150:0/3800602915 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6affd640 1 --2- 172.21.15.150:0/3800602915 >> v2:172.21.15.148:3300/0 conn(0x7f5b6c06e570 0x7f5b6c06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6b7fe640 1 -- 172.21.15.150:0/3800602915 >> v2:172.21.15.150:3300/0 conn(0x7f5b6c0c79a0 msgr2=0x7f5b6c06f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6b7fe640 1 --2- 172.21.15.150:0/3800602915 >> v2:172.21.15.150:3300/0 conn(0x7f5b6c0c79a0 0x7f5b6c06f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6b7fe640 1 -- 172.21.15.150:0/3800602915 >> v2:172.21.15.148:3300/0 conn(0x7f5b6c06e570 msgr2=0x7f5b6c06d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6b7fe640 1 --2- 172.21.15.150:0/3800602915 >> v2:172.21.15.148:3300/0 conn(0x7f5b6c06e570 0x7f5b6c06d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6b7fe640 1 -- 172.21.15.150:0/3800602915 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c06d6f0 con 0x7f5b6c151900 2023-10-30T13:10:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6bfff640 1 --2- 172.21.15.150:0/3800602915 >> v2:172.21.15.150:3300/0 conn(0x7f5b6c0c79a0 0x7f5b6c06f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:10.519 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6b7fe640 1 --2- 172.21.15.150:0/3800602915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c151900 0x7f5b6c0c22f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0b2700 tx=0x7f5b5c01c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fadf09c8a68fb11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:10.520 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6a7fc640 1 -- 172.21.15.150:0/3800602915 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c0a5820 con 0x7f5b6c151900 2023-10-30T13:10:10.520 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6a7fc640 1 -- 172.21.15.150:0/3800602915 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c0a5980 con 0x7f5b6c151900 2023-10-30T13:10:10.520 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b6a7fc640 1 -- 172.21.15.150:0/3800602915 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c0a5d10 con 0x7f5b6c151900 2023-10-30T13:10:10.520 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 -- 172.21.15.150:0/3800602915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c151900 msgr2=0x7f5b6c0c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.520 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 --2- 172.21.15.150:0/3800602915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c151900 0x7f5b6c0c22f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0b2700 tx=0x7f5b5c01c9a0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.521 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 -- 172.21.15.150:0/3800602915 shutdown_connections 2023-10-30T13:10:10.521 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 --2- 172.21.15.150:0/3800602915 >> v2:172.21.15.150:3300/0 conn(0x7f5b6c0c79a0 0x7f5b6c06f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.521 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 --2- 172.21.15.150:0/3800602915 >> v2:172.21.15.148:3300/0 conn(0x7f5b6c06e570 0x7f5b6c06d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.521 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 --2- 172.21.15.150:0/3800602915 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c151900 0x7f5b6c0c22f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.521 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 -- 172.21.15.150:0/3800602915 >> 172.21.15.150:0/3800602915 conn(0x7f5b6c0ccfe0 msgr2=0x7f5b6c0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:10.522 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 -- 172.21.15.150:0/3800602915 shutdown_connections 2023-10-30T13:10:10.522 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.512+0000 7f5b71eac640 1 -- 172.21.15.150:0/3800602915 wait complete. 2023-10-30T13:10:10.522 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 Processor -- start 2023-10-30T13:10:10.522 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 -- start start 2023-10-30T13:10:10.522 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c06e570 0x7f5b6c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.523 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b6c0c79a0 0x7f5b6c156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.523 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b6c151900 0x7f5b6c1e9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.523 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c1e8e40 con 0x7f5b6c151900 2023-10-30T13:10:10.523 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c1efb00 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.523 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c1efde0 con 0x7f5b6c06e570 2023-10-30T13:10:10.523 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b6c0c79a0 0x7f5b6c156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c06e570 0x7f5b6c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c06e570 0x7f5b6c152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:51878/0 (socket says 172.21.15.150:51878) 2023-10-30T13:10:10.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6b7fe640 1 -- 172.21.15.150:0/2993846577 learned_addr learned my addr 172.21.15.150:0/2993846577 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:10.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6bfff640 1 --2- 172.21.15.150:0/2993846577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b6c151900 0x7f5b6c1e9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6affd640 1 -- 172.21.15.150:0/2993846577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c06e570 msgr2=0x7f5b6c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6affd640 1 --2- 172.21.15.150:0/2993846577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c06e570 0x7f5b6c152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6affd640 1 -- 172.21.15.150:0/2993846577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b6c151900 msgr2=0x7f5b6c1e9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6affd640 1 --2- 172.21.15.150:0/2993846577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b6c151900 0x7f5b6c1e9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6affd640 1 -- 172.21.15.150:0/2993846577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b64009420 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b6affd640 1 --2- 172.21.15.150:0/2993846577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b6c0c79a0 0x7f5b6c156ab0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000d5f0 tx=0x7f5b6000d9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:10.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b53fff640 1 -- 172.21.15.150:0/2993846577 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60014620 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b53fff640 1 -- 172.21.15.150:0/2993846577 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b60014780 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 -- 172.21.15.150:0/2993846577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c009b30 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b53fff640 1 -- 172.21.15.150:0/2993846577 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60014b30 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 -- 172.21.15.150:0/2993846577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c1f03a0 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b53fff640 1 -- 172.21.15.150:0/2993846577 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5b6001d400 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b53fff640 1 -- 172.21.15.150:0/2993846577 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f5b600128c0 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.527 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.516+0000 7f5b71eac640 1 -- 172.21.15.150:0/2993846577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c04e170 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.527 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.520+0000 7f5b53fff640 1 -- 172.21.15.150:0/2993846577 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f5b6003cb40 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.583 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:10.581+0000 7f0eda53a180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T13:10:10.591 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.588+0000 7f5b71eac640 1 -- 172.21.15.150:0/2993846577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4fc28859-6374-44ef-96b4-9f7c0c4ddcb0", "id": 1} v 0) v1 -- 0x7f5b6c06ef60 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.602 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b53fff640 1 -- 172.21.15.150:0/2993846577 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4fc28859-6374-44ef-96b4-9f7c0c4ddcb0", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5b6002b030 con 0x7f5b6c0c79a0 2023-10-30T13:10:10.602 INFO:teuthology.orchestra.run.smithi150.stdout:1 2023-10-30T13:10:10.603 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b71eac640 1 -- 172.21.15.150:0/2993846577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b6c0c79a0 msgr2=0x7f5b6c156ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.603 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b71eac640 1 --2- 172.21.15.150:0/2993846577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b6c0c79a0 0x7f5b6c156ab0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000d5f0 tx=0x7f5b6000d9f0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.604 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b71eac640 1 -- 172.21.15.150:0/2993846577 shutdown_connections 2023-10-30T13:10:10.604 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b71eac640 1 --2- 172.21.15.150:0/2993846577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b6c151900 0x7f5b6c1e9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.604 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b71eac640 1 --2- 172.21.15.150:0/2993846577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b6c0c79a0 0x7f5b6c156ab0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.604 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b71eac640 1 --2- 172.21.15.150:0/2993846577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b6c06e570 0x7f5b6c152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.604 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b71eac640 1 -- 172.21.15.150:0/2993846577 >> 172.21.15.150:0/2993846577 conn(0x7f5b6c0ccfe0 msgr2=0x7f5b6c157760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:10.605 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b71eac640 1 -- 172.21.15.150:0/2993846577 shutdown_connections 2023-10-30T13:10:10.605 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.600+0000 7f5b71eac640 1 -- 172.21.15.150:0/2993846577 wait complete. 2023-10-30T13:10:10.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 21c3986b-480c-4bdd-a85b-3d51710c7822 2 2023-10-30T13:10:10.653 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:10.652+0000 7ff1d1a2c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T13:10:10.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.732+0000 7f7987fc7640 1 Processor -- start 2023-10-30T13:10:10.735 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.732+0000 7f7987fc7640 1 -- start start 2023-10-30T13:10:10.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.732+0000 7f7987fc7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79800c2e20 0x7f79800c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.732+0000 7f7987fc7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7980159be0 con 0x7f7980150800 2023-10-30T13:10:10.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.732+0000 7f7987fc7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7980156850 con 0x7f798015a070 2023-10-30T13:10:10.737 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.732+0000 7f7987fc7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79800c7020 con 0x7f79800c2e20 2023-10-30T13:10:10.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.732+0000 7f7985562640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79800c2e20 0x7f79800c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7985562640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79800c2e20 0x7f79800c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42382/0 (socket says 172.21.15.150:42382) 2023-10-30T13:10:10.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7985562640 1 -- 172.21.15.150:0/317852376 learned_addr learned my addr 172.21.15.150:0/317852376 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:10.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7985562640 1 -- 172.21.15.150:0/317852376 >> v1:172.21.15.150:6789/0 conn(0x7f798015a070 legacy=0x7f7980160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.738 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7985562640 1 -- 172.21.15.150:0/317852376 >> v1:172.21.15.148:6789/0 conn(0x7f7980150800 legacy=0x7f79800cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7985562640 1 -- 172.21.15.150:0/317852376 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7980156b30 con 0x7f79800c2e20 2023-10-30T13:10:10.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7985562640 1 --2- 172.21.15.150:0/317852376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79800c2e20 0x7f79800c6ae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f797000bb10 tx=0x7f7970021400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2527ce0a1bf1564d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:10.739 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7984d61640 1 -- 172.21.15.150:0/317852376 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79700b1600 con 0x7f79800c2e20 2023-10-30T13:10:10.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7984d61640 1 -- 172.21.15.150:0/317852376 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79700ad050 con 0x7f79800c2e20 2023-10-30T13:10:10.740 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7984d61640 1 -- 172.21.15.150:0/317852376 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79700b1980 con 0x7f79800c2e20 2023-10-30T13:10:10.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7987fc7640 1 -- 172.21.15.150:0/317852376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79800c2e20 msgr2=0x7f79800c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7987fc7640 1 --2- 172.21.15.150:0/317852376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79800c2e20 0x7f79800c6ae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f797000bb10 tx=0x7f7970021400 comp rx=0 tx=0).stop 2023-10-30T13:10:10.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7987fc7640 1 -- 172.21.15.150:0/317852376 shutdown_connections 2023-10-30T13:10:10.741 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7987fc7640 1 --2- 172.21.15.150:0/317852376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79800c2e20 0x7f79800c6ae0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7987fc7640 1 -- 172.21.15.150:0/317852376 >> 172.21.15.150:0/317852376 conn(0x7f798006c850 msgr2=0x7f798006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:10.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7987fc7640 1 -- 172.21.15.150:0/317852376 shutdown_connections 2023-10-30T13:10:10.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.736+0000 7f7987fc7640 1 -- 172.21.15.150:0/317852376 wait complete. 2023-10-30T13:10:10.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 Processor -- start 2023-10-30T13:10:10.742 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 -- start start 2023-10-30T13:10:10.744 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79800c2e20 0x7f7980142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.744 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7980150800 0x7f798014e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f798015a070 0x7f79800c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79801f2730 con 0x7f79800c2e20 2023-10-30T13:10:10.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798014ee80 con 0x7f798015a070 2023-10-30T13:10:10.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79801f2980 con 0x7f7980150800 2023-10-30T13:10:10.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7986564640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f798015a070 0x7f79800c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7985562640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7980150800 0x7f798014e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7986564640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f798015a070 0x7f79800c73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42384/0 (socket says 172.21.15.150:42384) 2023-10-30T13:10:10.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7986564640 1 -- 172.21.15.150:0/431385677 learned_addr learned my addr 172.21.15.150:0/431385677 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:10.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7985d63640 1 --2- 172.21.15.150:0/431385677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79800c2e20 0x7f7980142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.746 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7986564640 1 -- 172.21.15.150:0/431385677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7980150800 msgr2=0x7f798014e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7986564640 1 --2- 172.21.15.150:0/431385677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7980150800 0x7f798014e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7986564640 1 -- 172.21.15.150:0/431385677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79800c2e20 msgr2=0x7f7980142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7986564640 1 --2- 172.21.15.150:0/431385677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79800c2e20 0x7f7980142c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7986564640 1 -- 172.21.15.150:0/431385677 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797000b770 con 0x7f798015a070 2023-10-30T13:10:10.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7985562640 1 --2- 172.21.15.150:0/431385677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7980150800 0x7f798014e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:10.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7986564640 1 --2- 172.21.15.150:0/431385677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f798015a070 0x7f79800c73d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f797800c730 tx=0x7f797800cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:10.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f79767fc640 1 -- 172.21.15.150:0/431385677 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978003b80 con 0x7f798015a070 2023-10-30T13:10:10.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f79767fc640 1 -- 172.21.15.150:0/431385677 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7978003ce0 con 0x7f798015a070 2023-10-30T13:10:10.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 -- 172.21.15.150:0/431385677 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79801f2c60 con 0x7f798015a070 2023-10-30T13:10:10.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f79767fc640 1 -- 172.21.15.150:0/431385677 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797800aa30 con 0x7f798015a070 2023-10-30T13:10:10.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 -- 172.21.15.150:0/431385677 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79801f3220 con 0x7f798015a070 2023-10-30T13:10:10.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f79767fc640 1 -- 172.21.15.150:0/431385677 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f79780050d0 con 0x7f798015a070 2023-10-30T13:10:10.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f79767fc640 1 -- 172.21.15.150:0/431385677 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f7978012840 con 0x7f798015a070 2023-10-30T13:10:10.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.740+0000 7f7987fc7640 1 -- 172.21.15.150:0/431385677 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79801ef210 con 0x7f798015a070 2023-10-30T13:10:10.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.744+0000 7f79767fc640 1 -- 172.21.15.150:0/431385677 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f7978012420 con 0x7f798015a070 2023-10-30T13:10:10.812 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.812+0000 7f7987fc7640 1 -- 172.21.15.150:0/431385677 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "21c3986b-480c-4bdd-a85b-3d51710c7822", "id": 2} v 0) v1 -- 0x7f798004e170 con 0x7f798015a070 2023-10-30T13:10:10.820 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.816+0000 7f79767fc640 1 -- 172.21.15.150:0/431385677 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "21c3986b-480c-4bdd-a85b-3d51710c7822", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7978005560 con 0x7f798015a070 2023-10-30T13:10:10.820 INFO:teuthology.orchestra.run.smithi150.stdout:2 2023-10-30T13:10:10.821 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.820+0000 7f7987fc7640 1 -- 172.21.15.150:0/431385677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f798015a070 msgr2=0x7f79800c73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.821 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.820+0000 7f7987fc7640 1 --2- 172.21.15.150:0/431385677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f798015a070 0x7f79800c73d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f797800c730 tx=0x7f797800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.821 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.820+0000 7f7987fc7640 1 -- 172.21.15.150:0/431385677 shutdown_connections 2023-10-30T13:10:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.820+0000 7f7987fc7640 1 --2- 172.21.15.150:0/431385677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f798015a070 0x7f79800c73d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.820+0000 7f7987fc7640 1 --2- 172.21.15.150:0/431385677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7980150800 0x7f798014e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.820+0000 7f7987fc7640 1 --2- 172.21.15.150:0/431385677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79800c2e20 0x7f7980142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.820+0000 7f7987fc7640 1 -- 172.21.15.150:0/431385677 >> 172.21.15.150:0/431385677 conn(0x7f798006c850 msgr2=0x7f79800cb170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:10.822 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.820+0000 7f7987fc7640 1 -- 172.21.15.150:0/431385677 shutdown_connections 2023-10-30T13:10:10.823 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.820+0000 7f7987fc7640 1 -- 172.21.15.150:0/431385677 wait complete. 2023-10-30T13:10:10.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba 3 2023-10-30T13:10:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.992+0000 7f41ccec7640 1 Processor -- start 2023-10-30T13:10:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.992+0000 7f41ccec7640 1 -- start start 2023-10-30T13:10:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41ccec7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c815a780 0x7f41c80c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41ccec7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c8159be0 con 0x7f41c80c2e20 2023-10-30T13:10:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41ccec7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c8156850 con 0x7f41c8150800 2023-10-30T13:10:10.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41ccec7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80c7020 con 0x7f41c815a780 2023-10-30T13:10:10.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f41c8150800 0x7f41c80cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34510/0 (socket says 172.21.15.150:34510) 2023-10-30T13:10:10.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c659c640 1 -- 172.21.15.150:0/3255504323 learned_addr learned my addr 172.21.15.150:0/3255504323 (peer_addr_for_me v1:172.21.15.150:0/0) 2023-10-30T13:10:10.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c6d9d640 1 --2- 172.21.15.150:0/3255504323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c815a780 0x7f41c80c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c559a640 1 -- 172.21.15.150:0/3255504323 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237420033 0 0) 0x7f41c8156850 con 0x7f41c8150800 2023-10-30T13:10:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c559a640 1 -- 172.21.15.150:0/3255504323 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b0003530 con 0x7f41c8150800 2023-10-30T13:10:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c6d9d640 1 -- 172.21.15.150:0/3255504323 >> v1:172.21.15.150:6789/0 conn(0x7f41c8150800 legacy=0x7f41c80cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c6d9d640 1 -- 172.21.15.150:0/3255504323 >> v1:172.21.15.148:6789/0 conn(0x7f41c80c2e20 legacy=0x7f41c815a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c6d9d640 1 -- 172.21.15.150:0/3255504323 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c8156b30 con 0x7f41c815a780 2023-10-30T13:10:11.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c6d9d640 1 --2- 172.21.15.150:0/3255504323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c815a780 0x7f41c80c6ae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009dc0 tx=0x7f41c0021360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa07671c0feaf4b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c559a640 1 -- 172.21.15.150:0/3255504323 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c00b1600 con 0x7f41c815a780 2023-10-30T13:10:11.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c559a640 1 -- 172.21.15.150:0/3255504323 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c00ad050 con 0x7f41c815a780 2023-10-30T13:10:11.001 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41c559a640 1 -- 172.21.15.150:0/3255504323 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c00b1980 con 0x7f41c815a780 2023-10-30T13:10:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41ccec7640 1 -- 172.21.15.150:0/3255504323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c815a780 msgr2=0x7f41c80c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:10.996+0000 7f41ccec7640 1 --2- 172.21.15.150:0/3255504323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c815a780 0x7f41c80c6ae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009dc0 tx=0x7f41c0021360 comp rx=0 tx=0).stop 2023-10-30T13:10:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- 172.21.15.150:0/3255504323 shutdown_connections 2023-10-30T13:10:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 --2- 172.21.15.150:0/3255504323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c815a780 0x7f41c80c6ae0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- 172.21.15.150:0/3255504323 >> 172.21.15.150:0/3255504323 conn(0x7f41c806c850 msgr2=0x7f41c806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.003 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- 172.21.15.150:0/3255504323 shutdown_connections 2023-10-30T13:10:11.003 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- 172.21.15.150:0/3255504323 wait complete. 2023-10-30T13:10:11.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 Processor -- start 2023-10-30T13:10:11.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- start start 2023-10-30T13:10:11.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c80c2e20 0x7f41c8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.005 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c8150800 0x7f41c8157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41c815a780 0x7f41c8157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c81e0760 con 0x7f41c815a780 2023-10-30T13:10:11.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c814f900 con 0x7f41c80c2e20 2023-10-30T13:10:11.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c8158380 con 0x7f41c8150800 2023-10-30T13:10:11.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c5d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c8150800 0x7f41c8157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.006 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c80c2e20 0x7f41c8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c80c2e20 0x7f41c8069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42390/0 (socket says 172.21.15.150:42390) 2023-10-30T13:10:11.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c659c640 1 -- 172.21.15.150:0/1333421043 learned_addr learned my addr 172.21.15.150:0/1333421043 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:11.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c5d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c8150800 0x7f41c8157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:51916/0 (socket says 172.21.15.150:51916) 2023-10-30T13:10:11.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c6d9d640 1 --2- 172.21.15.150:0/1333421043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41c815a780 0x7f41c8157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.007 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c659c640 1 -- 172.21.15.150:0/1333421043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c8150800 msgr2=0x7f41c8157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.008 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c659c640 1 --2- 172.21.15.150:0/1333421043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c8150800 0x7f41c8157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.008 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c659c640 1 -- 172.21.15.150:0/1333421043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41c815a780 msgr2=0x7f41c8157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.008 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c659c640 1 --2- 172.21.15.150:0/1333421043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41c815a780 0x7f41c8157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.008 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c659c640 1 -- 172.21.15.150:0/1333421043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c0009a70 con 0x7f41c80c2e20 2023-10-30T13:10:11.008 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c5d9b640 1 --2- 172.21.15.150:0/1333421043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c8150800 0x7f41c8157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:11.008 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41c659c640 1 --2- 172.21.15.150:0/1333421043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c80c2e20 0x7f41c8069e90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c4d0 tx=0x7f41b800c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41b6ffd640 1 -- 172.21.15.150:0/1333421043 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8005cf0 con 0x7f41c80c2e20 2023-10-30T13:10:11.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41b6ffd640 1 -- 172.21.15.150:0/1333421043 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b8005e50 con 0x7f41c80c2e20 2023-10-30T13:10:11.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- 172.21.15.150:0/1333421043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c814ea10 con 0x7f41c80c2e20 2023-10-30T13:10:11.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41b6ffd640 1 -- 172.21.15.150:0/1333421043 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8007270 con 0x7f41c80c2e20 2023-10-30T13:10:11.009 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- 172.21.15.150:0/1333421043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c814efd0 con 0x7f41c80c2e20 2023-10-30T13:10:11.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41b6ffd640 1 -- 172.21.15.150:0/1333421043 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f41b801c440 con 0x7f41c80c2e20 2023-10-30T13:10:11.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41b6ffd640 1 -- 172.21.15.150:0/1333421043 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f41b8012890 con 0x7f41c80c2e20 2023-10-30T13:10:11.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.000+0000 7f41ccec7640 1 -- 172.21.15.150:0/1333421043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c804e170 con 0x7f41c80c2e20 2023-10-30T13:10:11.010 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.004+0000 7f41b6ffd640 1 -- 172.21.15.150:0/1333421043 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f41b8021060 con 0x7f41c80c2e20 2023-10-30T13:10:11.073 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.072+0000 7f41ccec7640 1 -- 172.21.15.150:0/1333421043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba", "id": 3} v 0) v1 -- 0x7f41c81570c0 con 0x7f41c80c2e20 2023-10-30T13:10:11.080 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.076+0000 7f41b6ffd640 1 -- 172.21.15.150:0/1333421043 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f41b8017070 con 0x7f41c80c2e20 2023-10-30T13:10:11.080 INFO:teuthology.orchestra.run.smithi150.stdout:3 2023-10-30T13:10:11.081 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.080+0000 7f41ccec7640 1 -- 172.21.15.150:0/1333421043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c80c2e20 msgr2=0x7f41c8069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.081 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.080+0000 7f41ccec7640 1 --2- 172.21.15.150:0/1333421043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c80c2e20 0x7f41c8069e90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c4d0 tx=0x7f41b800c990 comp rx=0 tx=0).stop 2023-10-30T13:10:11.081 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.080+0000 7f41ccec7640 1 -- 172.21.15.150:0/1333421043 shutdown_connections 2023-10-30T13:10:11.082 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.080+0000 7f41ccec7640 1 --2- 172.21.15.150:0/1333421043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41c815a780 0x7f41c8157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.082 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.080+0000 7f41ccec7640 1 --2- 172.21.15.150:0/1333421043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41c8150800 0x7f41c8157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.082 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.080+0000 7f41ccec7640 1 --2- 172.21.15.150:0/1333421043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41c80c2e20 0x7f41c8069e90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.082 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.080+0000 7f41ccec7640 1 -- 172.21.15.150:0/1333421043 >> 172.21.15.150:0/1333421043 conn(0x7f41c806c850 msgr2=0x7f41c80c7170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.082 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.080+0000 7f41ccec7640 1 -- 172.21.15.150:0/1333421043 shutdown_connections 2023-10-30T13:10:11.083 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.080+0000 7f41ccec7640 1 -- 172.21.15.150:0/1333421043 wait complete. 2023-10-30T13:10:11.096 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 8929676f-29ab-4520-acb5-cfe61a0a3063 4 2023-10-30T13:10:11.208 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:11.205+0000 7f0eda53a180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T13:10:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0f2079640 1 Processor -- start 2023-10-30T13:10:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0f2079640 1 -- start start 2023-10-30T13:10:11.211 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0f2079640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0ec151900 0x7ff0ec0c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.211 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0f2079640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff0ec06e570 0x7ff0ec06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.211 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0f2079640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ec06e360 con 0x7ff0ec0c79a0 2023-10-30T13:10:11.211 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0f2079640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ec06eab0 con 0x7ff0ec06e570 2023-10-30T13:10:11.211 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0f2079640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ec0c7480 con 0x7ff0ec151900 2023-10-30T13:10:11.212 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0eb7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0ec151900 0x7ff0ec0c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.212 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0eb7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0ec151900 0x7ff0ec0c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42394/0 (socket says 172.21.15.150:42394) 2023-10-30T13:10:11.212 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0eb7fe640 1 -- 172.21.15.150:0/3790910950 learned_addr learned my addr 172.21.15.150:0/3790910950 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:11.212 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0eaffd640 1 --2- 172.21.15.150:0/3790910950 >> v2:172.21.15.148:3300/0 conn(0x7ff0ec06e570 0x7ff0ec06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.212 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0eb7fe640 1 -- 172.21.15.150:0/3790910950 >> v2:172.21.15.148:3300/0 conn(0x7ff0ec06e570 msgr2=0x7ff0ec06d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:11.213 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0eb7fe640 1 --2- 172.21.15.150:0/3790910950 >> v2:172.21.15.148:3300/0 conn(0x7ff0ec06e570 0x7ff0ec06d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.213 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0eb7fe640 1 -- 172.21.15.150:0/3790910950 >> v1:172.21.15.148:6789/0 conn(0x7ff0ec0c79a0 legacy=0x7ff0ec06f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.213 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.208+0000 7ff0eb7fe640 1 -- 172.21.15.150:0/3790910950 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec06d6f0 con 0x7ff0ec151900 2023-10-30T13:10:11.214 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eb7fe640 1 --2- 172.21.15.150:0/3790910950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec151900 0x7ff0ec0c22f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec070990 tx=0x7ff0dc01c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fd2329c62aa71af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.214 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0ea7fc640 1 -- 172.21.15.150:0/3790910950 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc0aa430 con 0x7ff0ec151900 2023-10-30T13:10:11.214 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0ea7fc640 1 -- 172.21.15.150:0/3790910950 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc0aa590 con 0x7ff0ec151900 2023-10-30T13:10:11.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0ea7fc640 1 -- 172.21.15.150:0/3790910950 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc0aa920 con 0x7ff0ec151900 2023-10-30T13:10:11.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- 172.21.15.150:0/3790910950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec151900 msgr2=0x7ff0ec0c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 --2- 172.21.15.150:0/3790910950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec151900 0x7ff0ec0c22f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec070990 tx=0x7ff0dc01c4d0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- 172.21.15.150:0/3790910950 shutdown_connections 2023-10-30T13:10:11.215 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 --2- 172.21.15.150:0/3790910950 >> v2:172.21.15.148:3300/0 conn(0x7ff0ec06e570 0x7ff0ec06d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.216 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 --2- 172.21.15.150:0/3790910950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec151900 0x7ff0ec0c22f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.216 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- 172.21.15.150:0/3790910950 >> 172.21.15.150:0/3790910950 conn(0x7ff0ec0ccfe0 msgr2=0x7ff0ec0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.216 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- 172.21.15.150:0/3790910950 shutdown_connections 2023-10-30T13:10:11.216 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- 172.21.15.150:0/3790910950 wait complete. 2023-10-30T13:10:11.218 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 Processor -- start 2023-10-30T13:10:11.219 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- start start 2023-10-30T13:10:11.219 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0ec06e570 0x7ff0ec156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.219 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec0c79a0 0x7ff0ec14f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.219 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0ec151900 0x7ff0ec06c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.219 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec1f1f10 con 0x7ff0ec151900 2023-10-30T13:10:11.219 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec1577f0 con 0x7ff0ec0c79a0 2023-10-30T13:10:11.220 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec157ad0 con 0x7ff0ec06e570 2023-10-30T13:10:11.220 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec0c79a0 0x7ff0ec14f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.220 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eb7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0ec06e570 0x7ff0ec156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.220 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec0c79a0 0x7ff0ec14f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42410/0 (socket says 172.21.15.150:42410) 2023-10-30T13:10:11.220 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eaffd640 1 -- 172.21.15.150:0/1498394570 learned_addr learned my addr 172.21.15.150:0/1498394570 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:11.221 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0ebfff640 1 --2- 172.21.15.150:0/1498394570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0ec151900 0x7ff0ec06c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.221 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eb7fe640 1 -- 172.21.15.150:0/1498394570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec0c79a0 msgr2=0x7ff0ec14f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.221 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eb7fe640 1 --2- 172.21.15.150:0/1498394570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec0c79a0 0x7ff0ec14f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.221 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eb7fe640 1 -- 172.21.15.150:0/1498394570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0ec151900 msgr2=0x7ff0ec06c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.221 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eb7fe640 1 --2- 172.21.15.150:0/1498394570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0ec151900 0x7ff0ec06c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.222 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eb7fe640 1 -- 172.21.15.150:0/1498394570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc0094d0 con 0x7ff0ec06e570 2023-10-30T13:10:11.222 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eaffd640 1 --2- 172.21.15.150:0/1498394570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec0c79a0 0x7ff0ec14f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:11.222 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0eb7fe640 1 --2- 172.21.15.150:0/1498394570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0ec06e570 0x7ff0ec156660 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0ab000 tx=0x7ff0dc01ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.222 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0e3fff640 1 -- 172.21.15.150:0/1498394570 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc0aa430 con 0x7ff0ec06e570 2023-10-30T13:10:11.222 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0e3fff640 1 -- 172.21.15.150:0/1498394570 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc0b3820 con 0x7ff0ec06e570 2023-10-30T13:10:11.222 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.212+0000 7ff0f2079640 1 -- 172.21.15.150:0/1498394570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ec1f1850 con 0x7ff0ec06e570 2023-10-30T13:10:11.223 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.216+0000 7ff0e3fff640 1 -- 172.21.15.150:0/1498394570 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc0b3bd0 con 0x7ff0ec06e570 2023-10-30T13:10:11.223 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.216+0000 7ff0f2079640 1 -- 172.21.15.150:0/1498394570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ec1f1500 con 0x7ff0ec06e570 2023-10-30T13:10:11.223 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.216+0000 7ff0e3fff640 1 -- 172.21.15.150:0/1498394570 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff0dc0bd6f0 con 0x7ff0ec06e570 2023-10-30T13:10:11.223 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.216+0000 7ff0e3fff640 1 -- 172.21.15.150:0/1498394570 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7ff0dc0b2890 con 0x7ff0ec06e570 2023-10-30T13:10:11.223 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.216+0000 7ff0f2079640 1 -- 172.21.15.150:0/1498394570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ec1f1690 con 0x7ff0ec06e570 2023-10-30T13:10:11.224 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.216+0000 7ff0e3fff640 1 -- 172.21.15.150:0/1498394570 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ff0dc0b2af0 con 0x7ff0ec06e570 2023-10-30T13:10:11.232 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:11.228+0000 7ff1d1a2c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T13:10:11.270 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:11.265+0000 7f0eda53a180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T13:10:11.286 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:11.284+0000 7ff1d1a2c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T13:10:11.287 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.284+0000 7ff0f2079640 1 -- 172.21.15.150:0/1498394570 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8929676f-29ab-4520-acb5-cfe61a0a3063", "id": 4} v 0) v1 -- 0x7ff0ec04e170 con 0x7ff0ec06e570 2023-10-30T13:10:11.294 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0e3fff640 1 -- 172.21.15.150:0/1498394570 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8929676f-29ab-4520-acb5-cfe61a0a3063", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff0dc0cc0b0 con 0x7ff0ec06e570 2023-10-30T13:10:11.295 INFO:teuthology.orchestra.run.smithi150.stdout:4 2023-10-30T13:10:11.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0f2079640 1 -- 172.21.15.150:0/1498394570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0ec06e570 msgr2=0x7ff0ec156660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0f2079640 1 --2- 172.21.15.150:0/1498394570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0ec06e570 0x7ff0ec156660 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0ab000 tx=0x7ff0dc01ad10 comp rx=0 tx=0).stop 2023-10-30T13:10:11.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0f2079640 1 -- 172.21.15.150:0/1498394570 shutdown_connections 2023-10-30T13:10:11.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0f2079640 1 --2- 172.21.15.150:0/1498394570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0ec151900 0x7ff0ec06c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0f2079640 1 --2- 172.21.15.150:0/1498394570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0ec0c79a0 0x7ff0ec14f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0f2079640 1 --2- 172.21.15.150:0/1498394570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0ec06e570 0x7ff0ec156660 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0f2079640 1 -- 172.21.15.150:0/1498394570 >> 172.21.15.150:0/1498394570 conn(0x7ff0ec0ccfe0 msgr2=0x7ff0ec0c5780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0f2079640 1 -- 172.21.15.150:0/1498394570 shutdown_connections 2023-10-30T13:10:11.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.292+0000 7ff0f2079640 1 -- 172.21.15.150:0/1498394570 wait complete. 2023-10-30T13:10:11.311 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 468a4500-4bc3-4397-a579-372ced468db3 5 2023-10-30T13:10:11.330 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:11.325+0000 7f0eda53a180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T13:10:11.342 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:11.340+0000 7ff1d1a2c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T13:10:11.417 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:11.413+0000 7f0eda53a180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T13:10:11.421 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:11.420+0000 7ff1d1a2c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T13:10:11.424 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.420+0000 7f4733772640 1 Processor -- start 2023-10-30T13:10:11.425 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4733772640 1 -- start start 2023-10-30T13:10:11.426 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4733772640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f472c06a4d0 0x7f472c1515e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.426 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4733772640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f472c06eb10 0x7f472c151b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4733772640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c070e60 con 0x7f472c06a0f0 2023-10-30T13:10:11.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4733772640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c06c360 con 0x7f472c06a4d0 2023-10-30T13:10:11.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4733772640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c06c920 con 0x7f472c06eb10 2023-10-30T13:10:11.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4731d0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f472c06eb10 0x7f472c151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.427 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4731d0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f472c06eb10 0x7f472c151b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42422/0 (socket says 172.21.15.150:42422) 2023-10-30T13:10:11.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4731d0f640 1 -- 172.21.15.150:0/1642178856 learned_addr learned my addr 172.21.15.150:0/1642178856 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:11.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4730d0d640 1 --2- 172.21.15.150:0/1642178856 >> v2:172.21.15.148:3300/0 conn(0x7f472c06a4d0 0x7f472c1515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4731d0f640 1 -- 172.21.15.150:0/1642178856 >> v2:172.21.15.148:3300/0 conn(0x7f472c06a4d0 msgr2=0x7f472c1515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:11.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4731d0f640 1 --2- 172.21.15.150:0/1642178856 >> v2:172.21.15.148:3300/0 conn(0x7f472c06a4d0 0x7f472c1515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.428 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4731d0f640 1 -- 172.21.15.150:0/1642178856 >> v1:172.21.15.148:6789/0 conn(0x7f472c06a0f0 legacy=0x7f472c1577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:11.429 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4731d0f640 1 -- 172.21.15.150:0/1642178856 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c157ff0 con 0x7f472c06eb10 2023-10-30T13:10:11.430 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4731d0f640 1 --2- 172.21.15.150:0/1642178856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 0x7f472c151b20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f472c070600 tx=0x7f472401c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=307db5f75874f515 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.430 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.424+0000 7f4723fff640 1 -- 172.21.15.150:0/1642178856 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47240a5850 con 0x7f472c06eb10 2023-10-30T13:10:11.430 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4723fff640 1 -- 172.21.15.150:0/1642178856 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47240a59b0 con 0x7f472c06eb10 2023-10-30T13:10:11.430 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4723fff640 1 -- 172.21.15.150:0/1642178856 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47240a5d40 con 0x7f472c06eb10 2023-10-30T13:10:11.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- 172.21.15.150:0/1642178856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 msgr2=0x7f472c151b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 --2- 172.21.15.150:0/1642178856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 0x7f472c151b20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f472c070600 tx=0x7f472401c990 comp rx=0 tx=0).stop 2023-10-30T13:10:11.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- 172.21.15.150:0/1642178856 shutdown_connections 2023-10-30T13:10:11.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 --2- 172.21.15.150:0/1642178856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 0x7f472c151b20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.431 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 --2- 172.21.15.150:0/1642178856 >> v2:172.21.15.148:3300/0 conn(0x7f472c06a4d0 0x7f472c1515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.432 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- 172.21.15.150:0/1642178856 >> 172.21.15.150:0/1642178856 conn(0x7f472c06dff0 msgr2=0x7f472c06e400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.432 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- 172.21.15.150:0/1642178856 shutdown_connections 2023-10-30T13:10:11.432 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- 172.21.15.150:0/1642178856 wait complete. 2023-10-30T13:10:11.434 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 Processor -- start 2023-10-30T13:10:11.434 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- start start 2023-10-30T13:10:11.434 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f472c06a0f0 0x7f472c156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f472c06a4d0 0x7f472c1448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 0x7f472c144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c1f04b0 con 0x7f472c06a4d0 2023-10-30T13:10:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c1ef930 con 0x7f472c06eb10 2023-10-30T13:10:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c1efc10 con 0x7f472c06a0f0 2023-10-30T13:10:11.435 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4731d0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 0x7f472c144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.436 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4731d0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 0x7f472c144e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42436/0 (socket says 172.21.15.150:42436) 2023-10-30T13:10:11.436 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4731d0f640 1 -- 172.21.15.150:0/2744682819 learned_addr learned my addr 172.21.15.150:0/2744682819 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:11.436 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f473150e640 1 --2- 172.21.15.150:0/2744682819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f472c06a0f0 0x7f472c156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.436 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4730d0d640 1 --2- 172.21.15.150:0/2744682819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f472c06a4d0 0x7f472c1448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.436 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4731d0f640 1 -- 172.21.15.150:0/2744682819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f472c06a0f0 msgr2=0x7f472c156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4731d0f640 1 --2- 172.21.15.150:0/2744682819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f472c06a0f0 0x7f472c156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4731d0f640 1 -- 172.21.15.150:0/2744682819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f472c06a4d0 msgr2=0x7f472c1448f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4731d0f640 1 --2- 172.21.15.150:0/2744682819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f472c06a4d0 0x7f472c1448f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4731d0f640 1 -- 172.21.15.150:0/2744682819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724009b30 con 0x7f472c06eb10 2023-10-30T13:10:11.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4731d0f640 1 --2- 172.21.15.150:0/2744682819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 0x7f472c144e30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f472401a5c0 tx=0x7f4724019730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.437 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4721ffb640 1 -- 172.21.15.150:0/2744682819 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47240b8670 con 0x7f472c06eb10 2023-10-30T13:10:11.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4721ffb640 1 -- 172.21.15.150:0/2744682819 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47240b87f0 con 0x7f472c06eb10 2023-10-30T13:10:11.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- 172.21.15.150:0/2744682819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c1efef0 con 0x7f472c06eb10 2023-10-30T13:10:11.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4721ffb640 1 -- 172.21.15.150:0/2744682819 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47240b8ba0 con 0x7f472c06eb10 2023-10-30T13:10:11.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4733772640 1 -- 172.21.15.150:0/2744682819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c145c80 con 0x7f472c06eb10 2023-10-30T13:10:11.438 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4721ffb640 1 -- 172.21.15.150:0/2744682819 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f47240c3460 con 0x7f472c06eb10 2023-10-30T13:10:11.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.428+0000 7f4721ffb640 1 -- 172.21.15.150:0/2744682819 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f47240b6a10 con 0x7f472c06eb10 2023-10-30T13:10:11.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.432+0000 7f4733772640 1 -- 172.21.15.150:0/2744682819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c145ea0 con 0x7f472c06eb10 2023-10-30T13:10:11.439 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.432+0000 7f4721ffb640 1 -- 172.21.15.150:0/2744682819 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f47240b6cf0 con 0x7f472c06eb10 2023-10-30T13:10:11.475 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:11.472+0000 7ff1d1a2c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T13:10:11.478 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:11.473+0000 7f0eda53a180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T13:10:11.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.500+0000 7f4733772640 1 -- 172.21.15.150:0/2744682819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "468a4500-4bc3-4397-a579-372ced468db3", "id": 5} v 0) v1 -- 0x7f472c145ea0 con 0x7f472c06eb10 2023-10-30T13:10:11.509 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4721ffb640 1 -- 172.21.15.150:0/2744682819 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "468a4500-4bc3-4397-a579-372ced468db3", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f47240b6490 con 0x7f472c06eb10 2023-10-30T13:10:11.509 INFO:teuthology.orchestra.run.smithi150.stdout:5 2023-10-30T13:10:11.510 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4733772640 1 -- 172.21.15.150:0/2744682819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 msgr2=0x7f472c144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.511 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4733772640 1 --2- 172.21.15.150:0/2744682819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 0x7f472c144e30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f472401a5c0 tx=0x7f4724019730 comp rx=0 tx=0).stop 2023-10-30T13:10:11.511 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4733772640 1 -- 172.21.15.150:0/2744682819 shutdown_connections 2023-10-30T13:10:11.511 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4733772640 1 --2- 172.21.15.150:0/2744682819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f472c06a4d0 0x7f472c1448f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.511 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4733772640 1 --2- 172.21.15.150:0/2744682819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f472c06eb10 0x7f472c144e30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.511 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4733772640 1 --2- 172.21.15.150:0/2744682819 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f472c06a0f0 0x7f472c156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4733772640 1 -- 172.21.15.150:0/2744682819 >> 172.21.15.150:0/2744682819 conn(0x7f472c06dff0 msgr2=0x7f472c14e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4733772640 1 -- 172.21.15.150:0/2744682819 shutdown_connections 2023-10-30T13:10:11.512 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.508+0000 7f4733772640 1 -- 172.21.15.150:0/2744682819 wait complete. 2023-10-30T13:10:11.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 752c1053-247f-4e90-8054-5ace15b90f2b 6 2023-10-30T13:10:11.562 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:11.560+0000 7ff1d1a2c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T13:10:11.566 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:11.565+0000 7f0eda53a180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T13:10:11.616 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:11.612+0000 7ff1d1a2c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T13:10:11.626 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:11.625+0000 7f0eda53a180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T13:10:11.671 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.668+0000 7f2ab45bc640 1 Processor -- start 2023-10-30T13:10:11.671 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.668+0000 7f2ab45bc640 1 -- start start 2023-10-30T13:10:11.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.668+0000 7f2ab45bc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2aac15d290 0x7f2aac15d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.668+0000 7f2ab45bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac0c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.668+0000 7f2ab45bc640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac070150 con 0x7f2aac0c22e0 2023-10-30T13:10:11.673 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.668+0000 7f2ab45bc640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac06fbd0 con 0x7f2aac15d290 2023-10-30T13:10:11.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.668+0000 7f2ab45bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac06fe70 con 0x7f2aac15dbb0 2023-10-30T13:10:11.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1b57640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac0c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1b57640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac0c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:51938/0 (socket says 172.21.15.150:51938) 2023-10-30T13:10:11.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1b57640 1 -- 172.21.15.150:0/117481503 learned_addr learned my addr 172.21.15.150:0/117481503 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:11.674 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab2358640 1 --2- 172.21.15.150:0/117481503 >> v2:172.21.15.148:3300/0 conn(0x7f2aac15d290 0x7f2aac15d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1b57640 1 -- 172.21.15.150:0/117481503 >> v2:172.21.15.148:3300/0 conn(0x7f2aac15d290 msgr2=0x7f2aac15d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:11.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1b57640 1 --2- 172.21.15.150:0/117481503 >> v2:172.21.15.148:3300/0 conn(0x7f2aac15d290 0x7f2aac15d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1b57640 1 -- 172.21.15.150:0/117481503 >> v1:172.21.15.148:6789/0 conn(0x7f2aac0c22e0 legacy=0x7f2aac0c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.675 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1b57640 1 -- 172.21.15.150:0/117481503 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac0c2e20 con 0x7f2aac15dbb0 2023-10-30T13:10:11.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1b57640 1 --2- 172.21.15.150:0/117481503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac0c3900 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009870 tx=0x7f2a9c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c67a51a5f74009a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1356640 1 -- 172.21.15.150:0/117481503 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c0a8a30 con 0x7f2aac15dbb0 2023-10-30T13:10:11.676 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1356640 1 -- 172.21.15.150:0/117481503 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c0a8b90 con 0x7f2aac15dbb0 2023-10-30T13:10:11.677 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab1356640 1 -- 172.21.15.150:0/117481503 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c0b0440 con 0x7f2aac15dbb0 2023-10-30T13:10:11.677 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab45bc640 1 -- 172.21.15.150:0/117481503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 msgr2=0x7f2aac0c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.677 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab45bc640 1 --2- 172.21.15.150:0/117481503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac0c3900 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009870 tx=0x7f2a9c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:10:11.678 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab45bc640 1 -- 172.21.15.150:0/117481503 shutdown_connections 2023-10-30T13:10:11.678 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab45bc640 1 --2- 172.21.15.150:0/117481503 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac0c3900 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.678 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab45bc640 1 --2- 172.21.15.150:0/117481503 >> v2:172.21.15.148:3300/0 conn(0x7f2aac15d290 0x7f2aac15d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.678 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.672+0000 7f2ab45bc640 1 -- 172.21.15.150:0/117481503 >> 172.21.15.150:0/117481503 conn(0x7f2aac0ccfe0 msgr2=0x7f2aac0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.678 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 -- 172.21.15.150:0/117481503 shutdown_connections 2023-10-30T13:10:11.679 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 -- 172.21.15.150:0/117481503 wait complete. 2023-10-30T13:10:11.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 Processor -- start 2023-10-30T13:10:11.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 -- start start 2023-10-30T13:10:11.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2aac0c22e0 0x7f2aac14fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.681 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aac15d290 0x7f2aac1f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac150300 con 0x7f2aac0c22e0 2023-10-30T13:10:11.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac1f2eb0 con 0x7f2aac15d290 2023-10-30T13:10:11.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac06ded0 con 0x7f2aac15dbb0 2023-10-30T13:10:11.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab1b57640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aac15d290 0x7f2aac1f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.682 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab2b59640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.683 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab1b57640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aac15d290 0x7f2aac1f2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42444/0 (socket says 172.21.15.150:42444) 2023-10-30T13:10:11.683 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab1b57640 1 -- 172.21.15.150:0/366503391 learned_addr learned my addr 172.21.15.150:0/366503391 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:11.683 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab2358640 1 --2- 172.21.15.150:0/366503391 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2aac0c22e0 0x7f2aac14fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.683 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab2b59640 1 -- 172.21.15.150:0/366503391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aac15d290 msgr2=0x7f2aac1f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.683 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab2b59640 1 --2- 172.21.15.150:0/366503391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aac15d290 0x7f2aac1f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.684 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab2b59640 1 -- 172.21.15.150:0/366503391 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2aac0c22e0 msgr2=0x7f2aac14fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.684 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab2b59640 1 --2- 172.21.15.150:0/366503391 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2aac0c22e0 0x7f2aac14fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.684 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab2b59640 1 -- 172.21.15.150:0/366503391 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0094d0 con 0x7f2aac15dbb0 2023-10-30T13:10:11.684 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab1b57640 1 --2- 172.21.15.150:0/366503391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aac15d290 0x7f2aac1f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:11.684 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab2b59640 1 --2- 172.21.15.150:0/366503391 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac06bb40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400c700 tx=0x7f2aa400cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.684 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2aa2ffd640 1 -- 172.21.15.150:0/366503391 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4003b80 con 0x7f2aac15dbb0 2023-10-30T13:10:11.685 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2aa2ffd640 1 -- 172.21.15.150:0/366503391 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa4003ce0 con 0x7f2aac15dbb0 2023-10-30T13:10:11.685 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 -- 172.21.15.150:0/366503391 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac159710 con 0x7f2aac15dbb0 2023-10-30T13:10:11.685 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2aa2ffd640 1 -- 172.21.15.150:0/366503391 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa400aa30 con 0x7f2aac15dbb0 2023-10-30T13:10:11.685 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 -- 172.21.15.150:0/366503391 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac06b430 con 0x7f2aac15dbb0 2023-10-30T13:10:11.685 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2aa2ffd640 1 -- 172.21.15.150:0/366503391 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2aa40050d0 con 0x7f2aac15dbb0 2023-10-30T13:10:11.685 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2aa2ffd640 1 -- 172.21.15.150:0/366503391 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f2aa4012a10 con 0x7f2aac15dbb0 2023-10-30T13:10:11.686 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.676+0000 7f2ab45bc640 1 -- 172.21.15.150:0/366503391 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac06b5c0 con 0x7f2aac15dbb0 2023-10-30T13:10:11.686 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.680+0000 7f2aa2ffd640 1 -- 172.21.15.150:0/366503391 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f2aa4012cb0 con 0x7f2aac15dbb0 2023-10-30T13:10:11.745 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.744+0000 7f2ab45bc640 1 -- 172.21.15.150:0/366503391 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd new", "uuid": "752c1053-247f-4e90-8054-5ace15b90f2b", "id": 6} v 0) v1 -- 0x7f2aac0c3100 con 0x7f2aac15dbb0 2023-10-30T13:10:11.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2aa2ffd640 1 -- 172.21.15.150:0/366503391 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "752c1053-247f-4e90-8054-5ace15b90f2b", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2aa4012e50 con 0x7f2aac15dbb0 2023-10-30T13:10:11.753 INFO:teuthology.orchestra.run.smithi150.stdout:6 2023-10-30T13:10:11.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2ab45bc640 1 -- 172.21.15.150:0/366503391 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 msgr2=0x7f2aac06bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.754 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2ab45bc640 1 --2- 172.21.15.150:0/366503391 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac06bb40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400c700 tx=0x7f2aa400cbc0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2ab45bc640 1 -- 172.21.15.150:0/366503391 shutdown_connections 2023-10-30T13:10:11.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2ab45bc640 1 --2- 172.21.15.150:0/366503391 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2aac15dbb0 0x7f2aac06bb40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2ab45bc640 1 --2- 172.21.15.150:0/366503391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2aac15d290 0x7f2aac1f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2ab45bc640 1 --2- 172.21.15.150:0/366503391 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2aac0c22e0 0x7f2aac14fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2ab45bc640 1 -- 172.21.15.150:0/366503391 >> 172.21.15.150:0/366503391 conn(0x7f2aac0ccfe0 msgr2=0x7f2aac06ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2ab45bc640 1 -- 172.21.15.150:0/366503391 shutdown_connections 2023-10-30T13:10:11.756 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.752+0000 7f2ab45bc640 1 -- 172.21.15.150:0/366503391 wait complete. 2023-10-30T13:10:11.761 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:11.760+0000 7ff1d1a2c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T13:10:11.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 1151f789-6221-4eae-9398-c4a4ac83e170 7 2023-10-30T13:10:11.783 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:11.781+0000 7f0eda53a180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T13:10:11.824 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:11.820+0000 7ff1d1a2c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T13:10:11.850 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:11.845+0000 7f0eda53a180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T13:10:11.881 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3ea27e4640 1 Processor -- start 2023-10-30T13:10:11.882 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3ea27e4640 1 -- start start 2023-10-30T13:10:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3ea27e4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e9c151900 0x7f3e9c0c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3ea27e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c06e570 0x7f3e9c06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3ea27e4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3e9c0c79a0 0x7f3e9c06f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3ea27e4640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9c06eab0 con 0x7f3e9c0c79a0 2023-10-30T13:10:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3ea27e4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9c06e360 con 0x7f3e9c151900 2023-10-30T13:10:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3ea27e4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c0c7480 con 0x7f3e9c06e570 2023-10-30T13:10:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3e9b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c06e570 0x7f3e9c06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3e9bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e9c151900 0x7f3e9c0c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3e9b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c06e570 0x7f3e9c06d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.150:51956/0 (socket says 172.21.15.150:51956) 2023-10-30T13:10:11.885 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3e9b7fe640 1 -- 172.21.15.150:0/1186135469 learned_addr learned my addr 172.21.15.150:0/1186135469 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:11.885 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3ea0d81640 1 --2- 172.21.15.150:0/1186135469 >> v2:172.21.15.148:3300/0 conn(0x7f3e9c0c79a0 0x7f3e9c06f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.885 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3e9b7fe640 1 -- 172.21.15.150:0/1186135469 >> v2:172.21.15.150:3300/0 conn(0x7f3e9c151900 msgr2=0x7f3e9c0c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:11.885 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3e9b7fe640 1 --2- 172.21.15.150:0/1186135469 >> v2:172.21.15.150:3300/0 conn(0x7f3e9c151900 0x7f3e9c0c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.885 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3e9b7fe640 1 -- 172.21.15.150:0/1186135469 >> v2:172.21.15.148:3300/0 conn(0x7f3e9c0c79a0 msgr2=0x7f3e9c06f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:11.886 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3e9b7fe640 1 --2- 172.21.15.150:0/1186135469 >> v2:172.21.15.148:3300/0 conn(0x7f3e9c0c79a0 0x7f3e9c06f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.886 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.880+0000 7f3e9b7fe640 1 -- 172.21.15.150:0/1186135469 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c06d6f0 con 0x7f3e9c06e570 2023-10-30T13:10:11.889 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3e9b7fe640 1 --2- 172.21.15.150:0/1186135469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c06e570 0x7f3e9c06d1b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c070990 tx=0x7f3e9001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d773a2e673df300 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.889 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3e9affd640 1 -- 172.21.15.150:0/1186135469 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900ac600 con 0x7f3e9c06e570 2023-10-30T13:10:11.889 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3e9affd640 1 -- 172.21.15.150:0/1186135469 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e900a8050 con 0x7f3e9c06e570 2023-10-30T13:10:11.890 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3e9affd640 1 -- 172.21.15.150:0/1186135469 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900ac980 con 0x7f3e9c06e570 2023-10-30T13:10:11.890 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 -- 172.21.15.150:0/1186135469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c06e570 msgr2=0x7f3e9c06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.890 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 --2- 172.21.15.150:0/1186135469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c06e570 0x7f3e9c06d1b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c070990 tx=0x7f3e9001c4d0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.890 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 -- 172.21.15.150:0/1186135469 shutdown_connections 2023-10-30T13:10:11.890 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 --2- 172.21.15.150:0/1186135469 >> v2:172.21.15.148:3300/0 conn(0x7f3e9c0c79a0 0x7f3e9c06f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.891 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 --2- 172.21.15.150:0/1186135469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c06e570 0x7f3e9c06d1b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.891 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 --2- 172.21.15.150:0/1186135469 >> v2:172.21.15.150:3300/0 conn(0x7f3e9c151900 0x7f3e9c0c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.891 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 -- 172.21.15.150:0/1186135469 >> 172.21.15.150:0/1186135469 conn(0x7f3e9c0ccfe0 msgr2=0x7f3e9c0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.891 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 -- 172.21.15.150:0/1186135469 shutdown_connections 2023-10-30T13:10:11.891 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 -- 172.21.15.150:0/1186135469 wait complete. 2023-10-30T13:10:11.892 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 Processor -- start 2023-10-30T13:10:11.892 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 -- start start 2023-10-30T13:10:11.892 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e9c06e570 0x7f3e9c1f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.892 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c0c79a0 0x7f3e9c1617e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.892 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e9c151900 0x7f3e9c15b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:11.892 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c161d20 con 0x7f3e9c06e570 2023-10-30T13:10:11.893 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c154590 con 0x7f3e9c151900 2023-10-30T13:10:11.893 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea27e4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c1520d0 con 0x7f3e9c0c79a0 2023-10-30T13:10:11.893 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea0d81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e9c151900 0x7f3e9c15b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.893 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3e9b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c0c79a0 0x7f3e9c1617e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.893 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea0d81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e9c151900 0x7f3e9c15b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42470/0 (socket says 172.21.15.150:42470) 2023-10-30T13:10:11.893 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea0d81640 1 -- 172.21.15.150:0/950637836 learned_addr learned my addr 172.21.15.150:0/950637836 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-30T13:10:11.894 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3e9bfff640 1 --2- 172.21.15.150:0/950637836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e9c06e570 0x7f3e9c1f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:11.894 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea0d81640 1 -- 172.21.15.150:0/950637836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c0c79a0 msgr2=0x7f3e9c1617e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.894 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea0d81640 1 --2- 172.21.15.150:0/950637836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c0c79a0 0x7f3e9c1617e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.894 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea0d81640 1 -- 172.21.15.150:0/950637836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e9c06e570 msgr2=0x7f3e9c1f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.894 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea0d81640 1 --2- 172.21.15.150:0/950637836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e9c06e570 0x7f3e9c1f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.895 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea0d81640 1 -- 172.21.15.150:0/950637836 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e900094d0 con 0x7f3e9c151900 2023-10-30T13:10:11.895 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3ea0d81640 1 --2- 172.21.15.150:0/950637836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e9c151900 0x7f3e9c15b4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400cd20 tx=0x7f3e9400a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:11.895 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.884+0000 7f3e98ff9640 1 -- 172.21.15.150:0/950637836 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e94003ba0 con 0x7f3e9c151900 2023-10-30T13:10:11.895 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.888+0000 7f3e98ff9640 1 -- 172.21.15.150:0/950637836 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e94003d00 con 0x7f3e9c151900 2023-10-30T13:10:11.895 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.888+0000 7f3e98ff9640 1 -- 172.21.15.150:0/950637836 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e94004240 con 0x7f3e9c151900 2023-10-30T13:10:11.895 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.888+0000 7f3ea27e4640 1 -- 172.21.15.150:0/950637836 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c154850 con 0x7f3e9c151900 2023-10-30T13:10:11.896 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.888+0000 7f3ea27e4640 1 -- 172.21.15.150:0/950637836 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c154e10 con 0x7f3e9c151900 2023-10-30T13:10:11.896 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.888+0000 7f3e98ff9640 1 -- 172.21.15.150:0/950637836 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3e940043a0 con 0x7f3e9c151900 2023-10-30T13:10:11.896 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.888+0000 7f3e98ff9640 1 -- 172.21.15.150:0/950637836 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f3e94012070 con 0x7f3e9c151900 2023-10-30T13:10:11.897 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.888+0000 7f3ea27e4640 1 -- 172.21.15.150:0/950637836 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9c04e170 con 0x7f3e9c151900 2023-10-30T13:10:11.897 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.888+0000 7f3e98ff9640 1 -- 172.21.15.150:0/950637836 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f3e94003e70 con 0x7f3e9c151900 2023-10-30T13:10:11.954 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.952+0000 7f3ea27e4640 1 -- 172.21.15.150:0/950637836 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1151f789-6221-4eae-9398-c4a4ac83e170", "id": 7} v 0) v1 -- 0x7f3e9c1527e0 con 0x7f3e9c151900 2023-10-30T13:10:11.962 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3e98ff9640 1 -- 172.21.15.150:0/950637836 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1151f789-6221-4eae-9398-c4a4ac83e170", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3e94004830 con 0x7f3e9c151900 2023-10-30T13:10:11.963 INFO:teuthology.orchestra.run.smithi150.stdout:7 2023-10-30T13:10:11.964 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3ea27e4640 1 -- 172.21.15.150:0/950637836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e9c151900 msgr2=0x7f3e9c15b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:11.964 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3ea27e4640 1 --2- 172.21.15.150:0/950637836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e9c151900 0x7f3e9c15b4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400cd20 tx=0x7f3e9400a660 comp rx=0 tx=0).stop 2023-10-30T13:10:11.964 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3ea27e4640 1 -- 172.21.15.150:0/950637836 shutdown_connections 2023-10-30T13:10:11.965 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3ea27e4640 1 --2- 172.21.15.150:0/950637836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e9c151900 0x7f3e9c15b4c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.965 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3ea27e4640 1 --2- 172.21.15.150:0/950637836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e9c0c79a0 0x7f3e9c1617e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.965 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3ea27e4640 1 --2- 172.21.15.150:0/950637836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e9c06e570 0x7f3e9c1f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:11.965 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3ea27e4640 1 -- 172.21.15.150:0/950637836 >> 172.21.15.150:0/950637836 conn(0x7f3e9c0ccfe0 msgr2=0x7f3e9c0c4880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:11.965 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3ea27e4640 1 -- 172.21.15.150:0/950637836 shutdown_connections 2023-10-30T13:10:11.965 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-30T13:10:11.960+0000 7f3ea27e4640 1 -- 172.21.15.150:0/950637836 wait complete. 2023-10-30T13:10:11.978 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T13:10:11.978 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-30T13:10:11.982 INFO:tasks.ceph.osd.0:Started 2023-10-30T13:10:11.982 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T13:10:11.983 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-30T13:10:11.986 INFO:tasks.ceph.osd.1:Started 2023-10-30T13:10:11.986 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T13:10:11.987 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-30T13:10:11.990 INFO:tasks.ceph.osd.2:Started 2023-10-30T13:10:11.990 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T13:10:11.991 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-30T13:10:11.994 INFO:tasks.ceph.osd.3:Started 2023-10-30T13:10:11.995 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T13:10:11.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-30T13:10:11.997 INFO:tasks.ceph.osd.4:Started 2023-10-30T13:10:11.998 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T13:10:11.998 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-30T13:10:12.000 INFO:tasks.ceph.osd.5:Started 2023-10-30T13:10:12.002 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T13:10:12.002 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-30T13:10:12.004 INFO:tasks.ceph.osd.6:Started 2023-10-30T13:10:12.004 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T13:10:12.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-30T13:10:12.006 INFO:tasks.ceph.osd.7:Started 2023-10-30T13:10:12.007 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:12.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.185+0000 7f6698f62640 1 Processor -- start 2023-10-30T13:10:12.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.185+0000 7f6698f62640 1 -- start start 2023-10-30T13:10:12.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f6698f62640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66941538f0 0x7f6694159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:12.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f6698f62640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f669415cd50 0x7f66940c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:12.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f6698f62640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:12.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f6698f62640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669415d300 con 0x7f669415cd50 2023-10-30T13:10:12.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f6698f62640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669415d5e0 con 0x7f66941538f0 2023-10-30T13:10:12.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f6698f62640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669415d8c0 con 0x7f66940cefa0 2023-10-30T13:10:12.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f66937fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66941538f0 0x7f6694159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:12.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f66937fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66941538f0 0x7f6694159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:49958/0 (socket says 172.21.15.148:49958) 2023-10-30T13:10:12.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f66937fe640 1 -- 172.21.15.148:0/3397745255 learned_addr learned my addr 172.21.15.148:0/3397745255 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:12.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f668affd640 1 --2- 172.21.15.148:0/3397745255 >> v2:172.21.15.148:3300/0 conn(0x7f669415cd50 0x7f66940c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:12.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f6693fff640 1 --2- 172.21.15.148:0/3397745255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:12.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f66937fe640 1 -- 172.21.15.148:0/3397745255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 msgr2=0x7f66940cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:12.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f66937fe640 1 --2- 172.21.15.148:0/3397745255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f66937fe640 1 -- 172.21.15.148:0/3397745255 >> v2:172.21.15.148:3300/0 conn(0x7f669415cd50 msgr2=0x7f66940c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:12.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f66937fe640 1 --2- 172.21.15.148:0/3397745255 >> v2:172.21.15.148:3300/0 conn(0x7f669415cd50 0x7f66940c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f66937fe640 1 -- 172.21.15.148:0/3397745255 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669415dba0 con 0x7f66941538f0 2023-10-30T13:10:12.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.189+0000 7f668affd640 1 --2- 172.21.15.148:0/3397745255 >> v2:172.21.15.148:3300/0 conn(0x7f669415cd50 0x7f66940c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:12.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f66937fe640 1 --2- 172.21.15.148:0/3397745255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66941538f0 0x7f6694159fb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f66940c9cb0 tx=0x7f668401f1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=757ab87cc1e30be2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:12.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6692ffd640 1 -- 172.21.15.148:0/3397745255 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66840af400 con 0x7f66941538f0 2023-10-30T13:10:12.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6692ffd640 1 -- 172.21.15.148:0/3397745255 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66840ab050 con 0x7f66941538f0 2023-10-30T13:10:12.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6692ffd640 1 -- 172.21.15.148:0/3397745255 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66840af780 con 0x7f66941538f0 2023-10-30T13:10:12.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6698f62640 1 -- 172.21.15.148:0/3397745255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66941538f0 msgr2=0x7f6694159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:12.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6698f62640 1 --2- 172.21.15.148:0/3397745255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66941538f0 0x7f6694159fb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f66940c9cb0 tx=0x7f668401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6698f62640 1 -- 172.21.15.148:0/3397745255 shutdown_connections 2023-10-30T13:10:12.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6698f62640 1 --2- 172.21.15.148:0/3397745255 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6698f62640 1 --2- 172.21.15.148:0/3397745255 >> v2:172.21.15.148:3300/0 conn(0x7f669415cd50 0x7f66940c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6698f62640 1 --2- 172.21.15.148:0/3397745255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66941538f0 0x7f6694159fb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6698f62640 1 -- 172.21.15.148:0/3397745255 >> 172.21.15.148:0/3397745255 conn(0x7f669406a640 msgr2=0x7f6694065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:12.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6698f62640 1 -- 172.21.15.148:0/3397745255 shutdown_connections 2023-10-30T13:10:12.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.197+0000 7f6698f62640 1 -- 172.21.15.148:0/3397745255 wait complete. 2023-10-30T13:10:12.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.201+0000 7f6698f62640 1 Processor -- start 2023-10-30T13:10:12.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.201+0000 7f6698f62640 1 -- start start 2023-10-30T13:10:12.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f6698f62640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:12.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f6698f62640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66941538f0 0x7f66940b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:12.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f6698f62640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f669415cd50 0x7f669415b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:12.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f66937fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:12.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f66937fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940b3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:42456/0 (socket says 172.21.15.148:42456) 2023-10-30T13:10:12.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f6698f62640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66940c43d0 con 0x7f66941538f0 2023-10-30T13:10:12.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f66937fe640 1 -- 172.21.15.148:0/2851617410 learned_addr learned my addr 172.21.15.148:0/2851617410 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:12.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f6693fff640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f669415cd50 0x7f669415b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:12.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66940b44a0 con 0x7f669415cd50 2023-10-30T13:10:12.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669415a9c0 con 0x7f66940cefa0 2023-10-30T13:10:12.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.205+0000 7f668affd640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66941538f0 0x7f66940b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:12.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f66937fe640 1 -- 172.21.15.148:0/2851617410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f669415cd50 msgr2=0x7f669415b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f66937fe640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f669415cd50 0x7f669415b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f66937fe640 1 -- 172.21.15.148:0/2851617410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66941538f0 msgr2=0x7f66940b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f66937fe640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66941538f0 0x7f66940b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f66937fe640 1 -- 172.21.15.148:0/2851617410 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6684009b30 con 0x7f66940cefa0 2023-10-30T13:10:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f668affd640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66941538f0 0x7f66940b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:12.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f66937fe640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940b3610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f668401cdf0 tx=0x7f668401bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:12.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f6690ff9640 1 -- 172.21.15.148:0/2851617410 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66840afd50 con 0x7f66940cefa0 2023-10-30T13:10:12.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f6690ff9640 1 -- 172.21.15.148:0/2851617410 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66840b9410 con 0x7f66940cefa0 2023-10-30T13:10:12.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66940b41c0 con 0x7f66940cefa0 2023-10-30T13:10:12.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669415af60 con 0x7f66940cefa0 2023-10-30T13:10:12.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f6690ff9640 1 -- 172.21.15.148:0/2851617410 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66840b97c0 con 0x7f66940cefa0 2023-10-30T13:10:12.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f6690ff9640 1 -- 172.21.15.148:0/2851617410 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f66840b6070 con 0x7f66940cefa0 2023-10-30T13:10:12.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.209+0000 7f6690ff9640 1 -- 172.21.15.148:0/2851617410 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f66840b9c20 con 0x7f66940cefa0 2023-10-30T13:10:12.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.213+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6678005260 con 0x7f66940cefa0 2023-10-30T13:10:12.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.213+0000 7f6690ff9640 1 -- 172.21.15.148:0/2851617410 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f66840b4070 con 0x7f66940cefa0 2023-10-30T13:10:12.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.285+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66780057e0 con 0x7f66940cefa0 2023-10-30T13:10:12.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.285+0000 7f6690ff9640 1 -- 172.21.15.148:0/2851617410 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f66840cd430 con 0x7f66940cefa0 2023-10-30T13:10:12.290 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:12.290 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":10,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:11.955160+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:12.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.289+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 msgr2=0x7f66940b3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:12.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.289+0000 7f6698f62640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940b3610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f668401cdf0 tx=0x7f668401bf60 comp rx=0 tx=0).stop 2023-10-30T13:10:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.289+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 shutdown_connections 2023-10-30T13:10:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.289+0000 7f6698f62640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66941538f0 0x7f66940b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.289+0000 7f6698f62640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f669415cd50 0x7f669415b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.289+0000 7f6698f62640 1 --2- 172.21.15.148:0/2851617410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66940cefa0 0x7f66940b3610 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:12.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.289+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 >> 172.21.15.148:0/2851617410 conn(0x7f669406a640 msgr2=0x7f66940cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:12.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.289+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 shutdown_connections 2023-10-30T13:10:12.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:12.289+0000 7f6698f62640 1 -- 172.21.15.148:0/2851617410 wait complete. 2023-10-30T13:10:12.309 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T13:10:12.310 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T13:10:12.401 INFO:tasks.ceph.mgr.x.smithi150.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T13:10:12.402 INFO:tasks.ceph.mgr.x.smithi150.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T13:10:12.402 INFO:tasks.ceph.mgr.x.smithi150.stderr: from numpy import show_config as show_numpy_config 2023-10-30T13:10:12.441 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:12.440+0000 7ff1d1a2c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T13:10:12.487 INFO:tasks.ceph.mgr.y.smithi148.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T13:10:12.488 INFO:tasks.ceph.mgr.y.smithi148.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T13:10:12.488 INFO:tasks.ceph.mgr.y.smithi148.stderr: from numpy import show_config as show_numpy_config 2023-10-30T13:10:12.535 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:12.533+0000 7f0eda53a180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T13:10:12.574 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:12.572+0000 7ff1d1a2c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:10:12.614 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T13:10:12.612+0000 7f3ba9946640 -1 Falling back to public interface 2023-10-30T13:10:12.614 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T13:10:12.612+0000 7fbec3f87640 -1 Falling back to public interface 2023-10-30T13:10:12.615 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T13:10:12.612+0000 7f11a36a5640 -1 Falling back to public interface 2023-10-30T13:10:12.615 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T13:10:12.612+0000 7f11a2666640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2023-10-30T13:10:12.629 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T13:10:12.628+0000 7fcbd4d34640 -1 Falling back to public interface 2023-10-30T13:10:12.640 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T13:10:12.637+0000 7fafc204a640 -1 Falling back to public interface 2023-10-30T13:10:12.640 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T13:10:12.637+0000 7f1af9dab640 -1 Falling back to public interface 2023-10-30T13:10:12.641 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T13:10:12.637+0000 7f1af8d6c640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-10-30T13:10:12.644 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T13:10:12.641+0000 7f11ad4c1640 -1 Falling back to public interface 2023-10-30T13:10:12.667 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T13:10:12.665+0000 7fb61528e640 -1 Falling back to public interface 2023-10-30T13:10:12.676 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:12.673+0000 7f0eda53a180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:10:12.761 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:12.760+0000 7ff1d1a2c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T13:10:12.830 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:12.828+0000 7ff1d1a2c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T13:10:12.849 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:12.845+0000 7f0eda53a180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T13:10:12.921 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:12.917+0000 7f0eda53a180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T13:10:12.955 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:12.952+0000 7ff1d1a2c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:10:13.059 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:13.057+0000 7f0eda53a180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T13:10:13.113 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:13.241 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.237+0000 7efc51e79640 1 Processor -- start 2023-10-30T13:10:13.242 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.237+0000 7efc51e79640 1 -- start start 2023-10-30T13:10:13.243 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc51e79640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efc4c1539f0 0x7efc4c0c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:13.243 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc51e79640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc4c0cf080 0x7efc4c0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:13.244 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc51e79640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c0c5390 con 0x7efc4c0c9f90 2023-10-30T13:10:13.244 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc51e79640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4c0ca980 con 0x7efc4c1539f0 2023-10-30T13:10:13.244 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc51e79640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4c15a240 con 0x7efc4c0cf080 2023-10-30T13:10:13.244 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc4b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efc4c1539f0 0x7efc4c0c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:13.244 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc4bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc4c0cf080 0x7efc4c0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:13.245 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc4b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efc4c1539f0 0x7efc4c0c5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57052/0 (socket says 172.21.15.148:57052) 2023-10-30T13:10:13.245 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc4b7fe640 1 -- 172.21.15.148:0/840939998 learned_addr learned my addr 172.21.15.148:0/840939998 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:13.245 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc4b7fe640 1 -- 172.21.15.148:0/840939998 >> v2:172.21.15.150:3300/0 conn(0x7efc4c0cf080 msgr2=0x7efc4c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:13.245 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc4b7fe640 1 --2- 172.21.15.148:0/840939998 >> v2:172.21.15.150:3300/0 conn(0x7efc4c0cf080 0x7efc4c0cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:13.245 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc4b7fe640 1 -- 172.21.15.148:0/840939998 >> v1:172.21.15.150:6789/0 conn(0x7efc4c0c9f90 legacy=0x7efc4c159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:13.246 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc4b7fe640 1 -- 172.21.15.148:0/840939998 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c15a520 con 0x7efc4c1539f0 2023-10-30T13:10:13.246 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.241+0000 7efc4b7fe640 1 --2- 172.21.15.148:0/840939998 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc4c1539f0 0x7efc4c0c5820 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0c6410 tx=0x7efc3c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374708cfd40bb820 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:13.247 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc4a7fc640 1 -- 172.21.15.148:0/840939998 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0ab5f0 con 0x7efc4c1539f0 2023-10-30T13:10:13.247 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc4a7fc640 1 -- 172.21.15.148:0/840939998 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c0ab750 con 0x7efc4c1539f0 2023-10-30T13:10:13.247 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc4a7fc640 1 -- 172.21.15.148:0/840939998 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0abae0 con 0x7efc4c1539f0 2023-10-30T13:10:13.248 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 -- 172.21.15.148:0/840939998 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc4c1539f0 msgr2=0x7efc4c0c5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:13.248 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 --2- 172.21.15.148:0/840939998 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc4c1539f0 0x7efc4c0c5820 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0c6410 tx=0x7efc3c01c500 comp rx=0 tx=0).stop 2023-10-30T13:10:13.248 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 -- 172.21.15.148:0/840939998 shutdown_connections 2023-10-30T13:10:13.249 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 --2- 172.21.15.148:0/840939998 >> v2:172.21.15.150:3300/0 conn(0x7efc4c0cf080 0x7efc4c0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:13.249 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 --2- 172.21.15.148:0/840939998 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc4c1539f0 0x7efc4c0c5820 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:13.249 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 -- 172.21.15.148:0/840939998 >> 172.21.15.148:0/840939998 conn(0x7efc4c065bf0 msgr2=0x7efc4c066000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:13.249 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 -- 172.21.15.148:0/840939998 shutdown_connections 2023-10-30T13:10:13.249 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 -- 172.21.15.148:0/840939998 wait complete. 2023-10-30T13:10:13.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 Processor -- start 2023-10-30T13:10:13.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 -- start start 2023-10-30T13:10:13.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc4c0c9f90 0x7efc4c158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:13.252 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc4c0cf080 0x7efc4c13c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:13.253 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc4c1539f0 0x7efc4c13cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:13.253 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c1529e0 con 0x7efc4c0cf080 2023-10-30T13:10:13.253 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c1585e0 con 0x7efc4c1539f0 2023-10-30T13:10:13.253 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc51e79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c145e70 con 0x7efc4c0c9f90 2023-10-30T13:10:13.253 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc4b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc4c0c9f90 0x7efc4c158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:13.254 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc4b7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc4c0c9f90 0x7efc4c158a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:42466/0 (socket says 172.21.15.148:42466) 2023-10-30T13:10:13.254 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc4b7fe640 1 -- 172.21.15.148:0/2936150646 learned_addr learned my addr 172.21.15.148:0/2936150646 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:13.254 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc4bfff640 1 --2- 172.21.15.148:0/2936150646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc4c1539f0 0x7efc4c13cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:13.254 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.245+0000 7efc4affd640 1 --2- 172.21.15.148:0/2936150646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc4c0cf080 0x7efc4c13c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:13.254 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc4b7fe640 1 -- 172.21.15.148:0/2936150646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc4c1539f0 msgr2=0x7efc4c13cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:13.255 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc4b7fe640 1 --2- 172.21.15.148:0/2936150646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc4c1539f0 0x7efc4c13cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:13.255 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc4b7fe640 1 -- 172.21.15.148:0/2936150646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc4c0cf080 msgr2=0x7efc4c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:13.255 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc4b7fe640 1 --2- 172.21.15.148:0/2936150646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc4c0cf080 0x7efc4c13c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:13.255 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc4b7fe640 1 -- 172.21.15.148:0/2936150646 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c0094d0 con 0x7efc4c0c9f90 2023-10-30T13:10:13.255 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc4b7fe640 1 --2- 172.21.15.148:0/2936150646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc4c0c9f90 0x7efc4c158a70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efc3c019110 tx=0x7efc3c01ad40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:13.255 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc33fff640 1 -- 172.21.15.148:0/2936150646 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0be5f0 con 0x7efc4c0c9f90 2023-10-30T13:10:13.256 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc33fff640 1 -- 172.21.15.148:0/2936150646 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c0be770 con 0x7efc4c0c9f90 2023-10-30T13:10:13.256 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc51e79640 1 -- 172.21.15.148:0/2936150646 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4c1458b0 con 0x7efc4c0c9f90 2023-10-30T13:10:13.256 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc33fff640 1 -- 172.21.15.148:0/2936150646 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0beb20 con 0x7efc4c0c9f90 2023-10-30T13:10:13.256 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc51e79640 1 -- 172.21.15.148:0/2936150646 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4c157f90 con 0x7efc4c0c9f90 2023-10-30T13:10:13.256 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc33fff640 1 -- 172.21.15.148:0/2936150646 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efc3c0b43f0 con 0x7efc4c0c9f90 2023-10-30T13:10:13.257 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc33fff640 1 -- 172.21.15.148:0/2936150646 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7efc3c0a8050 con 0x7efc4c0c9f90 2023-10-30T13:10:13.257 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.249+0000 7efc51e79640 1 -- 172.21.15.148:0/2936150646 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4c04e170 con 0x7efc4c0c9f90 2023-10-30T13:10:13.257 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.253+0000 7efc33fff640 1 -- 172.21.15.148:0/2936150646 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7efc3c0b46a0 con 0x7efc4c0c9f90 2023-10-30T13:10:13.311 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 -- 172.21.15.148:0/2936150646 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc4c0caad0 con 0x7efc4c0c9f90 2023-10-30T13:10:13.312 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc33fff640 1 -- 172.21.15.148:0/2936150646 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7efc3c0b4840 con 0x7efc4c0c9f90 2023-10-30T13:10:13.313 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T13:10:13.313 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":10,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:11.955160+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:13.314 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 -- 172.21.15.148:0/2936150646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc4c0c9f90 msgr2=0x7efc4c158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:13.314 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 --2- 172.21.15.148:0/2936150646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc4c0c9f90 0x7efc4c158a70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efc3c019110 tx=0x7efc3c01ad40 comp rx=0 tx=0).stop 2023-10-30T13:10:13.314 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 -- 172.21.15.148:0/2936150646 shutdown_connections 2023-10-30T13:10:13.314 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 --2- 172.21.15.148:0/2936150646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc4c0cf080 0x7efc4c13c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:13.315 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 --2- 172.21.15.148:0/2936150646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc4c1539f0 0x7efc4c13cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:13.315 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 --2- 172.21.15.148:0/2936150646 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc4c0c9f90 0x7efc4c158a70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:13.315 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 -- 172.21.15.148:0/2936150646 >> 172.21.15.148:0/2936150646 conn(0x7efc4c065bf0 msgr2=0x7efc4c065fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:13.315 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 -- 172.21.15.148:0/2936150646 shutdown_connections 2023-10-30T13:10:13.315 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:13.309+0000 7efc51e79640 1 -- 172.21.15.148:0/2936150646 wait complete. 2023-10-30T13:10:13.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T13:10:13.559 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:13.556+0000 7ff1d1a2c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T13:10:13.635 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:13.632+0000 7ff1d1a2c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T13:10:13.739 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:13.737+0000 7f0eda53a180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T13:10:13.804 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:13.800+0000 7ff1d1a2c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T13:10:13.825 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:13.821+0000 7f0eda53a180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T13:10:13.877 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:13.876+0000 7ff1d1a2c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T13:10:14.015 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:14.013+0000 7f0eda53a180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T13:10:14.099 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:14.097+0000 7f0eda53a180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T13:10:14.163 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:14.160+0000 7ff1d1a2c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T13:10:14.222 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:14.220+0000 7ff1d1a2c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T13:10:14.433 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:14.429+0000 7f0eda53a180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T13:10:14.501 INFO:tasks.ceph.mgr.y.smithi148.stderr:2023-10-30T13:10:14.497+0000 7f0eda53a180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T13:10:15.266 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:15.264+0000 7ff1792a2640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T13:10:15.267 INFO:tasks.ceph.mgr.x.smithi150.stderr:2023-10-30T13:10:15.264+0000 7ff1792a2640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T13:10:16.447 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T13:10:16.444+0000 7fbec3f87640 -1 osd.6 0 log_to_monitors true 2023-10-30T13:10:16.531 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T13:10:16.529+0000 7f1af9dab640 -1 osd.2 0 log_to_monitors true 2023-10-30T13:10:16.537 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T13:10:16.536+0000 7f3ba9946640 -1 osd.4 0 log_to_monitors true 2023-10-30T13:10:16.548 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T13:10:16.545+0000 7fafc204a640 -1 osd.0 0 log_to_monitors true 2023-10-30T13:10:16.612 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T13:10:16.608+0000 7fcbd4d34640 -1 osd.7 0 log_to_monitors true 2023-10-30T13:10:16.624 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T13:10:16.624+0000 7f11a36a5640 -1 osd.5 0 log_to_monitors true 2023-10-30T13:10:16.677 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T13:10:16.673+0000 7f11ad4c1640 -1 osd.1 0 log_to_monitors true 2023-10-30T13:10:16.688 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T13:10:16.685+0000 7fb61528e640 -1 osd.3 0 log_to_monitors true 2023-10-30T13:10:20.137 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:20.337 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.333+0000 7f8dc71d3640 1 Processor -- start 2023-10-30T13:10:20.338 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.333+0000 7f8dc71d3640 1 -- start start 2023-10-30T13:10:20.339 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc71d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc0157e80 0x7f8dc00cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:20.339 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc71d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8dc015e5d0 0x7f8dc00cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:20.339 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc71d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc0158260 con 0x7f8dc006b4d0 2023-10-30T13:10:20.340 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc71d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc00ca560 con 0x7f8dc015e5d0 2023-10-30T13:10:20.340 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc71d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00ca840 con 0x7f8dc0157e80 2023-10-30T13:10:20.340 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc4f6f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc0157e80 0x7f8dc00cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:20.340 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc4f6f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc0157e80 0x7f8dc00cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:42512/0 (socket says 172.21.15.148:42512) 2023-10-30T13:10:20.341 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc5770640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8dc006b4d0 0x7f8dc006a170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:51130/0 (socket says 172.21.15.148:51130) 2023-10-30T13:10:20.341 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc4f6f640 1 -- 172.21.15.148:0/1188770829 learned_addr learned my addr 172.21.15.148:0/1188770829 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:20.341 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc4f6f640 1 -- 172.21.15.148:0/1188770829 >> v2:172.21.15.148:3300/0 conn(0x7f8dc015e5d0 msgr2=0x7f8dc00cbf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:10:20.341 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc4f6f640 1 --2- 172.21.15.148:0/1188770829 >> v2:172.21.15.148:3300/0 conn(0x7f8dc015e5d0 0x7f8dc00cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:20.341 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc4f6f640 1 -- 172.21.15.148:0/1188770829 >> v1:172.21.15.150:6789/0 conn(0x7f8dc006b4d0 legacy=0x7f8dc006a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:20.342 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.337+0000 7f8dc4f6f640 1 -- 172.21.15.148:0/1188770829 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc00cab20 con 0x7f8dc0157e80 2023-10-30T13:10:20.345 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.341+0000 7f8dc4f6f640 1 --2- 172.21.15.148:0/1188770829 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc0157e80 0x7f8dc00cba00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009870 tx=0x7f8db401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3176c1239129b4cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:20.346 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.341+0000 7f8dbf7fe640 1 -- 172.21.15.148:0/1188770829 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40a9400 con 0x7f8dc0157e80 2023-10-30T13:10:20.346 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.341+0000 7f8dbf7fe640 1 -- 172.21.15.148:0/1188770829 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db40a9560 con 0x7f8dc0157e80 2023-10-30T13:10:20.346 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.341+0000 7f8dbf7fe640 1 -- 172.21.15.148:0/1188770829 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40a98f0 con 0x7f8dc0157e80 2023-10-30T13:10:20.347 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.345+0000 7f8dc71d3640 1 -- 172.21.15.148:0/1188770829 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc0157e80 msgr2=0x7f8dc00cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:20.347 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.345+0000 7f8dc71d3640 1 --2- 172.21.15.148:0/1188770829 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc0157e80 0x7f8dc00cba00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009870 tx=0x7f8db401c540 comp rx=0 tx=0).stop 2023-10-30T13:10:20.348 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.345+0000 7f8dc71d3640 1 -- 172.21.15.148:0/1188770829 shutdown_connections 2023-10-30T13:10:20.348 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.345+0000 7f8dc71d3640 1 --2- 172.21.15.148:0/1188770829 >> v2:172.21.15.148:3300/0 conn(0x7f8dc015e5d0 0x7f8dc00cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:20.348 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.345+0000 7f8dc71d3640 1 --2- 172.21.15.148:0/1188770829 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc0157e80 0x7f8dc00cba00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:20.348 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.345+0000 7f8dc71d3640 1 -- 172.21.15.148:0/1188770829 >> 172.21.15.148:0/1188770829 conn(0x7f8dc00ceff0 msgr2=0x7f8dc00cf400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:20.349 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.345+0000 7f8dc71d3640 1 -- 172.21.15.148:0/1188770829 shutdown_connections 2023-10-30T13:10:20.349 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.345+0000 7f8dc71d3640 1 -- 172.21.15.148:0/1188770829 wait complete. 2023-10-30T13:10:20.350 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.345+0000 7f8dc71d3640 1 Processor -- start 2023-10-30T13:10:20.351 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.349+0000 7f8dc71d3640 1 -- start start 2023-10-30T13:10:20.353 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.349+0000 7f8dc71d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dc006b4d0 0x7f8dc014c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:20.355 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.349+0000 7f8dc71d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc0157e80 0x7f8dc01f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:20.355 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.349+0000 7f8dc71d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc015e5d0 0x7f8dc015d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:20.355 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.349+0000 7f8dc4f6f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dc006b4d0 0x7f8dc014c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:20.355 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.349+0000 7f8dc4f6f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dc006b4d0 0x7f8dc014c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57088/0 (socket says 172.21.15.148:57088) 2023-10-30T13:10:20.355 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc4f6f640 1 -- 172.21.15.148:0/3483357223 learned_addr learned my addr 172.21.15.148:0/3483357223 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:20.356 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dbffff640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc0157e80 0x7f8dc01f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:20.356 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc71d3640 1 -- 172.21.15.148:0/3483357223 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc015a200 con 0x7f8dc006b4d0 2023-10-30T13:10:20.356 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc71d3640 1 -- 172.21.15.148:0/3483357223 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc0157020 con 0x7f8dc0157e80 2023-10-30T13:10:20.356 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc71d3640 1 -- 172.21.15.148:0/3483357223 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc015ac60 con 0x7f8dc015e5d0 2023-10-30T13:10:20.356 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc5770640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc015e5d0 0x7f8dc015d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:20.357 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc4f6f640 1 -- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc015e5d0 msgr2=0x7f8dc015d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:20.358 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc4f6f640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc015e5d0 0x7f8dc015d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:20.358 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc4f6f640 1 -- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc0157e80 msgr2=0x7f8dc01f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:20.358 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc4f6f640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc0157e80 0x7f8dc01f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:20.358 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc4f6f640 1 -- 172.21.15.148:0/3483357223 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db40094d0 con 0x7f8dc006b4d0 2023-10-30T13:10:20.358 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dbffff640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc0157e80 0x7f8dc01f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:20.359 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc4f6f640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dc006b4d0 0x7f8dc014c500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8db401abf0 tx=0x7f8db401ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:20.359 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dbd7fa640 1 -- 172.21.15.148:0/3483357223 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40b1800 con 0x7f8dc006b4d0 2023-10-30T13:10:20.359 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc71d3640 1 -- 172.21.15.148:0/3483357223 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc015af40 con 0x7f8dc006b4d0 2023-10-30T13:10:20.359 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dc71d3640 1 -- 172.21.15.148:0/3483357223 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc01569d0 con 0x7f8dc006b4d0 2023-10-30T13:10:20.360 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dbd7fa640 1 -- 172.21.15.148:0/3483357223 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db40b1980 con 0x7f8dc006b4d0 2023-10-30T13:10:20.360 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.353+0000 7f8dbd7fa640 1 -- 172.21.15.148:0/3483357223 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40b1d30 con 0x7f8dc006b4d0 2023-10-30T13:10:20.360 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.357+0000 7f8daaffd640 1 -- 172.21.15.148:0/3483357223 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da0005260 con 0x7f8dc006b4d0 2023-10-30T13:10:20.361 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.357+0000 7f8dbd7fa640 1 -- 172.21.15.148:0/3483357223 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 4) v1 ==== 85794+0+0 (secure 0 0 0) 0x7f8db40bc450 con 0x7f8dc006b4d0 2023-10-30T13:10:20.366 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.357+0000 7f8dbd7fa640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8da4066510 0x7f8da40689d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:20.366 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.357+0000 7f8dbd7fa640 1 -- 172.21.15.148:0/3483357223 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f8db411d9b0 con 0x7f8dc006b4d0 2023-10-30T13:10:20.367 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.361+0000 7f8dbffff640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8da4066510 0x7f8da40689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:20.367 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.361+0000 7f8dbffff640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8da4066510 0x7f8da40689d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00cd930 tx=0x7f8dac009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:20.372 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.369+0000 7f8dbd7fa640 1 -- 172.21.15.148:0/3483357223 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8db40ebd30 con 0x7f8dc006b4d0 2023-10-30T13:10:20.479 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T13:10:20.476+0000 7f3ba50cb640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T13:10:20.502 INFO:tasks.ceph.osd.4.smithi150.stderr:2023-10-30T13:10:20.500+0000 7f3b9d712640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:10:20.576 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T13:10:20.572+0000 7fcbd04b9640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T13:10:20.601 INFO:tasks.ceph.osd.7.smithi150.stderr:2023-10-30T13:10:20.600+0000 7fcbc8b00640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:10:20.669 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T13:10:20.668+0000 7fbec0f44640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T13:10:20.672 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.669+0000 7f8daaffd640 1 -- 172.21.15.148:0/3483357223 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8da0005510 con 0x7f8dc006b4d0 2023-10-30T13:10:20.674 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.669+0000 7f8dbd7fa640 1 -- 172.21.15.148:0/3483357223 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f8db40a6020 con 0x7f8dc006b4d0 2023-10-30T13:10:20.678 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T13:10:20.679 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":12,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:18.269354+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:20.682 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T13:10:20.680+0000 7f119ee2a640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T13:10:20.682 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 -- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8da4066510 msgr2=0x7f8da40689d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:20.684 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8da4066510 0x7f8da40689d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00cd930 tx=0x7f8dac009110 comp rx=0 tx=0).stop 2023-10-30T13:10:20.684 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 -- 172.21.15.148:0/3483357223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dc006b4d0 msgr2=0x7f8dc014c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:20.684 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dc006b4d0 0x7f8dc014c500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8db401abf0 tx=0x7f8db401ad80 comp rx=0 tx=0).stop 2023-10-30T13:10:20.685 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 -- 172.21.15.148:0/3483357223 shutdown_connections 2023-10-30T13:10:20.685 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8da4066510 0x7f8da40689d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:20.685 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8dc015e5d0 0x7f8dc015d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:20.685 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dc0157e80 0x7f8dc01f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:20.685 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 --2- 172.21.15.148:0/3483357223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8dc006b4d0 0x7f8dc014c500 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:20.685 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 -- 172.21.15.148:0/3483357223 >> 172.21.15.148:0/3483357223 conn(0x7f8dc00ceff0 msgr2=0x7f8dc00c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:20.686 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 -- 172.21.15.148:0/3483357223 shutdown_connections 2023-10-30T13:10:20.686 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:20.681+0000 7f8daaffd640 1 -- 172.21.15.148:0/3483357223 wait complete. 2023-10-30T13:10:20.689 INFO:tasks.ceph.osd.6.smithi150.stderr:2023-10-30T13:10:20.688+0000 7fbeb7d53640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:10:20.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T13:10:20.721 INFO:tasks.ceph.osd.5.smithi150.stderr:2023-10-30T13:10:20.720+0000 7f1197471640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:10:20.777 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T13:10:20.773+0000 7fb610a15640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T13:10:20.782 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T13:10:20.777+0000 7fafbd7cf640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T13:10:20.791 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T13:10:20.789+0000 7f1af5530640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T13:10:20.794 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T13:10:20.789+0000 7f11a8c46640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T13:10:20.812 INFO:tasks.ceph.osd.3.smithi148.stderr:2023-10-30T13:10:20.809+0000 7fb60905c640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:10:20.828 INFO:tasks.ceph.osd.0.smithi148.stderr:2023-10-30T13:10:20.825+0000 7fafb5e16640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:10:20.828 INFO:tasks.ceph.osd.1.smithi148.stderr:2023-10-30T13:10:20.825+0000 7f11a128d640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:10:20.843 INFO:tasks.ceph.osd.2.smithi148.stderr:2023-10-30T13:10:20.841+0000 7f1aedb77640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T13:10:22.602 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:10:22.606 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:10:27.525 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:27.664 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cf90e640 1 Processor -- start 2023-10-30T13:10:27.665 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cf90e640 1 -- start start 2023-10-30T13:10:27.666 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cf90e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb5c81515d0 0x7fb5c80c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:27.667 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cf90e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb5c80708e0 0x7fb5c8157c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:27.667 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cf90e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c8070070 con 0x7fb5c806fb80 2023-10-30T13:10:27.667 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cf90e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c8070350 con 0x7fb5c80708e0 2023-10-30T13:10:27.667 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cf90e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c80b0a10 con 0x7fb5c81515d0 2023-10-30T13:10:27.667 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cdeab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb5c80708e0 0x7fb5c8157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:27.668 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cdeab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb5c80708e0 0x7fb5c8157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40964/0 (socket says 172.21.15.148:40964) 2023-10-30T13:10:27.668 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.660+0000 7fb5cdeab640 1 -- 172.21.15.148:0/2786313362 learned_addr learned my addr 172.21.15.148:0/2786313362 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:27.668 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5cd6aa640 1 --2- 172.21.15.148:0/2786313362 >> v2:172.21.15.150:3300/0 conn(0x7fb5c81515d0 0x7fb5c80c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:27.668 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5cdeab640 1 -- 172.21.15.148:0/2786313362 >> v2:172.21.15.150:3300/0 conn(0x7fb5c81515d0 msgr2=0x7fb5c80c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:27.668 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5cdeab640 1 --2- 172.21.15.148:0/2786313362 >> v2:172.21.15.150:3300/0 conn(0x7fb5c81515d0 0x7fb5c80c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.668 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5cdeab640 1 -- 172.21.15.148:0/2786313362 >> v1:172.21.15.150:6789/0 conn(0x7fb5c806fb80 legacy=0x7fb5c80c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:27.669 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5cdeab640 1 -- 172.21.15.148:0/2786313362 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c80b0cf0 con 0x7fb5c80708e0 2023-10-30T13:10:27.669 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5cdeab640 1 --2- 172.21.15.148:0/2786313362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c80708e0 0x7fb5c8157c90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb5c806c4c0 tx=0x7fb5c001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f010bbb974e9f881 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:27.670 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5bffff640 1 -- 172.21.15.148:0/2786313362 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00af400 con 0x7fb5c80708e0 2023-10-30T13:10:27.671 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5bffff640 1 -- 172.21.15.148:0/2786313362 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c00ab050 con 0x7fb5c80708e0 2023-10-30T13:10:27.671 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5bffff640 1 -- 172.21.15.148:0/2786313362 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00af780 con 0x7fb5c80708e0 2023-10-30T13:10:27.671 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2786313362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c80708e0 msgr2=0x7fb5c8157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:27.671 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.664+0000 7fb5cf90e640 1 --2- 172.21.15.148:0/2786313362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c80708e0 0x7fb5c8157c90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb5c806c4c0 tx=0x7fb5c001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.671 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2786313362 shutdown_connections 2023-10-30T13:10:27.672 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 --2- 172.21.15.148:0/2786313362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c80708e0 0x7fb5c8157c90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.672 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 --2- 172.21.15.148:0/2786313362 >> v2:172.21.15.150:3300/0 conn(0x7fb5c81515d0 0x7fb5c80c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.672 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2786313362 >> 172.21.15.148:0/2786313362 conn(0x7fb5c80cb0b0 msgr2=0x7fb5c80cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:27.672 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2786313362 shutdown_connections 2023-10-30T13:10:27.672 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2786313362 wait complete. 2023-10-30T13:10:27.673 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 Processor -- start 2023-10-30T13:10:27.673 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 -- start start 2023-10-30T13:10:27.675 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c806fb80 0x7fb5c8152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:27.675 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c80708e0 0x7fb5c80bee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:27.675 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c81515d0 0x7fb5c814ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:27.675 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c8151f80 con 0x7fb5c81515d0 2023-10-30T13:10:27.676 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c80aeab0 con 0x7fb5c806fb80 2023-10-30T13:10:27.676 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cf90e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c81ef7f0 con 0x7fb5c80708e0 2023-10-30T13:10:27.676 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cdeab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c81515d0 0x7fb5c814ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:27.676 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cdeab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c81515d0 0x7fb5c814ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40976/0 (socket says 172.21.15.148:40976) 2023-10-30T13:10:27.676 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cdeab640 1 -- 172.21.15.148:0/2355154603 learned_addr learned my addr 172.21.15.148:0/2355154603 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:27.677 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cd6aa640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c806fb80 0x7fb5c8152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:27.677 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5ccea9640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c80708e0 0x7fb5c80bee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:27.677 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cdeab640 1 -- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c80708e0 msgr2=0x7fb5c80bee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:27.677 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cdeab640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c80708e0 0x7fb5c80bee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.677 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cdeab640 1 -- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c806fb80 msgr2=0x7fb5c8152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:27.678 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cdeab640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c806fb80 0x7fb5c8152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.678 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.668+0000 7fb5cdeab640 1 -- 172.21.15.148:0/2355154603 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c0009b30 con 0x7fb5c81515d0 2023-10-30T13:10:27.678 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5cdeab640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c81515d0 0x7fb5c814ca00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb5c001bdd0 tx=0x7fb5c001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:27.678 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5bdffb640 1 -- 172.21.15.148:0/2355154603 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00bf070 con 0x7fb5c81515d0 2023-10-30T13:10:27.678 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5bdffb640 1 -- 172.21.15.148:0/2355154603 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c0009510 con 0x7fb5c81515d0 2023-10-30T13:10:27.678 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c81efad0 con 0x7fb5c81515d0 2023-10-30T13:10:27.679 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5bdffb640 1 -- 172.21.15.148:0/2355154603 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00afe40 con 0x7fb5c81515d0 2023-10-30T13:10:27.679 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c80af050 con 0x7fb5c81515d0 2023-10-30T13:10:27.680 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c80af240 con 0x7fb5c81515d0 2023-10-30T13:10:27.680 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5bdffb640 1 -- 172.21.15.148:0/2355154603 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5c00b6070 con 0x7fb5c81515d0 2023-10-30T13:10:27.680 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5bdffb640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5ac0664a0 0x7fb5ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:27.680 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.672+0000 7fb5bdffb640 1 -- 172.21.15.148:0/2355154603 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb5c0121300 con 0x7fb5c81515d0 2023-10-30T13:10:27.680 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.676+0000 7fb5cd6aa640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5ac0664a0 0x7fb5ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:27.681 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.676+0000 7fb5cd6aa640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5ac0664a0 0x7fb5ac068960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4007a60 tx=0x7fb5c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:27.682 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.680+0000 7fb5bdffb640 1 -- 172.21.15.148:0/2355154603 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5c00eead0 con 0x7fb5c81515d0 2023-10-30T13:10:27.832 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.828+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5c804e170 con 0x7fb5c81515d0 2023-10-30T13:10:27.833 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.828+0000 7fb5bdffb640 1 -- 172.21.15.148:0/2355154603 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12886 (secure 0 0 0) 0x7fb5c00f20f0 con 0x7fb5c81515d0 2023-10-30T13:10:27.833 INFO:teuthology.misc.health.smithi148.stdout: 2023-10-30T13:10:27.833 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":16,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:25.308810+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:27.835 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5ac0664a0 msgr2=0x7fb5ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:27.835 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5ac0664a0 0x7fb5ac068960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4007a60 tx=0x7fb5c4008040 comp rx=0 tx=0).stop 2023-10-30T13:10:27.836 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c81515d0 msgr2=0x7fb5c814ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:27.836 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c81515d0 0x7fb5c814ca00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb5c001bdd0 tx=0x7fb5c001bf60 comp rx=0 tx=0).stop 2023-10-30T13:10:27.836 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 shutdown_connections 2023-10-30T13:10:27.836 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5ac0664a0 0x7fb5ac068960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.836 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5c81515d0 0x7fb5c814ca00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb5c80708e0 0x7fb5c80bee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 --2- 172.21.15.148:0/2355154603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5c806fb80 0x7fb5c8152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:27.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 >> 172.21.15.148:0/2355154603 conn(0x7fb5c80cb0b0 msgr2=0x7fb5c80c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:27.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 shutdown_connections 2023-10-30T13:10:27.837 INFO:teuthology.misc.health.smithi148.stderr:2023-10-30T13:10:27.832+0000 7fb5cf90e640 1 -- 172.21.15.148:0/2355154603 wait complete. 2023-10-30T13:10:27.852 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T13:10:27.853 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T13:10:27.853 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:10:27.854 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:10:27.857 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:10:27.858 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:10:27.858 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:10:27.862 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:10:27.862 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:10:27.862 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:10:27.865 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:10:27.866 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T13:10:27.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T13:10:27.869 INFO:tasks.ceph.mds.b:Started 2023-10-30T13:10:27.869 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:10:27.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:10:27.872 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:10:27.872 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T13:10:27.872 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T13:10:27.875 INFO:tasks.ceph.mds.f:Started 2023-10-30T13:10:27.875 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-30T13:10:27.875 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:27.955 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2023-10-30T13:10:27.956 INFO:tasks.ceph.mds.f.smithi150.stdout:starting mds.f at 2023-10-30T13:10:27.956 INFO:tasks.ceph.mds.b.smithi150.stdout:starting mds.b at 2023-10-30T13:10:27.999 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:10:28.000 INFO:tasks.ceph.mds.e.smithi148.stdout:starting mds.e at 2023-10-30T13:10:28.000 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:10:28.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.024+0000 7f8e8aadc640 1 Processor -- start 2023-10-30T13:10:28.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.024+0000 7f8e8aadc640 1 -- start start 2023-10-30T13:10:28.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e8aadc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e841538f0 0x7f8e84159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e8aadc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8e8415cd50 0x7f8e840c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e8aadc640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8415d300 con 0x7f8e840cefa0 2023-10-30T13:10:28.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e8aadc640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8415d5e0 con 0x7f8e8415cd50 2023-10-30T13:10:28.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e8aadc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8415d8c0 con 0x7f8e841538f0 2023-10-30T13:10:28.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e892d9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8e8415cd50 0x7f8e840c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e8a2db640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8e840cefa0 0x7f8e840cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43956/0 (socket says 172.21.15.148:43956) 2023-10-30T13:10:28.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e8a2db640 1 -- 172.21.15.148:0/448408908 learned_addr learned my addr 172.21.15.148:0/448408908 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:28.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e89ada640 1 --2- 172.21.15.148:0/448408908 >> v2:172.21.15.150:3300/0 conn(0x7f8e841538f0 0x7f8e84159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e88ad8640 1 -- 172.21.15.148:0/448408908 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449093786 0 0) 0x7f8e8415d300 con 0x7f8e840cefa0 2023-10-30T13:10:28.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e88ad8640 1 -- 172.21.15.148:0/448408908 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e70003530 con 0x7f8e840cefa0 2023-10-30T13:10:28.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e892d9640 1 -- 172.21.15.148:0/448408908 >> v2:172.21.15.150:3300/0 conn(0x7f8e841538f0 msgr2=0x7f8e84159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:28.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e892d9640 1 --2- 172.21.15.148:0/448408908 >> v2:172.21.15.150:3300/0 conn(0x7f8e841538f0 0x7f8e84159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e892d9640 1 -- 172.21.15.148:0/448408908 >> v1:172.21.15.148:6789/0 conn(0x7f8e840cefa0 legacy=0x7f8e840cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e892d9640 1 -- 172.21.15.148:0/448408908 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8415dba0 con 0x7f8e8415cd50 2023-10-30T13:10:28.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.028+0000 7f8e89ada640 1 --2- 172.21.15.148:0/448408908 >> v2:172.21.15.150:3300/0 conn(0x7f8e841538f0 0x7f8e84159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:28.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e892d9640 1 --2- 172.21.15.148:0/448408908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 0x7f8e840c9f90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8e840c9cb0 tx=0x7f8e7401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25795749ecbec321 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:28.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e88ad8640 1 -- 172.21.15.148:0/448408908 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e740a8950 con 0x7f8e8415cd50 2023-10-30T13:10:28.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e88ad8640 1 -- 172.21.15.148:0/448408908 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e740a8ab0 con 0x7f8e8415cd50 2023-10-30T13:10:28.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e88ad8640 1 -- 172.21.15.148:0/448408908 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e740a8e40 con 0x7f8e8415cd50 2023-10-30T13:10:28.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 -- 172.21.15.148:0/448408908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 msgr2=0x7f8e840c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 --2- 172.21.15.148:0/448408908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 0x7f8e840c9f90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8e840c9cb0 tx=0x7f8e7401ec50 comp rx=0 tx=0).stop 2023-10-30T13:10:28.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 -- 172.21.15.148:0/448408908 shutdown_connections 2023-10-30T13:10:28.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 --2- 172.21.15.148:0/448408908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 0x7f8e840c9f90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 --2- 172.21.15.148:0/448408908 >> v2:172.21.15.150:3300/0 conn(0x7f8e841538f0 0x7f8e84159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 -- 172.21.15.148:0/448408908 >> 172.21.15.148:0/448408908 conn(0x7f8e8406a640 msgr2=0x7f8e84065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:28.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 -- 172.21.15.148:0/448408908 shutdown_connections 2023-10-30T13:10:28.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 -- 172.21.15.148:0/448408908 wait complete. 2023-10-30T13:10:28.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 Processor -- start 2023-10-30T13:10:28.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.032+0000 7f8e8aadc640 1 -- start start 2023-10-30T13:10:28.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8aadc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e840cefa0 0x7f8e84158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8aadc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e841538f0 0x7f8e84152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8aadc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 0x7f8e840b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8aadc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8415c1c0 con 0x7f8e8415cd50 2023-10-30T13:10:28.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8aadc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8415aac0 con 0x7f8e840cefa0 2023-10-30T13:10:28.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8aadc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8415ada0 con 0x7f8e841538f0 2023-10-30T13:10:28.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8a2db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 0x7f8e840b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e89ada640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e840cefa0 0x7f8e84158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e892d9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e841538f0 0x7f8e84152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e89ada640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e840cefa0 0x7f8e84158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:55060/0 (socket says 172.21.15.148:55060) 2023-10-30T13:10:28.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e89ada640 1 -- 172.21.15.148:0/1345261930 learned_addr learned my addr 172.21.15.148:0/1345261930 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:28.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8a2db640 1 -- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e841538f0 msgr2=0x7f8e84152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8a2db640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e841538f0 0x7f8e84152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8a2db640 1 -- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e840cefa0 msgr2=0x7f8e84158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8a2db640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e840cefa0 0x7f8e84158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8a2db640 1 -- 172.21.15.148:0/1345261930 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e80009420 con 0x7f8e8415cd50 2023-10-30T13:10:28.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e89ada640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e840cefa0 0x7f8e84158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:28.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8a2db640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 0x7f8e840b2fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00ca80 tx=0x7f8e7c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:28.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e7a7fc640 1 -- 172.21.15.148:0/1345261930 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c00ac30 con 0x7f8e8415cd50 2023-10-30T13:10:28.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e7a7fc640 1 -- 172.21.15.148:0/1345261930 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c00ad90 con 0x7f8e8415cd50 2023-10-30T13:10:28.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e74009420 con 0x7f8e8415cd50 2023-10-30T13:10:28.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e7a7fc640 1 -- 172.21.15.148:0/1345261930 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c012600 con 0x7f8e8415cd50 2023-10-30T13:10:28.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8415ca60 con 0x7f8e8415cd50 2023-10-30T13:10:28.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8404e170 con 0x7f8e8415cd50 2023-10-30T13:10:28.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.036+0000 7f8e7a7fc640 1 -- 172.21.15.148:0/1345261930 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e7c007070 con 0x7f8e8415cd50 2023-10-30T13:10:28.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.040+0000 7f8e7a7fc640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8e68066470 0x7f8e68068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.040+0000 7f8e7a7fc640 1 -- 172.21.15.148:0/1345261930 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8e7c083940 con 0x7f8e8415cd50 2023-10-30T13:10:28.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.040+0000 7f8e89ada640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8e68066470 0x7f8e68068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.040+0000 7f8e89ada640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8e68066470 0x7f8e68068930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009fd0 tx=0x7f8e80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:28.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.044+0000 7f8e7a7fc640 1 -- 172.21.15.148:0/1345261930 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e7c0511c0 con 0x7f8e8415cd50 2023-10-30T13:10:28.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.196+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e84065f40 con 0x7f8e8415cd50 2023-10-30T13:10:28.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.196+0000 7f8e7a7fc640 1 -- 172.21.15.148:0/1345261930 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12886 (secure 0 0 0) 0x7f8e7c017090 con 0x7f8e8415cd50 2023-10-30T13:10:28.203 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:28.203 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":16,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:25.308810+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:28.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8e68066470 msgr2=0x7f8e68068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8e68066470 0x7f8e68068930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009fd0 tx=0x7f8e80009190 comp rx=0 tx=0).stop 2023-10-30T13:10:28.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 msgr2=0x7f8e840b2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 0x7f8e840b2fa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00ca80 tx=0x7f8e7c00cf40 comp rx=0 tx=0).stop 2023-10-30T13:10:28.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 shutdown_connections 2023-10-30T13:10:28.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8e68066470 0x7f8e68068930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e8415cd50 0x7f8e840b2fa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8e841538f0 0x7f8e84152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 --2- 172.21.15.148:0/1345261930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e840cefa0 0x7f8e84158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 >> 172.21.15.148:0/1345261930 conn(0x7f8e8406a640 msgr2=0x7f8e8406bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:28.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 shutdown_connections 2023-10-30T13:10:28.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.200+0000 7f8e8aadc640 1 -- 172.21.15.148:0/1345261930 wait complete. 2023-10-30T13:10:28.223 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:10:22.349322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:10:28.224 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:10:28.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.360+0000 7f83ed832640 1 Processor -- start 2023-10-30T13:10:28.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.360+0000 7f83ed832640 1 -- start start 2023-10-30T13:10:28.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.360+0000 7f83ed832640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e8159930 0x7f83e80cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.360+0000 7f83ed832640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f83e80cf6e0 0x7f83e815d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.360+0000 7f83ed832640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e815a920 con 0x7f83e8153980 2023-10-30T13:10:28.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.360+0000 7f83ed832640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e80cb930 con 0x7f83e80cf6e0 2023-10-30T13:10:28.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.360+0000 7f83ed832640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8159dc0 con 0x7f83e8159930 2023-10-30T13:10:28.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e77fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f83e80cf6e0 0x7f83e815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e6ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f83e8153980 0x7f83e80caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43972/0 (socket says 172.21.15.148:43972) 2023-10-30T13:10:28.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e6ffd640 1 -- 172.21.15.148:0/2143784173 learned_addr learned my addr 172.21.15.148:0/2143784173 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:28.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83dffff640 1 --2- 172.21.15.148:0/2143784173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e8159930 0x7f83e80cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e77fe640 1 -- 172.21.15.148:0/2143784173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e8159930 msgr2=0x7f83e80cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:28.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e77fe640 1 --2- 172.21.15.148:0/2143784173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e8159930 0x7f83e80cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e77fe640 1 -- 172.21.15.148:0/2143784173 >> v1:172.21.15.148:6789/0 conn(0x7f83e8153980 legacy=0x7f83e80caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e77fe640 1 -- 172.21.15.148:0/2143784173 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e815a080 con 0x7f83e80cf6e0 2023-10-30T13:10:28.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e77fe640 1 --2- 172.21.15.148:0/2143784173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e80cf6e0 0x7f83e815d570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f83e0009d90 tx=0x7f83e001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5518a479354b1582 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:28.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e67fc640 1 -- 172.21.15.148:0/2143784173 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e00a58f0 con 0x7f83e80cf6e0 2023-10-30T13:10:28.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e67fc640 1 -- 172.21.15.148:0/2143784173 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e00a5a50 con 0x7f83e80cf6e0 2023-10-30T13:10:28.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.364+0000 7f83e67fc640 1 -- 172.21.15.148:0/2143784173 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e00a5de0 con 0x7f83e80cf6e0 2023-10-30T13:10:28.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 -- 172.21.15.148:0/2143784173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e80cf6e0 msgr2=0x7f83e815d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 --2- 172.21.15.148:0/2143784173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e80cf6e0 0x7f83e815d570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f83e0009d90 tx=0x7f83e001c920 comp rx=0 tx=0).stop 2023-10-30T13:10:28.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 -- 172.21.15.148:0/2143784173 shutdown_connections 2023-10-30T13:10:28.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 --2- 172.21.15.148:0/2143784173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e80cf6e0 0x7f83e815d570 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 --2- 172.21.15.148:0/2143784173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e8159930 0x7f83e80cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 -- 172.21.15.148:0/2143784173 >> 172.21.15.148:0/2143784173 conn(0x7f83e806c0f0 msgr2=0x7f83e806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:28.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 -- 172.21.15.148:0/2143784173 shutdown_connections 2023-10-30T13:10:28.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 -- 172.21.15.148:0/2143784173 wait complete. 2023-10-30T13:10:28.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 Processor -- start 2023-10-30T13:10:28.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 -- start start 2023-10-30T13:10:28.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e80cf6e0 0x7f83e815c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e8153980 0x7f83e8148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e815b100 0x7f83e815b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e815cda0 con 0x7f83e8153980 2023-10-30T13:10:28.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e815cb30 con 0x7f83e815b100 2023-10-30T13:10:28.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83ed832640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e815bab0 con 0x7f83e80cf6e0 2023-10-30T13:10:28.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83dffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e8153980 0x7f83e8148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83dffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e8153980 0x7f83e8148250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41060/0 (socket says 172.21.15.148:41060) 2023-10-30T13:10:28.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83dffff640 1 -- 172.21.15.148:0/3054146999 learned_addr learned my addr 172.21.15.148:0/3054146999 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:28.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83e6ffd640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e80cf6e0 0x7f83e815c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83e77fe640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e815b100 0x7f83e815b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83dffff640 1 -- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e80cf6e0 msgr2=0x7f83e815c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83dffff640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e80cf6e0 0x7f83e815c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83dffff640 1 -- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e815b100 msgr2=0x7f83e815b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83dffff640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e815b100 0x7f83e815b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.368+0000 7f83dffff640 1 -- 172.21.15.148:0/3054146999 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e0009a70 con 0x7f83e8153980 2023-10-30T13:10:28.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83dffff640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e8153980 0x7f83e8148250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83d000d830 tx=0x7f83d000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:28.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83e4ff9640 1 -- 172.21.15.148:0/3054146999 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d000b6c0 con 0x7f83e8153980 2023-10-30T13:10:28.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83e4ff9640 1 -- 172.21.15.148:0/3054146999 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d000b820 con 0x7f83e8153980 2023-10-30T13:10:28.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e81541a0 con 0x7f83e8153980 2023-10-30T13:10:28.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83e4ff9640 1 -- 172.21.15.148:0/3054146999 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d000bb90 con 0x7f83e8153980 2023-10-30T13:10:28.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e8154440 con 0x7f83e8153980 2023-10-30T13:10:28.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e804e170 con 0x7f83e8153980 2023-10-30T13:10:28.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83e4ff9640 1 -- 172.21.15.148:0/3054146999 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83d001e460 con 0x7f83e8153980 2023-10-30T13:10:28.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83e4ff9640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f83c80664a0 0x7f83c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83e4ff9640 1 -- 172.21.15.148:0/3054146999 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f83d0084370 con 0x7f83e8153980 2023-10-30T13:10:28.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.372+0000 7f83e6ffd640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f83c80664a0 0x7f83c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.376+0000 7f83e6ffd640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f83c80664a0 0x7f83c8068960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f83d8005890 tx=0x7f83d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:28.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.376+0000 7f83e4ff9640 1 -- 172.21.15.148:0/3054146999 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83d0051aa0 con 0x7f83e8153980 2023-10-30T13:10:28.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83e80caa80 con 0x7f83e8153980 2023-10-30T13:10:28.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83e4ff9640 1 -- 172.21.15.148:0/3054146999 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83d00550c0 con 0x7f83e8153980 2023-10-30T13:10:28.532 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2023-10-30T13:10:28.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f83c80664a0 msgr2=0x7f83c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f83c80664a0 0x7f83c8068960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f83d8005890 tx=0x7f83d8009040 comp rx=0 tx=0).stop 2023-10-30T13:10:28.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e8153980 msgr2=0x7f83e8148250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e8153980 0x7f83e8148250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83d000d830 tx=0x7f83d000dcf0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 shutdown_connections 2023-10-30T13:10:28.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f83c80664a0 0x7f83c8068960 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83e8153980 0x7f83e8148250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e815b100 0x7f83e815b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 --2- 172.21.15.148:0/3054146999 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83e80cf6e0 0x7f83e815c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 >> 172.21.15.148:0/3054146999 conn(0x7f83e806c0f0 msgr2=0x7f83e80caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:28.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 shutdown_connections 2023-10-30T13:10:28.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.528+0000 7f83ed832640 1 -- 172.21.15.148:0/3054146999 wait complete. 2023-10-30T13:10:28.553 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:10:28.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc353194640 1 Processor -- start 2023-10-30T13:10:28.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc353194640 1 -- start start 2023-10-30T13:10:28.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc353194640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc34c06f7a0 0x7fc34c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc353194640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc34c071110 0x7fc34c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc353194640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc34c066cf0 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc353194640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc34c0700c0 con 0x7fc34c06f7a0 2023-10-30T13:10:28.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc353194640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc34c0c70d0 con 0x7fc34c071110 2023-10-30T13:10:28.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc350f30640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc34c06f7a0 0x7fc34c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc343fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc34c0c6bd0 0x7fc34c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43978/0 (socket says 172.21.15.148:43978) 2023-10-30T13:10:28.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc343fff640 1 -- 172.21.15.148:0/3367716570 learned_addr learned my addr 172.21.15.148:0/3367716570 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:28.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc351731640 1 --2- 172.21.15.148:0/3367716570 >> v2:172.21.15.150:3300/0 conn(0x7fc34c071110 0x7fc34c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc350f30640 1 -- 172.21.15.148:0/3367716570 >> v2:172.21.15.150:3300/0 conn(0x7fc34c071110 msgr2=0x7fc34c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:28.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc350f30640 1 --2- 172.21.15.148:0/3367716570 >> v2:172.21.15.150:3300/0 conn(0x7fc34c071110 0x7fc34c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc350f30640 1 -- 172.21.15.148:0/3367716570 >> v1:172.21.15.148:6789/0 conn(0x7fc34c0c6bd0 legacy=0x7fc34c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc350f30640 1 -- 172.21.15.148:0/3367716570 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c0c7390 con 0x7fc34c06f7a0 2023-10-30T13:10:28.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.688+0000 7fc3437fe640 1 -- 172.21.15.148:0/3367716570 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291819053 0 0) 0x7fc34c066cf0 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc350f30640 1 --2- 172.21.15.148:0/3367716570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c06f7a0 0x7fc34c06fb80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc34c06cdd0 tx=0x7fc34801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5580f729b5f6983 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:28.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc3437fe640 1 -- 172.21.15.148:0/3367716570 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3480a8950 con 0x7fc34c06f7a0 2023-10-30T13:10:28.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc3437fe640 1 -- 172.21.15.148:0/3367716570 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3480a8ab0 con 0x7fc34c06f7a0 2023-10-30T13:10:28.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc3437fe640 1 -- 172.21.15.148:0/3367716570 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3480a8e40 con 0x7fc34c06f7a0 2023-10-30T13:10:28.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc353194640 1 -- 172.21.15.148:0/3367716570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c06f7a0 msgr2=0x7fc34c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc353194640 1 --2- 172.21.15.148:0/3367716570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c06f7a0 0x7fc34c06fb80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc34c06cdd0 tx=0x7fc34801ec50 comp rx=0 tx=0).stop 2023-10-30T13:10:28.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc353194640 1 -- 172.21.15.148:0/3367716570 shutdown_connections 2023-10-30T13:10:28.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc353194640 1 --2- 172.21.15.148:0/3367716570 >> v2:172.21.15.150:3300/0 conn(0x7fc34c071110 0x7fc34c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc353194640 1 --2- 172.21.15.148:0/3367716570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c06f7a0 0x7fc34c06fb80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc353194640 1 -- 172.21.15.148:0/3367716570 >> 172.21.15.148:0/3367716570 conn(0x7fc34c0c7910 msgr2=0x7fc34c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:28.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc353194640 1 -- 172.21.15.148:0/3367716570 shutdown_connections 2023-10-30T13:10:28.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.692+0000 7fc353194640 1 -- 172.21.15.148:0/3367716570 wait complete. 2023-10-30T13:10:28.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc353194640 1 Processor -- start 2023-10-30T13:10:28.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc353194640 1 -- start start 2023-10-30T13:10:28.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc353194640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc34c06f7a0 0x7fc34c14d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc353194640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc34c071110 0x7fc34c1ddf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc353194640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c0c6bd0 0x7fc34c1de480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc353194640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c155010 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc353194640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c14d9c0 con 0x7fc34c06f7a0 2023-10-30T13:10:28.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc353194640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c1deb50 con 0x7fc34c071110 2023-10-30T13:10:28.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc351731640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c0c6bd0 0x7fc34c1de480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc351731640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c0c6bd0 0x7fc34c1de480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41064/0 (socket says 172.21.15.148:41064) 2023-10-30T13:10:28.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc351731640 1 -- 172.21.15.148:0/3965794596 learned_addr learned my addr 172.21.15.148:0/3965794596 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:28.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc343fff640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc34c071110 0x7fc34c1ddf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc350f30640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc34c06f7a0 0x7fc34c14d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc351731640 1 -- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc34c071110 msgr2=0x7fc34c1ddf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc351731640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc34c071110 0x7fc34c1ddf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc351731640 1 -- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc34c06f7a0 msgr2=0x7fc34c14d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc351731640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc34c06f7a0 0x7fc34c14d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc351731640 1 -- 172.21.15.148:0/3965794596 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc348009420 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc351731640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c0c6bd0 0x7fc34c1de480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc344007e60 tx=0x7fc34400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:28.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc3417fa640 1 -- 172.21.15.148:0/3965794596 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344014420 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc3417fa640 1 -- 172.21.15.148:0/3965794596 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3440145a0 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc3417fa640 1 -- 172.21.15.148:0/3965794596 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344014950 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.696+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34c154d30 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.700+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34c1e4420 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.700+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34c1e45e0 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.700+0000 7fc3417fa640 1 -- 172.21.15.148:0/3965794596 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc344014ab0 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.700+0000 7fc3417fa640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3380664a0 0x7fc338068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:28.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.700+0000 7fc3417fa640 1 -- 172.21.15.148:0/3965794596 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc3440843b0 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.700+0000 7fc350f30640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3380664a0 0x7fc338068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:28.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.700+0000 7fc350f30640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3380664a0 0x7fc338068960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc3480b2000 tx=0x7fc3480aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:28.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.704+0000 7fc3417fa640 1 -- 172.21.15.148:0/3965794596 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc344051ba0 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.884+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc34c1e3fa0 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.884+0000 7fc3417fa640 1 -- 172.21.15.148:0/3965794596 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1327 (secure 0 0 0) 0x7fc34c1e3fa0 con 0x7fc34c0c6bd0 2023-10-30T13:10:28.890 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:28.891 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-30T13:10:28.891 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 2 2023-10-30T13:10:28.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3380664a0 msgr2=0x7fc338068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3380664a0 0x7fc338068960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc3480b2000 tx=0x7fc3480aa070 comp rx=0 tx=0).stop 2023-10-30T13:10:28.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c0c6bd0 msgr2=0x7fc34c1de480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:28.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c0c6bd0 0x7fc34c1de480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc344007e60 tx=0x7fc34400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 shutdown_connections 2023-10-30T13:10:28.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3380664a0 0x7fc338068960 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc34c0c6bd0 0x7fc34c1de480 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc34c071110 0x7fc34c1ddf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 --2- 172.21.15.148:0/3965794596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc34c06f7a0 0x7fc34c14d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:28.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 >> 172.21.15.148:0/3965794596 conn(0x7fc34c0c7910 msgr2=0x7fc34c0c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:28.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 shutdown_connections 2023-10-30T13:10:28.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:28.888+0000 7fc353194640 1 -- 172.21.15.148:0/3965794596 wait complete. 2023-10-30T13:10:30.912 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:10:31.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9ce846640 1 Processor -- start 2023-10-30T13:10:31.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9ce846640 1 -- start start 2023-10-30T13:10:31.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9ce846640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa9c81538a0 0x7fa9c815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9ce846640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c80cb800 con 0x7fa9c80cf160 2023-10-30T13:10:31.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9ce846640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c815dc30 con 0x7fa9c80ca0e0 2023-10-30T13:10:31.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9ce846640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c80cb330 con 0x7fa9c81538a0 2023-10-30T13:10:31.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9c7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa9c81538a0 0x7fa9c815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9c7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa9c81538a0 0x7fa9c815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41068/0 (socket says 172.21.15.148:41068) 2023-10-30T13:10:31.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9ccde3640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa9c80cf160 0x7fa9c8164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43982/0 (socket says 172.21.15.148:43982) 2023-10-30T13:10:31.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9c7fff640 1 -- 172.21.15.148:0/3701447565 learned_addr learned my addr 172.21.15.148:0/3701447565 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:31.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9c7fff640 1 -- 172.21.15.148:0/3701447565 >> v1:172.21.15.150:6789/0 conn(0x7fa9c80ca0e0 legacy=0x7fa9c815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9c7fff640 1 -- 172.21.15.148:0/3701447565 >> v1:172.21.15.148:6789/0 conn(0x7fa9c80cf160 legacy=0x7fa9c8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9c7fff640 1 -- 172.21.15.148:0/3701447565 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c80cad70 con 0x7fa9c81538a0 2023-10-30T13:10:31.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.056+0000 7fa9c6ffd640 1 -- 172.21.15.148:0/3701447565 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999885372 0 0) 0x7fa9c80cb800 con 0x7fa9c80cf160 2023-10-30T13:10:31.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9c7fff640 1 --2- 172.21.15.148:0/3701447565 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 0x7fa9c815d6f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009870 tx=0x7fa9b801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e231aba74d35cd6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:31.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9c6ffd640 1 -- 172.21.15.148:0/3701447565 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b80a8a30 con 0x7fa9c81538a0 2023-10-30T13:10:31.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9c6ffd640 1 -- 172.21.15.148:0/3701447565 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b80a8b90 con 0x7fa9c81538a0 2023-10-30T13:10:31.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9c6ffd640 1 -- 172.21.15.148:0/3701447565 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b80b0440 con 0x7fa9c81538a0 2023-10-30T13:10:31.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9ce846640 1 -- 172.21.15.148:0/3701447565 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 msgr2=0x7fa9c815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9ce846640 1 --2- 172.21.15.148:0/3701447565 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 0x7fa9c815d6f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009870 tx=0x7fa9b801ed30 comp rx=0 tx=0).stop 2023-10-30T13:10:31.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9ce846640 1 -- 172.21.15.148:0/3701447565 shutdown_connections 2023-10-30T13:10:31.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9ce846640 1 --2- 172.21.15.148:0/3701447565 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 0x7fa9c815d6f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9ce846640 1 -- 172.21.15.148:0/3701447565 >> 172.21.15.148:0/3701447565 conn(0x7fa9c806bfe0 msgr2=0x7fa9c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:31.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9ce846640 1 -- 172.21.15.148:0/3701447565 shutdown_connections 2023-10-30T13:10:31.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.060+0000 7fa9ce846640 1 -- 172.21.15.148:0/3701447565 wait complete. 2023-10-30T13:10:31.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 Processor -- start 2023-10-30T13:10:31.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 -- start start 2023-10-30T13:10:31.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9c80ca0e0 0x7fa9c80b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c80cf160 0x7fa9c80b3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 0x7fa9c8143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c81e3130 con 0x7fa9c81538a0 2023-10-30T13:10:31.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c8157f30 con 0x7fa9c80cf160 2023-10-30T13:10:31.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c8153100 con 0x7fa9c80ca0e0 2023-10-30T13:10:31.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ccde3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 0x7fa9c8143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ccde3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 0x7fa9c8143920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41072/0 (socket says 172.21.15.148:41072) 2023-10-30T13:10:31.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ccde3640 1 -- 172.21.15.148:0/4021339933 learned_addr learned my addr 172.21.15.148:0/4021339933 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:31.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9c77fe640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c80cf160 0x7fa9c80b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9c7fff640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9c80ca0e0 0x7fa9c80b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ccde3640 1 -- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9c80ca0e0 msgr2=0x7fa9c80b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ccde3640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9c80ca0e0 0x7fa9c80b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ccde3640 1 -- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c80cf160 msgr2=0x7fa9c80b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ccde3640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c80cf160 0x7fa9c80b3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ccde3640 1 -- 172.21.15.148:0/4021339933 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b80094d0 con 0x7fa9c81538a0 2023-10-30T13:10:31.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9c77fe640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c80cf160 0x7fa9c80b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:31.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ccde3640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 0x7fa9c8143920 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000c780 tx=0x7fa9c000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:31.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9c4ff9640 1 -- 172.21.15.148:0/4021339933 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c000ce30 con 0x7fa9c81538a0 2023-10-30T13:10:31.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9c4ff9640 1 -- 172.21.15.148:0/4021339933 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c000a640 con 0x7fa9c81538a0 2023-10-30T13:10:31.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9c4ff9640 1 -- 172.21.15.148:0/4021339933 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c000aa30 con 0x7fa9c81538a0 2023-10-30T13:10:31.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c8147190 con 0x7fa9c81538a0 2023-10-30T13:10:31.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.064+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c8158460 con 0x7fa9c81538a0 2023-10-30T13:10:31.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.068+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c804e170 con 0x7fa9c81538a0 2023-10-30T13:10:31.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.068+0000 7fa9c4ff9640 1 -- 172.21.15.148:0/4021339933 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9c000ab90 con 0x7fa9c81538a0 2023-10-30T13:10:31.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.068+0000 7fa9c4ff9640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9b40664a0 0x7fa9b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.068+0000 7fa9c4ff9640 1 -- 172.21.15.148:0/4021339933 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa9c0082f90 con 0x7fa9c81538a0 2023-10-30T13:10:31.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.068+0000 7fa9c7fff640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9b40664a0 0x7fa9b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.068+0000 7fa9c7fff640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9b40664a0 0x7fa9b4068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80b2000 tx=0x7fa9b80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:31.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.072+0000 7fa9c4ff9640 1 -- 172.21.15.148:0/4021339933 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9c0050770 con 0x7fa9c81538a0 2023-10-30T13:10:31.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.252+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9c80c46c0 con 0x7fa9c81538a0 2023-10-30T13:10:31.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.252+0000 7fa9c4ff9640 1 -- 172.21.15.148:0/4021339933 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5292 (secure 0 0 0) 0x7fa9c0018020 con 0x7fa9c81538a0 2023-10-30T13:10:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:31.259 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4285,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-30T13:10:31.259 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 3 2023-10-30T13:10:31.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9b40664a0 msgr2=0x7fa9b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9b40664a0 0x7fa9b4068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa9b80b2000 tx=0x7fa9b80aa070 comp rx=0 tx=0).stop 2023-10-30T13:10:31.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 msgr2=0x7fa9c8143920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 0x7fa9c8143920 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000c780 tx=0x7fa9c000cc40 comp rx=0 tx=0).stop 2023-10-30T13:10:31.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 shutdown_connections 2023-10-30T13:10:31.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9b40664a0 0x7fa9b4068960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9c81538a0 0x7fa9c8143920 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9c80cf160 0x7fa9c80b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 --2- 172.21.15.148:0/4021339933 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9c80ca0e0 0x7fa9c80b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 >> 172.21.15.148:0/4021339933 conn(0x7fa9c806bfe0 msgr2=0x7fa9c8149660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:31.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 shutdown_connections 2023-10-30T13:10:31.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.256+0000 7fa9ce846640 1 -- 172.21.15.148:0/4021339933 wait complete. 2023-10-30T13:10:31.279 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:31.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.412+0000 7fd89cf3e640 1 Processor -- start 2023-10-30T13:10:31.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.412+0000 7fd89cf3e640 1 -- start start 2023-10-30T13:10:31.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.412+0000 7fd89cf3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8981570f0 0x7fd89806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.412+0000 7fd89cf3e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd89806f370 0x7fd8981640d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.412+0000 7fd89cf3e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89806b3b0 con 0x7fd8980b2300 2023-10-30T13:10:31.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.412+0000 7fd89cf3e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8980c38a0 con 0x7fd89806f370 2023-10-30T13:10:31.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.412+0000 7fd89cf3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8980c3b20 con 0x7fd8981570f0 2023-10-30T13:10:31.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd88ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd8980b2300 0x7fd898071b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43988/0 (socket says 172.21.15.148:43988) 2023-10-30T13:10:31.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd88ffff640 1 -- 172.21.15.148:0/3031640813 learned_addr learned my addr 172.21.15.148:0/3031640813 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:31.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd896d9d640 1 --2- 172.21.15.148:0/3031640813 >> v2:172.21.15.148:3300/0 conn(0x7fd89806f370 0x7fd8981640d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd89659c640 1 --2- 172.21.15.148:0/3031640813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8981570f0 0x7fd89806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd895d9b640 1 -- 172.21.15.148:0/3031640813 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469758850 0 0) 0x7fd89806b3b0 con 0x7fd8980b2300 2023-10-30T13:10:31.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd895d9b640 1 -- 172.21.15.148:0/3031640813 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd884003530 con 0x7fd8980b2300 2023-10-30T13:10:31.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd896d9d640 1 -- 172.21.15.148:0/3031640813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8981570f0 msgr2=0x7fd89806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:31.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd896d9d640 1 --2- 172.21.15.148:0/3031640813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8981570f0 0x7fd89806ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd896d9d640 1 -- 172.21.15.148:0/3031640813 >> v1:172.21.15.148:6789/0 conn(0x7fd8980b2300 legacy=0x7fd898071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd896d9d640 1 -- 172.21.15.148:0/3031640813 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8980ca9f0 con 0x7fd89806f370 2023-10-30T13:10:31.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd896d9d640 1 --2- 172.21.15.148:0/3031640813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd89806f370 0x7fd8981640d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd890009d90 tx=0x7fd89001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d1d1b884fb708b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:31.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd895d9b640 1 -- 172.21.15.148:0/3031640813 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8900a5900 con 0x7fd89806f370 2023-10-30T13:10:31.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd895d9b640 1 -- 172.21.15.148:0/3031640813 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8900a5a60 con 0x7fd89806f370 2023-10-30T13:10:31.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.416+0000 7fd895d9b640 1 -- 172.21.15.148:0/3031640813 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8900a5df0 con 0x7fd89806f370 2023-10-30T13:10:31.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3031640813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd89806f370 msgr2=0x7fd8981640d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 --2- 172.21.15.148:0/3031640813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd89806f370 0x7fd8981640d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd890009d90 tx=0x7fd89001c950 comp rx=0 tx=0).stop 2023-10-30T13:10:31.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3031640813 shutdown_connections 2023-10-30T13:10:31.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 --2- 172.21.15.148:0/3031640813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd89806f370 0x7fd8981640d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 --2- 172.21.15.148:0/3031640813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd8981570f0 0x7fd89806ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3031640813 >> 172.21.15.148:0/3031640813 conn(0x7fd898069e90 msgr2=0x7fd89806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:31.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3031640813 shutdown_connections 2023-10-30T13:10:31.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3031640813 wait complete. 2023-10-30T13:10:31.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 Processor -- start 2023-10-30T13:10:31.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 -- start start 2023-10-30T13:10:31.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd89806f370 0x7fd8981f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8980b2300 0x7fd898148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8981570f0 0x7fd898146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd898144540 con 0x7fd8980b2300 2023-10-30T13:10:31.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8981f1eb0 con 0x7fd8981570f0 2023-10-30T13:10:31.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89cf3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8981f2430 con 0x7fd89806f370 2023-10-30T13:10:31.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd88ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8980b2300 0x7fd898148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd88ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8980b2300 0x7fd898148a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41092/0 (socket says 172.21.15.148:41092) 2023-10-30T13:10:31.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd88ffff640 1 -- 172.21.15.148:0/3764207158 learned_addr learned my addr 172.21.15.148:0/3764207158 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:31.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89659c640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd89806f370 0x7fd8981f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd896d9d640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8981570f0 0x7fd898146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd88ffff640 1 -- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd89806f370 msgr2=0x7fd8981f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd88ffff640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd89806f370 0x7fd8981f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd88ffff640 1 -- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8981570f0 msgr2=0x7fd898146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd88ffff640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8981570f0 0x7fd898146210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd88ffff640 1 -- 172.21.15.148:0/3764207158 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd890009a70 con 0x7fd8980b2300 2023-10-30T13:10:31.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd89659c640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd89806f370 0x7fd8981f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:31.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.420+0000 7fd896d9d640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8981570f0 0x7fd898146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:31.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd88ffff640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8980b2300 0x7fd898148a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd88000c420 tx=0x7fd88000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:31.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd88effd640 1 -- 172.21.15.148:0/3764207158 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880005cf0 con 0x7fd8980b2300 2023-10-30T13:10:31.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd88effd640 1 -- 172.21.15.148:0/3764207158 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd880005e50 con 0x7fd8980b2300 2023-10-30T13:10:31.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8981f2e00 con 0x7fd8980b2300 2023-10-30T13:10:31.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd88effd640 1 -- 172.21.15.148:0/3764207158 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8800072d0 con 0x7fd8980b2300 2023-10-30T13:10:31.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd898144d20 con 0x7fd8980b2300 2023-10-30T13:10:31.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89804e170 con 0x7fd8980b2300 2023-10-30T13:10:31.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd88effd640 1 -- 172.21.15.148:0/3764207158 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd880007430 con 0x7fd8980b2300 2023-10-30T13:10:31.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd88effd640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd8780664a0 0x7fd878068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd88effd640 1 -- 172.21.15.148:0/3764207158 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd880083160 con 0x7fd8980b2300 2023-10-30T13:10:31.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.424+0000 7fd89659c640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd8780664a0 0x7fd878068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.428+0000 7fd89659c640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd8780664a0 0x7fd878068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd888009bb0 tx=0x7fd888008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:31.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.428+0000 7fd88effd640 1 -- 172.21.15.148:0/3764207158 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd8800508b0 con 0x7fd8980b2300 2023-10-30T13:10:31.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd89806fbd0 con 0x7fd8980b2300 2023-10-30T13:10:31.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd88effd640 1 -- 172.21.15.148:0/3764207158 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12886 (secure 0 0 0) 0x7fd880017090 con 0x7fd8980b2300 2023-10-30T13:10:31.584 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:31.585 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":16,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:25.308810+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd8780664a0 msgr2=0x7fd878068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd8780664a0 0x7fd878068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd888009bb0 tx=0x7fd888008040 comp rx=0 tx=0).stop 2023-10-30T13:10:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8980b2300 msgr2=0x7fd898148a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8980b2300 0x7fd898148a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd88000c420 tx=0x7fd88000c8e0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 shutdown_connections 2023-10-30T13:10:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd8780664a0 0x7fd878068960 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd8980b2300 0x7fd898148a40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8981570f0 0x7fd898146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 --2- 172.21.15.148:0/3764207158 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd89806f370 0x7fd8981f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 >> 172.21.15.148:0/3764207158 conn(0x7fd898069e90 msgr2=0x7fd8981550b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 shutdown_connections 2023-10-30T13:10:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.580+0000 7fd89cf3e640 1 -- 172.21.15.148:0/3764207158 wait complete. 2023-10-30T13:10:31.605 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:10:22.349322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:10:31.606 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:10:31.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.740+0000 7f970fdb8640 1 Processor -- start 2023-10-30T13:10:31.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.740+0000 7f970fdb8640 1 -- start start 2023-10-30T13:10:31.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970fdb8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159fb0 0x7f970806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970fdb8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970806b2e0 con 0x7f970806fda0 2023-10-30T13:10:31.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970fdb8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970815d020 con 0x7f9708156fd0 2023-10-30T13:10:31.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970fdb8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970806bbe0 con 0x7f9708159fb0 2023-10-30T13:10:31.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970d353640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f970806fda0 0x7f97081598a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44002/0 (socket says 172.21.15.148:44002) 2023-10-30T13:10:31.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970d353640 1 -- 172.21.15.148:0/568073924 learned_addr learned my addr 172.21.15.148:0/568073924 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:31.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970e355640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159fb0 0x7f970806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075324359 0 0) 0x7f970806b2e0 con 0x7f970806fda0 2023-10-30T13:10:31.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f4003530 con 0x7f970806fda0 2023-10-30T13:10:31.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421519872 0 0) 0x7f96f4003530 con 0x7f970806fda0 2023-10-30T13:10:31.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159fb0 msgr2=0x7f970806ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:31.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159fb0 0x7f970806ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 >> v1:172.21.15.150:6789/0 conn(0x7f9708156fd0 legacy=0x7f9708071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970815d300 con 0x7f970806fda0 2023-10-30T13:10:31.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f96f8001680 con 0x7f970806fda0 2023-10-30T13:10:31.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 >> v1:172.21.15.148:6789/0 conn(0x7f970806fda0 legacy=0x7f97081598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96f40039b0 0x7f96f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96f4004310 0x7f96f40047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f96f400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970806bbe0 con 0x7f96f4004ff0 2023-10-30T13:10:31.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f400bf00 con 0x7f96f4004310 2023-10-30T13:10:31.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f400c0c0 con 0x7f96f40039b0 2023-10-30T13:10:31.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970e355640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f96f400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970d353640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96f4004310 0x7f96f40047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970db54640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96f40039b0 0x7f96f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970e355640 1 -- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96f40039b0 msgr2=0x7f96f4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:31.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970e355640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f96f40039b0 0x7f96f4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970e355640 1 -- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96f4004310 msgr2=0x7f96f40047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:31.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970e355640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96f4004310 0x7f96f40047f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970e355640 1 -- 172.21.15.148:0/568073924 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f4003530 con 0x7f96f4004ff0 2023-10-30T13:10:31.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.744+0000 7f970d353640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96f4004310 0x7f96f40047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:31.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970e355640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f96f400b990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9700009fc0 tx=0x7f970001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed324fe8cf7319cb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:31.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97000a5c70 con 0x7f96f4004ff0 2023-10-30T13:10:31.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970000f040 con 0x7f96f4004ff0 2023-10-30T13:10:31.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970cb52640 1 -- 172.21.15.148:0/568073924 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97000ad440 con 0x7f96f4004ff0 2023-10-30T13:10:31.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970fdb8640 1 -- 172.21.15.148:0/568073924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 msgr2=0x7f96f400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970fdb8640 1 --2- 172.21.15.148:0/568073924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f96f400b990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9700009fc0 tx=0x7f970001cc60 comp rx=0 tx=0).stop 2023-10-30T13:10:31.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970e355640 1 -- 172.21.15.148:0/568073924 reap_dead start 2023-10-30T13:10:31.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970fdb8640 1 -- 172.21.15.148:0/568073924 shutdown_connections 2023-10-30T13:10:31.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970fdb8640 1 -- 172.21.15.148:0/568073924 >> 172.21.15.148:0/568073924 conn(0x7f9708069e90 msgr2=0x7f970806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:31.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970fdb8640 1 -- 172.21.15.148:0/568073924 shutdown_connections 2023-10-30T13:10:31.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.748+0000 7f970fdb8640 1 -- 172.21.15.148:0/568073924 wait complete. 2023-10-30T13:10:31.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 Processor -- start 2023-10-30T13:10:31.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 -- start start 2023-10-30T13:10:31.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f97080cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159a50 0x7f97080c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97081468b0 0x7f9708146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97081472a0 con 0x7f96f4004ff0 2023-10-30T13:10:31.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9708158c10 con 0x7f97081468b0 2023-10-30T13:10:31.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97081f2120 con 0x7f9708159a50 2023-10-30T13:10:31.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970db54640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f97080cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970db54640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f97080cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41106/0 (socket says 172.21.15.148:41106) 2023-10-30T13:10:31.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970db54640 1 -- 172.21.15.148:0/3953484940 learned_addr learned my addr 172.21.15.148:0/3953484940 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:31.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970d353640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159a50 0x7f97080c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970e355640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97081468b0 0x7f9708146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970db54640 1 -- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159a50 msgr2=0x7f97080c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970db54640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159a50 0x7f97080c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970db54640 1 -- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97081468b0 msgr2=0x7f9708146d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970db54640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97081468b0 0x7f9708146d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970db54640 1 -- 172.21.15.148:0/3953484940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97081f2400 con 0x7f96f4004ff0 2023-10-30T13:10:31.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970d353640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159a50 0x7f97080c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:31.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970db54640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f97080cd8b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f970400e580 tx=0x7f970400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:31.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f96fe7fc640 1 -- 172.21.15.148:0/3953484940 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97040052e0 con 0x7f96f4004ff0 2023-10-30T13:10:31.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f96fe7fc640 1 -- 172.21.15.148:0/3953484940 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9704005440 con 0x7f96f4004ff0 2023-10-30T13:10:31.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f96fe7fc640 1 -- 172.21.15.148:0/3953484940 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704003820 con 0x7f96f4004ff0 2023-10-30T13:10:31.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97081f26e0 con 0x7f96f4004ff0 2023-10-30T13:10:31.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97081f2ca0 con 0x7f96f4004ff0 2023-10-30T13:10:31.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.752+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97081f2e30 con 0x7f96f4004ff0 2023-10-30T13:10:31.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.756+0000 7f96fe7fc640 1 -- 172.21.15.148:0/3953484940 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9704003980 con 0x7f96f4004ff0 2023-10-30T13:10:31.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.756+0000 7f96fe7fc640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96ec0664a0 0x7f96ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:31.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.756+0000 7f96fe7fc640 1 -- 172.21.15.148:0/3953484940 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9704083980 con 0x7f96f4004ff0 2023-10-30T13:10:31.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.756+0000 7f970d353640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96ec0664a0 0x7f96ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:31.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.756+0000 7f970d353640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96ec0664a0 0x7f96ec068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f97080c3cf0 tx=0x7f96f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:31.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.760+0000 7f96fe7fc640 1 -- 172.21.15.148:0/3953484940 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9704051180 con 0x7f96f4004ff0 2023-10-30T13:10:31.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.908+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f97080b2460 con 0x7f96f4004ff0 2023-10-30T13:10:31.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.908+0000 7f96fe7fc640 1 -- 172.21.15.148:0/3953484940 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97040547a0 con 0x7f96f4004ff0 2023-10-30T13:10:31.914 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2023-10-30T13:10:31.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96ec0664a0 msgr2=0x7f96ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96ec0664a0 0x7f96ec068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f97080c3cf0 tx=0x7f96f8009040 comp rx=0 tx=0).stop 2023-10-30T13:10:31.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 msgr2=0x7f97080cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:31.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f97080cd8b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f970400e580 tx=0x7f970400ea40 comp rx=0 tx=0).stop 2023-10-30T13:10:31.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 shutdown_connections 2023-10-30T13:10:31.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96ec0664a0 0x7f96ec068960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97081468b0 0x7f9708146d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9708159a50 0x7f97080c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 --2- 172.21.15.148:0/3953484940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96f4004ff0 0x7f97080cd8b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:31.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 >> 172.21.15.148:0/3953484940 conn(0x7f9708069e90 msgr2=0x7f97080c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:31.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 shutdown_connections 2023-10-30T13:10:31.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:31.912+0000 7f970fdb8640 1 -- 172.21.15.148:0/3953484940 wait complete. 2023-10-30T13:10:31.934 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-30T13:10:32.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9fe46640 1 Processor -- start 2023-10-30T13:10:32.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9fe46640 1 -- start start 2023-10-30T13:10:32.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9fe46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab98153d70 0x7fab9806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9fe46640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab9806e470 0x7fab980cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9fe46640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9806afa0 con 0x7fab980c8490 2023-10-30T13:10:32.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9fe46640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9806d780 con 0x7fab9806e470 2023-10-30T13:10:32.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9fe46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab98154150 con 0x7fab98153d70 2023-10-30T13:10:32.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9d3e1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab9806e470 0x7fab980cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9e3e3640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fab980c8490 0x7fab980c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34284/0 (socket says 172.21.15.148:34284) 2023-10-30T13:10:32.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9e3e3640 1 -- 172.21.15.148:0/534097020 learned_addr learned my addr 172.21.15.148:0/534097020 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:32.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9dbe2640 1 --2- 172.21.15.148:0/534097020 >> v2:172.21.15.150:3300/0 conn(0x7fab98153d70 0x7fab9806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9d3e1640 1 -- 172.21.15.148:0/534097020 >> v2:172.21.15.150:3300/0 conn(0x7fab98153d70 msgr2=0x7fab9806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:32.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9d3e1640 1 --2- 172.21.15.148:0/534097020 >> v2:172.21.15.150:3300/0 conn(0x7fab98153d70 0x7fab9806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9d3e1640 1 -- 172.21.15.148:0/534097020 >> v1:172.21.15.148:6789/0 conn(0x7fab980c8490 legacy=0x7fab980c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9d3e1640 1 -- 172.21.15.148:0/534097020 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9806e030 con 0x7fab9806e470 2023-10-30T13:10:32.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.068+0000 7fab9cbe0640 1 -- 172.21.15.148:0/534097020 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997521210 0 0) 0x7fab9806afa0 con 0x7fab980c8490 2023-10-30T13:10:32.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9d3e1640 1 --2- 172.21.15.148:0/534097020 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab9806e470 0x7fab980cce00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fab9806d180 tx=0x7fab8801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f57873a198a47b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:32.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9cbe0640 1 -- 172.21.15.148:0/534097020 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab880a7070 con 0x7fab9806e470 2023-10-30T13:10:32.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9cbe0640 1 -- 172.21.15.148:0/534097020 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab880ab430 con 0x7fab9806e470 2023-10-30T13:10:32.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9cbe0640 1 -- 172.21.15.148:0/534097020 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab880ab7c0 con 0x7fab9806e470 2023-10-30T13:10:32.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9fe46640 1 -- 172.21.15.148:0/534097020 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab9806e470 msgr2=0x7fab980cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9fe46640 1 --2- 172.21.15.148:0/534097020 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab9806e470 0x7fab980cce00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fab9806d180 tx=0x7fab8801c430 comp rx=0 tx=0).stop 2023-10-30T13:10:32.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9fe46640 1 -- 172.21.15.148:0/534097020 shutdown_connections 2023-10-30T13:10:32.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9fe46640 1 --2- 172.21.15.148:0/534097020 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab9806e470 0x7fab980cce00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9fe46640 1 --2- 172.21.15.148:0/534097020 >> v2:172.21.15.150:3300/0 conn(0x7fab98153d70 0x7fab9806ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9fe46640 1 -- 172.21.15.148:0/534097020 >> 172.21.15.148:0/534097020 conn(0x7fab98071ea0 msgr2=0x7fab98065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:32.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9fe46640 1 -- 172.21.15.148:0/534097020 shutdown_connections 2023-10-30T13:10:32.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.072+0000 7fab9fe46640 1 -- 172.21.15.148:0/534097020 wait complete. 2023-10-30T13:10:32.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 Processor -- start 2023-10-30T13:10:32.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 -- start start 2023-10-30T13:10:32.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9806e470 0x7fab9814f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab980c8490 0x7fab981f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab98153d70 0x7fab981f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab981f7000 con 0x7fab98153d70 2023-10-30T13:10:32.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab98142f40 con 0x7fab9806e470 2023-10-30T13:10:32.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab981431e0 con 0x7fab980c8490 2023-10-30T13:10:32.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9e3e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab98153d70 0x7fab981f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9e3e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab98153d70 0x7fab981f3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50452/0 (socket says 172.21.15.148:50452) 2023-10-30T13:10:32.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9e3e3640 1 -- 172.21.15.148:0/744908770 learned_addr learned my addr 172.21.15.148:0/744908770 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:32.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9dbe2640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9806e470 0x7fab9814f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9d3e1640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab980c8490 0x7fab981f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9e3e3640 1 -- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab980c8490 msgr2=0x7fab981f2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9e3e3640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab980c8490 0x7fab981f2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9e3e3640 1 -- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9806e470 msgr2=0x7fab9814f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9e3e3640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9806e470 0x7fab9814f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9e3e3640 1 -- 172.21.15.148:0/744908770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab88009420 con 0x7fab98153d70 2023-10-30T13:10:32.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9d3e1640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab980c8490 0x7fab981f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:32.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9e3e3640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab98153d70 0x7fab981f3420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab90001570 tx=0x7fab9000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:32.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab8e7fc640 1 -- 172.21.15.148:0/744908770 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9000a640 con 0x7fab98153d70 2023-10-30T13:10:32.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab8e7fc640 1 -- 172.21.15.148:0/744908770 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9000a7c0 con 0x7fab98153d70 2023-10-30T13:10:32.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab8e7fc640 1 -- 172.21.15.148:0/744908770 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9000ab70 con 0x7fab98153d70 2023-10-30T13:10:32.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab981f63f0 con 0x7fab98153d70 2023-10-30T13:10:32.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.076+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab981f69b0 con 0x7fab98153d70 2023-10-30T13:10:32.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.080+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab981f7170 con 0x7fab98153d70 2023-10-30T13:10:32.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.080+0000 7fab8e7fc640 1 -- 172.21.15.148:0/744908770 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab9000acd0 con 0x7fab98153d70 2023-10-30T13:10:32.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.080+0000 7fab8e7fc640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab7c0664a0 0x7fab7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.080+0000 7fab8e7fc640 1 -- 172.21.15.148:0/744908770 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fab900830f0 con 0x7fab98153d70 2023-10-30T13:10:32.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.080+0000 7fab9dbe2640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab7c0664a0 0x7fab7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.080+0000 7fab9dbe2640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab7c0664a0 0x7fab7c068960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fab940079c0 tx=0x7fab94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:32.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.084+0000 7fab8e7fc640 1 -- 172.21.15.148:0/744908770 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab900508d0 con 0x7fab98153d70 2023-10-30T13:10:32.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.232+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fab9804e170 con 0x7fab98153d70 2023-10-30T13:10:32.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.232+0000 7fab8e7fc640 1 -- 172.21.15.148:0/744908770 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7fab90018090 con 0x7fab98153d70 2023-10-30T13:10:32.239 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:32.239 INFO:teuthology.orchestra.run.smithi148.stdout:[ 2023-10-30T13:10:32.239 INFO:teuthology.orchestra.run.smithi148.stdout: { 2023-10-30T13:10:32.239 INFO:teuthology.orchestra.run.smithi148.stdout: "poolnum": 1, 2023-10-30T13:10:32.239 INFO:teuthology.orchestra.run.smithi148.stdout: "poolname": ".mgr" 2023-10-30T13:10:32.240 INFO:teuthology.orchestra.run.smithi148.stdout: } 2023-10-30T13:10:32.240 INFO:teuthology.orchestra.run.smithi148.stdout:] 2023-10-30T13:10:32.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab7c0664a0 msgr2=0x7fab7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab7c0664a0 0x7fab7c068960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fab940079c0 tx=0x7fab94008040 comp rx=0 tx=0).stop 2023-10-30T13:10:32.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab98153d70 msgr2=0x7fab981f3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab98153d70 0x7fab981f3420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab90001570 tx=0x7fab9000cca0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 shutdown_connections 2023-10-30T13:10:32.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab7c0664a0 0x7fab7c068960 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab98153d70 0x7fab981f3420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab980c8490 0x7fab981f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 --2- 172.21.15.148:0/744908770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9806e470 0x7fab9814f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 >> 172.21.15.148:0/744908770 conn(0x7fab98071ea0 msgr2=0x7fab980c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:32.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 shutdown_connections 2023-10-30T13:10:32.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.236+0000 7fab9fe46640 1 -- 172.21.15.148:0/744908770 wait complete. 2023-10-30T13:10:32.259 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-30T13:10:32.260 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-30T13:10:32.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.396+0000 7f4f4c8d5640 1 Processor -- start 2023-10-30T13:10:32.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.396+0000 7f4f4c8d5640 1 -- start start 2023-10-30T13:10:32.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f4c8d5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4f4815cd50 0x7f4f480c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f4c8d5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 0x7f4f480cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f4c8d5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4815a6c0 con 0x7f4f481538f0 2023-10-30T13:10:32.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f4c8d5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4815d300 con 0x7f4f4815cd50 2023-10-30T13:10:32.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f4c8d5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4815d5e0 con 0x7f4f480cefa0 2023-10-30T13:10:32.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f3e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4f4815cd50 0x7f4f480c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f46d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4f481538f0 0x7f4f48159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:37278/0 (socket says 172.21.15.148:37278) 2023-10-30T13:10:32.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f4759e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 0x7f4f480cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f46d9d640 1 -- 172.21.15.148:0/1184617774 learned_addr learned my addr 172.21.15.148:0/1184617774 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:32.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f3e59c640 1 -- 172.21.15.148:0/1184617774 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 msgr2=0x7f4f480cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:32.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f3e59c640 1 --2- 172.21.15.148:0/1184617774 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 0x7f4f480cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f3e59c640 1 -- 172.21.15.148:0/1184617774 >> v1:172.21.15.150:6789/0 conn(0x7f4f481538f0 legacy=0x7f4f48159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f3e59c640 1 -- 172.21.15.148:0/1184617774 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4815d8c0 con 0x7f4f4815cd50 2023-10-30T13:10:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.400+0000 7f4f3e59c640 1 --2- 172.21.15.148:0/1184617774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4815cd50 0x7f4f480c9f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009d90 tx=0x7f4f3001f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc5e31bb26b0211c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4659c640 1 -- 172.21.15.148:0/1184617774 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f300af600 con 0x7f4f4815cd50 2023-10-30T13:10:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4659c640 1 -- 172.21.15.148:0/1184617774 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f300ab050 con 0x7f4f4815cd50 2023-10-30T13:10:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4659c640 1 -- 172.21.15.148:0/1184617774 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f300af980 con 0x7f4f4815cd50 2023-10-30T13:10:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/1184617774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4815cd50 msgr2=0x7f4f480c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 --2- 172.21.15.148:0/1184617774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4815cd50 0x7f4f480c9f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009d90 tx=0x7f4f3001f130 comp rx=0 tx=0).stop 2023-10-30T13:10:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/1184617774 shutdown_connections 2023-10-30T13:10:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 --2- 172.21.15.148:0/1184617774 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 0x7f4f480cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 --2- 172.21.15.148:0/1184617774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4815cd50 0x7f4f480c9f90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/1184617774 >> 172.21.15.148:0/1184617774 conn(0x7f4f4806a640 msgr2=0x7f4f48065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/1184617774 shutdown_connections 2023-10-30T13:10:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/1184617774 wait complete. 2023-10-30T13:10:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 Processor -- start 2023-10-30T13:10:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.404+0000 7f4f4c8d5640 1 -- start start 2023-10-30T13:10:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4c8d5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 0x7f4f481ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4c8d5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f481538f0 0x7f4f48152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4c8d5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f4815cd50 0x7f4f48152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4c8d5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f481ef5c0 con 0x7f4f4815cd50 2023-10-30T13:10:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4c8d5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4815c730 con 0x7f4f481538f0 2023-10-30T13:10:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4c8d5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f480cea00 con 0x7f4f480cefa0 2023-10-30T13:10:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f4815cd50 0x7f4f48152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f4815cd50 0x7f4f48152b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50456/0 (socket says 172.21.15.148:50456) 2023-10-30T13:10:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4759e640 1 -- 172.21.15.148:0/97984816 learned_addr learned my addr 172.21.15.148:0/97984816 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f46d9d640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 0x7f4f481ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f3e59c640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f481538f0 0x7f4f48152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4759e640 1 -- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 msgr2=0x7f4f481ef9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4759e640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 0x7f4f481ef9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4759e640 1 -- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f481538f0 msgr2=0x7f4f48152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4759e640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f481538f0 0x7f4f48152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4759e640 1 -- 172.21.15.148:0/97984816 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30009a70 con 0x7f4f4815cd50 2023-10-30T13:10:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f3e59c640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f481538f0 0x7f4f48152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4759e640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f4815cd50 0x7f4f48152b60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000e930 tx=0x7f4f4000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f3ffff640 1 -- 172.21.15.148:0/97984816 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f400096b0 con 0x7f4f4815cd50 2023-10-30T13:10:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f3ffff640 1 -- 172.21.15.148:0/97984816 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4000c9a0 con 0x7f4f4815cd50 2023-10-30T13:10:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f3ffff640 1 -- 172.21.15.148:0/97984816 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4000cd30 con 0x7f4f4815cd50 2023-10-30T13:10:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4815aaa0 con 0x7f4f4815cd50 2023-10-30T13:10:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4815ad40 con 0x7f4f4815cd50 2023-10-30T13:10:32.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f481ef700 con 0x7f4f4815cd50 2023-10-30T13:10:32.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.408+0000 7f4f3ffff640 1 -- 172.21.15.148:0/97984816 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f40013470 con 0x7f4f4815cd50 2023-10-30T13:10:32.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.412+0000 7f4f3ffff640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4f280664a0 0x7f4f28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.412+0000 7f4f3ffff640 1 -- 172.21.15.148:0/97984816 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4f40083cd0 con 0x7f4f4815cd50 2023-10-30T13:10:32.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.412+0000 7f4f46d9d640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4f280664a0 0x7f4f28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.412+0000 7f4f46d9d640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4f280664a0 0x7f4f28068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005890 tx=0x7f4f38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:32.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.416+0000 7f4f3ffff640 1 -- 172.21.15.148:0/97984816 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f481ef700 con 0x7f4f4815cd50 2023-10-30T13:10:32.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.564+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4f481ef700 con 0x7f4f4815cd50 2023-10-30T13:10:32.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.660+0000 7f4f3ffff640 1 -- 172.21.15.148:0/97984816 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4f481ef700 con 0x7f4f4815cd50 2023-10-30T13:10:32.665 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'cephfs_metadata' created 2023-10-30T13:10:32.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4f280664a0 msgr2=0x7f4f28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4f280664a0 0x7f4f28068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005890 tx=0x7f4f38009040 comp rx=0 tx=0).stop 2023-10-30T13:10:32.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f4815cd50 msgr2=0x7f4f48152b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f4815cd50 0x7f4f48152b60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000e930 tx=0x7f4f4000edf0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 shutdown_connections 2023-10-30T13:10:32.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4f280664a0 0x7f4f28068960 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f4815cd50 0x7f4f48152b60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f481538f0 0x7f4f48152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 --2- 172.21.15.148:0/97984816 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4f480cefa0 0x7f4f481ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 >> 172.21.15.148:0/97984816 conn(0x7f4f4806a640 msgr2=0x7f4f480c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:32.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 shutdown_connections 2023-10-30T13:10:32.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.664+0000 7f4f4c8d5640 1 -- 172.21.15.148:0/97984816 wait complete. 2023-10-30T13:10:32.692 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-30T13:10:32.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1ca5446640 1 Processor -- start 2023-10-30T13:10:32.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1ca5446640 1 -- start start 2023-10-30T13:10:32.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1ca5446640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1ca00c5810 0x7f1ca00c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1ca5446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 0x7f1ca0161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1ca5446640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca00c6cb0 con 0x7f1ca00c6130 2023-10-30T13:10:32.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1ca5446640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca0071d90 con 0x7f1ca00c5810 2023-10-30T13:10:32.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1ca5446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca01585d0 con 0x7f1ca006a980 2023-10-30T13:10:32.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1c9effd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1ca00c5810 0x7f1ca00c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1c9effd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1ca00c5810 0x7f1ca00c5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50466/0 (socket says 172.21.15.148:50466) 2023-10-30T13:10:32.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.832+0000 7f1c9effd640 1 -- 172.21.15.148:0/1275906677 learned_addr learned my addr 172.21.15.148:0/1275906677 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:32.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.836+0000 7f1c9f7fe640 1 --2- 172.21.15.148:0/1275906677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 0x7f1ca0161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.836+0000 7f1c9effd640 1 -- 172.21.15.148:0/1275906677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 msgr2=0x7f1ca0161f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:32.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.836+0000 7f1c9effd640 1 --2- 172.21.15.148:0/1275906677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 0x7f1ca0161f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.836+0000 7f1c9effd640 1 -- 172.21.15.148:0/1275906677 >> v1:172.21.15.150:6789/0 conn(0x7f1ca00c6130 legacy=0x7f1ca006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.836+0000 7f1c9effd640 1 -- 172.21.15.148:0/1275906677 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca00c6620 con 0x7f1ca00c5810 2023-10-30T13:10:32.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.836+0000 7f1c9effd640 1 --2- 172.21.15.148:0/1275906677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c5810 0x7f1ca00c5bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009870 tx=0x7f1c9001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41efef31d6c2c30a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:32.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.836+0000 7f1c9dffb640 1 -- 172.21.15.148:0/1275906677 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c900ac640 con 0x7f1ca00c5810 2023-10-30T13:10:32.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.836+0000 7f1c9dffb640 1 -- 172.21.15.148:0/1275906677 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c900ac7a0 con 0x7f1ca00c5810 2023-10-30T13:10:32.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.836+0000 7f1c9dffb640 1 -- 172.21.15.148:0/1275906677 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c900acb30 con 0x7f1ca00c5810 2023-10-30T13:10:32.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 -- 172.21.15.148:0/1275906677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c5810 msgr2=0x7f1ca00c5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 --2- 172.21.15.148:0/1275906677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c5810 0x7f1ca00c5bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009870 tx=0x7f1c9001c400 comp rx=0 tx=0).stop 2023-10-30T13:10:32.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 -- 172.21.15.148:0/1275906677 shutdown_connections 2023-10-30T13:10:32.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 --2- 172.21.15.148:0/1275906677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 0x7f1ca0161f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 --2- 172.21.15.148:0/1275906677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c5810 0x7f1ca00c5bf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 -- 172.21.15.148:0/1275906677 >> 172.21.15.148:0/1275906677 conn(0x7f1ca00c7910 msgr2=0x7f1ca00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:32.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 -- 172.21.15.148:0/1275906677 shutdown_connections 2023-10-30T13:10:32.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 -- 172.21.15.148:0/1275906677 wait complete. 2023-10-30T13:10:32.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 Processor -- start 2023-10-30T13:10:32.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 -- start start 2023-10-30T13:10:32.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 0x7f1ca00c4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca00c5810 0x7f1ca01f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c6130 0x7f1ca006da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca006dfb0 con 0x7f1ca00c6130 2023-10-30T13:10:32.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca00c4070 con 0x7f1ca00c5810 2023-10-30T13:10:32.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1ca5446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca0147df0 con 0x7f1ca006a980 2023-10-30T13:10:32.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c6130 0x7f1ca006da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c6130 0x7f1ca006da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50476/0 (socket says 172.21.15.148:50476) 2023-10-30T13:10:32.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9f7fe640 1 -- 172.21.15.148:0/1070656384 learned_addr learned my addr 172.21.15.148:0/1070656384 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:32.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9e7fc640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca00c5810 0x7f1ca01f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9effd640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 0x7f1ca00c4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9f7fe640 1 -- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 msgr2=0x7f1ca00c4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9f7fe640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 0x7f1ca00c4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9f7fe640 1 -- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca00c5810 msgr2=0x7f1ca01f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:32.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9f7fe640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca00c5810 0x7f1ca01f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:32.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9f7fe640 1 -- 172.21.15.148:0/1070656384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c90009480 con 0x7f1ca00c6130 2023-10-30T13:10:32.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9e7fc640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca00c5810 0x7f1ca01f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:32.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.840+0000 7f1c9f7fe640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c6130 0x7f1ca006da70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c98007e90 tx=0x7f1c9800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:32.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1c877fe640 1 -- 172.21.15.148:0/1070656384 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98014420 con 0x7f1ca00c6130 2023-10-30T13:10:32.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1c877fe640 1 -- 172.21.15.148:0/1070656384 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c98014580 con 0x7f1ca00c6130 2023-10-30T13:10:32.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca00c3d40 con 0x7f1ca00c6130 2023-10-30T13:10:32.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1c877fe640 1 -- 172.21.15.148:0/1070656384 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98014930 con 0x7f1ca00c6130 2023-10-30T13:10:32.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca01588f0 con 0x7f1ca00c6130 2023-10-30T13:10:32.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca00c4220 con 0x7f1ca00c6130 2023-10-30T13:10:32.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1c877fe640 1 -- 172.21.15.148:0/1070656384 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c98014a90 con 0x7f1ca00c6130 2023-10-30T13:10:32.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1c877fe640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c8c0664a0 0x7f1c8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:32.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1c877fe640 1 -- 172.21.15.148:0/1070656384 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f1c98084700 con 0x7f1ca00c6130 2023-10-30T13:10:32.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.844+0000 7f1c9effd640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c8c0664a0 0x7f1c8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:32.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.848+0000 7f1c9effd640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c8c0664a0 0x7f1c8c068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1c900ae000 tx=0x7f1c900a6070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:32.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:32.848+0000 7f1c877fe640 1 -- 172.21.15.148:0/1070656384 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ca00c4220 con 0x7f1ca00c6130 2023-10-30T13:10:33.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.000+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1ca00c41e0 con 0x7f1ca00c6130 2023-10-30T13:10:33.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.664+0000 7f1c877fe640 1 -- 172.21.15.148:0/1070656384 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1c98051c30 con 0x7f1ca00c6130 2023-10-30T13:10:33.669 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'cephfs_data' created 2023-10-30T13:10:33.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c8c0664a0 msgr2=0x7f1c8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:33.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c8c0664a0 0x7f1c8c068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1c900ae000 tx=0x7f1c900a6070 comp rx=0 tx=0).stop 2023-10-30T13:10:33.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c6130 msgr2=0x7f1ca006da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:33.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c6130 0x7f1ca006da70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c98007e90 tx=0x7f1c9800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 shutdown_connections 2023-10-30T13:10:33.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c8c0664a0 0x7f1c8c068960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ca00c6130 0x7f1ca006da70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ca00c5810 0x7f1ca01f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 --2- 172.21.15.148:0/1070656384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1ca006a980 0x7f1ca00c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 >> 172.21.15.148:0/1070656384 conn(0x7f1ca00c7910 msgr2=0x7f1ca0070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:33.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 shutdown_connections 2023-10-30T13:10:33.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.668+0000 7f1ca5446640 1 -- 172.21.15.148:0/1070656384 wait complete. 2023-10-30T13:10:33.715 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-30T13:10:33.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f893ccd6640 1 Processor -- start 2023-10-30T13:10:33.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f893ccd6640 1 -- start start 2023-10-30T13:10:33.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f893ccd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938159930 0x7f89380cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:33.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f893ccd6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89380cf6e0 0x7f893815d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:33.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f893ccd6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893815a920 con 0x7f8938153980 2023-10-30T13:10:33.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f893ccd6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89380cb930 con 0x7f89380cf6e0 2023-10-30T13:10:33.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f893ccd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8938159dc0 con 0x7f8938159930 2023-10-30T13:10:33.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f8936d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89380cf6e0 0x7f893815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:33.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f8936d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89380cf6e0 0x7f893815d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50492/0 (socket says 172.21.15.148:50492) 2023-10-30T13:10:33.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f8936d9d640 1 -- 172.21.15.148:0/4057316502 learned_addr learned my addr 172.21.15.148:0/4057316502 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:33.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f892dd9b640 1 --2- 172.21.15.148:0/4057316502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938159930 0x7f89380cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:33.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f8936d9d640 1 -- 172.21.15.148:0/4057316502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938159930 msgr2=0x7f89380cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:33.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f8936d9d640 1 --2- 172.21.15.148:0/4057316502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938159930 0x7f89380cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f8936d9d640 1 -- 172.21.15.148:0/4057316502 >> v1:172.21.15.150:6789/0 conn(0x7f8938153980 legacy=0x7f89380caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:33.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.856+0000 7f8936d9d640 1 -- 172.21.15.148:0/4057316502 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893815a080 con 0x7f89380cf6e0 2023-10-30T13:10:33.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f8936d9d640 1 --2- 172.21.15.148:0/4057316502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 0x7f893815d570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8930009870 tx=0x7f893001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e9dd1135ba4627 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:33.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f8935d9b640 1 -- 172.21.15.148:0/4057316502 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300ab620 con 0x7f89380cf6e0 2023-10-30T13:10:33.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f8935d9b640 1 -- 172.21.15.148:0/4057316502 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89300ab780 con 0x7f89380cf6e0 2023-10-30T13:10:33.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f8935d9b640 1 -- 172.21.15.148:0/4057316502 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300abb10 con 0x7f89380cf6e0 2023-10-30T13:10:33.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f893ccd6640 1 -- 172.21.15.148:0/4057316502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 msgr2=0x7f893815d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:33.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f893ccd6640 1 --2- 172.21.15.148:0/4057316502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 0x7f893815d570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8930009870 tx=0x7f893001c510 comp rx=0 tx=0).stop 2023-10-30T13:10:33.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f893ccd6640 1 -- 172.21.15.148:0/4057316502 shutdown_connections 2023-10-30T13:10:33.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f893ccd6640 1 --2- 172.21.15.148:0/4057316502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 0x7f893815d570 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f893ccd6640 1 --2- 172.21.15.148:0/4057316502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938159930 0x7f89380cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f893ccd6640 1 -- 172.21.15.148:0/4057316502 >> 172.21.15.148:0/4057316502 conn(0x7f893806c0f0 msgr2=0x7f893806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:33.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f893ccd6640 1 -- 172.21.15.148:0/4057316502 shutdown_connections 2023-10-30T13:10:33.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.860+0000 7f893ccd6640 1 -- 172.21.15.148:0/4057316502 wait complete. 2023-10-30T13:10:33.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 Processor -- start 2023-10-30T13:10:33.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 -- start start 2023-10-30T13:10:33.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 0x7f89380c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:33.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938153980 0x7f89380c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:33.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f893815b820 0x7f8938152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:33.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893815aaf0 con 0x7f89380cf6e0 2023-10-30T13:10:33.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89380c6530 con 0x7f893815b820 2023-10-30T13:10:33.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89380b32c0 con 0x7f8938153980 2023-10-30T13:10:33.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 0x7f89380c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:33.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 0x7f89380c5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50502/0 (socket says 172.21.15.148:50502) 2023-10-30T13:10:33.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893659c640 1 -- 172.21.15.148:0/1891380189 learned_addr learned my addr 172.21.15.148:0/1891380189 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:33.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f892dd9b640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938153980 0x7f89380c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:33.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f8936d9d640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f893815b820 0x7f8938152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:33.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893659c640 1 -- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938153980 msgr2=0x7f89380c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:33.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893659c640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938153980 0x7f89380c79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893659c640 1 -- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f893815b820 msgr2=0x7f8938152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:33.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893659c640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f893815b820 0x7f8938152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:33.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893659c640 1 -- 172.21.15.148:0/1891380189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300094d0 con 0x7f89380cf6e0 2023-10-30T13:10:33.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893659c640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 0x7f89380c5ff0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f892800c530 tx=0x7f892800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:33.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f892f7fe640 1 -- 172.21.15.148:0/1891380189 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8928005220 con 0x7f89380cf6e0 2023-10-30T13:10:33.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f892f7fe640 1 -- 172.21.15.148:0/1891380189 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8928005380 con 0x7f89380cf6e0 2023-10-30T13:10:33.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f892f7fe640 1 -- 172.21.15.148:0/1891380189 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8928007270 con 0x7f89380cf6e0 2023-10-30T13:10:33.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89380b35a0 con 0x7f89380cf6e0 2023-10-30T13:10:33.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893815b020 con 0x7f89380cf6e0 2023-10-30T13:10:33.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.864+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893804e170 con 0x7f89380cf6e0 2023-10-30T13:10:33.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.868+0000 7f892f7fe640 1 -- 172.21.15.148:0/1891380189 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f89280073d0 con 0x7f89380cf6e0 2023-10-30T13:10:33.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.868+0000 7f892f7fe640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664a0 0x7f8918068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:33.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.868+0000 7f892f7fe640 1 -- 172.21.15.148:0/1891380189 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f89280835e0 con 0x7f89380cf6e0 2023-10-30T13:10:33.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.868+0000 7f892dd9b640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664a0 0x7f8918068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:33.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.868+0000 7f892dd9b640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664a0 0x7f8918068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8938159790 tx=0x7f89200098a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:33.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:33.872+0000 7f892f7fe640 1 -- 172.21.15.148:0/1891380189 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8928050b60 con 0x7f89380cf6e0 2023-10-30T13:10:34.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.052+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f89380caa80 con 0x7f89380cf6e0 2023-10-30T13:10:34.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f892f7fe640 1 -- 172.21.15.148:0/1891380189 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8928054180 con 0x7f89380cf6e0 2023-10-30T13:10:34.679 INFO:teuthology.orchestra.run.smithi148.stderr:new fs with metadata pool 2 and data pool 3 2023-10-30T13:10:34.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664a0 msgr2=0x7f8918068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:34.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664a0 0x7f8918068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8938159790 tx=0x7f89200098a0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 msgr2=0x7f89380c5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:34.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 0x7f89380c5ff0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f892800c530 tx=0x7f892800c9f0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 shutdown_connections 2023-10-30T13:10:34.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664a0 0x7f8918068960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f893815b820 0x7f8938152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8938153980 0x7f89380c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 --2- 172.21.15.148:0/1891380189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89380cf6e0 0x7f89380c5ff0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 >> 172.21.15.148:0/1891380189 conn(0x7f893806c0f0 msgr2=0x7f89380c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:34.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 shutdown_connections 2023-10-30T13:10:34.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.676+0000 7f893ccd6640 1 -- 172.21.15.148:0/1891380189 wait complete. 2023-10-30T13:10:34.702 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:34.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.832+0000 7f25f36d6640 1 Processor -- start 2023-10-30T13:10:34.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f36d6640 1 -- start start 2023-10-30T13:10:34.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f36d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 0x7f25ec15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:34.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f36d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f25ec0cf160 0x7f25ec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:34.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f36d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec0cb800 con 0x7f25ec1538a0 2023-10-30T13:10:34.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f36d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec0cb330 con 0x7f25ec0cf160 2023-10-30T13:10:34.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f36d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0cad70 con 0x7f25ec15d6f0 2023-10-30T13:10:34.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f1c73640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f25ec0cf160 0x7f25ec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:34.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f0c71640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 0x7f25ec15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:34.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f1472640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f25ec1538a0 0x7f25ec0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:37296/0 (socket says 172.21.15.148:37296) 2023-10-30T13:10:34.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f1472640 1 -- 172.21.15.148:0/471470177 learned_addr learned my addr 172.21.15.148:0/471470177 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:34.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f1c73640 1 -- 172.21.15.148:0/471470177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 msgr2=0x7f25ec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:34.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f1c73640 1 --2- 172.21.15.148:0/471470177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 0x7f25ec15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f1c73640 1 -- 172.21.15.148:0/471470177 >> v1:172.21.15.150:6789/0 conn(0x7f25ec1538a0 legacy=0x7f25ec0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:34.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.836+0000 7f25f1c73640 1 -- 172.21.15.148:0/471470177 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0cb050 con 0x7f25ec0cf160 2023-10-30T13:10:34.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25f1c73640 1 --2- 172.21.15.148:0/471470177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0cf160 0x7f25ec0cf590 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f25e4009d90 tx=0x7f25e401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68fa657dfa528459 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:34.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25e3fff640 1 -- 172.21.15.148:0/471470177 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e40af600 con 0x7f25ec0cf160 2023-10-30T13:10:34.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25e3fff640 1 -- 172.21.15.148:0/471470177 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e40ab050 con 0x7f25ec0cf160 2023-10-30T13:10:34.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25e3fff640 1 -- 172.21.15.148:0/471470177 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e40af980 con 0x7f25ec0cf160 2023-10-30T13:10:34.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25f36d6640 1 -- 172.21.15.148:0/471470177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0cf160 msgr2=0x7f25ec0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:34.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25f36d6640 1 --2- 172.21.15.148:0/471470177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0cf160 0x7f25ec0cf590 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f25e4009d90 tx=0x7f25e401f130 comp rx=0 tx=0).stop 2023-10-30T13:10:34.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25f36d6640 1 -- 172.21.15.148:0/471470177 shutdown_connections 2023-10-30T13:10:34.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25f36d6640 1 --2- 172.21.15.148:0/471470177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 0x7f25ec15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25f36d6640 1 --2- 172.21.15.148:0/471470177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0cf160 0x7f25ec0cf590 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25f36d6640 1 -- 172.21.15.148:0/471470177 >> 172.21.15.148:0/471470177 conn(0x7f25ec06bfe0 msgr2=0x7f25ec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:34.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25f36d6640 1 -- 172.21.15.148:0/471470177 shutdown_connections 2023-10-30T13:10:34.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.840+0000 7f25f36d6640 1 -- 172.21.15.148:0/471470177 wait complete. 2023-10-30T13:10:34.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 Processor -- start 2023-10-30T13:10:34.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 -- start start 2023-10-30T13:10:34.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25ec0cf160 0x7f25ec0c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:34.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec1538a0 0x7f25ec146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:34.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 0x7f25ec144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:34.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0cd4c0 con 0x7f25ec0cf160 2023-10-30T13:10:34.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0cd950 con 0x7f25ec1538a0 2023-10-30T13:10:34.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0c9830 con 0x7f25ec15d6f0 2023-10-30T13:10:34.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1472640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25ec0cf160 0x7f25ec0c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:34.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1472640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25ec0cf160 0x7f25ec0c4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50512/0 (socket says 172.21.15.148:50512) 2023-10-30T13:10:34.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1472640 1 -- 172.21.15.148:0/1044157445 learned_addr learned my addr 172.21.15.148:0/1044157445 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:34.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f0c71640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec1538a0 0x7f25ec146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:34.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1c73640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 0x7f25ec144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:34.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1472640 1 -- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 msgr2=0x7f25ec144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:34.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1472640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 0x7f25ec144f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1472640 1 -- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec1538a0 msgr2=0x7f25ec146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:34.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1472640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec1538a0 0x7f25ec146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:34.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1472640 1 -- 172.21.15.148:0/1044157445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e4009a70 con 0x7f25ec0cf160 2023-10-30T13:10:34.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f1472640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25ec0cf160 0x7f25ec0c4c50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f25e800c500 tx=0x7f25e800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:34.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25e1ffb640 1 -- 172.21.15.148:0/1044157445 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8005cf0 con 0x7f25ec0cf160 2023-10-30T13:10:34.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25e1ffb640 1 -- 172.21.15.148:0/1044157445 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e8005e50 con 0x7f25ec0cf160 2023-10-30T13:10:34.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25e1ffb640 1 -- 172.21.15.148:0/1044157445 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e80072f0 con 0x7f25ec0cf160 2023-10-30T13:10:34.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec0b3240 con 0x7f25ec0cf160 2023-10-30T13:10:34.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec0b3800 con 0x7f25ec0cf160 2023-10-30T13:10:34.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.844+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec0b2a50 con 0x7f25ec0cf160 2023-10-30T13:10:34.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.848+0000 7f25e1ffb640 1 -- 172.21.15.148:0/1044157445 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25e8007450 con 0x7f25ec0cf160 2023-10-30T13:10:34.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.848+0000 7f25e1ffb640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25d00664a0 0x7f25d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:34.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.848+0000 7f25e1ffb640 1 -- 172.21.15.148:0/1044157445 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25e80833e0 con 0x7f25ec0cf160 2023-10-30T13:10:34.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.848+0000 7f25f0c71640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25d00664a0 0x7f25d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:34.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.848+0000 7f25f0c71640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25d00664a0 0x7f25d0068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f25ec159610 tx=0x7f25dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:34.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:34.852+0000 7f25e1ffb640 1 -- 172.21.15.148:0/1044157445 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25e8050900 con 0x7f25ec0cf160 2023-10-30T13:10:35.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25ec0b39f0 con 0x7f25ec0cf160 2023-10-30T13:10:35.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25e1ffb640 1 -- 172.21.15.148:0/1044157445 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16554 (secure 0 0 0) 0x7f25e8053f20 con 0x7f25ec0cf160 2023-10-30T13:10:35.009 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:35.009 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:34.057890+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:35.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25d00664a0 msgr2=0x7f25d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25d00664a0 0x7f25d0068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f25ec159610 tx=0x7f25dc009190 comp rx=0 tx=0).stop 2023-10-30T13:10:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25ec0cf160 msgr2=0x7f25ec0c4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25ec0cf160 0x7f25ec0c4c50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f25e800c500 tx=0x7f25e800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 shutdown_connections 2023-10-30T13:10:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25d00664a0 0x7f25d0068960 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25ec15d6f0 0x7f25ec144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec1538a0 0x7f25ec146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 --2- 172.21.15.148:0/1044157445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25ec0cf160 0x7f25ec0c4c50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.004+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 >> 172.21.15.148:0/1044157445 conn(0x7f25ec06bfe0 msgr2=0x7f25ec065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:35.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.008+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 shutdown_connections 2023-10-30T13:10:35.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.008+0000 7f25f36d6640 1 -- 172.21.15.148:0/1044157445 wait complete. 2023-10-30T13:10:35.030 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:35.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f657e10f640 1 Processor -- start 2023-10-30T13:10:35.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f657e10f640 1 -- start start 2023-10-30T13:10:35.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f657e10f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f65781538a0 0x7f657815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f657e10f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65780cf160 0x7f65780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f657e10f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65780cb800 con 0x7f65780ca0e0 2023-10-30T13:10:35.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f657e10f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657815dc30 con 0x7f65781538a0 2023-10-30T13:10:35.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f657e10f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65780cb330 con 0x7f65780cf160 2023-10-30T13:10:35.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f65777fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f65781538a0 0x7f657815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f65777fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f65781538a0 0x7f657815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50518/0 (socket says 172.21.15.148:50518) 2023-10-30T13:10:35.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f65777fe640 1 -- 172.21.15.148:0/3335924658 learned_addr learned my addr 172.21.15.148:0/3335924658 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:35.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f6577fff640 1 --2- 172.21.15.148:0/3335924658 >> v2:172.21.15.150:3300/0 conn(0x7f65780cf160 0x7f65780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f65777fe640 1 -- 172.21.15.148:0/3335924658 >> v2:172.21.15.150:3300/0 conn(0x7f65780cf160 msgr2=0x7f65780cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:35.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f65777fe640 1 --2- 172.21.15.148:0/3335924658 >> v2:172.21.15.150:3300/0 conn(0x7f65780cf160 0x7f65780cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f65777fe640 1 -- 172.21.15.148:0/3335924658 >> v1:172.21.15.150:6789/0 conn(0x7f65780ca0e0 legacy=0x7f657815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.164+0000 7f65777fe640 1 -- 172.21.15.148:0/3335924658 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65780cad70 con 0x7f65781538a0 2023-10-30T13:10:35.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f65777fe640 1 --2- 172.21.15.148:0/3335924658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65781538a0 0x7f657815d6f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f65780b5330 tx=0x7f656801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c37174378d4f4aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:35.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f65767fc640 1 -- 172.21.15.148:0/3335924658 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65680a89f0 con 0x7f65781538a0 2023-10-30T13:10:35.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f65767fc640 1 -- 172.21.15.148:0/3335924658 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65680a8b50 con 0x7f65781538a0 2023-10-30T13:10:35.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f65767fc640 1 -- 172.21.15.148:0/3335924658 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65680b0440 con 0x7f65781538a0 2023-10-30T13:10:35.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f657e10f640 1 -- 172.21.15.148:0/3335924658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65781538a0 msgr2=0x7f657815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f657e10f640 1 --2- 172.21.15.148:0/3335924658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65781538a0 0x7f657815d6f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f65780b5330 tx=0x7f656801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f657e10f640 1 -- 172.21.15.148:0/3335924658 shutdown_connections 2023-10-30T13:10:35.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f657e10f640 1 --2- 172.21.15.148:0/3335924658 >> v2:172.21.15.150:3300/0 conn(0x7f65780cf160 0x7f65780cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f657e10f640 1 --2- 172.21.15.148:0/3335924658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65781538a0 0x7f657815d6f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f657e10f640 1 -- 172.21.15.148:0/3335924658 >> 172.21.15.148:0/3335924658 conn(0x7f657806bfe0 msgr2=0x7f657806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:35.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f657e10f640 1 -- 172.21.15.148:0/3335924658 shutdown_connections 2023-10-30T13:10:35.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.168+0000 7f657e10f640 1 -- 172.21.15.148:0/3335924658 wait complete. 2023-10-30T13:10:35.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 Processor -- start 2023-10-30T13:10:35.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 -- start start 2023-10-30T13:10:35.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65780ca0e0 0x7f657815cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65780cf160 0x7f6578158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65781538a0 0x7f65781e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65781595d0 con 0x7f65780ca0e0 2023-10-30T13:10:35.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657815d7a0 con 0x7f65780cf160 2023-10-30T13:10:35.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657815ba60 con 0x7f65781538a0 2023-10-30T13:10:35.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f65777fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65780ca0e0 0x7f657815cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f65777fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65780ca0e0 0x7f657815cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50524/0 (socket says 172.21.15.148:50524) 2023-10-30T13:10:35.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f65777fe640 1 -- 172.21.15.148:0/3722705025 learned_addr learned my addr 172.21.15.148:0/3722705025 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:35.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f6576ffd640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65780cf160 0x7f6578158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f6577fff640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65781538a0 0x7f65781e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f65777fe640 1 -- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65781538a0 msgr2=0x7f65781e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f65777fe640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65781538a0 0x7f65781e3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f65777fe640 1 -- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65780cf160 msgr2=0x7f6578158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f65777fe640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65780cf160 0x7f6578158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f65777fe640 1 -- 172.21.15.148:0/3722705025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680094d0 con 0x7f65780ca0e0 2023-10-30T13:10:35.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f65777fe640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65780ca0e0 0x7f657815cdd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f656801b900 tx=0x7f656801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:35.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f655ffff640 1 -- 172.21.15.148:0/3722705025 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65680ba600 con 0x7f65780ca0e0 2023-10-30T13:10:35.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f655ffff640 1 -- 172.21.15.148:0/3722705025 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65680ba760 con 0x7f65780ca0e0 2023-10-30T13:10:35.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f655ffff640 1 -- 172.21.15.148:0/3722705025 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65680bab10 con 0x7f65780ca0e0 2023-10-30T13:10:35.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657815bd40 con 0x7f65780ca0e0 2023-10-30T13:10:35.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65781e58d0 con 0x7f65780ca0e0 2023-10-30T13:10:35.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.172+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65780b2a50 con 0x7f65780ca0e0 2023-10-30T13:10:35.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.176+0000 7f655ffff640 1 -- 172.21.15.148:0/3722705025 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65680bac70 con 0x7f65780ca0e0 2023-10-30T13:10:35.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.176+0000 7f655ffff640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65640664a0 0x7f6564068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.176+0000 7f655ffff640 1 -- 172.21.15.148:0/3722705025 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6568122480 con 0x7f65780ca0e0 2023-10-30T13:10:35.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.176+0000 7f6576ffd640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65640664a0 0x7f6564068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.176+0000 7f6576ffd640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65640664a0 0x7f6564068960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65780c58d0 tx=0x7f656c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:35.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.180+0000 7f655ffff640 1 -- 172.21.15.148:0/3722705025 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65680ef8f0 con 0x7f65780ca0e0 2023-10-30T13:10:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.332+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f657804e170 con 0x7f65780ca0e0 2023-10-30T13:10:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.332+0000 7f655ffff640 1 -- 172.21.15.148:0/3722705025 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f65680f2f10 con 0x7f65780ca0e0 2023-10-30T13:10:35.337 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:35.337 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:35.311442+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65640664a0 msgr2=0x7f6564068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65640664a0 0x7f6564068960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65780c58d0 tx=0x7f656c009040 comp rx=0 tx=0).stop 2023-10-30T13:10:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65780ca0e0 msgr2=0x7f657815cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65780ca0e0 0x7f657815cdd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f656801b900 tx=0x7f656801ba90 comp rx=0 tx=0).stop 2023-10-30T13:10:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 shutdown_connections 2023-10-30T13:10:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65640664a0 0x7f6564068960 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65781538a0 0x7f65781e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65780cf160 0x7f6578158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 --2- 172.21.15.148:0/3722705025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65780ca0e0 0x7f657815cdd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 >> 172.21.15.148:0/3722705025 conn(0x7f657806bfe0 msgr2=0x7f6578065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 shutdown_connections 2023-10-30T13:10:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.336+0000 7f657e10f640 1 -- 172.21.15.148:0/3722705025 wait complete. 2023-10-30T13:10:35.358 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-30T13:10:35.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.492+0000 7ff8d5cdc640 1 Processor -- start 2023-10-30T13:10:35.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.492+0000 7ff8d5cdc640 1 -- start start 2023-10-30T13:10:35.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8d5cdc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff8d00cb4c0 0x7ff8d015a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8d5cdc640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d015a6b0 con 0x7ff8d0153c20 2023-10-30T13:10:35.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8d5cdc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d015d730 con 0x7ff8d0153840 2023-10-30T13:10:35.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8d5cdc640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d015dbc0 con 0x7ff8d00cb4c0 2023-10-30T13:10:35.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8c7fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff8d0153c20 0x7ff8d00cef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34290/0 (socket says 172.21.15.148:34290) 2023-10-30T13:10:35.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8c7fff640 1 -- 172.21.15.148:0/2809103713 learned_addr learned my addr 172.21.15.148:0/2809103713 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:35.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8d54db640 1 --2- 172.21.15.148:0/2809103713 >> v2:172.21.15.148:3300/0 conn(0x7ff8d00cb4c0 0x7ff8d015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8c77fe640 1 -- 172.21.15.148:0/2809103713 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879822042 0 0) 0x7ff8c0001120 con 0x7ff8d0153c20 2023-10-30T13:10:35.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8c77fe640 1 -- 172.21.15.148:0/2809103713 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b8003530 con 0x7ff8d0153c20 2023-10-30T13:10:35.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8d54db640 1 -- 172.21.15.148:0/2809103713 >> v1:172.21.15.150:6789/0 conn(0x7ff8d0153840 legacy=0x7ff8d00c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8d54db640 1 -- 172.21.15.148:0/2809103713 >> v1:172.21.15.148:6789/0 conn(0x7ff8d0153c20 legacy=0x7ff8d00cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.496+0000 7ff8d54db640 1 -- 172.21.15.148:0/2809103713 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d00ca8f0 con 0x7ff8d00cb4c0 2023-10-30T13:10:35.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d54db640 1 --2- 172.21.15.148:0/2809103713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d00cb4c0 0x7ff8d015a170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8009d90 tx=0x7ff8c801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d66e923490cfefc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:35.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8c77fe640 1 -- 172.21.15.148:0/2809103713 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c80af600 con 0x7ff8d00cb4c0 2023-10-30T13:10:35.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8c77fe640 1 -- 172.21.15.148:0/2809103713 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c80ab050 con 0x7ff8d00cb4c0 2023-10-30T13:10:35.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8c77fe640 1 -- 172.21.15.148:0/2809103713 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c80af980 con 0x7ff8d00cb4c0 2023-10-30T13:10:35.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/2809103713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d00cb4c0 msgr2=0x7ff8d015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d5cdc640 1 --2- 172.21.15.148:0/2809103713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d00cb4c0 0x7ff8d015a170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8009d90 tx=0x7ff8c801f130 comp rx=0 tx=0).stop 2023-10-30T13:10:35.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/2809103713 shutdown_connections 2023-10-30T13:10:35.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d5cdc640 1 --2- 172.21.15.148:0/2809103713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d00cb4c0 0x7ff8d015a170 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/2809103713 >> 172.21.15.148:0/2809103713 conn(0x7ff8d006be40 msgr2=0x7ff8d006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:35.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/2809103713 shutdown_connections 2023-10-30T13:10:35.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/2809103713 wait complete. 2023-10-30T13:10:35.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d5cdc640 1 Processor -- start 2023-10-30T13:10:35.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.500+0000 7ff8d5cdc640 1 -- start start 2023-10-30T13:10:35.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d5cdc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00cb4c0 0x7ff8d01ef700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d5cdc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d0153840 0x7ff8d00c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d5cdc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d0153c20 0x7ff8d0157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d5cdc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00c62e0 con 0x7ff8d0153c20 2023-10-30T13:10:35.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d5cdc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01595a0 con 0x7ff8d0153840 2023-10-30T13:10:35.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d5cdc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00c8130 con 0x7ff8d00cb4c0 2023-10-30T13:10:35.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d54db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d0153c20 0x7ff8d0157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d54db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d0153c20 0x7ff8d0157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50544/0 (socket says 172.21.15.148:50544) 2023-10-30T13:10:35.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d54db640 1 -- 172.21.15.148:0/109401403 learned_addr learned my addr 172.21.15.148:0/109401403 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:35.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d4cda640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00cb4c0 0x7ff8d01ef700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8c7fff640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d0153840 0x7ff8d00c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d54db640 1 -- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00cb4c0 msgr2=0x7ff8d01ef700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d54db640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00cb4c0 0x7ff8d01ef700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d54db640 1 -- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d0153840 msgr2=0x7ff8d00c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:35.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d54db640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d0153840 0x7ff8d00c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:35.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d54db640 1 -- 172.21.15.148:0/109401403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c8009a70 con 0x7ff8d0153c20 2023-10-30T13:10:35.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d54db640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d0153c20 0x7ff8d0157520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80b2040 tx=0x7ff8c801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:35.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8c57fa640 1 -- 172.21.15.148:0/109401403 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8009450 con 0x7ff8d0153c20 2023-10-30T13:10:35.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8c57fa640 1 -- 172.21.15.148:0/109401403 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c80b9410 con 0x7ff8d0153c20 2023-10-30T13:10:35.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d00c8410 con 0x7ff8d0153c20 2023-10-30T13:10:35.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8c57fa640 1 -- 172.21.15.148:0/109401403 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c80b97c0 con 0x7ff8d0153c20 2023-10-30T13:10:35.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d0159ad0 con 0x7ff8d0153c20 2023-10-30T13:10:35.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d0159cc0 con 0x7ff8d0153c20 2023-10-30T13:10:35.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.504+0000 7ff8c57fa640 1 -- 172.21.15.148:0/109401403 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff8c80b6050 con 0x7ff8d0153c20 2023-10-30T13:10:35.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.508+0000 7ff8c57fa640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8bc0664d0 0x7ff8bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:35.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.508+0000 7ff8c57fa640 1 -- 172.21.15.148:0/109401403 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8c81223c0 con 0x7ff8d0153c20 2023-10-30T13:10:35.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.508+0000 7ff8d4cda640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8bc0664d0 0x7ff8bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:35.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.508+0000 7ff8d4cda640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8bc0664d0 0x7ff8bc068990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc000f70 tx=0x7ff8cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:35.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.512+0000 7ff8c57fa640 1 -- 172.21.15.148:0/109401403 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff8c80ef7a0 con 0x7ff8d0153c20 2023-10-30T13:10:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:35.696+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff8d004e9c0 con 0x7ff8d0153c20 2023-10-30T13:10:36.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.692+0000 7ff8c57fa640 1 -- 172.21.15.148:0/109401403 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff8c80f2dc0 con 0x7ff8d0153c20 2023-10-30T13:10:36.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.692+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8bc0664d0 msgr2=0x7ff8bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:36.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.692+0000 7ff8d5cdc640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8bc0664d0 0x7ff8bc068990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc000f70 tx=0x7ff8cc009040 comp rx=0 tx=0).stop 2023-10-30T13:10:36.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.692+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d0153c20 msgr2=0x7ff8d0157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:36.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.692+0000 7ff8d5cdc640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d0153c20 0x7ff8d0157520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80b2040 tx=0x7ff8c801bed0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.696+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 shutdown_connections 2023-10-30T13:10:36.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.696+0000 7ff8d5cdc640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8bc0664d0 0x7ff8bc068990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.696+0000 7ff8d5cdc640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8d0153c20 0x7ff8d0157520 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.696+0000 7ff8d5cdc640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d0153840 0x7ff8d00c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.696+0000 7ff8d5cdc640 1 --2- 172.21.15.148:0/109401403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8d00cb4c0 0x7ff8d01ef700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.696+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 >> 172.21.15.148:0/109401403 conn(0x7ff8d006be40 msgr2=0x7ff8d01525c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:36.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.696+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 shutdown_connections 2023-10-30T13:10:36.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.696+0000 7ff8d5cdc640 1 -- 172.21.15.148:0/109401403 wait complete. 2023-10-30T13:10:36.719 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300} 2023-10-30T13:10:36.720 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-30T13:10:36.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8bed780640 1 Processor -- start 2023-10-30T13:10:36.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8bed780640 1 -- start start 2023-10-30T13:10:36.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8bed780640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8be8153980 0x7f8be80caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:36.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8bed780640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8be80cb1e0 0x7f8be815d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:36.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8bed780640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be815a920 con 0x7f8be8159930 2023-10-30T13:10:36.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8bed780640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be80cb930 con 0x7f8be8153980 2023-10-30T13:10:36.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8bed780640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be8159dc0 con 0x7f8be80cb1e0 2023-10-30T13:10:36.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8be6ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8be8153980 0x7f8be80caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:36.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8be6ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8be8153980 0x7f8be80caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50550/0 (socket says 172.21.15.148:50550) 2023-10-30T13:10:36.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8be6ffd640 1 -- 172.21.15.148:0/2415457436 learned_addr learned my addr 172.21.15.148:0/2415457436 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:36.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8be77fe640 1 --2- 172.21.15.148:0/2415457436 >> v2:172.21.15.150:3300/0 conn(0x7f8be80cb1e0 0x7f8be815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:36.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8be6ffd640 1 -- 172.21.15.148:0/2415457436 >> v2:172.21.15.150:3300/0 conn(0x7f8be80cb1e0 msgr2=0x7f8be815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:36.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8be6ffd640 1 --2- 172.21.15.148:0/2415457436 >> v2:172.21.15.150:3300/0 conn(0x7f8be80cb1e0 0x7f8be815d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8be6ffd640 1 -- 172.21.15.148:0/2415457436 >> v1:172.21.15.150:6789/0 conn(0x7f8be8159930 legacy=0x7f8be80cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:36.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.860+0000 7f8be6ffd640 1 -- 172.21.15.148:0/2415457436 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be815a080 con 0x7f8be8153980 2023-10-30T13:10:36.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8be6ffd640 1 --2- 172.21.15.148:0/2415457436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be8153980 0x7f8be80caca0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8be80b1cf0 tx=0x7f8bd801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e3c2dc6a8eeb6f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:36.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8be67fc640 1 -- 172.21.15.148:0/2415457436 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80ac600 con 0x7f8be8153980 2023-10-30T13:10:36.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8be67fc640 1 -- 172.21.15.148:0/2415457436 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd80a8050 con 0x7f8be8153980 2023-10-30T13:10:36.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8be67fc640 1 -- 172.21.15.148:0/2415457436 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80ac980 con 0x7f8be8153980 2023-10-30T13:10:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 -- 172.21.15.148:0/2415457436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be8153980 msgr2=0x7f8be80caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 --2- 172.21.15.148:0/2415457436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be8153980 0x7f8be80caca0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8be80b1cf0 tx=0x7f8bd801c490 comp rx=0 tx=0).stop 2023-10-30T13:10:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 -- 172.21.15.148:0/2415457436 shutdown_connections 2023-10-30T13:10:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 --2- 172.21.15.148:0/2415457436 >> v2:172.21.15.150:3300/0 conn(0x7f8be80cb1e0 0x7f8be815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 --2- 172.21.15.148:0/2415457436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be8153980 0x7f8be80caca0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 -- 172.21.15.148:0/2415457436 >> 172.21.15.148:0/2415457436 conn(0x7f8be806c0f0 msgr2=0x7f8be806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:36.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 -- 172.21.15.148:0/2415457436 shutdown_connections 2023-10-30T13:10:36.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 -- 172.21.15.148:0/2415457436 wait complete. 2023-10-30T13:10:36.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 Processor -- start 2023-10-30T13:10:36.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.864+0000 7f8bed780640 1 -- start start 2023-10-30T13:10:36.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bed780640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80cb1e0 0x7f8be815ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:36.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bed780640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be8153980 0x7f8be80b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:36.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bed780640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be81f2840 0x7f8be8145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:36.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bed780640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be81f2420 con 0x7f8be80cb1e0 2023-10-30T13:10:36.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bed780640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be81f1b50 con 0x7f8be81f2840 2023-10-30T13:10:36.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bed780640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be815b370 con 0x7f8be8153980 2023-10-30T13:10:36.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be6ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80cb1e0 0x7f8be815ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:36.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be6ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80cb1e0 0x7f8be815ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50560/0 (socket says 172.21.15.148:50560) 2023-10-30T13:10:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be6ffd640 1 -- 172.21.15.148:0/3433169956 learned_addr learned my addr 172.21.15.148:0/3433169956 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be77fe640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be81f2840 0x7f8be8145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bdffff640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be8153980 0x7f8be80b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be6ffd640 1 -- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be8153980 msgr2=0x7f8be80b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:36.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be6ffd640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be8153980 0x7f8be80b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be6ffd640 1 -- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be81f2840 msgr2=0x7f8be8145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:36.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be6ffd640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be81f2840 0x7f8be8145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:36.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be6ffd640 1 -- 172.21.15.148:0/3433169956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd80094d0 con 0x7f8be80cb1e0 2023-10-30T13:10:36.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8be6ffd640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80cb1e0 0x7f8be815ba50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8bd801ab40 tx=0x7f8bd801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:36.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bdf7fe640 1 -- 172.21.15.148:0/3433169956 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80b7650 con 0x7f8be80cb1e0 2023-10-30T13:10:36.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bdf7fe640 1 -- 172.21.15.148:0/3433169956 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd80b77b0 con 0x7f8be80cb1e0 2023-10-30T13:10:36.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be815aaf0 con 0x7f8be80cb1e0 2023-10-30T13:10:36.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bdf7fe640 1 -- 172.21.15.148:0/3433169956 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80b7b20 con 0x7f8be80cb1e0 2023-10-30T13:10:36.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be815b090 con 0x7f8be80cb1e0 2023-10-30T13:10:36.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be81f2690 con 0x7f8be80cb1e0 2023-10-30T13:10:36.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.868+0000 7f8bdf7fe640 1 -- 172.21.15.148:0/3433169956 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8bd80b3070 con 0x7f8be80cb1e0 2023-10-30T13:10:36.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.872+0000 7f8bdf7fe640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bc80664a0 0x7f8bc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:36.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.872+0000 7f8bdf7fe640 1 -- 172.21.15.148:0/3433169956 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bd811e520 con 0x7f8be80cb1e0 2023-10-30T13:10:36.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.872+0000 7f8bdffff640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bc80664a0 0x7f8bc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:36.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.872+0000 7f8bdffff640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bc80664a0 0x7f8bc8068960 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8be815b700 tx=0x7f8bd0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:36.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:36.876+0000 7f8bdf7fe640 1 -- 172.21.15.148:0/3433169956 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8bd80eb990 con 0x7f8be80cb1e0 2023-10-30T13:10:37.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.056+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f8be804e170 con 0x7f8be80cb1e0 2023-10-30T13:10:37.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.692+0000 7f8bdf7fe640 1 -- 172.21.15.148:0/3433169956 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f8bd80eefb0 con 0x7f8be80cb1e0 2023-10-30T13:10:37.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bc80664a0 msgr2=0x7f8bc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:37.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bc80664a0 0x7f8bc8068960 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8be815b700 tx=0x7f8bd0008040 comp rx=0 tx=0).stop 2023-10-30T13:10:37.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80cb1e0 msgr2=0x7f8be815ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:37.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80cb1e0 0x7f8be815ba50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8bd801ab40 tx=0x7f8bd801acd0 comp rx=0 tx=0).stop 2023-10-30T13:10:37.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 shutdown_connections 2023-10-30T13:10:37.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bc80664a0 0x7f8bc8068960 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:37.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be81f2840 0x7f8be8145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:37.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be8153980 0x7f8be80b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:37.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 --2- 172.21.15.148:0/3433169956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80cb1e0 0x7f8be815ba50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:37.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 >> 172.21.15.148:0/3433169956 conn(0x7f8be806c0f0 msgr2=0x7f8be8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:37.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 shutdown_connections 2023-10-30T13:10:37.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.696+0000 7f8bed780640 1 -- 172.21.15.148:0/3433169956 wait complete. 2023-10-30T13:10:37.720 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-30T13:10:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.856+0000 7fda822a9640 1 Processor -- start 2023-10-30T13:10:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.856+0000 7fda822a9640 1 -- start start 2023-10-30T13:10:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda822a9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fda7c0c2870 0x7fda7c0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda822a9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c0c5910 con 0x7fda7c15d200 2023-10-30T13:10:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda822a9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c0c3520 con 0x7fda7c06fbd0 2023-10-30T13:10:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda822a9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7c0c2620 con 0x7fda7c0c2870 2023-10-30T13:10:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda80846640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fda7c0c2870 0x7fda7c0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda80846640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fda7c0c2870 0x7fda7c0c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50570/0 (socket says 172.21.15.148:50570) 2023-10-30T13:10:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda80846640 1 -- 172.21.15.148:0/331439861 learned_addr learned my addr 172.21.15.148:0/331439861 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda7bfff640 1 --1- 172.21.15.148:0/331439861 >> v1:172.21.15.148:6789/0 conn(0x7fda7c15d200 0x7fda7c0c39b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34292/0 (socket says 172.21.15.148:34292) 2023-10-30T13:10:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda80846640 1 -- 172.21.15.148:0/331439861 >> v1:172.21.15.150:6789/0 conn(0x7fda7c06fbd0 legacy=0x7fda7c15d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda80846640 1 -- 172.21.15.148:0/331439861 >> v1:172.21.15.148:6789/0 conn(0x7fda7c15d200 legacy=0x7fda7c0c39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda7affd640 1 -- 172.21.15.148:0/331439861 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741613032 0 0) 0x7fda7c0c5910 con 0x7fda7c15d200 2023-10-30T13:10:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda80846640 1 -- 172.21.15.148:0/331439861 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c070130 con 0x7fda7c0c2870 2023-10-30T13:10:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda80846640 1 --2- 172.21.15.148:0/331439861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c0c2870 0x7fda7c0c8ce0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fda74009e50 tx=0x7fda7401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe157c3df0cd7aaf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda7affd640 1 -- 172.21.15.148:0/331439861 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda740a58d0 con 0x7fda7c0c2870 2023-10-30T13:10:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda7affd640 1 -- 172.21.15.148:0/331439861 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda740a5a30 con 0x7fda7c0c2870 2023-10-30T13:10:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.860+0000 7fda7affd640 1 -- 172.21.15.148:0/331439861 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda740a5dc0 con 0x7fda7c0c2870 2023-10-30T13:10:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 -- 172.21.15.148:0/331439861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c0c2870 msgr2=0x7fda7c0c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 --2- 172.21.15.148:0/331439861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c0c2870 0x7fda7c0c8ce0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fda74009e50 tx=0x7fda7401c9e0 comp rx=0 tx=0).stop 2023-10-30T13:10:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 -- 172.21.15.148:0/331439861 shutdown_connections 2023-10-30T13:10:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 --2- 172.21.15.148:0/331439861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c0c2870 0x7fda7c0c8ce0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 -- 172.21.15.148:0/331439861 >> 172.21.15.148:0/331439861 conn(0x7fda7c0ccfe0 msgr2=0x7fda7c0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 -- 172.21.15.148:0/331439861 shutdown_connections 2023-10-30T13:10:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 -- 172.21.15.148:0/331439861 wait complete. 2023-10-30T13:10:37.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 Processor -- start 2023-10-30T13:10:37.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 -- start start 2023-10-30T13:10:37.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c06fbd0 0x7fda7c06da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:37.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda7c0c2870 0x7fda7c1f5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:37.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda7c15d200 0x7fda7c15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:37.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c14bff0 con 0x7fda7c06fbd0 2023-10-30T13:10:37.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c06df80 con 0x7fda7c0c2870 2023-10-30T13:10:37.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda822a9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c06b9b0 con 0x7fda7c15d200 2023-10-30T13:10:37.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda7bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c06fbd0 0x7fda7c06da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda7bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c06fbd0 0x7fda7c06da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50578/0 (socket says 172.21.15.148:50578) 2023-10-30T13:10:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda7bfff640 1 -- 172.21.15.148:0/102454065 learned_addr learned my addr 172.21.15.148:0/102454065 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda7b7fe640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda7c0c2870 0x7fda7c1f5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda80846640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda7c15d200 0x7fda7c15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda7bfff640 1 -- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda7c15d200 msgr2=0x7fda7c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda7bfff640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda7c15d200 0x7fda7c15bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda7bfff640 1 -- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda7c0c2870 msgr2=0x7fda7c1f5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda7bfff640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda7c0c2870 0x7fda7c1f5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.864+0000 7fda7bfff640 1 -- 172.21.15.148:0/102454065 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda74009b30 con 0x7fda7c06fbd0 2023-10-30T13:10:37.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda7bfff640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c06fbd0 0x7fda7c06da40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00c410 tx=0x7fda6c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:37.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda78ff9640 1 -- 172.21.15.148:0/102454065 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c0074e0 con 0x7fda7c06fbd0 2023-10-30T13:10:37.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda78ff9640 1 -- 172.21.15.148:0/102454065 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6c007640 con 0x7fda7c06fbd0 2023-10-30T13:10:37.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda78ff9640 1 -- 172.21.15.148:0/102454065 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c004a30 con 0x7fda7c06fbd0 2023-10-30T13:10:37.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7c1f5640 con 0x7fda7c06fbd0 2023-10-30T13:10:37.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7c149f70 con 0x7fda7c06fbd0 2023-10-30T13:10:37.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7c04e170 con 0x7fda7c06fbd0 2023-10-30T13:10:37.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda78ff9640 1 -- 172.21.15.148:0/102454065 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda6c004b90 con 0x7fda7c06fbd0 2023-10-30T13:10:37.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda78ff9640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda680664a0 0x7fda68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:37.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.868+0000 7fda78ff9640 1 -- 172.21.15.148:0/102454065 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda6c083d60 con 0x7fda7c06fbd0 2023-10-30T13:10:37.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.872+0000 7fda7b7fe640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda680664a0 0x7fda68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:37.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.872+0000 7fda7b7fe640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda680664a0 0x7fda68068960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fda7c15bc60 tx=0x7fda70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:37.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:37.872+0000 7fda78ff9640 1 -- 172.21.15.148:0/102454065 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda6c051140 con 0x7fda7c06fbd0 2023-10-30T13:10:38.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.052+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fda7c0c6790 con 0x7fda7c06fbd0 2023-10-30T13:10:38.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.700+0000 7fda78ff9640 1 -- 172.21.15.148:0/102454065 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7fda6c054760 con 0x7fda7c06fbd0 2023-10-30T13:10:38.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda680664a0 msgr2=0x7fda68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda680664a0 0x7fda68068960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fda7c15bc60 tx=0x7fda70009040 comp rx=0 tx=0).stop 2023-10-30T13:10:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c06fbd0 msgr2=0x7fda7c06da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c06fbd0 0x7fda7c06da40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00c410 tx=0x7fda6c00c8d0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 shutdown_connections 2023-10-30T13:10:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda680664a0 0x7fda68068960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda7c15d200 0x7fda7c15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda7c0c2870 0x7fda7c1f5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 --2- 172.21.15.148:0/102454065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda7c06fbd0 0x7fda7c06da40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 >> 172.21.15.148:0/102454065 conn(0x7fda7c0ccfe0 msgr2=0x7fda7c14e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 shutdown_connections 2023-10-30T13:10:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.704+0000 7fda822a9640 1 -- 172.21.15.148:0/102454065 wait complete. 2023-10-30T13:10:38.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-30T13:10:38.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f734133e640 1 Processor -- start 2023-10-30T13:10:38.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f734133e640 1 -- start start 2023-10-30T13:10:38.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f734133e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f733c15d570 0x7f733c15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:38.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f734133e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c15a920 con 0x7f733c153980 2023-10-30T13:10:38.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f734133e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c0cb930 con 0x7f733c159930 2023-10-30T13:10:38.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f734133e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733c159dc0 con 0x7f733c15d570 2023-10-30T13:10:38.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733affd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f733c153980 0x7f733c0caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34308/0 (socket says 172.21.15.148:34308) 2023-10-30T13:10:38.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733affd640 1 -- 172.21.15.148:0/3458280631 learned_addr learned my addr 172.21.15.148:0/3458280631 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:38.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733b7fe640 1 --2- 172.21.15.148:0/3458280631 >> v2:172.21.15.150:3300/0 conn(0x7f733c15d570 0x7f733c15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:38.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592377173 0 0) 0x7f733c15a920 con 0x7f733c153980 2023-10-30T13:10:38.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7324003530 con 0x7f733c153980 2023-10-30T13:10:38.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118162906 0 0) 0x7f733c0cb930 con 0x7f733c159930 2023-10-30T13:10:38.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f733c15a920 con 0x7f733c159930 2023-10-30T13:10:38.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2213759712 0 0) 0x7f7324003530 con 0x7f733c153980 2023-10-30T13:10:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 >> v2:172.21.15.150:3300/0 conn(0x7f733c15d570 msgr2=0x7f733c15d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 --2- 172.21.15.148:0/3458280631 >> v2:172.21.15.150:3300/0 conn(0x7f733c15d570 0x7f733c15d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 >> v1:172.21.15.150:6789/0 conn(0x7f733c159930 legacy=0x7f733c0cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c15a080 con 0x7f733c153980 2023-10-30T13:10:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f732c001680 con 0x7f733c153980 2023-10-30T13:10:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733b7fe640 1 --2- 172.21.15.148:0/3458280631 >> v2:172.21.15.150:3300/0 conn(0x7f733c15d570 0x7f733c15d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 >> v1:172.21.15.148:6789/0 conn(0x7f733c153980 legacy=0x7f733c0caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324003b10 0x7f7324003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73240047d0 0x7f7324004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f732400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c159dc0 con 0x7f7324005190 2023-10-30T13:10:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732400c010 con 0x7f7324003b10 2023-10-30T13:10:38.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732400c1d0 con 0x7f73240047d0 2023-10-30T13:10:38.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733b7fe640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f732400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:38.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733affd640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324003b10 0x7f7324003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:38.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733a7fc640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73240047d0 0x7f7324004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:38.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733b7fe640 1 -- 172.21.15.148:0/3458280631 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73240047d0 msgr2=0x7f7324004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:38.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733b7fe640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f73240047d0 0x7f7324004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733b7fe640 1 -- 172.21.15.148:0/3458280631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324003b10 msgr2=0x7f7324003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:38.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733b7fe640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324003b10 0x7f7324003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.868+0000 7f733b7fe640 1 -- 172.21.15.148:0/3458280631 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7324003530 con 0x7f7324005190 2023-10-30T13:10:38.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f733b7fe640 1 -- 172.21.15.148:0/3458280631 reap_dead start 2023-10-30T13:10:38.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f733b7fe640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f732400b990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f733400e350 tx=0x7f733401ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=243d753d6cc2c31a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:38.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c007ea0 con 0x7f7324005190 2023-10-30T13:10:38.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733400f040 con 0x7f7324005190 2023-10-30T13:10:38.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f7339ffb640 1 -- 172.21.15.148:0/3458280631 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73340ad440 con 0x7f7324005190 2023-10-30T13:10:38.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f734133e640 1 -- 172.21.15.148:0/3458280631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 msgr2=0x7f732400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:38.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f734133e640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f732400b990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f733400e350 tx=0x7f733401ce60 comp rx=0 tx=0).stop 2023-10-30T13:10:38.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f734133e640 1 -- 172.21.15.148:0/3458280631 shutdown_connections 2023-10-30T13:10:38.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f734133e640 1 --2- 172.21.15.148:0/3458280631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f732400b990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f734133e640 1 -- 172.21.15.148:0/3458280631 >> 172.21.15.148:0/3458280631 conn(0x7f733c06c0f0 msgr2=0x7f733c06c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:38.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f734133e640 1 -- 172.21.15.148:0/3458280631 shutdown_connections 2023-10-30T13:10:38.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.872+0000 7f734133e640 1 -- 172.21.15.148:0/3458280631 wait complete. 2023-10-30T13:10:38.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 Processor -- start 2023-10-30T13:10:38.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 -- start start 2023-10-30T13:10:38.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f733c150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:38.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f733c0ca370 0x7f733c14da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:38.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c153fd0 0x7f733c14ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:38.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c14fc40 con 0x7f7324005190 2023-10-30T13:10:38.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1efa50 con 0x7f733c0ca370 2023-10-30T13:10:38.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c14f030 con 0x7f733c153fd0 2023-10-30T13:10:38.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f733c150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:38.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f733c150460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50600/0 (socket says 172.21.15.148:50600) 2023-10-30T13:10:38.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733affd640 1 -- 172.21.15.148:0/2873145558 learned_addr learned my addr 172.21.15.148:0/2873145558 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:38.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733b7fe640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c153fd0 0x7f733c14ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:38.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733a7fc640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f733c0ca370 0x7f733c14da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:38.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733affd640 1 -- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c153fd0 msgr2=0x7f733c14ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:38.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733affd640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c153fd0 0x7f733c14ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733affd640 1 -- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f733c0ca370 msgr2=0x7f733c14da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:38.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733affd640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f733c0ca370 0x7f733c14da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:38.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733affd640 1 -- 172.21.15.148:0/2873145558 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c14f310 con 0x7f7324005190 2023-10-30T13:10:38.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f733affd640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f733c150460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f732c012540 tx=0x7f732c012a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:38.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f73237fe640 1 -- 172.21.15.148:0/2873145558 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c014750 con 0x7f7324005190 2023-10-30T13:10:38.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f73237fe640 1 -- 172.21.15.148:0/2873145558 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c0148d0 con 0x7f7324005190 2023-10-30T13:10:38.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f73237fe640 1 -- 172.21.15.148:0/2873145558 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c014c80 con 0x7f7324005190 2023-10-30T13:10:38.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c14f5f0 con 0x7f7324005190 2023-10-30T13:10:38.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.876+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c0ca0e0 con 0x7f7324005190 2023-10-30T13:10:38.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.880+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c04e170 con 0x7f7324005190 2023-10-30T13:10:38.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.880+0000 7f73237fe640 1 -- 172.21.15.148:0/2873145558 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f732c0153e0 con 0x7f7324005190 2023-10-30T13:10:38.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.880+0000 7f73237fe640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f73280664d0 0x7f7328068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:38.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.880+0000 7f73237fe640 1 -- 172.21.15.148:0/2873145558 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f732c0867a0 con 0x7f7324005190 2023-10-30T13:10:38.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.880+0000 7f733a7fc640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f73280664d0 0x7f7328068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:38.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.880+0000 7f733a7fc640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f73280664d0 0x7f7328068990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7330007860 tx=0x7f7330009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:38.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:38.884+0000 7f73237fe640 1 -- 172.21.15.148:0/2873145558 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f732c053bb0 con 0x7f7324005190 2023-10-30T13:10:39.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.064+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f733c0b1f10 con 0x7f7324005190 2023-10-30T13:10:39.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.708+0000 7f73237fe640 1 -- 172.21.15.148:0/2873145558 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7f732c01c020 con 0x7f7324005190 2023-10-30T13:10:39.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f73280664d0 msgr2=0x7f7328068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:39.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f73280664d0 0x7f7328068990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7330007860 tx=0x7f7330009040 comp rx=0 tx=0).stop 2023-10-30T13:10:39.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 msgr2=0x7f733c150460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:39.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f733c150460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f732c012540 tx=0x7f732c012a00 comp rx=0 tx=0).stop 2023-10-30T13:10:39.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 shutdown_connections 2023-10-30T13:10:39.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f73280664d0 0x7f7328068990 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f733c153fd0 0x7f733c14ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f733c0ca370 0x7f733c14da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 --2- 172.21.15.148:0/2873145558 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7324005190 0x7f733c150460 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 >> 172.21.15.148:0/2873145558 conn(0x7f733c06c0f0 msgr2=0x7f733c152860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 shutdown_connections 2023-10-30T13:10:39.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.712+0000 7f734133e640 1 -- 172.21.15.148:0/2873145558 wait complete. 2023-10-30T13:10:39.737 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-30T13:10:39.737 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:10:39.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34f958d640 1 Processor -- start 2023-10-30T13:10:39.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34f958d640 1 -- start start 2023-10-30T13:10:39.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34f958d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34f41538a0 0x7f34f415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:39.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34f958d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f40ca0e0 0x7f34f415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:39.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34f958d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f40cb800 con 0x7f34f40cf160 2023-10-30T13:10:39.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34f958d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f415dc30 con 0x7f34f41538a0 2023-10-30T13:10:39.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34f958d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f40cb330 con 0x7f34f40ca0e0 2023-10-30T13:10:39.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34ea7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f40ca0e0 0x7f34f415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:39.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34f2ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34f41538a0 0x7f34f415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:39.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34ea7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f40ca0e0 0x7f34f415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:44626/0 (socket says 172.21.15.148:44626) 2023-10-30T13:10:39.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34ea7fc640 1 -- 172.21.15.148:0/535408003 learned_addr learned my addr 172.21.15.148:0/535408003 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:39.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34f27fc640 1 -- 172.21.15.148:0/535408003 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391773437 0 0) 0x7f34f40cb800 con 0x7f34f40cf160 2023-10-30T13:10:39.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34ea7fc640 1 -- 172.21.15.148:0/535408003 >> v2:172.21.15.150:3300/0 conn(0x7f34f41538a0 msgr2=0x7f34f415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:39.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34ea7fc640 1 --2- 172.21.15.148:0/535408003 >> v2:172.21.15.150:3300/0 conn(0x7f34f41538a0 0x7f34f415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:39.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34ea7fc640 1 -- 172.21.15.148:0/535408003 >> v1:172.21.15.150:6789/0 conn(0x7f34f40cf160 legacy=0x7f34f40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:39.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.872+0000 7f34ea7fc640 1 -- 172.21.15.148:0/535408003 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f40cad70 con 0x7f34f40ca0e0 2023-10-30T13:10:39.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34ea7fc640 1 --2- 172.21.15.148:0/535408003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f40ca0e0 0x7f34f415a090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009870 tx=0x7f34dc01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af2fe641a6aed452 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:39.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f27fc640 1 -- 172.21.15.148:0/535408003 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0a8a30 con 0x7f34f40ca0e0 2023-10-30T13:10:39.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f27fc640 1 -- 172.21.15.148:0/535408003 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc0a8b90 con 0x7f34f40ca0e0 2023-10-30T13:10:39.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f27fc640 1 -- 172.21.15.148:0/535408003 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0b0440 con 0x7f34f40ca0e0 2023-10-30T13:10:39.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f958d640 1 -- 172.21.15.148:0/535408003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f40ca0e0 msgr2=0x7f34f415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:39.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f958d640 1 --2- 172.21.15.148:0/535408003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f40ca0e0 0x7f34f415a090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009870 tx=0x7f34dc01ed30 comp rx=0 tx=0).stop 2023-10-30T13:10:39.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f958d640 1 -- 172.21.15.148:0/535408003 shutdown_connections 2023-10-30T13:10:39.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f958d640 1 --2- 172.21.15.148:0/535408003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f40ca0e0 0x7f34f415a090 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:39.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f958d640 1 --2- 172.21.15.148:0/535408003 >> v2:172.21.15.150:3300/0 conn(0x7f34f41538a0 0x7f34f415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:39.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f958d640 1 -- 172.21.15.148:0/535408003 >> 172.21.15.148:0/535408003 conn(0x7f34f406bfe0 msgr2=0x7f34f406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:39.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f958d640 1 -- 172.21.15.148:0/535408003 shutdown_connections 2023-10-30T13:10:39.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.876+0000 7f34f958d640 1 -- 172.21.15.148:0/535408003 wait complete. 2023-10-30T13:10:39.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f958d640 1 Processor -- start 2023-10-30T13:10:39.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f958d640 1 -- start start 2023-10-30T13:10:39.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f958d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34f40ca0e0 0x7f34f41f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:39.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f958d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34f40cf160 0x7f34f4157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:39.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f958d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f41538a0 0x7f34f40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:39.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f958d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4159700 con 0x7f34f40cf160 2023-10-30T13:10:39.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f958d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f41efb50 con 0x7f34f40ca0e0 2023-10-30T13:10:39.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f958d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f41ef280 con 0x7f34f41538a0 2023-10-30T13:10:39.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ea7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34f40cf160 0x7f34f4157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:39.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ea7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34f40cf160 0x7f34f4157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50604/0 (socket says 172.21.15.148:50604) 2023-10-30T13:10:39.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ea7fc640 1 -- 172.21.15.148:0/2268947688 learned_addr learned my addr 172.21.15.148:0/2268947688 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:39.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f2ffd640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34f40ca0e0 0x7f34f41f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:39.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34f37fe640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f41538a0 0x7f34f40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:39.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ea7fc640 1 -- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f41538a0 msgr2=0x7f34f40ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:39.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ea7fc640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f41538a0 0x7f34f40ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:39.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ea7fc640 1 -- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34f40ca0e0 msgr2=0x7f34f41f31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:39.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ea7fc640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34f40ca0e0 0x7f34f41f31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:39.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ea7fc640 1 -- 172.21.15.148:0/2268947688 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc0094d0 con 0x7f34f40cf160 2023-10-30T13:10:39.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ea7fc640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34f40cf160 0x7f34f4157520 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0b3040 tx=0x7f34dc01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:39.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ebfff640 1 -- 172.21.15.148:0/2268947688 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0b0dd0 con 0x7f34f40cf160 2023-10-30T13:10:39.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ebfff640 1 -- 172.21.15.148:0/2268947688 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc0bb460 con 0x7f34f40cf160 2023-10-30T13:10:39.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.880+0000 7f34ebfff640 1 -- 172.21.15.148:0/2268947688 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0bb810 con 0x7f34f40cf160 2023-10-30T13:10:39.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.884+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f4159140 con 0x7f34f40cf160 2023-10-30T13:10:39.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.884+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f41ef490 con 0x7f34f40cf160 2023-10-30T13:10:39.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.884+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f40b2a50 con 0x7f34f40cf160 2023-10-30T13:10:39.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.884+0000 7f34ebfff640 1 -- 172.21.15.148:0/2268947688 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34dc0b83f0 con 0x7f34f40cf160 2023-10-30T13:10:39.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.884+0000 7f34ebfff640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34d40664a0 0x7f34d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:39.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.884+0000 7f34ebfff640 1 -- 172.21.15.148:0/2268947688 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34dc1216e0 con 0x7f34f40cf160 2023-10-30T13:10:39.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.884+0000 7f34f2ffd640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34d40664a0 0x7f34d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:39.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.888+0000 7f34f2ffd640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34d40664a0 0x7f34d4068960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34e4011340 tx=0x7f34e400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:39.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:39.888+0000 7f34ebfff640 1 -- 172.21.15.148:0/2268947688 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34dc0eec10 con 0x7f34f40cf160 2023-10-30T13:10:40.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.072+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34f41efc90 con 0x7f34f40cf160 2023-10-30T13:10:40.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.072+0000 7f34ebfff640 1 -- 172.21.15.148:0/2268947688 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6508 (secure 0 0 0) 0x7f34f41efc90 con 0x7f34f40cf160 2023-10-30T13:10:40.077 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:40.077 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4256,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:39.068853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:10:40.078 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 8 2023-10-30T13:10:40.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34d40664a0 msgr2=0x7f34d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34d40664a0 0x7f34d4068960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34e4011340 tx=0x7f34e400b040 comp rx=0 tx=0).stop 2023-10-30T13:10:40.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34f40cf160 msgr2=0x7f34f4157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34f40cf160 0x7f34f4157520 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0b3040 tx=0x7f34dc01bad0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 shutdown_connections 2023-10-30T13:10:40.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34d40664a0 0x7f34d4068960 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34f40cf160 0x7f34f4157520 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34f41538a0 0x7f34f40ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 --2- 172.21.15.148:0/2268947688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34f40ca0e0 0x7f34f41f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 >> 172.21.15.148:0/2268947688 conn(0x7f34f406bfe0 msgr2=0x7f34f4066310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:40.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 shutdown_connections 2023-10-30T13:10:40.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.076+0000 7f34f958d640 1 -- 172.21.15.148:0/2268947688 wait complete. 2023-10-30T13:10:40.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:40.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea0a1d3640 1 Processor -- start 2023-10-30T13:10:40.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea0a1d3640 1 -- start start 2023-10-30T13:10:40.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea0a1d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fea0406f7a0 0x7fea0406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea0a1d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea04066cf0 con 0x7fea040c6bd0 2023-10-30T13:10:40.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea0a1d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea040c70d0 con 0x7fea04071110 2023-10-30T13:10:40.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea0a1d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea040700c0 con 0x7fea0406f7a0 2023-10-30T13:10:40.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea037fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fea0406f7a0 0x7fea0406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea037fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fea0406f7a0 0x7fea0406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50614/0 (socket says 172.21.15.148:50614) 2023-10-30T13:10:40.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea02ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fea040c6bd0 0x7fea04069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34320/0 (socket says 172.21.15.148:34320) 2023-10-30T13:10:40.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea037fe640 1 -- 172.21.15.148:0/3880840989 learned_addr learned my addr 172.21.15.148:0/3880840989 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:40.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea037fe640 1 -- 172.21.15.148:0/3880840989 >> v1:172.21.15.150:6789/0 conn(0x7fea04071110 legacy=0x7fea04158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea037fe640 1 -- 172.21.15.148:0/3880840989 >> v1:172.21.15.148:6789/0 conn(0x7fea040c6bd0 legacy=0x7fea04069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.232+0000 7fea037fe640 1 -- 172.21.15.148:0/3880840989 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea040c7390 con 0x7fea0406f7a0 2023-10-30T13:10:40.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea037fe640 1 --2- 172.21.15.148:0/3880840989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea0406f7a0 0x7fea0406fb80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009870 tx=0x7fe9f401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12be2b14bb370cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:40.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea027fc640 1 -- 172.21.15.148:0/3880840989 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f40a8a30 con 0x7fea0406f7a0 2023-10-30T13:10:40.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea027fc640 1 -- 172.21.15.148:0/3880840989 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f40a8b90 con 0x7fea0406f7a0 2023-10-30T13:10:40.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea027fc640 1 -- 172.21.15.148:0/3880840989 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f40b0440 con 0x7fea0406f7a0 2023-10-30T13:10:40.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea0a1d3640 1 -- 172.21.15.148:0/3880840989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea0406f7a0 msgr2=0x7fea0406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea0a1d3640 1 --2- 172.21.15.148:0/3880840989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea0406f7a0 0x7fea0406fb80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009870 tx=0x7fe9f401ed30 comp rx=0 tx=0).stop 2023-10-30T13:10:40.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea0a1d3640 1 -- 172.21.15.148:0/3880840989 shutdown_connections 2023-10-30T13:10:40.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea0a1d3640 1 --2- 172.21.15.148:0/3880840989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea0406f7a0 0x7fea0406fb80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea0a1d3640 1 -- 172.21.15.148:0/3880840989 >> 172.21.15.148:0/3880840989 conn(0x7fea040c7910 msgr2=0x7fea040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:40.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea0a1d3640 1 -- 172.21.15.148:0/3880840989 shutdown_connections 2023-10-30T13:10:40.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.236+0000 7fea0a1d3640 1 -- 172.21.15.148:0/3880840989 wait complete. 2023-10-30T13:10:40.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 Processor -- start 2023-10-30T13:10:40.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 -- start start 2023-10-30T13:10:40.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea0406f7a0 0x7fea04155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea04071110 0x7fea04150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea040c6bd0 0x7fea04154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea04156780 con 0x7fea04071110 2023-10-30T13:10:40.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea040c96d0 con 0x7fea040c6bd0 2023-10-30T13:10:40.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea04146910 con 0x7fea0406f7a0 2023-10-30T13:10:40.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea02ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea04071110 0x7fea04150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea02ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea04071110 0x7fea04150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50628/0 (socket says 172.21.15.148:50628) 2023-10-30T13:10:40.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea02ffd640 1 -- 172.21.15.148:0/2333668936 learned_addr learned my addr 172.21.15.148:0/2333668936 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:40.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea037fe640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea0406f7a0 0x7fea04155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea03fff640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea040c6bd0 0x7fea04154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea02ffd640 1 -- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea0406f7a0 msgr2=0x7fea04155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea02ffd640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea0406f7a0 0x7fea04155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea02ffd640 1 -- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea040c6bd0 msgr2=0x7fea04154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea02ffd640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea040c6bd0 0x7fea04154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea02ffd640 1 -- 172.21.15.148:0/2333668936 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f40094d0 con 0x7fea04071110 2023-10-30T13:10:40.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea037fe640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea0406f7a0 0x7fea04155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:40.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea02ffd640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea04071110 0x7fea04150b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800c530 tx=0x7fe9f800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:40.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fe9ebfff640 1 -- 172.21.15.148:0/2333668936 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8007500 con 0x7fea04071110 2023-10-30T13:10:40.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fe9ebfff640 1 -- 172.21.15.148:0/2333668936 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f8007660 con 0x7fea04071110 2023-10-30T13:10:40.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea040c93f0 con 0x7fea04071110 2023-10-30T13:10:40.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fe9ebfff640 1 -- 172.21.15.148:0/2333668936 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8004a10 con 0x7fea04071110 2023-10-30T13:10:40.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.240+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea04150640 con 0x7fea04071110 2023-10-30T13:10:40.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.244+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0404e170 con 0x7fea04071110 2023-10-30T13:10:40.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.244+0000 7fe9ebfff640 1 -- 172.21.15.148:0/2333668936 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe9f8004b70 con 0x7fea04071110 2023-10-30T13:10:40.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.244+0000 7fe9ebfff640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe9f00664d0 0x7fe9f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.244+0000 7fe9ebfff640 1 -- 172.21.15.148:0/2333668936 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9f8083010 con 0x7fea04071110 2023-10-30T13:10:40.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.244+0000 7fea037fe640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe9f00664d0 0x7fe9f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.244+0000 7fea037fe640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe9f00664d0 0x7fe9f0068990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40b2000 tx=0x7fe9f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:40.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.248+0000 7fe9ebfff640 1 -- 172.21.15.148:0/2333668936 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe9f8050520 con 0x7fea04071110 2023-10-30T13:10:40.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.396+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea040c7eb0 con 0x7fea04071110 2023-10-30T13:10:40.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fe9ebfff640 1 -- 172.21.15.148:0/2333668936 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fe9f8018090 con 0x7fea04071110 2023-10-30T13:10:40.403 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:40.403 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:35.311442+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:40.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe9f00664d0 msgr2=0x7fe9f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe9f00664d0 0x7fe9f0068990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40b2000 tx=0x7fe9f4008040 comp rx=0 tx=0).stop 2023-10-30T13:10:40.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea04071110 msgr2=0x7fea04150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea04071110 0x7fea04150b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800c530 tx=0x7fe9f800c9f0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 shutdown_connections 2023-10-30T13:10:40.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe9f00664d0 0x7fe9f0068990 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea04071110 0x7fea04150b60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea040c6bd0 0x7fea04154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 --2- 172.21.15.148:0/2333668936 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fea0406f7a0 0x7fea04155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 >> 172.21.15.148:0/2333668936 conn(0x7fea040c7910 msgr2=0x7fea040c3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:40.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 shutdown_connections 2023-10-30T13:10:40.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.400+0000 7fea0a1d3640 1 -- 172.21.15.148:0/2333668936 wait complete. 2023-10-30T13:10:40.424 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-30T13:10:40.424 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:10:40.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb4646640 1 Processor -- start 2023-10-30T13:10:40.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb4646640 1 -- start start 2023-10-30T13:10:40.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb4646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac1538a0 0x7f1eac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb4646640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1eac0cf160 0x7f1eac0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb4646640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eac0cb800 con 0x7f1eac0ca0e0 2023-10-30T13:10:40.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb4646640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac15dc30 con 0x7f1eac0cf160 2023-10-30T13:10:40.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb4646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac0cb330 con 0x7f1eac1538a0 2023-10-30T13:10:40.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb1be1640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f1eac0ca0e0 0x7f1eac15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34328/0 (socket says 172.21.15.148:34328) 2023-10-30T13:10:40.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb1be1640 1 -- 172.21.15.148:0/3908454376 learned_addr learned my addr 172.21.15.148:0/3908454376 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:40.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb2be3640 1 --2- 172.21.15.148:0/3908454376 >> v2:172.21.15.148:3300/0 conn(0x7f1eac0cf160 0x7f1eac0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb23e2640 1 --2- 172.21.15.148:0/3908454376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac1538a0 0x7f1eac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb13e0640 1 -- 172.21.15.148:0/3908454376 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225522671 0 0) 0x7f1eac0cb800 con 0x7f1eac0ca0e0 2023-10-30T13:10:40.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb2be3640 1 -- 172.21.15.148:0/3908454376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac1538a0 msgr2=0x7f1eac15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:40.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb2be3640 1 --2- 172.21.15.148:0/3908454376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac1538a0 0x7f1eac15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb2be3640 1 -- 172.21.15.148:0/3908454376 >> v1:172.21.15.148:6789/0 conn(0x7f1eac0ca0e0 legacy=0x7f1eac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.560+0000 7f1eb2be3640 1 -- 172.21.15.148:0/3908454376 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac0cad70 con 0x7f1eac0cf160 2023-10-30T13:10:40.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb2be3640 1 --2- 172.21.15.148:0/3908454376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 0x7f1eac0cf590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009d90 tx=0x7f1ea401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171b1aaf8c7dde95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:40.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb13e0640 1 -- 172.21.15.148:0/3908454376 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea40af600 con 0x7f1eac0cf160 2023-10-30T13:10:40.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb13e0640 1 -- 172.21.15.148:0/3908454376 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea40ab050 con 0x7f1eac0cf160 2023-10-30T13:10:40.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb13e0640 1 -- 172.21.15.148:0/3908454376 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea40af980 con 0x7f1eac0cf160 2023-10-30T13:10:40.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb4646640 1 -- 172.21.15.148:0/3908454376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 msgr2=0x7f1eac0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb4646640 1 --2- 172.21.15.148:0/3908454376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 0x7f1eac0cf590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009d90 tx=0x7f1ea401f130 comp rx=0 tx=0).stop 2023-10-30T13:10:40.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb4646640 1 -- 172.21.15.148:0/3908454376 shutdown_connections 2023-10-30T13:10:40.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb4646640 1 --2- 172.21.15.148:0/3908454376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 0x7f1eac0cf590 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb4646640 1 --2- 172.21.15.148:0/3908454376 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac1538a0 0x7f1eac15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb4646640 1 -- 172.21.15.148:0/3908454376 >> 172.21.15.148:0/3908454376 conn(0x7f1eac06bfe0 msgr2=0x7f1eac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:40.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb4646640 1 -- 172.21.15.148:0/3908454376 shutdown_connections 2023-10-30T13:10:40.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.564+0000 7f1eb4646640 1 -- 172.21.15.148:0/3908454376 wait complete. 2023-10-30T13:10:40.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 Processor -- start 2023-10-30T13:10:40.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 -- start start 2023-10-30T13:10:40.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac0ca0e0 0x7f1eac0c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 0x7f1eac15c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eac1538a0 0x7f1eac15ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac15ac80 con 0x7f1eac0cf160 2023-10-30T13:10:40.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac15aee0 con 0x7f1eac1538a0 2023-10-30T13:10:40.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac1f2530 con 0x7f1eac0ca0e0 2023-10-30T13:10:40.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb1be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 0x7f1eac15c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb1be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 0x7f1eac15c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50644/0 (socket says 172.21.15.148:50644) 2023-10-30T13:10:40.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb1be1640 1 -- 172.21.15.148:0/970929384 learned_addr learned my addr 172.21.15.148:0/970929384 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:40.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb23e2640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac0ca0e0 0x7f1eac0c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb2be3640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eac1538a0 0x7f1eac15ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb1be1640 1 -- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac0ca0e0 msgr2=0x7f1eac0c57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb1be1640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac0ca0e0 0x7f1eac0c57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb1be1640 1 -- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eac1538a0 msgr2=0x7f1eac15ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb1be1640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eac1538a0 0x7f1eac15ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb1be1640 1 -- 172.21.15.148:0/970929384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4009a70 con 0x7f1eac0cf160 2023-10-30T13:10:40.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb1be1640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 0x7f1eac15c510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0011f0 tx=0x7f1e9c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:40.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1ea2ffd640 1 -- 172.21.15.148:0/970929384 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c0074e0 con 0x7f1eac0cf160 2023-10-30T13:10:40.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1ea2ffd640 1 -- 172.21.15.148:0/970929384 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c007640 con 0x7f1eac0cf160 2023-10-30T13:10:40.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1ea2ffd640 1 -- 172.21.15.148:0/970929384 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c004a10 con 0x7f1eac0cf160 2023-10-30T13:10:40.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac1f2810 con 0x7f1eac0cf160 2023-10-30T13:10:40.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.568+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac1f2dd0 con 0x7f1eac0cf160 2023-10-30T13:10:40.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.572+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eac0b2a50 con 0x7f1eac0cf160 2023-10-30T13:10:40.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.572+0000 7f1ea2ffd640 1 -- 172.21.15.148:0/970929384 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1e9c004b70 con 0x7f1eac0cf160 2023-10-30T13:10:40.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.572+0000 7f1ea2ffd640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e900664d0 0x7f1e90068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.572+0000 7f1ea2ffd640 1 -- 172.21.15.148:0/970929384 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e9c082fe0 con 0x7f1eac0cf160 2023-10-30T13:10:40.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.572+0000 7f1eb23e2640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e900664d0 0x7f1e90068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.572+0000 7f1eb23e2640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e900664d0 0x7f1e90068990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009a50 tx=0x7f1ea8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:40.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.576+0000 7f1ea2ffd640 1 -- 172.21.15.148:0/970929384 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1e9c0504b0 con 0x7f1eac0cf160 2023-10-30T13:10:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.724+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1eac159890 con 0x7f1e900664d0 2023-10-30T13:10:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.748+0000 7f1ea2ffd640 1 -- 172.21.15.148:0/970929384 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264523 (secure 0 0 0) 0x7f1ea80119a0 con 0x7f1e900664d0 2023-10-30T13:10:40.754 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:40.755 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:10:40.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e900664d0 msgr2=0x7f1e90068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e900664d0 0x7f1e90068990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009a50 tx=0x7f1ea8008040 comp rx=0 tx=0).stop 2023-10-30T13:10:40.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 msgr2=0x7f1eac15c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 0x7f1eac15c510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0011f0 tx=0x7f1e9c00c9b0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 shutdown_connections 2023-10-30T13:10:40.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e900664d0 0x7f1e90068990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eac0cf160 0x7f1eac15c510 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eac1538a0 0x7f1eac15ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 --2- 172.21.15.148:0/970929384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eac0ca0e0 0x7f1eac0c57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 >> 172.21.15.148:0/970929384 conn(0x7f1eac06bfe0 msgr2=0x7f1eac1523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:40.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 shutdown_connections 2023-10-30T13:10:40.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.752+0000 7f1eb4646640 1 -- 172.21.15.148:0/970929384 wait complete. 2023-10-30T13:10:40.774 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-30T13:10:40.262673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218928,"kb_used_data":2756,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755792,"statfs":{"total":773094113280,"available":772869931008,"internally_reserved":0,"allocated":2822144,"data_stored":1746901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.585543"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322509+0000","last_change":"2023-10-30T13:10:34.379031+0000","last_active":"2023-10-30T13:10:35.322509+0000","last_peered":"2023-10-30T13:10:35.322509+0000","last_clean":"2023-10-30T13:10:35.322509+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:10:35.322509+0000","last_undegraded":"2023-10-30T13:10:35.322509+0000","last_fullsized":"2023-10-30T13:10:35.322509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:32.529312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919896+0000","last_change":"2023-10-30T13:10:34.324308+0000","last_active":"2023-10-30T13:10:35.919896+0000","last_peered":"2023-10-30T13:10:35.919896+0000","last_clean":"2023-10-30T13:10:35.919896+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:10:35.919896+0000","last_undegraded":"2023-10-30T13:10:35.919896+0000","last_fullsized":"2023-10-30T13:10:35.919896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:21.885716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919373+0000","last_change":"2023-10-30T13:10:34.334122+0000","last_active":"2023-10-30T13:10:35.919373+0000","last_peered":"2023-10-30T13:10:35.919373+0000","last_clean":"2023-10-30T13:10:35.919373+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:10:35.919373+0000","last_undegraded":"2023-10-30T13:10:35.919373+0000","last_fullsized":"2023-10-30T13:10:35.919373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:55.045024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324579+0000","last_change":"2023-10-30T13:10:34.346795+0000","last_active":"2023-10-30T13:10:35.324579+0000","last_peered":"2023-10-30T13:10:35.324579+0000","last_clean":"2023-10-30T13:10:35.324579+0000","last_became_active":"2023-10-30T13:10:34.346079+0000","last_became_peered":"2023-10-30T13:10:34.346079+0000","last_unstale":"2023-10-30T13:10:35.324579+0000","last_undegraded":"2023-10-30T13:10:35.324579+0000","last_fullsized":"2023-10-30T13:10:35.324579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:09.546205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322339+0000","last_change":"2023-10-30T13:10:34.371670+0000","last_active":"2023-10-30T13:10:35.322339+0000","last_peered":"2023-10-30T13:10:35.322339+0000","last_clean":"2023-10-30T13:10:35.322339+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:10:35.322339+0000","last_undegraded":"2023-10-30T13:10:35.322339+0000","last_fullsized":"2023-10-30T13:10:35.322339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:03:54.773965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323062+0000","last_change":"2023-10-30T13:10:34.335752+0000","last_active":"2023-10-30T13:10:35.323062+0000","last_peered":"2023-10-30T13:10:35.323062+0000","last_clean":"2023-10-30T13:10:35.323062+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:10:35.323062+0000","last_undegraded":"2023-10-30T13:10:35.323062+0000","last_fullsized":"2023-10-30T13:10:35.323062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:33.278920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321759+0000","last_change":"2023-10-30T13:10:34.371817+0000","last_active":"2023-10-30T13:10:35.321759+0000","last_peered":"2023-10-30T13:10:35.321759+0000","last_clean":"2023-10-30T13:10:35.321759+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:10:35.321759+0000","last_undegraded":"2023-10-30T13:10:35.321759+0000","last_fullsized":"2023-10-30T13:10:35.321759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:33.797113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318581+0000","last_change":"2023-10-30T13:10:34.348205+0000","last_active":"2023-10-30T13:10:35.318581+0000","last_peered":"2023-10-30T13:10:35.318581+0000","last_clean":"2023-10-30T13:10:35.318581+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:10:35.318581+0000","last_undegraded":"2023-10-30T13:10:35.318581+0000","last_fullsized":"2023-10-30T13:10:35.318581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:57.492738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322864+0000","last_change":"2023-10-30T13:10:34.381052+0000","last_active":"2023-10-30T13:10:35.322864+0000","last_peered":"2023-10-30T13:10:35.322864+0000","last_clean":"2023-10-30T13:10:35.322864+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:10:35.322864+0000","last_undegraded":"2023-10-30T13:10:35.322864+0000","last_fullsized":"2023-10-30T13:10:35.322864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:16.697332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759029+0000","last_change":"2023-10-30T13:10:34.356932+0000","last_active":"2023-10-30T13:10:35.759029+0000","last_peered":"2023-10-30T13:10:35.759029+0000","last_clean":"2023-10-30T13:10:35.759029+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:10:35.759029+0000","last_undegraded":"2023-10-30T13:10:35.759029+0000","last_fullsized":"2023-10-30T13:10:35.759029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:54.091993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758774+0000","last_change":"2023-10-30T13:10:34.357358+0000","last_active":"2023-10-30T13:10:35.758774+0000","last_peered":"2023-10-30T13:10:35.758774+0000","last_clean":"2023-10-30T13:10:35.758774+0000","last_became_active":"2023-10-30T13:10:34.356466+0000","last_became_peered":"2023-10-30T13:10:34.356466+0000","last_unstale":"2023-10-30T13:10:35.758774+0000","last_undegraded":"2023-10-30T13:10:35.758774+0000","last_fullsized":"2023-10-30T13:10:35.758774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:06:24.029159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924241+0000","last_change":"2023-10-30T13:10:34.365161+0000","last_active":"2023-10-30T13:10:35.924241+0000","last_peered":"2023-10-30T13:10:35.924241+0000","last_clean":"2023-10-30T13:10:35.924241+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:10:35.924241+0000","last_undegraded":"2023-10-30T13:10:35.924241+0000","last_fullsized":"2023-10-30T13:10:35.924241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:51.721060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318828+0000","last_change":"2023-10-30T13:10:34.351901+0000","last_active":"2023-10-30T13:10:35.318828+0000","last_peered":"2023-10-30T13:10:35.318828+0000","last_clean":"2023-10-30T13:10:35.318828+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:10:35.318828+0000","last_undegraded":"2023-10-30T13:10:35.318828+0000","last_fullsized":"2023-10-30T13:10:35.318828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:11.576546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920191+0000","last_change":"2023-10-30T13:10:34.323358+0000","last_active":"2023-10-30T13:10:35.920191+0000","last_peered":"2023-10-30T13:10:35.920191+0000","last_clean":"2023-10-30T13:10:35.920191+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:10:35.920191+0000","last_undegraded":"2023-10-30T13:10:35.920191+0000","last_fullsized":"2023-10-30T13:10:35.920191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:03.178822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919173+0000","last_change":"2023-10-30T13:10:34.360120+0000","last_active":"2023-10-30T13:10:35.919173+0000","last_peered":"2023-10-30T13:10:35.919173+0000","last_clean":"2023-10-30T13:10:35.919173+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:10:35.919173+0000","last_undegraded":"2023-10-30T13:10:35.919173+0000","last_fullsized":"2023-10-30T13:10:35.919173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:30.731746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322343+0000","last_change":"2023-10-30T13:10:34.333752+0000","last_active":"2023-10-30T13:10:35.322343+0000","last_peered":"2023-10-30T13:10:35.322343+0000","last_clean":"2023-10-30T13:10:35.322343+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:10:35.322343+0000","last_undegraded":"2023-10-30T13:10:35.322343+0000","last_fullsized":"2023-10-30T13:10:35.322343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:45.390603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324451+0000","last_change":"2023-10-30T13:10:34.371931+0000","last_active":"2023-10-30T13:10:35.324451+0000","last_peered":"2023-10-30T13:10:35.324451+0000","last_clean":"2023-10-30T13:10:35.324451+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:10:35.324451+0000","last_undegraded":"2023-10-30T13:10:35.324451+0000","last_fullsized":"2023-10-30T13:10:35.324451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:51.514990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919968+0000","last_change":"2023-10-30T13:10:34.352747+0000","last_active":"2023-10-30T13:10:35.919968+0000","last_peered":"2023-10-30T13:10:35.919968+0000","last_clean":"2023-10-30T13:10:35.919968+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:10:35.919968+0000","last_undegraded":"2023-10-30T13:10:35.919968+0000","last_fullsized":"2023-10-30T13:10:35.919968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:21.543351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323598+0000","last_change":"2023-10-30T13:10:34.376135+0000","last_active":"2023-10-30T13:10:35.323598+0000","last_peered":"2023-10-30T13:10:35.323598+0000","last_clean":"2023-10-30T13:10:35.323598+0000","last_became_active":"2023-10-30T13:10:34.375497+0000","last_became_peered":"2023-10-30T13:10:34.375497+0000","last_unstale":"2023-10-30T13:10:35.323598+0000","last_undegraded":"2023-10-30T13:10:35.323598+0000","last_fullsized":"2023-10-30T13:10:35.323598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:14.816946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758767+0000","last_change":"2023-10-30T13:10:34.366575+0000","last_active":"2023-10-30T13:10:35.758767+0000","last_peered":"2023-10-30T13:10:35.758767+0000","last_clean":"2023-10-30T13:10:35.758767+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:10:35.758767+0000","last_undegraded":"2023-10-30T13:10:35.758767+0000","last_fullsized":"2023-10-30T13:10:35.758767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:23.540645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920318+0000","last_change":"2023-10-30T13:10:34.368761+0000","last_active":"2023-10-30T13:10:35.920318+0000","last_peered":"2023-10-30T13:10:35.920318+0000","last_clean":"2023-10-30T13:10:35.920318+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:10:35.920318+0000","last_undegraded":"2023-10-30T13:10:35.920318+0000","last_fullsized":"2023-10-30T13:10:35.920318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:32.861756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323430+0000","last_change":"2023-10-30T13:10:34.372881+0000","last_active":"2023-10-30T13:10:35.323430+0000","last_peered":"2023-10-30T13:10:35.323430+0000","last_clean":"2023-10-30T13:10:35.323430+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:10:35.323430+0000","last_undegraded":"2023-10-30T13:10:35.323430+0000","last_fullsized":"2023-10-30T13:10:35.323430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:23.855779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318722+0000","last_change":"2023-10-30T13:10:34.359359+0000","last_active":"2023-10-30T13:10:35.318722+0000","last_peered":"2023-10-30T13:10:35.318722+0000","last_clean":"2023-10-30T13:10:35.318722+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:10:35.318722+0000","last_undegraded":"2023-10-30T13:10:35.318722+0000","last_fullsized":"2023-10-30T13:10:35.318722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:08.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322387+0000","last_change":"2023-10-30T13:10:34.347431+0000","last_active":"2023-10-30T13:10:35.322387+0000","last_peered":"2023-10-30T13:10:35.322387+0000","last_clean":"2023-10-30T13:10:35.322387+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:10:35.322387+0000","last_undegraded":"2023-10-30T13:10:35.322387+0000","last_fullsized":"2023-10-30T13:10:35.322387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:25.094617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319296+0000","last_change":"2023-10-30T13:10:34.359717+0000","last_active":"2023-10-30T13:10:35.319296+0000","last_peered":"2023-10-30T13:10:35.319296+0000","last_clean":"2023-10-30T13:10:35.319296+0000","last_became_active":"2023-10-30T13:10:34.359063+0000","last_became_peered":"2023-10-30T13:10:34.359063+0000","last_unstale":"2023-10-30T13:10:35.319296+0000","last_undegraded":"2023-10-30T13:10:35.319296+0000","last_fullsized":"2023-10-30T13:10:35.319296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:22.525493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759209+0000","last_change":"2023-10-30T13:10:34.379878+0000","last_active":"2023-10-30T13:10:35.759209+0000","last_peered":"2023-10-30T13:10:35.759209+0000","last_clean":"2023-10-30T13:10:35.759209+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:10:35.759209+0000","last_undegraded":"2023-10-30T13:10:35.759209+0000","last_fullsized":"2023-10-30T13:10:35.759209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:42.280634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919210+0000","last_change":"2023-10-30T13:10:34.332020+0000","last_active":"2023-10-30T13:10:35.919210+0000","last_peered":"2023-10-30T13:10:35.919210+0000","last_clean":"2023-10-30T13:10:35.919210+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:10:35.919210+0000","last_undegraded":"2023-10-30T13:10:35.919210+0000","last_fullsized":"2023-10-30T13:10:35.919210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:53.057012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319283+0000","last_change":"2023-10-30T13:10:34.363154+0000","last_active":"2023-10-30T13:10:35.319283+0000","last_peered":"2023-10-30T13:10:35.319283+0000","last_clean":"2023-10-30T13:10:35.319283+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:10:35.319283+0000","last_undegraded":"2023-10-30T13:10:35.319283+0000","last_fullsized":"2023-10-30T13:10:35.319283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:04.359629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920381+0000","last_change":"2023-10-30T13:10:34.332160+0000","last_active":"2023-10-30T13:10:35.920381+0000","last_peered":"2023-10-30T13:10:35.920381+0000","last_clean":"2023-10-30T13:10:35.920381+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:10:35.920381+0000","last_undegraded":"2023-10-30T13:10:35.920381+0000","last_fullsized":"2023-10-30T13:10:35.920381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:28.128344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323753+0000","last_change":"2023-10-30T13:10:34.337617+0000","last_active":"2023-10-30T13:10:35.323753+0000","last_peered":"2023-10-30T13:10:35.323753+0000","last_clean":"2023-10-30T13:10:35.323753+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:10:35.323753+0000","last_undegraded":"2023-10-30T13:10:35.323753+0000","last_fullsized":"2023-10-30T13:10:35.323753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:10.287441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920225+0000","last_change":"2023-10-30T13:10:34.359681+0000","last_active":"2023-10-30T13:10:35.920225+0000","last_peered":"2023-10-30T13:10:35.920225+0000","last_clean":"2023-10-30T13:10:35.920225+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:10:35.920225+0000","last_undegraded":"2023-10-30T13:10:35.920225+0000","last_fullsized":"2023-10-30T13:10:35.920225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:02.812352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758296+0000","last_change":"2023-10-30T13:10:34.344864+0000","last_active":"2023-10-30T13:10:35.758296+0000","last_peered":"2023-10-30T13:10:35.758296+0000","last_clean":"2023-10-30T13:10:35.758296+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:10:35.758296+0000","last_undegraded":"2023-10-30T13:10:35.758296+0000","last_fullsized":"2023-10-30T13:10:35.758296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:13.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923616+0000","last_change":"2023-10-30T13:10:34.364069+0000","last_active":"2023-10-30T13:10:35.923616+0000","last_peered":"2023-10-30T13:10:35.923616+0000","last_clean":"2023-10-30T13:10:35.923616+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:10:35.923616+0000","last_undegraded":"2023-10-30T13:10:35.923616+0000","last_fullsized":"2023-10-30T13:10:35.923616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:15.717336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319281+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:10:35.319281+0000","last_peered":"2023-10-30T13:10:35.319281+0000","last_clean":"2023-10-30T13:10:35.319281+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:10:35.319281+0000","last_undegraded":"2023-10-30T13:10:35.319281+0000","last_fullsized":"2023-10-30T13:10:35.319281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920436+0000","last_change":"2023-10-30T13:10:34.360935+0000","last_active":"2023-10-30T13:10:35.920436+0000","last_peered":"2023-10-30T13:10:35.920436+0000","last_clean":"2023-10-30T13:10:35.920436+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:10:35.920436+0000","last_undegraded":"2023-10-30T13:10:35.920436+0000","last_fullsized":"2023-10-30T13:10:35.920436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:58.639753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758265+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:10:35.758265+0000","last_peered":"2023-10-30T13:10:35.758265+0000","last_clean":"2023-10-30T13:10:35.758265+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:10:35.758265+0000","last_undegraded":"2023-10-30T13:10:35.758265+0000","last_fullsized":"2023-10-30T13:10:35.758265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318706+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:10:35.318706+0000","last_peered":"2023-10-30T13:10:35.318706+0000","last_clean":"2023-10-30T13:10:35.318706+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:10:35.318706+0000","last_undegraded":"2023-10-30T13:10:35.318706+0000","last_fullsized":"2023-10-30T13:10:35.318706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318630+0000","last_change":"2023-10-30T13:10:34.329935+0000","last_active":"2023-10-30T13:10:35.318630+0000","last_peered":"2023-10-30T13:10:35.318630+0000","last_clean":"2023-10-30T13:10:35.318630+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:10:35.318630+0000","last_undegraded":"2023-10-30T13:10:35.318630+0000","last_fullsized":"2023-10-30T13:10:35.318630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:07.795163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322575+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:10:35.322575+0000","last_peered":"2023-10-30T13:10:35.322575+0000","last_clean":"2023-10-30T13:10:35.322575+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:10:35.322575+0000","last_undegraded":"2023-10-30T13:10:35.322575+0000","last_fullsized":"2023-10-30T13:10:35.322575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919125+0000","last_change":"2023-10-30T13:10:34.331351+0000","last_active":"2023-10-30T13:10:35.919125+0000","last_peered":"2023-10-30T13:10:35.919125+0000","last_clean":"2023-10-30T13:10:35.919125+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:10:35.919125+0000","last_undegraded":"2023-10-30T13:10:35.919125+0000","last_fullsized":"2023-10-30T13:10:35.919125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:27.864073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759011+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:10:35.759011+0000","last_peered":"2023-10-30T13:10:35.759011+0000","last_clean":"2023-10-30T13:10:35.759011+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:10:35.759011+0000","last_undegraded":"2023-10-30T13:10:35.759011+0000","last_fullsized":"2023-10-30T13:10:35.759011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323392+0000","last_change":"2023-10-30T13:10:34.371882+0000","last_active":"2023-10-30T13:10:35.323392+0000","last_peered":"2023-10-30T13:10:35.323392+0000","last_clean":"2023-10-30T13:10:35.323392+0000","last_became_active":"2023-10-30T13:10:34.371138+0000","last_became_peered":"2023-10-30T13:10:34.371138+0000","last_unstale":"2023-10-30T13:10:35.323392+0000","last_undegraded":"2023-10-30T13:10:35.323392+0000","last_fullsized":"2023-10-30T13:10:35.323392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:16.287528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323317+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:10:35.323317+0000","last_peered":"2023-10-30T13:10:35.323317+0000","last_clean":"2023-10-30T13:10:35.323317+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:10:35.323317+0000","last_undegraded":"2023-10-30T13:10:35.323317+0000","last_fullsized":"2023-10-30T13:10:35.323317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920254+0000","last_change":"2023-10-30T13:10:34.335243+0000","last_active":"2023-10-30T13:10:35.920254+0000","last_peered":"2023-10-30T13:10:35.920254+0000","last_clean":"2023-10-30T13:10:35.920254+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:10:35.920254+0000","last_undegraded":"2023-10-30T13:10:35.920254+0000","last_fullsized":"2023-10-30T13:10:35.920254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:02.970675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919295+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:10:35.919295+0000","last_peered":"2023-10-30T13:10:35.919295+0000","last_clean":"2023-10-30T13:10:35.919295+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:10:35.919295+0000","last_undegraded":"2023-10-30T13:10:35.919295+0000","last_fullsized":"2023-10-30T13:10:35.919295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924235+0000","last_change":"2023-10-30T13:10:34.381099+0000","last_active":"2023-10-30T13:10:35.924235+0000","last_peered":"2023-10-30T13:10:35.924235+0000","last_clean":"2023-10-30T13:10:35.924235+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:10:35.924235+0000","last_undegraded":"2023-10-30T13:10:35.924235+0000","last_fullsized":"2023-10-30T13:10:35.924235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:16.694101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758757+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:10:35.758757+0000","last_peered":"2023-10-30T13:10:35.758757+0000","last_clean":"2023-10-30T13:10:35.758757+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:10:35.758757+0000","last_undegraded":"2023-10-30T13:10:35.758757+0000","last_fullsized":"2023-10-30T13:10:35.758757+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322690+0000","last_change":"2023-10-30T13:10:34.372219+0000","last_active":"2023-10-30T13:10:35.322690+0000","last_peered":"2023-10-30T13:10:35.322690+0000","last_clean":"2023-10-30T13:10:35.322690+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:10:35.322690+0000","last_undegraded":"2023-10-30T13:10:35.322690+0000","last_fullsized":"2023-10-30T13:10:35.322690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:38.329952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920461+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:10:35.920461+0000","last_peered":"2023-10-30T13:10:35.920461+0000","last_clean":"2023-10-30T13:10:35.920461+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:10:35.920461+0000","last_undegraded":"2023-10-30T13:10:35.920461+0000","last_fullsized":"2023-10-30T13:10:35.920461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920288+0000","last_change":"2023-10-30T13:10:34.368736+0000","last_active":"2023-10-30T13:10:35.920288+0000","last_peered":"2023-10-30T13:10:35.920288+0000","last_clean":"2023-10-30T13:10:35.920288+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:10:35.920288+0000","last_undegraded":"2023-10-30T13:10:35.920288+0000","last_fullsized":"2023-10-30T13:10:35.920288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:11.128769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323758+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:10:35.323758+0000","last_peered":"2023-10-30T13:10:35.323758+0000","last_clean":"2023-10-30T13:10:35.323758+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:10:35.323758+0000","last_undegraded":"2023-10-30T13:10:35.323758+0000","last_fullsized":"2023-10-30T13:10:35.323758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322654+0000","last_change":"2023-10-30T13:10:34.376512+0000","last_active":"2023-10-30T13:10:35.322654+0000","last_peered":"2023-10-30T13:10:35.322654+0000","last_clean":"2023-10-30T13:10:35.322654+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:10:35.322654+0000","last_undegraded":"2023-10-30T13:10:35.322654+0000","last_fullsized":"2023-10-30T13:10:35.322654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.856690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758265+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:10:35.758265+0000","last_peered":"2023-10-30T13:10:35.758265+0000","last_clean":"2023-10-30T13:10:35.758265+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:10:35.758265+0000","last_undegraded":"2023-10-30T13:10:35.758265+0000","last_fullsized":"2023-10-30T13:10:35.758265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758371+0000","last_change":"2023-10-30T13:10:34.344863+0000","last_active":"2023-10-30T13:10:35.758371+0000","last_peered":"2023-10-30T13:10:35.758371+0000","last_clean":"2023-10-30T13:10:35.758371+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:10:35.758371+0000","last_undegraded":"2023-10-30T13:10:35.758371+0000","last_fullsized":"2023-10-30T13:10:35.758371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:25.199813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919770+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:10:35.919770+0000","last_peered":"2023-10-30T13:10:35.919770+0000","last_clean":"2023-10-30T13:10:35.919770+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:10:35.919770+0000","last_undegraded":"2023-10-30T13:10:35.919770+0000","last_fullsized":"2023-10-30T13:10:35.919770+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323513+0000","last_change":"2023-10-30T13:10:34.371813+0000","last_active":"2023-10-30T13:10:35.323513+0000","last_peered":"2023-10-30T13:10:35.323513+0000","last_clean":"2023-10-30T13:10:35.323513+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:10:35.323513+0000","last_undegraded":"2023-10-30T13:10:35.323513+0000","last_fullsized":"2023-10-30T13:10:35.323513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:58.719747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924105+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:10:35.924105+0000","last_peered":"2023-10-30T13:10:35.924105+0000","last_clean":"2023-10-30T13:10:35.924105+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:10:35.924105+0000","last_undegraded":"2023-10-30T13:10:35.924105+0000","last_fullsized":"2023-10-30T13:10:35.924105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322746+0000","last_change":"2023-10-30T13:10:34.375164+0000","last_active":"2023-10-30T13:10:35.322746+0000","last_peered":"2023-10-30T13:10:35.322746+0000","last_clean":"2023-10-30T13:10:35.322746+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:10:35.322746+0000","last_undegraded":"2023-10-30T13:10:35.322746+0000","last_fullsized":"2023-10-30T13:10:35.322746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:05.305348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920036+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:10:35.920036+0000","last_peered":"2023-10-30T13:10:35.920036+0000","last_clean":"2023-10-30T13:10:35.920036+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:10:35.920036+0000","last_undegraded":"2023-10-30T13:10:35.920036+0000","last_fullsized":"2023-10-30T13:10:35.920036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323211+0000","last_change":"2023-10-30T13:10:34.382785+0000","last_active":"2023-10-30T13:10:35.323211+0000","last_peered":"2023-10-30T13:10:35.323211+0000","last_clean":"2023-10-30T13:10:35.323211+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:10:35.323211+0000","last_undegraded":"2023-10-30T13:10:35.323211+0000","last_fullsized":"2023-10-30T13:10:35.323211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:50.203770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919847+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:10:35.919847+0000","last_peered":"2023-10-30T13:10:35.919847+0000","last_clean":"2023-10-30T13:10:35.919847+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:10:35.919847+0000","last_undegraded":"2023-10-30T13:10:35.919847+0000","last_fullsized":"2023-10-30T13:10:35.919847+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324805+0000","last_change":"2023-10-30T13:10:34.342901+0000","last_active":"2023-10-30T13:10:35.324805+0000","last_peered":"2023-10-30T13:10:35.324805+0000","last_clean":"2023-10-30T13:10:35.324805+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:10:35.324805+0000","last_undegraded":"2023-10-30T13:10:35.324805+0000","last_fullsized":"2023-10-30T13:10:35.324805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:27.227907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919963+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:10:35.919963+0000","last_peered":"2023-10-30T13:10:35.919963+0000","last_clean":"2023-10-30T13:10:35.919963+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:10:35.919963+0000","last_undegraded":"2023-10-30T13:10:35.919963+0000","last_fullsized":"2023-10-30T13:10:35.919963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323034+0000","last_change":"2023-10-30T13:10:34.373854+0000","last_active":"2023-10-30T13:10:35.323034+0000","last_peered":"2023-10-30T13:10:35.323034+0000","last_clean":"2023-10-30T13:10:35.323034+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:10:35.323034+0000","last_undegraded":"2023-10-30T13:10:35.323034+0000","last_fullsized":"2023-10-30T13:10:35.323034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:12.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:10:34.334682+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:10:34.334682+0000","last_peered":"2023-10-30T13:10:34.334682+0000","last_clean":"2023-10-30T13:10:34.334682+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:10:34.334682+0000","last_undegraded":"2023-10-30T13:10:34.334682+0000","last_fullsized":"2023-10-30T13:10:34.334682+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324107+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:10:35.324107+0000","last_peered":"2023-10-30T13:10:35.324107+0000","last_clean":"2023-10-30T13:10:35.324107+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:10:35.324107+0000","last_undegraded":"2023-10-30T13:10:35.324107+0000","last_fullsized":"2023-10-30T13:10:35.324107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318835+0000","last_change":"2023-10-30T13:10:34.348470+0000","last_active":"2023-10-30T13:10:35.318835+0000","last_peered":"2023-10-30T13:10:35.318835+0000","last_clean":"2023-10-30T13:10:35.318835+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:10:35.318835+0000","last_undegraded":"2023-10-30T13:10:35.318835+0000","last_fullsized":"2023-10-30T13:10:35.318835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:24.818359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321505+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:10:35.321505+0000","last_peered":"2023-10-30T13:10:35.321505+0000","last_clean":"2023-10-30T13:10:35.321505+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:10:35.321505+0000","last_undegraded":"2023-10-30T13:10:35.321505+0000","last_fullsized":"2023-10-30T13:10:35.321505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919520+0000","last_change":"2023-10-30T13:10:34.359740+0000","last_active":"2023-10-30T13:10:35.919520+0000","last_peered":"2023-10-30T13:10:35.919520+0000","last_clean":"2023-10-30T13:10:35.919520+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:10:35.919520+0000","last_undegraded":"2023-10-30T13:10:35.919520+0000","last_fullsized":"2023-10-30T13:10:35.919520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:05:25.883485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919976+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:10:35.919976+0000","last_peered":"2023-10-30T13:10:35.919976+0000","last_clean":"2023-10-30T13:10:35.919976+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:10:35.919976+0000","last_undegraded":"2023-10-30T13:10:35.919976+0000","last_fullsized":"2023-10-30T13:10:35.919976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919216+0000","last_change":"2023-10-30T13:10:34.332955+0000","last_active":"2023-10-30T13:10:35.919216+0000","last_peered":"2023-10-30T13:10:35.919216+0000","last_clean":"2023-10-30T13:10:35.919216+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:10:35.919216+0000","last_undegraded":"2023-10-30T13:10:35.919216+0000","last_fullsized":"2023-10-30T13:10:35.919216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:05.776190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323974+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:10:35.323974+0000","last_peered":"2023-10-30T13:10:35.323974+0000","last_clean":"2023-10-30T13:10:35.323974+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:10:35.323974+0000","last_undegraded":"2023-10-30T13:10:35.323974+0000","last_fullsized":"2023-10-30T13:10:35.323974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923916+0000","last_change":"2023-10-30T13:10:34.362313+0000","last_active":"2023-10-30T13:10:35.923916+0000","last_peered":"2023-10-30T13:10:35.923916+0000","last_clean":"2023-10-30T13:10:35.923916+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:10:35.923916+0000","last_undegraded":"2023-10-30T13:10:35.923916+0000","last_fullsized":"2023-10-30T13:10:35.923916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:45.213916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919650+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:10:35.919650+0000","last_peered":"2023-10-30T13:10:35.919650+0000","last_clean":"2023-10-30T13:10:35.919650+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:10:35.919650+0000","last_undegraded":"2023-10-30T13:10:35.919650+0000","last_fullsized":"2023-10-30T13:10:35.919650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323629+0000","last_change":"2023-10-30T13:10:34.348653+0000","last_active":"2023-10-30T13:10:35.323629+0000","last_peered":"2023-10-30T13:10:35.323629+0000","last_clean":"2023-10-30T13:10:35.323629+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:10:35.323629+0000","last_undegraded":"2023-10-30T13:10:35.323629+0000","last_fullsized":"2023-10-30T13:10:35.323629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:54:00.948062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323802+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:10:35.323802+0000","last_peered":"2023-10-30T13:10:35.323802+0000","last_clean":"2023-10-30T13:10:35.323802+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:10:35.323802+0000","last_undegraded":"2023-10-30T13:10:35.323802+0000","last_fullsized":"2023-10-30T13:10:35.323802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919933+0000","last_change":"2023-10-30T13:10:34.332001+0000","last_active":"2023-10-30T13:10:35.919933+0000","last_peered":"2023-10-30T13:10:35.919933+0000","last_clean":"2023-10-30T13:10:35.919933+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:10:35.919933+0000","last_undegraded":"2023-10-30T13:10:35.919933+0000","last_fullsized":"2023-10-30T13:10:35.919933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:16.341054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920027+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:10:35.920027+0000","last_peered":"2023-10-30T13:10:35.920027+0000","last_clean":"2023-10-30T13:10:35.920027+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:10:35.920027+0000","last_undegraded":"2023-10-30T13:10:35.920027+0000","last_fullsized":"2023-10-30T13:10:35.920027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319328+0000","last_change":"2023-10-30T13:10:34.369845+0000","last_active":"2023-10-30T13:10:35.319328+0000","last_peered":"2023-10-30T13:10:35.319328+0000","last_clean":"2023-10-30T13:10:35.319328+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:10:35.319328+0000","last_undegraded":"2023-10-30T13:10:35.319328+0000","last_fullsized":"2023-10-30T13:10:35.319328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:30.468845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759001+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:10:35.759001+0000","last_peered":"2023-10-30T13:10:35.759001+0000","last_clean":"2023-10-30T13:10:35.759001+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:10:35.759001+0000","last_undegraded":"2023-10-30T13:10:35.759001+0000","last_fullsized":"2023-10-30T13:10:35.759001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920190+0000","last_change":"2023-10-30T13:10:34.323336+0000","last_active":"2023-10-30T13:10:35.920190+0000","last_peered":"2023-10-30T13:10:35.920190+0000","last_clean":"2023-10-30T13:10:35.920190+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:10:35.920190+0000","last_undegraded":"2023-10-30T13:10:35.920190+0000","last_fullsized":"2023-10-30T13:10:35.920190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:21.642241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321410+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:10:35.321410+0000","last_peered":"2023-10-30T13:10:35.321410+0000","last_clean":"2023-10-30T13:10:35.321410+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:10:35.321410+0000","last_undegraded":"2023-10-30T13:10:35.321410+0000","last_fullsized":"2023-10-30T13:10:35.321410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923797+0000","last_change":"2023-10-30T13:10:34.363313+0000","last_active":"2023-10-30T13:10:35.923797+0000","last_peered":"2023-10-30T13:10:35.923797+0000","last_clean":"2023-10-30T13:10:35.923797+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:10:35.923797+0000","last_undegraded":"2023-10-30T13:10:35.923797+0000","last_fullsized":"2023-10-30T13:10:35.923797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:13.027610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919211+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:10:35.919211+0000","last_peered":"2023-10-30T13:10:35.919211+0000","last_clean":"2023-10-30T13:10:35.919211+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:10:35.919211+0000","last_undegraded":"2023-10-30T13:10:35.919211+0000","last_fullsized":"2023-10-30T13:10:35.919211+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318724+0000","last_change":"2023-10-30T13:10:34.330677+0000","last_active":"2023-10-30T13:10:35.318724+0000","last_peered":"2023-10-30T13:10:35.318724+0000","last_clean":"2023-10-30T13:10:35.318724+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:10:35.318724+0000","last_undegraded":"2023-10-30T13:10:35.318724+0000","last_fullsized":"2023-10-30T13:10:35.318724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:01.360813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919383+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:10:35.919383+0000","last_peered":"2023-10-30T13:10:35.919383+0000","last_clean":"2023-10-30T13:10:35.919383+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:10:35.919383+0000","last_undegraded":"2023-10-30T13:10:35.919383+0000","last_fullsized":"2023-10-30T13:10:35.919383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319182+0000","last_change":"2023-10-30T13:10:34.331117+0000","last_active":"2023-10-30T13:10:35.319182+0000","last_peered":"2023-10-30T13:10:35.319182+0000","last_clean":"2023-10-30T13:10:35.319182+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:10:35.319182+0000","last_undegraded":"2023-10-30T13:10:35.319182+0000","last_fullsized":"2023-10-30T13:10:35.319182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:56.948470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323017+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:10:35.323017+0000","last_peered":"2023-10-30T13:10:35.323017+0000","last_clean":"2023-10-30T13:10:35.323017+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:10:35.323017+0000","last_undegraded":"2023-10-30T13:10:35.323017+0000","last_fullsized":"2023-10-30T13:10:35.323017+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758411+0000","last_change":"2023-10-30T13:10:34.365315+0000","last_active":"2023-10-30T13:10:35.758411+0000","last_peered":"2023-10-30T13:10:35.758411+0000","last_clean":"2023-10-30T13:10:35.758411+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:10:35.758411+0000","last_undegraded":"2023-10-30T13:10:35.758411+0000","last_fullsized":"2023-10-30T13:10:35.758411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:37.078775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323303+0000","last_change":"2023-10-30T13:10:34.366044+0000","last_active":"2023-10-30T13:10:35.323303+0000","last_peered":"2023-10-30T13:10:35.323303+0000","last_clean":"2023-10-30T13:10:35.323303+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:10:35.323303+0000","last_undegraded":"2023-10-30T13:10:35.323303+0000","last_fullsized":"2023-10-30T13:10:35.323303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:31.831020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322405+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:10:35.322405+0000","last_peered":"2023-10-30T13:10:35.322405+0000","last_clean":"2023-10-30T13:10:35.322405+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:10:35.322405+0000","last_undegraded":"2023-10-30T13:10:35.322405+0000","last_fullsized":"2023-10-30T13:10:35.322405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324679+0000","last_change":"2023-10-30T13:10:34.369814+0000","last_active":"2023-10-30T13:10:35.324679+0000","last_peered":"2023-10-30T13:10:35.324679+0000","last_clean":"2023-10-30T13:10:35.324679+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:10:35.324679+0000","last_undegraded":"2023-10-30T13:10:35.324679+0000","last_fullsized":"2023-10-30T13:10:35.324679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:26.879717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323260+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:10:35.323260+0000","last_peered":"2023-10-30T13:10:35.323260+0000","last_clean":"2023-10-30T13:10:35.323260+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:10:35.323260+0000","last_undegraded":"2023-10-30T13:10:35.323260+0000","last_fullsized":"2023-10-30T13:10:35.323260+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919435+0000","last_change":"2023-10-30T13:10:34.344766+0000","last_active":"2023-10-30T13:10:35.919435+0000","last_peered":"2023-10-30T13:10:35.919435+0000","last_clean":"2023-10-30T13:10:35.919435+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:10:35.919435+0000","last_undegraded":"2023-10-30T13:10:35.919435+0000","last_fullsized":"2023-10-30T13:10:35.919435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.073346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319295+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:10:35.319295+0000","last_peered":"2023-10-30T13:10:35.319295+0000","last_clean":"2023-10-30T13:10:35.319295+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:10:35.319295+0000","last_undegraded":"2023-10-30T13:10:35.319295+0000","last_fullsized":"2023-10-30T13:10:35.319295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319357+0000","last_change":"2023-10-30T13:10:34.354305+0000","last_active":"2023-10-30T13:10:35.319357+0000","last_peered":"2023-10-30T13:10:35.319357+0000","last_clean":"2023-10-30T13:10:35.319357+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:10:35.319357+0000","last_undegraded":"2023-10-30T13:10:35.319357+0000","last_fullsized":"2023-10-30T13:10:35.319357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:53.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920370+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:10:35.920370+0000","last_peered":"2023-10-30T13:10:35.920370+0000","last_clean":"2023-10-30T13:10:35.920370+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:10:35.920370+0000","last_undegraded":"2023-10-30T13:10:35.920370+0000","last_fullsized":"2023-10-30T13:10:35.920370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":798720,"data_stored":661266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.097}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":794624,"data_stored":660247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:10:40.780 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:10:40.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.948+0000 7f345309e640 1 Processor -- start 2023-10-30T13:10:40.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f345309e640 1 -- start start 2023-10-30T13:10:40.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f345309e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f344c0c05d0 0x7f344c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f345309e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f344c0c6f60 0x7f344c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f345309e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344c0c0a50 con 0x7f344c0c6b80 2023-10-30T13:10:40.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f345309e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344c06f6e0 con 0x7f344c0c05d0 2023-10-30T13:10:40.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f345309e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344c06f9c0 con 0x7f344c0c6f60 2023-10-30T13:10:40.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f3443fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f344c0c05d0 0x7f344c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f3443fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f344c0c05d0 0x7f344c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50646/0 (socket says 172.21.15.148:50646) 2023-10-30T13:10:40.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f3443fff640 1 -- 172.21.15.148:0/3925354 learned_addr learned my addr 172.21.15.148:0/3925354 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:40.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f345163b640 1 --2- 172.21.15.148:0/3925354 >> v2:172.21.15.150:3300/0 conn(0x7f344c0c6f60 0x7f344c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f3443fff640 1 -- 172.21.15.148:0/3925354 >> v2:172.21.15.150:3300/0 conn(0x7f344c0c6f60 msgr2=0x7f344c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:40.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f3443fff640 1 --2- 172.21.15.148:0/3925354 >> v2:172.21.15.150:3300/0 conn(0x7f344c0c6f60 0x7f344c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f3443fff640 1 -- 172.21.15.148:0/3925354 >> v1:172.21.15.150:6789/0 conn(0x7f344c0c6b80 legacy=0x7f344c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f3443fff640 1 -- 172.21.15.148:0/3925354 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c06fca0 con 0x7f344c0c05d0 2023-10-30T13:10:40.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.952+0000 7f345163b640 1 --2- 172.21.15.148:0/3925354 >> v2:172.21.15.150:3300/0 conn(0x7f344c0c6f60 0x7f344c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:40.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f3443fff640 1 --2- 172.21.15.148:0/3925354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c05d0 0x7f344c13ed90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f344c0c89b0 tx=0x7f343c01ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ae8aa11d65c7b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:40.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f34437fe640 1 -- 172.21.15.148:0/3925354 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c0a7bb0 con 0x7f344c0c05d0 2023-10-30T13:10:40.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f34437fe640 1 -- 172.21.15.148:0/3925354 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343c0a7d10 con 0x7f344c0c05d0 2023-10-30T13:10:40.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f34437fe640 1 -- 172.21.15.148:0/3925354 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c0af660 con 0x7f344c0c05d0 2023-10-30T13:10:40.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 -- 172.21.15.148:0/3925354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c05d0 msgr2=0x7f344c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 --2- 172.21.15.148:0/3925354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c05d0 0x7f344c13ed90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f344c0c89b0 tx=0x7f343c01ebd0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 -- 172.21.15.148:0/3925354 shutdown_connections 2023-10-30T13:10:40.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 --2- 172.21.15.148:0/3925354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c05d0 0x7f344c13ed90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 --2- 172.21.15.148:0/3925354 >> v2:172.21.15.150:3300/0 conn(0x7f344c0c6f60 0x7f344c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 -- 172.21.15.148:0/3925354 >> 172.21.15.148:0/3925354 conn(0x7f344c0c7910 msgr2=0x7f344c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:40.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 -- 172.21.15.148:0/3925354 shutdown_connections 2023-10-30T13:10:40.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 -- 172.21.15.148:0/3925354 wait complete. 2023-10-30T13:10:40.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 Processor -- start 2023-10-30T13:10:40.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.956+0000 7f345309e640 1 -- start start 2023-10-30T13:10:40.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345309e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f344c0c05d0 0x7f344c0ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345309e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f344c0c6b80 0x7f344c150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345309e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c6f60 0x7f344c158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345309e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0ca3f0 con 0x7f344c0c6f60 2023-10-30T13:10:40.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345309e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c13e8a0 con 0x7f344c0c05d0 2023-10-30T13:10:40.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345309e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0c9730 con 0x7f344c0c6b80 2023-10-30T13:10:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345163b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c6f60 0x7f344c158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345163b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c6f60 0x7f344c158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50650/0 (socket says 172.21.15.148:50650) 2023-10-30T13:10:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345163b640 1 -- 172.21.15.148:0/1279627434 learned_addr learned my addr 172.21.15.148:0/1279627434 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f3443fff640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f344c0c6b80 0x7f344c150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f3450e3a640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f344c0c05d0 0x7f344c0ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345163b640 1 -- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f344c0c6b80 msgr2=0x7f344c150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345163b640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f344c0c6b80 0x7f344c150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345163b640 1 -- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f344c0c05d0 msgr2=0x7f344c0ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:40.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345163b640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f344c0c05d0 0x7f344c0ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:40.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345163b640 1 -- 172.21.15.148:0/1279627434 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c00b770 con 0x7f344c0c6f60 2023-10-30T13:10:40.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345163b640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c6f60 0x7f344c158c50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3444007df0 tx=0x7f344400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:40.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f34417fa640 1 -- 172.21.15.148:0/1279627434 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34440039a0 con 0x7f344c0c6f60 2023-10-30T13:10:40.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f34417fa640 1 -- 172.21.15.148:0/1279627434 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344400f040 con 0x7f344c0c6f60 2023-10-30T13:10:40.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344c0c99f0 con 0x7f344c0c6f60 2023-10-30T13:10:40.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f34417fa640 1 -- 172.21.15.148:0/1279627434 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444003d40 con 0x7f344c0c6f60 2023-10-30T13:10:40.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344c159810 con 0x7f344c0c6f60 2023-10-30T13:10:40.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344c0ca530 con 0x7f344c0c6f60 2023-10-30T13:10:40.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.960+0000 7f34417fa640 1 -- 172.21.15.148:0/1279627434 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3444004030 con 0x7f344c0c6f60 2023-10-30T13:10:40.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.964+0000 7f34417fa640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34380664d0 0x7f3438068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:40.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.964+0000 7f34417fa640 1 -- 172.21.15.148:0/1279627434 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3444084130 con 0x7f344c0c6f60 2023-10-30T13:10:40.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.964+0000 7f3450e3a640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34380664d0 0x7f3438068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:40.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.964+0000 7f3450e3a640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34380664d0 0x7f3438068990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3448000f70 tx=0x7f3448009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:40.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:40.968+0000 7f34417fa640 1 -- 172.21.15.148:0/1279627434 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f344c0ca530 con 0x7f344c0c6f60 2023-10-30T13:10:41.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.116+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f344c0ca530 con 0x7f34380664d0 2023-10-30T13:10:41.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.132+0000 7f34417fa640 1 -- 172.21.15.148:0/1279627434 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264523 (secure 0 0 0) 0x7f344c0ca530 con 0x7f34380664d0 2023-10-30T13:10:41.141 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:41.142 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:10:41.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34380664d0 msgr2=0x7f3438068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34380664d0 0x7f3438068990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3448000f70 tx=0x7f3448009040 comp rx=0 tx=0).stop 2023-10-30T13:10:41.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c6f60 msgr2=0x7f344c158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c6f60 0x7f344c158c50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3444007df0 tx=0x7f344400d4a0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 shutdown_connections 2023-10-30T13:10:41.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34380664d0 0x7f3438068990 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f344c0c6f60 0x7f344c158c50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f344c0c6b80 0x7f344c150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 --2- 172.21.15.148:0/1279627434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f344c0c05d0 0x7f344c0ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 >> 172.21.15.148:0/1279627434 conn(0x7f344c0c7910 msgr2=0x7f344c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:41.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 shutdown_connections 2023-10-30T13:10:41.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.136+0000 7f345309e640 1 -- 172.21.15.148:0/1279627434 wait complete. 2023-10-30T13:10:41.161 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-30T13:10:40.262673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218928,"kb_used_data":2756,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755792,"statfs":{"total":773094113280,"available":772869931008,"internally_reserved":0,"allocated":2822144,"data_stored":1746901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.585543"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322509+0000","last_change":"2023-10-30T13:10:34.379031+0000","last_active":"2023-10-30T13:10:35.322509+0000","last_peered":"2023-10-30T13:10:35.322509+0000","last_clean":"2023-10-30T13:10:35.322509+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:10:35.322509+0000","last_undegraded":"2023-10-30T13:10:35.322509+0000","last_fullsized":"2023-10-30T13:10:35.322509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:32.529312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919896+0000","last_change":"2023-10-30T13:10:34.324308+0000","last_active":"2023-10-30T13:10:35.919896+0000","last_peered":"2023-10-30T13:10:35.919896+0000","last_clean":"2023-10-30T13:10:35.919896+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:10:35.919896+0000","last_undegraded":"2023-10-30T13:10:35.919896+0000","last_fullsized":"2023-10-30T13:10:35.919896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:21.885716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919373+0000","last_change":"2023-10-30T13:10:34.334122+0000","last_active":"2023-10-30T13:10:35.919373+0000","last_peered":"2023-10-30T13:10:35.919373+0000","last_clean":"2023-10-30T13:10:35.919373+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:10:35.919373+0000","last_undegraded":"2023-10-30T13:10:35.919373+0000","last_fullsized":"2023-10-30T13:10:35.919373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:55.045024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324579+0000","last_change":"2023-10-30T13:10:34.346795+0000","last_active":"2023-10-30T13:10:35.324579+0000","last_peered":"2023-10-30T13:10:35.324579+0000","last_clean":"2023-10-30T13:10:35.324579+0000","last_became_active":"2023-10-30T13:10:34.346079+0000","last_became_peered":"2023-10-30T13:10:34.346079+0000","last_unstale":"2023-10-30T13:10:35.324579+0000","last_undegraded":"2023-10-30T13:10:35.324579+0000","last_fullsized":"2023-10-30T13:10:35.324579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:09.546205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322339+0000","last_change":"2023-10-30T13:10:34.371670+0000","last_active":"2023-10-30T13:10:35.322339+0000","last_peered":"2023-10-30T13:10:35.322339+0000","last_clean":"2023-10-30T13:10:35.322339+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:10:35.322339+0000","last_undegraded":"2023-10-30T13:10:35.322339+0000","last_fullsized":"2023-10-30T13:10:35.322339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:03:54.773965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323062+0000","last_change":"2023-10-30T13:10:34.335752+0000","last_active":"2023-10-30T13:10:35.323062+0000","last_peered":"2023-10-30T13:10:35.323062+0000","last_clean":"2023-10-30T13:10:35.323062+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:10:35.323062+0000","last_undegraded":"2023-10-30T13:10:35.323062+0000","last_fullsized":"2023-10-30T13:10:35.323062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:33.278920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321759+0000","last_change":"2023-10-30T13:10:34.371817+0000","last_active":"2023-10-30T13:10:35.321759+0000","last_peered":"2023-10-30T13:10:35.321759+0000","last_clean":"2023-10-30T13:10:35.321759+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:10:35.321759+0000","last_undegraded":"2023-10-30T13:10:35.321759+0000","last_fullsized":"2023-10-30T13:10:35.321759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:33.797113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318581+0000","last_change":"2023-10-30T13:10:34.348205+0000","last_active":"2023-10-30T13:10:35.318581+0000","last_peered":"2023-10-30T13:10:35.318581+0000","last_clean":"2023-10-30T13:10:35.318581+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:10:35.318581+0000","last_undegraded":"2023-10-30T13:10:35.318581+0000","last_fullsized":"2023-10-30T13:10:35.318581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:57.492738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322864+0000","last_change":"2023-10-30T13:10:34.381052+0000","last_active":"2023-10-30T13:10:35.322864+0000","last_peered":"2023-10-30T13:10:35.322864+0000","last_clean":"2023-10-30T13:10:35.322864+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:10:35.322864+0000","last_undegraded":"2023-10-30T13:10:35.322864+0000","last_fullsized":"2023-10-30T13:10:35.322864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:16.697332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759029+0000","last_change":"2023-10-30T13:10:34.356932+0000","last_active":"2023-10-30T13:10:35.759029+0000","last_peered":"2023-10-30T13:10:35.759029+0000","last_clean":"2023-10-30T13:10:35.759029+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:10:35.759029+0000","last_undegraded":"2023-10-30T13:10:35.759029+0000","last_fullsized":"2023-10-30T13:10:35.759029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:54.091993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758774+0000","last_change":"2023-10-30T13:10:34.357358+0000","last_active":"2023-10-30T13:10:35.758774+0000","last_peered":"2023-10-30T13:10:35.758774+0000","last_clean":"2023-10-30T13:10:35.758774+0000","last_became_active":"2023-10-30T13:10:34.356466+0000","last_became_peered":"2023-10-30T13:10:34.356466+0000","last_unstale":"2023-10-30T13:10:35.758774+0000","last_undegraded":"2023-10-30T13:10:35.758774+0000","last_fullsized":"2023-10-30T13:10:35.758774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:06:24.029159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924241+0000","last_change":"2023-10-30T13:10:34.365161+0000","last_active":"2023-10-30T13:10:35.924241+0000","last_peered":"2023-10-30T13:10:35.924241+0000","last_clean":"2023-10-30T13:10:35.924241+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:10:35.924241+0000","last_undegraded":"2023-10-30T13:10:35.924241+0000","last_fullsized":"2023-10-30T13:10:35.924241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:51.721060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318828+0000","last_change":"2023-10-30T13:10:34.351901+0000","last_active":"2023-10-30T13:10:35.318828+0000","last_peered":"2023-10-30T13:10:35.318828+0000","last_clean":"2023-10-30T13:10:35.318828+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:10:35.318828+0000","last_undegraded":"2023-10-30T13:10:35.318828+0000","last_fullsized":"2023-10-30T13:10:35.318828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:11.576546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920191+0000","last_change":"2023-10-30T13:10:34.323358+0000","last_active":"2023-10-30T13:10:35.920191+0000","last_peered":"2023-10-30T13:10:35.920191+0000","last_clean":"2023-10-30T13:10:35.920191+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:10:35.920191+0000","last_undegraded":"2023-10-30T13:10:35.920191+0000","last_fullsized":"2023-10-30T13:10:35.920191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:03.178822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919173+0000","last_change":"2023-10-30T13:10:34.360120+0000","last_active":"2023-10-30T13:10:35.919173+0000","last_peered":"2023-10-30T13:10:35.919173+0000","last_clean":"2023-10-30T13:10:35.919173+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:10:35.919173+0000","last_undegraded":"2023-10-30T13:10:35.919173+0000","last_fullsized":"2023-10-30T13:10:35.919173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:30.731746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322343+0000","last_change":"2023-10-30T13:10:34.333752+0000","last_active":"2023-10-30T13:10:35.322343+0000","last_peered":"2023-10-30T13:10:35.322343+0000","last_clean":"2023-10-30T13:10:35.322343+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:10:35.322343+0000","last_undegraded":"2023-10-30T13:10:35.322343+0000","last_fullsized":"2023-10-30T13:10:35.322343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:45.390603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324451+0000","last_change":"2023-10-30T13:10:34.371931+0000","last_active":"2023-10-30T13:10:35.324451+0000","last_peered":"2023-10-30T13:10:35.324451+0000","last_clean":"2023-10-30T13:10:35.324451+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:10:35.324451+0000","last_undegraded":"2023-10-30T13:10:35.324451+0000","last_fullsized":"2023-10-30T13:10:35.324451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:51.514990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919968+0000","last_change":"2023-10-30T13:10:34.352747+0000","last_active":"2023-10-30T13:10:35.919968+0000","last_peered":"2023-10-30T13:10:35.919968+0000","last_clean":"2023-10-30T13:10:35.919968+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:10:35.919968+0000","last_undegraded":"2023-10-30T13:10:35.919968+0000","last_fullsized":"2023-10-30T13:10:35.919968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:21.543351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323598+0000","last_change":"2023-10-30T13:10:34.376135+0000","last_active":"2023-10-30T13:10:35.323598+0000","last_peered":"2023-10-30T13:10:35.323598+0000","last_clean":"2023-10-30T13:10:35.323598+0000","last_became_active":"2023-10-30T13:10:34.375497+0000","last_became_peered":"2023-10-30T13:10:34.375497+0000","last_unstale":"2023-10-30T13:10:35.323598+0000","last_undegraded":"2023-10-30T13:10:35.323598+0000","last_fullsized":"2023-10-30T13:10:35.323598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:14.816946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758767+0000","last_change":"2023-10-30T13:10:34.366575+0000","last_active":"2023-10-30T13:10:35.758767+0000","last_peered":"2023-10-30T13:10:35.758767+0000","last_clean":"2023-10-30T13:10:35.758767+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:10:35.758767+0000","last_undegraded":"2023-10-30T13:10:35.758767+0000","last_fullsized":"2023-10-30T13:10:35.758767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:23.540645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920318+0000","last_change":"2023-10-30T13:10:34.368761+0000","last_active":"2023-10-30T13:10:35.920318+0000","last_peered":"2023-10-30T13:10:35.920318+0000","last_clean":"2023-10-30T13:10:35.920318+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:10:35.920318+0000","last_undegraded":"2023-10-30T13:10:35.920318+0000","last_fullsized":"2023-10-30T13:10:35.920318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:32.861756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323430+0000","last_change":"2023-10-30T13:10:34.372881+0000","last_active":"2023-10-30T13:10:35.323430+0000","last_peered":"2023-10-30T13:10:35.323430+0000","last_clean":"2023-10-30T13:10:35.323430+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:10:35.323430+0000","last_undegraded":"2023-10-30T13:10:35.323430+0000","last_fullsized":"2023-10-30T13:10:35.323430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:23.855779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318722+0000","last_change":"2023-10-30T13:10:34.359359+0000","last_active":"2023-10-30T13:10:35.318722+0000","last_peered":"2023-10-30T13:10:35.318722+0000","last_clean":"2023-10-30T13:10:35.318722+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:10:35.318722+0000","last_undegraded":"2023-10-30T13:10:35.318722+0000","last_fullsized":"2023-10-30T13:10:35.318722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:08.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322387+0000","last_change":"2023-10-30T13:10:34.347431+0000","last_active":"2023-10-30T13:10:35.322387+0000","last_peered":"2023-10-30T13:10:35.322387+0000","last_clean":"2023-10-30T13:10:35.322387+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:10:35.322387+0000","last_undegraded":"2023-10-30T13:10:35.322387+0000","last_fullsized":"2023-10-30T13:10:35.322387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:25.094617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319296+0000","last_change":"2023-10-30T13:10:34.359717+0000","last_active":"2023-10-30T13:10:35.319296+0000","last_peered":"2023-10-30T13:10:35.319296+0000","last_clean":"2023-10-30T13:10:35.319296+0000","last_became_active":"2023-10-30T13:10:34.359063+0000","last_became_peered":"2023-10-30T13:10:34.359063+0000","last_unstale":"2023-10-30T13:10:35.319296+0000","last_undegraded":"2023-10-30T13:10:35.319296+0000","last_fullsized":"2023-10-30T13:10:35.319296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:22.525493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759209+0000","last_change":"2023-10-30T13:10:34.379878+0000","last_active":"2023-10-30T13:10:35.759209+0000","last_peered":"2023-10-30T13:10:35.759209+0000","last_clean":"2023-10-30T13:10:35.759209+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:10:35.759209+0000","last_undegraded":"2023-10-30T13:10:35.759209+0000","last_fullsized":"2023-10-30T13:10:35.759209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:42.280634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919210+0000","last_change":"2023-10-30T13:10:34.332020+0000","last_active":"2023-10-30T13:10:35.919210+0000","last_peered":"2023-10-30T13:10:35.919210+0000","last_clean":"2023-10-30T13:10:35.919210+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:10:35.919210+0000","last_undegraded":"2023-10-30T13:10:35.919210+0000","last_fullsized":"2023-10-30T13:10:35.919210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:53.057012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319283+0000","last_change":"2023-10-30T13:10:34.363154+0000","last_active":"2023-10-30T13:10:35.319283+0000","last_peered":"2023-10-30T13:10:35.319283+0000","last_clean":"2023-10-30T13:10:35.319283+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:10:35.319283+0000","last_undegraded":"2023-10-30T13:10:35.319283+0000","last_fullsized":"2023-10-30T13:10:35.319283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:04.359629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920381+0000","last_change":"2023-10-30T13:10:34.332160+0000","last_active":"2023-10-30T13:10:35.920381+0000","last_peered":"2023-10-30T13:10:35.920381+0000","last_clean":"2023-10-30T13:10:35.920381+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:10:35.920381+0000","last_undegraded":"2023-10-30T13:10:35.920381+0000","last_fullsized":"2023-10-30T13:10:35.920381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:28.128344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323753+0000","last_change":"2023-10-30T13:10:34.337617+0000","last_active":"2023-10-30T13:10:35.323753+0000","last_peered":"2023-10-30T13:10:35.323753+0000","last_clean":"2023-10-30T13:10:35.323753+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:10:35.323753+0000","last_undegraded":"2023-10-30T13:10:35.323753+0000","last_fullsized":"2023-10-30T13:10:35.323753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:10.287441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920225+0000","last_change":"2023-10-30T13:10:34.359681+0000","last_active":"2023-10-30T13:10:35.920225+0000","last_peered":"2023-10-30T13:10:35.920225+0000","last_clean":"2023-10-30T13:10:35.920225+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:10:35.920225+0000","last_undegraded":"2023-10-30T13:10:35.920225+0000","last_fullsized":"2023-10-30T13:10:35.920225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:02.812352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758296+0000","last_change":"2023-10-30T13:10:34.344864+0000","last_active":"2023-10-30T13:10:35.758296+0000","last_peered":"2023-10-30T13:10:35.758296+0000","last_clean":"2023-10-30T13:10:35.758296+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:10:35.758296+0000","last_undegraded":"2023-10-30T13:10:35.758296+0000","last_fullsized":"2023-10-30T13:10:35.758296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:13.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923616+0000","last_change":"2023-10-30T13:10:34.364069+0000","last_active":"2023-10-30T13:10:35.923616+0000","last_peered":"2023-10-30T13:10:35.923616+0000","last_clean":"2023-10-30T13:10:35.923616+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:10:35.923616+0000","last_undegraded":"2023-10-30T13:10:35.923616+0000","last_fullsized":"2023-10-30T13:10:35.923616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:15.717336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319281+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:10:35.319281+0000","last_peered":"2023-10-30T13:10:35.319281+0000","last_clean":"2023-10-30T13:10:35.319281+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:10:35.319281+0000","last_undegraded":"2023-10-30T13:10:35.319281+0000","last_fullsized":"2023-10-30T13:10:35.319281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920436+0000","last_change":"2023-10-30T13:10:34.360935+0000","last_active":"2023-10-30T13:10:35.920436+0000","last_peered":"2023-10-30T13:10:35.920436+0000","last_clean":"2023-10-30T13:10:35.920436+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:10:35.920436+0000","last_undegraded":"2023-10-30T13:10:35.920436+0000","last_fullsized":"2023-10-30T13:10:35.920436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:58.639753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758265+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:10:35.758265+0000","last_peered":"2023-10-30T13:10:35.758265+0000","last_clean":"2023-10-30T13:10:35.758265+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:10:35.758265+0000","last_undegraded":"2023-10-30T13:10:35.758265+0000","last_fullsized":"2023-10-30T13:10:35.758265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318706+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:10:35.318706+0000","last_peered":"2023-10-30T13:10:35.318706+0000","last_clean":"2023-10-30T13:10:35.318706+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:10:35.318706+0000","last_undegraded":"2023-10-30T13:10:35.318706+0000","last_fullsized":"2023-10-30T13:10:35.318706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318630+0000","last_change":"2023-10-30T13:10:34.329935+0000","last_active":"2023-10-30T13:10:35.318630+0000","last_peered":"2023-10-30T13:10:35.318630+0000","last_clean":"2023-10-30T13:10:35.318630+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:10:35.318630+0000","last_undegraded":"2023-10-30T13:10:35.318630+0000","last_fullsized":"2023-10-30T13:10:35.318630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:07.795163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322575+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:10:35.322575+0000","last_peered":"2023-10-30T13:10:35.322575+0000","last_clean":"2023-10-30T13:10:35.322575+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:10:35.322575+0000","last_undegraded":"2023-10-30T13:10:35.322575+0000","last_fullsized":"2023-10-30T13:10:35.322575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919125+0000","last_change":"2023-10-30T13:10:34.331351+0000","last_active":"2023-10-30T13:10:35.919125+0000","last_peered":"2023-10-30T13:10:35.919125+0000","last_clean":"2023-10-30T13:10:35.919125+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:10:35.919125+0000","last_undegraded":"2023-10-30T13:10:35.919125+0000","last_fullsized":"2023-10-30T13:10:35.919125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:27.864073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759011+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:10:35.759011+0000","last_peered":"2023-10-30T13:10:35.759011+0000","last_clean":"2023-10-30T13:10:35.759011+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:10:35.759011+0000","last_undegraded":"2023-10-30T13:10:35.759011+0000","last_fullsized":"2023-10-30T13:10:35.759011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323392+0000","last_change":"2023-10-30T13:10:34.371882+0000","last_active":"2023-10-30T13:10:35.323392+0000","last_peered":"2023-10-30T13:10:35.323392+0000","last_clean":"2023-10-30T13:10:35.323392+0000","last_became_active":"2023-10-30T13:10:34.371138+0000","last_became_peered":"2023-10-30T13:10:34.371138+0000","last_unstale":"2023-10-30T13:10:35.323392+0000","last_undegraded":"2023-10-30T13:10:35.323392+0000","last_fullsized":"2023-10-30T13:10:35.323392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:16.287528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323317+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:10:35.323317+0000","last_peered":"2023-10-30T13:10:35.323317+0000","last_clean":"2023-10-30T13:10:35.323317+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:10:35.323317+0000","last_undegraded":"2023-10-30T13:10:35.323317+0000","last_fullsized":"2023-10-30T13:10:35.323317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920254+0000","last_change":"2023-10-30T13:10:34.335243+0000","last_active":"2023-10-30T13:10:35.920254+0000","last_peered":"2023-10-30T13:10:35.920254+0000","last_clean":"2023-10-30T13:10:35.920254+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:10:35.920254+0000","last_undegraded":"2023-10-30T13:10:35.920254+0000","last_fullsized":"2023-10-30T13:10:35.920254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:02.970675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919295+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:10:35.919295+0000","last_peered":"2023-10-30T13:10:35.919295+0000","last_clean":"2023-10-30T13:10:35.919295+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:10:35.919295+0000","last_undegraded":"2023-10-30T13:10:35.919295+0000","last_fullsized":"2023-10-30T13:10:35.919295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924235+0000","last_change":"2023-10-30T13:10:34.381099+0000","last_active":"2023-10-30T13:10:35.924235+0000","last_peered":"2023-10-30T13:10:35.924235+0000","last_clean":"2023-10-30T13:10:35.924235+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:10:35.924235+0000","last_undegraded":"2023-10-30T13:10:35.924235+0000","last_fullsized":"2023-10-30T13:10:35.924235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:16.694101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758757+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:10:35.758757+0000","last_peered":"2023-10-30T13:10:35.758757+0000","last_clean":"2023-10-30T13:10:35.758757+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:10:35.758757+0000","last_undegraded":"2023-10-30T13:10:35.758757+0000","last_fullsized":"2023-10-30T13:10:35.758757+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322690+0000","last_change":"2023-10-30T13:10:34.372219+0000","last_active":"2023-10-30T13:10:35.322690+0000","last_peered":"2023-10-30T13:10:35.322690+0000","last_clean":"2023-10-30T13:10:35.322690+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:10:35.322690+0000","last_undegraded":"2023-10-30T13:10:35.322690+0000","last_fullsized":"2023-10-30T13:10:35.322690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:38.329952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920461+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:10:35.920461+0000","last_peered":"2023-10-30T13:10:35.920461+0000","last_clean":"2023-10-30T13:10:35.920461+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:10:35.920461+0000","last_undegraded":"2023-10-30T13:10:35.920461+0000","last_fullsized":"2023-10-30T13:10:35.920461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920288+0000","last_change":"2023-10-30T13:10:34.368736+0000","last_active":"2023-10-30T13:10:35.920288+0000","last_peered":"2023-10-30T13:10:35.920288+0000","last_clean":"2023-10-30T13:10:35.920288+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:10:35.920288+0000","last_undegraded":"2023-10-30T13:10:35.920288+0000","last_fullsized":"2023-10-30T13:10:35.920288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:11.128769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323758+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:10:35.323758+0000","last_peered":"2023-10-30T13:10:35.323758+0000","last_clean":"2023-10-30T13:10:35.323758+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:10:35.323758+0000","last_undegraded":"2023-10-30T13:10:35.323758+0000","last_fullsized":"2023-10-30T13:10:35.323758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322654+0000","last_change":"2023-10-30T13:10:34.376512+0000","last_active":"2023-10-30T13:10:35.322654+0000","last_peered":"2023-10-30T13:10:35.322654+0000","last_clean":"2023-10-30T13:10:35.322654+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:10:35.322654+0000","last_undegraded":"2023-10-30T13:10:35.322654+0000","last_fullsized":"2023-10-30T13:10:35.322654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.856690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758265+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:10:35.758265+0000","last_peered":"2023-10-30T13:10:35.758265+0000","last_clean":"2023-10-30T13:10:35.758265+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:10:35.758265+0000","last_undegraded":"2023-10-30T13:10:35.758265+0000","last_fullsized":"2023-10-30T13:10:35.758265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758371+0000","last_change":"2023-10-30T13:10:34.344863+0000","last_active":"2023-10-30T13:10:35.758371+0000","last_peered":"2023-10-30T13:10:35.758371+0000","last_clean":"2023-10-30T13:10:35.758371+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:10:35.758371+0000","last_undegraded":"2023-10-30T13:10:35.758371+0000","last_fullsized":"2023-10-30T13:10:35.758371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:25.199813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919770+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:10:35.919770+0000","last_peered":"2023-10-30T13:10:35.919770+0000","last_clean":"2023-10-30T13:10:35.919770+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:10:35.919770+0000","last_undegraded":"2023-10-30T13:10:35.919770+0000","last_fullsized":"2023-10-30T13:10:35.919770+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323513+0000","last_change":"2023-10-30T13:10:34.371813+0000","last_active":"2023-10-30T13:10:35.323513+0000","last_peered":"2023-10-30T13:10:35.323513+0000","last_clean":"2023-10-30T13:10:35.323513+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:10:35.323513+0000","last_undegraded":"2023-10-30T13:10:35.323513+0000","last_fullsized":"2023-10-30T13:10:35.323513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:58.719747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924105+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:10:35.924105+0000","last_peered":"2023-10-30T13:10:35.924105+0000","last_clean":"2023-10-30T13:10:35.924105+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:10:35.924105+0000","last_undegraded":"2023-10-30T13:10:35.924105+0000","last_fullsized":"2023-10-30T13:10:35.924105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322746+0000","last_change":"2023-10-30T13:10:34.375164+0000","last_active":"2023-10-30T13:10:35.322746+0000","last_peered":"2023-10-30T13:10:35.322746+0000","last_clean":"2023-10-30T13:10:35.322746+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:10:35.322746+0000","last_undegraded":"2023-10-30T13:10:35.322746+0000","last_fullsized":"2023-10-30T13:10:35.322746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:05.305348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920036+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:10:35.920036+0000","last_peered":"2023-10-30T13:10:35.920036+0000","last_clean":"2023-10-30T13:10:35.920036+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:10:35.920036+0000","last_undegraded":"2023-10-30T13:10:35.920036+0000","last_fullsized":"2023-10-30T13:10:35.920036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323211+0000","last_change":"2023-10-30T13:10:34.382785+0000","last_active":"2023-10-30T13:10:35.323211+0000","last_peered":"2023-10-30T13:10:35.323211+0000","last_clean":"2023-10-30T13:10:35.323211+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:10:35.323211+0000","last_undegraded":"2023-10-30T13:10:35.323211+0000","last_fullsized":"2023-10-30T13:10:35.323211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:50.203770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919847+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:10:35.919847+0000","last_peered":"2023-10-30T13:10:35.919847+0000","last_clean":"2023-10-30T13:10:35.919847+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:10:35.919847+0000","last_undegraded":"2023-10-30T13:10:35.919847+0000","last_fullsized":"2023-10-30T13:10:35.919847+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324805+0000","last_change":"2023-10-30T13:10:34.342901+0000","last_active":"2023-10-30T13:10:35.324805+0000","last_peered":"2023-10-30T13:10:35.324805+0000","last_clean":"2023-10-30T13:10:35.324805+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:10:35.324805+0000","last_undegraded":"2023-10-30T13:10:35.324805+0000","last_fullsized":"2023-10-30T13:10:35.324805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:27.227907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919963+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:10:35.919963+0000","last_peered":"2023-10-30T13:10:35.919963+0000","last_clean":"2023-10-30T13:10:35.919963+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:10:35.919963+0000","last_undegraded":"2023-10-30T13:10:35.919963+0000","last_fullsized":"2023-10-30T13:10:35.919963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323034+0000","last_change":"2023-10-30T13:10:34.373854+0000","last_active":"2023-10-30T13:10:35.323034+0000","last_peered":"2023-10-30T13:10:35.323034+0000","last_clean":"2023-10-30T13:10:35.323034+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:10:35.323034+0000","last_undegraded":"2023-10-30T13:10:35.323034+0000","last_fullsized":"2023-10-30T13:10:35.323034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:12.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:10:34.334682+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:10:34.334682+0000","last_peered":"2023-10-30T13:10:34.334682+0000","last_clean":"2023-10-30T13:10:34.334682+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:10:34.334682+0000","last_undegraded":"2023-10-30T13:10:34.334682+0000","last_fullsized":"2023-10-30T13:10:34.334682+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324107+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:10:35.324107+0000","last_peered":"2023-10-30T13:10:35.324107+0000","last_clean":"2023-10-30T13:10:35.324107+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:10:35.324107+0000","last_undegraded":"2023-10-30T13:10:35.324107+0000","last_fullsized":"2023-10-30T13:10:35.324107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318835+0000","last_change":"2023-10-30T13:10:34.348470+0000","last_active":"2023-10-30T13:10:35.318835+0000","last_peered":"2023-10-30T13:10:35.318835+0000","last_clean":"2023-10-30T13:10:35.318835+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:10:35.318835+0000","last_undegraded":"2023-10-30T13:10:35.318835+0000","last_fullsized":"2023-10-30T13:10:35.318835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:24.818359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321505+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:10:35.321505+0000","last_peered":"2023-10-30T13:10:35.321505+0000","last_clean":"2023-10-30T13:10:35.321505+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:10:35.321505+0000","last_undegraded":"2023-10-30T13:10:35.321505+0000","last_fullsized":"2023-10-30T13:10:35.321505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919520+0000","last_change":"2023-10-30T13:10:34.359740+0000","last_active":"2023-10-30T13:10:35.919520+0000","last_peered":"2023-10-30T13:10:35.919520+0000","last_clean":"2023-10-30T13:10:35.919520+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:10:35.919520+0000","last_undegraded":"2023-10-30T13:10:35.919520+0000","last_fullsized":"2023-10-30T13:10:35.919520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:05:25.883485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919976+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:10:35.919976+0000","last_peered":"2023-10-30T13:10:35.919976+0000","last_clean":"2023-10-30T13:10:35.919976+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:10:35.919976+0000","last_undegraded":"2023-10-30T13:10:35.919976+0000","last_fullsized":"2023-10-30T13:10:35.919976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919216+0000","last_change":"2023-10-30T13:10:34.332955+0000","last_active":"2023-10-30T13:10:35.919216+0000","last_peered":"2023-10-30T13:10:35.919216+0000","last_clean":"2023-10-30T13:10:35.919216+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:10:35.919216+0000","last_undegraded":"2023-10-30T13:10:35.919216+0000","last_fullsized":"2023-10-30T13:10:35.919216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:05.776190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323974+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:10:35.323974+0000","last_peered":"2023-10-30T13:10:35.323974+0000","last_clean":"2023-10-30T13:10:35.323974+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:10:35.323974+0000","last_undegraded":"2023-10-30T13:10:35.323974+0000","last_fullsized":"2023-10-30T13:10:35.323974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923916+0000","last_change":"2023-10-30T13:10:34.362313+0000","last_active":"2023-10-30T13:10:35.923916+0000","last_peered":"2023-10-30T13:10:35.923916+0000","last_clean":"2023-10-30T13:10:35.923916+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:10:35.923916+0000","last_undegraded":"2023-10-30T13:10:35.923916+0000","last_fullsized":"2023-10-30T13:10:35.923916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:45.213916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919650+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:10:35.919650+0000","last_peered":"2023-10-30T13:10:35.919650+0000","last_clean":"2023-10-30T13:10:35.919650+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:10:35.919650+0000","last_undegraded":"2023-10-30T13:10:35.919650+0000","last_fullsized":"2023-10-30T13:10:35.919650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323629+0000","last_change":"2023-10-30T13:10:34.348653+0000","last_active":"2023-10-30T13:10:35.323629+0000","last_peered":"2023-10-30T13:10:35.323629+0000","last_clean":"2023-10-30T13:10:35.323629+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:10:35.323629+0000","last_undegraded":"2023-10-30T13:10:35.323629+0000","last_fullsized":"2023-10-30T13:10:35.323629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:54:00.948062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323802+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:10:35.323802+0000","last_peered":"2023-10-30T13:10:35.323802+0000","last_clean":"2023-10-30T13:10:35.323802+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:10:35.323802+0000","last_undegraded":"2023-10-30T13:10:35.323802+0000","last_fullsized":"2023-10-30T13:10:35.323802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919933+0000","last_change":"2023-10-30T13:10:34.332001+0000","last_active":"2023-10-30T13:10:35.919933+0000","last_peered":"2023-10-30T13:10:35.919933+0000","last_clean":"2023-10-30T13:10:35.919933+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:10:35.919933+0000","last_undegraded":"2023-10-30T13:10:35.919933+0000","last_fullsized":"2023-10-30T13:10:35.919933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:16.341054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920027+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:10:35.920027+0000","last_peered":"2023-10-30T13:10:35.920027+0000","last_clean":"2023-10-30T13:10:35.920027+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:10:35.920027+0000","last_undegraded":"2023-10-30T13:10:35.920027+0000","last_fullsized":"2023-10-30T13:10:35.920027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319328+0000","last_change":"2023-10-30T13:10:34.369845+0000","last_active":"2023-10-30T13:10:35.319328+0000","last_peered":"2023-10-30T13:10:35.319328+0000","last_clean":"2023-10-30T13:10:35.319328+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:10:35.319328+0000","last_undegraded":"2023-10-30T13:10:35.319328+0000","last_fullsized":"2023-10-30T13:10:35.319328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:30.468845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759001+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:10:35.759001+0000","last_peered":"2023-10-30T13:10:35.759001+0000","last_clean":"2023-10-30T13:10:35.759001+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:10:35.759001+0000","last_undegraded":"2023-10-30T13:10:35.759001+0000","last_fullsized":"2023-10-30T13:10:35.759001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920190+0000","last_change":"2023-10-30T13:10:34.323336+0000","last_active":"2023-10-30T13:10:35.920190+0000","last_peered":"2023-10-30T13:10:35.920190+0000","last_clean":"2023-10-30T13:10:35.920190+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:10:35.920190+0000","last_undegraded":"2023-10-30T13:10:35.920190+0000","last_fullsized":"2023-10-30T13:10:35.920190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:21.642241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321410+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:10:35.321410+0000","last_peered":"2023-10-30T13:10:35.321410+0000","last_clean":"2023-10-30T13:10:35.321410+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:10:35.321410+0000","last_undegraded":"2023-10-30T13:10:35.321410+0000","last_fullsized":"2023-10-30T13:10:35.321410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923797+0000","last_change":"2023-10-30T13:10:34.363313+0000","last_active":"2023-10-30T13:10:35.923797+0000","last_peered":"2023-10-30T13:10:35.923797+0000","last_clean":"2023-10-30T13:10:35.923797+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:10:35.923797+0000","last_undegraded":"2023-10-30T13:10:35.923797+0000","last_fullsized":"2023-10-30T13:10:35.923797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:13.027610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919211+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:10:35.919211+0000","last_peered":"2023-10-30T13:10:35.919211+0000","last_clean":"2023-10-30T13:10:35.919211+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:10:35.919211+0000","last_undegraded":"2023-10-30T13:10:35.919211+0000","last_fullsized":"2023-10-30T13:10:35.919211+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318724+0000","last_change":"2023-10-30T13:10:34.330677+0000","last_active":"2023-10-30T13:10:35.318724+0000","last_peered":"2023-10-30T13:10:35.318724+0000","last_clean":"2023-10-30T13:10:35.318724+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:10:35.318724+0000","last_undegraded":"2023-10-30T13:10:35.318724+0000","last_fullsized":"2023-10-30T13:10:35.318724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:01.360813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919383+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:10:35.919383+0000","last_peered":"2023-10-30T13:10:35.919383+0000","last_clean":"2023-10-30T13:10:35.919383+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:10:35.919383+0000","last_undegraded":"2023-10-30T13:10:35.919383+0000","last_fullsized":"2023-10-30T13:10:35.919383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319182+0000","last_change":"2023-10-30T13:10:34.331117+0000","last_active":"2023-10-30T13:10:35.319182+0000","last_peered":"2023-10-30T13:10:35.319182+0000","last_clean":"2023-10-30T13:10:35.319182+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:10:35.319182+0000","last_undegraded":"2023-10-30T13:10:35.319182+0000","last_fullsized":"2023-10-30T13:10:35.319182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:56.948470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323017+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:10:35.323017+0000","last_peered":"2023-10-30T13:10:35.323017+0000","last_clean":"2023-10-30T13:10:35.323017+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:10:35.323017+0000","last_undegraded":"2023-10-30T13:10:35.323017+0000","last_fullsized":"2023-10-30T13:10:35.323017+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758411+0000","last_change":"2023-10-30T13:10:34.365315+0000","last_active":"2023-10-30T13:10:35.758411+0000","last_peered":"2023-10-30T13:10:35.758411+0000","last_clean":"2023-10-30T13:10:35.758411+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:10:35.758411+0000","last_undegraded":"2023-10-30T13:10:35.758411+0000","last_fullsized":"2023-10-30T13:10:35.758411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:37.078775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323303+0000","last_change":"2023-10-30T13:10:34.366044+0000","last_active":"2023-10-30T13:10:35.323303+0000","last_peered":"2023-10-30T13:10:35.323303+0000","last_clean":"2023-10-30T13:10:35.323303+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:10:35.323303+0000","last_undegraded":"2023-10-30T13:10:35.323303+0000","last_fullsized":"2023-10-30T13:10:35.323303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:31.831020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322405+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:10:35.322405+0000","last_peered":"2023-10-30T13:10:35.322405+0000","last_clean":"2023-10-30T13:10:35.322405+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:10:35.322405+0000","last_undegraded":"2023-10-30T13:10:35.322405+0000","last_fullsized":"2023-10-30T13:10:35.322405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324679+0000","last_change":"2023-10-30T13:10:34.369814+0000","last_active":"2023-10-30T13:10:35.324679+0000","last_peered":"2023-10-30T13:10:35.324679+0000","last_clean":"2023-10-30T13:10:35.324679+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:10:35.324679+0000","last_undegraded":"2023-10-30T13:10:35.324679+0000","last_fullsized":"2023-10-30T13:10:35.324679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:26.879717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323260+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:10:35.323260+0000","last_peered":"2023-10-30T13:10:35.323260+0000","last_clean":"2023-10-30T13:10:35.323260+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:10:35.323260+0000","last_undegraded":"2023-10-30T13:10:35.323260+0000","last_fullsized":"2023-10-30T13:10:35.323260+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919435+0000","last_change":"2023-10-30T13:10:34.344766+0000","last_active":"2023-10-30T13:10:35.919435+0000","last_peered":"2023-10-30T13:10:35.919435+0000","last_clean":"2023-10-30T13:10:35.919435+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:10:35.919435+0000","last_undegraded":"2023-10-30T13:10:35.919435+0000","last_fullsized":"2023-10-30T13:10:35.919435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.073346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319295+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:10:35.319295+0000","last_peered":"2023-10-30T13:10:35.319295+0000","last_clean":"2023-10-30T13:10:35.319295+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:10:35.319295+0000","last_undegraded":"2023-10-30T13:10:35.319295+0000","last_fullsized":"2023-10-30T13:10:35.319295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319357+0000","last_change":"2023-10-30T13:10:34.354305+0000","last_active":"2023-10-30T13:10:35.319357+0000","last_peered":"2023-10-30T13:10:35.319357+0000","last_clean":"2023-10-30T13:10:35.319357+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:10:35.319357+0000","last_undegraded":"2023-10-30T13:10:35.319357+0000","last_fullsized":"2023-10-30T13:10:35.319357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:53.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920370+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:10:35.920370+0000","last_peered":"2023-10-30T13:10:35.920370+0000","last_clean":"2023-10-30T13:10:35.920370+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:10:35.920370+0000","last_undegraded":"2023-10-30T13:10:35.920370+0000","last_fullsized":"2023-10-30T13:10:35.920370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":798720,"data_stored":661266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.097}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":794624,"data_stored":660247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:10:41.167 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-30T13:10:41.167 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T13:10:41.167 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T13:10:41.168 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T13:10:41.168 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T13:10:41.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe75c5640 1 Processor -- start 2023-10-30T13:10:41.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe75c5640 1 -- start start 2023-10-30T13:10:41.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe75c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffbe01538a0 0x7ffbe015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe75c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe00ca0e0 0x7ffbe015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe75c5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe00cb800 con 0x7ffbe00cf160 2023-10-30T13:10:41.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe75c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe015dc30 con 0x7ffbe01538a0 2023-10-30T13:10:41.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe75c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe00cb330 con 0x7ffbe00ca0e0 2023-10-30T13:10:41.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe5b62640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ffbe00cf160 0x7ffbe00cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34336/0 (socket says 172.21.15.148:34336) 2023-10-30T13:10:41.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe5b62640 1 -- 172.21.15.148:0/3314056427 learned_addr learned my addr 172.21.15.148:0/3314056427 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:41.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe5361640 1 --2- 172.21.15.148:0/3314056427 >> v2:172.21.15.150:3300/0 conn(0x7ffbe01538a0 0x7ffbe015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe4b60640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe00ca0e0 0x7ffbe015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738570076 0 0) 0x7ffbe00cb800 con 0x7ffbe00cf160 2023-10-30T13:10:41.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc003530 con 0x7ffbe00cf160 2023-10-30T13:10:41.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 802628209 0 0) 0x7ffbcc003530 con 0x7ffbe00cf160 2023-10-30T13:10:41.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe00ca0e0 msgr2=0x7ffbe015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:41.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe00ca0e0 0x7ffbe015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 >> v2:172.21.15.150:3300/0 conn(0x7ffbe01538a0 msgr2=0x7ffbe015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:41.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 --2- 172.21.15.148:0/3314056427 >> v2:172.21.15.150:3300/0 conn(0x7ffbe01538a0 0x7ffbe015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe00cad70 con 0x7ffbe00cf160 2023-10-30T13:10:41.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7ffbd8001a00 con 0x7ffbe00cf160 2023-10-30T13:10:41.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 >> v1:172.21.15.148:6789/0 conn(0x7ffbe00cf160 legacy=0x7ffbe00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe00ca0e0 0x7ffbcc003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbcc003f00 0x7ffbcc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbcc004d10 0x7ffbcc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe015dc30 con 0x7ffbcc004d10 2023-10-30T13:10:41.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe00cb330 con 0x7ffbcc003f00 2023-10-30T13:10:41.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc00b9c0 con 0x7ffbe00ca0e0 2023-10-30T13:10:41.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe5361640 1 --2- 172.21.15.148:0/3314056427 >> v2:172.21.15.150:3300/0 conn(0x7ffbe01538a0 0x7ffbe015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:41.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe5b62640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbcc004d10 0x7ffbcc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe4b60640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbcc003f00 0x7ffbcc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.336+0000 7ffbe5361640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe00ca0e0 0x7ffbcc003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe5b62640 1 -- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe00ca0e0 msgr2=0x7ffbcc003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:41.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe5b62640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe00ca0e0 0x7ffbcc003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe5b62640 1 -- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbcc003f00 msgr2=0x7ffbcc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:41.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe5b62640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbcc003f00 0x7ffbcc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe5b62640 1 -- 172.21.15.148:0/3314056427 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc003530 con 0x7ffbcc004d10 2023-10-30T13:10:41.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe5b62640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbcc004d10 0x7ffbcc005190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800a8d0 tx=0x7ffbd801d3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d6d9a17f7ae517 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:41.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd80ad630 con 0x7ffbcc004d10 2023-10-30T13:10:41.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd80a8070 con 0x7ffbcc004d10 2023-10-30T13:10:41.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbd7fff640 1 -- 172.21.15.148:0/3314056427 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd80ad9d0 con 0x7ffbcc004d10 2023-10-30T13:10:41.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe75c5640 1 -- 172.21.15.148:0/3314056427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbcc004d10 msgr2=0x7ffbcc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe75c5640 1 --2- 172.21.15.148:0/3314056427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbcc004d10 0x7ffbcc005190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800a8d0 tx=0x7ffbd801d3e0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe5b62640 1 -- 172.21.15.148:0/3314056427 reap_dead start 2023-10-30T13:10:41.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe75c5640 1 -- 172.21.15.148:0/3314056427 shutdown_connections 2023-10-30T13:10:41.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe75c5640 1 -- 172.21.15.148:0/3314056427 >> 172.21.15.148:0/3314056427 conn(0x7ffbe006bfe0 msgr2=0x7ffbe006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:41.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe75c5640 1 -- 172.21.15.148:0/3314056427 shutdown_connections 2023-10-30T13:10:41.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.340+0000 7ffbe75c5640 1 -- 172.21.15.148:0/3314056427 wait complete. 2023-10-30T13:10:41.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe75c5640 1 Processor -- start 2023-10-30T13:10:41.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe75c5640 1 -- start start 2023-10-30T13:10:41.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe75c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbcc004d10 0x7ffbe0158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe75c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe0149200 0x7ffbe015bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe75c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbe0149640 0x7ffbe015c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe75c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe015d1b0 con 0x7ffbe0149640 2023-10-30T13:10:41.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe75c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe01f2df0 con 0x7ffbcc004d10 2023-10-30T13:10:41.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe75c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe0158480 con 0x7ffbe0149200 2023-10-30T13:10:41.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5b62640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbe0149640 0x7ffbe015c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5b62640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbe0149640 0x7ffbe015c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50680/0 (socket says 172.21.15.148:50680) 2023-10-30T13:10:41.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5b62640 1 -- 172.21.15.148:0/94984987 learned_addr learned my addr 172.21.15.148:0/94984987 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:41.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5361640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbcc004d10 0x7ffbe0158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe4b60640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe0149200 0x7ffbe015bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5b62640 1 -- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe0149200 msgr2=0x7ffbe015bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5b62640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe0149200 0x7ffbe015bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5b62640 1 -- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbcc004d10 msgr2=0x7ffbe0158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5b62640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbcc004d10 0x7ffbe0158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5b62640 1 -- 172.21.15.148:0/94984987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe0158760 con 0x7ffbe0149640 2023-10-30T13:10:41.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe5b62640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbe0149640 0x7ffbe015c490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffbd801d210 tx=0x7ffbcc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:41.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbd5ffb640 1 -- 172.21.15.148:0/94984987 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd80017c0 con 0x7ffbe0149640 2023-10-30T13:10:41.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbd5ffb640 1 -- 172.21.15.148:0/94984987 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd8001920 con 0x7ffbe0149640 2023-10-30T13:10:41.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbd5ffb640 1 -- 172.21.15.148:0/94984987 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8004d70 con 0x7ffbe0149640 2023-10-30T13:10:41.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.344+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe015a8b0 con 0x7ffbe0149640 2023-10-30T13:10:41.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.348+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe01f2ae0 con 0x7ffbe0149640 2023-10-30T13:10:41.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.348+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe004e170 con 0x7ffbe0149640 2023-10-30T13:10:41.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.348+0000 7ffbd5ffb640 1 -- 172.21.15.148:0/94984987 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffbd80aa030 con 0x7ffbe0149640 2023-10-30T13:10:41.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.348+0000 7ffbd5ffb640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbc40664d0 0x7ffbc4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.348+0000 7ffbd5ffb640 1 -- 172.21.15.148:0/94984987 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbd811dbf0 con 0x7ffbe0149640 2023-10-30T13:10:41.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.348+0000 7ffbe5361640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbc40664d0 0x7ffbc4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.352+0000 7ffbe5361640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbc40664d0 0x7ffbc4068990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009c40 tx=0x7ffbdc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:41.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.352+0000 7ffbd5ffb640 1 -- 172.21.15.148:0/94984987 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffbd80eb110 con 0x7ffbe0149640 2023-10-30T13:10:41.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.536+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ffbe004e170 con 0x7ffbe0149640 2023-10-30T13:10:41.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.540+0000 7ffbd5ffb640 1 -- 172.21.15.148:0/94984987 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164436 (secure 0 0 0) 0x7ffbd80ee730 con 0x7ffbe0149640 2023-10-30T13:10:41.546 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:41.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbc40664d0 msgr2=0x7ffbc4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbc40664d0 0x7ffbc4068990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009c40 tx=0x7ffbdc008040 comp rx=0 tx=0).stop 2023-10-30T13:10:41.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbe0149640 msgr2=0x7ffbe015c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbe0149640 0x7ffbe015c490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffbd801d210 tx=0x7ffbcc005800 comp rx=0 tx=0).stop 2023-10-30T13:10:41.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 shutdown_connections 2023-10-30T13:10:41.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbc40664d0 0x7ffbc4068990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbe0149640 0x7ffbe015c490 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbe0149200 0x7ffbe015bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 --2- 172.21.15.148:0/94984987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbcc004d10 0x7ffbe0158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 >> 172.21.15.148:0/94984987 conn(0x7ffbe006bfe0 msgr2=0x7ffbe0152130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:41.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 shutdown_connections 2023-10-30T13:10:41.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.544+0000 7ffbe75c5640 1 -- 172.21.15.148:0/94984987 wait complete. 2023-10-30T13:10:41.567 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":5,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1299332652},{"type":"v1","addr":"172.21.15.150:6833","nonce":1299332652}]},"active_addr":"172.21.15.150:6833/1299332652","active_change":"2023-10-30T13:10:14.242988+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4103,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3343641556}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2076728322}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":902004440}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2944212554}]}]} 2023-10-30T13:10:41.570 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T13:10:41.571 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T13:10:41.571 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14ae548640 1 Processor -- start 2023-10-30T13:10:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14ae548640 1 -- start start 2023-10-30T13:10:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14ae548640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14a81538f0 0x7f14a8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14ae548640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f14a815cd50 0x7f14a80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14ae548640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a80cefa0 0x7f14a80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14ae548640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a815d300 con 0x7f14a815cd50 2023-10-30T13:10:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14ae548640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a815d5e0 con 0x7f14a81538f0 2023-10-30T13:10:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14ae548640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a815d8c0 con 0x7f14a80cefa0 2023-10-30T13:10:41.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14acd45640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f14a815cd50 0x7f14a80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14acd45640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f14a815cd50 0x7f14a80c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50696/0 (socket says 172.21.15.148:50696) 2023-10-30T13:10:41.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14acd45640 1 -- 172.21.15.148:0/446279955 learned_addr learned my addr 172.21.15.148:0/446279955 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:41.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14add47640 1 --2- 172.21.15.148:0/446279955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a80cefa0 0x7f14a80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14ad546640 1 --2- 172.21.15.148:0/446279955 >> v2:172.21.15.150:3300/0 conn(0x7f14a81538f0 0x7f14a8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14acd45640 1 -- 172.21.15.148:0/446279955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a80cefa0 msgr2=0x7f14a80cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:41.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14acd45640 1 --2- 172.21.15.148:0/446279955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a80cefa0 0x7f14a80cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14acd45640 1 -- 172.21.15.148:0/446279955 >> v2:172.21.15.150:3300/0 conn(0x7f14a81538f0 msgr2=0x7f14a8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:41.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14acd45640 1 --2- 172.21.15.148:0/446279955 >> v2:172.21.15.150:3300/0 conn(0x7f14a81538f0 0x7f14a8159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.708+0000 7f14acd45640 1 -- 172.21.15.148:0/446279955 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a815dba0 con 0x7f14a815cd50 2023-10-30T13:10:41.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14acd45640 1 --2- 172.21.15.148:0/446279955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 0x7f14a80c9f90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f14a80c9cb0 tx=0x7f149801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f0c8b0ac4e5a77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:41.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f149ffff640 1 -- 172.21.15.148:0/446279955 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14980a89b0 con 0x7f14a815cd50 2023-10-30T13:10:41.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f149ffff640 1 -- 172.21.15.148:0/446279955 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14980a8b10 con 0x7f14a815cd50 2023-10-30T13:10:41.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f149ffff640 1 -- 172.21.15.148:0/446279955 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14980a8ea0 con 0x7f14a815cd50 2023-10-30T13:10:41.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 -- 172.21.15.148:0/446279955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 msgr2=0x7f14a80c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 --2- 172.21.15.148:0/446279955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 0x7f14a80c9f90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f14a80c9cb0 tx=0x7f149801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 -- 172.21.15.148:0/446279955 shutdown_connections 2023-10-30T13:10:41.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 --2- 172.21.15.148:0/446279955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a80cefa0 0x7f14a80cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 --2- 172.21.15.148:0/446279955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 0x7f14a80c9f90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 --2- 172.21.15.148:0/446279955 >> v2:172.21.15.150:3300/0 conn(0x7f14a81538f0 0x7f14a8159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 -- 172.21.15.148:0/446279955 >> 172.21.15.148:0/446279955 conn(0x7f14a806a640 msgr2=0x7f14a8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:41.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 -- 172.21.15.148:0/446279955 shutdown_connections 2023-10-30T13:10:41.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 -- 172.21.15.148:0/446279955 wait complete. 2023-10-30T13:10:41.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.712+0000 7f14ae548640 1 Processor -- start 2023-10-30T13:10:41.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 -- start start 2023-10-30T13:10:41.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14a80cefa0 0x7f14a80c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a81538f0 0x7f14a80b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 0x7f14a80b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a81f00e0 con 0x7f14a815cd50 2023-10-30T13:10:41.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a815a9b0 con 0x7f14a80cefa0 2023-10-30T13:10:41.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a81481e0 con 0x7f14a81538f0 2023-10-30T13:10:41.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14add47640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 0x7f14a80b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14add47640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 0x7f14a80b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50702/0 (socket says 172.21.15.148:50702) 2023-10-30T13:10:41.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14add47640 1 -- 172.21.15.148:0/878177625 learned_addr learned my addr 172.21.15.148:0/878177625 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:41.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ad546640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14a80cefa0 0x7f14a80c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14acd45640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a81538f0 0x7f14a80b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14add47640 1 -- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a81538f0 msgr2=0x7f14a80b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14add47640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a81538f0 0x7f14a80b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14add47640 1 -- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14a80cefa0 msgr2=0x7f14a80c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14add47640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14a80cefa0 0x7f14a80c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14add47640 1 -- 172.21.15.148:0/878177625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980094d0 con 0x7f14a815cd50 2023-10-30T13:10:41.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14add47640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 0x7f14a80b44e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f14a0007e60 tx=0x7f14a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:41.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f149dffb640 1 -- 172.21.15.148:0/878177625 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0014620 con 0x7f14a815cd50 2023-10-30T13:10:41.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f149dffb640 1 -- 172.21.15.148:0/878177625 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a000ee90 con 0x7f14a815cd50 2023-10-30T13:10:41.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f149dffb640 1 -- 172.21.15.148:0/878177625 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a00149e0 con 0x7f14a815cd50 2023-10-30T13:10:41.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a81efe20 con 0x7f14a815cd50 2023-10-30T13:10:41.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a8148480 con 0x7f14a815cd50 2023-10-30T13:10:41.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.716+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a80cb730 con 0x7f14a815cd50 2023-10-30T13:10:41.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.720+0000 7f149dffb640 1 -- 172.21.15.148:0/878177625 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f14a0014b40 con 0x7f14a815cd50 2023-10-30T13:10:41.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.720+0000 7f149dffb640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f148c0664a0 0x7f148c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:41.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.720+0000 7f149dffb640 1 -- 172.21.15.148:0/878177625 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14a0084640 con 0x7f14a815cd50 2023-10-30T13:10:41.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.720+0000 7f14ad546640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f148c0664a0 0x7f148c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:41.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.720+0000 7f14ad546640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f148c0664a0 0x7f148c068960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f14a4007a60 tx=0x7f14a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:41.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.724+0000 7f149dffb640 1 -- 172.21.15.148:0/878177625 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f14a80cb730 con 0x7f14a815cd50 2023-10-30T13:10:41.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.880+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14a80cb730 con 0x7f14a815cd50 2023-10-30T13:10:41.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.880+0000 7f149dffb640 1 -- 172.21.15.148:0/878177625 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f14a80cb730 con 0x7f14a815cd50 2023-10-30T13:10:41.885 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:41.885 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:35.311442+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:41.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f148c0664a0 msgr2=0x7f148c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f148c0664a0 0x7f148c068960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f14a4007a60 tx=0x7f14a4008040 comp rx=0 tx=0).stop 2023-10-30T13:10:41.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 msgr2=0x7f14a80b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 0x7f14a80b44e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f14a0007e60 tx=0x7f14a000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 shutdown_connections 2023-10-30T13:10:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f148c0664a0 0x7f148c068960 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14a815cd50 0x7f14a80b44e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14a81538f0 0x7f14a80b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 --2- 172.21.15.148:0/878177625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14a80cefa0 0x7f14a80c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:41.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 >> 172.21.15.148:0/878177625 conn(0x7f14a806a640 msgr2=0x7f14a80cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:41.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 shutdown_connections 2023-10-30T13:10:41.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:41.884+0000 7f14ae548640 1 -- 172.21.15.148:0/878177625 wait complete. 2023-10-30T13:10:41.906 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T13:10:41.906 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:42.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a9380640 1 Processor -- start 2023-10-30T13:10:42.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a9380640 1 -- start start 2023-10-30T13:10:42.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a9380640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3a41538a0 0x7fa3a415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a9380640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a40ca0e0 0x7fa3a415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a9380640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a40cb800 con 0x7fa3a40cf160 2023-10-30T13:10:42.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a9380640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a415dc30 con 0x7fa3a41538a0 2023-10-30T13:10:42.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a9380640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a40cb330 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa39bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a40ca0e0 0x7fa3a415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a2ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3a41538a0 0x7fa3a415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa3a40cf160 0x7fa3a40cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:59504/0 (socket says 172.21.15.148:59504) 2023-10-30T13:10:42.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a37fe640 1 -- 172.21.15.148:0/267854174 learned_addr learned my addr 172.21.15.148:0/267854174 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:42.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a2ffd640 1 -- 172.21.15.148:0/267854174 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a40ca0e0 msgr2=0x7fa3a415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a2ffd640 1 --2- 172.21.15.148:0/267854174 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a40ca0e0 0x7fa3a415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a2ffd640 1 -- 172.21.15.148:0/267854174 >> v1:172.21.15.150:6789/0 conn(0x7fa3a40cf160 legacy=0x7fa3a40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa3a2ffd640 1 -- 172.21.15.148:0/267854174 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a40cad70 con 0x7fa3a41538a0 2023-10-30T13:10:42.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.040+0000 7fa39bfff640 1 --2- 172.21.15.148:0/267854174 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a40ca0e0 0x7fa3a415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:42.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.044+0000 7fa3a2ffd640 1 --2- 172.21.15.148:0/267854174 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a41538a0 0x7fa3a415d6f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3940097c0 tx=0x7fa39401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=564e1a18e9efb55a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.044+0000 7fa3a27fc640 1 -- 172.21.15.148:0/267854174 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3940a8970 con 0x7fa3a41538a0 2023-10-30T13:10:42.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.044+0000 7fa3a27fc640 1 -- 172.21.15.148:0/267854174 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3940a8ad0 con 0x7fa3a41538a0 2023-10-30T13:10:42.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.044+0000 7fa3a27fc640 1 -- 172.21.15.148:0/267854174 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3940a8e60 con 0x7fa3a41538a0 2023-10-30T13:10:42.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- 172.21.15.148:0/267854174 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a41538a0 msgr2=0x7fa3a415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 --2- 172.21.15.148:0/267854174 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a41538a0 0x7fa3a415d6f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3940097c0 tx=0x7fa39401ec70 comp rx=0 tx=0).stop 2023-10-30T13:10:42.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- 172.21.15.148:0/267854174 shutdown_connections 2023-10-30T13:10:42.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 --2- 172.21.15.148:0/267854174 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a40ca0e0 0x7fa3a415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 --2- 172.21.15.148:0/267854174 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a41538a0 0x7fa3a415d6f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- 172.21.15.148:0/267854174 >> 172.21.15.148:0/267854174 conn(0x7fa3a406bfe0 msgr2=0x7fa3a406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- 172.21.15.148:0/267854174 shutdown_connections 2023-10-30T13:10:42.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- 172.21.15.148:0/267854174 wait complete. 2023-10-30T13:10:42.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 Processor -- start 2023-10-30T13:10:42.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- start start 2023-10-30T13:10:42.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a40ca0e0 0x7fa3a4065de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40cf160 0x7fa3a415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a41538a0 0x7fa3a415bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a40b3fe0 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a40b4b60 con 0x7fa3a40cf160 2023-10-30T13:10:42.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a41e4df0 con 0x7fa3a41538a0 2023-10-30T13:10:42.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a40ca0e0 0x7fa3a4065de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a40ca0e0 0x7fa3a4065de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41730/0 (socket says 172.21.15.148:41730) 2023-10-30T13:10:42.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a2ffd640 1 -- 172.21.15.148:0/3486837647 learned_addr learned my addr 172.21.15.148:0/3486837647 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a37fe640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a41538a0 0x7fa3a415bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa39bfff640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40cf160 0x7fa3a415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a2ffd640 1 -- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a41538a0 msgr2=0x7fa3a415bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a2ffd640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a41538a0 0x7fa3a415bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a2ffd640 1 -- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40cf160 msgr2=0x7fa3a415b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a2ffd640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40cf160 0x7fa3a415b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a2ffd640 1 -- 172.21.15.148:0/3486837647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c009420 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa39bfff640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40cf160 0x7fa3a415b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:42.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a2ffd640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a40ca0e0 0x7fa3a4065de0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa3940b3040 tx=0x7fa39401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa39b7fe640 1 -- 172.21.15.148:0/3486837647 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3940b9630 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa39b7fe640 1 -- 172.21.15.148:0/3486837647 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3940b97b0 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.048+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa394009420 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.052+0000 7fa39b7fe640 1 -- 172.21.15.148:0/3486837647 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3940b9b60 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.052+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a40b4510 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.052+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a40b2a50 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.052+0000 7fa39b7fe640 1 -- 172.21.15.148:0/3486837647 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3940b7040 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.052+0000 7fa39b7fe640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3840664a0 0x7fa384068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.052+0000 7fa39b7fe640 1 -- 172.21.15.148:0/3486837647 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa394121ed0 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.052+0000 7fa39bfff640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3840664a0 0x7fa384068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.056+0000 7fa39bfff640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3840664a0 0x7fa384068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa3a406c710 tx=0x7fa38c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.056+0000 7fa39b7fe640 1 -- 172.21.15.148:0/3486837647 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3940ef340 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3a404e170 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa39b7fe640 1 -- 172.21.15.148:0/3486837647 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fa3940b5070 con 0x7fa3a40ca0e0 2023-10-30T13:10:42.212 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:42.213 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:35.311442+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:42.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3840664a0 msgr2=0x7fa384068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3840664a0 0x7fa384068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa3a406c710 tx=0x7fa38c008040 comp rx=0 tx=0).stop 2023-10-30T13:10:42.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a40ca0e0 msgr2=0x7fa3a4065de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a40ca0e0 0x7fa3a4065de0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa3940b3040 tx=0x7fa39401ba10 comp rx=0 tx=0).stop 2023-10-30T13:10:42.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 shutdown_connections 2023-10-30T13:10:42.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3840664a0 0x7fa384068960 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3a41538a0 0x7fa3a415bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40cf160 0x7fa3a415b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 --2- 172.21.15.148:0/3486837647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a40ca0e0 0x7fa3a4065de0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 >> 172.21.15.148:0/3486837647 conn(0x7fa3a406bfe0 msgr2=0x7fa3a414de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 shutdown_connections 2023-10-30T13:10:42.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.208+0000 7fa3a9380640 1 -- 172.21.15.148:0/3486837647 wait complete. 2023-10-30T13:10:42.234 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T13:10:42.234 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T13:10:42.235 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T13:10:42.235 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T13:10:42.236 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T13:10:42.236 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T13:10:42.237 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T13:10:42.237 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T13:10:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.476+0000 7f4a72691640 1 Processor -- start 2023-10-30T13:10:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.476+0000 7f4a72691640 1 -- start start 2023-10-30T13:10:42.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.476+0000 7f4a72691640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4a6c15ef20 0x7f4a6c161310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.476+0000 7f4a72691640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6c0c6090 con 0x7f4a6c14c140 2023-10-30T13:10:42.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.476+0000 7f4a72691640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6c158d30 con 0x7f4a6c156020 2023-10-30T13:10:42.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.480+0000 7f4a72691640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6c0c6370 con 0x7f4a6c15ef20 2023-10-30T13:10:42.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.480+0000 7f4a6bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4a6c14c140 0x7f4a6c0c2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39268/0 (socket says 172.21.15.148:39268) 2023-10-30T13:10:42.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.480+0000 7f4a70c2e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4a6c15ef20 0x7f4a6c161310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.480+0000 7f4a70c2e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4a6c15ef20 0x7f4a6c161310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37522/0 (socket says 172.21.15.148:37522) 2023-10-30T13:10:42.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.480+0000 7f4a70c2e640 1 -- 172.21.15.148:0/1101690859 learned_addr learned my addr 172.21.15.148:0/1101690859 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.480+0000 7f4a70c2e640 1 -- 172.21.15.148:0/1101690859 >> v1:172.21.15.150:6789/0 conn(0x7f4a6c156020 legacy=0x7f4a6c06d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.480+0000 7f4a70c2e640 1 -- 172.21.15.148:0/1101690859 >> v1:172.21.15.148:6789/0 conn(0x7f4a6c14c140 legacy=0x7f4a6c0c2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.480+0000 7f4a6b7fe640 1 -- 172.21.15.148:0/1101690859 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142334399 0 0) 0x7f4a6c158d30 con 0x7f4a6c156020 2023-10-30T13:10:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.480+0000 7f4a70c2e640 1 -- 172.21.15.148:0/1101690859 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c159010 con 0x7f4a6c15ef20 2023-10-30T13:10:42.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.484+0000 7f577e7e4640 1 Processor -- start 2023-10-30T13:10:42.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.484+0000 7f577e7e4640 1 -- start start 2023-10-30T13:10:42.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.484+0000 7f4a70c2e640 1 --2- 172.21.15.148:0/1101690859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c15ef20 0x7f4a6c161310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009870 tx=0x7f4a6401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a90e7dedc8a3a9c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.484+0000 7f4a6b7fe640 1 -- 172.21.15.148:0/1101690859 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a640a8a30 con 0x7f4a6c15ef20 2023-10-30T13:10:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f4a6b7fe640 1 -- 172.21.15.148:0/1101690859 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a640a8b90 con 0x7f4a6c15ef20 2023-10-30T13:10:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577e7e4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f57780c6f00 0x7f57780c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577e7e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5778155700 0x7f577806cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f4a72691640 1 -- 172.21.15.148:0/1101690859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c15ef20 msgr2=0x7f4a6c161310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f4a72691640 1 --2- 172.21.15.148:0/1101690859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c15ef20 0x7f4a6c161310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009870 tx=0x7f4a6401ed30 comp rx=0 tx=0).stop 2023-10-30T13:10:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577d7e2640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f57781598f0 0x7f577814f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39278/0 (socket says 172.21.15.148:39278) 2023-10-30T13:10:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577d7e2640 1 -- 172.21.15.148:0/498842070 learned_addr learned my addr 172.21.15.148:0/498842070 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577e7e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57780c2e90 con 0x7f57781598f0 2023-10-30T13:10:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577e7e4640 1 -- 172.21.15.148:0/498842070 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5778156700 con 0x7f57780c6f00 2023-10-30T13:10:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577e7e4640 1 -- 172.21.15.148:0/498842070 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5778156140 con 0x7f5778155700 2023-10-30T13:10:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577dfe3640 1 --2- 172.21.15.148:0/498842070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5778155700 0x7f577806cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577cfe1640 1 --2- 172.21.15.148:0/498842070 >> v2:172.21.15.150:3300/0 conn(0x7f57780c6f00 0x7f57780c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7ff3fb59e640 1 Processor -- start 2023-10-30T13:10:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f4a72691640 1 -- 172.21.15.148:0/1101690859 shutdown_connections 2023-10-30T13:10:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f4a72691640 1 --2- 172.21.15.148:0/1101690859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c15ef20 0x7f4a6c161310 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f4a72691640 1 -- 172.21.15.148:0/1101690859 >> 172.21.15.148:0/1101690859 conn(0x7f4a6c0cac50 msgr2=0x7f4a6c0cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f4a72691640 1 -- 172.21.15.148:0/1101690859 shutdown_connections 2023-10-30T13:10:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f4a72691640 1 -- 172.21.15.148:0/1101690859 wait complete. 2023-10-30T13:10:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f4a72691640 1 Processor -- start 2023-10-30T13:10:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577dfe3640 1 -- 172.21.15.148:0/498842070 >> v2:172.21.15.150:3300/0 conn(0x7f57780c6f00 msgr2=0x7f57780c2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577dfe3640 1 --2- 172.21.15.148:0/498842070 >> v2:172.21.15.150:3300/0 conn(0x7f57780c6f00 0x7f57780c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577dfe3640 1 -- 172.21.15.148:0/498842070 >> v1:172.21.15.148:6789/0 conn(0x7f57781598f0 legacy=0x7f577814f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f576ffff640 1 -- 172.21.15.148:0/498842070 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770550300 0 0) 0x7f57780c2e90 con 0x7f57781598f0 2023-10-30T13:10:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7f577dfe3640 1 -- 172.21.15.148:0/498842070 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5778156420 con 0x7f5778155700 2023-10-30T13:10:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.488+0000 7ff3fb59e640 1 -- start start 2023-10-30T13:10:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fb59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff3fc153850 0x7ff3fc0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fb59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3fc15a2c0 0x7ff3fc15aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fb59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff3fc15cbc0 0x7ff3fc15cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fa59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff3fc153850 0x7ff3fc0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fa59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff3fc153850 0x7ff3fc0c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37546/0 (socket says 172.21.15.148:37546) 2023-10-30T13:10:42.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff3fc15cbc0 0x7ff3fc15cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff3fc15cbc0 0x7ff3fc15cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41740/0 (socket says 172.21.15.148:41740) 2023-10-30T13:10:42.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fb59e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3fc15dba0 con 0x7ff3fc15cbc0 2023-10-30T13:10:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fb59e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3fc1595d0 con 0x7ff3fc153850 2023-10-30T13:10:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fb59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc0c7380 con 0x7ff3fc15a2c0 2023-10-30T13:10:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fa59c640 1 -- 172.21.15.148:0/3545467356 learned_addr learned my addr 172.21.15.148:0/3545467356 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7f4a72691640 1 -- start start 2023-10-30T13:10:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3f9d9b640 1 --2- 172.21.15.148:0/3545467356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3fc15a2c0 0x7ff3fc15aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fa59c640 1 -- 172.21.15.148:0/3545467356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3fc15a2c0 msgr2=0x7ff3fc15aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fa59c640 1 --2- 172.21.15.148:0/3545467356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3fc15a2c0 0x7ff3fc15aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fa59c640 1 -- 172.21.15.148:0/3545467356 >> v2:172.21.15.148:3300/0 conn(0x7ff3fc15cbc0 msgr2=0x7ff3fc15cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fa59c640 1 --2- 172.21.15.148:0/3545467356 >> v2:172.21.15.148:3300/0 conn(0x7ff3fc15cbc0 0x7ff3fc15cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7ff3fa59c640 1 -- 172.21.15.148:0/3545467356 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc15b000 con 0x7ff3fc153850 2023-10-30T13:10:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7fec2f217640 1 Processor -- start 2023-10-30T13:10:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7fec2f217640 1 -- start start 2023-10-30T13:10:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.492+0000 7f4a72691640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a6c14c140 0x7f4a6c1f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a72691640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c156020 0x7f4a6c0c3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a72691640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a6c15ef20 0x7f4a6c0c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a72691640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c146d60 con 0x7f4a6c14c140 2023-10-30T13:10:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a72691640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0709a0 con 0x7f4a6c156020 2023-10-30T13:10:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a72691640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c146a80 con 0x7f4a6c15ef20 2023-10-30T13:10:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a70c2e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a6c15ef20 0x7f4a6c0c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a70c2e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a6c15ef20 0x7f4a6c0c3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45504/0 (socket says 172.21.15.148:45504) 2023-10-30T13:10:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a70c2e640 1 -- 172.21.15.148:0/396813469 learned_addr learned my addr 172.21.15.148:0/396813469 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a6bfff640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a6c14c140 0x7f4a6c1f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec2f217640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec2004c6c0 0x7fec200e31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec2f217640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fec200e36f0 0x7fec200462f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec2f217640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec200e3b60 con 0x7fec200dd440 2023-10-30T13:10:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec2f217640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec20035d50 con 0x7fec200e36f0 2023-10-30T13:10:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec2f217640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec200e3d90 con 0x7fec2004c6c0 2023-10-30T13:10:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a637fe640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c156020 0x7f4a6c0c3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a70c2e640 1 -- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c156020 msgr2=0x7f4a6c0c3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec2cfb3640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fec200dd440 0x7fec2004ae20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39286/0 (socket says 172.21.15.148:39286) 2023-10-30T13:10:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec2cfb3640 1 -- 172.21.15.148:0/2090763572 learned_addr learned my addr 172.21.15.148:0/2090763572 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec27fff640 1 --2- 172.21.15.148:0/2090763572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec2004c6c0 0x7fec200e31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f577dfe3640 1 --2- 172.21.15.148:0/498842070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5778155700 0x7f577806cec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f57680097c0 tx=0x7f576801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ec6150a220890e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f576ffff640 1 -- 172.21.15.148:0/498842070 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57680a8970 con 0x7f5778155700 2023-10-30T13:10:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f576ffff640 1 -- 172.21.15.148:0/498842070 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57680a8ad0 con 0x7f5778155700 2023-10-30T13:10:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f576ffff640 1 -- 172.21.15.148:0/498842070 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57680a8e60 con 0x7f5778155700 2023-10-30T13:10:42.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a70c2e640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c156020 0x7f4a6c0c3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a70c2e640 1 -- 172.21.15.148:0/396813469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a6c14c140 msgr2=0x7f4a6c1f5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec2d7b4640 1 --2- 172.21.15.148:0/2090763572 >> v2:172.21.15.148:3300/0 conn(0x7fec200e36f0 0x7fec200462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec27fff640 1 -- 172.21.15.148:0/2090763572 >> v2:172.21.15.148:3300/0 conn(0x7fec200e36f0 msgr2=0x7fec200462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec27fff640 1 --2- 172.21.15.148:0/2090763572 >> v2:172.21.15.148:3300/0 conn(0x7fec200e36f0 0x7fec200462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec27fff640 1 -- 172.21.15.148:0/2090763572 >> v1:172.21.15.148:6789/0 conn(0x7fec200dd440 legacy=0x7fec2004ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec27fff640 1 -- 172.21.15.148:0/2090763572 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec20045550 con 0x7fec2004c6c0 2023-10-30T13:10:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fec2d7b4640 1 --2- 172.21.15.148:0/2090763572 >> v2:172.21.15.148:3300/0 conn(0x7fec200e36f0 0x7fec200462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:10:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fa023e32640 1 Processor -- start 2023-10-30T13:10:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7fa023e32640 1 -- start start 2023-10-30T13:10:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae838c9640 1 Processor -- start 2023-10-30T13:10:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.496+0000 7f4a70c2e640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a6c14c140 0x7f4a6c1f5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f577e7e4640 1 -- 172.21.15.148:0/498842070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5778155700 msgr2=0x7f577806cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f577e7e4640 1 --2- 172.21.15.148:0/498842070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5778155700 0x7f577806cec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f57680097c0 tx=0x7f576801ec90 comp rx=0 tx=0).stop 2023-10-30T13:10:42.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f4a70c2e640 1 -- 172.21.15.148:0/396813469 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a640094d0 con 0x7f4a6c15ef20 2023-10-30T13:10:42.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f4a6bfff640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a6c14c140 0x7f4a6c1f5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:42.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae838c9640 1 -- start start 2023-10-30T13:10:42.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa023e32640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa01c153c60 0x7fa01c0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f577e7e4640 1 -- 172.21.15.148:0/498842070 shutdown_connections 2023-10-30T13:10:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f577e7e4640 1 --2- 172.21.15.148:0/498842070 >> v2:172.21.15.150:3300/0 conn(0x7f57780c6f00 0x7f57780c2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f577e7e4640 1 --2- 172.21.15.148:0/498842070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5778155700 0x7f577806cec0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f577e7e4640 1 -- 172.21.15.148:0/498842070 >> 172.21.15.148:0/498842070 conn(0x7f57780cae10 msgr2=0x7f57780cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7ff3fa59c640 1 --2- 172.21.15.148:0/3545467356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 0x7ff3fc0c6e40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0c5ca0 tx=0x7ff3ec01c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da9c7946434d929c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f4a70c2e640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a6c15ef20 0x7f4a6c0c3bb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a6401c960 tx=0x7f4a6401bad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa023e32640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa01c0c4430 0x7fa01c0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa023e32640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa01c159830 0x7fa01c159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa023e32640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c154040 con 0x7fa01c159830 2023-10-30T13:10:42.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa023e32640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c0cb780 con 0x7fa01c0c4430 2023-10-30T13:10:42.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa023e32640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0cc9f0 con 0x7fa01c153c60 2023-10-30T13:10:42.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f4a697fa640 1 -- 172.21.15.148:0/396813469 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a640b9610 con 0x7f4a6c15ef20 2023-10-30T13:10:42.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f4a697fa640 1 -- 172.21.15.148:0/396813469 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a640b9770 con 0x7f4a6c15ef20 2023-10-30T13:10:42.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f4a697fa640 1 -- 172.21.15.148:0/396813469 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a640b9b20 con 0x7f4a6c15ef20 2023-10-30T13:10:42.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0223cf640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa01c159830 0x7fa01c159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa021bce640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa01c153c60 0x7fa01c0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0223cf640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa01c159830 0x7fa01c159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41774/0 (socket says 172.21.15.148:41774) 2023-10-30T13:10:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0223cf640 1 -- 172.21.15.148:0/2533807896 learned_addr learned my addr 172.21.15.148:0/2533807896 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0213cd640 1 --2- 172.21.15.148:0/2533807896 >> v2:172.21.15.150:3300/0 conn(0x7fa01c0c4430 0x7fa01c0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7ff3f959a640 1 -- 172.21.15.148:0/3545467356 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0ab600 con 0x7ff3fc153850 2023-10-30T13:10:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7ff3f959a640 1 -- 172.21.15.148:0/3545467356 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ec0a7050 con 0x7ff3fc153850 2023-10-30T13:10:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7ff3f959a640 1 -- 172.21.15.148:0/3545467356 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0ab600 con 0x7ff3fc153850 2023-10-30T13:10:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f577e7e4640 1 -- 172.21.15.148:0/498842070 shutdown_connections 2023-10-30T13:10:42.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae838c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fae740d7050 0x7fae740e02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae838c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae7404e260 0x7fae74009fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae838c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7404dff0 con 0x7fae74052c40 2023-10-30T13:10:42.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae838c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae74048720 con 0x7fae740d7050 2023-10-30T13:10:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae838c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae740479a0 con 0x7fae7404e260 2023-10-30T13:10:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0213cd640 1 -- 172.21.15.148:0/2533807896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa01c153c60 msgr2=0x7fa01c0ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0213cd640 1 --2- 172.21.15.148:0/2533807896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa01c153c60 0x7fa01c0ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0213cd640 1 -- 172.21.15.148:0/2533807896 >> v2:172.21.15.148:3300/0 conn(0x7fa01c159830 msgr2=0x7fa01c159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0213cd640 1 --2- 172.21.15.148:0/2533807896 >> v2:172.21.15.148:3300/0 conn(0x7fa01c159830 0x7fa01c159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f4a72691640 1 -- 172.21.15.148:0/396813469 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c153500 con 0x7f4a6c15ef20 2023-10-30T13:10:42.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0213cd640 1 -- 172.21.15.148:0/2533807896 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c15a2e0 con 0x7fa01c0c4430 2023-10-30T13:10:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fa0223cf640 1 --2- 172.21.15.148:0/2533807896 >> v2:172.21.15.148:3300/0 conn(0x7fa01c159830 0x7fa01c159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7ff3fb59e640 1 -- 172.21.15.148:0/3545467356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 msgr2=0x7ff3fc0c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7ff3fb59e640 1 --2- 172.21.15.148:0/3545467356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 0x7ff3fc0c6e40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0c5ca0 tx=0x7ff3ec01c490 comp rx=0 tx=0).stop 2023-10-30T13:10:42.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae81e66640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fae74052c40 0x7fae7400c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:59512/0 (socket says 172.21.15.148:59512) 2023-10-30T13:10:42.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae81e66640 1 -- 172.21.15.148:0/464211473 learned_addr learned my addr 172.21.15.148:0/464211473 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:42.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae81665640 1 --2- 172.21.15.148:0/464211473 >> v2:172.21.15.150:3300/0 conn(0x7fae740d7050 0x7fae740e02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7fae80e64640 1 --2- 172.21.15.148:0/464211473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae7404e260 0x7fae74009fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7f4a72691640 1 -- 172.21.15.148:0/396813469 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c1f40c0 con 0x7f4a6c15ef20 2023-10-30T13:10:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fec27fff640 1 --2- 172.21.15.148:0/2090763572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec2004c6c0 0x7fec200e31b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0097c0 tx=0x7fec1c01ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8219e81c5d45bd16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7ff3fb59e640 1 -- 172.21.15.148:0/3545467356 shutdown_connections 2023-10-30T13:10:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7ff3fb59e640 1 --2- 172.21.15.148:0/3545467356 >> v2:172.21.15.148:3300/0 conn(0x7ff3fc15cbc0 0x7ff3fc15cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7ff3fb59e640 1 --2- 172.21.15.148:0/3545467356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3fc15a2c0 0x7ff3fc15aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7ff3fb59e640 1 --2- 172.21.15.148:0/3545467356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 0x7ff3fc0c6e40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7ff3fb59e640 1 -- 172.21.15.148:0/3545467356 >> 172.21.15.148:0/3545467356 conn(0x7ff3fc0cae10 msgr2=0x7ff3fc0cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fae73fff640 1 -- 172.21.15.148:0/464211473 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703707403 0 0) 0x7fae7404dff0 con 0x7fae74052c40 2023-10-30T13:10:42.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fae73fff640 1 -- 172.21.15.148:0/464211473 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae68003530 con 0x7fae74052c40 2023-10-30T13:10:42.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7ff3fb59e640 1 -- 172.21.15.148:0/3545467356 shutdown_connections 2023-10-30T13:10:42.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7ff3fb59e640 1 -- 172.21.15.148:0/3545467356 wait complete. 2023-10-30T13:10:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fae80e64640 1 -- 172.21.15.148:0/464211473 >> v2:172.21.15.150:3300/0 conn(0x7fae740d7050 msgr2=0x7fae740e02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fae80e64640 1 --2- 172.21.15.148:0/464211473 >> v2:172.21.15.150:3300/0 conn(0x7fae740d7050 0x7fae740e02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fae80e64640 1 -- 172.21.15.148:0/464211473 >> v1:172.21.15.150:6789/0 conn(0x7fae74052c40 legacy=0x7fae7400c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fae80e64640 1 -- 172.21.15.148:0/464211473 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae74009dc0 con 0x7fae7404e260 2023-10-30T13:10:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.500+0000 7f577e7e4640 1 -- 172.21.15.148:0/498842070 wait complete. 2023-10-30T13:10:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fae81665640 1 --2- 172.21.15.148:0/464211473 >> v2:172.21.15.150:3300/0 conn(0x7fae740d7050 0x7fae740e02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:42.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7f4a72691640 1 -- 172.21.15.148:0/396813469 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4a6c1595d0 con 0x7f4a6c15ef20 2023-10-30T13:10:42.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fec277fe640 1 -- 172.21.15.148:0/2090763572 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0a87b0 con 0x7fec2004c6c0 2023-10-30T13:10:42.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fec277fe640 1 -- 172.21.15.148:0/2090763572 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c0a8910 con 0x7fec2004c6c0 2023-10-30T13:10:42.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fec277fe640 1 -- 172.21.15.148:0/2090763572 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0a8ca0 con 0x7fec2004c6c0 2023-10-30T13:10:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7f577e7e4640 1 Processor -- start 2023-10-30T13:10:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fec2f217640 1 -- 172.21.15.148:0/2090763572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec2004c6c0 msgr2=0x7fec200e31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fec2f217640 1 --2- 172.21.15.148:0/2090763572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec2004c6c0 0x7fec200e31b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0097c0 tx=0x7fec1c01ec10 comp rx=0 tx=0).stop 2023-10-30T13:10:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7f577e7e4640 1 -- start start 2023-10-30T13:10:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7ff3fb59e640 1 Processor -- start 2023-10-30T13:10:42.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7ff3fb59e640 1 -- start start 2023-10-30T13:10:42.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fb454217640 1 Processor -- start 2023-10-30T13:10:42.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fec2f217640 1 -- 172.21.15.148:0/2090763572 shutdown_connections 2023-10-30T13:10:42.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fec2f217640 1 --2- 172.21.15.148:0/2090763572 >> v2:172.21.15.148:3300/0 conn(0x7fec200e36f0 0x7fec200462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fec2f217640 1 --2- 172.21.15.148:0/2090763572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec2004c6c0 0x7fec200e31b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fec2f217640 1 -- 172.21.15.148:0/2090763572 >> 172.21.15.148:0/2090763572 conn(0x7fec20050310 msgr2=0x7fec20050720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577e7e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57780c6f00 0x7f5778143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577e7e4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5778155700 0x7f57781f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577e7e4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57781598f0 0x7f57780c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577e7e4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778144710 con 0x7f5778155700 2023-10-30T13:10:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577e7e4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57781441c0 con 0x7f57781598f0 2023-10-30T13:10:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fec2f217640 1 -- 172.21.15.148:0/2090763572 shutdown_connections 2023-10-30T13:10:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577e7e4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57781f3410 con 0x7f57780c6f00 2023-10-30T13:10:42.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fec2f217640 1 -- 172.21.15.148:0/2090763572 wait complete. 2023-10-30T13:10:42.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa0213cd640 1 --2- 172.21.15.148:0/2533807896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c0c4430 0x7fa01c0cc4b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0ccd20 tx=0x7fa01401c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a113573375390027 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.504+0000 7fb454217640 1 -- start start 2023-10-30T13:10:42.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fb59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 0x7ff3fc1515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fb59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3fc15a2c0 0x7ff3fc14e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577cfe1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5778155700 0x7f57781f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577d7e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57780c6f00 0x7f5778143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f4a697fa640 1 -- 172.21.15.148:0/396813469 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4a640b7040 con 0x7f4a6c15ef20 2023-10-30T13:10:42.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577dfe3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57781598f0 0x7f57780c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577d7e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57780c6f00 0x7f5778143c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45532/0 (socket says 172.21.15.148:45532) 2023-10-30T13:10:42.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577d7e2640 1 -- 172.21.15.148:0/962744439 learned_addr learned my addr 172.21.15.148:0/962744439 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fb454217640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb44c153c60 0x7fb44c0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fb454217640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb44c0c4430 0x7fb44c0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa020bcc640 1 -- 172.21.15.148:0/2533807896 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0140a5880 con 0x7fa01c0c4430 2023-10-30T13:10:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa020bcc640 1 -- 172.21.15.148:0/2533807896 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0140a59e0 con 0x7fa01c0c4430 2023-10-30T13:10:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa020bcc640 1 -- 172.21.15.148:0/2533807896 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0140a5d70 con 0x7fa01c0c4430 2023-10-30T13:10:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fae80e64640 1 --2- 172.21.15.148:0/464211473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae7404e260 0x7fae74009fc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0097c0 tx=0x7fae6c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67aa0122f5baefb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fbce47d3640 1 Processor -- start 2023-10-30T13:10:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa023e32640 1 -- 172.21.15.148:0/2533807896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c0c4430 msgr2=0x7fa01c0cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa023e32640 1 --2- 172.21.15.148:0/2533807896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c0c4430 0x7fa01c0cc4b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0ccd20 tx=0x7fa01401c910 comp rx=0 tx=0).stop 2023-10-30T13:10:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fb59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3fc15cbc0 0x7ff3fc070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fb59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc14ed60 con 0x7ff3fc15a2c0 2023-10-30T13:10:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fb59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc0c3c80 con 0x7ff3fc153850 2023-10-30T13:10:42.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fa59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 0x7ff3fc1515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fa59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 0x7ff3fc1515c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37590/0 (socket says 172.21.15.148:37590) 2023-10-30T13:10:42.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fa59c640 1 -- 172.21.15.148:0/1134457080 learned_addr learned my addr 172.21.15.148:0/1134457080 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc0c2dc0 con 0x7ff3fc15cbc0 2023-10-30T13:10:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fb451fb3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb44c153c60 0x7fb44c0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3f9d9b640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3fc15a2c0 0x7ff3fc14e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577d7e2640 1 -- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57781598f0 msgr2=0x7f57780c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577d7e2640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57781598f0 0x7f57780c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577d7e2640 1 -- 172.21.15.148:0/962744439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5778155700 msgr2=0x7f57781f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577d7e2640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5778155700 0x7f57781f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f577d7e2640 1 -- 172.21.15.148:0/962744439 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5768009420 con 0x7f57780c6f00 2023-10-30T13:10:42.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa023e32640 1 -- 172.21.15.148:0/2533807896 shutdown_connections 2023-10-30T13:10:42.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa023e32640 1 --2- 172.21.15.148:0/2533807896 >> v2:172.21.15.148:3300/0 conn(0x7fa01c159830 0x7fa01c159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa023e32640 1 --2- 172.21.15.148:0/2533807896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c0c4430 0x7fa01c0cc4b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa023e32640 1 --2- 172.21.15.148:0/2533807896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa01c153c60 0x7fa01c0ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fa023e32640 1 -- 172.21.15.148:0/2533807896 >> 172.21.15.148:0/2533807896 conn(0x7fa01c06bfe0 msgr2=0x7fa01c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fb451fb3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb44c153c60 0x7fb44c0ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37604/0 (socket says 172.21.15.148:37604) 2023-10-30T13:10:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f4a697fa640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a500664a0 0x7f4a50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f4a697fa640 1 -- 172.21.15.148:0/396813469 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a641222d0 con 0x7f4a6c15ef20 2023-10-30T13:10:42.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fae73fff640 1 -- 172.21.15.148:0/464211473 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c0ab650 con 0x7fae7404e260 2023-10-30T13:10:42.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fbce47d3640 1 -- start start 2023-10-30T13:10:42.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fad9d640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3fc15cbc0 0x7ff3fc070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f4a6bfff640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a500664a0 0x7f4a50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fec2f217640 1 Processor -- start 2023-10-30T13:10:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fb454217640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0cb780 con 0x7fb44c159830 2023-10-30T13:10:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fb454217640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44c154040 con 0x7fb44c0c4430 2023-10-30T13:10:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fb454217640 1 -- 172.21.15.148:0/804160076 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44c0cc9f0 con 0x7fb44c153c60 2023-10-30T13:10:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fb451fb3640 1 -- 172.21.15.148:0/804160076 learned_addr learned my addr 172.21.15.148:0/804160076 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fae73fff640 1 -- 172.21.15.148:0/464211473 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c0ab7b0 con 0x7fae7404e260 2023-10-30T13:10:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fae73fff640 1 -- 172.21.15.148:0/464211473 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c0abb40 con 0x7fae7404e260 2023-10-30T13:10:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f4a697fa640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f4a5006e640 0x7f4a50070a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7f4a697fa640 1 -- 172.21.15.148:0/396813469 --> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a50071110 con 0x7f4a5006e640 2023-10-30T13:10:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f4a697fa640 1 -- 172.21.15.148:0/396813469 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a64122720 con 0x7f4a6c15ef20 2023-10-30T13:10:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7fec2f217640 1 -- start start 2023-10-30T13:10:42.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fa023e32640 1 -- 172.21.15.148:0/2533807896 shutdown_connections 2023-10-30T13:10:42.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fa59c640 1 -- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3fc15cbc0 msgr2=0x7ff3fc070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fa59c640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3fc15cbc0 0x7ff3fc070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.508+0000 7ff3fa59c640 1 -- 172.21.15.148:0/1134457080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3fc15a2c0 msgr2=0x7ff3fc14e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7ff3fa59c640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3fc15a2c0 0x7ff3fc14e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7ff3fa59c640 1 -- 172.21.15.148:0/1134457080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0094d0 con 0x7ff3fc153850 2023-10-30T13:10:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fa023e32640 1 -- 172.21.15.148:0/2533807896 wait complete. 2023-10-30T13:10:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7ff3f9d9b640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3fc15a2c0 0x7ff3fc14e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f577d7e2640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57780c6f00 0x7f5778143c80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f577400c4d0 tx=0x7f577400c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fae838c9640 1 -- 172.21.15.148:0/464211473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae7404e260 msgr2=0x7fae74009fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fae838c9640 1 --2- 172.21.15.148:0/464211473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae7404e260 0x7fae74009fc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0097c0 tx=0x7fae6c01c450 comp rx=0 tx=0).stop 2023-10-30T13:10:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f4a6bfff640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a500664a0 0x7f4a50068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c004df0 tx=0x7f4a5c007e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f4a637fe640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f4a5006e640 0x7f4a50070a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fb451fb3640 1 -- 172.21.15.148:0/804160076 >> v2:172.21.15.148:3300/0 conn(0x7fb44c0c4430 msgr2=0x7fb44c0cc4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:10:42.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fb451fb3640 1 --2- 172.21.15.148:0/804160076 >> v2:172.21.15.148:3300/0 conn(0x7fb44c0c4430 0x7fb44c0cc4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f576dffb640 1 -- 172.21.15.148:0/962744439 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57740074e0 con 0x7f57780c6f00 2023-10-30T13:10:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fae838c9640 1 -- 172.21.15.148:0/464211473 shutdown_connections 2023-10-30T13:10:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fae838c9640 1 --2- 172.21.15.148:0/464211473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae7404e260 0x7fae74009fc0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0097c0 tx=0x7fae6c01c450 comp rx=0 tx=0).stop 2023-10-30T13:10:42.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fae838c9640 1 --2- 172.21.15.148:0/464211473 >> v2:172.21.15.150:3300/0 conn(0x7fae740d7050 0x7fae740e02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fae838c9640 1 -- 172.21.15.148:0/464211473 >> 172.21.15.148:0/464211473 conn(0x7fae7400e9c0 msgr2=0x7fae74052530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f576dffb640 1 -- 172.21.15.148:0/962744439 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5774007640 con 0x7f57780c6f00 2023-10-30T13:10:42.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f576dffb640 1 -- 172.21.15.148:0/962744439 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5774004a30 con 0x7f57780c6f00 2023-10-30T13:10:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce47d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbcdc0c6b40 0x7fbcdc161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce47d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcdc0c72b0 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce47d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcdc06acb0 con 0x7fbcdc158090 2023-10-30T13:10:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce47d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcdc162120 con 0x7fbcdc0c6b40 2023-10-30T13:10:42.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5778144430 con 0x7f57780c6f00 2023-10-30T13:10:42.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce256f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbcdc158090 0x7fbcdc069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:59518/0 (socket says 172.21.15.148:59518) 2023-10-30T13:10:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce256f640 1 -- 172.21.15.148:0/1231678805 learned_addr learned my addr 172.21.15.148:0/1231678805 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57781401d0 con 0x7f57780c6f00 2023-10-30T13:10:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f4a637fe640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f4a5006e640 0x7f4a50070a60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fae838c9640 1 -- 172.21.15.148:0/464211473 shutdown_connections 2023-10-30T13:10:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce1d6e640 1 --2- 172.21.15.148:0/1231678805 >> v2:172.21.15.148:3300/0 conn(0x7fbcdc0c6b40 0x7fbcdc161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce1d6e640 1 -- 172.21.15.148:0/1231678805 >> v1:172.21.15.150:6789/0 conn(0x7fbcdc158090 legacy=0x7fbcdc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce1d6e640 1 -- 172.21.15.148:0/1231678805 >> v1:172.21.15.148:6789/0 conn(0x7fbcdc0c0ed0 legacy=0x7fbcdc06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fbce1d6e640 1 -- 172.21.15.148:0/1231678805 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc0c6fd0 con 0x7fbcdc0c6b40 2023-10-30T13:10:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57781590f0 con 0x7f57780c6f00 2023-10-30T13:10:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fa023e32640 1 Processor -- start 2023-10-30T13:10:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7ff3fa59c640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 0x7ff3fc1515c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec01ab40 tx=0x7ff3ec019230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7f4a697fa640 1 -- 172.21.15.148:0/396813469 <== osd.0 v2:172.21.15.148:6800/1454599971 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4a50071110 con 0x7f4a5006e640 2023-10-30T13:10:42.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fa023e32640 1 -- start start 2023-10-30T13:10:42.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fae838c9640 1 -- 172.21.15.148:0/464211473 wait complete. 2023-10-30T13:10:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7ff3eaffd640 1 -- 172.21.15.148:0/1134457080 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec003a30 con 0x7ff3fc153850 2023-10-30T13:10:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2f217640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec2004c6c0 0x7fec2017c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2f217640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec200dd440 0x7fec200d5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2f217640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec200e36f0 0x7fec201791d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2f217640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2017ceb0 con 0x7fec2004c6c0 2023-10-30T13:10:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2f217640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec20178d40 con 0x7fec200e36f0 2023-10-30T13:10:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2f217640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec200dbdb0 con 0x7fec200dd440 2023-10-30T13:10:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2cfb3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec2004c6c0 0x7fec2017c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2d7b4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec200e36f0 0x7fec201791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2d7b4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec200e36f0 0x7fec201791d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37616/0 (socket says 172.21.15.148:37616) 2023-10-30T13:10:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2d7b4640 1 -- 172.21.15.148:0/1881676803 learned_addr learned my addr 172.21.15.148:0/1881676803 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec27fff640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec200dd440 0x7fec200d5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2cfb3640 1 -- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec200dd440 msgr2=0x7fec200d5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2cfb3640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec200dd440 0x7fec200d5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2cfb3640 1 -- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec200e36f0 msgr2=0x7fec201791d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2cfb3640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec200e36f0 0x7fec201791d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fec2cfb3640 1 -- 172.21.15.148:0/1881676803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c009420 con 0x7fec2004c6c0 2023-10-30T13:10:42.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fb451fb3640 1 -- 172.21.15.148:0/804160076 >> v1:172.21.15.150:6789/0 conn(0x7fb44c159830 legacy=0x7fb44c0cac40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T13:10:42.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.512+0000 7fb451fb3640 1 -- 172.21.15.148:0/804160076 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c15a2e0 con 0x7fb44c153c60 2023-10-30T13:10:42.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fec2cfb3640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec2004c6c0 0x7fec2017c970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fec2804fab0 tx=0x7fec280674c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fec257fa640 1 -- 172.21.15.148:0/1881676803 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec280675f0 con 0x7fec2004c6c0 2023-10-30T13:10:42.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fec257fa640 1 -- 172.21.15.148:0/1881676803 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec28069e90 con 0x7fec2004c6c0 2023-10-30T13:10:42.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fec257fa640 1 -- 172.21.15.148:0/1881676803 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec28072400 con 0x7fec2004c6c0 2023-10-30T13:10:42.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec200d9660 con 0x7fec2004c6c0 2023-10-30T13:10:42.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec201789f0 con 0x7fec2004c6c0 2023-10-30T13:10:42.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fae838c9640 1 Processor -- start 2023-10-30T13:10:42.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7ff3eaffd640 1 -- 172.21.15.148:0/1134457080 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ec003b90 con 0x7ff3fc153850 2023-10-30T13:10:42.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7ff3eaffd640 1 -- 172.21.15.148:0/1134457080 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec0b6440 con 0x7ff3fc153850 2023-10-30T13:10:42.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7f576dffb640 1 -- 172.21.15.148:0/962744439 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5774004b90 con 0x7f57780c6f00 2023-10-30T13:10:42.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc1f9e90 con 0x7ff3fc153850 2023-10-30T13:10:42.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc0c2ff0 con 0x7ff3fc153850 2023-10-30T13:10:42.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fae838c9640 1 -- start start 2023-10-30T13:10:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff3fc159ac0 con 0x7ff3fc153850 2023-10-30T13:10:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa023e32640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa01c0c4430 0x7fa01c156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa023e32640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa01c153c60 0x7fa01c148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa023e32640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c159830 0x7fa01c155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa023e32640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c155600 con 0x7fa01c0c4430 2023-10-30T13:10:42.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa023e32640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0c9200 con 0x7fa01c159830 2023-10-30T13:10:42.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa023e32640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c1571a0 con 0x7fa01c153c60 2023-10-30T13:10:42.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fb451fb3640 1 --2- 172.21.15.148:0/804160076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 0x7fb44c0ca360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb44c072ce0 tx=0x7fb44801c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5be4951b08614fa9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa0223cf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c159830 0x7fa01c155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa0213cd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa01c153c60 0x7fa01c148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa0223cf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c159830 0x7fa01c155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37628/0 (socket says 172.21.15.148:37628) 2023-10-30T13:10:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa0223cf640 1 -- 172.21.15.148:0/747672180 learned_addr learned my addr 172.21.15.148:0/747672180 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7fa021bce640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa01c0c4430 0x7fa01c156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7f576dffb640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f575c0664a0 0x7f575c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.516+0000 7f576dffb640 1 -- 172.21.15.148:0/962744439 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57740838d0 con 0x7f57780c6f00 2023-10-30T13:10:42.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7f576dffb640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7f575c06e6c0 0x7f575c070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7f576dffb640 1 -- 172.21.15.148:0/962744439 --> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f575c071190 con 0x7f575c06e6c0 2023-10-30T13:10:42.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7f576dffb640 1 -- 172.21.15.148:0/962744439 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5774083c80 con 0x7f57780c6f00 2023-10-30T13:10:42.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa0223cf640 1 -- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa01c153c60 msgr2=0x7fa01c148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa0223cf640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa01c153c60 0x7fa01c148f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa0223cf640 1 -- 172.21.15.148:0/747672180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa01c0c4430 msgr2=0x7fa01c156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa0223cf640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa01c0c4430 0x7fa01c156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce1d6e640 1 --2- 172.21.15.148:0/1231678805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0c6b40 0x7fbcdc161be0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0097c0 tx=0x7fbccc01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843e9ee5c56bb208 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa0223cf640 1 -- 172.21.15.148:0/747672180 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0040094d0 con 0x7fa01c159830 2023-10-30T13:10:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7f577cfe1640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f575c0664a0 0x7f575c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae838c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae74052c40 0x7fae740d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae838c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae740d7050 0x7fae740d0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae838c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae740499a0 0x7fae740d0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae80e64640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae740d7050 0x7fae740d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce156d640 1 -- 172.21.15.148:0/1231678805 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc0a8930 con 0x7fbcdc0c6b40 2023-10-30T13:10:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7f577dfe3640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7f575c06e6c0 0x7f575c070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce156d640 1 -- 172.21.15.148:0/1231678805 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc0a8a90 con 0x7fbcdc0c6b40 2023-10-30T13:10:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce156d640 1 -- 172.21.15.148:0/1231678805 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc0a8e20 con 0x7fbcdc0c6b40 2023-10-30T13:10:42.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa0223cf640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c159830 0x7fa01c155950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa00400e930 tx=0x7fa00400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fec257fa640 1 -- 172.21.15.148:0/1881676803 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fec280a53c0 con 0x7fec2004c6c0 2023-10-30T13:10:42.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec20045810 con 0x7fec2004c6c0 2023-10-30T13:10:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae80e64640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae740d7050 0x7fae740d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41824/0 (socket says 172.21.15.148:41824) 2023-10-30T13:10:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae838c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae740cfbc0 con 0x7fae740d7050 2023-10-30T13:10:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae838c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae740cecd0 con 0x7fae740499a0 2023-10-30T13:10:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae838c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae740cefb0 con 0x7fae74052c40 2023-10-30T13:10:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae80e64640 1 -- 172.21.15.148:0/1814128382 learned_addr learned my addr 172.21.15.148:0/1814128382 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae81665640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae74052c40 0x7fae740d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7f577cfe1640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f575c0664a0 0x7f575c068960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57780c1af0 tx=0x7f5770007710 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa01a7fc640 1 -- 172.21.15.148:0/747672180 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004004020 con 0x7fa01c159830 2023-10-30T13:10:42.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fb450fb1640 1 -- 172.21.15.148:0/804160076 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4480ab600 con 0x7fb44c153c60 2023-10-30T13:10:42.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa01a7fc640 1 -- 172.21.15.148:0/747672180 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa004004180 con 0x7fa01c159830 2023-10-30T13:10:42.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae81e66640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae740499a0 0x7fae740d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7f577dfe3640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7f575c06e6c0 0x7f575c070ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa014009a70 con 0x7fa01c159830 2023-10-30T13:10:42.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce47d3640 1 -- 172.21.15.148:0/1231678805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0c6b40 msgr2=0x7fbcdc161be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce47d3640 1 --2- 172.21.15.148:0/1231678805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0c6b40 0x7fbcdc161be0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0097c0 tx=0x7fbccc01ec30 comp rx=0 tx=0).stop 2023-10-30T13:10:42.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae81e66640 1 -- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae74052c40 msgr2=0x7fae740d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae81e66640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae74052c40 0x7fae740d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae81e66640 1 -- 172.21.15.148:0/1814128382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae740d7050 msgr2=0x7fae740d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae81e66640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae740d7050 0x7fae740d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce47d3640 1 -- 172.21.15.148:0/1231678805 shutdown_connections 2023-10-30T13:10:42.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce47d3640 1 --2- 172.21.15.148:0/1231678805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0c6b40 0x7fbcdc161be0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce47d3640 1 -- 172.21.15.148:0/1231678805 >> 172.21.15.148:0/1231678805 conn(0x7fbcdc0c7910 msgr2=0x7fbcdc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae81e66640 1 -- 172.21.15.148:0/1814128382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae78009420 con 0x7fae740499a0 2023-10-30T13:10:42.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7ff3eaffd640 1 -- 172.21.15.148:0/1134457080 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3ec0b2040 con 0x7ff3fc153850 2023-10-30T13:10:42.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa01a7fc640 1 -- 172.21.15.148:0/747672180 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004004510 con 0x7fa01c159830 2023-10-30T13:10:42.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae81e66640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae740499a0 0x7fae740d0590 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae7c04fab0 tx=0x7fae7c074bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce47d3640 1 -- 172.21.15.148:0/1231678805 shutdown_connections 2023-10-30T13:10:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fb450fb1640 1 -- 172.21.15.148:0/804160076 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4480a7050 con 0x7fb44c153c60 2023-10-30T13:10:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fb450fb1640 1 -- 172.21.15.148:0/804160076 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4480ab980 con 0x7fb44c153c60 2023-10-30T13:10:42.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae71ffb640 1 -- 172.21.15.148:0/1814128382 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c072970 con 0x7fae740499a0 2023-10-30T13:10:42.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae71ffb640 1 -- 172.21.15.148:0/1814128382 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c069e90 con 0x7fae740499a0 2023-10-30T13:10:42.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae71ffb640 1 -- 172.21.15.148:0/1814128382 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c072d70 con 0x7fae740499a0 2023-10-30T13:10:42.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fbce47d3640 1 -- 172.21.15.148:0/1231678805 wait complete. 2023-10-30T13:10:42.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6c009420 con 0x7fae740499a0 2023-10-30T13:10:42.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7ff3eaffd640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3dc0664a0 0x7ff3dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7f576dffb640 1 -- 172.21.15.148:0/962744439 <== osd.2 v2:172.21.15.148:6802/2893462459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f575c071190 con 0x7f575c06e6c0 2023-10-30T13:10:42.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7ff3eaffd640 1 -- 172.21.15.148:0/1134457080 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3ec11ebf0 con 0x7ff3fc153850 2023-10-30T13:10:42.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.520+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c1494d0 con 0x7fa01c159830 2023-10-30T13:10:42.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7ff3eaffd640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ff3dc06e6c0 0x7ff3dc070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7ff3eaffd640 1 -- 172.21.15.148:0/1134457080 --> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3dc071190 con 0x7ff3dc06e6c0 2023-10-30T13:10:42.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7ff3eaffd640 1 -- 172.21.15.148:0/1134457080 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3ec11efa0 con 0x7ff3fc153850 2023-10-30T13:10:42.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7ff3f9d9b640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3dc0664a0 0x7ff3dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7ff3fad9d640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ff3dc06e6c0 0x7ff3dc070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa01c0cbf90 con 0x7fa01c159830 2023-10-30T13:10:42.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae740cf570 con 0x7fae740499a0 2023-10-30T13:10:42.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fec257fa640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec0c0665a0 0x7fec0c068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae74048200 con 0x7fae740499a0 2023-10-30T13:10:42.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7ff3fad9d640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ff3dc06e6c0 0x7ff3dc070ae0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fec257fa640 1 -- 172.21.15.148:0/1881676803 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec2810f100 con 0x7fec2004c6c0 2023-10-30T13:10:42.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fec27fff640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec0c0665a0 0x7fec0c068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fec257fa640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7fec0c06e7c0 0x7fec0c070be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fae71ffb640 1 -- 172.21.15.148:0/1814128382 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae7c095420 con 0x7fae740499a0 2023-10-30T13:10:42.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fec257fa640 1 -- 172.21.15.148:0/1881676803 --> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec0c071290 con 0x7fec0c06e7c0 2023-10-30T13:10:42.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fec257fa640 1 -- 172.21.15.148:0/1881676803 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec28111660 con 0x7fec2004c6c0 2023-10-30T13:10:42.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7ff3eaffd640 1 -- 172.21.15.148:0/1134457080 <== osd.3 v2:172.21.15.148:6824/1150735205 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7ff3dc071190 con 0x7ff3dc06e6c0 2023-10-30T13:10:42.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fec2d7b4640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7fec0c06e7c0 0x7fec0c070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fa01a7fc640 1 -- 172.21.15.148:0/747672180 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa00400c940 con 0x7fa01c159830 2023-10-30T13:10:42.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fb454217640 1 -- 172.21.15.148:0/804160076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 msgr2=0x7fb44c0ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.524+0000 7fb454217640 1 --2- 172.21.15.148:0/804160076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 0x7fb44c0ca360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb44c072ce0 tx=0x7fb44801c4e0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fa01a7fc640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ffc066520 0x7f9ffc0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fb454217640 1 -- 172.21.15.148:0/804160076 shutdown_connections 2023-10-30T13:10:42.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fb454217640 1 --2- 172.21.15.148:0/804160076 >> v2:172.21.15.148:3300/0 conn(0x7fb44c0c4430 0x7fb44c0cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fb454217640 1 --2- 172.21.15.148:0/804160076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 0x7fb44c0ca360 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fb454217640 1 -- 172.21.15.148:0/804160076 >> 172.21.15.148:0/804160076 conn(0x7fb44c06bfe0 msgr2=0x7fb44c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fb454217640 1 -- 172.21.15.148:0/804160076 shutdown_connections 2023-10-30T13:10:42.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fec2d7b4640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7fec0c06e7c0 0x7fec0c070be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fb454217640 1 -- 172.21.15.148:0/804160076 wait complete. 2023-10-30T13:10:42.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fa021bce640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ffc066520 0x7f9ffc0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fa01a7fc640 1 -- 172.21.15.148:0/747672180 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa004083e80 con 0x7fa01c159830 2023-10-30T13:10:42.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fa01a7fc640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f9ffc06e740 0x7f9ffc070b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fa01a7fc640 1 -- 172.21.15.148:0/747672180 --> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ffc071210 con 0x7f9ffc06e740 2023-10-30T13:10:42.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fa01a7fc640 1 -- 172.21.15.148:0/747672180 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa004086590 con 0x7fa01c159830 2023-10-30T13:10:42.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fa021bce640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ffc066520 0x7f9ffc0689e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0045f0 tx=0x7fa00c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce47d3640 1 Processor -- start 2023-10-30T13:10:42.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fa0213cd640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f9ffc06e740 0x7f9ffc070b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fae71ffb640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae600664f0 0x7fae600689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fec27fff640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec0c0665a0 0x7fec0c068a60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0b2000 tx=0x7fec1c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7ff3f9d9b640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3dc0664a0 0x7ff3dc068960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc072090 tx=0x7ff3f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce47d3640 1 -- start start 2023-10-30T13:10:42.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fae71ffb640 1 -- 172.21.15.148:0/1814128382 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae7c090020 con 0x7fae740499a0 2023-10-30T13:10:42.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fae71ffb640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fae6006e710 0x7fae60070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fec257fa640 1 -- 172.21.15.148:0/1881676803 <== osd.1 v2:172.21.15.148:6815/1053568651 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fec0c071290 con 0x7fec0c06e7c0 2023-10-30T13:10:42.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fae71ffb640 1 -- 172.21.15.148:0/1814128382 --> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae600711e0 con 0x7fae6006e710 2023-10-30T13:10:42.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fae71ffb640 1 -- 172.21.15.148:0/1814128382 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae7c1022a0 con 0x7fae740499a0 2023-10-30T13:10:42.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fa0213cd640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f9ffc06e740 0x7f9ffc070b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce47d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc0c0ed0 0x7fbcdc153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce47d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc158090 0x7fbcdc15eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce47d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc149790 0x7fbcdc15f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce47d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc14f6f0 con 0x7fbcdc149790 2023-10-30T13:10:42.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce47d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc161d40 con 0x7fbcdc158090 2023-10-30T13:10:42.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce47d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc14d950 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce256f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc0c0ed0 0x7fbcdc153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce1d6e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc158090 0x7fbcdc15eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce256f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc0c0ed0 0x7fbcdc153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45576/0 (socket says 172.21.15.148:45576) 2023-10-30T13:10:42.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fbce256f640 1 -- 172.21.15.148:0/3301140172 learned_addr learned my addr 172.21.15.148:0/3301140172 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fae81665640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae600664f0 0x7fae600689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fae80e64640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fae6006e710 0x7fae60070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fae80e64640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fae6006e710 0x7fae60070b30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.528+0000 7fae81665640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae600664f0 0x7fae600689b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fae78009c50 tx=0x7fae78009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb454217640 1 Processor -- start 2023-10-30T13:10:42.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce2d70640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc149790 0x7fbcdc15f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce256f640 1 -- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc158090 msgr2=0x7fbcdc15eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce256f640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc158090 0x7fbcdc15eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce256f640 1 -- 172.21.15.148:0/3301140172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc149790 msgr2=0x7fbcdc15f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb454217640 1 -- start start 2023-10-30T13:10:42.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce256f640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc149790 0x7fbcdc15f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce256f640 1 -- 172.21.15.148:0/3301140172 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc009420 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce2d70640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc149790 0x7fbcdc15f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:10:42.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce256f640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc0c0ed0 0x7fbcdc153de0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400cbf0 tx=0x7fbcd400a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbcd2ffd640 1 -- 172.21.15.148:0/3301140172 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd400aca0 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc14e570 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc1604a0 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fa01a7fc640 1 -- 172.21.15.148:0/747672180 <== osd.5 v2:172.21.15.150:6802/2046109052 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f9ffc071210 con 0x7f9ffc06e740 2023-10-30T13:10:42.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb454217640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb44c0c4430 0x7fb44c0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbcdc15e010 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb454217640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 0x7fb44c142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb454217640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb44c159830 0x7fb44c0c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb454217640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c0ce990 con 0x7fb44c0c4430 2023-10-30T13:10:42.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb454217640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c1f6470 con 0x7fb44c153c60 2023-10-30T13:10:42.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb454217640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c1f6980 con 0x7fb44c159830 2023-10-30T13:10:42.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb4517b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 0x7fb44c142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbcd2ffd640 1 -- 172.21.15.148:0/3301140172 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd400ae00 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fbcd2ffd640 1 -- 172.21.15.148:0/3301140172 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4012600 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fae71ffb640 1 -- 172.21.15.148:0/1814128382 <== osd.6 v2:172.21.15.150:6804/397365804 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fae600711e0 con 0x7fae6006e710 2023-10-30T13:10:42.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb4527b4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb44c159830 0x7fb44c0c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb4527b4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb44c159830 0x7fb44c0c5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45586/0 (socket says 172.21.15.148:45586) 2023-10-30T13:10:42.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb4527b4640 1 -- 172.21.15.148:0/1693785808 learned_addr learned my addr 172.21.15.148:0/1693785808 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb4517b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 0x7fb44c142cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37648/0 (socket says 172.21.15.148:37648) 2023-10-30T13:10:42.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.532+0000 7fb451fb3640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb44c0c4430 0x7fb44c0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb4517b2640 1 -- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb44c159830 msgr2=0x7fb44c0c5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb4517b2640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb44c159830 0x7fb44c0c5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb4517b2640 1 -- 172.21.15.148:0/1693785808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb44c0c4430 msgr2=0x7fb44c0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb4517b2640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb44c0c4430 0x7fb44c0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb4517b2640 1 -- 172.21.15.148:0/1693785808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480094d0 con 0x7fb44c153c60 2023-10-30T13:10:42.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb451fb3640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb44c0c4430 0x7fb44c0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:42.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fbcd2ffd640 1 -- 172.21.15.148:0/3301140172 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbcd4007070 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb4517b2640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 0x7fb44c142cd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb43c009fd0 tx=0x7fb43c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb4427fc640 1 -- 172.21.15.148:0/1693785808 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c011830 con 0x7fb44c153c60 2023-10-30T13:10:42.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb4427fc640 1 -- 172.21.15.148:0/1693785808 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43c011990 con 0x7fb44c153c60 2023-10-30T13:10:42.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.536+0000 7fb4427fc640 1 -- 172.21.15.148:0/1693785808 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c011d20 con 0x7fb44c153c60 2023-10-30T13:10:42.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c1f66a0 con 0x7fb44c153c60 2023-10-30T13:10:42.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c0cd6b0 con 0x7fb44c153c60 2023-10-30T13:10:42.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fbcd2ffd640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcc00664d0 0x7fbcc0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fbcd2ffd640 1 -- 172.21.15.148:0/3301140172 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbcd4084620 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fbcd2ffd640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fbcc006e6f0 0x7fbcc0070b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fbce1d6e640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcc00664d0 0x7fbcc0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fbcd2ffd640 1 -- 172.21.15.148:0/3301140172 --> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbcc00711c0 con 0x7fbcc006e6f0 2023-10-30T13:10:42.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fbcd2ffd640 1 -- 172.21.15.148:0/3301140172 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcd40849d0 con 0x7fbcdc0c0ed0 2023-10-30T13:10:42.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fbce2d70640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fbcc006e6f0 0x7fbcc0070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb44c0cbf90 con 0x7fb44c153c60 2023-10-30T13:10:42.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7f4a72691640 1 -- 172.21.15.148:0/396813469 --> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4a6c1592f0 con 0x7f4a5006e640 2023-10-30T13:10:42.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.540+0000 7fbce1d6e640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcc00664d0 0x7fbcc0068990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0b2000 tx=0x7fbccc0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7fbce2d70640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fbcc006e6f0 0x7fbcc0070b10 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7fb4427fc640 1 -- 172.21.15.148:0/1693785808 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb43c01a400 con 0x7fb44c153c60 2023-10-30T13:10:42.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a697fa640 1 -- 172.21.15.148:0/396813469 <== osd.0 v2:172.21.15.148:6800/1454599971 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4a540020f0 con 0x7f4a5006e640 2023-10-30T13:10:42.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 -- 172.21.15.148:0/396813469 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f4a5006e640 msgr2=0x7f4a50070a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f4a5006e640 0x7f4a50070a60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 -- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a500664a0 msgr2=0x7f4a50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a500664a0 0x7f4a50068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c004df0 tx=0x7f4a5c007e50 comp rx=0 tx=0).stop 2023-10-30T13:10:42.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 -- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a6c15ef20 msgr2=0x7f4a6c0c3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 --2- 172.21.15.148:0/396813469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a6c15ef20 0x7f4a6c0c3bb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4a6401c960 tx=0x7f4a6401bad0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7fbcd2ffd640 1 -- 172.21.15.148:0/3301140172 <== osd.7 v2:172.21.15.150:6824/1265253901 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fbcc00711c0 con 0x7fbcc006e6f0 2023-10-30T13:10:42.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a70c2e640 1 -- 172.21.15.148:0/396813469 reap_dead start 2023-10-30T13:10:42.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 -- 172.21.15.148:0/396813469 shutdown_connections 2023-10-30T13:10:42.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 -- 172.21.15.148:0/396813469 >> 172.21.15.148:0/396813469 conn(0x7f4a6c0cac50 msgr2=0x7f4a6c06d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7fb4427fc640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4300664f0 0x7fb4300689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7fb4427fc640 1 -- 172.21.15.148:0/1693785808 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb43c0844e0 con 0x7fb44c153c60 2023-10-30T13:10:42.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7fb4427fc640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7fb43006e710 0x7fb430070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7fb451fb3640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4300664f0 0x7fb4300689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 -- 172.21.15.148:0/396813469 shutdown_connections 2023-10-30T13:10:42.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7f4a627fc640 1 -- 172.21.15.148:0/396813469 wait complete. 2023-10-30T13:10:42.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7fb4427fc640 1 -- 172.21.15.148:0/1693785808 --> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4300711e0 con 0x7fb43006e710 2023-10-30T13:10:42.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.544+0000 7fb4427fc640 1 -- 172.21.15.148:0/1693785808 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb43c084980 con 0x7fb44c153c60 2023-10-30T13:10:42.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.548+0000 7fb4527b4640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7fb43006e710 0x7fb430070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.548+0000 7fb451fb3640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4300664f0 0x7fb4300689b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb4480ad000 tx=0x7fb448008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.548+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 --> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fec200e4050 con 0x7fec0c06e7c0 2023-10-30T13:10:42.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.548+0000 7fb4527b4640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7fb43006e710 0x7fb430070b30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.548+0000 7fb4427fc640 1 -- 172.21.15.148:0/1693785808 <== osd.4 v2:172.21.15.150:6800/381432752 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fb4300711e0 con 0x7fb43006e710 2023-10-30T13:10:42.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.552+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 --> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff3fc1597e0 con 0x7ff3dc06e6c0 2023-10-30T13:10:42.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7ff3eaffd640 1 -- 172.21.15.148:0/1134457080 <== osd.3 v2:172.21.15.148:6824/1150735205 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff3f400f040 con 0x7ff3dc06e6c0 2023-10-30T13:10:42.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec257fa640 1 -- 172.21.15.148:0/1881676803 <== osd.1 v2:172.21.15.148:6815/1053568651 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fec1400c040 con 0x7fec0c06e7c0 2023-10-30T13:10:42.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7fec0c06e7c0 msgr2=0x7fec0c070be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7fec0c06e7c0 0x7fec0c070be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec0c0665a0 msgr2=0x7fec0c068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec0c0665a0 0x7fec0c068a60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0b2000 tx=0x7fec1c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:10:42.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec2004c6c0 msgr2=0x7fec2017c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 --2- 172.21.15.148:0/1881676803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec2004c6c0 0x7fec2017c970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fec2804fab0 tx=0x7fec280674c0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2d7b4640 1 -- 172.21.15.148:0/1881676803 reap_dead start 2023-10-30T13:10:42.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 shutdown_connections 2023-10-30T13:10:42.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 >> 172.21.15.148:0/1881676803 conn(0x7fec20050310 msgr2=0x7fec20034360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 shutdown_connections 2023-10-30T13:10:42.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7fec2f217640 1 -- 172.21.15.148:0/1881676803 wait complete. 2023-10-30T13:10:42.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.556+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 --> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5778158e10 con 0x7f575c06e6c0 2023-10-30T13:10:42.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ff3dc06e6c0 msgr2=0x7ff3dc070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ff3dc06e6c0 0x7ff3dc070ae0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f576dffb640 1 -- 172.21.15.148:0/962744439 <== osd.2 v2:172.21.15.148:6802/2893462459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f575c071190 con 0x7f575c06e6c0 2023-10-30T13:10:42.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3dc0664a0 msgr2=0x7ff3dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3dc0664a0 0x7ff3dc068960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc072090 tx=0x7ff3f0009190 comp rx=0 tx=0).stop 2023-10-30T13:10:42.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 msgr2=0x7ff3fc1515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 --2- 172.21.15.148:0/1134457080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3fc153850 0x7ff3fc1515c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec01ab40 tx=0x7ff3ec019230 comp rx=0 tx=0).stop 2023-10-30T13:10:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fad9d640 1 -- 172.21.15.148:0/1134457080 reap_dead start 2023-10-30T13:10:42.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 --> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae74047f20 con 0x7fae6006e710 2023-10-30T13:10:42.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 shutdown_connections 2023-10-30T13:10:42.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7f575c06e6c0 msgr2=0x7f575c070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f577e7e4640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7f575c06e6c0 0x7f575c070ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 >> 172.21.15.148:0/1134457080 conn(0x7ff3fc0cae10 msgr2=0x7ff3fc0c5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae71ffb640 1 -- 172.21.15.148:0/1814128382 <== osd.6 v2:172.21.15.150:6804/397365804 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fae6c0ab920 con 0x7fae6006e710 2023-10-30T13:10:42.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 shutdown_connections 2023-10-30T13:10:42.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7ff3fb59e640 1 -- 172.21.15.148:0/1134457080 wait complete. 2023-10-30T13:10:42.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fae6006e710 msgr2=0x7fae60070b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fae6006e710 0x7fae60070b30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 --> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa01c0cbcb0 con 0x7f9ffc06e740 2023-10-30T13:10:42.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f575c0664a0 msgr2=0x7f575c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f577e7e4640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f575c0664a0 0x7f575c068960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57780c1af0 tx=0x7f5770007710 comp rx=0 tx=0).stop 2023-10-30T13:10:42.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57780c6f00 msgr2=0x7f5778143c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f577e7e4640 1 --2- 172.21.15.148:0/962744439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57780c6f00 0x7f5778143c80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f577400c4d0 tx=0x7f577400c990 comp rx=0 tx=0).stop 2023-10-30T13:10:42.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f577dfe3640 1 -- 172.21.15.148:0/962744439 reap_dead start 2023-10-30T13:10:42.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 shutdown_connections 2023-10-30T13:10:42.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 >> 172.21.15.148:0/962744439 conn(0x7f57780cae10 msgr2=0x7f577806ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae600664f0 msgr2=0x7fae600689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae600664f0 0x7fae600689b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fae78009c50 tx=0x7fae78009190 comp rx=0 tx=0).stop 2023-10-30T13:10:42.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae740499a0 msgr2=0x7fae740d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 --2- 172.21.15.148:0/1814128382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae740499a0 0x7fae740d0590 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae7c04fab0 tx=0x7fae7c074bb0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae81e66640 1 -- 172.21.15.148:0/1814128382 reap_dead start 2023-10-30T13:10:42.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 shutdown_connections 2023-10-30T13:10:42.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 >> 172.21.15.148:0/1814128382 conn(0x7fae7400e9c0 msgr2=0x7fae740d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 shutdown_connections 2023-10-30T13:10:42.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fae838c9640 1 -- 172.21.15.148:0/1814128382 wait complete. 2023-10-30T13:10:42.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.560+0000 7fa01a7fc640 1 -- 172.21.15.148:0/747672180 <== osd.5 v2:172.21.15.150:6802/2046109052 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa0140a5bd0 con 0x7f9ffc06e740 2023-10-30T13:10:42.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f9ffc06e740 msgr2=0x7f9ffc070b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f9ffc06e740 0x7f9ffc070b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ffc066520 msgr2=0x7f9ffc0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ffc066520 0x7f9ffc0689e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0045f0 tx=0x7fa00c008040 comp rx=0 tx=0).stop 2023-10-30T13:10:42.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c159830 msgr2=0x7fa01c155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 --2- 172.21.15.148:0/747672180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa01c159830 0x7fa01c155950 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa00400e930 tx=0x7fa00400edf0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa0223cf640 1 -- 172.21.15.148:0/747672180 reap_dead start 2023-10-30T13:10:42.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 shutdown_connections 2023-10-30T13:10:42.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 >> 172.21.15.148:0/747672180 conn(0x7fa01c06bfe0 msgr2=0x7fa01c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 shutdown_connections 2023-10-30T13:10:42.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.564+0000 7fa023e32640 1 -- 172.21.15.148:0/747672180 wait complete. 2023-10-30T13:10:42.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.568+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 shutdown_connections 2023-10-30T13:10:42.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.572+0000 7f577e7e4640 1 -- 172.21.15.148:0/962744439 wait complete. 2023-10-30T13:10:42.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.572+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 --> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbcdc15dd30 con 0x7fbcc006e6f0 2023-10-30T13:10:42.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.576+0000 7fbcd2ffd640 1 -- 172.21.15.148:0/3301140172 <== osd.7 v2:172.21.15.150:6824/1265253901 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbcd8003990 con 0x7fbcc006e6f0 2023-10-30T13:10:42.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.576+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 --> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb44c0cbcb0 con 0x7fb43006e710 2023-10-30T13:10:42.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.576+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fbcc006e6f0 msgr2=0x7fbcc0070b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fbce47d3640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fbcc006e6f0 0x7fbcc0070b10 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcc00664d0 msgr2=0x7fbcc0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fbce47d3640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcc00664d0 0x7fbcc0068990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0b2000 tx=0x7fbccc0aa070 comp rx=0 tx=0).stop 2023-10-30T13:10:42.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb4427fc640 1 -- 172.21.15.148:0/1693785808 <== osd.4 v2:172.21.15.150:6800/381432752 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb4300711e0 con 0x7fb43006e710 2023-10-30T13:10:42.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc0c0ed0 msgr2=0x7fbcdc153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fbce47d3640 1 --2- 172.21.15.148:0/3301140172 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc0c0ed0 0x7fbcdc153de0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400cbf0 tx=0x7fbcd400a060 comp rx=0 tx=0).stop 2023-10-30T13:10:42.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fbce2d70640 1 -- 172.21.15.148:0/3301140172 reap_dead start 2023-10-30T13:10:42.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 shutdown_connections 2023-10-30T13:10:42.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 >> 172.21.15.148:0/3301140172 conn(0x7fbcdc0c7910 msgr2=0x7fbcdc06c120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7fb43006e710 msgr2=0x7fb430070b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7fb43006e710 0x7fb430070b30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4300664f0 msgr2=0x7fb4300689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4300664f0 0x7fb4300689b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb4480ad000 tx=0x7fb448008040 comp rx=0 tx=0).stop 2023-10-30T13:10:42.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 msgr2=0x7fb44c142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 --2- 172.21.15.148:0/1693785808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb44c153c60 0x7fb44c142cd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb43c009fd0 tx=0x7fb43c00b870 comp rx=0 tx=0).stop 2023-10-30T13:10:42.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb4527b4640 1 -- 172.21.15.148:0/1693785808 reap_dead start 2023-10-30T13:10:42.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 shutdown_connections 2023-10-30T13:10:42.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 >> 172.21.15.148:0/1693785808 conn(0x7fb44c06bfe0 msgr2=0x7fb44c151c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 shutdown_connections 2023-10-30T13:10:42.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.580+0000 7fb454217640 1 -- 172.21.15.148:0/1693785808 wait complete. 2023-10-30T13:10:42.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.584+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 shutdown_connections 2023-10-30T13:10:42.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.584+0000 7fbce47d3640 1 -- 172.21.15.148:0/3301140172 wait complete. 2023-10-30T13:10:42.669 INFO:teuthology.orchestra.run.smithi148.stdout:55834574857 2023-10-30T13:10:42.669 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T13:10:42.670 INFO:teuthology.orchestra.run.smithi148.stdout:55834574856 2023-10-30T13:10:42.670 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T13:10:42.671 INFO:teuthology.orchestra.run.smithi148.stdout:55834574856 2023-10-30T13:10:42.672 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T13:10:42.672 INFO:teuthology.orchestra.run.smithi148.stdout:55834574856 2023-10-30T13:10:42.673 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T13:10:42.674 INFO:teuthology.orchestra.run.smithi148.stdout:55834574857 2023-10-30T13:10:42.674 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T13:10:42.675 INFO:teuthology.orchestra.run.smithi148.stdout:55834574857 2023-10-30T13:10:42.675 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T13:10:42.676 INFO:teuthology.orchestra.run.smithi148.stdout:55834574856 2023-10-30T13:10:42.676 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T13:10:42.677 INFO:teuthology.orchestra.run.smithi148.stdout:55834574857 2023-10-30T13:10:42.678 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T13:10:42.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.948+0000 7fa48a7b5640 1 Processor -- start 2023-10-30T13:10:42.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.948+0000 7fa48a7b5640 1 -- start start 2023-10-30T13:10:42.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa48a7b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa484157e00 0x7fa48406ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa48a7b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48415e540 0x7fa4841619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa48a7b5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48415e0b0 con 0x7fa4840c5d10 2023-10-30T13:10:42.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa48a7b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4841581e0 con 0x7fa484157e00 2023-10-30T13:10:42.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa48a7b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa484161ee0 con 0x7fa48415e540 2023-10-30T13:10:42.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa489fb4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48415e540 0x7fa4841619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa489fb4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48415e540 0x7fa4841619a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45590/0 (socket says 172.21.15.148:45590) 2023-10-30T13:10:42.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa489fb4640 1 -- 172.21.15.148:0/1451725847 learned_addr learned my addr 172.21.15.148:0/1451725847 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa4897b3640 1 --2- 172.21.15.148:0/1451725847 >> v2:172.21.15.150:3300/0 conn(0x7fa484157e00 0x7fa48406ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa489fb4640 1 -- 172.21.15.148:0/1451725847 >> v2:172.21.15.150:3300/0 conn(0x7fa484157e00 msgr2=0x7fa48406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa489fb4640 1 --2- 172.21.15.148:0/1451725847 >> v2:172.21.15.150:3300/0 conn(0x7fa484157e00 0x7fa48406ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa489fb4640 1 -- 172.21.15.148:0/1451725847 >> v1:172.21.15.148:6789/0 conn(0x7fa4840c5d10 legacy=0x7fa48406cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.952+0000 7fa489fb4640 1 -- 172.21.15.148:0/1451725847 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48415e9d0 con 0x7fa48415e540 2023-10-30T13:10:42.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7f334eb65640 1 Processor -- start 2023-10-30T13:10:42.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7f334eb65640 1 -- start start 2023-10-30T13:10:42.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7fa489fb4640 1 --2- 172.21.15.148:0/1451725847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48415e540 0x7fa4841619a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009e50 tx=0x7fa47c01ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df90de38419a9ba5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7fa47bfff640 1 -- 172.21.15.148:0/1451725847 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0a57d0 con 0x7fa48415e540 2023-10-30T13:10:42.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7fa47bfff640 1 -- 172.21.15.148:0/1451725847 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c0a5930 con 0x7fa48415e540 2023-10-30T13:10:42.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7fa47bfff640 1 -- 172.21.15.148:0/1451725847 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0a5cc0 con 0x7fa48415e540 2023-10-30T13:10:42.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7f334eb65640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f334814fc00 0x7f334806a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7f334eb65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33480c0060 con 0x7f3348156350 2023-10-30T13:10:42.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7f334eb65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33480c0340 con 0x7f33480c74e0 2023-10-30T13:10:42.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7f334eb65640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3348071090 con 0x7f334814fc00 2023-10-30T13:10:42.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7feafce46640 1 Processor -- start 2023-10-30T13:10:42.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7f334c901640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f334814fc00 0x7f334806a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1451725847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48415e540 msgr2=0x7fa4841619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.956+0000 7fa48a7b5640 1 --2- 172.21.15.148:0/1451725847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48415e540 0x7fa4841619a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009e50 tx=0x7fa47c01ca10 comp rx=0 tx=0).stop 2023-10-30T13:10:42.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7f334c901640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f334814fc00 0x7f334806a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41852/0 (socket says 172.21.15.148:41852) 2023-10-30T13:10:42.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7f334c901640 1 -- 172.21.15.148:0/3403894506 learned_addr learned my addr 172.21.15.148:0/3403894506 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1451725847 shutdown_connections 2023-10-30T13:10:42.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7fa48a7b5640 1 --2- 172.21.15.148:0/1451725847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa48415e540 0x7fa4841619a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7fa48a7b5640 1 --2- 172.21.15.148:0/1451725847 >> v2:172.21.15.150:3300/0 conn(0x7fa484157e00 0x7fa48406ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1451725847 >> 172.21.15.148:0/1451725847 conn(0x7fa4840cae10 msgr2=0x7fa4840cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7f334c901640 1 -- 172.21.15.148:0/3403894506 >> v1:172.21.15.150:6789/0 conn(0x7f33480c74e0 legacy=0x7f3348155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7f334c901640 1 -- 172.21.15.148:0/3403894506 >> v1:172.21.15.148:6789/0 conn(0x7f3348156350 legacy=0x7f33481605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7f334c901640 1 -- 172.21.15.148:0/3403894506 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3348071370 con 0x7f334814fc00 2023-10-30T13:10:42.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1451725847 shutdown_connections 2023-10-30T13:10:42.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1451725847 wait complete. 2023-10-30T13:10:42.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7feafce46640 1 -- start start 2023-10-30T13:10:42.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7feafce46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feaf8158090 0x7feaf8161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7feafce46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf80c6b40 0x7feaf8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7feafce46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf80c72b0 con 0x7feaf806a3d0 2023-10-30T13:10:42.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7feafce46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf8162120 con 0x7feaf8158090 2023-10-30T13:10:42.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7feafce46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf80c6fd0 con 0x7feaf80c6b40 2023-10-30T13:10:42.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7fa48a7b5640 1 Processor -- start 2023-10-30T13:10:42.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.960+0000 7fa48a7b5640 1 -- start start 2023-10-30T13:10:42.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7feaf659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feaf8158090 0x7feaf8161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7feaf659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feaf8158090 0x7feaf8161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37674/0 (socket says 172.21.15.148:37674) 2023-10-30T13:10:42.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7feaf659c640 1 -- 172.21.15.148:0/3095334544 learned_addr learned my addr 172.21.15.148:0/3095334544 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7f334c901640 1 --2- 172.21.15.148:0/3403894506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f334814fc00 0x7f334806a720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f333c009870 tx=0x7f333c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a30682e3785578 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7f33477fe640 1 -- 172.21.15.148:0/3403894506 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0a89f0 con 0x7f334814fc00 2023-10-30T13:10:42.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7f33477fe640 1 -- 172.21.15.148:0/3403894506 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c0a8b50 con 0x7f334814fc00 2023-10-30T13:10:42.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7f33477fe640 1 -- 172.21.15.148:0/3403894506 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0b0440 con 0x7f334814fc00 2023-10-30T13:10:42.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa48a7b5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4840c5d10 0x7fa48415d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa48a7b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa484157e00 0x7fa4841f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa48a7b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48415e540 0x7fa48414da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7feaf5d9b640 1 --2- 172.21.15.148:0/3095334544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf80c6b40 0x7feaf8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa48a7b5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4841505c0 con 0x7fa4840c5d10 2023-10-30T13:10:42.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7f334eb65640 1 -- 172.21.15.148:0/3403894506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f334814fc00 msgr2=0x7f334806a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7f334eb65640 1 --2- 172.21.15.148:0/3403894506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f334814fc00 0x7f334806a720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f333c009870 tx=0x7f333c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa48a7b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4841f6610 con 0x7fa48415e540 2023-10-30T13:10:42.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa48a7b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4841f9a90 con 0x7fa484157e00 2023-10-30T13:10:42.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa488fb2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa484157e00 0x7fa4841f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa4897b3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4840c5d10 0x7fa48415d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa489fb4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48415e540 0x7fa48414da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa4897b3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4840c5d10 0x7fa48415d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41864/0 (socket says 172.21.15.148:41864) 2023-10-30T13:10:42.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa4897b3640 1 -- 172.21.15.148:0/1132346934 learned_addr learned my addr 172.21.15.148:0/1132346934 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7f487d3d3640 1 Processor -- start 2023-10-30T13:10:42.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7f487d3d3640 1 -- start start 2023-10-30T13:10:42.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa4897b3640 1 -- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa484157e00 msgr2=0x7fa4841f5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa4897b3640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa484157e00 0x7fa4841f5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa4897b3640 1 -- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48415e540 msgr2=0x7fa48414da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7fa4897b3640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48415e540 0x7fa48414da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4897b3640 1 -- 172.21.15.148:0/1132346934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c009b30 con 0x7fa4840c5d10 2023-10-30T13:10:42.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa489fb4640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48415e540 0x7fa48414da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:42.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7feaf659c640 1 -- 172.21.15.148:0/3095334544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf80c6b40 msgr2=0x7feaf8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7feaf659c640 1 --2- 172.21.15.148:0/3095334544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf80c6b40 0x7feaf8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7feaf659c640 1 -- 172.21.15.148:0/3095334544 >> v1:172.21.15.150:6789/0 conn(0x7feaf806a3d0 legacy=0x7feaf80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.964+0000 7feaf659c640 1 -- 172.21.15.148:0/3095334544 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf815dd30 con 0x7feaf8158090 2023-10-30T13:10:42.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4897b3640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4840c5d10 0x7fa48415d600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa48000e930 tx=0x7fa48000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4e3972640 1 Processor -- start 2023-10-30T13:10:42.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f478a809640 1 Processor -- start 2023-10-30T13:10:42.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa479ffb640 1 -- 172.21.15.148:0/1132346934 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48000ca00 con 0x7fa4840c5d10 2023-10-30T13:10:42.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa479ffb640 1 -- 172.21.15.148:0/1132346934 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48000cb60 con 0x7fa4840c5d10 2023-10-30T13:10:42.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa479ffb640 1 -- 172.21.15.148:0/1132346934 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480013470 con 0x7fa4840c5d10 2023-10-30T13:10:42.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4e3972640 1 -- start start 2023-10-30T13:10:42.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f334eb65640 1 -- 172.21.15.148:0/3403894506 shutdown_connections 2023-10-30T13:10:42.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f334eb65640 1 --2- 172.21.15.148:0/3403894506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f334814fc00 0x7f334806a720 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f334eb65640 1 -- 172.21.15.148:0/3403894506 >> 172.21.15.148:0/3403894506 conn(0x7f33480c7910 msgr2=0x7f33480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f487d3d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4878153c60 0x7f48780ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f487d3d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4878159830 0x7f4878159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f487d3d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48780cb780 con 0x7f48780c4430 2023-10-30T13:10:42.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f487d3d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4878154040 con 0x7f4878159830 2023-10-30T13:10:42.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f487d3d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487815a2e0 con 0x7f4878153c60 2023-10-30T13:10:42.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4841f9d70 con 0x7fa4840c5d10 2023-10-30T13:10:42.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f48767fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f48780c4430 0x7f48780cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39320/0 (socket says 172.21.15.148:39320) 2023-10-30T13:10:42.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f48767fc640 1 -- 172.21.15.148:0/175704424 learned_addr learned my addr 172.21.15.148:0/175704424 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:42.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4841528b0 con 0x7fa4840c5d10 2023-10-30T13:10:42.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f48777fe640 1 --2- 172.21.15.148:0/175704424 >> v2:172.21.15.150:3300/0 conn(0x7f4878159830 0x7f4878159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4e3972640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 0x7fa4dc0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4e3972640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa4dc0c4430 0x7fa4dc0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4e3972640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa4dc159830 0x7fa4dc159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4e3972640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4dc154040 con 0x7fa4dc0c4430 2023-10-30T13:10:42.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4e3972640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4dc0cb780 con 0x7fa4dc159830 2023-10-30T13:10:42.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa4e3972640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc0cc9f0 con 0x7fa4dc153c60 2023-10-30T13:10:42.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f478a809640 1 -- start start 2023-10-30T13:10:42.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f478a809640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4784158090 0x7f4784161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f478a809640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f47840c6b40 0x7f4784069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f478a809640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47840c72b0 con 0x7f478406a3d0 2023-10-30T13:10:42.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f478a809640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4784162120 con 0x7f47840c6b40 2023-10-30T13:10:42.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f478a809640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47840c6fd0 con 0x7f4784158090 2023-10-30T13:10:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f48777fe640 1 -- 172.21.15.148:0/175704424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4878153c60 msgr2=0x7f48780ca360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:10:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f48777fe640 1 --2- 172.21.15.148:0/175704424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4878153c60 0x7f48780ca360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7f48777fe640 1 -- 172.21.15.148:0/175704424 >> v1:172.21.15.148:6789/0 conn(0x7f48780c4430 legacy=0x7f48780cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.968+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa484152430 con 0x7fa4840c5d10 2023-10-30T13:10:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e1f0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa4dc159830 0x7fa4dc159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f48777fe640 1 -- 172.21.15.148:0/175704424 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487815a5c0 con 0x7f4878159830 2023-10-30T13:10:42.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f4788da6640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f478406a3d0 0x7f47840bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39328/0 (socket says 172.21.15.148:39328) 2023-10-30T13:10:42.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f4788da6640 1 -- 172.21.15.148:0/3821523238 learned_addr learned my addr 172.21.15.148:0/3821523238 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:42.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f4783fff640 1 --2- 172.21.15.148:0/3821523238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4784158090 0x7f4784161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f47837fe640 1 --2- 172.21.15.148:0/3821523238 >> v2:172.21.15.148:3300/0 conn(0x7f47840c6b40 0x7f4784069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e0f0d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa4dc0c4430 0x7fa4dc0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e170e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 0x7fa4dc0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e0f0d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa4dc0c4430 0x7fa4dc0cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41880/0 (socket says 172.21.15.148:41880) 2023-10-30T13:10:42.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e0f0d640 1 -- 172.21.15.148:0/4107183751 learned_addr learned my addr 172.21.15.148:0/4107183751 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:42.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7feaf659c640 1 --2- 172.21.15.148:0/3095334544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf8158090 0x7feaf8161be0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feae8009870 tx=0x7feae801ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=531665ab7a8181e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f47837fe640 1 -- 172.21.15.148:0/3821523238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4784158090 msgr2=0x7f4784161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f47837fe640 1 --2- 172.21.15.148:0/3821523238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4784158090 0x7f4784161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f47837fe640 1 -- 172.21.15.148:0/3821523238 >> v1:172.21.15.148:6789/0 conn(0x7f478406a3d0 legacy=0x7f47840bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f47837fe640 1 -- 172.21.15.148:0/3821523238 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478415dd30 con 0x7f47840c6b40 2023-10-30T13:10:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7feaf559a640 1 -- 172.21.15.148:0/3095334544 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae80a89b0 con 0x7feaf8158090 2023-10-30T13:10:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7feaf559a640 1 -- 172.21.15.148:0/3095334544 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae80a8b10 con 0x7feaf8158090 2023-10-30T13:10:42.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7feaf559a640 1 -- 172.21.15.148:0/3095334544 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae80a8ea0 con 0x7feaf8158090 2023-10-30T13:10:42.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e1f0f640 1 -- 172.21.15.148:0/4107183751 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 msgr2=0x7fa4dc0ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e1f0f640 1 --2- 172.21.15.148:0/4107183751 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 0x7fa4dc0ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e1f0f640 1 -- 172.21.15.148:0/4107183751 >> v2:172.21.15.148:3300/0 conn(0x7fa4dc0c4430 msgr2=0x7fa4dc0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:42.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e1f0f640 1 --2- 172.21.15.148:0/4107183751 >> v2:172.21.15.148:3300/0 conn(0x7fa4dc0c4430 0x7fa4dc0cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa4e1f0f640 1 -- 172.21.15.148:0/4107183751 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc15a2e0 con 0x7fa4dc159830 2023-10-30T13:10:42.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa479ffb640 1 -- 172.21.15.148:0/1132346934 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4800135d0 con 0x7fa4840c5d10 2023-10-30T13:10:42.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f334eb65640 1 -- 172.21.15.148:0/3403894506 shutdown_connections 2023-10-30T13:10:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7fa479ffb640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa468066570 0x7fa468068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7fa479ffb640 1 -- 172.21.15.148:0/1132346934 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa480083fc0 con 0x7fa4840c5d10 2023-10-30T13:10:42.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7fa488fb2640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa468066570 0x7fa468068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:42.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7feafce46640 1 -- 172.21.15.148:0/3095334544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf8158090 msgr2=0x7feaf8161be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:42.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7feafce46640 1 --2- 172.21.15.148:0/3095334544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf8158090 0x7feaf8161be0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feae8009870 tx=0x7feae801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.972+0000 7f334eb65640 1 -- 172.21.15.148:0/3403894506 wait complete. 2023-10-30T13:10:42.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f48777fe640 1 --2- 172.21.15.148:0/175704424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878159830 0x7f4878159ce0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4870009d90 tx=0x7f487001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed681958e514696 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:42.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7feafce46640 1 -- 172.21.15.148:0/3095334544 shutdown_connections 2023-10-30T13:10:42.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7feafce46640 1 --2- 172.21.15.148:0/3095334544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf80c6b40 0x7feaf8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7feafce46640 1 --2- 172.21.15.148:0/3095334544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf8158090 0x7feaf8161be0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:42.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7feafce46640 1 -- 172.21.15.148:0/3095334544 >> 172.21.15.148:0/3095334544 conn(0x7feaf80c7910 msgr2=0x7feaf80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:42.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7feafce46640 1 -- 172.21.15.148:0/3095334544 shutdown_connections 2023-10-30T13:10:42.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7feafce46640 1 -- 172.21.15.148:0/3095334544 wait complete. 2023-10-30T13:10:43.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f4875ffb640 1 -- 172.21.15.148:0/175704424 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700a5920 con 0x7f4878159830 2023-10-30T13:10:43.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f4875ffb640 1 -- 172.21.15.148:0/175704424 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48700a5a80 con 0x7f4878159830 2023-10-30T13:10:43.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f4875ffb640 1 -- 172.21.15.148:0/175704424 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700a5e10 con 0x7f4878159830 2023-10-30T13:10:43.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7fa4e1f0f640 1 --2- 172.21.15.148:0/4107183751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4dc159830 0x7fa4dc159ce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc08d1a0 tx=0x7fa4d401c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84a5136950a8aaf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f334eb65640 1 Processor -- start 2023-10-30T13:10:43.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f334eb65640 1 -- start start 2023-10-30T13:10:43.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7fa488fb2640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa468066570 0x7fa468068a30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4840c13e0 tx=0x7fa474009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f487d3d3640 1 -- 172.21.15.148:0/175704424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878159830 msgr2=0x7f4878159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f487d3d3640 1 --2- 172.21.15.148:0/175704424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878159830 0x7f4878159ce0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4870009d90 tx=0x7f487001c970 comp rx=0 tx=0).stop 2023-10-30T13:10:43.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7fa4d3fff640 1 -- 172.21.15.148:0/4107183751 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40a5880 con 0x7fa4dc159830 2023-10-30T13:10:43.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f47837fe640 1 --2- 172.21.15.148:0/3821523238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47840c6b40 0x7f4784069e90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f47740097c0 tx=0x7f477401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7482cbf99c316260 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7fa4d3fff640 1 -- 172.21.15.148:0/4107183751 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d40a59e0 con 0x7fa4dc159830 2023-10-30T13:10:43.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7fa4d3fff640 1 -- 172.21.15.148:0/4107183751 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40a5880 con 0x7fa4dc159830 2023-10-30T13:10:43.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f487d3d3640 1 -- 172.21.15.148:0/175704424 shutdown_connections 2023-10-30T13:10:43.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f487d3d3640 1 --2- 172.21.15.148:0/175704424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878159830 0x7f4878159ce0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f487d3d3640 1 --2- 172.21.15.148:0/175704424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4878153c60 0x7f48780ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f487d3d3640 1 -- 172.21.15.148:0/175704424 >> 172.21.15.148:0/175704424 conn(0x7f487806bfe0 msgr2=0x7f487806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f487d3d3640 1 -- 172.21.15.148:0/175704424 shutdown_connections 2023-10-30T13:10:43.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334eb65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33480c74e0 0x7f3348166000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7fa4e3972640 1 -- 172.21.15.148:0/4107183751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4dc159830 msgr2=0x7fa4dc159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7fa4e3972640 1 --2- 172.21.15.148:0/4107183751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4dc159830 0x7fa4dc159ce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc08d1a0 tx=0x7fa4d401c910 comp rx=0 tx=0).stop 2023-10-30T13:10:43.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feafce46640 1 Processor -- start 2023-10-30T13:10:43.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334eb65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f334814fc00 0x7f334813a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7fa4e3972640 1 -- 172.21.15.148:0/4107183751 shutdown_connections 2023-10-30T13:10:43.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feafce46640 1 -- start start 2023-10-30T13:10:43.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334eb65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3348156350 0x7f334813a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334eb65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3348145050 con 0x7f33480c74e0 2023-10-30T13:10:43.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334eb65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334814f0d0 con 0x7f3348156350 2023-10-30T13:10:43.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334eb65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33481f2650 con 0x7f334814fc00 2023-10-30T13:10:43.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7fa4e3972640 1 --2- 172.21.15.148:0/4107183751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4dc159830 0x7fa4dc159ce0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334c901640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33480c74e0 0x7f3348166000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334c901640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33480c74e0 0x7f3348166000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41884/0 (socket says 172.21.15.148:41884) 2023-10-30T13:10:43.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334c901640 1 -- 172.21.15.148:0/1363214476 learned_addr learned my addr 172.21.15.148:0/1363214476 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:43.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7fa4e3972640 1 --2- 172.21.15.148:0/4107183751 >> v2:172.21.15.148:3300/0 conn(0x7fa4dc0c4430 0x7fa4dc0cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7fa4e3972640 1 --2- 172.21.15.148:0/4107183751 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 0x7fa4dc0ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7fa4e3972640 1 -- 172.21.15.148:0/4107183751 >> 172.21.15.148:0/4107183751 conn(0x7fa4dc06bfe0 msgr2=0x7fa4dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feafce46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf806a3d0 0x7feaf80c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334d102640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3348156350 0x7f334813a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334c901640 1 -- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f334814fc00 msgr2=0x7f334813a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feafce46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feaf8158090 0x7feaf80c0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feafce46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf8153490 0x7feaf814d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feafce46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf815a550 con 0x7feaf8158090 2023-10-30T13:10:43.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feafce46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf815ab50 con 0x7feaf8153490 2023-10-30T13:10:43.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feafce46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf80c90a0 con 0x7feaf806a3d0 2023-10-30T13:10:43.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf659c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf806a3d0 0x7feaf80c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf659c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf806a3d0 0x7feaf80c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45662/0 (socket says 172.21.15.148:45662) 2023-10-30T13:10:43.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf659c640 1 -- 172.21.15.148:0/330111808 learned_addr learned my addr 172.21.15.148:0/330111808 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:43.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf6d9d640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf8153490 0x7feaf814d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f3347fff640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f334814fc00 0x7f334813a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334c901640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f334814fc00 0x7f334813a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334c901640 1 -- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3348156350 msgr2=0x7f334813a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334c901640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3348156350 0x7f334813a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334c901640 1 -- 172.21.15.148:0/1363214476 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c0094d0 con 0x7f33480c74e0 2023-10-30T13:10:43.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334d102640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3348156350 0x7f334813a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:43.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.976+0000 7f487d3d3640 1 -- 172.21.15.148:0/175704424 wait complete. 2023-10-30T13:10:43.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f487d3d3640 1 Processor -- start 2023-10-30T13:10:43.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf5d9b640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feaf8158090 0x7feaf80c0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7fa4e3972640 1 -- 172.21.15.148:0/4107183751 shutdown_connections 2023-10-30T13:10:43.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7fa4e3972640 1 -- 172.21.15.148:0/4107183751 wait complete. 2023-10-30T13:10:43.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf659c640 1 -- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf8153490 msgr2=0x7feaf814d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf659c640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf8153490 0x7feaf814d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf659c640 1 -- 172.21.15.148:0/330111808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feaf8158090 msgr2=0x7feaf80c0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf659c640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feaf8158090 0x7feaf80c0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f487d3d3640 1 -- start start 2023-10-30T13:10:43.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf659c640 1 -- 172.21.15.148:0/330111808 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80094d0 con 0x7feaf806a3d0 2023-10-30T13:10:43.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7f334c901640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33480c74e0 0x7f3348166000 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f333c01b900 tx=0x7f333c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf5d9b640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feaf8158090 0x7feaf80c0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:43.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.980+0000 7feaf659c640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf806a3d0 0x7feaf80c9af0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feae801c920 tx=0x7feae801ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7feae6ffd640 1 -- 172.21.15.148:0/330111808 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae80b1df0 con 0x7feaf806a3d0 2023-10-30T13:10:43.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f33457fa640 1 -- 172.21.15.148:0/1363214476 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0b0e40 con 0x7f33480c74e0 2023-10-30T13:10:43.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf80c8dd0 con 0x7feaf806a3d0 2023-10-30T13:10:43.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf815b2b0 con 0x7feaf806a3d0 2023-10-30T13:10:43.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7feae6ffd640 1 -- 172.21.15.148:0/330111808 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae80bb470 con 0x7feaf806a3d0 2023-10-30T13:10:43.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7feae6ffd640 1 -- 172.21.15.148:0/330111808 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae80bb7e0 con 0x7feaf806a3d0 2023-10-30T13:10:43.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33481f28a0 con 0x7f33480c74e0 2023-10-30T13:10:43.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f33457fa640 1 -- 172.21.15.148:0/1363214476 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c0ba440 con 0x7f33480c74e0 2023-10-30T13:10:43.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f33457fa640 1 -- 172.21.15.148:0/1363214476 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0ba7f0 con 0x7f33480c74e0 2023-10-30T13:10:43.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f4782ffd640 1 -- 172.21.15.148:0/3821523238 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740a88f0 con 0x7f47840c6b40 2023-10-30T13:10:43.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33481f2e60 con 0x7f33480c74e0 2023-10-30T13:10:43.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f4782ffd640 1 -- 172.21.15.148:0/3821523238 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47740a8a50 con 0x7f47840c6b40 2023-10-30T13:10:43.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f4782ffd640 1 -- 172.21.15.148:0/3821523238 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740a88f0 con 0x7f47840c6b40 2023-10-30T13:10:43.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f487d3d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48780c4430 0x7f487813ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa479ffb640 1 -- 172.21.15.148:0/1132346934 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa484152430 con 0x7fa4840c5d10 2023-10-30T13:10:43.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e3972640 1 Processor -- start 2023-10-30T13:10:43.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e3972640 1 -- start start 2023-10-30T13:10:43.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f487d3d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878153c60 0x7f487806f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33480b0d00 con 0x7f33480c74e0 2023-10-30T13:10:43.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f487d3d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4878159830 0x7f4878164680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f487d3d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48780cef80 con 0x7f4878159830 2023-10-30T13:10:43.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f487d3d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48780cd200 con 0x7f4878153c60 2023-10-30T13:10:43.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f487d3d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48780cd4e0 con 0x7f48780c4430 2023-10-30T13:10:43.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f48767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878153c60 0x7f487806f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f478a809640 1 -- 172.21.15.148:0/3821523238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47840c6b40 msgr2=0x7f4784069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f478a809640 1 --2- 172.21.15.148:0/3821523238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47840c6b40 0x7f4784069e90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f47740097c0 tx=0x7f477401ec30 comp rx=0 tx=0).stop 2023-10-30T13:10:43.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f48767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878153c60 0x7f487806f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37714/0 (socket says 172.21.15.148:37714) 2023-10-30T13:10:43.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f48767fc640 1 -- 172.21.15.148:0/181985372 learned_addr learned my addr 172.21.15.148:0/181985372 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:43.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feadc005260 con 0x7feaf806a3d0 2023-10-30T13:10:43.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e3972640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4dc0c4430 0x7fa4dc0b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f478a809640 1 -- 172.21.15.148:0/3821523238 shutdown_connections 2023-10-30T13:10:43.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f478a809640 1 --2- 172.21.15.148:0/3821523238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47840c6b40 0x7f4784069e90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f478a809640 1 --2- 172.21.15.148:0/3821523238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4784158090 0x7f4784161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f478a809640 1 -- 172.21.15.148:0/3821523238 >> 172.21.15.148:0/3821523238 conn(0x7f47840c7910 msgr2=0x7f47840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f478a809640 1 -- 172.21.15.148:0/3821523238 shutdown_connections 2023-10-30T13:10:43.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e3972640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 0x7fa4dc142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e3972640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4dc159830 0x7fa4dc0cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e3972640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc1435e0 con 0x7fa4dc0c4430 2023-10-30T13:10:43.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e3972640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc0b4c30 con 0x7fa4dc159830 2023-10-30T13:10:43.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e3972640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc0ce370 con 0x7fa4dc153c60 2023-10-30T13:10:43.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e170e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4dc0c4430 0x7fa4dc0b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e170e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4dc0c4430 0x7fa4dc0b50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41908/0 (socket says 172.21.15.148:41908) 2023-10-30T13:10:43.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7fa4e170e640 1 -- 172.21.15.148:0/3148509183 learned_addr learned my addr 172.21.15.148:0/3148509183 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:43.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fa4e1f0f640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4dc159830 0x7fa4dc0cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f4876ffd640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48780c4430 0x7f487813ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f48767fc640 1 -- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48780c4430 msgr2=0x7f487813ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f48767fc640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48780c4430 0x7f487813ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fa4e0f0d640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 0x7fa4dc142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f48767fc640 1 -- 172.21.15.148:0/181985372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4878159830 msgr2=0x7f4878164680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f48767fc640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4878159830 0x7f4878164680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fa4e170e640 1 -- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 msgr2=0x7fa4dc142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fa4e170e640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 0x7fa4dc142ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fa4e170e640 1 -- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4dc159830 msgr2=0x7fa4dc0cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fa4e170e640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4dc159830 0x7fa4dc0cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fa4e170e640 1 -- 172.21.15.148:0/3148509183 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d4009a70 con 0x7fa4dc0c4430 2023-10-30T13:10:43.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.984+0000 7f478a809640 1 -- 172.21.15.148:0/3821523238 wait complete. 2023-10-30T13:10:43.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f48767fc640 1 -- 172.21.15.148:0/181985372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4870009a70 con 0x7f4878153c60 2023-10-30T13:10:43.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fa4e170e640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4dc0c4430 0x7fa4dc0b50c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800e960 tx=0x7fa4d800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7feae6ffd640 1 -- 172.21.15.148:0/330111808 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feae80b7070 con 0x7feaf806a3d0 2023-10-30T13:10:43.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f48767fc640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878153c60 0x7f487806f1c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f486c00c430 tx=0x7f486c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f478a809640 1 Processor -- start 2023-10-30T13:10:43.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fee3572a640 1 Processor -- start 2023-10-30T13:10:43.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f33457fa640 1 -- 172.21.15.148:0/1363214476 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f333c0ba950 con 0x7f33480c74e0 2023-10-30T13:10:43.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f33457fa640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3334066570 0x7f3334068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fee3572a640 1 -- start start 2023-10-30T13:10:43.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f485f7fe640 1 -- 172.21.15.148:0/181985372 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c005cf0 con 0x7f4878153c60 2023-10-30T13:10:43.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f478a809640 1 -- start start 2023-10-30T13:10:43.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48780cd7c0 con 0x7f4878153c60 2023-10-30T13:10:43.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f485f7fe640 1 -- 172.21.15.148:0/181985372 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c005e50 con 0x7f4878153c60 2023-10-30T13:10:43.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f485f7fe640 1 -- 172.21.15.148:0/181985372 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c005cf0 con 0x7f4878153c60 2023-10-30T13:10:43.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f33457fa640 1 -- 172.21.15.148:0/1363214476 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f333c121ed0 con 0x7f33480c74e0 2023-10-30T13:10:43.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7feae6ffd640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fead80664a0 0x7fead8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f3347fff640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3334066570 0x7f3334068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7feae6ffd640 1 -- 172.21.15.148:0/330111808 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feae8121e90 con 0x7feaf806a3d0 2023-10-30T13:10:43.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f478a809640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f478406a3d0 0x7f4784147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f478a809640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4784158090 0x7f47840c4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7feaf5d9b640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fead80664a0 0x7fead8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48780cdd80 con 0x7f4878153c60 2023-10-30T13:10:43.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fee3572a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee30153c60 0x7fee300ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fee3572a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee30159830 0x7fee30159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fee3572a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee300cb780 con 0x7fee300c4430 2023-10-30T13:10:43.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fee3572a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee30154040 con 0x7fee30153c60 2023-10-30T13:10:43.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.988+0000 7fee3572a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3015a2e0 con 0x7fee30159830 2023-10-30T13:10:43.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f4783fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f478406a3d0 0x7f4784147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f478a809640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784149490 0x7f47840c2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f4783fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f478406a3d0 0x7f4784147a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45698/0 (socket says 172.21.15.148:45698) 2023-10-30T13:10:43.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f47837fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4784158090 0x7f47840c4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7fee2f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee30159830 0x7fee30159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7fee2f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee30159830 0x7fee30159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37746/0 (socket says 172.21.15.148:37746) 2023-10-30T13:10:43.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7fee2f7fe640 1 -- 172.21.15.148:0/3437525980 learned_addr learned my addr 172.21.15.148:0/3437525980 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:43.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f47837fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4784158090 0x7f47840c4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41920/0 (socket says 172.21.15.148:41920) 2023-10-30T13:10:43.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f4783fff640 1 -- 172.21.15.148:0/2653343588 learned_addr learned my addr 172.21.15.148:0/2653343588 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:43.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47840c3d00 con 0x7f4784158090 2023-10-30T13:10:43.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47840c51d0 con 0x7f4784149490 2023-10-30T13:10:43.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4784140a70 con 0x7f478406a3d0 2023-10-30T13:10:43.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392f59e640 1 Processor -- start 2023-10-30T13:10:43.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392f59e640 1 -- start start 2023-10-30T13:10:43.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f4788da6640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784149490 0x7f47840c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f47837fe640 1 -- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f478406a3d0 msgr2=0x7f4784147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f47837fe640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f478406a3d0 0x7f4784147a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f47837fe640 1 -- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784149490 msgr2=0x7f47840c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f47837fe640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784149490 0x7f47840c2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f47837fe640 1 -- 172.21.15.148:0/2653343588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4774009420 con 0x7f4784158090 2023-10-30T13:10:43.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f4788da6640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784149490 0x7f47840c2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:43.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392f59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f39300c5550 0x7f39300cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392f59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cb050 0x7f39300cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392f59e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39300cba30 con 0x7f39300cab00 2023-10-30T13:10:43.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392f59e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39300ca290 con 0x7f39300c5550 2023-10-30T13:10:43.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392f59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39300c9cf0 con 0x7f39300cb050 2023-10-30T13:10:43.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7fee2dffb640 1 -- 172.21.15.148:0/3437525980 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594725839 0 0) 0x7fee300cb780 con 0x7fee300c4430 2023-10-30T13:10:43.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7fee2dffb640 1 -- 172.21.15.148:0/3437525980 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee18003530 con 0x7fee300c4430 2023-10-30T13:10:43.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48780cdf70 con 0x7f4878153c60 2023-10-30T13:10:43.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f39300c5550 0x7f39300cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f47837fe640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4784158090 0x7f47840c4890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f47740af000 tx=0x7f477401b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f39300c5550 0x7f39300cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41922/0 (socket says 172.21.15.148:41922) 2023-10-30T13:10:43.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f392e59c640 1 -- 172.21.15.148:0/3535687448 learned_addr learned my addr 172.21.15.148:0/3535687448 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:43.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f3925d9b640 1 --2- 172.21.15.148:0/3535687448 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cb050 0x7f39300cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7f485f7fe640 1 -- 172.21.15.148:0/181985372 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f486c0047f0 con 0x7f4878153c60 2023-10-30T13:10:43.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7fee2f7fe640 1 -- 172.21.15.148:0/3437525980 >> v2:172.21.15.148:3300/0 conn(0x7fee30153c60 msgr2=0x7fee300ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:43.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7fee2f7fe640 1 --2- 172.21.15.148:0/3437525980 >> v2:172.21.15.148:3300/0 conn(0x7fee30153c60 0x7fee300ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7fee2f7fe640 1 -- 172.21.15.148:0/3437525980 >> v1:172.21.15.150:6789/0 conn(0x7fee300c4430 legacy=0x7fee300cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.992+0000 7fee2f7fe640 1 -- 172.21.15.148:0/3437525980 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3015a5c0 con 0x7fee30159830 2023-10-30T13:10:43.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f3347fff640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3334066570 0x7f3334068a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3348070cf0 tx=0x7f3338008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7feaf5d9b640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fead80664a0 0x7fead8068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feaf81620f0 tx=0x7feaec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f485f7fe640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4864066560 0x7f4864068a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f485f7fe640 1 -- 172.21.15.148:0/181985372 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f486c083280 con 0x7f4878153c60 2023-10-30T13:10:43.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f4876ffd640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4864066560 0x7f4864068a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f4876ffd640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4864066560 0x7f4864068a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4868009980 tx=0x7f48680090e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f392e59c640 1 -- 172.21.15.148:0/3535687448 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cb050 msgr2=0x7f39300cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:43.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f392e59c640 1 --2- 172.21.15.148:0/3535687448 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cb050 0x7f39300cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f392e59c640 1 -- 172.21.15.148:0/3535687448 >> v1:172.21.15.148:6789/0 conn(0x7f39300cab00 legacy=0x7f39300cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:43.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f392e59c640 1 -- 172.21.15.148:0/3535687448 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393015d990 con 0x7f39300c5550 2023-10-30T13:10:43.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7fa4d1ffb640 1 -- 172.21.15.148:0/3148509183 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d800ca00 con 0x7fa4dc0c4430 2023-10-30T13:10:43.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc0ce610 con 0x7fa4dc0c4430 2023-10-30T13:10:43.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7fa4d1ffb640 1 -- 172.21.15.148:0/3148509183 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d800cb60 con 0x7fa4dc0c4430 2023-10-30T13:10:43.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7fa4d1ffb640 1 -- 172.21.15.148:0/3148509183 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d800ced0 con 0x7fa4dc0c4430 2023-10-30T13:10:43.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f4780ff9640 1 -- 172.21.15.148:0/2653343588 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774004510 con 0x7f4784158090 2023-10-30T13:10:43.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f4780ff9640 1 -- 172.21.15.148:0/2653343588 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4774003620 con 0x7f4784158090 2023-10-30T13:10:43.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f4780ff9640 1 -- 172.21.15.148:0/2653343588 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774003990 con 0x7f4784158090 2023-10-30T13:10:43.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4dc0cee80 con 0x7fa4dc0c4430 2023-10-30T13:10:43.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4784140d50 con 0x7f4784158090 2023-10-30T13:10:43.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:42.996+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4784141310 con 0x7f4784158090 2023-10-30T13:10:43.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f392e59c640 1 --2- 172.21.15.148:0/3535687448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 0x7f39300cef00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3920009e50 tx=0x7f392001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737d3dd478d95ba2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f33457fa640 1 -- 172.21.15.148:0/1363214476 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f333c0ef2a0 con 0x7f33480c74e0 2023-10-30T13:10:43.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7fee2f7fe640 1 --2- 172.21.15.148:0/3437525980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30159830 0x7fee30159cc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fee30073da0 tx=0x7fee2801c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=635cd7ca3b446c8e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478404e170 con 0x7f4784158090 2023-10-30T13:10:43.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc04e170 con 0x7fa4dc0c4430 2023-10-30T13:10:43.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f392dd9b640 1 -- 172.21.15.148:0/3535687448 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200a5740 con 0x7f39300c5550 2023-10-30T13:10:43.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f392dd9b640 1 -- 172.21.15.148:0/3535687448 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39200a58a0 con 0x7f39300c5550 2023-10-30T13:10:43.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f392dd9b640 1 -- 172.21.15.148:0/3535687448 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200a5c30 con 0x7f39300c5550 2023-10-30T13:10:43.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f4780ff9640 1 -- 172.21.15.148:0/2653343588 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47740b7040 con 0x7f4784158090 2023-10-30T13:10:43.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f4780ff9640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47640664f0 0x7f47640689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f392f59e640 1 -- 172.21.15.148:0/3535687448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 msgr2=0x7f39300cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f392f59e640 1 --2- 172.21.15.148:0/3535687448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 0x7f39300cef00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3920009e50 tx=0x7f392001c9d0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7fa4d1ffb640 1 -- 172.21.15.148:0/3148509183 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4d8013470 con 0x7fa4dc0c4430 2023-10-30T13:10:43.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f4780ff9640 1 -- 172.21.15.148:0/2653343588 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4774121600 con 0x7f4784158090 2023-10-30T13:10:43.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f392f59e640 1 -- 172.21.15.148:0/3535687448 shutdown_connections 2023-10-30T13:10:43.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.000+0000 7f392f59e640 1 --2- 172.21.15.148:0/3535687448 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cb050 0x7f39300cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7f392f59e640 1 --2- 172.21.15.148:0/3535687448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 0x7f39300cef00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7f392f59e640 1 -- 172.21.15.148:0/3535687448 >> 172.21.15.148:0/3535687448 conn(0x7f393006bed0 msgr2=0x7f393006c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7f4783fff640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47640664f0 0x7f47640689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7f392f59e640 1 -- 172.21.15.148:0/3535687448 shutdown_connections 2023-10-30T13:10:43.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7f392f59e640 1 -- 172.21.15.148:0/3535687448 wait complete. 2023-10-30T13:10:43.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7fa4d1ffb640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4c00664a0 0x7fa4c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7fa4d1ffb640 1 -- 172.21.15.148:0/3148509183 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4d80848d0 con 0x7fa4dc0c4430 2023-10-30T13:10:43.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7fa4e0f0d640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4c00664a0 0x7fa4c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7feae6ffd640 1 -- 172.21.15.148:0/330111808 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feae80ef290 con 0x7feaf806a3d0 2023-10-30T13:10:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7f4783fff640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47640664f0 0x7f47640689b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f477c0094c0 tx=0x7f477c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7fa4e0f0d640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4c00664a0 0x7fa4c0068960 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc158be0 tx=0x7fa4cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7f392f59e640 1 Processor -- start 2023-10-30T13:10:43.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.004+0000 7f392f59e640 1 -- start start 2023-10-30T13:10:43.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392f59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 0x7f393015cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392f59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cab00 0x7f39301f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392f59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39300cb050 0x7f39301f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392f59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39301f1770 con 0x7f39300c5550 2023-10-30T13:10:43.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f3925d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cab00 0x7f39301f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f3925d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cab00 0x7f39301f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45718/0 (socket says 172.21.15.148:45718) 2023-10-30T13:10:43.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f3925d9b640 1 -- 172.21.15.148:0/1639105932 learned_addr learned my addr 172.21.15.148:0/1639105932 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:43.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39301537d0 con 0x7f39300cb050 2023-10-30T13:10:43.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392ed9d640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39300cb050 0x7f39301f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393014f8b0 con 0x7f39300cab00 2023-10-30T13:10:43.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392e59c640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 0x7f393015cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392e59c640 1 -- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cab00 msgr2=0x7f39301f1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392e59c640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cab00 0x7f39301f1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392e59c640 1 -- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39300cb050 msgr2=0x7f39301f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392e59c640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39300cb050 0x7f39301f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7fee2dffb640 1 -- 172.21.15.148:0/3437525980 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee280a5780 con 0x7fee30159830 2023-10-30T13:10:43.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392e59c640 1 -- 172.21.15.148:0/1639105932 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920009b30 con 0x7f39300c5550 2023-10-30T13:10:43.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f3925d9b640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cab00 0x7f39301f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:43.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f485f7fe640 1 -- 172.21.15.148:0/181985372 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f486c0506a0 con 0x7f4878153c60 2023-10-30T13:10:43.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7fee2dffb640 1 -- 172.21.15.148:0/3437525980 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee280a58e0 con 0x7fee30159830 2023-10-30T13:10:43.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7fee2dffb640 1 -- 172.21.15.148:0/3437525980 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee280a5c70 con 0x7fee30159830 2023-10-30T13:10:43.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7f392e59c640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 0x7f393015cb40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f39200ab000 tx=0x7f3920019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7fee3572a640 1 -- 172.21.15.148:0/3437525980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30159830 msgr2=0x7fee30159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.008+0000 7fee3572a640 1 --2- 172.21.15.148:0/3437525980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30159830 0x7fee30159cc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fee30073da0 tx=0x7fee2801c9f0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.016+0000 7fee3572a640 1 -- 172.21.15.148:0/3437525980 shutdown_connections 2023-10-30T13:10:43.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.016+0000 7fee3572a640 1 --2- 172.21.15.148:0/3437525980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30159830 0x7fee30159cc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.016+0000 7fee3572a640 1 --2- 172.21.15.148:0/3437525980 >> v2:172.21.15.148:3300/0 conn(0x7fee30153c60 0x7fee300ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.016+0000 7fee3572a640 1 -- 172.21.15.148:0/3437525980 >> 172.21.15.148:0/3437525980 conn(0x7fee3006bfe0 msgr2=0x7fee3006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.016+0000 7fee3572a640 1 -- 172.21.15.148:0/3437525980 shutdown_connections 2023-10-30T13:10:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.016+0000 7fee3572a640 1 -- 172.21.15.148:0/3437525980 wait complete. 2023-10-30T13:10:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.016+0000 7fa4d1ffb640 1 -- 172.21.15.148:0/3148509183 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4d8051d30 con 0x7fa4dc0c4430 2023-10-30T13:10:43.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.016+0000 7fee3572a640 1 Processor -- start 2023-10-30T13:10:43.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.016+0000 7fee3572a640 1 -- start start 2023-10-30T13:10:43.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7f39277fe640 1 -- 172.21.15.148:0/1639105932 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200a5740 con 0x7f39300c5550 2023-10-30T13:10:43.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7f39277fe640 1 -- 172.21.15.148:0/1639105932 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39200b8670 con 0x7f39300c5550 2023-10-30T13:10:43.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7f39277fe640 1 -- 172.21.15.148:0/1639105932 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200b89e0 con 0x7f39300c5550 2023-10-30T13:10:43.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39301f5820 con 0x7f39300c5550 2023-10-30T13:10:43.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee3572a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee300c4430 0x7fee30144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7f4780ff9640 1 -- 172.21.15.148:0/2653343588 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47740eeaf0 con 0x7f4784158090 2023-10-30T13:10:43.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee2effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee300c4430 0x7fee30144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee3572a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30153c60 0x7fee301451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee3572a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee30159830 0x7fee3014ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee3572a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee300b3d60 con 0x7fee300c4430 2023-10-30T13:10:43.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee3572a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee300b3220 con 0x7fee30153c60 2023-10-30T13:10:43.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee3572a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee300b34c0 con 0x7fee30159830 2023-10-30T13:10:43.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee2e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30153c60 0x7fee301451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee2e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30153c60 0x7fee301451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37776/0 (socket says 172.21.15.148:37776) 2023-10-30T13:10:43.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee2e7fc640 1 -- 172.21.15.148:0/3397345323 learned_addr learned my addr 172.21.15.148:0/3397345323 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:43.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee2f7fe640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee30159830 0x7fee3014ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39301505f0 con 0x7f39300c5550 2023-10-30T13:10:43.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee2e7fc640 1 -- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee30159830 msgr2=0x7fee3014ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee2e7fc640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee30159830 0x7fee3014ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.020+0000 7fee2e7fc640 1 -- 172.21.15.148:0/3397345323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee300c4430 msgr2=0x7fee30144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee2e7fc640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee300c4430 0x7fee30144c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee2e7fc640 1 -- 172.21.15.148:0/3397345323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee28009b30 con 0x7fee30153c60 2023-10-30T13:10:43.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee2f7fe640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee30159830 0x7fee3014ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:43.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee2effd640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee300c4430 0x7fee30144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:43.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee2e7fc640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30153c60 0x7fee301451c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee2000c430 tx=0x7fee2000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee177fe640 1 -- 172.21.15.148:0/3397345323 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200052d0 con 0x7fee30153c60 2023-10-30T13:10:43.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee177fe640 1 -- 172.21.15.148:0/3397345323 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee20005430 con 0x7fee30153c60 2023-10-30T13:10:43.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee177fe640 1 -- 172.21.15.148:0/3397345323 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee20011890 con 0x7fee30153c60 2023-10-30T13:10:43.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee300b37a0 con 0x7fee30153c60 2023-10-30T13:10:43.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee301eb260 con 0x7fee30153c60 2023-10-30T13:10:43.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39300c8610 con 0x7f39300c5550 2023-10-30T13:10:43.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7f39277fe640 1 -- 172.21.15.148:0/1639105932 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39200b8b40 con 0x7f39300c5550 2023-10-30T13:10:43.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.028+0000 7f39277fe640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39100664f0 0x7f39100689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.028+0000 7f39277fe640 1 -- 172.21.15.148:0/1639105932 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f392011e720 con 0x7f39300c5550 2023-10-30T13:10:43.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.028+0000 7f3925d9b640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39100664f0 0x7f39100689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.024+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3004e170 con 0x7fee30153c60 2023-10-30T13:10:43.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.028+0000 7fee177fe640 1 -- 172.21.15.148:0/3397345323 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee200119f0 con 0x7fee30153c60 2023-10-30T13:10:43.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.028+0000 7fee177fe640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee1c0664a0 0x7fee1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:43.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.028+0000 7fee177fe640 1 -- 172.21.15.148:0/3397345323 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee200833a0 con 0x7fee30153c60 2023-10-30T13:10:43.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.032+0000 7fee2effd640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee1c0664a0 0x7fee1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:43.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.032+0000 7f3925d9b640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39100664f0 0x7f39100689b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f39300cd9c0 tx=0x7f3918006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.032+0000 7fee2effd640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee1c0664a0 0x7fee1c068960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fee24009d60 tx=0x7fee24009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:43.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.036+0000 7f39277fe640 1 -- 172.21.15.148:0/1639105932 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39200ebcd0 con 0x7f39300c5550 2023-10-30T13:10:43.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.040+0000 7fee177fe640 1 -- 172.21.15.148:0/3397345323 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee20050830 con 0x7fee30153c60 2023-10-30T13:10:43.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.312+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa484152430 con 0x7fa4840c5d10 2023-10-30T13:10:43.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.312+0000 7fa479ffb640 1 -- 172.21.15.148:0/1132346934 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa480021020 con 0x7fa4840c5d10 2023-10-30T13:10:43.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.312+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f33480b0d00 con 0x7f33480c74e0 2023-10-30T13:10:43.318 INFO:teuthology.orchestra.run.smithi148.stdout:55834574855 2023-10-30T13:10:43.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.312+0000 7f33457fa640 1 -- 172.21.15.148:0/1363214476 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f333c0b5030 con 0x7f33480c74e0 2023-10-30T13:10:43.319 INFO:teuthology.orchestra.run.smithi148.stdout:55834574854 2023-10-30T13:10:43.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.316+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3334066570 msgr2=0x7f3334068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.316+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa468066570 msgr2=0x7fa468068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.316+0000 7f334eb65640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3334066570 0x7f3334068a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3348070cf0 tx=0x7f3338008040 comp rx=0 tx=0).stop 2023-10-30T13:10:43.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.316+0000 7fa48a7b5640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa468066570 0x7fa468068a30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4840c13e0 tx=0x7fa474009040 comp rx=0 tx=0).stop 2023-10-30T13:10:43.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.316+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33480c74e0 msgr2=0x7f3348166000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7f334eb65640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33480c74e0 0x7f3348166000 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f333c01b900 tx=0x7f333c01ba90 comp rx=0 tx=0).stop 2023-10-30T13:10:43.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4840c5d10 msgr2=0x7fa48415d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4840c5d10 0x7fa48415d600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa48000e930 tx=0x7fa48000edf0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 shutdown_connections 2023-10-30T13:10:43.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7f334eb65640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3334066570 0x7f3334068a30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7f334eb65640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3348156350 0x7f334813a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7f334eb65640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f334814fc00 0x7f334813a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7f334eb65640 1 --2- 172.21.15.148:0/1363214476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33480c74e0 0x7f3348166000 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 >> 172.21.15.148:0/1363214476 conn(0x7f33480c7910 msgr2=0x7f334806a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 shutdown_connections 2023-10-30T13:10:43.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 shutdown_connections 2023-10-30T13:10:43.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa468066570 0x7fa468068a30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7f334eb65640 1 -- 172.21.15.148:0/1363214476 wait complete. 2023-10-30T13:10:43.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa48415e540 0x7fa48414da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa484157e00 0x7fa4841f5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 --2- 172.21.15.148:0/1132346934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4840c5d10 0x7fa48415d600 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 >> 172.21.15.148:0/1132346934 conn(0x7fa4840cae10 msgr2=0x7fa48406ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 shutdown_connections 2023-10-30T13:10:43.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.320+0000 7fa48a7b5640 1 -- 172.21.15.148:0/1132346934 wait complete. 2023-10-30T13:10:43.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.332+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f487804e170 con 0x7f4878153c60 2023-10-30T13:10:43.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.332+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7feadc005510 con 0x7feaf806a3d0 2023-10-30T13:10:43.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.336+0000 7feae6ffd640 1 -- 172.21.15.148:0/330111808 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feae80b5030 con 0x7feaf806a3d0 2023-10-30T13:10:43.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.336+0000 7f485f7fe640 1 -- 172.21.15.148:0/181985372 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f486c053cc0 con 0x7f4878153c60 2023-10-30T13:10:43.340 INFO:teuthology.orchestra.run.smithi148.stdout:55834574855 2023-10-30T13:10:43.341 INFO:teuthology.orchestra.run.smithi148.stdout:55834574854 2023-10-30T13:10:43.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.336+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa4dc0caa20 con 0x7fa4dc0c4430 2023-10-30T13:10:43.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.336+0000 7fa4d1ffb640 1 -- 172.21.15.148:0/3148509183 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa4d8021020 con 0x7fa4dc0c4430 2023-10-30T13:10:43.342 INFO:teuthology.orchestra.run.smithi148.stdout:55834574854 2023-10-30T13:10:43.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4864066560 msgr2=0x7f4864068a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4864066560 0x7f4864068a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4868009980 tx=0x7f48680090e0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878153c60 msgr2=0x7f487806f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878153c60 0x7f487806f1c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f486c00c430 tx=0x7f486c00c8f0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 shutdown_connections 2023-10-30T13:10:43.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4864066560 0x7f4864068a20 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4868009980 tx=0x7f48680090e0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4878159830 0x7f4878164680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4878153c60 0x7f487806f1c0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f486c00c430 tx=0x7f486c00c8f0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 --2- 172.21.15.148:0/181985372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48780c4430 0x7f487813ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 >> 172.21.15.148:0/181985372 conn(0x7f487806bfe0 msgr2=0x7f4878152f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fead80664a0 msgr2=0x7fead8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fead80664a0 0x7fead8068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feaf81620f0 tx=0x7feaec008040 comp rx=0 tx=0).stop 2023-10-30T13:10:43.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf806a3d0 msgr2=0x7feaf80c9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf806a3d0 0x7feaf80c9af0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feae801c920 tx=0x7feae801ba90 comp rx=0 tx=0).stop 2023-10-30T13:10:43.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 shutdown_connections 2023-10-30T13:10:43.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f487d3d3640 1 -- 172.21.15.148:0/181985372 wait complete. 2023-10-30T13:10:43.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 shutdown_connections 2023-10-30T13:10:43.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fead80664a0 0x7fead8068960 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feaf8158090 0x7feaf80c0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf8153490 0x7feaf814d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 --2- 172.21.15.148:0/330111808 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feaf806a3d0 0x7feaf80c9af0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feae801c920 tx=0x7feae801ba90 comp rx=0 tx=0).stop 2023-10-30T13:10:43.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 >> 172.21.15.148:0/330111808 conn(0x7feaf80c7910 msgr2=0x7feaf815ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 shutdown_connections 2023-10-30T13:10:43.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7feafce46640 1 -- 172.21.15.148:0/330111808 wait complete. 2023-10-30T13:10:43.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f478404e170 con 0x7f4784158090 2023-10-30T13:10:43.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4c00664a0 msgr2=0x7fa4c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7fa4e3972640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4c00664a0 0x7fa4c0068960 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc158be0 tx=0x7fa4cc008040 comp rx=0 tx=0).stop 2023-10-30T13:10:43.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4dc0c4430 msgr2=0x7fa4dc0b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.340+0000 7fa4e3972640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4dc0c4430 0x7fa4dc0b50c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800e960 tx=0x7fa4d800ee20 comp rx=0 tx=0).stop 2023-10-30T13:10:43.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.344+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 shutdown_connections 2023-10-30T13:10:43.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.344+0000 7fa4e3972640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4c00664a0 0x7fa4c0068960 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.344+0000 7fa4e3972640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4dc159830 0x7fa4dc0cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.344+0000 7fa4e3972640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4dc153c60 0x7fa4dc142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.344+0000 7fa4e3972640 1 --2- 172.21.15.148:0/3148509183 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4dc0c4430 0x7fa4dc0b50c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.344+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 >> 172.21.15.148:0/3148509183 conn(0x7fa4dc06bfe0 msgr2=0x7fa4dc0c5110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.344+0000 7f4780ff9640 1 -- 172.21.15.148:0/2653343588 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f47740f2110 con 0x7f4784158090 2023-10-30T13:10:43.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.344+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 shutdown_connections 2023-10-30T13:10:43.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.344+0000 7fa4e3972640 1 -- 172.21.15.148:0/3148509183 wait complete. 2023-10-30T13:10:43.353 INFO:teuthology.orchestra.run.smithi148.stdout:55834574855 2023-10-30T13:10:43.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47640664f0 msgr2=0x7f47640689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47640664f0 0x7f47640689b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f477c0094c0 tx=0x7f477c00b040 comp rx=0 tx=0).stop 2023-10-30T13:10:43.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4784158090 msgr2=0x7f47840c4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4784158090 0x7f47840c4890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f47740af000 tx=0x7f477401b9d0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 shutdown_connections 2023-10-30T13:10:43.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47640664f0 0x7f47640689b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4784158090 0x7f47840c4890 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4784149490 0x7f47840c2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 --2- 172.21.15.148:0/2653343588 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f478406a3d0 0x7f4784147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 >> 172.21.15.148:0/2653343588 conn(0x7f47840c7910 msgr2=0x7f478415c810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 shutdown_connections 2023-10-30T13:10:43.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.352+0000 7f478a809640 1 -- 172.21.15.148:0/2653343588 wait complete. 2023-10-30T13:10:43.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.0 2023-10-30T13:10:43.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.1 2023-10-30T13:10:43.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.360+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fee300caa20 con 0x7fee30153c60 2023-10-30T13:10:43.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee177fe640 1 -- 172.21.15.148:0/3397345323 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fee20017020 con 0x7fee30153c60 2023-10-30T13:10:43.367 INFO:teuthology.orchestra.run.smithi148.stdout:55834574855 2023-10-30T13:10:43.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f393004e170 con 0x7f39300c5550 2023-10-30T13:10:43.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7f39277fe640 1 -- 172.21.15.148:0/1639105932 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f39200bc090 con 0x7f39300c5550 2023-10-30T13:10:43.368 INFO:teuthology.orchestra.run.smithi148.stdout:55834574854 2023-10-30T13:10:43.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee1c0664a0 msgr2=0x7fee1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee1c0664a0 0x7fee1c068960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fee24009d60 tx=0x7fee24009110 comp rx=0 tx=0).stop 2023-10-30T13:10:43.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30153c60 msgr2=0x7fee301451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30153c60 0x7fee301451c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee2000c430 tx=0x7fee2000c8f0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 shutdown_connections 2023-10-30T13:10:43.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee1c0664a0 0x7fee1c068960 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee30159830 0x7fee3014ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee30153c60 0x7fee301451c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 --2- 172.21.15.148:0/3397345323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee300c4430 0x7fee30144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 >> 172.21.15.148:0/3397345323 conn(0x7fee3006bfe0 msgr2=0x7fee3006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 shutdown_connections 2023-10-30T13:10:43.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.364+0000 7fee3572a640 1 -- 172.21.15.148:0/3397345323 wait complete. 2023-10-30T13:10:43.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.368+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39100664f0 msgr2=0x7f39100689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.368+0000 7f392f59e640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39100664f0 0x7f39100689b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f39300cd9c0 tx=0x7f3918006f90 comp rx=0 tx=0).stop 2023-10-30T13:10:43.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.368+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 msgr2=0x7f393015cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:43.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.368+0000 7f392f59e640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 0x7f393015cb40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f39200ab000 tx=0x7f3920019770 comp rx=0 tx=0).stop 2023-10-30T13:10:43.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.372+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 shutdown_connections 2023-10-30T13:10:43.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.372+0000 7f392f59e640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39100664f0 0x7f39100689b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.372+0000 7f392f59e640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39300cb050 0x7f39301f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.372+0000 7f392f59e640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39300cab00 0x7f39301f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.372+0000 7f392f59e640 1 --2- 172.21.15.148:0/1639105932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39300c5550 0x7f393015cb40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:43.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.372+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 >> 172.21.15.148:0/1639105932 conn(0x7f393006bed0 msgr2=0x7f39300c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:43.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.372+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 shutdown_connections 2023-10-30T13:10:43.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:43.372+0000 7f392f59e640 1 -- 172.21.15.148:0/1639105932 wait complete. 2023-10-30T13:10:43.381 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.3 2023-10-30T13:10:43.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.2 2023-10-30T13:10:43.385 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.5 2023-10-30T13:10:43.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.4 2023-10-30T13:10:43.395 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.7 2023-10-30T13:10:43.401 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.6 2023-10-30T13:10:44.363 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T13:10:44.364 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T13:10:44.382 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T13:10:44.386 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T13:10:44.386 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T13:10:44.394 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T13:10:44.397 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T13:10:44.402 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T13:10:44.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.588+0000 7f231fd2a640 1 Processor -- start 2023-10-30T13:10:44.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.588+0000 7f231fd2a640 1 -- start start 2023-10-30T13:10:44.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.588+0000 7fddbdc17640 1 Processor -- start 2023-10-30T13:10:44.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.588+0000 7fddbdc17640 1 -- start start 2023-10-30T13:10:44.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7f231fd2a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2318156cb0 0x7f231813fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7f231fd2a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2318072030 con 0x7f23180c08d0 2023-10-30T13:10:44.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7f231fd2a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2318070010 con 0x7f231813f600 2023-10-30T13:10:44.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7f231fd2a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231813ffa0 con 0x7f2318156cb0 2023-10-30T13:10:44.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddbdc17640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fddb8153c60 0x7fddb80ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddbdc17640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fddb80c4430 0x7fddb80cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddbdc17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb8159830 0x7fddb8159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddbdc17640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb8154040 con 0x7fddb80c4430 2023-10-30T13:10:44.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddbdc17640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb80cb780 con 0x7fddb8153c60 2023-10-30T13:10:44.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddbdc17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb80cc9f0 con 0x7fddb8159830 2023-10-30T13:10:44.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddb7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb8159830 0x7fddb8159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddb77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fddb8153c60 0x7fddb80ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddb7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb8159830 0x7fddb8159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45728/0 (socket says 172.21.15.148:45728) 2023-10-30T13:10:44.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7f231d2c5640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f231813f600 0x7f23181565a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:59558/0 (socket says 172.21.15.148:59558) 2023-10-30T13:10:44.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7f231d2c5640 1 -- 172.21.15.148:0/1369830023 learned_addr learned my addr 172.21.15.148:0/1369830023 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:44.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddb7fff640 1 -- 172.21.15.148:0/782707259 learned_addr learned my addr 172.21.15.148:0/782707259 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7f231e2c7640 1 --2- 172.21.15.148:0/1369830023 >> v2:172.21.15.148:3300/0 conn(0x7f2318156cb0 0x7f231813fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddb7fff640 1 -- 172.21.15.148:0/782707259 >> v2:172.21.15.150:3300/0 conn(0x7fddb8153c60 msgr2=0x7fddb80ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddb7fff640 1 --2- 172.21.15.148:0/782707259 >> v2:172.21.15.150:3300/0 conn(0x7fddb8153c60 0x7fddb80ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddb7fff640 1 -- 172.21.15.148:0/782707259 >> v2:172.21.15.148:3300/0 conn(0x7fddb80c4430 msgr2=0x7fddb80cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddb7fff640 1 --2- 172.21.15.148:0/782707259 >> v2:172.21.15.148:3300/0 conn(0x7fddb80c4430 0x7fddb80cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.592+0000 7fddb7fff640 1 -- 172.21.15.148:0/782707259 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb815a2e0 con 0x7fddb8159830 2023-10-30T13:10:44.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166327095 0 0) 0x7f2318070010 con 0x7f231813f600 2023-10-30T13:10:44.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304003530 con 0x7f231813f600 2023-10-30T13:10:44.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173022922 0 0) 0x7f2318072030 con 0x7f23180c08d0 2023-10-30T13:10:44.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304003850 con 0x7f23180c08d0 2023-10-30T13:10:44.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3700045428 0 0) 0x7f2304003530 con 0x7f231813f600 2023-10-30T13:10:44.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 >> v2:172.21.15.148:3300/0 conn(0x7f2318156cb0 msgr2=0x7f231813fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 --2- 172.21.15.148:0/1369830023 >> v2:172.21.15.148:3300/0 conn(0x7f2318156cb0 0x7f231813fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 >> v1:172.21.15.148:6789/0 conn(0x7f23180c08d0 legacy=0x7f2318071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231815ce80 con 0x7f231813f600 2023-10-30T13:10:44.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f2308001660 con 0x7f231813f600 2023-10-30T13:10:44.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 >> v1:172.21.15.150:6789/0 conn(0x7f231813f600 legacy=0x7f23181565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2304003cc0 0x7f23040040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23040047d0 0x7f2304004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2304005350 0x7f230400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231813ffa0 con 0x7f2304003cc0 2023-10-30T13:10:44.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230400c270 con 0x7f23040047d0 2023-10-30T13:10:44.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230400c430 con 0x7f2304005350 2023-10-30T13:10:44.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231e2c7640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2304005350 0x7f230400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.596+0000 7f231dac6640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2304003cc0 0x7f23040040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.600+0000 7f231e2c7640 1 -- 172.21.15.148:0/1369830023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23040047d0 msgr2=0x7f2304004c50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:10:44.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.600+0000 7f231e2c7640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23040047d0 0x7f2304004c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.600+0000 7f231e2c7640 1 -- 172.21.15.148:0/1369830023 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2304003cc0 msgr2=0x7f23040040c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.600+0000 7f231e2c7640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2304003cc0 0x7f23040040c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.600+0000 7f231e2c7640 1 -- 172.21.15.148:0/1369830023 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2304003530 con 0x7f2304005350 2023-10-30T13:10:44.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.600+0000 7fddb7fff640 1 --2- 172.21.15.148:0/782707259 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb8159830 0x7fddb8159ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fddb8072ce0 tx=0x7fddac02d530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55e2c5e65ed386d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.600+0000 7fddb67fc640 1 -- 172.21.15.148:0/782707259 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0bb650 con 0x7fddb8159830 2023-10-30T13:10:44.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.600+0000 7fddb67fc640 1 -- 172.21.15.148:0/782707259 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac0bb7b0 con 0x7fddb8159830 2023-10-30T13:10:44.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.600+0000 7fddb67fc640 1 -- 172.21.15.148:0/782707259 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0bbb40 con 0x7fddb8159830 2023-10-30T13:10:44.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231e2c7640 1 -- 172.21.15.148:0/1369830023 reap_dead start 2023-10-30T13:10:44.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 -- 172.21.15.148:0/782707259 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb8159830 msgr2=0x7fddb8159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 --2- 172.21.15.148:0/782707259 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb8159830 0x7fddb8159ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fddb8072ce0 tx=0x7fddac02d530 comp rx=0 tx=0).stop 2023-10-30T13:10:44.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 -- 172.21.15.148:0/782707259 shutdown_connections 2023-10-30T13:10:44.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 --2- 172.21.15.148:0/782707259 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb8159830 0x7fddb8159ce0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231e2c7640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2304005350 0x7f230400b990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2310007360 tx=0x7f231001ce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bde9837d63dfc9c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 --2- 172.21.15.148:0/782707259 >> v2:172.21.15.148:3300/0 conn(0x7fddb80c4430 0x7fddb80cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 --2- 172.21.15.148:0/782707259 >> v2:172.21.15.150:3300/0 conn(0x7fddb8153c60 0x7fddb80ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 -- 172.21.15.148:0/782707259 >> 172.21.15.148:0/782707259 conn(0x7fddb806bfe0 msgr2=0x7fddb806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 -- 172.21.15.148:0/782707259 shutdown_connections 2023-10-30T13:10:44.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 -- 172.21.15.148:0/782707259 wait complete. 2023-10-30T13:10:44.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7fddbdc17640 1 Processor -- start 2023-10-30T13:10:44.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23100a5d10 con 0x7f2304005350 2023-10-30T13:10:44.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23100a5e70 con 0x7f2304005350 2023-10-30T13:10:44.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231cac4640 1 -- 172.21.15.148:0/1369830023 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310003bc0 con 0x7f2304005350 2023-10-30T13:10:44.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231fd2a640 1 -- 172.21.15.148:0/1369830023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2304005350 msgr2=0x7f230400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231fd2a640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2304005350 0x7f230400b990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2310007360 tx=0x7f231001ce50 comp rx=0 tx=0).stop 2023-10-30T13:10:44.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231fd2a640 1 -- 172.21.15.148:0/1369830023 shutdown_connections 2023-10-30T13:10:44.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231fd2a640 1 --2- 172.21.15.148:0/1369830023 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2304005350 0x7f230400b990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231fd2a640 1 -- 172.21.15.148:0/1369830023 >> 172.21.15.148:0/1369830023 conn(0x7f23180c7910 msgr2=0x7f23180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231fd2a640 1 -- 172.21.15.148:0/1369830023 shutdown_connections 2023-10-30T13:10:44.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.604+0000 7f231fd2a640 1 -- 172.21.15.148:0/1369830023 wait complete. 2023-10-30T13:10:44.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddbdc17640 1 -- start start 2023-10-30T13:10:44.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddbdc17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb80c4430 0x7fddb80b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddbdc17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddb8153c60 0x7fddb8146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddbdc17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fddb8159830 0x7fddb8146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddbdc17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb814f790 con 0x7fddb8159830 2023-10-30T13:10:44.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddbdc17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb80b5080 con 0x7fddb8153c60 2023-10-30T13:10:44.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddbdc17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb80705d0 con 0x7fddb80c4430 2023-10-30T13:10:44.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddb7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fddb8159830 0x7fddb8146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddb77fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb80c4430 0x7fddb80b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddb77fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb80c4430 0x7fddb80b3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45752/0 (socket says 172.21.15.148:45752) 2023-10-30T13:10:44.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7fddb77fe640 1 -- 172.21.15.148:0/893449920 learned_addr learned my addr 172.21.15.148:0/893449920 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7f231fd2a640 1 Processor -- start 2023-10-30T13:10:44.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.608+0000 7f231fd2a640 1 -- start start 2023-10-30T13:10:44.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231fd2a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2304005350 0x7f231813ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231fd2a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2318155210 0x7f23181357b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231fd2a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23181541b0 0x7f2318135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231fd2a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2318155bf0 con 0x7f2304005350 2023-10-30T13:10:44.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231fd2a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2318158dd0 con 0x7f23181541b0 2023-10-30T13:10:44.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231fd2a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2318158110 con 0x7f2318155210 2023-10-30T13:10:44.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7fddb6ffd640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddb8153c60 0x7fddb8146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7fddb7fff640 1 -- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb80c4430 msgr2=0x7fddb80b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7fddb7fff640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb80c4430 0x7fddb80b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7fddb7fff640 1 -- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddb8153c60 msgr2=0x7fddb8146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7fddb7fff640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddb8153c60 0x7fddb8146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7fddb7fff640 1 -- 172.21.15.148:0/893449920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac0094d0 con 0x7fddb8159830 2023-10-30T13:10:44.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231e2c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23181541b0 0x7f2318135cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231e2c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23181541b0 0x7f2318135cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37808/0 (socket says 172.21.15.148:37808) 2023-10-30T13:10:44.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231e2c7640 1 -- 172.21.15.148:0/794856997 learned_addr learned my addr 172.21.15.148:0/794856997 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231d2c5640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2318155210 0x7f23181357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231e2c7640 1 -- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2318155210 msgr2=0x7f23181357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231e2c7640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2318155210 0x7f23181357b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231e2c7640 1 -- 172.21.15.148:0/794856997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2304005350 msgr2=0x7f231813ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231e2c7640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2304005350 0x7f231813ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231e2c7640 1 -- 172.21.15.148:0/794856997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2318155e40 con 0x7f23181541b0 2023-10-30T13:10:44.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7fddb7fff640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fddb8159830 0x7fddb8146f70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fddb80b1950 tx=0x7fddac02a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f231e2c7640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23181541b0 0x7f2318135cf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f231001cc80 tx=0x7f2310019bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7fdd9ffff640 1 -- 172.21.15.148:0/893449920 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0c5810 con 0x7fddb8159830 2023-10-30T13:10:44.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7fdd9ffff640 1 -- 172.21.15.148:0/893449920 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac0c5970 con 0x7fddb8159830 2023-10-30T13:10:44.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.616+0000 7fdd9ffff640 1 -- 172.21.15.148:0/893449920 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0c5810 con 0x7fddb8159830 2023-10-30T13:10:44.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.616+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb814f1d0 con 0x7fddb8159830 2023-10-30T13:10:44.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.612+0000 7f230e7fc640 1 -- 172.21.15.148:0/794856997 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310003ea0 con 0x7f23181541b0 2023-10-30T13:10:44.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.616+0000 7f230e7fc640 1 -- 172.21.15.148:0/794856997 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2310004000 con 0x7f23181541b0 2023-10-30T13:10:44.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.616+0000 7f230e7fc640 1 -- 172.21.15.148:0/794856997 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23100043b0 con 0x7f23181541b0 2023-10-30T13:10:44.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.616+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2318156120 con 0x7f23181541b0 2023-10-30T13:10:44.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.616+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2318159300 con 0x7f23181541b0 2023-10-30T13:10:44.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.616+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb8149960 con 0x7fddb8159830 2023-10-30T13:10:44.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.616+0000 7fdd9ffff640 1 -- 172.21.15.148:0/893449920 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fddac0c5ae0 con 0x7fddb8159830 2023-10-30T13:10:44.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7fdd9ffff640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdda4066500 0x7fdda40689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7fdd9ffff640 1 -- 172.21.15.148:0/893449920 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddac12f080 con 0x7fddb8159830 2023-10-30T13:10:44.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7f230e7fc640 1 -- 172.21.15.148:0/794856997 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2310004510 con 0x7f23181541b0 2023-10-30T13:10:44.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2318156400 con 0x7f23181541b0 2023-10-30T13:10:44.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7f230e7fc640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22fc0664d0 0x7f22fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7f230e7fc640 1 -- 172.21.15.148:0/794856997 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f231011ddb0 con 0x7f23181541b0 2023-10-30T13:10:44.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7fddb77fe640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdda4066500 0x7fdda40689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7f231dac6640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22fc0664d0 0x7f22fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb804e170 con 0x7fddb8159830 2023-10-30T13:10:44.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7fddb77fe640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdda4066500 0x7fdda40689c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009a40 tx=0x7fddb000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.620+0000 7f231dac6640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22fc0664d0 0x7f22fc068990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f23140016c0 tx=0x7f2314009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.632+0000 7fdd9ffff640 1 -- 172.21.15.148:0/893449920 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fddac0fc640 con 0x7fddb8159830 2023-10-30T13:10:44.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.640+0000 7f230e7fc640 1 -- 172.21.15.148:0/794856997 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2318156400 con 0x7f23181541b0 2023-10-30T13:10:44.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.644+0000 7fe29f165640 1 Processor -- start 2023-10-30T13:10:44.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.644+0000 7fe29f165640 1 -- start start 2023-10-30T13:10:44.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29f165640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe2980cad80 0x7fe2980cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29f165640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe2980cf160 0x7fe2980cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29f165640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe298158350 con 0x7fe298157f70 2023-10-30T13:10:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29f165640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2980ca8b0 con 0x7fe2980cad80 2023-10-30T13:10:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29f165640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2980ca2f0 con 0x7fe2980cf160 2023-10-30T13:10:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29d702640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe2980cf160 0x7fe2980cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29d702640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe2980cf160 0x7fe2980cf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37814/0 (socket says 172.21.15.148:37814) 2023-10-30T13:10:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29d702640 1 -- 172.21.15.148:0/3867496203 learned_addr learned my addr 172.21.15.148:0/3867496203 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29d702640 1 -- 172.21.15.148:0/3867496203 >> v2:172.21.15.148:3300/0 conn(0x7fe2980cad80 msgr2=0x7fe2980cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29d702640 1 --2- 172.21.15.148:0/3867496203 >> v2:172.21.15.148:3300/0 conn(0x7fe2980cad80 0x7fe2980cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29d702640 1 -- 172.21.15.148:0/3867496203 >> v1:172.21.15.148:6789/0 conn(0x7fe298157f70 legacy=0x7fe298161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.648+0000 7fe29d702640 1 -- 172.21.15.148:0/3867496203 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2980ca5d0 con 0x7fe2980cf160 2023-10-30T13:10:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe29d702640 1 --2- 172.21.15.148:0/3867496203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2980cf160 0x7fe2980cf5d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe2980cacd0 tx=0x7fe28801ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c69a029b37dd6e02 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7ff4eb59e640 1 Processor -- start 2023-10-30T13:10:44.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe2977fe640 1 -- 172.21.15.148:0/3867496203 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2880a89b0 con 0x7fe2980cf160 2023-10-30T13:10:44.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe2977fe640 1 -- 172.21.15.148:0/3867496203 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2880a8b10 con 0x7fe2980cf160 2023-10-30T13:10:44.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe29f165640 1 -- 172.21.15.148:0/3867496203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2980cf160 msgr2=0x7fe2980cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe29f165640 1 --2- 172.21.15.148:0/3867496203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2980cf160 0x7fe2980cf5d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe2980cacd0 tx=0x7fe28801ecd0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe29f165640 1 -- 172.21.15.148:0/3867496203 shutdown_connections 2023-10-30T13:10:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe29f165640 1 --2- 172.21.15.148:0/3867496203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2980cf160 0x7fe2980cf5d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe29f165640 1 --2- 172.21.15.148:0/3867496203 >> v2:172.21.15.148:3300/0 conn(0x7fe2980cad80 0x7fe2980cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe29f165640 1 -- 172.21.15.148:0/3867496203 >> 172.21.15.148:0/3867496203 conn(0x7fe29806bfe0 msgr2=0x7fe29806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7ff4eb59e640 1 -- start start 2023-10-30T13:10:44.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe29f165640 1 -- 172.21.15.148:0/3867496203 shutdown_connections 2023-10-30T13:10:44.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7fe29f165640 1 -- 172.21.15.148:0/3867496203 wait complete. 2023-10-30T13:10:44.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7ff4eb59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff4ec152eb0 0x7ff4ec0c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.656+0000 7ff4eb59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff4ec0700c0 0x7ff4ec0713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4eb59e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ec156670 con 0x7ff4ec155720 2023-10-30T13:10:44.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4eb59e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ec1560b0 con 0x7ff4ec152eb0 2023-10-30T13:10:44.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7fe29f165640 1 Processor -- start 2023-10-30T13:10:44.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4eb59e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ec156390 con 0x7ff4ec0700c0 2023-10-30T13:10:44.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7fe29f165640 1 -- start start 2023-10-30T13:10:44.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4ea59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff4ec152eb0 0x7ff4ec0c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4ea59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff4ec152eb0 0x7ff4ec0c2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41992/0 (socket says 172.21.15.148:41992) 2023-10-30T13:10:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4ead9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff4ec155720 0x7ff4ec069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39370/0 (socket says 172.21.15.148:39370) 2023-10-30T13:10:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4ead9d640 1 -- 172.21.15.148:0/3227429719 learned_addr learned my addr 172.21.15.148:0/3227429719 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:44.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4ea59c640 1 -- 172.21.15.148:0/3227429719 >> v2:172.21.15.150:3300/0 conn(0x7ff4ec0700c0 msgr2=0x7ff4ec0713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7fe29f165640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2980cad80 0x7fe2981525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7fe29f165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2980cf160 0x7fe2980b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe29f165640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe298157f70 0x7fe2980b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe297fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2980cf160 0x7fe2980b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe297fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2980cf160 0x7fe2980b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45772/0 (socket says 172.21.15.148:45772) 2023-10-30T13:10:44.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe29cf01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2980cad80 0x7fe2981525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe29cf01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2980cad80 0x7fe2981525e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42000/0 (socket says 172.21.15.148:42000) 2023-10-30T13:10:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe297fff640 1 -- 172.21.15.148:0/1049300821 learned_addr learned my addr 172.21.15.148:0/1049300821 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980c6560 con 0x7fe2980cad80 2023-10-30T13:10:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980cd050 con 0x7fe298157f70 2023-10-30T13:10:44.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29813e040 con 0x7fe2980cf160 2023-10-30T13:10:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe29d702640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe298157f70 0x7fe2980b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe297fff640 1 -- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe298157f70 msgr2=0x7fe2980b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe297fff640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe298157f70 0x7fe2980b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe297fff640 1 -- 172.21.15.148:0/1049300821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2980cad80 msgr2=0x7fe2981525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe297fff640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2980cad80 0x7fe2981525e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe297fff640 1 -- 172.21.15.148:0/1049300821 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2880094d0 con 0x7fe2980cf160 2023-10-30T13:10:44.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe297fff640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2980cf160 0x7fe2980b4260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe29000ba50 tx=0x7fe29000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4e9d9b640 1 -- 172.21.15.148:0/3227429719 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570474695 0 0) 0x7ff4ec156670 con 0x7ff4ec155720 2023-10-30T13:10:44.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4ea59c640 1 --2- 172.21.15.148:0/3227429719 >> v2:172.21.15.150:3300/0 conn(0x7ff4ec0700c0 0x7ff4ec0713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4ea59c640 1 -- 172.21.15.148:0/3227429719 >> v1:172.21.15.148:6789/0 conn(0x7ff4ec155720 legacy=0x7ff4ec069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.660+0000 7ff4ea59c640 1 -- 172.21.15.148:0/3227429719 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec0708c0 con 0x7ff4ec152eb0 2023-10-30T13:10:44.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7f9f4f31d640 1 Processor -- start 2023-10-30T13:10:44.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7f9f4f31d640 1 -- start start 2023-10-30T13:10:44.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7f78cf59e640 1 Processor -- start 2023-10-30T13:10:44.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7fe2957fa640 1 -- 172.21.15.148:0/1049300821 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29000cbe0 con 0x7fe2980cf160 2023-10-30T13:10:44.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.664+0000 7f78cf59e640 1 -- start start 2023-10-30T13:10:44.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7fe2957fa640 1 -- 172.21.15.148:0/1049300821 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29000cd60 con 0x7fe2980cf160 2023-10-30T13:10:44.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e35079640 1 Processor -- start 2023-10-30T13:10:44.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7fe2957fa640 1 -- 172.21.15.148:0/1049300821 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290014680 con 0x7fe2980cf160 2023-10-30T13:10:44.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2980cd2f0 con 0x7fe2980cf160 2023-10-30T13:10:44.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e35079640 1 -- start start 2023-10-30T13:10:44.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4c8b8640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9f480c4430 0x7f9f48159830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39384/0 (socket says 172.21.15.148:39384) 2023-10-30T13:10:44.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4c8b8640 1 -- 172.21.15.148:0/2902181568 learned_addr learned my addr 172.21.15.148:0/2902181568 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:44.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4f31d640 1 --2- 172.21.15.148:0/2902181568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f480cac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2980cd8b0 con 0x7fe2980cf160 2023-10-30T13:10:44.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f78cf59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78d0157c80 0x7f78d0161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f78cf59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f78d006fef0 0x7f78d006f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f78cf59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 0x7f78d006f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f78cf59e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d015ef30 con 0x7f78d006fef0 2023-10-30T13:10:44.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f78cf59e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d0160e50 con 0x7f78d0157c80 2023-10-30T13:10:44.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f78cf59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0161f60 con 0x7f78d006ab20 2023-10-30T13:10:44.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4d8ba640 1 --2- 172.21.15.148:0/2902181568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f480cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2902181568 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f480cb780 con 0x7f9f480c4430 2023-10-30T13:10:44.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2902181568 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f480ca810 con 0x7f9f48153c60 2023-10-30T13:10:44.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2902181568 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f48154040 con 0x7f9f480ca360 2023-10-30T13:10:44.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e35079640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e300c4430 0x7f2e300ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e35079640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e30159830 0x7f2e30159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e35079640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e300cb780 con 0x7f2e30153c60 2023-10-30T13:10:44.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e35079640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e30154040 con 0x7f2e30159830 2023-10-30T13:10:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e35079640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3015a2e0 con 0x7f2e300c4430 2023-10-30T13:10:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e2e59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e300c4430 0x7f2e300ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e2e59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e300c4430 0x7f2e300ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45796/0 (socket says 172.21.15.148:45796) 2023-10-30T13:10:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e2e59c640 1 -- 172.21.15.148:0/556451071 learned_addr learned my addr 172.21.15.148:0/556451071 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e2f59e640 1 --2- 172.21.15.148:0/556451071 >> v2:172.21.15.150:3300/0 conn(0x7f2e30159830 0x7f2e30159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e2e59c640 1 -- 172.21.15.148:0/556451071 >> v2:172.21.15.150:3300/0 conn(0x7f2e30159830 msgr2=0x7f2e30159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e2e59c640 1 --2- 172.21.15.148:0/556451071 >> v2:172.21.15.150:3300/0 conn(0x7f2e30159830 0x7f2e30159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e2e59c640 1 -- 172.21.15.148:0/556451071 >> v1:172.21.15.148:6789/0 conn(0x7f2e30153c60 legacy=0x7f2e300cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f2e2e59c640 1 -- 172.21.15.148:0/556451071 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3015a5c0 con 0x7f2e300c4430 2023-10-30T13:10:44.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f78cdd9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f78d006fef0 0x7f78d006f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4d8ba640 1 -- 172.21.15.148:0/2902181568 >> v1:172.21.15.150:6789/0 conn(0x7f9f48153c60 legacy=0x7f9f480cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4d8ba640 1 -- 172.21.15.148:0/2902181568 >> v1:172.21.15.148:6789/0 conn(0x7f9f480c4430 legacy=0x7f9f48159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7f9f4d8ba640 1 -- 172.21.15.148:0/2902181568 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4815a2e0 con 0x7f9f480ca360 2023-10-30T13:10:44.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7f78ce59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78d0157c80 0x7f78d0161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7f78ced9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 0x7f78d006f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7f78cdd9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f78d006fef0 0x7f78d006f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42006/0 (socket says 172.21.15.148:42006) 2023-10-30T13:10:44.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7f78cdd9b640 1 -- 172.21.15.148:0/3861705586 learned_addr learned my addr 172.21.15.148:0/3861705586 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.668+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29804e170 con 0x7fe2980cf160 2023-10-30T13:10:44.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4ea59c640 1 --2- 172.21.15.148:0/3227429719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 0x7ff4ec0c2890 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0668d0 tx=0x7ff4dc01e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac8ca83bdafe989 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7f78cdd9b640 1 -- 172.21.15.148:0/3861705586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 msgr2=0x7f78d006f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7f78cdd9b640 1 --2- 172.21.15.148:0/3861705586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 0x7f78d006f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7f78cdd9b640 1 -- 172.21.15.148:0/3861705586 >> v2:172.21.15.150:3300/0 conn(0x7f78d0157c80 msgr2=0x7f78d0161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7f78cdd9b640 1 --2- 172.21.15.148:0/3861705586 >> v2:172.21.15.150:3300/0 conn(0x7f78d0157c80 0x7f78d0161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7f78cdd9b640 1 -- 172.21.15.148:0/3861705586 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d01616c0 con 0x7f78d006fef0 2023-10-30T13:10:44.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7fe2957fa640 1 -- 172.21.15.148:0/1049300821 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe29001d3b0 con 0x7fe2980cf160 2023-10-30T13:10:44.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4e9d9b640 1 -- 172.21.15.148:0/3227429719 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc004230 con 0x7ff4ec152eb0 2023-10-30T13:10:44.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4e9d9b640 1 -- 172.21.15.148:0/3227429719 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc0ab070 con 0x7ff4ec152eb0 2023-10-30T13:10:44.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4e9d9b640 1 -- 172.21.15.148:0/3227429719 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0a7790 con 0x7ff4ec152eb0 2023-10-30T13:10:44.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7fe2957fa640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2840664f0 0x7fe2840689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3227429719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 msgr2=0x7ff4ec0c2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4eb59e640 1 --2- 172.21.15.148:0/3227429719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 0x7ff4ec0c2890 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec0668d0 tx=0x7ff4dc01e8e0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7fe2957fa640 1 -- 172.21.15.148:0/1049300821 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe290083c60 con 0x7fe2980cf160 2023-10-30T13:10:44.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7fe29cf01640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2840664f0 0x7fe2840689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3227429719 shutdown_connections 2023-10-30T13:10:44.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4eb59e640 1 --2- 172.21.15.148:0/3227429719 >> v2:172.21.15.150:3300/0 conn(0x7ff4ec0700c0 0x7ff4ec0713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4eb59e640 1 --2- 172.21.15.148:0/3227429719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 0x7ff4ec0c2890 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.672+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3227429719 >> 172.21.15.148:0/3227429719 conn(0x7ff4ec0c75d0 msgr2=0x7ff4ec0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3227429719 shutdown_connections 2023-10-30T13:10:44.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3227429719 wait complete. 2023-10-30T13:10:44.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f2e2e59c640 1 --2- 172.21.15.148:0/556451071 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e300c4430 0x7f2e300ca360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009e50 tx=0x7f2e2801c9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c7e06d399ee8f4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f9f4d8ba640 1 --2- 172.21.15.148:0/2902181568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f480cac40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009dc0 tx=0x7f9f4001c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64efa63ccf58b2c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7fe29cf01640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2840664f0 0x7fe2840689b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0098e0 tx=0x7fe28c009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70658dc640 1 Processor -- start 2023-10-30T13:10:44.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70658dc640 1 -- start start 2023-10-30T13:10:44.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7ff4eb59e640 1 Processor -- start 2023-10-30T13:10:44.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7ff4eb59e640 1 -- start start 2023-10-30T13:10:44.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f78cdd9b640 1 --2- 172.21.15.148:0/3861705586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d006fef0 0x7f78d006f340 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f78d00c86f0 tx=0x7f78c801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda578aafe973226 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70658dc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f705800e210 0x7f70580dca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70658dc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7058033710 0x7f70580dcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70658dc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580d6ef0 con 0x7f70580d6b10 2023-10-30T13:10:44.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70658dc640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705804a110 con 0x7f705800e210 2023-10-30T13:10:44.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70658dc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058033cf0 con 0x7f7058033710 2023-10-30T13:10:44.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7ff4eb59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec0700c0 0x7ff4ec1428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7ff4eb59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 0x7ff4ec0c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7ff4eb59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec155720 0x7ff4ec1f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7ff4eb59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec148af0 con 0x7ff4ec152eb0 2023-10-30T13:10:44.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70648da640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f70580d6b10 0x7f705804ab50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:59570/0 (socket says 172.21.15.148:59570) 2023-10-30T13:10:44.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70648da640 1 -- 172.21.15.148:0/2345665948 learned_addr learned my addr 172.21.15.148:0/2345665948 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:44.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f70650db640 1 --2- 172.21.15.148:0/2345665948 >> v2:172.21.15.150:3300/0 conn(0x7f7058033710 0x7f70580dcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.676+0000 7f705ffff640 1 --2- 172.21.15.148:0/2345665948 >> v2:172.21.15.148:3300/0 conn(0x7f705800e210 0x7f70580dca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4eb59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec0c5910 con 0x7ff4ec155720 2023-10-30T13:10:44.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4eb59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec14d7c0 con 0x7ff4ec0700c0 2023-10-30T13:10:44.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cd59a640 1 -- 172.21.15.148:0/3861705586 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c80af400 con 0x7f78d006fef0 2023-10-30T13:10:44.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cd59a640 1 -- 172.21.15.148:0/3861705586 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c80ab050 con 0x7f78d006fef0 2023-10-30T13:10:44.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cd59a640 1 -- 172.21.15.148:0/3861705586 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c80af780 con 0x7f78d006fef0 2023-10-30T13:10:44.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f705ffff640 1 -- 172.21.15.148:0/2345665948 >> v2:172.21.15.150:3300/0 conn(0x7f7058033710 msgr2=0x7f70580dcf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:44.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f705ffff640 1 --2- 172.21.15.148:0/2345665948 >> v2:172.21.15.150:3300/0 conn(0x7f7058033710 0x7f70580dcf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f705ffff640 1 -- 172.21.15.148:0/2345665948 >> v1:172.21.15.150:6789/0 conn(0x7f70580d6b10 legacy=0x7f705804ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f705ffff640 1 -- 172.21.15.148:0/2345665948 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70580dd4d0 con 0x7f705800e210 2023-10-30T13:10:44.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4ead9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec155720 0x7ff4ec1f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 0x7ff4ec0c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4ead9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec155720 0x7ff4ec1f4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37886/0 (socket says 172.21.15.148:37886) 2023-10-30T13:10:44.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4ead9d640 1 -- 172.21.15.148:0/3574270206 learned_addr learned my addr 172.21.15.148:0/3574270206 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4ea59c640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec0700c0 0x7ff4ec1428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e3fff640 1 -- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec0700c0 msgr2=0x7ff4ec1428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e3fff640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec0700c0 0x7ff4ec1428f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e3fff640 1 -- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec155720 msgr2=0x7ff4ec1f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e3fff640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec155720 0x7ff4ec1f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e3fff640 1 -- 172.21.15.148:0/3574270206 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc00b650 con 0x7ff4ec152eb0 2023-10-30T13:10:44.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4ead9d640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec155720 0x7ff4ec1f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:44.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4ea59c640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec0700c0 0x7ff4ec1428f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:44.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cf59e640 1 -- 172.21.15.148:0/3861705586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d006fef0 msgr2=0x7f78d006f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cf59e640 1 --2- 172.21.15.148:0/3861705586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d006fef0 0x7f78d006f340 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f78d00c86f0 tx=0x7f78c801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e3fff640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 0x7ff4ec0c53d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4013b90 tx=0x7ff4d4013c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e2ffd640 1 -- 172.21.15.148:0/3574270206 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4014d30 con 0x7ff4ec152eb0 2023-10-30T13:10:44.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cf59e640 1 -- 172.21.15.148:0/3861705586 shutdown_connections 2023-10-30T13:10:44.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cf59e640 1 --2- 172.21.15.148:0/3861705586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 0x7f78d006f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec148d30 con 0x7ff4ec152eb0 2023-10-30T13:10:44.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cf59e640 1 --2- 172.21.15.148:0/3861705586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d006fef0 0x7f78d006f340 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cf59e640 1 --2- 172.21.15.148:0/3861705586 >> v2:172.21.15.150:3300/0 conn(0x7f78d0157c80 0x7f78d0161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cf59e640 1 -- 172.21.15.148:0/3861705586 >> 172.21.15.148:0/3861705586 conn(0x7f78d00c75d0 msgr2=0x7f78d00c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cf59e640 1 -- 172.21.15.148:0/3861705586 shutdown_connections 2023-10-30T13:10:44.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec0c71b0 con 0x7ff4ec152eb0 2023-10-30T13:10:44.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e2ffd640 1 -- 172.21.15.148:0/3574270206 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d4014e90 con 0x7ff4ec152eb0 2023-10-30T13:10:44.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4e2ffd640 1 -- 172.21.15.148:0/3574270206 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4011310 con 0x7ff4ec152eb0 2023-10-30T13:10:44.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7f78cf59e640 1 -- 172.21.15.148:0/3861705586 wait complete. 2023-10-30T13:10:44.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.680+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d0005260 con 0x7ff4ec152eb0 2023-10-30T13:10:44.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78cf59e640 1 Processor -- start 2023-10-30T13:10:44.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7ff4e2ffd640 1 -- 172.21.15.148:0/3574270206 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4d4011470 con 0x7ff4ec152eb0 2023-10-30T13:10:44.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78cf59e640 1 -- start start 2023-10-30T13:10:44.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f705ffff640 1 --2- 172.21.15.148:0/2345665948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f705800e210 0x7f70580dca50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f70580ddb20 tx=0x7f704c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c83f0c06270b0a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7ff4e2ffd640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4cc066520 0x7ff4cc0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78cf59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 0x7f78d00c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78cf59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78d006fef0 0x7f78d014cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78cf59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d0157c80 0x7f78d014d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78cf59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0152eb0 con 0x7f78d0157c80 2023-10-30T13:10:44.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78cf59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d01f2de0 con 0x7f78d006fef0 2023-10-30T13:10:44.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78cf59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0158fa0 con 0x7f78d006ab20 2023-10-30T13:10:44.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7ff4e2ffd640 1 -- 172.21.15.148:0/3574270206 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4d408b810 con 0x7ff4ec152eb0 2023-10-30T13:10:44.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78ce59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 0x7f78d00c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78ced9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d0157c80 0x7f78d014d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78ce59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 0x7f78d00c61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45826/0 (socket says 172.21.15.148:45826) 2023-10-30T13:10:44.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78ce59c640 1 -- 172.21.15.148:0/489811525 learned_addr learned my addr 172.21.15.148:0/489811525 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7ff4ea59c640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4cc066520 0x7ff4cc0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78ced9d640 1 -- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 msgr2=0x7f78d00c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78ced9d640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 0x7f78d00c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78ced9d640 1 -- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78d006fef0 msgr2=0x7f78d014cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T13:10:44.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78ced9d640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78d006fef0 0x7f78d014cc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.684+0000 7f78ced9d640 1 -- 172.21.15.148:0/489811525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8009b30 con 0x7f78d0157c80 2023-10-30T13:10:44.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f78ced9d640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d0157c80 0x7f78d014d140 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78c000d530 tx=0x7f78c000d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7ff4ea59c640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4cc066520 0x7ff4cc0689e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0b1000 tx=0x7ff4dc0a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f3ffff640 1 -- 172.21.15.148:0/2902181568 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f400a5860 con 0x7f9f480ca360 2023-10-30T13:10:44.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f3ffff640 1 -- 172.21.15.148:0/2902181568 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f400a59c0 con 0x7f9f480ca360 2023-10-30T13:10:44.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f3ffff640 1 -- 172.21.15.148:0/2902181568 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f400a5d50 con 0x7f9f480ca360 2023-10-30T13:10:44.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7fe2957fa640 1 -- 172.21.15.148:0/1049300821 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe290051300 con 0x7fe2980cf160 2023-10-30T13:10:44.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2902181568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 msgr2=0x7f9f480cac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f4f31d640 1 --2- 172.21.15.148:0/2902181568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f480cac40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009dc0 tx=0x7f9f4001c800 comp rx=0 tx=0).stop 2023-10-30T13:10:44.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e2dd9b640 1 -- 172.21.15.148:0/556451071 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e280a5740 con 0x7f2e300c4430 2023-10-30T13:10:44.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e2dd9b640 1 -- 172.21.15.148:0/556451071 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e280a58a0 con 0x7f2e300c4430 2023-10-30T13:10:44.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e2dd9b640 1 -- 172.21.15.148:0/556451071 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e280a5c30 con 0x7f2e300c4430 2023-10-30T13:10:44.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2902181568 shutdown_connections 2023-10-30T13:10:44.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f4f31d640 1 --2- 172.21.15.148:0/2902181568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f480cac40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2902181568 >> 172.21.15.148:0/2902181568 conn(0x7f9f4806bfe0 msgr2=0x7f9f4806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e35079640 1 -- 172.21.15.148:0/556451071 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e300c4430 msgr2=0x7f2e300ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e35079640 1 --2- 172.21.15.148:0/556451071 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e300c4430 0x7f2e300ca360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009e50 tx=0x7f2e2801c9d0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e35079640 1 -- 172.21.15.148:0/556451071 shutdown_connections 2023-10-30T13:10:44.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e35079640 1 --2- 172.21.15.148:0/556451071 >> v2:172.21.15.150:3300/0 conn(0x7f2e30159830 0x7f2e30159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e35079640 1 --2- 172.21.15.148:0/556451071 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e300c4430 0x7f2e300ca360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e35079640 1 -- 172.21.15.148:0/556451071 >> 172.21.15.148:0/556451071 conn(0x7f2e3006bfe0 msgr2=0x7f2e3006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e35079640 1 -- 172.21.15.148:0/556451071 shutdown_connections 2023-10-30T13:10:44.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2902181568 shutdown_connections 2023-10-30T13:10:44.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f2e35079640 1 -- 172.21.15.148:0/556451071 wait complete. 2023-10-30T13:10:44.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.688+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2902181568 wait complete. 2023-10-30T13:10:44.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f705f7fe640 1 -- 172.21.15.148:0/2345665948 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c0a8950 con 0x7f705800e210 2023-10-30T13:10:44.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f705f7fe640 1 -- 172.21.15.148:0/2345665948 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704c0a8ab0 con 0x7f705800e210 2023-10-30T13:10:44.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f705f7fe640 1 -- 172.21.15.148:0/2345665948 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c0a8e40 con 0x7f705800e210 2023-10-30T13:10:44.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f78beffd640 1 -- 172.21.15.148:0/489811525 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c00125f0 con 0x7f78d0157c80 2023-10-30T13:10:44.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f78beffd640 1 -- 172.21.15.148:0/489811525 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c0012750 con 0x7f78d0157c80 2023-10-30T13:10:44.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f78beffd640 1 -- 172.21.15.148:0/489811525 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c0012b00 con 0x7f78d0157c80 2023-10-30T13:10:44.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f9f4f31d640 1 Processor -- start 2023-10-30T13:10:44.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f2e35079640 1 Processor -- start 2023-10-30T13:10:44.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f2e35079640 1 -- start start 2023-10-30T13:10:44.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f70658dc640 1 -- 172.21.15.148:0/2345665948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f705800e210 msgr2=0x7f70580dca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f70658dc640 1 --2- 172.21.15.148:0/2345665948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f705800e210 0x7f70580dca50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f70580ddb20 tx=0x7f704c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:10:44.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f70658dc640 1 -- 172.21.15.148:0/2345665948 shutdown_connections 2023-10-30T13:10:44.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f70658dc640 1 --2- 172.21.15.148:0/2345665948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f705800e210 0x7f70580dca50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f70658dc640 1 --2- 172.21.15.148:0/2345665948 >> v2:172.21.15.150:3300/0 conn(0x7f7058033710 0x7f70580dcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f70658dc640 1 -- 172.21.15.148:0/2345665948 >> 172.21.15.148:0/2345665948 conn(0x7f70580501d0 msgr2=0x7f70580505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7ff4e2ffd640 1 -- 172.21.15.148:0/3574270206 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4d4058db0 con 0x7ff4ec152eb0 2023-10-30T13:10:44.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f9f4f31d640 1 -- start start 2023-10-30T13:10:44.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d01f2ff0 con 0x7f78d0157c80 2023-10-30T13:10:44.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d01f35b0 con 0x7f78d0157c80 2023-10-30T13:10:44.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b4005260 con 0x7f78d0157c80 2023-10-30T13:10:44.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f70658dc640 1 -- 172.21.15.148:0/2345665948 shutdown_connections 2023-10-30T13:10:44.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f2e35079640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e300c4430 0x7f2e30152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e35079640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e30153c60 0x7f2e301f4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e35079640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e30159830 0x7f2e30143be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e300c4430 0x7f2e30152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e300c4430 0x7f2e30152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37916/0 (socket says 172.21.15.148:37916) 2023-10-30T13:10:44.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2e59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e30153c60 0x7f2e301f4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2e59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e30153c60 0x7f2e301f4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45836/0 (socket says 172.21.15.148:45836) 2023-10-30T13:10:44.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2f59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e30159830 0x7f2e30143be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2f59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e30159830 0x7f2e30143be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42042/0 (socket says 172.21.15.148:42042) 2023-10-30T13:10:44.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2ed9d640 1 -- 172.21.15.148:0/2170461078 learned_addr learned my addr 172.21.15.148:0/2170461078 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3014f9b0 con 0x7f2e30159830 2023-10-30T13:10:44.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300cec80 con 0x7f2e300c4430 2023-10-30T13:10:44.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300ce6c0 con 0x7f2e30153c60 2023-10-30T13:10:44.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2ed9d640 1 -- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e30153c60 msgr2=0x7f2e301f4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2ed9d640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e30153c60 0x7f2e301f4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2ed9d640 1 -- 172.21.15.148:0/2170461078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e30159830 msgr2=0x7f2e30143be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2ed9d640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e30159830 0x7f2e30143be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2ed9d640 1 -- 172.21.15.148:0/2170461078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e28009b30 con 0x7f2e300c4430 2023-10-30T13:10:44.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2f59e640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e30159830 0x7f2e30143be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:44.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f78beffd640 1 -- 172.21.15.148:0/489811525 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78c0012c60 con 0x7f78d0157c80 2023-10-30T13:10:44.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.692+0000 7f70658dc640 1 -- 172.21.15.148:0/2345665948 wait complete. 2023-10-30T13:10:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f2e2ed9d640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e300c4430 0x7f2e30152ac0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b3e0 tx=0x7f2e2400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.700+0000 7f2e177fe640 1 -- 172.21.15.148:0/2170461078 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24011830 con 0x7f2e300c4430 2023-10-30T13:10:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.700+0000 7f2e177fe640 1 -- 172.21.15.148:0/2170461078 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e240119b0 con 0x7f2e300c4430 2023-10-30T13:10:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.700+0000 7f2e177fe640 1 -- 172.21.15.148:0/2170461078 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24011d60 con 0x7f2e300c4430 2023-10-30T13:10:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.700+0000 7f78beffd640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78b0066570 0x7f78b0068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.700+0000 7f78beffd640 1 -- 172.21.15.148:0/489811525 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78c0084db0 con 0x7f78d0157c80 2023-10-30T13:10:44.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.700+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e300ce9a0 con 0x7f2e300c4430 2023-10-30T13:10:44.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.700+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3014f5d0 con 0x7f2e300c4430 2023-10-30T13:10:44.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.700+0000 7f78ce59c640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78b0066570 0x7f78b0068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f78beffd640 1 -- 172.21.15.148:0/489811525 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78c0052180 con 0x7f78d0157c80 2023-10-30T13:10:44.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f78ce59c640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78b0066570 0x7f78b0068a30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009b20 tx=0x7f78c4006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f9f4f31d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f480c4430 0x7f9f481495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f9f4f31d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f48146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f9f4f31d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f48153c60 0x7f9f481506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f9f4f31d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f480c88d0 con 0x7f9f480c4430 2023-10-30T13:10:44.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f9f4f31d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f480c79c0 con 0x7f9f48153c60 2023-10-30T13:10:44.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.696+0000 7f9f4f31d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f48151f30 con 0x7f9f480ca360 2023-10-30T13:10:44.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4d8ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f48153c60 0x7f9f481506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4d8ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f48153c60 0x7f9f481506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37922/0 (socket says 172.21.15.148:37922) 2023-10-30T13:10:44.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4d8ba640 1 -- 172.21.15.148:0/2627435085 learned_addr learned my addr 172.21.15.148:0/2627435085 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4c8b8640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f48146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.700+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3004e170 con 0x7f2e300c4430 2023-10-30T13:10:44.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f70658dc640 1 Processor -- start 2023-10-30T13:10:44.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4d0b9640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f480c4430 0x7f9f481495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4d8ba640 1 -- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 msgr2=0x7f9f48146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4d8ba640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f48146850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4d8ba640 1 -- 172.21.15.148:0/2627435085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f480c4430 msgr2=0x7f9f481495f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4d8ba640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f480c4430 0x7f9f481495f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4d8ba640 1 -- 172.21.15.148:0/2627435085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f40009a70 con 0x7f9f48153c60 2023-10-30T13:10:44.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f9f4c8b8640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f48146850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f2e177fe640 1 -- 172.21.15.148:0/2170461078 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e2401a400 con 0x7f2e300c4430 2023-10-30T13:10:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f2e177fe640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e1c0664d0 0x7f2e1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f2e177fe640 1 -- 172.21.15.148:0/2170461078 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e24083f50 con 0x7f2e300c4430 2023-10-30T13:10:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f2e2e59c640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e1c0664d0 0x7f2e1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.704+0000 7f70658dc640 1 -- start start 2023-10-30T13:10:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f9f4d8ba640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f48153c60 0x7f9f481506f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9f4001a430 tx=0x7f9f400195a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f2e2e59c640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e1c0664d0 0x7f2e1c068990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2e280ae000 tx=0x7f2e280a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70658dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f705800e210 0x7f705804aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70658dc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7058033710 0x7f7058173100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70658dc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70580d6b10 0x7f7058173640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70648da640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f705800e210 0x7f705804aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70648da640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f705800e210 0x7f705804aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37924/0 (socket says 172.21.15.148:37924) 2023-10-30T13:10:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70648da640 1 -- 172.21.15.148:0/2338179161 learned_addr learned my addr 172.21.15.148:0/2338179161 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f705ffff640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7058033710 0x7f7058173100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70650db640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70580d6b10 0x7f7058173640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70580e06e0 con 0x7f7058033710 2023-10-30T13:10:44.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70580ddf70 con 0x7f705800e210 2023-10-30T13:10:44.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70580de250 con 0x7f70580d6b10 2023-10-30T13:10:44.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70650db640 1 -- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f705800e210 msgr2=0x7f705804aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70650db640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f705800e210 0x7f705804aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70650db640 1 -- 172.21.15.148:0/2338179161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7058033710 msgr2=0x7f7058173100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70650db640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7058033710 0x7f7058173100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.708+0000 7f70650db640 1 -- 172.21.15.148:0/2338179161 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c009420 con 0x7f70580d6b10 2023-10-30T13:10:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.712+0000 7f70650db640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70580d6b10 0x7f7058173640 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7060000b70 tx=0x7f70600674c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.712+0000 7f705d7fa640 1 -- 172.21.15.148:0/2338179161 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060072770 con 0x7f70580d6b10 2023-10-30T13:10:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.712+0000 7f705d7fa640 1 -- 172.21.15.148:0/2338179161 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7060090070 con 0x7f70580d6b10 2023-10-30T13:10:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.712+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70580dfc00 con 0x7f70580d6b10 2023-10-30T13:10:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.712+0000 7f705d7fa640 1 -- 172.21.15.148:0/2338179161 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060072b70 con 0x7f70580d6b10 2023-10-30T13:10:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.712+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70580e01c0 con 0x7f70580d6b10 2023-10-30T13:10:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70580dfa60 con 0x7f70580d6b10 2023-10-30T13:10:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f705d7fa640 1 -- 172.21.15.148:0/2338179161 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7060092070 con 0x7f70580d6b10 2023-10-30T13:10:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f705d7fa640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70440664a0 0x7f7044068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f9f3dffb640 1 -- 172.21.15.148:0/2627435085 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f400aed50 con 0x7f9f48153c60 2023-10-30T13:10:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f9f3dffb640 1 -- 172.21.15.148:0/2627435085 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f400b7410 con 0x7f9f48153c60 2023-10-30T13:10:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f9f3dffb640 1 -- 172.21.15.148:0/2627435085 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f400b77c0 con 0x7f9f48153c60 2023-10-30T13:10:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f705d7fa640 1 -- 172.21.15.148:0/2338179161 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7060104af0 con 0x7f70580d6b10 2023-10-30T13:10:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f70648da640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70440664a0 0x7f7044068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f481521d0 con 0x7f9f48153c60 2023-10-30T13:10:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.716+0000 7f2e177fe640 1 -- 172.21.15.148:0/2170461078 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e240513e0 con 0x7f2e300c4430 2023-10-30T13:10:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.720+0000 7f70648da640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70440664a0 0x7f7044068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7054005890 tx=0x7f7054009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.720+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f480c8280 con 0x7f9f48153c60 2023-10-30T13:10:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.720+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4804e170 con 0x7f9f48153c60 2023-10-30T13:10:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.724+0000 7f9f3dffb640 1 -- 172.21.15.148:0/2627435085 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f400b4050 con 0x7f9f48153c60 2023-10-30T13:10:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.724+0000 7f9f3dffb640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f2c066570 0x7f9f2c068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.724+0000 7f9f3dffb640 1 -- 172.21.15.148:0/2627435085 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f4011ef60 con 0x7f9f48153c60 2023-10-30T13:10:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.724+0000 7f9f4d0b9640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f2c066570 0x7f9f2c068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.724+0000 7f9f4d0b9640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f2c066570 0x7f9f2c068a30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f44000f70 tx=0x7f9f44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.728+0000 7f705d7fa640 1 -- 172.21.15.148:0/2338179161 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70580dfa60 con 0x7f70580d6b10 2023-10-30T13:10:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.740+0000 7f9f3dffb640 1 -- 172.21.15.148:0/2627435085 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f400ec330 con 0x7f9f48153c60 2023-10-30T13:10:44.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.936+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2318156400 con 0x7f23181541b0 2023-10-30T13:10:44.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.936+0000 7f230e7fc640 1 -- 172.21.15.148:0/794856997 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2318156400 con 0x7f23181541b0 2023-10-30T13:10:44.941 INFO:teuthology.orchestra.run.smithi148.stdout:55834574856 2023-10-30T13:10:44.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22fc0664d0 msgr2=0x7f22fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22fc0664d0 0x7f22fc068990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f23140016c0 tx=0x7f2314009040 comp rx=0 tx=0).stop 2023-10-30T13:10:44.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23181541b0 msgr2=0x7f2318135cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23181541b0 0x7f2318135cf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f231001cc80 tx=0x7f2310019bf0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 shutdown_connections 2023-10-30T13:10:44.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22fc0664d0 0x7f22fc068990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23181541b0 0x7f2318135cf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2318155210 0x7f23181357b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 --2- 172.21.15.148:0/794856997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2304005350 0x7f231813ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 >> 172.21.15.148:0/794856997 conn(0x7f23180c7910 msgr2=0x7f2318159af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 shutdown_connections 2023-10-30T13:10:44.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.940+0000 7f231fd2a640 1 -- 172.21.15.148:0/794856997 wait complete. 2023-10-30T13:10:44.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.964+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fddb80caa20 con 0x7fddb8159830 2023-10-30T13:10:44.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.968+0000 7fdd9ffff640 1 -- 172.21.15.148:0/893449920 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fddac0bb610 con 0x7fddb8159830 2023-10-30T13:10:44.971 INFO:teuthology.orchestra.run.smithi148.stdout:55834574857 2023-10-30T13:10:44.976 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2023-10-30T13:10:44.977 DEBUG:teuthology.parallel:result is None 2023-10-30T13:10:44.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdda4066500 msgr2=0x7fdda40689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdda4066500 0x7fdda40689c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009a40 tx=0x7fddb000b040 comp rx=0 tx=0).stop 2023-10-30T13:10:44.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fddb8159830 msgr2=0x7fddb8146f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:44.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fddb8159830 0x7fddb8146f70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fddb80b1950 tx=0x7fddac02a2d0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 shutdown_connections 2023-10-30T13:10:44.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdda4066500 0x7fdda40689c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fddb8159830 0x7fddb8146f70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddb8153c60 0x7fddb8146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 --2- 172.21.15.148:0/893449920 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fddb80c4430 0x7fddb80b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:44.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 >> 172.21.15.148:0/893449920 conn(0x7fddb806bfe0 msgr2=0x7fddb806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:44.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 shutdown_connections 2023-10-30T13:10:44.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.972+0000 7fddbdc17640 1 -- 172.21.15.148:0/893449920 wait complete. 2023-10-30T13:10:44.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.992+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe2981411e0 con 0x7fe2980cf160 2023-10-30T13:10:44.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.992+0000 7fe2957fa640 1 -- 172.21.15.148:0/1049300821 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe290019090 con 0x7fe2980cf160 2023-10-30T13:10:44.997 INFO:teuthology.orchestra.run.smithi148.stdout:55834574856 2023-10-30T13:10:45.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2840664f0 msgr2=0x7fe2840689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2840664f0 0x7fe2840689b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0098e0 tx=0x7fe28c009060 comp rx=0 tx=0).stop 2023-10-30T13:10:45.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2980cf160 msgr2=0x7fe2980b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2980cf160 0x7fe2980b4260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe29000ba50 tx=0x7fe29000bf10 comp rx=0 tx=0).stop 2023-10-30T13:10:45.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 shutdown_connections 2023-10-30T13:10:45.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2840664f0 0x7fe2840689b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe298157f70 0x7fe2980b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2980cf160 0x7fe2980b4260 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 --2- 172.21.15.148:0/1049300821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2980cad80 0x7fe2981525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 >> 172.21.15.148:0/1049300821 conn(0x7fe29806bfe0 msgr2=0x7fe2980c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 shutdown_connections 2023-10-30T13:10:45.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:44.996+0000 7fe29f165640 1 -- 172.21.15.148:0/1049300821 wait complete. 2023-10-30T13:10:45.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.004+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff4d0005510 con 0x7ff4ec152eb0 2023-10-30T13:10:45.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.004+0000 7ff4e2ffd640 1 -- 172.21.15.148:0/3574270206 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff4d405c3d0 con 0x7ff4ec152eb0 2023-10-30T13:10:45.009 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2023-10-30T13:10:45.009 INFO:teuthology.orchestra.run.smithi148.stdout:55834574857 2023-10-30T13:10:45.010 DEBUG:teuthology.parallel:result is None 2023-10-30T13:10:45.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4cc066520 msgr2=0x7ff4cc0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4cc066520 0x7ff4cc0689e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0b1000 tx=0x7ff4dc0a9070 comp rx=0 tx=0).stop 2023-10-30T13:10:45.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 msgr2=0x7ff4ec0c53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 0x7ff4ec0c53d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4013b90 tx=0x7ff4d4013c80 comp rx=0 tx=0).stop 2023-10-30T13:10:45.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 shutdown_connections 2023-10-30T13:10:45.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4cc066520 0x7ff4cc0689e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec152eb0 0x7ff4ec0c53d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec155720 0x7ff4ec1f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 --2- 172.21.15.148:0/3574270206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec0700c0 0x7ff4ec1428f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 >> 172.21.15.148:0/3574270206 conn(0x7ff4ec0c75d0 msgr2=0x7ff4ec0c40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 shutdown_connections 2023-10-30T13:10:45.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.008+0000 7ff4eb59e640 1 -- 172.21.15.148:0/3574270206 wait complete. 2023-10-30T13:10:45.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.012+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f78b40057e0 con 0x7f78d0157c80 2023-10-30T13:10:45.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78beffd640 1 -- 172.21.15.148:0/489811525 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f78c00557a0 con 0x7f78d0157c80 2023-10-30T13:10:45.019 INFO:teuthology.orchestra.run.smithi148.stdout:55834574856 2023-10-30T13:10:45.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78b0066570 msgr2=0x7f78b0068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78b0066570 0x7f78b0068a30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009b20 tx=0x7f78c4006f90 comp rx=0 tx=0).stop 2023-10-30T13:10:45.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d0157c80 msgr2=0x7f78d014d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d0157c80 0x7f78d014d140 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78c000d530 tx=0x7f78c000d9f0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 shutdown_connections 2023-10-30T13:10:45.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78b0066570 0x7f78b0068a30 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009b20 tx=0x7f78c4006f90 comp rx=0 tx=0).stop 2023-10-30T13:10:45.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78d0157c80 0x7f78d014d140 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78d006fef0 0x7f78d014cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 --2- 172.21.15.148:0/489811525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78d006ab20 0x7f78d00c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 >> 172.21.15.148:0/489811525 conn(0x7f78d00c75d0 msgr2=0x7f78d00c8930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 shutdown_connections 2023-10-30T13:10:45.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.016+0000 7f78cf59e640 1 -- 172.21.15.148:0/489811525 wait complete. 2023-10-30T13:10:45.030 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2023-10-30T13:10:45.030 DEBUG:teuthology.parallel:result is None 2023-10-30T13:10:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.028+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2e300caa20 con 0x7f2e300c4430 2023-10-30T13:10:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.028+0000 7f2e177fe640 1 -- 172.21.15.148:0/2170461078 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e24015090 con 0x7f2e300c4430 2023-10-30T13:10:45.034 INFO:teuthology.orchestra.run.smithi148.stdout:55834574857 2023-10-30T13:10:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e1c0664d0 msgr2=0x7f2e1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e1c0664d0 0x7f2e1c068990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2e280ae000 tx=0x7f2e280a7070 comp rx=0 tx=0).stop 2023-10-30T13:10:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e300c4430 msgr2=0x7f2e30152ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e300c4430 0x7f2e30152ac0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b3e0 tx=0x7f2e2400b8a0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 shutdown_connections 2023-10-30T13:10:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e1c0664d0 0x7f2e1c068990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e30159830 0x7f2e30143be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e30153c60 0x7f2e301f4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 --2- 172.21.15.148:0/2170461078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e300c4430 0x7f2e30152ac0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 >> 172.21.15.148:0/2170461078 conn(0x7f2e3006bfe0 msgr2=0x7f2e30146ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 shutdown_connections 2023-10-30T13:10:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f2e35079640 1 -- 172.21.15.148:0/2170461078 wait complete. 2023-10-30T13:10:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.032+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f70580dfa60 con 0x7f70580d6b10 2023-10-30T13:10:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f705d7fa640 1 -- 172.21.15.148:0/2338179161 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f70580dfa60 con 0x7f70580d6b10 2023-10-30T13:10:45.040 INFO:teuthology.orchestra.run.smithi148.stdout:55834574856 2023-10-30T13:10:45.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70440664a0 msgr2=0x7f7044068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70440664a0 0x7f7044068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7054005890 tx=0x7f7054009040 comp rx=0 tx=0).stop 2023-10-30T13:10:45.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70580d6b10 msgr2=0x7f7058173640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70580d6b10 0x7f7058173640 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7060000b70 tx=0x7f70600674c0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 shutdown_connections 2023-10-30T13:10:45.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70440664a0 0x7f7044068960 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7058033710 0x7f7058173100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70580d6b10 0x7f7058173640 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 --2- 172.21.15.148:0/2338179161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f705800e210 0x7f705804aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 >> 172.21.15.148:0/2338179161 conn(0x7f70580501d0 msgr2=0x7f7058005560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 shutdown_connections 2023-10-30T13:10:45.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.036+0000 7f70658dc640 1 -- 172.21.15.148:0/2338179161 wait complete. 2023-10-30T13:10:45.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2023-10-30T13:10:45.044 DEBUG:teuthology.parallel:result is None 2023-10-30T13:10:45.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2023-10-30T13:10:45.048 DEBUG:teuthology.parallel:result is None 2023-10-30T13:10:45.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.052+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9f480caa20 con 0x7f9f48153c60 2023-10-30T13:10:45.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.052+0000 7f9f3dffb640 1 -- 172.21.15.148:0/2627435085 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9f400b2020 con 0x7f9f48153c60 2023-10-30T13:10:45.057 INFO:teuthology.orchestra.run.smithi148.stdout:55834574857 2023-10-30T13:10:45.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f2c066570 msgr2=0x7f9f2c068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f2c066570 0x7f9f2c068a30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f44000f70 tx=0x7f9f44009040 comp rx=0 tx=0).stop 2023-10-30T13:10:45.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f48153c60 msgr2=0x7f9f481506f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f48153c60 0x7f9f481506f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9f4001a430 tx=0x7f9f400195a0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 shutdown_connections 2023-10-30T13:10:45.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f2c066570 0x7f9f2c068a30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f48153c60 0x7f9f481506f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f480ca360 0x7f9f48146850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 --2- 172.21.15.148:0/2627435085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f480c4430 0x7f9f481495f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 >> 172.21.15.148:0/2627435085 conn(0x7f9f4806bfe0 msgr2=0x7f9f48152d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 shutdown_connections 2023-10-30T13:10:45.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.056+0000 7f9f4f31d640 1 -- 172.21.15.148:0/2627435085 wait complete. 2023-10-30T13:10:45.063 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2023-10-30T13:10:45.064 DEBUG:teuthology.parallel:result is None 2023-10-30T13:10:45.066 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2023-10-30T13:10:45.066 DEBUG:teuthology.parallel:result is None 2023-10-30T13:10:45.079 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2023-10-30T13:10:45.079 DEBUG:teuthology.parallel:result is None 2023-10-30T13:10:45.079 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T13:10:45.080 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:10:45.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3da26d6640 1 Processor -- start 2023-10-30T13:10:45.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3da26d6640 1 -- start start 2023-10-30T13:10:45.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3da26d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3d9c06f7a0 0x7f3d9c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3da26d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c071110 0x7f3d9c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3da26d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c066cf0 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3da26d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9c0700c0 con 0x7f3d9c06f7a0 2023-10-30T13:10:45.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3da26d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0c70d0 con 0x7f3d9c071110 2023-10-30T13:10:45.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3d9bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3d9c06f7a0 0x7f3d9c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3d9bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3d9c06f7a0 0x7f3d9c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42078/0 (socket says 172.21.15.148:42078) 2023-10-30T13:10:45.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3d9b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3d9c0c6bd0 0x7f3d9c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39402/0 (socket says 172.21.15.148:39402) 2023-10-30T13:10:45.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3d9bfff640 1 -- 172.21.15.148:0/3551454660 learned_addr learned my addr 172.21.15.148:0/3551454660 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:45.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3da0c73640 1 --2- 172.21.15.148:0/3551454660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c071110 0x7f3d9c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3d9bfff640 1 -- 172.21.15.148:0/3551454660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c071110 msgr2=0x7f3d9c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:45.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3d9bfff640 1 --2- 172.21.15.148:0/3551454660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c071110 0x7f3d9c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3d9bfff640 1 -- 172.21.15.148:0/3551454660 >> v1:172.21.15.148:6789/0 conn(0x7f3d9c0c6bd0 legacy=0x7f3d9c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3d9bfff640 1 -- 172.21.15.148:0/3551454660 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c0c7390 con 0x7f3d9c06f7a0 2023-10-30T13:10:45.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.260+0000 7f3d9affd640 1 -- 172.21.15.148:0/3551454660 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792798128 0 0) 0x7f3d9c066cf0 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3d9bfff640 1 --2- 172.21.15.148:0/3551454660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c06f7a0 0x7f3d9c06fb80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009870 tx=0x7f3d8c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661ad5cabf2a1d03 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:45.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3d9affd640 1 -- 172.21.15.148:0/3551454660 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c0a8a30 con 0x7f3d9c06f7a0 2023-10-30T13:10:45.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3d9affd640 1 -- 172.21.15.148:0/3551454660 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c0a8b90 con 0x7f3d9c06f7a0 2023-10-30T13:10:45.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3d9affd640 1 -- 172.21.15.148:0/3551454660 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c0b0440 con 0x7f3d9c06f7a0 2023-10-30T13:10:45.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3da26d6640 1 -- 172.21.15.148:0/3551454660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c06f7a0 msgr2=0x7f3d9c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3da26d6640 1 --2- 172.21.15.148:0/3551454660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c06f7a0 0x7f3d9c06fb80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009870 tx=0x7f3d8c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:10:45.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3da26d6640 1 -- 172.21.15.148:0/3551454660 shutdown_connections 2023-10-30T13:10:45.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3da26d6640 1 --2- 172.21.15.148:0/3551454660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c071110 0x7f3d9c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3da26d6640 1 --2- 172.21.15.148:0/3551454660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c06f7a0 0x7f3d9c06fb80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3da26d6640 1 -- 172.21.15.148:0/3551454660 >> 172.21.15.148:0/3551454660 conn(0x7f3d9c0c7910 msgr2=0x7f3d9c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3da26d6640 1 -- 172.21.15.148:0/3551454660 shutdown_connections 2023-10-30T13:10:45.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.264+0000 7f3da26d6640 1 -- 172.21.15.148:0/3551454660 wait complete. 2023-10-30T13:10:45.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 Processor -- start 2023-10-30T13:10:45.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 -- start start 2023-10-30T13:10:45.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c06f7a0 0x7f3d9c0c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d9c071110 0x7f3d9c14fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c0c6bd0 0x7f3d9c1500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c1e1450 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c1f0670 con 0x7f3d9c071110 2023-10-30T13:10:45.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c1efa60 con 0x7f3d9c06f7a0 2023-10-30T13:10:45.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da0c73640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c0c6bd0 0x7f3d9c1500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da0c73640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c0c6bd0 0x7f3d9c1500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42082/0 (socket says 172.21.15.148:42082) 2023-10-30T13:10:45.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da0c73640 1 -- 172.21.15.148:0/1161073604 learned_addr learned my addr 172.21.15.148:0/1161073604 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:45.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3d9bfff640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c06f7a0 0x7f3d9c0c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3d9b7fe640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d9c071110 0x7f3d9c14fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da0c73640 1 -- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c06f7a0 msgr2=0x7f3d9c0c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da0c73640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c06f7a0 0x7f3d9c0c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da0c73640 1 -- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d9c071110 msgr2=0x7f3d9c14fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da0c73640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d9c071110 0x7f3d9c14fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da0c73640 1 -- 172.21.15.148:0/1161073604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c0094d0 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3d9b7fe640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d9c071110 0x7f3d9c14fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da0c73640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c0c6bd0 0x7f3d9c1500b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3d94007e70 tx=0x7f3d9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3d98ff9640 1 -- 172.21.15.148:0/1161073604 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94014620 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3d98ff9640 1 -- 172.21.15.148:0/1161073604 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d940147a0 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3d98ff9640 1 -- 172.21.15.148:0/1161073604 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d94014b50 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c1efd40 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.268+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c1f0300 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.272+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c1f04c0 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.272+0000 7f3d98ff9640 1 -- 172.21.15.148:0/1161073604 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d94014cb0 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.272+0000 7f3d98ff9640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d880664d0 0x7f3d88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.272+0000 7f3d98ff9640 1 -- 172.21.15.148:0/1161073604 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d940847a0 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.272+0000 7f3d9bfff640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d880664d0 0x7f3d88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.272+0000 7f3d9bfff640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d880664d0 0x7f3d88068990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0b2000 tx=0x7f3d8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:45.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.276+0000 7f3d98ff9640 1 -- 172.21.15.148:0/1161073604 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d94051b80 con 0x7f3d9c0c6bd0 2023-10-30T13:10:45.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.428+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d9c070350 con 0x7f3d880664d0 2023-10-30T13:10:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.448+0000 7f3d98ff9640 1 -- 172.21.15.148:0/1161073604 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264490 (secure 0 0 0) 0x7f3d8c0b04c0 con 0x7f3d880664d0 2023-10-30T13:10:45.454 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:10:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d880664d0 msgr2=0x7f3d88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d880664d0 0x7f3d88068990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0b2000 tx=0x7f3d8c009040 comp rx=0 tx=0).stop 2023-10-30T13:10:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c0c6bd0 msgr2=0x7f3d9c1500b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c0c6bd0 0x7f3d9c1500b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3d94007e70 tx=0x7f3d9400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 shutdown_connections 2023-10-30T13:10:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d880664d0 0x7f3d88068990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d9c0c6bd0 0x7f3d9c1500b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d9c071110 0x7f3d9c14fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 --2- 172.21.15.148:0/1161073604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d9c06f7a0 0x7f3d9c0c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 >> 172.21.15.148:0/1161073604 conn(0x7f3d9c0c7910 msgr2=0x7f3d9c0c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 shutdown_connections 2023-10-30T13:10:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.452+0000 7f3da26d6640 1 -- 172.21.15.148:0/1161073604 wait complete. 2023-10-30T13:10:45.474 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-30T13:10:44.263848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218948,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755772,"statfs":{"total":773094113280,"available":772869910528,"internally_reserved":0,"allocated":2887680,"data_stored":1751721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.947746"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322509+0000","last_change":"2023-10-30T13:10:34.379031+0000","last_active":"2023-10-30T13:10:35.322509+0000","last_peered":"2023-10-30T13:10:35.322509+0000","last_clean":"2023-10-30T13:10:35.322509+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:10:35.322509+0000","last_undegraded":"2023-10-30T13:10:35.322509+0000","last_fullsized":"2023-10-30T13:10:35.322509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:32.529312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919896+0000","last_change":"2023-10-30T13:10:34.324308+0000","last_active":"2023-10-30T13:10:35.919896+0000","last_peered":"2023-10-30T13:10:35.919896+0000","last_clean":"2023-10-30T13:10:35.919896+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:10:35.919896+0000","last_undegraded":"2023-10-30T13:10:35.919896+0000","last_fullsized":"2023-10-30T13:10:35.919896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:21.885716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919373+0000","last_change":"2023-10-30T13:10:34.334122+0000","last_active":"2023-10-30T13:10:35.919373+0000","last_peered":"2023-10-30T13:10:35.919373+0000","last_clean":"2023-10-30T13:10:35.919373+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:10:35.919373+0000","last_undegraded":"2023-10-30T13:10:35.919373+0000","last_fullsized":"2023-10-30T13:10:35.919373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:55.045024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324579+0000","last_change":"2023-10-30T13:10:34.346795+0000","last_active":"2023-10-30T13:10:35.324579+0000","last_peered":"2023-10-30T13:10:35.324579+0000","last_clean":"2023-10-30T13:10:35.324579+0000","last_became_active":"2023-10-30T13:10:34.346079+0000","last_became_peered":"2023-10-30T13:10:34.346079+0000","last_unstale":"2023-10-30T13:10:35.324579+0000","last_undegraded":"2023-10-30T13:10:35.324579+0000","last_fullsized":"2023-10-30T13:10:35.324579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:09.546205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322339+0000","last_change":"2023-10-30T13:10:34.371670+0000","last_active":"2023-10-30T13:10:35.322339+0000","last_peered":"2023-10-30T13:10:35.322339+0000","last_clean":"2023-10-30T13:10:35.322339+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:10:35.322339+0000","last_undegraded":"2023-10-30T13:10:35.322339+0000","last_fullsized":"2023-10-30T13:10:35.322339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:03:54.773965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323062+0000","last_change":"2023-10-30T13:10:34.335752+0000","last_active":"2023-10-30T13:10:35.323062+0000","last_peered":"2023-10-30T13:10:35.323062+0000","last_clean":"2023-10-30T13:10:35.323062+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:10:35.323062+0000","last_undegraded":"2023-10-30T13:10:35.323062+0000","last_fullsized":"2023-10-30T13:10:35.323062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:33.278920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321759+0000","last_change":"2023-10-30T13:10:34.371817+0000","last_active":"2023-10-30T13:10:35.321759+0000","last_peered":"2023-10-30T13:10:35.321759+0000","last_clean":"2023-10-30T13:10:35.321759+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:10:35.321759+0000","last_undegraded":"2023-10-30T13:10:35.321759+0000","last_fullsized":"2023-10-30T13:10:35.321759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:33.797113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318581+0000","last_change":"2023-10-30T13:10:34.348205+0000","last_active":"2023-10-30T13:10:35.318581+0000","last_peered":"2023-10-30T13:10:35.318581+0000","last_clean":"2023-10-30T13:10:35.318581+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:10:35.318581+0000","last_undegraded":"2023-10-30T13:10:35.318581+0000","last_fullsized":"2023-10-30T13:10:35.318581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:57.492738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322864+0000","last_change":"2023-10-30T13:10:34.381052+0000","last_active":"2023-10-30T13:10:35.322864+0000","last_peered":"2023-10-30T13:10:35.322864+0000","last_clean":"2023-10-30T13:10:35.322864+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:10:35.322864+0000","last_undegraded":"2023-10-30T13:10:35.322864+0000","last_fullsized":"2023-10-30T13:10:35.322864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:16.697332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759029+0000","last_change":"2023-10-30T13:10:34.356932+0000","last_active":"2023-10-30T13:10:35.759029+0000","last_peered":"2023-10-30T13:10:35.759029+0000","last_clean":"2023-10-30T13:10:35.759029+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:10:35.759029+0000","last_undegraded":"2023-10-30T13:10:35.759029+0000","last_fullsized":"2023-10-30T13:10:35.759029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:54.091993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758774+0000","last_change":"2023-10-30T13:10:34.357358+0000","last_active":"2023-10-30T13:10:35.758774+0000","last_peered":"2023-10-30T13:10:35.758774+0000","last_clean":"2023-10-30T13:10:35.758774+0000","last_became_active":"2023-10-30T13:10:34.356466+0000","last_became_peered":"2023-10-30T13:10:34.356466+0000","last_unstale":"2023-10-30T13:10:35.758774+0000","last_undegraded":"2023-10-30T13:10:35.758774+0000","last_fullsized":"2023-10-30T13:10:35.758774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:06:24.029159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924241+0000","last_change":"2023-10-30T13:10:34.365161+0000","last_active":"2023-10-30T13:10:35.924241+0000","last_peered":"2023-10-30T13:10:35.924241+0000","last_clean":"2023-10-30T13:10:35.924241+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:10:35.924241+0000","last_undegraded":"2023-10-30T13:10:35.924241+0000","last_fullsized":"2023-10-30T13:10:35.924241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:51.721060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318828+0000","last_change":"2023-10-30T13:10:34.351901+0000","last_active":"2023-10-30T13:10:35.318828+0000","last_peered":"2023-10-30T13:10:35.318828+0000","last_clean":"2023-10-30T13:10:35.318828+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:10:35.318828+0000","last_undegraded":"2023-10-30T13:10:35.318828+0000","last_fullsized":"2023-10-30T13:10:35.318828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:11.576546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920191+0000","last_change":"2023-10-30T13:10:34.323358+0000","last_active":"2023-10-30T13:10:35.920191+0000","last_peered":"2023-10-30T13:10:35.920191+0000","last_clean":"2023-10-30T13:10:35.920191+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:10:35.920191+0000","last_undegraded":"2023-10-30T13:10:35.920191+0000","last_fullsized":"2023-10-30T13:10:35.920191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:03.178822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919173+0000","last_change":"2023-10-30T13:10:34.360120+0000","last_active":"2023-10-30T13:10:35.919173+0000","last_peered":"2023-10-30T13:10:35.919173+0000","last_clean":"2023-10-30T13:10:35.919173+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:10:35.919173+0000","last_undegraded":"2023-10-30T13:10:35.919173+0000","last_fullsized":"2023-10-30T13:10:35.919173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:30.731746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322343+0000","last_change":"2023-10-30T13:10:34.333752+0000","last_active":"2023-10-30T13:10:35.322343+0000","last_peered":"2023-10-30T13:10:35.322343+0000","last_clean":"2023-10-30T13:10:35.322343+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:10:35.322343+0000","last_undegraded":"2023-10-30T13:10:35.322343+0000","last_fullsized":"2023-10-30T13:10:35.322343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:45.390603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324451+0000","last_change":"2023-10-30T13:10:34.371931+0000","last_active":"2023-10-30T13:10:35.324451+0000","last_peered":"2023-10-30T13:10:35.324451+0000","last_clean":"2023-10-30T13:10:35.324451+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:10:35.324451+0000","last_undegraded":"2023-10-30T13:10:35.324451+0000","last_fullsized":"2023-10-30T13:10:35.324451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:51.514990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919968+0000","last_change":"2023-10-30T13:10:34.352747+0000","last_active":"2023-10-30T13:10:35.919968+0000","last_peered":"2023-10-30T13:10:35.919968+0000","last_clean":"2023-10-30T13:10:35.919968+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:10:35.919968+0000","last_undegraded":"2023-10-30T13:10:35.919968+0000","last_fullsized":"2023-10-30T13:10:35.919968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:21.543351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323598+0000","last_change":"2023-10-30T13:10:34.376135+0000","last_active":"2023-10-30T13:10:35.323598+0000","last_peered":"2023-10-30T13:10:35.323598+0000","last_clean":"2023-10-30T13:10:35.323598+0000","last_became_active":"2023-10-30T13:10:34.375497+0000","last_became_peered":"2023-10-30T13:10:34.375497+0000","last_unstale":"2023-10-30T13:10:35.323598+0000","last_undegraded":"2023-10-30T13:10:35.323598+0000","last_fullsized":"2023-10-30T13:10:35.323598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:14.816946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758767+0000","last_change":"2023-10-30T13:10:34.366575+0000","last_active":"2023-10-30T13:10:35.758767+0000","last_peered":"2023-10-30T13:10:35.758767+0000","last_clean":"2023-10-30T13:10:35.758767+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:10:35.758767+0000","last_undegraded":"2023-10-30T13:10:35.758767+0000","last_fullsized":"2023-10-30T13:10:35.758767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:23.540645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920318+0000","last_change":"2023-10-30T13:10:34.368761+0000","last_active":"2023-10-30T13:10:35.920318+0000","last_peered":"2023-10-30T13:10:35.920318+0000","last_clean":"2023-10-30T13:10:35.920318+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:10:35.920318+0000","last_undegraded":"2023-10-30T13:10:35.920318+0000","last_fullsized":"2023-10-30T13:10:35.920318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:32.861756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323430+0000","last_change":"2023-10-30T13:10:34.372881+0000","last_active":"2023-10-30T13:10:35.323430+0000","last_peered":"2023-10-30T13:10:35.323430+0000","last_clean":"2023-10-30T13:10:35.323430+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:10:35.323430+0000","last_undegraded":"2023-10-30T13:10:35.323430+0000","last_fullsized":"2023-10-30T13:10:35.323430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:23.855779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318722+0000","last_change":"2023-10-30T13:10:34.359359+0000","last_active":"2023-10-30T13:10:35.318722+0000","last_peered":"2023-10-30T13:10:35.318722+0000","last_clean":"2023-10-30T13:10:35.318722+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:10:35.318722+0000","last_undegraded":"2023-10-30T13:10:35.318722+0000","last_fullsized":"2023-10-30T13:10:35.318722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:08.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322387+0000","last_change":"2023-10-30T13:10:34.347431+0000","last_active":"2023-10-30T13:10:35.322387+0000","last_peered":"2023-10-30T13:10:35.322387+0000","last_clean":"2023-10-30T13:10:35.322387+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:10:35.322387+0000","last_undegraded":"2023-10-30T13:10:35.322387+0000","last_fullsized":"2023-10-30T13:10:35.322387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:25.094617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319296+0000","last_change":"2023-10-30T13:10:34.359717+0000","last_active":"2023-10-30T13:10:35.319296+0000","last_peered":"2023-10-30T13:10:35.319296+0000","last_clean":"2023-10-30T13:10:35.319296+0000","last_became_active":"2023-10-30T13:10:34.359063+0000","last_became_peered":"2023-10-30T13:10:34.359063+0000","last_unstale":"2023-10-30T13:10:35.319296+0000","last_undegraded":"2023-10-30T13:10:35.319296+0000","last_fullsized":"2023-10-30T13:10:35.319296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:22.525493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759209+0000","last_change":"2023-10-30T13:10:34.379878+0000","last_active":"2023-10-30T13:10:35.759209+0000","last_peered":"2023-10-30T13:10:35.759209+0000","last_clean":"2023-10-30T13:10:35.759209+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:10:35.759209+0000","last_undegraded":"2023-10-30T13:10:35.759209+0000","last_fullsized":"2023-10-30T13:10:35.759209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:42.280634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919210+0000","last_change":"2023-10-30T13:10:34.332020+0000","last_active":"2023-10-30T13:10:35.919210+0000","last_peered":"2023-10-30T13:10:35.919210+0000","last_clean":"2023-10-30T13:10:35.919210+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:10:35.919210+0000","last_undegraded":"2023-10-30T13:10:35.919210+0000","last_fullsized":"2023-10-30T13:10:35.919210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:53.057012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319283+0000","last_change":"2023-10-30T13:10:34.363154+0000","last_active":"2023-10-30T13:10:35.319283+0000","last_peered":"2023-10-30T13:10:35.319283+0000","last_clean":"2023-10-30T13:10:35.319283+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:10:35.319283+0000","last_undegraded":"2023-10-30T13:10:35.319283+0000","last_fullsized":"2023-10-30T13:10:35.319283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:04.359629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920381+0000","last_change":"2023-10-30T13:10:34.332160+0000","last_active":"2023-10-30T13:10:35.920381+0000","last_peered":"2023-10-30T13:10:35.920381+0000","last_clean":"2023-10-30T13:10:35.920381+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:10:35.920381+0000","last_undegraded":"2023-10-30T13:10:35.920381+0000","last_fullsized":"2023-10-30T13:10:35.920381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:28.128344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323753+0000","last_change":"2023-10-30T13:10:34.337617+0000","last_active":"2023-10-30T13:10:35.323753+0000","last_peered":"2023-10-30T13:10:35.323753+0000","last_clean":"2023-10-30T13:10:35.323753+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:10:35.323753+0000","last_undegraded":"2023-10-30T13:10:35.323753+0000","last_fullsized":"2023-10-30T13:10:35.323753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:10.287441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920225+0000","last_change":"2023-10-30T13:10:34.359681+0000","last_active":"2023-10-30T13:10:35.920225+0000","last_peered":"2023-10-30T13:10:35.920225+0000","last_clean":"2023-10-30T13:10:35.920225+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:10:35.920225+0000","last_undegraded":"2023-10-30T13:10:35.920225+0000","last_fullsized":"2023-10-30T13:10:35.920225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:02.812352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758296+0000","last_change":"2023-10-30T13:10:34.344864+0000","last_active":"2023-10-30T13:10:35.758296+0000","last_peered":"2023-10-30T13:10:35.758296+0000","last_clean":"2023-10-30T13:10:35.758296+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:10:35.758296+0000","last_undegraded":"2023-10-30T13:10:35.758296+0000","last_fullsized":"2023-10-30T13:10:35.758296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:13.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923616+0000","last_change":"2023-10-30T13:10:34.364069+0000","last_active":"2023-10-30T13:10:35.923616+0000","last_peered":"2023-10-30T13:10:35.923616+0000","last_clean":"2023-10-30T13:10:35.923616+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:10:35.923616+0000","last_undegraded":"2023-10-30T13:10:35.923616+0000","last_fullsized":"2023-10-30T13:10:35.923616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:15.717336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.283789+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:10:37.283789+0000","last_peered":"2023-10-30T13:10:37.283789+0000","last_clean":"2023-10-30T13:10:37.283789+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:10:37.283789+0000","last_undegraded":"2023-10-30T13:10:37.283789+0000","last_fullsized":"2023-10-30T13:10:37.283789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920436+0000","last_change":"2023-10-30T13:10:34.360935+0000","last_active":"2023-10-30T13:10:35.920436+0000","last_peered":"2023-10-30T13:10:35.920436+0000","last_clean":"2023-10-30T13:10:35.920436+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:10:35.920436+0000","last_undegraded":"2023-10-30T13:10:35.920436+0000","last_fullsized":"2023-10-30T13:10:35.920436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:58.639753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.330349+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:10:37.330349+0000","last_peered":"2023-10-30T13:10:37.330349+0000","last_clean":"2023-10-30T13:10:37.330349+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:10:37.330349+0000","last_undegraded":"2023-10-30T13:10:37.330349+0000","last_fullsized":"2023-10-30T13:10:37.330349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318706+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:10:35.318706+0000","last_peered":"2023-10-30T13:10:35.318706+0000","last_clean":"2023-10-30T13:10:35.318706+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:10:35.318706+0000","last_undegraded":"2023-10-30T13:10:35.318706+0000","last_fullsized":"2023-10-30T13:10:35.318706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318630+0000","last_change":"2023-10-30T13:10:34.329935+0000","last_active":"2023-10-30T13:10:35.318630+0000","last_peered":"2023-10-30T13:10:35.318630+0000","last_clean":"2023-10-30T13:10:35.318630+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:10:35.318630+0000","last_undegraded":"2023-10-30T13:10:35.318630+0000","last_fullsized":"2023-10-30T13:10:35.318630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:07.795163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322575+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:10:35.322575+0000","last_peered":"2023-10-30T13:10:35.322575+0000","last_clean":"2023-10-30T13:10:35.322575+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:10:35.322575+0000","last_undegraded":"2023-10-30T13:10:35.322575+0000","last_fullsized":"2023-10-30T13:10:35.322575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919125+0000","last_change":"2023-10-30T13:10:34.331351+0000","last_active":"2023-10-30T13:10:35.919125+0000","last_peered":"2023-10-30T13:10:35.919125+0000","last_clean":"2023-10-30T13:10:35.919125+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:10:35.919125+0000","last_undegraded":"2023-10-30T13:10:35.919125+0000","last_fullsized":"2023-10-30T13:10:35.919125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:27.864073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759011+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:10:35.759011+0000","last_peered":"2023-10-30T13:10:35.759011+0000","last_clean":"2023-10-30T13:10:35.759011+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:10:35.759011+0000","last_undegraded":"2023-10-30T13:10:35.759011+0000","last_fullsized":"2023-10-30T13:10:35.759011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323392+0000","last_change":"2023-10-30T13:10:34.371882+0000","last_active":"2023-10-30T13:10:35.323392+0000","last_peered":"2023-10-30T13:10:35.323392+0000","last_clean":"2023-10-30T13:10:35.323392+0000","last_became_active":"2023-10-30T13:10:34.371138+0000","last_became_peered":"2023-10-30T13:10:34.371138+0000","last_unstale":"2023-10-30T13:10:35.323392+0000","last_undegraded":"2023-10-30T13:10:35.323392+0000","last_fullsized":"2023-10-30T13:10:35.323392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:16.287528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323317+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:10:35.323317+0000","last_peered":"2023-10-30T13:10:35.323317+0000","last_clean":"2023-10-30T13:10:35.323317+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:10:35.323317+0000","last_undegraded":"2023-10-30T13:10:35.323317+0000","last_fullsized":"2023-10-30T13:10:35.323317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920254+0000","last_change":"2023-10-30T13:10:34.335243+0000","last_active":"2023-10-30T13:10:35.920254+0000","last_peered":"2023-10-30T13:10:35.920254+0000","last_clean":"2023-10-30T13:10:35.920254+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:10:35.920254+0000","last_undegraded":"2023-10-30T13:10:35.920254+0000","last_fullsized":"2023-10-30T13:10:35.920254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:02.970675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.694619+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:10:39.694619+0000","last_peered":"2023-10-30T13:10:39.694619+0000","last_clean":"2023-10-30T13:10:39.694619+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:10:39.694619+0000","last_undegraded":"2023-10-30T13:10:39.694619+0000","last_fullsized":"2023-10-30T13:10:39.694619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924235+0000","last_change":"2023-10-30T13:10:34.381099+0000","last_active":"2023-10-30T13:10:35.924235+0000","last_peered":"2023-10-30T13:10:35.924235+0000","last_clean":"2023-10-30T13:10:35.924235+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:10:35.924235+0000","last_undegraded":"2023-10-30T13:10:35.924235+0000","last_fullsized":"2023-10-30T13:10:35.924235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:16.694101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.902621+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:10:39.902621+0000","last_peered":"2023-10-30T13:10:39.902621+0000","last_clean":"2023-10-30T13:10:39.902621+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:10:39.902621+0000","last_undegraded":"2023-10-30T13:10:39.902621+0000","last_fullsized":"2023-10-30T13:10:39.902621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322690+0000","last_change":"2023-10-30T13:10:34.372219+0000","last_active":"2023-10-30T13:10:35.322690+0000","last_peered":"2023-10-30T13:10:35.322690+0000","last_clean":"2023-10-30T13:10:35.322690+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:10:35.322690+0000","last_undegraded":"2023-10-30T13:10:35.322690+0000","last_fullsized":"2023-10-30T13:10:35.322690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:38.329952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.694091+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:10:39.694091+0000","last_peered":"2023-10-30T13:10:39.694091+0000","last_clean":"2023-10-30T13:10:39.694091+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:10:39.694091+0000","last_undegraded":"2023-10-30T13:10:39.694091+0000","last_fullsized":"2023-10-30T13:10:39.694091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920288+0000","last_change":"2023-10-30T13:10:34.368736+0000","last_active":"2023-10-30T13:10:35.920288+0000","last_peered":"2023-10-30T13:10:35.920288+0000","last_clean":"2023-10-30T13:10:35.920288+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:10:35.920288+0000","last_undegraded":"2023-10-30T13:10:35.920288+0000","last_fullsized":"2023-10-30T13:10:35.920288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:11.128769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323758+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:10:35.323758+0000","last_peered":"2023-10-30T13:10:35.323758+0000","last_clean":"2023-10-30T13:10:35.323758+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:10:35.323758+0000","last_undegraded":"2023-10-30T13:10:35.323758+0000","last_fullsized":"2023-10-30T13:10:35.323758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322654+0000","last_change":"2023-10-30T13:10:34.376512+0000","last_active":"2023-10-30T13:10:35.322654+0000","last_peered":"2023-10-30T13:10:35.322654+0000","last_clean":"2023-10-30T13:10:35.322654+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:10:35.322654+0000","last_undegraded":"2023-10-30T13:10:35.322654+0000","last_fullsized":"2023-10-30T13:10:35.322654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.856690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758265+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:10:35.758265+0000","last_peered":"2023-10-30T13:10:35.758265+0000","last_clean":"2023-10-30T13:10:35.758265+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:10:35.758265+0000","last_undegraded":"2023-10-30T13:10:35.758265+0000","last_fullsized":"2023-10-30T13:10:35.758265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758371+0000","last_change":"2023-10-30T13:10:34.344863+0000","last_active":"2023-10-30T13:10:35.758371+0000","last_peered":"2023-10-30T13:10:35.758371+0000","last_clean":"2023-10-30T13:10:35.758371+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:10:35.758371+0000","last_undegraded":"2023-10-30T13:10:35.758371+0000","last_fullsized":"2023-10-30T13:10:35.758371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:25.199813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.694395+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:10:39.694395+0000","last_peered":"2023-10-30T13:10:39.694395+0000","last_clean":"2023-10-30T13:10:39.694395+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:10:39.694395+0000","last_undegraded":"2023-10-30T13:10:39.694395+0000","last_fullsized":"2023-10-30T13:10:39.694395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323513+0000","last_change":"2023-10-30T13:10:34.371813+0000","last_active":"2023-10-30T13:10:35.323513+0000","last_peered":"2023-10-30T13:10:35.323513+0000","last_clean":"2023-10-30T13:10:35.323513+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:10:35.323513+0000","last_undegraded":"2023-10-30T13:10:35.323513+0000","last_fullsized":"2023-10-30T13:10:35.323513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:58.719747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.294675+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:10:37.294675+0000","last_peered":"2023-10-30T13:10:37.294675+0000","last_clean":"2023-10-30T13:10:37.294675+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:10:37.294675+0000","last_undegraded":"2023-10-30T13:10:37.294675+0000","last_fullsized":"2023-10-30T13:10:37.294675+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322746+0000","last_change":"2023-10-30T13:10:34.375164+0000","last_active":"2023-10-30T13:10:35.322746+0000","last_peered":"2023-10-30T13:10:35.322746+0000","last_clean":"2023-10-30T13:10:35.322746+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:10:35.322746+0000","last_undegraded":"2023-10-30T13:10:35.322746+0000","last_fullsized":"2023-10-30T13:10:35.322746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:05.305348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.692714+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:10:39.692714+0000","last_peered":"2023-10-30T13:10:39.692714+0000","last_clean":"2023-10-30T13:10:39.692714+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:10:39.692714+0000","last_undegraded":"2023-10-30T13:10:39.692714+0000","last_fullsized":"2023-10-30T13:10:39.692714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323211+0000","last_change":"2023-10-30T13:10:34.382785+0000","last_active":"2023-10-30T13:10:35.323211+0000","last_peered":"2023-10-30T13:10:35.323211+0000","last_clean":"2023-10-30T13:10:35.323211+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:10:35.323211+0000","last_undegraded":"2023-10-30T13:10:35.323211+0000","last_fullsized":"2023-10-30T13:10:35.323211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:50.203770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.291570+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:10:37.291570+0000","last_peered":"2023-10-30T13:10:37.291570+0000","last_clean":"2023-10-30T13:10:37.291570+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:10:37.291570+0000","last_undegraded":"2023-10-30T13:10:37.291570+0000","last_fullsized":"2023-10-30T13:10:37.291570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324805+0000","last_change":"2023-10-30T13:10:34.342901+0000","last_active":"2023-10-30T13:10:35.324805+0000","last_peered":"2023-10-30T13:10:35.324805+0000","last_clean":"2023-10-30T13:10:35.324805+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:10:35.324805+0000","last_undegraded":"2023-10-30T13:10:35.324805+0000","last_fullsized":"2023-10-30T13:10:35.324805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:27.227907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919963+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:10:35.919963+0000","last_peered":"2023-10-30T13:10:35.919963+0000","last_clean":"2023-10-30T13:10:35.919963+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:10:35.919963+0000","last_undegraded":"2023-10-30T13:10:35.919963+0000","last_fullsized":"2023-10-30T13:10:35.919963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323034+0000","last_change":"2023-10-30T13:10:34.373854+0000","last_active":"2023-10-30T13:10:35.323034+0000","last_peered":"2023-10-30T13:10:35.323034+0000","last_clean":"2023-10-30T13:10:35.323034+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:10:35.323034+0000","last_undegraded":"2023-10-30T13:10:35.323034+0000","last_fullsized":"2023-10-30T13:10:35.323034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:12.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:10:34.334682+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:10:34.334682+0000","last_peered":"2023-10-30T13:10:34.334682+0000","last_clean":"2023-10-30T13:10:34.334682+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:10:34.334682+0000","last_undegraded":"2023-10-30T13:10:34.334682+0000","last_fullsized":"2023-10-30T13:10:34.334682+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324107+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:10:35.324107+0000","last_peered":"2023-10-30T13:10:35.324107+0000","last_clean":"2023-10-30T13:10:35.324107+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:10:35.324107+0000","last_undegraded":"2023-10-30T13:10:35.324107+0000","last_fullsized":"2023-10-30T13:10:35.324107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318835+0000","last_change":"2023-10-30T13:10:34.348470+0000","last_active":"2023-10-30T13:10:35.318835+0000","last_peered":"2023-10-30T13:10:35.318835+0000","last_clean":"2023-10-30T13:10:35.318835+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:10:35.318835+0000","last_undegraded":"2023-10-30T13:10:35.318835+0000","last_fullsized":"2023-10-30T13:10:35.318835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:24.818359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321505+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:10:35.321505+0000","last_peered":"2023-10-30T13:10:35.321505+0000","last_clean":"2023-10-30T13:10:35.321505+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:10:35.321505+0000","last_undegraded":"2023-10-30T13:10:35.321505+0000","last_fullsized":"2023-10-30T13:10:35.321505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919520+0000","last_change":"2023-10-30T13:10:34.359740+0000","last_active":"2023-10-30T13:10:35.919520+0000","last_peered":"2023-10-30T13:10:35.919520+0000","last_clean":"2023-10-30T13:10:35.919520+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:10:35.919520+0000","last_undegraded":"2023-10-30T13:10:35.919520+0000","last_fullsized":"2023-10-30T13:10:35.919520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:05:25.883485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919976+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:10:35.919976+0000","last_peered":"2023-10-30T13:10:35.919976+0000","last_clean":"2023-10-30T13:10:35.919976+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:10:35.919976+0000","last_undegraded":"2023-10-30T13:10:35.919976+0000","last_fullsized":"2023-10-30T13:10:35.919976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919216+0000","last_change":"2023-10-30T13:10:34.332955+0000","last_active":"2023-10-30T13:10:35.919216+0000","last_peered":"2023-10-30T13:10:35.919216+0000","last_clean":"2023-10-30T13:10:35.919216+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:10:35.919216+0000","last_undegraded":"2023-10-30T13:10:35.919216+0000","last_fullsized":"2023-10-30T13:10:35.919216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:05.776190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323974+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:10:35.323974+0000","last_peered":"2023-10-30T13:10:35.323974+0000","last_clean":"2023-10-30T13:10:35.323974+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:10:35.323974+0000","last_undegraded":"2023-10-30T13:10:35.323974+0000","last_fullsized":"2023-10-30T13:10:35.323974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923916+0000","last_change":"2023-10-30T13:10:34.362313+0000","last_active":"2023-10-30T13:10:35.923916+0000","last_peered":"2023-10-30T13:10:35.923916+0000","last_clean":"2023-10-30T13:10:35.923916+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:10:35.923916+0000","last_undegraded":"2023-10-30T13:10:35.923916+0000","last_fullsized":"2023-10-30T13:10:35.923916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:45.213916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919650+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:10:35.919650+0000","last_peered":"2023-10-30T13:10:35.919650+0000","last_clean":"2023-10-30T13:10:35.919650+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:10:35.919650+0000","last_undegraded":"2023-10-30T13:10:35.919650+0000","last_fullsized":"2023-10-30T13:10:35.919650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323629+0000","last_change":"2023-10-30T13:10:34.348653+0000","last_active":"2023-10-30T13:10:35.323629+0000","last_peered":"2023-10-30T13:10:35.323629+0000","last_clean":"2023-10-30T13:10:35.323629+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:10:35.323629+0000","last_undegraded":"2023-10-30T13:10:35.323629+0000","last_fullsized":"2023-10-30T13:10:35.323629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:54:00.948062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323802+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:10:35.323802+0000","last_peered":"2023-10-30T13:10:35.323802+0000","last_clean":"2023-10-30T13:10:35.323802+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:10:35.323802+0000","last_undegraded":"2023-10-30T13:10:35.323802+0000","last_fullsized":"2023-10-30T13:10:35.323802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919933+0000","last_change":"2023-10-30T13:10:34.332001+0000","last_active":"2023-10-30T13:10:35.919933+0000","last_peered":"2023-10-30T13:10:35.919933+0000","last_clean":"2023-10-30T13:10:35.919933+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:10:35.919933+0000","last_undegraded":"2023-10-30T13:10:35.919933+0000","last_fullsized":"2023-10-30T13:10:35.919933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:16.341054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920027+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:10:35.920027+0000","last_peered":"2023-10-30T13:10:35.920027+0000","last_clean":"2023-10-30T13:10:35.920027+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:10:35.920027+0000","last_undegraded":"2023-10-30T13:10:35.920027+0000","last_fullsized":"2023-10-30T13:10:35.920027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319328+0000","last_change":"2023-10-30T13:10:34.369845+0000","last_active":"2023-10-30T13:10:35.319328+0000","last_peered":"2023-10-30T13:10:35.319328+0000","last_clean":"2023-10-30T13:10:35.319328+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:10:35.319328+0000","last_undegraded":"2023-10-30T13:10:35.319328+0000","last_fullsized":"2023-10-30T13:10:35.319328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:30.468845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.897907+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:10:39.897907+0000","last_peered":"2023-10-30T13:10:39.897907+0000","last_clean":"2023-10-30T13:10:39.897907+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:10:39.897907+0000","last_undegraded":"2023-10-30T13:10:39.897907+0000","last_fullsized":"2023-10-30T13:10:39.897907+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920190+0000","last_change":"2023-10-30T13:10:34.323336+0000","last_active":"2023-10-30T13:10:35.920190+0000","last_peered":"2023-10-30T13:10:35.920190+0000","last_clean":"2023-10-30T13:10:35.920190+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:10:35.920190+0000","last_undegraded":"2023-10-30T13:10:35.920190+0000","last_fullsized":"2023-10-30T13:10:35.920190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:21.642241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321410+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:10:35.321410+0000","last_peered":"2023-10-30T13:10:35.321410+0000","last_clean":"2023-10-30T13:10:35.321410+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:10:35.321410+0000","last_undegraded":"2023-10-30T13:10:35.321410+0000","last_fullsized":"2023-10-30T13:10:35.321410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923797+0000","last_change":"2023-10-30T13:10:34.363313+0000","last_active":"2023-10-30T13:10:35.923797+0000","last_peered":"2023-10-30T13:10:35.923797+0000","last_clean":"2023-10-30T13:10:35.923797+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:10:35.923797+0000","last_undegraded":"2023-10-30T13:10:35.923797+0000","last_fullsized":"2023-10-30T13:10:35.923797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:13.027610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919211+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:10:35.919211+0000","last_peered":"2023-10-30T13:10:35.919211+0000","last_clean":"2023-10-30T13:10:35.919211+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:10:35.919211+0000","last_undegraded":"2023-10-30T13:10:35.919211+0000","last_fullsized":"2023-10-30T13:10:35.919211+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318724+0000","last_change":"2023-10-30T13:10:34.330677+0000","last_active":"2023-10-30T13:10:35.318724+0000","last_peered":"2023-10-30T13:10:35.318724+0000","last_clean":"2023-10-30T13:10:35.318724+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:10:35.318724+0000","last_undegraded":"2023-10-30T13:10:35.318724+0000","last_fullsized":"2023-10-30T13:10:35.318724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:01.360813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919383+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:10:35.919383+0000","last_peered":"2023-10-30T13:10:35.919383+0000","last_clean":"2023-10-30T13:10:35.919383+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:10:35.919383+0000","last_undegraded":"2023-10-30T13:10:35.919383+0000","last_fullsized":"2023-10-30T13:10:35.919383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319182+0000","last_change":"2023-10-30T13:10:34.331117+0000","last_active":"2023-10-30T13:10:35.319182+0000","last_peered":"2023-10-30T13:10:35.319182+0000","last_clean":"2023-10-30T13:10:35.319182+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:10:35.319182+0000","last_undegraded":"2023-10-30T13:10:35.319182+0000","last_fullsized":"2023-10-30T13:10:35.319182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:56.948470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.697876+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:10:39.697876+0000","last_peered":"2023-10-30T13:10:39.697876+0000","last_clean":"2023-10-30T13:10:39.697876+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:10:39.697876+0000","last_undegraded":"2023-10-30T13:10:39.697876+0000","last_fullsized":"2023-10-30T13:10:39.697876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758411+0000","last_change":"2023-10-30T13:10:34.365315+0000","last_active":"2023-10-30T13:10:35.758411+0000","last_peered":"2023-10-30T13:10:35.758411+0000","last_clean":"2023-10-30T13:10:35.758411+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:10:35.758411+0000","last_undegraded":"2023-10-30T13:10:35.758411+0000","last_fullsized":"2023-10-30T13:10:35.758411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:37.078775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323303+0000","last_change":"2023-10-30T13:10:34.366044+0000","last_active":"2023-10-30T13:10:35.323303+0000","last_peered":"2023-10-30T13:10:35.323303+0000","last_clean":"2023-10-30T13:10:35.323303+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:10:35.323303+0000","last_undegraded":"2023-10-30T13:10:35.323303+0000","last_fullsized":"2023-10-30T13:10:35.323303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:31.831020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.697453+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:10:39.697453+0000","last_peered":"2023-10-30T13:10:39.697453+0000","last_clean":"2023-10-30T13:10:39.697453+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:10:39.697453+0000","last_undegraded":"2023-10-30T13:10:39.697453+0000","last_fullsized":"2023-10-30T13:10:39.697453+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324679+0000","last_change":"2023-10-30T13:10:34.369814+0000","last_active":"2023-10-30T13:10:35.324679+0000","last_peered":"2023-10-30T13:10:35.324679+0000","last_clean":"2023-10-30T13:10:35.324679+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:10:35.324679+0000","last_undegraded":"2023-10-30T13:10:35.324679+0000","last_fullsized":"2023-10-30T13:10:35.324679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:26.879717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.697444+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:10:39.697444+0000","last_peered":"2023-10-30T13:10:39.697444+0000","last_clean":"2023-10-30T13:10:39.697444+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:10:39.697444+0000","last_undegraded":"2023-10-30T13:10:39.697444+0000","last_fullsized":"2023-10-30T13:10:39.697444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919435+0000","last_change":"2023-10-30T13:10:34.344766+0000","last_active":"2023-10-30T13:10:35.919435+0000","last_peered":"2023-10-30T13:10:35.919435+0000","last_clean":"2023-10-30T13:10:35.919435+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:10:35.919435+0000","last_undegraded":"2023-10-30T13:10:35.919435+0000","last_fullsized":"2023-10-30T13:10:35.919435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.073346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.283829+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:10:37.283829+0000","last_peered":"2023-10-30T13:10:37.283829+0000","last_clean":"2023-10-30T13:10:37.283829+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:10:37.283829+0000","last_undegraded":"2023-10-30T13:10:37.283829+0000","last_fullsized":"2023-10-30T13:10:37.283829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319357+0000","last_change":"2023-10-30T13:10:34.354305+0000","last_active":"2023-10-30T13:10:35.319357+0000","last_peered":"2023-10-30T13:10:35.319357+0000","last_clean":"2023-10-30T13:10:35.319357+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:10:35.319357+0000","last_undegraded":"2023-10-30T13:10:35.319357+0000","last_fullsized":"2023-10-30T13:10:35.319357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:53.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.696166+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:10:39.696166+0000","last_peered":"2023-10-30T13:10:39.696166+0000","last_clean":"2023-10-30T13:10:39.696166+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:10:39.696166+0000","last_undegraded":"2023-10-30T13:10:39.696166+0000","last_fullsized":"2023-10-30T13:10:39.696166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6870000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6469999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6840000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.51}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2569999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9159999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.117}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":806912,"data_stored":660885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:10:45.479 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:10:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e8217640 1 Processor -- start 2023-10-30T13:10:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e8217640 1 -- start start 2023-10-30T13:10:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e8217640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56e015d6f0 0x7f56e015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e8217640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e00cb800 con 0x7f56e00cf160 2023-10-30T13:10:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e8217640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e00cb330 con 0x7f56e01538a0 2023-10-30T13:10:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e8217640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e00cad70 con 0x7f56e015d6f0 2023-10-30T13:10:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e67b4640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f56e00cf160 0x7f56e0164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39410/0 (socket says 172.21.15.148:39410) 2023-10-30T13:10:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e67b4640 1 -- 172.21.15.148:0/1533978679 learned_addr learned my addr 172.21.15.148:0/1533978679 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e57b2640 1 --2- 172.21.15.148:0/1533978679 >> v2:172.21.15.150:3300/0 conn(0x7f56e015d6f0 0x7f56e015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e4fb1640 1 -- 172.21.15.148:0/1533978679 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374038635 0 0) 0x7f56e00cb800 con 0x7f56e00cf160 2023-10-30T13:10:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e4fb1640 1 -- 172.21.15.148:0/1533978679 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc003530 con 0x7f56e00cf160 2023-10-30T13:10:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e4fb1640 1 -- 172.21.15.148:0/1533978679 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3652135494 0 0) 0x7f56cc003530 con 0x7f56e00cf160 2023-10-30T13:10:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e57b2640 1 -- 172.21.15.148:0/1533978679 >> v1:172.21.15.150:6789/0 conn(0x7f56e01538a0 legacy=0x7f56e00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e57b2640 1 -- 172.21.15.148:0/1533978679 >> v1:172.21.15.148:6789/0 conn(0x7f56e00cf160 legacy=0x7f56e0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.656+0000 7f56e57b2640 1 -- 172.21.15.148:0/1533978679 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e00cb050 con 0x7f56e015d6f0 2023-10-30T13:10:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e57b2640 1 --2- 172.21.15.148:0/1533978679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e015d6f0 0x7f56e015a090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56d00097c0 tx=0x7f56d001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fd3dc6f9d0b6aad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e4fb1640 1 -- 172.21.15.148:0/1533978679 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8001a00 con 0x7f56e015d6f0 2023-10-30T13:10:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e4fb1640 1 -- 172.21.15.148:0/1533978679 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d00a8610 con 0x7f56e015d6f0 2023-10-30T13:10:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e4fb1640 1 -- 172.21.15.148:0/1533978679 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d00a8980 con 0x7f56e015d6f0 2023-10-30T13:10:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e8217640 1 -- 172.21.15.148:0/1533978679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e015d6f0 msgr2=0x7f56e015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e8217640 1 --2- 172.21.15.148:0/1533978679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e015d6f0 0x7f56e015a090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56d00097c0 tx=0x7f56d001eb90 comp rx=0 tx=0).stop 2023-10-30T13:10:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e8217640 1 -- 172.21.15.148:0/1533978679 shutdown_connections 2023-10-30T13:10:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e8217640 1 --2- 172.21.15.148:0/1533978679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e015d6f0 0x7f56e015a090 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e8217640 1 -- 172.21.15.148:0/1533978679 >> 172.21.15.148:0/1533978679 conn(0x7f56e006bfe0 msgr2=0x7f56e006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e8217640 1 -- 172.21.15.148:0/1533978679 shutdown_connections 2023-10-30T13:10:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.660+0000 7f56e8217640 1 -- 172.21.15.148:0/1533978679 wait complete. 2023-10-30T13:10:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 Processor -- start 2023-10-30T13:10:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 -- start start 2023-10-30T13:10:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56e00cf160 0x7f56e00cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56e01538a0 0x7f56e00cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e015d6f0 0x7f56e014cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00cd2b0 con 0x7f56e01538a0 2023-10-30T13:10:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e014e880 con 0x7f56e015d6f0 2023-10-30T13:10:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e0158ab0 con 0x7f56e00cf160 2023-10-30T13:10:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e57b2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56e01538a0 0x7f56e00cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e57b2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56e01538a0 0x7f56e00cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42094/0 (socket says 172.21.15.148:42094) 2023-10-30T13:10:45.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e57b2640 1 -- 172.21.15.148:0/1572658862 learned_addr learned my addr 172.21.15.148:0/1572658862 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:45.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e67b4640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e015d6f0 0x7f56e014cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e5fb3640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56e00cf160 0x7f56e00cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e57b2640 1 -- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56e00cf160 msgr2=0x7f56e00cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e57b2640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56e00cf160 0x7f56e00cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e57b2640 1 -- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e015d6f0 msgr2=0x7f56e014cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e57b2640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e015d6f0 0x7f56e014cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e57b2640 1 -- 172.21.15.148:0/1572658862 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d0009420 con 0x7f56e01538a0 2023-10-30T13:10:45.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e5fb3640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56e00cf160 0x7f56e00cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:45.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e57b2640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56e01538a0 0x7f56e00cd990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56d0000c10 tx=0x7f56d001b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:45.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56d67fc640 1 -- 172.21.15.148:0/1572658862 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d00a9d20 con 0x7f56e01538a0 2023-10-30T13:10:45.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56d67fc640 1 -- 172.21.15.148:0/1572658862 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d00a9e80 con 0x7f56e01538a0 2023-10-30T13:10:45.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e0157f00 con 0x7f56e01538a0 2023-10-30T13:10:45.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56d67fc640 1 -- 172.21.15.148:0/1572658862 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0003820 con 0x7f56e01538a0 2023-10-30T13:10:45.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.664+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e0158180 con 0x7f56e01538a0 2023-10-30T13:10:45.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.668+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e004e170 con 0x7f56e01538a0 2023-10-30T13:10:45.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.668+0000 7f56d67fc640 1 -- 172.21.15.148:0/1572658862 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56d00b7040 con 0x7f56e01538a0 2023-10-30T13:10:45.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.668+0000 7f56d67fc640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f56c40664a0 0x7f56c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:45.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.668+0000 7f56d67fc640 1 -- 172.21.15.148:0/1572658862 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56d0121670 con 0x7f56e01538a0 2023-10-30T13:10:45.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.668+0000 7f56e5fb3640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f56c40664a0 0x7f56c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:45.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.668+0000 7f56e5fb3640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f56c40664a0 0x7f56c4068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f56dc000f70 tx=0x7f56dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:45.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.672+0000 7f56d67fc640 1 -- 172.21.15.148:0/1572658862 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56d00eeb70 con 0x7f56e01538a0 2023-10-30T13:10:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.824+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56e0159b70 con 0x7f56c40664a0 2023-10-30T13:10:45.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56d67fc640 1 -- 172.21.15.148:0/1572658862 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264490 (secure 0 0 0) 0x7f56dc0013f0 con 0x7f56c40664a0 2023-10-30T13:10:45.849 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:45.851 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:10:45.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f56c40664a0 msgr2=0x7f56c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f56c40664a0 0x7f56c4068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f56dc000f70 tx=0x7f56dc009040 comp rx=0 tx=0).stop 2023-10-30T13:10:45.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56e01538a0 msgr2=0x7f56e00cd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:45.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56e01538a0 0x7f56e00cd990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56d0000c10 tx=0x7f56d001b930 comp rx=0 tx=0).stop 2023-10-30T13:10:45.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 shutdown_connections 2023-10-30T13:10:45.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f56c40664a0 0x7f56c4068960 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56e01538a0 0x7f56e00cd990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e015d6f0 0x7f56e014cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 --2- 172.21.15.148:0/1572658862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f56e00cf160 0x7f56e00cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:45.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 >> 172.21.15.148:0/1572658862 conn(0x7f56e006bfe0 msgr2=0x7f56e015b640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:45.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 shutdown_connections 2023-10-30T13:10:45.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:45.844+0000 7f56e8217640 1 -- 172.21.15.148:0/1572658862 wait complete. 2023-10-30T13:10:45.869 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-30T13:10:44.263848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218948,"kb_used_data":2820,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755772,"statfs":{"total":773094113280,"available":772869910528,"internally_reserved":0,"allocated":2887680,"data_stored":1751721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.947746"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322509+0000","last_change":"2023-10-30T13:10:34.379031+0000","last_active":"2023-10-30T13:10:35.322509+0000","last_peered":"2023-10-30T13:10:35.322509+0000","last_clean":"2023-10-30T13:10:35.322509+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:10:35.322509+0000","last_undegraded":"2023-10-30T13:10:35.322509+0000","last_fullsized":"2023-10-30T13:10:35.322509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:32.529312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919896+0000","last_change":"2023-10-30T13:10:34.324308+0000","last_active":"2023-10-30T13:10:35.919896+0000","last_peered":"2023-10-30T13:10:35.919896+0000","last_clean":"2023-10-30T13:10:35.919896+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:10:35.919896+0000","last_undegraded":"2023-10-30T13:10:35.919896+0000","last_fullsized":"2023-10-30T13:10:35.919896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:21.885716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919373+0000","last_change":"2023-10-30T13:10:34.334122+0000","last_active":"2023-10-30T13:10:35.919373+0000","last_peered":"2023-10-30T13:10:35.919373+0000","last_clean":"2023-10-30T13:10:35.919373+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:10:35.919373+0000","last_undegraded":"2023-10-30T13:10:35.919373+0000","last_fullsized":"2023-10-30T13:10:35.919373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:55.045024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324579+0000","last_change":"2023-10-30T13:10:34.346795+0000","last_active":"2023-10-30T13:10:35.324579+0000","last_peered":"2023-10-30T13:10:35.324579+0000","last_clean":"2023-10-30T13:10:35.324579+0000","last_became_active":"2023-10-30T13:10:34.346079+0000","last_became_peered":"2023-10-30T13:10:34.346079+0000","last_unstale":"2023-10-30T13:10:35.324579+0000","last_undegraded":"2023-10-30T13:10:35.324579+0000","last_fullsized":"2023-10-30T13:10:35.324579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:09.546205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322339+0000","last_change":"2023-10-30T13:10:34.371670+0000","last_active":"2023-10-30T13:10:35.322339+0000","last_peered":"2023-10-30T13:10:35.322339+0000","last_clean":"2023-10-30T13:10:35.322339+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:10:35.322339+0000","last_undegraded":"2023-10-30T13:10:35.322339+0000","last_fullsized":"2023-10-30T13:10:35.322339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:03:54.773965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323062+0000","last_change":"2023-10-30T13:10:34.335752+0000","last_active":"2023-10-30T13:10:35.323062+0000","last_peered":"2023-10-30T13:10:35.323062+0000","last_clean":"2023-10-30T13:10:35.323062+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:10:35.323062+0000","last_undegraded":"2023-10-30T13:10:35.323062+0000","last_fullsized":"2023-10-30T13:10:35.323062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:33.278920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321759+0000","last_change":"2023-10-30T13:10:34.371817+0000","last_active":"2023-10-30T13:10:35.321759+0000","last_peered":"2023-10-30T13:10:35.321759+0000","last_clean":"2023-10-30T13:10:35.321759+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:10:35.321759+0000","last_undegraded":"2023-10-30T13:10:35.321759+0000","last_fullsized":"2023-10-30T13:10:35.321759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:33.797113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318581+0000","last_change":"2023-10-30T13:10:34.348205+0000","last_active":"2023-10-30T13:10:35.318581+0000","last_peered":"2023-10-30T13:10:35.318581+0000","last_clean":"2023-10-30T13:10:35.318581+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:10:35.318581+0000","last_undegraded":"2023-10-30T13:10:35.318581+0000","last_fullsized":"2023-10-30T13:10:35.318581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:57.492738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322864+0000","last_change":"2023-10-30T13:10:34.381052+0000","last_active":"2023-10-30T13:10:35.322864+0000","last_peered":"2023-10-30T13:10:35.322864+0000","last_clean":"2023-10-30T13:10:35.322864+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:10:35.322864+0000","last_undegraded":"2023-10-30T13:10:35.322864+0000","last_fullsized":"2023-10-30T13:10:35.322864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:16.697332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759029+0000","last_change":"2023-10-30T13:10:34.356932+0000","last_active":"2023-10-30T13:10:35.759029+0000","last_peered":"2023-10-30T13:10:35.759029+0000","last_clean":"2023-10-30T13:10:35.759029+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:10:35.759029+0000","last_undegraded":"2023-10-30T13:10:35.759029+0000","last_fullsized":"2023-10-30T13:10:35.759029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:54.091993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758774+0000","last_change":"2023-10-30T13:10:34.357358+0000","last_active":"2023-10-30T13:10:35.758774+0000","last_peered":"2023-10-30T13:10:35.758774+0000","last_clean":"2023-10-30T13:10:35.758774+0000","last_became_active":"2023-10-30T13:10:34.356466+0000","last_became_peered":"2023-10-30T13:10:34.356466+0000","last_unstale":"2023-10-30T13:10:35.758774+0000","last_undegraded":"2023-10-30T13:10:35.758774+0000","last_fullsized":"2023-10-30T13:10:35.758774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:06:24.029159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924241+0000","last_change":"2023-10-30T13:10:34.365161+0000","last_active":"2023-10-30T13:10:35.924241+0000","last_peered":"2023-10-30T13:10:35.924241+0000","last_clean":"2023-10-30T13:10:35.924241+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:10:35.924241+0000","last_undegraded":"2023-10-30T13:10:35.924241+0000","last_fullsized":"2023-10-30T13:10:35.924241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:51.721060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318828+0000","last_change":"2023-10-30T13:10:34.351901+0000","last_active":"2023-10-30T13:10:35.318828+0000","last_peered":"2023-10-30T13:10:35.318828+0000","last_clean":"2023-10-30T13:10:35.318828+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:10:35.318828+0000","last_undegraded":"2023-10-30T13:10:35.318828+0000","last_fullsized":"2023-10-30T13:10:35.318828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:11.576546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920191+0000","last_change":"2023-10-30T13:10:34.323358+0000","last_active":"2023-10-30T13:10:35.920191+0000","last_peered":"2023-10-30T13:10:35.920191+0000","last_clean":"2023-10-30T13:10:35.920191+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:10:35.920191+0000","last_undegraded":"2023-10-30T13:10:35.920191+0000","last_fullsized":"2023-10-30T13:10:35.920191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:03.178822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919173+0000","last_change":"2023-10-30T13:10:34.360120+0000","last_active":"2023-10-30T13:10:35.919173+0000","last_peered":"2023-10-30T13:10:35.919173+0000","last_clean":"2023-10-30T13:10:35.919173+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:10:35.919173+0000","last_undegraded":"2023-10-30T13:10:35.919173+0000","last_fullsized":"2023-10-30T13:10:35.919173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:30.731746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322343+0000","last_change":"2023-10-30T13:10:34.333752+0000","last_active":"2023-10-30T13:10:35.322343+0000","last_peered":"2023-10-30T13:10:35.322343+0000","last_clean":"2023-10-30T13:10:35.322343+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:10:35.322343+0000","last_undegraded":"2023-10-30T13:10:35.322343+0000","last_fullsized":"2023-10-30T13:10:35.322343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:45.390603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324451+0000","last_change":"2023-10-30T13:10:34.371931+0000","last_active":"2023-10-30T13:10:35.324451+0000","last_peered":"2023-10-30T13:10:35.324451+0000","last_clean":"2023-10-30T13:10:35.324451+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:10:35.324451+0000","last_undegraded":"2023-10-30T13:10:35.324451+0000","last_fullsized":"2023-10-30T13:10:35.324451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:51.514990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919968+0000","last_change":"2023-10-30T13:10:34.352747+0000","last_active":"2023-10-30T13:10:35.919968+0000","last_peered":"2023-10-30T13:10:35.919968+0000","last_clean":"2023-10-30T13:10:35.919968+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:10:35.919968+0000","last_undegraded":"2023-10-30T13:10:35.919968+0000","last_fullsized":"2023-10-30T13:10:35.919968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:21.543351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323598+0000","last_change":"2023-10-30T13:10:34.376135+0000","last_active":"2023-10-30T13:10:35.323598+0000","last_peered":"2023-10-30T13:10:35.323598+0000","last_clean":"2023-10-30T13:10:35.323598+0000","last_became_active":"2023-10-30T13:10:34.375497+0000","last_became_peered":"2023-10-30T13:10:34.375497+0000","last_unstale":"2023-10-30T13:10:35.323598+0000","last_undegraded":"2023-10-30T13:10:35.323598+0000","last_fullsized":"2023-10-30T13:10:35.323598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:14.816946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758767+0000","last_change":"2023-10-30T13:10:34.366575+0000","last_active":"2023-10-30T13:10:35.758767+0000","last_peered":"2023-10-30T13:10:35.758767+0000","last_clean":"2023-10-30T13:10:35.758767+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:10:35.758767+0000","last_undegraded":"2023-10-30T13:10:35.758767+0000","last_fullsized":"2023-10-30T13:10:35.758767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:23.540645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920318+0000","last_change":"2023-10-30T13:10:34.368761+0000","last_active":"2023-10-30T13:10:35.920318+0000","last_peered":"2023-10-30T13:10:35.920318+0000","last_clean":"2023-10-30T13:10:35.920318+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:10:35.920318+0000","last_undegraded":"2023-10-30T13:10:35.920318+0000","last_fullsized":"2023-10-30T13:10:35.920318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:32.861756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323430+0000","last_change":"2023-10-30T13:10:34.372881+0000","last_active":"2023-10-30T13:10:35.323430+0000","last_peered":"2023-10-30T13:10:35.323430+0000","last_clean":"2023-10-30T13:10:35.323430+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:10:35.323430+0000","last_undegraded":"2023-10-30T13:10:35.323430+0000","last_fullsized":"2023-10-30T13:10:35.323430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:23.855779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318722+0000","last_change":"2023-10-30T13:10:34.359359+0000","last_active":"2023-10-30T13:10:35.318722+0000","last_peered":"2023-10-30T13:10:35.318722+0000","last_clean":"2023-10-30T13:10:35.318722+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:10:35.318722+0000","last_undegraded":"2023-10-30T13:10:35.318722+0000","last_fullsized":"2023-10-30T13:10:35.318722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:08.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322387+0000","last_change":"2023-10-30T13:10:34.347431+0000","last_active":"2023-10-30T13:10:35.322387+0000","last_peered":"2023-10-30T13:10:35.322387+0000","last_clean":"2023-10-30T13:10:35.322387+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:10:35.322387+0000","last_undegraded":"2023-10-30T13:10:35.322387+0000","last_fullsized":"2023-10-30T13:10:35.322387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:25.094617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319296+0000","last_change":"2023-10-30T13:10:34.359717+0000","last_active":"2023-10-30T13:10:35.319296+0000","last_peered":"2023-10-30T13:10:35.319296+0000","last_clean":"2023-10-30T13:10:35.319296+0000","last_became_active":"2023-10-30T13:10:34.359063+0000","last_became_peered":"2023-10-30T13:10:34.359063+0000","last_unstale":"2023-10-30T13:10:35.319296+0000","last_undegraded":"2023-10-30T13:10:35.319296+0000","last_fullsized":"2023-10-30T13:10:35.319296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:42:22.525493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759209+0000","last_change":"2023-10-30T13:10:34.379878+0000","last_active":"2023-10-30T13:10:35.759209+0000","last_peered":"2023-10-30T13:10:35.759209+0000","last_clean":"2023-10-30T13:10:35.759209+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:10:35.759209+0000","last_undegraded":"2023-10-30T13:10:35.759209+0000","last_fullsized":"2023-10-30T13:10:35.759209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:42.280634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919210+0000","last_change":"2023-10-30T13:10:34.332020+0000","last_active":"2023-10-30T13:10:35.919210+0000","last_peered":"2023-10-30T13:10:35.919210+0000","last_clean":"2023-10-30T13:10:35.919210+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:10:35.919210+0000","last_undegraded":"2023-10-30T13:10:35.919210+0000","last_fullsized":"2023-10-30T13:10:35.919210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:53.057012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319283+0000","last_change":"2023-10-30T13:10:34.363154+0000","last_active":"2023-10-30T13:10:35.319283+0000","last_peered":"2023-10-30T13:10:35.319283+0000","last_clean":"2023-10-30T13:10:35.319283+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:10:35.319283+0000","last_undegraded":"2023-10-30T13:10:35.319283+0000","last_fullsized":"2023-10-30T13:10:35.319283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:04.359629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920381+0000","last_change":"2023-10-30T13:10:34.332160+0000","last_active":"2023-10-30T13:10:35.920381+0000","last_peered":"2023-10-30T13:10:35.920381+0000","last_clean":"2023-10-30T13:10:35.920381+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:10:35.920381+0000","last_undegraded":"2023-10-30T13:10:35.920381+0000","last_fullsized":"2023-10-30T13:10:35.920381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:28.128344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323753+0000","last_change":"2023-10-30T13:10:34.337617+0000","last_active":"2023-10-30T13:10:35.323753+0000","last_peered":"2023-10-30T13:10:35.323753+0000","last_clean":"2023-10-30T13:10:35.323753+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:10:35.323753+0000","last_undegraded":"2023-10-30T13:10:35.323753+0000","last_fullsized":"2023-10-30T13:10:35.323753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:10.287441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920225+0000","last_change":"2023-10-30T13:10:34.359681+0000","last_active":"2023-10-30T13:10:35.920225+0000","last_peered":"2023-10-30T13:10:35.920225+0000","last_clean":"2023-10-30T13:10:35.920225+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:10:35.920225+0000","last_undegraded":"2023-10-30T13:10:35.920225+0000","last_fullsized":"2023-10-30T13:10:35.920225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:02.812352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758296+0000","last_change":"2023-10-30T13:10:34.344864+0000","last_active":"2023-10-30T13:10:35.758296+0000","last_peered":"2023-10-30T13:10:35.758296+0000","last_clean":"2023-10-30T13:10:35.758296+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:10:35.758296+0000","last_undegraded":"2023-10-30T13:10:35.758296+0000","last_fullsized":"2023-10-30T13:10:35.758296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:13.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923616+0000","last_change":"2023-10-30T13:10:34.364069+0000","last_active":"2023-10-30T13:10:35.923616+0000","last_peered":"2023-10-30T13:10:35.923616+0000","last_clean":"2023-10-30T13:10:35.923616+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:10:35.923616+0000","last_undegraded":"2023-10-30T13:10:35.923616+0000","last_fullsized":"2023-10-30T13:10:35.923616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:15.717336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.283789+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:10:37.283789+0000","last_peered":"2023-10-30T13:10:37.283789+0000","last_clean":"2023-10-30T13:10:37.283789+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:10:37.283789+0000","last_undegraded":"2023-10-30T13:10:37.283789+0000","last_fullsized":"2023-10-30T13:10:37.283789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920436+0000","last_change":"2023-10-30T13:10:34.360935+0000","last_active":"2023-10-30T13:10:35.920436+0000","last_peered":"2023-10-30T13:10:35.920436+0000","last_clean":"2023-10-30T13:10:35.920436+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:10:35.920436+0000","last_undegraded":"2023-10-30T13:10:35.920436+0000","last_fullsized":"2023-10-30T13:10:35.920436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:58.639753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.330349+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:10:37.330349+0000","last_peered":"2023-10-30T13:10:37.330349+0000","last_clean":"2023-10-30T13:10:37.330349+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:10:37.330349+0000","last_undegraded":"2023-10-30T13:10:37.330349+0000","last_fullsized":"2023-10-30T13:10:37.330349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318706+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:10:35.318706+0000","last_peered":"2023-10-30T13:10:35.318706+0000","last_clean":"2023-10-30T13:10:35.318706+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:10:35.318706+0000","last_undegraded":"2023-10-30T13:10:35.318706+0000","last_fullsized":"2023-10-30T13:10:35.318706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318630+0000","last_change":"2023-10-30T13:10:34.329935+0000","last_active":"2023-10-30T13:10:35.318630+0000","last_peered":"2023-10-30T13:10:35.318630+0000","last_clean":"2023-10-30T13:10:35.318630+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:10:35.318630+0000","last_undegraded":"2023-10-30T13:10:35.318630+0000","last_fullsized":"2023-10-30T13:10:35.318630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:07.795163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322575+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:10:35.322575+0000","last_peered":"2023-10-30T13:10:35.322575+0000","last_clean":"2023-10-30T13:10:35.322575+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:10:35.322575+0000","last_undegraded":"2023-10-30T13:10:35.322575+0000","last_fullsized":"2023-10-30T13:10:35.322575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919125+0000","last_change":"2023-10-30T13:10:34.331351+0000","last_active":"2023-10-30T13:10:35.919125+0000","last_peered":"2023-10-30T13:10:35.919125+0000","last_clean":"2023-10-30T13:10:35.919125+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:10:35.919125+0000","last_undegraded":"2023-10-30T13:10:35.919125+0000","last_fullsized":"2023-10-30T13:10:35.919125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:27.864073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.759011+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:10:35.759011+0000","last_peered":"2023-10-30T13:10:35.759011+0000","last_clean":"2023-10-30T13:10:35.759011+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:10:35.759011+0000","last_undegraded":"2023-10-30T13:10:35.759011+0000","last_fullsized":"2023-10-30T13:10:35.759011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323392+0000","last_change":"2023-10-30T13:10:34.371882+0000","last_active":"2023-10-30T13:10:35.323392+0000","last_peered":"2023-10-30T13:10:35.323392+0000","last_clean":"2023-10-30T13:10:35.323392+0000","last_became_active":"2023-10-30T13:10:34.371138+0000","last_became_peered":"2023-10-30T13:10:34.371138+0000","last_unstale":"2023-10-30T13:10:35.323392+0000","last_undegraded":"2023-10-30T13:10:35.323392+0000","last_fullsized":"2023-10-30T13:10:35.323392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:16.287528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323317+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:10:35.323317+0000","last_peered":"2023-10-30T13:10:35.323317+0000","last_clean":"2023-10-30T13:10:35.323317+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:10:35.323317+0000","last_undegraded":"2023-10-30T13:10:35.323317+0000","last_fullsized":"2023-10-30T13:10:35.323317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920254+0000","last_change":"2023-10-30T13:10:34.335243+0000","last_active":"2023-10-30T13:10:35.920254+0000","last_peered":"2023-10-30T13:10:35.920254+0000","last_clean":"2023-10-30T13:10:35.920254+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:10:35.920254+0000","last_undegraded":"2023-10-30T13:10:35.920254+0000","last_fullsized":"2023-10-30T13:10:35.920254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:02.970675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.694619+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:10:39.694619+0000","last_peered":"2023-10-30T13:10:39.694619+0000","last_clean":"2023-10-30T13:10:39.694619+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:10:39.694619+0000","last_undegraded":"2023-10-30T13:10:39.694619+0000","last_fullsized":"2023-10-30T13:10:39.694619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.924235+0000","last_change":"2023-10-30T13:10:34.381099+0000","last_active":"2023-10-30T13:10:35.924235+0000","last_peered":"2023-10-30T13:10:35.924235+0000","last_clean":"2023-10-30T13:10:35.924235+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:10:35.924235+0000","last_undegraded":"2023-10-30T13:10:35.924235+0000","last_fullsized":"2023-10-30T13:10:35.924235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:16.694101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.902621+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:10:39.902621+0000","last_peered":"2023-10-30T13:10:39.902621+0000","last_clean":"2023-10-30T13:10:39.902621+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:10:39.902621+0000","last_undegraded":"2023-10-30T13:10:39.902621+0000","last_fullsized":"2023-10-30T13:10:39.902621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322690+0000","last_change":"2023-10-30T13:10:34.372219+0000","last_active":"2023-10-30T13:10:35.322690+0000","last_peered":"2023-10-30T13:10:35.322690+0000","last_clean":"2023-10-30T13:10:35.322690+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:10:35.322690+0000","last_undegraded":"2023-10-30T13:10:35.322690+0000","last_fullsized":"2023-10-30T13:10:35.322690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:38.329952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.694091+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:10:39.694091+0000","last_peered":"2023-10-30T13:10:39.694091+0000","last_clean":"2023-10-30T13:10:39.694091+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:10:39.694091+0000","last_undegraded":"2023-10-30T13:10:39.694091+0000","last_fullsized":"2023-10-30T13:10:39.694091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920288+0000","last_change":"2023-10-30T13:10:34.368736+0000","last_active":"2023-10-30T13:10:35.920288+0000","last_peered":"2023-10-30T13:10:35.920288+0000","last_clean":"2023-10-30T13:10:35.920288+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:10:35.920288+0000","last_undegraded":"2023-10-30T13:10:35.920288+0000","last_fullsized":"2023-10-30T13:10:35.920288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:11.128769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323758+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:10:35.323758+0000","last_peered":"2023-10-30T13:10:35.323758+0000","last_clean":"2023-10-30T13:10:35.323758+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:10:35.323758+0000","last_undegraded":"2023-10-30T13:10:35.323758+0000","last_fullsized":"2023-10-30T13:10:35.323758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322654+0000","last_change":"2023-10-30T13:10:34.376512+0000","last_active":"2023-10-30T13:10:35.322654+0000","last_peered":"2023-10-30T13:10:35.322654+0000","last_clean":"2023-10-30T13:10:35.322654+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:10:35.322654+0000","last_undegraded":"2023-10-30T13:10:35.322654+0000","last_fullsized":"2023-10-30T13:10:35.322654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.856690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758265+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:10:35.758265+0000","last_peered":"2023-10-30T13:10:35.758265+0000","last_clean":"2023-10-30T13:10:35.758265+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:10:35.758265+0000","last_undegraded":"2023-10-30T13:10:35.758265+0000","last_fullsized":"2023-10-30T13:10:35.758265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758371+0000","last_change":"2023-10-30T13:10:34.344863+0000","last_active":"2023-10-30T13:10:35.758371+0000","last_peered":"2023-10-30T13:10:35.758371+0000","last_clean":"2023-10-30T13:10:35.758371+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:10:35.758371+0000","last_undegraded":"2023-10-30T13:10:35.758371+0000","last_fullsized":"2023-10-30T13:10:35.758371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:25.199813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.694395+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:10:39.694395+0000","last_peered":"2023-10-30T13:10:39.694395+0000","last_clean":"2023-10-30T13:10:39.694395+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:10:39.694395+0000","last_undegraded":"2023-10-30T13:10:39.694395+0000","last_fullsized":"2023-10-30T13:10:39.694395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323513+0000","last_change":"2023-10-30T13:10:34.371813+0000","last_active":"2023-10-30T13:10:35.323513+0000","last_peered":"2023-10-30T13:10:35.323513+0000","last_clean":"2023-10-30T13:10:35.323513+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:10:35.323513+0000","last_undegraded":"2023-10-30T13:10:35.323513+0000","last_fullsized":"2023-10-30T13:10:35.323513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:58.719747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.294675+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:10:37.294675+0000","last_peered":"2023-10-30T13:10:37.294675+0000","last_clean":"2023-10-30T13:10:37.294675+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:10:37.294675+0000","last_undegraded":"2023-10-30T13:10:37.294675+0000","last_fullsized":"2023-10-30T13:10:37.294675+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.322746+0000","last_change":"2023-10-30T13:10:34.375164+0000","last_active":"2023-10-30T13:10:35.322746+0000","last_peered":"2023-10-30T13:10:35.322746+0000","last_clean":"2023-10-30T13:10:35.322746+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:10:35.322746+0000","last_undegraded":"2023-10-30T13:10:35.322746+0000","last_fullsized":"2023-10-30T13:10:35.322746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:05.305348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.692714+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:10:39.692714+0000","last_peered":"2023-10-30T13:10:39.692714+0000","last_clean":"2023-10-30T13:10:39.692714+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:10:39.692714+0000","last_undegraded":"2023-10-30T13:10:39.692714+0000","last_fullsized":"2023-10-30T13:10:39.692714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323211+0000","last_change":"2023-10-30T13:10:34.382785+0000","last_active":"2023-10-30T13:10:35.323211+0000","last_peered":"2023-10-30T13:10:35.323211+0000","last_clean":"2023-10-30T13:10:35.323211+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:10:35.323211+0000","last_undegraded":"2023-10-30T13:10:35.323211+0000","last_fullsized":"2023-10-30T13:10:35.323211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:50.203770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.291570+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:10:37.291570+0000","last_peered":"2023-10-30T13:10:37.291570+0000","last_clean":"2023-10-30T13:10:37.291570+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:10:37.291570+0000","last_undegraded":"2023-10-30T13:10:37.291570+0000","last_fullsized":"2023-10-30T13:10:37.291570+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324805+0000","last_change":"2023-10-30T13:10:34.342901+0000","last_active":"2023-10-30T13:10:35.324805+0000","last_peered":"2023-10-30T13:10:35.324805+0000","last_clean":"2023-10-30T13:10:35.324805+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:10:35.324805+0000","last_undegraded":"2023-10-30T13:10:35.324805+0000","last_fullsized":"2023-10-30T13:10:35.324805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:27.227907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919963+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:10:35.919963+0000","last_peered":"2023-10-30T13:10:35.919963+0000","last_clean":"2023-10-30T13:10:35.919963+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:10:35.919963+0000","last_undegraded":"2023-10-30T13:10:35.919963+0000","last_fullsized":"2023-10-30T13:10:35.919963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323034+0000","last_change":"2023-10-30T13:10:34.373854+0000","last_active":"2023-10-30T13:10:35.323034+0000","last_peered":"2023-10-30T13:10:35.323034+0000","last_clean":"2023-10-30T13:10:35.323034+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:10:35.323034+0000","last_undegraded":"2023-10-30T13:10:35.323034+0000","last_fullsized":"2023-10-30T13:10:35.323034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:12.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-30T13:10:34.334682+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:10:34.334682+0000","last_peered":"2023-10-30T13:10:34.334682+0000","last_clean":"2023-10-30T13:10:34.334682+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:10:34.334682+0000","last_undegraded":"2023-10-30T13:10:34.334682+0000","last_fullsized":"2023-10-30T13:10:34.334682+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324107+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:10:35.324107+0000","last_peered":"2023-10-30T13:10:35.324107+0000","last_clean":"2023-10-30T13:10:35.324107+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:10:35.324107+0000","last_undegraded":"2023-10-30T13:10:35.324107+0000","last_fullsized":"2023-10-30T13:10:35.324107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318835+0000","last_change":"2023-10-30T13:10:34.348470+0000","last_active":"2023-10-30T13:10:35.318835+0000","last_peered":"2023-10-30T13:10:35.318835+0000","last_clean":"2023-10-30T13:10:35.318835+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:10:35.318835+0000","last_undegraded":"2023-10-30T13:10:35.318835+0000","last_fullsized":"2023-10-30T13:10:35.318835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:24.818359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321505+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:10:35.321505+0000","last_peered":"2023-10-30T13:10:35.321505+0000","last_clean":"2023-10-30T13:10:35.321505+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:10:35.321505+0000","last_undegraded":"2023-10-30T13:10:35.321505+0000","last_fullsized":"2023-10-30T13:10:35.321505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919520+0000","last_change":"2023-10-30T13:10:34.359740+0000","last_active":"2023-10-30T13:10:35.919520+0000","last_peered":"2023-10-30T13:10:35.919520+0000","last_clean":"2023-10-30T13:10:35.919520+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:10:35.919520+0000","last_undegraded":"2023-10-30T13:10:35.919520+0000","last_fullsized":"2023-10-30T13:10:35.919520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:05:25.883485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919976+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:10:35.919976+0000","last_peered":"2023-10-30T13:10:35.919976+0000","last_clean":"2023-10-30T13:10:35.919976+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:10:35.919976+0000","last_undegraded":"2023-10-30T13:10:35.919976+0000","last_fullsized":"2023-10-30T13:10:35.919976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919216+0000","last_change":"2023-10-30T13:10:34.332955+0000","last_active":"2023-10-30T13:10:35.919216+0000","last_peered":"2023-10-30T13:10:35.919216+0000","last_clean":"2023-10-30T13:10:35.919216+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:10:35.919216+0000","last_undegraded":"2023-10-30T13:10:35.919216+0000","last_fullsized":"2023-10-30T13:10:35.919216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:05.776190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323974+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:10:35.323974+0000","last_peered":"2023-10-30T13:10:35.323974+0000","last_clean":"2023-10-30T13:10:35.323974+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:10:35.323974+0000","last_undegraded":"2023-10-30T13:10:35.323974+0000","last_fullsized":"2023-10-30T13:10:35.323974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923916+0000","last_change":"2023-10-30T13:10:34.362313+0000","last_active":"2023-10-30T13:10:35.923916+0000","last_peered":"2023-10-30T13:10:35.923916+0000","last_clean":"2023-10-30T13:10:35.923916+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:10:35.923916+0000","last_undegraded":"2023-10-30T13:10:35.923916+0000","last_fullsized":"2023-10-30T13:10:35.923916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:45.213916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919650+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:10:35.919650+0000","last_peered":"2023-10-30T13:10:35.919650+0000","last_clean":"2023-10-30T13:10:35.919650+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:10:35.919650+0000","last_undegraded":"2023-10-30T13:10:35.919650+0000","last_fullsized":"2023-10-30T13:10:35.919650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323629+0000","last_change":"2023-10-30T13:10:34.348653+0000","last_active":"2023-10-30T13:10:35.323629+0000","last_peered":"2023-10-30T13:10:35.323629+0000","last_clean":"2023-10-30T13:10:35.323629+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:10:35.323629+0000","last_undegraded":"2023-10-30T13:10:35.323629+0000","last_fullsized":"2023-10-30T13:10:35.323629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:54:00.948062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323802+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:10:35.323802+0000","last_peered":"2023-10-30T13:10:35.323802+0000","last_clean":"2023-10-30T13:10:35.323802+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:10:35.323802+0000","last_undegraded":"2023-10-30T13:10:35.323802+0000","last_fullsized":"2023-10-30T13:10:35.323802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919933+0000","last_change":"2023-10-30T13:10:34.332001+0000","last_active":"2023-10-30T13:10:35.919933+0000","last_peered":"2023-10-30T13:10:35.919933+0000","last_clean":"2023-10-30T13:10:35.919933+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:10:35.919933+0000","last_undegraded":"2023-10-30T13:10:35.919933+0000","last_fullsized":"2023-10-30T13:10:35.919933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:16.341054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920027+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:10:35.920027+0000","last_peered":"2023-10-30T13:10:35.920027+0000","last_clean":"2023-10-30T13:10:35.920027+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:10:35.920027+0000","last_undegraded":"2023-10-30T13:10:35.920027+0000","last_fullsized":"2023-10-30T13:10:35.920027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319328+0000","last_change":"2023-10-30T13:10:34.369845+0000","last_active":"2023-10-30T13:10:35.319328+0000","last_peered":"2023-10-30T13:10:35.319328+0000","last_clean":"2023-10-30T13:10:35.319328+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:10:35.319328+0000","last_undegraded":"2023-10-30T13:10:35.319328+0000","last_fullsized":"2023-10-30T13:10:35.319328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:30.468845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.897907+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:10:39.897907+0000","last_peered":"2023-10-30T13:10:39.897907+0000","last_clean":"2023-10-30T13:10:39.897907+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:10:39.897907+0000","last_undegraded":"2023-10-30T13:10:39.897907+0000","last_fullsized":"2023-10-30T13:10:39.897907+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.920190+0000","last_change":"2023-10-30T13:10:34.323336+0000","last_active":"2023-10-30T13:10:35.920190+0000","last_peered":"2023-10-30T13:10:35.920190+0000","last_clean":"2023-10-30T13:10:35.920190+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:10:35.920190+0000","last_undegraded":"2023-10-30T13:10:35.920190+0000","last_fullsized":"2023-10-30T13:10:35.920190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:21.642241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.321410+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:10:35.321410+0000","last_peered":"2023-10-30T13:10:35.321410+0000","last_clean":"2023-10-30T13:10:35.321410+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:10:35.321410+0000","last_undegraded":"2023-10-30T13:10:35.321410+0000","last_fullsized":"2023-10-30T13:10:35.321410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.923797+0000","last_change":"2023-10-30T13:10:34.363313+0000","last_active":"2023-10-30T13:10:35.923797+0000","last_peered":"2023-10-30T13:10:35.923797+0000","last_clean":"2023-10-30T13:10:35.923797+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:10:35.923797+0000","last_undegraded":"2023-10-30T13:10:35.923797+0000","last_fullsized":"2023-10-30T13:10:35.923797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:13.027610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919211+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:10:35.919211+0000","last_peered":"2023-10-30T13:10:35.919211+0000","last_clean":"2023-10-30T13:10:35.919211+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:10:35.919211+0000","last_undegraded":"2023-10-30T13:10:35.919211+0000","last_fullsized":"2023-10-30T13:10:35.919211+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.318724+0000","last_change":"2023-10-30T13:10:34.330677+0000","last_active":"2023-10-30T13:10:35.318724+0000","last_peered":"2023-10-30T13:10:35.318724+0000","last_clean":"2023-10-30T13:10:35.318724+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:10:35.318724+0000","last_undegraded":"2023-10-30T13:10:35.318724+0000","last_fullsized":"2023-10-30T13:10:35.318724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:01.360813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919383+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:10:35.919383+0000","last_peered":"2023-10-30T13:10:35.919383+0000","last_clean":"2023-10-30T13:10:35.919383+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:10:35.919383+0000","last_undegraded":"2023-10-30T13:10:35.919383+0000","last_fullsized":"2023-10-30T13:10:35.919383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319182+0000","last_change":"2023-10-30T13:10:34.331117+0000","last_active":"2023-10-30T13:10:35.319182+0000","last_peered":"2023-10-30T13:10:35.319182+0000","last_clean":"2023-10-30T13:10:35.319182+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:10:35.319182+0000","last_undegraded":"2023-10-30T13:10:35.319182+0000","last_fullsized":"2023-10-30T13:10:35.319182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:56.948470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.697876+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:10:39.697876+0000","last_peered":"2023-10-30T13:10:39.697876+0000","last_clean":"2023-10-30T13:10:39.697876+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:10:39.697876+0000","last_undegraded":"2023-10-30T13:10:39.697876+0000","last_fullsized":"2023-10-30T13:10:39.697876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.758411+0000","last_change":"2023-10-30T13:10:34.365315+0000","last_active":"2023-10-30T13:10:35.758411+0000","last_peered":"2023-10-30T13:10:35.758411+0000","last_clean":"2023-10-30T13:10:35.758411+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:10:35.758411+0000","last_undegraded":"2023-10-30T13:10:35.758411+0000","last_fullsized":"2023-10-30T13:10:35.758411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:37.078775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.323303+0000","last_change":"2023-10-30T13:10:34.366044+0000","last_active":"2023-10-30T13:10:35.323303+0000","last_peered":"2023-10-30T13:10:35.323303+0000","last_clean":"2023-10-30T13:10:35.323303+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:10:35.323303+0000","last_undegraded":"2023-10-30T13:10:35.323303+0000","last_fullsized":"2023-10-30T13:10:35.323303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:31.831020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.697453+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:10:39.697453+0000","last_peered":"2023-10-30T13:10:39.697453+0000","last_clean":"2023-10-30T13:10:39.697453+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:10:39.697453+0000","last_undegraded":"2023-10-30T13:10:39.697453+0000","last_fullsized":"2023-10-30T13:10:39.697453+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.324679+0000","last_change":"2023-10-30T13:10:34.369814+0000","last_active":"2023-10-30T13:10:35.324679+0000","last_peered":"2023-10-30T13:10:35.324679+0000","last_clean":"2023-10-30T13:10:35.324679+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:10:35.324679+0000","last_undegraded":"2023-10-30T13:10:35.324679+0000","last_fullsized":"2023-10-30T13:10:35.324679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:26.879717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.697444+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:10:39.697444+0000","last_peered":"2023-10-30T13:10:39.697444+0000","last_clean":"2023-10-30T13:10:39.697444+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:10:39.697444+0000","last_undegraded":"2023-10-30T13:10:39.697444+0000","last_fullsized":"2023-10-30T13:10:39.697444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.919435+0000","last_change":"2023-10-30T13:10:34.344766+0000","last_active":"2023-10-30T13:10:35.919435+0000","last_peered":"2023-10-30T13:10:35.919435+0000","last_clean":"2023-10-30T13:10:35.919435+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:10:35.919435+0000","last_undegraded":"2023-10-30T13:10:35.919435+0000","last_fullsized":"2023-10-30T13:10:35.919435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.073346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:37.283829+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:10:37.283829+0000","last_peered":"2023-10-30T13:10:37.283829+0000","last_clean":"2023-10-30T13:10:37.283829+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:10:37.283829+0000","last_undegraded":"2023-10-30T13:10:37.283829+0000","last_fullsized":"2023-10-30T13:10:37.283829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:35.319357+0000","last_change":"2023-10-30T13:10:34.354305+0000","last_active":"2023-10-30T13:10:35.319357+0000","last_peered":"2023-10-30T13:10:35.319357+0000","last_clean":"2023-10-30T13:10:35.319357+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:10:35.319357+0000","last_undegraded":"2023-10-30T13:10:35.319357+0000","last_fullsized":"2023-10-30T13:10:35.319357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:53.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T13:10:39.696166+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:10:39.696166+0000","last_peered":"2023-10-30T13:10:39.696166+0000","last_clean":"2023-10-30T13:10:39.696166+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:10:39.696166+0000","last_undegraded":"2023-10-30T13:10:39.696166+0000","last_fullsized":"2023-10-30T13:10:39.696166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6870000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.879}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6469999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6840000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.51}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2569999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9159999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.117}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":806912,"data_stored":660885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:10:45.874 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T13:10:45.875 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T13:10:45.875 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T13:10:45.875 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T13:10:46.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.040+0000 7fc13429e640 1 Processor -- start 2023-10-30T13:10:46.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.040+0000 7fc13429e640 1 -- start start 2023-10-30T13:10:46.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13429e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc12c0cce00 0x7fc12c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:46.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13429e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc12c0c7830 0x7fc12c0c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:46.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13429e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0c2220 con 0x7fc12c06e430 2023-10-30T13:10:46.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13429e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c0c4040 con 0x7fc12c0cce00 2023-10-30T13:10:46.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13429e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c0cd720 con 0x7fc12c0c7830 2023-10-30T13:10:46.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13203a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc12c0cce00 0x7fc12c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:46.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13283b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc12c06e430 0x7fc12c15ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39426/0 (socket says 172.21.15.148:39426) 2023-10-30T13:10:46.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13203a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc12c0cce00 0x7fc12c0cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42104/0 (socket says 172.21.15.148:42104) 2023-10-30T13:10:46.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13283b640 1 -- 172.21.15.148:0/3584134928 learned_addr learned my addr 172.21.15.148:0/3584134928 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:46.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc131839640 1 --2- 172.21.15.148:0/3584134928 >> v2:172.21.15.150:3300/0 conn(0x7fc12c0c7830 0x7fc12c0c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:46.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13203a640 1 -- 172.21.15.148:0/3584134928 >> v2:172.21.15.150:3300/0 conn(0x7fc12c0c7830 msgr2=0x7fc12c0c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:46.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13203a640 1 --2- 172.21.15.148:0/3584134928 >> v2:172.21.15.150:3300/0 conn(0x7fc12c0c7830 0x7fc12c0c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13203a640 1 -- 172.21.15.148:0/3584134928 >> v1:172.21.15.148:6789/0 conn(0x7fc12c06e430 legacy=0x7fc12c15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:46.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13203a640 1 -- 172.21.15.148:0/3584134928 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c0c3ac0 con 0x7fc12c0cce00 2023-10-30T13:10:46.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc131038640 1 -- 172.21.15.148:0/3584134928 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257226314 0 0) 0x7fc12c0c2220 con 0x7fc12c06e430 2023-10-30T13:10:46.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc13203a640 1 --2- 172.21.15.148:0/3584134928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0cce00 0x7fc12c0cd1e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0cdef0 tx=0x7fc12801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12629a71c5c385b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:46.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc131038640 1 -- 172.21.15.148:0/3584134928 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1280ab600 con 0x7fc12c0cce00 2023-10-30T13:10:46.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc131038640 1 -- 172.21.15.148:0/3584134928 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1280a7050 con 0x7fc12c0cce00 2023-10-30T13:10:46.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.044+0000 7fc131038640 1 -- 172.21.15.148:0/3584134928 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1280ab980 con 0x7fc12c0cce00 2023-10-30T13:10:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 -- 172.21.15.148:0/3584134928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0cce00 msgr2=0x7fc12c0cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 --2- 172.21.15.148:0/3584134928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0cce00 0x7fc12c0cd1e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0cdef0 tx=0x7fc12801c3d0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 -- 172.21.15.148:0/3584134928 shutdown_connections 2023-10-30T13:10:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 --2- 172.21.15.148:0/3584134928 >> v2:172.21.15.150:3300/0 conn(0x7fc12c0c7830 0x7fc12c0c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 --2- 172.21.15.148:0/3584134928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0cce00 0x7fc12c0cd1e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 -- 172.21.15.148:0/3584134928 >> 172.21.15.148:0/3584134928 conn(0x7fc12c071ea0 msgr2=0x7fc12c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:46.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 -- 172.21.15.148:0/3584134928 shutdown_connections 2023-10-30T13:10:46.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 -- 172.21.15.148:0/3584134928 wait complete. 2023-10-30T13:10:46.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 Processor -- start 2023-10-30T13:10:46.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 -- start start 2023-10-30T13:10:46.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc12c06e430 0x7fc12c06c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:46.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0c7830 0x7fc12c146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:46.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc12c0cce00 0x7fc12c1474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:46.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c159120 con 0x7fc12c0c7830 2023-10-30T13:10:46.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c159d30 con 0x7fc12c06e430 2023-10-30T13:10:46.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13429e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c1566e0 con 0x7fc12c0cce00 2023-10-30T13:10:46.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc131839640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0c7830 0x7fc12c146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:46.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc131839640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0c7830 0x7fc12c146fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42108/0 (socket says 172.21.15.148:42108) 2023-10-30T13:10:46.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc131839640 1 -- 172.21.15.148:0/3806790500 learned_addr learned my addr 172.21.15.148:0/3806790500 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:46.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13203a640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc12c06e430 0x7fc12c06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:46.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13283b640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc12c0cce00 0x7fc12c1474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:46.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc131839640 1 -- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc12c0cce00 msgr2=0x7fc12c1474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:46.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc131839640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc12c0cce00 0x7fc12c1474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc131839640 1 -- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc12c06e430 msgr2=0x7fc12c06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:46.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc131839640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc12c06e430 0x7fc12c06c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc131839640 1 -- 172.21.15.148:0/3806790500 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc128009420 con 0x7fc12c0c7830 2023-10-30T13:10:46.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.048+0000 7fc13283b640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc12c0cce00 0x7fc12c1474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:46.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc131839640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0c7830 0x7fc12c146fb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00d830 tx=0x7fc11c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:46.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc122ffd640 1 -- 172.21.15.148:0/3806790500 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c00b6c0 con 0x7fc12c0c7830 2023-10-30T13:10:46.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc122ffd640 1 -- 172.21.15.148:0/3806790500 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c00b820 con 0x7fc12c0c7830 2023-10-30T13:10:46.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c06cf10 con 0x7fc12c0c7830 2023-10-30T13:10:46.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc122ffd640 1 -- 172.21.15.148:0/3806790500 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c00bb90 con 0x7fc12c0c7830 2023-10-30T13:10:46.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c1596e0 con 0x7fc12c0c7830 2023-10-30T13:10:46.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c04e170 con 0x7fc12c0c7830 2023-10-30T13:10:46.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc122ffd640 1 -- 172.21.15.148:0/3806790500 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc11c01e460 con 0x7fc12c0c7830 2023-10-30T13:10:46.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc122ffd640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1100664d0 0x7fc110068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:46.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc122ffd640 1 -- 172.21.15.148:0/3806790500 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc11c0846d0 con 0x7fc12c0c7830 2023-10-30T13:10:46.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.052+0000 7fc13203a640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1100664d0 0x7fc110068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:46.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.056+0000 7fc13203a640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1100664d0 0x7fc110068990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1280ad000 tx=0x7fc128008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:46.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.056+0000 7fc122ffd640 1 -- 172.21.15.148:0/3806790500 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc11c051a20 con 0x7fc12c0c7830 2023-10-30T13:10:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.236+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc12c154f40 con 0x7fc12c0c7830 2023-10-30T13:10:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.236+0000 7fc122ffd640 1 -- 172.21.15.148:0/3806790500 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7fc11c055040 con 0x7fc12c0c7830 2023-10-30T13:10:46.243 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:46.243 INFO:teuthology.orchestra.run.smithi148.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-30T13:10:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1100664d0 msgr2=0x7fc110068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1100664d0 0x7fc110068990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc1280ad000 tx=0x7fc128008040 comp rx=0 tx=0).stop 2023-10-30T13:10:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0c7830 msgr2=0x7fc12c146fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0c7830 0x7fc12c146fb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00d830 tx=0x7fc11c00dcf0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 shutdown_connections 2023-10-30T13:10:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1100664d0 0x7fc110068990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc12c0c7830 0x7fc12c146fb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc12c0cce00 0x7fc12c1474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 --2- 172.21.15.148:0/3806790500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc12c06e430 0x7fc12c06c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 >> 172.21.15.148:0/3806790500 conn(0x7fc12c071ea0 msgr2=0x7fc12c0c94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 shutdown_connections 2023-10-30T13:10:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:46.240+0000 7fc13429e640 1 -- 172.21.15.148:0/3806790500 wait complete. 2023-10-30T13:10:49.264 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T13:10:49.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4dbb94640 1 Processor -- start 2023-10-30T13:10:49.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4dbb94640 1 -- start start 2023-10-30T13:10:49.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4dbb94640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 0x7fe4d4161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4dbb94640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4d40c6b40 0x7fe4d4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4dbb94640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d40c72b0 con 0x7fe4d406a3d0 2023-10-30T13:10:49.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4dbb94640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d4162120 con 0x7fe4d40c6b40 2023-10-30T13:10:49.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4dbb94640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40c6fd0 con 0x7fe4d4158090 2023-10-30T13:10:49.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4d912f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4d40c6b40 0x7fe4d4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4d912f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4d40c6b40 0x7fe4d4069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42122/0 (socket says 172.21.15.148:42122) 2023-10-30T13:10:49.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4d912f640 1 -- 172.21.15.148:0/124985997 learned_addr learned my addr 172.21.15.148:0/124985997 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:49.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.408+0000 7fe4da131640 1 --1- 172.21.15.148:0/124985997 >> v1:172.21.15.148:6789/0 conn(0x7fe4d406a3d0 0x7fe4d40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39440/0 (socket says 172.21.15.148:39440) 2023-10-30T13:10:49.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d9930640 1 --2- 172.21.15.148:0/124985997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 0x7fe4d4161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d912f640 1 -- 172.21.15.148:0/124985997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 msgr2=0x7fe4d4161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:49.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d912f640 1 --2- 172.21.15.148:0/124985997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 0x7fe4d4161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d912f640 1 -- 172.21.15.148:0/124985997 >> v1:172.21.15.148:6789/0 conn(0x7fe4d406a3d0 legacy=0x7fe4d40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d912f640 1 -- 172.21.15.148:0/124985997 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d415dd30 con 0x7fe4d40c6b40 2023-10-30T13:10:49.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d892e640 1 -- 172.21.15.148:0/124985997 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685493340 0 0) 0x7fe4d40c72b0 con 0x7fe4d406a3d0 2023-10-30T13:10:49.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d912f640 1 --2- 172.21.15.148:0/124985997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c6b40 0x7fe4d4069e90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009870 tx=0x7fe4c401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66aaab2402ea95f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:49.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d892e640 1 -- 172.21.15.148:0/124985997 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40a8a30 con 0x7fe4d40c6b40 2023-10-30T13:10:49.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d892e640 1 -- 172.21.15.148:0/124985997 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c40a8b90 con 0x7fe4d40c6b40 2023-10-30T13:10:49.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4d892e640 1 -- 172.21.15.148:0/124985997 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40b0440 con 0x7fe4d40c6b40 2023-10-30T13:10:49.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4dbb94640 1 -- 172.21.15.148:0/124985997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c6b40 msgr2=0x7fe4d4069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.412+0000 7fe4dbb94640 1 --2- 172.21.15.148:0/124985997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c6b40 0x7fe4d4069e90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009870 tx=0x7fe4c401ed30 comp rx=0 tx=0).stop 2023-10-30T13:10:49.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 -- 172.21.15.148:0/124985997 shutdown_connections 2023-10-30T13:10:49.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 --2- 172.21.15.148:0/124985997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c6b40 0x7fe4d4069e90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 --2- 172.21.15.148:0/124985997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 0x7fe4d4161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 -- 172.21.15.148:0/124985997 >> 172.21.15.148:0/124985997 conn(0x7fe4d40c7910 msgr2=0x7fe4d40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:49.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 -- 172.21.15.148:0/124985997 shutdown_connections 2023-10-30T13:10:49.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 -- 172.21.15.148:0/124985997 wait complete. 2023-10-30T13:10:49.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 Processor -- start 2023-10-30T13:10:49.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 -- start start 2023-10-30T13:10:49.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4d406a3d0 0x7fe4d40c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 0x7fe4d40c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c1040 0x7fe4d414c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d40c41a0 con 0x7fe4d40c1040 2023-10-30T13:10:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d406a980 con 0x7fe4d406a3d0 2023-10-30T13:10:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4dbb94640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d41513e0 con 0x7fe4d4158090 2023-10-30T13:10:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4da131640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c1040 0x7fe4d414c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4da131640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c1040 0x7fe4d414c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42132/0 (socket says 172.21.15.148:42132) 2023-10-30T13:10:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4da131640 1 -- 172.21.15.148:0/3029931043 learned_addr learned my addr 172.21.15.148:0/3029931043 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4d9930640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4d406a3d0 0x7fe4d40c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4d912f640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 0x7fe4d40c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4da131640 1 -- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 msgr2=0x7fe4d40c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4da131640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 0x7fe4d40c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4da131640 1 -- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4d406a3d0 msgr2=0x7fe4d40c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4da131640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4d406a3d0 0x7fe4d40c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.416+0000 7fe4da131640 1 -- 172.21.15.148:0/3029931043 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c40094d0 con 0x7fe4d40c1040 2023-10-30T13:10:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4da131640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c1040 0x7fe4d414c5f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc001570 tx=0x7fe4cc00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4ca7fc640 1 -- 172.21.15.148:0/3029931043 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc00a640 con 0x7fe4d40c1040 2023-10-30T13:10:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4ca7fc640 1 -- 172.21.15.148:0/3029931043 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc00a7c0 con 0x7fe4d40c1040 2023-10-30T13:10:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d41516a0 con 0x7fe4d40c1040 2023-10-30T13:10:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4ca7fc640 1 -- 172.21.15.148:0/3029931043 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc00ab70 con 0x7fe4d40c1040 2023-10-30T13:10:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d4151c60 con 0x7fe4d40c1040 2023-10-30T13:10:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d404e170 con 0x7fe4d40c1040 2023-10-30T13:10:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4ca7fc640 1 -- 172.21.15.148:0/3029931043 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe4cc00acd0 con 0x7fe4d40c1040 2023-10-30T13:10:49.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4ca7fc640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4b80664a0 0x7fe4b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4ca7fc640 1 -- 172.21.15.148:0/3029931043 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4cc0837b0 con 0x7fe4d40c1040 2023-10-30T13:10:49.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.420+0000 7fe4d9930640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4b80664a0 0x7fe4b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.424+0000 7fe4d9930640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4b80664a0 0x7fe4b8068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000b930 tx=0x7fe4d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.424+0000 7fe4ca7fc640 1 -- 172.21.15.148:0/3029931043 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe4cc050b00 con 0x7fe4d40c1040 2023-10-30T13:10:49.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe4d40b0860 con 0x7fe4d40c1040 2023-10-30T13:10:49.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4ca7fc640 1 -- 172.21.15.148:0/3029931043 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe4cc018080 con 0x7fe4d40c1040 2023-10-30T13:10:49.612 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:49.613 INFO:teuthology.orchestra.run.smithi148.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T13:10:49.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4b80664a0 msgr2=0x7fe4b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4b80664a0 0x7fe4b8068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000b930 tx=0x7fe4d0008040 comp rx=0 tx=0).stop 2023-10-30T13:10:49.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c1040 msgr2=0x7fe4d414c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c1040 0x7fe4d414c5f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc001570 tx=0x7fe4cc00cca0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 shutdown_connections 2023-10-30T13:10:49.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4b80664a0 0x7fe4b8068960 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4d40c1040 0x7fe4d414c5f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4d4158090 0x7fe4d40c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 --2- 172.21.15.148:0/3029931043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4d406a3d0 0x7fe4d40c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 >> 172.21.15.148:0/3029931043 conn(0x7fe4d40c7910 msgr2=0x7fe4d406bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:49.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 shutdown_connections 2023-10-30T13:10:49.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.608+0000 7fe4dbb94640 1 -- 172.21.15.148:0/3029931043 wait complete. 2023-10-30T13:10:49.632 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T13:10:49.633 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:49.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.768+0000 7f9bc7b3e640 1 Processor -- start 2023-10-30T13:10:49.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.768+0000 7f9bc7b3e640 1 -- start start 2023-10-30T13:10:49.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.768+0000 7f9bc7b3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc0153d70 0x7f9bc006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.768+0000 7f9bc7b3e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9bc006e470 0x7f9bc00cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.768+0000 7f9bc7b3e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc006afa0 con 0x7f9bc00c8490 2023-10-30T13:10:49.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.768+0000 7f9bc7b3e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc006d780 con 0x7f9bc006e470 2023-10-30T13:10:49.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.768+0000 7f9bc7b3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc0154150 con 0x7f9bc0153d70 2023-10-30T13:10:49.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc50d9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9bc006e470 0x7f9bc00cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc60db640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9bc00c8490 0x7f9bc00c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39456/0 (socket says 172.21.15.148:39456) 2023-10-30T13:10:49.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc50d9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9bc006e470 0x7f9bc00cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42136/0 (socket says 172.21.15.148:42136) 2023-10-30T13:10:49.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc60db640 1 -- 172.21.15.148:0/998307501 learned_addr learned my addr 172.21.15.148:0/998307501 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:49.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc58da640 1 --2- 172.21.15.148:0/998307501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc0153d70 0x7f9bc006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc50d9640 1 -- 172.21.15.148:0/998307501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc0153d70 msgr2=0x7f9bc006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:49.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc50d9640 1 --2- 172.21.15.148:0/998307501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc0153d70 0x7f9bc006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc50d9640 1 -- 172.21.15.148:0/998307501 >> v1:172.21.15.148:6789/0 conn(0x7f9bc00c8490 legacy=0x7f9bc00c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc50d9640 1 -- 172.21.15.148:0/998307501 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc006e030 con 0x7f9bc006e470 2023-10-30T13:10:49.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc48d8640 1 -- 172.21.15.148:0/998307501 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427473211 0 0) 0x7f9bc006afa0 con 0x7f9bc00c8490 2023-10-30T13:10:49.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc50d9640 1 --2- 172.21.15.148:0/998307501 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 0x7f9bc00cce00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00097c0 tx=0x7f9bb001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4dacbdcddb42bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:49.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc48d8640 1 -- 172.21.15.148:0/998307501 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb00ab650 con 0x7f9bc006e470 2023-10-30T13:10:49.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc48d8640 1 -- 172.21.15.148:0/998307501 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb00ab7b0 con 0x7f9bc006e470 2023-10-30T13:10:49.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.772+0000 7f9bc48d8640 1 -- 172.21.15.148:0/998307501 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb00abb40 con 0x7f9bc006e470 2023-10-30T13:10:49.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/998307501 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 msgr2=0x7f9bc00cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 --2- 172.21.15.148:0/998307501 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 0x7f9bc00cce00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00097c0 tx=0x7f9bb001c450 comp rx=0 tx=0).stop 2023-10-30T13:10:49.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/998307501 shutdown_connections 2023-10-30T13:10:49.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 --2- 172.21.15.148:0/998307501 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 0x7f9bc00cce00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 --2- 172.21.15.148:0/998307501 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc0153d70 0x7f9bc006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/998307501 >> 172.21.15.148:0/998307501 conn(0x7f9bc0071ea0 msgr2=0x7f9bc0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:49.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/998307501 shutdown_connections 2023-10-30T13:10:49.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/998307501 wait complete. 2023-10-30T13:10:49.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 Processor -- start 2023-10-30T13:10:49.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 -- start start 2023-10-30T13:10:49.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 0x7f9bc0156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc00c8490 0x7f9bc01f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bc0153d70 0x7f9bc01f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00c3060 con 0x7f9bc006e470 2023-10-30T13:10:49.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc0147bd0 con 0x7f9bc0153d70 2023-10-30T13:10:49.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc7b3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00b15c0 con 0x7f9bc00c8490 2023-10-30T13:10:49.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc58da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 0x7f9bc0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc58da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 0x7f9bc0156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42144/0 (socket says 172.21.15.148:42144) 2023-10-30T13:10:49.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc58da640 1 -- 172.21.15.148:0/3551049336 learned_addr learned my addr 172.21.15.148:0/3551049336 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:49.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc50d9640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc00c8490 0x7f9bc01f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc60db640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bc0153d70 0x7f9bc01f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc58da640 1 -- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc00c8490 msgr2=0x7f9bc01f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc58da640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc00c8490 0x7f9bc01f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc58da640 1 -- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bc0153d70 msgr2=0x7f9bc01f5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc58da640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bc0153d70 0x7f9bc01f5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc58da640 1 -- 172.21.15.148:0/3551049336 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb0009420 con 0x7f9bc006e470 2023-10-30T13:10:49.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc50d9640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc00c8490 0x7f9bc01f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:49.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.776+0000 7f9bc60db640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bc0153d70 0x7f9bc01f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:10:49.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bc58da640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 0x7f9bc0156500 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00e960 tx=0x7f9bbc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:49.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bb67fc640 1 -- 172.21.15.148:0/3551049336 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc009690 con 0x7f9bc006e470 2023-10-30T13:10:49.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bb67fc640 1 -- 172.21.15.148:0/3551049336 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bbc00ca00 con 0x7f9bc006e470 2023-10-30T13:10:49.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bb67fc640 1 -- 172.21.15.148:0/3551049336 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc00cd70 con 0x7f9bc006e470 2023-10-30T13:10:49.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc00c7320 con 0x7f9bc006e470 2023-10-30T13:10:49.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc00c6d60 con 0x7f9bc006e470 2023-10-30T13:10:49.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc004e170 con 0x7f9bc006e470 2023-10-30T13:10:49.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bb67fc640 1 -- 172.21.15.148:0/3551049336 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9bbc013470 con 0x7f9bc006e470 2023-10-30T13:10:49.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bb67fc640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ba40664a0 0x7f9ba4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:49.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bb67fc640 1 -- 172.21.15.148:0/3551049336 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bbc084400 con 0x7f9bc006e470 2023-10-30T13:10:49.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.780+0000 7f9bc50d9640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ba40664a0 0x7f9ba4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:49.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.784+0000 7f9bc50d9640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ba40664a0 0x7f9ba4068960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00af000 tx=0x7f9bb00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:49.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.784+0000 7f9bb67fc640 1 -- 172.21.15.148:0/3551049336 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9bbc051910 con 0x7f9bc006e470 2023-10-30T13:10:49.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.936+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bc006dd20 con 0x7f9bc006e470 2023-10-30T13:10:49.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.936+0000 7f9bb67fc640 1 -- 172.21.15.148:0/3551049336 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f9bbc054f30 con 0x7f9bc006e470 2023-10-30T13:10:49.941 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:49.942 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:35.311442+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:49.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ba40664a0 msgr2=0x7f9ba4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ba40664a0 0x7f9ba4068960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00af000 tx=0x7f9bb00a7000 comp rx=0 tx=0).stop 2023-10-30T13:10:49.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 msgr2=0x7f9bc0156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:49.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 0x7f9bc0156500 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00e960 tx=0x7f9bbc00ee20 comp rx=0 tx=0).stop 2023-10-30T13:10:49.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 shutdown_connections 2023-10-30T13:10:49.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ba40664a0 0x7f9ba4068960 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bc0153d70 0x7f9bc01f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bc00c8490 0x7f9bc01f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 --2- 172.21.15.148:0/3551049336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bc006e470 0x7f9bc0156500 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:49.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 >> 172.21.15.148:0/3551049336 conn(0x7f9bc0071ea0 msgr2=0x7f9bc0150ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:49.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 shutdown_connections 2023-10-30T13:10:49.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:49.940+0000 7f9bc7b3e640 1 -- 172.21.15.148:0/3551049336 wait complete. 2023-10-30T13:10:49.962 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:10:22.349322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:10:32.611151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:10:33.044931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:10:49.963 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:10:49.990 INFO:tasks.ceph.mds.e.smithi148.stderr:2023-10-30T13:10:49.984+0000 7ff756f04640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:10:49.990 INFO:tasks.ceph.mds.e.smithi148.stderr:2023-10-30T13:10:49.984+0000 7ff756f04640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:10:50.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6fb3e640 1 Processor -- start 2023-10-30T13:10:50.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6fb3e640 1 -- start start 2023-10-30T13:10:50.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6fb3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee681539e0 0x7fee680c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6fb3e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee6806d420 0x7fee681640a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6fb3e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6815a670 con 0x7fee6815a1a0 2023-10-30T13:10:50.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6fb3e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee680c0f70 con 0x7fee6806d420 2023-10-30T13:10:50.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6fb3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee680c1250 con 0x7fee681539e0 2023-10-30T13:10:50.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6e0db640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee6806d420 0x7fee681640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6d0d9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fee6815a1a0 0x7fee680c78f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39468/0 (socket says 172.21.15.148:39468) 2023-10-30T13:10:50.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6e0db640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee6806d420 0x7fee681640a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42156/0 (socket says 172.21.15.148:42156) 2023-10-30T13:10:50.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6d0d9640 1 -- 172.21.15.148:0/477706275 learned_addr learned my addr 172.21.15.148:0/477706275 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:50.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.100+0000 7fee6d8da640 1 --2- 172.21.15.148:0/477706275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee681539e0 0x7fee680c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6e0db640 1 -- 172.21.15.148:0/477706275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee681539e0 msgr2=0x7fee680c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:50.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6e0db640 1 --2- 172.21.15.148:0/477706275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee681539e0 0x7fee680c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6e0db640 1 -- 172.21.15.148:0/477706275 >> v1:172.21.15.148:6789/0 conn(0x7fee6815a1a0 legacy=0x7fee680c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6e0db640 1 -- 172.21.15.148:0/477706275 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee680c1530 con 0x7fee6806d420 2023-10-30T13:10:50.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6c8d8640 1 -- 172.21.15.148:0/477706275 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718326772 0 0) 0x7fee6815a670 con 0x7fee6815a1a0 2023-10-30T13:10:50.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6e0db640 1 --2- 172.21.15.148:0/477706275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee6806d420 0x7fee681640a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fee60009d90 tx=0x7fee6001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a87eb21b89dc14a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:50.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6c8d8640 1 -- 172.21.15.148:0/477706275 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee600af600 con 0x7fee6806d420 2023-10-30T13:10:50.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6c8d8640 1 -- 172.21.15.148:0/477706275 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee600ab050 con 0x7fee6806d420 2023-10-30T13:10:50.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6c8d8640 1 -- 172.21.15.148:0/477706275 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee600af980 con 0x7fee6806d420 2023-10-30T13:10:50.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6fb3e640 1 -- 172.21.15.148:0/477706275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee6806d420 msgr2=0x7fee681640a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.104+0000 7fee6fb3e640 1 --2- 172.21.15.148:0/477706275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee6806d420 0x7fee681640a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fee60009d90 tx=0x7fee6001f130 comp rx=0 tx=0).stop 2023-10-30T13:10:50.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 -- 172.21.15.148:0/477706275 shutdown_connections 2023-10-30T13:10:50.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 --2- 172.21.15.148:0/477706275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee6806d420 0x7fee681640a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 --2- 172.21.15.148:0/477706275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee681539e0 0x7fee680c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 -- 172.21.15.148:0/477706275 >> 172.21.15.148:0/477706275 conn(0x7fee6806c850 msgr2=0x7fee6806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:50.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 -- 172.21.15.148:0/477706275 shutdown_connections 2023-10-30T13:10:50.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 -- 172.21.15.148:0/477706275 wait complete. 2023-10-30T13:10:50.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 Processor -- start 2023-10-30T13:10:50.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 -- start start 2023-10-30T13:10:50.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee6806d420 0x7fee6815ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee681539e0 0x7fee681f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6815a1a0 0x7fee680c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6815ac30 con 0x7fee681539e0 2023-10-30T13:10:50.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680c37d0 con 0x7fee6815a1a0 2023-10-30T13:10:50.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6fb3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6815a8e0 con 0x7fee6806d420 2023-10-30T13:10:50.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d0d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee681539e0 0x7fee681f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d0d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee681539e0 0x7fee681f2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42172/0 (socket says 172.21.15.148:42172) 2023-10-30T13:10:50.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d0d9640 1 -- 172.21.15.148:0/211603651 learned_addr learned my addr 172.21.15.148:0/211603651 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:50.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d8da640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee6806d420 0x7fee6815ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6e0db640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6815a1a0 0x7fee680c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d0d9640 1 -- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee6806d420 msgr2=0x7fee6815ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d0d9640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee6806d420 0x7fee6815ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d0d9640 1 -- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6815a1a0 msgr2=0x7fee680c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d0d9640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6815a1a0 0x7fee680c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d0d9640 1 -- 172.21.15.148:0/211603651 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee60009a70 con 0x7fee681539e0 2023-10-30T13:10:50.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.108+0000 7fee6d0d9640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee681539e0 0x7fee681f2550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c450 tx=0x7fee5800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:50.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee5e7fc640 1 -- 172.21.15.148:0/211603651 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee580074e0 con 0x7fee681539e0 2023-10-30T13:10:50.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee5e7fc640 1 -- 172.21.15.148:0/211603651 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58007640 con 0x7fee681539e0 2023-10-30T13:10:50.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee5e7fc640 1 -- 172.21.15.148:0/211603651 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee580049f0 con 0x7fee681539e0 2023-10-30T13:10:50.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee68155180 con 0x7fee681539e0 2023-10-30T13:10:50.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6815d510 con 0x7fee681539e0 2023-10-30T13:10:50.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6815ad70 con 0x7fee681539e0 2023-10-30T13:10:50.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee5e7fc640 1 -- 172.21.15.148:0/211603651 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee58004b50 con 0x7fee681539e0 2023-10-30T13:10:50.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee5e7fc640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee4c0664f0 0x7fee4c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee5e7fc640 1 -- 172.21.15.148:0/211603651 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee58083210 con 0x7fee681539e0 2023-10-30T13:10:50.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.112+0000 7fee6d8da640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee4c0664f0 0x7fee4c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.116+0000 7fee6d8da640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee4c0664f0 0x7fee4c0689b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee64009b60 tx=0x7fee64009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:50.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.116+0000 7fee5e7fc640 1 -- 172.21.15.148:0/211603651 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee58050820 con 0x7fee681539e0 2023-10-30T13:10:50.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.264+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee6804e170 con 0x7fee681539e0 2023-10-30T13:10:50.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.264+0000 7fee5e7fc640 1 -- 172.21.15.148:0/211603651 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee580180a0 con 0x7fee681539e0 2023-10-30T13:10:50.269 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2023-10-30T13:10:50.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee4c0664f0 msgr2=0x7fee4c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee4c0664f0 0x7fee4c0689b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee64009b60 tx=0x7fee64009190 comp rx=0 tx=0).stop 2023-10-30T13:10:50.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee681539e0 msgr2=0x7fee681f2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee681539e0 0x7fee681f2550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c450 tx=0x7fee5800c910 comp rx=0 tx=0).stop 2023-10-30T13:10:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 shutdown_connections 2023-10-30T13:10:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee4c0664f0 0x7fee4c0689b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee681539e0 0x7fee681f2550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6815a1a0 0x7fee680c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 --2- 172.21.15.148:0/211603651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee6806d420 0x7fee6815ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 >> 172.21.15.148:0/211603651 conn(0x7fee6806c850 msgr2=0x7fee680706c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:50.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 shutdown_connections 2023-10-30T13:10:50.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.268+0000 7fee6fb3e640 1 -- 172.21.15.148:0/211603651 wait complete. 2023-10-30T13:10:50.289 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:10:50.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.468+0000 7fee318ea640 1 Processor -- start 2023-10-30T13:10:50.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.468+0000 7fee318ea640 1 -- start start 2023-10-30T13:10:50.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee318ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c153980 0x7fee2c0caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee318ea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee2c0cb1e0 0x7fee2c15d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee318ea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c15a920 con 0x7fee2c159930 2023-10-30T13:10:50.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee318ea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee2c0cb930 con 0x7fee2c0cb1e0 2023-10-30T13:10:50.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee318ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c159dc0 con 0x7fee2c153980 2023-10-30T13:10:50.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee23fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fee2c159930 0x7fee2c0cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39470/0 (socket says 172.21.15.148:39470) 2023-10-30T13:10:50.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee23fff640 1 -- 172.21.15.148:0/1065586442 learned_addr learned my addr 172.21.15.148:0/1065586442 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:50.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee2affd640 1 --2- 172.21.15.148:0/1065586442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c153980 0x7fee2c0caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee2b7fe640 1 --2- 172.21.15.148:0/1065586442 >> v2:172.21.15.150:3300/0 conn(0x7fee2c0cb1e0 0x7fee2c15d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee2a7fc640 1 -- 172.21.15.148:0/1065586442 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036769546 0 0) 0x7fee2c15a920 con 0x7fee2c159930 2023-10-30T13:10:50.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee2a7fc640 1 -- 172.21.15.148:0/1065586442 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee18003530 con 0x7fee2c159930 2023-10-30T13:10:50.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee2affd640 1 -- 172.21.15.148:0/1065586442 >> v2:172.21.15.150:3300/0 conn(0x7fee2c0cb1e0 msgr2=0x7fee2c15d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:50.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee2affd640 1 --2- 172.21.15.148:0/1065586442 >> v2:172.21.15.150:3300/0 conn(0x7fee2c0cb1e0 0x7fee2c15d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee2affd640 1 -- 172.21.15.148:0/1065586442 >> v1:172.21.15.148:6789/0 conn(0x7fee2c159930 legacy=0x7fee2c0cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee2affd640 1 -- 172.21.15.148:0/1065586442 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c15a080 con 0x7fee2c153980 2023-10-30T13:10:50.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.472+0000 7fee2b7fe640 1 --2- 172.21.15.148:0/1065586442 >> v2:172.21.15.150:3300/0 conn(0x7fee2c0cb1e0 0x7fee2c15d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:50.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee2affd640 1 --2- 172.21.15.148:0/1065586442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c153980 0x7fee2c0caca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0097c0 tx=0x7fee1c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c788f026c07fd14 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:50.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee2a7fc640 1 -- 172.21.15.148:0/1065586442 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0ad660 con 0x7fee2c153980 2023-10-30T13:10:50.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee2a7fc640 1 -- 172.21.15.148:0/1065586442 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c0ad7c0 con 0x7fee2c153980 2023-10-30T13:10:50.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee2a7fc640 1 -- 172.21.15.148:0/1065586442 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0adb50 con 0x7fee2c153980 2023-10-30T13:10:50.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 -- 172.21.15.148:0/1065586442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c153980 msgr2=0x7fee2c0caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 --2- 172.21.15.148:0/1065586442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c153980 0x7fee2c0caca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0097c0 tx=0x7fee1c01c450 comp rx=0 tx=0).stop 2023-10-30T13:10:50.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 -- 172.21.15.148:0/1065586442 shutdown_connections 2023-10-30T13:10:50.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 --2- 172.21.15.148:0/1065586442 >> v2:172.21.15.150:3300/0 conn(0x7fee2c0cb1e0 0x7fee2c15d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 --2- 172.21.15.148:0/1065586442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c153980 0x7fee2c0caca0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 -- 172.21.15.148:0/1065586442 >> 172.21.15.148:0/1065586442 conn(0x7fee2c06c0f0 msgr2=0x7fee2c06c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:50.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 -- 172.21.15.148:0/1065586442 shutdown_connections 2023-10-30T13:10:50.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 -- 172.21.15.148:0/1065586442 wait complete. 2023-10-30T13:10:50.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 Processor -- start 2023-10-30T13:10:50.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 -- start start 2023-10-30T13:10:50.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2c0cb1e0 0x7fee2c14b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee2c153980 0x7fee2c0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c0ce760 0x7fee2c14bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c0ce5d0 con 0x7fee2c153980 2023-10-30T13:10:50.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c154b50 con 0x7fee2c0cb1e0 2023-10-30T13:10:50.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.476+0000 7fee318ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c14b900 con 0x7fee2c0ce760 2023-10-30T13:10:50.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee23fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee2c153980 0x7fee2c0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee23fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee2c153980 0x7fee2c0b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42178/0 (socket says 172.21.15.148:42178) 2023-10-30T13:10:50.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee23fff640 1 -- 172.21.15.148:0/3365650454 learned_addr learned my addr 172.21.15.148:0/3365650454 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:50.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee2affd640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2c0cb1e0 0x7fee2c14b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee2b7fe640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c0ce760 0x7fee2c14bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee23fff640 1 -- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c0ce760 msgr2=0x7fee2c14bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee23fff640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c0ce760 0x7fee2c14bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee23fff640 1 -- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2c0cb1e0 msgr2=0x7fee2c14b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee23fff640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2c0cb1e0 0x7fee2c14b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee23fff640 1 -- 172.21.15.148:0/3365650454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee24009a70 con 0x7fee2c153980 2023-10-30T13:10:50.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee23fff640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee2c153980 0x7fee2c0b4330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fee1400c420 tx=0x7fee1400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:50.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee28ff9640 1 -- 172.21.15.148:0/3365650454 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14005cf0 con 0x7fee2c153980 2023-10-30T13:10:50.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee28ff9640 1 -- 172.21.15.148:0/3365650454 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee14005e50 con 0x7fee2c153980 2023-10-30T13:10:50.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c009420 con 0x7fee2c153980 2023-10-30T13:10:50.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee28ff9640 1 -- 172.21.15.148:0/3365650454 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee140072d0 con 0x7fee2c153980 2023-10-30T13:10:50.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c0cf080 con 0x7fee2c153980 2023-10-30T13:10:50.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c0cdd70 con 0x7fee2c153980 2023-10-30T13:10:50.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.480+0000 7fee28ff9640 1 -- 172.21.15.148:0/3365650454 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee14007470 con 0x7fee2c153980 2023-10-30T13:10:50.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.484+0000 7fee28ff9640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee0c0664a0 0x7fee0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.484+0000 7fee28ff9640 1 -- 172.21.15.148:0/3365650454 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee140834d0 con 0x7fee2c153980 2023-10-30T13:10:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.484+0000 7fee2affd640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee0c0664a0 0x7fee0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.484+0000 7fee2affd640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee0c0664a0 0x7fee0c068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0af000 tx=0x7fee1c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.488+0000 7fee28ff9640 1 -- 172.21.15.148:0/3365650454 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee2c0cdd70 con 0x7fee2c153980 2023-10-30T13:10:50.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.636+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fee2c0cdd70 con 0x7fee2c153980 2023-10-30T13:10:50.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.636+0000 7fee28ff9640 1 -- 172.21.15.148:0/3365650454 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fee2c0cdd70 con 0x7fee2c153980 2023-10-30T13:10:50.641 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 32 2023-10-30T13:10:50.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee0c0664a0 msgr2=0x7fee0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee0c0664a0 0x7fee0c068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0af000 tx=0x7fee1c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:10:50.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee2c153980 msgr2=0x7fee2c0b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee2c153980 0x7fee2c0b4330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fee1400c420 tx=0x7fee1400c820 comp rx=0 tx=0).stop 2023-10-30T13:10:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 shutdown_connections 2023-10-30T13:10:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee0c0664a0 0x7fee0c068960 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee2c153980 0x7fee2c0b4330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee2c0ce760 0x7fee2c14bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 --2- 172.21.15.148:0/3365650454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2c0cb1e0 0x7fee2c14b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 >> 172.21.15.148:0/3365650454 conn(0x7fee2c06c0f0 msgr2=0x7fee2c15c320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 shutdown_connections 2023-10-30T13:10:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.640+0000 7fee318ea640 1 -- 172.21.15.148:0/3365650454 wait complete. 2023-10-30T13:10:50.662 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:10:50.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc2a0c09640 1 Processor -- start 2023-10-30T13:10:50.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc2a0c09640 1 -- start start 2023-10-30T13:10:50.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc2a0c09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0722a0 0x7fc29c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc2a0c09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc29c0c0400 0x7fc29c156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc2a0c09640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0c0a30 con 0x7fc29c1508c0 2023-10-30T13:10:50.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc2a0c09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29c0c85b0 con 0x7fc29c0c0400 2023-10-30T13:10:50.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc2a0c09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c06f650 con 0x7fc29c0722a0 2023-10-30T13:10:50.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc29a59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc29c1508c0 0x7fc29c0c6920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39476/0 (socket says 172.21.15.148:39476) 2023-10-30T13:10:50.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc29a59c640 1 -- 172.21.15.148:0/3559747338 learned_addr learned my addr 172.21.15.148:0/3559747338 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:50.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc299d9b640 1 --2- 172.21.15.148:0/3559747338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0722a0 0x7fc29c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc29ad9d640 1 --2- 172.21.15.148:0/3559747338 >> v2:172.21.15.150:3300/0 conn(0x7fc29c0c0400 0x7fc29c156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc29959a640 1 -- 172.21.15.148:0/3559747338 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153816695 0 0) 0x7fc29c0c0a30 con 0x7fc29c1508c0 2023-10-30T13:10:50.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc29959a640 1 -- 172.21.15.148:0/3559747338 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc284003530 con 0x7fc29c1508c0 2023-10-30T13:10:50.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc29959a640 1 -- 172.21.15.148:0/3559747338 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1695424423 0 0) 0x7fc284003530 con 0x7fc29c1508c0 2023-10-30T13:10:50.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc299d9b640 1 -- 172.21.15.148:0/3559747338 >> v2:172.21.15.150:3300/0 conn(0x7fc29c0c0400 msgr2=0x7fc29c156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:50.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc299d9b640 1 --2- 172.21.15.148:0/3559747338 >> v2:172.21.15.150:3300/0 conn(0x7fc29c0c0400 0x7fc29c156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc299d9b640 1 -- 172.21.15.148:0/3559747338 >> v1:172.21.15.148:6789/0 conn(0x7fc29c1508c0 legacy=0x7fc29c0c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.800+0000 7fc299d9b640 1 -- 172.21.15.148:0/3559747338 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c06f930 con 0x7fc29c0722a0 2023-10-30T13:10:50.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc299d9b640 1 --2- 172.21.15.148:0/3559747338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0722a0 0x7fc29c0bfec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc2900097c0 tx=0x7fc29001c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e47561741af5731 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:50.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc29959a640 1 -- 172.21.15.148:0/3559747338 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c001800 con 0x7fc29c0722a0 2023-10-30T13:10:50.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc29959a640 1 -- 172.21.15.148:0/3559747338 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2900a7050 con 0x7fc29c0722a0 2023-10-30T13:10:50.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc29959a640 1 -- 172.21.15.148:0/3559747338 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900ab620 con 0x7fc29c0722a0 2023-10-30T13:10:50.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc2a0c09640 1 -- 172.21.15.148:0/3559747338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0722a0 msgr2=0x7fc29c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc2a0c09640 1 --2- 172.21.15.148:0/3559747338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0722a0 0x7fc29c0bfec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc2900097c0 tx=0x7fc29001c310 comp rx=0 tx=0).stop 2023-10-30T13:10:50.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc2a0c09640 1 -- 172.21.15.148:0/3559747338 shutdown_connections 2023-10-30T13:10:50.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc2a0c09640 1 --2- 172.21.15.148:0/3559747338 >> v2:172.21.15.150:3300/0 conn(0x7fc29c0c0400 0x7fc29c156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc2a0c09640 1 --2- 172.21.15.148:0/3559747338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0722a0 0x7fc29c0bfec0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc2a0c09640 1 -- 172.21.15.148:0/3559747338 >> 172.21.15.148:0/3559747338 conn(0x7fc29c0c7910 msgr2=0x7fc29c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:50.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc2a0c09640 1 -- 172.21.15.148:0/3559747338 shutdown_connections 2023-10-30T13:10:50.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.804+0000 7fc2a0c09640 1 -- 172.21.15.148:0/3559747338 wait complete. 2023-10-30T13:10:50.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 Processor -- start 2023-10-30T13:10:50.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 -- start start 2023-10-30T13:10:50.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc29c0722a0 0x7fc29c1f0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0c0400 0x7fc29c14bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc29c1508c0 0x7fc29c14f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c14ec40 con 0x7fc29c0722a0 2023-10-30T13:10:50.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c0c68c0 con 0x7fc29c1508c0 2023-10-30T13:10:50.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c0c2b70 con 0x7fc29c0c0400 2023-10-30T13:10:50.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc29c0722a0 0x7fc29c1f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc29c0722a0 0x7fc29c1f0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42192/0 (socket says 172.21.15.148:42192) 2023-10-30T13:10:50.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29a59c640 1 -- 172.21.15.148:0/2566539272 learned_addr learned my addr 172.21.15.148:0/2566539272 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:50.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc299d9b640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0c0400 0x7fc29c14bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29ad9d640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc29c1508c0 0x7fc29c14f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29a59c640 1 -- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0c0400 msgr2=0x7fc29c14bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29a59c640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0c0400 0x7fc29c14bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29a59c640 1 -- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc29c1508c0 msgr2=0x7fc29c14f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29a59c640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc29c1508c0 0x7fc29c14f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29a59c640 1 -- 172.21.15.148:0/2566539272 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc290009420 con 0x7fc29c0722a0 2023-10-30T13:10:50.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29ad9d640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc29c1508c0 0x7fc29c14f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:50.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc29a59c640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc29c0722a0 0x7fc29c1f0730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00c780 tx=0x7fc28c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:50.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc28affd640 1 -- 172.21.15.148:0/2566539272 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c007270 con 0x7fc29c0722a0 2023-10-30T13:10:50.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc28affd640 1 -- 172.21.15.148:0/2566539272 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c0073f0 con 0x7fc29c0722a0 2023-10-30T13:10:50.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c14e340 con 0x7fc29c0722a0 2023-10-30T13:10:50.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc28affd640 1 -- 172.21.15.148:0/2566539272 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0057b0 con 0x7fc29c0722a0 2023-10-30T13:10:50.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.808+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c0c6af0 con 0x7fc29c0722a0 2023-10-30T13:10:50.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.812+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c14edb0 con 0x7fc29c0722a0 2023-10-30T13:10:50.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.812+0000 7fc28affd640 1 -- 172.21.15.148:0/2566539272 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc28c005910 con 0x7fc29c0722a0 2023-10-30T13:10:50.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.812+0000 7fc28affd640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc27c0664a0 0x7fc27c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:50.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.812+0000 7fc28affd640 1 -- 172.21.15.148:0/2566539272 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc28c083c20 con 0x7fc29c0722a0 2023-10-30T13:10:50.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.812+0000 7fc299d9b640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc27c0664a0 0x7fc27c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:50.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.812+0000 7fc299d9b640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc27c0664a0 0x7fc27c068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2900ad000 tx=0x7fc2900a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:50.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.816+0000 7fc28affd640 1 -- 172.21.15.148:0/2566539272 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc28c051000 con 0x7fc29c0722a0 2023-10-30T13:10:50.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.964+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc29c04e170 con 0x7fc29c0722a0 2023-10-30T13:10:50.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.964+0000 7fc28affd640 1 -- 172.21.15.148:0/2566539272 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc28c054620 con 0x7fc29c0722a0 2023-10-30T13:10:50.969 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 64 2023-10-30T13:10:50.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.964+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc27c0664a0 msgr2=0x7fc27c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.964+0000 7fc2a0c09640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc27c0664a0 0x7fc27c068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2900ad000 tx=0x7fc2900a6000 comp rx=0 tx=0).stop 2023-10-30T13:10:50.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc29c0722a0 msgr2=0x7fc29c1f0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:50.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc29c0722a0 0x7fc29c1f0730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00c780 tx=0x7fc28c00cc40 comp rx=0 tx=0).stop 2023-10-30T13:10:50.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 shutdown_connections 2023-10-30T13:10:50.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc27c0664a0 0x7fc27c068960 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc29c1508c0 0x7fc29c14f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc29c0c0400 0x7fc29c14bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 --2- 172.21.15.148:0/2566539272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc29c0722a0 0x7fc29c1f0730 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:50.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 >> 172.21.15.148:0/2566539272 conn(0x7fc29c0c7910 msgr2=0x7fc29c146a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:50.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 shutdown_connections 2023-10-30T13:10:50.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:50.968+0000 7fc2a0c09640 1 -- 172.21.15.148:0/2566539272 wait complete. 2023-10-30T13:10:50.989 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:10:51.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6cb59e640 1 Processor -- start 2023-10-30T13:10:51.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6cb59e640 1 -- start start 2023-10-30T13:10:51.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6cb59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6cc0c6840 0x7fd6cc15a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6cb59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6cc0c3640 0x7fd6cc156590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6cb59e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc0c3b30 con 0x7fd6cc0c6c20 2023-10-30T13:10:51.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6cb59e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc0c2aa0 con 0x7fd6cc0c6840 2023-10-30T13:10:51.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6cb59e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc0c3df0 con 0x7fd6cc0c3640 2023-10-30T13:10:51.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6ca59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6cc0c6840 0x7fd6cc15a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6ca59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6cc0c6840 0x7fd6cc15a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42194/0 (socket says 172.21.15.148:42194) 2023-10-30T13:10:51.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6ca59c640 1 -- 172.21.15.148:0/2112082024 learned_addr learned my addr 172.21.15.148:0/2112082024 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:51.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6c9d9b640 1 --2- 172.21.15.148:0/2112082024 >> v2:172.21.15.150:3300/0 conn(0x7fd6cc0c3640 0x7fd6cc156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6ca59c640 1 -- 172.21.15.148:0/2112082024 >> v2:172.21.15.150:3300/0 conn(0x7fd6cc0c3640 msgr2=0x7fd6cc156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:51.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6ca59c640 1 --2- 172.21.15.148:0/2112082024 >> v2:172.21.15.150:3300/0 conn(0x7fd6cc0c3640 0x7fd6cc156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6ca59c640 1 -- 172.21.15.148:0/2112082024 >> v1:172.21.15.150:6789/0 conn(0x7fd6cc0c6c20 legacy=0x7fd6cc153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.124+0000 7fd6ca59c640 1 -- 172.21.15.148:0/2112082024 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc159ba0 con 0x7fd6cc0c6840 2023-10-30T13:10:51.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6ca59c640 1 --2- 172.21.15.148:0/2112082024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 0x7fd6cc15a680 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc06ef20 tx=0x7fd6bc01e710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987dd12df66b6942 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:51.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6c959a640 1 -- 172.21.15.148:0/2112082024 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0ab5f0 con 0x7fd6cc0c6840 2023-10-30T13:10:51.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6c959a640 1 -- 172.21.15.148:0/2112082024 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc0ab750 con 0x7fd6cc0c6840 2023-10-30T13:10:51.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6c959a640 1 -- 172.21.15.148:0/2112082024 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0abae0 con 0x7fd6cc0c6840 2023-10-30T13:10:51.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6cb59e640 1 -- 172.21.15.148:0/2112082024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 msgr2=0x7fd6cc15a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6cb59e640 1 --2- 172.21.15.148:0/2112082024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 0x7fd6cc15a680 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc06ef20 tx=0x7fd6bc01e710 comp rx=0 tx=0).stop 2023-10-30T13:10:51.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6cb59e640 1 -- 172.21.15.148:0/2112082024 shutdown_connections 2023-10-30T13:10:51.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6cb59e640 1 --2- 172.21.15.148:0/2112082024 >> v2:172.21.15.150:3300/0 conn(0x7fd6cc0c3640 0x7fd6cc156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6cb59e640 1 --2- 172.21.15.148:0/2112082024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 0x7fd6cc15a680 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6cb59e640 1 -- 172.21.15.148:0/2112082024 >> 172.21.15.148:0/2112082024 conn(0x7fd6cc0c75d0 msgr2=0x7fd6cc0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:51.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6cb59e640 1 -- 172.21.15.148:0/2112082024 shutdown_connections 2023-10-30T13:10:51.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.128+0000 7fd6cb59e640 1 -- 172.21.15.148:0/2112082024 wait complete. 2023-10-30T13:10:51.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 Processor -- start 2023-10-30T13:10:51.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 -- start start 2023-10-30T13:10:51.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc0c3640 0x7fd6cc145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 0x7fd6cc146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6cc0c6c20 0x7fd6cc1478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc14c550 con 0x7fd6cc0c6840 2023-10-30T13:10:51.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc0c1750 con 0x7fd6cc0c3640 2023-10-30T13:10:51.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc14a090 con 0x7fd6cc0c6c20 2023-10-30T13:10:51.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6c9d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 0x7fd6cc146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6c9d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 0x7fd6cc146ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42208/0 (socket says 172.21.15.148:42208) 2023-10-30T13:10:51.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6c9d9b640 1 -- 172.21.15.148:0/3630975410 learned_addr learned my addr 172.21.15.148:0/3630975410 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:51.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6ca59c640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc0c3640 0x7fd6cc145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cad9d640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6cc0c6c20 0x7fd6cc1478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6c9d9b640 1 -- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6cc0c6c20 msgr2=0x7fd6cc1478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6c9d9b640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6cc0c6c20 0x7fd6cc1478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6c9d9b640 1 -- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc0c3640 msgr2=0x7fd6cc145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6c9d9b640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc0c3640 0x7fd6cc145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6c9d9b640 1 -- 172.21.15.148:0/3630975410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc009480 con 0x7fd6cc0c6840 2023-10-30T13:10:51.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cad9d640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6cc0c6c20 0x7fd6cc1478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:51.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6c9d9b640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 0x7fd6cc146ed0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000e960 tx=0x7fd6c000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:51.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6baffd640 1 -- 172.21.15.148:0/3630975410 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0009690 con 0x7fd6cc0c6840 2023-10-30T13:10:51.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6baffd640 1 -- 172.21.15.148:0/3630975410 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c000ca00 con 0x7fd6cc0c6840 2023-10-30T13:10:51.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6baffd640 1 -- 172.21.15.148:0/3630975410 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c000cd70 con 0x7fd6cc0c6840 2023-10-30T13:10:51.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc1489d0 con 0x7fd6cc0c6840 2023-10-30T13:10:51.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc146110 con 0x7fd6cc0c6840 2023-10-30T13:10:51.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.132+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc04e170 con 0x7fd6cc0c6840 2023-10-30T13:10:51.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.136+0000 7fd6baffd640 1 -- 172.21.15.148:0/3630975410 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6c0013470 con 0x7fd6cc0c6840 2023-10-30T13:10:51.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.136+0000 7fd6baffd640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6ac0664a0 0x7fd6ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.136+0000 7fd6baffd640 1 -- 172.21.15.148:0/3630975410 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6c0084360 con 0x7fd6cc0c6840 2023-10-30T13:10:51.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.136+0000 7fd6ca59c640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6ac0664a0 0x7fd6ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.136+0000 7fd6ca59c640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6ac0664a0 0x7fd6ac068960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0b0000 tx=0x7fd6bc0a9000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:51.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.140+0000 7fd6baffd640 1 -- 172.21.15.148:0/3630975410 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6c0051760 con 0x7fd6cc0c6840 2023-10-30T13:10:51.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6cc1489d0 con 0x7fd6cc0c6840 2023-10-30T13:10:51.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6baffd640 1 -- 172.21.15.148:0/3630975410 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6540 (secure 0 0 0) 0x7fd6cc1489d0 con 0x7fd6cc0c6840 2023-10-30T13:10:51.324 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:51.324 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:46.327103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:creating","state_seq":1,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:10:51.324 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 12 2023-10-30T13:10:51.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6ac0664a0 msgr2=0x7fd6ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6ac0664a0 0x7fd6ac068960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0b0000 tx=0x7fd6bc0a9000 comp rx=0 tx=0).stop 2023-10-30T13:10:51.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 msgr2=0x7fd6cc146ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 0x7fd6cc146ed0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000e960 tx=0x7fd6c000ee20 comp rx=0 tx=0).stop 2023-10-30T13:10:51.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 shutdown_connections 2023-10-30T13:10:51.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6ac0664a0 0x7fd6ac068960 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc0c6840 0x7fd6cc146ed0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6cc0c6c20 0x7fd6cc1478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 --2- 172.21.15.148:0/3630975410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6cc0c3640 0x7fd6cc145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 >> 172.21.15.148:0/3630975410 conn(0x7fd6cc0c75d0 msgr2=0x7fd6cc14c700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:51.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 shutdown_connections 2023-10-30T13:10:51.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.320+0000 7fd6cb59e640 1 -- 172.21.15.148:0/3630975410 wait complete. 2023-10-30T13:10:51.351 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:51.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.484+0000 7fe7de054640 1 Processor -- start 2023-10-30T13:10:51.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.484+0000 7fe7de054640 1 -- start start 2023-10-30T13:10:51.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.484+0000 7fe7de054640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe7d80c6b80 0x7fe7d80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.484+0000 7fe7de054640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe7d813f4a0 0x7fe7d80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.484+0000 7fe7de054640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d80c0a50 con 0x7fe7d80c6f60 2023-10-30T13:10:51.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.484+0000 7fe7de054640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d806f6e0 con 0x7fe7d813f4a0 2023-10-30T13:10:51.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.484+0000 7fe7de054640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d806f9c0 con 0x7fe7d80c6b80 2023-10-30T13:10:51.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe7d813f4a0 0x7fe7d80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe7d813f4a0 0x7fe7d80c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42216/0 (socket says 172.21.15.148:42216) 2023-10-30T13:10:51.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d7fff640 1 -- 172.21.15.148:0/1613398727 learned_addr learned my addr 172.21.15.148:0/1613398727 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:51.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d77fe640 1 --2- 172.21.15.148:0/1613398727 >> v2:172.21.15.150:3300/0 conn(0x7fe7d80c6b80 0x7fe7d80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d7fff640 1 -- 172.21.15.148:0/1613398727 >> v2:172.21.15.150:3300/0 conn(0x7fe7d80c6b80 msgr2=0x7fe7d80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:51.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d7fff640 1 --2- 172.21.15.148:0/1613398727 >> v2:172.21.15.150:3300/0 conn(0x7fe7d80c6b80 0x7fe7d80bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d7fff640 1 -- 172.21.15.148:0/1613398727 >> v1:172.21.15.150:6789/0 conn(0x7fe7d80c6f60 legacy=0x7fe7d813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d7fff640 1 -- 172.21.15.148:0/1613398727 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d806fca0 con 0x7fe7d813f4a0 2023-10-30T13:10:51.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d77fe640 1 --2- 172.21.15.148:0/1613398727 >> v2:172.21.15.150:3300/0 conn(0x7fe7d80c6b80 0x7fe7d80bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:51.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d7fff640 1 --2- 172.21.15.148:0/1613398727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 0x7fe7d80c0400 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe7d806f1b0 tx=0x7fe7d001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c466b7d355c575f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:51.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d67fc640 1 -- 172.21.15.148:0/1613398727 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00a57a0 con 0x7fe7d813f4a0 2023-10-30T13:10:51.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d67fc640 1 -- 172.21.15.148:0/1613398727 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d00a5900 con 0x7fe7d813f4a0 2023-10-30T13:10:51.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.488+0000 7fe7d67fc640 1 -- 172.21.15.148:0/1613398727 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00a5c90 con 0x7fe7d813f4a0 2023-10-30T13:10:51.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 -- 172.21.15.148:0/1613398727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 msgr2=0x7fe7d80c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 --2- 172.21.15.148:0/1613398727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 0x7fe7d80c0400 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe7d806f1b0 tx=0x7fe7d001c9d0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 -- 172.21.15.148:0/1613398727 shutdown_connections 2023-10-30T13:10:51.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 --2- 172.21.15.148:0/1613398727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 0x7fe7d80c0400 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 --2- 172.21.15.148:0/1613398727 >> v2:172.21.15.150:3300/0 conn(0x7fe7d80c6b80 0x7fe7d80bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 -- 172.21.15.148:0/1613398727 >> 172.21.15.148:0/1613398727 conn(0x7fe7d80c7910 msgr2=0x7fe7d80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:51.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 -- 172.21.15.148:0/1613398727 shutdown_connections 2023-10-30T13:10:51.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 -- 172.21.15.148:0/1613398727 wait complete. 2023-10-30T13:10:51.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 Processor -- start 2023-10-30T13:10:51.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 -- start start 2023-10-30T13:10:51.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c6b80 0x7fe7d8135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d80c6f60 0x7fe7d8134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 0x7fe7d8152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d813adb0 con 0x7fe7d813f4a0 2023-10-30T13:10:51.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d813b240 con 0x7fe7d80c6b80 2023-10-30T13:10:51.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7de054640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d8158be0 con 0x7fe7d80c6f60 2023-10-30T13:10:51.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 0x7fe7d8152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 0x7fe7d8152f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42226/0 (socket says 172.21.15.148:42226) 2023-10-30T13:10:51.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d7fff640 1 -- 172.21.15.148:0/404341403 learned_addr learned my addr 172.21.15.148:0/404341403 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:51.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d77fe640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c6b80 0x7fe7d8135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d6ffd640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d80c6f60 0x7fe7d8134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d7fff640 1 -- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d80c6f60 msgr2=0x7fe7d8134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d7fff640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d80c6f60 0x7fe7d8134a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d7fff640 1 -- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c6b80 msgr2=0x7fe7d8135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d7fff640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c6b80 0x7fe7d8135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.492+0000 7fe7d7fff640 1 -- 172.21.15.148:0/404341403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d0009b30 con 0x7fe7d813f4a0 2023-10-30T13:10:51.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7d7fff640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 0x7fe7d8152f40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00af040 tx=0x7fe7d0019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:51.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7bffff640 1 -- 172.21.15.148:0/404341403 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00b8880 con 0x7fe7d813f4a0 2023-10-30T13:10:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7bffff640 1 -- 172.21.15.148:0/404341403 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d00b8a00 con 0x7fe7d813f4a0 2023-10-30T13:10:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d8154520 con 0x7fe7d813f4a0 2023-10-30T13:10:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7bffff640 1 -- 172.21.15.148:0/404341403 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00b8db0 con 0x7fe7d813f4a0 2023-10-30T13:10:51.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d8154ae0 con 0x7fe7d813f4a0 2023-10-30T13:10:51.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d804e170 con 0x7fe7d813f4a0 2023-10-30T13:10:51.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7bffff640 1 -- 172.21.15.148:0/404341403 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7d00b6440 con 0x7fe7d813f4a0 2023-10-30T13:10:51.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7bffff640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe7c40664a0 0x7fe7c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7bffff640 1 -- 172.21.15.148:0/404341403 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7d011f5f0 con 0x7fe7d813f4a0 2023-10-30T13:10:51.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.496+0000 7fe7d77fe640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe7c40664a0 0x7fe7c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.500+0000 7fe7d77fe640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe7c40664a0 0x7fe7c4068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800c900 tx=0x7fe7c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:51.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.500+0000 7fe7bffff640 1 -- 172.21.15.148:0/404341403 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7d00ecc20 con 0x7fe7d813f4a0 2023-10-30T13:10:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.652+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7d813aef0 con 0x7fe7d813f4a0 2023-10-30T13:10:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.652+0000 7fe7bffff640 1 -- 172.21.15.148:0/404341403 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fe7d00f0240 con 0x7fe7d813f4a0 2023-10-30T13:10:51.657 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:51.657 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:35.311442+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:51.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe7c40664a0 msgr2=0x7fe7c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe7c40664a0 0x7fe7c4068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800c900 tx=0x7fe7c8008040 comp rx=0 tx=0).stop 2023-10-30T13:10:51.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 msgr2=0x7fe7d8152f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 0x7fe7d8152f40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00af040 tx=0x7fe7d0019770 comp rx=0 tx=0).stop 2023-10-30T13:10:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 shutdown_connections 2023-10-30T13:10:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe7c40664a0 0x7fe7c4068960 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7d813f4a0 0x7fe7d8152f40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7d80c6f60 0x7fe7d8134a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 --2- 172.21.15.148:0/404341403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d80c6b80 0x7fe7d8135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 >> 172.21.15.148:0/404341403 conn(0x7fe7d80c7910 msgr2=0x7fe7d806a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:51.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 shutdown_connections 2023-10-30T13:10:51.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.656+0000 7fe7de054640 1 -- 172.21.15.148:0/404341403 wait complete. 2023-10-30T13:10:51.678 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:10:22.349322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:10:32.611151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:10:33.044931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:10:51.679 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:10:51.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bf7f8640 1 Processor -- start 2023-10-30T13:10:51.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bf7f8640 1 -- start start 2023-10-30T13:10:51.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bf7f8640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f38b8153980 0x7f38b80caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bf7f8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f38b8159930 0x7f38b80cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bf7f8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b815a920 con 0x7f38b80cf1a0 2023-10-30T13:10:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bf7f8640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b80cb930 con 0x7f38b8153980 2023-10-30T13:10:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bf7f8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b8159dc0 con 0x7f38b8159930 2023-10-30T13:10:51.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bd594640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f38b8153980 0x7f38b80caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bd594640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f38b8153980 0x7f38b80caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48510/0 (socket says 172.21.15.148:48510) 2023-10-30T13:10:51.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bd594640 1 -- 172.21.15.148:0/3551109061 learned_addr learned my addr 172.21.15.148:0/3551109061 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:51.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bcd93640 1 --2- 172.21.15.148:0/3551109061 >> v2:172.21.15.150:3300/0 conn(0x7f38b8159930 0x7f38b80cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bd594640 1 -- 172.21.15.148:0/3551109061 >> v2:172.21.15.150:3300/0 conn(0x7f38b8159930 msgr2=0x7f38b80cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:51.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bd594640 1 --2- 172.21.15.148:0/3551109061 >> v2:172.21.15.150:3300/0 conn(0x7f38b8159930 0x7f38b80cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bd594640 1 -- 172.21.15.148:0/3551109061 >> v1:172.21.15.150:6789/0 conn(0x7f38b80cf1a0 legacy=0x7f38b815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:51.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bd594640 1 -- 172.21.15.148:0/3551109061 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b815a080 con 0x7f38b8153980 2023-10-30T13:10:51.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.816+0000 7f38bcd93640 1 --2- 172.21.15.148:0/3551109061 >> v2:172.21.15.150:3300/0 conn(0x7f38b8159930 0x7f38b80cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:10:51.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38bd594640 1 --2- 172.21.15.148:0/3551109061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 0x7f38b80caca0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f38b80742c0 tx=0x7f38b401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18941f1ae6be789 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:51.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38affff640 1 -- 172.21.15.148:0/3551109061 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b40aa610 con 0x7f38b8153980 2023-10-30T13:10:51.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38affff640 1 -- 172.21.15.148:0/3551109061 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b40a6040 con 0x7f38b8153980 2023-10-30T13:10:51.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38affff640 1 -- 172.21.15.148:0/3551109061 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b40aa990 con 0x7f38b8153980 2023-10-30T13:10:51.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38bf7f8640 1 -- 172.21.15.148:0/3551109061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 msgr2=0x7f38b80caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38bf7f8640 1 --2- 172.21.15.148:0/3551109061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 0x7f38b80caca0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f38b80742c0 tx=0x7f38b401c490 comp rx=0 tx=0).stop 2023-10-30T13:10:51.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38bf7f8640 1 -- 172.21.15.148:0/3551109061 shutdown_connections 2023-10-30T13:10:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38bf7f8640 1 --2- 172.21.15.148:0/3551109061 >> v2:172.21.15.150:3300/0 conn(0x7f38b8159930 0x7f38b80cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38bf7f8640 1 --2- 172.21.15.148:0/3551109061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 0x7f38b80caca0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38bf7f8640 1 -- 172.21.15.148:0/3551109061 >> 172.21.15.148:0/3551109061 conn(0x7f38b806c0f0 msgr2=0x7f38b806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38bf7f8640 1 -- 172.21.15.148:0/3551109061 shutdown_connections 2023-10-30T13:10:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.820+0000 7f38bf7f8640 1 -- 172.21.15.148:0/3551109061 wait complete. 2023-10-30T13:10:51.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bf7f8640 1 Processor -- start 2023-10-30T13:10:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bf7f8640 1 -- start start 2023-10-30T13:10:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bf7f8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38b80cf1a0 0x7f38b815c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bf7f8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 0x7f38b815aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bf7f8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38b8158390 0x7f38b8151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bf7f8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80cd3c0 con 0x7f38b8153980 2023-10-30T13:10:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bf7f8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80cd100 con 0x7f38b80cf1a0 2023-10-30T13:10:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bf7f8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b81f25e0 con 0x7f38b8158390 2023-10-30T13:10:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bcd93640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 0x7f38b815aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bcd93640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 0x7f38b815aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48516/0 (socket says 172.21.15.148:48516) 2023-10-30T13:10:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bcd93640 1 -- 172.21.15.148:0/4122719643 learned_addr learned my addr 172.21.15.148:0/4122719643 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bd594640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38b80cf1a0 0x7f38b815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bdd95640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38b8158390 0x7f38b8151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bcd93640 1 -- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38b8158390 msgr2=0x7f38b8151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bcd93640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38b8158390 0x7f38b8151c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bcd93640 1 -- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38b80cf1a0 msgr2=0x7f38b815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bcd93640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38b80cf1a0 0x7f38b815c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bcd93640 1 -- 172.21.15.148:0/4122719643 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b40094d0 con 0x7f38b8153980 2023-10-30T13:10:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bdd95640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38b8158390 0x7f38b8151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bcd93640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 0x7f38b815aaf0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f38a800c7f0 tx=0x7f38a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38adffb640 1 -- 172.21.15.148:0/4122719643 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8007820 con 0x7f38b8153980 2023-10-30T13:10:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38adffb640 1 -- 172.21.15.148:0/4122719643 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a8007980 con 0x7f38b8153980 2023-10-30T13:10:51.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b81f28c0 con 0x7f38b8153980 2023-10-30T13:10:51.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.824+0000 7f38adffb640 1 -- 172.21.15.148:0/4122719643 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8007d30 con 0x7f38b8153980 2023-10-30T13:10:51.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.828+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b8151490 con 0x7f38b8153980 2023-10-30T13:10:51.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.828+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b804e170 con 0x7f38b8153980 2023-10-30T13:10:51.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.828+0000 7f38adffb640 1 -- 172.21.15.148:0/4122719643 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f38a80113f0 con 0x7f38b8153980 2023-10-30T13:10:51.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.828+0000 7f38adffb640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f389c0664a0 0x7f389c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:51.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.828+0000 7f38adffb640 1 -- 172.21.15.148:0/4122719643 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38a8084220 con 0x7f38b8153980 2023-10-30T13:10:51.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.828+0000 7f38bd594640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f389c0664a0 0x7f389c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:51.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.828+0000 7f38bd594640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f389c0664a0 0x7f389c068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f38b40ae000 tx=0x7f38b40b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:51.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.832+0000 7f38adffb640 1 -- 172.21.15.148:0/4122719643 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f38a8051620 con 0x7f38b8153980 2023-10-30T13:10:51.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38b80caa80 con 0x7f38b8153980 2023-10-30T13:10:51.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38adffb640 1 -- 172.21.15.148:0/4122719643 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38a8054c40 con 0x7f38b8153980 2023-10-30T13:10:51.987 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2023-10-30T13:10:51.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f389c0664a0 msgr2=0x7f389c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f389c0664a0 0x7f389c068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f38b40ae000 tx=0x7f38b40b5040 comp rx=0 tx=0).stop 2023-10-30T13:10:51.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 msgr2=0x7f38b815aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:51.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 0x7f38b815aaf0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f38a800c7f0 tx=0x7f38a800ccb0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 shutdown_connections 2023-10-30T13:10:51.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f389c0664a0 0x7f389c068960 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f38b8153980 0x7f38b815aaf0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f38b8158390 0x7f38b8151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 --2- 172.21.15.148:0/4122719643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38b80cf1a0 0x7f38b815c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:51.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 >> 172.21.15.148:0/4122719643 conn(0x7f38b806c0f0 msgr2=0x7f38b8158e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:51.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 shutdown_connections 2023-10-30T13:10:51.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:51.984+0000 7f38bf7f8640 1 -- 172.21.15.148:0/4122719643 wait complete. 2023-10-30T13:10:52.008 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:10:52.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.144+0000 7f254d70f640 1 Processor -- start 2023-10-30T13:10:52.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.144+0000 7f254d70f640 1 -- start start 2023-10-30T13:10:52.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.144+0000 7f254d70f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2548159930 0x7f25480cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.144+0000 7f254d70f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f25480cf6e0 0x7f254815d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.144+0000 7f254d70f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254815a920 con 0x7f2548153980 2023-10-30T13:10:52.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.144+0000 7f254d70f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25480cb930 con 0x7f25480cf6e0 2023-10-30T13:10:52.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.144+0000 7f254d70f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2548159dc0 con 0x7f2548159930 2023-10-30T13:10:52.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25477fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f25480cf6e0 0x7f254815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f2546ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2548153980 0x7f25480caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37362/0 (socket says 172.21.15.148:37362) 2023-10-30T13:10:52.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f2546ffd640 1 -- 172.21.15.148:0/1660528347 learned_addr learned my addr 172.21.15.148:0/1660528347 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:52.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25477fe640 1 --2- 172.21.15.148:0/1660528347 >> v2:172.21.15.148:3300/0 conn(0x7f25480cf6e0 0x7f254815d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48526/0 (socket says 172.21.15.148:48526) 2023-10-30T13:10:52.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f253e7fc640 1 --2- 172.21.15.148:0/1660528347 >> v2:172.21.15.150:3300/0 conn(0x7f2548159930 0x7f25480cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25477fe640 1 -- 172.21.15.148:0/1660528347 >> v2:172.21.15.150:3300/0 conn(0x7f2548159930 msgr2=0x7f25480cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:52.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25477fe640 1 --2- 172.21.15.148:0/1660528347 >> v2:172.21.15.150:3300/0 conn(0x7f2548159930 0x7f25480cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25477fe640 1 -- 172.21.15.148:0/1660528347 >> v1:172.21.15.148:6789/0 conn(0x7f2548153980 legacy=0x7f25480caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25477fe640 1 -- 172.21.15.148:0/1660528347 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254815a080 con 0x7f25480cf6e0 2023-10-30T13:10:52.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25477fe640 1 --2- 172.21.15.148:0/1660528347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 0x7f254815d570 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f25480742c0 tx=0x7f254001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5701cdca12d53a3c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:52.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25467fc640 1 -- 172.21.15.148:0/1660528347 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25400a58b0 con 0x7f25480cf6e0 2023-10-30T13:10:52.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25467fc640 1 -- 172.21.15.148:0/1660528347 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25400a5a10 con 0x7f25480cf6e0 2023-10-30T13:10:52.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.148+0000 7f25467fc640 1 -- 172.21.15.148:0/1660528347 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25400a5da0 con 0x7f25480cf6e0 2023-10-30T13:10:52.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 -- 172.21.15.148:0/1660528347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 msgr2=0x7f254815d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 --2- 172.21.15.148:0/1660528347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 0x7f254815d570 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f25480742c0 tx=0x7f254001c8e0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 -- 172.21.15.148:0/1660528347 shutdown_connections 2023-10-30T13:10:52.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 --2- 172.21.15.148:0/1660528347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 0x7f254815d570 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 --2- 172.21.15.148:0/1660528347 >> v2:172.21.15.150:3300/0 conn(0x7f2548159930 0x7f25480cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 -- 172.21.15.148:0/1660528347 >> 172.21.15.148:0/1660528347 conn(0x7f254806c0f0 msgr2=0x7f254806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:52.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 -- 172.21.15.148:0/1660528347 shutdown_connections 2023-10-30T13:10:52.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 -- 172.21.15.148:0/1660528347 wait complete. 2023-10-30T13:10:52.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 Processor -- start 2023-10-30T13:10:52.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 -- start start 2023-10-30T13:10:52.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 0x7f254815c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2548153980 0x7f254815c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f254815bbe0 0x7f25481434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25481523c0 con 0x7f25480cf6e0 2023-10-30T13:10:52.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25481430b0 con 0x7f2548153980 2023-10-30T13:10:52.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f254d70f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2548153150 con 0x7f254815bbe0 2023-10-30T13:10:52.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 0x7f254815c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f2546ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 0x7f254815c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48534/0 (socket says 172.21.15.148:48534) 2023-10-30T13:10:52.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f2546ffd640 1 -- 172.21.15.148:0/295730615 learned_addr learned my addr 172.21.15.148:0/295730615 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:52.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f253e7fc640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2548153980 0x7f254815c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f25477fe640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f254815bbe0 0x7f25481434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f2546ffd640 1 -- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f254815bbe0 msgr2=0x7f25481434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f2546ffd640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f254815bbe0 0x7f25481434e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f2546ffd640 1 -- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2548153980 msgr2=0x7f254815c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f2546ffd640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2548153980 0x7f254815c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.152+0000 7f2546ffd640 1 -- 172.21.15.148:0/295730615 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2540009a70 con 0x7f25480cf6e0 2023-10-30T13:10:52.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f2546ffd640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 0x7f254815c480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f253000c530 tx=0x7f253000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:52.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f253ffff640 1 -- 172.21.15.148:0/295730615 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530005cf0 con 0x7f25480cf6e0 2023-10-30T13:10:52.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f253ffff640 1 -- 172.21.15.148:0/295730615 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2530005e50 con 0x7f25480cf6e0 2023-10-30T13:10:52.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2548152bb0 con 0x7f25480cf6e0 2023-10-30T13:10:52.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f253ffff640 1 -- 172.21.15.148:0/295730615 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530007270 con 0x7f25480cf6e0 2023-10-30T13:10:52.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25481525d0 con 0x7f25480cf6e0 2023-10-30T13:10:52.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254804e170 con 0x7f25480cf6e0 2023-10-30T13:10:52.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f253ffff640 1 -- 172.21.15.148:0/295730615 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25300073d0 con 0x7f25480cf6e0 2023-10-30T13:10:52.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f253ffff640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25280664a0 0x7f2528068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f253ffff640 1 -- 172.21.15.148:0/295730615 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2530083380 con 0x7f25480cf6e0 2023-10-30T13:10:52.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.156+0000 7f253e7fc640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25280664a0 0x7f2528068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.160+0000 7f253e7fc640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25280664a0 0x7f2528068960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25481495c0 tx=0x7f2538008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:52.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.160+0000 7f253ffff640 1 -- 172.21.15.148:0/295730615 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2530050810 con 0x7f25480cf6e0 2023-10-30T13:10:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.308+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f25480cf440 con 0x7f25480cf6e0 2023-10-30T13:10:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.308+0000 7f253ffff640 1 -- 172.21.15.148:0/295730615 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2530053e30 con 0x7f25480cf6e0 2023-10-30T13:10:52.313 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 32 2023-10-30T13:10:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25280664a0 msgr2=0x7f2528068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25280664a0 0x7f2528068960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25481495c0 tx=0x7f2538008040 comp rx=0 tx=0).stop 2023-10-30T13:10:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 msgr2=0x7f254815c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 0x7f254815c480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f253000c530 tx=0x7f253000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 shutdown_connections 2023-10-30T13:10:52.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25280664a0 0x7f2528068960 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f254815bbe0 0x7f25481434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2548153980 0x7f254815c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 --2- 172.21.15.148:0/295730615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25480cf6e0 0x7f254815c480 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 >> 172.21.15.148:0/295730615 conn(0x7f254806c0f0 msgr2=0x7f2548158ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:52.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 shutdown_connections 2023-10-30T13:10:52.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.312+0000 7f254d70f640 1 -- 172.21.15.148:0/295730615 wait complete. 2023-10-30T13:10:52.334 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:10:52.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d413e640 1 Processor -- start 2023-10-30T13:10:52.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d413e640 1 -- start start 2023-10-30T13:10:52.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d413e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc6cc06fbd0 0x7fc6cc15d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d413e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc0c2870 0x7fc6cc15db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d413e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6cc0c5910 con 0x7fc6cc15d200 2023-10-30T13:10:52.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d413e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6cc0c3520 con 0x7fc6cc06fbd0 2023-10-30T13:10:52.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d413e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc0c2620 con 0x7fc6cc0c2870 2023-10-30T13:10:52.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d16d9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc6cc06fbd0 0x7fc6cc15d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d16d9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc6cc06fbd0 0x7fc6cc15d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48544/0 (socket says 172.21.15.148:48544) 2023-10-30T13:10:52.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d16d9640 1 -- 172.21.15.148:0/1611451525 learned_addr learned my addr 172.21.15.148:0/1611451525 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:52.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d26db640 1 --2- 172.21.15.148:0/1611451525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc0c2870 0x7fc6cc15db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d16d9640 1 -- 172.21.15.148:0/1611451525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc0c2870 msgr2=0x7fc6cc15db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:52.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d16d9640 1 --2- 172.21.15.148:0/1611451525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc0c2870 0x7fc6cc15db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d16d9640 1 -- 172.21.15.148:0/1611451525 >> v1:172.21.15.150:6789/0 conn(0x7fc6cc15d200 legacy=0x7fc6cc0c39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.472+0000 7fc6d16d9640 1 -- 172.21.15.148:0/1611451525 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc070130 con 0x7fc6cc06fbd0 2023-10-30T13:10:52.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d16d9640 1 --2- 172.21.15.148:0/1611451525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc06fbd0 0x7fc6cc15d5e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc009870 tx=0x7fc6bc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2722c4d41ac60cb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:52.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d0ed8640 1 -- 172.21.15.148:0/1611451525 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0aa670 con 0x7fc6cc06fbd0 2023-10-30T13:10:52.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d0ed8640 1 -- 172.21.15.148:0/1611451525 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6bc0aa7d0 con 0x7fc6cc06fbd0 2023-10-30T13:10:52.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d0ed8640 1 -- 172.21.15.148:0/1611451525 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0aab60 con 0x7fc6cc06fbd0 2023-10-30T13:10:52.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 -- 172.21.15.148:0/1611451525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc06fbd0 msgr2=0x7fc6cc15d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 --2- 172.21.15.148:0/1611451525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc06fbd0 0x7fc6cc15d5e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc009870 tx=0x7fc6bc01c510 comp rx=0 tx=0).stop 2023-10-30T13:10:52.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 -- 172.21.15.148:0/1611451525 shutdown_connections 2023-10-30T13:10:52.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 --2- 172.21.15.148:0/1611451525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc0c2870 0x7fc6cc15db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 --2- 172.21.15.148:0/1611451525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc06fbd0 0x7fc6cc15d5e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 -- 172.21.15.148:0/1611451525 >> 172.21.15.148:0/1611451525 conn(0x7fc6cc0ccfe0 msgr2=0x7fc6cc0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:52.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 -- 172.21.15.148:0/1611451525 shutdown_connections 2023-10-30T13:10:52.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 -- 172.21.15.148:0/1611451525 wait complete. 2023-10-30T13:10:52.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 Processor -- start 2023-10-30T13:10:52.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.476+0000 7fc6d413e640 1 -- start start 2023-10-30T13:10:52.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d413e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc06fbd0 0x7fc6cc1f55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d413e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc0c2870 0x7fc6cc0c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d413e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc15d200 0x7fc6cc1f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d413e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc06d7d0 con 0x7fc6cc0c2870 2023-10-30T13:10:52.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d413e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc159d90 con 0x7fc6cc15d200 2023-10-30T13:10:52.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d413e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc15a050 con 0x7fc6cc06fbd0 2023-10-30T13:10:52.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d16d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc0c2870 0x7fc6cc0c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d16d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc0c2870 0x7fc6cc0c7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48546/0 (socket says 172.21.15.148:48546) 2023-10-30T13:10:52.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d16d9640 1 -- 172.21.15.148:0/511298676 learned_addr learned my addr 172.21.15.148:0/511298676 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:52.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d1eda640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc06fbd0 0x7fc6cc1f55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d26db640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc15d200 0x7fc6cc1f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d16d9640 1 -- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc06fbd0 msgr2=0x7fc6cc1f55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d16d9640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc06fbd0 0x7fc6cc1f55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d16d9640 1 -- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc15d200 msgr2=0x7fc6cc1f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d16d9640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc15d200 0x7fc6cc1f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d16d9640 1 -- 172.21.15.148:0/511298676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc0094d0 con 0x7fc6cc0c2870 2023-10-30T13:10:52.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d16d9640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc0c2870 0x7fc6cc0c7ff0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc01a140 tx=0x7fc6bc01ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:52.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6c27fc640 1 -- 172.21.15.148:0/511298676 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0b3800 con 0x7fc6cc0c2870 2023-10-30T13:10:52.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6c27fc640 1 -- 172.21.15.148:0/511298676 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6bc0b3960 con 0x7fc6cc0c2870 2023-10-30T13:10:52.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6c27fc640 1 -- 172.21.15.148:0/511298676 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0b3d10 con 0x7fc6cc0c2870 2023-10-30T13:10:52.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6cc15a330 con 0x7fc6cc0c2870 2023-10-30T13:10:52.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6cc1f1740 con 0x7fc6cc0c2870 2023-10-30T13:10:52.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6cc04e170 con 0x7fc6cc0c2870 2023-10-30T13:10:52.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.480+0000 7fc6c27fc640 1 -- 172.21.15.148:0/511298676 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6bc0b2380 con 0x7fc6cc0c2870 2023-10-30T13:10:52.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.484+0000 7fc6c27fc640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6b00664a0 0x7fc6b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.484+0000 7fc6c27fc640 1 -- 172.21.15.148:0/511298676 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6bc11e700 con 0x7fc6cc0c2870 2023-10-30T13:10:52.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.484+0000 7fc6d1eda640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6b00664a0 0x7fc6b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.484+0000 7fc6d1eda640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6b00664a0 0x7fc6b0068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8007fd0 tx=0x7fc6c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:52.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.488+0000 7fc6c27fc640 1 -- 172.21.15.148:0/511298676 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6bc0ebb90 con 0x7fc6cc0c2870 2023-10-30T13:10:52.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.636+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc6cc0c61c0 con 0x7fc6cc0c2870 2023-10-30T13:10:52.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.636+0000 7fc6c27fc640 1 -- 172.21.15.148:0/511298676 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc6bc0ef1b0 con 0x7fc6cc0c2870 2023-10-30T13:10:52.641 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 64 2023-10-30T13:10:52.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6b00664a0 msgr2=0x7fc6b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6b00664a0 0x7fc6b0068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8007fd0 tx=0x7fc6c8009040 comp rx=0 tx=0).stop 2023-10-30T13:10:52.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc0c2870 msgr2=0x7fc6cc0c7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc0c2870 0x7fc6cc0c7ff0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc01a140 tx=0x7fc6bc01ad50 comp rx=0 tx=0).stop 2023-10-30T13:10:52.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 shutdown_connections 2023-10-30T13:10:52.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6b00664a0 0x7fc6b0068960 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6cc0c2870 0x7fc6cc0c7ff0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc15d200 0x7fc6cc1f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 --2- 172.21.15.148:0/511298676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6cc06fbd0 0x7fc6cc1f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 >> 172.21.15.148:0/511298676 conn(0x7fc6cc0ccfe0 msgr2=0x7fc6cc156c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:52.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 shutdown_connections 2023-10-30T13:10:52.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.640+0000 7fc6d413e640 1 -- 172.21.15.148:0/511298676 wait complete. 2023-10-30T13:10:52.658 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:10:52.661 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:10:52.663 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:10:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed1730f640 1 Processor -- start 2023-10-30T13:10:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed1730f640 1 -- start start 2023-10-30T13:10:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed1730f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fed101538a0 0x7fed1015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed1730f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed100cf160 0x7fed100cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed1730f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed100cb800 con 0x7fed100ca0e0 2023-10-30T13:10:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed1730f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1015dc30 con 0x7fed101538a0 2023-10-30T13:10:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed1730f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed100cb330 con 0x7fed100cf160 2023-10-30T13:10:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed150ab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fed101538a0 0x7fed1015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed148aa640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fed100ca0e0 0x7fed1015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:58754/0 (socket says 172.21.15.148:58754) 2023-10-30T13:10:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed148aa640 1 -- 172.21.15.148:0/1096400317 learned_addr learned my addr 172.21.15.148:0/1096400317 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed158ac640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed100cf160 0x7fed100cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed07fff640 1 -- 172.21.15.148:0/1096400317 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277475663 0 0) 0x7fed100cb800 con 0x7fed100ca0e0 2023-10-30T13:10:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed158ac640 1 -- 172.21.15.148:0/1096400317 >> v2:172.21.15.150:3300/0 conn(0x7fed101538a0 msgr2=0x7fed1015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed158ac640 1 --2- 172.21.15.148:0/1096400317 >> v2:172.21.15.150:3300/0 conn(0x7fed101538a0 0x7fed1015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed158ac640 1 -- 172.21.15.148:0/1096400317 >> v1:172.21.15.150:6789/0 conn(0x7fed100ca0e0 legacy=0x7fed1015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.844+0000 7fed158ac640 1 -- 172.21.15.148:0/1096400317 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed100cad70 con 0x7fed100cf160 2023-10-30T13:10:52.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed158ac640 1 --2- 172.21.15.148:0/1096400317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed100cf160 0x7fed100cf590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed08009d90 tx=0x7fed0801f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caadf10998a2e9a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed07fff640 1 -- 172.21.15.148:0/1096400317 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080af600 con 0x7fed100cf160 2023-10-30T13:10:52.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed07fff640 1 -- 172.21.15.148:0/1096400317 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed080ab050 con 0x7fed100cf160 2023-10-30T13:10:52.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed07fff640 1 -- 172.21.15.148:0/1096400317 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080af980 con 0x7fed100cf160 2023-10-30T13:10:52.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed1730f640 1 -- 172.21.15.148:0/1096400317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed100cf160 msgr2=0x7fed100cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed1730f640 1 --2- 172.21.15.148:0/1096400317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed100cf160 0x7fed100cf590 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed08009d90 tx=0x7fed0801f130 comp rx=0 tx=0).stop 2023-10-30T13:10:52.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed1730f640 1 -- 172.21.15.148:0/1096400317 shutdown_connections 2023-10-30T13:10:52.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed1730f640 1 --2- 172.21.15.148:0/1096400317 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed100cf160 0x7fed100cf590 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed1730f640 1 --2- 172.21.15.148:0/1096400317 >> v2:172.21.15.150:3300/0 conn(0x7fed101538a0 0x7fed1015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.848+0000 7fed1730f640 1 -- 172.21.15.148:0/1096400317 >> 172.21.15.148:0/1096400317 conn(0x7fed1006bfe0 msgr2=0x7fed1006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:52.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 -- 172.21.15.148:0/1096400317 shutdown_connections 2023-10-30T13:10:52.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 -- 172.21.15.148:0/1096400317 wait complete. 2023-10-30T13:10:52.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 Processor -- start 2023-10-30T13:10:52.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 -- start start 2023-10-30T13:10:52.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed100ca0e0 0x7fed10148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fed100cf160 0x7fed100b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed101538a0 0x7fed101f0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed101485a0 con 0x7fed100cf160 2023-10-30T13:10:52.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed10159520 con 0x7fed100ca0e0 2023-10-30T13:10:52.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed100c4710 con 0x7fed101538a0 2023-10-30T13:10:52.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed148aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fed100cf160 0x7fed100b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed148aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fed100cf160 0x7fed100b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48556/0 (socket says 172.21.15.148:48556) 2023-10-30T13:10:52.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed148aa640 1 -- 172.21.15.148:0/1009023170 learned_addr learned my addr 172.21.15.148:0/1009023170 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed158ac640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed101538a0 0x7fed101f0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed150ab640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed100ca0e0 0x7fed10148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed148aa640 1 -- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed101538a0 msgr2=0x7fed101f0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed148aa640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed101538a0 0x7fed101f0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed148aa640 1 -- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed100ca0e0 msgr2=0x7fed10148060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed148aa640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed100ca0e0 0x7fed10148060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed148aa640 1 -- 172.21.15.148:0/1009023170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08009a70 con 0x7fed100cf160 2023-10-30T13:10:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed150ab640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed100ca0e0 0x7fed10148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed148aa640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fed100cf160 0x7fed100b4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fed000011f0 tx=0x7fed0000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:52.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed05ffb640 1 -- 172.21.15.148:0/1009023170 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed000074e0 con 0x7fed100cf160 2023-10-30T13:10:52.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed05ffb640 1 -- 172.21.15.148:0/1009023170 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed00007640 con 0x7fed100cf160 2023-10-30T13:10:52.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed101487a0 con 0x7fed100cf160 2023-10-30T13:10:52.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.852+0000 7fed05ffb640 1 -- 172.21.15.148:0/1009023170 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00004a10 con 0x7fed100cf160 2023-10-30T13:10:52.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.856+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed10146bb0 con 0x7fed100cf160 2023-10-30T13:10:52.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.856+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed100b2a50 con 0x7fed100cf160 2023-10-30T13:10:52.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.856+0000 7fed05ffb640 1 -- 172.21.15.148:0/1009023170 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed00004b70 con 0x7fed100cf160 2023-10-30T13:10:52.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.856+0000 7fed05ffb640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecf40664a0 0x7fecf4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:52.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.856+0000 7fed05ffb640 1 -- 172.21.15.148:0/1009023170 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed000834e0 con 0x7fed100cf160 2023-10-30T13:10:52.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.856+0000 7fed150ab640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecf40664a0 0x7fecf4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:52.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.860+0000 7fed150ab640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecf40664a0 0x7fecf4068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009b10 tx=0x7fed0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:52.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:52.860+0000 7fed05ffb640 1 -- 172.21.15.148:0/1009023170 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed000508e0 con 0x7fed100cf160 2023-10-30T13:10:53.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed1004e170 con 0x7fed100cf160 2023-10-30T13:10:53.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed05ffb640 1 -- 172.21.15.148:0/1009023170 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fed00018090 con 0x7fed100cf160 2023-10-30T13:10:53.044 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:53.045 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:10:53.045 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:10:53.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecf40664a0 msgr2=0x7fecf4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecf40664a0 0x7fecf4068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009b10 tx=0x7fed0c008040 comp rx=0 tx=0).stop 2023-10-30T13:10:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fed100cf160 msgr2=0x7fed100b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fed100cf160 0x7fed100b4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fed000011f0 tx=0x7fed0000c9b0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 shutdown_connections 2023-10-30T13:10:53.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecf40664a0 0x7fecf4068960 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fed100cf160 0x7fed100b4580 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fed101538a0 0x7fed101f0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 --2- 172.21.15.148:0/1009023170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed100ca0e0 0x7fed10148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.040+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 >> 172.21.15.148:0/1009023170 conn(0x7fed1006bfe0 msgr2=0x7fed10152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:53.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.044+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 shutdown_connections 2023-10-30T13:10:53.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.044+0000 7fed1730f640 1 -- 172.21.15.148:0/1009023170 wait complete. 2023-10-30T13:10:53.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:10:53.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.196+0000 7f610726b640 1 Processor -- start 2023-10-30T13:10:53.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f610726b640 1 -- start start 2023-10-30T13:10:53.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f610726b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f61001538a0 0x7f610015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f610726b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61000cb800 con 0x7f61000cf160 2023-10-30T13:10:53.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f610726b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610015dc30 con 0x7f61000ca0e0 2023-10-30T13:10:53.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f610726b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61000cb330 con 0x7f61001538a0 2023-10-30T13:10:53.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105808640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f61000cf160 0x7f6100164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37366/0 (socket says 172.21.15.148:37366) 2023-10-30T13:10:53.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105808640 1 -- 172.21.15.148:0/2057726300 learned_addr learned my addr 172.21.15.148:0/2057726300 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:53.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105007640 1 --2- 172.21.15.148:0/2057726300 >> v2:172.21.15.150:3300/0 conn(0x7f61001538a0 0x7f610015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654827988 0 0) 0x7f61000cb800 con 0x7f61000cf160 2023-10-30T13:10:53.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60ec003530 con 0x7f61000cf160 2023-10-30T13:10:53.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3696419879 0 0) 0x7f60ec003530 con 0x7f61000cf160 2023-10-30T13:10:53.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 >> v2:172.21.15.150:3300/0 conn(0x7f61001538a0 msgr2=0x7f610015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:53.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 --2- 172.21.15.148:0/2057726300 >> v2:172.21.15.150:3300/0 conn(0x7f61001538a0 0x7f610015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 >> v1:172.21.15.150:6789/0 conn(0x7f61000ca0e0 legacy=0x7f610015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000cad70 con 0x7f61000cf160 2023-10-30T13:10:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f60f8001a00 con 0x7f61000cf160 2023-10-30T13:10:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 >> v1:172.21.15.148:6789/0 conn(0x7f61000cf160 legacy=0x7f6100164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec003980 0x7f60ec003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60ec004380 0x7f60ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ec005050 0x7f60ec00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61000cb330 con 0x7f60ec005050 2023-10-30T13:10:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec00bf00 con 0x7f60ec003980 2023-10-30T13:10:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec00c0c0 con 0x7f60ec004380 2023-10-30T13:10:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105808640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ec005050 0x7f60ec00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105007640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec003980 0x7f60ec003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6104806640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60ec004380 0x7f60ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105808640 1 -- 172.21.15.148:0/2057726300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60ec004380 msgr2=0x7f60ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:53.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105808640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f60ec004380 0x7f60ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105808640 1 -- 172.21.15.148:0/2057726300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec003980 msgr2=0x7f60ec003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:53.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105808640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec003980 0x7f60ec003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.200+0000 7f6105808640 1 -- 172.21.15.148:0/2057726300 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec003530 con 0x7f60ec005050 2023-10-30T13:10:53.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f6105808640 1 -- 172.21.15.148:0/2057726300 reap_dead start 2023-10-30T13:10:53.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f6105808640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ec005050 0x7f60ec00b990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f60f800a7a0 tx=0x7f60f801d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cd5468c98e97fe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:53.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f80ad630 con 0x7f60ec005050 2023-10-30T13:10:53.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f80a8070 con 0x7f60ec005050 2023-10-30T13:10:53.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f60f7fff640 1 -- 172.21.15.148:0/2057726300 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f80ad9f0 con 0x7f60ec005050 2023-10-30T13:10:53.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f610726b640 1 -- 172.21.15.148:0/2057726300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ec005050 msgr2=0x7f60ec00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f610726b640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ec005050 0x7f60ec00b990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f60f800a7a0 tx=0x7f60f801d2a0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f610726b640 1 -- 172.21.15.148:0/2057726300 shutdown_connections 2023-10-30T13:10:53.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f610726b640 1 --2- 172.21.15.148:0/2057726300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ec005050 0x7f60ec00b990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f610726b640 1 -- 172.21.15.148:0/2057726300 >> 172.21.15.148:0/2057726300 conn(0x7f610006bfe0 msgr2=0x7f610006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:53.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f610726b640 1 -- 172.21.15.148:0/2057726300 shutdown_connections 2023-10-30T13:10:53.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.204+0000 7f610726b640 1 -- 172.21.15.148:0/2057726300 wait complete. 2023-10-30T13:10:53.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 Processor -- start 2023-10-30T13:10:53.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 -- start start 2023-10-30T13:10:53.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec005050 0x7f61001ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61000ca550 0x7f61001e3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61001e4760 0x7f61000b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610015be00 con 0x7f61001e4760 2023-10-30T13:10:53.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61001e55b0 con 0x7f60ec005050 2023-10-30T13:10:53.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61001e5890 con 0x7f61000ca550 2023-10-30T13:10:53.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105808640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61001e4760 0x7f61000b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105808640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61001e4760 0x7f61000b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48572/0 (socket says 172.21.15.148:48572) 2023-10-30T13:10:53.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105007640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec005050 0x7f61001ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105808640 1 -- 172.21.15.148:0/4023723305 learned_addr learned my addr 172.21.15.148:0/4023723305 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:53.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6104806640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61000ca550 0x7f61001e3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105808640 1 -- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61000ca550 msgr2=0x7f61001e3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105808640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61000ca550 0x7f61001e3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105808640 1 -- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec005050 msgr2=0x7f61001ef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105808640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec005050 0x7f61001ef0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105808640 1 -- 172.21.15.148:0/4023723305 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61001e36c0 con 0x7f61001e4760 2023-10-30T13:10:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6104806640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61000ca550 0x7f61001e3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f6105808640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61001e4760 0x7f61000b47d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f60f800a7a0 tx=0x7f60ec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f60f5ffb640 1 -- 172.21.15.148:0/4023723305 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f80b9670 con 0x7f61001e4760 2023-10-30T13:10:53.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f60f5ffb640 1 -- 172.21.15.148:0/4023723305 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f80b97d0 con 0x7f61001e4760 2023-10-30T13:10:53.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f60f5ffb640 1 -- 172.21.15.148:0/4023723305 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f80b9b60 con 0x7f61001e4760 2023-10-30T13:10:53.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61001e39a0 con 0x7f61001e4760 2023-10-30T13:10:53.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.208+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61000b4d10 con 0x7f61001e4760 2023-10-30T13:10:53.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.212+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61001e5a20 con 0x7f61001e4760 2023-10-30T13:10:53.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.212+0000 7f60f5ffb640 1 -- 172.21.15.148:0/4023723305 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60f80b6070 con 0x7f61001e4760 2023-10-30T13:10:53.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.212+0000 7f60f5ffb640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60e40664d0 0x7f60e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.212+0000 7f60f5ffb640 1 -- 172.21.15.148:0/4023723305 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60f811dd40 con 0x7f61001e4760 2023-10-30T13:10:53.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.212+0000 7f6105007640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60e40664d0 0x7f60e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.212+0000 7f6105007640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60e40664d0 0x7f60e4068990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f60fc009b60 tx=0x7f60fc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:53.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.216+0000 7f60f5ffb640 1 -- 172.21.15.148:0/4023723305 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60f80eb270 con 0x7f61001e4760 2023-10-30T13:10:53.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f610004e170 con 0x7f61001e4760 2023-10-30T13:10:53.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f60f5ffb640 1 -- 172.21.15.148:0/4023723305 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f60f80ee890 con 0x7f61001e4760 2023-10-30T13:10:53.372 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:53.373 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:10:35.311442+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:10:53.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60e40664d0 msgr2=0x7f60e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60e40664d0 0x7f60e4068990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f60fc009b60 tx=0x7f60fc009190 comp rx=0 tx=0).stop 2023-10-30T13:10:53.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61001e4760 msgr2=0x7f61000b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61001e4760 0x7f61000b47d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f60f800a7a0 tx=0x7f60ec005800 comp rx=0 tx=0).stop 2023-10-30T13:10:53.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 shutdown_connections 2023-10-30T13:10:53.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60e40664d0 0x7f60e4068990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61001e4760 0x7f61000b47d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61000ca550 0x7f61001e3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 --2- 172.21.15.148:0/4023723305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec005050 0x7f61001ef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.368+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 >> 172.21.15.148:0/4023723305 conn(0x7f610006bfe0 msgr2=0x7f61000cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:53.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.372+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 shutdown_connections 2023-10-30T13:10:53.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.372+0000 7f610726b640 1 -- 172.21.15.148:0/4023723305 wait complete. 2023-10-30T13:10:53.394 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:10:53.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff848d0f640 1 Processor -- start 2023-10-30T13:10:53.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff848d0f640 1 -- start start 2023-10-30T13:10:53.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff848d0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff8441538a0 0x7ff84415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff848d0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 0x7ff84415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff848d0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff8440cf160 0x7ff8440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff848d0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8440cb800 con 0x7ff8441538a0 2023-10-30T13:10:53.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff848d0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84415dc30 con 0x7ff8440cf160 2023-10-30T13:10:53.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff848d0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8440cb330 con 0x7ff8440ca0e0 2023-10-30T13:10:53.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff84259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff8441538a0 0x7ff84415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff84259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff8441538a0 0x7ff84415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48584/0 (socket says 172.21.15.148:48584) 2023-10-30T13:10:53.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff84259c640 1 -- 172.21.15.148:0/2659386850 learned_addr learned my addr 172.21.15.148:0/2659386850 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:53.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff839d9b640 1 --2- 172.21.15.148:0/2659386850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 0x7ff84415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff842d9d640 1 --2- 172.21.15.148:0/2659386850 >> v2:172.21.15.150:3300/0 conn(0x7ff8440cf160 0x7ff8440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff84259c640 1 -- 172.21.15.148:0/2659386850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 msgr2=0x7ff84415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:53.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff84259c640 1 --2- 172.21.15.148:0/2659386850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 0x7ff84415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff84259c640 1 -- 172.21.15.148:0/2659386850 >> v2:172.21.15.150:3300/0 conn(0x7ff8440cf160 msgr2=0x7ff8440cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:53.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff84259c640 1 --2- 172.21.15.148:0/2659386850 >> v2:172.21.15.150:3300/0 conn(0x7ff8440cf160 0x7ff8440cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff84259c640 1 -- 172.21.15.148:0/2659386850 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8440cad70 con 0x7ff8441538a0 2023-10-30T13:10:53.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.528+0000 7ff842d9d640 1 --2- 172.21.15.148:0/2659386850 >> v2:172.21.15.150:3300/0 conn(0x7ff8440cf160 0x7ff8440cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:53.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff84259c640 1 --2- 172.21.15.148:0/2659386850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 0x7ff84415d6f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff84406cb00 tx=0x7ff83401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7578ab463488569f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:53.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff841d9b640 1 -- 172.21.15.148:0/2659386850 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8340a89b0 con 0x7ff8441538a0 2023-10-30T13:10:53.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff841d9b640 1 -- 172.21.15.148:0/2659386850 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8340a8b10 con 0x7ff8441538a0 2023-10-30T13:10:53.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff841d9b640 1 -- 172.21.15.148:0/2659386850 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8340a8ea0 con 0x7ff8441538a0 2023-10-30T13:10:53.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff848d0f640 1 -- 172.21.15.148:0/2659386850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 msgr2=0x7ff84415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff848d0f640 1 --2- 172.21.15.148:0/2659386850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 0x7ff84415d6f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff84406cb00 tx=0x7ff83401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff848d0f640 1 -- 172.21.15.148:0/2659386850 shutdown_connections 2023-10-30T13:10:53.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff848d0f640 1 --2- 172.21.15.148:0/2659386850 >> v2:172.21.15.150:3300/0 conn(0x7ff8440cf160 0x7ff8440cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff848d0f640 1 --2- 172.21.15.148:0/2659386850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 0x7ff84415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff848d0f640 1 --2- 172.21.15.148:0/2659386850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 0x7ff84415d6f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff848d0f640 1 -- 172.21.15.148:0/2659386850 >> 172.21.15.148:0/2659386850 conn(0x7ff84406bfe0 msgr2=0x7ff84406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:53.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff848d0f640 1 -- 172.21.15.148:0/2659386850 shutdown_connections 2023-10-30T13:10:53.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.532+0000 7ff848d0f640 1 -- 172.21.15.148:0/2659386850 wait complete. 2023-10-30T13:10:53.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 Processor -- start 2023-10-30T13:10:53.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 -- start start 2023-10-30T13:10:53.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 0x7ff8441f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8440cf160 0x7ff8440c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 0x7ff84414cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff844159420 con 0x7ff8441538a0 2023-10-30T13:10:53.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8441f2c10 con 0x7ff8440cf160 2023-10-30T13:10:53.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff844159650 con 0x7ff8440ca0e0 2023-10-30T13:10:53.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff842d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 0x7ff84414cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff842d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 0x7ff84414cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48588/0 (socket says 172.21.15.148:48588) 2023-10-30T13:10:53.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff842d9d640 1 -- 172.21.15.148:0/743560297 learned_addr learned my addr 172.21.15.148:0/743560297 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:53.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff839d9b640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8440cf160 0x7ff8440c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff84259c640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 0x7ff8441f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff842d9d640 1 -- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 msgr2=0x7ff8441f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff842d9d640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 0x7ff8441f30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff842d9d640 1 -- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8440cf160 msgr2=0x7ff8440c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff842d9d640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8440cf160 0x7ff8440c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff842d9d640 1 -- 172.21.15.148:0/743560297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8340094d0 con 0x7ff8441538a0 2023-10-30T13:10:53.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff842d9d640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 0x7ff84414cab0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff83c007de0 tx=0x7ff83c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:53.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff83b7fe640 1 -- 172.21.15.148:0/743560297 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c0039a0 con 0x7ff8441538a0 2023-10-30T13:10:53.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff83b7fe640 1 -- 172.21.15.148:0/743560297 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83c00f040 con 0x7ff8441538a0 2023-10-30T13:10:53.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8441e2f50 con 0x7ff8441538a0 2023-10-30T13:10:53.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff83b7fe640 1 -- 172.21.15.148:0/743560297 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c003d40 con 0x7ff8441538a0 2023-10-30T13:10:53.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8441f28a0 con 0x7ff8441538a0 2023-10-30T13:10:53.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.536+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8440b2a50 con 0x7ff8441538a0 2023-10-30T13:10:53.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.540+0000 7ff83b7fe640 1 -- 172.21.15.148:0/743560297 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff83c004030 con 0x7ff8441538a0 2023-10-30T13:10:53.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.540+0000 7ff83b7fe640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8240664a0 0x7ff824068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:53.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.540+0000 7ff83b7fe640 1 -- 172.21.15.148:0/743560297 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff83c084190 con 0x7ff8441538a0 2023-10-30T13:10:53.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.540+0000 7ff84259c640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8240664a0 0x7ff824068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:53.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.540+0000 7ff84259c640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8240664a0 0x7ff824068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff8340b2000 tx=0x7ff834009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:53.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.544+0000 7ff83b7fe640 1 -- 172.21.15.148:0/743560297 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff83c0516b0 con 0x7ff8441538a0 2023-10-30T13:10:53.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.724+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8441f2d80 con 0x7ff8441538a0 2023-10-30T13:10:53.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.724+0000 7ff83b7fe640 1 -- 172.21.15.148:0/743560297 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7ff83c054cd0 con 0x7ff8441538a0 2023-10-30T13:10:53.730 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:10:53.731 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:10:53.731 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:10:53.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8240664a0 msgr2=0x7ff824068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8240664a0 0x7ff824068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff8340b2000 tx=0x7ff834009040 comp rx=0 tx=0).stop 2023-10-30T13:10:53.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 msgr2=0x7ff84414cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:53.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 0x7ff84414cab0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff83c007de0 tx=0x7ff83c00d4a0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 shutdown_connections 2023-10-30T13:10:53.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff8240664a0 0x7ff824068960 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8441538a0 0x7ff84414cab0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8440cf160 0x7ff8440c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 --2- 172.21.15.148:0/743560297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff8440ca0e0 0x7ff8441f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:53.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 >> 172.21.15.148:0/743560297 conn(0x7ff84406bfe0 msgr2=0x7ff844151d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:53.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 shutdown_connections 2023-10-30T13:10:53.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:53.728+0000 7ff848d0f640 1 -- 172.21.15.148:0/743560297 wait complete. 2023-10-30T13:10:54.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T13:10:34.058265+0000', 'modified': '2023-10-30T13:10:51.357311+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4285, 'mds_1': 4270, 'mds_2': 4256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4256': {'gid': 4256, 'name': 'c', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.148:6837/1844602519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6836', 'nonce': 1844602519}, {'type': 'v1', 'addr': '172.21.15.148:6837', 'nonce': 1844602519}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4270': {'gid': 4270, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.150:6836/1620877178', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 1620877178}, {'type': 'v1', 'addr': '172.21.15.150:6836', 'nonce': 1620877178}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4285': {'gid': 4285, 'name': 'e', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.148:6834/4162935019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6832', 'nonce': 4162935019}, {'type': 'v1', 'addr': '172.21.15.148:6834', 'nonce': 4162935019}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T13:10:54.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-30T13:10:54.356 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-30T13:10:54.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.492+0000 7f26560c9640 1 Processor -- start 2023-10-30T13:10:54.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.492+0000 7f26560c9640 1 -- start start 2023-10-30T13:10:54.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.492+0000 7f26560c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2650150130 0x7f26500c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.492+0000 7f26560c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26500c18c0 0x7f26500c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f26560c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2650155ce0 con 0x7f2650069e90 2023-10-30T13:10:54.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f26560c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2650150510 con 0x7f2650150130 2023-10-30T13:10:54.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f26560c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26500c2620 con 0x7f26500c18c0 2023-10-30T13:10:54.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2650069e90 0x7f26501561a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37372/0 (socket says 172.21.15.148:37372) 2023-10-30T13:10:54.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264ffff640 1 -- 172.21.15.148:0/750978349 learned_addr learned my addr 172.21.15.148:0/750978349 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:10:54.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264effd640 1 --2- 172.21.15.148:0/750978349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26500c18c0 0x7f26500c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264f7fe640 1 --2- 172.21.15.148:0/750978349 >> v2:172.21.15.150:3300/0 conn(0x7f2650150130 0x7f26500c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264e7fc640 1 -- 172.21.15.148:0/750978349 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279441043 0 0) 0x7f2650155ce0 con 0x7f2650069e90 2023-10-30T13:10:54.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264e7fc640 1 -- 172.21.15.148:0/750978349 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2638003530 con 0x7f2650069e90 2023-10-30T13:10:54.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264effd640 1 -- 172.21.15.148:0/750978349 >> v2:172.21.15.150:3300/0 conn(0x7f2650150130 msgr2=0x7f26500c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:54.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264effd640 1 --2- 172.21.15.148:0/750978349 >> v2:172.21.15.150:3300/0 conn(0x7f2650150130 0x7f26500c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264effd640 1 -- 172.21.15.148:0/750978349 >> v1:172.21.15.148:6789/0 conn(0x7f2650069e90 legacy=0x7f26501561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:54.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264e7fc640 1 -- 172.21.15.148:0/750978349 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1739436136 0 0) 0x7f2638003530 con 0x7f2650069e90 2023-10-30T13:10:54.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264effd640 1 -- 172.21.15.148:0/750978349 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26500c3f60 con 0x7f26500c18c0 2023-10-30T13:10:54.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264f7fe640 1 --2- 172.21.15.148:0/750978349 >> v2:172.21.15.150:3300/0 conn(0x7f2650150130 0x7f26500c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:10:54.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.496+0000 7f264effd640 1 --2- 172.21.15.148:0/750978349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26500c18c0 0x7f26500c20e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26440097c0 tx=0x7f264401c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d63c0862a5a84036 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:54.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f264e7fc640 1 -- 172.21.15.148:0/750978349 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648001a00 con 0x7f26500c18c0 2023-10-30T13:10:54.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f264e7fc640 1 -- 172.21.15.148:0/750978349 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26440a7050 con 0x7f26500c18c0 2023-10-30T13:10:54.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f264e7fc640 1 -- 172.21.15.148:0/750978349 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440ab620 con 0x7f26500c18c0 2023-10-30T13:10:54.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 -- 172.21.15.148:0/750978349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26500c18c0 msgr2=0x7f26500c20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:54.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 --2- 172.21.15.148:0/750978349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26500c18c0 0x7f26500c20e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26440097c0 tx=0x7f264401c310 comp rx=0 tx=0).stop 2023-10-30T13:10:54.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 -- 172.21.15.148:0/750978349 shutdown_connections 2023-10-30T13:10:54.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 --2- 172.21.15.148:0/750978349 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26500c18c0 0x7f26500c20e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 --2- 172.21.15.148:0/750978349 >> v2:172.21.15.150:3300/0 conn(0x7f2650150130 0x7f26500c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 -- 172.21.15.148:0/750978349 >> 172.21.15.148:0/750978349 conn(0x7f26500cace0 msgr2=0x7f26500cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:54.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 -- 172.21.15.148:0/750978349 shutdown_connections 2023-10-30T13:10:54.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 -- 172.21.15.148:0/750978349 wait complete. 2023-10-30T13:10:54.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 Processor -- start 2023-10-30T13:10:54.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.500+0000 7f26560c9640 1 -- start start 2023-10-30T13:10:54.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f26560c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2650069e90 0x7f26501effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f26560c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26500c18c0 0x7f26501f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f26560c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2650150130 0x7f2650155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f26560c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26501f1fe0 con 0x7f2650069e90 2023-10-30T13:10:54.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f26560c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2650071770 con 0x7f26500c18c0 2023-10-30T13:10:54.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f26560c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2650071a50 con 0x7f2650150130 2023-10-30T13:10:54.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2650069e90 0x7f26501effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2650069e90 0x7f26501effc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48602/0 (socket says 172.21.15.148:48602) 2023-10-30T13:10:54.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264f7fe640 1 -- 172.21.15.148:0/3440341369 learned_addr learned my addr 172.21.15.148:0/3440341369 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:54.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264ffff640 1 --2- 172.21.15.148:0/3440341369 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2650150130 0x7f2650155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264effd640 1 --2- 172.21.15.148:0/3440341369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26500c18c0 0x7f26501f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264f7fe640 1 -- 172.21.15.148:0/3440341369 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2650150130 msgr2=0x7f2650155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:54.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264f7fe640 1 --2- 172.21.15.148:0/3440341369 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2650150130 0x7f2650155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264f7fe640 1 -- 172.21.15.148:0/3440341369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26500c18c0 msgr2=0x7f26501f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:54.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264f7fe640 1 --2- 172.21.15.148:0/3440341369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26500c18c0 0x7f26501f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264f7fe640 1 -- 172.21.15.148:0/3440341369 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2640009420 con 0x7f2650069e90 2023-10-30T13:10:54.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f264f7fe640 1 --2- 172.21.15.148:0/3440341369 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2650069e90 0x7f26501effc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f264000e980 tx=0x7f264000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:54.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f2637fff640 1 -- 172.21.15.148:0/3440341369 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264000ca00 con 0x7f2650069e90 2023-10-30T13:10:54.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f2637fff640 1 -- 172.21.15.148:0/3440341369 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264000cb60 con 0x7f2650069e90 2023-10-30T13:10:54.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f26560c9640 1 -- 172.21.15.148:0/3440341369 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2644009420 con 0x7f2650069e90 2023-10-30T13:10:54.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f2637fff640 1 -- 172.21.15.148:0/3440341369 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2640013470 con 0x7f2650069e90 2023-10-30T13:10:54.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f26560c9640 1 -- 172.21.15.148:0/3440341369 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26501f2580 con 0x7f2650069e90 2023-10-30T13:10:54.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.504+0000 7f2637fff640 1 -- 172.21.15.148:0/3440341369 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f26400135d0 con 0x7f2650069e90 2023-10-30T13:10:54.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f2637fff640 1 --2- 172.21.15.148:0/3440341369 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f263c0664a0 0x7f263c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f2637fff640 1 -- 172.21.15.148:0/3440341369 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2640084190 con 0x7f2650069e90 2023-10-30T13:10:54.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f265943e000 1 Processor -- start 2023-10-30T13:10:54.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f265943e000 1 -- start start 2023-10-30T13:10:54.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264effd640 1 --2- 172.21.15.148:0/3440341369 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f263c0664a0 0x7f263c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264effd640 1 --2- 172.21.15.148:0/3440341369 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f263c0664a0 0x7f263c068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f26440ad000 tx=0x7f26440a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:54.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f265943e000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d38810 0x55c137d38bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f265943e000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d39130 0x55c137d39510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f265943e000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 0x55c137d43aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f265943e000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c137aeccc0 con 0x55c137d38810 2023-10-30T13:10:54.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f265943e000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c137b4f920 con 0x55c137d39a50 2023-10-30T13:10:54.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f265943e000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c137b4ce50 con 0x55c137d39130 2023-10-30T13:10:54.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d38810 0x55c137d38bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d38810 0x55c137d38bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48604/0 (socket says 172.21.15.148:48604) 2023-10-30T13:10:54.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264f7fe640 1 -- 172.21.15.148:0/4132800705 learned_addr learned my addr 172.21.15.148:0/4132800705 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:54.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264effd640 1 --2- 172.21.15.148:0/4132800705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d39130 0x55c137d39510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264ffff640 1 --2- 172.21.15.148:0/4132800705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 0x55c137d43aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264f7fe640 1 -- 172.21.15.148:0/4132800705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d39130 msgr2=0x55c137d39510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:54.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264f7fe640 1 --2- 172.21.15.148:0/4132800705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d39130 0x55c137d39510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264f7fe640 1 -- 172.21.15.148:0/4132800705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 msgr2=0x55c137d43aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:54.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264f7fe640 1 --2- 172.21.15.148:0/4132800705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 0x55c137d43aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264f7fe640 1 -- 172.21.15.148:0/4132800705 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c137841890 con 0x55c137d38810 2023-10-30T13:10:54.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f264f7fe640 1 --2- 172.21.15.148:0/4132800705 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d38810 0x55c137d38bf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2640051730 tx=0x7f2640054c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e1af749808109f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:54.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f26357fa640 1 -- 172.21.15.148:0/4132800705 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2640059170 con 0x55c137d38810 2023-10-30T13:10:54.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f26357fa640 1 -- 172.21.15.148:0/4132800705 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2640059380 con 0x55c137d38810 2023-10-30T13:10:54.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f26357fa640 1 -- 172.21.15.148:0/4132800705 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264005d530 con 0x55c137d38810 2023-10-30T13:10:54.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.508+0000 7f265943e000 1 -- 172.21.15.148:0/4132800705 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d38810 msgr2=0x55c137d38bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:54.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 --2- 172.21.15.148:0/4132800705 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d38810 0x55c137d38bf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2640051730 tx=0x7f2640054c50 comp rx=0 tx=0).stop 2023-10-30T13:10:54.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- 172.21.15.148:0/4132800705 shutdown_connections 2023-10-30T13:10:54.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 --2- 172.21.15.148:0/4132800705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 0x55c137d43aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 --2- 172.21.15.148:0/4132800705 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d39130 0x55c137d39510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 --2- 172.21.15.148:0/4132800705 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d38810 0x55c137d38bf0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- 172.21.15.148:0/4132800705 >> 172.21.15.148:0/4132800705 conn(0x55c137ceefc0 msgr2=0x55c137d365d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:54.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- 172.21.15.148:0/4132800705 shutdown_connections 2023-10-30T13:10:54.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- 172.21.15.148:0/4132800705 wait complete. 2023-10-30T13:10:54.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- start start 2023-10-30T13:10:54.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 Processor -- start 2023-10-30T13:10:54.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 0x55c137d32f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d33490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 0x55c137d41a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c137b48740 con 0x55c137d39130 2023-10-30T13:10:54.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c137b07d10 con 0x55c137d39a50 2023-10-30T13:10:54.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c137b079b0 con 0x55c137d38810 2023-10-30T13:10:54.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 10 client.0 _ll_register_callbacks cb 0x55c137ce3390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:10:54.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d33490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d33490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48612/0 (socket says 172.21.15.148:48612) 2023-10-30T13:10:54.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264effd640 1 -- 172.21.15.148:0/1833176942 learned_addr learned my addr 172.21.15.148:0/1833176942 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:54.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264f7fe640 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 0x55c137d32f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264ffff640 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 0x55c137d41a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264effd640 1 -- 172.21.15.148:0/1833176942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 msgr2=0x55c137d32f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:54.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264effd640 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 0x55c137d32f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264effd640 1 -- 172.21.15.148:0/1833176942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 msgr2=0x55c137d41a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:54.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264effd640 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 0x55c137d41a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:54.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264effd640 1 -- 172.21.15.148:0/1833176942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26501f2580 con 0x55c137d39130 2023-10-30T13:10:54.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f264effd640 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d33490 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f26440097c0 tx=0x7f264401ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28070e4e2e5cb8e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:54.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f262e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2023-10-30T13:10:54.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f262e7fc640 1 -- 172.21.15.148:0/1833176942 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440b6600 con 0x55c137d39130 2023-10-30T13:10:54.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f262e7fc640 1 -- 172.21.15.148:0/1833176942 <== mon.0 v2:172.21.15.148:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26440b6760 con 0x55c137d39130 2023-10-30T13:10:54.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- 172.21.15.148:0/1833176942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c137afc370 con 0x55c137d39130 2023-10-30T13:10:54.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f262e7fc640 1 -- 172.21.15.148:0/1833176942 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26440b6920 con 0x55c137d39130 2023-10-30T13:10:54.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 10 client.4671 fetch_fsmap learned FSMap version 13 2023-10-30T13:10:54.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.512+0000 7f265943e000 1 -- 172.21.15.148:0/1833176942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x55c137b0a460 con 0x55c137d39130 2023-10-30T13:10:54.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f262e7fc640 1 -- 172.21.15.148:0/1833176942 <== mon.0 v2:172.21.15.148:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c137afc370 con 0x55c137d39130 2023-10-30T13:10:54.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f262e7fc640 1 -- 172.21.15.148:0/1833176942 <== mon.0 v2:172.21.15.148:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f26440c2070 con 0x55c137d39130 2023-10-30T13:10:54.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f265943e000 10 client.4671 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:10:54.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f265943e000 10 client.4671 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-30T13:10:54.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f265943e000 20 client.4671 populate_metadata read hostname 'smithi148' 2023-10-30T13:10:54.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f265943e000 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d427f0 0x55c137d42bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:54.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f265943e000 4 client.4671 mds_command: new command op to 4256 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:10:54.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f265943e000 1 -- 172.21.15.148:0/1833176942 --> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c137ae22d0 con 0x55c137d427f0 2023-10-30T13:10:54.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f264f7fe640 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d427f0 0x55c137d42bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:54.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f264f7fe640 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d427f0 0x55c137d42bd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17c32a77bd2608b7 server_cookie=f3a9960c36d19130 in_seq=0 out_seq=0 2023-10-30T13:10:54.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:54.516+0000 7f262e7fc640 10 client.4671 ms_handle_connect on v2:172.21.15.148:6836/1844602519 2023-10-30T13:10:54.990 INFO:tasks.ceph.mds.e.smithi148.stderr:2023-10-30T13:10:54.988+0000 7ff756f04640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:10:57.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f262e7fc640 1 -- 172.21.15.148:0/1833176942 <== mds.2 v2:172.21.15.148:6836/1844602519 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c137ae22d0 con 0x55c137d427f0 2023-10-30T13:10:57.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f262e7fc640 10 client.4671 handle_command_reply: tid=0 2023-10-30T13:10:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 client.4671 shutdown 2023-10-30T13:10:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 -- 172.21.15.148:0/1833176942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 msgr2=0x55c137d33490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d33490 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f26440097c0 tx=0x7f264401ab50 comp rx=0 tx=0).stop 2023-10-30T13:10:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 -- 172.21.15.148:0/1833176942 shutdown_connections 2023-10-30T13:10:57.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d427f0 0x55c137d42bd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d33490 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39a50 0x55c137d41a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 --2- 172.21.15.148:0/1833176942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 0x55c137d32f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f262e7fc640 0 client.4671 ms_handle_reset on v2:172.21.15.148:6836/1844602519 2023-10-30T13:10:57.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 -- 172.21.15.148:0/1833176942 >> 172.21.15.148:0/1833176942 conn(0x55c137ceefc0 msgr2=0x55c137d38090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:57.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 -- 172.21.15.148:0/1833176942 shutdown_connections 2023-10-30T13:10:57.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 1 -- 172.21.15.148:0/1833176942 wait complete. 2023-10-30T13:10:57.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.360+0000 7f265943e000 20 client.4671 trim_cache size 0 max 16384 2023-10-30T13:10:57.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.376+0000 7f265943e000 1 Processor -- start 2023-10-30T13:10:57.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.376+0000 7f265943e000 1 -- start start 2023-10-30T13:10:57.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.376+0000 7f265943e000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 0x55c137cf4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:57.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.376+0000 7f265943e000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39130 0x55c137d40d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:57.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.376+0000 7f265943e000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39a50 0x55c137d41250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:57.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.376+0000 7f265943e000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c137802940 con 0x55c137d39a50 2023-10-30T13:10:57.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.376+0000 7f265943e000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c137b45b40 con 0x55c137d39130 2023-10-30T13:10:57.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.376+0000 7f265943e000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c137b4d8b0 con 0x55c137d38810 2023-10-30T13:10:57.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39a50 0x55c137d41250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:57.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39a50 0x55c137d41250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48626/0 (socket says 172.21.15.148:48626) 2023-10-30T13:10:57.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264ffff640 1 -- 172.21.15.148:0/1312495814 learned_addr learned my addr 172.21.15.148:0/1312495814 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:57.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264f7fe640 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 0x55c137cf4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:57.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264effd640 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39130 0x55c137d40d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:57.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264ffff640 1 -- 172.21.15.148:0/1312495814 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 msgr2=0x55c137cf4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:57.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264ffff640 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 0x55c137cf4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264ffff640 1 -- 172.21.15.148:0/1312495814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39130 msgr2=0x55c137d40d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:57.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264ffff640 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39130 0x55c137d40d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264ffff640 1 -- 172.21.15.148:0/1312495814 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c137af0760 con 0x55c137d39a50 2023-10-30T13:10:57.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264effd640 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39130 0x55c137d40d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:10:57.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f264ffff640 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39a50 0x55c137d41250 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2648007050 tx=0x7f2648007510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73028ec20dcc484b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:57.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f2635ffb640 1 -- 172.21.15.148:0/1312495814 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648015470 con 0x55c137d39a50 2023-10-30T13:10:57.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f2635ffb640 1 -- 172.21.15.148:0/1312495814 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26480155d0 con 0x55c137d39a50 2023-10-30T13:10:57.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f2635ffb640 1 -- 172.21.15.148:0/1312495814 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648015940 con 0x55c137d39a50 2023-10-30T13:10:57.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 -- 172.21.15.148:0/1312495814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39a50 msgr2=0x55c137d41250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:10:57.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39a50 0x55c137d41250 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2648007050 tx=0x7f2648007510 comp rx=0 tx=0).stop 2023-10-30T13:10:57.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 -- 172.21.15.148:0/1312495814 shutdown_connections 2023-10-30T13:10:57.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39a50 0x55c137d41250 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d39130 0x55c137d40d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 --2- 172.21.15.148:0/1312495814 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d38810 0x55c137cf4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 -- 172.21.15.148:0/1312495814 >> 172.21.15.148:0/1312495814 conn(0x55c137ceefc0 msgr2=0x55c137d40600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:10:57.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 -- 172.21.15.148:0/1312495814 shutdown_connections 2023-10-30T13:10:57.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 -- 172.21.15.148:0/1312495814 wait complete. 2023-10-30T13:10:57.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 -- start start 2023-10-30T13:10:57.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.380+0000 7f265943e000 1 Processor -- start 2023-10-30T13:10:57.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d38810 0x55c137d98820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:57.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d98d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:57.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d39a50 0x55c137d50230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:57.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c137b0bc00 con 0x55c137d39130 2023-10-30T13:10:57.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c137b074d0 con 0x55c137d38810 2023-10-30T13:10:57.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c137b42a40 con 0x55c137d39a50 2023-10-30T13:10:57.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 10 client.0 _ll_register_callbacks cb 0x7f264000cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:10:57.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d98d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:57.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d38810 0x55c137d98820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:57.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d98d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48640/0 (socket says 172.21.15.148:48640) 2023-10-30T13:10:57.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264effd640 1 -- 172.21.15.148:0/178289624 learned_addr learned my addr 172.21.15.148:0/178289624 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:10:57.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264ffff640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d39a50 0x55c137d50230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:57.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264effd640 1 -- 172.21.15.148:0/178289624 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d39a50 msgr2=0x55c137d50230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:57.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264effd640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x55c137d39a50 0x55c137d50230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264effd640 1 -- 172.21.15.148:0/178289624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d38810 msgr2=0x55c137d98820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:10:57.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264effd640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c137d38810 0x55c137d98820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:10:57.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264effd640 1 -- 172.21.15.148:0/178289624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264800f040 con 0x55c137d39130 2023-10-30T13:10:57.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264effd640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c137d39130 0x55c137d98d60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f264401c350 tx=0x7f2644008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca502391fd615e37 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:10:57.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f262e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2023-10-30T13:10:57.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f262e7fc640 1 -- 172.21.15.148:0/178289624 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440b6800 con 0x55c137d39130 2023-10-30T13:10:57.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 -- 172.21.15.148:0/178289624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c137ae06f0 con 0x55c137d39130 2023-10-30T13:10:57.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f262e7fc640 1 -- 172.21.15.148:0/178289624 <== mon.0 v2:172.21.15.148:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26440b6960 con 0x55c137d39130 2023-10-30T13:10:57.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f262e7fc640 1 -- 172.21.15.148:0/178289624 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26440c4590 con 0x55c137d39130 2023-10-30T13:10:57.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 10 client.4677 fetch_fsmap learned FSMap version 13 2023-10-30T13:10:57.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 -- 172.21.15.148:0/178289624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x55c137841cc0 con 0x55c137d39130 2023-10-30T13:10:57.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f262e7fc640 1 -- 172.21.15.148:0/178289624 <== mon.0 v2:172.21.15.148:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26440e6020 con 0x55c137d39130 2023-10-30T13:10:57.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f262e7fc640 1 -- 172.21.15.148:0/178289624 <== mon.0 v2:172.21.15.148:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f26440c46f0 con 0x55c137d39130 2023-10-30T13:10:57.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 10 client.4677 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:10:57.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 10 client.4677 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-30T13:10:57.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 20 client.4677 populate_metadata read hostname 'smithi148' 2023-10-30T13:10:57.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 0x55c137d50fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:10:57.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 4 client.4677 mds_command: new command op to 4256 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:10:57.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f265943e000 1 -- 172.21.15.148:0/178289624 --> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c137b41720 con 0x55c137d50bf0 2023-10-30T13:10:57.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264f7fe640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 0x55c137d50fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:57.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f262e7fc640 10 client.4677 ms_handle_connect on v2:172.21.15.148:6836/1844602519 2023-10-30T13:10:57.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.384+0000 7f264f7fe640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 0x55c137d50fd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d10ababaeab65a8 server_cookie=3a8f234cac715aa0 in_seq=0 out_seq=0 2023-10-30T13:10:57.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f264f7fe640 1 -- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 msgr2=0x55c137d50fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2023-10-30T13:10:57.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f264f7fe640 1 -- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 msgr2=0x55c137d50fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T13:10:57.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f264f7fe640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 0x55c137d50fd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:10:57.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f264f7fe640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 0x55c137d50fd0 unknown :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2023-10-30T13:10:57.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f264f7fe640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 0x55c137d50fd0 unknown :-1 s=BANNER_CONNECTING pgs=16 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:10:57.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f264f7fe640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 0x55c137d50fd0 crc :-1 s=SESSION_RECONNECTING pgs=16 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_session_reset received session reset full=1 2023-10-30T13:10:57.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f264f7fe640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 0x55c137d50fd0 crc :-1 s=SESSION_RECONNECTING pgs=16 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).reset_session 2023-10-30T13:10:57.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f262e7fc640 0 client.4677 ms_handle_remote_reset on v2:172.21.15.148:6836/1844602519 2023-10-30T13:10:57.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f262e7fc640 10 client.4677 ms_handle_connect on v2:172.21.15.148:6836/1844602519 2023-10-30T13:10:57.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:10:57.388+0000 7f264f7fe640 1 --2- 172.21.15.148:0/178289624 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x55c137d50bf0 0x55c137d50fd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd48e2ad0312dc5b server_cookie=680d7919f8df027e in_seq=0 out_seq=0 2023-10-30T13:11:22.799 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:11:22.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:11:52.854 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:11:52.859 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:12:22.942 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:12:22.947 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:12:53.015 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:12:53.020 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:12:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-30T13:12:55.393 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:55.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.537+0000 7f50bdda1640 1 Processor -- start 2023-10-30T13:12:55.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.537+0000 7f50bdda1640 1 -- start start 2023-10-30T13:12:55.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50bdda1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f50b8070650 0x7f50b8071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50bdda1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070c90 0x7f50b80720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50bdda1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b80c2f20 con 0x7f50b8156f80 2023-10-30T13:12:55.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50bdda1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b8157360 con 0x7f50b8070650 2023-10-30T13:12:55.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50bdda1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80c85c0 con 0x7f50b8070c90 2023-10-30T13:12:55.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f50b8070650 0x7f50b8071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070c90 0x7f50b80720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b77fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f50b8156f80 0x7f50b8159880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:41952/0 (socket says 172.21.15.148:41952) 2023-10-30T13:12:55.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b77fe640 1 -- 172.21.15.148:0/2434489882 learned_addr learned my addr 172.21.15.148:0/2434489882 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:12:55.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b67fc640 1 -- 172.21.15.148:0/2434489882 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259110302 0 0) 0x7f50b80c2f20 con 0x7f50b8156f80 2023-10-30T13:12:55.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b67fc640 1 -- 172.21.15.148:0/2434489882 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a0003530 con 0x7f50b8156f80 2023-10-30T13:12:55.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b7fff640 1 -- 172.21.15.148:0/2434489882 >> v2:172.21.15.150:3300/0 conn(0x7f50b8070650 msgr2=0x7f50b8071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:55.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b7fff640 1 --2- 172.21.15.148:0/2434489882 >> v2:172.21.15.150:3300/0 conn(0x7f50b8070650 0x7f50b8071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b7fff640 1 -- 172.21.15.148:0/2434489882 >> v1:172.21.15.150:6789/0 conn(0x7f50b8156f80 legacy=0x7f50b8159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b7fff640 1 -- 172.21.15.148:0/2434489882 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b8071230 con 0x7f50b8070c90 2023-10-30T13:12:55.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.541+0000 7f50b6ffd640 1 --2- 172.21.15.148:0/2434489882 >> v2:172.21.15.150:3300/0 conn(0x7f50b8070650 0x7f50b8071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:12:55.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50b7fff640 1 --2- 172.21.15.148:0/2434489882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070c90 0x7f50b80720f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009d90 tx=0x7f50b001c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46d0a6523d6d58f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:55.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50b67fc640 1 -- 172.21.15.148:0/2434489882 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00a59a0 con 0x7f50b8070c90 2023-10-30T13:12:55.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50b67fc640 1 -- 172.21.15.148:0/2434489882 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b00a5b00 con 0x7f50b8070c90 2023-10-30T13:12:55.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50b67fc640 1 -- 172.21.15.148:0/2434489882 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00a5e70 con 0x7f50b8070c90 2023-10-30T13:12:55.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50bdda1640 1 -- 172.21.15.148:0/2434489882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070c90 msgr2=0x7f50b80720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50bdda1640 1 --2- 172.21.15.148:0/2434489882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070c90 0x7f50b80720f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009d90 tx=0x7f50b001c950 comp rx=0 tx=0).stop 2023-10-30T13:12:55.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50bdda1640 1 -- 172.21.15.148:0/2434489882 shutdown_connections 2023-10-30T13:12:55.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50bdda1640 1 --2- 172.21.15.148:0/2434489882 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070c90 0x7f50b80720f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50bdda1640 1 --2- 172.21.15.148:0/2434489882 >> v2:172.21.15.150:3300/0 conn(0x7f50b8070650 0x7f50b8071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50bdda1640 1 -- 172.21.15.148:0/2434489882 >> 172.21.15.148:0/2434489882 conn(0x7f50b80cbeb0 msgr2=0x7f50b80cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:55.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50bdda1640 1 -- 172.21.15.148:0/2434489882 shutdown_connections 2023-10-30T13:12:55.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50bdda1640 1 -- 172.21.15.148:0/2434489882 wait complete. 2023-10-30T13:12:55.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.545+0000 7f50bdda1640 1 Processor -- start 2023-10-30T13:12:55.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 -- start start 2023-10-30T13:12:55.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070650 0x7f50b8152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50b8070c90 0x7f50b8145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50b8156f80 0x7f50b81f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b8158e60 con 0x7f50b8156f80 2023-10-30T13:12:55.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b81f1e40 con 0x7f50b8070c90 2023-10-30T13:12:55.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b806c660 con 0x7f50b8070650 2023-10-30T13:12:55.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50b8156f80 0x7f50b81f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50b8156f80 0x7f50b81f5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42426/0 (socket says 172.21.15.148:42426) 2023-10-30T13:12:55.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b7fff640 1 -- 172.21.15.148:0/2486673919 learned_addr learned my addr 172.21.15.148:0/2486673919 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:55.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b77fe640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070650 0x7f50b8152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b6ffd640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50b8070c90 0x7f50b8145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b7fff640 1 -- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070650 msgr2=0x7f50b8152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b7fff640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070650 0x7f50b8152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b7fff640 1 -- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50b8070c90 msgr2=0x7f50b8145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b7fff640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50b8070c90 0x7f50b8145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b7fff640 1 -- 172.21.15.148:0/2486673919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac009420 con 0x7f50b8156f80 2023-10-30T13:12:55.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50b7fff640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50b8156f80 0x7f50b81f5a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f50b001a580 tx=0x7f50b00196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:55.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f509ffff640 1 -- 172.21.15.148:0/2486673919 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00bd070 con 0x7f50b8156f80 2023-10-30T13:12:55.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f509ffff640 1 -- 172.21.15.148:0/2486673919 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0009450 con 0x7f50b8156f80 2023-10-30T13:12:55.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b0009a70 con 0x7f50b8156f80 2023-10-30T13:12:55.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f509ffff640 1 -- 172.21.15.148:0/2486673919 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0004240 con 0x7f50b8156f80 2023-10-30T13:12:55.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b81567a0 con 0x7f50b8156f80 2023-10-30T13:12:55.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.549+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b804e170 con 0x7f50b8156f80 2023-10-30T13:12:55.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.553+0000 7f509ffff640 1 -- 172.21.15.148:0/2486673919 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f50b00b4030 con 0x7f50b8156f80 2023-10-30T13:12:55.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.553+0000 7f509ffff640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50a40664d0 0x7f50a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.553+0000 7f509ffff640 1 -- 172.21.15.148:0/2486673919 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f50b011e580 con 0x7f50b8156f80 2023-10-30T13:12:55.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.553+0000 7f50b77fe640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50a40664d0 0x7f50a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.553+0000 7f50b77fe640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50a40664d0 0x7f50a4068990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f50a8007fd0 tx=0x7f50a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:55.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.557+0000 7f509ffff640 1 -- 172.21.15.148:0/2486673919 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f50b00eb980 con 0x7f50b8156f80 2023-10-30T13:12:55.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.737+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50b81567a0 con 0x7f50b8156f80 2023-10-30T13:12:55.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.737+0000 7f509ffff640 1 -- 172.21.15.148:0/2486673919 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f50b81567a0 con 0x7f50b8156f80 2023-10-30T13:12:55.742 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:12:55.742 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:55.743 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:12:55.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50a40664d0 msgr2=0x7f50a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50a40664d0 0x7f50a4068990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f50a8007fd0 tx=0x7f50a8009040 comp rx=0 tx=0).stop 2023-10-30T13:12:55.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50b8156f80 msgr2=0x7f50b81f5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50b8156f80 0x7f50b81f5a00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f50b001a580 tx=0x7f50b00196f0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 shutdown_connections 2023-10-30T13:12:55.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50a40664d0 0x7f50a4068990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50b8156f80 0x7f50b81f5a00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50b8070c90 0x7f50b8145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 --2- 172.21.15.148:0/2486673919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50b8070650 0x7f50b8152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 >> 172.21.15.148:0/2486673919 conn(0x7f50b80cbeb0 msgr2=0x7f50b8153d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:55.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 shutdown_connections 2023-10-30T13:12:55.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:55.741+0000 7f50bdda1640 1 -- 172.21.15.148:0/2486673919 wait complete. 2023-10-30T13:12:56.366 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T13:10:34.058265+0000', 'modified': '2023-10-30T13:10:51.357311+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4285, 'mds_1': 4270, 'mds_2': 4256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4256': {'gid': 4256, 'name': 'c', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.148:6837/1844602519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6836', 'nonce': 1844602519}, {'type': 'v1', 'addr': '172.21.15.148:6837', 'nonce': 1844602519}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4270': {'gid': 4270, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.150:6836/1620877178', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 1620877178}, {'type': 'v1', 'addr': '172.21.15.150:6836', 'nonce': 1620877178}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4285': {'gid': 4285, 'name': 'e', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.148:6834/4162935019', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6832', 'nonce': 4162935019}, {'type': 'v1', 'addr': '172.21.15.148:6834', 'nonce': 4162935019}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T13:12:56.366 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-30T13:12:56.367 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-30T13:12:56.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99ce817640 1 Processor -- start 2023-10-30T13:12:56.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99ce817640 1 -- start start 2023-10-30T13:12:56.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99ce817640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f99c80cce00 0x7f99c806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99ce817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 0x7f99c80cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99ce817640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c806d710 con 0x7f99c806e430 2023-10-30T13:12:56.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99ce817640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c806dd30 con 0x7f99c80cce00 2023-10-30T13:12:56.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99ce817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c806beb0 con 0x7f99c80c7830 2023-10-30T13:12:56.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99c7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f99c80cce00 0x7f99c806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99c7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f99c80cce00 0x7f99c806ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42436/0 (socket says 172.21.15.148:42436) 2023-10-30T13:12:56.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99c7fff640 1 -- 172.21.15.148:0/1762709082 learned_addr learned my addr 172.21.15.148:0/1762709082 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99ccdb4640 1 --1- 172.21.15.148:0/1762709082 >> v1:172.21.15.148:6789/0 conn(0x7f99c806e430 0x7f99c8159850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44846/0 (socket says 172.21.15.148:44846) 2023-10-30T13:12:56.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99c77fe640 1 --2- 172.21.15.148:0/1762709082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 0x7f99c80cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99c7fff640 1 -- 172.21.15.148:0/1762709082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 msgr2=0x7f99c80cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99c7fff640 1 --2- 172.21.15.148:0/1762709082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 0x7f99c80cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99c7fff640 1 -- 172.21.15.148:0/1762709082 >> v1:172.21.15.148:6789/0 conn(0x7f99c806e430 legacy=0x7f99c8159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.509+0000 7f99c7fff640 1 -- 172.21.15.148:0/1762709082 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c806b560 con 0x7f99c80cce00 2023-10-30T13:12:56.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99c7fff640 1 --2- 172.21.15.148:0/1762709082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80cce00 0x7f99c806ca50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f99b8009870 tx=0x7f99b801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e6cc3b713a8546 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99c6ffd640 1 -- 172.21.15.148:0/1762709082 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c815a770 con 0x7f99c80cce00 2023-10-30T13:12:56.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99c6ffd640 1 -- 172.21.15.148:0/1762709082 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b80a7050 con 0x7f99c80cce00 2023-10-30T13:12:56.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99c6ffd640 1 -- 172.21.15.148:0/1762709082 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b80ab820 con 0x7f99c80cce00 2023-10-30T13:12:56.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99ce817640 1 -- 172.21.15.148:0/1762709082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80cce00 msgr2=0x7f99c806ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99ce817640 1 --2- 172.21.15.148:0/1762709082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80cce00 0x7f99c806ca50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f99b8009870 tx=0x7f99b801c540 comp rx=0 tx=0).stop 2023-10-30T13:12:56.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99ce817640 1 -- 172.21.15.148:0/1762709082 shutdown_connections 2023-10-30T13:12:56.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99ce817640 1 --2- 172.21.15.148:0/1762709082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 0x7f99c80cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99ce817640 1 --2- 172.21.15.148:0/1762709082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80cce00 0x7f99c806ca50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99ce817640 1 -- 172.21.15.148:0/1762709082 >> 172.21.15.148:0/1762709082 conn(0x7f99c8071ea0 msgr2=0x7f99c8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99ce817640 1 -- 172.21.15.148:0/1762709082 shutdown_connections 2023-10-30T13:12:56.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.513+0000 7f99ce817640 1 -- 172.21.15.148:0/1762709082 wait complete. 2023-10-30T13:12:56.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 Processor -- start 2023-10-30T13:12:56.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 -- start start 2023-10-30T13:12:56.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c806e430 0x7f99c8154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 0x7f99c80c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c80cce00 0x7f99c8154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c8151c40 con 0x7f99c806e430 2023-10-30T13:12:56.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c81520a0 con 0x7f99c80cce00 2023-10-30T13:12:56.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c8152640 con 0x7f99c80c7830 2023-10-30T13:12:56.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c806e430 0x7f99c8154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c806e430 0x7f99c8154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42444/0 (socket says 172.21.15.148:42444) 2023-10-30T13:12:56.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c7fff640 1 -- 172.21.15.148:0/1232452202 learned_addr learned my addr 172.21.15.148:0/1232452202 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c77fe640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 0x7f99c80c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c80cce00 0x7f99c8154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c7fff640 1 -- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 msgr2=0x7f99c80c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c7fff640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 0x7f99c80c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c7fff640 1 -- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c80cce00 msgr2=0x7f99c8154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c7fff640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c80cce00 0x7f99c8154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c7fff640 1 -- 172.21.15.148:0/1232452202 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b80094d0 con 0x7f99c806e430 2023-10-30T13:12:56.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c77fe640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 0x7f99c80c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:56.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c7fff640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c806e430 0x7f99c8154370 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99b801a170 tx=0x7f99b80192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c4ff9640 1 -- 172.21.15.148:0/1232452202 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b80aa7b0 con 0x7f99c806e430 2023-10-30T13:12:56.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c4ff9640 1 -- 172.21.15.148:0/1232452202 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b80aa910 con 0x7f99c806e430 2023-10-30T13:12:56.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99c4ff9640 1 -- 172.21.15.148:0/1232452202 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b80aac80 con 0x7f99c806e430 2023-10-30T13:12:56.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 -- 172.21.15.148:0/1232452202 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c8152360 con 0x7f99c806e430 2023-10-30T13:12:56.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.517+0000 7f99ce817640 1 -- 172.21.15.148:0/1232452202 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c8151440 con 0x7f99c806e430 2023-10-30T13:12:56.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99c4ff9640 1 -- 172.21.15.148:0/1232452202 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99b80b2070 con 0x7f99c806e430 2023-10-30T13:12:56.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99c4ff9640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f99b40664d0 0x7f99b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99c4ff9640 1 -- 172.21.15.148:0/1232452202 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f99b811d550 con 0x7f99c806e430 2023-10-30T13:12:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99c77fe640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f99b40664d0 0x7f99b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99d1b9a000 1 Processor -- start 2023-10-30T13:12:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99d1b9a000 1 -- start start 2023-10-30T13:12:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99c77fe640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f99b40664d0 0x7f99b4068990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f99c815ce60 tx=0x7f99bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdc810 0x56110ffdcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdd130 0x56110ffdd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdda50 0x56110ffe7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110fd90cc0 con 0x56110ffdd130 2023-10-30T13:12:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110fdf3920 con 0x56110ffdc810 2023-10-30T13:12:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56110fdf0e50 con 0x56110ffdda50 2023-10-30T13:12:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdd130 0x56110ffdd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdd130 0x56110ffdd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42450/0 (socket says 172.21.15.148:42450) 2023-10-30T13:12:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99c77fe640 1 -- 172.21.15.148:0/670635294 learned_addr learned my addr 172.21.15.148:0/670635294 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.521+0000 7f99c7fff640 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdc810 0x56110ffdcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdda50 0x56110ffe7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c77fe640 1 -- 172.21.15.148:0/670635294 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdda50 msgr2=0x56110ffe7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c77fe640 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdda50 0x56110ffe7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c77fe640 1 -- 172.21.15.148:0/670635294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdc810 msgr2=0x56110ffdcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c77fe640 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdc810 0x56110ffdcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c77fe640 1 -- 172.21.15.148:0/670635294 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56110fae5890 con 0x56110ffdd130 2023-10-30T13:12:56.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c7fff640 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdc810 0x56110ffdcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:56.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c77fe640 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdd130 0x56110ffdd510 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f99bc00fab0 tx=0x7f99bc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b136f9fa5be3636 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99adffb640 1 -- 172.21.15.148:0/670635294 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc014ca0 con 0x56110ffdd130 2023-10-30T13:12:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99adffb640 1 -- 172.21.15.148:0/670635294 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99bc019070 con 0x56110ffdd130 2023-10-30T13:12:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99adffb640 1 -- 172.21.15.148:0/670635294 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc020440 con 0x56110ffdd130 2023-10-30T13:12:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 -- 172.21.15.148:0/670635294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdd130 msgr2=0x56110ffdd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdd130 0x56110ffdd510 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f99bc00fab0 tx=0x7f99bc012000 comp rx=0 tx=0).stop 2023-10-30T13:12:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 -- 172.21.15.148:0/670635294 shutdown_connections 2023-10-30T13:12:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdd130 0x56110ffdd510 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdda50 0x56110ffe7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/670635294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdc810 0x56110ffdcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 -- 172.21.15.148:0/670635294 >> 172.21.15.148:0/670635294 conn(0x56110ff92fc0 msgr2=0x56110ffda5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 -- 172.21.15.148:0/670635294 shutdown_connections 2023-10-30T13:12:56.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 -- 172.21.15.148:0/670635294 wait complete. 2023-10-30T13:12:56.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 -- start start 2023-10-30T13:12:56.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 Processor -- start 2023-10-30T13:12:56.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdc810 0x56110ffd6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdd130 0x56110ffd7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdda50 0x56110ffe2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110fdec740 con 0x56110ffdc810 2023-10-30T13:12:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110fdabd10 con 0x56110ffdda50 2023-10-30T13:12:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56110fdab9b0 con 0x56110ffdd130 2023-10-30T13:12:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99d1b9a000 10 client.0 _ll_register_callbacks cb 0x56110ff87390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:12:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdc810 0x56110ffd6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdc810 0x56110ffd6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42452/0 (socket says 172.21.15.148:42452) 2023-10-30T13:12:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c7fff640 1 -- 172.21.15.148:0/3085368318 learned_addr learned my addr 172.21.15.148:0/3085368318 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c77fe640 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdd130 0x56110ffd7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdda50 0x56110ffe2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c7fff640 1 -- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdd130 msgr2=0x56110ffd7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c7fff640 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdd130 0x56110ffd7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c7fff640 1 -- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdda50 msgr2=0x56110ffe2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c7fff640 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdda50 0x56110ffe2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c7fff640 1 -- 172.21.15.148:0/3085368318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99bc00e040 con 0x56110ffdc810 2023-10-30T13:12:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99c77fe640 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdd130 0x56110ffd7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:12:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.525+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdda50 0x56110ffe2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:12:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99c7fff640 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdc810 0x56110ffd6f50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f99b80eaaf0 tx=0x7f99b80ee5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3fd306a972103a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2023-10-30T13:12:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3085368318 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b80edae0 con 0x56110ffdc810 2023-10-30T13:12:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3085368318 <== mon.0 v2:172.21.15.148:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f99b80edc40 con 0x56110ffdc810 2023-10-30T13:12:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3085368318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56110fda0370 con 0x56110ffdc810 2023-10-30T13:12:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3085368318 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99b811d0c0 con 0x56110ffdc810 2023-10-30T13:12:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 10 client.4695 fetch_fsmap learned FSMap version 13 2023-10-30T13:12:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3085368318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x56110fdae460 con 0x56110ffdc810 2023-10-30T13:12:56.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3085368318 <== mon.0 v2:172.21.15.148:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56110fda0370 con 0x56110ffdc810 2023-10-30T13:12:56.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3085368318 <== mon.0 v2:172.21.15.148:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f99b80f6460 con 0x56110ffdc810 2023-10-30T13:12:56.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 10 client.4695 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:12:56.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 10 client.4695 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-30T13:12:56.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 20 client.4695 populate_metadata read hostname 'smithi148' 2023-10-30T13:12:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x56110ffe39d0 0x56110ffe3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 4 client.4695 mds_command: new command op to 4256 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:12:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3085368318 --> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56110fd862d0 con 0x56110ffe39d0 2023-10-30T13:12:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99c77fe640 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x56110ffe39d0 0x56110ffe3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 10 client.4695 ms_handle_connect on v2:172.21.15.148:6836/1844602519 2023-10-30T13:12:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99c77fe640 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x56110ffe39d0 0x56110ffe3db0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a6f0c9a379003f server_cookie=4e4ce3abfabc2aa2 in_seq=0 out_seq=0 2023-10-30T13:12:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3085368318 <== mds.2 v2:172.21.15.148:6836/1844602519 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56110fd862d0 con 0x56110ffe39d0 2023-10-30T13:12:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 10 client.4695 handle_command_reply: tid=0 2023-10-30T13:12:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 client.4695 shutdown 2023-10-30T13:12:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3085368318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdc810 msgr2=0x56110ffd6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdc810 0x56110ffd6f50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f99b80eaaf0 tx=0x7f99b80ee5f0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3085368318 shutdown_connections 2023-10-30T13:12:56.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x56110ffe39d0 0x56110ffe3db0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdda50 0x56110ffe2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdd130 0x56110ffd7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3085368318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdc810 0x56110ffd6f50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99a6ffd640 0 client.4695 ms_handle_reset on v2:172.21.15.148:6836/1844602519 2023-10-30T13:12:56.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3085368318 >> 172.21.15.148:0/3085368318 conn(0x56110ff92fc0 msgr2=0x56110ffdc090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3085368318 shutdown_connections 2023-10-30T13:12:56.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3085368318 wait complete. 2023-10-30T13:12:56.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.529+0000 7f99d1b9a000 20 client.4695 trim_cache size 0 max 16384 2023-10-30T13:12:56.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99d1b9a000 1 Processor -- start 2023-10-30T13:12:56.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99d1b9a000 1 -- start start 2023-10-30T13:12:56.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 0x56110ffe2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 0x56110ffe03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56110ffe08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110faa6940 con 0x56110ffdda50 2023-10-30T13:12:56.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110fde80b0 con 0x56110ffdd130 2023-10-30T13:12:56.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56110fe2dba0 con 0x56110ffdc810 2023-10-30T13:12:56.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ccdb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56110ffe08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ccdb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56110ffe08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42460/0 (socket says 172.21.15.148:42460) 2023-10-30T13:12:56.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ccdb4640 1 -- 172.21.15.148:0/1028871887 learned_addr learned my addr 172.21.15.148:0/1028871887 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99c7fff640 1 --2- 172.21.15.148:0/1028871887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 0x56110ffe2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99c77fe640 1 --2- 172.21.15.148:0/1028871887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 0x56110ffe03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ccdb4640 1 -- 172.21.15.148:0/1028871887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 msgr2=0x56110ffe2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/1028871887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 0x56110ffe2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ccdb4640 1 -- 172.21.15.148:0/1028871887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 msgr2=0x56110ffe03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/1028871887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 0x56110ffe03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ccdb4640 1 -- 172.21.15.148:0/1028871887 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56110ffeef50 con 0x56110ffdda50 2023-10-30T13:12:56.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/1028871887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56110ffe08e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001b50 tx=0x7f99c0002010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cea5ec8aa6b635 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.545+0000 7f99ae7fc640 1 -- 172.21.15.148:0/1028871887 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0007830 con 0x56110ffdda50 2023-10-30T13:12:56.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ae7fc640 1 -- 172.21.15.148:0/1028871887 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c0007990 con 0x56110ffdda50 2023-10-30T13:12:56.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ae7fc640 1 -- 172.21.15.148:0/1028871887 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0015650 con 0x56110ffdda50 2023-10-30T13:12:56.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- 172.21.15.148:0/1028871887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 msgr2=0x56110ffe08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/1028871887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56110ffe08e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f99c0001b50 tx=0x7f99c0002010 comp rx=0 tx=0).stop 2023-10-30T13:12:56.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- 172.21.15.148:0/1028871887 shutdown_connections 2023-10-30T13:12:56.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/1028871887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56110ffe08e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/1028871887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 0x56110ffe03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/1028871887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 0x56110ffe2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- 172.21.15.148:0/1028871887 >> 172.21.15.148:0/1028871887 conn(0x56110ff92fc0 msgr2=0x56110ffa14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- 172.21.15.148:0/1028871887 shutdown_connections 2023-10-30T13:12:56.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- 172.21.15.148:0/1028871887 wait complete. 2023-10-30T13:12:56.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- start start 2023-10-30T13:12:56.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 Processor -- start 2023-10-30T13:12:56.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 0x561110039a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 0x561110039fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56111003a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110fdf6200 con 0x56110ffdda50 2023-10-30T13:12:56.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56110fdab4d0 con 0x56110ffdd130 2023-10-30T13:12:56.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x56110fdf0740 con 0x56110ffdc810 2023-10-30T13:12:56.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 10 client.0 _ll_register_callbacks cb 0x7f99bc007660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:12:56.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ccdb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56111003a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ccdb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56111003a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42470/0 (socket says 172.21.15.148:42470) 2023-10-30T13:12:56.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ccdb4640 1 -- 172.21.15.148:0/3350854400 learned_addr learned my addr 172.21.15.148:0/3350854400 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99c7fff640 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 0x561110039a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99c77fe640 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 0x561110039fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ccdb4640 1 -- 172.21.15.148:0/3350854400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 msgr2=0x561110039a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 0x561110039a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ccdb4640 1 -- 172.21.15.148:0/3350854400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 msgr2=0x561110039fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 0x561110039fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ccdb4640 1 -- 172.21.15.148:0/3350854400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c806d710 con 0x56110ffdda50 2023-10-30T13:12:56.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99c7fff640 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 0x561110039a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:56.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99ccdb4640 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56111003a730 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f99c0000f90 tx=0x7f99c00012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f159af6128af1ebd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2023-10-30T13:12:56.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3350854400 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0015c80 con 0x56110ffdda50 2023-10-30T13:12:56.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3350854400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56110fd846f0 con 0x56110ffdda50 2023-10-30T13:12:56.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.549+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3350854400 <== mon.0 v2:172.21.15.148:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f99c0015de0 con 0x56110ffdda50 2023-10-30T13:12:56.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3350854400 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99c0029470 con 0x56110ffdda50 2023-10-30T13:12:56.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 10 client.4701 fetch_fsmap learned FSMap version 13 2023-10-30T13:12:56.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3350854400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x56110fd94760 con 0x56110ffdda50 2023-10-30T13:12:56.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3350854400 <== mon.0 v2:172.21.15.148:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56110fd846f0 con 0x56110ffdda50 2023-10-30T13:12:56.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3350854400 <== mon.0 v2:172.21.15.148:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f99c00295d0 con 0x56110ffdda50 2023-10-30T13:12:56.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 10 client.4701 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:12:56.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 10 client.4701 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-30T13:12:56.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 20 client.4701 populate_metadata read hostname 'smithi148' 2023-10-30T13:12:56.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x561110046be0 0x56111003b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 4 client.4701 mds_command: new command op to 4256 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:12:56.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3350854400 --> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56110fde5720 con 0x561110046be0 2023-10-30T13:12:56.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99c7fff640 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x561110046be0 0x56111003b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99c7fff640 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x561110046be0 0x56111003b4a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=58cc0cac9579510b server_cookie=84b1ece667aa8023 in_seq=0 out_seq=0 2023-10-30T13:12:56.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99a6ffd640 10 client.4701 ms_handle_connect on v2:172.21.15.148:6836/1844602519 2023-10-30T13:12:56.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99a6ffd640 1 -- 172.21.15.148:0/3350854400 <== mds.2 v2:172.21.15.148:6836/1844602519 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x56110fde5720 con 0x561110046be0 2023-10-30T13:12:56.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99a6ffd640 10 client.4701 handle_command_reply: tid=0 2023-10-30T13:12:56.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 client.4701 shutdown 2023-10-30T13:12:56.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3350854400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 msgr2=0x56111003a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56111003a730 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f99c0000f90 tx=0x7f99c00012b0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3350854400 shutdown_connections 2023-10-30T13:12:56.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.148:6836/1844602519,v1:172.21.15.148:6837/1844602519] conn(0x561110046be0 0x56111003b4a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56110ffdda50 0x56111003a730 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56110ffdd130 0x561110039fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 --2- 172.21.15.148:0/3350854400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x56110ffdc810 0x561110039a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3350854400 >> 172.21.15.148:0/3350854400 conn(0x56110ff92fc0 msgr2=0x56110ffa0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99a6ffd640 0 client.4701 ms_handle_reset on v2:172.21.15.148:6836/1844602519 2023-10-30T13:12:56.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3350854400 shutdown_connections 2023-10-30T13:12:56.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 1 -- 172.21.15.148:0/3350854400 wait complete. 2023-10-30T13:12:56.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99d1b9a000 20 client.4701 trim_cache size 0 max 16384 2023-10-30T13:12:56.569 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:12:56.569 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_fsid": "23a3d51e-f2ab-4ffa-93fe-42135bef502c", 2023-10-30T13:12:56.569 INFO:teuthology.orchestra.run.smithi148.stdout: "whoami": 2, 2023-10-30T13:12:56.569 INFO:teuthology.orchestra.run.smithi148.stdout: "id": 4256, 2023-10-30T13:12:56.569 INFO:teuthology.orchestra.run.smithi148.stdout: "want_state": "up:active", 2023-10-30T13:12:56.570 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "up:active", 2023-10-30T13:12:56.570 INFO:teuthology.orchestra.run.smithi148.stdout: "fs_name": "cephfs", 2023-10-30T13:12:56.570 INFO:teuthology.orchestra.run.smithi148.stdout: "rank_uptime": 130.22012554200001, 2023-10-30T13:12:56.570 INFO:teuthology.orchestra.run.smithi148.stdout: "mdsmap_epoch": 13, 2023-10-30T13:12:56.570 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_epoch": 22, 2023-10-30T13:12:56.570 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_epoch_barrier": 20, 2023-10-30T13:12:56.571 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 148.55079467300001 2023-10-30T13:12:56.571 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:12:56.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 -- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f99b40664d0 msgr2=0x7f99b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f99b40664d0 0x7f99b4068990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f99c815ce60 tx=0x7f99bc008040 comp rx=0 tx=0).stop 2023-10-30T13:12:56.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 -- 172.21.15.148:0/1232452202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c806e430 msgr2=0x7f99c8154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c806e430 0x7f99c8154370 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99b801a170 tx=0x7f99b80192e0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 -- 172.21.15.148:0/1232452202 shutdown_connections 2023-10-30T13:12:56.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f99b40664d0 0x7f99b4068990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c80cce00 0x7f99c8154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f99c80c7830 0x7f99c80c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 --2- 172.21.15.148:0/1232452202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c806e430 0x7f99c8154370 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 -- 172.21.15.148:0/1232452202 >> 172.21.15.148:0/1232452202 conn(0x7f99c8071ea0 msgr2=0x7f99c80cc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 -- 172.21.15.148:0/1232452202 shutdown_connections 2023-10-30T13:12:56.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.553+0000 7f99ce817640 1 -- 172.21.15.148:0/1232452202 wait complete. 2023-10-30T13:12:56.576 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-30T13:12:56.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.709+0000 7f00cc291640 1 Processor -- start 2023-10-30T13:12:56.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.709+0000 7f00cc291640 1 -- start start 2023-10-30T13:12:56.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00cc291640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00c40c6b80 0x7f00c4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00cc291640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c6f60 0x7f00c413f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00cc291640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c4070010 con 0x7f00c40c08d0 2023-10-30T13:12:56.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00cc291640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c4072260 con 0x7f00c40c6b80 2023-10-30T13:12:56.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00cc291640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c406f6e0 con 0x7f00c40c6f60 2023-10-30T13:12:56.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00c982c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c6f60 0x7f00c413f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00ca02d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00c40c6b80 0x7f00c4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00c982c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c6f60 0x7f00c413f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:60184/0 (socket says 172.21.15.148:60184) 2023-10-30T13:12:56.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00c982c640 1 -- 172.21.15.148:0/868724533 learned_addr learned my addr 172.21.15.148:0/868724533 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00c982c640 1 -- 172.21.15.148:0/868724533 >> v2:172.21.15.150:3300/0 conn(0x7f00c40c6b80 msgr2=0x7f00c4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00c982c640 1 --2- 172.21.15.148:0/868724533 >> v2:172.21.15.150:3300/0 conn(0x7f00c40c6b80 0x7f00c4071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00c982c640 1 -- 172.21.15.148:0/868724533 >> v1:172.21.15.150:6789/0 conn(0x7f00c40c08d0 legacy=0x7f00c41565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00c982c640 1 -- 172.21.15.148:0/868724533 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c406f9c0 con 0x7f00c40c6f60 2023-10-30T13:12:56.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.713+0000 7f00ca02d640 1 --2- 172.21.15.148:0/868724533 >> v2:172.21.15.150:3300/0 conn(0x7f00c40c6b80 0x7f00c4071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:12:56.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00c982c640 1 --2- 172.21.15.148:0/868724533 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c6f60 0x7f00c413f600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009870 tx=0x7f00b401e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cae2a1fe1e6b2e05 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00c902b640 1 -- 172.21.15.148:0/868724533 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40a8610 con 0x7f00c40c6f60 2023-10-30T13:12:56.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00c902b640 1 -- 172.21.15.148:0/868724533 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b40a8770 con 0x7f00c40c6f60 2023-10-30T13:12:56.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00c902b640 1 -- 172.21.15.148:0/868724533 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40a8b00 con 0x7f00c40c6f60 2023-10-30T13:12:56.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00cc291640 1 -- 172.21.15.148:0/868724533 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c6f60 msgr2=0x7f00c413f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00cc291640 1 --2- 172.21.15.148:0/868724533 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c6f60 0x7f00c413f600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009870 tx=0x7f00b401e720 comp rx=0 tx=0).stop 2023-10-30T13:12:56.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00cc291640 1 -- 172.21.15.148:0/868724533 shutdown_connections 2023-10-30T13:12:56.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00cc291640 1 --2- 172.21.15.148:0/868724533 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c6f60 0x7f00c413f600 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00cc291640 1 --2- 172.21.15.148:0/868724533 >> v2:172.21.15.150:3300/0 conn(0x7f00c40c6b80 0x7f00c4071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00cc291640 1 -- 172.21.15.148:0/868724533 >> 172.21.15.148:0/868724533 conn(0x7f00c40c7910 msgr2=0x7f00c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00cc291640 1 -- 172.21.15.148:0/868724533 shutdown_connections 2023-10-30T13:12:56.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.717+0000 7f00cc291640 1 -- 172.21.15.148:0/868724533 wait complete. 2023-10-30T13:12:56.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 Processor -- start 2023-10-30T13:12:56.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 -- start start 2023-10-30T13:12:56.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c08d0 0x7f00c413c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c40c6b80 0x7f00c4152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00c40c6f60 0x7f00c40c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c41f23b0 con 0x7f00c40c6b80 2023-10-30T13:12:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c41f6290 con 0x7f00c40c6f60 2023-10-30T13:12:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c41f5680 con 0x7f00c40c08d0 2023-10-30T13:12:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00c982c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c40c6b80 0x7f00c4152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00c982c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c40c6b80 0x7f00c4152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42486/0 (socket says 172.21.15.148:42486) 2023-10-30T13:12:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00c982c640 1 -- 172.21.15.148:0/2838937762 learned_addr learned my addr 172.21.15.148:0/2838937762 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00ca02d640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c08d0 0x7f00c413c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00ca82e640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00c40c6f60 0x7f00c40c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00c982c640 1 -- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c08d0 msgr2=0x7f00c413c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00c982c640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c08d0 0x7f00c413c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00c982c640 1 -- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00c40c6f60 msgr2=0x7f00c40c2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00c982c640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00c40c6f60 0x7f00c40c2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00c982c640 1 -- 172.21.15.148:0/2838937762 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c0009420 con 0x7f00c40c6b80 2023-10-30T13:12:56.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00c982c640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c40c6b80 0x7f00c4152290 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f00b401cdd0 tx=0x7f00b401b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00baffd640 1 -- 172.21.15.148:0/2838937762 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40b3650 con 0x7f00c40c6b80 2023-10-30T13:12:56.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00baffd640 1 -- 172.21.15.148:0/2838937762 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b40b37b0 con 0x7f00c40c6b80 2023-10-30T13:12:56.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00baffd640 1 -- 172.21.15.148:0/2838937762 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40b3b40 con 0x7f00c40c6b80 2023-10-30T13:12:56.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 -- 172.21.15.148:0/2838937762 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b40094d0 con 0x7f00c40c6b80 2023-10-30T13:12:56.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.721+0000 7f00cc291640 1 -- 172.21.15.148:0/2838937762 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c41f5c40 con 0x7f00c40c6b80 2023-10-30T13:12:56.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00baffd640 1 -- 172.21.15.148:0/2838937762 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00b40b3ca0 con 0x7f00c40c6b80 2023-10-30T13:12:56.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00baffd640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00a80664a0 0x7f00a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00baffd640 1 -- 172.21.15.148:0/2838937762 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f00b411e130 con 0x7f00c40c6b80 2023-10-30T13:12:56.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00cf612000 1 Processor -- start 2023-10-30T13:12:56.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00ca02d640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00a80664a0 0x7f00a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00cf612000 1 -- start start 2023-10-30T13:12:56.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00ca02d640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00a80664a0 0x7f00a8068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009c80 tx=0x7f00c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00cf612000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c391cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00cf612000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c391d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00cf612000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 0x5620c3927aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00cf612000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620c36d0cc0 con 0x5620c391d130 2023-10-30T13:12:56.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00cf612000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620c3733920 con 0x5620c391c810 2023-10-30T13:12:56.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00cf612000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5620c3730e50 con 0x5620c391da50 2023-10-30T13:12:56.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00c982c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c391d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00c982c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c391d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42498/0 (socket says 172.21.15.148:42498) 2023-10-30T13:12:56.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00c982c640 1 -- 172.21.15.148:0/2216670016 learned_addr learned my addr 172.21.15.148:0/2216670016 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00ca02d640 1 --2- 172.21.15.148:0/2216670016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c391cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.725+0000 7f00ca82e640 1 --2- 172.21.15.148:0/2216670016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 0x5620c3927aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00c982c640 1 -- 172.21.15.148:0/2216670016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 msgr2=0x5620c3927aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00c982c640 1 --2- 172.21.15.148:0/2216670016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 0x5620c3927aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00c982c640 1 -- 172.21.15.148:0/2216670016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 msgr2=0x5620c391cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00c982c640 1 --2- 172.21.15.148:0/2216670016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c391cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00c982c640 1 -- 172.21.15.148:0/2216670016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620c3425890 con 0x5620c391d130 2023-10-30T13:12:56.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00c982c640 1 --2- 172.21.15.148:0/2216670016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c391d510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f00b40f7040 tx=0x7f00b40ef320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50303da3dda6177 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00a7fff640 1 -- 172.21.15.148:0/2216670016 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40f7570 con 0x5620c391d130 2023-10-30T13:12:56.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00a7fff640 1 -- 172.21.15.148:0/2216670016 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b40ed6c0 con 0x5620c391d130 2023-10-30T13:12:56.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00a7fff640 1 -- 172.21.15.148:0/2216670016 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40eba30 con 0x5620c391d130 2023-10-30T13:12:56.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 -- 172.21.15.148:0/2216670016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 msgr2=0x5620c391d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 --2- 172.21.15.148:0/2216670016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c391d510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f00b40f7040 tx=0x7f00b40ef320 comp rx=0 tx=0).stop 2023-10-30T13:12:56.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 -- 172.21.15.148:0/2216670016 shutdown_connections 2023-10-30T13:12:56.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 --2- 172.21.15.148:0/2216670016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c391d510 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 --2- 172.21.15.148:0/2216670016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 0x5620c3927aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 --2- 172.21.15.148:0/2216670016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c391cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 -- 172.21.15.148:0/2216670016 >> 172.21.15.148:0/2216670016 conn(0x5620c38d2fc0 msgr2=0x5620c391a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 -- 172.21.15.148:0/2216670016 shutdown_connections 2023-10-30T13:12:56.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 -- 172.21.15.148:0/2216670016 wait complete. 2023-10-30T13:12:56.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 -- start start 2023-10-30T13:12:56.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 Processor -- start 2023-10-30T13:12:56.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391c810 0x5620c38de240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 0x5620c38de780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391da50 0x5620c38decc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620c372c740 con 0x5620c391c810 2023-10-30T13:12:56.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620c36ebd10 con 0x5620c391da50 2023-10-30T13:12:56.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5620c36eb9b0 con 0x5620c391d130 2023-10-30T13:12:56.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00cf612000 10 client.0 _ll_register_callbacks cb 0x5620c38c7390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:12:56.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00ca02d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391c810 0x5620c38de240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00ca02d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391c810 0x5620c38de240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42510/0 (socket says 172.21.15.148:42510) 2023-10-30T13:12:56.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00ca02d640 1 -- 172.21.15.148:0/2803904185 learned_addr learned my addr 172.21.15.148:0/2803904185 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00ca82e640 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391da50 0x5620c38decc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00c982c640 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 0x5620c38de780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00ca02d640 1 -- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 msgr2=0x5620c38de780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00ca02d640 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 0x5620c38de780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00ca02d640 1 -- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391da50 msgr2=0x5620c38decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.729+0000 7f00ca02d640 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391da50 0x5620c38decc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00ca02d640 1 -- 172.21.15.148:0/2803904185 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c41f5c40 con 0x5620c391c810 2023-10-30T13:12:56.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00ca02d640 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391c810 0x5620c38de240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f00c0015920 tx=0x7f00c00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afec2f18b6741d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2023-10-30T13:12:56.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00a57fa640 1 -- 172.21.15.148:0/2803904185 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c0015ea0 con 0x5620c391c810 2023-10-30T13:12:56.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00a57fa640 1 -- 172.21.15.148:0/2803904185 <== mon.0 v2:172.21.15.148:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f00c0014af0 con 0x5620c391c810 2023-10-30T13:12:56.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00cf612000 1 -- 172.21.15.148:0/2803904185 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5620c36e0370 con 0x5620c391c810 2023-10-30T13:12:56.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00a57fa640 1 -- 172.21.15.148:0/2803904185 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00c0014cb0 con 0x5620c391c810 2023-10-30T13:12:56.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00cf612000 10 client.4710 fetch_fsmap learned FSMap version 13 2023-10-30T13:12:56.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00cf612000 1 -- 172.21.15.148:0/2803904185 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x5620c36ee460 con 0x5620c391c810 2023-10-30T13:12:56.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00a57fa640 1 -- 172.21.15.148:0/2803904185 <== mon.0 v2:172.21.15.148:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5620c36e0370 con 0x5620c391c810 2023-10-30T13:12:56.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00a57fa640 1 -- 172.21.15.148:0/2803904185 <== mon.0 v2:172.21.15.148:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f00c002b440 con 0x5620c391c810 2023-10-30T13:12:56.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00cf612000 10 client.4710 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:12:56.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00cf612000 10 client.4710 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-30T13:12:56.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00cf612000 20 client.4710 populate_metadata read hostname 'smithi148' 2023-10-30T13:12:56.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00cf612000 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] conn(0x5620c38df6d0 0x5620c3931900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00cf612000 4 client.4710 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:12:56.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00cf612000 1 -- 172.21.15.148:0/2803904185 --> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5620c36c62d0 con 0x5620c38df6d0 2023-10-30T13:12:56.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00c982c640 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] conn(0x5620c38df6d0 0x5620c3931900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00c982c640 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] conn(0x5620c38df6d0 0x5620c3931900 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b35a83fb29b6e1 server_cookie=35972f79daffe189 in_seq=0 out_seq=0 2023-10-30T13:12:56.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00a57fa640 10 client.4710 ms_handle_connect on v2:172.21.15.150:6834/1620877178 2023-10-30T13:12:56.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00a57fa640 1 -- 172.21.15.148:0/2803904185 <== mds.1 v2:172.21.15.150:6834/1620877178 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5620c36c62d0 con 0x5620c38df6d0 2023-10-30T13:12:56.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.733+0000 7f00a57fa640 10 client.4710 handle_command_reply: tid=0 2023-10-30T13:12:56.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 client.4710 shutdown 2023-10-30T13:12:56.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 -- 172.21.15.148:0/2803904185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391c810 msgr2=0x5620c38de240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391c810 0x5620c38de240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f00c0015920 tx=0x7f00c00046d0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 -- 172.21.15.148:0/2803904185 shutdown_connections 2023-10-30T13:12:56.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] conn(0x5620c38df6d0 0x5620c3931900 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391da50 0x5620c38decc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 0x5620c38de780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00a57fa640 0 client.4710 ms_handle_reset on v2:172.21.15.150:6834/1620877178 2023-10-30T13:12:56.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 --2- 172.21.15.148:0/2803904185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391c810 0x5620c38de240 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 -- 172.21.15.148:0/2803904185 >> 172.21.15.148:0/2803904185 conn(0x5620c38d2fc0 msgr2=0x5620c391c090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 -- 172.21.15.148:0/2803904185 shutdown_connections 2023-10-30T13:12:56.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 1 -- 172.21.15.148:0/2803904185 wait complete. 2023-10-30T13:12:56.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.737+0000 7f00cf612000 20 client.4710 trim_cache size 0 max 16384 2023-10-30T13:12:56.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.749+0000 7f00cf612000 1 Processor -- start 2023-10-30T13:12:56.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.749+0000 7f00cf612000 1 -- start start 2023-10-30T13:12:56.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c38da770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 0x5620c38dacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391da50 0x5620c38db1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620c33e6940 con 0x5620c391da50 2023-10-30T13:12:56.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620c37280b0 con 0x5620c391c810 2023-10-30T13:12:56.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5620c372f2a0 con 0x5620c391d130 2023-10-30T13:12:56.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca82e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391da50 0x5620c38db1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca82e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391da50 0x5620c38db1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42522/0 (socket says 172.21.15.148:42522) 2023-10-30T13:12:56.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca82e640 1 -- 172.21.15.148:0/1245212518 learned_addr learned my addr 172.21.15.148:0/1245212518 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca02d640 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c38da770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00c982c640 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 0x5620c38dacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca82e640 1 -- 172.21.15.148:0/1245212518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 msgr2=0x5620c38dacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca82e640 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 0x5620c38dacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca82e640 1 -- 172.21.15.148:0/1245212518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 msgr2=0x5620c38da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca82e640 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c38da770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca82e640 1 -- 172.21.15.148:0/1245212518 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620c36d4760 con 0x5620c391da50 2023-10-30T13:12:56.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00c982c640 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 0x5620c38dacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:56.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00ca82e640 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391da50 0x5620c38db1f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0051b0 tx=0x7f00bc005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660413af1341ee79 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00b8ff9640 1 -- 172.21.15.148:0/1245212518 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc001b30 con 0x5620c391da50 2023-10-30T13:12:56.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00b8ff9640 1 -- 172.21.15.148:0/1245212518 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc001c90 con 0x5620c391da50 2023-10-30T13:12:56.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00b8ff9640 1 -- 172.21.15.148:0/1245212518 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc002000 con 0x5620c391da50 2023-10-30T13:12:56.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 -- 172.21.15.148:0/1245212518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391da50 msgr2=0x5620c38db1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391da50 0x5620c38db1f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0051b0 tx=0x7f00bc005670 comp rx=0 tx=0).stop 2023-10-30T13:12:56.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 -- 172.21.15.148:0/1245212518 shutdown_connections 2023-10-30T13:12:56.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391da50 0x5620c38db1f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391d130 0x5620c38dacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 --2- 172.21.15.148:0/1245212518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c38da770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 -- 172.21.15.148:0/1245212518 >> 172.21.15.148:0/1245212518 conn(0x5620c38d2fc0 msgr2=0x5620c3919280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 -- 172.21.15.148:0/1245212518 shutdown_connections 2023-10-30T13:12:56.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 -- 172.21.15.148:0/1245212518 wait complete. 2023-10-30T13:12:56.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 -- start start 2023-10-30T13:12:56.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 Processor -- start 2023-10-30T13:12:56.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c397f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.753+0000 7f00cf612000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c397f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 0x5620c3965fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620c376dba0 con 0x5620c391d130 2023-10-30T13:12:56.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620c3730740 con 0x5620c391c810 2023-10-30T13:12:56.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x5620c36f6720 con 0x5620c391da50 2023-10-30T13:12:56.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 10 client.0 _ll_register_callbacks cb 0x7f00b40eb660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:12:56.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00c982c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c397f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00c982c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c397f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42532/0 (socket says 172.21.15.148:42532) 2023-10-30T13:12:56.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00c982c640 1 -- 172.21.15.148:0/1283900268 learned_addr learned my addr 172.21.15.148:0/1283900268 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00ca02d640 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c397f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00ca82e640 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 0x5620c3965fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00c982c640 1 -- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 msgr2=0x5620c3965fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00c982c640 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 0x5620c3965fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00c982c640 1 -- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 msgr2=0x5620c397f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00c982c640 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c397f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00c982c640 1 -- 172.21.15.148:0/1283900268 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00bc00f040 con 0x5620c391d130 2023-10-30T13:12:56.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00ca02d640 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c397f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:56.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00c982c640 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c397f6a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f00b40ef2b0 tx=0x7f00b40ab040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b034affde8b16d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2023-10-30T13:12:56.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 1 -- 172.21.15.148:0/1283900268 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b411c200 con 0x5620c391d130 2023-10-30T13:12:56.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 1 -- 172.21.15.148:0/1283900268 <== mon.0 v2:172.21.15.148:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f00b411c360 con 0x5620c391d130 2023-10-30T13:12:56.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- 172.21.15.148:0/1283900268 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5620c38df090 con 0x5620c391d130 2023-10-30T13:12:56.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 1 -- 172.21.15.148:0/1283900268 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00b40eba30 con 0x5620c391d130 2023-10-30T13:12:56.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 10 client.4716 fetch_fsmap learned FSMap version 13 2023-10-30T13:12:56.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- 172.21.15.148:0/1283900268 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x5620c3425cc0 con 0x5620c391d130 2023-10-30T13:12:56.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 1 -- 172.21.15.148:0/1283900268 <== mon.0 v2:172.21.15.148:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5620c38df090 con 0x5620c391d130 2023-10-30T13:12:56.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 1 -- 172.21.15.148:0/1283900268 <== mon.0 v2:172.21.15.148:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f00b40f70e0 con 0x5620c391d130 2023-10-30T13:12:56.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 10 client.4716 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:12:56.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 10 client.4716 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-30T13:12:56.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 20 client.4716 populate_metadata read hostname 'smithi148' 2023-10-30T13:12:56.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] conn(0x5620c397bc60 0x5620c397c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 4 client.4716 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:12:56.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- 172.21.15.148:0/1283900268 --> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5620c36c46f0 con 0x5620c397bc60 2023-10-30T13:12:56.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00ca02d640 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] conn(0x5620c397bc60 0x5620c397c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00ca02d640 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] conn(0x5620c397bc60 0x5620c397c040 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d2c67ff080ee278 server_cookie=ba792d5de94103a3 in_seq=0 out_seq=0 2023-10-30T13:12:56.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 10 client.4716 ms_handle_connect on v2:172.21.15.150:6834/1620877178 2023-10-30T13:12:56.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 1 -- 172.21.15.148:0/1283900268 <== mds.1 v2:172.21.15.150:6834/1620877178 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5620c36c46f0 con 0x5620c397bc60 2023-10-30T13:12:56.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 10 client.4716 handle_command_reply: tid=0 2023-10-30T13:12:56.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 client.4716 shutdown 2023-10-30T13:12:56.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- 172.21.15.148:0/1283900268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 msgr2=0x5620c397f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c397f6a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f00b40ef2b0 tx=0x7f00b40ab040 comp rx=0 tx=0).stop 2023-10-30T13:12:56.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- 172.21.15.148:0/1283900268 shutdown_connections 2023-10-30T13:12:56.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:6834/1620877178,v1:172.21.15.150:6836/1620877178] conn(0x5620c397bc60 0x5620c397c040 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5620c391d130 0x5620c397f6a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x5620c391da50 0x5620c3965fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 --2- 172.21.15.148:0/1283900268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5620c391c810 0x5620c397f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00a57fa640 0 client.4716 ms_handle_reset on v2:172.21.15.150:6834/1620877178 2023-10-30T13:12:56.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- 172.21.15.148:0/1283900268 >> 172.21.15.148:0/1283900268 conn(0x5620c38d2fc0 msgr2=0x5620c3919b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- 172.21.15.148:0/1283900268 shutdown_connections 2023-10-30T13:12:56.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 1 -- 172.21.15.148:0/1283900268 wait complete. 2023-10-30T13:12:56.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.757+0000 7f00cf612000 20 client.4716 trim_cache size 0 max 16384 2023-10-30T13:12:56.775 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:12:56.775 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_fsid": "23a3d51e-f2ab-4ffa-93fe-42135bef502c", 2023-10-30T13:12:56.776 INFO:teuthology.orchestra.run.smithi148.stdout: "whoami": 1, 2023-10-30T13:12:56.776 INFO:teuthology.orchestra.run.smithi148.stdout: "id": 4270, 2023-10-30T13:12:56.776 INFO:teuthology.orchestra.run.smithi148.stdout: "want_state": "up:active", 2023-10-30T13:12:56.776 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "up:active", 2023-10-30T13:12:56.776 INFO:teuthology.orchestra.run.smithi148.stdout: "fs_name": "cephfs", 2023-10-30T13:12:56.776 INFO:teuthology.orchestra.run.smithi148.stdout: "rank_uptime": 136.02835322799999, 2023-10-30T13:12:56.777 INFO:teuthology.orchestra.run.smithi148.stdout: "mdsmap_epoch": 13, 2023-10-30T13:12:56.777 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_epoch": 22, 2023-10-30T13:12:56.777 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_epoch_barrier": 20, 2023-10-30T13:12:56.777 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 148.80459731600001 2023-10-30T13:12:56.777 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:12:56.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 -- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00a80664a0 msgr2=0x7f00a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00a80664a0 0x7f00a8068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009c80 tx=0x7f00c0008040 comp rx=0 tx=0).stop 2023-10-30T13:12:56.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 -- 172.21.15.148:0/2838937762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c40c6b80 msgr2=0x7f00c4152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c40c6b80 0x7f00c4152290 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f00b401cdd0 tx=0x7f00b401b4c0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 -- 172.21.15.148:0/2838937762 shutdown_connections 2023-10-30T13:12:56.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00a80664a0 0x7f00a8068960 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c40c6b80 0x7f00c4152290 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00c40c6f60 0x7f00c40c2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 --2- 172.21.15.148:0/2838937762 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00c40c08d0 0x7f00c413c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 -- 172.21.15.148:0/2838937762 >> 172.21.15.148:0/2838937762 conn(0x7f00c40c7910 msgr2=0x7f00c413e720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 -- 172.21.15.148:0/2838937762 shutdown_connections 2023-10-30T13:12:56.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.761+0000 7f00cc291640 1 -- 172.21.15.148:0/2838937762 wait complete. 2023-10-30T13:12:56.781 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-10-30T13:12:56.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb689310640 1 Processor -- start 2023-10-30T13:12:56.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb689310640 1 -- start start 2023-10-30T13:12:56.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb689310640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840c5550 0x7fb6840cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb689310640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb6840cab00 0x7fb6840cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb689310640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6840cba30 con 0x7fb6840cb050 2023-10-30T13:12:56.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb689310640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6840ca290 con 0x7fb6840cab00 2023-10-30T13:12:56.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb689310640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840c9cf0 con 0x7fb6840c5550 2023-10-30T13:12:56.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb683fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840c5550 0x7fb6840cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb688b0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb6840cab00 0x7fb6840cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb683fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840c5550 0x7fb6840cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:60224/0 (socket says 172.21.15.148:60224) 2023-10-30T13:12:56.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb683fff640 1 -- 172.21.15.148:0/991999917 learned_addr learned my addr 172.21.15.148:0/991999917 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb683fff640 1 -- 172.21.15.148:0/991999917 >> v2:172.21.15.150:3300/0 conn(0x7fb6840cab00 msgr2=0x7fb6840cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb683fff640 1 --2- 172.21.15.148:0/991999917 >> v2:172.21.15.150:3300/0 conn(0x7fb6840cab00 0x7fb6840cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb683fff640 1 -- 172.21.15.148:0/991999917 >> v1:172.21.15.150:6789/0 conn(0x7fb6840cb050 legacy=0x7fb6840cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb683fff640 1 -- 172.21.15.148:0/991999917 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68415d990 con 0x7fb6840c5550 2023-10-30T13:12:56.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.917+0000 7fb688b0f640 1 --2- 172.21.15.148:0/991999917 >> v2:172.21.15.150:3300/0 conn(0x7fb6840cab00 0x7fb6840cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:12:56.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb683fff640 1 --2- 172.21.15.148:0/991999917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840c5550 0x7fb6840cef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb674009870 tx=0x7fb67401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=850e4e36482a91a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb682ffd640 1 -- 172.21.15.148:0/991999917 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740a7070 con 0x7fb6840c5550 2023-10-30T13:12:56.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb682ffd640 1 -- 172.21.15.148:0/991999917 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6740ab430 con 0x7fb6840c5550 2023-10-30T13:12:56.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb682ffd640 1 -- 172.21.15.148:0/991999917 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740ab7c0 con 0x7fb6840c5550 2023-10-30T13:12:56.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb689310640 1 -- 172.21.15.148:0/991999917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840c5550 msgr2=0x7fb6840cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb689310640 1 --2- 172.21.15.148:0/991999917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840c5550 0x7fb6840cef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb674009870 tx=0x7fb67401c540 comp rx=0 tx=0).stop 2023-10-30T13:12:56.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb689310640 1 -- 172.21.15.148:0/991999917 shutdown_connections 2023-10-30T13:12:56.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb689310640 1 --2- 172.21.15.148:0/991999917 >> v2:172.21.15.150:3300/0 conn(0x7fb6840cab00 0x7fb6840cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb689310640 1 --2- 172.21.15.148:0/991999917 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840c5550 0x7fb6840cef00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb689310640 1 -- 172.21.15.148:0/991999917 >> 172.21.15.148:0/991999917 conn(0x7fb68406bed0 msgr2=0x7fb68406c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb689310640 1 -- 172.21.15.148:0/991999917 shutdown_connections 2023-10-30T13:12:56.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.921+0000 7fb689310640 1 -- 172.21.15.148:0/991999917 wait complete. 2023-10-30T13:12:56.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb689310640 1 Processor -- start 2023-10-30T13:12:56.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb689310640 1 -- start start 2023-10-30T13:12:56.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb689310640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6840c5550 0x7fb68414f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb689310640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840cab00 0x7fb6840c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb689310640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6840cb050 0x7fb684147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb689310640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb684153900 con 0x7fb6840cb050 2023-10-30T13:12:56.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb689310640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68414fe40 con 0x7fb6840c5550 2023-10-30T13:12:56.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb689310640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb684150120 con 0x7fb6840cab00 2023-10-30T13:12:56.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb688b0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6840cb050 0x7fb684147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb688b0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6840cb050 0x7fb684147330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42538/0 (socket says 172.21.15.148:42538) 2023-10-30T13:12:56.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb688b0f640 1 -- 172.21.15.148:0/695262243 learned_addr learned my addr 172.21.15.148:0/695262243 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb683fff640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6840c5550 0x7fb68414f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb6837fe640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840cab00 0x7fb6840c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb688b0f640 1 -- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840cab00 msgr2=0x7fb6840c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb688b0f640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840cab00 0x7fb6840c3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb688b0f640 1 -- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6840c5550 msgr2=0x7fb68414f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb688b0f640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6840c5550 0x7fb68414f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb688b0f640 1 -- 172.21.15.148:0/695262243 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c009a70 con 0x7fb6840cb050 2023-10-30T13:12:56.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb683fff640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6840c5550 0x7fb68414f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:56.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb688b0f640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6840cb050 0x7fb684147330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00efc0 tx=0x7fb67c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb680ff9640 1 -- 172.21.15.148:0/695262243 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c009d70 con 0x7fb6840cb050 2023-10-30T13:12:56.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb680ff9640 1 -- 172.21.15.148:0/695262243 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c00f040 con 0x7fb6840cb050 2023-10-30T13:12:56.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb680ff9640 1 -- 172.21.15.148:0/695262243 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0039a0 con 0x7fb6840cb050 2023-10-30T13:12:56.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.925+0000 7fb689310640 1 -- 172.21.15.148:0/695262243 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6740094d0 con 0x7fb6840cb050 2023-10-30T13:12:56.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.929+0000 7fb689310640 1 -- 172.21.15.148:0/695262243 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6841506e0 con 0x7fb6840cb050 2023-10-30T13:12:56.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.929+0000 7fb680ff9640 1 -- 172.21.15.148:0/695262243 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb67c0144a0 con 0x7fb6840cb050 2023-10-30T13:12:56.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.929+0000 7fb680ff9640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6700664a0 0x7fb670068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.929+0000 7fb680ff9640 1 -- 172.21.15.148:0/695262243 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb67c084d40 con 0x7fb6840cb050 2023-10-30T13:12:56.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.929+0000 7fb683fff640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6700664a0 0x7fb670068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.929+0000 7fb68d900000 1 Processor -- start 2023-10-30T13:12:56.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.929+0000 7fb68d900000 1 -- start start 2023-10-30T13:12:56.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.929+0000 7fb683fff640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6700664a0 0x7fb670068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb6740af000 tx=0x7fb674008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df096ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df096f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 0x559df0979aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x559df0722cc0 con 0x559df096e810 2023-10-30T13:12:56.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559df0785920 con 0x559df096f130 2023-10-30T13:12:56.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559df0782e50 con 0x559df096fa50 2023-10-30T13:12:56.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb683fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df096ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb683fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df096ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42546/0 (socket says 172.21.15.148:42546) 2023-10-30T13:12:56.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb683fff640 1 -- 172.21.15.148:0/60286225 learned_addr learned my addr 172.21.15.148:0/60286225 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb688b0f640 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 0x559df0979aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb6837fe640 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df096f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb683fff640 1 -- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 msgr2=0x559df0979aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb683fff640 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 0x559df0979aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb683fff640 1 -- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 msgr2=0x559df096f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb683fff640 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df096f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb683fff640 1 -- 172.21.15.148:0/60286225 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559df0477890 con 0x559df096e810 2023-10-30T13:12:56.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb688b0f640 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 0x559df0979aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:12:56.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb6837fe640 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df096f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:12:56.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb683fff640 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df096ebf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb674000c10 tx=0x7fb6740a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e45551dba9a5bf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb66a7fc640 1 -- 172.21.15.148:0/60286225 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740b4b40 con 0x559df096e810 2023-10-30T13:12:56.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb66a7fc640 1 -- 172.21.15.148:0/60286225 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6740bc070 con 0x559df096e810 2023-10-30T13:12:56.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb66a7fc640 1 -- 172.21.15.148:0/60286225 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740abe70 con 0x559df096e810 2023-10-30T13:12:56.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 -- 172.21.15.148:0/60286225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 msgr2=0x559df096ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df096ebf0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb674000c10 tx=0x7fb6740a7000 comp rx=0 tx=0).stop 2023-10-30T13:12:56.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 -- 172.21.15.148:0/60286225 shutdown_connections 2023-10-30T13:12:56.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 0x559df0979aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df096f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 --2- 172.21.15.148:0/60286225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df096ebf0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 -- 172.21.15.148:0/60286225 >> 172.21.15.148:0/60286225 conn(0x559df0924fc0 msgr2=0x559df096c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 -- 172.21.15.148:0/60286225 shutdown_connections 2023-10-30T13:12:56.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 -- 172.21.15.148:0/60286225 wait complete. 2023-10-30T13:12:56.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 -- start start 2023-10-30T13:12:56.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.933+0000 7fb68d900000 1 Processor -- start 2023-10-30T13:12:56.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 0x559df0968b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df09690a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09695e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x559df077e740 con 0x559df096fa50 2023-10-30T13:12:56.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559df073dd10 con 0x559df096f130 2023-10-30T13:12:56.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559df073d9b0 con 0x559df096e810 2023-10-30T13:12:56.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 10 client.0 _ll_register_callbacks cb 0x559df0919390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:12:56.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb688b0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb688b0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09695e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42554/0 (socket says 172.21.15.148:42554) 2023-10-30T13:12:56.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb688b0f640 1 -- 172.21.15.148:0/2048348562 learned_addr learned my addr 172.21.15.148:0/2048348562 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb683fff640 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 0x559df0968b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb6837fe640 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df09690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb688b0f640 1 -- 172.21.15.148:0/2048348562 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 msgr2=0x559df0968b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb688b0f640 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 0x559df0968b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb688b0f640 1 -- 172.21.15.148:0/2048348562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 msgr2=0x559df09690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb688b0f640 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df09690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb688b0f640 1 -- 172.21.15.148:0/2048348562 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68415d990 con 0x559df096fa50 2023-10-30T13:12:56.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb683fff640 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 0x559df0968b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:56.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb688b0f640 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09695e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0830e0 tx=0x7fb67c059ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b597e3013725dc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2023-10-30T13:12:56.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 1 -- 172.21.15.148:0/2048348562 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c054120 con 0x559df096fa50 2023-10-30T13:12:56.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 1 -- 172.21.15.148:0/2048348562 <== mon.0 v2:172.21.15.148:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb67c05dbc0 con 0x559df096fa50 2023-10-30T13:12:56.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- 172.21.15.148:0/2048348562 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559df0732370 con 0x559df096fa50 2023-10-30T13:12:56.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 1 -- 172.21.15.148:0/2048348562 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb67c05de00 con 0x559df096fa50 2023-10-30T13:12:56.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 10 client.4725 fetch_fsmap learned FSMap version 13 2023-10-30T13:12:56.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- 172.21.15.148:0/2048348562 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x559df0740460 con 0x559df096fa50 2023-10-30T13:12:56.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 1 -- 172.21.15.148:0/2048348562 <== mon.0 v2:172.21.15.148:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559df0732370 con 0x559df096fa50 2023-10-30T13:12:56.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 1 -- 172.21.15.148:0/2048348562 <== mon.0 v2:172.21.15.148:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fb67c005d20 con 0x559df096fa50 2023-10-30T13:12:56.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 10 client.4725 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:12:56.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 10 client.4725 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-30T13:12:56.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 20 client.4725 populate_metadata read hostname 'smithi148' 2023-10-30T13:12:56.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] conn(0x559df0982260 0x559df0984650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 4 client.4725 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T13:12:56.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- 172.21.15.148:0/2048348562 --> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559df07182d0 con 0x559df0982260 2023-10-30T13:12:56.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb683fff640 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] conn(0x559df0982260 0x559df0984650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 10 client.4725 ms_handle_connect on v2:172.21.15.148:6832/4162935019 2023-10-30T13:12:56.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb683fff640 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] conn(0x559df0982260 0x559df0984650 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df11ee5e282cb81f server_cookie=3bad8e8b101f22f in_seq=0 out_seq=0 2023-10-30T13:12:56.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 1 -- 172.21.15.148:0/2048348562 <== mds.0 v2:172.21.15.148:6832/4162935019 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559df07182d0 con 0x559df0982260 2023-10-30T13:12:56.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 10 client.4725 handle_command_reply: tid=0 2023-10-30T13:12:56.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 client.4725 shutdown 2023-10-30T13:12:56.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- 172.21.15.148:0/2048348562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 msgr2=0x559df09695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09695e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0830e0 tx=0x7fb67c059ed0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- 172.21.15.148:0/2048348562 shutdown_connections 2023-10-30T13:12:56.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] conn(0x559df0982260 0x559df0984650 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09695e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df09690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb662ffd640 0 client.4725 ms_handle_reset on v2:172.21.15.148:6832/4162935019 2023-10-30T13:12:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 --2- 172.21.15.148:0/2048348562 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 0x559df0968b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- 172.21.15.148:0/2048348562 >> 172.21.15.148:0/2048348562 conn(0x559df0924fc0 msgr2=0x559df096e090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- 172.21.15.148:0/2048348562 shutdown_connections 2023-10-30T13:12:56.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 1 -- 172.21.15.148:0/2048348562 wait complete. 2023-10-30T13:12:56.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.937+0000 7fb68d900000 20 client.4725 trim_cache size 0 max 16384 2023-10-30T13:12:56.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb68d900000 1 Processor -- start 2023-10-30T13:12:56.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb68d900000 1 -- start start 2023-10-30T13:12:56.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb68d900000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 0x559df0971e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb68d900000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df0972380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb68d900000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb68d900000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x559df0438940 con 0x559df096fa50 2023-10-30T13:12:56.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb68d900000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559df071f1a0 con 0x559df096f130 2023-10-30T13:12:56.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb68d900000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559df077a0b0 con 0x559df096e810 2023-10-30T13:12:56.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb688b0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb688b0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09728c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42570/0 (socket says 172.21.15.148:42570) 2023-10-30T13:12:56.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb688b0f640 1 -- 172.21.15.148:0/719136024 learned_addr learned my addr 172.21.15.148:0/719136024 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb683fff640 1 --2- 172.21.15.148:0/719136024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 0x559df0971e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb6837fe640 1 --2- 172.21.15.148:0/719136024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df0972380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb688b0f640 1 -- 172.21.15.148:0/719136024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 msgr2=0x559df0971e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb688b0f640 1 --2- 172.21.15.148:0/719136024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 0x559df0971e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb688b0f640 1 -- 172.21.15.148:0/719136024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 msgr2=0x559df0972380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.953+0000 7fb688b0f640 1 --2- 172.21.15.148:0/719136024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df0972380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb688b0f640 1 -- 172.21.15.148:0/719136024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559df0980f50 con 0x559df096fa50 2023-10-30T13:12:56.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb688b0f640 1 --2- 172.21.15.148:0/719136024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09728c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0526e0 tx=0x7fb67c05e1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510cc7a477dc3cc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb66affd640 1 -- 172.21.15.148:0/719136024 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c054120 con 0x559df096fa50 2023-10-30T13:12:56.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb66affd640 1 -- 172.21.15.148:0/719136024 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c053e00 con 0x559df096fa50 2023-10-30T13:12:56.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb66affd640 1 -- 172.21.15.148:0/719136024 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c05dbc0 con 0x559df096fa50 2023-10-30T13:12:56.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- 172.21.15.148:0/719136024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 msgr2=0x559df09728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 --2- 172.21.15.148:0/719136024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09728c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0526e0 tx=0x7fb67c05e1d0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- 172.21.15.148:0/719136024 shutdown_connections 2023-10-30T13:12:56.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 --2- 172.21.15.148:0/719136024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096fa50 0x559df09728c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 --2- 172.21.15.148:0/719136024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df0972380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 --2- 172.21.15.148:0/719136024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096e810 0x559df0971e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- 172.21.15.148:0/719136024 >> 172.21.15.148:0/719136024 conn(0x559df0924fc0 msgr2=0x559df0933470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- 172.21.15.148:0/719136024 shutdown_connections 2023-10-30T13:12:56.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- 172.21.15.148:0/719136024 wait complete. 2023-10-30T13:12:56.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- start start 2023-10-30T13:12:56.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 Processor -- start 2023-10-30T13:12:56.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df09cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df09cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 0x559df09cc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x559df096dc60 con 0x559df096e810 2023-10-30T13:12:56.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559df07812a0 con 0x559df096f130 2023-10-30T13:12:56.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x559df097fe70 con 0x559df096fa50 2023-10-30T13:12:56.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 10 client.0 _ll_register_callbacks cb 0x7fb67c0095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T13:12:56.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb683fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df09cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb683fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df09cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42582/0 (socket says 172.21.15.148:42582) 2023-10-30T13:12:56.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb683fff640 1 -- 172.21.15.148:0/1300010377 learned_addr learned my addr 172.21.15.148:0/1300010377 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:56.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb688b0f640 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 0x559df09cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb6837fe640 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df09cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb683fff640 1 -- 172.21.15.148:0/1300010377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 msgr2=0x559df09cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb683fff640 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 0x559df09cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb683fff640 1 -- 172.21.15.148:0/1300010377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 msgr2=0x559df09cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:56.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb683fff640 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df09cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb683fff640 1 -- 172.21.15.148:0/1300010377 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559df0740460 con 0x559df096e810 2023-10-30T13:12:56.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb683fff640 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df09cb660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb674019150 tx=0x7fb67401ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42fd6fb10d95b92 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:56.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb662ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2023-10-30T13:12:56.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb662ffd640 1 -- 172.21.15.148:0/1300010377 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740d1d40 con 0x559df096e810 2023-10-30T13:12:56.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb662ffd640 1 -- 172.21.15.148:0/1300010377 <== mon.0 v2:172.21.15.148:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb6740b9040 con 0x559df096e810 2023-10-30T13:12:56.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.957+0000 7fb68d900000 1 -- 172.21.15.148:0/1300010377 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559df07166f0 con 0x559df096e810 2023-10-30T13:12:56.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb662ffd640 1 -- 172.21.15.148:0/1300010377 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6740d1ea0 con 0x559df096e810 2023-10-30T13:12:56.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 10 client.4731 fetch_fsmap learned FSMap version 13 2023-10-30T13:12:56.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 -- 172.21.15.148:0/1300010377 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x559df0726760 con 0x559df096e810 2023-10-30T13:12:56.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb662ffd640 1 -- 172.21.15.148:0/1300010377 <== mon.0 v2:172.21.15.148:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559df07166f0 con 0x559df096e810 2023-10-30T13:12:56.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb662ffd640 1 -- 172.21.15.148:0/1300010377 <== mon.0 v2:172.21.15.148:3300/0 5 ==== fsmap(e 13) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fb6740d1130 con 0x559df096e810 2023-10-30T13:12:56.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 10 client.4731 fetch_fsmap finished waiting for FSMap version 13 2023-10-30T13:12:56.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 10 client.4731 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-30T13:12:56.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 20 client.4731 populate_metadata read hostname 'smithi148' 2023-10-30T13:12:56.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] conn(0x559df09ccf00 0x559df09cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:56.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 4 client.4731 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T13:12:56.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 -- 172.21.15.148:0/1300010377 --> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559df0777720 con 0x559df09ccf00 2023-10-30T13:12:56.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb6837fe640 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] conn(0x559df09ccf00 0x559df09cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:56.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb6837fe640 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] conn(0x559df09ccf00 0x559df09cd2e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9263ee51740284d server_cookie=20866b6cafd43786 in_seq=0 out_seq=0 2023-10-30T13:12:56.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb662ffd640 10 client.4731 ms_handle_connect on v2:172.21.15.148:6832/4162935019 2023-10-30T13:12:56.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb662ffd640 1 -- 172.21.15.148:0/1300010377 <== mds.0 v2:172.21.15.148:6832/4162935019 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x559df0777720 con 0x559df09ccf00 2023-10-30T13:12:56.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb662ffd640 10 client.4731 handle_command_reply: tid=0 2023-10-30T13:12:56.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 client.4731 shutdown 2023-10-30T13:12:56.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 -- 172.21.15.148:0/1300010377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 msgr2=0x559df09cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df09cb660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb674019150 tx=0x7fb67401ad80 comp rx=0 tx=0).stop 2023-10-30T13:12:56.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 -- 172.21.15.148:0/1300010377 shutdown_connections 2023-10-30T13:12:56.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.148:6832/4162935019,v1:172.21.15.148:6834/4162935019] conn(0x559df09ccf00 0x559df09cd2e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x559df096fa50 0x559df09cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559df096f130 0x559df09cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 --2- 172.21.15.148:0/1300010377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559df096e810 0x559df09cb660 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb662ffd640 0 client.4731 ms_handle_reset on v2:172.21.15.148:6832/4162935019 2023-10-30T13:12:56.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 -- 172.21.15.148:0/1300010377 >> 172.21.15.148:0/1300010377 conn(0x559df0924fc0 msgr2=0x559df0932be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 -- 172.21.15.148:0/1300010377 shutdown_connections 2023-10-30T13:12:56.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 1 -- 172.21.15.148:0/1300010377 wait complete. 2023-10-30T13:12:56.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb68d900000 20 client.4731 trim_cache size 0 max 16384 2023-10-30T13:12:56.978 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:12:56.978 INFO:teuthology.orchestra.run.smithi148.stdout: "cluster_fsid": "23a3d51e-f2ab-4ffa-93fe-42135bef502c", 2023-10-30T13:12:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "whoami": 0, 2023-10-30T13:12:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "id": 4285, 2023-10-30T13:12:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "want_state": "up:active", 2023-10-30T13:12:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "up:active", 2023-10-30T13:12:56.979 INFO:teuthology.orchestra.run.smithi148.stdout: "fs_name": "cephfs", 2023-10-30T13:12:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: "rank_uptime": 142.28281571599999, 2023-10-30T13:12:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: "mdsmap_epoch": 13, 2023-10-30T13:12:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_epoch": 22, 2023-10-30T13:12:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_epoch_barrier": 20, 2023-10-30T13:12:56.980 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 148.961806556 2023-10-30T13:12:56.981 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:12:56.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 -- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6700664a0 msgr2=0x7fb670068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6700664a0 0x7fb670068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb6740af000 tx=0x7fb674008040 comp rx=0 tx=0).stop 2023-10-30T13:12:56.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 -- 172.21.15.148:0/695262243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6840cb050 msgr2=0x7fb684147330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6840cb050 0x7fb684147330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00efc0 tx=0x7fb67c009f90 comp rx=0 tx=0).stop 2023-10-30T13:12:56.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 -- 172.21.15.148:0/695262243 shutdown_connections 2023-10-30T13:12:56.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6700664a0 0x7fb670068960 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6840cb050 0x7fb684147330 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6840cab00 0x7fb6840c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 --2- 172.21.15.148:0/695262243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6840c5550 0x7fb68414f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 -- 172.21.15.148:0/695262243 >> 172.21.15.148:0/695262243 conn(0x7fb68406bed0 msgr2=0x7fb6840cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 -- 172.21.15.148:0/695262243 shutdown_connections 2023-10-30T13:12:56.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:56.961+0000 7fb689310640 1 -- 172.21.15.148:0/695262243 wait complete. 2023-10-30T13:12:56.985 INFO:teuthology.run_tasks:Running task kclient... 2023-10-30T13:12:57.007 INFO:tasks.kclient:Mounting kernel clients... 2023-10-30T13:12:57.007 INFO:tasks.kclient:config is {'client.0': {}} 2023-10-30T13:12:57.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.008 DEBUG:teuthology.orchestra.run.smithi148:> ip netns list 2023-10-30T13:12:57.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.018 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip link delete ceph-brx 2023-10-30T13:12:57.085 INFO:teuthology.orchestra.run.smithi148.stderr:Cannot find device "ceph-brx" 2023-10-30T13:12:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:12:57.087 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-30T13:12:57.087 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-30T13:12:57.088 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T13:12:57.088 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-30T13:12:57.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f686069c640 1 Processor -- start 2023-10-30T13:12:57.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f686069c640 1 -- start start 2023-10-30T13:12:57.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f686069c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68580cf160 0x7f68580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:57.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f686069c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68580cb800 con 0x7f68581538a0 2023-10-30T13:12:57.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f686069c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68580cb330 con 0x7f685815d6f0 2023-10-30T13:12:57.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f686069c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68580cad70 con 0x7f68580cf160 2023-10-30T13:12:57.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685e438640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f68581538a0 0x7f68580ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44854/0 (socket says 172.21.15.148:44854) 2023-10-30T13:12:57.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685e438640 1 -- 172.21.15.148:0/1094769923 learned_addr learned my addr 172.21.15.148:0/1094769923 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:12:57.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685ec39640 1 --2- 172.21.15.148:0/1094769923 >> v2:172.21.15.150:3300/0 conn(0x7f68580cf160 0x7f68580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:57.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204183023 0 0) 0x7f68580cb800 con 0x7f68581538a0 2023-10-30T13:12:57.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6844003530 con 0x7f68581538a0 2023-10-30T13:12:57.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 922659851 0 0) 0x7f6844003530 con 0x7f68581538a0 2023-10-30T13:12:57.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 >> v2:172.21.15.150:3300/0 conn(0x7f68580cf160 msgr2=0x7f68580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:57.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 --2- 172.21.15.148:0/1094769923 >> v2:172.21.15.150:3300/0 conn(0x7f68580cf160 0x7f68580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 >> v1:172.21.15.150:6789/0 conn(0x7f685815d6f0 legacy=0x7f685815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:57.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68580cb050 con 0x7f68581538a0 2023-10-30T13:12:57.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f6854001680 con 0x7f68581538a0 2023-10-30T13:12:57.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 >> v1:172.21.15.148:6789/0 conn(0x7f68581538a0 legacy=0x7f68580ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:57.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6844003980 0x7f6844003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:57.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68440047d0 0x7f6844004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:57.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6844005190 0x7f684400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:57.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580cad70 con 0x7f6844005190 2023-10-30T13:12:57.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684400c040 con 0x7f68440047d0 2023-10-30T13:12:57.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684400c200 con 0x7f6844003980 2023-10-30T13:12:57.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685ec39640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6844005190 0x7f684400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:57.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685e438640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6844003980 0x7f6844003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:57.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685dc37640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68440047d0 0x7f6844004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:57.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685ec39640 1 -- 172.21.15.148:0/1094769923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6844003980 msgr2=0x7f6844003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:57.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685ec39640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6844003980 0x7f6844003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685ec39640 1 -- 172.21.15.148:0/1094769923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68440047d0 msgr2=0x7f6844004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:57.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685ec39640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68440047d0 0x7f6844004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.265+0000 7f685ec39640 1 -- 172.21.15.148:0/1094769923 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6844003530 con 0x7f6844005190 2023-10-30T13:12:57.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f685ec39640 1 -- 172.21.15.148:0/1094769923 reap_dead start 2023-10-30T13:12:57.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f685ec39640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6844005190 0x7f684400b990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6850009fc0 tx=0x7f685001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9033ae4c2303f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:57.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68500a5cb0 con 0x7f6844005190 2023-10-30T13:12:57.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685000f040 con 0x7f6844005190 2023-10-30T13:12:57.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f685d436640 1 -- 172.21.15.148:0/1094769923 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850003be0 con 0x7f6844005190 2023-10-30T13:12:57.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f686069c640 1 -- 172.21.15.148:0/1094769923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6844005190 msgr2=0x7f684400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:57.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f686069c640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6844005190 0x7f684400b990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6850009fc0 tx=0x7f685001cca0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f686069c640 1 -- 172.21.15.148:0/1094769923 shutdown_connections 2023-10-30T13:12:57.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f686069c640 1 --2- 172.21.15.148:0/1094769923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6844005190 0x7f684400b990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f686069c640 1 -- 172.21.15.148:0/1094769923 >> 172.21.15.148:0/1094769923 conn(0x7f685806bfe0 msgr2=0x7f685806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:57.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f686069c640 1 -- 172.21.15.148:0/1094769923 shutdown_connections 2023-10-30T13:12:57.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.269+0000 7f686069c640 1 -- 172.21.15.148:0/1094769923 wait complete. 2023-10-30T13:12:57.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 Processor -- start 2023-10-30T13:12:57.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 -- start start 2023-10-30T13:12:57.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6844005190 0x7f6858149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:57.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68580ce160 0x7f68580cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:57.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68581f2960 0x7f68581f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:57.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6858146d60 con 0x7f68581f2960 2023-10-30T13:12:57.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6858149510 con 0x7f6844005190 2023-10-30T13:12:57.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68581f24d0 con 0x7f68580ce160 2023-10-30T13:12:57.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685ec39640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68581f2960 0x7f68581f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:57.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685ec39640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68581f2960 0x7f68581f2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42608/0 (socket says 172.21.15.148:42608) 2023-10-30T13:12:57.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685ec39640 1 -- 172.21.15.148:0/1855887131 learned_addr learned my addr 172.21.15.148:0/1855887131 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:57.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685e438640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6844005190 0x7f6858149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:57.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685dc37640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68580ce160 0x7f68580cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:57.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685ec39640 1 -- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68580ce160 msgr2=0x7f68580cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:57.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685ec39640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68580ce160 0x7f68580cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685ec39640 1 -- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6844005190 msgr2=0x7f6858149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:57.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685ec39640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6844005190 0x7f6858149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685ec39640 1 -- 172.21.15.148:0/1855887131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68581eed30 con 0x7f68581f2960 2023-10-30T13:12:57.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685dc37640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68580ce160 0x7f68580cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:12:57.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f685ec39640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68581f2960 0x7f68581f2de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f685001d180 tx=0x7f6850009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:57.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f684effd640 1 -- 172.21.15.148:0/1855887131 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68500aecb0 con 0x7f68581f2960 2023-10-30T13:12:57.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f684effd640 1 -- 172.21.15.148:0/1855887131 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68500aee30 con 0x7f68581f2960 2023-10-30T13:12:57.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6858149a20 con 0x7f68581f2960 2023-10-30T13:12:57.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f684effd640 1 -- 172.21.15.148:0/1855887131 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850003be0 con 0x7f68581f2960 2023-10-30T13:12:57.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.273+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68581467a0 con 0x7f68581f2960 2023-10-30T13:12:57.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.277+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685804e170 con 0x7f68581f2960 2023-10-30T13:12:57.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.277+0000 7f684effd640 1 -- 172.21.15.148:0/1855887131 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6850003d40 con 0x7f68581f2960 2023-10-30T13:12:57.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.277+0000 7f684effd640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f683c0664a0 0x7f683c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:57.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.277+0000 7f684effd640 1 -- 172.21.15.148:0/1855887131 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f68500a6e40 con 0x7f68581f2960 2023-10-30T13:12:57.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.277+0000 7f685e438640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f683c0664a0 0x7f683c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:57.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.277+0000 7f685e438640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f683c0664a0 0x7f683c068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6854000c10 tx=0x7f6854009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:57.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.281+0000 7f684effd640 1 -- 172.21.15.148:0/1855887131 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f68500ec6f0 con 0x7f68581f2960 2023-10-30T13:12:57.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.465+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f68580b1dd0 con 0x7f68581f2960 2023-10-30T13:12:57.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.465+0000 7f684effd640 1 -- 172.21.15.148:0/1855887131 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+73 (secure 0 0 0) 0x7f68500efd10 con 0x7f68581f2960 2023-10-30T13:12:57.470 INFO:teuthology.orchestra.run.smithi148.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-30T13:12:57.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f683c0664a0 msgr2=0x7f683c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:57.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f683c0664a0 0x7f683c068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6854000c10 tx=0x7f6854009040 comp rx=0 tx=0).stop 2023-10-30T13:12:57.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68581f2960 msgr2=0x7f68581f2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:57.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68581f2960 0x7f68581f2de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f685001d180 tx=0x7f6850009750 comp rx=0 tx=0).stop 2023-10-30T13:12:57.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 shutdown_connections 2023-10-30T13:12:57.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f683c0664a0 0x7f683c068960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68581f2960 0x7f68581f2de0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68580ce160 0x7f68580cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 --2- 172.21.15.148:0/1855887131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6844005190 0x7f6858149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:57.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 >> 172.21.15.148:0/1855887131 conn(0x7f685806bfe0 msgr2=0x7f68581587a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:57.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 shutdown_connections 2023-10-30T13:12:57.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:57.469+0000 7f686069c640 1 -- 172.21.15.148:0/1855887131 wait complete. 2023-10-30T13:12:57.491 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-30T13:12:57.491 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-30T13:12:57.492 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi148.front.sepia.ceph.com 2023-10-30T13:12:57.492 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-30T13:12:57.492 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T13:12:57.492 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-30T13:12:57.493 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-30T13:12:57.493 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-30T13:12:57.493 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-30T13:12:57.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.494 DEBUG:teuthology.orchestra.run.smithi148:> ip addr 2023-10-30T13:12:57.503 INFO:teuthology.orchestra.run.smithi148.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-30T13:12:57.503 INFO:teuthology.orchestra.run.smithi148.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-30T13:12:57.503 INFO:teuthology.orchestra.run.smithi148.stdout: inet 127.0.0.1/8 scope host lo 2023-10-30T13:12:57.503 INFO:teuthology.orchestra.run.smithi148.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:12:57.504 INFO:teuthology.orchestra.run.smithi148.stdout: inet6 ::1/128 scope host 2023-10-30T13:12:57.504 INFO:teuthology.orchestra.run.smithi148.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:12:57.504 INFO:teuthology.orchestra.run.smithi148.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T13:12:57.504 INFO:teuthology.orchestra.run.smithi148.stdout: link/ether 0c:c4:7a:da:6a:a4 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:12:57.504 INFO:teuthology.orchestra.run.smithi148.stdout: altname enp7s0f0 2023-10-30T13:12:57.504 INFO:teuthology.orchestra.run.smithi148.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T13:12:57.505 INFO:teuthology.orchestra.run.smithi148.stdout: link/ether 0c:c4:7a:da:6a:a5 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:12:57.505 INFO:teuthology.orchestra.run.smithi148.stdout: altname enp7s0f1 2023-10-30T13:12:57.505 INFO:teuthology.orchestra.run.smithi148.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T13:12:57.505 INFO:teuthology.orchestra.run.smithi148.stdout: link/ether 0c:c4:7a:88:72:56 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:12:57.505 INFO:teuthology.orchestra.run.smithi148.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-30T13:12:57.506 INFO:teuthology.orchestra.run.smithi148.stdout: link/ether 0c:c4:7a:88:72:57 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:12:57.506 INFO:teuthology.orchestra.run.smithi148.stdout: inet 172.21.15.148/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-30T13:12:57.506 INFO:teuthology.orchestra.run.smithi148.stdout: valid_lft 42252sec preferred_lft 42252sec 2023-10-30T13:12:57.506 INFO:teuthology.orchestra.run.smithi148.stdout: inet6 fe80::ec4:7aff:fe88:7257/64 scope link 2023-10-30T13:12:57.506 INFO:teuthology.orchestra.run.smithi148.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:12:57.506 INFO:teuthology.orchestra.run.smithi148.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-30T13:12:57.507 INFO:teuthology.orchestra.run.smithi148.stdout: link/ether 02:42:73:6c:d0:23 brd ff:ff:ff:ff:ff:ff 2023-10-30T13:12:57.507 INFO:teuthology.orchestra.run.smithi148.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-30T13:12:57.507 INFO:teuthology.orchestra.run.smithi148.stdout: valid_lft forever preferred_lft forever 2023-10-30T13:12:57.508 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-30T13:12:57.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.509 DEBUG:teuthology.orchestra.run.smithi148:> (cd / && exec bash -c ' 2023-10-30T13:12:57.509 DEBUG:teuthology.orchestra.run.smithi148:> set -e 2023-10-30T13:12:57.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip link add name ceph-brx type bridge 2023-10-30T13:12:57.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip addr flush dev ceph-brx 2023-10-30T13:12:57.510 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip link set ceph-brx up 2023-10-30T13:12:57.510 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-30T13:12:57.510 DEBUG:teuthology.orchestra.run.smithi148:> ') 2023-10-30T13:12:57.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.621 DEBUG:teuthology.orchestra.run.smithi148:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-30T13:12:57.640 INFO:teuthology.orchestra.run.smithi148.stdout:1 2023-10-30T13:12:57.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.643 DEBUG:teuthology.orchestra.run.smithi148:> route 2023-10-30T13:12:57.717 INFO:teuthology.orchestra.run.smithi148.stdout:Kernel IP routing table 2023-10-30T13:12:57.717 INFO:teuthology.orchestra.run.smithi148.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-30T13:12:57.717 INFO:teuthology.orchestra.run.smithi148.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-30T13:12:57.717 INFO:teuthology.orchestra.run.smithi148.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-30T13:12:57.718 INFO:teuthology.orchestra.run.smithi148.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-30T13:12:57.718 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-30T13:12:57.718 INFO:teuthology.orchestra.run.smithi148.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-30T13:12:57.718 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-30T13:12:57.719 INFO:teuthology.orchestra.run.smithi148.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-30T13:12:57.719 INFO:teuthology.orchestra.run.smithi148.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-30T13:12:57.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.720 DEBUG:teuthology.orchestra.run.smithi148:> (cd / && exec bash -c ' 2023-10-30T13:12:57.720 DEBUG:teuthology.orchestra.run.smithi148:> set -e 2023-10-30T13:12:57.720 DEBUG:teuthology.orchestra.run.smithi148:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-30T13:12:57.720 DEBUG:teuthology.orchestra.run.smithi148:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-30T13:12:57.721 DEBUG:teuthology.orchestra.run.smithi148:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-30T13:12:57.721 DEBUG:teuthology.orchestra.run.smithi148:> ') 2023-10-30T13:12:57.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.809 DEBUG:teuthology.orchestra.run.smithi148:> ip netns list 2023-10-30T13:12:57.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.819 DEBUG:teuthology.orchestra.run.smithi148:> ip netns list-id 2023-10-30T13:12:57.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.871 DEBUG:teuthology.orchestra.run.smithi148:> (cd / && exec bash -c ' 2023-10-30T13:12:57.871 DEBUG:teuthology.orchestra.run.smithi148:> set -e 2023-10-30T13:12:57.872 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T13:12:57.872 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-30T13:12:57.872 DEBUG:teuthology.orchestra.run.smithi148:> ') 2023-10-30T13:12:57.953 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-30T13:12:57.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:57.953 DEBUG:teuthology.orchestra.run.smithi148:> (cd / && exec bash -c ' 2023-10-30T13:12:57.954 DEBUG:teuthology.orchestra.run.smithi148:> set -e 2023-10-30T13:12:57.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-30T13:12:57.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-30T13:12:57.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-30T13:12:57.955 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-30T13:12:57.955 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-30T13:12:57.955 DEBUG:teuthology.orchestra.run.smithi148:> ') 2023-10-30T13:12:58.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:58.117 DEBUG:teuthology.orchestra.run.smithi148:> (cd / && exec bash -c ' 2023-10-30T13:12:58.118 DEBUG:teuthology.orchestra.run.smithi148:> set -e 2023-10-30T13:12:58.118 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip link set brx.0 up 2023-10-30T13:12:58.118 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip link set dev brx.0 master ceph-brx 2023-10-30T13:12:58.119 DEBUG:teuthology.orchestra.run.smithi148:> ') 2023-10-30T13:12:58.163 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T13:12:58.163 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-30T13:12:58.219 INFO:teuthology.orchestra.run.smithi148.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-30T13:12:58.220 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T13:12:58.220 DEBUG:teuthology.orchestra.run.smithi148:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-30T13:12:58.273 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-30T13:12:58.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:58.274 DEBUG:teuthology.orchestra.run.smithi148:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:12:58.387 INFO:teuthology.orchestra.run.smithi148.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:12:58.388 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T13:12:58.388 INFO:teuthology.orchestra.run.smithi148.stdout:invalid new device string format 2023-10-30T13:12:58.504 INFO:teuthology.orchestra.run.smithi148.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:12:58.504 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T13:12:58.505 INFO:teuthology.orchestra.run.smithi148.stdout:invalid new device string format 2023-10-30T13:12:58.505 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: resolved to: "172.21.15.148:3300,172.21.15.150:3300,172.21.15.150:3301" 2023-10-30T13:12:58.505 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T13:12:58.518 INFO:teuthology.orchestra.run.smithi148.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T13:12:58.518 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T13:12:58.518 INFO:teuthology.orchestra.run.smithi148.stdout:invalid new device string format 2023-10-30T13:12:58.519 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: resolved to: "172.21.15.148:3300,172.21.15.150:3300,172.21.15.150:3301" 2023-10-30T13:12:58.519 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T13:12:58.519 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: resolved to: "172.21.15.148:6789,172.21.15.150:6789,172.21.15.150:6790" 2023-10-30T13:12:58.519 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.148:6789,172.21.15.150:6789,172.21.15.150:6790:/ 2023-10-30T13:12:58.519 INFO:teuthology.orchestra.run.smithi148.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=23a3d51e-f2ab-4ffa-93fe-42135bef502c" will pass to kernel 2023-10-30T13:12:58.522 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-30T13:12:58.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:58.523 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-30T13:12:58.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:58.550 DEBUG:teuthology.orchestra.run.smithi148:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T13:12:58.625 INFO:teuthology.orchestra.run.smithi148.stdout:client4738 2023-10-30T13:12:58.626 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:12:58.626 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T13:12:58.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:58.675 DEBUG:teuthology.orchestra.run.smithi148:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T13:12:58.734 INFO:teuthology.orchestra.run.smithi148.stdout:client4738 2023-10-30T13:12:58.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:58.735 DEBUG:teuthology.orchestra.run.smithi148:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T13:12:58.798 INFO:teuthology.orchestra.run.smithi148.stdout:client4738 2023-10-30T13:12:58.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:12:58.799 DEBUG:teuthology.orchestra.run.smithi148:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/23a3d51e-f2ab-4ffa-93fe-42135bef502c.client4738/status') 2023-10-30T13:12:59.125 INFO:teuthology.orchestra.run.smithi148.stdout:instance: client.4738 (3)172.21.15.148:0/365720273 2023-10-30T13:12:59.125 INFO:teuthology.orchestra.run.smithi148.stdout:blocklisted: false 2023-10-30T13:12:59.125 INFO:teuthology.orchestra.run.smithi148.stderr:0+1 records in 2023-10-30T13:12:59.126 INFO:teuthology.orchestra.run.smithi148.stderr:0+1 records out 2023-10-30T13:12:59.126 INFO:teuthology.orchestra.run.smithi148.stderr:70 bytes copied, 5.2481e-05 s, 1.3 MB/s 2023-10-30T13:12:59.127 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-10-30T13:12:59.146 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:12:59.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82e2dab640 1 Processor -- start 2023-10-30T13:12:59.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82e2dab640 1 -- start start 2023-10-30T13:12:59.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82e2dab640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc1538a0 0x7f82dc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82e2dab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82dc0ca0e0 0x7f82dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82e2dab640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82dc0cb800 con 0x7f82dc0cf160 2023-10-30T13:12:59.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82e2dab640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82dc15dc30 con 0x7f82dc0ca0e0 2023-10-30T13:12:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82e2dab640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc0cb330 con 0x7f82dc1538a0 2023-10-30T13:12:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82dbfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82dc0ca0e0 0x7f82dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82dbfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82dc0ca0e0 0x7f82dc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42614/0 (socket says 172.21.15.148:42614) 2023-10-30T13:12:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82e1348640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f82dc0cf160 0x7f82dc0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44868/0 (socket says 172.21.15.148:44868) 2023-10-30T13:12:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82dbfff640 1 -- 172.21.15.148:0/3937358461 learned_addr learned my addr 172.21.15.148:0/3937358461 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:59.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82e0b47640 1 --2- 172.21.15.148:0/3937358461 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc1538a0 0x7f82dc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82dbfff640 1 -- 172.21.15.148:0/3937358461 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc1538a0 msgr2=0x7f82dc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82dbfff640 1 --2- 172.21.15.148:0/3937358461 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc1538a0 0x7f82dc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82db7fe640 1 -- 172.21.15.148:0/3937358461 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713908382 0 0) 0x7f82dc0cb800 con 0x7f82dc0cf160 2023-10-30T13:12:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82dbfff640 1 -- 172.21.15.148:0/3937358461 >> v1:172.21.15.148:6789/0 conn(0x7f82dc0cf160 legacy=0x7f82dc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.281+0000 7f82dbfff640 1 -- 172.21.15.148:0/3937358461 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc0cad70 con 0x7f82dc0ca0e0 2023-10-30T13:12:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82dbfff640 1 --2- 172.21.15.148:0/3937358461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0ca0e0 0x7f82dc15a090 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009870 tx=0x7f82c801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fee09e7c9eb5a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82db7fe640 1 -- 172.21.15.148:0/3937358461 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80a8a30 con 0x7f82dc0ca0e0 2023-10-30T13:12:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82db7fe640 1 -- 172.21.15.148:0/3937358461 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c80a8b90 con 0x7f82dc0ca0e0 2023-10-30T13:12:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82db7fe640 1 -- 172.21.15.148:0/3937358461 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80b0440 con 0x7f82dc0ca0e0 2023-10-30T13:12:59.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82e2dab640 1 -- 172.21.15.148:0/3937358461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0ca0e0 msgr2=0x7f82dc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82e2dab640 1 --2- 172.21.15.148:0/3937358461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0ca0e0 0x7f82dc15a090 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009870 tx=0x7f82c801ed30 comp rx=0 tx=0).stop 2023-10-30T13:12:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82e2dab640 1 -- 172.21.15.148:0/3937358461 shutdown_connections 2023-10-30T13:12:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82e2dab640 1 --2- 172.21.15.148:0/3937358461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0ca0e0 0x7f82dc15a090 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82e2dab640 1 --2- 172.21.15.148:0/3937358461 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc1538a0 0x7f82dc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82e2dab640 1 -- 172.21.15.148:0/3937358461 >> 172.21.15.148:0/3937358461 conn(0x7f82dc06bfe0 msgr2=0x7f82dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:59.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82e2dab640 1 -- 172.21.15.148:0/3937358461 shutdown_connections 2023-10-30T13:12:59.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82e2dab640 1 -- 172.21.15.148:0/3937358461 wait complete. 2023-10-30T13:12:59.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.285+0000 7f82e2dab640 1 Processor -- start 2023-10-30T13:12:59.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 -- start start 2023-10-30T13:12:59.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc0ca0e0 0x7f82dc0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0cf160 0x7f82dc144550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82dc1538a0 0x7f82dc144a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc145b30 con 0x7f82dc0cf160 2023-10-30T13:12:59.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1f0380 con 0x7f82dc1538a0 2023-10-30T13:12:59.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc1ef490 con 0x7f82dc0ca0e0 2023-10-30T13:12:59.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0cf160 0x7f82dc144550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82dbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0cf160 0x7f82dc144550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42624/0 (socket says 172.21.15.148:42624) 2023-10-30T13:12:59.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82dbfff640 1 -- 172.21.15.148:0/546382557 learned_addr learned my addr 172.21.15.148:0/546382557 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:59.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e1348640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82dc1538a0 0x7f82dc144a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e0b47640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc0ca0e0 0x7f82dc0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82dbfff640 1 -- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc0ca0e0 msgr2=0x7f82dc0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82dbfff640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc0ca0e0 0x7f82dc0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82dbfff640 1 -- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82dc1538a0 msgr2=0x7f82dc144a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82dbfff640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82dc1538a0 0x7f82dc144a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82dbfff640 1 -- 172.21.15.148:0/546382557 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80094d0 con 0x7f82dc0cf160 2023-10-30T13:12:59.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e0b47640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc0ca0e0 0x7f82dc0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:12:59.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82dbfff640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0cf160 0x7f82dc144550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f82c80b3040 tx=0x7f82c801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:59.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82d97fa640 1 -- 172.21.15.148:0/546382557 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80b0dd0 con 0x7f82dc0cf160 2023-10-30T13:12:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82d97fa640 1 -- 172.21.15.148:0/546382557 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c80bb460 con 0x7f82dc0cf160 2023-10-30T13:12:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82dc1ef770 con 0x7f82dc0cf160 2023-10-30T13:12:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82d97fa640 1 -- 172.21.15.148:0/546382557 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80bb810 con 0x7f82dc0cf160 2023-10-30T13:12:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82dc1efd30 con 0x7f82dc0cf160 2023-10-30T13:12:59.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.289+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82dc0b2a50 con 0x7f82dc0cf160 2023-10-30T13:12:59.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.293+0000 7f82d97fa640 1 -- 172.21.15.148:0/546382557 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82c80b83f0 con 0x7f82dc0cf160 2023-10-30T13:12:59.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.293+0000 7f82d97fa640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82c00664a0 0x7f82c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.293+0000 7f82d97fa640 1 -- 172.21.15.148:0/546382557 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f82c81218c0 con 0x7f82dc0cf160 2023-10-30T13:12:59.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.293+0000 7f82e0b47640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82c00664a0 0x7f82c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.293+0000 7f82e0b47640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82c00664a0 0x7f82c0068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009a40 tx=0x7f82d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:59.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.297+0000 7f82d97fa640 1 -- 172.21.15.148:0/546382557 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82c80eece0 con 0x7f82dc0cf160 2023-10-30T13:12:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.449+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82dc04e170 con 0x7f82dc0cf160 2023-10-30T13:12:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.449+0000 7f82d97fa640 1 -- 172.21.15.148:0/546382557 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16782 (secure 0 0 0) 0x7f82c80c0090 con 0x7f82dc0cf160 2023-10-30T13:12:59.453 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:12:59.453 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":22,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:11:15.424384+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:12:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82c00664a0 msgr2=0x7f82c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82c00664a0 0x7f82c0068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009a40 tx=0x7f82d0008040 comp rx=0 tx=0).stop 2023-10-30T13:12:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0cf160 msgr2=0x7f82dc144550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0cf160 0x7f82dc144550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f82c80b3040 tx=0x7f82c801bad0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 shutdown_connections 2023-10-30T13:12:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82c00664a0 0x7f82c0068960 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82dc0cf160 0x7f82dc144550 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82dc1538a0 0x7f82dc144a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 --2- 172.21.15.148:0/546382557 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82dc0ca0e0 0x7f82dc0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 >> 172.21.15.148:0/546382557 conn(0x7f82dc06bfe0 msgr2=0x7f82dc06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 shutdown_connections 2023-10-30T13:12:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.453+0000 7f82e2dab640 1 -- 172.21.15.148:0/546382557 wait complete. 2023-10-30T13:12:59.474 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:10:22.349322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:10:32.611151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:10:33.044931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:12:59.475 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:12:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.613+0000 7f82c2ba7640 1 Processor -- start 2023-10-30T13:12:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c2ba7640 1 -- start start 2023-10-30T13:12:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c2ba7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc153840 0x7f82bc15a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c2ba7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f82bc0cb4c0 0x7f82bc0cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c2ba7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc15a6b0 con 0x7f82bc153c20 2023-10-30T13:12:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c2ba7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82bc15dbc0 con 0x7f82bc0cb4c0 2023-10-30T13:12:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c2ba7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc15d730 con 0x7f82bc153840 2023-10-30T13:12:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c13a4640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f82bc153c20 0x7f82bc0c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44874/0 (socket says 172.21.15.148:44874) 2023-10-30T13:12:59.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c13a4640 1 -- 172.21.15.148:0/4245347659 learned_addr learned my addr 172.21.15.148:0/4245347659 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:12:59.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c23a6640 1 --2- 172.21.15.148:0/4245347659 >> v2:172.21.15.150:3300/0 conn(0x7f82bc0cb4c0 0x7f82bc0cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c1ba5640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc153840 0x7f82bc15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082548850 0 0) 0x7f82ac001120 con 0x7f82bc153c20 2023-10-30T13:12:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a8003530 con 0x7f82bc153c20 2023-10-30T13:12:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1815642904 0 0) 0x7f82a8003530 con 0x7f82bc153c20 2023-10-30T13:12:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc153840 msgr2=0x7f82bc15a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc153840 0x7f82bc15a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 >> v2:172.21.15.150:3300/0 conn(0x7f82bc0cb4c0 msgr2=0x7f82bc0cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 --2- 172.21.15.148:0/4245347659 >> v2:172.21.15.150:3300/0 conn(0x7f82bc0cb4c0 0x7f82bc0cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0ca8f0 con 0x7f82bc153c20 2023-10-30T13:12:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f82ac001810 con 0x7f82bc153c20 2023-10-30T13:12:59.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c23a6640 1 --2- 172.21.15.148:0/4245347659 >> v2:172.21.15.150:3300/0 conn(0x7f82bc0cb4c0 0x7f82bc0cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:12:59.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 >> v1:172.21.15.148:6789/0 conn(0x7f82bc153c20 legacy=0x7f82bc0c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82a8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc153840 0x7f82a80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a8004d10 0x7f82a8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc15dbc0 con 0x7f82a8003980 2023-10-30T13:12:59.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc15d730 con 0x7f82a8004d10 2023-10-30T13:12:59.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a800b990 con 0x7f82bc153840 2023-10-30T13:12:59.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c1ba5640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82a8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c13a4640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc153840 0x7f82a80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c23a6640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a8004d10 0x7f82a8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c1ba5640 1 -- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc153840 msgr2=0x7f82a80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:59.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c1ba5640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc153840 0x7f82a80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c1ba5640 1 -- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a8004d10 msgr2=0x7f82a8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:59.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c1ba5640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a8004d10 0x7f82a8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.617+0000 7f82c1ba5640 1 -- 172.21.15.148:0/4245347659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a8003530 con 0x7f82a8003980 2023-10-30T13:12:59.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c1ba5640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82a8003da0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009ae0 tx=0x7f82b801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee55cdf6cd5f7b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:59.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b80a5640 con 0x7f82a8003980 2023-10-30T13:12:59.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b80a57a0 con 0x7f82a8003980 2023-10-30T13:12:59.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c0ba3640 1 -- 172.21.15.148:0/4245347659 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b80a5b50 con 0x7f82a8003980 2023-10-30T13:12:59.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c2ba7640 1 -- 172.21.15.148:0/4245347659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 msgr2=0x7f82a8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c2ba7640 1 --2- 172.21.15.148:0/4245347659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82a8003da0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009ae0 tx=0x7f82b801ca50 comp rx=0 tx=0).stop 2023-10-30T13:12:59.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c23a6640 1 -- 172.21.15.148:0/4245347659 reap_dead start 2023-10-30T13:12:59.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c2ba7640 1 -- 172.21.15.148:0/4245347659 shutdown_connections 2023-10-30T13:12:59.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c2ba7640 1 -- 172.21.15.148:0/4245347659 >> 172.21.15.148:0/4245347659 conn(0x7f82bc06be40 msgr2=0x7f82bc06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:59.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c2ba7640 1 -- 172.21.15.148:0/4245347659 shutdown_connections 2023-10-30T13:12:59.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.621+0000 7f82c2ba7640 1 -- 172.21.15.148:0/4245347659 wait complete. 2023-10-30T13:12:59.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 Processor -- start 2023-10-30T13:12:59.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 -- start start 2023-10-30T13:12:59.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82bc156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc0b3ef0 0x7f82bc156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc158810 0x7f82bc158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc155330 con 0x7f82a8003980 2023-10-30T13:12:59.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc15c360 con 0x7f82bc0b3ef0 2023-10-30T13:12:59.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc1f2290 con 0x7f82bc158810 2023-10-30T13:12:59.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c1ba5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82bc156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c1ba5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82bc156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42642/0 (socket says 172.21.15.148:42642) 2023-10-30T13:12:59.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c1ba5640 1 -- 172.21.15.148:0/3048112095 learned_addr learned my addr 172.21.15.148:0/3048112095 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:59.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c13a4640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc0b3ef0 0x7f82bc156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c23a6640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc158810 0x7f82bc158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c1ba5640 1 -- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc158810 msgr2=0x7f82bc158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c1ba5640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc158810 0x7f82bc158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c1ba5640 1 -- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc0b3ef0 msgr2=0x7f82bc156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c1ba5640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc0b3ef0 0x7f82bc156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c1ba5640 1 -- 172.21.15.148:0/3048112095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc1f2570 con 0x7f82a8003980 2023-10-30T13:12:59.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c1ba5640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82bc156850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009ae0 tx=0x7f82a8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:59.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82b27fc640 1 -- 172.21.15.148:0/3048112095 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b8004230 con 0x7f82a8003980 2023-10-30T13:12:59.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82b27fc640 1 -- 172.21.15.148:0/3048112095 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b8004390 con 0x7f82a8003980 2023-10-30T13:12:59.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82b27fc640 1 -- 172.21.15.148:0/3048112095 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b80ad850 con 0x7f82a8003980 2023-10-30T13:12:59.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc1f2850 con 0x7f82a8003980 2023-10-30T13:12:59.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc0ce520 con 0x7f82a8003980 2023-10-30T13:12:59.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.625+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc04e9c0 con 0x7f82a8003980 2023-10-30T13:12:59.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.629+0000 7f82b27fc640 1 -- 172.21.15.148:0/3048112095 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82b80b4040 con 0x7f82a8003980 2023-10-30T13:12:59.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.629+0000 7f82b27fc640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82a00664d0 0x7f82a0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.629+0000 7f82b27fc640 1 -- 172.21.15.148:0/3048112095 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f82b811da40 con 0x7f82a8003980 2023-10-30T13:12:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.629+0000 7f82c13a4640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82a00664d0 0x7f82a0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.629+0000 7f82c13a4640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82a00664d0 0x7f82a0068990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f82bc1efd20 tx=0x7f82ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.633+0000 7f82b27fc640 1 -- 172.21.15.148:0/3048112095 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82b80eae40 con 0x7f82a8003980 2023-10-30T13:12:59.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.785+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f82bc0cb000 con 0x7f82a8003980 2023-10-30T13:12:59.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.785+0000 7f82b27fc640 1 -- 172.21.15.148:0/3048112095 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f82b80c00a0 con 0x7f82a8003980 2023-10-30T13:12:59.790 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2023-10-30T13:12:59.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82a00664d0 msgr2=0x7f82a0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82a00664d0 0x7f82a0068990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f82bc1efd20 tx=0x7f82ac009040 comp rx=0 tx=0).stop 2023-10-30T13:12:59.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 msgr2=0x7f82bc156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82bc156850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009ae0 tx=0x7f82a8005800 comp rx=0 tx=0).stop 2023-10-30T13:12:59.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 shutdown_connections 2023-10-30T13:12:59.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82a00664d0 0x7f82a0068990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82bc158810 0x7f82bc158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc0b3ef0 0x7f82bc156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 --2- 172.21.15.148:0/3048112095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82a8003980 0x7f82bc156850 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 >> 172.21.15.148:0/3048112095 conn(0x7f82bc06be40 msgr2=0x7f82bc14d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:59.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 shutdown_connections 2023-10-30T13:12:59.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.789+0000 7f82c2ba7640 1 -- 172.21.15.148:0/3048112095 wait complete. 2023-10-30T13:12:59.811 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:12:59.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa353dd3640 1 Processor -- start 2023-10-30T13:12:59.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa353dd3640 1 -- start start 2023-10-30T13:12:59.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa353dd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa34c0722a0 0x7fa34c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa353dd3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c0c0a30 con 0x7fa34c0c0400 2023-10-30T13:12:59.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa353dd3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c06f650 con 0x7fa34c1508c0 2023-10-30T13:12:59.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa353dd3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34c0c85b0 con 0x7fa34c0722a0 2023-10-30T13:12:59.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa352370640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa34c0c0400 0x7fa34c156930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44880/0 (socket says 172.21.15.148:44880) 2023-10-30T13:12:59.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa352370640 1 -- 172.21.15.148:0/1911798464 learned_addr learned my addr 172.21.15.148:0/1911798464 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:12:59.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa35136e640 1 --2- 172.21.15.148:0/1911798464 >> v2:172.21.15.150:3300/0 conn(0x7fa34c0722a0 0x7fa34c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa350b6d640 1 -- 172.21.15.148:0/1911798464 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168171025 0 0) 0x7fa34c0c0a30 con 0x7fa34c0c0400 2023-10-30T13:12:59.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa350b6d640 1 -- 172.21.15.148:0/1911798464 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa338003530 con 0x7fa34c0c0400 2023-10-30T13:12:59.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa350b6d640 1 -- 172.21.15.148:0/1911798464 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 328524665 0 0) 0x7fa338003530 con 0x7fa34c0c0400 2023-10-30T13:12:59.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa35136e640 1 -- 172.21.15.148:0/1911798464 >> v1:172.21.15.150:6789/0 conn(0x7fa34c1508c0 legacy=0x7fa34c0c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa35136e640 1 -- 172.21.15.148:0/1911798464 >> v1:172.21.15.148:6789/0 conn(0x7fa34c0c0400 legacy=0x7fa34c156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.949+0000 7fa35136e640 1 -- 172.21.15.148:0/1911798464 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c06f930 con 0x7fa34c0722a0 2023-10-30T13:12:59.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa35136e640 1 --2- 172.21.15.148:0/1911798464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa34c0722a0 0x7fa34c0bfec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0097c0 tx=0x7fa33c01c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0f8248f2272810d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:59.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa350b6d640 1 -- 172.21.15.148:0/1911798464 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344001a00 con 0x7fa34c0722a0 2023-10-30T13:12:59.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa350b6d640 1 -- 172.21.15.148:0/1911798464 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c0a7050 con 0x7fa34c0722a0 2023-10-30T13:12:59.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa350b6d640 1 -- 172.21.15.148:0/1911798464 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0ab620 con 0x7fa34c0722a0 2023-10-30T13:12:59.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa353dd3640 1 -- 172.21.15.148:0/1911798464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa34c0722a0 msgr2=0x7fa34c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa353dd3640 1 --2- 172.21.15.148:0/1911798464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa34c0722a0 0x7fa34c0bfec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0097c0 tx=0x7fa33c01c310 comp rx=0 tx=0).stop 2023-10-30T13:12:59.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa353dd3640 1 -- 172.21.15.148:0/1911798464 shutdown_connections 2023-10-30T13:12:59.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa353dd3640 1 --2- 172.21.15.148:0/1911798464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa34c0722a0 0x7fa34c0bfec0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa353dd3640 1 -- 172.21.15.148:0/1911798464 >> 172.21.15.148:0/1911798464 conn(0x7fa34c0c7910 msgr2=0x7fa34c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:59.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa353dd3640 1 -- 172.21.15.148:0/1911798464 shutdown_connections 2023-10-30T13:12:59.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.953+0000 7fa353dd3640 1 -- 172.21.15.148:0/1911798464 wait complete. 2023-10-30T13:12:59.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 Processor -- start 2023-10-30T13:12:59.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 -- start start 2023-10-30T13:12:59.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa34c0722a0 0x7fa34c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa34c0c0400 0x7fa34c14e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa34c1508c0 0x7fa34c14e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0c0850 con 0x7fa34c0c0400 2023-10-30T13:12:59.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c1f2c00 con 0x7fa34c0722a0 2023-10-30T13:12:59.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0ca7b0 con 0x7fa34c1508c0 2023-10-30T13:12:59.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa35136e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa34c0c0400 0x7fa34c14e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa35136e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa34c0c0400 0x7fa34c14e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42654/0 (socket says 172.21.15.148:42654) 2023-10-30T13:12:59.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa35136e640 1 -- 172.21.15.148:0/2337694823 learned_addr learned my addr 172.21.15.148:0/2337694823 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:12:59.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa351b6f640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa34c0722a0 0x7fa34c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa352370640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa34c1508c0 0x7fa34c14e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa35136e640 1 -- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa34c1508c0 msgr2=0x7fa34c14e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa35136e640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa34c1508c0 0x7fa34c14e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa35136e640 1 -- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa34c0722a0 msgr2=0x7fa34c0c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:59.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa35136e640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa34c0722a0 0x7fa34c0c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:59.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa35136e640 1 -- 172.21.15.148:0/2337694823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c009420 con 0x7fa34c0c0400 2023-10-30T13:12:59.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa352370640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa34c1508c0 0x7fa34c14e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:59.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa35136e640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa34c0c0400 0x7fa34c14e190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0097c0 tx=0x7fa33c0190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:59.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa3427fc640 1 -- 172.21.15.148:0/2337694823 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0bb070 con 0x7fa34c0c0400 2023-10-30T13:12:59.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa3427fc640 1 -- 172.21.15.148:0/2337694823 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c0abda0 con 0x7fa34c0c0400 2023-10-30T13:12:59.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c1f1ff0 con 0x7fa34c0c0400 2023-10-30T13:12:59.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa3427fc640 1 -- 172.21.15.148:0/2337694823 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c004220 con 0x7fa34c0c0400 2023-10-30T13:12:59.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.957+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c1f25b0 con 0x7fa34c0c0400 2023-10-30T13:12:59.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.961+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c1f2db0 con 0x7fa34c0c0400 2023-10-30T13:12:59.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.961+0000 7fa3427fc640 1 -- 172.21.15.148:0/2337694823 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa33c0b2040 con 0x7fa34c0c0400 2023-10-30T13:12:59.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.961+0000 7fa3427fc640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3300664a0 0x7fa330068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:59.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.961+0000 7fa3427fc640 1 -- 172.21.15.148:0/2337694823 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fa33c11d690 con 0x7fa34c0c0400 2023-10-30T13:12:59.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.961+0000 7fa351b6f640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3300664a0 0x7fa330068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:59.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.961+0000 7fa351b6f640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3300664a0 0x7fa330068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa348000fc0 tx=0x7fa348009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:59.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:12:59.965+0000 7fa3427fc640 1 -- 172.21.15.148:0/2337694823 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa34c1f2db0 con 0x7fa34c0c0400 2023-10-30T13:13:00.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa34c1f2db0 con 0x7fa34c0c0400 2023-10-30T13:13:00.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa3427fc640 1 -- 172.21.15.148:0/2337694823 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa34c1f2db0 con 0x7fa34c0c0400 2023-10-30T13:13:00.120 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 32 2023-10-30T13:13:00.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3300664a0 msgr2=0x7fa330068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3300664a0 0x7fa330068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa348000fc0 tx=0x7fa348009040 comp rx=0 tx=0).stop 2023-10-30T13:13:00.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa34c0c0400 msgr2=0x7fa34c14e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa34c0c0400 0x7fa34c14e190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0097c0 tx=0x7fa33c0190b0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 shutdown_connections 2023-10-30T13:13:00.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3300664a0 0x7fa330068960 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa34c0c0400 0x7fa34c14e190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa34c1508c0 0x7fa34c14e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 --2- 172.21.15.148:0/2337694823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa34c0722a0 0x7fa34c0c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 >> 172.21.15.148:0/2337694823 conn(0x7fa34c0c7910 msgr2=0x7fa34c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:00.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 shutdown_connections 2023-10-30T13:13:00.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.117+0000 7fa353dd3640 1 -- 172.21.15.148:0/2337694823 wait complete. 2023-10-30T13:13:00.140 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:13:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.277+0000 7f9f97ec9640 1 Processor -- start 2023-10-30T13:13:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.277+0000 7f9f97ec9640 1 -- start start 2023-10-30T13:13:00.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f97ec9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 0x7f9f900caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f97ec9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9015a920 con 0x7f9f900cb1e0 2023-10-30T13:13:00.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f97ec9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f900cb930 con 0x7f9f90159930 2023-10-30T13:13:00.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f97ec9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f90159dc0 con 0x7f9f90153980 2023-10-30T13:13:00.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f96466640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9f900cb1e0 0x7f9f9015d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44886/0 (socket says 172.21.15.148:44886) 2023-10-30T13:13:00.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f96466640 1 -- 172.21.15.148:0/3890870687 learned_addr learned my addr 172.21.15.148:0/3890870687 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:00.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f95c65640 1 --2- 172.21.15.148:0/3890870687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 0x7f9f900caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f94c63640 1 -- 172.21.15.148:0/3890870687 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743522356 0 0) 0x7f9f9015a920 con 0x7f9f900cb1e0 2023-10-30T13:13:00.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f94c63640 1 -- 172.21.15.148:0/3890870687 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f7c003530 con 0x7f9f900cb1e0 2023-10-30T13:13:00.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f94c63640 1 -- 172.21.15.148:0/3890870687 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962868859 0 0) 0x7f9f900cb930 con 0x7f9f90159930 2023-10-30T13:13:00.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f94c63640 1 -- 172.21.15.148:0/3890870687 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f9015a920 con 0x7f9f90159930 2023-10-30T13:13:00.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f95c65640 1 -- 172.21.15.148:0/3890870687 >> v1:172.21.15.150:6789/0 conn(0x7f9f90159930 legacy=0x7f9f900cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f95c65640 1 -- 172.21.15.148:0/3890870687 >> v1:172.21.15.148:6789/0 conn(0x7f9f900cb1e0 legacy=0x7f9f9015d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f94c63640 1 -- 172.21.15.148:0/3890870687 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 739273671 0 0) 0x7f9f7c003530 con 0x7f9f900cb1e0 2023-10-30T13:13:00.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.281+0000 7f9f95c65640 1 -- 172.21.15.148:0/3890870687 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9015a080 con 0x7f9f90153980 2023-10-30T13:13:00.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f95c65640 1 --2- 172.21.15.148:0/3890870687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 0x7f9f900caca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0097f0 tx=0x7f9f8c01c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa731d51cd4b42bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f94c63640 1 -- 172.21.15.148:0/3890870687 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f88001a00 con 0x7f9f90153980 2023-10-30T13:13:00.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f94c63640 1 -- 172.21.15.148:0/3890870687 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c0a7050 con 0x7f9f90153980 2023-10-30T13:13:00.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f94c63640 1 -- 172.21.15.148:0/3890870687 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0ab600 con 0x7f9f90153980 2023-10-30T13:13:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3890870687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 msgr2=0x7f9f900caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 --2- 172.21.15.148:0/3890870687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 0x7f9f900caca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0097f0 tx=0x7f9f8c01c3b0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3890870687 shutdown_connections 2023-10-30T13:13:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 --2- 172.21.15.148:0/3890870687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 0x7f9f900caca0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3890870687 >> 172.21.15.148:0/3890870687 conn(0x7f9f9006c0f0 msgr2=0x7f9f9006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3890870687 shutdown_connections 2023-10-30T13:13:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3890870687 wait complete. 2023-10-30T13:13:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 Processor -- start 2023-10-30T13:13:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 -- start start 2023-10-30T13:13:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f900cb1e0 0x7f9f900c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 0x7f9f900c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f900c5200 0x7f9f90147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f90155300 con 0x7f9f900c5200 2023-10-30T13:13:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f900c7ad0 con 0x7f9f900cb1e0 2023-10-30T13:13:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.285+0000 7f9f97ec9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f900c84f0 con 0x7f9f90153980 2023-10-30T13:13:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f96466640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f900c5200 0x7f9f90147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f96466640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f900c5200 0x7f9f90147d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42668/0 (socket says 172.21.15.148:42668) 2023-10-30T13:13:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f95c65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f900cb1e0 0x7f9f900c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f96466640 1 -- 172.21.15.148:0/3823826619 learned_addr learned my addr 172.21.15.148:0/3823826619 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f95464640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 0x7f9f900c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f96466640 1 -- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 msgr2=0x7f9f900c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f96466640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 0x7f9f900c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f96466640 1 -- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f900cb1e0 msgr2=0x7f9f900c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f96466640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f900cb1e0 0x7f9f900c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f96466640 1 -- 172.21.15.148:0/3823826619 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c009420 con 0x7f9f900c5200 2023-10-30T13:13:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f96466640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f900c5200 0x7f9f90147d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9f8800cc20 tx=0x7f9f8800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f867fc640 1 -- 172.21.15.148:0/3823826619 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8800ce30 con 0x7f9f900c5200 2023-10-30T13:13:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f867fc640 1 -- 172.21.15.148:0/3823826619 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f88012400 con 0x7f9f900c5200 2023-10-30T13:13:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f867fc640 1 -- 172.21.15.148:0/3823826619 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f880127b0 con 0x7f9f900c5200 2023-10-30T13:13:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f900c9470 con 0x7f9f900c5200 2023-10-30T13:13:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f900c68b0 con 0x7f9f900c5200 2023-10-30T13:13:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f90153d60 con 0x7f9f900c5200 2023-10-30T13:13:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.289+0000 7f9f867fc640 1 -- 172.21.15.148:0/3823826619 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f88012910 con 0x7f9f900c5200 2023-10-30T13:13:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.293+0000 7f9f867fc640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f740664d0 0x7f9f74068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.293+0000 7f9f867fc640 1 -- 172.21.15.148:0/3823826619 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9f880844e0 con 0x7f9f900c5200 2023-10-30T13:13:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.293+0000 7f9f95c65640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f740664d0 0x7f9f74068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.293+0000 7f9f95c65640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f740664d0 0x7f9f74068990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0ad000 tx=0x7f9f8c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.297+0000 7f9f867fc640 1 -- 172.21.15.148:0/3823826619 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f88051970 con 0x7f9f900c5200 2023-10-30T13:13:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.445+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9f900c7c90 con 0x7f9f900c5200 2023-10-30T13:13:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.445+0000 7f9f867fc640 1 -- 172.21.15.148:0/3823826619 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9f88054f90 con 0x7f9f900c5200 2023-10-30T13:13:00.451 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 64 2023-10-30T13:13:00.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f740664d0 msgr2=0x7f9f74068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f740664d0 0x7f9f74068990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0ad000 tx=0x7f9f8c0a6000 comp rx=0 tx=0).stop 2023-10-30T13:13:00.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f900c5200 msgr2=0x7f9f90147d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f900c5200 0x7f9f90147d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9f8800cc20 tx=0x7f9f8800a430 comp rx=0 tx=0).stop 2023-10-30T13:13:00.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 shutdown_connections 2023-10-30T13:13:00.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f740664d0 0x7f9f74068990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f900c5200 0x7f9f90147d40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f90153980 0x7f9f900c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 --2- 172.21.15.148:0/3823826619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f900cb1e0 0x7f9f900c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 >> 172.21.15.148:0/3823826619 conn(0x7f9f9006c0f0 msgr2=0x7f9f900b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:00.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 shutdown_connections 2023-10-30T13:13:00.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.449+0000 7f9f97ec9640 1 -- 172.21.15.148:0/3823826619 wait complete. 2023-10-30T13:13:00.474 INFO:tasks.mds_thrash:mds thrasher using random seed: 1698671580 2023-10-30T13:13:00.474 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:13:00.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.609+0000 7f7d977d9640 1 Processor -- start 2023-10-30T13:13:00.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.609+0000 7f7d977d9640 1 -- start start 2023-10-30T13:13:00.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.609+0000 7f7d977d9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7d900c93d0 0x7f7d9006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.609+0000 7f7d977d9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9006afa0 con 0x7f7d90153d70 2023-10-30T13:13:00.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.609+0000 7f7d977d9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d90154150 con 0x7f7d9006e470 2023-10-30T13:13:00.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.609+0000 7f7d977d9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9006d780 con 0x7f7d900c93d0 2023-10-30T13:13:00.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.609+0000 7f7d95575640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7d90153d70 0x7f7d900cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44900/0 (socket says 172.21.15.148:44900) 2023-10-30T13:13:00.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.609+0000 7f7d95575640 1 -- 172.21.15.148:0/274615256 learned_addr learned my addr 172.21.15.148:0/274615256 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:00.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.609+0000 7f7d95d76640 1 --2- 172.21.15.148:0/274615256 >> v2:172.21.15.150:3300/0 conn(0x7f7d900c93d0 0x7f7d9006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78175788 0 0) 0x7f7d9006afa0 con 0x7f7d90153d70 2023-10-30T13:13:00.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d7c003530 con 0x7f7d90153d70 2023-10-30T13:13:00.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 877748431 0 0) 0x7f7d7c003530 con 0x7f7d90153d70 2023-10-30T13:13:00.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 >> v2:172.21.15.150:3300/0 conn(0x7f7d900c93d0 msgr2=0x7f7d9006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:00.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 --2- 172.21.15.148:0/274615256 >> v2:172.21.15.150:3300/0 conn(0x7f7d900c93d0 0x7f7d9006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 >> v1:172.21.15.150:6789/0 conn(0x7f7d9006e470 legacy=0x7f7d900c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9006e030 con 0x7f7d90153d70 2023-10-30T13:13:00.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f7d8c001680 con 0x7f7d90153d70 2023-10-30T13:13:00.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d95d76640 1 --2- 172.21.15.148:0/274615256 >> v2:172.21.15.150:3300/0 conn(0x7f7d900c93d0 0x7f7d9006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:13:00.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 >> v1:172.21.15.148:6789/0 conn(0x7f7d90153d70 legacy=0x7f7d900cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c003980 0x7f7d7c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d7c0047d0 0x7f7d7c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d7c005190 0x7f7d7c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9006d780 con 0x7f7d7c0047d0 2023-10-30T13:13:00.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7c00c030 con 0x7f7d7c003980 2023-10-30T13:13:00.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7c00c1f0 con 0x7f7d7c005190 2023-10-30T13:13:00.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d94d74640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d7c0047d0 0x7f7d7c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d95575640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c003980 0x7f7d7c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d95d76640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d7c005190 0x7f7d7c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d94d74640 1 -- 172.21.15.148:0/274615256 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d7c005190 msgr2=0x7f7d7c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:00.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d94d74640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d7c005190 0x7f7d7c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d94d74640 1 -- 172.21.15.148:0/274615256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c003980 msgr2=0x7f7d7c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:00.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d94d74640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c003980 0x7f7d7c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d94d74640 1 -- 172.21.15.148:0/274615256 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c003530 con 0x7f7d7c0047d0 2023-10-30T13:13:00.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.613+0000 7f7d95d76640 1 -- 172.21.15.148:0/274615256 reap_dead start 2023-10-30T13:13:00.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d94d74640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d7c0047d0 0x7f7d7c004c50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7d80001780 tx=0x7f7d8001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abacfc6f2549a3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d800052d0 con 0x7f7d7c0047d0 2023-10-30T13:13:00.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d800a7050 con 0x7f7d7c0047d0 2023-10-30T13:13:00.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d87fff640 1 -- 172.21.15.148:0/274615256 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80003600 con 0x7f7d7c0047d0 2023-10-30T13:13:00.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 -- 172.21.15.148:0/274615256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d7c0047d0 msgr2=0x7f7d7c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d7c0047d0 0x7f7d7c004c50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7d80001780 tx=0x7f7d8001c3e0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 -- 172.21.15.148:0/274615256 shutdown_connections 2023-10-30T13:13:00.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 --2- 172.21.15.148:0/274615256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d7c0047d0 0x7f7d7c004c50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 -- 172.21.15.148:0/274615256 >> 172.21.15.148:0/274615256 conn(0x7f7d90071ea0 msgr2=0x7f7d90065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:00.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 -- 172.21.15.148:0/274615256 shutdown_connections 2023-10-30T13:13:00.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 -- 172.21.15.148:0/274615256 wait complete. 2023-10-30T13:13:00.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 Processor -- start 2023-10-30T13:13:00.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 -- start start 2023-10-30T13:13:00.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c0047d0 0x7f7d90156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d9014fbb0 0x7f7d90150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d901f2a80 0x7f7d901f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d901f0530 con 0x7f7d9014fbb0 2023-10-30T13:13:00.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d901e6de0 con 0x7f7d7c0047d0 2023-10-30T13:13:00.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.617+0000 7f7d977d9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90150550 con 0x7f7d901f2a80 2023-10-30T13:13:00.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d94d74640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d9014fbb0 0x7f7d90150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d94d74640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d9014fbb0 0x7f7d90150010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42690/0 (socket says 172.21.15.148:42690) 2023-10-30T13:13:00.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d94d74640 1 -- 172.21.15.148:0/193144214 learned_addr learned my addr 172.21.15.148:0/193144214 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:00.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d95575640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c0047d0 0x7f7d90156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d95d76640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d901f2a80 0x7f7d901f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d94d74640 1 -- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d901f2a80 msgr2=0x7f7d901f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d94d74640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d901f2a80 0x7f7d901f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d94d74640 1 -- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c0047d0 msgr2=0x7f7d90156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d94d74640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c0047d0 0x7f7d90156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d94d74640 1 -- 172.21.15.148:0/193144214 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d901eff90 con 0x7f7d9014fbb0 2023-10-30T13:13:00.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d95575640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c0047d0 0x7f7d90156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:00.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d94d74640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d9014fbb0 0x7f7d90150010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d8001c210 tx=0x7f7d7c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d85ffb640 1 -- 172.21.15.148:0/193144214 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80007500 con 0x7f7d9014fbb0 2023-10-30T13:13:00.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d85ffb640 1 -- 172.21.15.148:0/193144214 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d80007660 con 0x7f7d9014fbb0 2023-10-30T13:13:00.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d85ffb640 1 -- 172.21.15.148:0/193144214 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80006a00 con 0x7f7d9014fbb0 2023-10-30T13:13:00.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d901f0250 con 0x7f7d9014fbb0 2023-10-30T13:13:00.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9015af30 con 0x7f7d9014fbb0 2023-10-30T13:13:00.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d90073480 con 0x7f7d9014fbb0 2023-10-30T13:13:00.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.621+0000 7f7d85ffb640 1 -- 172.21.15.148:0/193144214 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d80006b60 con 0x7f7d9014fbb0 2023-10-30T13:13:00.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.625+0000 7f7d85ffb640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d740664a0 0x7f7d74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.625+0000 7f7d85ffb640 1 -- 172.21.15.148:0/193144214 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7d8011e220 con 0x7f7d9014fbb0 2023-10-30T13:13:00.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.625+0000 7f7d95575640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d740664a0 0x7f7d74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.625+0000 7f7d95575640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d740664a0 0x7f7d74068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c007cc0 tx=0x7f7d8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.629+0000 7f7d85ffb640 1 -- 172.21.15.148:0/193144214 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d800eb570 con 0x7f7d9014fbb0 2023-10-30T13:13:00.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.777+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d9015b120 con 0x7f7d9014fbb0 2023-10-30T13:13:00.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d85ffb640 1 -- 172.21.15.148:0/193144214 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16782 (secure 0 0 0) 0x7f7d800eeb90 con 0x7f7d9014fbb0 2023-10-30T13:13:00.783 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:13:00.784 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":22,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:11:15.424384+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:13:00.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d740664a0 msgr2=0x7f7d74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d740664a0 0x7f7d74068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c007cc0 tx=0x7f7d8c009040 comp rx=0 tx=0).stop 2023-10-30T13:13:00.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d9014fbb0 msgr2=0x7f7d90150010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d9014fbb0 0x7f7d90150010 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d8001c210 tx=0x7f7d7c005800 comp rx=0 tx=0).stop 2023-10-30T13:13:00.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 shutdown_connections 2023-10-30T13:13:00.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d740664a0 0x7f7d74068960 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d9014fbb0 0x7f7d90150010 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d901f2a80 0x7f7d901f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 --2- 172.21.15.148:0/193144214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d7c0047d0 0x7f7d90156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 >> 172.21.15.148:0/193144214 conn(0x7f7d90071ea0 msgr2=0x7f7d9006f640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:00.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 shutdown_connections 2023-10-30T13:13:00.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.781+0000 7f7d977d9640 1 -- 172.21.15.148:0/193144214 wait complete. 2023-10-30T13:13:00.805 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:10:22.349322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:10:32.611151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:10:33.044931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:13:00.806 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:13:00.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bb59e640 1 Processor -- start 2023-10-30T13:13:00.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bb59e640 1 -- start start 2023-10-30T13:13:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bb59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153c20 0x7f04bc15a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bb59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f04bc0cb4c0 0x7f04bc0cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bb59e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc15a6b0 con 0x7f04bc153840 2023-10-30T13:13:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bb59e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04bc15dbc0 con 0x7f04bc0cb4c0 2023-10-30T13:13:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bb59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc15d730 con 0x7f04bc153c20 2023-10-30T13:13:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04ba59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f04bc153840 0x7f04bc0c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44904/0 (socket says 172.21.15.148:44904) 2023-10-30T13:13:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04ba59c640 1 -- 172.21.15.148:0/1087500387 learned_addr learned my addr 172.21.15.148:0/1087500387 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:00.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bad9d640 1 --2- 172.21.15.148:0/1087500387 >> v2:172.21.15.150:3300/0 conn(0x7f04bc0cb4c0 0x7f04bc0cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04b9d9b640 1 --2- 172.21.15.148:0/1087500387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153c20 0x7f04bc15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04b959a640 1 -- 172.21.15.148:0/1087500387 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722496391 0 0) 0x7f04ac001120 con 0x7f04bc153840 2023-10-30T13:13:00.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04b959a640 1 -- 172.21.15.148:0/1087500387 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a4003530 con 0x7f04bc153840 2023-10-30T13:13:00.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04b959a640 1 -- 172.21.15.148:0/1087500387 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439571587 0 0) 0x7f04a4003530 con 0x7f04bc153840 2023-10-30T13:13:00.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bad9d640 1 -- 172.21.15.148:0/1087500387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153c20 msgr2=0x7f04bc15a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:00.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bad9d640 1 --2- 172.21.15.148:0/1087500387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153c20 0x7f04bc15a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bad9d640 1 -- 172.21.15.148:0/1087500387 >> v1:172.21.15.148:6789/0 conn(0x7f04bc153840 legacy=0x7f04bc0c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.945+0000 7f04bad9d640 1 -- 172.21.15.148:0/1087500387 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc0ca8f0 con 0x7f04bc0cb4c0 2023-10-30T13:13:00.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04bad9d640 1 --2- 172.21.15.148:0/1087500387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04bc0cb4c0 0x7f04bc0cef40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009d90 tx=0x7f04b401f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=814e775087d591 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04b959a640 1 -- 172.21.15.148:0/1087500387 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac001810 con 0x7f04bc0cb4c0 2023-10-30T13:13:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04b959a640 1 -- 172.21.15.148:0/1087500387 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b40a8bb0 con 0x7f04bc0cb4c0 2023-10-30T13:13:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04b959a640 1 -- 172.21.15.148:0/1087500387 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40b0440 con 0x7f04bc0cb4c0 2023-10-30T13:13:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04bb59e640 1 -- 172.21.15.148:0/1087500387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04bc0cb4c0 msgr2=0x7f04bc0cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04bb59e640 1 --2- 172.21.15.148:0/1087500387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04bc0cb4c0 0x7f04bc0cef40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009d90 tx=0x7f04b401f060 comp rx=0 tx=0).stop 2023-10-30T13:13:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04bb59e640 1 -- 172.21.15.148:0/1087500387 shutdown_connections 2023-10-30T13:13:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04bb59e640 1 --2- 172.21.15.148:0/1087500387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04bc0cb4c0 0x7f04bc0cef40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04bb59e640 1 --2- 172.21.15.148:0/1087500387 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153c20 0x7f04bc15a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04bb59e640 1 -- 172.21.15.148:0/1087500387 >> 172.21.15.148:0/1087500387 conn(0x7f04bc06be40 msgr2=0x7f04bc06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04bb59e640 1 -- 172.21.15.148:0/1087500387 shutdown_connections 2023-10-30T13:13:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.949+0000 7f04bb59e640 1 -- 172.21.15.148:0/1087500387 wait complete. 2023-10-30T13:13:00.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 Processor -- start 2023-10-30T13:13:00.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 -- start start 2023-10-30T13:13:00.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04bc0cb4c0 0x7f04bc0c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153840 0x7f04bc1f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04bc153c20 0x7f04bc1f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0c87b0 con 0x7f04bc153c20 2023-10-30T13:13:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc1f3370 con 0x7f04bc0cb4c0 2023-10-30T13:13:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc154170 con 0x7f04bc153840 2023-10-30T13:13:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04bc153c20 0x7f04bc1f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04bc153c20 0x7f04bc1f2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42706/0 (socket says 172.21.15.148:42706) 2023-10-30T13:13:00.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bad9d640 1 -- 172.21.15.148:0/2184250877 learned_addr learned my addr 172.21.15.148:0/2184250877 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:00.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04ba59c640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04bc0cb4c0 0x7f04bc0c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04b9d9b640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153840 0x7f04bc1f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bad9d640 1 -- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153840 msgr2=0x7f04bc1f2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bad9d640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153840 0x7f04bc1f2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bad9d640 1 -- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04bc0cb4c0 msgr2=0x7f04bc0c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bad9d640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04bc0cb4c0 0x7f04bc0c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bad9d640 1 -- 172.21.15.148:0/2184250877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009a70 con 0x7f04bc153c20 2023-10-30T13:13:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bad9d640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04bc153c20 0x7f04bc1f2d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009d90 tx=0x7f04b401be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04aaffd640 1 -- 172.21.15.148:0/2184250877 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40c0070 con 0x7f04bc153c20 2023-10-30T13:13:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04aaffd640 1 -- 172.21.15.148:0/2184250877 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b4004250 con 0x7f04bc153c20 2023-10-30T13:13:00.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04aaffd640 1 -- 172.21.15.148:0/2184250877 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40045c0 con 0x7f04bc153c20 2023-10-30T13:13:00.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc154410 con 0x7f04bc153c20 2023-10-30T13:13:00.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.953+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc0c8440 con 0x7f04bc153c20 2023-10-30T13:13:00.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.957+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc0c8960 con 0x7f04bc153c20 2023-10-30T13:13:00.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.957+0000 7f04aaffd640 1 -- 172.21.15.148:0/2184250877 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04b40b7030 con 0x7f04bc153c20 2023-10-30T13:13:00.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.957+0000 7f04aaffd640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f049c0664a0 0x7f049c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.957+0000 7f04aaffd640 1 -- 172.21.15.148:0/2184250877 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f04b4121c50 con 0x7f04bc153c20 2023-10-30T13:13:00.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.957+0000 7f04ba59c640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f049c0664a0 0x7f049c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.957+0000 7f04ba59c640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f049c0664a0 0x7f049c068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f04ac001520 tx=0x7f04ac0014b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:00.961+0000 7f04aaffd640 1 -- 172.21.15.148:0/2184250877 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f04bc0c8960 con 0x7f04bc153c20 2023-10-30T13:13:01.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f04bc04e9c0 con 0x7f04bc153c20 2023-10-30T13:13:01.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04aaffd640 1 -- 172.21.15.148:0/2184250877 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f04b40ef050 con 0x7f04bc153c20 2023-10-30T13:13:01.113 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2023-10-30T13:13:01.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f049c0664a0 msgr2=0x7f049c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f049c0664a0 0x7f049c068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f04ac001520 tx=0x7f04ac0014b0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04bc153c20 msgr2=0x7f04bc1f2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04bc153c20 0x7f04bc1f2d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009d90 tx=0x7f04b401be00 comp rx=0 tx=0).stop 2023-10-30T13:13:01.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 shutdown_connections 2023-10-30T13:13:01.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f049c0664a0 0x7f049c068960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04bc153c20 0x7f04bc1f2d40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04bc153840 0x7f04bc1f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 --2- 172.21.15.148:0/2184250877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04bc0cb4c0 0x7f04bc0c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.109+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 >> 172.21.15.148:0/2184250877 conn(0x7f04bc06be40 msgr2=0x7f04bc06c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:01.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.113+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 shutdown_connections 2023-10-30T13:13:01.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.113+0000 7f04bb59e640 1 -- 172.21.15.148:0/2184250877 wait complete. 2023-10-30T13:13:01.133 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:13:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.269+0000 7f18a24d6640 1 Processor -- start 2023-10-30T13:13:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.269+0000 7f18a24d6640 1 -- start start 2023-10-30T13:13:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f18a24d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f189c06d420 0x7f189c1650b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f18a24d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f189c15a670 con 0x7f189c15a1a0 2023-10-30T13:13:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f18a24d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f189c0c0f70 con 0x7f189c1539e0 2023-10-30T13:13:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f18a24d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f189c0ca3c0 con 0x7f189c06d420 2023-10-30T13:13:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f18a0a73640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f189c06d420 0x7f189c1650b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f189b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f189c15a1a0 0x7f189c0c78f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44914/0 (socket says 172.21.15.148:44914) 2023-10-30T13:13:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f189b7fe640 1 -- 172.21.15.148:0/2985885054 learned_addr learned my addr 172.21.15.148:0/2985885054 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f18a0a73640 1 -- 172.21.15.148:0/2985885054 >> v1:172.21.15.150:6789/0 conn(0x7f189c1539e0 legacy=0x7f189c0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f18a0a73640 1 -- 172.21.15.148:0/2985885054 >> v1:172.21.15.148:6789/0 conn(0x7f189c15a1a0 legacy=0x7f189c0c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.273+0000 7f18a0a73640 1 -- 172.21.15.148:0/2985885054 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189c0c1250 con 0x7f189c06d420 2023-10-30T13:13:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a0a73640 1 --2- 172.21.15.148:0/2985885054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 0x7f189c1650b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1894009d90 tx=0x7f189401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7cbabddb9f8e85 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f189affd640 1 -- 172.21.15.148:0/2985885054 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18940af600 con 0x7f189c06d420 2023-10-30T13:13:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f189affd640 1 -- 172.21.15.148:0/2985885054 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18940ab050 con 0x7f189c06d420 2023-10-30T13:13:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f189affd640 1 -- 172.21.15.148:0/2985885054 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18940af980 con 0x7f189c06d420 2023-10-30T13:13:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a24d6640 1 -- 172.21.15.148:0/2985885054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 msgr2=0x7f189c1650b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a24d6640 1 --2- 172.21.15.148:0/2985885054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 0x7f189c1650b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1894009d90 tx=0x7f189401f130 comp rx=0 tx=0).stop 2023-10-30T13:13:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a24d6640 1 -- 172.21.15.148:0/2985885054 shutdown_connections 2023-10-30T13:13:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a24d6640 1 --2- 172.21.15.148:0/2985885054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 0x7f189c1650b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a24d6640 1 -- 172.21.15.148:0/2985885054 >> 172.21.15.148:0/2985885054 conn(0x7f189c06c850 msgr2=0x7f189c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a24d6640 1 -- 172.21.15.148:0/2985885054 shutdown_connections 2023-10-30T13:13:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a24d6640 1 -- 172.21.15.148:0/2985885054 wait complete. 2023-10-30T13:13:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a24d6640 1 Processor -- start 2023-10-30T13:13:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.277+0000 7f18a24d6640 1 -- start start 2023-10-30T13:13:01.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a24d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 0x7f189c155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a24d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f189c1539e0 0x7f189c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a24d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c15a1a0 0x7f189c15c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a24d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c14cf70 con 0x7f189c06d420 2023-10-30T13:13:01.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a24d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c15abf0 con 0x7f189c15a1a0 2023-10-30T13:13:01.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a24d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189c157010 con 0x7f189c1539e0 2023-10-30T13:13:01.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 0x7f189c155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 0x7f189c155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42722/0 (socket says 172.21.15.148:42722) 2023-10-30T13:13:01.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189bfff640 1 -- 172.21.15.148:0/4146628479 learned_addr learned my addr 172.21.15.148:0/4146628479 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:01.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a0a73640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c15a1a0 0x7f189c15c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189b7fe640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f189c1539e0 0x7f189c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189bfff640 1 -- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f189c1539e0 msgr2=0x7f189c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189bfff640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f189c1539e0 0x7f189c15be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189bfff640 1 -- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c15a1a0 msgr2=0x7f189c15c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189bfff640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c15a1a0 0x7f189c15c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189bfff640 1 -- 172.21.15.148:0/4146628479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1894009a70 con 0x7f189c06d420 2023-10-30T13:13:01.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a0a73640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c15a1a0 0x7f189c15c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:01.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f189bfff640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 0x7f189c155ec0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f188c00c500 tx=0x7f188c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:01.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f1898ff9640 1 -- 172.21.15.148:0/4146628479 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c005220 con 0x7f189c06d420 2023-10-30T13:13:01.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f1898ff9640 1 -- 172.21.15.148:0/4146628479 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c005380 con 0x7f189c06d420 2023-10-30T13:13:01.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f1898ff9640 1 -- 172.21.15.148:0/4146628479 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c007270 con 0x7f189c06d420 2023-10-30T13:13:01.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f189c1569d0 con 0x7f189c06d420 2023-10-30T13:13:01.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189c15b390 con 0x7f189c06d420 2023-10-30T13:13:01.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f189c04e170 con 0x7f189c06d420 2023-10-30T13:13:01.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.281+0000 7f1898ff9640 1 -- 172.21.15.148:0/4146628479 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f188c0073d0 con 0x7f189c06d420 2023-10-30T13:13:01.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.285+0000 7f1898ff9640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f18880664a0 0x7f1888068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.285+0000 7f1898ff9640 1 -- 172.21.15.148:0/4146628479 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f188c0837e0 con 0x7f189c06d420 2023-10-30T13:13:01.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.285+0000 7f189b7fe640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f18880664a0 0x7f1888068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.285+0000 7f189b7fe640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f18880664a0 0x7f1888068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f189c155bf0 tx=0x7f1890009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:01.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.289+0000 7f1898ff9640 1 -- 172.21.15.148:0/4146628479 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f188c050be0 con 0x7f189c06d420 2023-10-30T13:13:01.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.437+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f189c152200 con 0x7f189c06d420 2023-10-30T13:13:01.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.437+0000 7f1898ff9640 1 -- 172.21.15.148:0/4146628479 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f188c054200 con 0x7f189c06d420 2023-10-30T13:13:01.443 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 32 2023-10-30T13:13:01.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f18880664a0 msgr2=0x7f1888068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f18880664a0 0x7f1888068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f189c155bf0 tx=0x7f1890009040 comp rx=0 tx=0).stop 2023-10-30T13:13:01.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 msgr2=0x7f189c155ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 0x7f189c155ec0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f188c00c500 tx=0x7f188c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 shutdown_connections 2023-10-30T13:13:01.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f18880664a0 0x7f1888068960 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c15a1a0 0x7f189c15c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f189c1539e0 0x7f189c15be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 --2- 172.21.15.148:0/4146628479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f189c06d420 0x7f189c155ec0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 >> 172.21.15.148:0/4146628479 conn(0x7f189c06c850 msgr2=0x7f189c071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:01.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 shutdown_connections 2023-10-30T13:13:01.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.441+0000 7f18a24d6640 1 -- 172.21.15.148:0/4146628479 wait complete. 2023-10-30T13:13:01.462 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:13:01.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b31d3e640 1 Processor -- start 2023-10-30T13:13:01.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b31d3e640 1 -- start start 2023-10-30T13:13:01.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b31d3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b2c153980 0x7f0b2c0caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b31d3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c159930 0x7f0b2c0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b31d3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c15a920 con 0x7f0b2c0cf1a0 2023-10-30T13:13:01.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b31d3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2c0cb930 con 0x7f0b2c153980 2023-10-30T13:13:01.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b31d3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c159dc0 con 0x7f0b2c159930 2023-10-30T13:13:01.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b2affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c159930 0x7f0b2c0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b2b7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b2c153980 0x7f0b2c0caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b2bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0b2c0cf1a0 0x7f0b2c15d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:42034/0 (socket says 172.21.15.148:42034) 2023-10-30T13:13:01.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b2bfff640 1 -- 172.21.15.148:0/1598753431 learned_addr learned my addr 172.21.15.148:0/1598753431 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:01.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b2affd640 1 -- 172.21.15.148:0/1598753431 >> v2:172.21.15.150:3300/0 conn(0x7f0b2c153980 msgr2=0x7f0b2c0caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:01.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b2affd640 1 --2- 172.21.15.148:0/1598753431 >> v2:172.21.15.150:3300/0 conn(0x7f0b2c153980 0x7f0b2c0caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b2affd640 1 -- 172.21.15.148:0/1598753431 >> v1:172.21.15.150:6789/0 conn(0x7f0b2c0cf1a0 legacy=0x7f0b2c15d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.601+0000 7f0b2affd640 1 -- 172.21.15.148:0/1598753431 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c15a080 con 0x7f0b2c159930 2023-10-30T13:13:01.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b2affd640 1 --2- 172.21.15.148:0/1598753431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c159930 0x7f0b2c0cb1e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b200097c0 tx=0x7f0b2001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4f503ef4fd9ca9a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:01.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b2a7fc640 1 -- 172.21.15.148:0/1598753431 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b200ab650 con 0x7f0b2c159930 2023-10-30T13:13:01.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b2a7fc640 1 -- 172.21.15.148:0/1598753431 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b200ab7b0 con 0x7f0b2c159930 2023-10-30T13:13:01.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b2a7fc640 1 -- 172.21.15.148:0/1598753431 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b200abb40 con 0x7f0b2c159930 2023-10-30T13:13:01.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b31d3e640 1 -- 172.21.15.148:0/1598753431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c159930 msgr2=0x7f0b2c0cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b31d3e640 1 --2- 172.21.15.148:0/1598753431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c159930 0x7f0b2c0cb1e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b200097c0 tx=0x7f0b2001c450 comp rx=0 tx=0).stop 2023-10-30T13:13:01.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b31d3e640 1 -- 172.21.15.148:0/1598753431 shutdown_connections 2023-10-30T13:13:01.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b31d3e640 1 --2- 172.21.15.148:0/1598753431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c159930 0x7f0b2c0cb1e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b31d3e640 1 --2- 172.21.15.148:0/1598753431 >> v2:172.21.15.150:3300/0 conn(0x7f0b2c153980 0x7f0b2c0caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b31d3e640 1 -- 172.21.15.148:0/1598753431 >> 172.21.15.148:0/1598753431 conn(0x7f0b2c06c0f0 msgr2=0x7f0b2c06c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:01.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b31d3e640 1 -- 172.21.15.148:0/1598753431 shutdown_connections 2023-10-30T13:13:01.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.605+0000 7f0b31d3e640 1 -- 172.21.15.148:0/1598753431 wait complete. 2023-10-30T13:13:01.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 Processor -- start 2023-10-30T13:13:01.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 -- start start 2023-10-30T13:13:01.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c0cf1a0 0x7f0b2c14af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c153980 0x7f0b2c152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b2c14ca40 0x7f0b2c1f9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c14d090 con 0x7f0b2c14ca40 2023-10-30T13:13:01.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c14d650 con 0x7f0b2c0cf1a0 2023-10-30T13:13:01.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c14d350 con 0x7f0b2c153980 2023-10-30T13:13:01.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b2c14ca40 0x7f0b2c1f9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c0cf1a0 0x7f0b2c14af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b2c14ca40 0x7f0b2c1f9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42732/0 (socket says 172.21.15.148:42732) 2023-10-30T13:13:01.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2bfff640 1 -- 172.21.15.148:0/2259007994 learned_addr learned my addr 172.21.15.148:0/2259007994 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:01.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2affd640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c153980 0x7f0b2c152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2bfff640 1 -- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c153980 msgr2=0x7f0b2c152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2bfff640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c153980 0x7f0b2c152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2bfff640 1 -- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c0cf1a0 msgr2=0x7f0b2c14af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2bfff640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c0cf1a0 0x7f0b2c14af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2bfff640 1 -- 172.21.15.148:0/2259007994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20009420 con 0x7f0b2c14ca40 2023-10-30T13:13:01.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b2bfff640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b2c14ca40 0x7f0b2c1f9e00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400cbf0 tx=0x7f0b2400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:01.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b13fff640 1 -- 172.21.15.148:0/2259007994 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2400ad40 con 0x7f0b2c14ca40 2023-10-30T13:13:01.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b13fff640 1 -- 172.21.15.148:0/2259007994 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b24012400 con 0x7f0b2c14ca40 2023-10-30T13:13:01.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b13fff640 1 -- 172.21.15.148:0/2259007994 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2400aea0 con 0x7f0b2c14ca40 2023-10-30T13:13:01.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c14b8f0 con 0x7f0b2c14ca40 2023-10-30T13:13:01.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.609+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c14bbb0 con 0x7f0b2c14ca40 2023-10-30T13:13:01.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.613+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c04e170 con 0x7f0b2c14ca40 2023-10-30T13:13:01.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.613+0000 7f0b13fff640 1 -- 172.21.15.148:0/2259007994 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b24007070 con 0x7f0b2c14ca40 2023-10-30T13:13:01.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.613+0000 7f0b13fff640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b180664a0 0x7f0b18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.613+0000 7f0b13fff640 1 -- 172.21.15.148:0/2259007994 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0b24083dc0 con 0x7f0b2c14ca40 2023-10-30T13:13:01.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.613+0000 7f0b2b7fe640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b180664a0 0x7f0b18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.613+0000 7f0b2b7fe640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b180664a0 0x7f0b18068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c009b60 tx=0x7f0b1c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:01.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.617+0000 7f0b13fff640 1 -- 172.21.15.148:0/2259007994 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b24051250 con 0x7f0b2c14ca40 2023-10-30T13:13:01.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0b2c14d790 con 0x7f0b2c14ca40 2023-10-30T13:13:01.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b13fff640 1 -- 172.21.15.148:0/2259007994 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0b240170a0 con 0x7f0b2c14ca40 2023-10-30T13:13:01.768 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 64 2023-10-30T13:13:01.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b180664a0 msgr2=0x7f0b18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b180664a0 0x7f0b18068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c009b60 tx=0x7f0b1c009190 comp rx=0 tx=0).stop 2023-10-30T13:13:01.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b2c14ca40 msgr2=0x7f0b2c1f9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b2c14ca40 0x7f0b2c1f9e00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400cbf0 tx=0x7f0b2400a430 comp rx=0 tx=0).stop 2023-10-30T13:13:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 shutdown_connections 2023-10-30T13:13:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b180664a0 0x7f0b18068960 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b2c14ca40 0x7f0b2c1f9e00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b2c153980 0x7f0b2c152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 --2- 172.21.15.148:0/2259007994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c0cf1a0 0x7f0b2c14af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 >> 172.21.15.148:0/2259007994 conn(0x7f0b2c06c0f0 msgr2=0x7f0b2c149770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:01.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 shutdown_connections 2023-10-30T13:13:01.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.765+0000 7f0b31d3e640 1 -- 172.21.15.148:0/2259007994 wait complete. 2023-10-30T13:13:01.789 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-10-30T13:13:01.789 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.921+0000 7f2b70f0f640 1 Processor -- start 2023-10-30T13:13:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.921+0000 7f2b70f0f640 1 -- start start 2023-10-30T13:13:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b70f0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b6c06e470 0x7f2b6c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b70f0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6c06afa0 con 0x7f2b6c0c8490 2023-10-30T13:13:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b70f0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6c154150 con 0x7f2b6c153d70 2023-10-30T13:13:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b70f0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c06d780 con 0x7f2b6c06e470 2023-10-30T13:13:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b6ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2b6c0c8490 0x7f2b6c0c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42634/0 (socket says 172.21.15.148:42634) 2023-10-30T13:13:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b6ad9d640 1 -- 172.21.15.148:0/1510262066 learned_addr learned my addr 172.21.15.148:0/1510262066 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b61d9b640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b6c06e470 0x7f2b6c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059669910 0 0) 0x7f2b6c06afa0 con 0x7f2b6c0c8490 2023-10-30T13:13:01.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b58003530 con 0x7f2b6c0c8490 2023-10-30T13:13:01.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854503490 0 0) 0x7f2b6c154150 con 0x7f2b6c153d70 2023-10-30T13:13:01.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b6c06afa0 con 0x7f2b6c153d70 2023-10-30T13:13:01.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3346369303 0 0) 0x7f2b58003530 con 0x7f2b6c0c8490 2023-10-30T13:13:01.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b6c06e470 msgr2=0x7f2b6c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:01.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b6c06e470 0x7f2b6c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 >> v1:172.21.15.150:6789/0 conn(0x7f2b6c153d70 legacy=0x7f2b6c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c06e030 con 0x7f2b6c0c8490 2023-10-30T13:13:01.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f2b64001680 con 0x7f2b6c0c8490 2023-10-30T13:13:01.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b61d9b640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b6c06e470 0x7f2b6c06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:13:01.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 >> v1:172.21.15.148:6789/0 conn(0x7f2b6c0c8490 legacy=0x7f2b6c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b6c06e470 0x7f2b58003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b580040d0 0x7f2b580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b58004d10 0x7f2b58005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c06d780 con 0x7f2b58004d10 2023-10-30T13:13:01.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5800baa0 con 0x7f2b580040d0 2023-10-30T13:13:01.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5800bc60 con 0x7f2b6c06e470 2023-10-30T13:13:01.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b6ad9d640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b58004d10 0x7f2b58005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b61d9b640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b580040d0 0x7f2b580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b6a59c640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b6c06e470 0x7f2b58003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b6ad9d640 1 -- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b6c06e470 msgr2=0x7f2b58003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:01.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b6ad9d640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b6c06e470 0x7f2b58003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b6ad9d640 1 -- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b580040d0 msgr2=0x7f2b580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:01.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b6ad9d640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b580040d0 0x7f2b580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.925+0000 7f2b6ad9d640 1 -- 172.21.15.148:0/1510262066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b54009420 con 0x7f2b58004d10 2023-10-30T13:13:01.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b6ad9d640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b58004d10 0x7f2b58005190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2b64001ec0 tx=0x7f2b6401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df02c1cbdffade36 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:01.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b640a8e30 con 0x7f2b58004d10 2023-10-30T13:13:01.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b640058e0 con 0x7f2b58004d10 2023-10-30T13:13:01.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b69d9b640 1 -- 172.21.15.148:0/1510262066 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64005cb0 con 0x7f2b58004d10 2023-10-30T13:13:01.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b70f0f640 1 -- 172.21.15.148:0/1510262066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b58004d10 msgr2=0x7f2b58005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b70f0f640 1 --2- 172.21.15.148:0/1510262066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b58004d10 0x7f2b58005190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2b64001ec0 tx=0x7f2b6401f210 comp rx=0 tx=0).stop 2023-10-30T13:13:01.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b6ad9d640 1 -- 172.21.15.148:0/1510262066 reap_dead start 2023-10-30T13:13:01.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b70f0f640 1 -- 172.21.15.148:0/1510262066 shutdown_connections 2023-10-30T13:13:01.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b70f0f640 1 -- 172.21.15.148:0/1510262066 >> 172.21.15.148:0/1510262066 conn(0x7f2b6c071ea0 msgr2=0x7f2b6c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:01.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b70f0f640 1 -- 172.21.15.148:0/1510262066 shutdown_connections 2023-10-30T13:13:01.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.929+0000 7f2b70f0f640 1 -- 172.21.15.148:0/1510262066 wait complete. 2023-10-30T13:13:01.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 Processor -- start 2023-10-30T13:13:01.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 -- start start 2023-10-30T13:13:01.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b58004d10 0x7f2b6c1e51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b6c1e5a70 0x7f2b6c1e6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b6c1efad0 0x7f2b6c1eff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c1ef3c0 con 0x7f2b6c1efad0 2023-10-30T13:13:01.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c14c6e0 con 0x7f2b6c1e5a70 2023-10-30T13:13:01.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c14ead0 con 0x7f2b58004d10 2023-10-30T13:13:01.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b6c1efad0 0x7f2b6c1eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b6c1efad0 0x7f2b6c1eff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38336/0 (socket says 172.21.15.148:38336) 2023-10-30T13:13:01.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6ad9d640 1 -- 172.21.15.148:0/907984681 learned_addr learned my addr 172.21.15.148:0/907984681 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:01.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6a59c640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b58004d10 0x7f2b6c1e51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b61d9b640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b6c1e5a70 0x7f2b6c1e6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6ad9d640 1 -- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b58004d10 msgr2=0x7f2b6c1e51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6ad9d640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b58004d10 0x7f2b6c1e51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6ad9d640 1 -- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b6c1e5a70 msgr2=0x7f2b6c1e6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:01.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6ad9d640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b6c1e5a70 0x7f2b6c1e6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:01.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6ad9d640 1 -- 172.21.15.148:0/907984681 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c1ef640 con 0x7f2b6c1efad0 2023-10-30T13:13:01.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b6ad9d640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b6c1efad0 0x7f2b6c1eff50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2b64001ec0 tx=0x7f2b58005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:01.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b637fe640 1 -- 172.21.15.148:0/907984681 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b640b7070 con 0x7f2b6c1efad0 2023-10-30T13:13:01.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b637fe640 1 -- 172.21.15.148:0/907984681 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b64001480 con 0x7f2b6c1efad0 2023-10-30T13:13:01.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6c1e4030 con 0x7f2b6c1efad0 2023-10-30T13:13:01.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b637fe640 1 -- 172.21.15.148:0/907984681 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64000e90 con 0x7f2b6c1efad0 2023-10-30T13:13:01.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.933+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6c1e45f0 con 0x7f2b6c1efad0 2023-10-30T13:13:01.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.937+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6c067280 con 0x7f2b6c1efad0 2023-10-30T13:13:01.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.937+0000 7f2b637fe640 1 -- 172.21.15.148:0/907984681 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b64004a10 con 0x7f2b6c1efad0 2023-10-30T13:13:01.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.937+0000 7f2b637fe640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b4c0664a0 0x7f2b4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:01.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.937+0000 7f2b637fe640 1 -- 172.21.15.148:0/907984681 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2b64120550 con 0x7f2b6c1efad0 2023-10-30T13:13:01.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.937+0000 7f2b6a59c640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b4c0664a0 0x7f2b4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:01.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.937+0000 7f2b6a59c640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b4c0664a0 0x7f2b4c068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c000f90 tx=0x7f2b5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:01.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:01.941+0000 7f2b637fe640 1 -- 172.21.15.148:0/907984681 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b640ed810 con 0x7f2b6c1efad0 2023-10-30T13:13:02.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b6c04e170 con 0x7f2b6c1efad0 2023-10-30T13:13:02.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b637fe640 1 -- 172.21.15.148:0/907984681 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f2b640f0e30 con 0x7f2b6c1efad0 2023-10-30T13:13:02.124 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:13:02.124 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:02.125 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:13:02.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b4c0664a0 msgr2=0x7f2b4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b4c0664a0 0x7f2b4c068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c000f90 tx=0x7f2b5c009040 comp rx=0 tx=0).stop 2023-10-30T13:13:02.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b6c1efad0 msgr2=0x7f2b6c1eff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b6c1efad0 0x7f2b6c1eff50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2b64001ec0 tx=0x7f2b58005800 comp rx=0 tx=0).stop 2023-10-30T13:13:02.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 shutdown_connections 2023-10-30T13:13:02.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b4c0664a0 0x7f2b4c068960 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b6c1efad0 0x7f2b6c1eff50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b6c1e5a70 0x7f2b6c1e6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 --2- 172.21.15.148:0/907984681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b58004d10 0x7f2b6c1e51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 >> 172.21.15.148:0/907984681 conn(0x7f2b6c071ea0 msgr2=0x7f2b6c0c97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:02.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 shutdown_connections 2023-10-30T13:13:02.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.121+0000 7f2b70f0f640 1 -- 172.21.15.148:0/907984681 wait complete. 2023-10-30T13:13:02.144 INFO:tasks.mds_thrash:Ready to start thrashing 2023-10-30T13:13:02.144 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-10-30T13:13:02.145 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:13:02.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f98223d3640 1 Processor -- start 2023-10-30T13:13:02.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f98223d3640 1 -- start start 2023-10-30T13:13:02.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f98223d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f981c15d6f0 0x7f981c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f98223d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981c0cb800 con 0x7f981c1538a0 2023-10-30T13:13:02.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f98223d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981c0cb330 con 0x7f981c0cf160 2023-10-30T13:13:02.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f98223d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981c0cad70 con 0x7f981c15d6f0 2023-10-30T13:13:02.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f981b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f981c15d6f0 0x7f981c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f981bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f981c1538a0 0x7f981c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42646/0 (socket says 172.21.15.148:42646) 2023-10-30T13:13:02.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f981bfff640 1 -- 172.21.15.148:0/995694991 learned_addr learned my addr 172.21.15.148:0/995694991 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:02.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f981b7fe640 1 -- 172.21.15.148:0/995694991 >> v1:172.21.15.150:6789/0 conn(0x7f981c0cf160 legacy=0x7f981c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f981b7fe640 1 -- 172.21.15.148:0/995694991 >> v1:172.21.15.148:6789/0 conn(0x7f981c1538a0 legacy=0x7f981c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f981affd640 1 -- 172.21.15.148:0/995694991 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094484449 0 0) 0x7f981c0cb800 con 0x7f981c1538a0 2023-10-30T13:13:02.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.281+0000 7f981b7fe640 1 -- 172.21.15.148:0/995694991 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c0cb050 con 0x7f981c15d6f0 2023-10-30T13:13:02.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f981b7fe640 1 --2- 172.21.15.148:0/995694991 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c15d6f0 0x7f981c15a090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f98100097c0 tx=0x7f981001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c3cfda55f945b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:02.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f981affd640 1 -- 172.21.15.148:0/995694991 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100a8970 con 0x7f981c15d6f0 2023-10-30T13:13:02.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f981affd640 1 -- 172.21.15.148:0/995694991 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98100a8ad0 con 0x7f981c15d6f0 2023-10-30T13:13:02.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f981affd640 1 -- 172.21.15.148:0/995694991 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100a8e60 con 0x7f981c15d6f0 2023-10-30T13:13:02.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f98223d3640 1 -- 172.21.15.148:0/995694991 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c15d6f0 msgr2=0x7f981c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f98223d3640 1 --2- 172.21.15.148:0/995694991 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c15d6f0 0x7f981c15a090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f98100097c0 tx=0x7f981001ec70 comp rx=0 tx=0).stop 2023-10-30T13:13:02.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f98223d3640 1 -- 172.21.15.148:0/995694991 shutdown_connections 2023-10-30T13:13:02.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f98223d3640 1 --2- 172.21.15.148:0/995694991 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c15d6f0 0x7f981c15a090 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f98223d3640 1 -- 172.21.15.148:0/995694991 >> 172.21.15.148:0/995694991 conn(0x7f981c06bfe0 msgr2=0x7f981c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:02.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f98223d3640 1 -- 172.21.15.148:0/995694991 shutdown_connections 2023-10-30T13:13:02.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.285+0000 7f98223d3640 1 -- 172.21.15.148:0/995694991 wait complete. 2023-10-30T13:13:02.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 Processor -- start 2023-10-30T13:13:02.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 -- start start 2023-10-30T13:13:02.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f981c0cf160 0x7f981c14cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c1538a0 0x7f981c150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f981c15d6f0 0x7f981c1ef920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0c7d50 con 0x7f981c1538a0 2023-10-30T13:13:02.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c1ef490 con 0x7f981c0cf160 2023-10-30T13:13:02.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c14db50 con 0x7f981c15d6f0 2023-10-30T13:13:02.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c1538a0 0x7f981c150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c1538a0 0x7f981c150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38344/0 (socket says 172.21.15.148:38344) 2023-10-30T13:13:02.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981b7fe640 1 -- 172.21.15.148:0/4146434524 learned_addr learned my addr 172.21.15.148:0/4146434524 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:02.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981bfff640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f981c0cf160 0x7f981c14cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f9820970640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f981c15d6f0 0x7f981c1ef920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981b7fe640 1 -- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f981c15d6f0 msgr2=0x7f981c1ef920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981b7fe640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f981c15d6f0 0x7f981c1ef920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981b7fe640 1 -- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f981c0cf160 msgr2=0x7f981c14cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981b7fe640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f981c0cf160 0x7f981c14cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981b7fe640 1 -- 172.21.15.148:0/4146434524 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810009420 con 0x7f981c1538a0 2023-10-30T13:13:02.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981bfff640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f981c0cf160 0x7f981c14cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:02.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f981b7fe640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c1538a0 0x7f981c150040 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f981001b880 tx=0x7f981001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:02.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f9818ff9640 1 -- 172.21.15.148:0/4146434524 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100b9610 con 0x7f981c1538a0 2023-10-30T13:13:02.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f9818ff9640 1 -- 172.21.15.148:0/4146434524 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98100b9790 con 0x7f981c1538a0 2023-10-30T13:13:02.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f9818ff9640 1 -- 172.21.15.148:0/4146434524 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100b9b40 con 0x7f981c1538a0 2023-10-30T13:13:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c1ef140 con 0x7f981c1538a0 2023-10-30T13:13:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.289+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c14f720 con 0x7f981c1538a0 2023-10-30T13:13:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.293+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c1ef640 con 0x7f981c1538a0 2023-10-30T13:13:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.293+0000 7f9818ff9640 1 -- 172.21.15.148:0/4146434524 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f98100b7040 con 0x7f981c1538a0 2023-10-30T13:13:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.293+0000 7f9818ff9640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f98080664a0 0x7f9808068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.293+0000 7f9818ff9640 1 -- 172.21.15.148:0/4146434524 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9810121d60 con 0x7f981c1538a0 2023-10-30T13:13:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.293+0000 7f981bfff640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f98080664a0 0x7f9808068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.293+0000 7f981bfff640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f98080664a0 0x7f9808068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f980c007fd0 tx=0x7f980c007c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.297+0000 7f9818ff9640 1 -- 172.21.15.148:0/4146434524 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f98100ef020 con 0x7f981c1538a0 2023-10-30T13:13:02.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.445+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f981c159b70 con 0x7f98080664a0 2023-10-30T13:13:02.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.469+0000 7f9818ff9640 1 -- 172.21.15.148:0/4146434524 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288228 (secure 0 0 0) 0x7f980c005c90 con 0x7f98080664a0 2023-10-30T13:13:02.475 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:13:02.477 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:13:02.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f98080664a0 msgr2=0x7f9808068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f98080664a0 0x7f9808068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f980c007fd0 tx=0x7f980c007c20 comp rx=0 tx=0).stop 2023-10-30T13:13:02.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c1538a0 msgr2=0x7f981c150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c1538a0 0x7f981c150040 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f981001b880 tx=0x7f981001ba10 comp rx=0 tx=0).stop 2023-10-30T13:13:02.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 shutdown_connections 2023-10-30T13:13:02.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f98080664a0 0x7f9808068960 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f981c1538a0 0x7f981c150040 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f981c15d6f0 0x7f981c1ef920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 --2- 172.21.15.148:0/4146434524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f981c0cf160 0x7f981c14cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 >> 172.21.15.148:0/4146434524 conn(0x7f981c06bfe0 msgr2=0x7f981c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:02.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 shutdown_connections 2023-10-30T13:13:02.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.473+0000 7f98223d3640 1 -- 172.21.15.148:0/4146434524 wait complete. 2023-10-30T13:13:02.496 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2023-10-30T13:13:02.314085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":285,"num_write_kb":2455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219284,"kb_used_data":3108,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755436,"statfs":{"total":773094113280,"available":772869566464,"internally_reserved":0,"allocated":3182592,"data_stored":1857794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003910"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530410+0000","last_change":"2023-10-30T13:10:34.379031+0000","last_active":"2023-10-30T13:11:15.530410+0000","last_peered":"2023-10-30T13:11:15.530410+0000","last_clean":"2023-10-30T13:11:15.530410+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:11:15.530410+0000","last_undegraded":"2023-10-30T13:11:15.530410+0000","last_fullsized":"2023-10-30T13:11:15.530410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:32.529312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434689+0000","last_change":"2023-10-30T13:10:34.324308+0000","last_active":"2023-10-30T13:11:15.434689+0000","last_peered":"2023-10-30T13:11:15.434689+0000","last_clean":"2023-10-30T13:11:15.434689+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:11:15.434689+0000","last_undegraded":"2023-10-30T13:11:15.434689+0000","last_fullsized":"2023-10-30T13:11:15.434689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:21.885716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561039+0000","last_change":"2023-10-30T13:10:34.334122+0000","last_active":"2023-10-30T13:11:15.561039+0000","last_peered":"2023-10-30T13:11:15.561039+0000","last_clean":"2023-10-30T13:11:15.561039+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:11:15.561039+0000","last_undegraded":"2023-10-30T13:11:15.561039+0000","last_fullsized":"2023-10-30T13:11:15.561039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:55.045024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437588+0000","last_change":"2023-10-30T13:11:15.435943+0000","last_active":"2023-10-30T13:11:15.437588+0000","last_peered":"2023-10-30T13:11:15.437588+0000","last_clean":"2023-10-30T13:11:15.437588+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:11:15.437588+0000","last_undegraded":"2023-10-30T13:11:15.437588+0000","last_fullsized":"2023-10-30T13:11:15.437588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:03.152688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441733+0000","last_change":"2023-10-30T13:10:34.371670+0000","last_active":"2023-10-30T13:11:14.441733+0000","last_peered":"2023-10-30T13:11:14.441733+0000","last_clean":"2023-10-30T13:11:14.441733+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:11:14.441733+0000","last_undegraded":"2023-10-30T13:11:14.441733+0000","last_fullsized":"2023-10-30T13:11:14.441733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:03:54.773965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441333+0000","last_change":"2023-10-30T13:10:34.335752+0000","last_active":"2023-10-30T13:11:14.441333+0000","last_peered":"2023-10-30T13:11:14.441333+0000","last_clean":"2023-10-30T13:11:14.441333+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:11:14.441333+0000","last_undegraded":"2023-10-30T13:11:14.441333+0000","last_fullsized":"2023-10-30T13:11:14.441333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:33.278920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.440809+0000","last_change":"2023-10-30T13:10:34.371817+0000","last_active":"2023-10-30T13:11:14.440809+0000","last_peered":"2023-10-30T13:11:14.440809+0000","last_clean":"2023-10-30T13:11:14.440809+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:11:14.440809+0000","last_undegraded":"2023-10-30T13:11:14.440809+0000","last_fullsized":"2023-10-30T13:11:14.440809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:33.797113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.427171+0000","last_change":"2023-10-30T13:10:34.348205+0000","last_active":"2023-10-30T13:11:14.427171+0000","last_peered":"2023-10-30T13:11:14.427171+0000","last_clean":"2023-10-30T13:11:14.427171+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:11:14.427171+0000","last_undegraded":"2023-10-30T13:11:14.427171+0000","last_fullsized":"2023-10-30T13:11:14.427171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:57.492738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.442942+0000","last_change":"2023-10-30T13:10:34.381052+0000","last_active":"2023-10-30T13:11:14.442942+0000","last_peered":"2023-10-30T13:11:14.442942+0000","last_clean":"2023-10-30T13:11:14.442942+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:11:14.442942+0000","last_undegraded":"2023-10-30T13:11:14.442942+0000","last_fullsized":"2023-10-30T13:11:14.442942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:16.697332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.447936+0000","last_change":"2023-10-30T13:10:34.356932+0000","last_active":"2023-10-30T13:11:14.447936+0000","last_peered":"2023-10-30T13:11:14.447936+0000","last_clean":"2023-10-30T13:11:14.447936+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:11:14.447936+0000","last_undegraded":"2023-10-30T13:11:14.447936+0000","last_fullsized":"2023-10-30T13:11:14.447936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:54.091993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.447369+0000","last_change":"2023-10-30T13:11:15.447369+0000","last_active":"2023-10-30T13:11:15.447369+0000","last_peered":"2023-10-30T13:11:15.447369+0000","last_clean":"2023-10-30T13:11:15.447369+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:11:15.447369+0000","last_undegraded":"2023-10-30T13:11:15.447369+0000","last_fullsized":"2023-10-30T13:11:15.447369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:32.683227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.439648+0000","last_change":"2023-10-30T13:10:34.365161+0000","last_active":"2023-10-30T13:11:14.439648+0000","last_peered":"2023-10-30T13:11:14.439648+0000","last_clean":"2023-10-30T13:11:14.439648+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:11:14.439648+0000","last_undegraded":"2023-10-30T13:11:14.439648+0000","last_fullsized":"2023-10-30T13:11:14.439648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:51.721060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430223+0000","last_change":"2023-10-30T13:10:34.351901+0000","last_active":"2023-10-30T13:11:15.430223+0000","last_peered":"2023-10-30T13:11:15.430223+0000","last_clean":"2023-10-30T13:11:15.430223+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:11:15.430223+0000","last_undegraded":"2023-10-30T13:11:15.430223+0000","last_fullsized":"2023-10-30T13:11:15.430223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:11.576546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560710+0000","last_change":"2023-10-30T13:10:34.323358+0000","last_active":"2023-10-30T13:11:15.560710+0000","last_peered":"2023-10-30T13:11:15.560710+0000","last_clean":"2023-10-30T13:11:15.560710+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:11:15.560710+0000","last_undegraded":"2023-10-30T13:11:15.560710+0000","last_fullsized":"2023-10-30T13:11:15.560710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:03.178822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433306+0000","last_change":"2023-10-30T13:10:34.360120+0000","last_active":"2023-10-30T13:11:15.433306+0000","last_peered":"2023-10-30T13:11:15.433306+0000","last_clean":"2023-10-30T13:11:15.433306+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:11:15.433306+0000","last_undegraded":"2023-10-30T13:11:15.433306+0000","last_fullsized":"2023-10-30T13:11:15.433306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:30.731746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437472+0000","last_change":"2023-10-30T13:10:34.333752+0000","last_active":"2023-10-30T13:11:15.437472+0000","last_peered":"2023-10-30T13:11:15.437472+0000","last_clean":"2023-10-30T13:11:15.437472+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:11:15.437472+0000","last_undegraded":"2023-10-30T13:11:15.437472+0000","last_fullsized":"2023-10-30T13:11:15.437472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:45.390603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.441064+0000","last_change":"2023-10-30T13:10:34.371931+0000","last_active":"2023-10-30T13:11:15.441064+0000","last_peered":"2023-10-30T13:11:15.441064+0000","last_clean":"2023-10-30T13:11:15.441064+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:11:15.441064+0000","last_undegraded":"2023-10-30T13:11:15.441064+0000","last_fullsized":"2023-10-30T13:11:15.441064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:51.514990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433093+0000","last_change":"2023-10-30T13:10:34.352747+0000","last_active":"2023-10-30T13:11:15.433093+0000","last_peered":"2023-10-30T13:11:15.433093+0000","last_clean":"2023-10-30T13:11:15.433093+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:11:15.433093+0000","last_undegraded":"2023-10-30T13:11:15.433093+0000","last_fullsized":"2023-10-30T13:11:15.433093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:21.543351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437590+0000","last_change":"2023-10-30T13:11:15.436291+0000","last_active":"2023-10-30T13:11:15.437590+0000","last_peered":"2023-10-30T13:11:15.437590+0000","last_clean":"2023-10-30T13:11:15.437590+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:11:15.437590+0000","last_undegraded":"2023-10-30T13:11:15.437590+0000","last_fullsized":"2023-10-30T13:11:15.437590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:22.745942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446501+0000","last_change":"2023-10-30T13:10:34.366575+0000","last_active":"2023-10-30T13:11:15.446501+0000","last_peered":"2023-10-30T13:11:15.446501+0000","last_clean":"2023-10-30T13:11:15.446501+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:11:15.446501+0000","last_undegraded":"2023-10-30T13:11:15.446501+0000","last_fullsized":"2023-10-30T13:11:15.446501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:23.540645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560863+0000","last_change":"2023-10-30T13:10:34.368761+0000","last_active":"2023-10-30T13:11:15.560863+0000","last_peered":"2023-10-30T13:11:15.560863+0000","last_clean":"2023-10-30T13:11:15.560863+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:11:15.560863+0000","last_undegraded":"2023-10-30T13:11:15.560863+0000","last_fullsized":"2023-10-30T13:11:15.560863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:32.861756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438076+0000","last_change":"2023-10-30T13:10:34.372881+0000","last_active":"2023-10-30T13:11:15.438076+0000","last_peered":"2023-10-30T13:11:15.438076+0000","last_clean":"2023-10-30T13:11:15.438076+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:11:15.438076+0000","last_undegraded":"2023-10-30T13:11:15.438076+0000","last_fullsized":"2023-10-30T13:11:15.438076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:23.855779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430039+0000","last_change":"2023-10-30T13:10:34.359359+0000","last_active":"2023-10-30T13:11:15.430039+0000","last_peered":"2023-10-30T13:11:15.430039+0000","last_clean":"2023-10-30T13:11:15.430039+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:11:15.430039+0000","last_undegraded":"2023-10-30T13:11:15.430039+0000","last_fullsized":"2023-10-30T13:11:15.430039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:08.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437617+0000","last_change":"2023-10-30T13:10:34.347431+0000","last_active":"2023-10-30T13:11:15.437617+0000","last_peered":"2023-10-30T13:11:15.437617+0000","last_clean":"2023-10-30T13:11:15.437617+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:11:15.437617+0000","last_undegraded":"2023-10-30T13:11:15.437617+0000","last_fullsized":"2023-10-30T13:11:15.437617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:25.094617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.442207+0000","last_change":"2023-10-30T13:11:15.440541+0000","last_active":"2023-10-30T13:11:15.442207+0000","last_peered":"2023-10-30T13:11:15.442207+0000","last_clean":"2023-10-30T13:11:15.442207+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:11:15.442207+0000","last_undegraded":"2023-10-30T13:11:15.442207+0000","last_fullsized":"2023-10-30T13:11:15.442207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:16.858272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.447245+0000","last_change":"2023-10-30T13:10:34.379878+0000","last_active":"2023-10-30T13:11:15.447245+0000","last_peered":"2023-10-30T13:11:15.447245+0000","last_clean":"2023-10-30T13:11:15.447245+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:11:15.447245+0000","last_undegraded":"2023-10-30T13:11:15.447245+0000","last_fullsized":"2023-10-30T13:11:15.447245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:42.280634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434611+0000","last_change":"2023-10-30T13:10:34.332020+0000","last_active":"2023-10-30T13:11:15.434611+0000","last_peered":"2023-10-30T13:11:15.434611+0000","last_clean":"2023-10-30T13:11:15.434611+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:11:15.434611+0000","last_undegraded":"2023-10-30T13:11:15.434611+0000","last_fullsized":"2023-10-30T13:11:15.434611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:53.057012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430366+0000","last_change":"2023-10-30T13:10:34.363154+0000","last_active":"2023-10-30T13:11:15.430366+0000","last_peered":"2023-10-30T13:11:15.430366+0000","last_clean":"2023-10-30T13:11:15.430366+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:11:15.430366+0000","last_undegraded":"2023-10-30T13:11:15.430366+0000","last_fullsized":"2023-10-30T13:11:15.430366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:04.359629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560893+0000","last_change":"2023-10-30T13:10:34.332160+0000","last_active":"2023-10-30T13:11:15.560893+0000","last_peered":"2023-10-30T13:11:15.560893+0000","last_clean":"2023-10-30T13:11:15.560893+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:11:15.560893+0000","last_undegraded":"2023-10-30T13:11:15.560893+0000","last_fullsized":"2023-10-30T13:11:15.560893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:28.128344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438275+0000","last_change":"2023-10-30T13:10:34.337617+0000","last_active":"2023-10-30T13:11:15.438275+0000","last_peered":"2023-10-30T13:11:15.438275+0000","last_clean":"2023-10-30T13:11:15.438275+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:11:15.438275+0000","last_undegraded":"2023-10-30T13:11:15.438275+0000","last_fullsized":"2023-10-30T13:11:15.438275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:10.287441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.432768+0000","last_change":"2023-10-30T13:10:34.359681+0000","last_active":"2023-10-30T13:11:15.432768+0000","last_peered":"2023-10-30T13:11:15.432768+0000","last_clean":"2023-10-30T13:11:15.432768+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:11:15.432768+0000","last_undegraded":"2023-10-30T13:11:15.432768+0000","last_fullsized":"2023-10-30T13:11:15.432768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:02.812352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.443436+0000","last_change":"2023-10-30T13:10:34.344864+0000","last_active":"2023-10-30T13:11:15.443436+0000","last_peered":"2023-10-30T13:11:15.443436+0000","last_clean":"2023-10-30T13:11:15.443436+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:11:15.443436+0000","last_undegraded":"2023-10-30T13:11:15.443436+0000","last_fullsized":"2023-10-30T13:11:15.443436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:13.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.439520+0000","last_change":"2023-10-30T13:10:34.364069+0000","last_active":"2023-10-30T13:11:14.439520+0000","last_peered":"2023-10-30T13:11:14.439520+0000","last_clean":"2023-10-30T13:11:14.439520+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:11:14.439520+0000","last_undegraded":"2023-10-30T13:11:14.439520+0000","last_fullsized":"2023-10-30T13:11:14.439520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:15.717336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.427598+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:11:14.427598+0000","last_peered":"2023-10-30T13:11:14.427598+0000","last_clean":"2023-10-30T13:11:14.427598+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:11:14.427598+0000","last_undegraded":"2023-10-30T13:11:14.427598+0000","last_fullsized":"2023-10-30T13:11:14.427598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.503227+0000","last_change":"2023-10-30T13:10:34.360935+0000","last_active":"2023-10-30T13:11:14.503227+0000","last_peered":"2023-10-30T13:11:14.503227+0000","last_clean":"2023-10-30T13:11:14.503227+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:11:14.503227+0000","last_undegraded":"2023-10-30T13:11:14.503227+0000","last_fullsized":"2023-10-30T13:11:14.503227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:58.639753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.447505+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:11:14.447505+0000","last_peered":"2023-10-30T13:11:14.447505+0000","last_clean":"2023-10-30T13:11:14.447505+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:11:14.447505+0000","last_undegraded":"2023-10-30T13:11:14.447505+0000","last_fullsized":"2023-10-30T13:11:14.447505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430030+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:11:15.430030+0000","last_peered":"2023-10-30T13:11:15.430030+0000","last_clean":"2023-10-30T13:11:15.430030+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:11:15.430030+0000","last_undegraded":"2023-10-30T13:11:15.430030+0000","last_fullsized":"2023-10-30T13:11:15.430030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430018+0000","last_change":"2023-10-30T13:10:34.329935+0000","last_active":"2023-10-30T13:11:15.430018+0000","last_peered":"2023-10-30T13:11:15.430018+0000","last_clean":"2023-10-30T13:11:15.430018+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:11:15.430018+0000","last_undegraded":"2023-10-30T13:11:15.430018+0000","last_fullsized":"2023-10-30T13:11:15.430018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:07.795163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.434429+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:11:14.434429+0000","last_peered":"2023-10-30T13:11:14.434429+0000","last_clean":"2023-10-30T13:11:14.434429+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:11:14.434429+0000","last_undegraded":"2023-10-30T13:11:14.434429+0000","last_fullsized":"2023-10-30T13:11:14.434429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433082+0000","last_change":"2023-10-30T13:10:34.331351+0000","last_active":"2023-10-30T13:11:15.433082+0000","last_peered":"2023-10-30T13:11:15.433082+0000","last_clean":"2023-10-30T13:11:15.433082+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:11:15.433082+0000","last_undegraded":"2023-10-30T13:11:15.433082+0000","last_fullsized":"2023-10-30T13:11:15.433082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:27.864073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.448058+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:11:14.448058+0000","last_peered":"2023-10-30T13:11:14.448058+0000","last_clean":"2023-10-30T13:11:14.448058+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:11:14.448058+0000","last_undegraded":"2023-10-30T13:11:14.448058+0000","last_fullsized":"2023-10-30T13:11:14.448058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531489+0000","last_change":"2023-10-30T13:11:15.530240+0000","last_active":"2023-10-30T13:11:15.531489+0000","last_peered":"2023-10-30T13:11:15.531489+0000","last_clean":"2023-10-30T13:11:15.531489+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:11:15.531489+0000","last_undegraded":"2023-10-30T13:11:15.531489+0000","last_fullsized":"2023-10-30T13:11:15.531489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:40:51.011001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.438554+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:11:14.438554+0000","last_peered":"2023-10-30T13:11:14.438554+0000","last_clean":"2023-10-30T13:11:14.438554+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:11:14.438554+0000","last_undegraded":"2023-10-30T13:11:14.438554+0000","last_fullsized":"2023-10-30T13:11:14.438554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561157+0000","last_change":"2023-10-30T13:10:34.335243+0000","last_active":"2023-10-30T13:11:15.561157+0000","last_peered":"2023-10-30T13:11:15.561157+0000","last_clean":"2023-10-30T13:11:15.561157+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:11:15.561157+0000","last_undegraded":"2023-10-30T13:11:15.561157+0000","last_fullsized":"2023-10-30T13:11:15.561157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:02.970675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434664+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:11:15.434664+0000","last_peered":"2023-10-30T13:11:15.434664+0000","last_clean":"2023-10-30T13:11:15.434664+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:11:15.434664+0000","last_undegraded":"2023-10-30T13:11:15.434664+0000","last_fullsized":"2023-10-30T13:11:15.434664+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440790+0000","last_change":"2023-10-30T13:10:34.381099+0000","last_active":"2023-10-30T13:11:15.440790+0000","last_peered":"2023-10-30T13:11:15.440790+0000","last_clean":"2023-10-30T13:11:15.440790+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:11:15.440790+0000","last_undegraded":"2023-10-30T13:11:15.440790+0000","last_fullsized":"2023-10-30T13:11:15.440790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:16.694101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446654+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:11:15.446654+0000","last_peered":"2023-10-30T13:11:15.446654+0000","last_clean":"2023-10-30T13:11:15.446654+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:11:15.446654+0000","last_undegraded":"2023-10-30T13:11:15.446654+0000","last_fullsized":"2023-10-30T13:11:15.446654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441447+0000","last_change":"2023-10-30T13:10:34.372219+0000","last_active":"2023-10-30T13:11:14.441447+0000","last_peered":"2023-10-30T13:11:14.441447+0000","last_clean":"2023-10-30T13:11:14.441447+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:11:14.441447+0000","last_undegraded":"2023-10-30T13:11:14.441447+0000","last_fullsized":"2023-10-30T13:11:14.441447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:38.329952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561046+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:11:15.561046+0000","last_peered":"2023-10-30T13:11:15.561046+0000","last_clean":"2023-10-30T13:11:15.561046+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:11:15.561046+0000","last_undegraded":"2023-10-30T13:11:15.561046+0000","last_fullsized":"2023-10-30T13:11:15.561046+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560795+0000","last_change":"2023-10-30T13:10:34.368736+0000","last_active":"2023-10-30T13:11:15.560795+0000","last_peered":"2023-10-30T13:11:15.560795+0000","last_clean":"2023-10-30T13:11:15.560795+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:11:15.560795+0000","last_undegraded":"2023-10-30T13:11:15.560795+0000","last_fullsized":"2023-10-30T13:11:15.560795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:11.128769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438505+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:11:15.438505+0000","last_peered":"2023-10-30T13:11:15.438505+0000","last_clean":"2023-10-30T13:11:15.438505+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:11:15.438505+0000","last_undegraded":"2023-10-30T13:11:15.438505+0000","last_fullsized":"2023-10-30T13:11:15.438505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531029+0000","last_change":"2023-10-30T13:10:34.376512+0000","last_active":"2023-10-30T13:11:15.531029+0000","last_peered":"2023-10-30T13:11:15.531029+0000","last_clean":"2023-10-30T13:11:15.531029+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:11:15.531029+0000","last_undegraded":"2023-10-30T13:11:15.531029+0000","last_fullsized":"2023-10-30T13:11:15.531029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.856690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446249+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:11:15.446249+0000","last_peered":"2023-10-30T13:11:15.446249+0000","last_clean":"2023-10-30T13:11:15.446249+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:11:15.446249+0000","last_undegraded":"2023-10-30T13:11:15.446249+0000","last_fullsized":"2023-10-30T13:11:15.446249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.443575+0000","last_change":"2023-10-30T13:10:34.344863+0000","last_active":"2023-10-30T13:11:15.443575+0000","last_peered":"2023-10-30T13:11:15.443575+0000","last_clean":"2023-10-30T13:11:15.443575+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:11:15.443575+0000","last_undegraded":"2023-10-30T13:11:15.443575+0000","last_fullsized":"2023-10-30T13:11:15.443575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:25.199813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'15","reported_seq":30,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.432685+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:11:15.432685+0000","last_peered":"2023-10-30T13:11:15.432685+0000","last_clean":"2023-10-30T13:11:15.432685+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:11:15.432685+0000","last_undegraded":"2023-10-30T13:11:15.432685+0000","last_fullsized":"2023-10-30T13:11:15.432685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530363+0000","last_change":"2023-10-30T13:10:34.371813+0000","last_active":"2023-10-30T13:11:15.530363+0000","last_peered":"2023-10-30T13:11:15.530363+0000","last_clean":"2023-10-30T13:11:15.530363+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:11:15.530363+0000","last_undegraded":"2023-10-30T13:11:15.530363+0000","last_fullsized":"2023-10-30T13:11:15.530363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:58.719747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440825+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:11:15.440825+0000","last_peered":"2023-10-30T13:11:15.440825+0000","last_clean":"2023-10-30T13:11:15.440825+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:11:15.440825+0000","last_undegraded":"2023-10-30T13:11:15.440825+0000","last_fullsized":"2023-10-30T13:11:15.440825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530665+0000","last_change":"2023-10-30T13:10:34.375164+0000","last_active":"2023-10-30T13:11:15.530665+0000","last_peered":"2023-10-30T13:11:15.530665+0000","last_clean":"2023-10-30T13:11:15.530665+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:11:15.530665+0000","last_undegraded":"2023-10-30T13:11:15.530665+0000","last_fullsized":"2023-10-30T13:11:15.530665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:05.305348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433212+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:11:15.433212+0000","last_peered":"2023-10-30T13:11:15.433212+0000","last_clean":"2023-10-30T13:11:15.433212+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:11:15.433212+0000","last_undegraded":"2023-10-30T13:11:15.433212+0000","last_fullsized":"2023-10-30T13:11:15.433212+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531192+0000","last_change":"2023-10-30T13:10:34.382785+0000","last_active":"2023-10-30T13:11:15.531192+0000","last_peered":"2023-10-30T13:11:15.531192+0000","last_clean":"2023-10-30T13:11:15.531192+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:11:15.531192+0000","last_undegraded":"2023-10-30T13:11:15.531192+0000","last_fullsized":"2023-10-30T13:11:15.531192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:50.203770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434778+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:11:15.434778+0000","last_peered":"2023-10-30T13:11:15.434778+0000","last_clean":"2023-10-30T13:11:15.434778+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:11:15.434778+0000","last_undegraded":"2023-10-30T13:11:15.434778+0000","last_fullsized":"2023-10-30T13:11:15.434778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.439160+0000","last_change":"2023-10-30T13:10:34.342901+0000","last_active":"2023-10-30T13:11:15.439160+0000","last_peered":"2023-10-30T13:11:15.439160+0000","last_clean":"2023-10-30T13:11:15.439160+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:11:15.439160+0000","last_undegraded":"2023-10-30T13:11:15.439160+0000","last_fullsized":"2023-10-30T13:11:15.439160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:27.227907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433265+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:11:15.433265+0000","last_peered":"2023-10-30T13:11:15.433265+0000","last_clean":"2023-10-30T13:11:15.433265+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:11:15.433265+0000","last_undegraded":"2023-10-30T13:11:15.433265+0000","last_fullsized":"2023-10-30T13:11:15.433265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530590+0000","last_change":"2023-10-30T13:10:34.373854+0000","last_active":"2023-10-30T13:11:15.530590+0000","last_peered":"2023-10-30T13:11:15.530590+0000","last_clean":"2023-10-30T13:11:15.530590+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:11:15.530590+0000","last_undegraded":"2023-10-30T13:11:15.530590+0000","last_fullsized":"2023-10-30T13:11:15.530590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:12.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530831+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:11:15.530831+0000","last_peered":"2023-10-30T13:11:15.530831+0000","last_clean":"2023-10-30T13:11:15.530831+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:11:15.530831+0000","last_undegraded":"2023-10-30T13:11:15.530831+0000","last_fullsized":"2023-10-30T13:11:15.530831+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438091+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:11:15.438091+0000","last_peered":"2023-10-30T13:11:15.438091+0000","last_clean":"2023-10-30T13:11:15.438091+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:11:15.438091+0000","last_undegraded":"2023-10-30T13:11:15.438091+0000","last_fullsized":"2023-10-30T13:11:15.438091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430506+0000","last_change":"2023-10-30T13:10:34.348470+0000","last_active":"2023-10-30T13:11:15.430506+0000","last_peered":"2023-10-30T13:11:15.430506+0000","last_clean":"2023-10-30T13:11:15.430506+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:11:15.430506+0000","last_undegraded":"2023-10-30T13:11:15.430506+0000","last_fullsized":"2023-10-30T13:11:15.430506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:24.818359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530008+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:11:15.530008+0000","last_peered":"2023-10-30T13:11:15.530008+0000","last_clean":"2023-10-30T13:11:15.530008+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:11:15.530008+0000","last_undegraded":"2023-10-30T13:11:15.530008+0000","last_fullsized":"2023-10-30T13:11:15.530008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560701+0000","last_change":"2023-10-30T13:10:34.359740+0000","last_active":"2023-10-30T13:11:15.560701+0000","last_peered":"2023-10-30T13:11:15.560701+0000","last_clean":"2023-10-30T13:11:15.560701+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:11:15.560701+0000","last_undegraded":"2023-10-30T13:11:15.560701+0000","last_fullsized":"2023-10-30T13:11:15.560701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:05:25.883485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:12:42.307293+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:12:42.307293+0000","last_peered":"2023-10-30T13:12:42.307293+0000","last_clean":"2023-10-30T13:12:42.307293+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:12:42.307293+0000","last_undegraded":"2023-10-30T13:12:42.307293+0000","last_fullsized":"2023-10-30T13:12:42.307293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433030+0000","last_change":"2023-10-30T13:10:34.332955+0000","last_active":"2023-10-30T13:11:15.433030+0000","last_peered":"2023-10-30T13:11:15.433030+0000","last_clean":"2023-10-30T13:11:15.433030+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:11:15.433030+0000","last_undegraded":"2023-10-30T13:11:15.433030+0000","last_fullsized":"2023-10-30T13:11:15.433030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:05.776190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438730+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:11:15.438730+0000","last_peered":"2023-10-30T13:11:15.438730+0000","last_clean":"2023-10-30T13:11:15.438730+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:11:15.438730+0000","last_undegraded":"2023-10-30T13:11:15.438730+0000","last_fullsized":"2023-10-30T13:11:15.438730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440305+0000","last_change":"2023-10-30T13:10:34.362313+0000","last_active":"2023-10-30T13:11:15.440305+0000","last_peered":"2023-10-30T13:11:15.440305+0000","last_clean":"2023-10-30T13:11:15.440305+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:11:15.440305+0000","last_undegraded":"2023-10-30T13:11:15.440305+0000","last_fullsized":"2023-10-30T13:11:15.440305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:45.213916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434563+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:11:15.434563+0000","last_peered":"2023-10-30T13:11:15.434563+0000","last_clean":"2023-10-30T13:11:15.434563+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:11:15.434563+0000","last_undegraded":"2023-10-30T13:11:15.434563+0000","last_fullsized":"2023-10-30T13:11:15.434563+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.439222+0000","last_change":"2023-10-30T13:10:34.348653+0000","last_active":"2023-10-30T13:11:15.439222+0000","last_peered":"2023-10-30T13:11:15.439222+0000","last_clean":"2023-10-30T13:11:15.439222+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:11:15.439222+0000","last_undegraded":"2023-10-30T13:11:15.439222+0000","last_fullsized":"2023-10-30T13:11:15.439222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:54:00.948062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531132+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:11:15.531132+0000","last_peered":"2023-10-30T13:11:15.531132+0000","last_clean":"2023-10-30T13:11:15.531132+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:11:15.531132+0000","last_undegraded":"2023-10-30T13:11:15.531132+0000","last_fullsized":"2023-10-30T13:11:15.531132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433306+0000","last_change":"2023-10-30T13:10:34.332001+0000","last_active":"2023-10-30T13:11:15.433306+0000","last_peered":"2023-10-30T13:11:15.433306+0000","last_clean":"2023-10-30T13:11:15.433306+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:11:15.433306+0000","last_undegraded":"2023-10-30T13:11:15.433306+0000","last_fullsized":"2023-10-30T13:11:15.433306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:16.341054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434819+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:11:15.434819+0000","last_peered":"2023-10-30T13:11:15.434819+0000","last_clean":"2023-10-30T13:11:15.434819+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:11:15.434819+0000","last_undegraded":"2023-10-30T13:11:15.434819+0000","last_fullsized":"2023-10-30T13:11:15.434819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430358+0000","last_change":"2023-10-30T13:10:34.369845+0000","last_active":"2023-10-30T13:11:15.430358+0000","last_peered":"2023-10-30T13:11:15.430358+0000","last_clean":"2023-10-30T13:11:15.430358+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:11:15.430358+0000","last_undegraded":"2023-10-30T13:11:15.430358+0000","last_fullsized":"2023-10-30T13:11:15.430358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:30.468845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.447484+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:11:15.447484+0000","last_peered":"2023-10-30T13:11:15.447484+0000","last_clean":"2023-10-30T13:11:15.447484+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:11:15.447484+0000","last_undegraded":"2023-10-30T13:11:15.447484+0000","last_fullsized":"2023-10-30T13:11:15.447484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561059+0000","last_change":"2023-10-30T13:10:34.323336+0000","last_active":"2023-10-30T13:11:15.561059+0000","last_peered":"2023-10-30T13:11:15.561059+0000","last_clean":"2023-10-30T13:11:15.561059+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:11:15.561059+0000","last_undegraded":"2023-10-30T13:11:15.561059+0000","last_fullsized":"2023-10-30T13:11:15.561059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:21.642241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.529931+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:11:15.529931+0000","last_peered":"2023-10-30T13:11:15.529931+0000","last_clean":"2023-10-30T13:11:15.529931+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:11:15.529931+0000","last_undegraded":"2023-10-30T13:11:15.529931+0000","last_fullsized":"2023-10-30T13:11:15.529931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440140+0000","last_change":"2023-10-30T13:10:34.363313+0000","last_active":"2023-10-30T13:11:15.440140+0000","last_peered":"2023-10-30T13:11:15.440140+0000","last_clean":"2023-10-30T13:11:15.440140+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:11:15.440140+0000","last_undegraded":"2023-10-30T13:11:15.440140+0000","last_fullsized":"2023-10-30T13:11:15.440140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:13.027610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433415+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:11:15.433415+0000","last_peered":"2023-10-30T13:11:15.433415+0000","last_clean":"2023-10-30T13:11:15.433415+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:11:15.433415+0000","last_undegraded":"2023-10-30T13:11:15.433415+0000","last_fullsized":"2023-10-30T13:11:15.433415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430057+0000","last_change":"2023-10-30T13:10:34.330677+0000","last_active":"2023-10-30T13:11:15.430057+0000","last_peered":"2023-10-30T13:11:15.430057+0000","last_clean":"2023-10-30T13:11:15.430057+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:11:15.430057+0000","last_undegraded":"2023-10-30T13:11:15.430057+0000","last_fullsized":"2023-10-30T13:11:15.430057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:01.360813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433573+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:11:15.433573+0000","last_peered":"2023-10-30T13:11:15.433573+0000","last_clean":"2023-10-30T13:11:15.433573+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:11:15.433573+0000","last_undegraded":"2023-10-30T13:11:15.433573+0000","last_fullsized":"2023-10-30T13:11:15.433573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430362+0000","last_change":"2023-10-30T13:10:34.331117+0000","last_active":"2023-10-30T13:11:15.430362+0000","last_peered":"2023-10-30T13:11:15.430362+0000","last_clean":"2023-10-30T13:11:15.430362+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:11:15.430362+0000","last_undegraded":"2023-10-30T13:11:15.430362+0000","last_fullsized":"2023-10-30T13:11:15.430362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:56.948470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531341+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:11:15.531341+0000","last_peered":"2023-10-30T13:11:15.531341+0000","last_clean":"2023-10-30T13:11:15.531341+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:11:15.531341+0000","last_undegraded":"2023-10-30T13:11:15.531341+0000","last_fullsized":"2023-10-30T13:11:15.531341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446526+0000","last_change":"2023-10-30T13:10:34.365315+0000","last_active":"2023-10-30T13:11:15.446526+0000","last_peered":"2023-10-30T13:11:15.446526+0000","last_clean":"2023-10-30T13:11:15.446526+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:11:15.446526+0000","last_undegraded":"2023-10-30T13:11:15.446526+0000","last_fullsized":"2023-10-30T13:11:15.446526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:37.078775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.442500+0000","last_change":"2023-10-30T13:10:34.366044+0000","last_active":"2023-10-30T13:11:14.442500+0000","last_peered":"2023-10-30T13:11:14.442500+0000","last_clean":"2023-10-30T13:11:14.442500+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:11:14.442500+0000","last_undegraded":"2023-10-30T13:11:14.442500+0000","last_fullsized":"2023-10-30T13:11:14.442500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:31.831020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.442338+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:11:14.442338+0000","last_peered":"2023-10-30T13:11:14.442338+0000","last_clean":"2023-10-30T13:11:14.442338+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:11:14.442338+0000","last_undegraded":"2023-10-30T13:11:14.442338+0000","last_fullsized":"2023-10-30T13:11:14.442338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.436137+0000","last_change":"2023-10-30T13:10:34.369814+0000","last_active":"2023-10-30T13:11:14.436137+0000","last_peered":"2023-10-30T13:11:14.436137+0000","last_clean":"2023-10-30T13:11:14.436137+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:11:14.436137+0000","last_undegraded":"2023-10-30T13:11:14.436137+0000","last_fullsized":"2023-10-30T13:11:14.436137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:26.879717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441916+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:11:14.441916+0000","last_peered":"2023-10-30T13:11:14.441916+0000","last_clean":"2023-10-30T13:11:14.441916+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:11:14.441916+0000","last_undegraded":"2023-10-30T13:11:14.441916+0000","last_fullsized":"2023-10-30T13:11:14.441916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561102+0000","last_change":"2023-10-30T13:10:34.344766+0000","last_active":"2023-10-30T13:11:15.561102+0000","last_peered":"2023-10-30T13:11:15.561102+0000","last_clean":"2023-10-30T13:11:15.561102+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:11:15.561102+0000","last_undegraded":"2023-10-30T13:11:15.561102+0000","last_fullsized":"2023-10-30T13:11:15.561102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.073346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430538+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:11:15.430538+0000","last_peered":"2023-10-30T13:11:15.430538+0000","last_clean":"2023-10-30T13:11:15.430538+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:11:15.430538+0000","last_undegraded":"2023-10-30T13:11:15.430538+0000","last_fullsized":"2023-10-30T13:11:15.430538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430467+0000","last_change":"2023-10-30T13:10:34.354305+0000","last_active":"2023-10-30T13:11:15.430467+0000","last_peered":"2023-10-30T13:11:15.430467+0000","last_clean":"2023-10-30T13:11:15.430467+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:11:15.430467+0000","last_undegraded":"2023-10-30T13:11:15.430467+0000","last_fullsized":"2023-10-30T13:11:15.430467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:53.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'9","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561228+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:11:15.561228+0000","last_peered":"2023-10-30T13:11:15.561228+0000","last_clean":"2023-10-30T13:11:15.561228+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:11:15.561228+0000","last_undegraded":"2023-10-30T13:11:15.561228+0000","last_fullsized":"2023-10-30T13:11:15.561228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":843776,"data_stored":675004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.81999999999999995,"15min":0.81399999999999995},"min":{"1min":0.57099999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.159,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85899999999999999,"15min":0.84899999999999998},"min":{"1min":0.54000000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.284,"5min":1.379,"15min":1.379},"last":1.071}]},{"osd":1,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.94299999999999995,"15min":0.95499999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.3879999999999999,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.97699999999999998,"15min":0.997},"min":{"1min":0.64100000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.052,"5min":4.5229999999999997,"15min":4.5229999999999997},"last":1.0369999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.87,"15min":0.88400000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.0369999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.111},{"interface":"front","average":{"1min":0.995,"5min":1.0149999999999999,"15min":1.018},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.306,"5min":1.359,"15min":1.359},"last":0.79800000000000004}]},{"osd":3,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.98199999999999998,"15min":0.98799999999999999},"min":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1499999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.075,"15min":1.083},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.2529999999999999,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":1.1499999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.085,"15min":1.1120000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.306,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.02,"5min":1.071,"15min":1.079},"min":{"1min":0.67500000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3169999999999999,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.1870000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3049999999999999,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.22},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.2030000000000001,"15min":1.2350000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.208,"5min":4.9269999999999996,"15min":4.9269999999999996},"last":0.96599999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.153,"15min":1.1870000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2350000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.3160000000000001,"15min":1.3640000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.2410000000000001,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.83499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":85610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.0980000000000001,"15min":1.1399999999999999},"min":{"1min":0.61499999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0429999999999999,"5min":4.1699999999999999,"15min":4.1699999999999999},"last":0.62},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.93799999999999994,"15min":0.94599999999999995},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0860000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.94899999999999995}]},{"osd":1,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.625,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1279999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.92800000000000005,"15min":0.93100000000000005},"min":{"1min":0.67300000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.177,"5min":1.458,"15min":1.458},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.1870000000000001,"15min":1.2270000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1519999999999999,"5min":5.4269999999999996,"15min":5.4269999999999996},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.034,"15min":1.048},"min":{"1min":0.63600000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.258,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.82699999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0269999999999999,"15min":1.044},"min":{"1min":0.629,"5min":0.622,"15min":0.622},"max":{"1min":1.181,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":1.052},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.92200000000000004,"15min":0.93200000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.093,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.76900000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.008,"15min":1.014},"min":{"1min":0.65700000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.359,"5min":1.546,"15min":1.546},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.044,"15min":1.0429999999999999},"min":{"1min":0.69999999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.67800000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.024,"15min":1.026},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.1970000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.167,"15min":1.177},"min":{"1min":0.78500000000000003,"5min":0.749,"15min":0.749},"max":{"1min":1.423,"5min":1.653,"15min":1.653},"last":1.0760000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.177,"15min":1.179},"min":{"1min":0.83899999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.091,"5min":1.1399999999999999,"15min":1.1479999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.46,"5min":1.704,"15min":1.704},"last":1.127}]}]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":84698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.012,"5min":1.155,"15min":1.155},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.80000000000000004,"15min":0.78400000000000003},"min":{"1min":0.59899999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.079}]},{"osd":1,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.82299999999999995,"15min":0.81100000000000005},"min":{"1min":0.58799999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0820000000000001,"5min":1.24,"15min":1.24},"last":1.0600000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.64400000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.157,"5min":1.258,"15min":1.258},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.95699999999999996,"15min":0.96699999999999997},"min":{"1min":0.67900000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1699999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.03}]},{"osd":3,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.73899999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.1739999999999999,"5min":1.218,"15min":1.218},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.028,"15min":1.03},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.2849999999999999,"5min":1.601,"15min":1.601},"last":1.1479999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94999999999999996,"15min":0.96999999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.097,"5min":1.379,"15min":1.379},"last":0.751},{"interface":"front","average":{"1min":1.147,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.3320000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.24}]},{"osd":6,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.131,"15min":1.147},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.194,"5min":1.875,"15min":1.875},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.042,"15min":1.0580000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.2230000000000001,"5min":1.554,"15min":1.554},"last":1.117}]},{"osd":7,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.083,"15min":1.083},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.3480000000000001,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.171},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.175,"15min":1.1850000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.401,"5min":1.821,"15min":1.821},"last":1.2769999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.91600000000000004,"15min":0.92400000000000004},"min":{"1min":0.53900000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2609999999999999,"5min":3.6469999999999998,"15min":3.6469999999999998},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.90000000000000002,"15min":0.89200000000000002},"min":{"1min":0.54600000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.351,"5min":1.669,"15min":1.669},"last":0.55500000000000005}]},{"osd":1,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.2330000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.90100000000000002,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.242,"5min":1.413,"15min":1.413},"last":0.83199999999999996}]},{"osd":2,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93400000000000005,"15min":0.94199999999999995},"min":{"1min":0.59499999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.302,"5min":3.6840000000000002,"15min":3.6840000000000002},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93000000000000005,"15min":0.92200000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.90000000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.95099999999999996,"15min":0.93899999999999995},"min":{"1min":0.69699999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2749999999999999,"5min":1.321,"15min":1.321},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.88200000000000001,"15min":0.877},"min":{"1min":0.58599999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.145,"5min":1.46,"15min":1.46},"last":1}]},{"osd":5,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.004,"15min":1.006},"min":{"1min":0.64500000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3620000000000001,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0229999999999999,"15min":1.02},"min":{"1min":0.72599999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.3220000000000001,"5min":1.514,"15min":1.514},"last":1.1379999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.92900000000000005,"15min":0.92300000000000004},"min":{"1min":0.70799999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.091},{"interface":"front","average":{"1min":1.024,"5min":1.1000000000000001,"15min":1.1120000000000001},"min":{"1min":0.749,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.2789999999999999,"5min":2.0379999999999998,"15min":2.0379999999999998},"last":1.0680000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.72399999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.569,"5min":1.744,"15min":1.744},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.6000000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.0429999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":86717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.59799999999999998,"15min":0.61899999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65200000000000002,"5min":1.923,"15min":1.923},"last":0.308},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70599999999999996,"5min":2.085,"15min":2.085},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.65900000000000003,"15min":0.68000000000000005},"min":{"1min":0.33500000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73799999999999999,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.38100000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.754,"15min":0.78800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.751,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63800000000000001,"15min":0.64600000000000002},"min":{"1min":0.41999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79400000000000004,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91200000000000003,"15min":0.92300000000000004},"min":{"1min":0.72699999999999998,"5min":0.627,"15min":0.627},"max":{"1min":1.0720000000000001,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.876,"5min":0.95999999999999996,"15min":0.97399999999999998},"min":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.0569999999999999,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.996,"15min":1.026},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.98699999999999999,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.97899999999999998,"15min":0.98499999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1439999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.96499999999999997,"15min":0.98299999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.111,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":0.623},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.97299999999999998,"15min":0.98799999999999999},"min":{"1min":0.66200000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.149,"5min":1.839,"15min":1.839},"last":0.63700000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.0009999999999999,"15min":1.0169999999999999},"min":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.1160000000000001,"5min":2.117,"15min":2.117},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.026,"15min":1.048},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.1100000000000001,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":0.65000000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":84138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.33400000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.436},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.38500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58399999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.48899999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.34999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67200000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.755,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.80900000000000005,"15min":0.79900000000000004},"min":{"1min":0.70099999999999996,"5min":0.629,"15min":0.629},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.81999999999999995,"15min":0.81799999999999995},"min":{"1min":0.67500000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0469999999999999,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.78500000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85399999999999998},"min":{"1min":0.68200000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":0.98599999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.90900000000000003,"15min":0.91900000000000004},"min":{"1min":0.68300000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.1220000000000001,"5min":1.339,"15min":1.339},"last":0.68300000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.84499999999999997,"15min":0.84199999999999997},"min":{"1min":0.68300000000000005,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.0449999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.628},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.93000000000000005,"15min":0.94499999999999995},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":0.95399999999999996,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.82599999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98999999999999999,"15min":1},"min":{"1min":0.74099999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.091,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.94699999999999995,"15min":0.95699999999999996},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.075,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.70599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":84264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72999999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.502},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.60799999999999998,"5min":1.006,"15min":1.006},"last":0.59999999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63300000000000001,"5min":1.095,"15min":1.095},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.39400000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.63,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.69399999999999995,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.39800000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67800000000000005,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.42099999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.63300000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.0800000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.94499999999999995,"15min":0.96899999999999997},"min":{"1min":0.67300000000000004,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.03,"5min":4.093,"15min":4.093},"last":0.94899999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85099999999999998,"15min":0.85399999999999998},"min":{"1min":0.67800000000000005,"5min":0.629,"15min":0.629},"max":{"1min":1.0609999999999999,"5min":1.796,"15min":1.796},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.61399999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.012,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.86799999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.88600000000000001,"15min":0.89800000000000002},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.1919999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.89900000000000002,"15min":0.90300000000000002},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":0.997,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.88300000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":1,"15min":1.028},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1379999999999999,"5min":4.1470000000000002,"15min":4.1470000000000002},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.90700000000000003,"15min":0.91700000000000004},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.016,"5min":1.976,"15min":1.976},"last":0.80100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":835584,"data_stored":673329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.755,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51300000000000001,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.71199999999999997,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.497}]},{"osd":2,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73599999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.748,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.78000000000000003,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78900000000000003,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.629,"5min":0.62,"15min":0.62},"max":{"1min":1.0660000000000001,"5min":1.222,"15min":1.222},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.84599999999999997,"15min":0.85199999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.94499999999999995,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.94899999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86599999999999999,"15min":0.86899999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.044,"5min":1.306,"15min":1.306},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.873,"5min":0.90500000000000003,"15min":0.91000000000000003},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.1459999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.88,"15min":0.88600000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.0129999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.878,"5min":0.92000000000000004,"15min":0.92700000000000005},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.1319999999999999,"5min":1.446,"15min":1.446},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.89100000000000001,"15min":0.90300000000000002},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.109,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92700000000000005,"15min":0.93500000000000005},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0880000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.67200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:13:02.502 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:13:02.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a2d3e640 1 Processor -- start 2023-10-30T13:13:02.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a2d3e640 1 -- start start 2023-10-30T13:13:02.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a2d3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f649c1538a0 0x7f649c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a2d3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0cf160 0x7f649c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a2d3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0cb800 con 0x7f649c0ca0e0 2023-10-30T13:13:02.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a2d3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f649c15dc30 con 0x7f649c1538a0 2023-10-30T13:13:02.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a2d3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0cb330 con 0x7f649c0cf160 2023-10-30T13:13:02.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a12db640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0cf160 0x7f649c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a0ada640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f649c1538a0 0x7f649c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a12db640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0cf160 0x7f649c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:56624/0 (socket says 172.21.15.148:56624) 2023-10-30T13:13:02.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a12db640 1 -- 172.21.15.148:0/3189953072 learned_addr learned my addr 172.21.15.148:0/3189953072 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:02.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a0ada640 1 -- 172.21.15.148:0/3189953072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0cf160 msgr2=0x7f649c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:02.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a0ada640 1 --2- 172.21.15.148:0/3189953072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0cf160 0x7f649c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a0ada640 1 -- 172.21.15.148:0/3189953072 >> v1:172.21.15.150:6789/0 conn(0x7f649c0ca0e0 legacy=0x7f649c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.677+0000 7f64a0ada640 1 -- 172.21.15.148:0/3189953072 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0cad70 con 0x7f649c1538a0 2023-10-30T13:13:02.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64a0ada640 1 --2- 172.21.15.148:0/3189953072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f649c1538a0 0x7f649c15d6f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f64980097c0 tx=0x7f649801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=433630aaa8929c53 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:02.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64937fe640 1 -- 172.21.15.148:0/3189953072 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64980a8970 con 0x7f649c1538a0 2023-10-30T13:13:02.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64937fe640 1 -- 172.21.15.148:0/3189953072 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64980a8ad0 con 0x7f649c1538a0 2023-10-30T13:13:02.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64937fe640 1 -- 172.21.15.148:0/3189953072 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64980a8e60 con 0x7f649c1538a0 2023-10-30T13:13:02.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3189953072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f649c1538a0 msgr2=0x7f649c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64a2d3e640 1 --2- 172.21.15.148:0/3189953072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f649c1538a0 0x7f649c15d6f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f64980097c0 tx=0x7f649801ec70 comp rx=0 tx=0).stop 2023-10-30T13:13:02.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3189953072 shutdown_connections 2023-10-30T13:13:02.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64a2d3e640 1 --2- 172.21.15.148:0/3189953072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0cf160 0x7f649c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64a2d3e640 1 --2- 172.21.15.148:0/3189953072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f649c1538a0 0x7f649c15d6f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3189953072 >> 172.21.15.148:0/3189953072 conn(0x7f649c06bfe0 msgr2=0x7f649c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:02.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3189953072 shutdown_connections 2023-10-30T13:13:02.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.681+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3189953072 wait complete. 2023-10-30T13:13:02.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 Processor -- start 2023-10-30T13:13:02.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 -- start start 2023-10-30T13:13:02.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0ca0e0 0x7f649c1e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f649c0cf160 0x7f649c15b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f649c1538a0 0x7f649c15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c14d4f0 con 0x7f649c1538a0 2023-10-30T13:13:02.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c1e4d40 con 0x7f649c0cf160 2023-10-30T13:13:02.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649c15b2a0 con 0x7f649c0ca0e0 2023-10-30T13:13:02.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a12db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f649c1538a0 0x7f649c15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a12db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f649c1538a0 0x7f649c15bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38350/0 (socket says 172.21.15.148:38350) 2023-10-30T13:13:02.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a12db640 1 -- 172.21.15.148:0/3197130834 learned_addr learned my addr 172.21.15.148:0/3197130834 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:02.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a0ada640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0ca0e0 0x7f649c1e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f6493fff640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f649c0cf160 0x7f649c15b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a12db640 1 -- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0ca0e0 msgr2=0x7f649c1e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a12db640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0ca0e0 0x7f649c1e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a12db640 1 -- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f649c0cf160 msgr2=0x7f649c15b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a12db640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f649c0cf160 0x7f649c15b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a12db640 1 -- 172.21.15.148:0/3197130834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6498009420 con 0x7f649c1538a0 2023-10-30T13:13:02.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a12db640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f649c1538a0 0x7f649c15bb70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f649400efc0 tx=0x7f649400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:02.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64917fa640 1 -- 172.21.15.148:0/3197130834 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649400cd20 con 0x7f649c1538a0 2023-10-30T13:13:02.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64917fa640 1 -- 172.21.15.148:0/3197130834 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649400f040 con 0x7f649c1538a0 2023-10-30T13:13:02.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64917fa640 1 -- 172.21.15.148:0/3197130834 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494003bc0 con 0x7f649c1538a0 2023-10-30T13:13:02.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c0cef10 con 0x7f649c1538a0 2023-10-30T13:13:02.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.685+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c0c50b0 con 0x7f649c1538a0 2023-10-30T13:13:02.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.689+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c0b2a50 con 0x7f649c1538a0 2023-10-30T13:13:02.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.689+0000 7f64917fa640 1 -- 172.21.15.148:0/3197130834 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6494003d20 con 0x7f649c1538a0 2023-10-30T13:13:02.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.689+0000 7f64917fa640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64880664a0 0x7f6488068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:02.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.689+0000 7f64917fa640 1 -- 172.21.15.148:0/3197130834 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6494084310 con 0x7f649c1538a0 2023-10-30T13:13:02.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.689+0000 7f64a0ada640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64880664a0 0x7f6488068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:02.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.693+0000 7f64a0ada640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64880664a0 0x7f6488068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f64980b2000 tx=0x7f6498008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:02.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.693+0000 7f64917fa640 1 -- 172.21.15.148:0/3197130834 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64940517b0 con 0x7f649c1538a0 2023-10-30T13:13:02.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.841+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f649c0cb050 con 0x7f64880664a0 2023-10-30T13:13:02.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.861+0000 7f64917fa640 1 -- 172.21.15.148:0/3197130834 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288228 (secure 0 0 0) 0x7f64980b0440 con 0x7f64880664a0 2023-10-30T13:13:02.867 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:13:02.868 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:13:02.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64880664a0 msgr2=0x7f6488068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64880664a0 0x7f6488068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f64980b2000 tx=0x7f6498008040 comp rx=0 tx=0).stop 2023-10-30T13:13:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f649c1538a0 msgr2=0x7f649c15bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f649c1538a0 0x7f649c15bb70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f649400efc0 tx=0x7f649400c490 comp rx=0 tx=0).stop 2023-10-30T13:13:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 shutdown_connections 2023-10-30T13:13:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64880664a0 0x7f6488068960 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f649c1538a0 0x7f649c15bb70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f649c0cf160 0x7f649c15b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 --2- 172.21.15.148:0/3197130834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f649c0ca0e0 0x7f649c1e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 >> 172.21.15.148:0/3197130834 conn(0x7f649c06bfe0 msgr2=0x7f649c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 shutdown_connections 2023-10-30T13:13:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:02.865+0000 7f64a2d3e640 1 -- 172.21.15.148:0/3197130834 wait complete. 2023-10-30T13:13:02.888 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2023-10-30T13:13:02.314085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":285,"num_write_kb":2455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219284,"kb_used_data":3108,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755436,"statfs":{"total":773094113280,"available":772869566464,"internally_reserved":0,"allocated":3182592,"data_stored":1857794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003910"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530410+0000","last_change":"2023-10-30T13:10:34.379031+0000","last_active":"2023-10-30T13:11:15.530410+0000","last_peered":"2023-10-30T13:11:15.530410+0000","last_clean":"2023-10-30T13:11:15.530410+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:11:15.530410+0000","last_undegraded":"2023-10-30T13:11:15.530410+0000","last_fullsized":"2023-10-30T13:11:15.530410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:32.529312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434689+0000","last_change":"2023-10-30T13:10:34.324308+0000","last_active":"2023-10-30T13:11:15.434689+0000","last_peered":"2023-10-30T13:11:15.434689+0000","last_clean":"2023-10-30T13:11:15.434689+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:11:15.434689+0000","last_undegraded":"2023-10-30T13:11:15.434689+0000","last_fullsized":"2023-10-30T13:11:15.434689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:21.885716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561039+0000","last_change":"2023-10-30T13:10:34.334122+0000","last_active":"2023-10-30T13:11:15.561039+0000","last_peered":"2023-10-30T13:11:15.561039+0000","last_clean":"2023-10-30T13:11:15.561039+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:11:15.561039+0000","last_undegraded":"2023-10-30T13:11:15.561039+0000","last_fullsized":"2023-10-30T13:11:15.561039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:55.045024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437588+0000","last_change":"2023-10-30T13:11:15.435943+0000","last_active":"2023-10-30T13:11:15.437588+0000","last_peered":"2023-10-30T13:11:15.437588+0000","last_clean":"2023-10-30T13:11:15.437588+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:11:15.437588+0000","last_undegraded":"2023-10-30T13:11:15.437588+0000","last_fullsized":"2023-10-30T13:11:15.437588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:03.152688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441733+0000","last_change":"2023-10-30T13:10:34.371670+0000","last_active":"2023-10-30T13:11:14.441733+0000","last_peered":"2023-10-30T13:11:14.441733+0000","last_clean":"2023-10-30T13:11:14.441733+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:11:14.441733+0000","last_undegraded":"2023-10-30T13:11:14.441733+0000","last_fullsized":"2023-10-30T13:11:14.441733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:03:54.773965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441333+0000","last_change":"2023-10-30T13:10:34.335752+0000","last_active":"2023-10-30T13:11:14.441333+0000","last_peered":"2023-10-30T13:11:14.441333+0000","last_clean":"2023-10-30T13:11:14.441333+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:11:14.441333+0000","last_undegraded":"2023-10-30T13:11:14.441333+0000","last_fullsized":"2023-10-30T13:11:14.441333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:33.278920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.440809+0000","last_change":"2023-10-30T13:10:34.371817+0000","last_active":"2023-10-30T13:11:14.440809+0000","last_peered":"2023-10-30T13:11:14.440809+0000","last_clean":"2023-10-30T13:11:14.440809+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:11:14.440809+0000","last_undegraded":"2023-10-30T13:11:14.440809+0000","last_fullsized":"2023-10-30T13:11:14.440809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:33.797113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.427171+0000","last_change":"2023-10-30T13:10:34.348205+0000","last_active":"2023-10-30T13:11:14.427171+0000","last_peered":"2023-10-30T13:11:14.427171+0000","last_clean":"2023-10-30T13:11:14.427171+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:11:14.427171+0000","last_undegraded":"2023-10-30T13:11:14.427171+0000","last_fullsized":"2023-10-30T13:11:14.427171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:57.492738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.442942+0000","last_change":"2023-10-30T13:10:34.381052+0000","last_active":"2023-10-30T13:11:14.442942+0000","last_peered":"2023-10-30T13:11:14.442942+0000","last_clean":"2023-10-30T13:11:14.442942+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:11:14.442942+0000","last_undegraded":"2023-10-30T13:11:14.442942+0000","last_fullsized":"2023-10-30T13:11:14.442942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:16.697332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.447936+0000","last_change":"2023-10-30T13:10:34.356932+0000","last_active":"2023-10-30T13:11:14.447936+0000","last_peered":"2023-10-30T13:11:14.447936+0000","last_clean":"2023-10-30T13:11:14.447936+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:11:14.447936+0000","last_undegraded":"2023-10-30T13:11:14.447936+0000","last_fullsized":"2023-10-30T13:11:14.447936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:54.091993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.447369+0000","last_change":"2023-10-30T13:11:15.447369+0000","last_active":"2023-10-30T13:11:15.447369+0000","last_peered":"2023-10-30T13:11:15.447369+0000","last_clean":"2023-10-30T13:11:15.447369+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:11:15.447369+0000","last_undegraded":"2023-10-30T13:11:15.447369+0000","last_fullsized":"2023-10-30T13:11:15.447369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:32.683227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.439648+0000","last_change":"2023-10-30T13:10:34.365161+0000","last_active":"2023-10-30T13:11:14.439648+0000","last_peered":"2023-10-30T13:11:14.439648+0000","last_clean":"2023-10-30T13:11:14.439648+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:11:14.439648+0000","last_undegraded":"2023-10-30T13:11:14.439648+0000","last_fullsized":"2023-10-30T13:11:14.439648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:51.721060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430223+0000","last_change":"2023-10-30T13:10:34.351901+0000","last_active":"2023-10-30T13:11:15.430223+0000","last_peered":"2023-10-30T13:11:15.430223+0000","last_clean":"2023-10-30T13:11:15.430223+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:11:15.430223+0000","last_undegraded":"2023-10-30T13:11:15.430223+0000","last_fullsized":"2023-10-30T13:11:15.430223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:11.576546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560710+0000","last_change":"2023-10-30T13:10:34.323358+0000","last_active":"2023-10-30T13:11:15.560710+0000","last_peered":"2023-10-30T13:11:15.560710+0000","last_clean":"2023-10-30T13:11:15.560710+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:11:15.560710+0000","last_undegraded":"2023-10-30T13:11:15.560710+0000","last_fullsized":"2023-10-30T13:11:15.560710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:03.178822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433306+0000","last_change":"2023-10-30T13:10:34.360120+0000","last_active":"2023-10-30T13:11:15.433306+0000","last_peered":"2023-10-30T13:11:15.433306+0000","last_clean":"2023-10-30T13:11:15.433306+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:11:15.433306+0000","last_undegraded":"2023-10-30T13:11:15.433306+0000","last_fullsized":"2023-10-30T13:11:15.433306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:30.731746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437472+0000","last_change":"2023-10-30T13:10:34.333752+0000","last_active":"2023-10-30T13:11:15.437472+0000","last_peered":"2023-10-30T13:11:15.437472+0000","last_clean":"2023-10-30T13:11:15.437472+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:11:15.437472+0000","last_undegraded":"2023-10-30T13:11:15.437472+0000","last_fullsized":"2023-10-30T13:11:15.437472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:45.390603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.441064+0000","last_change":"2023-10-30T13:10:34.371931+0000","last_active":"2023-10-30T13:11:15.441064+0000","last_peered":"2023-10-30T13:11:15.441064+0000","last_clean":"2023-10-30T13:11:15.441064+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:11:15.441064+0000","last_undegraded":"2023-10-30T13:11:15.441064+0000","last_fullsized":"2023-10-30T13:11:15.441064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:51.514990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433093+0000","last_change":"2023-10-30T13:10:34.352747+0000","last_active":"2023-10-30T13:11:15.433093+0000","last_peered":"2023-10-30T13:11:15.433093+0000","last_clean":"2023-10-30T13:11:15.433093+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:11:15.433093+0000","last_undegraded":"2023-10-30T13:11:15.433093+0000","last_fullsized":"2023-10-30T13:11:15.433093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:21.543351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437590+0000","last_change":"2023-10-30T13:11:15.436291+0000","last_active":"2023-10-30T13:11:15.437590+0000","last_peered":"2023-10-30T13:11:15.437590+0000","last_clean":"2023-10-30T13:11:15.437590+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:11:15.437590+0000","last_undegraded":"2023-10-30T13:11:15.437590+0000","last_fullsized":"2023-10-30T13:11:15.437590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:22.745942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446501+0000","last_change":"2023-10-30T13:10:34.366575+0000","last_active":"2023-10-30T13:11:15.446501+0000","last_peered":"2023-10-30T13:11:15.446501+0000","last_clean":"2023-10-30T13:11:15.446501+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:11:15.446501+0000","last_undegraded":"2023-10-30T13:11:15.446501+0000","last_fullsized":"2023-10-30T13:11:15.446501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:23.540645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560863+0000","last_change":"2023-10-30T13:10:34.368761+0000","last_active":"2023-10-30T13:11:15.560863+0000","last_peered":"2023-10-30T13:11:15.560863+0000","last_clean":"2023-10-30T13:11:15.560863+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:11:15.560863+0000","last_undegraded":"2023-10-30T13:11:15.560863+0000","last_fullsized":"2023-10-30T13:11:15.560863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:32.861756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438076+0000","last_change":"2023-10-30T13:10:34.372881+0000","last_active":"2023-10-30T13:11:15.438076+0000","last_peered":"2023-10-30T13:11:15.438076+0000","last_clean":"2023-10-30T13:11:15.438076+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:11:15.438076+0000","last_undegraded":"2023-10-30T13:11:15.438076+0000","last_fullsized":"2023-10-30T13:11:15.438076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:23.855779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430039+0000","last_change":"2023-10-30T13:10:34.359359+0000","last_active":"2023-10-30T13:11:15.430039+0000","last_peered":"2023-10-30T13:11:15.430039+0000","last_clean":"2023-10-30T13:11:15.430039+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:11:15.430039+0000","last_undegraded":"2023-10-30T13:11:15.430039+0000","last_fullsized":"2023-10-30T13:11:15.430039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:08.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437617+0000","last_change":"2023-10-30T13:10:34.347431+0000","last_active":"2023-10-30T13:11:15.437617+0000","last_peered":"2023-10-30T13:11:15.437617+0000","last_clean":"2023-10-30T13:11:15.437617+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:11:15.437617+0000","last_undegraded":"2023-10-30T13:11:15.437617+0000","last_fullsized":"2023-10-30T13:11:15.437617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:25.094617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.442207+0000","last_change":"2023-10-30T13:11:15.440541+0000","last_active":"2023-10-30T13:11:15.442207+0000","last_peered":"2023-10-30T13:11:15.442207+0000","last_clean":"2023-10-30T13:11:15.442207+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:11:15.442207+0000","last_undegraded":"2023-10-30T13:11:15.442207+0000","last_fullsized":"2023-10-30T13:11:15.442207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:16.858272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.447245+0000","last_change":"2023-10-30T13:10:34.379878+0000","last_active":"2023-10-30T13:11:15.447245+0000","last_peered":"2023-10-30T13:11:15.447245+0000","last_clean":"2023-10-30T13:11:15.447245+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:11:15.447245+0000","last_undegraded":"2023-10-30T13:11:15.447245+0000","last_fullsized":"2023-10-30T13:11:15.447245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:42.280634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434611+0000","last_change":"2023-10-30T13:10:34.332020+0000","last_active":"2023-10-30T13:11:15.434611+0000","last_peered":"2023-10-30T13:11:15.434611+0000","last_clean":"2023-10-30T13:11:15.434611+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:11:15.434611+0000","last_undegraded":"2023-10-30T13:11:15.434611+0000","last_fullsized":"2023-10-30T13:11:15.434611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:53.057012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430366+0000","last_change":"2023-10-30T13:10:34.363154+0000","last_active":"2023-10-30T13:11:15.430366+0000","last_peered":"2023-10-30T13:11:15.430366+0000","last_clean":"2023-10-30T13:11:15.430366+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:11:15.430366+0000","last_undegraded":"2023-10-30T13:11:15.430366+0000","last_fullsized":"2023-10-30T13:11:15.430366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:04.359629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560893+0000","last_change":"2023-10-30T13:10:34.332160+0000","last_active":"2023-10-30T13:11:15.560893+0000","last_peered":"2023-10-30T13:11:15.560893+0000","last_clean":"2023-10-30T13:11:15.560893+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:11:15.560893+0000","last_undegraded":"2023-10-30T13:11:15.560893+0000","last_fullsized":"2023-10-30T13:11:15.560893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:28.128344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438275+0000","last_change":"2023-10-30T13:10:34.337617+0000","last_active":"2023-10-30T13:11:15.438275+0000","last_peered":"2023-10-30T13:11:15.438275+0000","last_clean":"2023-10-30T13:11:15.438275+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:11:15.438275+0000","last_undegraded":"2023-10-30T13:11:15.438275+0000","last_fullsized":"2023-10-30T13:11:15.438275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:10.287441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.432768+0000","last_change":"2023-10-30T13:10:34.359681+0000","last_active":"2023-10-30T13:11:15.432768+0000","last_peered":"2023-10-30T13:11:15.432768+0000","last_clean":"2023-10-30T13:11:15.432768+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:11:15.432768+0000","last_undegraded":"2023-10-30T13:11:15.432768+0000","last_fullsized":"2023-10-30T13:11:15.432768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:02.812352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.443436+0000","last_change":"2023-10-30T13:10:34.344864+0000","last_active":"2023-10-30T13:11:15.443436+0000","last_peered":"2023-10-30T13:11:15.443436+0000","last_clean":"2023-10-30T13:11:15.443436+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:11:15.443436+0000","last_undegraded":"2023-10-30T13:11:15.443436+0000","last_fullsized":"2023-10-30T13:11:15.443436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:13.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.439520+0000","last_change":"2023-10-30T13:10:34.364069+0000","last_active":"2023-10-30T13:11:14.439520+0000","last_peered":"2023-10-30T13:11:14.439520+0000","last_clean":"2023-10-30T13:11:14.439520+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:11:14.439520+0000","last_undegraded":"2023-10-30T13:11:14.439520+0000","last_fullsized":"2023-10-30T13:11:14.439520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:15.717336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.427598+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:11:14.427598+0000","last_peered":"2023-10-30T13:11:14.427598+0000","last_clean":"2023-10-30T13:11:14.427598+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:11:14.427598+0000","last_undegraded":"2023-10-30T13:11:14.427598+0000","last_fullsized":"2023-10-30T13:11:14.427598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.503227+0000","last_change":"2023-10-30T13:10:34.360935+0000","last_active":"2023-10-30T13:11:14.503227+0000","last_peered":"2023-10-30T13:11:14.503227+0000","last_clean":"2023-10-30T13:11:14.503227+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:11:14.503227+0000","last_undegraded":"2023-10-30T13:11:14.503227+0000","last_fullsized":"2023-10-30T13:11:14.503227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:58.639753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.447505+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:11:14.447505+0000","last_peered":"2023-10-30T13:11:14.447505+0000","last_clean":"2023-10-30T13:11:14.447505+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:11:14.447505+0000","last_undegraded":"2023-10-30T13:11:14.447505+0000","last_fullsized":"2023-10-30T13:11:14.447505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430030+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:11:15.430030+0000","last_peered":"2023-10-30T13:11:15.430030+0000","last_clean":"2023-10-30T13:11:15.430030+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:11:15.430030+0000","last_undegraded":"2023-10-30T13:11:15.430030+0000","last_fullsized":"2023-10-30T13:11:15.430030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430018+0000","last_change":"2023-10-30T13:10:34.329935+0000","last_active":"2023-10-30T13:11:15.430018+0000","last_peered":"2023-10-30T13:11:15.430018+0000","last_clean":"2023-10-30T13:11:15.430018+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:11:15.430018+0000","last_undegraded":"2023-10-30T13:11:15.430018+0000","last_fullsized":"2023-10-30T13:11:15.430018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:07.795163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.434429+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:11:14.434429+0000","last_peered":"2023-10-30T13:11:14.434429+0000","last_clean":"2023-10-30T13:11:14.434429+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:11:14.434429+0000","last_undegraded":"2023-10-30T13:11:14.434429+0000","last_fullsized":"2023-10-30T13:11:14.434429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433082+0000","last_change":"2023-10-30T13:10:34.331351+0000","last_active":"2023-10-30T13:11:15.433082+0000","last_peered":"2023-10-30T13:11:15.433082+0000","last_clean":"2023-10-30T13:11:15.433082+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:11:15.433082+0000","last_undegraded":"2023-10-30T13:11:15.433082+0000","last_fullsized":"2023-10-30T13:11:15.433082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:27.864073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.448058+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:11:14.448058+0000","last_peered":"2023-10-30T13:11:14.448058+0000","last_clean":"2023-10-30T13:11:14.448058+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:11:14.448058+0000","last_undegraded":"2023-10-30T13:11:14.448058+0000","last_fullsized":"2023-10-30T13:11:14.448058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531489+0000","last_change":"2023-10-30T13:11:15.530240+0000","last_active":"2023-10-30T13:11:15.531489+0000","last_peered":"2023-10-30T13:11:15.531489+0000","last_clean":"2023-10-30T13:11:15.531489+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:11:15.531489+0000","last_undegraded":"2023-10-30T13:11:15.531489+0000","last_fullsized":"2023-10-30T13:11:15.531489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:40:51.011001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.438554+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:11:14.438554+0000","last_peered":"2023-10-30T13:11:14.438554+0000","last_clean":"2023-10-30T13:11:14.438554+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:11:14.438554+0000","last_undegraded":"2023-10-30T13:11:14.438554+0000","last_fullsized":"2023-10-30T13:11:14.438554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561157+0000","last_change":"2023-10-30T13:10:34.335243+0000","last_active":"2023-10-30T13:11:15.561157+0000","last_peered":"2023-10-30T13:11:15.561157+0000","last_clean":"2023-10-30T13:11:15.561157+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:11:15.561157+0000","last_undegraded":"2023-10-30T13:11:15.561157+0000","last_fullsized":"2023-10-30T13:11:15.561157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:02.970675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434664+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:11:15.434664+0000","last_peered":"2023-10-30T13:11:15.434664+0000","last_clean":"2023-10-30T13:11:15.434664+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:11:15.434664+0000","last_undegraded":"2023-10-30T13:11:15.434664+0000","last_fullsized":"2023-10-30T13:11:15.434664+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440790+0000","last_change":"2023-10-30T13:10:34.381099+0000","last_active":"2023-10-30T13:11:15.440790+0000","last_peered":"2023-10-30T13:11:15.440790+0000","last_clean":"2023-10-30T13:11:15.440790+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:11:15.440790+0000","last_undegraded":"2023-10-30T13:11:15.440790+0000","last_fullsized":"2023-10-30T13:11:15.440790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:16.694101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446654+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:11:15.446654+0000","last_peered":"2023-10-30T13:11:15.446654+0000","last_clean":"2023-10-30T13:11:15.446654+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:11:15.446654+0000","last_undegraded":"2023-10-30T13:11:15.446654+0000","last_fullsized":"2023-10-30T13:11:15.446654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441447+0000","last_change":"2023-10-30T13:10:34.372219+0000","last_active":"2023-10-30T13:11:14.441447+0000","last_peered":"2023-10-30T13:11:14.441447+0000","last_clean":"2023-10-30T13:11:14.441447+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:11:14.441447+0000","last_undegraded":"2023-10-30T13:11:14.441447+0000","last_fullsized":"2023-10-30T13:11:14.441447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:38.329952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561046+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:11:15.561046+0000","last_peered":"2023-10-30T13:11:15.561046+0000","last_clean":"2023-10-30T13:11:15.561046+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:11:15.561046+0000","last_undegraded":"2023-10-30T13:11:15.561046+0000","last_fullsized":"2023-10-30T13:11:15.561046+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560795+0000","last_change":"2023-10-30T13:10:34.368736+0000","last_active":"2023-10-30T13:11:15.560795+0000","last_peered":"2023-10-30T13:11:15.560795+0000","last_clean":"2023-10-30T13:11:15.560795+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:11:15.560795+0000","last_undegraded":"2023-10-30T13:11:15.560795+0000","last_fullsized":"2023-10-30T13:11:15.560795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:11.128769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438505+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:11:15.438505+0000","last_peered":"2023-10-30T13:11:15.438505+0000","last_clean":"2023-10-30T13:11:15.438505+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:11:15.438505+0000","last_undegraded":"2023-10-30T13:11:15.438505+0000","last_fullsized":"2023-10-30T13:11:15.438505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531029+0000","last_change":"2023-10-30T13:10:34.376512+0000","last_active":"2023-10-30T13:11:15.531029+0000","last_peered":"2023-10-30T13:11:15.531029+0000","last_clean":"2023-10-30T13:11:15.531029+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:11:15.531029+0000","last_undegraded":"2023-10-30T13:11:15.531029+0000","last_fullsized":"2023-10-30T13:11:15.531029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.856690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446249+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:11:15.446249+0000","last_peered":"2023-10-30T13:11:15.446249+0000","last_clean":"2023-10-30T13:11:15.446249+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:11:15.446249+0000","last_undegraded":"2023-10-30T13:11:15.446249+0000","last_fullsized":"2023-10-30T13:11:15.446249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.443575+0000","last_change":"2023-10-30T13:10:34.344863+0000","last_active":"2023-10-30T13:11:15.443575+0000","last_peered":"2023-10-30T13:11:15.443575+0000","last_clean":"2023-10-30T13:11:15.443575+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:11:15.443575+0000","last_undegraded":"2023-10-30T13:11:15.443575+0000","last_fullsized":"2023-10-30T13:11:15.443575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:25.199813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'15","reported_seq":30,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.432685+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:11:15.432685+0000","last_peered":"2023-10-30T13:11:15.432685+0000","last_clean":"2023-10-30T13:11:15.432685+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:11:15.432685+0000","last_undegraded":"2023-10-30T13:11:15.432685+0000","last_fullsized":"2023-10-30T13:11:15.432685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530363+0000","last_change":"2023-10-30T13:10:34.371813+0000","last_active":"2023-10-30T13:11:15.530363+0000","last_peered":"2023-10-30T13:11:15.530363+0000","last_clean":"2023-10-30T13:11:15.530363+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:11:15.530363+0000","last_undegraded":"2023-10-30T13:11:15.530363+0000","last_fullsized":"2023-10-30T13:11:15.530363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:58.719747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440825+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:11:15.440825+0000","last_peered":"2023-10-30T13:11:15.440825+0000","last_clean":"2023-10-30T13:11:15.440825+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:11:15.440825+0000","last_undegraded":"2023-10-30T13:11:15.440825+0000","last_fullsized":"2023-10-30T13:11:15.440825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530665+0000","last_change":"2023-10-30T13:10:34.375164+0000","last_active":"2023-10-30T13:11:15.530665+0000","last_peered":"2023-10-30T13:11:15.530665+0000","last_clean":"2023-10-30T13:11:15.530665+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:11:15.530665+0000","last_undegraded":"2023-10-30T13:11:15.530665+0000","last_fullsized":"2023-10-30T13:11:15.530665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:05.305348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433212+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:11:15.433212+0000","last_peered":"2023-10-30T13:11:15.433212+0000","last_clean":"2023-10-30T13:11:15.433212+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:11:15.433212+0000","last_undegraded":"2023-10-30T13:11:15.433212+0000","last_fullsized":"2023-10-30T13:11:15.433212+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531192+0000","last_change":"2023-10-30T13:10:34.382785+0000","last_active":"2023-10-30T13:11:15.531192+0000","last_peered":"2023-10-30T13:11:15.531192+0000","last_clean":"2023-10-30T13:11:15.531192+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:11:15.531192+0000","last_undegraded":"2023-10-30T13:11:15.531192+0000","last_fullsized":"2023-10-30T13:11:15.531192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:50.203770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434778+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:11:15.434778+0000","last_peered":"2023-10-30T13:11:15.434778+0000","last_clean":"2023-10-30T13:11:15.434778+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:11:15.434778+0000","last_undegraded":"2023-10-30T13:11:15.434778+0000","last_fullsized":"2023-10-30T13:11:15.434778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.439160+0000","last_change":"2023-10-30T13:10:34.342901+0000","last_active":"2023-10-30T13:11:15.439160+0000","last_peered":"2023-10-30T13:11:15.439160+0000","last_clean":"2023-10-30T13:11:15.439160+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:11:15.439160+0000","last_undegraded":"2023-10-30T13:11:15.439160+0000","last_fullsized":"2023-10-30T13:11:15.439160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:27.227907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433265+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:11:15.433265+0000","last_peered":"2023-10-30T13:11:15.433265+0000","last_clean":"2023-10-30T13:11:15.433265+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:11:15.433265+0000","last_undegraded":"2023-10-30T13:11:15.433265+0000","last_fullsized":"2023-10-30T13:11:15.433265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530590+0000","last_change":"2023-10-30T13:10:34.373854+0000","last_active":"2023-10-30T13:11:15.530590+0000","last_peered":"2023-10-30T13:11:15.530590+0000","last_clean":"2023-10-30T13:11:15.530590+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:11:15.530590+0000","last_undegraded":"2023-10-30T13:11:15.530590+0000","last_fullsized":"2023-10-30T13:11:15.530590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:12.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530831+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:11:15.530831+0000","last_peered":"2023-10-30T13:11:15.530831+0000","last_clean":"2023-10-30T13:11:15.530831+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:11:15.530831+0000","last_undegraded":"2023-10-30T13:11:15.530831+0000","last_fullsized":"2023-10-30T13:11:15.530831+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438091+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:11:15.438091+0000","last_peered":"2023-10-30T13:11:15.438091+0000","last_clean":"2023-10-30T13:11:15.438091+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:11:15.438091+0000","last_undegraded":"2023-10-30T13:11:15.438091+0000","last_fullsized":"2023-10-30T13:11:15.438091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430506+0000","last_change":"2023-10-30T13:10:34.348470+0000","last_active":"2023-10-30T13:11:15.430506+0000","last_peered":"2023-10-30T13:11:15.430506+0000","last_clean":"2023-10-30T13:11:15.430506+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:11:15.430506+0000","last_undegraded":"2023-10-30T13:11:15.430506+0000","last_fullsized":"2023-10-30T13:11:15.430506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:24.818359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530008+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:11:15.530008+0000","last_peered":"2023-10-30T13:11:15.530008+0000","last_clean":"2023-10-30T13:11:15.530008+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:11:15.530008+0000","last_undegraded":"2023-10-30T13:11:15.530008+0000","last_fullsized":"2023-10-30T13:11:15.530008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560701+0000","last_change":"2023-10-30T13:10:34.359740+0000","last_active":"2023-10-30T13:11:15.560701+0000","last_peered":"2023-10-30T13:11:15.560701+0000","last_clean":"2023-10-30T13:11:15.560701+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:11:15.560701+0000","last_undegraded":"2023-10-30T13:11:15.560701+0000","last_fullsized":"2023-10-30T13:11:15.560701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:05:25.883485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:12:42.307293+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:12:42.307293+0000","last_peered":"2023-10-30T13:12:42.307293+0000","last_clean":"2023-10-30T13:12:42.307293+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:12:42.307293+0000","last_undegraded":"2023-10-30T13:12:42.307293+0000","last_fullsized":"2023-10-30T13:12:42.307293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433030+0000","last_change":"2023-10-30T13:10:34.332955+0000","last_active":"2023-10-30T13:11:15.433030+0000","last_peered":"2023-10-30T13:11:15.433030+0000","last_clean":"2023-10-30T13:11:15.433030+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:11:15.433030+0000","last_undegraded":"2023-10-30T13:11:15.433030+0000","last_fullsized":"2023-10-30T13:11:15.433030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:05.776190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438730+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:11:15.438730+0000","last_peered":"2023-10-30T13:11:15.438730+0000","last_clean":"2023-10-30T13:11:15.438730+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:11:15.438730+0000","last_undegraded":"2023-10-30T13:11:15.438730+0000","last_fullsized":"2023-10-30T13:11:15.438730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440305+0000","last_change":"2023-10-30T13:10:34.362313+0000","last_active":"2023-10-30T13:11:15.440305+0000","last_peered":"2023-10-30T13:11:15.440305+0000","last_clean":"2023-10-30T13:11:15.440305+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:11:15.440305+0000","last_undegraded":"2023-10-30T13:11:15.440305+0000","last_fullsized":"2023-10-30T13:11:15.440305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:45.213916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434563+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:11:15.434563+0000","last_peered":"2023-10-30T13:11:15.434563+0000","last_clean":"2023-10-30T13:11:15.434563+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:11:15.434563+0000","last_undegraded":"2023-10-30T13:11:15.434563+0000","last_fullsized":"2023-10-30T13:11:15.434563+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.439222+0000","last_change":"2023-10-30T13:10:34.348653+0000","last_active":"2023-10-30T13:11:15.439222+0000","last_peered":"2023-10-30T13:11:15.439222+0000","last_clean":"2023-10-30T13:11:15.439222+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:11:15.439222+0000","last_undegraded":"2023-10-30T13:11:15.439222+0000","last_fullsized":"2023-10-30T13:11:15.439222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:54:00.948062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531132+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:11:15.531132+0000","last_peered":"2023-10-30T13:11:15.531132+0000","last_clean":"2023-10-30T13:11:15.531132+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:11:15.531132+0000","last_undegraded":"2023-10-30T13:11:15.531132+0000","last_fullsized":"2023-10-30T13:11:15.531132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433306+0000","last_change":"2023-10-30T13:10:34.332001+0000","last_active":"2023-10-30T13:11:15.433306+0000","last_peered":"2023-10-30T13:11:15.433306+0000","last_clean":"2023-10-30T13:11:15.433306+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:11:15.433306+0000","last_undegraded":"2023-10-30T13:11:15.433306+0000","last_fullsized":"2023-10-30T13:11:15.433306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:16.341054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434819+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:11:15.434819+0000","last_peered":"2023-10-30T13:11:15.434819+0000","last_clean":"2023-10-30T13:11:15.434819+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:11:15.434819+0000","last_undegraded":"2023-10-30T13:11:15.434819+0000","last_fullsized":"2023-10-30T13:11:15.434819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430358+0000","last_change":"2023-10-30T13:10:34.369845+0000","last_active":"2023-10-30T13:11:15.430358+0000","last_peered":"2023-10-30T13:11:15.430358+0000","last_clean":"2023-10-30T13:11:15.430358+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:11:15.430358+0000","last_undegraded":"2023-10-30T13:11:15.430358+0000","last_fullsized":"2023-10-30T13:11:15.430358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:30.468845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.447484+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:11:15.447484+0000","last_peered":"2023-10-30T13:11:15.447484+0000","last_clean":"2023-10-30T13:11:15.447484+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:11:15.447484+0000","last_undegraded":"2023-10-30T13:11:15.447484+0000","last_fullsized":"2023-10-30T13:11:15.447484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561059+0000","last_change":"2023-10-30T13:10:34.323336+0000","last_active":"2023-10-30T13:11:15.561059+0000","last_peered":"2023-10-30T13:11:15.561059+0000","last_clean":"2023-10-30T13:11:15.561059+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:11:15.561059+0000","last_undegraded":"2023-10-30T13:11:15.561059+0000","last_fullsized":"2023-10-30T13:11:15.561059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:21.642241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.529931+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:11:15.529931+0000","last_peered":"2023-10-30T13:11:15.529931+0000","last_clean":"2023-10-30T13:11:15.529931+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:11:15.529931+0000","last_undegraded":"2023-10-30T13:11:15.529931+0000","last_fullsized":"2023-10-30T13:11:15.529931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440140+0000","last_change":"2023-10-30T13:10:34.363313+0000","last_active":"2023-10-30T13:11:15.440140+0000","last_peered":"2023-10-30T13:11:15.440140+0000","last_clean":"2023-10-30T13:11:15.440140+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:11:15.440140+0000","last_undegraded":"2023-10-30T13:11:15.440140+0000","last_fullsized":"2023-10-30T13:11:15.440140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:13.027610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433415+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:11:15.433415+0000","last_peered":"2023-10-30T13:11:15.433415+0000","last_clean":"2023-10-30T13:11:15.433415+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:11:15.433415+0000","last_undegraded":"2023-10-30T13:11:15.433415+0000","last_fullsized":"2023-10-30T13:11:15.433415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430057+0000","last_change":"2023-10-30T13:10:34.330677+0000","last_active":"2023-10-30T13:11:15.430057+0000","last_peered":"2023-10-30T13:11:15.430057+0000","last_clean":"2023-10-30T13:11:15.430057+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:11:15.430057+0000","last_undegraded":"2023-10-30T13:11:15.430057+0000","last_fullsized":"2023-10-30T13:11:15.430057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:01.360813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433573+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:11:15.433573+0000","last_peered":"2023-10-30T13:11:15.433573+0000","last_clean":"2023-10-30T13:11:15.433573+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:11:15.433573+0000","last_undegraded":"2023-10-30T13:11:15.433573+0000","last_fullsized":"2023-10-30T13:11:15.433573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430362+0000","last_change":"2023-10-30T13:10:34.331117+0000","last_active":"2023-10-30T13:11:15.430362+0000","last_peered":"2023-10-30T13:11:15.430362+0000","last_clean":"2023-10-30T13:11:15.430362+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:11:15.430362+0000","last_undegraded":"2023-10-30T13:11:15.430362+0000","last_fullsized":"2023-10-30T13:11:15.430362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:56.948470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531341+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:11:15.531341+0000","last_peered":"2023-10-30T13:11:15.531341+0000","last_clean":"2023-10-30T13:11:15.531341+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:11:15.531341+0000","last_undegraded":"2023-10-30T13:11:15.531341+0000","last_fullsized":"2023-10-30T13:11:15.531341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446526+0000","last_change":"2023-10-30T13:10:34.365315+0000","last_active":"2023-10-30T13:11:15.446526+0000","last_peered":"2023-10-30T13:11:15.446526+0000","last_clean":"2023-10-30T13:11:15.446526+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:11:15.446526+0000","last_undegraded":"2023-10-30T13:11:15.446526+0000","last_fullsized":"2023-10-30T13:11:15.446526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:37.078775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.442500+0000","last_change":"2023-10-30T13:10:34.366044+0000","last_active":"2023-10-30T13:11:14.442500+0000","last_peered":"2023-10-30T13:11:14.442500+0000","last_clean":"2023-10-30T13:11:14.442500+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:11:14.442500+0000","last_undegraded":"2023-10-30T13:11:14.442500+0000","last_fullsized":"2023-10-30T13:11:14.442500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:31.831020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.442338+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:11:14.442338+0000","last_peered":"2023-10-30T13:11:14.442338+0000","last_clean":"2023-10-30T13:11:14.442338+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:11:14.442338+0000","last_undegraded":"2023-10-30T13:11:14.442338+0000","last_fullsized":"2023-10-30T13:11:14.442338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.436137+0000","last_change":"2023-10-30T13:10:34.369814+0000","last_active":"2023-10-30T13:11:14.436137+0000","last_peered":"2023-10-30T13:11:14.436137+0000","last_clean":"2023-10-30T13:11:14.436137+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:11:14.436137+0000","last_undegraded":"2023-10-30T13:11:14.436137+0000","last_fullsized":"2023-10-30T13:11:14.436137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:26.879717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441916+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:11:14.441916+0000","last_peered":"2023-10-30T13:11:14.441916+0000","last_clean":"2023-10-30T13:11:14.441916+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:11:14.441916+0000","last_undegraded":"2023-10-30T13:11:14.441916+0000","last_fullsized":"2023-10-30T13:11:14.441916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561102+0000","last_change":"2023-10-30T13:10:34.344766+0000","last_active":"2023-10-30T13:11:15.561102+0000","last_peered":"2023-10-30T13:11:15.561102+0000","last_clean":"2023-10-30T13:11:15.561102+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:11:15.561102+0000","last_undegraded":"2023-10-30T13:11:15.561102+0000","last_fullsized":"2023-10-30T13:11:15.561102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.073346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430538+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:11:15.430538+0000","last_peered":"2023-10-30T13:11:15.430538+0000","last_clean":"2023-10-30T13:11:15.430538+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:11:15.430538+0000","last_undegraded":"2023-10-30T13:11:15.430538+0000","last_fullsized":"2023-10-30T13:11:15.430538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430467+0000","last_change":"2023-10-30T13:10:34.354305+0000","last_active":"2023-10-30T13:11:15.430467+0000","last_peered":"2023-10-30T13:11:15.430467+0000","last_clean":"2023-10-30T13:11:15.430467+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:11:15.430467+0000","last_undegraded":"2023-10-30T13:11:15.430467+0000","last_fullsized":"2023-10-30T13:11:15.430467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:53.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'9","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561228+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:11:15.561228+0000","last_peered":"2023-10-30T13:11:15.561228+0000","last_clean":"2023-10-30T13:11:15.561228+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:11:15.561228+0000","last_undegraded":"2023-10-30T13:11:15.561228+0000","last_fullsized":"2023-10-30T13:11:15.561228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":843776,"data_stored":675004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.81999999999999995,"15min":0.81399999999999995},"min":{"1min":0.57099999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.159,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85899999999999999,"15min":0.84899999999999998},"min":{"1min":0.54000000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.284,"5min":1.379,"15min":1.379},"last":1.071}]},{"osd":1,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.94299999999999995,"15min":0.95499999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.3879999999999999,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.97699999999999998,"15min":0.997},"min":{"1min":0.64100000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.052,"5min":4.5229999999999997,"15min":4.5229999999999997},"last":1.0369999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.87,"15min":0.88400000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.0369999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.111},{"interface":"front","average":{"1min":0.995,"5min":1.0149999999999999,"15min":1.018},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.306,"5min":1.359,"15min":1.359},"last":0.79800000000000004}]},{"osd":3,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.98199999999999998,"15min":0.98799999999999999},"min":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1499999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.075,"15min":1.083},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.2529999999999999,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":1.1499999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.085,"15min":1.1120000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.306,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.02,"5min":1.071,"15min":1.079},"min":{"1min":0.67500000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3169999999999999,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.1870000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3049999999999999,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.22},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.2030000000000001,"15min":1.2350000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.208,"5min":4.9269999999999996,"15min":4.9269999999999996},"last":0.96599999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.153,"15min":1.1870000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2350000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.3160000000000001,"15min":1.3640000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.2410000000000001,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.83499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":85610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.0980000000000001,"15min":1.1399999999999999},"min":{"1min":0.61499999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0429999999999999,"5min":4.1699999999999999,"15min":4.1699999999999999},"last":0.62},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.93799999999999994,"15min":0.94599999999999995},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0860000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.94899999999999995}]},{"osd":1,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.625,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1279999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.92800000000000005,"15min":0.93100000000000005},"min":{"1min":0.67300000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.177,"5min":1.458,"15min":1.458},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.1870000000000001,"15min":1.2270000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1519999999999999,"5min":5.4269999999999996,"15min":5.4269999999999996},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.034,"15min":1.048},"min":{"1min":0.63600000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.258,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.82699999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0269999999999999,"15min":1.044},"min":{"1min":0.629,"5min":0.622,"15min":0.622},"max":{"1min":1.181,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":1.052},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.92200000000000004,"15min":0.93200000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.093,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.76900000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.008,"15min":1.014},"min":{"1min":0.65700000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.359,"5min":1.546,"15min":1.546},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.044,"15min":1.0429999999999999},"min":{"1min":0.69999999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.67800000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.024,"15min":1.026},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.1970000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.167,"15min":1.177},"min":{"1min":0.78500000000000003,"5min":0.749,"15min":0.749},"max":{"1min":1.423,"5min":1.653,"15min":1.653},"last":1.0760000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.177,"15min":1.179},"min":{"1min":0.83899999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.091,"5min":1.1399999999999999,"15min":1.1479999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.46,"5min":1.704,"15min":1.704},"last":1.127}]}]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":84698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.012,"5min":1.155,"15min":1.155},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.80000000000000004,"15min":0.78400000000000003},"min":{"1min":0.59899999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.079}]},{"osd":1,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.82299999999999995,"15min":0.81100000000000005},"min":{"1min":0.58799999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0820000000000001,"5min":1.24,"15min":1.24},"last":1.0600000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.64400000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.157,"5min":1.258,"15min":1.258},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.95699999999999996,"15min":0.96699999999999997},"min":{"1min":0.67900000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1699999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.03}]},{"osd":3,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.73899999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.1739999999999999,"5min":1.218,"15min":1.218},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.028,"15min":1.03},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.2849999999999999,"5min":1.601,"15min":1.601},"last":1.1479999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94999999999999996,"15min":0.96999999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.097,"5min":1.379,"15min":1.379},"last":0.751},{"interface":"front","average":{"1min":1.147,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.3320000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.24}]},{"osd":6,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.131,"15min":1.147},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.194,"5min":1.875,"15min":1.875},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.042,"15min":1.0580000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.2230000000000001,"5min":1.554,"15min":1.554},"last":1.117}]},{"osd":7,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.083,"15min":1.083},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.3480000000000001,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.171},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.175,"15min":1.1850000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.401,"5min":1.821,"15min":1.821},"last":1.2769999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.91600000000000004,"15min":0.92400000000000004},"min":{"1min":0.53900000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2609999999999999,"5min":3.6469999999999998,"15min":3.6469999999999998},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.90000000000000002,"15min":0.89200000000000002},"min":{"1min":0.54600000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.351,"5min":1.669,"15min":1.669},"last":0.55500000000000005}]},{"osd":1,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.2330000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.90100000000000002,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.242,"5min":1.413,"15min":1.413},"last":0.83199999999999996}]},{"osd":2,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93400000000000005,"15min":0.94199999999999995},"min":{"1min":0.59499999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.302,"5min":3.6840000000000002,"15min":3.6840000000000002},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93000000000000005,"15min":0.92200000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.90000000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.95099999999999996,"15min":0.93899999999999995},"min":{"1min":0.69699999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2749999999999999,"5min":1.321,"15min":1.321},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.88200000000000001,"15min":0.877},"min":{"1min":0.58599999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.145,"5min":1.46,"15min":1.46},"last":1}]},{"osd":5,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.004,"15min":1.006},"min":{"1min":0.64500000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3620000000000001,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0229999999999999,"15min":1.02},"min":{"1min":0.72599999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.3220000000000001,"5min":1.514,"15min":1.514},"last":1.1379999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.92900000000000005,"15min":0.92300000000000004},"min":{"1min":0.70799999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.091},{"interface":"front","average":{"1min":1.024,"5min":1.1000000000000001,"15min":1.1120000000000001},"min":{"1min":0.749,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.2789999999999999,"5min":2.0379999999999998,"15min":2.0379999999999998},"last":1.0680000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.72399999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.569,"5min":1.744,"15min":1.744},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.6000000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.0429999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":86717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.59799999999999998,"15min":0.61899999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65200000000000002,"5min":1.923,"15min":1.923},"last":0.308},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70599999999999996,"5min":2.085,"15min":2.085},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.65900000000000003,"15min":0.68000000000000005},"min":{"1min":0.33500000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73799999999999999,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.38100000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.754,"15min":0.78800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.751,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63800000000000001,"15min":0.64600000000000002},"min":{"1min":0.41999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79400000000000004,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91200000000000003,"15min":0.92300000000000004},"min":{"1min":0.72699999999999998,"5min":0.627,"15min":0.627},"max":{"1min":1.0720000000000001,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.876,"5min":0.95999999999999996,"15min":0.97399999999999998},"min":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.0569999999999999,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.996,"15min":1.026},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.98699999999999999,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.97899999999999998,"15min":0.98499999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1439999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.96499999999999997,"15min":0.98299999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.111,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":0.623},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.97299999999999998,"15min":0.98799999999999999},"min":{"1min":0.66200000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.149,"5min":1.839,"15min":1.839},"last":0.63700000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.0009999999999999,"15min":1.0169999999999999},"min":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.1160000000000001,"5min":2.117,"15min":2.117},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.026,"15min":1.048},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.1100000000000001,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":0.65000000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":84138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.33400000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.436},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.38500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58399999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.48899999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.34999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67200000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.755,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.80900000000000005,"15min":0.79900000000000004},"min":{"1min":0.70099999999999996,"5min":0.629,"15min":0.629},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.81999999999999995,"15min":0.81799999999999995},"min":{"1min":0.67500000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0469999999999999,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.78500000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85399999999999998},"min":{"1min":0.68200000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":0.98599999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.90900000000000003,"15min":0.91900000000000004},"min":{"1min":0.68300000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.1220000000000001,"5min":1.339,"15min":1.339},"last":0.68300000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.84499999999999997,"15min":0.84199999999999997},"min":{"1min":0.68300000000000005,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.0449999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.628},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.93000000000000005,"15min":0.94499999999999995},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":0.95399999999999996,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.82599999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98999999999999999,"15min":1},"min":{"1min":0.74099999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.091,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.94699999999999995,"15min":0.95699999999999996},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.075,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.70599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":84264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72999999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.502},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.60799999999999998,"5min":1.006,"15min":1.006},"last":0.59999999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63300000000000001,"5min":1.095,"15min":1.095},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.39400000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.63,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.69399999999999995,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.39800000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67800000000000005,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.42099999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.63300000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.0800000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.94499999999999995,"15min":0.96899999999999997},"min":{"1min":0.67300000000000004,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.03,"5min":4.093,"15min":4.093},"last":0.94899999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85099999999999998,"15min":0.85399999999999998},"min":{"1min":0.67800000000000005,"5min":0.629,"15min":0.629},"max":{"1min":1.0609999999999999,"5min":1.796,"15min":1.796},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.61399999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.012,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.86799999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.88600000000000001,"15min":0.89800000000000002},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.1919999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.89900000000000002,"15min":0.90300000000000002},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":0.997,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.88300000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":1,"15min":1.028},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1379999999999999,"5min":4.1470000000000002,"15min":4.1470000000000002},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.90700000000000003,"15min":0.91700000000000004},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.016,"5min":1.976,"15min":1.976},"last":0.80100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":835584,"data_stored":673329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.755,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51300000000000001,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.71199999999999997,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.497}]},{"osd":2,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73599999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.748,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.78000000000000003,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78900000000000003,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.629,"5min":0.62,"15min":0.62},"max":{"1min":1.0660000000000001,"5min":1.222,"15min":1.222},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.84599999999999997,"15min":0.85199999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.94499999999999995,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.94899999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86599999999999999,"15min":0.86899999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.044,"5min":1.306,"15min":1.306},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.873,"5min":0.90500000000000003,"15min":0.91000000000000003},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.1459999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.88,"15min":0.88600000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.0129999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.878,"5min":0.92000000000000004,"15min":0.92700000000000005},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.1319999999999999,"5min":1.446,"15min":1.446},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.89100000000000001,"15min":0.90300000000000002},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.109,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92700000000000005,"15min":0.93500000000000005},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0880000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.67200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:13:02.893 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-10-30T13:13:02.894 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:13:03.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00fccad640 1 Processor -- start 2023-10-30T13:13:03.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00fccad640 1 -- start start 2023-10-30T13:13:03.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00fccad640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f00f81538f0 0x7f00f8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00fccad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f815cd50 0x7f00f80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00fccad640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f815d300 con 0x7f00f80cefa0 2023-10-30T13:13:03.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00fccad640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f815d5e0 con 0x7f00f81538f0 2023-10-30T13:13:03.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00fccad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f815d8c0 con 0x7f00f815cd50 2023-10-30T13:13:03.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f77fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f00f81538f0 0x7f00f8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f77fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f00f81538f0 0x7f00f8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38366/0 (socket says 172.21.15.148:38366) 2023-10-30T13:13:03.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f77fe640 1 -- 172.21.15.148:0/1318807863 learned_addr learned my addr 172.21.15.148:0/1318807863 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:03.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f7fff640 1 --1- 172.21.15.148:0/1318807863 >> v1:172.21.15.148:6789/0 conn(0x7f00f80cefa0 0x7f00f80cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42648/0 (socket says 172.21.15.148:42648) 2023-10-30T13:13:03.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f6ffd640 1 --2- 172.21.15.148:0/1318807863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f815cd50 0x7f00f80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f77fe640 1 -- 172.21.15.148:0/1318807863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f815cd50 msgr2=0x7f00f80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:03.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f77fe640 1 --2- 172.21.15.148:0/1318807863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f815cd50 0x7f00f80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f77fe640 1 -- 172.21.15.148:0/1318807863 >> v1:172.21.15.148:6789/0 conn(0x7f00f80cefa0 legacy=0x7f00f80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f77fe640 1 -- 172.21.15.148:0/1318807863 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f815dba0 con 0x7f00f81538f0 2023-10-30T13:13:03.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.065+0000 7f00f67fc640 1 -- 172.21.15.148:0/1318807863 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717944844 0 0) 0x7f00f815d300 con 0x7f00f80cefa0 2023-10-30T13:13:03.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00f77fe640 1 --2- 172.21.15.148:0/1318807863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f81538f0 0x7f00f8159fb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009870 tx=0x7f00e801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470ed824a2e67789 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00f67fc640 1 -- 172.21.15.148:0/1318807863 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e80a8a30 con 0x7f00f81538f0 2023-10-30T13:13:03.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00f67fc640 1 -- 172.21.15.148:0/1318807863 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e80a8b90 con 0x7f00f81538f0 2023-10-30T13:13:03.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00f67fc640 1 -- 172.21.15.148:0/1318807863 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e80b0440 con 0x7f00f81538f0 2023-10-30T13:13:03.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00fccad640 1 -- 172.21.15.148:0/1318807863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f81538f0 msgr2=0x7f00f8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00fccad640 1 --2- 172.21.15.148:0/1318807863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f81538f0 0x7f00f8159fb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009870 tx=0x7f00e801ed30 comp rx=0 tx=0).stop 2023-10-30T13:13:03.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00fccad640 1 -- 172.21.15.148:0/1318807863 shutdown_connections 2023-10-30T13:13:03.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00fccad640 1 --2- 172.21.15.148:0/1318807863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f815cd50 0x7f00f80c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00fccad640 1 --2- 172.21.15.148:0/1318807863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f81538f0 0x7f00f8159fb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00fccad640 1 -- 172.21.15.148:0/1318807863 >> 172.21.15.148:0/1318807863 conn(0x7f00f806a640 msgr2=0x7f00f8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:03.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00fccad640 1 -- 172.21.15.148:0/1318807863 shutdown_connections 2023-10-30T13:13:03.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00fccad640 1 -- 172.21.15.148:0/1318807863 wait complete. 2023-10-30T13:13:03.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.069+0000 7f00fccad640 1 Processor -- start 2023-10-30T13:13:03.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 -- start start 2023-10-30T13:13:03.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f80cefa0 0x7f00f80b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f81538f0 0x7f00f81efb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f815cd50 0x7f00f8147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f81f3480 con 0x7f00f815cd50 2023-10-30T13:13:03.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f81ef0d0 con 0x7f00f80cefa0 2023-10-30T13:13:03.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f81e6410 con 0x7f00f81538f0 2023-10-30T13:13:03.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f815cd50 0x7f00f8147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f815cd50 0x7f00f8147bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38376/0 (socket says 172.21.15.148:38376) 2023-10-30T13:13:03.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f7fff640 1 -- 172.21.15.148:0/3558345469 learned_addr learned my addr 172.21.15.148:0/3558345469 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:03.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f77fe640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f80cefa0 0x7f00f80b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f6ffd640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f81538f0 0x7f00f81efb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f7fff640 1 -- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f81538f0 msgr2=0x7f00f81efb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f7fff640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f81538f0 0x7f00f81efb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f7fff640 1 -- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f80cefa0 msgr2=0x7f00f80b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f7fff640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f80cefa0 0x7f00f80b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f7fff640 1 -- 172.21.15.148:0/3558345469 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e80094d0 con 0x7f00f815cd50 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00f7fff640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f815cd50 0x7f00f8147bd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f00f0001570 tx=0x7f00f000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00dffff640 1 -- 172.21.15.148:0/3558345469 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f000a640 con 0x7f00f815cd50 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00dffff640 1 -- 172.21.15.148:0/3558345469 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f000a7c0 con 0x7f00f815cd50 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00dffff640 1 -- 172.21.15.148:0/3558345469 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f000ab70 con 0x7f00f815cd50 2023-10-30T13:13:03.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f81e66f0 con 0x7f00f815cd50 2023-10-30T13:13:03.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f81ef390 con 0x7f00f815cd50 2023-10-30T13:13:03.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.073+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f804e170 con 0x7f00f815cd50 2023-10-30T13:13:03.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.077+0000 7f00dffff640 1 -- 172.21.15.148:0/3558345469 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00f000acd0 con 0x7f00f815cd50 2023-10-30T13:13:03.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.077+0000 7f00dffff640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00e40664a0 0x7f00e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.077+0000 7f00dffff640 1 -- 172.21.15.148:0/3558345469 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f00f00833a0 con 0x7f00f815cd50 2023-10-30T13:13:03.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.077+0000 7f00f77fe640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00e40664a0 0x7f00e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.077+0000 7f00f77fe640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00e40664a0 0x7f00e4068960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f00e80b2000 tx=0x7f00e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.081+0000 7f00dffff640 1 -- 172.21.15.148:0/3558345469 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00f00508d0 con 0x7f00f815cd50 2023-10-30T13:13:03.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.229+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00f81f35c0 con 0x7f00e40664a0 2023-10-30T13:13:03.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.249+0000 7f00dffff640 1 -- 172.21.15.148:0/3558345469 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288228 (secure 0 0 0) 0x7f00f81f35c0 con 0x7f00e40664a0 2023-10-30T13:13:03.256 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:13:03.257 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:13:03.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00e40664a0 msgr2=0x7f00e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00e40664a0 0x7f00e4068960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f00e80b2000 tx=0x7f00e8009040 comp rx=0 tx=0).stop 2023-10-30T13:13:03.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f815cd50 msgr2=0x7f00f8147bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f815cd50 0x7f00f8147bd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f00f0001570 tx=0x7f00f000cca0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 shutdown_connections 2023-10-30T13:13:03.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f00e40664a0 0x7f00e4068960 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00f815cd50 0x7f00f8147bd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f00f81538f0 0x7f00f81efb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 --2- 172.21.15.148:0/3558345469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f80cefa0 0x7f00f80b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 >> 172.21.15.148:0/3558345469 conn(0x7f00f806a640 msgr2=0x7f00f80caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:03.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 shutdown_connections 2023-10-30T13:13:03.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.253+0000 7f00fccad640 1 -- 172.21.15.148:0/3558345469 wait complete. 2023-10-30T13:13:03.276 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2023-10-30T13:13:02.314085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":285,"num_write_kb":2455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219284,"kb_used_data":3108,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755436,"statfs":{"total":773094113280,"available":772869566464,"internally_reserved":0,"allocated":3182592,"data_stored":1857794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003910"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530410+0000","last_change":"2023-10-30T13:10:34.379031+0000","last_active":"2023-10-30T13:11:15.530410+0000","last_peered":"2023-10-30T13:11:15.530410+0000","last_clean":"2023-10-30T13:11:15.530410+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:11:15.530410+0000","last_undegraded":"2023-10-30T13:11:15.530410+0000","last_fullsized":"2023-10-30T13:11:15.530410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:32.529312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434689+0000","last_change":"2023-10-30T13:10:34.324308+0000","last_active":"2023-10-30T13:11:15.434689+0000","last_peered":"2023-10-30T13:11:15.434689+0000","last_clean":"2023-10-30T13:11:15.434689+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:11:15.434689+0000","last_undegraded":"2023-10-30T13:11:15.434689+0000","last_fullsized":"2023-10-30T13:11:15.434689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:21.885716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561039+0000","last_change":"2023-10-30T13:10:34.334122+0000","last_active":"2023-10-30T13:11:15.561039+0000","last_peered":"2023-10-30T13:11:15.561039+0000","last_clean":"2023-10-30T13:11:15.561039+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:11:15.561039+0000","last_undegraded":"2023-10-30T13:11:15.561039+0000","last_fullsized":"2023-10-30T13:11:15.561039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:55.045024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437588+0000","last_change":"2023-10-30T13:11:15.435943+0000","last_active":"2023-10-30T13:11:15.437588+0000","last_peered":"2023-10-30T13:11:15.437588+0000","last_clean":"2023-10-30T13:11:15.437588+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:11:15.437588+0000","last_undegraded":"2023-10-30T13:11:15.437588+0000","last_fullsized":"2023-10-30T13:11:15.437588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:03.152688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441733+0000","last_change":"2023-10-30T13:10:34.371670+0000","last_active":"2023-10-30T13:11:14.441733+0000","last_peered":"2023-10-30T13:11:14.441733+0000","last_clean":"2023-10-30T13:11:14.441733+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:11:14.441733+0000","last_undegraded":"2023-10-30T13:11:14.441733+0000","last_fullsized":"2023-10-30T13:11:14.441733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:03:54.773965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441333+0000","last_change":"2023-10-30T13:10:34.335752+0000","last_active":"2023-10-30T13:11:14.441333+0000","last_peered":"2023-10-30T13:11:14.441333+0000","last_clean":"2023-10-30T13:11:14.441333+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:11:14.441333+0000","last_undegraded":"2023-10-30T13:11:14.441333+0000","last_fullsized":"2023-10-30T13:11:14.441333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:33.278920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.440809+0000","last_change":"2023-10-30T13:10:34.371817+0000","last_active":"2023-10-30T13:11:14.440809+0000","last_peered":"2023-10-30T13:11:14.440809+0000","last_clean":"2023-10-30T13:11:14.440809+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:11:14.440809+0000","last_undegraded":"2023-10-30T13:11:14.440809+0000","last_fullsized":"2023-10-30T13:11:14.440809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:33.797113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.427171+0000","last_change":"2023-10-30T13:10:34.348205+0000","last_active":"2023-10-30T13:11:14.427171+0000","last_peered":"2023-10-30T13:11:14.427171+0000","last_clean":"2023-10-30T13:11:14.427171+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:11:14.427171+0000","last_undegraded":"2023-10-30T13:11:14.427171+0000","last_fullsized":"2023-10-30T13:11:14.427171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:57.492738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.442942+0000","last_change":"2023-10-30T13:10:34.381052+0000","last_active":"2023-10-30T13:11:14.442942+0000","last_peered":"2023-10-30T13:11:14.442942+0000","last_clean":"2023-10-30T13:11:14.442942+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:11:14.442942+0000","last_undegraded":"2023-10-30T13:11:14.442942+0000","last_fullsized":"2023-10-30T13:11:14.442942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:16.697332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.447936+0000","last_change":"2023-10-30T13:10:34.356932+0000","last_active":"2023-10-30T13:11:14.447936+0000","last_peered":"2023-10-30T13:11:14.447936+0000","last_clean":"2023-10-30T13:11:14.447936+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:11:14.447936+0000","last_undegraded":"2023-10-30T13:11:14.447936+0000","last_fullsized":"2023-10-30T13:11:14.447936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:54.091993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.447369+0000","last_change":"2023-10-30T13:11:15.447369+0000","last_active":"2023-10-30T13:11:15.447369+0000","last_peered":"2023-10-30T13:11:15.447369+0000","last_clean":"2023-10-30T13:11:15.447369+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:11:15.447369+0000","last_undegraded":"2023-10-30T13:11:15.447369+0000","last_fullsized":"2023-10-30T13:11:15.447369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:32.683227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.439648+0000","last_change":"2023-10-30T13:10:34.365161+0000","last_active":"2023-10-30T13:11:14.439648+0000","last_peered":"2023-10-30T13:11:14.439648+0000","last_clean":"2023-10-30T13:11:14.439648+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:11:14.439648+0000","last_undegraded":"2023-10-30T13:11:14.439648+0000","last_fullsized":"2023-10-30T13:11:14.439648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:51.721060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430223+0000","last_change":"2023-10-30T13:10:34.351901+0000","last_active":"2023-10-30T13:11:15.430223+0000","last_peered":"2023-10-30T13:11:15.430223+0000","last_clean":"2023-10-30T13:11:15.430223+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:11:15.430223+0000","last_undegraded":"2023-10-30T13:11:15.430223+0000","last_fullsized":"2023-10-30T13:11:15.430223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:11.576546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560710+0000","last_change":"2023-10-30T13:10:34.323358+0000","last_active":"2023-10-30T13:11:15.560710+0000","last_peered":"2023-10-30T13:11:15.560710+0000","last_clean":"2023-10-30T13:11:15.560710+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:11:15.560710+0000","last_undegraded":"2023-10-30T13:11:15.560710+0000","last_fullsized":"2023-10-30T13:11:15.560710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:03.178822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433306+0000","last_change":"2023-10-30T13:10:34.360120+0000","last_active":"2023-10-30T13:11:15.433306+0000","last_peered":"2023-10-30T13:11:15.433306+0000","last_clean":"2023-10-30T13:11:15.433306+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:11:15.433306+0000","last_undegraded":"2023-10-30T13:11:15.433306+0000","last_fullsized":"2023-10-30T13:11:15.433306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:30.731746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437472+0000","last_change":"2023-10-30T13:10:34.333752+0000","last_active":"2023-10-30T13:11:15.437472+0000","last_peered":"2023-10-30T13:11:15.437472+0000","last_clean":"2023-10-30T13:11:15.437472+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:11:15.437472+0000","last_undegraded":"2023-10-30T13:11:15.437472+0000","last_fullsized":"2023-10-30T13:11:15.437472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:45.390603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.441064+0000","last_change":"2023-10-30T13:10:34.371931+0000","last_active":"2023-10-30T13:11:15.441064+0000","last_peered":"2023-10-30T13:11:15.441064+0000","last_clean":"2023-10-30T13:11:15.441064+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:11:15.441064+0000","last_undegraded":"2023-10-30T13:11:15.441064+0000","last_fullsized":"2023-10-30T13:11:15.441064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:51.514990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433093+0000","last_change":"2023-10-30T13:10:34.352747+0000","last_active":"2023-10-30T13:11:15.433093+0000","last_peered":"2023-10-30T13:11:15.433093+0000","last_clean":"2023-10-30T13:11:15.433093+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:11:15.433093+0000","last_undegraded":"2023-10-30T13:11:15.433093+0000","last_fullsized":"2023-10-30T13:11:15.433093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:14:21.543351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437590+0000","last_change":"2023-10-30T13:11:15.436291+0000","last_active":"2023-10-30T13:11:15.437590+0000","last_peered":"2023-10-30T13:11:15.437590+0000","last_clean":"2023-10-30T13:11:15.437590+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:11:15.437590+0000","last_undegraded":"2023-10-30T13:11:15.437590+0000","last_fullsized":"2023-10-30T13:11:15.437590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:11:22.745942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446501+0000","last_change":"2023-10-30T13:10:34.366575+0000","last_active":"2023-10-30T13:11:15.446501+0000","last_peered":"2023-10-30T13:11:15.446501+0000","last_clean":"2023-10-30T13:11:15.446501+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:11:15.446501+0000","last_undegraded":"2023-10-30T13:11:15.446501+0000","last_fullsized":"2023-10-30T13:11:15.446501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:30:23.540645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560863+0000","last_change":"2023-10-30T13:10:34.368761+0000","last_active":"2023-10-30T13:11:15.560863+0000","last_peered":"2023-10-30T13:11:15.560863+0000","last_clean":"2023-10-30T13:11:15.560863+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:11:15.560863+0000","last_undegraded":"2023-10-30T13:11:15.560863+0000","last_fullsized":"2023-10-30T13:11:15.560863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:32.861756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438076+0000","last_change":"2023-10-30T13:10:34.372881+0000","last_active":"2023-10-30T13:11:15.438076+0000","last_peered":"2023-10-30T13:11:15.438076+0000","last_clean":"2023-10-30T13:11:15.438076+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:11:15.438076+0000","last_undegraded":"2023-10-30T13:11:15.438076+0000","last_fullsized":"2023-10-30T13:11:15.438076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:23.855779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430039+0000","last_change":"2023-10-30T13:10:34.359359+0000","last_active":"2023-10-30T13:11:15.430039+0000","last_peered":"2023-10-30T13:11:15.430039+0000","last_clean":"2023-10-30T13:11:15.430039+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:11:15.430039+0000","last_undegraded":"2023-10-30T13:11:15.430039+0000","last_fullsized":"2023-10-30T13:11:15.430039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:34:08.365523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.437617+0000","last_change":"2023-10-30T13:10:34.347431+0000","last_active":"2023-10-30T13:11:15.437617+0000","last_peered":"2023-10-30T13:11:15.437617+0000","last_clean":"2023-10-30T13:11:15.437617+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:11:15.437617+0000","last_undegraded":"2023-10-30T13:11:15.437617+0000","last_fullsized":"2023-10-30T13:11:15.437617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:25.094617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.442207+0000","last_change":"2023-10-30T13:11:15.440541+0000","last_active":"2023-10-30T13:11:15.442207+0000","last_peered":"2023-10-30T13:11:15.442207+0000","last_clean":"2023-10-30T13:11:15.442207+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:11:15.442207+0000","last_undegraded":"2023-10-30T13:11:15.442207+0000","last_fullsized":"2023-10-30T13:11:15.442207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:28:16.858272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.447245+0000","last_change":"2023-10-30T13:10:34.379878+0000","last_active":"2023-10-30T13:11:15.447245+0000","last_peered":"2023-10-30T13:11:15.447245+0000","last_clean":"2023-10-30T13:11:15.447245+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:11:15.447245+0000","last_undegraded":"2023-10-30T13:11:15.447245+0000","last_fullsized":"2023-10-30T13:11:15.447245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:19:42.280634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434611+0000","last_change":"2023-10-30T13:10:34.332020+0000","last_active":"2023-10-30T13:11:15.434611+0000","last_peered":"2023-10-30T13:11:15.434611+0000","last_clean":"2023-10-30T13:11:15.434611+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:11:15.434611+0000","last_undegraded":"2023-10-30T13:11:15.434611+0000","last_fullsized":"2023-10-30T13:11:15.434611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:53.057012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430366+0000","last_change":"2023-10-30T13:10:34.363154+0000","last_active":"2023-10-30T13:11:15.430366+0000","last_peered":"2023-10-30T13:11:15.430366+0000","last_clean":"2023-10-30T13:11:15.430366+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:11:15.430366+0000","last_undegraded":"2023-10-30T13:11:15.430366+0000","last_fullsized":"2023-10-30T13:11:15.430366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:20:04.359629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560893+0000","last_change":"2023-10-30T13:10:34.332160+0000","last_active":"2023-10-30T13:11:15.560893+0000","last_peered":"2023-10-30T13:11:15.560893+0000","last_clean":"2023-10-30T13:11:15.560893+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:11:15.560893+0000","last_undegraded":"2023-10-30T13:11:15.560893+0000","last_fullsized":"2023-10-30T13:11:15.560893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:48:28.128344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438275+0000","last_change":"2023-10-30T13:10:34.337617+0000","last_active":"2023-10-30T13:11:15.438275+0000","last_peered":"2023-10-30T13:11:15.438275+0000","last_clean":"2023-10-30T13:11:15.438275+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:11:15.438275+0000","last_undegraded":"2023-10-30T13:11:15.438275+0000","last_fullsized":"2023-10-30T13:11:15.438275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:30:10.287441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.432768+0000","last_change":"2023-10-30T13:10:34.359681+0000","last_active":"2023-10-30T13:11:15.432768+0000","last_peered":"2023-10-30T13:11:15.432768+0000","last_clean":"2023-10-30T13:11:15.432768+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:11:15.432768+0000","last_undegraded":"2023-10-30T13:11:15.432768+0000","last_fullsized":"2023-10-30T13:11:15.432768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:02.812352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.443436+0000","last_change":"2023-10-30T13:10:34.344864+0000","last_active":"2023-10-30T13:11:15.443436+0000","last_peered":"2023-10-30T13:11:15.443436+0000","last_clean":"2023-10-30T13:11:15.443436+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:11:15.443436+0000","last_undegraded":"2023-10-30T13:11:15.443436+0000","last_fullsized":"2023-10-30T13:11:15.443436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:40:13.520306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.439520+0000","last_change":"2023-10-30T13:10:34.364069+0000","last_active":"2023-10-30T13:11:14.439520+0000","last_peered":"2023-10-30T13:11:14.439520+0000","last_clean":"2023-10-30T13:11:14.439520+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:11:14.439520+0000","last_undegraded":"2023-10-30T13:11:14.439520+0000","last_fullsized":"2023-10-30T13:11:14.439520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:15.717336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.427598+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:11:14.427598+0000","last_peered":"2023-10-30T13:11:14.427598+0000","last_clean":"2023-10-30T13:11:14.427598+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:11:14.427598+0000","last_undegraded":"2023-10-30T13:11:14.427598+0000","last_fullsized":"2023-10-30T13:11:14.427598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.503227+0000","last_change":"2023-10-30T13:10:34.360935+0000","last_active":"2023-10-30T13:11:14.503227+0000","last_peered":"2023-10-30T13:11:14.503227+0000","last_clean":"2023-10-30T13:11:14.503227+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:11:14.503227+0000","last_undegraded":"2023-10-30T13:11:14.503227+0000","last_fullsized":"2023-10-30T13:11:14.503227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:58.639753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.447505+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:11:14.447505+0000","last_peered":"2023-10-30T13:11:14.447505+0000","last_clean":"2023-10-30T13:11:14.447505+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:11:14.447505+0000","last_undegraded":"2023-10-30T13:11:14.447505+0000","last_fullsized":"2023-10-30T13:11:14.447505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430030+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:11:15.430030+0000","last_peered":"2023-10-30T13:11:15.430030+0000","last_clean":"2023-10-30T13:11:15.430030+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:11:15.430030+0000","last_undegraded":"2023-10-30T13:11:15.430030+0000","last_fullsized":"2023-10-30T13:11:15.430030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430018+0000","last_change":"2023-10-30T13:10:34.329935+0000","last_active":"2023-10-30T13:11:15.430018+0000","last_peered":"2023-10-30T13:11:15.430018+0000","last_clean":"2023-10-30T13:11:15.430018+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:11:15.430018+0000","last_undegraded":"2023-10-30T13:11:15.430018+0000","last_fullsized":"2023-10-30T13:11:15.430018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:07.795163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.434429+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:11:14.434429+0000","last_peered":"2023-10-30T13:11:14.434429+0000","last_clean":"2023-10-30T13:11:14.434429+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:11:14.434429+0000","last_undegraded":"2023-10-30T13:11:14.434429+0000","last_fullsized":"2023-10-30T13:11:14.434429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433082+0000","last_change":"2023-10-30T13:10:34.331351+0000","last_active":"2023-10-30T13:11:15.433082+0000","last_peered":"2023-10-30T13:11:15.433082+0000","last_clean":"2023-10-30T13:11:15.433082+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:11:15.433082+0000","last_undegraded":"2023-10-30T13:11:15.433082+0000","last_fullsized":"2023-10-30T13:11:15.433082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:27.864073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.448058+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:11:14.448058+0000","last_peered":"2023-10-30T13:11:14.448058+0000","last_clean":"2023-10-30T13:11:14.448058+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:11:14.448058+0000","last_undegraded":"2023-10-30T13:11:14.448058+0000","last_fullsized":"2023-10-30T13:11:14.448058+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531489+0000","last_change":"2023-10-30T13:11:15.530240+0000","last_active":"2023-10-30T13:11:15.531489+0000","last_peered":"2023-10-30T13:11:15.531489+0000","last_clean":"2023-10-30T13:11:15.531489+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:11:15.531489+0000","last_undegraded":"2023-10-30T13:11:15.531489+0000","last_fullsized":"2023-10-30T13:11:15.531489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:40:51.011001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.438554+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:11:14.438554+0000","last_peered":"2023-10-30T13:11:14.438554+0000","last_clean":"2023-10-30T13:11:14.438554+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:11:14.438554+0000","last_undegraded":"2023-10-30T13:11:14.438554+0000","last_fullsized":"2023-10-30T13:11:14.438554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561157+0000","last_change":"2023-10-30T13:10:34.335243+0000","last_active":"2023-10-30T13:11:15.561157+0000","last_peered":"2023-10-30T13:11:15.561157+0000","last_clean":"2023-10-30T13:11:15.561157+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:11:15.561157+0000","last_undegraded":"2023-10-30T13:11:15.561157+0000","last_fullsized":"2023-10-30T13:11:15.561157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:02.970675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434664+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:11:15.434664+0000","last_peered":"2023-10-30T13:11:15.434664+0000","last_clean":"2023-10-30T13:11:15.434664+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:11:15.434664+0000","last_undegraded":"2023-10-30T13:11:15.434664+0000","last_fullsized":"2023-10-30T13:11:15.434664+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440790+0000","last_change":"2023-10-30T13:10:34.381099+0000","last_active":"2023-10-30T13:11:15.440790+0000","last_peered":"2023-10-30T13:11:15.440790+0000","last_clean":"2023-10-30T13:11:15.440790+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:11:15.440790+0000","last_undegraded":"2023-10-30T13:11:15.440790+0000","last_fullsized":"2023-10-30T13:11:15.440790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:16.694101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446654+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:11:15.446654+0000","last_peered":"2023-10-30T13:11:15.446654+0000","last_clean":"2023-10-30T13:11:15.446654+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:11:15.446654+0000","last_undegraded":"2023-10-30T13:11:15.446654+0000","last_fullsized":"2023-10-30T13:11:15.446654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441447+0000","last_change":"2023-10-30T13:10:34.372219+0000","last_active":"2023-10-30T13:11:14.441447+0000","last_peered":"2023-10-30T13:11:14.441447+0000","last_clean":"2023-10-30T13:11:14.441447+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:11:14.441447+0000","last_undegraded":"2023-10-30T13:11:14.441447+0000","last_fullsized":"2023-10-30T13:11:14.441447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:26:38.329952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561046+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:11:15.561046+0000","last_peered":"2023-10-30T13:11:15.561046+0000","last_clean":"2023-10-30T13:11:15.561046+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:11:15.561046+0000","last_undegraded":"2023-10-30T13:11:15.561046+0000","last_fullsized":"2023-10-30T13:11:15.561046+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560795+0000","last_change":"2023-10-30T13:10:34.368736+0000","last_active":"2023-10-30T13:11:15.560795+0000","last_peered":"2023-10-30T13:11:15.560795+0000","last_clean":"2023-10-30T13:11:15.560795+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:11:15.560795+0000","last_undegraded":"2023-10-30T13:11:15.560795+0000","last_fullsized":"2023-10-30T13:11:15.560795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:11.128769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438505+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:11:15.438505+0000","last_peered":"2023-10-30T13:11:15.438505+0000","last_clean":"2023-10-30T13:11:15.438505+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:11:15.438505+0000","last_undegraded":"2023-10-30T13:11:15.438505+0000","last_fullsized":"2023-10-30T13:11:15.438505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531029+0000","last_change":"2023-10-30T13:10:34.376512+0000","last_active":"2023-10-30T13:11:15.531029+0000","last_peered":"2023-10-30T13:11:15.531029+0000","last_clean":"2023-10-30T13:11:15.531029+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:11:15.531029+0000","last_undegraded":"2023-10-30T13:11:15.531029+0000","last_fullsized":"2023-10-30T13:11:15.531029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:40.856690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446249+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:11:15.446249+0000","last_peered":"2023-10-30T13:11:15.446249+0000","last_clean":"2023-10-30T13:11:15.446249+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:11:15.446249+0000","last_undegraded":"2023-10-30T13:11:15.446249+0000","last_fullsized":"2023-10-30T13:11:15.446249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.443575+0000","last_change":"2023-10-30T13:10:34.344863+0000","last_active":"2023-10-30T13:11:15.443575+0000","last_peered":"2023-10-30T13:11:15.443575+0000","last_clean":"2023-10-30T13:11:15.443575+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:11:15.443575+0000","last_undegraded":"2023-10-30T13:11:15.443575+0000","last_fullsized":"2023-10-30T13:11:15.443575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:25.199813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'15","reported_seq":30,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.432685+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:11:15.432685+0000","last_peered":"2023-10-30T13:11:15.432685+0000","last_clean":"2023-10-30T13:11:15.432685+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:11:15.432685+0000","last_undegraded":"2023-10-30T13:11:15.432685+0000","last_fullsized":"2023-10-30T13:11:15.432685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530363+0000","last_change":"2023-10-30T13:10:34.371813+0000","last_active":"2023-10-30T13:11:15.530363+0000","last_peered":"2023-10-30T13:11:15.530363+0000","last_clean":"2023-10-30T13:11:15.530363+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:11:15.530363+0000","last_undegraded":"2023-10-30T13:11:15.530363+0000","last_fullsized":"2023-10-30T13:11:15.530363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:58.719747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440825+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:11:15.440825+0000","last_peered":"2023-10-30T13:11:15.440825+0000","last_clean":"2023-10-30T13:11:15.440825+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:11:15.440825+0000","last_undegraded":"2023-10-30T13:11:15.440825+0000","last_fullsized":"2023-10-30T13:11:15.440825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530665+0000","last_change":"2023-10-30T13:10:34.375164+0000","last_active":"2023-10-30T13:11:15.530665+0000","last_peered":"2023-10-30T13:11:15.530665+0000","last_clean":"2023-10-30T13:11:15.530665+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:11:15.530665+0000","last_undegraded":"2023-10-30T13:11:15.530665+0000","last_fullsized":"2023-10-30T13:11:15.530665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:05.305348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433212+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:11:15.433212+0000","last_peered":"2023-10-30T13:11:15.433212+0000","last_clean":"2023-10-30T13:11:15.433212+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:11:15.433212+0000","last_undegraded":"2023-10-30T13:11:15.433212+0000","last_fullsized":"2023-10-30T13:11:15.433212+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531192+0000","last_change":"2023-10-30T13:10:34.382785+0000","last_active":"2023-10-30T13:11:15.531192+0000","last_peered":"2023-10-30T13:11:15.531192+0000","last_clean":"2023-10-30T13:11:15.531192+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:11:15.531192+0000","last_undegraded":"2023-10-30T13:11:15.531192+0000","last_fullsized":"2023-10-30T13:11:15.531192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:50.203770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434778+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:11:15.434778+0000","last_peered":"2023-10-30T13:11:15.434778+0000","last_clean":"2023-10-30T13:11:15.434778+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:11:15.434778+0000","last_undegraded":"2023-10-30T13:11:15.434778+0000","last_fullsized":"2023-10-30T13:11:15.434778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.439160+0000","last_change":"2023-10-30T13:10:34.342901+0000","last_active":"2023-10-30T13:11:15.439160+0000","last_peered":"2023-10-30T13:11:15.439160+0000","last_clean":"2023-10-30T13:11:15.439160+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:11:15.439160+0000","last_undegraded":"2023-10-30T13:11:15.439160+0000","last_fullsized":"2023-10-30T13:11:15.439160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:27.227907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433265+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:11:15.433265+0000","last_peered":"2023-10-30T13:11:15.433265+0000","last_clean":"2023-10-30T13:11:15.433265+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:11:15.433265+0000","last_undegraded":"2023-10-30T13:11:15.433265+0000","last_fullsized":"2023-10-30T13:11:15.433265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530590+0000","last_change":"2023-10-30T13:10:34.373854+0000","last_active":"2023-10-30T13:11:15.530590+0000","last_peered":"2023-10-30T13:11:15.530590+0000","last_clean":"2023-10-30T13:11:15.530590+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:11:15.530590+0000","last_undegraded":"2023-10-30T13:11:15.530590+0000","last_fullsized":"2023-10-30T13:11:15.530590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:12.702507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530831+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:11:15.530831+0000","last_peered":"2023-10-30T13:11:15.530831+0000","last_clean":"2023-10-30T13:11:15.530831+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:11:15.530831+0000","last_undegraded":"2023-10-30T13:11:15.530831+0000","last_fullsized":"2023-10-30T13:11:15.530831+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438091+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:11:15.438091+0000","last_peered":"2023-10-30T13:11:15.438091+0000","last_clean":"2023-10-30T13:11:15.438091+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:11:15.438091+0000","last_undegraded":"2023-10-30T13:11:15.438091+0000","last_fullsized":"2023-10-30T13:11:15.438091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430506+0000","last_change":"2023-10-30T13:10:34.348470+0000","last_active":"2023-10-30T13:11:15.430506+0000","last_peered":"2023-10-30T13:11:15.430506+0000","last_clean":"2023-10-30T13:11:15.430506+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:11:15.430506+0000","last_undegraded":"2023-10-30T13:11:15.430506+0000","last_fullsized":"2023-10-30T13:11:15.430506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:24.818359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.530008+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:11:15.530008+0000","last_peered":"2023-10-30T13:11:15.530008+0000","last_clean":"2023-10-30T13:11:15.530008+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:11:15.530008+0000","last_undegraded":"2023-10-30T13:11:15.530008+0000","last_fullsized":"2023-10-30T13:11:15.530008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.560701+0000","last_change":"2023-10-30T13:10:34.359740+0000","last_active":"2023-10-30T13:11:15.560701+0000","last_peered":"2023-10-30T13:11:15.560701+0000","last_clean":"2023-10-30T13:11:15.560701+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:11:15.560701+0000","last_undegraded":"2023-10-30T13:11:15.560701+0000","last_fullsized":"2023-10-30T13:11:15.560701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:05:25.883485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:12:42.307293+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:12:42.307293+0000","last_peered":"2023-10-30T13:12:42.307293+0000","last_clean":"2023-10-30T13:12:42.307293+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:12:42.307293+0000","last_undegraded":"2023-10-30T13:12:42.307293+0000","last_fullsized":"2023-10-30T13:12:42.307293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433030+0000","last_change":"2023-10-30T13:10:34.332955+0000","last_active":"2023-10-30T13:11:15.433030+0000","last_peered":"2023-10-30T13:11:15.433030+0000","last_clean":"2023-10-30T13:11:15.433030+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:11:15.433030+0000","last_undegraded":"2023-10-30T13:11:15.433030+0000","last_fullsized":"2023-10-30T13:11:15.433030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:05.776190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.438730+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:11:15.438730+0000","last_peered":"2023-10-30T13:11:15.438730+0000","last_clean":"2023-10-30T13:11:15.438730+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:11:15.438730+0000","last_undegraded":"2023-10-30T13:11:15.438730+0000","last_fullsized":"2023-10-30T13:11:15.438730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440305+0000","last_change":"2023-10-30T13:10:34.362313+0000","last_active":"2023-10-30T13:11:15.440305+0000","last_peered":"2023-10-30T13:11:15.440305+0000","last_clean":"2023-10-30T13:11:15.440305+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:11:15.440305+0000","last_undegraded":"2023-10-30T13:11:15.440305+0000","last_fullsized":"2023-10-30T13:11:15.440305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:20:45.213916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434563+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:11:15.434563+0000","last_peered":"2023-10-30T13:11:15.434563+0000","last_clean":"2023-10-30T13:11:15.434563+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:11:15.434563+0000","last_undegraded":"2023-10-30T13:11:15.434563+0000","last_fullsized":"2023-10-30T13:11:15.434563+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.439222+0000","last_change":"2023-10-30T13:10:34.348653+0000","last_active":"2023-10-30T13:11:15.439222+0000","last_peered":"2023-10-30T13:11:15.439222+0000","last_clean":"2023-10-30T13:11:15.439222+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:11:15.439222+0000","last_undegraded":"2023-10-30T13:11:15.439222+0000","last_fullsized":"2023-10-30T13:11:15.439222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:54:00.948062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531132+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:11:15.531132+0000","last_peered":"2023-10-30T13:11:15.531132+0000","last_clean":"2023-10-30T13:11:15.531132+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:11:15.531132+0000","last_undegraded":"2023-10-30T13:11:15.531132+0000","last_fullsized":"2023-10-30T13:11:15.531132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433306+0000","last_change":"2023-10-30T13:10:34.332001+0000","last_active":"2023-10-30T13:11:15.433306+0000","last_peered":"2023-10-30T13:11:15.433306+0000","last_clean":"2023-10-30T13:11:15.433306+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:11:15.433306+0000","last_undegraded":"2023-10-30T13:11:15.433306+0000","last_fullsized":"2023-10-30T13:11:15.433306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:16.341054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.434819+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:11:15.434819+0000","last_peered":"2023-10-30T13:11:15.434819+0000","last_clean":"2023-10-30T13:11:15.434819+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:11:15.434819+0000","last_undegraded":"2023-10-30T13:11:15.434819+0000","last_fullsized":"2023-10-30T13:11:15.434819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430358+0000","last_change":"2023-10-30T13:10:34.369845+0000","last_active":"2023-10-30T13:11:15.430358+0000","last_peered":"2023-10-30T13:11:15.430358+0000","last_clean":"2023-10-30T13:11:15.430358+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:11:15.430358+0000","last_undegraded":"2023-10-30T13:11:15.430358+0000","last_fullsized":"2023-10-30T13:11:15.430358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:30.468845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.447484+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:11:15.447484+0000","last_peered":"2023-10-30T13:11:15.447484+0000","last_clean":"2023-10-30T13:11:15.447484+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:11:15.447484+0000","last_undegraded":"2023-10-30T13:11:15.447484+0000","last_fullsized":"2023-10-30T13:11:15.447484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561059+0000","last_change":"2023-10-30T13:10:34.323336+0000","last_active":"2023-10-30T13:11:15.561059+0000","last_peered":"2023-10-30T13:11:15.561059+0000","last_clean":"2023-10-30T13:11:15.561059+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:11:15.561059+0000","last_undegraded":"2023-10-30T13:11:15.561059+0000","last_fullsized":"2023-10-30T13:11:15.561059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:21.642241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.529931+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:11:15.529931+0000","last_peered":"2023-10-30T13:11:15.529931+0000","last_clean":"2023-10-30T13:11:15.529931+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:11:15.529931+0000","last_undegraded":"2023-10-30T13:11:15.529931+0000","last_fullsized":"2023-10-30T13:11:15.529931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.440140+0000","last_change":"2023-10-30T13:10:34.363313+0000","last_active":"2023-10-30T13:11:15.440140+0000","last_peered":"2023-10-30T13:11:15.440140+0000","last_clean":"2023-10-30T13:11:15.440140+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:11:15.440140+0000","last_undegraded":"2023-10-30T13:11:15.440140+0000","last_fullsized":"2023-10-30T13:11:15.440140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:18:13.027610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433415+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:11:15.433415+0000","last_peered":"2023-10-30T13:11:15.433415+0000","last_clean":"2023-10-30T13:11:15.433415+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:11:15.433415+0000","last_undegraded":"2023-10-30T13:11:15.433415+0000","last_fullsized":"2023-10-30T13:11:15.433415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430057+0000","last_change":"2023-10-30T13:10:34.330677+0000","last_active":"2023-10-30T13:11:15.430057+0000","last_peered":"2023-10-30T13:11:15.430057+0000","last_clean":"2023-10-30T13:11:15.430057+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:11:15.430057+0000","last_undegraded":"2023-10-30T13:11:15.430057+0000","last_fullsized":"2023-10-30T13:11:15.430057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:14:01.360813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.433573+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:11:15.433573+0000","last_peered":"2023-10-30T13:11:15.433573+0000","last_clean":"2023-10-30T13:11:15.433573+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:11:15.433573+0000","last_undegraded":"2023-10-30T13:11:15.433573+0000","last_fullsized":"2023-10-30T13:11:15.433573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430362+0000","last_change":"2023-10-30T13:10:34.331117+0000","last_active":"2023-10-30T13:11:15.430362+0000","last_peered":"2023-10-30T13:11:15.430362+0000","last_clean":"2023-10-30T13:11:15.430362+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:11:15.430362+0000","last_undegraded":"2023-10-30T13:11:15.430362+0000","last_fullsized":"2023-10-30T13:11:15.430362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:56.948470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.531341+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:11:15.531341+0000","last_peered":"2023-10-30T13:11:15.531341+0000","last_clean":"2023-10-30T13:11:15.531341+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:11:15.531341+0000","last_undegraded":"2023-10-30T13:11:15.531341+0000","last_fullsized":"2023-10-30T13:11:15.531341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.446526+0000","last_change":"2023-10-30T13:10:34.365315+0000","last_active":"2023-10-30T13:11:15.446526+0000","last_peered":"2023-10-30T13:11:15.446526+0000","last_clean":"2023-10-30T13:11:15.446526+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:11:15.446526+0000","last_undegraded":"2023-10-30T13:11:15.446526+0000","last_fullsized":"2023-10-30T13:11:15.446526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:37.078775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.442500+0000","last_change":"2023-10-30T13:10:34.366044+0000","last_active":"2023-10-30T13:11:14.442500+0000","last_peered":"2023-10-30T13:11:14.442500+0000","last_clean":"2023-10-30T13:11:14.442500+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:11:14.442500+0000","last_undegraded":"2023-10-30T13:11:14.442500+0000","last_fullsized":"2023-10-30T13:11:14.442500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:31.831020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.442338+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:11:14.442338+0000","last_peered":"2023-10-30T13:11:14.442338+0000","last_clean":"2023-10-30T13:11:14.442338+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:11:14.442338+0000","last_undegraded":"2023-10-30T13:11:14.442338+0000","last_fullsized":"2023-10-30T13:11:14.442338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.436137+0000","last_change":"2023-10-30T13:10:34.369814+0000","last_active":"2023-10-30T13:11:14.436137+0000","last_peered":"2023-10-30T13:11:14.436137+0000","last_clean":"2023-10-30T13:11:14.436137+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:11:14.436137+0000","last_undegraded":"2023-10-30T13:11:14.436137+0000","last_fullsized":"2023-10-30T13:11:14.436137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:21:26.879717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T13:11:14.441916+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:11:14.441916+0000","last_peered":"2023-10-30T13:11:14.441916+0000","last_clean":"2023-10-30T13:11:14.441916+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:11:14.441916+0000","last_undegraded":"2023-10-30T13:11:14.441916+0000","last_fullsized":"2023-10-30T13:11:14.441916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561102+0000","last_change":"2023-10-30T13:10:34.344766+0000","last_active":"2023-10-30T13:11:15.561102+0000","last_peered":"2023-10-30T13:11:15.561102+0000","last_clean":"2023-10-30T13:11:15.561102+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:11:15.561102+0000","last_undegraded":"2023-10-30T13:11:15.561102+0000","last_fullsized":"2023-10-30T13:11:15.561102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:42:11.073346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430538+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:11:15.430538+0000","last_peered":"2023-10-30T13:11:15.430538+0000","last_clean":"2023-10-30T13:11:15.430538+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:11:15.430538+0000","last_undegraded":"2023-10-30T13:11:15.430538+0000","last_fullsized":"2023-10-30T13:11:15.430538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.430467+0000","last_change":"2023-10-30T13:10:34.354305+0000","last_active":"2023-10-30T13:11:15.430467+0000","last_peered":"2023-10-30T13:11:15.430467+0000","last_clean":"2023-10-30T13:11:15.430467+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:11:15.430467+0000","last_undegraded":"2023-10-30T13:11:15.430467+0000","last_fullsized":"2023-10-30T13:11:15.430467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:53.449789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'9","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T13:11:15.561228+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:11:15.561228+0000","last_peered":"2023-10-30T13:11:15.561228+0000","last_clean":"2023-10-30T13:11:15.561228+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:11:15.561228+0000","last_undegraded":"2023-10-30T13:11:15.561228+0000","last_fullsized":"2023-10-30T13:11:15.561228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":843776,"data_stored":675004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.81999999999999995,"15min":0.81399999999999995},"min":{"1min":0.57099999999999995,"5min":0.496,"15min":0.496},"max":{"1min":1.159,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.85899999999999999,"15min":0.84899999999999998},"min":{"1min":0.54000000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.284,"5min":1.379,"15min":1.379},"last":1.071}]},{"osd":1,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.94299999999999995,"15min":0.95499999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.3879999999999999,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.97699999999999998,"15min":0.997},"min":{"1min":0.64100000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.052,"5min":4.5229999999999997,"15min":4.5229999999999997},"last":1.0369999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.87,"15min":0.88400000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.0369999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":1.111},{"interface":"front","average":{"1min":0.995,"5min":1.0149999999999999,"15min":1.018},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.306,"5min":1.359,"15min":1.359},"last":0.79800000000000004}]},{"osd":3,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.98199999999999998,"15min":0.98799999999999999},"min":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1499999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.075,"15min":1.083},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.2529999999999999,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":1.1499999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.085,"15min":1.1120000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.306,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.02,"5min":1.071,"15min":1.079},"min":{"1min":0.67500000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3169999999999999,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.1870000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1779999999999999,"15min":1.204},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.3049999999999999,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.22},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.2030000000000001,"15min":1.2350000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.208,"5min":4.9269999999999996,"15min":4.9269999999999996},"last":0.96599999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.153,"15min":1.1870000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2350000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.3160000000000001,"15min":1.3640000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.2410000000000001,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.83499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":85610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.0980000000000001,"15min":1.1399999999999999},"min":{"1min":0.61499999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0429999999999999,"5min":4.1699999999999999,"15min":4.1699999999999999},"last":0.62},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.93799999999999994,"15min":0.94599999999999995},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0860000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.94899999999999995}]},{"osd":1,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.625,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1279999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.92800000000000005,"15min":0.93100000000000005},"min":{"1min":0.67300000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.177,"5min":1.458,"15min":1.458},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.1870000000000001,"15min":1.2270000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1519999999999999,"5min":5.4269999999999996,"15min":5.4269999999999996},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.034,"15min":1.048},"min":{"1min":0.63600000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.258,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.82699999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.0269999999999999,"15min":1.044},"min":{"1min":0.629,"5min":0.622,"15min":0.622},"max":{"1min":1.181,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":1.052},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.92200000000000004,"15min":0.93200000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.093,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.76900000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.008,"15min":1.014},"min":{"1min":0.65700000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.359,"5min":1.546,"15min":1.546},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.044,"15min":1.0429999999999999},"min":{"1min":0.69999999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.67800000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.024,"15min":1.026},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.1970000000000001,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.167,"15min":1.177},"min":{"1min":0.78500000000000003,"5min":0.749,"15min":0.749},"max":{"1min":1.423,"5min":1.653,"15min":1.653},"last":1.0760000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:24 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.177,"15min":1.179},"min":{"1min":0.83899999999999997,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.091,"5min":1.1399999999999999,"15min":1.1479999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.46,"5min":1.704,"15min":1.704},"last":1.127}]}]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":84698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.012,"5min":1.155,"15min":1.155},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.80000000000000004,"15min":0.78400000000000003},"min":{"1min":0.59899999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.079}]},{"osd":1,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.82299999999999995,"15min":0.81100000000000005},"min":{"1min":0.58799999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0820000000000001,"5min":1.24,"15min":1.24},"last":1.0600000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.64400000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.157,"5min":1.258,"15min":1.258},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.95699999999999996,"15min":0.96699999999999997},"min":{"1min":0.67900000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1699999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.03}]},{"osd":3,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.95599999999999996,"15min":0.95699999999999996},"min":{"1min":0.73899999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.1739999999999999,"5min":1.218,"15min":1.218},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.028,"15min":1.03},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.2849999999999999,"5min":1.601,"15min":1.601},"last":1.1479999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94999999999999996,"15min":0.96999999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.097,"5min":1.379,"15min":1.379},"last":0.751},{"interface":"front","average":{"1min":1.147,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"max":{"1min":1.3320000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.24}]},{"osd":6,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.131,"15min":1.147},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.194,"5min":1.875,"15min":1.875},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.042,"15min":1.0580000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.2230000000000001,"5min":1.554,"15min":1.554},"last":1.117}]},{"osd":7,"last update":"Mon Oct 30 13:12:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.083,"15min":1.083},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.3480000000000001,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.171},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.175,"15min":1.1850000000000001},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.401,"5min":1.821,"15min":1.821},"last":1.2769999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":84034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.91600000000000004,"15min":0.92400000000000004},"min":{"1min":0.53900000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2609999999999999,"5min":3.6469999999999998,"15min":3.6469999999999998},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.90000000000000002,"15min":0.89200000000000002},"min":{"1min":0.54600000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.351,"5min":1.669,"15min":1.669},"last":0.55500000000000005}]},{"osd":1,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.2330000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.90100000000000002,"15min":0.89100000000000001},"min":{"1min":0.59199999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.242,"5min":1.413,"15min":1.413},"last":0.83199999999999996}]},{"osd":2,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93400000000000005,"15min":0.94199999999999995},"min":{"1min":0.59499999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.302,"5min":3.6840000000000002,"15min":3.6840000000000002},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93000000000000005,"15min":0.92200000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.90000000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.95099999999999996,"15min":0.93899999999999995},"min":{"1min":0.69699999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2749999999999999,"5min":1.321,"15min":1.321},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.88200000000000001,"15min":0.877},"min":{"1min":0.58599999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.145,"5min":1.46,"15min":1.46},"last":1}]},{"osd":5,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.004,"15min":1.006},"min":{"1min":0.64500000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3620000000000001,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0229999999999999,"15min":1.02},"min":{"1min":0.72599999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.3220000000000001,"5min":1.514,"15min":1.514},"last":1.1379999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.92900000000000005,"15min":0.92300000000000004},"min":{"1min":0.70799999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.091},{"interface":"front","average":{"1min":1.024,"5min":1.1000000000000001,"15min":1.1120000000000001},"min":{"1min":0.749,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.2789999999999999,"5min":2.0379999999999998,"15min":2.0379999999999998},"last":1.0680000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:26 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.72399999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.569,"5min":1.744,"15min":1.744},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.6000000000000001,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":1.0429999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":86717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.59799999999999998,"15min":0.61899999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65200000000000002,"5min":1.923,"15min":1.923},"last":0.308},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.66600000000000004,"15min":0.68799999999999994},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70599999999999996,"5min":2.085,"15min":2.085},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.65900000000000003,"15min":0.68000000000000005},"min":{"1min":0.33500000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73799999999999999,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.38100000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.754,"15min":0.78800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.751,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.63800000000000001,"15min":0.64600000000000002},"min":{"1min":0.41999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79400000000000004,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.91200000000000003,"15min":0.92300000000000004},"min":{"1min":0.72699999999999998,"5min":0.627,"15min":0.627},"max":{"1min":1.0720000000000001,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.876,"5min":0.95999999999999996,"15min":0.97399999999999998},"min":{"1min":0.63300000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.0569999999999999,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":0.68300000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.996,"15min":1.026},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.98699999999999999,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.97899999999999998,"15min":0.98499999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1439999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.96499999999999997,"15min":0.98299999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.111,"5min":2.0609999999999999,"15min":2.0609999999999999},"last":0.623},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.97299999999999998,"15min":0.98799999999999999},"min":{"1min":0.66200000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.149,"5min":1.839,"15min":1.839},"last":0.63700000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.0009999999999999,"15min":1.0169999999999999},"min":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.1160000000000001,"5min":2.117,"15min":2.117},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.026,"15min":1.048},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.1100000000000001,"5min":2.2229999999999999,"15min":2.2229999999999999},"last":0.65000000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":84138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48599999999999999,"15min":0.48999999999999999},"min":{"1min":0.33400000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.436},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.38500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58399999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70299999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.48899999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.34999999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67200000000000004,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60299999999999998,"15min":0.60699999999999998},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.755,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.80900000000000005,"15min":0.79900000000000004},"min":{"1min":0.70099999999999996,"5min":0.629,"15min":0.629},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.81999999999999995,"15min":0.81799999999999995},"min":{"1min":0.67500000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0469999999999999,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.78500000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85399999999999998},"min":{"1min":0.68200000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":0.98599999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.90900000000000003,"15min":0.91900000000000004},"min":{"1min":0.68300000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.1220000000000001,"5min":1.339,"15min":1.339},"last":0.68300000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.84499999999999997,"15min":0.84199999999999997},"min":{"1min":0.68300000000000005,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.0449999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.628},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.93000000000000005,"15min":0.94499999999999995},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":0.95399999999999996,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.82599999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98999999999999999,"15min":1},"min":{"1min":0.74099999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.091,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.94699999999999995,"15min":0.95699999999999996},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.075,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.70599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":84264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72999999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.502,"15min":0.502},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.60799999999999998,"5min":1.006,"15min":1.006},"last":0.59999999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.51300000000000001,"15min":0.52200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63300000000000001,"5min":1.095,"15min":1.095},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.39400000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.63,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.70599999999999996,"15min":0.72899999999999998},"min":{"1min":0.442,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.69399999999999995,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.39800000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67800000000000005,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.42099999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.63300000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.0800000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.94499999999999995,"15min":0.96899999999999997},"min":{"1min":0.67300000000000004,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.03,"5min":4.093,"15min":4.093},"last":0.94899999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85099999999999998,"15min":0.85399999999999998},"min":{"1min":0.67800000000000005,"5min":0.629,"15min":0.629},"max":{"1min":1.0609999999999999,"5min":1.796,"15min":1.796},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.61399999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.012,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.86799999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.88600000000000001,"15min":0.89800000000000002},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.1919999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.89900000000000002,"15min":0.90300000000000002},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":0.997,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.88300000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":1,"15min":1.028},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.1379999999999999,"5min":4.1470000000000002,"15min":4.1470000000000002},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.90700000000000003,"15min":0.91700000000000004},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.016,"5min":1.976,"15min":1.976},"last":0.80100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":835584,"data_stored":673329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.755,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51300000000000001,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.71199999999999997,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.497}]},{"osd":2,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73599999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.748,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.48199999999999998,"5min":0.434,"15min":0.434},"max":{"1min":0.78000000000000003,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58899999999999997,"15min":0.58699999999999997},"min":{"1min":0.44600000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78900000000000003,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.629,"5min":0.62,"15min":0.62},"max":{"1min":1.0660000000000001,"5min":1.222,"15min":1.222},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.84599999999999997,"15min":0.85199999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.94499999999999995,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.94899999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86599999999999999,"15min":0.86899999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.044,"5min":1.306,"15min":1.306},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.873,"5min":0.90500000000000003,"15min":0.91000000000000003},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.1459999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.88,"15min":0.88600000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.0129999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.878,"5min":0.92000000000000004,"15min":0.92700000000000005},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.1319999999999999,"5min":1.446,"15min":1.446},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.89100000000000001,"15min":0.90300000000000002},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.109,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92700000000000005,"15min":0.93500000000000005},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.0880000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.67200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:13:03.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:13:03.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4fbd3640 1 Processor -- start 2023-10-30T13:13:03.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4fbd3640 1 -- start start 2023-10-30T13:13:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4fbd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e4815d6f0 0x7f0e4815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4fbd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0e480cf160 0x7f0e480cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4fbd3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e480cb800 con 0x7f0e481538a0 2023-10-30T13:13:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4fbd3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e480cb330 con 0x7f0e480cf160 2023-10-30T13:13:03.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4fbd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480cad70 con 0x7f0e4815d6f0 2023-10-30T13:13:03.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4d96f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0e481538a0 0x7f0e480ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:40904/0 (socket says 172.21.15.148:40904) 2023-10-30T13:13:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4d96f640 1 -- 172.21.15.148:0/3510792308 learned_addr learned my addr 172.21.15.148:0/3510792308 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4d16e640 1 --2- 172.21.15.148:0/3510792308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e4815d6f0 0x7f0e4815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.457+0000 7f0e4e170640 1 --2- 172.21.15.148:0/3510792308 >> v2:172.21.15.150:3300/0 conn(0x7f0e480cf160 0x7f0e480cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.461+0000 7f0e4c96d640 1 -- 172.21.15.148:0/3510792308 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969114048 0 0) 0x7f0e480cb800 con 0x7f0e481538a0 2023-10-30T13:13:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.461+0000 7f0e4c96d640 1 -- 172.21.15.148:0/3510792308 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e34003530 con 0x7f0e481538a0 2023-10-30T13:13:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.461+0000 7f0e4e170640 1 -- 172.21.15.148:0/3510792308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e4815d6f0 msgr2=0x7f0e4815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.461+0000 7f0e4e170640 1 --2- 172.21.15.148:0/3510792308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e4815d6f0 0x7f0e4815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.461+0000 7f0e4e170640 1 -- 172.21.15.148:0/3510792308 >> v1:172.21.15.150:6789/0 conn(0x7f0e481538a0 legacy=0x7f0e480ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.461+0000 7f0e4e170640 1 -- 172.21.15.148:0/3510792308 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e480cb050 con 0x7f0e480cf160 2023-10-30T13:13:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.461+0000 7f0e4d16e640 1 --2- 172.21.15.148:0/3510792308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e4815d6f0 0x7f0e4815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:13:03.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.461+0000 7f0e4e170640 1 --2- 172.21.15.148:0/3510792308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e480cf160 0x7f0e480cf590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e40009d90 tx=0x7f0e4001f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a51529ed8caf443 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.461+0000 7f0e4c96d640 1 -- 172.21.15.148:0/3510792308 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e400af600 con 0x7f0e480cf160 2023-10-30T13:13:03.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4c96d640 1 -- 172.21.15.148:0/3510792308 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e400ab050 con 0x7f0e480cf160 2023-10-30T13:13:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4c96d640 1 -- 172.21.15.148:0/3510792308 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e400af980 con 0x7f0e480cf160 2023-10-30T13:13:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3510792308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e480cf160 msgr2=0x7f0e480cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 --2- 172.21.15.148:0/3510792308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e480cf160 0x7f0e480cf590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e40009d90 tx=0x7f0e4001f130 comp rx=0 tx=0).stop 2023-10-30T13:13:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3510792308 shutdown_connections 2023-10-30T13:13:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 --2- 172.21.15.148:0/3510792308 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e4815d6f0 0x7f0e4815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 --2- 172.21.15.148:0/3510792308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e480cf160 0x7f0e480cf590 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3510792308 >> 172.21.15.148:0/3510792308 conn(0x7f0e4806bfe0 msgr2=0x7f0e4806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3510792308 shutdown_connections 2023-10-30T13:13:03.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3510792308 wait complete. 2023-10-30T13:13:03.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 Processor -- start 2023-10-30T13:13:03.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 -- start start 2023-10-30T13:13:03.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e480cf160 0x7f0e480c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e481538a0 0x7f0e481f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e4815d6f0 0x7f0e481e65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e481f28e0 con 0x7f0e480cf160 2023-10-30T13:13:03.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e480cd130 con 0x7f0e4815d6f0 2023-10-30T13:13:03.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4fbd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e481f32b0 con 0x7f0e481538a0 2023-10-30T13:13:03.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4d96f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e480cf160 0x7f0e480c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4d96f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e480cf160 0x7f0e480c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38388/0 (socket says 172.21.15.148:38388) 2023-10-30T13:13:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4d96f640 1 -- 172.21.15.148:0/3676716424 learned_addr learned my addr 172.21.15.148:0/3676716424 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4e170640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e4815d6f0 0x7f0e481e65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.465+0000 7f0e4d16e640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e481538a0 0x7f0e481f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4d96f640 1 -- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e481538a0 msgr2=0x7f0e481f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4d96f640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e481538a0 0x7f0e481f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4d96f640 1 -- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e4815d6f0 msgr2=0x7f0e481e65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4d96f640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e4815d6f0 0x7f0e481e65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4d96f640 1 -- 172.21.15.148:0/3676716424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e38009420 con 0x7f0e480cf160 2023-10-30T13:13:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4d96f640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e480cf160 0x7f0e480c65d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400c4d0 tx=0x7f0e4400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e3e7fc640 1 -- 172.21.15.148:0/3676716424 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e44005cf0 con 0x7f0e480cf160 2023-10-30T13:13:03.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e3e7fc640 1 -- 172.21.15.148:0/3676716424 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e44005e50 con 0x7f0e480cf160 2023-10-30T13:13:03.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e3e7fc640 1 -- 172.21.15.148:0/3676716424 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e44007270 con 0x7f0e480cf160 2023-10-30T13:13:03.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e40009a70 con 0x7f0e480cf160 2023-10-30T13:13:03.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e480cd6d0 con 0x7f0e480cf160 2023-10-30T13:13:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e480b2a50 con 0x7f0e480cf160 2023-10-30T13:13:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e3e7fc640 1 -- 172.21.15.148:0/3676716424 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e440073d0 con 0x7f0e480cf160 2023-10-30T13:13:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e3e7fc640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e2c0664a0 0x7f0e2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e3e7fc640 1 -- 172.21.15.148:0/3676716424 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0e440833d0 con 0x7f0e480cf160 2023-10-30T13:13:03.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.469+0000 7f0e4d16e640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e2c0664a0 0x7f0e2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.473+0000 7f0e4d16e640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e2c0664a0 0x7f0e2c068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0e480c90b0 tx=0x7f0e38009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.473+0000 7f0e3e7fc640 1 -- 172.21.15.148:0/3676716424 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e44050860 con 0x7f0e480cf160 2023-10-30T13:13:03.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.625+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e480cd8c0 con 0x7f0e480cf160 2023-10-30T13:13:03.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.625+0000 7f0e3e7fc640 1 -- 172.21.15.148:0/3676716424 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16782 (secure 0 0 0) 0x7f0e44053e80 con 0x7f0e480cf160 2023-10-30T13:13:03.630 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:13:03.630 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":22,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:11:15.424384+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:13:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e2c0664a0 msgr2=0x7f0e2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e2c0664a0 0x7f0e2c068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0e480c90b0 tx=0x7f0e38009190 comp rx=0 tx=0).stop 2023-10-30T13:13:03.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e480cf160 msgr2=0x7f0e480c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e480cf160 0x7f0e480c65d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400c4d0 tx=0x7f0e4400c990 comp rx=0 tx=0).stop 2023-10-30T13:13:03.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 shutdown_connections 2023-10-30T13:13:03.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e2c0664a0 0x7f0e2c068960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e4815d6f0 0x7f0e481e65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e481538a0 0x7f0e481f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 --2- 172.21.15.148:0/3676716424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e480cf160 0x7f0e480c65d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 >> 172.21.15.148:0/3676716424 conn(0x7f0e4806bfe0 msgr2=0x7f0e481e2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 shutdown_connections 2023-10-30T13:13:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.629+0000 7f0e4fbd3640 1 -- 172.21.15.148:0/3676716424 wait complete. 2023-10-30T13:13:03.651 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:10:22.349322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:10:32.611151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:10:33.044931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:13:03.652 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:13:03.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.789+0000 7f9b2359e640 1 Processor -- start 2023-10-30T13:13:03.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.789+0000 7f9b2359e640 1 -- start start 2023-10-30T13:13:03.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2359e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b24153840 0x7f9b2415a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2359e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b24153c20 0x7f9b240c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2359e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2415a6b0 con 0x7f9b240cb4c0 2023-10-30T13:13:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2359e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2415dbc0 con 0x7f9b24153840 2023-10-30T13:13:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2359e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2415d730 con 0x7f9b24153c20 2023-10-30T13:13:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b24153840 0x7f9b2415a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b24153840 0x7f9b2415a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38398/0 (socket says 172.21.15.148:38398) 2023-10-30T13:13:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2259c640 1 -- 172.21.15.148:0/1689235130 learned_addr learned my addr 172.21.15.148:0/1689235130 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b1bfff640 1 --2- 172.21.15.148:0/1689235130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b24153c20 0x7f9b240c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2259c640 1 -- 172.21.15.148:0/1689235130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b24153c20 msgr2=0x7f9b240c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2259c640 1 --2- 172.21.15.148:0/1689235130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b24153c20 0x7f9b240c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2259c640 1 -- 172.21.15.148:0/1689235130 >> v1:172.21.15.150:6789/0 conn(0x7f9b240cb4c0 legacy=0x7f9b240cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2259c640 1 -- 172.21.15.148:0/1689235130 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b240ca8f0 con 0x7f9b24153840 2023-10-30T13:13:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b2259c640 1 --2- 172.21.15.148:0/1689235130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153840 0x7f9b2415a170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009870 tx=0x7f9b1401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28abf1523e3164b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b21d9b640 1 -- 172.21.15.148:0/1689235130 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b140a8a30 con 0x7f9b24153840 2023-10-30T13:13:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b21d9b640 1 -- 172.21.15.148:0/1689235130 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b140a8b90 con 0x7f9b24153840 2023-10-30T13:13:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.793+0000 7f9b21d9b640 1 -- 172.21.15.148:0/1689235130 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b140b0440 con 0x7f9b24153840 2023-10-30T13:13:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 -- 172.21.15.148:0/1689235130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153840 msgr2=0x7f9b2415a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 --2- 172.21.15.148:0/1689235130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153840 0x7f9b2415a170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9b14009870 tx=0x7f9b1401ed30 comp rx=0 tx=0).stop 2023-10-30T13:13:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 -- 172.21.15.148:0/1689235130 shutdown_connections 2023-10-30T13:13:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 --2- 172.21.15.148:0/1689235130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b24153c20 0x7f9b240c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 --2- 172.21.15.148:0/1689235130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153840 0x7f9b2415a170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 -- 172.21.15.148:0/1689235130 >> 172.21.15.148:0/1689235130 conn(0x7f9b2406be40 msgr2=0x7f9b2406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 -- 172.21.15.148:0/1689235130 shutdown_connections 2023-10-30T13:13:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 -- 172.21.15.148:0/1689235130 wait complete. 2023-10-30T13:13:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 Processor -- start 2023-10-30T13:13:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 -- start start 2023-10-30T13:13:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b240cb4c0 0x7f9b241efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b24153840 0x7f9b24157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153c20 0x7f9b24154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24154d50 con 0x7f9b24153c20 2023-10-30T13:13:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b241545a0 con 0x7f9b24153840 2023-10-30T13:13:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2359e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b241557a0 con 0x7f9b240cb4c0 2023-10-30T13:13:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b22d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153c20 0x7f9b24154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b22d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153c20 0x7f9b24154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38414/0 (socket says 172.21.15.148:38414) 2023-10-30T13:13:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b22d9d640 1 -- 172.21.15.148:0/334166062 learned_addr learned my addr 172.21.15.148:0/334166062 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b1bfff640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b24153840 0x7f9b24157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.797+0000 7f9b2259c640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b240cb4c0 0x7f9b241efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b22d9d640 1 -- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b240cb4c0 msgr2=0x7f9b241efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b22d9d640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b240cb4c0 0x7f9b241efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b22d9d640 1 -- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b24153840 msgr2=0x7f9b24157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b22d9d640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b24153840 0x7f9b24157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b22d9d640 1 -- 172.21.15.148:0/334166062 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b140094d0 con 0x7f9b24153c20 2023-10-30T13:13:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b22d9d640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153c20 0x7f9b24154060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c00c770 tx=0x7f9b1c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b1affd640 1 -- 172.21.15.148:0/334166062 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c003b80 con 0x7f9b24153c20 2023-10-30T13:13:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b1affd640 1 -- 172.21.15.148:0/334166062 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1c003ce0 con 0x7f9b24153c20 2023-10-30T13:13:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b1affd640 1 -- 172.21.15.148:0/334166062 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c00aa90 con 0x7f9b24153c20 2023-10-30T13:13:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2415c400 con 0x7f9b24153c20 2023-10-30T13:13:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b241ef080 con 0x7f9b24153c20 2023-10-30T13:13:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2404e9c0 con 0x7f9b24153c20 2023-10-30T13:13:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b1affd640 1 -- 172.21.15.148:0/334166062 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b1c00abf0 con 0x7f9b24153c20 2023-10-30T13:13:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b1affd640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b040664a0 0x7f9b04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.801+0000 7f9b1affd640 1 -- 172.21.15.148:0/334166062 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9b1c083a80 con 0x7f9b24153c20 2023-10-30T13:13:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.805+0000 7f9b2259c640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b040664a0 0x7f9b04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.805+0000 7f9b2259c640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b040664a0 0x7f9b04068960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9b140b2000 tx=0x7f9b14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.805+0000 7f9b1affd640 1 -- 172.21.15.148:0/334166062 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b1c050f10 con 0x7f9b24153c20 2023-10-30T13:13:03.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9b2415a6b0 con 0x7f9b24153c20 2023-10-30T13:13:03.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b1affd640 1 -- 172.21.15.148:0/334166062 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9b2415a6b0 con 0x7f9b24153c20 2023-10-30T13:13:03.961 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2023-10-30T13:13:03.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b040664a0 msgr2=0x7f9b04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b040664a0 0x7f9b04068960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9b140b2000 tx=0x7f9b14009040 comp rx=0 tx=0).stop 2023-10-30T13:13:03.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153c20 msgr2=0x7f9b24154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153c20 0x7f9b24154060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c00c770 tx=0x7f9b1c00cc30 comp rx=0 tx=0).stop 2023-10-30T13:13:03.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 shutdown_connections 2023-10-30T13:13:03.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b040664a0 0x7f9b04068960 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b24153c20 0x7f9b24154060 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b24153840 0x7f9b24157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 --2- 172.21.15.148:0/334166062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b240cb4c0 0x7f9b241efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 >> 172.21.15.148:0/334166062 conn(0x7f9b2406be40 msgr2=0x7f9b24150640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:03.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 shutdown_connections 2023-10-30T13:13:03.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:03.957+0000 7f9b2359e640 1 -- 172.21.15.148:0/334166062 wait complete. 2023-10-30T13:13:03.982 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:13:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae924ea640 1 Processor -- start 2023-10-30T13:13:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae924ea640 1 -- start start 2023-10-30T13:13:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae924ea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fae8c156fd0 0x7fae8c06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae924ea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fae8c06fda0 0x7fae8c071b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae924ea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c06b2e0 con 0x7fae8c072090 2023-10-30T13:13:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae924ea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c06bbe0 con 0x7fae8c06fda0 2023-10-30T13:13:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae924ea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c0724a0 con 0x7fae8c156fd0 2023-10-30T13:13:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae8b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fae8c06fda0 0x7fae8c071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae90a87640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fae8c072090 0x7fae8c1598a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42656/0 (socket says 172.21.15.148:42656) 2023-10-30T13:13:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae90a87640 1 -- 172.21.15.148:0/1263226855 learned_addr learned my addr 172.21.15.148:0/1263226855 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae8bfff640 1 --2- 172.21.15.148:0/1263226855 >> v2:172.21.15.150:3300/0 conn(0x7fae8c156fd0 0x7fae8c06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae8b7fe640 1 -- 172.21.15.148:0/1263226855 >> v2:172.21.15.150:3300/0 conn(0x7fae8c156fd0 msgr2=0x7fae8c06ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:04.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae8b7fe640 1 --2- 172.21.15.148:0/1263226855 >> v2:172.21.15.150:3300/0 conn(0x7fae8c156fd0 0x7fae8c06ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae8b7fe640 1 -- 172.21.15.148:0/1263226855 >> v1:172.21.15.148:6789/0 conn(0x7fae8c072090 legacy=0x7fae8c1598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae8b7fe640 1 -- 172.21.15.148:0/1263226855 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c15d020 con 0x7fae8c06fda0 2023-10-30T13:13:04.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.121+0000 7fae8affd640 1 -- 172.21.15.148:0/1263226855 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198054044 0 0) 0x7fae8c06b2e0 con 0x7fae8c072090 2023-10-30T13:13:04.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae8b7fe640 1 --2- 172.21.15.148:0/1263226855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c06fda0 0x7fae8c071b50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0ca6a0 tx=0x7fae8001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fad5d3a3ee52e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae8affd640 1 -- 172.21.15.148:0/1263226855 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae800a8950 con 0x7fae8c06fda0 2023-10-30T13:13:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae8affd640 1 -- 172.21.15.148:0/1263226855 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae800a8ab0 con 0x7fae8c06fda0 2023-10-30T13:13:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae8affd640 1 -- 172.21.15.148:0/1263226855 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae800a8e40 con 0x7fae8c06fda0 2023-10-30T13:13:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae924ea640 1 -- 172.21.15.148:0/1263226855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c06fda0 msgr2=0x7fae8c071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae924ea640 1 --2- 172.21.15.148:0/1263226855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c06fda0 0x7fae8c071b50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0ca6a0 tx=0x7fae8001ec50 comp rx=0 tx=0).stop 2023-10-30T13:13:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae924ea640 1 -- 172.21.15.148:0/1263226855 shutdown_connections 2023-10-30T13:13:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae924ea640 1 --2- 172.21.15.148:0/1263226855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c06fda0 0x7fae8c071b50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae924ea640 1 --2- 172.21.15.148:0/1263226855 >> v2:172.21.15.150:3300/0 conn(0x7fae8c156fd0 0x7fae8c06ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae924ea640 1 -- 172.21.15.148:0/1263226855 >> 172.21.15.148:0/1263226855 conn(0x7fae8c069e90 msgr2=0x7fae8c06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:04.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae924ea640 1 -- 172.21.15.148:0/1263226855 shutdown_connections 2023-10-30T13:13:04.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.125+0000 7fae924ea640 1 -- 172.21.15.148:0/1263226855 wait complete. 2023-10-30T13:13:04.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 Processor -- start 2023-10-30T13:13:04.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 -- start start 2023-10-30T13:13:04.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae8c06fda0 0x7fae8c1478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c072090 0x7fae8c1f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae8c156fd0 0x7fae8c0c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c157850 con 0x7fae8c072090 2023-10-30T13:13:04.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c15cd70 con 0x7fae8c06fda0 2023-10-30T13:13:04.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c159a70 con 0x7fae8c156fd0 2023-10-30T13:13:04.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c072090 0x7fae8c1f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c072090 0x7fae8c1f2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38420/0 (socket says 172.21.15.148:38420) 2023-10-30T13:13:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8b7fe640 1 -- 172.21.15.148:0/4294251995 learned_addr learned my addr 172.21.15.148:0/4294251995 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8bfff640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae8c06fda0 0x7fae8c1478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae90a87640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae8c156fd0 0x7fae8c0c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8b7fe640 1 -- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae8c156fd0 msgr2=0x7fae8c0c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8b7fe640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae8c156fd0 0x7fae8c0c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8b7fe640 1 -- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae8c06fda0 msgr2=0x7fae8c1478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8b7fe640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae8c06fda0 0x7fae8c1478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8b7fe640 1 -- 172.21.15.148:0/4294251995 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae80009420 con 0x7fae8c072090 2023-10-30T13:13:04.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae90a87640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae8c156fd0 0x7fae8c0c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:13:04.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae8b7fe640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c072090 0x7fae8c1f2530 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fae800b3040 tx=0x7fae8001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:04.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae88ff9640 1 -- 172.21.15.148:0/4294251995 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae800b0dd0 con 0x7fae8c072090 2023-10-30T13:13:04.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae88ff9640 1 -- 172.21.15.148:0/4294251995 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae800b9430 con 0x7fae8c072090 2023-10-30T13:13:04.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c1594f0 con 0x7fae8c072090 2023-10-30T13:13:04.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae88ff9640 1 -- 172.21.15.148:0/4294251995 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae800b97e0 con 0x7fae8c072090 2023-10-30T13:13:04.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c1574e0 con 0x7fae8c072090 2023-10-30T13:13:04.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.129+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8c04e170 con 0x7fae8c072090 2023-10-30T13:13:04.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.133+0000 7fae88ff9640 1 -- 172.21.15.148:0/4294251995 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae800b7040 con 0x7fae8c072090 2023-10-30T13:13:04.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.133+0000 7fae88ff9640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae780664a0 0x7fae78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.133+0000 7fae88ff9640 1 -- 172.21.15.148:0/4294251995 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fae80121b00 con 0x7fae8c072090 2023-10-30T13:13:04.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.133+0000 7fae8bfff640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae780664a0 0x7fae78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.133+0000 7fae8bfff640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae780664a0 0x7fae78068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009a50 tx=0x7fae7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:04.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.137+0000 7fae88ff9640 1 -- 172.21.15.148:0/4294251995 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae80123e60 con 0x7fae8c072090 2023-10-30T13:13:04.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.285+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae8c156800 con 0x7fae8c072090 2023-10-30T13:13:04.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.285+0000 7fae88ff9640 1 -- 172.21.15.148:0/4294251995 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae800b50a0 con 0x7fae8c072090 2023-10-30T13:13:04.289 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 32 2023-10-30T13:13:04.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae780664a0 msgr2=0x7fae78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae780664a0 0x7fae78068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009a50 tx=0x7fae7c008040 comp rx=0 tx=0).stop 2023-10-30T13:13:04.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c072090 msgr2=0x7fae8c1f2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c072090 0x7fae8c1f2530 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fae800b3040 tx=0x7fae8001b9f0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 shutdown_connections 2023-10-30T13:13:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae780664a0 0x7fae78068960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8c072090 0x7fae8c1f2530 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae8c156fd0 0x7fae8c0c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 --2- 172.21.15.148:0/4294251995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae8c06fda0 0x7fae8c1478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 >> 172.21.15.148:0/4294251995 conn(0x7fae8c069e90 msgr2=0x7fae8c15a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:04.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 shutdown_connections 2023-10-30T13:13:04.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.289+0000 7fae924ea640 1 -- 172.21.15.148:0/4294251995 wait complete. 2023-10-30T13:13:04.310 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:13:04.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.449+0000 7f822b3ab640 1 Processor -- start 2023-10-30T13:13:04.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.449+0000 7f822b3ab640 1 -- start start 2023-10-30T13:13:04.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.449+0000 7f822b3ab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f822415d570 0x7f822415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.449+0000 7f822b3ab640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822415a920 con 0x7f8224153980 2023-10-30T13:13:04.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.449+0000 7f822b3ab640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82240cb930 con 0x7f8224159930 2023-10-30T13:13:04.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.449+0000 7f822b3ab640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8224159dc0 con 0x7f822415d570 2023-10-30T13:13:04.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.449+0000 7f8229147640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8224153980 0x7f82240caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42658/0 (socket says 172.21.15.148:42658) 2023-10-30T13:13:04.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.449+0000 7f8229147640 1 -- 172.21.15.148:0/2267689068 learned_addr learned my addr 172.21.15.148:0/2267689068 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:04.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.449+0000 7f8229948640 1 --2- 172.21.15.148:0/2267689068 >> v2:172.21.15.148:3300/0 conn(0x7f822415d570 0x7f822415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f821bfff640 1 -- 172.21.15.148:0/2267689068 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418045737 0 0) 0x7f822415a920 con 0x7f8224153980 2023-10-30T13:13:04.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f821bfff640 1 -- 172.21.15.148:0/2267689068 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8210003530 con 0x7f8224153980 2023-10-30T13:13:04.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f8229948640 1 -- 172.21.15.148:0/2267689068 >> v1:172.21.15.150:6789/0 conn(0x7f8224159930 legacy=0x7f82240cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f8229948640 1 -- 172.21.15.148:0/2267689068 >> v1:172.21.15.148:6789/0 conn(0x7f8224153980 legacy=0x7f82240caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f8229948640 1 -- 172.21.15.148:0/2267689068 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822415a080 con 0x7f822415d570 2023-10-30T13:13:04.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f8229948640 1 --2- 172.21.15.148:0/2267689068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 0x7f822415d9e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f821c009d90 tx=0x7f821c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169474f87ec55225 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:04.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f821bfff640 1 -- 172.21.15.148:0/2267689068 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c0a5900 con 0x7f822415d570 2023-10-30T13:13:04.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f821bfff640 1 -- 172.21.15.148:0/2267689068 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821c0a5a60 con 0x7f822415d570 2023-10-30T13:13:04.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f821bfff640 1 -- 172.21.15.148:0/2267689068 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c0a5df0 con 0x7f822415d570 2023-10-30T13:13:04.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f822b3ab640 1 -- 172.21.15.148:0/2267689068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 msgr2=0x7f822415d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.453+0000 7f822b3ab640 1 --2- 172.21.15.148:0/2267689068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 0x7f822415d9e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f821c009d90 tx=0x7f821c01c950 comp rx=0 tx=0).stop 2023-10-30T13:13:04.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 -- 172.21.15.148:0/2267689068 shutdown_connections 2023-10-30T13:13:04.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 --2- 172.21.15.148:0/2267689068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 0x7f822415d9e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 -- 172.21.15.148:0/2267689068 >> 172.21.15.148:0/2267689068 conn(0x7f822406c0f0 msgr2=0x7f822406c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 -- 172.21.15.148:0/2267689068 shutdown_connections 2023-10-30T13:13:04.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 -- 172.21.15.148:0/2267689068 wait complete. 2023-10-30T13:13:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 Processor -- start 2023-10-30T13:13:04.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 -- start start 2023-10-30T13:13:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8224153980 0x7f8224066220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 0x7f822415c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f822415ca80 0x7f822413fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82240cdc70 con 0x7f822415d570 2023-10-30T13:13:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822413f760 con 0x7f822415ca80 2023-10-30T13:13:04.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f822b3ab640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822414dbf0 con 0x7f8224153980 2023-10-30T13:13:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8228946640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 0x7f822415c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8228946640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 0x7f822415c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38448/0 (socket says 172.21.15.148:38448) 2023-10-30T13:13:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8228946640 1 -- 172.21.15.148:0/1234926287 learned_addr learned my addr 172.21.15.148:0/1234926287 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8229147640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8224153980 0x7f8224066220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8229948640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f822415ca80 0x7f822413fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8228946640 1 -- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8224153980 msgr2=0x7f8224066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8228946640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8224153980 0x7f8224066220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8228946640 1 -- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f822415ca80 msgr2=0x7f822413fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8228946640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f822415ca80 0x7f822413fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8228946640 1 -- 172.21.15.148:0/1234926287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c009a70 con 0x7f822415d570 2023-10-30T13:13:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8229948640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f822415ca80 0x7f822413fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.457+0000 7f8228946640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 0x7f822415c540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f821400c500 tx=0x7f821400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f8219ffb640 1 -- 172.21.15.148:0/1234926287 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214005220 con 0x7f822415d570 2023-10-30T13:13:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f8219ffb640 1 -- 172.21.15.148:0/1234926287 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8214005380 con 0x7f822415d570 2023-10-30T13:13:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f8219ffb640 1 -- 172.21.15.148:0/1234926287 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214007270 con 0x7f822415d570 2023-10-30T13:13:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822415baa0 con 0x7f822415d570 2023-10-30T13:13:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82240ce1a0 con 0x7f822415d570 2023-10-30T13:13:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822404e170 con 0x7f822415d570 2023-10-30T13:13:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f8219ffb640 1 -- 172.21.15.148:0/1234926287 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82140073d0 con 0x7f822415d570 2023-10-30T13:13:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f8219ffb640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82080664a0 0x7f8208068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f8219ffb640 1 -- 172.21.15.148:0/1234926287 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8214083810 con 0x7f822415d570 2023-10-30T13:13:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.461+0000 7f8229147640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82080664a0 0x7f8208068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.465+0000 7f8229147640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82080664a0 0x7f8208068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f82200011f0 tx=0x7f8220007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.465+0000 7f8219ffb640 1 -- 172.21.15.148:0/1234926287 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8214050b70 con 0x7f822415d570 2023-10-30T13:13:04.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f82240caa80 con 0x7f822415d570 2023-10-30T13:13:04.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f8219ffb640 1 -- 172.21.15.148:0/1234926287 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8214054190 con 0x7f822415d570 2023-10-30T13:13:04.620 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 64 2023-10-30T13:13:04.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82080664a0 msgr2=0x7f8208068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82080664a0 0x7f8208068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f82200011f0 tx=0x7f8220007d40 comp rx=0 tx=0).stop 2023-10-30T13:13:04.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 msgr2=0x7f822415c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 0x7f822415c540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f821400c500 tx=0x7f821400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 shutdown_connections 2023-10-30T13:13:04.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82080664a0 0x7f8208068960 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f822415d570 0x7f822415c540 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f822415ca80 0x7f822413fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 --2- 172.21.15.148:0/1234926287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8224153980 0x7f8224066220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 >> 172.21.15.148:0/1234926287 conn(0x7f822406c0f0 msgr2=0x7f82240caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 shutdown_connections 2023-10-30T13:13:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.617+0000 7f822b3ab640 1 -- 172.21.15.148:0/1234926287 wait complete. 2023-10-30T13:13:04.640 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7ad6617640 1 Processor -- start 2023-10-30T13:13:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7ad6617640 1 -- start start 2023-10-30T13:13:04.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7ad6617640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7ad0153d70 0x7f7ad006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7ad6617640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7ad00c8490 0x7f7ad00c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7ad6617640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad006afa0 con 0x7f7ad006e470 2023-10-30T13:13:04.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7ad6617640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad006d780 con 0x7f7ad0153d70 2023-10-30T13:13:04.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7ad6617640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad0154150 con 0x7f7ad00c8490 2023-10-30T13:13:04.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7acffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7ad0153d70 0x7f7ad006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7acffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7ad0153d70 0x7f7ad006ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38456/0 (socket says 172.21.15.148:38456) 2023-10-30T13:13:04.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7acffff640 1 -- 172.21.15.148:0/36482790 learned_addr learned my addr 172.21.15.148:0/36482790 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:04.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7ad4bb4640 1 --2- 172.21.15.148:0/36482790 >> v2:172.21.15.150:3300/0 conn(0x7f7ad00c8490 0x7f7ad00c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7acffff640 1 -- 172.21.15.148:0/36482790 >> v2:172.21.15.150:3300/0 conn(0x7f7ad00c8490 msgr2=0x7f7ad00c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:04.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7acffff640 1 --2- 172.21.15.148:0/36482790 >> v2:172.21.15.150:3300/0 conn(0x7f7ad00c8490 0x7f7ad00c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7acffff640 1 -- 172.21.15.148:0/36482790 >> v1:172.21.15.150:6789/0 conn(0x7f7ad006e470 legacy=0x7f7ad00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.777+0000 7f7acffff640 1 -- 172.21.15.148:0/36482790 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad006e030 con 0x7f7ad0153d70 2023-10-30T13:13:04.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7acffff640 1 --2- 172.21.15.148:0/36482790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad0153d70 0x7f7ad006ca80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00cddf0 tx=0x7f7ac001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91804db3b6dbb12d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:04.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7aceffd640 1 -- 172.21.15.148:0/36482790 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac00aa430 con 0x7f7ad0153d70 2023-10-30T13:13:04.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7aceffd640 1 -- 172.21.15.148:0/36482790 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac00aa590 con 0x7f7ad0153d70 2023-10-30T13:13:04.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7aceffd640 1 -- 172.21.15.148:0/36482790 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac00aa920 con 0x7f7ad0153d70 2023-10-30T13:13:04.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7ad6617640 1 -- 172.21.15.148:0/36482790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad0153d70 msgr2=0x7f7ad006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7ad6617640 1 --2- 172.21.15.148:0/36482790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad0153d70 0x7f7ad006ca80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00cddf0 tx=0x7f7ac001c4d0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7ad6617640 1 -- 172.21.15.148:0/36482790 shutdown_connections 2023-10-30T13:13:04.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7ad6617640 1 --2- 172.21.15.148:0/36482790 >> v2:172.21.15.150:3300/0 conn(0x7f7ad00c8490 0x7f7ad00c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7ad6617640 1 --2- 172.21.15.148:0/36482790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad0153d70 0x7f7ad006ca80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7ad6617640 1 -- 172.21.15.148:0/36482790 >> 172.21.15.148:0/36482790 conn(0x7f7ad0071ea0 msgr2=0x7f7ad0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7ad6617640 1 -- 172.21.15.148:0/36482790 shutdown_connections 2023-10-30T13:13:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.781+0000 7f7ad6617640 1 -- 172.21.15.148:0/36482790 wait complete. 2023-10-30T13:13:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 Processor -- start 2023-10-30T13:13:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 -- start start 2023-10-30T13:13:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad006e470 0x7f7ad00cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad00c8490 0x7f7ad01495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ad0153d70 0x7f7ad0149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad00ccc00 con 0x7f7ad00c8490 2023-10-30T13:13:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad00cbae0 con 0x7f7ad006e470 2023-10-30T13:13:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad014c2c0 con 0x7f7ad0153d70 2023-10-30T13:13:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad00c8490 0x7f7ad01495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad00c8490 0x7f7ad01495b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38466/0 (socket says 172.21.15.148:38466) 2023-10-30T13:13:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acf7fe640 1 -- 172.21.15.148:0/1153023519 learned_addr learned my addr 172.21.15.148:0/1153023519 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acffff640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad006e470 0x7f7ad00cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad4bb4640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ad0153d70 0x7f7ad0149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acf7fe640 1 -- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ad0153d70 msgr2=0x7f7ad0149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acf7fe640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ad0153d70 0x7f7ad0149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acf7fe640 1 -- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad006e470 msgr2=0x7f7ad00cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acf7fe640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad006e470 0x7f7ad00cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acf7fe640 1 -- 172.21.15.148:0/1153023519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac00094d0 con 0x7f7ad00c8490 2023-10-30T13:13:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad4bb4640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ad0153d70 0x7f7ad0149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acffff640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad006e470 0x7f7ad00cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7acf7fe640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad00c8490 0x7f7ad01495b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400c3c0 tx=0x7f7ac400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:04.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7accff9640 1 -- 172.21.15.148:0/1153023519 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac40052d0 con 0x7f7ad00c8490 2023-10-30T13:13:04.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7accff9640 1 -- 172.21.15.148:0/1153023519 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac4005430 con 0x7f7ad00c8490 2023-10-30T13:13:04.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad0143db0 con 0x7f7ad00c8490 2023-10-30T13:13:04.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7accff9640 1 -- 172.21.15.148:0/1153023519 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4003840 con 0x7f7ad00c8490 2023-10-30T13:13:04.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.785+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad00cc640 con 0x7f7ad00c8490 2023-10-30T13:13:04.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.789+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad004e170 con 0x7f7ad00c8490 2023-10-30T13:13:04.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.789+0000 7f7accff9640 1 -- 172.21.15.148:0/1153023519 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7ac40039a0 con 0x7f7ad00c8490 2023-10-30T13:13:04.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.789+0000 7f7accff9640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7abc0664a0 0x7f7abc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:04.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.789+0000 7f7accff9640 1 -- 172.21.15.148:0/1153023519 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7ac4083470 con 0x7f7ad00c8490 2023-10-30T13:13:04.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.789+0000 7f7acffff640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7abc0664a0 0x7f7abc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:04.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.789+0000 7f7acffff640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7abc0664a0 0x7f7abc068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00ae000 tx=0x7f7ac00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:04.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.793+0000 7f7accff9640 1 -- 172.21.15.148:0/1153023519 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ac4050850 con 0x7f7ad00c8490 2023-10-30T13:13:04.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ad006c2a0 con 0x7f7ad00c8490 2023-10-30T13:13:04.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7accff9640 1 -- 172.21.15.148:0/1153023519 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f7ac4053e70 con 0x7f7ad00c8490 2023-10-30T13:13:04.976 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:13:04.976 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:04.976 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:13:04.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7abc0664a0 msgr2=0x7f7abc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7abc0664a0 0x7f7abc068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00ae000 tx=0x7f7ac00a7000 comp rx=0 tx=0).stop 2023-10-30T13:13:04.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad00c8490 msgr2=0x7f7ad01495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:04.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad00c8490 0x7f7ad01495b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400c3c0 tx=0x7f7ac400c880 comp rx=0 tx=0).stop 2023-10-30T13:13:04.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 shutdown_connections 2023-10-30T13:13:04.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7abc0664a0 0x7f7abc068960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ad00c8490 0x7f7ad01495b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ad0153d70 0x7f7ad0149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 --2- 172.21.15.148:0/1153023519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ad006e470 0x7f7ad00cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:04.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 >> 172.21.15.148:0/1153023519 conn(0x7f7ad0071ea0 msgr2=0x7f7ad014e640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:04.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 shutdown_connections 2023-10-30T13:13:04.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:04.973+0000 7f7ad6617640 1 -- 172.21.15.148:0/1153023519 wait complete. 2023-10-30T13:13:04.997 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:13:05.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.129+0000 7f09ee5f8640 1 Processor -- start 2023-10-30T13:13:05.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.129+0000 7f09ee5f8640 1 -- start start 2023-10-30T13:13:05.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09ee5f8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09e806e470 0x7f09e806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09ee5f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e806afa0 con 0x7f09e80c8490 2023-10-30T13:13:05.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09ee5f8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e8154150 con 0x7f09e8153d70 2023-10-30T13:13:05.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09ee5f8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e806d780 con 0x7f09e806e470 2023-10-30T13:13:05.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09ecb95640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f09e80c8490 0x7f09e80c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42666/0 (socket says 172.21.15.148:42666) 2023-10-30T13:13:05.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09ecb95640 1 -- 172.21.15.148:0/3187972928 learned_addr learned my addr 172.21.15.148:0/3187972928 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:13:05.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e77fe640 1 --2- 172.21.15.148:0/3187972928 >> v2:172.21.15.150:3300/0 conn(0x7f09e806e470 0x7f09e806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840851488 0 0) 0x7f09e806afa0 con 0x7f09e80c8490 2023-10-30T13:13:05.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d0003530 con 0x7f09e80c8490 2023-10-30T13:13:05.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 525048163 0 0) 0x7f09d0003530 con 0x7f09e80c8490 2023-10-30T13:13:05.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 >> v2:172.21.15.150:3300/0 conn(0x7f09e806e470 msgr2=0x7f09e806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:05.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 --2- 172.21.15.148:0/3187972928 >> v2:172.21.15.150:3300/0 conn(0x7f09e806e470 0x7f09e806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 >> v1:172.21.15.150:6789/0 conn(0x7f09e8153d70 legacy=0x7f09e80cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e806e030 con 0x7f09e80c8490 2023-10-30T13:13:05.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f09e0001a00 con 0x7f09e80c8490 2023-10-30T13:13:05.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e77fe640 1 --2- 172.21.15.148:0/3187972928 >> v2:172.21.15.150:3300/0 conn(0x7f09e806e470 0x7f09e806ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:13:05.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 >> v1:172.21.15.148:6789/0 conn(0x7f09e80c8490 legacy=0x7f09e80c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d0003980 0x7f09d0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09d00047d0 0x7f09d0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09d0005190 0x7f09d000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e806d780 con 0x7f09d00047d0 2023-10-30T13:13:05.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d000c030 con 0x7f09d0005190 2023-10-30T13:13:05.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d000c1f0 con 0x7f09d0003980 2023-10-30T13:13:05.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e77fe640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09d00047d0 0x7f09d0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e7fff640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d0003980 0x7f09d0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09ecb95640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09d0005190 0x7f09d000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e77fe640 1 -- 172.21.15.148:0/3187972928 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d0003980 msgr2=0x7f09d0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:05.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e77fe640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d0003980 0x7f09d0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e77fe640 1 -- 172.21.15.148:0/3187972928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09d0005190 msgr2=0x7f09d000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:05.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e77fe640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09d0005190 0x7f09d000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09e77fe640 1 -- 172.21.15.148:0/3187972928 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d0003530 con 0x7f09d00047d0 2023-10-30T13:13:05.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09ecb95640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09d0005190 0x7f09d000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:05.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.133+0000 7f09ecb95640 1 -- 172.21.15.148:0/3187972928 reap_dead start 2023-10-30T13:13:05.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09e77fe640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09d00047d0 0x7f09d0004c50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009cb0 tx=0x7f09dc01cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6377047aaa1059 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:05.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc0a58b0 con 0x7f09d00047d0 2023-10-30T13:13:05.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc0a5a10 con 0x7f09d00047d0 2023-10-30T13:13:05.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09e6ffd640 1 -- 172.21.15.148:0/3187972928 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc0a5e00 con 0x7f09d00047d0 2023-10-30T13:13:05.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09ee5f8640 1 -- 172.21.15.148:0/3187972928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09d00047d0 msgr2=0x7f09d0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09ee5f8640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09d00047d0 0x7f09d0004c50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009cb0 tx=0x7f09dc01cbf0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09ee5f8640 1 -- 172.21.15.148:0/3187972928 shutdown_connections 2023-10-30T13:13:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09ee5f8640 1 --2- 172.21.15.148:0/3187972928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09d00047d0 0x7f09d0004c50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09ee5f8640 1 -- 172.21.15.148:0/3187972928 >> 172.21.15.148:0/3187972928 conn(0x7f09e8071ea0 msgr2=0x7f09e8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09ee5f8640 1 -- 172.21.15.148:0/3187972928 shutdown_connections 2023-10-30T13:13:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09ee5f8640 1 -- 172.21.15.148:0/3187972928 wait complete. 2023-10-30T13:13:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.137+0000 7f09ee5f8640 1 Processor -- start 2023-10-30T13:13:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 -- start start 2023-10-30T13:13:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d00047d0 0x7f09e8149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09e814d6e0 0x7f09e814bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09e81efca0 0x7f09e814c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e8154650 con 0x7f09e81efca0 2023-10-30T13:13:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e81548f0 con 0x7f09e814d6e0 2023-10-30T13:13:05.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e8156f80 con 0x7f09d00047d0 2023-10-30T13:13:05.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ecb95640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09e81efca0 0x7f09e814c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ecb95640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09e81efca0 0x7f09e814c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38476/0 (socket says 172.21.15.148:38476) 2023-10-30T13:13:05.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ecb95640 1 -- 172.21.15.148:0/2311559210 learned_addr learned my addr 172.21.15.148:0/2311559210 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:13:05.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09e77fe640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09e814d6e0 0x7f09e814bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09e7fff640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d00047d0 0x7f09e8149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ecb95640 1 -- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d00047d0 msgr2=0x7f09e8149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ecb95640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d00047d0 0x7f09e8149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ecb95640 1 -- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09e814d6e0 msgr2=0x7f09e814bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ecb95640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09e814d6e0 0x7f09e814bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ecb95640 1 -- 172.21.15.148:0/2311559210 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e8157260 con 0x7f09e81efca0 2023-10-30T13:13:05.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09e77fe640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09e814d6e0 0x7f09e814bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:05.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09e7fff640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d00047d0 0x7f09e8149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:05.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ecb95640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09e81efca0 0x7f09e814c910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09e80700c0 tx=0x7f09e0011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:05.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09e4ff9640 1 -- 172.21.15.148:0/2311559210 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e000f8a0 con 0x7f09e81efca0 2023-10-30T13:13:05.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09e4ff9640 1 -- 172.21.15.148:0/2311559210 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e000fa00 con 0x7f09e81efca0 2023-10-30T13:13:05.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e81ef780 con 0x7f09e81efca0 2023-10-30T13:13:05.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09e4ff9640 1 -- 172.21.15.148:0/2311559210 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e000fdb0 con 0x7f09e81efca0 2023-10-30T13:13:05.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e81ef430 con 0x7f09e81efca0 2023-10-30T13:13:05.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.141+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e8067280 con 0x7f09e81efca0 2023-10-30T13:13:05.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.145+0000 7f09e4ff9640 1 -- 172.21.15.148:0/2311559210 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09e001f460 con 0x7f09e81efca0 2023-10-30T13:13:05.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.145+0000 7f09e4ff9640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f09d40664d0 0x7f09d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.145+0000 7f09e4ff9640 1 -- 172.21.15.148:0/2311559210 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f09e0086e30 con 0x7f09e81efca0 2023-10-30T13:13:05.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.145+0000 7f09e7fff640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f09d40664d0 0x7f09d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.145+0000 7f09e7fff640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f09d40664d0 0x7f09d4068990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09d8007fd0 tx=0x7f09d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:05.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.149+0000 7f09e4ff9640 1 -- 172.21.15.148:0/2311559210 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09e0054290 con 0x7f09e81efca0 2023-10-30T13:13:05.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09e8156de0 con 0x7f09e81efca0 2023-10-30T13:13:05.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09e4ff9640 1 -- 172.21.15.148:0/2311559210 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16782 (secure 0 0 0) 0x7f09e8156de0 con 0x7f09e81efca0 2023-10-30T13:13:05.304 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:13:05.304 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":22,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:11:15.424384+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:13:05.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f09d40664d0 msgr2=0x7f09d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f09d40664d0 0x7f09d4068990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09d8007fd0 tx=0x7f09d8009040 comp rx=0 tx=0).stop 2023-10-30T13:13:05.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09e81efca0 msgr2=0x7f09e814c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09e81efca0 0x7f09e814c910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09e80700c0 tx=0x7f09e0011c20 comp rx=0 tx=0).stop 2023-10-30T13:13:05.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 shutdown_connections 2023-10-30T13:13:05.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f09d40664d0 0x7f09d4068990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09e81efca0 0x7f09e814c910 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09e814d6e0 0x7f09e814bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 --2- 172.21.15.148:0/2311559210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09d00047d0 0x7f09e8149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 >> 172.21.15.148:0/2311559210 conn(0x7f09e8071ea0 msgr2=0x7f09e8152e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:05.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 shutdown_connections 2023-10-30T13:13:05.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:13:05.301+0000 7f09ee5f8640 1 -- 172.21.15.148:0/2311559210 wait complete. 2023-10-30T13:13:05.326 DEBUG:tasks.mds_thrash:Yielding 2023-10-30T13:13:05.326 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T13:13:05.341 INFO:tasks.workunit:Pulling workunits from ref 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:13:05.341 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T13:13:05.342 INFO:tasks.workunit:timeout=6h 2023-10-30T13:13:05.342 INFO:tasks.workunit:cleanup=True 2023-10-30T13:13:05.342 DEBUG:teuthology.orchestra.run.smithi148:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T13:13:05.343 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-10-30T13:13:05.343 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 57 secs before thrashing 2023-10-30T13:13:05.353 INFO:teuthology.orchestra.run.smithi148.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-30T13:13:05.353 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T13:13:05.353 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-10-30T13:13:05.353 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T13:13:05.354 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-30T13:13:05.354 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2023-10-30 13:10:39.682176602 +0000 2023-10-30T13:13:05.354 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2023-10-30 13:12:58.537454994 +0000 2023-10-30T13:13:05.354 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2023-10-30 13:10:39.682176602 +0000 2023-10-30T13:13:05.354 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-30T13:13:05.355 DEBUG:teuthology.orchestra.run.smithi148:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-30T13:13:06.299 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T13:13:06.312 INFO:tasks.workunit.client.0.smithi148.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T13:13:23.331 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:13:23.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:13:53.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:13:53.390 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:14:02.345 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc73b46b640 1 Processor -- start 2023-10-30T13:14:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc73b46b640 1 -- start start 2023-10-30T13:14:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc73b46b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc73415d6f0 0x7fc73415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc73b46b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7340cf160 0x7fc7340cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc73b46b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7340cb800 con 0x7fc7341538a0 2023-10-30T13:14:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc73b46b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7340cb330 con 0x7fc73415d6f0 2023-10-30T13:14:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc73b46b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7340cad70 con 0x7fc7340cf160 2023-10-30T13:14:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc739207640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc7341538a0 0x7fc7340ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:56112/0 (socket says 172.21.15.148:56112) 2023-10-30T13:14:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc739207640 1 -- 172.21.15.148:0/1794715429 learned_addr learned my addr 172.21.15.148:0/1794715429 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc739a08640 1 --2- 172.21.15.148:0/1794715429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7340cf160 0x7fc7340cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc738a06640 1 --2- 172.21.15.148:0/1794715429 >> v2:172.21.15.148:3300/0 conn(0x7fc73415d6f0 0x7fc73415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc738a06640 1 -- 172.21.15.148:0/1794715429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7340cf160 msgr2=0x7fc7340cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc738a06640 1 --2- 172.21.15.148:0/1794715429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7340cf160 0x7fc7340cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc738a06640 1 -- 172.21.15.148:0/1794715429 >> v1:172.21.15.150:6789/0 conn(0x7fc7341538a0 legacy=0x7fc7340ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc738a06640 1 -- 172.21.15.148:0/1794715429 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7340cb050 con 0x7fc73415d6f0 2023-10-30T13:14:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.491+0000 7fc72bfff640 1 -- 172.21.15.148:0/1794715429 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575005209 0 0) 0x7fc7340cb800 con 0x7fc7341538a0 2023-10-30T13:14:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc738a06640 1 --2- 172.21.15.148:0/1794715429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc73415d6f0 0x7fc73415a090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc7240097c0 tx=0x7fc72401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e9e3eab33d8658 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc72bfff640 1 -- 172.21.15.148:0/1794715429 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7240a8970 con 0x7fc73415d6f0 2023-10-30T13:14:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc72bfff640 1 -- 172.21.15.148:0/1794715429 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7240a8ad0 con 0x7fc73415d6f0 2023-10-30T13:14:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc72bfff640 1 -- 172.21.15.148:0/1794715429 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7240a8e60 con 0x7fc73415d6f0 2023-10-30T13:14:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc73b46b640 1 -- 172.21.15.148:0/1794715429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc73415d6f0 msgr2=0x7fc73415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc73b46b640 1 --2- 172.21.15.148:0/1794715429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc73415d6f0 0x7fc73415a090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc7240097c0 tx=0x7fc72401ec70 comp rx=0 tx=0).stop 2023-10-30T13:14:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc73b46b640 1 -- 172.21.15.148:0/1794715429 shutdown_connections 2023-10-30T13:14:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc73b46b640 1 --2- 172.21.15.148:0/1794715429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7340cf160 0x7fc7340cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc73b46b640 1 --2- 172.21.15.148:0/1794715429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc73415d6f0 0x7fc73415a090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc73b46b640 1 -- 172.21.15.148:0/1794715429 >> 172.21.15.148:0/1794715429 conn(0x7fc73406bfe0 msgr2=0x7fc73406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc73b46b640 1 -- 172.21.15.148:0/1794715429 shutdown_connections 2023-10-30T13:14:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.499+0000 7fc73b46b640 1 -- 172.21.15.148:0/1794715429 wait complete. 2023-10-30T13:14:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc73b46b640 1 Processor -- start 2023-10-30T13:14:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc73b46b640 1 -- start start 2023-10-30T13:14:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc73b46b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7340cf160 0x7fc7341f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc73b46b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7341538a0 0x7fc7340c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc73b46b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc73415d6f0 0x7fc73414dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc73b46b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7341e5900 con 0x7fc7340cf160 2023-10-30T13:14:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc73b46b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7341f3190 con 0x7fc7341538a0 2023-10-30T13:14:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc73b46b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc734150160 con 0x7fc73415d6f0 2023-10-30T13:14:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc739207640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7340cf160 0x7fc7341f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc739207640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7340cf160 0x7fc7341f2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40966/0 (socket says 172.21.15.148:40966) 2023-10-30T13:14:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc739207640 1 -- 172.21.15.148:0/3578317082 learned_addr learned my addr 172.21.15.148:0/3578317082 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.503+0000 7fc738a06640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7341538a0 0x7fc7340c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc739a08640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc73415d6f0 0x7fc73414dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc739207640 1 -- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc73415d6f0 msgr2=0x7fc73414dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc739207640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc73415d6f0 0x7fc73414dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc739207640 1 -- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7341538a0 msgr2=0x7fc7340c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc739207640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7341538a0 0x7fc7340c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc739207640 1 -- 172.21.15.148:0/3578317082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724009420 con 0x7fc7340cf160 2023-10-30T13:14:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc739207640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7340cf160 0x7fc7341f2c50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7300011f0 tx=0x7fc73000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc729ffb640 1 -- 172.21.15.148:0/3578317082 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7300074e0 con 0x7fc7340cf160 2023-10-30T13:14:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc729ffb640 1 -- 172.21.15.148:0/3578317082 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc730007640 con 0x7fc7340cf160 2023-10-30T13:14:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc729ffb640 1 -- 172.21.15.148:0/3578317082 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc730004a10 con 0x7fc7340cf160 2023-10-30T13:14:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7341e5380 con 0x7fc7340cf160 2023-10-30T13:14:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc734150600 con 0x7fc7340cf160 2023-10-30T13:14:02.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.507+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7340b2a50 con 0x7fc7340cf160 2023-10-30T13:14:02.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.511+0000 7fc729ffb640 1 -- 172.21.15.148:0/3578317082 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc730004b70 con 0x7fc7340cf160 2023-10-30T13:14:02.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.511+0000 7fc729ffb640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7180664d0 0x7fc718068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.511+0000 7fc729ffb640 1 -- 172.21.15.148:0/3578317082 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc730082ff0 con 0x7fc7340cf160 2023-10-30T13:14:02.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.511+0000 7fc738a06640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7180664d0 0x7fc718068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.511+0000 7fc738a06640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7180664d0 0x7fc718068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc7240b2000 tx=0x7fc724008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.519+0000 7fc729ffb640 1 -- 172.21.15.148:0/3578317082 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7300b4090 con 0x7fc7340cf160 2023-10-30T13:14:02.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.711+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc734073bd0 con 0x7fc7340cf160 2023-10-30T13:14:02.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.711+0000 7fc729ffb640 1 -- 172.21.15.148:0/3578317082 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fc730018090 con 0x7fc7340cf160 2023-10-30T13:14:02.718 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:02.718 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:02.719 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:14:02.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7180664d0 msgr2=0x7fc718068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7180664d0 0x7fc718068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc7240b2000 tx=0x7fc724008040 comp rx=0 tx=0).stop 2023-10-30T13:14:02.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7340cf160 msgr2=0x7fc7341f2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7340cf160 0x7fc7341f2c50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7300011f0 tx=0x7fc73000c9b0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 shutdown_connections 2023-10-30T13:14:02.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7180664d0 0x7fc718068990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc73415d6f0 0x7fc73414dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7341538a0 0x7fc7340c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 --2- 172.21.15.148:0/3578317082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7340cf160 0x7fc7341f2c50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 >> 172.21.15.148:0/3578317082 conn(0x7fc73406bfe0 msgr2=0x7fc7340c5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:02.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 shutdown_connections 2023-10-30T13:14:02.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.715+0000 7fc73b46b640 1 -- 172.21.15.148:0/3578317082 wait complete. 2023-10-30T13:14:02.749 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2023-10-30T13:14:02.749 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:14:02.749 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:14:02.767 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:14:02.767 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:02.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.947+0000 7fe59c5a1640 1 Processor -- start 2023-10-30T13:14:02.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.947+0000 7fe59c5a1640 1 -- start start 2023-10-30T13:14:02.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe59c5a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe59415d6f0 0x7fe59415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe59c5a1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940cb800 con 0x7fe5940cf160 2023-10-30T13:14:02.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe59c5a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940cb330 con 0x7fe5941538a0 2023-10-30T13:14:02.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe59c5a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5940cad70 con 0x7fe59415d6f0 2023-10-30T13:14:02.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe59a33d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe5941538a0 0x7fe5940ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:56128/0 (socket says 172.21.15.148:56128) 2023-10-30T13:14:02.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe59a33d640 1 -- 172.21.15.148:0/3391176627 learned_addr learned my addr 172.21.15.148:0/3391176627 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:02.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe599b3c640 1 --2- 172.21.15.148:0/3391176627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe59415d6f0 0x7fe59415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe59933b640 1 -- 172.21.15.148:0/3391176627 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192838172 0 0) 0x7fe5940cb330 con 0x7fe5941538a0 2023-10-30T13:14:02.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe59933b640 1 -- 172.21.15.148:0/3391176627 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe580003530 con 0x7fe5941538a0 2023-10-30T13:14:02.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe599b3c640 1 -- 172.21.15.148:0/3391176627 >> v1:172.21.15.150:6789/0 conn(0x7fe5941538a0 legacy=0x7fe5940ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe599b3c640 1 -- 172.21.15.148:0/3391176627 >> v1:172.21.15.148:6789/0 conn(0x7fe5940cf160 legacy=0x7fe594164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe599b3c640 1 -- 172.21.15.148:0/3391176627 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5940cb050 con 0x7fe59415d6f0 2023-10-30T13:14:02.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.951+0000 7fe59933b640 1 -- 172.21.15.148:0/3391176627 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338924062 0 0) 0x7fe5940cb800 con 0x7fe5940cf160 2023-10-30T13:14:02.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe599b3c640 1 --2- 172.21.15.148:0/3391176627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe59415d6f0 0x7fe59415a090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe5840097f0 tx=0x7fe58401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3246b38cb679c5d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:02.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59933b640 1 -- 172.21.15.148:0/3391176627 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5840a89b0 con 0x7fe59415d6f0 2023-10-30T13:14:02.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59933b640 1 -- 172.21.15.148:0/3391176627 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5840a8b10 con 0x7fe59415d6f0 2023-10-30T13:14:02.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59933b640 1 -- 172.21.15.148:0/3391176627 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5840a8e80 con 0x7fe59415d6f0 2023-10-30T13:14:02.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59c5a1640 1 -- 172.21.15.148:0/3391176627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe59415d6f0 msgr2=0x7fe59415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59c5a1640 1 --2- 172.21.15.148:0/3391176627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe59415d6f0 0x7fe59415a090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe5840097f0 tx=0x7fe58401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59c5a1640 1 -- 172.21.15.148:0/3391176627 shutdown_connections 2023-10-30T13:14:02.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59c5a1640 1 --2- 172.21.15.148:0/3391176627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe59415d6f0 0x7fe59415a090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59c5a1640 1 -- 172.21.15.148:0/3391176627 >> 172.21.15.148:0/3391176627 conn(0x7fe59406bfe0 msgr2=0x7fe59406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:02.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59c5a1640 1 -- 172.21.15.148:0/3391176627 shutdown_connections 2023-10-30T13:14:02.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.955+0000 7fe59c5a1640 1 -- 172.21.15.148:0/3391176627 wait complete. 2023-10-30T13:14:02.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 Processor -- start 2023-10-30T13:14:02.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 -- start start 2023-10-30T13:14:02.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5940cf160 0x7fe59414ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5941538a0 0x7fe5941e3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe59415d6f0 0x7fe5941e4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5941e3af0 con 0x7fe59415d6f0 2023-10-30T13:14:02.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5941f33e0 con 0x7fe5941538a0 2023-10-30T13:14:02.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59414e8f0 con 0x7fe5940cf160 2023-10-30T13:14:02.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59ab3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe59415d6f0 0x7fe5941e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59a33d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5940cf160 0x7fe59414ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59ab3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe59415d6f0 0x7fe5941e4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40970/0 (socket says 172.21.15.148:40970) 2023-10-30T13:14:02.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59ab3e640 1 -- 172.21.15.148:0/1283733626 learned_addr learned my addr 172.21.15.148:0/1283733626 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:02.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe599b3c640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5941538a0 0x7fe5941e3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59ab3e640 1 -- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5940cf160 msgr2=0x7fe59414ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59ab3e640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5940cf160 0x7fe59414ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59ab3e640 1 -- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5941538a0 msgr2=0x7fe5941e3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:02.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59ab3e640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5941538a0 0x7fe5941e3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:02.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59ab3e640 1 -- 172.21.15.148:0/1283733626 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe584009420 con 0x7fe59415d6f0 2023-10-30T13:14:02.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59ab3e640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe59415d6f0 0x7fe5941e4480 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe58c001570 tx=0x7fe58c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:02.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe58affd640 1 -- 172.21.15.148:0/1283733626 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c003db0 con 0x7fe59415d6f0 2023-10-30T13:14:02.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe58affd640 1 -- 172.21.15.148:0/1283733626 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58c00ce30 con 0x7fe59415d6f0 2023-10-30T13:14:02.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe58affd640 1 -- 172.21.15.148:0/1283733626 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c0052b0 con 0x7fe59415d6f0 2023-10-30T13:14:02.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59414ebd0 con 0x7fe59415d6f0 2023-10-30T13:14:02.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.959+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59414f190 con 0x7fe59415d6f0 2023-10-30T13:14:02.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.963+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5941e3c60 con 0x7fe59415d6f0 2023-10-30T13:14:02.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.963+0000 7fe58affd640 1 -- 172.21.15.148:0/1283733626 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe58c005410 con 0x7fe59415d6f0 2023-10-30T13:14:02.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.963+0000 7fe58affd640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5780664a0 0x7fe578068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:02.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.963+0000 7fe58affd640 1 -- 172.21.15.148:0/1283733626 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe58c083710 con 0x7fe59415d6f0 2023-10-30T13:14:02.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.963+0000 7fe59a33d640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5780664a0 0x7fe578068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:02.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.967+0000 7fe59a33d640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5780664a0 0x7fe578068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe590007fd0 tx=0x7fe590009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:02.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:02.971+0000 7fe58affd640 1 -- 172.21.15.148:0/1283733626 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5941e3c60 con 0x7fe59415d6f0 2023-10-30T13:14:03.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.163+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5941e3c30 con 0x7fe59415d6f0 2023-10-30T13:14:03.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.163+0000 7fe58affd640 1 -- 172.21.15.148:0/1283733626 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fe58c017090 con 0x7fe59415d6f0 2023-10-30T13:14:03.169 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:03.169 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:03.169 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:14:03.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5780664a0 msgr2=0x7fe578068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:03.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5780664a0 0x7fe578068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe590007fd0 tx=0x7fe590009040 comp rx=0 tx=0).stop 2023-10-30T13:14:03.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe59415d6f0 msgr2=0x7fe5941e4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:03.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe59415d6f0 0x7fe5941e4480 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe58c001570 tx=0x7fe58c00cbe0 comp rx=0 tx=0).stop 2023-10-30T13:14:03.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 shutdown_connections 2023-10-30T13:14:03.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5780664a0 0x7fe578068960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:03.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe59415d6f0 0x7fe5941e4480 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:03.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5941538a0 0x7fe5941e3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:03.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 --2- 172.21.15.148:0/1283733626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5940cf160 0x7fe59414ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:03.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 >> 172.21.15.148:0/1283733626 conn(0x7fe59406bfe0 msgr2=0x7fe5941510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:03.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 shutdown_connections 2023-10-30T13:14:03.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:03.167+0000 7fe59c5a1640 1 -- 172.21.15.148:0/1283733626 wait complete. 2023-10-30T13:14:03.200 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:14:05.201 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:05.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.431+0000 7fb8ab1a0640 1 Processor -- start 2023-10-30T13:14:05.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.431+0000 7fb8ab1a0640 1 -- start start 2023-10-30T13:14:05.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.431+0000 7fb8ab1a0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a41538f0 0x7fb8a4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:05.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.431+0000 7fb8ab1a0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb8a415cd50 0x7fb8a40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:05.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.431+0000 7fb8ab1a0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a415d300 con 0x7fb8a40cefa0 2023-10-30T13:14:05.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.431+0000 7fb8ab1a0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a415d5e0 con 0x7fb8a415cd50 2023-10-30T13:14:05.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.431+0000 7fb8ab1a0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a415d8c0 con 0x7fb8a41538f0 2023-10-30T13:14:05.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.435+0000 7fb8aa19e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a41538f0 0x7fb8a4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:05.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.435+0000 7fb8aa19e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a41538f0 0x7fb8a4159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:34836/0 (socket says 172.21.15.148:34836) 2023-10-30T13:14:05.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.435+0000 7fb8aa19e640 1 -- 172.21.15.148:0/1041235636 learned_addr learned my addr 172.21.15.148:0/1041235636 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:05.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.435+0000 7fb8a999d640 1 --2- 172.21.15.148:0/1041235636 >> v2:172.21.15.150:3300/0 conn(0x7fb8a415cd50 0x7fb8a40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:05.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.435+0000 7fb8aa19e640 1 -- 172.21.15.148:0/1041235636 >> v2:172.21.15.150:3300/0 conn(0x7fb8a415cd50 msgr2=0x7fb8a40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:05.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.435+0000 7fb8aa19e640 1 --2- 172.21.15.148:0/1041235636 >> v2:172.21.15.150:3300/0 conn(0x7fb8a415cd50 0x7fb8a40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.435+0000 7fb8aa19e640 1 -- 172.21.15.148:0/1041235636 >> v1:172.21.15.150:6789/0 conn(0x7fb8a40cefa0 legacy=0x7fb8a40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:05.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.435+0000 7fb8aa19e640 1 -- 172.21.15.148:0/1041235636 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a415dba0 con 0x7fb8a41538f0 2023-10-30T13:14:05.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8aa19e640 1 --2- 172.21.15.148:0/1041235636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a41538f0 0x7fb8a4159fb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009870 tx=0x7fb8a001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16f518520e7523a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:05.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8a919c640 1 -- 172.21.15.148:0/1041235636 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a00a8a30 con 0x7fb8a41538f0 2023-10-30T13:14:05.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8a919c640 1 -- 172.21.15.148:0/1041235636 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a00a8b90 con 0x7fb8a41538f0 2023-10-30T13:14:05.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8a919c640 1 -- 172.21.15.148:0/1041235636 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a00b0440 con 0x7fb8a41538f0 2023-10-30T13:14:05.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/1041235636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a41538f0 msgr2=0x7fb8a4159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:05.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8ab1a0640 1 --2- 172.21.15.148:0/1041235636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a41538f0 0x7fb8a4159fb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009870 tx=0x7fb8a001ed30 comp rx=0 tx=0).stop 2023-10-30T13:14:05.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/1041235636 shutdown_connections 2023-10-30T13:14:05.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8ab1a0640 1 --2- 172.21.15.148:0/1041235636 >> v2:172.21.15.150:3300/0 conn(0x7fb8a415cd50 0x7fb8a40c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8ab1a0640 1 --2- 172.21.15.148:0/1041235636 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a41538f0 0x7fb8a4159fb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/1041235636 >> 172.21.15.148:0/1041235636 conn(0x7fb8a406a640 msgr2=0x7fb8a4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:05.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/1041235636 shutdown_connections 2023-10-30T13:14:05.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.439+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/1041235636 wait complete. 2023-10-30T13:14:05.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.443+0000 7fb8ab1a0640 1 Processor -- start 2023-10-30T13:14:05.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.443+0000 7fb8ab1a0640 1 -- start start 2023-10-30T13:14:05.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.443+0000 7fb8ab1a0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a40cefa0 0x7fb8a41517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:05.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.443+0000 7fb8ab1a0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8a41538f0 0x7fb8a41f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:05.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.443+0000 7fb8ab1a0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a415cd50 0x7fb8a40b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:05.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.443+0000 7fb8ab1a0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40b5130 con 0x7fb8a41538f0 2023-10-30T13:14:05.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.443+0000 7fb8ab1a0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a41514e0 con 0x7fb8a40cefa0 2023-10-30T13:14:05.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.443+0000 7fb8ab1a0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a41f28a0 con 0x7fb8a415cd50 2023-10-30T13:14:05.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8a999d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8a41538f0 0x7fb8a41f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:05.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8a999d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8a41538f0 0x7fb8a41f32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40974/0 (socket says 172.21.15.148:40974) 2023-10-30T13:14:05.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8a999d640 1 -- 172.21.15.148:0/902549666 learned_addr learned my addr 172.21.15.148:0/902549666 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:05.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8aa19e640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a40cefa0 0x7fb8a41517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:05.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8aa99f640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a415cd50 0x7fb8a40b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:05.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8a999d640 1 -- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a415cd50 msgr2=0x7fb8a40b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:05.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8a999d640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a415cd50 0x7fb8a40b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8a999d640 1 -- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a40cefa0 msgr2=0x7fb8a41517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:05.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8a999d640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a40cefa0 0x7fb8a41517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8a999d640 1 -- 172.21.15.148:0/902549666 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00094d0 con 0x7fb8a41538f0 2023-10-30T13:14:05.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8a999d640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8a41538f0 0x7fb8a41f32f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb89400e8a0 tx=0x7fb89400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:05.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb89affd640 1 -- 172.21.15.148:0/902549666 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb894009510 con 0x7fb8a41538f0 2023-10-30T13:14:05.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb89affd640 1 -- 172.21.15.148:0/902549666 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89400c8e0 con 0x7fb8a41538f0 2023-10-30T13:14:05.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a415ac40 con 0x7fb8a41538f0 2023-10-30T13:14:05.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb89affd640 1 -- 172.21.15.148:0/902549666 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89400cc50 con 0x7fb8a41538f0 2023-10-30T13:14:05.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a415b1e0 con 0x7fb8a41538f0 2023-10-30T13:14:05.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.447+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a40c5d90 con 0x7fb8a41538f0 2023-10-30T13:14:05.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.451+0000 7fb89affd640 1 -- 172.21.15.148:0/902549666 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb894013470 con 0x7fb8a41538f0 2023-10-30T13:14:05.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.451+0000 7fb89affd640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8880664d0 0x7fb888068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:05.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.451+0000 7fb89affd640 1 -- 172.21.15.148:0/902549666 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb894083fa0 con 0x7fb8a41538f0 2023-10-30T13:14:05.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.451+0000 7fb8aa19e640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8880664d0 0x7fb888068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:05.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.451+0000 7fb8aa19e640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8880664d0 0x7fb888068990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00b2000 tx=0x7fb8a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:05.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.459+0000 7fb89affd640 1 -- 172.21.15.148:0/902549666 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8940513a0 con 0x7fb8a41538f0 2023-10-30T13:14:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.839+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8a404e170 con 0x7fb8a41538f0 2023-10-30T13:14:05.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.839+0000 7fb89affd640 1 -- 172.21.15.148:0/902549666 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fb8940549c0 con 0x7fb8a41538f0 2023-10-30T13:14:05.844 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:05.845 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:05.845 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:14:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8880664d0 msgr2=0x7fb888068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8880664d0 0x7fb888068990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00b2000 tx=0x7fb8a0009040 comp rx=0 tx=0).stop 2023-10-30T13:14:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8a41538f0 msgr2=0x7fb8a41f32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8a41538f0 0x7fb8a41f32f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb89400e8a0 tx=0x7fb89400eca0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 shutdown_connections 2023-10-30T13:14:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8880664d0 0x7fb888068990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8a41538f0 0x7fb8a41f32f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8a415cd50 0x7fb8a40b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 --2- 172.21.15.148:0/902549666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8a40cefa0 0x7fb8a41517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 >> 172.21.15.148:0/902549666 conn(0x7fb8a406a640 msgr2=0x7fb8a40c9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:05.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 shutdown_connections 2023-10-30T13:14:05.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:05.843+0000 7fb8ab1a0640 1 -- 172.21.15.148:0/902549666 wait complete. 2023-10-30T13:14:05.876 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:14:07.877 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:08.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.103+0000 7f107759e640 1 Processor -- start 2023-10-30T13:14:08.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.103+0000 7f107759e640 1 -- start start 2023-10-30T13:14:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107759e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f10781538f0 0x7f1078159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107759e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10780cefa0 0x7f10780cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107759e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10780ca6a0 con 0x7f107815cd50 2023-10-30T13:14:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107759e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10780cf8f0 con 0x7f10781538f0 2023-10-30T13:14:08.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107759e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107815d300 con 0x7f10780cefa0 2023-10-30T13:14:08.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f10781538f0 0x7f1078159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:08.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f10781538f0 0x7f1078159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40984/0 (socket says 172.21.15.148:40984) 2023-10-30T13:14:08.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107659c640 1 -- 172.21.15.148:0/2881450652 learned_addr learned my addr 172.21.15.148:0/2881450652 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:08.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f1076d9d640 1 --2- 172.21.15.148:0/2881450652 >> v2:172.21.15.150:3300/0 conn(0x7f10780cefa0 0x7f10780cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:08.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107659c640 1 -- 172.21.15.148:0/2881450652 >> v2:172.21.15.150:3300/0 conn(0x7f10780cefa0 msgr2=0x7f10780cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:08.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107659c640 1 --2- 172.21.15.148:0/2881450652 >> v2:172.21.15.150:3300/0 conn(0x7f10780cefa0 0x7f10780cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107659c640 1 -- 172.21.15.148:0/2881450652 >> v1:172.21.15.150:6789/0 conn(0x7f107815cd50 legacy=0x7f10780c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:08.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.107+0000 7f107659c640 1 -- 172.21.15.148:0/2881450652 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107815d5e0 con 0x7f10781538f0 2023-10-30T13:14:08.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.111+0000 7f107659c640 1 --2- 172.21.15.148:0/2881450652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 0x7f1078159fb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1078066a40 tx=0x7f106801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4a3d95c8fc60c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:08.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107559a640 1 -- 172.21.15.148:0/2881450652 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10680a89f0 con 0x7f10781538f0 2023-10-30T13:14:08.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107559a640 1 -- 172.21.15.148:0/2881450652 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10680a8b50 con 0x7f10781538f0 2023-10-30T13:14:08.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107559a640 1 -- 172.21.15.148:0/2881450652 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10680b0440 con 0x7f10781538f0 2023-10-30T13:14:08.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107759e640 1 -- 172.21.15.148:0/2881450652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 msgr2=0x7f1078159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:08.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107759e640 1 --2- 172.21.15.148:0/2881450652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 0x7f1078159fb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1078066a40 tx=0x7f106801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107759e640 1 -- 172.21.15.148:0/2881450652 shutdown_connections 2023-10-30T13:14:08.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107759e640 1 --2- 172.21.15.148:0/2881450652 >> v2:172.21.15.150:3300/0 conn(0x7f10780cefa0 0x7f10780cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107759e640 1 --2- 172.21.15.148:0/2881450652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 0x7f1078159fb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107759e640 1 -- 172.21.15.148:0/2881450652 >> 172.21.15.148:0/2881450652 conn(0x7f107806a640 msgr2=0x7f1078065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:08.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107759e640 1 -- 172.21.15.148:0/2881450652 shutdown_connections 2023-10-30T13:14:08.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.115+0000 7f107759e640 1 -- 172.21.15.148:0/2881450652 wait complete. 2023-10-30T13:14:08.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f107759e640 1 Processor -- start 2023-10-30T13:14:08.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f107759e640 1 -- start start 2023-10-30T13:14:08.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f107759e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10780cefa0 0x7f1078158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:08.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f107759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 0x7f1078157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:08.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f107759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107815cd50 0x7f1078157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:08.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f107759e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1078148a10 con 0x7f10781538f0 2023-10-30T13:14:08.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f107759e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10781438a0 con 0x7f107815cd50 2023-10-30T13:14:08.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f107759e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1078146800 con 0x7f10780cefa0 2023-10-30T13:14:08.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1075d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 0x7f1078157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:08.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1075d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 0x7f1078157680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41000/0 (socket says 172.21.15.148:41000) 2023-10-30T13:14:08.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1075d9b640 1 -- 172.21.15.148:0/2060443423 learned_addr learned my addr 172.21.15.148:0/2060443423 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:08.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f107659c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10780cefa0 0x7f1078158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:08.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1076d9d640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107815cd50 0x7f1078157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:08.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1075d9b640 1 -- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10780cefa0 msgr2=0x7f1078158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1075d9b640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10780cefa0 0x7f1078158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1075d9b640 1 -- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107815cd50 msgr2=0x7f1078157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1075d9b640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107815cd50 0x7f1078157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1075d9b640 1 -- 172.21.15.148:0/2060443423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10680094d0 con 0x7f10781538f0 2023-10-30T13:14:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.119+0000 7f1075d9b640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 0x7f1078157680 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f106c00c3c0 tx=0x7f106c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.123+0000 7f1066ffd640 1 -- 172.21.15.148:0/2060443423 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c0052d0 con 0x7f10781538f0 2023-10-30T13:14:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.123+0000 7f1066ffd640 1 -- 172.21.15.148:0/2060443423 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c005430 con 0x7f10781538f0 2023-10-30T13:14:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.123+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1078153110 con 0x7f10781538f0 2023-10-30T13:14:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.123+0000 7f1066ffd640 1 -- 172.21.15.148:0/2060443423 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c003840 con 0x7f10781538f0 2023-10-30T13:14:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.123+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1078148730 con 0x7f10781538f0 2023-10-30T13:14:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.123+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107804e170 con 0x7f10781538f0 2023-10-30T13:14:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.123+0000 7f1066ffd640 1 -- 172.21.15.148:0/2060443423 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f106c0039a0 con 0x7f10781538f0 2023-10-30T13:14:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.127+0000 7f1066ffd640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10580664f0 0x7f10580689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:08.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.127+0000 7f1066ffd640 1 -- 172.21.15.148:0/2060443423 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f106c083480 con 0x7f10781538f0 2023-10-30T13:14:08.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.127+0000 7f107659c640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10580664f0 0x7f10580689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:08.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.127+0000 7f107659c640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10580664f0 0x7f10580689b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f10680b2000 tx=0x7f1068009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:08.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.135+0000 7f1066ffd640 1 -- 172.21.15.148:0/2060443423 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f106c050870 con 0x7f10781538f0 2023-10-30T13:14:08.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.511+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f107814fcc0 con 0x7f10781538f0 2023-10-30T13:14:08.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.511+0000 7f1066ffd640 1 -- 172.21.15.148:0/2060443423 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f106c053e90 con 0x7f10781538f0 2023-10-30T13:14:08.517 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:08.517 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:08.517 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:14:08.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10580664f0 msgr2=0x7f10580689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:08.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10580664f0 0x7f10580689b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f10680b2000 tx=0x7f1068009040 comp rx=0 tx=0).stop 2023-10-30T13:14:08.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 msgr2=0x7f1078157680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:08.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 0x7f1078157680 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f106c00c3c0 tx=0x7f106c00c880 comp rx=0 tx=0).stop 2023-10-30T13:14:08.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 shutdown_connections 2023-10-30T13:14:08.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10580664f0 0x7f10580689b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10781538f0 0x7f1078157680 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107815cd50 0x7f1078157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 --2- 172.21.15.148:0/2060443423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10780cefa0 0x7f1078158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:08.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 >> 172.21.15.148:0/2060443423 conn(0x7f107806a640 msgr2=0x7f1078065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:08.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 shutdown_connections 2023-10-30T13:14:08.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:08.515+0000 7f107759e640 1 -- 172.21.15.148:0/2060443423 wait complete. 2023-10-30T13:14:08.549 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:14:10.550 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:10.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.683+0000 7f8cb406b640 1 Processor -- start 2023-10-30T13:14:10.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb406b640 1 -- start start 2023-10-30T13:14:10.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb406b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6b80 0x7f8cac0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:10.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb406b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8cac13f4a0 0x7f8cac0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:10.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb406b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0c0a50 con 0x7f8cac0c6f60 2023-10-30T13:14:10.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb406b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac06f6e0 con 0x7f8cac13f4a0 2023-10-30T13:14:10.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb406b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac06f9c0 con 0x7f8cac0c6b80 2023-10-30T13:14:10.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb1e07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6b80 0x7f8cac0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:10.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb2608640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8cac13f4a0 0x7f8cac0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:10.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb1e07640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6b80 0x7f8cac0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:34860/0 (socket says 172.21.15.148:34860) 2023-10-30T13:14:10.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb1e07640 1 -- 172.21.15.148:0/220349210 learned_addr learned my addr 172.21.15.148:0/220349210 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:10.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb2608640 1 -- 172.21.15.148:0/220349210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6b80 msgr2=0x7f8cac0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:10.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb2608640 1 --2- 172.21.15.148:0/220349210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6b80 0x7f8cac0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb2608640 1 -- 172.21.15.148:0/220349210 >> v1:172.21.15.150:6789/0 conn(0x7f8cac0c6f60 legacy=0x7f8cac13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:10.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb2608640 1 -- 172.21.15.148:0/220349210 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac06fca0 con 0x7f8cac13f4a0 2023-10-30T13:14:10.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.687+0000 7f8cb1e07640 1 --2- 172.21.15.148:0/220349210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6b80 0x7f8cac0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:14:10.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb2608640 1 --2- 172.21.15.148:0/220349210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac13f4a0 0x7f8cac0c0400 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009d90 tx=0x7f8ca401c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=165229eb7f33f91f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:10.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb0e05640 1 -- 172.21.15.148:0/220349210 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40a5920 con 0x7f8cac13f4a0 2023-10-30T13:14:10.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb0e05640 1 -- 172.21.15.148:0/220349210 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca40a5a80 con 0x7f8cac13f4a0 2023-10-30T13:14:10.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb0e05640 1 -- 172.21.15.148:0/220349210 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40a5e10 con 0x7f8cac13f4a0 2023-10-30T13:14:10.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb406b640 1 -- 172.21.15.148:0/220349210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac13f4a0 msgr2=0x7f8cac0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:10.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb406b640 1 --2- 172.21.15.148:0/220349210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac13f4a0 0x7f8cac0c0400 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009d90 tx=0x7f8ca401c950 comp rx=0 tx=0).stop 2023-10-30T13:14:10.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb406b640 1 -- 172.21.15.148:0/220349210 shutdown_connections 2023-10-30T13:14:10.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb406b640 1 --2- 172.21.15.148:0/220349210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac13f4a0 0x7f8cac0c0400 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb406b640 1 --2- 172.21.15.148:0/220349210 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6b80 0x7f8cac0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb406b640 1 -- 172.21.15.148:0/220349210 >> 172.21.15.148:0/220349210 conn(0x7f8cac0c7910 msgr2=0x7f8cac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:10.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb406b640 1 -- 172.21.15.148:0/220349210 shutdown_connections 2023-10-30T13:14:10.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.691+0000 7f8cb406b640 1 -- 172.21.15.148:0/220349210 wait complete. 2023-10-30T13:14:10.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 Processor -- start 2023-10-30T13:14:10.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 -- start start 2023-10-30T13:14:10.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac0c6b80 0x7f8cac155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:10.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6f60 0x7f8cac155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:10.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac13f4a0 0x7f8cac134d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:10.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac06a6f0 con 0x7f8cac0c6b80 2023-10-30T13:14:10.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac13afa0 con 0x7f8cac13f4a0 2023-10-30T13:14:10.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac157520 con 0x7f8cac0c6f60 2023-10-30T13:14:10.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1e07640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac0c6b80 0x7f8cac155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:10.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1e07640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac0c6b80 0x7f8cac155500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41016/0 (socket says 172.21.15.148:41016) 2023-10-30T13:14:10.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1e07640 1 -- 172.21.15.148:0/2805172836 learned_addr learned my addr 172.21.15.148:0/2805172836 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:10.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb2608640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac13f4a0 0x7f8cac134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:10.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1606640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6f60 0x7f8cac155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:10.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1e07640 1 -- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6f60 msgr2=0x7f8cac155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:10.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1e07640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6f60 0x7f8cac155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1e07640 1 -- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac13f4a0 msgr2=0x7f8cac134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:10.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1e07640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac13f4a0 0x7f8cac134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1e07640 1 -- 172.21.15.148:0/2805172836 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca80094d0 con 0x7f8cac0c6b80 2023-10-30T13:14:10.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb1e07640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac0c6b80 0x7f8cac155500 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800ea00 tx=0x7f8ca800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:10.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8ca27fc640 1 -- 172.21.15.148:0/2805172836 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca800ca60 con 0x7f8cac0c6b80 2023-10-30T13:14:10.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8ca27fc640 1 -- 172.21.15.148:0/2805172836 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca800cbc0 con 0x7f8cac0c6b80 2023-10-30T13:14:10.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca4009a70 con 0x7f8cac0c6b80 2023-10-30T13:14:10.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8ca27fc640 1 -- 172.21.15.148:0/2805172836 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8013470 con 0x7f8cac0c6b80 2023-10-30T13:14:10.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac157790 con 0x7f8cac0c6b80 2023-10-30T13:14:10.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.695+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac04e170 con 0x7f8cac0c6b80 2023-10-30T13:14:10.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.699+0000 7f8ca27fc640 1 -- 172.21.15.148:0/2805172836 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ca80135d0 con 0x7f8cac0c6b80 2023-10-30T13:14:10.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.699+0000 7f8ca27fc640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c900664a0 0x7f8c90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:10.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.699+0000 7f8ca27fc640 1 -- 172.21.15.148:0/2805172836 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8ca8084170 con 0x7f8cac0c6b80 2023-10-30T13:14:10.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.699+0000 7f8cb1606640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c900664a0 0x7f8c90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:10.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.699+0000 7f8cb1606640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c900664a0 0x7f8c90068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0c0f70 tx=0x7f8c9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:10.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.703+0000 7f8ca27fc640 1 -- 172.21.15.148:0/2805172836 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ca8051600 con 0x7f8cac0c6b80 2023-10-30T13:14:10.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.899+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cac156380 con 0x7f8cac0c6b80 2023-10-30T13:14:10.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.899+0000 7f8ca27fc640 1 -- 172.21.15.148:0/2805172836 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f8ca8054c20 con 0x7f8cac0c6b80 2023-10-30T13:14:10.905 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:10.906 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:10.906 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:14:10.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c900664a0 msgr2=0x7f8c90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:10.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c900664a0 0x7f8c90068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0c0f70 tx=0x7f8c9c009040 comp rx=0 tx=0).stop 2023-10-30T13:14:10.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac0c6b80 msgr2=0x7f8cac155500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:10.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac0c6b80 0x7f8cac155500 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800ea00 tx=0x7f8ca800eec0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 shutdown_connections 2023-10-30T13:14:10.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c900664a0 0x7f8c90068960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac13f4a0 0x7f8cac134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cac0c6f60 0x7f8cac155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 --2- 172.21.15.148:0/2805172836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cac0c6b80 0x7f8cac155500 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:10.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 >> 172.21.15.148:0/2805172836 conn(0x7f8cac0c7910 msgr2=0x7f8cac1330f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:10.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 shutdown_connections 2023-10-30T13:14:10.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:10.903+0000 7f8cb406b640 1 -- 172.21.15.148:0/2805172836 wait complete. 2023-10-30T13:14:10.926 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:14:12.658 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 69% (8333/12071) Updating files: 70% (8450/12071) Updating files: 71% (8571/12071) Updating files: 72% (8692/12071) Updating files: 73% (8812/12071) Updating files: 74% (8933/12071) Updating files: 75% (9054/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 78% (9416/12071) Updating files: 79% (9537/12071) Updating files: 80% (9657/12071) Updating files: 81% (9778/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 86% (10382/12071) Updating files: 87% (10502/12071) Updating files: 88% (10623/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 91% (10985/12071) Updating files: 92% (11106/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T13:14:12.927 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:13.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f6454846640 1 Processor -- start 2023-10-30T13:14:13.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f6454846640 1 -- start start 2023-10-30T13:14:13.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f6454846640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64500b2300 0x7f6450071b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:13.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f6454846640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645006b3b0 con 0x7f64501570f0 2023-10-30T13:14:13.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f6454846640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64500c38a0 con 0x7f645006f370 2023-10-30T13:14:13.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f6454846640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64500c3b20 con 0x7f64500b2300 2023-10-30T13:14:13.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644e59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f64501570f0 0x7f645006ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41202/0 (socket says 172.21.15.148:41202) 2023-10-30T13:14:13.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644e59c640 1 -- 172.21.15.148:0/4288274568 learned_addr learned my addr 172.21.15.148:0/4288274568 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:13.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644dd9b640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64500b2300 0x7f6450071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:13.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925184285 0 0) 0x7f645006b3b0 con 0x7f64501570f0 2023-10-30T13:14:13.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6438003530 con 0x7f64501570f0 2023-10-30T13:14:13.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376284706 0 0) 0x7f6438003530 con 0x7f64501570f0 2023-10-30T13:14:13.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64500b2300 msgr2=0x7f6450071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:13.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64500b2300 0x7f6450071b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:13.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 >> v1:172.21.15.150:6789/0 conn(0x7f645006f370 legacy=0x7f64501640d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:13.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64500ca9f0 con 0x7f64501570f0 2023-10-30T13:14:13.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f6440001680 con 0x7f64501570f0 2023-10-30T13:14:13.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 >> v1:172.21.15.148:6789/0 conn(0x7f64501570f0 legacy=0x7f645006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:13.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64380039b0 0x7f6438003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:13.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6438004360 0x7f6438004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:13.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64500b2300 0x7f6438005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:13.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64500c3b20 con 0x7f6438004360 2023-10-30T13:14:13.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643800b990 con 0x7f64380039b0 2023-10-30T13:14:13.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643800bb50 con 0x7f64500b2300 2023-10-30T13:14:13.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644dd9b640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6438004360 0x7f6438004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:13.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644e59c640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64380039b0 0x7f6438003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:13.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644ed9d640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64500b2300 0x7f6438005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:13.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644dd9b640 1 -- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64500b2300 msgr2=0x7f6438005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:13.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644dd9b640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f64500b2300 0x7f6438005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:13.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644dd9b640 1 -- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64380039b0 msgr2=0x7f6438003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:13.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644dd9b640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64380039b0 0x7f6438003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:13.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644dd9b640 1 -- 172.21.15.148:0/4288274568 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6438003530 con 0x7f6438004360 2023-10-30T13:14:13.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.063+0000 7f644e59c640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64380039b0 0x7f6438003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:13.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f644dd9b640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6438004360 0x7f6438004830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6444009ae0 tx=0x7f644401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62262e26e8a8b78c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:13.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64440a5640 con 0x7f6438004360 2023-10-30T13:14:13.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64440a57a0 con 0x7f6438004360 2023-10-30T13:14:13.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f644d59a640 1 -- 172.21.15.148:0/4288274568 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64440a5b50 con 0x7f6438004360 2023-10-30T13:14:13.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f6454846640 1 -- 172.21.15.148:0/4288274568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6438004360 msgr2=0x7f6438004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:13.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f6454846640 1 --2- 172.21.15.148:0/4288274568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6438004360 0x7f6438004830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6444009ae0 tx=0x7f644401ca50 comp rx=0 tx=0).stop 2023-10-30T13:14:13.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f644ed9d640 1 -- 172.21.15.148:0/4288274568 reap_dead start 2023-10-30T13:14:13.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f6454846640 1 -- 172.21.15.148:0/4288274568 shutdown_connections 2023-10-30T13:14:13.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f6454846640 1 -- 172.21.15.148:0/4288274568 >> 172.21.15.148:0/4288274568 conn(0x7f6450069e90 msgr2=0x7f645006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:13.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f6454846640 1 -- 172.21.15.148:0/4288274568 shutdown_connections 2023-10-30T13:14:13.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.067+0000 7f6454846640 1 -- 172.21.15.148:0/4288274568 wait complete. 2023-10-30T13:14:13.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 Processor -- start 2023-10-30T13:14:13.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 -- start start 2023-10-30T13:14:13.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6438004360 0x7f6450157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:13.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64500c85f0 0x7f645014bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:13.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6450157710 0x7f6450153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:13.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6450159ea0 con 0x7f64500c85f0 2023-10-30T13:14:13.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64500caf10 con 0x7f6450157710 2023-10-30T13:14:13.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6450158360 con 0x7f6438004360 2023-10-30T13:14:13.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64500c85f0 0x7f645014bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:13.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64500c85f0 0x7f645014bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60950/0 (socket says 172.21.15.148:60950) 2023-10-30T13:14:13.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644dd9b640 1 -- 172.21.15.148:0/2797630852 learned_addr learned my addr 172.21.15.148:0/2797630852 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:13.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644e59c640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6438004360 0x7f6450157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:13.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644ed9d640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6450157710 0x7f6450153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:13.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644dd9b640 1 -- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6438004360 msgr2=0x7f6450157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:13.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644dd9b640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6438004360 0x7f6450157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:13.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644dd9b640 1 -- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6450157710 msgr2=0x7f6450153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:13.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644dd9b640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6450157710 0x7f6450153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:13.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644dd9b640 1 -- 172.21.15.148:0/2797630852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645015a0f0 con 0x7f64500c85f0 2023-10-30T13:14:13.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f644dd9b640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64500c85f0 0x7f645014bd20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f64440b0040 tx=0x7f6438005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:13.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f643effd640 1 -- 172.21.15.148:0/2797630852 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444004220 con 0x7f64500c85f0 2023-10-30T13:14:13.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f643effd640 1 -- 172.21.15.148:0/2797630852 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6444004380 con 0x7f64500c85f0 2023-10-30T13:14:13.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f643effd640 1 -- 172.21.15.148:0/2797630852 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64440ad890 con 0x7f64500c85f0 2023-10-30T13:14:13.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645015a3d0 con 0x7f64500c85f0 2023-10-30T13:14:13.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.071+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64500cb440 con 0x7f64500c85f0 2023-10-30T13:14:13.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.075+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645006c7e0 con 0x7f64500c85f0 2023-10-30T13:14:13.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.075+0000 7f643effd640 1 -- 172.21.15.148:0/2797630852 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64440b4040 con 0x7f64500c85f0 2023-10-30T13:14:13.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.075+0000 7f643effd640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64300664d0 0x7f6430068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:13.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.075+0000 7f643effd640 1 -- 172.21.15.148:0/2797630852 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f644411e520 con 0x7f64500c85f0 2023-10-30T13:14:13.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.075+0000 7f644e59c640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64300664d0 0x7f6430068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:13.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.075+0000 7f644e59c640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64300664d0 0x7f6430068990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6440001cc0 tx=0x7f6440009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:13.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.079+0000 7f643effd640 1 -- 172.21.15.148:0/2797630852 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64440eba40 con 0x7f64500c85f0 2023-10-30T13:14:13.195 INFO:tasks.workunit.client.0.smithi148.stderr:Note: switching to '00c1c13cea21a686b046584596fbeac5315a7d71'. 2023-10-30T13:14:13.196 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T13:14:13.196 INFO:tasks.workunit.client.0.smithi148.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T13:14:13.196 INFO:tasks.workunit.client.0.smithi148.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T13:14:13.196 INFO:tasks.workunit.client.0.smithi148.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T13:14:13.196 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T13:14:13.197 INFO:tasks.workunit.client.0.smithi148.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T13:14:13.197 INFO:tasks.workunit.client.0.smithi148.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T13:14:13.197 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T13:14:13.197 INFO:tasks.workunit.client.0.smithi148.stderr: git switch -c 2023-10-30T13:14:13.197 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T13:14:13.197 INFO:tasks.workunit.client.0.smithi148.stderr:Or undo this operation with: 2023-10-30T13:14:13.198 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T13:14:13.198 INFO:tasks.workunit.client.0.smithi148.stderr: git switch - 2023-10-30T13:14:13.198 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T13:14:13.198 INFO:tasks.workunit.client.0.smithi148.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T13:14:13.198 INFO:tasks.workunit.client.0.smithi148.stderr: 2023-10-30T13:14:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:HEAD is now at 00c1c13cea2 mds/MDBalancer: ignore queued callbacks if MDS is not active 2023-10-30T13:14:13.201 DEBUG:teuthology.orchestra.run.smithi148:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T13:14:13.210 INFO:tasks.workunit.client.0.smithi148.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T13:14:13.213 INFO:tasks.workunit.client.0.smithi148.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T13:14:13.213 INFO:tasks.workunit.client.0.smithi148.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T13:14:13.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.275+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6450159ea0 con 0x7f64500c85f0 2023-10-30T13:14:13.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.275+0000 7f643effd640 1 -- 172.21.15.148:0/2797630852 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f64440044f0 con 0x7f64500c85f0 2023-10-30T13:14:13.281 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:13.281 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:13.282 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:14:13.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64300664d0 msgr2=0x7f6430068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64300664d0 0x7f6430068990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6440001cc0 tx=0x7f6440009040 comp rx=0 tx=0).stop 2023-10-30T13:14:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64500c85f0 msgr2=0x7f645014bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64500c85f0 0x7f645014bd20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f64440b0040 tx=0x7f6438005800 comp rx=0 tx=0).stop 2023-10-30T13:14:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 shutdown_connections 2023-10-30T13:14:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f64300664d0 0x7f6430068990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64500c85f0 0x7f645014bd20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6450157710 0x7f6450153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 --2- 172.21.15.148:0/2797630852 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6438004360 0x7f6450157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 >> 172.21.15.148:0/2797630852 conn(0x7f6450069e90 msgr2=0x7f64500c8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 shutdown_connections 2023-10-30T13:14:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:13.279+0000 7f6454846640 1 -- 172.21.15.148:0/2797630852 wait complete. 2023-10-30T13:14:13.303 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:14:13.975 INFO:tasks.workunit.client.0.smithi148.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T13:14:14.067 INFO:tasks.workunit.client.0.smithi148.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T13:14:14.122 INFO:tasks.workunit.client.0.smithi148.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T13:14:14.125 INFO:tasks.workunit.client.0.smithi148.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T13:14:14.125 INFO:tasks.workunit.client.0.smithi148.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T13:14:14.178 INFO:tasks.workunit.client.0.smithi148.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T13:14:14.183 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:14:14.183 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T13:14:14.193 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-10-30T13:14:14.194 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-10-30T13:14:14.194 DEBUG:teuthology.orchestra.run.smithi148:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-10-30T13:14:15.304 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:15.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b771d640 1 Processor -- start 2023-10-30T13:14:15.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b771d640 1 -- start start 2023-10-30T13:14:15.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b771d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b006f7a0 0x7f67b006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:15.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b771d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67b0071110 0x7f67b0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:15.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b771d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b0066cf0 con 0x7f67b00c6bd0 2023-10-30T13:14:15.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b771d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b00700c0 con 0x7f67b0071110 2023-10-30T13:14:15.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b771d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00c70d0 con 0x7f67b006f7a0 2023-10-30T13:14:15.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b54b9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b006f7a0 0x7f67b006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:15.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b5cba640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67b0071110 0x7f67b0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:15.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b54b9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b006f7a0 0x7f67b006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:56646/0 (socket says 172.21.15.148:56646) 2023-10-30T13:14:15.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b4cb8640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f67b00c6bd0 0x7f67b0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:41940/0 (socket says 172.21.15.148:41940) 2023-10-30T13:14:15.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b5cba640 1 --2- 172.21.15.148:0/3964513463 >> v2:172.21.15.150:3300/0 conn(0x7f67b0071110 0x7f67b0158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:41452/0 (socket says 172.21.15.148:41452) 2023-10-30T13:14:15.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.447+0000 7f67b54b9640 1 -- 172.21.15.148:0/3964513463 learned_addr learned my addr 172.21.15.148:0/3964513463 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:15.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67b5cba640 1 -- 172.21.15.148:0/3964513463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b006f7a0 msgr2=0x7f67b006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:15.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67b5cba640 1 --2- 172.21.15.148:0/3964513463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b006f7a0 0x7f67b006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67b5cba640 1 -- 172.21.15.148:0/3964513463 >> v1:172.21.15.150:6789/0 conn(0x7f67b00c6bd0 legacy=0x7f67b0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:15.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67b5cba640 1 -- 172.21.15.148:0/3964513463 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b00c7390 con 0x7f67b0071110 2023-10-30T13:14:15.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67b5cba640 1 --2- 172.21.15.148:0/3964513463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b0071110 0x7f67b0158110 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009d90 tx=0x7f67a801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7760a3f7f5d91062 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:15.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67a7fff640 1 -- 172.21.15.148:0/3964513463 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a80af600 con 0x7f67b0071110 2023-10-30T13:14:15.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67a7fff640 1 -- 172.21.15.148:0/3964513463 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a80ab050 con 0x7f67b0071110 2023-10-30T13:14:15.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67a7fff640 1 -- 172.21.15.148:0/3964513463 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a80af980 con 0x7f67b0071110 2023-10-30T13:14:15.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67b771d640 1 -- 172.21.15.148:0/3964513463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b0071110 msgr2=0x7f67b0158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:15.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.451+0000 7f67b771d640 1 --2- 172.21.15.148:0/3964513463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b0071110 0x7f67b0158110 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009d90 tx=0x7f67a801f130 comp rx=0 tx=0).stop 2023-10-30T13:14:15.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 -- 172.21.15.148:0/3964513463 shutdown_connections 2023-10-30T13:14:15.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 --2- 172.21.15.148:0/3964513463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b0071110 0x7f67b0158110 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 --2- 172.21.15.148:0/3964513463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b006f7a0 0x7f67b006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 -- 172.21.15.148:0/3964513463 >> 172.21.15.148:0/3964513463 conn(0x7f67b00c7910 msgr2=0x7f67b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:15.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 -- 172.21.15.148:0/3964513463 shutdown_connections 2023-10-30T13:14:15.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 -- 172.21.15.148:0/3964513463 wait complete. 2023-10-30T13:14:15.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 Processor -- start 2023-10-30T13:14:15.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 -- start start 2023-10-30T13:14:15.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b006f7a0 0x7f67b01f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:15.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b0071110 0x7f67b014e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67b00c6bd0 0x7f67b014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b014fac0 con 0x7f67b00c6bd0 2023-10-30T13:14:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b006a900 con 0x7f67b006f7a0 2023-10-30T13:14:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b771d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00af830 con 0x7f67b0071110 2023-10-30T13:14:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b5cba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67b00c6bd0 0x7f67b014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:15.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b5cba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67b00c6bd0 0x7f67b014e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60960/0 (socket says 172.21.15.148:60960) 2023-10-30T13:14:15.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b5cba640 1 -- 172.21.15.148:0/803918224 learned_addr learned my addr 172.21.15.148:0/803918224 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:15.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b54b9640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b006f7a0 0x7f67b01f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:15.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b4cb8640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b0071110 0x7f67b014e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:15.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b5cba640 1 -- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b0071110 msgr2=0x7f67b014e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:15.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b5cba640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b0071110 0x7f67b014e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b5cba640 1 -- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b006f7a0 msgr2=0x7f67b01f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:15.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b5cba640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b006f7a0 0x7f67b01f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b5cba640 1 -- 172.21.15.148:0/803918224 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a8009a70 con 0x7f67b00c6bd0 2023-10-30T13:14:15.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.455+0000 7f67b5cba640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67b00c6bd0 0x7f67b014e9e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009d90 tx=0x7f67a801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:15.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67a5ffb640 1 -- 172.21.15.148:0/803918224 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8009450 con 0x7f67b00c6bd0 2023-10-30T13:14:15.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67a5ffb640 1 -- 172.21.15.148:0/803918224 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a80b9410 con 0x7f67b00c6bd0 2023-10-30T13:14:15.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67a5ffb640 1 -- 172.21.15.148:0/803918224 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a80b97c0 con 0x7f67b00c6bd0 2023-10-30T13:14:15.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b014e010 con 0x7f67b00c6bd0 2023-10-30T13:14:15.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b01f2af0 con 0x7f67b00c6bd0 2023-10-30T13:14:15.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b014e1a0 con 0x7f67b00c6bd0 2023-10-30T13:14:15.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67a5ffb640 1 -- 172.21.15.148:0/803918224 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67a80b6050 con 0x7f67b00c6bd0 2023-10-30T13:14:15.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67a5ffb640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67940664a0 0x7f6794068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:15.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67a5ffb640 1 -- 172.21.15.148:0/803918224 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f67a8122710 con 0x7f67b00c6bd0 2023-10-30T13:14:15.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.459+0000 7f67b54b9640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67940664a0 0x7f6794068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:15.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.463+0000 7f67b54b9640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67940664a0 0x7f6794068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009ab0 tx=0x7f67ac009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:15.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.463+0000 7f67a5ffb640 1 -- 172.21.15.148:0/803918224 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67a80efcc0 con 0x7f67b00c6bd0 2023-10-30T13:14:15.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.643+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67b004e170 con 0x7f67b00c6bd0 2023-10-30T13:14:15.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.643+0000 7f67a5ffb640 1 -- 172.21.15.148:0/803918224 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f67a80f32e0 con 0x7f67b00c6bd0 2023-10-30T13:14:15.651 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:15.651 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:15.651 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:14:15.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67940664a0 msgr2=0x7f6794068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:15.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67940664a0 0x7f6794068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009ab0 tx=0x7f67ac009240 comp rx=0 tx=0).stop 2023-10-30T13:14:15.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67b00c6bd0 msgr2=0x7f67b014e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:15.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67b00c6bd0 0x7f67b014e9e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009d90 tx=0x7f67a801bed0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 shutdown_connections 2023-10-30T13:14:15.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67940664a0 0x7f6794068960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67b00c6bd0 0x7f67b014e9e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67b0071110 0x7f67b014e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 --2- 172.21.15.148:0/803918224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b006f7a0 0x7f67b01f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:15.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 >> 172.21.15.148:0/803918224 conn(0x7f67b00c7910 msgr2=0x7f67b00c24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:15.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 shutdown_connections 2023-10-30T13:14:15.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:15.647+0000 7f67b771d640 1 -- 172.21.15.148:0/803918224 wait complete. 2023-10-30T13:14:15.672 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:14:17.673 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:17.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.815+0000 7f7828879640 1 Processor -- start 2023-10-30T13:14:17.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.815+0000 7f7828879640 1 -- start start 2023-10-30T13:14:17.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7828879640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78240c6b80 0x7f78240bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:17.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7828879640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f78240c6f60 0x7f78240c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:17.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7828879640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 0x7f782413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:17.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7828879640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78240c0a50 con 0x7f78240c6f60 2023-10-30T13:14:17.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7828879640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782406f6e0 con 0x7f78240c6b80 2023-10-30T13:14:17.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7828879640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782406f9c0 con 0x7f78240c8430 2023-10-30T13:14:17.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7821d9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f78240c6f60 0x7f78240c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:17.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7821d9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f78240c6f60 0x7f78240c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60968/0 (socket says 172.21.15.148:60968) 2023-10-30T13:14:17.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7821d9b640 1 -- 172.21.15.148:0/4172975350 learned_addr learned my addr 172.21.15.148:0/4172975350 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:17.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7822d9d640 1 --2- 172.21.15.148:0/4172975350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 0x7f782413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:17.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f782259c640 1 --2- 172.21.15.148:0/4172975350 >> v2:172.21.15.150:3300/0 conn(0x7f78240c6b80 0x7f78240bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:17.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7821d9b640 1 -- 172.21.15.148:0/4172975350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 msgr2=0x7f782413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:17.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7821d9b640 1 --2- 172.21.15.148:0/4172975350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 0x7f782413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:17.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7821d9b640 1 -- 172.21.15.148:0/4172975350 >> v2:172.21.15.150:3300/0 conn(0x7f78240c6b80 msgr2=0x7f78240bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:17.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7821d9b640 1 --2- 172.21.15.148:0/4172975350 >> v2:172.21.15.150:3300/0 conn(0x7f78240c6b80 0x7f78240bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:17.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.819+0000 7f7821d9b640 1 -- 172.21.15.148:0/4172975350 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782406fca0 con 0x7f78240c6f60 2023-10-30T13:14:17.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7821d9b640 1 --2- 172.21.15.148:0/4172975350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 0x7f78240c0400 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f782406f1b0 tx=0x7f781801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256d332932bf6e8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:17.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f782159a640 1 -- 172.21.15.148:0/4172975350 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78180ac630 con 0x7f78240c6f60 2023-10-30T13:14:17.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f782159a640 1 -- 172.21.15.148:0/4172975350 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78180ac790 con 0x7f78240c6f60 2023-10-30T13:14:17.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f782159a640 1 -- 172.21.15.148:0/4172975350 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78180acb20 con 0x7f78240c6f60 2023-10-30T13:14:17.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 -- 172.21.15.148:0/4172975350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 msgr2=0x7f78240c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:17.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 --2- 172.21.15.148:0/4172975350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 0x7f78240c0400 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f782406f1b0 tx=0x7f781801c500 comp rx=0 tx=0).stop 2023-10-30T13:14:17.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 -- 172.21.15.148:0/4172975350 shutdown_connections 2023-10-30T13:14:17.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 --2- 172.21.15.148:0/4172975350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 0x7f782413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:17.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 --2- 172.21.15.148:0/4172975350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 0x7f78240c0400 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:17.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 --2- 172.21.15.148:0/4172975350 >> v2:172.21.15.150:3300/0 conn(0x7f78240c6b80 0x7f78240bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:17.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 -- 172.21.15.148:0/4172975350 >> 172.21.15.148:0/4172975350 conn(0x7f78240c7910 msgr2=0x7f78240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:17.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 -- 172.21.15.148:0/4172975350 shutdown_connections 2023-10-30T13:14:17.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 -- 172.21.15.148:0/4172975350 wait complete. 2023-10-30T13:14:17.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 Processor -- start 2023-10-30T13:14:17.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.823+0000 7f7828879640 1 -- start start 2023-10-30T13:14:17.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7828879640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78240c6b80 0x7f7824155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:17.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7828879640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 0x7f782413a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:17.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7828879640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 0x7f78241f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:17.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7828879640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78240ae9f0 con 0x7f78240c6f60 2023-10-30T13:14:17.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7828879640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78240af550 con 0x7f78240c6b80 2023-10-30T13:14:17.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7828879640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78241562a0 con 0x7f78240c8430 2023-10-30T13:14:17.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7821d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 0x7f782413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:17.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7821d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 0x7f782413a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60978/0 (socket says 172.21.15.148:60978) 2023-10-30T13:14:17.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7821d9b640 1 -- 172.21.15.148:0/1678081702 learned_addr learned my addr 172.21.15.148:0/1678081702 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:17.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7822d9d640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 0x7f78241f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:17.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f782259c640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78240c6b80 0x7f7824155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:17.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7821d9b640 1 -- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 msgr2=0x7f78241f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:17.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7821d9b640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 0x7f78241f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:17.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7821d9b640 1 -- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78240c6b80 msgr2=0x7f7824155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:17.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7821d9b640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78240c6b80 0x7f7824155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:17.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7821d9b640 1 -- 172.21.15.148:0/1678081702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78180094d0 con 0x7f78240c6f60 2023-10-30T13:14:17.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7821d9b640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 0x7f782413a830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f781801abb0 tx=0x7f78180192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:17.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7812ffd640 1 -- 172.21.15.148:0/1678081702 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78180b5640 con 0x7f78240c6f60 2023-10-30T13:14:17.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7812ffd640 1 -- 172.21.15.148:0/1678081702 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78180b57a0 con 0x7f78240c6f60 2023-10-30T13:14:17.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78240aec20 con 0x7f78240c6f60 2023-10-30T13:14:17.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7812ffd640 1 -- 172.21.15.148:0/1678081702 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78180b5b10 con 0x7f78240c6f60 2023-10-30T13:14:17.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78240af1e0 con 0x7f78240c6f60 2023-10-30T13:14:17.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78240b0f40 con 0x7f78240c6f60 2023-10-30T13:14:17.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.827+0000 7f7812ffd640 1 -- 172.21.15.148:0/1678081702 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78180b3070 con 0x7f78240c6f60 2023-10-30T13:14:17.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.831+0000 7f7812ffd640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78040664a0 0x7f7804068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:17.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.831+0000 7f7812ffd640 1 -- 172.21.15.148:0/1678081702 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f781811e150 con 0x7f78240c6f60 2023-10-30T13:14:17.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.831+0000 7f782259c640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78040664a0 0x7f7804068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:17.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.831+0000 7f782259c640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78040664a0 0x7f7804068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7814009a70 tx=0x7f7814008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:17.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:17.831+0000 7f7812ffd640 1 -- 172.21.15.148:0/1678081702 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78180eb4a0 con 0x7f78240c6f60 2023-10-30T13:14:18.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f782404e170 con 0x7f78240c6f60 2023-10-30T13:14:18.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7812ffd640 1 -- 172.21.15.148:0/1678081702 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f78180eeac0 con 0x7f78240c6f60 2023-10-30T13:14:18.019 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:18.020 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:10:51.357311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":7,"addr":"172.21.15.148:6837/1844602519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1844602519},{"type":"v1","addr":"172.21.15.148:6837","nonce":1844602519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:18.020 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 13 2023-10-30T13:14:18.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78040664a0 msgr2=0x7f7804068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:18.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78040664a0 0x7f7804068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7814009a70 tx=0x7f7814008040 comp rx=0 tx=0).stop 2023-10-30T13:14:18.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 msgr2=0x7f782413a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 0x7f782413a830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f781801abb0 tx=0x7f78180192a0 comp rx=0 tx=0).stop 2023-10-30T13:14:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 shutdown_connections 2023-10-30T13:14:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78040664a0 0x7f7804068960 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78240c6f60 0x7f782413a830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78240c8430 0x7f78241f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 --2- 172.21.15.148:0/1678081702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78240c6b80 0x7f7824155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 >> 172.21.15.148:0/1678081702 conn(0x7f78240c7910 msgr2=0x7f78240c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 shutdown_connections 2023-10-30T13:14:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:18.015+0000 7f7828879640 1 -- 172.21.15.148:0/1678081702 wait complete. 2023-10-30T13:14:18.040 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:14:20.042 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:20.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.183+0000 7f8064ec9640 1 Processor -- start 2023-10-30T13:14:20.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.183+0000 7f8064ec9640 1 -- start start 2023-10-30T13:14:20.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8064ec9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600715c0 0x7f8060070650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8064ec9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80600c1530 con 0x7f80600c8460 2023-10-30T13:14:20.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8064ec9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80600c0f70 con 0x7f8060157010 2023-10-30T13:14:20.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8064ec9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80600c21b0 con 0x7f80600715c0 2023-10-30T13:14:20.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8055d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f80600c8460 0x7f80601620a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41204/0 (socket says 172.21.15.148:41204) 2023-10-30T13:14:20.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8055d9b640 1 -- 172.21.15.148:0/1440162167 learned_addr learned my addr 172.21.15.148:0/1440162167 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:20.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805ed9d640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600715c0 0x7f8060070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000602799 0 0) 0x7f80600c1530 con 0x7f80600c8460 2023-10-30T13:14:20.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f804c003530 con 0x7f80600c8460 2023-10-30T13:14:20.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4020676345 0 0) 0x7f804c003530 con 0x7f80600c8460 2023-10-30T13:14:20.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600715c0 msgr2=0x7f8060070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:20.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600715c0 0x7f8060070650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 >> v1:172.21.15.150:6789/0 conn(0x7f8060157010 legacy=0x7f8060071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80600c1250 con 0x7f80600c8460 2023-10-30T13:14:20.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f8048001680 con 0x7f80600c8460 2023-10-30T13:14:20.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 >> v1:172.21.15.148:6789/0 conn(0x7f80600c8460 legacy=0x7f80601620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0039b0 0x7f804c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f804c0047d0 0x7f804c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600715c0 0x7f804c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80600c21b0 con 0x7f804c0047d0 2023-10-30T13:14:20.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804c00ba70 con 0x7f804c0039b0 2023-10-30T13:14:20.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804c00bc30 con 0x7f80600715c0 2023-10-30T13:14:20.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8055d9b640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f804c0047d0 0x7f804c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805e59c640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0039b0 0x7f804c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805ed9d640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600715c0 0x7f804c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8055d9b640 1 -- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600715c0 msgr2=0x7f804c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:20.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8055d9b640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600715c0 0x7f804c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8055d9b640 1 -- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0039b0 msgr2=0x7f804c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:20.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8055d9b640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0039b0 0x7f804c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f8055d9b640 1 -- 172.21.15.148:0/1440162167 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c003530 con 0x7f804c0047d0 2023-10-30T13:14:20.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.187+0000 7f805e59c640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0039b0 0x7f804c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:20.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f8055d9b640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f804c0047d0 0x7f804c004c50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f806015dd60 tx=0x7f804801f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6cc43f6fdd666ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:20.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048005ae0 con 0x7f804c0047d0 2023-10-30T13:14:20.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80480ab070 con 0x7f804c0047d0 2023-10-30T13:14:20.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f805dd9b640 1 -- 172.21.15.148:0/1440162167 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048005e80 con 0x7f804c0047d0 2023-10-30T13:14:20.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f8064ec9640 1 -- 172.21.15.148:0/1440162167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f804c0047d0 msgr2=0x7f804c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f8064ec9640 1 --2- 172.21.15.148:0/1440162167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f804c0047d0 0x7f804c004c50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f806015dd60 tx=0x7f804801f330 comp rx=0 tx=0).stop 2023-10-30T13:14:20.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f805ed9d640 1 -- 172.21.15.148:0/1440162167 reap_dead start 2023-10-30T13:14:20.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f8064ec9640 1 -- 172.21.15.148:0/1440162167 shutdown_connections 2023-10-30T13:14:20.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f8064ec9640 1 -- 172.21.15.148:0/1440162167 >> 172.21.15.148:0/1440162167 conn(0x7f80600cbeb0 msgr2=0x7f80600cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:20.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f8064ec9640 1 -- 172.21.15.148:0/1440162167 shutdown_connections 2023-10-30T13:14:20.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f8064ec9640 1 -- 172.21.15.148:0/1440162167 wait complete. 2023-10-30T13:14:20.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f8064ec9640 1 Processor -- start 2023-10-30T13:14:20.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.191+0000 7f8064ec9640 1 -- start start 2023-10-30T13:14:20.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8064ec9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0047d0 0x7f80600c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8064ec9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80601efe70 0x7f80601f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8064ec9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600ae2e0 0x7f80600ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8064ec9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80600af080 con 0x7f80601efe70 2023-10-30T13:14:20.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8064ec9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80601f0810 con 0x7f804c0047d0 2023-10-30T13:14:20.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8064ec9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806006bba0 con 0x7f80600ae2e0 2023-10-30T13:14:20.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8055d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80601efe70 0x7f80601f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8055d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80601efe70 0x7f80601f02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32772/0 (socket says 172.21.15.148:32772) 2023-10-30T13:14:20.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8055d9b640 1 -- 172.21.15.148:0/1402476544 learned_addr learned my addr 172.21.15.148:0/1402476544 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:20.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f805e59c640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0047d0 0x7f80600c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f805ed9d640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600ae2e0 0x7f80600ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8055d9b640 1 -- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600ae2e0 msgr2=0x7f80600ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8055d9b640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600ae2e0 0x7f80600ae760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8055d9b640 1 -- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0047d0 msgr2=0x7f80600c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8055d9b640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0047d0 0x7f80600c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8055d9b640 1 -- 172.21.15.148:0/1402476544 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80600b0730 con 0x7f80601efe70 2023-10-30T13:14:20.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8055d9b640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80601efe70 0x7f80601f02d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8048025fa0 tx=0x7f804c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:20.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f80577fe640 1 -- 172.21.15.148:0/1402476544 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048001640 con 0x7f80601efe70 2023-10-30T13:14:20.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f80577fe640 1 -- 172.21.15.148:0/1402476544 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8048007070 con 0x7f80601efe70 2023-10-30T13:14:20.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80601ef5b0 con 0x7f80601efe70 2023-10-30T13:14:20.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f80577fe640 1 -- 172.21.15.148:0/1402476544 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048007420 con 0x7f80601efe70 2023-10-30T13:14:20.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80600c5e80 con 0x7f80601efe70 2023-10-30T13:14:20.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.195+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80601efcc0 con 0x7f80601efe70 2023-10-30T13:14:20.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.199+0000 7f80577fe640 1 -- 172.21.15.148:0/1402476544 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8048007580 con 0x7f80601efe70 2023-10-30T13:14:20.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.199+0000 7f80577fe640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f80400664d0 0x7f8040068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.199+0000 7f80577fe640 1 -- 172.21.15.148:0/1402476544 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f8048121390 con 0x7f80601efe70 2023-10-30T13:14:20.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.199+0000 7f805e59c640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f80400664d0 0x7f8040068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.199+0000 7f805e59c640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f80400664d0 0x7f8040068990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8050000c10 tx=0x7f8050009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:20.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.203+0000 7f80577fe640 1 -- 172.21.15.148:0/1402476544 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f80480ee6a0 con 0x7f80601efe70 2023-10-30T13:14:20.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.383+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f806004e170 con 0x7f80601efe70 2023-10-30T13:14:20.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.383+0000 7f80577fe640 1 -- 172.21.15.148:0/1402476544 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f80480f1cc0 con 0x7f80601efe70 2023-10-30T13:14:20.389 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:20.390 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:19.364120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:20.390 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 14 2023-10-30T13:14:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f80400664d0 msgr2=0x7f8040068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f80400664d0 0x7f8040068990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8050000c10 tx=0x7f8050009040 comp rx=0 tx=0).stop 2023-10-30T13:14:20.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80601efe70 msgr2=0x7f80601f02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80601efe70 0x7f80601f02d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8048025fa0 tx=0x7f804c005800 comp rx=0 tx=0).stop 2023-10-30T13:14:20.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 shutdown_connections 2023-10-30T13:14:20.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f80400664d0 0x7f8040068990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80601efe70 0x7f80601f02d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80600ae2e0 0x7f80600ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 --2- 172.21.15.148:0/1402476544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f804c0047d0 0x7f80600c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 >> 172.21.15.148:0/1402476544 conn(0x7f80600cbeb0 msgr2=0x7f806014f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 shutdown_connections 2023-10-30T13:14:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.387+0000 7f8064ec9640 1 -- 172.21.15.148:0/1402476544 wait complete. 2023-10-30T13:14:20.410 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:14:20.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:14:20.411 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:20.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72fd9d640 1 Processor -- start 2023-10-30T13:14:20.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72fd9d640 1 -- start start 2023-10-30T13:14:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72fd9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7280c05d0 0x7fe72813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72fd9d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7280c0a50 con 0x7fe7280c6b80 2023-10-30T13:14:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72fd9d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72806f6e0 con 0x7fe7280c6f60 2023-10-30T13:14:20.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72fd9d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72806f9c0 con 0x7fe7280c05d0 2023-10-30T13:14:20.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72db39640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe7280c6b80 0x7fe7280bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41216/0 (socket says 172.21.15.148:41216) 2023-10-30T13:14:20.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72db39640 1 -- 172.21.15.148:0/4220087813 learned_addr learned my addr 172.21.15.148:0/4220087813 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:20.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72d338640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7280c05d0 0x7fe72813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402146476 0 0) 0x7fe7280c0a50 con 0x7fe7280c6b80 2023-10-30T13:14:20.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe714003530 con 0x7fe7280c6b80 2023-10-30T13:14:20.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4114219468 0 0) 0x7fe714003530 con 0x7fe7280c6b80 2023-10-30T13:14:20.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7280c05d0 msgr2=0x7fe72813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:20.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7280c05d0 0x7fe72813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 >> v1:172.21.15.150:6789/0 conn(0x7fe7280c6f60 legacy=0x7fe7281566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72806fca0 con 0x7fe7280c6b80 2023-10-30T13:14:20.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fe724001680 con 0x7fe7280c6b80 2023-10-30T13:14:20.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 >> v1:172.21.15.148:6789/0 conn(0x7fe7280c6b80 legacy=0x7fe7280bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7280c05d0 0x7fe7140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe714003ef0 0x7fe7140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe714004d10 0x7fe714005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72806f9c0 con 0x7fe714003ef0 2023-10-30T13:14:20.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71400b9f0 con 0x7fe714004d10 2023-10-30T13:14:20.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71400bbb0 con 0x7fe7280c05d0 2023-10-30T13:14:20.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.543+0000 7fe72d338640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe714003ef0 0x7fe7140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72db39640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7280c05d0 0x7fe7140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72e33a640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe714004d10 0x7fe714005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72d338640 1 -- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7280c05d0 msgr2=0x7fe7140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:20.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72d338640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe7280c05d0 0x7fe7140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72d338640 1 -- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe714004d10 msgr2=0x7fe714005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:20.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72d338640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe714004d10 0x7fe714005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72d338640 1 -- 172.21.15.148:0/4220087813 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe714003530 con 0x7fe714003ef0 2023-10-30T13:14:20.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72e33a640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe714004d10 0x7fe714005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:20.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72d338640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe714003ef0 0x7fe7140047d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe718009ae0 tx=0x7fe71801c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7971e22461cd4ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:20.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7180a5580 con 0x7fe714003ef0 2023-10-30T13:14:20.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7180a56e0 con 0x7fe714003ef0 2023-10-30T13:14:20.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72cb37640 1 -- 172.21.15.148:0/4220087813 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7180a5a90 con 0x7fe714003ef0 2023-10-30T13:14:20.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72fd9d640 1 -- 172.21.15.148:0/4220087813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe714003ef0 msgr2=0x7fe7140047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72fd9d640 1 --2- 172.21.15.148:0/4220087813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe714003ef0 0x7fe7140047d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe718009ae0 tx=0x7fe71801c990 comp rx=0 tx=0).stop 2023-10-30T13:14:20.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.547+0000 7fe72e33a640 1 -- 172.21.15.148:0/4220087813 reap_dead start 2023-10-30T13:14:20.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 -- 172.21.15.148:0/4220087813 shutdown_connections 2023-10-30T13:14:20.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 -- 172.21.15.148:0/4220087813 >> 172.21.15.148:0/4220087813 conn(0x7fe7280c7910 msgr2=0x7fe7280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:20.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 -- 172.21.15.148:0/4220087813 shutdown_connections 2023-10-30T13:14:20.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 -- 172.21.15.148:0/4220087813 wait complete. 2023-10-30T13:14:20.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 Processor -- start 2023-10-30T13:14:20.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 -- start start 2023-10-30T13:14:20.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe714003ef0 0x7fe7280c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe72813e560 0x7fe7281544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe728135ab0 0x7fe728135f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72813ddf0 con 0x7fe72813e560 2023-10-30T13:14:20.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe728158ab0 con 0x7fe728135ab0 2023-10-30T13:14:20.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72fd9d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7281549e0 con 0x7fe714003ef0 2023-10-30T13:14:20.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72d338640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe72813e560 0x7fe7281544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72d338640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe72813e560 0x7fe7281544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32792/0 (socket says 172.21.15.148:32792) 2023-10-30T13:14:20.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72d338640 1 -- 172.21.15.148:0/1311633291 learned_addr learned my addr 172.21.15.148:0/1311633291 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:20.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72db39640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe714003ef0 0x7fe7280c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72e33a640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe728135ab0 0x7fe728135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72d338640 1 -- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe714003ef0 msgr2=0x7fe7280c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72d338640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe714003ef0 0x7fe7280c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72d338640 1 -- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe728135ab0 msgr2=0x7fe728135f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72d338640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe728135ab0 0x7fe728135f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.551+0000 7fe72d338640 1 -- 172.21.15.148:0/1311633291 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe728156780 con 0x7fe72813e560 2023-10-30T13:14:20.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.555+0000 7fe72d338640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe72813e560 0x7fe7281544a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe718009ae0 tx=0x7fe714005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:20.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.555+0000 7fe71e7fc640 1 -- 172.21.15.148:0/1311633291 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe718004230 con 0x7fe72813e560 2023-10-30T13:14:20.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.555+0000 7fe71e7fc640 1 -- 172.21.15.148:0/1311633291 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe718004390 con 0x7fe72813e560 2023-10-30T13:14:20.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.555+0000 7fe71e7fc640 1 -- 172.21.15.148:0/1311633291 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe718003820 con 0x7fe72813e560 2023-10-30T13:14:20.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.555+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72813c8c0 con 0x7fe72813e560 2023-10-30T13:14:20.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.555+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72813ce80 con 0x7fe72813e560 2023-10-30T13:14:20.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.555+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72813d040 con 0x7fe72813e560 2023-10-30T13:14:20.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.555+0000 7fe71e7fc640 1 -- 172.21.15.148:0/1311633291 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7180b4040 con 0x7fe72813e560 2023-10-30T13:14:20.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.555+0000 7fe71e7fc640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe70c0664f0 0x7fe70c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:20.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.559+0000 7fe71e7fc640 1 -- 172.21.15.148:0/1311633291 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fe71811dbf0 con 0x7fe72813e560 2023-10-30T13:14:20.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.559+0000 7fe72db39640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe70c0664f0 0x7fe70c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:20.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.559+0000 7fe72db39640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe70c0664f0 0x7fe70c0689b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe724005b20 tx=0x7fe724007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:20.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.559+0000 7fe71e7fc640 1 -- 172.21.15.148:0/1311633291 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7180eb080 con 0x7fe72813e560 2023-10-30T13:14:20.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.747+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe72804e170 con 0x7fe72813e560 2023-10-30T13:14:20.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.747+0000 7fe71e7fc640 1 -- 172.21.15.148:0/1311633291 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fe7180b2020 con 0x7fe72813e560 2023-10-30T13:14:20.753 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:20.753 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:19.364120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:20.753 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 14 2023-10-30T13:14:20.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe70c0664f0 msgr2=0x7fe70c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe70c0664f0 0x7fe70c0689b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe724005b20 tx=0x7fe724007040 comp rx=0 tx=0).stop 2023-10-30T13:14:20.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe72813e560 msgr2=0x7fe7281544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:20.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe72813e560 0x7fe7281544a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe718009ae0 tx=0x7fe714005800 comp rx=0 tx=0).stop 2023-10-30T13:14:20.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 shutdown_connections 2023-10-30T13:14:20.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe70c0664f0 0x7fe70c0689b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe72813e560 0x7fe7281544a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe728135ab0 0x7fe728135f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 --2- 172.21.15.148:0/1311633291 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe714003ef0 0x7fe7280c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:20.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 >> 172.21.15.148:0/1311633291 conn(0x7fe7280c7910 msgr2=0x7fe7280c1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:20.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 shutdown_connections 2023-10-30T13:14:20.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:20.751+0000 7fe72fd9d640 1 -- 172.21.15.148:0/1311633291 wait complete. 2023-10-30T13:14:20.774 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 14, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 14, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4250": { "addr": "172.21.15.148:6835/3749586444", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 3749586444, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 3749586444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_4270": { "addr": "172.21.15.150:6836/1620877178", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1620877178, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 1620877178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4270, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4285": { "addr": "172.21.15.148:6834/4162935019", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 4162935019, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 4162935019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 23, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:14:19.364120+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4270, "mds_2": 4250 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:14:22.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:22.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.915+0000 7f0b5c965640 1 Processor -- start 2023-10-30T13:14:22.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.915+0000 7f0b5c965640 1 -- start start 2023-10-30T13:14:22.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.915+0000 7f0b5c965640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0b580c6b80 0x7f0b580bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:22.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.915+0000 7f0b5c965640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b580c6f60 0x7f0b580c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:22.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.915+0000 7f0b5c965640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580c0a50 con 0x7f0b580c8430 2023-10-30T13:14:22.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.915+0000 7f0b5c965640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b5806f6e0 con 0x7f0b580c6b80 2023-10-30T13:14:22.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.915+0000 7f0b5c965640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b5806f9c0 con 0x7f0b580c6f60 2023-10-30T13:14:22.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b5659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0b580c6b80 0x7f0b580bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:22.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b56d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0b580c8430 0x7f0b5813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47444/0 (socket says 172.21.15.148:47444) 2023-10-30T13:14:22.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b5659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0b580c6b80 0x7f0b580bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46874/0 (socket says 172.21.15.148:46874) 2023-10-30T13:14:22.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b56d9d640 1 -- 172.21.15.148:0/1659429749 learned_addr learned my addr 172.21.15.148:0/1659429749 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:22.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b4dd9b640 1 --2- 172.21.15.148:0/1659429749 >> v2:172.21.15.150:3300/0 conn(0x7f0b580c6f60 0x7f0b580c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:22.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b5659c640 1 -- 172.21.15.148:0/1659429749 >> v2:172.21.15.150:3300/0 conn(0x7f0b580c6f60 msgr2=0x7f0b580c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:22.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b5659c640 1 --2- 172.21.15.148:0/1659429749 >> v2:172.21.15.150:3300/0 conn(0x7f0b580c6f60 0x7f0b580c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:22.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b5659c640 1 -- 172.21.15.148:0/1659429749 >> v1:172.21.15.148:6789/0 conn(0x7f0b580c8430 legacy=0x7f0b5813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:22.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b55d9b640 1 -- 172.21.15.148:0/1659429749 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996096862 0 0) 0x7f0b580c0a50 con 0x7f0b580c8430 2023-10-30T13:14:22.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b5659c640 1 -- 172.21.15.148:0/1659429749 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5806fca0 con 0x7f0b580c6b80 2023-10-30T13:14:22.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b5659c640 1 --2- 172.21.15.148:0/1659429749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 0x7f0b580bfec0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0b58159b90 tx=0x7f0b4001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4258fe4abbc7e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:22.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b55d9b640 1 -- 172.21.15.148:0/1659429749 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b400ab600 con 0x7f0b580c6b80 2023-10-30T13:14:22.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b55d9b640 1 -- 172.21.15.148:0/1659429749 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b400a7050 con 0x7f0b580c6b80 2023-10-30T13:14:22.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.919+0000 7f0b55d9b640 1 -- 172.21.15.148:0/1659429749 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b400ab980 con 0x7f0b580c6b80 2023-10-30T13:14:22.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 -- 172.21.15.148:0/1659429749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 msgr2=0x7f0b580bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:22.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 --2- 172.21.15.148:0/1659429749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 0x7f0b580bfec0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0b58159b90 tx=0x7f0b4001c460 comp rx=0 tx=0).stop 2023-10-30T13:14:22.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 -- 172.21.15.148:0/1659429749 shutdown_connections 2023-10-30T13:14:22.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 --2- 172.21.15.148:0/1659429749 >> v2:172.21.15.150:3300/0 conn(0x7f0b580c6f60 0x7f0b580c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:22.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 --2- 172.21.15.148:0/1659429749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 0x7f0b580bfec0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:22.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 -- 172.21.15.148:0/1659429749 >> 172.21.15.148:0/1659429749 conn(0x7f0b580c7910 msgr2=0x7f0b580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:22.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 -- 172.21.15.148:0/1659429749 shutdown_connections 2023-10-30T13:14:22.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 -- 172.21.15.148:0/1659429749 wait complete. 2023-10-30T13:14:22.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 Processor -- start 2023-10-30T13:14:22.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 -- start start 2023-10-30T13:14:22.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 0x7f0b5813a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:22.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b580c6f60 0x7f0b5813a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:22.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8430 0x7f0b58154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:22.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b580ca770 con 0x7f0b580c6b80 2023-10-30T13:14:22.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b581594d0 con 0x7f0b580c6f60 2023-10-30T13:14:22.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5c965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b58159720 con 0x7f0b580c8430 2023-10-30T13:14:22.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 0x7f0b5813a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:22.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 0x7f0b5813a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46882/0 (socket says 172.21.15.148:46882) 2023-10-30T13:14:22.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.923+0000 7f0b5659c640 1 -- 172.21.15.148:0/1855277400 learned_addr learned my addr 172.21.15.148:0/1855277400 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:22.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b4dd9b640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b580c6f60 0x7f0b5813a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:22.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b56d9d640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8430 0x7f0b58154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:22.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b5659c640 1 -- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8430 msgr2=0x7f0b58154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:22.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b5659c640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8430 0x7f0b58154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:22.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b5659c640 1 -- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b580c6f60 msgr2=0x7f0b5813a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:22.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b5659c640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b580c6f60 0x7f0b5813a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:22.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b5659c640 1 -- 172.21.15.148:0/1855277400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b40009420 con 0x7f0b580c6b80 2023-10-30T13:14:22.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b4dd9b640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b580c6f60 0x7f0b5813a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:22.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b5659c640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 0x7f0b5813a320 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0b40019070 tx=0x7f0b40019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:22.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b4f7fe640 1 -- 172.21.15.148:0/1855277400 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b40004230 con 0x7f0b580c6b80 2023-10-30T13:14:22.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b4f7fe640 1 -- 172.21.15.148:0/1855277400 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b40004390 con 0x7f0b580c6b80 2023-10-30T13:14:22.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b4f7fe640 1 -- 172.21.15.148:0/1855277400 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b40003820 con 0x7f0b580c6b80 2023-10-30T13:14:22.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b58071ae0 con 0x7f0b580c6b80 2023-10-30T13:14:22.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b581f2270 con 0x7f0b580c6b80 2023-10-30T13:14:22.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b58154c80 con 0x7f0b580c6b80 2023-10-30T13:14:22.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.927+0000 7f0b4f7fe640 1 -- 172.21.15.148:0/1855277400 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b400b2040 con 0x7f0b580c6b80 2023-10-30T13:14:22.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.931+0000 7f0b4f7fe640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b380664a0 0x7f0b38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:22.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.931+0000 7f0b4f7fe640 1 -- 172.21.15.148:0/1855277400 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f0b4011d430 con 0x7f0b580c6b80 2023-10-30T13:14:22.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.931+0000 7f0b4dd9b640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b380664a0 0x7f0b38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:22.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.931+0000 7f0b4dd9b640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b380664a0 0x7f0b38068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0b58072310 tx=0x7f0b48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:22.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:22.935+0000 7f0b4f7fe640 1 -- 172.21.15.148:0/1855277400 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b400ea720 con 0x7f0b580c6b80 2023-10-30T13:14:23.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b5804e170 con 0x7f0b580c6b80 2023-10-30T13:14:23.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b4f7fe640 1 -- 172.21.15.148:0/1855277400 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f0b40006e20 con 0x7f0b580c6b80 2023-10-30T13:14:23.120 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:23.121 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:19.364120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:23.121 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 14 2023-10-30T13:14:23.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b380664a0 msgr2=0x7f0b38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:23.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b380664a0 0x7f0b38068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0b58072310 tx=0x7f0b48008040 comp rx=0 tx=0).stop 2023-10-30T13:14:23.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 msgr2=0x7f0b5813a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:23.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 0x7f0b5813a320 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0b40019070 tx=0x7f0b40019200 comp rx=0 tx=0).stop 2023-10-30T13:14:23.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 shutdown_connections 2023-10-30T13:14:23.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b380664a0 0x7f0b38068960 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:23.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b580c8430 0x7f0b58154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:23.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b580c6f60 0x7f0b5813a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:23.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 --2- 172.21.15.148:0/1855277400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b580c6b80 0x7f0b5813a320 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:23.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 >> 172.21.15.148:0/1855277400 conn(0x7f0b580c7910 msgr2=0x7f0b5813ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:23.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 shutdown_connections 2023-10-30T13:14:23.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:23.115+0000 7f0b5c965640 1 -- 172.21.15.148:0/1855277400 wait complete. 2023-10-30T13:14:23.141 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:14:23.459 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:14:23.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:14:25.142 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:25.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.283+0000 7f89ffcea640 1 Processor -- start 2023-10-30T13:14:25.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.283+0000 7f89ffcea640 1 -- start start 2023-10-30T13:14:25.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89ffcea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89f80c6b80 0x7f89f80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:25.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89ffcea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f89f80c6f60 0x7f89f80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:25.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89ffcea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 0x7f89f813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:25.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89ffcea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f80c0a50 con 0x7f89f80c6b80 2023-10-30T13:14:25.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89ffcea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f806f6e0 con 0x7f89f80c6f60 2023-10-30T13:14:25.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89ffcea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f806f9c0 con 0x7f89f80c8430 2023-10-30T13:14:25.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fda86640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89f80c6b80 0x7f89f80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:25.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fda86640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89f80c6b80 0x7f89f80bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46890/0 (socket says 172.21.15.148:46890) 2023-10-30T13:14:25.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fda86640 1 -- 172.21.15.148:0/4092644569 learned_addr learned my addr 172.21.15.148:0/4092644569 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:25.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fe287640 1 --2- 172.21.15.148:0/4092644569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 0x7f89f813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:25.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fd285640 1 --2- 172.21.15.148:0/4092644569 >> v2:172.21.15.150:3300/0 conn(0x7f89f80c6f60 0x7f89f80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:25.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fda86640 1 -- 172.21.15.148:0/4092644569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 msgr2=0x7f89f813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:25.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fda86640 1 --2- 172.21.15.148:0/4092644569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 0x7f89f813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fda86640 1 -- 172.21.15.148:0/4092644569 >> v2:172.21.15.150:3300/0 conn(0x7f89f80c6f60 msgr2=0x7f89f80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:25.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fda86640 1 --2- 172.21.15.148:0/4092644569 >> v2:172.21.15.150:3300/0 conn(0x7f89f80c6f60 0x7f89f80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.287+0000 7f89fda86640 1 -- 172.21.15.148:0/4092644569 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f806fca0 con 0x7f89f80c6b80 2023-10-30T13:14:25.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89fda86640 1 --2- 172.21.15.148:0/4092644569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 0x7f89f80bfec0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f89f8159b90 tx=0x7f89f401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabe2e172c92da92 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:25.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89fca84640 1 -- 172.21.15.148:0/4092644569 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f40ac630 con 0x7f89f80c6b80 2023-10-30T13:14:25.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89fca84640 1 -- 172.21.15.148:0/4092644569 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f40ac790 con 0x7f89f80c6b80 2023-10-30T13:14:25.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89fca84640 1 -- 172.21.15.148:0/4092644569 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f40acb20 con 0x7f89f80c6b80 2023-10-30T13:14:25.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 -- 172.21.15.148:0/4092644569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 msgr2=0x7f89f80bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:25.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 --2- 172.21.15.148:0/4092644569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 0x7f89f80bfec0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f89f8159b90 tx=0x7f89f401c500 comp rx=0 tx=0).stop 2023-10-30T13:14:25.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 -- 172.21.15.148:0/4092644569 shutdown_connections 2023-10-30T13:14:25.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 --2- 172.21.15.148:0/4092644569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 0x7f89f813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 --2- 172.21.15.148:0/4092644569 >> v2:172.21.15.150:3300/0 conn(0x7f89f80c6f60 0x7f89f80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 --2- 172.21.15.148:0/4092644569 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 0x7f89f80bfec0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 -- 172.21.15.148:0/4092644569 >> 172.21.15.148:0/4092644569 conn(0x7f89f80c7910 msgr2=0x7f89f80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:25.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 -- 172.21.15.148:0/4092644569 shutdown_connections 2023-10-30T13:14:25.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 -- 172.21.15.148:0/4092644569 wait complete. 2023-10-30T13:14:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 Processor -- start 2023-10-30T13:14:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.291+0000 7f89ffcea640 1 -- start start 2023-10-30T13:14:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ffcea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 0x7f89f8132090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ffcea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89f80c6f60 0x7f89f80c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ffcea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 0x7f89f8133100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ffcea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f813bd50 con 0x7f89f80c6b80 2023-10-30T13:14:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ffcea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f813b8c0 con 0x7f89f80c6f60 2023-10-30T13:14:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ffcea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80cabd0 con 0x7f89f80c8430 2023-10-30T13:14:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fda86640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 0x7f89f8132090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fda86640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 0x7f89f8132090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46892/0 (socket says 172.21.15.148:46892) 2023-10-30T13:14:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fda86640 1 -- 172.21.15.148:0/1299182570 learned_addr learned my addr 172.21.15.148:0/1299182570 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fd285640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89f80c6f60 0x7f89f80c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fe287640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 0x7f89f8133100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fda86640 1 -- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 msgr2=0x7f89f8133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fda86640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 0x7f89f8133100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fda86640 1 -- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89f80c6f60 msgr2=0x7f89f80c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fda86640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89f80c6f60 0x7f89f80c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fda86640 1 -- 172.21.15.148:0/1299182570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f40094d0 con 0x7f89f80c6b80 2023-10-30T13:14:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89fda86640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 0x7f89f8132090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f89f401abb0 tx=0x7f89f40192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ee7fc640 1 -- 172.21.15.148:0/1299182570 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f40b5850 con 0x7f89f80c6b80 2023-10-30T13:14:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ee7fc640 1 -- 172.21.15.148:0/1299182570 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f40b59b0 con 0x7f89f80c6b80 2023-10-30T13:14:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f80c8d40 con 0x7f89f80c6b80 2023-10-30T13:14:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ee7fc640 1 -- 172.21.15.148:0/1299182570 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f40b5d40 con 0x7f89f80c6b80 2023-10-30T13:14:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f80ca8f0 con 0x7f89f80c6b80 2023-10-30T13:14:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f813b690 con 0x7f89f80c6b80 2023-10-30T13:14:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.295+0000 7f89ee7fc640 1 -- 172.21.15.148:0/1299182570 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f89f40b3070 con 0x7f89f80c6b80 2023-10-30T13:14:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.299+0000 7f89ee7fc640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89dc0664a0 0x7f89dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.299+0000 7f89ee7fc640 1 -- 172.21.15.148:0/1299182570 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f89f411e3f0 con 0x7f89f80c6b80 2023-10-30T13:14:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.299+0000 7f89fd285640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89dc0664a0 0x7f89dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.299+0000 7f89fd285640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89dc0664a0 0x7f89dc068960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f89f81316f0 tx=0x7f89e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.303+0000 7f89ee7fc640 1 -- 172.21.15.148:0/1299182570 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f89f813b690 con 0x7f89f80c6b80 2023-10-30T13:14:25.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.483+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89f813b690 con 0x7f89f80c6b80 2023-10-30T13:14:25.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.483+0000 7f89ee7fc640 1 -- 172.21.15.148:0/1299182570 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f89f813b690 con 0x7f89f80c6b80 2023-10-30T13:14:25.491 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:25.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:24.588545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:resolve","state_seq":61,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:25.491 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 15 2023-10-30T13:14:25.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89dc0664a0 msgr2=0x7f89dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:25.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89dc0664a0 0x7f89dc068960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f89f81316f0 tx=0x7f89e8008040 comp rx=0 tx=0).stop 2023-10-30T13:14:25.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 msgr2=0x7f89f8132090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:25.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 0x7f89f8132090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f89f401abb0 tx=0x7f89f40192a0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 shutdown_connections 2023-10-30T13:14:25.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89dc0664a0 0x7f89dc068960 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f89f80c8430 0x7f89f8133100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89f80c6f60 0x7f89f80c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 --2- 172.21.15.148:0/1299182570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89f80c6b80 0x7f89f8132090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:25.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 >> 172.21.15.148:0/1299182570 conn(0x7f89f80c7910 msgr2=0x7f89f80c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:25.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 shutdown_connections 2023-10-30T13:14:25.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:25.487+0000 7f89ffcea640 1 -- 172.21.15.148:0/1299182570 wait complete. 2023-10-30T13:14:25.511 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:14:27.512 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:27.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f49409f5640 1 Processor -- start 2023-10-30T13:14:27.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f49409f5640 1 -- start start 2023-10-30T13:14:27.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f49409f5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f493c1538a0 0x7f493c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:27.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f49409f5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c0cf160 0x7f493c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:27.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f49409f5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c0cb800 con 0x7f493c0ca0e0 2023-10-30T13:14:27.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f49409f5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c15dc30 con 0x7f493c1538a0 2023-10-30T13:14:27.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f49409f5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0cb330 con 0x7f493c0cf160 2023-10-30T13:14:27.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f493ad9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c0cf160 0x7f493c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:27.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f493ad9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c0cf160 0x7f493c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45644/0 (socket says 172.21.15.148:45644) 2023-10-30T13:14:27.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f493ad9d640 1 -- 172.21.15.148:0/2446041312 learned_addr learned my addr 172.21.15.148:0/2446041312 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:27.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f493a59c640 1 --2- 172.21.15.148:0/2446041312 >> v2:172.21.15.150:3300/0 conn(0x7f493c1538a0 0x7f493c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:27.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f493ad9d640 1 -- 172.21.15.148:0/2446041312 >> v2:172.21.15.150:3300/0 conn(0x7f493c1538a0 msgr2=0x7f493c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:27.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f493ad9d640 1 --2- 172.21.15.148:0/2446041312 >> v2:172.21.15.150:3300/0 conn(0x7f493c1538a0 0x7f493c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f493ad9d640 1 -- 172.21.15.148:0/2446041312 >> v1:172.21.15.150:6789/0 conn(0x7f493c0ca0e0 legacy=0x7f493c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:27.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.655+0000 7f493ad9d640 1 -- 172.21.15.148:0/2446041312 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0cad70 con 0x7f493c0cf160 2023-10-30T13:14:27.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f493ad9d640 1 --2- 172.21.15.148:0/2446041312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c0cf160 0x7f493c0cf590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4934009e50 tx=0x7f493401f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7893fa0293f03e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:27.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f4939d9b640 1 -- 172.21.15.148:0/2446041312 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49340af400 con 0x7f493c0cf160 2023-10-30T13:14:27.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f4939d9b640 1 -- 172.21.15.148:0/2446041312 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49340ab050 con 0x7f493c0cf160 2023-10-30T13:14:27.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f4939d9b640 1 -- 172.21.15.148:0/2446041312 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49340af780 con 0x7f493c0cf160 2023-10-30T13:14:27.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f49409f5640 1 -- 172.21.15.148:0/2446041312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c0cf160 msgr2=0x7f493c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:27.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f49409f5640 1 --2- 172.21.15.148:0/2446041312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c0cf160 0x7f493c0cf590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4934009e50 tx=0x7f493401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f49409f5640 1 -- 172.21.15.148:0/2446041312 shutdown_connections 2023-10-30T13:14:27.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f49409f5640 1 --2- 172.21.15.148:0/2446041312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c0cf160 0x7f493c0cf590 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f49409f5640 1 --2- 172.21.15.148:0/2446041312 >> v2:172.21.15.150:3300/0 conn(0x7f493c1538a0 0x7f493c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f49409f5640 1 -- 172.21.15.148:0/2446041312 >> 172.21.15.148:0/2446041312 conn(0x7f493c06bfe0 msgr2=0x7f493c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:27.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f49409f5640 1 -- 172.21.15.148:0/2446041312 shutdown_connections 2023-10-30T13:14:27.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.659+0000 7f49409f5640 1 -- 172.21.15.148:0/2446041312 wait complete. 2023-10-30T13:14:27.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 Processor -- start 2023-10-30T13:14:27.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 -- start start 2023-10-30T13:14:27.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f493c0ca0e0 0x7f493c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:27.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c0cf160 0x7f493c142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:27.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c1538a0 0x7f493c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:27.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0ce820 con 0x7f493c0ca0e0 2023-10-30T13:14:27.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0ceae0 con 0x7f493c0cf160 2023-10-30T13:14:27.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0cedc0 con 0x7f493c1538a0 2023-10-30T13:14:27.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f493c0ca0e0 0x7f493c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:27.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f493c0ca0e0 0x7f493c0ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46896/0 (socket says 172.21.15.148:46896) 2023-10-30T13:14:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493a59c640 1 -- 172.21.15.148:0/2563396390 learned_addr learned my addr 172.21.15.148:0/2563396390 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493ad9d640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c1538a0 0x7f493c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f4933fff640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c0cf160 0x7f493c142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493a59c640 1 -- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c1538a0 msgr2=0x7f493c157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493a59c640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c1538a0 0x7f493c157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493a59c640 1 -- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c0cf160 msgr2=0x7f493c142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493a59c640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c0cf160 0x7f493c142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493a59c640 1 -- 172.21.15.148:0/2563396390 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4934009b30 con 0x7f493c0ca0e0 2023-10-30T13:14:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f4933fff640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c0cf160 0x7f493c142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f493a59c640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f493c0ca0e0 0x7f493c0ccfb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f492c00e870 tx=0x7f492c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f4932ffd640 1 -- 172.21.15.148:0/2563396390 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c009510 con 0x7f493c0ca0e0 2023-10-30T13:14:27.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f4932ffd640 1 -- 172.21.15.148:0/2563396390 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c00eec0 con 0x7f493c0ca0e0 2023-10-30T13:14:27.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493c1efb80 con 0x7f493c0ca0e0 2023-10-30T13:14:27.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f4932ffd640 1 -- 172.21.15.148:0/2563396390 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c00cb40 con 0x7f493c0ca0e0 2023-10-30T13:14:27.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.663+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c1f0140 con 0x7f493c0ca0e0 2023-10-30T13:14:27.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.667+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493c0b2a50 con 0x7f493c0ca0e0 2023-10-30T13:14:27.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.667+0000 7f4932ffd640 1 -- 172.21.15.148:0/2563396390 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f492c00cca0 con 0x7f493c0ca0e0 2023-10-30T13:14:27.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.667+0000 7f4932ffd640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f491c0664a0 0x7f491c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:27.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.667+0000 7f4932ffd640 1 -- 172.21.15.148:0/2563396390 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f492c084040 con 0x7f493c0ca0e0 2023-10-30T13:14:27.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.667+0000 7f4933fff640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f491c0664a0 0x7f491c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:27.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.667+0000 7f4933fff640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f491c0664a0 0x7f491c068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f493c146660 tx=0x7f4924009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:27.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.671+0000 7f4932ffd640 1 -- 172.21.15.148:0/2563396390 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f492c051480 con 0x7f493c0ca0e0 2023-10-30T13:14:27.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f493c159850 con 0x7f493c0ca0e0 2023-10-30T13:14:27.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f4932ffd640 1 -- 172.21.15.148:0/2563396390 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f492c054aa0 con 0x7f493c0ca0e0 2023-10-30T13:14:27.856 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:27.856 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:24.588545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:resolve","state_seq":61,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:27.857 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 15 2023-10-30T13:14:27.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f491c0664a0 msgr2=0x7f491c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:27.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f491c0664a0 0x7f491c068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f493c146660 tx=0x7f4924009040 comp rx=0 tx=0).stop 2023-10-30T13:14:27.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f493c0ca0e0 msgr2=0x7f493c0ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:27.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f493c0ca0e0 0x7f493c0ccfb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f492c00e870 tx=0x7f492c00ec70 comp rx=0 tx=0).stop 2023-10-30T13:14:27.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 shutdown_connections 2023-10-30T13:14:27.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f491c0664a0 0x7f491c068960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f493c1538a0 0x7f493c157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c0cf160 0x7f493c142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 --2- 172.21.15.148:0/2563396390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f493c0ca0e0 0x7f493c0ccfb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:27.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 >> 172.21.15.148:0/2563396390 conn(0x7f493c06bfe0 msgr2=0x7f493c0c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:27.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 shutdown_connections 2023-10-30T13:14:27.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:27.851+0000 7f49409f5640 1 -- 172.21.15.148:0/2563396390 wait complete. 2023-10-30T13:14:27.877 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:14:29.878 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:30.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.019+0000 7f1377fc4640 1 Processor -- start 2023-10-30T13:14:30.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.019+0000 7f1377fc4640 1 -- start start 2023-10-30T13:14:30.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1377fc4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f13700c6b80 0x7f13700bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:30.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1377fc4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f137013f4a0 0x7f13700c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:30.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1377fc4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13700c0a50 con 0x7f13700c6f60 2023-10-30T13:14:30.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1377fc4640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f137006f6e0 con 0x7f13700c6b80 2023-10-30T13:14:30.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1377fc4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f137006f9c0 con 0x7f137013f4a0 2023-10-30T13:14:30.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1375d60640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f13700c6b80 0x7f13700bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:30.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1375d60640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f13700c6b80 0x7f13700bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46906/0 (socket says 172.21.15.148:46906) 2023-10-30T13:14:30.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1375d60640 1 -- 172.21.15.148:0/1237186785 learned_addr learned my addr 172.21.15.148:0/1237186785 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:30.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1376561640 1 --2- 172.21.15.148:0/1237186785 >> v2:172.21.15.150:3300/0 conn(0x7f137013f4a0 0x7f13700c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:30.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1375d60640 1 -- 172.21.15.148:0/1237186785 >> v2:172.21.15.150:3300/0 conn(0x7f137013f4a0 msgr2=0x7f13700c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:30.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1375d60640 1 --2- 172.21.15.148:0/1237186785 >> v2:172.21.15.150:3300/0 conn(0x7f137013f4a0 0x7f13700c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:30.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1375d60640 1 -- 172.21.15.148:0/1237186785 >> v1:172.21.15.150:6789/0 conn(0x7f13700c6f60 legacy=0x7f137013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:30.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.023+0000 7f1375d60640 1 -- 172.21.15.148:0/1237186785 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137006fca0 con 0x7f13700c6b80 2023-10-30T13:14:30.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1375d60640 1 --2- 172.21.15.148:0/1237186785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 0x7f13700bfec0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f13700c8980 tx=0x7f136c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5441adf9f9d29d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:30.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1374d5e640 1 -- 172.21.15.148:0/1237186785 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c0ab610 con 0x7f13700c6b80 2023-10-30T13:14:30.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1374d5e640 1 -- 172.21.15.148:0/1237186785 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136c0ab770 con 0x7f13700c6b80 2023-10-30T13:14:30.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1374d5e640 1 -- 172.21.15.148:0/1237186785 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c0abb00 con 0x7f13700c6b80 2023-10-30T13:14:30.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 -- 172.21.15.148:0/1237186785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 msgr2=0x7f13700bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:30.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 --2- 172.21.15.148:0/1237186785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 0x7f13700bfec0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f13700c8980 tx=0x7f136c01c500 comp rx=0 tx=0).stop 2023-10-30T13:14:30.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 -- 172.21.15.148:0/1237186785 shutdown_connections 2023-10-30T13:14:30.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 --2- 172.21.15.148:0/1237186785 >> v2:172.21.15.150:3300/0 conn(0x7f137013f4a0 0x7f13700c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:30.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 --2- 172.21.15.148:0/1237186785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 0x7f13700bfec0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:30.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 -- 172.21.15.148:0/1237186785 >> 172.21.15.148:0/1237186785 conn(0x7f13700c7910 msgr2=0x7f13700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:30.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 -- 172.21.15.148:0/1237186785 shutdown_connections 2023-10-30T13:14:30.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 -- 172.21.15.148:0/1237186785 wait complete. 2023-10-30T13:14:30.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 Processor -- start 2023-10-30T13:14:30.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.027+0000 7f1377fc4640 1 -- start start 2023-10-30T13:14:30.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1377fc4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 0x7f13700ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:30.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1377fc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13700c6f60 0x7f13700c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:30.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1377fc4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f137013f4a0 0x7f13700c2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:30.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1377fc4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137013e120 con 0x7f13700c6b80 2023-10-30T13:14:30.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1377fc4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13700c52f0 con 0x7f13700c6f60 2023-10-30T13:14:30.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1377fc4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13700c5050 con 0x7f137013f4a0 2023-10-30T13:14:30.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1375d60640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 0x7f13700ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:30.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1375d60640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 0x7f13700ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46920/0 (socket says 172.21.15.148:46920) 2023-10-30T13:14:30.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1375d60640 1 -- 172.21.15.148:0/3686988458 learned_addr learned my addr 172.21.15.148:0/3686988458 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:30.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1376561640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f137013f4a0 0x7f13700c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:30.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f137555f640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13700c6f60 0x7f13700c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:30.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1375d60640 1 -- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f137013f4a0 msgr2=0x7f13700c2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:30.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1375d60640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f137013f4a0 0x7f13700c2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:30.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1375d60640 1 -- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13700c6f60 msgr2=0x7f13700c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:30.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1375d60640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13700c6f60 0x7f13700c43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:30.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1375d60640 1 -- 172.21.15.148:0/3686988458 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c0094d0 con 0x7f13700c6b80 2023-10-30T13:14:30.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1376561640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f137013f4a0 0x7f13700c2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:30.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f137555f640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13700c6f60 0x7f13700c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:30.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1375d60640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 0x7f13700ca590 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f136c019110 tx=0x7f136c01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:30.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f13667fc640 1 -- 172.21.15.148:0/3686988458 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c0b4800 con 0x7f13700c6b80 2023-10-30T13:14:30.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f13667fc640 1 -- 172.21.15.148:0/3686988458 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136c0b4960 con 0x7f13700c6b80 2023-10-30T13:14:30.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137013d7f0 con 0x7f13700c6b80 2023-10-30T13:14:30.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f13667fc640 1 -- 172.21.15.148:0/3686988458 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c0b4d10 con 0x7f13700c6b80 2023-10-30T13:14:30.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137013ddb0 con 0x7f13700c6b80 2023-10-30T13:14:30.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137004e170 con 0x7f13700c6b80 2023-10-30T13:14:30.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.031+0000 7f13667fc640 1 -- 172.21.15.148:0/3686988458 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f136c0b2410 con 0x7f13700c6b80 2023-10-30T13:14:30.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.035+0000 7f13667fc640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13540664a0 0x7f1354068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:30.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.035+0000 7f13667fc640 1 -- 172.21.15.148:0/3686988458 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f136c11e7b0 con 0x7f13700c6b80 2023-10-30T13:14:30.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.035+0000 7f137555f640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13540664a0 0x7f1354068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:30.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.035+0000 7f137555f640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13540664a0 0x7f1354068960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f137006c3c0 tx=0x7f1360009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:30.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.039+0000 7f13667fc640 1 -- 172.21.15.148:0/3686988458 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f136c0ebb50 con 0x7f13700c6b80 2023-10-30T13:14:30.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.219+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f137013f0d0 con 0x7f13700c6b80 2023-10-30T13:14:30.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.219+0000 7f13667fc640 1 -- 172.21.15.148:0/3686988458 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f136c0ef170 con 0x7f13700c6b80 2023-10-30T13:14:30.226 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:30.226 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:24.588545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:resolve","state_seq":61,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:30.226 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 15 2023-10-30T13:14:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13540664a0 msgr2=0x7f1354068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13540664a0 0x7f1354068960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f137006c3c0 tx=0x7f1360009040 comp rx=0 tx=0).stop 2023-10-30T13:14:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 msgr2=0x7f13700ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 0x7f13700ca590 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f136c019110 tx=0x7f136c01ad40 comp rx=0 tx=0).stop 2023-10-30T13:14:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 shutdown_connections 2023-10-30T13:14:30.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13540664a0 0x7f1354068960 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:30.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f137013f4a0 0x7f13700c2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:30.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13700c6f60 0x7f13700c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:30.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 --2- 172.21.15.148:0/3686988458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13700c6b80 0x7f13700ca590 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:30.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 >> 172.21.15.148:0/3686988458 conn(0x7f13700c7910 msgr2=0x7f13700c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:30.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 shutdown_connections 2023-10-30T13:14:30.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:30.223+0000 7f1377fc4640 1 -- 172.21.15.148:0/3686988458 wait complete. 2023-10-30T13:14:30.247 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:14:32.247 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:32.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd99a09a640 1 Processor -- start 2023-10-30T13:14:32.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd99a09a640 1 -- start start 2023-10-30T13:14:32.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd99a09a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6b80 0x7fd9940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:32.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd99a09a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd99413f4a0 0x7fd9940c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:32.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd99a09a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9940c0a50 con 0x7fd9940c6f60 2023-10-30T13:14:32.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd99a09a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99406f6e0 con 0x7fd99413f4a0 2023-10-30T13:14:32.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd99a09a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99406f9c0 con 0x7fd9940c6b80 2023-10-30T13:14:32.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd9937fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6b80 0x7fd9940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:32.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd9937fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6b80 0x7fd9940bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:43294/0 (socket says 172.21.15.148:43294) 2023-10-30T13:14:32.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd9937fe640 1 -- 172.21.15.148:0/871863905 learned_addr learned my addr 172.21.15.148:0/871863905 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:32.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd993fff640 1 --2- 172.21.15.148:0/871863905 >> v2:172.21.15.150:3300/0 conn(0x7fd99413f4a0 0x7fd9940c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:32.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd9937fe640 1 -- 172.21.15.148:0/871863905 >> v2:172.21.15.150:3300/0 conn(0x7fd99413f4a0 msgr2=0x7fd9940c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:32.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd9937fe640 1 --2- 172.21.15.148:0/871863905 >> v2:172.21.15.150:3300/0 conn(0x7fd99413f4a0 0x7fd9940c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:32.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd9937fe640 1 -- 172.21.15.148:0/871863905 >> v1:172.21.15.150:6789/0 conn(0x7fd9940c6f60 legacy=0x7fd99413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:32.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.391+0000 7fd9937fe640 1 -- 172.21.15.148:0/871863905 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99406fca0 con 0x7fd9940c6b80 2023-10-30T13:14:32.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd9937fe640 1 --2- 172.21.15.148:0/871863905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6b80 0x7fd9940bfec0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd984009870 tx=0x7fd98401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e005bd21b190bd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:32.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd9927fc640 1 -- 172.21.15.148:0/871863905 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9840ab650 con 0x7fd9940c6b80 2023-10-30T13:14:32.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd9927fc640 1 -- 172.21.15.148:0/871863905 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9840ab7b0 con 0x7fd9940c6b80 2023-10-30T13:14:32.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd9927fc640 1 -- 172.21.15.148:0/871863905 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9840abb40 con 0x7fd9940c6b80 2023-10-30T13:14:32.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd99a09a640 1 -- 172.21.15.148:0/871863905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6b80 msgr2=0x7fd9940bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:32.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd99a09a640 1 --2- 172.21.15.148:0/871863905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6b80 0x7fd9940bfec0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd984009870 tx=0x7fd98401c540 comp rx=0 tx=0).stop 2023-10-30T13:14:32.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd99a09a640 1 -- 172.21.15.148:0/871863905 shutdown_connections 2023-10-30T13:14:32.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd99a09a640 1 --2- 172.21.15.148:0/871863905 >> v2:172.21.15.150:3300/0 conn(0x7fd99413f4a0 0x7fd9940c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:32.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd99a09a640 1 --2- 172.21.15.148:0/871863905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6b80 0x7fd9940bfec0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:32.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd99a09a640 1 -- 172.21.15.148:0/871863905 >> 172.21.15.148:0/871863905 conn(0x7fd9940c7910 msgr2=0x7fd9940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:32.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd99a09a640 1 -- 172.21.15.148:0/871863905 shutdown_connections 2023-10-30T13:14:32.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.395+0000 7fd99a09a640 1 -- 172.21.15.148:0/871863905 wait complete. 2023-10-30T13:14:32.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd99a09a640 1 Processor -- start 2023-10-30T13:14:32.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd99a09a640 1 -- start start 2023-10-30T13:14:32.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd99a09a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9940c6b80 0x7fd9940c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd99a09a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6f60 0x7fd9940c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd99a09a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd99413f4a0 0x7fd994134250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd99a09a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9940c94a0 con 0x7fd99413f4a0 2023-10-30T13:14:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd99a09a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9940b0170 con 0x7fd9940c6b80 2023-10-30T13:14:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd99a09a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd994133de0 con 0x7fd9940c6f60 2023-10-30T13:14:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd993fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd99413f4a0 0x7fd994134250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd993fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd99413f4a0 0x7fd994134250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46976/0 (socket says 172.21.15.148:46976) 2023-10-30T13:14:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd993fff640 1 -- 172.21.15.148:0/578253632 learned_addr learned my addr 172.21.15.148:0/578253632 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd992ffd640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6f60 0x7fd9940c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd9937fe640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9940c6b80 0x7fd9940c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd993fff640 1 -- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6f60 msgr2=0x7fd9940c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd993fff640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6f60 0x7fd9940c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd993fff640 1 -- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9940c6b80 msgr2=0x7fd9940c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd993fff640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9940c6b80 0x7fd9940c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd993fff640 1 -- 172.21.15.148:0/578253632 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9840094d0 con 0x7fd99413f4a0 2023-10-30T13:14:32.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.399+0000 7fd993fff640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd99413f4a0 0x7fd994134250 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00ef90 tx=0x7fd98c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:32.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.403+0000 7fd97bfff640 1 -- 172.21.15.148:0/578253632 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c009b20 con 0x7fd99413f4a0 2023-10-30T13:14:32.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.403+0000 7fd97bfff640 1 -- 172.21.15.148:0/578253632 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98c00f040 con 0x7fd99413f4a0 2023-10-30T13:14:32.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.403+0000 7fd97bfff640 1 -- 172.21.15.148:0/578253632 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0146c0 con 0x7fd99413f4a0 2023-10-30T13:14:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.403+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9940c1dc0 con 0x7fd99413f4a0 2023-10-30T13:14:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.403+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9940b03c0 con 0x7fd99413f4a0 2023-10-30T13:14:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.403+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9940b0b20 con 0x7fd99413f4a0 2023-10-30T13:14:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.403+0000 7fd97bfff640 1 -- 172.21.15.148:0/578253632 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd98c014820 con 0x7fd99413f4a0 2023-10-30T13:14:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.403+0000 7fd97bfff640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9800664d0 0x7fd980068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.403+0000 7fd97bfff640 1 -- 172.21.15.148:0/578253632 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fd98c084c70 con 0x7fd99413f4a0 2023-10-30T13:14:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.407+0000 7fd9937fe640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9800664d0 0x7fd980068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.407+0000 7fd9937fe640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9800664d0 0x7fd980068990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd9840af000 tx=0x7fd9840b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.407+0000 7fd97bfff640 1 -- 172.21.15.148:0/578253632 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd98c0520a0 con 0x7fd99413f4a0 2023-10-30T13:14:32.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.587+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd99404e170 con 0x7fd99413f4a0 2023-10-30T13:14:32.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd97bfff640 1 -- 172.21.15.148:0/578253632 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5750 (secure 0 0 0) 0x7fd98c018090 con 0x7fd99413f4a0 2023-10-30T13:14:32.595 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:32.595 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:30.381219+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:reconnect","state_seq":63,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:32.596 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 16 2023-10-30T13:14:32.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9800664d0 msgr2=0x7fd980068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:32.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9800664d0 0x7fd980068990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd9840af000 tx=0x7fd9840b6040 comp rx=0 tx=0).stop 2023-10-30T13:14:32.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd99413f4a0 msgr2=0x7fd994134250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:32.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd99413f4a0 0x7fd994134250 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00ef90 tx=0x7fd98c00c490 comp rx=0 tx=0).stop 2023-10-30T13:14:32.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 shutdown_connections 2023-10-30T13:14:32.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9800664d0 0x7fd980068990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:32.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd99413f4a0 0x7fd994134250 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:32.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9940c6f60 0x7fd9940c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:32.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 --2- 172.21.15.148:0/578253632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9940c6b80 0x7fd9940c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:32.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 >> 172.21.15.148:0/578253632 conn(0x7fd9940c7910 msgr2=0x7fd99413ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:32.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 shutdown_connections 2023-10-30T13:14:32.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:32.591+0000 7fd99a09a640 1 -- 172.21.15.148:0/578253632 wait complete. 2023-10-30T13:14:32.617 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 16, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 16, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4250": { "addr": "172.21.15.148:6835/3749586444", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 3749586444, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 3749586444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4250, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:reconnect", "state_seq": 63 }, "gid_4270": { "addr": "172.21.15.150:6836/1620877178", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1620877178, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 1620877178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4270, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4285": { "addr": "172.21.15.148:6834/4162935019", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 4162935019, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 4162935019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 23, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:14:30.381219+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4270, "mds_2": 4250 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:14:34.618 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:34.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ed510f640 1 Processor -- start 2023-10-30T13:14:34.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ed510f640 1 -- start start 2023-10-30T13:14:34.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ed510f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed00c6f60 0x7f3ed01566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:34.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ed510f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed00c0a50 con 0x7f3ed00c6b80 2023-10-30T13:14:34.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ed510f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed006f6e0 con 0x7f3ed00c05d0 2023-10-30T13:14:34.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ed510f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed006f9c0 con 0x7f3ed00c6f60 2023-10-30T13:14:34.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3eced9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3ed00c6b80 0x7f3ed00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40884/0 (socket says 172.21.15.148:40884) 2023-10-30T13:14:34.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3eced9d640 1 -- 172.21.15.148:0/2083144498 learned_addr learned my addr 172.21.15.148:0/2083144498 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:34.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ecf59e640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed00c6f60 0x7f3ed01566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:34.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698177933 0 0) 0x7f3ed00c0a50 con 0x7f3ed00c6b80 2023-10-30T13:14:34.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc005740 con 0x7f3ed00c6b80 2023-10-30T13:14:34.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735635850 0 0) 0x7f3ed006f6e0 con 0x7f3ed00c05d0 2023-10-30T13:14:34.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed00c0a50 con 0x7f3ed00c05d0 2023-10-30T13:14:34.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2537556520 0 0) 0x7f3ebc005740 con 0x7f3ed00c6b80 2023-10-30T13:14:34.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed00c6f60 msgr2=0x7f3ed01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:34.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed00c6f60 0x7f3ed01566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:34.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 >> v1:172.21.15.150:6789/0 conn(0x7f3ed00c05d0 legacy=0x7f3ed013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:34.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed006fca0 con 0x7f3ed00c6b80 2023-10-30T13:14:34.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f3ec0001680 con 0x7f3ed00c6b80 2023-10-30T13:14:34.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 >> v1:172.21.15.148:6789/0 conn(0x7f3ed00c6b80 legacy=0x7f3ed00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:34.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed00c6f60 0x7f3ebc005d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:34.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ebc0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:34.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ebc006f20 0x7f3ebc0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:34.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed006f9c0 con 0x7f3ebc006300 2023-10-30T13:14:34.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00dce0 con 0x7f3ebc006f20 2023-10-30T13:14:34.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00dea0 con 0x7f3ed00c6f60 2023-10-30T13:14:34.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ec659c640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ebc0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:34.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3eced9d640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed00c6f60 0x7f3ebc005d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:34.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.763+0000 7f3ecf59e640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ebc006f20 0x7f3ebc0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:34.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ec659c640 1 -- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed00c6f60 msgr2=0x7f3ebc005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:34.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ec659c640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed00c6f60 0x7f3ebc005d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:34.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ec659c640 1 -- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ebc006f20 msgr2=0x7f3ebc0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:34.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ec659c640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ebc006f20 0x7f3ebc0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:34.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ec659c640 1 -- 172.21.15.148:0/2083144498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc005740 con 0x7f3ebc006300 2023-10-30T13:14:34.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ecf59e640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ebc006f20 0x7f3ebc0073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:34.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ec659c640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ebc0069e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3ed006c9a0 tx=0x7f3eb801ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f518c770073bc3eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80a8b90 con 0x7f3ebc006300 2023-10-30T13:14:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb80a8d10 con 0x7f3ebc006300 2023-10-30T13:14:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ece59c640 1 -- 172.21.15.148:0/2083144498 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8003bc0 con 0x7f3ebc006300 2023-10-30T13:14:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ed510f640 1 -- 172.21.15.148:0/2083144498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 msgr2=0x7f3ebc0069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ed510f640 1 --2- 172.21.15.148:0/2083144498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ebc0069e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3ed006c9a0 tx=0x7f3eb801ef70 comp rx=0 tx=0).stop 2023-10-30T13:14:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ecf59e640 1 -- 172.21.15.148:0/2083144498 reap_dead start 2023-10-30T13:14:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ed510f640 1 -- 172.21.15.148:0/2083144498 shutdown_connections 2023-10-30T13:14:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ed510f640 1 -- 172.21.15.148:0/2083144498 >> 172.21.15.148:0/2083144498 conn(0x7f3ed00c7910 msgr2=0x7f3ed00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ed510f640 1 -- 172.21.15.148:0/2083144498 shutdown_connections 2023-10-30T13:14:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.767+0000 7f3ed510f640 1 -- 172.21.15.148:0/2083144498 wait complete. 2023-10-30T13:14:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ed510f640 1 Processor -- start 2023-10-30T13:14:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ed510f640 1 -- start start 2023-10-30T13:14:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ed510f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ed01f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ed510f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed014dfd0 0x7f3ed01f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ed510f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ed014f2b0 0x7f3ed014f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ed510f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed01392b0 con 0x7f3ebc006300 2023-10-30T13:14:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ed510f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed01371a0 con 0x7f3ed014f2b0 2023-10-30T13:14:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ed510f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed014eea0 con 0x7f3ed014dfd0 2023-10-30T13:14:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3eced9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ed01f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3eced9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ed01f2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47000/0 (socket says 172.21.15.148:47000) 2023-10-30T13:14:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3eced9d640 1 -- 172.21.15.148:0/2745433741 learned_addr learned my addr 172.21.15.148:0/2745433741 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ecf59e640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ed014f2b0 0x7f3ed014f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ec659c640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed014dfd0 0x7f3ed01f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3eced9d640 1 -- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed014dfd0 msgr2=0x7f3ed01f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3eced9d640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed014dfd0 0x7f3ed01f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3eced9d640 1 -- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ed014f2b0 msgr2=0x7f3ed014f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3eced9d640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ed014f2b0 0x7f3ed014f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3eced9d640 1 -- 172.21.15.148:0/2745433741 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed01f2090 con 0x7f3ebc006300 2023-10-30T13:14:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ecf59e640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ed014f2b0 0x7f3ed014f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3eced9d640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ed01f2520 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00072c0 tx=0x7f3ec0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ec7fff640 1 -- 172.21.15.148:0/2745433741 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec000f960 con 0x7f3ebc006300 2023-10-30T13:14:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ec7fff640 1 -- 172.21.15.148:0/2745433741 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec000fb00 con 0x7f3ebc006300 2023-10-30T13:14:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed0153fb0 con 0x7f3ebc006300 2023-10-30T13:14:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.771+0000 7f3ec7fff640 1 -- 172.21.15.148:0/2745433741 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec000feb0 con 0x7f3ebc006300 2023-10-30T13:14:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.775+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed014e8e0 con 0x7f3ebc006300 2023-10-30T13:14:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.775+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed004e170 con 0x7f3ebc006300 2023-10-30T13:14:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.775+0000 7f3ec7fff640 1 -- 172.21.15.148:0/2745433741 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3ec001d460 con 0x7f3ebc006300 2023-10-30T13:14:34.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.775+0000 7f3ec7fff640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3eb00664a0 0x7f3eb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:34.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.775+0000 7f3ec7fff640 1 -- 172.21.15.148:0/2745433741 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f3ec0084200 con 0x7f3ebc006300 2023-10-30T13:14:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.775+0000 7f3ec659c640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3eb00664a0 0x7f3eb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.775+0000 7f3ec659c640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3eb00664a0 0x7f3eb0068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3eb80b0000 tx=0x7f3eb80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.779+0000 7f3ec7fff640 1 -- 172.21.15.148:0/2745433741 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3ec0051580 con 0x7f3ebc006300 2023-10-30T13:14:34.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.959+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ed00c7e20 con 0x7f3ebc006300 2023-10-30T13:14:34.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.959+0000 7f3ec7fff640 1 -- 172.21.15.148:0/2745433741 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5750 (secure 0 0 0) 0x7f3ec0054ba0 con 0x7f3ebc006300 2023-10-30T13:14:34.965 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:34.966 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:30.381219+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:reconnect","state_seq":63,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:34.966 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 16 2023-10-30T13:14:34.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3eb00664a0 msgr2=0x7f3eb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:34.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3eb00664a0 0x7f3eb0068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3eb80b0000 tx=0x7f3eb80aa070 comp rx=0 tx=0).stop 2023-10-30T13:14:34.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 msgr2=0x7f3ed01f2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:34.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ed01f2520 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00072c0 tx=0x7f3ec0007780 comp rx=0 tx=0).stop 2023-10-30T13:14:34.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 shutdown_connections 2023-10-30T13:14:34.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3eb00664a0 0x7f3eb0068960 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3eb80b0000 tx=0x7f3eb80aa070 comp rx=0 tx=0).stop 2023-10-30T13:14:34.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ed014f2b0 0x7f3ed014f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:34.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ed014dfd0 0x7f3ed01f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:34.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 --2- 172.21.15.148:0/2745433741 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ebc006300 0x7f3ed01f2520 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:34.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 >> 172.21.15.148:0/2745433741 conn(0x7f3ed00c7910 msgr2=0x7f3ed00c5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:34.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 shutdown_connections 2023-10-30T13:14:34.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:34.963+0000 7f3ed510f640 1 -- 172.21.15.148:0/2745433741 wait complete. 2023-10-30T13:14:34.987 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:14:36.988 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:37.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f9345468640 1 Processor -- start 2023-10-30T13:14:37.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f9345468640 1 -- start start 2023-10-30T13:14:37.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f9345468640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9340069e90 0x7f934015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:37.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f9345468640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f934006cb80 0x7f934006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:37.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f9345468640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934006d530 con 0x7f934006a270 2023-10-30T13:14:37.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f9345468640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93400c8570 con 0x7f9340069e90 2023-10-30T13:14:37.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f9345468640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93400c6c10 con 0x7f934006cb80 2023-10-30T13:14:37.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f933effd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9340069e90 0x7f934015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:37.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f933effd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9340069e90 0x7f934015a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47004/0 (socket says 172.21.15.148:47004) 2023-10-30T13:14:37.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f933effd640 1 -- 172.21.15.148:0/466270019 learned_addr learned my addr 172.21.15.148:0/466270019 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:37.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f933f7fe640 1 --2- 172.21.15.148:0/466270019 >> v2:172.21.15.150:3300/0 conn(0x7f934006cb80 0x7f934006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:37.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f933effd640 1 -- 172.21.15.148:0/466270019 >> v2:172.21.15.150:3300/0 conn(0x7f934006cb80 msgr2=0x7f934006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:37.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f933effd640 1 --2- 172.21.15.148:0/466270019 >> v2:172.21.15.150:3300/0 conn(0x7f934006cb80 0x7f934006cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f933effd640 1 -- 172.21.15.148:0/466270019 >> v1:172.21.15.150:6789/0 conn(0x7f934006a270 legacy=0x7f93400c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:37.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.126+0000 7f933effd640 1 -- 172.21.15.148:0/466270019 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93400c6ef0 con 0x7f9340069e90 2023-10-30T13:14:37.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f933effd640 1 --2- 172.21.15.148:0/466270019 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9340069e90 0x7f934015a410 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f93400ca5c0 tx=0x7f933001c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d20400b0c56a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:37.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f933e7fc640 1 -- 172.21.15.148:0/466270019 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93300ac410 con 0x7f9340069e90 2023-10-30T13:14:37.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f933e7fc640 1 -- 172.21.15.148:0/466270019 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93300ac570 con 0x7f9340069e90 2023-10-30T13:14:37.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f933e7fc640 1 -- 172.21.15.148:0/466270019 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93300ac900 con 0x7f9340069e90 2023-10-30T13:14:37.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f9345468640 1 -- 172.21.15.148:0/466270019 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9340069e90 msgr2=0x7f934015a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:37.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f9345468640 1 --2- 172.21.15.148:0/466270019 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9340069e90 0x7f934015a410 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f93400ca5c0 tx=0x7f933001c4c0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f9345468640 1 -- 172.21.15.148:0/466270019 shutdown_connections 2023-10-30T13:14:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f9345468640 1 --2- 172.21.15.148:0/466270019 >> v2:172.21.15.150:3300/0 conn(0x7f934006cb80 0x7f934006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f9345468640 1 --2- 172.21.15.148:0/466270019 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9340069e90 0x7f934015a410 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f9345468640 1 -- 172.21.15.148:0/466270019 >> 172.21.15.148:0/466270019 conn(0x7f934006dfd0 msgr2=0x7f934006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:37.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f9345468640 1 -- 172.21.15.148:0/466270019 shutdown_connections 2023-10-30T13:14:37.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f9345468640 1 -- 172.21.15.148:0/466270019 wait complete. 2023-10-30T13:14:37.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.130+0000 7f9345468640 1 Processor -- start 2023-10-30T13:14:37.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 -- start start 2023-10-30T13:14:37.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9340069e90 0x7f93401f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:37.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f934006a270 0x7f934014e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:37.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934006cb80 0x7f934014f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:37.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93400c4180 con 0x7f934006a270 2023-10-30T13:14:37.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93400cb6a0 con 0x7f934006cb80 2023-10-30T13:14:37.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93401478b0 con 0x7f9340069e90 2023-10-30T13:14:37.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f93367fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f934006a270 0x7f934014e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:37.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f93367fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f934006a270 0x7f934014e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47014/0 (socket says 172.21.15.148:47014) 2023-10-30T13:14:37.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f93367fc640 1 -- 172.21.15.148:0/2916078268 learned_addr learned my addr 172.21.15.148:0/2916078268 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:37.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f933effd640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9340069e90 0x7f93401f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:37.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f933f7fe640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934006cb80 0x7f934014f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:37.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f93367fc640 1 -- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9340069e90 msgr2=0x7f93401f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:37.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f93367fc640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9340069e90 0x7f93401f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f93367fc640 1 -- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934006cb80 msgr2=0x7f934014f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:37.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f93367fc640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934006cb80 0x7f934014f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f93367fc640 1 -- 172.21.15.148:0/2916078268 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93300094d0 con 0x7f934006a270 2023-10-30T13:14:37.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f93367fc640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f934006a270 0x7f934014e4b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f932800c7f0 tx=0x7f932800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:37.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9337fff640 1 -- 172.21.15.148:0/2916078268 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932800aa50 con 0x7f934006a270 2023-10-30T13:14:37.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9337fff640 1 -- 172.21.15.148:0/2916078268 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932800abb0 con 0x7f934006a270 2023-10-30T13:14:37.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93400cb8f0 con 0x7f934006a270 2023-10-30T13:14:37.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9337fff640 1 -- 172.21.15.148:0/2916078268 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328012420 con 0x7f934006a270 2023-10-30T13:14:37.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93401f1c70 con 0x7f934006a270 2023-10-30T13:14:37.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.134+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934004e170 con 0x7f934006a270 2023-10-30T13:14:37.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.138+0000 7f9337fff640 1 -- 172.21.15.148:0/2916078268 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9328012580 con 0x7f934006a270 2023-10-30T13:14:37.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.138+0000 7f9337fff640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93200664d0 0x7f9320068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:37.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.138+0000 7f9337fff640 1 -- 172.21.15.148:0/2916078268 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f9328084330 con 0x7f934006a270 2023-10-30T13:14:37.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.138+0000 7f933effd640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93200664d0 0x7f9320068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:37.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.138+0000 7f933effd640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93200664d0 0x7f9320068990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f93300ae000 tx=0x7f9330008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:37.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.142+0000 7f9337fff640 1 -- 172.21.15.148:0/2916078268 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9328051590 con 0x7f934006a270 2023-10-30T13:14:37.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.322+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9340070c30 con 0x7f934006a270 2023-10-30T13:14:37.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.322+0000 7f9337fff640 1 -- 172.21.15.148:0/2916078268 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f9328016090 con 0x7f934006a270 2023-10-30T13:14:37.329 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:37.329 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:35.416986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:rejoin","state_seq":65,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:37.330 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 17 2023-10-30T13:14:37.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93200664d0 msgr2=0x7f9320068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:37.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93200664d0 0x7f9320068990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f93300ae000 tx=0x7f9330008040 comp rx=0 tx=0).stop 2023-10-30T13:14:37.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f934006a270 msgr2=0x7f934014e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:37.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f934006a270 0x7f934014e4b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f932800c7f0 tx=0x7f932800ccb0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 shutdown_connections 2023-10-30T13:14:37.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93200664d0 0x7f9320068990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f934006a270 0x7f934014e4b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934006cb80 0x7f934014f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 --2- 172.21.15.148:0/2916078268 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9340069e90 0x7f93401f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:37.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 >> 172.21.15.148:0/2916078268 conn(0x7f934006dfd0 msgr2=0x7f934014c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:37.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 shutdown_connections 2023-10-30T13:14:37.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:37.326+0000 7f9345468640 1 -- 172.21.15.148:0/2916078268 wait complete. 2023-10-30T13:14:37.350 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:14:39.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.494+0000 7f9783701640 1 Processor -- start 2023-10-30T13:14:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.494+0000 7f9783701640 1 -- start start 2023-10-30T13:14:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9783701640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f977c153bf0 0x7f977c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9783701640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c0c6fe0 0x7f977c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9783701640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0700b0 con 0x7f977c0c79e0 2023-10-30T13:14:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9783701640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f977c0c6810 con 0x7f977c153bf0 2023-10-30T13:14:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9783701640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c159900 con 0x7f977c0c6fe0 2023-10-30T13:14:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9780c9c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c0c6fe0 0x7f977c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9780c9c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c0c6fe0 0x7f977c0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:43334/0 (socket says 172.21.15.148:43334) 2023-10-30T13:14:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9780c9c640 1 -- 172.21.15.148:0/4180168853 learned_addr learned my addr 172.21.15.148:0/4180168853 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f978149d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f977c153bf0 0x7f977c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9780c9c640 1 -- 172.21.15.148:0/4180168853 >> v2:172.21.15.150:3300/0 conn(0x7f977c153bf0 msgr2=0x7f977c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9780c9c640 1 --2- 172.21.15.148:0/4180168853 >> v2:172.21.15.150:3300/0 conn(0x7f977c153bf0 0x7f977c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9780c9c640 1 -- 172.21.15.148:0/4180168853 >> v1:172.21.15.150:6789/0 conn(0x7f977c0c79e0 legacy=0x7f977c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.498+0000 7f9780c9c640 1 -- 172.21.15.148:0/4180168853 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c0c6530 con 0x7f977c0c6fe0 2023-10-30T13:14:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9780c9c640 1 --2- 172.21.15.148:0/4180168853 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c0c6fe0 0x7f977c0c74a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f976c009870 tx=0x7f976c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8c8fe1e24268f96 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9773fff640 1 -- 172.21.15.148:0/4180168853 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c0a8a30 con 0x7f977c0c6fe0 2023-10-30T13:14:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9773fff640 1 -- 172.21.15.148:0/4180168853 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c0a8b90 con 0x7f977c0c6fe0 2023-10-30T13:14:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9773fff640 1 -- 172.21.15.148:0/4180168853 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c0b0440 con 0x7f977c0c6fe0 2023-10-30T13:14:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 -- 172.21.15.148:0/4180168853 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c0c6fe0 msgr2=0x7f977c0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 --2- 172.21.15.148:0/4180168853 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c0c6fe0 0x7f977c0c74a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f976c009870 tx=0x7f976c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:14:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 -- 172.21.15.148:0/4180168853 shutdown_connections 2023-10-30T13:14:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 --2- 172.21.15.148:0/4180168853 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c0c6fe0 0x7f977c0c74a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 --2- 172.21.15.148:0/4180168853 >> v2:172.21.15.150:3300/0 conn(0x7f977c153bf0 0x7f977c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 -- 172.21.15.148:0/4180168853 >> 172.21.15.148:0/4180168853 conn(0x7f977c06c6e0 msgr2=0x7f977c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 -- 172.21.15.148:0/4180168853 shutdown_connections 2023-10-30T13:14:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 -- 172.21.15.148:0/4180168853 wait complete. 2023-10-30T13:14:39.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 Processor -- start 2023-10-30T13:14:39.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.502+0000 7f9783701640 1 -- start start 2023-10-30T13:14:39.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9783701640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c0c6fe0 0x7f977c1f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:39.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9783701640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f977c0c79e0 0x7f977c0c3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:39.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9783701640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c153bf0 0x7f977c0ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:39.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9783701640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0c9bd0 con 0x7f977c0c6fe0 2023-10-30T13:14:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9783701640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0c4910 con 0x7f977c0c79e0 2023-10-30T13:14:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9783701640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0c4bd0 con 0x7f977c153bf0 2023-10-30T13:14:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f978149d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c0c6fe0 0x7f977c1f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f978149d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c0c6fe0 0x7f977c1f2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47018/0 (socket says 172.21.15.148:47018) 2023-10-30T13:14:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f978149d640 1 -- 172.21.15.148:0/1431572001 learned_addr learned my addr 172.21.15.148:0/1431572001 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9780c9c640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f977c0c79e0 0x7f977c0c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9781c9e640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c153bf0 0x7f977c0ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f978149d640 1 -- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c153bf0 msgr2=0x7f977c0ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f978149d640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c153bf0 0x7f977c0ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f978149d640 1 -- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f977c0c79e0 msgr2=0x7f977c0c3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f978149d640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f977c0c79e0 0x7f977c0c3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f978149d640 1 -- 172.21.15.148:0/1431572001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976c0094d0 con 0x7f977c0c6fe0 2023-10-30T13:14:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f978149d640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c0c6fe0 0x7f977c1f2d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f977800e870 tx=0x7f977800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9771ffb640 1 -- 172.21.15.148:0/1431572001 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9778009510 con 0x7f977c0c6fe0 2023-10-30T13:14:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9771ffb640 1 -- 172.21.15.148:0/1431572001 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977800eec0 con 0x7f977c0c6fe0 2023-10-30T13:14:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c0c4eb0 con 0x7f977c0c6fe0 2023-10-30T13:14:39.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9771ffb640 1 -- 172.21.15.148:0/1431572001 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977800cb40 con 0x7f977c0c6fe0 2023-10-30T13:14:39.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c0c9860 con 0x7f977c0c6fe0 2023-10-30T13:14:39.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c04e170 con 0x7f977c0c6fe0 2023-10-30T13:14:39.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.506+0000 7f9771ffb640 1 -- 172.21.15.148:0/1431572001 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f977800cca0 con 0x7f977c0c6fe0 2023-10-30T13:14:39.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.510+0000 7f9771ffb640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97600664a0 0x7f9760068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:39.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.510+0000 7f9771ffb640 1 -- 172.21.15.148:0/1431572001 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f9778083fe0 con 0x7f977c0c6fe0 2023-10-30T13:14:39.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.510+0000 7f9780c9c640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97600664a0 0x7f9760068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:39.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.510+0000 7f9780c9c640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97600664a0 0x7f9760068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f976c0b2000 tx=0x7f976c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:39.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.514+0000 7f9771ffb640 1 -- 172.21.15.148:0/1431572001 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9778051430 con 0x7f977c0c6fe0 2023-10-30T13:14:39.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.694+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f977c06fe90 con 0x7f977c0c6fe0 2023-10-30T13:14:39.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.694+0000 7f9771ffb640 1 -- 172.21.15.148:0/1431572001 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f9778054a50 con 0x7f977c0c6fe0 2023-10-30T13:14:39.701 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:39.702 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:35.416986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:rejoin","state_seq":65,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:39.702 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 17 2023-10-30T13:14:39.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97600664a0 msgr2=0x7f9760068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:39.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97600664a0 0x7f9760068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f976c0b2000 tx=0x7f976c009040 comp rx=0 tx=0).stop 2023-10-30T13:14:39.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c0c6fe0 msgr2=0x7f977c1f2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:39.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c0c6fe0 0x7f977c1f2d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f977800e870 tx=0x7f977800ec70 comp rx=0 tx=0).stop 2023-10-30T13:14:39.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 shutdown_connections 2023-10-30T13:14:39.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97600664a0 0x7f9760068960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:39.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f977c153bf0 0x7f977c0ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:39.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f977c0c79e0 0x7f977c0c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:39.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 --2- 172.21.15.148:0/1431572001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f977c0c6fe0 0x7f977c1f2d80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:39.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 >> 172.21.15.148:0/1431572001 conn(0x7f977c06c6e0 msgr2=0x7f977c070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:39.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 shutdown_connections 2023-10-30T13:14:39.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:39.698+0000 7f9783701640 1 -- 172.21.15.148:0/1431572001 wait complete. 2023-10-30T13:14:39.722 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:14:40.435 INFO:tasks.workunit.client.0.smithi148.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-10-30T13:14:41.722 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:41.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc71f71640 1 Processor -- start 2023-10-30T13:14:41.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc71f71640 1 -- start start 2023-10-30T13:14:41.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc71f71640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc6c1538f0 0x7efc6c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:41.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc71f71640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c15cd50 0x7efc6c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:41.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc71f71640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc6c15d300 con 0x7efc6c0cefa0 2023-10-30T13:14:41.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc71f71640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc6c15d5e0 con 0x7efc6c1538f0 2023-10-30T13:14:41.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc71f71640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c15d8c0 con 0x7efc6c15cd50 2023-10-30T13:14:41.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc71770640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efc6c0cefa0 0x7efc6c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53160/0 (socket says 172.21.15.148:53160) 2023-10-30T13:14:41.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc71770640 1 -- 172.21.15.148:0/3089228028 learned_addr learned my addr 172.21.15.148:0/3089228028 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:41.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc70f6f640 1 --2- 172.21.15.148:0/3089228028 >> v2:172.21.15.150:3300/0 conn(0x7efc6c1538f0 0x7efc6c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:41.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc63fff640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c15cd50 0x7efc6c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:41.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.874+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982880147 0 0) 0x7efc6c15d300 con 0x7efc6c0cefa0 2023-10-30T13:14:41.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc54003530 con 0x7efc6c0cefa0 2023-10-30T13:14:41.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1371057097 0 0) 0x7efc54003530 con 0x7efc6c0cefa0 2023-10-30T13:14:41.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c15cd50 msgr2=0x7efc6c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:41.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c15cd50 0x7efc6c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:41.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 >> v2:172.21.15.150:3300/0 conn(0x7efc6c1538f0 msgr2=0x7efc6c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:41.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 --2- 172.21.15.148:0/3089228028 >> v2:172.21.15.150:3300/0 conn(0x7efc6c1538f0 0x7efc6c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:41.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c15dba0 con 0x7efc6c0cefa0 2023-10-30T13:14:41.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7efc64001a00 con 0x7efc6c0cefa0 2023-10-30T13:14:41.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 >> v1:172.21.15.148:6789/0 conn(0x7efc6c0cefa0 legacy=0x7efc6c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc54003980 0x7efc54003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c15cd50 0x7efc540047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc54005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c15d5e0 con 0x7efc54004d10 2023-10-30T13:14:41.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c15d8c0 con 0x7efc54003980 2023-10-30T13:14:41.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5400b990 con 0x7efc6c15cd50 2023-10-30T13:14:41.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc71770640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc54005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:41.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc70f6f640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc54003980 0x7efc54003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:41.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc63fff640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c15cd50 0x7efc540047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:41.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc71770640 1 -- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c15cd50 msgr2=0x7efc540047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:41.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc71770640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c15cd50 0x7efc540047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:41.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc71770640 1 -- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc54003980 msgr2=0x7efc54003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:41.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc71770640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc54003980 0x7efc54003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:41.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.878+0000 7efc71770640 1 -- 172.21.15.148:0/3089228028 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc54003530 con 0x7efc54004d10 2023-10-30T13:14:41.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71770640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc54005190 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efc6400a860 tx=0x7efc6401d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07f464e45c7927d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:41.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc640ad630 con 0x7efc54004d10 2023-10-30T13:14:41.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc640a8070 con 0x7efc54004d10 2023-10-30T13:14:41.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc637fe640 1 -- 172.21.15.148:0/3089228028 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc640ad9f0 con 0x7efc54004d10 2023-10-30T13:14:41.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 -- 172.21.15.148:0/3089228028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 msgr2=0x7efc54005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:41.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 --2- 172.21.15.148:0/3089228028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc54005190 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efc6400a860 tx=0x7efc6401d360 comp rx=0 tx=0).stop 2023-10-30T13:14:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71770640 1 -- 172.21.15.148:0/3089228028 reap_dead start 2023-10-30T13:14:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 -- 172.21.15.148:0/3089228028 shutdown_connections 2023-10-30T13:14:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 -- 172.21.15.148:0/3089228028 >> 172.21.15.148:0/3089228028 conn(0x7efc6c06a640 msgr2=0x7efc6c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 -- 172.21.15.148:0/3089228028 shutdown_connections 2023-10-30T13:14:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 -- 172.21.15.148:0/3089228028 wait complete. 2023-10-30T13:14:41.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 Processor -- start 2023-10-30T13:14:41.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 -- start start 2023-10-30T13:14:41.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc6c1f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:41.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc6c15a830 0x7efc6c14f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:41.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c1f01f0 0x7efc6c15c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:41.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c15bbe0 con 0x7efc54004d10 2023-10-30T13:14:41.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c15bf40 con 0x7efc6c15a830 2023-10-30T13:14:41.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.882+0000 7efc71f71640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c15ad00 con 0x7efc6c1f01f0 2023-10-30T13:14:41.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc70f6f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc6c1f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:41.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc70f6f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc6c1f2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60342/0 (socket says 172.21.15.148:60342) 2023-10-30T13:14:41.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc70f6f640 1 -- 172.21.15.148:0/732682720 learned_addr learned my addr 172.21.15.148:0/732682720 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:41.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc63fff640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc6c15a830 0x7efc6c14f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:41.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc71770640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c1f01f0 0x7efc6c15c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:41.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc70f6f640 1 -- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c1f01f0 msgr2=0x7efc6c15c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:41.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc70f6f640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c1f01f0 0x7efc6c15c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:41.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc70f6f640 1 -- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc6c15a830 msgr2=0x7efc6c14f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc70f6f640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc6c15a830 0x7efc6c14f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc70f6f640 1 -- 172.21.15.148:0/732682720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c1f06a0 con 0x7efc54004d10 2023-10-30T13:14:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc70f6f640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc6c1f2e00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efc680115e0 tx=0x7efc68011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc617fa640 1 -- 172.21.15.148:0/732682720 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6800f8f0 con 0x7efc54004d10 2023-10-30T13:14:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc617fa640 1 -- 172.21.15.148:0/732682720 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6800fa50 con 0x7efc54004d10 2023-10-30T13:14:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc617fa640 1 -- 172.21.15.148:0/732682720 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6800fdc0 con 0x7efc54004d10 2023-10-30T13:14:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6c14f050 con 0x7efc54004d10 2023-10-30T13:14:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6c1f2320 con 0x7efc54004d10 2023-10-30T13:14:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6c04e170 con 0x7efc54004d10 2023-10-30T13:14:41.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc617fa640 1 -- 172.21.15.148:0/732682720 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efc68015460 con 0x7efc54004d10 2023-10-30T13:14:41.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc617fa640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc580664d0 0x7efc58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:41.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.886+0000 7efc617fa640 1 -- 172.21.15.148:0/732682720 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7efc68084450 con 0x7efc54004d10 2023-10-30T13:14:41.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.890+0000 7efc63fff640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc580664d0 0x7efc58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:41.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.890+0000 7efc63fff640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc580664d0 0x7efc58068990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efc6c15ac30 tx=0x7efc5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:41.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:41.890+0000 7efc617fa640 1 -- 172.21.15.148:0/732682720 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efc680517d0 con 0x7efc54004d10 2023-10-30T13:14:42.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.098+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc6c159700 con 0x7efc54004d10 2023-10-30T13:14:42.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.098+0000 7efc617fa640 1 -- 172.21.15.148:0/732682720 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5747 (secure 0 0 0) 0x7efc68054df0 con 0x7efc54004d10 2023-10-30T13:14:42.105 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:42.105 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:42.106 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 18 2023-10-30T13:14:42.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc580664d0 msgr2=0x7efc58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:42.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc580664d0 0x7efc58068990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efc6c15ac30 tx=0x7efc5c008040 comp rx=0 tx=0).stop 2023-10-30T13:14:42.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 msgr2=0x7efc6c1f2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:42.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc6c1f2e00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efc680115e0 tx=0x7efc68011aa0 comp rx=0 tx=0).stop 2023-10-30T13:14:42.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 shutdown_connections 2023-10-30T13:14:42.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc580664d0 0x7efc58068990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:42.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc6c1f01f0 0x7efc6c15c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:42.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc6c15a830 0x7efc6c14f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 --2- 172.21.15.148:0/732682720 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc54004d10 0x7efc6c1f2e00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 >> 172.21.15.148:0/732682720 conn(0x7efc6c06a640 msgr2=0x7efc6c0caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 shutdown_connections 2023-10-30T13:14:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:42.102+0000 7efc71f71640 1 -- 172.21.15.148:0/732682720 wait complete. 2023-10-30T13:14:42.127 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=2, replacing gid=4256 2023-10-30T13:14:42.127 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.c 2023-10-30T13:14:42.939 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir foo 2023-10-30T13:14:42.946 INFO:tasks.workunit.client.0.smithi148.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-10-30T13:14:43.040 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir foo/.snap/barsnap 2023-10-30T13:14:45.470 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-10-30T13:14:45.484 INFO:tasks.workunit.client.0.smithi148.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-10-30T13:14:46.108 INFO:tasks.workunit.client.0.smithi148.stdout:passed: files are identical 2023-10-30T13:14:46.109 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo passed: files are identical 2023-10-30T13:14:46.109 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir foo/.snap/barsnap 2023-10-30T13:14:46.150 INFO:tasks.workunit.client.0.smithi148.stdout:OK 2023-10-30T13:14:46.151 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo OK 2023-10-30T13:14:46.152 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:14:46.152 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:14:47.944 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-10-30T13:14:47.944 DEBUG:teuthology.orchestra.run.smithi148:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-10-30T13:14:50.128 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:14:50.128 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:14:50.129 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:14:50.133 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:14:50.133 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:50.228 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:14:50.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab6db5640 1 Processor -- start 2023-10-30T13:14:50.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab6db5640 1 -- start start 2023-10-30T13:14:50.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab6db5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffab0153750 0x7ffab00c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:50.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab6db5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab0153b30 con 0x7ffab006afa0 2023-10-30T13:14:50.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab6db5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab00b2480 con 0x7ffab015a040 2023-10-30T13:14:50.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab6db5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab006fdd0 con 0x7ffab0153750 2023-10-30T13:14:50.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab55b2640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ffab006afa0 0x7ffab0159930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53174/0 (socket says 172.21.15.148:53174) 2023-10-30T13:14:50.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab55b2640 1 -- 172.21.15.148:0/3787697510 learned_addr learned my addr 172.21.15.148:0/3787697510 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab5db3640 1 --2- 172.21.15.148:0/3787697510 >> v2:172.21.15.150:3300/0 conn(0x7ffab0153750 0x7ffab00c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab4db1640 1 -- 172.21.15.148:0/3787697510 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294328215 0 0) 0x7ffab0153b30 con 0x7ffab006afa0 2023-10-30T13:14:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab4db1640 1 -- 172.21.15.148:0/3787697510 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c003530 con 0x7ffab006afa0 2023-10-30T13:14:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab5db3640 1 -- 172.21.15.148:0/3787697510 >> v1:172.21.15.150:6789/0 conn(0x7ffab015a040 legacy=0x7ffab0163fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab5db3640 1 -- 172.21.15.148:0/3787697510 >> v1:172.21.15.148:6789/0 conn(0x7ffab006afa0 legacy=0x7ffab0159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.278+0000 7ffab5db3640 1 -- 172.21.15.148:0/3787697510 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab006cfc0 con 0x7ffab0153750 2023-10-30T13:14:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab5db3640 1 --2- 172.21.15.148:0/3787697510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab0153750 0x7ffab00c9db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0097c0 tx=0x7ffaac01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=120dc53c43e94d29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab4db1640 1 -- 172.21.15.148:0/3787697510 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0ab600 con 0x7ffab0153750 2023-10-30T13:14:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab4db1640 1 -- 172.21.15.148:0/3787697510 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaac0a7050 con 0x7ffab0153750 2023-10-30T13:14:50.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab4db1640 1 -- 172.21.15.148:0/3787697510 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac0ab960 con 0x7ffab0153750 2023-10-30T13:14:50.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab6db5640 1 -- 172.21.15.148:0/3787697510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab0153750 msgr2=0x7ffab00c9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:50.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab6db5640 1 --2- 172.21.15.148:0/3787697510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab0153750 0x7ffab00c9db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0097c0 tx=0x7ffaac01c480 comp rx=0 tx=0).stop 2023-10-30T13:14:50.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab6db5640 1 -- 172.21.15.148:0/3787697510 shutdown_connections 2023-10-30T13:14:50.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab6db5640 1 --2- 172.21.15.148:0/3787697510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab0153750 0x7ffab00c9db0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:50.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab6db5640 1 -- 172.21.15.148:0/3787697510 >> 172.21.15.148:0/3787697510 conn(0x7ffab006d560 msgr2=0x7ffab006d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:50.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab6db5640 1 -- 172.21.15.148:0/3787697510 shutdown_connections 2023-10-30T13:14:50.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.282+0000 7ffab6db5640 1 -- 172.21.15.148:0/3787697510 wait complete. 2023-10-30T13:14:50.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 Processor -- start 2023-10-30T13:14:50.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 -- start start 2023-10-30T13:14:50.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab006afa0 0x7ffab00cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:50.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab0153750 0x7ffab00af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:50.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab015a040 0x7ffab00af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:50.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab015d690 con 0x7ffab015a040 2023-10-30T13:14:50.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab015bac0 con 0x7ffab0153750 2023-10-30T13:14:50.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab015cd30 con 0x7ffab006afa0 2023-10-30T13:14:50.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab65b4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab015a040 0x7ffab00af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:50.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab5db3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab006afa0 0x7ffab00cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:50.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab55b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab0153750 0x7ffab00af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:50.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab65b4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab015a040 0x7ffab00af860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60358/0 (socket says 172.21.15.148:60358) 2023-10-30T13:14:50.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab65b4640 1 -- 172.21.15.148:0/2906857975 learned_addr learned my addr 172.21.15.148:0/2906857975 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:50.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab65b4640 1 -- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab006afa0 msgr2=0x7ffab00cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:50.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab65b4640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab006afa0 0x7ffab00cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:50.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab65b4640 1 -- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab0153750 msgr2=0x7ffab00af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:50.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab65b4640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab0153750 0x7ffab00af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:50.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab65b4640 1 -- 172.21.15.148:0/2906857975 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac009420 con 0x7ffab015a040 2023-10-30T13:14:50.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab65b4640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab015a040 0x7ffab00af860 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800c730 tx=0x7ffaa800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:50.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffaa67fc640 1 -- 172.21.15.148:0/2906857975 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8003b80 con 0x7ffab015a040 2023-10-30T13:14:50.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffaa67fc640 1 -- 172.21.15.148:0/2906857975 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa8003ce0 con 0x7ffab015a040 2023-10-30T13:14:50.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffaa67fc640 1 -- 172.21.15.148:0/2906857975 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa800aa30 con 0x7ffab015a040 2023-10-30T13:14:50.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab015cfd0 con 0x7ffab015a040 2023-10-30T13:14:50.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab015bd60 con 0x7ffab015a040 2023-10-30T13:14:50.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.286+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab006f400 con 0x7ffab015a040 2023-10-30T13:14:50.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.290+0000 7ffaa67fc640 1 -- 172.21.15.148:0/2906857975 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffaa800ab90 con 0x7ffab015a040 2023-10-30T13:14:50.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.290+0000 7ffaa67fc640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:50.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.290+0000 7ffaa67fc640 1 -- 172.21.15.148:0/2906857975 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5940+0+0 (secure 0 0 0) 0x7ffaa8083d90 con 0x7ffab015a040 2023-10-30T13:14:50.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.290+0000 7ffab5db3640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:50.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.290+0000 7ffab5db3640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0ad000 tx=0x7ffaac0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:50.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.294+0000 7ffaa67fc640 1 -- 172.21.15.148:0/2906857975 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffaa80510a0 con 0x7ffab015a040 2023-10-30T13:14:50.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.478+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffab004e170 con 0x7ffab015a040 2023-10-30T13:14:50.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffaa67fc640 1 -- 172.21.15.148:0/2906857975 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5747 (secure 0 0 0) 0x7ffaa80546c0 con 0x7ffab015a040 2023-10-30T13:14:50.487 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:50.487 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:50.488 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 18 2023-10-30T13:14:50.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 msgr2=0x7ffa94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:50.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0ad000 tx=0x7ffaac0a6000 comp rx=0 tx=0).stop 2023-10-30T13:14:50.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab015a040 msgr2=0x7ffab00af860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab015a040 0x7ffab00af860 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800c730 tx=0x7ffaa800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:14:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 shutdown_connections 2023-10-30T13:14:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab015a040 0x7ffab00af860 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab0153750 0x7ffab00af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:50.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 --2- 172.21.15.148:0/2906857975 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab006afa0 0x7ffab00cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:50.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 >> 172.21.15.148:0/2906857975 conn(0x7ffab006d560 msgr2=0x7ffab0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:50.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 shutdown_connections 2023-10-30T13:14:50.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:50.482+0000 7ffab6db5640 1 -- 172.21.15.148:0/2906857975 wait complete. 2023-10-30T13:14:50.509 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:14:50.750 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 1 2023-10-30T13:14:50.751 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 2 2023-10-30T13:14:50.751 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 3 2023-10-30T13:14:50.751 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' -e file4 ']' 2023-10-30T13:14:50.752 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/snap1 2023-10-30T13:14:50.767 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 4 2023-10-30T13:14:50.769 INFO:tasks.workunit.client.0.smithi148.stderr:++ ls 2023-10-30T13:14:50.775 INFO:tasks.workunit.client.0.smithi148.stderr:+ now='file1 2023-10-30T13:14:50.775 INFO:tasks.workunit.client.0.smithi148.stderr:file2 2023-10-30T13:14:50.776 INFO:tasks.workunit.client.0.smithi148.stderr:file3 2023-10-30T13:14:50.776 INFO:tasks.workunit.client.0.smithi148.stderr:file4' 2023-10-30T13:14:50.776 INFO:tasks.workunit.client.0.smithi148.stderr:++ ls .snap/snap1 2023-10-30T13:14:50.783 INFO:tasks.workunit.client.0.smithi148.stderr:+ then='file1 2023-10-30T13:14:50.783 INFO:tasks.workunit.client.0.smithi148.stderr:file2 2023-10-30T13:14:50.783 INFO:tasks.workunit.client.0.smithi148.stderr:file3' 2023-10-30T13:14:50.784 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir .snap/snap1 2023-10-30T13:14:50.797 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' 'file1 2023-10-30T13:14:50.797 INFO:tasks.workunit.client.0.smithi148.stderr:file2 2023-10-30T13:14:50.797 INFO:tasks.workunit.client.0.smithi148.stderr:file3 2023-10-30T13:14:50.798 INFO:tasks.workunit.client.0.smithi148.stderr:file4' = 'file1 2023-10-30T13:14:50.798 INFO:tasks.workunit.client.0.smithi148.stderr:file2 2023-10-30T13:14:50.798 INFO:tasks.workunit.client.0.smithi148.stderr:file3' ']' 2023-10-30T13:14:50.798 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 1 2023-10-30T13:14:50.806 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 2 2023-10-30T13:14:50.817 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 3 2023-10-30T13:14:50.830 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/snap1 2023-10-30T13:14:50.845 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo 4 2023-10-30T13:14:50.855 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir .snap/snap1 2023-10-30T13:14:50.867 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm file1 file2 file3 file4 2023-10-30T13:14:50.874 INFO:tasks.workunit.client.0.smithi148.stdout:OK 2023-10-30T13:14:50.875 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo OK 2023-10-30T13:14:50.875 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:14:50.876 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:14:50.901 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-10-30T13:14:50.902 DEBUG:teuthology.orchestra.run.smithi148:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-10-30T13:14:52.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:52.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffb0506b640 1 Processor -- start 2023-10-30T13:14:52.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffb0506b640 1 -- start start 2023-10-30T13:14:52.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffb0506b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffb0015d6f0 0x7ffb0015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:52.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffb0506b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb000cf160 0x7ffb000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:52.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffb0506b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb000cb800 con 0x7ffb001538a0 2023-10-30T13:14:52.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffb0506b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb000cb330 con 0x7ffb0015d6f0 2023-10-30T13:14:52.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffb0506b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb000cad70 con 0x7ffb000cf160 2023-10-30T13:14:52.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffafed9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ffb001538a0 0x7ffb000ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39538/0 (socket says 172.21.15.148:39538) 2023-10-30T13:14:52.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffafed9d640 1 -- 172.21.15.148:0/331113025 learned_addr learned my addr 172.21.15.148:0/331113025 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:14:52.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffaff59e640 1 --2- 172.21.15.148:0/331113025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb000cf160 0x7ffb000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:52.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffaf659c640 1 --2- 172.21.15.148:0/331113025 >> v2:172.21.15.150:3300/0 conn(0x7ffb0015d6f0 0x7ffb0015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:52.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffafe59c640 1 -- 172.21.15.148:0/331113025 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033295817 0 0) 0x7ffb000cb800 con 0x7ffb001538a0 2023-10-30T13:14:52.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffafe59c640 1 -- 172.21.15.148:0/331113025 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaec003530 con 0x7ffb001538a0 2023-10-30T13:14:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffafe59c640 1 -- 172.21.15.148:0/331113025 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1562136768 0 0) 0x7ffaec003530 con 0x7ffb001538a0 2023-10-30T13:14:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffaff59e640 1 -- 172.21.15.148:0/331113025 >> v2:172.21.15.150:3300/0 conn(0x7ffb0015d6f0 msgr2=0x7ffb0015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffaff59e640 1 --2- 172.21.15.148:0/331113025 >> v2:172.21.15.150:3300/0 conn(0x7ffb0015d6f0 0x7ffb0015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffaff59e640 1 -- 172.21.15.148:0/331113025 >> v1:172.21.15.148:6789/0 conn(0x7ffb001538a0 legacy=0x7ffb000ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.666+0000 7ffaff59e640 1 -- 172.21.15.148:0/331113025 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb000cb050 con 0x7ffb000cf160 2023-10-30T13:14:52.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffaff59e640 1 --2- 172.21.15.148:0/331113025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb000cf160 0x7ffb000cf590 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009d90 tx=0x7ffaf801f060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a561afeecb509d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:52.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffafe59c640 1 -- 172.21.15.148:0/331113025 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0001680 con 0x7ffb000cf160 2023-10-30T13:14:52.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffafe59c640 1 -- 172.21.15.148:0/331113025 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf80a8bb0 con 0x7ffb000cf160 2023-10-30T13:14:52.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffafe59c640 1 -- 172.21.15.148:0/331113025 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80b0440 con 0x7ffb000cf160 2023-10-30T13:14:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffb0506b640 1 -- 172.21.15.148:0/331113025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb000cf160 msgr2=0x7ffb000cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffb0506b640 1 --2- 172.21.15.148:0/331113025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb000cf160 0x7ffb000cf590 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009d90 tx=0x7ffaf801f060 comp rx=0 tx=0).stop 2023-10-30T13:14:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffb0506b640 1 -- 172.21.15.148:0/331113025 shutdown_connections 2023-10-30T13:14:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffb0506b640 1 --2- 172.21.15.148:0/331113025 >> v2:172.21.15.150:3300/0 conn(0x7ffb0015d6f0 0x7ffb0015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffb0506b640 1 --2- 172.21.15.148:0/331113025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb000cf160 0x7ffb000cf590 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:52.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffb0506b640 1 -- 172.21.15.148:0/331113025 >> 172.21.15.148:0/331113025 conn(0x7ffb0006bfe0 msgr2=0x7ffb0006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:52.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffb0506b640 1 -- 172.21.15.148:0/331113025 shutdown_connections 2023-10-30T13:14:52.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.670+0000 7ffb0506b640 1 -- 172.21.15.148:0/331113025 wait complete. 2023-10-30T13:14:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 Processor -- start 2023-10-30T13:14:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 -- start start 2023-10-30T13:14:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb000cf160 0x7ffb001e5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb001538a0 0x7ffb00157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0015d6f0 0x7ffb001f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb001e2e70 con 0x7ffb000cf160 2023-10-30T13:14:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb00158d20 con 0x7ffb0015d6f0 2023-10-30T13:14:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb00158a80 con 0x7ffb001538a0 2023-10-30T13:14:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffafed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb000cf160 0x7ffb001e5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffafed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb000cf160 0x7ffb001e5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53792/0 (socket says 172.21.15.148:53792) 2023-10-30T13:14:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffafed9d640 1 -- 172.21.15.148:0/247450052 learned_addr learned my addr 172.21.15.148:0/247450052 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:14:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffaff59e640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0015d6f0 0x7ffb001f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffaf659c640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb001538a0 0x7ffb00157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffafed9d640 1 -- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb001538a0 msgr2=0x7ffb00157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffafed9d640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb001538a0 0x7ffb00157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffafed9d640 1 -- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0015d6f0 msgr2=0x7ffb001f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffafed9d640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0015d6f0 0x7ffb001f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffafed9d640 1 -- 172.21.15.148:0/247450052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf8009a70 con 0x7ffb000cf160 2023-10-30T13:14:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffafed9d640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb000cf160 0x7ffb001e5c10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000c7b0 tx=0x7ffaf000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffaf7fff640 1 -- 172.21.15.148:0/247450052 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0005cf0 con 0x7ffb000cf160 2023-10-30T13:14:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffaf7fff640 1 -- 172.21.15.148:0/247450052 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf0005e50 con 0x7ffb000cf160 2023-10-30T13:14:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffaf7fff640 1 -- 172.21.15.148:0/247450052 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf00072b0 con 0x7ffb000cf160 2023-10-30T13:14:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb001592b0 con 0x7ffb000cf160 2023-10-30T13:14:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.674+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb001efcd0 con 0x7ffb000cf160 2023-10-30T13:14:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.678+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb000b2a50 con 0x7ffb000cf160 2023-10-30T13:14:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.678+0000 7ffaf7fff640 1 -- 172.21.15.148:0/247450052 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffaf0007410 con 0x7ffb000cf160 2023-10-30T13:14:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.678+0000 7ffaf7fff640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffae00664d0 0x7ffae0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.678+0000 7ffaf7fff640 1 -- 172.21.15.148:0/247450052 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5996+0+0 (secure 0 0 0) 0x7ffaf0083d50 con 0x7ffb000cf160 2023-10-30T13:14:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.678+0000 7ffaf659c640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffae00664d0 0x7ffae0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.678+0000 7ffaf659c640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffae00664d0 0x7ffae0068990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffb0014dde0 tx=0x7ffae8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.682+0000 7ffaf7fff640 1 -- 172.21.15.148:0/247450052 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffaf00510a0 con 0x7ffb000cf160 2023-10-30T13:14:52.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb00159b40 con 0x7ffb000cf160 2023-10-30T13:14:52.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffaf7fff640 1 -- 172.21.15.148:0/247450052 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7ffaf000ce60 con 0x7ffb000cf160 2023-10-30T13:14:52.872 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:14:52.873 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:52.873 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:14:52.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffae00664d0 msgr2=0x7ffae0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:52.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffae00664d0 0x7ffae0068990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffb0014dde0 tx=0x7ffae8008040 comp rx=0 tx=0).stop 2023-10-30T13:14:52.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb000cf160 msgr2=0x7ffb001e5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:52.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb000cf160 0x7ffb001e5c10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000c7b0 tx=0x7ffaf000cc70 comp rx=0 tx=0).stop 2023-10-30T13:14:52.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 shutdown_connections 2023-10-30T13:14:52.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffae00664d0 0x7ffae0068990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:52.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0015d6f0 0x7ffb001f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:52.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb001538a0 0x7ffb00157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:52.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 --2- 172.21.15.148:0/247450052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb000cf160 0x7ffb001e5c10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:52.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 >> 172.21.15.148:0/247450052 conn(0x7ffb0006bfe0 msgr2=0x7ffb0015b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:52.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 shutdown_connections 2023-10-30T13:14:52.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:14:52.866+0000 7ffb0506b640 1 -- 172.21.15.148:0/247450052 wait complete. 2023-10-30T13:14:52.893 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:14:52.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before thrashing 2023-10-30T13:14:53.535 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:14:53.539 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:14:55.752 INFO:tasks.workunit.client.0.smithi148.stdout:Create dir 100 to 199 ... 2023-10-30T13:14:55.755 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 100 2023-10-30T13:14:55.761 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 101 2023-10-30T13:14:55.771 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 102 2023-10-30T13:14:55.781 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 103 2023-10-30T13:14:55.791 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 104 2023-10-30T13:14:55.836 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 105 2023-10-30T13:14:56.405 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 106 2023-10-30T13:14:56.416 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 107 2023-10-30T13:14:56.833 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 108 2023-10-30T13:14:56.843 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 109 2023-10-30T13:14:56.854 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 110 2023-10-30T13:14:56.864 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 111 2023-10-30T13:14:56.874 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 112 2023-10-30T13:14:57.497 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 113 2023-10-30T13:14:57.507 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 114 2023-10-30T13:14:57.517 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 115 2023-10-30T13:14:57.527 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 116 2023-10-30T13:14:57.539 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 117 2023-10-30T13:14:57.549 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 118 2023-10-30T13:14:57.563 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 119 2023-10-30T13:14:57.573 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 120 2023-10-30T13:14:57.583 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 121 2023-10-30T13:14:57.593 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 122 2023-10-30T13:14:57.603 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 123 2023-10-30T13:14:57.613 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 124 2023-10-30T13:14:57.623 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 125 2023-10-30T13:14:57.633 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 126 2023-10-30T13:14:57.642 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 127 2023-10-30T13:14:57.652 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 128 2023-10-30T13:14:57.662 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 129 2023-10-30T13:14:57.671 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 130 2023-10-30T13:14:57.681 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 131 2023-10-30T13:14:57.691 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 132 2023-10-30T13:14:57.701 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 133 2023-10-30T13:14:57.710 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 134 2023-10-30T13:14:57.720 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 135 2023-10-30T13:14:57.729 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 136 2023-10-30T13:14:57.738 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 137 2023-10-30T13:14:57.747 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 138 2023-10-30T13:14:57.756 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 139 2023-10-30T13:14:57.820 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 140 2023-10-30T13:14:58.468 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 141 2023-10-30T13:14:58.478 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 142 2023-10-30T13:14:58.812 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 143 2023-10-30T13:14:59.264 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 144 2023-10-30T13:14:59.276 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 145 2023-10-30T13:14:59.485 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 146 2023-10-30T13:14:59.500 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 147 2023-10-30T13:14:59.511 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 148 2023-10-30T13:14:59.522 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 149 2023-10-30T13:14:59.533 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 150 2023-10-30T13:14:59.543 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 151 2023-10-30T13:14:59.553 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 152 2023-10-30T13:14:59.563 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 153 2023-10-30T13:14:59.573 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 154 2023-10-30T13:14:59.584 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 155 2023-10-30T13:14:59.840 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 156 2023-10-30T13:14:59.849 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 157 2023-10-30T13:15:00.698 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 158 2023-10-30T13:15:00.894 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:01.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.030+0000 7fecf7f94640 1 Processor -- start 2023-10-30T13:15:01.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf7f94640 1 -- start start 2023-10-30T13:15:01.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf7f94640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fecf0069e90 0x7fecf015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf7f94640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fecf006cb80 0x7fecf006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf7f94640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf006d530 con 0x7fecf006a270 2023-10-30T13:15:01.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf7f94640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf00c8570 con 0x7fecf006cb80 2023-10-30T13:15:01.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf7f94640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf00c6c10 con 0x7fecf0069e90 2023-10-30T13:15:01.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf6531640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fecf006cb80 0x7fecf006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf552f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fecf006a270 0x7fecf00c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39544/0 (socket says 172.21.15.148:39544) 2023-10-30T13:15:01.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf552f640 1 -- 172.21.15.148:0/1580403155 learned_addr learned my addr 172.21.15.148:0/1580403155 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:01.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf5d30640 1 --2- 172.21.15.148:0/1580403155 >> v2:172.21.15.150:3300/0 conn(0x7fecf0069e90 0x7fecf015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf6531640 1 -- 172.21.15.148:0/1580403155 >> v2:172.21.15.150:3300/0 conn(0x7fecf0069e90 msgr2=0x7fecf015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:01.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf6531640 1 --2- 172.21.15.148:0/1580403155 >> v2:172.21.15.150:3300/0 conn(0x7fecf0069e90 0x7fecf015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf6531640 1 -- 172.21.15.148:0/1580403155 >> v1:172.21.15.148:6789/0 conn(0x7fecf006a270 legacy=0x7fecf00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf6531640 1 -- 172.21.15.148:0/1580403155 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf00c6ef0 con 0x7fecf006cb80 2023-10-30T13:15:01.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf5d30640 1 --2- 172.21.15.148:0/1580403155 >> v2:172.21.15.150:3300/0 conn(0x7fecf0069e90 0x7fecf015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:01.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.034+0000 7fecf4d2e640 1 -- 172.21.15.148:0/1580403155 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480421788 0 0) 0x7fecf006d530 con 0x7fecf006a270 2023-10-30T13:15:01.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf6531640 1 --2- 172.21.15.148:0/1580403155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 0x7fecf006cff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fecf006ed20 tx=0x7fece801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce665fa80cc7f75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:01.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf4d2e640 1 -- 172.21.15.148:0/1580403155 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece80a58c0 con 0x7fecf006cb80 2023-10-30T13:15:01.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf4d2e640 1 -- 172.21.15.148:0/1580403155 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece80a5a20 con 0x7fecf006cb80 2023-10-30T13:15:01.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf4d2e640 1 -- 172.21.15.148:0/1580403155 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece80a5db0 con 0x7fecf006cb80 2023-10-30T13:15:01.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 -- 172.21.15.148:0/1580403155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 msgr2=0x7fecf006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 --2- 172.21.15.148:0/1580403155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 0x7fecf006cff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fecf006ed20 tx=0x7fece801c910 comp rx=0 tx=0).stop 2023-10-30T13:15:01.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 -- 172.21.15.148:0/1580403155 shutdown_connections 2023-10-30T13:15:01.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 --2- 172.21.15.148:0/1580403155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 0x7fecf006cff0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 --2- 172.21.15.148:0/1580403155 >> v2:172.21.15.150:3300/0 conn(0x7fecf0069e90 0x7fecf015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 -- 172.21.15.148:0/1580403155 >> 172.21.15.148:0/1580403155 conn(0x7fecf006dfd0 msgr2=0x7fecf006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:01.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 -- 172.21.15.148:0/1580403155 shutdown_connections 2023-10-30T13:15:01.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 -- 172.21.15.148:0/1580403155 wait complete. 2023-10-30T13:15:01.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 Processor -- start 2023-10-30T13:15:01.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.038+0000 7fecf7f94640 1 -- start start 2023-10-30T13:15:01.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf7f94640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf0069e90 0x7fecf0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf7f94640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecf006a270 0x7fecf01592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf7f94640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 0x7fecf014d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf7f94640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf0146400 con 0x7fecf006cb80 2023-10-30T13:15:01.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf7f94640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf01f5ce0 con 0x7fecf0069e90 2023-10-30T13:15:01.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf7f94640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf01f5a00 con 0x7fecf006a270 2023-10-30T13:15:01.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf6531640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 0x7fecf014d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf6531640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 0x7fecf014d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53806/0 (socket says 172.21.15.148:53806) 2023-10-30T13:15:01.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf6531640 1 -- 172.21.15.148:0/2767085422 learned_addr learned my addr 172.21.15.148:0/2767085422 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:01.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf5d30640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf0069e90 0x7fecf0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf552f640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecf006a270 0x7fecf01592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf6531640 1 -- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecf006a270 msgr2=0x7fecf01592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf6531640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecf006a270 0x7fecf01592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf6531640 1 -- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf0069e90 msgr2=0x7fecf0071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf6531640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf0069e90 0x7fecf0071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf6531640 1 -- 172.21.15.148:0/2767085422 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece8009a70 con 0x7fecf006cb80 2023-10-30T13:15:01.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf5d30640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf0069e90 0x7fecf0071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:01.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf6531640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 0x7fecf014d520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fece8019520 tx=0x7fece80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:01.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fece67fc640 1 -- 172.21.15.148:0/2767085422 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece80b9860 con 0x7fecf006cb80 2023-10-30T13:15:01.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fece67fc640 1 -- 172.21.15.148:0/2767085422 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece80b99c0 con 0x7fecf006cb80 2023-10-30T13:15:01.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf0158520 con 0x7fecf006cb80 2023-10-30T13:15:01.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fece67fc640 1 -- 172.21.15.148:0/2767085422 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece80b9d70 con 0x7fecf006cb80 2023-10-30T13:15:01.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf0158ae0 con 0x7fecf006cb80 2023-10-30T13:15:01.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf006a650 con 0x7fecf006cb80 2023-10-30T13:15:01.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.042+0000 7fece67fc640 1 -- 172.21.15.148:0/2767085422 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fece80c4460 con 0x7fecf006cb80 2023-10-30T13:15:01.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.046+0000 7fece67fc640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecd40664a0 0x7fecd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.046+0000 7fece67fc640 1 -- 172.21.15.148:0/2767085422 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fece811f410 con 0x7fecf006cb80 2023-10-30T13:15:01.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.046+0000 7fecf5d30640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecd40664a0 0x7fecd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.046+0000 7fecf5d30640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecd40664a0 0x7fecd4068960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fecec009970 tx=0x7fecec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:01.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.050+0000 7fece67fc640 1 -- 172.21.15.148:0/2767085422 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fecf006a650 con 0x7fecf006cb80 2023-10-30T13:15:01.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.230+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecf006a650 con 0x7fecf006cb80 2023-10-30T13:15:01.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.230+0000 7fece67fc640 1 -- 172.21.15.148:0/2767085422 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7fecf006a650 con 0x7fecf006cb80 2023-10-30T13:15:01.237 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:01.238 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:01.238 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:15:01.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecd40664a0 msgr2=0x7fecd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecd40664a0 0x7fecd4068960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fecec009970 tx=0x7fecec008040 comp rx=0 tx=0).stop 2023-10-30T13:15:01.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 msgr2=0x7fecf014d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 0x7fecf014d520 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fece8019520 tx=0x7fece80196b0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 shutdown_connections 2023-10-30T13:15:01.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecd40664a0 0x7fecd4068960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecf006cb80 0x7fecf014d520 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecf006a270 0x7fecf01592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 --2- 172.21.15.148:0/2767085422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf0069e90 0x7fecf0071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 >> 172.21.15.148:0/2767085422 conn(0x7fecf006dfd0 msgr2=0x7fecf0070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:01.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 shutdown_connections 2023-10-30T13:15:01.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.234+0000 7fecf7f94640 1 -- 172.21.15.148:0/2767085422 wait complete. 2023-10-30T13:15:01.258 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=2) 2023-10-30T13:15:01.258 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:15:01.259 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:15:01.273 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:15:01.273 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:01.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893c646640 1 Processor -- start 2023-10-30T13:15:01.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893c646640 1 -- start start 2023-10-30T13:15:01.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893c646640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89341518a0 0x7f8934158a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893c646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 0x7f89340c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893c646640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8934070260 con 0x7f89341581b0 2023-10-30T13:15:01.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893c646640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89340c30c0 con 0x7f89341518a0 2023-10-30T13:15:01.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893c646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89340c2490 con 0x7f893406fdd0 2023-10-30T13:15:01.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893a3e2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89341518a0 0x7f8934158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893a3e2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89341518a0 0x7f8934158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53822/0 (socket says 172.21.15.148:53822) 2023-10-30T13:15:01.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893a3e2640 1 -- 172.21.15.148:0/2331494881 learned_addr learned my addr 172.21.15.148:0/2331494881 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:01.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893abe3640 1 --2- 172.21.15.148:0/2331494881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 0x7f89340c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893a3e2640 1 -- 172.21.15.148:0/2331494881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 msgr2=0x7f89340c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:01.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893a3e2640 1 --2- 172.21.15.148:0/2331494881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 0x7f89340c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893a3e2640 1 -- 172.21.15.148:0/2331494881 >> v1:172.21.15.150:6789/0 conn(0x7f89341581b0 legacy=0x7f89340ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.450+0000 7f893a3e2640 1 -- 172.21.15.148:0/2331494881 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934151c80 con 0x7f89341518a0 2023-10-30T13:15:01.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f893a3e2640 1 --2- 172.21.15.148:0/2331494881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 0x7f8934158a20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8930009870 tx=0x7f893001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fa0f819e3a82a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:01.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f89393e0640 1 -- 172.21.15.148:0/2331494881 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300a8a30 con 0x7f89341518a0 2023-10-30T13:15:01.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f89393e0640 1 -- 172.21.15.148:0/2331494881 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89300a8b90 con 0x7f89341518a0 2023-10-30T13:15:01.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f89393e0640 1 -- 172.21.15.148:0/2331494881 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300b0440 con 0x7f89341518a0 2023-10-30T13:15:01.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f893c646640 1 -- 172.21.15.148:0/2331494881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 msgr2=0x7f8934158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f893c646640 1 --2- 172.21.15.148:0/2331494881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 0x7f8934158a20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8930009870 tx=0x7f893001ed30 comp rx=0 tx=0).stop 2023-10-30T13:15:01.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f893c646640 1 -- 172.21.15.148:0/2331494881 shutdown_connections 2023-10-30T13:15:01.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f893c646640 1 --2- 172.21.15.148:0/2331494881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 0x7f89340c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f893c646640 1 --2- 172.21.15.148:0/2331494881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 0x7f8934158a20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f893c646640 1 -- 172.21.15.148:0/2331494881 >> 172.21.15.148:0/2331494881 conn(0x7f8934069e90 msgr2=0x7f893406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:01.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f893c646640 1 -- 172.21.15.148:0/2331494881 shutdown_connections 2023-10-30T13:15:01.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.454+0000 7f893c646640 1 -- 172.21.15.148:0/2331494881 wait complete. 2023-10-30T13:15:01.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 Processor -- start 2023-10-30T13:15:01.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 -- start start 2023-10-30T13:15:01.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 0x7f89340c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 0x7f89341ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89341581b0 0x7f89341ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89340caa00 con 0x7f89341518a0 2023-10-30T13:15:01.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8934153590 con 0x7f89341581b0 2023-10-30T13:15:01.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8934152980 con 0x7f893406fdd0 2023-10-30T13:15:01.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f8939be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 0x7f89341ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f8939be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 0x7f89341ee130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53824/0 (socket says 172.21.15.148:53824) 2023-10-30T13:15:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f8939be1640 1 -- 172.21.15.148:0/2572733589 learned_addr learned my addr 172.21.15.148:0/2572733589 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893a3e2640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 0x7f89340c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893abe3640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89341581b0 0x7f89341ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f8939be1640 1 -- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 msgr2=0x7f89340c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f8939be1640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 0x7f89340c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f8939be1640 1 -- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89341581b0 msgr2=0x7f89341ee670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f8939be1640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89341581b0 0x7f89341ee670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f8939be1640 1 -- 172.21.15.148:0/2572733589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300094d0 con 0x7f89341518a0 2023-10-30T13:15:01.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f8939be1640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 0x7f89341ee130 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f892400c3c0 tx=0x7f892400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:01.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f892affd640 1 -- 172.21.15.148:0/2572733589 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89240052d0 con 0x7f89341518a0 2023-10-30T13:15:01.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f892affd640 1 -- 172.21.15.148:0/2572733589 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924005430 con 0x7f89341518a0 2023-10-30T13:15:01.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8934152c60 con 0x7f89341518a0 2023-10-30T13:15:01.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f892affd640 1 -- 172.21.15.148:0/2572733589 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924003840 con 0x7f89341518a0 2023-10-30T13:15:01.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.458+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8934153220 con 0x7f89341518a0 2023-10-30T13:15:01.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.462+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893404e170 con 0x7f89341518a0 2023-10-30T13:15:01.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.462+0000 7f892affd640 1 -- 172.21.15.148:0/2572733589 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f89240039a0 con 0x7f89341518a0 2023-10-30T13:15:01.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.462+0000 7f892affd640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664d0 0x7f8918068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:01.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.462+0000 7f892affd640 1 -- 172.21.15.148:0/2572733589 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f8924083180 con 0x7f89341518a0 2023-10-30T13:15:01.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.462+0000 7f893a3e2640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664d0 0x7f8918068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:01.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.462+0000 7f893a3e2640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664d0 0x7f8918068990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f89300b2000 tx=0x7f8930009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:01.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.466+0000 7f892affd640 1 -- 172.21.15.148:0/2572733589 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8924050640 con 0x7f89341518a0 2023-10-30T13:15:01.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.646+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8934070260 con 0x7f89341518a0 2023-10-30T13:15:01.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f892affd640 1 -- 172.21.15.148:0/2572733589 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f8934070260 con 0x7f89341518a0 2023-10-30T13:15:01.655 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:01.655 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:01.655 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:15:01.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664d0 msgr2=0x7f8918068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664d0 0x7f8918068990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f89300b2000 tx=0x7f8930009040 comp rx=0 tx=0).stop 2023-10-30T13:15:01.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 msgr2=0x7f89341ee130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 0x7f89341ee130 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f892400c3c0 tx=0x7f892400c880 comp rx=0 tx=0).stop 2023-10-30T13:15:01.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 shutdown_connections 2023-10-30T13:15:01.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f89180664d0 0x7f8918068990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89341518a0 0x7f89341ee130 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89341581b0 0x7f89341ee670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 --2- 172.21.15.148:0/2572733589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f893406fdd0 0x7f89340c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 >> 172.21.15.148:0/2572733589 conn(0x7f8934069e90 msgr2=0x7f8934155710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:01.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 shutdown_connections 2023-10-30T13:15:01.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:01.650+0000 7f893c646640 1 -- 172.21.15.148:0/2572733589 wait complete. 2023-10-30T13:15:01.675 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:15:01.839 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 159 2023-10-30T13:15:02.097 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 160 2023-10-30T13:15:02.106 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 161 2023-10-30T13:15:02.115 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 162 2023-10-30T13:15:02.124 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 163 2023-10-30T13:15:02.135 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 164 2023-10-30T13:15:02.144 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 165 2023-10-30T13:15:02.153 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 166 2023-10-30T13:15:02.163 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 167 2023-10-30T13:15:02.172 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 168 2023-10-30T13:15:02.181 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 169 2023-10-30T13:15:02.191 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 170 2023-10-30T13:15:02.200 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 171 2023-10-30T13:15:02.210 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 172 2023-10-30T13:15:02.219 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 173 2023-10-30T13:15:02.228 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 174 2023-10-30T13:15:02.238 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 175 2023-10-30T13:15:02.247 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 176 2023-10-30T13:15:02.256 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 177 2023-10-30T13:15:02.336 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 178 2023-10-30T13:15:02.568 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 179 2023-10-30T13:15:02.578 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 180 2023-10-30T13:15:02.587 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 181 2023-10-30T13:15:02.596 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 182 2023-10-30T13:15:02.857 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 183 2023-10-30T13:15:02.867 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 184 2023-10-30T13:15:02.877 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 185 2023-10-30T13:15:02.886 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 186 2023-10-30T13:15:02.896 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 187 2023-10-30T13:15:02.906 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 188 2023-10-30T13:15:03.167 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 189 2023-10-30T13:15:03.177 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 190 2023-10-30T13:15:03.186 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 191 2023-10-30T13:15:03.196 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 192 2023-10-30T13:15:03.206 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 193 2023-10-30T13:15:03.222 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 194 2023-10-30T13:15:03.236 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 195 2023-10-30T13:15:03.249 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 196 2023-10-30T13:15:03.261 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 197 2023-10-30T13:15:03.274 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 198 2023-10-30T13:15:03.288 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 199 2023-10-30T13:15:03.588 INFO:tasks.workunit.client.0.smithi148.stdout:Take first snapshot .snap/test1 2023-10-30T13:15:03.676 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:03.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92f606b640 1 Processor -- start 2023-10-30T13:15:03.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92f606b640 1 -- start start 2023-10-30T13:15:03.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92f606b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92f015d6f0 0x7f92f015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92f606b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 0x7f92f00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92f606b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f00cb800 con 0x7f92f01538a0 2023-10-30T13:15:03.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92f606b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f00cb330 con 0x7f92f015d6f0 2023-10-30T13:15:03.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92f606b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00cad70 con 0x7f92f00cf160 2023-10-30T13:15:03.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92eeffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92f015d6f0 0x7f92f015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92ef7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f92f01538a0 0x7f92f00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51070/0 (socket says 172.21.15.148:51070) 2023-10-30T13:15:03.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92ef7fe640 1 -- 172.21.15.148:0/1630342563 learned_addr learned my addr 172.21.15.148:0/1630342563 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:03.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92eeffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92f015d6f0 0x7f92f015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36960/0 (socket says 172.21.15.148:36960) 2023-10-30T13:15:03.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92effff640 1 --2- 172.21.15.148:0/1630342563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 0x7f92f00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92eeffd640 1 -- 172.21.15.148:0/1630342563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 msgr2=0x7f92f00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:03.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92eeffd640 1 --2- 172.21.15.148:0/1630342563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 0x7f92f00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92eeffd640 1 -- 172.21.15.148:0/1630342563 >> v1:172.21.15.148:6789/0 conn(0x7f92f01538a0 legacy=0x7f92f00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.810+0000 7f92eeffd640 1 -- 172.21.15.148:0/1630342563 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f00cb050 con 0x7f92f015d6f0 2023-10-30T13:15:03.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92eeffd640 1 --2- 172.21.15.148:0/1630342563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f015d6f0 0x7f92f015a090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f92e40097c0 tx=0x7f92e401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702e240509c6b3c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:03.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92ee7fc640 1 -- 172.21.15.148:0/1630342563 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40a8970 con 0x7f92f015d6f0 2023-10-30T13:15:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92ee7fc640 1 -- 172.21.15.148:0/1630342563 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e40a8ad0 con 0x7f92f015d6f0 2023-10-30T13:15:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92ee7fc640 1 -- 172.21.15.148:0/1630342563 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40a8e60 con 0x7f92f015d6f0 2023-10-30T13:15:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92f606b640 1 -- 172.21.15.148:0/1630342563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f015d6f0 msgr2=0x7f92f015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92f606b640 1 --2- 172.21.15.148:0/1630342563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f015d6f0 0x7f92f015a090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f92e40097c0 tx=0x7f92e401ec70 comp rx=0 tx=0).stop 2023-10-30T13:15:03.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92f606b640 1 -- 172.21.15.148:0/1630342563 shutdown_connections 2023-10-30T13:15:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92f606b640 1 --2- 172.21.15.148:0/1630342563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f015d6f0 0x7f92f015a090 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92f606b640 1 --2- 172.21.15.148:0/1630342563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 0x7f92f00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92f606b640 1 -- 172.21.15.148:0/1630342563 >> 172.21.15.148:0/1630342563 conn(0x7f92f006bfe0 msgr2=0x7f92f006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92f606b640 1 -- 172.21.15.148:0/1630342563 shutdown_connections 2023-10-30T13:15:03.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.814+0000 7f92f606b640 1 -- 172.21.15.148:0/1630342563 wait complete. 2023-10-30T13:15:03.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 Processor -- start 2023-10-30T13:15:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 -- start start 2023-10-30T13:15:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 0x7f92f0159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f01538a0 0x7f92f0148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f015d6f0 0x7f92f00c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f0158f60 con 0x7f92f01538a0 2023-10-30T13:15:03.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f01e4660 con 0x7f92f015d6f0 2023-10-30T13:15:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f0158a20 con 0x7f92f00cf160 2023-10-30T13:15:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92eeffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f01538a0 0x7f92f0148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92eeffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f01538a0 0x7f92f0148a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36970/0 (socket says 172.21.15.148:36970) 2023-10-30T13:15:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92eeffd640 1 -- 172.21.15.148:0/2871347337 learned_addr learned my addr 172.21.15.148:0/2871347337 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:03.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92ef7fe640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 0x7f92f0159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92effff640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f015d6f0 0x7f92f00c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92eeffd640 1 -- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 msgr2=0x7f92f0159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92eeffd640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 0x7f92f0159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92eeffd640 1 -- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f015d6f0 msgr2=0x7f92f00c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92eeffd640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f015d6f0 0x7f92f00c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92eeffd640 1 -- 172.21.15.148:0/2871347337 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e4009420 con 0x7f92f01538a0 2023-10-30T13:15:03.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92effff640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f015d6f0 0x7f92f00c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:03.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92eeffd640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f01538a0 0x7f92f0148a20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f92e401c8a0 tx=0x7f92e401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:03.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92d7fff640 1 -- 172.21.15.148:0/2871347337 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40b9630 con 0x7f92f01538a0 2023-10-30T13:15:03.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92d7fff640 1 -- 172.21.15.148:0/2871347337 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e40b97b0 con 0x7f92f01538a0 2023-10-30T13:15:03.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92d7fff640 1 -- 172.21.15.148:0/2871347337 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40b9b60 con 0x7f92f01538a0 2023-10-30T13:15:03.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f00c6950 con 0x7f92f01538a0 2023-10-30T13:15:03.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f0158460 con 0x7f92f01538a0 2023-10-30T13:15:03.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.818+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f00b2a50 con 0x7f92f01538a0 2023-10-30T13:15:03.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.822+0000 7f92d7fff640 1 -- 172.21.15.148:0/2871347337 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f92e40b7040 con 0x7f92f01538a0 2023-10-30T13:15:03.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.822+0000 7f92d7fff640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92dc0664a0 0x7f92dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.822+0000 7f92d7fff640 1 -- 172.21.15.148:0/2871347337 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f92e4121ed0 con 0x7f92f01538a0 2023-10-30T13:15:03.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.822+0000 7f92ef7fe640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92dc0664a0 0x7f92dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.822+0000 7f92ef7fe640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92dc0664a0 0x7f92dc068960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f92e0005890 tx=0x7f92e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:03.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:03.826+0000 7f92d7fff640 1 -- 172.21.15.148:0/2871347337 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92e40ef1d0 con 0x7f92f01538a0 2023-10-30T13:15:04.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.006+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92f004e170 con 0x7f92f01538a0 2023-10-30T13:15:04.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.006+0000 7f92d7fff640 1 -- 172.21.15.148:0/2871347337 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f92e40f27f0 con 0x7f92f01538a0 2023-10-30T13:15:04.013 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:04.013 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:04.014 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:15:04.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92dc0664a0 msgr2=0x7f92dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:04.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92dc0664a0 0x7f92dc068960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f92e0005890 tx=0x7f92e0009040 comp rx=0 tx=0).stop 2023-10-30T13:15:04.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f01538a0 msgr2=0x7f92f0148a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:04.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f01538a0 0x7f92f0148a20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f92e401c8a0 tx=0x7f92e401ba10 comp rx=0 tx=0).stop 2023-10-30T13:15:04.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 shutdown_connections 2023-10-30T13:15:04.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92dc0664a0 0x7f92dc068960 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:04.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92f01538a0 0x7f92f0148a20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:04.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f015d6f0 0x7f92f00c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:04.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 --2- 172.21.15.148:0/2871347337 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92f00cf160 0x7f92f0159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:04.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 >> 172.21.15.148:0/2871347337 conn(0x7f92f006bfe0 msgr2=0x7f92f00cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:04.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 shutdown_connections 2023-10-30T13:15:04.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:04.010+0000 7f92f606b640 1 -- 172.21.15.148:0/2871347337 wait complete. 2023-10-30T13:15:04.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:15:06.034 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:06.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.174+0000 7f63c7eea640 1 Processor -- start 2023-10-30T13:15:06.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.174+0000 7f63c7eea640 1 -- start start 2023-10-30T13:15:06.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c7eea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f63c0153d70 0x7f63c006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c7eea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f63c00c8490 0x7f63c00c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c7eea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c006afa0 con 0x7f63c006e470 2023-10-30T13:15:06.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c7eea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c006d780 con 0x7f63c00c8490 2023-10-30T13:15:06.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c7eea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c0154150 con 0x7f63c0153d70 2023-10-30T13:15:06.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c6487640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f63c00c8490 0x7f63c00c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c6487640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f63c00c8490 0x7f63c00c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36978/0 (socket says 172.21.15.148:36978) 2023-10-30T13:15:06.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c6487640 1 -- 172.21.15.148:0/2505712960 learned_addr learned my addr 172.21.15.148:0/2505712960 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:06.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c5485640 1 --1- 172.21.15.148:0/2505712960 >> v1:172.21.15.148:6789/0 conn(0x7f63c006e470 0x7f63c00cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51080/0 (socket says 172.21.15.148:51080) 2023-10-30T13:15:06.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c5c86640 1 --2- 172.21.15.148:0/2505712960 >> v2:172.21.15.150:3300/0 conn(0x7f63c0153d70 0x7f63c006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c6487640 1 -- 172.21.15.148:0/2505712960 >> v2:172.21.15.150:3300/0 conn(0x7f63c0153d70 msgr2=0x7f63c006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:06.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c6487640 1 --2- 172.21.15.148:0/2505712960 >> v2:172.21.15.150:3300/0 conn(0x7f63c0153d70 0x7f63c006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c6487640 1 -- 172.21.15.148:0/2505712960 >> v1:172.21.15.148:6789/0 conn(0x7f63c006e470 legacy=0x7f63c00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c6487640 1 -- 172.21.15.148:0/2505712960 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c006e030 con 0x7f63c00c8490 2023-10-30T13:15:06.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.178+0000 7f63c4c84640 1 -- 172.21.15.148:0/2505712960 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829238562 0 0) 0x7f63c006afa0 con 0x7f63c006e470 2023-10-30T13:15:06.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c6487640 1 --2- 172.21.15.148:0/2505712960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c00c8490 0x7f63c00c8cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f63c006d180 tx=0x7f63b801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56af33f6b673a42 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:06.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c4c84640 1 -- 172.21.15.148:0/2505712960 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80a5860 con 0x7f63c00c8490 2023-10-30T13:15:06.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c4c84640 1 -- 172.21.15.148:0/2505712960 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b80a59c0 con 0x7f63c00c8490 2023-10-30T13:15:06.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c4c84640 1 -- 172.21.15.148:0/2505712960 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80a5d50 con 0x7f63c00c8490 2023-10-30T13:15:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 -- 172.21.15.148:0/2505712960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c00c8490 msgr2=0x7f63c00c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 --2- 172.21.15.148:0/2505712960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c00c8490 0x7f63c00c8cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f63c006d180 tx=0x7f63b801c9a0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 -- 172.21.15.148:0/2505712960 shutdown_connections 2023-10-30T13:15:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 --2- 172.21.15.148:0/2505712960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c00c8490 0x7f63c00c8cc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 --2- 172.21.15.148:0/2505712960 >> v2:172.21.15.150:3300/0 conn(0x7f63c0153d70 0x7f63c006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 -- 172.21.15.148:0/2505712960 >> 172.21.15.148:0/2505712960 conn(0x7f63c0071ea0 msgr2=0x7f63c0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:06.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 -- 172.21.15.148:0/2505712960 shutdown_connections 2023-10-30T13:15:06.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 -- 172.21.15.148:0/2505712960 wait complete. 2023-10-30T13:15:06.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 Processor -- start 2023-10-30T13:15:06.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.182+0000 7f63c7eea640 1 -- start start 2023-10-30T13:15:06.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c7eea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63c006e470 0x7f63c006f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c7eea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63c00c8490 0x7f63c006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c7eea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c0153d70 0x7f63c0142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c7eea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c01f6b90 con 0x7f63c0153d70 2023-10-30T13:15:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c7eea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c00cc7b0 con 0x7f63c006e470 2023-10-30T13:15:06.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c7eea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c01f5d30 con 0x7f63c00c8490 2023-10-30T13:15:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c6487640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c0153d70 0x7f63c0142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c6487640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c0153d70 0x7f63c0142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36990/0 (socket says 172.21.15.148:36990) 2023-10-30T13:15:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c6487640 1 -- 172.21.15.148:0/4108173120 learned_addr learned my addr 172.21.15.148:0/4108173120 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c5c86640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63c006e470 0x7f63c006f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c5485640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63c00c8490 0x7f63c006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c6487640 1 -- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63c00c8490 msgr2=0x7f63c006f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c6487640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63c00c8490 0x7f63c006f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c6487640 1 -- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63c006e470 msgr2=0x7f63c006f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c6487640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63c006e470 0x7f63c006f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c6487640 1 -- 172.21.15.148:0/4108173120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8009b30 con 0x7f63c0153d70 2023-10-30T13:15:06.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c5485640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63c00c8490 0x7f63c006f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c6487640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c0153d70 0x7f63c0142360 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f63b80195b0 tx=0x7f63b8019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63b67fc640 1 -- 172.21.15.148:0/4108173120 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80bd070 con 0x7f63c0153d70 2023-10-30T13:15:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63b67fc640 1 -- 172.21.15.148:0/4108173120 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b80b8470 con 0x7f63c0153d70 2023-10-30T13:15:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c01f6010 con 0x7f63c0153d70 2023-10-30T13:15:06.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63b67fc640 1 -- 172.21.15.148:0/4108173120 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80b8840 con 0x7f63c0153d70 2023-10-30T13:15:06.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c01f65d0 con 0x7f63c0153d70 2023-10-30T13:15:06.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c004e170 con 0x7f63c0153d70 2023-10-30T13:15:06.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.186+0000 7f63b67fc640 1 -- 172.21.15.148:0/4108173120 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f63b80b89a0 con 0x7f63c0153d70 2023-10-30T13:15:06.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.190+0000 7f63b67fc640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f63a40664a0 0x7f63a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.190+0000 7f63b67fc640 1 -- 172.21.15.148:0/4108173120 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f63b811ec30 con 0x7f63c0153d70 2023-10-30T13:15:06.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.190+0000 7f63c5c86640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f63a40664a0 0x7f63a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.190+0000 7f63c5c86640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f63a40664a0 0x7f63a4068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009b60 tx=0x7f63bc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:06.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.194+0000 7f63b67fc640 1 -- 172.21.15.148:0/4108173120 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f63b80ebea0 con 0x7f63c0153d70 2023-10-30T13:15:06.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63c0153160 con 0x7f63c0153d70 2023-10-30T13:15:06.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63b67fc640 1 -- 172.21.15.148:0/4108173120 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f63b80ef4c0 con 0x7f63c0153d70 2023-10-30T13:15:06.384 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:06.384 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:06.385 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:15:06.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f63a40664a0 msgr2=0x7f63a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f63a40664a0 0x7f63a4068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009b60 tx=0x7f63bc009190 comp rx=0 tx=0).stop 2023-10-30T13:15:06.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c0153d70 msgr2=0x7f63c0142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c0153d70 0x7f63c0142360 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f63b80195b0 tx=0x7f63b8019740 comp rx=0 tx=0).stop 2023-10-30T13:15:06.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 shutdown_connections 2023-10-30T13:15:06.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f63a40664a0 0x7f63a4068960 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f63c0153d70 0x7f63c0142360 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63c00c8490 0x7f63c006f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 --2- 172.21.15.148:0/4108173120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63c006e470 0x7f63c006f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 >> 172.21.15.148:0/4108173120 conn(0x7f63c0071ea0 msgr2=0x7f63c0151d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:06.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 shutdown_connections 2023-10-30T13:15:06.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:06.378+0000 7f63c7eea640 1 -- 172.21.15.148:0/4108173120 wait complete. 2023-10-30T13:15:06.405 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:15:08.406 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:08.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcf1d3e640 1 Processor -- start 2023-10-30T13:15:08.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcf1d3e640 1 -- start start 2023-10-30T13:15:08.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcf1d3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcf1d3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efcec0cf160 0x7efcec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcf1d3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec0cb800 con 0x7efcec0ca0e0 2023-10-30T13:15:08.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcf1d3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec15dc30 con 0x7efcec0cf160 2023-10-30T13:15:08.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcf1d3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0cb330 con 0x7efcec1538a0 2023-10-30T13:15:08.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efceb7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcebfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efcec0cf160 0x7efcec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efceb7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:56042/0 (socket says 172.21.15.148:56042) 2023-10-30T13:15:08.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efceb7fe640 1 -- 172.21.15.148:0/1922342182 learned_addr learned my addr 172.21.15.148:0/1922342182 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:08.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcebfff640 1 -- 172.21.15.148:0/1922342182 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 msgr2=0x7efcec15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:08.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.550+0000 7efcebfff640 1 --2- 172.21.15.148:0/1922342182 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.554+0000 7efcebfff640 1 -- 172.21.15.148:0/1922342182 >> v1:172.21.15.150:6789/0 conn(0x7efcec0ca0e0 legacy=0x7efcec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.554+0000 7efcebfff640 1 -- 172.21.15.148:0/1922342182 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0cad70 con 0x7efcec0cf160 2023-10-30T13:15:08.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.554+0000 7efceb7fe640 1 --2- 172.21.15.148:0/1922342182 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:15:08.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.554+0000 7efcebfff640 1 --2- 172.21.15.148:0/1922342182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0cf160 0x7efcec0cf590 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efce4009d90 tx=0x7efce401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b7b27d556fd183d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:08.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.554+0000 7efcea7fc640 1 -- 172.21.15.148:0/1922342182 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce40af600 con 0x7efcec0cf160 2023-10-30T13:15:08.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.554+0000 7efcea7fc640 1 -- 172.21.15.148:0/1922342182 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce40ab050 con 0x7efcec0cf160 2023-10-30T13:15:08.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.554+0000 7efcea7fc640 1 -- 172.21.15.148:0/1922342182 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce40af980 con 0x7efcec0cf160 2023-10-30T13:15:08.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.554+0000 7efcf1d3e640 1 -- 172.21.15.148:0/1922342182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0cf160 msgr2=0x7efcec0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.554+0000 7efcf1d3e640 1 --2- 172.21.15.148:0/1922342182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0cf160 0x7efcec0cf590 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efce4009d90 tx=0x7efce401f130 comp rx=0 tx=0).stop 2023-10-30T13:15:08.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 -- 172.21.15.148:0/1922342182 shutdown_connections 2023-10-30T13:15:08.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 --2- 172.21.15.148:0/1922342182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0cf160 0x7efcec0cf590 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 --2- 172.21.15.148:0/1922342182 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 -- 172.21.15.148:0/1922342182 >> 172.21.15.148:0/1922342182 conn(0x7efcec06bfe0 msgr2=0x7efcec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:08.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 -- 172.21.15.148:0/1922342182 shutdown_connections 2023-10-30T13:15:08.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 -- 172.21.15.148:0/1922342182 wait complete. 2023-10-30T13:15:08.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 Processor -- start 2023-10-30T13:15:08.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 -- start start 2023-10-30T13:15:08.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0ca0e0 0x7efcec0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcec0cf160 0x7efcec15c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec1442e0 con 0x7efcec0cf160 2023-10-30T13:15:08.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec143fe0 con 0x7efcec0ca0e0 2023-10-30T13:15:08.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcf1d3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec14c570 con 0x7efcec1538a0 2023-10-30T13:15:08.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcec0cf160 0x7efcec15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcec0cf160 0x7efcec15c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37004/0 (socket says 172.21.15.148:37004) 2023-10-30T13:15:08.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceaffd640 1 -- 172.21.15.148:0/3242648660 learned_addr learned my addr 172.21.15.148:0/3242648660 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:08.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efcebfff640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceb7fe640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0ca0e0 0x7efcec0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceaffd640 1 -- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 msgr2=0x7efcec144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceaffd640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec144770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceaffd640 1 -- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0ca0e0 msgr2=0x7efcec0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceaffd640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0ca0e0 0x7efcec0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceaffd640 1 -- 172.21.15.148:0/3242648660 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0094d0 con 0x7efcec0cf160 2023-10-30T13:15:08.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.558+0000 7efceb7fe640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0ca0e0 0x7efcec0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:08.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efceaffd640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcec0cf160 0x7efcec15c920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efce000c530 tx=0x7efce000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:08.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efcd3fff640 1 -- 172.21.15.148:0/3242648660 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0005220 con 0x7efcec0cf160 2023-10-30T13:15:08.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efcd3fff640 1 -- 172.21.15.148:0/3242648660 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce0005380 con 0x7efcec0cf160 2023-10-30T13:15:08.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efcd3fff640 1 -- 172.21.15.148:0/3242648660 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0007270 con 0x7efcec0cf160 2023-10-30T13:15:08.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce4009a70 con 0x7efcec0cf160 2023-10-30T13:15:08.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec14cb30 con 0x7efcec0cf160 2023-10-30T13:15:08.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec0b2a50 con 0x7efcec0cf160 2023-10-30T13:15:08.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efcd3fff640 1 -- 172.21.15.148:0/3242648660 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efce00073d0 con 0x7efcec0cf160 2023-10-30T13:15:08.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efcd3fff640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efcd80664a0 0x7efcd8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efcd3fff640 1 -- 172.21.15.148:0/3242648660 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7efce0083870 con 0x7efcec0cf160 2023-10-30T13:15:08.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.562+0000 7efceb7fe640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efcd80664a0 0x7efcd8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.566+0000 7efceb7fe640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efcd80664a0 0x7efcd8068960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009c90 tx=0x7efcdc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:08.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.566+0000 7efcd3fff640 1 -- 172.21.15.148:0/3242648660 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efce0050c00 con 0x7efcec0cf160 2023-10-30T13:15:08.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcec04e170 con 0x7efcec0cf160 2023-10-30T13:15:08.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcd3fff640 1 -- 172.21.15.148:0/3242648660 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7efce0054220 con 0x7efcec0cf160 2023-10-30T13:15:08.755 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:08.755 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:08.756 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:15:08.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efcd80664a0 msgr2=0x7efcd8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efcd80664a0 0x7efcd8068960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009c90 tx=0x7efcdc008040 comp rx=0 tx=0).stop 2023-10-30T13:15:08.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcec0cf160 msgr2=0x7efcec15c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcec0cf160 0x7efcec15c920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efce000c530 tx=0x7efce000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 shutdown_connections 2023-10-30T13:15:08.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efcd80664a0 0x7efcd8068960 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efcec0cf160 0x7efcec15c920 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efcec1538a0 0x7efcec144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 --2- 172.21.15.148:0/3242648660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcec0ca0e0 0x7efcec0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 >> 172.21.15.148:0/3242648660 conn(0x7efcec06bfe0 msgr2=0x7efcec151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:08.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 shutdown_connections 2023-10-30T13:15:08.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:08.750+0000 7efcf1d3e640 1 -- 172.21.15.148:0/3242648660 wait complete. 2023-10-30T13:15:08.776 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:15:10.777 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:10.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.918+0000 7ff566654640 1 Processor -- start 2023-10-30T13:15:10.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.918+0000 7ff566654640 1 -- start start 2023-10-30T13:15:10.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.918+0000 7ff566654640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff5600c14f0 0x7ff5600c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.918+0000 7ff566654640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff56006c100 0x7ff5600c4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.918+0000 7ff566654640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5601584c0 con 0x7ff5600c18d0 2023-10-30T13:15:10.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff566654640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff560157610 con 0x7ff56006c100 2023-10-30T13:15:10.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff566654640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5601578b0 con 0x7ff5600c14f0 2023-10-30T13:15:10.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff564bf1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff56006c100 0x7ff5600c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff564bf1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff56006c100 0x7ff5600c4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37020/0 (socket says 172.21.15.148:37020) 2023-10-30T13:15:10.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff564bf1640 1 -- 172.21.15.148:0/2648844127 learned_addr learned my addr 172.21.15.148:0/2648844127 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:10.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff55ffff640 1 --2- 172.21.15.148:0/2648844127 >> v2:172.21.15.150:3300/0 conn(0x7ff5600c14f0 0x7ff5600c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff564bf1640 1 -- 172.21.15.148:0/2648844127 >> v2:172.21.15.150:3300/0 conn(0x7ff5600c14f0 msgr2=0x7ff5600c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:10.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff564bf1640 1 --2- 172.21.15.148:0/2648844127 >> v2:172.21.15.150:3300/0 conn(0x7ff5600c14f0 0x7ff5600c42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff564bf1640 1 -- 172.21.15.148:0/2648844127 >> v1:172.21.15.150:6789/0 conn(0x7ff5600c18d0 legacy=0x7ff56015add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff564bf1640 1 -- 172.21.15.148:0/2648844127 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff560157b90 con 0x7ff56006c100 2023-10-30T13:15:10.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff564bf1640 1 --2- 172.21.15.148:0/2648844127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 0x7ff5600c4830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff56006f630 tx=0x7ff55801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e671134c7c1dcd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:10.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff55effd640 1 -- 172.21.15.148:0/2648844127 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5580af400 con 0x7ff56006c100 2023-10-30T13:15:10.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff55effd640 1 -- 172.21.15.148:0/2648844127 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5580ab050 con 0x7ff56006c100 2023-10-30T13:15:10.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.922+0000 7ff55effd640 1 -- 172.21.15.148:0/2648844127 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5580af780 con 0x7ff56006c100 2023-10-30T13:15:10.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 -- 172.21.15.148:0/2648844127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 msgr2=0x7ff5600c4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 --2- 172.21.15.148:0/2648844127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 0x7ff5600c4830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff56006f630 tx=0x7ff55801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 -- 172.21.15.148:0/2648844127 shutdown_connections 2023-10-30T13:15:10.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 --2- 172.21.15.148:0/2648844127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 0x7ff5600c4830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 --2- 172.21.15.148:0/2648844127 >> v2:172.21.15.150:3300/0 conn(0x7ff5600c14f0 0x7ff5600c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 -- 172.21.15.148:0/2648844127 >> 172.21.15.148:0/2648844127 conn(0x7ff5600cace0 msgr2=0x7ff5600cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:10.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 -- 172.21.15.148:0/2648844127 shutdown_connections 2023-10-30T13:15:10.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 -- 172.21.15.148:0/2648844127 wait complete. 2023-10-30T13:15:10.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 Processor -- start 2023-10-30T13:15:10.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 -- start start 2023-10-30T13:15:10.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 0x7ff5600afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5600c14f0 0x7ff5601f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5600c18d0 0x7ff5600c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff560071b70 con 0x7ff56006c100 2023-10-30T13:15:10.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff560070ff0 con 0x7ff5600c14f0 2023-10-30T13:15:10.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff566654640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5600c2d10 con 0x7ff5600c18d0 2023-10-30T13:15:10.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff55ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 0x7ff5600afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff55ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 0x7ff5600afef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37028/0 (socket says 172.21.15.148:37028) 2023-10-30T13:15:10.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff55ffff640 1 -- 172.21.15.148:0/1057903241 learned_addr learned my addr 172.21.15.148:0/1057903241 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:10.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff55f7fe640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5600c14f0 0x7ff5601f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff564bf1640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5600c18d0 0x7ff5600c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff55ffff640 1 -- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5600c18d0 msgr2=0x7ff5600c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff55ffff640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5600c18d0 0x7ff5600c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff55ffff640 1 -- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5600c14f0 msgr2=0x7ff5601f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff55ffff640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5600c14f0 0x7ff5601f1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.926+0000 7ff55ffff640 1 -- 172.21.15.148:0/1057903241 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff558009b30 con 0x7ff56006c100 2023-10-30T13:15:10.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff55ffff640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 0x7ff5600afef0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff55000d830 tx=0x7ff55000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:10.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff55cff9640 1 -- 172.21.15.148:0/1057903241 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55000b660 con 0x7ff56006c100 2023-10-30T13:15:10.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff55cff9640 1 -- 172.21.15.148:0/1057903241 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55000b7c0 con 0x7ff56006c100 2023-10-30T13:15:10.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff55cff9640 1 -- 172.21.15.148:0/1057903241 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff55000bb30 con 0x7ff56006c100 2023-10-30T13:15:10.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff560155020 con 0x7ff56006c100 2023-10-30T13:15:10.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5600b0c20 con 0x7ff56006c100 2023-10-30T13:15:10.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56004e170 con 0x7ff56006c100 2023-10-30T13:15:10.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff55cff9640 1 -- 172.21.15.148:0/1057903241 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff55001e460 con 0x7ff56006c100 2023-10-30T13:15:10.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff55cff9640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff54c0664a0 0x7ff54c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff55cff9640 1 -- 172.21.15.148:0/1057903241 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7ff5500845e0 con 0x7ff56006c100 2023-10-30T13:15:10.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.930+0000 7ff55f7fe640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff54c0664a0 0x7ff54c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.934+0000 7ff55f7fe640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff54c0664a0 0x7ff54c068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff5601efa30 tx=0x7ff554009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:10.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:10.934+0000 7ff55cff9640 1 -- 172.21.15.148:0/1057903241 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5500518f0 con 0x7ff56006c100 2023-10-30T13:15:11.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff560155020 con 0x7ff56006c100 2023-10-30T13:15:11.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff55cff9640 1 -- 172.21.15.148:0/1057903241 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7ff560155020 con 0x7ff56006c100 2023-10-30T13:15:11.124 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:11.124 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:11.125 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:15:11.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff54c0664a0 msgr2=0x7ff54c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:11.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff54c0664a0 0x7ff54c068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff5601efa30 tx=0x7ff554009040 comp rx=0 tx=0).stop 2023-10-30T13:15:11.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 msgr2=0x7ff5600afef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:11.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 0x7ff5600afef0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff55000d830 tx=0x7ff55000dcf0 comp rx=0 tx=0).stop 2023-10-30T13:15:11.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 shutdown_connections 2023-10-30T13:15:11.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff54c0664a0 0x7ff54c068960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:11.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5600c18d0 0x7ff5600c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:11.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5600c14f0 0x7ff5601f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:11.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 --2- 172.21.15.148:0/1057903241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff56006c100 0x7ff5600afef0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:11.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 >> 172.21.15.148:0/1057903241 conn(0x7ff5600cace0 msgr2=0x7ff56006a830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:11.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 shutdown_connections 2023-10-30T13:15:11.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:11.118+0000 7ff566654640 1 -- 172.21.15.148:0/1057903241 wait complete. 2023-10-30T13:15:11.145 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:15:13.146 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:13.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f237483a640 1 Processor -- start 2023-10-30T13:15:13.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f237483a640 1 -- start start 2023-10-30T13:15:13.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f237483a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23701538f0 0x7f2370159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f237483a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f23700cefa0 0x7f23700cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f237483a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23700ca6a0 con 0x7f237015cd50 2023-10-30T13:15:13.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f237483a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237015d300 con 0x7f23700cefa0 2023-10-30T13:15:13.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f237483a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237015d5e0 con 0x7f23701538f0 2023-10-30T13:15:13.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f236ffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f23700cefa0 0x7f23700cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f236ffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f23700cefa0 0x7f23700cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55170/0 (socket says 172.21.15.148:55170) 2023-10-30T13:15:13.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f236ffff640 1 -- 172.21.15.148:0/2879097156 learned_addr learned my addr 172.21.15.148:0/2879097156 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:13.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f236f7fe640 1 --2- 172.21.15.148:0/2879097156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23701538f0 0x7f2370159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f236ffff640 1 -- 172.21.15.148:0/2879097156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23701538f0 msgr2=0x7f2370159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:13.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f236ffff640 1 --2- 172.21.15.148:0/2879097156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23701538f0 0x7f2370159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f236ffff640 1 -- 172.21.15.148:0/2879097156 >> v1:172.21.15.150:6789/0 conn(0x7f237015cd50 legacy=0x7f23700c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f236ffff640 1 -- 172.21.15.148:0/2879097156 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237015d8c0 con 0x7f23700cefa0 2023-10-30T13:15:13.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.290+0000 7f236f7fe640 1 --2- 172.21.15.148:0/2879097156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23701538f0 0x7f2370159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:13.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f236ffff640 1 --2- 172.21.15.148:0/2879097156 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23700cefa0 0x7f23700cf3f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2368009e50 tx=0x7f236801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e56e5c1fcb279ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:13.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f236e7fc640 1 -- 172.21.15.148:0/2879097156 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23680af400 con 0x7f23700cefa0 2023-10-30T13:15:13.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f236e7fc640 1 -- 172.21.15.148:0/2879097156 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23680ab050 con 0x7f23700cefa0 2023-10-30T13:15:13.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f236e7fc640 1 -- 172.21.15.148:0/2879097156 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23680af780 con 0x7f23700cefa0 2023-10-30T13:15:13.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f237483a640 1 -- 172.21.15.148:0/2879097156 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23700cefa0 msgr2=0x7f23700cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f237483a640 1 --2- 172.21.15.148:0/2879097156 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23700cefa0 0x7f23700cf3f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2368009e50 tx=0x7f236801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f237483a640 1 -- 172.21.15.148:0/2879097156 shutdown_connections 2023-10-30T13:15:13.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f237483a640 1 --2- 172.21.15.148:0/2879097156 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23700cefa0 0x7f23700cf3f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f237483a640 1 --2- 172.21.15.148:0/2879097156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23701538f0 0x7f2370159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f237483a640 1 -- 172.21.15.148:0/2879097156 >> 172.21.15.148:0/2879097156 conn(0x7f237006a640 msgr2=0x7f2370065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:13.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f237483a640 1 -- 172.21.15.148:0/2879097156 shutdown_connections 2023-10-30T13:15:13.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f237483a640 1 -- 172.21.15.148:0/2879097156 wait complete. 2023-10-30T13:15:13.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.294+0000 7f237483a640 1 Processor -- start 2023-10-30T13:15:13.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 -- start start 2023-10-30T13:15:13.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23700cefa0 0x7f23700b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23701538f0 0x7f23701e4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f237015cd50 0x7f23701458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23700b43a0 con 0x7f237015cd50 2023-10-30T13:15:13.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23701e6ce0 con 0x7f23701538f0 2023-10-30T13:15:13.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23701e6740 con 0x7f23700cefa0 2023-10-30T13:15:13.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f237015cd50 0x7f23701458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f237015cd50 0x7f23701458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55174/0 (socket says 172.21.15.148:55174) 2023-10-30T13:15:13.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236ffff640 1 -- 172.21.15.148:0/1409934498 learned_addr learned my addr 172.21.15.148:0/1409934498 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:13.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23700cefa0 0x7f23700b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236effd640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23701538f0 0x7f23701e4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236ffff640 1 -- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23700cefa0 msgr2=0x7f23700b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236ffff640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23700cefa0 0x7f23700b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236ffff640 1 -- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23701538f0 msgr2=0x7f23701e4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236ffff640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23701538f0 0x7f23701e4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236ffff640 1 -- 172.21.15.148:0/1409934498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2368009b30 con 0x7f237015cd50 2023-10-30T13:15:13.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f236ffff640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f237015cd50 0x7f23701458a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f236801ce20 tx=0x7f236801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:13.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f2357fff640 1 -- 172.21.15.148:0/1409934498 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23680afde0 con 0x7f237015cd50 2023-10-30T13:15:13.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f2357fff640 1 -- 172.21.15.148:0/1409934498 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2368009510 con 0x7f237015cd50 2023-10-30T13:15:13.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f2357fff640 1 -- 172.21.15.148:0/1409934498 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23680ba660 con 0x7f237015cd50 2023-10-30T13:15:13.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23701e6a00 con 0x7f237015cd50 2023-10-30T13:15:13.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23700b4e80 con 0x7f237015cd50 2023-10-30T13:15:13.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.298+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237004e170 con 0x7f237015cd50 2023-10-30T13:15:13.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.302+0000 7f2357fff640 1 -- 172.21.15.148:0/1409934498 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23680b6040 con 0x7f237015cd50 2023-10-30T13:15:13.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.302+0000 7f2357fff640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f235c0664a0 0x7f235c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.302+0000 7f2357fff640 1 -- 172.21.15.148:0/1409934498 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f2368122050 con 0x7f237015cd50 2023-10-30T13:15:13.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.302+0000 7f236f7fe640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f235c0664a0 0x7f235c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.302+0000 7f236f7fe640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f235c0664a0 0x7f235c068960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2360009950 tx=0x7f2360008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:13.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.306+0000 7f2357fff640 1 -- 172.21.15.148:0/1409934498 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23680ef540 con 0x7f237015cd50 2023-10-30T13:15:13.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.482+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23700b2290 con 0x7f237015cd50 2023-10-30T13:15:13.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f2357fff640 1 -- 172.21.15.148:0/1409934498 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f23680f2b60 con 0x7f237015cd50 2023-10-30T13:15:13.491 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:13.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:13.492 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:15:13.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f235c0664a0 msgr2=0x7f235c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f235c0664a0 0x7f235c068960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2360009950 tx=0x7f2360008040 comp rx=0 tx=0).stop 2023-10-30T13:15:13.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f237015cd50 msgr2=0x7f23701458a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f237015cd50 0x7f23701458a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f236801ce20 tx=0x7f236801bf90 comp rx=0 tx=0).stop 2023-10-30T13:15:13.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 shutdown_connections 2023-10-30T13:15:13.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f235c0664a0 0x7f235c068960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f237015cd50 0x7f23701458a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23701538f0 0x7f23701e4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 --2- 172.21.15.148:0/1409934498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23700cefa0 0x7f23700b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 >> 172.21.15.148:0/1409934498 conn(0x7f237006a640 msgr2=0x7f2370158f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:13.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 shutdown_connections 2023-10-30T13:15:13.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:13.486+0000 7f237483a640 1 -- 172.21.15.148:0/1409934498 wait complete. 2023-10-30T13:15:13.512 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:15:15.513 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:15.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb9782f7640 1 Processor -- start 2023-10-30T13:15:15.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb9782f7640 1 -- start start 2023-10-30T13:15:15.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb9782f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9701538a0 0x7fb97015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb9782f7640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb9700cf160 0x7fb9700cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb9782f7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9700cb800 con 0x7fb9700ca0e0 2023-10-30T13:15:15.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb9782f7640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97015dc30 con 0x7fb9700cf160 2023-10-30T13:15:15.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb9782f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9700cb330 con 0x7fb9701538a0 2023-10-30T13:15:15.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb976894640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb9700cf160 0x7fb9700cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb976894640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb9700cf160 0x7fb9700cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55188/0 (socket says 172.21.15.148:55188) 2023-10-30T13:15:15.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb976894640 1 -- 172.21.15.148:0/593291964 learned_addr learned my addr 172.21.15.148:0/593291964 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:15.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb976093640 1 --2- 172.21.15.148:0/593291964 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9701538a0 0x7fb97015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb976894640 1 -- 172.21.15.148:0/593291964 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9701538a0 msgr2=0x7fb97015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:15.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb976894640 1 --2- 172.21.15.148:0/593291964 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9701538a0 0x7fb97015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb976894640 1 -- 172.21.15.148:0/593291964 >> v1:172.21.15.150:6789/0 conn(0x7fb9700ca0e0 legacy=0x7fb97015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.658+0000 7fb976894640 1 -- 172.21.15.148:0/593291964 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9700cad70 con 0x7fb9700cf160 2023-10-30T13:15:15.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb976894640 1 --2- 172.21.15.148:0/593291964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9700cf160 0x7fb9700cf590 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb968009e50 tx=0x7fb96801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46a9de495a14dfc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:15.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb975091640 1 -- 172.21.15.148:0/593291964 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9680af400 con 0x7fb9700cf160 2023-10-30T13:15:15.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb975091640 1 -- 172.21.15.148:0/593291964 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9680ab050 con 0x7fb9700cf160 2023-10-30T13:15:15.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb975091640 1 -- 172.21.15.148:0/593291964 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9680af780 con 0x7fb9700cf160 2023-10-30T13:15:15.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb9782f7640 1 -- 172.21.15.148:0/593291964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9700cf160 msgr2=0x7fb9700cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb9782f7640 1 --2- 172.21.15.148:0/593291964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9700cf160 0x7fb9700cf590 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb968009e50 tx=0x7fb96801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb9782f7640 1 -- 172.21.15.148:0/593291964 shutdown_connections 2023-10-30T13:15:15.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb9782f7640 1 --2- 172.21.15.148:0/593291964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9700cf160 0x7fb9700cf590 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb9782f7640 1 --2- 172.21.15.148:0/593291964 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9701538a0 0x7fb97015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb9782f7640 1 -- 172.21.15.148:0/593291964 >> 172.21.15.148:0/593291964 conn(0x7fb97006bfe0 msgr2=0x7fb97006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:15.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb9782f7640 1 -- 172.21.15.148:0/593291964 shutdown_connections 2023-10-30T13:15:15.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb9782f7640 1 -- 172.21.15.148:0/593291964 wait complete. 2023-10-30T13:15:15.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.662+0000 7fb9782f7640 1 Processor -- start 2023-10-30T13:15:15.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 -- start start 2023-10-30T13:15:15.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9700ca0e0 0x7fb9700ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9700cf160 0x7fb97015c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9701538a0 0x7fb970144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9701442e0 con 0x7fb9701538a0 2023-10-30T13:15:15.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb970143fe0 con 0x7fb9700cf160 2023-10-30T13:15:15.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97014c570 con 0x7fb9700ca0e0 2023-10-30T13:15:15.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976894640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9701538a0 0x7fb970144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976894640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9701538a0 0x7fb970144770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55202/0 (socket says 172.21.15.148:55202) 2023-10-30T13:15:15.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976894640 1 -- 172.21.15.148:0/3846303987 learned_addr learned my addr 172.21.15.148:0/3846303987 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:15.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976093640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9700ca0e0 0x7fb9700ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb975892640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9700cf160 0x7fb97015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976894640 1 -- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9700ca0e0 msgr2=0x7fb9700ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976894640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9700ca0e0 0x7fb9700ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976894640 1 -- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9700cf160 msgr2=0x7fb97015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976894640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9700cf160 0x7fb97015c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976894640 1 -- 172.21.15.148:0/3846303987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb968009b30 con 0x7fb9701538a0 2023-10-30T13:15:15.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb976894640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9701538a0 0x7fb970144770 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb9680b2040 tx=0x7fb96801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:15.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb966ffd640 1 -- 172.21.15.148:0/3846303987 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9680afd30 con 0x7fb9701538a0 2023-10-30T13:15:15.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb966ffd640 1 -- 172.21.15.148:0/3846303987 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9680afe90 con 0x7fb9701538a0 2023-10-30T13:15:15.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb966ffd640 1 -- 172.21.15.148:0/3846303987 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9680ba680 con 0x7fb9701538a0 2023-10-30T13:15:15.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97014c850 con 0x7fb9701538a0 2023-10-30T13:15:15.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97014ce10 con 0x7fb9701538a0 2023-10-30T13:15:15.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.666+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9700b2a50 con 0x7fb9701538a0 2023-10-30T13:15:15.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.670+0000 7fb966ffd640 1 -- 172.21.15.148:0/3846303987 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9680b6040 con 0x7fb9701538a0 2023-10-30T13:15:15.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.670+0000 7fb966ffd640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9540664a0 0x7fb954068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.670+0000 7fb966ffd640 1 -- 172.21.15.148:0/3846303987 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fb968122150 con 0x7fb9701538a0 2023-10-30T13:15:15.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.670+0000 7fb976093640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9540664a0 0x7fb954068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.670+0000 7fb976093640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9540664a0 0x7fb954068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb96c0079c0 tx=0x7fb96c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:15.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.674+0000 7fb966ffd640 1 -- 172.21.15.148:0/3846303987 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9680ef6c0 con 0x7fb9701538a0 2023-10-30T13:15:15.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.854+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb970144490 con 0x7fb9701538a0 2023-10-30T13:15:15.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb966ffd640 1 -- 172.21.15.148:0/3846303987 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7fb970144490 con 0x7fb9701538a0 2023-10-30T13:15:15.863 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:15.863 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:15.863 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:15:15.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9540664a0 msgr2=0x7fb954068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9540664a0 0x7fb954068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb96c0079c0 tx=0x7fb96c008040 comp rx=0 tx=0).stop 2023-10-30T13:15:15.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9701538a0 msgr2=0x7fb970144770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9701538a0 0x7fb970144770 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb9680b2040 tx=0x7fb96801bf90 comp rx=0 tx=0).stop 2023-10-30T13:15:15.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 shutdown_connections 2023-10-30T13:15:15.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9540664a0 0x7fb954068960 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9701538a0 0x7fb970144770 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9700cf160 0x7fb97015c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 --2- 172.21.15.148:0/3846303987 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb9700ca0e0 0x7fb9700ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 >> 172.21.15.148:0/3846303987 conn(0x7fb97006bfe0 msgr2=0x7fb970151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:15.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 shutdown_connections 2023-10-30T13:15:15.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:15.858+0000 7fb9782f7640 1 -- 172.21.15.148:0/3846303987 wait complete. 2023-10-30T13:15:15.883 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:15:17.884 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:18.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5184194640 1 Processor -- start 2023-10-30T13:15:18.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5184194640 1 -- start start 2023-10-30T13:15:18.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5184194640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f517c153bf0 0x7f517c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:18.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5184194640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c0c6fe0 0x7f517c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:18.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5184194640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f517c0c79e0 0x7f517c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:18.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5184194640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c0c6810 con 0x7f517c153bf0 2023-10-30T13:15:18.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5184194640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c159900 con 0x7f517c0c79e0 2023-10-30T13:15:18.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5184194640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c0700b0 con 0x7f517c0c6fe0 2023-10-30T13:15:18.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5181f30640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f517c153bf0 0x7f517c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:18.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5181f30640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f517c153bf0 0x7f517c071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55214/0 (socket says 172.21.15.148:55214) 2023-10-30T13:15:18.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5181f30640 1 -- 172.21.15.148:0/3694789683 learned_addr learned my addr 172.21.15.148:0/3694789683 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:18.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f5182731640 1 --2- 172.21.15.148:0/3694789683 >> v2:172.21.15.150:3300/0 conn(0x7f517c0c79e0 0x7f517c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:18.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.026+0000 7f518172f640 1 --2- 172.21.15.148:0/3694789683 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c0c6fe0 0x7f517c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:18.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5181f30640 1 -- 172.21.15.148:0/3694789683 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c0c6fe0 msgr2=0x7f517c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:18.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5181f30640 1 --2- 172.21.15.148:0/3694789683 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c0c6fe0 0x7f517c0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5181f30640 1 -- 172.21.15.148:0/3694789683 >> v2:172.21.15.150:3300/0 conn(0x7f517c0c79e0 msgr2=0x7f517c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:18.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5181f30640 1 --2- 172.21.15.148:0/3694789683 >> v2:172.21.15.150:3300/0 conn(0x7f517c0c79e0 0x7f517c0cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5181f30640 1 -- 172.21.15.148:0/3694789683 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c0c6530 con 0x7f517c153bf0 2023-10-30T13:15:18.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5182731640 1 --2- 172.21.15.148:0/3694789683 >> v2:172.21.15.150:3300/0 conn(0x7f517c0c79e0 0x7f517c0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:18.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5181f30640 1 --2- 172.21.15.148:0/3694789683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c153bf0 0x7f517c071260 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f517c0c7400 tx=0x7f517801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51957d2314795d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:18.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5180f2e640 1 -- 172.21.15.148:0/3694789683 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51780a89b0 con 0x7f517c153bf0 2023-10-30T13:15:18.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5180f2e640 1 -- 172.21.15.148:0/3694789683 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51780a8b10 con 0x7f517c153bf0 2023-10-30T13:15:18.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5180f2e640 1 -- 172.21.15.148:0/3694789683 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51780a8ea0 con 0x7f517c153bf0 2023-10-30T13:15:18.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5184194640 1 -- 172.21.15.148:0/3694789683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c153bf0 msgr2=0x7f517c071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:18.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.030+0000 7f5184194640 1 --2- 172.21.15.148:0/3694789683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c153bf0 0x7f517c071260 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f517c0c7400 tx=0x7f517801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 -- 172.21.15.148:0/3694789683 shutdown_connections 2023-10-30T13:15:18.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 --2- 172.21.15.148:0/3694789683 >> v2:172.21.15.150:3300/0 conn(0x7f517c0c79e0 0x7f517c0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 --2- 172.21.15.148:0/3694789683 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c0c6fe0 0x7f517c0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 --2- 172.21.15.148:0/3694789683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c153bf0 0x7f517c071260 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 -- 172.21.15.148:0/3694789683 >> 172.21.15.148:0/3694789683 conn(0x7f517c06c6e0 msgr2=0x7f517c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:18.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 -- 172.21.15.148:0/3694789683 shutdown_connections 2023-10-30T13:15:18.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 -- 172.21.15.148:0/3694789683 wait complete. 2023-10-30T13:15:18.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 Processor -- start 2023-10-30T13:15:18.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 -- start start 2023-10-30T13:15:18.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c0c6fe0 0x7f517c1f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:18.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f517c0c79e0 0x7f517c154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:18.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c153bf0 0x7f517c1e63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:18.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c1555d0 con 0x7f517c0c6fe0 2023-10-30T13:15:18.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c14c420 con 0x7f517c0c79e0 2023-10-30T13:15:18.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5184194640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c155180 con 0x7f517c153bf0 2023-10-30T13:15:18.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5181f30640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c0c6fe0 0x7f517c1f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:18.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5181f30640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c0c6fe0 0x7f517c1f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55216/0 (socket says 172.21.15.148:55216) 2023-10-30T13:15:18.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5181f30640 1 -- 172.21.15.148:0/3224497799 learned_addr learned my addr 172.21.15.148:0/3224497799 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:18.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5182731640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c153bf0 0x7f517c1e63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:18.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f518172f640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f517c0c79e0 0x7f517c154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:18.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5181f30640 1 -- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c153bf0 msgr2=0x7f517c1e63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:18.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5181f30640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c153bf0 0x7f517c1e63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5181f30640 1 -- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f517c0c79e0 msgr2=0x7f517c154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:18.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5181f30640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f517c0c79e0 0x7f517c154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5181f30640 1 -- 172.21.15.148:0/3224497799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51780094d0 con 0x7f517c0c6fe0 2023-10-30T13:15:18.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f5181f30640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c0c6fe0 0x7f517c1f25c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51780b3040 tx=0x7f517801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:18.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.034+0000 7f51727fc640 1 -- 172.21.15.148:0/3224497799 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51780b0d80 con 0x7f517c0c6fe0 2023-10-30T13:15:18.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.038+0000 7f51727fc640 1 -- 172.21.15.148:0/3224497799 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51780bb470 con 0x7f517c0c6fe0 2023-10-30T13:15:18.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.038+0000 7f51727fc640 1 -- 172.21.15.148:0/3224497799 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51780bb820 con 0x7f517c0c6fe0 2023-10-30T13:15:18.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.038+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517c14b2b0 con 0x7f517c0c6fe0 2023-10-30T13:15:18.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.038+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517c14b870 con 0x7f517c0c6fe0 2023-10-30T13:15:18.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.038+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517c04e170 con 0x7f517c0c6fe0 2023-10-30T13:15:18.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.038+0000 7f51727fc640 1 -- 172.21.15.148:0/3224497799 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51780b7070 con 0x7f517c0c6fe0 2023-10-30T13:15:18.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.038+0000 7f51727fc640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f51600664d0 0x7f5160068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:18.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.038+0000 7f51727fc640 1 -- 172.21.15.148:0/3224497799 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f51781221f0 con 0x7f517c0c6fe0 2023-10-30T13:15:18.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.038+0000 7f518172f640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f51600664d0 0x7f5160068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:18.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.042+0000 7f518172f640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f51600664d0 0x7f5160068990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f517c1e6d20 tx=0x7f516c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:18.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.042+0000 7f51727fc640 1 -- 172.21.15.148:0/3224497799 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51780ef580 con 0x7f517c0c6fe0 2023-10-30T13:15:18.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.222+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f517c06fe90 con 0x7f517c0c6fe0 2023-10-30T13:15:18.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.222+0000 7f51727fc640 1 -- 172.21.15.148:0/3224497799 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f51780f2ba0 con 0x7f517c0c6fe0 2023-10-30T13:15:18.229 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:18.230 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:14:40.430696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":67,"addr":"172.21.15.148:6835/3749586444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3749586444},{"type":"v1","addr":"172.21.15.148:6835","nonce":3749586444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:18.230 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 19 2023-10-30T13:15:18.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f51600664d0 msgr2=0x7f5160068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:18.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f51600664d0 0x7f5160068990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f517c1e6d20 tx=0x7f516c008040 comp rx=0 tx=0).stop 2023-10-30T13:15:18.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c0c6fe0 msgr2=0x7f517c1f25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:18.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c0c6fe0 0x7f517c1f25c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51780b3040 tx=0x7f517801ba90 comp rx=0 tx=0).stop 2023-10-30T13:15:18.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 shutdown_connections 2023-10-30T13:15:18.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f51600664d0 0x7f5160068990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f517c153bf0 0x7f517c1e63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f517c0c79e0 0x7f517c154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 --2- 172.21.15.148:0/3224497799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f517c0c6fe0 0x7f517c1f25c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 >> 172.21.15.148:0/3224497799 conn(0x7f517c06c6e0 msgr2=0x7f517c151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:18.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 shutdown_connections 2023-10-30T13:15:18.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:18.226+0000 7f5184194640 1 -- 172.21.15.148:0/3224497799 wait complete. 2023-10-30T13:15:18.251 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:15:20.252 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.394+0000 7fc37d9c5640 1 Processor -- start 2023-10-30T13:15:20.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.394+0000 7fc37d9c5640 1 -- start start 2023-10-30T13:15:20.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.394+0000 7fc37d9c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.394+0000 7fc37d9c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3780ca0e0 0x7fc37815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.394+0000 7fc37d9c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780cb800 con 0x7fc3780cf160 2023-10-30T13:15:20.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.394+0000 7fc37d9c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37815dc30 con 0x7fc3780ca0e0 2023-10-30T13:15:20.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.394+0000 7fc37d9c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3780cb330 con 0x7fc3781538a0 2023-10-30T13:15:20.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc376ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc36ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3780ca0e0 0x7fc37815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc3777fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc3780cf160 0x7fc3780cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:56458/0 (socket says 172.21.15.148:56458) 2023-10-30T13:15:20.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc3777fe640 1 -- 172.21.15.148:0/4160855738 learned_addr learned my addr 172.21.15.148:0/4160855738 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:20.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc376ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:54382/0 (socket says 172.21.15.148:54382) 2023-10-30T13:15:20.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc36ffff640 1 -- 172.21.15.148:0/4160855738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3781538a0 msgr2=0x7fc37815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:20.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc36ffff640 1 --2- 172.21.15.148:0/4160855738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc36ffff640 1 -- 172.21.15.148:0/4160855738 >> v1:172.21.15.150:6789/0 conn(0x7fc3780cf160 legacy=0x7fc3780cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc36ffff640 1 -- 172.21.15.148:0/4160855738 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780cad70 con 0x7fc3780ca0e0 2023-10-30T13:15:20.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.398+0000 7fc376ffd640 1 --2- 172.21.15.148:0/4160855738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:15:20.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc36ffff640 1 --2- 172.21.15.148:0/4160855738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3780ca0e0 0x7fc37815a090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc3600097c0 tx=0x7fc36001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bf2f38106d616ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:20.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc3767fc640 1 -- 172.21.15.148:0/4160855738 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3600a8970 con 0x7fc3780ca0e0 2023-10-30T13:15:20.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc3767fc640 1 -- 172.21.15.148:0/4160855738 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3600a8ad0 con 0x7fc3780ca0e0 2023-10-30T13:15:20.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc3767fc640 1 -- 172.21.15.148:0/4160855738 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3600a8e60 con 0x7fc3780ca0e0 2023-10-30T13:15:20.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 -- 172.21.15.148:0/4160855738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3780ca0e0 msgr2=0x7fc37815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 --2- 172.21.15.148:0/4160855738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3780ca0e0 0x7fc37815a090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc3600097c0 tx=0x7fc36001ec70 comp rx=0 tx=0).stop 2023-10-30T13:15:20.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 -- 172.21.15.148:0/4160855738 shutdown_connections 2023-10-30T13:15:20.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 --2- 172.21.15.148:0/4160855738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3780ca0e0 0x7fc37815a090 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 --2- 172.21.15.148:0/4160855738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3781538a0 0x7fc37815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 -- 172.21.15.148:0/4160855738 >> 172.21.15.148:0/4160855738 conn(0x7fc37806bfe0 msgr2=0x7fc37806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:20.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 -- 172.21.15.148:0/4160855738 shutdown_connections 2023-10-30T13:15:20.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 -- 172.21.15.148:0/4160855738 wait complete. 2023-10-30T13:15:20.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 Processor -- start 2023-10-30T13:15:20.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 -- start start 2023-10-30T13:15:20.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3780ca0e0 0x7fc378147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3780cf160 0x7fc37815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3781538a0 0x7fc37815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3781f0c50 con 0x7fc3780ca0e0 2023-10-30T13:15:20.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3781436f0 con 0x7fc3781538a0 2023-10-30T13:15:20.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.402+0000 7fc37d9c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37815acf0 con 0x7fc3780cf160 2023-10-30T13:15:20.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc376ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3780ca0e0 0x7fc378147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc376ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3780ca0e0 0x7fc378147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55232/0 (socket says 172.21.15.148:55232) 2023-10-30T13:15:20.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc376ffd640 1 -- 172.21.15.148:0/3164747045 learned_addr learned my addr 172.21.15.148:0/3164747045 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:20.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc3777fe640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3781538a0 0x7fc37815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc36ffff640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3780cf160 0x7fc37815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc376ffd640 1 -- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3780cf160 msgr2=0x7fc37815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc376ffd640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3780cf160 0x7fc37815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc376ffd640 1 -- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3781538a0 msgr2=0x7fc37815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc376ffd640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3781538a0 0x7fc37815ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc376ffd640 1 -- 172.21.15.148:0/3164747045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc368009420 con 0x7fc3780ca0e0 2023-10-30T13:15:20.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc3777fe640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3781538a0 0x7fc37815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:20.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc376ffd640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3780ca0e0 0x7fc378147e90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e980 tx=0x7fc36800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:20.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc374ff9640 1 -- 172.21.15.148:0/3164747045 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36800c9a0 con 0x7fc3780ca0e0 2023-10-30T13:15:20.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc374ff9640 1 -- 172.21.15.148:0/3164747045 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36800cb00 con 0x7fc3780ca0e0 2023-10-30T13:15:20.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc374ff9640 1 -- 172.21.15.148:0/3164747045 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36800ce90 con 0x7fc3780ca0e0 2023-10-30T13:15:20.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc360009420 con 0x7fc3780ca0e0 2023-10-30T13:15:20.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37815b2b0 con 0x7fc3780ca0e0 2023-10-30T13:15:20.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3780b2a50 con 0x7fc3780ca0e0 2023-10-30T13:15:20.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.406+0000 7fc374ff9640 1 -- 172.21.15.148:0/3164747045 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc368013470 con 0x7fc3780ca0e0 2023-10-30T13:15:20.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.410+0000 7fc374ff9640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3580664a0 0x7fc358068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.410+0000 7fc374ff9640 1 -- 172.21.15.148:0/3164747045 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fc3680842c0 con 0x7fc3780ca0e0 2023-10-30T13:15:20.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.410+0000 7fc36ffff640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3580664a0 0x7fc358068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.410+0000 7fc36ffff640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3580664a0 0x7fc358068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc3600b2000 tx=0x7fc360008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:20.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.414+0000 7fc374ff9640 1 -- 172.21.15.148:0/3164747045 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3680514d0 con 0x7fc3780ca0e0 2023-10-30T13:15:20.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.590+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc37804e170 con 0x7fc3780ca0e0 2023-10-30T13:15:20.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc374ff9640 1 -- 172.21.15.148:0/3164747045 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fc368054af0 con 0x7fc3780ca0e0 2023-10-30T13:15:20.598 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:20.599 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:19.379891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:20.600 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 20 2023-10-30T13:15:20.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3580664a0 msgr2=0x7fc358068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3580664a0 0x7fc358068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc3600b2000 tx=0x7fc360008040 comp rx=0 tx=0).stop 2023-10-30T13:15:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3780ca0e0 msgr2=0x7fc378147e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3780ca0e0 0x7fc378147e90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e980 tx=0x7fc36800ee40 comp rx=0 tx=0).stop 2023-10-30T13:15:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 shutdown_connections 2023-10-30T13:15:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3580664a0 0x7fc358068960 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3781538a0 0x7fc37815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3780cf160 0x7fc37815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 --2- 172.21.15.148:0/3164747045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3780ca0e0 0x7fc378147e90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 >> 172.21.15.148:0/3164747045 conn(0x7fc37806bfe0 msgr2=0x7fc3780cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:20.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 shutdown_connections 2023-10-30T13:15:20.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.594+0000 7fc37d9c5640 1 -- 172.21.15.148:0/3164747045 wait complete. 2023-10-30T13:15:20.620 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:15:20.620 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:15:20.620 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:20.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f5735446640 1 Processor -- start 2023-10-30T13:15:20.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f5735446640 1 -- start start 2023-10-30T13:15:20.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f5735446640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f573015d6f0 0x7f573015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f5735446640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f57300cf160 0x7f57300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f5735446640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57300cb800 con 0x7f57301538a0 2023-10-30T13:15:20.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f5735446640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57300cb330 con 0x7f573015d6f0 2023-10-30T13:15:20.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f5735446640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57300cad70 con 0x7f57300cf160 2023-10-30T13:15:20.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f57267fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f573015d6f0 0x7f573015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f57267fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f573015d6f0 0x7f573015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55246/0 (socket says 172.21.15.148:55246) 2023-10-30T13:15:20.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f57267fc640 1 -- 172.21.15.148:0/317933148 learned_addr learned my addr 172.21.15.148:0/317933148 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:20.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f572f7fe640 1 --2- 172.21.15.148:0/317933148 >> v2:172.21.15.150:3300/0 conn(0x7f57300cf160 0x7f57300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f57267fc640 1 -- 172.21.15.148:0/317933148 >> v2:172.21.15.150:3300/0 conn(0x7f57300cf160 msgr2=0x7f57300cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:20.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f57267fc640 1 --2- 172.21.15.148:0/317933148 >> v2:172.21.15.150:3300/0 conn(0x7f57300cf160 0x7f57300cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f57267fc640 1 -- 172.21.15.148:0/317933148 >> v1:172.21.15.150:6789/0 conn(0x7f57301538a0 legacy=0x7f57300ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.754+0000 7f57267fc640 1 -- 172.21.15.148:0/317933148 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57300cb050 con 0x7f573015d6f0 2023-10-30T13:15:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f57267fc640 1 --2- 172.21.15.148:0/317933148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f573015d6f0 0x7f573015a090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f57300b5330 tx=0x7f572001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea27e3e0992782b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f572e7fc640 1 -- 172.21.15.148:0/317933148 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57200a89f0 con 0x7f573015d6f0 2023-10-30T13:15:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f572e7fc640 1 -- 172.21.15.148:0/317933148 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57200a8b50 con 0x7f573015d6f0 2023-10-30T13:15:20.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f572e7fc640 1 -- 172.21.15.148:0/317933148 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57200b0440 con 0x7f573015d6f0 2023-10-30T13:15:20.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f5735446640 1 -- 172.21.15.148:0/317933148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f573015d6f0 msgr2=0x7f573015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f5735446640 1 --2- 172.21.15.148:0/317933148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f573015d6f0 0x7f573015a090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f57300b5330 tx=0x7f572001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f5735446640 1 -- 172.21.15.148:0/317933148 shutdown_connections 2023-10-30T13:15:20.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f5735446640 1 --2- 172.21.15.148:0/317933148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f573015d6f0 0x7f573015a090 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f5735446640 1 --2- 172.21.15.148:0/317933148 >> v2:172.21.15.150:3300/0 conn(0x7f57300cf160 0x7f57300cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f5735446640 1 -- 172.21.15.148:0/317933148 >> 172.21.15.148:0/317933148 conn(0x7f573006bfe0 msgr2=0x7f573006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f5735446640 1 -- 172.21.15.148:0/317933148 shutdown_connections 2023-10-30T13:15:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.758+0000 7f5735446640 1 -- 172.21.15.148:0/317933148 wait complete. 2023-10-30T13:15:20.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 Processor -- start 2023-10-30T13:15:20.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 -- start start 2023-10-30T13:15:20.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57300cf160 0x7f57301e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57301538a0 0x7f57300b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573015d6f0 0x7f57300b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573014e2d0 con 0x7f57301538a0 2023-10-30T13:15:20.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57300b33b0 con 0x7f573015d6f0 2023-10-30T13:15:20.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57300cd9e0 con 0x7f57300cf160 2023-10-30T13:15:20.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f57267fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57301538a0 0x7f57300b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f57267fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57301538a0 0x7f57300b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55250/0 (socket says 172.21.15.148:55250) 2023-10-30T13:15:20.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f57267fc640 1 -- 172.21.15.148:0/2571373843 learned_addr learned my addr 172.21.15.148:0/2571373843 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:20.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f572f7fe640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573015d6f0 0x7f57300b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f572effd640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57300cf160 0x7f57301e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f57267fc640 1 -- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57300cf160 msgr2=0x7f57301e3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f57267fc640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57300cf160 0x7f57301e3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f57267fc640 1 -- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573015d6f0 msgr2=0x7f57300b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f57267fc640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573015d6f0 0x7f57300b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f57267fc640 1 -- 172.21.15.148:0/2571373843 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57200094d0 con 0x7f57301538a0 2023-10-30T13:15:20.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f572f7fe640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573015d6f0 0x7f57300b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:20.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f57267fc640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57301538a0 0x7f57300b41e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f572001b900 tx=0x7f572001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:20.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5727fff640 1 -- 172.21.15.148:0/2571373843 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57200ba600 con 0x7f57301538a0 2023-10-30T13:15:20.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5727fff640 1 -- 172.21.15.148:0/2571373843 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57200ba760 con 0x7f57301538a0 2023-10-30T13:15:20.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5727fff640 1 -- 172.21.15.148:0/2571373843 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57200bab10 con 0x7f57301538a0 2023-10-30T13:15:20.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57300ceaa0 con 0x7f57301538a0 2023-10-30T13:15:20.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.762+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57300cdd80 con 0x7f57301538a0 2023-10-30T13:15:20.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.766+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57300b2a50 con 0x7f57301538a0 2023-10-30T13:15:20.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.766+0000 7f5727fff640 1 -- 172.21.15.148:0/2571373843 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57200bac70 con 0x7f57301538a0 2023-10-30T13:15:20.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.766+0000 7f5727fff640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57100664a0 0x7f5710068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.766+0000 7f5727fff640 1 -- 172.21.15.148:0/2571373843 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f57201225d0 con 0x7f57301538a0 2023-10-30T13:15:20.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.766+0000 7f572effd640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57100664a0 0x7f5710068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.766+0000 7f572effd640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57100664a0 0x7f5710068960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5718000f70 tx=0x7f5718009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:20.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.770+0000 7f5727fff640 1 -- 172.21.15.148:0/2571373843 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57200ef7e0 con 0x7f57301538a0 2023-10-30T13:15:20.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5730159850 con 0x7f57301538a0 2023-10-30T13:15:20.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5727fff640 1 -- 172.21.15.148:0/2571373843 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f57200b5090 con 0x7f57301538a0 2023-10-30T13:15:20.956 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:20.956 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:19.379891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:20.957 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 20 2023-10-30T13:15:20.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57100664a0 msgr2=0x7f5710068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57100664a0 0x7f5710068960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5718000f70 tx=0x7f5718009040 comp rx=0 tx=0).stop 2023-10-30T13:15:20.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57301538a0 msgr2=0x7f57300b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57301538a0 0x7f57300b41e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f572001b900 tx=0x7f572001ba90 comp rx=0 tx=0).stop 2023-10-30T13:15:20.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 shutdown_connections 2023-10-30T13:15:20.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57100664a0 0x7f5710068960 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57301538a0 0x7f57300b41e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573015d6f0 0x7f57300b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 --2- 172.21.15.148:0/2571373843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f57300cf160 0x7f57301e3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 >> 172.21.15.148:0/2571373843 conn(0x7f573006bfe0 msgr2=0x7f57301e5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:20.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 shutdown_connections 2023-10-30T13:15:20.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:20.950+0000 7f5735446640 1 -- 172.21.15.148:0/2571373843 wait complete. 2023-10-30T13:15:20.977 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4270": { "addr": "172.21.15.150:6836/1620877178", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1620877178, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 1620877178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4270, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4285": { "addr": "172.21.15.148:6834/4162935019", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 4162935019, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 4162935019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4959": { "addr": "172.21.15.148:6837/95380877", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 95380877, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 95380877, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4959, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:15:19.379891+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4270, "mds_2": 4959 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:15:22.978 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d2990f640 1 Processor -- start 2023-10-30T13:15:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d2990f640 1 -- start start 2023-10-30T13:15:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d2990f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d24071800 0x7f8d2415a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d2990f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d240c91b0 0x7f8d24071be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d2990f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2406ac40 con 0x7f8d24072120 2023-10-30T13:15:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d2990f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2406bce0 con 0x7f8d24071800 2023-10-30T13:15:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d2990f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2406a030 con 0x7f8d240c91b0 2023-10-30T13:15:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d237fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8d24072120 0x7f8d240c6550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52270/0 (socket says 172.21.15.148:52270) 2023-10-30T13:15:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d237fe640 1 -- 172.21.15.148:0/860672951 learned_addr learned my addr 172.21.15.148:0/860672951 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d22ffd640 1 --2- 172.21.15.148:0/860672951 >> v2:172.21.15.150:3300/0 conn(0x7f8d24071800 0x7f8d2415a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d227fc640 1 --2- 172.21.15.148:0/860672951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d240c91b0 0x7f8d24071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d21ffb640 1 -- 172.21.15.148:0/860672951 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260474632 0 0) 0x7f8d2406ac40 con 0x7f8d24072120 2023-10-30T13:15:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d21ffb640 1 -- 172.21.15.148:0/860672951 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d0c003530 con 0x7f8d24072120 2023-10-30T13:15:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d21ffb640 1 -- 172.21.15.148:0/860672951 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2088784052 0 0) 0x7f8d0c003530 con 0x7f8d24072120 2023-10-30T13:15:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d22ffd640 1 -- 172.21.15.148:0/860672951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d240c91b0 msgr2=0x7f8d24071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d22ffd640 1 --2- 172.21.15.148:0/860672951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d240c91b0 0x7f8d24071be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d22ffd640 1 -- 172.21.15.148:0/860672951 >> v1:172.21.15.148:6789/0 conn(0x7f8d24072120 legacy=0x7f8d240c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.122+0000 7f8d22ffd640 1 -- 172.21.15.148:0/860672951 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2406a310 con 0x7f8d24071800 2023-10-30T13:15:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d22ffd640 1 --2- 172.21.15.148:0/860672951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d24071800 0x7f8d2415a050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8d140097c0 tx=0x7f8d1401c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=333bf24c7c51e777 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d21ffb640 1 -- 172.21.15.148:0/860672951 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c001a00 con 0x7f8d24071800 2023-10-30T13:15:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d21ffb640 1 -- 172.21.15.148:0/860672951 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d140a7050 con 0x7f8d24071800 2023-10-30T13:15:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d21ffb640 1 -- 172.21.15.148:0/860672951 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140ab600 con 0x7f8d24071800 2023-10-30T13:15:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d2990f640 1 -- 172.21.15.148:0/860672951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d24071800 msgr2=0x7f8d2415a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d2990f640 1 --2- 172.21.15.148:0/860672951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d24071800 0x7f8d2415a050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8d140097c0 tx=0x7f8d1401c3a0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d2990f640 1 -- 172.21.15.148:0/860672951 shutdown_connections 2023-10-30T13:15:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d2990f640 1 --2- 172.21.15.148:0/860672951 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d240c91b0 0x7f8d24071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d2990f640 1 --2- 172.21.15.148:0/860672951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d24071800 0x7f8d2415a050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d2990f640 1 -- 172.21.15.148:0/860672951 >> 172.21.15.148:0/860672951 conn(0x7f8d240c8650 msgr2=0x7f8d240c8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d2990f640 1 -- 172.21.15.148:0/860672951 shutdown_connections 2023-10-30T13:15:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.126+0000 7f8d2990f640 1 -- 172.21.15.148:0/860672951 wait complete. 2023-10-30T13:15:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 Processor -- start 2023-10-30T13:15:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 -- start start 2023-10-30T13:15:23.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d24071800 0x7f8d240c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:23.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d24072120 0x7f8d240c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:23.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d240c91b0 0x7f8d240bd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:23.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240bf4f0 con 0x7f8d24071800 2023-10-30T13:15:23.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240bc680 con 0x7f8d240c91b0 2023-10-30T13:15:23.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d240bc3e0 con 0x7f8d24072120 2023-10-30T13:15:23.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d22ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d24071800 0x7f8d240c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:23.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d22ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d24071800 0x7f8d240c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37098/0 (socket says 172.21.15.148:37098) 2023-10-30T13:15:23.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d22ffd640 1 -- 172.21.15.148:0/1383961191 learned_addr learned my addr 172.21.15.148:0/1383961191 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:23.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d227fc640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d24072120 0x7f8d240c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:23.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d237fe640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d240c91b0 0x7f8d240bd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:23.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d22ffd640 1 -- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d24072120 msgr2=0x7f8d240c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:23.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d22ffd640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d24072120 0x7f8d240c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d22ffd640 1 -- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d240c91b0 msgr2=0x7f8d240bd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:23.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d22ffd640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d240c91b0 0x7f8d240bd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d22ffd640 1 -- 172.21.15.148:0/1383961191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d14009420 con 0x7f8d24071800 2023-10-30T13:15:23.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d237fe640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d240c91b0 0x7f8d240bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:23.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d22ffd640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d24071800 0x7f8d240c59f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8d14000c10 tx=0x7f8d14019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:23.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d0bfff640 1 -- 172.21.15.148:0/1383961191 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140b4400 con 0x7f8d24071800 2023-10-30T13:15:23.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d0bfff640 1 -- 172.21.15.148:0/1383961191 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d140b4560 con 0x7f8d24071800 2023-10-30T13:15:23.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d240bef30 con 0x7f8d24071800 2023-10-30T13:15:23.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d0bfff640 1 -- 172.21.15.148:0/1383961191 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140b48f0 con 0x7f8d24071800 2023-10-30T13:15:23.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d240c45c0 con 0x7f8d24071800 2023-10-30T13:15:23.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.130+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2406eec0 con 0x7f8d24071800 2023-10-30T13:15:23.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.134+0000 7f8d0bfff640 1 -- 172.21.15.148:0/1383961191 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d140b2040 con 0x7f8d24071800 2023-10-30T13:15:23.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.134+0000 7f8d0bfff640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8d100664d0 0x7f8d10068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:23.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.134+0000 7f8d0bfff640 1 -- 172.21.15.148:0/1383961191 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f8d1411d680 con 0x7f8d24071800 2023-10-30T13:15:23.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.134+0000 7f8d227fc640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8d100664d0 0x7f8d10068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:23.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.134+0000 7f8d227fc640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8d100664d0 0x7f8d10068990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8d2415dbb0 tx=0x7f8d18009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:23.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.138+0000 7f8d0bfff640 1 -- 172.21.15.148:0/1383961191 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8d140ea9b0 con 0x7f8d24071800 2023-10-30T13:15:23.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.318+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d240c47b0 con 0x7f8d24071800 2023-10-30T13:15:23.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.318+0000 7f8d0bfff640 1 -- 172.21.15.148:0/1383961191 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f8d140edfd0 con 0x7f8d24071800 2023-10-30T13:15:23.326 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:23.326 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:19.379891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:23.326 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 20 2023-10-30T13:15:23.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8d100664d0 msgr2=0x7f8d10068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:23.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8d100664d0 0x7f8d10068990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8d2415dbb0 tx=0x7f8d18009190 comp rx=0 tx=0).stop 2023-10-30T13:15:23.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d24071800 msgr2=0x7f8d240c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:23.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d24071800 0x7f8d240c59f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8d14000c10 tx=0x7f8d14019140 comp rx=0 tx=0).stop 2023-10-30T13:15:23.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 shutdown_connections 2023-10-30T13:15:23.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8d100664d0 0x7f8d10068990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d240c91b0 0x7f8d240bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8d24072120 0x7f8d240c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 --2- 172.21.15.148:0/1383961191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d24071800 0x7f8d240c59f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:23.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 >> 172.21.15.148:0/1383961191 conn(0x7f8d240c8650 msgr2=0x7f8d24152bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:23.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 shutdown_connections 2023-10-30T13:15:23.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:23.322+0000 7f8d2990f640 1 -- 172.21.15.148:0/1383961191 wait complete. 2023-10-30T13:15:23.346 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:23.888 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:15:23.892 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:15:25.347 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:25.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f724e7c5640 1 Processor -- start 2023-10-30T13:15:25.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f724e7c5640 1 -- start start 2023-10-30T13:15:25.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f724e7c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248156f80 0x7f7248159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:25.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f724e7c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7248070c90 0x7f7248164120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:25.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f724e7c5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72480c2f20 con 0x7f7248070650 2023-10-30T13:15:25.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f724e7c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7248157360 con 0x7f7248070c90 2023-10-30T13:15:25.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f724e7c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72480c85c0 con 0x7f7248156f80 2023-10-30T13:15:25.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f72477fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7248070650 0x7f7248071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52282/0 (socket says 172.21.15.148:52282) 2023-10-30T13:15:25.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f72477fe640 1 -- 172.21.15.148:0/4107686644 learned_addr learned my addr 172.21.15.148:0/4107686644 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:25.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f7247fff640 1 --2- 172.21.15.148:0/4107686644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248156f80 0x7f7248159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:25.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f724cd62640 1 --2- 172.21.15.148:0/4107686644 >> v2:172.21.15.150:3300/0 conn(0x7f7248070c90 0x7f7248164120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:25.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f7246ffd640 1 -- 172.21.15.148:0/4107686644 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025864883 0 0) 0x7f72480c2f20 con 0x7f7248070650 2023-10-30T13:15:25.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.490+0000 7f7246ffd640 1 -- 172.21.15.148:0/4107686644 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7230003530 con 0x7f7248070650 2023-10-30T13:15:25.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f7247fff640 1 -- 172.21.15.148:0/4107686644 >> v2:172.21.15.150:3300/0 conn(0x7f7248070c90 msgr2=0x7f7248164120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:25.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f7246ffd640 1 -- 172.21.15.148:0/4107686644 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3979101674 0 0) 0x7f7230003530 con 0x7f7248070650 2023-10-30T13:15:25.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f7247fff640 1 --2- 172.21.15.148:0/4107686644 >> v2:172.21.15.150:3300/0 conn(0x7f7248070c90 0x7f7248164120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f7247fff640 1 -- 172.21.15.148:0/4107686644 >> v1:172.21.15.148:6789/0 conn(0x7f7248070650 legacy=0x7f7248071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:25.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f7247fff640 1 -- 172.21.15.148:0/4107686644 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7248071230 con 0x7f7248156f80 2023-10-30T13:15:25.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f7247fff640 1 --2- 172.21.15.148:0/4107686644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248156f80 0x7f7248159880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f72380097c0 tx=0x7f723801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae689a2571207f6e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:25.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f7246ffd640 1 -- 172.21.15.148:0/4107686644 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723c001680 con 0x7f7248156f80 2023-10-30T13:15:25.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f7246ffd640 1 -- 172.21.15.148:0/4107686644 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72380a7050 con 0x7f7248156f80 2023-10-30T13:15:25.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f7246ffd640 1 -- 172.21.15.148:0/4107686644 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72380ab600 con 0x7f7248156f80 2023-10-30T13:15:25.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f724e7c5640 1 -- 172.21.15.148:0/4107686644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248156f80 msgr2=0x7f7248159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:25.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.494+0000 7f724e7c5640 1 --2- 172.21.15.148:0/4107686644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248156f80 0x7f7248159880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f72380097c0 tx=0x7f723801c3a0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 -- 172.21.15.148:0/4107686644 shutdown_connections 2023-10-30T13:15:25.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 --2- 172.21.15.148:0/4107686644 >> v2:172.21.15.150:3300/0 conn(0x7f7248070c90 0x7f7248164120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 --2- 172.21.15.148:0/4107686644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248156f80 0x7f7248159880 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 -- 172.21.15.148:0/4107686644 >> 172.21.15.148:0/4107686644 conn(0x7f72480cbeb0 msgr2=0x7f72480cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:25.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 -- 172.21.15.148:0/4107686644 shutdown_connections 2023-10-30T13:15:25.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 -- 172.21.15.148:0/4107686644 wait complete. 2023-10-30T13:15:25.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 Processor -- start 2023-10-30T13:15:25.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 -- start start 2023-10-30T13:15:25.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248070650 0x7f7248158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:25.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7248070c90 0x7f7248155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:25.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7248156f80 0x7f72481452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:25.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7248158a10 con 0x7f7248070c90 2023-10-30T13:15:25.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7248164150 con 0x7f7248156f80 2023-10-30T13:15:25.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724e7c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7248159370 con 0x7f7248070650 2023-10-30T13:15:25.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f72477fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7248070c90 0x7f7248155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:25.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f72477fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7248070c90 0x7f7248155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37100/0 (socket says 172.21.15.148:37100) 2023-10-30T13:15:25.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f72477fe640 1 -- 172.21.15.148:0/3874481852 learned_addr learned my addr 172.21.15.148:0/3874481852 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:25.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f7247fff640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248070650 0x7f7248158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:25.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f724cd62640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7248156f80 0x7f72481452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:25.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f72477fe640 1 -- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248070650 msgr2=0x7f7248158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:25.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f72477fe640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248070650 0x7f7248158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f72477fe640 1 -- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7248156f80 msgr2=0x7f72481452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:25.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f72477fe640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7248156f80 0x7f72481452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.498+0000 7f72477fe640 1 -- 172.21.15.148:0/3874481852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7238009420 con 0x7f7248070c90 2023-10-30T13:15:25.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f72477fe640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7248070c90 0x7f7248155b20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f723c0017b0 tx=0x7f723c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:25.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f7244ff9640 1 -- 172.21.15.148:0/3874481852 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723c005ae0 con 0x7f7248070c90 2023-10-30T13:15:25.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f7244ff9640 1 -- 172.21.15.148:0/3874481852 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723c005c40 con 0x7f7248070c90 2023-10-30T13:15:25.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f7244ff9640 1 -- 172.21.15.148:0/3874481852 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723c007290 con 0x7f7248070c90 2023-10-30T13:15:25.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7248156660 con 0x7f7248070c90 2023-10-30T13:15:25.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7248158630 con 0x7f7248070c90 2023-10-30T13:15:25.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7248158b50 con 0x7f7248070c90 2023-10-30T13:15:25.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f7244ff9640 1 -- 172.21.15.148:0/3874481852 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f723c0073f0 con 0x7f7248070c90 2023-10-30T13:15:25.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f7244ff9640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72340664a0 0x7f7234068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:25.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f7244ff9640 1 -- 172.21.15.148:0/3874481852 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f723c083cb0 con 0x7f7248070c90 2023-10-30T13:15:25.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.502+0000 7f7247fff640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72340664a0 0x7f7234068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:25.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.506+0000 7f7247fff640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72340664a0 0x7f7234068960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f72380ad000 tx=0x7f72380a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:25.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.506+0000 7f7244ff9640 1 -- 172.21.15.148:0/3874481852 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7248158b50 con 0x7f7248070c90 2023-10-30T13:15:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7248158b50 con 0x7f7248070c90 2023-10-30T13:15:25.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f7244ff9640 1 -- 172.21.15.148:0/3874481852 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f7248158b50 con 0x7f7248070c90 2023-10-30T13:15:25.695 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:25.695 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:24.394103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:resolve","state_seq":10,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:25.696 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 21 2023-10-30T13:15:25.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72340664a0 msgr2=0x7f7234068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:25.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72340664a0 0x7f7234068960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f72380ad000 tx=0x7f72380a6000 comp rx=0 tx=0).stop 2023-10-30T13:15:25.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7248070c90 msgr2=0x7f7248155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:25.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7248070c90 0x7f7248155b20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f723c0017b0 tx=0x7f723c00cbe0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 shutdown_connections 2023-10-30T13:15:25.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72340664a0 0x7f7234068960 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7248070c90 0x7f7248155b20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7248156f80 0x7f72481452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 --2- 172.21.15.148:0/3874481852 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7248070650 0x7f7248158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 >> 172.21.15.148:0/3874481852 conn(0x7f72480cbeb0 msgr2=0x7f72480cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:25.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 shutdown_connections 2023-10-30T13:15:25.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:25.690+0000 7f724e7c5640 1 -- 172.21.15.148:0/3874481852 wait complete. 2023-10-30T13:15:25.716 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:27.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:27.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1f73e640 1 Processor -- start 2023-10-30T13:15:27.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1f73e640 1 -- start start 2023-10-30T13:15:27.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1f73e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf181538a0 0x7fdf1815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1f73e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdf180cf160 0x7fdf180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1f73e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf180cb800 con 0x7fdf180ca0e0 2023-10-30T13:15:27.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1f73e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1815dc30 con 0x7fdf180cf160 2023-10-30T13:15:27.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1f73e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180cb330 con 0x7fdf181538a0 2023-10-30T13:15:27.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1ccd9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fdf180ca0e0 0x7fdf1815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52290/0 (socket says 172.21.15.148:52290) 2023-10-30T13:15:27.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1ccd9640 1 -- 172.21.15.148:0/3396519929 learned_addr learned my addr 172.21.15.148:0/3396519929 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:27.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1d4da640 1 --2- 172.21.15.148:0/3396519929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf181538a0 0x7fdf1815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1dcdb640 1 --2- 172.21.15.148:0/3396519929 >> v2:172.21.15.150:3300/0 conn(0x7fdf180cf160 0x7fdf180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf0ffff640 1 -- 172.21.15.148:0/3396519929 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806727864 0 0) 0x7fdf180cb800 con 0x7fdf180ca0e0 2023-10-30T13:15:27.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf0ffff640 1 -- 172.21.15.148:0/3396519929 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf04003530 con 0x7fdf180ca0e0 2023-10-30T13:15:27.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1dcdb640 1 -- 172.21.15.148:0/3396519929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf181538a0 msgr2=0x7fdf1815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:27.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1dcdb640 1 --2- 172.21.15.148:0/3396519929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf181538a0 0x7fdf1815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1dcdb640 1 -- 172.21.15.148:0/3396519929 >> v1:172.21.15.148:6789/0 conn(0x7fdf180ca0e0 legacy=0x7fdf1815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf0ffff640 1 -- 172.21.15.148:0/3396519929 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1042817290 0 0) 0x7fdf04003530 con 0x7fdf180ca0e0 2023-10-30T13:15:27.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.862+0000 7fdf1dcdb640 1 -- 172.21.15.148:0/3396519929 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf180cad70 con 0x7fdf180cf160 2023-10-30T13:15:27.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf1dcdb640 1 --2- 172.21.15.148:0/3396519929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf180cf160 0x7fdf180cf590 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009d90 tx=0x7fdf1001f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d251916788ff66a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:27.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf0ffff640 1 -- 172.21.15.148:0/3396519929 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100af600 con 0x7fdf180cf160 2023-10-30T13:15:27.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf0ffff640 1 -- 172.21.15.148:0/3396519929 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf100ab050 con 0x7fdf180cf160 2023-10-30T13:15:27.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf0ffff640 1 -- 172.21.15.148:0/3396519929 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100af960 con 0x7fdf180cf160 2023-10-30T13:15:27.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf1f73e640 1 -- 172.21.15.148:0/3396519929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf180cf160 msgr2=0x7fdf180cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf1f73e640 1 --2- 172.21.15.148:0/3396519929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf180cf160 0x7fdf180cf590 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009d90 tx=0x7fdf1001f130 comp rx=0 tx=0).stop 2023-10-30T13:15:27.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf1f73e640 1 -- 172.21.15.148:0/3396519929 shutdown_connections 2023-10-30T13:15:27.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf1f73e640 1 --2- 172.21.15.148:0/3396519929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf180cf160 0x7fdf180cf590 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf1f73e640 1 --2- 172.21.15.148:0/3396519929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf181538a0 0x7fdf1815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf1f73e640 1 -- 172.21.15.148:0/3396519929 >> 172.21.15.148:0/3396519929 conn(0x7fdf1806bfe0 msgr2=0x7fdf1806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:27.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf1f73e640 1 -- 172.21.15.148:0/3396519929 shutdown_connections 2023-10-30T13:15:27.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.866+0000 7fdf1f73e640 1 -- 172.21.15.148:0/3396519929 wait complete. 2023-10-30T13:15:27.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 Processor -- start 2023-10-30T13:15:27.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 -- start start 2023-10-30T13:15:27.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf180ca0e0 0x7fdf18152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf180cf160 0x7fdf181e4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf181538a0 0x7fdf181e4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180b3240 con 0x7fdf180cf160 2023-10-30T13:15:27.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180b3520 con 0x7fdf181538a0 2023-10-30T13:15:27.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180b3800 con 0x7fdf180ca0e0 2023-10-30T13:15:27.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1ccd9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf180cf160 0x7fdf181e4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1d4da640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf180ca0e0 0x7fdf18152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1dcdb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf181538a0 0x7fdf181e4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1ccd9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf180cf160 0x7fdf181e4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37102/0 (socket says 172.21.15.148:37102) 2023-10-30T13:15:27.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1ccd9640 1 -- 172.21.15.148:0/2109339702 learned_addr learned my addr 172.21.15.148:0/2109339702 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:27.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1ccd9640 1 -- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf180ca0e0 msgr2=0x7fdf18152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1ccd9640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf180ca0e0 0x7fdf18152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1ccd9640 1 -- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf181538a0 msgr2=0x7fdf181e4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1ccd9640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf181538a0 0x7fdf181e4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1ccd9640 1 -- 172.21.15.148:0/2109339702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf10009a70 con 0x7fdf180cf160 2023-10-30T13:15:27.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1ccd9640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf180cf160 0x7fdf181e4560 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800c780 tx=0x7fdf0800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:27.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf0dffb640 1 -- 172.21.15.148:0/2109339702 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08005ac0 con 0x7fdf180cf160 2023-10-30T13:15:27.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf0dffb640 1 -- 172.21.15.148:0/2109339702 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf08005c20 con 0x7fdf180cf160 2023-10-30T13:15:27.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf0dffb640 1 -- 172.21.15.148:0/2109339702 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf080072b0 con 0x7fdf180cf160 2023-10-30T13:15:27.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf180b3ae0 con 0x7fdf180cf160 2023-10-30T13:15:27.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.870+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf180b40a0 con 0x7fdf180cf160 2023-10-30T13:15:27.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.874+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf180b2a50 con 0x7fdf180cf160 2023-10-30T13:15:27.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.874+0000 7fdf0dffb640 1 -- 172.21.15.148:0/2109339702 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdf08007410 con 0x7fdf180cf160 2023-10-30T13:15:27.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.874+0000 7fdf0dffb640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdefc0664a0 0x7fdefc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.874+0000 7fdf0dffb640 1 -- 172.21.15.148:0/2109339702 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fdf08083b60 con 0x7fdf180cf160 2023-10-30T13:15:27.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.874+0000 7fdf1d4da640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdefc0664a0 0x7fdefc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.874+0000 7fdf1d4da640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdefc0664a0 0x7fdefc068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009ac0 tx=0x7fdf14009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:27.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:27.878+0000 7fdf0dffb640 1 -- 172.21.15.148:0/2109339702 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdf08050ea0 con 0x7fdf180cf160 2023-10-30T13:15:28.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.058+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf18159860 con 0x7fdf180cf160 2023-10-30T13:15:28.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.058+0000 7fdf0dffb640 1 -- 172.21.15.148:0/2109339702 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fdf080544c0 con 0x7fdf180cf160 2023-10-30T13:15:28.065 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:28.065 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:24.394103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:resolve","state_seq":10,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:28.065 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 21 2023-10-30T13:15:28.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdefc0664a0 msgr2=0x7fdefc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:28.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdefc0664a0 0x7fdefc068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdf14009ac0 tx=0x7fdf14009190 comp rx=0 tx=0).stop 2023-10-30T13:15:28.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf180cf160 msgr2=0x7fdf181e4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:28.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf180cf160 0x7fdf181e4560 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800c780 tx=0x7fdf0800cc40 comp rx=0 tx=0).stop 2023-10-30T13:15:28.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 shutdown_connections 2023-10-30T13:15:28.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdefc0664a0 0x7fdefc068960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:28.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdf180cf160 0x7fdf181e4560 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:28.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf181538a0 0x7fdf181e4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:28.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 --2- 172.21.15.148:0/2109339702 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdf180ca0e0 0x7fdf18152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:28.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 >> 172.21.15.148:0/2109339702 conn(0x7fdf1806bfe0 msgr2=0x7fdf1815a000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:28.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 shutdown_connections 2023-10-30T13:15:28.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:28.062+0000 7fdf1f73e640 1 -- 172.21.15.148:0/2109339702 wait complete. 2023-10-30T13:15:28.085 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:30.087 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.250+0000 7fdeb2c88640 1 Processor -- start 2023-10-30T13:15:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.250+0000 7fdeb2c88640 1 -- start start 2023-10-30T13:15:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb2c88640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdeac157c80 0x7fdeac161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb2c88640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeac161f60 con 0x7fdeac06fef0 2023-10-30T13:15:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb2c88640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeac1616c0 con 0x7fdeac06ab20 2023-10-30T13:15:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb2c88640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeac15ef30 con 0x7fdeac157c80 2023-10-30T13:15:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1485640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fdeac06fef0 0x7fdeac06f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52306/0 (socket says 172.21.15.148:52306) 2023-10-30T13:15:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1485640 1 -- 172.21.15.148:0/289794697 learned_addr learned my addr 172.21.15.148:0/289794697 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1c86640 1 --2- 172.21.15.148:0/289794697 >> v2:172.21.15.150:3300/0 conn(0x7fdeac157c80 0x7fdeac161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:30.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928519369 0 0) 0x7fdeac161f60 con 0x7fdeac06fef0 2023-10-30T13:15:30.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde98003530 con 0x7fdeac06fef0 2023-10-30T13:15:30.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2850030933 0 0) 0x7fde98003530 con 0x7fdeac06fef0 2023-10-30T13:15:30.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 >> v2:172.21.15.150:3300/0 conn(0x7fdeac157c80 msgr2=0x7fdeac161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:30.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 --2- 172.21.15.148:0/289794697 >> v2:172.21.15.150:3300/0 conn(0x7fdeac157c80 0x7fdeac161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 >> v1:172.21.15.150:6789/0 conn(0x7fdeac06ab20 legacy=0x7fdeac070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:30.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac1619a0 con 0x7fdeac06fef0 2023-10-30T13:15:30.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fde9c001680 con 0x7fdeac06fef0 2023-10-30T13:15:30.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 >> v1:172.21.15.148:6789/0 conn(0x7fdeac06fef0 legacy=0x7fdeac06f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:30.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde98003980 0x7fde98003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:30.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde98004380 0x7fde98004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:30.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98005050 0x7fde9800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:30.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9800bed0 con 0x7fde98004380 2023-10-30T13:15:30.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9800c040 con 0x7fde98003980 2023-10-30T13:15:30.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9800c200 con 0x7fde98005050 2023-10-30T13:15:30.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1485640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde98004380 0x7fde98004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:30.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1c86640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde98003980 0x7fde98003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:30.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1485640 1 -- 172.21.15.148:0/289794697 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98005050 msgr2=0x7fde9800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:30.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb2487640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98005050 0x7fde9800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:30.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1485640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98005050 0x7fde9800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1485640 1 -- 172.21.15.148:0/289794697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde98003980 msgr2=0x7fde98003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:30.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1485640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde98003980 0x7fde98003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb1485640 1 -- 172.21.15.148:0/289794697 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde98003530 con 0x7fde98004380 2023-10-30T13:15:30.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb2487640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98005050 0x7fde9800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:15:30.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.254+0000 7fdeb2487640 1 -- 172.21.15.148:0/289794697 reap_dead start 2023-10-30T13:15:30.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb1485640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde98004380 0x7fde98004850 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0702f0 tx=0x7fde9c01f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d73f09937ad2d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:30.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c005ae0 con 0x7fde98004380 2023-10-30T13:15:30.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9c0ab070 con 0x7fde98004380 2023-10-30T13:15:30.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb0c84640 1 -- 172.21.15.148:0/289794697 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c007070 con 0x7fde98004380 2023-10-30T13:15:30.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb2c88640 1 -- 172.21.15.148:0/289794697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde98004380 msgr2=0x7fde98004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:30.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb2c88640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde98004380 0x7fde98004850 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdeac0702f0 tx=0x7fde9c01f370 comp rx=0 tx=0).stop 2023-10-30T13:15:30.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb2c88640 1 -- 172.21.15.148:0/289794697 shutdown_connections 2023-10-30T13:15:30.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb2c88640 1 --2- 172.21.15.148:0/289794697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde98004380 0x7fde98004850 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb2c88640 1 -- 172.21.15.148:0/289794697 >> 172.21.15.148:0/289794697 conn(0x7fdeac0c75d0 msgr2=0x7fdeac0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:30.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb2c88640 1 -- 172.21.15.148:0/289794697 shutdown_connections 2023-10-30T13:15:30.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb2c88640 1 -- 172.21.15.148:0/289794697 wait complete. 2023-10-30T13:15:30.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.258+0000 7fdeb2c88640 1 Processor -- start 2023-10-30T13:15:30.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 -- start start 2023-10-30T13:15:30.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98004380 0x7fdeac15b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:30.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeac151080 0x7fdeac1f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:30.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdeac1514e0 0x7fdeac1f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:30.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac15be50 con 0x7fdeac1514e0 2023-10-30T13:15:30.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac15c030 con 0x7fdeac151080 2023-10-30T13:15:30.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac1f6440 con 0x7fde98004380 2023-10-30T13:15:30.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2487640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdeac1514e0 0x7fdeac1f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:30.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2487640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdeac1514e0 0x7fdeac1f6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37130/0 (socket says 172.21.15.148:37130) 2023-10-30T13:15:30.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2487640 1 -- 172.21.15.148:0/3953611472 learned_addr learned my addr 172.21.15.148:0/3953611472 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:30.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb1c86640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98004380 0x7fdeac15b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:30.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb1485640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeac151080 0x7fdeac1f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:30.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2487640 1 -- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98004380 msgr2=0x7fdeac15b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:30.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2487640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98004380 0x7fdeac15b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2487640 1 -- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeac151080 msgr2=0x7fdeac1f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:30.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2487640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeac151080 0x7fdeac1f6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2487640 1 -- 172.21.15.148:0/3953611472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac0c4b90 con 0x7fdeac1514e0 2023-10-30T13:15:30.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2487640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdeac1514e0 0x7fdeac1f6dd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdeac06b3d0 tx=0x7fdea400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:30.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdea27fc640 1 -- 172.21.15.148:0/3953611472 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4003d90 con 0x7fdeac1514e0 2023-10-30T13:15:30.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdea27fc640 1 -- 172.21.15.148:0/3953611472 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea4005070 con 0x7fdeac1514e0 2023-10-30T13:15:30.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdea27fc640 1 -- 172.21.15.148:0/3953611472 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea4005420 con 0x7fdeac1514e0 2023-10-30T13:15:30.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeac15d4e0 con 0x7fdeac1514e0 2023-10-30T13:15:30.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac15d9f0 con 0x7fdeac1514e0 2023-10-30T13:15:30.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.262+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeac1f65b0 con 0x7fdeac1514e0 2023-10-30T13:15:30.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.266+0000 7fdea27fc640 1 -- 172.21.15.148:0/3953611472 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdea4005580 con 0x7fdeac1514e0 2023-10-30T13:15:30.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.266+0000 7fdea27fc640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde900664a0 0x7fde90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:30.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.266+0000 7fdea27fc640 1 -- 172.21.15.148:0/3953611472 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fdea4083910 con 0x7fdeac1514e0 2023-10-30T13:15:30.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.266+0000 7fdeb1c86640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde900664a0 0x7fde90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:30.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.266+0000 7fdeb1c86640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde900664a0 0x7fde90068960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009c40 tx=0x7fdea8006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:30.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.270+0000 7fdea27fc640 1 -- 172.21.15.148:0/3953611472 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdeac1f65b0 con 0x7fdeac1514e0 2023-10-30T13:15:30.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.454+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdeac070460 con 0x7fdeac1514e0 2023-10-30T13:15:30.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdea27fc640 1 -- 172.21.15.148:0/3953611472 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fdea4018070 con 0x7fdeac1514e0 2023-10-30T13:15:30.462 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:30.463 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:24.394103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:resolve","state_seq":10,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:30.463 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 21 2023-10-30T13:15:30.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde900664a0 msgr2=0x7fde90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:30.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde900664a0 0x7fde90068960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009c40 tx=0x7fdea8006f90 comp rx=0 tx=0).stop 2023-10-30T13:15:30.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdeac1514e0 msgr2=0x7fdeac1f6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:30.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdeac1514e0 0x7fdeac1f6dd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdeac06b3d0 tx=0x7fdea400dac0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 shutdown_connections 2023-10-30T13:15:30.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde900664a0 0x7fde90068960 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdeac1514e0 0x7fdeac1f6dd0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeac151080 0x7fdeac1f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 --2- 172.21.15.148:0/3953611472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde98004380 0x7fdeac15b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:30.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 >> 172.21.15.148:0/3953611472 conn(0x7fdeac0c75d0 msgr2=0x7fdeac0c9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:30.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 shutdown_connections 2023-10-30T13:15:30.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:30.458+0000 7fdeb2c88640 1 -- 172.21.15.148:0/3953611472 wait complete. 2023-10-30T13:15:30.484 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:32.485 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:32.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.618+0000 7f67789d3640 1 Processor -- start 2023-10-30T13:15:32.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.618+0000 7f67789d3640 1 -- start start 2023-10-30T13:15:32.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f67789d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6774153ba0 0x7f677415a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:32.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f67789d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f67740c74b0 0x7f6774069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:32.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f67789d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6774153f80 con 0x7f67740c7df0 2023-10-30T13:15:32.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f67789d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67740c1f50 con 0x7f67740c74b0 2023-10-30T13:15:32.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f67789d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67740b0e60 con 0x7f6774153ba0 2023-10-30T13:15:32.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f676bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f67740c74b0 0x7f6774069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:32.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f6772d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f67740c7df0 0x7f677415cdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49212/0 (socket says 172.21.15.148:49212) 2023-10-30T13:15:32.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f6772d9d640 1 -- 172.21.15.148:0/1790117319 learned_addr learned my addr 172.21.15.148:0/1790117319 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:32.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f677259c640 1 --2- 172.21.15.148:0/1790117319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6774153ba0 0x7f677415a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:32.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f676bfff640 1 -- 172.21.15.148:0/1790117319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6774153ba0 msgr2=0x7f677415a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:32.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f676bfff640 1 --2- 172.21.15.148:0/1790117319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6774153ba0 0x7f677415a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f676bfff640 1 -- 172.21.15.148:0/1790117319 >> v1:172.21.15.148:6789/0 conn(0x7f67740c7df0 legacy=0x7f677415cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:32.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.622+0000 7f676bfff640 1 -- 172.21.15.148:0/1790117319 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740b1140 con 0x7f67740c74b0 2023-10-30T13:15:32.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f676bfff640 1 --2- 172.21.15.148:0/1790117319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 0x7f6774069e90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f675c0097c0 tx=0x7f675c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3921791aa63dcb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:32.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f6771d9b640 1 -- 172.21.15.148:0/1790117319 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c0a8970 con 0x7f67740c74b0 2023-10-30T13:15:32.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f6771d9b640 1 -- 172.21.15.148:0/1790117319 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c0a8ad0 con 0x7f67740c74b0 2023-10-30T13:15:32.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f6771d9b640 1 -- 172.21.15.148:0/1790117319 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c0a8e60 con 0x7f67740c74b0 2023-10-30T13:15:32.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 -- 172.21.15.148:0/1790117319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 msgr2=0x7f6774069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:32.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 --2- 172.21.15.148:0/1790117319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 0x7f6774069e90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f675c0097c0 tx=0x7f675c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:15:32.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 -- 172.21.15.148:0/1790117319 shutdown_connections 2023-10-30T13:15:32.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 --2- 172.21.15.148:0/1790117319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 0x7f6774069e90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 --2- 172.21.15.148:0/1790117319 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6774153ba0 0x7f677415a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 -- 172.21.15.148:0/1790117319 >> 172.21.15.148:0/1790117319 conn(0x7f677406d770 msgr2=0x7f677406db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:32.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 -- 172.21.15.148:0/1790117319 shutdown_connections 2023-10-30T13:15:32.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 -- 172.21.15.148:0/1790117319 wait complete. 2023-10-30T13:15:32.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 Processor -- start 2023-10-30T13:15:32.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.626+0000 7f67789d3640 1 -- start start 2023-10-30T13:15:32.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f67789d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 0x7f67741562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:32.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f67789d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740c7df0 0x7f67741f2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:32.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f67789d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6774153ba0 0x7f67740c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:32.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f67789d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6774156820 con 0x7f67740c74b0 2023-10-30T13:15:32.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f67789d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67741f2dc0 con 0x7f6774153ba0 2023-10-30T13:15:32.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f67789d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67740c3fd0 con 0x7f67740c7df0 2023-10-30T13:15:32.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f677259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 0x7f67741562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:32.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f677259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 0x7f67741562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54522/0 (socket says 172.21.15.148:54522) 2023-10-30T13:15:32.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f677259c640 1 -- 172.21.15.148:0/645677090 learned_addr learned my addr 172.21.15.148:0/645677090 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:32.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f676bfff640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740c7df0 0x7f67741f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:32.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f6772d9d640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6774153ba0 0x7f67740c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:32.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f677259c640 1 -- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740c7df0 msgr2=0x7f67741f2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:32.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f677259c640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740c7df0 0x7f67741f2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f677259c640 1 -- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6774153ba0 msgr2=0x7f67740c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:32.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f677259c640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6774153ba0 0x7f67740c3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f677259c640 1 -- 172.21.15.148:0/645677090 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c009420 con 0x7f67740c74b0 2023-10-30T13:15:32.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f6772d9d640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6774153ba0 0x7f67740c3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:32.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f677259c640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 0x7f67741562e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f676400d5f0 tx=0x7f676400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:32.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f676affd640 1 -- 172.21.15.148:0/645677090 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67640125f0 con 0x7f67740c74b0 2023-10-30T13:15:32.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f676affd640 1 -- 172.21.15.148:0/645677090 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6764012750 con 0x7f67740c74b0 2023-10-30T13:15:32.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f676affd640 1 -- 172.21.15.148:0/645677090 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6764012ae0 con 0x7f67740c74b0 2023-10-30T13:15:32.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67741ee950 con 0x7f67740c74b0 2023-10-30T13:15:32.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67741eeef0 con 0x7f67740c74b0 2023-10-30T13:15:32.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6774074250 con 0x7f67740c74b0 2023-10-30T13:15:32.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.630+0000 7f676affd640 1 -- 172.21.15.148:0/645677090 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6764012c40 con 0x7f67740c74b0 2023-10-30T13:15:32.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.634+0000 7f676affd640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67540664a0 0x7f6754068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:32.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.634+0000 7f676affd640 1 -- 172.21.15.148:0/645677090 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f6764084460 con 0x7f67740c74b0 2023-10-30T13:15:32.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.634+0000 7f676bfff640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67540664a0 0x7f6754068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:32.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.634+0000 7f676bfff640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67540664a0 0x7f6754068960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f675c0b2000 tx=0x7f675c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:32.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.638+0000 7f676affd640 1 -- 172.21.15.148:0/645677090 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6764051700 con 0x7f67740c74b0 2023-10-30T13:15:32.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.818+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f677404e170 con 0x7f67740c74b0 2023-10-30T13:15:32.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f676affd640 1 -- 172.21.15.148:0/645677090 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f6764054d20 con 0x7f67740c74b0 2023-10-30T13:15:32.827 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:32.827 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:30.388924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:32.827 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 22 2023-10-30T13:15:32.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67540664a0 msgr2=0x7f6754068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:32.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67540664a0 0x7f6754068960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f675c0b2000 tx=0x7f675c008040 comp rx=0 tx=0).stop 2023-10-30T13:15:32.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 msgr2=0x7f67741562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:32.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 0x7f67741562e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f676400d5f0 tx=0x7f676400dab0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 shutdown_connections 2023-10-30T13:15:32.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67540664a0 0x7f6754068960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6774153ba0 0x7f67740c3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67740c7df0 0x7f67741f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 --2- 172.21.15.148:0/645677090 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67740c74b0 0x7f67741562e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:32.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 >> 172.21.15.148:0/645677090 conn(0x7f677406d770 msgr2=0x7f677406db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:32.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 shutdown_connections 2023-10-30T13:15:32.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:32.822+0000 7f67789d3640 1 -- 172.21.15.148:0/645677090 wait complete. 2023-10-30T13:15:32.847 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4270": { "addr": "172.21.15.150:6836/1620877178", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1620877178, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 1620877178, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4270, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4285": { "addr": "172.21.15.148:6834/4162935019", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 4162935019, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 4162935019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4959": { "addr": "172.21.15.148:6837/95380877", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 95380877, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 95380877, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4959, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:15:30.388924+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4270, "mds_2": 4959 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:15:34.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:34.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.990+0000 7fda48617640 1 Processor -- start 2023-10-30T13:15:34.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.990+0000 7fda48617640 1 -- start start 2023-10-30T13:15:35.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda48617640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fda400c05d0 0x7fda4013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:35.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda48617640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fda400c6f60 0x7fda4013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:35.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda48617640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda400c0a50 con 0x7fda400c6b80 2023-10-30T13:15:35.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda48617640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4006f6e0 con 0x7fda400c05d0 2023-10-30T13:15:35.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda48617640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4006f9c0 con 0x7fda400c6f60 2023-10-30T13:15:35.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda45bb2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fda400c05d0 0x7fda4013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:35.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda45bb2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fda400c05d0 0x7fda4013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54524/0 (socket says 172.21.15.148:54524) 2023-10-30T13:15:35.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda45bb2640 1 -- 172.21.15.148:0/2857898590 learned_addr learned my addr 172.21.15.148:0/2857898590 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:35.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda46bb4640 1 --2- 172.21.15.148:0/2857898590 >> v2:172.21.15.150:3300/0 conn(0x7fda400c6f60 0x7fda4013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:35.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda45bb2640 1 -- 172.21.15.148:0/2857898590 >> v2:172.21.15.150:3300/0 conn(0x7fda400c6f60 msgr2=0x7fda4013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:35.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda45bb2640 1 --2- 172.21.15.148:0/2857898590 >> v2:172.21.15.150:3300/0 conn(0x7fda400c6f60 0x7fda4013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:35.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda45bb2640 1 -- 172.21.15.148:0/2857898590 >> v1:172.21.15.150:6789/0 conn(0x7fda400c6b80 legacy=0x7fda400bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:35.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.994+0000 7fda45bb2640 1 -- 172.21.15.148:0/2857898590 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4006fca0 con 0x7fda400c05d0 2023-10-30T13:15:35.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda45bb2640 1 --2- 172.21.15.148:0/2857898590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c05d0 0x7fda4013ed90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fda400c89b0 tx=0x7fda3001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3daf12726b0efe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:35.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda453b1640 1 -- 172.21.15.148:0/2857898590 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda300ab610 con 0x7fda400c05d0 2023-10-30T13:15:35.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda453b1640 1 -- 172.21.15.148:0/2857898590 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda300ab770 con 0x7fda400c05d0 2023-10-30T13:15:35.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda453b1640 1 -- 172.21.15.148:0/2857898590 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda300abb00 con 0x7fda400c05d0 2023-10-30T13:15:35.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 -- 172.21.15.148:0/2857898590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c05d0 msgr2=0x7fda4013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:35.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 --2- 172.21.15.148:0/2857898590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c05d0 0x7fda4013ed90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fda400c89b0 tx=0x7fda3001c500 comp rx=0 tx=0).stop 2023-10-30T13:15:35.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 -- 172.21.15.148:0/2857898590 shutdown_connections 2023-10-30T13:15:35.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 --2- 172.21.15.148:0/2857898590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c05d0 0x7fda4013ed90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:35.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 --2- 172.21.15.148:0/2857898590 >> v2:172.21.15.150:3300/0 conn(0x7fda400c6f60 0x7fda4013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:35.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 -- 172.21.15.148:0/2857898590 >> 172.21.15.148:0/2857898590 conn(0x7fda400c7910 msgr2=0x7fda400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:35.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 -- 172.21.15.148:0/2857898590 shutdown_connections 2023-10-30T13:15:35.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 -- 172.21.15.148:0/2857898590 wait complete. 2023-10-30T13:15:35.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 Processor -- start 2023-10-30T13:15:35.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:34.998+0000 7fda48617640 1 -- start start 2023-10-30T13:15:35.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda48617640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda400c05d0 0x7fda401f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:35.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda48617640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c6b80 0x7fda400c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:35.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda48617640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda400c6f60 0x7fda401f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:35.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda48617640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda401f2670 con 0x7fda400c6b80 2023-10-30T13:15:35.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda48617640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4015f9c0 con 0x7fda400c05d0 2023-10-30T13:15:35.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda48617640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4015fef0 con 0x7fda400c6f60 2023-10-30T13:15:35.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda45bb2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c6b80 0x7fda400c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:35.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda45bb2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c6b80 0x7fda400c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54540/0 (socket says 172.21.15.148:54540) 2023-10-30T13:15:35.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda45bb2640 1 -- 172.21.15.148:0/1879606286 learned_addr learned my addr 172.21.15.148:0/1879606286 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:35.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda46bb4640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda400c6f60 0x7fda401f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:35.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda463b3640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda400c05d0 0x7fda401f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:35.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda45bb2640 1 -- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda400c6f60 msgr2=0x7fda401f5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:35.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda45bb2640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda400c6f60 0x7fda401f5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:35.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda45bb2640 1 -- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda400c05d0 msgr2=0x7fda401f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda45bb2640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda400c05d0 0x7fda401f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda45bb2640 1 -- 172.21.15.148:0/1879606286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda300094d0 con 0x7fda400c6b80 2023-10-30T13:15:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda45bb2640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c6b80 0x7fda400c9650 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fda30019110 tx=0x7fda3001ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda36ffd640 1 -- 172.21.15.148:0/1879606286 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda300b4800 con 0x7fda400c6b80 2023-10-30T13:15:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda36ffd640 1 -- 172.21.15.148:0/1879606286 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda300b4960 con 0x7fda400c6b80 2023-10-30T13:15:35.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda36ffd640 1 -- 172.21.15.148:0/1879606286 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda300b4d10 con 0x7fda400c6b80 2023-10-30T13:15:35.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4015fc10 con 0x7fda400c6b80 2023-10-30T13:15:35.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda401f2290 con 0x7fda400c6b80 2023-10-30T13:15:35.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.002+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda401f28e0 con 0x7fda400c6b80 2023-10-30T13:15:35.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.006+0000 7fda36ffd640 1 -- 172.21.15.148:0/1879606286 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda300b2410 con 0x7fda400c6b80 2023-10-30T13:15:35.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.006+0000 7fda36ffd640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda240664a0 0x7fda24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:35.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.006+0000 7fda36ffd640 1 -- 172.21.15.148:0/1879606286 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fda3011e7c0 con 0x7fda400c6b80 2023-10-30T13:15:35.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.006+0000 7fda463b3640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda240664a0 0x7fda24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:35.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.006+0000 7fda463b3640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda240664a0 0x7fda24068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fda3c000f70 tx=0x7fda3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:35.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.010+0000 7fda36ffd640 1 -- 172.21.15.148:0/1879606286 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda300ebb10 con 0x7fda400c6b80 2023-10-30T13:15:35.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda4004e170 con 0x7fda400c6b80 2023-10-30T13:15:35.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda36ffd640 1 -- 172.21.15.148:0/1879606286 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5744 (secure 0 0 0) 0x7fda300ef130 con 0x7fda400c6b80 2023-10-30T13:15:35.194 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:35.195 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:30.388924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:35.195 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 22 2023-10-30T13:15:35.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda240664a0 msgr2=0x7fda24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:35.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda240664a0 0x7fda24068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fda3c000f70 tx=0x7fda3c009040 comp rx=0 tx=0).stop 2023-10-30T13:15:35.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c6b80 msgr2=0x7fda400c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:35.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c6b80 0x7fda400c9650 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fda30019110 tx=0x7fda3001ad40 comp rx=0 tx=0).stop 2023-10-30T13:15:35.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 shutdown_connections 2023-10-30T13:15:35.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda240664a0 0x7fda24068960 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:35.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda400c6b80 0x7fda400c9650 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:35.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda400c6f60 0x7fda401f5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:35.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 --2- 172.21.15.148:0/1879606286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda400c05d0 0x7fda401f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:35.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 >> 172.21.15.148:0/1879606286 conn(0x7fda400c7910 msgr2=0x7fda400c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:35.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 shutdown_connections 2023-10-30T13:15:35.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:35.190+0000 7fda48617640 1 -- 172.21.15.148:0/1879606286 wait complete. 2023-10-30T13:15:35.215 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:37.216 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:37.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f4274046640 1 Processor -- start 2023-10-30T13:15:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f4274046640 1 -- start start 2023-10-30T13:15:37.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f4274046640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f426c0c93d0 0x7f426c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f4274046640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c06afa0 con 0x7f426c06e470 2023-10-30T13:15:37.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f4274046640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c154150 con 0x7f426c153d70 2023-10-30T13:15:37.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f4274046640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426c06d780 con 0x7f426c0c93d0 2023-10-30T13:15:37.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f42715e1640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f426c06e470 0x7f426c0c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49226/0 (socket says 172.21.15.148:49226) 2023-10-30T13:15:37.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f42715e1640 1 -- 172.21.15.148:0/1367889593 learned_addr learned my addr 172.21.15.148:0/1367889593 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:37.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f42725e3640 1 --2- 172.21.15.148:0/1367889593 >> v2:172.21.15.150:3300/0 conn(0x7f426c0c93d0 0x7f426c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.350+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205077050 0 0) 0x7f426c06afa0 con 0x7f426c06e470 2023-10-30T13:15:37.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4258003530 con 0x7f426c06e470 2023-10-30T13:15:37.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4105276731 0 0) 0x7f4258003530 con 0x7f426c06e470 2023-10-30T13:15:37.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 >> v2:172.21.15.150:3300/0 conn(0x7f426c0c93d0 msgr2=0x7f426c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:37.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 --2- 172.21.15.148:0/1367889593 >> v2:172.21.15.150:3300/0 conn(0x7f426c0c93d0 0x7f426c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 >> v1:172.21.15.150:6789/0 conn(0x7f426c153d70 legacy=0x7f426c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c06e030 con 0x7f426c06e470 2023-10-30T13:15:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f425c001680 con 0x7f426c06e470 2023-10-30T13:15:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 >> v1:172.21.15.148:6789/0 conn(0x7f426c06e470 legacy=0x7f426c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258003980 0x7f4258003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4258004380 0x7f4258004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4258005050 0x7f425800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c06d780 con 0x7f4258005050 2023-10-30T13:15:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425800bfb0 con 0x7f4258003980 2023-10-30T13:15:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425800c170 con 0x7f4258004380 2023-10-30T13:15:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f42725e3640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4258005050 0x7f425800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f42715e1640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4258004380 0x7f4258004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4271de2640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258003980 0x7f4258003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f42725e3640 1 -- 172.21.15.148:0/1367889593 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4258004380 msgr2=0x7f4258004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f42725e3640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4258004380 0x7f4258004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f42725e3640 1 -- 172.21.15.148:0/1367889593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258003980 msgr2=0x7f4258003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f42725e3640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258003980 0x7f4258003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f42725e3640 1 -- 172.21.15.148:0/1367889593 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4258003530 con 0x7f4258005050 2023-10-30T13:15:37.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f4271de2640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258003980 0x7f4258003d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:37.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.354+0000 7f42725e3640 1 -- 172.21.15.148:0/1367889593 reap_dead start 2023-10-30T13:15:37.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f42725e3640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4258005050 0x7f425800b990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4264009fc0 tx=0x7f426401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd02bfc0caeea3fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:37.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c007ea0 con 0x7f4258005050 2023-10-30T13:15:37.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426400f040 con 0x7f4258005050 2023-10-30T13:15:37.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4270de0640 1 -- 172.21.15.148:0/1367889593 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42640039a0 con 0x7f4258005050 2023-10-30T13:15:37.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 -- 172.21.15.148:0/1367889593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4258005050 msgr2=0x7f425800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4258005050 0x7f425800b990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4264009fc0 tx=0x7f426401cca0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 -- 172.21.15.148:0/1367889593 shutdown_connections 2023-10-30T13:15:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 --2- 172.21.15.148:0/1367889593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4258005050 0x7f425800b990 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 -- 172.21.15.148:0/1367889593 >> 172.21.15.148:0/1367889593 conn(0x7f426c071ea0 msgr2=0x7f426c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 -- 172.21.15.148:0/1367889593 shutdown_connections 2023-10-30T13:15:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 -- 172.21.15.148:0/1367889593 wait complete. 2023-10-30T13:15:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 Processor -- start 2023-10-30T13:15:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 -- start start 2023-10-30T13:15:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258005050 0x7f426c1f0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1eff10 0x7f426c1f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c15d8b0 0x7f426c151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c1efaf0 con 0x7f426c15d8b0 2023-10-30T13:15:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c1f0c40 con 0x7f4258005050 2023-10-30T13:15:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.358+0000 7f4274046640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c1f3720 con 0x7f426c1eff10 2023-10-30T13:15:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42725e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c15d8b0 0x7f426c151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42725e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c15d8b0 0x7f426c151a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54562/0 (socket says 172.21.15.148:54562) 2023-10-30T13:15:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42725e3640 1 -- 172.21.15.148:0/241304017 learned_addr learned my addr 172.21.15.148:0/241304017 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42715e1640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1eff10 0x7f426c1f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f4271de2640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258005050 0x7f426c1f0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42725e3640 1 -- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1eff10 msgr2=0x7f426c1f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42725e3640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1eff10 0x7f426c1f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42725e3640 1 -- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258005050 msgr2=0x7f426c1f0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42725e3640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258005050 0x7f426c1f0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42725e3640 1 -- 172.21.15.148:0/241304017 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c1462b0 con 0x7f426c15d8b0 2023-10-30T13:15:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42725e3640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c15d8b0 0x7f426c151a10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f426401cce0 tx=0x7f4264009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42627fc640 1 -- 172.21.15.148:0/241304017 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264003db0 con 0x7f426c15d8b0 2023-10-30T13:15:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42627fc640 1 -- 172.21.15.148:0/241304017 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42640a5b70 con 0x7f426c15d8b0 2023-10-30T13:15:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42627fc640 1 -- 172.21.15.148:0/241304017 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42640ae960 con 0x7f426c15d8b0 2023-10-30T13:15:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426c145cf0 con 0x7f426c15d8b0 2023-10-30T13:15:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c1ef430 con 0x7f426c15d8b0 2023-10-30T13:15:37.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c1ef290 con 0x7f426c15d8b0 2023-10-30T13:15:37.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42627fc640 1 -- 172.21.15.148:0/241304017 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42640aeac0 con 0x7f426c15d8b0 2023-10-30T13:15:37.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42627fc640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42500664d0 0x7f4250068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.362+0000 7f42627fc640 1 -- 172.21.15.148:0/241304017 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f426411f1e0 con 0x7f426c15d8b0 2023-10-30T13:15:37.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.366+0000 7f4271de2640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42500664d0 0x7f4250068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.366+0000 7f4271de2640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42500664d0 0x7f4250068990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f42680054c0 tx=0x7f4268009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:37.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.366+0000 7f42627fc640 1 -- 172.21.15.148:0/241304017 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f426c1ef290 con 0x7f426c15d8b0 2023-10-30T13:15:37.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.566+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f426c1ef290 con 0x7f426c15d8b0 2023-10-30T13:15:37.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.566+0000 7f42627fc640 1 -- 172.21.15.148:0/241304017 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f426c1ef290 con 0x7f426c15d8b0 2023-10-30T13:15:37.573 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:37.573 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:35.670674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:37.574 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 23 2023-10-30T13:15:37.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42500664d0 msgr2=0x7f4250068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42500664d0 0x7f4250068990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f42680054c0 tx=0x7f4268009040 comp rx=0 tx=0).stop 2023-10-30T13:15:37.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c15d8b0 msgr2=0x7f426c151a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c15d8b0 0x7f426c151a10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f426401cce0 tx=0x7f4264009750 comp rx=0 tx=0).stop 2023-10-30T13:15:37.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 shutdown_connections 2023-10-30T13:15:37.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42500664d0 0x7f4250068990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c15d8b0 0x7f426c151a10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1eff10 0x7f426c1f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 --2- 172.21.15.148:0/241304017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258005050 0x7f426c1f0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 >> 172.21.15.148:0/241304017 conn(0x7f426c071ea0 msgr2=0x7f426c06ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:37.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 shutdown_connections 2023-10-30T13:15:37.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:37.570+0000 7f4274046640 1 -- 172.21.15.148:0/241304017 wait complete. 2023-10-30T13:15:37.593 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:39.595 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:39.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.738+0000 7fcf5b301640 1 Processor -- start 2023-10-30T13:15:39.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.738+0000 7fcf5b301640 1 -- start start 2023-10-30T13:15:39.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5b301640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf541539a0 0x7fcf5415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5b301640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 0x7fcf54070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5b301640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5415a8c0 con 0x7fcf54069e90 2023-10-30T13:15:39.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5b301640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf5406fbb0 con 0x7fcf541539a0 2023-10-30T13:15:39.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5b301640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf54071380 con 0x7fcf54070a00 2023-10-30T13:15:39.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5909d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf541539a0 0x7fcf5415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5989e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcf54069e90 0x7fcf5406a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:54452/0 (socket says 172.21.15.148:54452) 2023-10-30T13:15:39.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5989e640 1 -- 172.21.15.148:0/2812501949 learned_addr learned my addr 172.21.15.148:0/2812501949 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:39.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5889c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 0x7fcf54070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf4bfff640 1 -- 172.21.15.148:0/2812501949 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827736646 0 0) 0x7fcf5415a8c0 con 0x7fcf54069e90 2023-10-30T13:15:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf4bfff640 1 -- 172.21.15.148:0/2812501949 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf40003530 con 0x7fcf54069e90 2023-10-30T13:15:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5889c640 1 -- 172.21.15.148:0/2812501949 >> v2:172.21.15.150:3300/0 conn(0x7fcf541539a0 msgr2=0x7fcf5415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5889c640 1 --2- 172.21.15.148:0/2812501949 >> v2:172.21.15.150:3300/0 conn(0x7fcf541539a0 0x7fcf5415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5889c640 1 -- 172.21.15.148:0/2812501949 >> v1:172.21.15.150:6789/0 conn(0x7fcf54069e90 legacy=0x7fcf5406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.742+0000 7fcf5889c640 1 -- 172.21.15.148:0/2812501949 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf54153d80 con 0x7fcf54070a00 2023-10-30T13:15:39.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5889c640 1 --2- 172.21.15.148:0/2812501949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 0x7fcf54070e40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcf440097c0 tx=0x7fcf4401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48ba5c72632ea1cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:39.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf4bfff640 1 -- 172.21.15.148:0/2812501949 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf440ac620 con 0x7fcf54070a00 2023-10-30T13:15:39.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf4bfff640 1 -- 172.21.15.148:0/2812501949 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf440ac780 con 0x7fcf54070a00 2023-10-30T13:15:39.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf4bfff640 1 -- 172.21.15.148:0/2812501949 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf440acb10 con 0x7fcf54070a00 2023-10-30T13:15:39.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5b301640 1 -- 172.21.15.148:0/2812501949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 msgr2=0x7fcf54070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5b301640 1 --2- 172.21.15.148:0/2812501949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 0x7fcf54070e40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcf440097c0 tx=0x7fcf4401c3c0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5b301640 1 -- 172.21.15.148:0/2812501949 shutdown_connections 2023-10-30T13:15:39.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5b301640 1 --2- 172.21.15.148:0/2812501949 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 0x7fcf54070e40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5b301640 1 --2- 172.21.15.148:0/2812501949 >> v2:172.21.15.150:3300/0 conn(0x7fcf541539a0 0x7fcf5415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5b301640 1 -- 172.21.15.148:0/2812501949 >> 172.21.15.148:0/2812501949 conn(0x7fcf540c7910 msgr2=0x7fcf540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:39.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5b301640 1 -- 172.21.15.148:0/2812501949 shutdown_connections 2023-10-30T13:15:39.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5b301640 1 -- 172.21.15.148:0/2812501949 wait complete. 2023-10-30T13:15:39.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.746+0000 7fcf5b301640 1 Processor -- start 2023-10-30T13:15:39.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 -- start start 2023-10-30T13:15:39.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54069e90 0x7fcf54156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 0x7fcf541f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf541539a0 0x7fcf5414f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf541f5950 con 0x7fcf541539a0 2023-10-30T13:15:39.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf54157310 con 0x7fcf54069e90 2023-10-30T13:15:39.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540c3a80 con 0x7fcf54070a00 2023-10-30T13:15:39.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5989e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf541539a0 0x7fcf5414f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5989e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf541539a0 0x7fcf5414f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54572/0 (socket says 172.21.15.148:54572) 2023-10-30T13:15:39.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5989e640 1 -- 172.21.15.148:0/30545982 learned_addr learned my addr 172.21.15.148:0/30545982 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:39.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5889c640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 0x7fcf541f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5909d640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54069e90 0x7fcf54156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5989e640 1 -- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 msgr2=0x7fcf541f5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5989e640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 0x7fcf541f5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5989e640 1 -- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54069e90 msgr2=0x7fcf54156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5989e640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54069e90 0x7fcf54156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5989e640 1 -- 172.21.15.148:0/30545982 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf44009420 con 0x7fcf541539a0 2023-10-30T13:15:39.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5909d640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54069e90 0x7fcf54156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:39.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5989e640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf541539a0 0x7fcf5414f360 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00cb40 tx=0x7fcf4c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:39.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf49ffb640 1 -- 172.21.15.148:0/30545982 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c00ac40 con 0x7fcf541539a0 2023-10-30T13:15:39.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf49ffb640 1 -- 172.21.15.148:0/30545982 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4c00ada0 con 0x7fcf541539a0 2023-10-30T13:15:39.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf54159810 con 0x7fcf541539a0 2023-10-30T13:15:39.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf49ffb640 1 -- 172.21.15.148:0/30545982 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c012620 con 0x7fcf541539a0 2023-10-30T13:15:39.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf541f5570 con 0x7fcf541539a0 2023-10-30T13:15:39.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.750+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf541f5a90 con 0x7fcf541539a0 2023-10-30T13:15:39.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.754+0000 7fcf49ffb640 1 -- 172.21.15.148:0/30545982 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcf4c007070 con 0x7fcf541539a0 2023-10-30T13:15:39.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.754+0000 7fcf49ffb640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf380664a0 0x7fcf38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.754+0000 7fcf49ffb640 1 -- 172.21.15.148:0/30545982 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fcf4c083fc0 con 0x7fcf541539a0 2023-10-30T13:15:39.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.754+0000 7fcf5909d640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf380664a0 0x7fcf38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.754+0000 7fcf5909d640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf380664a0 0x7fcf38068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009b10 tx=0x7fcf50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:39.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.758+0000 7fcf49ffb640 1 -- 172.21.15.148:0/30545982 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcf541f5a90 con 0x7fcf541539a0 2023-10-30T13:15:39.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf541f5a90 con 0x7fcf541539a0 2023-10-30T13:15:39.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf49ffb640 1 -- 172.21.15.148:0/30545982 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcf541f5a90 con 0x7fcf541539a0 2023-10-30T13:15:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:39.947 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:35.670674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:39.947 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 23 2023-10-30T13:15:39.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf380664a0 msgr2=0x7fcf38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf380664a0 0x7fcf38068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009b10 tx=0x7fcf50008040 comp rx=0 tx=0).stop 2023-10-30T13:15:39.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf541539a0 msgr2=0x7fcf5414f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf541539a0 0x7fcf5414f360 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00cb40 tx=0x7fcf4c00a060 comp rx=0 tx=0).stop 2023-10-30T13:15:39.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 shutdown_connections 2023-10-30T13:15:39.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf380664a0 0x7fcf38068960 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf541539a0 0x7fcf5414f360 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf54070a00 0x7fcf541f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 --2- 172.21.15.148:0/30545982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54069e90 0x7fcf54156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 >> 172.21.15.148:0/30545982 conn(0x7fcf540c7910 msgr2=0x7fcf54150410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:39.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 shutdown_connections 2023-10-30T13:15:39.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:39.942+0000 7fcf5b301640 1 -- 172.21.15.148:0/30545982 wait complete. 2023-10-30T13:15:39.967 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:40.712 INFO:tasks.workunit.client.0.smithi148.stdout:Create dir 200 to 299 ... 2023-10-30T13:15:40.714 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 200 2023-10-30T13:15:40.721 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 201 2023-10-30T13:15:40.732 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 202 2023-10-30T13:15:40.749 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 203 2023-10-30T13:15:40.761 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 204 2023-10-30T13:15:40.771 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 205 2023-10-30T13:15:40.782 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 206 2023-10-30T13:15:40.793 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 207 2023-10-30T13:15:41.053 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 208 2023-10-30T13:15:41.065 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 209 2023-10-30T13:15:41.076 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 210 2023-10-30T13:15:41.088 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 211 2023-10-30T13:15:41.099 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 212 2023-10-30T13:15:41.110 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 213 2023-10-30T13:15:41.121 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 214 2023-10-30T13:15:41.132 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 215 2023-10-30T13:15:41.144 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 216 2023-10-30T13:15:41.154 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 217 2023-10-30T13:15:41.166 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 218 2023-10-30T13:15:41.177 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 219 2023-10-30T13:15:41.188 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 220 2023-10-30T13:15:41.199 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 221 2023-10-30T13:15:41.210 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 222 2023-10-30T13:15:41.222 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 223 2023-10-30T13:15:41.481 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 224 2023-10-30T13:15:41.741 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 225 2023-10-30T13:15:41.794 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 226 2023-10-30T13:15:41.805 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 227 2023-10-30T13:15:41.816 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 228 2023-10-30T13:15:41.828 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 229 2023-10-30T13:15:41.839 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 230 2023-10-30T13:15:41.850 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 231 2023-10-30T13:15:41.861 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 232 2023-10-30T13:15:41.876 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 233 2023-10-30T13:15:41.892 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 234 2023-10-30T13:15:41.908 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 235 2023-10-30T13:15:41.923 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 236 2023-10-30T13:15:41.934 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 237 2023-10-30T13:15:41.944 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 238 2023-10-30T13:15:41.968 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:42.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.098+0000 7fee21f94640 1 Processor -- start 2023-10-30T13:15:42.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee21f94640 1 -- start start 2023-10-30T13:15:42.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee21f94640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee1c0c6b80 0x7fee1c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:42.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee21f94640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c0c0a50 con 0x7fee1c13f4a0 2023-10-30T13:15:42.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee21f94640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c06f6e0 con 0x7fee1c0c6f60 2023-10-30T13:15:42.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee21f94640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1c06f9c0 con 0x7fee1c0c6b80 2023-10-30T13:15:42.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee1b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee1c0c6b80 0x7fee1c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:42.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee1b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee1c0c6b80 0x7fee1c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43310/0 (socket says 172.21.15.148:43310) 2023-10-30T13:15:42.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee1b7fe640 1 -- 172.21.15.148:0/100353952 learned_addr learned my addr 172.21.15.148:0/100353952 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee1b7fe640 1 -- 172.21.15.148:0/100353952 >> v1:172.21.15.150:6789/0 conn(0x7fee1c0c6f60 legacy=0x7fee1c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee1b7fe640 1 -- 172.21.15.148:0/100353952 >> v1:172.21.15.148:6789/0 conn(0x7fee1c13f4a0 legacy=0x7fee1c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.102+0000 7fee1b7fe640 1 -- 172.21.15.148:0/100353952 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c06fca0 con 0x7fee1c0c6b80 2023-10-30T13:15:42.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee1b7fe640 1 --2- 172.21.15.148:0/100353952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 0x7fee1c0bfec0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009870 tx=0x7fee0c01e750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50bbb834e587b7bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee1a7fc640 1 -- 172.21.15.148:0/100353952 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c0ab630 con 0x7fee1c0c6b80 2023-10-30T13:15:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee1a7fc640 1 -- 172.21.15.148:0/100353952 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c0ab790 con 0x7fee1c0c6b80 2023-10-30T13:15:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee1a7fc640 1 -- 172.21.15.148:0/100353952 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c0abb20 con 0x7fee1c0c6b80 2023-10-30T13:15:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee21f94640 1 -- 172.21.15.148:0/100353952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 msgr2=0x7fee1c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee21f94640 1 --2- 172.21.15.148:0/100353952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 0x7fee1c0bfec0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009870 tx=0x7fee0c01e750 comp rx=0 tx=0).stop 2023-10-30T13:15:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee21f94640 1 -- 172.21.15.148:0/100353952 shutdown_connections 2023-10-30T13:15:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee21f94640 1 --2- 172.21.15.148:0/100353952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 0x7fee1c0bfec0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:42.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee21f94640 1 -- 172.21.15.148:0/100353952 >> 172.21.15.148:0/100353952 conn(0x7fee1c0c7910 msgr2=0x7fee1c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:42.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee21f94640 1 -- 172.21.15.148:0/100353952 shutdown_connections 2023-10-30T13:15:42.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee21f94640 1 -- 172.21.15.148:0/100353952 wait complete. 2023-10-30T13:15:42.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee21f94640 1 Processor -- start 2023-10-30T13:15:42.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.106+0000 7fee21f94640 1 -- start start 2023-10-30T13:15:42.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee21f94640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 0x7fee1c1f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:42.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee21f94640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee1c0c6f60 0x7fee1c13e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:42.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee21f94640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee1c13f4a0 0x7fee1c0ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:42.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee21f94640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c0c2d70 con 0x7fee1c0c6b80 2023-10-30T13:15:42.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee21f94640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c13df50 con 0x7fee1c0c6f60 2023-10-30T13:15:42.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee21f94640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c13dc20 con 0x7fee1c13f4a0 2023-10-30T13:15:42.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 0x7fee1c1f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:42.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 0x7fee1c1f5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43326/0 (socket says 172.21.15.148:43326) 2023-10-30T13:15:42.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1b7fe640 1 -- 172.21.15.148:0/3127763309 learned_addr learned my addr 172.21.15.148:0/3127763309 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:42.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1affd640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee1c0c6f60 0x7fee1c13e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:42.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1bfff640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee1c13f4a0 0x7fee1c0ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:42.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1b7fe640 1 -- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee1c13f4a0 msgr2=0x7fee1c0ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1b7fe640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee1c13f4a0 0x7fee1c0ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1b7fe640 1 -- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee1c0c6f60 msgr2=0x7fee1c13e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1b7fe640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee1c0c6f60 0x7fee1c13e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1b7fe640 1 -- 172.21.15.148:0/3127763309 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0094d0 con 0x7fee1c0c6b80 2023-10-30T13:15:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1affd640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee1c0c6f60 0x7fee1c13e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:15:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1bfff640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee1c13f4a0 0x7fee1c0ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:15:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee1b7fe640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 0x7fee1c1f5ee0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0ad040 tx=0x7fee0c01cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee03fff640 1 -- 172.21.15.148:0/3127763309 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c0b3800 con 0x7fee1c0c6b80 2023-10-30T13:15:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee03fff640 1 -- 172.21.15.148:0/3127763309 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c0b3980 con 0x7fee1c0c6b80 2023-10-30T13:15:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c0c4850 con 0x7fee1c0c6b80 2023-10-30T13:15:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee03fff640 1 -- 172.21.15.148:0/3127763309 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c0b3d30 con 0x7fee1c0c6b80 2023-10-30T13:15:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c0c4290 con 0x7fee1c0c6b80 2023-10-30T13:15:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.110+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1c13dd90 con 0x7fee1c0c6b80 2023-10-30T13:15:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.114+0000 7fee03fff640 1 -- 172.21.15.148:0/3127763309 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee0c0aa390 con 0x7fee1c0c6b80 2023-10-30T13:15:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.114+0000 7fee03fff640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee080664a0 0x7fee08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.114+0000 7fee03fff640 1 -- 172.21.15.148:0/3127763309 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fee0c11e7e0 con 0x7fee1c0c6b80 2023-10-30T13:15:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.114+0000 7fee1affd640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee080664a0 0x7fee08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.114+0000 7fee1affd640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee080664a0 0x7fee08068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fee1c1f6ff0 tx=0x7fee10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.118+0000 7fee03fff640 1 -- 172.21.15.148:0/3127763309 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee1c13dd90 con 0x7fee1c0c6b80 2023-10-30T13:15:42.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee1c13dd90 con 0x7fee1c0c6b80 2023-10-30T13:15:42.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee03fff640 1 -- 172.21.15.148:0/3127763309 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fee1c13dd90 con 0x7fee1c0c6b80 2023-10-30T13:15:42.303 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:42.303 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:40.696092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:42.303 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 24 2023-10-30T13:15:42.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee080664a0 msgr2=0x7fee08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:42.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee080664a0 0x7fee08068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fee1c1f6ff0 tx=0x7fee10009040 comp rx=0 tx=0).stop 2023-10-30T13:15:42.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 msgr2=0x7fee1c1f5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:42.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 0x7fee1c1f5ee0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0ad040 tx=0x7fee0c01cf90 comp rx=0 tx=0).stop 2023-10-30T13:15:42.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 shutdown_connections 2023-10-30T13:15:42.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee080664a0 0x7fee08068960 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:42.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee1c13f4a0 0x7fee1c0ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:42.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee1c0c6f60 0x7fee1c13e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:42.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 --2- 172.21.15.148:0/3127763309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee1c0c6b80 0x7fee1c1f5ee0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:42.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 >> 172.21.15.148:0/3127763309 conn(0x7fee1c0c7910 msgr2=0x7fee1c06a970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:42.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 shutdown_connections 2023-10-30T13:15:42.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:42.298+0000 7fee21f94640 1 -- 172.21.15.148:0/3127763309 wait complete. 2023-10-30T13:15:42.324 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=4250 2023-10-30T13:15:42.325 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.a 2023-10-30T13:15:42.340 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 239 2023-10-30T13:15:42.351 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 240 2023-10-30T13:15:42.368 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 241 2023-10-30T13:15:42.379 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 242 2023-10-30T13:15:42.395 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 243 2023-10-30T13:15:42.411 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 244 2023-10-30T13:15:42.429 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 245 2023-10-30T13:15:42.698 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 246 2023-10-30T13:15:42.714 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 247 2023-10-30T13:15:42.724 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 248 2023-10-30T13:15:42.734 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 249 2023-10-30T13:15:42.744 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 250 2023-10-30T13:15:42.772 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 251 2023-10-30T13:15:45.006 INFO:tasks.ceph.mds.e.smithi148.stderr:2023-10-30T13:15:45.002+0000 7ff756f04640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:15:45.325 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:15:45.326 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:15:45.326 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:15:45.330 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:15:45.331 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:45.427 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:15:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.474+0000 7ff28251d640 1 Processor -- start 2023-10-30T13:15:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.474+0000 7ff28251d640 1 -- start start 2023-10-30T13:15:45.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff28251d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c0c6b80 0x7ff27c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff28251d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27c0c0a50 con 0x7ff27c13f4a0 2023-10-30T13:15:45.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff28251d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27c06f6e0 con 0x7ff27c0c6f60 2023-10-30T13:15:45.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff28251d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c06f9c0 con 0x7ff27c0c6b80 2023-10-30T13:15:45.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff280aba640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff27c13f4a0 0x7ff27c1566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39794/0 (socket says 172.21.15.148:39794) 2023-10-30T13:15:45.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff280aba640 1 -- 172.21.15.148:0/3258054253 learned_addr learned my addr 172.21.15.148:0/3258054253 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:45.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27bfff640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c0c6b80 0x7ff27c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498991394 0 0) 0x7ff27c0c0a50 con 0x7ff27c13f4a0 2023-10-30T13:15:45.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff264003530 con 0x7ff27c13f4a0 2023-10-30T13:15:45.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943045095 0 0) 0x7ff27c06f6e0 con 0x7ff27c0c6f60 2023-10-30T13:15:45.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff27c0c0a50 con 0x7ff27c0c6f60 2023-10-30T13:15:45.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2276130507 0 0) 0x7ff264003530 con 0x7ff27c13f4a0 2023-10-30T13:15:45.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c0c6b80 msgr2=0x7ff27c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:45.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c0c6b80 0x7ff27c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 >> v1:172.21.15.150:6789/0 conn(0x7ff27c0c6f60 legacy=0x7ff27c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c06fca0 con 0x7ff27c13f4a0 2023-10-30T13:15:45.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7ff274001980 con 0x7ff27c13f4a0 2023-10-30T13:15:45.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 >> v1:172.21.15.148:6789/0 conn(0x7ff27c13f4a0 legacy=0x7ff27c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff264003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2640047d0 0x7ff264004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c0c6b80 0x7ff264005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c06f9c0 con 0x7ff264003b40 2023-10-30T13:15:45.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26400ba40 con 0x7ff2640047d0 2023-10-30T13:15:45.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26400bc00 con 0x7ff27c0c6b80 2023-10-30T13:15:45.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27bfff640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff264003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27b7fe640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2640047d0 0x7ff264004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff280aba640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c0c6b80 0x7ff264005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27bfff640 1 -- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c0c6b80 msgr2=0x7ff264005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:45.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27bfff640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c0c6b80 0x7ff264005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27bfff640 1 -- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2640047d0 msgr2=0x7ff264004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:45.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27bfff640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2640047d0 0x7ff264004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.478+0000 7ff27bfff640 1 -- 172.21.15.148:0/3258054253 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff264003530 con 0x7ff264003b40 2023-10-30T13:15:45.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff27bfff640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff264003f40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009670 tx=0x7ff26c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8394572b717e85f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:45.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c0a5550 con 0x7ff264003b40 2023-10-30T13:15:45.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c0a56b0 con 0x7ff264003b40 2023-10-30T13:15:45.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff27affd640 1 -- 172.21.15.148:0/3258054253 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c0a5a60 con 0x7ff264003b40 2023-10-30T13:15:45.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff28251d640 1 -- 172.21.15.148:0/3258054253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 msgr2=0x7ff264003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff28251d640 1 --2- 172.21.15.148:0/3258054253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff264003f40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009670 tx=0x7ff26c01ca70 comp rx=0 tx=0).stop 2023-10-30T13:15:45.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff280aba640 1 -- 172.21.15.148:0/3258054253 reap_dead start 2023-10-30T13:15:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff28251d640 1 -- 172.21.15.148:0/3258054253 shutdown_connections 2023-10-30T13:15:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff28251d640 1 -- 172.21.15.148:0/3258054253 >> 172.21.15.148:0/3258054253 conn(0x7ff27c0c7910 msgr2=0x7ff27c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff28251d640 1 -- 172.21.15.148:0/3258054253 shutdown_connections 2023-10-30T13:15:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.482+0000 7ff28251d640 1 -- 172.21.15.148:0/3258054253 wait complete. 2023-10-30T13:15:45.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 Processor -- start 2023-10-30T13:15:45.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 -- start start 2023-10-30T13:15:45.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff27c13c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff27c139170 0x7ff27c1f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c13af30 0x7ff27c152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c138140 con 0x7ff264003b40 2023-10-30T13:15:45.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c1f6660 con 0x7ff27c139170 2023-10-30T13:15:45.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c1f6920 con 0x7ff27c13af30 2023-10-30T13:15:45.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff27c13c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff27c13c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43358/0 (socket says 172.21.15.148:43358) 2023-10-30T13:15:45.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27bfff640 1 -- 172.21.15.148:0/500113449 learned_addr learned my addr 172.21.15.148:0/500113449 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:45.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27b7fe640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff27c139170 0x7ff27c1f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff280aba640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c13af30 0x7ff27c152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27bfff640 1 -- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c13af30 msgr2=0x7ff27c152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27bfff640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c13af30 0x7ff27c152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27bfff640 1 -- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff27c139170 msgr2=0x7ff27c1f5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27bfff640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff27c139170 0x7ff27c1f5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27bfff640 1 -- 172.21.15.148:0/500113449 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c13a8b0 con 0x7ff264003b40 2023-10-30T13:15:45.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff27bfff640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff27c13c820 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0b0040 tx=0x7ff264005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:45.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff278ff9640 1 -- 172.21.15.148:0/500113449 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c0be070 con 0x7ff264003b40 2023-10-30T13:15:45.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff278ff9640 1 -- 172.21.15.148:0/500113449 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c00f000 con 0x7ff264003b40 2023-10-30T13:15:45.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff278ff9640 1 -- 172.21.15.148:0/500113449 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c004020 con 0x7ff264003b40 2023-10-30T13:15:45.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27c13ab90 con 0x7ff264003b40 2023-10-30T13:15:45.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27c0c2ea0 con 0x7ff264003b40 2023-10-30T13:15:45.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.486+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27c13ad80 con 0x7ff264003b40 2023-10-30T13:15:45.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.490+0000 7ff278ff9640 1 -- 172.21.15.148:0/500113449 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff26c0b4040 con 0x7ff264003b40 2023-10-30T13:15:45.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.490+0000 7ff278ff9640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2680664a0 0x7ff268068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.490+0000 7ff278ff9640 1 -- 172.21.15.148:0/500113449 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7ff26c11d860 con 0x7ff264003b40 2023-10-30T13:15:45.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.490+0000 7ff27b7fe640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2680664a0 0x7ff268068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.490+0000 7ff27b7fe640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2680664a0 0x7ff268068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff270007af0 tx=0x7ff270009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:45.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.494+0000 7ff278ff9640 1 -- 172.21.15.148:0/500113449 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff26c0eab90 con 0x7ff264003b40 2023-10-30T13:15:45.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff27c0c2190 con 0x7ff264003b40 2023-10-30T13:15:45.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff278ff9640 1 -- 172.21.15.148:0/500113449 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff26c0ee1b0 con 0x7ff264003b40 2023-10-30T13:15:45.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:45.679 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:40.696092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:45.680 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 24 2023-10-30T13:15:45.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2680664a0 msgr2=0x7ff268068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2680664a0 0x7ff268068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff270007af0 tx=0x7ff270009040 comp rx=0 tx=0).stop 2023-10-30T13:15:45.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 msgr2=0x7ff27c13c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff27c13c820 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0b0040 tx=0x7ff264005800 comp rx=0 tx=0).stop 2023-10-30T13:15:45.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 shutdown_connections 2023-10-30T13:15:45.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2680664a0 0x7ff268068960 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff27c13af30 0x7ff27c152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff27c139170 0x7ff27c1f5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 --2- 172.21.15.148:0/500113449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff264003b40 0x7ff27c13c820 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 >> 172.21.15.148:0/500113449 conn(0x7ff27c0c7910 msgr2=0x7ff27c06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:45.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 shutdown_connections 2023-10-30T13:15:45.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:45.674+0000 7ff28251d640 1 -- 172.21.15.148:0/500113449 wait complete. 2023-10-30T13:15:45.700 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:15:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 252 2023-10-30T13:15:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 253 2023-10-30T13:15:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 254 2023-10-30T13:15:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 255 2023-10-30T13:15:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 256 2023-10-30T13:15:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 257 2023-10-30T13:15:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 258 2023-10-30T13:15:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 259 2023-10-30T13:15:45.848 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 260 2023-10-30T13:15:45.858 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 261 2023-10-30T13:15:46.202 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 262 2023-10-30T13:15:46.212 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 263 2023-10-30T13:15:46.223 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 264 2023-10-30T13:15:46.234 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 265 2023-10-30T13:15:46.244 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 266 2023-10-30T13:15:46.255 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 267 2023-10-30T13:15:46.266 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 268 2023-10-30T13:15:46.277 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 269 2023-10-30T13:15:46.288 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 270 2023-10-30T13:15:46.370 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 271 2023-10-30T13:15:46.381 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 272 2023-10-30T13:15:46.391 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 273 2023-10-30T13:15:46.402 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 274 2023-10-30T13:15:46.415 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 275 2023-10-30T13:15:46.427 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 276 2023-10-30T13:15:46.438 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 277 2023-10-30T13:15:46.449 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 278 2023-10-30T13:15:46.459 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 279 2023-10-30T13:15:46.469 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 280 2023-10-30T13:15:47.380 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 281 2023-10-30T13:15:47.701 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:47.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.834+0000 7f479fcd6640 1 Processor -- start 2023-10-30T13:15:47.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.834+0000 7f479fcd6640 1 -- start start 2023-10-30T13:15:47.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479fcd6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f479815d6f0 0x7f479815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479fcd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 0x7f47980cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479fcd6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47980cb800 con 0x7f47981538a0 2023-10-30T13:15:47.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479fcd6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47980cb330 con 0x7f479815d6f0 2023-10-30T13:15:47.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479fcd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47980cad70 con 0x7f47980cf160 2023-10-30T13:15:47.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479d271640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f479815d6f0 0x7f479815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479da72640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f47981538a0 0x7f47980ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:36498/0 (socket says 172.21.15.148:36498) 2023-10-30T13:15:47.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479da72640 1 -- 172.21.15.148:0/3051199740 learned_addr learned my addr 172.21.15.148:0/3051199740 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:15:47.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479e273640 1 --2- 172.21.15.148:0/3051199740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 0x7f47980cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479d271640 1 -- 172.21.15.148:0/3051199740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 msgr2=0x7f47980cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:47.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479d271640 1 --2- 172.21.15.148:0/3051199740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 0x7f47980cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479d271640 1 -- 172.21.15.148:0/3051199740 >> v1:172.21.15.150:6789/0 conn(0x7f47981538a0 legacy=0x7f47980ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:47.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.838+0000 7f479d271640 1 -- 172.21.15.148:0/3051199740 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47980cb050 con 0x7f479815d6f0 2023-10-30T13:15:47.846 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 282 2023-10-30T13:15:47.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479d271640 1 --2- 172.21.15.148:0/3051199740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f479815d6f0 0x7f479815a090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f47880097c0 tx=0x7f478801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df42607f80d3f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:47.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479ca70640 1 -- 172.21.15.148:0/3051199740 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47880a8970 con 0x7f479815d6f0 2023-10-30T13:15:47.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479ca70640 1 -- 172.21.15.148:0/3051199740 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47880a8ad0 con 0x7f479815d6f0 2023-10-30T13:15:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479ca70640 1 -- 172.21.15.148:0/3051199740 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47880a8e60 con 0x7f479815d6f0 2023-10-30T13:15:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 -- 172.21.15.148:0/3051199740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f479815d6f0 msgr2=0x7f479815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 --2- 172.21.15.148:0/3051199740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f479815d6f0 0x7f479815a090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f47880097c0 tx=0x7f478801ec70 comp rx=0 tx=0).stop 2023-10-30T13:15:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 -- 172.21.15.148:0/3051199740 shutdown_connections 2023-10-30T13:15:47.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 --2- 172.21.15.148:0/3051199740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 0x7f47980cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 --2- 172.21.15.148:0/3051199740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f479815d6f0 0x7f479815a090 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 -- 172.21.15.148:0/3051199740 >> 172.21.15.148:0/3051199740 conn(0x7f479806bfe0 msgr2=0x7f479806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:47.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 -- 172.21.15.148:0/3051199740 shutdown_connections 2023-10-30T13:15:47.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 -- 172.21.15.148:0/3051199740 wait complete. 2023-10-30T13:15:47.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 Processor -- start 2023-10-30T13:15:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.842+0000 7f479fcd6640 1 -- start start 2023-10-30T13:15:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479fcd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 0x7f4798159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479fcd6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47981538a0 0x7f47980c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479fcd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f479815d6f0 0x7f47981e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479fcd6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47981e3bf0 con 0x7f47981538a0 2023-10-30T13:15:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479fcd6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47981580f0 con 0x7f479815d6f0 2023-10-30T13:15:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479fcd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47980c59a0 con 0x7f47980cf160 2023-10-30T13:15:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479d271640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47981538a0 0x7f47980c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479d271640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47981538a0 0x7f47980c52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43370/0 (socket says 172.21.15.148:43370) 2023-10-30T13:15:47.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479d271640 1 -- 172.21.15.148:0/2389915165 learned_addr learned my addr 172.21.15.148:0/2389915165 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:15:47.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479e273640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f479815d6f0 0x7f47981e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479da72640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 0x7f4798159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479d271640 1 -- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 msgr2=0x7f4798159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:47.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479d271640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 0x7f4798159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479d271640 1 -- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f479815d6f0 msgr2=0x7f47981e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:47.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479d271640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f479815d6f0 0x7f47981e2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479d271640 1 -- 172.21.15.148:0/2389915165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4788009420 con 0x7f47981538a0 2023-10-30T13:15:47.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479d271640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47981538a0 0x7f47980c52b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f47880b3040 tx=0x7f478801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:47.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f478e7fc640 1 -- 172.21.15.148:0/2389915165 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47880b9630 con 0x7f47981538a0 2023-10-30T13:15:47.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f478e7fc640 1 -- 172.21.15.148:0/2389915165 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47880b97b0 con 0x7f47981538a0 2023-10-30T13:15:47.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47981584d0 con 0x7f47981538a0 2023-10-30T13:15:47.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f478e7fc640 1 -- 172.21.15.148:0/2389915165 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47880b9b60 con 0x7f47981538a0 2023-10-30T13:15:47.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47980c4f60 con 0x7f47981538a0 2023-10-30T13:15:47.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.846+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47980b2a50 con 0x7f47981538a0 2023-10-30T13:15:47.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.850+0000 7f478e7fc640 1 -- 172.21.15.148:0/2389915165 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47880b7040 con 0x7f47981538a0 2023-10-30T13:15:47.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.850+0000 7f478e7fc640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f477c0664d0 0x7f477c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.850+0000 7f478e7fc640 1 -- 172.21.15.148:0/2389915165 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f4788121e20 con 0x7f47981538a0 2023-10-30T13:15:47.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.850+0000 7f479da72640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f477c0664d0 0x7f477c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.850+0000 7f479da72640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f477c0664d0 0x7f477c068990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4794005890 tx=0x7f4794009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:47.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:47.854+0000 7f478e7fc640 1 -- 172.21.15.148:0/2389915165 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47880ef200 con 0x7f47981538a0 2023-10-30T13:15:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f479804e170 con 0x7f47981538a0 2023-10-30T13:15:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f478e7fc640 1 -- 172.21.15.148:0/2389915165 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f47880b5030 con 0x7f47981538a0 2023-10-30T13:15:48.040 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:15:48.041 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:15:40.696092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:48.042 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 25 2023-10-30T13:15:48.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f477c0664d0 msgr2=0x7f477c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:48.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f477c0664d0 0x7f477c068990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4794005890 tx=0x7f4794009040 comp rx=0 tx=0).stop 2023-10-30T13:15:48.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47981538a0 msgr2=0x7f47980c52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:48.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47981538a0 0x7f47980c52b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f47880b3040 tx=0x7f478801ba10 comp rx=0 tx=0).stop 2023-10-30T13:15:48.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 shutdown_connections 2023-10-30T13:15:48.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f477c0664d0 0x7f477c068990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:48.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47981538a0 0x7f47980c52b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:48.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f479815d6f0 0x7f47981e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:48.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 --2- 172.21.15.148:0/2389915165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47980cf160 0x7f4798159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:48.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 >> 172.21.15.148:0/2389915165 conn(0x7f479806bfe0 msgr2=0x7f479814ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:48.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 shutdown_connections 2023-10-30T13:15:48.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:15:48.034+0000 7f479fcd6640 1 -- 172.21.15.148:0/2389915165 wait complete. 2023-10-30T13:15:48.060 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:15:48.060 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-10-30T13:15:48.581 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 283 2023-10-30T13:15:48.753 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 284 2023-10-30T13:15:49.744 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 285 2023-10-30T13:15:49.766 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 286 2023-10-30T13:15:49.790 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 287 2023-10-30T13:15:49.812 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 288 2023-10-30T13:15:49.833 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 289 2023-10-30T13:15:50.107 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 290 2023-10-30T13:15:50.564 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 291 2023-10-30T13:15:50.588 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 292 2023-10-30T13:15:50.613 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 293 2023-10-30T13:15:50.637 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 294 2023-10-30T13:15:50.661 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 295 2023-10-30T13:15:50.685 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 296 2023-10-30T13:15:50.708 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 297 2023-10-30T13:15:50.731 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 298 2023-10-30T13:15:50.755 INFO:tasks.workunit.client.0.smithi148.stdout: create dir 299 2023-10-30T13:15:50.777 INFO:tasks.workunit.client.0.smithi148.stdout:Create a snapshot in every first level dir ... 2023-10-30T13:15:50.792 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/.snap/snap-subdir-test 2023-10-30T13:15:50.821 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/30 file after the snapshot 2023-10-30T13:15:50.822 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/31 file after the snapshot 2023-10-30T13:15:50.823 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/32 file after the snapshot 2023-10-30T13:15:50.824 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/33 file after the snapshot 2023-10-30T13:15:50.825 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/34 file after the snapshot 2023-10-30T13:15:50.825 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/35 file after the snapshot 2023-10-30T13:15:50.826 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/36 file after the snapshot 2023-10-30T13:15:50.827 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/37 file after the snapshot 2023-10-30T13:15:50.827 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/38 file after the snapshot 2023-10-30T13:15:50.828 INFO:tasks.workunit.client.0.smithi148.stdout: create 100/39 file after the snapshot 2023-10-30T13:15:50.829 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/.snap/snap-subdir-test 2023-10-30T13:15:50.850 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/30 file after the snapshot 2023-10-30T13:15:50.851 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/31 file after the snapshot 2023-10-30T13:15:50.851 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/32 file after the snapshot 2023-10-30T13:15:50.852 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/33 file after the snapshot 2023-10-30T13:15:50.853 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/34 file after the snapshot 2023-10-30T13:15:50.853 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/35 file after the snapshot 2023-10-30T13:15:50.854 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/36 file after the snapshot 2023-10-30T13:15:50.855 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/37 file after the snapshot 2023-10-30T13:15:50.855 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/38 file after the snapshot 2023-10-30T13:15:50.856 INFO:tasks.workunit.client.0.smithi148.stdout: create 101/39 file after the snapshot 2023-10-30T13:15:50.857 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/.snap/snap-subdir-test 2023-10-30T13:15:50.879 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/30 file after the snapshot 2023-10-30T13:15:50.880 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/31 file after the snapshot 2023-10-30T13:15:50.880 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/32 file after the snapshot 2023-10-30T13:15:50.881 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/33 file after the snapshot 2023-10-30T13:15:50.882 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/34 file after the snapshot 2023-10-30T13:15:50.882 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/35 file after the snapshot 2023-10-30T13:15:50.883 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/36 file after the snapshot 2023-10-30T13:15:50.884 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/37 file after the snapshot 2023-10-30T13:15:50.884 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/38 file after the snapshot 2023-10-30T13:15:50.885 INFO:tasks.workunit.client.0.smithi148.stdout: create 102/39 file after the snapshot 2023-10-30T13:15:50.886 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/.snap/snap-subdir-test 2023-10-30T13:15:50.907 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/30 file after the snapshot 2023-10-30T13:15:50.908 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/31 file after the snapshot 2023-10-30T13:15:50.909 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/32 file after the snapshot 2023-10-30T13:15:50.910 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/33 file after the snapshot 2023-10-30T13:15:50.910 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/34 file after the snapshot 2023-10-30T13:15:50.911 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/35 file after the snapshot 2023-10-30T13:15:50.912 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/36 file after the snapshot 2023-10-30T13:15:50.912 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/37 file after the snapshot 2023-10-30T13:15:50.913 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/38 file after the snapshot 2023-10-30T13:15:50.914 INFO:tasks.workunit.client.0.smithi148.stdout: create 103/39 file after the snapshot 2023-10-30T13:15:50.915 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/.snap/snap-subdir-test 2023-10-30T13:15:50.936 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/30 file after the snapshot 2023-10-30T13:15:50.938 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/31 file after the snapshot 2023-10-30T13:15:50.938 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/32 file after the snapshot 2023-10-30T13:15:50.939 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/33 file after the snapshot 2023-10-30T13:15:50.940 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/34 file after the snapshot 2023-10-30T13:15:50.940 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/35 file after the snapshot 2023-10-30T13:15:50.941 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/36 file after the snapshot 2023-10-30T13:15:50.942 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/37 file after the snapshot 2023-10-30T13:15:50.942 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/38 file after the snapshot 2023-10-30T13:15:50.943 INFO:tasks.workunit.client.0.smithi148.stdout: create 104/39 file after the snapshot 2023-10-30T13:15:50.944 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/.snap/snap-subdir-test 2023-10-30T13:15:50.965 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/30 file after the snapshot 2023-10-30T13:15:50.966 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/31 file after the snapshot 2023-10-30T13:15:50.966 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/32 file after the snapshot 2023-10-30T13:15:50.967 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/33 file after the snapshot 2023-10-30T13:15:50.968 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/34 file after the snapshot 2023-10-30T13:15:50.968 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/35 file after the snapshot 2023-10-30T13:15:50.969 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/36 file after the snapshot 2023-10-30T13:15:50.970 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/37 file after the snapshot 2023-10-30T13:15:50.970 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/38 file after the snapshot 2023-10-30T13:15:50.971 INFO:tasks.workunit.client.0.smithi148.stdout: create 105/39 file after the snapshot 2023-10-30T13:15:50.972 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/.snap/snap-subdir-test 2023-10-30T13:15:50.992 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/30 file after the snapshot 2023-10-30T13:15:50.993 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/31 file after the snapshot 2023-10-30T13:15:50.994 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/32 file after the snapshot 2023-10-30T13:15:50.994 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/33 file after the snapshot 2023-10-30T13:15:50.995 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/34 file after the snapshot 2023-10-30T13:15:50.996 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/35 file after the snapshot 2023-10-30T13:15:50.996 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/36 file after the snapshot 2023-10-30T13:15:50.997 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/37 file after the snapshot 2023-10-30T13:15:50.998 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/38 file after the snapshot 2023-10-30T13:15:50.999 INFO:tasks.workunit.client.0.smithi148.stdout: create 106/39 file after the snapshot 2023-10-30T13:15:50.999 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/.snap/snap-subdir-test 2023-10-30T13:15:51.021 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/30 file after the snapshot 2023-10-30T13:15:51.022 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/31 file after the snapshot 2023-10-30T13:15:51.023 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/32 file after the snapshot 2023-10-30T13:15:51.023 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/33 file after the snapshot 2023-10-30T13:15:51.024 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/34 file after the snapshot 2023-10-30T13:15:51.025 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/35 file after the snapshot 2023-10-30T13:15:51.026 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/36 file after the snapshot 2023-10-30T13:15:51.026 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/37 file after the snapshot 2023-10-30T13:15:51.027 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/38 file after the snapshot 2023-10-30T13:15:51.028 INFO:tasks.workunit.client.0.smithi148.stdout: create 107/39 file after the snapshot 2023-10-30T13:15:51.028 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/.snap/snap-subdir-test 2023-10-30T13:15:51.050 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/30 file after the snapshot 2023-10-30T13:15:51.051 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/31 file after the snapshot 2023-10-30T13:15:51.052 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/32 file after the snapshot 2023-10-30T13:15:51.052 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/33 file after the snapshot 2023-10-30T13:15:51.053 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/34 file after the snapshot 2023-10-30T13:15:51.054 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/35 file after the snapshot 2023-10-30T13:15:51.055 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/36 file after the snapshot 2023-10-30T13:15:51.055 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/37 file after the snapshot 2023-10-30T13:15:51.056 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/38 file after the snapshot 2023-10-30T13:15:51.057 INFO:tasks.workunit.client.0.smithi148.stdout: create 108/39 file after the snapshot 2023-10-30T13:15:51.057 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/.snap/snap-subdir-test 2023-10-30T13:15:51.079 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/30 file after the snapshot 2023-10-30T13:15:51.081 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/31 file after the snapshot 2023-10-30T13:15:51.082 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/32 file after the snapshot 2023-10-30T13:15:51.083 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/33 file after the snapshot 2023-10-30T13:15:51.084 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/34 file after the snapshot 2023-10-30T13:15:51.085 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/35 file after the snapshot 2023-10-30T13:15:51.086 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/36 file after the snapshot 2023-10-30T13:15:51.087 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/37 file after the snapshot 2023-10-30T13:15:51.088 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/38 file after the snapshot 2023-10-30T13:15:51.089 INFO:tasks.workunit.client.0.smithi148.stdout: create 109/39 file after the snapshot 2023-10-30T13:15:51.090 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/.snap/snap-subdir-test 2023-10-30T13:15:51.113 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/30 file after the snapshot 2023-10-30T13:15:51.115 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/31 file after the snapshot 2023-10-30T13:15:51.116 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/32 file after the snapshot 2023-10-30T13:15:51.117 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/33 file after the snapshot 2023-10-30T13:15:51.118 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/34 file after the snapshot 2023-10-30T13:15:51.119 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/35 file after the snapshot 2023-10-30T13:15:51.120 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/36 file after the snapshot 2023-10-30T13:15:51.121 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/37 file after the snapshot 2023-10-30T13:15:51.122 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/38 file after the snapshot 2023-10-30T13:15:51.123 INFO:tasks.workunit.client.0.smithi148.stdout: create 110/39 file after the snapshot 2023-10-30T13:15:51.124 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/.snap/snap-subdir-test 2023-10-30T13:15:51.147 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/30 file after the snapshot 2023-10-30T13:15:51.149 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/31 file after the snapshot 2023-10-30T13:15:51.150 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/32 file after the snapshot 2023-10-30T13:15:51.151 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/33 file after the snapshot 2023-10-30T13:15:51.152 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/34 file after the snapshot 2023-10-30T13:15:51.152 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/35 file after the snapshot 2023-10-30T13:15:51.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/36 file after the snapshot 2023-10-30T13:15:51.155 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/37 file after the snapshot 2023-10-30T13:15:51.156 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/38 file after the snapshot 2023-10-30T13:15:51.156 INFO:tasks.workunit.client.0.smithi148.stdout: create 111/39 file after the snapshot 2023-10-30T13:15:51.157 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/.snap/snap-subdir-test 2023-10-30T13:15:51.181 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/30 file after the snapshot 2023-10-30T13:15:51.182 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/31 file after the snapshot 2023-10-30T13:15:51.183 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/32 file after the snapshot 2023-10-30T13:15:51.184 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/33 file after the snapshot 2023-10-30T13:15:51.185 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/34 file after the snapshot 2023-10-30T13:15:51.460 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/35 file after the snapshot 2023-10-30T13:15:51.466 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/36 file after the snapshot 2023-10-30T13:15:51.558 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/37 file after the snapshot 2023-10-30T13:15:51.585 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/38 file after the snapshot 2023-10-30T13:15:51.834 INFO:tasks.workunit.client.0.smithi148.stdout: create 112/39 file after the snapshot 2023-10-30T13:15:51.995 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/.snap/snap-subdir-test 2023-10-30T13:15:52.573 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/30 file after the snapshot 2023-10-30T13:15:52.579 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/31 file after the snapshot 2023-10-30T13:15:52.582 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/32 file after the snapshot 2023-10-30T13:15:52.584 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/33 file after the snapshot 2023-10-30T13:15:52.585 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/34 file after the snapshot 2023-10-30T13:15:52.587 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/35 file after the snapshot 2023-10-30T13:15:52.589 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/36 file after the snapshot 2023-10-30T13:15:52.590 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/37 file after the snapshot 2023-10-30T13:15:52.593 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/38 file after the snapshot 2023-10-30T13:15:52.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 113/39 file after the snapshot 2023-10-30T13:15:53.337 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/.snap/snap-subdir-test 2023-10-30T13:15:53.359 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/30 file after the snapshot 2023-10-30T13:15:53.360 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/31 file after the snapshot 2023-10-30T13:15:53.361 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/32 file after the snapshot 2023-10-30T13:15:53.362 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/33 file after the snapshot 2023-10-30T13:15:53.363 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/34 file after the snapshot 2023-10-30T13:15:53.364 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/35 file after the snapshot 2023-10-30T13:15:53.365 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/36 file after the snapshot 2023-10-30T13:15:53.365 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/37 file after the snapshot 2023-10-30T13:15:53.366 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/38 file after the snapshot 2023-10-30T13:15:53.367 INFO:tasks.workunit.client.0.smithi148.stdout: create 114/39 file after the snapshot 2023-10-30T13:15:53.368 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/.snap/snap-subdir-test 2023-10-30T13:15:53.391 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/30 file after the snapshot 2023-10-30T13:15:53.392 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/31 file after the snapshot 2023-10-30T13:15:53.393 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/32 file after the snapshot 2023-10-30T13:15:53.394 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/33 file after the snapshot 2023-10-30T13:15:53.395 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/34 file after the snapshot 2023-10-30T13:15:53.395 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/35 file after the snapshot 2023-10-30T13:15:53.396 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/36 file after the snapshot 2023-10-30T13:15:53.397 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/37 file after the snapshot 2023-10-30T13:15:53.398 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/38 file after the snapshot 2023-10-30T13:15:53.399 INFO:tasks.workunit.client.0.smithi148.stdout: create 115/39 file after the snapshot 2023-10-30T13:15:53.400 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/.snap/snap-subdir-test 2023-10-30T13:15:53.947 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:15:53.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:15:54.121 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/30 file after the snapshot 2023-10-30T13:15:54.122 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/31 file after the snapshot 2023-10-30T13:15:54.123 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/32 file after the snapshot 2023-10-30T13:15:54.124 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/33 file after the snapshot 2023-10-30T13:15:54.125 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/34 file after the snapshot 2023-10-30T13:15:54.126 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/35 file after the snapshot 2023-10-30T13:15:54.127 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/36 file after the snapshot 2023-10-30T13:15:54.127 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/37 file after the snapshot 2023-10-30T13:15:54.128 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/38 file after the snapshot 2023-10-30T13:15:54.129 INFO:tasks.workunit.client.0.smithi148.stdout: create 116/39 file after the snapshot 2023-10-30T13:15:54.130 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/.snap/snap-subdir-test 2023-10-30T13:15:54.681 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/30 file after the snapshot 2023-10-30T13:15:54.682 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/31 file after the snapshot 2023-10-30T13:15:54.683 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/32 file after the snapshot 2023-10-30T13:15:54.684 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/33 file after the snapshot 2023-10-30T13:15:54.684 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/34 file after the snapshot 2023-10-30T13:15:54.685 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/35 file after the snapshot 2023-10-30T13:15:54.686 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/36 file after the snapshot 2023-10-30T13:15:54.687 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/37 file after the snapshot 2023-10-30T13:15:54.688 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/38 file after the snapshot 2023-10-30T13:15:54.689 INFO:tasks.workunit.client.0.smithi148.stdout: create 117/39 file after the snapshot 2023-10-30T13:15:54.690 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/.snap/snap-subdir-test 2023-10-30T13:15:54.722 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/30 file after the snapshot 2023-10-30T13:15:54.725 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/31 file after the snapshot 2023-10-30T13:15:54.727 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/32 file after the snapshot 2023-10-30T13:15:54.729 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/33 file after the snapshot 2023-10-30T13:15:54.730 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/34 file after the snapshot 2023-10-30T13:15:54.732 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/35 file after the snapshot 2023-10-30T13:15:54.734 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/36 file after the snapshot 2023-10-30T13:15:54.736 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/37 file after the snapshot 2023-10-30T13:15:54.738 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/38 file after the snapshot 2023-10-30T13:15:54.740 INFO:tasks.workunit.client.0.smithi148.stdout: create 118/39 file after the snapshot 2023-10-30T13:15:54.742 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/.snap/snap-subdir-test 2023-10-30T13:15:54.763 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/30 file after the snapshot 2023-10-30T13:15:54.764 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/31 file after the snapshot 2023-10-30T13:15:54.765 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/32 file after the snapshot 2023-10-30T13:15:54.765 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/33 file after the snapshot 2023-10-30T13:15:54.766 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/34 file after the snapshot 2023-10-30T13:15:54.767 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/35 file after the snapshot 2023-10-30T13:15:54.768 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/36 file after the snapshot 2023-10-30T13:15:54.769 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/37 file after the snapshot 2023-10-30T13:15:54.770 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/38 file after the snapshot 2023-10-30T13:15:54.770 INFO:tasks.workunit.client.0.smithi148.stdout: create 119/39 file after the snapshot 2023-10-30T13:15:54.771 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/.snap/snap-subdir-test 2023-10-30T13:15:54.795 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/30 file after the snapshot 2023-10-30T13:15:54.796 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/31 file after the snapshot 2023-10-30T13:15:54.797 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/32 file after the snapshot 2023-10-30T13:15:54.797 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/33 file after the snapshot 2023-10-30T13:15:54.798 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/34 file after the snapshot 2023-10-30T13:15:54.799 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/35 file after the snapshot 2023-10-30T13:15:54.800 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/36 file after the snapshot 2023-10-30T13:15:54.801 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/37 file after the snapshot 2023-10-30T13:15:54.801 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/38 file after the snapshot 2023-10-30T13:15:54.802 INFO:tasks.workunit.client.0.smithi148.stdout: create 120/39 file after the snapshot 2023-10-30T13:15:54.803 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/.snap/snap-subdir-test 2023-10-30T13:15:54.833 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/30 file after the snapshot 2023-10-30T13:15:54.835 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/31 file after the snapshot 2023-10-30T13:15:54.837 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/32 file after the snapshot 2023-10-30T13:15:54.839 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/33 file after the snapshot 2023-10-30T13:15:54.841 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/34 file after the snapshot 2023-10-30T13:15:54.843 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/35 file after the snapshot 2023-10-30T13:15:54.845 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/36 file after the snapshot 2023-10-30T13:15:54.847 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/37 file after the snapshot 2023-10-30T13:15:54.849 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/38 file after the snapshot 2023-10-30T13:15:54.851 INFO:tasks.workunit.client.0.smithi148.stdout: create 121/39 file after the snapshot 2023-10-30T13:15:54.853 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/.snap/snap-subdir-test 2023-10-30T13:15:56.129 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/30 file after the snapshot 2023-10-30T13:15:56.139 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/31 file after the snapshot 2023-10-30T13:15:56.142 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/32 file after the snapshot 2023-10-30T13:15:56.145 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/33 file after the snapshot 2023-10-30T13:15:56.148 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/34 file after the snapshot 2023-10-30T13:15:56.165 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/35 file after the snapshot 2023-10-30T13:15:56.189 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/36 file after the snapshot 2023-10-30T13:15:56.218 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/37 file after the snapshot 2023-10-30T13:15:56.279 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/38 file after the snapshot 2023-10-30T13:15:56.342 INFO:tasks.workunit.client.0.smithi148.stdout: create 122/39 file after the snapshot 2023-10-30T13:15:56.697 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/.snap/snap-subdir-test 2023-10-30T13:15:59.904 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/30 file after the snapshot 2023-10-30T13:15:59.906 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/31 file after the snapshot 2023-10-30T13:15:59.908 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/32 file after the snapshot 2023-10-30T13:15:59.910 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/33 file after the snapshot 2023-10-30T13:15:59.912 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/34 file after the snapshot 2023-10-30T13:15:59.913 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/35 file after the snapshot 2023-10-30T13:15:59.915 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/36 file after the snapshot 2023-10-30T13:15:59.917 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/37 file after the snapshot 2023-10-30T13:15:59.919 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/38 file after the snapshot 2023-10-30T13:15:59.920 INFO:tasks.workunit.client.0.smithi148.stdout: create 123/39 file after the snapshot 2023-10-30T13:15:59.922 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/.snap/snap-subdir-test 2023-10-30T13:15:59.948 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/30 file after the snapshot 2023-10-30T13:15:59.949 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/31 file after the snapshot 2023-10-30T13:15:59.952 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/32 file after the snapshot 2023-10-30T13:15:59.955 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/33 file after the snapshot 2023-10-30T13:15:59.957 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/34 file after the snapshot 2023-10-30T13:15:59.960 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/35 file after the snapshot 2023-10-30T13:15:59.963 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/36 file after the snapshot 2023-10-30T13:15:59.965 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/37 file after the snapshot 2023-10-30T13:15:59.968 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/38 file after the snapshot 2023-10-30T13:15:59.971 INFO:tasks.workunit.client.0.smithi148.stdout: create 124/39 file after the snapshot 2023-10-30T13:15:59.973 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/.snap/snap-subdir-test 2023-10-30T13:15:59.990 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/30 file after the snapshot 2023-10-30T13:15:59.995 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/31 file after the snapshot 2023-10-30T13:15:59.997 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/32 file after the snapshot 2023-10-30T13:15:59.998 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/33 file after the snapshot 2023-10-30T13:15:59.999 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/34 file after the snapshot 2023-10-30T13:16:00.000 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/35 file after the snapshot 2023-10-30T13:16:00.001 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/36 file after the snapshot 2023-10-30T13:16:00.002 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/37 file after the snapshot 2023-10-30T13:16:00.003 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/38 file after the snapshot 2023-10-30T13:16:00.005 INFO:tasks.workunit.client.0.smithi148.stdout: create 125/39 file after the snapshot 2023-10-30T13:16:00.006 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/.snap/snap-subdir-test 2023-10-30T13:16:00.024 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/30 file after the snapshot 2023-10-30T13:16:00.025 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/31 file after the snapshot 2023-10-30T13:16:00.026 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/32 file after the snapshot 2023-10-30T13:16:00.028 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/33 file after the snapshot 2023-10-30T13:16:00.029 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/34 file after the snapshot 2023-10-30T13:16:00.030 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/35 file after the snapshot 2023-10-30T13:16:00.031 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/36 file after the snapshot 2023-10-30T13:16:00.032 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/37 file after the snapshot 2023-10-30T13:16:00.033 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/38 file after the snapshot 2023-10-30T13:16:00.034 INFO:tasks.workunit.client.0.smithi148.stdout: create 126/39 file after the snapshot 2023-10-30T13:16:00.035 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/.snap/snap-subdir-test 2023-10-30T13:16:00.061 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/30 file after the snapshot 2023-10-30T13:16:00.569 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/31 file after the snapshot 2023-10-30T13:16:00.572 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/32 file after the snapshot 2023-10-30T13:16:00.575 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/33 file after the snapshot 2023-10-30T13:16:00.577 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/34 file after the snapshot 2023-10-30T13:16:00.580 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/35 file after the snapshot 2023-10-30T13:16:00.583 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/36 file after the snapshot 2023-10-30T13:16:00.586 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/37 file after the snapshot 2023-10-30T13:16:00.588 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/38 file after the snapshot 2023-10-30T13:16:00.591 INFO:tasks.workunit.client.0.smithi148.stdout: create 127/39 file after the snapshot 2023-10-30T13:16:00.593 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/.snap/snap-subdir-test 2023-10-30T13:16:01.544 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/30 file after the snapshot 2023-10-30T13:16:01.545 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/31 file after the snapshot 2023-10-30T13:16:01.546 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/32 file after the snapshot 2023-10-30T13:16:01.547 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/33 file after the snapshot 2023-10-30T13:16:01.548 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/34 file after the snapshot 2023-10-30T13:16:01.548 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/35 file after the snapshot 2023-10-30T13:16:01.549 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/36 file after the snapshot 2023-10-30T13:16:01.550 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/37 file after the snapshot 2023-10-30T13:16:01.551 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/38 file after the snapshot 2023-10-30T13:16:01.551 INFO:tasks.workunit.client.0.smithi148.stdout: create 128/39 file after the snapshot 2023-10-30T13:16:01.552 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/.snap/snap-subdir-test 2023-10-30T13:16:01.573 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/30 file after the snapshot 2023-10-30T13:16:01.576 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/31 file after the snapshot 2023-10-30T13:16:01.578 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/32 file after the snapshot 2023-10-30T13:16:01.580 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/33 file after the snapshot 2023-10-30T13:16:01.581 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/34 file after the snapshot 2023-10-30T13:16:01.583 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/35 file after the snapshot 2023-10-30T13:16:01.585 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/36 file after the snapshot 2023-10-30T13:16:01.587 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/37 file after the snapshot 2023-10-30T13:16:01.589 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/38 file after the snapshot 2023-10-30T13:16:01.591 INFO:tasks.workunit.client.0.smithi148.stdout: create 129/39 file after the snapshot 2023-10-30T13:16:01.592 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/.snap/snap-subdir-test 2023-10-30T13:16:01.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/30 file after the snapshot 2023-10-30T13:16:01.620 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/31 file after the snapshot 2023-10-30T13:16:01.622 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/32 file after the snapshot 2023-10-30T13:16:01.624 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/33 file after the snapshot 2023-10-30T13:16:01.625 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/34 file after the snapshot 2023-10-30T13:16:01.627 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/35 file after the snapshot 2023-10-30T13:16:01.629 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/36 file after the snapshot 2023-10-30T13:16:01.631 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/37 file after the snapshot 2023-10-30T13:16:01.633 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/38 file after the snapshot 2023-10-30T13:16:01.635 INFO:tasks.workunit.client.0.smithi148.stdout: create 130/39 file after the snapshot 2023-10-30T13:16:01.636 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/.snap/snap-subdir-test 2023-10-30T13:16:01.662 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/30 file after the snapshot 2023-10-30T13:16:01.665 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/31 file after the snapshot 2023-10-30T13:16:01.666 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/32 file after the snapshot 2023-10-30T13:16:01.668 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/33 file after the snapshot 2023-10-30T13:16:01.670 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/34 file after the snapshot 2023-10-30T13:16:01.672 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/35 file after the snapshot 2023-10-30T13:16:02.106 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/36 file after the snapshot 2023-10-30T13:16:02.108 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/37 file after the snapshot 2023-10-30T13:16:02.110 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/38 file after the snapshot 2023-10-30T13:16:02.112 INFO:tasks.workunit.client.0.smithi148.stdout: create 131/39 file after the snapshot 2023-10-30T13:16:02.113 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/.snap/snap-subdir-test 2023-10-30T13:16:02.141 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/30 file after the snapshot 2023-10-30T13:16:02.144 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/31 file after the snapshot 2023-10-30T13:16:02.146 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/32 file after the snapshot 2023-10-30T13:16:02.148 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/33 file after the snapshot 2023-10-30T13:16:02.150 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/34 file after the snapshot 2023-10-30T13:16:02.152 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/35 file after the snapshot 2023-10-30T13:16:02.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/36 file after the snapshot 2023-10-30T13:16:02.155 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/37 file after the snapshot 2023-10-30T13:16:02.157 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/38 file after the snapshot 2023-10-30T13:16:02.160 INFO:tasks.workunit.client.0.smithi148.stdout: create 132/39 file after the snapshot 2023-10-30T13:16:02.161 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/.snap/snap-subdir-test 2023-10-30T13:16:02.178 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/30 file after the snapshot 2023-10-30T13:16:02.179 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/31 file after the snapshot 2023-10-30T13:16:02.180 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/32 file after the snapshot 2023-10-30T13:16:02.181 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/33 file after the snapshot 2023-10-30T13:16:02.182 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/34 file after the snapshot 2023-10-30T13:16:02.182 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/35 file after the snapshot 2023-10-30T13:16:02.183 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/36 file after the snapshot 2023-10-30T13:16:02.184 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/37 file after the snapshot 2023-10-30T13:16:02.185 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/38 file after the snapshot 2023-10-30T13:16:02.186 INFO:tasks.workunit.client.0.smithi148.stdout: create 133/39 file after the snapshot 2023-10-30T13:16:02.187 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/.snap/snap-subdir-test 2023-10-30T13:16:02.212 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/30 file after the snapshot 2023-10-30T13:16:02.214 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/31 file after the snapshot 2023-10-30T13:16:02.215 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/32 file after the snapshot 2023-10-30T13:16:02.216 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/33 file after the snapshot 2023-10-30T13:16:02.217 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/34 file after the snapshot 2023-10-30T13:16:02.217 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/35 file after the snapshot 2023-10-30T13:16:02.218 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/36 file after the snapshot 2023-10-30T13:16:02.218 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/37 file after the snapshot 2023-10-30T13:16:02.219 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/38 file after the snapshot 2023-10-30T13:16:02.220 INFO:tasks.workunit.client.0.smithi148.stdout: create 134/39 file after the snapshot 2023-10-30T13:16:02.221 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/.snap/snap-subdir-test 2023-10-30T13:16:02.239 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/30 file after the snapshot 2023-10-30T13:16:02.240 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/31 file after the snapshot 2023-10-30T13:16:02.240 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/32 file after the snapshot 2023-10-30T13:16:02.241 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/33 file after the snapshot 2023-10-30T13:16:02.242 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/34 file after the snapshot 2023-10-30T13:16:02.243 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/35 file after the snapshot 2023-10-30T13:16:02.243 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/36 file after the snapshot 2023-10-30T13:16:02.244 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/37 file after the snapshot 2023-10-30T13:16:02.245 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/38 file after the snapshot 2023-10-30T13:16:02.245 INFO:tasks.workunit.client.0.smithi148.stdout: create 135/39 file after the snapshot 2023-10-30T13:16:02.246 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/.snap/snap-subdir-test 2023-10-30T13:16:02.263 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/30 file after the snapshot 2023-10-30T13:16:02.264 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/31 file after the snapshot 2023-10-30T13:16:02.265 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/32 file after the snapshot 2023-10-30T13:16:02.265 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/33 file after the snapshot 2023-10-30T13:16:02.266 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/34 file after the snapshot 2023-10-30T13:16:02.267 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/35 file after the snapshot 2023-10-30T13:16:02.268 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/36 file after the snapshot 2023-10-30T13:16:02.268 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/37 file after the snapshot 2023-10-30T13:16:02.269 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/38 file after the snapshot 2023-10-30T13:16:02.270 INFO:tasks.workunit.client.0.smithi148.stdout: create 136/39 file after the snapshot 2023-10-30T13:16:02.270 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/.snap/snap-subdir-test 2023-10-30T13:16:02.288 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/30 file after the snapshot 2023-10-30T13:16:02.288 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/31 file after the snapshot 2023-10-30T13:16:02.289 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/32 file after the snapshot 2023-10-30T13:16:02.290 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/33 file after the snapshot 2023-10-30T13:16:02.291 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/34 file after the snapshot 2023-10-30T13:16:02.291 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/35 file after the snapshot 2023-10-30T13:16:02.292 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/36 file after the snapshot 2023-10-30T13:16:02.293 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/37 file after the snapshot 2023-10-30T13:16:02.293 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/38 file after the snapshot 2023-10-30T13:16:02.294 INFO:tasks.workunit.client.0.smithi148.stdout: create 137/39 file after the snapshot 2023-10-30T13:16:02.295 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/.snap/snap-subdir-test 2023-10-30T13:16:02.312 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/30 file after the snapshot 2023-10-30T13:16:02.313 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/31 file after the snapshot 2023-10-30T13:16:02.314 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/32 file after the snapshot 2023-10-30T13:16:02.314 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/33 file after the snapshot 2023-10-30T13:16:02.315 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/34 file after the snapshot 2023-10-30T13:16:02.316 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/35 file after the snapshot 2023-10-30T13:16:02.316 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/36 file after the snapshot 2023-10-30T13:16:02.317 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/37 file after the snapshot 2023-10-30T13:16:02.318 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/38 file after the snapshot 2023-10-30T13:16:02.319 INFO:tasks.workunit.client.0.smithi148.stdout: create 138/39 file after the snapshot 2023-10-30T13:16:02.319 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/.snap/snap-subdir-test 2023-10-30T13:16:02.346 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/30 file after the snapshot 2023-10-30T13:16:02.349 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/31 file after the snapshot 2023-10-30T13:16:02.350 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/32 file after the snapshot 2023-10-30T13:16:02.352 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/33 file after the snapshot 2023-10-30T13:16:02.355 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/34 file after the snapshot 2023-10-30T13:16:02.356 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/35 file after the snapshot 2023-10-30T13:16:02.358 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/36 file after the snapshot 2023-10-30T13:16:02.360 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/37 file after the snapshot 2023-10-30T13:16:02.362 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/38 file after the snapshot 2023-10-30T13:16:02.364 INFO:tasks.workunit.client.0.smithi148.stdout: create 139/39 file after the snapshot 2023-10-30T13:16:02.366 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/.snap/snap-subdir-test 2023-10-30T13:16:02.383 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/30 file after the snapshot 2023-10-30T13:16:02.384 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/31 file after the snapshot 2023-10-30T13:16:02.385 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/32 file after the snapshot 2023-10-30T13:16:02.386 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/33 file after the snapshot 2023-10-30T13:16:02.386 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/34 file after the snapshot 2023-10-30T13:16:02.387 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/35 file after the snapshot 2023-10-30T13:16:02.388 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/36 file after the snapshot 2023-10-30T13:16:02.389 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/37 file after the snapshot 2023-10-30T13:16:02.389 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/38 file after the snapshot 2023-10-30T13:16:02.390 INFO:tasks.workunit.client.0.smithi148.stdout: create 140/39 file after the snapshot 2023-10-30T13:16:02.391 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/.snap/snap-subdir-test 2023-10-30T13:16:02.412 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/30 file after the snapshot 2023-10-30T13:16:02.414 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/31 file after the snapshot 2023-10-30T13:16:02.415 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/32 file after the snapshot 2023-10-30T13:16:02.416 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/33 file after the snapshot 2023-10-30T13:16:02.417 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/34 file after the snapshot 2023-10-30T13:16:02.418 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/35 file after the snapshot 2023-10-30T13:16:02.419 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/36 file after the snapshot 2023-10-30T13:16:02.420 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/37 file after the snapshot 2023-10-30T13:16:02.421 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/38 file after the snapshot 2023-10-30T13:16:02.422 INFO:tasks.workunit.client.0.smithi148.stdout: create 141/39 file after the snapshot 2023-10-30T13:16:02.423 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/.snap/snap-subdir-test 2023-10-30T13:16:02.438 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/30 file after the snapshot 2023-10-30T13:16:02.439 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/31 file after the snapshot 2023-10-30T13:16:02.440 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/32 file after the snapshot 2023-10-30T13:16:02.441 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/33 file after the snapshot 2023-10-30T13:16:02.441 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/34 file after the snapshot 2023-10-30T13:16:02.810 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/35 file after the snapshot 2023-10-30T13:16:03.255 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/36 file after the snapshot 2023-10-30T13:16:03.255 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/37 file after the snapshot 2023-10-30T13:16:03.256 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/38 file after the snapshot 2023-10-30T13:16:03.257 INFO:tasks.workunit.client.0.smithi148.stdout: create 142/39 file after the snapshot 2023-10-30T13:16:03.258 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/.snap/snap-subdir-test 2023-10-30T13:16:04.750 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/30 file after the snapshot 2023-10-30T13:16:04.751 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/31 file after the snapshot 2023-10-30T13:16:04.752 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/32 file after the snapshot 2023-10-30T13:16:04.753 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/33 file after the snapshot 2023-10-30T13:16:04.754 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/34 file after the snapshot 2023-10-30T13:16:04.755 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/35 file after the snapshot 2023-10-30T13:16:04.756 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/36 file after the snapshot 2023-10-30T13:16:04.757 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/37 file after the snapshot 2023-10-30T13:16:04.758 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/38 file after the snapshot 2023-10-30T13:16:04.759 INFO:tasks.workunit.client.0.smithi148.stdout: create 143/39 file after the snapshot 2023-10-30T13:16:04.759 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/.snap/snap-subdir-test 2023-10-30T13:16:04.777 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/30 file after the snapshot 2023-10-30T13:16:04.779 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/31 file after the snapshot 2023-10-30T13:16:04.779 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/32 file after the snapshot 2023-10-30T13:16:04.780 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/33 file after the snapshot 2023-10-30T13:16:04.781 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/34 file after the snapshot 2023-10-30T13:16:04.782 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/35 file after the snapshot 2023-10-30T13:16:04.783 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/36 file after the snapshot 2023-10-30T13:16:04.783 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/37 file after the snapshot 2023-10-30T13:16:04.784 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/38 file after the snapshot 2023-10-30T13:16:04.785 INFO:tasks.workunit.client.0.smithi148.stdout: create 144/39 file after the snapshot 2023-10-30T13:16:04.785 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/.snap/snap-subdir-test 2023-10-30T13:16:04.812 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/30 file after the snapshot 2023-10-30T13:16:04.815 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/31 file after the snapshot 2023-10-30T13:16:04.817 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/32 file after the snapshot 2023-10-30T13:16:04.819 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/33 file after the snapshot 2023-10-30T13:16:04.820 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/34 file after the snapshot 2023-10-30T13:16:04.822 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/35 file after the snapshot 2023-10-30T13:16:04.824 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/36 file after the snapshot 2023-10-30T13:16:04.826 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/37 file after the snapshot 2023-10-30T13:16:04.828 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/38 file after the snapshot 2023-10-30T13:16:04.830 INFO:tasks.workunit.client.0.smithi148.stdout: create 145/39 file after the snapshot 2023-10-30T13:16:04.832 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/.snap/snap-subdir-test 2023-10-30T13:16:04.850 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/30 file after the snapshot 2023-10-30T13:16:04.851 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/31 file after the snapshot 2023-10-30T13:16:04.852 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/32 file after the snapshot 2023-10-30T13:16:04.853 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/33 file after the snapshot 2023-10-30T13:16:04.854 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/34 file after the snapshot 2023-10-30T13:16:04.854 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/35 file after the snapshot 2023-10-30T13:16:04.855 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/36 file after the snapshot 2023-10-30T13:16:04.856 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/37 file after the snapshot 2023-10-30T13:16:05.106 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/38 file after the snapshot 2023-10-30T13:16:05.107 INFO:tasks.workunit.client.0.smithi148.stdout: create 146/39 file after the snapshot 2023-10-30T13:16:05.108 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/.snap/snap-subdir-test 2023-10-30T13:16:05.124 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/30 file after the snapshot 2023-10-30T13:16:05.125 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/31 file after the snapshot 2023-10-30T13:16:05.125 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/32 file after the snapshot 2023-10-30T13:16:05.126 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/33 file after the snapshot 2023-10-30T13:16:05.127 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/34 file after the snapshot 2023-10-30T13:16:05.128 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/35 file after the snapshot 2023-10-30T13:16:05.130 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/36 file after the snapshot 2023-10-30T13:16:05.130 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/37 file after the snapshot 2023-10-30T13:16:05.131 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/38 file after the snapshot 2023-10-30T13:16:05.131 INFO:tasks.workunit.client.0.smithi148.stdout: create 147/39 file after the snapshot 2023-10-30T13:16:05.132 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/.snap/snap-subdir-test 2023-10-30T13:16:06.017 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/30 file after the snapshot 2023-10-30T13:16:06.019 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/31 file after the snapshot 2023-10-30T13:16:06.021 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/32 file after the snapshot 2023-10-30T13:16:06.023 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/33 file after the snapshot 2023-10-30T13:16:06.025 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/34 file after the snapshot 2023-10-30T13:16:06.027 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/35 file after the snapshot 2023-10-30T13:16:06.029 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/36 file after the snapshot 2023-10-30T13:16:06.031 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/37 file after the snapshot 2023-10-30T13:16:06.033 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/38 file after the snapshot 2023-10-30T13:16:06.035 INFO:tasks.workunit.client.0.smithi148.stdout: create 148/39 file after the snapshot 2023-10-30T13:16:06.037 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/.snap/snap-subdir-test 2023-10-30T13:16:06.053 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/30 file after the snapshot 2023-10-30T13:16:06.054 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/31 file after the snapshot 2023-10-30T13:16:06.055 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/32 file after the snapshot 2023-10-30T13:16:06.056 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/33 file after the snapshot 2023-10-30T13:16:06.057 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/34 file after the snapshot 2023-10-30T13:16:06.058 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/35 file after the snapshot 2023-10-30T13:16:06.059 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/36 file after the snapshot 2023-10-30T13:16:06.059 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/37 file after the snapshot 2023-10-30T13:16:06.060 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/38 file after the snapshot 2023-10-30T13:16:06.061 INFO:tasks.workunit.client.0.smithi148.stdout: create 149/39 file after the snapshot 2023-10-30T13:16:06.062 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/.snap/snap-subdir-test 2023-10-30T13:16:06.080 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/30 file after the snapshot 2023-10-30T13:16:06.082 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/31 file after the snapshot 2023-10-30T13:16:06.082 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/32 file after the snapshot 2023-10-30T13:16:06.083 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/33 file after the snapshot 2023-10-30T13:16:06.084 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/34 file after the snapshot 2023-10-30T13:16:06.085 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/35 file after the snapshot 2023-10-30T13:16:06.086 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/36 file after the snapshot 2023-10-30T13:16:06.086 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/37 file after the snapshot 2023-10-30T13:16:06.087 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/38 file after the snapshot 2023-10-30T13:16:06.088 INFO:tasks.workunit.client.0.smithi148.stdout: create 150/39 file after the snapshot 2023-10-30T13:16:06.089 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/.snap/snap-subdir-test 2023-10-30T13:16:06.358 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/30 file after the snapshot 2023-10-30T13:16:06.360 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/31 file after the snapshot 2023-10-30T13:16:06.361 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/32 file after the snapshot 2023-10-30T13:16:06.362 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/33 file after the snapshot 2023-10-30T13:16:06.363 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/34 file after the snapshot 2023-10-30T13:16:06.364 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/35 file after the snapshot 2023-10-30T13:16:06.365 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/36 file after the snapshot 2023-10-30T13:16:06.366 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/37 file after the snapshot 2023-10-30T13:16:06.367 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/38 file after the snapshot 2023-10-30T13:16:06.368 INFO:tasks.workunit.client.0.smithi148.stdout: create 151/39 file after the snapshot 2023-10-30T13:16:06.369 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/.snap/snap-subdir-test 2023-10-30T13:16:06.393 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/30 file after the snapshot 2023-10-30T13:16:06.396 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/31 file after the snapshot 2023-10-30T13:16:06.398 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/32 file after the snapshot 2023-10-30T13:16:06.400 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/33 file after the snapshot 2023-10-30T13:16:06.401 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/34 file after the snapshot 2023-10-30T13:16:06.403 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/35 file after the snapshot 2023-10-30T13:16:06.405 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/36 file after the snapshot 2023-10-30T13:16:06.407 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/37 file after the snapshot 2023-10-30T13:16:06.409 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/38 file after the snapshot 2023-10-30T13:16:06.411 INFO:tasks.workunit.client.0.smithi148.stdout: create 152/39 file after the snapshot 2023-10-30T13:16:06.414 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/.snap/snap-subdir-test 2023-10-30T13:16:06.440 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/30 file after the snapshot 2023-10-30T13:16:06.444 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/31 file after the snapshot 2023-10-30T13:16:06.447 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/32 file after the snapshot 2023-10-30T13:16:06.449 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/33 file after the snapshot 2023-10-30T13:16:06.452 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/34 file after the snapshot 2023-10-30T13:16:06.455 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/35 file after the snapshot 2023-10-30T13:16:06.458 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/36 file after the snapshot 2023-10-30T13:16:06.460 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/37 file after the snapshot 2023-10-30T13:16:06.463 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/38 file after the snapshot 2023-10-30T13:16:06.466 INFO:tasks.workunit.client.0.smithi148.stdout: create 153/39 file after the snapshot 2023-10-30T13:16:06.469 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/.snap/snap-subdir-test 2023-10-30T13:16:06.496 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/30 file after the snapshot 2023-10-30T13:16:06.500 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/31 file after the snapshot 2023-10-30T13:16:06.502 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/32 file after the snapshot 2023-10-30T13:16:06.505 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/33 file after the snapshot 2023-10-30T13:16:06.508 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/34 file after the snapshot 2023-10-30T13:16:06.511 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/35 file after the snapshot 2023-10-30T13:16:06.513 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/36 file after the snapshot 2023-10-30T13:16:06.516 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/37 file after the snapshot 2023-10-30T13:16:06.518 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/38 file after the snapshot 2023-10-30T13:16:06.520 INFO:tasks.workunit.client.0.smithi148.stdout: create 154/39 file after the snapshot 2023-10-30T13:16:06.522 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/.snap/snap-subdir-test 2023-10-30T13:16:06.537 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/30 file after the snapshot 2023-10-30T13:16:06.538 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/31 file after the snapshot 2023-10-30T13:16:06.538 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/32 file after the snapshot 2023-10-30T13:16:06.539 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/33 file after the snapshot 2023-10-30T13:16:06.540 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/34 file after the snapshot 2023-10-30T13:16:06.540 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/35 file after the snapshot 2023-10-30T13:16:06.541 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/36 file after the snapshot 2023-10-30T13:16:06.542 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/37 file after the snapshot 2023-10-30T13:16:06.542 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/38 file after the snapshot 2023-10-30T13:16:06.543 INFO:tasks.workunit.client.0.smithi148.stdout: create 155/39 file after the snapshot 2023-10-30T13:16:06.544 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/.snap/snap-subdir-test 2023-10-30T13:16:06.570 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/30 file after the snapshot 2023-10-30T13:16:06.572 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/31 file after the snapshot 2023-10-30T13:16:06.574 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/32 file after the snapshot 2023-10-30T13:16:06.576 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/33 file after the snapshot 2023-10-30T13:16:06.578 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/34 file after the snapshot 2023-10-30T13:16:06.580 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/35 file after the snapshot 2023-10-30T13:16:06.582 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/36 file after the snapshot 2023-10-30T13:16:06.584 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/37 file after the snapshot 2023-10-30T13:16:06.586 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/38 file after the snapshot 2023-10-30T13:16:06.588 INFO:tasks.workunit.client.0.smithi148.stdout: create 156/39 file after the snapshot 2023-10-30T13:16:06.589 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/.snap/snap-subdir-test 2023-10-30T13:16:06.609 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/30 file after the snapshot 2023-10-30T13:16:06.610 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/31 file after the snapshot 2023-10-30T13:16:06.611 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/32 file after the snapshot 2023-10-30T13:16:06.612 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/33 file after the snapshot 2023-10-30T13:16:06.613 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/34 file after the snapshot 2023-10-30T13:16:06.613 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/35 file after the snapshot 2023-10-30T13:16:06.614 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/36 file after the snapshot 2023-10-30T13:16:06.614 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/37 file after the snapshot 2023-10-30T13:16:06.615 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/38 file after the snapshot 2023-10-30T13:16:06.616 INFO:tasks.workunit.client.0.smithi148.stdout: create 157/39 file after the snapshot 2023-10-30T13:16:06.617 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/.snap/snap-subdir-test 2023-10-30T13:16:06.632 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/30 file after the snapshot 2023-10-30T13:16:06.633 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/31 file after the snapshot 2023-10-30T13:16:06.634 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/32 file after the snapshot 2023-10-30T13:16:06.634 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/33 file after the snapshot 2023-10-30T13:16:06.635 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/34 file after the snapshot 2023-10-30T13:16:06.636 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/35 file after the snapshot 2023-10-30T13:16:06.636 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/36 file after the snapshot 2023-10-30T13:16:06.637 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/37 file after the snapshot 2023-10-30T13:16:06.637 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/38 file after the snapshot 2023-10-30T13:16:06.638 INFO:tasks.workunit.client.0.smithi148.stdout: create 158/39 file after the snapshot 2023-10-30T13:16:06.639 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/.snap/snap-subdir-test 2023-10-30T13:16:06.661 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/30 file after the snapshot 2023-10-30T13:16:06.662 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/31 file after the snapshot 2023-10-30T13:16:06.663 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/32 file after the snapshot 2023-10-30T13:16:06.663 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/33 file after the snapshot 2023-10-30T13:16:06.664 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/34 file after the snapshot 2023-10-30T13:16:06.665 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/35 file after the snapshot 2023-10-30T13:16:06.666 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/36 file after the snapshot 2023-10-30T13:16:06.666 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/37 file after the snapshot 2023-10-30T13:16:06.667 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/38 file after the snapshot 2023-10-30T13:16:06.667 INFO:tasks.workunit.client.0.smithi148.stdout: create 159/39 file after the snapshot 2023-10-30T13:16:06.668 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/.snap/snap-subdir-test 2023-10-30T13:16:06.694 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/30 file after the snapshot 2023-10-30T13:16:06.697 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/31 file after the snapshot 2023-10-30T13:16:06.699 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/32 file after the snapshot 2023-10-30T13:16:06.701 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/33 file after the snapshot 2023-10-30T13:16:06.702 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/34 file after the snapshot 2023-10-30T13:16:06.705 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/35 file after the snapshot 2023-10-30T13:16:06.708 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/36 file after the snapshot 2023-10-30T13:16:06.711 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/37 file after the snapshot 2023-10-30T13:16:06.713 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/38 file after the snapshot 2023-10-30T13:16:06.716 INFO:tasks.workunit.client.0.smithi148.stdout: create 160/39 file after the snapshot 2023-10-30T13:16:06.718 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/.snap/snap-subdir-test 2023-10-30T13:16:06.740 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/30 file after the snapshot 2023-10-30T13:16:06.741 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/31 file after the snapshot 2023-10-30T13:16:06.742 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/32 file after the snapshot 2023-10-30T13:16:06.743 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/33 file after the snapshot 2023-10-30T13:16:06.744 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/34 file after the snapshot 2023-10-30T13:16:06.745 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/35 file after the snapshot 2023-10-30T13:16:06.746 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/36 file after the snapshot 2023-10-30T13:16:06.747 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/37 file after the snapshot 2023-10-30T13:16:06.748 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/38 file after the snapshot 2023-10-30T13:16:06.750 INFO:tasks.workunit.client.0.smithi148.stdout: create 161/39 file after the snapshot 2023-10-30T13:16:06.751 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/.snap/snap-subdir-test 2023-10-30T13:16:06.777 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/30 file after the snapshot 2023-10-30T13:16:07.049 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/31 file after the snapshot 2023-10-30T13:16:07.052 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/32 file after the snapshot 2023-10-30T13:16:07.054 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/33 file after the snapshot 2023-10-30T13:16:07.057 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/34 file after the snapshot 2023-10-30T13:16:07.060 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/35 file after the snapshot 2023-10-30T13:16:07.062 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/36 file after the snapshot 2023-10-30T13:16:07.065 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/37 file after the snapshot 2023-10-30T13:16:07.068 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/38 file after the snapshot 2023-10-30T13:16:07.071 INFO:tasks.workunit.client.0.smithi148.stdout: create 162/39 file after the snapshot 2023-10-30T13:16:07.073 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/.snap/snap-subdir-test 2023-10-30T13:16:07.100 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/30 file after the snapshot 2023-10-30T13:16:07.104 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/31 file after the snapshot 2023-10-30T13:16:07.107 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/32 file after the snapshot 2023-10-30T13:16:07.109 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/33 file after the snapshot 2023-10-30T13:16:07.112 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/34 file after the snapshot 2023-10-30T13:16:07.115 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/35 file after the snapshot 2023-10-30T13:16:07.117 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/36 file after the snapshot 2023-10-30T13:16:07.120 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/37 file after the snapshot 2023-10-30T13:16:07.123 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/38 file after the snapshot 2023-10-30T13:16:07.126 INFO:tasks.workunit.client.0.smithi148.stdout: create 163/39 file after the snapshot 2023-10-30T13:16:07.128 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/.snap/snap-subdir-test 2023-10-30T13:16:07.146 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/30 file after the snapshot 2023-10-30T13:16:07.147 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/31 file after the snapshot 2023-10-30T13:16:07.148 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/32 file after the snapshot 2023-10-30T13:16:07.149 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/33 file after the snapshot 2023-10-30T13:16:07.149 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/34 file after the snapshot 2023-10-30T13:16:07.150 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/35 file after the snapshot 2023-10-30T13:16:07.151 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/36 file after the snapshot 2023-10-30T13:16:07.151 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/37 file after the snapshot 2023-10-30T13:16:07.152 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/38 file after the snapshot 2023-10-30T13:16:07.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 164/39 file after the snapshot 2023-10-30T13:16:07.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/.snap/snap-subdir-test 2023-10-30T13:16:07.443 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/30 file after the snapshot 2023-10-30T13:16:07.447 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/31 file after the snapshot 2023-10-30T13:16:07.449 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/32 file after the snapshot 2023-10-30T13:16:07.452 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/33 file after the snapshot 2023-10-30T13:16:07.455 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/34 file after the snapshot 2023-10-30T13:16:07.458 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/35 file after the snapshot 2023-10-30T13:16:07.460 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/36 file after the snapshot 2023-10-30T13:16:07.463 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/37 file after the snapshot 2023-10-30T13:16:07.466 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/38 file after the snapshot 2023-10-30T13:16:07.468 INFO:tasks.workunit.client.0.smithi148.stdout: create 165/39 file after the snapshot 2023-10-30T13:16:07.471 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/.snap/snap-subdir-test 2023-10-30T13:16:07.486 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/30 file after the snapshot 2023-10-30T13:16:07.487 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/31 file after the snapshot 2023-10-30T13:16:07.488 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/32 file after the snapshot 2023-10-30T13:16:07.488 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/33 file after the snapshot 2023-10-30T13:16:07.489 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/34 file after the snapshot 2023-10-30T13:16:07.490 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/35 file after the snapshot 2023-10-30T13:16:07.491 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/36 file after the snapshot 2023-10-30T13:16:07.491 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/37 file after the snapshot 2023-10-30T13:16:07.492 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/38 file after the snapshot 2023-10-30T13:16:07.493 INFO:tasks.workunit.client.0.smithi148.stdout: create 166/39 file after the snapshot 2023-10-30T13:16:07.494 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/.snap/snap-subdir-test 2023-10-30T13:16:07.511 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/30 file after the snapshot 2023-10-30T13:16:07.512 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/31 file after the snapshot 2023-10-30T13:16:07.513 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/32 file after the snapshot 2023-10-30T13:16:07.514 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/33 file after the snapshot 2023-10-30T13:16:07.515 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/34 file after the snapshot 2023-10-30T13:16:07.515 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/35 file after the snapshot 2023-10-30T13:16:07.516 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/36 file after the snapshot 2023-10-30T13:16:07.517 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/37 file after the snapshot 2023-10-30T13:16:07.517 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/38 file after the snapshot 2023-10-30T13:16:07.518 INFO:tasks.workunit.client.0.smithi148.stdout: create 167/39 file after the snapshot 2023-10-30T13:16:07.519 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/.snap/snap-subdir-test 2023-10-30T13:16:07.548 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/30 file after the snapshot 2023-10-30T13:16:07.552 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/31 file after the snapshot 2023-10-30T13:16:07.554 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/32 file after the snapshot 2023-10-30T13:16:07.557 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/33 file after the snapshot 2023-10-30T13:16:07.560 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/34 file after the snapshot 2023-10-30T13:16:07.562 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/35 file after the snapshot 2023-10-30T13:16:07.565 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/36 file after the snapshot 2023-10-30T13:16:07.568 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/37 file after the snapshot 2023-10-30T13:16:07.570 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/38 file after the snapshot 2023-10-30T13:16:07.573 INFO:tasks.workunit.client.0.smithi148.stdout: create 168/39 file after the snapshot 2023-10-30T13:16:07.576 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/.snap/snap-subdir-test 2023-10-30T13:16:07.602 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/30 file after the snapshot 2023-10-30T13:16:07.606 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/31 file after the snapshot 2023-10-30T13:16:07.608 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/32 file after the snapshot 2023-10-30T13:16:07.609 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/33 file after the snapshot 2023-10-30T13:16:07.611 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/34 file after the snapshot 2023-10-30T13:16:07.613 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/35 file after the snapshot 2023-10-30T13:16:07.615 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/36 file after the snapshot 2023-10-30T13:16:07.617 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/37 file after the snapshot 2023-10-30T13:16:07.619 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/38 file after the snapshot 2023-10-30T13:16:07.621 INFO:tasks.workunit.client.0.smithi148.stdout: create 169/39 file after the snapshot 2023-10-30T13:16:07.622 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/.snap/snap-subdir-test 2023-10-30T13:16:07.636 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/30 file after the snapshot 2023-10-30T13:16:07.638 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/31 file after the snapshot 2023-10-30T13:16:07.638 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/32 file after the snapshot 2023-10-30T13:16:07.639 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/33 file after the snapshot 2023-10-30T13:16:07.639 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/34 file after the snapshot 2023-10-30T13:16:07.640 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/35 file after the snapshot 2023-10-30T13:16:07.641 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/36 file after the snapshot 2023-10-30T13:16:07.642 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/37 file after the snapshot 2023-10-30T13:16:07.642 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/38 file after the snapshot 2023-10-30T13:16:07.643 INFO:tasks.workunit.client.0.smithi148.stdout: create 170/39 file after the snapshot 2023-10-30T13:16:07.644 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/.snap/snap-subdir-test 2023-10-30T13:16:07.672 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/30 file after the snapshot 2023-10-30T13:16:07.676 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/31 file after the snapshot 2023-10-30T13:16:07.678 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/32 file after the snapshot 2023-10-30T13:16:07.681 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/33 file after the snapshot 2023-10-30T13:16:07.684 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/34 file after the snapshot 2023-10-30T13:16:07.686 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/35 file after the snapshot 2023-10-30T13:16:07.689 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/36 file after the snapshot 2023-10-30T13:16:07.692 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/37 file after the snapshot 2023-10-30T13:16:07.695 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/38 file after the snapshot 2023-10-30T13:16:07.697 INFO:tasks.workunit.client.0.smithi148.stdout: create 171/39 file after the snapshot 2023-10-30T13:16:07.700 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/.snap/snap-subdir-test 2023-10-30T13:16:07.721 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/30 file after the snapshot 2023-10-30T13:16:07.722 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/31 file after the snapshot 2023-10-30T13:16:07.723 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/32 file after the snapshot 2023-10-30T13:16:07.724 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/33 file after the snapshot 2023-10-30T13:16:07.724 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/34 file after the snapshot 2023-10-30T13:16:07.725 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/35 file after the snapshot 2023-10-30T13:16:07.726 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/36 file after the snapshot 2023-10-30T13:16:07.726 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/37 file after the snapshot 2023-10-30T13:16:07.727 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/38 file after the snapshot 2023-10-30T13:16:07.728 INFO:tasks.workunit.client.0.smithi148.stdout: create 172/39 file after the snapshot 2023-10-30T13:16:07.728 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/.snap/snap-subdir-test 2023-10-30T13:16:07.743 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/30 file after the snapshot 2023-10-30T13:16:07.744 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/31 file after the snapshot 2023-10-30T13:16:07.744 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/32 file after the snapshot 2023-10-30T13:16:07.745 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/33 file after the snapshot 2023-10-30T13:16:07.746 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/34 file after the snapshot 2023-10-30T13:16:07.746 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/35 file after the snapshot 2023-10-30T13:16:07.747 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/36 file after the snapshot 2023-10-30T13:16:07.748 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/37 file after the snapshot 2023-10-30T13:16:07.748 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/38 file after the snapshot 2023-10-30T13:16:07.749 INFO:tasks.workunit.client.0.smithi148.stdout: create 173/39 file after the snapshot 2023-10-30T13:16:07.749 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/.snap/snap-subdir-test 2023-10-30T13:16:07.766 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/30 file after the snapshot 2023-10-30T13:16:07.767 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/31 file after the snapshot 2023-10-30T13:16:07.768 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/32 file after the snapshot 2023-10-30T13:16:07.768 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/33 file after the snapshot 2023-10-30T13:16:07.769 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/34 file after the snapshot 2023-10-30T13:16:07.770 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/35 file after the snapshot 2023-10-30T13:16:07.770 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/36 file after the snapshot 2023-10-30T13:16:07.771 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/37 file after the snapshot 2023-10-30T13:16:07.772 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/38 file after the snapshot 2023-10-30T13:16:07.773 INFO:tasks.workunit.client.0.smithi148.stdout: create 174/39 file after the snapshot 2023-10-30T13:16:07.773 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/.snap/snap-subdir-test 2023-10-30T13:16:07.803 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/30 file after the snapshot 2023-10-30T13:16:07.805 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/31 file after the snapshot 2023-10-30T13:16:07.807 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/32 file after the snapshot 2023-10-30T13:16:07.809 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/33 file after the snapshot 2023-10-30T13:16:08.589 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/34 file after the snapshot 2023-10-30T13:16:08.591 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/35 file after the snapshot 2023-10-30T13:16:08.593 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/36 file after the snapshot 2023-10-30T13:16:08.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/37 file after the snapshot 2023-10-30T13:16:08.597 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/38 file after the snapshot 2023-10-30T13:16:08.599 INFO:tasks.workunit.client.0.smithi148.stdout: create 175/39 file after the snapshot 2023-10-30T13:16:08.601 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/.snap/snap-subdir-test 2023-10-30T13:16:08.621 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/30 file after the snapshot 2023-10-30T13:16:08.622 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/31 file after the snapshot 2023-10-30T13:16:08.623 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/32 file after the snapshot 2023-10-30T13:16:08.624 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/33 file after the snapshot 2023-10-30T13:16:08.624 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/34 file after the snapshot 2023-10-30T13:16:08.625 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/35 file after the snapshot 2023-10-30T13:16:08.626 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/36 file after the snapshot 2023-10-30T13:16:08.627 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/37 file after the snapshot 2023-10-30T13:16:08.627 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/38 file after the snapshot 2023-10-30T13:16:08.628 INFO:tasks.workunit.client.0.smithi148.stdout: create 176/39 file after the snapshot 2023-10-30T13:16:08.629 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/.snap/snap-subdir-test 2023-10-30T13:16:08.656 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/30 file after the snapshot 2023-10-30T13:16:08.658 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/31 file after the snapshot 2023-10-30T13:16:08.660 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/32 file after the snapshot 2023-10-30T13:16:08.663 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/33 file after the snapshot 2023-10-30T13:16:08.664 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/34 file after the snapshot 2023-10-30T13:16:08.666 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/35 file after the snapshot 2023-10-30T13:16:08.668 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/36 file after the snapshot 2023-10-30T13:16:08.671 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/37 file after the snapshot 2023-10-30T13:16:08.673 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/38 file after the snapshot 2023-10-30T13:16:08.675 INFO:tasks.workunit.client.0.smithi148.stdout: create 177/39 file after the snapshot 2023-10-30T13:16:08.676 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/.snap/snap-subdir-test 2023-10-30T13:16:08.703 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/30 file after the snapshot 2023-10-30T13:16:08.706 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/31 file after the snapshot 2023-10-30T13:16:08.708 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/32 file after the snapshot 2023-10-30T13:16:08.710 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/33 file after the snapshot 2023-10-30T13:16:08.712 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/34 file after the snapshot 2023-10-30T13:16:08.714 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/35 file after the snapshot 2023-10-30T13:16:08.716 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/36 file after the snapshot 2023-10-30T13:16:08.718 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/37 file after the snapshot 2023-10-30T13:16:08.720 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/38 file after the snapshot 2023-10-30T13:16:08.722 INFO:tasks.workunit.client.0.smithi148.stdout: create 178/39 file after the snapshot 2023-10-30T13:16:08.724 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/.snap/snap-subdir-test 2023-10-30T13:16:08.750 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/30 file after the snapshot 2023-10-30T13:16:08.753 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/31 file after the snapshot 2023-10-30T13:16:08.755 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/32 file after the snapshot 2023-10-30T13:16:08.756 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/33 file after the snapshot 2023-10-30T13:16:08.758 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/34 file after the snapshot 2023-10-30T13:16:08.760 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/35 file after the snapshot 2023-10-30T13:16:08.762 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/36 file after the snapshot 2023-10-30T13:16:08.764 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/37 file after the snapshot 2023-10-30T13:16:08.765 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/38 file after the snapshot 2023-10-30T13:16:08.767 INFO:tasks.workunit.client.0.smithi148.stdout: create 179/39 file after the snapshot 2023-10-30T13:16:08.769 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/.snap/snap-subdir-test 2023-10-30T13:16:08.909 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/30 file after the snapshot 2023-10-30T13:16:08.910 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/31 file after the snapshot 2023-10-30T13:16:08.911 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/32 file after the snapshot 2023-10-30T13:16:08.912 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/33 file after the snapshot 2023-10-30T13:16:08.913 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/34 file after the snapshot 2023-10-30T13:16:08.913 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/35 file after the snapshot 2023-10-30T13:16:08.914 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/36 file after the snapshot 2023-10-30T13:16:08.915 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/37 file after the snapshot 2023-10-30T13:16:08.916 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/38 file after the snapshot 2023-10-30T13:16:08.917 INFO:tasks.workunit.client.0.smithi148.stdout: create 180/39 file after the snapshot 2023-10-30T13:16:08.918 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/.snap/snap-subdir-test 2023-10-30T13:16:08.944 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/30 file after the snapshot 2023-10-30T13:16:08.947 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/31 file after the snapshot 2023-10-30T13:16:08.949 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/32 file after the snapshot 2023-10-30T13:16:08.950 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/33 file after the snapshot 2023-10-30T13:16:08.952 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/34 file after the snapshot 2023-10-30T13:16:08.954 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/35 file after the snapshot 2023-10-30T13:16:08.956 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/36 file after the snapshot 2023-10-30T13:16:08.958 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/37 file after the snapshot 2023-10-30T13:16:08.960 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/38 file after the snapshot 2023-10-30T13:16:08.962 INFO:tasks.workunit.client.0.smithi148.stdout: create 181/39 file after the snapshot 2023-10-30T13:16:08.964 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/.snap/snap-subdir-test 2023-10-30T13:16:08.982 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/30 file after the snapshot 2023-10-30T13:16:08.983 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/31 file after the snapshot 2023-10-30T13:16:08.984 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/32 file after the snapshot 2023-10-30T13:16:08.985 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/33 file after the snapshot 2023-10-30T13:16:08.986 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/34 file after the snapshot 2023-10-30T13:16:08.987 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/35 file after the snapshot 2023-10-30T13:16:08.988 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/36 file after the snapshot 2023-10-30T13:16:08.989 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/37 file after the snapshot 2023-10-30T13:16:08.990 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/38 file after the snapshot 2023-10-30T13:16:08.991 INFO:tasks.workunit.client.0.smithi148.stdout: create 182/39 file after the snapshot 2023-10-30T13:16:08.992 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/.snap/snap-subdir-test 2023-10-30T13:16:09.019 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/30 file after the snapshot 2023-10-30T13:16:09.021 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/31 file after the snapshot 2023-10-30T13:16:09.023 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/32 file after the snapshot 2023-10-30T13:16:09.025 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/33 file after the snapshot 2023-10-30T13:16:09.027 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/34 file after the snapshot 2023-10-30T13:16:09.029 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/35 file after the snapshot 2023-10-30T13:16:09.031 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/36 file after the snapshot 2023-10-30T13:16:09.033 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/37 file after the snapshot 2023-10-30T13:16:09.035 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/38 file after the snapshot 2023-10-30T13:16:09.037 INFO:tasks.workunit.client.0.smithi148.stdout: create 183/39 file after the snapshot 2023-10-30T13:16:09.039 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/.snap/snap-subdir-test 2023-10-30T13:16:09.060 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/30 file after the snapshot 2023-10-30T13:16:09.061 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/31 file after the snapshot 2023-10-30T13:16:09.061 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/32 file after the snapshot 2023-10-30T13:16:09.062 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/33 file after the snapshot 2023-10-30T13:16:09.063 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/34 file after the snapshot 2023-10-30T13:16:09.064 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/35 file after the snapshot 2023-10-30T13:16:09.065 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/36 file after the snapshot 2023-10-30T13:16:09.065 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/37 file after the snapshot 2023-10-30T13:16:09.066 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/38 file after the snapshot 2023-10-30T13:16:09.067 INFO:tasks.workunit.client.0.smithi148.stdout: create 184/39 file after the snapshot 2023-10-30T13:16:09.068 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/.snap/snap-subdir-test 2023-10-30T13:16:09.918 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/30 file after the snapshot 2023-10-30T13:16:09.919 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/31 file after the snapshot 2023-10-30T13:16:10.171 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/32 file after the snapshot 2023-10-30T13:16:10.171 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/33 file after the snapshot 2023-10-30T13:16:10.173 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/34 file after the snapshot 2023-10-30T13:16:10.174 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/35 file after the snapshot 2023-10-30T13:16:10.175 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/36 file after the snapshot 2023-10-30T13:16:10.175 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/37 file after the snapshot 2023-10-30T13:16:10.176 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/38 file after the snapshot 2023-10-30T13:16:10.177 INFO:tasks.workunit.client.0.smithi148.stdout: create 185/39 file after the snapshot 2023-10-30T13:16:10.178 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/.snap/snap-subdir-test 2023-10-30T13:16:10.201 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/30 file after the snapshot 2023-10-30T13:16:10.202 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/31 file after the snapshot 2023-10-30T13:16:10.203 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/32 file after the snapshot 2023-10-30T13:16:10.204 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/33 file after the snapshot 2023-10-30T13:16:10.204 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/34 file after the snapshot 2023-10-30T13:16:10.205 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/35 file after the snapshot 2023-10-30T13:16:10.206 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/36 file after the snapshot 2023-10-30T13:16:10.207 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/37 file after the snapshot 2023-10-30T13:16:10.208 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/38 file after the snapshot 2023-10-30T13:16:10.209 INFO:tasks.workunit.client.0.smithi148.stdout: create 186/39 file after the snapshot 2023-10-30T13:16:10.209 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/.snap/snap-subdir-test 2023-10-30T13:16:10.230 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/30 file after the snapshot 2023-10-30T13:16:10.232 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/31 file after the snapshot 2023-10-30T13:16:10.232 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/32 file after the snapshot 2023-10-30T13:16:10.233 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/33 file after the snapshot 2023-10-30T13:16:10.234 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/34 file after the snapshot 2023-10-30T13:16:10.235 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/35 file after the snapshot 2023-10-30T13:16:10.235 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/36 file after the snapshot 2023-10-30T13:16:10.236 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/37 file after the snapshot 2023-10-30T13:16:10.237 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/38 file after the snapshot 2023-10-30T13:16:10.238 INFO:tasks.workunit.client.0.smithi148.stdout: create 187/39 file after the snapshot 2023-10-30T13:16:10.238 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/.snap/snap-subdir-test 2023-10-30T13:16:10.266 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/30 file after the snapshot 2023-10-30T13:16:10.268 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/31 file after the snapshot 2023-10-30T13:16:10.270 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/32 file after the snapshot 2023-10-30T13:16:10.272 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/33 file after the snapshot 2023-10-30T13:16:10.274 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/34 file after the snapshot 2023-10-30T13:16:10.276 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/35 file after the snapshot 2023-10-30T13:16:10.278 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/36 file after the snapshot 2023-10-30T13:16:10.280 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/37 file after the snapshot 2023-10-30T13:16:10.282 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/38 file after the snapshot 2023-10-30T13:16:10.284 INFO:tasks.workunit.client.0.smithi148.stdout: create 188/39 file after the snapshot 2023-10-30T13:16:10.287 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/.snap/snap-subdir-test 2023-10-30T13:16:10.304 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/30 file after the snapshot 2023-10-30T13:16:10.305 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/31 file after the snapshot 2023-10-30T13:16:10.306 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/32 file after the snapshot 2023-10-30T13:16:10.307 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/33 file after the snapshot 2023-10-30T13:16:10.307 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/34 file after the snapshot 2023-10-30T13:16:10.308 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/35 file after the snapshot 2023-10-30T13:16:10.309 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/36 file after the snapshot 2023-10-30T13:16:10.310 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/37 file after the snapshot 2023-10-30T13:16:10.310 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/38 file after the snapshot 2023-10-30T13:16:10.311 INFO:tasks.workunit.client.0.smithi148.stdout: create 189/39 file after the snapshot 2023-10-30T13:16:10.312 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/.snap/snap-subdir-test 2023-10-30T13:16:10.336 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/30 file after the snapshot 2023-10-30T13:16:10.338 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/31 file after the snapshot 2023-10-30T13:16:10.339 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/32 file after the snapshot 2023-10-30T13:16:10.339 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/33 file after the snapshot 2023-10-30T13:16:10.340 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/34 file after the snapshot 2023-10-30T13:16:10.341 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/35 file after the snapshot 2023-10-30T13:16:10.342 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/36 file after the snapshot 2023-10-30T13:16:10.343 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/37 file after the snapshot 2023-10-30T13:16:10.343 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/38 file after the snapshot 2023-10-30T13:16:10.344 INFO:tasks.workunit.client.0.smithi148.stdout: create 190/39 file after the snapshot 2023-10-30T13:16:10.345 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/.snap/snap-subdir-test 2023-10-30T13:16:10.982 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/30 file after the snapshot 2023-10-30T13:16:10.983 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/31 file after the snapshot 2023-10-30T13:16:10.984 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/32 file after the snapshot 2023-10-30T13:16:10.985 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/33 file after the snapshot 2023-10-30T13:16:10.986 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/34 file after the snapshot 2023-10-30T13:16:10.987 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/35 file after the snapshot 2023-10-30T13:16:10.988 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/36 file after the snapshot 2023-10-30T13:16:10.989 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/37 file after the snapshot 2023-10-30T13:16:10.990 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/38 file after the snapshot 2023-10-30T13:16:10.991 INFO:tasks.workunit.client.0.smithi148.stdout: create 191/39 file after the snapshot 2023-10-30T13:16:10.991 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/.snap/snap-subdir-test 2023-10-30T13:16:11.013 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/30 file after the snapshot 2023-10-30T13:16:11.014 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/31 file after the snapshot 2023-10-30T13:16:11.015 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/32 file after the snapshot 2023-10-30T13:16:11.016 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/33 file after the snapshot 2023-10-30T13:16:11.017 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/34 file after the snapshot 2023-10-30T13:16:11.017 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/35 file after the snapshot 2023-10-30T13:16:11.018 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/36 file after the snapshot 2023-10-30T13:16:11.019 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/37 file after the snapshot 2023-10-30T13:16:11.020 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/38 file after the snapshot 2023-10-30T13:16:11.021 INFO:tasks.workunit.client.0.smithi148.stdout: create 192/39 file after the snapshot 2023-10-30T13:16:11.022 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/.snap/snap-subdir-test 2023-10-30T13:16:11.042 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/30 file after the snapshot 2023-10-30T13:16:11.046 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/31 file after the snapshot 2023-10-30T13:16:11.049 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/32 file after the snapshot 2023-10-30T13:16:11.052 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/33 file after the snapshot 2023-10-30T13:16:11.054 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/34 file after the snapshot 2023-10-30T13:16:11.057 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/35 file after the snapshot 2023-10-30T13:16:11.060 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/36 file after the snapshot 2023-10-30T13:16:11.063 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/37 file after the snapshot 2023-10-30T13:16:11.315 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/38 file after the snapshot 2023-10-30T13:16:11.318 INFO:tasks.workunit.client.0.smithi148.stdout: create 193/39 file after the snapshot 2023-10-30T13:16:11.321 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/.snap/snap-subdir-test 2023-10-30T13:16:11.344 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/30 file after the snapshot 2023-10-30T13:16:11.345 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/31 file after the snapshot 2023-10-30T13:16:11.346 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/32 file after the snapshot 2023-10-30T13:16:11.347 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/33 file after the snapshot 2023-10-30T13:16:11.348 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/34 file after the snapshot 2023-10-30T13:16:11.349 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/35 file after the snapshot 2023-10-30T13:16:11.350 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/36 file after the snapshot 2023-10-30T13:16:11.351 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/37 file after the snapshot 2023-10-30T13:16:11.351 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/38 file after the snapshot 2023-10-30T13:16:11.352 INFO:tasks.workunit.client.0.smithi148.stdout: create 194/39 file after the snapshot 2023-10-30T13:16:11.353 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/.snap/snap-subdir-test 2023-10-30T13:16:11.375 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/30 file after the snapshot 2023-10-30T13:16:11.376 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/31 file after the snapshot 2023-10-30T13:16:11.376 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/32 file after the snapshot 2023-10-30T13:16:11.377 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/33 file after the snapshot 2023-10-30T13:16:11.378 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/34 file after the snapshot 2023-10-30T13:16:11.379 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/35 file after the snapshot 2023-10-30T13:16:11.380 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/36 file after the snapshot 2023-10-30T13:16:11.381 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/37 file after the snapshot 2023-10-30T13:16:11.381 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/38 file after the snapshot 2023-10-30T13:16:11.383 INFO:tasks.workunit.client.0.smithi148.stdout: create 195/39 file after the snapshot 2023-10-30T13:16:11.383 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/.snap/snap-subdir-test 2023-10-30T13:16:11.403 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/30 file after the snapshot 2023-10-30T13:16:11.406 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/31 file after the snapshot 2023-10-30T13:16:11.408 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/32 file after the snapshot 2023-10-30T13:16:11.410 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/33 file after the snapshot 2023-10-30T13:16:11.411 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/34 file after the snapshot 2023-10-30T13:16:11.414 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/35 file after the snapshot 2023-10-30T13:16:11.416 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/36 file after the snapshot 2023-10-30T13:16:11.419 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/37 file after the snapshot 2023-10-30T13:16:11.421 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/38 file after the snapshot 2023-10-30T13:16:11.438 INFO:tasks.workunit.client.0.smithi148.stdout: create 196/39 file after the snapshot 2023-10-30T13:16:11.457 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/.snap/snap-subdir-test 2023-10-30T13:16:14.752 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/30 file after the snapshot 2023-10-30T13:16:14.754 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/31 file after the snapshot 2023-10-30T13:16:14.756 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/32 file after the snapshot 2023-10-30T13:16:14.758 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/33 file after the snapshot 2023-10-30T13:16:14.759 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/34 file after the snapshot 2023-10-30T13:16:14.761 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/35 file after the snapshot 2023-10-30T13:16:14.763 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/36 file after the snapshot 2023-10-30T13:16:14.765 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/37 file after the snapshot 2023-10-30T13:16:14.767 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/38 file after the snapshot 2023-10-30T13:16:14.769 INFO:tasks.workunit.client.0.smithi148.stdout: create 197/39 file after the snapshot 2023-10-30T13:16:14.771 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/.snap/snap-subdir-test 2023-10-30T13:16:14.789 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/30 file after the snapshot 2023-10-30T13:16:14.791 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/31 file after the snapshot 2023-10-30T13:16:14.791 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/32 file after the snapshot 2023-10-30T13:16:14.792 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/33 file after the snapshot 2023-10-30T13:16:14.793 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/34 file after the snapshot 2023-10-30T13:16:14.794 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/35 file after the snapshot 2023-10-30T13:16:14.795 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/36 file after the snapshot 2023-10-30T13:16:14.795 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/37 file after the snapshot 2023-10-30T13:16:14.796 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/38 file after the snapshot 2023-10-30T13:16:14.797 INFO:tasks.workunit.client.0.smithi148.stdout: create 198/39 file after the snapshot 2023-10-30T13:16:14.798 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/.snap/snap-subdir-test 2023-10-30T13:16:15.462 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/30 file after the snapshot 2023-10-30T13:16:15.464 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/31 file after the snapshot 2023-10-30T13:16:15.465 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/32 file after the snapshot 2023-10-30T13:16:15.467 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/33 file after the snapshot 2023-10-30T13:16:15.469 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/34 file after the snapshot 2023-10-30T13:16:15.471 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/35 file after the snapshot 2023-10-30T13:16:15.472 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/36 file after the snapshot 2023-10-30T13:16:15.475 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/37 file after the snapshot 2023-10-30T13:16:15.476 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/38 file after the snapshot 2023-10-30T13:16:15.478 INFO:tasks.workunit.client.0.smithi148.stdout: create 199/39 file after the snapshot 2023-10-30T13:16:15.481 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/.snap/snap-subdir-test 2023-10-30T13:16:15.501 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/30 file after the snapshot 2023-10-30T13:16:15.503 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/31 file after the snapshot 2023-10-30T13:16:15.503 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/32 file after the snapshot 2023-10-30T13:16:15.503 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/33 file after the snapshot 2023-10-30T13:16:15.503 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/34 file after the snapshot 2023-10-30T13:16:15.503 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/35 file after the snapshot 2023-10-30T13:16:15.504 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/36 file after the snapshot 2023-10-30T13:16:15.504 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/37 file after the snapshot 2023-10-30T13:16:15.504 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/38 file after the snapshot 2023-10-30T13:16:15.504 INFO:tasks.workunit.client.0.smithi148.stdout: create 200/39 file after the snapshot 2023-10-30T13:16:15.504 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/.snap/snap-subdir-test 2023-10-30T13:16:15.529 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/30 file after the snapshot 2023-10-30T13:16:15.532 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/31 file after the snapshot 2023-10-30T13:16:15.533 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/32 file after the snapshot 2023-10-30T13:16:15.533 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/33 file after the snapshot 2023-10-30T13:16:15.533 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/34 file after the snapshot 2023-10-30T13:16:15.533 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/35 file after the snapshot 2023-10-30T13:16:15.533 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/36 file after the snapshot 2023-10-30T13:16:15.534 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/37 file after the snapshot 2023-10-30T13:16:15.534 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/38 file after the snapshot 2023-10-30T13:16:15.534 INFO:tasks.workunit.client.0.smithi148.stdout: create 201/39 file after the snapshot 2023-10-30T13:16:15.534 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/.snap/snap-subdir-test 2023-10-30T13:16:15.558 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/30 file after the snapshot 2023-10-30T13:16:15.559 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/31 file after the snapshot 2023-10-30T13:16:15.559 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/32 file after the snapshot 2023-10-30T13:16:15.560 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/33 file after the snapshot 2023-10-30T13:16:15.560 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/34 file after the snapshot 2023-10-30T13:16:15.560 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/35 file after the snapshot 2023-10-30T13:16:15.560 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/36 file after the snapshot 2023-10-30T13:16:15.560 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/37 file after the snapshot 2023-10-30T13:16:15.561 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/38 file after the snapshot 2023-10-30T13:16:15.561 INFO:tasks.workunit.client.0.smithi148.stdout: create 202/39 file after the snapshot 2023-10-30T13:16:15.561 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/.snap/snap-subdir-test 2023-10-30T13:16:15.590 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/30 file after the snapshot 2023-10-30T13:16:15.593 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/31 file after the snapshot 2023-10-30T13:16:15.594 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/32 file after the snapshot 2023-10-30T13:16:15.594 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/33 file after the snapshot 2023-10-30T13:16:15.594 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/34 file after the snapshot 2023-10-30T13:16:15.594 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/35 file after the snapshot 2023-10-30T13:16:15.594 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/36 file after the snapshot 2023-10-30T13:16:15.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/37 file after the snapshot 2023-10-30T13:16:15.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/38 file after the snapshot 2023-10-30T13:16:15.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 203/39 file after the snapshot 2023-10-30T13:16:15.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/.snap/snap-subdir-test 2023-10-30T13:16:15.616 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/30 file after the snapshot 2023-10-30T13:16:15.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/31 file after the snapshot 2023-10-30T13:16:15.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/32 file after the snapshot 2023-10-30T13:16:15.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/33 file after the snapshot 2023-10-30T13:16:15.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/34 file after the snapshot 2023-10-30T13:16:15.619 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/35 file after the snapshot 2023-10-30T13:16:15.619 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/36 file after the snapshot 2023-10-30T13:16:15.619 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/37 file after the snapshot 2023-10-30T13:16:15.619 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/38 file after the snapshot 2023-10-30T13:16:15.619 INFO:tasks.workunit.client.0.smithi148.stdout: create 204/39 file after the snapshot 2023-10-30T13:16:15.619 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/.snap/snap-subdir-test 2023-10-30T13:16:15.646 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/30 file after the snapshot 2023-10-30T13:16:15.649 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/31 file after the snapshot 2023-10-30T13:16:15.649 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/32 file after the snapshot 2023-10-30T13:16:15.649 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/33 file after the snapshot 2023-10-30T13:16:15.649 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/34 file after the snapshot 2023-10-30T13:16:15.650 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/35 file after the snapshot 2023-10-30T13:16:15.650 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/36 file after the snapshot 2023-10-30T13:16:15.650 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/37 file after the snapshot 2023-10-30T13:16:15.650 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/38 file after the snapshot 2023-10-30T13:16:15.650 INFO:tasks.workunit.client.0.smithi148.stdout: create 205/39 file after the snapshot 2023-10-30T13:16:15.651 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/.snap/snap-subdir-test 2023-10-30T13:16:15.696 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/30 file after the snapshot 2023-10-30T13:16:15.699 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/31 file after the snapshot 2023-10-30T13:16:15.699 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/32 file after the snapshot 2023-10-30T13:16:15.699 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/33 file after the snapshot 2023-10-30T13:16:15.699 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/34 file after the snapshot 2023-10-30T13:16:15.700 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/35 file after the snapshot 2023-10-30T13:16:15.700 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/36 file after the snapshot 2023-10-30T13:16:15.700 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/37 file after the snapshot 2023-10-30T13:16:15.700 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/38 file after the snapshot 2023-10-30T13:16:15.700 INFO:tasks.workunit.client.0.smithi148.stdout: create 206/39 file after the snapshot 2023-10-30T13:16:15.701 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/.snap/snap-subdir-test 2023-10-30T13:16:15.749 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/30 file after the snapshot 2023-10-30T13:16:15.752 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/31 file after the snapshot 2023-10-30T13:16:15.752 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/32 file after the snapshot 2023-10-30T13:16:15.752 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/33 file after the snapshot 2023-10-30T13:16:15.752 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/34 file after the snapshot 2023-10-30T13:16:15.753 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/35 file after the snapshot 2023-10-30T13:16:15.753 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/36 file after the snapshot 2023-10-30T13:16:15.753 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/37 file after the snapshot 2023-10-30T13:16:15.753 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/38 file after the snapshot 2023-10-30T13:16:15.753 INFO:tasks.workunit.client.0.smithi148.stdout: create 207/39 file after the snapshot 2023-10-30T13:16:15.754 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/.snap/snap-subdir-test 2023-10-30T13:16:15.770 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/30 file after the snapshot 2023-10-30T13:16:15.771 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/31 file after the snapshot 2023-10-30T13:16:15.772 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/32 file after the snapshot 2023-10-30T13:16:15.772 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/33 file after the snapshot 2023-10-30T13:16:15.772 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/34 file after the snapshot 2023-10-30T13:16:15.772 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/35 file after the snapshot 2023-10-30T13:16:15.772 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/36 file after the snapshot 2023-10-30T13:16:15.772 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/37 file after the snapshot 2023-10-30T13:16:15.773 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/38 file after the snapshot 2023-10-30T13:16:15.773 INFO:tasks.workunit.client.0.smithi148.stdout: create 208/39 file after the snapshot 2023-10-30T13:16:15.773 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/.snap/snap-subdir-test 2023-10-30T13:16:16.115 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/30 file after the snapshot 2023-10-30T13:16:16.117 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/31 file after the snapshot 2023-10-30T13:16:16.117 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/32 file after the snapshot 2023-10-30T13:16:16.117 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/33 file after the snapshot 2023-10-30T13:16:16.118 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/34 file after the snapshot 2023-10-30T13:16:16.118 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/35 file after the snapshot 2023-10-30T13:16:16.118 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/36 file after the snapshot 2023-10-30T13:16:16.118 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/37 file after the snapshot 2023-10-30T13:16:16.118 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/38 file after the snapshot 2023-10-30T13:16:16.119 INFO:tasks.workunit.client.0.smithi148.stdout: create 209/39 file after the snapshot 2023-10-30T13:16:16.119 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/.snap/snap-subdir-test 2023-10-30T13:16:16.141 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/30 file after the snapshot 2023-10-30T13:16:16.143 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/31 file after the snapshot 2023-10-30T13:16:16.143 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/32 file after the snapshot 2023-10-30T13:16:16.143 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/33 file after the snapshot 2023-10-30T13:16:16.143 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/34 file after the snapshot 2023-10-30T13:16:16.144 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/35 file after the snapshot 2023-10-30T13:16:16.144 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/36 file after the snapshot 2023-10-30T13:16:16.144 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/37 file after the snapshot 2023-10-30T13:16:16.144 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/38 file after the snapshot 2023-10-30T13:16:16.144 INFO:tasks.workunit.client.0.smithi148.stdout: create 210/39 file after the snapshot 2023-10-30T13:16:16.144 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/.snap/snap-subdir-test 2023-10-30T13:16:16.363 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/30 file after the snapshot 2023-10-30T13:16:16.366 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/31 file after the snapshot 2023-10-30T13:16:16.366 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/32 file after the snapshot 2023-10-30T13:16:16.366 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/33 file after the snapshot 2023-10-30T13:16:16.366 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/34 file after the snapshot 2023-10-30T13:16:16.367 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/35 file after the snapshot 2023-10-30T13:16:16.367 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/36 file after the snapshot 2023-10-30T13:16:16.367 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/37 file after the snapshot 2023-10-30T13:16:16.367 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/38 file after the snapshot 2023-10-30T13:16:16.367 INFO:tasks.workunit.client.0.smithi148.stdout: create 211/39 file after the snapshot 2023-10-30T13:16:16.367 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/.snap/snap-subdir-test 2023-10-30T13:16:16.753 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/30 file after the snapshot 2023-10-30T13:16:16.755 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/31 file after the snapshot 2023-10-30T13:16:16.756 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/32 file after the snapshot 2023-10-30T13:16:16.756 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/33 file after the snapshot 2023-10-30T13:16:16.756 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/34 file after the snapshot 2023-10-30T13:16:16.756 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/35 file after the snapshot 2023-10-30T13:16:16.756 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/36 file after the snapshot 2023-10-30T13:16:16.757 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/37 file after the snapshot 2023-10-30T13:16:16.757 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/38 file after the snapshot 2023-10-30T13:16:16.757 INFO:tasks.workunit.client.0.smithi148.stdout: create 212/39 file after the snapshot 2023-10-30T13:16:16.757 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/.snap/snap-subdir-test 2023-10-30T13:16:16.776 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/30 file after the snapshot 2023-10-30T13:16:16.778 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/31 file after the snapshot 2023-10-30T13:16:16.778 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/32 file after the snapshot 2023-10-30T13:16:16.778 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/33 file after the snapshot 2023-10-30T13:16:16.778 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/34 file after the snapshot 2023-10-30T13:16:16.778 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/35 file after the snapshot 2023-10-30T13:16:16.779 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/36 file after the snapshot 2023-10-30T13:16:16.779 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/37 file after the snapshot 2023-10-30T13:16:16.779 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/38 file after the snapshot 2023-10-30T13:16:16.779 INFO:tasks.workunit.client.0.smithi148.stdout: create 213/39 file after the snapshot 2023-10-30T13:16:16.779 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/.snap/snap-subdir-test 2023-10-30T13:16:16.934 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/30 file after the snapshot 2023-10-30T13:16:16.935 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/31 file after the snapshot 2023-10-30T13:16:16.935 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/32 file after the snapshot 2023-10-30T13:16:16.935 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/33 file after the snapshot 2023-10-30T13:16:16.935 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/34 file after the snapshot 2023-10-30T13:16:16.936 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/35 file after the snapshot 2023-10-30T13:16:16.936 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/36 file after the snapshot 2023-10-30T13:16:16.936 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/37 file after the snapshot 2023-10-30T13:16:16.936 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/38 file after the snapshot 2023-10-30T13:16:16.936 INFO:tasks.workunit.client.0.smithi148.stdout: create 214/39 file after the snapshot 2023-10-30T13:16:16.937 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/.snap/snap-subdir-test 2023-10-30T13:16:17.733 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/30 file after the snapshot 2023-10-30T13:16:17.735 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/31 file after the snapshot 2023-10-30T13:16:17.735 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/32 file after the snapshot 2023-10-30T13:16:17.735 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/33 file after the snapshot 2023-10-30T13:16:17.735 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/34 file after the snapshot 2023-10-30T13:16:17.735 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/35 file after the snapshot 2023-10-30T13:16:17.736 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/36 file after the snapshot 2023-10-30T13:16:17.736 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/37 file after the snapshot 2023-10-30T13:16:17.736 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/38 file after the snapshot 2023-10-30T13:16:17.736 INFO:tasks.workunit.client.0.smithi148.stdout: create 215/39 file after the snapshot 2023-10-30T13:16:17.736 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/.snap/snap-subdir-test 2023-10-30T13:16:18.720 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/30 file after the snapshot 2023-10-30T13:16:18.724 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/31 file after the snapshot 2023-10-30T13:16:18.724 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/32 file after the snapshot 2023-10-30T13:16:18.724 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/33 file after the snapshot 2023-10-30T13:16:18.724 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/34 file after the snapshot 2023-10-30T13:16:18.724 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/35 file after the snapshot 2023-10-30T13:16:18.725 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/36 file after the snapshot 2023-10-30T13:16:18.725 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/37 file after the snapshot 2023-10-30T13:16:18.725 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/38 file after the snapshot 2023-10-30T13:16:18.725 INFO:tasks.workunit.client.0.smithi148.stdout: create 216/39 file after the snapshot 2023-10-30T13:16:18.725 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/.snap/snap-subdir-test 2023-10-30T13:16:19.591 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/30 file after the snapshot 2023-10-30T13:16:19.594 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/31 file after the snapshot 2023-10-30T13:16:19.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/32 file after the snapshot 2023-10-30T13:16:19.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/33 file after the snapshot 2023-10-30T13:16:19.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/34 file after the snapshot 2023-10-30T13:16:19.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/35 file after the snapshot 2023-10-30T13:16:19.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/36 file after the snapshot 2023-10-30T13:16:19.596 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/37 file after the snapshot 2023-10-30T13:16:19.596 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/38 file after the snapshot 2023-10-30T13:16:19.596 INFO:tasks.workunit.client.0.smithi148.stdout: create 217/39 file after the snapshot 2023-10-30T13:16:19.596 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/.snap/snap-subdir-test 2023-10-30T13:16:19.614 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/30 file after the snapshot 2023-10-30T13:16:19.617 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/31 file after the snapshot 2023-10-30T13:16:19.617 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/32 file after the snapshot 2023-10-30T13:16:19.617 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/33 file after the snapshot 2023-10-30T13:16:19.617 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/34 file after the snapshot 2023-10-30T13:16:19.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/35 file after the snapshot 2023-10-30T13:16:19.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/36 file after the snapshot 2023-10-30T13:16:19.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/37 file after the snapshot 2023-10-30T13:16:19.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/38 file after the snapshot 2023-10-30T13:16:19.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 218/39 file after the snapshot 2023-10-30T13:16:19.618 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/.snap/snap-subdir-test 2023-10-30T13:16:19.642 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/30 file after the snapshot 2023-10-30T13:16:19.644 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/31 file after the snapshot 2023-10-30T13:16:19.644 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/32 file after the snapshot 2023-10-30T13:16:19.644 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/33 file after the snapshot 2023-10-30T13:16:19.645 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/34 file after the snapshot 2023-10-30T13:16:19.645 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/35 file after the snapshot 2023-10-30T13:16:19.645 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/36 file after the snapshot 2023-10-30T13:16:19.645 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/37 file after the snapshot 2023-10-30T13:16:19.645 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/38 file after the snapshot 2023-10-30T13:16:19.645 INFO:tasks.workunit.client.0.smithi148.stdout: create 219/39 file after the snapshot 2023-10-30T13:16:19.646 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/.snap/snap-subdir-test 2023-10-30T13:16:19.678 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/30 file after the snapshot 2023-10-30T13:16:19.931 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/31 file after the snapshot 2023-10-30T13:16:19.932 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/32 file after the snapshot 2023-10-30T13:16:19.932 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/33 file after the snapshot 2023-10-30T13:16:19.932 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/34 file after the snapshot 2023-10-30T13:16:19.932 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/35 file after the snapshot 2023-10-30T13:16:19.932 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/36 file after the snapshot 2023-10-30T13:16:19.932 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/37 file after the snapshot 2023-10-30T13:16:19.933 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/38 file after the snapshot 2023-10-30T13:16:19.933 INFO:tasks.workunit.client.0.smithi148.stdout: create 220/39 file after the snapshot 2023-10-30T13:16:19.933 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/.snap/snap-subdir-test 2023-10-30T13:16:19.950 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/30 file after the snapshot 2023-10-30T13:16:19.952 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/31 file after the snapshot 2023-10-30T13:16:19.952 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/32 file after the snapshot 2023-10-30T13:16:19.952 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/33 file after the snapshot 2023-10-30T13:16:19.953 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/34 file after the snapshot 2023-10-30T13:16:19.953 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/35 file after the snapshot 2023-10-30T13:16:19.953 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/36 file after the snapshot 2023-10-30T13:16:19.953 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/37 file after the snapshot 2023-10-30T13:16:19.953 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/38 file after the snapshot 2023-10-30T13:16:19.953 INFO:tasks.workunit.client.0.smithi148.stdout: create 221/39 file after the snapshot 2023-10-30T13:16:19.954 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/.snap/snap-subdir-test 2023-10-30T13:16:19.985 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/30 file after the snapshot 2023-10-30T13:16:19.988 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/31 file after the snapshot 2023-10-30T13:16:19.988 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/32 file after the snapshot 2023-10-30T13:16:19.988 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/33 file after the snapshot 2023-10-30T13:16:19.988 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/34 file after the snapshot 2023-10-30T13:16:19.988 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/35 file after the snapshot 2023-10-30T13:16:19.988 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/36 file after the snapshot 2023-10-30T13:16:19.989 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/37 file after the snapshot 2023-10-30T13:16:19.989 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/38 file after the snapshot 2023-10-30T13:16:19.989 INFO:tasks.workunit.client.0.smithi148.stdout: create 222/39 file after the snapshot 2023-10-30T13:16:19.989 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/.snap/snap-subdir-test 2023-10-30T13:16:20.248 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/30 file after the snapshot 2023-10-30T13:16:20.251 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/31 file after the snapshot 2023-10-30T13:16:20.251 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/32 file after the snapshot 2023-10-30T13:16:20.251 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/33 file after the snapshot 2023-10-30T13:16:20.251 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/34 file after the snapshot 2023-10-30T13:16:20.252 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/35 file after the snapshot 2023-10-30T13:16:20.252 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/36 file after the snapshot 2023-10-30T13:16:20.252 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/37 file after the snapshot 2023-10-30T13:16:20.252 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/38 file after the snapshot 2023-10-30T13:16:20.252 INFO:tasks.workunit.client.0.smithi148.stdout: create 223/39 file after the snapshot 2023-10-30T13:16:20.252 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/.snap/snap-subdir-test 2023-10-30T13:16:21.149 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/30 file after the snapshot 2023-10-30T13:16:21.152 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/31 file after the snapshot 2023-10-30T13:16:21.152 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/32 file after the snapshot 2023-10-30T13:16:21.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/33 file after the snapshot 2023-10-30T13:16:21.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/34 file after the snapshot 2023-10-30T13:16:21.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/35 file after the snapshot 2023-10-30T13:16:21.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/36 file after the snapshot 2023-10-30T13:16:21.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/37 file after the snapshot 2023-10-30T13:16:21.153 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/38 file after the snapshot 2023-10-30T13:16:21.154 INFO:tasks.workunit.client.0.smithi148.stdout: create 224/39 file after the snapshot 2023-10-30T13:16:21.154 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/.snap/snap-subdir-test 2023-10-30T13:16:21.171 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/30 file after the snapshot 2023-10-30T13:16:21.172 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/31 file after the snapshot 2023-10-30T13:16:21.173 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/32 file after the snapshot 2023-10-30T13:16:21.173 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/33 file after the snapshot 2023-10-30T13:16:21.173 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/34 file after the snapshot 2023-10-30T13:16:21.173 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/35 file after the snapshot 2023-10-30T13:16:21.173 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/36 file after the snapshot 2023-10-30T13:16:21.173 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/37 file after the snapshot 2023-10-30T13:16:21.174 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/38 file after the snapshot 2023-10-30T13:16:21.174 INFO:tasks.workunit.client.0.smithi148.stdout: create 225/39 file after the snapshot 2023-10-30T13:16:21.174 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/.snap/snap-subdir-test 2023-10-30T13:16:21.189 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/30 file after the snapshot 2023-10-30T13:16:21.190 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/31 file after the snapshot 2023-10-30T13:16:21.191 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/32 file after the snapshot 2023-10-30T13:16:21.191 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/33 file after the snapshot 2023-10-30T13:16:21.191 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/34 file after the snapshot 2023-10-30T13:16:21.191 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/35 file after the snapshot 2023-10-30T13:16:21.191 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/36 file after the snapshot 2023-10-30T13:16:21.191 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/37 file after the snapshot 2023-10-30T13:16:21.192 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/38 file after the snapshot 2023-10-30T13:16:21.192 INFO:tasks.workunit.client.0.smithi148.stdout: create 226/39 file after the snapshot 2023-10-30T13:16:21.192 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/.snap/snap-subdir-test 2023-10-30T13:16:21.216 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/30 file after the snapshot 2023-10-30T13:16:21.218 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/31 file after the snapshot 2023-10-30T13:16:21.218 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/32 file after the snapshot 2023-10-30T13:16:21.218 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/33 file after the snapshot 2023-10-30T13:16:21.218 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/34 file after the snapshot 2023-10-30T13:16:21.218 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/35 file after the snapshot 2023-10-30T13:16:21.219 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/36 file after the snapshot 2023-10-30T13:16:21.219 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/37 file after the snapshot 2023-10-30T13:16:21.219 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/38 file after the snapshot 2023-10-30T13:16:21.219 INFO:tasks.workunit.client.0.smithi148.stdout: create 227/39 file after the snapshot 2023-10-30T13:16:21.219 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/.snap/snap-subdir-test 2023-10-30T13:16:21.491 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/30 file after the snapshot 2023-10-30T13:16:21.493 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/31 file after the snapshot 2023-10-30T13:16:21.493 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/32 file after the snapshot 2023-10-30T13:16:21.493 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/33 file after the snapshot 2023-10-30T13:16:21.493 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/34 file after the snapshot 2023-10-30T13:16:21.494 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/35 file after the snapshot 2023-10-30T13:16:21.494 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/36 file after the snapshot 2023-10-30T13:16:21.494 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/37 file after the snapshot 2023-10-30T13:16:21.494 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/38 file after the snapshot 2023-10-30T13:16:21.494 INFO:tasks.workunit.client.0.smithi148.stdout: create 228/39 file after the snapshot 2023-10-30T13:16:21.495 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/.snap/snap-subdir-test 2023-10-30T13:16:24.003 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:16:24.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:16:24.989 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/30 file after the snapshot 2023-10-30T13:16:24.991 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/31 file after the snapshot 2023-10-30T13:16:24.991 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/32 file after the snapshot 2023-10-30T13:16:24.992 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/33 file after the snapshot 2023-10-30T13:16:24.992 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/34 file after the snapshot 2023-10-30T13:16:24.992 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/35 file after the snapshot 2023-10-30T13:16:24.992 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/36 file after the snapshot 2023-10-30T13:16:24.992 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/37 file after the snapshot 2023-10-30T13:16:24.992 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/38 file after the snapshot 2023-10-30T13:16:24.993 INFO:tasks.workunit.client.0.smithi148.stdout: create 229/39 file after the snapshot 2023-10-30T13:16:24.993 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/.snap/snap-subdir-test 2023-10-30T13:16:25.268 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/30 file after the snapshot 2023-10-30T13:16:25.270 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/31 file after the snapshot 2023-10-30T13:16:25.270 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/32 file after the snapshot 2023-10-30T13:16:25.270 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/33 file after the snapshot 2023-10-30T13:16:25.270 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/34 file after the snapshot 2023-10-30T13:16:25.271 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/35 file after the snapshot 2023-10-30T13:16:25.271 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/36 file after the snapshot 2023-10-30T13:16:25.271 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/37 file after the snapshot 2023-10-30T13:16:25.271 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/38 file after the snapshot 2023-10-30T13:16:25.271 INFO:tasks.workunit.client.0.smithi148.stdout: create 230/39 file after the snapshot 2023-10-30T13:16:25.272 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/.snap/snap-subdir-test 2023-10-30T13:16:25.287 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/30 file after the snapshot 2023-10-30T13:16:25.290 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/31 file after the snapshot 2023-10-30T13:16:25.290 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/32 file after the snapshot 2023-10-30T13:16:25.290 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/33 file after the snapshot 2023-10-30T13:16:25.290 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/34 file after the snapshot 2023-10-30T13:16:25.290 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/35 file after the snapshot 2023-10-30T13:16:25.290 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/36 file after the snapshot 2023-10-30T13:16:25.291 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/37 file after the snapshot 2023-10-30T13:16:25.291 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/38 file after the snapshot 2023-10-30T13:16:25.291 INFO:tasks.workunit.client.0.smithi148.stdout: create 231/39 file after the snapshot 2023-10-30T13:16:25.291 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/.snap/snap-subdir-test 2023-10-30T13:16:25.315 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/30 file after the snapshot 2023-10-30T13:16:25.317 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/31 file after the snapshot 2023-10-30T13:16:25.317 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/32 file after the snapshot 2023-10-30T13:16:25.318 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/33 file after the snapshot 2023-10-30T13:16:25.318 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/34 file after the snapshot 2023-10-30T13:16:25.318 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/35 file after the snapshot 2023-10-30T13:16:25.318 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/36 file after the snapshot 2023-10-30T13:16:25.318 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/37 file after the snapshot 2023-10-30T13:16:25.319 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/38 file after the snapshot 2023-10-30T13:16:25.319 INFO:tasks.workunit.client.0.smithi148.stdout: create 232/39 file after the snapshot 2023-10-30T13:16:25.319 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/.snap/snap-subdir-test 2023-10-30T13:16:25.344 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/30 file after the snapshot 2023-10-30T13:16:25.345 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/31 file after the snapshot 2023-10-30T13:16:25.346 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/32 file after the snapshot 2023-10-30T13:16:25.346 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/33 file after the snapshot 2023-10-30T13:16:25.346 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/34 file after the snapshot 2023-10-30T13:16:25.346 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/35 file after the snapshot 2023-10-30T13:16:25.346 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/36 file after the snapshot 2023-10-30T13:16:25.346 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/37 file after the snapshot 2023-10-30T13:16:25.347 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/38 file after the snapshot 2023-10-30T13:16:25.347 INFO:tasks.workunit.client.0.smithi148.stdout: create 233/39 file after the snapshot 2023-10-30T13:16:25.347 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/.snap/snap-subdir-test 2023-10-30T13:16:25.892 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/30 file after the snapshot 2023-10-30T13:16:25.895 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/31 file after the snapshot 2023-10-30T13:16:25.895 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/32 file after the snapshot 2023-10-30T13:16:25.895 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/33 file after the snapshot 2023-10-30T13:16:25.895 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/34 file after the snapshot 2023-10-30T13:16:25.896 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/35 file after the snapshot 2023-10-30T13:16:25.896 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/36 file after the snapshot 2023-10-30T13:16:25.896 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/37 file after the snapshot 2023-10-30T13:16:25.896 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/38 file after the snapshot 2023-10-30T13:16:25.896 INFO:tasks.workunit.client.0.smithi148.stdout: create 234/39 file after the snapshot 2023-10-30T13:16:25.897 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/.snap/snap-subdir-test 2023-10-30T13:16:25.915 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/30 file after the snapshot 2023-10-30T13:16:25.918 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/31 file after the snapshot 2023-10-30T13:16:25.919 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/32 file after the snapshot 2023-10-30T13:16:25.919 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/33 file after the snapshot 2023-10-30T13:16:25.919 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/34 file after the snapshot 2023-10-30T13:16:25.919 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/35 file after the snapshot 2023-10-30T13:16:25.919 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/36 file after the snapshot 2023-10-30T13:16:25.920 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/37 file after the snapshot 2023-10-30T13:16:25.920 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/38 file after the snapshot 2023-10-30T13:16:25.920 INFO:tasks.workunit.client.0.smithi148.stdout: create 235/39 file after the snapshot 2023-10-30T13:16:25.920 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/.snap/snap-subdir-test 2023-10-30T13:16:25.963 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/30 file after the snapshot 2023-10-30T13:16:25.966 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/31 file after the snapshot 2023-10-30T13:16:25.967 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/32 file after the snapshot 2023-10-30T13:16:25.967 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/33 file after the snapshot 2023-10-30T13:16:25.967 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/34 file after the snapshot 2023-10-30T13:16:25.967 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/35 file after the snapshot 2023-10-30T13:16:25.967 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/36 file after the snapshot 2023-10-30T13:16:25.968 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/37 file after the snapshot 2023-10-30T13:16:25.968 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/38 file after the snapshot 2023-10-30T13:16:25.968 INFO:tasks.workunit.client.0.smithi148.stdout: create 236/39 file after the snapshot 2023-10-30T13:16:25.968 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/.snap/snap-subdir-test 2023-10-30T13:16:26.433 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/30 file after the snapshot 2023-10-30T13:16:26.434 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/31 file after the snapshot 2023-10-30T13:16:26.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/32 file after the snapshot 2023-10-30T13:16:26.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/33 file after the snapshot 2023-10-30T13:16:26.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/34 file after the snapshot 2023-10-30T13:16:26.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/35 file after the snapshot 2023-10-30T13:16:26.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/36 file after the snapshot 2023-10-30T13:16:26.436 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/37 file after the snapshot 2023-10-30T13:16:26.436 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/38 file after the snapshot 2023-10-30T13:16:26.436 INFO:tasks.workunit.client.0.smithi148.stdout: create 237/39 file after the snapshot 2023-10-30T13:16:26.436 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/.snap/snap-subdir-test 2023-10-30T13:16:26.714 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/30 file after the snapshot 2023-10-30T13:16:26.715 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/31 file after the snapshot 2023-10-30T13:16:26.715 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/32 file after the snapshot 2023-10-30T13:16:26.716 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/33 file after the snapshot 2023-10-30T13:16:26.716 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/34 file after the snapshot 2023-10-30T13:16:26.716 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/35 file after the snapshot 2023-10-30T13:16:26.716 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/36 file after the snapshot 2023-10-30T13:16:26.716 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/37 file after the snapshot 2023-10-30T13:16:26.717 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/38 file after the snapshot 2023-10-30T13:16:26.717 INFO:tasks.workunit.client.0.smithi148.stdout: create 238/39 file after the snapshot 2023-10-30T13:16:26.717 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/.snap/snap-subdir-test 2023-10-30T13:16:26.735 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/30 file after the snapshot 2023-10-30T13:16:26.738 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/31 file after the snapshot 2023-10-30T13:16:26.738 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/32 file after the snapshot 2023-10-30T13:16:26.738 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/33 file after the snapshot 2023-10-30T13:16:26.738 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/34 file after the snapshot 2023-10-30T13:16:26.739 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/35 file after the snapshot 2023-10-30T13:16:26.739 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/36 file after the snapshot 2023-10-30T13:16:26.739 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/37 file after the snapshot 2023-10-30T13:16:26.739 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/38 file after the snapshot 2023-10-30T13:16:26.739 INFO:tasks.workunit.client.0.smithi148.stdout: create 239/39 file after the snapshot 2023-10-30T13:16:26.739 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/.snap/snap-subdir-test 2023-10-30T13:16:26.783 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/30 file after the snapshot 2023-10-30T13:16:26.787 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/31 file after the snapshot 2023-10-30T13:16:26.787 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/32 file after the snapshot 2023-10-30T13:16:26.787 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/33 file after the snapshot 2023-10-30T13:16:26.788 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/34 file after the snapshot 2023-10-30T13:16:26.788 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/35 file after the snapshot 2023-10-30T13:16:26.788 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/36 file after the snapshot 2023-10-30T13:16:26.789 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/37 file after the snapshot 2023-10-30T13:16:26.789 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/38 file after the snapshot 2023-10-30T13:16:26.789 INFO:tasks.workunit.client.0.smithi148.stdout: create 240/39 file after the snapshot 2023-10-30T13:16:26.793 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/.snap/snap-subdir-test 2023-10-30T13:16:26.921 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/30 file after the snapshot 2023-10-30T13:16:26.922 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/31 file after the snapshot 2023-10-30T13:16:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/32 file after the snapshot 2023-10-30T13:16:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/33 file after the snapshot 2023-10-30T13:16:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/34 file after the snapshot 2023-10-30T13:16:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/35 file after the snapshot 2023-10-30T13:16:26.923 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/36 file after the snapshot 2023-10-30T13:16:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/37 file after the snapshot 2023-10-30T13:16:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/38 file after the snapshot 2023-10-30T13:16:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: create 241/39 file after the snapshot 2023-10-30T13:16:26.924 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/.snap/snap-subdir-test 2023-10-30T13:16:27.047 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/30 file after the snapshot 2023-10-30T13:16:27.048 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/31 file after the snapshot 2023-10-30T13:16:27.048 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/32 file after the snapshot 2023-10-30T13:16:27.049 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/33 file after the snapshot 2023-10-30T13:16:27.049 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/34 file after the snapshot 2023-10-30T13:16:27.049 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/35 file after the snapshot 2023-10-30T13:16:27.049 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/36 file after the snapshot 2023-10-30T13:16:27.049 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/37 file after the snapshot 2023-10-30T13:16:27.050 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/38 file after the snapshot 2023-10-30T13:16:27.050 INFO:tasks.workunit.client.0.smithi148.stdout: create 242/39 file after the snapshot 2023-10-30T13:16:27.050 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/.snap/snap-subdir-test 2023-10-30T13:16:27.071 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/30 file after the snapshot 2023-10-30T13:16:27.073 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/31 file after the snapshot 2023-10-30T13:16:27.073 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/32 file after the snapshot 2023-10-30T13:16:27.073 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/33 file after the snapshot 2023-10-30T13:16:27.073 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/34 file after the snapshot 2023-10-30T13:16:27.074 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/35 file after the snapshot 2023-10-30T13:16:27.074 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/36 file after the snapshot 2023-10-30T13:16:27.074 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/37 file after the snapshot 2023-10-30T13:16:27.074 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/38 file after the snapshot 2023-10-30T13:16:27.074 INFO:tasks.workunit.client.0.smithi148.stdout: create 243/39 file after the snapshot 2023-10-30T13:16:27.074 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/.snap/snap-subdir-test 2023-10-30T13:16:27.095 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/30 file after the snapshot 2023-10-30T13:16:27.097 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/31 file after the snapshot 2023-10-30T13:16:27.097 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/32 file after the snapshot 2023-10-30T13:16:27.097 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/33 file after the snapshot 2023-10-30T13:16:27.097 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/34 file after the snapshot 2023-10-30T13:16:27.098 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/35 file after the snapshot 2023-10-30T13:16:27.098 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/36 file after the snapshot 2023-10-30T13:16:27.098 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/37 file after the snapshot 2023-10-30T13:16:27.098 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/38 file after the snapshot 2023-10-30T13:16:27.098 INFO:tasks.workunit.client.0.smithi148.stdout: create 244/39 file after the snapshot 2023-10-30T13:16:27.099 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/.snap/snap-subdir-test 2023-10-30T13:16:27.121 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/30 file after the snapshot 2023-10-30T13:16:27.122 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/31 file after the snapshot 2023-10-30T13:16:27.123 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/32 file after the snapshot 2023-10-30T13:16:27.123 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/33 file after the snapshot 2023-10-30T13:16:27.123 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/34 file after the snapshot 2023-10-30T13:16:27.123 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/35 file after the snapshot 2023-10-30T13:16:27.123 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/36 file after the snapshot 2023-10-30T13:16:27.123 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/37 file after the snapshot 2023-10-30T13:16:27.124 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/38 file after the snapshot 2023-10-30T13:16:27.124 INFO:tasks.workunit.client.0.smithi148.stdout: create 245/39 file after the snapshot 2023-10-30T13:16:27.124 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/.snap/snap-subdir-test 2023-10-30T13:16:27.585 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/30 file after the snapshot 2023-10-30T13:16:27.586 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/31 file after the snapshot 2023-10-30T13:16:27.586 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/32 file after the snapshot 2023-10-30T13:16:27.586 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/33 file after the snapshot 2023-10-30T13:16:27.587 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/34 file after the snapshot 2023-10-30T13:16:27.587 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/35 file after the snapshot 2023-10-30T13:16:27.587 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/36 file after the snapshot 2023-10-30T13:16:27.588 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/37 file after the snapshot 2023-10-30T13:16:27.588 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/38 file after the snapshot 2023-10-30T13:16:27.588 INFO:tasks.workunit.client.0.smithi148.stdout: create 246/39 file after the snapshot 2023-10-30T13:16:27.588 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/.snap/snap-subdir-test 2023-10-30T13:16:27.604 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/30 file after the snapshot 2023-10-30T13:16:27.606 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/31 file after the snapshot 2023-10-30T13:16:27.606 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/32 file after the snapshot 2023-10-30T13:16:27.606 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/33 file after the snapshot 2023-10-30T13:16:27.606 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/34 file after the snapshot 2023-10-30T13:16:27.606 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/35 file after the snapshot 2023-10-30T13:16:27.607 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/36 file after the snapshot 2023-10-30T13:16:27.607 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/37 file after the snapshot 2023-10-30T13:16:27.607 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/38 file after the snapshot 2023-10-30T13:16:27.607 INFO:tasks.workunit.client.0.smithi148.stdout: create 247/39 file after the snapshot 2023-10-30T13:16:27.607 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/.snap/snap-subdir-test 2023-10-30T13:16:27.629 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/30 file after the snapshot 2023-10-30T13:16:27.632 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/31 file after the snapshot 2023-10-30T13:16:27.632 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/32 file after the snapshot 2023-10-30T13:16:27.632 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/33 file after the snapshot 2023-10-30T13:16:27.632 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/34 file after the snapshot 2023-10-30T13:16:27.633 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/35 file after the snapshot 2023-10-30T13:16:27.633 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/36 file after the snapshot 2023-10-30T13:16:27.633 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/37 file after the snapshot 2023-10-30T13:16:27.633 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/38 file after the snapshot 2023-10-30T13:16:27.633 INFO:tasks.workunit.client.0.smithi148.stdout: create 248/39 file after the snapshot 2023-10-30T13:16:27.633 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/.snap/snap-subdir-test 2023-10-30T13:16:27.658 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/30 file after the snapshot 2023-10-30T13:16:27.660 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/31 file after the snapshot 2023-10-30T13:16:27.660 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/32 file after the snapshot 2023-10-30T13:16:27.660 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/33 file after the snapshot 2023-10-30T13:16:27.660 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/34 file after the snapshot 2023-10-30T13:16:27.660 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/35 file after the snapshot 2023-10-30T13:16:27.661 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/36 file after the snapshot 2023-10-30T13:16:27.661 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/37 file after the snapshot 2023-10-30T13:16:27.661 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/38 file after the snapshot 2023-10-30T13:16:27.661 INFO:tasks.workunit.client.0.smithi148.stdout: create 249/39 file after the snapshot 2023-10-30T13:16:27.661 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/.snap/snap-subdir-test 2023-10-30T13:16:27.684 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/30 file after the snapshot 2023-10-30T13:16:27.686 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/31 file after the snapshot 2023-10-30T13:16:27.686 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/32 file after the snapshot 2023-10-30T13:16:27.686 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/33 file after the snapshot 2023-10-30T13:16:27.687 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/34 file after the snapshot 2023-10-30T13:16:27.687 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/35 file after the snapshot 2023-10-30T13:16:27.687 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/36 file after the snapshot 2023-10-30T13:16:27.687 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/37 file after the snapshot 2023-10-30T13:16:27.687 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/38 file after the snapshot 2023-10-30T13:16:27.687 INFO:tasks.workunit.client.0.smithi148.stdout: create 250/39 file after the snapshot 2023-10-30T13:16:27.688 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/.snap/snap-subdir-test 2023-10-30T13:16:27.711 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/30 file after the snapshot 2023-10-30T13:16:27.713 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/31 file after the snapshot 2023-10-30T13:16:27.714 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/32 file after the snapshot 2023-10-30T13:16:27.714 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/33 file after the snapshot 2023-10-30T13:16:27.714 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/34 file after the snapshot 2023-10-30T13:16:27.714 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/35 file after the snapshot 2023-10-30T13:16:27.715 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/36 file after the snapshot 2023-10-30T13:16:27.715 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/37 file after the snapshot 2023-10-30T13:16:27.715 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/38 file after the snapshot 2023-10-30T13:16:27.715 INFO:tasks.workunit.client.0.smithi148.stdout: create 251/39 file after the snapshot 2023-10-30T13:16:27.715 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/.snap/snap-subdir-test 2023-10-30T13:16:27.760 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/30 file after the snapshot 2023-10-30T13:16:27.763 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/31 file after the snapshot 2023-10-30T13:16:27.763 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/32 file after the snapshot 2023-10-30T13:16:27.763 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/33 file after the snapshot 2023-10-30T13:16:27.764 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/34 file after the snapshot 2023-10-30T13:16:27.764 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/35 file after the snapshot 2023-10-30T13:16:27.764 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/36 file after the snapshot 2023-10-30T13:16:27.764 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/37 file after the snapshot 2023-10-30T13:16:27.764 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/38 file after the snapshot 2023-10-30T13:16:27.764 INFO:tasks.workunit.client.0.smithi148.stdout: create 252/39 file after the snapshot 2023-10-30T13:16:27.765 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/.snap/snap-subdir-test 2023-10-30T13:16:27.786 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/30 file after the snapshot 2023-10-30T13:16:27.788 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/31 file after the snapshot 2023-10-30T13:16:27.788 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/32 file after the snapshot 2023-10-30T13:16:27.788 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/33 file after the snapshot 2023-10-30T13:16:27.788 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/34 file after the snapshot 2023-10-30T13:16:27.789 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/35 file after the snapshot 2023-10-30T13:16:27.789 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/36 file after the snapshot 2023-10-30T13:16:27.789 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/37 file after the snapshot 2023-10-30T13:16:27.789 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/38 file after the snapshot 2023-10-30T13:16:27.789 INFO:tasks.workunit.client.0.smithi148.stdout: create 253/39 file after the snapshot 2023-10-30T13:16:27.790 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/.snap/snap-subdir-test 2023-10-30T13:16:27.821 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/30 file after the snapshot 2023-10-30T13:16:27.824 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/31 file after the snapshot 2023-10-30T13:16:27.824 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/32 file after the snapshot 2023-10-30T13:16:27.824 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/33 file after the snapshot 2023-10-30T13:16:27.824 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/34 file after the snapshot 2023-10-30T13:16:27.825 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/35 file after the snapshot 2023-10-30T13:16:27.825 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/36 file after the snapshot 2023-10-30T13:16:27.825 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/37 file after the snapshot 2023-10-30T13:16:27.825 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/38 file after the snapshot 2023-10-30T13:16:27.825 INFO:tasks.workunit.client.0.smithi148.stdout: create 254/39 file after the snapshot 2023-10-30T13:16:27.825 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/.snap/snap-subdir-test 2023-10-30T13:16:27.979 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/30 file after the snapshot 2023-10-30T13:16:27.980 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/31 file after the snapshot 2023-10-30T13:16:27.981 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/32 file after the snapshot 2023-10-30T13:16:27.981 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/33 file after the snapshot 2023-10-30T13:16:27.981 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/34 file after the snapshot 2023-10-30T13:16:27.981 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/35 file after the snapshot 2023-10-30T13:16:27.982 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/36 file after the snapshot 2023-10-30T13:16:27.982 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/37 file after the snapshot 2023-10-30T13:16:27.982 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/38 file after the snapshot 2023-10-30T13:16:27.982 INFO:tasks.workunit.client.0.smithi148.stdout: create 255/39 file after the snapshot 2023-10-30T13:16:27.982 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/.snap/snap-subdir-test 2023-10-30T13:16:27.998 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/30 file after the snapshot 2023-10-30T13:16:28.000 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/31 file after the snapshot 2023-10-30T13:16:28.000 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/32 file after the snapshot 2023-10-30T13:16:28.000 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/33 file after the snapshot 2023-10-30T13:16:28.000 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/34 file after the snapshot 2023-10-30T13:16:28.001 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/35 file after the snapshot 2023-10-30T13:16:28.001 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/36 file after the snapshot 2023-10-30T13:16:28.001 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/37 file after the snapshot 2023-10-30T13:16:28.001 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/38 file after the snapshot 2023-10-30T13:16:28.001 INFO:tasks.workunit.client.0.smithi148.stdout: create 256/39 file after the snapshot 2023-10-30T13:16:28.002 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/.snap/snap-subdir-test 2023-10-30T13:16:28.032 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/30 file after the snapshot 2023-10-30T13:16:28.035 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/31 file after the snapshot 2023-10-30T13:16:28.035 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/32 file after the snapshot 2023-10-30T13:16:28.036 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/33 file after the snapshot 2023-10-30T13:16:28.036 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/34 file after the snapshot 2023-10-30T13:16:28.036 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/35 file after the snapshot 2023-10-30T13:16:28.036 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/36 file after the snapshot 2023-10-30T13:16:28.036 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/37 file after the snapshot 2023-10-30T13:16:28.037 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/38 file after the snapshot 2023-10-30T13:16:28.037 INFO:tasks.workunit.client.0.smithi148.stdout: create 257/39 file after the snapshot 2023-10-30T13:16:28.037 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/.snap/snap-subdir-test 2023-10-30T13:16:28.274 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/30 file after the snapshot 2023-10-30T13:16:28.275 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/31 file after the snapshot 2023-10-30T13:16:28.276 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/32 file after the snapshot 2023-10-30T13:16:28.276 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/33 file after the snapshot 2023-10-30T13:16:28.276 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/34 file after the snapshot 2023-10-30T13:16:28.276 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/35 file after the snapshot 2023-10-30T13:16:28.276 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/36 file after the snapshot 2023-10-30T13:16:28.277 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/37 file after the snapshot 2023-10-30T13:16:28.277 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/38 file after the snapshot 2023-10-30T13:16:28.277 INFO:tasks.workunit.client.0.smithi148.stdout: create 258/39 file after the snapshot 2023-10-30T13:16:28.277 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/.snap/snap-subdir-test 2023-10-30T13:16:28.311 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/30 file after the snapshot 2023-10-30T13:16:28.314 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/31 file after the snapshot 2023-10-30T13:16:28.314 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/32 file after the snapshot 2023-10-30T13:16:28.314 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/33 file after the snapshot 2023-10-30T13:16:28.315 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/34 file after the snapshot 2023-10-30T13:16:28.315 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/35 file after the snapshot 2023-10-30T13:16:28.315 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/36 file after the snapshot 2023-10-30T13:16:28.315 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/37 file after the snapshot 2023-10-30T13:16:28.315 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/38 file after the snapshot 2023-10-30T13:16:28.316 INFO:tasks.workunit.client.0.smithi148.stdout: create 259/39 file after the snapshot 2023-10-30T13:16:28.316 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/.snap/snap-subdir-test 2023-10-30T13:16:28.402 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/30 file after the snapshot 2023-10-30T13:16:28.403 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/31 file after the snapshot 2023-10-30T13:16:28.403 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/32 file after the snapshot 2023-10-30T13:16:28.404 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/33 file after the snapshot 2023-10-30T13:16:28.404 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/34 file after the snapshot 2023-10-30T13:16:28.404 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/35 file after the snapshot 2023-10-30T13:16:28.404 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/36 file after the snapshot 2023-10-30T13:16:28.404 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/37 file after the snapshot 2023-10-30T13:16:28.404 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/38 file after the snapshot 2023-10-30T13:16:28.405 INFO:tasks.workunit.client.0.smithi148.stdout: create 260/39 file after the snapshot 2023-10-30T13:16:28.405 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/.snap/snap-subdir-test 2023-10-30T13:16:28.432 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/30 file after the snapshot 2023-10-30T13:16:28.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/31 file after the snapshot 2023-10-30T13:16:28.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/32 file after the snapshot 2023-10-30T13:16:28.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/33 file after the snapshot 2023-10-30T13:16:28.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/34 file after the snapshot 2023-10-30T13:16:28.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/35 file after the snapshot 2023-10-30T13:16:28.436 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/36 file after the snapshot 2023-10-30T13:16:28.436 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/37 file after the snapshot 2023-10-30T13:16:28.436 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/38 file after the snapshot 2023-10-30T13:16:28.436 INFO:tasks.workunit.client.0.smithi148.stdout: create 261/39 file after the snapshot 2023-10-30T13:16:28.436 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/.snap/snap-subdir-test 2023-10-30T13:16:29.285 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/30 file after the snapshot 2023-10-30T13:16:29.287 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/31 file after the snapshot 2023-10-30T13:16:29.287 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/32 file after the snapshot 2023-10-30T13:16:29.287 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/33 file after the snapshot 2023-10-30T13:16:29.288 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/34 file after the snapshot 2023-10-30T13:16:29.288 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/35 file after the snapshot 2023-10-30T13:16:29.288 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/36 file after the snapshot 2023-10-30T13:16:29.288 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/37 file after the snapshot 2023-10-30T13:16:29.288 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/38 file after the snapshot 2023-10-30T13:16:29.289 INFO:tasks.workunit.client.0.smithi148.stdout: create 262/39 file after the snapshot 2023-10-30T13:16:29.289 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/.snap/snap-subdir-test 2023-10-30T13:16:29.320 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/30 file after the snapshot 2023-10-30T13:16:29.323 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/31 file after the snapshot 2023-10-30T13:16:29.324 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/32 file after the snapshot 2023-10-30T13:16:29.324 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/33 file after the snapshot 2023-10-30T13:16:29.324 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/34 file after the snapshot 2023-10-30T13:16:29.324 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/35 file after the snapshot 2023-10-30T13:16:29.324 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/36 file after the snapshot 2023-10-30T13:16:29.325 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/37 file after the snapshot 2023-10-30T13:16:29.325 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/38 file after the snapshot 2023-10-30T13:16:29.325 INFO:tasks.workunit.client.0.smithi148.stdout: create 263/39 file after the snapshot 2023-10-30T13:16:29.325 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/.snap/snap-subdir-test 2023-10-30T13:16:29.371 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/30 file after the snapshot 2023-10-30T13:16:29.374 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/31 file after the snapshot 2023-10-30T13:16:29.374 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/32 file after the snapshot 2023-10-30T13:16:29.375 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/33 file after the snapshot 2023-10-30T13:16:29.375 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/34 file after the snapshot 2023-10-30T13:16:29.375 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/35 file after the snapshot 2023-10-30T13:16:29.375 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/36 file after the snapshot 2023-10-30T13:16:29.375 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/37 file after the snapshot 2023-10-30T13:16:29.376 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/38 file after the snapshot 2023-10-30T13:16:29.376 INFO:tasks.workunit.client.0.smithi148.stdout: create 264/39 file after the snapshot 2023-10-30T13:16:29.376 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/.snap/snap-subdir-test 2023-10-30T13:16:29.397 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/30 file after the snapshot 2023-10-30T13:16:29.399 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/31 file after the snapshot 2023-10-30T13:16:29.399 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/32 file after the snapshot 2023-10-30T13:16:29.399 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/33 file after the snapshot 2023-10-30T13:16:29.400 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/34 file after the snapshot 2023-10-30T13:16:29.400 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/35 file after the snapshot 2023-10-30T13:16:29.400 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/36 file after the snapshot 2023-10-30T13:16:29.400 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/37 file after the snapshot 2023-10-30T13:16:29.400 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/38 file after the snapshot 2023-10-30T13:16:29.401 INFO:tasks.workunit.client.0.smithi148.stdout: create 265/39 file after the snapshot 2023-10-30T13:16:29.401 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/.snap/snap-subdir-test 2023-10-30T13:16:29.435 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/30 file after the snapshot 2023-10-30T13:16:29.438 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/31 file after the snapshot 2023-10-30T13:16:29.438 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/32 file after the snapshot 2023-10-30T13:16:29.438 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/33 file after the snapshot 2023-10-30T13:16:29.439 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/34 file after the snapshot 2023-10-30T13:16:29.439 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/35 file after the snapshot 2023-10-30T13:16:29.439 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/36 file after the snapshot 2023-10-30T13:16:29.439 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/37 file after the snapshot 2023-10-30T13:16:29.439 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/38 file after the snapshot 2023-10-30T13:16:29.440 INFO:tasks.workunit.client.0.smithi148.stdout: create 266/39 file after the snapshot 2023-10-30T13:16:29.440 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/.snap/snap-subdir-test 2023-10-30T13:16:29.462 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/30 file after the snapshot 2023-10-30T13:16:29.465 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/31 file after the snapshot 2023-10-30T13:16:29.465 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/32 file after the snapshot 2023-10-30T13:16:29.465 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/33 file after the snapshot 2023-10-30T13:16:29.465 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/34 file after the snapshot 2023-10-30T13:16:29.466 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/35 file after the snapshot 2023-10-30T13:16:29.466 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/36 file after the snapshot 2023-10-30T13:16:29.466 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/37 file after the snapshot 2023-10-30T13:16:29.466 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/38 file after the snapshot 2023-10-30T13:16:29.467 INFO:tasks.workunit.client.0.smithi148.stdout: create 267/39 file after the snapshot 2023-10-30T13:16:29.467 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/.snap/snap-subdir-test 2023-10-30T13:16:30.135 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/30 file after the snapshot 2023-10-30T13:16:30.136 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/31 file after the snapshot 2023-10-30T13:16:30.136 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/32 file after the snapshot 2023-10-30T13:16:30.136 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/33 file after the snapshot 2023-10-30T13:16:30.137 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/34 file after the snapshot 2023-10-30T13:16:30.137 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/35 file after the snapshot 2023-10-30T13:16:30.138 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/36 file after the snapshot 2023-10-30T13:16:30.138 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/37 file after the snapshot 2023-10-30T13:16:30.138 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/38 file after the snapshot 2023-10-30T13:16:30.138 INFO:tasks.workunit.client.0.smithi148.stdout: create 268/39 file after the snapshot 2023-10-30T13:16:30.138 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/.snap/snap-subdir-test 2023-10-30T13:16:30.169 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/30 file after the snapshot 2023-10-30T13:16:30.171 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/31 file after the snapshot 2023-10-30T13:16:30.171 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/32 file after the snapshot 2023-10-30T13:16:30.171 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/33 file after the snapshot 2023-10-30T13:16:30.172 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/34 file after the snapshot 2023-10-30T13:16:30.172 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/35 file after the snapshot 2023-10-30T13:16:30.172 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/36 file after the snapshot 2023-10-30T13:16:30.172 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/37 file after the snapshot 2023-10-30T13:16:30.172 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/38 file after the snapshot 2023-10-30T13:16:30.173 INFO:tasks.workunit.client.0.smithi148.stdout: create 269/39 file after the snapshot 2023-10-30T13:16:30.173 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/.snap/snap-subdir-test 2023-10-30T13:16:30.198 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/30 file after the snapshot 2023-10-30T13:16:30.199 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/31 file after the snapshot 2023-10-30T13:16:30.199 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/32 file after the snapshot 2023-10-30T13:16:30.200 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/33 file after the snapshot 2023-10-30T13:16:30.200 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/34 file after the snapshot 2023-10-30T13:16:30.200 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/35 file after the snapshot 2023-10-30T13:16:30.200 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/36 file after the snapshot 2023-10-30T13:16:30.200 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/37 file after the snapshot 2023-10-30T13:16:30.201 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/38 file after the snapshot 2023-10-30T13:16:30.201 INFO:tasks.workunit.client.0.smithi148.stdout: create 270/39 file after the snapshot 2023-10-30T13:16:30.201 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/.snap/snap-subdir-test 2023-10-30T13:16:30.233 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/30 file after the snapshot 2023-10-30T13:16:30.237 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/31 file after the snapshot 2023-10-30T13:16:30.237 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/32 file after the snapshot 2023-10-30T13:16:30.237 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/33 file after the snapshot 2023-10-30T13:16:30.238 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/34 file after the snapshot 2023-10-30T13:16:30.238 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/35 file after the snapshot 2023-10-30T13:16:30.238 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/36 file after the snapshot 2023-10-30T13:16:30.238 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/37 file after the snapshot 2023-10-30T13:16:30.238 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/38 file after the snapshot 2023-10-30T13:16:30.239 INFO:tasks.workunit.client.0.smithi148.stdout: create 271/39 file after the snapshot 2023-10-30T13:16:30.239 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/.snap/snap-subdir-test 2023-10-30T13:16:30.507 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/30 file after the snapshot 2023-10-30T13:16:30.508 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/31 file after the snapshot 2023-10-30T13:16:30.509 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/32 file after the snapshot 2023-10-30T13:16:30.509 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/33 file after the snapshot 2023-10-30T13:16:30.509 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/34 file after the snapshot 2023-10-30T13:16:30.509 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/35 file after the snapshot 2023-10-30T13:16:30.509 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/36 file after the snapshot 2023-10-30T13:16:30.510 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/37 file after the snapshot 2023-10-30T13:16:30.510 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/38 file after the snapshot 2023-10-30T13:16:30.510 INFO:tasks.workunit.client.0.smithi148.stdout: create 272/39 file after the snapshot 2023-10-30T13:16:30.511 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/.snap/snap-subdir-test 2023-10-30T13:16:30.527 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/30 file after the snapshot 2023-10-30T13:16:30.529 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/31 file after the snapshot 2023-10-30T13:16:30.529 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/32 file after the snapshot 2023-10-30T13:16:30.529 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/33 file after the snapshot 2023-10-30T13:16:30.530 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/34 file after the snapshot 2023-10-30T13:16:30.530 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/35 file after the snapshot 2023-10-30T13:16:30.530 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/36 file after the snapshot 2023-10-30T13:16:30.530 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/37 file after the snapshot 2023-10-30T13:16:30.530 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/38 file after the snapshot 2023-10-30T13:16:30.530 INFO:tasks.workunit.client.0.smithi148.stdout: create 273/39 file after the snapshot 2023-10-30T13:16:30.531 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/.snap/snap-subdir-test 2023-10-30T13:16:30.556 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/30 file after the snapshot 2023-10-30T13:16:30.557 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/31 file after the snapshot 2023-10-30T13:16:30.557 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/32 file after the snapshot 2023-10-30T13:16:30.558 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/33 file after the snapshot 2023-10-30T13:16:30.558 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/34 file after the snapshot 2023-10-30T13:16:30.558 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/35 file after the snapshot 2023-10-30T13:16:30.558 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/36 file after the snapshot 2023-10-30T13:16:30.558 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/37 file after the snapshot 2023-10-30T13:16:30.559 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/38 file after the snapshot 2023-10-30T13:16:30.559 INFO:tasks.workunit.client.0.smithi148.stdout: create 274/39 file after the snapshot 2023-10-30T13:16:30.559 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/.snap/snap-subdir-test 2023-10-30T13:16:30.590 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/30 file after the snapshot 2023-10-30T13:16:30.594 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/31 file after the snapshot 2023-10-30T13:16:30.594 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/32 file after the snapshot 2023-10-30T13:16:30.594 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/33 file after the snapshot 2023-10-30T13:16:30.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/34 file after the snapshot 2023-10-30T13:16:30.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/35 file after the snapshot 2023-10-30T13:16:30.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/36 file after the snapshot 2023-10-30T13:16:30.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/37 file after the snapshot 2023-10-30T13:16:30.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/38 file after the snapshot 2023-10-30T13:16:30.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 275/39 file after the snapshot 2023-10-30T13:16:30.596 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/.snap/snap-subdir-test 2023-10-30T13:16:31.061 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:31.179 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/30 file after the snapshot 2023-10-30T13:16:31.183 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/31 file after the snapshot 2023-10-30T13:16:31.183 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/32 file after the snapshot 2023-10-30T13:16:31.183 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/33 file after the snapshot 2023-10-30T13:16:31.184 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/34 file after the snapshot 2023-10-30T13:16:31.184 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/35 file after the snapshot 2023-10-30T13:16:31.184 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/36 file after the snapshot 2023-10-30T13:16:31.184 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/37 file after the snapshot 2023-10-30T13:16:31.184 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/38 file after the snapshot 2023-10-30T13:16:31.185 INFO:tasks.workunit.client.0.smithi148.stdout: create 276/39 file after the snapshot 2023-10-30T13:16:31.185 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/.snap/snap-subdir-test 2023-10-30T13:16:31.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.201+0000 7f3d670d6640 1 Processor -- start 2023-10-30T13:16:31.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.201+0000 7f3d670d6640 1 -- start start 2023-10-30T13:16:31.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d670d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3d601538a0 0x7f3d6015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d670d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3d600cf160 0x7f3d600cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d670d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d600cb800 con 0x7f3d600ca0e0 2023-10-30T13:16:31.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d670d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6015dc30 con 0x7f3d601538a0 2023-10-30T13:16:31.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d670d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d600cb330 con 0x7f3d600cf160 2023-10-30T13:16:31.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d64e72640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3d601538a0 0x7f3d6015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d5ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3d600ca0e0 0x7f3d6015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36790/0 (socket says 172.21.15.148:36790) 2023-10-30T13:16:31.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d5ffff640 1 -- 172.21.15.148:0/601721263 learned_addr learned my addr 172.21.15.148:0/601721263 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:16:31.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d65673640 1 --2- 172.21.15.148:0/601721263 >> v2:172.21.15.150:3300/0 conn(0x7f3d600cf160 0x7f3d600cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d5f7fe640 1 -- 172.21.15.148:0/601721263 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431862797 0 0) 0x7f3d600cb800 con 0x7f3d600ca0e0 2023-10-30T13:16:31.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d64e72640 1 -- 172.21.15.148:0/601721263 >> v2:172.21.15.150:3300/0 conn(0x7f3d600cf160 msgr2=0x7f3d600cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:31.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d64e72640 1 --2- 172.21.15.148:0/601721263 >> v2:172.21.15.150:3300/0 conn(0x7f3d600cf160 0x7f3d600cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d64e72640 1 -- 172.21.15.148:0/601721263 >> v1:172.21.15.148:6789/0 conn(0x7f3d600ca0e0 legacy=0x7f3d6015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.205+0000 7f3d64e72640 1 -- 172.21.15.148:0/601721263 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d600cad70 con 0x7f3d601538a0 2023-10-30T13:16:31.209 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/30 file after the snapshot 2023-10-30T13:16:31.209 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/31 file after the snapshot 2023-10-30T13:16:31.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d64e72640 1 --2- 172.21.15.148:0/601721263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d601538a0 0x7f3d6015d6f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3d600b5330 tx=0x7f3d5401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b2f5ae8d9d88 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:31.210 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/32 file after the snapshot 2023-10-30T13:16:31.210 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/33 file after the snapshot 2023-10-30T13:16:31.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d5f7fe640 1 -- 172.21.15.148:0/601721263 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d540a8950 con 0x7f3d601538a0 2023-10-30T13:16:31.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d5f7fe640 1 -- 172.21.15.148:0/601721263 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d540a8ab0 con 0x7f3d601538a0 2023-10-30T13:16:31.211 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/34 file after the snapshot 2023-10-30T13:16:31.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d5f7fe640 1 -- 172.21.15.148:0/601721263 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d540a8e40 con 0x7f3d601538a0 2023-10-30T13:16:31.212 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/35 file after the snapshot 2023-10-30T13:16:31.212 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/36 file after the snapshot 2023-10-30T13:16:31.213 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/37 file after the snapshot 2023-10-30T13:16:31.213 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/38 file after the snapshot 2023-10-30T13:16:31.213 INFO:tasks.workunit.client.0.smithi148.stdout: create 277/39 file after the snapshot 2023-10-30T13:16:31.213 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/.snap/snap-subdir-test 2023-10-30T13:16:31.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d670d6640 1 -- 172.21.15.148:0/601721263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d601538a0 msgr2=0x7f3d6015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d670d6640 1 --2- 172.21.15.148:0/601721263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d601538a0 0x7f3d6015d6f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3d600b5330 tx=0x7f3d5401ec50 comp rx=0 tx=0).stop 2023-10-30T13:16:31.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d670d6640 1 -- 172.21.15.148:0/601721263 shutdown_connections 2023-10-30T13:16:31.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d670d6640 1 --2- 172.21.15.148:0/601721263 >> v2:172.21.15.150:3300/0 conn(0x7f3d600cf160 0x7f3d600cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d670d6640 1 --2- 172.21.15.148:0/601721263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d601538a0 0x7f3d6015d6f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d670d6640 1 -- 172.21.15.148:0/601721263 >> 172.21.15.148:0/601721263 conn(0x7f3d6006bfe0 msgr2=0x7f3d6006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:31.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d670d6640 1 -- 172.21.15.148:0/601721263 shutdown_connections 2023-10-30T13:16:31.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.209+0000 7f3d670d6640 1 -- 172.21.15.148:0/601721263 wait complete. 2023-10-30T13:16:31.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 Processor -- start 2023-10-30T13:16:31.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 -- start start 2023-10-30T13:16:31.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d600ca0e0 0x7f3d6015b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d600cf160 0x7f3d600b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d601538a0 0x7f3d601ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6014ecd0 con 0x7f3d600cf160 2023-10-30T13:16:31.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6014e750 con 0x7f3d600ca0e0 2023-10-30T13:16:31.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d601f3380 con 0x7f3d601538a0 2023-10-30T13:16:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d600cf160 0x7f3d600b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d600cf160 0x7f3d600b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42276/0 (socket says 172.21.15.148:42276) 2023-10-30T13:16:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5ffff640 1 -- 172.21.15.148:0/2196211589 learned_addr learned my addr 172.21.15.148:0/2196211589 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d64e72640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d600ca0e0 0x7f3d6015b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d65673640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d601538a0 0x7f3d601ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5ffff640 1 -- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d601538a0 msgr2=0x7f3d601ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5ffff640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d601538a0 0x7f3d601ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5ffff640 1 -- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d600ca0e0 msgr2=0x7f3d6015b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5ffff640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d600ca0e0 0x7f3d6015b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5ffff640 1 -- 172.21.15.148:0/2196211589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d54009420 con 0x7f3d600cf160 2023-10-30T13:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d65673640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d601538a0 0x7f3d601ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:31.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5ffff640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d600cf160 0x7f3d600b4580 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c480 tx=0x7f3d4c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:31.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5d7fa640 1 -- 172.21.15.148:0/2196211589 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c0074e0 con 0x7f3d600cf160 2023-10-30T13:16:31.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5d7fa640 1 -- 172.21.15.148:0/2196211589 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c007660 con 0x7f3d600cf160 2023-10-30T13:16:31.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d5d7fa640 1 -- 172.21.15.148:0/2196211589 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c004a30 con 0x7f3d600cf160 2023-10-30T13:16:31.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6014e9f0 con 0x7f3d600cf160 2023-10-30T13:16:31.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.213+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d600b1b60 con 0x7f3d600cf160 2023-10-30T13:16:31.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.217+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d600b2a50 con 0x7f3d600cf160 2023-10-30T13:16:31.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.217+0000 7f3d5d7fa640 1 -- 172.21.15.148:0/2196211589 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d4c004b90 con 0x7f3d600cf160 2023-10-30T13:16:31.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.217+0000 7f3d5d7fa640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d440664a0 0x7f3d44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.217+0000 7f3d5d7fa640 1 -- 172.21.15.148:0/2196211589 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f3d4c083070 con 0x7f3d600cf160 2023-10-30T13:16:31.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.217+0000 7f3d64e72640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d440664a0 0x7f3d44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.217+0000 7f3d64e72640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d440664a0 0x7f3d44068960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3d540b2000 tx=0x7f3d54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:31.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.221+0000 7f3d5d7fa640 1 -- 172.21.15.148:0/2196211589 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d4c053f80 con 0x7f3d600cf160 2023-10-30T13:16:31.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.405+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d601ef280 con 0x7f3d600cf160 2023-10-30T13:16:31.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.405+0000 7f3d5d7fa640 1 -- 172.21.15.148:0/2196211589 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6539 (secure 0 0 0) 0x7f3d601ef280 con 0x7f3d600cf160 2023-10-30T13:16:31.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:31.409 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:11.413204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:31.409 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 28 2023-10-30T13:16:31.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d440664a0 msgr2=0x7f3d44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d440664a0 0x7f3d44068960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3d540b2000 tx=0x7f3d54008040 comp rx=0 tx=0).stop 2023-10-30T13:16:31.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d600cf160 msgr2=0x7f3d600b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d600cf160 0x7f3d600b4580 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c480 tx=0x7f3d4c00c940 comp rx=0 tx=0).stop 2023-10-30T13:16:31.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 shutdown_connections 2023-10-30T13:16:31.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d440664a0 0x7f3d44068960 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d600cf160 0x7f3d600b4580 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d601538a0 0x7f3d601ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 --2- 172.21.15.148:0/2196211589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d600ca0e0 0x7f3d6015b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 >> 172.21.15.148:0/2196211589 conn(0x7f3d6006bfe0 msgr2=0x7f3d60065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:31.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 shutdown_connections 2023-10-30T13:16:31.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.409+0000 7f3d670d6640 1 -- 172.21.15.148:0/2196211589 wait complete. 2023-10-30T13:16:31.430 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=1) 2023-10-30T13:16:31.431 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:16:31.431 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:16:31.446 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:16:31.446 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:31.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.581+0000 7fa68a594640 1 Processor -- start 2023-10-30T13:16:31.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.581+0000 7fa68a594640 1 -- start start 2023-10-30T13:16:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa68a594640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6841539a0 0x7fa68415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa68a594640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa68415a8c0 con 0x7fa684070a00 2023-10-30T13:16:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa68a594640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa684153d80 con 0x7fa684069e90 2023-10-30T13:16:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa68a594640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68406fbb0 con 0x7fa6841539a0 2023-10-30T13:16:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa6837fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa684070a00 0x7fa684070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36802/0 (socket says 172.21.15.148:36802) 2023-10-30T13:16:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa6837fe640 1 -- 172.21.15.148:0/4181597803 learned_addr learned my addr 172.21.15.148:0/4181597803 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:16:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa683fff640 1 --2- 172.21.15.148:0/4181597803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6841539a0 0x7fa68415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa682ffd640 1 -- 172.21.15.148:0/4181597803 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284051243 0 0) 0x7fa68415a8c0 con 0x7fa684070a00 2023-10-30T13:16:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa682ffd640 1 -- 172.21.15.148:0/4181597803 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa66c003530 con 0x7fa684070a00 2023-10-30T13:16:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa682ffd640 1 -- 172.21.15.148:0/4181597803 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966028448 0 0) 0x7fa684153d80 con 0x7fa684069e90 2023-10-30T13:16:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa683fff640 1 -- 172.21.15.148:0/4181597803 >> v1:172.21.15.150:6789/0 conn(0x7fa684069e90 legacy=0x7fa68406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa683fff640 1 -- 172.21.15.148:0/4181597803 >> v1:172.21.15.148:6789/0 conn(0x7fa684070a00 legacy=0x7fa684070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.585+0000 7fa683fff640 1 -- 172.21.15.148:0/4181597803 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6840c5c50 con 0x7fa6841539a0 2023-10-30T13:16:31.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.589+0000 7fa683fff640 1 --2- 172.21.15.148:0/4181597803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6841539a0 0x7fa68415a380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa6740097f0 tx=0x7fa67401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c4813b16eebe13f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:31.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa682ffd640 1 -- 172.21.15.148:0/4181597803 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6740ab600 con 0x7fa6841539a0 2023-10-30T13:16:31.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa682ffd640 1 -- 172.21.15.148:0/4181597803 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6740ab760 con 0x7fa6841539a0 2023-10-30T13:16:31.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa682ffd640 1 -- 172.21.15.148:0/4181597803 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6740abaf0 con 0x7fa6841539a0 2023-10-30T13:16:31.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa68a594640 1 -- 172.21.15.148:0/4181597803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6841539a0 msgr2=0x7fa68415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa68a594640 1 --2- 172.21.15.148:0/4181597803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6841539a0 0x7fa68415a380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa6740097f0 tx=0x7fa67401c400 comp rx=0 tx=0).stop 2023-10-30T13:16:31.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa68a594640 1 -- 172.21.15.148:0/4181597803 shutdown_connections 2023-10-30T13:16:31.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa68a594640 1 --2- 172.21.15.148:0/4181597803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6841539a0 0x7fa68415a380 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa68a594640 1 -- 172.21.15.148:0/4181597803 >> 172.21.15.148:0/4181597803 conn(0x7fa6840c7910 msgr2=0x7fa6840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:31.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa68a594640 1 -- 172.21.15.148:0/4181597803 shutdown_connections 2023-10-30T13:16:31.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa68a594640 1 -- 172.21.15.148:0/4181597803 wait complete. 2023-10-30T13:16:31.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.593+0000 7fa68a594640 1 Processor -- start 2023-10-30T13:16:31.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 -- start start 2023-10-30T13:16:31.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa684069e90 0x7fa6841f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa684070a00 0x7fa6841456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841539a0 0x7fa684143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6841f2190 con 0x7fa6841539a0 2023-10-30T13:16:31.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6841f1e60 con 0x7fa684069e90 2023-10-30T13:16:31.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6841462a0 con 0x7fa684070a00 2023-10-30T13:16:31.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa688b31640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841539a0 0x7fa684143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa688b31640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841539a0 0x7fa684143560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42280/0 (socket says 172.21.15.148:42280) 2023-10-30T13:16:31.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa688b31640 1 -- 172.21.15.148:0/2584360841 learned_addr learned my addr 172.21.15.148:0/2584360841 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:31.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa683fff640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa684069e90 0x7fa6841f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa6837fe640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa684070a00 0x7fa6841456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa688b31640 1 -- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa684070a00 msgr2=0x7fa6841456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa688b31640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa684070a00 0x7fa6841456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa688b31640 1 -- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa684069e90 msgr2=0x7fa6841f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa688b31640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa684069e90 0x7fa6841f2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa688b31640 1 -- 172.21.15.148:0/2584360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa674009420 con 0x7fa6841539a0 2023-10-30T13:16:31.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa688b31640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841539a0 0x7fa684143560 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa67c001570 tx=0x7fa67c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:31.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa680ff9640 1 -- 172.21.15.148:0/2584360841 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c003db0 con 0x7fa6841539a0 2023-10-30T13:16:31.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa680ff9640 1 -- 172.21.15.148:0/2584360841 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67c005070 con 0x7fa6841539a0 2023-10-30T13:16:31.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6841f5890 con 0x7fa6841539a0 2023-10-30T13:16:31.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa680ff9640 1 -- 172.21.15.148:0/2584360841 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c005420 con 0x7fa6841539a0 2023-10-30T13:16:31.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6841f5e50 con 0x7fa6841539a0 2023-10-30T13:16:31.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.597+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6841f2340 con 0x7fa6841539a0 2023-10-30T13:16:31.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.601+0000 7fa680ff9640 1 -- 172.21.15.148:0/2584360841 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa67c005580 con 0x7fa6841539a0 2023-10-30T13:16:31.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.601+0000 7fa680ff9640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6700664f0 0x7fa6700689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:31.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.601+0000 7fa680ff9640 1 -- 172.21.15.148:0/2584360841 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fa67c083760 con 0x7fa6841539a0 2023-10-30T13:16:31.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.601+0000 7fa683fff640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6700664f0 0x7fa6700689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:31.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.601+0000 7fa683fff640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6700664f0 0x7fa6700689b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6740af000 tx=0x7fa6740b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:31.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.605+0000 7fa680ff9640 1 -- 172.21.15.148:0/2584360841 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6841f2340 con 0x7fa6841539a0 2023-10-30T13:16:31.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.785+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6840c64c0 con 0x7fa6841539a0 2023-10-30T13:16:31.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.785+0000 7fa680ff9640 1 -- 172.21.15.148:0/2584360841 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6539 (secure 0 0 0) 0x7fa67c017090 con 0x7fa6841539a0 2023-10-30T13:16:31.789 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:31.789 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:11.413204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:31.789 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 28 2023-10-30T13:16:31.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6700664f0 msgr2=0x7fa6700689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6700664f0 0x7fa6700689b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6740af000 tx=0x7fa6740b6040 comp rx=0 tx=0).stop 2023-10-30T13:16:31.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841539a0 msgr2=0x7fa684143560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:31.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841539a0 0x7fa684143560 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa67c001570 tx=0x7fa67c00cd20 comp rx=0 tx=0).stop 2023-10-30T13:16:31.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 shutdown_connections 2023-10-30T13:16:31.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6700664f0 0x7fa6700689b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6841539a0 0x7fa684143560 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa684070a00 0x7fa6841456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 --2- 172.21.15.148:0/2584360841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa684069e90 0x7fa6841f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:31.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 >> 172.21.15.148:0/2584360841 conn(0x7fa6840c7910 msgr2=0x7fa684151cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:31.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 shutdown_connections 2023-10-30T13:16:31.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:31.789+0000 7fa68a594640 1 -- 172.21.15.148:0/2584360841 wait complete. 2023-10-30T13:16:31.809 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:16:33.810 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:33.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.957+0000 7efd46268640 1 Processor -- start 2023-10-30T13:16:33.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.957+0000 7efd46268640 1 -- start start 2023-10-30T13:16:33.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.957+0000 7efd46268640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd400c5e70 0x7efd40165200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:33.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.957+0000 7efd46268640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd400c74f0 con 0x7efd400b0dc0 2023-10-30T13:16:33.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.957+0000 7efd46268640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd400c77b0 con 0x7efd40153b10 2023-10-30T13:16:33.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.957+0000 7efd46268640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd400c1f20 con 0x7efd400c5e70 2023-10-30T13:16:33.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.957+0000 7efd3f7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efd400b0dc0 0x7efd400c7e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43312/0 (socket says 172.21.15.148:43312) 2023-10-30T13:16:33.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.957+0000 7efd3f7fe640 1 -- 172.21.15.148:0/4113961667 learned_addr learned my addr 172.21.15.148:0/4113961667 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:16:33.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd44805640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd400c5e70 0x7efd40165200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:33.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92164966 0 0) 0x7efd400c74f0 con 0x7efd400b0dc0 2023-10-30T13:16:33.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd28003530 con 0x7efd400b0dc0 2023-10-30T13:16:33.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105253148 0 0) 0x7efd400c77b0 con 0x7efd40153b10 2023-10-30T13:16:33.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd400c74f0 con 0x7efd40153b10 2023-10-30T13:16:33.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289909243 0 0) 0x7efd28003530 con 0x7efd400b0dc0 2023-10-30T13:16:33.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd400c5e70 msgr2=0x7efd40165200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:33.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd400c5e70 0x7efd40165200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:33.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 >> v1:172.21.15.150:6789/0 conn(0x7efd40153b10 legacy=0x7efd4015ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:33.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd40159b80 con 0x7efd400b0dc0 2023-10-30T13:16:33.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7efd34001680 con 0x7efd400b0dc0 2023-10-30T13:16:33.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 >> v1:172.21.15.148:6789/0 conn(0x7efd400b0dc0 legacy=0x7efd400c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:33.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd28003b40 0x7efd28003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:33.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd400c5e70 0x7efd280047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:33.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd28004d10 0x7efd28005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:33.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2800ba40 con 0x7efd28004d10 2023-10-30T13:16:33.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2800bb80 con 0x7efd28003b40 2023-10-30T13:16:33.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2800bcf0 con 0x7efd400c5e70 2023-10-30T13:16:33.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd44805640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd28004d10 0x7efd28005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:33.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3ffff640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd28003b40 0x7efd28003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:33.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3f7fe640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd400c5e70 0x7efd280047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:33.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd44805640 1 -- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd400c5e70 msgr2=0x7efd280047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:33.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd44805640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd400c5e70 0x7efd280047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:33.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd44805640 1 -- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd28003b40 msgr2=0x7efd28003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:33.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd44805640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd28003b40 0x7efd28003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:33.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd44805640 1 -- 172.21.15.148:0/4113961667 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd28003530 con 0x7efd28004d10 2023-10-30T13:16:33.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.961+0000 7efd3f7fe640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd400c5e70 0x7efd280047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:33.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd44805640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd28004d10 0x7efd28005190 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efd38009fc0 tx=0x7efd3801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c84c93a317fe37 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:33.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd380a5c70 con 0x7efd28004d10 2023-10-30T13:16:33.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3800f040 con 0x7efd28004d10 2023-10-30T13:16:33.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd3effd640 1 -- 172.21.15.148:0/4113961667 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd380ad440 con 0x7efd28004d10 2023-10-30T13:16:33.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 -- 172.21.15.148:0/4113961667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd28004d10 msgr2=0x7efd28005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:33.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 --2- 172.21.15.148:0/4113961667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd28004d10 0x7efd28005190 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efd38009fc0 tx=0x7efd3801cc60 comp rx=0 tx=0).stop 2023-10-30T13:16:33.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd44805640 1 -- 172.21.15.148:0/4113961667 reap_dead start 2023-10-30T13:16:33.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 -- 172.21.15.148:0/4113961667 shutdown_connections 2023-10-30T13:16:33.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 -- 172.21.15.148:0/4113961667 >> 172.21.15.148:0/4113961667 conn(0x7efd4006d770 msgr2=0x7efd4006db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:33.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 -- 172.21.15.148:0/4113961667 shutdown_connections 2023-10-30T13:16:33.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 -- 172.21.15.148:0/4113961667 wait complete. 2023-10-30T13:16:33.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 Processor -- start 2023-10-30T13:16:33.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 -- start start 2023-10-30T13:16:33.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd28004d10 0x7efd40155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:33.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.965+0000 7efd46268640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd401560f0 0x7efd401efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:33.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd46268640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd401f00a0 0x7efd40149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:33.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd46268640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4015b850 con 0x7efd401560f0 2023-10-30T13:16:33.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd46268640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4015aa30 con 0x7efd28004d10 2023-10-30T13:16:33.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd46268640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4015b200 con 0x7efd401f00a0 2023-10-30T13:16:33.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd401560f0 0x7efd401efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:33.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd401560f0 0x7efd401efb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38090/0 (socket says 172.21.15.148:38090) 2023-10-30T13:16:33.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3f7fe640 1 -- 172.21.15.148:0/387808013 learned_addr learned my addr 172.21.15.148:0/387808013 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:33.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3ffff640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd28004d10 0x7efd40155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:33.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd44805640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd401f00a0 0x7efd40149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:33.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3f7fe640 1 -- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd401f00a0 msgr2=0x7efd40149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:33.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3f7fe640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd401f00a0 0x7efd40149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:33.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3f7fe640 1 -- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd28004d10 msgr2=0x7efd40155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:33.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3f7fe640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd28004d10 0x7efd40155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:33.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3f7fe640 1 -- 172.21.15.148:0/387808013 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd401ef6d0 con 0x7efd401560f0 2023-10-30T13:16:33.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3f7fe640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd401560f0 0x7efd401efb60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efd34001c30 tx=0x7efd3400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:33.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3cff9640 1 -- 172.21.15.148:0/387808013 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3400ee80 con 0x7efd401560f0 2023-10-30T13:16:33.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3cff9640 1 -- 172.21.15.148:0/387808013 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3400c870 con 0x7efd401560f0 2023-10-30T13:16:33.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3cff9640 1 -- 172.21.15.148:0/387808013 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3400cbe0 con 0x7efd401560f0 2023-10-30T13:16:33.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4015c2d0 con 0x7efd401560f0 2023-10-30T13:16:33.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4015c6d0 con 0x7efd401560f0 2023-10-30T13:16:33.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd40074250 con 0x7efd401560f0 2023-10-30T13:16:33.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.969+0000 7efd3cff9640 1 -- 172.21.15.148:0/387808013 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd3400cd40 con 0x7efd401560f0 2023-10-30T13:16:33.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.973+0000 7efd3cff9640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd2c0664d0 0x7efd2c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:33.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.973+0000 7efd3cff9640 1 -- 172.21.15.148:0/387808013 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7efd34083bf0 con 0x7efd401560f0 2023-10-30T13:16:33.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.973+0000 7efd3ffff640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd2c0664d0 0x7efd2c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:33.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.973+0000 7efd3ffff640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd2c0664d0 0x7efd2c068990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efd30007fd0 tx=0x7efd30009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:33.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:33.973+0000 7efd3cff9640 1 -- 172.21.15.148:0/387808013 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd34050e90 con 0x7efd401560f0 2023-10-30T13:16:34.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.153+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd401ef860 con 0x7efd401560f0 2023-10-30T13:16:34.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.153+0000 7efd3cff9640 1 -- 172.21.15.148:0/387808013 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6539 (secure 0 0 0) 0x7efd340544b0 con 0x7efd401560f0 2023-10-30T13:16:34.157 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:34.157 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:11.413204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:34.157 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 28 2023-10-30T13:16:34.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd2c0664d0 msgr2=0x7efd2c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:34.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd2c0664d0 0x7efd2c068990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efd30007fd0 tx=0x7efd30009040 comp rx=0 tx=0).stop 2023-10-30T13:16:34.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd401560f0 msgr2=0x7efd401efb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:34.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd401560f0 0x7efd401efb60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efd34001c30 tx=0x7efd3400ec90 comp rx=0 tx=0).stop 2023-10-30T13:16:34.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 shutdown_connections 2023-10-30T13:16:34.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd2c0664d0 0x7efd2c068990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:34.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd401560f0 0x7efd401efb60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:34.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd401f00a0 0x7efd40149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:34.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 --2- 172.21.15.148:0/387808013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd28004d10 0x7efd40155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:34.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 >> 172.21.15.148:0/387808013 conn(0x7efd4006d770 msgr2=0x7efd400c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:34.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 shutdown_connections 2023-10-30T13:16:34.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:34.157+0000 7efd46268640 1 -- 172.21.15.148:0/387808013 wait complete. 2023-10-30T13:16:34.177 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:16:35.741 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/30 file after the snapshot 2023-10-30T13:16:35.743 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/31 file after the snapshot 2023-10-30T13:16:35.743 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/32 file after the snapshot 2023-10-30T13:16:35.743 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/33 file after the snapshot 2023-10-30T13:16:35.743 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/34 file after the snapshot 2023-10-30T13:16:35.744 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/35 file after the snapshot 2023-10-30T13:16:35.744 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/36 file after the snapshot 2023-10-30T13:16:35.744 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/37 file after the snapshot 2023-10-30T13:16:35.744 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/38 file after the snapshot 2023-10-30T13:16:35.744 INFO:tasks.workunit.client.0.smithi148.stdout: create 278/39 file after the snapshot 2023-10-30T13:16:35.745 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/.snap/snap-subdir-test 2023-10-30T13:16:36.178 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:36.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.325+0000 7f06c2aea640 1 Processor -- start 2023-10-30T13:16:36.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.325+0000 7f06c2aea640 1 -- start start 2023-10-30T13:16:36.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06c2aea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f06bc070650 0x7f06bc071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:36.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06c2aea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070c90 0x7f06bc0720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:36.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06c2aea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06bc0c2f20 con 0x7f06bc156f80 2023-10-30T13:16:36.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06c2aea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06bc157360 con 0x7f06bc070650 2023-10-30T13:16:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06c2aea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc0c85c0 con 0x7f06bc070c90 2023-10-30T13:16:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06bbfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f06bc070650 0x7f06bc071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06c0886640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f06bc156f80 0x7f06bc159880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43322/0 (socket says 172.21.15.148:43322) 2023-10-30T13:16:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06bbfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f06bc070650 0x7f06bc071bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38100/0 (socket says 172.21.15.148:38100) 2023-10-30T13:16:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06c0886640 1 -- 172.21.15.148:0/3013649473 learned_addr learned my addr 172.21.15.148:0/3013649473 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:16:36.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06c1087640 1 --2- 172.21.15.148:0/3013649473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070c90 0x7f06bc0720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06bbfff640 1 -- 172.21.15.148:0/3013649473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070c90 msgr2=0x7f06bc0720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06bbfff640 1 --2- 172.21.15.148:0/3013649473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070c90 0x7f06bc0720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06bbfff640 1 -- 172.21.15.148:0/3013649473 >> v1:172.21.15.148:6789/0 conn(0x7f06bc156f80 legacy=0x7f06bc159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.329+0000 7f06bbfff640 1 -- 172.21.15.148:0/3013649473 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc071230 con 0x7f06bc070650 2023-10-30T13:16:36.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06bbfff640 1 --2- 172.21.15.148:0/3013649473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc070650 0x7f06bc071bb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f06a80097c0 tx=0x7f06a801c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7374f23437ed3e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:36.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06bb7fe640 1 -- 172.21.15.148:0/3013649473 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a80ab600 con 0x7f06bc070650 2023-10-30T13:16:36.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06bb7fe640 1 -- 172.21.15.148:0/3013649473 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a80a7050 con 0x7f06bc070650 2023-10-30T13:16:36.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06bb7fe640 1 -- 172.21.15.148:0/3013649473 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a80ab980 con 0x7f06bc070650 2023-10-30T13:16:36.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 -- 172.21.15.148:0/3013649473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc070650 msgr2=0x7f06bc071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 --2- 172.21.15.148:0/3013649473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc070650 0x7f06bc071bb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f06a80097c0 tx=0x7f06a801c310 comp rx=0 tx=0).stop 2023-10-30T13:16:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 -- 172.21.15.148:0/3013649473 shutdown_connections 2023-10-30T13:16:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 --2- 172.21.15.148:0/3013649473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc070650 0x7f06bc071bb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 --2- 172.21.15.148:0/3013649473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070c90 0x7f06bc0720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 -- 172.21.15.148:0/3013649473 >> 172.21.15.148:0/3013649473 conn(0x7f06bc0cbeb0 msgr2=0x7f06bc0cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:36.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 -- 172.21.15.148:0/3013649473 shutdown_connections 2023-10-30T13:16:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 -- 172.21.15.148:0/3013649473 wait complete. 2023-10-30T13:16:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 Processor -- start 2023-10-30T13:16:36.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.333+0000 7f06c2aea640 1 -- start start 2023-10-30T13:16:36.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c2aea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070650 0x7f06bc1f2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:36.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c2aea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06bc070c90 0x7f06bc152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c2aea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc156f80 0x7f06bc14ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c2aea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc14df70 con 0x7f06bc156f80 2023-10-30T13:16:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c2aea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc149670 con 0x7f06bc070c90 2023-10-30T13:16:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c2aea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc1f33b0 con 0x7f06bc070650 2023-10-30T13:16:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c1087640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc156f80 0x7f06bc14ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:36.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c1087640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc156f80 0x7f06bc14ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38110/0 (socket says 172.21.15.148:38110) 2023-10-30T13:16:36.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c1087640 1 -- 172.21.15.148:0/3784384003 learned_addr learned my addr 172.21.15.148:0/3784384003 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:36.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c0886640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070650 0x7f06bc1f2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:36.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06bbfff640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06bc070c90 0x7f06bc152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:36.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c1087640 1 -- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070650 msgr2=0x7f06bc1f2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:36.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c1087640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070650 0x7f06bc1f2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c1087640 1 -- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06bc070c90 msgr2=0x7f06bc152e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c1087640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06bc070c90 0x7f06bc152e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c1087640 1 -- 172.21.15.148:0/3784384003 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a8009420 con 0x7f06bc156f80 2023-10-30T13:16:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06bbfff640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06bc070c90 0x7f06bc152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c1087640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc156f80 0x7f06bc14ab80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f06b400efc0 tx=0x7f06b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:36.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06b97fa640 1 -- 172.21.15.148:0/3784384003 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4009ba0 con 0x7f06bc156f80 2023-10-30T13:16:36.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06b97fa640 1 -- 172.21.15.148:0/3784384003 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b400f040 con 0x7f06bc156f80 2023-10-30T13:16:36.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06bc153360 con 0x7f06bc156f80 2023-10-30T13:16:36.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06b97fa640 1 -- 172.21.15.148:0/3784384003 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b40146c0 con 0x7f06bc156f80 2023-10-30T13:16:36.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06bc14d2f0 con 0x7f06bc156f80 2023-10-30T13:16:36.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06bc04e170 con 0x7f06bc156f80 2023-10-30T13:16:36.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.337+0000 7f06b97fa640 1 -- 172.21.15.148:0/3784384003 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06b4014820 con 0x7f06bc156f80 2023-10-30T13:16:36.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.341+0000 7f06b97fa640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06a00664a0 0x7f06a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:36.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.341+0000 7f06b97fa640 1 -- 172.21.15.148:0/3784384003 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f06b4084bc0 con 0x7f06bc156f80 2023-10-30T13:16:36.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.341+0000 7f06c0886640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06a00664a0 0x7f06a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:36.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.341+0000 7f06c0886640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06a00664a0 0x7f06a0068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f06b0000f70 tx=0x7f06b0007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:36.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.345+0000 7f06b97fa640 1 -- 172.21.15.148:0/3784384003 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06b4051dd0 con 0x7f06bc156f80 2023-10-30T13:16:36.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.529+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06bc1567b0 con 0x7f06bc156f80 2023-10-30T13:16:36.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.529+0000 7f06b97fa640 1 -- 172.21.15.148:0/3784384003 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6539 (secure 0 0 0) 0x7f06b4018020 con 0x7f06bc156f80 2023-10-30T13:16:36.531 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:36.531 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:11.413204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:36.532 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 28 2023-10-30T13:16:36.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.529+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06a00664a0 msgr2=0x7f06a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:36.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.529+0000 7f06c2aea640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06a00664a0 0x7f06a0068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f06b0000f70 tx=0x7f06b0007e10 comp rx=0 tx=0).stop 2023-10-30T13:16:36.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.529+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc156f80 msgr2=0x7f06bc14ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:36.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.529+0000 7f06c2aea640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc156f80 0x7f06bc14ab80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f06b400efc0 tx=0x7f06b400c490 comp rx=0 tx=0).stop 2023-10-30T13:16:36.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.533+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 shutdown_connections 2023-10-30T13:16:36.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.533+0000 7f06c2aea640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06a00664a0 0x7f06a0068960 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:36.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.533+0000 7f06c2aea640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06bc156f80 0x7f06bc14ab80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:36.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.533+0000 7f06c2aea640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06bc070c90 0x7f06bc152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:36.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.533+0000 7f06c2aea640 1 --2- 172.21.15.148:0/3784384003 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06bc070650 0x7f06bc1f2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:36.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.533+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 >> 172.21.15.148:0/3784384003 conn(0x7f06bc0cbeb0 msgr2=0x7f06bc14c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:36.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.533+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 shutdown_connections 2023-10-30T13:16:36.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:36.533+0000 7f06c2aea640 1 -- 172.21.15.148:0/3784384003 wait complete. 2023-10-30T13:16:36.551 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:16:38.552 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:38.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeac73e640 1 Processor -- start 2023-10-30T13:16:38.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeac73e640 1 -- start start 2023-10-30T13:16:38.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeac73e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7feea4153d70 0x7feea406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:38.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeac73e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feea406e470 0x7feea40cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:38.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeac73e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea40c8490 0x7feea40cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:38.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeac73e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea406d780 con 0x7feea4153d70 2023-10-30T13:16:38.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeac73e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea406afa0 con 0x7feea406e470 2023-10-30T13:16:38.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeac73e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea4154150 con 0x7feea40c8490 2023-10-30T13:16:38.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaa4da640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7feea4153d70 0x7feea406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:38.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaa4da640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7feea4153d70 0x7feea406ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38122/0 (socket says 172.21.15.148:38122) 2023-10-30T13:16:38.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaa4da640 1 -- 172.21.15.148:0/642231781 learned_addr learned my addr 172.21.15.148:0/642231781 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:38.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feea9cd9640 1 --2- 172.21.15.148:0/642231781 >> v2:172.21.15.150:3300/0 conn(0x7feea406e470 0x7feea40cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:38.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaacdb640 1 --2- 172.21.15.148:0/642231781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea40c8490 0x7feea40cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:38.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaa4da640 1 -- 172.21.15.148:0/642231781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea40c8490 msgr2=0x7feea40cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:38.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaa4da640 1 --2- 172.21.15.148:0/642231781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea40c8490 0x7feea40cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaa4da640 1 -- 172.21.15.148:0/642231781 >> v2:172.21.15.150:3300/0 conn(0x7feea406e470 msgr2=0x7feea40cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:38.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaa4da640 1 --2- 172.21.15.148:0/642231781 >> v2:172.21.15.150:3300/0 conn(0x7feea406e470 0x7feea40cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaa4da640 1 -- 172.21.15.148:0/642231781 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea406e030 con 0x7feea4153d70 2023-10-30T13:16:38.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.701+0000 7feeaacdb640 1 --2- 172.21.15.148:0/642231781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea40c8490 0x7feea40cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:16:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeaa4da640 1 --2- 172.21.15.148:0/642231781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea4153d70 0x7feea406ca80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7feea4070230 tx=0x7feea001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55357993fead3ec8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feea94d8640 1 -- 172.21.15.148:0/642231781 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea00aa5f0 con 0x7feea4153d70 2023-10-30T13:16:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feea94d8640 1 -- 172.21.15.148:0/642231781 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea00aa750 con 0x7feea4153d70 2023-10-30T13:16:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feea94d8640 1 -- 172.21.15.148:0/642231781 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea00aaae0 con 0x7feea4153d70 2023-10-30T13:16:38.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeac73e640 1 -- 172.21.15.148:0/642231781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea4153d70 msgr2=0x7feea406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeac73e640 1 --2- 172.21.15.148:0/642231781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea4153d70 0x7feea406ca80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7feea4070230 tx=0x7feea001c4d0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeac73e640 1 -- 172.21.15.148:0/642231781 shutdown_connections 2023-10-30T13:16:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeac73e640 1 --2- 172.21.15.148:0/642231781 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea40c8490 0x7feea40cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeac73e640 1 --2- 172.21.15.148:0/642231781 >> v2:172.21.15.150:3300/0 conn(0x7feea406e470 0x7feea40cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeac73e640 1 --2- 172.21.15.148:0/642231781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea4153d70 0x7feea406ca80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeac73e640 1 -- 172.21.15.148:0/642231781 >> 172.21.15.148:0/642231781 conn(0x7feea4071ea0 msgr2=0x7feea4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:38.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeac73e640 1 -- 172.21.15.148:0/642231781 shutdown_connections 2023-10-30T13:16:38.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.705+0000 7feeac73e640 1 -- 172.21.15.148:0/642231781 wait complete. 2023-10-30T13:16:38.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 Processor -- start 2023-10-30T13:16:38.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 -- start start 2023-10-30T13:16:38.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea406e470 0x7feea414d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:38.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea40c8490 0x7feea414f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:38.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feea4153d70 0x7feea41f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:38.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea414d5b0 con 0x7feea40c8490 2023-10-30T13:16:38.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea41f5c40 con 0x7feea4153d70 2023-10-30T13:16:38.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea40c7e20 con 0x7feea406e470 2023-10-30T13:16:38.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feea9cd9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea40c8490 0x7feea414f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:38.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feea9cd9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea40c8490 0x7feea414f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38138/0 (socket says 172.21.15.148:38138) 2023-10-30T13:16:38.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feea9cd9640 1 -- 172.21.15.148:0/167140979 learned_addr learned my addr 172.21.15.148:0/167140979 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:38.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeaa4da640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea406e470 0x7feea414d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:38.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeaacdb640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feea4153d70 0x7feea41f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:38.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feea9cd9640 1 -- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea406e470 msgr2=0x7feea414d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:38.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feea9cd9640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea406e470 0x7feea414d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feea9cd9640 1 -- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feea4153d70 msgr2=0x7feea41f26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:38.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feea9cd9640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feea4153d70 0x7feea41f26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feea9cd9640 1 -- 172.21.15.148:0/167140979 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea00094d0 con 0x7feea40c8490 2023-10-30T13:16:38.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeaacdb640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feea4153d70 0x7feea41f26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:38.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feea9cd9640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea40c8490 0x7feea414f670 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fee9400d800 tx=0x7fee9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:38.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7fee9affd640 1 -- 172.21.15.148:0/167140979 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9400b6c0 con 0x7feea40c8490 2023-10-30T13:16:38.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7fee9affd640 1 -- 172.21.15.148:0/167140979 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9400b820 con 0x7feea40c8490 2023-10-30T13:16:38.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea414d7b0 con 0x7feea40c8490 2023-10-30T13:16:38.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7fee9affd640 1 -- 172.21.15.148:0/167140979 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9400bb90 con 0x7feea40c8490 2023-10-30T13:16:38.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.709+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea41f5e90 con 0x7feea40c8490 2023-10-30T13:16:38.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.713+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea404e170 con 0x7feea40c8490 2023-10-30T13:16:38.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.713+0000 7fee9affd640 1 -- 172.21.15.148:0/167140979 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee9400bcf0 con 0x7feea40c8490 2023-10-30T13:16:38.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.713+0000 7fee9affd640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee880664a0 0x7fee88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:38.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.713+0000 7fee9affd640 1 -- 172.21.15.148:0/167140979 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fee940844e0 con 0x7feea40c8490 2023-10-30T13:16:38.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.713+0000 7feeaa4da640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee880664a0 0x7fee88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:38.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.713+0000 7feeaa4da640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee880664a0 0x7fee88068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7feea00ae000 tx=0x7feea00b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:38.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.717+0000 7fee9affd640 1 -- 172.21.15.148:0/167140979 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee94051800 con 0x7feea40c8490 2023-10-30T13:16:38.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.897+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feea41561b0 con 0x7feea40c8490 2023-10-30T13:16:38.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.897+0000 7fee9affd640 1 -- 172.21.15.148:0/167140979 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6539 (secure 0 0 0) 0x7fee94054e20 con 0x7feea40c8490 2023-10-30T13:16:38.900 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:38.900 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:11.413204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:38.901 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 28 2023-10-30T13:16:38.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee880664a0 msgr2=0x7fee88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:38.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee880664a0 0x7fee88068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7feea00ae000 tx=0x7feea00b5040 comp rx=0 tx=0).stop 2023-10-30T13:16:38.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea40c8490 msgr2=0x7feea414f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:38.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea40c8490 0x7feea414f670 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fee9400d800 tx=0x7fee9400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 shutdown_connections 2023-10-30T13:16:38.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee880664a0 0x7fee88068960 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feea40c8490 0x7feea414f670 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feea4153d70 0x7feea41f26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 --2- 172.21.15.148:0/167140979 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feea406e470 0x7feea414d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:38.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 >> 172.21.15.148:0/167140979 conn(0x7feea4071ea0 msgr2=0x7feea41509e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:38.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 shutdown_connections 2023-10-30T13:16:38.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:38.901+0000 7feeac73e640 1 -- 172.21.15.148:0/167140979 wait complete. 2023-10-30T13:16:38.920 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:16:40.921 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:41.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.069+0000 7fbc3d66b640 1 Processor -- start 2023-10-30T13:16:41.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.069+0000 7fbc3d66b640 1 -- start start 2023-10-30T13:16:41.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.069+0000 7fbc3d66b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc381538a0 0x7fbc3815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:41.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.069+0000 7fbc3d66b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbc380ca0e0 0x7fbc3815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:41.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.069+0000 7fbc3d66b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc380cf160 0x7fbc380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:41.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.069+0000 7fbc3d66b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380cb800 con 0x7fbc380ca0e0 2023-10-30T13:16:41.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.069+0000 7fbc3d66b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3815dc30 con 0x7fbc380cf160 2023-10-30T13:16:41.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.069+0000 7fbc3d66b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380cb330 con 0x7fbc381538a0 2023-10-30T13:16:41.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.069+0000 7fbc2e7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbc380ca0e0 0x7fbc3815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:41.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc2e7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbc380ca0e0 0x7fbc3815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38152/0 (socket says 172.21.15.148:38152) 2023-10-30T13:16:41.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc2e7fc640 1 -- 172.21.15.148:0/2766284843 learned_addr learned my addr 172.21.15.148:0/2766284843 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:41.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc36ffd640 1 --2- 172.21.15.148:0/2766284843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc381538a0 0x7fbc3815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:41.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc377fe640 1 --2- 172.21.15.148:0/2766284843 >> v2:172.21.15.150:3300/0 conn(0x7fbc380cf160 0x7fbc380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:41.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc2e7fc640 1 -- 172.21.15.148:0/2766284843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc381538a0 msgr2=0x7fbc3815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:41.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc2e7fc640 1 --2- 172.21.15.148:0/2766284843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc381538a0 0x7fbc3815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc2e7fc640 1 -- 172.21.15.148:0/2766284843 >> v2:172.21.15.150:3300/0 conn(0x7fbc380cf160 msgr2=0x7fbc380cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:41.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc2e7fc640 1 --2- 172.21.15.148:0/2766284843 >> v2:172.21.15.150:3300/0 conn(0x7fbc380cf160 0x7fbc380cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc2e7fc640 1 -- 172.21.15.148:0/2766284843 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc380cad70 con 0x7fbc380ca0e0 2023-10-30T13:16:41.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc377fe640 1 --2- 172.21.15.148:0/2766284843 >> v2:172.21.15.150:3300/0 conn(0x7fbc380cf160 0x7fbc380cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:16:41.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc2e7fc640 1 --2- 172.21.15.148:0/2766284843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380ca0e0 0x7fbc3815a090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806cb00 tx=0x7fbc2001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4ee3e6c1709ab5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:41.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc367fc640 1 -- 172.21.15.148:0/2766284843 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc200a89b0 con 0x7fbc380ca0e0 2023-10-30T13:16:41.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc367fc640 1 -- 172.21.15.148:0/2766284843 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc200a8b10 con 0x7fbc380ca0e0 2023-10-30T13:16:41.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.073+0000 7fbc367fc640 1 -- 172.21.15.148:0/2766284843 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc200a8ea0 con 0x7fbc380ca0e0 2023-10-30T13:16:41.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 -- 172.21.15.148:0/2766284843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380ca0e0 msgr2=0x7fbc3815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:41.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/2766284843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380ca0e0 0x7fbc3815a090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbc3806cb00 tx=0x7fbc2001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 -- 172.21.15.148:0/2766284843 shutdown_connections 2023-10-30T13:16:41.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/2766284843 >> v2:172.21.15.150:3300/0 conn(0x7fbc380cf160 0x7fbc380cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/2766284843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380ca0e0 0x7fbc3815a090 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/2766284843 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc381538a0 0x7fbc3815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 -- 172.21.15.148:0/2766284843 >> 172.21.15.148:0/2766284843 conn(0x7fbc3806bfe0 msgr2=0x7fbc3806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:41.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 -- 172.21.15.148:0/2766284843 shutdown_connections 2023-10-30T13:16:41.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 -- 172.21.15.148:0/2766284843 wait complete. 2023-10-30T13:16:41.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 Processor -- start 2023-10-30T13:16:41.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 -- start start 2023-10-30T13:16:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc380ca0e0 0x7fbc381e5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380cf160 0x7fbc381e3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc381538a0 0x7fbc381ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380ccfb0 con 0x7fbc380cf160 2023-10-30T13:16:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380cd1c0 con 0x7fbc381538a0 2023-10-30T13:16:41.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc3d66b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380cd4a0 con 0x7fbc380ca0e0 2023-10-30T13:16:41.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc2e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380cf160 0x7fbc381e3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:41.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc2e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380cf160 0x7fbc381e3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38154/0 (socket says 172.21.15.148:38154) 2023-10-30T13:16:41.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc2e7fc640 1 -- 172.21.15.148:0/1742545418 learned_addr learned my addr 172.21.15.148:0/1742545418 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:41.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc36ffd640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc380ca0e0 0x7fbc381e5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:41.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc377fe640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc381538a0 0x7fbc381ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:41.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc2e7fc640 1 -- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc380ca0e0 msgr2=0x7fbc381e5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:41.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.077+0000 7fbc2e7fc640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc380ca0e0 0x7fbc381e5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2e7fc640 1 -- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc381538a0 msgr2=0x7fbc381ef8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:41.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2e7fc640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc381538a0 0x7fbc381ef8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2e7fc640 1 -- 172.21.15.148:0/1742545418 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc200094d0 con 0x7fbc380cf160 2023-10-30T13:16:41.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2e7fc640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380cf160 0x7fbc381e3250 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbc2001d3a0 tx=0x7fbc2001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:41.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2ffff640 1 -- 172.21.15.148:0/1742545418 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc200ba600 con 0x7fbc380cf160 2023-10-30T13:16:41.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2ffff640 1 -- 172.21.15.148:0/1742545418 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc200ba760 con 0x7fbc380cf160 2023-10-30T13:16:41.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2ffff640 1 -- 172.21.15.148:0/1742545418 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc200bab10 con 0x7fbc380cf160 2023-10-30T13:16:41.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc380cd780 con 0x7fbc380cf160 2023-10-30T13:16:41.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc381ef360 con 0x7fbc380cf160 2023-10-30T13:16:41.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc380b2a50 con 0x7fbc380cf160 2023-10-30T13:16:41.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2ffff640 1 -- 172.21.15.148:0/1742545418 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbc200bac70 con 0x7fbc380cf160 2023-10-30T13:16:41.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2ffff640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc180664d0 0x7fbc18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:41.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc2ffff640 1 -- 172.21.15.148:0/1742545418 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fbc20122730 con 0x7fbc380cf160 2023-10-30T13:16:41.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.081+0000 7fbc36ffd640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc180664d0 0x7fbc18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:41.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.085+0000 7fbc36ffd640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc180664d0 0x7fbc18068990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800b930 tx=0x7fbc28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:41.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.085+0000 7fbc2ffff640 1 -- 172.21.15.148:0/1742545418 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbc200ef940 con 0x7fbc380cf160 2023-10-30T13:16:41.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.265+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc3804e170 con 0x7fbc380cf160 2023-10-30T13:16:41.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc2ffff640 1 -- 172.21.15.148:0/1742545418 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6539 (secure 0 0 0) 0x7fbc200f2f60 con 0x7fbc380cf160 2023-10-30T13:16:41.269 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:41.270 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:11.413204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:41.270 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 28 2023-10-30T13:16:41.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc180664d0 msgr2=0x7fbc18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:41.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc180664d0 0x7fbc18068990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800b930 tx=0x7fbc28008040 comp rx=0 tx=0).stop 2023-10-30T13:16:41.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380cf160 msgr2=0x7fbc381e3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:41.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380cf160 0x7fbc381e3250 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbc2001d3a0 tx=0x7fbc2001ba90 comp rx=0 tx=0).stop 2023-10-30T13:16:41.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 shutdown_connections 2023-10-30T13:16:41.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc180664d0 0x7fbc18068990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc380cf160 0x7fbc381e3250 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc381538a0 0x7fbc381ef8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 --2- 172.21.15.148:0/1742545418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc380ca0e0 0x7fbc381e5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:41.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 >> 172.21.15.148:0/1742545418 conn(0x7fbc3806bfe0 msgr2=0x7fbc38151770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:41.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 shutdown_connections 2023-10-30T13:16:41.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:41.269+0000 7fbc3d66b640 1 -- 172.21.15.148:0/1742545418 wait complete. 2023-10-30T13:16:41.290 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:16:43.291 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:43.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.437+0000 7f22a7646640 1 Processor -- start 2023-10-30T13:16:43.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.437+0000 7f22a7646640 1 -- start start 2023-10-30T13:16:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a7646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 0x7f22a00cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a7646640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f22a0071250 0x7f22a00716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a7646640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a0071be0 con 0x7f22a014fd50 2023-10-30T13:16:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a7646640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a01560a0 con 0x7f22a0071250 2023-10-30T13:16:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a7646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0156380 con 0x7f22a00cb550 2023-10-30T13:16:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a5be3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f22a0071250 0x7f22a00716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a5be3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f22a0071250 0x7f22a00716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50376/0 (socket says 172.21.15.148:50376) 2023-10-30T13:16:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a5be3640 1 -- 172.21.15.148:0/606473811 learned_addr learned my addr 172.21.15.148:0/606473811 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a4be1640 1 --2- 172.21.15.148:0/606473811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 0x7f22a00cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a5be3640 1 -- 172.21.15.148:0/606473811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 msgr2=0x7f22a00cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a5be3640 1 --2- 172.21.15.148:0/606473811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 0x7f22a00cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a5be3640 1 -- 172.21.15.148:0/606473811 >> v1:172.21.15.150:6789/0 conn(0x7f22a014fd50 legacy=0x7f22a00c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a5be3640 1 -- 172.21.15.148:0/606473811 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a0156660 con 0x7f22a0071250 2023-10-30T13:16:43.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.441+0000 7f22a5be3640 1 --2- 172.21.15.148:0/606473811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 0x7f22a00716a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2298009e50 tx=0x7f229801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b3c8b9dbb145c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:43.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f2297fff640 1 -- 172.21.15.148:0/606473811 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22980a57f0 con 0x7f22a0071250 2023-10-30T13:16:43.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f2297fff640 1 -- 172.21.15.148:0/606473811 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22980a5950 con 0x7f22a0071250 2023-10-30T13:16:43.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f2297fff640 1 -- 172.21.15.148:0/606473811 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22980a5ce0 con 0x7f22a0071250 2023-10-30T13:16:43.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 -- 172.21.15.148:0/606473811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 msgr2=0x7f22a00716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:43.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 --2- 172.21.15.148:0/606473811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 0x7f22a00716a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2298009e50 tx=0x7f229801c9e0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 -- 172.21.15.148:0/606473811 shutdown_connections 2023-10-30T13:16:43.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 --2- 172.21.15.148:0/606473811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 0x7f22a00716a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 --2- 172.21.15.148:0/606473811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 0x7f22a00cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 -- 172.21.15.148:0/606473811 >> 172.21.15.148:0/606473811 conn(0x7f22a006c6e0 msgr2=0x7f22a0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:43.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 -- 172.21.15.148:0/606473811 shutdown_connections 2023-10-30T13:16:43.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 -- 172.21.15.148:0/606473811 wait complete. 2023-10-30T13:16:43.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 Processor -- start 2023-10-30T13:16:43.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 -- start start 2023-10-30T13:16:43.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 0x7f22a00c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:43.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 0x7f22a013a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:43.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22a014fd50 0x7f22a014e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:43.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a013f160 con 0x7f22a0071250 2023-10-30T13:16:43.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a013f440 con 0x7f22a014fd50 2023-10-30T13:16:43.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a7646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a013a8c0 con 0x7f22a00cb550 2023-10-30T13:16:43.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.445+0000 7f22a53e2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 0x7f22a00c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:43.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a53e2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 0x7f22a00c45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50384/0 (socket says 172.21.15.148:50384) 2023-10-30T13:16:43.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a53e2640 1 -- 172.21.15.148:0/3929621822 learned_addr learned my addr 172.21.15.148:0/3929621822 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:43.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a4be1640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 0x7f22a013a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:43.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a5be3640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22a014fd50 0x7f22a014e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:43.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a53e2640 1 -- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 msgr2=0x7f22a013a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:43.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a53e2640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 0x7f22a013a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a53e2640 1 -- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22a014fd50 msgr2=0x7f22a014e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:43.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a53e2640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22a014fd50 0x7f22a014e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a53e2640 1 -- 172.21.15.148:0/3929621822 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2298009b30 con 0x7f22a0071250 2023-10-30T13:16:43.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a53e2640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 0x7f22a00c45c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f229c00c3c0 tx=0x7f229c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:43.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f2295ffb640 1 -- 172.21.15.148:0/3929621822 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c0052d0 con 0x7f22a0071250 2023-10-30T13:16:43.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f2295ffb640 1 -- 172.21.15.148:0/3929621822 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229c005430 con 0x7f22a0071250 2023-10-30T13:16:43.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f2295ffb640 1 -- 172.21.15.148:0/3929621822 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c003840 con 0x7f22a0071250 2023-10-30T13:16:43.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a00c4c50 con 0x7f22a0071250 2023-10-30T13:16:43.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a00c5210 con 0x7f22a0071250 2023-10-30T13:16:43.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a004e170 con 0x7f22a0071250 2023-10-30T13:16:43.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f2295ffb640 1 -- 172.21.15.148:0/3929621822 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f229c0039a0 con 0x7f22a0071250 2023-10-30T13:16:43.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f2295ffb640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22840664a0 0x7f2284068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:43.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.449+0000 7f2295ffb640 1 -- 172.21.15.148:0/3929621822 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f229c0834f0 con 0x7f22a0071250 2023-10-30T13:16:43.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.453+0000 7f22a4be1640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22840664a0 0x7f2284068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:43.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.453+0000 7f22a4be1640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22840664a0 0x7f2284068960 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f22a01eea20 tx=0x7f229000b580 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:43.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.457+0000 7f2295ffb640 1 -- 172.21.15.148:0/3929621822 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f229c050840 con 0x7f22a0071250 2023-10-30T13:16:43.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.637+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22a01604f0 con 0x7f22a0071250 2023-10-30T13:16:43.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f2295ffb640 1 -- 172.21.15.148:0/3929621822 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6539 (secure 0 0 0) 0x7f229c053e60 con 0x7f22a0071250 2023-10-30T13:16:43.641 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:43.641 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:11.413204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:43.642 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 28 2023-10-30T13:16:43.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22840664a0 msgr2=0x7f2284068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:43.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22840664a0 0x7f2284068960 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f22a01eea20 tx=0x7f229000b580 comp rx=0 tx=0).stop 2023-10-30T13:16:43.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 msgr2=0x7f22a00c45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:43.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 0x7f22a00c45c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f229c00c3c0 tx=0x7f229c00c880 comp rx=0 tx=0).stop 2023-10-30T13:16:43.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 shutdown_connections 2023-10-30T13:16:43.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22840664a0 0x7f2284068960 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22a014fd50 0x7f22a014e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22a00cb550 0x7f22a013a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 --2- 172.21.15.148:0/3929621822 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22a0071250 0x7f22a00c45c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:43.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 >> 172.21.15.148:0/3929621822 conn(0x7f22a006c6e0 msgr2=0x7f22a00cb930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:43.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 shutdown_connections 2023-10-30T13:16:43.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:43.641+0000 7f22a7646640 1 -- 172.21.15.148:0/3929621822 wait complete. 2023-10-30T13:16:43.662 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:16:45.663 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:45.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf9668640 1 Processor -- start 2023-10-30T13:16:45.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf9668640 1 -- start start 2023-10-30T13:16:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf9668640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf415d6f0 0x7fbdf415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf9668640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40cb800 con 0x7fbdf41538a0 2023-10-30T13:16:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf9668640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40cb330 con 0x7fbdf40cf160 2023-10-30T13:16:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf9668640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40cad70 con 0x7fbdf415d6f0 2023-10-30T13:16:45.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf2ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbdf41538a0 0x7fbdf40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39718/0 (socket says 172.21.15.148:39718) 2023-10-30T13:16:45.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf2ffd640 1 -- 172.21.15.148:0/2029441051 learned_addr learned my addr 172.21.15.148:0/2029441051 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:16:45.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf27fc640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf415d6f0 0x7fbdf415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:45.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537272937 0 0) 0x7fbdf40cb800 con 0x7fbdf41538a0 2023-10-30T13:16:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbddc003530 con 0x7fbdf41538a0 2023-10-30T13:16:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1117825924 0 0) 0x7fbddc003530 con 0x7fbdf41538a0 2023-10-30T13:16:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf415d6f0 msgr2=0x7fbdf415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:45.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf415d6f0 0x7fbdf415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 >> v1:172.21.15.150:6789/0 conn(0x7fbdf40cf160 legacy=0x7fbdf4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40cb050 con 0x7fbdf41538a0 2023-10-30T13:16:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fbde4001680 con 0x7fbdf41538a0 2023-10-30T13:16:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 >> v1:172.21.15.148:6789/0 conn(0x7fbdf41538a0 legacy=0x7fbdf40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf415d6f0 0x7fbddc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:45.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbddc003f30 0x7fbddc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc004d10 0x7fbddc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40cad70 con 0x7fbddc003f30 2023-10-30T13:16:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc00bb00 con 0x7fbddc004d10 2023-10-30T13:16:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc00bcc0 con 0x7fbdf415d6f0 2023-10-30T13:16:45.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf27fc640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbddc003f30 0x7fbddc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf2ffd640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf415d6f0 0x7fbddc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf37fe640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc004d10 0x7fbddc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf27fc640 1 -- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf415d6f0 msgr2=0x7fbddc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf27fc640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf415d6f0 0x7fbddc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:45.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf27fc640 1 -- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc004d10 msgr2=0x7fbddc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:45.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf27fc640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc004d10 0x7fbddc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:45.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf27fc640 1 -- 172.21.15.148:0/2029441051 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc003530 con 0x7fbddc003f30 2023-10-30T13:16:45.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.809+0000 7fbdf37fe640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc004d10 0x7fbddc005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:45.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf27fc640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbddc003f30 0x7fbddc0047d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009ae0 tx=0x7fbde801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93f7aefe845142c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:45.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80a5640 con 0x7fbddc003f30 2023-10-30T13:16:45.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde80a57a0 con 0x7fbddc003f30 2023-10-30T13:16:45.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf1ffb640 1 -- 172.21.15.148:0/2029441051 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80a5b50 con 0x7fbddc003f30 2023-10-30T13:16:45.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf9668640 1 -- 172.21.15.148:0/2029441051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbddc003f30 msgr2=0x7fbddc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:45.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf9668640 1 --2- 172.21.15.148:0/2029441051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbddc003f30 0x7fbddc0047d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009ae0 tx=0x7fbde801ca50 comp rx=0 tx=0).stop 2023-10-30T13:16:45.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf37fe640 1 -- 172.21.15.148:0/2029441051 reap_dead start 2023-10-30T13:16:45.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf9668640 1 -- 172.21.15.148:0/2029441051 shutdown_connections 2023-10-30T13:16:45.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf9668640 1 -- 172.21.15.148:0/2029441051 >> 172.21.15.148:0/2029441051 conn(0x7fbdf406bfe0 msgr2=0x7fbdf406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf9668640 1 -- 172.21.15.148:0/2029441051 shutdown_connections 2023-10-30T13:16:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.813+0000 7fbdf9668640 1 -- 172.21.15.148:0/2029441051 wait complete. 2023-10-30T13:16:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 Processor -- start 2023-10-30T13:16:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 -- start start 2023-10-30T13:16:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc003f30 0x7fbdf41f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf414ee00 0x7fbdf40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf40ced40 0x7fbdf414c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf414e9e0 con 0x7fbdf414ee00 2023-10-30T13:16:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf41e56b0 con 0x7fbddc003f30 2023-10-30T13:16:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf4150bd0 con 0x7fbdf40ced40 2023-10-30T13:16:45.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf414ee00 0x7fbdf40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf414ee00 0x7fbdf40ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50408/0 (socket says 172.21.15.148:50408) 2023-10-30T13:16:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf27fc640 1 -- 172.21.15.148:0/2039821483 learned_addr learned my addr 172.21.15.148:0/2039821483 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf2ffd640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc003f30 0x7fbdf41f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf37fe640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf40ced40 0x7fbdf414c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:45.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf27fc640 1 -- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf40ced40 msgr2=0x7fbdf414c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf27fc640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf40ced40 0x7fbdf414c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf27fc640 1 -- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc003f30 msgr2=0x7fbdf41f03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf27fc640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc003f30 0x7fbdf41f03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf27fc640 1 -- 172.21.15.148:0/2039821483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf4150610 con 0x7fbdf414ee00 2023-10-30T13:16:45.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf27fc640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf414ee00 0x7fbdf40ce260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009ae0 tx=0x7fbddc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbddb7fe640 1 -- 172.21.15.148:0/2039821483 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8004230 con 0x7fbdf414ee00 2023-10-30T13:16:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbddb7fe640 1 -- 172.21.15.148:0/2039821483 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8004390 con 0x7fbdf414ee00 2023-10-30T13:16:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbddb7fe640 1 -- 172.21.15.148:0/2039821483 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80ad850 con 0x7fbdf414ee00 2023-10-30T13:16:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf41508f0 con 0x7fbdf414ee00 2023-10-30T13:16:45.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.817+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf41e5380 con 0x7fbdf414ee00 2023-10-30T13:16:45.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.821+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf404e170 con 0x7fbdf414ee00 2023-10-30T13:16:45.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.821+0000 7fbddb7fe640 1 -- 172.21.15.148:0/2039821483 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbde80b4040 con 0x7fbdf414ee00 2023-10-30T13:16:45.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.821+0000 7fbddb7fe640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbde00664f0 0x7fbde00689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:45.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.821+0000 7fbddb7fe640 1 -- 172.21.15.148:0/2039821483 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fbde811e730 con 0x7fbdf414ee00 2023-10-30T13:16:45.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.821+0000 7fbdf2ffd640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbde00664f0 0x7fbde00689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:45.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.821+0000 7fbdf2ffd640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbde00664f0 0x7fbde00689b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbde4007d30 tx=0x7fbde4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:45.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:45.825+0000 7fbddb7fe640 1 -- 172.21.15.148:0/2039821483 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbde80ebb90 con 0x7fbdf414ee00 2023-10-30T13:16:46.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdf40b1dd0 con 0x7fbdf414ee00 2023-10-30T13:16:46.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbddb7fe640 1 -- 172.21.15.148:0/2039821483 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6539 (secure 0 0 0) 0x7fbde80ef1b0 con 0x7fbdf414ee00 2023-10-30T13:16:46.011 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:46.011 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:11.413204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:46.012 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 28 2023-10-30T13:16:46.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbde00664f0 msgr2=0x7fbde00689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:46.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbde00664f0 0x7fbde00689b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbde4007d30 tx=0x7fbde4009040 comp rx=0 tx=0).stop 2023-10-30T13:16:46.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf414ee00 msgr2=0x7fbdf40ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:46.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf414ee00 0x7fbdf40ce260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009ae0 tx=0x7fbddc005800 comp rx=0 tx=0).stop 2023-10-30T13:16:46.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 shutdown_connections 2023-10-30T13:16:46.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbde00664f0 0x7fbde00689b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:46.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf414ee00 0x7fbdf40ce260 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:46.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf40ced40 0x7fbdf414c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:46.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 --2- 172.21.15.148:0/2039821483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc003f30 0x7fbdf41f03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:46.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 >> 172.21.15.148:0/2039821483 conn(0x7fbdf406bfe0 msgr2=0x7fbdf40c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:46.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 shutdown_connections 2023-10-30T13:16:46.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:46.009+0000 7fbdf9668640 1 -- 172.21.15.148:0/2039821483 wait complete. 2023-10-30T13:16:46.032 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:16:48.033 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:48.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09b1017640 1 Processor -- start 2023-10-30T13:16:48.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09b1017640 1 -- start start 2023-10-30T13:16:48.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09b1017640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac1538a0 0x7f09ac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:48.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09b1017640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09ac0ca0e0 0x7f09ac15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:48.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09b1017640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f09ac0cf160 0x7f09ac0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:48.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09b1017640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac0cb800 con 0x7f09ac0cf160 2023-10-30T13:16:48.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09b1017640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac15dc30 con 0x7f09ac0ca0e0 2023-10-30T13:16:48.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09b1017640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0cb330 con 0x7f09ac1538a0 2023-10-30T13:16:48.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09ab59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f09ac0cf160 0x7f09ac0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:48.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09ab59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f09ac0cf160 0x7f09ac0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50412/0 (socket says 172.21.15.148:50412) 2023-10-30T13:16:48.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09ab59e640 1 -- 172.21.15.148:0/808815134 learned_addr learned my addr 172.21.15.148:0/808815134 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:48.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09aad9d640 1 --2- 172.21.15.148:0/808815134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac1538a0 0x7f09ac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:48.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09a3fff640 1 --2- 172.21.15.148:0/808815134 >> v2:172.21.15.150:3300/0 conn(0x7f09ac0ca0e0 0x7f09ac15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:48.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09ab59e640 1 -- 172.21.15.148:0/808815134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac1538a0 msgr2=0x7f09ac15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:48.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09ab59e640 1 --2- 172.21.15.148:0/808815134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac1538a0 0x7f09ac15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09ab59e640 1 -- 172.21.15.148:0/808815134 >> v2:172.21.15.150:3300/0 conn(0x7f09ac0ca0e0 msgr2=0x7f09ac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:48.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09ab59e640 1 --2- 172.21.15.148:0/808815134 >> v2:172.21.15.150:3300/0 conn(0x7f09ac0ca0e0 0x7f09ac15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.181+0000 7f09ab59e640 1 -- 172.21.15.148:0/808815134 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac0cad70 con 0x7f09ac0cf160 2023-10-30T13:16:48.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09ab59e640 1 --2- 172.21.15.148:0/808815134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 0x7f09ac0cf590 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0ccd10 tx=0x7f09a401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58741dd4fcd6834 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:48.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09aa59c640 1 -- 172.21.15.148:0/808815134 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a40af400 con 0x7f09ac0cf160 2023-10-30T13:16:48.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09aa59c640 1 -- 172.21.15.148:0/808815134 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a40ab050 con 0x7f09ac0cf160 2023-10-30T13:16:48.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09aa59c640 1 -- 172.21.15.148:0/808815134 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a40af780 con 0x7f09ac0cf160 2023-10-30T13:16:48.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09b1017640 1 -- 172.21.15.148:0/808815134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 msgr2=0x7f09ac0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:48.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09b1017640 1 --2- 172.21.15.148:0/808815134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 0x7f09ac0cf590 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0ccd10 tx=0x7f09a401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09b1017640 1 -- 172.21.15.148:0/808815134 shutdown_connections 2023-10-30T13:16:48.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09b1017640 1 --2- 172.21.15.148:0/808815134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 0x7f09ac0cf590 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09b1017640 1 --2- 172.21.15.148:0/808815134 >> v2:172.21.15.150:3300/0 conn(0x7f09ac0ca0e0 0x7f09ac15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09b1017640 1 --2- 172.21.15.148:0/808815134 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac1538a0 0x7f09ac15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09b1017640 1 -- 172.21.15.148:0/808815134 >> 172.21.15.148:0/808815134 conn(0x7f09ac06bfe0 msgr2=0x7f09ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:48.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09b1017640 1 -- 172.21.15.148:0/808815134 shutdown_connections 2023-10-30T13:16:48.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.185+0000 7f09b1017640 1 -- 172.21.15.148:0/808815134 wait complete. 2023-10-30T13:16:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 Processor -- start 2023-10-30T13:16:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 -- start start 2023-10-30T13:16:48.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac0ca0e0 0x7f09ac0cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:48.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 0x7f09ac15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:48.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac1538a0 0x7f09ac15adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:48.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac1f3060 con 0x7f09ac0cf160 2023-10-30T13:16:48.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac15cd20 con 0x7f09ac1538a0 2023-10-30T13:16:48.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac1f2450 con 0x7f09ac0ca0e0 2023-10-30T13:16:48.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 0x7f09ac15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:48.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09aad9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac0ca0e0 0x7f09ac0cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:48.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 0x7f09ac15a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50414/0 (socket says 172.21.15.148:50414) 2023-10-30T13:16:48.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a3fff640 1 -- 172.21.15.148:0/690910007 learned_addr learned my addr 172.21.15.148:0/690910007 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:48.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09ab59e640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac1538a0 0x7f09ac15adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:48.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a3fff640 1 -- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac0ca0e0 msgr2=0x7f09ac0cc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:48.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a3fff640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac0ca0e0 0x7f09ac0cc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a3fff640 1 -- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac1538a0 msgr2=0x7f09ac15adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:48.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a3fff640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac1538a0 0x7f09ac15adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a3fff640 1 -- 172.21.15.148:0/690910007 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4009b30 con 0x7f09ac0cf160 2023-10-30T13:16:48.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a3fff640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 0x7f09ac15a8b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f099400d800 tx=0x7f099400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:48.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a37fe640 1 -- 172.21.15.148:0/690910007 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09940096a0 con 0x7f09ac0cf160 2023-10-30T13:16:48.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a37fe640 1 -- 172.21.15.148:0/690910007 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0994009800 con 0x7f09ac0cf160 2023-10-30T13:16:48.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09a37fe640 1 -- 172.21.15.148:0/690910007 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0994009b70 con 0x7f09ac0cf160 2023-10-30T13:16:48.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac1f2730 con 0x7f09ac0cf160 2023-10-30T13:16:48.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac1f2cf0 con 0x7f09ac0cf160 2023-10-30T13:16:48.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.189+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac0b2a50 con 0x7f09ac0cf160 2023-10-30T13:16:48.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.193+0000 7f09a37fe640 1 -- 172.21.15.148:0/690910007 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0994009cd0 con 0x7f09ac0cf160 2023-10-30T13:16:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.193+0000 7f09a37fe640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f098c0664a0 0x7f098c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.193+0000 7f09a37fe640 1 -- 172.21.15.148:0/690910007 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f09940847b0 con 0x7f09ac0cf160 2023-10-30T13:16:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.193+0000 7f09aad9d640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f098c0664a0 0x7f098c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.193+0000 7f09aad9d640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f098c0664a0 0x7f098c068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f099c0079c0 tx=0x7f099c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:48.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.197+0000 7f09a37fe640 1 -- 172.21.15.148:0/690910007 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0994051a70 con 0x7f09ac0cf160 2023-10-30T13:16:48.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.377+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09ac04e170 con 0x7f09ac0cf160 2023-10-30T13:16:48.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.377+0000 7f09a37fe640 1 -- 172.21.15.148:0/690910007 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6539 (secure 0 0 0) 0x7f0994055090 con 0x7f09ac0cf160 2023-10-30T13:16:48.380 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:48.380 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:11.413204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":4270,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6836/1620877178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1620877178},{"type":"v1","addr":"172.21.15.150:6836","nonce":1620877178}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 28 2023-10-30T13:16:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f098c0664a0 msgr2=0x7f098c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f098c0664a0 0x7f098c068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f099c0079c0 tx=0x7f099c008040 comp rx=0 tx=0).stop 2023-10-30T13:16:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 msgr2=0x7f09ac15a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 0x7f09ac15a8b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f099400d800 tx=0x7f099400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 shutdown_connections 2023-10-30T13:16:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f098c0664a0 0x7f098c068960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09ac0cf160 0x7f09ac15a8b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09ac1538a0 0x7f09ac15adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 --2- 172.21.15.148:0/690910007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f09ac0ca0e0 0x7f09ac0cc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:48.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 >> 172.21.15.148:0/690910007 conn(0x7f09ac06bfe0 msgr2=0x7f09ac073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:48.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 shutdown_connections 2023-10-30T13:16:48.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:48.381+0000 7f09b1017640 1 -- 172.21.15.148:0/690910007 wait complete. 2023-10-30T13:16:48.400 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:16:50.402 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc669a9c640 1 Processor -- start 2023-10-30T13:16:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc669a9c640 1 -- start start 2023-10-30T13:16:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc669a9c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc66415d6f0 0x7fc66415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc669a9c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6640cf160 0x7fc6640cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc669a9c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6640cb800 con 0x7fc6641538a0 2023-10-30T13:16:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc669a9c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6640cb330 con 0x7fc66415d6f0 2023-10-30T13:16:50.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc669a9c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640cad70 con 0x7fc6640cf160 2023-10-30T13:16:50.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc6637fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc6641538a0 0x7fc6640ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39724/0 (socket says 172.21.15.148:39724) 2023-10-30T13:16:50.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc6637fe640 1 -- 172.21.15.148:0/1119380330 learned_addr learned my addr 172.21.15.148:0/1119380330 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:16:50.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc663fff640 1 --2- 172.21.15.148:0/1119380330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6640cf160 0x7fc6640cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc65bfff640 1 --2- 172.21.15.148:0/1119380330 >> v2:172.21.15.150:3300/0 conn(0x7fc66415d6f0 0x7fc66415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc662ffd640 1 -- 172.21.15.148:0/1119380330 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507173274 0 0) 0x7fc6640cb800 con 0x7fc6641538a0 2023-10-30T13:16:50.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc662ffd640 1 -- 172.21.15.148:0/1119380330 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc650003530 con 0x7fc6641538a0 2023-10-30T13:16:50.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc663fff640 1 -- 172.21.15.148:0/1119380330 >> v2:172.21.15.150:3300/0 conn(0x7fc66415d6f0 msgr2=0x7fc66415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:50.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc663fff640 1 --2- 172.21.15.148:0/1119380330 >> v2:172.21.15.150:3300/0 conn(0x7fc66415d6f0 0x7fc66415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc663fff640 1 -- 172.21.15.148:0/1119380330 >> v1:172.21.15.148:6789/0 conn(0x7fc6641538a0 legacy=0x7fc6640ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc662ffd640 1 -- 172.21.15.148:0/1119380330 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3124267137 0 0) 0x7fc650003530 con 0x7fc6641538a0 2023-10-30T13:16:50.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc663fff640 1 -- 172.21.15.148:0/1119380330 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6640cb050 con 0x7fc6640cf160 2023-10-30T13:16:50.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.549+0000 7fc65bfff640 1 --2- 172.21.15.148:0/1119380330 >> v2:172.21.15.150:3300/0 conn(0x7fc66415d6f0 0x7fc66415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:16:50.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc663fff640 1 --2- 172.21.15.148:0/1119380330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6640cf160 0x7fc6640cf590 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009d90 tx=0x7fc65c01f060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af130754d8435e58 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:50.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc662ffd640 1 -- 172.21.15.148:0/1119380330 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654001680 con 0x7fc6640cf160 2023-10-30T13:16:50.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc662ffd640 1 -- 172.21.15.148:0/1119380330 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c0a8bb0 con 0x7fc6640cf160 2023-10-30T13:16:50.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc662ffd640 1 -- 172.21.15.148:0/1119380330 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c0b0440 con 0x7fc6640cf160 2023-10-30T13:16:50.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc669a9c640 1 -- 172.21.15.148:0/1119380330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6640cf160 msgr2=0x7fc6640cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc669a9c640 1 --2- 172.21.15.148:0/1119380330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6640cf160 0x7fc6640cf590 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009d90 tx=0x7fc65c01f060 comp rx=0 tx=0).stop 2023-10-30T13:16:50.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc669a9c640 1 -- 172.21.15.148:0/1119380330 shutdown_connections 2023-10-30T13:16:50.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc669a9c640 1 --2- 172.21.15.148:0/1119380330 >> v2:172.21.15.150:3300/0 conn(0x7fc66415d6f0 0x7fc66415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc669a9c640 1 --2- 172.21.15.148:0/1119380330 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6640cf160 0x7fc6640cf590 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc669a9c640 1 -- 172.21.15.148:0/1119380330 >> 172.21.15.148:0/1119380330 conn(0x7fc66406bfe0 msgr2=0x7fc66406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:50.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc669a9c640 1 -- 172.21.15.148:0/1119380330 shutdown_connections 2023-10-30T13:16:50.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.553+0000 7fc669a9c640 1 -- 172.21.15.148:0/1119380330 wait complete. 2023-10-30T13:16:50.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 Processor -- start 2023-10-30T13:16:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 -- start start 2023-10-30T13:16:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6640cf160 0x7fc6641e6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6641538a0 0x7fc6641e3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66415d6f0 0x7fc66414cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6641f24d0 con 0x7fc6640cf160 2023-10-30T13:16:50.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6641f3470 con 0x7fc66415d6f0 2023-10-30T13:16:50.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66414e1f0 con 0x7fc6641538a0 2023-10-30T13:16:50.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc6637fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6640cf160 0x7fc6641e6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc65bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6641538a0 0x7fc6641e3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc6637fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6640cf160 0x7fc6641e6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50430/0 (socket says 172.21.15.148:50430) 2023-10-30T13:16:50.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc6637fe640 1 -- 172.21.15.148:0/756729005 learned_addr learned my addr 172.21.15.148:0/756729005 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:50.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc663fff640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66415d6f0 0x7fc66414cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc6637fe640 1 -- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6641538a0 msgr2=0x7fc6641e3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc6637fe640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6641538a0 0x7fc6641e3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc6637fe640 1 -- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66415d6f0 msgr2=0x7fc66414cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc6637fe640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66415d6f0 0x7fc66414cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc6637fe640 1 -- 172.21.15.148:0/756729005 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c009420 con 0x7fc6640cf160 2023-10-30T13:16:50.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc663fff640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66415d6f0 0x7fc66414cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:50.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc6637fe640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6640cf160 0x7fc6641e6250 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c750 tx=0x7fc65400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:50.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc660ff9640 1 -- 172.21.15.148:0/756729005 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654005ae0 con 0x7fc6640cf160 2023-10-30T13:16:50.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc660ff9640 1 -- 172.21.15.148:0/756729005 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc654005c40 con 0x7fc6640cf160 2023-10-30T13:16:50.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c009a70 con 0x7fc6640cf160 2023-10-30T13:16:50.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc660ff9640 1 -- 172.21.15.148:0/756729005 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654007070 con 0x7fc6640cf160 2023-10-30T13:16:50.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.557+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6641f26b0 con 0x7fc6640cf160 2023-10-30T13:16:50.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.561+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6640b2a50 con 0x7fc6640cf160 2023-10-30T13:16:50.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.561+0000 7fc660ff9640 1 -- 172.21.15.148:0/756729005 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6540071d0 con 0x7fc6640cf160 2023-10-30T13:16:50.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.561+0000 7fc660ff9640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6440664a0 0x7fc644068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.561+0000 7fc660ff9640 1 -- 172.21.15.148:0/756729005 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fc6540836e0 con 0x7fc6640cf160 2023-10-30T13:16:50.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.561+0000 7fc65bfff640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6440664a0 0x7fc644068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.561+0000 7fc65bfff640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6440664a0 0x7fc644068960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6641582f0 tx=0x7fc64c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:50.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.565+0000 7fc660ff9640 1 -- 172.21.15.148:0/756729005 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc654050a90 con 0x7fc6640cf160 2023-10-30T13:16:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.745+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc66404e170 con 0x7fc6640cf160 2023-10-30T13:16:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.745+0000 7fc660ff9640 1 -- 172.21.15.148:0/756729005 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fc6540540b0 con 0x7fc6640cf160 2023-10-30T13:16:50.748 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:50.748 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:49.403529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 29 2023-10-30T13:16:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6440664a0 msgr2=0x7fc644068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6440664a0 0x7fc644068960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6641582f0 tx=0x7fc64c009190 comp rx=0 tx=0).stop 2023-10-30T13:16:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6640cf160 msgr2=0x7fc6641e6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6640cf160 0x7fc6641e6250 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc65400c750 tx=0x7fc65400cc10 comp rx=0 tx=0).stop 2023-10-30T13:16:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 shutdown_connections 2023-10-30T13:16:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6440664a0 0x7fc644068960 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66415d6f0 0x7fc66414cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6641538a0 0x7fc6641e3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 --2- 172.21.15.148:0/756729005 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6640cf160 0x7fc6641e6250 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 >> 172.21.15.148:0/756729005 conn(0x7fc66406bfe0 msgr2=0x7fc6640c5170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:50.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 shutdown_connections 2023-10-30T13:16:50.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.749+0000 7fc669a9c640 1 -- 172.21.15.148:0/756729005 wait complete. 2023-10-30T13:16:50.769 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:16:50.769 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:16:50.769 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:50.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.905+0000 7fccbb365640 1 Processor -- start 2023-10-30T13:16:50.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.905+0000 7fccbb365640 1 -- start start 2023-10-30T13:16:50.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.905+0000 7fccbb365640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fccb41580e0 0x7fccb4070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.905+0000 7fccbb365640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fccb40c6540 0x7fccb415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.905+0000 7fccbb365640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb40c5810 con 0x7fccb4069e90 2023-10-30T13:16:50.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.905+0000 7fccbb365640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb40c6990 con 0x7fccb40c6540 2023-10-30T13:16:50.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.905+0000 7fccbb365640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb40c6bb0 con 0x7fccb41580e0 2023-10-30T13:16:50.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccb8900640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fccb40c6540 0x7fccb415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccb8900640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fccb40c6540 0x7fccb415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50438/0 (socket says 172.21.15.148:50438) 2023-10-30T13:16:50.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccb8900640 1 -- 172.21.15.148:0/2236711992 learned_addr learned my addr 172.21.15.148:0/2236711992 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:50.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccb9101640 1 --2- 172.21.15.148:0/2236711992 >> v2:172.21.15.150:3300/0 conn(0x7fccb41580e0 0x7fccb4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccb8900640 1 -- 172.21.15.148:0/2236711992 >> v2:172.21.15.150:3300/0 conn(0x7fccb41580e0 msgr2=0x7fccb4070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:50.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccb8900640 1 --2- 172.21.15.148:0/2236711992 >> v2:172.21.15.150:3300/0 conn(0x7fccb41580e0 0x7fccb4070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccb8900640 1 -- 172.21.15.148:0/2236711992 >> v1:172.21.15.150:6789/0 conn(0x7fccb4069e90 legacy=0x7fccb406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccb8900640 1 -- 172.21.15.148:0/2236711992 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40c6e70 con 0x7fccb40c6540 2023-10-30T13:16:50.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccb8900640 1 --2- 172.21.15.148:0/2236711992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb40c6540 0x7fccb415e700 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fccb406eea0 tx=0x7fcca401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca2bd7e6e05e476 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:50.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccabfff640 1 -- 172.21.15.148:0/2236711992 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca40aa660 con 0x7fccb40c6540 2023-10-30T13:16:50.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccabfff640 1 -- 172.21.15.148:0/2236711992 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca40aa7c0 con 0x7fccb40c6540 2023-10-30T13:16:50.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.909+0000 7fccabfff640 1 -- 172.21.15.148:0/2236711992 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca40aab50 con 0x7fccb40c6540 2023-10-30T13:16:50.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 -- 172.21.15.148:0/2236711992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb40c6540 msgr2=0x7fccb415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 --2- 172.21.15.148:0/2236711992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb40c6540 0x7fccb415e700 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fccb406eea0 tx=0x7fcca401c500 comp rx=0 tx=0).stop 2023-10-30T13:16:50.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 -- 172.21.15.148:0/2236711992 shutdown_connections 2023-10-30T13:16:50.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 --2- 172.21.15.148:0/2236711992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb40c6540 0x7fccb415e700 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 --2- 172.21.15.148:0/2236711992 >> v2:172.21.15.150:3300/0 conn(0x7fccb41580e0 0x7fccb4070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 -- 172.21.15.148:0/2236711992 >> 172.21.15.148:0/2236711992 conn(0x7fccb40c7910 msgr2=0x7fccb40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:50.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 -- 172.21.15.148:0/2236711992 shutdown_connections 2023-10-30T13:16:50.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 -- 172.21.15.148:0/2236711992 wait complete. 2023-10-30T13:16:50.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 Processor -- start 2023-10-30T13:16:50.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 -- start start 2023-10-30T13:16:50.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb4069e90 0x7fccb414dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccb40c6540 0x7fccb41450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccb41580e0 0x7fccb41442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40ca790 con 0x7fccb4069e90 2023-10-30T13:16:50.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb4149f30 con 0x7fccb41580e0 2023-10-30T13:16:50.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccbb365640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb414a180 con 0x7fccb40c6540 2023-10-30T13:16:50.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb9101640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb4069e90 0x7fccb414dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb9101640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb4069e90 0x7fccb414dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50440/0 (socket says 172.21.15.148:50440) 2023-10-30T13:16:50.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb9101640 1 -- 172.21.15.148:0/2374770354 learned_addr learned my addr 172.21.15.148:0/2374770354 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:50.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb9902640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccb41580e0 0x7fccb41442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb8900640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccb40c6540 0x7fccb41450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb9101640 1 -- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccb40c6540 msgr2=0x7fccb41450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb9101640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccb40c6540 0x7fccb41450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb9101640 1 -- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccb41580e0 msgr2=0x7fccb41442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:50.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb9101640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccb41580e0 0x7fccb41442b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:50.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.913+0000 7fccb9101640 1 -- 172.21.15.148:0/2374770354 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40094d0 con 0x7fccb4069e90 2023-10-30T13:16:50.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fccb9101640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb4069e90 0x7fccb414dbf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fccb000e870 tx=0x7fccb000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:50.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fcca9ffb640 1 -- 172.21.15.148:0/2374770354 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb0009510 con 0x7fccb4069e90 2023-10-30T13:16:50.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fcca9ffb640 1 -- 172.21.15.148:0/2374770354 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb000eec0 con 0x7fccb4069e90 2023-10-30T13:16:50.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb414a460 con 0x7fccb4069e90 2023-10-30T13:16:50.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fcca9ffb640 1 -- 172.21.15.148:0/2374770354 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb000cb40 con 0x7fccb4069e90 2023-10-30T13:16:50.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb414d3f0 con 0x7fccb4069e90 2023-10-30T13:16:50.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb404e170 con 0x7fccb4069e90 2023-10-30T13:16:50.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fcca9ffb640 1 -- 172.21.15.148:0/2374770354 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fccb000cca0 con 0x7fccb4069e90 2023-10-30T13:16:50.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fcca9ffb640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc980664a0 0x7fcc98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:50.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fcca9ffb640 1 -- 172.21.15.148:0/2374770354 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fccb00841c0 con 0x7fccb4069e90 2023-10-30T13:16:50.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.917+0000 7fccb8900640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc980664a0 0x7fcc98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:50.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.921+0000 7fccb8900640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc980664a0 0x7fcc98068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcca40ae000 tx=0x7fcca40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:50.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:50.921+0000 7fcca9ffb640 1 -- 172.21.15.148:0/2374770354 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fccb0051430 con 0x7fccb4069e90 2023-10-30T13:16:51.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.101+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccb406aed0 con 0x7fccb4069e90 2023-10-30T13:16:51.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.101+0000 7fcca9ffb640 1 -- 172.21.15.148:0/2374770354 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fccb0054a50 con 0x7fccb4069e90 2023-10-30T13:16:51.104 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:51.104 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:49.403529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:51.105 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 29 2023-10-30T13:16:51.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc980664a0 msgr2=0x7fcc98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:51.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc980664a0 0x7fcc98068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcca40ae000 tx=0x7fcca40a7000 comp rx=0 tx=0).stop 2023-10-30T13:16:51.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb4069e90 msgr2=0x7fccb414dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:51.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb4069e90 0x7fccb414dbf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fccb000e870 tx=0x7fccb000ec70 comp rx=0 tx=0).stop 2023-10-30T13:16:51.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 shutdown_connections 2023-10-30T13:16:51.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc980664a0 0x7fcc98068960 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:51.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccb41580e0 0x7fccb41442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:51.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fccb40c6540 0x7fccb41450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:51.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 --2- 172.21.15.148:0/2374770354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fccb4069e90 0x7fccb414dbf0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:51.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 >> 172.21.15.148:0/2374770354 conn(0x7fccb40c7910 msgr2=0x7fccb415e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:51.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 shutdown_connections 2023-10-30T13:16:51.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:51.105+0000 7fccbb365640 1 -- 172.21.15.148:0/2374770354 wait complete. 2023-10-30T13:16:51.125 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 29, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 29, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4285": { "addr": "172.21.15.148:6834/4162935019", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 4162935019, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 4162935019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4959": { "addr": "172.21.15.148:6837/95380877", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 95380877, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 95380877, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4959, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 16 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:16:49.403529+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 5103, "mds_2": 4959 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:16:53.127 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5843646640 1 Processor -- start 2023-10-30T13:16:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5843646640 1 -- start start 2023-10-30T13:16:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5843646640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f583c1538a0 0x7f583c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5843646640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f583c0ca0e0 0x7f583c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5843646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c0cf160 0x7f583c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5843646640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583c0cb800 con 0x7f583c1538a0 2023-10-30T13:16:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5843646640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583c15dc30 con 0x7f583c0ca0e0 2023-10-30T13:16:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5843646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0cb330 con 0x7f583c0cf160 2023-10-30T13:16:53.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f58413e2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f583c1538a0 0x7f583c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:53.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f58413e2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f583c1538a0 0x7f583c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40152/0 (socket says 172.21.15.148:40152) 2023-10-30T13:16:53.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f58413e2640 1 -- 172.21.15.148:0/2818483227 learned_addr learned my addr 172.21.15.148:0/2818483227 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:53.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5841be3640 1 --2- 172.21.15.148:0/2818483227 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c0cf160 0x7f583c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:53.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5840be1640 1 --2- 172.21.15.148:0/2818483227 >> v2:172.21.15.150:3300/0 conn(0x7f583c0ca0e0 0x7f583c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:53.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f58413e2640 1 -- 172.21.15.148:0/2818483227 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c0cf160 msgr2=0x7f583c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:53.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f58413e2640 1 --2- 172.21.15.148:0/2818483227 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c0cf160 0x7f583c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f58413e2640 1 -- 172.21.15.148:0/2818483227 >> v2:172.21.15.150:3300/0 conn(0x7f583c0ca0e0 msgr2=0x7f583c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:53.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f58413e2640 1 --2- 172.21.15.148:0/2818483227 >> v2:172.21.15.150:3300/0 conn(0x7f583c0ca0e0 0x7f583c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f58413e2640 1 -- 172.21.15.148:0/2818483227 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c0cad70 con 0x7f583c1538a0 2023-10-30T13:16:53.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.273+0000 7f5841be3640 1 --2- 172.21.15.148:0/2818483227 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c0cf160 0x7f583c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:53.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f58413e2640 1 --2- 172.21.15.148:0/2818483227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c1538a0 0x7f583c15d6f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f583c0b30a0 tx=0x7f583801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abf5a77c69bfc30 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:53.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5833fff640 1 -- 172.21.15.148:0/2818483227 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58380a89b0 con 0x7f583c1538a0 2023-10-30T13:16:53.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5833fff640 1 -- 172.21.15.148:0/2818483227 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58380a8b10 con 0x7f583c1538a0 2023-10-30T13:16:53.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5833fff640 1 -- 172.21.15.148:0/2818483227 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58380a8ea0 con 0x7f583c1538a0 2023-10-30T13:16:53.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5843646640 1 -- 172.21.15.148:0/2818483227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c1538a0 msgr2=0x7f583c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:53.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5843646640 1 --2- 172.21.15.148:0/2818483227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c1538a0 0x7f583c15d6f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f583c0b30a0 tx=0x7f583801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5843646640 1 -- 172.21.15.148:0/2818483227 shutdown_connections 2023-10-30T13:16:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5843646640 1 --2- 172.21.15.148:0/2818483227 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c0cf160 0x7f583c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5843646640 1 --2- 172.21.15.148:0/2818483227 >> v2:172.21.15.150:3300/0 conn(0x7f583c0ca0e0 0x7f583c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5843646640 1 --2- 172.21.15.148:0/2818483227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c1538a0 0x7f583c15d6f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5843646640 1 -- 172.21.15.148:0/2818483227 >> 172.21.15.148:0/2818483227 conn(0x7f583c06bfe0 msgr2=0x7f583c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5843646640 1 -- 172.21.15.148:0/2818483227 shutdown_connections 2023-10-30T13:16:53.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.277+0000 7f5843646640 1 -- 172.21.15.148:0/2818483227 wait complete. 2023-10-30T13:16:53.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 Processor -- start 2023-10-30T13:16:53.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 -- start start 2023-10-30T13:16:53.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f583c0ca0e0 0x7f583c1f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:53.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c0cf160 0x7f583c15b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:53.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c1538a0 0x7f583c065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:53.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c1f3750 con 0x7f583c0cf160 2023-10-30T13:16:53.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0665a0 con 0x7f583c0ca0e0 2023-10-30T13:16:53.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c1ef290 con 0x7f583c1538a0 2023-10-30T13:16:53.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5840be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c0cf160 0x7f583c15b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:53.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5840be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c0cf160 0x7f583c15b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40162/0 (socket says 172.21.15.148:40162) 2023-10-30T13:16:53.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5840be1640 1 -- 172.21.15.148:0/1656500406 learned_addr learned my addr 172.21.15.148:0/1656500406 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:53.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f58413e2640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f583c0ca0e0 0x7f583c1f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:53.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5841be3640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c1538a0 0x7f583c065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:53.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5840be1640 1 -- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c1538a0 msgr2=0x7f583c065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:53.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5840be1640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c1538a0 0x7f583c065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5840be1640 1 -- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f583c0ca0e0 msgr2=0x7f583c1f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:53.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5840be1640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f583c0ca0e0 0x7f583c1f3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5840be1640 1 -- 172.21.15.148:0/1656500406 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58380094d0 con 0x7f583c0cf160 2023-10-30T13:16:53.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f58413e2640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f583c0ca0e0 0x7f583c1f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:53.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5841be3640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c1538a0 0x7f583c065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:53.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5840be1640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c0cf160 0x7f583c15b670 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f582c00d800 tx=0x7f582c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:53.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5831ffb640 1 -- 172.21.15.148:0/1656500406 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c0096a0 con 0x7f583c0cf160 2023-10-30T13:16:53.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5831ffb640 1 -- 172.21.15.148:0/1656500406 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c009800 con 0x7f583c0cf160 2023-10-30T13:16:53.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5831ffb640 1 -- 172.21.15.148:0/1656500406 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c009b70 con 0x7f583c0cf160 2023-10-30T13:16:53.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c15abf0 con 0x7f583c0cf160 2023-10-30T13:16:53.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.281+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c1ef550 con 0x7f583c0cf160 2023-10-30T13:16:53.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.285+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c0b2a50 con 0x7f583c0cf160 2023-10-30T13:16:53.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.285+0000 7f5831ffb640 1 -- 172.21.15.148:0/1656500406 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f582c009cd0 con 0x7f583c0cf160 2023-10-30T13:16:53.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.285+0000 7f5831ffb640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58200664a0 0x7f5820068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:53.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.285+0000 7f5831ffb640 1 -- 172.21.15.148:0/1656500406 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f582c084760 con 0x7f583c0cf160 2023-10-30T13:16:53.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.285+0000 7f58413e2640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58200664a0 0x7f5820068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:53.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.285+0000 7f58413e2640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58200664a0 0x7f5820068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f58380b2000 tx=0x7f5838008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:53.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.289+0000 7f5831ffb640 1 -- 172.21.15.148:0/1656500406 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f582c051a60 con 0x7f583c0cf160 2023-10-30T13:16:53.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.469+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f583c04e170 con 0x7f583c0cf160 2023-10-30T13:16:53.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5831ffb640 1 -- 172.21.15.148:0/1656500406 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f582c055080 con 0x7f583c0cf160 2023-10-30T13:16:53.473 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:53.473 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:49.403529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 29 2023-10-30T13:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58200664a0 msgr2=0x7f5820068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58200664a0 0x7f5820068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f58380b2000 tx=0x7f5838008040 comp rx=0 tx=0).stop 2023-10-30T13:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c0cf160 msgr2=0x7f583c15b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:53.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c0cf160 0x7f583c15b670 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f582c00d800 tx=0x7f582c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 shutdown_connections 2023-10-30T13:16:53.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58200664a0 0x7f5820068960 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f583c0cf160 0x7f583c15b670 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f583c1538a0 0x7f583c065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 --2- 172.21.15.148:0/1656500406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f583c0ca0e0 0x7f583c1f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:53.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 >> 172.21.15.148:0/1656500406 conn(0x7f583c06bfe0 msgr2=0x7f583c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:53.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 shutdown_connections 2023-10-30T13:16:53.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:53.473+0000 7f5843646640 1 -- 172.21.15.148:0/1656500406 wait complete. 2023-10-30T13:16:53.494 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:54.059 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:16:54.063 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:16:55.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:55.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.633+0000 7f824a1d3640 1 Processor -- start 2023-10-30T13:16:55.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.633+0000 7f824a1d3640 1 -- start start 2023-10-30T13:16:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f824a1d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f82440c6b80 0x7f82440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f824a1d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82440c6f60 0x7f82440c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f824a1d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 0x7f824413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f824a1d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440c0a50 con 0x7f82440c6f60 2023-10-30T13:16:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f824a1d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824406f6e0 con 0x7f82440c6b80 2023-10-30T13:16:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f824a1d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824406f9c0 con 0x7f82440c8430 2023-10-30T13:16:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f8242ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82440c6f60 0x7f82440c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f8242ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82440c6f60 0x7f82440c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40176/0 (socket says 172.21.15.148:40176) 2023-10-30T13:16:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f8242ffd640 1 -- 172.21.15.148:0/2123020041 learned_addr learned my addr 172.21.15.148:0/2123020041 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f82437fe640 1 --2- 172.21.15.148:0/2123020041 >> v2:172.21.15.150:3300/0 conn(0x7f82440c6b80 0x7f82440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f8243fff640 1 --2- 172.21.15.148:0/2123020041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 0x7f824413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f8242ffd640 1 -- 172.21.15.148:0/2123020041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 msgr2=0x7f824413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f8242ffd640 1 --2- 172.21.15.148:0/2123020041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 0x7f824413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f8242ffd640 1 -- 172.21.15.148:0/2123020041 >> v2:172.21.15.150:3300/0 conn(0x7f82440c6b80 msgr2=0x7f82440bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f8242ffd640 1 --2- 172.21.15.148:0/2123020041 >> v2:172.21.15.150:3300/0 conn(0x7f82440c6b80 0x7f82440bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f8242ffd640 1 -- 172.21.15.148:0/2123020041 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824406fca0 con 0x7f82440c6f60 2023-10-30T13:16:55.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.637+0000 7f82437fe640 1 --2- 172.21.15.148:0/2123020041 >> v2:172.21.15.150:3300/0 conn(0x7f82440c6b80 0x7f82440bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:55.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f8242ffd640 1 --2- 172.21.15.148:0/2123020041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6f60 0x7f82440c0400 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f824406f1b0 tx=0x7f823801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8c6dab7ab9ebde server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:55.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f82427fc640 1 -- 172.21.15.148:0/2123020041 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82380ac630 con 0x7f82440c6f60 2023-10-30T13:16:55.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f82427fc640 1 -- 172.21.15.148:0/2123020041 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82380ac790 con 0x7f82440c6f60 2023-10-30T13:16:55.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f82427fc640 1 -- 172.21.15.148:0/2123020041 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82380acb20 con 0x7f82440c6f60 2023-10-30T13:16:55.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 -- 172.21.15.148:0/2123020041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6f60 msgr2=0x7f82440c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 --2- 172.21.15.148:0/2123020041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6f60 0x7f82440c0400 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f824406f1b0 tx=0x7f823801c500 comp rx=0 tx=0).stop 2023-10-30T13:16:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 -- 172.21.15.148:0/2123020041 shutdown_connections 2023-10-30T13:16:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 --2- 172.21.15.148:0/2123020041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 0x7f824413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 --2- 172.21.15.148:0/2123020041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6f60 0x7f82440c0400 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 --2- 172.21.15.148:0/2123020041 >> v2:172.21.15.150:3300/0 conn(0x7f82440c6b80 0x7f82440bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 -- 172.21.15.148:0/2123020041 >> 172.21.15.148:0/2123020041 conn(0x7f82440c7910 msgr2=0x7f82440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:55.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 -- 172.21.15.148:0/2123020041 shutdown_connections 2023-10-30T13:16:55.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 -- 172.21.15.148:0/2123020041 wait complete. 2023-10-30T13:16:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 Processor -- start 2023-10-30T13:16:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.641+0000 7f824a1d3640 1 -- start start 2023-10-30T13:16:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f824a1d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6b80 0x7f8244138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f824a1d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82440c6f60 0x7f8244135540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f824a1d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 0x7f8244135a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f824a1d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8244158ae0 con 0x7f82440c6b80 2023-10-30T13:16:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f824a1d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82441f1d80 con 0x7f82440c6f60 2023-10-30T13:16:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f824a1d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82441f6260 con 0x7f82440c8430 2023-10-30T13:16:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f82437fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6b80 0x7f8244138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f82437fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6b80 0x7f8244138910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40178/0 (socket says 172.21.15.148:40178) 2023-10-30T13:16:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f82437fe640 1 -- 172.21.15.148:0/1181683867 learned_addr learned my addr 172.21.15.148:0/1181683867 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f8242ffd640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82440c6f60 0x7f8244135540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f8243fff640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 0x7f8244135a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f82437fe640 1 -- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 msgr2=0x7f8244135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f82437fe640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 0x7f8244135a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f82437fe640 1 -- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82440c6f60 msgr2=0x7f8244135540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:55.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f82437fe640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82440c6f60 0x7f8244135540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f82437fe640 1 -- 172.21.15.148:0/1181683867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82380094d0 con 0x7f82440c6b80 2023-10-30T13:16:55.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f82437fe640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6b80 0x7f8244138910 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f823400e840 tx=0x7f823400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:55.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f822bfff640 1 -- 172.21.15.148:0/1181683867 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234004030 con 0x7f82440c6b80 2023-10-30T13:16:55.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f822bfff640 1 -- 172.21.15.148:0/1181683867 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8234004190 con 0x7f82440c6b80 2023-10-30T13:16:55.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f822bfff640 1 -- 172.21.15.148:0/1181683867 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234004520 con 0x7f82440c6b80 2023-10-30T13:16:55.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82441f2b60 con 0x7f82440c6b80 2023-10-30T13:16:55.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82441f1f90 con 0x7f82440c6b80 2023-10-30T13:16:55.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824404e170 con 0x7f82440c6b80 2023-10-30T13:16:55.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.645+0000 7f822bfff640 1 -- 172.21.15.148:0/1181683867 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f823400c6d0 con 0x7f82440c6b80 2023-10-30T13:16:55.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.649+0000 7f822bfff640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82300664a0 0x7f8230068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:55.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.649+0000 7f822bfff640 1 -- 172.21.15.148:0/1181683867 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f8234083c40 con 0x7f82440c6b80 2023-10-30T13:16:55.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.649+0000 7f8242ffd640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82300664a0 0x7f8230068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:55.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.649+0000 7f8242ffd640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82300664a0 0x7f8230068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f82380ae000 tx=0x7f8238008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:55.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.649+0000 7f822bfff640 1 -- 172.21.15.148:0/1181683867 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8234050ea0 con 0x7f82440c6b80 2023-10-30T13:16:55.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8244154060 con 0x7f82440c6b80 2023-10-30T13:16:55.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f822bfff640 1 -- 172.21.15.148:0/1181683867 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f8234020090 con 0x7f82440c6b80 2023-10-30T13:16:55.834 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:55.835 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:54.560258+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:resolve","state_seq":19,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:55.835 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 30 2023-10-30T13:16:55.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82300664a0 msgr2=0x7f8230068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:55.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82300664a0 0x7f8230068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f82380ae000 tx=0x7f8238008040 comp rx=0 tx=0).stop 2023-10-30T13:16:55.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6b80 msgr2=0x7f8244138910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:55.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6b80 0x7f8244138910 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f823400e840 tx=0x7f823400ed00 comp rx=0 tx=0).stop 2023-10-30T13:16:55.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 shutdown_connections 2023-10-30T13:16:55.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82300664a0 0x7f8230068960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82440c8430 0x7f8244135a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82440c6f60 0x7f8244135540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 --2- 172.21.15.148:0/1181683867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82440c6b80 0x7f8244138910 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:55.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 >> 172.21.15.148:0/1181683867 conn(0x7f82440c7910 msgr2=0x7f824406a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:55.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 shutdown_connections 2023-10-30T13:16:55.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:55.833+0000 7f824a1d3640 1 -- 172.21.15.148:0/1181683867 wait complete. 2023-10-30T13:16:55.855 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:57.856 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:58.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd3a6c9640 1 Processor -- start 2023-10-30T13:16:58.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd3a6c9640 1 -- start start 2023-10-30T13:16:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd3a6c9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcd341538a0 0x7fcd3415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd3a6c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcd340ca0e0 0x7fcd3415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd3a6c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd340cb800 con 0x7fcd340cf160 2023-10-30T13:16:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd3a6c9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3415dc30 con 0x7fcd341538a0 2023-10-30T13:16:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd3a6c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd340cb330 con 0x7fcd340ca0e0 2023-10-30T13:16:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd33fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcd341538a0 0x7fcd3415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd33fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcd341538a0 0x7fcd3415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40186/0 (socket says 172.21.15.148:40186) 2023-10-30T13:16:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd33fff640 1 -- 172.21.15.148:0/3616217582 learned_addr learned my addr 172.21.15.148:0/3616217582 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd337fe640 1 --2- 172.21.15.148:0/3616217582 >> v2:172.21.15.150:3300/0 conn(0x7fcd340ca0e0 0x7fcd3415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd33fff640 1 -- 172.21.15.148:0/3616217582 >> v2:172.21.15.150:3300/0 conn(0x7fcd340ca0e0 msgr2=0x7fcd3415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd33fff640 1 --2- 172.21.15.148:0/3616217582 >> v2:172.21.15.150:3300/0 conn(0x7fcd340ca0e0 0x7fcd3415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd33fff640 1 -- 172.21.15.148:0/3616217582 >> v1:172.21.15.150:6789/0 conn(0x7fcd340cf160 legacy=0x7fcd340cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:58.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.005+0000 7fcd33fff640 1 -- 172.21.15.148:0/3616217582 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd340cad70 con 0x7fcd341538a0 2023-10-30T13:16:58.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd33fff640 1 --2- 172.21.15.148:0/3616217582 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 0x7fcd3415d6f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcd340b30a0 tx=0x7fcd2401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8939a244a8b44d9a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:58.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd32ffd640 1 -- 172.21.15.148:0/3616217582 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240a89b0 con 0x7fcd341538a0 2023-10-30T13:16:58.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd32ffd640 1 -- 172.21.15.148:0/3616217582 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd240a8b10 con 0x7fcd341538a0 2023-10-30T13:16:58.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd32ffd640 1 -- 172.21.15.148:0/3616217582 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240a8ea0 con 0x7fcd341538a0 2023-10-30T13:16:58.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/3616217582 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 msgr2=0x7fcd3415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:58.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd3a6c9640 1 --2- 172.21.15.148:0/3616217582 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 0x7fcd3415d6f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcd340b30a0 tx=0x7fcd2401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/3616217582 shutdown_connections 2023-10-30T13:16:58.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd3a6c9640 1 --2- 172.21.15.148:0/3616217582 >> v2:172.21.15.150:3300/0 conn(0x7fcd340ca0e0 0x7fcd3415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd3a6c9640 1 --2- 172.21.15.148:0/3616217582 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 0x7fcd3415d6f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/3616217582 >> 172.21.15.148:0/3616217582 conn(0x7fcd3406bfe0 msgr2=0x7fcd3406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:58.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/3616217582 shutdown_connections 2023-10-30T13:16:58.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.009+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/3616217582 wait complete. 2023-10-30T13:16:58.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 Processor -- start 2023-10-30T13:16:58.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 -- start start 2023-10-30T13:16:58.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd340ca0e0 0x7fcd340b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:58.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd340cf160 0x7fcd341466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 0x7fcd341498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340cda00 con 0x7fcd341538a0 2023-10-30T13:16:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd341e2ae0 con 0x7fcd340cf160 2023-10-30T13:16:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340b3570 con 0x7fcd340ca0e0 2023-10-30T13:16:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd38c66640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 0x7fcd341498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:58.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd38c66640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 0x7fcd341498e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40188/0 (socket says 172.21.15.148:40188) 2023-10-30T13:16:58.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd38c66640 1 -- 172.21.15.148:0/674489947 learned_addr learned my addr 172.21.15.148:0/674489947 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:16:58.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd33fff640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd340ca0e0 0x7fcd340b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:58.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd337fe640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd340cf160 0x7fcd341466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:58.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd38c66640 1 -- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd340ca0e0 msgr2=0x7fcd340b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:58.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd38c66640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd340ca0e0 0x7fcd340b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd38c66640 1 -- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd340cf160 msgr2=0x7fcd341466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:58.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd38c66640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd340cf160 0x7fcd341466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd38c66640 1 -- 172.21.15.148:0/674489947 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd240094d0 con 0x7fcd341538a0 2023-10-30T13:16:58.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd38c66640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 0x7fcd341498e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00c780 tx=0x7fcd2c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:58.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd30ff9640 1 -- 172.21.15.148:0/674489947 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c004b20 con 0x7fcd341538a0 2023-10-30T13:16:58.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd30ff9640 1 -- 172.21.15.148:0/674489947 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c004c80 con 0x7fcd341538a0 2023-10-30T13:16:58.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd34145ff0 con 0x7fcd341538a0 2023-10-30T13:16:58.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd30ff9640 1 -- 172.21.15.148:0/674489947 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c00aa90 con 0x7fcd341538a0 2023-10-30T13:16:58.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.013+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd340ccfb0 con 0x7fcd341538a0 2023-10-30T13:16:58.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.017+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd340b2a50 con 0x7fcd341538a0 2023-10-30T13:16:58.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.017+0000 7fcd30ff9640 1 -- 172.21.15.148:0/674489947 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd2c00abf0 con 0x7fcd341538a0 2023-10-30T13:16:58.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.017+0000 7fcd30ff9640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd200664a0 0x7fcd20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:58.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.017+0000 7fcd30ff9640 1 -- 172.21.15.148:0/674489947 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fcd2c084070 con 0x7fcd341538a0 2023-10-30T13:16:58.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.017+0000 7fcd33fff640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd200664a0 0x7fcd20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:58.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.017+0000 7fcd33fff640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd200664a0 0x7fcd20068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcd240b2000 tx=0x7fcd24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:58.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.021+0000 7fcd30ff9640 1 -- 172.21.15.148:0/674489947 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd2c0b5090 con 0x7fcd341538a0 2023-10-30T13:16:58.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd3404e170 con 0x7fcd341538a0 2023-10-30T13:16:58.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd30ff9640 1 -- 172.21.15.148:0/674489947 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5747 (secure 0 0 0) 0x7fcd2c051370 con 0x7fcd341538a0 2023-10-30T13:16:58.202 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:16:58.202 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:54.560258+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:resolve","state_seq":19,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:58.202 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 30 2023-10-30T13:16:58.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd200664a0 msgr2=0x7fcd20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:58.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd200664a0 0x7fcd20068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcd240b2000 tx=0x7fcd24009040 comp rx=0 tx=0).stop 2023-10-30T13:16:58.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 msgr2=0x7fcd341498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:58.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 0x7fcd341498e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00c780 tx=0x7fcd2c00cc40 comp rx=0 tx=0).stop 2023-10-30T13:16:58.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 shutdown_connections 2023-10-30T13:16:58.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd200664a0 0x7fcd20068960 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd341538a0 0x7fcd341498e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd340cf160 0x7fcd341466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 --2- 172.21.15.148:0/674489947 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd340ca0e0 0x7fcd340b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:58.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 >> 172.21.15.148:0/674489947 conn(0x7fcd3406bfe0 msgr2=0x7fcd340c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:58.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 shutdown_connections 2023-10-30T13:16:58.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:16:58.201+0000 7fcd3a6c9640 1 -- 172.21.15.148:0/674489947 wait complete. 2023-10-30T13:16:58.222 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:17:00.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:00.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.361+0000 7f9292cd6640 1 Processor -- start 2023-10-30T13:17:00.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.361+0000 7f9292cd6640 1 -- start start 2023-10-30T13:17:00.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9292cd6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f928c06f7a0 0x7f928c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:00.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9292cd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 0x7f928c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:00.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9292cd6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f928c071110 0x7f928c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:00.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9292cd6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928c0700c0 con 0x7f928c06f7a0 2023-10-30T13:17:00.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9292cd6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f928c06a3d0 con 0x7f928c071110 2023-10-30T13:17:00.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9292cd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928c066cf0 con 0x7f928c0c6bd0 2023-10-30T13:17:00.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9290a72640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f928c06f7a0 0x7f928c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:00.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9290a72640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f928c06f7a0 0x7f928c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40194/0 (socket says 172.21.15.148:40194) 2023-10-30T13:17:00.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9290a72640 1 -- 172.21.15.148:0/4061556638 learned_addr learned my addr 172.21.15.148:0/4061556638 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:00.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9291273640 1 --2- 172.21.15.148:0/4061556638 >> v2:172.21.15.150:3300/0 conn(0x7f928c071110 0x7f928c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:00.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f928bfff640 1 --2- 172.21.15.148:0/4061556638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 0x7f928c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:00.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9290a72640 1 -- 172.21.15.148:0/4061556638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 msgr2=0x7f928c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:00.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9290a72640 1 --2- 172.21.15.148:0/4061556638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 0x7f928c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9290a72640 1 -- 172.21.15.148:0/4061556638 >> v2:172.21.15.150:3300/0 conn(0x7f928c071110 msgr2=0x7f928c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:00.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9290a72640 1 --2- 172.21.15.148:0/4061556638 >> v2:172.21.15.150:3300/0 conn(0x7f928c071110 0x7f928c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.365+0000 7f9290a72640 1 -- 172.21.15.148:0/4061556638 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c0c70d0 con 0x7f928c06f7a0 2023-10-30T13:17:00.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9290a72640 1 --2- 172.21.15.148:0/4061556638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 0x7f928c06fb80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f928c06c7d0 tx=0x7f928001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ee53208d6f569e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:00.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f928b7fe640 1 -- 172.21.15.148:0/4061556638 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800a89b0 con 0x7f928c06f7a0 2023-10-30T13:17:00.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f928b7fe640 1 -- 172.21.15.148:0/4061556638 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92800a8b10 con 0x7f928c06f7a0 2023-10-30T13:17:00.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f928b7fe640 1 -- 172.21.15.148:0/4061556638 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800a8ea0 con 0x7f928c06f7a0 2023-10-30T13:17:00.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 -- 172.21.15.148:0/4061556638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 msgr2=0x7f928c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:00.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 --2- 172.21.15.148:0/4061556638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 0x7f928c06fb80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f928c06c7d0 tx=0x7f928001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 -- 172.21.15.148:0/4061556638 shutdown_connections 2023-10-30T13:17:00.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 --2- 172.21.15.148:0/4061556638 >> v2:172.21.15.150:3300/0 conn(0x7f928c071110 0x7f928c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 --2- 172.21.15.148:0/4061556638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 0x7f928c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 --2- 172.21.15.148:0/4061556638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 0x7f928c06fb80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 -- 172.21.15.148:0/4061556638 >> 172.21.15.148:0/4061556638 conn(0x7f928c0c7910 msgr2=0x7f928c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:00.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 -- 172.21.15.148:0/4061556638 shutdown_connections 2023-10-30T13:17:00.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 -- 172.21.15.148:0/4061556638 wait complete. 2023-10-30T13:17:00.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 Processor -- start 2023-10-30T13:17:00.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.369+0000 7f9292cd6640 1 -- start start 2023-10-30T13:17:00.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9292cd6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 0x7f928c149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:00.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9292cd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928c071110 0x7f928c0c1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:00.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9292cd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 0x7f928c0c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:00.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9292cd6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c0c0f50 con 0x7f928c06f7a0 2023-10-30T13:17:00.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9292cd6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928c14a120 con 0x7f928c071110 2023-10-30T13:17:00.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9292cd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928c0af3c0 con 0x7f928c0c6bd0 2023-10-30T13:17:00.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9290a72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 0x7f928c149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:00.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9290a72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 0x7f928c149be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40196/0 (socket says 172.21.15.148:40196) 2023-10-30T13:17:00.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9290a72640 1 -- 172.21.15.148:0/1723462942 learned_addr learned my addr 172.21.15.148:0/1723462942 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:00.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9291273640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 0x7f928c0c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:00.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f928bfff640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928c071110 0x7f928c0c1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:00.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9290a72640 1 -- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 msgr2=0x7f928c0c18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:00.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9290a72640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 0x7f928c0c18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9290a72640 1 -- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928c071110 msgr2=0x7f928c0c1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:00.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9290a72640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928c071110 0x7f928c0c1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9290a72640 1 -- 172.21.15.148:0/1723462942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92800094d0 con 0x7f928c06f7a0 2023-10-30T13:17:00.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9291273640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 0x7f928c0c18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:00.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f928bfff640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928c071110 0x7f928c0c1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:00.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9290a72640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 0x7f928c149be0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f928001d3a0 tx=0x7f928001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:00.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f92897fa640 1 -- 172.21.15.148:0/1723462942 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800b0d40 con 0x7f928c06f7a0 2023-10-30T13:17:00.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f92897fa640 1 -- 172.21.15.148:0/1723462942 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92800bb460 con 0x7f928c06f7a0 2023-10-30T13:17:00.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f928c0aee00 con 0x7f928c06f7a0 2023-10-30T13:17:00.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f92897fa640 1 -- 172.21.15.148:0/1723462942 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800b0ea0 con 0x7f928c06f7a0 2023-10-30T13:17:00.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928c0c0890 con 0x7f928c06f7a0 2023-10-30T13:17:00.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928c0c06f0 con 0x7f928c06f7a0 2023-10-30T13:17:00.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.373+0000 7f92897fa640 1 -- 172.21.15.148:0/1723462942 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f92800b83f0 con 0x7f928c06f7a0 2023-10-30T13:17:00.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.377+0000 7f92897fa640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92700664a0 0x7f9270068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.377+0000 7f92897fa640 1 -- 172.21.15.148:0/1723462942 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f9280121830 con 0x7f928c06f7a0 2023-10-30T13:17:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.377+0000 7f928bfff640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92700664a0 0x7f9270068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.377+0000 7f928bfff640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92700664a0 0x7f9270068960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f928c0c4640 tx=0x7f9278008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:00.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.381+0000 7f92897fa640 1 -- 172.21.15.148:0/1723462942 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f928c0c06f0 con 0x7f928c06f7a0 2023-10-30T13:17:00.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.561+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f928c0c06f0 con 0x7f928c06f7a0 2023-10-30T13:17:00.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.561+0000 7f92897fa640 1 -- 172.21.15.148:0/1723462942 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f928c0c06f0 con 0x7f928c06f7a0 2023-10-30T13:17:00.564 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:17:00.564 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:16:54.560258+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:resolve","state_seq":19,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:00.564 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 30 2023-10-30T13:17:00.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92700664a0 msgr2=0x7f9270068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:00.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92700664a0 0x7f9270068960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f928c0c4640 tx=0x7f9278008040 comp rx=0 tx=0).stop 2023-10-30T13:17:00.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 msgr2=0x7f928c149be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:00.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 0x7f928c149be0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f928001d3a0 tx=0x7f928001ba90 comp rx=0 tx=0).stop 2023-10-30T13:17:00.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 shutdown_connections 2023-10-30T13:17:00.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92700664a0 0x7f9270068960 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f928c0c6bd0 0x7f928c0c18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928c071110 0x7f928c0c1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 --2- 172.21.15.148:0/1723462942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f928c06f7a0 0x7f928c149be0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:00.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 >> 172.21.15.148:0/1723462942 conn(0x7f928c0c7910 msgr2=0x7f928c150180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:00.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 shutdown_connections 2023-10-30T13:17:00.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:00.565+0000 7f9292cd6640 1 -- 172.21.15.148:0/1723462942 wait complete. 2023-10-30T13:17:00.585 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:17:02.585 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:02.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f548269a640 1 Processor -- start 2023-10-30T13:17:02.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f548269a640 1 -- start start 2023-10-30T13:17:02.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f548269a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f547c1538a0 0x7f547c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:02.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f548269a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f547c0ca0e0 0x7f547c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:02.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f548269a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 0x7f547c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:02.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f548269a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f547c0cb800 con 0x7f547c0ca0e0 2023-10-30T13:17:02.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f548269a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f547c15dc30 con 0x7f547c1538a0 2023-10-30T13:17:02.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f548269a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0cb330 con 0x7f547c0cf160 2023-10-30T13:17:02.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f547b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f547c0ca0e0 0x7f547c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:02.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f547bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f547c1538a0 0x7f547c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:02.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f547b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f547c0ca0e0 0x7f547c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60930/0 (socket says 172.21.15.148:60930) 2023-10-30T13:17:02.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f547b7fe640 1 -- 172.21.15.148:0/3104222672 learned_addr learned my addr 172.21.15.148:0/3104222672 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:02.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f5480c37640 1 --2- 172.21.15.148:0/3104222672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 0x7f547c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:02.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f547b7fe640 1 -- 172.21.15.148:0/3104222672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 msgr2=0x7f547c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:02.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f547b7fe640 1 --2- 172.21.15.148:0/3104222672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 0x7f547c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f547b7fe640 1 -- 172.21.15.148:0/3104222672 >> v2:172.21.15.150:3300/0 conn(0x7f547c1538a0 msgr2=0x7f547c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:02.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f547b7fe640 1 --2- 172.21.15.148:0/3104222672 >> v2:172.21.15.150:3300/0 conn(0x7f547c1538a0 0x7f547c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.733+0000 7f547b7fe640 1 -- 172.21.15.148:0/3104222672 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c0cad70 con 0x7f547c0ca0e0 2023-10-30T13:17:02.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f547b7fe640 1 --2- 172.21.15.148:0/3104222672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 0x7f547c15a090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f547c0cfd10 tx=0x7f547001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6afec6c94d94991 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:02.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f547affd640 1 -- 172.21.15.148:0/3104222672 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54700a89b0 con 0x7f547c0ca0e0 2023-10-30T13:17:02.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f547affd640 1 -- 172.21.15.148:0/3104222672 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54700a8b10 con 0x7f547c0ca0e0 2023-10-30T13:17:02.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f547affd640 1 -- 172.21.15.148:0/3104222672 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54700a8ea0 con 0x7f547c0ca0e0 2023-10-30T13:17:02.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f548269a640 1 -- 172.21.15.148:0/3104222672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 msgr2=0x7f547c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:02.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f548269a640 1 --2- 172.21.15.148:0/3104222672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 0x7f547c15a090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f547c0cfd10 tx=0x7f547001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f548269a640 1 -- 172.21.15.148:0/3104222672 shutdown_connections 2023-10-30T13:17:02.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f548269a640 1 --2- 172.21.15.148:0/3104222672 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 0x7f547c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f548269a640 1 --2- 172.21.15.148:0/3104222672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 0x7f547c15a090 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f548269a640 1 --2- 172.21.15.148:0/3104222672 >> v2:172.21.15.150:3300/0 conn(0x7f547c1538a0 0x7f547c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f548269a640 1 -- 172.21.15.148:0/3104222672 >> 172.21.15.148:0/3104222672 conn(0x7f547c06bfe0 msgr2=0x7f547c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:02.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f548269a640 1 -- 172.21.15.148:0/3104222672 shutdown_connections 2023-10-30T13:17:02.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.737+0000 7f548269a640 1 -- 172.21.15.148:0/3104222672 wait complete. 2023-10-30T13:17:02.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 Processor -- start 2023-10-30T13:17:02.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 -- start start 2023-10-30T13:17:02.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 0x7f547c1e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:02.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 0x7f547c1e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:02.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f547c1538a0 0x7f547c142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:02.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c151a30 con 0x7f547c0ca0e0 2023-10-30T13:17:02.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c1f0230 con 0x7f547c1538a0 2023-10-30T13:17:02.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c1e32c0 con 0x7f547c0cf160 2023-10-30T13:17:02.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 0x7f547c1e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:02.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 0x7f547c1e2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60942/0 (socket says 172.21.15.148:60942) 2023-10-30T13:17:02.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547bfff640 1 -- 172.21.15.148:0/4020496809 learned_addr learned my addr 172.21.15.148:0/4020496809 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:02.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547b7fe640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 0x7f547c1e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:02.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f5480c37640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f547c1538a0 0x7f547c142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:02.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547bfff640 1 -- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 msgr2=0x7f547c1e3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:02.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547bfff640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 0x7f547c1e3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547bfff640 1 -- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f547c1538a0 msgr2=0x7f547c142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:02.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547bfff640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f547c1538a0 0x7f547c142bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547bfff640 1 -- 172.21.15.148:0/4020496809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54700094d0 con 0x7f547c0ca0e0 2023-10-30T13:17:02.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f547bfff640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 0x7f547c1e2a80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f546c00e8a0 tx=0x7f546c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:02.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f5478ff9640 1 -- 172.21.15.148:0/4020496809 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c009510 con 0x7f547c0ca0e0 2023-10-30T13:17:02.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f5478ff9640 1 -- 172.21.15.148:0/4020496809 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c00c940 con 0x7f547c0ca0e0 2023-10-30T13:17:02.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f5478ff9640 1 -- 172.21.15.148:0/4020496809 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00ccb0 con 0x7f547c0ca0e0 2023-10-30T13:17:02.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547c1f2de0 con 0x7f547c0ca0e0 2023-10-30T13:17:02.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.741+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547c1e3510 con 0x7f547c0ca0e0 2023-10-30T13:17:02.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.745+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f547c0b2a50 con 0x7f547c0ca0e0 2023-10-30T13:17:02.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.745+0000 7f5478ff9640 1 -- 172.21.15.148:0/4020496809 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f546c013470 con 0x7f547c0ca0e0 2023-10-30T13:17:02.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.745+0000 7f5478ff9640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54680664d0 0x7f5468068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:02.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.745+0000 7f5478ff9640 1 -- 172.21.15.148:0/4020496809 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f546c084370 con 0x7f547c0ca0e0 2023-10-30T13:17:02.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.745+0000 7f547b7fe640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54680664d0 0x7f5468068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:02.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.745+0000 7f547b7fe640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54680664d0 0x7f5468068990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f54700b2000 tx=0x7f5470009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.749+0000 7f5478ff9640 1 -- 172.21.15.148:0/4020496809 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f546c051720 con 0x7f547c0ca0e0 2023-10-30T13:17:02.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.929+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f547c04e170 con 0x7f547c0ca0e0 2023-10-30T13:17:02.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.929+0000 7f5478ff9640 1 -- 172.21.15.148:0/4020496809 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5749 (secure 0 0 0) 0x7f546c054d40 con 0x7f547c0ca0e0 2023-10-30T13:17:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:17:02.933 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:00.417912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:02.933 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 31 2023-10-30T13:17:02.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54680664d0 msgr2=0x7f5468068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:02.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54680664d0 0x7f5468068990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f54700b2000 tx=0x7f5470009040 comp rx=0 tx=0).stop 2023-10-30T13:17:02.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 msgr2=0x7f547c1e2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:02.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 0x7f547c1e2a80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f546c00e8a0 tx=0x7f546c00eca0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 shutdown_connections 2023-10-30T13:17:02.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54680664d0 0x7f5468068990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f547c1538a0 0x7f547c142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f547c0cf160 0x7f547c1e3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 --2- 172.21.15.148:0/4020496809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f547c0ca0e0 0x7f547c1e2a80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:02.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 >> 172.21.15.148:0/4020496809 conn(0x7f547c06bfe0 msgr2=0x7f547c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:02.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 shutdown_connections 2023-10-30T13:17:02.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:02.933+0000 7f548269a640 1 -- 172.21.15.148:0/4020496809 wait complete. 2023-10-30T13:17:02.953 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 31, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 31, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4285": { "addr": "172.21.15.148:6834/4162935019", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 4162935019, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 4162935019, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4959": { "addr": "172.21.15.148:6837/95380877", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 95380877, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 95380877, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4959, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 16 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:17:00.417912+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 5103, "mds_2": 4959 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:17:04.955 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:05.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.101+0000 7fa22189a640 1 Processor -- start 2023-10-30T13:17:05.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.101+0000 7fa22189a640 1 -- start start 2023-10-30T13:17:05.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa22189a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa21c06f7a0 0x7fa21c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:05.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa22189a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa21c0c6bd0 0x7fa21c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:05.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa22189a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c06a3d0 con 0x7fa21c071110 2023-10-30T13:17:05.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa22189a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21c0700c0 con 0x7fa21c06f7a0 2023-10-30T13:17:05.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa22189a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21c066cf0 con 0x7fa21c0c6bd0 2023-10-30T13:17:05.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa21affd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa21c06f7a0 0x7fa21c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:05.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa21affd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa21c06f7a0 0x7fa21c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60946/0 (socket says 172.21.15.148:60946) 2023-10-30T13:17:05.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa21affd640 1 -- 172.21.15.148:0/26323537 learned_addr learned my addr 172.21.15.148:0/26323537 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:05.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa213fff640 1 --2- 172.21.15.148:0/26323537 >> v2:172.21.15.150:3300/0 conn(0x7fa21c0c6bd0 0x7fa21c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:05.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa21affd640 1 -- 172.21.15.148:0/26323537 >> v2:172.21.15.150:3300/0 conn(0x7fa21c0c6bd0 msgr2=0x7fa21c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:05.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa21affd640 1 --2- 172.21.15.148:0/26323537 >> v2:172.21.15.150:3300/0 conn(0x7fa21c0c6bd0 0x7fa21c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa21affd640 1 -- 172.21.15.148:0/26323537 >> v1:172.21.15.150:6789/0 conn(0x7fa21c071110 legacy=0x7fa21c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:05.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.105+0000 7fa21affd640 1 -- 172.21.15.148:0/26323537 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c0c70d0 con 0x7fa21c06f7a0 2023-10-30T13:17:05.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa21affd640 1 --2- 172.21.15.148:0/26323537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c06f7a0 0x7fa21c06fb80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa21c06f1c0 tx=0x7fa20c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e833e89fb1adf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:05.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa21a7fc640 1 -- 172.21.15.148:0/26323537 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c0a89f0 con 0x7fa21c06f7a0 2023-10-30T13:17:05.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa21a7fc640 1 -- 172.21.15.148:0/26323537 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c0a8b50 con 0x7fa21c06f7a0 2023-10-30T13:17:05.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa21a7fc640 1 -- 172.21.15.148:0/26323537 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c0b0440 con 0x7fa21c06f7a0 2023-10-30T13:17:05.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 -- 172.21.15.148:0/26323537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c06f7a0 msgr2=0x7fa21c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:05.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 --2- 172.21.15.148:0/26323537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c06f7a0 0x7fa21c06fb80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa21c06f1c0 tx=0x7fa20c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 -- 172.21.15.148:0/26323537 shutdown_connections 2023-10-30T13:17:05.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 --2- 172.21.15.148:0/26323537 >> v2:172.21.15.150:3300/0 conn(0x7fa21c0c6bd0 0x7fa21c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 --2- 172.21.15.148:0/26323537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c06f7a0 0x7fa21c06fb80 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 -- 172.21.15.148:0/26323537 >> 172.21.15.148:0/26323537 conn(0x7fa21c0c7910 msgr2=0x7fa21c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:05.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 -- 172.21.15.148:0/26323537 shutdown_connections 2023-10-30T13:17:05.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 -- 172.21.15.148:0/26323537 wait complete. 2023-10-30T13:17:05.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 Processor -- start 2023-10-30T13:17:05.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 -- start start 2023-10-30T13:17:05.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa21c06f7a0 0x7fa21c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:05.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c071110 0x7fa21c0adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:05.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa21c0c6bd0 0x7fa21c0ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:05.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c0adb00 con 0x7fa21c071110 2023-10-30T13:17:05.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c1efc00 con 0x7fa21c0c6bd0 2023-10-30T13:17:05.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.109+0000 7fa22189a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c1efee0 con 0x7fa21c06f7a0 2023-10-30T13:17:05.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa213fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c071110 0x7fa21c0adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:05.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa213fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c071110 0x7fa21c0adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60952/0 (socket says 172.21.15.148:60952) 2023-10-30T13:17:05.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa213fff640 1 -- 172.21.15.148:0/483455834 learned_addr learned my addr 172.21.15.148:0/483455834 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:05.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa21affd640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa21c06f7a0 0x7fa21c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:05.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa21b7fe640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa21c0c6bd0 0x7fa21c0ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:05.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa213fff640 1 -- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa21c06f7a0 msgr2=0x7fa21c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:05.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa213fff640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa21c06f7a0 0x7fa21c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa213fff640 1 -- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa21c0c6bd0 msgr2=0x7fa21c0ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:05.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa213fff640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa21c0c6bd0 0x7fa21c0ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa213fff640 1 -- 172.21.15.148:0/483455834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c0094d0 con 0x7fa21c071110 2023-10-30T13:17:05.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa21affd640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa21c06f7a0 0x7fa21c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:05.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa213fff640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c071110 0x7fa21c0adf90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa20400d5d0 tx=0x7fa20400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:05.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa218ff9640 1 -- 172.21.15.148:0/483455834 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa204004240 con 0x7fa21c071110 2023-10-30T13:17:05.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa218ff9640 1 -- 172.21.15.148:0/483455834 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2040043a0 con 0x7fa21c071110 2023-10-30T13:17:05.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c0af960 con 0x7fa21c071110 2023-10-30T13:17:05.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa218ff9640 1 -- 172.21.15.148:0/483455834 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa204003840 con 0x7fa21c071110 2023-10-30T13:17:05.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c0c81e0 con 0x7fa21c071110 2023-10-30T13:17:05.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21c1f00b0 con 0x7fa21c071110 2023-10-30T13:17:05.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.113+0000 7fa218ff9640 1 -- 172.21.15.148:0/483455834 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2040039a0 con 0x7fa21c071110 2023-10-30T13:17:05.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.117+0000 7fa218ff9640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1fc0664a0 0x7fa1fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:05.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.117+0000 7fa218ff9640 1 -- 172.21.15.148:0/483455834 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fa204084290 con 0x7fa21c071110 2023-10-30T13:17:05.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.117+0000 7fa21affd640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1fc0664a0 0x7fa1fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:05.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.117+0000 7fa21affd640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1fc0664a0 0x7fa1fc068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0b2000 tx=0x7fa20c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:05.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.121+0000 7fa218ff9640 1 -- 172.21.15.148:0/483455834 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2040514f0 con 0x7fa21c071110 2023-10-30T13:17:05.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa21c04e170 con 0x7fa21c071110 2023-10-30T13:17:05.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa218ff9640 1 -- 172.21.15.148:0/483455834 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5749 (secure 0 0 0) 0x7fa204054b10 con 0x7fa21c071110 2023-10-30T13:17:05.302 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:17:05.302 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:00.417912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:05.303 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 31 2023-10-30T13:17:05.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1fc0664a0 msgr2=0x7fa1fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:05.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1fc0664a0 0x7fa1fc068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0b2000 tx=0x7fa20c008040 comp rx=0 tx=0).stop 2023-10-30T13:17:05.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c071110 msgr2=0x7fa21c0adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:05.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c071110 0x7fa21c0adf90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa20400d5d0 tx=0x7fa20400da90 comp rx=0 tx=0).stop 2023-10-30T13:17:05.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 shutdown_connections 2023-10-30T13:17:05.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1fc0664a0 0x7fa1fc068960 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa21c071110 0x7fa21c0adf90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa21c0c6bd0 0x7fa21c0ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 --2- 172.21.15.148:0/483455834 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa21c06f7a0 0x7fa21c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:05.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 >> 172.21.15.148:0/483455834 conn(0x7fa21c0c7910 msgr2=0x7fa21c1e5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:05.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 shutdown_connections 2023-10-30T13:17:05.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:05.301+0000 7fa22189a640 1 -- 172.21.15.148:0/483455834 wait complete. 2023-10-30T13:17:05.323 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:17:07.324 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:07.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.473+0000 7fba2413e640 1 Processor -- start 2023-10-30T13:17:07.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.473+0000 7fba2413e640 1 -- start start 2023-10-30T13:17:07.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.473+0000 7fba2413e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fba1c151800 0x7fba1c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:07.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.473+0000 7fba2413e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 0x7fba1c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:07.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.473+0000 7fba2413e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1c06e090 con 0x7fba1c0c9230 2023-10-30T13:17:07.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.473+0000 7fba2413e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c06dc70 con 0x7fba1c151800 2023-10-30T13:17:07.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.473+0000 7fba2413e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0c8a50 con 0x7fba1c0c1ee0 2023-10-30T13:17:07.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.473+0000 7fba21eda640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fba1c151800 0x7fba1c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:07.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba21eda640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fba1c151800 0x7fba1c06e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60964/0 (socket says 172.21.15.148:60964) 2023-10-30T13:17:07.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba21eda640 1 -- 172.21.15.148:0/801645135 learned_addr learned my addr 172.21.15.148:0/801645135 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:07.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba216d9640 1 --2- 172.21.15.148:0/801645135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 0x7fba1c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:07.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba21eda640 1 -- 172.21.15.148:0/801645135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 msgr2=0x7fba1c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:07.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba21eda640 1 --2- 172.21.15.148:0/801645135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 0x7fba1c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:07.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba21eda640 1 -- 172.21.15.148:0/801645135 >> v1:172.21.15.150:6789/0 conn(0x7fba1c0c9230 legacy=0x7fba1c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:07.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba21eda640 1 -- 172.21.15.148:0/801645135 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c0c3b10 con 0x7fba1c151800 2023-10-30T13:17:07.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba216d9640 1 --2- 172.21.15.148:0/801645135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 0x7fba1c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:07.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba21eda640 1 --2- 172.21.15.148:0/801645135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c151800 0x7fba1c06e470 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fba18009870 tx=0x7fba1801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85030472fe5035c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:07.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba20ed8640 1 -- 172.21.15.148:0/801645135 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180aa670 con 0x7fba1c151800 2023-10-30T13:17:07.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba20ed8640 1 -- 172.21.15.148:0/801645135 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba180aa7d0 con 0x7fba1c151800 2023-10-30T13:17:07.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.477+0000 7fba20ed8640 1 -- 172.21.15.148:0/801645135 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180aab60 con 0x7fba1c151800 2023-10-30T13:17:07.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 -- 172.21.15.148:0/801645135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c151800 msgr2=0x7fba1c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:07.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 --2- 172.21.15.148:0/801645135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c151800 0x7fba1c06e470 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fba18009870 tx=0x7fba1801c510 comp rx=0 tx=0).stop 2023-10-30T13:17:07.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 -- 172.21.15.148:0/801645135 shutdown_connections 2023-10-30T13:17:07.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 --2- 172.21.15.148:0/801645135 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 0x7fba1c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:07.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 --2- 172.21.15.148:0/801645135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c151800 0x7fba1c06e470 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:07.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 -- 172.21.15.148:0/801645135 >> 172.21.15.148:0/801645135 conn(0x7fba1c071ea0 msgr2=0x7fba1c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:07.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 -- 172.21.15.148:0/801645135 shutdown_connections 2023-10-30T13:17:07.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 -- 172.21.15.148:0/801645135 wait complete. 2023-10-30T13:17:07.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 Processor -- start 2023-10-30T13:17:07.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 -- start start 2023-10-30T13:17:07.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 0x7fba1c0c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:07.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c0c9230 0x7fba1c13b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:07.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba1c151800 0x7fba1c13be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:07.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0c5a10 con 0x7fba1c0c9230 2023-10-30T13:17:07.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0c4b80 con 0x7fba1c151800 2023-10-30T13:17:07.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba2413e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c14b480 con 0x7fba1c0c1ee0 2023-10-30T13:17:07.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba216d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c0c9230 0x7fba1c13b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:07.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba216d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c0c9230 0x7fba1c13b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60976/0 (socket says 172.21.15.148:60976) 2023-10-30T13:17:07.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba216d9640 1 -- 172.21.15.148:0/1492257881 learned_addr learned my addr 172.21.15.148:0/1492257881 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:07.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba21eda640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 0x7fba1c0c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:07.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba226db640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba1c151800 0x7fba1c13be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:07.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba216d9640 1 -- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 msgr2=0x7fba1c0c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:07.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba216d9640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 0x7fba1c0c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:07.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba216d9640 1 -- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba1c151800 msgr2=0x7fba1c13be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:07.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba216d9640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba1c151800 0x7fba1c13be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:07.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba216d9640 1 -- 172.21.15.148:0/1492257881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba180094d0 con 0x7fba1c0c9230 2023-10-30T13:17:07.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.481+0000 7fba226db640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba1c151800 0x7fba1c13be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:07.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba216d9640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c0c9230 0x7fba1c13b920 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00d7d0 tx=0x7fba0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:07.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba127fc640 1 -- 172.21.15.148:0/1492257881 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0096a0 con 0x7fba1c0c9230 2023-10-30T13:17:07.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba127fc640 1 -- 172.21.15.148:0/1492257881 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c009800 con 0x7fba1c0c9230 2023-10-30T13:17:07.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1c0c4e00 con 0x7fba1c0c9230 2023-10-30T13:17:07.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba127fc640 1 -- 172.21.15.148:0/1492257881 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c009bb0 con 0x7fba1c0c9230 2023-10-30T13:17:07.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c0c53c0 con 0x7fba1c0c9230 2023-10-30T13:17:07.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1c04e170 con 0x7fba1c0c9230 2023-10-30T13:17:07.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba127fc640 1 -- 172.21.15.148:0/1492257881 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fba0c009d10 con 0x7fba1c0c9230 2023-10-30T13:17:07.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba127fc640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fba000664a0 0x7fba00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:07.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba127fc640 1 -- 172.21.15.148:0/1492257881 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fba0c0847a0 con 0x7fba1c0c9230 2023-10-30T13:17:07.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.485+0000 7fba21eda640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fba000664a0 0x7fba00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:07.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.489+0000 7fba21eda640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fba000664a0 0x7fba00068960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fba180ae000 tx=0x7fba180b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:07.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.489+0000 7fba127fc640 1 -- 172.21.15.148:0/1492257881 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fba0c051980 con 0x7fba1c0c9230 2023-10-30T13:17:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba1c0c6de0 con 0x7fba1c0c9230 2023-10-30T13:17:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba127fc640 1 -- 172.21.15.148:0/1492257881 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fba0c054fa0 con 0x7fba1c0c9230 2023-10-30T13:17:07.670 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:17:07.671 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:05.726711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 32 2023-10-30T13:17:07.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fba000664a0 msgr2=0x7fba00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:07.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fba000664a0 0x7fba00068960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fba180ae000 tx=0x7fba180b5040 comp rx=0 tx=0).stop 2023-10-30T13:17:07.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c0c9230 msgr2=0x7fba1c13b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c0c9230 0x7fba1c13b920 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00d7d0 tx=0x7fba0c00dc90 comp rx=0 tx=0).stop 2023-10-30T13:17:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 shutdown_connections 2023-10-30T13:17:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fba000664a0 0x7fba00068960 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1c0c9230 0x7fba1c13b920 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba1c151800 0x7fba1c13be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:07.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 --2- 172.21.15.148:0/1492257881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fba1c0c1ee0 0x7fba1c0c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:07.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 >> 172.21.15.148:0/1492257881 conn(0x7fba1c071ea0 msgr2=0x7fba1c14f230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:07.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 shutdown_connections 2023-10-30T13:17:07.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:07.669+0000 7fba2413e640 1 -- 172.21.15.148:0/1492257881 wait complete. 2023-10-30T13:17:07.691 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:17:09.692 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:09.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbcb38d640 1 Processor -- start 2023-10-30T13:17:09.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbcb38d640 1 -- start start 2023-10-30T13:17:09.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbcb38d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbbc41538a0 0x7fbbc415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:09.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbcb38d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40cf160 0x7fbbc40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:09.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbcb38d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc40cb800 con 0x7fbbc40ca0e0 2023-10-30T13:17:09.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbcb38d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc415dc30 con 0x7fbbc41538a0 2023-10-30T13:17:09.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbcb38d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc40cb330 con 0x7fbbc40cf160 2023-10-30T13:17:09.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbc992a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40cf160 0x7fbbc40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:09.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbc992a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40cf160 0x7fbbc40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:35224/0 (socket says 172.21.15.148:35224) 2023-10-30T13:17:09.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbc992a640 1 -- 172.21.15.148:0/3510333354 learned_addr learned my addr 172.21.15.148:0/3510333354 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:09.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbc9129640 1 --2- 172.21.15.148:0/3510333354 >> v2:172.21.15.150:3300/0 conn(0x7fbbc41538a0 0x7fbbc415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:09.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbc992a640 1 -- 172.21.15.148:0/3510333354 >> v2:172.21.15.150:3300/0 conn(0x7fbbc41538a0 msgr2=0x7fbbc415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:09.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbc992a640 1 --2- 172.21.15.148:0/3510333354 >> v2:172.21.15.150:3300/0 conn(0x7fbbc41538a0 0x7fbbc415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:09.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbc992a640 1 -- 172.21.15.148:0/3510333354 >> v1:172.21.15.150:6789/0 conn(0x7fbbc40ca0e0 legacy=0x7fbbc415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:09.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.841+0000 7fbbc992a640 1 -- 172.21.15.148:0/3510333354 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc40cad70 con 0x7fbbc40cf160 2023-10-30T13:17:09.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.845+0000 7fbbc992a640 1 --2- 172.21.15.148:0/3510333354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40cf160 0x7fbbc40cf590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009e50 tx=0x7fbbbc01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7accba9a42acc65e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:09.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.845+0000 7fbbbbfff640 1 -- 172.21.15.148:0/3510333354 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0af400 con 0x7fbbc40cf160 2023-10-30T13:17:09.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.845+0000 7fbbbbfff640 1 -- 172.21.15.148:0/3510333354 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc0ab050 con 0x7fbbc40cf160 2023-10-30T13:17:09.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.845+0000 7fbbbbfff640 1 -- 172.21.15.148:0/3510333354 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0af780 con 0x7fbbc40cf160 2023-10-30T13:17:09.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 -- 172.21.15.148:0/3510333354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40cf160 msgr2=0x7fbbc40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:09.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 --2- 172.21.15.148:0/3510333354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40cf160 0x7fbbc40cf590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009e50 tx=0x7fbbbc01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:17:09.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 -- 172.21.15.148:0/3510333354 shutdown_connections 2023-10-30T13:17:09.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 --2- 172.21.15.148:0/3510333354 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40cf160 0x7fbbc40cf590 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:09.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 --2- 172.21.15.148:0/3510333354 >> v2:172.21.15.150:3300/0 conn(0x7fbbc41538a0 0x7fbbc415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:09.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 -- 172.21.15.148:0/3510333354 >> 172.21.15.148:0/3510333354 conn(0x7fbbc406bfe0 msgr2=0x7fbbc406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:09.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 -- 172.21.15.148:0/3510333354 shutdown_connections 2023-10-30T13:17:09.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 -- 172.21.15.148:0/3510333354 wait complete. 2023-10-30T13:17:09.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 Processor -- start 2023-10-30T13:17:09.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 -- start start 2023-10-30T13:17:09.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40ca0e0 0x7fbbc41f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:09.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbc40cf160 0x7fbbc4146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:09.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbc41538a0 0x7fbbc41ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:09.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc415d260 con 0x7fbbc40cf160 2023-10-30T13:17:09.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc41436f0 con 0x7fbbc41538a0 2023-10-30T13:17:09.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbcb38d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc40cee90 con 0x7fbbc40ca0e0 2023-10-30T13:17:09.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbc8928640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbc40cf160 0x7fbbc4146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:09.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbc8928640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbc40cf160 0x7fbbc4146160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60980/0 (socket says 172.21.15.148:60980) 2023-10-30T13:17:09.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbc8928640 1 -- 172.21.15.148:0/1442903665 learned_addr learned my addr 172.21.15.148:0/1442903665 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbc9129640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40ca0e0 0x7fbbc41f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.849+0000 7fbbc992a640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbc41538a0 0x7fbbc41ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbc8928640 1 -- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40ca0e0 msgr2=0x7fbbc41f2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbc8928640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40ca0e0 0x7fbbc41f2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbc8928640 1 -- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbc41538a0 msgr2=0x7fbbc41ef000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:09.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbc8928640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbc41538a0 0x7fbbc41ef000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:09.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbc8928640 1 -- 172.21.15.148:0/1442903665 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc009b30 con 0x7fbbc40cf160 2023-10-30T13:17:09.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbc8928640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbc40cf160 0x7fbbc4146160 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c420 tx=0x7fbbb400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:09.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbb9ffb640 1 -- 172.21.15.148:0/1442903665 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40052d0 con 0x7fbbc40cf160 2023-10-30T13:17:09.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbb9ffb640 1 -- 172.21.15.148:0/1442903665 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb4005430 con 0x7fbbc40cf160 2023-10-30T13:17:09.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbb9ffb640 1 -- 172.21.15.148:0/1442903665 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4007260 con 0x7fbbc40cf160 2023-10-30T13:17:09.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc41466a0 con 0x7fbbc40cf160 2023-10-30T13:17:09.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc41590c0 con 0x7fbbc40cf160 2023-10-30T13:17:09.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc40b2a50 con 0x7fbbc40cf160 2023-10-30T13:17:09.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbb9ffb640 1 -- 172.21.15.148:0/1442903665 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbbb40073c0 con 0x7fbbc40cf160 2023-10-30T13:17:09.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbb9ffb640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbba80664a0 0x7fbba8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:09.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.853+0000 7fbbb9ffb640 1 -- 172.21.15.148:0/1442903665 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fbbb40833a0 con 0x7fbbc40cf160 2023-10-30T13:17:09.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.857+0000 7fbbc9129640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbba80664a0 0x7fbba8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:09.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.857+0000 7fbbc9129640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbba80664a0 0x7fbba8068960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009ac0 tx=0x7fbbc0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:09.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:09.857+0000 7fbbb9ffb640 1 -- 172.21.15.148:0/1442903665 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbbb4050740 con 0x7fbbc40cf160 2023-10-30T13:17:10.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbc4159850 con 0x7fbbc40cf160 2023-10-30T13:17:10.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbb9ffb640 1 -- 172.21.15.148:0/1442903665 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fbbb4053d60 con 0x7fbbc40cf160 2023-10-30T13:17:10.041 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:17:10.042 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:05.726711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:10.042 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 32 2023-10-30T13:17:10.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbba80664a0 msgr2=0x7fbba8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbba80664a0 0x7fbba8068960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009ac0 tx=0x7fbbc0009190 comp rx=0 tx=0).stop 2023-10-30T13:17:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbc40cf160 msgr2=0x7fbbc4146160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbc40cf160 0x7fbbc4146160 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c420 tx=0x7fbbb400c8e0 comp rx=0 tx=0).stop 2023-10-30T13:17:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 shutdown_connections 2023-10-30T13:17:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbba80664a0 0x7fbba8068960 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:10.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbc40cf160 0x7fbbc4146160 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:10.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbc41538a0 0x7fbbc41ef000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:10.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 --2- 172.21.15.148:0/1442903665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbc40ca0e0 0x7fbbc41f2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:10.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 >> 172.21.15.148:0/1442903665 conn(0x7fbbc406bfe0 msgr2=0x7fbbc40c5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:10.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 shutdown_connections 2023-10-30T13:17:10.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:10.041+0000 7fbbcb38d640 1 -- 172.21.15.148:0/1442903665 wait complete. 2023-10-30T13:17:10.062 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:17:12.063 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:12.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.209+0000 7fdc3c0ea640 1 Processor -- start 2023-10-30T13:17:12.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.209+0000 7fdc3c0ea640 1 -- start start 2023-10-30T13:17:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.209+0000 7fdc3c0ea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdc341538a0 0x7fdc3415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.209+0000 7fdc3c0ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc340ca0e0 0x7fdc3415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.209+0000 7fdc3c0ea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdc340cf160 0x7fdc340cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3c0ea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc340cb800 con 0x7fdc340cf160 2023-10-30T13:17:12.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3c0ea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc3415dc30 con 0x7fdc341538a0 2023-10-30T13:17:12.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3c0ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340cb330 con 0x7fdc340ca0e0 2023-10-30T13:17:12.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3a687640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdc340cf160 0x7fdc340cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:12.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3a687640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdc340cf160 0x7fdc340cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55644/0 (socket says 172.21.15.148:55644) 2023-10-30T13:17:12.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3a687640 1 -- 172.21.15.148:0/160974905 learned_addr learned my addr 172.21.15.148:0/160974905 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:12.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc39e86640 1 --2- 172.21.15.148:0/160974905 >> v2:172.21.15.150:3300/0 conn(0x7fdc341538a0 0x7fdc3415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:12.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc39685640 1 --2- 172.21.15.148:0/160974905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc340ca0e0 0x7fdc3415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:12.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3a687640 1 -- 172.21.15.148:0/160974905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc340ca0e0 msgr2=0x7fdc3415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:12.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3a687640 1 --2- 172.21.15.148:0/160974905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc340ca0e0 0x7fdc3415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3a687640 1 -- 172.21.15.148:0/160974905 >> v2:172.21.15.150:3300/0 conn(0x7fdc341538a0 msgr2=0x7fdc3415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:12.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3a687640 1 --2- 172.21.15.148:0/160974905 >> v2:172.21.15.150:3300/0 conn(0x7fdc341538a0 0x7fdc3415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3a687640 1 -- 172.21.15.148:0/160974905 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc340cad70 con 0x7fdc340cf160 2023-10-30T13:17:12.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc39685640 1 --2- 172.21.15.148:0/160974905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc340ca0e0 0x7fdc3415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:12.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc39e86640 1 --2- 172.21.15.148:0/160974905 >> v2:172.21.15.150:3300/0 conn(0x7fdc341538a0 0x7fdc3415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:12.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc3a687640 1 --2- 172.21.15.148:0/160974905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340cf160 0x7fdc340cf590 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdc340cfd10 tx=0x7fdc2c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d45474cc9abd70 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:12.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc38e84640 1 -- 172.21.15.148:0/160974905 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c0af400 con 0x7fdc340cf160 2023-10-30T13:17:12.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc38e84640 1 -- 172.21.15.148:0/160974905 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2c0ab050 con 0x7fdc340cf160 2023-10-30T13:17:12.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.213+0000 7fdc38e84640 1 -- 172.21.15.148:0/160974905 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c0af780 con 0x7fdc340cf160 2023-10-30T13:17:12.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/160974905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340cf160 msgr2=0x7fdc340cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:12.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/160974905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340cf160 0x7fdc340cf590 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdc340cfd10 tx=0x7fdc2c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/160974905 shutdown_connections 2023-10-30T13:17:12.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/160974905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340cf160 0x7fdc340cf590 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/160974905 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc340ca0e0 0x7fdc3415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/160974905 >> v2:172.21.15.150:3300/0 conn(0x7fdc341538a0 0x7fdc3415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/160974905 >> 172.21.15.148:0/160974905 conn(0x7fdc3406bfe0 msgr2=0x7fdc3406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:12.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/160974905 shutdown_connections 2023-10-30T13:17:12.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/160974905 wait complete. 2023-10-30T13:17:12.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 Processor -- start 2023-10-30T13:17:12.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 -- start start 2023-10-30T13:17:12.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340ca0e0 0x7fdc341e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:12.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc340cf160 0x7fdc340cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:12.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc341538a0 0x7fdc340b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:12.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341e31e0 con 0x7fdc340ca0e0 2023-10-30T13:17:12.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341e34b0 con 0x7fdc340cf160 2023-10-30T13:17:12.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3c0ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341ef390 con 0x7fdc341538a0 2023-10-30T13:17:12.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc39e86640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340ca0e0 0x7fdc341e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:12.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc39e86640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340ca0e0 0x7fdc341e2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55650/0 (socket says 172.21.15.148:55650) 2023-10-30T13:17:12.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc39e86640 1 -- 172.21.15.148:0/862830251 learned_addr learned my addr 172.21.15.148:0/862830251 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:12.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc39685640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc340cf160 0x7fdc340cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:12.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.217+0000 7fdc3a687640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc341538a0 0x7fdc340b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:12.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc39e86640 1 -- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc341538a0 msgr2=0x7fdc340b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:12.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc39e86640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc341538a0 0x7fdc340b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc39e86640 1 -- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc340cf160 msgr2=0x7fdc340cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:12.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc39e86640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc340cf160 0x7fdc340cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc39e86640 1 -- 172.21.15.148:0/862830251 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c009b30 con 0x7fdc340ca0e0 2023-10-30T13:17:12.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc39e86640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340ca0e0 0x7fdc341e2ca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000d800 tx=0x7fdc3000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:12.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc2a7fc640 1 -- 172.21.15.148:0/862830251 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc300096c0 con 0x7fdc340ca0e0 2023-10-30T13:17:12.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc2a7fc640 1 -- 172.21.15.148:0/862830251 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc30009820 con 0x7fdc340ca0e0 2023-10-30T13:17:12.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc2a7fc640 1 -- 172.21.15.148:0/862830251 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30009bd0 con 0x7fdc340ca0e0 2023-10-30T13:17:12.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc340b3240 con 0x7fdc340ca0e0 2023-10-30T13:17:12.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc340b3800 con 0x7fdc340ca0e0 2023-10-30T13:17:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc340b2a50 con 0x7fdc340ca0e0 2023-10-30T13:17:12.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc2a7fc640 1 -- 172.21.15.148:0/862830251 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdc30009d30 con 0x7fdc340ca0e0 2023-10-30T13:17:12.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc2a7fc640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc180664a0 0x7fdc18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:12.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc2a7fc640 1 -- 172.21.15.148:0/862830251 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7fdc300845d0 con 0x7fdc340ca0e0 2023-10-30T13:17:12.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.221+0000 7fdc39685640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc180664a0 0x7fdc18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:12.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.225+0000 7fdc39685640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc180664a0 0x7fdc18068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdc340b4e70 tx=0x7fdc24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:12.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.225+0000 7fdc2a7fc640 1 -- 172.21.15.148:0/862830251 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdc30051980 con 0x7fdc340ca0e0 2023-10-30T13:17:12.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.405+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc3404e170 con 0x7fdc340ca0e0 2023-10-30T13:17:12.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.405+0000 7fdc2a7fc640 1 -- 172.21.15.148:0/862830251 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fdc30054fa0 con 0x7fdc340ca0e0 2023-10-30T13:17:12.408 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:17:12.408 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:11.446467+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:12.409 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 33 2023-10-30T13:17:12.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc180664a0 msgr2=0x7fdc18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc180664a0 0x7fdc18068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdc340b4e70 tx=0x7fdc24008040 comp rx=0 tx=0).stop 2023-10-30T13:17:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340ca0e0 msgr2=0x7fdc341e2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340ca0e0 0x7fdc341e2ca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000d800 tx=0x7fdc3000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 shutdown_connections 2023-10-30T13:17:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc180664a0 0x7fdc18068960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc341538a0 0x7fdc340b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc340cf160 0x7fdc340cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 --2- 172.21.15.148:0/862830251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc340ca0e0 0x7fdc341e2ca0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:12.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 >> 172.21.15.148:0/862830251 conn(0x7fdc3406bfe0 msgr2=0x7fdc34151880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:12.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 shutdown_connections 2023-10-30T13:17:12.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:12.409+0000 7fdc3c0ea640 1 -- 172.21.15.148:0/862830251 wait complete. 2023-10-30T13:17:12.429 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=4270 2023-10-30T13:17:12.429 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.d 2023-10-30T13:17:15.011 INFO:tasks.ceph.mds.e.smithi148.stderr:2023-10-30T13:17:15.009+0000 7ff756f04640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:17:17.465 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/30 file after the snapshot 2023-10-30T13:17:17.466 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/31 file after the snapshot 2023-10-30T13:17:17.468 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/32 file after the snapshot 2023-10-30T13:17:17.468 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/33 file after the snapshot 2023-10-30T13:17:17.469 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/34 file after the snapshot 2023-10-30T13:17:17.470 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/35 file after the snapshot 2023-10-30T13:17:17.471 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/36 file after the snapshot 2023-10-30T13:17:17.471 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/37 file after the snapshot 2023-10-30T13:17:17.473 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/38 file after the snapshot 2023-10-30T13:17:17.473 INFO:tasks.workunit.client.0.smithi148.stdout: create 279/39 file after the snapshot 2023-10-30T13:17:17.474 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/.snap/snap-subdir-test 2023-10-30T13:17:17.489 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/30 file after the snapshot 2023-10-30T13:17:17.490 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/31 file after the snapshot 2023-10-30T13:17:17.491 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/32 file after the snapshot 2023-10-30T13:17:17.492 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/33 file after the snapshot 2023-10-30T13:17:17.492 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/34 file after the snapshot 2023-10-30T13:17:17.493 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/35 file after the snapshot 2023-10-30T13:17:17.494 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/36 file after the snapshot 2023-10-30T13:17:17.495 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/37 file after the snapshot 2023-10-30T13:17:17.496 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/38 file after the snapshot 2023-10-30T13:17:17.496 INFO:tasks.workunit.client.0.smithi148.stdout: create 280/39 file after the snapshot 2023-10-30T13:17:17.497 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/.snap/snap-subdir-test 2023-10-30T13:17:17.514 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/30 file after the snapshot 2023-10-30T13:17:17.515 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/31 file after the snapshot 2023-10-30T13:17:17.516 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/32 file after the snapshot 2023-10-30T13:17:17.516 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/33 file after the snapshot 2023-10-30T13:17:17.517 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/34 file after the snapshot 2023-10-30T13:17:17.518 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/35 file after the snapshot 2023-10-30T13:17:17.518 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/36 file after the snapshot 2023-10-30T13:17:17.519 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/37 file after the snapshot 2023-10-30T13:17:17.520 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/38 file after the snapshot 2023-10-30T13:17:17.521 INFO:tasks.workunit.client.0.smithi148.stdout: create 281/39 file after the snapshot 2023-10-30T13:17:17.521 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/.snap/snap-subdir-test 2023-10-30T13:17:17.541 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/30 file after the snapshot 2023-10-30T13:17:17.543 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/31 file after the snapshot 2023-10-30T13:17:17.544 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/32 file after the snapshot 2023-10-30T13:17:17.545 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/33 file after the snapshot 2023-10-30T13:17:17.546 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/34 file after the snapshot 2023-10-30T13:17:17.547 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/35 file after the snapshot 2023-10-30T13:17:17.548 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/36 file after the snapshot 2023-10-30T13:17:17.549 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/37 file after the snapshot 2023-10-30T13:17:17.550 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/38 file after the snapshot 2023-10-30T13:17:17.551 INFO:tasks.workunit.client.0.smithi148.stdout: create 282/39 file after the snapshot 2023-10-30T13:17:17.552 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/.snap/snap-subdir-test 2023-10-30T13:17:17.570 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/30 file after the snapshot 2023-10-30T13:17:17.572 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/31 file after the snapshot 2023-10-30T13:17:17.573 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/32 file after the snapshot 2023-10-30T13:17:17.573 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/33 file after the snapshot 2023-10-30T13:17:17.574 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/34 file after the snapshot 2023-10-30T13:17:17.574 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/35 file after the snapshot 2023-10-30T13:17:17.575 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/36 file after the snapshot 2023-10-30T13:17:17.576 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/37 file after the snapshot 2023-10-30T13:17:17.576 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/38 file after the snapshot 2023-10-30T13:17:17.577 INFO:tasks.workunit.client.0.smithi148.stdout: create 283/39 file after the snapshot 2023-10-30T13:17:17.578 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/.snap/snap-subdir-test 2023-10-30T13:17:17.595 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/30 file after the snapshot 2023-10-30T13:17:17.597 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/31 file after the snapshot 2023-10-30T13:17:17.598 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/32 file after the snapshot 2023-10-30T13:17:17.599 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/33 file after the snapshot 2023-10-30T13:17:17.600 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/34 file after the snapshot 2023-10-30T13:17:17.601 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/35 file after the snapshot 2023-10-30T13:17:17.602 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/36 file after the snapshot 2023-10-30T13:17:17.603 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/37 file after the snapshot 2023-10-30T13:17:17.604 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/38 file after the snapshot 2023-10-30T13:17:17.605 INFO:tasks.workunit.client.0.smithi148.stdout: create 284/39 file after the snapshot 2023-10-30T13:17:17.606 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/.snap/snap-subdir-test 2023-10-30T13:17:17.622 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/30 file after the snapshot 2023-10-30T13:17:17.623 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/31 file after the snapshot 2023-10-30T13:17:17.623 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/32 file after the snapshot 2023-10-30T13:17:17.624 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/33 file after the snapshot 2023-10-30T13:17:17.625 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/34 file after the snapshot 2023-10-30T13:17:17.626 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/35 file after the snapshot 2023-10-30T13:17:17.626 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/36 file after the snapshot 2023-10-30T13:17:17.627 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/37 file after the snapshot 2023-10-30T13:17:17.628 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/38 file after the snapshot 2023-10-30T13:17:17.629 INFO:tasks.workunit.client.0.smithi148.stdout: create 285/39 file after the snapshot 2023-10-30T13:17:17.629 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/.snap/snap-subdir-test 2023-10-30T13:17:17.651 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/30 file after the snapshot 2023-10-30T13:17:17.653 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/31 file after the snapshot 2023-10-30T13:17:17.654 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/32 file after the snapshot 2023-10-30T13:17:17.655 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/33 file after the snapshot 2023-10-30T13:17:17.656 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/34 file after the snapshot 2023-10-30T13:17:17.656 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/35 file after the snapshot 2023-10-30T13:17:17.657 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/36 file after the snapshot 2023-10-30T13:17:17.658 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/37 file after the snapshot 2023-10-30T13:17:17.659 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/38 file after the snapshot 2023-10-30T13:17:17.660 INFO:tasks.workunit.client.0.smithi148.stdout: create 286/39 file after the snapshot 2023-10-30T13:17:17.661 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/.snap/snap-subdir-test 2023-10-30T13:17:17.680 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/30 file after the snapshot 2023-10-30T13:17:17.681 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/31 file after the snapshot 2023-10-30T13:17:17.682 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/32 file after the snapshot 2023-10-30T13:17:17.683 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/33 file after the snapshot 2023-10-30T13:17:17.684 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/34 file after the snapshot 2023-10-30T13:17:17.684 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/35 file after the snapshot 2023-10-30T13:17:17.685 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/36 file after the snapshot 2023-10-30T13:17:17.686 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/37 file after the snapshot 2023-10-30T13:17:17.687 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/38 file after the snapshot 2023-10-30T13:17:17.687 INFO:tasks.workunit.client.0.smithi148.stdout: create 287/39 file after the snapshot 2023-10-30T13:17:17.688 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/.snap/snap-subdir-test 2023-10-30T13:17:17.704 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/30 file after the snapshot 2023-10-30T13:17:17.704 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/31 file after the snapshot 2023-10-30T13:17:17.705 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/32 file after the snapshot 2023-10-30T13:17:17.706 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/33 file after the snapshot 2023-10-30T13:17:17.706 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/34 file after the snapshot 2023-10-30T13:17:17.707 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/35 file after the snapshot 2023-10-30T13:17:17.708 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/36 file after the snapshot 2023-10-30T13:17:17.709 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/37 file after the snapshot 2023-10-30T13:17:17.709 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/38 file after the snapshot 2023-10-30T13:17:17.710 INFO:tasks.workunit.client.0.smithi148.stdout: create 288/39 file after the snapshot 2023-10-30T13:17:17.710 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/.snap/snap-subdir-test 2023-10-30T13:17:17.730 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/30 file after the snapshot 2023-10-30T13:17:17.731 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/31 file after the snapshot 2023-10-30T13:17:17.731 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/32 file after the snapshot 2023-10-30T13:17:17.732 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/33 file after the snapshot 2023-10-30T13:17:17.733 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/34 file after the snapshot 2023-10-30T13:17:17.817 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/35 file after the snapshot 2023-10-30T13:17:18.521 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/36 file after the snapshot 2023-10-30T13:17:18.521 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/37 file after the snapshot 2023-10-30T13:17:18.522 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/38 file after the snapshot 2023-10-30T13:17:18.523 INFO:tasks.workunit.client.0.smithi148.stdout: create 289/39 file after the snapshot 2023-10-30T13:17:18.524 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/.snap/snap-subdir-test 2023-10-30T13:17:18.544 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/30 file after the snapshot 2023-10-30T13:17:18.545 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/31 file after the snapshot 2023-10-30T13:17:18.546 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/32 file after the snapshot 2023-10-30T13:17:18.547 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/33 file after the snapshot 2023-10-30T13:17:18.547 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/34 file after the snapshot 2023-10-30T13:17:18.548 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/35 file after the snapshot 2023-10-30T13:17:18.549 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/36 file after the snapshot 2023-10-30T13:17:18.549 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/37 file after the snapshot 2023-10-30T13:17:18.550 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/38 file after the snapshot 2023-10-30T13:17:18.551 INFO:tasks.workunit.client.0.smithi148.stdout: create 290/39 file after the snapshot 2023-10-30T13:17:18.551 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/.snap/snap-subdir-test 2023-10-30T13:17:18.569 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/30 file after the snapshot 2023-10-30T13:17:18.570 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/31 file after the snapshot 2023-10-30T13:17:18.571 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/32 file after the snapshot 2023-10-30T13:17:18.572 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/33 file after the snapshot 2023-10-30T13:17:18.572 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/34 file after the snapshot 2023-10-30T13:17:18.573 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/35 file after the snapshot 2023-10-30T13:17:18.574 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/36 file after the snapshot 2023-10-30T13:17:18.575 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/37 file after the snapshot 2023-10-30T13:17:18.575 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/38 file after the snapshot 2023-10-30T13:17:18.576 INFO:tasks.workunit.client.0.smithi148.stdout: create 291/39 file after the snapshot 2023-10-30T13:17:18.577 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/.snap/snap-subdir-test 2023-10-30T13:17:18.596 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/30 file after the snapshot 2023-10-30T13:17:18.597 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/31 file after the snapshot 2023-10-30T13:17:18.598 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/32 file after the snapshot 2023-10-30T13:17:18.599 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/33 file after the snapshot 2023-10-30T13:17:18.600 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/34 file after the snapshot 2023-10-30T13:17:18.601 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/35 file after the snapshot 2023-10-30T13:17:18.602 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/36 file after the snapshot 2023-10-30T13:17:18.603 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/37 file after the snapshot 2023-10-30T13:17:18.604 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/38 file after the snapshot 2023-10-30T13:17:18.605 INFO:tasks.workunit.client.0.smithi148.stdout: create 292/39 file after the snapshot 2023-10-30T13:17:18.606 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/.snap/snap-subdir-test 2023-10-30T13:17:18.624 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/30 file after the snapshot 2023-10-30T13:17:18.625 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/31 file after the snapshot 2023-10-30T13:17:18.626 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/32 file after the snapshot 2023-10-30T13:17:18.627 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/33 file after the snapshot 2023-10-30T13:17:18.628 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/34 file after the snapshot 2023-10-30T13:17:18.628 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/35 file after the snapshot 2023-10-30T13:17:18.629 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/36 file after the snapshot 2023-10-30T13:17:18.630 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/37 file after the snapshot 2023-10-30T13:17:18.631 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/38 file after the snapshot 2023-10-30T13:17:18.631 INFO:tasks.workunit.client.0.smithi148.stdout: create 293/39 file after the snapshot 2023-10-30T13:17:18.632 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/.snap/snap-subdir-test 2023-10-30T13:17:18.650 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/30 file after the snapshot 2023-10-30T13:17:18.651 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/31 file after the snapshot 2023-10-30T13:17:18.651 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/32 file after the snapshot 2023-10-30T13:17:18.652 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/33 file after the snapshot 2023-10-30T13:17:18.653 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/34 file after the snapshot 2023-10-30T13:17:18.654 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/35 file after the snapshot 2023-10-30T13:17:18.654 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/36 file after the snapshot 2023-10-30T13:17:18.655 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/37 file after the snapshot 2023-10-30T13:17:18.656 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/38 file after the snapshot 2023-10-30T13:17:18.656 INFO:tasks.workunit.client.0.smithi148.stdout: create 294/39 file after the snapshot 2023-10-30T13:17:18.657 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/.snap/snap-subdir-test 2023-10-30T13:17:19.516 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/30 file after the snapshot 2023-10-30T13:17:19.518 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/31 file after the snapshot 2023-10-30T13:17:19.519 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/32 file after the snapshot 2023-10-30T13:17:19.520 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/33 file after the snapshot 2023-10-30T13:17:19.521 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/34 file after the snapshot 2023-10-30T13:17:19.522 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/35 file after the snapshot 2023-10-30T13:17:19.523 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/36 file after the snapshot 2023-10-30T13:17:19.524 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/37 file after the snapshot 2023-10-30T13:17:19.525 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/38 file after the snapshot 2023-10-30T13:17:19.525 INFO:tasks.workunit.client.0.smithi148.stdout: create 295/39 file after the snapshot 2023-10-30T13:17:19.526 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/.snap/snap-subdir-test 2023-10-30T13:17:19.546 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/30 file after the snapshot 2023-10-30T13:17:19.547 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/31 file after the snapshot 2023-10-30T13:17:19.548 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/32 file after the snapshot 2023-10-30T13:17:19.549 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/33 file after the snapshot 2023-10-30T13:17:19.550 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/34 file after the snapshot 2023-10-30T13:17:19.551 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/35 file after the snapshot 2023-10-30T13:17:19.552 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/36 file after the snapshot 2023-10-30T13:17:19.552 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/37 file after the snapshot 2023-10-30T13:17:19.553 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/38 file after the snapshot 2023-10-30T13:17:19.554 INFO:tasks.workunit.client.0.smithi148.stdout: create 296/39 file after the snapshot 2023-10-30T13:17:19.555 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/.snap/snap-subdir-test 2023-10-30T13:17:19.573 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/30 file after the snapshot 2023-10-30T13:17:19.575 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/31 file after the snapshot 2023-10-30T13:17:19.576 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/32 file after the snapshot 2023-10-30T13:17:19.576 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/33 file after the snapshot 2023-10-30T13:17:19.577 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/34 file after the snapshot 2023-10-30T13:17:19.578 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/35 file after the snapshot 2023-10-30T13:17:19.579 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/36 file after the snapshot 2023-10-30T13:17:19.579 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/37 file after the snapshot 2023-10-30T13:17:19.580 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/38 file after the snapshot 2023-10-30T13:17:19.581 INFO:tasks.workunit.client.0.smithi148.stdout: create 297/39 file after the snapshot 2023-10-30T13:17:19.582 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/.snap/snap-subdir-test 2023-10-30T13:17:19.600 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/30 file after the snapshot 2023-10-30T13:17:19.601 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/31 file after the snapshot 2023-10-30T13:17:19.602 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/32 file after the snapshot 2023-10-30T13:17:19.602 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/33 file after the snapshot 2023-10-30T13:17:19.603 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/34 file after the snapshot 2023-10-30T13:17:19.604 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/35 file after the snapshot 2023-10-30T13:17:19.604 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/36 file after the snapshot 2023-10-30T13:17:19.605 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/37 file after the snapshot 2023-10-30T13:17:19.606 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/38 file after the snapshot 2023-10-30T13:17:19.607 INFO:tasks.workunit.client.0.smithi148.stdout: create 298/39 file after the snapshot 2023-10-30T13:17:19.608 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/.snap/snap-subdir-test 2023-10-30T13:17:19.624 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/30 file after the snapshot 2023-10-30T13:17:19.625 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/31 file after the snapshot 2023-10-30T13:17:19.626 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/32 file after the snapshot 2023-10-30T13:17:19.627 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/33 file after the snapshot 2023-10-30T13:17:19.628 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/34 file after the snapshot 2023-10-30T13:17:19.628 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/35 file after the snapshot 2023-10-30T13:17:19.629 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/36 file after the snapshot 2023-10-30T13:17:19.630 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/37 file after the snapshot 2023-10-30T13:17:19.631 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/38 file after the snapshot 2023-10-30T13:17:19.631 INFO:tasks.workunit.client.0.smithi148.stdout: create 299/39 file after the snapshot 2023-10-30T13:17:19.632 INFO:tasks.workunit.client.0.smithi148.stdout:Take second snapshot .snap/test2 2023-10-30T13:17:19.654 INFO:tasks.workunit.client.0.smithi148.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-10-30T13:17:20.430 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:17:20.431 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:17:20.431 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:17:20.434 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:17:20.434 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:20.512 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2023-10-30T13:17:20.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.573+0000 7f6e60f72640 1 Processor -- start 2023-10-30T13:17:20.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.573+0000 7f6e60f72640 1 -- start start 2023-10-30T13:17:20.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e60f72640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6e5c0c6b80 0x7f6e5c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:20.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e60f72640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0c0a50 con 0x7f6e5c13f4a0 2023-10-30T13:17:20.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e60f72640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c06f6e0 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e60f72640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5c06f9c0 con 0x7f6e5c0c6b80 2023-10-30T13:17:20.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e5ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6e5c13f4a0 0x7f6e5c1566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48286/0 (socket says 172.21.15.148:48286) 2023-10-30T13:17:20.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e5ad9d640 1 -- 172.21.15.148:0/1708428206 learned_addr learned my addr 172.21.15.148:0/1708428206 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:17:20.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e5a59c640 1 --2- 172.21.15.148:0/1708428206 >> v2:172.21.15.150:3300/0 conn(0x7f6e5c0c6b80 0x7f6e5c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:20.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e59d9b640 1 -- 172.21.15.148:0/1708428206 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623739960 0 0) 0x7f6e5c0c0a50 con 0x7f6e5c13f4a0 2023-10-30T13:17:20.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e59d9b640 1 -- 172.21.15.148:0/1708428206 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48003530 con 0x7f6e5c13f4a0 2023-10-30T13:17:20.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e59d9b640 1 -- 172.21.15.148:0/1708428206 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143223768 0 0) 0x7f6e5c06f6e0 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e59d9b640 1 -- 172.21.15.148:0/1708428206 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e5c0c0a50 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e59d9b640 1 -- 172.21.15.148:0/1708428206 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3696071873 0 0) 0x7f6e48003530 con 0x7f6e5c13f4a0 2023-10-30T13:17:20.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e5a59c640 1 -- 172.21.15.148:0/1708428206 >> v1:172.21.15.150:6789/0 conn(0x7f6e5c0c6f60 legacy=0x7f6e5c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:20.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e5a59c640 1 -- 172.21.15.148:0/1708428206 >> v1:172.21.15.148:6789/0 conn(0x7f6e5c13f4a0 legacy=0x7f6e5c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:20.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.577+0000 7f6e5a59c640 1 -- 172.21.15.148:0/1708428206 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c06fca0 con 0x7f6e5c0c6b80 2023-10-30T13:17:20.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e5a59c640 1 --2- 172.21.15.148:0/1708428206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e5c0c6b80 0x7f6e5c0bfec0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0097c0 tx=0x7f6e4c01c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74137f8e07ea485b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:20.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e59d9b640 1 -- 172.21.15.148:0/1708428206 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54001a00 con 0x7f6e5c0c6b80 2023-10-30T13:17:20.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e59d9b640 1 -- 172.21.15.148:0/1708428206 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4c0a7050 con 0x7f6e5c0c6b80 2023-10-30T13:17:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e59d9b640 1 -- 172.21.15.148:0/1708428206 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c0ab620 con 0x7f6e5c0c6b80 2023-10-30T13:17:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e60f72640 1 -- 172.21.15.148:0/1708428206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e5c0c6b80 msgr2=0x7f6e5c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e60f72640 1 --2- 172.21.15.148:0/1708428206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e5c0c6b80 0x7f6e5c0bfec0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0097c0 tx=0x7f6e4c01c3a0 comp rx=0 tx=0).stop 2023-10-30T13:17:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e60f72640 1 -- 172.21.15.148:0/1708428206 shutdown_connections 2023-10-30T13:17:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e60f72640 1 --2- 172.21.15.148:0/1708428206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e5c0c6b80 0x7f6e5c0bfec0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:20.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e60f72640 1 -- 172.21.15.148:0/1708428206 >> 172.21.15.148:0/1708428206 conn(0x7f6e5c0c7910 msgr2=0x7f6e5c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e60f72640 1 -- 172.21.15.148:0/1708428206 shutdown_connections 2023-10-30T13:17:20.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e60f72640 1 -- 172.21.15.148:0/1708428206 wait complete. 2023-10-30T13:17:20.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e60f72640 1 Processor -- start 2023-10-30T13:17:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.581+0000 7f6e60f72640 1 -- start start 2023-10-30T13:17:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e60f72640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e5c0c6b80 0x7f6e5c0c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e60f72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e5c0c6f60 0x7f6e5c0c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e60f72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e5c13f4a0 0x7f6e5c0c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e60f72640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c138020 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e60f72640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0c3100 con 0x7f6e5c13f4a0 2023-10-30T13:17:20.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e60f72640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c135080 con 0x7f6e5c0c6b80 2023-10-30T13:17:20.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e53fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e5c0c6f60 0x7f6e5c0c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:20.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e53fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e5c0c6f60 0x7f6e5c0c38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55660/0 (socket says 172.21.15.148:55660) 2023-10-30T13:17:20.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e53fff640 1 -- 172.21.15.148:0/518112009 learned_addr learned my addr 172.21.15.148:0/518112009 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e5ad9d640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e5c13f4a0 0x7f6e5c0c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e5a59c640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e5c0c6b80 0x7f6e5c0c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e53fff640 1 -- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e5c0c6b80 msgr2=0x7f6e5c0c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e53fff640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e5c0c6b80 0x7f6e5c0c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e53fff640 1 -- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e5c13f4a0 msgr2=0x7f6e5c0c3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:20.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e53fff640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e5c13f4a0 0x7f6e5c0c3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:20.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e53fff640 1 -- 172.21.15.148:0/518112009 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c009420 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e53fff640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e5c0c6f60 0x7f6e5c0c38a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400c490 tx=0x7f6e4400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:20.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e52ffd640 1 -- 172.21.15.148:0/518112009 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e440052d0 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e52ffd640 1 -- 172.21.15.148:0/518112009 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e44005430 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e52ffd640 1 -- 172.21.15.148:0/518112009 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e44012870 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c1371c0 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c137780 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.585+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c0c0220 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.589+0000 7f6e52ffd640 1 -- 172.21.15.148:0/518112009 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e440129d0 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.589+0000 7f6e52ffd640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e3c0664a0 0x7f6e3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:20.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.589+0000 7f6e52ffd640 1 -- 172.21.15.148:0/518112009 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f6e44083ae0 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.589+0000 7f6e5a59c640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e3c0664a0 0x7f6e3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:20.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.589+0000 7f6e5a59c640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e3c0664a0 0x7f6e3c068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0ad000 tx=0x7f6e4c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:20.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.593+0000 7f6e52ffd640 1 -- 172.21.15.148:0/518112009 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e44050df0 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e5c04e170 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e52ffd640 1 -- 172.21.15.148:0/518112009 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f6e44054410 con 0x7f6e5c0c6f60 2023-10-30T13:17:20.794 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:17:20.794 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:11.446467+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:20.794 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 33 2023-10-30T13:17:20.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e3c0664a0 msgr2=0x7f6e3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:20.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e3c0664a0 0x7f6e3c068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0ad000 tx=0x7f6e4c0a6000 comp rx=0 tx=0).stop 2023-10-30T13:17:20.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e5c0c6f60 msgr2=0x7f6e5c0c38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:20.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e5c0c6f60 0x7f6e5c0c38a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400c490 tx=0x7f6e4400c950 comp rx=0 tx=0).stop 2023-10-30T13:17:20.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 shutdown_connections 2023-10-30T13:17:20.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e3c0664a0 0x7f6e3c068960 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:20.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6e5c0c6f60 0x7f6e5c0c38a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:20.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e5c13f4a0 0x7f6e5c0c3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:20.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 --2- 172.21.15.148:0/518112009 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6e5c0c6b80 0x7f6e5c0c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:20.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 >> 172.21.15.148:0/518112009 conn(0x7f6e5c0c7910 msgr2=0x7f6e5c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:20.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 shutdown_connections 2023-10-30T13:17:20.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:20.793+0000 7f6e60f72640 1 -- 172.21.15.148:0/518112009 wait complete. 2023-10-30T13:17:20.815 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:17:21.320 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-10-30T13:17:21.321 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-10-30T13:17:21.321 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-10-30T13:17:21.321 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-10-30T13:17:21.321 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-10-30T13:17:21.321 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-10-30T13:17:21.322 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-10-30T13:17:21.322 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-10-30T13:17:21.322 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-10-30T13:17:21.322 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-10-30T13:17:21.322 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-10-30T13:17:21.322 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-10-30T13:17:21.323 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-10-30T13:17:21.323 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-10-30T13:17:21.323 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-10-30T13:17:21.323 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-10-30T13:17:21.323 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-10-30T13:17:21.323 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-10-30T13:17:21.324 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-10-30T13:17:21.324 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-10-30T13:17:21.324 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-10-30T13:17:21.324 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-10-30T13:17:21.324 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-10-30T13:17:21.324 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-10-30T13:17:21.325 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-10-30T13:17:21.325 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-10-30T13:17:21.325 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-10-30T13:17:21.325 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-10-30T13:17:21.325 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-10-30T13:17:21.325 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-10-30T13:17:21.326 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-10-30T13:17:21.326 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-10-30T13:17:21.326 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-10-30T13:17:21.326 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-10-30T13:17:21.326 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-10-30T13:17:21.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-10-30T13:17:21.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-10-30T13:17:21.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-10-30T13:17:21.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-10-30T13:17:21.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-10-30T13:17:21.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-10-30T13:17:21.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-10-30T13:17:21.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-10-30T13:17:21.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-10-30T13:17:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-10-30T13:17:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-10-30T13:17:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-10-30T13:17:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-10-30T13:17:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-10-30T13:17:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-10-30T13:17:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-10-30T13:17:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-10-30T13:17:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-10-30T13:17:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-10-30T13:17:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-10-30T13:17:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-10-30T13:17:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-10-30T13:17:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-10-30T13:17:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-10-30T13:17:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-10-30T13:17:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-10-30T13:17:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-10-30T13:17:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-10-30T13:17:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-10-30T13:17:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-10-30T13:17:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-10-30T13:17:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-10-30T13:17:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-10-30T13:17:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-10-30T13:17:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-10-30T13:17:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-10-30T13:17:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-10-30T13:17:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-10-30T13:17:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-10-30T13:17:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-10-30T13:17:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-10-30T13:17:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-10-30T13:17:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-10-30T13:17:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-10-30T13:17:21.336 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-10-30T13:17:21.336 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-10-30T13:17:22.389 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-10-30T13:17:22.389 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-10-30T13:17:22.390 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-10-30T13:17:22.390 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-10-30T13:17:22.390 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-10-30T13:17:22.390 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-10-30T13:17:22.390 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-10-30T13:17:22.391 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-10-30T13:17:22.391 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-10-30T13:17:22.391 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-10-30T13:17:22.391 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-10-30T13:17:22.391 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-10-30T13:17:22.391 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-10-30T13:17:22.392 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-10-30T13:17:22.392 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-10-30T13:17:22.392 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-10-30T13:17:22.392 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-10-30T13:17:22.392 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-10-30T13:17:22.393 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-10-30T13:17:22.393 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-10-30T13:17:22.393 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-10-30T13:17:22.393 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-10-30T13:17:22.393 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-10-30T13:17:22.393 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-10-30T13:17:22.394 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-10-30T13:17:22.394 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-10-30T13:17:22.394 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-10-30T13:17:22.394 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-10-30T13:17:22.394 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-10-30T13:17:22.395 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-10-30T13:17:22.395 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-10-30T13:17:22.395 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-10-30T13:17:22.395 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-10-30T13:17:22.395 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-10-30T13:17:22.396 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-10-30T13:17:22.396 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-10-30T13:17:22.396 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-10-30T13:17:22.396 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-10-30T13:17:22.396 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-10-30T13:17:22.396 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-10-30T13:17:22.397 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-10-30T13:17:22.397 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-10-30T13:17:22.397 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-10-30T13:17:22.397 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-10-30T13:17:22.397 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-10-30T13:17:22.398 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-10-30T13:17:22.398 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-10-30T13:17:22.398 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-10-30T13:17:22.398 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-10-30T13:17:22.398 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-10-30T13:17:22.399 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-10-30T13:17:22.399 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-10-30T13:17:22.399 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-10-30T13:17:22.399 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-10-30T13:17:22.399 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-10-30T13:17:22.399 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-10-30T13:17:22.400 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-10-30T13:17:22.400 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-10-30T13:17:22.400 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-10-30T13:17:22.400 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-10-30T13:17:22.400 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-10-30T13:17:22.400 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-10-30T13:17:22.401 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-10-30T13:17:22.401 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-10-30T13:17:22.401 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-10-30T13:17:22.401 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-10-30T13:17:22.401 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-10-30T13:17:22.401 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-10-30T13:17:22.402 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-10-30T13:17:22.402 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-10-30T13:17:22.402 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-10-30T13:17:22.402 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-10-30T13:17:22.402 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-10-30T13:17:22.402 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-10-30T13:17:22.403 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-10-30T13:17:22.403 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-10-30T13:17:22.403 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-10-30T13:17:22.403 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-10-30T13:17:22.403 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-10-30T13:17:22.403 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-10-30T13:17:22.404 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-10-30T13:17:22.816 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:22.921 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-10-30T13:17:22.921 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-10-30T13:17:22.921 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-10-30T13:17:22.921 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-10-30T13:17:22.922 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-10-30T13:17:22.922 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-10-30T13:17:22.922 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-10-30T13:17:22.922 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-10-30T13:17:22.922 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-10-30T13:17:22.923 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-10-30T13:17:22.923 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-10-30T13:17:22.923 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-10-30T13:17:22.923 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-10-30T13:17:22.923 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-10-30T13:17:22.923 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-10-30T13:17:22.924 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-10-30T13:17:22.924 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-10-30T13:17:22.924 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-10-30T13:17:22.924 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-10-30T13:17:22.924 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-10-30T13:17:22.924 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-10-30T13:17:22.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-10-30T13:17:22.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-10-30T13:17:22.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-10-30T13:17:22.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-10-30T13:17:22.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-10-30T13:17:22.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-10-30T13:17:22.926 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-10-30T13:17:22.926 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-10-30T13:17:22.926 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-10-30T13:17:22.926 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-10-30T13:17:22.926 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-10-30T13:17:22.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-10-30T13:17:22.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-10-30T13:17:22.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-10-30T13:17:22.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-10-30T13:17:22.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-10-30T13:17:22.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-10-30T13:17:22.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-10-30T13:17:22.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-10-30T13:17:22.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-10-30T13:17:22.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-10-30T13:17:22.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-10-30T13:17:22.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-10-30T13:17:22.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-10-30T13:17:22.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-10-30T13:17:22.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-10-30T13:17:22.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-10-30T13:17:22.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-10-30T13:17:22.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-10-30T13:17:22.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-10-30T13:17:22.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-10-30T13:17:22.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-10-30T13:17:22.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-10-30T13:17:22.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-10-30T13:17:22.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-10-30T13:17:22.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-10-30T13:17:22.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-10-30T13:17:22.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-10-30T13:17:22.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-10-30T13:17:22.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-10-30T13:17:22.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-10-30T13:17:22.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-10-30T13:17:22.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-10-30T13:17:22.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-10-30T13:17:22.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-10-30T13:17:22.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-10-30T13:17:22.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-10-30T13:17:22.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-10-30T13:17:22.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-10-30T13:17:22.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-10-30T13:17:22.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-10-30T13:17:22.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-10-30T13:17:22.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-10-30T13:17:22.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-10-30T13:17:22.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-10-30T13:17:22.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-10-30T13:17:22.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-10-30T13:17:22.935 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-10-30T13:17:22.935 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-10-30T13:17:22.935 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-10-30T13:17:22.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2d89f640 1 Processor -- start 2023-10-30T13:17:22.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2d89f640 1 -- start start 2023-10-30T13:17:22.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2d89f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c281538f0 0x7f6c28159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:22.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2d89f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6c2815cd50 0x7f6c280c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:22.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2d89f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c280cefa0 0x7f6c280cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:22.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2d89f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c2815d300 con 0x7f6c2815cd50 2023-10-30T13:17:22.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2d89f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c2815d5e0 con 0x7f6c280cefa0 2023-10-30T13:17:22.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2d89f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2815d8c0 con 0x7f6c281538f0 2023-10-30T13:17:22.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c27fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6c2815cd50 0x7f6c280c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:22.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c27fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6c2815cd50 0x7f6c280c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43328/0 (socket says 172.21.15.148:43328) 2023-10-30T13:17:22.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c27fff640 1 -- 172.21.15.148:0/4180174522 learned_addr learned my addr 172.21.15.148:0/4180174522 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:22.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2d09e640 1 --2- 172.21.15.148:0/4180174522 >> v2:172.21.15.150:3300/0 conn(0x7f6c280cefa0 0x7f6c280cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:22.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2c89d640 1 --2- 172.21.15.148:0/4180174522 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c281538f0 0x7f6c28159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:22.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c27fff640 1 -- 172.21.15.148:0/4180174522 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c281538f0 msgr2=0x7f6c28159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:22.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c27fff640 1 --2- 172.21.15.148:0/4180174522 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c281538f0 0x7f6c28159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:22.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c27fff640 1 -- 172.21.15.148:0/4180174522 >> v2:172.21.15.150:3300/0 conn(0x7f6c280cefa0 msgr2=0x7f6c280cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:22.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c27fff640 1 --2- 172.21.15.148:0/4180174522 >> v2:172.21.15.150:3300/0 conn(0x7f6c280cefa0 0x7f6c280cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:22.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c27fff640 1 -- 172.21.15.148:0/4180174522 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2815dba0 con 0x7f6c2815cd50 2023-10-30T13:17:22.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.977+0000 7f6c2c89d640 1 --2- 172.21.15.148:0/4180174522 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c281538f0 0x7f6c28159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:22.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c27fff640 1 --2- 172.21.15.148:0/4180174522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c2815cd50 0x7f6c280c9f90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6c280b1ae0 tx=0x7f6c1401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc13319b700dcc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:22.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c277fe640 1 -- 172.21.15.148:0/4180174522 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c140a89b0 con 0x7f6c2815cd50 2023-10-30T13:17:22.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c277fe640 1 -- 172.21.15.148:0/4180174522 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c140a8b10 con 0x7f6c2815cd50 2023-10-30T13:17:22.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c277fe640 1 -- 172.21.15.148:0/4180174522 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c140a8ea0 con 0x7f6c2815cd50 2023-10-30T13:17:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c2d89f640 1 -- 172.21.15.148:0/4180174522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c2815cd50 msgr2=0x7f6c280c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/4180174522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c2815cd50 0x7f6c280c9f90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6c280b1ae0 tx=0x7f6c1401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:17:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c2d89f640 1 -- 172.21.15.148:0/4180174522 shutdown_connections 2023-10-30T13:17:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/4180174522 >> v2:172.21.15.150:3300/0 conn(0x7f6c280cefa0 0x7f6c280cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/4180174522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c2815cd50 0x7f6c280c9f90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/4180174522 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c281538f0 0x7f6c28159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c2d89f640 1 -- 172.21.15.148:0/4180174522 >> 172.21.15.148:0/4180174522 conn(0x7f6c2806a640 msgr2=0x7f6c28065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c2d89f640 1 -- 172.21.15.148:0/4180174522 shutdown_connections 2023-10-30T13:17:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.981+0000 7f6c2d89f640 1 -- 172.21.15.148:0/4180174522 wait complete. 2023-10-30T13:17:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 Processor -- start 2023-10-30T13:17:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 -- start start 2023-10-30T13:17:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c280cefa0 0x7f6c28147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c281538f0 0x7f6c28157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c2815cd50 0x7f6c28157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2815c3a0 con 0x7f6c280cefa0 2023-10-30T13:17:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2815c6f0 con 0x7f6c281538f0 2023-10-30T13:17:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2815b4a0 con 0x7f6c2815cd50 2023-10-30T13:17:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2c89d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c280cefa0 0x7f6c28147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2c89d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c280cefa0 0x7f6c28147ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43336/0 (socket says 172.21.15.148:43336) 2023-10-30T13:17:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2c89d640 1 -- 172.21.15.148:0/2562375725 learned_addr learned my addr 172.21.15.148:0/2562375725 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:17:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c27fff640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c281538f0 0x7f6c28157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d09e640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c2815cd50 0x7f6c28157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2c89d640 1 -- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c2815cd50 msgr2=0x7f6c28157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2c89d640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c2815cd50 0x7f6c28157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2c89d640 1 -- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c281538f0 msgr2=0x7f6c28157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2c89d640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c281538f0 0x7f6c28157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2c89d640 1 -- 172.21.15.148:0/2562375725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c140094d0 con 0x7f6c280cefa0 2023-10-30T13:17:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2c89d640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c280cefa0 0x7f6c28147ad0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00e810 tx=0x7f6c1c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c257fa640 1 -- 172.21.15.148:0/2562375725 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c00c8d0 con 0x7f6c280cefa0 2023-10-30T13:17:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c257fa640 1 -- 172.21.15.148:0/2562375725 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1c00ca30 con 0x7f6c280cefa0 2023-10-30T13:17:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c257fa640 1 -- 172.21.15.148:0/2562375725 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c00cdc0 con 0x7f6c280cefa0 2023-10-30T13:17:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2815a920 con 0x7f6c280cefa0 2023-10-30T13:17:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.985+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2815aee0 con 0x7f6c280cefa0 2023-10-30T13:17:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.989+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2804e170 con 0x7f6c280cefa0 2023-10-30T13:17:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.989+0000 7f6c257fa640 1 -- 172.21.15.148:0/2562375725 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c1c013410 con 0x7f6c280cefa0 2023-10-30T13:17:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.989+0000 7f6c257fa640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c0c0664a0 0x7f6c0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.989+0000 7f6c257fa640 1 -- 172.21.15.148:0/2562375725 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6112+0+0 (secure 0 0 0) 0x7f6c1c083e00 con 0x7f6c280cefa0 2023-10-30T13:17:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.989+0000 7f6c27fff640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c0c0664a0 0x7f6c0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.989+0000 7f6c27fff640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c0c0664a0 0x7f6c0c068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6c140b2000 tx=0x7f6c140aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:22.993+0000 7f6c257fa640 1 -- 172.21.15.148:0/2562375725 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c1c051100 con 0x7f6c280cefa0 2023-10-30T13:17:23.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.185+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c280c5d40 con 0x7f6c280cefa0 2023-10-30T13:17:23.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.185+0000 7f6c257fa640 1 -- 172.21.15.148:0/2562375725 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f6c1c054720 con 0x7f6c280cefa0 2023-10-30T13:17:23.187 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:17:23.187 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:21.531094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:23.187 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 35 2023-10-30T13:17:23.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c0c0664a0 msgr2=0x7f6c0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:23.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c0c0664a0 0x7f6c0c068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6c140b2000 tx=0x7f6c140aa070 comp rx=0 tx=0).stop 2023-10-30T13:17:23.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c280cefa0 msgr2=0x7f6c28147ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:23.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c280cefa0 0x7f6c28147ad0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00e810 tx=0x7f6c1c00ecd0 comp rx=0 tx=0).stop 2023-10-30T13:17:23.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 shutdown_connections 2023-10-30T13:17:23.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c0c0664a0 0x7f6c0c068960 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:23.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c2815cd50 0x7f6c28157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:23.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c281538f0 0x7f6c28157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:23.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 --2- 172.21.15.148:0/2562375725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c280cefa0 0x7f6c28147ad0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:23.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 >> 172.21.15.148:0/2562375725 conn(0x7f6c2806a640 msgr2=0x7f6c280c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:23.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 shutdown_connections 2023-10-30T13:17:23.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:17:23.189+0000 7f6c2d89f640 1 -- 172.21.15.148:0/2562375725 wait complete. 2023-10-30T13:17:23.207 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:17:23.208 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 59 secs before thrashing 2023-10-30T13:17:23.860 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-10-30T13:17:23.860 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-10-30T13:17:23.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-10-30T13:17:23.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-10-30T13:17:23.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-10-30T13:17:23.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-10-30T13:17:23.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-10-30T13:17:23.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-10-30T13:17:23.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-10-30T13:17:23.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-10-30T13:17:23.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-10-30T13:17:23.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-10-30T13:17:23.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-10-30T13:17:23.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-10-30T13:17:23.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-10-30T13:17:23.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-10-30T13:17:23.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-10-30T13:17:23.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-10-30T13:17:23.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-10-30T13:17:23.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-10-30T13:17:23.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-10-30T13:17:23.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-10-30T13:17:23.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-10-30T13:17:23.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-10-30T13:17:23.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-10-30T13:17:23.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-10-30T13:17:23.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-10-30T13:17:23.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-10-30T13:17:23.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-10-30T13:17:23.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-10-30T13:17:23.866 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-10-30T13:17:23.866 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-10-30T13:17:23.866 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-10-30T13:17:23.866 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-10-30T13:17:23.866 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-10-30T13:17:23.867 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-10-30T13:17:23.867 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-10-30T13:17:23.867 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-10-30T13:17:23.867 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-10-30T13:17:23.867 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-10-30T13:17:23.868 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-10-30T13:17:23.868 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-10-30T13:17:23.868 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-10-30T13:17:23.868 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-10-30T13:17:23.868 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-10-30T13:17:23.868 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-10-30T13:17:23.869 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-10-30T13:17:23.869 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-10-30T13:17:23.869 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-10-30T13:17:23.869 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-10-30T13:17:23.869 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-10-30T13:17:23.869 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-10-30T13:17:23.870 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-10-30T13:17:23.870 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-10-30T13:17:23.870 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-10-30T13:17:23.870 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-10-30T13:17:23.870 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-10-30T13:17:23.870 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-10-30T13:17:23.871 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-10-30T13:17:23.871 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-10-30T13:17:23.871 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-10-30T13:17:23.871 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-10-30T13:17:23.871 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-10-30T13:17:23.871 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-10-30T13:17:23.872 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-10-30T13:17:23.872 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-10-30T13:17:23.872 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-10-30T13:17:23.872 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-10-30T13:17:23.872 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-10-30T13:17:23.872 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-10-30T13:17:23.873 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-10-30T13:17:23.873 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-10-30T13:17:23.873 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-10-30T13:17:23.873 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-10-30T13:17:23.873 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-10-30T13:17:23.873 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-10-30T13:17:23.874 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-10-30T13:17:23.874 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-10-30T13:17:23.874 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-10-30T13:17:23.874 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-10-30T13:17:23.874 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-10-30T13:17:24.342 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:17:24.347 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:17:24.728 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-10-30T13:17:24.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-10-30T13:17:24.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-10-30T13:17:24.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-10-30T13:17:24.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-10-30T13:17:24.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-10-30T13:17:24.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-10-30T13:17:24.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-10-30T13:17:24.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-10-30T13:17:24.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-10-30T13:17:24.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-10-30T13:17:24.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-10-30T13:17:24.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-10-30T13:17:24.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-10-30T13:17:24.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-10-30T13:17:24.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-10-30T13:17:24.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-10-30T13:17:24.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-10-30T13:17:24.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-10-30T13:17:24.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-10-30T13:17:24.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-10-30T13:17:24.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-10-30T13:17:24.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-10-30T13:17:24.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-10-30T13:17:24.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-10-30T13:17:24.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-10-30T13:17:24.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-10-30T13:17:24.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-10-30T13:17:24.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-10-30T13:17:24.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-10-30T13:17:24.734 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-10-30T13:17:24.734 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-10-30T13:17:24.734 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-10-30T13:17:24.734 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-10-30T13:17:24.734 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-10-30T13:17:24.734 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-10-30T13:17:24.735 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-10-30T13:17:24.735 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-10-30T13:17:24.735 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-10-30T13:17:24.735 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-10-30T13:17:24.735 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-10-30T13:17:24.735 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-10-30T13:17:24.736 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-10-30T13:17:24.736 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-10-30T13:17:24.736 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-10-30T13:17:24.736 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-10-30T13:17:24.736 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-10-30T13:17:24.736 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-10-30T13:17:24.737 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-10-30T13:17:24.737 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-10-30T13:17:24.737 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-10-30T13:17:24.737 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-10-30T13:17:24.737 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-10-30T13:17:24.737 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-10-30T13:17:24.738 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-10-30T13:17:24.738 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-10-30T13:17:24.738 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-10-30T13:17:24.738 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-10-30T13:17:24.738 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-10-30T13:17:24.739 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-10-30T13:17:24.739 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-10-30T13:17:24.739 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-10-30T13:17:24.739 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-10-30T13:17:24.739 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-10-30T13:17:24.739 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-10-30T13:17:24.740 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-10-30T13:17:24.740 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-10-30T13:17:24.740 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-10-30T13:17:24.740 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-10-30T13:17:24.740 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-10-30T13:17:24.741 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-10-30T13:17:24.741 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-10-30T13:17:24.741 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-10-30T13:17:24.741 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-10-30T13:17:24.741 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-10-30T13:17:24.741 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-10-30T13:17:24.742 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-10-30T13:17:24.742 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-10-30T13:17:24.742 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-10-30T13:17:24.742 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-10-30T13:17:24.742 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-10-30T13:17:25.924 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-10-30T13:17:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-10-30T13:17:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-10-30T13:17:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-10-30T13:17:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-10-30T13:17:25.926 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-10-30T13:17:25.926 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-10-30T13:17:25.926 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-10-30T13:17:25.926 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-10-30T13:17:25.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-10-30T13:17:25.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-10-30T13:17:25.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-10-30T13:17:25.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-10-30T13:17:25.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-10-30T13:17:25.927 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-10-30T13:17:25.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-10-30T13:17:25.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-10-30T13:17:25.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-10-30T13:17:25.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-10-30T13:17:25.928 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-10-30T13:17:25.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-10-30T13:17:25.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-10-30T13:17:25.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-10-30T13:17:25.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-10-30T13:17:25.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-10-30T13:17:25.929 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-10-30T13:17:25.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-10-30T13:17:25.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-10-30T13:17:25.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-10-30T13:17:25.930 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-10-30T13:17:25.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-10-30T13:17:25.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-10-30T13:17:25.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-10-30T13:17:25.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-10-30T13:17:25.931 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-10-30T13:17:25.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-10-30T13:17:25.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-10-30T13:17:25.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-10-30T13:17:25.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-10-30T13:17:25.932 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-10-30T13:17:25.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-10-30T13:17:25.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-10-30T13:17:25.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-10-30T13:17:25.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-10-30T13:17:25.933 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-10-30T13:17:25.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-10-30T13:17:25.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-10-30T13:17:25.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-10-30T13:17:25.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-10-30T13:17:25.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-10-30T13:17:25.934 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-10-30T13:17:25.935 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-10-30T13:17:25.935 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-10-30T13:17:25.935 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-10-30T13:17:25.935 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-10-30T13:17:25.935 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-10-30T13:17:25.936 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-10-30T13:17:25.936 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-10-30T13:17:25.936 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-10-30T13:17:25.936 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-10-30T13:17:25.936 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-10-30T13:17:25.937 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-10-30T13:17:25.937 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-10-30T13:17:25.937 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-10-30T13:17:25.937 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-10-30T13:17:25.937 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-10-30T13:17:25.937 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-10-30T13:17:25.938 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-10-30T13:17:25.938 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-10-30T13:17:25.938 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-10-30T13:17:25.938 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-10-30T13:17:25.938 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-10-30T13:17:25.939 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-10-30T13:17:25.939 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-10-30T13:17:25.939 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-10-30T13:17:25.939 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-10-30T13:17:25.939 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-10-30T13:17:25.939 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-10-30T13:17:25.940 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-10-30T13:17:25.940 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-10-30T13:17:25.940 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-10-30T13:17:26.325 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-10-30T13:17:26.325 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-10-30T13:17:26.326 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-10-30T13:17:26.326 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-10-30T13:17:26.326 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-10-30T13:17:26.326 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-10-30T13:17:26.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-10-30T13:17:26.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-10-30T13:17:26.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-10-30T13:17:26.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-10-30T13:17:26.327 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-10-30T13:17:26.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-10-30T13:17:26.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-10-30T13:17:26.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-10-30T13:17:26.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-10-30T13:17:26.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-10-30T13:17:26.328 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-10-30T13:17:26.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-10-30T13:17:26.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-10-30T13:17:26.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-10-30T13:17:26.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-10-30T13:17:26.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-10-30T13:17:26.329 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-10-30T13:17:26.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-10-30T13:17:26.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-10-30T13:17:26.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-10-30T13:17:26.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-10-30T13:17:26.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-10-30T13:17:26.330 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-10-30T13:17:26.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-10-30T13:17:26.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-10-30T13:17:26.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-10-30T13:17:26.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-10-30T13:17:26.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-10-30T13:17:26.331 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-10-30T13:17:26.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-10-30T13:17:26.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-10-30T13:17:26.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-10-30T13:17:26.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-10-30T13:17:26.332 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-10-30T13:17:26.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-10-30T13:17:26.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-10-30T13:17:26.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-10-30T13:17:26.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-10-30T13:17:26.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-10-30T13:17:26.333 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-10-30T13:17:26.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-10-30T13:17:26.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-10-30T13:17:26.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-10-30T13:17:26.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-10-30T13:17:26.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-10-30T13:17:26.334 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-10-30T13:17:26.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-10-30T13:17:26.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-10-30T13:17:26.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-10-30T13:17:26.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-10-30T13:17:26.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-10-30T13:17:26.335 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-10-30T13:17:26.336 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-10-30T13:17:26.336 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-10-30T13:17:26.336 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-10-30T13:17:26.336 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-10-30T13:17:26.336 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-10-30T13:17:26.337 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-10-30T13:17:26.337 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-10-30T13:17:26.337 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-10-30T13:17:26.337 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-10-30T13:17:26.337 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-10-30T13:17:26.337 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-10-30T13:17:26.338 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-10-30T13:17:26.338 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-10-30T13:17:26.338 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-10-30T13:17:26.338 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-10-30T13:17:26.338 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-10-30T13:17:26.339 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-10-30T13:17:26.339 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-10-30T13:17:26.339 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-10-30T13:17:26.339 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-10-30T13:17:26.339 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-10-30T13:17:26.339 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-10-30T13:17:26.340 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-10-30T13:17:26.340 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-10-30T13:17:26.718 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-10-30T13:17:26.718 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-10-30T13:17:26.718 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-10-30T13:17:26.719 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-10-30T13:17:26.719 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-10-30T13:17:26.719 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-10-30T13:17:26.719 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-10-30T13:17:26.719 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-10-30T13:17:26.720 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-10-30T13:17:26.720 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-10-30T13:17:26.720 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-10-30T13:17:26.720 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-10-30T13:17:26.720 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-10-30T13:17:26.721 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-10-30T13:17:26.721 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-10-30T13:17:26.721 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-10-30T13:17:26.721 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-10-30T13:17:26.721 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-10-30T13:17:26.721 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-10-30T13:17:26.722 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-10-30T13:17:26.722 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-10-30T13:17:26.722 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-10-30T13:17:26.722 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-10-30T13:17:26.722 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-10-30T13:17:26.723 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-10-30T13:17:26.723 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-10-30T13:17:26.723 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-10-30T13:17:26.723 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-10-30T13:17:26.723 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-10-30T13:17:26.724 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-10-30T13:17:26.724 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-10-30T13:17:26.724 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-10-30T13:17:26.724 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-10-30T13:17:26.724 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-10-30T13:17:26.725 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-10-30T13:17:26.725 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-10-30T13:17:26.725 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-10-30T13:17:26.725 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-10-30T13:17:26.725 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-10-30T13:17:26.725 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-10-30T13:17:26.726 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-10-30T13:17:26.726 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-10-30T13:17:26.726 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-10-30T13:17:26.727 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-10-30T13:17:26.727 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-10-30T13:17:26.727 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-10-30T13:17:26.727 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-10-30T13:17:26.728 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-10-30T13:17:26.728 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-10-30T13:17:26.728 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-10-30T13:17:26.728 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-10-30T13:17:26.728 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-10-30T13:17:26.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-10-30T13:17:26.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-10-30T13:17:26.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-10-30T13:17:26.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-10-30T13:17:26.729 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-10-30T13:17:26.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-10-30T13:17:26.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-10-30T13:17:26.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-10-30T13:17:26.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-10-30T13:17:26.730 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-10-30T13:17:26.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-10-30T13:17:26.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-10-30T13:17:26.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-10-30T13:17:26.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-10-30T13:17:26.731 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-10-30T13:17:26.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-10-30T13:17:26.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-10-30T13:17:26.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-10-30T13:17:26.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-10-30T13:17:26.732 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-10-30T13:17:26.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-10-30T13:17:26.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-10-30T13:17:26.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-10-30T13:17:26.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-10-30T13:17:26.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-10-30T13:17:26.733 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-10-30T13:17:26.734 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-10-30T13:17:26.734 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-10-30T13:17:26.734 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-10-30T13:17:26.851 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-10-30T13:17:26.851 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-10-30T13:17:26.852 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-10-30T13:17:26.852 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-10-30T13:17:26.852 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-10-30T13:17:26.852 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-10-30T13:17:26.852 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-10-30T13:17:26.853 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-10-30T13:17:26.853 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-10-30T13:17:26.853 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-10-30T13:17:26.853 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-10-30T13:17:26.853 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-10-30T13:17:26.853 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-10-30T13:17:26.854 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-10-30T13:17:26.854 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-10-30T13:17:26.854 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-10-30T13:17:26.854 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-10-30T13:17:26.854 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-10-30T13:17:26.854 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-10-30T13:17:26.855 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-10-30T13:17:26.855 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-10-30T13:17:26.855 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-10-30T13:17:26.855 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-10-30T13:17:26.855 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-10-30T13:17:26.855 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-10-30T13:17:26.856 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-10-30T13:17:26.856 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-10-30T13:17:26.856 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-10-30T13:17:26.856 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-10-30T13:17:26.856 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-10-30T13:17:26.857 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-10-30T13:17:26.857 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-10-30T13:17:26.857 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-10-30T13:17:26.857 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-10-30T13:17:26.857 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-10-30T13:17:26.858 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-10-30T13:17:26.858 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-10-30T13:17:26.858 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-10-30T13:17:26.858 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-10-30T13:17:26.858 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-10-30T13:17:26.859 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-10-30T13:17:26.859 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-10-30T13:17:26.859 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-10-30T13:17:26.859 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-10-30T13:17:26.859 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-10-30T13:17:26.860 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-10-30T13:17:26.860 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-10-30T13:17:26.860 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-10-30T13:17:26.860 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-10-30T13:17:26.860 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-10-30T13:17:26.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-10-30T13:17:26.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-10-30T13:17:26.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-10-30T13:17:26.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-10-30T13:17:26.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-10-30T13:17:26.861 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-10-30T13:17:26.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-10-30T13:17:26.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-10-30T13:17:26.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-10-30T13:17:26.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-10-30T13:17:26.862 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-10-30T13:17:26.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-10-30T13:17:26.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-10-30T13:17:26.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-10-30T13:17:26.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-10-30T13:17:26.863 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-10-30T13:17:26.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-10-30T13:17:26.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-10-30T13:17:26.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-10-30T13:17:26.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-10-30T13:17:26.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-10-30T13:17:26.864 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-10-30T13:17:26.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-10-30T13:17:26.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-10-30T13:17:26.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-10-30T13:17:26.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-10-30T13:17:26.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-10-30T13:17:26.865 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-10-30T13:17:26.866 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-10-30T13:17:26.866 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-10-30T13:17:26.866 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-10-30T13:17:27.514 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-10-30T13:17:27.514 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-10-30T13:17:27.514 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-10-30T13:17:27.515 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-10-30T13:17:27.515 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-10-30T13:17:27.515 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-10-30T13:17:27.515 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-10-30T13:17:27.515 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-10-30T13:17:27.516 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-10-30T13:17:27.516 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-10-30T13:17:27.516 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-10-30T13:17:27.516 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-10-30T13:17:27.516 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-10-30T13:17:27.517 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-10-30T13:17:27.517 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-10-30T13:17:27.517 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-10-30T13:17:27.517 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-10-30T13:17:27.517 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-10-30T13:17:27.517 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-10-30T13:17:27.518 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-10-30T13:17:27.518 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-10-30T13:17:27.518 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-10-30T13:17:27.518 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-10-30T13:17:27.518 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-10-30T13:17:27.519 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-10-30T13:17:27.519 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-10-30T13:17:27.519 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-10-30T13:17:27.519 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-10-30T13:17:27.519 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-10-30T13:17:27.520 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-10-30T13:17:27.520 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-10-30T13:17:27.520 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-10-30T13:17:27.520 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-10-30T13:17:27.520 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-10-30T13:17:27.520 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-10-30T13:17:27.521 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-10-30T13:17:27.521 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-10-30T13:17:27.521 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-10-30T13:17:27.521 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-10-30T13:17:27.521 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-10-30T13:17:27.521 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-10-30T13:17:27.522 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-10-30T13:17:27.522 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-10-30T13:17:27.522 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-10-30T13:17:27.522 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-10-30T13:17:27.522 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-10-30T13:17:27.523 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-10-30T13:17:27.523 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-10-30T13:17:27.523 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-10-30T13:17:27.523 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-10-30T13:17:27.523 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-10-30T13:17:27.524 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-10-30T13:17:27.524 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-10-30T13:17:27.524 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-10-30T13:17:27.524 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-10-30T13:17:27.524 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-10-30T13:17:27.524 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-10-30T13:17:27.525 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-10-30T13:17:27.525 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-10-30T13:17:27.525 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-10-30T13:17:27.525 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-10-30T13:17:27.525 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-10-30T13:17:27.525 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-10-30T13:17:27.526 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-10-30T13:17:27.526 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-10-30T13:17:27.526 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-10-30T13:17:27.526 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-10-30T13:17:27.527 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-10-30T13:17:27.527 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-10-30T13:17:27.527 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-10-30T13:17:27.527 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-10-30T13:17:27.527 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-10-30T13:17:27.527 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-10-30T13:17:27.528 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-10-30T13:17:27.528 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-10-30T13:17:27.528 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-10-30T13:17:27.528 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-10-30T13:17:27.528 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-10-30T13:17:27.528 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-10-30T13:17:27.529 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-10-30T13:17:27.529 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-10-30T13:17:31.000 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-10-30T13:17:31.001 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-10-30T13:17:31.001 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-10-30T13:17:31.001 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-10-30T13:17:31.001 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-10-30T13:17:31.002 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-10-30T13:17:31.002 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-10-30T13:17:31.002 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-10-30T13:17:31.002 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-10-30T13:17:31.002 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-10-30T13:17:31.002 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-10-30T13:17:31.003 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-10-30T13:17:31.003 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-10-30T13:17:31.003 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-10-30T13:17:31.003 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-10-30T13:17:31.003 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-10-30T13:17:31.004 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-10-30T13:17:31.004 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-10-30T13:17:31.004 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-10-30T13:17:31.004 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-10-30T13:17:31.004 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-10-30T13:17:31.005 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-10-30T13:17:31.005 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-10-30T13:17:31.005 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-10-30T13:17:31.005 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-10-30T13:17:31.005 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-10-30T13:17:31.006 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-10-30T13:17:31.006 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-10-30T13:17:31.006 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-10-30T13:17:31.006 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-10-30T13:17:31.007 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-10-30T13:17:31.007 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-10-30T13:17:31.007 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-10-30T13:17:31.007 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-10-30T13:17:31.007 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-10-30T13:17:31.008 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-10-30T13:17:31.008 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-10-30T13:17:31.008 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-10-30T13:17:31.008 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-10-30T13:17:31.008 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-10-30T13:17:31.009 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-10-30T13:17:31.009 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-10-30T13:17:31.009 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-10-30T13:17:31.009 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-10-30T13:17:31.009 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-10-30T13:17:31.010 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-10-30T13:17:31.010 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-10-30T13:17:31.010 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-10-30T13:17:31.010 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-10-30T13:17:31.010 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-10-30T13:17:31.011 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-10-30T13:17:31.011 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-10-30T13:17:31.011 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-10-30T13:17:31.011 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-10-30T13:17:31.011 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-10-30T13:17:31.012 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-10-30T13:17:31.012 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-10-30T13:17:31.012 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-10-30T13:17:31.012 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-10-30T13:17:31.012 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-10-30T13:17:31.012 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-10-30T13:17:31.013 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-10-30T13:17:31.013 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-10-30T13:17:31.013 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-10-30T13:17:31.013 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-10-30T13:17:31.013 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-10-30T13:17:31.014 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-10-30T13:17:31.014 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-10-30T13:17:31.014 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-10-30T13:17:31.014 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-10-30T13:17:31.014 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-10-30T13:17:31.015 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-10-30T13:17:31.015 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-10-30T13:17:31.015 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-10-30T13:17:31.015 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-10-30T13:17:31.015 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-10-30T13:17:31.015 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-10-30T13:17:31.016 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-10-30T13:17:31.016 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-10-30T13:17:31.016 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-10-30T13:17:31.016 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-10-30T13:17:31.753 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-10-30T13:17:31.754 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-10-30T13:17:31.754 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-10-30T13:17:31.754 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-10-30T13:17:31.754 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-10-30T13:17:31.754 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-10-30T13:17:31.755 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-10-30T13:17:31.755 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-10-30T13:17:31.755 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-10-30T13:17:31.755 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-10-30T13:17:31.755 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-10-30T13:17:31.755 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-10-30T13:17:31.756 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-10-30T13:17:31.756 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-10-30T13:17:31.756 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-10-30T13:17:31.756 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-10-30T13:17:31.756 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-10-30T13:17:31.756 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-10-30T13:17:31.757 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-10-30T13:17:31.757 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-10-30T13:17:31.757 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-10-30T13:17:31.757 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-10-30T13:17:31.757 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-10-30T13:17:31.758 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-10-30T13:17:31.758 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-10-30T13:17:31.758 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-10-30T13:17:31.758 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-10-30T13:17:31.758 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-10-30T13:17:31.758 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-10-30T13:17:31.759 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-10-30T13:17:31.759 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-10-30T13:17:31.759 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-10-30T13:17:31.759 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-10-30T13:17:31.759 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-10-30T13:17:31.760 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-10-30T13:17:31.760 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-10-30T13:17:31.760 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-10-30T13:17:31.760 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-10-30T13:17:31.760 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-10-30T13:17:31.761 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-10-30T13:17:31.761 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-10-30T13:17:31.761 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-10-30T13:17:31.761 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-10-30T13:17:31.761 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-10-30T13:17:31.762 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-10-30T13:17:31.762 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-10-30T13:17:31.762 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-10-30T13:17:31.762 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-10-30T13:17:31.762 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-10-30T13:17:31.763 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-10-30T13:17:31.763 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-10-30T13:17:31.763 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-10-30T13:17:31.763 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-10-30T13:17:31.763 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-10-30T13:17:31.763 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-10-30T13:17:31.764 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-10-30T13:17:31.764 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-10-30T13:17:31.764 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-10-30T13:17:31.764 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-10-30T13:17:31.764 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-10-30T13:17:31.764 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-10-30T13:17:31.765 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-10-30T13:17:31.765 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-10-30T13:17:31.765 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-10-30T13:17:31.765 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-10-30T13:17:31.765 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-10-30T13:17:31.765 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-10-30T13:17:31.766 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-10-30T13:17:31.766 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-10-30T13:17:31.766 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-10-30T13:17:31.766 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-10-30T13:17:31.766 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-10-30T13:17:31.767 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-10-30T13:17:31.767 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-10-30T13:17:31.767 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-10-30T13:17:31.767 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-10-30T13:17:31.767 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-10-30T13:17:31.767 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-10-30T13:17:31.768 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-10-30T13:17:31.768 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-10-30T13:17:31.768 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-10-30T13:17:32.469 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-10-30T13:17:32.470 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-10-30T13:17:32.470 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-10-30T13:17:32.470 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-10-30T13:17:32.470 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-10-30T13:17:32.470 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-10-30T13:17:32.471 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-10-30T13:17:32.471 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-10-30T13:17:32.471 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-10-30T13:17:32.471 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-10-30T13:17:32.471 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-10-30T13:17:32.471 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-10-30T13:17:32.472 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-10-30T13:17:32.472 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-10-30T13:17:32.472 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-10-30T13:17:32.472 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-10-30T13:17:32.472 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-10-30T13:17:32.472 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-10-30T13:17:32.473 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-10-30T13:17:32.473 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-10-30T13:17:32.473 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-10-30T13:17:32.473 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-10-30T13:17:32.473 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-10-30T13:17:32.473 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-10-30T13:17:32.474 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-10-30T13:17:32.474 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-10-30T13:17:32.474 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-10-30T13:17:32.474 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-10-30T13:17:32.475 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-10-30T13:17:32.475 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-10-30T13:17:32.475 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-10-30T13:17:32.475 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-10-30T13:17:32.475 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-10-30T13:17:32.475 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-10-30T13:17:32.476 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-10-30T13:17:32.476 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-10-30T13:17:32.476 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-10-30T13:17:32.476 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-10-30T13:17:32.476 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-10-30T13:17:32.477 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-10-30T13:17:32.477 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-10-30T13:17:32.477 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-10-30T13:17:32.477 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-10-30T13:17:32.477 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-10-30T13:17:32.477 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-10-30T13:17:32.478 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-10-30T13:17:32.478 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-10-30T13:17:32.478 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-10-30T13:17:32.478 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-10-30T13:17:32.478 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-10-30T13:17:32.478 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-10-30T13:17:32.479 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-10-30T13:17:32.479 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-10-30T13:17:32.479 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-10-30T13:17:32.479 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-10-30T13:17:32.479 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-10-30T13:17:32.480 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-10-30T13:17:32.480 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-10-30T13:17:32.480 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-10-30T13:17:32.480 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-10-30T13:17:32.480 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-10-30T13:17:32.481 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-10-30T13:17:32.481 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-10-30T13:17:32.481 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-10-30T13:17:32.481 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-10-30T13:17:32.481 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-10-30T13:17:32.481 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-10-30T13:17:32.482 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-10-30T13:17:32.482 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-10-30T13:17:32.482 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-10-30T13:17:32.482 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-10-30T13:17:32.482 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-10-30T13:17:32.483 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-10-30T13:17:32.483 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-10-30T13:17:32.483 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-10-30T13:17:32.483 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-10-30T13:17:32.483 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-10-30T13:17:32.484 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-10-30T13:17:32.484 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-10-30T13:17:32.484 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-10-30T13:17:32.484 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-10-30T13:17:32.604 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-10-30T13:17:32.605 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-10-30T13:17:32.605 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-10-30T13:17:32.605 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-10-30T13:17:32.605 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-10-30T13:17:32.605 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-10-30T13:17:32.606 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-10-30T13:17:32.606 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-10-30T13:17:32.606 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-10-30T13:17:32.606 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-10-30T13:17:32.606 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-10-30T13:17:32.607 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-10-30T13:17:32.607 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-10-30T13:17:32.607 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-10-30T13:17:32.607 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-10-30T13:17:32.607 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-10-30T13:17:32.607 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-10-30T13:17:32.608 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-10-30T13:17:32.608 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-10-30T13:17:32.608 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-10-30T13:17:32.608 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-10-30T13:17:32.608 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-10-30T13:17:32.609 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-10-30T13:17:32.609 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-10-30T13:17:32.609 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-10-30T13:17:32.609 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-10-30T13:17:32.609 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-10-30T13:17:32.610 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-10-30T13:17:32.610 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-10-30T13:17:32.610 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-10-30T13:17:32.610 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-10-30T13:17:32.610 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-10-30T13:17:32.611 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-10-30T13:17:32.611 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-10-30T13:17:32.611 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-10-30T13:17:32.611 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-10-30T13:17:32.611 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-10-30T13:17:32.612 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-10-30T13:17:32.612 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-10-30T13:17:32.612 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-10-30T13:17:32.612 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-10-30T13:17:32.612 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-10-30T13:17:32.612 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-10-30T13:17:32.613 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-10-30T13:17:32.613 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-10-30T13:17:32.613 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-10-30T13:17:32.613 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-10-30T13:17:32.613 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-10-30T13:17:32.614 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-10-30T13:17:32.614 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-10-30T13:17:32.614 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-10-30T13:17:32.614 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-10-30T13:17:32.614 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-10-30T13:17:32.614 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-10-30T13:17:32.615 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-10-30T13:17:32.615 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-10-30T13:17:32.615 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-10-30T13:17:32.615 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-10-30T13:17:32.615 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-10-30T13:17:32.616 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-10-30T13:17:32.616 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-10-30T13:17:32.616 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-10-30T13:17:32.616 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-10-30T13:17:32.616 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-10-30T13:17:32.617 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-10-30T13:17:32.617 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-10-30T13:17:32.617 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-10-30T13:17:32.617 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-10-30T13:17:32.617 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-10-30T13:17:32.618 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-10-30T13:17:32.618 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-10-30T13:17:32.618 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-10-30T13:17:32.618 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-10-30T13:17:32.618 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-10-30T13:17:32.619 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-10-30T13:17:32.619 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-10-30T13:17:32.619 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-10-30T13:17:32.619 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-10-30T13:17:32.619 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-10-30T13:17:32.619 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-10-30T13:17:32.620 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-10-30T13:17:33.071 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-10-30T13:17:33.072 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-10-30T13:17:33.072 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-10-30T13:17:33.072 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-10-30T13:17:33.072 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-10-30T13:17:33.073 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-10-30T13:17:33.073 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-10-30T13:17:33.073 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-10-30T13:17:33.073 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-10-30T13:17:33.073 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-10-30T13:17:33.074 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-10-30T13:17:33.074 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-10-30T13:17:33.074 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-10-30T13:17:33.074 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-10-30T13:17:33.074 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-10-30T13:17:33.075 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-10-30T13:17:33.075 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-10-30T13:17:33.075 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-10-30T13:17:33.075 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-10-30T13:17:33.075 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-10-30T13:17:33.075 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-10-30T13:17:33.076 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-10-30T13:17:33.076 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-10-30T13:17:33.076 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-10-30T13:17:33.076 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-10-30T13:17:33.076 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-10-30T13:17:33.076 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-10-30T13:17:33.077 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-10-30T13:17:33.077 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-10-30T13:17:33.077 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-10-30T13:17:33.077 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-10-30T13:17:33.077 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-10-30T13:17:33.077 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-10-30T13:17:33.078 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-10-30T13:17:33.078 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-10-30T13:17:33.078 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-10-30T13:17:33.078 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-10-30T13:17:33.078 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-10-30T13:17:33.079 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-10-30T13:17:33.079 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-10-30T13:17:33.079 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-10-30T13:17:33.079 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-10-30T13:17:33.079 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-10-30T13:17:33.079 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-10-30T13:17:33.080 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-10-30T13:17:33.080 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-10-30T13:17:33.080 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-10-30T13:17:33.080 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-10-30T13:17:33.080 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-10-30T13:17:33.080 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-10-30T13:17:33.081 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-10-30T13:17:33.081 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-10-30T13:17:33.081 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-10-30T13:17:33.081 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-10-30T13:17:33.081 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-10-30T13:17:33.081 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-10-30T13:17:33.082 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-10-30T13:17:33.082 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-10-30T13:17:33.082 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-10-30T13:17:33.082 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-10-30T13:17:33.082 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-10-30T13:17:33.083 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-10-30T13:17:33.083 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-10-30T13:17:33.083 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-10-30T13:17:33.083 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-10-30T13:17:33.083 INFO:tasks.workunit.client.0.smithi148.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-10-30T13:17:33.084 INFO:tasks.workunit.client.0.smithi148.stdout:Take third snapshot .snap/test3 2023-10-30T13:17:33.084 INFO:tasks.workunit.client.0.smithi148.stdout:Delete the snapshots... 2023-10-30T13:17:38.672 INFO:tasks.workunit.client.0.smithi148.stdout:.//.snap/test1 2023-10-30T13:17:38.672 INFO:tasks.workunit.client.0.smithi148.stdout:.//.snap/test2 2023-10-30T13:17:38.672 INFO:tasks.workunit.client.0.smithi148.stdout:.//.snap/test3 2023-10-30T13:17:38.680 INFO:tasks.workunit.client.0.smithi148.stdout:./274/.snap/snap-subdir-test 2023-10-30T13:17:38.691 INFO:tasks.workunit.client.0.smithi148.stdout:./166/.snap/snap-subdir-test 2023-10-30T13:17:38.699 INFO:tasks.workunit.client.0.smithi148.stdout:./256/.snap/snap-subdir-test 2023-10-30T13:17:38.707 INFO:tasks.workunit.client.0.smithi148.stdout:./247/.snap/snap-subdir-test 2023-10-30T13:17:38.713 INFO:tasks.workunit.client.0.smithi148.stdout:./248/.snap/snap-subdir-test 2023-10-30T13:17:38.722 INFO:tasks.workunit.client.0.smithi148.stdout:./147/.snap/snap-subdir-test 2023-10-30T13:17:38.730 INFO:tasks.workunit.client.0.smithi148.stdout:./151/.snap/snap-subdir-test 2023-10-30T13:17:38.739 INFO:tasks.workunit.client.0.smithi148.stdout:./170/.snap/snap-subdir-test 2023-10-30T13:17:38.747 INFO:tasks.workunit.client.0.smithi148.stdout:./167/.snap/snap-subdir-test 2023-10-30T13:17:38.751 INFO:tasks.workunit.client.0.smithi148.stdout:./140/.snap/snap-subdir-test 2023-10-30T13:17:38.757 INFO:tasks.workunit.client.0.smithi148.stdout:./142/.snap/snap-subdir-test 2023-10-30T13:17:38.764 INFO:tasks.workunit.client.0.smithi148.stdout:./150/.snap/snap-subdir-test 2023-10-30T13:17:38.772 INFO:tasks.workunit.client.0.smithi148.stdout:./273/.snap/snap-subdir-test 2023-10-30T13:17:38.779 INFO:tasks.workunit.client.0.smithi148.stdout:./269/.snap/snap-subdir-test 2023-10-30T13:17:38.786 INFO:tasks.workunit.client.0.smithi148.stdout:./105/.snap/snap-subdir-test 2023-10-30T13:17:38.792 INFO:tasks.workunit.client.0.smithi148.stdout:./210/.snap/snap-subdir-test 2023-10-30T13:17:38.799 INFO:tasks.workunit.client.0.smithi148.stdout:./265/.snap/snap-subdir-test 2023-10-30T13:17:38.806 INFO:tasks.workunit.client.0.smithi148.stdout:./218/.snap/snap-subdir-test 2023-10-30T13:17:38.812 INFO:tasks.workunit.client.0.smithi148.stdout:./290/.snap/snap-subdir-test 2023-10-30T13:17:38.819 INFO:tasks.workunit.client.0.smithi148.stdout:./149/.snap/snap-subdir-test 2023-10-30T13:17:38.826 INFO:tasks.workunit.client.0.smithi148.stdout:./109/.snap/snap-subdir-test 2023-10-30T13:17:38.831 INFO:tasks.workunit.client.0.smithi148.stdout:./292/.snap/snap-subdir-test 2023-10-30T13:17:38.838 INFO:tasks.workunit.client.0.smithi148.stdout:./242/.snap/snap-subdir-test 2023-10-30T13:17:38.844 INFO:tasks.workunit.client.0.smithi148.stdout:./158/.snap/snap-subdir-test 2023-10-30T13:17:38.851 INFO:tasks.workunit.client.0.smithi148.stdout:./243/.snap/snap-subdir-test 2023-10-30T13:17:38.857 INFO:tasks.workunit.client.0.smithi148.stdout:./106/.snap/snap-subdir-test 2023-10-30T13:17:38.863 INFO:tasks.workunit.client.0.smithi148.stdout:./250/.snap/snap-subdir-test 2023-10-30T13:17:38.868 INFO:tasks.workunit.client.0.smithi148.stdout:./221/.snap/snap-subdir-test 2023-10-30T13:17:38.876 INFO:tasks.workunit.client.0.smithi148.stdout:./244/.snap/snap-subdir-test 2023-10-30T13:17:38.882 INFO:tasks.workunit.client.0.smithi148.stdout:./258/.snap/snap-subdir-test 2023-10-30T13:17:38.889 INFO:tasks.workunit.client.0.smithi148.stdout:./268/.snap/snap-subdir-test 2023-10-30T13:17:38.895 INFO:tasks.workunit.client.0.smithi148.stdout:./189/.snap/snap-subdir-test 2023-10-30T13:17:38.900 INFO:tasks.workunit.client.0.smithi148.stdout:./285/.snap/snap-subdir-test 2023-10-30T13:17:38.907 INFO:tasks.workunit.client.0.smithi148.stdout:./208/.snap/snap-subdir-test 2023-10-30T13:17:38.912 INFO:tasks.workunit.client.0.smithi148.stdout:./288/.snap/snap-subdir-test 2023-10-30T13:17:38.920 INFO:tasks.workunit.client.0.smithi148.stdout:./209/.snap/snap-subdir-test 2023-10-30T13:17:38.927 INFO:tasks.workunit.client.0.smithi148.stdout:./253/.snap/snap-subdir-test 2023-10-30T13:17:38.934 INFO:tasks.workunit.client.0.smithi148.stdout:./249/.snap/snap-subdir-test 2023-10-30T13:17:38.937 INFO:tasks.workunit.client.0.smithi148.stdout:./133/.snap/snap-subdir-test 2023-10-30T13:17:38.943 INFO:tasks.workunit.client.0.smithi148.stdout:./291/.snap/snap-subdir-test 2023-10-30T13:17:39.410 INFO:tasks.workunit.client.0.smithi148.stdout:./226/.snap/snap-subdir-test 2023-10-30T13:17:39.417 INFO:tasks.workunit.client.0.smithi148.stdout:./173/.snap/snap-subdir-test 2023-10-30T13:17:39.428 INFO:tasks.workunit.client.0.smithi148.stdout:./174/.snap/snap-subdir-test 2023-10-30T13:17:39.433 INFO:tasks.workunit.client.0.smithi148.stdout:./299/.snap/snap-subdir-test 2023-10-30T13:17:39.439 INFO:tasks.workunit.client.0.smithi148.stdout:./262/.snap/snap-subdir-test 2023-10-30T13:17:39.446 INFO:tasks.workunit.client.0.smithi148.stdout:./231/.snap/snap-subdir-test 2023-10-30T13:17:39.455 INFO:tasks.workunit.client.0.smithi148.stdout:./214/.snap/snap-subdir-test 2023-10-30T13:17:39.461 INFO:tasks.workunit.client.0.smithi148.stdout:./281/.snap/snap-subdir-test 2023-10-30T13:17:39.467 INFO:tasks.workunit.client.0.smithi148.stdout:./289/.snap/snap-subdir-test 2023-10-30T13:17:39.471 INFO:tasks.workunit.client.0.smithi148.stdout:./283/.snap/snap-subdir-test 2023-10-30T13:17:39.477 INFO:tasks.workunit.client.0.smithi148.stdout:./232/.snap/snap-subdir-test 2023-10-30T13:17:39.482 INFO:tasks.workunit.client.0.smithi148.stdout:./298/.snap/snap-subdir-test 2023-10-30T13:17:39.602 INFO:tasks.workunit.client.0.smithi148.stdout:./233/.snap/snap-subdir-test 2023-10-30T13:17:39.609 INFO:tasks.workunit.client.0.smithi148.stdout:./155/.snap/snap-subdir-test 2023-10-30T13:17:39.613 INFO:tasks.workunit.client.0.smithi148.stdout:./114/.snap/snap-subdir-test 2023-10-30T13:17:40.514 INFO:tasks.workunit.client.0.smithi148.stdout:./159/.snap/snap-subdir-test 2023-10-30T13:17:41.388 INFO:tasks.workunit.client.0.smithi148.stdout:./187/.snap/snap-subdir-test 2023-10-30T13:17:41.396 INFO:tasks.workunit.client.0.smithi148.stdout:./185/.snap/snap-subdir-test 2023-10-30T13:17:41.401 INFO:tasks.workunit.client.0.smithi148.stdout:./293/.snap/snap-subdir-test 2023-10-30T13:17:41.409 INFO:tasks.workunit.client.0.smithi148.stdout:./228/.snap/snap-subdir-test 2023-10-30T13:17:41.413 INFO:tasks.workunit.client.0.smithi148.stdout:./120/.snap/snap-subdir-test 2023-10-30T13:17:41.420 INFO:tasks.workunit.client.0.smithi148.stdout:./110/.snap/snap-subdir-test 2023-10-30T13:17:41.424 INFO:tasks.workunit.client.0.smithi148.stdout:./267/.snap/snap-subdir-test 2023-10-30T13:17:41.434 INFO:tasks.workunit.client.0.smithi148.stdout:./202/.snap/snap-subdir-test 2023-10-30T13:17:41.441 INFO:tasks.workunit.client.0.smithi148.stdout:./137/.snap/snap-subdir-test 2023-10-30T13:17:41.447 INFO:tasks.workunit.client.0.smithi148.stdout:./246/.snap/snap-subdir-test 2023-10-30T13:17:41.455 INFO:tasks.workunit.client.0.smithi148.stdout:./219/.snap/snap-subdir-test 2023-10-30T13:17:41.462 INFO:tasks.workunit.client.0.smithi148.stdout:./198/.snap/snap-subdir-test 2023-10-30T13:17:41.469 INFO:tasks.workunit.client.0.smithi148.stdout:./136/.snap/snap-subdir-test 2023-10-30T13:17:41.473 INFO:tasks.workunit.client.0.smithi148.stdout:./270/.snap/snap-subdir-test 2023-10-30T13:17:41.481 INFO:tasks.workunit.client.0.smithi148.stdout:./125/.snap/snap-subdir-test 2023-10-30T13:17:41.819 INFO:tasks.workunit.client.0.smithi148.stdout:./194/.snap/snap-subdir-test 2023-10-30T13:17:41.823 INFO:tasks.workunit.client.0.smithi148.stdout:./119/.snap/snap-subdir-test 2023-10-30T13:17:41.826 INFO:tasks.workunit.client.0.smithi148.stdout:./297/.snap/snap-subdir-test 2023-10-30T13:17:41.833 INFO:tasks.workunit.client.0.smithi148.stdout:./230/.snap/snap-subdir-test 2023-10-30T13:17:41.836 INFO:tasks.workunit.client.0.smithi148.stdout:./260/.snap/snap-subdir-test 2023-10-30T13:17:41.843 INFO:tasks.workunit.client.0.smithi148.stdout:./157/.snap/snap-subdir-test 2023-10-30T13:17:41.850 INFO:tasks.workunit.client.0.smithi148.stdout:./172/.snap/snap-subdir-test 2023-10-30T13:17:41.854 INFO:tasks.workunit.client.0.smithi148.stdout:./277/.snap/snap-subdir-test 2023-10-30T13:17:41.861 INFO:tasks.workunit.client.0.smithi148.stdout:./186/.snap/snap-subdir-test 2023-10-30T13:17:42.677 INFO:tasks.workunit.client.0.smithi148.stdout:./237/.snap/snap-subdir-test 2023-10-30T13:17:42.683 INFO:tasks.workunit.client.0.smithi148.stdout:./190/.snap/snap-subdir-test 2023-10-30T13:17:42.691 INFO:tasks.workunit.client.0.smithi148.stdout:./213/.snap/snap-subdir-test 2023-10-30T13:17:42.695 INFO:tasks.workunit.client.0.smithi148.stdout:./104/.snap/snap-subdir-test 2023-10-30T13:17:42.703 INFO:tasks.workunit.client.0.smithi148.stdout:./164/.snap/snap-subdir-test 2023-10-30T13:17:42.707 INFO:tasks.workunit.client.0.smithi148.stdout:./112/.snap/snap-subdir-test 2023-10-30T13:17:42.710 INFO:tasks.workunit.client.0.smithi148.stdout:./295/.snap/snap-subdir-test 2023-10-30T13:17:42.714 INFO:tasks.workunit.client.0.smithi148.stdout:./272/.snap/snap-subdir-test 2023-10-30T13:17:43.539 INFO:tasks.workunit.client.0.smithi148.stdout:./100/.snap/snap-subdir-test 2023-10-30T13:17:43.548 INFO:tasks.workunit.client.0.smithi148.stdout:./134/.snap/snap-subdir-test 2023-10-30T13:17:43.557 INFO:tasks.workunit.client.0.smithi148.stdout:./200/.snap/snap-subdir-test 2023-10-30T13:17:43.562 INFO:tasks.workunit.client.0.smithi148.stdout:./115/.snap/snap-subdir-test 2023-10-30T13:17:43.569 INFO:tasks.workunit.client.0.smithi148.stdout:./284/.snap/snap-subdir-test 2023-10-30T13:17:43.573 INFO:tasks.workunit.client.0.smithi148.stdout:./280/.snap/snap-subdir-test 2023-10-30T13:17:43.578 INFO:tasks.workunit.client.0.smithi148.stdout:./116/.snap/snap-subdir-test 2023-10-30T13:17:43.585 INFO:tasks.workunit.client.0.smithi148.stdout:./245/.snap/snap-subdir-test 2023-10-30T13:17:43.600 INFO:tasks.workunit.client.0.smithi148.stdout:./195/.snap/snap-subdir-test 2023-10-30T13:17:43.607 INFO:tasks.workunit.client.0.smithi148.stdout:./138/.snap/snap-subdir-test 2023-10-30T13:17:43.616 INFO:tasks.workunit.client.0.smithi148.stdout:./135/.snap/snap-subdir-test 2023-10-30T13:17:43.623 INFO:tasks.workunit.client.0.smithi148.stdout:./117/.snap/snap-subdir-test 2023-10-30T13:17:44.557 INFO:tasks.workunit.client.0.smithi148.stdout:./241/.snap/snap-subdir-test 2023-10-30T13:17:44.722 INFO:tasks.workunit.client.0.smithi148.stdout:./143/.snap/snap-subdir-test 2023-10-30T13:17:44.727 INFO:tasks.workunit.client.0.smithi148.stdout:./180/.snap/snap-subdir-test 2023-10-30T13:17:44.737 INFO:tasks.workunit.client.0.smithi148.stdout:./126/.snap/snap-subdir-test 2023-10-30T13:17:44.745 INFO:tasks.workunit.client.0.smithi148.stdout:./161/.snap/snap-subdir-test 2023-10-30T13:17:44.826 INFO:tasks.workunit.client.0.smithi148.stdout:./204/.snap/snap-subdir-test 2023-10-30T13:17:45.468 INFO:tasks.workunit.client.0.smithi148.stdout:./176/.snap/snap-subdir-test 2023-10-30T13:17:45.476 INFO:tasks.workunit.client.0.smithi148.stdout:./255/.snap/snap-subdir-test 2023-10-30T13:17:45.484 INFO:tasks.workunit.client.0.smithi148.stdout:./215/.snap/snap-subdir-test 2023-10-30T13:17:45.576 INFO:tasks.workunit.client.0.smithi148.stdout:./111/.snap/snap-subdir-test 2023-10-30T13:17:45.585 INFO:tasks.workunit.client.0.smithi148.stdout:./238/.snap/snap-subdir-test 2023-10-30T13:17:45.594 INFO:tasks.workunit.client.0.smithi148.stdout:./191/.snap/snap-subdir-test 2023-10-30T13:17:45.599 INFO:tasks.workunit.client.0.smithi148.stdout:./184/.snap/snap-subdir-test 2023-10-30T13:17:45.607 INFO:tasks.workunit.client.0.smithi148.stdout:./192/.snap/snap-subdir-test 2023-10-30T13:17:45.612 INFO:tasks.workunit.client.0.smithi148.stdout:./103/.snap/snap-subdir-test 2023-10-30T13:17:45.620 INFO:tasks.workunit.client.0.smithi148.stdout:./128/.snap/snap-subdir-test 2023-10-30T13:17:45.628 INFO:tasks.workunit.client.0.smithi148.stdout:./229/.snap/snap-subdir-test 2023-10-30T13:17:46.089 INFO:tasks.workunit.client.0.smithi148.stdout:./144/.snap/snap-subdir-test 2023-10-30T13:17:46.098 INFO:tasks.workunit.client.0.smithi148.stdout:./146/.snap/snap-subdir-test 2023-10-30T13:17:46.105 INFO:tasks.workunit.client.0.smithi148.stdout:./227/.snap/snap-subdir-test 2023-10-30T13:17:46.112 INFO:tasks.workunit.client.0.smithi148.stdout:./278/.snap/snap-subdir-test 2023-10-30T13:17:46.121 INFO:tasks.workunit.client.0.smithi148.stdout:./141/.snap/snap-subdir-test 2023-10-30T13:17:46.130 INFO:tasks.workunit.client.0.smithi148.stdout:./234/.snap/snap-subdir-test 2023-10-30T13:17:46.137 INFO:tasks.workunit.client.0.smithi148.stdout:./225/.snap/snap-subdir-test 2023-10-30T13:17:46.141 INFO:tasks.workunit.client.0.smithi148.stdout:./182/.snap/snap-subdir-test 2023-10-30T13:17:46.402 INFO:tasks.workunit.client.0.smithi148.stdout:./107/.snap/snap-subdir-test 2023-10-30T13:17:46.409 INFO:tasks.workunit.client.0.smithi148.stdout:./178/.snap/snap-subdir-test 2023-10-30T13:17:46.418 INFO:tasks.workunit.client.0.smithi148.stdout:./266/.snap/snap-subdir-test 2023-10-30T13:17:46.427 INFO:tasks.workunit.client.0.smithi148.stdout:./240/.snap/snap-subdir-test 2023-10-30T13:17:46.483 INFO:tasks.workunit.client.0.smithi148.stdout:./156/.snap/snap-subdir-test 2023-10-30T13:17:46.490 INFO:tasks.workunit.client.0.smithi148.stdout:./131/.snap/snap-subdir-test 2023-10-30T13:17:46.499 INFO:tasks.workunit.client.0.smithi148.stdout:./123/.snap/snap-subdir-test 2023-10-30T13:17:46.506 INFO:tasks.workunit.client.0.smithi148.stdout:./160/.snap/snap-subdir-test 2023-10-30T13:17:47.065 INFO:tasks.workunit.client.0.smithi148.stdout:./165/.snap/snap-subdir-test 2023-10-30T13:17:47.071 INFO:tasks.workunit.client.0.smithi148.stdout:./101/.snap/snap-subdir-test 2023-10-30T13:17:47.081 INFO:tasks.workunit.client.0.smithi148.stdout:./148/.snap/snap-subdir-test 2023-10-30T13:17:47.088 INFO:tasks.workunit.client.0.smithi148.stdout:./168/.snap/snap-subdir-test 2023-10-30T13:17:47.095 INFO:tasks.workunit.client.0.smithi148.stdout:./179/.snap/snap-subdir-test 2023-10-30T13:17:47.103 INFO:tasks.workunit.client.0.smithi148.stdout:./257/.snap/snap-subdir-test 2023-10-30T13:17:47.111 INFO:tasks.workunit.client.0.smithi148.stdout:./252/.snap/snap-subdir-test 2023-10-30T13:17:47.118 INFO:tasks.workunit.client.0.smithi148.stdout:./132/.snap/snap-subdir-test 2023-10-30T13:17:47.125 INFO:tasks.workunit.client.0.smithi148.stdout:./236/.snap/snap-subdir-test 2023-10-30T13:17:47.133 INFO:tasks.workunit.client.0.smithi148.stdout:./211/.snap/snap-subdir-test 2023-10-30T13:17:47.142 INFO:tasks.workunit.client.0.smithi148.stdout:./199/.snap/snap-subdir-test 2023-10-30T13:17:47.150 INFO:tasks.workunit.client.0.smithi148.stdout:./205/.snap/snap-subdir-test 2023-10-30T13:17:47.157 INFO:tasks.workunit.client.0.smithi148.stdout:./163/.snap/snap-subdir-test 2023-10-30T13:17:47.164 INFO:tasks.workunit.client.0.smithi148.stdout:./261/.snap/snap-subdir-test 2023-10-30T13:17:47.172 INFO:tasks.workunit.client.0.smithi148.stdout:./201/.snap/snap-subdir-test 2023-10-30T13:17:47.174 INFO:tasks.workunit.client.0.smithi148.stdout:./282/.snap/snap-subdir-test 2023-10-30T13:17:47.177 INFO:tasks.workunit.client.0.smithi148.stdout:./296/.snap/snap-subdir-test 2023-10-30T13:17:47.184 INFO:tasks.workunit.client.0.smithi148.stdout:./217/.snap/snap-subdir-test 2023-10-30T13:17:47.193 INFO:tasks.workunit.client.0.smithi148.stdout:./153/.snap/snap-subdir-test 2023-10-30T13:17:47.201 INFO:tasks.workunit.client.0.smithi148.stdout:./206/.snap/snap-subdir-test 2023-10-30T13:17:47.208 INFO:tasks.workunit.client.0.smithi148.stdout:./122/.snap/snap-subdir-test 2023-10-30T13:17:47.215 INFO:tasks.workunit.client.0.smithi148.stdout:./162/.snap/snap-subdir-test 2023-10-30T13:17:47.222 INFO:tasks.workunit.client.0.smithi148.stdout:./197/.snap/snap-subdir-test 2023-10-30T13:17:47.228 INFO:tasks.workunit.client.0.smithi148.stdout:./216/.snap/snap-subdir-test 2023-10-30T13:17:50.654 INFO:tasks.workunit.client.0.smithi148.stdout:./254/.snap/snap-subdir-test 2023-10-30T13:17:50.662 INFO:tasks.workunit.client.0.smithi148.stdout:./113/.snap/snap-subdir-test 2023-10-30T13:17:50.670 INFO:tasks.workunit.client.0.smithi148.stdout:./152/.snap/snap-subdir-test 2023-10-30T13:17:50.677 INFO:tasks.workunit.client.0.smithi148.stdout:./121/.snap/snap-subdir-test 2023-10-30T13:17:50.685 INFO:tasks.workunit.client.0.smithi148.stdout:./118/.snap/snap-subdir-test 2023-10-30T13:17:51.145 INFO:tasks.workunit.client.0.smithi148.stdout:./196/.snap/snap-subdir-test 2023-10-30T13:17:51.153 INFO:tasks.workunit.client.0.smithi148.stdout:./124/.snap/snap-subdir-test 2023-10-30T13:17:51.159 INFO:tasks.workunit.client.0.smithi148.stdout:./239/.snap/snap-subdir-test 2023-10-30T13:17:51.167 INFO:tasks.workunit.client.0.smithi148.stdout:./188/.snap/snap-subdir-test 2023-10-30T13:17:51.170 INFO:tasks.workunit.client.0.smithi148.stdout:./287/.snap/snap-subdir-test 2023-10-30T13:17:51.174 INFO:tasks.workunit.client.0.smithi148.stdout:./275/.snap/snap-subdir-test 2023-10-30T13:17:51.178 INFO:tasks.workunit.client.0.smithi148.stdout:./259/.snap/snap-subdir-test 2023-10-30T13:17:51.186 INFO:tasks.workunit.client.0.smithi148.stdout:./224/.snap/snap-subdir-test 2023-10-30T13:17:51.189 INFO:tasks.workunit.client.0.smithi148.stdout:./264/.snap/snap-subdir-test 2023-10-30T13:17:51.197 INFO:tasks.workunit.client.0.smithi148.stdout:./145/.snap/snap-subdir-test 2023-10-30T13:17:51.200 INFO:tasks.workunit.client.0.smithi148.stdout:./279/.snap/snap-subdir-test 2023-10-30T13:17:51.208 INFO:tasks.workunit.client.0.smithi148.stdout:./263/.snap/snap-subdir-test 2023-10-30T13:17:51.214 INFO:tasks.workunit.client.0.smithi148.stdout:./130/.snap/snap-subdir-test 2023-10-30T13:17:51.220 INFO:tasks.workunit.client.0.smithi148.stdout:./193/.snap/snap-subdir-test 2023-10-30T13:17:51.229 INFO:tasks.workunit.client.0.smithi148.stdout:./223/.snap/snap-subdir-test 2023-10-30T13:17:51.235 INFO:tasks.workunit.client.0.smithi148.stdout:./207/.snap/snap-subdir-test 2023-10-30T13:17:51.340 INFO:tasks.workunit.client.0.smithi148.stdout:./276/.snap/snap-subdir-test 2023-10-30T13:17:51.494 INFO:tasks.workunit.client.0.smithi148.stdout:./271/.snap/snap-subdir-test 2023-10-30T13:17:51.501 INFO:tasks.workunit.client.0.smithi148.stdout:./251/.snap/snap-subdir-test 2023-10-30T13:17:51.697 INFO:tasks.workunit.client.0.smithi148.stdout:./175/.snap/snap-subdir-test 2023-10-30T13:17:51.704 INFO:tasks.workunit.client.0.smithi148.stdout:./222/.snap/snap-subdir-test 2023-10-30T13:17:51.709 INFO:tasks.workunit.client.0.smithi148.stdout:./220/.snap/snap-subdir-test 2023-10-30T13:17:51.716 INFO:tasks.workunit.client.0.smithi148.stdout:./203/.snap/snap-subdir-test 2023-10-30T13:17:51.722 INFO:tasks.workunit.client.0.smithi148.stdout:./181/.snap/snap-subdir-test 2023-10-30T13:17:51.725 INFO:tasks.workunit.client.0.smithi148.stdout:./286/.snap/snap-subdir-test 2023-10-30T13:17:51.733 INFO:tasks.workunit.client.0.smithi148.stdout:./127/.snap/snap-subdir-test 2023-10-30T13:17:51.739 INFO:tasks.workunit.client.0.smithi148.stdout:./108/.snap/snap-subdir-test 2023-10-30T13:17:51.745 INFO:tasks.workunit.client.0.smithi148.stdout:./183/.snap/snap-subdir-test 2023-10-30T13:17:52.002 INFO:tasks.workunit.client.0.smithi148.stdout:./294/.snap/snap-subdir-test 2023-10-30T13:17:52.357 INFO:tasks.workunit.client.0.smithi148.stdout:./235/.snap/snap-subdir-test 2023-10-30T13:17:52.365 INFO:tasks.workunit.client.0.smithi148.stdout:./177/.snap/snap-subdir-test 2023-10-30T13:17:52.625 INFO:tasks.workunit.client.0.smithi148.stdout:./169/.snap/snap-subdir-test 2023-10-30T13:17:52.633 INFO:tasks.workunit.client.0.smithi148.stdout:./212/.snap/snap-subdir-test 2023-10-30T13:17:53.488 INFO:tasks.workunit.client.0.smithi148.stdout:./154/.snap/snap-subdir-test 2023-10-30T13:17:53.497 INFO:tasks.workunit.client.0.smithi148.stdout:./171/.snap/snap-subdir-test 2023-10-30T13:17:53.507 INFO:tasks.workunit.client.0.smithi148.stdout:./139/.snap/snap-subdir-test 2023-10-30T13:17:53.515 INFO:tasks.workunit.client.0.smithi148.stdout:./102/.snap/snap-subdir-test 2023-10-30T13:17:53.523 INFO:tasks.workunit.client.0.smithi148.stdout:./129/.snap/snap-subdir-test 2023-10-30T13:17:54.405 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:17:54.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:18:02.721 INFO:tasks.workunit.client.0.smithi148.stdout:Delete all the files and directories ... 2023-10-30T13:18:08.661 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/19' 2023-10-30T13:18:08.662 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/15' 2023-10-30T13:18:08.662 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/31' 2023-10-30T13:18:08.662 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/36' 2023-10-30T13:18:08.662 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/20' 2023-10-30T13:18:08.663 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/12' 2023-10-30T13:18:08.663 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/39' 2023-10-30T13:18:08.663 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/10' 2023-10-30T13:18:08.663 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/34' 2023-10-30T13:18:08.663 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/38' 2023-10-30T13:18:08.664 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/11' 2023-10-30T13:18:08.664 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/16' 2023-10-30T13:18:08.664 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/32' 2023-10-30T13:18:08.664 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/30' 2023-10-30T13:18:08.664 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/14' 2023-10-30T13:18:08.664 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/18' 2023-10-30T13:18:08.665 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/13' 2023-10-30T13:18:08.665 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/33' 2023-10-30T13:18:08.665 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/17' 2023-10-30T13:18:08.665 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/35' 2023-10-30T13:18:08.665 INFO:tasks.workunit.client.0.smithi148.stdout:removed './100/37' 2023-10-30T13:18:08.666 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './100' 2023-10-30T13:18:08.666 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/19' 2023-10-30T13:18:08.666 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/15' 2023-10-30T13:18:08.666 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/31' 2023-10-30T13:18:08.666 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/36' 2023-10-30T13:18:08.667 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/20' 2023-10-30T13:18:08.667 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/12' 2023-10-30T13:18:08.667 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/39' 2023-10-30T13:18:08.667 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/10' 2023-10-30T13:18:08.667 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/34' 2023-10-30T13:18:08.667 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/38' 2023-10-30T13:18:08.668 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/11' 2023-10-30T13:18:08.668 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/16' 2023-10-30T13:18:08.668 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/32' 2023-10-30T13:18:08.668 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/30' 2023-10-30T13:18:08.668 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/14' 2023-10-30T13:18:08.668 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/18' 2023-10-30T13:18:08.669 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/13' 2023-10-30T13:18:08.669 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/33' 2023-10-30T13:18:08.669 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/17' 2023-10-30T13:18:08.669 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/35' 2023-10-30T13:18:08.669 INFO:tasks.workunit.client.0.smithi148.stdout:removed './101/37' 2023-10-30T13:18:08.670 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './101' 2023-10-30T13:18:08.670 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/19' 2023-10-30T13:18:08.670 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/15' 2023-10-30T13:18:08.670 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/31' 2023-10-30T13:18:08.670 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/36' 2023-10-30T13:18:08.670 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/20' 2023-10-30T13:18:08.671 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/12' 2023-10-30T13:18:08.671 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/39' 2023-10-30T13:18:08.671 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/10' 2023-10-30T13:18:08.671 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/34' 2023-10-30T13:18:08.671 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/38' 2023-10-30T13:18:08.671 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/11' 2023-10-30T13:18:08.672 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/16' 2023-10-30T13:18:08.672 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/32' 2023-10-30T13:18:08.672 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/30' 2023-10-30T13:18:08.672 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/14' 2023-10-30T13:18:08.672 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/18' 2023-10-30T13:18:08.673 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/13' 2023-10-30T13:18:08.673 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/33' 2023-10-30T13:18:08.673 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/17' 2023-10-30T13:18:08.673 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/35' 2023-10-30T13:18:08.673 INFO:tasks.workunit.client.0.smithi148.stdout:removed './102/37' 2023-10-30T13:18:08.673 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './102' 2023-10-30T13:18:08.674 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/19' 2023-10-30T13:18:08.674 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/15' 2023-10-30T13:18:08.674 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/31' 2023-10-30T13:18:08.674 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/36' 2023-10-30T13:18:08.674 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/20' 2023-10-30T13:18:08.675 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/12' 2023-10-30T13:18:08.675 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/39' 2023-10-30T13:18:08.675 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/10' 2023-10-30T13:18:08.675 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/34' 2023-10-30T13:18:08.675 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/38' 2023-10-30T13:18:08.675 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/11' 2023-10-30T13:18:08.676 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/16' 2023-10-30T13:18:08.676 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/32' 2023-10-30T13:18:08.676 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/30' 2023-10-30T13:18:08.676 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/14' 2023-10-30T13:18:08.676 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/18' 2023-10-30T13:18:08.677 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/13' 2023-10-30T13:18:08.677 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/33' 2023-10-30T13:18:08.677 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/17' 2023-10-30T13:18:08.677 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/35' 2023-10-30T13:18:08.677 INFO:tasks.workunit.client.0.smithi148.stdout:removed './103/37' 2023-10-30T13:18:08.678 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './103' 2023-10-30T13:18:08.678 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/19' 2023-10-30T13:18:08.678 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/15' 2023-10-30T13:18:08.678 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/31' 2023-10-30T13:18:08.678 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/36' 2023-10-30T13:18:08.678 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/20' 2023-10-30T13:18:08.679 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/12' 2023-10-30T13:18:08.679 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/39' 2023-10-30T13:18:08.679 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/10' 2023-10-30T13:18:08.679 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/34' 2023-10-30T13:18:08.679 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/38' 2023-10-30T13:18:08.680 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/11' 2023-10-30T13:18:08.680 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/16' 2023-10-30T13:18:08.680 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/32' 2023-10-30T13:18:08.680 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/30' 2023-10-30T13:18:08.680 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/14' 2023-10-30T13:18:08.680 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/18' 2023-10-30T13:18:08.681 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/13' 2023-10-30T13:18:08.681 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/33' 2023-10-30T13:18:08.681 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/17' 2023-10-30T13:18:08.681 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/35' 2023-10-30T13:18:08.681 INFO:tasks.workunit.client.0.smithi148.stdout:removed './104/37' 2023-10-30T13:18:08.681 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './104' 2023-10-30T13:18:08.682 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/19' 2023-10-30T13:18:08.682 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/15' 2023-10-30T13:18:08.682 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/31' 2023-10-30T13:18:08.682 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/36' 2023-10-30T13:18:08.682 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/20' 2023-10-30T13:18:08.682 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/12' 2023-10-30T13:18:08.683 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/39' 2023-10-30T13:18:08.683 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/10' 2023-10-30T13:18:08.683 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/34' 2023-10-30T13:18:08.683 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/38' 2023-10-30T13:18:08.683 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/11' 2023-10-30T13:18:08.684 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/16' 2023-10-30T13:18:08.684 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/32' 2023-10-30T13:18:08.684 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/30' 2023-10-30T13:18:08.684 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/14' 2023-10-30T13:18:08.684 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/18' 2023-10-30T13:18:08.684 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/13' 2023-10-30T13:18:08.685 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/33' 2023-10-30T13:18:08.685 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/17' 2023-10-30T13:18:08.685 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/35' 2023-10-30T13:18:08.685 INFO:tasks.workunit.client.0.smithi148.stdout:removed './105/37' 2023-10-30T13:18:08.685 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './105' 2023-10-30T13:18:08.685 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/19' 2023-10-30T13:18:08.686 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/15' 2023-10-30T13:18:08.686 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/31' 2023-10-30T13:18:08.686 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/36' 2023-10-30T13:18:08.686 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/20' 2023-10-30T13:18:08.686 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/12' 2023-10-30T13:18:08.687 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/39' 2023-10-30T13:18:08.687 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/10' 2023-10-30T13:18:08.687 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/34' 2023-10-30T13:18:08.687 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/38' 2023-10-30T13:18:08.687 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/11' 2023-10-30T13:18:08.687 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/16' 2023-10-30T13:18:08.688 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/32' 2023-10-30T13:18:08.688 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/30' 2023-10-30T13:18:08.688 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/14' 2023-10-30T13:18:08.688 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/18' 2023-10-30T13:18:08.688 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/13' 2023-10-30T13:18:08.689 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/33' 2023-10-30T13:18:08.689 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/17' 2023-10-30T13:18:08.689 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/35' 2023-10-30T13:18:08.689 INFO:tasks.workunit.client.0.smithi148.stdout:removed './106/37' 2023-10-30T13:18:08.689 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './106' 2023-10-30T13:18:08.689 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/19' 2023-10-30T13:18:08.690 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/15' 2023-10-30T13:18:08.690 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/31' 2023-10-30T13:18:08.690 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/36' 2023-10-30T13:18:08.690 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/20' 2023-10-30T13:18:08.690 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/12' 2023-10-30T13:18:08.691 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/39' 2023-10-30T13:18:08.691 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/10' 2023-10-30T13:18:08.691 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/34' 2023-10-30T13:18:08.691 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/38' 2023-10-30T13:18:08.691 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/11' 2023-10-30T13:18:08.691 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/16' 2023-10-30T13:18:08.692 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/32' 2023-10-30T13:18:08.692 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/30' 2023-10-30T13:18:08.692 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/14' 2023-10-30T13:18:08.692 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/18' 2023-10-30T13:18:08.692 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/13' 2023-10-30T13:18:08.692 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/33' 2023-10-30T13:18:08.693 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/17' 2023-10-30T13:18:08.693 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/35' 2023-10-30T13:18:08.693 INFO:tasks.workunit.client.0.smithi148.stdout:removed './107/37' 2023-10-30T13:18:08.693 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './107' 2023-10-30T13:18:08.693 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/19' 2023-10-30T13:18:08.694 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/15' 2023-10-30T13:18:08.694 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/31' 2023-10-30T13:18:08.694 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/36' 2023-10-30T13:18:08.694 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/20' 2023-10-30T13:18:08.694 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/12' 2023-10-30T13:18:08.694 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/39' 2023-10-30T13:18:08.695 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/10' 2023-10-30T13:18:08.695 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/34' 2023-10-30T13:18:08.695 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/38' 2023-10-30T13:18:08.695 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/11' 2023-10-30T13:18:08.695 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/16' 2023-10-30T13:18:08.696 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/32' 2023-10-30T13:18:08.696 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/30' 2023-10-30T13:18:08.696 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/14' 2023-10-30T13:18:08.696 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/18' 2023-10-30T13:18:08.696 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/13' 2023-10-30T13:18:08.696 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/33' 2023-10-30T13:18:08.697 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/17' 2023-10-30T13:18:08.697 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/35' 2023-10-30T13:18:08.697 INFO:tasks.workunit.client.0.smithi148.stdout:removed './108/37' 2023-10-30T13:18:08.697 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './108' 2023-10-30T13:18:08.697 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/19' 2023-10-30T13:18:08.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/15' 2023-10-30T13:18:08.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/31' 2023-10-30T13:18:08.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/36' 2023-10-30T13:18:08.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/20' 2023-10-30T13:18:08.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/12' 2023-10-30T13:18:08.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/39' 2023-10-30T13:18:08.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/10' 2023-10-30T13:18:08.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/34' 2023-10-30T13:18:08.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/38' 2023-10-30T13:18:08.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/11' 2023-10-30T13:18:08.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/16' 2023-10-30T13:18:08.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/32' 2023-10-30T13:18:08.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/30' 2023-10-30T13:18:12.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/14' 2023-10-30T13:18:12.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/18' 2023-10-30T13:18:12.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/13' 2023-10-30T13:18:12.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/33' 2023-10-30T13:18:12.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/17' 2023-10-30T13:18:12.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/35' 2023-10-30T13:18:12.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './109/37' 2023-10-30T13:18:12.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './109' 2023-10-30T13:18:12.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/19' 2023-10-30T13:18:12.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/15' 2023-10-30T13:18:12.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/31' 2023-10-30T13:18:12.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/36' 2023-10-30T13:18:12.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/20' 2023-10-30T13:18:12.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/12' 2023-10-30T13:18:12.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/39' 2023-10-30T13:18:12.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/10' 2023-10-30T13:18:12.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/34' 2023-10-30T13:18:12.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/38' 2023-10-30T13:18:12.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/11' 2023-10-30T13:18:12.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/16' 2023-10-30T13:18:12.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/32' 2023-10-30T13:18:12.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/30' 2023-10-30T13:18:12.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/14' 2023-10-30T13:18:12.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/18' 2023-10-30T13:18:12.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/13' 2023-10-30T13:18:12.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/33' 2023-10-30T13:18:12.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/17' 2023-10-30T13:18:12.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/35' 2023-10-30T13:18:12.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed './110/37' 2023-10-30T13:18:12.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './110' 2023-10-30T13:18:12.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/19' 2023-10-30T13:18:12.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/15' 2023-10-30T13:18:12.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/31' 2023-10-30T13:18:12.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/36' 2023-10-30T13:18:12.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/20' 2023-10-30T13:18:12.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/12' 2023-10-30T13:18:12.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/39' 2023-10-30T13:18:12.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/10' 2023-10-30T13:18:12.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/34' 2023-10-30T13:18:12.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/38' 2023-10-30T13:18:12.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/11' 2023-10-30T13:18:12.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/16' 2023-10-30T13:18:12.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/32' 2023-10-30T13:18:12.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/30' 2023-10-30T13:18:12.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/14' 2023-10-30T13:18:12.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/18' 2023-10-30T13:18:12.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/13' 2023-10-30T13:18:12.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/33' 2023-10-30T13:18:12.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/17' 2023-10-30T13:18:12.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/35' 2023-10-30T13:18:12.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './111/37' 2023-10-30T13:18:12.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './111' 2023-10-30T13:18:12.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/19' 2023-10-30T13:18:12.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/15' 2023-10-30T13:18:12.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/31' 2023-10-30T13:18:12.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/36' 2023-10-30T13:18:12.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/20' 2023-10-30T13:18:12.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/12' 2023-10-30T13:18:12.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/39' 2023-10-30T13:18:12.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/10' 2023-10-30T13:18:12.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/34' 2023-10-30T13:18:12.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/38' 2023-10-30T13:18:12.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/11' 2023-10-30T13:18:12.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/16' 2023-10-30T13:18:12.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/32' 2023-10-30T13:18:12.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/30' 2023-10-30T13:18:12.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/14' 2023-10-30T13:18:12.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/18' 2023-10-30T13:18:12.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/13' 2023-10-30T13:18:12.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/33' 2023-10-30T13:18:12.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/17' 2023-10-30T13:18:12.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/35' 2023-10-30T13:18:12.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './112/37' 2023-10-30T13:18:12.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './112' 2023-10-30T13:18:12.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/19' 2023-10-30T13:18:12.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/15' 2023-10-30T13:18:12.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/31' 2023-10-30T13:18:12.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/36' 2023-10-30T13:18:12.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/20' 2023-10-30T13:18:12.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/12' 2023-10-30T13:18:12.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/39' 2023-10-30T13:18:12.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/10' 2023-10-30T13:18:12.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/34' 2023-10-30T13:18:12.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/38' 2023-10-30T13:18:12.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/11' 2023-10-30T13:18:12.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/16' 2023-10-30T13:18:12.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/32' 2023-10-30T13:18:12.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/30' 2023-10-30T13:18:12.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/14' 2023-10-30T13:18:12.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/18' 2023-10-30T13:18:12.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/13' 2023-10-30T13:18:12.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/33' 2023-10-30T13:18:12.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/17' 2023-10-30T13:18:12.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/35' 2023-10-30T13:18:12.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './113/37' 2023-10-30T13:18:12.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './113' 2023-10-30T13:18:12.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/19' 2023-10-30T13:18:12.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/15' 2023-10-30T13:18:12.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/31' 2023-10-30T13:18:12.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/36' 2023-10-30T13:18:12.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/20' 2023-10-30T13:18:12.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/12' 2023-10-30T13:18:12.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/39' 2023-10-30T13:18:12.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/10' 2023-10-30T13:18:12.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/34' 2023-10-30T13:18:12.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/38' 2023-10-30T13:18:12.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/11' 2023-10-30T13:18:12.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/16' 2023-10-30T13:18:12.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/32' 2023-10-30T13:18:12.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/30' 2023-10-30T13:18:12.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/14' 2023-10-30T13:18:12.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/18' 2023-10-30T13:18:12.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/13' 2023-10-30T13:18:12.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/33' 2023-10-30T13:18:12.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/17' 2023-10-30T13:18:12.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/35' 2023-10-30T13:18:12.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './114/37' 2023-10-30T13:18:12.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './114' 2023-10-30T13:18:12.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/19' 2023-10-30T13:18:12.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/15' 2023-10-30T13:18:12.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/31' 2023-10-30T13:18:12.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/36' 2023-10-30T13:18:12.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/20' 2023-10-30T13:18:12.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/12' 2023-10-30T13:18:12.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/39' 2023-10-30T13:18:12.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/10' 2023-10-30T13:18:12.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/34' 2023-10-30T13:18:12.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/38' 2023-10-30T13:18:12.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/11' 2023-10-30T13:18:12.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/16' 2023-10-30T13:18:12.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/32' 2023-10-30T13:18:12.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/30' 2023-10-30T13:18:12.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/14' 2023-10-30T13:18:12.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/18' 2023-10-30T13:18:12.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/13' 2023-10-30T13:18:12.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/33' 2023-10-30T13:18:12.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/17' 2023-10-30T13:18:12.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/35' 2023-10-30T13:18:12.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './115/37' 2023-10-30T13:18:12.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './115' 2023-10-30T13:18:12.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/19' 2023-10-30T13:18:12.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/15' 2023-10-30T13:18:12.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/31' 2023-10-30T13:18:12.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/36' 2023-10-30T13:18:12.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/20' 2023-10-30T13:18:12.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/12' 2023-10-30T13:18:12.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/39' 2023-10-30T13:18:12.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/10' 2023-10-30T13:18:12.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/34' 2023-10-30T13:18:12.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/38' 2023-10-30T13:18:12.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/11' 2023-10-30T13:18:12.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/16' 2023-10-30T13:18:12.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/32' 2023-10-30T13:18:12.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/30' 2023-10-30T13:18:12.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/14' 2023-10-30T13:18:12.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/18' 2023-10-30T13:18:12.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/13' 2023-10-30T13:18:12.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/33' 2023-10-30T13:18:12.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/17' 2023-10-30T13:18:12.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/35' 2023-10-30T13:18:12.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './116/37' 2023-10-30T13:18:12.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './116' 2023-10-30T13:18:12.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/19' 2023-10-30T13:18:12.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/15' 2023-10-30T13:18:12.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/31' 2023-10-30T13:18:12.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/36' 2023-10-30T13:18:12.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/20' 2023-10-30T13:18:12.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/12' 2023-10-30T13:18:12.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/39' 2023-10-30T13:18:12.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/10' 2023-10-30T13:18:12.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/34' 2023-10-30T13:18:12.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/38' 2023-10-30T13:18:12.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/11' 2023-10-30T13:18:12.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/16' 2023-10-30T13:18:12.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/32' 2023-10-30T13:18:12.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/30' 2023-10-30T13:18:12.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/14' 2023-10-30T13:18:12.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/18' 2023-10-30T13:18:12.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/13' 2023-10-30T13:18:12.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/33' 2023-10-30T13:18:12.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/17' 2023-10-30T13:18:12.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/35' 2023-10-30T13:18:12.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './117/37' 2023-10-30T13:18:12.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './117' 2023-10-30T13:18:12.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/19' 2023-10-30T13:18:12.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/15' 2023-10-30T13:18:12.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/31' 2023-10-30T13:18:12.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/36' 2023-10-30T13:18:12.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/20' 2023-10-30T13:18:12.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/12' 2023-10-30T13:18:12.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/39' 2023-10-30T13:18:12.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/10' 2023-10-30T13:18:12.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/34' 2023-10-30T13:18:12.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/38' 2023-10-30T13:18:12.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/11' 2023-10-30T13:18:12.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/16' 2023-10-30T13:18:12.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/32' 2023-10-30T13:18:12.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/30' 2023-10-30T13:18:12.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/14' 2023-10-30T13:18:12.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/18' 2023-10-30T13:18:12.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/13' 2023-10-30T13:18:12.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/33' 2023-10-30T13:18:12.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/17' 2023-10-30T13:18:12.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/35' 2023-10-30T13:18:12.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './118/37' 2023-10-30T13:18:12.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './118' 2023-10-30T13:18:12.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/19' 2023-10-30T13:18:12.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/15' 2023-10-30T13:18:12.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/31' 2023-10-30T13:18:12.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/36' 2023-10-30T13:18:12.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/20' 2023-10-30T13:18:12.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/12' 2023-10-30T13:18:22.186 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/39' 2023-10-30T13:18:22.186 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/10' 2023-10-30T13:18:22.187 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/34' 2023-10-30T13:18:22.187 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/38' 2023-10-30T13:18:22.187 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/11' 2023-10-30T13:18:22.187 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/16' 2023-10-30T13:18:22.187 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/32' 2023-10-30T13:18:22.187 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/30' 2023-10-30T13:18:22.188 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/14' 2023-10-30T13:18:22.188 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/18' 2023-10-30T13:18:22.188 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/13' 2023-10-30T13:18:22.188 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/33' 2023-10-30T13:18:22.188 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/17' 2023-10-30T13:18:22.188 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/35' 2023-10-30T13:18:22.189 INFO:tasks.workunit.client.0.smithi148.stdout:removed './119/37' 2023-10-30T13:18:22.189 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './119' 2023-10-30T13:18:22.189 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/19' 2023-10-30T13:18:22.189 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/15' 2023-10-30T13:18:22.189 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/31' 2023-10-30T13:18:22.190 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/36' 2023-10-30T13:18:22.190 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/20' 2023-10-30T13:18:22.190 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/12' 2023-10-30T13:18:22.190 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/39' 2023-10-30T13:18:22.190 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/10' 2023-10-30T13:18:22.190 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/34' 2023-10-30T13:18:22.191 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/38' 2023-10-30T13:18:22.191 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/11' 2023-10-30T13:18:22.191 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/16' 2023-10-30T13:18:22.191 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/32' 2023-10-30T13:18:22.191 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/30' 2023-10-30T13:18:22.191 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/14' 2023-10-30T13:18:22.192 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/18' 2023-10-30T13:18:22.192 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/13' 2023-10-30T13:18:22.192 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/33' 2023-10-30T13:18:22.192 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/17' 2023-10-30T13:18:22.192 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/35' 2023-10-30T13:18:22.192 INFO:tasks.workunit.client.0.smithi148.stdout:removed './120/37' 2023-10-30T13:18:22.193 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './120' 2023-10-30T13:18:22.193 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/19' 2023-10-30T13:18:22.193 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/15' 2023-10-30T13:18:22.193 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/31' 2023-10-30T13:18:22.193 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/36' 2023-10-30T13:18:22.193 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/20' 2023-10-30T13:18:22.193 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/12' 2023-10-30T13:18:22.194 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/39' 2023-10-30T13:18:22.194 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/10' 2023-10-30T13:18:22.194 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/34' 2023-10-30T13:18:22.194 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/38' 2023-10-30T13:18:22.194 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/11' 2023-10-30T13:18:22.194 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/16' 2023-10-30T13:18:22.195 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/32' 2023-10-30T13:18:22.195 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/30' 2023-10-30T13:18:22.195 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/14' 2023-10-30T13:18:22.195 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/18' 2023-10-30T13:18:22.195 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/13' 2023-10-30T13:18:22.196 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/33' 2023-10-30T13:18:22.196 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/17' 2023-10-30T13:18:22.196 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/35' 2023-10-30T13:18:22.196 INFO:tasks.workunit.client.0.smithi148.stdout:removed './121/37' 2023-10-30T13:18:22.196 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './121' 2023-10-30T13:18:22.196 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/19' 2023-10-30T13:18:22.197 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/15' 2023-10-30T13:18:22.197 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/31' 2023-10-30T13:18:22.197 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/36' 2023-10-30T13:18:22.197 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/20' 2023-10-30T13:18:22.197 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/12' 2023-10-30T13:18:22.197 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/39' 2023-10-30T13:18:22.198 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/10' 2023-10-30T13:18:22.198 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/34' 2023-10-30T13:18:22.198 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/38' 2023-10-30T13:18:22.198 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/11' 2023-10-30T13:18:22.198 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/16' 2023-10-30T13:18:22.199 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/32' 2023-10-30T13:18:22.199 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/30' 2023-10-30T13:18:22.199 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/14' 2023-10-30T13:18:22.199 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/18' 2023-10-30T13:18:22.199 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/13' 2023-10-30T13:18:22.199 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/33' 2023-10-30T13:18:22.200 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/17' 2023-10-30T13:18:22.200 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/35' 2023-10-30T13:18:22.200 INFO:tasks.workunit.client.0.smithi148.stdout:removed './122/37' 2023-10-30T13:18:22.200 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './122' 2023-10-30T13:18:22.200 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/19' 2023-10-30T13:18:22.201 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/15' 2023-10-30T13:18:22.201 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/31' 2023-10-30T13:18:22.201 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/36' 2023-10-30T13:18:22.201 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/20' 2023-10-30T13:18:22.201 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/12' 2023-10-30T13:18:22.201 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/39' 2023-10-30T13:18:22.202 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/10' 2023-10-30T13:18:22.202 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/34' 2023-10-30T13:18:22.202 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/38' 2023-10-30T13:18:22.202 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/11' 2023-10-30T13:18:22.202 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/16' 2023-10-30T13:18:22.202 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/32' 2023-10-30T13:18:22.203 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/30' 2023-10-30T13:18:22.203 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/14' 2023-10-30T13:18:22.203 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/18' 2023-10-30T13:18:22.203 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/13' 2023-10-30T13:18:22.203 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/33' 2023-10-30T13:18:22.204 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/17' 2023-10-30T13:18:22.204 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/35' 2023-10-30T13:18:22.204 INFO:tasks.workunit.client.0.smithi148.stdout:removed './123/37' 2023-10-30T13:18:22.204 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './123' 2023-10-30T13:18:22.204 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/19' 2023-10-30T13:18:22.204 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/15' 2023-10-30T13:18:22.205 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/31' 2023-10-30T13:18:22.205 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/36' 2023-10-30T13:18:22.205 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/20' 2023-10-30T13:18:22.205 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/12' 2023-10-30T13:18:22.205 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/39' 2023-10-30T13:18:22.205 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/10' 2023-10-30T13:18:22.206 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/34' 2023-10-30T13:18:22.206 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/38' 2023-10-30T13:18:22.206 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/11' 2023-10-30T13:18:22.206 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/16' 2023-10-30T13:18:22.206 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/32' 2023-10-30T13:18:22.207 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/30' 2023-10-30T13:18:22.207 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/14' 2023-10-30T13:18:22.207 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/18' 2023-10-30T13:18:22.207 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/13' 2023-10-30T13:18:22.207 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/33' 2023-10-30T13:18:22.207 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/17' 2023-10-30T13:18:22.208 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/35' 2023-10-30T13:18:22.208 INFO:tasks.workunit.client.0.smithi148.stdout:removed './124/37' 2023-10-30T13:18:22.208 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './124' 2023-10-30T13:18:22.208 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/19' 2023-10-30T13:18:22.208 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/15' 2023-10-30T13:18:22.209 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/31' 2023-10-30T13:18:22.209 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/36' 2023-10-30T13:18:22.209 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/20' 2023-10-30T13:18:22.209 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/12' 2023-10-30T13:18:22.209 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/39' 2023-10-30T13:18:22.209 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/10' 2023-10-30T13:18:22.210 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/34' 2023-10-30T13:18:22.210 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/38' 2023-10-30T13:18:22.210 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/11' 2023-10-30T13:18:22.210 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/16' 2023-10-30T13:18:22.210 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/32' 2023-10-30T13:18:22.210 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/30' 2023-10-30T13:18:22.211 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/14' 2023-10-30T13:18:22.211 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/18' 2023-10-30T13:18:22.211 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/13' 2023-10-30T13:18:22.211 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/33' 2023-10-30T13:18:22.211 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/17' 2023-10-30T13:18:22.212 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/35' 2023-10-30T13:18:22.212 INFO:tasks.workunit.client.0.smithi148.stdout:removed './125/37' 2023-10-30T13:18:22.212 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './125' 2023-10-30T13:18:22.212 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/19' 2023-10-30T13:18:22.212 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/15' 2023-10-30T13:18:22.212 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/31' 2023-10-30T13:18:22.213 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/36' 2023-10-30T13:18:22.213 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/20' 2023-10-30T13:18:22.213 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/12' 2023-10-30T13:18:22.213 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/39' 2023-10-30T13:18:22.214 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/10' 2023-10-30T13:18:22.214 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/34' 2023-10-30T13:18:22.214 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/38' 2023-10-30T13:18:22.214 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/11' 2023-10-30T13:18:22.214 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/16' 2023-10-30T13:18:22.214 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/32' 2023-10-30T13:18:22.215 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/30' 2023-10-30T13:18:22.215 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/14' 2023-10-30T13:18:22.215 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/18' 2023-10-30T13:18:22.215 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/13' 2023-10-30T13:18:22.215 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/33' 2023-10-30T13:18:22.216 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/17' 2023-10-30T13:18:22.216 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/35' 2023-10-30T13:18:22.216 INFO:tasks.workunit.client.0.smithi148.stdout:removed './126/37' 2023-10-30T13:18:22.216 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './126' 2023-10-30T13:18:22.216 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/19' 2023-10-30T13:18:22.216 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/15' 2023-10-30T13:18:22.217 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/31' 2023-10-30T13:18:22.217 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/36' 2023-10-30T13:18:22.217 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/20' 2023-10-30T13:18:22.217 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/12' 2023-10-30T13:18:22.217 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/39' 2023-10-30T13:18:22.217 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/10' 2023-10-30T13:18:22.218 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/34' 2023-10-30T13:18:22.218 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/38' 2023-10-30T13:18:22.218 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/11' 2023-10-30T13:18:22.218 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/16' 2023-10-30T13:18:22.218 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/32' 2023-10-30T13:18:22.218 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/30' 2023-10-30T13:18:22.219 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/14' 2023-10-30T13:18:22.219 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/18' 2023-10-30T13:18:22.219 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/13' 2023-10-30T13:18:22.219 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/33' 2023-10-30T13:18:22.219 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/17' 2023-10-30T13:18:22.219 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/35' 2023-10-30T13:18:22.220 INFO:tasks.workunit.client.0.smithi148.stdout:removed './127/37' 2023-10-30T13:18:22.220 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './127' 2023-10-30T13:18:22.220 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/19' 2023-10-30T13:18:22.220 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/15' 2023-10-30T13:18:22.220 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/31' 2023-10-30T13:18:22.220 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/36' 2023-10-30T13:18:22.221 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/20' 2023-10-30T13:18:22.221 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/12' 2023-10-30T13:18:22.221 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/39' 2023-10-30T13:18:22.221 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/10' 2023-10-30T13:18:22.221 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/34' 2023-10-30T13:18:22.221 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/38' 2023-10-30T13:18:22.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/11' 2023-10-30T13:18:22.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/16' 2023-10-30T13:18:22.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/32' 2023-10-30T13:18:22.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/30' 2023-10-30T13:18:22.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/14' 2023-10-30T13:18:22.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/18' 2023-10-30T13:18:22.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/13' 2023-10-30T13:18:22.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/33' 2023-10-30T13:18:22.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/17' 2023-10-30T13:18:22.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/35' 2023-10-30T13:18:22.224 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:22.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.368+0000 7f31c303a640 1 Processor -- start 2023-10-30T13:18:22.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.368+0000 7f31c303a640 1 -- start start 2023-10-30T13:18:22.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.368+0000 7f31c303a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f31bc1538f0 0x7f31bc159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.368+0000 7f31c303a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0cefa0 0x7f31bc0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.368+0000 7f31c303a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc0ca6a0 con 0x7f31bc15cd50 2023-10-30T13:18:22.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.368+0000 7f31c303a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31bc15d300 con 0x7f31bc1538f0 2023-10-30T13:18:22.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.368+0000 7f31c303a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc15d5e0 con 0x7f31bc0cefa0 2023-10-30T13:18:22.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.368+0000 7f31c1837640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f31bc15cd50 0x7f31bc0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51062/0 (socket says 172.21.15.148:51062) 2023-10-30T13:18:22.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.368+0000 7f31c1837640 1 -- 172.21.15.148:0/3077613549 learned_addr learned my addr 172.21.15.148:0/3077613549 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:22.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c2038640 1 --2- 172.21.15.148:0/3077613549 >> v2:172.21.15.150:3300/0 conn(0x7f31bc1538f0 0x7f31bc159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c2839640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0cefa0 0x7f31bc0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542104875 0 0) 0x7f31bc0ca6a0 con 0x7f31bc15cd50 2023-10-30T13:18:22.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a8003530 con 0x7f31bc15cd50 2023-10-30T13:18:22.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2310618703 0 0) 0x7f31a8003530 con 0x7f31bc15cd50 2023-10-30T13:18:22.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0cefa0 msgr2=0x7f31bc0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:22.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0cefa0 0x7f31bc0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 >> v2:172.21.15.150:3300/0 conn(0x7f31bc1538f0 msgr2=0x7f31bc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:22.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 --2- 172.21.15.148:0/3077613549 >> v2:172.21.15.150:3300/0 conn(0x7f31bc1538f0 0x7f31bc159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc15d8c0 con 0x7f31bc15cd50 2023-10-30T13:18:22.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f31ac001680 con 0x7f31bc15cd50 2023-10-30T13:18:22.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 >> v1:172.21.15.148:6789/0 conn(0x7f31bc15cd50 legacy=0x7f31bc0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003980 0x7f31a8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0cefa0 0x7f31a80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31a8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc15d300 con 0x7f31a8004d10 2023-10-30T13:18:22.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc15d5e0 con 0x7f31a8003980 2023-10-30T13:18:22.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a800b990 con 0x7f31bc0cefa0 2023-10-30T13:18:22.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c2839640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31a8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c2038640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003980 0x7f31a8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c1837640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0cefa0 0x7f31a80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c2839640 1 -- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0cefa0 msgr2=0x7f31a80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:22.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c2839640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0cefa0 0x7f31a80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c2839640 1 -- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003980 msgr2=0x7f31a8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:22.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c2839640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a8003980 0x7f31a8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.372+0000 7f31c2839640 1 -- 172.21.15.148:0/3077613549 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8003530 con 0x7f31a8004d10 2023-10-30T13:18:22.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c2839640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31a8005190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f31b4009fc0 tx=0x7f31b401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709312e34d2f9bfb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:22.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b40a5c70 con 0x7f31a8004d10 2023-10-30T13:18:22.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b400f040 con 0x7f31a8004d10 2023-10-30T13:18:22.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c1036640 1 -- 172.21.15.148:0/3077613549 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b40ad440 con 0x7f31a8004d10 2023-10-30T13:18:22.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c303a640 1 -- 172.21.15.148:0/3077613549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 msgr2=0x7f31a8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c303a640 1 --2- 172.21.15.148:0/3077613549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31a8005190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f31b4009fc0 tx=0x7f31b401cc60 comp rx=0 tx=0).stop 2023-10-30T13:18:22.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c2839640 1 -- 172.21.15.148:0/3077613549 reap_dead start 2023-10-30T13:18:22.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c303a640 1 -- 172.21.15.148:0/3077613549 shutdown_connections 2023-10-30T13:18:22.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c303a640 1 -- 172.21.15.148:0/3077613549 >> 172.21.15.148:0/3077613549 conn(0x7f31bc06a640 msgr2=0x7f31bc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:22.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c303a640 1 -- 172.21.15.148:0/3077613549 shutdown_connections 2023-10-30T13:18:22.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c303a640 1 -- 172.21.15.148:0/3077613549 wait complete. 2023-10-30T13:18:22.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c303a640 1 Processor -- start 2023-10-30T13:18:22.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.376+0000 7f31c303a640 1 -- start start 2023-10-30T13:18:22.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c303a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31bc0b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c303a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31bc0c4a50 0x7f31bc147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c303a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0b3c10 0x7f31bc1586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c303a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0b4690 con 0x7f31a8004d10 2023-10-30T13:18:22.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c303a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0b4820 con 0x7f31bc0c4a50 2023-10-30T13:18:22.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c303a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0b3640 con 0x7f31bc0b3c10 2023-10-30T13:18:22.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2038640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31bc0b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2038640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31bc0b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59354/0 (socket says 172.21.15.148:59354) 2023-10-30T13:18:22.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2038640 1 -- 172.21.15.148:0/4283601591 learned_addr learned my addr 172.21.15.148:0/4283601591 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:22.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2839640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0b3c10 0x7f31bc1586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c1837640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31bc0c4a50 0x7f31bc147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2038640 1 -- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0b3c10 msgr2=0x7f31bc1586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2038640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0b3c10 0x7f31bc1586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2038640 1 -- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31bc0c4a50 msgr2=0x7f31bc147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2038640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31bc0c4a50 0x7f31bc147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2038640 1 -- 172.21.15.148:0/4283601591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0b38e0 con 0x7f31a8004d10 2023-10-30T13:18:22.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c1837640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31bc0c4a50 0x7f31bc147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:22.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c2038640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31bc0b4150 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f31b80119d0 tx=0x7f31b8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:22.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31b2ffd640 1 -- 172.21.15.148:0/4283601591 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b800fb50 con 0x7f31a8004d10 2023-10-30T13:18:22.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31b2ffd640 1 -- 172.21.15.148:0/4283601591 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b800fcb0 con 0x7f31a8004d10 2023-10-30T13:18:22.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31b2ffd640 1 -- 172.21.15.148:0/4283601591 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8014440 con 0x7f31a8004d10 2023-10-30T13:18:22.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31bc1f29d0 con 0x7f31a8004d10 2023-10-30T13:18:22.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31bc148020 con 0x7f31a8004d10 2023-10-30T13:18:22.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31bc1f2e80 con 0x7f31a8004d10 2023-10-30T13:18:22.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.380+0000 7f31b2ffd640 1 -- 172.21.15.148:0/4283601591 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31b80145a0 con 0x7f31a8004d10 2023-10-30T13:18:22.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.384+0000 7f31b2ffd640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31a00664a0 0x7f31a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.384+0000 7f31b2ffd640 1 -- 172.21.15.148:0/4283601591 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f31b8084a60 con 0x7f31a8004d10 2023-10-30T13:18:22.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.384+0000 7f31c1837640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31a00664a0 0x7f31a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.384+0000 7f31c1837640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31a00664a0 0x7f31a0068960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f31bc153280 tx=0x7f31ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:22.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.388+0000 7f31b2ffd640 1 -- 172.21.15.148:0/4283601591 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31b8051cb0 con 0x7f31a8004d10 2023-10-30T13:18:22.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.572+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31bc065cf0 con 0x7f31a8004d10 2023-10-30T13:18:22.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.572+0000 7f31b2ffd640 1 -- 172.21.15.148:0/4283601591 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f31b8022070 con 0x7f31a8004d10 2023-10-30T13:18:22.575 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:22.575 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:21.531094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:22.575 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 35 2023-10-30T13:18:22.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31a00664a0 msgr2=0x7f31a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31a00664a0 0x7f31a0068960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f31bc153280 tx=0x7f31ac009040 comp rx=0 tx=0).stop 2023-10-30T13:18:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 msgr2=0x7f31bc0b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31bc0b4150 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f31b80119d0 tx=0x7f31b8011e90 comp rx=0 tx=0).stop 2023-10-30T13:18:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 shutdown_connections 2023-10-30T13:18:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31a00664a0 0x7f31a0068960 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f31bc0b3c10 0x7f31bc1586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31bc0c4a50 0x7f31bc147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 --2- 172.21.15.148:0/4283601591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31a8004d10 0x7f31bc0b4150 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 >> 172.21.15.148:0/4283601591 conn(0x7f31bc06a640 msgr2=0x7f31bc0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:22.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 shutdown_connections 2023-10-30T13:18:22.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.576+0000 7f31c303a640 1 -- 172.21.15.148:0/4283601591 wait complete. 2023-10-30T13:18:22.595 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T13:18:22.596 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:18:22.596 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:18:22.616 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:18:22.617 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:22.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f271d93e640 1 Processor -- start 2023-10-30T13:18:22.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f271d93e640 1 -- start start 2023-10-30T13:18:22.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f271d93e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2718071110 0x7f271806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f271d93e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2718066cf0 con 0x7f271806f7a0 2023-10-30T13:18:22.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f271d93e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27180c70d0 con 0x7f27180c6bd0 2023-10-30T13:18:22.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f271d93e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27180700c0 con 0x7f2718071110 2023-10-30T13:18:22.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f2716ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f271806f7a0 0x7f2718069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51068/0 (socket says 172.21.15.148:51068) 2023-10-30T13:18:22.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f2716ffd640 1 -- 172.21.15.148:0/2532758527 learned_addr learned my addr 172.21.15.148:0/2532758527 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:22.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f27177fe640 1 --2- 172.21.15.148:0/2532758527 >> v2:172.21.15.148:3300/0 conn(0x7f2718071110 0x7f271806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f27167fc640 1 -- 172.21.15.148:0/2532758527 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440692316 0 0) 0x7f2718066cf0 con 0x7f271806f7a0 2023-10-30T13:18:22.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f27167fc640 1 -- 172.21.15.148:0/2532758527 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2704003530 con 0x7f271806f7a0 2023-10-30T13:18:22.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f27177fe640 1 -- 172.21.15.148:0/2532758527 >> v1:172.21.15.150:6789/0 conn(0x7f27180c6bd0 legacy=0x7f2718158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f27177fe640 1 -- 172.21.15.148:0/2532758527 >> v1:172.21.15.148:6789/0 conn(0x7f271806f7a0 legacy=0x7f2718069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.796+0000 7f27177fe640 1 -- 172.21.15.148:0/2532758527 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27180c7390 con 0x7f2718071110 2023-10-30T13:18:22.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f27177fe640 1 --2- 172.21.15.148:0/2532758527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2718071110 0x7f271806fb80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f27100097c0 tx=0x7f271001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd54cf002074eda3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:22.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f27167fc640 1 -- 172.21.15.148:0/2532758527 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100a8970 con 0x7f2718071110 2023-10-30T13:18:22.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f27167fc640 1 -- 172.21.15.148:0/2532758527 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27100a8ad0 con 0x7f2718071110 2023-10-30T13:18:22.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f27167fc640 1 -- 172.21.15.148:0/2532758527 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100a8e60 con 0x7f2718071110 2023-10-30T13:18:22.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f271d93e640 1 -- 172.21.15.148:0/2532758527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2718071110 msgr2=0x7f271806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f271d93e640 1 --2- 172.21.15.148:0/2532758527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2718071110 0x7f271806fb80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f27100097c0 tx=0x7f271001ec70 comp rx=0 tx=0).stop 2023-10-30T13:18:22.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f271d93e640 1 -- 172.21.15.148:0/2532758527 shutdown_connections 2023-10-30T13:18:22.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f271d93e640 1 --2- 172.21.15.148:0/2532758527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2718071110 0x7f271806fb80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f271d93e640 1 -- 172.21.15.148:0/2532758527 >> 172.21.15.148:0/2532758527 conn(0x7f27180c7910 msgr2=0x7f27180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:22.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f271d93e640 1 -- 172.21.15.148:0/2532758527 shutdown_connections 2023-10-30T13:18:22.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.800+0000 7f271d93e640 1 -- 172.21.15.148:0/2532758527 wait complete. 2023-10-30T13:18:22.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 Processor -- start 2023-10-30T13:18:22.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 -- start start 2023-10-30T13:18:22.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f271806f7a0 0x7f2718154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718071110 0x7f2718152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27180c6bd0 0x7f27180c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27181f1f00 con 0x7f271806f7a0 2023-10-30T13:18:22.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718155090 con 0x7f2718071110 2023-10-30T13:18:22.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27181f2760 con 0x7f27180c6bd0 2023-10-30T13:18:22.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2716ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f271806f7a0 0x7f2718154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2716ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f271806f7a0 0x7f2718154b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59376/0 (socket says 172.21.15.148:59376) 2023-10-30T13:18:22.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2716ffd640 1 -- 172.21.15.148:0/2845076795 learned_addr learned my addr 172.21.15.148:0/2845076795 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:22.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f270e7fc640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718071110 0x7f2718152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f27177fe640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27180c6bd0 0x7f27180c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2716ffd640 1 -- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27180c6bd0 msgr2=0x7f27180c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2716ffd640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27180c6bd0 0x7f27180c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2716ffd640 1 -- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718071110 msgr2=0x7f2718152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2716ffd640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718071110 0x7f2718152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2716ffd640 1 -- 172.21.15.148:0/2845076795 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2710009420 con 0x7f271806f7a0 2023-10-30T13:18:22.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2716ffd640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f271806f7a0 0x7f2718154b50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f270800c420 tx=0x7f270800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:22.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2714ff9640 1 -- 172.21.15.148:0/2845076795 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2708005cf0 con 0x7f271806f7a0 2023-10-30T13:18:22.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2714ff9640 1 -- 172.21.15.148:0/2845076795 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2708005e50 con 0x7f271806f7a0 2023-10-30T13:18:22.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f2714ff9640 1 -- 172.21.15.148:0/2845076795 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27080072d0 con 0x7f271806f7a0 2023-10-30T13:18:22.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2718151f20 con 0x7f271806f7a0 2023-10-30T13:18:22.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.804+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27181f23f0 con 0x7f271806f7a0 2023-10-30T13:18:22.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.808+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271804e170 con 0x7f271806f7a0 2023-10-30T13:18:22.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.808+0000 7f2714ff9640 1 -- 172.21.15.148:0/2845076795 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2708007430 con 0x7f271806f7a0 2023-10-30T13:18:22.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.808+0000 7f2714ff9640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26f80664a0 0x7f26f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.808+0000 7f2714ff9640 1 -- 172.21.15.148:0/2845076795 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f2708083560 con 0x7f271806f7a0 2023-10-30T13:18:22.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.808+0000 7f270e7fc640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26f80664a0 0x7f26f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.808+0000 7f270e7fc640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26f80664a0 0x7f26f8068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f27180c8610 tx=0x7f2700009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:22.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.812+0000 7f2714ff9640 1 -- 172.21.15.148:0/2845076795 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f27080508c0 con 0x7f271806f7a0 2023-10-30T13:18:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.992+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27180c3650 con 0x7f271806f7a0 2023-10-30T13:18:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.992+0000 7f2714ff9640 1 -- 172.21.15.148:0/2845076795 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f2708053ee0 con 0x7f271806f7a0 2023-10-30T13:18:22.995 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:22.995 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:21.531094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 35 2023-10-30T13:18:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26f80664a0 msgr2=0x7f26f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26f80664a0 0x7f26f8068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f27180c8610 tx=0x7f2700009040 comp rx=0 tx=0).stop 2023-10-30T13:18:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f271806f7a0 msgr2=0x7f2718154b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f271806f7a0 0x7f2718154b50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f270800c420 tx=0x7f270800c8e0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 shutdown_connections 2023-10-30T13:18:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26f80664a0 0x7f26f8068960 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27180c6bd0 0x7f27180c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718071110 0x7f2718152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 --2- 172.21.15.148:0/2845076795 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f271806f7a0 0x7f2718154b50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 >> 172.21.15.148:0/2845076795 conn(0x7f27180c7910 msgr2=0x7f27180c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 shutdown_connections 2023-10-30T13:18:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:22.996+0000 7f271d93e640 1 -- 172.21.15.148:0/2845076795 wait complete. 2023-10-30T13:18:23.016 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:18:24.461 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:18:24.466 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:18:25.017 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:25.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.164+0000 7f724113e640 1 Processor -- start 2023-10-30T13:18:25.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.164+0000 7f724113e640 1 -- start start 2023-10-30T13:18:25.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f724113e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f723c0c6f60 0x7f723c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:25.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f724113e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723c0c0a50 con 0x7f723c0c6b80 2023-10-30T13:18:25.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f724113e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723c06f6e0 con 0x7f723c0c05d0 2023-10-30T13:18:25.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f724113e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723c06f9c0 con 0x7f723c0c6f60 2023-10-30T13:18:25.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f723c0c6b80 0x7f723c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51076/0 (socket says 172.21.15.148:51076) 2023-10-30T13:18:25.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723ad9d640 1 -- 172.21.15.148:0/3322526429 learned_addr learned my addr 172.21.15.148:0/3322526429 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:25.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723b59e640 1 --2- 172.21.15.148:0/3322526429 >> v2:172.21.15.150:3300/0 conn(0x7f723c0c6f60 0x7f723c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:25.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245549794 0 0) 0x7f723c0c0a50 con 0x7f723c0c6b80 2023-10-30T13:18:25.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7224003530 con 0x7f723c0c6b80 2023-10-30T13:18:25.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1839975307 0 0) 0x7f7224003530 con 0x7f723c0c6b80 2023-10-30T13:18:25.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 >> v2:172.21.15.150:3300/0 conn(0x7f723c0c6f60 msgr2=0x7f723c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:25.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 --2- 172.21.15.148:0/3322526429 >> v2:172.21.15.150:3300/0 conn(0x7f723c0c6f60 0x7f723c1566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 >> v1:172.21.15.150:6789/0 conn(0x7f723c0c05d0 legacy=0x7f723c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:25.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c06fca0 con 0x7f723c0c6b80 2023-10-30T13:18:25.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f722c001680 con 0x7f723c0c6b80 2023-10-30T13:18:25.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 >> v1:172.21.15.148:6789/0 conn(0x7f723c0c6b80 legacy=0x7f723c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:25.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224003980 0x7f7224003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:25.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f7224004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:25.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7224005190 0x7f722400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:25.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c06f9c0 con 0x7f72240047d0 2023-10-30T13:18:25.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722400c030 con 0x7f7224003980 2023-10-30T13:18:25.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722400c1f0 con 0x7f7224005190 2023-10-30T13:18:25.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723a59c640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f7224004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:25.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723b59e640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7224005190 0x7f722400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:25.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723ad9d640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224003980 0x7f7224003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:25.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723a59c640 1 -- 172.21.15.148:0/3322526429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7224005190 msgr2=0x7f722400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:25.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723a59c640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7224005190 0x7f722400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723a59c640 1 -- 172.21.15.148:0/3322526429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224003980 msgr2=0x7f7224003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:25.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723a59c640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224003980 0x7f7224003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723a59c640 1 -- 172.21.15.148:0/3322526429 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7224003530 con 0x7f72240047d0 2023-10-30T13:18:25.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.168+0000 7f723b59e640 1 -- 172.21.15.148:0/3322526429 reap_dead start 2023-10-30T13:18:25.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f723a59c640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f7224004c50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7230001360 tx=0x7f723001e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc4f65a5b014de3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:25.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72300052d0 con 0x7f72240047d0 2023-10-30T13:18:25.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7230005450 con 0x7f72240047d0 2023-10-30T13:18:25.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f7239d9b640 1 -- 172.21.15.148:0/3322526429 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7230003840 con 0x7f72240047d0 2023-10-30T13:18:25.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f724113e640 1 -- 172.21.15.148:0/3322526429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 msgr2=0x7f7224004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:25.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f724113e640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f7224004c50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7230001360 tx=0x7f723001e5d0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f724113e640 1 -- 172.21.15.148:0/3322526429 shutdown_connections 2023-10-30T13:18:25.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f724113e640 1 --2- 172.21.15.148:0/3322526429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f7224004c50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f724113e640 1 -- 172.21.15.148:0/3322526429 >> 172.21.15.148:0/3322526429 conn(0x7f723c0c7910 msgr2=0x7f723c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:25.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f724113e640 1 -- 172.21.15.148:0/3322526429 shutdown_connections 2023-10-30T13:18:25.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.172+0000 7f724113e640 1 -- 172.21.15.148:0/3322526429 wait complete. 2023-10-30T13:18:25.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 Processor -- start 2023-10-30T13:18:25.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 -- start start 2023-10-30T13:18:25.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f723c13dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:25.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723c0c9240 0x7f723c1587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:25.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f723c158cf0 0x7f723c0c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:25.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c151f60 con 0x7f72240047d0 2023-10-30T13:18:25.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c159a30 con 0x7f723c0c9240 2023-10-30T13:18:25.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723c157c50 con 0x7f723c158cf0 2023-10-30T13:18:25.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f723c13dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:25.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f723c13dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59398/0 (socket says 172.21.15.148:59398) 2023-10-30T13:18:25.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723c0c9240 0x7f723c1587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:25.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723ad9d640 1 -- 172.21.15.148:0/3380997508 learned_addr learned my addr 172.21.15.148:0/3380997508 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:25.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723b59e640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f723c158cf0 0x7f723c0c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:25.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723ad9d640 1 -- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f723c158cf0 msgr2=0x7f723c0c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:25.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723ad9d640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f723c158cf0 0x7f723c0c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723ad9d640 1 -- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723c0c9240 msgr2=0x7f723c1587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:25.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723ad9d640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723c0c9240 0x7f723c1587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723ad9d640 1 -- 172.21.15.148:0/3380997508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c1530e0 con 0x7f72240047d0 2023-10-30T13:18:25.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f723ad9d640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f723c13dcc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f722c0118a0 tx=0x7f722c011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:25.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f72237fe640 1 -- 172.21.15.148:0/3380997508 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c00fb60 con 0x7f72240047d0 2023-10-30T13:18:25.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f72237fe640 1 -- 172.21.15.148:0/3380997508 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c00fce0 con 0x7f72240047d0 2023-10-30T13:18:25.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f72237fe640 1 -- 172.21.15.148:0/3380997508 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c015650 con 0x7f72240047d0 2023-10-30T13:18:25.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723c152de0 con 0x7f72240047d0 2023-10-30T13:18:25.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723c153570 con 0x7f72240047d0 2023-10-30T13:18:25.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.176+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723c04e170 con 0x7f72240047d0 2023-10-30T13:18:25.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.180+0000 7f72237fe640 1 -- 172.21.15.148:0/3380997508 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f722c0157b0 con 0x7f72240047d0 2023-10-30T13:18:25.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.180+0000 7f72237fe640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72280664d0 0x7f7228068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:25.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.180+0000 7f72237fe640 1 -- 172.21.15.148:0/3380997508 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f722c087230 con 0x7f72240047d0 2023-10-30T13:18:25.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.180+0000 7f723a59c640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72280664d0 0x7f7228068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:25.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.180+0000 7f723a59c640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72280664d0 0x7f7228068990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f723c06c9a0 tx=0x7f72300011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:25.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.184+0000 7f72237fe640 1 -- 172.21.15.148:0/3380997508 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f722c0544e0 con 0x7f72240047d0 2023-10-30T13:18:25.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.364+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f723c0c54f0 con 0x7f72240047d0 2023-10-30T13:18:25.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.364+0000 7f72237fe640 1 -- 172.21.15.148:0/3380997508 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f722c057b00 con 0x7f72240047d0 2023-10-30T13:18:25.367 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:25.367 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:21.531094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:25.368 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 35 2023-10-30T13:18:25.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72280664d0 msgr2=0x7f7228068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:25.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72280664d0 0x7f7228068990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f723c06c9a0 tx=0x7f72300011a0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 msgr2=0x7f723c13dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:25.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f723c13dcc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f722c0118a0 tx=0x7f722c011d60 comp rx=0 tx=0).stop 2023-10-30T13:18:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 shutdown_connections 2023-10-30T13:18:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72280664d0 0x7f7228068990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f723c158cf0 0x7f723c0c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723c0c9240 0x7f723c1587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 --2- 172.21.15.148:0/3380997508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72240047d0 0x7f723c13dcc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:25.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 >> 172.21.15.148:0/3380997508 conn(0x7f723c0c7910 msgr2=0x7f723c138ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:25.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 shutdown_connections 2023-10-30T13:18:25.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:25.368+0000 7f724113e640 1 -- 172.21.15.148:0/3380997508 wait complete. 2023-10-30T13:18:25.388 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:18:27.390 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:27.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.540+0000 7fbc88246640 1 Processor -- start 2023-10-30T13:18:27.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.540+0000 7fbc88246640 1 -- start start 2023-10-30T13:18:27.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.540+0000 7fbc88246640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 0x7fbc800c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:27.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.540+0000 7fbc88246640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbc8015e700 0x7fbc8006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:27.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc88246640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc800c5810 con 0x7fbc801580e0 2023-10-30T13:18:27.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc88246640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc80070b60 con 0x7fbc8015e700 2023-10-30T13:18:27.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc88246640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8015eb90 con 0x7fbc80070740 2023-10-30T13:18:27.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc867e3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbc8015e700 0x7fbc8006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:27.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc85fe2640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbc801580e0 0x7fbc80069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51086/0 (socket says 172.21.15.148:51086) 2023-10-30T13:18:27.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc85fe2640 1 -- 172.21.15.148:0/2006526049 learned_addr learned my addr 172.21.15.148:0/2006526049 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:27.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc857e1640 1 --2- 172.21.15.148:0/2006526049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 0x7fbc800c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:27.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc867e3640 1 -- 172.21.15.148:0/2006526049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 msgr2=0x7fbc800c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:27.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc867e3640 1 --2- 172.21.15.148:0/2006526049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 0x7fbc800c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc867e3640 1 -- 172.21.15.148:0/2006526049 >> v1:172.21.15.148:6789/0 conn(0x7fbc801580e0 legacy=0x7fbc80069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:27.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc867e3640 1 -- 172.21.15.148:0/2006526049 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc800c6bb0 con 0x7fbc8015e700 2023-10-30T13:18:27.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc84fe0640 1 -- 172.21.15.148:0/2006526049 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664298968 0 0) 0x7fbc800c5810 con 0x7fbc801580e0 2023-10-30T13:18:27.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc867e3640 1 --2- 172.21.15.148:0/2006526049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc8015e700 0x7fbc8006a5a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009d90 tx=0x7fbc7801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef22505faf38660 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:27.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc84fe0640 1 -- 172.21.15.148:0/2006526049 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc780a59e0 con 0x7fbc8015e700 2023-10-30T13:18:27.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc84fe0640 1 -- 172.21.15.148:0/2006526049 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc780a5b40 con 0x7fbc8015e700 2023-10-30T13:18:27.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.544+0000 7fbc84fe0640 1 -- 172.21.15.148:0/2006526049 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc780ad440 con 0x7fbc8015e700 2023-10-30T13:18:27.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 -- 172.21.15.148:0/2006526049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc8015e700 msgr2=0x7fbc8006a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:27.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 --2- 172.21.15.148:0/2006526049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc8015e700 0x7fbc8006a5a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009d90 tx=0x7fbc7801c920 comp rx=0 tx=0).stop 2023-10-30T13:18:27.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 -- 172.21.15.148:0/2006526049 shutdown_connections 2023-10-30T13:18:27.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 --2- 172.21.15.148:0/2006526049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc8015e700 0x7fbc8006a5a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 --2- 172.21.15.148:0/2006526049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 0x7fbc800c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 -- 172.21.15.148:0/2006526049 >> 172.21.15.148:0/2006526049 conn(0x7fbc800c7910 msgr2=0x7fbc800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:27.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 -- 172.21.15.148:0/2006526049 shutdown_connections 2023-10-30T13:18:27.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 -- 172.21.15.148:0/2006526049 wait complete. 2023-10-30T13:18:27.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 Processor -- start 2023-10-30T13:18:27.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 -- start start 2023-10-30T13:18:27.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 0x7fbc800c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:27.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc801580e0 0x7fbc800c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:27.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8015e700 0x7fbc800c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:27.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c3980 con 0x7fbc801580e0 2023-10-30T13:18:27.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c3310 con 0x7fbc8015e700 2023-10-30T13:18:27.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc88246640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c20b0 con 0x7fbc80070740 2023-10-30T13:18:27.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc857e1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc801580e0 0x7fbc800c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:27.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc857e1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc801580e0 0x7fbc800c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59416/0 (socket says 172.21.15.148:59416) 2023-10-30T13:18:27.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc857e1640 1 -- 172.21.15.148:0/3083171244 learned_addr learned my addr 172.21.15.148:0/3083171244 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:27.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc85fe2640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 0x7fbc800c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:27.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.548+0000 7fbc867e3640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8015e700 0x7fbc800c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:27.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc857e1640 1 -- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 msgr2=0x7fbc800c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:27.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc857e1640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 0x7fbc800c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc857e1640 1 -- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8015e700 msgr2=0x7fbc800c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:27.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc857e1640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8015e700 0x7fbc800c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc857e1640 1 -- 172.21.15.148:0/3083171244 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78009a70 con 0x7fbc801580e0 2023-10-30T13:18:27.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc857e1640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc801580e0 0x7fbc800c3e10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000d800 tx=0x7fbc7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:27.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc767fc640 1 -- 172.21.15.148:0/3083171244 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7000b6c0 con 0x7fbc801580e0 2023-10-30T13:18:27.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc767fc640 1 -- 172.21.15.148:0/3083171244 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7000b820 con 0x7fbc801580e0 2023-10-30T13:18:27.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc767fc640 1 -- 172.21.15.148:0/3083171244 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7000bb90 con 0x7fbc801580e0 2023-10-30T13:18:27.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8014dd40 con 0x7fbc801580e0 2023-10-30T13:18:27.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8015f120 con 0x7fbc801580e0 2023-10-30T13:18:27.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc800c6000 con 0x7fbc801580e0 2023-10-30T13:18:27.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.552+0000 7fbc767fc640 1 -- 172.21.15.148:0/3083171244 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbc7000bcf0 con 0x7fbc801580e0 2023-10-30T13:18:27.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.556+0000 7fbc767fc640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc640664a0 0x7fbc64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:27.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.556+0000 7fbc767fc640 1 -- 172.21.15.148:0/3083171244 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fbc700846a0 con 0x7fbc801580e0 2023-10-30T13:18:27.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.556+0000 7fbc85fe2640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc640664a0 0x7fbc64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:27.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.556+0000 7fbc85fe2640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc640664a0 0x7fbc64068960 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c000f70 tx=0x7fbc7c007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:27.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.560+0000 7fbc767fc640 1 -- 172.21.15.148:0/3083171244 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbc800c6000 con 0x7fbc801580e0 2023-10-30T13:18:27.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc800c3b30 con 0x7fbc801580e0 2023-10-30T13:18:27.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc767fc640 1 -- 172.21.15.148:0/3083171244 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fbc800c3b30 con 0x7fbc801580e0 2023-10-30T13:18:27.745 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:27.746 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:21.531094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:27.746 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 35 2023-10-30T13:18:27.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc640664a0 msgr2=0x7fbc64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:27.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc640664a0 0x7fbc64068960 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c000f70 tx=0x7fbc7c007b60 comp rx=0 tx=0).stop 2023-10-30T13:18:27.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc801580e0 msgr2=0x7fbc800c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:27.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc801580e0 0x7fbc800c3e10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000d800 tx=0x7fbc7000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 shutdown_connections 2023-10-30T13:18:27.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc640664a0 0x7fbc64068960 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbc801580e0 0x7fbc800c3e10 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8015e700 0x7fbc800c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 --2- 172.21.15.148:0/3083171244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbc80070740 0x7fbc800c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:27.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 >> 172.21.15.148:0/3083171244 conn(0x7fbc800c7910 msgr2=0x7fbc80155910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:27.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 shutdown_connections 2023-10-30T13:18:27.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:27.744+0000 7fbc88246640 1 -- 172.21.15.148:0/3083171244 wait complete. 2023-10-30T13:18:27.767 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:18:29.767 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:29.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb33730f640 1 Processor -- start 2023-10-30T13:18:29.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb33730f640 1 -- start start 2023-10-30T13:18:29.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb33730f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6b80 0x7fb3300bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:29.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb33730f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3300c0a50 con 0x7fb3300c6f60 2023-10-30T13:18:29.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb33730f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33006f6e0 con 0x7fb33013f4a0 2023-10-30T13:18:29.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb33730f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33006f9c0 con 0x7fb3300c6b80 2023-10-30T13:18:29.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb3348aa640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb3300c6f60 0x7fb33013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51100/0 (socket says 172.21.15.148:51100) 2023-10-30T13:18:29.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb3348aa640 1 -- 172.21.15.148:0/1450975919 learned_addr learned my addr 172.21.15.148:0/1450975919 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:29.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb3350ab640 1 --2- 172.21.15.148:0/1450975919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6b80 0x7fb3300bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:29.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb327fff640 1 -- 172.21.15.148:0/1450975919 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810733949 0 0) 0x7fb3300c0a50 con 0x7fb3300c6f60 2023-10-30T13:18:29.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb327fff640 1 -- 172.21.15.148:0/1450975919 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb31c003530 con 0x7fb3300c6f60 2023-10-30T13:18:29.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb3350ab640 1 -- 172.21.15.148:0/1450975919 >> v1:172.21.15.150:6789/0 conn(0x7fb33013f4a0 legacy=0x7fb3301566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:29.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb327fff640 1 -- 172.21.15.148:0/1450975919 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547171458 0 0) 0x7fb33006f6e0 con 0x7fb33013f4a0 2023-10-30T13:18:29.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb3350ab640 1 -- 172.21.15.148:0/1450975919 >> v1:172.21.15.148:6789/0 conn(0x7fb3300c6f60 legacy=0x7fb33013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:29.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.916+0000 7fb3350ab640 1 -- 172.21.15.148:0/1450975919 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33006fca0 con 0x7fb3300c6b80 2023-10-30T13:18:29.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb3350ab640 1 --2- 172.21.15.148:0/1450975919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6b80 0x7fb3300bfec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0097f0 tx=0x7fb32c01c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c50e3e6840109cae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:29.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb327fff640 1 -- 172.21.15.148:0/1450975919 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320001680 con 0x7fb3300c6b80 2023-10-30T13:18:29.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb327fff640 1 -- 172.21.15.148:0/1450975919 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c0a7050 con 0x7fb3300c6b80 2023-10-30T13:18:29.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb327fff640 1 -- 172.21.15.148:0/1450975919 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0ab600 con 0x7fb3300c6b80 2023-10-30T13:18:29.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb33730f640 1 -- 172.21.15.148:0/1450975919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6b80 msgr2=0x7fb3300bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:29.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb33730f640 1 --2- 172.21.15.148:0/1450975919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6b80 0x7fb3300bfec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0097f0 tx=0x7fb32c01c300 comp rx=0 tx=0).stop 2023-10-30T13:18:29.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb33730f640 1 -- 172.21.15.148:0/1450975919 shutdown_connections 2023-10-30T13:18:29.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb33730f640 1 --2- 172.21.15.148:0/1450975919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6b80 0x7fb3300bfec0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:29.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb33730f640 1 -- 172.21.15.148:0/1450975919 >> 172.21.15.148:0/1450975919 conn(0x7fb3300c7910 msgr2=0x7fb3300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:29.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb33730f640 1 -- 172.21.15.148:0/1450975919 shutdown_connections 2023-10-30T13:18:29.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.920+0000 7fb33730f640 1 -- 172.21.15.148:0/1450975919 wait complete. 2023-10-30T13:18:29.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 Processor -- start 2023-10-30T13:18:29.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 -- start start 2023-10-30T13:18:29.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3300c6b80 0x7fb33013cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:29.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6f60 0x7fb33013d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:29.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb33013f4a0 0x7fb3300ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:29.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33013d120 con 0x7fb33013f4a0 2023-10-30T13:18:29.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3300b0e70 con 0x7fb3300c6b80 2023-10-30T13:18:29.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33013e730 con 0x7fb3300c6f60 2023-10-30T13:18:29.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3358ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb33013f4a0 0x7fb3300ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:29.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3350ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3300c6b80 0x7fb33013cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:29.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3358ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb33013f4a0 0x7fb3300ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59428/0 (socket says 172.21.15.148:59428) 2023-10-30T13:18:29.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3358ac640 1 -- 172.21.15.148:0/562681076 learned_addr learned my addr 172.21.15.148:0/562681076 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:29.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3348aa640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6f60 0x7fb33013d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:29.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3358ac640 1 -- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6f60 msgr2=0x7fb33013d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:29.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3358ac640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6f60 0x7fb33013d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:29.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3358ac640 1 -- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3300c6b80 msgr2=0x7fb33013cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:29.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3358ac640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3300c6b80 0x7fb33013cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:29.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3358ac640 1 -- 172.21.15.148:0/562681076 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c009420 con 0x7fb33013f4a0 2023-10-30T13:18:29.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb3358ac640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb33013f4a0 0x7fb3300ad8d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb32800c6d0 tx=0x7fb32800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:29.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb325ffb640 1 -- 172.21.15.148:0/562681076 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328003b80 con 0x7fb33013f4a0 2023-10-30T13:18:29.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb325ffb640 1 -- 172.21.15.148:0/562681076 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb328003ce0 con 0x7fb33013f4a0 2023-10-30T13:18:29.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3300b0b90 con 0x7fb33013f4a0 2023-10-30T13:18:29.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb325ffb640 1 -- 172.21.15.148:0/562681076 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3280052b0 con 0x7fb33013f4a0 2023-10-30T13:18:29.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.924+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb330154720 con 0x7fb33013f4a0 2023-10-30T13:18:29.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.928+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb330156100 con 0x7fb33013f4a0 2023-10-30T13:18:29.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.928+0000 7fb325ffb640 1 -- 172.21.15.148:0/562681076 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb328005620 con 0x7fb33013f4a0 2023-10-30T13:18:29.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.928+0000 7fb325ffb640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3140664d0 0x7fb314068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:29.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.928+0000 7fb325ffb640 1 -- 172.21.15.148:0/562681076 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb328083a10 con 0x7fb33013f4a0 2023-10-30T13:18:29.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.928+0000 7fb3350ab640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3140664d0 0x7fb314068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:29.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.928+0000 7fb3350ab640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3140664d0 0x7fb314068990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0ad000 tx=0x7fb32c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:29.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:29.932+0000 7fb325ffb640 1 -- 172.21.15.148:0/562681076 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb330156100 con 0x7fb33013f4a0 2023-10-30T13:18:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.112+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb330156100 con 0x7fb33013f4a0 2023-10-30T13:18:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.112+0000 7fb325ffb640 1 -- 172.21.15.148:0/562681076 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fb330156100 con 0x7fb33013f4a0 2023-10-30T13:18:30.115 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:30.116 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:21.531094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 35 2023-10-30T13:18:30.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3140664d0 msgr2=0x7fb314068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:30.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3140664d0 0x7fb314068990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0ad000 tx=0x7fb32c0a6000 comp rx=0 tx=0).stop 2023-10-30T13:18:30.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb33013f4a0 msgr2=0x7fb3300ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:30.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb33013f4a0 0x7fb3300ad8d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb32800c6d0 tx=0x7fb32800cb90 comp rx=0 tx=0).stop 2023-10-30T13:18:30.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 shutdown_connections 2023-10-30T13:18:30.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3140664d0 0x7fb314068990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:30.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb33013f4a0 0x7fb3300ad8d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:30.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3300c6f60 0x7fb33013d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:30.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 --2- 172.21.15.148:0/562681076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3300c6b80 0x7fb33013cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:30.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 >> 172.21.15.148:0/562681076 conn(0x7fb3300c7910 msgr2=0x7fb33006a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:30.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 shutdown_connections 2023-10-30T13:18:30.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:30.116+0000 7fb33730f640 1 -- 172.21.15.148:0/562681076 wait complete. 2023-10-30T13:18:30.136 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:18:32.137 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb7a8d6640 1 Processor -- start 2023-10-30T13:18:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb7a8d6640 1 -- start start 2023-10-30T13:18:32.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb7a8d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbb741538a0 0x7fbb7415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:32.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb7a8d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbb740cf160 0x7fbb740cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:32.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb7a8d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740cb800 con 0x7fbb740ca0e0 2023-10-30T13:18:32.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb7a8d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7415dc30 con 0x7fbb740cf160 2023-10-30T13:18:32.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb7a8d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb740cb330 con 0x7fbb741538a0 2023-10-30T13:18:32.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb78e73640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbb740cf160 0x7fbb740cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:32.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb737fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbb740ca0e0 0x7fbb7415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48662/0 (socket says 172.21.15.148:48662) 2023-10-30T13:18:32.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb737fe640 1 -- 172.21.15.148:0/32456881 learned_addr learned my addr 172.21.15.148:0/32456881 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:32.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb73fff640 1 --2- 172.21.15.148:0/32456881 >> v2:172.21.15.150:3300/0 conn(0x7fbb741538a0 0x7fbb7415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:32.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb78e73640 1 -- 172.21.15.148:0/32456881 >> v2:172.21.15.150:3300/0 conn(0x7fbb741538a0 msgr2=0x7fbb7415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:32.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb78e73640 1 --2- 172.21.15.148:0/32456881 >> v2:172.21.15.150:3300/0 conn(0x7fbb741538a0 0x7fbb7415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb78e73640 1 -- 172.21.15.148:0/32456881 >> v1:172.21.15.148:6789/0 conn(0x7fbb740ca0e0 legacy=0x7fbb7415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:32.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb78e73640 1 -- 172.21.15.148:0/32456881 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740cad70 con 0x7fbb740cf160 2023-10-30T13:18:32.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.284+0000 7fbb72ffd640 1 -- 172.21.15.148:0/32456881 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086024490 0 0) 0x7fbb740cb800 con 0x7fbb740ca0e0 2023-10-30T13:18:32.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb78e73640 1 --2- 172.21.15.148:0/32456881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb740cf160 0x7fbb740cf590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbb740cfd10 tx=0x7fbb6c01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45a616ba1fd95e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:32.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb72ffd640 1 -- 172.21.15.148:0/32456881 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0b0440 con 0x7fbb740cf160 2023-10-30T13:18:32.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb72ffd640 1 -- 172.21.15.148:0/32456881 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c0b05a0 con 0x7fbb740cf160 2023-10-30T13:18:32.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb72ffd640 1 -- 172.21.15.148:0/32456881 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0b0930 con 0x7fbb740cf160 2023-10-30T13:18:32.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/32456881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb740cf160 msgr2=0x7fbb740cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:32.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb7a8d6640 1 --2- 172.21.15.148:0/32456881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb740cf160 0x7fbb740cf590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbb740cfd10 tx=0x7fbb6c01f0f0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/32456881 shutdown_connections 2023-10-30T13:18:32.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb7a8d6640 1 --2- 172.21.15.148:0/32456881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb740cf160 0x7fbb740cf590 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb7a8d6640 1 --2- 172.21.15.148:0/32456881 >> v2:172.21.15.150:3300/0 conn(0x7fbb741538a0 0x7fbb7415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/32456881 >> 172.21.15.148:0/32456881 conn(0x7fbb7406bfe0 msgr2=0x7fbb7406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:32.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/32456881 shutdown_connections 2023-10-30T13:18:32.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.288+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/32456881 wait complete. 2023-10-30T13:18:32.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 Processor -- start 2023-10-30T13:18:32.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 -- start start 2023-10-30T13:18:32.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb740ca0e0 0x7fbb740cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:32.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb740cf160 0x7fbb741e3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:32.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb741538a0 0x7fbb741e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:32.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb741e31d0 con 0x7fbb741538a0 2023-10-30T13:18:32.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74148450 con 0x7fbb740cf160 2023-10-30T13:18:32.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74148730 con 0x7fbb740ca0e0 2023-10-30T13:18:32.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb78e73640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb741538a0 0x7fbb741e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:32.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb78e73640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb741538a0 0x7fbb741e5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34680/0 (socket says 172.21.15.148:34680) 2023-10-30T13:18:32.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb78e73640 1 -- 172.21.15.148:0/3251806094 learned_addr learned my addr 172.21.15.148:0/3251806094 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:32.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb737fe640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb740cf160 0x7fbb741e3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:32.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb73fff640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb740ca0e0 0x7fbb740cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:32.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb78e73640 1 -- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb740ca0e0 msgr2=0x7fbb740cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:32.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb78e73640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb740ca0e0 0x7fbb740cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb78e73640 1 -- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb740cf160 msgr2=0x7fbb741e3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:32.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb78e73640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb740cf160 0x7fbb741e3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb78e73640 1 -- 172.21.15.148:0/3251806094 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c009a70 con 0x7fbb741538a0 2023-10-30T13:18:32.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb73fff640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb740ca0e0 0x7fbb740cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:32.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb78e73640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb741538a0 0x7fbb741e5260 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c01bd00 tx=0x7fbb6c01be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:32.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb70ff9640 1 -- 172.21.15.148:0/3251806094 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0b0440 con 0x7fbb741538a0 2023-10-30T13:18:32.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb70ff9640 1 -- 172.21.15.148:0/3251806094 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c0bc650 con 0x7fbb741538a0 2023-10-30T13:18:32.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb741e2b80 con 0x7fbb741538a0 2023-10-30T13:18:32.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb70ff9640 1 -- 172.21.15.148:0/3251806094 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0bca00 con 0x7fbb741538a0 2023-10-30T13:18:32.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.292+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb741e57a0 con 0x7fbb741538a0 2023-10-30T13:18:32.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.296+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb740b2a50 con 0x7fbb741538a0 2023-10-30T13:18:32.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.296+0000 7fbb70ff9640 1 -- 172.21.15.148:0/3251806094 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb6c0bcb60 con 0x7fbb741538a0 2023-10-30T13:18:32.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.296+0000 7fbb70ff9640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbb600664a0 0x7fbb60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:32.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.296+0000 7fbb70ff9640 1 -- 172.21.15.148:0/3251806094 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fbb6c122b60 con 0x7fbb741538a0 2023-10-30T13:18:32.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.296+0000 7fbb73fff640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbb600664a0 0x7fbb60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:32.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.296+0000 7fbb73fff640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbb600664a0 0x7fbb60068960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009bb0 tx=0x7fbb64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:32.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.300+0000 7fbb70ff9640 1 -- 172.21.15.148:0/3251806094 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb6c0efe90 con 0x7fbb741538a0 2023-10-30T13:18:32.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.480+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb74159850 con 0x7fbb741538a0 2023-10-30T13:18:32.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb70ff9640 1 -- 172.21.15.148:0/3251806094 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fbb6c009450 con 0x7fbb741538a0 2023-10-30T13:18:32.484 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:32.484 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:21.531094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:32.485 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 35 2023-10-30T13:18:32.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbb600664a0 msgr2=0x7fbb60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:32.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbb600664a0 0x7fbb60068960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009bb0 tx=0x7fbb64008040 comp rx=0 tx=0).stop 2023-10-30T13:18:32.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb741538a0 msgr2=0x7fbb741e5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:32.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb741538a0 0x7fbb741e5260 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c01bd00 tx=0x7fbb6c01be90 comp rx=0 tx=0).stop 2023-10-30T13:18:32.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 shutdown_connections 2023-10-30T13:18:32.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbb600664a0 0x7fbb60068960 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbb741538a0 0x7fbb741e5260 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb740cf160 0x7fbb741e3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 --2- 172.21.15.148:0/3251806094 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbb740ca0e0 0x7fbb740cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:32.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 >> 172.21.15.148:0/3251806094 conn(0x7fbb7406bfe0 msgr2=0x7fbb74066390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:32.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 shutdown_connections 2023-10-30T13:18:32.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:32.484+0000 7fbb7a8d6640 1 -- 172.21.15.148:0/3251806094 wait complete. 2023-10-30T13:18:32.506 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:18:34.507 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:34.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c6cad640 1 Processor -- start 2023-10-30T13:18:34.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c6cad640 1 -- start start 2023-10-30T13:18:34.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c6cad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c01538f0 0x7f33c0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:34.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c6cad640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f33c015cd50 0x7f33c00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:34.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c6cad640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c015d300 con 0x7f33c00cefa0 2023-10-30T13:18:34.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c6cad640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c015d5e0 con 0x7f33c015cd50 2023-10-30T13:18:34.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c6cad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c015d8c0 con 0x7f33c01538f0 2023-10-30T13:18:34.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c54aa640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f33c015cd50 0x7f33c00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:34.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c54aa640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f33c015cd50 0x7f33c00c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34688/0 (socket says 172.21.15.148:34688) 2023-10-30T13:18:34.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c54aa640 1 -- 172.21.15.148:0/1453982455 learned_addr learned my addr 172.21.15.148:0/1453982455 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:34.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c5cab640 1 --2- 172.21.15.148:0/1453982455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c01538f0 0x7f33c0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:34.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c54aa640 1 -- 172.21.15.148:0/1453982455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c01538f0 msgr2=0x7f33c0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:34.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c54aa640 1 --2- 172.21.15.148:0/1453982455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c01538f0 0x7f33c0159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c54aa640 1 -- 172.21.15.148:0/1453982455 >> v1:172.21.15.150:6789/0 conn(0x7f33c00cefa0 legacy=0x7f33c00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:34.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.656+0000 7f33c54aa640 1 -- 172.21.15.148:0/1453982455 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c015dba0 con 0x7f33c015cd50 2023-10-30T13:18:34.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c54aa640 1 --2- 172.21.15.148:0/1453982455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c015cd50 0x7f33c00c9f90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009870 tx=0x7f33b001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a4d12ce6b07b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:34.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c4ca9640 1 -- 172.21.15.148:0/1453982455 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b00a8a30 con 0x7f33c015cd50 2023-10-30T13:18:34.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c4ca9640 1 -- 172.21.15.148:0/1453982455 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b00a8b90 con 0x7f33c015cd50 2023-10-30T13:18:34.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c4ca9640 1 -- 172.21.15.148:0/1453982455 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b00b0440 con 0x7f33c015cd50 2023-10-30T13:18:34.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c6cad640 1 -- 172.21.15.148:0/1453982455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c015cd50 msgr2=0x7f33c00c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:34.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c6cad640 1 --2- 172.21.15.148:0/1453982455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c015cd50 0x7f33c00c9f90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009870 tx=0x7f33b001ed30 comp rx=0 tx=0).stop 2023-10-30T13:18:34.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c6cad640 1 -- 172.21.15.148:0/1453982455 shutdown_connections 2023-10-30T13:18:34.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c6cad640 1 --2- 172.21.15.148:0/1453982455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c015cd50 0x7f33c00c9f90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c6cad640 1 --2- 172.21.15.148:0/1453982455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c01538f0 0x7f33c0159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c6cad640 1 -- 172.21.15.148:0/1453982455 >> 172.21.15.148:0/1453982455 conn(0x7f33c006a640 msgr2=0x7f33c0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:34.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c6cad640 1 -- 172.21.15.148:0/1453982455 shutdown_connections 2023-10-30T13:18:34.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.660+0000 7f33c6cad640 1 -- 172.21.15.148:0/1453982455 wait complete. 2023-10-30T13:18:34.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 Processor -- start 2023-10-30T13:18:34.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 -- start start 2023-10-30T13:18:34.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c00cefa0 0x7f33c01484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:34.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c01538f0 0x7f33c00ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:34.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33c015cd50 0x7f33c01f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:34.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c01f26e0 con 0x7f33c01538f0 2023-10-30T13:18:34.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c014d750 con 0x7f33c015cd50 2023-10-30T13:18:34.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c0147f50 con 0x7f33c00cefa0 2023-10-30T13:18:34.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c54aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c01538f0 0x7f33c00ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:34.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c54aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c01538f0 0x7f33c00ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34690/0 (socket says 172.21.15.148:34690) 2023-10-30T13:18:34.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c54aa640 1 -- 172.21.15.148:0/1005969986 learned_addr learned my addr 172.21.15.148:0/1005969986 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:34.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c5cab640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c00cefa0 0x7f33c01484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:34.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c64ac640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33c015cd50 0x7f33c01f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:34.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c54aa640 1 -- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c00cefa0 msgr2=0x7f33c01484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:34.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c54aa640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c00cefa0 0x7f33c01484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c54aa640 1 -- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33c015cd50 msgr2=0x7f33c01f2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:34.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c54aa640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33c015cd50 0x7f33c01f2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c54aa640 1 -- 172.21.15.148:0/1005969986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b00094d0 con 0x7f33c01538f0 2023-10-30T13:18:34.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c54aa640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c01538f0 0x7f33c00ce1c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f33b00b3040 tx=0x7f33b001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:34.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33b67fc640 1 -- 172.21.15.148:0/1005969986 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b00b9620 con 0x7f33c01538f0 2023-10-30T13:18:34.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33b67fc640 1 -- 172.21.15.148:0/1005969986 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b00b97a0 con 0x7f33c01538f0 2023-10-30T13:18:34.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33b67fc640 1 -- 172.21.15.148:0/1005969986 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b00b9b50 con 0x7f33c01538f0 2023-10-30T13:18:34.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c01f23b0 con 0x7f33c01538f0 2023-10-30T13:18:34.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c01481f0 con 0x7f33c01538f0 2023-10-30T13:18:34.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.664+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c004e170 con 0x7f33c01538f0 2023-10-30T13:18:34.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.668+0000 7f33b67fc640 1 -- 172.21.15.148:0/1005969986 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33b00b7070 con 0x7f33c01538f0 2023-10-30T13:18:34.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.668+0000 7f33b67fc640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f33a40664a0 0x7f33a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:34.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.668+0000 7f33b67fc640 1 -- 172.21.15.148:0/1005969986 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f33b0121ff0 con 0x7f33c01538f0 2023-10-30T13:18:34.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.668+0000 7f33c5cab640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f33a40664a0 0x7f33a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:34.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.668+0000 7f33c5cab640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f33a40664a0 0x7f33a4068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00b9d0 tx=0x7f33bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:34.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.672+0000 7f33b67fc640 1 -- 172.21.15.148:0/1005969986 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33b00ef1a0 con 0x7f33c01538f0 2023-10-30T13:18:34.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.852+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33c00b2290 con 0x7f33c01538f0 2023-10-30T13:18:34.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.852+0000 7f33b67fc640 1 -- 172.21.15.148:0/1005969986 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f33b00b5070 con 0x7f33c01538f0 2023-10-30T13:18:34.854 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:34.854 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:21.531094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:34.855 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 35 2023-10-30T13:18:34.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f33a40664a0 msgr2=0x7f33a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:34.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f33a40664a0 0x7f33a4068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00b9d0 tx=0x7f33bc008040 comp rx=0 tx=0).stop 2023-10-30T13:18:34.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c01538f0 msgr2=0x7f33c00ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:34.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c01538f0 0x7f33c00ce1c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f33b00b3040 tx=0x7f33b001bad0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 shutdown_connections 2023-10-30T13:18:34.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f33a40664a0 0x7f33a4068960 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f33c01538f0 0x7f33c00ce1c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33c015cd50 0x7f33c01f2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 --2- 172.21.15.148:0/1005969986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f33c00cefa0 0x7f33c01484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 >> 172.21.15.148:0/1005969986 conn(0x7f33c006a640 msgr2=0x7f33c00c4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:34.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 shutdown_connections 2023-10-30T13:18:34.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:34.856+0000 7f33c6cad640 1 -- 172.21.15.148:0/1005969986 wait complete. 2023-10-30T13:18:34.875 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:18:36.876 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:37.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1437394640 1 Processor -- start 2023-10-30T13:18:37.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1437394640 1 -- start start 2023-10-30T13:18:37.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1437394640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14300cf160 0x7f14300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:37.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1437394640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14300cb800 con 0x7f14301538a0 2023-10-30T13:18:37.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1437394640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14300cb330 con 0x7f143015d6f0 2023-10-30T13:18:37.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1437394640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14300cad70 con 0x7f14300cf160 2023-10-30T13:18:37.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1435130640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f14301538a0 0x7f14300ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48666/0 (socket says 172.21.15.148:48666) 2023-10-30T13:18:37.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1435130640 1 -- 172.21.15.148:0/1713031603 learned_addr learned my addr 172.21.15.148:0/1713031603 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:37.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1435931640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14300cf160 0x7f14300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:37.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873156640 0 0) 0x7f14300cb800 con 0x7f14301538a0 2023-10-30T13:18:37.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141c003530 con 0x7f14301538a0 2023-10-30T13:18:37.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474416004 0 0) 0x7f14300cb330 con 0x7f143015d6f0 2023-10-30T13:18:37.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14300cb800 con 0x7f143015d6f0 2023-10-30T13:18:37.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 795297297 0 0) 0x7f141c003530 con 0x7f14301538a0 2023-10-30T13:18:37.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14300cf160 msgr2=0x7f14300cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:37.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14300cf160 0x7f14300cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 >> v1:172.21.15.150:6789/0 conn(0x7f143015d6f0 legacy=0x7f143015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:37.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14300cb050 con 0x7f14301538a0 2023-10-30T13:18:37.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f142c001680 con 0x7f14301538a0 2023-10-30T13:18:37.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 >> v1:172.21.15.148:6789/0 conn(0x7f14301538a0 legacy=0x7f14300ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:37.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c003b40 0x7f141c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:37.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f141c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:37.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14300cf160 0x7f141c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:37.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14300cad70 con 0x7f141c0047d0 2023-10-30T13:18:37.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c00ba40 con 0x7f141c003b40 2023-10-30T13:18:37.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141c00bbd0 con 0x7f14300cf160 2023-10-30T13:18:37.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f143492f640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f141c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:37.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.024+0000 7f1435130640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c003b40 0x7f141c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:37.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1435931640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14300cf160 0x7f141c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:37.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f143492f640 1 -- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14300cf160 msgr2=0x7f141c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:37.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f143492f640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14300cf160 0x7f141c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f143492f640 1 -- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c003b40 msgr2=0x7f141c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:37.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f143492f640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c003b40 0x7f141c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f143492f640 1 -- 172.21.15.148:0/1713031603 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c003530 con 0x7f141c0047d0 2023-10-30T13:18:37.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1435931640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14300cf160 0x7f141c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:37.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f143492f640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f141c004c50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f14200018d0 tx=0x7f142001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87a2c34d7f2b0bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:37.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c007ea0 con 0x7f141c0047d0 2023-10-30T13:18:37.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14200a7050 con 0x7f141c0047d0 2023-10-30T13:18:37.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1427fff640 1 -- 172.21.15.148:0/1713031603 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420003600 con 0x7f141c0047d0 2023-10-30T13:18:37.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1437394640 1 -- 172.21.15.148:0/1713031603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 msgr2=0x7f141c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:37.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1437394640 1 --2- 172.21.15.148:0/1713031603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f141c004c50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f14200018d0 tx=0x7f142001c500 comp rx=0 tx=0).stop 2023-10-30T13:18:37.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1435931640 1 -- 172.21.15.148:0/1713031603 reap_dead start 2023-10-30T13:18:37.038 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1437394640 1 -- 172.21.15.148:0/1713031603 shutdown_connections 2023-10-30T13:18:37.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.028+0000 7f1437394640 1 -- 172.21.15.148:0/1713031603 >> 172.21.15.148:0/1713031603 conn(0x7f143006bfe0 msgr2=0x7f143006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:37.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 -- 172.21.15.148:0/1713031603 shutdown_connections 2023-10-30T13:18:37.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 -- 172.21.15.148:0/1713031603 wait complete. 2023-10-30T13:18:37.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 Processor -- start 2023-10-30T13:18:37.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 -- start start 2023-10-30T13:18:37.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f14300b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:37.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f143014ef90 0x7f1430143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:37.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1430158ef0 0x7f143014f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:37.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143014db50 con 0x7f141c0047d0 2023-10-30T13:18:37.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1430150ba0 con 0x7f143014ef90 2023-10-30T13:18:37.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1437394640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143015a8b0 con 0x7f1430158ef0 2023-10-30T13:18:37.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435130640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f14300b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:37.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435130640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f14300b4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34716/0 (socket says 172.21.15.148:34716) 2023-10-30T13:18:37.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435130640 1 -- 172.21.15.148:0/1630017463 learned_addr learned my addr 172.21.15.148:0/1630017463 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:37.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f143492f640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f143014ef90 0x7f1430143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:37.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435931640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1430158ef0 0x7f143014f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:37.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435130640 1 -- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1430158ef0 msgr2=0x7f143014f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:37.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435130640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1430158ef0 0x7f143014f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435130640 1 -- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f143014ef90 msgr2=0x7f1430143fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:37.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435130640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f143014ef90 0x7f1430143fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435130640 1 -- 172.21.15.148:0/1630017463 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14301508c0 con 0x7f141c0047d0 2023-10-30T13:18:37.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.032+0000 7f1435130640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f14300b4d30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f142c001de0 tx=0x7f142c0129a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:37.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f1425ffb640 1 -- 172.21.15.148:0/1630017463 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c014750 con 0x7f141c0047d0 2023-10-30T13:18:37.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f1425ffb640 1 -- 172.21.15.148:0/1630017463 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c0148d0 con 0x7f141c0047d0 2023-10-30T13:18:37.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14300b4770 con 0x7f141c0047d0 2023-10-30T13:18:37.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f1425ffb640 1 -- 172.21.15.148:0/1630017463 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c014c80 con 0x7f141c0047d0 2023-10-30T13:18:37.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143014dda0 con 0x7f141c0047d0 2023-10-30T13:18:37.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143014df90 con 0x7f141c0047d0 2023-10-30T13:18:37.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f1425ffb640 1 -- 172.21.15.148:0/1630017463 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f142c0153e0 con 0x7f141c0047d0 2023-10-30T13:18:37.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f1425ffb640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f14140664a0 0x7f1414068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:37.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f1425ffb640 1 -- 172.21.15.148:0/1630017463 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f142c086aa0 con 0x7f141c0047d0 2023-10-30T13:18:37.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.036+0000 7f143492f640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f14140664a0 0x7f1414068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:37.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.040+0000 7f143492f640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f14140664a0 0x7f1414068960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f14200ac000 tx=0x7f14200a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:37.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.040+0000 7f1425ffb640 1 -- 172.21.15.148:0/1630017463 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f142c053e00 con 0x7f141c0047d0 2023-10-30T13:18:37.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.220+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f143004e170 con 0x7f141c0047d0 2023-10-30T13:18:37.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.220+0000 7f1425ffb640 1 -- 172.21.15.148:0/1630017463 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f142c01c020 con 0x7f141c0047d0 2023-10-30T13:18:37.223 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:37.224 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:17:21.531094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4285,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.148:6834/4162935019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":4162935019},{"type":"v1","addr":"172.21.15.148:6834","nonce":4162935019}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:37.224 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 35 2023-10-30T13:18:37.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f14140664a0 msgr2=0x7f1414068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:37.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f14140664a0 0x7f1414068960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f14200ac000 tx=0x7f14200a6000 comp rx=0 tx=0).stop 2023-10-30T13:18:37.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 msgr2=0x7f14300b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:37.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f14300b4d30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f142c001de0 tx=0x7f142c0129a0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 shutdown_connections 2023-10-30T13:18:37.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f14140664a0 0x7f1414068960 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1430158ef0 0x7f143014f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f143014ef90 0x7f1430143fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 --2- 172.21.15.148:0/1630017463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f141c0047d0 0x7f14300b4d30 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:37.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 >> 172.21.15.148:0/1630017463 conn(0x7f143006bfe0 msgr2=0x7f1430152170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:37.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 shutdown_connections 2023-10-30T13:18:37.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:37.224+0000 7f1437394640 1 -- 172.21.15.148:0/1630017463 wait complete. 2023-10-30T13:18:37.245 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:18:39.245 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:39.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.392+0000 7fb8f1db8640 1 Processor -- start 2023-10-30T13:18:39.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.392+0000 7fb8f1db8640 1 -- start start 2023-10-30T13:18:39.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.392+0000 7fb8f1db8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb8ec0c05d0 0x7fb8ec13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.392+0000 7fb8f1db8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec0c0a50 con 0x7fb8ec0c6f60 2023-10-30T13:18:39.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.392+0000 7fb8f1db8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec06f6e0 con 0x7fb8ec0c6b80 2023-10-30T13:18:39.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.392+0000 7fb8f1db8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ec06f9c0 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.392+0000 7fb8ebfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb8ec0c6f60 0x7fb8ec1566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48670/0 (socket says 172.21.15.148:48670) 2023-10-30T13:18:39.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.392+0000 7fb8ebfff640 1 -- 172.21.15.148:0/355079510 learned_addr learned my addr 172.21.15.148:0/355079510 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:39.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.396+0000 7fb8eaffd640 1 --2- 172.21.15.148:0/355079510 >> v2:172.21.15.150:3300/0 conn(0x7fb8ec0c05d0 0x7fb8ec13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.396+0000 7fb8ea7fc640 1 -- 172.21.15.148:0/355079510 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527202455 0 0) 0x7fb8ec0c0a50 con 0x7fb8ec0c6f60 2023-10-30T13:18:39.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.396+0000 7fb8ea7fc640 1 -- 172.21.15.148:0/355079510 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d4003530 con 0x7fb8ec0c6f60 2023-10-30T13:18:39.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.396+0000 7fb8ea7fc640 1 -- 172.21.15.148:0/355079510 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2395384575 0 0) 0x7fb8d4003530 con 0x7fb8ec0c6f60 2023-10-30T13:18:39.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.396+0000 7fb8eaffd640 1 -- 172.21.15.148:0/355079510 >> v1:172.21.15.150:6789/0 conn(0x7fb8ec0c6b80 legacy=0x7fb8ec0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.396+0000 7fb8eaffd640 1 -- 172.21.15.148:0/355079510 >> v1:172.21.15.148:6789/0 conn(0x7fb8ec0c6f60 legacy=0x7fb8ec1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.396+0000 7fb8eaffd640 1 -- 172.21.15.148:0/355079510 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec06fca0 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.396+0000 7fb8eaffd640 1 --2- 172.21.15.148:0/355079510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c05d0 0x7fb8ec13ed90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00097c0 tx=0x7fb8e001c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f44e08070cadb3c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:39.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.396+0000 7fb8ea7fc640 1 -- 172.21.15.148:0/355079510 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4001a00 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8ea7fc640 1 -- 172.21.15.148:0/355079510 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e00a7050 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8ea7fc640 1 -- 172.21.15.148:0/355079510 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e00ab620 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 -- 172.21.15.148:0/355079510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c05d0 msgr2=0x7fb8ec13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 --2- 172.21.15.148:0/355079510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c05d0 0x7fb8ec13ed90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00097c0 tx=0x7fb8e001c3a0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 -- 172.21.15.148:0/355079510 shutdown_connections 2023-10-30T13:18:39.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 --2- 172.21.15.148:0/355079510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c05d0 0x7fb8ec13ed90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 -- 172.21.15.148:0/355079510 >> 172.21.15.148:0/355079510 conn(0x7fb8ec0c7910 msgr2=0x7fb8ec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:39.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 -- 172.21.15.148:0/355079510 shutdown_connections 2023-10-30T13:18:39.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 -- 172.21.15.148:0/355079510 wait complete. 2023-10-30T13:18:39.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 Processor -- start 2023-10-30T13:18:39.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 -- start start 2023-10-30T13:18:39.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8ec0c05d0 0x7fb8ec155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c6b80 0x7fb8ec12f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8ec0c6f60 0x7fb8ec14f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec159250 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec157790 con 0x7fb8ec0c6b80 2023-10-30T13:18:39.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.400+0000 7fb8f1db8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec1588b0 con 0x7fb8ec0c6f60 2023-10-30T13:18:39.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8ec0c05d0 0x7fb8ec155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8ec0c05d0 0x7fb8ec155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34720/0 (socket says 172.21.15.148:34720) 2023-10-30T13:18:39.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eb7fe640 1 -- 172.21.15.148:0/3871481810 learned_addr learned my addr 172.21.15.148:0/3871481810 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:39.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eaffd640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c6b80 0x7fb8ec12f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8ebfff640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8ec0c6f60 0x7fb8ec14f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eb7fe640 1 -- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8ec0c6f60 msgr2=0x7fb8ec14f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eb7fe640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8ec0c6f60 0x7fb8ec14f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eb7fe640 1 -- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c6b80 msgr2=0x7fb8ec12f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eb7fe640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c6b80 0x7fb8ec12f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eb7fe640 1 -- 172.21.15.148:0/3871481810 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e0009420 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eaffd640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c6b80 0x7fb8ec12f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:39.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8eb7fe640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8ec0c05d0 0x7fb8ec155c70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00c530 tx=0x7fb8dc00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:39.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8d3fff640 1 -- 172.21.15.148:0/3871481810 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc005220 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8d3fff640 1 -- 172.21.15.148:0/3871481810 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc005380 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ec158d40 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8d3fff640 1 -- 172.21.15.148:0/3871481810 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc007270 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ec157ca0 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ec04e170 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8d3fff640 1 -- 172.21.15.148:0/3871481810 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8dc0073d0 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.404+0000 7fb8d3fff640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8d80664d0 0x7fb8d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.408+0000 7fb8d3fff640 1 -- 172.21.15.148:0/3871481810 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb8dc083a20 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.408+0000 7fb8eaffd640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8d80664d0 0x7fb8d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.408+0000 7fb8eaffd640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8d80664d0 0x7fb8d8068990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00ad000 tx=0x7fb8e00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:39.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.408+0000 7fb8d3fff640 1 -- 172.21.15.148:0/3871481810 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8dc050c40 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.600+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8ec0008e0 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8d3fff640 1 -- 172.21.15.148:0/3871481810 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fb8dc054260 con 0x7fb8ec0c05d0 2023-10-30T13:18:39.604 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:39.604 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:39.446234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:39.605 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 36 2023-10-30T13:18:39.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8d80664d0 msgr2=0x7fb8d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8d80664d0 0x7fb8d8068990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00ad000 tx=0x7fb8e00a6000 comp rx=0 tx=0).stop 2023-10-30T13:18:39.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8ec0c05d0 msgr2=0x7fb8ec155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8ec0c05d0 0x7fb8ec155c70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00c530 tx=0x7fb8dc00c9f0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 shutdown_connections 2023-10-30T13:18:39.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8d80664d0 0x7fb8d8068990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8ec0c6f60 0x7fb8ec14f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8ec0c6b80 0x7fb8ec12f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 --2- 172.21.15.148:0/3871481810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8ec0c05d0 0x7fb8ec155c70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 >> 172.21.15.148:0/3871481810 conn(0x7fb8ec0c7910 msgr2=0x7fb8ec0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:39.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 shutdown_connections 2023-10-30T13:18:39.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.604+0000 7fb8f1db8640 1 -- 172.21.15.148:0/3871481810 wait complete. 2023-10-30T13:18:39.626 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:18:39.626 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:18:39.627 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:39.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f2321580640 1 Processor -- start 2023-10-30T13:18:39.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f2321580640 1 -- start start 2023-10-30T13:18:39.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f2321580640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f231c0c8cf0 0x7f231c0c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f2321580640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c15ce30 con 0x7f231c0ca340 2023-10-30T13:18:39.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f2321580640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c159880 con 0x7f231c0c90d0 2023-10-30T13:18:39.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f2321580640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231c159ae0 con 0x7f231c0c8cf0 2023-10-30T13:18:39.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f231b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f231c0ca340 0x7f231c15d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48678/0 (socket says 172.21.15.148:48678) 2023-10-30T13:18:39.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f231b7fe640 1 -- 172.21.15.148:0/294219709 learned_addr learned my addr 172.21.15.148:0/294219709 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:39.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f231affd640 1 --2- 172.21.15.148:0/294219709 >> v2:172.21.15.148:3300/0 conn(0x7f231c0c8cf0 0x7f231c0c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f231affd640 1 -- 172.21.15.148:0/294219709 >> v1:172.21.15.150:6789/0 conn(0x7f231c0c90d0 legacy=0x7f231c0cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f231affd640 1 -- 172.21.15.148:0/294219709 >> v1:172.21.15.148:6789/0 conn(0x7f231c0ca340 legacy=0x7f231c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f231affd640 1 -- 172.21.15.148:0/294219709 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c159dc0 con 0x7f231c0c8cf0 2023-10-30T13:18:39.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.764+0000 7f2319ffb640 1 -- 172.21.15.148:0/294219709 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726002803 0 0) 0x7f231c15ce30 con 0x7f231c0ca340 2023-10-30T13:18:39.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f231affd640 1 --2- 172.21.15.148:0/294219709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0c8cf0 0x7f231c0c9e00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f230c0097c0 tx=0x7f230c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1cacab82a60989 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:39.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2319ffb640 1 -- 172.21.15.148:0/294219709 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c0ab670 con 0x7f231c0c8cf0 2023-10-30T13:18:39.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2319ffb640 1 -- 172.21.15.148:0/294219709 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c0ab7d0 con 0x7f231c0c8cf0 2023-10-30T13:18:39.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2319ffb640 1 -- 172.21.15.148:0/294219709 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c0abb60 con 0x7f231c0c8cf0 2023-10-30T13:18:39.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2321580640 1 -- 172.21.15.148:0/294219709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0c8cf0 msgr2=0x7f231c0c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2321580640 1 --2- 172.21.15.148:0/294219709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0c8cf0 0x7f231c0c9e00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f230c0097c0 tx=0x7f230c01c450 comp rx=0 tx=0).stop 2023-10-30T13:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2321580640 1 -- 172.21.15.148:0/294219709 shutdown_connections 2023-10-30T13:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2321580640 1 --2- 172.21.15.148:0/294219709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0c8cf0 0x7f231c0c9e00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2321580640 1 -- 172.21.15.148:0/294219709 >> 172.21.15.148:0/294219709 conn(0x7f231c06be40 msgr2=0x7f231c06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2321580640 1 -- 172.21.15.148:0/294219709 shutdown_connections 2023-10-30T13:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.768+0000 7f2321580640 1 -- 172.21.15.148:0/294219709 wait complete. 2023-10-30T13:18:39.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 Processor -- start 2023-10-30T13:18:39.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 -- start start 2023-10-30T13:18:39.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f231c0c8cf0 0x7f231c0c4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f231c0c90d0 0x7f231c152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0ca340 0x7f231c157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0c4230 con 0x7f231c0ca340 2023-10-30T13:18:39.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c164fc0 con 0x7f231c0c8cf0 2023-10-30T13:18:39.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0c3c00 con 0x7f231c0c90d0 2023-10-30T13:18:39.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0ca340 0x7f231c157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0ca340 0x7f231c157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34738/0 (socket says 172.21.15.148:34738) 2023-10-30T13:18:39.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231b7fe640 1 -- 172.21.15.148:0/937590097 learned_addr learned my addr 172.21.15.148:0/937590097 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:39.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231affd640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f231c0c8cf0 0x7f231c0c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231a7fc640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f231c0c90d0 0x7f231c152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231b7fe640 1 -- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f231c0c90d0 msgr2=0x7f231c152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231b7fe640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f231c0c90d0 0x7f231c152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231b7fe640 1 -- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f231c0c8cf0 msgr2=0x7f231c0c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231b7fe640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f231c0c8cf0 0x7f231c0c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231b7fe640 1 -- 172.21.15.148:0/937590097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c009420 con 0x7f231c0ca340 2023-10-30T13:18:39.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231affd640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f231c0c8cf0 0x7f231c0c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:39.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f231b7fe640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0ca340 0x7f231c157b40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f231400c810 tx=0x7f231400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:39.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f23037fe640 1 -- 172.21.15.148:0/937590097 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231400a5e0 con 0x7f231c0ca340 2023-10-30T13:18:39.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f23037fe640 1 -- 172.21.15.148:0/937590097 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231400a760 con 0x7f231c0ca340 2023-10-30T13:18:39.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c1f2370 con 0x7f231c0ca340 2023-10-30T13:18:39.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f23037fe640 1 -- 172.21.15.148:0/937590097 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231400ab10 con 0x7f231c0ca340 2023-10-30T13:18:39.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c0c3e50 con 0x7f231c0ca340 2023-10-30T13:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.772+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231c0c4370 con 0x7f231c0ca340 2023-10-30T13:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.776+0000 7f23037fe640 1 -- 172.21.15.148:0/937590097 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f231400ac70 con 0x7f231c0ca340 2023-10-30T13:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.776+0000 7f23037fe640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23080664d0 0x7f2308068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.776+0000 7f23037fe640 1 -- 172.21.15.148:0/937590097 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f2314083890 con 0x7f231c0ca340 2023-10-30T13:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.776+0000 7f231affd640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23080664d0 0x7f2308068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.776+0000 7f231affd640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23080664d0 0x7f2308068990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f230c0af000 tx=0x7f230c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.780+0000 7f23037fe640 1 -- 172.21.15.148:0/937590097 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f231c0c4370 con 0x7f231c0ca340 2023-10-30T13:18:39.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f231c0c4370 con 0x7f231c0ca340 2023-10-30T13:18:39.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f23037fe640 1 -- 172.21.15.148:0/937590097 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f231c0c4370 con 0x7f231c0ca340 2023-10-30T13:18:39.961 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:39.961 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:39.446234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:39.962 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 36 2023-10-30T13:18:39.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23080664d0 msgr2=0x7f2308068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23080664d0 0x7f2308068990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f230c0af000 tx=0x7f230c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:18:39.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0ca340 msgr2=0x7f231c157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:39.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0ca340 0x7f231c157b40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f231400c810 tx=0x7f231400ccd0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 shutdown_connections 2023-10-30T13:18:39.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23080664d0 0x7f2308068990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f231c0ca340 0x7f231c157b40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f231c0c90d0 0x7f231c152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 --2- 172.21.15.148:0/937590097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f231c0c8cf0 0x7f231c0c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:39.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 >> 172.21.15.148:0/937590097 conn(0x7f231c06be40 msgr2=0x7f231c15c480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:39.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 shutdown_connections 2023-10-30T13:18:39.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:39.960+0000 7f2321580640 1 -- 172.21.15.148:0/937590097 wait complete. 2023-10-30T13:18:39.982 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 36, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4959": { "addr": "172.21.15.148:6837/95380877", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 95380877, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 95380877, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4959, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 16 }, "gid_5033": { "addr": "172.21.15.150:6836/953984429", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 953984429, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 953984429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5033, "incarnation": 36, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:18:39.446234+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5033, "mds_1": 5103, "mds_2": 4959 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:18:41.984 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa37479d640 1 Processor -- start 2023-10-30T13:18:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa37479d640 1 -- start start 2023-10-30T13:18:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa37479d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa36c0c6b80 0x7fa36c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa37479d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c6f60 0x7fa36c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa37479d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa36c0c8430 0x7fa36c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa37479d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36c0c0a50 con 0x7fa36c0c6b80 2023-10-30T13:18:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa37479d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36c06f6e0 con 0x7fa36c0c8430 2023-10-30T13:18:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa37479d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c06f9c0 con 0x7fa36c0c6f60 2023-10-30T13:18:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa372539640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa36c0c6b80 0x7fa36c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa372539640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa36c0c6b80 0x7fa36c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59700/0 (socket says 172.21.15.148:59700) 2023-10-30T13:18:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa372539640 1 -- 172.21.15.148:0/3611163165 learned_addr learned my addr 172.21.15.148:0/3611163165 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa372d3a640 1 --2- 172.21.15.148:0/3611163165 >> v2:172.21.15.150:3300/0 conn(0x7fa36c0c8430 0x7fa36c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.132+0000 7fa371d38640 1 --2- 172.21.15.148:0/3611163165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c6f60 0x7fa36c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa372539640 1 -- 172.21.15.148:0/3611163165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c6f60 msgr2=0x7fa36c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa372539640 1 --2- 172.21.15.148:0/3611163165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c6f60 0x7fa36c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa372539640 1 -- 172.21.15.148:0/3611163165 >> v2:172.21.15.150:3300/0 conn(0x7fa36c0c8430 msgr2=0x7fa36c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:42.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa372539640 1 --2- 172.21.15.148:0/3611163165 >> v2:172.21.15.150:3300/0 conn(0x7fa36c0c8430 0x7fa36c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa372539640 1 -- 172.21.15.148:0/3611163165 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c06fca0 con 0x7fa36c0c6b80 2023-10-30T13:18:42.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa372539640 1 --2- 172.21.15.148:0/3611163165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 0x7fa36c0bfec0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa36c0b0860 tx=0x7fa36801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e73c1a1498d6723 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:42.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa371537640 1 -- 172.21.15.148:0/3611163165 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3680ab600 con 0x7fa36c0c6b80 2023-10-30T13:18:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa371537640 1 -- 172.21.15.148:0/3611163165 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3680a7050 con 0x7fa36c0c6b80 2023-10-30T13:18:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa371537640 1 -- 172.21.15.148:0/3611163165 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3680ab980 con 0x7fa36c0c6b80 2023-10-30T13:18:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa37479d640 1 -- 172.21.15.148:0/3611163165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 msgr2=0x7fa36c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa37479d640 1 --2- 172.21.15.148:0/3611163165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 0x7fa36c0bfec0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa36c0b0860 tx=0x7fa36801c470 comp rx=0 tx=0).stop 2023-10-30T13:18:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa37479d640 1 -- 172.21.15.148:0/3611163165 shutdown_connections 2023-10-30T13:18:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa37479d640 1 --2- 172.21.15.148:0/3611163165 >> v2:172.21.15.150:3300/0 conn(0x7fa36c0c8430 0x7fa36c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa37479d640 1 --2- 172.21.15.148:0/3611163165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c6f60 0x7fa36c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa37479d640 1 --2- 172.21.15.148:0/3611163165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 0x7fa36c0bfec0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.136+0000 7fa37479d640 1 -- 172.21.15.148:0/3611163165 >> 172.21.15.148:0/3611163165 conn(0x7fa36c0c7910 msgr2=0x7fa36c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 -- 172.21.15.148:0/3611163165 shutdown_connections 2023-10-30T13:18:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 -- 172.21.15.148:0/3611163165 wait complete. 2023-10-30T13:18:42.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 Processor -- start 2023-10-30T13:18:42.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 -- start start 2023-10-30T13:18:42.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 0x7fa36c1309d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:42.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36c0c6f60 0x7fa36c132880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:42.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c8430 0x7fa36c1334f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:42.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c134010 con 0x7fa36c0c6b80 2023-10-30T13:18:42.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c133110 con 0x7fa36c0c6f60 2023-10-30T13:18:42.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c158b10 con 0x7fa36c0c8430 2023-10-30T13:18:42.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372539640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 0x7fa36c1309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:42.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372539640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 0x7fa36c1309d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59712/0 (socket says 172.21.15.148:59712) 2023-10-30T13:18:42.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372539640 1 -- 172.21.15.148:0/1024354845 learned_addr learned my addr 172.21.15.148:0/1024354845 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:42.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa371d38640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36c0c6f60 0x7fa36c132880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:42.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372d3a640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c8430 0x7fa36c1334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:42.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372539640 1 -- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c8430 msgr2=0x7fa36c1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:42.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372539640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c8430 0x7fa36c1334f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372539640 1 -- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36c0c6f60 msgr2=0x7fa36c132880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:42.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372539640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36c0c6f60 0x7fa36c132880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372539640 1 -- 172.21.15.148:0/1024354845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3680094d0 con 0x7fa36c0c6b80 2023-10-30T13:18:42.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa372539640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 0x7fa36c1309d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa36801cd10 tx=0x7fa368019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:42.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa362ffd640 1 -- 172.21.15.148:0/1024354845 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3680aa5b0 con 0x7fa36c0c6b80 2023-10-30T13:18:42.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa362ffd640 1 -- 172.21.15.148:0/1024354845 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3680aa710 con 0x7fa36c0c6b80 2023-10-30T13:18:42.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c130540 con 0x7fa36c0c6b80 2023-10-30T13:18:42.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.140+0000 7fa362ffd640 1 -- 172.21.15.148:0/1024354845 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3680aaa80 con 0x7fa36c0c6b80 2023-10-30T13:18:42.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.144+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c158550 con 0x7fa36c0c6b80 2023-10-30T13:18:42.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.144+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c130f10 con 0x7fa36c0c6b80 2023-10-30T13:18:42.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.144+0000 7fa362ffd640 1 -- 172.21.15.148:0/1024354845 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3680b2070 con 0x7fa36c0c6b80 2023-10-30T13:18:42.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.144+0000 7fa362ffd640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3500664a0 0x7fa350068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:42.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.144+0000 7fa362ffd640 1 -- 172.21.15.148:0/1024354845 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fa36811d950 con 0x7fa36c0c6b80 2023-10-30T13:18:42.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.144+0000 7fa371d38640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3500664a0 0x7fa350068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:42.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.148+0000 7fa371d38640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3500664a0 0x7fa350068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa36c159190 tx=0x7fa35c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:42.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.148+0000 7fa362ffd640 1 -- 172.21.15.148:0/1024354845 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3680eabe0 con 0x7fa36c0c6b80 2023-10-30T13:18:42.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.328+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa36c0c6750 con 0x7fa36c0c6b80 2023-10-30T13:18:42.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.328+0000 7fa362ffd640 1 -- 172.21.15.148:0/1024354845 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa3680ee200 con 0x7fa36c0c6b80 2023-10-30T13:18:42.330 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:42.330 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:41.416272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:42.330 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 37 2023-10-30T13:18:42.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3500664a0 msgr2=0x7fa350068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:42.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3500664a0 0x7fa350068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa36c159190 tx=0x7fa35c008040 comp rx=0 tx=0).stop 2023-10-30T13:18:42.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 msgr2=0x7fa36c1309d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:42.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 0x7fa36c1309d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa36801cd10 tx=0x7fa368019210 comp rx=0 tx=0).stop 2023-10-30T13:18:42.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 shutdown_connections 2023-10-30T13:18:42.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3500664a0 0x7fa350068960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa36c0c8430 0x7fa36c1334f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa36c0c6f60 0x7fa36c132880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 --2- 172.21.15.148:0/1024354845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa36c0c6b80 0x7fa36c1309d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:42.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 >> 172.21.15.148:0/1024354845 conn(0x7fa36c0c7910 msgr2=0x7fa36c131520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:42.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 shutdown_connections 2023-10-30T13:18:42.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:42.332+0000 7fa37479d640 1 -- 172.21.15.148:0/1024354845 wait complete. 2023-10-30T13:18:42.352 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:44.354 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:44.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9039d0f640 1 Processor -- start 2023-10-30T13:18:44.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9039d0f640 1 -- start start 2023-10-30T13:18:44.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9039d0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f903415d6f0 0x7f903415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:44.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9039d0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90340cb800 con 0x7f90340cf160 2023-10-30T13:18:44.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9039d0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90340cb330 con 0x7f90341538a0 2023-10-30T13:18:44.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9039d0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90340cad70 con 0x7f903415d6f0 2023-10-30T13:18:44.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9033fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f90340cf160 0x7f9034164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45764/0 (socket says 172.21.15.148:45764) 2023-10-30T13:18:44.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9033fff640 1 -- 172.21.15.148:0/3643596331 learned_addr learned my addr 172.21.15.148:0/3643596331 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:44.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9032ffd640 1 --2- 172.21.15.148:0/3643596331 >> v2:172.21.15.148:3300/0 conn(0x7f903415d6f0 0x7f903415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:44.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f90327fc640 1 -- 172.21.15.148:0/3643596331 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559866063 0 0) 0x7f90340cb800 con 0x7f90340cf160 2023-10-30T13:18:44.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f90327fc640 1 -- 172.21.15.148:0/3643596331 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f901c003530 con 0x7f90340cf160 2023-10-30T13:18:44.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9032ffd640 1 -- 172.21.15.148:0/3643596331 >> v1:172.21.15.150:6789/0 conn(0x7f90341538a0 legacy=0x7f90340ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:44.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9032ffd640 1 -- 172.21.15.148:0/3643596331 >> v1:172.21.15.148:6789/0 conn(0x7f90340cf160 legacy=0x7f9034164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:44.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.512+0000 7f9032ffd640 1 -- 172.21.15.148:0/3643596331 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90340cb050 con 0x7f903415d6f0 2023-10-30T13:18:44.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f9032ffd640 1 --2- 172.21.15.148:0/3643596331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f903415d6f0 0x7f903415a090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f90280097c0 tx=0x7f902801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3209631cc46f3402 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:44.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f90327fc640 1 -- 172.21.15.148:0/3643596331 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90280a8970 con 0x7f903415d6f0 2023-10-30T13:18:44.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f90327fc640 1 -- 172.21.15.148:0/3643596331 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90280a8ad0 con 0x7f903415d6f0 2023-10-30T13:18:44.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f90327fc640 1 -- 172.21.15.148:0/3643596331 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90280a8e40 con 0x7f903415d6f0 2023-10-30T13:18:44.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f9039d0f640 1 -- 172.21.15.148:0/3643596331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f903415d6f0 msgr2=0x7f903415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:44.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f9039d0f640 1 --2- 172.21.15.148:0/3643596331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f903415d6f0 0x7f903415a090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f90280097c0 tx=0x7f902801ec70 comp rx=0 tx=0).stop 2023-10-30T13:18:44.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f9039d0f640 1 -- 172.21.15.148:0/3643596331 shutdown_connections 2023-10-30T13:18:44.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f9039d0f640 1 --2- 172.21.15.148:0/3643596331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f903415d6f0 0x7f903415a090 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:44.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f9039d0f640 1 -- 172.21.15.148:0/3643596331 >> 172.21.15.148:0/3643596331 conn(0x7f903406bfe0 msgr2=0x7f903406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:44.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f9039d0f640 1 -- 172.21.15.148:0/3643596331 shutdown_connections 2023-10-30T13:18:44.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.516+0000 7f9039d0f640 1 -- 172.21.15.148:0/3643596331 wait complete. 2023-10-30T13:18:44.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 Processor -- start 2023-10-30T13:18:44.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 -- start start 2023-10-30T13:18:44.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90340cf160 0x7f90341e2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:44.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90341538a0 0x7f90341e60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:44.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f903415d6f0 0x7f90341e3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:44.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90341e3500 con 0x7f90340cf160 2023-10-30T13:18:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90341f3120 con 0x7f903415d6f0 2023-10-30T13:18:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90340c6c00 con 0x7f90341538a0 2023-10-30T13:18:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f90337fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90340cf160 0x7f90341e2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f90337fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90340cf160 0x7f90341e2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59730/0 (socket says 172.21.15.148:59730) 2023-10-30T13:18:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f90337fe640 1 -- 172.21.15.148:0/3316167396 learned_addr learned my addr 172.21.15.148:0/3316167396 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9032ffd640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90341538a0 0x7f90341e60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9033fff640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f903415d6f0 0x7f90341e3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f90337fe640 1 -- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90341538a0 msgr2=0x7f90341e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f90337fe640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90341538a0 0x7f90341e60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f90337fe640 1 -- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f903415d6f0 msgr2=0x7f90341e3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f90337fe640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f903415d6f0 0x7f90341e3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f90337fe640 1 -- 172.21.15.148:0/3316167396 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028009420 con 0x7f90340cf160 2023-10-30T13:18:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9033fff640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f903415d6f0 0x7f90341e3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f90337fe640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90340cf160 0x7f90341e2e70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f902400c530 tx=0x7f902400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f901bfff640 1 -- 172.21.15.148:0/3316167396 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9024005220 con 0x7f90340cf160 2023-10-30T13:18:44.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f901bfff640 1 -- 172.21.15.148:0/3316167396 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9024005380 con 0x7f90340cf160 2023-10-30T13:18:44.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f901bfff640 1 -- 172.21.15.148:0/3316167396 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9024007270 con 0x7f90340cf160 2023-10-30T13:18:44.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90341f2550 con 0x7f90340cf160 2023-10-30T13:18:44.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90341f2ad0 con 0x7f90340cf160 2023-10-30T13:18:44.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.520+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903404e170 con 0x7f90340cf160 2023-10-30T13:18:44.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.524+0000 7f901bfff640 1 -- 172.21.15.148:0/3316167396 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90240073d0 con 0x7f90340cf160 2023-10-30T13:18:44.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.524+0000 7f901bfff640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90200664d0 0x7f9020068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:44.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.524+0000 7f901bfff640 1 -- 172.21.15.148:0/3316167396 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f9024083920 con 0x7f90340cf160 2023-10-30T13:18:44.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.524+0000 7f9032ffd640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90200664d0 0x7f9020068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:44.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.524+0000 7f9032ffd640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90200664d0 0x7f9020068990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f90280b2000 tx=0x7f90280aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:44.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.528+0000 7f901bfff640 1 -- 172.21.15.148:0/3316167396 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9024050ba0 con 0x7f90340cf160 2023-10-30T13:18:44.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90340ca9b0 con 0x7f90340cf160 2023-10-30T13:18:44.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f901bfff640 1 -- 172.21.15.148:0/3316167396 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f90240541c0 con 0x7f90340cf160 2023-10-30T13:18:44.713 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:44.713 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:41.416272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:44.713 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 37 2023-10-30T13:18:44.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90200664d0 msgr2=0x7f9020068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:44.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90200664d0 0x7f9020068990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f90280b2000 tx=0x7f90280aa070 comp rx=0 tx=0).stop 2023-10-30T13:18:44.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90340cf160 msgr2=0x7f90341e2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:44.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90340cf160 0x7f90341e2e70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f902400c530 tx=0x7f902400c9f0 comp rx=0 tx=0).stop 2023-10-30T13:18:44.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 shutdown_connections 2023-10-30T13:18:44.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90200664d0 0x7f9020068990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:44.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f903415d6f0 0x7f90341e3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:44.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90341538a0 0x7f90341e60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:44.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 --2- 172.21.15.148:0/3316167396 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90340cf160 0x7f90341e2e70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:44.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 >> 172.21.15.148:0/3316167396 conn(0x7f903406bfe0 msgr2=0x7f90340c5070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:44.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 shutdown_connections 2023-10-30T13:18:44.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:44.712+0000 7f9039d0f640 1 -- 172.21.15.148:0/3316167396 wait complete. 2023-10-30T13:18:44.734 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:46.736 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:46.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af3a9a640 1 Processor -- start 2023-10-30T13:18:46.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af3a9a640 1 -- start start 2023-10-30T13:18:46.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af3a9a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8aec06e470 0x7f8aec06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:46.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af3a9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8aec0c8490 0x7f8aec0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:46.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af3a9a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aec06afa0 con 0x7f8aec153d70 2023-10-30T13:18:46.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af3a9a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec06d780 con 0x7f8aec06e470 2023-10-30T13:18:46.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af3a9a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec154150 con 0x7f8aec0c8490 2023-10-30T13:18:46.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af1035640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8aec06e470 0x7f8aec06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:46.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af1836640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8aec153d70 0x7f8aec0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45766/0 (socket says 172.21.15.148:45766) 2023-10-30T13:18:46.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af1836640 1 -- 172.21.15.148:0/3017595199 learned_addr learned my addr 172.21.15.148:0/3017595199 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:46.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af2037640 1 --2- 172.21.15.148:0/3017595199 >> v2:172.21.15.150:3300/0 conn(0x7f8aec0c8490 0x7f8aec0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:46.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af1035640 1 -- 172.21.15.148:0/3017595199 >> v2:172.21.15.150:3300/0 conn(0x7f8aec0c8490 msgr2=0x7f8aec0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:46.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af1035640 1 --2- 172.21.15.148:0/3017595199 >> v2:172.21.15.150:3300/0 conn(0x7f8aec0c8490 0x7f8aec0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:46.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af1035640 1 -- 172.21.15.148:0/3017595199 >> v1:172.21.15.148:6789/0 conn(0x7f8aec153d70 legacy=0x7f8aec0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:46.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af1035640 1 -- 172.21.15.148:0/3017595199 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec06e030 con 0x7f8aec06e470 2023-10-30T13:18:46.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.884+0000 7f8af2037640 1 --2- 172.21.15.148:0/3017595199 >> v2:172.21.15.150:3300/0 conn(0x7f8aec0c8490 0x7f8aec0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:18:46.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af1035640 1 --2- 172.21.15.148:0/3017595199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec06e470 0x7f8aec06ca80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0cddf0 tx=0x7f8adc01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641482a66cca64e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:46.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af0834640 1 -- 172.21.15.148:0/3017595199 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0a7070 con 0x7f8aec06e470 2023-10-30T13:18:46.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af0834640 1 -- 172.21.15.148:0/3017595199 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc0ab430 con 0x7f8aec06e470 2023-10-30T13:18:46.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af0834640 1 -- 172.21.15.148:0/3017595199 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0ab7c0 con 0x7f8aec06e470 2023-10-30T13:18:46.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af3a9a640 1 -- 172.21.15.148:0/3017595199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec06e470 msgr2=0x7f8aec06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:46.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af3a9a640 1 --2- 172.21.15.148:0/3017595199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec06e470 0x7f8aec06ca80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0cddf0 tx=0x7f8adc01c430 comp rx=0 tx=0).stop 2023-10-30T13:18:46.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af3a9a640 1 -- 172.21.15.148:0/3017595199 shutdown_connections 2023-10-30T13:18:46.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af3a9a640 1 --2- 172.21.15.148:0/3017595199 >> v2:172.21.15.150:3300/0 conn(0x7f8aec0c8490 0x7f8aec0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:46.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af3a9a640 1 --2- 172.21.15.148:0/3017595199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec06e470 0x7f8aec06ca80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:46.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af3a9a640 1 -- 172.21.15.148:0/3017595199 >> 172.21.15.148:0/3017595199 conn(0x7f8aec071ea0 msgr2=0x7f8aec065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:46.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af3a9a640 1 -- 172.21.15.148:0/3017595199 shutdown_connections 2023-10-30T13:18:46.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.888+0000 7f8af3a9a640 1 -- 172.21.15.148:0/3017595199 wait complete. 2023-10-30T13:18:46.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 Processor -- start 2023-10-30T13:18:46.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 -- start start 2023-10-30T13:18:46.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8aec06e470 0x7f8aec144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:46.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec0c8490 0x7f8aec143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:46.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aec153d70 0x7f8aec143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:46.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0cb680 con 0x7f8aec0c8490 2023-10-30T13:18:46.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0cb960 con 0x7f8aec153d70 2023-10-30T13:18:46.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec1462d0 con 0x7f8aec06e470 2023-10-30T13:18:46.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1035640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec0c8490 0x7f8aec143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:46.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1035640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec0c8490 0x7f8aec143a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59746/0 (socket says 172.21.15.148:59746) 2023-10-30T13:18:46.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1035640 1 -- 172.21.15.148:0/2278369515 learned_addr learned my addr 172.21.15.148:0/2278369515 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:46.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af2037640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aec153d70 0x7f8aec143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:46.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1836640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8aec06e470 0x7f8aec144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:46.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1035640 1 -- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8aec06e470 msgr2=0x7f8aec144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:46.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1035640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8aec06e470 0x7f8aec144a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:46.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1035640 1 -- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aec153d70 msgr2=0x7f8aec143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:46.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1035640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aec153d70 0x7f8aec143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:46.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1035640 1 -- 172.21.15.148:0/2278369515 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc009420 con 0x7f8aec0c8490 2023-10-30T13:18:46.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af2037640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aec153d70 0x7f8aec143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:18:46.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af1035640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec0c8490 0x7f8aec143a30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0ad040 tx=0x7f8adc01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8ae27fc640 1 -- 172.21.15.148:0/2278369515 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0b5810 con 0x7f8aec0c8490 2023-10-30T13:18:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8ae27fc640 1 -- 172.21.15.148:0/2278369515 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc0b5990 con 0x7f8aec0c8490 2023-10-30T13:18:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8ae27fc640 1 -- 172.21.15.148:0/2278369515 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0b5d40 con 0x7f8aec0c8490 2023-10-30T13:18:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec0cbdf0 con 0x7f8aec0c8490 2023-10-30T13:18:46.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.892+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec146550 con 0x7f8aec0c8490 2023-10-30T13:18:46.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.896+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec04e170 con 0x7f8aec0c8490 2023-10-30T13:18:46.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.896+0000 7f8ae27fc640 1 -- 172.21.15.148:0/2278369515 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8adc0b4390 con 0x7f8aec0c8490 2023-10-30T13:18:46.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.896+0000 7f8ae27fc640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ad00664a0 0x7f8ad0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:46.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.896+0000 7f8ae27fc640 1 -- 172.21.15.148:0/2278369515 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f8adc11e520 con 0x7f8aec0c8490 2023-10-30T13:18:46.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.896+0000 7f8af1836640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ad00664a0 0x7f8ad0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:46.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.896+0000 7f8af1836640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ad00664a0 0x7f8ad0068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8005890 tx=0x7f8ae8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:46.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:46.900+0000 7f8ae27fc640 1 -- 172.21.15.148:0/2278369515 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8adc0eb7c0 con 0x7f8aec0c8490 2023-10-30T13:18:47.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.080+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8aec06c2a0 con 0x7f8aec0c8490 2023-10-30T13:18:47.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.080+0000 7f8ae27fc640 1 -- 172.21.15.148:0/2278369515 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f8adc0eede0 con 0x7f8aec0c8490 2023-10-30T13:18:47.083 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:47.084 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:45.174191+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:resolve","state_seq":23,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:47.084 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 38 2023-10-30T13:18:47.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ad00664a0 msgr2=0x7f8ad0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:47.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ad00664a0 0x7f8ad0068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8005890 tx=0x7f8ae8009040 comp rx=0 tx=0).stop 2023-10-30T13:18:47.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec0c8490 msgr2=0x7f8aec143a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:47.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec0c8490 0x7f8aec143a30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0ad040 tx=0x7f8adc01ac70 comp rx=0 tx=0).stop 2023-10-30T13:18:47.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 shutdown_connections 2023-10-30T13:18:47.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ad00664a0 0x7f8ad0068960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:47.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8aec0c8490 0x7f8aec143a30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:47.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aec153d70 0x7f8aec143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:47.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 --2- 172.21.15.148:0/2278369515 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8aec06e470 0x7f8aec144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:47.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 >> 172.21.15.148:0/2278369515 conn(0x7f8aec071ea0 msgr2=0x7f8aec150af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:47.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 shutdown_connections 2023-10-30T13:18:47.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:47.084+0000 7f8af3a9a640 1 -- 172.21.15.148:0/2278369515 wait complete. 2023-10-30T13:18:47.104 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:49.105 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:49.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f3210c46640 1 Processor -- start 2023-10-30T13:18:49.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f3210c46640 1 -- start start 2023-10-30T13:18:49.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f3210c46640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f320c0c6b80 0x7f320c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:49.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f3210c46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f320c0c6f60 0x7f320c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:49.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f3210c46640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c0c0a50 con 0x7f320c0c8430 2023-10-30T13:18:49.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f3210c46640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c06f6e0 con 0x7f320c0c6b80 2023-10-30T13:18:49.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f3210c46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c06f9c0 con 0x7f320c0c6f60 2023-10-30T13:18:49.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f320a59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f320c0c6b80 0x7f320c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:49.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f320ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f320c0c8430 0x7f320c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45776/0 (socket says 172.21.15.148:45776) 2023-10-30T13:18:49.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f320ad9d640 1 -- 172.21.15.148:0/544543908 learned_addr learned my addr 172.21.15.148:0/544543908 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:49.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.252+0000 7f3209d9b640 1 --2- 172.21.15.148:0/544543908 >> v2:172.21.15.150:3300/0 conn(0x7f320c0c6f60 0x7f320c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:49.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f320a59c640 1 -- 172.21.15.148:0/544543908 >> v2:172.21.15.150:3300/0 conn(0x7f320c0c6f60 msgr2=0x7f320c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:49.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f320a59c640 1 --2- 172.21.15.148:0/544543908 >> v2:172.21.15.150:3300/0 conn(0x7f320c0c6f60 0x7f320c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:49.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f320a59c640 1 -- 172.21.15.148:0/544543908 >> v1:172.21.15.148:6789/0 conn(0x7f320c0c8430 legacy=0x7f320c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:49.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f320a59c640 1 -- 172.21.15.148:0/544543908 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c06fca0 con 0x7f320c0c6b80 2023-10-30T13:18:49.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f3209d9b640 1 --2- 172.21.15.148:0/544543908 >> v2:172.21.15.150:3300/0 conn(0x7f320c0c6f60 0x7f320c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:49.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f320959a640 1 -- 172.21.15.148:0/544543908 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152929090 0 0) 0x7f320c0c0a50 con 0x7f320c0c8430 2023-10-30T13:18:49.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f320a59c640 1 --2- 172.21.15.148:0/544543908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 0x7f320c0bfec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f320c159b90 tx=0x7f31fc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c37debaff21dab7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:49.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f320959a640 1 -- 172.21.15.148:0/544543908 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0ab600 con 0x7f320c0c6b80 2023-10-30T13:18:49.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f320959a640 1 -- 172.21.15.148:0/544543908 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc0a7050 con 0x7f320c0c6b80 2023-10-30T13:18:49.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.256+0000 7f320959a640 1 -- 172.21.15.148:0/544543908 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0ab980 con 0x7f320c0c6b80 2023-10-30T13:18:49.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 -- 172.21.15.148:0/544543908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 msgr2=0x7f320c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:49.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 --2- 172.21.15.148:0/544543908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 0x7f320c0bfec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f320c159b90 tx=0x7f31fc01c460 comp rx=0 tx=0).stop 2023-10-30T13:18:49.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 -- 172.21.15.148:0/544543908 shutdown_connections 2023-10-30T13:18:49.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 --2- 172.21.15.148:0/544543908 >> v2:172.21.15.150:3300/0 conn(0x7f320c0c6f60 0x7f320c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:49.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 --2- 172.21.15.148:0/544543908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 0x7f320c0bfec0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:49.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 -- 172.21.15.148:0/544543908 >> 172.21.15.148:0/544543908 conn(0x7f320c0c7910 msgr2=0x7f320c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:49.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 -- 172.21.15.148:0/544543908 shutdown_connections 2023-10-30T13:18:49.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 -- 172.21.15.148:0/544543908 wait complete. 2023-10-30T13:18:49.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 Processor -- start 2023-10-30T13:18:49.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 -- start start 2023-10-30T13:18:49.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 0x7f320c06bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:49.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f320c0c6f60 0x7f320c0ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:49.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f320c0c8430 0x7f320c1f9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:49.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0c94f0 con 0x7f320c0c6b80 2023-10-30T13:18:49.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c1f9610 con 0x7f320c0c8430 2023-10-30T13:18:49.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3210c46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c13ce40 con 0x7f320c0c6f60 2023-10-30T13:18:49.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f320a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 0x7f320c06bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:49.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f320a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 0x7f320c06bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59758/0 (socket says 172.21.15.148:59758) 2023-10-30T13:18:49.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f320a59c640 1 -- 172.21.15.148:0/3216134464 learned_addr learned my addr 172.21.15.148:0/3216134464 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:49.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f320ad9d640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f320c0c8430 0x7f320c1f9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:49.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f3209d9b640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f320c0c6f60 0x7f320c0ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:49.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.260+0000 7f320a59c640 1 -- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f320c0c6f60 msgr2=0x7f320c0ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:49.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f320a59c640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f320c0c6f60 0x7f320c0ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:49.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f320a59c640 1 -- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f320c0c8430 msgr2=0x7f320c1f9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:49.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f320a59c640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f320c0c8430 0x7f320c1f9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:49.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f320a59c640 1 -- 172.21.15.148:0/3216134464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc009420 con 0x7f320c0c6b80 2023-10-30T13:18:49.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f320a59c640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 0x7f320c06bd60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f31fc019070 tx=0x7f31fc019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:49.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f31faffd640 1 -- 172.21.15.148:0/3216134464 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0b4840 con 0x7f320c0c6b80 2023-10-30T13:18:49.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f31faffd640 1 -- 172.21.15.148:0/3216134464 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc0b49a0 con 0x7f320c0c6b80 2023-10-30T13:18:49.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c1f92e0 con 0x7f320c0c6b80 2023-10-30T13:18:49.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f31faffd640 1 -- 172.21.15.148:0/3216134464 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0b4d30 con 0x7f320c0c6b80 2023-10-30T13:18:49.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c0c9a20 con 0x7f320c0c6b80 2023-10-30T13:18:49.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c04e170 con 0x7f320c0c6b80 2023-10-30T13:18:49.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f31faffd640 1 -- 172.21.15.148:0/3216134464 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31fc0b2040 con 0x7f320c0c6b80 2023-10-30T13:18:49.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.264+0000 7f31faffd640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31ec0664d0 0x7f31ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:49.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.268+0000 7f31faffd640 1 -- 172.21.15.148:0/3216134464 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f31fc11e3e0 con 0x7f320c0c6b80 2023-10-30T13:18:49.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.268+0000 7f3209d9b640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31ec0664d0 0x7f31ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:49.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.268+0000 7f3209d9b640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31ec0664d0 0x7f31ec068990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f320c0c1a00 tx=0x7f3200008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:49.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.268+0000 7f31faffd640 1 -- 172.21.15.148:0/3216134464 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31fc0eb5d0 con 0x7f320c0c6b80 2023-10-30T13:18:49.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.448+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f320c06adb0 con 0x7f320c0c6b80 2023-10-30T13:18:49.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.448+0000 7f31faffd640 1 -- 172.21.15.148:0/3216134464 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f31fc0b0090 con 0x7f320c0c6b80 2023-10-30T13:18:49.451 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:49.451 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:45.174191+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:resolve","state_seq":23,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:49.452 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 38 2023-10-30T13:18:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31ec0664d0 msgr2=0x7f31ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31ec0664d0 0x7f31ec068990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f320c0c1a00 tx=0x7f3200008040 comp rx=0 tx=0).stop 2023-10-30T13:18:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 msgr2=0x7f320c06bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 0x7f320c06bd60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f31fc019070 tx=0x7f31fc019200 comp rx=0 tx=0).stop 2023-10-30T13:18:49.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 shutdown_connections 2023-10-30T13:18:49.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f31ec0664d0 0x7f31ec068990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:49.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f320c0c8430 0x7f320c1f9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:49.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f320c0c6f60 0x7f320c0ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:49.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 --2- 172.21.15.148:0/3216134464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f320c0c6b80 0x7f320c06bd60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:49.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 >> 172.21.15.148:0/3216134464 conn(0x7f320c0c7910 msgr2=0x7f320c0b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:49.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 shutdown_connections 2023-10-30T13:18:49.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:49.452+0000 7f3210c46640 1 -- 172.21.15.148:0/3216134464 wait complete. 2023-10-30T13:18:49.472 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:51.472 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:51.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.620+0000 7fd3423b8640 1 Processor -- start 2023-10-30T13:18:51.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.620+0000 7fd3423b8640 1 -- start start 2023-10-30T13:18:51.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd3423b8640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd33c1538a0 0x7fd33c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:51.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd3423b8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd33c0ca0e0 0x7fd33c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:51.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd3423b8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0cb800 con 0x7fd33c0cf160 2023-10-30T13:18:51.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd3423b8640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c15dc30 con 0x7fd33c1538a0 2023-10-30T13:18:51.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd3423b8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0cb330 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd33bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd33c1538a0 0x7fd33c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:51.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd33bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd33c1538a0 0x7fd33c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59770/0 (socket says 172.21.15.148:59770) 2023-10-30T13:18:51.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd33bfff640 1 -- 172.21.15.148:0/607388842 learned_addr learned my addr 172.21.15.148:0/607388842 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:51.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd33b7fe640 1 --2- 172.21.15.148:0/607388842 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd33c0ca0e0 0x7fd33c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:51.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd33bfff640 1 -- 172.21.15.148:0/607388842 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd33c0ca0e0 msgr2=0x7fd33c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:51.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd33bfff640 1 --2- 172.21.15.148:0/607388842 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd33c0ca0e0 0x7fd33c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd33bfff640 1 -- 172.21.15.148:0/607388842 >> v1:172.21.15.150:6789/0 conn(0x7fd33c0cf160 legacy=0x7fd33c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:51.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.624+0000 7fd33bfff640 1 -- 172.21.15.148:0/607388842 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0cad70 con 0x7fd33c1538a0 2023-10-30T13:18:51.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd33bfff640 1 --2- 172.21.15.148:0/607388842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c1538a0 0x7fd33c15d6f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009870 tx=0x7fd32c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47932538fc963d7e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:51.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd33affd640 1 -- 172.21.15.148:0/607388842 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0a89b0 con 0x7fd33c1538a0 2023-10-30T13:18:51.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd33affd640 1 -- 172.21.15.148:0/607388842 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c0a8b10 con 0x7fd33c1538a0 2023-10-30T13:18:51.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd33affd640 1 -- 172.21.15.148:0/607388842 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0a8ea0 con 0x7fd33c1538a0 2023-10-30T13:18:51.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd3423b8640 1 -- 172.21.15.148:0/607388842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c1538a0 msgr2=0x7fd33c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:51.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd3423b8640 1 --2- 172.21.15.148:0/607388842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c1538a0 0x7fd33c15d6f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd32c009870 tx=0x7fd32c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd3423b8640 1 -- 172.21.15.148:0/607388842 shutdown_connections 2023-10-30T13:18:51.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd3423b8640 1 --2- 172.21.15.148:0/607388842 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd33c0ca0e0 0x7fd33c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd3423b8640 1 --2- 172.21.15.148:0/607388842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c1538a0 0x7fd33c15d6f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd3423b8640 1 -- 172.21.15.148:0/607388842 >> 172.21.15.148:0/607388842 conn(0x7fd33c06bfe0 msgr2=0x7fd33c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:51.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd3423b8640 1 -- 172.21.15.148:0/607388842 shutdown_connections 2023-10-30T13:18:51.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.628+0000 7fd3423b8640 1 -- 172.21.15.148:0/607388842 wait complete. 2023-10-30T13:18:51.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd3423b8640 1 Processor -- start 2023-10-30T13:18:51.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd3423b8640 1 -- start start 2023-10-30T13:18:51.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd3423b8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c0ca0e0 0x7fd33c065a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:51.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd3423b8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c0cf160 0x7fd33c1ef1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:51.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd3423b8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd33c1538a0 0x7fd33c1479f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:51.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd3423b8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0666b0 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd3423b8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0660f0 con 0x7fd33c0cf160 2023-10-30T13:18:51.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd3423b8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0663d0 con 0x7fd33c1538a0 2023-10-30T13:18:51.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd33bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c0ca0e0 0x7fd33c065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:51.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd33bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c0ca0e0 0x7fd33c065a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59772/0 (socket says 172.21.15.148:59772) 2023-10-30T13:18:51.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd33bfff640 1 -- 172.21.15.148:0/313248045 learned_addr learned my addr 172.21.15.148:0/313248045 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:51.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd33bfff640 1 -- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd33c1538a0 msgr2=0x7fd33c1479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:51.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd33bfff640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd33c1538a0 0x7fd33c1479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd33bfff640 1 -- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c0cf160 msgr2=0x7fd33c1ef1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:51.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd33bfff640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c0cf160 0x7fd33c1ef1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd33bfff640 1 -- 172.21.15.148:0/313248045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0094d0 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd33bfff640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c0ca0e0 0x7fd33c065a80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0b3040 tx=0x7fd32c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:51.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd338ff9640 1 -- 172.21.15.148:0/313248045 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0a89b0 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd338ff9640 1 -- 172.21.15.148:0/313248045 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c0bb480 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c1f2840 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.632+0000 7fd338ff9640 1 -- 172.21.15.148:0/313248045 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0bb830 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.636+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c1f2e00 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.636+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c0b2a50 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.636+0000 7fd338ff9640 1 -- 172.21.15.148:0/313248045 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd32c0bb990 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.640+0000 7fd338ff9640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3280664d0 0x7fd328068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:51.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.640+0000 7fd338ff9640 1 -- 172.21.15.148:0/313248045 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd32c123070 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.640+0000 7fd33b7fe640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3280664d0 0x7fd328068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:51.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.640+0000 7fd33b7fe640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3280664d0 0x7fd328068990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd33c150330 tx=0x7fd330008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:51.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.644+0000 7fd338ff9640 1 -- 172.21.15.148:0/313248045 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd32c0f0370 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd33c0b2a50 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd338ff9640 1 -- 172.21.15.148:0/313248045 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fd32c0b5020 con 0x7fd33c0ca0e0 2023-10-30T13:18:51.829 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:51.829 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:50.616844+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:51.829 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 39 2023-10-30T13:18:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3280664d0 msgr2=0x7fd328068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3280664d0 0x7fd328068990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd33c150330 tx=0x7fd330008040 comp rx=0 tx=0).stop 2023-10-30T13:18:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c0ca0e0 msgr2=0x7fd33c065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:51.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c0ca0e0 0x7fd33c065a80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0b3040 tx=0x7fd32c01ba90 comp rx=0 tx=0).stop 2023-10-30T13:18:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 shutdown_connections 2023-10-30T13:18:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3280664d0 0x7fd328068990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd33c1538a0 0x7fd33c1479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd33c0cf160 0x7fd33c1ef1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 --2- 172.21.15.148:0/313248045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd33c0ca0e0 0x7fd33c065a80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:51.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 >> 172.21.15.148:0/313248045 conn(0x7fd33c06bfe0 msgr2=0x7fd33c0c4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 shutdown_connections 2023-10-30T13:18:51.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:51.828+0000 7fd3423b8640 1 -- 172.21.15.148:0/313248045 wait complete. 2023-10-30T13:18:51.851 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4959": { "addr": "172.21.15.148:6837/95380877", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 95380877, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 95380877, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4959, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 16 }, "gid_5033": { "addr": "172.21.15.150:6836/953984429", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 953984429, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 953984429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5033, "incarnation": 36, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 25 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:18:50.616844+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5033, "mds_1": 5103, "mds_2": 4959 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:18:53.853 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.000+0000 7f9a47c09640 1 Processor -- start 2023-10-30T13:18:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.000+0000 7f9a47c09640 1 -- start start 2023-10-30T13:18:54.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a47c09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a4006f7a0 0x7f9a4006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:54.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a47c09640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9a400c6bd0 0x7f9a40069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:54.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a47c09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9a40071110 0x7f9a40158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:54.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a47c09640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a400700c0 con 0x7f9a400c6bd0 2023-10-30T13:18:54.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a47c09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4006a3d0 con 0x7f9a40071110 2023-10-30T13:18:54.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a47c09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a40066cf0 con 0x7f9a4006f7a0 2023-10-30T13:18:54.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a451a4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9a400c6bd0 0x7f9a40069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:54.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a451a4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9a400c6bd0 0x7f9a40069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60294/0 (socket says 172.21.15.148:60294) 2023-10-30T13:18:54.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a451a4640 1 -- 172.21.15.148:0/1398056844 learned_addr learned my addr 172.21.15.148:0/1398056844 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:54.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a459a5640 1 --2- 172.21.15.148:0/1398056844 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a4006f7a0 0x7f9a4006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:54.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a461a6640 1 --2- 172.21.15.148:0/1398056844 >> v2:172.21.15.150:3300/0 conn(0x7f9a40071110 0x7f9a40158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:54.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a451a4640 1 -- 172.21.15.148:0/1398056844 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a4006f7a0 msgr2=0x7f9a4006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:54.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a451a4640 1 --2- 172.21.15.148:0/1398056844 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a4006f7a0 0x7f9a4006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a451a4640 1 -- 172.21.15.148:0/1398056844 >> v2:172.21.15.150:3300/0 conn(0x7f9a40071110 msgr2=0x7f9a40158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:54.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a451a4640 1 --2- 172.21.15.148:0/1398056844 >> v2:172.21.15.150:3300/0 conn(0x7f9a40071110 0x7f9a40158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.004+0000 7f9a451a4640 1 -- 172.21.15.148:0/1398056844 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a400c70d0 con 0x7f9a400c6bd0 2023-10-30T13:18:54.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a451a4640 1 --2- 172.21.15.148:0/1398056844 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a400c6bd0 0x7f9a40069e90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9a4006c7d0 tx=0x7f9a3001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4555dbf5fa61c4d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:54.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a449a3640 1 -- 172.21.15.148:0/1398056844 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300a89b0 con 0x7f9a400c6bd0 2023-10-30T13:18:54.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a449a3640 1 -- 172.21.15.148:0/1398056844 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a300a8b10 con 0x7f9a400c6bd0 2023-10-30T13:18:54.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a449a3640 1 -- 172.21.15.148:0/1398056844 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300a8ea0 con 0x7f9a400c6bd0 2023-10-30T13:18:54.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 -- 172.21.15.148:0/1398056844 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a400c6bd0 msgr2=0x7f9a40069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:54.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 --2- 172.21.15.148:0/1398056844 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a400c6bd0 0x7f9a40069e90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9a4006c7d0 tx=0x7f9a3001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 -- 172.21.15.148:0/1398056844 shutdown_connections 2023-10-30T13:18:54.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 --2- 172.21.15.148:0/1398056844 >> v2:172.21.15.150:3300/0 conn(0x7f9a40071110 0x7f9a40158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 --2- 172.21.15.148:0/1398056844 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a400c6bd0 0x7f9a40069e90 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 --2- 172.21.15.148:0/1398056844 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a4006f7a0 0x7f9a4006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 -- 172.21.15.148:0/1398056844 >> 172.21.15.148:0/1398056844 conn(0x7f9a400c7910 msgr2=0x7f9a400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:54.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 -- 172.21.15.148:0/1398056844 shutdown_connections 2023-10-30T13:18:54.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 -- 172.21.15.148:0/1398056844 wait complete. 2023-10-30T13:18:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 Processor -- start 2023-10-30T13:18:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 -- start start 2023-10-30T13:18:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.008+0000 7f9a47c09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a4006f7a0 0x7f9a400c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:54.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a47c09640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a40071110 0x7f9a401f0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:54.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a47c09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a400c6bd0 0x7f9a401f0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:54.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a47c09640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4014a5a0 con 0x7f9a40071110 2023-10-30T13:18:54.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a47c09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a40149720 con 0x7f9a4006f7a0 2023-10-30T13:18:54.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a47c09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a40149990 con 0x7f9a400c6bd0 2023-10-30T13:18:54.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a451a4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a40071110 0x7f9a401f0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:54.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a451a4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a40071110 0x7f9a401f0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60308/0 (socket says 172.21.15.148:60308) 2023-10-30T13:18:54.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a451a4640 1 -- 172.21.15.148:0/3272677847 learned_addr learned my addr 172.21.15.148:0/3272677847 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:54.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a459a5640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a4006f7a0 0x7f9a400c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:54.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a461a6640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a400c6bd0 0x7f9a401f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:54.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a451a4640 1 -- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a400c6bd0 msgr2=0x7f9a401f0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:54.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a451a4640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a400c6bd0 0x7f9a401f0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a451a4640 1 -- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a4006f7a0 msgr2=0x7f9a400c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:54.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a451a4640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a4006f7a0 0x7f9a400c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a451a4640 1 -- 172.21.15.148:0/3272677847 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a300094d0 con 0x7f9a40071110 2023-10-30T13:18:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a459a5640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a4006f7a0 0x7f9a400c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a451a4640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a40071110 0x7f9a401f0130 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9a300b3040 tx=0x7f9a3001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a367fc640 1 -- 172.21.15.148:0/3272677847 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300b0d40 con 0x7f9a40071110 2023-10-30T13:18:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a367fc640 1 -- 172.21.15.148:0/3272677847 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a300bb460 con 0x7f9a40071110 2023-10-30T13:18:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a367fc640 1 -- 172.21.15.148:0/3272677847 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300b0ea0 con 0x7f9a40071110 2023-10-30T13:18:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a40149c70 con 0x7f9a40071110 2023-10-30T13:18:54.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4014a230 con 0x7f9a40071110 2023-10-30T13:18:54.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4006a1d0 con 0x7f9a40071110 2023-10-30T13:18:54.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.012+0000 7f9a367fc640 1 -- 172.21.15.148:0/3272677847 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9a300b83f0 con 0x7f9a40071110 2023-10-30T13:18:54.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.016+0000 7f9a367fc640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9a240664a0 0x7f9a24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:54.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.016+0000 7f9a367fc640 1 -- 172.21.15.148:0/3272677847 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f9a30121c50 con 0x7f9a40071110 2023-10-30T13:18:54.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.016+0000 7f9a459a5640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9a240664a0 0x7f9a24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:54.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.016+0000 7f9a459a5640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9a240664a0 0x7f9a24068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009a90 tx=0x7f9a3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:54.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.020+0000 7f9a367fc640 1 -- 172.21.15.148:0/3272677847 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9a300eee40 con 0x7f9a40071110 2023-10-30T13:18:54.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a4004e170 con 0x7f9a40071110 2023-10-30T13:18:54.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a367fc640 1 -- 172.21.15.148:0/3272677847 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f9a300f2460 con 0x7f9a40071110 2023-10-30T13:18:54.201 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:54.202 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:50.616844+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:54.202 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 39 2023-10-30T13:18:54.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9a240664a0 msgr2=0x7f9a24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:54.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9a240664a0 0x7f9a24068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009a90 tx=0x7f9a3c008040 comp rx=0 tx=0).stop 2023-10-30T13:18:54.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a40071110 msgr2=0x7f9a401f0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:54.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a40071110 0x7f9a401f0130 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9a300b3040 tx=0x7f9a3001ba90 comp rx=0 tx=0).stop 2023-10-30T13:18:54.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 shutdown_connections 2023-10-30T13:18:54.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9a240664a0 0x7f9a24068960 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9a40071110 0x7f9a401f0130 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9a400c6bd0 0x7f9a401f0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 --2- 172.21.15.148:0/3272677847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a4006f7a0 0x7f9a400c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:54.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.200+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 >> 172.21.15.148:0/3272677847 conn(0x7f9a400c7910 msgr2=0x7f9a400c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:54.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.204+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 shutdown_connections 2023-10-30T13:18:54.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:54.204+0000 7f9a47c09640 1 -- 172.21.15.148:0/3272677847 wait complete. 2023-10-30T13:18:54.222 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:54.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:18:54.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:18:56.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:56.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc370772640 1 Processor -- start 2023-10-30T13:18:56.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc370772640 1 -- start start 2023-10-30T13:18:56.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc370772640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc3680cf160 0x7fc3680cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:56.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc370772640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3680cb800 con 0x7fc3681538a0 2023-10-30T13:18:56.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc370772640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3680cb330 con 0x7fc36815d6f0 2023-10-30T13:18:56.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc370772640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3680cad70 con 0x7fc3680cf160 2023-10-30T13:18:56.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc36e50e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc3681538a0 0x7fc3680ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43900/0 (socket says 172.21.15.148:43900) 2023-10-30T13:18:56.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc36e50e640 1 -- 172.21.15.148:0/1345836566 learned_addr learned my addr 172.21.15.148:0/1345836566 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:18:56.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc36ed0f640 1 --2- 172.21.15.148:0/1345836566 >> v2:172.21.15.148:3300/0 conn(0x7fc3680cf160 0x7fc3680cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:56.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc36d50c640 1 -- 172.21.15.148:0/1345836566 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227966630 0 0) 0x7fc3680cb800 con 0x7fc3681538a0 2023-10-30T13:18:56.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc36d50c640 1 -- 172.21.15.148:0/1345836566 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc354003530 con 0x7fc3681538a0 2023-10-30T13:18:56.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc36ed0f640 1 -- 172.21.15.148:0/1345836566 >> v1:172.21.15.150:6789/0 conn(0x7fc36815d6f0 legacy=0x7fc36815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:56.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc36ed0f640 1 -- 172.21.15.148:0/1345836566 >> v1:172.21.15.148:6789/0 conn(0x7fc3681538a0 legacy=0x7fc3680ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:56.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.372+0000 7fc36ed0f640 1 -- 172.21.15.148:0/1345836566 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3680cb050 con 0x7fc3680cf160 2023-10-30T13:18:56.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc36ed0f640 1 --2- 172.21.15.148:0/1345836566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3680cf160 0x7fc3680cf590 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc360009d90 tx=0x7fc36001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f686e6ea4bdf0a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:56.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc36d50c640 1 -- 172.21.15.148:0/1345836566 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3600af600 con 0x7fc3680cf160 2023-10-30T13:18:56.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc36d50c640 1 -- 172.21.15.148:0/1345836566 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3600ab050 con 0x7fc3680cf160 2023-10-30T13:18:56.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc36d50c640 1 -- 172.21.15.148:0/1345836566 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3600af960 con 0x7fc3680cf160 2023-10-30T13:18:56.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc370772640 1 -- 172.21.15.148:0/1345836566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3680cf160 msgr2=0x7fc3680cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:56.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc370772640 1 --2- 172.21.15.148:0/1345836566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3680cf160 0x7fc3680cf590 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc360009d90 tx=0x7fc36001f130 comp rx=0 tx=0).stop 2023-10-30T13:18:56.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc370772640 1 -- 172.21.15.148:0/1345836566 shutdown_connections 2023-10-30T13:18:56.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc370772640 1 --2- 172.21.15.148:0/1345836566 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3680cf160 0x7fc3680cf590 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:56.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc370772640 1 -- 172.21.15.148:0/1345836566 >> 172.21.15.148:0/1345836566 conn(0x7fc36806bfe0 msgr2=0x7fc36806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:56.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc370772640 1 -- 172.21.15.148:0/1345836566 shutdown_connections 2023-10-30T13:18:56.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.376+0000 7fc370772640 1 -- 172.21.15.148:0/1345836566 wait complete. 2023-10-30T13:18:56.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 Processor -- start 2023-10-30T13:18:56.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 -- start start 2023-10-30T13:18:56.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3680cf160 0x7fc3681f0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:56.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3681538a0 0x7fc3681f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:56.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36815d6f0 0x7fc3681f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:56.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680b3bb0 con 0x7fc3681538a0 2023-10-30T13:18:56.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3680b4730 con 0x7fc36815d6f0 2023-10-30T13:18:56.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3681f3150 con 0x7fc3680cf160 2023-10-30T13:18:56.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36dd0d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3681538a0 0x7fc3681f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36dd0d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3681538a0 0x7fc3681f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60334/0 (socket says 172.21.15.148:60334) 2023-10-30T13:18:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36dd0d640 1 -- 172.21.15.148:0/3356681133 learned_addr learned my addr 172.21.15.148:0/3356681133 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36e50e640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3680cf160 0x7fc3681f0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36ed0f640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36815d6f0 0x7fc3681f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36dd0d640 1 -- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3680cf160 msgr2=0x7fc3681f0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:56.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36dd0d640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3680cf160 0x7fc3681f0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:56.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36dd0d640 1 -- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36815d6f0 msgr2=0x7fc3681f2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:56.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36dd0d640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36815d6f0 0x7fc3681f2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:56.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36dd0d640 1 -- 172.21.15.148:0/3356681133 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc360009a70 con 0x7fc3681538a0 2023-10-30T13:18:56.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc36dd0d640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3681538a0 0x7fc3681f25c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc35800c3a0 tx=0x7fc35800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:56.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc35effd640 1 -- 172.21.15.148:0/3356681133 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3580052d0 con 0x7fc3681538a0 2023-10-30T13:18:56.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc35effd640 1 -- 172.21.15.148:0/3356681133 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc358005430 con 0x7fc3681538a0 2023-10-30T13:18:56.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc35effd640 1 -- 172.21.15.148:0/3356681133 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358007260 con 0x7fc3681538a0 2023-10-30T13:18:56.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3680b3e00 con 0x7fc3681538a0 2023-10-30T13:18:56.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3680b43c0 con 0x7fc3681538a0 2023-10-30T13:18:56.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.380+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36804e170 con 0x7fc3681538a0 2023-10-30T13:18:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.384+0000 7fc35effd640 1 -- 172.21.15.148:0/3356681133 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3580073c0 con 0x7fc3681538a0 2023-10-30T13:18:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.384+0000 7fc35effd640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc34c0664a0 0x7fc34c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.384+0000 7fc35effd640 1 -- 172.21.15.148:0/3356681133 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fc358083430 con 0x7fc3681538a0 2023-10-30T13:18:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.384+0000 7fc36e50e640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc34c0664a0 0x7fc34c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.384+0000 7fc36e50e640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc34c0664a0 0x7fc34c068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc3640011f0 tx=0x7fc364007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:56.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.388+0000 7fc35effd640 1 -- 172.21.15.148:0/3356681133 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3580506d0 con 0x7fc3681538a0 2023-10-30T13:18:56.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.572+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc36815d4d0 con 0x7fc3681538a0 2023-10-30T13:18:56.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.572+0000 7fc35effd640 1 -- 172.21.15.148:0/3356681133 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fc35805a020 con 0x7fc3681538a0 2023-10-30T13:18:56.574 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:56.575 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:50.616844+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:56.575 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 39 2023-10-30T13:18:56.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc34c0664a0 msgr2=0x7fc34c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:56.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc34c0664a0 0x7fc34c068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc3640011f0 tx=0x7fc364007d40 comp rx=0 tx=0).stop 2023-10-30T13:18:56.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3681538a0 msgr2=0x7fc3681f25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:56.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3681538a0 0x7fc3681f25c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc35800c3a0 tx=0x7fc35800c860 comp rx=0 tx=0).stop 2023-10-30T13:18:56.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 shutdown_connections 2023-10-30T13:18:56.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc34c0664a0 0x7fc34c068960 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:56.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3681538a0 0x7fc3681f25c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:56.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc36815d6f0 0x7fc3681f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:56.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 --2- 172.21.15.148:0/3356681133 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3680cf160 0x7fc3681f0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:56.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 >> 172.21.15.148:0/3356681133 conn(0x7fc36806bfe0 msgr2=0x7fc368151b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:56.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 shutdown_connections 2023-10-30T13:18:56.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:56.576+0000 7fc370772640 1 -- 172.21.15.148:0/3356681133 wait complete. 2023-10-30T13:18:56.595 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:58.597 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:58.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.744+0000 7fd2edfaa640 1 Processor -- start 2023-10-30T13:18:58.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2edfaa640 1 -- start start 2023-10-30T13:18:58.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2edfaa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e80c6bd0 0x7fd2e806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:58.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2edfaa640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2e8071110 0x7fd2e8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:58.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2edfaa640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e8066cf0 con 0x7fd2e806f7a0 2023-10-30T13:18:58.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2edfaa640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e80700c0 con 0x7fd2e8071110 2023-10-30T13:18:58.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2edfaa640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80c70d0 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2e6ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e80c6bd0 0x7fd2e806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:58.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2e7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2e8071110 0x7fd2e8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:58.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2e6ffd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e80c6bd0 0x7fd2e806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:35122/0 (socket says 172.21.15.148:35122) 2023-10-30T13:18:58.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2e6ffd640 1 -- 172.21.15.148:0/3865606378 learned_addr learned my addr 172.21.15.148:0/3865606378 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:58.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2e6ffd640 1 -- 172.21.15.148:0/3865606378 >> v2:172.21.15.150:3300/0 conn(0x7fd2e8071110 msgr2=0x7fd2e8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:58.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2e6ffd640 1 --2- 172.21.15.148:0/3865606378 >> v2:172.21.15.150:3300/0 conn(0x7fd2e8071110 0x7fd2e8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:58.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2e6ffd640 1 -- 172.21.15.148:0/3865606378 >> v1:172.21.15.150:6789/0 conn(0x7fd2e806f7a0 legacy=0x7fd2e8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:58.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.748+0000 7fd2e6ffd640 1 -- 172.21.15.148:0/3865606378 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80c7390 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2e6ffd640 1 --2- 172.21.15.148:0/3865606378 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e80c6bd0 0x7fd2e806fb80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009870 tx=0x7fd2dc01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=206343fc287c3f5d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:58.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2e67fc640 1 -- 172.21.15.148:0/3865606378 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc0a8a30 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2e67fc640 1 -- 172.21.15.148:0/3865606378 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc0a8b90 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2e67fc640 1 -- 172.21.15.148:0/3865606378 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc0b0440 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2edfaa640 1 -- 172.21.15.148:0/3865606378 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e80c6bd0 msgr2=0x7fd2e806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:58.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2edfaa640 1 --2- 172.21.15.148:0/3865606378 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e80c6bd0 0x7fd2e806fb80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009870 tx=0x7fd2dc01ed30 comp rx=0 tx=0).stop 2023-10-30T13:18:58.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2edfaa640 1 -- 172.21.15.148:0/3865606378 shutdown_connections 2023-10-30T13:18:58.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2edfaa640 1 --2- 172.21.15.148:0/3865606378 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e80c6bd0 0x7fd2e806fb80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:58.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2edfaa640 1 --2- 172.21.15.148:0/3865606378 >> v2:172.21.15.150:3300/0 conn(0x7fd2e8071110 0x7fd2e8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:58.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2edfaa640 1 -- 172.21.15.148:0/3865606378 >> 172.21.15.148:0/3865606378 conn(0x7fd2e80c7910 msgr2=0x7fd2e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:58.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2edfaa640 1 -- 172.21.15.148:0/3865606378 shutdown_connections 2023-10-30T13:18:58.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2edfaa640 1 -- 172.21.15.148:0/3865606378 wait complete. 2023-10-30T13:18:58.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.752+0000 7fd2edfaa640 1 Processor -- start 2023-10-30T13:18:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 -- start start 2023-10-30T13:18:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e806f7a0 0x7fd2e80afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2e8071110 0x7fd2e80bff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2e80c6bd0 0x7fd2e80c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80b0170 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e81f2e20 con 0x7fd2e8071110 2023-10-30T13:18:58.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e81f30c0 con 0x7fd2e806f7a0 2023-10-30T13:18:58.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2e80c6bd0 0x7fd2e80c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:58.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2e80c6bd0 0x7fd2e80c3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60350/0 (socket says 172.21.15.148:60350) 2023-10-30T13:18:58.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e7fff640 1 -- 172.21.15.148:0/2993131721 learned_addr learned my addr 172.21.15.148:0/2993131721 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:18:58.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e77fe640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e806f7a0 0x7fd2e80afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:58.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e6ffd640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2e8071110 0x7fd2e80bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:58.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e7fff640 1 -- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e806f7a0 msgr2=0x7fd2e80afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:58.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e7fff640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e806f7a0 0x7fd2e80afc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:58.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e7fff640 1 -- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2e8071110 msgr2=0x7fd2e80bff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:58.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e7fff640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2e8071110 0x7fd2e80bff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:58.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e7fff640 1 -- 172.21.15.148:0/2993131721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc0094d0 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2e7fff640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2e80c6bd0 0x7fd2e80c3a20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000efc0 tx=0x7fd2e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:58.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2cffff640 1 -- 172.21.15.148:0/2993131721 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0009ba0 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2cffff640 1 -- 172.21.15.148:0/2993131721 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e000f040 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e81eee00 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2cffff640 1 -- 172.21.15.148:0/2993131721 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e00146c0 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e81ef3a0 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.756+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e804e170 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.760+0000 7fd2cffff640 1 -- 172.21.15.148:0/2993131721 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2e0014820 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.760+0000 7fd2cffff640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664f0 0x7fd2d40689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:58.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.760+0000 7fd2cffff640 1 -- 172.21.15.148:0/2993131721 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd2e0084c30 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.760+0000 7fd2e77fe640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664f0 0x7fd2d40689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:58.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.760+0000 7fd2e77fe640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664f0 0x7fd2d40689b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8001770 tx=0x7fd2d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:58.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.764+0000 7fd2cffff640 1 -- 172.21.15.148:0/2993131721 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2e0052050 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.944+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2e8150df0 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.944+0000 7fd2cffff640 1 -- 172.21.15.148:0/2993131721 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fd2e0018020 con 0x7fd2e80c6bd0 2023-10-30T13:18:58.946 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:18:58.946 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:56.622559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:58.946 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 40 2023-10-30T13:18:58.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664f0 msgr2=0x7fd2d40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:58.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664f0 0x7fd2d40689b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8001770 tx=0x7fd2d8009040 comp rx=0 tx=0).stop 2023-10-30T13:18:58.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2e80c6bd0 msgr2=0x7fd2e80c3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:58.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2e80c6bd0 0x7fd2e80c3a20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000efc0 tx=0x7fd2e000c490 comp rx=0 tx=0).stop 2023-10-30T13:18:58.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 shutdown_connections 2023-10-30T13:18:58.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664f0 0x7fd2d40689b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:58.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2e80c6bd0 0x7fd2e80c3a20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:58.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2e8071110 0x7fd2e80bff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:58.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 --2- 172.21.15.148:0/2993131721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2e806f7a0 0x7fd2e80afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:58.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 >> 172.21.15.148:0/2993131721 conn(0x7fd2e80c7910 msgr2=0x7fd2e80c9220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:58.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 shutdown_connections 2023-10-30T13:18:58.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:18:58.948+0000 7fd2edfaa640 1 -- 172.21.15.148:0/2993131721 wait complete. 2023-10-30T13:18:58.967 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:19:00.969 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:19:01.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.116+0000 7fc90caea640 1 Processor -- start 2023-10-30T13:19:01.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.116+0000 7fc90caea640 1 -- start start 2023-10-30T13:19:01.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.116+0000 7fc90caea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc9080bfec0 0x7fc9080c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:01.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.116+0000 7fc90caea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc908071290 con 0x7fc90806a720 2023-10-30T13:19:01.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.116+0000 7fc90caea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc908071570 con 0x7fc908159e60 2023-10-30T13:19:01.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.116+0000 7fc90caea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc908071850 con 0x7fc9080bfec0 2023-10-30T13:19:01.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc906d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc90806a720 0x7fc908160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43910/0 (socket says 172.21.15.148:43910) 2023-10-30T13:19:01.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc906d9d640 1 -- 172.21.15.148:0/2202731449 learned_addr learned my addr 172.21.15.148:0/2202731449 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:19:01.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc905d9b640 1 --2- 172.21.15.148:0/2202731449 >> v2:172.21.15.150:3300/0 conn(0x7fc9080bfec0 0x7fc9080c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:01.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589343387 0 0) 0x7fc908071290 con 0x7fc90806a720 2023-10-30T13:19:01.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f0003450 con 0x7fc90806a720 2023-10-30T13:19:01.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1350346206 0 0) 0x7fc8f0003450 con 0x7fc90806a720 2023-10-30T13:19:01.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 >> v2:172.21.15.150:3300/0 conn(0x7fc9080bfec0 msgr2=0x7fc9080c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:01.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 --2- 172.21.15.148:0/2202731449 >> v2:172.21.15.150:3300/0 conn(0x7fc9080bfec0 0x7fc9080c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 >> v1:172.21.15.150:6789/0 conn(0x7fc908159e60 legacy=0x7fc90806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:01.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc908071b30 con 0x7fc90806a720 2023-10-30T13:19:01.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fc900001a00 con 0x7fc90806a720 2023-10-30T13:19:01.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 >> v1:172.21.15.148:6789/0 conn(0x7fc90806a720 legacy=0x7fc908160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:01.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f0003a70 0x7fc8f0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:01.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8f0008860 0x7fc8f000ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:01.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8f000b410 0x7fc8f000d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:01.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc908071850 con 0x7fc8f000b410 2023-10-30T13:19:01.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f000e110 con 0x7fc8f0003a70 2023-10-30T13:19:01.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f000e2d0 con 0x7fc8f0008860 2023-10-30T13:19:01.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc906d9d640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8f000b410 0x7fc8f000d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:01.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc90659c640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f0003a70 0x7fc8f0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:01.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc905d9b640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8f0008860 0x7fc8f000ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:01.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc906d9d640 1 -- 172.21.15.148:0/2202731449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8f0008860 msgr2=0x7fc8f000ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:01.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc906d9d640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8f0008860 0x7fc8f000ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc906d9d640 1 -- 172.21.15.148:0/2202731449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f0003a70 msgr2=0x7fc8f0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:01.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc906d9d640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f0003a70 0x7fc8f0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.120+0000 7fc906d9d640 1 -- 172.21.15.148:0/2202731449 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f0003450 con 0x7fc8f000b410 2023-10-30T13:19:01.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc906d9d640 1 -- 172.21.15.148:0/2202731449 reap_dead start 2023-10-30T13:19:01.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc906d9d640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8f000b410 0x7fc8f000d8d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc90000a7a0 tx=0x7fc90001d260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540376880ea93816 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:01.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9000ad630 con 0x7fc8f000b410 2023-10-30T13:19:01.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9000a8070 con 0x7fc8f000b410 2023-10-30T13:19:01.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90559a640 1 -- 172.21.15.148:0/2202731449 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9000ad9f0 con 0x7fc8f000b410 2023-10-30T13:19:01.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90caea640 1 -- 172.21.15.148:0/2202731449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8f000b410 msgr2=0x7fc8f000d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:01.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90caea640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8f000b410 0x7fc8f000d8d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc90000a7a0 tx=0x7fc90001d260 comp rx=0 tx=0).stop 2023-10-30T13:19:01.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90caea640 1 -- 172.21.15.148:0/2202731449 shutdown_connections 2023-10-30T13:19:01.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90caea640 1 --2- 172.21.15.148:0/2202731449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8f000b410 0x7fc8f000d8d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90caea640 1 -- 172.21.15.148:0/2202731449 >> 172.21.15.148:0/2202731449 conn(0x7fc9080c7910 msgr2=0x7fc9080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:01.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90caea640 1 -- 172.21.15.148:0/2202731449 shutdown_connections 2023-10-30T13:19:01.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90caea640 1 -- 172.21.15.148:0/2202731449 wait complete. 2023-10-30T13:19:01.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90caea640 1 Processor -- start 2023-10-30T13:19:01.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.124+0000 7fc90caea640 1 -- start start 2023-10-30T13:19:01.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90caea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f000b410 0x7fc9081583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:01.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90caea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9081e0250 0x7fc90814e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:01.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90caea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908148f30 0x7fc9081493e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:01.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90caea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9081df3d0 con 0x7fc908148f30 2023-10-30T13:19:01.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90caea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc908158920 con 0x7fc8f000b410 2023-10-30T13:19:01.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90caea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9081576b0 con 0x7fc9081e0250 2023-10-30T13:19:01.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc906d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908148f30 0x7fc9081493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:01.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc906d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908148f30 0x7fc9081493e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60366/0 (socket says 172.21.15.148:60366) 2023-10-30T13:19:01.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc906d9d640 1 -- 172.21.15.148:0/1973792725 learned_addr learned my addr 172.21.15.148:0/1973792725 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:01.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90659c640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f000b410 0x7fc9081583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:01.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc905d9b640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9081e0250 0x7fc90814e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:01.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc906d9d640 1 -- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9081e0250 msgr2=0x7fc90814e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:01.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc906d9d640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9081e0250 0x7fc90814e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc906d9d640 1 -- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f000b410 msgr2=0x7fc9081583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:01.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc906d9d640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f000b410 0x7fc9081583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc906d9d640 1 -- 172.21.15.148:0/1973792725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90814f2b0 con 0x7fc908148f30 2023-10-30T13:19:01.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90659c640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f000b410 0x7fc9081583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:19:01.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc905d9b640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9081e0250 0x7fc90814e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:19:01.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc906d9d640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908148f30 0x7fc9081493e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc90000a7a0 tx=0x7fc8f0003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:01.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc8f6ffd640 1 -- 172.21.15.148:0/1973792725 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9000ad430 con 0x7fc908148f30 2023-10-30T13:19:01.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc8f6ffd640 1 -- 172.21.15.148:0/1973792725 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9000ad590 con 0x7fc908148f30 2023-10-30T13:19:01.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90814f590 con 0x7fc908148f30 2023-10-30T13:19:01.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc8f6ffd640 1 -- 172.21.15.148:0/1973792725 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9000ad900 con 0x7fc908148f30 2023-10-30T13:19:01.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc908157be0 con 0x7fc908148f30 2023-10-30T13:19:01.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90804e170 con 0x7fc908148f30 2023-10-30T13:19:01.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.128+0000 7fc8f6ffd640 1 -- 172.21.15.148:0/1973792725 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9000aa030 con 0x7fc908148f30 2023-10-30T13:19:01.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.132+0000 7fc8f6ffd640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8e80664d0 0x7fc8e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:01.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.132+0000 7fc8f6ffd640 1 -- 172.21.15.148:0/1973792725 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fc9000a5e40 con 0x7fc908148f30 2023-10-30T13:19:01.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.132+0000 7fc90659c640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8e80664d0 0x7fc8e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:01.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.132+0000 7fc90659c640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8e80664d0 0x7fc8e8068990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8007fd0 tx=0x7fc8f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:01.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.136+0000 7fc8f6ffd640 1 -- 172.21.15.148:0/1973792725 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9000eb1e0 con 0x7fc908148f30 2023-10-30T13:19:01.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.312+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc90806fa10 con 0x7fc908148f30 2023-10-30T13:19:01.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc8f6ffd640 1 -- 172.21.15.148:0/1973792725 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fc9000ee800 con 0x7fc908148f30 2023-10-30T13:19:01.316 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:19:01.316 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:18:56.622559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:19:01.316 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 40 2023-10-30T13:19:01.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8e80664d0 msgr2=0x7fc8e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:01.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8e80664d0 0x7fc8e8068990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8007fd0 tx=0x7fc8f8009040 comp rx=0 tx=0).stop 2023-10-30T13:19:01.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908148f30 msgr2=0x7fc9081493e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:01.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908148f30 0x7fc9081493e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc90000a7a0 tx=0x7fc8f0003700 comp rx=0 tx=0).stop 2023-10-30T13:19:01.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 shutdown_connections 2023-10-30T13:19:01.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8e80664d0 0x7fc8e8068990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908148f30 0x7fc9081493e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9081e0250 0x7fc90814e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 --2- 172.21.15.148:0/1973792725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f000b410 0x7fc9081583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:01.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 >> 172.21.15.148:0/1973792725 conn(0x7fc9080c7910 msgr2=0x7fc908073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:01.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 shutdown_connections 2023-10-30T13:19:01.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:01.316+0000 7fc90caea640 1 -- 172.21.15.148:0/1973792725 wait complete. 2023-10-30T13:19:01.336 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:19:03.338 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:19:03.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd40d6c9640 1 Processor -- start 2023-10-30T13:19:03.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd40d6c9640 1 -- start start 2023-10-30T13:19:03.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd40d6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c90d0 0x7fd4080cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:03.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd40d6c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4080cf4d0 0x7fd40815d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd40d6c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40815ce30 con 0x7fd4080c8cf0 2023-10-30T13:19:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd40d6c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd408159880 con 0x7fd4080cf4d0 2023-10-30T13:19:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd40d6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd408159ae0 con 0x7fd4080c90d0 2023-10-30T13:19:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd406ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd4080c8cf0 0x7fd4080c9e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45690/0 (socket says 172.21.15.148:45690) 2023-10-30T13:19:03.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd406ffd640 1 -- 172.21.15.148:0/1173296773 learned_addr learned my addr 172.21.15.148:0/1173296773 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:19:03.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4067fc640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c90d0 0x7fd4080cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:03.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4077fe640 1 --2- 172.21.15.148:0/1173296773 >> v2:172.21.15.150:3300/0 conn(0x7fd4080cf4d0 0x7fd40815d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:03.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226285796 0 0) 0x7fd40815ce30 con 0x7fd4080c8cf0 2023-10-30T13:19:03.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f0003530 con 0x7fd4080c8cf0 2023-10-30T13:19:03.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257496808 0 0) 0x7fd3f0003530 con 0x7fd4080c8cf0 2023-10-30T13:19:03.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c90d0 msgr2=0x7fd4080cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:03.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c90d0 0x7fd4080cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 >> v2:172.21.15.150:3300/0 conn(0x7fd4080cf4d0 msgr2=0x7fd40815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:03.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 --2- 172.21.15.148:0/1173296773 >> v2:172.21.15.150:3300/0 conn(0x7fd4080cf4d0 0x7fd40815d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd408159dc0 con 0x7fd4080c8cf0 2023-10-30T13:19:03.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fd3f8001680 con 0x7fd4080c8cf0 2023-10-30T13:19:03.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 >> v1:172.21.15.148:6789/0 conn(0x7fd4080c8cf0 legacy=0x7fd4080c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:03.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c90d0 0x7fd3f0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:03.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3f0003f00 0x7fd3f00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:03.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd3f0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:03.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd408159880 con 0x7fd3f0004d10 2023-10-30T13:19:03.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd408159ae0 con 0x7fd3f0003f00 2023-10-30T13:19:03.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f000b990 con 0x7fd4080c90d0 2023-10-30T13:19:03.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4077fe640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd3f0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:03.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd406ffd640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c90d0 0x7fd3f0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:03.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4067fc640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3f0003f00 0x7fd3f00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:03.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4077fe640 1 -- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c90d0 msgr2=0x7fd3f0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:03.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4077fe640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4080c90d0 0x7fd3f0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4077fe640 1 -- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3f0003f00 msgr2=0x7fd3f00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:03.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4077fe640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3f0003f00 0x7fd3f00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4077fe640 1 -- 172.21.15.148:0/1173296773 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f0003530 con 0x7fd3f0004d10 2023-10-30T13:19:03.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.480+0000 7fd4067fc640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3f0003f00 0x7fd3f00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:19:03.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd4077fe640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd3f0005190 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd400009fc0 tx=0x7fd40001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3572fe27cb4ca9b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:03.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4000a5cb0 con 0x7fd3f0004d10 2023-10-30T13:19:03.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40000f040 con 0x7fd3f0004d10 2023-10-30T13:19:03.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd405ffb640 1 -- 172.21.15.148:0/1173296773 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4000ad680 con 0x7fd3f0004d10 2023-10-30T13:19:03.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd40d6c9640 1 -- 172.21.15.148:0/1173296773 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 msgr2=0x7fd3f0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:03.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd40d6c9640 1 --2- 172.21.15.148:0/1173296773 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd3f0005190 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd400009fc0 tx=0x7fd40001cc40 comp rx=0 tx=0).stop 2023-10-30T13:19:03.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd4077fe640 1 -- 172.21.15.148:0/1173296773 reap_dead start 2023-10-30T13:19:03.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd40d6c9640 1 -- 172.21.15.148:0/1173296773 shutdown_connections 2023-10-30T13:19:03.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd40d6c9640 1 -- 172.21.15.148:0/1173296773 >> 172.21.15.148:0/1173296773 conn(0x7fd40806be40 msgr2=0x7fd40806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:03.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd40d6c9640 1 -- 172.21.15.148:0/1173296773 shutdown_connections 2023-10-30T13:19:03.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.484+0000 7fd40d6c9640 1 -- 172.21.15.148:0/1173296773 wait complete. 2023-10-30T13:19:03.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 Processor -- start 2023-10-30T13:19:03.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 -- start start 2023-10-30T13:19:03.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd4080c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:03.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081506f0 0x7fd408156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:03.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4081eef50 0x7fd4080c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:03.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081ef520 con 0x7fd3f0004d10 2023-10-30T13:19:03.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080c74e0 con 0x7fd4081506f0 2023-10-30T13:19:03.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40814a410 con 0x7fd4081eef50 2023-10-30T13:19:03.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd4080c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:03.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd4080c6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53374/0 (socket says 172.21.15.148:53374) 2023-10-30T13:19:03.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd406ffd640 1 -- 172.21.15.148:0/4136572764 learned_addr learned my addr 172.21.15.148:0/4136572764 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:03.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd4067fc640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081506f0 0x7fd408156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:03.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd4077fe640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4081eef50 0x7fd4080c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:03.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd406ffd640 1 -- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4081eef50 msgr2=0x7fd4080c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:03.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd406ffd640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4081eef50 0x7fd4080c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd406ffd640 1 -- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081506f0 msgr2=0x7fd408156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:03.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd406ffd640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081506f0 0x7fd408156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd406ffd640 1 -- 172.21.15.148:0/4136572764 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4080cd100 con 0x7fd3f0004d10 2023-10-30T13:19:03.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd4077fe640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4081eef50 0x7fd4080c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:19:03.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd4067fc640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081506f0 0x7fd408156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:19:03.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd406ffd640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd4080c6fa0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8007310 tx=0x7fd3f8007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:03.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd3ef7fe640 1 -- 172.21.15.148:0/4136572764 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f800fa80 con 0x7fd3f0004d10 2023-10-30T13:19:03.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd3ef7fe640 1 -- 172.21.15.148:0/4136572764 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f800fc20 con 0x7fd3f0004d10 2023-10-30T13:19:03.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd3ef7fe640 1 -- 172.21.15.148:0/4136572764 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f80133f0 con 0x7fd3f0004d10 2023-10-30T13:19:03.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4080ca100 con 0x7fd3f0004d10 2023-10-30T13:19:03.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.488+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4080cd680 con 0x7fd3f0004d10 2023-10-30T13:19:03.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.492+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40804e170 con 0x7fd3f0004d10 2023-10-30T13:19:03.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.492+0000 7fd3ef7fe640 1 -- 172.21.15.148:0/4136572764 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3f8013550 con 0x7fd3f0004d10 2023-10-30T13:19:03.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.492+0000 7fd3ef7fe640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3f40664a0 0x7fd3f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:03.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.492+0000 7fd3ef7fe640 1 -- 172.21.15.148:0/4136572764 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd3f8084480 con 0x7fd3f0004d10 2023-10-30T13:19:03.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.492+0000 7fd4067fc640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3f40664a0 0x7fd3f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:03.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.492+0000 7fd4067fc640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3f40664a0 0x7fd3f4068960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd408150430 tx=0x7fd3fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:03.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.496+0000 7fd3ef7fe640 1 -- 172.21.15.148:0/4136572764 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3f8051670 con 0x7fd3f0004d10 2023-10-30T13:19:03.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.676+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4080ca9d0 con 0x7fd3f0004d10 2023-10-30T13:19:03.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.676+0000 7fd3ef7fe640 1 -- 172.21.15.148:0/4136572764 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fd3f8054c90 con 0x7fd3f0004d10 2023-10-30T13:19:03.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:19:03.679 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:19:01.727168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:19:03.680 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 41 2023-10-30T13:19:03.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3f40664a0 msgr2=0x7fd3f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:03.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3f40664a0 0x7fd3f4068960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd408150430 tx=0x7fd3fc008040 comp rx=0 tx=0).stop 2023-10-30T13:19:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 msgr2=0x7fd4080c6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd4080c6fa0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8007310 tx=0x7fd3f8007710 comp rx=0 tx=0).stop 2023-10-30T13:19:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 shutdown_connections 2023-10-30T13:19:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3f40664a0 0x7fd3f4068960 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4081eef50 0x7fd4080c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4081506f0 0x7fd408156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 --2- 172.21.15.148:0/4136572764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f0004d10 0x7fd4080c6fa0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:03.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 >> 172.21.15.148:0/4136572764 conn(0x7fd40806be40 msgr2=0x7fd408157a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:03.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 shutdown_connections 2023-10-30T13:19:03.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:03.680+0000 7fd40d6c9640 1 -- 172.21.15.148:0/4136572764 wait complete. 2023-10-30T13:19:03.700 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4285 2023-10-30T13:19:03.701 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.e 2023-10-30T13:19:09.701 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:19:09.702 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:19:09.702 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:19:09.706 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:19:09.706 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:19:09.796 INFO:tasks.ceph.mds.e.smithi148.stdout:starting mds.e at 2023-10-30T13:19:09.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd3830d6640 1 Processor -- start 2023-10-30T13:19:09.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd3830d6640 1 -- start start 2023-10-30T13:19:09.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd3830d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd37c0c05d0 0x7fd37c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:09.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd3830d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c6f60 0x7fd37c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:09.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd3830d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c0c0a50 con 0x7fd37c0c6b80 2023-10-30T13:19:09.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd3830d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd37c06f6e0 con 0x7fd37c0c05d0 2023-10-30T13:19:09.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd3830d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c06f9c0 con 0x7fd37c0c6f60 2023-10-30T13:19:09.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd37bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd37c0c05d0 0x7fd37c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:09.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd37bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd37c0c05d0 0x7fd37c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53412/0 (socket says 172.21.15.148:53412) 2023-10-30T13:19:09.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd37bfff640 1 -- 172.21.15.148:0/2399082804 learned_addr learned my addr 172.21.15.148:0/2399082804 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd381673640 1 --2- 172.21.15.148:0/2399082804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c6f60 0x7fd37c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd37bfff640 1 -- 172.21.15.148:0/2399082804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c6f60 msgr2=0x7fd37c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd37bfff640 1 --2- 172.21.15.148:0/2399082804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c6f60 0x7fd37c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd37bfff640 1 -- 172.21.15.148:0/2399082804 >> v1:172.21.15.150:6789/0 conn(0x7fd37c0c6b80 legacy=0x7fd37c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:09.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.852+0000 7fd37bfff640 1 -- 172.21.15.148:0/2399082804 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c06fca0 con 0x7fd37c0c05d0 2023-10-30T13:19:09.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd37bfff640 1 --2- 172.21.15.148:0/2399082804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c05d0 0x7fd37c13ed90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd368009870 tx=0x7fd36801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591daafc7114047f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:09.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd37b7fe640 1 -- 172.21.15.148:0/2399082804 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680ab650 con 0x7fd37c0c05d0 2023-10-30T13:19:09.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd37b7fe640 1 -- 172.21.15.148:0/2399082804 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3680ab7b0 con 0x7fd37c0c05d0 2023-10-30T13:19:09.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd37b7fe640 1 -- 172.21.15.148:0/2399082804 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680abb40 con 0x7fd37c0c05d0 2023-10-30T13:19:09.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 -- 172.21.15.148:0/2399082804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c05d0 msgr2=0x7fd37c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:09.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 --2- 172.21.15.148:0/2399082804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c05d0 0x7fd37c13ed90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd368009870 tx=0x7fd36801c540 comp rx=0 tx=0).stop 2023-10-30T13:19:09.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 -- 172.21.15.148:0/2399082804 shutdown_connections 2023-10-30T13:19:09.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 --2- 172.21.15.148:0/2399082804 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c6f60 0x7fd37c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:09.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 --2- 172.21.15.148:0/2399082804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c05d0 0x7fd37c13ed90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:09.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 -- 172.21.15.148:0/2399082804 >> 172.21.15.148:0/2399082804 conn(0x7fd37c0c7910 msgr2=0x7fd37c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:09.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 -- 172.21.15.148:0/2399082804 shutdown_connections 2023-10-30T13:19:09.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 -- 172.21.15.148:0/2399082804 wait complete. 2023-10-30T13:19:09.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 Processor -- start 2023-10-30T13:19:09.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.856+0000 7fd3830d6640 1 -- start start 2023-10-30T13:19:09.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3830d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c05d0 0x7fd37c154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3830d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c6b80 0x7fd37c154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3830d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd37c0c6f60 0x7fd37c132550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3830d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0ca950 con 0x7fd37c0c6b80 2023-10-30T13:19:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3830d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0c8c10 con 0x7fd37c0c6f60 2023-10-30T13:19:09.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3830d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c14ee30 con 0x7fd37c0c05d0 2023-10-30T13:19:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd37bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c6b80 0x7fd37c154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd37bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c6b80 0x7fd37c154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53426/0 (socket says 172.21.15.148:53426) 2023-10-30T13:19:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd37bfff640 1 -- 172.21.15.148:0/2290425446 learned_addr learned my addr 172.21.15.148:0/2290425446 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd381673640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd37c0c6f60 0x7fd37c132550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd380e72640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c05d0 0x7fd37c154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:09.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd37bfff640 1 -- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c05d0 msgr2=0x7fd37c154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:09.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd37bfff640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c05d0 0x7fd37c154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:09.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd37bfff640 1 -- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd37c0c6f60 msgr2=0x7fd37c132550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:09.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd37bfff640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd37c0c6f60 0x7fd37c132550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:09.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd37bfff640 1 -- 172.21.15.148:0/2290425446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3680094d0 con 0x7fd37c0c6b80 2023-10-30T13:19:09.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd381673640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd37c0c6f60 0x7fd37c132550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:19:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd37bfff640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c6b80 0x7fd37c154c40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd36801a170 tx=0x7fd36801ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3797fa640 1 -- 172.21.15.148:0/2290425446 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680aa980 con 0x7fd37c0c6b80 2023-10-30T13:19:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3797fa640 1 -- 172.21.15.148:0/2290425446 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3680aab00 con 0x7fd37c0c6b80 2023-10-30T13:19:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3797fa640 1 -- 172.21.15.148:0/2290425446 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680aaeb0 con 0x7fd37c0c6b80 2023-10-30T13:19:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37c0c9e10 con 0x7fd37c0c6b80 2023-10-30T13:19:09.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37c0ca390 con 0x7fd37c0c6b80 2023-10-30T13:19:09.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.860+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37c14efa0 con 0x7fd37c0c6b80 2023-10-30T13:19:09.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.864+0000 7fd3797fa640 1 -- 172.21.15.148:0/2290425446 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3680b43f0 con 0x7fd37c0c6b80 2023-10-30T13:19:09.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.864+0000 7fd3797fa640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3600664d0 0x7fd360068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:09.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.864+0000 7fd3797fa640 1 -- 172.21.15.148:0/2290425446 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd36811edf0 con 0x7fd37c0c6b80 2023-10-30T13:19:09.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.864+0000 7fd380e72640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3600664d0 0x7fd360068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:09.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.864+0000 7fd380e72640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3600664d0 0x7fd360068990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd370000f70 tx=0x7fd370009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:09.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:09.868+0000 7fd3797fa640 1 -- 172.21.15.148:0/2290425446 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3680ec090 con 0x7fd37c0c6b80 2023-10-30T13:19:10.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.064+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd37c04e170 con 0x7fd37c0c6b80 2023-10-30T13:19:10.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.064+0000 7fd3797fa640 1 -- 172.21.15.148:0/2290425446 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fd3680ef6b0 con 0x7fd37c0c6b80 2023-10-30T13:19:10.066 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:19:10.067 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:19:01.727168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:19:10.067 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 41 2023-10-30T13:19:10.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3600664d0 msgr2=0x7fd360068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:10.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3600664d0 0x7fd360068990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd370000f70 tx=0x7fd370009040 comp rx=0 tx=0).stop 2023-10-30T13:19:10.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c6b80 msgr2=0x7fd37c154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:10.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c6b80 0x7fd37c154c40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd36801a170 tx=0x7fd36801ad80 comp rx=0 tx=0).stop 2023-10-30T13:19:10.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 shutdown_connections 2023-10-30T13:19:10.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3600664d0 0x7fd360068990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:10.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd37c0c6b80 0x7fd37c154c40 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:10.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd37c0c6f60 0x7fd37c132550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:10.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 --2- 172.21.15.148:0/2290425446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd37c0c05d0 0x7fd37c154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:10.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 >> 172.21.15.148:0/2290425446 conn(0x7fd37c0c7910 msgr2=0x7fd37c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:10.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 shutdown_connections 2023-10-30T13:19:10.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:10.068+0000 7fd3830d6640 1 -- 172.21.15.148:0/2290425446 wait complete. 2023-10-30T13:19:10.087 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T13:19:10.221 INFO:tasks.workunit.client.0.smithi148.stdout:removed './128/37' 2023-10-30T13:19:10.221 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './128' 2023-10-30T13:19:10.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/19' 2023-10-30T13:19:10.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/15' 2023-10-30T13:19:10.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/31' 2023-10-30T13:19:10.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/36' 2023-10-30T13:19:10.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/20' 2023-10-30T13:19:10.222 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/12' 2023-10-30T13:19:10.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/39' 2023-10-30T13:19:10.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/10' 2023-10-30T13:19:10.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/34' 2023-10-30T13:19:10.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/38' 2023-10-30T13:19:10.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/11' 2023-10-30T13:19:10.223 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/16' 2023-10-30T13:19:10.224 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/32' 2023-10-30T13:19:10.224 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/30' 2023-10-30T13:19:10.224 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/14' 2023-10-30T13:19:10.224 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/18' 2023-10-30T13:19:10.224 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/13' 2023-10-30T13:19:10.225 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/33' 2023-10-30T13:19:10.225 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/17' 2023-10-30T13:19:10.225 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/35' 2023-10-30T13:19:10.225 INFO:tasks.workunit.client.0.smithi148.stdout:removed './129/37' 2023-10-30T13:19:10.225 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './129' 2023-10-30T13:19:10.225 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/19' 2023-10-30T13:19:10.226 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/15' 2023-10-30T13:19:10.226 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/31' 2023-10-30T13:19:10.226 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/36' 2023-10-30T13:19:10.226 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/20' 2023-10-30T13:19:10.226 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/12' 2023-10-30T13:19:10.227 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/39' 2023-10-30T13:19:10.227 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/10' 2023-10-30T13:19:10.227 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/34' 2023-10-30T13:19:10.227 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/38' 2023-10-30T13:19:10.227 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/11' 2023-10-30T13:19:10.227 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/16' 2023-10-30T13:19:10.228 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/32' 2023-10-30T13:19:10.228 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/30' 2023-10-30T13:19:10.228 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/14' 2023-10-30T13:19:10.228 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/18' 2023-10-30T13:19:10.228 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/13' 2023-10-30T13:19:10.229 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/33' 2023-10-30T13:19:10.229 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/17' 2023-10-30T13:19:10.229 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/35' 2023-10-30T13:19:10.229 INFO:tasks.workunit.client.0.smithi148.stdout:removed './130/37' 2023-10-30T13:19:10.229 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './130' 2023-10-30T13:19:10.229 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/19' 2023-10-30T13:19:10.230 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/15' 2023-10-30T13:19:10.230 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/31' 2023-10-30T13:19:10.230 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/36' 2023-10-30T13:19:10.230 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/20' 2023-10-30T13:19:10.230 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/12' 2023-10-30T13:19:10.230 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/39' 2023-10-30T13:19:10.231 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/10' 2023-10-30T13:19:10.231 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/34' 2023-10-30T13:19:10.231 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/38' 2023-10-30T13:19:10.231 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/11' 2023-10-30T13:19:10.232 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/16' 2023-10-30T13:19:10.232 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/32' 2023-10-30T13:19:10.232 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/30' 2023-10-30T13:19:10.232 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/14' 2023-10-30T13:19:10.232 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/18' 2023-10-30T13:19:10.232 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/13' 2023-10-30T13:19:10.233 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/33' 2023-10-30T13:19:10.233 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/17' 2023-10-30T13:19:10.233 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/35' 2023-10-30T13:19:10.233 INFO:tasks.workunit.client.0.smithi148.stdout:removed './131/37' 2023-10-30T13:19:10.233 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './131' 2023-10-30T13:19:10.233 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/19' 2023-10-30T13:19:10.234 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/15' 2023-10-30T13:19:10.234 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/31' 2023-10-30T13:19:10.234 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/36' 2023-10-30T13:19:10.234 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/20' 2023-10-30T13:19:10.234 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/12' 2023-10-30T13:19:10.235 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/39' 2023-10-30T13:19:10.235 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/10' 2023-10-30T13:19:10.235 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/34' 2023-10-30T13:19:10.235 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/38' 2023-10-30T13:19:10.235 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/11' 2023-10-30T13:19:10.236 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/16' 2023-10-30T13:19:10.236 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/32' 2023-10-30T13:19:10.236 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/30' 2023-10-30T13:19:10.236 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/14' 2023-10-30T13:19:10.236 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/18' 2023-10-30T13:19:10.237 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/13' 2023-10-30T13:19:10.237 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/33' 2023-10-30T13:19:10.237 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/17' 2023-10-30T13:19:10.237 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/35' 2023-10-30T13:19:10.237 INFO:tasks.workunit.client.0.smithi148.stdout:removed './132/37' 2023-10-30T13:19:10.237 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './132' 2023-10-30T13:19:10.238 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/19' 2023-10-30T13:19:10.238 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/15' 2023-10-30T13:19:10.238 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/31' 2023-10-30T13:19:10.238 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/36' 2023-10-30T13:19:10.238 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/20' 2023-10-30T13:19:10.239 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/12' 2023-10-30T13:19:10.239 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/39' 2023-10-30T13:19:10.239 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/10' 2023-10-30T13:19:10.239 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/34' 2023-10-30T13:19:10.239 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/38' 2023-10-30T13:19:10.239 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/11' 2023-10-30T13:19:10.240 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/16' 2023-10-30T13:19:10.240 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/32' 2023-10-30T13:19:10.240 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/30' 2023-10-30T13:19:10.240 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/14' 2023-10-30T13:19:10.240 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/18' 2023-10-30T13:19:10.240 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/13' 2023-10-30T13:19:10.241 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/33' 2023-10-30T13:19:10.241 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/17' 2023-10-30T13:19:10.241 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/35' 2023-10-30T13:19:10.241 INFO:tasks.workunit.client.0.smithi148.stdout:removed './133/37' 2023-10-30T13:19:10.241 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './133' 2023-10-30T13:19:10.241 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/19' 2023-10-30T13:19:10.242 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/15' 2023-10-30T13:19:10.242 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/31' 2023-10-30T13:19:10.242 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/36' 2023-10-30T13:19:10.242 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/20' 2023-10-30T13:19:10.242 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/12' 2023-10-30T13:19:10.242 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/39' 2023-10-30T13:19:10.243 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/10' 2023-10-30T13:19:10.243 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/34' 2023-10-30T13:19:10.243 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/38' 2023-10-30T13:19:10.243 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/11' 2023-10-30T13:19:10.243 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/16' 2023-10-30T13:19:10.243 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/32' 2023-10-30T13:19:10.243 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/30' 2023-10-30T13:19:10.244 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/14' 2023-10-30T13:19:10.244 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/18' 2023-10-30T13:19:10.244 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/13' 2023-10-30T13:19:10.244 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/33' 2023-10-30T13:19:10.244 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/17' 2023-10-30T13:19:10.244 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/35' 2023-10-30T13:19:10.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './134/37' 2023-10-30T13:19:10.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './134' 2023-10-30T13:19:10.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/19' 2023-10-30T13:19:10.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/15' 2023-10-30T13:19:10.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/31' 2023-10-30T13:19:10.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/36' 2023-10-30T13:19:10.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/20' 2023-10-30T13:19:10.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/12' 2023-10-30T13:19:10.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/39' 2023-10-30T13:19:10.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/10' 2023-10-30T13:19:10.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/34' 2023-10-30T13:19:10.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/38' 2023-10-30T13:19:10.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/11' 2023-10-30T13:19:10.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/16' 2023-10-30T13:19:10.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/32' 2023-10-30T13:19:10.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/30' 2023-10-30T13:19:10.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/14' 2023-10-30T13:19:10.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/18' 2023-10-30T13:19:10.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/13' 2023-10-30T13:19:10.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/33' 2023-10-30T13:19:10.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/17' 2023-10-30T13:19:10.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/35' 2023-10-30T13:19:10.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './135/37' 2023-10-30T13:19:10.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './135' 2023-10-30T13:19:10.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/19' 2023-10-30T13:19:10.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/15' 2023-10-30T13:19:10.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/31' 2023-10-30T13:19:10.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/36' 2023-10-30T13:19:10.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/20' 2023-10-30T13:19:10.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/12' 2023-10-30T13:19:10.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/39' 2023-10-30T13:19:10.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/10' 2023-10-30T13:19:10.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/34' 2023-10-30T13:19:10.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/38' 2023-10-30T13:19:10.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/11' 2023-10-30T13:19:10.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/16' 2023-10-30T13:19:10.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/32' 2023-10-30T13:19:10.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/30' 2023-10-30T13:19:10.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/14' 2023-10-30T13:19:10.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/18' 2023-10-30T13:19:10.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/13' 2023-10-30T13:19:10.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/33' 2023-10-30T13:19:10.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/17' 2023-10-30T13:19:10.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/35' 2023-10-30T13:19:10.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './136/37' 2023-10-30T13:19:10.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './136' 2023-10-30T13:19:10.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/19' 2023-10-30T13:19:10.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/15' 2023-10-30T13:19:10.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/31' 2023-10-30T13:19:10.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/36' 2023-10-30T13:19:10.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/20' 2023-10-30T13:19:10.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/12' 2023-10-30T13:19:10.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/39' 2023-10-30T13:19:10.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/10' 2023-10-30T13:19:10.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/34' 2023-10-30T13:19:10.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/38' 2023-10-30T13:19:10.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/11' 2023-10-30T13:19:10.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/16' 2023-10-30T13:19:10.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/32' 2023-10-30T13:19:10.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/30' 2023-10-30T13:19:10.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/14' 2023-10-30T13:19:10.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/18' 2023-10-30T13:19:10.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/13' 2023-10-30T13:19:10.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/33' 2023-10-30T13:19:10.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/17' 2023-10-30T13:19:10.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/35' 2023-10-30T13:19:10.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './137/37' 2023-10-30T13:19:10.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './137' 2023-10-30T13:19:10.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/19' 2023-10-30T13:19:10.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/15' 2023-10-30T13:19:10.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/31' 2023-10-30T13:19:10.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/36' 2023-10-30T13:19:10.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/20' 2023-10-30T13:19:10.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/12' 2023-10-30T13:19:10.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/39' 2023-10-30T13:19:10.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/10' 2023-10-30T13:19:10.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/34' 2023-10-30T13:19:10.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/38' 2023-10-30T13:19:10.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/11' 2023-10-30T13:19:10.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/16' 2023-10-30T13:19:12.088 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:19:12.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.244+0000 7fabd5965640 1 Processor -- start 2023-10-30T13:19:12.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.244+0000 7fabd5965640 1 -- start start 2023-10-30T13:19:12.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabd5965640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fabd01538a0 0x7fabd015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:12.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabd5965640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fabd00cf160 0x7fabd00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:12.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabd5965640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd00cb800 con 0x7fabd00ca0e0 2023-10-30T13:19:12.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabd5965640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd015dc30 con 0x7fabd00cf160 2023-10-30T13:19:12.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabd5965640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd00cb330 con 0x7fabd01538a0 2023-10-30T13:19:12.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabcf7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fabd00cf160 0x7fabd00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:12.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabcf7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fabd00cf160 0x7fabd00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34202/0 (socket says 172.21.15.148:34202) 2023-10-30T13:19:12.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabcf7fe640 1 -- 172.21.15.148:0/155463160 learned_addr learned my addr 172.21.15.148:0/155463160 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:12.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabce7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fabd00ca0e0 0x7fabd015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41234/0 (socket says 172.21.15.148:41234) 2023-10-30T13:19:12.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabceffd640 1 --2- 172.21.15.148:0/155463160 >> v2:172.21.15.150:3300/0 conn(0x7fabd01538a0 0x7fabd015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:12.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabcf7fe640 1 -- 172.21.15.148:0/155463160 >> v2:172.21.15.150:3300/0 conn(0x7fabd01538a0 msgr2=0x7fabd015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:12.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabcf7fe640 1 --2- 172.21.15.148:0/155463160 >> v2:172.21.15.150:3300/0 conn(0x7fabd01538a0 0x7fabd015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabcf7fe640 1 -- 172.21.15.148:0/155463160 >> v1:172.21.15.148:6789/0 conn(0x7fabd00ca0e0 legacy=0x7fabd015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:12.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.248+0000 7fabcf7fe640 1 -- 172.21.15.148:0/155463160 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd00cad70 con 0x7fabd00cf160 2023-10-30T13:19:12.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabcf7fe640 1 --2- 172.21.15.148:0/155463160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00cf160 0x7fabd00cf590 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fabd00cfd10 tx=0x7fabc801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7caa07af0b19e33 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:12.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabcdffb640 1 -- 172.21.15.148:0/155463160 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc80af400 con 0x7fabd00cf160 2023-10-30T13:19:12.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabcdffb640 1 -- 172.21.15.148:0/155463160 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc80ab050 con 0x7fabd00cf160 2023-10-30T13:19:12.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabcdffb640 1 -- 172.21.15.148:0/155463160 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc80af780 con 0x7fabd00cf160 2023-10-30T13:19:12.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabd5965640 1 -- 172.21.15.148:0/155463160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00cf160 msgr2=0x7fabd00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:12.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabd5965640 1 --2- 172.21.15.148:0/155463160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00cf160 0x7fabd00cf590 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fabd00cfd10 tx=0x7fabc801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabd5965640 1 -- 172.21.15.148:0/155463160 shutdown_connections 2023-10-30T13:19:12.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabd5965640 1 --2- 172.21.15.148:0/155463160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00cf160 0x7fabd00cf590 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabd5965640 1 --2- 172.21.15.148:0/155463160 >> v2:172.21.15.150:3300/0 conn(0x7fabd01538a0 0x7fabd015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabd5965640 1 -- 172.21.15.148:0/155463160 >> 172.21.15.148:0/155463160 conn(0x7fabd006bfe0 msgr2=0x7fabd006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:12.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabd5965640 1 -- 172.21.15.148:0/155463160 shutdown_connections 2023-10-30T13:19:12.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.252+0000 7fabd5965640 1 -- 172.21.15.148:0/155463160 wait complete. 2023-10-30T13:19:12.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabd5965640 1 Processor -- start 2023-10-30T13:19:12.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabd5965640 1 -- start start 2023-10-30T13:19:12.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabd5965640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00ca0e0 0x7fabd01480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:12.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabd5965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fabd00cf160 0x7fabd0158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:12.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabd5965640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd01538a0 0x7fabd01f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:12.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabd5965640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01e5330 con 0x7fabd00ca0e0 2023-10-30T13:19:12.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabd5965640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd00b4f30 con 0x7fabd01538a0 2023-10-30T13:19:12.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabd5965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01e4d70 con 0x7fabd00cf160 2023-10-30T13:19:12.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabceffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00ca0e0 0x7fabd01480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:12.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabceffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00ca0e0 0x7fabd01480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34208/0 (socket says 172.21.15.148:34208) 2023-10-30T13:19:12.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabceffd640 1 -- 172.21.15.148:0/694134131 learned_addr learned my addr 172.21.15.148:0/694134131 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:12.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabcf7fe640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd01538a0 0x7fabd01f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:12.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.256+0000 7fabce7fc640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fabd00cf160 0x7fabd0158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:12.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabceffd640 1 -- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fabd00cf160 msgr2=0x7fabd0158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:12.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabceffd640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fabd00cf160 0x7fabd0158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabceffd640 1 -- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd01538a0 msgr2=0x7fabd01f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:12.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabceffd640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd01538a0 0x7fabd01f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabceffd640 1 -- 172.21.15.148:0/694134131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc8009b30 con 0x7fabd00ca0e0 2023-10-30T13:19:12.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabceffd640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00ca0e0 0x7fabd01480f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fabc000c7f0 tx=0x7fabc000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:12.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabb7fff640 1 -- 172.21.15.148:0/694134131 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0004220 con 0x7fabd00ca0e0 2023-10-30T13:19:12.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabb7fff640 1 -- 172.21.15.148:0/694134131 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc00043a0 con 0x7fabd00ca0e0 2023-10-30T13:19:12.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabb7fff640 1 -- 172.21.15.148:0/694134131 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0004510 con 0x7fabd00ca0e0 2023-10-30T13:19:12.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd01e5050 con 0x7fabd00ca0e0 2023-10-30T13:19:12.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd00b4bc0 con 0x7fabd00ca0e0 2023-10-30T13:19:12.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd00b4d80 con 0x7fabd00ca0e0 2023-10-30T13:19:12.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.260+0000 7fabb7fff640 1 -- 172.21.15.148:0/694134131 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fabc0003a70 con 0x7fabd00ca0e0 2023-10-30T13:19:12.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.264+0000 7fabb7fff640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabbc0664a0 0x7fabbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:12.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.264+0000 7fabb7fff640 1 -- 172.21.15.148:0/694134131 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fabc0083ff0 con 0x7fabd00ca0e0 2023-10-30T13:19:12.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.264+0000 7fabce7fc640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabbc0664a0 0x7fabbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:12.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.264+0000 7fabce7fc640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabbc0664a0 0x7fabbc068960 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fabd01e45c0 tx=0x7fabc4007e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:12.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.268+0000 7fabb7fff640 1 -- 172.21.15.148:0/694134131 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fabc0051260 con 0x7fabd00ca0e0 2023-10-30T13:19:12.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabd00b2a50 con 0x7fabd00ca0e0 2023-10-30T13:19:12.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabb7fff640 1 -- 172.21.15.148:0/694134131 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6533 (secure 0 0 0) 0x7fabc001e020 con 0x7fabd00ca0e0 2023-10-30T13:19:12.457 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:19:12.457 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:19:01.727168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:19:12.458 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 42 2023-10-30T13:19:12.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabbc0664a0 msgr2=0x7fabbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:12.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabbc0664a0 0x7fabbc068960 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fabd01e45c0 tx=0x7fabc4007e50 comp rx=0 tx=0).stop 2023-10-30T13:19:12.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00ca0e0 msgr2=0x7fabd01480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:12.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00ca0e0 0x7fabd01480f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fabc000c7f0 tx=0x7fabc000ccb0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 shutdown_connections 2023-10-30T13:19:12.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabbc0664a0 0x7fabbc068960 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd01538a0 0x7fabd01f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fabd00cf160 0x7fabd0158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 --2- 172.21.15.148:0/694134131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabd00ca0e0 0x7fabd01480f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:12.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 >> 172.21.15.148:0/694134131 conn(0x7fabd006bfe0 msgr2=0x7fabd006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:12.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 shutdown_connections 2023-10-30T13:19:12.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:12.456+0000 7fabd5965640 1 -- 172.21.15.148:0/694134131 wait complete. 2023-10-30T13:19:12.478 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:19:12.478 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2023-10-30T13:19:12.540 INFO:tasks.ceph.mds.d.smithi150.stderr:2023-10-30T13:19:12.540+0000 7fe56d311640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:19:12.541 INFO:tasks.ceph.mds.d.smithi150.stderr:2023-10-30T13:19:12.540+0000 7fe56d311640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:19:12.541 INFO:tasks.ceph.mds.d.smithi150.stderr:2023-10-30T13:19:12.540+0000 7fe56d311640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:19:14.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/32' 2023-10-30T13:19:14.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/30' 2023-10-30T13:19:14.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/14' 2023-10-30T13:19:14.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/18' 2023-10-30T13:19:14.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/13' 2023-10-30T13:19:14.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/33' 2023-10-30T13:19:14.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/17' 2023-10-30T13:19:14.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/35' 2023-10-30T13:19:14.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './138/37' 2023-10-30T13:19:14.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './138' 2023-10-30T13:19:14.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/19' 2023-10-30T13:19:14.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/15' 2023-10-30T13:19:14.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/31' 2023-10-30T13:19:14.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/36' 2023-10-30T13:19:14.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/20' 2023-10-30T13:19:14.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/12' 2023-10-30T13:19:14.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/39' 2023-10-30T13:19:14.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/10' 2023-10-30T13:19:14.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/34' 2023-10-30T13:19:14.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/38' 2023-10-30T13:19:14.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/11' 2023-10-30T13:19:14.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/16' 2023-10-30T13:19:14.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/32' 2023-10-30T13:19:14.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/30' 2023-10-30T13:19:14.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/14' 2023-10-30T13:19:14.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/18' 2023-10-30T13:19:14.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/13' 2023-10-30T13:19:14.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/33' 2023-10-30T13:19:14.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/17' 2023-10-30T13:19:14.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/35' 2023-10-30T13:19:14.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './139/37' 2023-10-30T13:19:14.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './139' 2023-10-30T13:19:14.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/19' 2023-10-30T13:19:14.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/15' 2023-10-30T13:19:14.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/31' 2023-10-30T13:19:14.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/36' 2023-10-30T13:19:14.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/20' 2023-10-30T13:19:14.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/12' 2023-10-30T13:19:14.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/39' 2023-10-30T13:19:14.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/10' 2023-10-30T13:19:14.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/34' 2023-10-30T13:19:14.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/38' 2023-10-30T13:19:14.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/11' 2023-10-30T13:19:14.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/16' 2023-10-30T13:19:14.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/32' 2023-10-30T13:19:14.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/30' 2023-10-30T13:19:14.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/14' 2023-10-30T13:19:14.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/18' 2023-10-30T13:19:14.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/13' 2023-10-30T13:19:14.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/33' 2023-10-30T13:19:14.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/17' 2023-10-30T13:19:14.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/35' 2023-10-30T13:19:14.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './140/37' 2023-10-30T13:19:14.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './140' 2023-10-30T13:19:14.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/19' 2023-10-30T13:19:14.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/15' 2023-10-30T13:19:14.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/31' 2023-10-30T13:19:14.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/36' 2023-10-30T13:19:14.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/20' 2023-10-30T13:19:14.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/12' 2023-10-30T13:19:14.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/39' 2023-10-30T13:19:14.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/10' 2023-10-30T13:19:14.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/34' 2023-10-30T13:19:14.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/38' 2023-10-30T13:19:14.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/11' 2023-10-30T13:19:14.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/16' 2023-10-30T13:19:14.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/32' 2023-10-30T13:19:14.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/30' 2023-10-30T13:19:14.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/14' 2023-10-30T13:19:14.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/18' 2023-10-30T13:19:14.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/13' 2023-10-30T13:19:14.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/33' 2023-10-30T13:19:14.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/17' 2023-10-30T13:19:14.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/35' 2023-10-30T13:19:14.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './141/37' 2023-10-30T13:19:14.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './141' 2023-10-30T13:19:14.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/19' 2023-10-30T13:19:14.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/15' 2023-10-30T13:19:14.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/31' 2023-10-30T13:19:14.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/36' 2023-10-30T13:19:14.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/20' 2023-10-30T13:19:14.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/12' 2023-10-30T13:19:14.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/39' 2023-10-30T13:19:14.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/10' 2023-10-30T13:19:14.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/34' 2023-10-30T13:19:14.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/38' 2023-10-30T13:19:14.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/11' 2023-10-30T13:19:14.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/16' 2023-10-30T13:19:14.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/32' 2023-10-30T13:19:14.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/30' 2023-10-30T13:19:14.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/14' 2023-10-30T13:19:14.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/18' 2023-10-30T13:19:14.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/13' 2023-10-30T13:19:14.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/33' 2023-10-30T13:19:14.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/17' 2023-10-30T13:19:14.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/35' 2023-10-30T13:19:14.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './142/37' 2023-10-30T13:19:14.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './142' 2023-10-30T13:19:14.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/19' 2023-10-30T13:19:14.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/15' 2023-10-30T13:19:14.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/31' 2023-10-30T13:19:14.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/36' 2023-10-30T13:19:14.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/20' 2023-10-30T13:19:14.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/12' 2023-10-30T13:19:14.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/39' 2023-10-30T13:19:14.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/10' 2023-10-30T13:19:14.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/34' 2023-10-30T13:19:14.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/38' 2023-10-30T13:19:14.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/11' 2023-10-30T13:19:14.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/16' 2023-10-30T13:19:14.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/32' 2023-10-30T13:19:14.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/30' 2023-10-30T13:19:14.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/14' 2023-10-30T13:19:14.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/18' 2023-10-30T13:19:14.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/13' 2023-10-30T13:19:14.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/33' 2023-10-30T13:19:14.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/17' 2023-10-30T13:19:14.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/35' 2023-10-30T13:19:14.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './143/37' 2023-10-30T13:19:14.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './143' 2023-10-30T13:19:14.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/19' 2023-10-30T13:19:14.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/15' 2023-10-30T13:19:14.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/31' 2023-10-30T13:19:14.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/36' 2023-10-30T13:19:14.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/20' 2023-10-30T13:19:14.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/12' 2023-10-30T13:19:14.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/39' 2023-10-30T13:19:14.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/10' 2023-10-30T13:19:14.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/34' 2023-10-30T13:19:14.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/38' 2023-10-30T13:19:14.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/11' 2023-10-30T13:19:14.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/16' 2023-10-30T13:19:14.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/32' 2023-10-30T13:19:14.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/30' 2023-10-30T13:19:14.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/14' 2023-10-30T13:19:14.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/18' 2023-10-30T13:19:14.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/13' 2023-10-30T13:19:14.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/33' 2023-10-30T13:19:14.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/17' 2023-10-30T13:19:14.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/35' 2023-10-30T13:19:14.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './144/37' 2023-10-30T13:19:14.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './144' 2023-10-30T13:19:14.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/19' 2023-10-30T13:19:14.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/15' 2023-10-30T13:19:14.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/31' 2023-10-30T13:19:14.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/36' 2023-10-30T13:19:14.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/20' 2023-10-30T13:19:14.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/12' 2023-10-30T13:19:14.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/39' 2023-10-30T13:19:14.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/10' 2023-10-30T13:19:14.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/34' 2023-10-30T13:19:14.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/38' 2023-10-30T13:19:14.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/11' 2023-10-30T13:19:14.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/16' 2023-10-30T13:19:14.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/32' 2023-10-30T13:19:14.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/30' 2023-10-30T13:19:14.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/14' 2023-10-30T13:19:14.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/18' 2023-10-30T13:19:14.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/13' 2023-10-30T13:19:14.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/33' 2023-10-30T13:19:14.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/17' 2023-10-30T13:19:14.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/35' 2023-10-30T13:19:14.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './145/37' 2023-10-30T13:19:14.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './145' 2023-10-30T13:19:14.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/19' 2023-10-30T13:19:14.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/15' 2023-10-30T13:19:14.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/31' 2023-10-30T13:19:14.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/36' 2023-10-30T13:19:14.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/20' 2023-10-30T13:19:14.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/12' 2023-10-30T13:19:14.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/39' 2023-10-30T13:19:14.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/10' 2023-10-30T13:19:14.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/34' 2023-10-30T13:19:14.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/38' 2023-10-30T13:19:14.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/11' 2023-10-30T13:19:14.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/16' 2023-10-30T13:19:14.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/32' 2023-10-30T13:19:14.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/30' 2023-10-30T13:19:14.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/14' 2023-10-30T13:19:14.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/18' 2023-10-30T13:19:14.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/13' 2023-10-30T13:19:14.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/33' 2023-10-30T13:19:14.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/17' 2023-10-30T13:19:14.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/35' 2023-10-30T13:19:14.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './146/37' 2023-10-30T13:19:14.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './146' 2023-10-30T13:19:14.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/19' 2023-10-30T13:19:14.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/15' 2023-10-30T13:19:14.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/31' 2023-10-30T13:19:14.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/36' 2023-10-30T13:19:14.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/20' 2023-10-30T13:19:14.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/12' 2023-10-30T13:19:14.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/39' 2023-10-30T13:19:14.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/10' 2023-10-30T13:19:14.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/34' 2023-10-30T13:19:14.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/38' 2023-10-30T13:19:14.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/11' 2023-10-30T13:19:14.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/16' 2023-10-30T13:19:14.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/32' 2023-10-30T13:19:14.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/30' 2023-10-30T13:19:14.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/14' 2023-10-30T13:19:14.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/18' 2023-10-30T13:19:14.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/13' 2023-10-30T13:19:14.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/33' 2023-10-30T13:19:14.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/17' 2023-10-30T13:19:14.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/35' 2023-10-30T13:19:14.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './147/37' 2023-10-30T13:19:14.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './147' 2023-10-30T13:19:14.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/19' 2023-10-30T13:19:14.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/15' 2023-10-30T13:19:14.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/31' 2023-10-30T13:19:14.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/36' 2023-10-30T13:19:17.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/20' 2023-10-30T13:19:17.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/12' 2023-10-30T13:19:17.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/39' 2023-10-30T13:19:17.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/10' 2023-10-30T13:19:17.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/34' 2023-10-30T13:19:17.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/38' 2023-10-30T13:19:17.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/11' 2023-10-30T13:19:17.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/16' 2023-10-30T13:19:17.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/32' 2023-10-30T13:19:17.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/30' 2023-10-30T13:19:17.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/14' 2023-10-30T13:19:17.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/18' 2023-10-30T13:19:17.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/13' 2023-10-30T13:19:17.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/33' 2023-10-30T13:19:17.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/17' 2023-10-30T13:19:17.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/35' 2023-10-30T13:19:17.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './148/37' 2023-10-30T13:19:17.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './148' 2023-10-30T13:19:17.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/19' 2023-10-30T13:19:17.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/15' 2023-10-30T13:19:17.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/31' 2023-10-30T13:19:17.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/36' 2023-10-30T13:19:17.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/20' 2023-10-30T13:19:17.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/12' 2023-10-30T13:19:17.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/39' 2023-10-30T13:19:17.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/10' 2023-10-30T13:19:17.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/34' 2023-10-30T13:19:17.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/38' 2023-10-30T13:19:17.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/11' 2023-10-30T13:19:17.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/16' 2023-10-30T13:19:17.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/32' 2023-10-30T13:19:17.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/30' 2023-10-30T13:19:17.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/14' 2023-10-30T13:19:17.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/18' 2023-10-30T13:19:17.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/13' 2023-10-30T13:19:17.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/33' 2023-10-30T13:19:17.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/17' 2023-10-30T13:19:17.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/35' 2023-10-30T13:19:17.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './149/37' 2023-10-30T13:19:17.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './149' 2023-10-30T13:19:17.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/19' 2023-10-30T13:19:17.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/15' 2023-10-30T13:19:17.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/31' 2023-10-30T13:19:17.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/36' 2023-10-30T13:19:17.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/20' 2023-10-30T13:19:17.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/12' 2023-10-30T13:19:17.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/39' 2023-10-30T13:19:17.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/10' 2023-10-30T13:19:17.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/34' 2023-10-30T13:19:17.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/38' 2023-10-30T13:19:17.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/11' 2023-10-30T13:19:17.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/16' 2023-10-30T13:19:17.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/32' 2023-10-30T13:19:17.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/30' 2023-10-30T13:19:17.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/14' 2023-10-30T13:19:17.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/18' 2023-10-30T13:19:17.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/13' 2023-10-30T13:19:17.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/33' 2023-10-30T13:19:17.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/17' 2023-10-30T13:19:17.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/35' 2023-10-30T13:19:17.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './150/37' 2023-10-30T13:19:17.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './150' 2023-10-30T13:19:17.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/19' 2023-10-30T13:19:17.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/15' 2023-10-30T13:19:17.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/31' 2023-10-30T13:19:17.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/36' 2023-10-30T13:19:17.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/20' 2023-10-30T13:19:17.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/12' 2023-10-30T13:19:17.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/39' 2023-10-30T13:19:17.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/10' 2023-10-30T13:19:17.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/34' 2023-10-30T13:19:17.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/38' 2023-10-30T13:19:17.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/11' 2023-10-30T13:19:17.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/16' 2023-10-30T13:19:17.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/32' 2023-10-30T13:19:17.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/30' 2023-10-30T13:19:17.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/14' 2023-10-30T13:19:17.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/18' 2023-10-30T13:19:17.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/13' 2023-10-30T13:19:17.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/33' 2023-10-30T13:19:17.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/17' 2023-10-30T13:19:17.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/35' 2023-10-30T13:19:17.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './151/37' 2023-10-30T13:19:17.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './151' 2023-10-30T13:19:17.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/19' 2023-10-30T13:19:17.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/15' 2023-10-30T13:19:17.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/31' 2023-10-30T13:19:17.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/36' 2023-10-30T13:19:17.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/20' 2023-10-30T13:19:17.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/12' 2023-10-30T13:19:17.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/39' 2023-10-30T13:19:17.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/10' 2023-10-30T13:19:17.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/34' 2023-10-30T13:19:17.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/38' 2023-10-30T13:19:17.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/11' 2023-10-30T13:19:17.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/16' 2023-10-30T13:19:17.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/32' 2023-10-30T13:19:17.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/30' 2023-10-30T13:19:17.750 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/14' 2023-10-30T13:19:17.750 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/18' 2023-10-30T13:19:17.750 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/13' 2023-10-30T13:19:17.750 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/33' 2023-10-30T13:19:17.750 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/17' 2023-10-30T13:19:17.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/35' 2023-10-30T13:19:17.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './152/37' 2023-10-30T13:19:17.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './152' 2023-10-30T13:19:17.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/19' 2023-10-30T13:19:17.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/15' 2023-10-30T13:19:17.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/31' 2023-10-30T13:19:17.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/36' 2023-10-30T13:19:17.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/20' 2023-10-30T13:19:17.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/12' 2023-10-30T13:19:17.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/39' 2023-10-30T13:19:17.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/10' 2023-10-30T13:19:17.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/34' 2023-10-30T13:19:17.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/38' 2023-10-30T13:19:17.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/11' 2023-10-30T13:19:17.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/16' 2023-10-30T13:19:17.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/32' 2023-10-30T13:19:17.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/30' 2023-10-30T13:19:17.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/14' 2023-10-30T13:19:17.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/18' 2023-10-30T13:19:17.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/13' 2023-10-30T13:19:17.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/33' 2023-10-30T13:19:17.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/17' 2023-10-30T13:19:17.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/35' 2023-10-30T13:19:17.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './153/37' 2023-10-30T13:19:17.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './153' 2023-10-30T13:19:17.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/19' 2023-10-30T13:19:17.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/15' 2023-10-30T13:19:17.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/31' 2023-10-30T13:19:17.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/36' 2023-10-30T13:19:17.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/20' 2023-10-30T13:19:17.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/12' 2023-10-30T13:19:17.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/39' 2023-10-30T13:19:17.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/10' 2023-10-30T13:19:17.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/34' 2023-10-30T13:19:17.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/38' 2023-10-30T13:19:17.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/11' 2023-10-30T13:19:17.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/16' 2023-10-30T13:19:17.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/32' 2023-10-30T13:19:17.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/30' 2023-10-30T13:19:17.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/14' 2023-10-30T13:19:17.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/18' 2023-10-30T13:19:17.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/13' 2023-10-30T13:19:17.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/33' 2023-10-30T13:19:17.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/17' 2023-10-30T13:19:17.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/35' 2023-10-30T13:19:17.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './154/37' 2023-10-30T13:19:17.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './154' 2023-10-30T13:19:17.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/19' 2023-10-30T13:19:17.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/15' 2023-10-30T13:19:17.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/31' 2023-10-30T13:19:17.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/36' 2023-10-30T13:19:17.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/20' 2023-10-30T13:19:17.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/12' 2023-10-30T13:19:17.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/39' 2023-10-30T13:19:17.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/10' 2023-10-30T13:19:17.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/34' 2023-10-30T13:19:17.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/38' 2023-10-30T13:19:17.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/11' 2023-10-30T13:19:17.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/16' 2023-10-30T13:19:17.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/32' 2023-10-30T13:19:17.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/30' 2023-10-30T13:19:17.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/14' 2023-10-30T13:19:17.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/18' 2023-10-30T13:19:17.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/13' 2023-10-30T13:19:17.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/33' 2023-10-30T13:19:17.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/17' 2023-10-30T13:19:17.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/35' 2023-10-30T13:19:17.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './155/37' 2023-10-30T13:19:17.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './155' 2023-10-30T13:19:17.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/19' 2023-10-30T13:19:17.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/15' 2023-10-30T13:19:17.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/31' 2023-10-30T13:19:17.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/36' 2023-10-30T13:19:17.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/20' 2023-10-30T13:19:17.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/12' 2023-10-30T13:19:17.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/39' 2023-10-30T13:19:17.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/10' 2023-10-30T13:19:17.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/34' 2023-10-30T13:19:17.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/38' 2023-10-30T13:19:17.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/11' 2023-10-30T13:19:17.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/16' 2023-10-30T13:19:17.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/32' 2023-10-30T13:19:17.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/30' 2023-10-30T13:19:17.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/14' 2023-10-30T13:19:17.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/18' 2023-10-30T13:19:17.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/13' 2023-10-30T13:19:17.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/33' 2023-10-30T13:19:17.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/17' 2023-10-30T13:19:17.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/35' 2023-10-30T13:19:17.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './156/37' 2023-10-30T13:19:17.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './156' 2023-10-30T13:19:17.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/19' 2023-10-30T13:19:17.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/15' 2023-10-30T13:19:17.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/31' 2023-10-30T13:19:17.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/36' 2023-10-30T13:19:17.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/20' 2023-10-30T13:19:17.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/12' 2023-10-30T13:19:17.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/39' 2023-10-30T13:19:17.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/10' 2023-10-30T13:19:17.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/34' 2023-10-30T13:19:17.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/38' 2023-10-30T13:19:17.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/11' 2023-10-30T13:19:17.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/16' 2023-10-30T13:19:17.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/32' 2023-10-30T13:19:17.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/30' 2023-10-30T13:19:17.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/14' 2023-10-30T13:19:17.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/18' 2023-10-30T13:19:17.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/13' 2023-10-30T13:19:17.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/33' 2023-10-30T13:19:19.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/17' 2023-10-30T13:19:19.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/35' 2023-10-30T13:19:19.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './157/37' 2023-10-30T13:19:19.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './157' 2023-10-30T13:19:19.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/19' 2023-10-30T13:19:19.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/15' 2023-10-30T13:19:19.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/31' 2023-10-30T13:19:19.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/36' 2023-10-30T13:19:19.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/20' 2023-10-30T13:19:19.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/12' 2023-10-30T13:19:19.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/39' 2023-10-30T13:19:19.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/10' 2023-10-30T13:19:19.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/34' 2023-10-30T13:19:19.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/38' 2023-10-30T13:19:19.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/11' 2023-10-30T13:19:19.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/16' 2023-10-30T13:19:19.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/32' 2023-10-30T13:19:19.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/30' 2023-10-30T13:19:19.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/14' 2023-10-30T13:19:19.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/18' 2023-10-30T13:19:19.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/13' 2023-10-30T13:19:19.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/33' 2023-10-30T13:19:19.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/17' 2023-10-30T13:19:19.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/35' 2023-10-30T13:19:19.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './158/37' 2023-10-30T13:19:19.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './158' 2023-10-30T13:19:19.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/19' 2023-10-30T13:19:19.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/15' 2023-10-30T13:19:19.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/31' 2023-10-30T13:19:19.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/36' 2023-10-30T13:19:19.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/20' 2023-10-30T13:19:19.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/12' 2023-10-30T13:19:19.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/39' 2023-10-30T13:19:19.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/10' 2023-10-30T13:19:19.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/34' 2023-10-30T13:19:19.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/38' 2023-10-30T13:19:19.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/11' 2023-10-30T13:19:19.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/16' 2023-10-30T13:19:19.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/32' 2023-10-30T13:19:19.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/30' 2023-10-30T13:19:19.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/14' 2023-10-30T13:19:19.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/18' 2023-10-30T13:19:19.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/13' 2023-10-30T13:19:19.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/33' 2023-10-30T13:19:19.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/17' 2023-10-30T13:19:19.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/35' 2023-10-30T13:19:19.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './159/37' 2023-10-30T13:19:19.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './159' 2023-10-30T13:19:19.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/19' 2023-10-30T13:19:19.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/15' 2023-10-30T13:19:19.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/31' 2023-10-30T13:19:19.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/36' 2023-10-30T13:19:19.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/20' 2023-10-30T13:19:19.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/12' 2023-10-30T13:19:19.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/39' 2023-10-30T13:19:19.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/10' 2023-10-30T13:19:19.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/34' 2023-10-30T13:19:19.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/38' 2023-10-30T13:19:19.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/11' 2023-10-30T13:19:19.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/16' 2023-10-30T13:19:19.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/32' 2023-10-30T13:19:19.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/30' 2023-10-30T13:19:19.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/14' 2023-10-30T13:19:19.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/18' 2023-10-30T13:19:19.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/13' 2023-10-30T13:19:19.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/33' 2023-10-30T13:19:19.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/17' 2023-10-30T13:19:19.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/35' 2023-10-30T13:19:19.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed './160/37' 2023-10-30T13:19:19.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './160' 2023-10-30T13:19:19.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/19' 2023-10-30T13:19:19.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/15' 2023-10-30T13:19:19.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/31' 2023-10-30T13:19:19.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/36' 2023-10-30T13:19:19.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/20' 2023-10-30T13:19:19.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/12' 2023-10-30T13:19:19.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/39' 2023-10-30T13:19:19.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/10' 2023-10-30T13:19:19.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/34' 2023-10-30T13:19:19.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/38' 2023-10-30T13:19:19.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/11' 2023-10-30T13:19:19.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/16' 2023-10-30T13:19:19.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/32' 2023-10-30T13:19:19.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/30' 2023-10-30T13:19:19.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/14' 2023-10-30T13:19:19.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/18' 2023-10-30T13:19:19.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/13' 2023-10-30T13:19:19.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/33' 2023-10-30T13:19:19.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/17' 2023-10-30T13:19:19.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/35' 2023-10-30T13:19:19.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed './161/37' 2023-10-30T13:19:19.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './161' 2023-10-30T13:19:19.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/19' 2023-10-30T13:19:19.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/15' 2023-10-30T13:19:19.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/31' 2023-10-30T13:19:19.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/36' 2023-10-30T13:19:19.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/20' 2023-10-30T13:19:19.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/12' 2023-10-30T13:19:19.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/39' 2023-10-30T13:19:19.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/10' 2023-10-30T13:19:19.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/34' 2023-10-30T13:19:19.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/38' 2023-10-30T13:19:19.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/11' 2023-10-30T13:19:19.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/16' 2023-10-30T13:19:19.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/32' 2023-10-30T13:19:19.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/30' 2023-10-30T13:19:19.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/14' 2023-10-30T13:19:19.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/18' 2023-10-30T13:19:19.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/13' 2023-10-30T13:19:19.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/33' 2023-10-30T13:19:19.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/17' 2023-10-30T13:19:19.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/35' 2023-10-30T13:19:19.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed './162/37' 2023-10-30T13:19:19.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './162' 2023-10-30T13:19:19.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/19' 2023-10-30T13:19:19.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/15' 2023-10-30T13:19:19.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/31' 2023-10-30T13:19:19.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/36' 2023-10-30T13:19:19.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/20' 2023-10-30T13:19:19.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/12' 2023-10-30T13:19:19.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/39' 2023-10-30T13:19:19.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/10' 2023-10-30T13:19:19.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/34' 2023-10-30T13:19:19.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/38' 2023-10-30T13:19:19.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/11' 2023-10-30T13:19:19.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/16' 2023-10-30T13:19:19.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/32' 2023-10-30T13:19:19.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/30' 2023-10-30T13:19:19.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/14' 2023-10-30T13:19:19.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/18' 2023-10-30T13:19:19.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/13' 2023-10-30T13:19:19.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/33' 2023-10-30T13:19:19.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/17' 2023-10-30T13:19:19.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/35' 2023-10-30T13:19:19.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed './163/37' 2023-10-30T13:19:19.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './163' 2023-10-30T13:19:19.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/19' 2023-10-30T13:19:19.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/15' 2023-10-30T13:19:19.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/31' 2023-10-30T13:19:19.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/36' 2023-10-30T13:19:19.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/20' 2023-10-30T13:19:19.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/12' 2023-10-30T13:19:19.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/39' 2023-10-30T13:19:19.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/10' 2023-10-30T13:19:19.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/34' 2023-10-30T13:19:19.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/38' 2023-10-30T13:19:19.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/11' 2023-10-30T13:19:19.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/16' 2023-10-30T13:19:19.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/32' 2023-10-30T13:19:19.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/30' 2023-10-30T13:19:19.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/14' 2023-10-30T13:19:19.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/18' 2023-10-30T13:19:19.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/13' 2023-10-30T13:19:19.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/33' 2023-10-30T13:19:19.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/17' 2023-10-30T13:19:19.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/35' 2023-10-30T13:19:19.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './164/37' 2023-10-30T13:19:19.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './164' 2023-10-30T13:19:19.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/19' 2023-10-30T13:19:19.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/15' 2023-10-30T13:19:19.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/31' 2023-10-30T13:19:19.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/36' 2023-10-30T13:19:19.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/20' 2023-10-30T13:19:19.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/12' 2023-10-30T13:19:19.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/39' 2023-10-30T13:19:19.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/10' 2023-10-30T13:19:19.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/34' 2023-10-30T13:19:19.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/38' 2023-10-30T13:19:19.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/11' 2023-10-30T13:19:19.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/16' 2023-10-30T13:19:19.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/32' 2023-10-30T13:19:19.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/30' 2023-10-30T13:19:19.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/14' 2023-10-30T13:19:19.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/18' 2023-10-30T13:19:19.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/13' 2023-10-30T13:19:19.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/33' 2023-10-30T13:19:19.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/17' 2023-10-30T13:19:19.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/35' 2023-10-30T13:19:19.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './165/37' 2023-10-30T13:19:19.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './165' 2023-10-30T13:19:19.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/19' 2023-10-30T13:19:19.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/15' 2023-10-30T13:19:19.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/31' 2023-10-30T13:19:19.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/36' 2023-10-30T13:19:19.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/20' 2023-10-30T13:19:19.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/12' 2023-10-30T13:19:19.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/39' 2023-10-30T13:19:19.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/10' 2023-10-30T13:19:19.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/34' 2023-10-30T13:19:19.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/38' 2023-10-30T13:19:19.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/11' 2023-10-30T13:19:19.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/16' 2023-10-30T13:19:19.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/32' 2023-10-30T13:19:19.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/30' 2023-10-30T13:19:19.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/14' 2023-10-30T13:19:19.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/18' 2023-10-30T13:19:19.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/13' 2023-10-30T13:19:19.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/33' 2023-10-30T13:19:19.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/17' 2023-10-30T13:19:19.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/35' 2023-10-30T13:19:19.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed './166/37' 2023-10-30T13:19:19.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './166' 2023-10-30T13:19:19.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/19' 2023-10-30T13:19:19.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/15' 2023-10-30T13:19:19.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/31' 2023-10-30T13:19:19.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/36' 2023-10-30T13:19:19.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/20' 2023-10-30T13:19:19.509 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/12' 2023-10-30T13:19:19.509 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/39' 2023-10-30T13:19:19.509 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/10' 2023-10-30T13:19:19.509 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/34' 2023-10-30T13:19:19.509 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/38' 2023-10-30T13:19:23.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/11' 2023-10-30T13:19:23.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/16' 2023-10-30T13:19:23.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/32' 2023-10-30T13:19:23.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/30' 2023-10-30T13:19:23.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/14' 2023-10-30T13:19:23.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/18' 2023-10-30T13:19:23.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/13' 2023-10-30T13:19:23.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/33' 2023-10-30T13:19:23.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/17' 2023-10-30T13:19:23.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/35' 2023-10-30T13:19:23.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './167/37' 2023-10-30T13:19:23.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './167' 2023-10-30T13:19:23.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/19' 2023-10-30T13:19:23.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/15' 2023-10-30T13:19:23.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/31' 2023-10-30T13:19:23.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/36' 2023-10-30T13:19:23.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/20' 2023-10-30T13:19:23.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/12' 2023-10-30T13:19:23.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/39' 2023-10-30T13:19:23.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/10' 2023-10-30T13:19:23.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/34' 2023-10-30T13:19:23.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/38' 2023-10-30T13:19:23.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/11' 2023-10-30T13:19:23.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/16' 2023-10-30T13:19:23.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/32' 2023-10-30T13:19:23.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/30' 2023-10-30T13:19:23.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/14' 2023-10-30T13:19:23.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/18' 2023-10-30T13:19:23.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/13' 2023-10-30T13:19:23.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/33' 2023-10-30T13:19:23.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/17' 2023-10-30T13:19:23.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/35' 2023-10-30T13:19:23.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './168/37' 2023-10-30T13:19:23.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './168' 2023-10-30T13:19:23.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/19' 2023-10-30T13:19:23.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/15' 2023-10-30T13:19:23.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/31' 2023-10-30T13:19:23.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/36' 2023-10-30T13:19:23.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/20' 2023-10-30T13:19:23.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/12' 2023-10-30T13:19:23.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/39' 2023-10-30T13:19:23.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/10' 2023-10-30T13:19:23.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/34' 2023-10-30T13:19:23.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/38' 2023-10-30T13:19:23.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/11' 2023-10-30T13:19:23.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/16' 2023-10-30T13:19:23.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/32' 2023-10-30T13:19:23.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/30' 2023-10-30T13:19:23.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/14' 2023-10-30T13:19:23.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/18' 2023-10-30T13:19:23.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/13' 2023-10-30T13:19:23.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/33' 2023-10-30T13:19:23.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/17' 2023-10-30T13:19:23.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/35' 2023-10-30T13:19:23.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './169/37' 2023-10-30T13:19:23.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './169' 2023-10-30T13:19:23.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/19' 2023-10-30T13:19:23.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/15' 2023-10-30T13:19:23.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/31' 2023-10-30T13:19:23.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/36' 2023-10-30T13:19:23.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/20' 2023-10-30T13:19:23.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/12' 2023-10-30T13:19:23.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/39' 2023-10-30T13:19:23.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/10' 2023-10-30T13:19:23.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/34' 2023-10-30T13:19:23.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/38' 2023-10-30T13:19:23.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/11' 2023-10-30T13:19:23.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/16' 2023-10-30T13:19:23.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/32' 2023-10-30T13:19:23.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/30' 2023-10-30T13:19:23.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/14' 2023-10-30T13:19:23.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/18' 2023-10-30T13:19:23.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/13' 2023-10-30T13:19:23.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/33' 2023-10-30T13:19:23.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/17' 2023-10-30T13:19:23.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/35' 2023-10-30T13:19:23.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './170/37' 2023-10-30T13:19:23.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './170' 2023-10-30T13:19:23.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/19' 2023-10-30T13:19:23.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/15' 2023-10-30T13:19:23.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/31' 2023-10-30T13:19:23.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/36' 2023-10-30T13:19:23.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/20' 2023-10-30T13:19:23.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/12' 2023-10-30T13:19:23.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/39' 2023-10-30T13:19:23.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/10' 2023-10-30T13:19:23.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/34' 2023-10-30T13:19:23.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/38' 2023-10-30T13:19:23.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/11' 2023-10-30T13:19:23.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/16' 2023-10-30T13:19:23.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/32' 2023-10-30T13:19:23.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/30' 2023-10-30T13:19:23.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/14' 2023-10-30T13:19:23.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/18' 2023-10-30T13:19:23.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/13' 2023-10-30T13:19:23.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/33' 2023-10-30T13:19:23.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/17' 2023-10-30T13:19:23.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/35' 2023-10-30T13:19:23.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './171/37' 2023-10-30T13:19:23.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './171' 2023-10-30T13:19:23.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/19' 2023-10-30T13:19:23.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/15' 2023-10-30T13:19:23.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/31' 2023-10-30T13:19:23.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/36' 2023-10-30T13:19:23.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/20' 2023-10-30T13:19:23.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/12' 2023-10-30T13:19:23.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/39' 2023-10-30T13:19:23.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/10' 2023-10-30T13:19:23.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/34' 2023-10-30T13:19:23.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/38' 2023-10-30T13:19:23.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/11' 2023-10-30T13:19:23.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/16' 2023-10-30T13:19:23.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/32' 2023-10-30T13:19:23.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/30' 2023-10-30T13:19:23.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/14' 2023-10-30T13:19:23.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/18' 2023-10-30T13:19:23.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/13' 2023-10-30T13:19:23.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/33' 2023-10-30T13:19:23.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/17' 2023-10-30T13:19:23.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/35' 2023-10-30T13:19:23.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './172/37' 2023-10-30T13:19:23.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './172' 2023-10-30T13:19:23.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/19' 2023-10-30T13:19:23.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/15' 2023-10-30T13:19:23.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/31' 2023-10-30T13:19:23.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/36' 2023-10-30T13:19:23.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/20' 2023-10-30T13:19:23.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/12' 2023-10-30T13:19:23.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/39' 2023-10-30T13:19:23.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/10' 2023-10-30T13:19:23.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/34' 2023-10-30T13:19:23.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/38' 2023-10-30T13:19:23.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/11' 2023-10-30T13:19:23.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/16' 2023-10-30T13:19:23.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/32' 2023-10-30T13:19:23.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/30' 2023-10-30T13:19:23.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/14' 2023-10-30T13:19:23.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/18' 2023-10-30T13:19:23.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/13' 2023-10-30T13:19:23.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/33' 2023-10-30T13:19:23.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/17' 2023-10-30T13:19:23.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/35' 2023-10-30T13:19:23.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './173/37' 2023-10-30T13:19:23.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './173' 2023-10-30T13:19:23.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/19' 2023-10-30T13:19:23.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/15' 2023-10-30T13:19:23.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/31' 2023-10-30T13:19:23.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/36' 2023-10-30T13:19:23.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/20' 2023-10-30T13:19:23.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/12' 2023-10-30T13:19:23.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/39' 2023-10-30T13:19:23.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/10' 2023-10-30T13:19:23.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/34' 2023-10-30T13:19:23.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/38' 2023-10-30T13:19:23.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/11' 2023-10-30T13:19:23.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/16' 2023-10-30T13:19:23.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/32' 2023-10-30T13:19:23.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/30' 2023-10-30T13:19:23.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/14' 2023-10-30T13:19:23.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/18' 2023-10-30T13:19:23.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/13' 2023-10-30T13:19:23.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/33' 2023-10-30T13:19:23.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/17' 2023-10-30T13:19:23.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/35' 2023-10-30T13:19:23.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './174/37' 2023-10-30T13:19:23.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './174' 2023-10-30T13:19:23.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/19' 2023-10-30T13:19:23.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/15' 2023-10-30T13:19:23.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/31' 2023-10-30T13:19:23.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/36' 2023-10-30T13:19:23.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/20' 2023-10-30T13:19:23.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/12' 2023-10-30T13:19:23.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/39' 2023-10-30T13:19:23.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/10' 2023-10-30T13:19:23.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/34' 2023-10-30T13:19:23.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/38' 2023-10-30T13:19:23.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/11' 2023-10-30T13:19:23.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/16' 2023-10-30T13:19:23.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/32' 2023-10-30T13:19:23.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/30' 2023-10-30T13:19:23.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/14' 2023-10-30T13:19:23.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/18' 2023-10-30T13:19:23.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/13' 2023-10-30T13:19:23.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/33' 2023-10-30T13:19:23.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/17' 2023-10-30T13:19:23.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/35' 2023-10-30T13:19:23.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './175/37' 2023-10-30T13:19:23.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './175' 2023-10-30T13:19:23.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/19' 2023-10-30T13:19:23.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/15' 2023-10-30T13:19:23.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/31' 2023-10-30T13:19:23.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/36' 2023-10-30T13:19:23.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/20' 2023-10-30T13:19:23.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/12' 2023-10-30T13:19:23.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/39' 2023-10-30T13:19:23.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/10' 2023-10-30T13:19:23.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/34' 2023-10-30T13:19:23.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/38' 2023-10-30T13:19:23.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/11' 2023-10-30T13:19:23.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/16' 2023-10-30T13:19:23.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/32' 2023-10-30T13:19:23.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/30' 2023-10-30T13:19:23.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/14' 2023-10-30T13:19:23.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/18' 2023-10-30T13:19:23.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/13' 2023-10-30T13:19:23.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/33' 2023-10-30T13:19:23.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/17' 2023-10-30T13:19:23.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/35' 2023-10-30T13:19:23.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './176/37' 2023-10-30T13:19:23.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './176' 2023-10-30T13:19:23.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/19' 2023-10-30T13:19:23.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/15' 2023-10-30T13:19:24.564 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:19:24.568 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:19:25.890 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/31' 2023-10-30T13:19:25.890 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/36' 2023-10-30T13:19:25.890 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/20' 2023-10-30T13:19:25.891 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/12' 2023-10-30T13:19:25.891 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/39' 2023-10-30T13:19:25.891 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/10' 2023-10-30T13:19:25.891 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/34' 2023-10-30T13:19:25.891 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/38' 2023-10-30T13:19:25.892 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/11' 2023-10-30T13:19:25.892 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/16' 2023-10-30T13:19:25.892 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/32' 2023-10-30T13:19:25.892 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/30' 2023-10-30T13:19:25.892 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/14' 2023-10-30T13:19:25.892 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/18' 2023-10-30T13:19:25.893 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/13' 2023-10-30T13:19:25.893 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/33' 2023-10-30T13:19:25.893 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/17' 2023-10-30T13:19:25.893 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/35' 2023-10-30T13:19:25.893 INFO:tasks.workunit.client.0.smithi148.stdout:removed './177/37' 2023-10-30T13:19:25.893 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './177' 2023-10-30T13:19:25.894 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/19' 2023-10-30T13:19:25.894 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/15' 2023-10-30T13:19:25.894 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/31' 2023-10-30T13:19:25.894 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/36' 2023-10-30T13:19:25.894 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/20' 2023-10-30T13:19:25.895 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/12' 2023-10-30T13:19:25.895 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/39' 2023-10-30T13:19:25.895 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/10' 2023-10-30T13:19:25.895 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/34' 2023-10-30T13:19:25.895 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/38' 2023-10-30T13:19:25.896 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/11' 2023-10-30T13:19:25.896 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/16' 2023-10-30T13:19:25.896 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/32' 2023-10-30T13:19:25.896 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/30' 2023-10-30T13:19:25.896 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/14' 2023-10-30T13:19:25.896 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/18' 2023-10-30T13:19:25.897 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/13' 2023-10-30T13:19:25.897 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/33' 2023-10-30T13:19:25.897 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/17' 2023-10-30T13:19:25.897 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/35' 2023-10-30T13:19:25.897 INFO:tasks.workunit.client.0.smithi148.stdout:removed './178/37' 2023-10-30T13:19:25.898 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './178' 2023-10-30T13:19:25.898 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/19' 2023-10-30T13:19:25.898 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/15' 2023-10-30T13:19:25.898 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/31' 2023-10-30T13:19:25.898 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/36' 2023-10-30T13:19:25.899 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/20' 2023-10-30T13:19:25.899 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/12' 2023-10-30T13:19:25.899 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/39' 2023-10-30T13:19:25.899 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/10' 2023-10-30T13:19:25.899 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/34' 2023-10-30T13:19:25.899 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/38' 2023-10-30T13:19:25.900 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/11' 2023-10-30T13:19:25.900 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/16' 2023-10-30T13:19:25.900 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/32' 2023-10-30T13:19:25.900 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/30' 2023-10-30T13:19:25.900 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/14' 2023-10-30T13:19:25.901 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/18' 2023-10-30T13:19:25.901 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/13' 2023-10-30T13:19:25.901 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/33' 2023-10-30T13:19:25.901 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/17' 2023-10-30T13:19:25.901 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/35' 2023-10-30T13:19:25.901 INFO:tasks.workunit.client.0.smithi148.stdout:removed './179/37' 2023-10-30T13:19:25.902 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './179' 2023-10-30T13:19:25.902 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/19' 2023-10-30T13:19:25.902 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/15' 2023-10-30T13:19:25.902 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/31' 2023-10-30T13:19:25.902 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/36' 2023-10-30T13:19:25.902 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/20' 2023-10-30T13:19:25.903 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/12' 2023-10-30T13:19:25.903 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/39' 2023-10-30T13:19:25.903 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/10' 2023-10-30T13:19:25.903 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/34' 2023-10-30T13:19:25.903 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/38' 2023-10-30T13:19:25.904 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/11' 2023-10-30T13:19:25.904 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/16' 2023-10-30T13:19:25.904 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/32' 2023-10-30T13:19:25.904 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/30' 2023-10-30T13:19:25.904 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/14' 2023-10-30T13:19:25.904 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/18' 2023-10-30T13:19:25.905 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/13' 2023-10-30T13:19:25.905 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/33' 2023-10-30T13:19:25.905 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/17' 2023-10-30T13:19:25.905 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/35' 2023-10-30T13:19:25.905 INFO:tasks.workunit.client.0.smithi148.stdout:removed './180/37' 2023-10-30T13:19:25.905 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './180' 2023-10-30T13:19:25.906 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/19' 2023-10-30T13:19:25.906 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/15' 2023-10-30T13:19:25.906 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/31' 2023-10-30T13:19:25.906 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/36' 2023-10-30T13:19:25.906 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/20' 2023-10-30T13:19:25.907 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/12' 2023-10-30T13:19:25.907 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/39' 2023-10-30T13:19:25.907 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/10' 2023-10-30T13:19:25.907 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/34' 2023-10-30T13:19:25.907 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/38' 2023-10-30T13:19:25.907 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/11' 2023-10-30T13:19:25.907 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/16' 2023-10-30T13:19:25.908 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/32' 2023-10-30T13:19:25.908 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/30' 2023-10-30T13:19:25.908 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/14' 2023-10-30T13:19:25.908 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/18' 2023-10-30T13:19:25.908 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/13' 2023-10-30T13:19:25.908 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/33' 2023-10-30T13:19:25.909 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/17' 2023-10-30T13:19:25.909 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/35' 2023-10-30T13:19:25.909 INFO:tasks.workunit.client.0.smithi148.stdout:removed './181/37' 2023-10-30T13:19:25.909 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './181' 2023-10-30T13:19:25.909 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/19' 2023-10-30T13:19:25.909 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/15' 2023-10-30T13:19:25.910 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/31' 2023-10-30T13:19:25.910 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/36' 2023-10-30T13:19:25.910 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/20' 2023-10-30T13:19:25.910 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/12' 2023-10-30T13:19:25.910 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/39' 2023-10-30T13:19:25.910 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/10' 2023-10-30T13:19:25.911 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/34' 2023-10-30T13:19:25.911 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/38' 2023-10-30T13:19:25.911 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/11' 2023-10-30T13:19:25.911 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/16' 2023-10-30T13:19:25.911 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/32' 2023-10-30T13:19:25.911 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/30' 2023-10-30T13:19:25.912 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/14' 2023-10-30T13:19:25.912 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/18' 2023-10-30T13:19:25.912 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/13' 2023-10-30T13:19:25.912 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/33' 2023-10-30T13:19:25.912 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/17' 2023-10-30T13:19:25.912 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/35' 2023-10-30T13:19:25.913 INFO:tasks.workunit.client.0.smithi148.stdout:removed './182/37' 2023-10-30T13:19:25.913 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './182' 2023-10-30T13:19:25.913 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/19' 2023-10-30T13:19:25.913 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/15' 2023-10-30T13:19:25.913 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/31' 2023-10-30T13:19:25.913 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/36' 2023-10-30T13:19:25.914 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/20' 2023-10-30T13:19:25.914 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/12' 2023-10-30T13:19:25.914 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/39' 2023-10-30T13:19:25.914 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/10' 2023-10-30T13:19:25.914 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/34' 2023-10-30T13:19:25.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/38' 2023-10-30T13:19:25.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/11' 2023-10-30T13:19:25.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/16' 2023-10-30T13:19:25.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/32' 2023-10-30T13:19:25.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/30' 2023-10-30T13:19:25.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/14' 2023-10-30T13:19:25.916 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/18' 2023-10-30T13:19:25.916 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/13' 2023-10-30T13:19:25.916 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/33' 2023-10-30T13:19:25.916 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/17' 2023-10-30T13:19:25.916 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/35' 2023-10-30T13:19:25.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed './183/37' 2023-10-30T13:19:25.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './183' 2023-10-30T13:19:25.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/19' 2023-10-30T13:19:25.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/15' 2023-10-30T13:19:25.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/31' 2023-10-30T13:19:25.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/36' 2023-10-30T13:19:25.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/20' 2023-10-30T13:19:25.918 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/12' 2023-10-30T13:19:25.918 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/39' 2023-10-30T13:19:25.918 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/10' 2023-10-30T13:19:25.918 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/34' 2023-10-30T13:19:25.918 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/38' 2023-10-30T13:19:25.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/11' 2023-10-30T13:19:25.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/16' 2023-10-30T13:19:25.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/32' 2023-10-30T13:19:25.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/30' 2023-10-30T13:19:25.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/14' 2023-10-30T13:19:25.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/18' 2023-10-30T13:19:25.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/13' 2023-10-30T13:19:25.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/33' 2023-10-30T13:19:25.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/17' 2023-10-30T13:19:25.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/35' 2023-10-30T13:19:25.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed './184/37' 2023-10-30T13:19:25.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './184' 2023-10-30T13:19:25.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/19' 2023-10-30T13:19:25.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/15' 2023-10-30T13:19:25.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/31' 2023-10-30T13:19:25.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/36' 2023-10-30T13:19:25.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/20' 2023-10-30T13:19:25.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/12' 2023-10-30T13:19:25.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/39' 2023-10-30T13:19:25.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/10' 2023-10-30T13:19:25.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/34' 2023-10-30T13:19:25.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/38' 2023-10-30T13:19:25.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/11' 2023-10-30T13:19:25.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/16' 2023-10-30T13:19:25.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/32' 2023-10-30T13:19:25.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/30' 2023-10-30T13:19:25.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/14' 2023-10-30T13:19:25.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/18' 2023-10-30T13:19:25.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/13' 2023-10-30T13:19:25.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/33' 2023-10-30T13:19:25.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/17' 2023-10-30T13:19:25.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/35' 2023-10-30T13:19:25.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed './185/37' 2023-10-30T13:19:25.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './185' 2023-10-30T13:19:25.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/19' 2023-10-30T13:19:25.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/15' 2023-10-30T13:19:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/31' 2023-10-30T13:19:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/36' 2023-10-30T13:19:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/20' 2023-10-30T13:19:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/12' 2023-10-30T13:19:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/39' 2023-10-30T13:19:25.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/10' 2023-10-30T13:19:25.926 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/34' 2023-10-30T13:19:25.926 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/38' 2023-10-30T13:19:25.926 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/11' 2023-10-30T13:19:25.926 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/16' 2023-10-30T13:19:25.926 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/32' 2023-10-30T13:19:25.927 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/30' 2023-10-30T13:19:25.927 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/14' 2023-10-30T13:19:25.927 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/18' 2023-10-30T13:19:28.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/13' 2023-10-30T13:19:28.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/33' 2023-10-30T13:19:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/17' 2023-10-30T13:19:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/35' 2023-10-30T13:19:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './186/37' 2023-10-30T13:19:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './186' 2023-10-30T13:19:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/19' 2023-10-30T13:19:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/15' 2023-10-30T13:19:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/31' 2023-10-30T13:19:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/36' 2023-10-30T13:19:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/20' 2023-10-30T13:19:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/12' 2023-10-30T13:19:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/39' 2023-10-30T13:19:28.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/10' 2023-10-30T13:19:28.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/34' 2023-10-30T13:19:28.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/38' 2023-10-30T13:19:28.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/11' 2023-10-30T13:19:28.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/16' 2023-10-30T13:19:28.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/32' 2023-10-30T13:19:28.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/30' 2023-10-30T13:19:28.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/14' 2023-10-30T13:19:28.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/18' 2023-10-30T13:19:28.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/13' 2023-10-30T13:19:28.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/33' 2023-10-30T13:19:28.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/17' 2023-10-30T13:19:28.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/35' 2023-10-30T13:19:28.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './187/37' 2023-10-30T13:19:28.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './187' 2023-10-30T13:19:28.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/19' 2023-10-30T13:19:28.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/15' 2023-10-30T13:19:28.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/31' 2023-10-30T13:19:28.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/36' 2023-10-30T13:19:28.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/20' 2023-10-30T13:19:28.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/12' 2023-10-30T13:19:28.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/39' 2023-10-30T13:19:28.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/10' 2023-10-30T13:19:28.453 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/34' 2023-10-30T13:19:28.453 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/38' 2023-10-30T13:19:28.453 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/11' 2023-10-30T13:19:28.453 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/16' 2023-10-30T13:19:28.453 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/32' 2023-10-30T13:19:28.453 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/30' 2023-10-30T13:19:28.454 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/14' 2023-10-30T13:19:28.454 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/18' 2023-10-30T13:19:28.454 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/13' 2023-10-30T13:19:28.454 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/33' 2023-10-30T13:19:28.454 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/17' 2023-10-30T13:19:28.454 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/35' 2023-10-30T13:19:28.455 INFO:tasks.workunit.client.0.smithi148.stdout:removed './188/37' 2023-10-30T13:19:28.455 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './188' 2023-10-30T13:19:28.455 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/19' 2023-10-30T13:19:28.455 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/15' 2023-10-30T13:19:28.455 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/31' 2023-10-30T13:19:28.455 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/36' 2023-10-30T13:19:28.456 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/20' 2023-10-30T13:19:28.456 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/12' 2023-10-30T13:19:28.456 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/39' 2023-10-30T13:19:28.456 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/10' 2023-10-30T13:19:28.456 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/34' 2023-10-30T13:19:28.457 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/38' 2023-10-30T13:19:28.457 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/11' 2023-10-30T13:19:28.457 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/16' 2023-10-30T13:19:28.457 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/32' 2023-10-30T13:19:28.457 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/30' 2023-10-30T13:19:28.457 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/14' 2023-10-30T13:19:28.458 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/18' 2023-10-30T13:19:28.458 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/13' 2023-10-30T13:19:28.458 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/33' 2023-10-30T13:19:28.458 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/17' 2023-10-30T13:19:28.458 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/35' 2023-10-30T13:19:28.459 INFO:tasks.workunit.client.0.smithi148.stdout:removed './189/37' 2023-10-30T13:19:28.459 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './189' 2023-10-30T13:19:28.459 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/19' 2023-10-30T13:19:28.459 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/15' 2023-10-30T13:19:28.459 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/31' 2023-10-30T13:19:28.459 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/36' 2023-10-30T13:19:28.460 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/20' 2023-10-30T13:19:28.460 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/12' 2023-10-30T13:19:28.460 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/39' 2023-10-30T13:19:28.460 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/10' 2023-10-30T13:19:28.460 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/34' 2023-10-30T13:19:28.460 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/38' 2023-10-30T13:19:28.461 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/11' 2023-10-30T13:19:28.461 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/16' 2023-10-30T13:19:28.461 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/32' 2023-10-30T13:19:28.461 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/30' 2023-10-30T13:19:28.461 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/14' 2023-10-30T13:19:28.461 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/18' 2023-10-30T13:19:28.462 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/13' 2023-10-30T13:19:28.462 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/33' 2023-10-30T13:19:28.462 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/17' 2023-10-30T13:19:28.462 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/35' 2023-10-30T13:19:28.462 INFO:tasks.workunit.client.0.smithi148.stdout:removed './190/37' 2023-10-30T13:19:28.462 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './190' 2023-10-30T13:19:28.463 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/19' 2023-10-30T13:19:28.463 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/15' 2023-10-30T13:19:28.463 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/31' 2023-10-30T13:19:28.463 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/36' 2023-10-30T13:19:28.463 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/20' 2023-10-30T13:19:28.463 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/12' 2023-10-30T13:19:28.464 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/39' 2023-10-30T13:19:28.464 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/10' 2023-10-30T13:19:28.464 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/34' 2023-10-30T13:19:28.464 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/38' 2023-10-30T13:19:28.464 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/11' 2023-10-30T13:19:28.464 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/16' 2023-10-30T13:19:28.465 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/32' 2023-10-30T13:19:28.465 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/30' 2023-10-30T13:19:28.465 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/14' 2023-10-30T13:19:28.465 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/18' 2023-10-30T13:19:28.465 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/13' 2023-10-30T13:19:28.465 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/33' 2023-10-30T13:19:28.466 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/17' 2023-10-30T13:19:28.466 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/35' 2023-10-30T13:19:28.466 INFO:tasks.workunit.client.0.smithi148.stdout:removed './191/37' 2023-10-30T13:19:28.466 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './191' 2023-10-30T13:19:28.466 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/19' 2023-10-30T13:19:28.467 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/15' 2023-10-30T13:19:28.467 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/31' 2023-10-30T13:19:28.467 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/36' 2023-10-30T13:19:28.467 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/20' 2023-10-30T13:19:28.467 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/12' 2023-10-30T13:19:28.467 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/39' 2023-10-30T13:19:28.468 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/10' 2023-10-30T13:19:28.468 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/34' 2023-10-30T13:19:28.468 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/38' 2023-10-30T13:19:28.468 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/11' 2023-10-30T13:19:28.468 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/16' 2023-10-30T13:19:28.468 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/32' 2023-10-30T13:19:28.469 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/30' 2023-10-30T13:19:28.469 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/14' 2023-10-30T13:19:28.469 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/18' 2023-10-30T13:19:28.469 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/13' 2023-10-30T13:19:28.469 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/33' 2023-10-30T13:19:28.469 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/17' 2023-10-30T13:19:28.470 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/35' 2023-10-30T13:19:28.470 INFO:tasks.workunit.client.0.smithi148.stdout:removed './192/37' 2023-10-30T13:19:28.470 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './192' 2023-10-30T13:19:28.470 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/19' 2023-10-30T13:19:28.470 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/15' 2023-10-30T13:19:28.470 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/31' 2023-10-30T13:19:28.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/36' 2023-10-30T13:19:28.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/20' 2023-10-30T13:19:28.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/12' 2023-10-30T13:19:28.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/39' 2023-10-30T13:19:28.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/10' 2023-10-30T13:19:28.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/34' 2023-10-30T13:19:28.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/38' 2023-10-30T13:19:28.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/11' 2023-10-30T13:19:28.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/16' 2023-10-30T13:19:28.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/32' 2023-10-30T13:19:28.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/30' 2023-10-30T13:19:28.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/14' 2023-10-30T13:19:28.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/18' 2023-10-30T13:19:28.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/13' 2023-10-30T13:19:28.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/33' 2023-10-30T13:19:28.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/17' 2023-10-30T13:19:28.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/35' 2023-10-30T13:19:28.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './193/37' 2023-10-30T13:19:28.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './193' 2023-10-30T13:19:28.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/19' 2023-10-30T13:19:28.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/15' 2023-10-30T13:19:28.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/31' 2023-10-30T13:19:28.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/36' 2023-10-30T13:19:28.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/20' 2023-10-30T13:19:28.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/12' 2023-10-30T13:19:28.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/39' 2023-10-30T13:19:28.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/10' 2023-10-30T13:19:28.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/34' 2023-10-30T13:19:28.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/38' 2023-10-30T13:19:28.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/11' 2023-10-30T13:19:28.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/16' 2023-10-30T13:19:28.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/32' 2023-10-30T13:19:28.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/30' 2023-10-30T13:19:28.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/14' 2023-10-30T13:19:28.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/18' 2023-10-30T13:19:28.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/13' 2023-10-30T13:19:28.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/33' 2023-10-30T13:19:28.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/17' 2023-10-30T13:19:28.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/35' 2023-10-30T13:19:28.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './194/37' 2023-10-30T13:19:28.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './194' 2023-10-30T13:19:28.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/19' 2023-10-30T13:19:28.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/15' 2023-10-30T13:19:28.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/31' 2023-10-30T13:19:28.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/36' 2023-10-30T13:19:28.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/20' 2023-10-30T13:19:28.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/12' 2023-10-30T13:19:28.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/39' 2023-10-30T13:19:28.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/10' 2023-10-30T13:19:28.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/34' 2023-10-30T13:19:28.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/38' 2023-10-30T13:19:28.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/11' 2023-10-30T13:19:28.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/16' 2023-10-30T13:19:28.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/32' 2023-10-30T13:19:28.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/30' 2023-10-30T13:19:28.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/14' 2023-10-30T13:19:28.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/18' 2023-10-30T13:19:28.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/13' 2023-10-30T13:19:28.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/33' 2023-10-30T13:19:28.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/17' 2023-10-30T13:19:28.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/35' 2023-10-30T13:19:28.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './195/37' 2023-10-30T13:19:28.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './195' 2023-10-30T13:19:28.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/19' 2023-10-30T13:19:28.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/15' 2023-10-30T13:19:28.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/31' 2023-10-30T13:19:28.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/36' 2023-10-30T13:19:28.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/20' 2023-10-30T13:19:28.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/12' 2023-10-30T13:19:28.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/39' 2023-10-30T13:19:28.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/10' 2023-10-30T13:19:36.470 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/34' 2023-10-30T13:19:36.470 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/38' 2023-10-30T13:19:36.470 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/11' 2023-10-30T13:19:36.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/16' 2023-10-30T13:19:36.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/32' 2023-10-30T13:19:36.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/30' 2023-10-30T13:19:36.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/14' 2023-10-30T13:19:36.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/18' 2023-10-30T13:19:36.471 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/13' 2023-10-30T13:19:36.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/33' 2023-10-30T13:19:36.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/17' 2023-10-30T13:19:36.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/35' 2023-10-30T13:19:36.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed './196/37' 2023-10-30T13:19:36.472 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './196' 2023-10-30T13:19:36.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/19' 2023-10-30T13:19:36.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/15' 2023-10-30T13:19:36.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/31' 2023-10-30T13:19:36.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/36' 2023-10-30T13:19:36.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/20' 2023-10-30T13:19:36.473 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/12' 2023-10-30T13:19:36.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/39' 2023-10-30T13:19:36.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/10' 2023-10-30T13:19:36.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/34' 2023-10-30T13:19:36.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/38' 2023-10-30T13:19:36.474 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/11' 2023-10-30T13:19:36.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/16' 2023-10-30T13:19:36.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/32' 2023-10-30T13:19:36.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/30' 2023-10-30T13:19:36.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/14' 2023-10-30T13:19:36.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/18' 2023-10-30T13:19:36.475 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/13' 2023-10-30T13:19:36.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/33' 2023-10-30T13:19:36.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/17' 2023-10-30T13:19:36.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/35' 2023-10-30T13:19:36.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed './197/37' 2023-10-30T13:19:36.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './197' 2023-10-30T13:19:36.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/19' 2023-10-30T13:19:36.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/15' 2023-10-30T13:19:36.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/31' 2023-10-30T13:19:36.477 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/36' 2023-10-30T13:19:36.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/20' 2023-10-30T13:19:36.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/12' 2023-10-30T13:19:36.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/39' 2023-10-30T13:19:36.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/10' 2023-10-30T13:19:36.478 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/34' 2023-10-30T13:19:36.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/38' 2023-10-30T13:19:36.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/11' 2023-10-30T13:19:36.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/16' 2023-10-30T13:19:36.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/32' 2023-10-30T13:19:36.479 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/30' 2023-10-30T13:19:36.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/14' 2023-10-30T13:19:36.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/18' 2023-10-30T13:19:36.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/13' 2023-10-30T13:19:36.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/33' 2023-10-30T13:19:36.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/17' 2023-10-30T13:19:36.480 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/35' 2023-10-30T13:19:36.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './198/37' 2023-10-30T13:19:36.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './198' 2023-10-30T13:19:36.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/19' 2023-10-30T13:19:36.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/15' 2023-10-30T13:19:36.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/31' 2023-10-30T13:19:36.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/36' 2023-10-30T13:19:36.481 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/20' 2023-10-30T13:19:36.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/12' 2023-10-30T13:19:36.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/39' 2023-10-30T13:19:36.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/10' 2023-10-30T13:19:36.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/34' 2023-10-30T13:19:36.482 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/38' 2023-10-30T13:19:36.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/11' 2023-10-30T13:19:36.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/16' 2023-10-30T13:19:36.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/32' 2023-10-30T13:19:36.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/30' 2023-10-30T13:19:36.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/14' 2023-10-30T13:19:36.483 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/18' 2023-10-30T13:19:36.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/13' 2023-10-30T13:19:36.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/33' 2023-10-30T13:19:36.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/17' 2023-10-30T13:19:36.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/35' 2023-10-30T13:19:36.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed './199/37' 2023-10-30T13:19:36.484 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './199' 2023-10-30T13:19:36.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/21' 2023-10-30T13:19:36.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/27' 2023-10-30T13:19:36.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/31' 2023-10-30T13:19:36.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/36' 2023-10-30T13:19:36.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/20' 2023-10-30T13:19:36.485 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/39' 2023-10-30T13:19:36.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/34' 2023-10-30T13:19:36.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/38' 2023-10-30T13:19:36.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/28' 2023-10-30T13:19:36.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/24' 2023-10-30T13:19:36.486 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/29' 2023-10-30T13:19:36.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/32' 2023-10-30T13:19:36.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/22' 2023-10-30T13:19:36.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/25' 2023-10-30T13:19:36.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/30' 2023-10-30T13:19:36.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/33' 2023-10-30T13:19:36.487 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/23' 2023-10-30T13:19:36.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/35' 2023-10-30T13:19:36.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/26' 2023-10-30T13:19:36.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed './200/37' 2023-10-30T13:19:36.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './200' 2023-10-30T13:19:36.488 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/21' 2023-10-30T13:19:36.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/27' 2023-10-30T13:19:36.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/31' 2023-10-30T13:19:36.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/36' 2023-10-30T13:19:36.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/20' 2023-10-30T13:19:36.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/39' 2023-10-30T13:19:36.489 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/34' 2023-10-30T13:19:36.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/38' 2023-10-30T13:19:36.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/28' 2023-10-30T13:19:36.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/24' 2023-10-30T13:19:36.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/29' 2023-10-30T13:19:36.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/32' 2023-10-30T13:19:36.490 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/22' 2023-10-30T13:19:36.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/25' 2023-10-30T13:19:36.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/30' 2023-10-30T13:19:36.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/33' 2023-10-30T13:19:36.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/23' 2023-10-30T13:19:36.491 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/35' 2023-10-30T13:19:36.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/26' 2023-10-30T13:19:36.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed './201/37' 2023-10-30T13:19:36.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './201' 2023-10-30T13:19:36.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/21' 2023-10-30T13:19:36.492 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/27' 2023-10-30T13:19:36.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/31' 2023-10-30T13:19:36.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/36' 2023-10-30T13:19:36.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/20' 2023-10-30T13:19:36.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/39' 2023-10-30T13:19:36.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/34' 2023-10-30T13:19:36.493 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/38' 2023-10-30T13:19:36.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/28' 2023-10-30T13:19:36.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/24' 2023-10-30T13:19:36.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/29' 2023-10-30T13:19:36.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/32' 2023-10-30T13:19:36.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/22' 2023-10-30T13:19:36.494 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/25' 2023-10-30T13:19:36.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/30' 2023-10-30T13:19:36.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/33' 2023-10-30T13:19:36.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/23' 2023-10-30T13:19:36.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/35' 2023-10-30T13:19:36.495 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/26' 2023-10-30T13:19:36.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed './202/37' 2023-10-30T13:19:36.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './202' 2023-10-30T13:19:36.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/21' 2023-10-30T13:19:36.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/27' 2023-10-30T13:19:36.496 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/31' 2023-10-30T13:19:36.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/36' 2023-10-30T13:19:36.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/20' 2023-10-30T13:19:36.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/39' 2023-10-30T13:19:36.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/34' 2023-10-30T13:19:36.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/38' 2023-10-30T13:19:36.497 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/28' 2023-10-30T13:19:36.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/24' 2023-10-30T13:19:36.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/29' 2023-10-30T13:19:36.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/32' 2023-10-30T13:19:36.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/22' 2023-10-30T13:19:36.498 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/25' 2023-10-30T13:19:36.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/30' 2023-10-30T13:19:36.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/33' 2023-10-30T13:19:36.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/23' 2023-10-30T13:19:36.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/35' 2023-10-30T13:19:36.499 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/26' 2023-10-30T13:19:36.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './203/37' 2023-10-30T13:19:36.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './203' 2023-10-30T13:19:36.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/21' 2023-10-30T13:19:36.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/27' 2023-10-30T13:19:36.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/31' 2023-10-30T13:19:36.500 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/36' 2023-10-30T13:19:36.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/20' 2023-10-30T13:19:36.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/39' 2023-10-30T13:19:36.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/34' 2023-10-30T13:19:36.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/38' 2023-10-30T13:19:36.501 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/28' 2023-10-30T13:19:36.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/24' 2023-10-30T13:19:36.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/29' 2023-10-30T13:19:36.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/32' 2023-10-30T13:19:36.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/22' 2023-10-30T13:19:36.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/25' 2023-10-30T13:19:36.502 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/30' 2023-10-30T13:19:36.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/33' 2023-10-30T13:19:36.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/23' 2023-10-30T13:19:36.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/35' 2023-10-30T13:19:36.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/26' 2023-10-30T13:19:36.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed './204/37' 2023-10-30T13:19:36.503 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './204' 2023-10-30T13:19:36.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/21' 2023-10-30T13:19:36.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/27' 2023-10-30T13:19:36.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/31' 2023-10-30T13:19:36.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/36' 2023-10-30T13:19:36.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/20' 2023-10-30T13:19:36.504 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/39' 2023-10-30T13:19:36.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/34' 2023-10-30T13:19:36.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/38' 2023-10-30T13:19:36.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/28' 2023-10-30T13:19:36.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/24' 2023-10-30T13:19:36.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/29' 2023-10-30T13:19:36.505 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/32' 2023-10-30T13:19:36.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/22' 2023-10-30T13:19:36.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/25' 2023-10-30T13:19:36.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/30' 2023-10-30T13:19:36.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/33' 2023-10-30T13:19:36.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/23' 2023-10-30T13:19:36.506 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/35' 2023-10-30T13:19:36.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/26' 2023-10-30T13:19:36.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed './205/37' 2023-10-30T13:19:36.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './205' 2023-10-30T13:19:36.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/21' 2023-10-30T13:19:36.507 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/27' 2023-10-30T13:19:36.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/31' 2023-10-30T13:19:36.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/36' 2023-10-30T13:19:36.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/20' 2023-10-30T13:19:36.508 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/39' 2023-10-30T13:19:38.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/34' 2023-10-30T13:19:38.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/38' 2023-10-30T13:19:38.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/28' 2023-10-30T13:19:38.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/24' 2023-10-30T13:19:38.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/29' 2023-10-30T13:19:38.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/32' 2023-10-30T13:19:38.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/22' 2023-10-30T13:19:38.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/25' 2023-10-30T13:19:38.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/30' 2023-10-30T13:19:38.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/33' 2023-10-30T13:19:38.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/23' 2023-10-30T13:19:38.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/35' 2023-10-30T13:19:38.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/26' 2023-10-30T13:19:38.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './206/37' 2023-10-30T13:19:38.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './206' 2023-10-30T13:19:38.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/21' 2023-10-30T13:19:38.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/27' 2023-10-30T13:19:38.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/31' 2023-10-30T13:19:38.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/36' 2023-10-30T13:19:38.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/20' 2023-10-30T13:19:38.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/39' 2023-10-30T13:19:38.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/34' 2023-10-30T13:19:38.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/38' 2023-10-30T13:19:38.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/28' 2023-10-30T13:19:38.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/24' 2023-10-30T13:19:38.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/29' 2023-10-30T13:19:38.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/32' 2023-10-30T13:19:38.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/22' 2023-10-30T13:19:38.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/25' 2023-10-30T13:19:38.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/30' 2023-10-30T13:19:38.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/33' 2023-10-30T13:19:38.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/23' 2023-10-30T13:19:38.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/35' 2023-10-30T13:19:38.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/26' 2023-10-30T13:19:38.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './207/37' 2023-10-30T13:19:38.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './207' 2023-10-30T13:19:38.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/21' 2023-10-30T13:19:38.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/27' 2023-10-30T13:19:38.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/31' 2023-10-30T13:19:38.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/36' 2023-10-30T13:19:38.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/20' 2023-10-30T13:19:38.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/39' 2023-10-30T13:19:38.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/34' 2023-10-30T13:19:38.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/38' 2023-10-30T13:19:38.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/28' 2023-10-30T13:19:38.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/24' 2023-10-30T13:19:38.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/29' 2023-10-30T13:19:38.772 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/32' 2023-10-30T13:19:38.772 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/22' 2023-10-30T13:19:38.772 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/25' 2023-10-30T13:19:38.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/30' 2023-10-30T13:19:38.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/33' 2023-10-30T13:19:38.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/23' 2023-10-30T13:19:38.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/35' 2023-10-30T13:19:38.774 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/26' 2023-10-30T13:19:38.774 INFO:tasks.workunit.client.0.smithi148.stdout:removed './208/37' 2023-10-30T13:19:38.774 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './208' 2023-10-30T13:19:38.774 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/21' 2023-10-30T13:19:38.775 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/27' 2023-10-30T13:19:38.775 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/31' 2023-10-30T13:19:38.775 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/36' 2023-10-30T13:19:38.775 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/20' 2023-10-30T13:19:38.776 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/39' 2023-10-30T13:19:38.776 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/34' 2023-10-30T13:19:38.776 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/38' 2023-10-30T13:19:38.776 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/28' 2023-10-30T13:19:38.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/24' 2023-10-30T13:19:38.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/29' 2023-10-30T13:19:38.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/32' 2023-10-30T13:19:38.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/22' 2023-10-30T13:19:38.778 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/25' 2023-10-30T13:19:38.778 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/30' 2023-10-30T13:19:38.778 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/33' 2023-10-30T13:19:38.778 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/23' 2023-10-30T13:19:38.779 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/35' 2023-10-30T13:19:38.779 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/26' 2023-10-30T13:19:38.779 INFO:tasks.workunit.client.0.smithi148.stdout:removed './209/37' 2023-10-30T13:19:38.779 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './209' 2023-10-30T13:19:38.780 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/21' 2023-10-30T13:19:38.780 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/27' 2023-10-30T13:19:38.780 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/31' 2023-10-30T13:19:38.780 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/36' 2023-10-30T13:19:38.781 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/20' 2023-10-30T13:19:38.781 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/39' 2023-10-30T13:19:38.781 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/34' 2023-10-30T13:19:38.781 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/38' 2023-10-30T13:19:38.781 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/28' 2023-10-30T13:19:38.782 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/24' 2023-10-30T13:19:38.782 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/29' 2023-10-30T13:19:38.782 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/32' 2023-10-30T13:19:38.782 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/22' 2023-10-30T13:19:38.783 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/25' 2023-10-30T13:19:38.783 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/30' 2023-10-30T13:19:38.783 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/33' 2023-10-30T13:19:38.783 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/23' 2023-10-30T13:19:38.784 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/35' 2023-10-30T13:19:38.784 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/26' 2023-10-30T13:19:38.784 INFO:tasks.workunit.client.0.smithi148.stdout:removed './210/37' 2023-10-30T13:19:38.784 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './210' 2023-10-30T13:19:38.785 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/21' 2023-10-30T13:19:38.785 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/27' 2023-10-30T13:19:38.785 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/31' 2023-10-30T13:19:38.785 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/36' 2023-10-30T13:19:38.786 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/20' 2023-10-30T13:19:38.786 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/39' 2023-10-30T13:19:38.786 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/34' 2023-10-30T13:19:38.786 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/38' 2023-10-30T13:19:38.787 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/28' 2023-10-30T13:19:38.787 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/24' 2023-10-30T13:19:38.787 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/29' 2023-10-30T13:19:38.787 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/32' 2023-10-30T13:19:38.788 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/22' 2023-10-30T13:19:38.788 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/25' 2023-10-30T13:19:38.788 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/30' 2023-10-30T13:19:38.788 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/33' 2023-10-30T13:19:38.789 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/23' 2023-10-30T13:19:38.789 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/35' 2023-10-30T13:19:38.789 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/26' 2023-10-30T13:19:38.789 INFO:tasks.workunit.client.0.smithi148.stdout:removed './211/37' 2023-10-30T13:19:38.790 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './211' 2023-10-30T13:19:38.790 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/21' 2023-10-30T13:19:38.790 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/27' 2023-10-30T13:19:38.790 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/31' 2023-10-30T13:19:38.791 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/36' 2023-10-30T13:19:38.791 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/20' 2023-10-30T13:19:38.791 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/39' 2023-10-30T13:19:38.791 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/34' 2023-10-30T13:19:38.792 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/38' 2023-10-30T13:19:38.792 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/28' 2023-10-30T13:19:38.792 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/24' 2023-10-30T13:19:38.792 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/29' 2023-10-30T13:19:38.793 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/32' 2023-10-30T13:19:38.793 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/22' 2023-10-30T13:19:38.793 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/25' 2023-10-30T13:19:38.793 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/30' 2023-10-30T13:19:38.794 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/33' 2023-10-30T13:19:38.794 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/23' 2023-10-30T13:19:38.794 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/35' 2023-10-30T13:19:38.794 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/26' 2023-10-30T13:19:38.794 INFO:tasks.workunit.client.0.smithi148.stdout:removed './212/37' 2023-10-30T13:19:38.795 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './212' 2023-10-30T13:19:38.795 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/21' 2023-10-30T13:19:38.795 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/27' 2023-10-30T13:19:38.795 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/31' 2023-10-30T13:19:38.796 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/36' 2023-10-30T13:19:38.796 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/20' 2023-10-30T13:19:38.796 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/39' 2023-10-30T13:19:38.796 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/34' 2023-10-30T13:19:38.797 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/38' 2023-10-30T13:19:38.797 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/28' 2023-10-30T13:19:38.797 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/24' 2023-10-30T13:19:38.797 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/29' 2023-10-30T13:19:38.798 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/32' 2023-10-30T13:19:38.798 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/22' 2023-10-30T13:19:38.798 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/25' 2023-10-30T13:19:38.798 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/30' 2023-10-30T13:19:38.799 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/33' 2023-10-30T13:19:38.799 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/23' 2023-10-30T13:19:38.799 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/35' 2023-10-30T13:19:38.799 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/26' 2023-10-30T13:19:38.799 INFO:tasks.workunit.client.0.smithi148.stdout:removed './213/37' 2023-10-30T13:19:38.800 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './213' 2023-10-30T13:19:38.800 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/21' 2023-10-30T13:19:38.800 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/27' 2023-10-30T13:19:38.800 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/31' 2023-10-30T13:19:38.801 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/36' 2023-10-30T13:19:38.801 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/20' 2023-10-30T13:19:38.801 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/39' 2023-10-30T13:19:38.801 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/34' 2023-10-30T13:19:38.802 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/38' 2023-10-30T13:19:38.802 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/28' 2023-10-30T13:19:38.802 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/24' 2023-10-30T13:19:38.802 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/29' 2023-10-30T13:19:38.802 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/32' 2023-10-30T13:19:38.803 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/22' 2023-10-30T13:19:38.803 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/25' 2023-10-30T13:19:38.803 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/30' 2023-10-30T13:19:38.803 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/33' 2023-10-30T13:19:38.804 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/23' 2023-10-30T13:19:38.804 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/35' 2023-10-30T13:19:38.804 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/26' 2023-10-30T13:19:38.804 INFO:tasks.workunit.client.0.smithi148.stdout:removed './214/37' 2023-10-30T13:19:38.805 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './214' 2023-10-30T13:19:38.805 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/21' 2023-10-30T13:19:38.805 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/27' 2023-10-30T13:19:38.805 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/31' 2023-10-30T13:19:38.805 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/36' 2023-10-30T13:19:38.806 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/20' 2023-10-30T13:19:38.806 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/39' 2023-10-30T13:19:38.806 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/34' 2023-10-30T13:19:38.806 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/38' 2023-10-30T13:19:38.806 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/28' 2023-10-30T13:19:38.807 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/24' 2023-10-30T13:19:38.807 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/29' 2023-10-30T13:19:38.807 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/32' 2023-10-30T13:19:38.807 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/22' 2023-10-30T13:19:38.808 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/25' 2023-10-30T13:19:38.808 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/30' 2023-10-30T13:19:38.808 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/33' 2023-10-30T13:19:38.808 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/23' 2023-10-30T13:19:38.808 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/35' 2023-10-30T13:19:38.809 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/26' 2023-10-30T13:19:38.809 INFO:tasks.workunit.client.0.smithi148.stdout:removed './215/37' 2023-10-30T13:19:38.809 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './215' 2023-10-30T13:19:38.809 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/21' 2023-10-30T13:19:38.810 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/27' 2023-10-30T13:19:38.810 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/31' 2023-10-30T13:19:38.810 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/36' 2023-10-30T13:19:38.810 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/20' 2023-10-30T13:19:38.810 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/39' 2023-10-30T13:19:38.811 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/34' 2023-10-30T13:19:38.811 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/38' 2023-10-30T13:19:41.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/28' 2023-10-30T13:19:41.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/24' 2023-10-30T13:19:41.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/29' 2023-10-30T13:19:41.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/32' 2023-10-30T13:19:41.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/22' 2023-10-30T13:19:41.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/25' 2023-10-30T13:19:41.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/30' 2023-10-30T13:19:41.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/33' 2023-10-30T13:19:41.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/23' 2023-10-30T13:19:41.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/35' 2023-10-30T13:19:41.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/26' 2023-10-30T13:19:41.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './216/37' 2023-10-30T13:19:41.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './216' 2023-10-30T13:19:41.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/21' 2023-10-30T13:19:41.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/27' 2023-10-30T13:19:41.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/31' 2023-10-30T13:19:41.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/36' 2023-10-30T13:19:41.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/20' 2023-10-30T13:19:41.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/39' 2023-10-30T13:19:41.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/34' 2023-10-30T13:19:41.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/38' 2023-10-30T13:19:41.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/28' 2023-10-30T13:19:41.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/24' 2023-10-30T13:19:41.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/29' 2023-10-30T13:19:41.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/32' 2023-10-30T13:19:41.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/22' 2023-10-30T13:19:41.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/25' 2023-10-30T13:19:41.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/30' 2023-10-30T13:19:41.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/33' 2023-10-30T13:19:41.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/23' 2023-10-30T13:19:41.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/35' 2023-10-30T13:19:41.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/26' 2023-10-30T13:19:41.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed './217/37' 2023-10-30T13:19:41.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './217' 2023-10-30T13:19:41.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/21' 2023-10-30T13:19:41.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/27' 2023-10-30T13:19:41.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/31' 2023-10-30T13:19:41.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/36' 2023-10-30T13:19:41.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/20' 2023-10-30T13:19:41.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/39' 2023-10-30T13:19:41.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/34' 2023-10-30T13:19:41.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/38' 2023-10-30T13:19:41.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/28' 2023-10-30T13:19:41.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/24' 2023-10-30T13:19:41.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/29' 2023-10-30T13:19:41.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/32' 2023-10-30T13:19:41.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/22' 2023-10-30T13:19:41.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/25' 2023-10-30T13:19:41.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/30' 2023-10-30T13:19:41.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/33' 2023-10-30T13:19:41.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/23' 2023-10-30T13:19:41.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/35' 2023-10-30T13:19:41.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/26' 2023-10-30T13:19:41.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './218/37' 2023-10-30T13:19:41.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './218' 2023-10-30T13:19:41.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/21' 2023-10-30T13:19:41.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/27' 2023-10-30T13:19:41.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/31' 2023-10-30T13:19:41.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/36' 2023-10-30T13:19:41.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/20' 2023-10-30T13:19:41.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/39' 2023-10-30T13:19:41.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/34' 2023-10-30T13:19:41.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/38' 2023-10-30T13:19:41.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/28' 2023-10-30T13:19:41.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/24' 2023-10-30T13:19:41.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/29' 2023-10-30T13:19:41.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/32' 2023-10-30T13:19:41.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/22' 2023-10-30T13:19:41.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/25' 2023-10-30T13:19:41.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/30' 2023-10-30T13:19:41.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/33' 2023-10-30T13:19:41.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/23' 2023-10-30T13:19:41.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/35' 2023-10-30T13:19:41.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/26' 2023-10-30T13:19:41.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './219/37' 2023-10-30T13:19:41.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './219' 2023-10-30T13:19:41.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/21' 2023-10-30T13:19:41.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/27' 2023-10-30T13:19:41.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/31' 2023-10-30T13:19:41.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/36' 2023-10-30T13:19:41.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/20' 2023-10-30T13:19:41.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/39' 2023-10-30T13:19:41.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/34' 2023-10-30T13:19:41.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/38' 2023-10-30T13:19:41.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/28' 2023-10-30T13:19:41.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/24' 2023-10-30T13:19:41.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/29' 2023-10-30T13:19:41.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/32' 2023-10-30T13:19:41.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/22' 2023-10-30T13:19:41.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/25' 2023-10-30T13:19:41.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/30' 2023-10-30T13:19:41.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/33' 2023-10-30T13:19:41.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/23' 2023-10-30T13:19:41.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/35' 2023-10-30T13:19:41.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/26' 2023-10-30T13:19:41.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './220/37' 2023-10-30T13:19:41.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './220' 2023-10-30T13:19:41.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/21' 2023-10-30T13:19:41.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/27' 2023-10-30T13:19:41.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/31' 2023-10-30T13:19:41.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/36' 2023-10-30T13:19:41.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/20' 2023-10-30T13:19:41.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/39' 2023-10-30T13:19:41.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/34' 2023-10-30T13:19:41.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/38' 2023-10-30T13:19:41.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/28' 2023-10-30T13:19:41.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/24' 2023-10-30T13:19:41.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/29' 2023-10-30T13:19:41.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/32' 2023-10-30T13:19:41.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/22' 2023-10-30T13:19:41.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/25' 2023-10-30T13:19:41.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/30' 2023-10-30T13:19:41.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/33' 2023-10-30T13:19:41.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/23' 2023-10-30T13:19:41.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/35' 2023-10-30T13:19:41.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/26' 2023-10-30T13:19:41.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './221/37' 2023-10-30T13:19:41.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './221' 2023-10-30T13:19:41.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/21' 2023-10-30T13:19:41.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/27' 2023-10-30T13:19:41.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/31' 2023-10-30T13:19:41.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/36' 2023-10-30T13:19:41.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/20' 2023-10-30T13:19:41.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/39' 2023-10-30T13:19:41.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/34' 2023-10-30T13:19:41.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/38' 2023-10-30T13:19:41.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/28' 2023-10-30T13:19:41.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/24' 2023-10-30T13:19:41.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/29' 2023-10-30T13:19:41.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/32' 2023-10-30T13:19:41.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/22' 2023-10-30T13:19:41.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/25' 2023-10-30T13:19:41.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/30' 2023-10-30T13:19:41.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/33' 2023-10-30T13:19:41.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/23' 2023-10-30T13:19:41.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/35' 2023-10-30T13:19:41.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/26' 2023-10-30T13:19:41.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './222/37' 2023-10-30T13:19:41.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './222' 2023-10-30T13:19:41.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/21' 2023-10-30T13:19:41.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/27' 2023-10-30T13:19:41.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/31' 2023-10-30T13:19:41.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/36' 2023-10-30T13:19:41.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/20' 2023-10-30T13:19:41.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/39' 2023-10-30T13:19:41.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/34' 2023-10-30T13:19:41.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/38' 2023-10-30T13:19:41.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/28' 2023-10-30T13:19:41.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/24' 2023-10-30T13:19:41.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/29' 2023-10-30T13:19:41.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/32' 2023-10-30T13:19:41.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/22' 2023-10-30T13:19:41.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/25' 2023-10-30T13:19:41.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/30' 2023-10-30T13:19:41.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/33' 2023-10-30T13:19:41.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/23' 2023-10-30T13:19:41.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/35' 2023-10-30T13:19:41.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/26' 2023-10-30T13:19:41.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './223/37' 2023-10-30T13:19:41.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './223' 2023-10-30T13:19:41.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/21' 2023-10-30T13:19:41.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/27' 2023-10-30T13:19:41.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/31' 2023-10-30T13:19:41.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/36' 2023-10-30T13:19:41.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/20' 2023-10-30T13:19:41.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/39' 2023-10-30T13:19:41.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/34' 2023-10-30T13:19:41.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/38' 2023-10-30T13:19:41.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/28' 2023-10-30T13:19:41.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/24' 2023-10-30T13:19:41.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/29' 2023-10-30T13:19:41.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/32' 2023-10-30T13:19:41.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/22' 2023-10-30T13:19:41.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/25' 2023-10-30T13:19:41.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/30' 2023-10-30T13:19:41.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/33' 2023-10-30T13:19:41.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/23' 2023-10-30T13:19:41.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/35' 2023-10-30T13:19:41.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/26' 2023-10-30T13:19:41.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './224/37' 2023-10-30T13:19:41.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './224' 2023-10-30T13:19:41.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/21' 2023-10-30T13:19:41.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/27' 2023-10-30T13:19:41.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/31' 2023-10-30T13:19:41.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/36' 2023-10-30T13:19:41.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/20' 2023-10-30T13:19:41.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/39' 2023-10-30T13:19:41.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/34' 2023-10-30T13:19:41.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/38' 2023-10-30T13:19:41.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/28' 2023-10-30T13:19:41.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/24' 2023-10-30T13:19:41.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/29' 2023-10-30T13:19:41.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/32' 2023-10-30T13:19:41.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/22' 2023-10-30T13:19:41.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/25' 2023-10-30T13:19:41.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/30' 2023-10-30T13:19:41.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/33' 2023-10-30T13:19:41.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/23' 2023-10-30T13:19:41.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/35' 2023-10-30T13:19:41.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/26' 2023-10-30T13:19:41.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './225/37' 2023-10-30T13:19:41.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './225' 2023-10-30T13:19:41.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/21' 2023-10-30T13:19:41.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/27' 2023-10-30T13:19:41.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/31' 2023-10-30T13:19:41.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/36' 2023-10-30T13:19:41.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/20' 2023-10-30T13:19:41.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/39' 2023-10-30T13:19:41.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/34' 2023-10-30T13:19:41.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/38' 2023-10-30T13:19:41.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/28' 2023-10-30T13:19:41.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/24' 2023-10-30T13:19:45.816 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/29' 2023-10-30T13:19:45.817 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/32' 2023-10-30T13:19:45.817 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/22' 2023-10-30T13:19:45.817 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/25' 2023-10-30T13:19:45.817 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/30' 2023-10-30T13:19:45.817 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/33' 2023-10-30T13:19:45.817 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/23' 2023-10-30T13:19:45.818 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/35' 2023-10-30T13:19:45.818 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/26' 2023-10-30T13:19:45.818 INFO:tasks.workunit.client.0.smithi148.stdout:removed './226/37' 2023-10-30T13:19:45.818 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './226' 2023-10-30T13:19:45.819 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/21' 2023-10-30T13:19:45.819 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/27' 2023-10-30T13:19:45.819 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/31' 2023-10-30T13:19:45.819 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/36' 2023-10-30T13:19:45.819 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/20' 2023-10-30T13:19:45.819 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/39' 2023-10-30T13:19:45.820 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/34' 2023-10-30T13:19:45.820 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/38' 2023-10-30T13:19:45.820 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/28' 2023-10-30T13:19:45.820 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/24' 2023-10-30T13:19:45.820 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/29' 2023-10-30T13:19:45.820 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/32' 2023-10-30T13:19:45.821 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/22' 2023-10-30T13:19:45.821 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/25' 2023-10-30T13:19:45.821 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/30' 2023-10-30T13:19:45.821 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/33' 2023-10-30T13:19:45.821 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/23' 2023-10-30T13:19:45.821 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/35' 2023-10-30T13:19:45.821 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/26' 2023-10-30T13:19:45.822 INFO:tasks.workunit.client.0.smithi148.stdout:removed './227/37' 2023-10-30T13:19:45.822 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './227' 2023-10-30T13:19:45.822 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/21' 2023-10-30T13:19:45.822 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/27' 2023-10-30T13:19:45.822 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/31' 2023-10-30T13:19:45.823 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/36' 2023-10-30T13:19:45.823 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/20' 2023-10-30T13:19:45.823 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/39' 2023-10-30T13:19:45.823 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/34' 2023-10-30T13:19:45.823 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/38' 2023-10-30T13:19:45.823 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/28' 2023-10-30T13:19:45.824 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/24' 2023-10-30T13:19:45.824 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/29' 2023-10-30T13:19:45.824 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/32' 2023-10-30T13:19:45.824 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/22' 2023-10-30T13:19:45.824 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/25' 2023-10-30T13:19:45.824 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/30' 2023-10-30T13:19:45.825 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/33' 2023-10-30T13:19:45.825 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/23' 2023-10-30T13:19:45.825 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/35' 2023-10-30T13:19:45.825 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/26' 2023-10-30T13:19:45.825 INFO:tasks.workunit.client.0.smithi148.stdout:removed './228/37' 2023-10-30T13:19:45.825 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './228' 2023-10-30T13:19:45.825 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/21' 2023-10-30T13:19:45.826 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/27' 2023-10-30T13:19:45.826 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/31' 2023-10-30T13:19:45.826 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/36' 2023-10-30T13:19:45.826 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/20' 2023-10-30T13:19:45.826 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/39' 2023-10-30T13:19:45.827 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/34' 2023-10-30T13:19:45.827 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/38' 2023-10-30T13:19:45.827 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/28' 2023-10-30T13:19:45.827 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/24' 2023-10-30T13:19:45.827 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/29' 2023-10-30T13:19:45.827 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/32' 2023-10-30T13:19:45.828 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/22' 2023-10-30T13:19:45.828 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/25' 2023-10-30T13:19:45.828 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/30' 2023-10-30T13:19:45.828 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/33' 2023-10-30T13:19:45.828 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/23' 2023-10-30T13:19:45.828 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/35' 2023-10-30T13:19:45.829 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/26' 2023-10-30T13:19:45.829 INFO:tasks.workunit.client.0.smithi148.stdout:removed './229/37' 2023-10-30T13:19:45.829 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './229' 2023-10-30T13:19:45.829 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/21' 2023-10-30T13:19:45.829 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/27' 2023-10-30T13:19:45.829 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/31' 2023-10-30T13:19:45.830 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/36' 2023-10-30T13:19:45.830 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/20' 2023-10-30T13:19:45.830 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/39' 2023-10-30T13:19:45.830 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/34' 2023-10-30T13:19:45.830 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/38' 2023-10-30T13:19:45.830 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/28' 2023-10-30T13:19:45.831 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/24' 2023-10-30T13:19:45.831 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/29' 2023-10-30T13:19:45.831 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/32' 2023-10-30T13:19:45.831 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/22' 2023-10-30T13:19:45.831 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/25' 2023-10-30T13:19:45.831 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/30' 2023-10-30T13:19:45.832 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/33' 2023-10-30T13:19:45.832 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/23' 2023-10-30T13:19:45.832 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/35' 2023-10-30T13:19:45.832 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/26' 2023-10-30T13:19:45.832 INFO:tasks.workunit.client.0.smithi148.stdout:removed './230/37' 2023-10-30T13:19:45.832 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './230' 2023-10-30T13:19:45.833 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/21' 2023-10-30T13:19:45.833 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/27' 2023-10-30T13:19:45.833 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/31' 2023-10-30T13:19:45.833 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/36' 2023-10-30T13:19:45.833 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/20' 2023-10-30T13:19:45.833 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/39' 2023-10-30T13:19:45.834 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/34' 2023-10-30T13:19:45.834 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/38' 2023-10-30T13:19:45.834 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/28' 2023-10-30T13:19:45.834 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/24' 2023-10-30T13:19:45.834 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/29' 2023-10-30T13:19:45.834 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/32' 2023-10-30T13:19:45.835 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/22' 2023-10-30T13:19:45.835 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/25' 2023-10-30T13:19:45.835 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/30' 2023-10-30T13:19:45.835 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/33' 2023-10-30T13:19:45.835 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/23' 2023-10-30T13:19:45.835 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/35' 2023-10-30T13:19:45.836 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/26' 2023-10-30T13:19:45.836 INFO:tasks.workunit.client.0.smithi148.stdout:removed './231/37' 2023-10-30T13:19:45.836 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './231' 2023-10-30T13:19:45.836 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/21' 2023-10-30T13:19:45.836 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/27' 2023-10-30T13:19:45.836 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/31' 2023-10-30T13:19:45.837 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/36' 2023-10-30T13:19:45.837 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/20' 2023-10-30T13:19:45.837 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/39' 2023-10-30T13:19:45.837 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/34' 2023-10-30T13:19:45.837 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/38' 2023-10-30T13:19:45.837 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/28' 2023-10-30T13:19:45.838 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/24' 2023-10-30T13:19:45.838 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/29' 2023-10-30T13:19:45.838 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/32' 2023-10-30T13:19:45.838 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/22' 2023-10-30T13:19:45.838 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/25' 2023-10-30T13:19:45.838 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/30' 2023-10-30T13:19:45.839 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/33' 2023-10-30T13:19:45.839 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/23' 2023-10-30T13:19:45.839 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/35' 2023-10-30T13:19:45.839 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/26' 2023-10-30T13:19:45.839 INFO:tasks.workunit.client.0.smithi148.stdout:removed './232/37' 2023-10-30T13:19:45.839 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './232' 2023-10-30T13:19:45.840 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/21' 2023-10-30T13:19:45.840 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/27' 2023-10-30T13:19:45.840 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/31' 2023-10-30T13:19:45.840 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/36' 2023-10-30T13:19:45.840 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/20' 2023-10-30T13:19:45.840 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/39' 2023-10-30T13:19:45.841 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/34' 2023-10-30T13:19:45.841 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/38' 2023-10-30T13:19:45.841 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/28' 2023-10-30T13:19:45.841 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/24' 2023-10-30T13:19:45.841 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/29' 2023-10-30T13:19:45.841 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/32' 2023-10-30T13:19:45.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/22' 2023-10-30T13:19:45.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/25' 2023-10-30T13:19:45.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/30' 2023-10-30T13:19:45.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/33' 2023-10-30T13:19:45.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/23' 2023-10-30T13:19:45.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/35' 2023-10-30T13:19:45.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/26' 2023-10-30T13:19:45.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed './233/37' 2023-10-30T13:19:45.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './233' 2023-10-30T13:19:45.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/21' 2023-10-30T13:19:45.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/27' 2023-10-30T13:19:45.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/31' 2023-10-30T13:19:45.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/36' 2023-10-30T13:19:45.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/20' 2023-10-30T13:19:45.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/39' 2023-10-30T13:19:45.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/34' 2023-10-30T13:19:45.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/38' 2023-10-30T13:19:45.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/28' 2023-10-30T13:19:45.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/24' 2023-10-30T13:19:45.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/29' 2023-10-30T13:19:45.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/32' 2023-10-30T13:19:45.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/22' 2023-10-30T13:19:45.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/25' 2023-10-30T13:19:45.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/30' 2023-10-30T13:19:45.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/33' 2023-10-30T13:19:45.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/23' 2023-10-30T13:19:45.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/35' 2023-10-30T13:19:45.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/26' 2023-10-30T13:19:45.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed './234/37' 2023-10-30T13:19:45.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './234' 2023-10-30T13:19:45.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/21' 2023-10-30T13:19:45.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/27' 2023-10-30T13:19:45.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/31' 2023-10-30T13:19:45.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/36' 2023-10-30T13:19:45.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/20' 2023-10-30T13:19:45.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/39' 2023-10-30T13:19:45.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/34' 2023-10-30T13:19:45.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/38' 2023-10-30T13:19:45.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/28' 2023-10-30T13:19:45.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/24' 2023-10-30T13:19:45.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/29' 2023-10-30T13:19:45.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/32' 2023-10-30T13:19:45.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/22' 2023-10-30T13:19:45.849 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/25' 2023-10-30T13:19:45.849 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/30' 2023-10-30T13:19:45.849 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/33' 2023-10-30T13:19:45.849 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/23' 2023-10-30T13:19:45.849 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/35' 2023-10-30T13:19:45.849 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/26' 2023-10-30T13:19:45.850 INFO:tasks.workunit.client.0.smithi148.stdout:removed './235/37' 2023-10-30T13:19:45.850 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './235' 2023-10-30T13:19:45.850 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/21' 2023-10-30T13:19:45.850 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/27' 2023-10-30T13:19:45.850 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/31' 2023-10-30T13:19:45.850 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/36' 2023-10-30T13:19:45.851 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/20' 2023-10-30T13:19:45.851 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/39' 2023-10-30T13:19:45.851 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/34' 2023-10-30T13:19:45.851 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/38' 2023-10-30T13:19:45.851 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/28' 2023-10-30T13:19:45.851 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/24' 2023-10-30T13:19:45.851 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/29' 2023-10-30T13:19:45.852 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/32' 2023-10-30T13:19:54.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/22' 2023-10-30T13:19:54.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/25' 2023-10-30T13:19:54.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/30' 2023-10-30T13:19:54.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/33' 2023-10-30T13:19:54.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/23' 2023-10-30T13:19:54.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/35' 2023-10-30T13:19:54.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/26' 2023-10-30T13:19:54.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './236/37' 2023-10-30T13:19:54.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './236' 2023-10-30T13:19:54.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/21' 2023-10-30T13:19:54.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/27' 2023-10-30T13:19:54.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/31' 2023-10-30T13:19:54.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/36' 2023-10-30T13:19:54.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/20' 2023-10-30T13:19:54.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/39' 2023-10-30T13:19:54.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/34' 2023-10-30T13:19:54.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/38' 2023-10-30T13:19:54.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/28' 2023-10-30T13:19:54.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/24' 2023-10-30T13:19:54.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/29' 2023-10-30T13:19:54.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/32' 2023-10-30T13:19:54.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/22' 2023-10-30T13:19:54.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/25' 2023-10-30T13:19:54.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/30' 2023-10-30T13:19:54.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/33' 2023-10-30T13:19:54.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/23' 2023-10-30T13:19:54.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/35' 2023-10-30T13:19:54.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/26' 2023-10-30T13:19:54.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './237/37' 2023-10-30T13:19:54.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './237' 2023-10-30T13:19:54.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/21' 2023-10-30T13:19:54.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/27' 2023-10-30T13:19:54.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/31' 2023-10-30T13:19:54.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/36' 2023-10-30T13:19:54.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/20' 2023-10-30T13:19:54.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/39' 2023-10-30T13:19:54.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/34' 2023-10-30T13:19:54.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/38' 2023-10-30T13:19:54.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/28' 2023-10-30T13:19:54.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/24' 2023-10-30T13:19:54.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/29' 2023-10-30T13:19:54.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/32' 2023-10-30T13:19:54.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/22' 2023-10-30T13:19:54.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/25' 2023-10-30T13:19:54.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/30' 2023-10-30T13:19:54.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/33' 2023-10-30T13:19:54.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/23' 2023-10-30T13:19:54.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/35' 2023-10-30T13:19:54.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/26' 2023-10-30T13:19:54.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './238/37' 2023-10-30T13:19:54.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './238' 2023-10-30T13:19:54.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/21' 2023-10-30T13:19:54.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/27' 2023-10-30T13:19:54.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/31' 2023-10-30T13:19:54.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/36' 2023-10-30T13:19:54.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/20' 2023-10-30T13:19:54.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/39' 2023-10-30T13:19:54.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/34' 2023-10-30T13:19:54.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/38' 2023-10-30T13:19:54.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/28' 2023-10-30T13:19:54.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/24' 2023-10-30T13:19:54.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/29' 2023-10-30T13:19:54.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/32' 2023-10-30T13:19:54.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/22' 2023-10-30T13:19:54.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/25' 2023-10-30T13:19:54.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/30' 2023-10-30T13:19:54.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/33' 2023-10-30T13:19:54.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/23' 2023-10-30T13:19:54.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/35' 2023-10-30T13:19:54.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/26' 2023-10-30T13:19:54.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './239/37' 2023-10-30T13:19:54.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './239' 2023-10-30T13:19:54.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/21' 2023-10-30T13:19:54.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/27' 2023-10-30T13:19:54.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/31' 2023-10-30T13:19:54.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/36' 2023-10-30T13:19:54.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/20' 2023-10-30T13:19:54.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/39' 2023-10-30T13:19:54.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/34' 2023-10-30T13:19:54.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/38' 2023-10-30T13:19:54.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/28' 2023-10-30T13:19:54.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/24' 2023-10-30T13:19:54.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/29' 2023-10-30T13:19:54.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/32' 2023-10-30T13:19:54.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/22' 2023-10-30T13:19:54.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/25' 2023-10-30T13:19:54.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/30' 2023-10-30T13:19:54.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/33' 2023-10-30T13:19:54.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/23' 2023-10-30T13:19:54.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/35' 2023-10-30T13:19:54.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/26' 2023-10-30T13:19:54.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './240/37' 2023-10-30T13:19:54.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './240' 2023-10-30T13:19:54.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/21' 2023-10-30T13:19:54.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/27' 2023-10-30T13:19:54.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/31' 2023-10-30T13:19:54.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/36' 2023-10-30T13:19:54.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/20' 2023-10-30T13:19:54.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/39' 2023-10-30T13:19:54.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/34' 2023-10-30T13:19:54.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/38' 2023-10-30T13:19:54.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/28' 2023-10-30T13:19:54.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/24' 2023-10-30T13:19:54.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/29' 2023-10-30T13:19:54.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/32' 2023-10-30T13:19:54.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/22' 2023-10-30T13:19:54.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/25' 2023-10-30T13:19:54.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/30' 2023-10-30T13:19:54.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/33' 2023-10-30T13:19:54.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/23' 2023-10-30T13:19:54.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/35' 2023-10-30T13:19:54.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/26' 2023-10-30T13:19:54.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './241/37' 2023-10-30T13:19:54.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './241' 2023-10-30T13:19:54.278 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/21' 2023-10-30T13:19:54.278 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/27' 2023-10-30T13:19:54.278 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/31' 2023-10-30T13:19:54.278 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/36' 2023-10-30T13:19:54.278 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/20' 2023-10-30T13:19:54.278 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/39' 2023-10-30T13:19:54.279 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/34' 2023-10-30T13:19:54.279 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/38' 2023-10-30T13:19:54.279 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/28' 2023-10-30T13:19:54.279 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/24' 2023-10-30T13:19:54.279 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/29' 2023-10-30T13:19:54.280 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/32' 2023-10-30T13:19:54.280 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/22' 2023-10-30T13:19:54.280 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/25' 2023-10-30T13:19:54.280 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/30' 2023-10-30T13:19:54.280 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/33' 2023-10-30T13:19:54.280 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/23' 2023-10-30T13:19:54.281 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/35' 2023-10-30T13:19:54.281 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/26' 2023-10-30T13:19:54.281 INFO:tasks.workunit.client.0.smithi148.stdout:removed './242/37' 2023-10-30T13:19:54.281 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './242' 2023-10-30T13:19:54.281 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/21' 2023-10-30T13:19:54.282 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/27' 2023-10-30T13:19:54.282 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/31' 2023-10-30T13:19:54.282 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/36' 2023-10-30T13:19:54.282 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/20' 2023-10-30T13:19:54.282 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/39' 2023-10-30T13:19:54.282 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/34' 2023-10-30T13:19:54.283 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/38' 2023-10-30T13:19:54.283 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/28' 2023-10-30T13:19:54.283 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/24' 2023-10-30T13:19:54.283 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/29' 2023-10-30T13:19:54.283 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/32' 2023-10-30T13:19:54.284 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/22' 2023-10-30T13:19:54.284 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/25' 2023-10-30T13:19:54.284 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/30' 2023-10-30T13:19:54.284 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/33' 2023-10-30T13:19:54.284 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/23' 2023-10-30T13:19:54.284 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/35' 2023-10-30T13:19:54.285 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/26' 2023-10-30T13:19:54.285 INFO:tasks.workunit.client.0.smithi148.stdout:removed './243/37' 2023-10-30T13:19:54.285 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './243' 2023-10-30T13:19:54.285 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/21' 2023-10-30T13:19:54.285 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/27' 2023-10-30T13:19:54.286 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/31' 2023-10-30T13:19:54.286 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/36' 2023-10-30T13:19:54.286 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/20' 2023-10-30T13:19:54.286 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/39' 2023-10-30T13:19:54.286 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/34' 2023-10-30T13:19:54.286 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/38' 2023-10-30T13:19:54.287 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/28' 2023-10-30T13:19:54.287 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/24' 2023-10-30T13:19:54.287 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/29' 2023-10-30T13:19:54.287 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/32' 2023-10-30T13:19:54.287 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/22' 2023-10-30T13:19:54.288 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/25' 2023-10-30T13:19:54.288 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/30' 2023-10-30T13:19:54.288 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/33' 2023-10-30T13:19:54.288 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/23' 2023-10-30T13:19:54.288 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/35' 2023-10-30T13:19:54.288 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/26' 2023-10-30T13:19:54.289 INFO:tasks.workunit.client.0.smithi148.stdout:removed './244/37' 2023-10-30T13:19:54.289 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './244' 2023-10-30T13:19:54.289 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/21' 2023-10-30T13:19:54.289 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/27' 2023-10-30T13:19:54.289 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/31' 2023-10-30T13:19:54.290 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/36' 2023-10-30T13:19:54.290 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/20' 2023-10-30T13:19:54.290 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/39' 2023-10-30T13:19:54.290 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/34' 2023-10-30T13:19:54.290 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/38' 2023-10-30T13:19:54.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/28' 2023-10-30T13:19:54.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/24' 2023-10-30T13:19:54.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/29' 2023-10-30T13:19:54.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/32' 2023-10-30T13:19:54.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/22' 2023-10-30T13:19:54.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/25' 2023-10-30T13:19:54.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/30' 2023-10-30T13:19:54.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/33' 2023-10-30T13:19:54.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/23' 2023-10-30T13:19:54.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/35' 2023-10-30T13:19:54.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/26' 2023-10-30T13:19:54.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './245/37' 2023-10-30T13:19:54.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './245' 2023-10-30T13:19:54.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/21' 2023-10-30T13:19:54.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/27' 2023-10-30T13:19:54.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/31' 2023-10-30T13:19:54.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/36' 2023-10-30T13:19:54.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/20' 2023-10-30T13:19:54.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/39' 2023-10-30T13:19:54.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/34' 2023-10-30T13:19:54.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/38' 2023-10-30T13:19:54.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/28' 2023-10-30T13:19:54.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/24' 2023-10-30T13:19:54.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/29' 2023-10-30T13:19:54.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/32' 2023-10-30T13:19:54.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/22' 2023-10-30T13:19:54.480 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:19:54.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa5524d6640 1 Processor -- start 2023-10-30T13:19:54.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa5524d6640 1 -- start start 2023-10-30T13:19:54.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa5524d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa54c15d6f0 0x7fa54c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:54.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa5524d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa54c0cf160 0x7fa54c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:54.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa5524d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c0cb800 con 0x7fa54c1538a0 2023-10-30T13:19:54.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa5524d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c0cb330 con 0x7fa54c15d6f0 2023-10-30T13:19:54.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa5524d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c0cad70 con 0x7fa54c0cf160 2023-10-30T13:19:54.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa543fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa54c15d6f0 0x7fa54c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:54.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa543fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa54c15d6f0 0x7fa54c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42274/0 (socket says 172.21.15.148:42274) 2023-10-30T13:19:54.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa543fff640 1 -- 172.21.15.148:0/4072095066 learned_addr learned my addr 172.21.15.148:0/4072095066 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:54.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa550a73640 1 --2- 172.21.15.148:0/4072095066 >> v2:172.21.15.150:3300/0 conn(0x7fa54c0cf160 0x7fa54c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:54.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa543fff640 1 -- 172.21.15.148:0/4072095066 >> v2:172.21.15.150:3300/0 conn(0x7fa54c0cf160 msgr2=0x7fa54c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:54.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa543fff640 1 --2- 172.21.15.148:0/4072095066 >> v2:172.21.15.150:3300/0 conn(0x7fa54c0cf160 0x7fa54c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa543fff640 1 -- 172.21.15.148:0/4072095066 >> v1:172.21.15.150:6789/0 conn(0x7fa54c1538a0 legacy=0x7fa54c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:54.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.619+0000 7fa543fff640 1 -- 172.21.15.148:0/4072095066 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c0cb050 con 0x7fa54c15d6f0 2023-10-30T13:19:54.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa543fff640 1 --2- 172.21.15.148:0/4072095066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c15d6f0 0x7fa54c15a090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0b5330 tx=0x7fa53401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5dfba3d6f15ec2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:54.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa54b7fe640 1 -- 172.21.15.148:0/4072095066 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340a89f0 con 0x7fa54c15d6f0 2023-10-30T13:19:54.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa54b7fe640 1 -- 172.21.15.148:0/4072095066 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5340a8b50 con 0x7fa54c15d6f0 2023-10-30T13:19:54.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa54b7fe640 1 -- 172.21.15.148:0/4072095066 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340b0440 con 0x7fa54c15d6f0 2023-10-30T13:19:54.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa5524d6640 1 -- 172.21.15.148:0/4072095066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c15d6f0 msgr2=0x7fa54c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:54.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa5524d6640 1 --2- 172.21.15.148:0/4072095066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c15d6f0 0x7fa54c15a090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0b5330 tx=0x7fa53401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa5524d6640 1 -- 172.21.15.148:0/4072095066 shutdown_connections 2023-10-30T13:19:54.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa5524d6640 1 --2- 172.21.15.148:0/4072095066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c15d6f0 0x7fa54c15a090 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa5524d6640 1 --2- 172.21.15.148:0/4072095066 >> v2:172.21.15.150:3300/0 conn(0x7fa54c0cf160 0x7fa54c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa5524d6640 1 -- 172.21.15.148:0/4072095066 >> 172.21.15.148:0/4072095066 conn(0x7fa54c06bfe0 msgr2=0x7fa54c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:54.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa5524d6640 1 -- 172.21.15.148:0/4072095066 shutdown_connections 2023-10-30T13:19:54.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.623+0000 7fa5524d6640 1 -- 172.21.15.148:0/4072095066 wait complete. 2023-10-30T13:19:54.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 Processor -- start 2023-10-30T13:19:54.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 -- start start 2023-10-30T13:19:54.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa54c0cf160 0x7fa54c158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:54.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c1538a0 0x7fa54c0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:54.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa54c15d6f0 0x7fa54c0c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:54.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c158d60 con 0x7fa54c1538a0 2023-10-30T13:19:54.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c1e5280 con 0x7fa54c15d6f0 2023-10-30T13:19:54.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0c9650 con 0x7fa54c0cf160 2023-10-30T13:19:54.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa543fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c1538a0 0x7fa54c0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:54.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa543fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c1538a0 0x7fa54c0b4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42286/0 (socket says 172.21.15.148:42286) 2023-10-30T13:19:54.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa543fff640 1 -- 172.21.15.148:0/567042939 learned_addr learned my addr 172.21.15.148:0/567042939 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:54.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa54bfff640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa54c0cf160 0x7fa54c158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:54.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa550a73640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa54c15d6f0 0x7fa54c0c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:54.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa543fff640 1 -- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa54c0cf160 msgr2=0x7fa54c158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:54.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa543fff640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa54c0cf160 0x7fa54c158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa543fff640 1 -- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa54c15d6f0 msgr2=0x7fa54c0c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:54.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa543fff640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa54c15d6f0 0x7fa54c0c71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa543fff640 1 -- 172.21.15.148:0/567042939 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5340094d0 con 0x7fa54c1538a0 2023-10-30T13:19:54.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa543fff640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c1538a0 0x7fa54c0b4960 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa53401b900 tx=0x7fa53401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:54.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5497fa640 1 -- 172.21.15.148:0/567042939 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340ba600 con 0x7fa54c1538a0 2023-10-30T13:19:54.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5497fa640 1 -- 172.21.15.148:0/567042939 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5340ba760 con 0x7fa54c1538a0 2023-10-30T13:19:54.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5497fa640 1 -- 172.21.15.148:0/567042939 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340bab10 con 0x7fa54c1538a0 2023-10-30T13:19:54.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c0c6ec0 con 0x7fa54c1538a0 2023-10-30T13:19:54.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.627+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c159050 con 0x7fa54c1538a0 2023-10-30T13:19:54.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.631+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54c0b2a50 con 0x7fa54c1538a0 2023-10-30T13:19:54.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.631+0000 7fa5497fa640 1 -- 172.21.15.148:0/567042939 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa5340bac70 con 0x7fa54c1538a0 2023-10-30T13:19:54.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.631+0000 7fa5497fa640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa52c0664a0 0x7fa52c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:54.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.631+0000 7fa5497fa640 1 -- 172.21.15.148:0/567042939 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fa534122670 con 0x7fa54c1538a0 2023-10-30T13:19:54.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.631+0000 7fa54bfff640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa52c0664a0 0x7fa52c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:54.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.631+0000 7fa54bfff640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa52c0664a0 0x7fa52c068960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa53c000f70 tx=0x7fa53c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:54.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.635+0000 7fa5497fa640 1 -- 172.21.15.148:0/567042939 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa5340ef850 con 0x7fa54c1538a0 2023-10-30T13:19:54.663 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:19:54.667 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:19:54.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.819+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa54c04e170 con 0x7fa54c1538a0 2023-10-30T13:19:54.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5497fa640 1 -- 172.21.15.148:0/567042939 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fa5340b5090 con 0x7fa54c1538a0 2023-10-30T13:19:54.823 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:19:54.823 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:19:51.418233+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:19:54.823 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 44 2023-10-30T13:19:54.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa52c0664a0 msgr2=0x7fa52c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:54.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa52c0664a0 0x7fa52c068960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa53c000f70 tx=0x7fa53c009040 comp rx=0 tx=0).stop 2023-10-30T13:19:54.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c1538a0 msgr2=0x7fa54c0b4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:54.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c1538a0 0x7fa54c0b4960 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa53401b900 tx=0x7fa53401ba90 comp rx=0 tx=0).stop 2023-10-30T13:19:54.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 shutdown_connections 2023-10-30T13:19:54.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa52c0664a0 0x7fa52c068960 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa54c1538a0 0x7fa54c0b4960 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa54c15d6f0 0x7fa54c0c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 --2- 172.21.15.148:0/567042939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa54c0cf160 0x7fa54c158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 >> 172.21.15.148:0/567042939 conn(0x7fa54c06bfe0 msgr2=0x7fa54c1488e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:54.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 shutdown_connections 2023-10-30T13:19:54.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.823+0000 7fa5524d6640 1 -- 172.21.15.148:0/567042939 wait complete. 2023-10-30T13:19:54.843 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 3 -> 1 2023-10-30T13:19:54.843 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2023-10-30T13:19:54.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.991+0000 7efef9162640 1 Processor -- start 2023-10-30T13:19:54.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.991+0000 7efef9162640 1 -- start start 2023-10-30T13:19:54.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef9162640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efef4153840 0x7efef415a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:54.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef9162640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef415a6b0 con 0x7efef40cb4c0 2023-10-30T13:19:54.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef9162640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef415d730 con 0x7efef4153c20 2023-10-30T13:19:54.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef9162640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef415dbc0 con 0x7efef4153840 2023-10-30T13:19:54.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef8961640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efef40cb4c0 0x7efef40cef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37932/0 (socket says 172.21.15.148:37932) 2023-10-30T13:19:54.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef8961640 1 -- 172.21.15.148:0/1912204084 learned_addr learned my addr 172.21.15.148:0/1912204084 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:19:54.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef3fff640 1 --2- 172.21.15.148:0/1912204084 >> v2:172.21.15.150:3300/0 conn(0x7efef4153840 0x7efef415a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:54.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091820288 0 0) 0x7efeec0014a0 con 0x7efef40cb4c0 2023-10-30T13:19:54.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efedc003530 con 0x7efef40cb4c0 2023-10-30T13:19:54.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 583020186 0 0) 0x7efedc003530 con 0x7efef40cb4c0 2023-10-30T13:19:54.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 >> v2:172.21.15.150:3300/0 conn(0x7efef4153840 msgr2=0x7efef415a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:54.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 --2- 172.21.15.148:0/1912204084 >> v2:172.21.15.150:3300/0 conn(0x7efef4153840 0x7efef415a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 >> v1:172.21.15.150:6789/0 conn(0x7efef4153c20 legacy=0x7efef40c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef40ca8f0 con 0x7efef40cb4c0 2023-10-30T13:19:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7efeec001b90 con 0x7efef40cb4c0 2023-10-30T13:19:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 >> v1:172.21.15.148:6789/0 conn(0x7efef40cb4c0 legacy=0x7efef40cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc003980 0x7efedc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc0047d0 0x7efedc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efedc005190 0x7efedc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef415dbc0 con 0x7efedc005190 2023-10-30T13:19:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc00c040 con 0x7efedc003980 2023-10-30T13:19:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efedc00c200 con 0x7efedc0047d0 2023-10-30T13:19:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef8961640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efedc005190 0x7efedc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef3fff640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc003980 0x7efedc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef37fe640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc0047d0 0x7efedc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef8961640 1 -- 172.21.15.148:0/1912204084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc0047d0 msgr2=0x7efedc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef8961640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc0047d0 0x7efedc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef8961640 1 -- 172.21.15.148:0/1912204084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc003980 msgr2=0x7efedc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:55.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef8961640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc003980 0x7efedc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.995+0000 7efef8961640 1 -- 172.21.15.148:0/1912204084 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efedc003530 con 0x7efedc005190 2023-10-30T13:19:55.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef8961640 1 -- 172.21.15.148:0/1912204084 reap_dead start 2023-10-30T13:19:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef8961640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efedc005190 0x7efedc00b990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efeec00a830 tx=0x7efeec01d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27007f8687675baf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec0ad630 con 0x7efedc005190 2023-10-30T13:19:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec0a8070 con 0x7efedc005190 2023-10-30T13:19:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef2ffd640 1 -- 172.21.15.148:0/1912204084 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec0ad9f0 con 0x7efedc005190 2023-10-30T13:19:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef9162640 1 -- 172.21.15.148:0/1912204084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efedc005190 msgr2=0x7efedc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef9162640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efedc005190 0x7efedc00b990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efeec00a830 tx=0x7efeec01d340 comp rx=0 tx=0).stop 2023-10-30T13:19:55.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef9162640 1 -- 172.21.15.148:0/1912204084 shutdown_connections 2023-10-30T13:19:55.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef9162640 1 --2- 172.21.15.148:0/1912204084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efedc005190 0x7efedc00b990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef9162640 1 -- 172.21.15.148:0/1912204084 >> 172.21.15.148:0/1912204084 conn(0x7efef406be40 msgr2=0x7efef406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:55.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef9162640 1 -- 172.21.15.148:0/1912204084 shutdown_connections 2023-10-30T13:19:55.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef9162640 1 -- 172.21.15.148:0/1912204084 wait complete. 2023-10-30T13:19:55.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:54.999+0000 7efef9162640 1 Processor -- start 2023-10-30T13:19:55.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 -- start start 2023-10-30T13:19:55.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc005190 0x7efef41ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efef4154060 0x7efef415c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efef41f2230 0x7efef41f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef41ef630 con 0x7efef41f2230 2023-10-30T13:19:55.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef4159d20 con 0x7efef4154060 2023-10-30T13:19:55.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef4154510 con 0x7efedc005190 2023-10-30T13:19:55.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef8961640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efef41f2230 0x7efef41f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef8961640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efef41f2230 0x7efef41f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42290/0 (socket says 172.21.15.148:42290) 2023-10-30T13:19:55.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef8961640 1 -- 172.21.15.148:0/346740782 learned_addr learned my addr 172.21.15.148:0/346740782 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:55.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef3fff640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc005190 0x7efef41ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef37fe640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efef4154060 0x7efef415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef8961640 1 -- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc005190 msgr2=0x7efef41ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef8961640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc005190 0x7efef41ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef8961640 1 -- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efef4154060 msgr2=0x7efef415c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef8961640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efef4154060 0x7efef415c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef8961640 1 -- 172.21.15.148:0/346740782 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef4159760 con 0x7efef41f2230 2023-10-30T13:19:55.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef3fff640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc005190 0x7efef41ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:19:55.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef8961640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efef41f2230 0x7efef41f26b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efeec01dbe0 tx=0x7efedc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:55.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef0ff9640 1 -- 172.21.15.148:0/346740782 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec0ad630 con 0x7efef41f2230 2023-10-30T13:19:55.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef0ff9640 1 -- 172.21.15.148:0/346740782 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec0ad790 con 0x7efef41f2230 2023-10-30T13:19:55.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef0ff9640 1 -- 172.21.15.148:0/346740782 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec0adb00 con 0x7efef41f2230 2023-10-30T13:19:55.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef4159a40 con 0x7efef41f2230 2023-10-30T13:19:55.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef414e6d0 con 0x7efef41f2230 2023-10-30T13:19:55.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.003+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef404e9c0 con 0x7efef41f2230 2023-10-30T13:19:55.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.007+0000 7efef0ff9640 1 -- 172.21.15.148:0/346740782 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efeec0aa030 con 0x7efef41f2230 2023-10-30T13:19:55.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.007+0000 7efef0ff9640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efee00664d0 0x7efee0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.007+0000 7efef0ff9640 1 -- 172.21.15.148:0/346740782 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7efeec11df40 con 0x7efef41f2230 2023-10-30T13:19:55.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.007+0000 7efef3fff640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efee00664d0 0x7efee0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.007+0000 7efef3fff640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efee00664d0 0x7efee0068990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efee4009b60 tx=0x7efee4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:55.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.011+0000 7efef0ff9640 1 -- 172.21.15.148:0/346740782 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efeec0eb0a0 con 0x7efef41f2230 2023-10-30T13:19:55.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.195+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7efef415a6b0 con 0x7efef41f2230 2023-10-30T13:19:55.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.607+0000 7efef0ff9640 1 -- 172.21.15.148:0/346740782 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v45) v1 ==== 105+0+0 (secure 0 0 0) 0x7efef415a6b0 con 0x7efef41f2230 2023-10-30T13:19:55.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efee00664d0 msgr2=0x7efee0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efee00664d0 0x7efee0068990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efee4009b60 tx=0x7efee4008040 comp rx=0 tx=0).stop 2023-10-30T13:19:55.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efef41f2230 msgr2=0x7efef41f26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efef41f2230 0x7efef41f26b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efeec01dbe0 tx=0x7efedc005800 comp rx=0 tx=0).stop 2023-10-30T13:19:55.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 shutdown_connections 2023-10-30T13:19:55.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efee00664d0 0x7efee0068990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efef41f2230 0x7efef41f26b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efef4154060 0x7efef415c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 --2- 172.21.15.148:0/346740782 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efedc005190 0x7efef41ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 >> 172.21.15.148:0/346740782 conn(0x7efef406be40 msgr2=0x7efef41524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:55.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 shutdown_connections 2023-10-30T13:19:55.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.611+0000 7efef9162640 1 -- 172.21.15.148:0/346740782 wait complete. 2023-10-30T13:19:55.634 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:19:55.634 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:19:55.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.771+0000 7fd2f8eea640 1 Processor -- start 2023-10-30T13:19:55.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.771+0000 7fd2f8eea640 1 -- start start 2023-10-30T13:19:55.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f8eea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2f40c6b80 0x7fd2f40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f8eea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f413f4a0 0x7fd2f40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f8eea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f40c0a50 con 0x7fd2f40c6f60 2023-10-30T13:19:55.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f8eea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f406f6e0 con 0x7fd2f40c6b80 2023-10-30T13:19:55.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f8eea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f406f9c0 con 0x7fd2f413f4a0 2023-10-30T13:19:55.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f2d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f413f4a0 0x7fd2f40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2f40c6b80 0x7fd2f40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2f40c6b80 0x7fd2f40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:36110/0 (socket says 172.21.15.148:36110) 2023-10-30T13:19:55.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f259c640 1 -- 172.21.15.148:0/798170569 learned_addr learned my addr 172.21.15.148:0/798170569 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:55.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2e9d9b640 1 --1- 172.21.15.148:0/798170569 >> v1:172.21.15.150:6789/0 conn(0x7fd2f40c6f60 0x7fd2f413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:47588/0 (socket says 172.21.15.148:47588) 2023-10-30T13:19:55.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f2d9d640 1 -- 172.21.15.148:0/798170569 >> v2:172.21.15.150:3300/0 conn(0x7fd2f40c6b80 msgr2=0x7fd2f40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:55.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f2d9d640 1 --2- 172.21.15.148:0/798170569 >> v2:172.21.15.150:3300/0 conn(0x7fd2f40c6b80 0x7fd2f40bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f2d9d640 1 -- 172.21.15.148:0/798170569 >> v1:172.21.15.150:6789/0 conn(0x7fd2f40c6f60 legacy=0x7fd2f413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f2d9d640 1 -- 172.21.15.148:0/798170569 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f406fca0 con 0x7fd2f413f4a0 2023-10-30T13:19:55.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.775+0000 7fd2f259c640 1 --2- 172.21.15.148:0/798170569 >> v2:172.21.15.150:3300/0 conn(0x7fd2f40c6b80 0x7fd2f40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:19:55.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f2d9d640 1 --2- 172.21.15.148:0/798170569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f413f4a0 0x7fd2f40c0400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009d90 tx=0x7fd2ec01c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f17168a6d3597f12 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:55.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f1d9b640 1 -- 172.21.15.148:0/798170569 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec0a5920 con 0x7fd2f413f4a0 2023-10-30T13:19:55.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f1d9b640 1 -- 172.21.15.148:0/798170569 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ec0a5a80 con 0x7fd2f413f4a0 2023-10-30T13:19:55.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f1d9b640 1 -- 172.21.15.148:0/798170569 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec0a5e10 con 0x7fd2f413f4a0 2023-10-30T13:19:55.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 -- 172.21.15.148:0/798170569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f413f4a0 msgr2=0x7fd2f40c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 --2- 172.21.15.148:0/798170569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f413f4a0 0x7fd2f40c0400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009d90 tx=0x7fd2ec01c950 comp rx=0 tx=0).stop 2023-10-30T13:19:55.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 -- 172.21.15.148:0/798170569 shutdown_connections 2023-10-30T13:19:55.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 --2- 172.21.15.148:0/798170569 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f413f4a0 0x7fd2f40c0400 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 --2- 172.21.15.148:0/798170569 >> v2:172.21.15.150:3300/0 conn(0x7fd2f40c6b80 0x7fd2f40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 -- 172.21.15.148:0/798170569 >> 172.21.15.148:0/798170569 conn(0x7fd2f40c7910 msgr2=0x7fd2f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:55.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 -- 172.21.15.148:0/798170569 shutdown_connections 2023-10-30T13:19:55.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 -- 172.21.15.148:0/798170569 wait complete. 2023-10-30T13:19:55.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 Processor -- start 2023-10-30T13:19:55.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.779+0000 7fd2f8eea640 1 -- start start 2023-10-30T13:19:55.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f8eea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f40c6b80 0x7fd2f41f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f8eea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f40c6f60 0x7fd2f4154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f8eea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f413f4a0 0x7fd2f413dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f8eea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f40c9400 con 0x7fd2f40c6b80 2023-10-30T13:19:55.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f8eea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f40c8c10 con 0x7fd2f413f4a0 2023-10-30T13:19:55.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f8eea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f41f2e60 con 0x7fd2f40c6f60 2023-10-30T13:19:55.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f40c6b80 0x7fd2f41f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f40c6b80 0x7fd2f41f2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42296/0 (socket says 172.21.15.148:42296) 2023-10-30T13:19:55.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f259c640 1 -- 172.21.15.148:0/2529330711 learned_addr learned my addr 172.21.15.148:0/2529330711 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:55.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2e9d9b640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f40c6f60 0x7fd2f4154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f2d9d640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f413f4a0 0x7fd2f413dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f259c640 1 -- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f40c6f60 msgr2=0x7fd2f4154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f259c640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f40c6f60 0x7fd2f4154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f259c640 1 -- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f413f4a0 msgr2=0x7fd2f413dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f259c640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f413f4a0 0x7fd2f413dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f259c640 1 -- 172.21.15.148:0/2529330711 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc0094d0 con 0x7fd2f40c6b80 2023-10-30T13:19:55.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f259c640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f40c6b80 0x7fd2f41f2920 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00e9d0 tx=0x7fd2dc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:55.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2eb7fe640 1 -- 172.21.15.148:0/2529330711 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc00ca60 con 0x7fd2f40c6b80 2023-10-30T13:19:55.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2eb7fe640 1 -- 172.21.15.148:0/2529330711 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc00cbc0 con 0x7fd2f40c6b80 2023-10-30T13:19:55.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2eb7fe640 1 -- 172.21.15.148:0/2529330711 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc013470 con 0x7fd2f40c6b80 2023-10-30T13:19:55.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec009a70 con 0x7fd2f40c6b80 2023-10-30T13:19:55.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f4154890 con 0x7fd2f40c6b80 2023-10-30T13:19:55.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.783+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f404e170 con 0x7fd2f40c6b80 2023-10-30T13:19:55.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.787+0000 7fd2eb7fe640 1 -- 172.21.15.148:0/2529330711 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2dc0135d0 con 0x7fd2f40c6b80 2023-10-30T13:19:55.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.787+0000 7fd2eb7fe640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664a0 0x7fd2d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:55.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.787+0000 7fd2eb7fe640 1 -- 172.21.15.148:0/2529330711 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd2dc084700 con 0x7fd2f40c6b80 2023-10-30T13:19:55.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.787+0000 7fd2e9d9b640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664a0 0x7fd2d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:55.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.787+0000 7fd2e9d9b640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664a0 0x7fd2d4068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd2f40c6720 tx=0x7fd2e4007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:55.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.791+0000 7fd2eb7fe640 1 -- 172.21.15.148:0/2529330711 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2dc0519a0 con 0x7fd2f40c6b80 2023-10-30T13:19:55.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2f406baa0 con 0x7fd2f40c6b80 2023-10-30T13:19:55.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2eb7fe640 1 -- 172.21.15.148:0/2529330711 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+6539 (secure 0 0 0) 0x7fd2dc054fc0 con 0x7fd2f40c6b80 2023-10-30T13:19:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:19:55.976 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:19:55.615821+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:19:55.976 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 46 2023-10-30T13:19:55.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664a0 msgr2=0x7fd2d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664a0 0x7fd2d4068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd2f40c6720 tx=0x7fd2e4007e90 comp rx=0 tx=0).stop 2023-10-30T13:19:55.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f40c6b80 msgr2=0x7fd2f41f2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:55.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f40c6b80 0x7fd2f41f2920 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00e9d0 tx=0x7fd2dc00ee90 comp rx=0 tx=0).stop 2023-10-30T13:19:55.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 shutdown_connections 2023-10-30T13:19:55.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2d40664a0 0x7fd2d4068960 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f413f4a0 0x7fd2f413dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2f40c6f60 0x7fd2f4154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 --2- 172.21.15.148:0/2529330711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f40c6b80 0x7fd2f41f2920 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:55.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 >> 172.21.15.148:0/2529330711 conn(0x7fd2f40c7910 msgr2=0x7fd2f40c14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:55.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 shutdown_connections 2023-10-30T13:19:55.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:55.975+0000 7fd2f8eea640 1 -- 172.21.15.148:0/2529330711 wait complete. 2023-10-30T13:19:55.997 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:19:55.998 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 46, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4959": { "addr": "172.21.15.148:6837/95380877", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 95380877, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 95380877, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4959, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:stopping", "state_seq": 16 }, "gid_5033": { "addr": "172.21.15.150:6836/953984429", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 953984429, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 953984429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5033, "incarnation": 36, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:19:55.615821+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5033, "mds_1": 5103, "mds_2": 4959 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:19:57.999 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:19:58.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.143+0000 7f84efd0f640 1 Processor -- start 2023-10-30T13:19:58.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84efd0f640 1 -- start start 2023-10-30T13:19:58.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84efd0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f84e815d6f0 0x7f84e815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:58.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84efd0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e80cf160 0x7f84e80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:58.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84efd0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e80cb800 con 0x7f84e81538a0 2023-10-30T13:19:58.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84efd0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e80cb330 con 0x7f84e815d6f0 2023-10-30T13:19:58.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84efd0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80cad70 con 0x7f84e80cf160 2023-10-30T13:19:58.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84ed2aa640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f84e815d6f0 0x7f84e815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:58.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84ed2aa640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f84e815d6f0 0x7f84e815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42298/0 (socket says 172.21.15.148:42298) 2023-10-30T13:19:58.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84ed2aa640 1 -- 172.21.15.148:0/3167643107 learned_addr learned my addr 172.21.15.148:0/3167643107 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:58.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84ee2ac640 1 --2- 172.21.15.148:0/3167643107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e80cf160 0x7f84e80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:58.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84ed2aa640 1 -- 172.21.15.148:0/3167643107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e80cf160 msgr2=0x7f84e80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:19:58.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84ed2aa640 1 --2- 172.21.15.148:0/3167643107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e80cf160 0x7f84e80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:58.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84ed2aa640 1 -- 172.21.15.148:0/3167643107 >> v1:172.21.15.150:6789/0 conn(0x7f84e81538a0 legacy=0x7f84e80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:58.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.147+0000 7f84ed2aa640 1 -- 172.21.15.148:0/3167643107 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e80cb050 con 0x7f84e815d6f0 2023-10-30T13:19:58.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84ed2aa640 1 --2- 172.21.15.148:0/3167643107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 0x7f84e815a090 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009870 tx=0x7f84d801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878e3d31f8709d0b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:58.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84ecaa9640 1 -- 172.21.15.148:0/3167643107 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d80a8a30 con 0x7f84e815d6f0 2023-10-30T13:19:58.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84ecaa9640 1 -- 172.21.15.148:0/3167643107 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d80a8b90 con 0x7f84e815d6f0 2023-10-30T13:19:58.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84ecaa9640 1 -- 172.21.15.148:0/3167643107 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d80b0440 con 0x7f84e815d6f0 2023-10-30T13:19:58.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 -- 172.21.15.148:0/3167643107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 msgr2=0x7f84e815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:58.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 --2- 172.21.15.148:0/3167643107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 0x7f84e815a090 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009870 tx=0x7f84d801ed30 comp rx=0 tx=0).stop 2023-10-30T13:19:58.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 -- 172.21.15.148:0/3167643107 shutdown_connections 2023-10-30T13:19:58.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 --2- 172.21.15.148:0/3167643107 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e80cf160 0x7f84e80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:58.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 --2- 172.21.15.148:0/3167643107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 0x7f84e815a090 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:58.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 -- 172.21.15.148:0/3167643107 >> 172.21.15.148:0/3167643107 conn(0x7f84e806bfe0 msgr2=0x7f84e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:58.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 -- 172.21.15.148:0/3167643107 shutdown_connections 2023-10-30T13:19:58.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 -- 172.21.15.148:0/3167643107 wait complete. 2023-10-30T13:19:58.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 Processor -- start 2023-10-30T13:19:58.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.151+0000 7f84efd0f640 1 -- start start 2023-10-30T13:19:58.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84efd0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84e80cf160 0x7f84e8159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:58.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84efd0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e81538a0 0x7f84e80c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:58.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84efd0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 0x7f84e81e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:58.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84efd0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e81e3bf0 con 0x7f84e815d6f0 2023-10-30T13:19:58.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84efd0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e81580f0 con 0x7f84e80cf160 2023-10-30T13:19:58.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84efd0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80c59a0 con 0x7f84e81538a0 2023-10-30T13:19:58.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ee2ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 0x7f84e81e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:58.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ee2ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 0x7f84e81e2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42310/0 (socket says 172.21.15.148:42310) 2023-10-30T13:19:58.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ee2ac640 1 -- 172.21.15.148:0/2646949246 learned_addr learned my addr 172.21.15.148:0/2646949246 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:19:58.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84edaab640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84e80cf160 0x7f84e8159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:58.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ed2aa640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e81538a0 0x7f84e80c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:58.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ee2ac640 1 -- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e81538a0 msgr2=0x7f84e80c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:58.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ee2ac640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e81538a0 0x7f84e80c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:58.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ee2ac640 1 -- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84e80cf160 msgr2=0x7f84e8159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:58.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ee2ac640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84e80cf160 0x7f84e8159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:58.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ee2ac640 1 -- 172.21.15.148:0/2646949246 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80094d0 con 0x7f84e815d6f0 2023-10-30T13:19:58.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84ee2ac640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 0x7f84e81e2a40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f84e000efc0 tx=0x7f84e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:58.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84de7fc640 1 -- 172.21.15.148:0/2646949246 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e0009b10 con 0x7f84e815d6f0 2023-10-30T13:19:58.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84de7fc640 1 -- 172.21.15.148:0/2646949246 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e000f040 con 0x7f84e815d6f0 2023-10-30T13:19:58.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84de7fc640 1 -- 172.21.15.148:0/2646949246 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00146c0 con 0x7f84e815d6f0 2023-10-30T13:19:58.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e81584d0 con 0x7f84e815d6f0 2023-10-30T13:19:58.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e80c4f60 con 0x7f84e815d6f0 2023-10-30T13:19:58.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.155+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e80b2a50 con 0x7f84e815d6f0 2023-10-30T13:19:58.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.159+0000 7f84de7fc640 1 -- 172.21.15.148:0/2646949246 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84e0014820 con 0x7f84e815d6f0 2023-10-30T13:19:58.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.159+0000 7f84de7fc640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84cc0664d0 0x7f84cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:19:58.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.159+0000 7f84de7fc640 1 -- 172.21.15.148:0/2646949246 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f84e0084bc0 con 0x7f84e815d6f0 2023-10-30T13:19:58.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.159+0000 7f84edaab640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84cc0664d0 0x7f84cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:19:58.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.159+0000 7f84edaab640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84cc0664d0 0x7f84cc068990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f84e4000f70 tx=0x7f84e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:19:58.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.163+0000 7f84de7fc640 1 -- 172.21.15.148:0/2646949246 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f84e0051dc0 con 0x7f84e815d6f0 2023-10-30T13:19:58.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.339+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84e804e170 con 0x7f84e815d6f0 2023-10-30T13:19:58.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84de7fc640 1 -- 172.21.15.148:0/2646949246 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+6539 (secure 0 0 0) 0x7f84e0018070 con 0x7f84e815d6f0 2023-10-30T13:19:58.343 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:19:58.343 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:19:55.615821+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:19:58.343 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 46 2023-10-30T13:19:58.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84cc0664d0 msgr2=0x7f84cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:58.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84cc0664d0 0x7f84cc068990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f84e4000f70 tx=0x7f84e4009040 comp rx=0 tx=0).stop 2023-10-30T13:19:58.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 msgr2=0x7f84e81e2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:19:58.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 0x7f84e81e2a40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f84e000efc0 tx=0x7f84e000c490 comp rx=0 tx=0).stop 2023-10-30T13:19:58.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 shutdown_connections 2023-10-30T13:19:58.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84cc0664d0 0x7f84cc068990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:58.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84e815d6f0 0x7f84e81e2a40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:58.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84e81538a0 0x7f84e80c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:58.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 --2- 172.21.15.148:0/2646949246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84e80cf160 0x7f84e8159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:19:58.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 >> 172.21.15.148:0/2646949246 conn(0x7f84e806bfe0 msgr2=0x7f84e814ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:19:58.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 shutdown_connections 2023-10-30T13:19:58.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:19:58.343+0000 7f84efd0f640 1 -- 172.21.15.148:0/2646949246 wait complete. 2023-10-30T13:19:58.364 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:00.364 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:00.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.507+0000 7f8499817640 1 Processor -- start 2023-10-30T13:20:00.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.507+0000 7f8499817640 1 -- start start 2023-10-30T13:20:00.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.507+0000 7f8499817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c6b80 0x7f84940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:00.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.507+0000 7f8499817640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84940c0a50 con 0x7f849413f4a0 2023-10-30T13:20:00.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.507+0000 7f8499817640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849406f6e0 con 0x7f84940c6f60 2023-10-30T13:20:00.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.507+0000 7f8499817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849406f9c0 con 0x7f84940c6b80 2023-10-30T13:20:00.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.507+0000 7f84937fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f849413f4a0 0x7f84941566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37936/0 (socket says 172.21.15.148:37936) 2023-10-30T13:20:00.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.507+0000 7f84937fe640 1 -- 172.21.15.148:0/3156085837 learned_addr learned my addr 172.21.15.148:0/3156085837 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:00.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.507+0000 7f8492ffd640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c6b80 0x7f84940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:00.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235544552 0 0) 0x7f84940c0a50 con 0x7f849413f4a0 2023-10-30T13:20:00.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f847c005740 con 0x7f849413f4a0 2023-10-30T13:20:00.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298091526 0 0) 0x7f847c005740 con 0x7f849413f4a0 2023-10-30T13:20:00.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c6b80 msgr2=0x7f84940bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:00.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c6b80 0x7f84940bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 >> v1:172.21.15.150:6789/0 conn(0x7f84940c6f60 legacy=0x7f849413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:00.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849406fca0 con 0x7f849413f4a0 2023-10-30T13:20:00.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f848c001a00 con 0x7f849413f4a0 2023-10-30T13:20:00.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 >> v1:172.21.15.148:6789/0 conn(0x7f849413f4a0 legacy=0x7f84941566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:00.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c6b80 0x7f847c005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:00.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f847c0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:00.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f847c006f20 0x7f847c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:00.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849406f9c0 con 0x7f847c006140 2023-10-30T13:20:00.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c00dc80 con 0x7f847c006f20 2023-10-30T13:20:00.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c00de40 con 0x7f84940c6b80 2023-10-30T13:20:00.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f84927fc640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f847c0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:00.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f84937fe640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f847c006f20 0x7f847c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:00.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8492ffd640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c6b80 0x7f847c005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:00.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f84927fc640 1 -- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c6b80 msgr2=0x7f847c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:00.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f84927fc640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c6b80 0x7f847c005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f84927fc640 1 -- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f847c006f20 msgr2=0x7f847c0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:00.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f84927fc640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f847c006f20 0x7f847c0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f84927fc640 1 -- 172.21.15.148:0/3156085837 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c005740 con 0x7f847c006140 2023-10-30T13:20:00.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f84937fe640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f847c006f20 0x7f847c0073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:00.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.511+0000 7f8492ffd640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84940c6b80 0x7f847c005bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:00.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f84927fc640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f847c0069e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8488001750 tx=0x7f848801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1386ae6b102f819c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:00.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84880052d0 con 0x7f847c006140 2023-10-30T13:20:00.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84880a7050 con 0x7f847c006140 2023-10-30T13:20:00.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8491ffb640 1 -- 172.21.15.148:0/3156085837 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488003600 con 0x7f847c006140 2023-10-30T13:20:00.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 -- 172.21.15.148:0/3156085837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 msgr2=0x7f847c0069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:00.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 --2- 172.21.15.148:0/3156085837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f847c0069e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8488001750 tx=0x7f848801c3c0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f84937fe640 1 -- 172.21.15.148:0/3156085837 reap_dead start 2023-10-30T13:20:00.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 -- 172.21.15.148:0/3156085837 shutdown_connections 2023-10-30T13:20:00.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 -- 172.21.15.148:0/3156085837 >> 172.21.15.148:0/3156085837 conn(0x7f84940c7910 msgr2=0x7f84940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:00.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 -- 172.21.15.148:0/3156085837 shutdown_connections 2023-10-30T13:20:00.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 -- 172.21.15.148:0/3156085837 wait complete. 2023-10-30T13:20:00.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 Processor -- start 2023-10-30T13:20:00.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 -- start start 2023-10-30T13:20:00.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f84940ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:00.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941ef6f0 0x7f84941547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:00.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494154cf0 0x7f84941f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:00.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84940adce0 con 0x7f847c006140 2023-10-30T13:20:00.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8494155170 con 0x7f8494154cf0 2023-10-30T13:20:00.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.515+0000 7f8499817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84940b0060 con 0x7f84941ef6f0 2023-10-30T13:20:00.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8492ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f84940ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:00.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8492ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f84940ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42322/0 (socket says 172.21.15.148:42322) 2023-10-30T13:20:00.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8492ffd640 1 -- 172.21.15.148:0/3055128883 learned_addr learned my addr 172.21.15.148:0/3055128883 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:00.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f84927fc640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941ef6f0 0x7f84941547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:00.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f84937fe640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494154cf0 0x7f84941f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:00.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8492ffd640 1 -- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941ef6f0 msgr2=0x7f84941547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:00.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8492ffd640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941ef6f0 0x7f84941547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8492ffd640 1 -- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494154cf0 msgr2=0x7f84941f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:00.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8492ffd640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494154cf0 0x7f84941f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8492ffd640 1 -- 172.21.15.148:0/3055128883 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84940b0340 con 0x7f847c006140 2023-10-30T13:20:00.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8492ffd640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f84940ae170 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8484009fd0 tx=0x7f8484011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:00.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f847bfff640 1 -- 172.21.15.148:0/3055128883 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8484004220 con 0x7f847c006140 2023-10-30T13:20:00.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f847bfff640 1 -- 172.21.15.148:0/3055128883 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8484004380 con 0x7f847c006140 2023-10-30T13:20:00.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84940b0620 con 0x7f847c006140 2023-10-30T13:20:00.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f847bfff640 1 -- 172.21.15.148:0/3055128883 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848400f9e0 con 0x7f847c006140 2023-10-30T13:20:00.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84940afa10 con 0x7f847c006140 2023-10-30T13:20:00.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849404e170 con 0x7f847c006140 2023-10-30T13:20:00.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f847bfff640 1 -- 172.21.15.148:0/3055128883 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f848400fb40 con 0x7f847c006140 2023-10-30T13:20:00.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f847bfff640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84800664a0 0x7f8480068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:00.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.519+0000 7f847bfff640 1 -- 172.21.15.148:0/3055128883 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f8484084700 con 0x7f847c006140 2023-10-30T13:20:00.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.523+0000 7f84927fc640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84800664a0 0x7f8480068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:00.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.523+0000 7f84927fc640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84800664a0 0x7f8480068960 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f84940c7380 tx=0x7f84880011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:00.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.527+0000 7f847bfff640 1 -- 172.21.15.148:0/3055128883 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8484051900 con 0x7f847c006140 2023-10-30T13:20:00.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.731+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f849406ac50 con 0x7f847c006140 2023-10-30T13:20:00.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.731+0000 7f847bfff640 1 -- 172.21.15.148:0/3055128883 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6541 (secure 0 0 0) 0x7f8484022090 con 0x7f847c006140 2023-10-30T13:20:00.732 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:00.732 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:00.624273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:00.733 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 47 2023-10-30T13:20:00.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84800664a0 msgr2=0x7f8480068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:00.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84800664a0 0x7f8480068960 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f84940c7380 tx=0x7f84880011a0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 msgr2=0x7f84940ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:00.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f84940ae170 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8484009fd0 tx=0x7f8484011b30 comp rx=0 tx=0).stop 2023-10-30T13:20:00.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 shutdown_connections 2023-10-30T13:20:00.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84800664a0 0x7f8480068960 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8494154cf0 0x7f84941f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941ef6f0 0x7f84941547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 --2- 172.21.15.148:0/3055128883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f847c006140 0x7f84940ae170 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:00.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 >> 172.21.15.148:0/3055128883 conn(0x7f84940c7910 msgr2=0x7f84940c16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:00.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 shutdown_connections 2023-10-30T13:20:00.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:00.735+0000 7f8499817640 1 -- 172.21.15.148:0/3055128883 wait complete. 2023-10-30T13:20:00.757 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:02.758 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.903+0000 7fbca673e640 1 Processor -- start 2023-10-30T13:20:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.903+0000 7fbca673e640 1 -- start start 2023-10-30T13:20:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.903+0000 7fbca673e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbca006f7a0 0x7fbca006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.903+0000 7fbca673e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca0071110 0x7fbca0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.903+0000 7fbca673e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca0066cf0 con 0x7fbca00c6bd0 2023-10-30T13:20:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.903+0000 7fbca673e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca00700c0 con 0x7fbca006f7a0 2023-10-30T13:20:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.903+0000 7fbca673e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca00c70d0 con 0x7fbca0071110 2023-10-30T13:20:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.907+0000 7fbca4cdb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca0071110 0x7fbca0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.907+0000 7fbc9ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbca006f7a0 0x7fbca006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.907+0000 7fbc9f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbca00c6bd0 0x7fbca0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:45100/0 (socket says 172.21.15.148:45100) 2023-10-30T13:20:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.907+0000 7fbc9f7fe640 1 -- 172.21.15.148:0/1804712419 learned_addr learned my addr 172.21.15.148:0/1804712419 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.907+0000 7fbc9ffff640 1 -- 172.21.15.148:0/1804712419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca0071110 msgr2=0x7fbca0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.907+0000 7fbc9ffff640 1 --2- 172.21.15.148:0/1804712419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca0071110 0x7fbca0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.907+0000 7fbc9ffff640 1 -- 172.21.15.148:0/1804712419 >> v1:172.21.15.150:6789/0 conn(0x7fbca00c6bd0 legacy=0x7fbca0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.907+0000 7fbc9ffff640 1 -- 172.21.15.148:0/1804712419 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca00c7390 con 0x7fbca006f7a0 2023-10-30T13:20:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbc9ffff640 1 --2- 172.21.15.148:0/1804712419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbca006f7a0 0x7fbca006fb80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc900097c0 tx=0x7fbc9001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e21a22c7bb5dde3a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbc9effd640 1 -- 172.21.15.148:0/1804712419 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900a8970 con 0x7fbca006f7a0 2023-10-30T13:20:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbc9effd640 1 -- 172.21.15.148:0/1804712419 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc900a8ad0 con 0x7fbca006f7a0 2023-10-30T13:20:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbc9effd640 1 -- 172.21.15.148:0/1804712419 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900a8e60 con 0x7fbca006f7a0 2023-10-30T13:20:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 -- 172.21.15.148:0/1804712419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbca006f7a0 msgr2=0x7fbca006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 --2- 172.21.15.148:0/1804712419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbca006f7a0 0x7fbca006fb80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc900097c0 tx=0x7fbc9001ec70 comp rx=0 tx=0).stop 2023-10-30T13:20:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 -- 172.21.15.148:0/1804712419 shutdown_connections 2023-10-30T13:20:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 --2- 172.21.15.148:0/1804712419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca0071110 0x7fbca0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 --2- 172.21.15.148:0/1804712419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbca006f7a0 0x7fbca006fb80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 -- 172.21.15.148:0/1804712419 >> 172.21.15.148:0/1804712419 conn(0x7fbca00c7910 msgr2=0x7fbca00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 -- 172.21.15.148:0/1804712419 shutdown_connections 2023-10-30T13:20:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 -- 172.21.15.148:0/1804712419 wait complete. 2023-10-30T13:20:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 Processor -- start 2023-10-30T13:20:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.911+0000 7fbca673e640 1 -- start start 2023-10-30T13:20:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca673e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbca006f7a0 0x7fbca00af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca673e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbca0071110 0x7fbca00c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca673e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca00c6bd0 0x7fbca01f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca673e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca01ef140 con 0x7fbca006f7a0 2023-10-30T13:20:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca673e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca0158020 con 0x7fbca0071110 2023-10-30T13:20:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca673e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca0158290 con 0x7fbca00c6bd0 2023-10-30T13:20:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbca006f7a0 0x7fbca00af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbca006f7a0 0x7fbca00af4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42720/0 (socket says 172.21.15.148:42720) 2023-10-30T13:20:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9ffff640 1 -- 172.21.15.148:0/3352344039 learned_addr learned my addr 172.21.15.148:0/3352344039 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:02.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9f7fe640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbca0071110 0x7fbca00c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:02.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca4cdb640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca00c6bd0 0x7fbca01f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:02.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9ffff640 1 -- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca00c6bd0 msgr2=0x7fbca01f1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:02.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9ffff640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca00c6bd0 0x7fbca01f1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:02.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9ffff640 1 -- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbca0071110 msgr2=0x7fbca00c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9ffff640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbca0071110 0x7fbca00c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9ffff640 1 -- 172.21.15.148:0/3352344039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc90009420 con 0x7fbca006f7a0 2023-10-30T13:20:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca4cdb640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca00c6bd0 0x7fbca01f1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9ffff640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbca006f7a0 0x7fbca00af4a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbc9001c8a0 tx=0x7fbc9001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9cff9640 1 -- 172.21.15.148:0/3352344039 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900b9630 con 0x7fbca006f7a0 2023-10-30T13:20:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9cff9640 1 -- 172.21.15.148:0/3352344039 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc900b97b0 con 0x7fbca006f7a0 2023-10-30T13:20:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9cff9640 1 -- 172.21.15.148:0/3352344039 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900b9b60 con 0x7fbca006f7a0 2023-10-30T13:20:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca01eeaf0 con 0x7fbca006f7a0 2023-10-30T13:20:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca01ee7a0 con 0x7fbca006f7a0 2023-10-30T13:20:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca004e170 con 0x7fbca006f7a0 2023-10-30T13:20:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.915+0000 7fbc9cff9640 1 -- 172.21.15.148:0/3352344039 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbc900b7040 con 0x7fbca006f7a0 2023-10-30T13:20:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.919+0000 7fbc9cff9640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc8c0664d0 0x7fbc8c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:02.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.919+0000 7fbc9cff9640 1 -- 172.21.15.148:0/3352344039 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fbc90121e90 con 0x7fbca006f7a0 2023-10-30T13:20:02.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.919+0000 7fbc9f7fe640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc8c0664d0 0x7fbc8c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:02.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.919+0000 7fbc9f7fe640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc8c0664d0 0x7fbc8c068990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbca00c66a0 tx=0x7fbc94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:02.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:02.923+0000 7fbc9cff9640 1 -- 172.21.15.148:0/3352344039 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbc900ef230 con 0x7fbca006f7a0 2023-10-30T13:20:03.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbca01585a0 con 0x7fbca006f7a0 2023-10-30T13:20:03.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbc9cff9640 1 -- 172.21.15.148:0/3352344039 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6541 (secure 0 0 0) 0x7fbc900f2850 con 0x7fbca006f7a0 2023-10-30T13:20:03.103 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:03.104 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:00.624273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:03.104 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 47 2023-10-30T13:20:03.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc8c0664d0 msgr2=0x7fbc8c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:03.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc8c0664d0 0x7fbc8c068990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbca00c66a0 tx=0x7fbc94009040 comp rx=0 tx=0).stop 2023-10-30T13:20:03.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbca006f7a0 msgr2=0x7fbca00af4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:03.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbca006f7a0 0x7fbca00af4a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fbc9001c8a0 tx=0x7fbc9001ba10 comp rx=0 tx=0).stop 2023-10-30T13:20:03.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 shutdown_connections 2023-10-30T13:20:03.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbc8c0664d0 0x7fbc8c068990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:03.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbca00c6bd0 0x7fbca01f1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:03.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbca0071110 0x7fbca00c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:03.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 --2- 172.21.15.148:0/3352344039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbca006f7a0 0x7fbca00af4a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:03.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 >> 172.21.15.148:0/3352344039 conn(0x7fbca00c7910 msgr2=0x7fbca00c93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:03.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 shutdown_connections 2023-10-30T13:20:03.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:03.103+0000 7fbca673e640 1 -- 172.21.15.148:0/3352344039 wait complete. 2023-10-30T13:20:03.127 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:03.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/25' 2023-10-30T13:20:03.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/30' 2023-10-30T13:20:03.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/33' 2023-10-30T13:20:03.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/23' 2023-10-30T13:20:03.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/35' 2023-10-30T13:20:03.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/26' 2023-10-30T13:20:03.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed './246/37' 2023-10-30T13:20:03.740 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './246' 2023-10-30T13:20:03.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/21' 2023-10-30T13:20:03.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/27' 2023-10-30T13:20:03.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/31' 2023-10-30T13:20:03.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/36' 2023-10-30T13:20:03.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/20' 2023-10-30T13:20:03.741 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/39' 2023-10-30T13:20:03.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/34' 2023-10-30T13:20:03.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/38' 2023-10-30T13:20:03.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/28' 2023-10-30T13:20:03.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/24' 2023-10-30T13:20:03.742 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/29' 2023-10-30T13:20:03.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/32' 2023-10-30T13:20:03.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/22' 2023-10-30T13:20:03.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/25' 2023-10-30T13:20:03.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/30' 2023-10-30T13:20:03.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/33' 2023-10-30T13:20:03.743 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/23' 2023-10-30T13:20:03.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/35' 2023-10-30T13:20:03.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/26' 2023-10-30T13:20:03.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './247/37' 2023-10-30T13:20:03.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './247' 2023-10-30T13:20:03.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/21' 2023-10-30T13:20:03.744 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/27' 2023-10-30T13:20:03.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/31' 2023-10-30T13:20:03.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/36' 2023-10-30T13:20:03.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/20' 2023-10-30T13:20:03.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/39' 2023-10-30T13:20:03.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/34' 2023-10-30T13:20:03.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/38' 2023-10-30T13:20:03.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/28' 2023-10-30T13:20:03.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/24' 2023-10-30T13:20:03.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/29' 2023-10-30T13:20:03.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/32' 2023-10-30T13:20:03.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/22' 2023-10-30T13:20:03.746 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/25' 2023-10-30T13:20:03.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/30' 2023-10-30T13:20:03.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/33' 2023-10-30T13:20:03.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/23' 2023-10-30T13:20:03.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/35' 2023-10-30T13:20:03.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/26' 2023-10-30T13:20:03.747 INFO:tasks.workunit.client.0.smithi148.stdout:removed './248/37' 2023-10-30T13:20:03.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './248' 2023-10-30T13:20:03.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/21' 2023-10-30T13:20:03.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/27' 2023-10-30T13:20:03.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/31' 2023-10-30T13:20:03.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/36' 2023-10-30T13:20:03.748 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/20' 2023-10-30T13:20:03.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/39' 2023-10-30T13:20:03.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/34' 2023-10-30T13:20:03.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/38' 2023-10-30T13:20:03.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/28' 2023-10-30T13:20:03.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/24' 2023-10-30T13:20:03.749 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/29' 2023-10-30T13:20:03.750 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/32' 2023-10-30T13:20:03.750 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/22' 2023-10-30T13:20:03.750 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/25' 2023-10-30T13:20:03.750 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/30' 2023-10-30T13:20:03.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/33' 2023-10-30T13:20:03.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/23' 2023-10-30T13:20:03.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/35' 2023-10-30T13:20:03.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/26' 2023-10-30T13:20:03.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed './249/37' 2023-10-30T13:20:03.751 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './249' 2023-10-30T13:20:03.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/21' 2023-10-30T13:20:03.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/27' 2023-10-30T13:20:03.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/31' 2023-10-30T13:20:03.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/36' 2023-10-30T13:20:03.752 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/20' 2023-10-30T13:20:03.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/39' 2023-10-30T13:20:03.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/34' 2023-10-30T13:20:03.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/38' 2023-10-30T13:20:03.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/28' 2023-10-30T13:20:03.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/24' 2023-10-30T13:20:03.753 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/29' 2023-10-30T13:20:03.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/32' 2023-10-30T13:20:03.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/22' 2023-10-30T13:20:03.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/25' 2023-10-30T13:20:03.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/30' 2023-10-30T13:20:03.754 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/33' 2023-10-30T13:20:03.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/23' 2023-10-30T13:20:03.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/35' 2023-10-30T13:20:03.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/26' 2023-10-30T13:20:03.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './250/37' 2023-10-30T13:20:03.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './250' 2023-10-30T13:20:03.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/21' 2023-10-30T13:20:03.755 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/27' 2023-10-30T13:20:03.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/31' 2023-10-30T13:20:03.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/36' 2023-10-30T13:20:03.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/20' 2023-10-30T13:20:03.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/39' 2023-10-30T13:20:03.756 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/34' 2023-10-30T13:20:03.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/38' 2023-10-30T13:20:03.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/28' 2023-10-30T13:20:03.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/24' 2023-10-30T13:20:03.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/29' 2023-10-30T13:20:03.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/32' 2023-10-30T13:20:03.757 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/22' 2023-10-30T13:20:03.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/25' 2023-10-30T13:20:03.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/30' 2023-10-30T13:20:03.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/33' 2023-10-30T13:20:03.758 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/23' 2023-10-30T13:20:03.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/35' 2023-10-30T13:20:03.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/26' 2023-10-30T13:20:03.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './251/37' 2023-10-30T13:20:03.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './251' 2023-10-30T13:20:03.759 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/21' 2023-10-30T13:20:03.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/27' 2023-10-30T13:20:03.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/31' 2023-10-30T13:20:03.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/36' 2023-10-30T13:20:03.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/20' 2023-10-30T13:20:03.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/39' 2023-10-30T13:20:03.760 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/34' 2023-10-30T13:20:03.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/38' 2023-10-30T13:20:03.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/28' 2023-10-30T13:20:03.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/24' 2023-10-30T13:20:03.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/29' 2023-10-30T13:20:03.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/32' 2023-10-30T13:20:03.761 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/22' 2023-10-30T13:20:03.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/25' 2023-10-30T13:20:03.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/30' 2023-10-30T13:20:03.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/33' 2023-10-30T13:20:03.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/23' 2023-10-30T13:20:03.762 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/35' 2023-10-30T13:20:03.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/26' 2023-10-30T13:20:03.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './252/37' 2023-10-30T13:20:03.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './252' 2023-10-30T13:20:03.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/21' 2023-10-30T13:20:03.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/27' 2023-10-30T13:20:03.763 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/31' 2023-10-30T13:20:03.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/36' 2023-10-30T13:20:03.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/20' 2023-10-30T13:20:03.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/39' 2023-10-30T13:20:03.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/34' 2023-10-30T13:20:03.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/38' 2023-10-30T13:20:03.764 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/28' 2023-10-30T13:20:03.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/24' 2023-10-30T13:20:03.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/29' 2023-10-30T13:20:03.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/32' 2023-10-30T13:20:03.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/22' 2023-10-30T13:20:03.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/25' 2023-10-30T13:20:03.765 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/30' 2023-10-30T13:20:03.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/33' 2023-10-30T13:20:03.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/23' 2023-10-30T13:20:03.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/35' 2023-10-30T13:20:03.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/26' 2023-10-30T13:20:03.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed './253/37' 2023-10-30T13:20:03.766 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './253' 2023-10-30T13:20:03.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/21' 2023-10-30T13:20:03.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/27' 2023-10-30T13:20:03.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/31' 2023-10-30T13:20:03.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/36' 2023-10-30T13:20:03.767 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/20' 2023-10-30T13:20:03.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/39' 2023-10-30T13:20:03.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/34' 2023-10-30T13:20:03.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/38' 2023-10-30T13:20:03.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/28' 2023-10-30T13:20:03.768 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/24' 2023-10-30T13:20:03.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/29' 2023-10-30T13:20:03.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/32' 2023-10-30T13:20:03.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/22' 2023-10-30T13:20:03.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/25' 2023-10-30T13:20:03.769 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/30' 2023-10-30T13:20:03.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/33' 2023-10-30T13:20:03.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/23' 2023-10-30T13:20:03.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/35' 2023-10-30T13:20:03.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/26' 2023-10-30T13:20:03.770 INFO:tasks.workunit.client.0.smithi148.stdout:removed './254/37' 2023-10-30T13:20:03.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './254' 2023-10-30T13:20:03.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/21' 2023-10-30T13:20:03.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/27' 2023-10-30T13:20:03.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/31' 2023-10-30T13:20:03.771 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/36' 2023-10-30T13:20:03.772 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/20' 2023-10-30T13:20:03.772 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/39' 2023-10-30T13:20:03.772 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/34' 2023-10-30T13:20:03.772 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/38' 2023-10-30T13:20:03.772 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/28' 2023-10-30T13:20:03.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/24' 2023-10-30T13:20:03.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/29' 2023-10-30T13:20:03.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/32' 2023-10-30T13:20:03.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/22' 2023-10-30T13:20:03.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/25' 2023-10-30T13:20:03.773 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/30' 2023-10-30T13:20:03.774 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/33' 2023-10-30T13:20:03.774 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/23' 2023-10-30T13:20:03.774 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/35' 2023-10-30T13:20:03.774 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/26' 2023-10-30T13:20:03.774 INFO:tasks.workunit.client.0.smithi148.stdout:removed './255/37' 2023-10-30T13:20:03.775 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './255' 2023-10-30T13:20:03.775 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/21' 2023-10-30T13:20:03.775 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/27' 2023-10-30T13:20:03.775 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/31' 2023-10-30T13:20:03.775 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/36' 2023-10-30T13:20:03.776 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/20' 2023-10-30T13:20:03.776 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/39' 2023-10-30T13:20:03.776 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/34' 2023-10-30T13:20:03.776 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/38' 2023-10-30T13:20:03.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/28' 2023-10-30T13:20:03.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/24' 2023-10-30T13:20:03.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/29' 2023-10-30T13:20:03.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/32' 2023-10-30T13:20:03.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/22' 2023-10-30T13:20:03.777 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/25' 2023-10-30T13:20:03.778 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/30' 2023-10-30T13:20:05.128 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:05.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.275+0000 7f8c4c1d3640 1 Processor -- start 2023-10-30T13:20:05.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.275+0000 7f8c4c1d3640 1 -- start start 2023-10-30T13:20:05.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4c1d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8c441538a0 0x7f8c4415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:05.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4c1d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8c440cf160 0x7f8c440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:05.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4c1d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c440cb800 con 0x7f8c440ca0e0 2023-10-30T13:20:05.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4c1d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4415dc30 con 0x7f8c440cf160 2023-10-30T13:20:05.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4c1d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c440cb330 con 0x7f8c441538a0 2023-10-30T13:20:05.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4a770640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8c440cf160 0x7f8c440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:05.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4976e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8c440ca0e0 0x7f8c4415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46090/0 (socket says 172.21.15.148:46090) 2023-10-30T13:20:05.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4a770640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8c440cf160 0x7f8c440cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42732/0 (socket says 172.21.15.148:42732) 2023-10-30T13:20:05.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4976e640 1 -- 172.21.15.148:0/4255703468 learned_addr learned my addr 172.21.15.148:0/4255703468 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:05.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c49f6f640 1 --2- 172.21.15.148:0/4255703468 >> v2:172.21.15.150:3300/0 conn(0x7f8c441538a0 0x7f8c4415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:05.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c48f6d640 1 -- 172.21.15.148:0/4255703468 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483694807 0 0) 0x7f8c440cb800 con 0x7f8c440ca0e0 2023-10-30T13:20:05.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c48f6d640 1 -- 172.21.15.148:0/4255703468 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c30003530 con 0x7f8c440ca0e0 2023-10-30T13:20:05.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4a770640 1 -- 172.21.15.148:0/4255703468 >> v2:172.21.15.150:3300/0 conn(0x7f8c441538a0 msgr2=0x7f8c4415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:05.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4a770640 1 --2- 172.21.15.148:0/4255703468 >> v2:172.21.15.150:3300/0 conn(0x7f8c441538a0 0x7f8c4415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:05.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4a770640 1 -- 172.21.15.148:0/4255703468 >> v1:172.21.15.148:6789/0 conn(0x7f8c440ca0e0 legacy=0x7f8c4415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:05.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.279+0000 7f8c4a770640 1 -- 172.21.15.148:0/4255703468 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c440cad70 con 0x7f8c440cf160 2023-10-30T13:20:05.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c4a770640 1 --2- 172.21.15.148:0/4255703468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 0x7f8c440cf590 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8c440cfd10 tx=0x7f8c3c01f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5cc470c156a7523 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:05.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c48f6d640 1 -- 172.21.15.148:0/4255703468 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c0b0440 con 0x7f8c440cf160 2023-10-30T13:20:05.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c48f6d640 1 -- 172.21.15.148:0/4255703468 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3c0b05a0 con 0x7f8c440cf160 2023-10-30T13:20:05.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c48f6d640 1 -- 172.21.15.148:0/4255703468 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c0b0930 con 0x7f8c440cf160 2023-10-30T13:20:05.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/4255703468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 msgr2=0x7f8c440cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:05.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c4c1d3640 1 --2- 172.21.15.148:0/4255703468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 0x7f8c440cf590 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8c440cfd10 tx=0x7f8c3c01f110 comp rx=0 tx=0).stop 2023-10-30T13:20:05.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/4255703468 shutdown_connections 2023-10-30T13:20:05.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c4c1d3640 1 --2- 172.21.15.148:0/4255703468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 0x7f8c440cf590 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:05.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c4c1d3640 1 --2- 172.21.15.148:0/4255703468 >> v2:172.21.15.150:3300/0 conn(0x7f8c441538a0 0x7f8c4415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:05.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/4255703468 >> 172.21.15.148:0/4255703468 conn(0x7f8c4406bfe0 msgr2=0x7f8c4406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:05.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/4255703468 shutdown_connections 2023-10-30T13:20:05.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.283+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/4255703468 wait complete. 2023-10-30T13:20:05.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 Processor -- start 2023-10-30T13:20:05.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 -- start start 2023-10-30T13:20:05.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c440ca0e0 0x7f8c4415a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:05.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 0x7f8c441f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:05.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c441538a0 0x7f8c44145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:05.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c44149300 con 0x7f8c440cf160 2023-10-30T13:20:05.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4415d3d0 con 0x7f8c440ca0e0 2023-10-30T13:20:05.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c44148840 con 0x7f8c441538a0 2023-10-30T13:20:05.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4976e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 0x7f8c441f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:05.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4976e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 0x7f8c441f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42746/0 (socket says 172.21.15.148:42746) 2023-10-30T13:20:05.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4976e640 1 -- 172.21.15.148:0/128457538 learned_addr learned my addr 172.21.15.148:0/128457538 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:05.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c49f6f640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c440ca0e0 0x7f8c4415a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:05.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4a770640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c441538a0 0x7f8c44145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:05.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4976e640 1 -- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c441538a0 msgr2=0x7f8c44145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:05.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4976e640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c441538a0 0x7f8c44145a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:05.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4976e640 1 -- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c440ca0e0 msgr2=0x7f8c4415a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:05.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4976e640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c440ca0e0 0x7f8c4415a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:05.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4976e640 1 -- 172.21.15.148:0/128457538 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c009a70 con 0x7f8c440cf160 2023-10-30T13:20:05.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c49f6f640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c440ca0e0 0x7f8c4415a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:05.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4976e640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 0x7f8c441f2660 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400c450 tx=0x7f8c3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:05.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c3a7fc640 1 -- 172.21.15.148:0/128457538 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34005cf0 con 0x7f8c440cf160 2023-10-30T13:20:05.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c3a7fc640 1 -- 172.21.15.148:0/128457538 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c34005e50 con 0x7f8c440cf160 2023-10-30T13:20:05.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c441f3030 con 0x7f8c440cf160 2023-10-30T13:20:05.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c3a7fc640 1 -- 172.21.15.148:0/128457538 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34007270 con 0x7f8c440cf160 2023-10-30T13:20:05.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4415c870 con 0x7f8c440cf160 2023-10-30T13:20:05.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.287+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c440b2a50 con 0x7f8c440cf160 2023-10-30T13:20:05.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.291+0000 7f8c3a7fc640 1 -- 172.21.15.148:0/128457538 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c340073d0 con 0x7f8c440cf160 2023-10-30T13:20:05.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.291+0000 7f8c3a7fc640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c280664a0 0x7f8c28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:05.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.291+0000 7f8c3a7fc640 1 -- 172.21.15.148:0/128457538 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f8c340834d0 con 0x7f8c440cf160 2023-10-30T13:20:05.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.291+0000 7f8c49f6f640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c280664a0 0x7f8c28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:05.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.291+0000 7f8c49f6f640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c280664a0 0x7f8c28068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8c40009bb0 tx=0x7f8c40008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:05.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.295+0000 7f8c3a7fc640 1 -- 172.21.15.148:0/128457538 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c34050780 con 0x7f8c440cf160 2023-10-30T13:20:05.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.487+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c4404e170 con 0x7f8c440cf160 2023-10-30T13:20:05.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c3a7fc640 1 -- 172.21.15.148:0/128457538 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6541 (secure 0 0 0) 0x7f8c34017090 con 0x7f8c440cf160 2023-10-30T13:20:05.491 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:05.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:00.624273+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:05.492 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 47 2023-10-30T13:20:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c280664a0 msgr2=0x7f8c28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c280664a0 0x7f8c28068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8c40009bb0 tx=0x7f8c40008040 comp rx=0 tx=0).stop 2023-10-30T13:20:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 msgr2=0x7f8c441f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 0x7f8c441f2660 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400c450 tx=0x7f8c3400c910 comp rx=0 tx=0).stop 2023-10-30T13:20:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 shutdown_connections 2023-10-30T13:20:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8c280664a0 0x7f8c28068960 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c440cf160 0x7f8c441f2660 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:05.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c441538a0 0x7f8c44145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:05.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 --2- 172.21.15.148:0/128457538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c440ca0e0 0x7f8c4415a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:05.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 >> 172.21.15.148:0/128457538 conn(0x7f8c4406bfe0 msgr2=0x7f8c440c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:05.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 shutdown_connections 2023-10-30T13:20:05.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:05.491+0000 7f8c4c1d3640 1 -- 172.21.15.148:0/128457538 wait complete. 2023-10-30T13:20:05.512 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:07.513 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:07.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.659+0000 7f9791817640 1 Processor -- start 2023-10-30T13:20:07.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.659+0000 7f9791817640 1 -- start start 2023-10-30T13:20:07.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f9791817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0bfec0 0x7f978c0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:07.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f9791817640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c071290 con 0x7f978c06a720 2023-10-30T13:20:07.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f9791817640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c071570 con 0x7f978c159e60 2023-10-30T13:20:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f9791817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c071850 con 0x7f978c0bfec0 2023-10-30T13:20:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f978c06a720 0x7f978c160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46106/0 (socket says 172.21.15.148:46106) 2023-10-30T13:20:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978b7fe640 1 -- 172.21.15.148:0/114079289 learned_addr learned my addr 172.21.15.148:0/114079289 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f97827fc640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0bfec0 0x7f978c0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:07.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145223249 0 0) 0x7f978c071290 con 0x7f978c06a720 2023-10-30T13:20:07.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9778003530 con 0x7f978c06a720 2023-10-30T13:20:07.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3400629490 0 0) 0x7f9778003530 con 0x7f978c06a720 2023-10-30T13:20:07.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0bfec0 msgr2=0x7f978c0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:07.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0bfec0 0x7f978c0c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 >> v1:172.21.15.150:6789/0 conn(0x7f978c159e60 legacy=0x7f978c06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:07.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c071b30 con 0x7f978c06a720 2023-10-30T13:20:07.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f9784001680 con 0x7f978c06a720 2023-10-30T13:20:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 >> v1:172.21.15.148:6789/0 conn(0x7f978c06a720 legacy=0x7f978c160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97780039b0 0x7f9778003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f9778004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0bfec0 0x7f9778005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:07.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c071850 con 0x7f97780047d0 2023-10-30T13:20:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977800bad0 con 0x7f97780039b0 2023-10-30T13:20:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977800bc90 con 0x7f978c0bfec0 2023-10-30T13:20:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f97827fc640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f9778004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978affd640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97780039b0 0x7f9778003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:07.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f978b7fe640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0bfec0 0x7f9778005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f97827fc640 1 -- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0bfec0 msgr2=0x7f9778005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f97827fc640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c0bfec0 0x7f9778005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f97827fc640 1 -- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97780039b0 msgr2=0x7f9778003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f97827fc640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97780039b0 0x7f9778003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.663+0000 7f97827fc640 1 -- 172.21.15.148:0/114079289 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778003530 con 0x7f97780047d0 2023-10-30T13:20:07.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f97827fc640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f9778004c50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f977400a810 tx=0x7f977401f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780ba902aadd2740 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:07.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740ae600 con 0x7f97780047d0 2023-10-30T13:20:07.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97740aa050 con 0x7f97780047d0 2023-10-30T13:20:07.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f978a7fc640 1 -- 172.21.15.148:0/114079289 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740ae9a0 con 0x7f97780047d0 2023-10-30T13:20:07.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f9791817640 1 -- 172.21.15.148:0/114079289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 msgr2=0x7f9778004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:07.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f9791817640 1 --2- 172.21.15.148:0/114079289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f9778004c50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f977400a810 tx=0x7f977401f050 comp rx=0 tx=0).stop 2023-10-30T13:20:07.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f978b7fe640 1 -- 172.21.15.148:0/114079289 reap_dead start 2023-10-30T13:20:07.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f9791817640 1 -- 172.21.15.148:0/114079289 shutdown_connections 2023-10-30T13:20:07.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f9791817640 1 -- 172.21.15.148:0/114079289 >> 172.21.15.148:0/114079289 conn(0x7f978c0c7910 msgr2=0x7f978c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:07.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f9791817640 1 -- 172.21.15.148:0/114079289 shutdown_connections 2023-10-30T13:20:07.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.667+0000 7f9791817640 1 -- 172.21.15.148:0/114079289 wait complete. 2023-10-30T13:20:07.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 Processor -- start 2023-10-30T13:20:07.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 -- start start 2023-10-30T13:20:07.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f978c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:07.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c156570 0x7f978c155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:07.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c1f27d0 0x7f978c1559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:07.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0c4430 con 0x7f97780047d0 2023-10-30T13:20:07.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1ef590 con 0x7f978c156570 2023-10-30T13:20:07.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1ef870 con 0x7f978c1f27d0 2023-10-30T13:20:07.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f978c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:07.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f978c0736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42754/0 (socket says 172.21.15.148:42754) 2023-10-30T13:20:07.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978affd640 1 -- 172.21.15.148:0/258079029 learned_addr learned my addr 172.21.15.148:0/258079029 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:07.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f97827fc640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c156570 0x7f978c155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:07.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978b7fe640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c1f27d0 0x7f978c1559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:07.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978affd640 1 -- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c1f27d0 msgr2=0x7f978c1559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:07.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978affd640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c1f27d0 0x7f978c1559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978affd640 1 -- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c156570 msgr2=0x7f978c155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:07.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978affd640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c156570 0x7f978c155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978affd640 1 -- 172.21.15.148:0/258079029 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c1efb50 con 0x7f97780047d0 2023-10-30T13:20:07.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978b7fe640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c1f27d0 0x7f978c1559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:07.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f978affd640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f978c0736e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f977c00fb90 tx=0x7f977c00d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:07.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9788ff9640 1 -- 172.21.15.148:0/258079029 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c00fe90 con 0x7f97780047d0 2023-10-30T13:20:07.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9788ff9640 1 -- 172.21.15.148:0/258079029 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c00dc30 con 0x7f97780047d0 2023-10-30T13:20:07.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9788ff9640 1 -- 172.21.15.148:0/258079029 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c003980 con 0x7f97780047d0 2023-10-30T13:20:07.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c1efe30 con 0x7f97780047d0 2023-10-30T13:20:07.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c1ef260 con 0x7f97780047d0 2023-10-30T13:20:07.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.671+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c04e170 con 0x7f97780047d0 2023-10-30T13:20:07.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.675+0000 7f9788ff9640 1 -- 172.21.15.148:0/258079029 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f977c003ae0 con 0x7f97780047d0 2023-10-30T13:20:07.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.675+0000 7f9788ff9640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f976c0664a0 0x7f976c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:07.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.675+0000 7f9788ff9640 1 -- 172.21.15.148:0/258079029 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f977c0842b0 con 0x7f97780047d0 2023-10-30T13:20:07.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.675+0000 7f97827fc640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f976c0664a0 0x7f976c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:07.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.675+0000 7f97827fc640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f976c0664a0 0x7f976c068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f97740b0000 tx=0x7f97740046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:07.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.679+0000 7f9788ff9640 1 -- 172.21.15.148:0/258079029 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f977c051530 con 0x7f97780047d0 2023-10-30T13:20:07.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.863+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f978c0c6460 con 0x7f97780047d0 2023-10-30T13:20:07.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.863+0000 7f9788ff9640 1 -- 172.21.15.148:0/258079029 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7f977c054b50 con 0x7f97780047d0 2023-10-30T13:20:07.866 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:07.866 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:07.867 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:07.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f976c0664a0 msgr2=0x7f976c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:07.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f976c0664a0 0x7f976c068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f97740b0000 tx=0x7f97740046d0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 msgr2=0x7f978c0736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:07.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f978c0736e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f977c00fb90 tx=0x7f977c00d520 comp rx=0 tx=0).stop 2023-10-30T13:20:07.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 shutdown_connections 2023-10-30T13:20:07.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f976c0664a0 0x7f976c068960 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f978c1f27d0 0x7f978c1559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f978c156570 0x7f978c155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 --2- 172.21.15.148:0/258079029 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97780047d0 0x7f978c0736e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:07.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 >> 172.21.15.148:0/258079029 conn(0x7f978c0c7910 msgr2=0x7f978c0c6980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 shutdown_connections 2023-10-30T13:20:07.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:07.867+0000 7f9791817640 1 -- 172.21.15.148:0/258079029 wait complete. 2023-10-30T13:20:07.886 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:20:07.887 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4959": { "addr": "172.21.15.148:6837/95380877", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 95380877, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 95380877, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4959, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:stopping", "state_seq": 16 }, "gid_5033": { "addr": "172.21.15.150:6836/953984429", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 953984429, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 953984429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5033, "incarnation": 36, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:20:06.522682+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5033, "mds_1": 5103, "mds_2": 4959 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:20:09.889 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:10.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd17409c640 1 Processor -- start 2023-10-30T13:20:10.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd17409c640 1 -- start start 2023-10-30T13:20:10.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd17409c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 0x7fd16c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:10.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd17409c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd16c0ca0e0 0x7fd16c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:10.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd17409c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd16c0cf160 0x7fd16c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:10.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd17409c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16c0cb800 con 0x7fd16c0cf160 2023-10-30T13:20:10.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd17409c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16c15dc30 con 0x7fd16c0ca0e0 2023-10-30T13:20:10.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd17409c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c0cb330 con 0x7fd16c1538a0 2023-10-30T13:20:10.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd172639640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd16c0cf160 0x7fd16c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:10.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd172639640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd16c0cf160 0x7fd16c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42758/0 (socket says 172.21.15.148:42758) 2023-10-30T13:20:10.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd172639640 1 -- 172.21.15.148:0/3707354311 learned_addr learned my addr 172.21.15.148:0/3707354311 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd171e38640 1 --2- 172.21.15.148:0/3707354311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 0x7fd16c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd171637640 1 --2- 172.21.15.148:0/3707354311 >> v2:172.21.15.150:3300/0 conn(0x7fd16c0ca0e0 0x7fd16c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd172639640 1 -- 172.21.15.148:0/3707354311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 msgr2=0x7fd16c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd172639640 1 --2- 172.21.15.148:0/3707354311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 0x7fd16c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd172639640 1 -- 172.21.15.148:0/3707354311 >> v2:172.21.15.150:3300/0 conn(0x7fd16c0ca0e0 msgr2=0x7fd16c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:10.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd172639640 1 --2- 172.21.15.148:0/3707354311 >> v2:172.21.15.150:3300/0 conn(0x7fd16c0ca0e0 0x7fd16c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd172639640 1 -- 172.21.15.148:0/3707354311 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c0cad70 con 0x7fd16c0cf160 2023-10-30T13:20:10.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd171e38640 1 --2- 172.21.15.148:0/3707354311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 0x7fd16c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:10.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.039+0000 7fd171637640 1 --2- 172.21.15.148:0/3707354311 >> v2:172.21.15.150:3300/0 conn(0x7fd16c0ca0e0 0x7fd16c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:10.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd172639640 1 --2- 172.21.15.148:0/3707354311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 0x7fd16c0cf590 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0ccd10 tx=0x7fd16401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f81c99705c81b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:10.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd170e36640 1 -- 172.21.15.148:0/3707354311 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1640af400 con 0x7fd16c0cf160 2023-10-30T13:20:10.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd170e36640 1 -- 172.21.15.148:0/3707354311 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1640ab050 con 0x7fd16c0cf160 2023-10-30T13:20:10.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd170e36640 1 -- 172.21.15.148:0/3707354311 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1640af780 con 0x7fd16c0cf160 2023-10-30T13:20:10.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd17409c640 1 -- 172.21.15.148:0/3707354311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 msgr2=0x7fd16c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:10.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd17409c640 1 --2- 172.21.15.148:0/3707354311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 0x7fd16c0cf590 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0ccd10 tx=0x7fd16401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd17409c640 1 -- 172.21.15.148:0/3707354311 shutdown_connections 2023-10-30T13:20:10.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd17409c640 1 --2- 172.21.15.148:0/3707354311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 0x7fd16c0cf590 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd17409c640 1 --2- 172.21.15.148:0/3707354311 >> v2:172.21.15.150:3300/0 conn(0x7fd16c0ca0e0 0x7fd16c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd17409c640 1 --2- 172.21.15.148:0/3707354311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 0x7fd16c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd17409c640 1 -- 172.21.15.148:0/3707354311 >> 172.21.15.148:0/3707354311 conn(0x7fd16c06bfe0 msgr2=0x7fd16c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:10.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd17409c640 1 -- 172.21.15.148:0/3707354311 shutdown_connections 2023-10-30T13:20:10.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.043+0000 7fd17409c640 1 -- 172.21.15.148:0/3707354311 wait complete. 2023-10-30T13:20:10.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 Processor -- start 2023-10-30T13:20:10.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 -- start start 2023-10-30T13:20:10.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd16c0ca0e0 0x7fd16c1efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:10.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 0x7fd16c1e5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:10.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 0x7fd16c1e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:10.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c15afa0 con 0x7fd16c0cf160 2023-10-30T13:20:10.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c15a9e0 con 0x7fd16c0ca0e0 2023-10-30T13:20:10.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c15acc0 con 0x7fd16c1538a0 2023-10-30T13:20:10.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171637640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 0x7fd16c1e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:10.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171637640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 0x7fd16c1e5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42772/0 (socket says 172.21.15.148:42772) 2023-10-30T13:20:10.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171637640 1 -- 172.21.15.148:0/2399658682 learned_addr learned my addr 172.21.15.148:0/2399658682 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:10.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171e38640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd16c0ca0e0 0x7fd16c1efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:10.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd172639640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 0x7fd16c1e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:10.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171637640 1 -- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 msgr2=0x7fd16c1e2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:10.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171637640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 0x7fd16c1e2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171637640 1 -- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd16c0ca0e0 msgr2=0x7fd16c1efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:10.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171637640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd16c0ca0e0 0x7fd16c1efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171637640 1 -- 172.21.15.148:0/2399658682 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164009b30 con 0x7fd16c0cf160 2023-10-30T13:20:10.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd171637640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 0x7fd16c1e5620 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00e810 tx=0x7fd15c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:10.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd1627fc640 1 -- 172.21.15.148:0/2399658682 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c004030 con 0x7fd16c0cf160 2023-10-30T13:20:10.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd1627fc640 1 -- 172.21.15.148:0/2399658682 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c004190 con 0x7fd16c0cf160 2023-10-30T13:20:10.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd1627fc640 1 -- 172.21.15.148:0/2399658682 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c004520 con 0x7fd16c0cf160 2023-10-30T13:20:10.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c1ef080 con 0x7fd16c0cf160 2023-10-30T13:20:10.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.047+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c1ef640 con 0x7fd16c0cf160 2023-10-30T13:20:10.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.051+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16c1ef800 con 0x7fd16c0cf160 2023-10-30T13:20:10.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.051+0000 7fd1627fc640 1 -- 172.21.15.148:0/2399658682 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd15c00c6d0 con 0x7fd16c0cf160 2023-10-30T13:20:10.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.051+0000 7fd1627fc640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1500664a0 0x7fd150068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:10.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.051+0000 7fd1627fc640 1 -- 172.21.15.148:0/2399658682 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd15c083c20 con 0x7fd16c0cf160 2023-10-30T13:20:10.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.051+0000 7fd171e38640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1500664a0 0x7fd150068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:10.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.051+0000 7fd171e38640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1500664a0 0x7fd150068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd16800ba10 tx=0x7fd168008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:10.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.055+0000 7fd1627fc640 1 -- 172.21.15.148:0/2399658682 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd15c050ea0 con 0x7fd16c0cf160 2023-10-30T13:20:10.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd16c0b2a50 con 0x7fd16c0cf160 2023-10-30T13:20:10.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd1627fc640 1 -- 172.21.15.148:0/2399658682 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7fd15c0544c0 con 0x7fd16c0cf160 2023-10-30T13:20:10.236 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:10.236 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:10.237 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:10.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1500664a0 msgr2=0x7fd150068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:10.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1500664a0 0x7fd150068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd16800ba10 tx=0x7fd168008040 comp rx=0 tx=0).stop 2023-10-30T13:20:10.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 msgr2=0x7fd16c1e5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:10.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 0x7fd16c1e5620 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00e810 tx=0x7fd15c00ecd0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 shutdown_connections 2023-10-30T13:20:10.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1500664a0 0x7fd150068960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16c0cf160 0x7fd16c1e5620 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd16c1538a0 0x7fd16c1e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 --2- 172.21.15.148:0/2399658682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd16c0ca0e0 0x7fd16c1efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:10.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 >> 172.21.15.148:0/2399658682 conn(0x7fd16c06bfe0 msgr2=0x7fd16c1526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:10.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 shutdown_connections 2023-10-30T13:20:10.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:10.235+0000 7fd17409c640 1 -- 172.21.15.148:0/2399658682 wait complete. 2023-10-30T13:20:10.257 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:12.258 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:12.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6acb817640 1 Processor -- start 2023-10-30T13:20:12.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6acb817640 1 -- start start 2023-10-30T13:20:12.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6acb817640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6ac40c6b80 0x7f6ac40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:12.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6acb817640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ac40c6f60 0x7f6ac40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:12.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6acb817640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac40c0a50 con 0x7f6ac40c8430 2023-10-30T13:20:12.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6acb817640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac406f6e0 con 0x7f6ac40c6b80 2023-10-30T13:20:12.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6acb817640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac406f9c0 con 0x7f6ac40c6f60 2023-10-30T13:20:12.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6ac95b3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6ac40c6b80 0x7f6ac40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:12.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6ac95b3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6ac40c6b80 0x7f6ac40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49748/0 (socket says 172.21.15.148:49748) 2023-10-30T13:20:12.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6ac95b3640 1 -- 172.21.15.148:0/2738293636 learned_addr learned my addr 172.21.15.148:0/2738293636 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:12.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6ac8db2640 1 --2- 172.21.15.148:0/2738293636 >> v2:172.21.15.150:3300/0 conn(0x7f6ac40c6f60 0x7f6ac40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6ac95b3640 1 -- 172.21.15.148:0/2738293636 >> v2:172.21.15.150:3300/0 conn(0x7f6ac40c6f60 msgr2=0x7f6ac40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6ac95b3640 1 --2- 172.21.15.148:0/2738293636 >> v2:172.21.15.150:3300/0 conn(0x7f6ac40c6f60 0x7f6ac40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6ac95b3640 1 -- 172.21.15.148:0/2738293636 >> v1:172.21.15.150:6789/0 conn(0x7f6ac40c8430 legacy=0x7f6ac413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6ac95b3640 1 -- 172.21.15.148:0/2738293636 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac406fca0 con 0x7f6ac40c6b80 2023-10-30T13:20:12.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.407+0000 7f6ac8db2640 1 --2- 172.21.15.148:0/2738293636 >> v2:172.21.15.150:3300/0 conn(0x7f6ac40c6f60 0x7f6ac40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:12.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6ac95b3640 1 --2- 172.21.15.148:0/2738293636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c6b80 0x7f6ac40bfec0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4159b90 tx=0x7f6ac001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7c8a4ab4c77e5a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:12.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6abbfff640 1 -- 172.21.15.148:0/2738293636 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00ab610 con 0x7f6ac40c6b80 2023-10-30T13:20:12.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6abbfff640 1 -- 172.21.15.148:0/2738293636 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac00ab770 con 0x7f6ac40c6b80 2023-10-30T13:20:12.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6abbfff640 1 -- 172.21.15.148:0/2738293636 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00abb00 con 0x7f6ac40c6b80 2023-10-30T13:20:12.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6acb817640 1 -- 172.21.15.148:0/2738293636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c6b80 msgr2=0x7f6ac40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:12.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6acb817640 1 --2- 172.21.15.148:0/2738293636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c6b80 0x7f6ac40bfec0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4159b90 tx=0x7f6ac001c500 comp rx=0 tx=0).stop 2023-10-30T13:20:12.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6acb817640 1 -- 172.21.15.148:0/2738293636 shutdown_connections 2023-10-30T13:20:12.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6acb817640 1 --2- 172.21.15.148:0/2738293636 >> v2:172.21.15.150:3300/0 conn(0x7f6ac40c6f60 0x7f6ac40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6acb817640 1 --2- 172.21.15.148:0/2738293636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c6b80 0x7f6ac40bfec0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6acb817640 1 -- 172.21.15.148:0/2738293636 >> 172.21.15.148:0/2738293636 conn(0x7f6ac40c7910 msgr2=0x7f6ac40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:12.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6acb817640 1 -- 172.21.15.148:0/2738293636 shutdown_connections 2023-10-30T13:20:12.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.411+0000 7f6acb817640 1 -- 172.21.15.148:0/2738293636 wait complete. 2023-10-30T13:20:12.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 Processor -- start 2023-10-30T13:20:12.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 -- start start 2023-10-30T13:20:12.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6b80 0x7f6ac4155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:12.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac40c6f60 0x7f6ac4161ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:12.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c8430 0x7f6ac4162530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:12.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac41635a0 con 0x7f6ac40c8430 2023-10-30T13:20:12.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac41f2180 con 0x7f6ac40c6f60 2023-10-30T13:20:12.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac41f2460 con 0x7f6ac40c6b80 2023-10-30T13:20:12.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac9db4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c8430 0x7f6ac4162530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:12.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac9db4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c8430 0x7f6ac4162530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49750/0 (socket says 172.21.15.148:49750) 2023-10-30T13:20:12.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac9db4640 1 -- 172.21.15.148:0/10101519 learned_addr learned my addr 172.21.15.148:0/10101519 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:12.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac8db2640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac40c6f60 0x7f6ac4161ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:12.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac95b3640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6b80 0x7f6ac4155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:12.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac9db4640 1 -- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6b80 msgr2=0x7f6ac4155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:12.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac9db4640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6b80 0x7f6ac4155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac9db4640 1 -- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac40c6f60 msgr2=0x7f6ac4161ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:12.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac9db4640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac40c6f60 0x7f6ac4161ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac9db4640 1 -- 172.21.15.148:0/10101519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac00094d0 con 0x7f6ac40c8430 2023-10-30T13:20:12.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac95b3640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6b80 0x7f6ac4155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:12.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ac9db4640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c8430 0x7f6ac4162530 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c730 tx=0x7f6abc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:12.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ab9ffb640 1 -- 172.21.15.148:0/10101519 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc003b80 con 0x7f6ac40c8430 2023-10-30T13:20:12.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ab9ffb640 1 -- 172.21.15.148:0/10101519 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc003ce0 con 0x7f6ac40c8430 2023-10-30T13:20:12.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac41f2740 con 0x7f6ac40c8430 2023-10-30T13:20:12.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6ab9ffb640 1 -- 172.21.15.148:0/10101519 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc00aa30 con 0x7f6ac40c8430 2023-10-30T13:20:12.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac41f2d00 con 0x7f6ac40c8430 2023-10-30T13:20:12.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.415+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac41f2f20 con 0x7f6ac40c8430 2023-10-30T13:20:12.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.419+0000 7f6ab9ffb640 1 -- 172.21.15.148:0/10101519 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6abc00ab90 con 0x7f6ac40c8430 2023-10-30T13:20:12.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.419+0000 7f6ab9ffb640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa80664d0 0x7f6aa8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:12.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.419+0000 7f6ab9ffb640 1 -- 172.21.15.148:0/10101519 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f6abc083e80 con 0x7f6ac40c8430 2023-10-30T13:20:12.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.419+0000 7f6ac95b3640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa80664d0 0x7f6aa8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:12.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.419+0000 7f6ac95b3640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa80664d0 0x7f6aa8068990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00af000 tx=0x7f6ac00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:12.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.423+0000 7f6ab9ffb640 1 -- 172.21.15.148:0/10101519 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6abc050fe0 con 0x7f6ac40c8430 2023-10-30T13:20:12.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ac404e170 con 0x7f6ac40c8430 2023-10-30T13:20:12.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6ab9ffb640 1 -- 172.21.15.148:0/10101519 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7f6abc054600 con 0x7f6ac40c8430 2023-10-30T13:20:12.603 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:12.604 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:12.604 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:12.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa80664d0 msgr2=0x7f6aa8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:12.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa80664d0 0x7f6aa8068990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00af000 tx=0x7f6ac00a7000 comp rx=0 tx=0).stop 2023-10-30T13:20:12.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c8430 msgr2=0x7f6ac4162530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:12.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c8430 0x7f6ac4162530 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c730 tx=0x7f6abc00cbf0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 shutdown_connections 2023-10-30T13:20:12.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa80664d0 0x7f6aa8068990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac40c8430 0x7f6ac4162530 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac40c6f60 0x7f6ac4161ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 --2- 172.21.15.148:0/10101519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6b80 0x7f6ac4155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:12.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 >> 172.21.15.148:0/10101519 conn(0x7f6ac40c7910 msgr2=0x7f6ac40c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:12.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 shutdown_connections 2023-10-30T13:20:12.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:12.603+0000 7f6acb817640 1 -- 172.21.15.148:0/10101519 wait complete. 2023-10-30T13:20:12.624 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:14.626 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:14.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.775+0000 7f9f1f565640 1 Processor -- start 2023-10-30T13:20:14.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.775+0000 7f9f1f565640 1 -- start start 2023-10-30T13:20:14.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1f565640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9f180c6540 0x7f9f1815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:14.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1f565640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f180c5810 con 0x7f9f181580e0 2023-10-30T13:20:14.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1f565640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f180c69b0 con 0x7f9f18070740 2023-10-30T13:20:14.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1f565640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1806acb0 con 0x7f9f180c6540 2023-10-30T13:20:14.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1db02640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9f180c6540 0x7f9f1815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:14.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1d301640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9f181580e0 0x7f9f18069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58716/0 (socket says 172.21.15.148:58716) 2023-10-30T13:20:14.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1d301640 1 -- 172.21.15.148:0/3038440693 learned_addr learned my addr 172.21.15.148:0/3038440693 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:14.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1db02640 1 -- 172.21.15.148:0/3038440693 >> v1:172.21.15.150:6789/0 conn(0x7f9f18070740 legacy=0x7f9f1806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:14.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1db02640 1 -- 172.21.15.148:0/3038440693 >> v1:172.21.15.148:6789/0 conn(0x7f9f181580e0 legacy=0x7f9f18069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:14.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f0ffff640 1 -- 172.21.15.148:0/3038440693 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102058248 0 0) 0x7f9f180c5810 con 0x7f9f181580e0 2023-10-30T13:20:14.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.779+0000 7f9f1db02640 1 -- 172.21.15.148:0/3038440693 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180c6bb0 con 0x7f9f180c6540 2023-10-30T13:20:14.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1db02640 1 --2- 172.21.15.148:0/3038440693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f180c6540 0x7f9f1815e700 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009d90 tx=0x7f9f1001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e727ba0a27066d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:14.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f0ffff640 1 -- 172.21.15.148:0/3038440693 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f100a5920 con 0x7f9f180c6540 2023-10-30T13:20:14.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f0ffff640 1 -- 172.21.15.148:0/3038440693 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f100a5a80 con 0x7f9f180c6540 2023-10-30T13:20:14.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f0ffff640 1 -- 172.21.15.148:0/3038440693 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f100a5e10 con 0x7f9f180c6540 2023-10-30T13:20:14.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1f565640 1 -- 172.21.15.148:0/3038440693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f180c6540 msgr2=0x7f9f1815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:14.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1f565640 1 --2- 172.21.15.148:0/3038440693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f180c6540 0x7f9f1815e700 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009d90 tx=0x7f9f1001c950 comp rx=0 tx=0).stop 2023-10-30T13:20:14.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1f565640 1 -- 172.21.15.148:0/3038440693 shutdown_connections 2023-10-30T13:20:14.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1f565640 1 --2- 172.21.15.148:0/3038440693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f180c6540 0x7f9f1815e700 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:14.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1f565640 1 -- 172.21.15.148:0/3038440693 >> 172.21.15.148:0/3038440693 conn(0x7f9f180c7910 msgr2=0x7f9f180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:14.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1f565640 1 -- 172.21.15.148:0/3038440693 shutdown_connections 2023-10-30T13:20:14.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1f565640 1 -- 172.21.15.148:0/3038440693 wait complete. 2023-10-30T13:20:14.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1f565640 1 Processor -- start 2023-10-30T13:20:14.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.783+0000 7f9f1f565640 1 -- start start 2023-10-30T13:20:14.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1f565640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f18070740 0x7f9f18146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:14.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1f565640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f180c6540 0x7f9f18148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:14.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1f565640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f181580e0 0x7f9f181494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:14.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1f565640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1813c9c0 con 0x7f9f181580e0 2023-10-30T13:20:14.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1f565640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181479d0 con 0x7f9f180c6540 2023-10-30T13:20:14.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1f565640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18147c90 con 0x7f9f18070740 2023-10-30T13:20:14.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1db02640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f181580e0 0x7f9f181494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:14.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1db02640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f181580e0 0x7f9f181494f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49778/0 (socket says 172.21.15.148:49778) 2023-10-30T13:20:14.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1db02640 1 -- 172.21.15.148:0/3302720491 learned_addr learned my addr 172.21.15.148:0/3302720491 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:14.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1cb00640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f180c6540 0x7f9f18148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:14.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1d301640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f18070740 0x7f9f18146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:14.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1db02640 1 -- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f18070740 msgr2=0x7f9f18146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:14.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1db02640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f18070740 0x7f9f18146c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:14.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1db02640 1 -- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f180c6540 msgr2=0x7f9f18148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:14.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1db02640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f180c6540 0x7f9f18148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:14.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1db02640 1 -- 172.21.15.148:0/3302720491 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f10009a70 con 0x7f9f181580e0 2023-10-30T13:20:14.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1db02640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f181580e0 0x7f9f181494f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9f1001a580 tx=0x7f9f100196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:14.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f0dffb640 1 -- 172.21.15.148:0/3302720491 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f100ade40 con 0x7f9f181580e0 2023-10-30T13:20:14.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f0dffb640 1 -- 172.21.15.148:0/3302720491 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f100b9450 con 0x7f9f181580e0 2023-10-30T13:20:14.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f0dffb640 1 -- 172.21.15.148:0/3302720491 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f100b97e0 con 0x7f9f181580e0 2023-10-30T13:20:14.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f18147f70 con 0x7f9f181580e0 2023-10-30T13:20:14.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f18148530 con 0x7f9f181580e0 2023-10-30T13:20:14.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1804e170 con 0x7f9f181580e0 2023-10-30T13:20:14.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.787+0000 7f9f0dffb640 1 -- 172.21.15.148:0/3302720491 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f100b9940 con 0x7f9f181580e0 2023-10-30T13:20:14.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.791+0000 7f9f0dffb640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9efc0664a0 0x7f9efc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:14.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.791+0000 7f9f0dffb640 1 -- 172.21.15.148:0/3302720491 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f9f1011f920 con 0x7f9f181580e0 2023-10-30T13:20:14.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.791+0000 7f9f1d301640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9efc0664a0 0x7f9efc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:14.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.791+0000 7f9f1d301640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9efc0664a0 0x7f9efc068960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9f14000f70 tx=0x7f9f14007d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:14.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.795+0000 7f9f0dffb640 1 -- 172.21.15.148:0/3302720491 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f100ecb40 con 0x7f9f181580e0 2023-10-30T13:20:14.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.975+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f181f7cb0 con 0x7f9f181580e0 2023-10-30T13:20:14.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.975+0000 7f9f0dffb640 1 -- 172.21.15.148:0/3302720491 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7f9f100b2090 con 0x7f9f181580e0 2023-10-30T13:20:14.977 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:14.977 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:14.977 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:14.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9efc0664a0 msgr2=0x7f9efc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:14.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9efc0664a0 0x7f9efc068960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9f14000f70 tx=0x7f9f14007d10 comp rx=0 tx=0).stop 2023-10-30T13:20:14.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f181580e0 msgr2=0x7f9f181494f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:14.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f181580e0 0x7f9f181494f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9f1001a580 tx=0x7f9f100196f0 comp rx=0 tx=0).stop 2023-10-30T13:20:14.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 shutdown_connections 2023-10-30T13:20:14.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9efc0664a0 0x7f9efc068960 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:14.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f181580e0 0x7f9f181494f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:14.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f180c6540 0x7f9f18148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:14.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 --2- 172.21.15.148:0/3302720491 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f18070740 0x7f9f18146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:14.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 >> 172.21.15.148:0/3302720491 conn(0x7f9f180c7910 msgr2=0x7f9f18153980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:14.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 shutdown_connections 2023-10-30T13:20:14.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:14.979+0000 7f9f1f565640 1 -- 172.21.15.148:0/3302720491 wait complete. 2023-10-30T13:20:14.997 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:16.998 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:17.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e8a268640 1 Processor -- start 2023-10-30T13:20:17.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e8a268640 1 -- start start 2023-10-30T13:20:17.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e8a268640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e84153c30 0x7f3e8406a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:17.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e8a268640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3e840b1ea0 0x7f3e840cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:17.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e8a268640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e8406b4a0 0x7f3e840cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:17.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e8a268640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e841596e0 con 0x7f3e840b1ea0 2023-10-30T13:20:17.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e8a268640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8406ade0 con 0x7f3e84153c30 2023-10-30T13:20:17.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e8a268640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e841598f0 con 0x7f3e8406b4a0 2023-10-30T13:20:17.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e837fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3e840b1ea0 0x7f3e840cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:17.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e837fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3e840b1ea0 0x7f3e840cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49780/0 (socket says 172.21.15.148:49780) 2023-10-30T13:20:17.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e837fe640 1 -- 172.21.15.148:0/2219995284 learned_addr learned my addr 172.21.15.148:0/2219995284 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:17.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e83fff640 1 --2- 172.21.15.148:0/2219995284 >> v2:172.21.15.150:3300/0 conn(0x7f3e84153c30 0x7f3e8406a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:17.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e88805640 1 --2- 172.21.15.148:0/2219995284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e8406b4a0 0x7f3e840cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:17.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e837fe640 1 -- 172.21.15.148:0/2219995284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e8406b4a0 msgr2=0x7f3e840cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:17.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e837fe640 1 --2- 172.21.15.148:0/2219995284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e8406b4a0 0x7f3e840cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e837fe640 1 -- 172.21.15.148:0/2219995284 >> v2:172.21.15.150:3300/0 conn(0x7f3e84153c30 msgr2=0x7f3e8406a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:17.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e837fe640 1 --2- 172.21.15.148:0/2219995284 >> v2:172.21.15.150:3300/0 conn(0x7f3e84153c30 0x7f3e8406a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.147+0000 7f3e837fe640 1 -- 172.21.15.148:0/2219995284 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84159b90 con 0x7f3e840b1ea0 2023-10-30T13:20:17.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e837fe640 1 --2- 172.21.15.148:0/2219995284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e840b1ea0 0x7f3e840cb9a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3e84067610 tx=0x7f3e7801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb44334a0809eed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:17.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e82ffd640 1 -- 172.21.15.148:0/2219995284 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e780a89b0 con 0x7f3e840b1ea0 2023-10-30T13:20:17.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e82ffd640 1 -- 172.21.15.148:0/2219995284 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e780a8b10 con 0x7f3e840b1ea0 2023-10-30T13:20:17.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e82ffd640 1 -- 172.21.15.148:0/2219995284 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e780a8ea0 con 0x7f3e840b1ea0 2023-10-30T13:20:17.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e8a268640 1 -- 172.21.15.148:0/2219995284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e840b1ea0 msgr2=0x7f3e840cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:17.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e8a268640 1 --2- 172.21.15.148:0/2219995284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e840b1ea0 0x7f3e840cb9a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3e84067610 tx=0x7f3e7801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e8a268640 1 -- 172.21.15.148:0/2219995284 shutdown_connections 2023-10-30T13:20:17.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e8a268640 1 --2- 172.21.15.148:0/2219995284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e8406b4a0 0x7f3e840cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e8a268640 1 --2- 172.21.15.148:0/2219995284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e840b1ea0 0x7f3e840cb9a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e8a268640 1 --2- 172.21.15.148:0/2219995284 >> v2:172.21.15.150:3300/0 conn(0x7f3e84153c30 0x7f3e8406a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e8a268640 1 -- 172.21.15.148:0/2219995284 >> 172.21.15.148:0/2219995284 conn(0x7f3e8406bf40 msgr2=0x7f3e8406c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:17.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e8a268640 1 -- 172.21.15.148:0/2219995284 shutdown_connections 2023-10-30T13:20:17.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.151+0000 7f3e8a268640 1 -- 172.21.15.148:0/2219995284 wait complete. 2023-10-30T13:20:17.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 Processor -- start 2023-10-30T13:20:17.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 -- start start 2023-10-30T13:20:17.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e8406b4a0 0x7f3e840cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:17.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e840b1ea0 0x7f3e840ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:17.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e84153c30 0x7f3e84151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:17.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84151230 con 0x7f3e8406b4a0 2023-10-30T13:20:17.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e841456a0 con 0x7f3e84153c30 2023-10-30T13:20:17.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84151a00 con 0x7f3e840b1ea0 2023-10-30T13:20:17.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e83fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e8406b4a0 0x7f3e840cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:17.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e83fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e8406b4a0 0x7f3e840cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49796/0 (socket says 172.21.15.148:49796) 2023-10-30T13:20:17.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e83fff640 1 -- 172.21.15.148:0/606065143 learned_addr learned my addr 172.21.15.148:0/606065143 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:17.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e837fe640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e840b1ea0 0x7f3e840ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:17.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e88805640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e84153c30 0x7f3e84151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:17.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e83fff640 1 -- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e840b1ea0 msgr2=0x7f3e840ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:17.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e83fff640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e840b1ea0 0x7f3e840ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e83fff640 1 -- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e84153c30 msgr2=0x7f3e84151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:17.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e83fff640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e84153c30 0x7f3e84151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e83fff640 1 -- 172.21.15.148:0/606065143 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e78009480 con 0x7f3e8406b4a0 2023-10-30T13:20:17.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e837fe640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e840b1ea0 0x7f3e840ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:17.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e88805640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e84153c30 0x7f3e84151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:17.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e83fff640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e8406b4a0 0x7f3e840cde40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400e870 tx=0x7f3e7400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:17.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e80ff9640 1 -- 172.21.15.148:0/606065143 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e74009510 con 0x7f3e8406b4a0 2023-10-30T13:20:17.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e80ff9640 1 -- 172.21.15.148:0/606065143 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7400eec0 con 0x7f3e8406b4a0 2023-10-30T13:20:17.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e80ff9640 1 -- 172.21.15.148:0/606065143 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7400cb40 con 0x7f3e8406b4a0 2023-10-30T13:20:17.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e84151440 con 0x7f3e8406b4a0 2023-10-30T13:20:17.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.155+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e84145330 con 0x7f3e8406b4a0 2023-10-30T13:20:17.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.159+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8404e170 con 0x7f3e8406b4a0 2023-10-30T13:20:17.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.159+0000 7f3e80ff9640 1 -- 172.21.15.148:0/606065143 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e7400cca0 con 0x7f3e8406b4a0 2023-10-30T13:20:17.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.159+0000 7f3e80ff9640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e700664a0 0x7f3e70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:17.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.159+0000 7f3e80ff9640 1 -- 172.21.15.148:0/606065143 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f3e740841a0 con 0x7f3e8406b4a0 2023-10-30T13:20:17.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.159+0000 7f3e837fe640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e700664a0 0x7f3e70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:17.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.159+0000 7f3e837fe640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e700664a0 0x7f3e70068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3e780b2000 tx=0x7f3e780aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:17.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.163+0000 7f3e80ff9640 1 -- 172.21.15.148:0/606065143 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e74051390 con 0x7f3e8406b4a0 2023-10-30T13:20:17.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e840c9440 con 0x7f3e8406b4a0 2023-10-30T13:20:17.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e80ff9640 1 -- 172.21.15.148:0/606065143 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7f3e740549b0 con 0x7f3e8406b4a0 2023-10-30T13:20:17.344 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:17.344 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:17.345 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:17.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e700664a0 msgr2=0x7f3e70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:17.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e700664a0 0x7f3e70068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3e780b2000 tx=0x7f3e780aa070 comp rx=0 tx=0).stop 2023-10-30T13:20:17.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e8406b4a0 msgr2=0x7f3e840cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:17.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e8406b4a0 0x7f3e840cde40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400e870 tx=0x7f3e7400ec70 comp rx=0 tx=0).stop 2023-10-30T13:20:17.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 shutdown_connections 2023-10-30T13:20:17.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e700664a0 0x7f3e70068960 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e84153c30 0x7f3e84151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e840b1ea0 0x7f3e840ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 --2- 172.21.15.148:0/606065143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e8406b4a0 0x7f3e840cde40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:17.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 >> 172.21.15.148:0/606065143 conn(0x7f3e8406bf40 msgr2=0x7f3e84158ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:17.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 shutdown_connections 2023-10-30T13:20:17.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:17.343+0000 7f3e8a268640 1 -- 172.21.15.148:0/606065143 wait complete. 2023-10-30T13:20:17.366 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:19.366 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:19.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.511+0000 7f8c0326b640 1 Processor -- start 2023-10-30T13:20:19.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.511+0000 7f8c0326b640 1 -- start start 2023-10-30T13:20:19.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.511+0000 7f8c0326b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8bfc0c2570 0x7f8bfc0cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:19.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.511+0000 7f8c0326b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0c68b0 con 0x7f8bfc150920 2023-10-30T13:20:19.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.511+0000 7f8c0326b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc150d00 con 0x7f8bfc0c29d0 2023-10-30T13:20:19.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.511+0000 7f8c0326b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bfc0c2e90 con 0x7f8bfc0c2570 2023-10-30T13:20:19.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c01007640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8bfc150920 0x7f8bfc06d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58718/0 (socket says 172.21.15.148:58718) 2023-10-30T13:20:19.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c01007640 1 -- 172.21.15.148:0/4060502525 learned_addr learned my addr 172.21.15.148:0/4060502525 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:19.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c00806640 1 --2- 172.21.15.148:0/4060502525 >> v2:172.21.15.150:3300/0 conn(0x7f8bfc0c2570 0x7f8bfc0cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:19.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034931724 0 0) 0x7f8bfc0c68b0 con 0x7f8bfc150920 2023-10-30T13:20:19.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be8005740 con 0x7f8bfc150920 2023-10-30T13:20:19.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 343032265 0 0) 0x7f8be8005740 con 0x7f8bfc150920 2023-10-30T13:20:19.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 >> v2:172.21.15.150:3300/0 conn(0x7f8bfc0c2570 msgr2=0x7f8bfc0cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:19.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 --2- 172.21.15.148:0/4060502525 >> v2:172.21.15.150:3300/0 conn(0x7f8bfc0c2570 0x7f8bfc0cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 >> v1:172.21.15.150:6789/0 conn(0x7f8bfc0c29d0 legacy=0x7f8bfc156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:19.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0c6b00 con 0x7f8bfc150920 2023-10-30T13:20:19.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f8bf8001680 con 0x7f8bfc150920 2023-10-30T13:20:19.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 >> v1:172.21.15.148:6789/0 conn(0x7f8bfc150920 legacy=0x7f8bfc06d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:19.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be8005b90 0x7f8be8005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:19.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80069e0 0x7f8be8006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:19.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80073a0 0x7f8be800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:19.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0c2e90 con 0x7f8be80073a0 2023-10-30T13:20:19.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be800e240 con 0x7f8be80069e0 2023-10-30T13:20:19.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be800e400 con 0x7f8be8005b90 2023-10-30T13:20:19.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c01808640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80073a0 0x7f8be800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:19.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c00806640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80069e0 0x7f8be8006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:19.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c01007640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be8005b90 0x7f8be8005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:19.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c01808640 1 -- 172.21.15.148:0/4060502525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be8005b90 msgr2=0x7f8be8005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:19.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c01808640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be8005b90 0x7f8be8005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c01808640 1 -- 172.21.15.148:0/4060502525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80069e0 msgr2=0x7f8be8006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:19.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c01808640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80069e0 0x7f8be8006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.515+0000 7f8c01808640 1 -- 172.21.15.148:0/4060502525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be8005740 con 0x7f8be80073a0 2023-10-30T13:20:19.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c01808640 1 -- 172.21.15.148:0/4060502525 reap_dead start 2023-10-30T13:20:19.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c01808640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80073a0 0x7f8be800dba0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc06db50 tx=0x7f8bf401c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca412515ccdb126 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:19.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4003980 con 0x7f8be80073a0 2023-10-30T13:20:19.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf4003ae0 con 0x7f8be80073a0 2023-10-30T13:20:19.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8bf3fff640 1 -- 172.21.15.148:0/4060502525 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4003e90 con 0x7f8be80073a0 2023-10-30T13:20:19.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c0326b640 1 -- 172.21.15.148:0/4060502525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80073a0 msgr2=0x7f8be800dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:19.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c0326b640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80073a0 0x7f8be800dba0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc06db50 tx=0x7f8bf401c700 comp rx=0 tx=0).stop 2023-10-30T13:20:19.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c0326b640 1 -- 172.21.15.148:0/4060502525 shutdown_connections 2023-10-30T13:20:19.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c0326b640 1 --2- 172.21.15.148:0/4060502525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8be80073a0 0x7f8be800dba0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c0326b640 1 -- 172.21.15.148:0/4060502525 >> 172.21.15.148:0/4060502525 conn(0x7f8bfc06dfd0 msgr2=0x7f8bfc06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:19.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c0326b640 1 -- 172.21.15.148:0/4060502525 shutdown_connections 2023-10-30T13:20:19.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c0326b640 1 -- 172.21.15.148:0/4060502525 wait complete. 2023-10-30T13:20:19.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c0326b640 1 Processor -- start 2023-10-30T13:20:19.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.519+0000 7f8c0326b640 1 -- start start 2023-10-30T13:20:19.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c0326b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be80073a0 0x7f8bfc1eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:19.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c0326b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bfc071ba0 0x7f8bfc151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:19.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c0326b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bfc071fe0 0x7f8bfc0c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:19.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c0326b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1ee7a0 con 0x7f8bfc071ba0 2023-10-30T13:20:19.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c0326b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1eb710 con 0x7f8bfc071fe0 2023-10-30T13:20:19.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c0326b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1500b0 con 0x7f8be80073a0 2023-10-30T13:20:19.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c00806640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bfc071ba0 0x7f8bfc151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:19.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c00806640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bfc071ba0 0x7f8bfc151780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49804/0 (socket says 172.21.15.148:49804) 2023-10-30T13:20:19.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c00806640 1 -- 172.21.15.148:0/735064612 learned_addr learned my addr 172.21.15.148:0/735064612 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:19.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c01007640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be80073a0 0x7f8bfc1eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:19.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c01808640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bfc071fe0 0x7f8bfc0c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:19.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c00806640 1 -- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be80073a0 msgr2=0x7f8bfc1eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:19.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c00806640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be80073a0 0x7f8bfc1eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c00806640 1 -- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bfc071fe0 msgr2=0x7f8bfc0c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:19.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c00806640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bfc071fe0 0x7f8bfc0c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c00806640 1 -- 172.21.15.148:0/735064612 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc14fdd0 con 0x7f8bfc071ba0 2023-10-30T13:20:19.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c01007640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be80073a0 0x7f8bfc1eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:19.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c01808640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bfc071fe0 0x7f8bfc0c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:19.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c00806640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bfc071ba0 0x7f8bfc151780 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8bec011960 tx=0x7f8bec011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:19.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8bf1ffb640 1 -- 172.21.15.148:0/735064612 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec00faf0 con 0x7f8bfc071ba0 2023-10-30T13:20:19.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8bf1ffb640 1 -- 172.21.15.148:0/735064612 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec00fc50 con 0x7f8bfc071ba0 2023-10-30T13:20:19.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8bf1ffb640 1 -- 172.21.15.148:0/735064612 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec014440 con 0x7f8bfc071ba0 2023-10-30T13:20:19.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc1ede00 con 0x7f8bfc071ba0 2023-10-30T13:20:19.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc1ee3c0 con 0x7f8bfc071ba0 2023-10-30T13:20:19.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc04e170 con 0x7f8bfc071ba0 2023-10-30T13:20:19.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.523+0000 7f8bf1ffb640 1 -- 172.21.15.148:0/735064612 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8bec0145a0 con 0x7f8bfc071ba0 2023-10-30T13:20:19.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.527+0000 7f8bf1ffb640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8be00664a0 0x7f8be0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:19.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.527+0000 7f8bf1ffb640 1 -- 172.21.15.148:0/735064612 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f8bec084910 con 0x7f8bfc071ba0 2023-10-30T13:20:19.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.527+0000 7f8c01007640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8be00664a0 0x7f8be0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:19.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.527+0000 7f8c01007640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8be00664a0 0x7f8be0068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8001060 tx=0x7f8bf8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:19.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.531+0000 7f8bf1ffb640 1 -- 172.21.15.148:0/735064612 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8bec051b00 con 0x7f8bfc071ba0 2023-10-30T13:20:19.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.707+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bfc0ca650 con 0x7f8bfc071ba0 2023-10-30T13:20:19.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8bf1ffb640 1 -- 172.21.15.148:0/735064612 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7f8bec055120 con 0x7f8bfc071ba0 2023-10-30T13:20:19.711 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:19.711 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:19.712 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:19.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8be00664a0 msgr2=0x7f8be0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:19.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8be00664a0 0x7f8be0068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8001060 tx=0x7f8bf8009040 comp rx=0 tx=0).stop 2023-10-30T13:20:19.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bfc071ba0 msgr2=0x7f8bfc151780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:19.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bfc071ba0 0x7f8bfc151780 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8bec011960 tx=0x7f8bec011e20 comp rx=0 tx=0).stop 2023-10-30T13:20:19.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 shutdown_connections 2023-10-30T13:20:19.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8be00664a0 0x7f8be0068960 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bfc071ba0 0x7f8bfc151780 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bfc071fe0 0x7f8bfc0c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 --2- 172.21.15.148:0/735064612 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8be80073a0 0x7f8bfc1eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:19.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 >> 172.21.15.148:0/735064612 conn(0x7f8bfc06dfd0 msgr2=0x7f8bfc144a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:19.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 shutdown_connections 2023-10-30T13:20:19.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:19.711+0000 7f8c0326b640 1 -- 172.21.15.148:0/735064612 wait complete. 2023-10-30T13:20:19.731 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:20:19.732 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4959": { "addr": "172.21.15.148:6837/95380877", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 95380877, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 95380877, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4959, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:stopping", "state_seq": 16 }, "gid_5033": { "addr": "172.21.15.150:6836/953984429", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 953984429, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 953984429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5033, "incarnation": 36, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:20:06.522682+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5033, "mds_1": 5103, "mds_2": 4959 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:20:21.733 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:21.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.879+0000 7fcd5afc4640 1 Processor -- start 2023-10-30T13:20:21.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd5afc4640 1 -- start start 2023-10-30T13:20:21.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd5afc4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcd540c6b80 0x7fcd540bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:21.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd5afc4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd5413f4a0 0x7fcd540c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:21.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd5afc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd540c0a50 con 0x7fcd540c6f60 2023-10-30T13:20:21.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd5afc4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd5406f6e0 con 0x7fcd540c6b80 2023-10-30T13:20:21.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd5afc4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5406f9c0 con 0x7fcd5413f4a0 2023-10-30T13:20:21.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd4bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fcd540c6f60 0x7fcd5413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44744/0 (socket says 172.21.15.148:44744) 2023-10-30T13:20:21.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd4bfff640 1 -- 172.21.15.148:0/597225613 learned_addr learned my addr 172.21.15.148:0/597225613 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:21.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd58d60640 1 --2- 172.21.15.148:0/597225613 >> v2:172.21.15.150:3300/0 conn(0x7fcd540c6b80 0x7fcd540bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:21.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd59561640 1 --2- 172.21.15.148:0/597225613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd5413f4a0 0x7fcd540c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:21.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd4b7fe640 1 -- 172.21.15.148:0/597225613 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488357228 0 0) 0x7fcd540c0a50 con 0x7fcd540c6f60 2023-10-30T13:20:21.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd4b7fe640 1 -- 172.21.15.148:0/597225613 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd3c003530 con 0x7fcd540c6f60 2023-10-30T13:20:21.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd58d60640 1 -- 172.21.15.148:0/597225613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd5413f4a0 msgr2=0x7fcd540c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:21.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd58d60640 1 --2- 172.21.15.148:0/597225613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd5413f4a0 0x7fcd540c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:21.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd58d60640 1 -- 172.21.15.148:0/597225613 >> v1:172.21.15.148:6789/0 conn(0x7fcd540c6f60 legacy=0x7fcd5413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:21.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.883+0000 7fcd58d60640 1 -- 172.21.15.148:0/597225613 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5406fca0 con 0x7fcd540c6b80 2023-10-30T13:20:21.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd58d60640 1 --2- 172.21.15.148:0/597225613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540c6b80 0x7fcd540bfec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcd500097c0 tx=0x7fcd5001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6cfc875cbaa411 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:21.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd4b7fe640 1 -- 172.21.15.148:0/597225613 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd500ad640 con 0x7fcd540c6b80 2023-10-30T13:20:21.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd4b7fe640 1 -- 172.21.15.148:0/597225613 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd500ad7a0 con 0x7fcd540c6b80 2023-10-30T13:20:21.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd4b7fe640 1 -- 172.21.15.148:0/597225613 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd500adb10 con 0x7fcd540c6b80 2023-10-30T13:20:21.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd5afc4640 1 -- 172.21.15.148:0/597225613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540c6b80 msgr2=0x7fcd540bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:21.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd5afc4640 1 --2- 172.21.15.148:0/597225613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540c6b80 0x7fcd540bfec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcd500097c0 tx=0x7fcd5001c480 comp rx=0 tx=0).stop 2023-10-30T13:20:21.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd5afc4640 1 -- 172.21.15.148:0/597225613 shutdown_connections 2023-10-30T13:20:21.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd5afc4640 1 --2- 172.21.15.148:0/597225613 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd5413f4a0 0x7fcd540c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:21.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd5afc4640 1 --2- 172.21.15.148:0/597225613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540c6b80 0x7fcd540bfec0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:21.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd5afc4640 1 -- 172.21.15.148:0/597225613 >> 172.21.15.148:0/597225613 conn(0x7fcd540c7910 msgr2=0x7fcd540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:21.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd5afc4640 1 -- 172.21.15.148:0/597225613 shutdown_connections 2023-10-30T13:20:21.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd5afc4640 1 -- 172.21.15.148:0/597225613 wait complete. 2023-10-30T13:20:21.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.887+0000 7fcd5afc4640 1 Processor -- start 2023-10-30T13:20:21.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 -- start start 2023-10-30T13:20:21.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd540c6b80 0x7fcd540ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:21.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd540c6f60 0x7fcd54155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:21.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd5413f4a0 0x7fcd5406a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:21.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd54158c60 con 0x7fcd540c6f60 2023-10-30T13:20:21.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd540cab30 con 0x7fcd5413f4a0 2023-10-30T13:20:21.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd541f1ae0 con 0x7fcd540c6b80 2023-10-30T13:20:21.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd4bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd540c6f60 0x7fcd54155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:21.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd4bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd540c6f60 0x7fcd54155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52338/0 (socket says 172.21.15.148:52338) 2023-10-30T13:20:21.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd4bfff640 1 -- 172.21.15.148:0/430268311 learned_addr learned my addr 172.21.15.148:0/430268311 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:21.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd58d60640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd540c6b80 0x7fcd540ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:21.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd59561640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd5413f4a0 0x7fcd5406a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:21.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd4bfff640 1 -- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd540c6b80 msgr2=0x7fcd540ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:21.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd4bfff640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd540c6b80 0x7fcd540ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:21.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd4bfff640 1 -- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd5413f4a0 msgr2=0x7fcd5406a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:21.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd4bfff640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd5413f4a0 0x7fcd5406a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:21.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd4bfff640 1 -- 172.21.15.148:0/430268311 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd50009420 con 0x7fcd540c6f60 2023-10-30T13:20:21.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd4bfff640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd540c6f60 0x7fcd54155e70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400c450 tx=0x7fcd4400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:21.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd497fa640 1 -- 172.21.15.148:0/430268311 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44005cf0 con 0x7fcd540c6f60 2023-10-30T13:20:21.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd497fa640 1 -- 172.21.15.148:0/430268311 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd44005e50 con 0x7fcd540c6f60 2023-10-30T13:20:21.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd541f1dc0 con 0x7fcd540c6f60 2023-10-30T13:20:21.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd497fa640 1 -- 172.21.15.148:0/430268311 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd44007290 con 0x7fcd540c6f60 2023-10-30T13:20:21.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd541f2380 con 0x7fcd540c6f60 2023-10-30T13:20:21.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.891+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5404e170 con 0x7fcd540c6f60 2023-10-30T13:20:21.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.895+0000 7fcd497fa640 1 -- 172.21.15.148:0/430268311 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd440073f0 con 0x7fcd540c6f60 2023-10-30T13:20:21.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.895+0000 7fcd497fa640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd400664d0 0x7fcd40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:21.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.895+0000 7fcd497fa640 1 -- 172.21.15.148:0/430268311 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fcd440835e0 con 0x7fcd540c6f60 2023-10-30T13:20:21.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.895+0000 7fcd58d60640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd400664d0 0x7fcd40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:21.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.895+0000 7fcd58d60640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd400664d0 0x7fcd40068990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcd500af000 tx=0x7fcd500a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:21.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:21.899+0000 7fcd497fa640 1 -- 172.21.15.148:0/430268311 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd44050880 con 0x7fcd540c6f60 2023-10-30T13:20:22.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.079+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd540c7e20 con 0x7fcd540c6f60 2023-10-30T13:20:22.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.079+0000 7fcd497fa640 1 -- 172.21.15.148:0/430268311 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7fcd44017090 con 0x7fcd540c6f60 2023-10-30T13:20:22.080 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:22.081 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:22.081 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:22.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.079+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd400664d0 msgr2=0x7fcd40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:22.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.079+0000 7fcd5afc4640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd400664d0 0x7fcd40068990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcd500af000 tx=0x7fcd500a7070 comp rx=0 tx=0).stop 2023-10-30T13:20:22.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.079+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd540c6f60 msgr2=0x7fcd54155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:22.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.079+0000 7fcd5afc4640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd540c6f60 0x7fcd54155e70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400c450 tx=0x7fcd4400c910 comp rx=0 tx=0).stop 2023-10-30T13:20:22.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.083+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 shutdown_connections 2023-10-30T13:20:22.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.083+0000 7fcd5afc4640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd400664d0 0x7fcd40068990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:22.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.083+0000 7fcd5afc4640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd540c6f60 0x7fcd54155e70 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:22.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.083+0000 7fcd5afc4640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd5413f4a0 0x7fcd5406a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:22.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.083+0000 7fcd5afc4640 1 --2- 172.21.15.148:0/430268311 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd540c6b80 0x7fcd540ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:22.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.083+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 >> 172.21.15.148:0/430268311 conn(0x7fcd540c7910 msgr2=0x7fcd540c5810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:22.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.083+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 shutdown_connections 2023-10-30T13:20:22.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:22.083+0000 7fcd5afc4640 1 -- 172.21.15.148:0/430268311 wait complete. 2023-10-30T13:20:22.101 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:24.103 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:24.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6df53e640 1 Processor -- start 2023-10-30T13:20:24.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6df53e640 1 -- start start 2023-10-30T13:20:24.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6df53e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6d80cf160 0x7fa6d80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:24.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6df53e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80cb800 con 0x7fa6d815d6f0 2023-10-30T13:20:24.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6df53e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80cb330 con 0x7fa6d81538a0 2023-10-30T13:20:24.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6df53e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d80cad70 con 0x7fa6d80cf160 2023-10-30T13:20:24.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6dcad9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa6d815d6f0 0x7fa6d815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44760/0 (socket says 172.21.15.148:44760) 2023-10-30T13:20:24.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6dcad9640 1 -- 172.21.15.148:0/1128360799 learned_addr learned my addr 172.21.15.148:0/1128360799 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:24.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6ddadb640 1 --2- 172.21.15.148:0/1128360799 >> v2:172.21.15.150:3300/0 conn(0x7fa6d80cf160 0x7fa6d80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:24.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682376671 0 0) 0x7fa6d80cb800 con 0x7fa6d815d6f0 2023-10-30T13:20:24.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c4003530 con 0x7fa6d815d6f0 2023-10-30T13:20:24.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2579978745 0 0) 0x7fa6c4003530 con 0x7fa6d815d6f0 2023-10-30T13:20:24.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 >> v2:172.21.15.150:3300/0 conn(0x7fa6d80cf160 msgr2=0x7fa6d80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:24.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 --2- 172.21.15.148:0/1128360799 >> v2:172.21.15.150:3300/0 conn(0x7fa6d80cf160 0x7fa6d80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 >> v1:172.21.15.150:6789/0 conn(0x7fa6d81538a0 legacy=0x7fa6d80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:24.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80cb050 con 0x7fa6d815d6f0 2023-10-30T13:20:24.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fa6c8001680 con 0x7fa6d815d6f0 2023-10-30T13:20:24.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 >> v1:172.21.15.148:6789/0 conn(0x7fa6d815d6f0 legacy=0x7fa6d815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:24.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c4003980 0x7fa6c4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:24.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6c4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:24.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c4005190 0x7fa6c400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:24.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d80cad70 con 0x7fa6c40047d0 2023-10-30T13:20:24.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c400c040 con 0x7fa6c4003980 2023-10-30T13:20:24.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c400c200 con 0x7fa6c4005190 2023-10-30T13:20:24.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.251+0000 7fa6dcad9640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6c4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:24.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6dd2da640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c4003980 0x7fa6c4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:24.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6ddadb640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c4005190 0x7fa6c400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:24.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6dcad9640 1 -- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c4005190 msgr2=0x7fa6c400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:24.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6dcad9640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c4005190 0x7fa6c400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6dcad9640 1 -- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c4003980 msgr2=0x7fa6c4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:24.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6dcad9640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c4003980 0x7fa6c4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6dcad9640 1 -- 172.21.15.148:0/1128360799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4003530 con 0x7fa6c40047d0 2023-10-30T13:20:24.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6dd2da640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c4003980 0x7fa6c4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:24.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6ddadb640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c4005190 0x7fa6c400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:20:24.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6ddadb640 1 -- 172.21.15.148:0/1128360799 reap_dead start 2023-10-30T13:20:24.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6dcad9640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6c4004c50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c240 tx=0x7fa6c801cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520382e6810124da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:24.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8005ae0 con 0x7fa6c40047d0 2023-10-30T13:20:24.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8005c40 con 0x7fa6c40047d0 2023-10-30T13:20:24.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6cffff640 1 -- 172.21.15.148:0/1128360799 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c80a5a80 con 0x7fa6c40047d0 2023-10-30T13:20:24.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6df53e640 1 -- 172.21.15.148:0/1128360799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 msgr2=0x7fa6c4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:24.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6df53e640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6c4004c50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c240 tx=0x7fa6c801cb80 comp rx=0 tx=0).stop 2023-10-30T13:20:24.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6df53e640 1 -- 172.21.15.148:0/1128360799 shutdown_connections 2023-10-30T13:20:24.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6df53e640 1 --2- 172.21.15.148:0/1128360799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6c4004c50 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.255+0000 7fa6df53e640 1 -- 172.21.15.148:0/1128360799 >> 172.21.15.148:0/1128360799 conn(0x7fa6d806bfe0 msgr2=0x7fa6d806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:24.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 -- 172.21.15.148:0/1128360799 shutdown_connections 2023-10-30T13:20:24.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 -- 172.21.15.148:0/1128360799 wait complete. 2023-10-30T13:20:24.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 Processor -- start 2023-10-30T13:20:24.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 -- start start 2023-10-30T13:20:24.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6d81591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:24.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6d8073610 0x7fa6d81e4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:24.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d80b3040 0x7fa6d8158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:24.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d81e3ba0 con 0x7fa6c40047d0 2023-10-30T13:20:24.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d815b750 con 0x7fa6d80b3040 2023-10-30T13:20:24.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d80b34c0 con 0x7fa6d8073610 2023-10-30T13:20:24.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dd2da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6d81591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:24.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dd2da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6d81591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52346/0 (socket says 172.21.15.148:52346) 2023-10-30T13:20:24.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dd2da640 1 -- 172.21.15.148:0/2057390895 learned_addr learned my addr 172.21.15.148:0/2057390895 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:24.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dcad9640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6d8073610 0x7fa6d81e4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:24.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6ddadb640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d80b3040 0x7fa6d8158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:24.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dd2da640 1 -- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6d8073610 msgr2=0x7fa6d81e4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:24.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dd2da640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6d8073610 0x7fa6d81e4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dd2da640 1 -- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d80b3040 msgr2=0x7fa6d8158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:24.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dd2da640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d80b3040 0x7fa6d8158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dd2da640 1 -- 172.21.15.148:0/2057390895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d815b9c0 con 0x7fa6c40047d0 2023-10-30T13:20:24.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6dd2da640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6d81591b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400e4d0 tx=0x7fa6d400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:24.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6cdffb640 1 -- 172.21.15.148:0/2057390895 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d40052e0 con 0x7fa6c40047d0 2023-10-30T13:20:24.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6cdffb640 1 -- 172.21.15.148:0/2057390895 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d4005440 con 0x7fa6c40047d0 2023-10-30T13:20:24.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6cdffb640 1 -- 172.21.15.148:0/2057390895 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4003820 con 0x7fa6c40047d0 2023-10-30T13:20:24.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.259+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d815bca0 con 0x7fa6c40047d0 2023-10-30T13:20:24.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.263+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d81e3830 con 0x7fa6c40047d0 2023-10-30T13:20:24.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.263+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d81e39f0 con 0x7fa6c40047d0 2023-10-30T13:20:24.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.263+0000 7fa6cdffb640 1 -- 172.21.15.148:0/2057390895 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6d4003980 con 0x7fa6c40047d0 2023-10-30T13:20:24.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.263+0000 7fa6cdffb640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6bc0664d0 0x7fa6bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:24.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.263+0000 7fa6cdffb640 1 -- 172.21.15.148:0/2057390895 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fa6d4083f00 con 0x7fa6c40047d0 2023-10-30T13:20:24.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.263+0000 7fa6dcad9640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6bc0664d0 0x7fa6bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:24.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.263+0000 7fa6dcad9640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6bc0664d0 0x7fa6bc068990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80ab000 tx=0x7fa6c80b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:24.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.267+0000 7fa6cdffb640 1 -- 172.21.15.148:0/2057390895 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6d4051060 con 0x7fa6c40047d0 2023-10-30T13:20:24.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.447+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6d804e170 con 0x7fa6c40047d0 2023-10-30T13:20:24.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.447+0000 7fa6cdffb640 1 -- 172.21.15.148:0/2057390895 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7fa6d4054680 con 0x7fa6c40047d0 2023-10-30T13:20:24.450 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:24.450 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:24.451 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:24.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6bc0664d0 msgr2=0x7fa6bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:24.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6bc0664d0 0x7fa6bc068990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80ab000 tx=0x7fa6c80b1040 comp rx=0 tx=0).stop 2023-10-30T13:20:24.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 msgr2=0x7fa6d81591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:24.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6d81591b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400e4d0 tx=0x7fa6d400e990 comp rx=0 tx=0).stop 2023-10-30T13:20:24.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 shutdown_connections 2023-10-30T13:20:24.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6bc0664d0 0x7fa6bc068990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d80b3040 0x7fa6d8158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6d8073610 0x7fa6d81e4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 --2- 172.21.15.148:0/2057390895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c40047d0 0x7fa6d81591b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:24.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 >> 172.21.15.148:0/2057390895 conn(0x7fa6d806bfe0 msgr2=0x7fa6d80c4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:24.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 shutdown_connections 2023-10-30T13:20:24.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:24.451+0000 7fa6df53e640 1 -- 172.21.15.148:0/2057390895 wait complete. 2023-10-30T13:20:24.471 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:24.752 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:20:24.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:20:26.472 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:26.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.619+0000 7f01bcfc5640 1 Processor -- start 2023-10-30T13:20:26.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.619+0000 7f01bcfc5640 1 -- start start 2023-10-30T13:20:26.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01bcfc5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01b80c05d0 0x7f01b813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:26.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01bcfc5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b80c0a50 con 0x7f01b80c6b80 2023-10-30T13:20:26.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01bcfc5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b806f6e0 con 0x7f01b80c6f60 2023-10-30T13:20:26.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01bcfc5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b806f9c0 con 0x7f01b80c05d0 2023-10-30T13:20:26.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f01b80c6b80 0x7f01b80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44772/0 (socket says 172.21.15.148:44772) 2023-10-30T13:20:26.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b659c640 1 -- 172.21.15.148:0/2019685938 learned_addr learned my addr 172.21.15.148:0/2019685938 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:26.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01add9b640 1 --2- 172.21.15.148:0/2019685938 >> v2:172.21.15.150:3300/0 conn(0x7f01b80c05d0 0x7f01b813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:26.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958911617 0 0) 0x7f01b80c0a50 con 0x7f01b80c6b80 2023-10-30T13:20:26.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a4003530 con 0x7f01b80c6b80 2023-10-30T13:20:26.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834107155 0 0) 0x7f01a4003530 con 0x7f01b80c6b80 2023-10-30T13:20:26.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 >> v2:172.21.15.150:3300/0 conn(0x7f01b80c05d0 msgr2=0x7f01b813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:26.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 --2- 172.21.15.148:0/2019685938 >> v2:172.21.15.150:3300/0 conn(0x7f01b80c05d0 0x7f01b813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 >> v1:172.21.15.150:6789/0 conn(0x7f01b80c6f60 legacy=0x7f01b81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:26.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b806fca0 con 0x7f01b80c6b80 2023-10-30T13:20:26.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f01a8001680 con 0x7f01b80c6b80 2023-10-30T13:20:26.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 >> v1:172.21.15.148:6789/0 conn(0x7f01b80c6b80 legacy=0x7f01b80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:26.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a4003980 0x7f01a4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:26.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a40047d0 0x7f01a4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:26.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a4005190 0x7f01a400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:26.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b806f9c0 con 0x7f01a40047d0 2023-10-30T13:20:26.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a400c030 con 0x7f01a4003980 2023-10-30T13:20:26.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a400c1f0 con 0x7f01a4005190 2023-10-30T13:20:26.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01add9b640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a40047d0 0x7f01a4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:26.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b659c640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a4003980 0x7f01a4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:26.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b6d9d640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a4005190 0x7f01a400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:26.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01add9b640 1 -- 172.21.15.148:0/2019685938 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a4005190 msgr2=0x7f01a400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:26.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01add9b640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a4005190 0x7f01a400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01add9b640 1 -- 172.21.15.148:0/2019685938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a4003980 msgr2=0x7f01a4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:26.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01add9b640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a4003980 0x7f01a4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01add9b640 1 -- 172.21.15.148:0/2019685938 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a4003530 con 0x7f01a40047d0 2023-10-30T13:20:26.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.623+0000 7f01b6d9d640 1 -- 172.21.15.148:0/2019685938 reap_dead start 2023-10-30T13:20:26.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01add9b640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a40047d0 0x7f01a4004c50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009f90 tx=0x7f01a001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb92d22e3a1bf59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:26.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a00a5a20 con 0x7f01a40047d0 2023-10-30T13:20:26.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a000f040 con 0x7f01a40047d0 2023-10-30T13:20:26.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01b5d9b640 1 -- 172.21.15.148:0/2019685938 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a00a5dc0 con 0x7f01a40047d0 2023-10-30T13:20:26.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01bcfc5640 1 -- 172.21.15.148:0/2019685938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a40047d0 msgr2=0x7f01a4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:26.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01bcfc5640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a40047d0 0x7f01a4004c50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009f90 tx=0x7f01a001cc20 comp rx=0 tx=0).stop 2023-10-30T13:20:26.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01bcfc5640 1 -- 172.21.15.148:0/2019685938 shutdown_connections 2023-10-30T13:20:26.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01bcfc5640 1 --2- 172.21.15.148:0/2019685938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01a40047d0 0x7f01a4004c50 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01bcfc5640 1 -- 172.21.15.148:0/2019685938 >> 172.21.15.148:0/2019685938 conn(0x7f01b80c7910 msgr2=0x7f01b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:26.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01bcfc5640 1 -- 172.21.15.148:0/2019685938 shutdown_connections 2023-10-30T13:20:26.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.627+0000 7f01bcfc5640 1 -- 172.21.15.148:0/2019685938 wait complete. 2023-10-30T13:20:26.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 Processor -- start 2023-10-30T13:20:26.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 -- start start 2023-10-30T13:20:26.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a40047d0 0x7f01b80c3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:26.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b80c4e80 0x7f01b813d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:26.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01b81f0a50 0x7f01b8153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:26.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b81f05c0 con 0x7f01b81f0a50 2023-10-30T13:20:26.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b81f0ed0 con 0x7f01b80c4e80 2023-10-30T13:20:26.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b813c8a0 con 0x7f01a40047d0 2023-10-30T13:20:26.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01b81f0a50 0x7f01b8153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:26.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01b81f0a50 0x7f01b8153510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52366/0 (socket says 172.21.15.148:52366) 2023-10-30T13:20:26.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b6d9d640 1 -- 172.21.15.148:0/1248091884 learned_addr learned my addr 172.21.15.148:0/1248091884 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:26.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01add9b640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b80c4e80 0x7f01b813d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:26.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b659c640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a40047d0 0x7f01b80c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:26.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b6d9d640 1 -- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a40047d0 msgr2=0x7f01b80c3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:26.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b6d9d640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a40047d0 0x7f01b80c3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b6d9d640 1 -- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b80c4e80 msgr2=0x7f01b813d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:26.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b6d9d640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b80c4e80 0x7f01b813d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b6d9d640 1 -- 172.21.15.148:0/1248091884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b813cb60 con 0x7f01b81f0a50 2023-10-30T13:20:26.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b659c640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a40047d0 0x7f01b80c3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:26.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01b6d9d640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01b81f0a50 0x7f01b8153510 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f01b8071660 tx=0x7f01b000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:26.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01af7fe640 1 -- 172.21.15.148:0/1248091884 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0007ea0 con 0x7f01b81f0a50 2023-10-30T13:20:26.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01af7fe640 1 -- 172.21.15.148:0/1248091884 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b00052e0 con 0x7f01b81f0a50 2023-10-30T13:20:26.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b813ce40 con 0x7f01b81f0a50 2023-10-30T13:20:26.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01af7fe640 1 -- 172.21.15.148:0/1248091884 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0003600 con 0x7f01b81f0a50 2023-10-30T13:20:26.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.631+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b81f0250 con 0x7f01b81f0a50 2023-10-30T13:20:26.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.635+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b81f0770 con 0x7f01b81f0a50 2023-10-30T13:20:26.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.635+0000 7f01af7fe640 1 -- 172.21.15.148:0/1248091884 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01b0003760 con 0x7f01b81f0a50 2023-10-30T13:20:26.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.635+0000 7f01af7fe640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f01980664d0 0x7f0198068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:26.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.635+0000 7f01af7fe640 1 -- 172.21.15.148:0/1248091884 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f01b00835f0 con 0x7f01b81f0a50 2023-10-30T13:20:26.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.635+0000 7f01b659c640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f01980664d0 0x7f0198068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:26.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.635+0000 7f01b659c640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f01980664d0 0x7f0198068990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f01a8007cc0 tx=0x7f01a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:26.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.639+0000 7f01af7fe640 1 -- 172.21.15.148:0/1248091884 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01b81f0770 con 0x7f01b81f0a50 2023-10-30T13:20:26.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.823+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01b81f0770 con 0x7f01b81f0a50 2023-10-30T13:20:26.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.823+0000 7f01af7fe640 1 -- 172.21.15.148:0/1248091884 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7f01b81f0770 con 0x7f01b81f0a50 2023-10-30T13:20:26.824 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:26.825 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:26.825 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:26.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f01980664d0 msgr2=0x7f0198068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:26.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f01980664d0 0x7f0198068990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f01a8007cc0 tx=0x7f01a8009040 comp rx=0 tx=0).stop 2023-10-30T13:20:26.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01b81f0a50 msgr2=0x7f01b8153510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:26.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01b81f0a50 0x7f01b8153510 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f01b8071660 tx=0x7f01b000e820 comp rx=0 tx=0).stop 2023-10-30T13:20:26.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 shutdown_connections 2023-10-30T13:20:26.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f01980664d0 0x7f0198068990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f01b81f0a50 0x7f01b8153510 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b80c4e80 0x7f01b813d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 --2- 172.21.15.148:0/1248091884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f01a40047d0 0x7f01b80c3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:26.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 >> 172.21.15.148:0/1248091884 conn(0x7f01b80c7910 msgr2=0x7f01b80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:26.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 shutdown_connections 2023-10-30T13:20:26.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:26.827+0000 7f01bcfc5640 1 -- 172.21.15.148:0/1248091884 wait complete. 2023-10-30T13:20:26.845 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:28.846 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:28.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f849949a640 1 Processor -- start 2023-10-30T13:20:28.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f849949a640 1 -- start start 2023-10-30T13:20:28.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f849949a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941575f0 0x7f84940c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:28.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f849949a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84940c6ce0 con 0x7f8494151850 2023-10-30T13:20:28.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f849949a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84940c5520 con 0x7f849406f900 2023-10-30T13:20:28.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f849949a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84940704b0 con 0x7f84941575f0 2023-10-30T13:20:28.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f8492ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8494151850 0x7f84940c7450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44776/0 (socket says 172.21.15.148:44776) 2023-10-30T13:20:28.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f8492ffd640 1 -- 172.21.15.148:0/1452093368 learned_addr learned my addr 172.21.15.148:0/1452093368 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:29.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f848a7fc640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941575f0 0x7f84940c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750068519 0 0) 0x7f84940c6ce0 con 0x7f8494151850 2023-10-30T13:20:29.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8480003530 con 0x7f8494151850 2023-10-30T13:20:29.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1128935465 0 0) 0x7f8480003530 con 0x7f8494151850 2023-10-30T13:20:29.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941575f0 msgr2=0x7f84940c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:29.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941575f0 0x7f84940c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 >> v1:172.21.15.150:6789/0 conn(0x7f849406f900 legacy=0x7f84940c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8494070790 con 0x7f8494151850 2023-10-30T13:20:29.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f8484001680 con 0x7f8494151850 2023-10-30T13:20:29.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 >> v1:172.21.15.148:6789/0 conn(0x7f8494151850 legacy=0x7f84940c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941575f0 0x7f84800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8480004d10 0x7f8480005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84940704b0 con 0x7f8480003ef0 2023-10-30T13:20:29.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848000baa0 con 0x7f8480004d10 2023-10-30T13:20:29.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848000bc60 con 0x7f84941575f0 2023-10-30T13:20:29.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f848a7fc640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f8492ffd640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941575f0 0x7f84800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f84937fe640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8480004d10 0x7f8480005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f848a7fc640 1 -- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941575f0 msgr2=0x7f84800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:29.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f848a7fc640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941575f0 0x7f84800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f848a7fc640 1 -- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8480004d10 msgr2=0x7f8480005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:29.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f848a7fc640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8480004d10 0x7f8480005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.995+0000 7f848a7fc640 1 -- 172.21.15.148:0/1452093368 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480003530 con 0x7f8480003ef0 2023-10-30T13:20:29.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f848a7fc640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84800047d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f847c009fc0 tx=0x7f847c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa57acf03adae1f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:29.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c0a5c70 con 0x7f8480003ef0 2023-10-30T13:20:29.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c00f040 con 0x7f8480003ef0 2023-10-30T13:20:29.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f84927fc640 1 -- 172.21.15.148:0/1452093368 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c0ad440 con 0x7f8480003ef0 2023-10-30T13:20:29.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f849949a640 1 -- 172.21.15.148:0/1452093368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 msgr2=0x7f84800047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f849949a640 1 --2- 172.21.15.148:0/1452093368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84800047d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f847c009fc0 tx=0x7f847c01cc60 comp rx=0 tx=0).stop 2023-10-30T13:20:29.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f84937fe640 1 -- 172.21.15.148:0/1452093368 reap_dead start 2023-10-30T13:20:29.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f849949a640 1 -- 172.21.15.148:0/1452093368 shutdown_connections 2023-10-30T13:20:29.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f849949a640 1 -- 172.21.15.148:0/1452093368 >> 172.21.15.148:0/1452093368 conn(0x7f849406d770 msgr2=0x7f849406db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:29.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f849949a640 1 -- 172.21.15.148:0/1452093368 shutdown_connections 2023-10-30T13:20:29.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:28.999+0000 7f849949a640 1 -- 172.21.15.148:0/1452093368 wait complete. 2023-10-30T13:20:29.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 Processor -- start 2023-10-30T13:20:29.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 -- start start 2023-10-30T13:20:29.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84941e3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84940c1c80 0x7f84941f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941f2bb0 0x7f8494150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84940c0080 con 0x7f8480003ef0 2023-10-30T13:20:29.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84940c43c0 con 0x7f84940c1c80 2023-10-30T13:20:29.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84940c3160 con 0x7f84941f2bb0 2023-10-30T13:20:29.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f8492ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84941e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f8492ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84941e3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52374/0 (socket says 172.21.15.148:52374) 2023-10-30T13:20:29.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f8492ffd640 1 -- 172.21.15.148:0/3597359570 learned_addr learned my addr 172.21.15.148:0/3597359570 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:29.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f84937fe640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941f2bb0 0x7f8494150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f848a7fc640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84940c1c80 0x7f84941f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f8492ffd640 1 -- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941f2bb0 msgr2=0x7f8494150ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f8492ffd640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941f2bb0 0x7f8494150ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f8492ffd640 1 -- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84940c1c80 msgr2=0x7f84941f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f8492ffd640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84940c1c80 0x7f84941f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f8492ffd640 1 -- 172.21.15.148:0/3597359570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84940c2640 con 0x7f8480003ef0 2023-10-30T13:20:29.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f8492ffd640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84941e3160 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f848400e780 tx=0x7f848400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:29.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f848bfff640 1 -- 172.21.15.148:0/3597359570 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848400ee90 con 0x7f8480003ef0 2023-10-30T13:20:29.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f848bfff640 1 -- 172.21.15.148:0/3597359570 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848400c8d0 con 0x7f8480003ef0 2023-10-30T13:20:29.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f848bfff640 1 -- 172.21.15.148:0/3597359570 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848400cc60 con 0x7f8480003ef0 2023-10-30T13:20:29.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84940c28c0 con 0x7f8480003ef0 2023-10-30T13:20:29.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.003+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84940c2e80 con 0x7f8480003ef0 2023-10-30T13:20:29.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.007+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8494074250 con 0x7f8480003ef0 2023-10-30T13:20:29.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.007+0000 7f848bfff640 1 -- 172.21.15.148:0/3597359570 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f848401d460 con 0x7f8480003ef0 2023-10-30T13:20:29.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.007+0000 7f848bfff640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84740664d0 0x7f8474068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.007+0000 7f848bfff640 1 -- 172.21.15.148:0/3597359570 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f8484083ec0 con 0x7f8480003ef0 2023-10-30T13:20:29.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.007+0000 7f848a7fc640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84740664d0 0x7f8474068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.011+0000 7f848a7fc640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84740664d0 0x7f8474068990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f847c0af000 tx=0x7f847c004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:29.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.011+0000 7f848bfff640 1 -- 172.21.15.148:0/3597359570 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8484051160 con 0x7f8480003ef0 2023-10-30T13:20:29.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8494070a70 con 0x7f8480003ef0 2023-10-30T13:20:29.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f848bfff640 1 -- 172.21.15.148:0/3597359570 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+6540 (secure 0 0 0) 0x7f8484054780 con 0x7f8480003ef0 2023-10-30T13:20:29.195 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:29.195 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:06.522682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2],"up":{"mds_0":5033,"mds_1":5103,"mds_2":4959},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4959":{"gid":4959,"name":"c","rank":2,"incarnation":20,"state":"up:stopping","state_seq":16,"addr":"172.21.15.148:6837/95380877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":95380877},{"type":"v1","addr":"172.21.15.148:6837","nonce":95380877}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:active","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:29.196 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 48 2023-10-30T13:20:29.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84740664d0 msgr2=0x7f8474068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84740664d0 0x7f8474068990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f847c0af000 tx=0x7f847c004ac0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 msgr2=0x7f84941e3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84941e3160 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f848400e780 tx=0x7f848400ec40 comp rx=0 tx=0).stop 2023-10-30T13:20:29.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 shutdown_connections 2023-10-30T13:20:29.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84740664d0 0x7f8474068990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84941f2bb0 0x7f8494150ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84940c1c80 0x7f84941f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 --2- 172.21.15.148:0/3597359570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8480003ef0 0x7f84941e3160 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 >> 172.21.15.148:0/3597359570 conn(0x7f849406d770 msgr2=0x7f84941e3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:29.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 shutdown_connections 2023-10-30T13:20:29.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:29.195+0000 7f849949a640 1 -- 172.21.15.148:0/3597359570 wait complete. 2023-10-30T13:20:29.216 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:31.217 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:31.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.363+0000 7fc6c8565640 1 Processor -- start 2023-10-30T13:20:31.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.363+0000 7fc6c8565640 1 -- start start 2023-10-30T13:20:31.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c8565640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc6c00c05d0 0x7fc6c013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:31.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c8565640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc6c00c6f60 0x7fc6c013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:31.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c8565640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c00c0a50 con 0x7fc6c00c6b80 2023-10-30T13:20:31.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c8565640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c006f6e0 con 0x7fc6c00c6f60 2023-10-30T13:20:31.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c8565640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c006f9c0 con 0x7fc6c00c05d0 2023-10-30T13:20:31.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c6b02640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc6c00c6f60 0x7fc6c013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:31.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c6301640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc6c00c6b80 0x7fc6c00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44786/0 (socket says 172.21.15.148:44786) 2023-10-30T13:20:31.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c6b02640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc6c00c6f60 0x7fc6c013f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52378/0 (socket says 172.21.15.148:52378) 2023-10-30T13:20:31.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c6301640 1 -- 172.21.15.148:0/1416821157 learned_addr learned my addr 172.21.15.148:0/1416821157 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:31.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c5b00640 1 --2- 172.21.15.148:0/1416821157 >> v2:172.21.15.150:3300/0 conn(0x7fc6c00c05d0 0x7fc6c013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:31.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c6b02640 1 -- 172.21.15.148:0/1416821157 >> v2:172.21.15.150:3300/0 conn(0x7fc6c00c05d0 msgr2=0x7fc6c013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:31.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c6b02640 1 --2- 172.21.15.148:0/1416821157 >> v2:172.21.15.150:3300/0 conn(0x7fc6c00c05d0 0x7fc6c013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c6b02640 1 -- 172.21.15.148:0/1416821157 >> v1:172.21.15.148:6789/0 conn(0x7fc6c00c6b80 legacy=0x7fc6c00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:31.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c6b02640 1 -- 172.21.15.148:0/1416821157 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c006fca0 con 0x7fc6c00c6f60 2023-10-30T13:20:31.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.367+0000 7fc6c52ff640 1 -- 172.21.15.148:0/1416821157 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994118080 0 0) 0x7fc6c00c0a50 con 0x7fc6c00c6b80 2023-10-30T13:20:31.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c6b02640 1 --2- 172.21.15.148:0/1416821157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c6f60 0x7fc6c013f2d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc6c006c7e0 tx=0x7fc6b801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ac64261117bf36 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:31.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c52ff640 1 -- 172.21.15.148:0/1416821157 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80a58e0 con 0x7fc6c00c6f60 2023-10-30T13:20:31.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c52ff640 1 -- 172.21.15.148:0/1416821157 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b80a5a40 con 0x7fc6c00c6f60 2023-10-30T13:20:31.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c52ff640 1 -- 172.21.15.148:0/1416821157 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80a5dd0 con 0x7fc6c00c6f60 2023-10-30T13:20:31.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 -- 172.21.15.148:0/1416821157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c6f60 msgr2=0x7fc6c013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:31.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 --2- 172.21.15.148:0/1416821157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c6f60 0x7fc6c013f2d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc6c006c7e0 tx=0x7fc6b801c910 comp rx=0 tx=0).stop 2023-10-30T13:20:31.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 -- 172.21.15.148:0/1416821157 shutdown_connections 2023-10-30T13:20:31.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 --2- 172.21.15.148:0/1416821157 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c6f60 0x7fc6c013f2d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 --2- 172.21.15.148:0/1416821157 >> v2:172.21.15.150:3300/0 conn(0x7fc6c00c05d0 0x7fc6c013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 -- 172.21.15.148:0/1416821157 >> 172.21.15.148:0/1416821157 conn(0x7fc6c00c7910 msgr2=0x7fc6c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 -- 172.21.15.148:0/1416821157 shutdown_connections 2023-10-30T13:20:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 -- 172.21.15.148:0/1416821157 wait complete. 2023-10-30T13:20:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 Processor -- start 2023-10-30T13:20:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 -- start start 2023-10-30T13:20:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c05d0 0x7fc6c00ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.371+0000 7fc6c8565640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6c00c6b80 0x7fc6c01579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c8565640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c00c6f60 0x7fc6c0133850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c8565640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c0158490 con 0x7fc6c00c05d0 2023-10-30T13:20:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c8565640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c00c9d20 con 0x7fc6c00c6f60 2023-10-30T13:20:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c8565640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c0158c30 con 0x7fc6c00c6b80 2023-10-30T13:20:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6301640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c05d0 0x7fc6c00ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6301640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c05d0 0x7fc6c00ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52386/0 (socket says 172.21.15.148:52386) 2023-10-30T13:20:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6301640 1 -- 172.21.15.148:0/2272396532 learned_addr learned my addr 172.21.15.148:0/2272396532 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6b02640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c00c6f60 0x7fc6c0133850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c5b00640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6c00c6b80 0x7fc6c01579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6301640 1 -- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6c00c6b80 msgr2=0x7fc6c01579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6301640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6c00c6b80 0x7fc6c01579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6301640 1 -- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c00c6f60 msgr2=0x7fc6c0133850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6301640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c00c6f60 0x7fc6c0133850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6301640 1 -- 172.21.15.148:0/2272396532 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b8009a70 con 0x7fc6c00c05d0 2023-10-30T13:20:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c6301640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c05d0 0x7fc6c00ca7d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc0011f0 tx=0x7fc6bc00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6b6ffd640 1 -- 172.21.15.148:0/2272396532 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc0074e0 con 0x7fc6c00c05d0 2023-10-30T13:20:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6b6ffd640 1 -- 172.21.15.148:0/2272396532 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6bc007640 con 0x7fc6c00c05d0 2023-10-30T13:20:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c00c9a40 con 0x7fc6c00c05d0 2023-10-30T13:20:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6b6ffd640 1 -- 172.21.15.148:0/2272396532 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc004a10 con 0x7fc6c00c05d0 2023-10-30T13:20:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c0158950 con 0x7fc6c00c05d0 2023-10-30T13:20:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c004e170 con 0x7fc6c00c05d0 2023-10-30T13:20:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6b6ffd640 1 -- 172.21.15.148:0/2272396532 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6bc004b70 con 0x7fc6c00c05d0 2023-10-30T13:20:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.375+0000 7fc6b6ffd640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6a40664a0 0x7fc6a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.379+0000 7fc6b6ffd640 1 -- 172.21.15.148:0/2272396532 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fc6bc0835d0 con 0x7fc6c00c05d0 2023-10-30T13:20:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.379+0000 7fc6c5b00640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6a40664a0 0x7fc6a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.379+0000 7fc6c5b00640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6a40664a0 0x7fc6a4068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc6c013bea0 tx=0x7fc6b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.379+0000 7fc6b6ffd640 1 -- 172.21.15.148:0/2272396532 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6bc0507c0 con 0x7fc6c00c05d0 2023-10-30T13:20:31.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6c0154070 con 0x7fc6c00c05d0 2023-10-30T13:20:31.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6b6ffd640 1 -- 172.21.15.148:0/2272396532 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fc6bc053de0 con 0x7fc6c00c05d0 2023-10-30T13:20:31.572 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:31.572 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:31.554781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:31.572 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 50 2023-10-30T13:20:31.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6a40664a0 msgr2=0x7fc6a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:31.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6a40664a0 0x7fc6a4068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc6c013bea0 tx=0x7fc6b0008040 comp rx=0 tx=0).stop 2023-10-30T13:20:31.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c05d0 msgr2=0x7fc6c00ca7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:31.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c05d0 0x7fc6c00ca7d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc0011f0 tx=0x7fc6bc00c9b0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 shutdown_connections 2023-10-30T13:20:31.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc6a40664a0 0x7fc6a4068960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c00c6f60 0x7fc6c0133850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc6c00c6b80 0x7fc6c01579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 --2- 172.21.15.148:0/2272396532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc6c00c05d0 0x7fc6c00ca7d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:31.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 >> 172.21.15.148:0/2272396532 conn(0x7fc6c00c7910 msgr2=0x7fc6c0135c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:31.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.571+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 shutdown_connections 2023-10-30T13:20:31.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:31.575+0000 7fc6c8565640 1 -- 172.21.15.148:0/2272396532 wait complete. 2023-10-30T13:20:31.594 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:20:31.595 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5033": { "addr": "172.21.15.150:6836/953984429", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 953984429, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 953984429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5033, "incarnation": 36, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:20:31.554781+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5033, "mds_1": 5103 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:20:31.617 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:20:33.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:33.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.743+0000 7f0141691640 1 Processor -- start 2023-10-30T13:20:33.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0141691640 1 -- start start 2023-10-30T13:20:33.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0141691640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f013c153c30 0x7f013c06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:33.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0141691640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c0b1ea0 0x7f013c0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0141691640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013c06ade0 con 0x7f013c06b4a0 2023-10-30T13:20:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0141691640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f013c1596e0 con 0x7f013c153c30 2023-10-30T13:20:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0141691640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013c1598f0 con 0x7f013c0b1ea0 2023-10-30T13:20:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f013b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f013c06b4a0 0x7f013c0cabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46168/0 (socket says 172.21.15.148:46168) 2023-10-30T13:20:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f013b7fe640 1 -- 172.21.15.148:0/1222638971 learned_addr learned my addr 172.21.15.148:0/1222638971 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0133fff640 1 --2- 172.21.15.148:0/1222638971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c0b1ea0 0x7f013c0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f013affd640 1 --2- 172.21.15.148:0/1222638971 >> v2:172.21.15.150:3300/0 conn(0x7f013c153c30 0x7f013c06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f013a7fc640 1 -- 172.21.15.148:0/1222638971 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159627836 0 0) 0x7f01340014a0 con 0x7f013c06b4a0 2023-10-30T13:20:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f013a7fc640 1 -- 172.21.15.148:0/1222638971 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0128003530 con 0x7f013c06b4a0 2023-10-30T13:20:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0133fff640 1 -- 172.21.15.148:0/1222638971 >> v2:172.21.15.150:3300/0 conn(0x7f013c153c30 msgr2=0x7f013c06a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0133fff640 1 --2- 172.21.15.148:0/1222638971 >> v2:172.21.15.150:3300/0 conn(0x7f013c153c30 0x7f013c06a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0133fff640 1 -- 172.21.15.148:0/1222638971 >> v1:172.21.15.148:6789/0 conn(0x7f013c06b4a0 legacy=0x7f013c0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.747+0000 7f0133fff640 1 -- 172.21.15.148:0/1222638971 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013c159b90 con 0x7f013c0b1ea0 2023-10-30T13:20:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f0133fff640 1 --2- 172.21.15.148:0/1222638971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c0b1ea0 0x7f013c0cb9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f01240097a0 tx=0x7f012401ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=365fd342c0cf1571 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f013a7fc640 1 -- 172.21.15.148:0/1222638971 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240a8950 con 0x7f013c0b1ea0 2023-10-30T13:20:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f013a7fc640 1 -- 172.21.15.148:0/1222638971 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01240a8ab0 con 0x7f013c0b1ea0 2023-10-30T13:20:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f013a7fc640 1 -- 172.21.15.148:0/1222638971 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240a8e40 con 0x7f013c0b1ea0 2023-10-30T13:20:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f0141691640 1 -- 172.21.15.148:0/1222638971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c0b1ea0 msgr2=0x7f013c0cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f0141691640 1 --2- 172.21.15.148:0/1222638971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c0b1ea0 0x7f013c0cb9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f01240097a0 tx=0x7f012401ec50 comp rx=0 tx=0).stop 2023-10-30T13:20:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f0141691640 1 -- 172.21.15.148:0/1222638971 shutdown_connections 2023-10-30T13:20:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f0141691640 1 --2- 172.21.15.148:0/1222638971 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c0b1ea0 0x7f013c0cb9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f0141691640 1 --2- 172.21.15.148:0/1222638971 >> v2:172.21.15.150:3300/0 conn(0x7f013c153c30 0x7f013c06a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f0141691640 1 -- 172.21.15.148:0/1222638971 >> 172.21.15.148:0/1222638971 conn(0x7f013c06bf40 msgr2=0x7f013c06c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f0141691640 1 -- 172.21.15.148:0/1222638971 shutdown_connections 2023-10-30T13:20:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.751+0000 7f0141691640 1 -- 172.21.15.148:0/1222638971 wait complete. 2023-10-30T13:20:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 Processor -- start 2023-10-30T13:20:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 -- start start 2023-10-30T13:20:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f013c06b4a0 0x7f013c1f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f013c0b1ea0 0x7f013c0cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c153c30 0x7f013c14f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c0c6d70 con 0x7f013c0b1ea0 2023-10-30T13:20:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c0c7370 con 0x7f013c06b4a0 2023-10-30T13:20:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013c14eb90 con 0x7f013c153c30 2023-10-30T13:20:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0133fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f013c0b1ea0 0x7f013c0cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f013affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f013c06b4a0 0x7f013c1f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0133fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f013c0b1ea0 0x7f013c0cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44806/0 (socket says 172.21.15.148:44806) 2023-10-30T13:20:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0133fff640 1 -- 172.21.15.148:0/2216599081 learned_addr learned my addr 172.21.15.148:0/2216599081 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f013b7fe640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c153c30 0x7f013c14f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0133fff640 1 -- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c153c30 msgr2=0x7f013c14f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0133fff640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c153c30 0x7f013c14f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0133fff640 1 -- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f013c06b4a0 msgr2=0x7f013c1f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0133fff640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f013c06b4a0 0x7f013c1f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0133fff640 1 -- 172.21.15.148:0/2216599081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01240093b0 con 0x7f013c0b1ea0 2023-10-30T13:20:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0133fff640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f013c0b1ea0 0x7f013c0cae30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f01240b3040 tx=0x7f012401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f01337fe640 1 -- 172.21.15.148:0/2216599081 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240b0d20 con 0x7f013c0b1ea0 2023-10-30T13:20:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f01337fe640 1 -- 172.21.15.148:0/2216599081 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01240b0e80 con 0x7f013c0b1ea0 2023-10-30T13:20:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f01337fe640 1 -- 172.21.15.148:0/2216599081 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124005be0 con 0x7f013c0b1ea0 2023-10-30T13:20:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013c1f2830 con 0x7f013c0b1ea0 2023-10-30T13:20:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.755+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013c14e260 con 0x7f013c0b1ea0 2023-10-30T13:20:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.759+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013c04e170 con 0x7f013c0b1ea0 2023-10-30T13:20:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.759+0000 7f01337fe640 1 -- 172.21.15.148:0/2216599081 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01240b7040 con 0x7f013c0b1ea0 2023-10-30T13:20:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.759+0000 7f01337fe640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f011c0664a0 0x7f011c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.759+0000 7f01337fe640 1 -- 172.21.15.148:0/2216599081 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f0124121140 con 0x7f013c0b1ea0 2023-10-30T13:20:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.759+0000 7f013affd640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f011c0664a0 0x7f011c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.759+0000 7f013affd640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f011c0664a0 0x7f011c068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f012c009ac0 tx=0x7f012c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.763+0000 7f01337fe640 1 -- 172.21.15.148:0/2216599081 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01240ee3f0 con 0x7f013c0b1ea0 2023-10-30T13:20:33.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.943+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f013c0c9440 con 0x7f013c0b1ea0 2023-10-30T13:20:33.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.943+0000 7f01337fe640 1 -- 172.21.15.148:0/2216599081 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f01240b5090 con 0x7f013c0b1ea0 2023-10-30T13:20:33.945 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:33.946 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:31.554781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:33.946 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 51 2023-10-30T13:20:33.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f011c0664a0 msgr2=0x7f011c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:33.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f011c0664a0 0x7f011c068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f012c009ac0 tx=0x7f012c009190 comp rx=0 tx=0).stop 2023-10-30T13:20:33.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f013c0b1ea0 msgr2=0x7f013c0cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:33.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f013c0b1ea0 0x7f013c0cae30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f01240b3040 tx=0x7f012401b9f0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 shutdown_connections 2023-10-30T13:20:33.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f011c0664a0 0x7f011c068960 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f013c0b1ea0 0x7f013c0cae30 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f013c153c30 0x7f013c14f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 --2- 172.21.15.148:0/2216599081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f013c06b4a0 0x7f013c1f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:33.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 >> 172.21.15.148:0/2216599081 conn(0x7f013c06bf40 msgr2=0x7f013c06c320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:33.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 shutdown_connections 2023-10-30T13:20:33.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:33.947+0000 7f0141691640 1 -- 172.21.15.148:0/2216599081 wait complete. 2023-10-30T13:20:33.966 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:35.967 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:36.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb66b6ea640 1 Processor -- start 2023-10-30T13:20:36.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb66b6ea640 1 -- start start 2023-10-30T13:20:36.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb66b6ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c6bd0 0x7fb66406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:36.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb66b6ea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb664066cf0 con 0x7fb664071110 2023-10-30T13:20:36.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb66b6ea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6640c70d0 con 0x7fb66406f7a0 2023-10-30T13:20:36.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb66b6ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6640700c0 con 0x7fb6640c6bd0 2023-10-30T13:20:36.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb669c87640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb664071110 0x7fb664158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46176/0 (socket says 172.21.15.148:46176) 2023-10-30T13:20:36.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb669c87640 1 -- 172.21.15.148:0/1224532079 learned_addr learned my addr 172.21.15.148:0/1224532079 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:36.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb668c85640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c6bd0 0x7fb66406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162731651 0 0) 0x7fb664066cf0 con 0x7fb664071110 2023-10-30T13:20:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb650003530 con 0x7fb664071110 2023-10-30T13:20:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3549185513 0 0) 0x7fb650003530 con 0x7fb664071110 2023-10-30T13:20:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c6bd0 msgr2=0x7fb66406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:36.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c6bd0 0x7fb66406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 >> v1:172.21.15.150:6789/0 conn(0x7fb66406f7a0 legacy=0x7fb664069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:36.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6640c7390 con 0x7fb664071110 2023-10-30T13:20:36.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fb65c001a00 con 0x7fb664071110 2023-10-30T13:20:36.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 >> v1:172.21.15.148:6789/0 conn(0x7fb664071110 legacy=0x7fb664158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:36.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb650003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:36.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6500047d0 0x7fb650004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:36.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c6bd0 0x7fb650005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:36.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6640700c0 con 0x7fb6500039b0 2023-10-30T13:20:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65000fee0 con 0x7fb6500047d0 2023-10-30T13:20:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.115+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500100a0 con 0x7fb6640c6bd0 2023-10-30T13:20:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb669486640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb650003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb669c87640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c6bd0 0x7fb650005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb668c85640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6500047d0 0x7fb650004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:36.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb669486640 1 -- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c6bd0 msgr2=0x7fb650005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:36.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb669486640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c6bd0 0x7fb650005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb669486640 1 -- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6500047d0 msgr2=0x7fb650004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:36.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb669486640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6500047d0 0x7fb650004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb669486640 1 -- 172.21.15.148:0/1224532079 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb650003530 con 0x7fb6500039b0 2023-10-30T13:20:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb669486640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb650003dd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb660001750 tx=0x7fb66001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15a0f94da04832e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6600052d0 con 0x7fb6500039b0 2023-10-30T13:20:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6600a7050 con 0x7fb6500039b0 2023-10-30T13:20:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.119+0000 7fb65bfff640 1 -- 172.21.15.148:0/1224532079 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb660003600 con 0x7fb6500039b0 2023-10-30T13:20:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 -- 172.21.15.148:0/1224532079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 msgr2=0x7fb650003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 --2- 172.21.15.148:0/1224532079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb650003dd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb660001750 tx=0x7fb66001c3a0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669c87640 1 -- 172.21.15.148:0/1224532079 reap_dead start 2023-10-30T13:20:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 -- 172.21.15.148:0/1224532079 shutdown_connections 2023-10-30T13:20:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 -- 172.21.15.148:0/1224532079 >> 172.21.15.148:0/1224532079 conn(0x7fb6640c7910 msgr2=0x7fb6640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 -- 172.21.15.148:0/1224532079 shutdown_connections 2023-10-30T13:20:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 -- 172.21.15.148:0/1224532079 wait complete. 2023-10-30T13:20:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 Processor -- start 2023-10-30T13:20:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 -- start start 2023-10-30T13:20:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb6641eac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c0a90 0x7fb6640c1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664145390 0x7fb664145810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb664145d50 con 0x7fb6500039b0 2023-10-30T13:20:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6641438a0 con 0x7fb664145390 2023-10-30T13:20:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb66b6ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb664143b80 con 0x7fb6640c0a90 2023-10-30T13:20:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669486640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb6641eac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669486640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb6641eac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44828/0 (socket says 172.21.15.148:44828) 2023-10-30T13:20:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669486640 1 -- 172.21.15.148:0/793644450 learned_addr learned my addr 172.21.15.148:0/793644450 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb668c85640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c0a90 0x7fb6640c1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669c87640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664145390 0x7fb664145810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669486640 1 -- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c0a90 msgr2=0x7fb6640c1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669486640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c0a90 0x7fb6640c1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669486640 1 -- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664145390 msgr2=0x7fb664145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669486640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664145390 0x7fb664145810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669486640 1 -- 172.21.15.148:0/793644450 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb664143e60 con 0x7fb6500039b0 2023-10-30T13:20:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb668c85640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c0a90 0x7fb6640c1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669c87640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664145390 0x7fb664145810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.123+0000 7fb669486640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb6641eac20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb660005040 tx=0x7fb650005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb659ffb640 1 -- 172.21.15.148:0/793644450 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb660000ee0 con 0x7fb6500039b0 2023-10-30T13:20:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb659ffb640 1 -- 172.21.15.148:0/793644450 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb660007680 con 0x7fb6500039b0 2023-10-30T13:20:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb659ffb640 1 -- 172.21.15.148:0/793644450 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6600069e0 con 0x7fb6500039b0 2023-10-30T13:20:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb664144140 con 0x7fb6500039b0 2023-10-30T13:20:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6640c8530 con 0x7fb6500039b0 2023-10-30T13:20:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6641445f0 con 0x7fb6500039b0 2023-10-30T13:20:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb659ffb640 1 -- 172.21.15.148:0/793644450 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb660006c00 con 0x7fb6500039b0 2023-10-30T13:20:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb659ffb640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6480664d0 0x7fb648068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb659ffb640 1 -- 172.21.15.148:0/793644450 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fb66011e630 con 0x7fb6500039b0 2023-10-30T13:20:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.127+0000 7fb668c85640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6480664d0 0x7fb648068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.131+0000 7fb668c85640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6480664d0 0x7fb648068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb664154ab0 tx=0x7fb654008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.131+0000 7fb659ffb640 1 -- 172.21.15.148:0/793644450 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb6600eb8b0 con 0x7fb6500039b0 2023-10-30T13:20:36.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.311+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6641451e0 con 0x7fb6500039b0 2023-10-30T13:20:36.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.311+0000 7fb659ffb640 1 -- 172.21.15.148:0/793644450 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fb6600eeed0 con 0x7fb6500039b0 2023-10-30T13:20:36.313 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:36.313 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:31.554781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:36.313 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 51 2023-10-30T13:20:36.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6480664d0 msgr2=0x7fb648068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:36.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6480664d0 0x7fb648068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb664154ab0 tx=0x7fb654008040 comp rx=0 tx=0).stop 2023-10-30T13:20:36.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 msgr2=0x7fb6641eac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:36.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb6641eac20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb660005040 tx=0x7fb650005800 comp rx=0 tx=0).stop 2023-10-30T13:20:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 shutdown_connections 2023-10-30T13:20:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6480664d0 0x7fb648068990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664145390 0x7fb664145810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6640c0a90 0x7fb6640c1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 --2- 172.21.15.148:0/793644450 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6500039b0 0x7fb6641eac20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:36.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 >> 172.21.15.148:0/793644450 conn(0x7fb6640c7910 msgr2=0x7fb6640ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:36.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 shutdown_connections 2023-10-30T13:20:36.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:36.315+0000 7fb66b6ea640 1 -- 172.21.15.148:0/793644450 wait complete. 2023-10-30T13:20:36.334 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:38.335 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:38.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff589d65640 1 Processor -- start 2023-10-30T13:20:38.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff589d65640 1 -- start start 2023-10-30T13:20:38.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff589d65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5840c6b80 0x7ff5840bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:38.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff589d65640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff58413f4a0 0x7ff5840c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:38.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff589d65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5840c0a50 con 0x7ff5840c6f60 2023-10-30T13:20:38.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff589d65640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58406f6e0 con 0x7ff58413f4a0 2023-10-30T13:20:38.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff589d65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58406f9c0 con 0x7ff5840c6b80 2023-10-30T13:20:38.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff583fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff58413f4a0 0x7ff5840c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:38.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff583fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff58413f4a0 0x7ff5840c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44836/0 (socket says 172.21.15.148:44836) 2023-10-30T13:20:38.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff583fff640 1 -- 172.21.15.148:0/419293970 learned_addr learned my addr 172.21.15.148:0/419293970 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:38.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff5837fe640 1 --2- 172.21.15.148:0/419293970 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5840c6b80 0x7ff5840bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:38.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff583fff640 1 -- 172.21.15.148:0/419293970 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5840c6b80 msgr2=0x7ff5840bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:38.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff583fff640 1 --2- 172.21.15.148:0/419293970 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5840c6b80 0x7ff5840bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff583fff640 1 -- 172.21.15.148:0/419293970 >> v1:172.21.15.150:6789/0 conn(0x7ff5840c6f60 legacy=0x7ff58413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:38.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.475+0000 7ff583fff640 1 -- 172.21.15.148:0/419293970 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58406fca0 con 0x7ff58413f4a0 2023-10-30T13:20:38.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff583fff640 1 --2- 172.21.15.148:0/419293970 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff58413f4a0 0x7ff5840c0400 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009e50 tx=0x7ff57c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446d6b1ec572df1a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:38.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff582ffd640 1 -- 172.21.15.148:0/419293970 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0a5760 con 0x7ff58413f4a0 2023-10-30T13:20:38.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff582ffd640 1 -- 172.21.15.148:0/419293970 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c0a58c0 con 0x7ff58413f4a0 2023-10-30T13:20:38.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff582ffd640 1 -- 172.21.15.148:0/419293970 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0a5c50 con 0x7ff58413f4a0 2023-10-30T13:20:38.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff589d65640 1 -- 172.21.15.148:0/419293970 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff58413f4a0 msgr2=0x7ff5840c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:38.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff589d65640 1 --2- 172.21.15.148:0/419293970 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff58413f4a0 0x7ff5840c0400 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009e50 tx=0x7ff57c01c9d0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff589d65640 1 -- 172.21.15.148:0/419293970 shutdown_connections 2023-10-30T13:20:38.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff589d65640 1 --2- 172.21.15.148:0/419293970 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff58413f4a0 0x7ff5840c0400 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff589d65640 1 --2- 172.21.15.148:0/419293970 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5840c6b80 0x7ff5840bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff589d65640 1 -- 172.21.15.148:0/419293970 >> 172.21.15.148:0/419293970 conn(0x7ff5840c7910 msgr2=0x7ff5840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff589d65640 1 -- 172.21.15.148:0/419293970 shutdown_connections 2023-10-30T13:20:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.479+0000 7ff589d65640 1 -- 172.21.15.148:0/419293970 wait complete. 2023-10-30T13:20:38.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 Processor -- start 2023-10-30T13:20:38.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 -- start start 2023-10-30T13:20:38.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5840c6b80 0x7ff584154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:38.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5840c6f60 0x7ff584151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:38.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff58413f4a0 0x7ff584159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:38.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5841519d0 con 0x7ff5840c6b80 2023-10-30T13:20:38.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5841523a0 con 0x7ff5840c6f60 2023-10-30T13:20:38.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff584137200 con 0x7ff58413f4a0 2023-10-30T13:20:38.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff5837fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5840c6b80 0x7ff584154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:38.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff5837fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5840c6b80 0x7ff584154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44840/0 (socket says 172.21.15.148:44840) 2023-10-30T13:20:38.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff5837fe640 1 -- 172.21.15.148:0/2114468359 learned_addr learned my addr 172.21.15.148:0/2114468359 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:38.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff583fff640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff58413f4a0 0x7ff584159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:38.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff57bfff640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5840c6f60 0x7ff584151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:38.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff5837fe640 1 -- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff58413f4a0 msgr2=0x7ff584159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:38.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff5837fe640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff58413f4a0 0x7ff584159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff5837fe640 1 -- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5840c6f60 msgr2=0x7ff584151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:38.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff5837fe640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5840c6f60 0x7ff584151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff5837fe640 1 -- 172.21.15.148:0/2114468359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c009b30 con 0x7ff5840c6b80 2023-10-30T13:20:38.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff5837fe640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5840c6b80 0x7ff584154980 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff57400d800 tx=0x7ff57400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:38.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff580ff9640 1 -- 172.21.15.148:0/2114468359 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57400b6c0 con 0x7ff5840c6b80 2023-10-30T13:20:38.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff580ff9640 1 -- 172.21.15.148:0/2114468359 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57400b820 con 0x7ff5840c6b80 2023-10-30T13:20:38.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff580ff9640 1 -- 172.21.15.148:0/2114468359 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57400bb90 con 0x7ff5840c6b80 2023-10-30T13:20:38.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5840c9240 con 0x7ff5840c6b80 2023-10-30T13:20:38.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5840ca920 con 0x7ff5840c6b80 2023-10-30T13:20:38.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.483+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58404e170 con 0x7ff5840c6b80 2023-10-30T13:20:38.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.487+0000 7ff580ff9640 1 -- 172.21.15.148:0/2114468359 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff57400bcf0 con 0x7ff5840c6b80 2023-10-30T13:20:38.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.487+0000 7ff580ff9640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff5640664d0 0x7ff564068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:38.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.487+0000 7ff580ff9640 1 -- 172.21.15.148:0/2114468359 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7ff574084710 con 0x7ff5840c6b80 2023-10-30T13:20:38.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.487+0000 7ff57bfff640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff5640664d0 0x7ff564068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:38.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.487+0000 7ff57bfff640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff5640664d0 0x7ff564068990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff584070950 tx=0x7ff56c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.491+0000 7ff580ff9640 1 -- 172.21.15.148:0/2114468359 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff574051870 con 0x7ff5840c6b80 2023-10-30T13:20:38.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.691+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5841572e0 con 0x7ff5840c6b80 2023-10-30T13:20:38.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.691+0000 7ff580ff9640 1 -- 172.21.15.148:0/2114468359 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6529 (secure 0 0 0) 0x7ff574011340 con 0x7ff5840c6b80 2023-10-30T13:20:38.692 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:38.692 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:31.554781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:38.693 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 51 2023-10-30T13:20:38.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff5640664d0 msgr2=0x7ff564068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:38.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff5640664d0 0x7ff564068990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff584070950 tx=0x7ff56c009040 comp rx=0 tx=0).stop 2023-10-30T13:20:38.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5840c6b80 msgr2=0x7ff584154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:38.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5840c6b80 0x7ff584154980 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff57400d800 tx=0x7ff57400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 shutdown_connections 2023-10-30T13:20:38.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff5640664d0 0x7ff564068990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff58413f4a0 0x7ff584159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5840c6f60 0x7ff584151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 --2- 172.21.15.148:0/2114468359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5840c6b80 0x7ff584154980 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:38.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 >> 172.21.15.148:0/2114468359 conn(0x7ff5840c7910 msgr2=0x7ff5840b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:38.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 shutdown_connections 2023-10-30T13:20:38.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:38.695+0000 7ff589d65640 1 -- 172.21.15.148:0/2114468359 wait complete. 2023-10-30T13:20:38.697 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/33' 2023-10-30T13:20:38.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/23' 2023-10-30T13:20:38.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/35' 2023-10-30T13:20:38.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/26' 2023-10-30T13:20:38.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed './256/37' 2023-10-30T13:20:38.698 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './256' 2023-10-30T13:20:38.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/21' 2023-10-30T13:20:38.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/27' 2023-10-30T13:20:38.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/31' 2023-10-30T13:20:38.699 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/36' 2023-10-30T13:20:38.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/20' 2023-10-30T13:20:38.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/39' 2023-10-30T13:20:38.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/34' 2023-10-30T13:20:38.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/38' 2023-10-30T13:20:38.700 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/28' 2023-10-30T13:20:38.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/24' 2023-10-30T13:20:38.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/29' 2023-10-30T13:20:38.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/32' 2023-10-30T13:20:38.701 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/22' 2023-10-30T13:20:38.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/25' 2023-10-30T13:20:38.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/30' 2023-10-30T13:20:38.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/33' 2023-10-30T13:20:38.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/23' 2023-10-30T13:20:38.702 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/35' 2023-10-30T13:20:38.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/26' 2023-10-30T13:20:38.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed './257/37' 2023-10-30T13:20:38.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './257' 2023-10-30T13:20:38.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/21' 2023-10-30T13:20:38.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/27' 2023-10-30T13:20:38.703 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/31' 2023-10-30T13:20:38.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/36' 2023-10-30T13:20:38.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/20' 2023-10-30T13:20:38.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/39' 2023-10-30T13:20:38.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/34' 2023-10-30T13:20:38.704 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/38' 2023-10-30T13:20:38.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/28' 2023-10-30T13:20:38.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/24' 2023-10-30T13:20:38.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/29' 2023-10-30T13:20:38.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/32' 2023-10-30T13:20:38.705 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/22' 2023-10-30T13:20:38.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/25' 2023-10-30T13:20:38.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/30' 2023-10-30T13:20:38.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/33' 2023-10-30T13:20:38.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/23' 2023-10-30T13:20:38.706 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/35' 2023-10-30T13:20:38.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/26' 2023-10-30T13:20:38.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './258/37' 2023-10-30T13:20:38.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './258' 2023-10-30T13:20:38.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/21' 2023-10-30T13:20:38.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/27' 2023-10-30T13:20:38.707 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/31' 2023-10-30T13:20:38.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/36' 2023-10-30T13:20:38.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/20' 2023-10-30T13:20:38.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/39' 2023-10-30T13:20:38.708 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/34' 2023-10-30T13:20:38.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/38' 2023-10-30T13:20:38.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/28' 2023-10-30T13:20:38.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/24' 2023-10-30T13:20:38.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/29' 2023-10-30T13:20:38.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/32' 2023-10-30T13:20:38.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/22' 2023-10-30T13:20:38.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/25' 2023-10-30T13:20:38.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/30' 2023-10-30T13:20:38.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/33' 2023-10-30T13:20:38.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/23' 2023-10-30T13:20:38.710 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/35' 2023-10-30T13:20:38.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/26' 2023-10-30T13:20:38.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './259/37' 2023-10-30T13:20:38.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './259' 2023-10-30T13:20:38.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/21' 2023-10-30T13:20:38.711 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/27' 2023-10-30T13:20:38.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/31' 2023-10-30T13:20:38.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/36' 2023-10-30T13:20:38.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/20' 2023-10-30T13:20:38.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/39' 2023-10-30T13:20:38.712 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/34' 2023-10-30T13:20:38.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/38' 2023-10-30T13:20:38.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/28' 2023-10-30T13:20:38.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/24' 2023-10-30T13:20:38.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/29' 2023-10-30T13:20:38.713 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/32' 2023-10-30T13:20:38.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/22' 2023-10-30T13:20:38.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/25' 2023-10-30T13:20:38.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/30' 2023-10-30T13:20:38.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/33' 2023-10-30T13:20:38.714 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/23' 2023-10-30T13:20:38.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/35' 2023-10-30T13:20:38.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/26' 2023-10-30T13:20:38.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './260/37' 2023-10-30T13:20:38.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './260' 2023-10-30T13:20:38.715 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/21' 2023-10-30T13:20:38.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/27' 2023-10-30T13:20:38.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/31' 2023-10-30T13:20:38.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/36' 2023-10-30T13:20:38.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/20' 2023-10-30T13:20:38.716 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/39' 2023-10-30T13:20:38.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/34' 2023-10-30T13:20:38.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/38' 2023-10-30T13:20:38.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/28' 2023-10-30T13:20:38.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/24' 2023-10-30T13:20:38.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/29' 2023-10-30T13:20:38.717 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/32' 2023-10-30T13:20:38.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/22' 2023-10-30T13:20:38.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/25' 2023-10-30T13:20:38.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/30' 2023-10-30T13:20:38.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/33' 2023-10-30T13:20:38.718 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/23' 2023-10-30T13:20:38.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/35' 2023-10-30T13:20:38.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/26' 2023-10-30T13:20:38.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './261/37' 2023-10-30T13:20:38.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './261' 2023-10-30T13:20:38.719 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/21' 2023-10-30T13:20:38.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/27' 2023-10-30T13:20:38.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/31' 2023-10-30T13:20:38.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/36' 2023-10-30T13:20:38.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/20' 2023-10-30T13:20:38.720 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/39' 2023-10-30T13:20:38.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/34' 2023-10-30T13:20:38.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/38' 2023-10-30T13:20:38.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/28' 2023-10-30T13:20:38.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/24' 2023-10-30T13:20:38.721 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/29' 2023-10-30T13:20:38.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/32' 2023-10-30T13:20:38.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/22' 2023-10-30T13:20:38.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/25' 2023-10-30T13:20:38.722 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/30' 2023-10-30T13:20:38.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/33' 2023-10-30T13:20:38.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/23' 2023-10-30T13:20:38.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/35' 2023-10-30T13:20:38.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/26' 2023-10-30T13:20:38.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed './262/37' 2023-10-30T13:20:38.723 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './262' 2023-10-30T13:20:38.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/21' 2023-10-30T13:20:38.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/27' 2023-10-30T13:20:38.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/31' 2023-10-30T13:20:38.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/36' 2023-10-30T13:20:38.724 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/20' 2023-10-30T13:20:38.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/39' 2023-10-30T13:20:38.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/34' 2023-10-30T13:20:38.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/38' 2023-10-30T13:20:38.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/28' 2023-10-30T13:20:38.725 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/24' 2023-10-30T13:20:38.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/29' 2023-10-30T13:20:38.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/32' 2023-10-30T13:20:38.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/22' 2023-10-30T13:20:38.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/25' 2023-10-30T13:20:38.726 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/30' 2023-10-30T13:20:38.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/33' 2023-10-30T13:20:38.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/23' 2023-10-30T13:20:38.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/35' 2023-10-30T13:20:38.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/26' 2023-10-30T13:20:38.727 INFO:tasks.workunit.client.0.smithi148.stdout:removed './263/37' 2023-10-30T13:20:38.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './263' 2023-10-30T13:20:38.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/21' 2023-10-30T13:20:38.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/27' 2023-10-30T13:20:38.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/31' 2023-10-30T13:20:38.728 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/36' 2023-10-30T13:20:38.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/20' 2023-10-30T13:20:38.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/39' 2023-10-30T13:20:38.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/34' 2023-10-30T13:20:38.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/38' 2023-10-30T13:20:38.729 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/28' 2023-10-30T13:20:38.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/24' 2023-10-30T13:20:38.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/29' 2023-10-30T13:20:38.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/32' 2023-10-30T13:20:38.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/22' 2023-10-30T13:20:38.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/25' 2023-10-30T13:20:38.730 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/30' 2023-10-30T13:20:38.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/33' 2023-10-30T13:20:38.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/23' 2023-10-30T13:20:38.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/35' 2023-10-30T13:20:38.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/26' 2023-10-30T13:20:38.731 INFO:tasks.workunit.client.0.smithi148.stdout:removed './264/37' 2023-10-30T13:20:38.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './264' 2023-10-30T13:20:38.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/21' 2023-10-30T13:20:38.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/27' 2023-10-30T13:20:38.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/31' 2023-10-30T13:20:38.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/36' 2023-10-30T13:20:38.732 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/20' 2023-10-30T13:20:38.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/39' 2023-10-30T13:20:38.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/34' 2023-10-30T13:20:38.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/38' 2023-10-30T13:20:38.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/28' 2023-10-30T13:20:38.733 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/24' 2023-10-30T13:20:38.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/29' 2023-10-30T13:20:38.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/32' 2023-10-30T13:20:38.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/22' 2023-10-30T13:20:38.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/25' 2023-10-30T13:20:38.734 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/30' 2023-10-30T13:20:38.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/33' 2023-10-30T13:20:38.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/23' 2023-10-30T13:20:38.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/35' 2023-10-30T13:20:38.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/26' 2023-10-30T13:20:38.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed './265/37' 2023-10-30T13:20:38.735 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './265' 2023-10-30T13:20:38.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/21' 2023-10-30T13:20:38.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/27' 2023-10-30T13:20:38.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/31' 2023-10-30T13:20:38.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/36' 2023-10-30T13:20:38.736 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/20' 2023-10-30T13:20:38.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/39' 2023-10-30T13:20:38.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/34' 2023-10-30T13:20:38.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/38' 2023-10-30T13:20:38.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/28' 2023-10-30T13:20:38.737 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/24' 2023-10-30T13:20:38.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/29' 2023-10-30T13:20:38.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/32' 2023-10-30T13:20:38.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/22' 2023-10-30T13:20:38.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/25' 2023-10-30T13:20:38.738 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/30' 2023-10-30T13:20:38.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/33' 2023-10-30T13:20:38.739 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/23' 2023-10-30T13:20:38.740 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:40.741 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:40.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.887+0000 7f4b3b59e640 1 Processor -- start 2023-10-30T13:20:40.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3b59e640 1 -- start start 2023-10-30T13:20:40.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3b59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b3c1538f0 0x7f4b3c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:40.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3b59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4b3c0cefa0 0x7f4b3c0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:40.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3b59e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0ca6a0 con 0x7f4b3c15cd50 2023-10-30T13:20:40.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3b59e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c0cf8f0 con 0x7f4b3c0cefa0 2023-10-30T13:20:40.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3b59e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c15d300 con 0x7f4b3c1538f0 2023-10-30T13:20:40.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3ad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4b3c0cefa0 0x7f4b3c0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:40.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3ad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4b3c0cefa0 0x7f4b3c0cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44848/0 (socket says 172.21.15.148:44848) 2023-10-30T13:20:40.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b39d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4b3c15cd50 0x7f4b3c0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46178/0 (socket says 172.21.15.148:46178) 2023-10-30T13:20:40.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3ad9d640 1 -- 172.21.15.148:0/197742005 learned_addr learned my addr 172.21.15.148:0/197742005 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:40.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3a59c640 1 --2- 172.21.15.148:0/197742005 >> v2:172.21.15.150:3300/0 conn(0x7f4b3c1538f0 0x7f4b3c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:40.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3ad9d640 1 -- 172.21.15.148:0/197742005 >> v2:172.21.15.150:3300/0 conn(0x7f4b3c1538f0 msgr2=0x7f4b3c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:40.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3ad9d640 1 --2- 172.21.15.148:0/197742005 >> v2:172.21.15.150:3300/0 conn(0x7f4b3c1538f0 0x7f4b3c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:40.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3ad9d640 1 -- 172.21.15.148:0/197742005 >> v1:172.21.15.148:6789/0 conn(0x7f4b3c15cd50 legacy=0x7f4b3c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:40.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3959a640 1 -- 172.21.15.148:0/197742005 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921902845 0 0) 0x7f4b3c0ca6a0 con 0x7f4b3c15cd50 2023-10-30T13:20:40.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3ad9d640 1 -- 172.21.15.148:0/197742005 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c15d5e0 con 0x7f4b3c0cefa0 2023-10-30T13:20:40.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.891+0000 7f4b3a59c640 1 --2- 172.21.15.148:0/197742005 >> v2:172.21.15.150:3300/0 conn(0x7f4b3c1538f0 0x7f4b3c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:40.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3ad9d640 1 --2- 172.21.15.148:0/197742005 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c0cefa0 0x7f4b3c0cf3b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0c9cb0 tx=0x7f4b3401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9d1c2f1ed789d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:40.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3959a640 1 -- 172.21.15.148:0/197742005 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b340af400 con 0x7f4b3c0cefa0 2023-10-30T13:20:40.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3959a640 1 -- 172.21.15.148:0/197742005 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b340ab050 con 0x7f4b3c0cefa0 2023-10-30T13:20:40.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3959a640 1 -- 172.21.15.148:0/197742005 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b340af780 con 0x7f4b3c0cefa0 2023-10-30T13:20:40.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3b59e640 1 -- 172.21.15.148:0/197742005 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c0cefa0 msgr2=0x7f4b3c0cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:40.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3b59e640 1 --2- 172.21.15.148:0/197742005 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c0cefa0 0x7f4b3c0cf3b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0c9cb0 tx=0x7f4b3401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:20:40.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3b59e640 1 -- 172.21.15.148:0/197742005 shutdown_connections 2023-10-30T13:20:40.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3b59e640 1 --2- 172.21.15.148:0/197742005 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c0cefa0 0x7f4b3c0cf3b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:40.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3b59e640 1 --2- 172.21.15.148:0/197742005 >> v2:172.21.15.150:3300/0 conn(0x7f4b3c1538f0 0x7f4b3c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:40.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3b59e640 1 -- 172.21.15.148:0/197742005 >> 172.21.15.148:0/197742005 conn(0x7f4b3c06a640 msgr2=0x7f4b3c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3b59e640 1 -- 172.21.15.148:0/197742005 shutdown_connections 2023-10-30T13:20:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3b59e640 1 -- 172.21.15.148:0/197742005 wait complete. 2023-10-30T13:20:40.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.895+0000 7f4b3b59e640 1 Processor -- start 2023-10-30T13:20:40.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 -- start start 2023-10-30T13:20:40.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b3c0cefa0 0x7f4b3c0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:40.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3c1538f0 0x7f4b3c15a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:40.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c15cd50 0x7f4b3c15b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:40.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c152000 con 0x7f4b3c15cd50 2023-10-30T13:20:40.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c151130 con 0x7f4b3c1538f0 2023-10-30T13:20:40.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c1513f0 con 0x7f4b3c0cefa0 2023-10-30T13:20:40.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c15cd50 0x7f4b3c15b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:40.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c15cd50 0x7f4b3c15b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44854/0 (socket says 172.21.15.148:44854) 2023-10-30T13:20:40.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3ad9d640 1 -- 172.21.15.148:0/411084455 learned_addr learned my addr 172.21.15.148:0/411084455 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:40.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b39d9b640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3c1538f0 0x7f4b3c15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:40.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3a59c640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b3c0cefa0 0x7f4b3c0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:40.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3ad9d640 1 -- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b3c0cefa0 msgr2=0x7f4b3c0b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:40.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3ad9d640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b3c0cefa0 0x7f4b3c0b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:40.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3ad9d640 1 -- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3c1538f0 msgr2=0x7f4b3c15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:40.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3ad9d640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3c1538f0 0x7f4b3c15a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:40.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3ad9d640 1 -- 172.21.15.148:0/411084455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34009b30 con 0x7f4b3c15cd50 2023-10-30T13:20:40.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b39d9b640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3c1538f0 0x7f4b3c15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:40.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3ad9d640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c15cd50 0x7f4b3c15b4e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4b340b2040 tx=0x7f4b3401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:40.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b2affd640 1 -- 172.21.15.148:0/411084455 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b340afde0 con 0x7f4b3c15cd50 2023-10-30T13:20:40.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b2affd640 1 -- 172.21.15.148:0/411084455 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b34009510 con 0x7f4b3c15cd50 2023-10-30T13:20:40.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b2affd640 1 -- 172.21.15.148:0/411084455 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b340ba660 con 0x7f4b3c15cd50 2023-10-30T13:20:40.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c1516d0 con 0x7f4b3c15cd50 2023-10-30T13:20:40.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c151c90 con 0x7f4b3c15cd50 2023-10-30T13:20:40.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.899+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c04e170 con 0x7f4b3c15cd50 2023-10-30T13:20:40.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.903+0000 7f4b2affd640 1 -- 172.21.15.148:0/411084455 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b340b6040 con 0x7f4b3c15cd50 2023-10-30T13:20:40.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.903+0000 7f4b2affd640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b1c0664a0 0x7f4b1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:40.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.903+0000 7f4b2affd640 1 -- 172.21.15.148:0/411084455 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f4b34122310 con 0x7f4b3c15cd50 2023-10-30T13:20:40.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.903+0000 7f4b3a59c640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b1c0664a0 0x7f4b1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:40.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.903+0000 7f4b3a59c640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b1c0664a0 0x7f4b1c068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00c900 tx=0x7f4b2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:40.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:40.907+0000 7f4b2affd640 1 -- 172.21.15.148:0/411084455 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b340ef590 con 0x7f4b3c15cd50 2023-10-30T13:20:41.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.103+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b3c0c48d0 con 0x7f4b3c15cd50 2023-10-30T13:20:41.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.103+0000 7f4b2affd640 1 -- 172.21.15.148:0/411084455 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f4b340f2bb0 con 0x7f4b3c15cd50 2023-10-30T13:20:41.104 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:41.104 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:31.554781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:41.105 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 51 2023-10-30T13:20:41.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.103+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b1c0664a0 msgr2=0x7f4b1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:41.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.103+0000 7f4b3b59e640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b1c0664a0 0x7f4b1c068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00c900 tx=0x7f4b2c008040 comp rx=0 tx=0).stop 2023-10-30T13:20:41.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c15cd50 msgr2=0x7f4b3c15b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:41.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c15cd50 0x7f4b3c15b4e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4b340b2040 tx=0x7f4b3401bf60 comp rx=0 tx=0).stop 2023-10-30T13:20:41.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 shutdown_connections 2023-10-30T13:20:41.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b1c0664a0 0x7f4b1c068960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:41.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b3c15cd50 0x7f4b3c15b4e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:41.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3c1538f0 0x7f4b3c15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:41.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 --2- 172.21.15.148:0/411084455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b3c0cefa0 0x7f4b3c0b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:41.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 >> 172.21.15.148:0/411084455 conn(0x7f4b3c06a640 msgr2=0x7f4b3c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:41.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 shutdown_connections 2023-10-30T13:20:41.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:41.107+0000 7f4b3b59e640 1 -- 172.21.15.148:0/411084455 wait complete. 2023-10-30T13:20:41.125 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:41.391 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/35' 2023-10-30T13:20:41.391 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/26' 2023-10-30T13:20:41.391 INFO:tasks.workunit.client.0.smithi148.stdout:removed './266/37' 2023-10-30T13:20:41.391 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './266' 2023-10-30T13:20:41.391 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/21' 2023-10-30T13:20:41.392 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/27' 2023-10-30T13:20:41.392 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/31' 2023-10-30T13:20:41.392 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/36' 2023-10-30T13:20:41.392 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/20' 2023-10-30T13:20:41.392 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/39' 2023-10-30T13:20:41.392 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/34' 2023-10-30T13:20:41.393 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/38' 2023-10-30T13:20:41.393 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/28' 2023-10-30T13:20:41.393 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/24' 2023-10-30T13:20:41.393 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/29' 2023-10-30T13:20:41.393 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/32' 2023-10-30T13:20:41.393 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/22' 2023-10-30T13:20:41.394 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/25' 2023-10-30T13:20:41.394 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/30' 2023-10-30T13:20:41.394 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/33' 2023-10-30T13:20:41.394 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/23' 2023-10-30T13:20:41.394 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/35' 2023-10-30T13:20:41.395 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/26' 2023-10-30T13:20:41.395 INFO:tasks.workunit.client.0.smithi148.stdout:removed './267/37' 2023-10-30T13:20:41.395 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './267' 2023-10-30T13:20:41.395 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/21' 2023-10-30T13:20:41.395 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/27' 2023-10-30T13:20:41.395 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/31' 2023-10-30T13:20:41.396 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/36' 2023-10-30T13:20:41.396 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/20' 2023-10-30T13:20:41.396 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/39' 2023-10-30T13:20:41.396 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/34' 2023-10-30T13:20:41.396 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/38' 2023-10-30T13:20:41.397 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/28' 2023-10-30T13:20:41.397 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/24' 2023-10-30T13:20:41.397 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/29' 2023-10-30T13:20:41.397 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/32' 2023-10-30T13:20:41.397 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/22' 2023-10-30T13:20:41.397 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/25' 2023-10-30T13:20:41.398 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/30' 2023-10-30T13:20:41.398 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/33' 2023-10-30T13:20:41.398 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/23' 2023-10-30T13:20:41.398 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/35' 2023-10-30T13:20:41.398 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/26' 2023-10-30T13:20:41.398 INFO:tasks.workunit.client.0.smithi148.stdout:removed './268/37' 2023-10-30T13:20:41.399 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './268' 2023-10-30T13:20:41.399 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/21' 2023-10-30T13:20:41.399 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/27' 2023-10-30T13:20:41.399 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/31' 2023-10-30T13:20:41.399 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/36' 2023-10-30T13:20:41.399 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/20' 2023-10-30T13:20:41.400 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/39' 2023-10-30T13:20:41.400 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/34' 2023-10-30T13:20:41.400 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/38' 2023-10-30T13:20:41.400 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/28' 2023-10-30T13:20:41.400 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/24' 2023-10-30T13:20:41.400 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/29' 2023-10-30T13:20:41.401 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/32' 2023-10-30T13:20:41.401 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/22' 2023-10-30T13:20:41.401 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/25' 2023-10-30T13:20:41.401 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/30' 2023-10-30T13:20:41.401 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/33' 2023-10-30T13:20:41.401 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/23' 2023-10-30T13:20:41.402 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/35' 2023-10-30T13:20:41.402 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/26' 2023-10-30T13:20:41.402 INFO:tasks.workunit.client.0.smithi148.stdout:removed './269/37' 2023-10-30T13:20:41.402 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './269' 2023-10-30T13:20:41.402 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/21' 2023-10-30T13:20:41.402 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/27' 2023-10-30T13:20:41.403 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/31' 2023-10-30T13:20:41.403 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/36' 2023-10-30T13:20:41.403 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/20' 2023-10-30T13:20:41.403 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/39' 2023-10-30T13:20:41.403 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/34' 2023-10-30T13:20:41.403 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/38' 2023-10-30T13:20:41.404 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/28' 2023-10-30T13:20:41.404 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/24' 2023-10-30T13:20:41.404 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/29' 2023-10-30T13:20:41.404 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/32' 2023-10-30T13:20:41.404 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/22' 2023-10-30T13:20:41.404 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/25' 2023-10-30T13:20:41.405 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/30' 2023-10-30T13:20:41.405 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/33' 2023-10-30T13:20:41.405 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/23' 2023-10-30T13:20:41.405 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/35' 2023-10-30T13:20:41.405 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/26' 2023-10-30T13:20:41.405 INFO:tasks.workunit.client.0.smithi148.stdout:removed './270/37' 2023-10-30T13:20:41.406 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './270' 2023-10-30T13:20:41.406 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/21' 2023-10-30T13:20:41.406 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/27' 2023-10-30T13:20:41.406 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/31' 2023-10-30T13:20:41.406 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/36' 2023-10-30T13:20:41.407 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/20' 2023-10-30T13:20:41.407 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/39' 2023-10-30T13:20:41.407 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/34' 2023-10-30T13:20:41.407 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/38' 2023-10-30T13:20:41.407 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/28' 2023-10-30T13:20:41.407 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/24' 2023-10-30T13:20:41.408 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/29' 2023-10-30T13:20:41.408 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/32' 2023-10-30T13:20:41.408 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/22' 2023-10-30T13:20:41.408 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/25' 2023-10-30T13:20:41.408 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/30' 2023-10-30T13:20:41.409 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/33' 2023-10-30T13:20:41.409 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/23' 2023-10-30T13:20:41.409 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/35' 2023-10-30T13:20:41.409 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/26' 2023-10-30T13:20:41.409 INFO:tasks.workunit.client.0.smithi148.stdout:removed './271/37' 2023-10-30T13:20:41.409 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './271' 2023-10-30T13:20:41.410 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/21' 2023-10-30T13:20:41.410 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/27' 2023-10-30T13:20:41.410 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/31' 2023-10-30T13:20:41.410 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/36' 2023-10-30T13:20:41.410 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/20' 2023-10-30T13:20:41.410 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/39' 2023-10-30T13:20:41.411 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/34' 2023-10-30T13:20:41.411 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/38' 2023-10-30T13:20:41.411 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/28' 2023-10-30T13:20:41.411 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/24' 2023-10-30T13:20:41.411 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/29' 2023-10-30T13:20:41.411 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/32' 2023-10-30T13:20:41.412 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/22' 2023-10-30T13:20:41.412 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/25' 2023-10-30T13:20:41.412 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/30' 2023-10-30T13:20:41.412 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/33' 2023-10-30T13:20:41.412 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/23' 2023-10-30T13:20:41.412 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/35' 2023-10-30T13:20:41.413 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/26' 2023-10-30T13:20:41.413 INFO:tasks.workunit.client.0.smithi148.stdout:removed './272/37' 2023-10-30T13:20:41.413 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './272' 2023-10-30T13:20:41.413 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/21' 2023-10-30T13:20:41.413 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/27' 2023-10-30T13:20:41.413 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/31' 2023-10-30T13:20:41.413 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/36' 2023-10-30T13:20:41.414 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/20' 2023-10-30T13:20:41.414 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/39' 2023-10-30T13:20:41.414 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/34' 2023-10-30T13:20:41.414 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/38' 2023-10-30T13:20:41.414 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/28' 2023-10-30T13:20:41.414 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/24' 2023-10-30T13:20:41.415 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/29' 2023-10-30T13:20:41.415 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/32' 2023-10-30T13:20:41.415 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/22' 2023-10-30T13:20:41.415 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/25' 2023-10-30T13:20:41.415 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/30' 2023-10-30T13:20:41.415 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/33' 2023-10-30T13:20:41.416 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/23' 2023-10-30T13:20:41.416 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/35' 2023-10-30T13:20:41.416 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/26' 2023-10-30T13:20:41.416 INFO:tasks.workunit.client.0.smithi148.stdout:removed './273/37' 2023-10-30T13:20:41.416 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './273' 2023-10-30T13:20:41.416 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/21' 2023-10-30T13:20:41.417 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/27' 2023-10-30T13:20:41.417 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/31' 2023-10-30T13:20:41.417 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/36' 2023-10-30T13:20:41.417 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/20' 2023-10-30T13:20:41.417 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/39' 2023-10-30T13:20:41.418 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/34' 2023-10-30T13:20:41.418 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/38' 2023-10-30T13:20:41.418 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/28' 2023-10-30T13:20:41.418 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/24' 2023-10-30T13:20:41.418 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/29' 2023-10-30T13:20:41.418 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/32' 2023-10-30T13:20:41.419 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/22' 2023-10-30T13:20:41.419 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/25' 2023-10-30T13:20:41.419 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/30' 2023-10-30T13:20:41.419 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/33' 2023-10-30T13:20:41.419 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/23' 2023-10-30T13:20:41.419 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/35' 2023-10-30T13:20:41.420 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/26' 2023-10-30T13:20:41.420 INFO:tasks.workunit.client.0.smithi148.stdout:removed './274/37' 2023-10-30T13:20:41.420 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './274' 2023-10-30T13:20:41.420 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/21' 2023-10-30T13:20:41.420 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/27' 2023-10-30T13:20:41.420 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/31' 2023-10-30T13:20:41.421 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/36' 2023-10-30T13:20:41.421 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/20' 2023-10-30T13:20:41.421 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/39' 2023-10-30T13:20:41.421 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/34' 2023-10-30T13:20:41.421 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/38' 2023-10-30T13:20:41.422 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/28' 2023-10-30T13:20:41.422 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/24' 2023-10-30T13:20:41.422 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/29' 2023-10-30T13:20:41.422 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/32' 2023-10-30T13:20:41.422 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/22' 2023-10-30T13:20:41.422 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/25' 2023-10-30T13:20:41.423 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/30' 2023-10-30T13:20:41.423 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/33' 2023-10-30T13:20:41.423 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/23' 2023-10-30T13:20:41.423 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/35' 2023-10-30T13:20:41.423 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/26' 2023-10-30T13:20:41.423 INFO:tasks.workunit.client.0.smithi148.stdout:removed './275/37' 2023-10-30T13:20:41.424 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './275' 2023-10-30T13:20:41.424 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/21' 2023-10-30T13:20:41.424 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/27' 2023-10-30T13:20:41.424 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/31' 2023-10-30T13:20:41.424 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/36' 2023-10-30T13:20:41.425 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/20' 2023-10-30T13:20:41.425 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/39' 2023-10-30T13:20:41.425 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/34' 2023-10-30T13:20:41.425 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/38' 2023-10-30T13:20:41.425 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/28' 2023-10-30T13:20:41.425 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/24' 2023-10-30T13:20:41.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/29' 2023-10-30T13:20:41.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/32' 2023-10-30T13:20:41.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/22' 2023-10-30T13:20:41.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/25' 2023-10-30T13:20:41.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/30' 2023-10-30T13:20:41.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/33' 2023-10-30T13:20:41.427 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/23' 2023-10-30T13:20:41.427 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/35' 2023-10-30T13:20:41.427 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/26' 2023-10-30T13:20:43.126 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.271+0000 7f8202017640 1 Processor -- start 2023-10-30T13:20:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.271+0000 7f8202017640 1 -- start start 2023-10-30T13:20:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.271+0000 7f8202017640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81fc15d6f0 0x7f81fc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f8202017640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f81fc0cf160 0x7f81fc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f8202017640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc0cb800 con 0x7f81fc1538a0 2023-10-30T13:20:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f8202017640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81fc0cb330 con 0x7f81fc0cf160 2023-10-30T13:20:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f8202017640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81fc0cad70 con 0x7f81fc15d6f0 2023-10-30T13:20:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fbfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f81fc0cf160 0x7f81fc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fbfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f81fc0cf160 0x7f81fc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37280/0 (socket says 172.21.15.148:37280) 2023-10-30T13:20:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fbfff640 1 -- 172.21.15.148:0/164093610 learned_addr learned my addr 172.21.15.148:0/164093610 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fb7fe640 1 --1- 172.21.15.148:0/164093610 >> v1:172.21.15.150:6789/0 conn(0x7f81fc1538a0 0x7f81fc0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:55624/0 (socket says 172.21.15.148:55624) 2023-10-30T13:20:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81faffd640 1 --2- 172.21.15.148:0/164093610 >> v2:172.21.15.150:3300/0 conn(0x7f81fc15d6f0 0x7f81fc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fbfff640 1 -- 172.21.15.148:0/164093610 >> v2:172.21.15.150:3300/0 conn(0x7f81fc15d6f0 msgr2=0x7f81fc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fbfff640 1 --2- 172.21.15.148:0/164093610 >> v2:172.21.15.150:3300/0 conn(0x7f81fc15d6f0 0x7f81fc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fbfff640 1 -- 172.21.15.148:0/164093610 >> v1:172.21.15.150:6789/0 conn(0x7f81fc1538a0 legacy=0x7f81fc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fbfff640 1 -- 172.21.15.148:0/164093610 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc0cb050 con 0x7f81fc0cf160 2023-10-30T13:20:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fbfff640 1 --2- 172.21.15.148:0/164093610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc0cf160 0x7f81fc0cf590 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0b30a0 tx=0x7f81f401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa998504ff89115 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fa7fc640 1 -- 172.21.15.148:0/164093610 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f40af400 con 0x7f81fc0cf160 2023-10-30T13:20:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.275+0000 7f81fa7fc640 1 -- 172.21.15.148:0/164093610 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f40ab050 con 0x7f81fc0cf160 2023-10-30T13:20:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f81fa7fc640 1 -- 172.21.15.148:0/164093610 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f40af780 con 0x7f81fc0cf160 2023-10-30T13:20:43.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 -- 172.21.15.148:0/164093610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc0cf160 msgr2=0x7f81fc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:43.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 --2- 172.21.15.148:0/164093610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc0cf160 0x7f81fc0cf590 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0b30a0 tx=0x7f81f401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 -- 172.21.15.148:0/164093610 shutdown_connections 2023-10-30T13:20:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 --2- 172.21.15.148:0/164093610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc0cf160 0x7f81fc0cf590 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 --2- 172.21.15.148:0/164093610 >> v2:172.21.15.150:3300/0 conn(0x7f81fc15d6f0 0x7f81fc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 -- 172.21.15.148:0/164093610 >> 172.21.15.148:0/164093610 conn(0x7f81fc06bfe0 msgr2=0x7f81fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 -- 172.21.15.148:0/164093610 shutdown_connections 2023-10-30T13:20:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 -- 172.21.15.148:0/164093610 wait complete. 2023-10-30T13:20:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 Processor -- start 2023-10-30T13:20:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 -- start start 2023-10-30T13:20:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81fc0cf160 0x7f81fc1f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc1538a0 0x7f81fc0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc15d6f0 0x7f81fc0b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc14dc30 con 0x7f81fc15d6f0 2023-10-30T13:20:43.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc1f2fc0 con 0x7f81fc1538a0 2023-10-30T13:20:43.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.279+0000 7f8202017640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc1e2a40 con 0x7f81fc0cf160 2023-10-30T13:20:43.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc15d6f0 0x7f81fc0b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:43.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc15d6f0 0x7f81fc0b3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37288/0 (socket says 172.21.15.148:37288) 2023-10-30T13:20:43.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fbfff640 1 -- 172.21.15.148:0/1690886024 learned_addr learned my addr 172.21.15.148:0/1690886024 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:43.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81faffd640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc1538a0 0x7f81fc0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:43.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fb7fe640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81fc0cf160 0x7f81fc1f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:43.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fbfff640 1 -- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81fc0cf160 msgr2=0x7f81fc1f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:43.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fbfff640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81fc0cf160 0x7f81fc1f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fbfff640 1 -- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc1538a0 msgr2=0x7f81fc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:43.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fbfff640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc1538a0 0x7f81fc0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fbfff640 1 -- 172.21.15.148:0/1690886024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f4009b30 con 0x7f81fc15d6f0 2023-10-30T13:20:43.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81faffd640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc1538a0 0x7f81fc0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:43.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fb7fe640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81fc0cf160 0x7f81fc1f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:43.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81fbfff640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc15d6f0 0x7f81fc0b3ac0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009f10 tx=0x7f81f401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:43.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81e3fff640 1 -- 172.21.15.148:0/1690886024 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f40afd90 con 0x7f81fc15d6f0 2023-10-30T13:20:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81e3fff640 1 -- 172.21.15.148:0/1690886024 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f4009510 con 0x7f81fc15d6f0 2023-10-30T13:20:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81e3fff640 1 -- 172.21.15.148:0/1690886024 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f40b9610 con 0x7f81fc15d6f0 2023-10-30T13:20:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81fc1e5790 con 0x7f81fc15d6f0 2023-10-30T13:20:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81fc1e2c90 con 0x7f81fc15d6f0 2023-10-30T13:20:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81fc0b2a50 con 0x7f81fc15d6f0 2023-10-30T13:20:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81e3fff640 1 -- 172.21.15.148:0/1690886024 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f81f40b6070 con 0x7f81fc15d6f0 2023-10-30T13:20:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.283+0000 7f81e3fff640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81e80664a0 0x7f81e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.287+0000 7f81e3fff640 1 -- 172.21.15.148:0/1690886024 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f81f4121f60 con 0x7f81fc15d6f0 2023-10-30T13:20:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.287+0000 7f81fb7fe640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81e80664a0 0x7f81e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.287+0000 7f81fb7fe640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81e80664a0 0x7f81e8068960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f81ec000f70 tx=0x7f81ec007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:43.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.287+0000 7f81e3fff640 1 -- 172.21.15.148:0/1690886024 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f81f40ef1e0 con 0x7f81fc15d6f0 2023-10-30T13:20:43.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.479+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81fc04e170 con 0x7f81fc15d6f0 2023-10-30T13:20:43.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.479+0000 7f81e3fff640 1 -- 172.21.15.148:0/1690886024 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f81f40f2800 con 0x7f81fc15d6f0 2023-10-30T13:20:43.482 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:43.482 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:43.482 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:20:43.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81e80664a0 msgr2=0x7f81e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:43.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81e80664a0 0x7f81e8068960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f81ec000f70 tx=0x7f81ec007d50 comp rx=0 tx=0).stop 2023-10-30T13:20:43.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc15d6f0 msgr2=0x7f81fc0b3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:43.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc15d6f0 0x7f81fc0b3ac0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009f10 tx=0x7f81f401bf60 comp rx=0 tx=0).stop 2023-10-30T13:20:43.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 shutdown_connections 2023-10-30T13:20:43.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81e80664a0 0x7f81e8068960 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81fc15d6f0 0x7f81fc0b3ac0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc1538a0 0x7f81fc0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 --2- 172.21.15.148:0/1690886024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81fc0cf160 0x7f81fc1f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:43.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 >> 172.21.15.148:0/1690886024 conn(0x7f81fc06bfe0 msgr2=0x7f81fc1514e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:43.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 shutdown_connections 2023-10-30T13:20:43.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:43.483+0000 7f8202017640 1 -- 172.21.15.148:0/1690886024 wait complete. 2023-10-30T13:20:43.502 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:20:43.503 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5033": { "addr": "172.21.15.150:6836/953984429", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 953984429, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 953984429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5033, "incarnation": 36, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:20:41.562473+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5033, "mds_1": 5103 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6837/2690362018", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 2690362018, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 2690362018, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 51, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5529, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:20:44.290 INFO:tasks.workunit.client.0.smithi148.stdout:removed './276/37' 2023-10-30T13:20:44.290 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './276' 2023-10-30T13:20:44.290 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/21' 2023-10-30T13:20:44.290 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/27' 2023-10-30T13:20:44.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/31' 2023-10-30T13:20:44.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/36' 2023-10-30T13:20:44.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/20' 2023-10-30T13:20:44.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/39' 2023-10-30T13:20:44.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/34' 2023-10-30T13:20:44.291 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/38' 2023-10-30T13:20:44.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/28' 2023-10-30T13:20:44.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/24' 2023-10-30T13:20:44.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/29' 2023-10-30T13:20:44.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/32' 2023-10-30T13:20:44.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/22' 2023-10-30T13:20:44.292 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/25' 2023-10-30T13:20:44.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/30' 2023-10-30T13:20:44.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/33' 2023-10-30T13:20:44.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/23' 2023-10-30T13:20:44.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/35' 2023-10-30T13:20:44.293 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/26' 2023-10-30T13:20:44.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './277/37' 2023-10-30T13:20:44.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './277' 2023-10-30T13:20:44.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/21' 2023-10-30T13:20:44.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/27' 2023-10-30T13:20:44.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/31' 2023-10-30T13:20:44.294 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/36' 2023-10-30T13:20:44.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/20' 2023-10-30T13:20:44.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/39' 2023-10-30T13:20:44.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/34' 2023-10-30T13:20:44.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/38' 2023-10-30T13:20:44.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/28' 2023-10-30T13:20:44.295 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/24' 2023-10-30T13:20:44.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/29' 2023-10-30T13:20:44.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/32' 2023-10-30T13:20:44.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/22' 2023-10-30T13:20:44.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/25' 2023-10-30T13:20:44.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/30' 2023-10-30T13:20:44.296 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/33' 2023-10-30T13:20:44.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/23' 2023-10-30T13:20:44.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/35' 2023-10-30T13:20:44.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/26' 2023-10-30T13:20:44.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './278/37' 2023-10-30T13:20:44.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './278' 2023-10-30T13:20:44.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/21' 2023-10-30T13:20:44.297 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/27' 2023-10-30T13:20:44.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/31' 2023-10-30T13:20:44.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/36' 2023-10-30T13:20:44.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/20' 2023-10-30T13:20:44.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/39' 2023-10-30T13:20:44.298 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/34' 2023-10-30T13:20:44.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/38' 2023-10-30T13:20:44.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/28' 2023-10-30T13:20:44.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/24' 2023-10-30T13:20:44.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/29' 2023-10-30T13:20:44.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/32' 2023-10-30T13:20:44.299 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/22' 2023-10-30T13:20:44.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/25' 2023-10-30T13:20:44.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/30' 2023-10-30T13:20:44.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/33' 2023-10-30T13:20:44.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/23' 2023-10-30T13:20:44.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/35' 2023-10-30T13:20:44.300 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/26' 2023-10-30T13:20:44.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './279/37' 2023-10-30T13:20:44.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './279' 2023-10-30T13:20:44.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/21' 2023-10-30T13:20:44.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/27' 2023-10-30T13:20:44.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/31' 2023-10-30T13:20:44.301 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/36' 2023-10-30T13:20:44.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/20' 2023-10-30T13:20:44.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/39' 2023-10-30T13:20:44.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/34' 2023-10-30T13:20:44.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/38' 2023-10-30T13:20:44.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/28' 2023-10-30T13:20:44.302 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/24' 2023-10-30T13:20:44.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/29' 2023-10-30T13:20:44.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/32' 2023-10-30T13:20:44.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/22' 2023-10-30T13:20:44.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/25' 2023-10-30T13:20:44.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/30' 2023-10-30T13:20:44.303 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/33' 2023-10-30T13:20:44.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/23' 2023-10-30T13:20:44.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/35' 2023-10-30T13:20:44.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/26' 2023-10-30T13:20:44.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './280/37' 2023-10-30T13:20:44.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './280' 2023-10-30T13:20:44.304 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/21' 2023-10-30T13:20:44.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/27' 2023-10-30T13:20:44.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/31' 2023-10-30T13:20:44.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/36' 2023-10-30T13:20:44.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/20' 2023-10-30T13:20:44.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/39' 2023-10-30T13:20:44.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/34' 2023-10-30T13:20:44.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/38' 2023-10-30T13:20:44.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/28' 2023-10-30T13:20:44.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/24' 2023-10-30T13:20:44.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/29' 2023-10-30T13:20:44.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/32' 2023-10-30T13:20:44.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/22' 2023-10-30T13:20:44.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/25' 2023-10-30T13:20:44.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/30' 2023-10-30T13:20:44.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/33' 2023-10-30T13:20:44.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/23' 2023-10-30T13:20:44.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/35' 2023-10-30T13:20:44.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/26' 2023-10-30T13:20:44.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './281/37' 2023-10-30T13:20:44.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './281' 2023-10-30T13:20:44.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/21' 2023-10-30T13:20:44.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/27' 2023-10-30T13:20:44.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/31' 2023-10-30T13:20:44.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/36' 2023-10-30T13:20:44.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/20' 2023-10-30T13:20:44.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/39' 2023-10-30T13:20:44.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/34' 2023-10-30T13:20:44.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/38' 2023-10-30T13:20:44.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/28' 2023-10-30T13:20:44.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/24' 2023-10-30T13:20:44.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/29' 2023-10-30T13:20:44.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/32' 2023-10-30T13:20:44.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/22' 2023-10-30T13:20:44.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/25' 2023-10-30T13:20:44.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/30' 2023-10-30T13:20:44.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/33' 2023-10-30T13:20:44.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/23' 2023-10-30T13:20:44.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/35' 2023-10-30T13:20:44.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/26' 2023-10-30T13:20:44.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './282/37' 2023-10-30T13:20:44.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './282' 2023-10-30T13:20:44.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/21' 2023-10-30T13:20:44.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/27' 2023-10-30T13:20:44.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/31' 2023-10-30T13:20:44.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/36' 2023-10-30T13:20:44.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/20' 2023-10-30T13:20:44.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/39' 2023-10-30T13:20:44.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/34' 2023-10-30T13:20:44.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/38' 2023-10-30T13:20:44.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/28' 2023-10-30T13:20:44.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/24' 2023-10-30T13:20:44.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/29' 2023-10-30T13:20:44.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/32' 2023-10-30T13:20:44.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/22' 2023-10-30T13:20:44.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/25' 2023-10-30T13:20:44.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/30' 2023-10-30T13:20:44.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/33' 2023-10-30T13:20:44.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/23' 2023-10-30T13:20:44.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/35' 2023-10-30T13:20:44.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/26' 2023-10-30T13:20:44.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './283/37' 2023-10-30T13:20:44.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './283' 2023-10-30T13:20:44.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/21' 2023-10-30T13:20:44.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/27' 2023-10-30T13:20:44.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/31' 2023-10-30T13:20:44.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/36' 2023-10-30T13:20:44.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/20' 2023-10-30T13:20:44.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/39' 2023-10-30T13:20:44.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/34' 2023-10-30T13:20:44.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/38' 2023-10-30T13:20:44.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/28' 2023-10-30T13:20:44.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/24' 2023-10-30T13:20:44.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/29' 2023-10-30T13:20:44.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/32' 2023-10-30T13:20:44.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/22' 2023-10-30T13:20:44.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/25' 2023-10-30T13:20:44.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/30' 2023-10-30T13:20:44.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/33' 2023-10-30T13:20:44.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/23' 2023-10-30T13:20:44.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/35' 2023-10-30T13:20:44.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/26' 2023-10-30T13:20:44.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './284/37' 2023-10-30T13:20:44.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './284' 2023-10-30T13:20:44.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/21' 2023-10-30T13:20:44.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/27' 2023-10-30T13:20:44.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/31' 2023-10-30T13:20:44.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/36' 2023-10-30T13:20:44.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/20' 2023-10-30T13:20:44.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/39' 2023-10-30T13:20:44.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/34' 2023-10-30T13:20:44.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/38' 2023-10-30T13:20:44.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/28' 2023-10-30T13:20:44.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/24' 2023-10-30T13:20:44.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/29' 2023-10-30T13:20:44.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/32' 2023-10-30T13:20:44.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/22' 2023-10-30T13:20:44.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/25' 2023-10-30T13:20:44.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/30' 2023-10-30T13:20:44.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/33' 2023-10-30T13:20:44.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/23' 2023-10-30T13:20:44.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/35' 2023-10-30T13:20:44.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/26' 2023-10-30T13:20:44.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './285/37' 2023-10-30T13:20:44.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './285' 2023-10-30T13:20:44.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/21' 2023-10-30T13:20:44.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/27' 2023-10-30T13:20:44.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/31' 2023-10-30T13:20:44.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/36' 2023-10-30T13:20:44.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/20' 2023-10-30T13:20:44.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/39' 2023-10-30T13:20:44.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/34' 2023-10-30T13:20:44.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/38' 2023-10-30T13:20:44.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/28' 2023-10-30T13:20:44.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/24' 2023-10-30T13:20:44.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/29' 2023-10-30T13:20:44.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/32' 2023-10-30T13:20:44.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/22' 2023-10-30T13:20:44.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/25' 2023-10-30T13:20:44.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/30' 2023-10-30T13:20:44.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/33' 2023-10-30T13:20:44.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/23' 2023-10-30T13:20:44.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/35' 2023-10-30T13:20:44.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/26' 2023-10-30T13:20:44.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './286/37' 2023-10-30T13:20:44.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './286' 2023-10-30T13:20:45.504 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:45.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.643+0000 7f27d9454640 1 Processor -- start 2023-10-30T13:20:45.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.643+0000 7f27d9454640 1 -- start start 2023-10-30T13:20:45.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d9454640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27d40c8cf0 0x7f27d40c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:45.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d9454640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c90d0 0x7f27d40ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:45.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d9454640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d415ce30 con 0x7f27d40cef90 2023-10-30T13:20:45.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d9454640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d4159880 con 0x7f27d40c8cf0 2023-10-30T13:20:45.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d9454640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4159ae0 con 0x7f27d40c90d0 2023-10-30T13:20:45.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27ca7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c90d0 0x7f27d40ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:45.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d2ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27d40c8cf0 0x7f27d40c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:45.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f27d40cef90 0x7f27d415d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:55626/0 (socket says 172.21.15.148:55626) 2023-10-30T13:20:45.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d37fe640 1 -- 172.21.15.148:0/3735533168 learned_addr learned my addr 172.21.15.148:0/3735533168 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:45.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d2ffd640 1 -- 172.21.15.148:0/3735533168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c90d0 msgr2=0x7f27d40ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:45.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d2ffd640 1 --2- 172.21.15.148:0/3735533168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c90d0 0x7f27d40ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d2ffd640 1 -- 172.21.15.148:0/3735533168 >> v1:172.21.15.150:6789/0 conn(0x7f27d40cef90 legacy=0x7f27d415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:45.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.647+0000 7f27d2ffd640 1 -- 172.21.15.148:0/3735533168 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d4159dc0 con 0x7f27d40c8cf0 2023-10-30T13:20:45.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d2ffd640 1 --2- 172.21.15.148:0/3735533168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d40c8cf0 0x7f27d40c9e00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f27c40097c0 tx=0x7f27c401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ccbce33dd6ef8b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d27fc640 1 -- 172.21.15.148:0/3735533168 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c40ab650 con 0x7f27d40c8cf0 2023-10-30T13:20:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d27fc640 1 -- 172.21.15.148:0/3735533168 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c40ab7b0 con 0x7f27d40c8cf0 2023-10-30T13:20:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d27fc640 1 -- 172.21.15.148:0/3735533168 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c40abb40 con 0x7f27d40c8cf0 2023-10-30T13:20:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d9454640 1 -- 172.21.15.148:0/3735533168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d40c8cf0 msgr2=0x7f27d40c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d9454640 1 --2- 172.21.15.148:0/3735533168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d40c8cf0 0x7f27d40c9e00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f27c40097c0 tx=0x7f27c401c450 comp rx=0 tx=0).stop 2023-10-30T13:20:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d9454640 1 -- 172.21.15.148:0/3735533168 shutdown_connections 2023-10-30T13:20:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d9454640 1 --2- 172.21.15.148:0/3735533168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c90d0 0x7f27d40ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d9454640 1 --2- 172.21.15.148:0/3735533168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d40c8cf0 0x7f27d40c9e00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d9454640 1 -- 172.21.15.148:0/3735533168 >> 172.21.15.148:0/3735533168 conn(0x7f27d406be40 msgr2=0x7f27d406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d9454640 1 -- 172.21.15.148:0/3735533168 shutdown_connections 2023-10-30T13:20:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d9454640 1 -- 172.21.15.148:0/3735533168 wait complete. 2023-10-30T13:20:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.651+0000 7f27d9454640 1 Processor -- start 2023-10-30T13:20:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 -- start start 2023-10-30T13:20:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c8cf0 0x7f27d41f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27d40c90d0 0x7f27d40b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d40cef90 0x7f27d41eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d41f63c0 con 0x7f27d40c90d0 2023-10-30T13:20:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d41f2040 con 0x7f27d40cef90 2023-10-30T13:20:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d41f2320 con 0x7f27d40c8cf0 2023-10-30T13:20:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27ca7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27d40c90d0 0x7f27d40b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27ca7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27d40c90d0 0x7f27d40b2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37290/0 (socket says 172.21.15.148:37290) 2023-10-30T13:20:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27ca7fc640 1 -- 172.21.15.148:0/2827052704 learned_addr learned my addr 172.21.15.148:0/2827052704 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d37fe640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d40cef90 0x7f27d41eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d2ffd640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c8cf0 0x7f27d41f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27ca7fc640 1 -- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c8cf0 msgr2=0x7f27d41f5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27ca7fc640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c8cf0 0x7f27d41f5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27ca7fc640 1 -- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d40cef90 msgr2=0x7f27d41eb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27ca7fc640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d40cef90 0x7f27d41eb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27ca7fc640 1 -- 172.21.15.148:0/2827052704 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c4009420 con 0x7f27d40c90d0 2023-10-30T13:20:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27ca7fc640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27d40c90d0 0x7f27d40b2d00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00e930 tx=0x7f27bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27cbfff640 1 -- 172.21.15.148:0/2827052704 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc009690 con 0x7f27d40c90d0 2023-10-30T13:20:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27cbfff640 1 -- 172.21.15.148:0/2827052704 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc00ca00 con 0x7f27d40c90d0 2023-10-30T13:20:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27cbfff640 1 -- 172.21.15.148:0/2827052704 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00cd70 con 0x7f27d40c90d0 2023-10-30T13:20:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d41f2600 con 0x7f27d40c90d0 2023-10-30T13:20:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d40c8770 con 0x7f27d40c90d0 2023-10-30T13:20:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.655+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d41f2e00 con 0x7f27d40c90d0 2023-10-30T13:20:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.659+0000 7f27cbfff640 1 -- 172.21.15.148:0/2827052704 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f27bc013470 con 0x7f27d40c90d0 2023-10-30T13:20:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.659+0000 7f27cbfff640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f27b40664a0 0x7f27b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.659+0000 7f27cbfff640 1 -- 172.21.15.148:0/2827052704 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f27bc0843c0 con 0x7f27d40c90d0 2023-10-30T13:20:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.659+0000 7f27d2ffd640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f27b40664a0 0x7f27b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.659+0000 7f27d2ffd640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f27b40664a0 0x7f27b4068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f27c40af000 tx=0x7f27c40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.663+0000 7f27cbfff640 1 -- 172.21.15.148:0/2827052704 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f27d41f2e00 con 0x7f27d40c90d0 2023-10-30T13:20:45.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.843+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27d41f2e00 con 0x7f27d40c90d0 2023-10-30T13:20:45.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.843+0000 7f27cbfff640 1 -- 172.21.15.148:0/2827052704 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f27d41f2e00 con 0x7f27d40c90d0 2023-10-30T13:20:45.845 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:45.846 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:45.846 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:20:45.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f27b40664a0 msgr2=0x7f27b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:45.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f27b40664a0 0x7f27b4068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f27c40af000 tx=0x7f27c40a7000 comp rx=0 tx=0).stop 2023-10-30T13:20:45.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27d40c90d0 msgr2=0x7f27d40b2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:45.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27d40c90d0 0x7f27d40b2d00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00e930 tx=0x7f27bc00edf0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 shutdown_connections 2023-10-30T13:20:45.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f27b40664a0 0x7f27b4068960 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27d40c90d0 0x7f27d40b2d00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d40cef90 0x7f27d41eb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 --2- 172.21.15.148:0/2827052704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27d40c8cf0 0x7f27d41f5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:45.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 >> 172.21.15.148:0/2827052704 conn(0x7f27d406be40 msgr2=0x7f27d41537f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:45.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 shutdown_connections 2023-10-30T13:20:45.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:45.847+0000 7f27d9454640 1 -- 172.21.15.148:0/2827052704 wait complete. 2023-10-30T13:20:45.866 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:47.867 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:48.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f434d965640 1 Processor -- start 2023-10-30T13:20:48.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f434d965640 1 -- start start 2023-10-30T13:20:48.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f434d965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43481538a0 0x7f434815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:48.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f434d965640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f43480ca0e0 0x7f434815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:48.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f434d965640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f43480cf160 0x7f43480cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:48.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f434d965640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43480cb800 con 0x7f43480cf160 2023-10-30T13:20:48.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f434d965640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434815dc30 con 0x7f43480ca0e0 2023-10-30T13:20:48.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f434d965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43480cb330 con 0x7f43481538a0 2023-10-30T13:20:48.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43477fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f43480cf160 0x7f43480cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:48.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43477fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f43480cf160 0x7f43480cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37306/0 (socket says 172.21.15.148:37306) 2023-10-30T13:20:48.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43477fe640 1 -- 172.21.15.148:0/1648428220 learned_addr learned my addr 172.21.15.148:0/1648428220 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:48.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f4346ffd640 1 --2- 172.21.15.148:0/1648428220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43481538a0 0x7f434815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:48.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43467fc640 1 --2- 172.21.15.148:0/1648428220 >> v2:172.21.15.150:3300/0 conn(0x7f43480ca0e0 0x7f434815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:48.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43477fe640 1 -- 172.21.15.148:0/1648428220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43481538a0 msgr2=0x7f434815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:48.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43477fe640 1 --2- 172.21.15.148:0/1648428220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43481538a0 0x7f434815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43477fe640 1 -- 172.21.15.148:0/1648428220 >> v2:172.21.15.150:3300/0 conn(0x7f43480ca0e0 msgr2=0x7f434815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:48.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43477fe640 1 --2- 172.21.15.148:0/1648428220 >> v2:172.21.15.150:3300/0 conn(0x7f43480ca0e0 0x7f434815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43477fe640 1 -- 172.21.15.148:0/1648428220 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43480cad70 con 0x7f43480cf160 2023-10-30T13:20:48.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.019+0000 7f43467fc640 1 --2- 172.21.15.148:0/1648428220 >> v2:172.21.15.150:3300/0 conn(0x7f43480ca0e0 0x7f434815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:48.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f43477fe640 1 --2- 172.21.15.148:0/1648428220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43480cf160 0x7f43480cf590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f43480ccd10 tx=0x7f434001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a9d41828a14a8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:48.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f4345ffb640 1 -- 172.21.15.148:0/1648428220 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400af400 con 0x7f43480cf160 2023-10-30T13:20:48.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f4345ffb640 1 -- 172.21.15.148:0/1648428220 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43400ab050 con 0x7f43480cf160 2023-10-30T13:20:48.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f4345ffb640 1 -- 172.21.15.148:0/1648428220 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400af780 con 0x7f43480cf160 2023-10-30T13:20:48.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f434d965640 1 -- 172.21.15.148:0/1648428220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43480cf160 msgr2=0x7f43480cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:48.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f434d965640 1 --2- 172.21.15.148:0/1648428220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43480cf160 0x7f43480cf590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f43480ccd10 tx=0x7f434001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f434d965640 1 -- 172.21.15.148:0/1648428220 shutdown_connections 2023-10-30T13:20:48.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f434d965640 1 --2- 172.21.15.148:0/1648428220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43480cf160 0x7f43480cf590 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f434d965640 1 --2- 172.21.15.148:0/1648428220 >> v2:172.21.15.150:3300/0 conn(0x7f43480ca0e0 0x7f434815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f434d965640 1 --2- 172.21.15.148:0/1648428220 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43481538a0 0x7f434815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f434d965640 1 -- 172.21.15.148:0/1648428220 >> 172.21.15.148:0/1648428220 conn(0x7f434806bfe0 msgr2=0x7f434806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:48.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f434d965640 1 -- 172.21.15.148:0/1648428220 shutdown_connections 2023-10-30T13:20:48.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.023+0000 7f434d965640 1 -- 172.21.15.148:0/1648428220 wait complete. 2023-10-30T13:20:48.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 Processor -- start 2023-10-30T13:20:48.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 -- start start 2023-10-30T13:20:48.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43480ca0e0 0x7f43481efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:48.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43480cf160 0x7f43481e5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:48.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43481538a0 0x7f43481e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:48.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434815afa0 con 0x7f43481538a0 2023-10-30T13:20:48.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434815a9e0 con 0x7f43480cf160 2023-10-30T13:20:48.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434815acc0 con 0x7f43480ca0e0 2023-10-30T13:20:48.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43477fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43481538a0 0x7f43481e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:48.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43477fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43481538a0 0x7f43481e2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37318/0 (socket says 172.21.15.148:37318) 2023-10-30T13:20:48.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43477fe640 1 -- 172.21.15.148:0/2097719095 learned_addr learned my addr 172.21.15.148:0/2097719095 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:48.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f4346ffd640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43480ca0e0 0x7f43481efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:48.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43467fc640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43480cf160 0x7f43481e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:48.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43477fe640 1 -- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43480ca0e0 msgr2=0x7f43481efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:48.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43477fe640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43480ca0e0 0x7f43481efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43477fe640 1 -- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43480cf160 msgr2=0x7f43481e5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:48.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43477fe640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43480cf160 0x7f43481e5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43477fe640 1 -- 172.21.15.148:0/2097719095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4340009b30 con 0x7f43481538a0 2023-10-30T13:20:48.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f43477fe640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43481538a0 0x7f43481e2ae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f43400b2040 tx=0x7f434001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:48.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f432ffff640 1 -- 172.21.15.148:0/2097719095 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400afd10 con 0x7f43481538a0 2023-10-30T13:20:48.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f432ffff640 1 -- 172.21.15.148:0/2097719095 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43400afe90 con 0x7f43481538a0 2023-10-30T13:20:48.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43481ef080 con 0x7f43481538a0 2023-10-30T13:20:48.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f432ffff640 1 -- 172.21.15.148:0/2097719095 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400ba660 con 0x7f43481538a0 2023-10-30T13:20:48.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.027+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43481ef640 con 0x7f43481538a0 2023-10-30T13:20:48.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.031+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43480b2a50 con 0x7f43481538a0 2023-10-30T13:20:48.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.031+0000 7f432ffff640 1 -- 172.21.15.148:0/2097719095 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f43400b6040 con 0x7f43481538a0 2023-10-30T13:20:48.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.031+0000 7f432ffff640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43340664a0 0x7f4334068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:48.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.031+0000 7f432ffff640 1 -- 172.21.15.148:0/2097719095 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f43401223c0 con 0x7f43481538a0 2023-10-30T13:20:48.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.031+0000 7f4346ffd640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43340664a0 0x7f4334068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:48.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.031+0000 7f4346ffd640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43340664a0 0x7f4334068960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f43380079c0 tx=0x7f4338008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:48.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.035+0000 7f432ffff640 1 -- 172.21.15.148:0/2097719095 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43400ef6f0 con 0x7f43481538a0 2023-10-30T13:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.215+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4348159830 con 0x7f43481538a0 2023-10-30T13:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.215+0000 7f432ffff640 1 -- 172.21.15.148:0/2097719095 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f4340009510 con 0x7f43481538a0 2023-10-30T13:20:48.216 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:48.216 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.215+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43340664a0 msgr2=0x7f4334068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.215+0000 7f434d965640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43340664a0 0x7f4334068960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f43380079c0 tx=0x7f4338008040 comp rx=0 tx=0).stop 2023-10-30T13:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.215+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43481538a0 msgr2=0x7f43481e2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.215+0000 7f434d965640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43481538a0 0x7f43481e2ae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f43400b2040 tx=0x7f434001bf60 comp rx=0 tx=0).stop 2023-10-30T13:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.219+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 shutdown_connections 2023-10-30T13:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.219+0000 7f434d965640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43340664a0 0x7f4334068960 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.219+0000 7f434d965640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43481538a0 0x7f43481e2ae0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.219+0000 7f434d965640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43480cf160 0x7f43481e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.219+0000 7f434d965640 1 --2- 172.21.15.148:0/2097719095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f43480ca0e0 0x7f43481efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.219+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 >> 172.21.15.148:0/2097719095 conn(0x7f434806bfe0 msgr2=0x7f43481526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.219+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 shutdown_connections 2023-10-30T13:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:48.219+0000 7f434d965640 1 -- 172.21.15.148:0/2097719095 wait complete. 2023-10-30T13:20:48.236 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:50.237 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b32c6b640 1 Processor -- start 2023-10-30T13:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b32c6b640 1 -- start start 2023-10-30T13:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b32c6b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b2c0c6b80 0x7f9b2c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b32c6b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b2c0c6f60 0x7f9b2c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b32c6b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c0c0a50 con 0x7f9b2c0c8430 2023-10-30T13:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b32c6b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2c06f6e0 con 0x7f9b2c0c6b80 2023-10-30T13:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b32c6b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b2c06f9c0 con 0x7f9b2c0c6f60 2023-10-30T13:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b30a07640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b2c0c6b80 0x7f9b2c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b30a07640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b2c0c6b80 0x7f9b2c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37334/0 (socket says 172.21.15.148:37334) 2023-10-30T13:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b30a07640 1 -- 172.21.15.148:0/1045530586 learned_addr learned my addr 172.21.15.148:0/1045530586 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b31208640 1 --1- 172.21.15.148:0/1045530586 >> v1:172.21.15.148:6789/0 conn(0x7f9b2c0c8430 0x7f9b2c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39628/0 (socket says 172.21.15.148:39628) 2023-10-30T13:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b23fff640 1 --2- 172.21.15.148:0/1045530586 >> v2:172.21.15.150:3300/0 conn(0x7f9b2c0c6f60 0x7f9b2c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b30a07640 1 -- 172.21.15.148:0/1045530586 >> v2:172.21.15.150:3300/0 conn(0x7f9b2c0c6f60 msgr2=0x7f9b2c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b30a07640 1 --2- 172.21.15.148:0/1045530586 >> v2:172.21.15.150:3300/0 conn(0x7f9b2c0c6f60 0x7f9b2c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b30a07640 1 -- 172.21.15.148:0/1045530586 >> v1:172.21.15.148:6789/0 conn(0x7f9b2c0c8430 legacy=0x7f9b2c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b30a07640 1 -- 172.21.15.148:0/1045530586 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c06fca0 con 0x7f9b2c0c6b80 2023-10-30T13:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b23fff640 1 --2- 172.21.15.148:0/1045530586 >> v2:172.21.15.150:3300/0 conn(0x7f9b2c0c6f60 0x7f9b2c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.391+0000 7f9b237fe640 1 -- 172.21.15.148:0/1045530586 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545851725 0 0) 0x7f9b2c0c0a50 con 0x7f9b2c0c8430 2023-10-30T13:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b30a07640 1 --2- 172.21.15.148:0/1045530586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c6b80 0x7f9b2c0bfec0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c159b90 tx=0x7f9b2801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3866d26d6b67994e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b237fe640 1 -- 172.21.15.148:0/1045530586 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b280ab610 con 0x7f9b2c0c6b80 2023-10-30T13:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b237fe640 1 -- 172.21.15.148:0/1045530586 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b280ab770 con 0x7f9b2c0c6b80 2023-10-30T13:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b237fe640 1 -- 172.21.15.148:0/1045530586 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b280abb00 con 0x7f9b2c0c6b80 2023-10-30T13:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b32c6b640 1 -- 172.21.15.148:0/1045530586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c6b80 msgr2=0x7f9b2c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b32c6b640 1 --2- 172.21.15.148:0/1045530586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c6b80 0x7f9b2c0bfec0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c159b90 tx=0x7f9b2801c500 comp rx=0 tx=0).stop 2023-10-30T13:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b32c6b640 1 -- 172.21.15.148:0/1045530586 shutdown_connections 2023-10-30T13:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b32c6b640 1 --2- 172.21.15.148:0/1045530586 >> v2:172.21.15.150:3300/0 conn(0x7f9b2c0c6f60 0x7f9b2c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b32c6b640 1 --2- 172.21.15.148:0/1045530586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c6b80 0x7f9b2c0bfec0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b32c6b640 1 -- 172.21.15.148:0/1045530586 >> 172.21.15.148:0/1045530586 conn(0x7f9b2c0c7910 msgr2=0x7f9b2c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b32c6b640 1 -- 172.21.15.148:0/1045530586 shutdown_connections 2023-10-30T13:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.395+0000 7f9b32c6b640 1 -- 172.21.15.148:0/1045530586 wait complete. 2023-10-30T13:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 Processor -- start 2023-10-30T13:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 -- start start 2023-10-30T13:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b2c0c6b80 0x7f9b2c155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b2c0c6f60 0x7f9b2c14a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c8430 0x7f9b2c14fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c0c93a0 con 0x7f9b2c0c8430 2023-10-30T13:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c0c9c70 con 0x7f9b2c0c6b80 2023-10-30T13:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2c14a210 con 0x7f9b2c0c6f60 2023-10-30T13:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b31208640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c8430 0x7f9b2c14fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:50.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b31208640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c8430 0x7f9b2c14fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37338/0 (socket says 172.21.15.148:37338) 2023-10-30T13:20:50.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b31208640 1 -- 172.21.15.148:0/3762170742 learned_addr learned my addr 172.21.15.148:0/3762170742 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:50.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b23fff640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b2c0c6f60 0x7f9b2c14a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:50.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b30a07640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b2c0c6b80 0x7f9b2c155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:50.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b31208640 1 -- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b2c0c6f60 msgr2=0x7f9b2c14a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:50.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b31208640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b2c0c6f60 0x7f9b2c14a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:50.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b31208640 1 -- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b2c0c6b80 msgr2=0x7f9b2c155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:50.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b31208640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b2c0c6b80 0x7f9b2c155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:50.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b31208640 1 -- 172.21.15.148:0/3762170742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b280094d0 con 0x7f9b2c0c8430 2023-10-30T13:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b31208640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c8430 0x7f9b2c14fe50 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9b2400cb20 tx=0x7f9b24002140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b217fa640 1 -- 172.21.15.148:0/3762170742 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2400a700 con 0x7f9b2c0c8430 2023-10-30T13:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b217fa640 1 -- 172.21.15.148:0/3762170742 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2400a860 con 0x7f9b2c0c8430 2023-10-30T13:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2c14e8f0 con 0x7f9b2c0c8430 2023-10-30T13:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b217fa640 1 -- 172.21.15.148:0/3762170742 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2400ac10 con 0x7f9b2c0c8430 2023-10-30T13:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2c149600 con 0x7f9b2c0c8430 2023-10-30T13:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.399+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2c14a3a0 con 0x7f9b2c0c8430 2023-10-30T13:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.403+0000 7f9b217fa640 1 -- 172.21.15.148:0/3762170742 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b24006070 con 0x7f9b2c0c8430 2023-10-30T13:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.403+0000 7f9b217fa640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b18066470 0x7f9b18068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.403+0000 7f9b217fa640 1 -- 172.21.15.148:0/3762170742 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f9b24083a30 con 0x7f9b2c0c8430 2023-10-30T13:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.403+0000 7f9b30a07640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b18066470 0x7f9b18068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.403+0000 7f9b30a07640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b18066470 0x7f9b18068930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9b280af000 tx=0x7f9b280b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:50.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.407+0000 7f9b217fa640 1 -- 172.21.15.148:0/3762170742 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b24050d60 con 0x7f9b2c0c8430 2023-10-30T13:20:50.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b2c071ae0 con 0x7f9b2c0c8430 2023-10-30T13:20:50.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b217fa640 1 -- 172.21.15.148:0/3762170742 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f9b24017070 con 0x7f9b2c0c8430 2023-10-30T13:20:50.587 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:50.587 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:50.587 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:20:50.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b18066470 msgr2=0x7f9b18068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:50.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b18066470 0x7f9b18068930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9b280af000 tx=0x7f9b280b6040 comp rx=0 tx=0).stop 2023-10-30T13:20:50.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c8430 msgr2=0x7f9b2c14fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:50.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c8430 0x7f9b2c14fe50 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9b2400cb20 tx=0x7f9b24002140 comp rx=0 tx=0).stop 2023-10-30T13:20:50.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 shutdown_connections 2023-10-30T13:20:50.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b18066470 0x7f9b18068930 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:50.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b2c0c8430 0x7f9b2c14fe50 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:50.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b2c0c6f60 0x7f9b2c14a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:50.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 --2- 172.21.15.148:0/3762170742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b2c0c6b80 0x7f9b2c155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:50.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 >> 172.21.15.148:0/3762170742 conn(0x7f9b2c0c7910 msgr2=0x7f9b2c135000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:50.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 shutdown_connections 2023-10-30T13:20:50.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:50.587+0000 7f9b32c6b640 1 -- 172.21.15.148:0/3762170742 wait complete. 2023-10-30T13:20:50.608 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:52.609 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:52.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60849e640 1 Processor -- start 2023-10-30T13:20:52.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60849e640 1 -- start start 2023-10-30T13:20:52.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60849e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe6001538a0 0x7fe60015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:52.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60849e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000cf160 0x7fe6000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:52.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60849e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6000cb800 con 0x7fe6000ca0e0 2023-10-30T13:20:52.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60849e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60015dc30 con 0x7fe6001538a0 2023-10-30T13:20:52.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60849e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000cb330 con 0x7fe6000cf160 2023-10-30T13:20:52.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60623a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe6001538a0 0x7fe60015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:52.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60623a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe6001538a0 0x7fe60015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43192/0 (socket says 172.21.15.148:43192) 2023-10-30T13:20:52.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60623a640 1 -- 172.21.15.148:0/3063210590 learned_addr learned my addr 172.21.15.148:0/3063210590 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:52.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe605a39640 1 --1- 172.21.15.148:0/3063210590 >> v1:172.21.15.148:6789/0 conn(0x7fe6000ca0e0 0x7fe60015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:50950/0 (socket says 172.21.15.148:50950) 2023-10-30T13:20:52.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe606a3b640 1 --2- 172.21.15.148:0/3063210590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000cf160 0x7fe6000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:52.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60623a640 1 -- 172.21.15.148:0/3063210590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000cf160 msgr2=0x7fe6000cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:52.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60623a640 1 --2- 172.21.15.148:0/3063210590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000cf160 0x7fe6000cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:52.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60623a640 1 -- 172.21.15.148:0/3063210590 >> v1:172.21.15.148:6789/0 conn(0x7fe6000ca0e0 legacy=0x7fe60015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:52.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.763+0000 7fe60623a640 1 -- 172.21.15.148:0/3063210590 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6000cad70 con 0x7fe6001538a0 2023-10-30T13:20:52.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe60623a640 1 --2- 172.21.15.148:0/3063210590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6001538a0 0x7fe60015d6f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009870 tx=0x7fe5fc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf7ac9dbcf0433b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:52.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe605238640 1 -- 172.21.15.148:0/3063210590 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0a8a30 con 0x7fe6001538a0 2023-10-30T13:20:52.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe605238640 1 -- 172.21.15.148:0/3063210590 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5fc0a8b90 con 0x7fe6001538a0 2023-10-30T13:20:52.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe605238640 1 -- 172.21.15.148:0/3063210590 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0b0440 con 0x7fe6001538a0 2023-10-30T13:20:52.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe60849e640 1 -- 172.21.15.148:0/3063210590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6001538a0 msgr2=0x7fe60015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:52.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe60849e640 1 --2- 172.21.15.148:0/3063210590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6001538a0 0x7fe60015d6f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009870 tx=0x7fe5fc01ed30 comp rx=0 tx=0).stop 2023-10-30T13:20:52.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe60849e640 1 -- 172.21.15.148:0/3063210590 shutdown_connections 2023-10-30T13:20:52.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe60849e640 1 --2- 172.21.15.148:0/3063210590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000cf160 0x7fe6000cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:52.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe60849e640 1 --2- 172.21.15.148:0/3063210590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6001538a0 0x7fe60015d6f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:52.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe60849e640 1 -- 172.21.15.148:0/3063210590 >> 172.21.15.148:0/3063210590 conn(0x7fe60006bfe0 msgr2=0x7fe60006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:52.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe60849e640 1 -- 172.21.15.148:0/3063210590 shutdown_connections 2023-10-30T13:20:52.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.767+0000 7fe60849e640 1 -- 172.21.15.148:0/3063210590 wait complete. 2023-10-30T13:20:52.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 Processor -- start 2023-10-30T13:20:52.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 -- start start 2023-10-30T13:20:52.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000ca0e0 0x7fe6001e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:52.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6000cf160 0x7fe60015b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:52.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6001538a0 0x7fe60015bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:52.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60014d4f0 con 0x7fe6000cf160 2023-10-30T13:20:52.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6001e4d40 con 0x7fe6001538a0 2023-10-30T13:20:52.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60015b2a0 con 0x7fe6000ca0e0 2023-10-30T13:20:52.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe605a39640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6000cf160 0x7fe60015b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:52.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe605a39640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6000cf160 0x7fe60015b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43206/0 (socket says 172.21.15.148:43206) 2023-10-30T13:20:52.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe605a39640 1 -- 172.21.15.148:0/3213772165 learned_addr learned my addr 172.21.15.148:0/3213772165 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:52.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe606a3b640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6001538a0 0x7fe60015bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:52.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60623a640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000ca0e0 0x7fe6001e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:52.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe605a39640 1 -- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000ca0e0 msgr2=0x7fe6001e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:52.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe605a39640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000ca0e0 0x7fe6001e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:52.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe605a39640 1 -- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6001538a0 msgr2=0x7fe60015bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:52.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe605a39640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6001538a0 0x7fe60015bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:52.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe605a39640 1 -- 172.21.15.148:0/3213772165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc0094d0 con 0x7fe6000cf160 2023-10-30T13:20:52.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe605a39640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6000cf160 0x7fe60015b630 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe5f000c490 tx=0x7fe5f000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:52.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe5f6ffd640 1 -- 172.21.15.148:0/3213772165 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0007480 con 0x7fe6000cf160 2023-10-30T13:20:52.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe5f6ffd640 1 -- 172.21.15.148:0/3213772165 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f00075e0 con 0x7fe6000cf160 2023-10-30T13:20:52.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe5f6ffd640 1 -- 172.21.15.148:0/3213772165 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0004a10 con 0x7fe6000cf160 2023-10-30T13:20:52.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6000cef10 con 0x7fe6000cf160 2023-10-30T13:20:52.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6000c50b0 con 0x7fe6000cf160 2023-10-30T13:20:52.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.771+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6000b2a50 con 0x7fe6000cf160 2023-10-30T13:20:52.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.775+0000 7fe5f6ffd640 1 -- 172.21.15.148:0/3213772165 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5f0004b70 con 0x7fe6000cf160 2023-10-30T13:20:52.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.775+0000 7fe5f6ffd640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5e40664a0 0x7fe5e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:52.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.775+0000 7fe5f6ffd640 1 -- 172.21.15.148:0/3213772165 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fe5f0083250 con 0x7fe6000cf160 2023-10-30T13:20:52.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.775+0000 7fe60623a640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5e40664a0 0x7fe5e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:52.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.775+0000 7fe60623a640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5e40664a0 0x7fe5e4068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0b2000 tx=0x7fe5fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:52.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.779+0000 7fe5f6ffd640 1 -- 172.21.15.148:0/3213772165 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5f00504c0 con 0x7fe6000cf160 2023-10-30T13:20:52.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe600151940 con 0x7fe6000cf160 2023-10-30T13:20:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe5f6ffd640 1 -- 172.21.15.148:0/3213772165 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fe5f0053ae0 con 0x7fe6000cf160 2023-10-30T13:20:52.959 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:52.960 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:20:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5e40664a0 msgr2=0x7fe5e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5e40664a0 0x7fe5e4068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0b2000 tx=0x7fe5fc008040 comp rx=0 tx=0).stop 2023-10-30T13:20:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6000cf160 msgr2=0x7fe60015b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6000cf160 0x7fe60015b630 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe5f000c490 tx=0x7fe5f000c950 comp rx=0 tx=0).stop 2023-10-30T13:20:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 shutdown_connections 2023-10-30T13:20:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe5e40664a0 0x7fe5e4068960 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe6000cf160 0x7fe60015b630 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6001538a0 0x7fe60015bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 --2- 172.21.15.148:0/3213772165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe6000ca0e0 0x7fe6001e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 >> 172.21.15.148:0/3213772165 conn(0x7fe60006bfe0 msgr2=0x7fe600065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 shutdown_connections 2023-10-30T13:20:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:52.959+0000 7fe60849e640 1 -- 172.21.15.148:0/3213772165 wait complete. 2023-10-30T13:20:52.985 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:54.828 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:20:54.832 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:20:54.986 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:55.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e886a640 1 Processor -- start 2023-10-30T13:20:55.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e886a640 1 -- start start 2023-10-30T13:20:55.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e886a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff4e40cefa0 0x7ff4e40cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:55.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e886a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e415a6c0 con 0x7ff4e415cd50 2023-10-30T13:20:55.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e886a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e40ca6a0 con 0x7ff4e41538f0 2023-10-30T13:20:55.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e886a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e40cf8f0 con 0x7ff4e40cefa0 2023-10-30T13:20:55.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e359e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff4e40cefa0 0x7ff4e40cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:55.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e259c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff4e415cd50 0x7ff4e40c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:50952/0 (socket says 172.21.15.148:50952) 2023-10-30T13:20:55.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e359e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff4e40cefa0 0x7ff4e40cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43220/0 (socket says 172.21.15.148:43220) 2023-10-30T13:20:55.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e259c640 1 -- 172.21.15.148:0/109440667 learned_addr learned my addr 172.21.15.148:0/109440667 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:55.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e359e640 1 -- 172.21.15.148:0/109440667 >> v1:172.21.15.150:6789/0 conn(0x7ff4e41538f0 legacy=0x7ff4e4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:55.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e359e640 1 -- 172.21.15.148:0/109440667 >> v1:172.21.15.148:6789/0 conn(0x7ff4e415cd50 legacy=0x7ff4e40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:55.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e359e640 1 -- 172.21.15.148:0/109440667 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e415d300 con 0x7ff4e40cefa0 2023-10-30T13:20:55.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.135+0000 7ff4e1d9b640 1 -- 172.21.15.148:0/109440667 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671565965 0 0) 0x7ff4e415a6c0 con 0x7ff4e415cd50 2023-10-30T13:20:55.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e359e640 1 --2- 172.21.15.148:0/109440667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e40cefa0 0x7ff4e40cf3b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009d90 tx=0x7ff4dc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58d92c26d08bb24 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:55.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e1d9b640 1 -- 172.21.15.148:0/109440667 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0af600 con 0x7ff4e40cefa0 2023-10-30T13:20:55.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e1d9b640 1 -- 172.21.15.148:0/109440667 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc0ab050 con 0x7ff4e40cefa0 2023-10-30T13:20:55.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e1d9b640 1 -- 172.21.15.148:0/109440667 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0af980 con 0x7ff4e40cefa0 2023-10-30T13:20:55.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e886a640 1 -- 172.21.15.148:0/109440667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e40cefa0 msgr2=0x7ff4e40cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:55.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e886a640 1 --2- 172.21.15.148:0/109440667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e40cefa0 0x7ff4e40cf3b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009d90 tx=0x7ff4dc01f130 comp rx=0 tx=0).stop 2023-10-30T13:20:55.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e886a640 1 -- 172.21.15.148:0/109440667 shutdown_connections 2023-10-30T13:20:55.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e886a640 1 --2- 172.21.15.148:0/109440667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e40cefa0 0x7ff4e40cf3b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:55.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e886a640 1 -- 172.21.15.148:0/109440667 >> 172.21.15.148:0/109440667 conn(0x7ff4e406a640 msgr2=0x7ff4e4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:55.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e886a640 1 -- 172.21.15.148:0/109440667 shutdown_connections 2023-10-30T13:20:55.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.139+0000 7ff4e886a640 1 -- 172.21.15.148:0/109440667 wait complete. 2023-10-30T13:20:55.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 Processor -- start 2023-10-30T13:20:55.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 -- start start 2023-10-30T13:20:55.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4e40cefa0 0x7ff4e40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:55.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e41538f0 0x7ff4e415a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:55.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e415cd50 0x7ff4e4144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:55.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e415c5c0 con 0x7ff4e415cd50 2023-10-30T13:20:55.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4152b60 con 0x7ff4e41538f0 2023-10-30T13:20:55.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40b4460 con 0x7ff4e40cefa0 2023-10-30T13:20:55.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e359e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e415cd50 0x7ff4e4144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:55.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e359e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e415cd50 0x7ff4e4144900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43228/0 (socket says 172.21.15.148:43228) 2023-10-30T13:20:55.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e359e640 1 -- 172.21.15.148:0/1053260811 learned_addr learned my addr 172.21.15.148:0/1053260811 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:55.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e259c640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e41538f0 0x7ff4e415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:55.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e2d9d640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4e40cefa0 0x7ff4e40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:55.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e359e640 1 -- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4e40cefa0 msgr2=0x7ff4e40b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:55.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e359e640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4e40cefa0 0x7ff4e40b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:55.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e359e640 1 -- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e41538f0 msgr2=0x7ff4e415a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:55.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e359e640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e41538f0 0x7ff4e415a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:55.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e359e640 1 -- 172.21.15.148:0/1053260811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc009a70 con 0x7ff4e415cd50 2023-10-30T13:20:55.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e259c640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e41538f0 0x7ff4e415a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:55.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e359e640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e415cd50 0x7ff4e4144900 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc01cd60 tx=0x7ff4dc01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:55.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4cb7fe640 1 -- 172.21.15.148:0/1053260811 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0039a0 con 0x7ff4e415cd50 2023-10-30T13:20:55.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4cb7fe640 1 -- 172.21.15.148:0/1053260811 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc003b00 con 0x7ff4e415cd50 2023-10-30T13:20:55.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e40b3240 con 0x7ff4e415cd50 2023-10-30T13:20:55.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4cb7fe640 1 -- 172.21.15.148:0/1053260811 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc003e70 con 0x7ff4e415cd50 2023-10-30T13:20:55.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e40b49e0 con 0x7ff4e415cd50 2023-10-30T13:20:55.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.143+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e404e170 con 0x7ff4e415cd50 2023-10-30T13:20:55.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.147+0000 7ff4cb7fe640 1 -- 172.21.15.148:0/1053260811 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4dc004030 con 0x7ff4e415cd50 2023-10-30T13:20:55.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.147+0000 7ff4cb7fe640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4d00664d0 0x7ff4d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:55.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.147+0000 7ff4cb7fe640 1 -- 172.21.15.148:0/1053260811 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7ff4dc122dc0 con 0x7ff4e415cd50 2023-10-30T13:20:55.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.147+0000 7ff4e2d9d640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4d00664d0 0x7ff4d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:55.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.147+0000 7ff4e2d9d640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4d00664d0 0x7ff4d0068990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4000f70 tx=0x7ff4d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:55.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.151+0000 7ff4cb7fe640 1 -- 172.21.15.148:0/1053260811 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4dc0f3bf0 con 0x7ff4e415cd50 2023-10-30T13:20:55.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.331+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4e40b2290 con 0x7ff4e415cd50 2023-10-30T13:20:55.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.331+0000 7ff4cb7fe640 1 -- 172.21.15.148:0/1053260811 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7ff4dc0b4090 con 0x7ff4e415cd50 2023-10-30T13:20:55.332 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:55.333 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:55.333 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:20:55.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4d00664d0 msgr2=0x7ff4d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:55.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4d00664d0 0x7ff4d0068990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4000f70 tx=0x7ff4d4009040 comp rx=0 tx=0).stop 2023-10-30T13:20:55.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e415cd50 msgr2=0x7ff4e4144900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:55.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e415cd50 0x7ff4e4144900 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc01cd60 tx=0x7ff4dc01bed0 comp rx=0 tx=0).stop 2023-10-30T13:20:55.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 shutdown_connections 2023-10-30T13:20:55.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4d00664d0 0x7ff4d0068990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:55.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4e415cd50 0x7ff4e4144900 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:55.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e41538f0 0x7ff4e415a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:55.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 --2- 172.21.15.148:0/1053260811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4e40cefa0 0x7ff4e40b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:55.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 >> 172.21.15.148:0/1053260811 conn(0x7ff4e406a640 msgr2=0x7ff4e40cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:55.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 shutdown_connections 2023-10-30T13:20:55.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:55.335+0000 7ff4e886a640 1 -- 172.21.15.148:0/1053260811 wait complete. 2023-10-30T13:20:55.353 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:20:55.354 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5033": { "addr": "172.21.15.150:6836/953984429", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 953984429, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 953984429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5033, "incarnation": 36, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:20:41.562473+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5033, "mds_1": 5103 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6837/2690362018", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 2690362018, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 2690362018, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 51, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5529, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:20:57.355 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:57.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3c5b9d640 1 Processor -- start 2023-10-30T13:20:57.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3c5b9d640 1 -- start start 2023-10-30T13:20:57.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3c5b9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c015d6f0 0x7fa3c015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:57.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3c5b9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3c00cf160 0x7fa3c00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:57.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3c5b9d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c00cb800 con 0x7fa3c01538a0 2023-10-30T13:20:57.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3c5b9d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c00cb330 con 0x7fa3c00cf160 2023-10-30T13:20:57.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3c5b9d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00cad70 con 0x7fa3c015d6f0 2023-10-30T13:20:57.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3bffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3c00cf160 0x7fa3c00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:57.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3b7fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c015d6f0 0x7fa3c015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:57.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3bf7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa3c01538a0 0x7fa3c00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:42858/0 (socket says 172.21.15.148:42858) 2023-10-30T13:20:57.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.502+0000 7fa3bf7fe640 1 -- 172.21.15.148:0/2349772040 learned_addr learned my addr 172.21.15.148:0/2349772040 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:20:57.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3bffff640 1 -- 172.21.15.148:0/2349772040 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c015d6f0 msgr2=0x7fa3c015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:57.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3bffff640 1 --2- 172.21.15.148:0/2349772040 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c015d6f0 0x7fa3c015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3bffff640 1 -- 172.21.15.148:0/2349772040 >> v1:172.21.15.150:6789/0 conn(0x7fa3c01538a0 legacy=0x7fa3c00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:57.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3bffff640 1 -- 172.21.15.148:0/2349772040 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c00cb050 con 0x7fa3c00cf160 2023-10-30T13:20:57.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3bffff640 1 --2- 172.21.15.148:0/2349772040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c00cf160 0x7fa3c00cf590 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8009d90 tx=0x7fa3b801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b49f129d9c8e3d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:57.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3beffd640 1 -- 172.21.15.148:0/2349772040 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b80af600 con 0x7fa3c00cf160 2023-10-30T13:20:57.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3beffd640 1 -- 172.21.15.148:0/2349772040 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b80ab050 con 0x7fa3c00cf160 2023-10-30T13:20:57.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3beffd640 1 -- 172.21.15.148:0/2349772040 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b80af980 con 0x7fa3c00cf160 2023-10-30T13:20:57.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2349772040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c00cf160 msgr2=0x7fa3c00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:57.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.506+0000 7fa3c5b9d640 1 --2- 172.21.15.148:0/2349772040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c00cf160 0x7fa3c00cf590 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8009d90 tx=0x7fa3b801f130 comp rx=0 tx=0).stop 2023-10-30T13:20:57.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2349772040 shutdown_connections 2023-10-30T13:20:57.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 --2- 172.21.15.148:0/2349772040 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c015d6f0 0x7fa3c015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 --2- 172.21.15.148:0/2349772040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c00cf160 0x7fa3c00cf590 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2349772040 >> 172.21.15.148:0/2349772040 conn(0x7fa3c006bfe0 msgr2=0x7fa3c006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:57.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2349772040 shutdown_connections 2023-10-30T13:20:57.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2349772040 wait complete. 2023-10-30T13:20:57.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 Processor -- start 2023-10-30T13:20:57.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 -- start start 2023-10-30T13:20:57.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c00cf160 0x7fa3c00b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:57.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3c01538a0 0x7fa3c0144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:57.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c015d6f0 0x7fa3c0140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:57.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00c98f0 con 0x7fa3c01538a0 2023-10-30T13:20:57.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00b46a0 con 0x7fa3c015d6f0 2023-10-30T13:20:57.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3c5b9d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00cdfe0 con 0x7fa3c00cf160 2023-10-30T13:20:57.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3b7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3c01538a0 0x7fa3c0144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:57.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3b7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3c01538a0 0x7fa3c0144f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43234/0 (socket says 172.21.15.148:43234) 2023-10-30T13:20:57.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3b7fff640 1 -- 172.21.15.148:0/2318945722 learned_addr learned my addr 172.21.15.148:0/2318945722 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:57.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3bffff640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c015d6f0 0x7fa3c0140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:57.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3bf7fe640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c00cf160 0x7fa3c00b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:57.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3b7fff640 1 -- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c00cf160 msgr2=0x7fa3c00b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:57.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3b7fff640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c00cf160 0x7fa3c00b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3b7fff640 1 -- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c015d6f0 msgr2=0x7fa3c0140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:57.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3b7fff640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c015d6f0 0x7fa3c0140e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3b7fff640 1 -- 172.21.15.148:0/2318945722 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8009a70 con 0x7fa3c01538a0 2023-10-30T13:20:57.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.510+0000 7fa3b7fff640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3c01538a0 0x7fa3c0144f10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800e8a0 tx=0x7fa3a800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:57.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3bcff9640 1 -- 172.21.15.148:0/2318945722 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8009510 con 0x7fa3c01538a0 2023-10-30T13:20:57.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3bcff9640 1 -- 172.21.15.148:0/2318945722 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a800c940 con 0x7fa3c01538a0 2023-10-30T13:20:57.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3bcff9640 1 -- 172.21.15.148:0/2318945722 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a800ccb0 con 0x7fa3c01538a0 2023-10-30T13:20:57.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c0146d20 con 0x7fa3c01538a0 2023-10-30T13:20:57.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c00cdd00 con 0x7fa3c01538a0 2023-10-30T13:20:57.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c00b2a50 con 0x7fa3c01538a0 2023-10-30T13:20:57.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3bcff9640 1 -- 172.21.15.148:0/2318945722 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3a8013470 con 0x7fa3c01538a0 2023-10-30T13:20:57.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3bcff9640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3a00664a0 0x7fa3a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:57.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3bcff9640 1 -- 172.21.15.148:0/2318945722 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fa3a80843d0 con 0x7fa3c01538a0 2023-10-30T13:20:57.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.514+0000 7fa3bf7fe640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3a00664a0 0x7fa3a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:57.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.518+0000 7fa3bf7fe640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3a00664a0 0x7fa3a0068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0005890 tx=0x7fa3b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:57.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.518+0000 7fa3bcff9640 1 -- 172.21.15.148:0/2318945722 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3a80517a0 con 0x7fa3c01538a0 2023-10-30T13:20:57.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.702+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3c00ce920 con 0x7fa3c01538a0 2023-10-30T13:20:57.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3bcff9640 1 -- 172.21.15.148:0/2318945722 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fa3c00ce920 con 0x7fa3c01538a0 2023-10-30T13:20:57.706 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:20:57.706 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:57.707 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:20:57.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3a00664a0 msgr2=0x7fa3a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:57.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3a00664a0 0x7fa3a0068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0005890 tx=0x7fa3b0009040 comp rx=0 tx=0).stop 2023-10-30T13:20:57.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3c01538a0 msgr2=0x7fa3c0144f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:57.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3c01538a0 0x7fa3c0144f10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800e8a0 tx=0x7fa3a800eca0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 shutdown_connections 2023-10-30T13:20:57.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3a00664a0 0x7fa3a0068960 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3c01538a0 0x7fa3c0144f10 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c015d6f0 0x7fa3c0140e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 --2- 172.21.15.148:0/2318945722 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3c00cf160 0x7fa3c00b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:57.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 >> 172.21.15.148:0/2318945722 conn(0x7fa3c006bfe0 msgr2=0x7fa3c0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:57.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 shutdown_connections 2023-10-30T13:20:57.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:57.706+0000 7fa3c5b9d640 1 -- 172.21.15.148:0/2318945722 wait complete. 2023-10-30T13:20:57.728 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:59.729 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:59.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd466a6b640 1 Processor -- start 2023-10-30T13:20:59.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd466a6b640 1 -- start start 2023-10-30T13:20:59.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd466a6b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd4601538a0 0x7fd46015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:59.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd466a6b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4600cb800 con 0x7fd4600ca0e0 2023-10-30T13:20:59.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd466a6b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46015dc30 con 0x7fd4600cf160 2023-10-30T13:20:59.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd466a6b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4600cb330 con 0x7fd4601538a0 2023-10-30T13:20:59.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd464807640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd4601538a0 0x7fd46015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:59.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd464807640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd4601538a0 0x7fd46015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43242/0 (socket says 172.21.15.148:43242) 2023-10-30T13:20:59.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd464807640 1 -- 172.21.15.148:0/242281730 learned_addr learned my addr 172.21.15.148:0/242281730 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:59.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd464807640 1 -- 172.21.15.148:0/242281730 >> v1:172.21.15.150:6789/0 conn(0x7fd4600cf160 legacy=0x7fd460164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:59.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd464807640 1 -- 172.21.15.148:0/242281730 >> v1:172.21.15.148:6789/0 conn(0x7fd4600ca0e0 legacy=0x7fd46015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:59.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd4577fe640 1 -- 172.21.15.148:0/242281730 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140887497 0 0) 0x7fd4600cb800 con 0x7fd4600ca0e0 2023-10-30T13:20:59.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.878+0000 7fd464807640 1 -- 172.21.15.148:0/242281730 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4600cad70 con 0x7fd4601538a0 2023-10-30T13:20:59.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd464807640 1 --2- 172.21.15.148:0/242281730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4601538a0 0x7fd46015d6f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009870 tx=0x7fd45c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb18ea7a755a0d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:59.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd4577fe640 1 -- 172.21.15.148:0/242281730 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c0a8a30 con 0x7fd4601538a0 2023-10-30T13:20:59.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd4577fe640 1 -- 172.21.15.148:0/242281730 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c0a8b90 con 0x7fd4601538a0 2023-10-30T13:20:59.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd4577fe640 1 -- 172.21.15.148:0/242281730 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c0b0440 con 0x7fd4601538a0 2023-10-30T13:20:59.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd466a6b640 1 -- 172.21.15.148:0/242281730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4601538a0 msgr2=0x7fd46015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:59.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd466a6b640 1 --2- 172.21.15.148:0/242281730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4601538a0 0x7fd46015d6f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009870 tx=0x7fd45c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:20:59.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd466a6b640 1 -- 172.21.15.148:0/242281730 shutdown_connections 2023-10-30T13:20:59.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd466a6b640 1 --2- 172.21.15.148:0/242281730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4601538a0 0x7fd46015d6f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:59.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd466a6b640 1 -- 172.21.15.148:0/242281730 >> 172.21.15.148:0/242281730 conn(0x7fd46006bfe0 msgr2=0x7fd46006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:59.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd466a6b640 1 -- 172.21.15.148:0/242281730 shutdown_connections 2023-10-30T13:20:59.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.882+0000 7fd466a6b640 1 -- 172.21.15.148:0/242281730 wait complete. 2023-10-30T13:20:59.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 Processor -- start 2023-10-30T13:20:59.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 -- start start 2023-10-30T13:20:59.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4600ca0e0 0x7fd460146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:59.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4600cf160 0x7fd4600ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:59.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4601538a0 0x7fd4600ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:59.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4601466c0 con 0x7fd4600ca0e0 2023-10-30T13:20:59.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4601e6600 con 0x7fd4601538a0 2023-10-30T13:20:59.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd460157b60 con 0x7fd4600cf160 2023-10-30T13:20:59.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd464807640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4600ca0e0 0x7fd460146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:59.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd464807640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4600ca0e0 0x7fd460146180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43248/0 (socket says 172.21.15.148:43248) 2023-10-30T13:20:59.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd464807640 1 -- 172.21.15.148:0/1891060015 learned_addr learned my addr 172.21.15.148:0/1891060015 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:20:59.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd465008640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4601538a0 0x7fd4600ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:59.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd457fff640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4600cf160 0x7fd4600ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:59.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd464807640 1 -- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4600cf160 msgr2=0x7fd4600ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:59.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd464807640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4600cf160 0x7fd4600ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:59.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd464807640 1 -- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4601538a0 msgr2=0x7fd4600ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:59.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd464807640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4601538a0 0x7fd4600ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:59.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd464807640 1 -- 172.21.15.148:0/1891060015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c0094d0 con 0x7fd4600ca0e0 2023-10-30T13:20:59.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd457fff640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4600cf160 0x7fd4600ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:20:59.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd465008640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4601538a0 0x7fd4600ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:20:59.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd464807640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4600ca0e0 0x7fd460146180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd45c0b3040 tx=0x7fd45c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:59.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd4557fa640 1 -- 172.21.15.148:0/1891060015 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c0a8a30 con 0x7fd4600ca0e0 2023-10-30T13:20:59.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd4557fa640 1 -- 172.21.15.148:0/1891060015 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c0bb480 con 0x7fd4600ca0e0 2023-10-30T13:20:59.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd460157e40 con 0x7fd4600ca0e0 2023-10-30T13:20:59.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd4557fa640 1 -- 172.21.15.148:0/1891060015 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c0bb850 con 0x7fd4600ca0e0 2023-10-30T13:20:59.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4601e2ae0 con 0x7fd4600ca0e0 2023-10-30T13:20:59.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.886+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46004e170 con 0x7fd4600ca0e0 2023-10-30T13:20:59.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.890+0000 7fd4557fa640 1 -- 172.21.15.148:0/1891060015 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd45c0bb9b0 con 0x7fd4600ca0e0 2023-10-30T13:20:59.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.890+0000 7fd4557fa640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd44c0664d0 0x7fd44c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:59.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.890+0000 7fd4557fa640 1 -- 172.21.15.148:0/1891060015 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd45c122810 con 0x7fd4600ca0e0 2023-10-30T13:20:59.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.890+0000 7fd457fff640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd44c0664d0 0x7fd44c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:59.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.890+0000 7fd457fff640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd44c0664d0 0x7fd44c068990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd4600b52c0 tx=0x7fd450009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:59.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:20:59.894+0000 7fd4557fa640 1 -- 172.21.15.148:0/1891060015 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd45c0efa00 con 0x7fd4600ca0e0 2023-10-30T13:21:00.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.074+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4600b1dd0 con 0x7fd4600ca0e0 2023-10-30T13:21:00.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.074+0000 7fd4557fa640 1 -- 172.21.15.148:0/1891060015 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fd45c0b5090 con 0x7fd4600ca0e0 2023-10-30T13:21:00.078 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:00.078 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:00.078 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:21:00.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd44c0664d0 msgr2=0x7fd44c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:00.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd44c0664d0 0x7fd44c068990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd4600b52c0 tx=0x7fd450009040 comp rx=0 tx=0).stop 2023-10-30T13:21:00.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4600ca0e0 msgr2=0x7fd460146180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:00.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4600ca0e0 0x7fd460146180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd45c0b3040 tx=0x7fd45c01bad0 comp rx=0 tx=0).stop 2023-10-30T13:21:00.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 shutdown_connections 2023-10-30T13:21:00.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd44c0664d0 0x7fd44c068990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:00.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4601538a0 0x7fd4600ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:00.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4600cf160 0x7fd4600ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:00.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 --2- 172.21.15.148:0/1891060015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4600ca0e0 0x7fd460146180 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:00.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 >> 172.21.15.148:0/1891060015 conn(0x7fd46006bfe0 msgr2=0x7fd4601526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:00.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 shutdown_connections 2023-10-30T13:21:00.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:00.078+0000 7fd466a6b640 1 -- 172.21.15.148:0/1891060015 wait complete. 2023-10-30T13:21:00.099 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:02.100 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:02.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.246+0000 7ff385a6b640 1 Processor -- start 2023-10-30T13:21:02.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.246+0000 7ff385a6b640 1 -- start start 2023-10-30T13:21:02.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.246+0000 7ff385a6b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff3800c6b80 0x7ff3800bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:02.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.246+0000 7ff385a6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 0x7ff3800c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:02.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.246+0000 7ff385a6b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800c0a50 con 0x7ff3800c6f60 2023-10-30T13:21:02.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.246+0000 7ff385a6b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38006f6e0 con 0x7ff3800c6b80 2023-10-30T13:21:02.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.246+0000 7ff385a6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38006f9c0 con 0x7ff38013f4a0 2023-10-30T13:21:02.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 0x7ff3800c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:02.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff3800c6b80 0x7ff3800bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:02.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37effd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff3800c6f60 0x7ff38013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:36184/0 (socket says 172.21.15.148:36184) 2023-10-30T13:21:02.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37effd640 1 -- 172.21.15.148:0/3874849563 learned_addr learned my addr 172.21.15.148:0/3874849563 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:21:02.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37e7fc640 1 -- 172.21.15.148:0/3874849563 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238884290 0 0) 0x7ff3800c0a50 con 0x7ff3800c6f60 2023-10-30T13:21:02.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37e7fc640 1 -- 172.21.15.148:0/3874849563 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff368005740 con 0x7ff3800c6f60 2023-10-30T13:21:02.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37f7fe640 1 -- 172.21.15.148:0/3874849563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 msgr2=0x7ff3800c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:02.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37f7fe640 1 --2- 172.21.15.148:0/3874849563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 0x7ff3800c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:02.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37f7fe640 1 -- 172.21.15.148:0/3874849563 >> v1:172.21.15.150:6789/0 conn(0x7ff3800c6f60 legacy=0x7ff38013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:02.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37f7fe640 1 -- 172.21.15.148:0/3874849563 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38006fca0 con 0x7ff3800c6b80 2023-10-30T13:21:02.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.250+0000 7ff37ffff640 1 --2- 172.21.15.148:0/3874849563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 0x7ff3800c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:21:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff37f7fe640 1 --2- 172.21.15.148:0/3874849563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3800c6b80 0x7ff3800bfec0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff3700097c0 tx=0x7ff37001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92f90b58ecce3507 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff37e7fc640 1 -- 172.21.15.148:0/3874849563 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700ad640 con 0x7ff3800c6b80 2023-10-30T13:21:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff37e7fc640 1 -- 172.21.15.148:0/3874849563 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3700ad7a0 con 0x7ff3800c6b80 2023-10-30T13:21:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff37e7fc640 1 -- 172.21.15.148:0/3874849563 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700adb30 con 0x7ff3800c6b80 2023-10-30T13:21:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 -- 172.21.15.148:0/3874849563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3800c6b80 msgr2=0x7ff3800bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:02.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 --2- 172.21.15.148:0/3874849563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3800c6b80 0x7ff3800bfec0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff3700097c0 tx=0x7ff37001c480 comp rx=0 tx=0).stop 2023-10-30T13:21:02.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 -- 172.21.15.148:0/3874849563 shutdown_connections 2023-10-30T13:21:02.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 --2- 172.21.15.148:0/3874849563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 0x7ff3800c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:02.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 --2- 172.21.15.148:0/3874849563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3800c6b80 0x7ff3800bfec0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:02.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 -- 172.21.15.148:0/3874849563 >> 172.21.15.148:0/3874849563 conn(0x7ff3800c7910 msgr2=0x7ff3800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:02.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 -- 172.21.15.148:0/3874849563 shutdown_connections 2023-10-30T13:21:02.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 -- 172.21.15.148:0/3874849563 wait complete. 2023-10-30T13:21:02.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 Processor -- start 2023-10-30T13:21:02.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.254+0000 7ff385a6b640 1 -- start start 2023-10-30T13:21:02.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff385a6b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3800c6b80 0x7ff3800c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:02.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff385a6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3800c6f60 0x7ff38006a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:02.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff385a6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 0x7ff38013ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:02.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff385a6b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380157790 con 0x7ff3800c6b80 2023-10-30T13:21:02.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff385a6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3801582d0 con 0x7ff3800c6f60 2023-10-30T13:21:02.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff385a6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3801395e0 con 0x7ff38013f4a0 2023-10-30T13:21:02.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3800c6b80 0x7ff3800c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:02.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3800c6b80 0x7ff3800c8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57010/0 (socket says 172.21.15.148:57010) 2023-10-30T13:21:02.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37f7fe640 1 -- 172.21.15.148:0/3093739757 learned_addr learned my addr 172.21.15.148:0/3093739757 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:02.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37effd640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3800c6f60 0x7ff38006a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:02.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37ffff640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 0x7ff38013ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:02.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37f7fe640 1 -- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 msgr2=0x7ff38013ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:02.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37f7fe640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 0x7ff38013ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:02.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37f7fe640 1 -- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3800c6f60 msgr2=0x7ff38006a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:02.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37f7fe640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3800c6f60 0x7ff38006a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:02.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37f7fe640 1 -- 172.21.15.148:0/3093739757 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378009a70 con 0x7ff3800c6b80 2023-10-30T13:21:02.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff37f7fe640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3800c6b80 0x7ff3800c8ec0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff37001a0b0 tx=0x7ff370019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:02.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff367fff640 1 -- 172.21.15.148:0/3093739757 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700b6820 con 0x7ff3800c6b80 2023-10-30T13:21:02.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff367fff640 1 -- 172.21.15.148:0/3093739757 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3700a6040 con 0x7ff3800c6b80 2023-10-30T13:21:02.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff367fff640 1 -- 172.21.15.148:0/3093739757 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700b6bc0 con 0x7ff3800c6b80 2023-10-30T13:21:02.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff370009420 con 0x7ff3800c6b80 2023-10-30T13:21:02.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff380157c80 con 0x7ff3800c6b80 2023-10-30T13:21:02.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38004e170 con 0x7ff3800c6b80 2023-10-30T13:21:02.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.258+0000 7ff367fff640 1 -- 172.21.15.148:0/3093739757 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3700b4040 con 0x7ff3800c6b80 2023-10-30T13:21:02.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.262+0000 7ff367fff640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff36c0664d0 0x7ff36c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:02.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.262+0000 7ff367fff640 1 -- 172.21.15.148:0/3093739757 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7ff37011e550 con 0x7ff3800c6b80 2023-10-30T13:21:02.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.262+0000 7ff37effd640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff36c0664d0 0x7ff36c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:02.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.262+0000 7ff37effd640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff36c0664d0 0x7ff36c068990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff380071f70 tx=0x7ff374009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:02.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.266+0000 7ff367fff640 1 -- 172.21.15.148:0/3093739757 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3700eb7d0 con 0x7ff3800c6b80 2023-10-30T13:21:02.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff38013df70 con 0x7ff3800c6b80 2023-10-30T13:21:02.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff367fff640 1 -- 172.21.15.148:0/3093739757 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7ff3700eedf0 con 0x7ff3800c6b80 2023-10-30T13:21:02.448 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:02.448 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:02.448 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:21:02.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff36c0664d0 msgr2=0x7ff36c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:02.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff36c0664d0 0x7ff36c068990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff380071f70 tx=0x7ff374009040 comp rx=0 tx=0).stop 2023-10-30T13:21:02.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3800c6b80 msgr2=0x7ff3800c8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:02.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3800c6b80 0x7ff3800c8ec0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff37001a0b0 tx=0x7ff370019220 comp rx=0 tx=0).stop 2023-10-30T13:21:02.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 shutdown_connections 2023-10-30T13:21:02.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff36c0664d0 0x7ff36c068990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:02.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff38013f4a0 0x7ff38013ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:02.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3800c6f60 0x7ff38006a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:02.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 --2- 172.21.15.148:0/3093739757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3800c6b80 0x7ff3800c8ec0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:02.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 >> 172.21.15.148:0/3093739757 conn(0x7ff3800c7910 msgr2=0x7ff3800c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:02.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 shutdown_connections 2023-10-30T13:21:02.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:02.446+0000 7ff385a6b640 1 -- 172.21.15.148:0/3093739757 wait complete. 2023-10-30T13:21:02.468 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:04.469 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:04.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125d71640 1 Processor -- start 2023-10-30T13:21:04.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125d71640 1 -- start start 2023-10-30T13:21:04.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125d71640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f912015cd50 0x7f91200c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:04.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125d71640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912015a6c0 con 0x7f91201538f0 2023-10-30T13:21:04.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125d71640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912015d300 con 0x7f91200cefa0 2023-10-30T13:21:04.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125d71640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912015d5e0 con 0x7f912015cd50 2023-10-30T13:21:04.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9124d6f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f91201538f0 0x7f9120159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48932/0 (socket says 172.21.15.148:48932) 2023-10-30T13:21:04.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9124d6f640 1 -- 172.21.15.148:0/2212919365 learned_addr learned my addr 172.21.15.148:0/2212919365 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:21:04.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911ffff640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f912015cd50 0x7f91200c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:04.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206266488 0 0) 0x7f912015a6c0 con 0x7f91201538f0 2023-10-30T13:21:04.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9110003530 con 0x7f91201538f0 2023-10-30T13:21:04.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 676528673 0 0) 0x7f9110003530 con 0x7f91201538f0 2023-10-30T13:21:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f912015cd50 msgr2=0x7f91200c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f912015cd50 0x7f91200c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 >> v1:172.21.15.150:6789/0 conn(0x7f91200cefa0 legacy=0x7f91201640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912015d8c0 con 0x7f91201538f0 2023-10-30T13:21:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f9114001680 con 0x7f91201538f0 2023-10-30T13:21:04.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 >> v1:172.21.15.148:6789/0 conn(0x7f91201538f0 legacy=0x7f9120159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:04.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f912015cd50 0x7f91100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:04.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9110003ef0 0x7f91100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:04.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9110004d10 0x7f9110005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:04.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912015d5e0 con 0x7f9110004d10 2023-10-30T13:21:04.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911000b9f0 con 0x7f9110003ef0 2023-10-30T13:21:04.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911000bbb0 con 0x7f912015cd50 2023-10-30T13:21:04.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125570640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9110004d10 0x7f9110005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:04.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9124d6f640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f912015cd50 0x7f91100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:04.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f911ffff640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9110003ef0 0x7f91100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:04.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125570640 1 -- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f912015cd50 msgr2=0x7f91100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:04.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125570640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f912015cd50 0x7f91100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:04.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125570640 1 -- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9110003ef0 msgr2=0x7f91100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:04.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125570640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9110003ef0 0x7f91100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:04.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.618+0000 7f9125570640 1 -- 172.21.15.148:0/2212919365 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9110003530 con 0x7f9110004d10 2023-10-30T13:21:04.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f9125570640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9110004d10 0x7f9110005190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9118001f60 tx=0x7f911801c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5227d3162088a13a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:04.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180a5750 con 0x7f9110004d10 2023-10-30T13:21:04.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91180a58b0 con 0x7f9110004d10 2023-10-30T13:21:04.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f911f7fe640 1 -- 172.21.15.148:0/2212919365 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180a5ca0 con 0x7f9110004d10 2023-10-30T13:21:04.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f9125d71640 1 -- 172.21.15.148:0/2212919365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9110004d10 msgr2=0x7f9110005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:04.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f9125d71640 1 --2- 172.21.15.148:0/2212919365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9110004d10 0x7f9110005190 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9118001f60 tx=0x7f911801c860 comp rx=0 tx=0).stop 2023-10-30T13:21:04.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f9125570640 1 -- 172.21.15.148:0/2212919365 reap_dead start 2023-10-30T13:21:04.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f9125d71640 1 -- 172.21.15.148:0/2212919365 shutdown_connections 2023-10-30T13:21:04.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f9125d71640 1 -- 172.21.15.148:0/2212919365 >> 172.21.15.148:0/2212919365 conn(0x7f912006a640 msgr2=0x7f9120065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:04.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f9125d71640 1 -- 172.21.15.148:0/2212919365 shutdown_connections 2023-10-30T13:21:04.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.622+0000 7f9125d71640 1 -- 172.21.15.148:0/2212919365 wait complete. 2023-10-30T13:21:04.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 Processor -- start 2023-10-30T13:21:04.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 -- start start 2023-10-30T13:21:04.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9110004d10 0x7f91201521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:04.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91201456e0 0x7f9120158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:04.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91201ef6e0 0x7f91201efb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:04.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91201573a0 con 0x7f91201ef6e0 2023-10-30T13:21:04.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91201eefd0 con 0x7f9110004d10 2023-10-30T13:21:04.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9120152710 con 0x7f91201456e0 2023-10-30T13:21:04.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125570640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91201ef6e0 0x7f91201efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:04.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125570640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91201ef6e0 0x7f91201efb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57020/0 (socket says 172.21.15.148:57020) 2023-10-30T13:21:04.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125570640 1 -- 172.21.15.148:0/1011024129 learned_addr learned my addr 172.21.15.148:0/1011024129 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:04.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9124d6f640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9110004d10 0x7f91201521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f911ffff640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91201456e0 0x7f9120158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125570640 1 -- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91201456e0 msgr2=0x7f9120158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125570640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91201456e0 0x7f9120158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125570640 1 -- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9110004d10 msgr2=0x7f91201521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125570640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9110004d10 0x7f91201521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:04.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125570640 1 -- 172.21.15.148:0/1011024129 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91201512a0 con 0x7f91201ef6e0 2023-10-30T13:21:04.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f911ffff640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91201456e0 0x7f9120158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:04.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125570640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91201ef6e0 0x7f91201efb40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9118098270 tx=0x7f9110005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:04.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f911d7fa640 1 -- 172.21.15.148:0/1011024129 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180a5e30 con 0x7f91201ef6e0 2023-10-30T13:21:04.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f911d7fa640 1 -- 172.21.15.148:0/1011024129 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91180011d0 con 0x7f91201ef6e0 2023-10-30T13:21:04.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9120152c80 con 0x7f91201ef6e0 2023-10-30T13:21:04.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f911d7fa640 1 -- 172.21.15.148:0/1011024129 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118003b80 con 0x7f91201ef6e0 2023-10-30T13:21:04.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.626+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91201ef1c0 con 0x7f91201ef6e0 2023-10-30T13:21:04.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.630+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91201ef3b0 con 0x7f91201ef6e0 2023-10-30T13:21:04.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.630+0000 7f911d7fa640 1 -- 172.21.15.148:0/1011024129 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f91180b4040 con 0x7f91201ef6e0 2023-10-30T13:21:04.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.630+0000 7f911d7fa640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91040664a0 0x7f9104068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:04.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.630+0000 7f911d7fa640 1 -- 172.21.15.148:0/1011024129 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f911811d820 con 0x7f91201ef6e0 2023-10-30T13:21:04.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.630+0000 7f9124d6f640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91040664a0 0x7f9104068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:04.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.630+0000 7f9124d6f640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91040664a0 0x7f9104068960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9114000f70 tx=0x7f9114000f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:04.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.634+0000 7f911d7fa640 1 -- 172.21.15.148:0/1011024129 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f91180eab50 con 0x7f91201ef6e0 2023-10-30T13:21:04.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.818+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9120153130 con 0x7f91201ef6e0 2023-10-30T13:21:04.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.818+0000 7f911d7fa640 1 -- 172.21.15.148:0/1011024129 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f91180ee170 con 0x7f91201ef6e0 2023-10-30T13:21:04.820 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:04.821 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:04.821 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:21:04.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91040664a0 msgr2=0x7f9104068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:04.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91040664a0 0x7f9104068960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9114000f70 tx=0x7f9114000f00 comp rx=0 tx=0).stop 2023-10-30T13:21:04.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91201ef6e0 msgr2=0x7f91201efb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:04.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91201ef6e0 0x7f91201efb40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9118098270 tx=0x7f9110005800 comp rx=0 tx=0).stop 2023-10-30T13:21:04.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 shutdown_connections 2023-10-30T13:21:04.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91040664a0 0x7f9104068960 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:04.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91201ef6e0 0x7f91201efb40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:04.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f91201456e0 0x7f9120158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:04.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 --2- 172.21.15.148:0/1011024129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9110004d10 0x7f91201521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:04.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 >> 172.21.15.148:0/1011024129 conn(0x7f912006a640 msgr2=0x7f9120148fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:04.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 shutdown_connections 2023-10-30T13:21:04.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:04.822+0000 7f9125d71640 1 -- 172.21.15.148:0/1011024129 wait complete. 2023-10-30T13:21:04.841 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:06.843 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:06.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.986+0000 7fd406e46640 1 Processor -- start 2023-10-30T13:21:06.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.986+0000 7fd406e46640 1 -- start start 2023-10-30T13:21:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd406e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4001539a0 0x7fd40015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd406e46640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd400070a00 0x7fd400070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd406e46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40015a8c0 con 0x7fd400069e90 2023-10-30T13:21:06.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd406e46640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40006fbb0 con 0x7fd400070a00 2023-10-30T13:21:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd406e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd400071380 con 0x7fd4001539a0 2023-10-30T13:21:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd3f7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd400070a00 0x7fd400070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd3f7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd400070a00 0x7fd400070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57034/0 (socket says 172.21.15.148:57034) 2023-10-30T13:21:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd3f7fff640 1 -- 172.21.15.148:0/2847429763 learned_addr learned my addr 172.21.15.148:0/2847429763 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd404be2640 1 --2- 172.21.15.148:0/2847429763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4001539a0 0x7fd40015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:06.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd3f7fff640 1 -- 172.21.15.148:0/2847429763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4001539a0 msgr2=0x7fd40015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd3f7fff640 1 --2- 172.21.15.148:0/2847429763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4001539a0 0x7fd40015a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd3f7fff640 1 -- 172.21.15.148:0/2847429763 >> v1:172.21.15.150:6789/0 conn(0x7fd400069e90 legacy=0x7fd40006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.990+0000 7fd3f7fff640 1 -- 172.21.15.148:0/2847429763 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd400153d80 con 0x7fd400070a00 2023-10-30T13:21:06.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd3f7fff640 1 --2- 172.21.15.148:0/2847429763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd400070a00 0x7fd400070e40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0009870 tx=0x7fd3f001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999adfe6a75a56e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:06.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd3f77fe640 1 -- 172.21.15.148:0/2847429763 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f00ab680 con 0x7fd400070a00 2023-10-30T13:21:06.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd3f77fe640 1 -- 172.21.15.148:0/2847429763 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f00ab7e0 con 0x7fd400070a00 2023-10-30T13:21:06.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd3f77fe640 1 -- 172.21.15.148:0/2847429763 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f00abb70 con 0x7fd400070a00 2023-10-30T13:21:06.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 -- 172.21.15.148:0/2847429763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd400070a00 msgr2=0x7fd400070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:06.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 --2- 172.21.15.148:0/2847429763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd400070a00 0x7fd400070e40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0009870 tx=0x7fd3f001c480 comp rx=0 tx=0).stop 2023-10-30T13:21:06.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 -- 172.21.15.148:0/2847429763 shutdown_connections 2023-10-30T13:21:06.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 --2- 172.21.15.148:0/2847429763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd400070a00 0x7fd400070e40 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:06.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 --2- 172.21.15.148:0/2847429763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4001539a0 0x7fd40015a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:06.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 -- 172.21.15.148:0/2847429763 >> 172.21.15.148:0/2847429763 conn(0x7fd4000c7910 msgr2=0x7fd4000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:06.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 -- 172.21.15.148:0/2847429763 shutdown_connections 2023-10-30T13:21:06.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 -- 172.21.15.148:0/2847429763 wait complete. 2023-10-30T13:21:06.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 Processor -- start 2023-10-30T13:21:06.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.994+0000 7fd406e46640 1 -- start start 2023-10-30T13:21:06.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd406e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd400069e90 0x7fd400146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:06.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd406e46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd400070a00 0x7fd4000c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:06.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd406e46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4001539a0 0x7fd4000c4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:06.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd406e46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4000c2210 con 0x7fd4001539a0 2023-10-30T13:21:06.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd406e46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4001590b0 con 0x7fd400070a00 2023-10-30T13:21:06.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd406e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4000ca020 con 0x7fd400069e90 2023-10-30T13:21:06.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd4053e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4001539a0 0x7fd4000c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:07.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd4053e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4001539a0 0x7fd4000c4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57036/0 (socket says 172.21.15.148:57036) 2023-10-30T13:21:07.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd4053e3640 1 -- 172.21.15.148:0/1977548095 learned_addr learned my addr 172.21.15.148:0/1977548095 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:07.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd404be2640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd400069e90 0x7fd400146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:07.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd3f7fff640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd400070a00 0x7fd4000c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:07.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd4053e3640 1 -- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd400069e90 msgr2=0x7fd400146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:07.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd4053e3640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd400069e90 0x7fd400146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:07.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd4053e3640 1 -- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd400070a00 msgr2=0x7fd4000c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:07.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd4053e3640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd400070a00 0x7fd4000c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:07.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd4053e3640 1 -- 172.21.15.148:0/1977548095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f00094d0 con 0x7fd4001539a0 2023-10-30T13:21:07.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd4053e3640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4001539a0 0x7fd4000c4550 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c770 tx=0x7fd3f800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd3f57fa640 1 -- 172.21.15.148:0/1977548095 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8003b80 con 0x7fd4001539a0 2023-10-30T13:21:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd3f57fa640 1 -- 172.21.15.148:0/1977548095 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f8003ce0 con 0x7fd4001539a0 2023-10-30T13:21:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd3f57fa640 1 -- 172.21.15.148:0/1977548095 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f800aa90 con 0x7fd4001539a0 2023-10-30T13:21:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd400159300 con 0x7fd4001539a0 2023-10-30T13:21:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4000c2420 con 0x7fd4001539a0 2023-10-30T13:21:07.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4001465b0 con 0x7fd4001539a0 2023-10-30T13:21:07.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:06.998+0000 7fd3f57fa640 1 -- 172.21.15.148:0/1977548095 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3f800abf0 con 0x7fd4001539a0 2023-10-30T13:21:07.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.002+0000 7fd3f57fa640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3ec0664d0 0x7fd3ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.002+0000 7fd3f57fa640 1 -- 172.21.15.148:0/1977548095 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fd3f8083dc0 con 0x7fd4001539a0 2023-10-30T13:21:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.002+0000 7fd404be2640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3ec0664d0 0x7fd3ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:07.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.002+0000 7fd404be2640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3ec0664d0 0x7fd3ec068990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0079c0 tx=0x7fd3fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:07.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.006+0000 7fd3f57fa640 1 -- 172.21.15.148:0/1977548095 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd4001465b0 con 0x7fd4001539a0 2023-10-30T13:21:07.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4001465b0 con 0x7fd4001539a0 2023-10-30T13:21:07.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd3f57fa640 1 -- 172.21.15.148:0/1977548095 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fd4001465b0 con 0x7fd4001539a0 2023-10-30T13:21:07.187 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:07.187 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:07.188 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:21:07.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3ec0664d0 msgr2=0x7fd3ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:07.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3ec0664d0 0x7fd3ec068990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0079c0 tx=0x7fd3fc008040 comp rx=0 tx=0).stop 2023-10-30T13:21:07.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4001539a0 msgr2=0x7fd4000c4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4001539a0 0x7fd4000c4550 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800c770 tx=0x7fd3f800cc30 comp rx=0 tx=0).stop 2023-10-30T13:21:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 shutdown_connections 2023-10-30T13:21:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3ec0664d0 0x7fd3ec068990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4001539a0 0x7fd4000c4550 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd400070a00 0x7fd4000c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 --2- 172.21.15.148:0/1977548095 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd400069e90 0x7fd400146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 >> 172.21.15.148:0/1977548095 conn(0x7fd4000c7910 msgr2=0x7fd400073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 shutdown_connections 2023-10-30T13:21:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:07.186+0000 7fd406e46640 1 -- 172.21.15.148:0/1977548095 wait complete. 2023-10-30T13:21:07.208 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T13:21:07.209 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5033": { "addr": "172.21.15.150:6836/953984429", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 953984429, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 953984429, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5033, "incarnation": 36, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5103": { "addr": "172.21.15.148:6835/4069583575", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 4069583575, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 4069583575, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5103, "incarnation": 29, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:stopping", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:20:41.562473+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5033, "mds_1": 5103 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6837/2690362018", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 2690362018, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 2690362018, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 51, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5529, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:21:09.210 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:09.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d6f65640 1 Processor -- start 2023-10-30T13:21:09.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d6f65640 1 -- start start 2023-10-30T13:21:09.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d6f65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d0069e90 0x7f79d015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:09.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d6f65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79d006a270 0x7f79d00c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:09.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d6f65640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f79d006cb80 0x7f79d006d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:09.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d6f65640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d00c8570 con 0x7f79d006cb80 2023-10-30T13:21:09.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d6f65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d006d550 con 0x7f79d006a270 2023-10-30T13:21:09.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d6f65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d00c6c10 con 0x7f79d0069e90 2023-10-30T13:21:09.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d5502640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f79d006cb80 0x7f79d006d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:09.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d5502640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f79d006cb80 0x7f79d006d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57052/0 (socket says 172.21.15.148:57052) 2023-10-30T13:21:09.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d5502640 1 -- 172.21.15.148:0/1275593952 learned_addr learned my addr 172.21.15.148:0/1275593952 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:09.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79c7fff640 1 --2- 172.21.15.148:0/1275593952 >> v2:172.21.15.150:3300/0 conn(0x7f79d006a270 0x7f79d00c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:09.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d4d01640 1 --2- 172.21.15.148:0/1275593952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d0069e90 0x7f79d015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:09.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d5502640 1 -- 172.21.15.148:0/1275593952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d0069e90 msgr2=0x7f79d015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:09.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d5502640 1 --2- 172.21.15.148:0/1275593952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d0069e90 0x7f79d015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d5502640 1 -- 172.21.15.148:0/1275593952 >> v2:172.21.15.150:3300/0 conn(0x7f79d006a270 msgr2=0x7f79d00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:09.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d5502640 1 --2- 172.21.15.148:0/1275593952 >> v2:172.21.15.150:3300/0 conn(0x7f79d006a270 0x7f79d00c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.358+0000 7f79d5502640 1 -- 172.21.15.148:0/1275593952 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d00c6ef0 con 0x7f79d006cb80 2023-10-30T13:21:09.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d5502640 1 --2- 172.21.15.148:0/1275593952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d006cb80 0x7f79d006d010 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f79d0070020 tx=0x7f79c801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3abc8acd75acb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:09.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79c77fe640 1 -- 172.21.15.148:0/1275593952 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c80a5740 con 0x7f79d006cb80 2023-10-30T13:21:09.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79c77fe640 1 -- 172.21.15.148:0/1275593952 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c80a58a0 con 0x7f79d006cb80 2023-10-30T13:21:09.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79c77fe640 1 -- 172.21.15.148:0/1275593952 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c80a5c30 con 0x7f79d006cb80 2023-10-30T13:21:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d6f65640 1 -- 172.21.15.148:0/1275593952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d006cb80 msgr2=0x7f79d006d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1275593952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d006cb80 0x7f79d006d010 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f79d0070020 tx=0x7f79c801c9d0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d6f65640 1 -- 172.21.15.148:0/1275593952 shutdown_connections 2023-10-30T13:21:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1275593952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d006cb80 0x7f79d006d010 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1275593952 >> v2:172.21.15.150:3300/0 conn(0x7f79d006a270 0x7f79d00c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1275593952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d0069e90 0x7f79d015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d6f65640 1 -- 172.21.15.148:0/1275593952 >> 172.21.15.148:0/1275593952 conn(0x7f79d006dfd0 msgr2=0x7f79d006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:09.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d6f65640 1 -- 172.21.15.148:0/1275593952 shutdown_connections 2023-10-30T13:21:09.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.362+0000 7f79d6f65640 1 -- 172.21.15.148:0/1275593952 wait complete. 2023-10-30T13:21:09.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 Processor -- start 2023-10-30T13:21:09.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 -- start start 2023-10-30T13:21:09.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d0069e90 0x7f79d01f9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:09.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79d006a270 0x7f79d0154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:09.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d006cb80 0x7f79d0155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:09.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d01f9f00 con 0x7f79d0069e90 2023-10-30T13:21:09.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d00b1b80 con 0x7f79d006a270 2023-10-30T13:21:09.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d00b1e20 con 0x7f79d006cb80 2023-10-30T13:21:09.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d4d01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d0069e90 0x7f79d01f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:09.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d4d01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d0069e90 0x7f79d01f9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57054/0 (socket says 172.21.15.148:57054) 2023-10-30T13:21:09.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d4d01640 1 -- 172.21.15.148:0/1625464561 learned_addr learned my addr 172.21.15.148:0/1625464561 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:09.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79c7fff640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79d006a270 0x7f79d0154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:09.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d5502640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d006cb80 0x7f79d0155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:09.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d4d01640 1 -- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d006cb80 msgr2=0x7f79d0155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:09.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d4d01640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d006cb80 0x7f79d0155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d4d01640 1 -- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79d006a270 msgr2=0x7f79d0154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:09.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d4d01640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79d006a270 0x7f79d0154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d4d01640 1 -- 172.21.15.148:0/1625464561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c8009b30 con 0x7f79d0069e90 2023-10-30T13:21:09.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79c7fff640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79d006a270 0x7f79d0154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:09.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d4d01640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d0069e90 0x7f79d01f9250 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00e870 tx=0x7f79cc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:09.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79c57fa640 1 -- 172.21.15.148:0/1625464561 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00ee60 con 0x7f79d0069e90 2023-10-30T13:21:09.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79c57fa640 1 -- 172.21.15.148:0/1625464561 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc00c940 con 0x7f79d0069e90 2023-10-30T13:21:09.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79c57fa640 1 -- 172.21.15.148:0/1625464561 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00ccf0 con 0x7f79d0069e90 2023-10-30T13:21:09.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d00cb900 con 0x7f79d0069e90 2023-10-30T13:21:09.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.366+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d0154900 con 0x7f79d0069e90 2023-10-30T13:21:09.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.370+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d0154ac0 con 0x7f79d0069e90 2023-10-30T13:21:09.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.370+0000 7f79c57fa640 1 -- 172.21.15.148:0/1625464561 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79cc013470 con 0x7f79d0069e90 2023-10-30T13:21:09.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.370+0000 7f79c57fa640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79bc0664d0 0x7f79bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:09.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.370+0000 7f79c57fa640 1 -- 172.21.15.148:0/1625464561 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f79cc083ff0 con 0x7f79d0069e90 2023-10-30T13:21:09.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.370+0000 7f79c7fff640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79bc0664d0 0x7f79bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:09.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.370+0000 7f79c7fff640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79bc0664d0 0x7f79bc068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f79d00cb760 tx=0x7f79c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:09.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.374+0000 7f79c57fa640 1 -- 172.21.15.148:0/1625464561 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f79cc051320 con 0x7f79d0069e90 2023-10-30T13:21:09.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.554+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79d00b1f90 con 0x7f79d0069e90 2023-10-30T13:21:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.554+0000 7f79c57fa640 1 -- 172.21.15.148:0/1625464561 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f79d00b1f90 con 0x7f79d0069e90 2023-10-30T13:21:09.556 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:09.556 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:20:41.562473+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":5033,"mds_1":5103},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5103":{"gid":5103,"name":"a","rank":1,"incarnation":29,"state":"up:stopping","state_seq":25,"addr":"172.21.15.148:6835/4069583575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":4069583575},{"type":"v1","addr":"172.21.15.148:6835","nonce":4069583575}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 52 2023-10-30T13:21:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79bc0664d0 msgr2=0x7f79bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79bc0664d0 0x7f79bc068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f79d00cb760 tx=0x7f79c0008040 comp rx=0 tx=0).stop 2023-10-30T13:21:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d0069e90 msgr2=0x7f79d01f9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d0069e90 0x7f79d01f9250 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00e870 tx=0x7f79cc00ed30 comp rx=0 tx=0).stop 2023-10-30T13:21:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 shutdown_connections 2023-10-30T13:21:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79bc0664d0 0x7f79bc068990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79d006cb80 0x7f79d0155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79d006a270 0x7f79d0154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 --2- 172.21.15.148:0/1625464561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79d0069e90 0x7f79d01f9250 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 >> 172.21.15.148:0/1625464561 conn(0x7f79d006dfd0 msgr2=0x7f79d006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 shutdown_connections 2023-10-30T13:21:09.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:09.558+0000 7f79d6f65640 1 -- 172.21.15.148:0/1625464561 wait complete. 2023-10-30T13:21:09.577 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:11.579 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:11.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b68b3e640 1 Processor -- start 2023-10-30T13:21:11.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b68b3e640 1 -- start start 2023-10-30T13:21:11.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b68b3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b641538a0 0x7f0b6415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:11.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b68b3e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b640cb800 con 0x7f0b640ca0e0 2023-10-30T13:21:11.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b68b3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6415dc30 con 0x7f0b640cf160 2023-10-30T13:21:11.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b68b3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b640cb330 con 0x7f0b641538a0 2023-10-30T13:21:11.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b59d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0b640ca0e0 0x7f0b6415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48938/0 (socket says 172.21.15.148:48938) 2023-10-30T13:21:11.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b59d9b640 1 -- 172.21.15.148:0/3968627170 learned_addr learned my addr 172.21.15.148:0/3968627170 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:21:11.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b6259c640 1 --2- 172.21.15.148:0/3968627170 >> v2:172.21.15.150:3300/0 conn(0x7f0b641538a0 0x7f0b6415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:11.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326190591 0 0) 0x7f0b640cb800 con 0x7f0b640ca0e0 2023-10-30T13:21:11.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b50003530 con 0x7f0b640ca0e0 2023-10-30T13:21:11.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881596285 0 0) 0x7f0b50003530 con 0x7f0b640ca0e0 2023-10-30T13:21:11.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 >> v2:172.21.15.150:3300/0 conn(0x7f0b641538a0 msgr2=0x7f0b6415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:11.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 --2- 172.21.15.148:0/3968627170 >> v2:172.21.15.150:3300/0 conn(0x7f0b641538a0 0x7f0b6415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 >> v1:172.21.15.150:6789/0 conn(0x7f0b640cf160 legacy=0x7f0b64164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:11.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b640cad70 con 0x7f0b640ca0e0 2023-10-30T13:21:11.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f0b54001680 con 0x7f0b640ca0e0 2023-10-30T13:21:11.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 >> v1:172.21.15.148:6789/0 conn(0x7f0b640ca0e0 legacy=0x7f0b6415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:11.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b50003980 0x7f0b50003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:11.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b500047d0 0x7f0b50004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:11.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b50005190 0x7f0b5000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:11.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b640cb330 con 0x7f0b50005190 2023-10-30T13:21:11.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5000c040 con 0x7f0b500047d0 2023-10-30T13:21:11.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.726+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5000c200 con 0x7f0b50003980 2023-10-30T13:21:11.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b62d9d640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b50005190 0x7f0b5000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:11.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b6259c640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b50003980 0x7f0b50003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:11.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b59d9b640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b500047d0 0x7f0b50004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:11.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b62d9d640 1 -- 172.21.15.148:0/3968627170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b50003980 msgr2=0x7f0b50003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:11.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b62d9d640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b50003980 0x7f0b50003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b62d9d640 1 -- 172.21.15.148:0/3968627170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b500047d0 msgr2=0x7f0b50004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:11.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b62d9d640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b500047d0 0x7f0b50004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b62d9d640 1 -- 172.21.15.148:0/3968627170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b50003530 con 0x7f0b50005190 2023-10-30T13:21:11.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b62d9d640 1 -- 172.21.15.148:0/3968627170 reap_dead start 2023-10-30T13:21:11.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b62d9d640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b50005190 0x7f0b5000b990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c001cd0 tx=0x7f0b5c01c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432bc7e3d1f27dc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:11.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c003b80 con 0x7f0b50005190 2023-10-30T13:21:11.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5c003d20 con 0x7f0b50005190 2023-10-30T13:21:11.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.730+0000 7f0b61d9b640 1 -- 172.21.15.148:0/3968627170 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c005310 con 0x7f0b50005190 2023-10-30T13:21:11.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 -- 172.21.15.148:0/3968627170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b50005190 msgr2=0x7f0b5000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:11.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b50005190 0x7f0b5000b990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c001cd0 tx=0x7f0b5c01c5c0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 -- 172.21.15.148:0/3968627170 shutdown_connections 2023-10-30T13:21:11.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 --2- 172.21.15.148:0/3968627170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b50005190 0x7f0b5000b990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 -- 172.21.15.148:0/3968627170 >> 172.21.15.148:0/3968627170 conn(0x7f0b6406bfe0 msgr2=0x7f0b6406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:11.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 -- 172.21.15.148:0/3968627170 shutdown_connections 2023-10-30T13:21:11.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 -- 172.21.15.148:0/3968627170 wait complete. 2023-10-30T13:21:11.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 Processor -- start 2023-10-30T13:21:11.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 -- start start 2023-10-30T13:21:11.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b50005190 0x7f0b640c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:11.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b6415b120 0x7f0b640b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:11.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b641513b0 0x7f0b64151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:11.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6415d2d0 con 0x7f0b6415b120 2023-10-30T13:21:11.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b64151d70 con 0x7f0b50005190 2023-10-30T13:21:11.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b68b3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6415c370 con 0x7f0b641513b0 2023-10-30T13:21:11.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b59d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b6415b120 0x7f0b640b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:11.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b59d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b6415b120 0x7f0b640b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57076/0 (socket says 172.21.15.148:57076) 2023-10-30T13:21:11.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b59d9b640 1 -- 172.21.15.148:0/1577703103 learned_addr learned my addr 172.21.15.148:0/1577703103 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:11.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b62d9d640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b641513b0 0x7f0b64151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:11.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b6259c640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b50005190 0x7f0b640c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:11.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b59d9b640 1 -- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b641513b0 msgr2=0x7f0b64151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:11.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.734+0000 7f0b59d9b640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b641513b0 0x7f0b64151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b59d9b640 1 -- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b50005190 msgr2=0x7f0b640c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:11.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b59d9b640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b50005190 0x7f0b640c6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b59d9b640 1 -- 172.21.15.148:0/1577703103 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6415c650 con 0x7f0b6415b120 2023-10-30T13:21:11.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b59d9b640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b6415b120 0x7f0b640b4450 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007290 tx=0x7f0b54007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:11.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b5b7fe640 1 -- 172.21.15.148:0/1577703103 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b540077c0 con 0x7f0b6415b120 2023-10-30T13:21:11.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b5b7fe640 1 -- 172.21.15.148:0/1577703103 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5400f940 con 0x7f0b6415b120 2023-10-30T13:21:11.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b5b7fe640 1 -- 172.21.15.148:0/1577703103 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5400fd10 con 0x7f0b6415b120 2023-10-30T13:21:11.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6415c930 con 0x7f0b6415b120 2023-10-30T13:21:11.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6415cef0 con 0x7f0b6415b120 2023-10-30T13:21:11.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b641510d0 con 0x7f0b6415b120 2023-10-30T13:21:11.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b5b7fe640 1 -- 172.21.15.148:0/1577703103 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b5401d460 con 0x7f0b6415b120 2023-10-30T13:21:11.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b5b7fe640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b440664a0 0x7f0b44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:11.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b5b7fe640 1 -- 172.21.15.148:0/1577703103 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f0b540840f0 con 0x7f0b6415b120 2023-10-30T13:21:11.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.738+0000 7f0b6259c640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b440664a0 0x7f0b44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:11.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.742+0000 7f0b6259c640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b440664a0 0x7f0b44068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009b60 tx=0x7f0b4c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:11.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.742+0000 7f0b5b7fe640 1 -- 172.21.15.148:0/1577703103 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b641510d0 con 0x7f0b6415b120 2023-10-30T13:21:11.816 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:21:11.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.934+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b641510d0 con 0x7f0b6415b120 2023-10-30T13:21:11.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.934+0000 7f0b5b7fe640 1 -- 172.21.15.148:0/1577703103 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f0b641510d0 con 0x7f0b6415b120 2023-10-30T13:21:11.937 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:11.937 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:11.563063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:11.938 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 53 2023-10-30T13:21:11.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b440664a0 msgr2=0x7f0b44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:11.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b440664a0 0x7f0b44068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009b60 tx=0x7f0b4c009190 comp rx=0 tx=0).stop 2023-10-30T13:21:11.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b6415b120 msgr2=0x7f0b640b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:11.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b6415b120 0x7f0b640b4450 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007290 tx=0x7f0b54007690 comp rx=0 tx=0).stop 2023-10-30T13:21:11.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 shutdown_connections 2023-10-30T13:21:11.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b440664a0 0x7f0b44068960 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b6415b120 0x7f0b640b4450 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b641513b0 0x7f0b64151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 --2- 172.21.15.148:0/1577703103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b50005190 0x7f0b640c6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:11.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 >> 172.21.15.148:0/1577703103 conn(0x7f0b6406bfe0 msgr2=0x7f0b640cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:11.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 shutdown_connections 2023-10-30T13:21:11.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:11.938+0000 7f0b68b3e640 1 -- 172.21.15.148:0/1577703103 wait complete. 2023-10-30T13:21:11.959 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 1 alive and active, now stable! 2023-10-30T13:21:11.959 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2023-10-30T13:21:11.960 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:21:11.960 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:21:11.971 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:21:11.971 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:12.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39e1b94640 1 Processor -- start 2023-10-30T13:21:12.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39e1b94640 1 -- start start 2023-10-30T13:21:12.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39e1b94640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc1538a0 0x7f39dc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:12.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39e1b94640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39dc0ca0e0 0x7f39dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:12.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39e1b94640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc0cb800 con 0x7f39dc0cf160 2023-10-30T13:21:12.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39e1b94640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39dc15dc30 con 0x7f39dc0ca0e0 2023-10-30T13:21:12.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39e1b94640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0cb330 con 0x7f39dc1538a0 2023-10-30T13:21:12.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39db7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc1538a0 0x7f39dc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:12.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39daffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39dc0ca0e0 0x7f39dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:12.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39db7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc1538a0 0x7f39dc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:40916/0 (socket says 172.21.15.148:40916) 2023-10-30T13:21:12.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39dbfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f39dc0cf160 0x7f39dc0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:39664/0 (socket says 172.21.15.148:39664) 2023-10-30T13:21:12.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39db7fe640 1 -- 172.21.15.148:0/3884545924 learned_addr learned my addr 172.21.15.148:0/3884545924 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:12.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39daffd640 1 -- 172.21.15.148:0/3884545924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc1538a0 msgr2=0x7f39dc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:12.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39daffd640 1 --2- 172.21.15.148:0/3884545924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc1538a0 0x7f39dc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:12.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39daffd640 1 -- 172.21.15.148:0/3884545924 >> v1:172.21.15.150:6789/0 conn(0x7f39dc0cf160 legacy=0x7f39dc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:12.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.162+0000 7f39daffd640 1 -- 172.21.15.148:0/3884545924 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc0cad70 con 0x7f39dc0ca0e0 2023-10-30T13:21:12.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39daffd640 1 --2- 172.21.15.148:0/3884545924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc0ca0e0 0x7f39dc15a090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f39d00097c0 tx=0x7f39d001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=432a9949b6c2373d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:12.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39da7fc640 1 -- 172.21.15.148:0/3884545924 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d00a8970 con 0x7f39dc0ca0e0 2023-10-30T13:21:12.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39da7fc640 1 -- 172.21.15.148:0/3884545924 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d00a8ad0 con 0x7f39dc0ca0e0 2023-10-30T13:21:12.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39da7fc640 1 -- 172.21.15.148:0/3884545924 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d00a8e60 con 0x7f39dc0ca0e0 2023-10-30T13:21:12.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39e1b94640 1 -- 172.21.15.148:0/3884545924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc0ca0e0 msgr2=0x7f39dc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:12.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39e1b94640 1 --2- 172.21.15.148:0/3884545924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc0ca0e0 0x7f39dc15a090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f39d00097c0 tx=0x7f39d001ec70 comp rx=0 tx=0).stop 2023-10-30T13:21:12.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39e1b94640 1 -- 172.21.15.148:0/3884545924 shutdown_connections 2023-10-30T13:21:12.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39e1b94640 1 --2- 172.21.15.148:0/3884545924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc0ca0e0 0x7f39dc15a090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:12.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39e1b94640 1 --2- 172.21.15.148:0/3884545924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc1538a0 0x7f39dc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:12.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39e1b94640 1 -- 172.21.15.148:0/3884545924 >> 172.21.15.148:0/3884545924 conn(0x7f39dc06bfe0 msgr2=0x7f39dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:12.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39e1b94640 1 -- 172.21.15.148:0/3884545924 shutdown_connections 2023-10-30T13:21:12.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.166+0000 7f39e1b94640 1 -- 172.21.15.148:0/3884545924 wait complete. 2023-10-30T13:21:12.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 Processor -- start 2023-10-30T13:21:12.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 -- start start 2023-10-30T13:21:12.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc0ca0e0 0x7f39dc1e2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:12.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39dc0cf160 0x7f39dc065d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:12.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc1538a0 0x7f39dc1e33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:12.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc1e4cf0 con 0x7f39dc0cf160 2023-10-30T13:21:12.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc1e30d0 con 0x7f39dc1538a0 2023-10-30T13:21:12.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc147790 con 0x7f39dc0ca0e0 2023-10-30T13:21:12.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39daffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39dc0cf160 0x7f39dc065d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:12.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39daffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39dc0cf160 0x7f39dc065d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58054/0 (socket says 172.21.15.148:58054) 2023-10-30T13:21:12.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39daffd640 1 -- 172.21.15.148:0/2616938799 learned_addr learned my addr 172.21.15.148:0/2616938799 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:12.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39db7fe640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc0ca0e0 0x7f39dc1e2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:12.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39dbfff640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc1538a0 0x7f39dc1e33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:12.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39daffd640 1 -- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc0ca0e0 msgr2=0x7f39dc1e2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:12.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39daffd640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc0ca0e0 0x7f39dc1e2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39daffd640 1 -- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc1538a0 msgr2=0x7f39dc1e33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39daffd640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc1538a0 0x7f39dc1e33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39daffd640 1 -- 172.21.15.148:0/2616938799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d0009420 con 0x7f39dc0cf160 2023-10-30T13:21:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39dbfff640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc1538a0 0x7f39dc1e33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39daffd640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39dc0cf160 0x7f39dc065d50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f39d001d320 tx=0x7f39d001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:12.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39c3fff640 1 -- 172.21.15.148:0/2616938799 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d00bf070 con 0x7f39dc0cf160 2023-10-30T13:21:12.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39c3fff640 1 -- 172.21.15.148:0/2616938799 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d0006e20 con 0x7f39dc0cf160 2023-10-30T13:21:12.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc1e5960 con 0x7f39dc0cf160 2023-10-30T13:21:12.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39c3fff640 1 -- 172.21.15.148:0/2616938799 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d00b1e20 con 0x7f39dc0cf160 2023-10-30T13:21:12.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.170+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc146e60 con 0x7f39dc0cf160 2023-10-30T13:21:12.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.174+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc0b2a50 con 0x7f39dc0cf160 2023-10-30T13:21:12.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.174+0000 7f39c3fff640 1 -- 172.21.15.148:0/2616938799 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39d00b7040 con 0x7f39dc0cf160 2023-10-30T13:21:12.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.174+0000 7f39c3fff640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39c80664a0 0x7f39c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:12.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.174+0000 7f39c3fff640 1 -- 172.21.15.148:0/2616938799 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f39d0121430 con 0x7f39dc0cf160 2023-10-30T13:21:12.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.174+0000 7f39db7fe640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39c80664a0 0x7f39c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:12.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.174+0000 7f39db7fe640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39c80664a0 0x7f39c8068960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f39cc009ae0 tx=0x7f39cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:12.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.178+0000 7f39c3fff640 1 -- 172.21.15.148:0/2616938799 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39d00ee590 con 0x7f39dc0cf160 2023-10-30T13:21:12.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/21' 2023-10-30T13:21:12.305 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/27' 2023-10-30T13:21:12.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/31' 2023-10-30T13:21:12.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/36' 2023-10-30T13:21:12.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/20' 2023-10-30T13:21:12.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/39' 2023-10-30T13:21:12.306 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/34' 2023-10-30T13:21:12.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/38' 2023-10-30T13:21:12.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/28' 2023-10-30T13:21:12.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/24' 2023-10-30T13:21:12.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/29' 2023-10-30T13:21:12.307 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/32' 2023-10-30T13:21:12.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/22' 2023-10-30T13:21:12.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/25' 2023-10-30T13:21:12.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/30' 2023-10-30T13:21:12.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/33' 2023-10-30T13:21:12.308 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/23' 2023-10-30T13:21:12.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/35' 2023-10-30T13:21:12.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/26' 2023-10-30T13:21:12.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './287/37' 2023-10-30T13:21:12.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './287' 2023-10-30T13:21:12.309 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/21' 2023-10-30T13:21:12.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/27' 2023-10-30T13:21:12.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/31' 2023-10-30T13:21:12.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/36' 2023-10-30T13:21:12.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/20' 2023-10-30T13:21:12.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/39' 2023-10-30T13:21:12.310 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/34' 2023-10-30T13:21:12.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/38' 2023-10-30T13:21:12.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/28' 2023-10-30T13:21:12.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/24' 2023-10-30T13:21:12.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/29' 2023-10-30T13:21:12.311 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/32' 2023-10-30T13:21:12.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/22' 2023-10-30T13:21:12.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/25' 2023-10-30T13:21:12.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/30' 2023-10-30T13:21:12.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/33' 2023-10-30T13:21:12.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/23' 2023-10-30T13:21:12.312 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/35' 2023-10-30T13:21:12.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/26' 2023-10-30T13:21:12.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './288/37' 2023-10-30T13:21:12.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './288' 2023-10-30T13:21:12.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/21' 2023-10-30T13:21:12.313 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/27' 2023-10-30T13:21:12.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/31' 2023-10-30T13:21:12.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/36' 2023-10-30T13:21:12.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/20' 2023-10-30T13:21:12.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/39' 2023-10-30T13:21:12.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/34' 2023-10-30T13:21:12.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/38' 2023-10-30T13:21:12.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/28' 2023-10-30T13:21:12.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/24' 2023-10-30T13:21:12.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/29' 2023-10-30T13:21:12.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/32' 2023-10-30T13:21:12.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/22' 2023-10-30T13:21:12.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/25' 2023-10-30T13:21:12.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/30' 2023-10-30T13:21:12.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/33' 2023-10-30T13:21:12.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/23' 2023-10-30T13:21:12.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/35' 2023-10-30T13:21:12.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/26' 2023-10-30T13:21:12.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './289/37' 2023-10-30T13:21:12.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './289' 2023-10-30T13:21:12.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/21' 2023-10-30T13:21:12.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/27' 2023-10-30T13:21:12.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/31' 2023-10-30T13:21:12.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/36' 2023-10-30T13:21:12.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/20' 2023-10-30T13:21:12.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/39' 2023-10-30T13:21:12.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/34' 2023-10-30T13:21:12.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/38' 2023-10-30T13:21:12.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/28' 2023-10-30T13:21:12.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/24' 2023-10-30T13:21:12.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/29' 2023-10-30T13:21:12.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/32' 2023-10-30T13:21:12.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/22' 2023-10-30T13:21:12.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/25' 2023-10-30T13:21:12.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/30' 2023-10-30T13:21:12.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/33' 2023-10-30T13:21:12.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/23' 2023-10-30T13:21:12.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/35' 2023-10-30T13:21:12.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/26' 2023-10-30T13:21:12.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './290/37' 2023-10-30T13:21:12.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './290' 2023-10-30T13:21:12.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/21' 2023-10-30T13:21:12.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/27' 2023-10-30T13:21:12.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/31' 2023-10-30T13:21:12.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/36' 2023-10-30T13:21:12.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/20' 2023-10-30T13:21:12.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/39' 2023-10-30T13:21:12.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/34' 2023-10-30T13:21:12.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/38' 2023-10-30T13:21:12.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/28' 2023-10-30T13:21:12.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/24' 2023-10-30T13:21:12.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/29' 2023-10-30T13:21:12.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/32' 2023-10-30T13:21:12.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/22' 2023-10-30T13:21:12.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/25' 2023-10-30T13:21:12.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/30' 2023-10-30T13:21:12.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/33' 2023-10-30T13:21:12.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/23' 2023-10-30T13:21:12.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/35' 2023-10-30T13:21:12.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/26' 2023-10-30T13:21:12.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './291/37' 2023-10-30T13:21:12.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './291' 2023-10-30T13:21:12.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/21' 2023-10-30T13:21:12.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/27' 2023-10-30T13:21:12.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/31' 2023-10-30T13:21:12.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/36' 2023-10-30T13:21:12.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/20' 2023-10-30T13:21:12.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/39' 2023-10-30T13:21:12.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/34' 2023-10-30T13:21:12.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/38' 2023-10-30T13:21:12.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/28' 2023-10-30T13:21:12.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/24' 2023-10-30T13:21:12.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/29' 2023-10-30T13:21:12.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/32' 2023-10-30T13:21:12.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/22' 2023-10-30T13:21:12.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/25' 2023-10-30T13:21:12.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/30' 2023-10-30T13:21:12.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/33' 2023-10-30T13:21:12.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/23' 2023-10-30T13:21:12.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/35' 2023-10-30T13:21:12.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/26' 2023-10-30T13:21:12.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './292/37' 2023-10-30T13:21:12.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './292' 2023-10-30T13:21:12.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/21' 2023-10-30T13:21:12.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/27' 2023-10-30T13:21:12.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/31' 2023-10-30T13:21:12.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/36' 2023-10-30T13:21:12.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/20' 2023-10-30T13:21:12.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/39' 2023-10-30T13:21:12.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/34' 2023-10-30T13:21:12.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/38' 2023-10-30T13:21:12.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/28' 2023-10-30T13:21:12.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/24' 2023-10-30T13:21:12.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/29' 2023-10-30T13:21:12.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/32' 2023-10-30T13:21:12.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/22' 2023-10-30T13:21:12.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/25' 2023-10-30T13:21:12.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/30' 2023-10-30T13:21:12.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/33' 2023-10-30T13:21:12.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/23' 2023-10-30T13:21:12.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/35' 2023-10-30T13:21:12.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/26' 2023-10-30T13:21:12.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './293/37' 2023-10-30T13:21:12.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './293' 2023-10-30T13:21:12.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/21' 2023-10-30T13:21:12.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/27' 2023-10-30T13:21:12.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/31' 2023-10-30T13:21:12.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/36' 2023-10-30T13:21:12.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/20' 2023-10-30T13:21:12.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/39' 2023-10-30T13:21:12.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/34' 2023-10-30T13:21:12.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/38' 2023-10-30T13:21:12.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/28' 2023-10-30T13:21:12.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/24' 2023-10-30T13:21:12.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/29' 2023-10-30T13:21:12.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/32' 2023-10-30T13:21:12.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/22' 2023-10-30T13:21:12.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/25' 2023-10-30T13:21:12.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/30' 2023-10-30T13:21:12.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/33' 2023-10-30T13:21:12.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/23' 2023-10-30T13:21:12.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/35' 2023-10-30T13:21:12.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/26' 2023-10-30T13:21:12.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './294/37' 2023-10-30T13:21:12.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './294' 2023-10-30T13:21:12.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/21' 2023-10-30T13:21:12.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/27' 2023-10-30T13:21:12.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/31' 2023-10-30T13:21:12.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/36' 2023-10-30T13:21:12.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/20' 2023-10-30T13:21:12.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/39' 2023-10-30T13:21:12.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/34' 2023-10-30T13:21:12.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/38' 2023-10-30T13:21:12.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/28' 2023-10-30T13:21:12.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/24' 2023-10-30T13:21:12.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/29' 2023-10-30T13:21:12.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/32' 2023-10-30T13:21:12.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/22' 2023-10-30T13:21:12.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/25' 2023-10-30T13:21:12.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/30' 2023-10-30T13:21:12.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/33' 2023-10-30T13:21:12.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/23' 2023-10-30T13:21:12.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/35' 2023-10-30T13:21:12.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/26' 2023-10-30T13:21:12.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './295/37' 2023-10-30T13:21:12.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './295' 2023-10-30T13:21:12.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/21' 2023-10-30T13:21:12.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/27' 2023-10-30T13:21:12.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/31' 2023-10-30T13:21:12.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/36' 2023-10-30T13:21:12.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/20' 2023-10-30T13:21:12.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/39' 2023-10-30T13:21:12.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/34' 2023-10-30T13:21:12.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/38' 2023-10-30T13:21:12.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/28' 2023-10-30T13:21:12.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/24' 2023-10-30T13:21:12.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/29' 2023-10-30T13:21:12.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/32' 2023-10-30T13:21:12.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/22' 2023-10-30T13:21:12.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/25' 2023-10-30T13:21:12.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/30' 2023-10-30T13:21:12.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/33' 2023-10-30T13:21:12.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/23' 2023-10-30T13:21:12.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/35' 2023-10-30T13:21:12.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/26' 2023-10-30T13:21:12.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './296/37' 2023-10-30T13:21:12.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './296' 2023-10-30T13:21:12.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/21' 2023-10-30T13:21:12.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/27' 2023-10-30T13:21:12.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.362+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39dc04e170 con 0x7f39dc0cf160 2023-10-30T13:21:12.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.362+0000 7f39c3fff640 1 -- 172.21.15.148:0/2616938799 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f39d00f1bb0 con 0x7f39dc0cf160 2023-10-30T13:21:12.365 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:12.366 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:11.563063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:12.366 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 53 2023-10-30T13:21:12.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39c80664a0 msgr2=0x7f39c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:12.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39c80664a0 0x7f39c8068960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f39cc009ae0 tx=0x7f39cc008040 comp rx=0 tx=0).stop 2023-10-30T13:21:12.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39dc0cf160 msgr2=0x7f39dc065d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:12.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39dc0cf160 0x7f39dc065d50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f39d001d320 tx=0x7f39d001ba10 comp rx=0 tx=0).stop 2023-10-30T13:21:12.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 shutdown_connections 2023-10-30T13:21:12.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39c80664a0 0x7f39c8068960 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39dc0cf160 0x7f39dc065d50 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39dc1538a0 0x7f39dc1e33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 --2- 172.21.15.148:0/2616938799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39dc0ca0e0 0x7f39dc1e2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 >> 172.21.15.148:0/2616938799 conn(0x7f39dc06bfe0 msgr2=0x7f39dc0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 shutdown_connections 2023-10-30T13:21:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:12.366+0000 7f39e1b94640 1 -- 172.21.15.148:0/2616938799 wait complete. 2023-10-30T13:21:12.386 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:21:14.314 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/31' 2023-10-30T13:21:14.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/36' 2023-10-30T13:21:14.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/20' 2023-10-30T13:21:14.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/39' 2023-10-30T13:21:14.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/34' 2023-10-30T13:21:14.315 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/38' 2023-10-30T13:21:14.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/28' 2023-10-30T13:21:14.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/24' 2023-10-30T13:21:14.316 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/29' 2023-10-30T13:21:14.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/32' 2023-10-30T13:21:14.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/22' 2023-10-30T13:21:14.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/25' 2023-10-30T13:21:14.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/30' 2023-10-30T13:21:14.317 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/33' 2023-10-30T13:21:14.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/23' 2023-10-30T13:21:14.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/35' 2023-10-30T13:21:14.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/26' 2023-10-30T13:21:14.318 INFO:tasks.workunit.client.0.smithi148.stdout:removed './297/37' 2023-10-30T13:21:14.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './297' 2023-10-30T13:21:14.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/21' 2023-10-30T13:21:14.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/27' 2023-10-30T13:21:14.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/31' 2023-10-30T13:21:14.319 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/36' 2023-10-30T13:21:14.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/20' 2023-10-30T13:21:14.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/39' 2023-10-30T13:21:14.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/34' 2023-10-30T13:21:14.320 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/38' 2023-10-30T13:21:14.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/28' 2023-10-30T13:21:14.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/24' 2023-10-30T13:21:14.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/29' 2023-10-30T13:21:14.321 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/32' 2023-10-30T13:21:14.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/22' 2023-10-30T13:21:14.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/25' 2023-10-30T13:21:14.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/30' 2023-10-30T13:21:14.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/33' 2023-10-30T13:21:14.322 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/23' 2023-10-30T13:21:14.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/35' 2023-10-30T13:21:14.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/26' 2023-10-30T13:21:14.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed './298/37' 2023-10-30T13:21:14.323 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './298' 2023-10-30T13:21:14.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/21' 2023-10-30T13:21:14.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/27' 2023-10-30T13:21:14.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/31' 2023-10-30T13:21:14.324 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/36' 2023-10-30T13:21:14.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/20' 2023-10-30T13:21:14.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/39' 2023-10-30T13:21:14.325 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/34' 2023-10-30T13:21:14.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/38' 2023-10-30T13:21:14.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/28' 2023-10-30T13:21:14.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/24' 2023-10-30T13:21:14.326 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/29' 2023-10-30T13:21:14.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/32' 2023-10-30T13:21:14.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/22' 2023-10-30T13:21:14.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/25' 2023-10-30T13:21:14.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/30' 2023-10-30T13:21:14.327 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/33' 2023-10-30T13:21:14.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/23' 2023-10-30T13:21:14.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/35' 2023-10-30T13:21:14.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/26' 2023-10-30T13:21:14.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './299/37' 2023-10-30T13:21:14.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './299' 2023-10-30T13:21:14.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/19' 2023-10-30T13:21:14.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/15' 2023-10-30T13:21:14.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/20' 2023-10-30T13:21:14.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/12' 2023-10-30T13:21:14.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/10' 2023-10-30T13:21:14.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/11' 2023-10-30T13:21:14.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/16' 2023-10-30T13:21:14.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/14' 2023-10-30T13:21:14.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/18' 2023-10-30T13:21:14.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/13' 2023-10-30T13:21:14.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/166/17' 2023-10-30T13:21:14.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/166' 2023-10-30T13:21:14.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/19' 2023-10-30T13:21:14.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/15' 2023-10-30T13:21:14.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/20' 2023-10-30T13:21:14.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/12' 2023-10-30T13:21:14.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/10' 2023-10-30T13:21:14.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/11' 2023-10-30T13:21:14.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/16' 2023-10-30T13:21:14.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/14' 2023-10-30T13:21:14.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/18' 2023-10-30T13:21:14.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/13' 2023-10-30T13:21:14.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/147/17' 2023-10-30T13:21:14.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/147' 2023-10-30T13:21:14.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/19' 2023-10-30T13:21:14.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/15' 2023-10-30T13:21:14.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/20' 2023-10-30T13:21:14.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/12' 2023-10-30T13:21:14.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/10' 2023-10-30T13:21:14.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/11' 2023-10-30T13:21:14.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/16' 2023-10-30T13:21:14.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/14' 2023-10-30T13:21:14.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/18' 2023-10-30T13:21:14.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/13' 2023-10-30T13:21:14.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/151/17' 2023-10-30T13:21:14.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/151' 2023-10-30T13:21:14.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/19' 2023-10-30T13:21:14.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/15' 2023-10-30T13:21:14.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/20' 2023-10-30T13:21:14.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/12' 2023-10-30T13:21:14.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/10' 2023-10-30T13:21:14.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/11' 2023-10-30T13:21:14.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/16' 2023-10-30T13:21:14.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/14' 2023-10-30T13:21:14.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/18' 2023-10-30T13:21:14.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/13' 2023-10-30T13:21:14.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/170/17' 2023-10-30T13:21:14.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/170' 2023-10-30T13:21:14.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/19' 2023-10-30T13:21:14.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/15' 2023-10-30T13:21:14.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/20' 2023-10-30T13:21:14.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/12' 2023-10-30T13:21:14.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/10' 2023-10-30T13:21:14.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/11' 2023-10-30T13:21:14.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/16' 2023-10-30T13:21:14.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/14' 2023-10-30T13:21:14.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/18' 2023-10-30T13:21:14.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/13' 2023-10-30T13:21:14.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/167/17' 2023-10-30T13:21:14.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/167' 2023-10-30T13:21:14.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/19' 2023-10-30T13:21:14.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/15' 2023-10-30T13:21:14.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/20' 2023-10-30T13:21:14.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/12' 2023-10-30T13:21:14.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/10' 2023-10-30T13:21:14.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/11' 2023-10-30T13:21:14.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/16' 2023-10-30T13:21:14.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/14' 2023-10-30T13:21:14.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/18' 2023-10-30T13:21:14.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/13' 2023-10-30T13:21:14.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/140/17' 2023-10-30T13:21:14.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/140' 2023-10-30T13:21:14.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/19' 2023-10-30T13:21:14.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/15' 2023-10-30T13:21:14.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/20' 2023-10-30T13:21:14.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/12' 2023-10-30T13:21:14.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/10' 2023-10-30T13:21:14.387 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:14.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d9572640 1 Processor -- start 2023-10-30T13:21:14.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d9572640 1 -- start start 2023-10-30T13:21:14.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d9572640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67d41538a0 0x7f67d415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:14.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d9572640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d40cf160 0x7f67d40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:14.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d9572640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d40cb800 con 0x7f67d40ca0e0 2023-10-30T13:21:14.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d9572640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d415dc30 con 0x7f67d41538a0 2023-10-30T13:21:14.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d9572640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d40cb330 con 0x7f67d40cf160 2023-10-30T13:21:14.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d27fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f67d40ca0e0 0x7f67d415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:35156/0 (socket says 172.21.15.148:35156) 2023-10-30T13:21:14.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d27fc640 1 -- 172.21.15.148:0/887243184 learned_addr learned my addr 172.21.15.148:0/887243184 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:21:14.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d2ffd640 1 --2- 172.21.15.148:0/887243184 >> v2:172.21.15.150:3300/0 conn(0x7f67d41538a0 0x7f67d415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:14.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d37fe640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d40cf160 0x7f67d40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:14.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145258678 0 0) 0x7f67d40cb800 con 0x7f67d40ca0e0 2023-10-30T13:21:14.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67bc003530 con 0x7f67d40ca0e0 2023-10-30T13:21:14.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3964500741 0 0) 0x7f67bc003530 con 0x7f67d40ca0e0 2023-10-30T13:21:14.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d40cf160 msgr2=0x7f67d40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:14.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d40cf160 0x7f67d40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 >> v2:172.21.15.150:3300/0 conn(0x7f67d41538a0 msgr2=0x7f67d415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:14.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 --2- 172.21.15.148:0/887243184 >> v2:172.21.15.150:3300/0 conn(0x7f67d41538a0 0x7f67d415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d40cad70 con 0x7f67d40ca0e0 2023-10-30T13:21:14.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f67c8001680 con 0x7f67d40ca0e0 2023-10-30T13:21:14.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 >> v1:172.21.15.148:6789/0 conn(0x7f67d40ca0e0 legacy=0x7f67d415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:14.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67bc003980 0x7f67bc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:14.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67bc004370 0x7f67bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:14.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67bc004ec0 0x7f67bc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:14.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d415dc30 con 0x7f67bc004ec0 2023-10-30T13:21:14.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d40cb330 con 0x7f67bc004370 2023-10-30T13:21:14.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67bc00bed0 con 0x7f67bc003980 2023-10-30T13:21:14.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d37fe640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d40cf160 0x7f67d40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:21:14.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d37fe640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67bc004ec0 0x7f67bc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:14.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d2ffd640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67bc003980 0x7f67bc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:14.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d27fc640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67bc004370 0x7f67bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:14.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d37fe640 1 -- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67bc003980 msgr2=0x7f67bc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:14.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d37fe640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67bc003980 0x7f67bc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d37fe640 1 -- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67bc004370 msgr2=0x7f67bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:14.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d37fe640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67bc004370 0x7f67bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.526+0000 7f67d37fe640 1 -- 172.21.15.148:0/887243184 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc003530 con 0x7f67bc004ec0 2023-10-30T13:21:14.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d37fe640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67bc004ec0 0x7f67bc00b990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f67cc00e1f0 tx=0x7f67cc01cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea7240483c27dbc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:14.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc0a5d50 con 0x7f67bc004ec0 2023-10-30T13:21:14.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67cc00f040 con 0x7f67bc004ec0 2023-10-30T13:21:14.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d1ffb640 1 -- 172.21.15.148:0/887243184 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc003bc0 con 0x7f67bc004ec0 2023-10-30T13:21:14.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d9572640 1 -- 172.21.15.148:0/887243184 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67bc004ec0 msgr2=0x7f67bc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:14.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d9572640 1 --2- 172.21.15.148:0/887243184 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67bc004ec0 0x7f67bc00b990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f67cc00e1f0 tx=0x7f67cc01cd00 comp rx=0 tx=0).stop 2023-10-30T13:21:14.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d37fe640 1 -- 172.21.15.148:0/887243184 reap_dead start 2023-10-30T13:21:14.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d9572640 1 -- 172.21.15.148:0/887243184 shutdown_connections 2023-10-30T13:21:14.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d9572640 1 -- 172.21.15.148:0/887243184 >> 172.21.15.148:0/887243184 conn(0x7f67d406bfe0 msgr2=0x7f67d406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:14.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d9572640 1 -- 172.21.15.148:0/887243184 shutdown_connections 2023-10-30T13:21:14.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.530+0000 7f67d9572640 1 -- 172.21.15.148:0/887243184 wait complete. 2023-10-30T13:21:14.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 Processor -- start 2023-10-30T13:21:14.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 -- start start 2023-10-30T13:21:14.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67bc004ec0 0x7f67d41f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:14.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67d41efdb0 0x7f67d40c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:14.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d41592c0 0x7f67d40ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:14.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d415a9d0 con 0x7f67d41efdb0 2023-10-30T13:21:14.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d40c7390 con 0x7f67bc004ec0 2023-10-30T13:21:14.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d40c64d0 con 0x7f67d41592c0 2023-10-30T13:21:14.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67d41efdb0 0x7f67d40c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:14.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67bc004ec0 0x7f67d41f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:14.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67d41efdb0 0x7f67d40c6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58082/0 (socket says 172.21.15.148:58082) 2023-10-30T13:21:14.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d27fc640 1 -- 172.21.15.148:0/1274994587 learned_addr learned my addr 172.21.15.148:0/1274994587 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:14.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d37fe640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d41592c0 0x7f67d40ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:14.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d27fc640 1 -- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d41592c0 msgr2=0x7f67d40ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:14.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d27fc640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d41592c0 0x7f67d40ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d27fc640 1 -- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67bc004ec0 msgr2=0x7f67d41f3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:14.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d27fc640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67bc004ec0 0x7f67d41f3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d27fc640 1 -- 172.21.15.148:0/1274994587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d40c6a30 con 0x7f67d41efdb0 2023-10-30T13:21:14.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d37fe640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d41592c0 0x7f67d40ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:14.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d27fc640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67d41efdb0 0x7f67d40c6e50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f67c8001cd0 tx=0x7f67c800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:14.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67bb7fe640 1 -- 172.21.15.148:0/1274994587 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c800c870 con 0x7f67d41efdb0 2023-10-30T13:21:14.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67bb7fe640 1 -- 172.21.15.148:0/1274994587 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c800c9d0 con 0x7f67d41efdb0 2023-10-30T13:21:14.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67bb7fe640 1 -- 172.21.15.148:0/1274994587 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c800cd80 con 0x7f67d41efdb0 2023-10-30T13:21:14.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d40c6770 con 0x7f67d41efdb0 2023-10-30T13:21:14.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.534+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d41ef5b0 con 0x7f67d41efdb0 2023-10-30T13:21:14.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.538+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d41ef130 con 0x7f67d41efdb0 2023-10-30T13:21:14.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.538+0000 7f67bb7fe640 1 -- 172.21.15.148:0/1274994587 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67c801d460 con 0x7f67d41efdb0 2023-10-30T13:21:14.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.538+0000 7f67bb7fe640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67c00664d0 0x7f67c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:14.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.538+0000 7f67bb7fe640 1 -- 172.21.15.148:0/1274994587 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f67c8084050 con 0x7f67d41efdb0 2023-10-30T13:21:14.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.538+0000 7f67d2ffd640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67c00664d0 0x7f67c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:14.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.538+0000 7f67d2ffd640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67c00664d0 0x7f67c0068990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009b60 tx=0x7f67c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:14.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.542+0000 7f67bb7fe640 1 -- 172.21.15.148:0/1274994587 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67d41ef130 con 0x7f67d41efdb0 2023-10-30T13:21:14.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.722+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67d4159870 con 0x7f67d41efdb0 2023-10-30T13:21:14.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.722+0000 7f67bb7fe640 1 -- 172.21.15.148:0/1274994587 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6514 (secure 0 0 0) 0x7f67d4159870 con 0x7f67d41efdb0 2023-10-30T13:21:14.724 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:14.725 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:11.563063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:14.725 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 54 2023-10-30T13:21:14.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67c00664d0 msgr2=0x7f67c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:14.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67c00664d0 0x7f67c0068990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009b60 tx=0x7f67c4008040 comp rx=0 tx=0).stop 2023-10-30T13:21:14.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67d41efdb0 msgr2=0x7f67d40c6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:14.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67d41efdb0 0x7f67d40c6e50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f67c8001cd0 tx=0x7f67c800ece0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 shutdown_connections 2023-10-30T13:21:14.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67c00664d0 0x7f67c0068990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67d41efdb0 0x7f67d40c6e50 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67d41592c0 0x7f67d40ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 --2- 172.21.15.148:0/1274994587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67bc004ec0 0x7f67d41f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:14.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 >> 172.21.15.148:0/1274994587 conn(0x7f67d406bfe0 msgr2=0x7f67d40cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:14.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 shutdown_connections 2023-10-30T13:21:14.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:14.726+0000 7f67d9572640 1 -- 172.21.15.148:0/1274994587 wait complete. 2023-10-30T13:21:14.745 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:21:16.244 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/11' 2023-10-30T13:21:16.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/16' 2023-10-30T13:21:16.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/14' 2023-10-30T13:21:16.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/18' 2023-10-30T13:21:16.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/13' 2023-10-30T13:21:16.245 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/142/17' 2023-10-30T13:21:16.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/142' 2023-10-30T13:21:16.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/19' 2023-10-30T13:21:16.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/15' 2023-10-30T13:21:16.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/20' 2023-10-30T13:21:16.246 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/12' 2023-10-30T13:21:16.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/10' 2023-10-30T13:21:16.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/11' 2023-10-30T13:21:16.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/16' 2023-10-30T13:21:16.247 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/14' 2023-10-30T13:21:16.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/18' 2023-10-30T13:21:16.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/13' 2023-10-30T13:21:16.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/150/17' 2023-10-30T13:21:16.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/150' 2023-10-30T13:21:16.248 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/19' 2023-10-30T13:21:16.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/15' 2023-10-30T13:21:16.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/20' 2023-10-30T13:21:16.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/12' 2023-10-30T13:21:16.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/10' 2023-10-30T13:21:16.249 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/11' 2023-10-30T13:21:16.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/16' 2023-10-30T13:21:16.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/14' 2023-10-30T13:21:16.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/18' 2023-10-30T13:21:16.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/13' 2023-10-30T13:21:16.250 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/105/17' 2023-10-30T13:21:16.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/105' 2023-10-30T13:21:16.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/19' 2023-10-30T13:21:16.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/15' 2023-10-30T13:21:16.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/20' 2023-10-30T13:21:16.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/12' 2023-10-30T13:21:16.251 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/10' 2023-10-30T13:21:16.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/11' 2023-10-30T13:21:16.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/16' 2023-10-30T13:21:16.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/14' 2023-10-30T13:21:16.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/18' 2023-10-30T13:21:16.252 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/13' 2023-10-30T13:21:16.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/149/17' 2023-10-30T13:21:16.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/149' 2023-10-30T13:21:16.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/19' 2023-10-30T13:21:16.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/15' 2023-10-30T13:21:16.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/20' 2023-10-30T13:21:16.253 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/12' 2023-10-30T13:21:16.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/10' 2023-10-30T13:21:16.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/11' 2023-10-30T13:21:16.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/16' 2023-10-30T13:21:16.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/14' 2023-10-30T13:21:16.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/18' 2023-10-30T13:21:16.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/13' 2023-10-30T13:21:16.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/109/17' 2023-10-30T13:21:16.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/109' 2023-10-30T13:21:16.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/19' 2023-10-30T13:21:16.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/15' 2023-10-30T13:21:16.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/20' 2023-10-30T13:21:16.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/12' 2023-10-30T13:21:16.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/10' 2023-10-30T13:21:16.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/11' 2023-10-30T13:21:16.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/16' 2023-10-30T13:21:16.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/14' 2023-10-30T13:21:16.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/18' 2023-10-30T13:21:16.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/13' 2023-10-30T13:21:16.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/158/17' 2023-10-30T13:21:16.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/158' 2023-10-30T13:21:16.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/19' 2023-10-30T13:21:16.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/15' 2023-10-30T13:21:16.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/20' 2023-10-30T13:21:16.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/12' 2023-10-30T13:21:16.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/10' 2023-10-30T13:21:16.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/11' 2023-10-30T13:21:16.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/16' 2023-10-30T13:21:16.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/14' 2023-10-30T13:21:16.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/18' 2023-10-30T13:21:16.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/13' 2023-10-30T13:21:16.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/106/17' 2023-10-30T13:21:16.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/106' 2023-10-30T13:21:16.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/19' 2023-10-30T13:21:16.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/15' 2023-10-30T13:21:16.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/20' 2023-10-30T13:21:16.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/12' 2023-10-30T13:21:16.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/10' 2023-10-30T13:21:16.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/11' 2023-10-30T13:21:16.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/16' 2023-10-30T13:21:16.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/14' 2023-10-30T13:21:16.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/18' 2023-10-30T13:21:16.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/13' 2023-10-30T13:21:16.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/189/17' 2023-10-30T13:21:16.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/189' 2023-10-30T13:21:16.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/19' 2023-10-30T13:21:16.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/15' 2023-10-30T13:21:16.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/20' 2023-10-30T13:21:16.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/12' 2023-10-30T13:21:16.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/10' 2023-10-30T13:21:16.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/11' 2023-10-30T13:21:16.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/16' 2023-10-30T13:21:16.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/14' 2023-10-30T13:21:16.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/18' 2023-10-30T13:21:16.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/13' 2023-10-30T13:21:16.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/133/17' 2023-10-30T13:21:16.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/133' 2023-10-30T13:21:16.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/19' 2023-10-30T13:21:16.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/15' 2023-10-30T13:21:16.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/20' 2023-10-30T13:21:16.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/12' 2023-10-30T13:21:16.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/10' 2023-10-30T13:21:16.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/11' 2023-10-30T13:21:16.746 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:16.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e9399d640 1 Processor -- start 2023-10-30T13:21:16.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e9399d640 1 -- start start 2023-10-30T13:21:16.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e9399d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5e8c1538a0 0x7f5e8c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:16.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e9399d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5e8c0ca0e0 0x7f5e8c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:16.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e9399d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8c0cb800 con 0x7f5e8c0cf160 2023-10-30T13:21:16.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e9399d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8c15dc30 con 0x7f5e8c1538a0 2023-10-30T13:21:16.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e9399d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8c0cb330 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e91739640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5e8c1538a0 0x7f5e8c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:16.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e91f3a640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5e8c0cf160 0x7f5e8c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:35162/0 (socket says 172.21.15.148:35162) 2023-10-30T13:21:16.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e91f3a640 1 -- 172.21.15.148:0/1729903191 learned_addr learned my addr 172.21.15.148:0/1729903191 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:21:16.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e90f38640 1 --2- 172.21.15.148:0/1729903191 >> v2:172.21.15.150:3300/0 conn(0x7f5e8c0ca0e0 0x7f5e8c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:16.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e91739640 1 -- 172.21.15.148:0/1729903191 >> v2:172.21.15.150:3300/0 conn(0x7f5e8c0ca0e0 msgr2=0x7f5e8c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:16.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e91739640 1 --2- 172.21.15.148:0/1729903191 >> v2:172.21.15.150:3300/0 conn(0x7f5e8c0ca0e0 0x7f5e8c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:16.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e91739640 1 -- 172.21.15.148:0/1729903191 >> v1:172.21.15.148:6789/0 conn(0x7f5e8c0cf160 legacy=0x7f5e8c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:16.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e91739640 1 -- 172.21.15.148:0/1729903191 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c0cad70 con 0x7f5e8c1538a0 2023-10-30T13:21:16.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.894+0000 7f5e83fff640 1 -- 172.21.15.148:0/1729903191 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980068571 0 0) 0x7f5e8c0cb800 con 0x7f5e8c0cf160 2023-10-30T13:21:16.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e91739640 1 --2- 172.21.15.148:0/1729903191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c1538a0 0x7f5e8c15d6f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0b30a0 tx=0x7f5e8801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9aa226abace8629 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:16.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e83fff640 1 -- 172.21.15.148:0/1729903191 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880a8950 con 0x7f5e8c1538a0 2023-10-30T13:21:16.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e83fff640 1 -- 172.21.15.148:0/1729903191 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e880a8ab0 con 0x7f5e8c1538a0 2023-10-30T13:21:16.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e83fff640 1 -- 172.21.15.148:0/1729903191 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880a8e40 con 0x7f5e8c1538a0 2023-10-30T13:21:16.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e9399d640 1 -- 172.21.15.148:0/1729903191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c1538a0 msgr2=0x7f5e8c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:16.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e9399d640 1 --2- 172.21.15.148:0/1729903191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c1538a0 0x7f5e8c15d6f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0b30a0 tx=0x7f5e8801ec50 comp rx=0 tx=0).stop 2023-10-30T13:21:16.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e9399d640 1 -- 172.21.15.148:0/1729903191 shutdown_connections 2023-10-30T13:21:16.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e9399d640 1 --2- 172.21.15.148:0/1729903191 >> v2:172.21.15.150:3300/0 conn(0x7f5e8c0ca0e0 0x7f5e8c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:16.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e9399d640 1 --2- 172.21.15.148:0/1729903191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c1538a0 0x7f5e8c15d6f0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:16.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e9399d640 1 -- 172.21.15.148:0/1729903191 >> 172.21.15.148:0/1729903191 conn(0x7f5e8c06bfe0 msgr2=0x7f5e8c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:16.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e9399d640 1 -- 172.21.15.148:0/1729903191 shutdown_connections 2023-10-30T13:21:16.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.898+0000 7f5e9399d640 1 -- 172.21.15.148:0/1729903191 wait complete. 2023-10-30T13:21:16.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 Processor -- start 2023-10-30T13:21:16.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 -- start start 2023-10-30T13:21:16.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c0ca0e0 0x7f5e8c15c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:16.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8c0cf160 0x7f5e8c158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:16.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e8c1538a0 0x7f5e8c14fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:16.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c1eff90 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c159000 con 0x7f5e8c0cf160 2023-10-30T13:21:16.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c1592a0 con 0x7f5e8c1538a0 2023-10-30T13:21:16.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91739640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c0ca0e0 0x7f5e8c15c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:16.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91739640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c0ca0e0 0x7f5e8c15c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58098/0 (socket says 172.21.15.148:58098) 2023-10-30T13:21:16.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91739640 1 -- 172.21.15.148:0/528637001 learned_addr learned my addr 172.21.15.148:0/528637001 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:16.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e90f38640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8c0cf160 0x7f5e8c158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:16.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91739640 1 -- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e8c1538a0 msgr2=0x7f5e8c14fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:16.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91f3a640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e8c1538a0 0x7f5e8c14fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:16.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91739640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e8c1538a0 0x7f5e8c14fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:16.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91739640 1 -- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8c0cf160 msgr2=0x7f5e8c158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:16.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91739640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8c0cf160 0x7f5e8c158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:16.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91739640 1 -- 172.21.15.148:0/528637001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88009420 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e90f38640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8c0cf160 0x7f5e8c158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:21:16.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e91739640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c0ca0e0 0x7f5e8c15c6d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5e880b3040 tx=0x7f5e8801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:16.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e81ffb640 1 -- 172.21.15.148:0/528637001 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880b9630 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e81ffb640 1 -- 172.21.15.148:0/528637001 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e880b97b0 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e81ffb640 1 -- 172.21.15.148:0/528637001 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880b9b60 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8c159580 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.902+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8c157930 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.906+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8c0b2a50 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.906+0000 7f5e81ffb640 1 -- 172.21.15.148:0/528637001 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5e880b7040 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.906+0000 7f5e81ffb640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e700664d0 0x7f5e70068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:16.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.906+0000 7f5e81ffb640 1 -- 172.21.15.148:0/528637001 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f5e88121f20 con 0x7f5e8c0ca0e0 2023-10-30T13:21:16.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.906+0000 7f5e90f38640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e700664d0 0x7f5e70068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:16.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.906+0000 7f5e90f38640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e700664d0 0x7f5e70068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c158c10 tx=0x7f5e7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:16.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:16.910+0000 7f5e81ffb640 1 -- 172.21.15.148:0/528637001 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5e880ef240 con 0x7f5e8c0ca0e0 2023-10-30T13:21:17.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.102+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e8c04e170 con 0x7f5e8c0ca0e0 2023-10-30T13:21:17.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.102+0000 7f5e81ffb640 1 -- 172.21.15.148:0/528637001 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6514 (secure 0 0 0) 0x7f5e880b5090 con 0x7f5e8c0ca0e0 2023-10-30T13:21:17.105 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:17.105 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:11.563063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:17.105 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 54 2023-10-30T13:21:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e700664d0 msgr2=0x7f5e70068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e700664d0 0x7f5e70068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c158c10 tx=0x7f5e7c008040 comp rx=0 tx=0).stop 2023-10-30T13:21:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c0ca0e0 msgr2=0x7f5e8c15c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c0ca0e0 0x7f5e8c15c6d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5e880b3040 tx=0x7f5e8801b9f0 comp rx=0 tx=0).stop 2023-10-30T13:21:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 shutdown_connections 2023-10-30T13:21:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e700664d0 0x7f5e70068990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e8c1538a0 0x7f5e8c14fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8c0cf160 0x7f5e8c158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 --2- 172.21.15.148:0/528637001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e8c0ca0e0 0x7f5e8c15c6d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 >> 172.21.15.148:0/528637001 conn(0x7f5e8c06bfe0 msgr2=0x7f5e8c150de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 shutdown_connections 2023-10-30T13:21:17.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:17.106+0000 7f5e9399d640 1 -- 172.21.15.148:0/528637001 wait complete. 2023-10-30T13:21:17.126 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:21:17.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/16' 2023-10-30T13:21:17.254 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/14' 2023-10-30T13:21:17.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/18' 2023-10-30T13:21:17.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/13' 2023-10-30T13:21:17.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/173/17' 2023-10-30T13:21:17.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/173' 2023-10-30T13:21:17.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/19' 2023-10-30T13:21:17.255 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/15' 2023-10-30T13:21:17.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/20' 2023-10-30T13:21:17.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/12' 2023-10-30T13:21:17.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/10' 2023-10-30T13:21:17.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/11' 2023-10-30T13:21:17.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/16' 2023-10-30T13:21:17.256 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/14' 2023-10-30T13:21:17.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/18' 2023-10-30T13:21:17.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/13' 2023-10-30T13:21:17.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/174/17' 2023-10-30T13:21:17.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/174' 2023-10-30T13:21:17.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/19' 2023-10-30T13:21:17.257 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/15' 2023-10-30T13:21:17.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/20' 2023-10-30T13:21:17.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/12' 2023-10-30T13:21:17.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/10' 2023-10-30T13:21:17.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/11' 2023-10-30T13:21:17.258 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/16' 2023-10-30T13:21:17.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/14' 2023-10-30T13:21:17.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/18' 2023-10-30T13:21:17.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/13' 2023-10-30T13:21:17.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/155/17' 2023-10-30T13:21:17.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/155' 2023-10-30T13:21:17.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/19' 2023-10-30T13:21:17.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/15' 2023-10-30T13:21:17.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/20' 2023-10-30T13:21:17.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/12' 2023-10-30T13:21:17.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/10' 2023-10-30T13:21:17.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/11' 2023-10-30T13:21:17.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/16' 2023-10-30T13:21:17.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/14' 2023-10-30T13:21:17.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/18' 2023-10-30T13:21:17.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/13' 2023-10-30T13:21:17.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/114/17' 2023-10-30T13:21:17.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/114' 2023-10-30T13:21:17.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/19' 2023-10-30T13:21:17.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/15' 2023-10-30T13:21:17.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/20' 2023-10-30T13:21:17.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/12' 2023-10-30T13:21:17.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/10' 2023-10-30T13:21:17.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/11' 2023-10-30T13:21:17.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/16' 2023-10-30T13:21:17.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/14' 2023-10-30T13:21:17.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/18' 2023-10-30T13:21:17.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/13' 2023-10-30T13:21:17.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/159/17' 2023-10-30T13:21:17.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/159' 2023-10-30T13:21:17.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/19' 2023-10-30T13:21:17.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/15' 2023-10-30T13:21:17.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/20' 2023-10-30T13:21:17.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/12' 2023-10-30T13:21:17.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/10' 2023-10-30T13:21:17.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/11' 2023-10-30T13:21:17.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/16' 2023-10-30T13:21:17.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/14' 2023-10-30T13:21:17.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/18' 2023-10-30T13:21:17.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/13' 2023-10-30T13:21:17.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/187/17' 2023-10-30T13:21:17.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/187' 2023-10-30T13:21:17.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/19' 2023-10-30T13:21:17.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/15' 2023-10-30T13:21:17.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/20' 2023-10-30T13:21:17.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/12' 2023-10-30T13:21:17.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/10' 2023-10-30T13:21:17.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/11' 2023-10-30T13:21:17.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/16' 2023-10-30T13:21:17.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/14' 2023-10-30T13:21:17.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/18' 2023-10-30T13:21:17.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/13' 2023-10-30T13:21:17.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/185/17' 2023-10-30T13:21:17.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/185' 2023-10-30T13:21:17.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/19' 2023-10-30T13:21:17.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/15' 2023-10-30T13:21:17.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/20' 2023-10-30T13:21:17.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/12' 2023-10-30T13:21:17.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/10' 2023-10-30T13:21:17.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/11' 2023-10-30T13:21:17.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/16' 2023-10-30T13:21:17.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/14' 2023-10-30T13:21:17.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/18' 2023-10-30T13:21:17.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/13' 2023-10-30T13:21:17.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/120/17' 2023-10-30T13:21:17.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/120' 2023-10-30T13:21:17.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/19' 2023-10-30T13:21:17.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/15' 2023-10-30T13:21:17.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/20' 2023-10-30T13:21:17.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/12' 2023-10-30T13:21:17.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/10' 2023-10-30T13:21:17.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/11' 2023-10-30T13:21:17.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/16' 2023-10-30T13:21:17.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/14' 2023-10-30T13:21:17.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/18' 2023-10-30T13:21:17.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/13' 2023-10-30T13:21:17.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/110/17' 2023-10-30T13:21:17.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/110' 2023-10-30T13:21:17.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/19' 2023-10-30T13:21:17.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/15' 2023-10-30T13:21:17.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/20' 2023-10-30T13:21:17.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/12' 2023-10-30T13:21:17.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/10' 2023-10-30T13:21:17.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/11' 2023-10-30T13:21:17.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/16' 2023-10-30T13:21:18.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/14' 2023-10-30T13:21:18.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/18' 2023-10-30T13:21:18.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/13' 2023-10-30T13:21:18.259 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/137/17' 2023-10-30T13:21:18.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/137' 2023-10-30T13:21:18.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/19' 2023-10-30T13:21:18.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/15' 2023-10-30T13:21:18.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/20' 2023-10-30T13:21:18.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/12' 2023-10-30T13:21:18.260 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/10' 2023-10-30T13:21:18.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/11' 2023-10-30T13:21:18.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/16' 2023-10-30T13:21:18.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/14' 2023-10-30T13:21:18.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/18' 2023-10-30T13:21:18.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/13' 2023-10-30T13:21:18.261 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/198/17' 2023-10-30T13:21:18.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/198' 2023-10-30T13:21:18.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/19' 2023-10-30T13:21:18.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/15' 2023-10-30T13:21:18.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/20' 2023-10-30T13:21:18.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/12' 2023-10-30T13:21:18.262 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/10' 2023-10-30T13:21:18.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/11' 2023-10-30T13:21:18.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/16' 2023-10-30T13:21:18.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/14' 2023-10-30T13:21:18.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/18' 2023-10-30T13:21:18.263 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/13' 2023-10-30T13:21:18.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/136/17' 2023-10-30T13:21:18.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/136' 2023-10-30T13:21:18.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/19' 2023-10-30T13:21:18.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/15' 2023-10-30T13:21:18.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/20' 2023-10-30T13:21:18.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/12' 2023-10-30T13:21:18.264 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/10' 2023-10-30T13:21:18.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/11' 2023-10-30T13:21:18.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/16' 2023-10-30T13:21:18.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/14' 2023-10-30T13:21:18.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/18' 2023-10-30T13:21:18.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/13' 2023-10-30T13:21:18.265 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/125/17' 2023-10-30T13:21:18.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/125' 2023-10-30T13:21:18.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/19' 2023-10-30T13:21:18.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/15' 2023-10-30T13:21:18.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/20' 2023-10-30T13:21:18.266 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/12' 2023-10-30T13:21:18.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/10' 2023-10-30T13:21:18.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/11' 2023-10-30T13:21:18.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/16' 2023-10-30T13:21:18.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/14' 2023-10-30T13:21:18.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/18' 2023-10-30T13:21:18.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/13' 2023-10-30T13:21:18.267 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/194/17' 2023-10-30T13:21:18.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/194' 2023-10-30T13:21:18.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/19' 2023-10-30T13:21:18.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/15' 2023-10-30T13:21:18.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/20' 2023-10-30T13:21:18.268 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/12' 2023-10-30T13:21:18.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/10' 2023-10-30T13:21:18.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/11' 2023-10-30T13:21:18.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/16' 2023-10-30T13:21:18.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/14' 2023-10-30T13:21:18.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/18' 2023-10-30T13:21:18.269 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/13' 2023-10-30T13:21:18.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/119/17' 2023-10-30T13:21:18.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/119' 2023-10-30T13:21:18.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/19' 2023-10-30T13:21:18.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/15' 2023-10-30T13:21:18.270 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/20' 2023-10-30T13:21:18.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/12' 2023-10-30T13:21:18.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/10' 2023-10-30T13:21:18.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/11' 2023-10-30T13:21:18.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/16' 2023-10-30T13:21:18.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/14' 2023-10-30T13:21:18.271 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/18' 2023-10-30T13:21:18.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/13' 2023-10-30T13:21:18.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/157/17' 2023-10-30T13:21:18.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/157' 2023-10-30T13:21:18.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/19' 2023-10-30T13:21:18.272 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/15' 2023-10-30T13:21:18.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/20' 2023-10-30T13:21:18.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/12' 2023-10-30T13:21:18.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/10' 2023-10-30T13:21:18.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/11' 2023-10-30T13:21:18.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/16' 2023-10-30T13:21:18.273 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/14' 2023-10-30T13:21:18.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/18' 2023-10-30T13:21:18.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/13' 2023-10-30T13:21:18.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/172/17' 2023-10-30T13:21:18.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/172' 2023-10-30T13:21:18.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/19' 2023-10-30T13:21:18.274 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/15' 2023-10-30T13:21:18.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/20' 2023-10-30T13:21:18.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/12' 2023-10-30T13:21:18.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/10' 2023-10-30T13:21:18.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/11' 2023-10-30T13:21:18.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/16' 2023-10-30T13:21:18.275 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/14' 2023-10-30T13:21:18.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/18' 2023-10-30T13:21:18.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/13' 2023-10-30T13:21:18.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/186/17' 2023-10-30T13:21:18.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/186' 2023-10-30T13:21:18.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/19' 2023-10-30T13:21:18.276 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/15' 2023-10-30T13:21:18.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/20' 2023-10-30T13:21:18.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/12' 2023-10-30T13:21:18.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/10' 2023-10-30T13:21:18.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/11' 2023-10-30T13:21:18.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/16' 2023-10-30T13:21:18.277 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/14' 2023-10-30T13:21:19.127 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:19.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.262+0000 7f94f5b65640 1 Processor -- start 2023-10-30T13:21:19.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.262+0000 7f94f5b65640 1 -- start start 2023-10-30T13:21:19.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94f5b65640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f94f0070650 0x7f94f0071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:19.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94f5b65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070c90 0x7f94f00720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:19.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94f5b65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f00c2f20 con 0x7f94f0156f80 2023-10-30T13:21:19.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94f5b65640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f0157360 con 0x7f94f0070650 2023-10-30T13:21:19.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94f5b65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f00c85c0 con 0x7f94f0070c90 2023-10-30T13:21:19.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94e7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f94f0070650 0x7f94f0071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:19.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94e7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f94f0070650 0x7f94f0071bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58100/0 (socket says 172.21.15.148:58100) 2023-10-30T13:21:19.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94e7fff640 1 -- 172.21.15.148:0/105288269 learned_addr learned my addr 172.21.15.148:0/105288269 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:19.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94effff640 1 --2- 172.21.15.148:0/105288269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070c90 0x7f94f00720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:19.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94e7fff640 1 -- 172.21.15.148:0/105288269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070c90 msgr2=0x7f94f00720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:19.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94e7fff640 1 --2- 172.21.15.148:0/105288269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070c90 0x7f94f00720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94e7fff640 1 -- 172.21.15.148:0/105288269 >> v1:172.21.15.150:6789/0 conn(0x7f94f0156f80 legacy=0x7f94f0159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:19.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.266+0000 7f94e7fff640 1 -- 172.21.15.148:0/105288269 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f0071230 con 0x7f94f0070650 2023-10-30T13:21:19.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94e7fff640 1 --2- 172.21.15.148:0/105288269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0070650 0x7f94f0071bb0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f94d8009870 tx=0x7f94d801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d0c8749af15c6e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:19.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94eeffd640 1 -- 172.21.15.148:0/105288269 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d80ab650 con 0x7f94f0070650 2023-10-30T13:21:19.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94eeffd640 1 -- 172.21.15.148:0/105288269 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d80ab7b0 con 0x7f94f0070650 2023-10-30T13:21:19.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94eeffd640 1 -- 172.21.15.148:0/105288269 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d80abb40 con 0x7f94f0070650 2023-10-30T13:21:19.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 -- 172.21.15.148:0/105288269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0070650 msgr2=0x7f94f0071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:19.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 --2- 172.21.15.148:0/105288269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0070650 0x7f94f0071bb0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f94d8009870 tx=0x7f94d801c500 comp rx=0 tx=0).stop 2023-10-30T13:21:19.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 -- 172.21.15.148:0/105288269 shutdown_connections 2023-10-30T13:21:19.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 --2- 172.21.15.148:0/105288269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070c90 0x7f94f00720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 --2- 172.21.15.148:0/105288269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0070650 0x7f94f0071bb0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 -- 172.21.15.148:0/105288269 >> 172.21.15.148:0/105288269 conn(0x7f94f00cbeb0 msgr2=0x7f94f00cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:19.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 -- 172.21.15.148:0/105288269 shutdown_connections 2023-10-30T13:21:19.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 -- 172.21.15.148:0/105288269 wait complete. 2023-10-30T13:21:19.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 Processor -- start 2023-10-30T13:21:19.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 -- start start 2023-10-30T13:21:19.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070650 0x7f94f0154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:19.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f0070c90 0x7f94f0159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:19.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0156f80 0x7f94f01f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:19.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f01f3240 con 0x7f94f0156f80 2023-10-30T13:21:19.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0157fe0 con 0x7f94f0070c90 2023-10-30T13:21:19.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.270+0000 7f94f5b65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f01587f0 con 0x7f94f0070650 2023-10-30T13:21:19.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94effff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0156f80 0x7f94f01f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:19.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94effff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0156f80 0x7f94f01f2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58112/0 (socket says 172.21.15.148:58112) 2023-10-30T13:21:19.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94effff640 1 -- 172.21.15.148:0/2334363036 learned_addr learned my addr 172.21.15.148:0/2334363036 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:19.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94e7fff640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f0070c90 0x7f94f0159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:19.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94ef7fe640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070650 0x7f94f0154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:19.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94effff640 1 -- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070650 msgr2=0x7f94f0154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:19.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94effff640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070650 0x7f94f0154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94effff640 1 -- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f0070c90 msgr2=0x7f94f0159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:19.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94effff640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f0070c90 0x7f94f0159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94effff640 1 -- 172.21.15.148:0/2334363036 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d80094d0 con 0x7f94f0156f80 2023-10-30T13:21:19.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94effff640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0156f80 0x7f94f01f2d00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f94e8007e60 tx=0x7f94e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:19.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94ecff9640 1 -- 172.21.15.148:0/2334363036 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8014420 con 0x7f94f0156f80 2023-10-30T13:21:19.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94ecff9640 1 -- 172.21.15.148:0/2334363036 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e80145a0 con 0x7f94f0156f80 2023-10-30T13:21:19.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94ecff9640 1 -- 172.21.15.148:0/2334363036 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8014950 con 0x7f94f0156f80 2023-10-30T13:21:19.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f0159290 con 0x7f94f0156f80 2023-10-30T13:21:19.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f0158510 con 0x7f94f0156f80 2023-10-30T13:21:19.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f0159420 con 0x7f94f0156f80 2023-10-30T13:21:19.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94ecff9640 1 -- 172.21.15.148:0/2334363036 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94e8014ab0 con 0x7f94f0156f80 2023-10-30T13:21:19.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94ecff9640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94d00664a0 0x7f94d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:19.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.274+0000 7f94ecff9640 1 -- 172.21.15.148:0/2334363036 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f94e8084900 con 0x7f94f0156f80 2023-10-30T13:21:19.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.278+0000 7f94ef7fe640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94d00664a0 0x7f94d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:19.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.278+0000 7f94ef7fe640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94d00664a0 0x7f94d0068960 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f94e00098e0 tx=0x7f94e0009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:19.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.278+0000 7f94ecff9640 1 -- 172.21.15.148:0/2334363036 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94e8051ba0 con 0x7f94f0156f80 2023-10-30T13:21:19.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.458+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94f004e170 con 0x7f94f0156f80 2023-10-30T13:21:19.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.458+0000 7f94ecff9640 1 -- 172.21.15.148:0/2334363036 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6514 (secure 0 0 0) 0x7f94e80551c0 con 0x7f94f0156f80 2023-10-30T13:21:19.460 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:19.460 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:11.563063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:19.461 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 54 2023-10-30T13:21:19.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94d00664a0 msgr2=0x7f94d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:19.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94d00664a0 0x7f94d0068960 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f94e00098e0 tx=0x7f94e0009060 comp rx=0 tx=0).stop 2023-10-30T13:21:19.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0156f80 msgr2=0x7f94f01f2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:19.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0156f80 0x7f94f01f2d00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f94e8007e60 tx=0x7f94e800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 shutdown_connections 2023-10-30T13:21:19.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94d00664a0 0x7f94d0068960 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94f0156f80 0x7f94f01f2d00 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f0070c90 0x7f94f0159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 --2- 172.21.15.148:0/2334363036 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94f0070650 0x7f94f0154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:19.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 >> 172.21.15.148:0/2334363036 conn(0x7f94f00cbeb0 msgr2=0x7f94f00c7590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:19.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 shutdown_connections 2023-10-30T13:21:19.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:19.462+0000 7f94f5b65640 1 -- 172.21.15.148:0/2334363036 wait complete. 2023-10-30T13:21:19.482 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:21:20.167 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/18' 2023-10-30T13:21:20.167 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/13' 2023-10-30T13:21:20.167 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/190/17' 2023-10-30T13:21:20.167 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/190' 2023-10-30T13:21:20.168 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/19' 2023-10-30T13:21:20.168 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/15' 2023-10-30T13:21:20.168 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/20' 2023-10-30T13:21:20.168 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/12' 2023-10-30T13:21:20.168 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/10' 2023-10-30T13:21:20.168 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/11' 2023-10-30T13:21:20.169 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/16' 2023-10-30T13:21:20.169 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/14' 2023-10-30T13:21:20.169 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/18' 2023-10-30T13:21:20.169 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/13' 2023-10-30T13:21:20.169 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/104/17' 2023-10-30T13:21:20.169 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/104' 2023-10-30T13:21:20.170 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/19' 2023-10-30T13:21:20.170 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/15' 2023-10-30T13:21:20.170 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/20' 2023-10-30T13:21:20.170 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/12' 2023-10-30T13:21:20.170 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/10' 2023-10-30T13:21:20.170 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/11' 2023-10-30T13:21:20.171 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/16' 2023-10-30T13:21:20.171 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/14' 2023-10-30T13:21:20.171 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/18' 2023-10-30T13:21:20.171 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/13' 2023-10-30T13:21:20.171 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/164/17' 2023-10-30T13:21:20.172 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/164' 2023-10-30T13:21:20.172 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/19' 2023-10-30T13:21:20.172 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/15' 2023-10-30T13:21:20.172 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/20' 2023-10-30T13:21:20.172 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/12' 2023-10-30T13:21:20.173 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/10' 2023-10-30T13:21:20.173 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/11' 2023-10-30T13:21:20.173 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/16' 2023-10-30T13:21:20.173 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/14' 2023-10-30T13:21:20.173 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/18' 2023-10-30T13:21:20.173 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/13' 2023-10-30T13:21:20.174 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/112/17' 2023-10-30T13:21:20.174 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/112' 2023-10-30T13:21:20.174 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/19' 2023-10-30T13:21:20.174 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/15' 2023-10-30T13:21:20.174 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/20' 2023-10-30T13:21:20.174 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/12' 2023-10-30T13:21:20.175 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/10' 2023-10-30T13:21:20.175 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/11' 2023-10-30T13:21:20.175 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/16' 2023-10-30T13:21:20.175 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/14' 2023-10-30T13:21:20.175 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/18' 2023-10-30T13:21:20.175 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/13' 2023-10-30T13:21:20.176 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/100/17' 2023-10-30T13:21:20.176 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/100' 2023-10-30T13:21:20.176 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/19' 2023-10-30T13:21:20.176 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/15' 2023-10-30T13:21:20.176 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/20' 2023-10-30T13:21:20.176 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/12' 2023-10-30T13:21:20.177 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/10' 2023-10-30T13:21:20.177 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/11' 2023-10-30T13:21:20.177 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/16' 2023-10-30T13:21:20.177 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/14' 2023-10-30T13:21:20.177 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/18' 2023-10-30T13:21:20.177 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/13' 2023-10-30T13:21:20.178 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/134/17' 2023-10-30T13:21:20.178 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/134' 2023-10-30T13:21:20.178 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/19' 2023-10-30T13:21:20.178 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/15' 2023-10-30T13:21:20.178 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/20' 2023-10-30T13:21:20.178 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/12' 2023-10-30T13:21:20.179 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/10' 2023-10-30T13:21:20.179 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/11' 2023-10-30T13:21:20.179 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/16' 2023-10-30T13:21:20.179 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/14' 2023-10-30T13:21:20.179 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/18' 2023-10-30T13:21:20.179 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/13' 2023-10-30T13:21:20.180 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/115/17' 2023-10-30T13:21:20.180 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/115' 2023-10-30T13:21:20.180 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/19' 2023-10-30T13:21:20.180 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/15' 2023-10-30T13:21:20.180 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/20' 2023-10-30T13:21:20.180 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/12' 2023-10-30T13:21:20.181 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/10' 2023-10-30T13:21:20.181 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/11' 2023-10-30T13:21:20.181 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/16' 2023-10-30T13:21:20.181 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/14' 2023-10-30T13:21:20.181 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/18' 2023-10-30T13:21:20.181 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/13' 2023-10-30T13:21:20.182 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/116/17' 2023-10-30T13:21:20.182 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/116' 2023-10-30T13:21:20.182 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/19' 2023-10-30T13:21:20.182 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/15' 2023-10-30T13:21:20.182 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/20' 2023-10-30T13:21:20.182 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/12' 2023-10-30T13:21:20.183 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/10' 2023-10-30T13:21:20.183 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/11' 2023-10-30T13:21:20.183 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/16' 2023-10-30T13:21:20.183 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/14' 2023-10-30T13:21:20.183 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/18' 2023-10-30T13:21:20.183 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/13' 2023-10-30T13:21:20.184 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/195/17' 2023-10-30T13:21:20.184 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/195' 2023-10-30T13:21:20.184 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/19' 2023-10-30T13:21:20.184 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/15' 2023-10-30T13:21:20.184 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/20' 2023-10-30T13:21:20.184 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/12' 2023-10-30T13:21:20.185 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/10' 2023-10-30T13:21:20.185 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/11' 2023-10-30T13:21:20.185 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/16' 2023-10-30T13:21:20.185 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/14' 2023-10-30T13:21:20.185 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/18' 2023-10-30T13:21:21.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/13' 2023-10-30T13:21:21.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/138/17' 2023-10-30T13:21:21.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/138' 2023-10-30T13:21:21.328 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/19' 2023-10-30T13:21:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/15' 2023-10-30T13:21:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/20' 2023-10-30T13:21:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/12' 2023-10-30T13:21:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/10' 2023-10-30T13:21:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/11' 2023-10-30T13:21:21.329 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/16' 2023-10-30T13:21:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/14' 2023-10-30T13:21:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/18' 2023-10-30T13:21:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/13' 2023-10-30T13:21:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/135/17' 2023-10-30T13:21:21.330 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/135' 2023-10-30T13:21:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/19' 2023-10-30T13:21:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/15' 2023-10-30T13:21:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/20' 2023-10-30T13:21:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/12' 2023-10-30T13:21:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/10' 2023-10-30T13:21:21.331 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/11' 2023-10-30T13:21:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/16' 2023-10-30T13:21:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/14' 2023-10-30T13:21:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/18' 2023-10-30T13:21:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/13' 2023-10-30T13:21:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/117/17' 2023-10-30T13:21:21.332 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/117' 2023-10-30T13:21:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/19' 2023-10-30T13:21:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/15' 2023-10-30T13:21:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/20' 2023-10-30T13:21:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/12' 2023-10-30T13:21:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/10' 2023-10-30T13:21:21.333 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/11' 2023-10-30T13:21:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/16' 2023-10-30T13:21:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/14' 2023-10-30T13:21:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/18' 2023-10-30T13:21:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/13' 2023-10-30T13:21:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/143/17' 2023-10-30T13:21:21.334 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/143' 2023-10-30T13:21:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/19' 2023-10-30T13:21:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/15' 2023-10-30T13:21:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/20' 2023-10-30T13:21:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/12' 2023-10-30T13:21:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/10' 2023-10-30T13:21:21.335 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/11' 2023-10-30T13:21:21.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/16' 2023-10-30T13:21:21.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/14' 2023-10-30T13:21:21.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/18' 2023-10-30T13:21:21.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/13' 2023-10-30T13:21:21.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/180/17' 2023-10-30T13:21:21.336 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/180' 2023-10-30T13:21:21.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/19' 2023-10-30T13:21:21.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/15' 2023-10-30T13:21:21.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/20' 2023-10-30T13:21:21.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/12' 2023-10-30T13:21:21.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/10' 2023-10-30T13:21:21.337 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/11' 2023-10-30T13:21:21.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/16' 2023-10-30T13:21:21.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/14' 2023-10-30T13:21:21.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/18' 2023-10-30T13:21:21.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/13' 2023-10-30T13:21:21.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/126/17' 2023-10-30T13:21:21.338 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/126' 2023-10-30T13:21:21.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/19' 2023-10-30T13:21:21.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/15' 2023-10-30T13:21:21.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/20' 2023-10-30T13:21:21.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/12' 2023-10-30T13:21:21.339 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/10' 2023-10-30T13:21:21.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/11' 2023-10-30T13:21:21.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/16' 2023-10-30T13:21:21.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/14' 2023-10-30T13:21:21.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/18' 2023-10-30T13:21:21.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/13' 2023-10-30T13:21:21.340 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/161/17' 2023-10-30T13:21:21.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/161' 2023-10-30T13:21:21.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/19' 2023-10-30T13:21:21.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/15' 2023-10-30T13:21:21.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/20' 2023-10-30T13:21:21.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/12' 2023-10-30T13:21:21.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/10' 2023-10-30T13:21:21.341 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/11' 2023-10-30T13:21:21.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/16' 2023-10-30T13:21:21.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/14' 2023-10-30T13:21:21.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/18' 2023-10-30T13:21:21.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/13' 2023-10-30T13:21:21.342 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/176/17' 2023-10-30T13:21:21.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/176' 2023-10-30T13:21:21.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/19' 2023-10-30T13:21:21.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/15' 2023-10-30T13:21:21.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/20' 2023-10-30T13:21:21.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/12' 2023-10-30T13:21:21.343 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/10' 2023-10-30T13:21:21.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/11' 2023-10-30T13:21:21.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/16' 2023-10-30T13:21:21.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/14' 2023-10-30T13:21:21.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/18' 2023-10-30T13:21:21.344 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/13' 2023-10-30T13:21:21.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/111/17' 2023-10-30T13:21:21.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/111' 2023-10-30T13:21:21.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/19' 2023-10-30T13:21:21.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/15' 2023-10-30T13:21:21.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/20' 2023-10-30T13:21:21.345 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/12' 2023-10-30T13:21:21.346 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/10' 2023-10-30T13:21:21.346 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/11' 2023-10-30T13:21:21.346 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/16' 2023-10-30T13:21:21.346 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/14' 2023-10-30T13:21:21.346 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/18' 2023-10-30T13:21:21.346 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/13' 2023-10-30T13:21:21.483 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:21.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.630+0000 7ffab709c640 1 Processor -- start 2023-10-30T13:21:21.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.630+0000 7ffab709c640 1 -- start start 2023-10-30T13:21:21.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab709c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab00c6bd0 0x7ffab006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:21.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab709c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffab0071110 0x7ffab0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:21.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab709c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab0066cf0 con 0x7ffab006f7a0 2023-10-30T13:21:21.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab709c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab00700c0 con 0x7ffab0071110 2023-10-30T13:21:21.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab709c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00c70d0 con 0x7ffab00c6bd0 2023-10-30T13:21:21.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab5639640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffab0071110 0x7ffab0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:21.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab5639640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffab0071110 0x7ffab0158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58130/0 (socket says 172.21.15.148:58130) 2023-10-30T13:21:21.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab5639640 1 -- 172.21.15.148:0/2655200346 learned_addr learned my addr 172.21.15.148:0/2655200346 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:21.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffaaffff640 1 --2- 172.21.15.148:0/2655200346 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab00c6bd0 0x7ffab006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:21.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab5639640 1 -- 172.21.15.148:0/2655200346 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab00c6bd0 msgr2=0x7ffab006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:21.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab5639640 1 --2- 172.21.15.148:0/2655200346 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab00c6bd0 0x7ffab006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab5639640 1 -- 172.21.15.148:0/2655200346 >> v1:172.21.15.150:6789/0 conn(0x7ffab006f7a0 legacy=0x7ffab0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:21.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.634+0000 7ffab5639640 1 -- 172.21.15.148:0/2655200346 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab00c7390 con 0x7ffab0071110 2023-10-30T13:21:21.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab5639640 1 --2- 172.21.15.148:0/2655200346 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab0071110 0x7ffab0158110 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009e50 tx=0x7ffaa801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a2a390dcba45ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:21.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffaaf7fe640 1 -- 172.21.15.148:0/2655200346 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80af400 con 0x7ffab0071110 2023-10-30T13:21:21.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffaaf7fe640 1 -- 172.21.15.148:0/2655200346 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa80ab050 con 0x7ffab0071110 2023-10-30T13:21:21.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffaaf7fe640 1 -- 172.21.15.148:0/2655200346 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80af780 con 0x7ffab0071110 2023-10-30T13:21:21.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 -- 172.21.15.148:0/2655200346 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab0071110 msgr2=0x7ffab0158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:21.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 --2- 172.21.15.148:0/2655200346 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab0071110 0x7ffab0158110 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009e50 tx=0x7ffaa801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 -- 172.21.15.148:0/2655200346 shutdown_connections 2023-10-30T13:21:21.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 --2- 172.21.15.148:0/2655200346 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab0071110 0x7ffab0158110 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 --2- 172.21.15.148:0/2655200346 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab00c6bd0 0x7ffab006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 -- 172.21.15.148:0/2655200346 >> 172.21.15.148:0/2655200346 conn(0x7ffab00c7910 msgr2=0x7ffab00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:21.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 -- 172.21.15.148:0/2655200346 shutdown_connections 2023-10-30T13:21:21.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 -- 172.21.15.148:0/2655200346 wait complete. 2023-10-30T13:21:21.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 Processor -- start 2023-10-30T13:21:21.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 -- start start 2023-10-30T13:21:21.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab006f7a0 0x7ffab0159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:21.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.638+0000 7ffab709c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab0071110 0x7ffab01f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:21.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab709c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab00c6bd0 0x7ffab00ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:21.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab709c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00c1f90 con 0x7ffab00c6bd0 2023-10-30T13:21:21.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab709c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab0158a40 con 0x7ffab006f7a0 2023-10-30T13:21:21.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab709c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab00c1cb0 con 0x7ffab0071110 2023-10-30T13:21:21.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab5639640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab00c6bd0 0x7ffab00ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:21.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab5639640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab00c6bd0 0x7ffab00ae920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58134/0 (socket says 172.21.15.148:58134) 2023-10-30T13:21:21.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab5639640 1 -- 172.21.15.148:0/806681803 learned_addr learned my addr 172.21.15.148:0/806681803 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:21.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffaaffff640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab0071110 0x7ffab01f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:21.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab4e38640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab006f7a0 0x7ffab0159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:21.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab5639640 1 -- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab0071110 msgr2=0x7ffab01f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:21.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab5639640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab0071110 0x7ffab01f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab5639640 1 -- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab006f7a0 msgr2=0x7ffab0159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:21.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab5639640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab006f7a0 0x7ffab0159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab5639640 1 -- 172.21.15.148:0/806681803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa8009b30 con 0x7ffab00c6bd0 2023-10-30T13:21:21.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab5639640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab00c6bd0 0x7ffab00ae920 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ffaa801ce20 tx=0x7ffaa801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:21.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffaad7fa640 1 -- 172.21.15.148:0/806681803 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80afde0 con 0x7ffab00c6bd0 2023-10-30T13:21:21.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffaad7fa640 1 -- 172.21.15.148:0/806681803 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa8009510 con 0x7ffab00c6bd0 2023-10-30T13:21:21.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffaad7fa640 1 -- 172.21.15.148:0/806681803 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80ba660 con 0x7ffab00c6bd0 2023-10-30T13:21:21.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab01f2e50 con 0x7ffab00c6bd0 2023-10-30T13:21:21.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab00c4190 con 0x7ffab00c6bd0 2023-10-30T13:21:21.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab01f2fe0 con 0x7ffab00c6bd0 2023-10-30T13:21:21.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.642+0000 7ffaad7fa640 1 -- 172.21.15.148:0/806681803 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffaa80b6040 con 0x7ffab00c6bd0 2023-10-30T13:21:21.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.646+0000 7ffaad7fa640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:21.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.646+0000 7ffaad7fa640 1 -- 172.21.15.148:0/806681803 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7ffaa8122190 con 0x7ffab00c6bd0 2023-10-30T13:21:21.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.646+0000 7ffab4e38640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:21.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.646+0000 7ffab4e38640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4007fd0 tx=0x7ffaa4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:21.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.650+0000 7ffaad7fa640 1 -- 172.21.15.148:0/806681803 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffaa80ef4f0 con 0x7ffab00c6bd0 2023-10-30T13:21:21.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.834+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffab00c13d0 con 0x7ffab00c6bd0 2023-10-30T13:21:21.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.834+0000 7ffaad7fa640 1 -- 172.21.15.148:0/806681803 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6514 (secure 0 0 0) 0x7ffaa80f2b10 con 0x7ffab00c6bd0 2023-10-30T13:21:21.836 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:21.836 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:11.563063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:21.836 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 54 2023-10-30T13:21:21.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 msgr2=0x7ffa94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:21.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4007fd0 tx=0x7ffaa4009040 comp rx=0 tx=0).stop 2023-10-30T13:21:21.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab00c6bd0 msgr2=0x7ffab00ae920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:21.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab00c6bd0 0x7ffab00ae920 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ffaa801ce20 tx=0x7ffaa801bf90 comp rx=0 tx=0).stop 2023-10-30T13:21:21.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 shutdown_connections 2023-10-30T13:21:21.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa940664a0 0x7ffa94068960 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab00c6bd0 0x7ffab00ae920 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffab0071110 0x7ffab01f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 --2- 172.21.15.148:0/806681803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffab006f7a0 0x7ffab0159dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:21.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 >> 172.21.15.148:0/806681803 conn(0x7ffab00c7910 msgr2=0x7ffab00c9290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:21.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 shutdown_connections 2023-10-30T13:21:21.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:21.838+0000 7ffab709c640 1 -- 172.21.15.148:0/806681803 wait complete. 2023-10-30T13:21:21.858 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:21:23.859 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:23.914 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/191/17' 2023-10-30T13:21:23.914 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/191' 2023-10-30T13:21:23.914 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/19' 2023-10-30T13:21:23.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/15' 2023-10-30T13:21:23.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/20' 2023-10-30T13:21:23.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/12' 2023-10-30T13:21:23.915 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/10' 2023-10-30T13:21:23.916 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/11' 2023-10-30T13:21:23.916 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/16' 2023-10-30T13:21:23.916 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/14' 2023-10-30T13:21:23.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/18' 2023-10-30T13:21:23.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/13' 2023-10-30T13:21:23.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/184/17' 2023-10-30T13:21:23.917 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/184' 2023-10-30T13:21:23.918 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/19' 2023-10-30T13:21:23.918 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/15' 2023-10-30T13:21:23.918 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/20' 2023-10-30T13:21:23.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/12' 2023-10-30T13:21:23.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/10' 2023-10-30T13:21:23.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/11' 2023-10-30T13:21:23.919 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/16' 2023-10-30T13:21:23.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/14' 2023-10-30T13:21:23.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/18' 2023-10-30T13:21:23.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/13' 2023-10-30T13:21:23.920 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/192/17' 2023-10-30T13:21:23.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/192' 2023-10-30T13:21:23.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/19' 2023-10-30T13:21:23.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/15' 2023-10-30T13:21:23.921 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/20' 2023-10-30T13:21:23.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/12' 2023-10-30T13:21:23.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/10' 2023-10-30T13:21:23.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/11' 2023-10-30T13:21:23.922 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/16' 2023-10-30T13:21:23.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/14' 2023-10-30T13:21:23.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/18' 2023-10-30T13:21:23.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/13' 2023-10-30T13:21:23.923 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/103/17' 2023-10-30T13:21:23.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/103' 2023-10-30T13:21:23.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/19' 2023-10-30T13:21:23.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/15' 2023-10-30T13:21:23.924 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/20' 2023-10-30T13:21:23.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/12' 2023-10-30T13:21:23.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/10' 2023-10-30T13:21:23.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/11' 2023-10-30T13:21:23.925 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/16' 2023-10-30T13:21:23.926 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/14' 2023-10-30T13:21:23.926 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/18' 2023-10-30T13:21:23.926 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/13' 2023-10-30T13:21:23.926 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/128/17' 2023-10-30T13:21:23.927 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/128' 2023-10-30T13:21:23.927 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/19' 2023-10-30T13:21:23.927 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/15' 2023-10-30T13:21:23.927 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/20' 2023-10-30T13:21:23.928 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/12' 2023-10-30T13:21:23.928 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/10' 2023-10-30T13:21:23.928 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/11' 2023-10-30T13:21:23.929 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/16' 2023-10-30T13:21:23.929 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/14' 2023-10-30T13:21:23.929 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/18' 2023-10-30T13:21:23.929 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/13' 2023-10-30T13:21:23.930 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/144/17' 2023-10-30T13:21:23.930 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/144' 2023-10-30T13:21:23.930 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/19' 2023-10-30T13:21:23.930 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/15' 2023-10-30T13:21:23.931 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/20' 2023-10-30T13:21:23.931 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/12' 2023-10-30T13:21:23.931 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/10' 2023-10-30T13:21:23.931 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/11' 2023-10-30T13:21:23.932 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/16' 2023-10-30T13:21:23.932 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/14' 2023-10-30T13:21:23.932 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/18' 2023-10-30T13:21:23.932 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/13' 2023-10-30T13:21:23.932 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/146/17' 2023-10-30T13:21:23.933 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/146' 2023-10-30T13:21:23.933 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/19' 2023-10-30T13:21:23.933 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/15' 2023-10-30T13:21:23.933 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/20' 2023-10-30T13:21:23.934 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/12' 2023-10-30T13:21:23.934 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/10' 2023-10-30T13:21:23.934 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/11' 2023-10-30T13:21:23.934 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/16' 2023-10-30T13:21:23.935 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/14' 2023-10-30T13:21:23.935 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/18' 2023-10-30T13:21:23.935 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/13' 2023-10-30T13:21:23.935 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/141/17' 2023-10-30T13:21:23.936 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/141' 2023-10-30T13:21:23.936 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/19' 2023-10-30T13:21:23.936 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/15' 2023-10-30T13:21:23.936 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/20' 2023-10-30T13:21:23.937 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/12' 2023-10-30T13:21:23.937 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/10' 2023-10-30T13:21:23.937 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/11' 2023-10-30T13:21:23.937 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/16' 2023-10-30T13:21:23.937 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/14' 2023-10-30T13:21:23.938 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/18' 2023-10-30T13:21:23.938 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/13' 2023-10-30T13:21:23.938 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/182/17' 2023-10-30T13:21:23.938 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/182' 2023-10-30T13:21:23.939 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/19' 2023-10-30T13:21:23.939 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/15' 2023-10-30T13:21:23.939 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/20' 2023-10-30T13:21:23.939 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/12' 2023-10-30T13:21:23.940 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/10' 2023-10-30T13:21:23.940 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/11' 2023-10-30T13:21:23.940 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/16' 2023-10-30T13:21:23.940 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/14' 2023-10-30T13:21:23.941 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/18' 2023-10-30T13:21:23.941 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/13' 2023-10-30T13:21:23.941 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/107/17' 2023-10-30T13:21:24.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.010+0000 7f715cc17640 1 Processor -- start 2023-10-30T13:21:24.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f715cc17640 1 -- start start 2023-10-30T13:21:24.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f715cc17640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f715815cd50 0x7f71580c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:24.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f715cc17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71580cefa0 0x7f71580cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:24.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f715cc17640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715815a6c0 con 0x7f71581538f0 2023-10-30T13:21:24.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f715cc17640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715815d300 con 0x7f715815cd50 2023-10-30T13:21:24.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f715cc17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715815d5e0 con 0x7f71580cefa0 2023-10-30T13:21:24.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f7156ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f715815cd50 0x7f71580c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:24.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f71577fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f71581538f0 0x7f7158159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:40272/0 (socket says 172.21.15.148:40272) 2023-10-30T13:21:24.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f71577fe640 1 -- 172.21.15.148:0/3976418429 learned_addr learned my addr 172.21.15.148:0/3976418429 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:21:24.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f7157fff640 1 --2- 172.21.15.148:0/3976418429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71580cefa0 0x7f71580cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:24.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f7156ffd640 1 -- 172.21.15.148:0/3976418429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71580cefa0 msgr2=0x7f71580cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:24.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f7156ffd640 1 --2- 172.21.15.148:0/3976418429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71580cefa0 0x7f71580cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f7156ffd640 1 -- 172.21.15.148:0/3976418429 >> v1:172.21.15.150:6789/0 conn(0x7f71581538f0 legacy=0x7f7158159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:24.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.014+0000 7f7156ffd640 1 -- 172.21.15.148:0/3976418429 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715815d8c0 con 0x7f715815cd50 2023-10-30T13:21:24.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f7156ffd640 1 --2- 172.21.15.148:0/3976418429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f715815cd50 0x7f71580c9f90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f714c0097c0 tx=0x7f714c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70626aefd92d74e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:24.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f71567fc640 1 -- 172.21.15.148:0/3976418429 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c0a8970 con 0x7f715815cd50 2023-10-30T13:21:24.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f71567fc640 1 -- 172.21.15.148:0/3976418429 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c0a8ad0 con 0x7f715815cd50 2023-10-30T13:21:24.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f71567fc640 1 -- 172.21.15.148:0/3976418429 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c0a8e60 con 0x7f715815cd50 2023-10-30T13:21:24.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f715cc17640 1 -- 172.21.15.148:0/3976418429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f715815cd50 msgr2=0x7f71580c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:24.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f715cc17640 1 --2- 172.21.15.148:0/3976418429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f715815cd50 0x7f71580c9f90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f714c0097c0 tx=0x7f714c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:21:24.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f715cc17640 1 -- 172.21.15.148:0/3976418429 shutdown_connections 2023-10-30T13:21:24.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f715cc17640 1 --2- 172.21.15.148:0/3976418429 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71580cefa0 0x7f71580cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f715cc17640 1 --2- 172.21.15.148:0/3976418429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f715815cd50 0x7f71580c9f90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f715cc17640 1 -- 172.21.15.148:0/3976418429 >> 172.21.15.148:0/3976418429 conn(0x7f715806a640 msgr2=0x7f7158065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:24.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f715cc17640 1 -- 172.21.15.148:0/3976418429 shutdown_connections 2023-10-30T13:21:24.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.018+0000 7f715cc17640 1 -- 172.21.15.148:0/3976418429 wait complete. 2023-10-30T13:21:24.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 Processor -- start 2023-10-30T13:21:24.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 -- start start 2023-10-30T13:21:24.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71580cefa0 0x7f71581e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:24.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71581538f0 0x7f71581f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:24.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f715815cd50 0x7f715813be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:24.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71581f3700 con 0x7f71580cefa0 2023-10-30T13:21:24.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71581e5bd0 con 0x7f71581538f0 2023-10-30T13:21:24.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71581e5650 con 0x7f715815cd50 2023-10-30T13:21:24.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f71577fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71580cefa0 0x7f71581e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:24.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f71577fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71580cefa0 0x7f71581e3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41242/0 (socket says 172.21.15.148:41242) 2023-10-30T13:21:24.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f71577fe640 1 -- 172.21.15.148:0/2586411200 learned_addr learned my addr 172.21.15.148:0/2586411200 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:24.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f7156ffd640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71581538f0 0x7f71581f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:24.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f7157fff640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f715815cd50 0x7f715813be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:24.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f71577fe640 1 -- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f715815cd50 msgr2=0x7f715813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:24.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f71577fe640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f715815cd50 0x7f715813be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f71577fe640 1 -- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71581538f0 msgr2=0x7f71581f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:24.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f71577fe640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71581538f0 0x7f71581f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f71577fe640 1 -- 172.21.15.148:0/2586411200 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c009420 con 0x7f71580cefa0 2023-10-30T13:21:24.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f7157fff640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f715815cd50 0x7f715813be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:24.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f71577fe640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71580cefa0 0x7f71581e3bc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f714800c500 tx=0x7f714800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:24.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f713ffff640 1 -- 172.21.15.148:0/2586411200 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148005cf0 con 0x7f71580cefa0 2023-10-30T13:21:24.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f713ffff640 1 -- 172.21.15.148:0/2586411200 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7148005e50 con 0x7f71580cefa0 2023-10-30T13:21:24.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f713ffff640 1 -- 172.21.15.148:0/2586411200 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71480072f0 con 0x7f71580cefa0 2023-10-30T13:21:24.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71581ef3e0 con 0x7f71580cefa0 2023-10-30T13:21:24.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.022+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71581ef680 con 0x7f71580cefa0 2023-10-30T13:21:24.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.026+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71581ef240 con 0x7f71580cefa0 2023-10-30T13:21:24.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.026+0000 7f713ffff640 1 -- 172.21.15.148:0/2586411200 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7148007450 con 0x7f71580cefa0 2023-10-30T13:21:24.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.026+0000 7f713ffff640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f71440664a0 0x7f7144068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:24.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.026+0000 7f713ffff640 1 -- 172.21.15.148:0/2586411200 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f7148083510 con 0x7f71580cefa0 2023-10-30T13:21:24.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.026+0000 7f7156ffd640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f71440664a0 0x7f7144068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:24.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.026+0000 7f7156ffd640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f71440664a0 0x7f7144068960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f714c0b2000 tx=0x7f714c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:24.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.030+0000 7f713ffff640 1 -- 172.21.15.148:0/2586411200 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71581ef240 con 0x7f71580cefa0 2023-10-30T13:21:24.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71581ef240 con 0x7f71580cefa0 2023-10-30T13:21:24.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f713ffff640 1 -- 172.21.15.148:0/2586411200 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6514 (secure 0 0 0) 0x7f71581ef240 con 0x7f71580cefa0 2023-10-30T13:21:24.243 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:24.243 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:11.563063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:24.245 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 54 2023-10-30T13:21:24.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f71440664a0 msgr2=0x7f7144068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:24.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f71440664a0 0x7f7144068960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f714c0b2000 tx=0x7f714c008040 comp rx=0 tx=0).stop 2023-10-30T13:21:24.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71580cefa0 msgr2=0x7f71581e3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:24.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71580cefa0 0x7f71581e3bc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f714800c500 tx=0x7f714800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 shutdown_connections 2023-10-30T13:21:24.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f71440664a0 0x7f7144068960 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f715815cd50 0x7f715813be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71581538f0 0x7f71581f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 --2- 172.21.15.148:0/2586411200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71580cefa0 0x7f71581e3bc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:24.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 >> 172.21.15.148:0/2586411200 conn(0x7f715806a640 msgr2=0x7f71580c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:24.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 shutdown_connections 2023-10-30T13:21:24.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:24.242+0000 7f715cc17640 1 -- 172.21.15.148:0/2586411200 wait complete. 2023-10-30T13:21:24.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:21:24.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/107' 2023-10-30T13:21:24.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/19' 2023-10-30T13:21:24.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/15' 2023-10-30T13:21:24.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/20' 2023-10-30T13:21:24.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/12' 2023-10-30T13:21:24.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/10' 2023-10-30T13:21:24.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/11' 2023-10-30T13:21:24.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/16' 2023-10-30T13:21:24.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/14' 2023-10-30T13:21:24.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/18' 2023-10-30T13:21:24.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/13' 2023-10-30T13:21:24.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/178/17' 2023-10-30T13:21:24.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/178' 2023-10-30T13:21:24.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/19' 2023-10-30T13:21:24.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/15' 2023-10-30T13:21:24.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/20' 2023-10-30T13:21:24.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/12' 2023-10-30T13:21:24.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/10' 2023-10-30T13:21:24.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/11' 2023-10-30T13:21:24.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/16' 2023-10-30T13:21:24.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/14' 2023-10-30T13:21:24.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/18' 2023-10-30T13:21:24.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/13' 2023-10-30T13:21:24.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/156/17' 2023-10-30T13:21:24.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/156' 2023-10-30T13:21:24.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/19' 2023-10-30T13:21:24.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/15' 2023-10-30T13:21:24.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/20' 2023-10-30T13:21:24.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/12' 2023-10-30T13:21:24.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/10' 2023-10-30T13:21:24.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/11' 2023-10-30T13:21:24.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/16' 2023-10-30T13:21:24.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/14' 2023-10-30T13:21:24.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/18' 2023-10-30T13:21:24.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/13' 2023-10-30T13:21:24.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/131/17' 2023-10-30T13:21:24.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/131' 2023-10-30T13:21:24.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/19' 2023-10-30T13:21:24.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/15' 2023-10-30T13:21:24.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/20' 2023-10-30T13:21:24.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/12' 2023-10-30T13:21:24.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/10' 2023-10-30T13:21:24.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/11' 2023-10-30T13:21:24.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/16' 2023-10-30T13:21:24.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/14' 2023-10-30T13:21:24.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/18' 2023-10-30T13:21:24.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/13' 2023-10-30T13:21:24.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/123/17' 2023-10-30T13:21:24.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/123' 2023-10-30T13:21:24.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/19' 2023-10-30T13:21:24.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/15' 2023-10-30T13:21:24.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/20' 2023-10-30T13:21:24.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/12' 2023-10-30T13:21:24.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/10' 2023-10-30T13:21:24.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/11' 2023-10-30T13:21:24.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/16' 2023-10-30T13:21:24.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/14' 2023-10-30T13:21:24.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/18' 2023-10-30T13:21:24.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/13' 2023-10-30T13:21:24.443 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/160/17' 2023-10-30T13:21:24.443 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/160' 2023-10-30T13:21:24.443 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/19' 2023-10-30T13:21:24.443 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/15' 2023-10-30T13:21:24.443 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/20' 2023-10-30T13:21:24.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/12' 2023-10-30T13:21:24.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/10' 2023-10-30T13:21:24.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/11' 2023-10-30T13:21:24.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/16' 2023-10-30T13:21:24.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/14' 2023-10-30T13:21:24.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/18' 2023-10-30T13:21:24.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/13' 2023-10-30T13:21:24.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/165/17' 2023-10-30T13:21:24.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/165' 2023-10-30T13:21:24.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/19' 2023-10-30T13:21:24.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/15' 2023-10-30T13:21:24.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/20' 2023-10-30T13:21:24.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/12' 2023-10-30T13:21:24.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/10' 2023-10-30T13:21:24.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/11' 2023-10-30T13:21:24.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/16' 2023-10-30T13:21:24.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/14' 2023-10-30T13:21:24.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/18' 2023-10-30T13:21:24.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/13' 2023-10-30T13:21:24.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/101/17' 2023-10-30T13:21:24.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/101' 2023-10-30T13:21:24.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/19' 2023-10-30T13:21:24.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/15' 2023-10-30T13:21:24.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/20' 2023-10-30T13:21:24.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/12' 2023-10-30T13:21:24.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/10' 2023-10-30T13:21:24.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/11' 2023-10-30T13:21:24.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/16' 2023-10-30T13:21:24.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/14' 2023-10-30T13:21:24.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/18' 2023-10-30T13:21:24.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/13' 2023-10-30T13:21:24.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/148/17' 2023-10-30T13:21:24.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/148' 2023-10-30T13:21:24.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/19' 2023-10-30T13:21:24.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/15' 2023-10-30T13:21:24.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/20' 2023-10-30T13:21:24.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/12' 2023-10-30T13:21:24.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/10' 2023-10-30T13:21:24.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/11' 2023-10-30T13:21:24.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/16' 2023-10-30T13:21:24.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/14' 2023-10-30T13:21:24.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/18' 2023-10-30T13:21:24.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/13' 2023-10-30T13:21:24.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/168/17' 2023-10-30T13:21:24.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/168' 2023-10-30T13:21:24.974 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:21:24.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:21:26.266 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:26.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff8838640 1 Processor -- start 2023-10-30T13:21:26.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff8838640 1 -- start start 2023-10-30T13:21:26.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff8838640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7ff41538d0 0x7f7ff415cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:26.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff8838640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff40c6790 0x7f7ff415a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:26.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff8838640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7ff40cb800 0x7f7ff40cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:26.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff8838640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff40c7430 con 0x7f7ff41538d0 2023-10-30T13:21:26.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff8838640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff406c1e0 con 0x7f7ff40cb800 2023-10-30T13:21:26.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff8838640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff4159d20 con 0x7f7ff40c6790 2023-10-30T13:21:26.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff37fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7ff41538d0 0x7f7ff415cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:26.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff37fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7ff41538d0 0x7f7ff415cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41254/0 (socket says 172.21.15.148:41254) 2023-10-30T13:21:26.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff37fe640 1 -- 172.21.15.148:0/2621633715 learned_addr learned my addr 172.21.15.148:0/2621633715 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:26.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff2ffd640 1 --2- 172.21.15.148:0/2621633715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff40c6790 0x7f7ff415a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:26.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.406+0000 7f7ff3fff640 1 --2- 172.21.15.148:0/2621633715 >> v2:172.21.15.150:3300/0 conn(0x7f7ff40cb800 0x7f7ff40cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:26.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff37fe640 1 -- 172.21.15.148:0/2621633715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff40c6790 msgr2=0x7f7ff415a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:26.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff37fe640 1 --2- 172.21.15.148:0/2621633715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff40c6790 0x7f7ff415a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff37fe640 1 -- 172.21.15.148:0/2621633715 >> v2:172.21.15.150:3300/0 conn(0x7f7ff40cb800 msgr2=0x7f7ff40cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:26.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff37fe640 1 --2- 172.21.15.148:0/2621633715 >> v2:172.21.15.150:3300/0 conn(0x7f7ff40cb800 0x7f7ff40cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff37fe640 1 -- 172.21.15.148:0/2621633715 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff406c410 con 0x7f7ff41538d0 2023-10-30T13:21:26.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff2ffd640 1 --2- 172.21.15.148:0/2621633715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff40c6790 0x7f7ff415a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:26.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff3fff640 1 --2- 172.21.15.148:0/2621633715 >> v2:172.21.15.150:3300/0 conn(0x7f7ff40cb800 0x7f7ff40cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:26.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff37fe640 1 --2- 172.21.15.148:0/2621633715 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff41538d0 0x7f7ff415cc70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40cc450 tx=0x7f7fe401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee83512a2492662 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:26.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff27fc640 1 -- 172.21.15.148:0/2621633715 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40a89b0 con 0x7f7ff41538d0 2023-10-30T13:21:26.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff27fc640 1 -- 172.21.15.148:0/2621633715 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe40a8b10 con 0x7f7ff41538d0 2023-10-30T13:21:26.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff27fc640 1 -- 172.21.15.148:0/2621633715 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40a8ea0 con 0x7f7ff41538d0 2023-10-30T13:21:26.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff8838640 1 -- 172.21.15.148:0/2621633715 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff41538d0 msgr2=0x7f7ff415cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:26.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff8838640 1 --2- 172.21.15.148:0/2621633715 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff41538d0 0x7f7ff415cc70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40cc450 tx=0x7f7fe401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff8838640 1 -- 172.21.15.148:0/2621633715 shutdown_connections 2023-10-30T13:21:26.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff8838640 1 --2- 172.21.15.148:0/2621633715 >> v2:172.21.15.150:3300/0 conn(0x7f7ff40cb800 0x7f7ff40cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff8838640 1 --2- 172.21.15.148:0/2621633715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff40c6790 0x7f7ff415a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff8838640 1 --2- 172.21.15.148:0/2621633715 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff41538d0 0x7f7ff415cc70 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff8838640 1 -- 172.21.15.148:0/2621633715 >> 172.21.15.148:0/2621633715 conn(0x7f7ff406a290 msgr2=0x7f7ff406a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:26.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.410+0000 7f7ff8838640 1 -- 172.21.15.148:0/2621633715 shutdown_connections 2023-10-30T13:21:26.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff8838640 1 -- 172.21.15.148:0/2621633715 wait complete. 2023-10-30T13:21:26.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff8838640 1 Processor -- start 2023-10-30T13:21:26.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff8838640 1 -- start start 2023-10-30T13:21:26.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff8838640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff40c6790 0x7f7ff415be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:26.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff8838640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff40cb800 0x7f7ff4154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:26.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff8838640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff41538d0 0x7f7ff4155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:26.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff8838640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff4154780 con 0x7f7ff40cb800 2023-10-30T13:21:26.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff8838640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff4159510 con 0x7f7ff40c6790 2023-10-30T13:21:26.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff8838640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff406dee0 con 0x7f7ff41538d0 2023-10-30T13:21:26.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff40cb800 0x7f7ff4154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:26.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff40cb800 0x7f7ff4154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41270/0 (socket says 172.21.15.148:41270) 2023-10-30T13:21:26.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff2ffd640 1 -- 172.21.15.148:0/3362652483 learned_addr learned my addr 172.21.15.148:0/3362652483 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:26.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff37fe640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff40c6790 0x7f7ff415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:26.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff3fff640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff41538d0 0x7f7ff4155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:26.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff2ffd640 1 -- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff41538d0 msgr2=0x7f7ff4155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:26.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff2ffd640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff41538d0 0x7f7ff4155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff2ffd640 1 -- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff40c6790 msgr2=0x7f7ff415be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:26.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff2ffd640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff40c6790 0x7f7ff415be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff2ffd640 1 -- 172.21.15.148:0/3362652483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe40094d0 con 0x7f7ff40cb800 2023-10-30T13:21:26.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff3fff640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff41538d0 0x7f7ff4155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:26.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.414+0000 7f7ff2ffd640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff40cb800 0x7f7ff4154bd0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800d800 tx=0x7f7fe800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:26.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7fdbfff640 1 -- 172.21.15.148:0/3362652483 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8004030 con 0x7f7ff40cb800 2023-10-30T13:21:26.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7fdbfff640 1 -- 172.21.15.148:0/3362652483 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe8004190 con 0x7f7ff40cb800 2023-10-30T13:21:26.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff41597b0 con 0x7f7ff40cb800 2023-10-30T13:21:26.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7fdbfff640 1 -- 172.21.15.148:0/3362652483 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8004520 con 0x7f7ff40cb800 2023-10-30T13:21:26.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff41ef520 con 0x7f7ff40cb800 2023-10-30T13:21:26.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff41ef0a0 con 0x7f7ff40cb800 2023-10-30T13:21:26.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7fdbfff640 1 -- 172.21.15.148:0/3362652483 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fe80096a0 con 0x7f7ff40cb800 2023-10-30T13:21:26.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7fdbfff640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fe00664a0 0x7f7fe0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:26.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7fdbfff640 1 -- 172.21.15.148:0/3362652483 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f7fe80842f0 con 0x7f7ff40cb800 2023-10-30T13:21:26.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.418+0000 7f7ff37fe640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fe00664a0 0x7f7fe0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:26.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.422+0000 7f7ff37fe640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fe00664a0 0x7f7fe0068960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40b2000 tx=0x7f7fe4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:26.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.422+0000 7f7fdbfff640 1 -- 172.21.15.148:0/3362652483 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ff41ef0a0 con 0x7f7ff40cb800 2023-10-30T13:21:26.429 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/19' 2023-10-30T13:21:26.429 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/15' 2023-10-30T13:21:26.429 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/20' 2023-10-30T13:21:26.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/12' 2023-10-30T13:21:26.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/10' 2023-10-30T13:21:26.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/11' 2023-10-30T13:21:26.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/16' 2023-10-30T13:21:26.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/14' 2023-10-30T13:21:26.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/18' 2023-10-30T13:21:26.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/13' 2023-10-30T13:21:26.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/179/17' 2023-10-30T13:21:26.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/179' 2023-10-30T13:21:26.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/19' 2023-10-30T13:21:26.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/15' 2023-10-30T13:21:26.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/20' 2023-10-30T13:21:26.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/12' 2023-10-30T13:21:26.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/10' 2023-10-30T13:21:26.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/11' 2023-10-30T13:21:26.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/16' 2023-10-30T13:21:26.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/14' 2023-10-30T13:21:26.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/18' 2023-10-30T13:21:26.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/13' 2023-10-30T13:21:26.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/132/17' 2023-10-30T13:21:26.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/132' 2023-10-30T13:21:26.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/19' 2023-10-30T13:21:26.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/15' 2023-10-30T13:21:26.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/20' 2023-10-30T13:21:26.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/12' 2023-10-30T13:21:26.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/10' 2023-10-30T13:21:26.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/11' 2023-10-30T13:21:26.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/16' 2023-10-30T13:21:26.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/14' 2023-10-30T13:21:26.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/18' 2023-10-30T13:21:26.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/13' 2023-10-30T13:21:26.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/199/17' 2023-10-30T13:21:26.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/199' 2023-10-30T13:21:26.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/19' 2023-10-30T13:21:26.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/15' 2023-10-30T13:21:26.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/20' 2023-10-30T13:21:26.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/12' 2023-10-30T13:21:26.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/10' 2023-10-30T13:21:26.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/11' 2023-10-30T13:21:26.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/16' 2023-10-30T13:21:26.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/14' 2023-10-30T13:21:26.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/18' 2023-10-30T13:21:26.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/13' 2023-10-30T13:21:26.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/163/17' 2023-10-30T13:21:26.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/163' 2023-10-30T13:21:26.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/19' 2023-10-30T13:21:26.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/15' 2023-10-30T13:21:26.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/20' 2023-10-30T13:21:26.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/12' 2023-10-30T13:21:26.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/10' 2023-10-30T13:21:26.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/11' 2023-10-30T13:21:26.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/16' 2023-10-30T13:21:26.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/14' 2023-10-30T13:21:26.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/18' 2023-10-30T13:21:26.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/13' 2023-10-30T13:21:26.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/153/17' 2023-10-30T13:21:26.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/153' 2023-10-30T13:21:26.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/19' 2023-10-30T13:21:26.443 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/15' 2023-10-30T13:21:26.443 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/20' 2023-10-30T13:21:26.443 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/12' 2023-10-30T13:21:26.443 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/10' 2023-10-30T13:21:26.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/11' 2023-10-30T13:21:26.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/16' 2023-10-30T13:21:26.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/14' 2023-10-30T13:21:26.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/18' 2023-10-30T13:21:26.444 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/13' 2023-10-30T13:21:26.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/122/17' 2023-10-30T13:21:26.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/122' 2023-10-30T13:21:26.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/19' 2023-10-30T13:21:26.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/15' 2023-10-30T13:21:26.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/20' 2023-10-30T13:21:26.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/12' 2023-10-30T13:21:26.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/10' 2023-10-30T13:21:26.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/11' 2023-10-30T13:21:26.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/16' 2023-10-30T13:21:26.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/14' 2023-10-30T13:21:26.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/18' 2023-10-30T13:21:26.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/13' 2023-10-30T13:21:26.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/162/17' 2023-10-30T13:21:26.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/162' 2023-10-30T13:21:26.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/19' 2023-10-30T13:21:26.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/15' 2023-10-30T13:21:26.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/20' 2023-10-30T13:21:26.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/12' 2023-10-30T13:21:26.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/10' 2023-10-30T13:21:26.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/11' 2023-10-30T13:21:26.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/16' 2023-10-30T13:21:26.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/14' 2023-10-30T13:21:26.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/18' 2023-10-30T13:21:26.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/13' 2023-10-30T13:21:26.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/197/17' 2023-10-30T13:21:26.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/197' 2023-10-30T13:21:26.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/19' 2023-10-30T13:21:26.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/15' 2023-10-30T13:21:26.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/20' 2023-10-30T13:21:26.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/12' 2023-10-30T13:21:26.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/10' 2023-10-30T13:21:26.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/11' 2023-10-30T13:21:26.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/16' 2023-10-30T13:21:26.451 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/14' 2023-10-30T13:21:26.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/18' 2023-10-30T13:21:26.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/13' 2023-10-30T13:21:26.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/113/17' 2023-10-30T13:21:26.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/113' 2023-10-30T13:21:26.452 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/19' 2023-10-30T13:21:26.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.610+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ff41ef0a0 con 0x7f7ff40cb800 2023-10-30T13:21:26.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.610+0000 7f7fdbfff640 1 -- 172.21.15.148:0/3362652483 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6514 (secure 0 0 0) 0x7f7ff41ef0a0 con 0x7f7ff40cb800 2023-10-30T13:21:26.612 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:26.612 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2690362018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2690362018},{"type":"v1","addr":"172.21.15.148:6837","nonce":2690362018}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:11.563063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:26.612 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 54 2023-10-30T13:21:26.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fe00664a0 msgr2=0x7f7fe0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:26.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fe00664a0 0x7f7fe0068960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40b2000 tx=0x7f7fe4008040 comp rx=0 tx=0).stop 2023-10-30T13:21:26.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff40cb800 msgr2=0x7f7ff4154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:26.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff40cb800 0x7f7ff4154bd0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800d800 tx=0x7f7fe800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 shutdown_connections 2023-10-30T13:21:26.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fe00664a0 0x7f7fe0068960 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7ff40cb800 0x7f7ff4154bd0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ff41538d0 0x7f7ff4155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 --2- 172.21.15.148:0/3362652483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff40c6790 0x7f7ff415be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:26.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 >> 172.21.15.148:0/3362652483 conn(0x7f7ff406a290 msgr2=0x7f7ff40701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:26.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 shutdown_connections 2023-10-30T13:21:26.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:26.614+0000 7f7ff8838640 1 -- 172.21.15.148:0/3362652483 wait complete. 2023-10-30T13:21:26.633 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:21:27.079 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/15' 2023-10-30T13:21:27.080 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/20' 2023-10-30T13:21:27.080 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/12' 2023-10-30T13:21:27.080 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/10' 2023-10-30T13:21:27.080 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/11' 2023-10-30T13:21:27.081 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/16' 2023-10-30T13:21:27.081 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/14' 2023-10-30T13:21:27.081 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/18' 2023-10-30T13:21:27.082 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/13' 2023-10-30T13:21:27.082 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/152/17' 2023-10-30T13:21:27.082 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/152' 2023-10-30T13:21:27.082 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/19' 2023-10-30T13:21:27.083 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/15' 2023-10-30T13:21:27.083 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/20' 2023-10-30T13:21:27.083 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/12' 2023-10-30T13:21:27.083 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/10' 2023-10-30T13:21:27.083 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/11' 2023-10-30T13:21:27.084 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/16' 2023-10-30T13:21:27.084 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/14' 2023-10-30T13:21:27.084 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/18' 2023-10-30T13:21:27.084 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/13' 2023-10-30T13:21:27.084 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/121/17' 2023-10-30T13:21:27.084 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/121' 2023-10-30T13:21:27.085 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/19' 2023-10-30T13:21:27.085 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/15' 2023-10-30T13:21:27.085 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/20' 2023-10-30T13:21:27.085 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/12' 2023-10-30T13:21:27.085 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/10' 2023-10-30T13:21:27.086 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/11' 2023-10-30T13:21:27.086 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/16' 2023-10-30T13:21:27.086 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/14' 2023-10-30T13:21:27.086 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/18' 2023-10-30T13:21:27.086 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/13' 2023-10-30T13:21:27.087 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/118/17' 2023-10-30T13:21:27.087 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/118' 2023-10-30T13:21:27.087 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/19' 2023-10-30T13:21:27.087 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/15' 2023-10-30T13:21:27.087 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/20' 2023-10-30T13:21:27.087 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/12' 2023-10-30T13:21:27.088 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/10' 2023-10-30T13:21:27.088 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/11' 2023-10-30T13:21:27.088 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/16' 2023-10-30T13:21:27.088 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/14' 2023-10-30T13:21:27.088 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/18' 2023-10-30T13:21:27.088 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/13' 2023-10-30T13:21:27.089 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/196/17' 2023-10-30T13:21:27.089 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/196' 2023-10-30T13:21:27.089 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/19' 2023-10-30T13:21:27.089 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/15' 2023-10-30T13:21:27.089 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/20' 2023-10-30T13:21:27.090 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/12' 2023-10-30T13:21:27.090 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/10' 2023-10-30T13:21:27.090 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/11' 2023-10-30T13:21:27.090 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/16' 2023-10-30T13:21:27.090 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/14' 2023-10-30T13:21:27.091 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/18' 2023-10-30T13:21:27.091 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/13' 2023-10-30T13:21:27.091 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/124/17' 2023-10-30T13:21:27.091 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/124' 2023-10-30T13:21:27.091 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/19' 2023-10-30T13:21:27.091 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/15' 2023-10-30T13:21:27.092 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/20' 2023-10-30T13:21:27.092 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/12' 2023-10-30T13:21:27.092 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/10' 2023-10-30T13:21:27.092 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/11' 2023-10-30T13:21:27.092 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/16' 2023-10-30T13:21:27.093 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/14' 2023-10-30T13:21:27.093 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/18' 2023-10-30T13:21:27.093 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/13' 2023-10-30T13:21:27.093 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/188/17' 2023-10-30T13:21:27.093 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/188' 2023-10-30T13:21:27.094 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/19' 2023-10-30T13:21:27.094 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/15' 2023-10-30T13:21:27.094 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/20' 2023-10-30T13:21:27.094 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/12' 2023-10-30T13:21:27.094 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/10' 2023-10-30T13:21:27.095 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/11' 2023-10-30T13:21:27.095 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/16' 2023-10-30T13:21:27.095 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/14' 2023-10-30T13:21:27.095 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/18' 2023-10-30T13:21:27.095 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/13' 2023-10-30T13:21:27.095 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/145/17' 2023-10-30T13:21:27.096 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/145' 2023-10-30T13:21:27.096 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/19' 2023-10-30T13:21:27.096 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/15' 2023-10-30T13:21:27.096 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/20' 2023-10-30T13:21:27.096 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/12' 2023-10-30T13:21:27.097 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/10' 2023-10-30T13:21:27.097 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/11' 2023-10-30T13:21:27.097 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/16' 2023-10-30T13:21:27.097 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/14' 2023-10-30T13:21:27.097 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/18' 2023-10-30T13:21:27.098 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/13' 2023-10-30T13:21:27.098 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/130/17' 2023-10-30T13:21:27.098 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/130' 2023-10-30T13:21:27.098 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/19' 2023-10-30T13:21:27.098 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/15' 2023-10-30T13:21:27.098 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/20' 2023-10-30T13:21:27.099 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/12' 2023-10-30T13:21:27.099 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/10' 2023-10-30T13:21:27.099 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/11' 2023-10-30T13:21:27.099 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/16' 2023-10-30T13:21:27.099 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/14' 2023-10-30T13:21:27.100 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/18' 2023-10-30T13:21:27.100 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/13' 2023-10-30T13:21:27.100 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/193/17' 2023-10-30T13:21:27.100 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/193' 2023-10-30T13:21:27.100 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/19' 2023-10-30T13:21:27.101 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/15' 2023-10-30T13:21:28.425 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/20' 2023-10-30T13:21:28.425 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/12' 2023-10-30T13:21:28.425 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/10' 2023-10-30T13:21:28.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/11' 2023-10-30T13:21:28.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/16' 2023-10-30T13:21:28.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/14' 2023-10-30T13:21:28.426 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/18' 2023-10-30T13:21:28.427 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/13' 2023-10-30T13:21:28.427 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/175/17' 2023-10-30T13:21:28.427 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/175' 2023-10-30T13:21:28.427 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/19' 2023-10-30T13:21:28.427 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/15' 2023-10-30T13:21:28.428 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/20' 2023-10-30T13:21:28.428 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/12' 2023-10-30T13:21:28.428 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/10' 2023-10-30T13:21:28.428 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/11' 2023-10-30T13:21:28.428 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/16' 2023-10-30T13:21:28.429 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/14' 2023-10-30T13:21:28.429 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/18' 2023-10-30T13:21:28.429 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/13' 2023-10-30T13:21:28.429 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/181/17' 2023-10-30T13:21:28.429 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/181' 2023-10-30T13:21:28.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/19' 2023-10-30T13:21:28.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/15' 2023-10-30T13:21:28.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/20' 2023-10-30T13:21:28.430 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/12' 2023-10-30T13:21:28.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/10' 2023-10-30T13:21:28.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/11' 2023-10-30T13:21:28.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/16' 2023-10-30T13:21:28.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/14' 2023-10-30T13:21:28.431 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/18' 2023-10-30T13:21:28.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/13' 2023-10-30T13:21:28.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/127/17' 2023-10-30T13:21:28.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/127' 2023-10-30T13:21:28.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/19' 2023-10-30T13:21:28.432 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/15' 2023-10-30T13:21:28.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/20' 2023-10-30T13:21:28.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/12' 2023-10-30T13:21:28.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/10' 2023-10-30T13:21:28.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/11' 2023-10-30T13:21:28.433 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/16' 2023-10-30T13:21:28.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/14' 2023-10-30T13:21:28.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/18' 2023-10-30T13:21:28.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/13' 2023-10-30T13:21:28.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/108/17' 2023-10-30T13:21:28.434 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/108' 2023-10-30T13:21:28.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/19' 2023-10-30T13:21:28.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/15' 2023-10-30T13:21:28.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/20' 2023-10-30T13:21:28.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/12' 2023-10-30T13:21:28.435 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/10' 2023-10-30T13:21:28.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/11' 2023-10-30T13:21:28.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/16' 2023-10-30T13:21:28.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/14' 2023-10-30T13:21:28.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/18' 2023-10-30T13:21:28.436 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/13' 2023-10-30T13:21:28.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/183/17' 2023-10-30T13:21:28.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/183' 2023-10-30T13:21:28.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/19' 2023-10-30T13:21:28.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/15' 2023-10-30T13:21:28.437 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/20' 2023-10-30T13:21:28.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/12' 2023-10-30T13:21:28.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/10' 2023-10-30T13:21:28.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/11' 2023-10-30T13:21:28.438 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/16' 2023-10-30T13:21:28.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/14' 2023-10-30T13:21:28.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/18' 2023-10-30T13:21:28.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/13' 2023-10-30T13:21:28.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/177/17' 2023-10-30T13:21:28.439 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/177' 2023-10-30T13:21:28.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/19' 2023-10-30T13:21:28.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/15' 2023-10-30T13:21:28.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/20' 2023-10-30T13:21:28.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/12' 2023-10-30T13:21:28.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/10' 2023-10-30T13:21:28.440 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/11' 2023-10-30T13:21:28.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/16' 2023-10-30T13:21:28.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/14' 2023-10-30T13:21:28.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/18' 2023-10-30T13:21:28.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/13' 2023-10-30T13:21:28.441 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/169/17' 2023-10-30T13:21:28.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/169' 2023-10-30T13:21:28.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/19' 2023-10-30T13:21:28.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/15' 2023-10-30T13:21:28.442 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/20' 2023-10-30T13:21:28.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/12' 2023-10-30T13:21:28.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/10' 2023-10-30T13:21:28.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/11' 2023-10-30T13:21:28.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/16' 2023-10-30T13:21:28.445 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/14' 2023-10-30T13:21:28.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/18' 2023-10-30T13:21:28.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/13' 2023-10-30T13:21:28.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/154/17' 2023-10-30T13:21:28.446 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/154' 2023-10-30T13:21:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/19' 2023-10-30T13:21:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/15' 2023-10-30T13:21:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/20' 2023-10-30T13:21:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/12' 2023-10-30T13:21:28.447 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/10' 2023-10-30T13:21:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/11' 2023-10-30T13:21:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/16' 2023-10-30T13:21:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/14' 2023-10-30T13:21:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/18' 2023-10-30T13:21:28.448 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/13' 2023-10-30T13:21:28.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/171/17' 2023-10-30T13:21:28.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/171' 2023-10-30T13:21:28.449 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/19' 2023-10-30T13:21:28.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/15' 2023-10-30T13:21:28.450 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/20' 2023-10-30T13:21:28.634 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.782+0000 7f630e468640 1 Processor -- start 2023-10-30T13:21:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.782+0000 7f630e468640 1 -- start start 2023-10-30T13:21:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f630e468640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f63080c05d0 0x7f630813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f630e468640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c6f60 0x7f630813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f630e468640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080c0a50 con 0x7f63080c6b80 2023-10-30T13:21:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f630e468640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630806f6e0 con 0x7f63080c05d0 2023-10-30T13:21:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f630e468640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630806f9c0 con 0x7f63080c6f60 2023-10-30T13:21:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6307fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f63080c6b80 0x7f63080bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37018/0 (socket says 172.21.15.148:37018) 2023-10-30T13:21:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6307fff640 1 -- 172.21.15.148:0/283614173 learned_addr learned my addr 172.21.15.148:0/283614173 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:21:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f630ca05640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c6f60 0x7f630813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f63077fe640 1 --2- 172.21.15.148:0/283614173 >> v2:172.21.15.150:3300/0 conn(0x7f63080c05d0 0x7f630813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500316040 0 0) 0x7f63080c0a50 con 0x7f63080c6b80 2023-10-30T13:21:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f0003530 con 0x7f63080c6b80 2023-10-30T13:21:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037616352 0 0) 0x7f62f0003530 con 0x7f63080c6b80 2023-10-30T13:21:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c6f60 msgr2=0x7f630813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c6f60 0x7f630813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 >> v2:172.21.15.150:3300/0 conn(0x7f63080c05d0 msgr2=0x7f630813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 --2- 172.21.15.148:0/283614173 >> v2:172.21.15.150:3300/0 conn(0x7f63080c05d0 0x7f630813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630806fca0 con 0x7f63080c6b80 2023-10-30T13:21:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f62f8001680 con 0x7f63080c6b80 2023-10-30T13:21:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 >> v1:172.21.15.148:6789/0 conn(0x7f63080c6b80 legacy=0x7f63080bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f62f0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c6f60 0x7f62f00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f0004d10 0x7f62f0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630806f6e0 con 0x7f62f0003980 2023-10-30T13:21:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630806f9c0 con 0x7f62f0004d10 2023-10-30T13:21:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f000b990 con 0x7f63080c6f60 2023-10-30T13:21:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6307fff640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f62f0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f630ca05640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f0004d10 0x7f62f0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f63077fe640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c6f60 0x7f62f00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6307fff640 1 -- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c6f60 msgr2=0x7f62f00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6307fff640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63080c6f60 0x7f62f00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6307fff640 1 -- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f0004d10 msgr2=0x7f62f0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:28.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6307fff640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62f0004d10 0x7f62f0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.786+0000 7f6307fff640 1 -- 172.21.15.148:0/283614173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f0003530 con 0x7f62f0003980 2023-10-30T13:21:28.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f6307fff640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f62f0003da0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f62f800c240 tx=0x7f62f801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa091da850d2462a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:28.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80058e0 con 0x7f62f0003980 2023-10-30T13:21:28.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f8005a40 con 0x7f62f0003980 2023-10-30T13:21:28.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f6306ffd640 1 -- 172.21.15.148:0/283614173 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8005e10 con 0x7f62f0003980 2023-10-30T13:21:28.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f630e468640 1 -- 172.21.15.148:0/283614173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 msgr2=0x7f62f0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:28.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f630e468640 1 --2- 172.21.15.148:0/283614173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f62f0003da0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f62f800c240 tx=0x7f62f801cc60 comp rx=0 tx=0).stop 2023-10-30T13:21:28.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f630ca05640 1 -- 172.21.15.148:0/283614173 reap_dead start 2023-10-30T13:21:28.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f630e468640 1 -- 172.21.15.148:0/283614173 shutdown_connections 2023-10-30T13:21:28.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f630e468640 1 -- 172.21.15.148:0/283614173 >> 172.21.15.148:0/283614173 conn(0x7f63080c7910 msgr2=0x7f63080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:28.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f630e468640 1 -- 172.21.15.148:0/283614173 shutdown_connections 2023-10-30T13:21:28.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f630e468640 1 -- 172.21.15.148:0/283614173 wait complete. 2023-10-30T13:21:28.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f630e468640 1 Processor -- start 2023-10-30T13:21:28.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.790+0000 7f630e468640 1 -- start start 2023-10-30T13:21:28.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630e468640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f63081f5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:28.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630e468640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6308159630 0x7f63081524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:28.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630e468640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63081339b0 0x7f6308133e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:28.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630e468640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308151d70 con 0x7f62f0003980 2023-10-30T13:21:28.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630e468640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308152050 con 0x7f6308159630 2023-10-30T13:21:28.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630e468640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6308153900 con 0x7f63081339b0 2023-10-30T13:21:28.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6307fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f63081f5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:28.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6307fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f63081f5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41286/0 (socket says 172.21.15.148:41286) 2023-10-30T13:21:28.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6307fff640 1 -- 172.21.15.148:0/1601751587 learned_addr learned my addr 172.21.15.148:0/1601751587 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:28.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f63077fe640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6308159630 0x7f63081524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:28.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630ca05640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63081339b0 0x7f6308133e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:28.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6307fff640 1 -- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63081339b0 msgr2=0x7f6308133e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:28.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6307fff640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63081339b0 0x7f6308133e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6307fff640 1 -- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6308159630 msgr2=0x7f63081524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:28.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6307fff640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6308159630 0x7f63081524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6307fff640 1 -- 172.21.15.148:0/1601751587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6308153bc0 con 0x7f62f0003980 2023-10-30T13:21:28.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f63077fe640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6308159630 0x7f63081524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:28.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6307fff640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f63081f5d20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f62f80ab040 tx=0x7f62f0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:28.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6304ff9640 1 -- 172.21.15.148:0/1601751587 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80a5a50 con 0x7f62f0003980 2023-10-30T13:21:28.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6304ff9640 1 -- 172.21.15.148:0/1601751587 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f80a5bb0 con 0x7f62f0003980 2023-10-30T13:21:28.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6304ff9640 1 -- 172.21.15.148:0/1601751587 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80047f0 con 0x7f62f0003980 2023-10-30T13:21:28.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6308131c50 con 0x7f62f0003980 2023-10-30T13:21:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6308132210 con 0x7f62f0003980 2023-10-30T13:21:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308132400 con 0x7f62f0003980 2023-10-30T13:21:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.794+0000 7f6304ff9640 1 -- 172.21.15.148:0/1601751587 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f62f8004950 con 0x7f62f0003980 2023-10-30T13:21:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.798+0000 7f6304ff9640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62f40664d0 0x7f62f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:28.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.798+0000 7f6304ff9640 1 -- 172.21.15.148:0/1601751587 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f62f811d5e0 con 0x7f62f0003980 2023-10-30T13:21:28.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.798+0000 7f63077fe640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62f40664d0 0x7f62f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:28.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.798+0000 7f63077fe640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62f40664d0 0x7f62f4068990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f63080b0750 tx=0x7f62fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:28.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.802+0000 7f6304ff9640 1 -- 172.21.15.148:0/1601751587 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f62f80ea7e0 con 0x7f62f0003980 2023-10-30T13:21:28.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f630804e170 con 0x7f62f0003980 2023-10-30T13:21:28.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f6304ff9640 1 -- 172.21.15.148:0/1601751587 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5718 (secure 0 0 0) 0x7f62f80ede00 con 0x7f62f0003980 2023-10-30T13:21:28.983 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:28.983 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:27.737025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:28.984 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 56 2023-10-30T13:21:28.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62f40664d0 msgr2=0x7f62f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:28.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62f40664d0 0x7f62f4068990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f63080b0750 tx=0x7f62fc008040 comp rx=0 tx=0).stop 2023-10-30T13:21:28.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 msgr2=0x7f63081f5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:28.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f63081f5d20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f62f80ab040 tx=0x7f62f0005800 comp rx=0 tx=0).stop 2023-10-30T13:21:28.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 shutdown_connections 2023-10-30T13:21:28.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62f40664d0 0x7f62f4068990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f63081339b0 0x7f6308133e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6308159630 0x7f63081524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 --2- 172.21.15.148:0/1601751587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62f0003980 0x7f63081f5d20 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:28.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 >> 172.21.15.148:0/1601751587 conn(0x7f63080c7910 msgr2=0x7f63080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:28.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 shutdown_connections 2023-10-30T13:21:28.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:28.982+0000 7f630e468640 1 -- 172.21.15.148:0/1601751587 wait complete. 2023-10-30T13:21:29.003 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:21:29.004 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:21:29.004 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:29.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83ab074640 1 Processor -- start 2023-10-30T13:21:29.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83ab074640 1 -- start start 2023-10-30T13:21:29.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83ab074640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f83a41538f0 0x7f83a4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:29.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83ab074640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a415cd50 0x7f83a40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:29.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83ab074640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a415d300 con 0x7f83a40cefa0 2023-10-30T13:21:29.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83ab074640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a415d5e0 con 0x7f83a41538f0 2023-10-30T13:21:29.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83ab074640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a415d8c0 con 0x7f83a415cd50 2023-10-30T13:21:29.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83aa072640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f83a41538f0 0x7f83a4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:29.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83aa072640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f83a41538f0 0x7f83a4159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:46678/0 (socket says 172.21.15.148:46678) 2023-10-30T13:21:29.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83aa072640 1 -- 172.21.15.148:0/1193989396 learned_addr learned my addr 172.21.15.148:0/1193989396 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:29.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83a9871640 1 --2- 172.21.15.148:0/1193989396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a415cd50 0x7f83a40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:29.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83a9871640 1 -- 172.21.15.148:0/1193989396 >> v2:172.21.15.150:3300/0 conn(0x7f83a41538f0 msgr2=0x7f83a4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:29.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83a9871640 1 --2- 172.21.15.148:0/1193989396 >> v2:172.21.15.150:3300/0 conn(0x7f83a41538f0 0x7f83a4159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83a9871640 1 -- 172.21.15.148:0/1193989396 >> v1:172.21.15.150:6789/0 conn(0x7f83a40cefa0 legacy=0x7f83a40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:29.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.150+0000 7f83a9871640 1 -- 172.21.15.148:0/1193989396 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a415dba0 con 0x7f83a415cd50 2023-10-30T13:21:29.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.158+0000 7f83a9871640 1 --2- 172.21.15.148:0/1193989396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a415cd50 0x7f83a40c9f90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f83940097c0 tx=0x7f839401ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2107b8807fa5bbd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:29.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83a9070640 1 -- 172.21.15.148:0/1193989396 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83940a88f0 con 0x7f83a415cd50 2023-10-30T13:21:29.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83a9070640 1 -- 172.21.15.148:0/1193989396 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83940a8a50 con 0x7f83a415cd50 2023-10-30T13:21:29.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83a9070640 1 -- 172.21.15.148:0/1193989396 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83940a8de0 con 0x7f83a415cd50 2023-10-30T13:21:29.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83ab074640 1 -- 172.21.15.148:0/1193989396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a415cd50 msgr2=0x7f83a40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:29.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83ab074640 1 --2- 172.21.15.148:0/1193989396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a415cd50 0x7f83a40c9f90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f83940097c0 tx=0x7f839401ec30 comp rx=0 tx=0).stop 2023-10-30T13:21:29.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83ab074640 1 -- 172.21.15.148:0/1193989396 shutdown_connections 2023-10-30T13:21:29.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83ab074640 1 --2- 172.21.15.148:0/1193989396 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a415cd50 0x7f83a40c9f90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83ab074640 1 --2- 172.21.15.148:0/1193989396 >> v2:172.21.15.150:3300/0 conn(0x7f83a41538f0 0x7f83a4159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83ab074640 1 -- 172.21.15.148:0/1193989396 >> 172.21.15.148:0/1193989396 conn(0x7f83a406a640 msgr2=0x7f83a4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:29.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83ab074640 1 -- 172.21.15.148:0/1193989396 shutdown_connections 2023-10-30T13:21:29.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83ab074640 1 -- 172.21.15.148:0/1193989396 wait complete. 2023-10-30T13:21:29.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.162+0000 7f83ab074640 1 Processor -- start 2023-10-30T13:21:29.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83ab074640 1 -- start start 2023-10-30T13:21:29.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83ab074640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a40cefa0 0x7f83a4151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:29.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83ab074640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83a41538f0 0x7f83a4147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:29.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83ab074640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83a415cd50 0x7f83a40b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:29.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83ab074640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a40b3c00 con 0x7f83a415cd50 2023-10-30T13:21:29.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83ab074640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a415cb00 con 0x7f83a41538f0 2023-10-30T13:21:29.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83ab074640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a40b2ff0 con 0x7f83a40cefa0 2023-10-30T13:21:29.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83aa873640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83a415cd50 0x7f83a40b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:29.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83aa873640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83a415cd50 0x7f83a40b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41296/0 (socket says 172.21.15.148:41296) 2023-10-30T13:21:29.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83aa873640 1 -- 172.21.15.148:0/1467930944 learned_addr learned my addr 172.21.15.148:0/1467930944 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:29.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83a9871640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83a41538f0 0x7f83a4147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:29.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83aa873640 1 -- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a40cefa0 msgr2=0x7f83a4151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:29.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83aa873640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a40cefa0 0x7f83a4151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83aa873640 1 -- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83a41538f0 msgr2=0x7f83a4147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:29.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83aa873640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83a41538f0 0x7f83a4147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83aa873640 1 -- 172.21.15.148:0/1467930944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8394009420 con 0x7f83a415cd50 2023-10-30T13:21:29.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.166+0000 7f83aa873640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83a415cd50 0x7f83a40b4090 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c780 tx=0x7f839c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:29.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.170+0000 7f839affd640 1 -- 172.21.15.148:0/1467930944 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c003d90 con 0x7f83a415cd50 2023-10-30T13:21:29.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.170+0000 7f839affd640 1 -- 172.21.15.148:0/1467930944 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c00ce90 con 0x7f83a415cd50 2023-10-30T13:21:29.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.170+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a40b32d0 con 0x7f83a415cd50 2023-10-30T13:21:29.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.170+0000 7f839affd640 1 -- 172.21.15.148:0/1467930944 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c0052b0 con 0x7f83a415cd50 2023-10-30T13:21:29.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.170+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a40b3890 con 0x7f83a415cd50 2023-10-30T13:21:29.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.170+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a404e170 con 0x7f83a415cd50 2023-10-30T13:21:29.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.174+0000 7f839affd640 1 -- 172.21.15.148:0/1467930944 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f839c005410 con 0x7f83a415cd50 2023-10-30T13:21:29.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.174+0000 7f839affd640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8388066520 0x7f83880689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:29.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.174+0000 7f839affd640 1 -- 172.21.15.148:0/1467930944 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f839c083cd0 con 0x7f83a415cd50 2023-10-30T13:21:29.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.174+0000 7f83aa072640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8388066520 0x7f83880689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:29.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.174+0000 7f83aa072640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8388066520 0x7f83880689e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f83a00094a0 tx=0x7f83a00093f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:29.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.178+0000 7f839affd640 1 -- 172.21.15.148:0/1467930944 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f839c050ff0 con 0x7f83a415cd50 2023-10-30T13:21:29.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83a40b3d40 con 0x7f83a415cd50 2023-10-30T13:21:29.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f839affd640 1 -- 172.21.15.148:0/1467930944 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5718 (secure 0 0 0) 0x7f839c054610 con 0x7f83a415cd50 2023-10-30T13:21:29.359 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:29.359 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:27.737025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:29.361 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 56 2023-10-30T13:21:29.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8388066520 msgr2=0x7f83880689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:29.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8388066520 0x7f83880689e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f83a00094a0 tx=0x7f83a00093f0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83a415cd50 msgr2=0x7f83a40b4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:29.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83a415cd50 0x7f83a40b4090 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c780 tx=0x7f839c00cc40 comp rx=0 tx=0).stop 2023-10-30T13:21:29.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 shutdown_connections 2023-10-30T13:21:29.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8388066520 0x7f83880689e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83a415cd50 0x7f83a40b4090 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83a41538f0 0x7f83a4147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 --2- 172.21.15.148:0/1467930944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f83a40cefa0 0x7f83a4151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:29.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 >> 172.21.15.148:0/1467930944 conn(0x7f83a406a640 msgr2=0x7f83a4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:29.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 shutdown_connections 2023-10-30T13:21:29.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:29.358+0000 7f83ab074640 1 -- 172.21.15.148:0/1467930944 wait complete. 2023-10-30T13:21:29.379 INFO:tasks.mds_thrash.fs.[cephfs]:cluster has 1 actives (max_mds is 1), no MDS can replace rank 2 2023-10-30T13:21:29.380 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.c 2023-10-30T13:21:29.841 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/12' 2023-10-30T13:21:29.841 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/10' 2023-10-30T13:21:29.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/11' 2023-10-30T13:21:29.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/16' 2023-10-30T13:21:29.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/14' 2023-10-30T13:21:29.842 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/18' 2023-10-30T13:21:29.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/13' 2023-10-30T13:21:29.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/139/17' 2023-10-30T13:21:29.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/139' 2023-10-30T13:21:29.843 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/19' 2023-10-30T13:21:29.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/15' 2023-10-30T13:21:29.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/20' 2023-10-30T13:21:29.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/12' 2023-10-30T13:21:29.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/10' 2023-10-30T13:21:29.844 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/11' 2023-10-30T13:21:29.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/16' 2023-10-30T13:21:29.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/14' 2023-10-30T13:21:29.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/18' 2023-10-30T13:21:29.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/13' 2023-10-30T13:21:29.845 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/102/17' 2023-10-30T13:21:29.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/102' 2023-10-30T13:21:29.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/19' 2023-10-30T13:21:29.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/15' 2023-10-30T13:21:29.846 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/20' 2023-10-30T13:21:29.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/12' 2023-10-30T13:21:29.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/10' 2023-10-30T13:21:29.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/11' 2023-10-30T13:21:29.847 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/16' 2023-10-30T13:21:29.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/14' 2023-10-30T13:21:29.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/18' 2023-10-30T13:21:29.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/13' 2023-10-30T13:21:29.848 INFO:tasks.workunit.client.0.smithi148.stdout:removed './copyofsnap1/test1/129/17' 2023-10-30T13:21:29.849 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1/129' 2023-10-30T13:21:29.849 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1/test1' 2023-10-30T13:21:29.849 INFO:tasks.workunit.client.0.smithi148.stdout:removed directory './copyofsnap1' 2023-10-30T13:21:29.849 INFO:tasks.workunit.client.0.smithi148.stdout:OK 2023-10-30T13:21:29.851 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:21:29.851 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:21:29.885 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-10-30T13:21:29.885 DEBUG:teuthology.orchestra.run.smithi148:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-10-30T13:21:29.945 INFO:tasks.workunit.client.0.smithi148.stderr:+ set -e 2023-10-30T13:21:29.946 INFO:tasks.workunit.client.0.smithi148.stderr:+ touch foo 2023-10-30T13:21:29.949 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod +x foo 2023-10-30T13:21:29.950 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/s 2023-10-30T13:21:30.076 INFO:tasks.workunit.client.0.smithi148.stderr:+ find .snap/s/foo -executable 2023-10-30T13:21:30.076 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep foo 2023-10-30T13:21:30.080 INFO:tasks.workunit.client.0.smithi148.stdout:.snap/s/foo 2023-10-30T13:21:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir .snap/s 2023-10-30T13:21:30.092 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm foo 2023-10-30T13:21:30.098 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo OK 2023-10-30T13:21:30.099 INFO:tasks.workunit.client.0.smithi148.stdout:OK 2023-10-30T13:21:30.099 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:21:30.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:21:30.172 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-10-30T13:21:30.173 DEBUG:teuthology.orchestra.run.smithi148:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-10-30T13:21:30.237 INFO:tasks.workunit.client.0.smithi148.stderr:+ set -e 2023-10-30T13:21:30.238 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir foo 2023-10-30T13:21:30.243 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo x 2023-10-30T13:21:30.243 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:30.700 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir foo/.snap/ss 2023-10-30T13:21:30.715 INFO:tasks.workunit.client.0.smithi148.stderr:+ ln foo/x foo/xx 2023-10-30T13:21:30.719 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat foo/.snap/ss/x 2023-10-30T13:21:30.730 INFO:tasks.workunit.client.0.smithi148.stdout:x 2023-10-30T13:21:30.730 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir foo/.snap/ss 2023-10-30T13:21:30.742 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo a 2023-10-30T13:21:30.743 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo b 2023-10-30T13:21:30.743 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir foo/.snap/s 2023-10-30T13:21:30.760 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat foo/.snap/s/a 2023-10-30T13:21:30.775 INFO:tasks.workunit.client.0.smithi148.stderr:+ r=a 2023-10-30T13:21:30.775 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ -z a ] 2023-10-30T13:21:30.776 INFO:tasks.workunit.client.0.smithi148.stderr:+ ln foo/b foo/b2 2023-10-30T13:21:30.778 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat foo/.snap/s/b 2023-10-30T13:21:30.795 INFO:tasks.workunit.client.0.smithi148.stdout:b 2023-10-30T13:21:30.796 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo this used to hang: 2023-10-30T13:21:30.796 INFO:tasks.workunit.client.0.smithi148.stdout:this used to hang: 2023-10-30T13:21:30.797 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo more 2023-10-30T13:21:30.797 INFO:tasks.workunit.client.0.smithi148.stdout:oh, it didn't hang! good job. 2023-10-30T13:21:30.798 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo oh, it didn't hang! good job. 2023-10-30T13:21:30.798 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat foo/b 2023-10-30T13:21:30.799 INFO:tasks.workunit.client.0.smithi148.stdout:b 2023-10-30T13:21:30.799 INFO:tasks.workunit.client.0.smithi148.stdout:more 2023-10-30T13:21:30.800 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir foo/.snap/s 2023-10-30T13:21:30.807 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm -r foo 2023-10-30T13:21:30.839 INFO:tasks.workunit.client.0.smithi148.stdout:OK 2023-10-30T13:21:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo OK 2023-10-30T13:21:30.840 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:21:30.841 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:21:30.913 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-10-30T13:21:30.913 DEBUG:teuthology.orchestra.run.smithi148:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-10-30T13:21:30.974 INFO:tasks.workunit.client.0.smithi148.stderr:+ set -e 2023-10-30T13:21:30.974 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:30.977 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/bar 2023-10-30T13:21:30.996 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/foo 2023-10-30T13:21:31.634 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/foo 2023-10-30T13:21:31.644 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:31.649 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/bar/a 2023-10-30T13:21:31.654 INFO:tasks.workunit.client.0.smithi148.stdout: File: .snap/bar/a 2023-10-30T13:21:31.654 INFO:tasks.workunit.client.0.smithi148.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T13:21:31.655 INFO:tasks.workunit.client.0.smithi148.stdout:Device: fah/250d Inode: 1099511632238 Links: 2 2023-10-30T13:21:31.655 INFO:tasks.workunit.client.0.smithi148.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T13:21:31.655 INFO:tasks.workunit.client.0.smithi148.stdout:Access: 2023-10-30 13:21:30.974629067 +0000 2023-10-30T13:21:31.655 INFO:tasks.workunit.client.0.smithi148.stdout:Modify: 2023-10-30 13:21:30.974629067 +0000 2023-10-30T13:21:31.656 INFO:tasks.workunit.client.0.smithi148.stdout:Change: 2023-10-30 13:21:30.974629067 +0000 2023-10-30T13:21:31.656 INFO:tasks.workunit.client.0.smithi148.stdout: Birth: 2023-10-30 13:21:30.974629067 +0000 2023-10-30T13:21:31.656 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir .snap/bar 2023-10-30T13:21:31.672 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo OK 2023-10-30T13:21:31.673 INFO:tasks.workunit.client.0.smithi148.stdout:OK 2023-10-30T13:21:31.673 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:21:31.673 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:21:31.746 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-10-30T13:21:31.747 DEBUG:teuthology.orchestra.run.smithi148:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-10-30T13:21:31.806 INFO:tasks.workunit.client.0.smithi148.stderr:+ set -e 2023-10-30T13:21:31.807 INFO:tasks.workunit.client.0.smithi148.stderr:+ seq 1 20 2023-10-30T13:21:31.808 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:31.812 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:31.814 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:31.835 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:31.847 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:31.848 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:31.990 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:32.006 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:32.006 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:32.006 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:32.007 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:32.007 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:32.007 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:32.007 INFO:tasks.workunit.client.0.smithi148.stdout: 51760 ? 00:00:00 ps 2023-10-30T13:21:32.008 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:32.028 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:32.045 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:32.049 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:32.051 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:32.069 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:32.080 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:32.081 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:32.114 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:32.130 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:32.130 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:32.130 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:32.130 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:32.131 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:32.131 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:32.131 INFO:tasks.workunit.client.0.smithi148.stdout: 51770 ? 00:00:00 ps 2023-10-30T13:21:32.131 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:32.151 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:32.156 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:32.169 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:32.172 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:32.175 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:32.196 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:32.209 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:32.210 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:32.238 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:32.253 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:32.253 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:32.253 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:32.253 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:32.254 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:32.254 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:32.254 INFO:tasks.workunit.client.0.smithi148.stdout: 51780 ? 00:00:00 ps 2023-10-30T13:21:32.255 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:32.276 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:32.281 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:32.295 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:32.299 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:32.301 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:32.323 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:32.335 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:32.336 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:32.344 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:32.359 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:32.359 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:32.359 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:32.359 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:32.360 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:32.360 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:32.360 INFO:tasks.workunit.client.0.smithi148.stdout: 51790 ? 00:00:00 ps 2023-10-30T13:21:32.361 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:32.382 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:32.386 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:32.399 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:32.403 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:32.405 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:32.424 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:32.435 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:32.437 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:32.472 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:32.487 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:32.487 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:32.487 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:32.487 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:32.488 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:32.488 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:32.488 INFO:tasks.workunit.client.0.smithi148.stdout: 51800 ? 00:00:00 ps 2023-10-30T13:21:32.489 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:32.510 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:32.515 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:32.528 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:32.532 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:32.534 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:32.556 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:32.568 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:32.569 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:32.605 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:32.620 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:32.621 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:32.621 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:32.621 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:32.621 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:32.621 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:32.622 INFO:tasks.workunit.client.0.smithi148.stdout: 51810 ? 00:00:00 ps 2023-10-30T13:21:32.622 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:32.642 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:32.647 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:32.660 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:32.915 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:32.918 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:32.937 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:32.950 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:32.951 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:33.048 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:33.064 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:33.064 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:33.064 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:33.065 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:33.065 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:33.065 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:33.065 INFO:tasks.workunit.client.0.smithi148.stdout: 51820 ? 00:00:00 ps 2023-10-30T13:21:33.066 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:33.087 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:33.091 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:33.103 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:33.108 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:33.110 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:33.129 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:33.141 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:33.142 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:33.172 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:33.186 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:33.187 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:33.187 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:33.187 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:33.188 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:33.188 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:33.188 INFO:tasks.workunit.client.0.smithi148.stdout: 51830 ? 00:00:00 ps 2023-10-30T13:21:33.189 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:33.725 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:33.728 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:34.381 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:21:34.381 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:21:34.381 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:21:34.385 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:21:34.386 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:34.472 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:21:34.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e0a394640 1 Processor -- start 2023-10-30T13:21:34.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e0a394640 1 -- start start 2023-10-30T13:21:34.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e0a394640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e0415d6f0 0x7f4e0415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:34.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e0a394640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040cb800 con 0x7f4e040cf160 2023-10-30T13:21:34.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e0a394640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040cb330 con 0x7f4e041538a0 2023-10-30T13:21:34.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e0a394640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e040cad70 con 0x7f4e0415d6f0 2023-10-30T13:21:34.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e08931640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4e040cf160 0x7f4e04164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43428/0 (socket says 172.21.15.148:43428) 2023-10-30T13:21:34.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e08931640 1 -- 172.21.15.148:0/3065276095 learned_addr learned my addr 172.21.15.148:0/3065276095 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:21:34.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e037fe640 1 --2- 172.21.15.148:0/3065276095 >> v2:172.21.15.150:3300/0 conn(0x7f4e0415d6f0 0x7f4e0415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:34.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e02ffd640 1 -- 172.21.15.148:0/3065276095 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875874639 0 0) 0x7f4e040cb800 con 0x7f4e040cf160 2023-10-30T13:21:34.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e02ffd640 1 -- 172.21.15.148:0/3065276095 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dec003530 con 0x7f4e040cf160 2023-10-30T13:21:34.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e037fe640 1 -- 172.21.15.148:0/3065276095 >> v1:172.21.15.150:6789/0 conn(0x7f4e041538a0 legacy=0x7f4e040ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:34.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e02ffd640 1 -- 172.21.15.148:0/3065276095 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4013663266 0 0) 0x7f4dec003530 con 0x7f4e040cf160 2023-10-30T13:21:34.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e037fe640 1 -- 172.21.15.148:0/3065276095 >> v1:172.21.15.148:6789/0 conn(0x7f4e040cf160 legacy=0x7f4e04164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:34.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.526+0000 7f4e037fe640 1 -- 172.21.15.148:0/3065276095 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e040cb050 con 0x7f4e0415d6f0 2023-10-30T13:21:34.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e037fe640 1 --2- 172.21.15.148:0/3065276095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0415d6f0 0x7f4e0415a090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097c0 tx=0x7f4df801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7a593a8506fb85e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:34.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e02ffd640 1 -- 172.21.15.148:0/3065276095 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc001a00 con 0x7f4e0415d6f0 2023-10-30T13:21:34.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e02ffd640 1 -- 172.21.15.148:0/3065276095 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df80a8610 con 0x7f4e0415d6f0 2023-10-30T13:21:34.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e02ffd640 1 -- 172.21.15.148:0/3065276095 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df80a8980 con 0x7f4e0415d6f0 2023-10-30T13:21:34.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e0a394640 1 -- 172.21.15.148:0/3065276095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0415d6f0 msgr2=0x7f4e0415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:34.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e0a394640 1 --2- 172.21.15.148:0/3065276095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0415d6f0 0x7f4e0415a090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097c0 tx=0x7f4df801eb90 comp rx=0 tx=0).stop 2023-10-30T13:21:34.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e0a394640 1 -- 172.21.15.148:0/3065276095 shutdown_connections 2023-10-30T13:21:34.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e0a394640 1 --2- 172.21.15.148:0/3065276095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0415d6f0 0x7f4e0415a090 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:34.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e0a394640 1 -- 172.21.15.148:0/3065276095 >> 172.21.15.148:0/3065276095 conn(0x7f4e0406bfe0 msgr2=0x7f4e0406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:34.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e0a394640 1 -- 172.21.15.148:0/3065276095 shutdown_connections 2023-10-30T13:21:34.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.530+0000 7f4e0a394640 1 -- 172.21.15.148:0/3065276095 wait complete. 2023-10-30T13:21:34.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 Processor -- start 2023-10-30T13:21:34.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 -- start start 2023-10-30T13:21:34.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e040cf160 0x7f4e040cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:34.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e041538a0 0x7f4e040cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:34.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e0415d6f0 0x7f4e0414cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:34.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040cd2b0 con 0x7f4e0415d6f0 2023-10-30T13:21:34.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0414e880 con 0x7f4e041538a0 2023-10-30T13:21:34.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04158ab0 con 0x7f4e040cf160 2023-10-30T13:21:34.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e08931640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e0415d6f0 0x7f4e0414cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:34.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e08931640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e0415d6f0 0x7f4e0414cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34546/0 (socket says 172.21.15.148:34546) 2023-10-30T13:21:34.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e08931640 1 -- 172.21.15.148:0/382836989 learned_addr learned my addr 172.21.15.148:0/382836989 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:21:34.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e037fe640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e041538a0 0x7f4e040cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:34.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e03fff640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e040cf160 0x7f4e040cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e08931640 1 -- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e040cf160 msgr2=0x7f4e040cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e08931640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e040cf160 0x7f4e040cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e08931640 1 -- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e041538a0 msgr2=0x7f4e040cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e08931640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e041538a0 0x7f4e040cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e08931640 1 -- 172.21.15.148:0/382836989 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df8009420 con 0x7f4e0415d6f0 2023-10-30T13:21:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e08931640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e0415d6f0 0x7f4e0414cbe0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00cc10 tx=0x7f4dfc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e00ff9640 1 -- 172.21.15.148:0/382836989 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc00ace0 con 0x7f4e0415d6f0 2023-10-30T13:21:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e00ff9640 1 -- 172.21.15.148:0/382836989 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc00ae40 con 0x7f4e0415d6f0 2023-10-30T13:21:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e04157f00 con 0x7f4e0415d6f0 2023-10-30T13:21:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e00ff9640 1 -- 172.21.15.148:0/382836989 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc012640 con 0x7f4e0415d6f0 2023-10-30T13:21:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.534+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e04158180 con 0x7f4e0415d6f0 2023-10-30T13:21:34.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.538+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e04158c20 con 0x7f4e0415d6f0 2023-10-30T13:21:34.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.538+0000 7f4e00ff9640 1 -- 172.21.15.148:0/382836989 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4dfc0127a0 con 0x7f4e0415d6f0 2023-10-30T13:21:34.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.538+0000 7f4e00ff9640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4df00664a0 0x7f4df0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:34.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.538+0000 7f4e00ff9640 1 -- 172.21.15.148:0/382836989 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f4dfc084670 con 0x7f4e0415d6f0 2023-10-30T13:21:34.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.538+0000 7f4e03fff640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4df00664a0 0x7f4df0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:34.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.538+0000 7f4e03fff640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4df00664a0 0x7f4df0068960 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4df4007fd0 tx=0x7f4df4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:34.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.542+0000 7f4e00ff9640 1 -- 172.21.15.148:0/382836989 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4dfc051860 con 0x7f4e0415d6f0 2023-10-30T13:21:34.575 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:34.580 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:34.587 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:34.622 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:34.630 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:34.631 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:34.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.742+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e0404e170 con 0x7f4e0415d6f0 2023-10-30T13:21:34.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.742+0000 7f4e00ff9640 1 -- 172.21.15.148:0/382836989 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+6512 (secure 0 0 0) 0x7f4dfc054e80 con 0x7f4e0415d6f0 2023-10-30T13:21:34.745 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:21:34.745 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:21:27.737025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:34.745 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 57 2023-10-30T13:21:34.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4df00664a0 msgr2=0x7f4df0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:34.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4df00664a0 0x7f4df0068960 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4df4007fd0 tx=0x7f4df4009040 comp rx=0 tx=0).stop 2023-10-30T13:21:34.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e0415d6f0 msgr2=0x7f4e0414cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:34.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e0415d6f0 0x7f4e0414cbe0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00cc10 tx=0x7f4dfc00a430 comp rx=0 tx=0).stop 2023-10-30T13:21:34.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 shutdown_connections 2023-10-30T13:21:34.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4df00664a0 0x7f4df0068960 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e0415d6f0 0x7f4e0414cbe0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e041538a0 0x7f4e040cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 --2- 172.21.15.148:0/382836989 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e040cf160 0x7f4e040cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 >> 172.21.15.148:0/382836989 conn(0x7f4e0406bfe0 msgr2=0x7f4e0415b640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 shutdown_connections 2023-10-30T13:21:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:21:34.746+0000 7f4e0a394640 1 -- 172.21.15.148:0/382836989 wait complete. 2023-10-30T13:21:34.766 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:21:34.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2023-10-30T13:21:34.925 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:34.939 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:34.940 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:34.940 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:34.940 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:34.940 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:34.941 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:34.941 INFO:tasks.workunit.client.0.smithi148.stdout: 51890 ? 00:00:00 ps 2023-10-30T13:21:34.941 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:34.962 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:34.967 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:34.979 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:34.983 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:34.985 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:35.004 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:35.017 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:35.018 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:35.133 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:35.147 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:35.147 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:35.147 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:35.147 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:35.148 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:35.148 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:35.148 INFO:tasks.workunit.client.0.smithi148.stdout: 51900 ? 00:00:00 ps 2023-10-30T13:21:35.148 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:35.171 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:35.176 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:35.188 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:35.192 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:35.194 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:35.212 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:35.224 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:35.225 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:35.290 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:35.305 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:35.305 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:35.305 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:35.305 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:35.306 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:35.306 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:35.306 INFO:tasks.workunit.client.0.smithi148.stdout: 51910 ? 00:00:00 ps 2023-10-30T13:21:35.306 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:35.328 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:35.332 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:35.345 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:35.349 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:35.351 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:35.370 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:35.382 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:35.383 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:35.440 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:35.454 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:35.454 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:35.454 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:35.455 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:35.455 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:35.455 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:35.455 INFO:tasks.workunit.client.0.smithi148.stdout: 51920 ? 00:00:00 ps 2023-10-30T13:21:35.456 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:35.477 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:35.481 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:35.764 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:35.769 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:35.771 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:35.790 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:35.803 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:35.804 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:35.933 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:35.947 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:35.947 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:35.947 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:35.947 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:35.948 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:35.948 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:35.948 INFO:tasks.workunit.client.0.smithi148.stdout: 51931 ? 00:00:00 ps 2023-10-30T13:21:35.949 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:35.974 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:35.981 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:35.998 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:36.003 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:36.006 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:36.027 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:36.040 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:36.042 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:36.083 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:36.097 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:36.098 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:36.098 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:36.098 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:36.098 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:36.098 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:36.099 INFO:tasks.workunit.client.0.smithi148.stdout: 51941 ? 00:00:00 ps 2023-10-30T13:21:36.099 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:36.124 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:36.128 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:36.141 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:36.144 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:36.147 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:36.168 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:36.182 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:36.215 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:36.228 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:36.228 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:36.229 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:36.229 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:36.229 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:36.229 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:36.229 INFO:tasks.workunit.client.0.smithi148.stdout: 51951 ? 00:00:00 ps 2023-10-30T13:21:36.230 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:36.261 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:36.277 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:36.282 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:36.286 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:36.308 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:36.322 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:36.323 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:36.383 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:36.397 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:36.398 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:36.398 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:36.398 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:36.398 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:36.398 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:36.399 INFO:tasks.workunit.client.0.smithi148.stdout: 51961 ? 00:00:00 ps 2023-10-30T13:21:36.399 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:36.424 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:36.431 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:36.445 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:36.449 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:36.451 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:36.471 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:36.483 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:36.484 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:36.541 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:36.555 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:36.555 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:36.556 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:36.556 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:36.556 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:36.556 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:36.556 INFO:tasks.workunit.client.0.smithi148.stdout: 51971 ? 00:00:00 ps 2023-10-30T13:21:36.557 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:36.578 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:36.582 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:36.598 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:36.604 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:36.614 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:36.697 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:36.707 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:36.708 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:36.823 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:36.823 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:36.823 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:36.823 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:36.824 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:36.824 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:36.824 INFO:tasks.workunit.client.0.smithi148.stdout: 51981 ? 00:00:00 ps 2023-10-30T13:21:36.824 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:36.854 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:36.858 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:36.872 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:36.885 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:36.918 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:36.931 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:36.932 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:37.010 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:37.024 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:37.024 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:37.024 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:37.024 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:37.025 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:37.025 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:37.025 INFO:tasks.workunit.client.0.smithi148.stdout: 51991 ? 00:00:00 ps 2023-10-30T13:21:37.025 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:37.055 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:37.068 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a 2023-10-30T13:21:37.072 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/one 2023-10-30T13:21:37.074 INFO:tasks.workunit.client.0.smithi148.stderr:+ cat 2023-10-30T13:21:37.096 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir a/.snap/two 2023-10-30T13:21:37.108 INFO:tasks.workunit.client.0.smithi148.stderr:+ chmod 777 a/foo 2023-10-30T13:21:37.110 INFO:tasks.workunit.client.0.smithi148.stderr:+ sync 2023-10-30T13:21:37.141 INFO:tasks.workunit.client.0.smithi148.stderr:+ ps 2023-10-30T13:21:37.155 INFO:tasks.workunit.client.0.smithi148.stdout: PID TTY TIME CMD 2023-10-30T13:21:37.156 INFO:tasks.workunit.client.0.smithi148.stdout: 1297 ? 00:00:00 systemd 2023-10-30T13:21:37.156 INFO:tasks.workunit.client.0.smithi148.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-30T13:21:37.156 INFO:tasks.workunit.client.0.smithi148.stdout: 27951 ? 00:00:02 sshd 2023-10-30T13:21:37.156 INFO:tasks.workunit.client.0.smithi148.stdout: 51749 ? 00:00:00 timeout 2023-10-30T13:21:37.156 INFO:tasks.workunit.client.0.smithi148.stdout: 51752 ? 00:00:00 snaptest-double 2023-10-30T13:21:37.157 INFO:tasks.workunit.client.0.smithi148.stdout: 52001 ? 00:00:00 ps 2023-10-30T13:21:37.157 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T13:21:37.179 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm a/foo 2023-10-30T13:21:37.184 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir a 2023-10-30T13:21:37.992 INFO:tasks.workunit.client.0.smithi148.stdout:OK 2023-10-30T13:21:37.993 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo OK 2023-10-30T13:21:37.993 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:21:37.993 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:21:38.028 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-10-30T13:21:38.028 DEBUG:teuthology.orchestra.run.smithi148:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-10-30T13:21:38.085 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/foo 2023-10-30T13:21:38.099 INFO:tasks.workunit.client.0.smithi148.stdout:We want ENOENT, not ESTALE, here. 2023-10-30T13:21:38.100 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo We want ENOENT, not ESTALE, here. 2023-10-30T13:21:38.100 INFO:tasks.workunit.client.0.smithi148.stderr:+ seq 1 100 2023-10-30T13:21:38.102 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/1 2023-10-30T13:21:38.102 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.105 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2023-10-30T13:21:38.106 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/2 2023-10-30T13:21:38.106 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.107 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2023-10-30T13:21:38.108 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/3 2023-10-30T13:21:38.109 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.110 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2023-10-30T13:21:38.111 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/4 2023-10-30T13:21:38.111 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.113 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2023-10-30T13:21:38.114 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/5 2023-10-30T13:21:38.114 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.115 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2023-10-30T13:21:38.116 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/6 2023-10-30T13:21:38.116 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.118 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2023-10-30T13:21:38.118 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/7 2023-10-30T13:21:38.119 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.120 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2023-10-30T13:21:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/8 2023-10-30T13:21:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.123 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2023-10-30T13:21:38.123 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/9 2023-10-30T13:21:38.124 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.125 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2023-10-30T13:21:38.126 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/10 2023-10-30T13:21:38.127 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.128 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2023-10-30T13:21:38.129 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/11 2023-10-30T13:21:38.129 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.130 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2023-10-30T13:21:38.131 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/12 2023-10-30T13:21:38.131 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.133 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2023-10-30T13:21:38.134 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/13 2023-10-30T13:21:38.134 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.135 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2023-10-30T13:21:38.136 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/14 2023-10-30T13:21:38.137 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.138 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2023-10-30T13:21:38.139 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/15 2023-10-30T13:21:38.139 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.140 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2023-10-30T13:21:38.142 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/16 2023-10-30T13:21:38.142 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.143 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2023-10-30T13:21:38.144 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/17 2023-10-30T13:21:38.144 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.145 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2023-10-30T13:21:38.146 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/18 2023-10-30T13:21:38.147 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.148 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2023-10-30T13:21:38.149 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/19 2023-10-30T13:21:38.149 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.151 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2023-10-30T13:21:38.152 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/20 2023-10-30T13:21:38.152 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.153 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2023-10-30T13:21:38.154 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/21 2023-10-30T13:21:38.154 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.156 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2023-10-30T13:21:38.156 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/22 2023-10-30T13:21:38.157 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.158 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2023-10-30T13:21:38.159 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/23 2023-10-30T13:21:38.159 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.160 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2023-10-30T13:21:38.161 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/24 2023-10-30T13:21:38.162 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.163 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2023-10-30T13:21:38.164 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/25 2023-10-30T13:21:38.164 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.166 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2023-10-30T13:21:38.166 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/26 2023-10-30T13:21:38.167 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.168 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2023-10-30T13:21:38.169 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/27 2023-10-30T13:21:38.169 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.170 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2023-10-30T13:21:38.171 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/28 2023-10-30T13:21:38.172 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.173 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2023-10-30T13:21:38.174 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/29 2023-10-30T13:21:38.175 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.176 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2023-10-30T13:21:38.177 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/30 2023-10-30T13:21:38.177 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.178 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2023-10-30T13:21:38.179 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/31 2023-10-30T13:21:38.180 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.181 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2023-10-30T13:21:38.182 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/32 2023-10-30T13:21:38.182 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.184 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2023-10-30T13:21:38.185 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/33 2023-10-30T13:21:38.185 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.187 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2023-10-30T13:21:38.187 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/34 2023-10-30T13:21:38.188 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.189 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2023-10-30T13:21:38.190 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/35 2023-10-30T13:21:38.190 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.192 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2023-10-30T13:21:38.192 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/36 2023-10-30T13:21:38.193 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.194 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2023-10-30T13:21:38.195 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/37 2023-10-30T13:21:38.195 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.197 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2023-10-30T13:21:38.198 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/38 2023-10-30T13:21:38.198 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.199 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2023-10-30T13:21:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/39 2023-10-30T13:21:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.202 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2023-10-30T13:21:38.203 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/40 2023-10-30T13:21:38.203 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.204 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2023-10-30T13:21:38.205 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/41 2023-10-30T13:21:38.206 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.207 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2023-10-30T13:21:38.208 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/42 2023-10-30T13:21:38.208 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.209 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2023-10-30T13:21:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/43 2023-10-30T13:21:38.211 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.212 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2023-10-30T13:21:38.213 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/44 2023-10-30T13:21:38.213 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.215 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2023-10-30T13:21:38.215 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/45 2023-10-30T13:21:38.216 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.217 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2023-10-30T13:21:38.218 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/46 2023-10-30T13:21:38.218 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.220 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2023-10-30T13:21:38.221 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/47 2023-10-30T13:21:38.221 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.222 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2023-10-30T13:21:38.223 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/48 2023-10-30T13:21:38.223 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.225 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2023-10-30T13:21:38.226 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/49 2023-10-30T13:21:38.226 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.227 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2023-10-30T13:21:38.228 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/50 2023-10-30T13:21:38.228 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.230 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2023-10-30T13:21:38.231 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/51 2023-10-30T13:21:38.231 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.232 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2023-10-30T13:21:38.233 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/52 2023-10-30T13:21:38.233 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.234 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2023-10-30T13:21:38.235 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/53 2023-10-30T13:21:38.236 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.237 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2023-10-30T13:21:38.238 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/54 2023-10-30T13:21:38.239 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.240 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2023-10-30T13:21:38.241 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/55 2023-10-30T13:21:38.241 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.242 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2023-10-30T13:21:38.243 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/56 2023-10-30T13:21:38.243 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.245 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2023-10-30T13:21:38.246 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/57 2023-10-30T13:21:38.246 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.247 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2023-10-30T13:21:38.248 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/58 2023-10-30T13:21:38.248 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.250 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2023-10-30T13:21:38.251 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/59 2023-10-30T13:21:38.251 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.252 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2023-10-30T13:21:38.253 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/60 2023-10-30T13:21:38.253 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.255 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2023-10-30T13:21:38.256 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/61 2023-10-30T13:21:38.256 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.258 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2023-10-30T13:21:38.258 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/62 2023-10-30T13:21:38.259 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.260 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2023-10-30T13:21:38.261 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/63 2023-10-30T13:21:38.261 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.263 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2023-10-30T13:21:38.263 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/64 2023-10-30T13:21:38.264 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.265 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2023-10-30T13:21:38.266 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/65 2023-10-30T13:21:38.266 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.268 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2023-10-30T13:21:38.268 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/66 2023-10-30T13:21:38.269 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.270 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2023-10-30T13:21:38.271 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/67 2023-10-30T13:21:38.271 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.273 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2023-10-30T13:21:38.274 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/68 2023-10-30T13:21:38.274 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.275 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2023-10-30T13:21:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/69 2023-10-30T13:21:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.278 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2023-10-30T13:21:38.279 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/70 2023-10-30T13:21:38.279 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.280 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2023-10-30T13:21:38.281 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/71 2023-10-30T13:21:38.282 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.283 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2023-10-30T13:21:38.284 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/72 2023-10-30T13:21:38.284 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.286 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2023-10-30T13:21:38.287 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/73 2023-10-30T13:21:38.287 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.288 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2023-10-30T13:21:38.289 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/74 2023-10-30T13:21:38.289 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.291 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2023-10-30T13:21:38.292 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/75 2023-10-30T13:21:38.292 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.639 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2023-10-30T13:21:38.644 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/76 2023-10-30T13:21:38.644 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.647 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2023-10-30T13:21:38.649 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat+ .snap/foo/77 2023-10-30T13:21:38.651 INFO:tasks.workunit.client.0.smithi148.stderr:grep No such file 2023-10-30T13:21:38.654 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2023-10-30T13:21:38.655 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/78 2023-10-30T13:21:38.657 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.659 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2023-10-30T13:21:38.661 INFO:tasks.workunit.client.0.smithi148.stderr:+ + grep No such file 2023-10-30T13:21:38.661 INFO:tasks.workunit.client.0.smithi148.stderr:stat .snap/foo/79 2023-10-30T13:21:38.664 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2023-10-30T13:21:38.665 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/80 2023-10-30T13:21:38.665 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.667 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2023-10-30T13:21:38.668 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/81 2023-10-30T13:21:38.668 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.670 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2023-10-30T13:21:38.670 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/82 2023-10-30T13:21:38.671 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.672 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2023-10-30T13:21:38.673 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/83 2023-10-30T13:21:38.674 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.675 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2023-10-30T13:21:38.676 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/84 2023-10-30T13:21:38.676 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.678 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2023-10-30T13:21:38.679 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/85 2023-10-30T13:21:38.679 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.681 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2023-10-30T13:21:38.682 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/86 2023-10-30T13:21:38.682 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.683 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2023-10-30T13:21:38.684 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/87 2023-10-30T13:21:38.685 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.686 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2023-10-30T13:21:38.687 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/88 2023-10-30T13:21:38.687 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.688 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2023-10-30T13:21:38.689 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/89 2023-10-30T13:21:38.689 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.691 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2023-10-30T13:21:38.692 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/90 2023-10-30T13:21:38.692 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.693 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2023-10-30T13:21:38.694 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/91 2023-10-30T13:21:38.694 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.696 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2023-10-30T13:21:38.696 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/92 2023-10-30T13:21:38.697 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.698 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2023-10-30T13:21:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/93 2023-10-30T13:21:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.700 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2023-10-30T13:21:38.701 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/94 2023-10-30T13:21:38.701 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.703 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2023-10-30T13:21:38.704 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/95 2023-10-30T13:21:38.704 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.706 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2023-10-30T13:21:38.706 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/96 2023-10-30T13:21:38.707 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.708 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2023-10-30T13:21:38.709 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/97 2023-10-30T13:21:38.709 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.711 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2023-10-30T13:21:38.712 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/98 2023-10-30T13:21:38.712 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.713 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2023-10-30T13:21:38.714 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/99 2023-10-30T13:21:38.714 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.715 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2023-10-30T13:21:38.716 INFO:tasks.workunit.client.0.smithi148.stderr:+ stat .snap/foo/100 2023-10-30T13:21:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:+ grep No such file 2023-10-30T13:21:38.718 INFO:tasks.workunit.client.0.smithi148.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2023-10-30T13:21:38.719 INFO:tasks.workunit.client.0.smithi148.stderr:+ rmdir .snap/foo 2023-10-30T13:21:38.758 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo OK 2023-10-30T13:21:38.759 INFO:tasks.workunit.client.0.smithi148.stdout:OK 2023-10-30T13:21:38.759 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T13:21:38.760 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T13:21:38.829 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-10-30T13:21:38.830 DEBUG:teuthology.orchestra.run.smithi148:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-10-30T13:21:38.894 INFO:tasks.workunit.client.0.smithi148.stderr:+ set -e 2023-10-30T13:21:38.894 INFO:tasks.workunit.client.0.smithi148.stderr:+ sudo git config --global http.sslVerify false 2023-10-30T13:21:38.912 INFO:tasks.workunit.client.0.smithi148.stderr:+ sudo git config --global http.postBuffer 1048576000 2023-10-30T13:21:38.926 INFO:tasks.workunit.client.0.smithi148.stderr:+ retried=false 2023-10-30T13:21:38.926 INFO:tasks.workunit.client.0.smithi148.stderr:+ trap -- retry EXIT 2023-10-30T13:21:38.926 INFO:tasks.workunit.client.0.smithi148.stderr:+ rm -rf ceph 2023-10-30T13:21:38.926 INFO:tasks.workunit.client.0.smithi148.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-10-30T13:21:38.931 INFO:tasks.workunit.client.0.smithi148.stderr:Cloning into 'ceph'... 2023-10-30T13:21:55.198 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:21:55.202 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:22.767 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:23.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.002+0000 7efe743c5640 1 Processor -- start 2023-10-30T13:22:23.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.002+0000 7efe743c5640 1 -- start start 2023-10-30T13:22:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe743c5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efe6c1538a0 0x7efe6c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe743c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0ca0e0 0x7efe6c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe743c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c0cb800 con 0x7efe6c0cf160 2023-10-30T13:22:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe743c5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6c15dc30 con 0x7efe6c1538a0 2023-10-30T13:22:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe743c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c0cb330 con 0x7efe6c0ca0e0 2023-10-30T13:22:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe72161640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efe6c1538a0 0x7efe6c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe72161640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efe6c1538a0 0x7efe6c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35186/0 (socket says 172.21.15.148:35186) 2023-10-30T13:22:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe72161640 1 -- 172.21.15.148:0/976564744 learned_addr learned my addr 172.21.15.148:0/976564744 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe71960640 1 --2- 172.21.15.148:0/976564744 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0ca0e0 0x7efe6c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe72161640 1 -- 172.21.15.148:0/976564744 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0ca0e0 msgr2=0x7efe6c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe72161640 1 --2- 172.21.15.148:0/976564744 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0ca0e0 0x7efe6c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe72161640 1 -- 172.21.15.148:0/976564744 >> v1:172.21.15.150:6789/0 conn(0x7efe6c0cf160 legacy=0x7efe6c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.006+0000 7efe72161640 1 -- 172.21.15.148:0/976564744 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c0cad70 con 0x7efe6c1538a0 2023-10-30T13:22:23.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.010+0000 7efe72161640 1 --2- 172.21.15.148:0/976564744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c1538a0 0x7efe6c15d6f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efe68009870 tx=0x7efe6801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6643329df2e694 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:23.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.010+0000 7efe7115f640 1 -- 172.21.15.148:0/976564744 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe680a8a30 con 0x7efe6c1538a0 2023-10-30T13:22:23.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.010+0000 7efe7115f640 1 -- 172.21.15.148:0/976564744 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe680a8b90 con 0x7efe6c1538a0 2023-10-30T13:22:23.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.010+0000 7efe7115f640 1 -- 172.21.15.148:0/976564744 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe680b0440 con 0x7efe6c1538a0 2023-10-30T13:22:23.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.010+0000 7efe743c5640 1 -- 172.21.15.148:0/976564744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c1538a0 msgr2=0x7efe6c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.010+0000 7efe743c5640 1 --2- 172.21.15.148:0/976564744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c1538a0 0x7efe6c15d6f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efe68009870 tx=0x7efe6801ed30 comp rx=0 tx=0).stop 2023-10-30T13:22:23.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.010+0000 7efe743c5640 1 -- 172.21.15.148:0/976564744 shutdown_connections 2023-10-30T13:22:23.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.010+0000 7efe743c5640 1 --2- 172.21.15.148:0/976564744 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0ca0e0 0x7efe6c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.014+0000 7efe743c5640 1 --2- 172.21.15.148:0/976564744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c1538a0 0x7efe6c15d6f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.014+0000 7efe743c5640 1 -- 172.21.15.148:0/976564744 >> 172.21.15.148:0/976564744 conn(0x7efe6c06bfe0 msgr2=0x7efe6c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:23.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.014+0000 7efe743c5640 1 -- 172.21.15.148:0/976564744 shutdown_connections 2023-10-30T13:22:23.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.014+0000 7efe743c5640 1 -- 172.21.15.148:0/976564744 wait complete. 2023-10-30T13:22:23.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.014+0000 7efe743c5640 1 Processor -- start 2023-10-30T13:22:23.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.014+0000 7efe743c5640 1 -- start start 2023-10-30T13:22:23.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe743c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0ca0e0 0x7efe6c15b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe743c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c0cf160 0x7efe6c1e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe743c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c1538a0 0x7efe6c1e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe743c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c1e5580 con 0x7efe6c0cf160 2023-10-30T13:22:23.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe743c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c15bc60 con 0x7efe6c0ca0e0 2023-10-30T13:22:23.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe743c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c1e4670 con 0x7efe6c1538a0 2023-10-30T13:22:23.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe71960640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c0cf160 0x7efe6c1e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe72161640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0ca0e0 0x7efe6c15b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe71960640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c0cf160 0x7efe6c1e4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35192/0 (socket says 172.21.15.148:35192) 2023-10-30T13:22:23.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe71960640 1 -- 172.21.15.148:0/3756953466 learned_addr learned my addr 172.21.15.148:0/3756953466 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:23.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe72161640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0ca0e0 0x7efe6c15b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:39320/0 (socket says 172.21.15.148:39320) 2023-10-30T13:22:23.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe72962640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c1538a0 0x7efe6c1e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe71960640 1 -- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c1538a0 msgr2=0x7efe6c1e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe71960640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c1538a0 0x7efe6c1e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe71960640 1 -- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0ca0e0 msgr2=0x7efe6c15b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe71960640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0ca0e0 0x7efe6c15b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe71960640 1 -- 172.21.15.148:0/3756953466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680094d0 con 0x7efe6c0cf160 2023-10-30T13:22:23.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe71960640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c0cf160 0x7efe6c1e4b00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00d860 tx=0x7efe5c00dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:23.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe62ffd640 1 -- 172.21.15.148:0/3756953466 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c00de50 con 0x7efe6c0cf160 2023-10-30T13:22:23.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe62ffd640 1 -- 172.21.15.148:0/3756953466 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5c00b8d0 con 0x7efe6c0cf160 2023-10-30T13:22:23.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c158840 con 0x7efe6c0cf160 2023-10-30T13:22:23.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe62ffd640 1 -- 172.21.15.148:0/3756953466 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c00bc40 con 0x7efe6c0cf160 2023-10-30T13:22:23.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.018+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6c158ae0 con 0x7efe6c0cf160 2023-10-30T13:22:23.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.022+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6c0b2a50 con 0x7efe6c0cf160 2023-10-30T13:22:23.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.022+0000 7efe62ffd640 1 -- 172.21.15.148:0/3756953466 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe5c01e460 con 0x7efe6c0cf160 2023-10-30T13:22:23.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.022+0000 7efe62ffd640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe500664a0 0x7efe50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.022+0000 7efe62ffd640 1 -- 172.21.15.148:0/3756953466 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6226+0+0 (secure 0 0 0) 0x7efe5c0845d0 con 0x7efe6c0cf160 2023-10-30T13:22:23.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.022+0000 7efe72161640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe500664a0 0x7efe50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.026+0000 7efe72161640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe500664a0 0x7efe50068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efe680b2000 tx=0x7efe680aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:23.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.030+0000 7efe62ffd640 1 -- 172.21.15.148:0/3756953466 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe5c0517c0 con 0x7efe6c0cf160 2023-10-30T13:22:23.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.414+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe6c04e170 con 0x7efe6c0cf160 2023-10-30T13:22:23.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.414+0000 7efe62ffd640 1 -- 172.21.15.148:0/3756953466 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6511 (secure 0 0 0) 0x7efe5c054de0 con 0x7efe6c0cf160 2023-10-30T13:22:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:23.414 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:02.735218+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:23.414 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 58 2023-10-30T13:22:23.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.414+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe500664a0 msgr2=0x7efe50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.414+0000 7efe743c5640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe500664a0 0x7efe50068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efe680b2000 tx=0x7efe680aa070 comp rx=0 tx=0).stop 2023-10-30T13:22:23.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c0cf160 msgr2=0x7efe6c1e4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c0cf160 0x7efe6c1e4b00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00d860 tx=0x7efe5c00dd20 comp rx=0 tx=0).stop 2023-10-30T13:22:23.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 shutdown_connections 2023-10-30T13:22:23.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe500664a0 0x7efe50068960 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c0cf160 0x7efe6c1e4b00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c1538a0 0x7efe6c1e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 --2- 172.21.15.148:0/3756953466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0ca0e0 0x7efe6c15b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 >> 172.21.15.148:0/3756953466 conn(0x7efe6c06bfe0 msgr2=0x7efe6c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:23.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 shutdown_connections 2023-10-30T13:22:23.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.418+0000 7efe743c5640 1 -- 172.21.15.148:0/3756953466 wait complete. 2023-10-30T13:22:23.446 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-30T13:22:23.447 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:22:23.447 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:22:23.467 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:22:23.468 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.702+0000 7f9b08b93640 1 Processor -- start 2023-10-30T13:22:23.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.702+0000 7f9b08b93640 1 -- start start 2023-10-30T13:22:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b08b93640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b041538f0 0x7f9b04159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b08b93640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b0415cd50 0x7f9b040c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b08b93640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0415d300 con 0x7f9b040cefa0 2023-10-30T13:22:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b08b93640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0415d5e0 con 0x7f9b041538f0 2023-10-30T13:22:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b08b93640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0415d8c0 con 0x7f9b0415cd50 2023-10-30T13:22:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b037fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b041538f0 0x7f9b04159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b037fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b041538f0 0x7f9b04159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35206/0 (socket says 172.21.15.148:35206) 2023-10-30T13:22:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b037fe640 1 -- 172.21.15.148:0/3156699017 learned_addr learned my addr 172.21.15.148:0/3156699017 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b02ffd640 1 --2- 172.21.15.148:0/3156699017 >> v2:172.21.15.150:3300/0 conn(0x7f9b0415cd50 0x7f9b040c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b037fe640 1 -- 172.21.15.148:0/3156699017 >> v2:172.21.15.150:3300/0 conn(0x7f9b0415cd50 msgr2=0x7f9b040c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b037fe640 1 --2- 172.21.15.148:0/3156699017 >> v2:172.21.15.150:3300/0 conn(0x7f9b0415cd50 0x7f9b040c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b037fe640 1 -- 172.21.15.148:0/3156699017 >> v1:172.21.15.150:6789/0 conn(0x7f9b040cefa0 legacy=0x7f9b040cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.706+0000 7f9b037fe640 1 -- 172.21.15.148:0/3156699017 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0415dba0 con 0x7f9b041538f0 2023-10-30T13:22:23.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.710+0000 7f9b037fe640 1 --2- 172.21.15.148:0/3156699017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 0x7f9b04159fb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9b04072e90 tx=0x7f9af401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571ff58f09e6884d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:23.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.710+0000 7f9b027fc640 1 -- 172.21.15.148:0/3156699017 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af40a89f0 con 0x7f9b041538f0 2023-10-30T13:22:23.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.710+0000 7f9b027fc640 1 -- 172.21.15.148:0/3156699017 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af40a8b50 con 0x7f9b041538f0 2023-10-30T13:22:23.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.710+0000 7f9b027fc640 1 -- 172.21.15.148:0/3156699017 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af40b0440 con 0x7f9b041538f0 2023-10-30T13:22:23.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.714+0000 7f9b08b93640 1 -- 172.21.15.148:0/3156699017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 msgr2=0x7f9b04159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.714+0000 7f9b08b93640 1 --2- 172.21.15.148:0/3156699017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 0x7f9b04159fb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9b04072e90 tx=0x7f9af401ed10 comp rx=0 tx=0).stop 2023-10-30T13:22:23.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.714+0000 7f9b08b93640 1 -- 172.21.15.148:0/3156699017 shutdown_connections 2023-10-30T13:22:23.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.714+0000 7f9b08b93640 1 --2- 172.21.15.148:0/3156699017 >> v2:172.21.15.150:3300/0 conn(0x7f9b0415cd50 0x7f9b040c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.714+0000 7f9b08b93640 1 --2- 172.21.15.148:0/3156699017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 0x7f9b04159fb0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.714+0000 7f9b08b93640 1 -- 172.21.15.148:0/3156699017 >> 172.21.15.148:0/3156699017 conn(0x7f9b0406a640 msgr2=0x7f9b04065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:23.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.714+0000 7f9b08b93640 1 -- 172.21.15.148:0/3156699017 shutdown_connections 2023-10-30T13:22:23.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.714+0000 7f9b08b93640 1 -- 172.21.15.148:0/3156699017 wait complete. 2023-10-30T13:22:23.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.714+0000 7f9b08b93640 1 Processor -- start 2023-10-30T13:22:23.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b08b93640 1 -- start start 2023-10-30T13:22:23.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b08b93640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b040cefa0 0x7f9b0415bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b08b93640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 0x7f9b041ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b08b93640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b0415cd50 0x7f9b041efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b08b93640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b041f39b0 con 0x7f9b041538f0 2023-10-30T13:22:23.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b08b93640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b041f2de0 con 0x7f9b040cefa0 2023-10-30T13:22:23.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b08b93640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b041f3080 con 0x7f9b0415cd50 2023-10-30T13:22:23.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b02ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 0x7f9b041ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b02ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 0x7f9b041ef820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35208/0 (socket says 172.21.15.148:35208) 2023-10-30T13:22:23.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b02ffd640 1 -- 172.21.15.148:0/2747485730 learned_addr learned my addr 172.21.15.148:0/2747485730 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:23.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b03fff640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b0415cd50 0x7f9b041efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b037fe640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b040cefa0 0x7f9b0415bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b02ffd640 1 -- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b0415cd50 msgr2=0x7f9b041efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b02ffd640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b0415cd50 0x7f9b041efd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b02ffd640 1 -- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b040cefa0 msgr2=0x7f9b0415bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:23.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b02ffd640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b040cefa0 0x7f9b0415bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:23.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b02ffd640 1 -- 172.21.15.148:0/2747485730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af40094d0 con 0x7f9b041538f0 2023-10-30T13:22:23.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b02ffd640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 0x7f9b041ef820 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9af800e870 tx=0x7f9af800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:23.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9aebfff640 1 -- 172.21.15.148:0/2747485730 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af8009510 con 0x7f9b041538f0 2023-10-30T13:22:23.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9aebfff640 1 -- 172.21.15.148:0/2747485730 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af800eec0 con 0x7f9b041538f0 2023-10-30T13:22:23.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9aebfff640 1 -- 172.21.15.148:0/2747485730 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af800cb40 con 0x7f9b041538f0 2023-10-30T13:22:23.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.718+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b041f3360 con 0x7f9b041538f0 2023-10-30T13:22:23.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.722+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0415c8f0 con 0x7f9b041538f0 2023-10-30T13:22:23.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.722+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b041f3b20 con 0x7f9b041538f0 2023-10-30T13:22:23.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.726+0000 7f9aebfff640 1 -- 172.21.15.148:0/2747485730 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9af800cca0 con 0x7f9b041538f0 2023-10-30T13:22:23.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.726+0000 7f9aebfff640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9af00664a0 0x7f9af0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:23.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.726+0000 7f9aebfff640 1 -- 172.21.15.148:0/2747485730 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6226+0+0 (secure 0 0 0) 0x7f9af8055900 con 0x7f9b041538f0 2023-10-30T13:22:23.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.726+0000 7f9b037fe640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9af00664a0 0x7f9af0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:23.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.726+0000 7f9b037fe640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9af00664a0 0x7f9af0068960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9af40b2000 tx=0x7f9af4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:23.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:23.730+0000 7f9aebfff640 1 -- 172.21.15.148:0/2747485730 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9af8051c40 con 0x7f9b041538f0 2023-10-30T13:22:24.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.106+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b0404e170 con 0x7f9b041538f0 2023-10-30T13:22:24.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9aebfff640 1 -- 172.21.15.148:0/2747485730 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f9af8017090 con 0x7f9b041538f0 2023-10-30T13:22:24.109 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:24.109 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:02.735218+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:24.110 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 58 2023-10-30T13:22:24.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9af00664a0 msgr2=0x7f9af0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:24.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9af00664a0 0x7f9af0068960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9af40b2000 tx=0x7f9af4009040 comp rx=0 tx=0).stop 2023-10-30T13:22:24.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 msgr2=0x7f9b041ef820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:24.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 0x7f9b041ef820 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9af800e870 tx=0x7f9af800ec70 comp rx=0 tx=0).stop 2023-10-30T13:22:24.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 shutdown_connections 2023-10-30T13:22:24.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9af00664a0 0x7f9af0068960 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:24.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b041538f0 0x7f9b041ef820 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:24.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b0415cd50 0x7f9b041efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:24.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 --2- 172.21.15.148:0/2747485730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b040cefa0 0x7f9b0415bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:24.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 >> 172.21.15.148:0/2747485730 conn(0x7f9b0406a640 msgr2=0x7f9b040cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:24.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.110+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 shutdown_connections 2023-10-30T13:22:24.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:24.114+0000 7f9b08b93640 1 -- 172.21.15.148:0/2747485730 wait complete. 2023-10-30T13:22:24.143 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:22:25.258 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:25.264 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:26.144 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:26.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.378+0000 7f7334565640 1 Processor -- start 2023-10-30T13:22:26.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.378+0000 7f7334565640 1 -- start start 2023-10-30T13:22:26.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.378+0000 7f7334565640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f732c1538a0 0x7f732c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:26.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.378+0000 7f7334565640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0ca0e0 0x7f732c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:26.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7334565640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732c0cb800 con 0x7f732c0cf160 2023-10-30T13:22:26.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7334565640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f732c15dc30 con 0x7f732c1538a0 2023-10-30T13:22:26.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7334565640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732c0cb330 con 0x7f732c0ca0e0 2023-10-30T13:22:26.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7332b02640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f732c0cf160 0x7f732c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:39422/0 (socket says 172.21.15.148:39422) 2023-10-30T13:22:26.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7332b02640 1 -- 172.21.15.148:0/1943405181 learned_addr learned my addr 172.21.15.148:0/1943405181 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:22:26.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7331b00640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0ca0e0 0x7f732c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:26.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7332301640 1 --2- 172.21.15.148:0/1943405181 >> v2:172.21.15.148:3300/0 conn(0x7f732c1538a0 0x7f732c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:26.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f73312ff640 1 -- 172.21.15.148:0/1943405181 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131424099 0 0) 0x7f732c0cb800 con 0x7f732c0cf160 2023-10-30T13:22:26.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f73312ff640 1 -- 172.21.15.148:0/1943405181 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7318003530 con 0x7f732c0cf160 2023-10-30T13:22:26.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7331b00640 1 -- 172.21.15.148:0/1943405181 >> v2:172.21.15.148:3300/0 conn(0x7f732c1538a0 msgr2=0x7f732c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:26.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7331b00640 1 --2- 172.21.15.148:0/1943405181 >> v2:172.21.15.148:3300/0 conn(0x7f732c1538a0 0x7f732c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:26.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7331b00640 1 -- 172.21.15.148:0/1943405181 >> v1:172.21.15.150:6789/0 conn(0x7f732c0cf160 legacy=0x7f732c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:26.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.382+0000 7f7331b00640 1 -- 172.21.15.148:0/1943405181 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c0cad70 con 0x7f732c0ca0e0 2023-10-30T13:22:26.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.386+0000 7f7331b00640 1 --2- 172.21.15.148:0/1943405181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0ca0e0 0x7f732c15a090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f731c0097c0 tx=0x7f731c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57e8956e6db41e36 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.386+0000 7f73312ff640 1 -- 172.21.15.148:0/1943405181 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c0a8970 con 0x7f732c0ca0e0 2023-10-30T13:22:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.386+0000 7f73312ff640 1 -- 172.21.15.148:0/1943405181 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c0a8ad0 con 0x7f732c0ca0e0 2023-10-30T13:22:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.386+0000 7f73312ff640 1 -- 172.21.15.148:0/1943405181 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c0a8e60 con 0x7f732c0ca0e0 2023-10-30T13:22:26.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.386+0000 7f7334565640 1 -- 172.21.15.148:0/1943405181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0ca0e0 msgr2=0x7f732c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:26.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.386+0000 7f7334565640 1 --2- 172.21.15.148:0/1943405181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0ca0e0 0x7f732c15a090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f731c0097c0 tx=0x7f731c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:22:26.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.390+0000 7f7334565640 1 -- 172.21.15.148:0/1943405181 shutdown_connections 2023-10-30T13:22:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.390+0000 7f7334565640 1 --2- 172.21.15.148:0/1943405181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0ca0e0 0x7f732c15a090 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.390+0000 7f7334565640 1 --2- 172.21.15.148:0/1943405181 >> v2:172.21.15.148:3300/0 conn(0x7f732c1538a0 0x7f732c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.390+0000 7f7334565640 1 -- 172.21.15.148:0/1943405181 >> 172.21.15.148:0/1943405181 conn(0x7f732c06bfe0 msgr2=0x7f732c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.390+0000 7f7334565640 1 -- 172.21.15.148:0/1943405181 shutdown_connections 2023-10-30T13:22:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.390+0000 7f7334565640 1 -- 172.21.15.148:0/1943405181 wait complete. 2023-10-30T13:22:26.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.390+0000 7f7334565640 1 Processor -- start 2023-10-30T13:22:26.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.390+0000 7f7334565640 1 -- start start 2023-10-30T13:22:26.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7334565640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732c0ca0e0 0x7f732c0b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:26.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7334565640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0cf160 0x7f732c15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:26.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7334565640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f732c1538a0 0x7f732c15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:26.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7334565640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c14f4c0 con 0x7f732c1538a0 2023-10-30T13:22:26.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7334565640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c1eefc0 con 0x7f732c0ca0e0 2023-10-30T13:22:26.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7334565640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732c15a8b0 con 0x7f732c0cf160 2023-10-30T13:22:26.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7331b00640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0cf160 0x7f732c15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:26.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7332301640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732c0ca0e0 0x7f732c0b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:26.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7331b00640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0cf160 0x7f732c15c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:59152/0 (socket says 172.21.15.148:59152) 2023-10-30T13:22:26.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7331b00640 1 -- 172.21.15.148:0/1408243787 learned_addr learned my addr 172.21.15.148:0/1408243787 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:26.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7332b02640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f732c1538a0 0x7f732c15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:26.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7332b02640 1 -- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0cf160 msgr2=0x7f732c15c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:26.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7332b02640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0cf160 0x7f732c15c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:26.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7332b02640 1 -- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732c0ca0e0 msgr2=0x7f732c0b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:26.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7332b02640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732c0ca0e0 0x7f732c0b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:26.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7332b02640 1 -- 172.21.15.148:0/1408243787 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c009420 con 0x7f732c1538a0 2023-10-30T13:22:26.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7332b02640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f732c1538a0 0x7f732c15cd40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f732400caa0 tx=0x7f732400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:26.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7322ffd640 1 -- 172.21.15.148:0/1408243787 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732400ab40 con 0x7f732c1538a0 2023-10-30T13:22:26.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7322ffd640 1 -- 172.21.15.148:0/1408243787 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732400aca0 con 0x7f732c1538a0 2023-10-30T13:22:26.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7322ffd640 1 -- 172.21.15.148:0/1408243787 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324012600 con 0x7f732c1538a0 2023-10-30T13:22:26.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732c1ef1d0 con 0x7f732c1538a0 2023-10-30T13:22:26.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732c15ade0 con 0x7f732c1538a0 2023-10-30T13:22:26.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.394+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732c0b2a30 con 0x7f732c1538a0 2023-10-30T13:22:26.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.398+0000 7f7322ffd640 1 -- 172.21.15.148:0/1408243787 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7324007070 con 0x7f732c1538a0 2023-10-30T13:22:26.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.402+0000 7f7322ffd640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7310066570 0x7f7310068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:26.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.402+0000 7f7322ffd640 1 -- 172.21.15.148:0/1408243787 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6226+0+0 (secure 0 0 0) 0x7f7324084340 con 0x7f732c1538a0 2023-10-30T13:22:26.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.402+0000 7f7332301640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7310066570 0x7f7310068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:26.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.402+0000 7f7332301640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7310066570 0x7f7310068a30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7328009ac0 tx=0x7f7328009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:26.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.406+0000 7f7322ffd640 1 -- 172.21.15.148:0/1408243787 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73240516d0 con 0x7f732c1538a0 2023-10-30T13:22:26.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.782+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f732c0cfdd0 con 0x7f732c1538a0 2023-10-30T13:22:26.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.782+0000 7f7322ffd640 1 -- 172.21.15.148:0/1408243787 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f732405b070 con 0x7f732c1538a0 2023-10-30T13:22:26.784 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:26.784 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:02.735218+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:26.785 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 58 2023-10-30T13:22:26.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7310066570 msgr2=0x7f7310068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:26.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7310066570 0x7f7310068a30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7328009ac0 tx=0x7f7328009190 comp rx=0 tx=0).stop 2023-10-30T13:22:26.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f732c1538a0 msgr2=0x7f732c15cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:26.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f732c1538a0 0x7f732c15cd40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f732400caa0 tx=0x7f732400cf60 comp rx=0 tx=0).stop 2023-10-30T13:22:26.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 shutdown_connections 2023-10-30T13:22:26.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7310066570 0x7f7310068a30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:26.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f732c1538a0 0x7f732c15cd40 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:26.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f732c0cf160 0x7f732c15c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:26.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 --2- 172.21.15.148:0/1408243787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732c0ca0e0 0x7f732c0b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:26.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 >> 172.21.15.148:0/1408243787 conn(0x7f732c06bfe0 msgr2=0x7f732c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:26.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 shutdown_connections 2023-10-30T13:22:26.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:26.786+0000 7f7334565640 1 -- 172.21.15.148:0/1408243787 wait complete. 2023-10-30T13:22:26.817 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:22:28.818 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:29.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.050+0000 7f03ed33b640 1 Processor -- start 2023-10-30T13:22:29.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.050+0000 7f03ed33b640 1 -- start start 2023-10-30T13:22:29.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03ed33b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e81538f0 0x7f03e8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:29.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03ed33b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f03e815cd50 0x7f03e80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:29.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03ed33b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e815d300 con 0x7f03e80cefa0 2023-10-30T13:22:29.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03ed33b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e815d5e0 con 0x7f03e815cd50 2023-10-30T13:22:29.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03ed33b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e815d8c0 con 0x7f03e81538f0 2023-10-30T13:22:29.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03ecb3a640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f03e80cefa0 0x7f03e80cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47184/0 (socket says 172.21.15.148:47184) 2023-10-30T13:22:29.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03ecb3a640 1 -- 172.21.15.148:0/4249661687 learned_addr learned my addr 172.21.15.148:0/4249661687 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:22:29.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03e77fe640 1 --2- 172.21.15.148:0/4249661687 >> v2:172.21.15.148:3300/0 conn(0x7f03e815cd50 0x7f03e80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:29.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03e7fff640 1 --2- 172.21.15.148:0/4249661687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e81538f0 0x7f03e8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:29.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03e77fe640 1 -- 172.21.15.148:0/4249661687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e81538f0 msgr2=0x7f03e8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:29.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03e77fe640 1 --2- 172.21.15.148:0/4249661687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e81538f0 0x7f03e8159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03e77fe640 1 -- 172.21.15.148:0/4249661687 >> v1:172.21.15.148:6789/0 conn(0x7f03e80cefa0 legacy=0x7f03e80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:29.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.054+0000 7f03e77fe640 1 -- 172.21.15.148:0/4249661687 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e815dba0 con 0x7f03e815cd50 2023-10-30T13:22:29.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.058+0000 7f03e77fe640 1 --2- 172.21.15.148:0/4249661687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e815cd50 0x7f03e80c9f90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0097c0 tx=0x7f03dc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21d3f6993eaa899 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:29.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.058+0000 7f03e6ffd640 1 -- 172.21.15.148:0/4249661687 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0ab600 con 0x7f03e815cd50 2023-10-30T13:22:29.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.058+0000 7f03e6ffd640 1 -- 172.21.15.148:0/4249661687 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc0a7050 con 0x7f03e815cd50 2023-10-30T13:22:29.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.058+0000 7f03e6ffd640 1 -- 172.21.15.148:0/4249661687 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0ab980 con 0x7f03e815cd50 2023-10-30T13:22:29.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 -- 172.21.15.148:0/4249661687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e815cd50 msgr2=0x7f03e80c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:29.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 --2- 172.21.15.148:0/4249661687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e815cd50 0x7f03e80c9f90 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0097c0 tx=0x7f03dc01c3a0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 -- 172.21.15.148:0/4249661687 shutdown_connections 2023-10-30T13:22:29.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 --2- 172.21.15.148:0/4249661687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e815cd50 0x7f03e80c9f90 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 --2- 172.21.15.148:0/4249661687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e81538f0 0x7f03e8159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 -- 172.21.15.148:0/4249661687 >> 172.21.15.148:0/4249661687 conn(0x7f03e806a640 msgr2=0x7f03e8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:29.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 -- 172.21.15.148:0/4249661687 shutdown_connections 2023-10-30T13:22:29.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 -- 172.21.15.148:0/4249661687 wait complete. 2023-10-30T13:22:29.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 Processor -- start 2023-10-30T13:22:29.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.062+0000 7f03ed33b640 1 -- start start 2023-10-30T13:22:29.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ed33b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e80cefa0 0x7f03e814fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:29.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ed33b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e81538f0 0x7f03e815b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:29.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ed33b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e815cd50 0x7f03e815b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:29.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ed33b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e815afa0 con 0x7f03e81538f0 2023-10-30T13:22:29.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ed33b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e80b5000 con 0x7f03e80cefa0 2023-10-30T13:22:29.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ed33b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e8146560 con 0x7f03e815cd50 2023-10-30T13:22:29.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ecb3a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e815cd50 0x7f03e815b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:29.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e81538f0 0x7f03e815b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:29.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ecb3a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e815cd50 0x7f03e815b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:59168/0 (socket says 172.21.15.148:59168) 2023-10-30T13:22:29.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ecb3a640 1 -- 172.21.15.148:0/339859871 learned_addr learned my addr 172.21.15.148:0/339859871 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:29.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e7fff640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e80cefa0 0x7f03e814fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:29.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e77fe640 1 -- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e815cd50 msgr2=0x7f03e815b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:29.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e77fe640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e815cd50 0x7f03e815b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e77fe640 1 -- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e80cefa0 msgr2=0x7f03e814fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:29.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e77fe640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e80cefa0 0x7f03e814fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e77fe640 1 -- 172.21.15.148:0/339859871 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc009420 con 0x7f03e81538f0 2023-10-30T13:22:29.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03ecb3a640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e815cd50 0x7f03e815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:29.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e77fe640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e81538f0 0x7f03e815b3c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0ab000 tx=0x7f03dc019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:29.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e4ff9640 1 -- 172.21.15.148:0/339859871 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0b4810 con 0x7f03e81538f0 2023-10-30T13:22:29.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e4ff9640 1 -- 172.21.15.148:0/339859871 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc0b49b0 con 0x7f03e81538f0 2023-10-30T13:22:29.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.066+0000 7f03e4ff9640 1 -- 172.21.15.148:0/339859871 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0b4d60 con 0x7f03e81538f0 2023-10-30T13:22:29.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.070+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e81f3610 con 0x7f03e81538f0 2023-10-30T13:22:29.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.070+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e80b4720 con 0x7f03e81538f0 2023-10-30T13:22:29.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.070+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e80b4910 con 0x7f03e81538f0 2023-10-30T13:22:29.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.070+0000 7f03e4ff9640 1 -- 172.21.15.148:0/339859871 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03dc0b2040 con 0x7f03e81538f0 2023-10-30T13:22:29.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.074+0000 7f03e4ff9640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03d40664a0 0x7f03d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:29.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.074+0000 7f03e4ff9640 1 -- 172.21.15.148:0/339859871 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6226+0+0 (secure 0 0 0) 0x7f03dc11d920 con 0x7f03e81538f0 2023-10-30T13:22:29.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.074+0000 7f03e7fff640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03d40664a0 0x7f03d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:29.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.074+0000 7f03e7fff640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03d40664a0 0x7f03d4068960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009b60 tx=0x7f03d8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:29.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.082+0000 7f03e4ff9640 1 -- 172.21.15.148:0/339859871 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03dc0eaba0 con 0x7f03e81538f0 2023-10-30T13:22:29.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.462+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03e804e170 con 0x7f03e81538f0 2023-10-30T13:22:29.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.462+0000 7f03e4ff9640 1 -- 172.21.15.148:0/339859871 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f03dc0ee1c0 con 0x7f03e81538f0 2023-10-30T13:22:29.467 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:29.467 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:02.735218+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:29.468 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 58 2023-10-30T13:22:29.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03d40664a0 msgr2=0x7f03d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:29.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03d40664a0 0x7f03d4068960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009b60 tx=0x7f03d8009190 comp rx=0 tx=0).stop 2023-10-30T13:22:29.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e81538f0 msgr2=0x7f03e815b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:29.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e81538f0 0x7f03e815b3c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0ab000 tx=0x7f03dc019140 comp rx=0 tx=0).stop 2023-10-30T13:22:29.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 shutdown_connections 2023-10-30T13:22:29.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03d40664a0 0x7f03d4068960 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03e81538f0 0x7f03e815b3c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f03e815cd50 0x7f03e815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 --2- 172.21.15.148:0/339859871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e80cefa0 0x7f03e814fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:29.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 >> 172.21.15.148:0/339859871 conn(0x7f03e806a640 msgr2=0x7f03e80c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:29.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 shutdown_connections 2023-10-30T13:22:29.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:29.466+0000 7f03ed33b640 1 -- 172.21.15.148:0/339859871 wait complete. 2023-10-30T13:22:29.495 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:22:31.496 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:31.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.730+0000 7fac08446640 1 Processor -- start 2023-10-30T13:22:31.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.730+0000 7fac08446640 1 -- start start 2023-10-30T13:22:31.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.730+0000 7fac08446640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fac001538a0 0x7fac0015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:31.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac08446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac000cf160 0x7fac000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:31.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac08446640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac000cb800 con 0x7fac000ca0e0 2023-10-30T13:22:31.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac08446640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0015dc30 con 0x7fac001538a0 2023-10-30T13:22:31.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac08446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac000cb330 con 0x7fac000cf160 2023-10-30T13:22:31.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac061e2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fac001538a0 0x7fac0015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:31.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac059e1640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fac000ca0e0 0x7fac0015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47200/0 (socket says 172.21.15.148:47200) 2023-10-30T13:22:31.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac059e1640 1 -- 172.21.15.148:0/2314497669 learned_addr learned my addr 172.21.15.148:0/2314497669 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:22:31.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac069e3640 1 --2- 172.21.15.148:0/2314497669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac000cf160 0x7fac000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:31.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac051e0640 1 -- 172.21.15.148:0/2314497669 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701148927 0 0) 0x7fac000cb800 con 0x7fac000ca0e0 2023-10-30T13:22:31.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac051e0640 1 -- 172.21.15.148:0/2314497669 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabec003530 con 0x7fac000ca0e0 2023-10-30T13:22:31.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac061e2640 1 -- 172.21.15.148:0/2314497669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac000cf160 msgr2=0x7fac000cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:31.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac061e2640 1 --2- 172.21.15.148:0/2314497669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac000cf160 0x7fac000cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:31.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac061e2640 1 -- 172.21.15.148:0/2314497669 >> v1:172.21.15.148:6789/0 conn(0x7fac000ca0e0 legacy=0x7fac0015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:31.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.734+0000 7fac061e2640 1 -- 172.21.15.148:0/2314497669 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac000cad70 con 0x7fac001538a0 2023-10-30T13:22:31.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac061e2640 1 --2- 172.21.15.148:0/2314497669 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac001538a0 0x7fac0015d6f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0097c0 tx=0x7fabfc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540ea89bccb832 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:31.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac051e0640 1 -- 172.21.15.148:0/2314497669 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0a8970 con 0x7fac001538a0 2023-10-30T13:22:31.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac051e0640 1 -- 172.21.15.148:0/2314497669 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc0a8ad0 con 0x7fac001538a0 2023-10-30T13:22:31.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac051e0640 1 -- 172.21.15.148:0/2314497669 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0a8e60 con 0x7fac001538a0 2023-10-30T13:22:31.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac08446640 1 -- 172.21.15.148:0/2314497669 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac001538a0 msgr2=0x7fac0015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:31.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac08446640 1 --2- 172.21.15.148:0/2314497669 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac001538a0 0x7fac0015d6f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0097c0 tx=0x7fabfc01ec70 comp rx=0 tx=0).stop 2023-10-30T13:22:31.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac08446640 1 -- 172.21.15.148:0/2314497669 shutdown_connections 2023-10-30T13:22:31.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac08446640 1 --2- 172.21.15.148:0/2314497669 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac000cf160 0x7fac000cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:31.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac08446640 1 --2- 172.21.15.148:0/2314497669 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac001538a0 0x7fac0015d6f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:31.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.738+0000 7fac08446640 1 -- 172.21.15.148:0/2314497669 >> 172.21.15.148:0/2314497669 conn(0x7fac0006bfe0 msgr2=0x7fac0006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:31.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.742+0000 7fac08446640 1 -- 172.21.15.148:0/2314497669 shutdown_connections 2023-10-30T13:22:31.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.742+0000 7fac08446640 1 -- 172.21.15.148:0/2314497669 wait complete. 2023-10-30T13:22:31.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.742+0000 7fac08446640 1 Processor -- start 2023-10-30T13:22:31.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.742+0000 7fac08446640 1 -- start start 2023-10-30T13:22:31.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac08446640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac000ca0e0 0x7fac000b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:31.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac08446640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac000cf160 0x7fac00149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:31.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac08446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac001538a0 0x7fac001497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:31.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac08446640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0015cfb0 con 0x7fac000ca0e0 2023-10-30T13:22:31.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac08446640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac001f3460 con 0x7fac000cf160 2023-10-30T13:22:31.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac08446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac001f3740 con 0x7fac001538a0 2023-10-30T13:22:31.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac061e2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac000ca0e0 0x7fac000b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:31.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac061e2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac000ca0e0 0x7fac000b48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35278/0 (socket says 172.21.15.148:35278) 2023-10-30T13:22:31.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac061e2640 1 -- 172.21.15.148:0/2943698510 learned_addr learned my addr 172.21.15.148:0/2943698510 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:31.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac069e3640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac001538a0 0x7fac001497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:31.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac059e1640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac000cf160 0x7fac00149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:31.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac061e2640 1 -- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac001538a0 msgr2=0x7fac001497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:31.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac061e2640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac001538a0 0x7fac001497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:31.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac061e2640 1 -- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac000cf160 msgr2=0x7fac00149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:31.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac061e2640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac000cf160 0x7fac00149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:31.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac061e2640 1 -- 172.21.15.148:0/2943698510 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc009420 con 0x7fac000ca0e0 2023-10-30T13:22:31.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac061e2640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac000ca0e0 0x7fac000b48d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0b3040 tx=0x7fabfc01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:31.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fabf6ffd640 1 -- 172.21.15.148:0/2943698510 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0b0d60 con 0x7fac000ca0e0 2023-10-30T13:22:31.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fabf6ffd640 1 -- 172.21.15.148:0/2943698510 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc0b9430 con 0x7fac000ca0e0 2023-10-30T13:22:31.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fabf6ffd640 1 -- 172.21.15.148:0/2943698510 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0b97e0 con 0x7fac000ca0e0 2023-10-30T13:22:31.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac001f3a20 con 0x7fac000ca0e0 2023-10-30T13:22:31.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac001f3fe0 con 0x7fac000ca0e0 2023-10-30T13:22:31.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.746+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac000b2a50 con 0x7fac000ca0e0 2023-10-30T13:22:31.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.750+0000 7fabf6ffd640 1 -- 172.21.15.148:0/2943698510 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fabfc0b7040 con 0x7fac000ca0e0 2023-10-30T13:22:31.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.750+0000 7fabf6ffd640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabe40664f0 0x7fabe40689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:31.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.750+0000 7fabf6ffd640 1 -- 172.21.15.148:0/2943698510 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6226+0+0 (secure 0 0 0) 0x7fabfc121aa0 con 0x7fac000ca0e0 2023-10-30T13:22:31.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.750+0000 7fac059e1640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabe40664f0 0x7fabe40689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:31.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.754+0000 7fac059e1640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabe40664f0 0x7fabe40689b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fac00151e70 tx=0x7fabf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:31.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:31.758+0000 7fabf6ffd640 1 -- 172.21.15.148:0/2943698510 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fabfc0eed80 con 0x7fac000ca0e0 2023-10-30T13:22:32.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.133+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac0004e170 con 0x7fac000ca0e0 2023-10-30T13:22:32.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.133+0000 7fabf6ffd640 1 -- 172.21.15.148:0/2943698510 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fabfc0b5020 con 0x7fac000ca0e0 2023-10-30T13:22:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:32.136 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:02.735218+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:32.137 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 58 2023-10-30T13:22:32.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabe40664f0 msgr2=0x7fabe40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:32.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabe40664f0 0x7fabe40689b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fac00151e70 tx=0x7fabf0009040 comp rx=0 tx=0).stop 2023-10-30T13:22:32.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac000ca0e0 msgr2=0x7fac000b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:32.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac000ca0e0 0x7fac000b48d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0b3040 tx=0x7fabfc01ba10 comp rx=0 tx=0).stop 2023-10-30T13:22:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 shutdown_connections 2023-10-30T13:22:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fabe40664f0 0x7fabe40689b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac001538a0 0x7fac001497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac000cf160 0x7fac00149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 --2- 172.21.15.148:0/2943698510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac000ca0e0 0x7fac000b48d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 >> 172.21.15.148:0/2943698510 conn(0x7fac0006bfe0 msgr2=0x7fac00159090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:32.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 shutdown_connections 2023-10-30T13:22:32.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:32.137+0000 7fac08446640 1 -- 172.21.15.148:0/2943698510 wait complete. 2023-10-30T13:22:32.170 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:22:34.171 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:34.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.405+0000 7f070d365640 1 Processor -- start 2023-10-30T13:22:34.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.405+0000 7f070d365640 1 -- start start 2023-10-30T13:22:34.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f070d365640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f07080cc5d0 0x7f07081652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:34.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f070d365640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070806ccf0 con 0x7f0708153d60 2023-10-30T13:22:34.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f070d365640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070815a540 con 0x7f07080ccd30 2023-10-30T13:22:34.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f070d365640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07080cb4c0 con 0x7f07080cc5d0 2023-10-30T13:22:34.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f0706ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0708153d60 0x7f07080cd280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51510/0 (socket says 172.21.15.148:51510) 2023-10-30T13:22:34.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f0706ffd640 1 -- 172.21.15.148:0/1580942417 learned_addr learned my addr 172.21.15.148:0/1580942417 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:22:34.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f0707fff640 1 --2- 172.21.15.148:0/1580942417 >> v2:172.21.15.150:3300/0 conn(0x7f07080cc5d0 0x7f07081652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:34.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f0705ffb640 1 -- 172.21.15.148:0/1580942417 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660201536 0 0) 0x7f070806ccf0 con 0x7f0708153d60 2023-10-30T13:22:34.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f0705ffb640 1 -- 172.21.15.148:0/1580942417 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f0003530 con 0x7f0708153d60 2023-10-30T13:22:34.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f0707fff640 1 -- 172.21.15.148:0/1580942417 >> v1:172.21.15.150:6789/0 conn(0x7f07080ccd30 legacy=0x7f070806d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:34.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f0707fff640 1 -- 172.21.15.148:0/1580942417 >> v1:172.21.15.148:6789/0 conn(0x7f0708153d60 legacy=0x7f07080cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:34.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f0705ffb640 1 -- 172.21.15.148:0/1580942417 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782397251 0 0) 0x7f06f0003530 con 0x7f0708153d60 2023-10-30T13:22:34.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.409+0000 7f0707fff640 1 -- 172.21.15.148:0/1580942417 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07080cb7a0 con 0x7f07080cc5d0 2023-10-30T13:22:34.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.413+0000 7f0707fff640 1 --2- 172.21.15.148:0/1580942417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07080cc5d0 0x7f07081652b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0700009dc0 tx=0x7f070001f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8877f585d552a09d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:34.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.413+0000 7f0705ffb640 1 -- 172.21.15.148:0/1580942417 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8001680 con 0x7f07080cc5d0 2023-10-30T13:22:34.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.413+0000 7f0705ffb640 1 -- 172.21.15.148:0/1580942417 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07000a8c10 con 0x7f07080cc5d0 2023-10-30T13:22:34.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.413+0000 7f0705ffb640 1 -- 172.21.15.148:0/1580942417 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07000b0440 con 0x7f07080cc5d0 2023-10-30T13:22:34.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.413+0000 7f070d365640 1 -- 172.21.15.148:0/1580942417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07080cc5d0 msgr2=0x7f07081652b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:34.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.413+0000 7f070d365640 1 --2- 172.21.15.148:0/1580942417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07080cc5d0 0x7f07081652b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0700009dc0 tx=0x7f070001f040 comp rx=0 tx=0).stop 2023-10-30T13:22:34.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.417+0000 7f070d365640 1 -- 172.21.15.148:0/1580942417 shutdown_connections 2023-10-30T13:22:34.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.417+0000 7f070d365640 1 --2- 172.21.15.148:0/1580942417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07080cc5d0 0x7f07081652b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:34.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.417+0000 7f070d365640 1 -- 172.21.15.148:0/1580942417 >> 172.21.15.148:0/1580942417 conn(0x7f070806e840 msgr2=0x7f070806ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:34.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.417+0000 7f070d365640 1 -- 172.21.15.148:0/1580942417 shutdown_connections 2023-10-30T13:22:34.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.417+0000 7f070d365640 1 -- 172.21.15.148:0/1580942417 wait complete. 2023-10-30T13:22:34.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.417+0000 7f070d365640 1 Processor -- start 2023-10-30T13:22:34.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.417+0000 7f070d365640 1 -- start start 2023-10-30T13:22:34.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f070d365640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080cc5d0 0x7f07080ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:34.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f070d365640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07080ccd30 0x7f07081f0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:34.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f070d365640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708153d60 0x7f07081f1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:34.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f070d365640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07080d08d0 con 0x7f07080cc5d0 2023-10-30T13:22:34.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f070d365640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07080cf4c0 con 0x7f0708153d60 2023-10-30T13:22:34.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f070d365640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07081f0880 con 0x7f07080ccd30 2023-10-30T13:22:34.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0706ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080cc5d0 0x7f07080ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:34.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0706ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080cc5d0 0x7f07080ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49408/0 (socket says 172.21.15.148:49408) 2023-10-30T13:22:34.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0706ffd640 1 -- 172.21.15.148:0/3954731886 learned_addr learned my addr 172.21.15.148:0/3954731886 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:34.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f07067fc640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07080ccd30 0x7f07081f0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:34.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0707fff640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708153d60 0x7f07081f1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:34.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0706ffd640 1 -- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07080ccd30 msgr2=0x7f07081f0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:34.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0706ffd640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07080ccd30 0x7f07081f0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:34.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0706ffd640 1 -- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708153d60 msgr2=0x7f07081f1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:34.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0706ffd640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708153d60 0x7f07081f1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:34.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0706ffd640 1 -- 172.21.15.148:0/3954731886 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700009a70 con 0x7f07080cc5d0 2023-10-30T13:22:34.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f0706ffd640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080cc5d0 0x7f07080ce8e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f06f800c780 tx=0x7f06f800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:34.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f06ef7fe640 1 -- 172.21.15.148:0/3954731886 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8005ae0 con 0x7f07080cc5d0 2023-10-30T13:22:34.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f06ef7fe640 1 -- 172.21.15.148:0/3954731886 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f8005c40 con 0x7f07080cc5d0 2023-10-30T13:22:34.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07080d1440 con 0x7f07080cc5d0 2023-10-30T13:22:34.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f06ef7fe640 1 -- 172.21.15.148:0/3954731886 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8007070 con 0x7f07080cc5d0 2023-10-30T13:22:34.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.421+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07081eff50 con 0x7f07080cc5d0 2023-10-30T13:22:34.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.425+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070804e170 con 0x7f07080cc5d0 2023-10-30T13:22:34.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.425+0000 7f06ef7fe640 1 -- 172.21.15.148:0/3954731886 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06f80071d0 con 0x7f07080cc5d0 2023-10-30T13:22:34.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.425+0000 7f06ef7fe640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06f40664a0 0x7f06f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:34.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.425+0000 7f06ef7fe640 1 -- 172.21.15.148:0/3954731886 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6226+0+0 (secure 0 0 0) 0x7f06f8083e10 con 0x7f07080cc5d0 2023-10-30T13:22:34.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.429+0000 7f07067fc640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06f40664a0 0x7f06f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:34.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.433+0000 7f07067fc640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06f40664a0 0x7f06f4068960 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f07081efa70 tx=0x7f06fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:34.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.433+0000 7f06ef7fe640 1 -- 172.21.15.148:0/3954731886 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06f8051130 con 0x7f07080cc5d0 2023-10-30T13:22:34.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.813+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07080cbfd0 con 0x7f07080cc5d0 2023-10-30T13:22:34.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.813+0000 7f06ef7fe640 1 -- 172.21.15.148:0/3954731886 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f06f8054750 con 0x7f07080cc5d0 2023-10-30T13:22:34.814 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:34.814 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:02.735218+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:34.815 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 58 2023-10-30T13:22:34.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06f40664a0 msgr2=0x7f06f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:34.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06f40664a0 0x7f06f4068960 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f07081efa70 tx=0x7f06fc009040 comp rx=0 tx=0).stop 2023-10-30T13:22:34.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080cc5d0 msgr2=0x7f07080ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:34.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080cc5d0 0x7f07080ce8e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f06f800c780 tx=0x7f06f800cc40 comp rx=0 tx=0).stop 2023-10-30T13:22:34.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 shutdown_connections 2023-10-30T13:22:34.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06f40664a0 0x7f06f4068960 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:34.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708153d60 0x7f07081f1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:34.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07080ccd30 0x7f07081f0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:34.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 --2- 172.21.15.148:0/3954731886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080cc5d0 0x7f07080ce8e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:34.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 >> 172.21.15.148:0/3954731886 conn(0x7f070806e840 msgr2=0x7f07081529b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:34.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 shutdown_connections 2023-10-30T13:22:34.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:34.817+0000 7f070d365640 1 -- 172.21.15.148:0/3954731886 wait complete. 2023-10-30T13:22:34.847 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:22:36.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:36.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa66d91d640 1 Processor -- start 2023-10-30T13:22:36.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa66d91d640 1 -- start start 2023-10-30T13:22:36.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa66d91d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 0x7fa66806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:36.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa66d91d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa668071110 0x7fa668158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:37.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa66d91d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa668066cf0 con 0x7fa66806f7a0 2023-10-30T13:22:37.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa66d91d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6680700c0 con 0x7fa668071110 2023-10-30T13:22:37.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa66d91d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6680c70d0 con 0x7fa6680c6bd0 2023-10-30T13:22:37.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa6677fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa668071110 0x7fa668158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:37.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa65e7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 0x7fa66806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:37.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa6677fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa668071110 0x7fa668158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:39906/0 (socket says 172.21.15.148:39906) 2023-10-30T13:22:37.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa65e7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 0x7fa66806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:54160/0 (socket says 172.21.15.148:54160) 2023-10-30T13:22:37.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa6677fe640 1 -- 172.21.15.148:0/1741714355 learned_addr learned my addr 172.21.15.148:0/1741714355 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:37.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.001+0000 7fa666ffd640 1 --1- 172.21.15.148:0/1741714355 >> v1:172.21.15.150:6789/0 conn(0x7fa66806f7a0 0x7fa668069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:32962/0 (socket says 172.21.15.148:32962) 2023-10-30T13:22:37.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa6677fe640 1 -- 172.21.15.148:0/1741714355 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 msgr2=0x7fa66806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:37.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa6677fe640 1 --2- 172.21.15.148:0/1741714355 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 0x7fa66806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:37.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa6677fe640 1 -- 172.21.15.148:0/1741714355 >> v1:172.21.15.150:6789/0 conn(0x7fa66806f7a0 legacy=0x7fa668069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:37.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa6677fe640 1 -- 172.21.15.148:0/1741714355 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6680c7390 con 0x7fa668071110 2023-10-30T13:22:37.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa6667fc640 1 -- 172.21.15.148:0/1741714355 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373432420 0 0) 0x7fa668066cf0 con 0x7fa66806f7a0 2023-10-30T13:22:37.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa6677fe640 1 --2- 172.21.15.148:0/1741714355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa668071110 0x7fa668158110 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa660009870 tx=0x7fa66001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a20379d49c7718b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:37.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa6667fc640 1 -- 172.21.15.148:0/1741714355 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6600a8a30 con 0x7fa668071110 2023-10-30T13:22:37.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa6667fc640 1 -- 172.21.15.148:0/1741714355 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6600a8b90 con 0x7fa668071110 2023-10-30T13:22:37.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa6667fc640 1 -- 172.21.15.148:0/1741714355 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6600b0440 con 0x7fa668071110 2023-10-30T13:22:37.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa66d91d640 1 -- 172.21.15.148:0/1741714355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa668071110 msgr2=0x7fa668158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:37.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.005+0000 7fa66d91d640 1 --2- 172.21.15.148:0/1741714355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa668071110 0x7fa668158110 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa660009870 tx=0x7fa66001ed30 comp rx=0 tx=0).stop 2023-10-30T13:22:37.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 -- 172.21.15.148:0/1741714355 shutdown_connections 2023-10-30T13:22:37.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 --2- 172.21.15.148:0/1741714355 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 0x7fa66806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:37.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 --2- 172.21.15.148:0/1741714355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa668071110 0x7fa668158110 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:37.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 -- 172.21.15.148:0/1741714355 >> 172.21.15.148:0/1741714355 conn(0x7fa6680c7910 msgr2=0x7fa6680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:37.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 -- 172.21.15.148:0/1741714355 shutdown_connections 2023-10-30T13:22:37.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 -- 172.21.15.148:0/1741714355 wait complete. 2023-10-30T13:22:37.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 Processor -- start 2023-10-30T13:22:37.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 -- start start 2023-10-30T13:22:37.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa66806f7a0 0x7fa668157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:37.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa668071110 0x7fa668153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:37.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 0x7fa668151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:37.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa668151700 con 0x7fa66806f7a0 2023-10-30T13:22:37.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa668151090 con 0x7fa668071110 2023-10-30T13:22:37.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa66d91d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6681ec870 con 0x7fa6680c6bd0 2023-10-30T13:22:37.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa666ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa66806f7a0 0x7fa668157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:37.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa666ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa66806f7a0 0x7fa668157980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49416/0 (socket says 172.21.15.148:49416) 2023-10-30T13:22:37.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa666ffd640 1 -- 172.21.15.148:0/246259512 learned_addr learned my addr 172.21.15.148:0/246259512 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:37.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa6677fe640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 0x7fa668151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:37.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa65e7fc640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa668071110 0x7fa668153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:37.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa666ffd640 1 -- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 msgr2=0x7fa668151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:37.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa666ffd640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 0x7fa668151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:37.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa666ffd640 1 -- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa668071110 msgr2=0x7fa668153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:37.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa666ffd640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa668071110 0x7fa668153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:37.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa666ffd640 1 -- 172.21.15.148:0/246259512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6600094d0 con 0x7fa66806f7a0 2023-10-30T13:22:37.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.009+0000 7fa666ffd640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa66806f7a0 0x7fa668157980 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa65800c4b0 tx=0x7fa65800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:37.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa664ff9640 1 -- 172.21.15.148:0/246259512 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6580074e0 con 0x7fa66806f7a0 2023-10-30T13:22:37.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa664ff9640 1 -- 172.21.15.148:0/246259512 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa658007660 con 0x7fa66806f7a0 2023-10-30T13:22:37.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa664ff9640 1 -- 172.21.15.148:0/246259512 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa658004a30 con 0x7fa66806f7a0 2023-10-30T13:22:37.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6681ecaf0 con 0x7fa66806f7a0 2023-10-30T13:22:37.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6681ed0b0 con 0x7fa66806f7a0 2023-10-30T13:22:37.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66804e170 con 0x7fa66806f7a0 2023-10-30T13:22:37.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa664ff9640 1 -- 172.21.15.148:0/246259512 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa658004b90 con 0x7fa66806f7a0 2023-10-30T13:22:37.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa664ff9640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6480664a0 0x7fa648068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:37.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa664ff9640 1 -- 172.21.15.148:0/246259512 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6226+0+0 (secure 0 0 0) 0x7fa658083220 con 0x7fa66806f7a0 2023-10-30T13:22:37.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.013+0000 7fa65e7fc640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6480664a0 0x7fa648068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:37.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.017+0000 7fa65e7fc640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6480664a0 0x7fa648068960 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa6681581d0 tx=0x7fa65000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:37.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.017+0000 7fa664ff9640 1 -- 172.21.15.148:0/246259512 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa658050410 con 0x7fa66806f7a0 2023-10-30T13:22:37.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.197+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6680c66d0 con 0x7fa66806f7a0 2023-10-30T13:22:37.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.197+0000 7fa664ff9640 1 -- 172.21.15.148:0/246259512 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fa658018090 con 0x7fa66806f7a0 2023-10-30T13:22:37.195 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:37.195 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":5694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:02.735218+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5033},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5033":{"gid":5033,"name":"d","rank":0,"incarnation":36,"state":"up:active","state_seq":29,"addr":"172.21.15.150:6836/953984429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":953984429},{"type":"v1","addr":"172.21.15.150:6836","nonce":953984429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:37.195 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 58 2023-10-30T13:22:37.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6480664a0 msgr2=0x7fa648068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:37.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6480664a0 0x7fa648068960 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa6681581d0 tx=0x7fa65000b040 comp rx=0 tx=0).stop 2023-10-30T13:22:37.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa66806f7a0 msgr2=0x7fa668157980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:37.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa66806f7a0 0x7fa668157980 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa65800c4b0 tx=0x7fa65800c970 comp rx=0 tx=0).stop 2023-10-30T13:22:37.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 shutdown_connections 2023-10-30T13:22:37.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6480664a0 0x7fa648068960 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:37.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6680c6bd0 0x7fa668151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:37.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa668071110 0x7fa668153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:37.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 --2- 172.21.15.148:0/246259512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa66806f7a0 0x7fa668157980 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:37.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 >> 172.21.15.148:0/246259512 conn(0x7fa6680c7910 msgr2=0x7fa6680ca660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:37.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 shutdown_connections 2023-10-30T13:22:37.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:37.201+0000 7fa66d91d640 1 -- 172.21.15.148:0/246259512 wait complete. 2023-10-30T13:22:37.217 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:22:39.219 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:39.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53acfa7640 1 Processor -- start 2023-10-30T13:22:39.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53acfa7640 1 -- start start 2023-10-30T13:22:39.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53acfa7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53a80cefa0 0x7f53a80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53acfa7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a815a6c0 con 0x7f53a815cd50 2023-10-30T13:22:39.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53acfa7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a80ca6a0 con 0x7f53a81538f0 2023-10-30T13:22:39.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53acfa7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a815d300 con 0x7f53a80cefa0 2023-10-30T13:22:39.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f539ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f53a815cd50 0x7f53a80c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51514/0 (socket says 172.21.15.148:51514) 2023-10-30T13:22:39.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f539ffff640 1 -- 172.21.15.148:0/3477458942 learned_addr learned my addr 172.21.15.148:0/3477458942 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:22:39.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a7fff640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53a80cefa0 0x7f53a80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228431992 0 0) 0x7f53a815a6c0 con 0x7f53a815cd50 2023-10-30T13:22:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5394003530 con 0x7f53a815cd50 2023-10-30T13:22:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2448419677 0 0) 0x7f5394003530 con 0x7f53a815cd50 2023-10-30T13:22:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53a80cefa0 msgr2=0x7f53a80cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53a80cefa0 0x7f53a80cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 >> v1:172.21.15.150:6789/0 conn(0x7f53a81538f0 legacy=0x7f53a8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a815d5e0 con 0x7f53a815cd50 2023-10-30T13:22:39.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f5390001680 con 0x7f53a815cd50 2023-10-30T13:22:39.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 >> v1:172.21.15.148:6789/0 conn(0x7f53a815cd50 legacy=0x7f53a80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53940039b0 0x7f5394003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53a80cefa0 0x7f53940047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5394004d10 0x7f5394005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a815d300 con 0x7f53940039b0 2023-10-30T13:22:39.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539400b990 con 0x7f5394004d10 2023-10-30T13:22:39.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539400bb50 con 0x7f53a80cefa0 2023-10-30T13:22:39.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a77fe640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53940039b0 0x7f5394003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a7fff640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5394004d10 0x7f5394005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f539ffff640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53a80cefa0 0x7f53940047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a77fe640 1 -- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53a80cefa0 msgr2=0x7f53940047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:39.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a77fe640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53a80cefa0 0x7f53940047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a77fe640 1 -- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5394004d10 msgr2=0x7f5394005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:39.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a77fe640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5394004d10 0x7f5394005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.373+0000 7f53a77fe640 1 -- 172.21.15.148:0/3477458942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5394003530 con 0x7f53940039b0 2023-10-30T13:22:39.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53a77fe640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53940039b0 0x7f5394003d90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5398001910 tx=0x7f539801ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4742c441c3f648a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:39.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53980a8830 con 0x7f53940039b0 2023-10-30T13:22:39.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53980a89b0 con 0x7f53940039b0 2023-10-30T13:22:39.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53a6ffd640 1 -- 172.21.15.148:0/3477458942 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53980a8d60 con 0x7f53940039b0 2023-10-30T13:22:39.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53acfa7640 1 -- 172.21.15.148:0/3477458942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53940039b0 msgr2=0x7f5394003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53acfa7640 1 --2- 172.21.15.148:0/3477458942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53940039b0 0x7f5394003d90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5398001910 tx=0x7f539801ec10 comp rx=0 tx=0).stop 2023-10-30T13:22:39.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53a7fff640 1 -- 172.21.15.148:0/3477458942 reap_dead start 2023-10-30T13:22:39.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53acfa7640 1 -- 172.21.15.148:0/3477458942 shutdown_connections 2023-10-30T13:22:39.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53acfa7640 1 -- 172.21.15.148:0/3477458942 >> 172.21.15.148:0/3477458942 conn(0x7f53a806a640 msgr2=0x7f53a8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:39.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53acfa7640 1 -- 172.21.15.148:0/3477458942 shutdown_connections 2023-10-30T13:22:39.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.377+0000 7f53acfa7640 1 -- 172.21.15.148:0/3477458942 wait complete. 2023-10-30T13:22:39.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 Processor -- start 2023-10-30T13:22:39.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 -- start start 2023-10-30T13:22:39.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53940039b0 0x7f53a81ef840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53a815c850 0x7f53a81526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53a81573a0 0x7f53a81f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a80b3a90 con 0x7f53a81573a0 2023-10-30T13:22:39.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a81efd80 con 0x7f53a815c850 2023-10-30T13:22:39.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a80b2ea0 con 0x7f53940039b0 2023-10-30T13:22:39.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53a81573a0 0x7f53a81f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53a81573a0 0x7f53a81f2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49442/0 (socket says 172.21.15.148:49442) 2023-10-30T13:22:39.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a7fff640 1 -- 172.21.15.148:0/2052628302 learned_addr learned my addr 172.21.15.148:0/2052628302 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:39.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a77fe640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53940039b0 0x7f53a81ef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f539ffff640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53a815c850 0x7f53a81526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a7fff640 1 -- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53940039b0 msgr2=0x7f53a81ef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a7fff640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53940039b0 0x7f53a81ef840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a7fff640 1 -- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53a815c850 msgr2=0x7f53a81526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a7fff640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53a815c850 0x7f53a81526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a7fff640 1 -- 172.21.15.148:0/2052628302 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a8157890 con 0x7f53a81573a0 2023-10-30T13:22:39.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f539ffff640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53a815c850 0x7f53a81526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:39.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a7fff640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53a81573a0 0x7f53a81f2330 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f53a8066500 tx=0x7f53a00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:39.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a4ff9640 1 -- 172.21.15.148:0/2052628302 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a000e070 con 0x7f53a81573a0 2023-10-30T13:22:39.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a4ff9640 1 -- 172.21.15.148:0/2052628302 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0011de0 con 0x7f53a81573a0 2023-10-30T13:22:39.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53a4ff9640 1 -- 172.21.15.148:0/2052628302 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a00039a0 con 0x7f53a81573a0 2023-10-30T13:22:39.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a80b30f0 con 0x7f53a81573a0 2023-10-30T13:22:39.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.381+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a80b36b0 con 0x7f53a81573a0 2023-10-30T13:22:39.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.385+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a80b3bd0 con 0x7f53a81573a0 2023-10-30T13:22:39.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.385+0000 7f53a4ff9640 1 -- 172.21.15.148:0/2052628302 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f53a0003b00 con 0x7f53a81573a0 2023-10-30T13:22:39.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.385+0000 7f53a4ff9640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53880664a0 0x7f5388068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.385+0000 7f53a4ff9640 1 -- 172.21.15.148:0/2052628302 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6226+0+0 (secure 0 0 0) 0x7f53a0084490 con 0x7f53a81573a0 2023-10-30T13:22:39.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.385+0000 7f53a77fe640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53880664a0 0x7f5388068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.385+0000 7f53a77fe640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53880664a0 0x7f5388068960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f53980b2000 tx=0x7f5398001330 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:39.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.389+0000 7f53a4ff9640 1 -- 172.21.15.148:0/2052628302 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53a80b3bd0 con 0x7f53a81573a0 2023-10-30T13:22:39.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53a80b3db0 con 0x7f53a81573a0 2023-10-30T13:22:39.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53a4ff9640 1 -- 172.21.15.148:0/2052628302 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5719 (secure 0 0 0) 0x7f53a80b3db0 con 0x7f53a81573a0 2023-10-30T13:22:39.573 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:39.573 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:39.518211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:39.574 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 59 2023-10-30T13:22:39.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53880664a0 msgr2=0x7f5388068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53880664a0 0x7f5388068960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f53980b2000 tx=0x7f5398001330 comp rx=0 tx=0).stop 2023-10-30T13:22:39.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53a81573a0 msgr2=0x7f53a81f2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53a81573a0 0x7f53a81f2330 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f53a8066500 tx=0x7f53a00115a0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 shutdown_connections 2023-10-30T13:22:39.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53880664a0 0x7f5388068960 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53a81573a0 0x7f53a81f2330 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53a815c850 0x7f53a81526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 --2- 172.21.15.148:0/2052628302 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53940039b0 0x7f53a81ef840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 >> 172.21.15.148:0/2052628302 conn(0x7f53a806a640 msgr2=0x7f53a80caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:39.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 shutdown_connections 2023-10-30T13:22:39.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.577+0000 7f53acfa7640 1 -- 172.21.15.148:0/2052628302 wait complete. 2023-10-30T13:22:39.594 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:22:39.594 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:22:39.595 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:39.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce359e640 1 Processor -- start 2023-10-30T13:22:39.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce359e640 1 -- start start 2023-10-30T13:22:39.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce359e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce415cd50 0x7f4ce40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce359e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4ce40cefa0 0x7f4ce40cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce359e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce415a6c0 con 0x7f4ce41538f0 2023-10-30T13:22:39.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce359e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce415d300 con 0x7f4ce40cefa0 2023-10-30T13:22:39.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce359e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce415d5e0 con 0x7f4ce415cd50 2023-10-30T13:22:39.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce1d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce415cd50 0x7f4ce40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce2d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4ce40cefa0 0x7f4ce40cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce1d9b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce415cd50 0x7f4ce40c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:54212/0 (socket says 172.21.15.148:54212) 2023-10-30T13:22:39.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce1d9b640 1 -- 172.21.15.148:0/3768115377 learned_addr learned my addr 172.21.15.148:0/3768115377 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:39.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce1d9b640 1 -- 172.21.15.148:0/3768115377 >> v2:172.21.15.150:3300/0 conn(0x7f4ce40cefa0 msgr2=0x7f4ce40cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:39.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce1d9b640 1 --2- 172.21.15.148:0/3768115377 >> v2:172.21.15.150:3300/0 conn(0x7f4ce40cefa0 0x7f4ce40cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce1d9b640 1 -- 172.21.15.148:0/3768115377 >> v1:172.21.15.150:6789/0 conn(0x7f4ce41538f0 legacy=0x7f4ce4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce1d9b640 1 -- 172.21.15.148:0/3768115377 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce415d8c0 con 0x7f4ce415cd50 2023-10-30T13:22:39.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.737+0000 7f4ce2d9d640 1 --2- 172.21.15.148:0/3768115377 >> v2:172.21.15.150:3300/0 conn(0x7f4ce40cefa0 0x7f4ce40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:39.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce1d9b640 1 --2- 172.21.15.148:0/3768115377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce415cd50 0x7f4ce40c9f90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009870 tx=0x7f4cd801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=894aebe0865e68c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:39.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce159a640 1 -- 172.21.15.148:0/3768115377 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd80a8a30 con 0x7f4ce415cd50 2023-10-30T13:22:39.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce159a640 1 -- 172.21.15.148:0/3768115377 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd80a8b90 con 0x7f4ce415cd50 2023-10-30T13:22:39.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce159a640 1 -- 172.21.15.148:0/3768115377 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd80b0440 con 0x7f4ce415cd50 2023-10-30T13:22:39.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce359e640 1 -- 172.21.15.148:0/3768115377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce415cd50 msgr2=0x7f4ce40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce359e640 1 --2- 172.21.15.148:0/3768115377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce415cd50 0x7f4ce40c9f90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009870 tx=0x7f4cd801ed30 comp rx=0 tx=0).stop 2023-10-30T13:22:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce359e640 1 -- 172.21.15.148:0/3768115377 shutdown_connections 2023-10-30T13:22:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce359e640 1 --2- 172.21.15.148:0/3768115377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce415cd50 0x7f4ce40c9f90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce359e640 1 --2- 172.21.15.148:0/3768115377 >> v2:172.21.15.150:3300/0 conn(0x7f4ce40cefa0 0x7f4ce40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce359e640 1 -- 172.21.15.148:0/3768115377 >> 172.21.15.148:0/3768115377 conn(0x7f4ce406a640 msgr2=0x7f4ce4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce359e640 1 -- 172.21.15.148:0/3768115377 shutdown_connections 2023-10-30T13:22:39.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.741+0000 7f4ce359e640 1 -- 172.21.15.148:0/3768115377 wait complete. 2023-10-30T13:22:39.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 Processor -- start 2023-10-30T13:22:39.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 -- start start 2023-10-30T13:22:39.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ce40cefa0 0x7f4ce40b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce41538f0 0x7f4ce40b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ce415cd50 0x7f4ce40b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce415c2a0 con 0x7f4ce415cd50 2023-10-30T13:22:39.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce415ad30 con 0x7f4ce40cefa0 2023-10-30T13:22:39.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce41e5670 con 0x7f4ce41538f0 2023-10-30T13:22:39.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce2d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ce415cd50 0x7f4ce40b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce2d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ce415cd50 0x7f4ce40b4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49446/0 (socket says 172.21.15.148:49446) 2023-10-30T13:22:39.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce2d9d640 1 -- 172.21.15.148:0/1423987749 learned_addr learned my addr 172.21.15.148:0/1423987749 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:39.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce259c640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ce40cefa0 0x7f4ce40b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce1d9b640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce41538f0 0x7f4ce40b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce2d9d640 1 -- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce41538f0 msgr2=0x7f4ce40b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce2d9d640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce41538f0 0x7f4ce40b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce2d9d640 1 -- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ce40cefa0 msgr2=0x7f4ce40b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce2d9d640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ce40cefa0 0x7f4ce40b31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce2d9d640 1 -- 172.21.15.148:0/1423987749 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cdc009a70 con 0x7f4ce415cd50 2023-10-30T13:22:39.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce1d9b640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce41538f0 0x7f4ce40b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:39.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce2d9d640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ce415cd50 0x7f4ce40b4c90 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00efc0 tx=0x7f4cdc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:39.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4cd2ffd640 1 -- 172.21.15.148:0/1423987749 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc009d70 con 0x7f4ce415cd50 2023-10-30T13:22:39.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4cd2ffd640 1 -- 172.21.15.148:0/1423987749 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc00f040 con 0x7f4ce415cd50 2023-10-30T13:22:39.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd80094d0 con 0x7f4ce415cd50 2023-10-30T13:22:39.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4cd2ffd640 1 -- 172.21.15.148:0/1423987749 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0039a0 con 0x7f4ce415cd50 2023-10-30T13:22:39.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce415cab0 con 0x7f4ce415cd50 2023-10-30T13:22:39.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.745+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce404e170 con 0x7f4ce415cd50 2023-10-30T13:22:39.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.749+0000 7f4cd2ffd640 1 -- 172.21.15.148:0/1423987749 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4cdc0144a0 con 0x7f4ce415cd50 2023-10-30T13:22:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.749+0000 7f4cd2ffd640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4cc40664a0 0x7f4cc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.749+0000 7f4cd2ffd640 1 -- 172.21.15.148:0/1423987749 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f4cdc084d40 con 0x7f4ce415cd50 2023-10-30T13:22:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.749+0000 7f4ce259c640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4cc40664a0 0x7f4cc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:39.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.749+0000 7f4ce259c640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4cc40664a0 0x7f4cc4068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4007fd0 tx=0x7f4cd4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:39.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.753+0000 7f4cd2ffd640 1 -- 172.21.15.148:0/1423987749 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4cdc051ee0 con 0x7f4ce415cd50 2023-10-30T13:22:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.937+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ce415ae70 con 0x7f4ce415cd50 2023-10-30T13:22:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.937+0000 7f4cd2ffd640 1 -- 172.21.15.148:0/1423987749 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5719 (secure 0 0 0) 0x7f4cdc055500 con 0x7f4ce415cd50 2023-10-30T13:22:39.936 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:39.936 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:39.518211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 59 2023-10-30T13:22:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4cc40664a0 msgr2=0x7f4cc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4cc40664a0 0x7f4cc4068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4007fd0 tx=0x7f4cd4009040 comp rx=0 tx=0).stop 2023-10-30T13:22:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ce415cd50 msgr2=0x7f4ce40b4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ce415cd50 0x7f4ce40b4c90 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00efc0 tx=0x7f4cdc009f90 comp rx=0 tx=0).stop 2023-10-30T13:22:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 shutdown_connections 2023-10-30T13:22:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4cc40664a0 0x7f4cc4068960 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ce415cd50 0x7f4ce40b4c90 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ce41538f0 0x7f4ce40b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 --2- 172.21.15.148:0/1423987749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ce40cefa0 0x7f4ce40b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:39.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 >> 172.21.15.148:0/1423987749 conn(0x7f4ce406a640 msgr2=0x7f4ce40c54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:39.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 shutdown_connections 2023-10-30T13:22:39.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:39.941+0000 7f4ce359e640 1 -- 172.21.15.148:0/1423987749 wait complete. 2023-10-30T13:22:39.957 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 59, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 59, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5694": { "addr": "172.21.15.148:6837/3781833565", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3781833565, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3781833565, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5694, "incarnation": 59, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:22:39.518211+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5694 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6835/3194894584", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 3194894584, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 3194894584, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 54, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5643, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:22:41.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:42.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.109+0000 7fa51269c640 1 Processor -- start 2023-10-30T13:22:42.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.109+0000 7fa51269c640 1 -- start start 2023-10-30T13:22:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa51269c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa50c0c6b80 0x7fa50c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa51269c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6f60 0x7fa50c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa51269c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa50c0c8430 0x7fa50c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa51269c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50c0c0a50 con 0x7fa50c0c6b80 2023-10-30T13:22:42.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa51269c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50c06f6e0 con 0x7fa50c0c8430 2023-10-30T13:22:42.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa51269c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c06f9c0 con 0x7fa50c0c6f60 2023-10-30T13:22:42.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa50bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa50c0c6b80 0x7fa50c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:42.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa50bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa50c0c6b80 0x7fa50c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42292/0 (socket says 172.21.15.148:42292) 2023-10-30T13:22:42.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa50bfff640 1 -- 172.21.15.148:0/3137274918 learned_addr learned my addr 172.21.15.148:0/3137274918 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa510c39640 1 --2- 172.21.15.148:0/3137274918 >> v2:172.21.15.150:3300/0 conn(0x7fa50c0c8430 0x7fa50c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa50b7fe640 1 --2- 172.21.15.148:0/3137274918 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6f60 0x7fa50c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa50bfff640 1 -- 172.21.15.148:0/3137274918 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6f60 msgr2=0x7fa50c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa50bfff640 1 --2- 172.21.15.148:0/3137274918 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6f60 0x7fa50c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa50bfff640 1 -- 172.21.15.148:0/3137274918 >> v2:172.21.15.150:3300/0 conn(0x7fa50c0c8430 msgr2=0x7fa50c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa50bfff640 1 --2- 172.21.15.148:0/3137274918 >> v2:172.21.15.150:3300/0 conn(0x7fa50c0c8430 0x7fa50c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.113+0000 7fa50bfff640 1 -- 172.21.15.148:0/3137274918 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c06fca0 con 0x7fa50c0c6b80 2023-10-30T13:22:42.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa50bfff640 1 --2- 172.21.15.148:0/3137274918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6b80 0x7fa50c0bfec0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0b0860 tx=0x7fa4fc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484ad060ae42ee25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:42.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa50affd640 1 -- 172.21.15.148:0/3137274918 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc0ab600 con 0x7fa50c0c6b80 2023-10-30T13:22:42.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa50affd640 1 -- 172.21.15.148:0/3137274918 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc0a7050 con 0x7fa50c0c6b80 2023-10-30T13:22:42.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa50affd640 1 -- 172.21.15.148:0/3137274918 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc0ab980 con 0x7fa50c0c6b80 2023-10-30T13:22:42.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 -- 172.21.15.148:0/3137274918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6b80 msgr2=0x7fa50c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:42.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 --2- 172.21.15.148:0/3137274918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6b80 0x7fa50c0bfec0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0b0860 tx=0x7fa4fc01c470 comp rx=0 tx=0).stop 2023-10-30T13:22:42.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 -- 172.21.15.148:0/3137274918 shutdown_connections 2023-10-30T13:22:42.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 --2- 172.21.15.148:0/3137274918 >> v2:172.21.15.150:3300/0 conn(0x7fa50c0c8430 0x7fa50c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 --2- 172.21.15.148:0/3137274918 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6f60 0x7fa50c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 --2- 172.21.15.148:0/3137274918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6b80 0x7fa50c0bfec0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 -- 172.21.15.148:0/3137274918 >> 172.21.15.148:0/3137274918 conn(0x7fa50c0c7910 msgr2=0x7fa50c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:42.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 -- 172.21.15.148:0/3137274918 shutdown_connections 2023-10-30T13:22:42.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 -- 172.21.15.148:0/3137274918 wait complete. 2023-10-30T13:22:42.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 Processor -- start 2023-10-30T13:22:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.117+0000 7fa51269c640 1 -- start start 2023-10-30T13:22:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa51269c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6b80 0x7fa50c135d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa51269c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6f60 0x7fa50c134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa51269c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0c8430 0x7fa50c135120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:42.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa51269c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c158e20 con 0x7fa50c0c6f60 2023-10-30T13:22:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa51269c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c1e7d80 con 0x7fa50c0c8430 2023-10-30T13:22:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa51269c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c1e8020 con 0x7fa50c0c6b80 2023-10-30T13:22:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6f60 0x7fa50c134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6f60 0x7fa50c134be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42306/0 (socket says 172.21.15.148:42306) 2023-10-30T13:22:42.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50b7fe640 1 -- 172.21.15.148:0/1543129794 learned_addr learned my addr 172.21.15.148:0/1543129794 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50bfff640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6b80 0x7fa50c135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa510c39640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0c8430 0x7fa50c135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50b7fe640 1 -- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6b80 msgr2=0x7fa50c135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50b7fe640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6b80 0x7fa50c135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50b7fe640 1 -- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0c8430 msgr2=0x7fa50c135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50b7fe640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0c8430 0x7fa50c135120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50b7fe640 1 -- 172.21.15.148:0/1543129794 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc0094d0 con 0x7fa50c0c6f60 2023-10-30T13:22:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa50b7fe640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6f60 0x7fa50c134be0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa50000d800 tx=0x7fa50000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa508ff9640 1 -- 172.21.15.148:0/1543129794 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5000096a0 con 0x7fa50c0c6f60 2023-10-30T13:22:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa508ff9640 1 -- 172.21.15.148:0/1543129794 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa500009800 con 0x7fa50c0c6f60 2023-10-30T13:22:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50c158250 con 0x7fa50c0c6f60 2023-10-30T13:22:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa508ff9640 1 -- 172.21.15.148:0/1543129794 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500009b70 con 0x7fa50c0c6f60 2023-10-30T13:22:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50c157f30 con 0x7fa50c0c6f60 2023-10-30T13:22:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50c134a30 con 0x7fa50c0c6f60 2023-10-30T13:22:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.121+0000 7fa508ff9640 1 -- 172.21.15.148:0/1543129794 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa500009cd0 con 0x7fa50c0c6f60 2023-10-30T13:22:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.125+0000 7fa508ff9640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4f80664d0 0x7fa4f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.125+0000 7fa508ff9640 1 -- 172.21.15.148:0/1543129794 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fa5000846c0 con 0x7fa50c0c6f60 2023-10-30T13:22:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.125+0000 7fa50bfff640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4f80664d0 0x7fa4f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.125+0000 7fa50bfff640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4f80664d0 0x7fa4f8068990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa50c06f1b0 tx=0x7fa4fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.125+0000 7fa508ff9640 1 -- 172.21.15.148:0/1543129794 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa5000519a0 con 0x7fa50c0c6f60 2023-10-30T13:22:42.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.309+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa50c158a50 con 0x7fa50c0c6f60 2023-10-30T13:22:42.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.309+0000 7fa508ff9640 1 -- 172.21.15.148:0/1543129794 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5719 (secure 0 0 0) 0x7fa500054fc0 con 0x7fa50c0c6f60 2023-10-30T13:22:42.307 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:42.307 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:39.518211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:42.307 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 59 2023-10-30T13:22:42.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4f80664d0 msgr2=0x7fa4f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:42.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4f80664d0 0x7fa4f8068990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa50c06f1b0 tx=0x7fa4fc008040 comp rx=0 tx=0).stop 2023-10-30T13:22:42.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6f60 msgr2=0x7fa50c134be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:42.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6f60 0x7fa50c134be0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa50000d800 tx=0x7fa50000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 shutdown_connections 2023-10-30T13:22:42.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa4f80664d0 0x7fa4f8068990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa50c0c6f60 0x7fa50c134be0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa50c0c8430 0x7fa50c135120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 --2- 172.21.15.148:0/1543129794 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa50c0c6b80 0x7fa50c135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:42.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 >> 172.21.15.148:0/1543129794 conn(0x7fa50c0c7910 msgr2=0x7fa50c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:42.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 shutdown_connections 2023-10-30T13:22:42.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:42.313+0000 7fa51269c640 1 -- 172.21.15.148:0/1543129794 wait complete. 2023-10-30T13:22:42.327 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:44.329 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:44.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4e33b640 1 Processor -- start 2023-10-30T13:22:44.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4e33b640 1 -- start start 2023-10-30T13:22:44.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4e33b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2b4815cd50 0x7f2b480c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:44.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4e33b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2b480cefa0 0x7f2b480cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:44.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4e33b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4815a6c0 con 0x7f2b481538f0 2023-10-30T13:22:44.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4e33b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b480cf8f0 con 0x7f2b4815cd50 2023-10-30T13:22:44.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4e33b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4815d300 con 0x7f2b480cefa0 2023-10-30T13:22:44.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4cb38640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2b4815cd50 0x7f2b480c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:44.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4cb38640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2b4815cd50 0x7f2b480c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42320/0 (socket says 172.21.15.148:42320) 2023-10-30T13:22:44.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4cb38640 1 -- 172.21.15.148:0/3291965704 learned_addr learned my addr 172.21.15.148:0/3291965704 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:44.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4db3a640 1 --2- 172.21.15.148:0/3291965704 >> v2:172.21.15.150:3300/0 conn(0x7f2b480cefa0 0x7f2b480cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:44.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4cb38640 1 -- 172.21.15.148:0/3291965704 >> v2:172.21.15.150:3300/0 conn(0x7f2b480cefa0 msgr2=0x7f2b480cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:44.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4cb38640 1 --2- 172.21.15.148:0/3291965704 >> v2:172.21.15.150:3300/0 conn(0x7f2b480cefa0 0x7f2b480cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4cb38640 1 -- 172.21.15.148:0/3291965704 >> v1:172.21.15.150:6789/0 conn(0x7f2b481538f0 legacy=0x7f2b48159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:44.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.481+0000 7f2b4cb38640 1 -- 172.21.15.148:0/3291965704 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4815d5e0 con 0x7f2b4815cd50 2023-10-30T13:22:44.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b4cb38640 1 --2- 172.21.15.148:0/3291965704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4815cd50 0x7f2b480c9f90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2b48066a40 tx=0x7f2b3801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f42ac7c87c3541 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:44.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b3ffff640 1 -- 172.21.15.148:0/3291965704 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b380a89f0 con 0x7f2b4815cd50 2023-10-30T13:22:44.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b3ffff640 1 -- 172.21.15.148:0/3291965704 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b380a8b50 con 0x7f2b4815cd50 2023-10-30T13:22:44.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b3ffff640 1 -- 172.21.15.148:0/3291965704 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b380b0440 con 0x7f2b4815cd50 2023-10-30T13:22:44.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b4e33b640 1 -- 172.21.15.148:0/3291965704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4815cd50 msgr2=0x7f2b480c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:44.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b4e33b640 1 --2- 172.21.15.148:0/3291965704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4815cd50 0x7f2b480c9f90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2b48066a40 tx=0x7f2b3801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b4e33b640 1 -- 172.21.15.148:0/3291965704 shutdown_connections 2023-10-30T13:22:44.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b4e33b640 1 --2- 172.21.15.148:0/3291965704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4815cd50 0x7f2b480c9f90 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b4e33b640 1 --2- 172.21.15.148:0/3291965704 >> v2:172.21.15.150:3300/0 conn(0x7f2b480cefa0 0x7f2b480cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b4e33b640 1 -- 172.21.15.148:0/3291965704 >> 172.21.15.148:0/3291965704 conn(0x7f2b4806a640 msgr2=0x7f2b48065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b4e33b640 1 -- 172.21.15.148:0/3291965704 shutdown_connections 2023-10-30T13:22:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.485+0000 7f2b4e33b640 1 -- 172.21.15.148:0/3291965704 wait complete. 2023-10-30T13:22:44.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 Processor -- start 2023-10-30T13:22:44.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 -- start start 2023-10-30T13:22:44.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b480cefa0 0x7f2b48158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:44.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b481538f0 0x7f2b481f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:44.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b4815cd50 0x7f2b481f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:44.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b48152060 con 0x7f2b481538f0 2023-10-30T13:22:44.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b481527b0 con 0x7f2b4815cd50 2023-10-30T13:22:44.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b48152270 con 0x7f2b480cefa0 2023-10-30T13:22:44.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4cb38640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b481538f0 0x7f2b481f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:44.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4cb38640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b481538f0 0x7f2b481f26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42322/0 (socket says 172.21.15.148:42322) 2023-10-30T13:22:44.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4cb38640 1 -- 172.21.15.148:0/2965839863 learned_addr learned my addr 172.21.15.148:0/2965839863 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:44.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4db3a640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b4815cd50 0x7f2b481f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:44.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4d339640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b480cefa0 0x7f2b48158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:44.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4cb38640 1 -- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b480cefa0 msgr2=0x7f2b48158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:44.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4cb38640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b480cefa0 0x7f2b48158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4cb38640 1 -- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b4815cd50 msgr2=0x7f2b481f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:44.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4cb38640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b4815cd50 0x7f2b481f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4cb38640 1 -- 172.21.15.148:0/2965839863 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b380094d0 con 0x7f2b481538f0 2023-10-30T13:22:44.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4db3a640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b4815cd50 0x7f2b481f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:44.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4cb38640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b481538f0 0x7f2b481f26f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2b3801b900 tx=0x7f2b3801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:44.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b3dffb640 1 -- 172.21.15.148:0/2965839863 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b380ba600 con 0x7f2b481538f0 2023-10-30T13:22:44.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b3dffb640 1 -- 172.21.15.148:0/2965839863 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b380ba760 con 0x7f2b481538f0 2023-10-30T13:22:44.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b481529c0 con 0x7f2b481538f0 2023-10-30T13:22:44.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b3dffb640 1 -- 172.21.15.148:0/2965839863 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b380bab10 con 0x7f2b481538f0 2023-10-30T13:22:44.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b480b5180 con 0x7f2b481538f0 2023-10-30T13:22:44.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.489+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4804e170 con 0x7f2b481538f0 2023-10-30T13:22:44.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.493+0000 7f2b3dffb640 1 -- 172.21.15.148:0/2965839863 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b380bac70 con 0x7f2b481538f0 2023-10-30T13:22:44.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.493+0000 7f2b3dffb640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b2c0664a0 0x7f2b2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:44.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.493+0000 7f2b3dffb640 1 -- 172.21.15.148:0/2965839863 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f2b38122660 con 0x7f2b481538f0 2023-10-30T13:22:44.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.493+0000 7f2b4d339640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b2c0664a0 0x7f2b2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:44.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.493+0000 7f2b4d339640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b2c0664a0 0x7f2b2c068960 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b44000f70 tx=0x7f2b44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:44.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.497+0000 7f2b3dffb640 1 -- 172.21.15.148:0/2965839863 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b380ef800 con 0x7f2b481538f0 2023-10-30T13:22:44.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b480b2290 con 0x7f2b481538f0 2023-10-30T13:22:44.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b3dffb640 1 -- 172.21.15.148:0/2965839863 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5719 (secure 0 0 0) 0x7f2b380f2e20 con 0x7f2b481538f0 2023-10-30T13:22:44.726 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:44.726 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:39.518211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:44.726 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 59 2023-10-30T13:22:44.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b2c0664a0 msgr2=0x7f2b2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:44.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b2c0664a0 0x7f2b2c068960 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b44000f70 tx=0x7f2b44009040 comp rx=0 tx=0).stop 2023-10-30T13:22:44.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b481538f0 msgr2=0x7f2b481f26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:44.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b481538f0 0x7f2b481f26f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2b3801b900 tx=0x7f2b3801ba90 comp rx=0 tx=0).stop 2023-10-30T13:22:44.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 shutdown_connections 2023-10-30T13:22:44.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b2c0664a0 0x7f2b2c068960 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b481538f0 0x7f2b481f26f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b4815cd50 0x7f2b481f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 --2- 172.21.15.148:0/2965839863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b480cefa0 0x7f2b48158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:44.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 >> 172.21.15.148:0/2965839863 conn(0x7f2b4806a640 msgr2=0x7f2b4814fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:44.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 shutdown_connections 2023-10-30T13:22:44.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:44.729+0000 7f2b4e33b640 1 -- 172.21.15.148:0/2965839863 wait complete. 2023-10-30T13:22:44.748 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:46.749 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:46.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a198c9640 1 Processor -- start 2023-10-30T13:22:46.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a198c9640 1 -- start start 2023-10-30T13:22:46.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a198c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a140c6f60 0x7f6a141566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:46.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a198c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a140c0a50 con 0x7f6a140c6b80 2023-10-30T13:22:46.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a198c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a1406f6e0 con 0x7f6a140c05d0 2023-10-30T13:22:46.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a198c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1406f9c0 con 0x7f6a140c6f60 2023-10-30T13:22:46.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a12ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6a140c6b80 0x7f6a140bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42264/0 (socket says 172.21.15.148:42264) 2023-10-30T13:22:46.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a12ffd640 1 -- 172.21.15.148:0/679047883 learned_addr learned my addr 172.21.15.148:0/679047883 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:22:46.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a137fe640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a140c6f60 0x7f6a141566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:46.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910377203 0 0) 0x7f6a140c0a50 con 0x7f6a140c6b80 2023-10-30T13:22:46.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a00003530 con 0x7f6a140c6b80 2023-10-30T13:22:46.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021451751 0 0) 0x7f6a1406f6e0 con 0x7f6a140c05d0 2023-10-30T13:22:46.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a140c0a50 con 0x7f6a140c05d0 2023-10-30T13:22:46.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2024760180 0 0) 0x7f6a00003530 con 0x7f6a140c6b80 2023-10-30T13:22:46.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a140c6f60 msgr2=0x7f6a141566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:46.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a140c6f60 0x7f6a141566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:46.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 >> v1:172.21.15.150:6789/0 conn(0x7f6a140c05d0 legacy=0x7f6a1413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:46.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1406fca0 con 0x7f6a140c6b80 2023-10-30T13:22:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f6a04001680 con 0x7f6a140c6b80 2023-10-30T13:22:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.901+0000 7f6a137fe640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a140c6f60 0x7f6a141566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 >> v1:172.21.15.148:6789/0 conn(0x7f6a140c6b80 legacy=0x7f6a140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a140c6f60 0x7f6a00003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a00004080 0x7f6a000047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:46.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a00004d10 0x7f6a00005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:46.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1406f9c0 con 0x7f6a00004d10 2023-10-30T13:22:46.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0000ba40 con 0x7f6a00004080 2023-10-30T13:22:46.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0000bc00 con 0x7f6a140c6f60 2023-10-30T13:22:46.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a137fe640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a00004d10 0x7f6a00005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:46.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a12ffd640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a140c6f60 0x7f6a00003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:46.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a0a7fc640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a00004080 0x7f6a000047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:46.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a137fe640 1 -- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a140c6f60 msgr2=0x7f6a00003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:46.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a137fe640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a140c6f60 0x7f6a00003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:46.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a137fe640 1 -- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a00004080 msgr2=0x7f6a000047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:46.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a137fe640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a00004080 0x7f6a000047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:46.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a137fe640 1 -- 172.21.15.148:0/679047883 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c009420 con 0x7f6a00004d10 2023-10-30T13:22:46.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a137fe640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a00004d10 0x7f6a00005190 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009c90 tx=0x7f6a0c01cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde1b461e29020e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:46.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c004030 con 0x7f6a00004d10 2023-10-30T13:22:46.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c004190 con 0x7f6a00004d10 2023-10-30T13:22:46.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.905+0000 7f6a127fc640 1 -- 172.21.15.148:0/679047883 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0a57a0 con 0x7f6a00004d10 2023-10-30T13:22:46.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 -- 172.21.15.148:0/679047883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a00004d10 msgr2=0x7f6a00005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:46.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 --2- 172.21.15.148:0/679047883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a00004d10 0x7f6a00005190 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009c90 tx=0x7f6a0c01cbf0 comp rx=0 tx=0).stop 2023-10-30T13:22:46.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a137fe640 1 -- 172.21.15.148:0/679047883 reap_dead start 2023-10-30T13:22:46.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 -- 172.21.15.148:0/679047883 shutdown_connections 2023-10-30T13:22:46.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 -- 172.21.15.148:0/679047883 >> 172.21.15.148:0/679047883 conn(0x7f6a140c7910 msgr2=0x7f6a140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:46.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 -- 172.21.15.148:0/679047883 shutdown_connections 2023-10-30T13:22:46.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 -- 172.21.15.148:0/679047883 wait complete. 2023-10-30T13:22:46.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 Processor -- start 2023-10-30T13:22:46.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 -- start start 2023-10-30T13:22:46.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a00004d10 0x7f6a141ef540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:46.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a141e1240 0x7f6a14139310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:46.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a14138620 0x7f6a14138aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1413adc0 con 0x7f6a14138620 2023-10-30T13:22:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a14138fe0 con 0x7f6a141e1240 2023-10-30T13:22:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a198c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a141e0db0 con 0x7f6a00004d10 2023-10-30T13:22:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a137fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a14138620 0x7f6a14138aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a137fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a14138620 0x7f6a14138aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42332/0 (socket says 172.21.15.148:42332) 2023-10-30T13:22:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a137fe640 1 -- 172.21.15.148:0/3502454424 learned_addr learned my addr 172.21.15.148:0/3502454424 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:46.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a12ffd640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a00004d10 0x7f6a141ef540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:46.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.909+0000 7f6a0a7fc640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a141e1240 0x7f6a14139310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:46.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a137fe640 1 -- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a00004d10 msgr2=0x7f6a141ef540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:46.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a137fe640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a00004d10 0x7f6a141ef540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:46.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a137fe640 1 -- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a141e1240 msgr2=0x7f6a14139310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a137fe640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a141e1240 0x7f6a14139310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a137fe640 1 -- 172.21.15.148:0/3502454424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a141e1dc0 con 0x7f6a14138620 2023-10-30T13:22:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a137fe640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a14138620 0x7f6a14138aa0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c098500 tx=0x7f6a00005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a10ff9640 1 -- 172.21.15.148:0/3502454424 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0a6f10 con 0x7f6a14138620 2023-10-30T13:22:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a10ff9640 1 -- 172.21.15.148:0/3502454424 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c006e20 con 0x7f6a14138620 2023-10-30T13:22:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a10ff9640 1 -- 172.21.15.148:0/3502454424 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c004500 con 0x7f6a14138620 2023-10-30T13:22:46.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a14139850 con 0x7f6a14138620 2023-10-30T13:22:46.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a141e01a0 con 0x7f6a14138620 2023-10-30T13:22:46.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1404e170 con 0x7f6a14138620 2023-10-30T13:22:46.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a10ff9640 1 -- 172.21.15.148:0/3502454424 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6a0c0059a0 con 0x7f6a14138620 2023-10-30T13:22:46.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a10ff9640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69f40664d0 0x7f69f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:46.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a10ff9640 1 -- 172.21.15.148:0/3502454424 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f6a0c11eac0 con 0x7f6a14138620 2023-10-30T13:22:46.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.913+0000 7f6a12ffd640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69f40664d0 0x7f69f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:46.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.917+0000 7f6a12ffd640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69f40664d0 0x7f69f4068990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6a04001060 tx=0x7f6a04014040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:46.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:46.917+0000 7f6a10ff9640 1 -- 172.21.15.148:0/3502454424 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6a0c0ebd10 con 0x7f6a14138620 2023-10-30T13:22:47.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.097+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a141e0ef0 con 0x7f6a14138620 2023-10-30T13:22:47.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.097+0000 7f6a10ff9640 1 -- 172.21.15.148:0/3502454424 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f6a0c0edc40 con 0x7f6a14138620 2023-10-30T13:22:47.096 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:47.097 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:45.705579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:47.097 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 60 2023-10-30T13:22:47.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69f40664d0 msgr2=0x7f69f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:47.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69f40664d0 0x7f69f4068990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6a04001060 tx=0x7f6a04014040 comp rx=0 tx=0).stop 2023-10-30T13:22:47.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a14138620 msgr2=0x7f6a14138aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:47.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a14138620 0x7f6a14138aa0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c098500 tx=0x7f6a00005800 comp rx=0 tx=0).stop 2023-10-30T13:22:47.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 shutdown_connections 2023-10-30T13:22:47.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69f40664d0 0x7f69f4068990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:47.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a14138620 0x7f6a14138aa0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:47.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a141e1240 0x7f6a14139310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:47.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 --2- 172.21.15.148:0/3502454424 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a00004d10 0x7f6a141ef540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:47.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 >> 172.21.15.148:0/3502454424 conn(0x7f6a140c7910 msgr2=0x7f6a1406be00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:47.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 shutdown_connections 2023-10-30T13:22:47.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:47.101+0000 7f6a198c9640 1 -- 172.21.15.148:0/3502454424 wait complete. 2023-10-30T13:22:47.117 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:49.118 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:49.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.269+0000 7f8134e6b640 1 Processor -- start 2023-10-30T13:22:49.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.269+0000 7f8134e6b640 1 -- start start 2023-10-30T13:22:49.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8134e6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c6b80 0x7f81300bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:49.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8134e6b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f81300c6f60 0x7f81300c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:49.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8134e6b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81300c0a50 con 0x7f81300c8430 2023-10-30T13:22:49.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8134e6b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813006f6e0 con 0x7f81300c6f60 2023-10-30T13:22:49.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8134e6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813006f9c0 con 0x7f81300c6b80 2023-10-30T13:22:49.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8125d9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f81300c6f60 0x7f81300c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:49.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8125d9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f81300c6f60 0x7f81300c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42348/0 (socket says 172.21.15.148:42348) 2023-10-30T13:22:49.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8125d9b640 1 -- 172.21.15.148:0/730723103 learned_addr learned my addr 172.21.15.148:0/730723103 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:49.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f812ed9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f81300c8430 0x7f813013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42276/0 (socket says 172.21.15.148:42276) 2023-10-30T13:22:49.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f812e59c640 1 --2- 172.21.15.148:0/730723103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c6b80 0x7f81300bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:49.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8125d9b640 1 -- 172.21.15.148:0/730723103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c6b80 msgr2=0x7f81300bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:49.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8125d9b640 1 --2- 172.21.15.148:0/730723103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c6b80 0x7f81300bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:49.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8125d9b640 1 -- 172.21.15.148:0/730723103 >> v1:172.21.15.148:6789/0 conn(0x7f81300c8430 legacy=0x7f813013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:49.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f8125d9b640 1 -- 172.21.15.148:0/730723103 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813006fca0 con 0x7f81300c6f60 2023-10-30T13:22:49.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.273+0000 7f812dd9b640 1 -- 172.21.15.148:0/730723103 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688388241 0 0) 0x7f81300c0a50 con 0x7f81300c8430 2023-10-30T13:22:49.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8125d9b640 1 --2- 172.21.15.148:0/730723103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 0x7f81300c0400 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8118009870 tx=0x7f811801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632597c66589df2d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:49.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f812dd9b640 1 -- 172.21.15.148:0/730723103 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81180ab650 con 0x7f81300c6f60 2023-10-30T13:22:49.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f812dd9b640 1 -- 172.21.15.148:0/730723103 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81180ab7b0 con 0x7f81300c6f60 2023-10-30T13:22:49.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f812dd9b640 1 -- 172.21.15.148:0/730723103 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81180abb40 con 0x7f81300c6f60 2023-10-30T13:22:49.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 -- 172.21.15.148:0/730723103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 msgr2=0x7f81300c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:49.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 --2- 172.21.15.148:0/730723103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 0x7f81300c0400 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8118009870 tx=0x7f811801c540 comp rx=0 tx=0).stop 2023-10-30T13:22:49.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 -- 172.21.15.148:0/730723103 shutdown_connections 2023-10-30T13:22:49.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 --2- 172.21.15.148:0/730723103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 0x7f81300c0400 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:49.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 --2- 172.21.15.148:0/730723103 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c6b80 0x7f81300bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:49.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 -- 172.21.15.148:0/730723103 >> 172.21.15.148:0/730723103 conn(0x7f81300c7910 msgr2=0x7f81300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:49.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 -- 172.21.15.148:0/730723103 shutdown_connections 2023-10-30T13:22:49.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 -- 172.21.15.148:0/730723103 wait complete. 2023-10-30T13:22:49.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 Processor -- start 2023-10-30T13:22:49.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.277+0000 7f8134e6b640 1 -- start start 2023-10-30T13:22:49.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8134e6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81300c6b80 0x7f81300c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:49.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8134e6b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 0x7f813013d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:49.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8134e6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c8430 0x7f813013d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:49.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8134e6b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81301f6210 con 0x7f81300c6f60 2023-10-30T13:22:49.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8134e6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81301561a0 con 0x7f81300c6b80 2023-10-30T13:22:49.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8134e6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81301f5c90 con 0x7f81300c8430 2023-10-30T13:22:49.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8125d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 0x7f813013d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:49.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8125d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 0x7f813013d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42358/0 (socket says 172.21.15.148:42358) 2023-10-30T13:22:49.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8125d9b640 1 -- 172.21.15.148:0/357467579 learned_addr learned my addr 172.21.15.148:0/357467579 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:49.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f812e59c640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81300c6b80 0x7f81300c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:49.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f812ed9d640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c8430 0x7f813013d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:49.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8125d9b640 1 -- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c8430 msgr2=0x7f813013d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:49.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8125d9b640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c8430 0x7f813013d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:49.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8125d9b640 1 -- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81300c6b80 msgr2=0x7f81300c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:49.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8125d9b640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81300c6b80 0x7f81300c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:49.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8125d9b640 1 -- 172.21.15.148:0/357467579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81180094d0 con 0x7f81300c6f60 2023-10-30T13:22:49.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8125d9b640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 0x7f813013d0b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f811801a170 tx=0x7f811801ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:49.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f81277fe640 1 -- 172.21.15.148:0/357467579 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81180b4800 con 0x7f81300c6f60 2023-10-30T13:22:49.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f81277fe640 1 -- 172.21.15.148:0/357467579 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81180b4960 con 0x7f81300c6f60 2023-10-30T13:22:49.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f81277fe640 1 -- 172.21.15.148:0/357467579 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81180b4cf0 con 0x7f81300c6f60 2023-10-30T13:22:49.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813013f070 con 0x7f81300c6f60 2023-10-30T13:22:49.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813013c8b0 con 0x7f81300c6f60 2023-10-30T13:22:49.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813004e170 con 0x7f81300c6f60 2023-10-30T13:22:49.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.281+0000 7f81277fe640 1 -- 172.21.15.148:0/357467579 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f81180aa430 con 0x7f81300c6f60 2023-10-30T13:22:49.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.285+0000 7f81277fe640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81100664a0 0x7f8110068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:49.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.285+0000 7f81277fe640 1 -- 172.21.15.148:0/357467579 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f811811ef90 con 0x7f81300c6f60 2023-10-30T13:22:49.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.285+0000 7f812e59c640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81100664a0 0x7f8110068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:49.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.285+0000 7f812e59c640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81100664a0 0x7f8110068960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8120009af0 tx=0x7f8120008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:49.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.289+0000 7f81277fe640 1 -- 172.21.15.148:0/357467579 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f81180ec1c0 con 0x7f81300c6f60 2023-10-30T13:22:49.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.469+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8130071c60 con 0x7f81300c6f60 2023-10-30T13:22:49.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.469+0000 7f81277fe640 1 -- 172.21.15.148:0/357467579 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f81180ef7e0 con 0x7f81300c6f60 2023-10-30T13:22:49.466 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:49.466 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:45.705579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:49.467 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 60 2023-10-30T13:22:49.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.469+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81100664a0 msgr2=0x7f8110068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:49.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.469+0000 7f8134e6b640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81100664a0 0x7f8110068960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8120009af0 tx=0x7f8120008040 comp rx=0 tx=0).stop 2023-10-30T13:22:49.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 msgr2=0x7f813013d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 0x7f813013d0b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f811801a170 tx=0x7f811801ad80 comp rx=0 tx=0).stop 2023-10-30T13:22:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 shutdown_connections 2023-10-30T13:22:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81100664a0 0x7f8110068960 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81300c6f60 0x7f813013d0b0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:49.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81300c8430 0x7f813013d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:49.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 --2- 172.21.15.148:0/357467579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81300c6b80 0x7f81300c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:49.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 >> 172.21.15.148:0/357467579 conn(0x7f81300c7910 msgr2=0x7f81301394a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:49.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 shutdown_connections 2023-10-30T13:22:49.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:49.473+0000 7f8134e6b640 1 -- 172.21.15.148:0/357467579 wait complete. 2023-10-30T13:22:49.487 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:51.488 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:51.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.641+0000 7f4ff55a1640 1 Processor -- start 2023-10-30T13:22:51.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.641+0000 7f4ff55a1640 1 -- start start 2023-10-30T13:22:51.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4ff55a1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4ff00c05d0 0x7f4ff013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4ff55a1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00c0a50 con 0x7f4ff00c6f60 2023-10-30T13:22:51.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4ff55a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff006f6e0 con 0x7f4ff00c6b80 2023-10-30T13:22:51.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4ff55a1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff006f9c0 con 0x7f4ff00c05d0 2023-10-30T13:22:51.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fef7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4ff00c6f60 0x7f4ff01566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42290/0 (socket says 172.21.15.148:42290) 2023-10-30T13:22:51.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fef7fe640 1 -- 172.21.15.148:0/3841137567 learned_addr learned my addr 172.21.15.148:0/3841137567 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:22:51.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fe67fc640 1 --2- 172.21.15.148:0/3841137567 >> v2:172.21.15.150:3300/0 conn(0x7f4ff00c05d0 0x7f4ff013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664847481 0 0) 0x7f4ff00c0a50 con 0x7f4ff00c6f60 2023-10-30T13:22:51.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fdc005740 con 0x7f4ff00c6f60 2023-10-30T13:22:51.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805116143 0 0) 0x7f4ff006f6e0 con 0x7f4ff00c6b80 2023-10-30T13:22:51.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff00c0a50 con 0x7f4ff00c6b80 2023-10-30T13:22:51.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1723356856 0 0) 0x7f4fdc005740 con 0x7f4ff00c6f60 2023-10-30T13:22:51.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 >> v2:172.21.15.150:3300/0 conn(0x7f4ff00c05d0 msgr2=0x7f4ff013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:51.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 --2- 172.21.15.148:0/3841137567 >> v2:172.21.15.150:3300/0 conn(0x7f4ff00c05d0 0x7f4ff013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 >> v1:172.21.15.150:6789/0 conn(0x7f4ff00c6b80 legacy=0x7f4ff00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:51.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff006fca0 con 0x7f4ff00c6f60 2023-10-30T13:22:51.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f4fe8001a00 con 0x7f4ff00c6f60 2023-10-30T13:22:51.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 >> v1:172.21.15.148:6789/0 conn(0x7f4ff00c6f60 legacy=0x7f4ff01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:51.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc005d20 0x7f4fdc006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0069e0 0x7f4fdc006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fdc0073a0 0x7f4fdc00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff006f9c0 con 0x7f4fdc0073a0 2023-10-30T13:22:51.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc00e220 con 0x7f4fdc005d20 2023-10-30T13:22:51.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc00e3e0 con 0x7f4fdc0069e0 2023-10-30T13:22:51.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fef7fe640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fdc0073a0 0x7f4fdc00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4feeffd640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc005d20 0x7f4fdc006120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fe67fc640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0069e0 0x7f4fdc006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fef7fe640 1 -- 172.21.15.148:0/3841137567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0069e0 msgr2=0x7f4fdc006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:51.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fef7fe640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4fdc0069e0 0x7f4fdc006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fef7fe640 1 -- 172.21.15.148:0/3841137567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc005d20 msgr2=0x7f4fdc006120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:51.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fef7fe640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc005d20 0x7f4fdc006120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.645+0000 7f4fef7fe640 1 -- 172.21.15.148:0/3841137567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc005740 con 0x7f4fdc0073a0 2023-10-30T13:22:51.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4fef7fe640 1 -- 172.21.15.148:0/3841137567 reap_dead start 2023-10-30T13:22:51.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4fef7fe640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fdc0073a0 0x7f4fdc00dba0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800a7c0 tx=0x7f4fe801fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c047acd6423064b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:51.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe80aa630 con 0x7f4fdc0073a0 2023-10-30T13:22:51.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe80aa7b0 con 0x7f4fdc0073a0 2023-10-30T13:22:51.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4fee7fc640 1 -- 172.21.15.148:0/3841137567 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe80aab60 con 0x7f4fdc0073a0 2023-10-30T13:22:51.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4ff55a1640 1 -- 172.21.15.148:0/3841137567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fdc0073a0 msgr2=0x7f4fdc00dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:51.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4ff55a1640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fdc0073a0 0x7f4fdc00dba0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800a7c0 tx=0x7f4fe801fa00 comp rx=0 tx=0).stop 2023-10-30T13:22:51.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4ff55a1640 1 -- 172.21.15.148:0/3841137567 shutdown_connections 2023-10-30T13:22:51.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4ff55a1640 1 --2- 172.21.15.148:0/3841137567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fdc0073a0 0x7f4fdc00dba0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4ff55a1640 1 -- 172.21.15.148:0/3841137567 >> 172.21.15.148:0/3841137567 conn(0x7f4ff00c7910 msgr2=0x7f4ff00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:51.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4ff55a1640 1 -- 172.21.15.148:0/3841137567 shutdown_connections 2023-10-30T13:22:51.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4ff55a1640 1 -- 172.21.15.148:0/3841137567 wait complete. 2023-10-30T13:22:51.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.649+0000 7f4ff55a1640 1 Processor -- start 2023-10-30T13:22:51.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 -- start start 2023-10-30T13:22:51.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc0073a0 0x7f4ff006a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ff01571c0 0x7f4ff013cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ff013d440 0x7f4ff01ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff01edaa0 con 0x7f4ff01571c0 2023-10-30T13:22:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff01ee900 con 0x7f4fdc0073a0 2023-10-30T13:22:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff013d8c0 con 0x7f4ff013d440 2023-10-30T13:22:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe67fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ff01571c0 0x7f4ff013cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe67fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ff01571c0 0x7f4ff013cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42376/0 (socket says 172.21.15.148:42376) 2023-10-30T13:22:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe67fc640 1 -- 172.21.15.148:0/1281666534 learned_addr learned my addr 172.21.15.148:0/1281666534 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4feeffd640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc0073a0 0x7f4ff006a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fef7fe640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ff013d440 0x7f4ff01ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe67fc640 1 -- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ff013d440 msgr2=0x7f4ff01ea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe67fc640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ff013d440 0x7f4ff01ea7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe67fc640 1 -- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc0073a0 msgr2=0x7f4ff006a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe67fc640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc0073a0 0x7f4ff006a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe67fc640 1 -- 172.21.15.148:0/1281666534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff01edcf0 con 0x7f4ff01571c0 2023-10-30T13:22:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe67fc640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ff01571c0 0x7f4ff013cf00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0011590 tx=0x7f4fe0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:51.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe7fff640 1 -- 172.21.15.148:0/1281666534 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0004220 con 0x7f4ff01571c0 2023-10-30T13:22:51.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe7fff640 1 -- 172.21.15.148:0/1281666534 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe0004380 con 0x7f4ff01571c0 2023-10-30T13:22:51.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff01edfd0 con 0x7f4ff01571c0 2023-10-30T13:22:51.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4fe7fff640 1 -- 172.21.15.148:0/1281666534 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0015870 con 0x7f4ff01571c0 2023-10-30T13:22:51.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff01ee590 con 0x7f4ff01571c0 2023-10-30T13:22:51.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.653+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff01ee750 con 0x7f4ff01571c0 2023-10-30T13:22:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.657+0000 7f4fe7fff640 1 -- 172.21.15.148:0/1281666534 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4fe00159d0 con 0x7f4ff01571c0 2023-10-30T13:22:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.657+0000 7f4fe7fff640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4fd00664a0 0x7f4fd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.657+0000 7f4fe7fff640 1 -- 172.21.15.148:0/1281666534 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f4fe00841e0 con 0x7f4ff01571c0 2023-10-30T13:22:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.657+0000 7f4feeffd640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4fd00664a0 0x7f4fd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.657+0000 7f4feeffd640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4fd00664a0 0x7f4fd0068960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8007860 tx=0x7f4fd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:51.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.661+0000 7f4fe7fff640 1 -- 172.21.15.148:0/1281666534 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4fe0051430 con 0x7f4ff01571c0 2023-10-30T13:22:51.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.841+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ff01ea530 con 0x7f4ff01571c0 2023-10-30T13:22:51.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.841+0000 7f4fe7fff640 1 -- 172.21.15.148:0/1281666534 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f4fe0054a50 con 0x7f4ff01571c0 2023-10-30T13:22:51.840 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:51.840 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:51.536464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:51.840 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 61 2023-10-30T13:22:51.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4fd00664a0 msgr2=0x7f4fd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:51.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4fd00664a0 0x7f4fd0068960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8007860 tx=0x7f4fd8009040 comp rx=0 tx=0).stop 2023-10-30T13:22:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ff01571c0 msgr2=0x7f4ff013cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ff01571c0 0x7f4ff013cf00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0011590 tx=0x7f4fe0011a50 comp rx=0 tx=0).stop 2023-10-30T13:22:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 shutdown_connections 2023-10-30T13:22:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4fd00664a0 0x7f4fd0068960 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4ff01571c0 0x7f4ff013cf00 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4ff013d440 0x7f4ff01ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 --2- 172.21.15.148:0/1281666534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fdc0073a0 0x7f4ff006a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 >> 172.21.15.148:0/1281666534 conn(0x7f4ff00c7910 msgr2=0x7f4ff00bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 shutdown_connections 2023-10-30T13:22:51.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:51.845+0000 7f4ff55a1640 1 -- 172.21.15.148:0/1281666534 wait complete. 2023-10-30T13:22:51.861 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 61, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 61, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5694": { "addr": "172.21.15.148:6837/3781833565", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3781833565, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3781833565, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5694, "incarnation": 59, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:22:51.536464+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5694 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6835/3194894584", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 3194894584, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 3194894584, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 54, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5643, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:22:53.862 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:54.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff38d446640 1 Processor -- start 2023-10-30T13:22:54.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff38d446640 1 -- start start 2023-10-30T13:22:54.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff38d446640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff388070a00 0x7ff388070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff38d446640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff388069e90 0x7ff38806a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff38d446640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3880713b0 con 0x7ff3881539a0 2023-10-30T13:22:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff38d446640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38806fbb0 con 0x7ff388070a00 2023-10-30T13:22:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff38d446640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff388153d80 con 0x7ff388069e90 2023-10-30T13:22:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff37e7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff388070a00 0x7ff388070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff37e7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff388070a00 0x7ff388070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48696/0 (socket says 172.21.15.148:48696) 2023-10-30T13:22:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.013+0000 7ff37e7fc640 1 -- 172.21.15.148:0/2455087664 learned_addr learned my addr 172.21.15.148:0/2455087664 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:54.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff3877fe640 1 --2- 172.21.15.148:0/2455087664 >> v2:172.21.15.150:3300/0 conn(0x7ff388069e90 0x7ff38806a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff37e7fc640 1 -- 172.21.15.148:0/2455087664 >> v2:172.21.15.150:3300/0 conn(0x7ff388069e90 msgr2=0x7ff38806a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:54.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff37e7fc640 1 --2- 172.21.15.148:0/2455087664 >> v2:172.21.15.150:3300/0 conn(0x7ff388069e90 0x7ff38806a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff37e7fc640 1 -- 172.21.15.148:0/2455087664 >> v1:172.21.15.150:6789/0 conn(0x7ff3881539a0 legacy=0x7ff38815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:54.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff37e7fc640 1 -- 172.21.15.148:0/2455087664 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880c5c50 con 0x7ff388070a00 2023-10-30T13:22:54.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff37e7fc640 1 --2- 172.21.15.148:0/2455087664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388070a00 0x7ff388070e70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff3880b1080 tx=0x7ff37001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfefef6fe30d70a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:54.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff3867fc640 1 -- 172.21.15.148:0/2455087664 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700a56a0 con 0x7ff388070a00 2023-10-30T13:22:54.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff3867fc640 1 -- 172.21.15.148:0/2455087664 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3700a5800 con 0x7ff388070a00 2023-10-30T13:22:54.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff3867fc640 1 -- 172.21.15.148:0/2455087664 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700a5b90 con 0x7ff388070a00 2023-10-30T13:22:54.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff38d446640 1 -- 172.21.15.148:0/2455087664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388070a00 msgr2=0x7ff388070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:54.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff38d446640 1 --2- 172.21.15.148:0/2455087664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388070a00 0x7ff388070e70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff3880b1080 tx=0x7ff37001c8d0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff38d446640 1 -- 172.21.15.148:0/2455087664 shutdown_connections 2023-10-30T13:22:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff38d446640 1 --2- 172.21.15.148:0/2455087664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388070a00 0x7ff388070e70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff38d446640 1 --2- 172.21.15.148:0/2455087664 >> v2:172.21.15.150:3300/0 conn(0x7ff388069e90 0x7ff38806a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff38d446640 1 -- 172.21.15.148:0/2455087664 >> 172.21.15.148:0/2455087664 conn(0x7ff3880c7910 msgr2=0x7ff3880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:54.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.017+0000 7ff38d446640 1 -- 172.21.15.148:0/2455087664 shutdown_connections 2023-10-30T13:22:54.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff38d446640 1 -- 172.21.15.148:0/2455087664 wait complete. 2023-10-30T13:22:54.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff38d446640 1 Processor -- start 2023-10-30T13:22:54.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff38d446640 1 -- start start 2023-10-30T13:22:54.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff38d446640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388069e90 0x7ff388143b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff38d446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff388070a00 0x7ff388144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff38d446640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3881539a0 0x7ff3881f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff38d446640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881f6300 con 0x7ff388069e90 2023-10-30T13:22:54.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff38d446640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff388144090 con 0x7ff3881539a0 2023-10-30T13:22:54.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff38d446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881f5d60 con 0x7ff388070a00 2023-10-30T13:22:54.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff386ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388069e90 0x7ff388143b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff386ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388069e90 0x7ff388143b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48708/0 (socket says 172.21.15.148:48708) 2023-10-30T13:22:54.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff386ffd640 1 -- 172.21.15.148:0/181535875 learned_addr learned my addr 172.21.15.148:0/181535875 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:54.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff3877fe640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3881539a0 0x7ff3881f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff37e7fc640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff388070a00 0x7ff388144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff386ffd640 1 -- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff388070a00 msgr2=0x7ff388144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:54.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff386ffd640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff388070a00 0x7ff388144820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff386ffd640 1 -- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3881539a0 msgr2=0x7ff3881f2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:54.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff386ffd640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3881539a0 0x7ff3881f2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff386ffd640 1 -- 172.21.15.148:0/181535875 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff370009b30 con 0x7ff388069e90 2023-10-30T13:22:54.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff3877fe640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3881539a0 0x7ff3881f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:54.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff386ffd640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388069e90 0x7ff388143b50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff37800b440 tx=0x7ff37800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:54.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.021+0000 7ff37ffff640 1 -- 172.21.15.148:0/181535875 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378011890 con 0x7ff388069e90 2023-10-30T13:22:54.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.025+0000 7ff37ffff640 1 -- 172.21.15.148:0/181535875 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3780119f0 con 0x7ff388069e90 2023-10-30T13:22:54.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.025+0000 7ff37ffff640 1 -- 172.21.15.148:0/181535875 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378011da0 con 0x7ff388069e90 2023-10-30T13:22:54.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.025+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3881f6020 con 0x7ff388069e90 2023-10-30T13:22:54.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.025+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3881f1ef0 con 0x7ff388069e90 2023-10-30T13:22:54.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.025+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3880c8c10 con 0x7ff388069e90 2023-10-30T13:22:54.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.025+0000 7ff37ffff640 1 -- 172.21.15.148:0/181535875 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff37801a400 con 0x7ff388069e90 2023-10-30T13:22:54.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.025+0000 7ff37ffff640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3680664d0 0x7ff368068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:54.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.025+0000 7ff37ffff640 1 -- 172.21.15.148:0/181535875 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7ff378084170 con 0x7ff388069e90 2023-10-30T13:22:54.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.025+0000 7ff37e7fc640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3680664d0 0x7ff368068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:54.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.029+0000 7ff37e7fc640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3680664d0 0x7ff368068990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff3700ae000 tx=0x7ff3700a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:54.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.029+0000 7ff37ffff640 1 -- 172.21.15.148:0/181535875 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3880c8c10 con 0x7ff388069e90 2023-10-30T13:22:54.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.209+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3880c8c10 con 0x7ff388069e90 2023-10-30T13:22:54.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.209+0000 7ff37ffff640 1 -- 172.21.15.148:0/181535875 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5720 (secure 0 0 0) 0x7ff3880c8c10 con 0x7ff388069e90 2023-10-30T13:22:54.207 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:54.207 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:51.536464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:54.207 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 61 2023-10-30T13:22:54.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.209+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3680664d0 msgr2=0x7ff368068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:54.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.209+0000 7ff38d446640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3680664d0 0x7ff368068990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff3700ae000 tx=0x7ff3700a7070 comp rx=0 tx=0).stop 2023-10-30T13:22:54.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.209+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388069e90 msgr2=0x7ff388143b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:54.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.209+0000 7ff38d446640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388069e90 0x7ff388143b50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff37800b440 tx=0x7ff37800b900 comp rx=0 tx=0).stop 2023-10-30T13:22:54.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.213+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 shutdown_connections 2023-10-30T13:22:54.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.213+0000 7ff38d446640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3680664d0 0x7ff368068990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.213+0000 7ff38d446640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3881539a0 0x7ff3881f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.213+0000 7ff38d446640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff388070a00 0x7ff388144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.213+0000 7ff38d446640 1 --2- 172.21.15.148:0/181535875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff388069e90 0x7ff388143b50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:54.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.213+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 >> 172.21.15.148:0/181535875 conn(0x7ff3880c7910 msgr2=0x7ff38814f510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:54.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.213+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 shutdown_connections 2023-10-30T13:22:54.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:54.213+0000 7ff38d446640 1 -- 172.21.15.148:0/181535875 wait complete. 2023-10-30T13:22:54.227 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:55.324 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:55.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:56.228 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:56.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d602c9640 1 Processor -- start 2023-10-30T13:22:56.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d602c9640 1 -- start start 2023-10-30T13:22:56.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d602c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d581538a0 0x7f3d5815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d602c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3d580ca0e0 0x7f3d5815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d602c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d580cb800 con 0x7f3d580cf160 2023-10-30T13:22:56.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d602c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5815dc30 con 0x7f3d580ca0e0 2023-10-30T13:22:56.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d602c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d580cb330 con 0x7f3d581538a0 2023-10-30T13:22:56.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5e065640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d581538a0 0x7f3d5815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5d864640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3d580ca0e0 0x7f3d5815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5e866640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3d580cf160 0x7f3d580cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:40096/0 (socket says 172.21.15.148:40096) 2023-10-30T13:22:56.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5e065640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d581538a0 0x7f3d5815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:48648/0 (socket says 172.21.15.148:48648) 2023-10-30T13:22:56.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5e866640 1 -- 172.21.15.148:0/825880422 learned_addr learned my addr 172.21.15.148:0/825880422 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:22:56.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5d864640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3d580ca0e0 0x7f3d5815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:37316/0 (socket says 172.21.15.148:37316) 2023-10-30T13:22:56.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5d864640 1 -- 172.21.15.148:0/825880422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d581538a0 msgr2=0x7f3d5815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:56.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5d864640 1 --2- 172.21.15.148:0/825880422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d581538a0 0x7f3d5815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5d864640 1 -- 172.21.15.148:0/825880422 >> v1:172.21.15.150:6789/0 conn(0x7f3d580cf160 legacy=0x7f3d580cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5d864640 1 -- 172.21.15.148:0/825880422 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580cad70 con 0x7f3d580ca0e0 2023-10-30T13:22:56.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.381+0000 7f3d5e065640 1 --2- 172.21.15.148:0/825880422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d581538a0 0x7f3d5815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:56.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d5d864640 1 --2- 172.21.15.148:0/825880422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d580ca0e0 0x7f3d5815a090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097c0 tx=0x7f3d4801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a09ce794b7dea05d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d5d063640 1 -- 172.21.15.148:0/825880422 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480a8970 con 0x7f3d580ca0e0 2023-10-30T13:22:56.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d5d063640 1 -- 172.21.15.148:0/825880422 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d480a8ad0 con 0x7f3d580ca0e0 2023-10-30T13:22:56.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d5d063640 1 -- 172.21.15.148:0/825880422 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480a8e60 con 0x7f3d580ca0e0 2023-10-30T13:22:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d602c9640 1 -- 172.21.15.148:0/825880422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d580ca0e0 msgr2=0x7f3d5815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d602c9640 1 --2- 172.21.15.148:0/825880422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d580ca0e0 0x7f3d5815a090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097c0 tx=0x7f3d4801ec70 comp rx=0 tx=0).stop 2023-10-30T13:22:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d602c9640 1 -- 172.21.15.148:0/825880422 shutdown_connections 2023-10-30T13:22:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d602c9640 1 --2- 172.21.15.148:0/825880422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d580ca0e0 0x7f3d5815a090 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d602c9640 1 --2- 172.21.15.148:0/825880422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d581538a0 0x7f3d5815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d602c9640 1 -- 172.21.15.148:0/825880422 >> 172.21.15.148:0/825880422 conn(0x7f3d5806bfe0 msgr2=0x7f3d5806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d602c9640 1 -- 172.21.15.148:0/825880422 shutdown_connections 2023-10-30T13:22:56.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.385+0000 7f3d602c9640 1 -- 172.21.15.148:0/825880422 wait complete. 2023-10-30T13:22:56.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 Processor -- start 2023-10-30T13:22:56.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 -- start start 2023-10-30T13:22:56.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d580ca0e0 0x7f3d58147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d580cf160 0x7f3d5815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d581538a0 0x7f3d5815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d581f0c50 con 0x7f3d581538a0 2023-10-30T13:22:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d581436f0 con 0x7f3d580ca0e0 2023-10-30T13:22:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5815acf0 con 0x7f3d580cf160 2023-10-30T13:22:56.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e866640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d581538a0 0x7f3d5815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5d864640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d580cf160 0x7f3d5815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e866640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d581538a0 0x7f3d5815ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48718/0 (socket says 172.21.15.148:48718) 2023-10-30T13:22:56.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e866640 1 -- 172.21.15.148:0/3348854963 learned_addr learned my addr 172.21.15.148:0/3348854963 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:56.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e065640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d580ca0e0 0x7f3d58147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e866640 1 -- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d580cf160 msgr2=0x7f3d5815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e866640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d580cf160 0x7f3d5815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e866640 1 -- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d580ca0e0 msgr2=0x7f3d58147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e866640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d580ca0e0 0x7f3d58147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e866640 1 -- 172.21.15.148:0/3348854963 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d54009420 con 0x7f3d581538a0 2023-10-30T13:22:56.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d5e866640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d581538a0 0x7f3d5815ca40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000ca70 tx=0x7f3d5000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d4effd640 1 -- 172.21.15.148:0/3348854963 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5000ab70 con 0x7f3d581538a0 2023-10-30T13:22:56.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d4effd640 1 -- 172.21.15.148:0/3348854963 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5000acd0 con 0x7f3d581538a0 2023-10-30T13:22:56.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d48009420 con 0x7f3d581538a0 2023-10-30T13:22:56.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d4effd640 1 -- 172.21.15.148:0/3348854963 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50012600 con 0x7f3d581538a0 2023-10-30T13:22:56.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.389+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5815b2b0 con 0x7f3d581538a0 2023-10-30T13:22:56.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.393+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d580b2a50 con 0x7f3d581538a0 2023-10-30T13:22:56.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.393+0000 7f3d4effd640 1 -- 172.21.15.148:0/3348854963 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d50007070 con 0x7f3d581538a0 2023-10-30T13:22:56.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.393+0000 7f3d4effd640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d3c0664a0 0x7f3d3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:56.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.393+0000 7f3d4effd640 1 -- 172.21.15.148:0/3348854963 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f3d50083f80 con 0x7f3d581538a0 2023-10-30T13:22:56.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.393+0000 7f3d5e065640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d3c0664a0 0x7f3d3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:56.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.393+0000 7f3d5e065640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d3c0664a0 0x7f3d3c068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3d54009c50 tx=0x7f3d54009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:56.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.397+0000 7f3d4effd640 1 -- 172.21.15.148:0/3348854963 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d50051140 con 0x7f3d581538a0 2023-10-30T13:22:56.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.581+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d581f0d90 con 0x7f3d581538a0 2023-10-30T13:22:56.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.581+0000 7f3d4effd640 1 -- 172.21.15.148:0/3348854963 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f3d581f0d90 con 0x7f3d581538a0 2023-10-30T13:22:56.580 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:56.581 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:51.536464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:56.581 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 61 2023-10-30T13:22:56.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d3c0664a0 msgr2=0x7f3d3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d3c0664a0 0x7f3d3c068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3d54009c50 tx=0x7f3d54009190 comp rx=0 tx=0).stop 2023-10-30T13:22:56.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d581538a0 msgr2=0x7f3d5815ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:56.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d581538a0 0x7f3d5815ca40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000ca70 tx=0x7f3d5000cf30 comp rx=0 tx=0).stop 2023-10-30T13:22:56.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 shutdown_connections 2023-10-30T13:22:56.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3d3c0664a0 0x7f3d3c068960 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d581538a0 0x7f3d5815ca40 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3d580cf160 0x7f3d5815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 --2- 172.21.15.148:0/3348854963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d580ca0e0 0x7f3d58147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:56.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 >> 172.21.15.148:0/3348854963 conn(0x7f3d5806bfe0 msgr2=0x7f3d580cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:56.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 shutdown_connections 2023-10-30T13:22:56.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:56.585+0000 7f3d602c9640 1 -- 172.21.15.148:0/3348854963 wait complete. 2023-10-30T13:22:56.601 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:58.602 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:58.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f41a53f8640 1 Processor -- start 2023-10-30T13:22:58.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f41a53f8640 1 -- start start 2023-10-30T13:22:58.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f41a53f8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41a01538a0 0x7f41a015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f41a53f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a00cb800 con 0x7f41a00ca0e0 2023-10-30T13:22:58.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f41a53f8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a015dc30 con 0x7f41a00cf160 2023-10-30T13:22:58.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f41a53f8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a00cb330 con 0x7f41a01538a0 2023-10-30T13:22:58.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f41967fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f41a00ca0e0 0x7f41a015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43876/0 (socket says 172.21.15.148:43876) 2023-10-30T13:22:58.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f41967fc640 1 -- 172.21.15.148:0/204168550 learned_addr learned my addr 172.21.15.148:0/204168550 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:22:58.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f419effd640 1 --2- 172.21.15.148:0/204168550 >> v2:172.21.15.150:3300/0 conn(0x7f41a01538a0 0x7f41a015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631222918 0 0) 0x7f41a00cb800 con 0x7f41a00ca0e0 2023-10-30T13:22:58.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.753+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f418c003530 con 0x7f41a00ca0e0 2023-10-30T13:22:58.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959866850 0 0) 0x7f418c003530 con 0x7f41a00ca0e0 2023-10-30T13:22:58.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 >> v2:172.21.15.150:3300/0 conn(0x7f41a01538a0 msgr2=0x7f41a015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:58.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 --2- 172.21.15.148:0/204168550 >> v2:172.21.15.150:3300/0 conn(0x7f41a01538a0 0x7f41a015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 >> v1:172.21.15.150:6789/0 conn(0x7f41a00cf160 legacy=0x7f41a0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a00cad70 con 0x7f41a00ca0e0 2023-10-30T13:22:58.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f4188001680 con 0x7f41a00ca0e0 2023-10-30T13:22:58.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 >> v1:172.21.15.148:6789/0 conn(0x7f41a00ca0e0 legacy=0x7f41a015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f418c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f418c0047d0 0x7f418c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f418c005190 0x7f418c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a00cb330 con 0x7f418c003980 2023-10-30T13:22:58.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c00c040 con 0x7f418c0047d0 2023-10-30T13:22:58.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c00c200 con 0x7f418c005190 2023-10-30T13:22:58.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419effd640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f418c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f41967fc640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f418c0047d0 0x7f418c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419f7fe640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f418c005190 0x7f418c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419effd640 1 -- 172.21.15.148:0/204168550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f418c005190 msgr2=0x7f418c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419effd640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f418c005190 0x7f418c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419effd640 1 -- 172.21.15.148:0/204168550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f418c0047d0 msgr2=0x7f418c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419effd640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f418c0047d0 0x7f418c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419effd640 1 -- 172.21.15.148:0/204168550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c003530 con 0x7f418c003980 2023-10-30T13:22:58.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f41967fc640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f418c0047d0 0x7f418c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:58.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419f7fe640 1 -- 172.21.15.148:0/204168550 reap_dead start 2023-10-30T13:22:58.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419effd640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f418c003da0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4190009ae0 tx=0x7f419001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d632834dc3ce4ac0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:58.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41900a5550 con 0x7f418c003980 2023-10-30T13:22:58.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41900a56b0 con 0x7f418c003980 2023-10-30T13:22:58.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.757+0000 7f419e7fc640 1 -- 172.21.15.148:0/204168550 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41900a5a60 con 0x7f418c003980 2023-10-30T13:22:58.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 -- 172.21.15.148:0/204168550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 msgr2=0x7f418c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f418c003da0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4190009ae0 tx=0x7f419001ca70 comp rx=0 tx=0).stop 2023-10-30T13:22:58.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 -- 172.21.15.148:0/204168550 shutdown_connections 2023-10-30T13:22:58.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 --2- 172.21.15.148:0/204168550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f418c003da0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 -- 172.21.15.148:0/204168550 >> 172.21.15.148:0/204168550 conn(0x7f41a006bfe0 msgr2=0x7f41a006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:58.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 -- 172.21.15.148:0/204168550 shutdown_connections 2023-10-30T13:22:58.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 -- 172.21.15.148:0/204168550 wait complete. 2023-10-30T13:22:58.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 Processor -- start 2023-10-30T13:22:58.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 -- start start 2023-10-30T13:22:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f41a00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41a00cebe0 0x7f41a00b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a01f3130 0x7f41a01409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a00b4830 con 0x7f418c003980 2023-10-30T13:22:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a01f3ae0 con 0x7f41a01f3130 2023-10-30T13:22:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41a53f8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a01f3da0 con 0x7f41a00cebe0 2023-10-30T13:22:58.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f41a00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f41a00ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48760/0 (socket says 172.21.15.148:48760) 2023-10-30T13:22:58.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419effd640 1 -- 172.21.15.148:0/1191946209 learned_addr learned my addr 172.21.15.148:0/1191946209 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:22:58.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419f7fe640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a01f3130 0x7f41a01409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41967fc640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41a00cebe0 0x7f41a00b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419effd640 1 -- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41a00cebe0 msgr2=0x7f41a00b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419effd640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41a00cebe0 0x7f41a00b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419effd640 1 -- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a01f3130 msgr2=0x7f41a01409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419effd640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a01f3130 0x7f41a01409f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419effd640 1 -- 172.21.15.148:0/1191946209 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a00b32b0 con 0x7f418c003980 2023-10-30T13:22:58.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f419f7fe640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a01f3130 0x7f41a01409f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:22:58.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.761+0000 7f41967fc640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41a00cebe0 0x7f41a00b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:22:58.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.765+0000 7f419effd640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f41a00ccfb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f419001c8a0 tx=0x7f4190019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:58.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.765+0000 7f4197fff640 1 -- 172.21.15.148:0/1191946209 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419000f0c0 con 0x7f418c003980 2023-10-30T13:22:58.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.765+0000 7f4197fff640 1 -- 172.21.15.148:0/1191946209 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4190004020 con 0x7f418c003980 2023-10-30T13:22:58.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.765+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a00b44c0 con 0x7f418c003980 2023-10-30T13:22:58.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.765+0000 7f4197fff640 1 -- 172.21.15.148:0/1191946209 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41900043d0 con 0x7f418c003980 2023-10-30T13:22:58.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.765+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a0150bb0 con 0x7f418c003980 2023-10-30T13:22:58.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.765+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a00b4650 con 0x7f418c003980 2023-10-30T13:22:58.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.765+0000 7f4197fff640 1 -- 172.21.15.148:0/1191946209 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41900b4070 con 0x7f418c003980 2023-10-30T13:22:58.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.765+0000 7f4197fff640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41800664d0 0x7f4180068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:58.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.769+0000 7f4197fff640 1 -- 172.21.15.148:0/1191946209 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f419011e610 con 0x7f418c003980 2023-10-30T13:22:58.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.769+0000 7f41967fc640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41800664d0 0x7f4180068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:58.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.769+0000 7f41967fc640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41800664d0 0x7f4180068990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f41880017b0 tx=0x7f41880047f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:58.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.773+0000 7f4197fff640 1 -- 172.21.15.148:0/1191946209 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41900eb7c0 con 0x7f418c003980 2023-10-30T13:22:58.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.953+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41a004e170 con 0x7f418c003980 2023-10-30T13:22:58.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.953+0000 7f4197fff640 1 -- 172.21.15.148:0/1191946209 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f41900eede0 con 0x7f418c003980 2023-10-30T13:22:58.951 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:22:58.952 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:58.952 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 62 2023-10-30T13:22:58.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41800664d0 msgr2=0x7f4180068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41800664d0 0x7f4180068990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f41880017b0 tx=0x7f41880047f0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 msgr2=0x7f41a00ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:58.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f41a00ccfb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f419001c8a0 tx=0x7f4190019810 comp rx=0 tx=0).stop 2023-10-30T13:22:58.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 shutdown_connections 2023-10-30T13:22:58.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41800664d0 0x7f4180068990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a01f3130 0x7f41a01409f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41a00cebe0 0x7f41a00b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 --2- 172.21.15.148:0/1191946209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f418c003980 0x7f41a00ccfb0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 >> 172.21.15.148:0/1191946209 conn(0x7f41a006bfe0 msgr2=0x7f41a0142d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:58.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 shutdown_connections 2023-10-30T13:22:58.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:22:58.957+0000 7f41a53f8640 1 -- 172.21.15.148:0/1191946209 wait complete. 2023-10-30T13:22:58.972 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=5033 2023-10-30T13:22:58.972 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2023-10-30T13:23:02.973 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:23:02.973 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:23:02.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:23:02.977 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:23:02.977 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:03.053 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2023-10-30T13:23:03.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.121+0000 7f086729f640 1 Processor -- start 2023-10-30T13:23:03.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086729f640 1 -- start start 2023-10-30T13:23:03.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086729f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086015cd50 0x7f08600c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:03.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086729f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f08600cefa0 0x7f08600cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:03.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086729f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086015a6c0 con 0x7f08601538f0 2023-10-30T13:23:03.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086729f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086015d300 con 0x7f08600cefa0 2023-10-30T13:23:03.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086729f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086015d5e0 con 0x7f086015cd50 2023-10-30T13:23:03.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086629d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f08601538f0 0x7f0860159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59572/0 (socket says 172.21.15.148:59572) 2023-10-30T13:23:03.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086629d640 1 -- 172.21.15.148:0/1228047923 learned_addr learned my addr 172.21.15.148:0/1228047923 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:03.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f0866a9e640 1 --2- 172.21.15.148:0/1228047923 >> v2:172.21.15.150:3300/0 conn(0x7f08600cefa0 0x7f08600cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:03.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f0865a9c640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086015cd50 0x7f08600c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:03.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274435734 0 0) 0x7f086015a6c0 con 0x7f08601538f0 2023-10-30T13:23:03.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f084c003530 con 0x7f08601538f0 2023-10-30T13:23:03.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3369632043 0 0) 0x7f084c003530 con 0x7f08601538f0 2023-10-30T13:23:03.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086015cd50 msgr2=0x7f08600c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:03.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086015cd50 0x7f08600c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 >> v2:172.21.15.150:3300/0 conn(0x7f08600cefa0 msgr2=0x7f08600cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:03.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 --2- 172.21.15.148:0/1228047923 >> v2:172.21.15.150:3300/0 conn(0x7f08600cefa0 0x7f08600cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086015d8c0 con 0x7f08601538f0 2023-10-30T13:23:03.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f085c001680 con 0x7f08601538f0 2023-10-30T13:23:03.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 >> v1:172.21.15.148:6789/0 conn(0x7f08601538f0 legacy=0x7f0860159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:03.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f084c003980 0x7f084c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:03.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086015cd50 0x7f084c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:03.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c004d10 0x7f084c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:03.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086015d300 con 0x7f084c003980 2023-10-30T13:23:03.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086015d5e0 con 0x7f084c004d10 2023-10-30T13:23:03.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c00b990 con 0x7f086015cd50 2023-10-30T13:23:03.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086629d640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f084c003980 0x7f084c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:03.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f0865a9c640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086015cd50 0x7f084c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:03.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f0866a9e640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c004d10 0x7f084c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:03.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086629d640 1 -- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086015cd50 msgr2=0x7f084c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:03.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086629d640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086015cd50 0x7f084c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086629d640 1 -- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c004d10 msgr2=0x7f084c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:03.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086629d640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c004d10 0x7f084c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.125+0000 7f086629d640 1 -- 172.21.15.148:0/1228047923 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c003530 con 0x7f084c003980 2023-10-30T13:23:03.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086629d640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f084c003980 0x7f084c003da0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f085c00c240 tx=0x7f085c01f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8cc6d94672e6b5b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:03.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c005ae0 con 0x7f084c003980 2023-10-30T13:23:03.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085c0ab070 con 0x7f084c003980 2023-10-30T13:23:03.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086529b640 1 -- 172.21.15.148:0/1228047923 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c005ea0 con 0x7f084c003980 2023-10-30T13:23:03.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086729f640 1 -- 172.21.15.148:0/1228047923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f084c003980 msgr2=0x7f084c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:03.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086729f640 1 --2- 172.21.15.148:0/1228047923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f084c003980 0x7f084c003da0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f085c00c240 tx=0x7f085c01f3d0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f0866a9e640 1 -- 172.21.15.148:0/1228047923 reap_dead start 2023-10-30T13:23:03.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086729f640 1 -- 172.21.15.148:0/1228047923 shutdown_connections 2023-10-30T13:23:03.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086729f640 1 -- 172.21.15.148:0/1228047923 >> 172.21.15.148:0/1228047923 conn(0x7f086006a640 msgr2=0x7f0860065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:03.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086729f640 1 -- 172.21.15.148:0/1228047923 shutdown_connections 2023-10-30T13:23:03.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.129+0000 7f086729f640 1 -- 172.21.15.148:0/1228047923 wait complete. 2023-10-30T13:23:03.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 Processor -- start 2023-10-30T13:23:03.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 -- start start 2023-10-30T13:23:03.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c003980 0x7f0860152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:03.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086014d480 0x7f08600ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:03.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086015abe0 0x7f086015b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:03.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08601f3290 con 0x7f086015abe0 2023-10-30T13:23:03.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08601f2430 con 0x7f084c003980 2023-10-30T13:23:03.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0860152fa0 con 0x7f086014d480 2023-10-30T13:23:03.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0866a9e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086015abe0 0x7f086015b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:03.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0866a9e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086015abe0 0x7f086015b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54064/0 (socket says 172.21.15.148:54064) 2023-10-30T13:23:03.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0866a9e640 1 -- 172.21.15.148:0/38967624 learned_addr learned my addr 172.21.15.148:0/38967624 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0865a9c640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086014d480 0x7f08600ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086629d640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c003980 0x7f0860152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0866a9e640 1 -- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086014d480 msgr2=0x7f08600ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0866a9e640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086014d480 0x7f08600ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0866a9e640 1 -- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c003980 msgr2=0x7f0860152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0866a9e640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c003980 0x7f0860152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0866a9e640 1 -- 172.21.15.148:0/38967624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08601f2680 con 0x7f086015abe0 2023-10-30T13:23:03.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0866a9e640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086015abe0 0x7f086015b060 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0860066500 tx=0x7f085800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:03.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0856ffd640 1 -- 172.21.15.148:0/38967624 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085800e040 con 0x7f086015abe0 2023-10-30T13:23:03.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0856ffd640 1 -- 172.21.15.148:0/38967624 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085800f040 con 0x7f086015abe0 2023-10-30T13:23:03.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08601f2960 con 0x7f086015abe0 2023-10-30T13:23:03.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f0856ffd640 1 -- 172.21.15.148:0/38967624 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0858015430 con 0x7f086015abe0 2023-10-30T13:23:03.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.133+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08601f2f20 con 0x7f086015abe0 2023-10-30T13:23:03.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.137+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0860150cd0 con 0x7f086015abe0 2023-10-30T13:23:03.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.137+0000 7f0856ffd640 1 -- 172.21.15.148:0/38967624 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0858015590 con 0x7f086015abe0 2023-10-30T13:23:03.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.137+0000 7f0856ffd640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08440664a0 0x7f0844068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:03.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.137+0000 7f0856ffd640 1 -- 172.21.15.148:0/38967624 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f08580846a0 con 0x7f086015abe0 2023-10-30T13:23:03.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.137+0000 7f086629d640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08440664a0 0x7f0844068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:03.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.137+0000 7f086629d640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08440664a0 0x7f0844068960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f085c0ae000 tx=0x7f085c0b4040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:03.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.141+0000 7f0856ffd640 1 -- 172.21.15.148:0/38967624 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0858051840 con 0x7f086015abe0 2023-10-30T13:23:03.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.321+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08600c5d50 con 0x7f086015abe0 2023-10-30T13:23:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.321+0000 7f0856ffd640 1 -- 172.21.15.148:0/38967624 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f0858054e60 con 0x7f086015abe0 2023-10-30T13:23:03.318 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:03.319 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:03.319 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 62 2023-10-30T13:23:03.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08440664a0 msgr2=0x7f0844068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:03.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08440664a0 0x7f0844068960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f085c0ae000 tx=0x7f085c0b4040 comp rx=0 tx=0).stop 2023-10-30T13:23:03.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086015abe0 msgr2=0x7f086015b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086015abe0 0x7f086015b060 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0860066500 tx=0x7f085800f1b0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 shutdown_connections 2023-10-30T13:23:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08440664a0 0x7f0844068960 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086015abe0 0x7f086015b060 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f086014d480 0x7f08600ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 --2- 172.21.15.148:0/38967624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f084c003980 0x7f0860152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 >> 172.21.15.148:0/38967624 conn(0x7f086006a640 msgr2=0x7f0860065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 shutdown_connections 2023-10-30T13:23:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:03.325+0000 7f086729f640 1 -- 172.21.15.148:0/38967624 wait complete. 2023-10-30T13:23:03.339 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:23:04.606 INFO:tasks.ceph.mds.c.smithi148.stderr:2023-10-30T13:23:04.597+0000 7f76db6df640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:23:05.341 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:05.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.481+0000 7f8cd69f5640 1 Processor -- start 2023-10-30T13:23:05.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.481+0000 7f8cd69f5640 1 -- start start 2023-10-30T13:23:05.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8cd69f5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8cd00c6bd0 0x7f8cd006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8cd69f5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd0071110 0x7f8cd0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8cd69f5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd0066cf0 con 0x7f8cd006f7a0 2023-10-30T13:23:05.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8cd69f5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd00700c0 con 0x7f8cd00c6bd0 2023-10-30T13:23:05.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8cd69f5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00c70d0 con 0x7f8cd0071110 2023-10-30T13:23:05.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8ccf7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8cd00c6bd0 0x7f8cd006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8ccf7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8cd00c6bd0 0x7f8cd006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54078/0 (socket says 172.21.15.148:54078) 2023-10-30T13:23:05.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8ccf7fe640 1 -- 172.21.15.148:0/1404032639 learned_addr learned my addr 172.21.15.148:0/1404032639 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:05.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8cd4f92640 1 --2- 172.21.15.148:0/1404032639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd0071110 0x7f8cd0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8ccf7fe640 1 -- 172.21.15.148:0/1404032639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd0071110 msgr2=0x7f8cd0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:05.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8ccf7fe640 1 --2- 172.21.15.148:0/1404032639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd0071110 0x7f8cd0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8ccf7fe640 1 -- 172.21.15.148:0/1404032639 >> v1:172.21.15.150:6789/0 conn(0x7f8cd006f7a0 legacy=0x7f8cd0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8ccf7fe640 1 -- 172.21.15.148:0/1404032639 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd00c7390 con 0x7f8cd00c6bd0 2023-10-30T13:23:05.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8ccf7fe640 1 --2- 172.21.15.148:0/1404032639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd00c6bd0 0x7f8cd006fb80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009870 tx=0x7f8cc401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8c240d56f7b01d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:05.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8cceffd640 1 -- 172.21.15.148:0/1404032639 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc40a8a30 con 0x7f8cd00c6bd0 2023-10-30T13:23:05.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8cceffd640 1 -- 172.21.15.148:0/1404032639 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc40a8b90 con 0x7f8cd00c6bd0 2023-10-30T13:23:05.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.485+0000 7f8cceffd640 1 -- 172.21.15.148:0/1404032639 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc40b0440 con 0x7f8cd00c6bd0 2023-10-30T13:23:05.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 -- 172.21.15.148:0/1404032639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd00c6bd0 msgr2=0x7f8cd006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 --2- 172.21.15.148:0/1404032639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd00c6bd0 0x7f8cd006fb80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009870 tx=0x7f8cc401ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:05.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 -- 172.21.15.148:0/1404032639 shutdown_connections 2023-10-30T13:23:05.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 --2- 172.21.15.148:0/1404032639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd0071110 0x7f8cd0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 --2- 172.21.15.148:0/1404032639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd00c6bd0 0x7f8cd006fb80 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 -- 172.21.15.148:0/1404032639 >> 172.21.15.148:0/1404032639 conn(0x7f8cd00c7910 msgr2=0x7f8cd00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:05.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 -- 172.21.15.148:0/1404032639 shutdown_connections 2023-10-30T13:23:05.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 -- 172.21.15.148:0/1404032639 wait complete. 2023-10-30T13:23:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 Processor -- start 2023-10-30T13:23:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 -- start start 2023-10-30T13:23:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd006f7a0 0x7f8cd01e4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cd0071110 0x7f8cd014fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd00c6bd0 0x7f8cd0153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01e30c0 con 0x7f8cd006f7a0 2023-10-30T13:23:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01e2810 con 0x7f8cd0071110 2023-10-30T13:23:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd69f5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00c6880 con 0x7f8cd00c6bd0 2023-10-30T13:23:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8ccffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd006f7a0 0x7f8cd01e4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8ccffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd006f7a0 0x7f8cd01e4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54092/0 (socket says 172.21.15.148:54092) 2023-10-30T13:23:05.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8ccffff640 1 -- 172.21.15.148:0/2737398534 learned_addr learned my addr 172.21.15.148:0/2737398534 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:05.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8ccf7fe640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cd0071110 0x7f8cd014fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.489+0000 7f8cd4f92640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd00c6bd0 0x7f8cd0153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8ccffff640 1 -- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd00c6bd0 msgr2=0x7f8cd0153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8ccffff640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd00c6bd0 0x7f8cd0153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8ccffff640 1 -- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cd0071110 msgr2=0x7f8cd014fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8ccffff640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cd0071110 0x7f8cd014fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8ccffff640 1 -- 172.21.15.148:0/2737398534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40094d0 con 0x7f8cd006f7a0 2023-10-30T13:23:05.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8ccf7fe640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cd0071110 0x7f8cd014fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:05.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8ccffff640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd006f7a0 0x7f8cd01e4210 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000c3c0 tx=0x7f8cc000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:05.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8cccff9640 1 -- 172.21.15.148:0/2737398534 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00052d0 con 0x7f8cd006f7a0 2023-10-30T13:23:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8cccff9640 1 -- 172.21.15.148:0/2737398534 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0005430 con 0x7f8cd006f7a0 2023-10-30T13:23:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8cccff9640 1 -- 172.21.15.148:0/2737398534 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0003840 con 0x7f8cd006f7a0 2023-10-30T13:23:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd01e2ca0 con 0x7f8cd006f7a0 2023-10-30T13:23:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd00b0320 con 0x7f8cd006f7a0 2023-10-30T13:23:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd004e170 con 0x7f8cd006f7a0 2023-10-30T13:23:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8cccff9640 1 -- 172.21.15.148:0/2737398534 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8cc00039a0 con 0x7f8cd006f7a0 2023-10-30T13:23:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8cccff9640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8cbc0664d0 0x7f8cbc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8cccff9640 1 -- 172.21.15.148:0/2737398534 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f8cc00836b0 con 0x7f8cd006f7a0 2023-10-30T13:23:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.493+0000 7f8ccf7fe640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8cbc0664d0 0x7f8cbc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.497+0000 7f8ccf7fe640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8cbc0664d0 0x7f8cbc068990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40b2000 tx=0x7f8cc40aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:05.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.497+0000 7f8cccff9640 1 -- 172.21.15.148:0/2737398534 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8cc0050850 con 0x7f8cd006f7a0 2023-10-30T13:23:05.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.697+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cd01e25e0 con 0x7f8cd006f7a0 2023-10-30T13:23:05.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.697+0000 7f8cccff9640 1 -- 172.21.15.148:0/2737398534 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f8cc0053e70 con 0x7f8cd006f7a0 2023-10-30T13:23:05.695 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:05.695 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:05.695 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 63 2023-10-30T13:23:05.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8cbc0664d0 msgr2=0x7f8cbc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8cbc0664d0 0x7f8cbc068990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40b2000 tx=0x7f8cc40aa070 comp rx=0 tx=0).stop 2023-10-30T13:23:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd006f7a0 msgr2=0x7f8cd01e4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd006f7a0 0x7f8cd01e4210 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000c3c0 tx=0x7f8cc000c880 comp rx=0 tx=0).stop 2023-10-30T13:23:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 shutdown_connections 2023-10-30T13:23:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8cbc0664d0 0x7f8cbc068990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cd00c6bd0 0x7f8cd0153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cd0071110 0x7f8cd014fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 --2- 172.21.15.148:0/2737398534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cd006f7a0 0x7f8cd01e4210 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 >> 172.21.15.148:0/2737398534 conn(0x7f8cd00c7910 msgr2=0x7f8cd00c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 shutdown_connections 2023-10-30T13:23:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:05.701+0000 7f8cd69f5640 1 -- 172.21.15.148:0/2737398534 wait complete. 2023-10-30T13:23:05.716 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:23:05.716 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2023-10-30T13:23:12.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:12.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.861+0000 7f670c9f8640 1 Processor -- start 2023-10-30T13:23:12.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.861+0000 7f670c9f8640 1 -- start start 2023-10-30T13:23:12.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.861+0000 7f670c9f8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67080c6bd0 0x7f670806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.861+0000 7f670c9f8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6708071110 0x7f6708158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.861+0000 7f670c9f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6708066cf0 con 0x7f670806f7a0 2023-10-30T13:23:12.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.861+0000 7f670c9f8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67080700c0 con 0x7f67080c6bd0 2023-10-30T13:23:12.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.861+0000 7f670c9f8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67080c70d0 con 0x7f6708071110 2023-10-30T13:23:12.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f670659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f670806f7a0 0x7f6708069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43134/0 (socket says 172.21.15.148:43134) 2023-10-30T13:23:12.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f670659c640 1 -- 172.21.15.148:0/3450389682 learned_addr learned my addr 172.21.15.148:0/3450389682 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:12.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f66fffff640 1 --2- 172.21.15.148:0/3450389682 >> v2:172.21.15.150:3300/0 conn(0x7f67080c6bd0 0x7f670806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f6706d9d640 1 --2- 172.21.15.148:0/3450389682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6708071110 0x7f6708158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f6705d9b640 1 -- 172.21.15.148:0/3450389682 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765316686 0 0) 0x7f6708066cf0 con 0x7f670806f7a0 2023-10-30T13:23:12.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f6705d9b640 1 -- 172.21.15.148:0/3450389682 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66f4003530 con 0x7f670806f7a0 2023-10-30T13:23:12.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f66fffff640 1 -- 172.21.15.148:0/3450389682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6708071110 msgr2=0x7f6708158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:12.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f66fffff640 1 --2- 172.21.15.148:0/3450389682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6708071110 0x7f6708158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f66fffff640 1 -- 172.21.15.148:0/3450389682 >> v1:172.21.15.148:6789/0 conn(0x7f670806f7a0 legacy=0x7f6708069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f6705d9b640 1 -- 172.21.15.148:0/3450389682 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1297623569 0 0) 0x7f66f4003530 con 0x7f670806f7a0 2023-10-30T13:23:12.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f66fffff640 1 -- 172.21.15.148:0/3450389682 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080c7390 con 0x7f67080c6bd0 2023-10-30T13:23:12.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f6706d9d640 1 --2- 172.21.15.148:0/3450389682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6708071110 0x7f6708158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:12.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.865+0000 7f66fffff640 1 --2- 172.21.15.148:0/3450389682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67080c6bd0 0x7f670806fb80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097c0 tx=0x7f66f001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd5866ed783fafb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:12.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f6705d9b640 1 -- 172.21.15.148:0/3450389682 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f8001680 con 0x7f67080c6bd0 2023-10-30T13:23:12.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f6705d9b640 1 -- 172.21.15.148:0/3450389682 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f00a85d0 con 0x7f67080c6bd0 2023-10-30T13:23:12.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f6705d9b640 1 -- 172.21.15.148:0/3450389682 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f00a8940 con 0x7f67080c6bd0 2023-10-30T13:23:12.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 -- 172.21.15.148:0/3450389682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67080c6bd0 msgr2=0x7f670806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 --2- 172.21.15.148:0/3450389682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67080c6bd0 0x7f670806fb80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097c0 tx=0x7f66f001eb90 comp rx=0 tx=0).stop 2023-10-30T13:23:12.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 -- 172.21.15.148:0/3450389682 shutdown_connections 2023-10-30T13:23:12.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 --2- 172.21.15.148:0/3450389682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67080c6bd0 0x7f670806fb80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 --2- 172.21.15.148:0/3450389682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6708071110 0x7f6708158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 -- 172.21.15.148:0/3450389682 >> 172.21.15.148:0/3450389682 conn(0x7f67080c7910 msgr2=0x7f67080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:12.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 -- 172.21.15.148:0/3450389682 shutdown_connections 2023-10-30T13:23:12.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 -- 172.21.15.148:0/3450389682 wait complete. 2023-10-30T13:23:12.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 Processor -- start 2023-10-30T13:23:12.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 -- start start 2023-10-30T13:23:12.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f670806f7a0 0x7f6708155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708071110 0x7f6708153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67080c6bd0 0x7f6708152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67080ca980 con 0x7f67080c6bd0 2023-10-30T13:23:12.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67080c8280 con 0x7f6708071110 2023-10-30T13:23:12.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f670c9f8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67080afe20 con 0x7f670806f7a0 2023-10-30T13:23:12.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f6706d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67080c6bd0 0x7f6708152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f6706d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67080c6bd0 0x7f6708152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51406/0 (socket says 172.21.15.148:51406) 2023-10-30T13:23:12.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.869+0000 7f6706d9d640 1 -- 172.21.15.148:0/3935041116 learned_addr learned my addr 172.21.15.148:0/3935041116 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:12.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f66fffff640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708071110 0x7f6708153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f670659c640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f670806f7a0 0x7f6708155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f6706d9d640 1 -- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f670806f7a0 msgr2=0x7f6708155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f6706d9d640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f670806f7a0 0x7f6708155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f6706d9d640 1 -- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708071110 msgr2=0x7f6708153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:12.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f6706d9d640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708071110 0x7f6708153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:12.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f6706d9d640 1 -- 172.21.15.148:0/3935041116 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6700009a70 con 0x7f67080c6bd0 2023-10-30T13:23:12.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f6706d9d640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67080c6bd0 0x7f6708152760 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6700009f00 tx=0x7f670000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:12.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f66feffd640 1 -- 172.21.15.148:0/3935041116 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6700009d70 con 0x7f67080c6bd0 2023-10-30T13:23:12.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f66feffd640 1 -- 172.21.15.148:0/3935041116 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670000f040 con 0x7f67080c6bd0 2023-10-30T13:23:12.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f66feffd640 1 -- 172.21.15.148:0/3935041116 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6700003bc0 con 0x7f67080c6bd0 2023-10-30T13:23:12.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f0009420 con 0x7f67080c6bd0 2023-10-30T13:23:12.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6708154c60 con 0x7f67080c6bd0 2023-10-30T13:23:12.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6708155af0 con 0x7f67080c6bd0 2023-10-30T13:23:12.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f66feffd640 1 -- 172.21.15.148:0/3935041116 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6700014480 con 0x7f67080c6bd0 2023-10-30T13:23:12.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f66feffd640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66e80664d0 0x7f66e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:12.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.873+0000 7f66feffd640 1 -- 172.21.15.148:0/3935041116 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f6700084bb0 con 0x7f67080c6bd0 2023-10-30T13:23:12.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.877+0000 7f670659c640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66e80664d0 0x7f66e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:12.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.877+0000 7f670659c640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66e80664d0 0x7f66e8068990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f66f80011f0 tx=0x7f66f80016e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:12.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:12.877+0000 7f66feffd640 1 -- 172.21.15.148:0/3935041116 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6700051e90 con 0x7f67080c6bd0 2023-10-30T13:23:13.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.061+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f670804e170 con 0x7f67080c6bd0 2023-10-30T13:23:13.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.061+0000 7f66feffd640 1 -- 172.21.15.148:0/3935041116 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f67000554b0 con 0x7f67080c6bd0 2023-10-30T13:23:13.059 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:13.059 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:13.060 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 63 2023-10-30T13:23:13.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.061+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66e80664d0 msgr2=0x7f66e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66e80664d0 0x7f66e8068990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f66f80011f0 tx=0x7f66f80016e0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67080c6bd0 msgr2=0x7f6708152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67080c6bd0 0x7f6708152760 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6700009f00 tx=0x7f670000c490 comp rx=0 tx=0).stop 2023-10-30T13:23:13.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 shutdown_connections 2023-10-30T13:23:13.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66e80664d0 0x7f66e8068990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67080c6bd0 0x7f6708152760 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6708071110 0x7f6708153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 --2- 172.21.15.148:0/3935041116 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f670806f7a0 0x7f6708155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 >> 172.21.15.148:0/3935041116 conn(0x7f67080c7910 msgr2=0x7f67080c1890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:13.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 shutdown_connections 2023-10-30T13:23:13.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.065+0000 7f670c9f8640 1 -- 172.21.15.148:0/3935041116 wait complete. 2023-10-30T13:23:13.079 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:23:13.080 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:23:13.080 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:23:13.098 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:23:13.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:13.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373d0d6640 1 Processor -- start 2023-10-30T13:23:13.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373d0d6640 1 -- start start 2023-10-30T13:23:13.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373d0d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f373806f7a0 0x7f373806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373d0d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3738066cf0 con 0x7f3738071110 2023-10-30T13:23:13.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373d0d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37380c70d0 con 0x7f37380c6bd0 2023-10-30T13:23:13.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373d0d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37380700c0 con 0x7f373806f7a0 2023-10-30T13:23:13.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373759e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3738071110 0x7f3738158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43140/0 (socket says 172.21.15.148:43140) 2023-10-30T13:23:13.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373759e640 1 -- 172.21.15.148:0/1124852257 learned_addr learned my addr 172.21.15.148:0/1124852257 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:13.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3736d9d640 1 --2- 172.21.15.148:0/1124852257 >> v2:172.21.15.150:3300/0 conn(0x7f373806f7a0 0x7f373806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530674793 0 0) 0x7f3738066cf0 con 0x7f3738071110 2023-10-30T13:23:13.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3720003530 con 0x7f3738071110 2023-10-30T13:23:13.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027848175 0 0) 0x7f3720003530 con 0x7f3738071110 2023-10-30T13:23:13.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 >> v2:172.21.15.150:3300/0 conn(0x7f373806f7a0 msgr2=0x7f373806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:13.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 --2- 172.21.15.148:0/1124852257 >> v2:172.21.15.150:3300/0 conn(0x7f373806f7a0 0x7f373806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 >> v1:172.21.15.150:6789/0 conn(0x7f37380c6bd0 legacy=0x7f3738069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37380c7390 con 0x7f3738071110 2023-10-30T13:23:13.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f3730001a00 con 0x7f3738071110 2023-10-30T13:23:13.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 >> v1:172.21.15.148:6789/0 conn(0x7f3738071110 legacy=0x7f3738158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3720003980 0x7f3720003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f3720004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3720005190 0x7f372000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37380700c0 con 0x7f37200047d0 2023-10-30T13:23:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372000c040 con 0x7f3720005190 2023-10-30T13:23:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372000c200 con 0x7f3720003980 2023-10-30T13:23:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373659c640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f3720004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f3736d9d640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3720003980 0x7f3720003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373759e640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3720005190 0x7f372000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373659c640 1 -- 172.21.15.148:0/1124852257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3720003980 msgr2=0x7f3720003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373659c640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3720003980 0x7f3720003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373659c640 1 -- 172.21.15.148:0/1124852257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3720005190 msgr2=0x7f372000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373659c640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3720005190 0x7f372000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373659c640 1 -- 172.21.15.148:0/1124852257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3720003530 con 0x7f37200047d0 2023-10-30T13:23:13.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.281+0000 7f373759e640 1 -- 172.21.15.148:0/1124852257 reap_dead start 2023-10-30T13:23:13.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f373659c640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f3720004c50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f372c001750 tx=0x7f372c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40064c06baba95a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:13.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c0052d0 con 0x7f37200047d0 2023-10-30T13:23:13.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372c0a7050 con 0x7f37200047d0 2023-10-30T13:23:13.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f3735d9b640 1 -- 172.21.15.148:0/1124852257 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c003600 con 0x7f37200047d0 2023-10-30T13:23:13.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f373d0d6640 1 -- 172.21.15.148:0/1124852257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 msgr2=0x7f3720004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f373d0d6640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f3720004c50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f372c001750 tx=0x7f372c01c3a0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f373d0d6640 1 -- 172.21.15.148:0/1124852257 shutdown_connections 2023-10-30T13:23:13.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f373d0d6640 1 --2- 172.21.15.148:0/1124852257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f3720004c50 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f373d0d6640 1 -- 172.21.15.148:0/1124852257 >> 172.21.15.148:0/1124852257 conn(0x7f37380c7910 msgr2=0x7f37380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:13.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f373d0d6640 1 -- 172.21.15.148:0/1124852257 shutdown_connections 2023-10-30T13:23:13.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.285+0000 7f373d0d6640 1 -- 172.21.15.148:0/1124852257 wait complete. 2023-10-30T13:23:13.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 Processor -- start 2023-10-30T13:23:13.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 -- start start 2023-10-30T13:23:13.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f373815a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37380af740 0x7f37380aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373814c6b0 0x7f373814cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373814d0a0 con 0x7f37200047d0 2023-10-30T13:23:13.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37381588a0 con 0x7f373814c6b0 2023-10-30T13:23:13.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3738158b80 con 0x7f37380af740 2023-10-30T13:23:13.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f3736d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f373815a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f3736d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f373815a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51418/0 (socket says 172.21.15.148:51418) 2023-10-30T13:23:13.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f3736d9d640 1 -- 172.21.15.148:0/3732789345 learned_addr learned my addr 172.21.15.148:0/3732789345 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:13.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373759e640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373814c6b0 0x7f373814cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373659c640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37380af740 0x7f37380aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f3736d9d640 1 -- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37380af740 msgr2=0x7f37380aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f3736d9d640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37380af740 0x7f37380aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f3736d9d640 1 -- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373814c6b0 msgr2=0x7f373814cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f3736d9d640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373814c6b0 0x7f373814cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f3736d9d640 1 -- 172.21.15.148:0/3732789345 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3738158e60 con 0x7f37200047d0 2023-10-30T13:23:13.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f3736d9d640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f373815a3e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3728011500 tx=0x7f37280119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:13.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f371f7fe640 1 -- 172.21.15.148:0/3732789345 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728015870 con 0x7f37200047d0 2023-10-30T13:23:13.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f371f7fe640 1 -- 172.21.15.148:0/3732789345 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37280159d0 con 0x7f37200047d0 2023-10-30T13:23:13.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3738159140 con 0x7f37200047d0 2023-10-30T13:23:13.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f371f7fe640 1 -- 172.21.15.148:0/3732789345 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728015d40 con 0x7f37200047d0 2023-10-30T13:23:13.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.289+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3738159700 con 0x7f37200047d0 2023-10-30T13:23:13.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.293+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f373804e170 con 0x7f37200047d0 2023-10-30T13:23:13.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.293+0000 7f371f7fe640 1 -- 172.21.15.148:0/3732789345 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3728013400 con 0x7f37200047d0 2023-10-30T13:23:13.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.293+0000 7f371f7fe640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f37240664d0 0x7f3724068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:13.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.293+0000 7f371f7fe640 1 -- 172.21.15.148:0/3732789345 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f3728084960 con 0x7f37200047d0 2023-10-30T13:23:13.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.293+0000 7f373659c640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f37240664d0 0x7f3724068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:13.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.293+0000 7f373659c640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f37240664d0 0x7f3724068990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f37380c7000 tx=0x7f372c0011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:13.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.297+0000 7f371f7fe640 1 -- 172.21.15.148:0/3732789345 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3728051a80 con 0x7f37200047d0 2023-10-30T13:23:13.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3738070600 con 0x7f37200047d0 2023-10-30T13:23:13.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f371f7fe640 1 -- 172.21.15.148:0/3732789345 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f37280550a0 con 0x7f37200047d0 2023-10-30T13:23:13.473 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:13.474 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:13.474 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 63 2023-10-30T13:23:13.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f37240664d0 msgr2=0x7f3724068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f37240664d0 0x7f3724068990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f37380c7000 tx=0x7f372c0011a0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 msgr2=0x7f373815a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:13.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f373815a3e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3728011500 tx=0x7f37280119c0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 shutdown_connections 2023-10-30T13:23:13.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f37240664d0 0x7f3724068990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373814c6b0 0x7f373814cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f37380af740 0x7f37380aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 --2- 172.21.15.148:0/3732789345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f37200047d0 0x7f373815a3e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:13.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 >> 172.21.15.148:0/3732789345 conn(0x7f37380c7910 msgr2=0x7f3738150350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:13.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 shutdown_connections 2023-10-30T13:23:13.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:13.477+0000 7f373d0d6640 1 -- 172.21.15.148:0/3732789345 wait complete. 2023-10-30T13:23:13.494 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:23:15.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:15.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb805017640 1 Processor -- start 2023-10-30T13:23:15.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb805017640 1 -- start start 2023-10-30T13:23:15.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb805017640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 0x7fb80015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:15.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb805017640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb8000cf160 0x7fb8000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:15.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb805017640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8000cb800 con 0x7fb8000ca0e0 2023-10-30T13:23:15.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb805017640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb80015dc30 con 0x7fb8000cf160 2023-10-30T13:23:15.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb805017640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8000cb330 con 0x7fb8001538a0 2023-10-30T13:23:15.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7fe59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb8000ca0e0 0x7fb80015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43154/0 (socket says 172.21.15.148:43154) 2023-10-30T13:23:15.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7fe59c640 1 -- 172.21.15.148:0/43947402 learned_addr learned my addr 172.21.15.148:0/43947402 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:15.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7ff59e640 1 --2- 172.21.15.148:0/43947402 >> v2:172.21.15.150:3300/0 conn(0x7fb8000cf160 0x7fb8000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:15.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7fed9d640 1 --2- 172.21.15.148:0/43947402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 0x7fb80015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:15.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7fdd9b640 1 -- 172.21.15.148:0/43947402 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246256267 0 0) 0x7fb8000cb800 con 0x7fb8000ca0e0 2023-10-30T13:23:15.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7fdd9b640 1 -- 172.21.15.148:0/43947402 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e8003530 con 0x7fb8000ca0e0 2023-10-30T13:23:15.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7fdd9b640 1 -- 172.21.15.148:0/43947402 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2931703608 0 0) 0x7fb7e8003530 con 0x7fb8000ca0e0 2023-10-30T13:23:15.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7ff59e640 1 -- 172.21.15.148:0/43947402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 msgr2=0x7fb80015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:15.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7ff59e640 1 --2- 172.21.15.148:0/43947402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 0x7fb80015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:15.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7ff59e640 1 -- 172.21.15.148:0/43947402 >> v1:172.21.15.148:6789/0 conn(0x7fb8000ca0e0 legacy=0x7fb80015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:15.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.649+0000 7fb7ff59e640 1 -- 172.21.15.148:0/43947402 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8000cad70 con 0x7fb8000cf160 2023-10-30T13:23:15.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb7ff59e640 1 --2- 172.21.15.148:0/43947402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000cf160 0x7fb8000cf590 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009d90 tx=0x7fb7f801f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=667dc966fe6d9e26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:15.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb7fdd9b640 1 -- 172.21.15.148:0/43947402 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4001680 con 0x7fb8000cf160 2023-10-30T13:23:15.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb7fdd9b640 1 -- 172.21.15.148:0/43947402 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f80a8bb0 con 0x7fb8000cf160 2023-10-30T13:23:15.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb7fdd9b640 1 -- 172.21.15.148:0/43947402 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f80b0440 con 0x7fb8000cf160 2023-10-30T13:23:15.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb805017640 1 -- 172.21.15.148:0/43947402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000cf160 msgr2=0x7fb8000cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:15.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb805017640 1 --2- 172.21.15.148:0/43947402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000cf160 0x7fb8000cf590 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009d90 tx=0x7fb7f801f060 comp rx=0 tx=0).stop 2023-10-30T13:23:15.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb805017640 1 -- 172.21.15.148:0/43947402 shutdown_connections 2023-10-30T13:23:15.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb805017640 1 --2- 172.21.15.148:0/43947402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000cf160 0x7fb8000cf590 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:15.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb805017640 1 --2- 172.21.15.148:0/43947402 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 0x7fb80015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:15.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb805017640 1 -- 172.21.15.148:0/43947402 >> 172.21.15.148:0/43947402 conn(0x7fb80006bfe0 msgr2=0x7fb80006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:15.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb805017640 1 -- 172.21.15.148:0/43947402 shutdown_connections 2023-10-30T13:23:15.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.653+0000 7fb805017640 1 -- 172.21.15.148:0/43947402 wait complete. 2023-10-30T13:23:15.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb805017640 1 Processor -- start 2023-10-30T13:23:15.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb805017640 1 -- start start 2023-10-30T13:23:15.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb805017640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000ca0e0 0x7fb8001592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:15.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb805017640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8000cf160 0x7fb8000c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:15.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb805017640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 0x7fb8001efa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:15.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb805017640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8001f2c50 con 0x7fb8000cf160 2023-10-30T13:23:15.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb805017640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8001f2f30 con 0x7fb8000ca0e0 2023-10-30T13:23:15.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb805017640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8001f3210 con 0x7fb8001538a0 2023-10-30T13:23:15.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fe59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8000cf160 0x7fb8000c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:15.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fe59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8000cf160 0x7fb8000c4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51432/0 (socket says 172.21.15.148:51432) 2023-10-30T13:23:15.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fe59c640 1 -- 172.21.15.148:0/2249267394 learned_addr learned my addr 172.21.15.148:0/2249267394 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:15.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fed9d640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000ca0e0 0x7fb8001592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:15.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7ff59e640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 0x7fb8001efa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:15.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fe59c640 1 -- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 msgr2=0x7fb8001efa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:15.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fe59c640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 0x7fb8001efa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:15.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fe59c640 1 -- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000ca0e0 msgr2=0x7fb8001592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:15.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fe59c640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000ca0e0 0x7fb8001592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:15.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fe59c640 1 -- 172.21.15.148:0/2249267394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f8009a70 con 0x7fb8000cf160 2023-10-30T13:23:15.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fed9d640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000ca0e0 0x7fb8001592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:15.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.657+0000 7fb7fe59c640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8000cf160 0x7fb8000c4d70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400c780 tx=0x7fb7f400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:15.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb7e77fe640 1 -- 172.21.15.148:0/2249267394 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4005cf0 con 0x7fb8000cf160 2023-10-30T13:23:15.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8001f34f0 con 0x7fb8000cf160 2023-10-30T13:23:15.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb7e77fe640 1 -- 172.21.15.148:0/2249267394 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f4005e50 con 0x7fb8000cf160 2023-10-30T13:23:15.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb7e77fe640 1 -- 172.21.15.148:0/2249267394 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f40072b0 con 0x7fb8000cf160 2023-10-30T13:23:15.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8001f3ab0 con 0x7fb8000cf160 2023-10-30T13:23:15.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8000b2a50 con 0x7fb8000cf160 2023-10-30T13:23:15.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb7e77fe640 1 -- 172.21.15.148:0/2249267394 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7f4007410 con 0x7fb8000cf160 2023-10-30T13:23:15.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb7e77fe640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7ec0664a0 0x7fb7ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:15.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb7e77fe640 1 -- 172.21.15.148:0/2249267394 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fb7f4083f10 con 0x7fb8000cf160 2023-10-30T13:23:15.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.661+0000 7fb7fed9d640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7ec0664a0 0x7fb7ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:15.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.665+0000 7fb7fed9d640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7ec0664a0 0x7fb7ec068960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009b10 tx=0x7fb7f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:15.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.665+0000 7fb7e77fe640 1 -- 172.21.15.148:0/2249267394 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7f4051030 con 0x7fb8000cf160 2023-10-30T13:23:15.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.845+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb80004e170 con 0x7fb8000cf160 2023-10-30T13:23:15.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb7e77fe640 1 -- 172.21.15.148:0/2249267394 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fb7f4054650 con 0x7fb8000cf160 2023-10-30T13:23:15.844 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:15.844 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:15.845 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 63 2023-10-30T13:23:15.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7ec0664a0 msgr2=0x7fb7ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:15.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7ec0664a0 0x7fb7ec068960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009b10 tx=0x7fb7f0008040 comp rx=0 tx=0).stop 2023-10-30T13:23:15.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8000cf160 msgr2=0x7fb8000c4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:15.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8000cf160 0x7fb8000c4d70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400c780 tx=0x7fb7f400cc40 comp rx=0 tx=0).stop 2023-10-30T13:23:15.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 shutdown_connections 2023-10-30T13:23:15.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7ec0664a0 0x7fb7ec068960 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:15.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8000cf160 0x7fb8000c4d70 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:15.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8001538a0 0x7fb8001efa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:15.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 --2- 172.21.15.148:0/2249267394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8000ca0e0 0x7fb8001592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:15.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 >> 172.21.15.148:0/2249267394 conn(0x7fb80006bfe0 msgr2=0x7fb800150ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:15.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 shutdown_connections 2023-10-30T13:23:15.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:15.849+0000 7fb805017640 1 -- 172.21.15.148:0/2249267394 wait complete. 2023-10-30T13:23:15.865 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:23:17.866 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.017+0000 7f26e3a46640 1 Processor -- start 2023-10-30T13:23:18.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.017+0000 7f26e3a46640 1 -- start start 2023-10-30T13:23:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e3a46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc070740 0x7f26dc0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e3a46640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0c5810 con 0x7f26dc15e700 2023-10-30T13:23:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e3a46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc15eb90 con 0x7f26dc1580e0 2023-10-30T13:23:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e3a46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc06acb0 con 0x7f26dc070740 2023-10-30T13:23:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e1fe3640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f26dc15e700 0x7f26dc06a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43168/0 (socket says 172.21.15.148:43168) 2023-10-30T13:23:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e1fe3640 1 -- 172.21.15.148:0/1288130863 learned_addr learned my addr 172.21.15.148:0/1288130863 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e0fe1640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc070740 0x7f26dc0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496849365 0 0) 0x7f26dc0c5810 con 0x7f26dc15e700 2023-10-30T13:23:18.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c8003530 con 0x7f26dc15e700 2023-10-30T13:23:18.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928364860 0 0) 0x7f26dc15eb90 con 0x7f26dc1580e0 2023-10-30T13:23:18.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc0c5810 con 0x7f26dc1580e0 2023-10-30T13:23:18.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1390807487 0 0) 0x7f26c8003530 con 0x7f26dc15e700 2023-10-30T13:23:18.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc070740 msgr2=0x7f26dc0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:18.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc070740 0x7f26dc0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 >> v1:172.21.15.150:6789/0 conn(0x7f26dc1580e0 legacy=0x7f26dc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0c6bb0 con 0x7f26dc15e700 2023-10-30T13:23:18.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f26d4001a00 con 0x7f26dc15e700 2023-10-30T13:23:18.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 >> v1:172.21.15.148:6789/0 conn(0x7f26dc15e700 legacy=0x7f26dc06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26c8003b40 0x7f26c8003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c80047d0 0x7f26c8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c8005190 0x7f26c800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c800bf80 con 0x7f26c8003b40 2023-10-30T13:23:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c800c0c0 con 0x7f26c80047d0 2023-10-30T13:23:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c800c250 con 0x7f26c8005190 2023-10-30T13:23:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e0fe1640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc070740 0x7f26dc0c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e17e2640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26c8003b40 0x7f26c8003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e1fe3640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c8005190 0x7f26c800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e0fe1640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c80047d0 0x7f26c8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e17e2640 1 -- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c8005190 msgr2=0x7f26c800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e17e2640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26c8005190 0x7f26c800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e17e2640 1 -- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c80047d0 msgr2=0x7f26c8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e17e2640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c80047d0 0x7f26c8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.021+0000 7f26e17e2640 1 -- 172.21.15.148:0/1288130863 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c8003530 con 0x7f26c8003b40 2023-10-30T13:23:18.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e17e2640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26c8003b40 0x7f26c8003f20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f26d80018f0 tx=0x7f26d801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267e77d0d7defe8d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:18.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d80052d0 con 0x7f26c8003b40 2023-10-30T13:23:18.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d80a7050 con 0x7f26c8003b40 2023-10-30T13:23:18.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26d3fff640 1 -- 172.21.15.148:0/1288130863 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8003600 con 0x7f26c8003b40 2023-10-30T13:23:18.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e3a46640 1 -- 172.21.15.148:0/1288130863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26c8003b40 msgr2=0x7f26c8003f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e3a46640 1 --2- 172.21.15.148:0/1288130863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26c8003b40 0x7f26c8003f20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f26d80018f0 tx=0x7f26d801c540 comp rx=0 tx=0).stop 2023-10-30T13:23:18.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e1fe3640 1 -- 172.21.15.148:0/1288130863 reap_dead start 2023-10-30T13:23:18.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e3a46640 1 -- 172.21.15.148:0/1288130863 shutdown_connections 2023-10-30T13:23:18.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e3a46640 1 -- 172.21.15.148:0/1288130863 >> 172.21.15.148:0/1288130863 conn(0x7f26dc0c7910 msgr2=0x7f26dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:18.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e3a46640 1 -- 172.21.15.148:0/1288130863 shutdown_connections 2023-10-30T13:23:18.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e3a46640 1 -- 172.21.15.148:0/1288130863 wait complete. 2023-10-30T13:23:18.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e3a46640 1 Processor -- start 2023-10-30T13:23:18.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.025+0000 7f26e3a46640 1 -- start start 2023-10-30T13:23:18.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e3a46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c8003b40 0x7f26dc15d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e3a46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc0b0740 0x7f26dc15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e3a46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26dc1f9cc0 0x7f26dc1fa140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e3a46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0c5370 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e3a46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc15ce10 con 0x7f26c8003b40 2023-10-30T13:23:18.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e3a46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0c4db0 con 0x7f26dc0b0740 2023-10-30T13:23:18.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e1fe3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26dc1f9cc0 0x7f26dc1fa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e1fe3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26dc1f9cc0 0x7f26dc1fa140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51458/0 (socket says 172.21.15.148:51458) 2023-10-30T13:23:18.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e1fe3640 1 -- 172.21.15.148:0/3124207884 learned_addr learned my addr 172.21.15.148:0/3124207884 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:18.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e0fe1640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc0b0740 0x7f26dc15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e17e2640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c8003b40 0x7f26dc15d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e1fe3640 1 -- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc0b0740 msgr2=0x7f26dc15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e1fe3640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc0b0740 0x7f26dc15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e1fe3640 1 -- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c8003b40 msgr2=0x7f26dc15d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e1fe3640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c8003b40 0x7f26dc15d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e1fe3640 1 -- 172.21.15.148:0/3124207884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0c5090 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e17e2640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c8003b40 0x7f26dc15d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:18.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e1fe3640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26dc1f9cc0 0x7f26dc1fa140 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f26dc15eae0 tx=0x7f26d4008bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:18.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26d1ffb640 1 -- 172.21.15.148:0/3124207884 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d400faa0 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26d1ffb640 1 -- 172.21.15.148:0/3124207884 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d400fc00 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc15c7c0 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26d1ffb640 1 -- 172.21.15.148:0/3124207884 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d4015480 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc0c1d40 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.029+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc15cfc0 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.033+0000 7f26d1ffb640 1 -- 172.21.15.148:0/3124207884 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f26d401e3b0 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.033+0000 7f26d1ffb640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26c00664d0 0x7f26c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:18.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.033+0000 7f26d1ffb640 1 -- 172.21.15.148:0/3124207884 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f26d4083f60 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.033+0000 7f26e17e2640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26c00664d0 0x7f26c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:18.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.033+0000 7f26e17e2640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26c00664d0 0x7f26c0068990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f26d80ac000 tx=0x7f26d8001350 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:18.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.037+0000 7f26d1ffb640 1 -- 172.21.15.148:0/3124207884 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f26dc15cfc0 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.217+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26dc15cfc0 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26d1ffb640 1 -- 172.21.15.148:0/3124207884 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f26dc15cfc0 con 0x7f26dc1f9cc0 2023-10-30T13:23:18.217 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:18.217 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:18.217 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 63 2023-10-30T13:23:18.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26c00664d0 msgr2=0x7f26c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26c00664d0 0x7f26c0068990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f26d80ac000 tx=0x7f26d8001350 comp rx=0 tx=0).stop 2023-10-30T13:23:18.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26dc1f9cc0 msgr2=0x7f26dc1fa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:18.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26dc1f9cc0 0x7f26dc1fa140 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f26dc15eae0 tx=0x7f26d4008bb0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 shutdown_connections 2023-10-30T13:23:18.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26c00664d0 0x7f26c0068990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26dc1f9cc0 0x7f26dc1fa140 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26dc0b0740 0x7f26dc15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 --2- 172.21.15.148:0/3124207884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26c8003b40 0x7f26dc15d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:18.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 >> 172.21.15.148:0/3124207884 conn(0x7f26dc0c7910 msgr2=0x7f26dc1561f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:18.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 shutdown_connections 2023-10-30T13:23:18.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:18.221+0000 7f26e3a46640 1 -- 172.21.15.148:0/3124207884 wait complete. 2023-10-30T13:23:18.237 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:23:20.238 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.389+0000 7fad5879d640 1 Processor -- start 2023-10-30T13:23:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.389+0000 7fad5879d640 1 -- start start 2023-10-30T13:23:20.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad5879d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad50070650 0x7fad50071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad5879d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fad50070c90 0x7fad500720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad5879d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad500c2f20 con 0x7fad50156f80 2023-10-30T13:23:20.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad5879d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad50157360 con 0x7fad50070c90 2023-10-30T13:23:20.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad5879d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad500c85c0 con 0x7fad50070650 2023-10-30T13:23:20.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad56539640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fad50156f80 0x7fad50159880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43170/0 (socket says 172.21.15.148:43170) 2023-10-30T13:23:20.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad56539640 1 -- 172.21.15.148:0/1531219982 learned_addr learned my addr 172.21.15.148:0/1531219982 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:20.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad56d3a640 1 --2- 172.21.15.148:0/1531219982 >> v2:172.21.15.148:3300/0 conn(0x7fad50070c90 0x7fad500720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad55d38640 1 --2- 172.21.15.148:0/1531219982 >> v2:172.21.15.150:3300/0 conn(0x7fad50070650 0x7fad50071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad55537640 1 -- 172.21.15.148:0/1531219982 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148597030 0 0) 0x7fad500c2f20 con 0x7fad50156f80 2023-10-30T13:23:20.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad55537640 1 -- 172.21.15.148:0/1531219982 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad3c003530 con 0x7fad50156f80 2023-10-30T13:23:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad56d3a640 1 -- 172.21.15.148:0/1531219982 >> v2:172.21.15.150:3300/0 conn(0x7fad50070650 msgr2=0x7fad50071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad56d3a640 1 --2- 172.21.15.148:0/1531219982 >> v2:172.21.15.150:3300/0 conn(0x7fad50070650 0x7fad50071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad56d3a640 1 -- 172.21.15.148:0/1531219982 >> v1:172.21.15.148:6789/0 conn(0x7fad50156f80 legacy=0x7fad50159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad56d3a640 1 -- 172.21.15.148:0/1531219982 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad50071230 con 0x7fad50070c90 2023-10-30T13:23:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad55d38640 1 --2- 172.21.15.148:0/1531219982 >> v2:172.21.15.150:3300/0 conn(0x7fad50070650 0x7fad50071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad56d3a640 1 --2- 172.21.15.148:0/1531219982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 0x7fad500720f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fad5015a9b0 tx=0x7fad4801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dafa64206da10400 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.393+0000 7fad55537640 1 -- 172.21.15.148:0/1531219982 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad480a5960 con 0x7fad50070c90 2023-10-30T13:23:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad55537640 1 -- 172.21.15.148:0/1531219982 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad480a5ac0 con 0x7fad50070c90 2023-10-30T13:23:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad55537640 1 -- 172.21.15.148:0/1531219982 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad480a5e50 con 0x7fad50070c90 2023-10-30T13:23:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 -- 172.21.15.148:0/1531219982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 msgr2=0x7fad500720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 --2- 172.21.15.148:0/1531219982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 0x7fad500720f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fad5015a9b0 tx=0x7fad4801c910 comp rx=0 tx=0).stop 2023-10-30T13:23:20.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 -- 172.21.15.148:0/1531219982 shutdown_connections 2023-10-30T13:23:20.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 --2- 172.21.15.148:0/1531219982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 0x7fad500720f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 --2- 172.21.15.148:0/1531219982 >> v2:172.21.15.150:3300/0 conn(0x7fad50070650 0x7fad50071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 -- 172.21.15.148:0/1531219982 >> 172.21.15.148:0/1531219982 conn(0x7fad500cbeb0 msgr2=0x7fad500cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:20.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 -- 172.21.15.148:0/1531219982 shutdown_connections 2023-10-30T13:23:20.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 -- 172.21.15.148:0/1531219982 wait complete. 2023-10-30T13:23:20.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 Processor -- start 2023-10-30T13:23:20.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 -- start start 2023-10-30T13:23:20.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad50070650 0x7fad5014e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 0x7fad500cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad50156f80 0x7fad501f5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad50159470 con 0x7fad50070c90 2023-10-30T13:23:20.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad50155a40 con 0x7fad50070650 2023-10-30T13:23:20.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad5879d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad50155d20 con 0x7fad50156f80 2023-10-30T13:23:20.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad55d38640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 0x7fad500cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad55d38640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 0x7fad500cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51480/0 (socket says 172.21.15.148:51480) 2023-10-30T13:23:20.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad55d38640 1 -- 172.21.15.148:0/584348495 learned_addr learned my addr 172.21.15.148:0/584348495 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:20.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad56539640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad50070650 0x7fad5014e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad56d3a640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad50156f80 0x7fad501f5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.397+0000 7fad55d38640 1 -- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad50156f80 msgr2=0x7fad501f5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad55d38640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad50156f80 0x7fad501f5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad55d38640 1 -- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad50070650 msgr2=0x7fad5014e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad55d38640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad50070650 0x7fad5014e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad55d38640 1 -- 172.21.15.148:0/584348495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48009a70 con 0x7fad50070c90 2023-10-30T13:23:20.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad55d38640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 0x7fad500cb530 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fad4000d7d0 tx=0x7fad4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad46ffd640 1 -- 172.21.15.148:0/584348495 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad400096a0 con 0x7fad50070c90 2023-10-30T13:23:20.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad46ffd640 1 -- 172.21.15.148:0/584348495 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40009800 con 0x7fad50070c90 2023-10-30T13:23:20.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad50156000 con 0x7fad50070c90 2023-10-30T13:23:20.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad46ffd640 1 -- 172.21.15.148:0/584348495 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40009bb0 con 0x7fad50070c90 2023-10-30T13:23:20.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad501565c0 con 0x7fad50070c90 2023-10-30T13:23:20.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad501567e0 con 0x7fad50070c90 2023-10-30T13:23:20.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad46ffd640 1 -- 172.21.15.148:0/584348495 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad40009d10 con 0x7fad50070c90 2023-10-30T13:23:20.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad46ffd640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fad340664d0 0x7fad34068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:20.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.401+0000 7fad46ffd640 1 -- 172.21.15.148:0/584348495 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fad40084990 con 0x7fad50070c90 2023-10-30T13:23:20.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.405+0000 7fad56539640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fad340664d0 0x7fad34068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:20.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.405+0000 7fad56539640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fad340664d0 0x7fad34068990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fad4c005890 tx=0x7fad4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:20.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.409+0000 7fad46ffd640 1 -- 172.21.15.148:0/584348495 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad40051aa0 con 0x7fad50070c90 2023-10-30T13:23:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.589+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad5004e170 con 0x7fad50070c90 2023-10-30T13:23:20.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.589+0000 7fad46ffd640 1 -- 172.21.15.148:0/584348495 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fad400550c0 con 0x7fad50070c90 2023-10-30T13:23:20.587 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:20.587 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:20.587 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 63 2023-10-30T13:23:20.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fad340664d0 msgr2=0x7fad34068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fad340664d0 0x7fad34068990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fad4c005890 tx=0x7fad4c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:20.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 msgr2=0x7fad500cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:20.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 0x7fad500cb530 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fad4000d7d0 tx=0x7fad4000dc90 comp rx=0 tx=0).stop 2023-10-30T13:23:20.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 shutdown_connections 2023-10-30T13:23:20.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fad340664d0 0x7fad34068990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad50070c90 0x7fad500cb530 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad50156f80 0x7fad501f5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 --2- 172.21.15.148:0/584348495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad50070650 0x7fad5014e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:20.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 >> 172.21.15.148:0/584348495 conn(0x7fad500cbeb0 msgr2=0x7fad50153a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:20.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 shutdown_connections 2023-10-30T13:23:20.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:20.593+0000 7fad5879d640 1 -- 172.21.15.148:0/584348495 wait complete. 2023-10-30T13:23:20.608 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:23:22.609 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:22.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.761+0000 7f47a5ed6640 1 Processor -- start 2023-10-30T13:23:22.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.761+0000 7f47a5ed6640 1 -- start start 2023-10-30T13:23:22.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f47a5ed6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 0x7f47a013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f47a5ed6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f47a00c6f60 0x7f47a013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f47a5ed6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a00c0a50 con 0x7f47a00c6b80 2023-10-30T13:23:22.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f47a5ed6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a006f6e0 con 0x7f47a00c6f60 2023-10-30T13:23:22.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f47a5ed6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a006f9c0 con 0x7f47a00c05d0 2023-10-30T13:23:22.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479f7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f47a00c6b80 0x7f47a00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34264/0 (socket says 172.21.15.148:34264) 2023-10-30T13:23:22.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479f7fe640 1 -- 172.21.15.148:0/1685415977 learned_addr learned my addr 172.21.15.148:0/1685415977 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:22.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479ffff640 1 --2- 172.21.15.148:0/1685415977 >> v2:172.21.15.148:3300/0 conn(0x7f47a00c6f60 0x7f47a013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479effd640 1 --2- 172.21.15.148:0/1685415977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 0x7f47a013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479e7fc640 1 -- 172.21.15.148:0/1685415977 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47480976 0 0) 0x7f47a00c0a50 con 0x7f47a00c6b80 2023-10-30T13:23:22.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479ffff640 1 -- 172.21.15.148:0/1685415977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 msgr2=0x7f47a013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:22.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479ffff640 1 --2- 172.21.15.148:0/1685415977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 0x7f47a013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479ffff640 1 -- 172.21.15.148:0/1685415977 >> v1:172.21.15.148:6789/0 conn(0x7f47a00c6b80 legacy=0x7f47a00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479ffff640 1 -- 172.21.15.148:0/1685415977 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a006fca0 con 0x7f47a00c6f60 2023-10-30T13:23:22.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.765+0000 7f479ffff640 1 --2- 172.21.15.148:0/1685415977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 0x7f47a013f2d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4798009d90 tx=0x7f479801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e720d7f01e01fa97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:22.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f479e7fc640 1 -- 172.21.15.148:0/1685415977 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980a5920 con 0x7f47a00c6f60 2023-10-30T13:23:22.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f479e7fc640 1 -- 172.21.15.148:0/1685415977 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47980a5a80 con 0x7f47a00c6f60 2023-10-30T13:23:22.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f479e7fc640 1 -- 172.21.15.148:0/1685415977 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980a5e10 con 0x7f47a00c6f60 2023-10-30T13:23:22.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1685415977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 msgr2=0x7f47a013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 --2- 172.21.15.148:0/1685415977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 0x7f47a013f2d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4798009d90 tx=0x7f479801c950 comp rx=0 tx=0).stop 2023-10-30T13:23:22.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1685415977 shutdown_connections 2023-10-30T13:23:22.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 --2- 172.21.15.148:0/1685415977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 0x7f47a013f2d0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 --2- 172.21.15.148:0/1685415977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 0x7f47a013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1685415977 >> 172.21.15.148:0/1685415977 conn(0x7f47a00c7910 msgr2=0x7f47a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:22.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1685415977 shutdown_connections 2023-10-30T13:23:22.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1685415977 wait complete. 2023-10-30T13:23:22.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 Processor -- start 2023-10-30T13:23:22.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 -- start start 2023-10-30T13:23:22.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 0x7f47a006a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00c6b80 0x7f47a01501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 0x7f47a0150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01627a0 con 0x7f47a00c6f60 2023-10-30T13:23:22.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0162450 con 0x7f47a00c6b80 2023-10-30T13:23:22.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f47a5ed6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0132e50 con 0x7f47a00c05d0 2023-10-30T13:23:22.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.769+0000 7f479ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 0x7f47a0150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 0x7f47a0150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55312/0 (socket says 172.21.15.148:55312) 2023-10-30T13:23:22.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479ffff640 1 -- 172.21.15.148:0/1670073320 learned_addr learned my addr 172.21.15.148:0/1670073320 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:22.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479f7fe640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 0x7f47a006a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479effd640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00c6b80 0x7f47a01501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479ffff640 1 -- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 msgr2=0x7f47a006a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479ffff640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 0x7f47a006a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479ffff640 1 -- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00c6b80 msgr2=0x7f47a01501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479ffff640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00c6b80 0x7f47a01501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479ffff640 1 -- 172.21.15.148:0/1670073320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798009a70 con 0x7f47a00c6f60 2023-10-30T13:23:22.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f479ffff640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 0x7f47a0150700 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f479801b000 tx=0x7f47980196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:22.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f4787fff640 1 -- 172.21.15.148:0/1670073320 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980ad9d0 con 0x7f47a00c6f60 2023-10-30T13:23:22.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f4787fff640 1 -- 172.21.15.148:0/1670073320 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47980adb30 con 0x7f47a00c6f60 2023-10-30T13:23:22.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a0133130 con 0x7f47a00c6f60 2023-10-30T13:23:22.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f4787fff640 1 -- 172.21.15.148:0/1670073320 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980adea0 con 0x7f47a00c6f60 2023-10-30T13:23:22.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a01f2750 con 0x7f47a00c6f60 2023-10-30T13:23:22.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a004e170 con 0x7f47a00c6f60 2023-10-30T13:23:22.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f4787fff640 1 -- 172.21.15.148:0/1670073320 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47980b4050 con 0x7f47a00c6f60 2023-10-30T13:23:22.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f4787fff640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f478c0664d0 0x7f478c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:22.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.773+0000 7f4787fff640 1 -- 172.21.15.148:0/1670073320 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f479811f230 con 0x7f47a00c6f60 2023-10-30T13:23:22.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.777+0000 7f479f7fe640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f478c0664d0 0x7f478c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:22.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.777+0000 7f479f7fe640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f478c0664d0 0x7f478c068990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4790000f70 tx=0x7f4790007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:22.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.777+0000 7f4787fff640 1 -- 172.21.15.148:0/1670073320 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47980ec580 con 0x7f47a00c6f60 2023-10-30T13:23:22.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.961+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47a0162590 con 0x7f47a00c6f60 2023-10-30T13:23:22.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.961+0000 7f4787fff640 1 -- 172.21.15.148:0/1670073320 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f47980efba0 con 0x7f47a00c6f60 2023-10-30T13:23:22.959 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:22.960 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:22.960 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 63 2023-10-30T13:23:22.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f478c0664d0 msgr2=0x7f478c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f478c0664d0 0x7f478c068990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4790000f70 tx=0x7f4790007b60 comp rx=0 tx=0).stop 2023-10-30T13:23:22.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 msgr2=0x7f47a0150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:22.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 0x7f47a0150700 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f479801b000 tx=0x7f47980196f0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 shutdown_connections 2023-10-30T13:23:22.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f478c0664d0 0x7f478c068990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00c6f60 0x7f47a0150700 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00c6b80 0x7f47a01501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 --2- 172.21.15.148:0/1670073320 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a00c05d0 0x7f47a006a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:22.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 >> 172.21.15.148:0/1670073320 conn(0x7f47a00c7910 msgr2=0x7f47a0133dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:22.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 shutdown_connections 2023-10-30T13:23:22.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:22.965+0000 7f47a5ed6640 1 -- 172.21.15.148:0/1670073320 wait complete. 2023-10-30T13:23:22.980 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:23:24.981 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:25.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac68409640 1 Processor -- start 2023-10-30T13:23:25.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac68409640 1 -- start start 2023-10-30T13:23:25.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac68409640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fac6006a720 0x7fac6006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac68409640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac60071290 con 0x7fac600bfec0 2023-10-30T13:23:25.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac68409640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac60071570 con 0x7fac60159e60 2023-10-30T13:23:25.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac68409640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac60071850 con 0x7fac6006a720 2023-10-30T13:23:25.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac659a4640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fac600bfec0 0x7fac600c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34270/0 (socket says 172.21.15.148:34270) 2023-10-30T13:23:25.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac659a4640 1 -- 172.21.15.148:0/2256658956 learned_addr learned my addr 172.21.15.148:0/2256658956 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:25.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac669a6640 1 --2- 172.21.15.148:0/2256658956 >> v2:172.21.15.150:3300/0 conn(0x7fac6006a720 0x7fac6006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707442513 0 0) 0x7fac60071290 con 0x7fac600bfec0 2023-10-30T13:23:25.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.133+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac4c003530 con 0x7fac600bfec0 2023-10-30T13:23:25.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 115010215 0 0) 0x7fac4c003530 con 0x7fac600bfec0 2023-10-30T13:23:25.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 >> v2:172.21.15.150:3300/0 conn(0x7fac6006a720 msgr2=0x7fac6006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:25.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 --2- 172.21.15.148:0/2256658956 >> v2:172.21.15.150:3300/0 conn(0x7fac6006a720 0x7fac6006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 >> v1:172.21.15.150:6789/0 conn(0x7fac60159e60 legacy=0x7fac6006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac60071b30 con 0x7fac600bfec0 2023-10-30T13:23:25.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fac50001680 con 0x7fac600bfec0 2023-10-30T13:23:25.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 >> v1:172.21.15.148:6789/0 conn(0x7fac600bfec0 legacy=0x7fac600c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac4c003980 0x7fac4c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac4c0047d0 0x7fac4c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c005190 0x7fac4c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac60071850 con 0x7fac4c0047d0 2023-10-30T13:23:25.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c00c040 con 0x7fac4c005190 2023-10-30T13:23:25.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c00e410 con 0x7fac4c003980 2023-10-30T13:23:25.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac659a4640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac4c0047d0 0x7fac4c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac661a5640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac4c003980 0x7fac4c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac659a4640 1 -- 172.21.15.148:0/2256658956 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac4c003980 msgr2=0x7fac4c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:25.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac659a4640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac4c003980 0x7fac4c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac659a4640 1 -- 172.21.15.148:0/2256658956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c005190 msgr2=0x7fac4c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:25.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac659a4640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c005190 0x7fac4c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac659a4640 1 -- 172.21.15.148:0/2256658956 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c003530 con 0x7fac4c0047d0 2023-10-30T13:23:25.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac669a6640 1 -- 172.21.15.148:0/2256658956 reap_dead start 2023-10-30T13:23:25.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.137+0000 7fac659a4640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac4c0047d0 0x7fac4c004c50 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c240 tx=0x7fac5001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68129e0d9c8906 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac500a5b40 con 0x7fac4c0047d0 2023-10-30T13:23:25.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac500a5ca0 con 0x7fac4c0047d0 2023-10-30T13:23:25.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac651a3640 1 -- 172.21.15.148:0/2256658956 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50005b20 con 0x7fac4c0047d0 2023-10-30T13:23:25.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 -- 172.21.15.148:0/2256658956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac4c0047d0 msgr2=0x7fac4c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac4c0047d0 0x7fac4c004c50 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c240 tx=0x7fac5001cc40 comp rx=0 tx=0).stop 2023-10-30T13:23:25.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 -- 172.21.15.148:0/2256658956 shutdown_connections 2023-10-30T13:23:25.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 --2- 172.21.15.148:0/2256658956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac4c0047d0 0x7fac4c004c50 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 -- 172.21.15.148:0/2256658956 >> 172.21.15.148:0/2256658956 conn(0x7fac600c7910 msgr2=0x7fac600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:25.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 -- 172.21.15.148:0/2256658956 shutdown_connections 2023-10-30T13:23:25.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 -- 172.21.15.148:0/2256658956 wait complete. 2023-10-30T13:23:25.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 Processor -- start 2023-10-30T13:23:25.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 -- start start 2023-10-30T13:23:25.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac4c0047d0 0x7fac601478e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac601579e0 0x7fac601df640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac601ee650 0x7fac601eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac601ee200 con 0x7fac601579e0 2023-10-30T13:23:25.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac601dffb0 con 0x7fac601ee650 2023-10-30T13:23:25.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.141+0000 7fac68409640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac601ef040 con 0x7fac4c0047d0 2023-10-30T13:23:25.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac659a4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac601579e0 0x7fac601df640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac659a4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac601579e0 0x7fac601df640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55332/0 (socket says 172.21.15.148:55332) 2023-10-30T13:23:25.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac661a5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac4c0047d0 0x7fac601478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac659a4640 1 -- 172.21.15.148:0/4196640046 learned_addr learned my addr 172.21.15.148:0/4196640046 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:25.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac669a6640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac601ee650 0x7fac601eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac659a4640 1 -- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac4c0047d0 msgr2=0x7fac601478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac659a4640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac4c0047d0 0x7fac601478e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac659a4640 1 -- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac601ee650 msgr2=0x7fac601eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac659a4640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac601ee650 0x7fac601eeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac659a4640 1 -- 172.21.15.148:0/4196640046 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6014c0d0 con 0x7fac601579e0 2023-10-30T13:23:25.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac669a6640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac601ee650 0x7fac601eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:25.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac659a4640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac601579e0 0x7fac601df640 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c240 tx=0x7fac4c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac56ffd640 1 -- 172.21.15.148:0/4196640046 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac500b3070 con 0x7fac601579e0 2023-10-30T13:23:25.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac56ffd640 1 -- 172.21.15.148:0/4196640046 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac50000ee0 con 0x7fac601579e0 2023-10-30T13:23:25.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6014c3b0 con 0x7fac601579e0 2023-10-30T13:23:25.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac56ffd640 1 -- 172.21.15.148:0/4196640046 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50001640 con 0x7fac601579e0 2023-10-30T13:23:25.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6014c970 con 0x7fac601579e0 2023-10-30T13:23:25.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6014cb60 con 0x7fac601579e0 2023-10-30T13:23:25.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac56ffd640 1 -- 172.21.15.148:0/4196640046 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac50007140 con 0x7fac601579e0 2023-10-30T13:23:25.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.145+0000 7fac56ffd640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fac440664a0 0x7fac44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:25.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.149+0000 7fac56ffd640 1 -- 172.21.15.148:0/4196640046 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fac5011dfb0 con 0x7fac601579e0 2023-10-30T13:23:25.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.149+0000 7fac661a5640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fac440664a0 0x7fac44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:25.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.149+0000 7fac661a5640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fac440664a0 0x7fac44068960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fac5c001660 tx=0x7fac5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:25.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.153+0000 7fac56ffd640 1 -- 172.21.15.148:0/4196640046 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac500eb150 con 0x7fac601579e0 2023-10-30T13:23:25.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.329+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac6004e170 con 0x7fac601579e0 2023-10-30T13:23:25.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.329+0000 7fac56ffd640 1 -- 172.21.15.148:0/4196640046 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fac500ee770 con 0x7fac601579e0 2023-10-30T13:23:25.327 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:25.327 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:25.328 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 63 2023-10-30T13:23:25.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fac440664a0 msgr2=0x7fac44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fac440664a0 0x7fac44068960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fac5c001660 tx=0x7fac5c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:25.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac601579e0 msgr2=0x7fac601df640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:25.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac601579e0 0x7fac601df640 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c240 tx=0x7fac4c005800 comp rx=0 tx=0).stop 2023-10-30T13:23:25.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 shutdown_connections 2023-10-30T13:23:25.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fac440664a0 0x7fac44068960 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac601579e0 0x7fac601df640 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac601ee650 0x7fac601eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 --2- 172.21.15.148:0/4196640046 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fac4c0047d0 0x7fac601478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:25.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 >> 172.21.15.148:0/4196640046 conn(0x7fac600c7910 msgr2=0x7fac60157060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:25.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 shutdown_connections 2023-10-30T13:23:25.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:25.333+0000 7fac68409640 1 -- 172.21.15.148:0/4196640046 wait complete. 2023-10-30T13:23:25.348 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:23:25.381 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:25.384 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:27.349 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:27.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe0205d3640 1 Processor -- start 2023-10-30T13:23:27.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe0205d3640 1 -- start start 2023-10-30T13:23:27.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe0205d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe0180c6bd0 0x7fe01806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe0205d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe018071110 0x7fe018158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe0205d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe018066cf0 con 0x7fe01806f7a0 2023-10-30T13:23:27.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe0205d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0180700c0 con 0x7fe0180c6bd0 2023-10-30T13:23:27.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe0205d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180c70d0 con 0x7fe018071110 2023-10-30T13:23:27.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe01db6e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe0180c6bd0 0x7fe01806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe01db6e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe0180c6bd0 0x7fe01806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55346/0 (socket says 172.21.15.148:55346) 2023-10-30T13:23:27.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe01db6e640 1 -- 172.21.15.148:0/2954487927 learned_addr learned my addr 172.21.15.148:0/2954487927 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:27.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe01eb70640 1 --2- 172.21.15.148:0/2954487927 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe018071110 0x7fe018158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe01db6e640 1 -- 172.21.15.148:0/2954487927 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe018071110 msgr2=0x7fe018158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:27.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe01db6e640 1 --2- 172.21.15.148:0/2954487927 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe018071110 0x7fe018158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe01db6e640 1 -- 172.21.15.148:0/2954487927 >> v1:172.21.15.150:6789/0 conn(0x7fe01806f7a0 legacy=0x7fe018069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.501+0000 7fe01db6e640 1 -- 172.21.15.148:0/2954487927 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180c7390 con 0x7fe0180c6bd0 2023-10-30T13:23:27.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe01db6e640 1 --2- 172.21.15.148:0/2954487927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0180c6bd0 0x7fe01806fb80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe008009870 tx=0x7fe00801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44852418eff6871c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe01d36d640 1 -- 172.21.15.148:0/2954487927 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0080a8a30 con 0x7fe0180c6bd0 2023-10-30T13:23:27.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe01d36d640 1 -- 172.21.15.148:0/2954487927 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0080a8b90 con 0x7fe0180c6bd0 2023-10-30T13:23:27.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe01d36d640 1 -- 172.21.15.148:0/2954487927 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0080b0440 con 0x7fe0180c6bd0 2023-10-30T13:23:27.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe0205d3640 1 -- 172.21.15.148:0/2954487927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0180c6bd0 msgr2=0x7fe01806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe0205d3640 1 --2- 172.21.15.148:0/2954487927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0180c6bd0 0x7fe01806fb80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe008009870 tx=0x7fe00801ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:27.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe0205d3640 1 -- 172.21.15.148:0/2954487927 shutdown_connections 2023-10-30T13:23:27.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe0205d3640 1 --2- 172.21.15.148:0/2954487927 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe018071110 0x7fe018158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe0205d3640 1 --2- 172.21.15.148:0/2954487927 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0180c6bd0 0x7fe01806fb80 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe0205d3640 1 -- 172.21.15.148:0/2954487927 >> 172.21.15.148:0/2954487927 conn(0x7fe0180c7910 msgr2=0x7fe0180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.505+0000 7fe0205d3640 1 -- 172.21.15.148:0/2954487927 shutdown_connections 2023-10-30T13:23:27.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 -- 172.21.15.148:0/2954487927 wait complete. 2023-10-30T13:23:27.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 Processor -- start 2023-10-30T13:23:27.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 -- start start 2023-10-30T13:23:27.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe01806f7a0 0x7fe0181e4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe018071110 0x7fe01814fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0180c6bd0 0x7fe018153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0181e30c0 con 0x7fe018071110 2023-10-30T13:23:27.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0181e2810 con 0x7fe0180c6bd0 2023-10-30T13:23:27.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180c6880 con 0x7fe01806f7a0 2023-10-30T13:23:27.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01db6e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe018071110 0x7fe01814fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01db6e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe018071110 0x7fe01814fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55362/0 (socket says 172.21.15.148:55362) 2023-10-30T13:23:27.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01db6e640 1 -- 172.21.15.148:0/909880874 learned_addr learned my addr 172.21.15.148:0/909880874 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:27.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01eb70640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0180c6bd0 0x7fe018153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01e36f640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe01806f7a0 0x7fe0181e4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01db6e640 1 -- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe01806f7a0 msgr2=0x7fe0181e4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01db6e640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe01806f7a0 0x7fe0181e4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01db6e640 1 -- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0180c6bd0 msgr2=0x7fe018153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01db6e640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0180c6bd0 0x7fe018153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01db6e640 1 -- 172.21.15.148:0/909880874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0080094d0 con 0x7fe018071110 2023-10-30T13:23:27.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe01db6e640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe018071110 0x7fe01814fcb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe0080b3040 tx=0x7fe00801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe00effd640 1 -- 172.21.15.148:0/909880874 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0080b9620 con 0x7fe018071110 2023-10-30T13:23:27.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe00effd640 1 -- 172.21.15.148:0/909880874 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0080b97c0 con 0x7fe018071110 2023-10-30T13:23:27.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0181e2ca0 con 0x7fe018071110 2023-10-30T13:23:27.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe00effd640 1 -- 172.21.15.148:0/909880874 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0080b9b70 con 0x7fe018071110 2023-10-30T13:23:27.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.509+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0180b0320 con 0x7fe018071110 2023-10-30T13:23:27.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.513+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0181e25e0 con 0x7fe018071110 2023-10-30T13:23:27.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.513+0000 7fe00effd640 1 -- 172.21.15.148:0/909880874 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe0080b7070 con 0x7fe018071110 2023-10-30T13:23:27.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.513+0000 7fe00effd640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdffc0664d0 0x7fdffc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:27.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.513+0000 7fe00effd640 1 -- 172.21.15.148:0/909880874 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fe008122090 con 0x7fe018071110 2023-10-30T13:23:27.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.513+0000 7fe01e36f640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdffc0664d0 0x7fdffc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:27.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.513+0000 7fe01e36f640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdffc0664d0 0x7fdffc068990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe014000f70 tx=0x7fe014009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:27.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.517+0000 7fe00effd640 1 -- 172.21.15.148:0/909880874 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0181e25e0 con 0x7fe018071110 2023-10-30T13:23:27.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.697+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0181e29c0 con 0x7fe018071110 2023-10-30T13:23:27.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.697+0000 7fe00effd640 1 -- 172.21.15.148:0/909880874 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fe0181e29c0 con 0x7fe018071110 2023-10-30T13:23:27.696 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:27.696 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:22:56.580330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5694},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5694":{"gid":5694,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6837/3781833565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3781833565},{"type":"v1","addr":"172.21.15.148:6837","nonce":3781833565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:27.697 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 63 2023-10-30T13:23:27.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdffc0664d0 msgr2=0x7fdffc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdffc0664d0 0x7fdffc068990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe014000f70 tx=0x7fe014009040 comp rx=0 tx=0).stop 2023-10-30T13:23:27.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe018071110 msgr2=0x7fe01814fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:27.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe018071110 0x7fe01814fcb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe0080b3040 tx=0x7fe00801bad0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 shutdown_connections 2023-10-30T13:23:27.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdffc0664d0 0x7fdffc068990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe018071110 0x7fe01814fcb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0180c6bd0 0x7fe018153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 --2- 172.21.15.148:0/909880874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe01806f7a0 0x7fe0181e4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:27.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 >> 172.21.15.148:0/909880874 conn(0x7fe0180c7910 msgr2=0x7fe0180c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:27.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 shutdown_connections 2023-10-30T13:23:27.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:27.701+0000 7fe0205d3640 1 -- 172.21.15.148:0/909880874 wait complete. 2023-10-30T13:23:27.716 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:23:29.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:29.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.865+0000 7f7c17b3e640 1 Processor -- start 2023-10-30T13:23:29.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.865+0000 7f7c17b3e640 1 -- start start 2023-10-30T13:23:29.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.865+0000 7f7c17b3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7c10070a00 0x7f7c10070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:29.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.865+0000 7f7c17b3e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7c10069e90 0x7f7c1006a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:29.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c17b3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c100713b0 con 0x7f7c101539a0 2023-10-30T13:23:29.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c17b3e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1006fbb0 con 0x7f7c10069e90 2023-10-30T13:23:29.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c17b3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c10153d80 con 0x7f7c10070a00 2023-10-30T13:23:29.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c160db640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7c10069e90 0x7f7c1006a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:29.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c160db640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7c10069e90 0x7f7c1006a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55378/0 (socket says 172.21.15.148:55378) 2023-10-30T13:23:29.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c160db640 1 -- 172.21.15.148:0/2912964642 learned_addr learned my addr 172.21.15.148:0/2912964642 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:29.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c150d9640 1 --2- 172.21.15.148:0/2912964642 >> v2:172.21.15.150:3300/0 conn(0x7f7c10070a00 0x7f7c10070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:29.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c160db640 1 -- 172.21.15.148:0/2912964642 >> v2:172.21.15.150:3300/0 conn(0x7f7c10070a00 msgr2=0x7f7c10070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:29.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c160db640 1 --2- 172.21.15.148:0/2912964642 >> v2:172.21.15.150:3300/0 conn(0x7f7c10070a00 0x7f7c10070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:29.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c160db640 1 -- 172.21.15.148:0/2912964642 >> v1:172.21.15.150:6789/0 conn(0x7f7c101539a0 legacy=0x7f7c1015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:29.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c160db640 1 -- 172.21.15.148:0/2912964642 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100c5c50 con 0x7f7c10069e90 2023-10-30T13:23:29.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c160db640 1 --2- 172.21.15.148:0/2912964642 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 0x7f7c1006a2c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7c10072640 tx=0x7f7c0801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe627a0fed04081 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:29.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c148d8640 1 -- 172.21.15.148:0/2912964642 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080a56a0 con 0x7f7c10069e90 2023-10-30T13:23:29.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c148d8640 1 -- 172.21.15.148:0/2912964642 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c080a5800 con 0x7f7c10069e90 2023-10-30T13:23:29.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.869+0000 7f7c148d8640 1 -- 172.21.15.148:0/2912964642 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c080a5b90 con 0x7f7c10069e90 2023-10-30T13:23:29.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2912964642 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 msgr2=0x7f7c1006a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:29.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 --2- 172.21.15.148:0/2912964642 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 0x7f7c1006a2c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7c10072640 tx=0x7f7c0801c8d0 comp rx=0 tx=0).stop 2023-10-30T13:23:29.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2912964642 shutdown_connections 2023-10-30T13:23:29.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 --2- 172.21.15.148:0/2912964642 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 0x7f7c1006a2c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:29.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 --2- 172.21.15.148:0/2912964642 >> v2:172.21.15.150:3300/0 conn(0x7f7c10070a00 0x7f7c10070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:29.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2912964642 >> 172.21.15.148:0/2912964642 conn(0x7f7c100c7910 msgr2=0x7f7c100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:29.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2912964642 shutdown_connections 2023-10-30T13:23:29.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2912964642 wait complete. 2023-10-30T13:23:29.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 Processor -- start 2023-10-30T13:23:29.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 -- start start 2023-10-30T13:23:29.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 0x7f7c10157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:29.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c10070a00 0x7f7c100c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:29.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c101539a0 0x7f7c10159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:29.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100ca160 con 0x7f7c10069e90 2023-10-30T13:23:29.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c101597c0 con 0x7f7c101539a0 2023-10-30T13:23:29.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c17b3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c10158df0 con 0x7f7c10070a00 2023-10-30T13:23:29.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c158da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 0x7f7c10157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:29.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c158da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 0x7f7c10157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55380/0 (socket says 172.21.15.148:55380) 2023-10-30T13:23:29.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c158da640 1 -- 172.21.15.148:0/2754523821 learned_addr learned my addr 172.21.15.148:0/2754523821 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:29.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c160db640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c101539a0 0x7f7c10159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:29.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c150d9640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c10070a00 0x7f7c100c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:29.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c158da640 1 -- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c10070a00 msgr2=0x7f7c100c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:29.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c158da640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c10070a00 0x7f7c100c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:29.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c158da640 1 -- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c101539a0 msgr2=0x7f7c10159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:29.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c158da640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c101539a0 0x7f7c10159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:29.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c158da640 1 -- 172.21.15.148:0/2754523821 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c08009b30 con 0x7f7c10069e90 2023-10-30T13:23:29.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.873+0000 7f7c160db640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c101539a0 0x7f7c10159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:29.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c158da640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 0x7f7c10157430 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00c480 tx=0x7f7c0c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:29.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c067fc640 1 -- 172.21.15.148:0/2754523821 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c012830 con 0x7f7c10069e90 2023-10-30T13:23:29.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c067fc640 1 -- 172.21.15.148:0/2754523821 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c0129b0 con 0x7f7c10069e90 2023-10-30T13:23:29.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c067fc640 1 -- 172.21.15.148:0/2754523821 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c012d60 con 0x7f7c10069e90 2023-10-30T13:23:29.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c100c9380 con 0x7f7c10069e90 2023-10-30T13:23:29.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c100c9940 con 0x7f7c10069e90 2023-10-30T13:23:29.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1004e170 con 0x7f7c10069e90 2023-10-30T13:23:29.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c067fc640 1 -- 172.21.15.148:0/2754523821 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7c0c01c460 con 0x7f7c10069e90 2023-10-30T13:23:29.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c067fc640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bf40664d0 0x7f7bf4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:29.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.877+0000 7f7c067fc640 1 -- 172.21.15.148:0/2754523821 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f7c0c083fe0 con 0x7f7c10069e90 2023-10-30T13:23:29.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.881+0000 7f7c150d9640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bf40664d0 0x7f7bf4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:29.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.881+0000 7f7c150d9640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bf40664d0 0x7f7bf4068990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7c100b0b90 tx=0x7f7c00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:29.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:29.881+0000 7f7c067fc640 1 -- 172.21.15.148:0/2754523821 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7c0c0511d0 con 0x7f7c10069e90 2023-10-30T13:23:30.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c100c6a80 con 0x7f7c10069e90 2023-10-30T13:23:30.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c067fc640 1 -- 172.21.15.148:0/2754523821 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f7c0c0547f0 con 0x7f7c10069e90 2023-10-30T13:23:30.066 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:30.066 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:29.531922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:30.066 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 64 2023-10-30T13:23:30.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bf40664d0 msgr2=0x7f7bf4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bf40664d0 0x7f7bf4068990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7c100b0b90 tx=0x7f7c00008040 comp rx=0 tx=0).stop 2023-10-30T13:23:30.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 msgr2=0x7f7c10157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 0x7f7c10157430 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00c480 tx=0x7f7c0c00c940 comp rx=0 tx=0).stop 2023-10-30T13:23:30.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 shutdown_connections 2023-10-30T13:23:30.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bf40664d0 0x7f7bf4068990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c101539a0 0x7f7c10159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c10070a00 0x7f7c100c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 --2- 172.21.15.148:0/2754523821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c10069e90 0x7f7c10157430 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 >> 172.21.15.148:0/2754523821 conn(0x7f7c100c7910 msgr2=0x7f7c10150790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 shutdown_connections 2023-10-30T13:23:30.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.069+0000 7f7c17b3e640 1 -- 172.21.15.148:0/2754523821 wait complete. 2023-10-30T13:23:30.086 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:23:30.087 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:23:30.088 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:30.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.229+0000 7fa93c5c5640 1 Processor -- start 2023-10-30T13:23:30.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.229+0000 7fa93c5c5640 1 -- start start 2023-10-30T13:23:30.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.229+0000 7fa93c5c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9340c6b80 0x7fa9340bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.229+0000 7fa93c5c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa93413f4a0 0x7fa9340c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.229+0000 7fa93c5c5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9340c0a50 con 0x7fa9340c6f60 2023-10-30T13:23:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.229+0000 7fa93c5c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93406f6e0 con 0x7fa93413f4a0 2023-10-30T13:23:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.229+0000 7fa93c5c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93406f9c0 con 0x7fa9340c6b80 2023-10-30T13:23:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa939b60640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa9340c6f60 0x7fa93413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34272/0 (socket says 172.21.15.148:34272) 2023-10-30T13:23:30.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa939b60640 1 -- 172.21.15.148:0/3667575887 learned_addr learned my addr 172.21.15.148:0/3667575887 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:30.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93a361640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9340c6b80 0x7fa9340bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93ab62640 1 --2- 172.21.15.148:0/3667575887 >> v2:172.21.15.150:3300/0 conn(0x7fa93413f4a0 0x7fa9340c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712911317 0 0) 0x7fa9340c0a50 con 0x7fa9340c6f60 2023-10-30T13:23:30.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa920003530 con 0x7fa9340c6f60 2023-10-30T13:23:30.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1094249439 0 0) 0x7fa920003530 con 0x7fa9340c6f60 2023-10-30T13:23:30.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9340c6b80 msgr2=0x7fa9340bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9340c6b80 0x7fa9340bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 >> v2:172.21.15.150:3300/0 conn(0x7fa93413f4a0 msgr2=0x7fa9340c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 --2- 172.21.15.148:0/3667575887 >> v2:172.21.15.150:3300/0 conn(0x7fa93413f4a0 0x7fa9340c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93406fca0 con 0x7fa9340c6f60 2023-10-30T13:23:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fa924001680 con 0x7fa9340c6f60 2023-10-30T13:23:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 >> v1:172.21.15.148:6789/0 conn(0x7fa9340c6f60 legacy=0x7fa93413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa920003980 0x7fa920003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9200047d0 0x7fa920004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9340c6b80 0x7fa920005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93406f6e0 con 0x7fa920003980 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93406f9c0 con 0x7fa9200047d0 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92000b990 con 0x7fa9340c6b80 2023-10-30T13:23:30.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93a361640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa920003980 0x7fa920003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93ab62640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9340c6b80 0x7fa920005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa939b60640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9200047d0 0x7fa920004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93a361640 1 -- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9340c6b80 msgr2=0x7fa920005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93a361640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9340c6b80 0x7fa920005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93a361640 1 -- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9200047d0 msgr2=0x7fa920004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:30.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93a361640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9200047d0 0x7fa920004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa93a361640 1 -- 172.21.15.148:0/3667575887 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa920003530 con 0x7fa920003980 2023-10-30T13:23:30.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.233+0000 7fa939b60640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9200047d0 0x7fa920004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:30.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93a361640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa920003980 0x7fa920003da0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa930009ac0 tx=0x7fa93001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17263cf4423f1593 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9300a6440 con 0x7fa920003980 2023-10-30T13:23:30.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9300a65a0 con 0x7fa920003980 2023-10-30T13:23:30.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93935f640 1 -- 172.21.15.148:0/3667575887 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9300a6930 con 0x7fa920003980 2023-10-30T13:23:30.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3667575887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa920003980 msgr2=0x7fa920003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93c5c5640 1 --2- 172.21.15.148:0/3667575887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa920003980 0x7fa920003da0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa930009ac0 tx=0x7fa93001ca30 comp rx=0 tx=0).stop 2023-10-30T13:23:30.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93ab62640 1 -- 172.21.15.148:0/3667575887 reap_dead start 2023-10-30T13:23:30.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3667575887 shutdown_connections 2023-10-30T13:23:30.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3667575887 >> 172.21.15.148:0/3667575887 conn(0x7fa9340c7910 msgr2=0x7fa9340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3667575887 shutdown_connections 2023-10-30T13:23:30.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3667575887 wait complete. 2023-10-30T13:23:30.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93c5c5640 1 Processor -- start 2023-10-30T13:23:30.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.237+0000 7fa93c5c5640 1 -- start start 2023-10-30T13:23:30.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93c5c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa920003980 0x7fa9341384b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93c5c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa934154480 0x7fa9341e93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93c5c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9341548c0 0x7fa9341e9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93c5c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9340c3d30 con 0x7fa934154480 2023-10-30T13:23:30.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93c5c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9341f2480 con 0x7fa920003980 2023-10-30T13:23:30.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93c5c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa934159570 con 0x7fa9341548c0 2023-10-30T13:23:30.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa939b60640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa934154480 0x7fa9341e93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa939b60640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa934154480 0x7fa9341e93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55398/0 (socket says 172.21.15.148:55398) 2023-10-30T13:23:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa939b60640 1 -- 172.21.15.148:0/3182855359 learned_addr learned my addr 172.21.15.148:0/3182855359 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93ab62640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9341548c0 0x7fa9341e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93a361640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa920003980 0x7fa9341384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa939b60640 1 -- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9341548c0 msgr2=0x7fa9341e9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa939b60640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9341548c0 0x7fa9341e9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa939b60640 1 -- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa920003980 msgr2=0x7fa9341384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa939b60640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa920003980 0x7fa9341384b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa939b60640 1 -- 172.21.15.148:0/3182855359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9341f2670 con 0x7fa934154480 2023-10-30T13:23:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa939b60640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa934154480 0x7fa9341e93d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa924001000 tx=0x7fa92400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa92affd640 1 -- 172.21.15.148:0/3182855359 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92400c660 con 0x7fa934154480 2023-10-30T13:23:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa92affd640 1 -- 172.21.15.148:0/3182855359 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92400c7c0 con 0x7fa934154480 2023-10-30T13:23:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa92affd640 1 -- 172.21.15.148:0/3182855359 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92400cb50 con 0x7fa934154480 2023-10-30T13:23:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9341f2910 con 0x7fa934154480 2023-10-30T13:23:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9341f2ed0 con 0x7fa934154480 2023-10-30T13:23:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93404e170 con 0x7fa934154480 2023-10-30T13:23:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.241+0000 7fa92affd640 1 -- 172.21.15.148:0/3182855359 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa92400ccb0 con 0x7fa934154480 2023-10-30T13:23:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.245+0000 7fa92affd640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9180664d0 0x7fa918068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.245+0000 7fa92affd640 1 -- 172.21.15.148:0/3182855359 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fa924083fe0 con 0x7fa934154480 2023-10-30T13:23:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.245+0000 7fa93a361640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9180664d0 0x7fa918068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.245+0000 7fa93a361640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9180664d0 0x7fa918068990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa9300ad000 tx=0x7fa9300b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.249+0000 7fa92affd640 1 -- 172.21.15.148:0/3182855359 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9240511d0 con 0x7fa934154480 2023-10-30T13:23:30.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.433+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa934157570 con 0x7fa934154480 2023-10-30T13:23:30.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.433+0000 7fa92affd640 1 -- 172.21.15.148:0/3182855359 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5716 (secure 0 0 0) 0x7fa9240547f0 con 0x7fa934154480 2023-10-30T13:23:30.431 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:30.431 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:29.531922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:30.432 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 64 2023-10-30T13:23:30.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9180664d0 msgr2=0x7fa918068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9180664d0 0x7fa918068990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa9300ad000 tx=0x7fa9300b2040 comp rx=0 tx=0).stop 2023-10-30T13:23:30.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa934154480 msgr2=0x7fa9341e93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:30.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa934154480 0x7fa9341e93d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa924001000 tx=0x7fa92400eba0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 shutdown_connections 2023-10-30T13:23:30.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9180664d0 0x7fa918068990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa934154480 0x7fa9341e93d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa9341548c0 0x7fa9341e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 --2- 172.21.15.148:0/3182855359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa920003980 0x7fa9341384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:30.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 >> 172.21.15.148:0/3182855359 conn(0x7fa9340c7910 msgr2=0x7fa9340b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:30.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 shutdown_connections 2023-10-30T13:23:30.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:30.437+0000 7fa93c5c5640 1 -- 172.21.15.148:0/3182855359 wait complete. 2023-10-30T13:23:30.453 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 64, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 64, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5643": { "addr": "172.21.15.148:6835/3194894584", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 3194894584, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 3194894584, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5643, "incarnation": 64, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:23:29.531922+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5643 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:23:32.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:32.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.605+0000 7f30f8b0f640 1 Processor -- start 2023-10-30T13:23:32.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.605+0000 7f30f8b0f640 1 -- start start 2023-10-30T13:23:32.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f8b0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f30f40c6b80 0x7f30f40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f8b0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f40c0a50 con 0x7f30f413f4a0 2023-10-30T13:23:32.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f8b0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f406f6e0 con 0x7f30f40c6f60 2023-10-30T13:23:32.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f8b0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f406f9c0 con 0x7f30f40c6b80 2023-10-30T13:23:32.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f30f40c6b80 0x7f30f40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f2d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f30f413f4a0 0x7f30f41566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58384/0 (socket says 172.21.15.148:58384) 2023-10-30T13:23:32.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f2d9d640 1 -- 172.21.15.148:0/1264226427 learned_addr learned my addr 172.21.15.148:0/1264226427 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:32.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f30f40c6b80 0x7f30f40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33590/0 (socket says 172.21.15.148:33590) 2023-10-30T13:23:32.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f259c640 1 -- 172.21.15.148:0/1264226427 >> v1:172.21.15.150:6789/0 conn(0x7f30f40c6f60 legacy=0x7f30f413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:32.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f259c640 1 -- 172.21.15.148:0/1264226427 >> v1:172.21.15.148:6789/0 conn(0x7f30f413f4a0 legacy=0x7f30f41566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f259c640 1 -- 172.21.15.148:0/1264226427 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f406fca0 con 0x7f30f40c6b80 2023-10-30T13:23:32.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.609+0000 7f30f1d9b640 1 -- 172.21.15.148:0/1264226427 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347377838 0 0) 0x7f30f40c0a50 con 0x7f30f413f4a0 2023-10-30T13:23:32.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f259c640 1 --2- 172.21.15.148:0/1264226427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6b80 0x7f30f40bfec0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f30e40097c0 tx=0x7f30e401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6f8a0b17e2db21 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f1d9b640 1 -- 172.21.15.148:0/1264226427 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e40a8050 con 0x7f30f40c6b80 2023-10-30T13:23:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f1d9b640 1 -- 172.21.15.148:0/1264226427 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e40ac5f0 con 0x7f30f40c6b80 2023-10-30T13:23:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f1d9b640 1 -- 172.21.15.148:0/1264226427 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e40ac980 con 0x7f30f40c6b80 2023-10-30T13:23:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f8b0f640 1 -- 172.21.15.148:0/1264226427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6b80 msgr2=0x7f30f40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f8b0f640 1 --2- 172.21.15.148:0/1264226427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6b80 0x7f30f40bfec0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f30e40097c0 tx=0x7f30e401c480 comp rx=0 tx=0).stop 2023-10-30T13:23:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f8b0f640 1 -- 172.21.15.148:0/1264226427 shutdown_connections 2023-10-30T13:23:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f8b0f640 1 --2- 172.21.15.148:0/1264226427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6b80 0x7f30f40bfec0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f8b0f640 1 -- 172.21.15.148:0/1264226427 >> 172.21.15.148:0/1264226427 conn(0x7f30f40c7910 msgr2=0x7f30f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:32.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f8b0f640 1 -- 172.21.15.148:0/1264226427 shutdown_connections 2023-10-30T13:23:32.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f8b0f640 1 -- 172.21.15.148:0/1264226427 wait complete. 2023-10-30T13:23:32.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.613+0000 7f30f8b0f640 1 Processor -- start 2023-10-30T13:23:32.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 -- start start 2023-10-30T13:23:32.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30f40c6b80 0x7f30f4154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6f60 0x7f30f4134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30f413f4a0 0x7f30f40c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f413e900 con 0x7f30f40c6f60 2023-10-30T13:23:32.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f41597a0 con 0x7f30f413f4a0 2023-10-30T13:23:32.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f413ae10 con 0x7f30f40c6b80 2023-10-30T13:23:32.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30e9d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6f60 0x7f30f4134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30e9d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6f60 0x7f30f4134560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33602/0 (socket says 172.21.15.148:33602) 2023-10-30T13:23:32.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30e9d9b640 1 -- 172.21.15.148:0/835697797 learned_addr learned my addr 172.21.15.148:0/835697797 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:32.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f2d9d640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30f413f4a0 0x7f30f40c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f259c640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30f40c6b80 0x7f30f4154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30e9d9b640 1 -- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30f40c6b80 msgr2=0x7f30f4154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30e9d9b640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30f40c6b80 0x7f30f4154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30e9d9b640 1 -- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30f413f4a0 msgr2=0x7f30f40c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30e9d9b640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30f413f4a0 0x7f30f40c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30e9d9b640 1 -- 172.21.15.148:0/835697797 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e4009420 con 0x7f30f40c6f60 2023-10-30T13:23:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30e9d9b640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6f60 0x7f30f4134560 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00c820 tx=0x7f30dc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30eb7fe640 1 -- 172.21.15.148:0/835697797 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc003b80 con 0x7f30f40c6f60 2023-10-30T13:23:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30eb7fe640 1 -- 172.21.15.148:0/835697797 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30dc003ce0 con 0x7f30f40c6f60 2023-10-30T13:23:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f40c8c10 con 0x7f30f40c6f60 2023-10-30T13:23:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30eb7fe640 1 -- 172.21.15.148:0/835697797 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00aa90 con 0x7f30f40c6f60 2023-10-30T13:23:32.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f413b060 con 0x7f30f40c6f60 2023-10-30T13:23:32.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.617+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f404e170 con 0x7f30f40c6f60 2023-10-30T13:23:32.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.621+0000 7f30eb7fe640 1 -- 172.21.15.148:0/835697797 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f30dc00abf0 con 0x7f30f40c6f60 2023-10-30T13:23:32.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.621+0000 7f30eb7fe640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30d40664d0 0x7f30d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:32.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.621+0000 7f30eb7fe640 1 -- 172.21.15.148:0/835697797 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f30dc084200 con 0x7f30f40c6f60 2023-10-30T13:23:32.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.621+0000 7f30f259c640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30d40664d0 0x7f30d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:32.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.621+0000 7f30f259c640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30d40664d0 0x7f30d4068990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f30e40ae000 tx=0x7f30e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:32.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.625+0000 7f30eb7fe640 1 -- 172.21.15.148:0/835697797 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f30dc0512b0 con 0x7f30f40c6f60 2023-10-30T13:23:32.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.805+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30f4136200 con 0x7f30f40c6f60 2023-10-30T13:23:32.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.805+0000 7f30eb7fe640 1 -- 172.21.15.148:0/835697797 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f30dc0548d0 con 0x7f30f40c6f60 2023-10-30T13:23:32.802 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:32.802 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:29.531922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:32.803 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 64 2023-10-30T13:23:32.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.805+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30d40664d0 msgr2=0x7f30d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.805+0000 7f30f8b0f640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30d40664d0 0x7f30d4068990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f30e40ae000 tx=0x7f30e4008040 comp rx=0 tx=0).stop 2023-10-30T13:23:32.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6f60 msgr2=0x7f30f4134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:32.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6f60 0x7f30f4134560 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00c820 tx=0x7f30dc00cce0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 shutdown_connections 2023-10-30T13:23:32.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30d40664d0 0x7f30d4068990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30f40c6f60 0x7f30f4134560 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30f413f4a0 0x7f30f40c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 --2- 172.21.15.148:0/835697797 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30f40c6b80 0x7f30f4154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:32.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 >> 172.21.15.148:0/835697797 conn(0x7f30f40c7910 msgr2=0x7f30f40c60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:32.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 shutdown_connections 2023-10-30T13:23:32.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:32.809+0000 7f30f8b0f640 1 -- 172.21.15.148:0/835697797 wait complete. 2023-10-30T13:23:32.823 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:23:34.824 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:34.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.973+0000 7f487eeea640 1 Processor -- start 2023-10-30T13:23:34.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.973+0000 7f487eeea640 1 -- start start 2023-10-30T13:23:34.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487eeea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f487806f7a0 0x7f487806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487eeea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48780c6bd0 0x7f4878069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487eeea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487806a3d0 con 0x7f4878071110 2023-10-30T13:23:34.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487eeea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48780700c0 con 0x7f48780c6bd0 2023-10-30T13:23:34.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487eeea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4878066cf0 con 0x7f487806f7a0 2023-10-30T13:23:34.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487cc86640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f487806f7a0 0x7f487806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f4877fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48780c6bd0 0x7f4878069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487cc86640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f487806f7a0 0x7f487806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:44818/0 (socket says 172.21.15.148:44818) 2023-10-30T13:23:34.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487cc86640 1 -- 172.21.15.148:0/826499360 learned_addr learned my addr 172.21.15.148:0/826499360 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:34.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487cc86640 1 -- 172.21.15.148:0/826499360 >> v2:172.21.15.150:3300/0 conn(0x7f48780c6bd0 msgr2=0x7f4878069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:34.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487cc86640 1 --2- 172.21.15.148:0/826499360 >> v2:172.21.15.150:3300/0 conn(0x7f48780c6bd0 0x7f4878069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487cc86640 1 -- 172.21.15.148:0/826499360 >> v1:172.21.15.150:6789/0 conn(0x7f4878071110 legacy=0x7f4878158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.977+0000 7f487cc86640 1 -- 172.21.15.148:0/826499360 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48780c70d0 con 0x7f487806f7a0 2023-10-30T13:23:34.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487cc86640 1 --2- 172.21.15.148:0/826499360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f487806f7a0 0x7f487806fb80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f486c009870 tx=0x7f486c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25fe569b21d70413 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f48777fe640 1 -- 172.21.15.148:0/826499360 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c0a8a30 con 0x7f487806f7a0 2023-10-30T13:23:34.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f48777fe640 1 -- 172.21.15.148:0/826499360 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c0a8b90 con 0x7f487806f7a0 2023-10-30T13:23:34.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f48777fe640 1 -- 172.21.15.148:0/826499360 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c0b0440 con 0x7f487806f7a0 2023-10-30T13:23:34.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 -- 172.21.15.148:0/826499360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f487806f7a0 msgr2=0x7f487806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 --2- 172.21.15.148:0/826499360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f487806f7a0 0x7f487806fb80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f486c009870 tx=0x7f486c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:34.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 -- 172.21.15.148:0/826499360 shutdown_connections 2023-10-30T13:23:34.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 --2- 172.21.15.148:0/826499360 >> v2:172.21.15.150:3300/0 conn(0x7f48780c6bd0 0x7f4878069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 --2- 172.21.15.148:0/826499360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f487806f7a0 0x7f487806fb80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 -- 172.21.15.148:0/826499360 >> 172.21.15.148:0/826499360 conn(0x7f48780c7910 msgr2=0x7f48780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:34.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 -- 172.21.15.148:0/826499360 shutdown_connections 2023-10-30T13:23:34.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 -- 172.21.15.148:0/826499360 wait complete. 2023-10-30T13:23:34.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 Processor -- start 2023-10-30T13:23:34.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.981+0000 7f487eeea640 1 -- start start 2023-10-30T13:23:34.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487eeea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f487806f7a0 0x7f487815b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487eeea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4878071110 0x7f48781f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487eeea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48780c6bd0 0x7f48780c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487eeea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48781ef5f0 con 0x7f48780c6bd0 2023-10-30T13:23:34.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487eeea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48781eed00 con 0x7f487806f7a0 2023-10-30T13:23:34.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487eeea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487815bb60 con 0x7f4878071110 2023-10-30T13:23:34.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487d487640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48780c6bd0 0x7f48780c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487cc86640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f487806f7a0 0x7f487815b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f4877fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4878071110 0x7f48781f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487d487640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48780c6bd0 0x7f48780c9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33614/0 (socket says 172.21.15.148:33614) 2023-10-30T13:23:34.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487d487640 1 -- 172.21.15.148:0/3068243347 learned_addr learned my addr 172.21.15.148:0/3068243347 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:34.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487d487640 1 -- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4878071110 msgr2=0x7f48781f21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487d487640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4878071110 0x7f48781f21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487d487640 1 -- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f487806f7a0 msgr2=0x7f487815b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:34.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487d487640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f487806f7a0 0x7f487815b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:34.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487d487640 1 -- 172.21.15.148:0/3068243347 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c0094d0 con 0x7f48780c6bd0 2023-10-30T13:23:34.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487d487640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48780c6bd0 0x7f48780c9120 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f487000c730 tx=0x7f487000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f48757fa640 1 -- 172.21.15.148:0/3068243347 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4870003b80 con 0x7f48780c6bd0 2023-10-30T13:23:34.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f48757fa640 1 -- 172.21.15.148:0/3068243347 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4870003ce0 con 0x7f48780c6bd0 2023-10-30T13:23:34.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f48757fa640 1 -- 172.21.15.148:0/3068243347 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487000aa30 con 0x7f48780c6bd0 2023-10-30T13:23:34.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48780adca0 con 0x7f48780c6bd0 2023-10-30T13:23:34.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48781ef280 con 0x7f48780c6bd0 2023-10-30T13:23:34.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.985+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48781ef440 con 0x7f48780c6bd0 2023-10-30T13:23:34.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.989+0000 7f48757fa640 1 -- 172.21.15.148:0/3068243347 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f487000ab90 con 0x7f48780c6bd0 2023-10-30T13:23:34.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.989+0000 7f48757fa640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f485c0664d0 0x7f485c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:34.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.989+0000 7f48757fa640 1 -- 172.21.15.148:0/3068243347 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f4870083eb0 con 0x7f48780c6bd0 2023-10-30T13:23:34.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.989+0000 7f487cc86640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f485c0664d0 0x7f485c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:34.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.989+0000 7f487cc86640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f485c0664d0 0x7f485c068990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f486c0b2000 tx=0x7f486c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:34.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:34.997+0000 7f48757fa640 1 -- 172.21.15.148:0/3068243347 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4870051010 con 0x7f48780c6bd0 2023-10-30T13:23:35.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.193+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f487804e170 con 0x7f48780c6bd0 2023-10-30T13:23:35.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.193+0000 7f48757fa640 1 -- 172.21.15.148:0/3068243347 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f4870054630 con 0x7f48780c6bd0 2023-10-30T13:23:35.190 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:35.191 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:29.531922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:35.191 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 64 2023-10-30T13:23:35.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f485c0664d0 msgr2=0x7f485c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f485c0664d0 0x7f485c068990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f486c0b2000 tx=0x7f486c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:23:35.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48780c6bd0 msgr2=0x7f48780c9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:35.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48780c6bd0 0x7f48780c9120 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f487000c730 tx=0x7f487000cbf0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 shutdown_connections 2023-10-30T13:23:35.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f485c0664d0 0x7f485c068990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48780c6bd0 0x7f48780c9120 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4878071110 0x7f48781f21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 --2- 172.21.15.148:0/3068243347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f487806f7a0 0x7f487815b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:35.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 >> 172.21.15.148:0/3068243347 conn(0x7f48780c7910 msgr2=0x7f48780c3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:35.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 shutdown_connections 2023-10-30T13:23:35.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:35.197+0000 7f487eeea640 1 -- 172.21.15.148:0/3068243347 wait complete. 2023-10-30T13:23:35.212 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:23:37.214 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:37.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.357+0000 7f4d9cd1d640 1 Processor -- start 2023-10-30T13:23:37.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.357+0000 7f4d9cd1d640 1 -- start start 2023-10-30T13:23:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9cd1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d980c6b80 0x7f4d980bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9cd1d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d980c0a50 con 0x7f4d9813f4a0 2023-10-30T13:23:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9cd1d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9806f6e0 con 0x7f4d980c6f60 2023-10-30T13:23:37.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9cd1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9806f9c0 con 0x7f4d980c6b80 2023-10-30T13:23:37.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d96d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4d9813f4a0 0x7f4d981566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58398/0 (socket says 172.21.15.148:58398) 2023-10-30T13:23:37.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d96d9d640 1 -- 172.21.15.148:0/24539740 learned_addr learned my addr 172.21.15.148:0/24539740 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:37.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9659c640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d980c6b80 0x7f4d980bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:37.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922498237 0 0) 0x7f4d980c0a50 con 0x7f4d9813f4a0 2023-10-30T13:23:37.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d84003530 con 0x7f4d9813f4a0 2023-10-30T13:23:37.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1890620291 0 0) 0x7f4d84003530 con 0x7f4d9813f4a0 2023-10-30T13:23:37.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d980c6b80 msgr2=0x7f4d980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:37.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d980c6b80 0x7f4d980bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 >> v1:172.21.15.150:6789/0 conn(0x7f4d980c6f60 legacy=0x7f4d9813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9806fca0 con 0x7f4d9813f4a0 2023-10-30T13:23:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f4d90001a00 con 0x7f4d9813f4a0 2023-10-30T13:23:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9659c640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d980c6b80 0x7f4d980bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 >> v1:172.21.15.148:6789/0 conn(0x7f4d9813f4a0 legacy=0x7f4d981566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d840039b0 0x7f4d84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:37.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d980c6b80 0x7f4d840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d84004d10 0x7f4d84005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9806f9c0 con 0x7f4d840039b0 2023-10-30T13:23:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8400b990 con 0x7f4d84004d10 2023-10-30T13:23:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8400bb00 con 0x7f4d980c6b80 2023-10-30T13:23:37.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9659c640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d840039b0 0x7f4d84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d96d9d640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d84004d10 0x7f4d84005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d8dd9b640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d980c6b80 0x7f4d840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9659c640 1 -- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d980c6b80 msgr2=0x7f4d840047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9659c640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d980c6b80 0x7f4d840047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9659c640 1 -- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d84004d10 msgr2=0x7f4d84005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:37.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9659c640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d84004d10 0x7f4d84005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.361+0000 7f4d9659c640 1 -- 172.21.15.148:0/24539740 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d88009420 con 0x7f4d840039b0 2023-10-30T13:23:37.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d9659c640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d840039b0 0x7f4d84003d90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4d88009840 tx=0x7f4d8801ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed50c425edbbea8c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:37.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d880a7ac0 con 0x7f4d840039b0 2023-10-30T13:23:37.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d880a7c20 con 0x7f4d840039b0 2023-10-30T13:23:37.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d95d9b640 1 -- 172.21.15.148:0/24539740 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d880af440 con 0x7f4d840039b0 2023-10-30T13:23:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/24539740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d840039b0 msgr2=0x7f4d84003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d9cd1d640 1 --2- 172.21.15.148:0/24539740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d840039b0 0x7f4d84003d90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4d88009840 tx=0x7f4d8801ee00 comp rx=0 tx=0).stop 2023-10-30T13:23:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d96d9d640 1 -- 172.21.15.148:0/24539740 reap_dead start 2023-10-30T13:23:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/24539740 shutdown_connections 2023-10-30T13:23:37.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/24539740 >> 172.21.15.148:0/24539740 conn(0x7f4d980c7910 msgr2=0x7f4d980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/24539740 shutdown_connections 2023-10-30T13:23:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.365+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/24539740 wait complete. 2023-10-30T13:23:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 Processor -- start 2023-10-30T13:23:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 -- start start 2023-10-30T13:23:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d840039b0 0x7f4d9813c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:37.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d9813a410 0x7f4d9813e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d981544a0 0x7f4d98154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9813dff0 con 0x7f4d981544a0 2023-10-30T13:23:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980c8510 con 0x7f4d9813a410 2023-10-30T13:23:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d98155780 con 0x7f4d840039b0 2023-10-30T13:23:37.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d96d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d981544a0 0x7f4d98154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d96d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d981544a0 0x7f4d98154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33622/0 (socket says 172.21.15.148:33622) 2023-10-30T13:23:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d96d9d640 1 -- 172.21.15.148:0/2848450577 learned_addr learned my addr 172.21.15.148:0/2848450577 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9659c640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d840039b0 0x7f4d9813c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d8dd9b640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d9813a410 0x7f4d9813e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:37.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d96d9d640 1 -- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d840039b0 msgr2=0x7f4d9813c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d96d9d640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d840039b0 0x7f4d9813c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d96d9d640 1 -- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d9813a410 msgr2=0x7f4d9813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d96d9d640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d9813a410 0x7f4d9813e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d96d9d640 1 -- 172.21.15.148:0/2848450577 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d98155a20 con 0x7f4d981544a0 2023-10-30T13:23:37.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d8dd9b640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d9813a410 0x7f4d9813e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d96d9d640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d981544a0 0x7f4d98154920 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4d9813f8d0 tx=0x7f4d9000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d8f7fe640 1 -- 172.21.15.148:0/2848450577 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9000ec10 con 0x7f4d981544a0 2023-10-30T13:23:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d8f7fe640 1 -- 172.21.15.148:0/2848450577 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9000ed70 con 0x7f4d981544a0 2023-10-30T13:23:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d98156090 con 0x7f4d981544a0 2023-10-30T13:23:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d8f7fe640 1 -- 172.21.15.148:0/2848450577 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9000eee0 con 0x7f4d981544a0 2023-10-30T13:23:37.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d98156650 con 0x7f4d981544a0 2023-10-30T13:23:37.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.369+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d9804e170 con 0x7f4d981544a0 2023-10-30T13:23:37.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.373+0000 7f4d8f7fe640 1 -- 172.21.15.148:0/2848450577 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4d90014670 con 0x7f4d981544a0 2023-10-30T13:23:37.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.373+0000 7f4d8f7fe640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4d780664d0 0x7f4d78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:37.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.373+0000 7f4d8f7fe640 1 -- 172.21.15.148:0/2848450577 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f4d90083ef0 con 0x7f4d981544a0 2023-10-30T13:23:37.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.373+0000 7f4d9659c640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4d780664d0 0x7f4d78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:37.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.373+0000 7f4d9659c640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4d780664d0 0x7f4d78068990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4d880b1000 tx=0x7f4d880046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:37.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.377+0000 7f4d8f7fe640 1 -- 172.21.15.148:0/2848450577 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4d90054b70 con 0x7f4d981544a0 2023-10-30T13:23:37.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d98138040 con 0x7f4d981544a0 2023-10-30T13:23:37.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d8f7fe640 1 -- 172.21.15.148:0/2848450577 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f4d90054700 con 0x7f4d981544a0 2023-10-30T13:23:37.553 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:37.553 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:36.309901+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:37.553 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 65 2023-10-30T13:23:37.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4d780664d0 msgr2=0x7f4d78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:37.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4d780664d0 0x7f4d78068990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4d880b1000 tx=0x7f4d880046d0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d981544a0 msgr2=0x7f4d98154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:37.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d981544a0 0x7f4d98154920 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4d9813f8d0 tx=0x7f4d9000e530 comp rx=0 tx=0).stop 2023-10-30T13:23:37.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 shutdown_connections 2023-10-30T13:23:37.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4d780664d0 0x7f4d78068990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d981544a0 0x7f4d98154920 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d9813a410 0x7f4d9813e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 --2- 172.21.15.148:0/2848450577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4d840039b0 0x7f4d9813c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:37.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 >> 172.21.15.148:0/2848450577 conn(0x7f4d980c7910 msgr2=0x7f4d980c10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:37.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 shutdown_connections 2023-10-30T13:23:37.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:37.557+0000 7f4d9cd1d640 1 -- 172.21.15.148:0/2848450577 wait complete. 2023-10-30T13:23:37.573 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:23:39.574 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.725+0000 7efd66d0f640 1 Processor -- start 2023-10-30T13:23:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.725+0000 7efd66d0f640 1 -- start start 2023-10-30T13:23:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.725+0000 7efd66d0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efd60159700 0x7efd60162300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.725+0000 7efd66d0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd600c6050 con 0x7efd600670f0 2023-10-30T13:23:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.725+0000 7efd66d0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd600c6e50 con 0x7efd60153200 2023-10-30T13:23:39.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.725+0000 7efd66d0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd600c62a0 con 0x7efd60159700 2023-10-30T13:23:39.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd57fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efd600670f0 0x7efd60158ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58404/0 (socket says 172.21.15.148:58404) 2023-10-30T13:23:39.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd57fff640 1 -- 172.21.15.148:0/2997029615 learned_addr learned my addr 172.21.15.148:0/2997029615 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:39.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd652ac640 1 --2- 172.21.15.148:0/2997029615 >> v2:172.21.15.148:3300/0 conn(0x7efd60159700 0x7efd60162300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd652ac640 1 -- 172.21.15.148:0/2997029615 >> v1:172.21.15.150:6789/0 conn(0x7efd60153200 legacy=0x7efd600c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd652ac640 1 -- 172.21.15.148:0/2997029615 >> v1:172.21.15.148:6789/0 conn(0x7efd600670f0 legacy=0x7efd60158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd652ac640 1 -- 172.21.15.148:0/2997029615 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd600c70f0 con 0x7efd60159700 2023-10-30T13:23:39.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd652ac640 1 --2- 172.21.15.148:0/2997029615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 0x7efd60162300 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7efd58009d90 tx=0x7efd5801d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e8a418f53ba82f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:39.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd577fe640 1 -- 172.21.15.148:0/2997029615 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd580a5a00 con 0x7efd60159700 2023-10-30T13:23:39.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd577fe640 1 -- 172.21.15.148:0/2997029615 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd580a5b60 con 0x7efd60159700 2023-10-30T13:23:39.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.729+0000 7efd577fe640 1 -- 172.21.15.148:0/2997029615 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd580ad440 con 0x7efd60159700 2023-10-30T13:23:39.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 -- 172.21.15.148:0/2997029615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 msgr2=0x7efd60162300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:39.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 --2- 172.21.15.148:0/2997029615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 0x7efd60162300 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7efd58009d90 tx=0x7efd5801d380 comp rx=0 tx=0).stop 2023-10-30T13:23:39.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 -- 172.21.15.148:0/2997029615 shutdown_connections 2023-10-30T13:23:39.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 --2- 172.21.15.148:0/2997029615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 0x7efd60162300 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:39.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 -- 172.21.15.148:0/2997029615 >> 172.21.15.148:0/2997029615 conn(0x7efd6006d770 msgr2=0x7efd6006db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:39.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 -- 172.21.15.148:0/2997029615 shutdown_connections 2023-10-30T13:23:39.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 -- 172.21.15.148:0/2997029615 wait complete. 2023-10-30T13:23:39.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 Processor -- start 2023-10-30T13:23:39.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 -- start start 2023-10-30T13:23:39.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd600670f0 0x7efd600c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:39.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd60153200 0x7efd600c2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:39.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 0x7efd600cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:39.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd601f21e0 con 0x7efd60159700 2023-10-30T13:23:39.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd600c3af0 con 0x7efd600670f0 2023-10-30T13:23:39.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd66d0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd60155160 con 0x7efd60153200 2023-10-30T13:23:39.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd652ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 0x7efd600cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:39.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd652ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 0x7efd600cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33644/0 (socket says 172.21.15.148:33644) 2023-10-30T13:23:39.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd652ac640 1 -- 172.21.15.148:0/770776441 learned_addr learned my addr 172.21.15.148:0/770776441 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:39.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd64aab640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd600670f0 0x7efd600c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:39.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd57fff640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd60153200 0x7efd600c2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:39.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd652ac640 1 -- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd60153200 msgr2=0x7efd600c2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:39.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd652ac640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd60153200 0x7efd600c2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:39.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd652ac640 1 -- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd600670f0 msgr2=0x7efd600c35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:39.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd652ac640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd600670f0 0x7efd600c35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:39.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd652ac640 1 -- 172.21.15.148:0/770776441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58009a70 con 0x7efd60159700 2023-10-30T13:23:39.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd64aab640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd600670f0 0x7efd600c35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:39.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.733+0000 7efd652ac640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 0x7efd600cb2a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7efd58019530 tx=0x7efd580196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd557fa640 1 -- 172.21.15.148:0/770776441 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd580ade40 con 0x7efd60159700 2023-10-30T13:23:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd557fa640 1 -- 172.21.15.148:0/770776441 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd580b8420 con 0x7efd60159700 2023-10-30T13:23:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd60155440 con 0x7efd60159700 2023-10-30T13:23:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd557fa640 1 -- 172.21.15.148:0/770776441 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd580b87d0 con 0x7efd60159700 2023-10-30T13:23:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd601f58f0 con 0x7efd60159700 2023-10-30T13:23:39.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd60074250 con 0x7efd60159700 2023-10-30T13:23:39.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd557fa640 1 -- 172.21.15.148:0/770776441 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd580b4040 con 0x7efd60159700 2023-10-30T13:23:39.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd557fa640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd4c0664a0 0x7efd4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:39.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd557fa640 1 -- 172.21.15.148:0/770776441 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7efd5811ef50 con 0x7efd60159700 2023-10-30T13:23:39.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.737+0000 7efd64aab640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd4c0664a0 0x7efd4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:39.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.741+0000 7efd64aab640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd4c0664a0 0x7efd4c068960 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efd5c000f70 tx=0x7efd5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:39.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.741+0000 7efd557fa640 1 -- 172.21.15.148:0/770776441 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd580ec1e0 con 0x7efd60159700 2023-10-30T13:23:39.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.925+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd601f5750 con 0x7efd60159700 2023-10-30T13:23:39.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.925+0000 7efd557fa640 1 -- 172.21.15.148:0/770776441 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5720 (secure 0 0 0) 0x7efd601f5750 con 0x7efd60159700 2023-10-30T13:23:39.922 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:39.922 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:36.309901+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 65 2023-10-30T13:23:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd4c0664a0 msgr2=0x7efd4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd4c0664a0 0x7efd4c068960 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efd5c000f70 tx=0x7efd5c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 msgr2=0x7efd600cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 0x7efd600cb2a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7efd58019530 tx=0x7efd580196c0 comp rx=0 tx=0).stop 2023-10-30T13:23:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 shutdown_connections 2023-10-30T13:23:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd4c0664a0 0x7efd4c068960 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd60159700 0x7efd600cb2a0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd60153200 0x7efd600c2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 --2- 172.21.15.148:0/770776441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd600670f0 0x7efd600c35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 >> 172.21.15.148:0/770776441 conn(0x7efd6006d770 msgr2=0x7efd60162030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 shutdown_connections 2023-10-30T13:23:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:39.929+0000 7efd66d0f640 1 -- 172.21.15.148:0/770776441 wait complete. 2023-10-30T13:23:39.943 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:23:41.944 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:42.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.089+0000 7f9f806bb640 1 Processor -- start 2023-10-30T13:23:42.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.089+0000 7f9f806bb640 1 -- start start 2023-10-30T13:23:42.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f806bb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9f7815d6f0 0x7f9f7815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:42.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f806bb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f780cf160 0x7f9f780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:42.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f806bb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f780cb800 con 0x7f9f781538a0 2023-10-30T13:23:42.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f806bb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f780cb330 con 0x7f9f7815d6f0 2023-10-30T13:23:42.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f806bb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f780cad70 con 0x7f9f780cf160 2023-10-30T13:23:42.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7dc56640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9f7815d6f0 0x7f9f7815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:42.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7ec58640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f780cf160 0x7f9f780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:42.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7dc56640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9f7815d6f0 0x7f9f7815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:35398/0 (socket says 172.21.15.148:35398) 2023-10-30T13:23:42.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7dc56640 1 -- 172.21.15.148:0/1400843295 learned_addr learned my addr 172.21.15.148:0/1400843295 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:42.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7e457640 1 --1- 172.21.15.148:0/1400843295 >> v1:172.21.15.150:6789/0 conn(0x7f9f781538a0 0x7f9f780ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:38040/0 (socket says 172.21.15.148:38040) 2023-10-30T13:23:42.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7dc56640 1 -- 172.21.15.148:0/1400843295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f780cf160 msgr2=0x7f9f780cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:42.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7dc56640 1 --2- 172.21.15.148:0/1400843295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f780cf160 0x7f9f780cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:42.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7dc56640 1 -- 172.21.15.148:0/1400843295 >> v1:172.21.15.150:6789/0 conn(0x7f9f781538a0 legacy=0x7f9f780ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:42.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7dc56640 1 -- 172.21.15.148:0/1400843295 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f780cb050 con 0x7f9f7815d6f0 2023-10-30T13:23:42.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.093+0000 7f9f7ec58640 1 --2- 172.21.15.148:0/1400843295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f780cf160 0x7f9f780cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:42.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f7dc56640 1 --2- 172.21.15.148:0/1400843295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f7815d6f0 0x7f9f7815a090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009870 tx=0x7f9f6801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e48574a7d9492b75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:42.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f7d455640 1 -- 172.21.15.148:0/1400843295 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f680a8a30 con 0x7f9f7815d6f0 2023-10-30T13:23:42.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f7d455640 1 -- 172.21.15.148:0/1400843295 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f680a8b90 con 0x7f9f7815d6f0 2023-10-30T13:23:42.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f7d455640 1 -- 172.21.15.148:0/1400843295 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f680b0440 con 0x7f9f7815d6f0 2023-10-30T13:23:42.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f806bb640 1 -- 172.21.15.148:0/1400843295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f7815d6f0 msgr2=0x7f9f7815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:42.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f806bb640 1 --2- 172.21.15.148:0/1400843295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f7815d6f0 0x7f9f7815a090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009870 tx=0x7f9f6801ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:42.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f806bb640 1 -- 172.21.15.148:0/1400843295 shutdown_connections 2023-10-30T13:23:42.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f806bb640 1 --2- 172.21.15.148:0/1400843295 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f780cf160 0x7f9f780cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:42.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f806bb640 1 --2- 172.21.15.148:0/1400843295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f7815d6f0 0x7f9f7815a090 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:42.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f806bb640 1 -- 172.21.15.148:0/1400843295 >> 172.21.15.148:0/1400843295 conn(0x7f9f7806bfe0 msgr2=0x7f9f7806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:42.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f806bb640 1 -- 172.21.15.148:0/1400843295 shutdown_connections 2023-10-30T13:23:42.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.097+0000 7f9f806bb640 1 -- 172.21.15.148:0/1400843295 wait complete. 2023-10-30T13:23:42.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 Processor -- start 2023-10-30T13:23:42.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 -- start start 2023-10-30T13:23:42.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f780cf160 0x7f9f781e52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:42.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f781538a0 0x7f9f781e3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:42.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f7815d6f0 0x7f9f78148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:42.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f781e3450 con 0x7f9f7815d6f0 2023-10-30T13:23:42.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f781e4430 con 0x7f9f780cf160 2023-10-30T13:23:42.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f781e4d40 con 0x7f9f781538a0 2023-10-30T13:23:42.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7ec58640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f7815d6f0 0x7f9f78148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:42.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7ec58640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f7815d6f0 0x7f9f78148a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36700/0 (socket says 172.21.15.148:36700) 2023-10-30T13:23:42.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7ec58640 1 -- 172.21.15.148:0/1470494733 learned_addr learned my addr 172.21.15.148:0/1470494733 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:42.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7dc56640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f781538a0 0x7f9f781e3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:42.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7e457640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f780cf160 0x7f9f781e52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:42.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7ec58640 1 -- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f781538a0 msgr2=0x7f9f781e3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:42.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7ec58640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f781538a0 0x7f9f781e3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:42.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7ec58640 1 -- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f780cf160 msgr2=0x7f9f781e52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:42.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7ec58640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f780cf160 0x7f9f781e52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:42.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7ec58640 1 -- 172.21.15.148:0/1470494733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f70009a70 con 0x7f9f7815d6f0 2023-10-30T13:23:42.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7e457640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f780cf160 0x7f9f781e52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:42.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f7ec58640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f7815d6f0 0x7f9f78148a40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000efc0 tx=0x7f9f7000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:42.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f6effd640 1 -- 172.21.15.148:0/1470494733 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70009cc0 con 0x7f9f7815d6f0 2023-10-30T13:23:42.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f6effd640 1 -- 172.21.15.148:0/1470494733 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7000f040 con 0x7f9f7815d6f0 2023-10-30T13:23:42.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f6effd640 1 -- 172.21.15.148:0/1470494733 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f700039a0 con 0x7f9f7815d6f0 2023-10-30T13:23:42.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f680094d0 con 0x7f9f7815d6f0 2023-10-30T13:23:42.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f781e36f0 con 0x7f9f7815d6f0 2023-10-30T13:23:42.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.101+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f780b2a50 con 0x7f9f7815d6f0 2023-10-30T13:23:42.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.105+0000 7f9f6effd640 1 -- 172.21.15.148:0/1470494733 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f70014480 con 0x7f9f7815d6f0 2023-10-30T13:23:42.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.105+0000 7f9f6effd640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f5c0664a0 0x7f9f5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:42.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.105+0000 7f9f6effd640 1 -- 172.21.15.148:0/1470494733 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f9f70084c90 con 0x7f9f7815d6f0 2023-10-30T13:23:42.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.105+0000 7f9f7e457640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f5c0664a0 0x7f9f5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:42.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.105+0000 7f9f7e457640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f5c0664a0 0x7f9f5c068960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9f74000f70 tx=0x7f9f74007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:42.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.109+0000 7f9f6effd640 1 -- 172.21.15.148:0/1470494733 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f70051e60 con 0x7f9f7815d6f0 2023-10-30T13:23:42.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.285+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f7804e170 con 0x7f9f7815d6f0 2023-10-30T13:23:42.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f6effd640 1 -- 172.21.15.148:0/1470494733 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f9f70018090 con 0x7f9f7815d6f0 2023-10-30T13:23:42.284 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:42.284 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:41.557180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:42.284 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 66 2023-10-30T13:23:42.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f5c0664a0 msgr2=0x7f9f5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:42.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f5c0664a0 0x7f9f5c068960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9f74000f70 tx=0x7f9f74007d50 comp rx=0 tx=0).stop 2023-10-30T13:23:42.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f7815d6f0 msgr2=0x7f9f78148a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:42.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f7815d6f0 0x7f9f78148a40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000efc0 tx=0x7f9f7000c490 comp rx=0 tx=0).stop 2023-10-30T13:23:42.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 shutdown_connections 2023-10-30T13:23:42.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9f5c0664a0 0x7f9f5c068960 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:42.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f7815d6f0 0x7f9f78148a40 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:42.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9f781538a0 0x7f9f781e3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:42.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 --2- 172.21.15.148:0/1470494733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f780cf160 0x7f9f781e52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:42.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 >> 172.21.15.148:0/1470494733 conn(0x7f9f7806bfe0 msgr2=0x7f9f781529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:42.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 shutdown_connections 2023-10-30T13:23:42.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:42.289+0000 7f9f806bb640 1 -- 172.21.15.148:0/1470494733 wait complete. 2023-10-30T13:23:42.305 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 66, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 66, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5643": { "addr": "172.21.15.148:6835/3194894584", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 3194894584, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 3194894584, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5643, "incarnation": 64, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 40 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:23:41.557180+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5643 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:23:44.306 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:44.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.457+0000 7fe26e99d640 1 Processor -- start 2023-10-30T13:23:44.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.457+0000 7fe26e99d640 1 -- start start 2023-10-30T13:23:44.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe26e99d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe268153ca0 0x7fe2680c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe26e99d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2681598f0 con 0x7fe2680c0a40 2023-10-30T13:23:44.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe26e99d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe268159b60 con 0x7fe2680c7510 2023-10-30T13:23:44.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe26e99d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe268159e40 con 0x7fe268153ca0 2023-10-30T13:23:44.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe26cf3a640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe2680c0a40 0x7fe2680c0e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34794/0 (socket says 172.21.15.148:34794) 2023-10-30T13:23:44.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe26cf3a640 1 -- 172.21.15.148:0/4005425728 learned_addr learned my addr 172.21.15.148:0/4005425728 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:44.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe267fff640 1 --2- 172.21.15.148:0/4005425728 >> v2:172.21.15.150:3300/0 conn(0x7fe268153ca0 0x7fe2680c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910009753 0 0) 0x7fe2681598f0 con 0x7fe2680c0a40 2023-10-30T13:23:44.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe250003530 con 0x7fe2680c0a40 2023-10-30T13:23:44.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 631466947 0 0) 0x7fe250003530 con 0x7fe2680c0a40 2023-10-30T13:23:44.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 >> v2:172.21.15.150:3300/0 conn(0x7fe268153ca0 msgr2=0x7fe2680c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:44.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 --2- 172.21.15.148:0/4005425728 >> v2:172.21.15.150:3300/0 conn(0x7fe268153ca0 0x7fe2680c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 >> v1:172.21.15.150:6789/0 conn(0x7fe2680c7510 legacy=0x7fe2680c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26815a120 con 0x7fe2680c0a40 2023-10-30T13:23:44.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fe260001a00 con 0x7fe2680c0a40 2023-10-30T13:23:44.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 >> v1:172.21.15.148:6789/0 conn(0x7fe2680c0a40 legacy=0x7fe2680c0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe250003980 0x7fe250003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe250004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe250005190 0x7fe25000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe268159e40 con 0x7fe2500047d0 2023-10-30T13:23:44.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25000c040 con 0x7fe250005190 2023-10-30T13:23:44.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25000c200 con 0x7fe250003980 2023-10-30T13:23:44.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe2677fe640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe250004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe267fff640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe250003980 0x7fe250003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe26cf3a640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe250005190 0x7fe25000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe2677fe640 1 -- 172.21.15.148:0/4005425728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe250003980 msgr2=0x7fe250003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:44.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe2677fe640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe250003980 0x7fe250003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe2677fe640 1 -- 172.21.15.148:0/4005425728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe250005190 msgr2=0x7fe25000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:44.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe2677fe640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe250005190 0x7fe25000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe2677fe640 1 -- 172.21.15.148:0/4005425728 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250003530 con 0x7fe2500047d0 2023-10-30T13:23:44.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe26cf3a640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe250005190 0x7fe25000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:44.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.461+0000 7fe26cf3a640 1 -- 172.21.15.148:0/4005425728 reap_dead start 2023-10-30T13:23:44.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe2677fe640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe250004c50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe25c001780 tx=0x7fe25c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757b811028c03f60 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0052d0 con 0x7fe2500047d0 2023-10-30T13:23:44.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c0a7050 con 0x7fe2500047d0 2023-10-30T13:23:44.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe266ffd640 1 -- 172.21.15.148:0/4005425728 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c003600 con 0x7fe2500047d0 2023-10-30T13:23:44.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe26e99d640 1 -- 172.21.15.148:0/4005425728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 msgr2=0x7fe250004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe26e99d640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe250004c50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe25c001780 tx=0x7fe25c01c3e0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe26e99d640 1 -- 172.21.15.148:0/4005425728 shutdown_connections 2023-10-30T13:23:44.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe26e99d640 1 --2- 172.21.15.148:0/4005425728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe250004c50 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe26e99d640 1 -- 172.21.15.148:0/4005425728 >> 172.21.15.148:0/4005425728 conn(0x7fe26806d770 msgr2=0x7fe26806db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:44.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe26e99d640 1 -- 172.21.15.148:0/4005425728 shutdown_connections 2023-10-30T13:23:44.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe26e99d640 1 -- 172.21.15.148:0/4005425728 wait complete. 2023-10-30T13:23:44.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.465+0000 7fe26e99d640 1 Processor -- start 2023-10-30T13:23:44.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 -- start start 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe2681582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2680cace0 0x7fe268147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe268148810 0x7fe2681481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26814fcb0 con 0x7fe2500047d0 2023-10-30T13:23:44.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681f2360 con 0x7fe268148810 2023-10-30T13:23:44.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681f2600 con 0x7fe2680cace0 2023-10-30T13:23:44.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe267fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe2681582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe267fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe2681582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36712/0 (socket says 172.21.15.148:36712) 2023-10-30T13:23:44.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe267fff640 1 -- 172.21.15.148:0/3053844113 learned_addr learned my addr 172.21.15.148:0/3053844113 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:44.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe2677fe640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2680cace0 0x7fe268147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26cf3a640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe268148810 0x7fe2681481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe267fff640 1 -- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2680cace0 msgr2=0x7fe268147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe267fff640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2680cace0 0x7fe268147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe267fff640 1 -- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe268148810 msgr2=0x7fe2681481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe267fff640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe268148810 0x7fe2681481a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe267fff640 1 -- 172.21.15.148:0/3053844113 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2681f28c0 con 0x7fe2500047d0 2023-10-30T13:23:44.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26cf3a640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe268148810 0x7fe2681481a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:23:44.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe267fff640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe2681582c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe258011660 tx=0x7fe258011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe264ff9640 1 -- 172.21.15.148:0/3053844113 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25800f9c0 con 0x7fe2500047d0 2023-10-30T13:23:44.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe264ff9640 1 -- 172.21.15.148:0/3053844113 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25800fb20 con 0x7fe2500047d0 2023-10-30T13:23:44.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2681f2ba0 con 0x7fe2500047d0 2023-10-30T13:23:44.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe264ff9640 1 -- 172.21.15.148:0/3053844113 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25800fe90 con 0x7fe2500047d0 2023-10-30T13:23:44.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2681f3160 con 0x7fe2500047d0 2023-10-30T13:23:44.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.469+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe268074250 con 0x7fe2500047d0 2023-10-30T13:23:44.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.473+0000 7fe264ff9640 1 -- 172.21.15.148:0/3053844113 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe258015460 con 0x7fe2500047d0 2023-10-30T13:23:44.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.473+0000 7fe264ff9640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2540664d0 0x7fe254068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:44.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.473+0000 7fe264ff9640 1 -- 172.21.15.148:0/3053844113 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fe258084a50 con 0x7fe2500047d0 2023-10-30T13:23:44.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.473+0000 7fe2677fe640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2540664d0 0x7fe254068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:44.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.473+0000 7fe2677fe640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2540664d0 0x7fe254068990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe25c001750 tx=0x7fe25c0011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:44.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.477+0000 7fe264ff9640 1 -- 172.21.15.148:0/3053844113 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe258051cd0 con 0x7fe2500047d0 2023-10-30T13:23:44.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.657+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe26806f900 con 0x7fe2500047d0 2023-10-30T13:23:44.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.657+0000 7fe264ff9640 1 -- 172.21.15.148:0/3053844113 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fe2580552f0 con 0x7fe2500047d0 2023-10-30T13:23:44.655 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:44.656 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:41.557180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:44.657 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 66 2023-10-30T13:23:44.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2540664d0 msgr2=0x7fe254068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2540664d0 0x7fe254068990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe25c001750 tx=0x7fe25c0011a0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 msgr2=0x7fe2681582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe2681582c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe258011660 tx=0x7fe258011b20 comp rx=0 tx=0).stop 2023-10-30T13:23:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 shutdown_connections 2023-10-30T13:23:44.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe2540664d0 0x7fe254068990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe268148810 0x7fe2681481a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2680cace0 0x7fe268147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 --2- 172.21.15.148:0/3053844113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2500047d0 0x7fe2681582c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:44.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 >> 172.21.15.148:0/3053844113 conn(0x7fe26806d770 msgr2=0x7fe2680c59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 shutdown_connections 2023-10-30T13:23:44.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:44.661+0000 7fe26e99d640 1 -- 172.21.15.148:0/3053844113 wait complete. 2023-10-30T13:23:44.676 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:23:46.677 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.821+0000 7f9b59a17640 1 Processor -- start 2023-10-30T13:23:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.821+0000 7f9b59a17640 1 -- start start 2023-10-30T13:23:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b59a17640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b5406f7a0 0x7f9b5406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b59a17640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b54066cf0 con 0x7f9b54071110 2023-10-30T13:23:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b59a17640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b540c70d0 con 0x7f9b540c6bd0 2023-10-30T13:23:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b59a17640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b540700c0 con 0x7f9b5406f7a0 2023-10-30T13:23:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b52ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b5406f7a0 0x7f9b5406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b537fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9b54071110 0x7f9b54158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34808/0 (socket says 172.21.15.148:34808) 2023-10-30T13:23:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b52ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b5406f7a0 0x7f9b5406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36724/0 (socket says 172.21.15.148:36724) 2023-10-30T13:23:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b537fe640 1 -- 172.21.15.148:0/2918445774 learned_addr learned my addr 172.21.15.148:0/2918445774 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b52ffd640 1 -- 172.21.15.148:0/2918445774 >> v1:172.21.15.150:6789/0 conn(0x7f9b540c6bd0 legacy=0x7f9b54069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b52ffd640 1 -- 172.21.15.148:0/2918445774 >> v1:172.21.15.148:6789/0 conn(0x7f9b54071110 legacy=0x7f9b54158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b51ffb640 1 -- 172.21.15.148:0/2918445774 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335040700 0 0) 0x7f9b54066cf0 con 0x7f9b54071110 2023-10-30T13:23:46.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.825+0000 7f9b52ffd640 1 -- 172.21.15.148:0/2918445774 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b540c7390 con 0x7f9b5406f7a0 2023-10-30T13:23:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b52ffd640 1 --2- 172.21.15.148:0/2918445774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b5406f7a0 0x7f9b5406fb80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9b440097c0 tx=0x7f9b4401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7456b70000f2e26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b51ffb640 1 -- 172.21.15.148:0/2918445774 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b440a8970 con 0x7f9b5406f7a0 2023-10-30T13:23:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b51ffb640 1 -- 172.21.15.148:0/2918445774 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b440a8ad0 con 0x7f9b5406f7a0 2023-10-30T13:23:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b51ffb640 1 -- 172.21.15.148:0/2918445774 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b440a8e60 con 0x7f9b5406f7a0 2023-10-30T13:23:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b59a17640 1 -- 172.21.15.148:0/2918445774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b5406f7a0 msgr2=0x7f9b5406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b59a17640 1 --2- 172.21.15.148:0/2918445774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b5406f7a0 0x7f9b5406fb80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9b440097c0 tx=0x7f9b4401ec70 comp rx=0 tx=0).stop 2023-10-30T13:23:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b59a17640 1 -- 172.21.15.148:0/2918445774 shutdown_connections 2023-10-30T13:23:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b59a17640 1 --2- 172.21.15.148:0/2918445774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b5406f7a0 0x7f9b5406fb80 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b59a17640 1 -- 172.21.15.148:0/2918445774 >> 172.21.15.148:0/2918445774 conn(0x7f9b540c7910 msgr2=0x7f9b540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b59a17640 1 -- 172.21.15.148:0/2918445774 shutdown_connections 2023-10-30T13:23:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b59a17640 1 -- 172.21.15.148:0/2918445774 wait complete. 2023-10-30T13:23:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b59a17640 1 Processor -- start 2023-10-30T13:23:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.829+0000 7f9b59a17640 1 -- start start 2023-10-30T13:23:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b59a17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b5406f7a0 0x7f9b540c0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b59a17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b54071110 0x7f9b540c0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b59a17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b540c6bd0 0x7f9b541e3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b59a17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b541e29f0 con 0x7f9b54071110 2023-10-30T13:23:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b59a17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b541e2cd0 con 0x7f9b540c6bd0 2023-10-30T13:23:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b59a17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b541e2050 con 0x7f9b5406f7a0 2023-10-30T13:23:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b527fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b54071110 0x7f9b540c0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b527fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b54071110 0x7f9b540c0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36730/0 (socket says 172.21.15.148:36730) 2023-10-30T13:23:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b527fc640 1 -- 172.21.15.148:0/177738366 learned_addr learned my addr 172.21.15.148:0/177738366 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b537fe640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b540c6bd0 0x7f9b541e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b52ffd640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b5406f7a0 0x7f9b540c0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b527fc640 1 -- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b5406f7a0 msgr2=0x7f9b540c0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b527fc640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b5406f7a0 0x7f9b540c0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b527fc640 1 -- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b540c6bd0 msgr2=0x7f9b541e3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b527fc640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b540c6bd0 0x7f9b541e3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b527fc640 1 -- 172.21.15.148:0/177738366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b44009420 con 0x7f9b54071110 2023-10-30T13:23:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b537fe640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b540c6bd0 0x7f9b541e3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b52ffd640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b5406f7a0 0x7f9b540c0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b527fc640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b54071110 0x7f9b540c0f70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c500 tx=0x7f9b4800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b3bfff640 1 -- 172.21.15.148:0/177738366 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b480052d0 con 0x7f9b54071110 2023-10-30T13:23:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b3bfff640 1 -- 172.21.15.148:0/177738366 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b48005430 con 0x7f9b54071110 2023-10-30T13:23:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b3bfff640 1 -- 172.21.15.148:0/177738366 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b48012830 con 0x7f9b54071110 2023-10-30T13:23:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b541e1d70 con 0x7f9b54071110 2023-10-30T13:23:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b541ef9e0 con 0x7f9b54071110 2023-10-30T13:23:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b541e2e80 con 0x7f9b54071110 2023-10-30T13:23:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.833+0000 7f9b3bfff640 1 -- 172.21.15.148:0/177738366 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b48012990 con 0x7f9b54071110 2023-10-30T13:23:46.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.837+0000 7f9b3bfff640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b400664d0 0x7f9b40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:46.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.837+0000 7f9b3bfff640 1 -- 172.21.15.148:0/177738366 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f9b48083cd0 con 0x7f9b54071110 2023-10-30T13:23:46.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.837+0000 7f9b52ffd640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b400664d0 0x7f9b40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:46.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.837+0000 7f9b52ffd640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b400664d0 0x7f9b40068990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9b440b2000 tx=0x7f9b44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:46.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:46.841+0000 7f9b3bfff640 1 -- 172.21.15.148:0/177738366 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b541e2e80 con 0x7f9b54071110 2023-10-30T13:23:47.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.021+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b541e2e80 con 0x7f9b54071110 2023-10-30T13:23:47.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.021+0000 7f9b3bfff640 1 -- 172.21.15.148:0/177738366 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f9b541e2e80 con 0x7f9b54071110 2023-10-30T13:23:47.018 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:47.018 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:41.557180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:47.019 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 66 2023-10-30T13:23:47.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b400664d0 msgr2=0x7f9b40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b400664d0 0x7f9b40068990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9b440b2000 tx=0x7f9b44008040 comp rx=0 tx=0).stop 2023-10-30T13:23:47.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b54071110 msgr2=0x7f9b540c0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:47.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b54071110 0x7f9b540c0f70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c500 tx=0x7f9b4800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 shutdown_connections 2023-10-30T13:23:47.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b400664d0 0x7f9b40068990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b54071110 0x7f9b540c0f70 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b540c6bd0 0x7f9b541e3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 --2- 172.21.15.148:0/177738366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b5406f7a0 0x7f9b540c0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:47.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 >> 172.21.15.148:0/177738366 conn(0x7f9b540c7910 msgr2=0x7f9b540c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:47.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 shutdown_connections 2023-10-30T13:23:47.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:47.025+0000 7f9b59a17640 1 -- 172.21.15.148:0/177738366 wait complete. 2023-10-30T13:23:47.039 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:23:49.040 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:49.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f93331c5640 1 Processor -- start 2023-10-30T13:23:49.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f93331c5640 1 -- start start 2023-10-30T13:23:49.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f93331c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f932c1538a0 0x7f932c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f93331c5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f932c0cf160 0x7f932c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f93331c5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c0cb800 con 0x7f932c0ca0e0 2023-10-30T13:23:49.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f93331c5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c15dc30 con 0x7f932c0cf160 2023-10-30T13:23:49.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f93331c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c0cb330 con 0x7f932c1538a0 2023-10-30T13:23:49.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f9331762640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f932c0cf160 0x7f932c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f9331762640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f932c0cf160 0x7f932c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36742/0 (socket says 172.21.15.148:36742) 2023-10-30T13:23:49.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f9331762640 1 -- 172.21.15.148:0/1340759592 learned_addr learned my addr 172.21.15.148:0/1340759592 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:49.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f932bfff640 1 --1- 172.21.15.148:0/1340759592 >> v1:172.21.15.148:6789/0 conn(0x7f932c0ca0e0 0x7f932c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34810/0 (socket says 172.21.15.148:34810) 2023-10-30T13:23:49.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f9330f61640 1 --2- 172.21.15.148:0/1340759592 >> v2:172.21.15.150:3300/0 conn(0x7f932c1538a0 0x7f932c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f9331762640 1 -- 172.21.15.148:0/1340759592 >> v2:172.21.15.150:3300/0 conn(0x7f932c1538a0 msgr2=0x7f932c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:49.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f9331762640 1 --2- 172.21.15.148:0/1340759592 >> v2:172.21.15.150:3300/0 conn(0x7f932c1538a0 0x7f932c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f9331762640 1 -- 172.21.15.148:0/1340759592 >> v1:172.21.15.148:6789/0 conn(0x7f932c0ca0e0 legacy=0x7f932c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.193+0000 7f9331762640 1 -- 172.21.15.148:0/1340759592 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c0cad70 con 0x7f932c0cf160 2023-10-30T13:23:49.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f9331762640 1 --2- 172.21.15.148:0/1340759592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0cf160 0x7f932c0cf590 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f932c0cfd10 tx=0x7f932401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b49d5468b7d144c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:49.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f932b7fe640 1 -- 172.21.15.148:0/1340759592 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93240af400 con 0x7f932c0cf160 2023-10-30T13:23:49.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f932b7fe640 1 -- 172.21.15.148:0/1340759592 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93240ab050 con 0x7f932c0cf160 2023-10-30T13:23:49.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f932b7fe640 1 -- 172.21.15.148:0/1340759592 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93240af780 con 0x7f932c0cf160 2023-10-30T13:23:49.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f93331c5640 1 -- 172.21.15.148:0/1340759592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0cf160 msgr2=0x7f932c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f93331c5640 1 --2- 172.21.15.148:0/1340759592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0cf160 0x7f932c0cf590 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f932c0cfd10 tx=0x7f932401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f93331c5640 1 -- 172.21.15.148:0/1340759592 shutdown_connections 2023-10-30T13:23:49.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f93331c5640 1 --2- 172.21.15.148:0/1340759592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0cf160 0x7f932c0cf590 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f93331c5640 1 --2- 172.21.15.148:0/1340759592 >> v2:172.21.15.150:3300/0 conn(0x7f932c1538a0 0x7f932c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f93331c5640 1 -- 172.21.15.148:0/1340759592 >> 172.21.15.148:0/1340759592 conn(0x7f932c06bfe0 msgr2=0x7f932c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:49.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f93331c5640 1 -- 172.21.15.148:0/1340759592 shutdown_connections 2023-10-30T13:23:49.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.197+0000 7f93331c5640 1 -- 172.21.15.148:0/1340759592 wait complete. 2023-10-30T13:23:49.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 Processor -- start 2023-10-30T13:23:49.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 -- start start 2023-10-30T13:23:49.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0ca0e0 0x7f932c1480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f932c0cf160 0x7f932c158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c1538a0 0x7f932c1f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c1e5330 con 0x7f932c0ca0e0 2023-10-30T13:23:49.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0b4f30 con 0x7f932c1538a0 2023-10-30T13:23:49.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c1e4d70 con 0x7f932c0cf160 2023-10-30T13:23:49.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9330f61640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0ca0e0 0x7f932c1480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9330f61640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0ca0e0 0x7f932c1480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36748/0 (socket says 172.21.15.148:36748) 2023-10-30T13:23:49.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9330f61640 1 -- 172.21.15.148:0/4124815273 learned_addr learned my addr 172.21.15.148:0/4124815273 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:49.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9331762640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c1538a0 0x7f932c1f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f932bfff640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f932c0cf160 0x7f932c158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9330f61640 1 -- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f932c0cf160 msgr2=0x7f932c158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9330f61640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f932c0cf160 0x7f932c158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9330f61640 1 -- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c1538a0 msgr2=0x7f932c1f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9330f61640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c1538a0 0x7f932c1f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9330f61640 1 -- 172.21.15.148:0/4124815273 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9324009b30 con 0x7f932c0ca0e0 2023-10-30T13:23:49.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f9330f61640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0ca0e0 0x7f932c1480f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f932000d800 tx=0x7f932000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:49.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93297fa640 1 -- 172.21.15.148:0/4124815273 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932000b6c0 con 0x7f932c0ca0e0 2023-10-30T13:23:49.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93297fa640 1 -- 172.21.15.148:0/4124815273 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932000b820 con 0x7f932c0ca0e0 2023-10-30T13:23:49.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93297fa640 1 -- 172.21.15.148:0/4124815273 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932000bb90 con 0x7f932c0ca0e0 2023-10-30T13:23:49.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c1e5050 con 0x7f932c0ca0e0 2023-10-30T13:23:49.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c0b4bc0 con 0x7f932c0ca0e0 2023-10-30T13:23:49.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.201+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932c0b4d80 con 0x7f932c0ca0e0 2023-10-30T13:23:49.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.205+0000 7f93297fa640 1 -- 172.21.15.148:0/4124815273 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f932000bcf0 con 0x7f932c0ca0e0 2023-10-30T13:23:49.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.205+0000 7f93297fa640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93100664a0 0x7f9310068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:49.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.205+0000 7f93297fa640 1 -- 172.21.15.148:0/4124815273 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f93200847c0 con 0x7f932c0ca0e0 2023-10-30T13:23:49.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.205+0000 7f932bfff640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93100664a0 0x7f9310068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:49.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.205+0000 7f932bfff640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93100664a0 0x7f9310068960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f932c1e45c0 tx=0x7f9318007c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:49.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.209+0000 7f93297fa640 1 -- 172.21.15.148:0/4124815273 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9320051870 con 0x7f932c0ca0e0 2023-10-30T13:23:49.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f932c0b2a50 con 0x7f932c0ca0e0 2023-10-30T13:23:49.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93297fa640 1 -- 172.21.15.148:0/4124815273 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f9320054e90 con 0x7f932c0ca0e0 2023-10-30T13:23:49.385 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:49.385 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:46.638009+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:clientreplay","state_seq":42,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:49.386 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 67 2023-10-30T13:23:49.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93100664a0 msgr2=0x7f9310068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93100664a0 0x7f9310068960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f932c1e45c0 tx=0x7f9318007c90 comp rx=0 tx=0).stop 2023-10-30T13:23:49.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0ca0e0 msgr2=0x7f932c1480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:49.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0ca0e0 0x7f932c1480f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f932000d800 tx=0x7f932000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 shutdown_connections 2023-10-30T13:23:49.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93100664a0 0x7f9310068960 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f932c1538a0 0x7f932c1f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f932c0cf160 0x7f932c158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 --2- 172.21.15.148:0/4124815273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f932c0ca0e0 0x7f932c1480f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:49.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 >> 172.21.15.148:0/4124815273 conn(0x7f932c06bfe0 msgr2=0x7f932c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:49.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 shutdown_connections 2023-10-30T13:23:49.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:49.389+0000 7f93331c5640 1 -- 172.21.15.148:0/4124815273 wait complete. 2023-10-30T13:23:49.407 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:23:51.409 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:51.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdefc4640 1 Processor -- start 2023-10-30T13:23:51.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdefc4640 1 -- start start 2023-10-30T13:23:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdefc4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4bd80c05d0 0x7f4bd813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:51.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdefc4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4bd80c6f60 0x7f4bd813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:51.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdefc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd80c0a50 con 0x7f4bd80c6b80 2023-10-30T13:23:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdefc4640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd806f6e0 con 0x7f4bd80c6f60 2023-10-30T13:23:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdefc4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd806f9c0 con 0x7f4bd80c05d0 2023-10-30T13:23:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdd561640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4bd80c6f60 0x7f4bd813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:51.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdcd60640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4bd80c6b80 0x7f4bd80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34824/0 (socket says 172.21.15.148:34824) 2023-10-30T13:23:51.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bdcd60640 1 -- 172.21.15.148:0/1264199794 learned_addr learned my addr 172.21.15.148:0/1264199794 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:51.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.561+0000 7f4bcffff640 1 --2- 172.21.15.148:0/1264199794 >> v2:172.21.15.150:3300/0 conn(0x7f4bd80c05d0 0x7f4bd813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:51.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdd561640 1 -- 172.21.15.148:0/1264199794 >> v2:172.21.15.150:3300/0 conn(0x7f4bd80c05d0 msgr2=0x7f4bd813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:51.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdd561640 1 --2- 172.21.15.148:0/1264199794 >> v2:172.21.15.150:3300/0 conn(0x7f4bd80c05d0 0x7f4bd813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdd561640 1 -- 172.21.15.148:0/1264199794 >> v1:172.21.15.148:6789/0 conn(0x7f4bd80c6b80 legacy=0x7f4bd80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:51.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdd561640 1 -- 172.21.15.148:0/1264199794 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd806fca0 con 0x7f4bd80c6f60 2023-10-30T13:23:51.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bcffff640 1 --2- 172.21.15.148:0/1264199794 >> v2:172.21.15.150:3300/0 conn(0x7f4bd80c05d0 0x7f4bd813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:51.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bcf7fe640 1 -- 172.21.15.148:0/1264199794 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167681094 0 0) 0x7f4bd80c0a50 con 0x7f4bd80c6b80 2023-10-30T13:23:51.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdd561640 1 --2- 172.21.15.148:0/1264199794 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6f60 0x7f4bd813f2d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4bd806c7e0 tx=0x7f4bd001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd7d91cd481694e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:51.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bcf7fe640 1 -- 172.21.15.148:0/1264199794 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00a58e0 con 0x7f4bd80c6f60 2023-10-30T13:23:51.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bcf7fe640 1 -- 172.21.15.148:0/1264199794 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd00a5a40 con 0x7f4bd80c6f60 2023-10-30T13:23:51.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bcf7fe640 1 -- 172.21.15.148:0/1264199794 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00a5dd0 con 0x7f4bd80c6f60 2023-10-30T13:23:51.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdefc4640 1 -- 172.21.15.148:0/1264199794 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6f60 msgr2=0x7f4bd813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:51.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdefc4640 1 --2- 172.21.15.148:0/1264199794 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6f60 0x7f4bd813f2d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4bd806c7e0 tx=0x7f4bd001c910 comp rx=0 tx=0).stop 2023-10-30T13:23:51.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdefc4640 1 -- 172.21.15.148:0/1264199794 shutdown_connections 2023-10-30T13:23:51.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdefc4640 1 --2- 172.21.15.148:0/1264199794 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6f60 0x7f4bd813f2d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdefc4640 1 --2- 172.21.15.148:0/1264199794 >> v2:172.21.15.150:3300/0 conn(0x7f4bd80c05d0 0x7f4bd813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.565+0000 7f4bdefc4640 1 -- 172.21.15.148:0/1264199794 >> 172.21.15.148:0/1264199794 conn(0x7f4bd80c7910 msgr2=0x7f4bd80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:51.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 -- 172.21.15.148:0/1264199794 shutdown_connections 2023-10-30T13:23:51.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 -- 172.21.15.148:0/1264199794 wait complete. 2023-10-30T13:23:51.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 Processor -- start 2023-10-30T13:23:51.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 -- start start 2023-10-30T13:23:51.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bd80c05d0 0x7f4bd813a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:51.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6b80 0x7f4bd8157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:51.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4bd80c6f60 0x7f4bd8138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:51.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd813dfa0 con 0x7f4bd80c6b80 2023-10-30T13:23:51.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd813e390 con 0x7f4bd80c05d0 2023-10-30T13:23:51.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdefc4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd813e650 con 0x7f4bd80c6f60 2023-10-30T13:23:51.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bcffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6b80 0x7f4bd8157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:51.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bcffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6b80 0x7f4bd8157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36770/0 (socket says 172.21.15.148:36770) 2023-10-30T13:23:51.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bcffff640 1 -- 172.21.15.148:0/2952848820 learned_addr learned my addr 172.21.15.148:0/2952848820 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:51.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdcd60640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bd80c05d0 0x7f4bd813a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:51.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdd561640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4bd80c6f60 0x7f4bd8138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:51.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bcffff640 1 -- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4bd80c6f60 msgr2=0x7f4bd8138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:51.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bcffff640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4bd80c6f60 0x7f4bd8138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bcffff640 1 -- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bd80c05d0 msgr2=0x7f4bd813a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:51.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bcffff640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bd80c05d0 0x7f4bd813a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bcffff640 1 -- 172.21.15.148:0/2952848820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd0009a70 con 0x7f4bd80c6b80 2023-10-30T13:23:51.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bdcd60640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bd80c05d0 0x7f4bd813a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:51.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.569+0000 7f4bcffff640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6b80 0x7f4bd8157790 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800e810 tx=0x7f4bc800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:51.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bcd7fa640 1 -- 172.21.15.148:0/2952848820 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800c6d0 con 0x7f4bd80c6b80 2023-10-30T13:23:51.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bcd7fa640 1 -- 172.21.15.148:0/2952848820 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc800c830 con 0x7f4bd80c6b80 2023-10-30T13:23:51.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bcd7fa640 1 -- 172.21.15.148:0/2952848820 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800cbe0 con 0x7f4bd80c6b80 2023-10-30T13:23:51.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd813c8c0 con 0x7f4bd80c6b80 2023-10-30T13:23:51.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd814fdd0 con 0x7f4bd80c6b80 2023-10-30T13:23:51.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd804e170 con 0x7f4bd80c6b80 2023-10-30T13:23:51.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bcd7fa640 1 -- 172.21.15.148:0/2952848820 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4bc800cd40 con 0x7f4bd80c6b80 2023-10-30T13:23:51.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bcd7fa640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664d0 0x7f4bc4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:51.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bcd7fa640 1 -- 172.21.15.148:0/2952848820 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f4bc8083e40 con 0x7f4bd80c6b80 2023-10-30T13:23:51.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.573+0000 7f4bdcd60640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664d0 0x7f4bc4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:51.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.577+0000 7f4bdcd60640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664d0 0x7f4bc4068990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4007fd0 tx=0x7f4bd4007c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:51.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.577+0000 7f4bcd7fa640 1 -- 172.21.15.148:0/2952848820 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4bc80510b0 con 0x7f4bd80c6b80 2023-10-30T13:23:51.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.761+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bd806af80 con 0x7f4bd80c6b80 2023-10-30T13:23:51.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.761+0000 7f4bcd7fa640 1 -- 172.21.15.148:0/2952848820 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f4bc80546d0 con 0x7f4bd80c6b80 2023-10-30T13:23:51.758 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:51.759 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:46.638009+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:clientreplay","state_seq":42,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:51.759 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 67 2023-10-30T13:23:51.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664d0 msgr2=0x7f4bc4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:51.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664d0 0x7f4bc4068990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4007fd0 tx=0x7f4bd4007c20 comp rx=0 tx=0).stop 2023-10-30T13:23:51.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6b80 msgr2=0x7f4bd8157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:51.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6b80 0x7f4bd8157790 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800e810 tx=0x7f4bc800ecd0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 shutdown_connections 2023-10-30T13:23:51.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664d0 0x7f4bc4068990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bd80c6b80 0x7f4bd8157790 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4bd80c6f60 0x7f4bd8138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 --2- 172.21.15.148:0/2952848820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bd80c05d0 0x7f4bd813a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:51.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 >> 172.21.15.148:0/2952848820 conn(0x7f4bd80c7910 msgr2=0x7f4bd80bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:51.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 shutdown_connections 2023-10-30T13:23:51.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:51.765+0000 7f4bdefc4640 1 -- 172.21.15.148:0/2952848820 wait complete. 2023-10-30T13:23:51.779 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:23:53.780 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:53.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb5adc640 1 Processor -- start 2023-10-30T13:23:53.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb5adc640 1 -- start start 2023-10-30T13:23:53.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb5adc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fceb00717b0 0x7fceb015ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:53.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb5adc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb0071b90 0x7fceb01513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:53.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb5adc640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb015b060 con 0x7fceb00c6290 2023-10-30T13:23:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb5adc640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb0157530 con 0x7fceb00717b0 2023-10-30T13:23:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb5adc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb00c4ab0 con 0x7fceb0071b90 2023-10-30T13:23:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb4ada640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fceb00717b0 0x7fceb015ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb52db640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fceb00c6290 0x7fceb0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60952/0 (socket says 172.21.15.148:60952) 2023-10-30T13:23:53.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb4ada640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fceb00717b0 0x7fceb015ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42134/0 (socket says 172.21.15.148:42134) 2023-10-30T13:23:53.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb52db640 1 -- 172.21.15.148:0/3730806 learned_addr learned my addr 172.21.15.148:0/3730806 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:23:53.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceaffff640 1 --2- 172.21.15.148:0/3730806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb0071b90 0x7fceb01513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:53.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb4ada640 1 -- 172.21.15.148:0/3730806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb0071b90 msgr2=0x7fceb01513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:53.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb4ada640 1 --2- 172.21.15.148:0/3730806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb0071b90 0x7fceb01513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:53.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb4ada640 1 -- 172.21.15.148:0/3730806 >> v1:172.21.15.148:6789/0 conn(0x7fceb00c6290 legacy=0x7fceb0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:53.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceb4ada640 1 -- 172.21.15.148:0/3730806 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb00c4d90 con 0x7fceb00717b0 2023-10-30T13:23:53.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.925+0000 7fceaf7fe640 1 -- 172.21.15.148:0/3730806 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106227010 0 0) 0x7fceb015b060 con 0x7fceb00c6290 2023-10-30T13:23:53.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceb4ada640 1 --2- 172.21.15.148:0/3730806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00717b0 0x7fceb015ab20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcea40097a0 tx=0x7fcea401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9f1d7d5c19222f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:53.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceaf7fe640 1 -- 172.21.15.148:0/3730806 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea40a8950 con 0x7fceb00717b0 2023-10-30T13:23:53.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceaf7fe640 1 -- 172.21.15.148:0/3730806 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea40a8ab0 con 0x7fceb00717b0 2023-10-30T13:23:53.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceaf7fe640 1 -- 172.21.15.148:0/3730806 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea40a8e40 con 0x7fceb00717b0 2023-10-30T13:23:53.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceb5adc640 1 -- 172.21.15.148:0/3730806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00717b0 msgr2=0x7fceb015ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:53.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceb5adc640 1 --2- 172.21.15.148:0/3730806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00717b0 0x7fceb015ab20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fcea40097a0 tx=0x7fcea401ec50 comp rx=0 tx=0).stop 2023-10-30T13:23:53.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceb5adc640 1 -- 172.21.15.148:0/3730806 shutdown_connections 2023-10-30T13:23:53.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceb5adc640 1 --2- 172.21.15.148:0/3730806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb0071b90 0x7fceb01513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:53.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceb5adc640 1 --2- 172.21.15.148:0/3730806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00717b0 0x7fceb015ab20 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:53.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceb5adc640 1 -- 172.21.15.148:0/3730806 >> 172.21.15.148:0/3730806 conn(0x7fceb00c75d0 msgr2=0x7fceb00c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:53.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceb5adc640 1 -- 172.21.15.148:0/3730806 shutdown_connections 2023-10-30T13:23:53.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.929+0000 7fceb5adc640 1 -- 172.21.15.148:0/3730806 wait complete. 2023-10-30T13:23:53.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 Processor -- start 2023-10-30T13:23:53.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 -- start start 2023-10-30T13:23:53.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb00717b0 0x7fceb014e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:53.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceb0071b90 0x7fceb00bc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00c6290 0x7fceb01f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb014d360 con 0x7fceb00c6290 2023-10-30T13:23:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb006d8a0 con 0x7fceb0071b90 2023-10-30T13:23:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb006db80 con 0x7fceb00717b0 2023-10-30T13:23:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb52db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00c6290 0x7fceb01f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:53.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb52db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00c6290 0x7fceb01f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42138/0 (socket says 172.21.15.148:42138) 2023-10-30T13:23:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb52db640 1 -- 172.21.15.148:0/2392451067 learned_addr learned my addr 172.21.15.148:0/2392451067 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceaffff640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceb0071b90 0x7fceb00bc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb4ada640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb00717b0 0x7fceb014e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb52db640 1 -- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb00717b0 msgr2=0x7fceb014e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:53.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb52db640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb00717b0 0x7fceb014e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb52db640 1 -- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceb0071b90 msgr2=0x7fceb00bc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb52db640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceb0071b90 0x7fceb00bc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb52db640 1 -- 172.21.15.148:0/2392451067 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40093b0 con 0x7fceb00c6290 2023-10-30T13:23:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb52db640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00c6290 0x7fceb01f1d80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcea8001570 tx=0x7fcea800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:53.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fcead7fa640 1 -- 172.21.15.148:0/2392451067 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea800a640 con 0x7fceb00c6290 2023-10-30T13:23:53.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fcead7fa640 1 -- 172.21.15.148:0/2392451067 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea800a7c0 con 0x7fceb00c6290 2023-10-30T13:23:53.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fcead7fa640 1 -- 172.21.15.148:0/2392451067 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea800ab70 con 0x7fceb00c6290 2023-10-30T13:23:53.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb014d5b0 con 0x7fceb00c6290 2023-10-30T13:23:53.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb014db70 con 0x7fceb00c6290 2023-10-30T13:23:53.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.933+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb014dd90 con 0x7fceb00c6290 2023-10-30T13:23:53.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.937+0000 7fcead7fa640 1 -- 172.21.15.148:0/2392451067 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcea800acd0 con 0x7fceb00c6290 2023-10-30T13:23:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.937+0000 7fcead7fa640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce940664d0 0x7fce94068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.937+0000 7fcead7fa640 1 -- 172.21.15.148:0/2392451067 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fcea8083580 con 0x7fceb00c6290 2023-10-30T13:23:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.937+0000 7fceb4ada640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce940664d0 0x7fce94068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:53.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.937+0000 7fceb4ada640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce940664d0 0x7fce94068990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcea40b2000 tx=0x7fcea4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:53.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:53.941+0000 7fcead7fa640 1 -- 172.21.15.148:0/2392451067 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcea8050810 con 0x7fceb00c6290 2023-10-30T13:23:54.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.125+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fceb004e170 con 0x7fceb00c6290 2023-10-30T13:23:54.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.125+0000 7fcead7fa640 1 -- 172.21.15.148:0/2392451067 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fcea8053e30 con 0x7fceb00c6290 2023-10-30T13:23:54.122 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:54.122 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:54.122 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 68 2023-10-30T13:23:54.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce940664d0 msgr2=0x7fce94068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:54.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce940664d0 0x7fce94068990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcea40b2000 tx=0x7fcea4008040 comp rx=0 tx=0).stop 2023-10-30T13:23:54.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00c6290 msgr2=0x7fceb01f1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:54.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00c6290 0x7fceb01f1d80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcea8001570 tx=0x7fcea800cca0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 shutdown_connections 2023-10-30T13:23:54.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce940664d0 0x7fce94068990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fceb00c6290 0x7fceb01f1d80 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fceb0071b90 0x7fceb00bc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 --2- 172.21.15.148:0/2392451067 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fceb00717b0 0x7fceb014e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 >> 172.21.15.148:0/2392451067 conn(0x7fceb00c75d0 msgr2=0x7fceb0151190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:54.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 shutdown_connections 2023-10-30T13:23:54.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.129+0000 7fceb5adc640 1 -- 172.21.15.148:0/2392451067 wait complete. 2023-10-30T13:23:54.143 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5694 2023-10-30T13:23:54.143 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2023-10-30T13:23:54.143 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:23:54.144 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:23:54.144 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:23:54.148 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:23:54.148 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:54.233 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:23:54.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.293+0000 7f4ea1301640 1 Processor -- start 2023-10-30T13:23:54.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.293+0000 7f4ea1301640 1 -- start start 2023-10-30T13:23:54.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.293+0000 7f4ea1301640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c06e470 0x7f4e9c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:54.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.293+0000 7f4ea1301640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e9c0c8490 0x7f4e9c0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:54.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.293+0000 7f4ea1301640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9c06afa0 con 0x7f4e9c153d70 2023-10-30T13:23:54.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.293+0000 7f4ea1301640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9c06d780 con 0x7f4e9c0c8490 2023-10-30T13:23:54.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.293+0000 7f4ea1301640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c154150 con 0x7f4e9c06e470 2023-10-30T13:23:54.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e927fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c06e470 0x7f4e9c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:54.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e9b7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e9c0c8490 0x7f4e9c0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:54.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e927fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c06e470 0x7f4e9c06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:45796/0 (socket says 172.21.15.148:45796) 2023-10-30T13:23:54.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e927fc640 1 -- 172.21.15.148:0/2062847994 learned_addr learned my addr 172.21.15.148:0/2062847994 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:54.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e9a7fc640 1 -- 172.21.15.148:0/2062847994 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827282907 0 0) 0x7f4e9c06afa0 con 0x7f4e9c153d70 2023-10-30T13:23:54.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e9a7fc640 1 -- 172.21.15.148:0/2062847994 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e88003530 con 0x7f4e9c153d70 2023-10-30T13:23:54.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e927fc640 1 -- 172.21.15.148:0/2062847994 >> v2:172.21.15.150:3300/0 conn(0x7f4e9c0c8490 msgr2=0x7f4e9c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:54.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e927fc640 1 --2- 172.21.15.148:0/2062847994 >> v2:172.21.15.150:3300/0 conn(0x7f4e9c0c8490 0x7f4e9c0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e927fc640 1 -- 172.21.15.148:0/2062847994 >> v1:172.21.15.150:6789/0 conn(0x7f4e9c153d70 legacy=0x7f4e9c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:54.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e927fc640 1 -- 172.21.15.148:0/2062847994 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c06e030 con 0x7f4e9c06e470 2023-10-30T13:23:54.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e9b7fe640 1 --2- 172.21.15.148:0/2062847994 >> v2:172.21.15.150:3300/0 conn(0x7f4e9c0c8490 0x7f4e9c0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:23:54.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.297+0000 7f4e927fc640 1 --2- 172.21.15.148:0/2062847994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c06e470 0x7f4e9c06ca80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009e50 tx=0x7f4e8401c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9186caba0ab770e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:54.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4e9a7fc640 1 -- 172.21.15.148:0/2062847994 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e840a58a0 con 0x7f4e9c06e470 2023-10-30T13:23:54.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4e9a7fc640 1 -- 172.21.15.148:0/2062847994 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e840a5a00 con 0x7f4e9c06e470 2023-10-30T13:23:54.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4e9a7fc640 1 -- 172.21.15.148:0/2062847994 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e840a5d90 con 0x7f4e9c06e470 2023-10-30T13:23:54.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 -- 172.21.15.148:0/2062847994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c06e470 msgr2=0x7f4e9c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:54.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 --2- 172.21.15.148:0/2062847994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c06e470 0x7f4e9c06ca80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009e50 tx=0x7f4e8401c9e0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 -- 172.21.15.148:0/2062847994 shutdown_connections 2023-10-30T13:23:54.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 --2- 172.21.15.148:0/2062847994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c06e470 0x7f4e9c06ca80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 --2- 172.21.15.148:0/2062847994 >> v2:172.21.15.150:3300/0 conn(0x7f4e9c0c8490 0x7f4e9c0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 -- 172.21.15.148:0/2062847994 >> 172.21.15.148:0/2062847994 conn(0x7f4e9c071ea0 msgr2=0x7f4e9c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:54.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 -- 172.21.15.148:0/2062847994 shutdown_connections 2023-10-30T13:23:54.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 -- 172.21.15.148:0/2062847994 wait complete. 2023-10-30T13:23:54.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 Processor -- start 2023-10-30T13:23:54.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 -- start start 2023-10-30T13:23:54.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e9c06e470 0x7f4e9c1f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:54.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c0c8490 0x7f4e9c154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:54.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e9c153d70 0x7f4e9c149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:54.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c1475b0 con 0x7f4e9c153d70 2023-10-30T13:23:54.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c146ff0 con 0x7f4e9c06e470 2023-10-30T13:23:54.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.301+0000 7f4ea1301640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c1472d0 con 0x7f4e9c0c8490 2023-10-30T13:23:54.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e9c153d70 0x7f4e9c149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:54.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e9c153d70 0x7f4e9c149660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42144/0 (socket says 172.21.15.148:42144) 2023-10-30T13:23:54.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9b7fe640 1 -- 172.21.15.148:0/1887492114 learned_addr learned my addr 172.21.15.148:0/1887492114 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:54.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e927fc640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c0c8490 0x7f4e9c154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:54.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9affd640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e9c06e470 0x7f4e9c1f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:54.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9b7fe640 1 -- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c0c8490 msgr2=0x7f4e9c154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:54.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9b7fe640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c0c8490 0x7f4e9c154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9b7fe640 1 -- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e9c06e470 msgr2=0x7f4e9c1f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:54.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9b7fe640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e9c06e470 0x7f4e9c1f3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9b7fe640 1 -- 172.21.15.148:0/1887492114 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e94009420 con 0x7f4e9c153d70 2023-10-30T13:23:54.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9affd640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e9c06e470 0x7f4e9c1f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:54.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e9b7fe640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e9c153d70 0x7f4e9c149660 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400e950 tx=0x7f4e9400ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:54.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e93fff640 1 -- 172.21.15.148:0/1887492114 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94009720 con 0x7f4e9c153d70 2023-10-30T13:23:54.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e93fff640 1 -- 172.21.15.148:0/1887492114 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9400ca00 con 0x7f4e9c153d70 2023-10-30T13:23:54.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e84009b30 con 0x7f4e9c153d70 2023-10-30T13:23:54.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e93fff640 1 -- 172.21.15.148:0/1887492114 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400cd90 con 0x7f4e9c153d70 2023-10-30T13:23:54.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9c147c80 con 0x7f4e9c153d70 2023-10-30T13:23:54.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9c04e170 con 0x7f4e9c153d70 2023-10-30T13:23:54.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.305+0000 7f4e93fff640 1 -- 172.21.15.148:0/1887492114 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e94013470 con 0x7f4e9c153d70 2023-10-30T13:23:54.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.309+0000 7f4e93fff640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e7c0664a0 0x7f4e7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:54.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.309+0000 7f4e93fff640 1 -- 172.21.15.148:0/1887492114 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f4e94083c80 con 0x7f4e9c153d70 2023-10-30T13:23:54.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.309+0000 7f4e9affd640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e7c0664a0 0x7f4e7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:54.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.309+0000 7f4e9affd640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e7c0664a0 0x7f4e7c068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c001740 tx=0x7f4e8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:54.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.309+0000 7f4e93fff640 1 -- 172.21.15.148:0/1887492114 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e94050f00 con 0x7f4e9c153d70 2023-10-30T13:23:54.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.497+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e9c150bf0 con 0x7f4e9c153d70 2023-10-30T13:23:54.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.497+0000 7f4e93fff640 1 -- 172.21.15.148:0/1887492114 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f4e94054520 con 0x7f4e9c153d70 2023-10-30T13:23:54.495 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:54.495 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:54.496 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 68 2023-10-30T13:23:54.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e7c0664a0 msgr2=0x7f4e7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:54.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e7c0664a0 0x7f4e7c068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c001740 tx=0x7f4e8c009040 comp rx=0 tx=0).stop 2023-10-30T13:23:54.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e9c153d70 msgr2=0x7f4e9c149660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:54.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e9c153d70 0x7f4e9c149660 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400e950 tx=0x7f4e9400ee10 comp rx=0 tx=0).stop 2023-10-30T13:23:54.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 shutdown_connections 2023-10-30T13:23:54.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e7c0664a0 0x7f4e7c068960 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e9c153d70 0x7f4e9c149660 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e9c0c8490 0x7f4e9c154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 --2- 172.21.15.148:0/1887492114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e9c06e470 0x7f4e9c1f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:54.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 >> 172.21.15.148:0/1887492114 conn(0x7f4e9c071ea0 msgr2=0x7f4e9c06ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:54.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 shutdown_connections 2023-10-30T13:23:54.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:54.501+0000 7f4ea1301640 1 -- 172.21.15.148:0/1887492114 wait complete. 2023-10-30T13:23:54.515 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:23:55.437 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:55.441 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:56.515 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:56.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b7e13e640 1 Processor -- start 2023-10-30T13:23:56.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b7e13e640 1 -- start start 2023-10-30T13:23:56.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b7e13e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b7806f7a0 0x7f9b7806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:56.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b7e13e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b78071110 0x7f9b78158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:56.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b7e13e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78066cf0 con 0x7f9b780c6bd0 2023-10-30T13:23:56.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b7e13e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b780700c0 con 0x7f9b7806f7a0 2023-10-30T13:23:56.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b7e13e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b780c70d0 con 0x7f9b78071110 2023-10-30T13:23:56.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b777fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b7806f7a0 0x7f9b7806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:56.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b777fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9b7806f7a0 0x7f9b7806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42156/0 (socket says 172.21.15.148:42156) 2023-10-30T13:23:56.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b777fe640 1 -- 172.21.15.148:0/741541737 learned_addr learned my addr 172.21.15.148:0/741541737 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:56.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b76ffd640 1 --1- 172.21.15.148:0/741541737 >> v1:172.21.15.148:6789/0 conn(0x7f9b780c6bd0 0x7f9b78069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60954/0 (socket says 172.21.15.148:60954) 2023-10-30T13:23:56.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b77fff640 1 --2- 172.21.15.148:0/741541737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b78071110 0x7f9b78158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:56.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b777fe640 1 -- 172.21.15.148:0/741541737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b78071110 msgr2=0x7f9b78158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:56.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b777fe640 1 --2- 172.21.15.148:0/741541737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b78071110 0x7f9b78158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b777fe640 1 -- 172.21.15.148:0/741541737 >> v1:172.21.15.148:6789/0 conn(0x7f9b780c6bd0 legacy=0x7f9b78069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:56.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.669+0000 7f9b777fe640 1 -- 172.21.15.148:0/741541737 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b780c7390 con 0x7f9b7806f7a0 2023-10-30T13:23:56.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b777fe640 1 --2- 172.21.15.148:0/741541737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b7806f7a0 0x7f9b7806fb80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009870 tx=0x7f9b6801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd5e7c3156f6fdd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:56.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b767fc640 1 -- 172.21.15.148:0/741541737 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b680a8a30 con 0x7f9b7806f7a0 2023-10-30T13:23:56.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b767fc640 1 -- 172.21.15.148:0/741541737 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b680a8b90 con 0x7f9b7806f7a0 2023-10-30T13:23:56.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b767fc640 1 -- 172.21.15.148:0/741541737 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b680b0440 con 0x7f9b7806f7a0 2023-10-30T13:23:56.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b7e13e640 1 -- 172.21.15.148:0/741541737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b7806f7a0 msgr2=0x7f9b7806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:56.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b7e13e640 1 --2- 172.21.15.148:0/741541737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b7806f7a0 0x7f9b7806fb80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009870 tx=0x7f9b6801ed30 comp rx=0 tx=0).stop 2023-10-30T13:23:56.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b7e13e640 1 -- 172.21.15.148:0/741541737 shutdown_connections 2023-10-30T13:23:56.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b7e13e640 1 --2- 172.21.15.148:0/741541737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b78071110 0x7f9b78158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b7e13e640 1 --2- 172.21.15.148:0/741541737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b7806f7a0 0x7f9b7806fb80 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b7e13e640 1 -- 172.21.15.148:0/741541737 >> 172.21.15.148:0/741541737 conn(0x7f9b780c7910 msgr2=0x7f9b780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:56.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b7e13e640 1 -- 172.21.15.148:0/741541737 shutdown_connections 2023-10-30T13:23:56.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.673+0000 7f9b7e13e640 1 -- 172.21.15.148:0/741541737 wait complete. 2023-10-30T13:23:56.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 Processor -- start 2023-10-30T13:23:56.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 -- start start 2023-10-30T13:23:56.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b7806f7a0 0x7f9b781520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:56.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b78071110 0x7f9b780c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:56.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b780c6bd0 0x7f9b781eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:56.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b781f1eb0 con 0x7f9b78071110 2023-10-30T13:23:56.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b781f2d10 con 0x7f9b7806f7a0 2023-10-30T13:23:56.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78152630 con 0x7f9b780c6bd0 2023-10-30T13:23:56.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b76ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b78071110 0x7f9b780c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:56.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b76ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b78071110 0x7f9b780c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42168/0 (socket says 172.21.15.148:42168) 2023-10-30T13:23:56.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b76ffd640 1 -- 172.21.15.148:0/2805691026 learned_addr learned my addr 172.21.15.148:0/2805691026 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:23:56.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b77fff640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b780c6bd0 0x7f9b781eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:56.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b777fe640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b7806f7a0 0x7f9b781520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:56.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b76ffd640 1 -- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b780c6bd0 msgr2=0x7f9b781eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:56.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b76ffd640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b780c6bd0 0x7f9b781eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b76ffd640 1 -- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b7806f7a0 msgr2=0x7f9b781520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:56.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b76ffd640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b7806f7a0 0x7f9b781520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b76ffd640 1 -- 172.21.15.148:0/2805691026 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b680094d0 con 0x7f9b78071110 2023-10-30T13:23:56.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b76ffd640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b78071110 0x7f9b780c5760 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c00c530 tx=0x7f9b6c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:56.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b5ffff640 1 -- 172.21.15.148:0/2805691026 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c007500 con 0x7f9b78071110 2023-10-30T13:23:56.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b5ffff640 1 -- 172.21.15.148:0/2805691026 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c007660 con 0x7f9b78071110 2023-10-30T13:23:56.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b781f2100 con 0x7f9b78071110 2023-10-30T13:23:56.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b5ffff640 1 -- 172.21.15.148:0/2805691026 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c004a10 con 0x7f9b78071110 2023-10-30T13:23:56.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.677+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b781f26c0 con 0x7f9b78071110 2023-10-30T13:23:56.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.681+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b781ee940 con 0x7f9b78071110 2023-10-30T13:23:56.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.681+0000 7f9b5ffff640 1 -- 172.21.15.148:0/2805691026 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b6c004b70 con 0x7f9b78071110 2023-10-30T13:23:56.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.681+0000 7f9b5ffff640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b640664d0 0x7f9b64068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:56.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.681+0000 7f9b5ffff640 1 -- 172.21.15.148:0/2805691026 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f9b6c083590 con 0x7f9b78071110 2023-10-30T13:23:56.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.681+0000 7f9b777fe640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b640664d0 0x7f9b64068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:56.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.681+0000 7f9b777fe640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b640664d0 0x7f9b64068990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9b680b2000 tx=0x7f9b68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:56.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.685+0000 7f9b5ffff640 1 -- 172.21.15.148:0/2805691026 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b6c050760 con 0x7f9b78071110 2023-10-30T13:23:56.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.865+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b7804e170 con 0x7f9b78071110 2023-10-30T13:23:56.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b5ffff640 1 -- 172.21.15.148:0/2805691026 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f9b6c053d80 con 0x7f9b78071110 2023-10-30T13:23:56.864 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:23:56.864 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:56.864 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 69 2023-10-30T13:23:56.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b640664d0 msgr2=0x7f9b64068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:56.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b640664d0 0x7f9b64068990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9b680b2000 tx=0x7f9b68008040 comp rx=0 tx=0).stop 2023-10-30T13:23:56.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b78071110 msgr2=0x7f9b780c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:56.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b78071110 0x7f9b780c5760 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c00c530 tx=0x7f9b6c00c9f0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 shutdown_connections 2023-10-30T13:23:56.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9b640664d0 0x7f9b64068990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9b78071110 0x7f9b780c5760 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9b780c6bd0 0x7f9b781eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 --2- 172.21.15.148:0/2805691026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b7806f7a0 0x7f9b781520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:56.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 >> 172.21.15.148:0/2805691026 conn(0x7f9b780c7910 msgr2=0x7f9b780c4140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:56.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 shutdown_connections 2023-10-30T13:23:56.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:23:56.869+0000 7f9b7e13e640 1 -- 172.21.15.148:0/2805691026 wait complete. 2023-10-30T13:23:56.885 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:23:56.885 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2023-10-30T13:23:58.777 INFO:tasks.ceph.mds.a.smithi148.stderr:2023-10-30T13:23:58.781+0000 7f4918a5f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:24:13.886 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:14.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9b03640 1 Processor -- start 2023-10-30T13:24:14.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9b03640 1 -- start start 2023-10-30T13:24:14.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9b03640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a415cd50 0x7f48a40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9b03640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f48a40cefa0 0x7f48a40cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9b03640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a415a6c0 con 0x7f48a41538f0 2023-10-30T13:24:14.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9b03640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a415d300 con 0x7f48a40cefa0 2023-10-30T13:24:14.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9b03640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a415d5e0 con 0x7f48a415cd50 2023-10-30T13:24:14.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9302640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f48a40cefa0 0x7f48a40cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9302640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f48a40cefa0 0x7f48a40cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52186/0 (socket says 172.21.15.148:52186) 2023-10-30T13:24:14.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a8b01640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f48a41538f0 0x7f48a4159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54858/0 (socket says 172.21.15.148:54858) 2023-10-30T13:24:14.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a9302640 1 -- 172.21.15.148:0/208843516 learned_addr learned my addr 172.21.15.148:0/208843516 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:14.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.064+0000 7f48a3fff640 1 --2- 172.21.15.148:0/208843516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a415cd50 0x7f48a40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.068+0000 7f48a9302640 1 -- 172.21.15.148:0/208843516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a415cd50 msgr2=0x7f48a40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:14.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.068+0000 7f48a9302640 1 --2- 172.21.15.148:0/208843516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a415cd50 0x7f48a40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.068+0000 7f48a9302640 1 -- 172.21.15.148:0/208843516 >> v1:172.21.15.148:6789/0 conn(0x7f48a41538f0 legacy=0x7f48a4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.068+0000 7f48a9302640 1 -- 172.21.15.148:0/208843516 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a415d8c0 con 0x7f48a40cefa0 2023-10-30T13:24:14.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.068+0000 7f48a9302640 1 --2- 172.21.15.148:0/208843516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a40cefa0 0x7f48a40cf3f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f489c009e50 tx=0x7f489c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff201a452ce7921 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:14.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.068+0000 7f48a37fe640 1 -- 172.21.15.148:0/208843516 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0af400 con 0x7f48a40cefa0 2023-10-30T13:24:14.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.068+0000 7f48a37fe640 1 -- 172.21.15.148:0/208843516 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c0ab050 con 0x7f48a40cefa0 2023-10-30T13:24:14.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.068+0000 7f48a37fe640 1 -- 172.21.15.148:0/208843516 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0af780 con 0x7f48a40cefa0 2023-10-30T13:24:14.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 -- 172.21.15.148:0/208843516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a40cefa0 msgr2=0x7f48a40cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 --2- 172.21.15.148:0/208843516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a40cefa0 0x7f48a40cf3f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f489c009e50 tx=0x7f489c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 -- 172.21.15.148:0/208843516 shutdown_connections 2023-10-30T13:24:14.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 --2- 172.21.15.148:0/208843516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a40cefa0 0x7f48a40cf3f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 --2- 172.21.15.148:0/208843516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a415cd50 0x7f48a40c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 -- 172.21.15.148:0/208843516 >> 172.21.15.148:0/208843516 conn(0x7f48a406a640 msgr2=0x7f48a4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:14.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 -- 172.21.15.148:0/208843516 shutdown_connections 2023-10-30T13:24:14.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 -- 172.21.15.148:0/208843516 wait complete. 2023-10-30T13:24:14.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 Processor -- start 2023-10-30T13:24:14.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.072+0000 7f48a9b03640 1 -- start start 2023-10-30T13:24:14.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9b03640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a40cefa0 0x7f48a41f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9b03640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a41538f0 0x7f48a4158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9b03640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a415cd50 0x7f48a40b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9b03640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a415c320 con 0x7f48a415cd50 2023-10-30T13:24:14.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9b03640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a40c6480 con 0x7f48a41538f0 2023-10-30T13:24:14.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9b03640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a41582d0 con 0x7f48a40cefa0 2023-10-30T13:24:14.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9302640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a415cd50 0x7f48a40b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a8b01640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a40cefa0 0x7f48a41f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9302640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a415cd50 0x7f48a40b3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52200/0 (socket says 172.21.15.148:52200) 2023-10-30T13:24:14.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9302640 1 -- 172.21.15.148:0/3568885144 learned_addr learned my addr 172.21.15.148:0/3568885144 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:14.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a3fff640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a41538f0 0x7f48a4158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9302640 1 -- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a40cefa0 msgr2=0x7f48a41f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9302640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a40cefa0 0x7f48a41f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9302640 1 -- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a41538f0 msgr2=0x7f48a4158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9302640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a41538f0 0x7f48a4158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9302640 1 -- 172.21.15.148:0/3568885144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c009b30 con 0x7f48a415cd50 2023-10-30T13:24:14.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9302640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a415cd50 0x7f48a40b3bc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f489c01ce20 tx=0x7f489c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:14.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a17fa640 1 -- 172.21.15.148:0/3568885144 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0afd30 con 0x7f48a415cd50 2023-10-30T13:24:14.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a17fa640 1 -- 172.21.15.148:0/3568885144 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c0afe90 con 0x7f48a415cd50 2023-10-30T13:24:14.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a41f3260 con 0x7f48a415cd50 2023-10-30T13:24:14.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a17fa640 1 -- 172.21.15.148:0/3568885144 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c0ba660 con 0x7f48a415cd50 2023-10-30T13:24:14.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.076+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a4158ca0 con 0x7f48a415cd50 2023-10-30T13:24:14.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.080+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a4158460 con 0x7f48a415cd50 2023-10-30T13:24:14.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.080+0000 7f48a17fa640 1 -- 172.21.15.148:0/3568885144 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f489c0b6040 con 0x7f48a415cd50 2023-10-30T13:24:14.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.080+0000 7f48a17fa640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f48880664a0 0x7f4888068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.080+0000 7f48a17fa640 1 -- 172.21.15.148:0/3568885144 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f489c122190 con 0x7f48a415cd50 2023-10-30T13:24:14.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.080+0000 7f48a8b01640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f48880664a0 0x7f4888068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.080+0000 7f48a8b01640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f48880664a0 0x7f4888068960 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4898000f70 tx=0x7f4898007e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:14.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.088+0000 7f48a17fa640 1 -- 172.21.15.148:0/3568885144 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f489c0ef440 con 0x7f48a415cd50 2023-10-30T13:24:14.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48a404e170 con 0x7f48a415cd50 2023-10-30T13:24:14.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a17fa640 1 -- 172.21.15.148:0/3568885144 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f489c009510 con 0x7f48a415cd50 2023-10-30T13:24:14.273 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:14.273 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:14.273 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 69 2023-10-30T13:24:14.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f48880664a0 msgr2=0x7f4888068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f48880664a0 0x7f4888068960 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4898000f70 tx=0x7f4898007e50 comp rx=0 tx=0).stop 2023-10-30T13:24:14.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a415cd50 msgr2=0x7f48a40b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a415cd50 0x7f48a40b3bc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f489c01ce20 tx=0x7f489c01bf90 comp rx=0 tx=0).stop 2023-10-30T13:24:14.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 shutdown_connections 2023-10-30T13:24:14.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f48880664a0 0x7f4888068960 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f48a415cd50 0x7f48a40b3bc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a41538f0 0x7f48a4158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 --2- 172.21.15.148:0/3568885144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f48a40cefa0 0x7f48a41f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 >> 172.21.15.148:0/3568885144 conn(0x7f48a406a640 msgr2=0x7f48a40cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:14.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 shutdown_connections 2023-10-30T13:24:14.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.276+0000 7f48a9b03640 1 -- 172.21.15.148:0/3568885144 wait complete. 2023-10-30T13:24:14.293 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:24:14.293 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:24:14.293 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:24:14.311 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:24:14.311 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:14.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.496+0000 7fdecd246640 1 Processor -- start 2023-10-30T13:24:14.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.496+0000 7fdecd246640 1 -- start start 2023-10-30T13:24:14.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdecd246640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdec80c6b80 0x7fdec80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdecd246640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec80c0a50 con 0x7fdec813f4a0 2023-10-30T13:24:14.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdecd246640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec806f6e0 con 0x7fdec80c6f60 2023-10-30T13:24:14.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdecd246640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec806f9c0 con 0x7fdec80c6b80 2023-10-30T13:24:14.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdec6d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdec80c6b80 0x7fdec80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdec6d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdec80c6b80 0x7fdec80bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52214/0 (socket says 172.21.15.148:52214) 2023-10-30T13:24:14.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdec6d9d640 1 -- 172.21.15.148:0/1407853655 learned_addr learned my addr 172.21.15.148:0/1407853655 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:14.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdec6d9d640 1 -- 172.21.15.148:0/1407853655 >> v1:172.21.15.150:6789/0 conn(0x7fdec80c6f60 legacy=0x7fdec813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdec6d9d640 1 -- 172.21.15.148:0/1407853655 >> v1:172.21.15.148:6789/0 conn(0x7fdec813f4a0 legacy=0x7fdec81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.500+0000 7fdec6d9d640 1 -- 172.21.15.148:0/1407853655 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec806fca0 con 0x7fdec80c6b80 2023-10-30T13:24:14.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdec6d9d640 1 --2- 172.21.15.148:0/1407853655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 0x7fdec80bfec0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009870 tx=0x7fdeb801e750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203c1aa8130450ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:14.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdec5d9b640 1 -- 172.21.15.148:0/1407853655 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80ab630 con 0x7fdec80c6b80 2023-10-30T13:24:14.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdec5d9b640 1 -- 172.21.15.148:0/1407853655 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb80ab790 con 0x7fdec80c6b80 2023-10-30T13:24:14.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdec5d9b640 1 -- 172.21.15.148:0/1407853655 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80abb20 con 0x7fdec80c6b80 2023-10-30T13:24:14.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdecd246640 1 -- 172.21.15.148:0/1407853655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 msgr2=0x7fdec80bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdecd246640 1 --2- 172.21.15.148:0/1407853655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 0x7fdec80bfec0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009870 tx=0x7fdeb801e750 comp rx=0 tx=0).stop 2023-10-30T13:24:14.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdecd246640 1 -- 172.21.15.148:0/1407853655 shutdown_connections 2023-10-30T13:24:14.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdecd246640 1 --2- 172.21.15.148:0/1407853655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 0x7fdec80bfec0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdecd246640 1 -- 172.21.15.148:0/1407853655 >> 172.21.15.148:0/1407853655 conn(0x7fdec80c7910 msgr2=0x7fdec80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:14.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdecd246640 1 -- 172.21.15.148:0/1407853655 shutdown_connections 2023-10-30T13:24:14.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdecd246640 1 -- 172.21.15.148:0/1407853655 wait complete. 2023-10-30T13:24:14.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdecd246640 1 Processor -- start 2023-10-30T13:24:14.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.504+0000 7fdecd246640 1 -- start start 2023-10-30T13:24:14.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdecd246640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 0x7fdec81f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdecd246640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdec80c6f60 0x7fdec813e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdecd246640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdec813f4a0 0x7fdec80ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdecd246640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80c2d70 con 0x7fdec80c6b80 2023-10-30T13:24:14.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdecd246640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec813df50 con 0x7fdec813f4a0 2023-10-30T13:24:14.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdecd246640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec813dc20 con 0x7fdec80c6f60 2023-10-30T13:24:14.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 0x7fdec81f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 0x7fdec81f5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52216/0 (socket says 172.21.15.148:52216) 2023-10-30T13:24:14.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec6d9d640 1 -- 172.21.15.148:0/2219804313 learned_addr learned my addr 172.21.15.148:0/2219804313 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:14.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec759e640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdec813f4a0 0x7fdec80ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec659c640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdec80c6f60 0x7fdec813e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec6d9d640 1 -- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdec80c6f60 msgr2=0x7fdec813e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec6d9d640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdec80c6f60 0x7fdec813e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec6d9d640 1 -- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdec813f4a0 msgr2=0x7fdec80ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec6d9d640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdec813f4a0 0x7fdec80ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec6d9d640 1 -- 172.21.15.148:0/2219804313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb80094d0 con 0x7fdec80c6b80 2023-10-30T13:24:14.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdec6d9d640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 0x7fdec81f5ee0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80ad040 tx=0x7fdeb801cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:14.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdeaf7fe640 1 -- 172.21.15.148:0/2219804313 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80b3800 con 0x7fdec80c6b80 2023-10-30T13:24:14.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdeaf7fe640 1 -- 172.21.15.148:0/2219804313 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb80b3980 con 0x7fdec80c6b80 2023-10-30T13:24:14.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec80c4850 con 0x7fdec80c6b80 2023-10-30T13:24:14.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdeaf7fe640 1 -- 172.21.15.148:0/2219804313 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80b3d30 con 0x7fdec80c6b80 2023-10-30T13:24:14.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec80c4290 con 0x7fdec80c6b80 2023-10-30T13:24:14.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec804e170 con 0x7fdec80c6b80 2023-10-30T13:24:14.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.508+0000 7fdeaf7fe640 1 -- 172.21.15.148:0/2219804313 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdeb80c6390 con 0x7fdec80c6b80 2023-10-30T13:24:14.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.512+0000 7fdeaf7fe640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdeb40664a0 0x7fdeb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:14.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.512+0000 7fdeaf7fe640 1 -- 172.21.15.148:0/2219804313 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fdeb811ecc0 con 0x7fdec80c6b80 2023-10-30T13:24:14.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.512+0000 7fdec659c640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdeb40664a0 0x7fdeb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:14.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.512+0000 7fdec659c640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdeb40664a0 0x7fdeb4068960 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdec81f6ff0 tx=0x7fdebc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:14.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.516+0000 7fdeaf7fe640 1 -- 172.21.15.148:0/2219804313 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdeb8125080 con 0x7fdec80c6b80 2023-10-30T13:24:14.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.696+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdec806a450 con 0x7fdec80c6b80 2023-10-30T13:24:14.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.696+0000 7fdeaf7fe640 1 -- 172.21.15.148:0/2219804313 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fdeb80ebe90 con 0x7fdec80c6b80 2023-10-30T13:24:14.695 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:14.695 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:14.695 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 69 2023-10-30T13:24:14.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdeb40664a0 msgr2=0x7fdeb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdeb40664a0 0x7fdeb4068960 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdec81f6ff0 tx=0x7fdebc009040 comp rx=0 tx=0).stop 2023-10-30T13:24:14.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 msgr2=0x7fdec81f5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:14.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 0x7fdec81f5ee0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80ad040 tx=0x7fdeb801cf90 comp rx=0 tx=0).stop 2023-10-30T13:24:14.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 shutdown_connections 2023-10-30T13:24:14.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdeb40664a0 0x7fdeb4068960 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdec813f4a0 0x7fdec80ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdec80c6f60 0x7fdec813e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 --2- 172.21.15.148:0/2219804313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdec80c6b80 0x7fdec81f5ee0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:14.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 >> 172.21.15.148:0/2219804313 conn(0x7fdec80c7910 msgr2=0x7fdec806a970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:14.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 shutdown_connections 2023-10-30T13:24:14.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:14.700+0000 7fdecd246640 1 -- 172.21.15.148:0/2219804313 wait complete. 2023-10-30T13:24:14.716 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:24:16.717 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:16.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e66f80640 1 Processor -- start 2023-10-30T13:24:16.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e66f80640 1 -- start start 2023-10-30T13:24:16.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e66f80640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 0x7f1e600bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e66f80640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e600c6f60 0x7f1e600c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e66f80640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e600c0a50 con 0x7f1e600c8430 2023-10-30T13:24:16.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e66f80640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e6006f6e0 con 0x7f1e600c6f60 2023-10-30T13:24:16.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e66f80640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6006f9c0 con 0x7f1e600c6b80 2023-10-30T13:24:16.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e5ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e600c6f60 0x7f1e600c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e64d1c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 0x7f1e600bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e6551d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1e600c8430 0x7f1e6013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:45238/0 (socket says 172.21.15.148:45238) 2023-10-30T13:24:16.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e6551d640 1 -- 172.21.15.148:0/3910959056 learned_addr learned my addr 172.21.15.148:0/3910959056 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:24:16.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e5ffff640 1 -- 172.21.15.148:0/3910959056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 msgr2=0x7f1e600bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:16.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e5ffff640 1 --2- 172.21.15.148:0/3910959056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 0x7f1e600bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e5ffff640 1 -- 172.21.15.148:0/3910959056 >> v1:172.21.15.150:6789/0 conn(0x7f1e600c8430 legacy=0x7f1e6013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.872+0000 7f1e5ffff640 1 -- 172.21.15.148:0/3910959056 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6006fca0 con 0x7f1e600c6f60 2023-10-30T13:24:16.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e5ffff640 1 --2- 172.21.15.148:0/3910959056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e600c6f60 0x7f1e600c0400 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0097c0 tx=0x7f1e4c01e690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57e964c4d80950b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:16.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e5f7fe640 1 -- 172.21.15.148:0/3910959056 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c0ab670 con 0x7f1e600c6f60 2023-10-30T13:24:16.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e5f7fe640 1 -- 172.21.15.148:0/3910959056 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4c0ab7d0 con 0x7f1e600c6f60 2023-10-30T13:24:16.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e5f7fe640 1 -- 172.21.15.148:0/3910959056 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c0abb60 con 0x7f1e600c6f60 2023-10-30T13:24:16.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e66f80640 1 -- 172.21.15.148:0/3910959056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e600c6f60 msgr2=0x7f1e600c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e66f80640 1 --2- 172.21.15.148:0/3910959056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e600c6f60 0x7f1e600c0400 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0097c0 tx=0x7f1e4c01e690 comp rx=0 tx=0).stop 2023-10-30T13:24:16.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e66f80640 1 -- 172.21.15.148:0/3910959056 shutdown_connections 2023-10-30T13:24:16.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e66f80640 1 --2- 172.21.15.148:0/3910959056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e600c6f60 0x7f1e600c0400 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e66f80640 1 --2- 172.21.15.148:0/3910959056 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 0x7f1e600bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e66f80640 1 -- 172.21.15.148:0/3910959056 >> 172.21.15.148:0/3910959056 conn(0x7f1e600c7910 msgr2=0x7f1e600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:16.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e66f80640 1 -- 172.21.15.148:0/3910959056 shutdown_connections 2023-10-30T13:24:16.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.876+0000 7f1e66f80640 1 -- 172.21.15.148:0/3910959056 wait complete. 2023-10-30T13:24:16.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 Processor -- start 2023-10-30T13:24:16.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 -- start start 2023-10-30T13:24:16.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 0x7f1e60133120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e600c6f60 0x7f1e60155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e600c8430 0x7f1e60155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e601f0f80 con 0x7f1e600c8430 2023-10-30T13:24:16.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e601594f0 con 0x7f1e600c6f60 2023-10-30T13:24:16.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e600c54e0 con 0x7f1e600c6b80 2023-10-30T13:24:16.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e6551d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e600c8430 0x7f1e60155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e6551d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e600c8430 0x7f1e60155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52224/0 (socket says 172.21.15.148:52224) 2023-10-30T13:24:16.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e6551d640 1 -- 172.21.15.148:0/716501553 learned_addr learned my addr 172.21.15.148:0/716501553 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:16.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e5ffff640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e600c6f60 0x7f1e60155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e64d1c640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 0x7f1e60133120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e6551d640 1 -- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 msgr2=0x7f1e60133120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e6551d640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 0x7f1e60133120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e6551d640 1 -- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e600c6f60 msgr2=0x7f1e60155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e6551d640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e600c6f60 0x7f1e60155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e6551d640 1 -- 172.21.15.148:0/716501553 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c009420 con 0x7f1e600c8430 2023-10-30T13:24:16.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e6551d640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e600c8430 0x7f1e60155da0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1e5800ca80 tx=0x7f1e5800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:16.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e5d7fa640 1 -- 172.21.15.148:0/716501553 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5800ac30 con 0x7f1e600c8430 2023-10-30T13:24:16.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e5d7fa640 1 -- 172.21.15.148:0/716501553 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5800ad90 con 0x7f1e600c8430 2023-10-30T13:24:16.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e5d7fa640 1 -- 172.21.15.148:0/716501553 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e58012600 con 0x7f1e600c8430 2023-10-30T13:24:16.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e60159980 con 0x7f1e600c8430 2023-10-30T13:24:16.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.880+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e601588e0 con 0x7f1e600c8430 2023-10-30T13:24:16.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.884+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e60159660 con 0x7f1e600c8430 2023-10-30T13:24:16.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.884+0000 7f1e5d7fa640 1 -- 172.21.15.148:0/716501553 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1e58007070 con 0x7f1e600c8430 2023-10-30T13:24:16.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.884+0000 7f1e5d7fa640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e440664a0 0x7f1e44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.884+0000 7f1e5d7fa640 1 -- 172.21.15.148:0/716501553 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f1e58083ed0 con 0x7f1e600c8430 2023-10-30T13:24:16.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.884+0000 7f1e64d1c640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e440664a0 0x7f1e44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.884+0000 7f1e64d1c640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e440664a0 0x7f1e44068960 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009b60 tx=0x7f1e54009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:16.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:16.888+0000 7f1e5d7fa640 1 -- 172.21.15.148:0/716501553 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1e58051160 con 0x7f1e600c8430 2023-10-30T13:24:17.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.068+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e6004e170 con 0x7f1e600c8430 2023-10-30T13:24:17.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.068+0000 7f1e5d7fa640 1 -- 172.21.15.148:0/716501553 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f1e58017090 con 0x7f1e600c8430 2023-10-30T13:24:17.067 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:17.067 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:17.067 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 69 2023-10-30T13:24:17.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e440664a0 msgr2=0x7f1e44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:17.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e440664a0 0x7f1e44068960 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009b60 tx=0x7f1e54009190 comp rx=0 tx=0).stop 2023-10-30T13:24:17.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e600c8430 msgr2=0x7f1e60155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:17.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e600c8430 0x7f1e60155da0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1e5800ca80 tx=0x7f1e5800cf40 comp rx=0 tx=0).stop 2023-10-30T13:24:17.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 shutdown_connections 2023-10-30T13:24:17.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e440664a0 0x7f1e44068960 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:17.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e600c8430 0x7f1e60155da0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:17.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e600c6f60 0x7f1e60155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:17.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 --2- 172.21.15.148:0/716501553 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e600c6b80 0x7f1e60133120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:17.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 >> 172.21.15.148:0/716501553 conn(0x7f1e600c7910 msgr2=0x7f1e600c10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:17.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 shutdown_connections 2023-10-30T13:24:17.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:17.072+0000 7f1e66f80640 1 -- 172.21.15.148:0/716501553 wait complete. 2023-10-30T13:24:17.088 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:24:19.089 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:19.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.240+0000 7f9812691640 1 Processor -- start 2023-10-30T13:24:19.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.240+0000 7f9812691640 1 -- start start 2023-10-30T13:24:19.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.240+0000 7f9812691640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f980c0c6b80 0x7f980c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:19.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.240+0000 7f9812691640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c13f4a0 0x7f980c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:19.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.240+0000 7f9812691640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c0c0a50 con 0x7f980c0c6f60 2023-10-30T13:24:19.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.240+0000 7f9812691640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c06f6e0 con 0x7f980c0c6b80 2023-10-30T13:24:19.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.240+0000 7f9812691640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c06f9c0 con 0x7f980c13f4a0 2023-10-30T13:24:19.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f980c0c6b80 0x7f980c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:19.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f980c0c6b80 0x7f980c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52240/0 (socket says 172.21.15.148:52240) 2023-10-30T13:24:19.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980bfff640 1 -- 172.21.15.148:0/484942386 learned_addr learned my addr 172.21.15.148:0/484942386 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:19.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f9810c2e640 1 --2- 172.21.15.148:0/484942386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c13f4a0 0x7f980c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:19.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980bfff640 1 -- 172.21.15.148:0/484942386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c13f4a0 msgr2=0x7f980c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:19.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980bfff640 1 --2- 172.21.15.148:0/484942386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c13f4a0 0x7f980c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980bfff640 1 -- 172.21.15.148:0/484942386 >> v1:172.21.15.150:6789/0 conn(0x7f980c0c6f60 legacy=0x7f980c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:19.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980bfff640 1 -- 172.21.15.148:0/484942386 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c06fca0 con 0x7f980c0c6b80 2023-10-30T13:24:19.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980bfff640 1 --2- 172.21.15.148:0/484942386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c0c6b80 0x7f980c0bfec0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009870 tx=0x7f97fc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4859096a79c2913 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:19.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980affd640 1 -- 172.21.15.148:0/484942386 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0ab650 con 0x7f980c0c6b80 2023-10-30T13:24:19.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980affd640 1 -- 172.21.15.148:0/484942386 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc0ab7b0 con 0x7f980c0c6b80 2023-10-30T13:24:19.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.244+0000 7f980affd640 1 -- 172.21.15.148:0/484942386 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0abb40 con 0x7f980c0c6b80 2023-10-30T13:24:19.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 -- 172.21.15.148:0/484942386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c0c6b80 msgr2=0x7f980c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:19.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 --2- 172.21.15.148:0/484942386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c0c6b80 0x7f980c0bfec0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009870 tx=0x7f97fc01c540 comp rx=0 tx=0).stop 2023-10-30T13:24:19.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 -- 172.21.15.148:0/484942386 shutdown_connections 2023-10-30T13:24:19.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 --2- 172.21.15.148:0/484942386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c13f4a0 0x7f980c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 --2- 172.21.15.148:0/484942386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c0c6b80 0x7f980c0bfec0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 -- 172.21.15.148:0/484942386 >> 172.21.15.148:0/484942386 conn(0x7f980c0c7910 msgr2=0x7f980c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:19.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 -- 172.21.15.148:0/484942386 shutdown_connections 2023-10-30T13:24:19.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 -- 172.21.15.148:0/484942386 wait complete. 2023-10-30T13:24:19.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 Processor -- start 2023-10-30T13:24:19.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 -- start start 2023-10-30T13:24:19.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c0c6b80 0x7f980c14fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:19.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f980c0c6f60 0x7f980c14ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:19.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c13f4a0 0x7f980c135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:19.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c1564c0 con 0x7f980c13f4a0 2023-10-30T13:24:19.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c134fa0 con 0x7f980c0c6f60 2023-10-30T13:24:19.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9812691640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c158240 con 0x7f980c0c6b80 2023-10-30T13:24:19.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9810c2e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c13f4a0 0x7f980c135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:19.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9810c2e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c13f4a0 0x7f980c135430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52256/0 (socket says 172.21.15.148:52256) 2023-10-30T13:24:19.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9810c2e640 1 -- 172.21.15.148:0/565872578 learned_addr learned my addr 172.21.15.148:0/565872578 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:19.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f980bfff640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c0c6b80 0x7f980c14fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:19.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f980b7fe640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f980c0c6f60 0x7f980c14ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:19.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9810c2e640 1 -- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c0c6b80 msgr2=0x7f980c14fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:19.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9810c2e640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c0c6b80 0x7f980c14fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9810c2e640 1 -- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f980c0c6f60 msgr2=0x7f980c14ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:19.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.248+0000 7f9810c2e640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f980c0c6f60 0x7f980c14ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9810c2e640 1 -- 172.21.15.148:0/565872578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc0094d0 con 0x7f980c13f4a0 2023-10-30T13:24:19.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9810c2e640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c13f4a0 0x7f980c135430 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9804007e90 tx=0x7f980400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:19.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9808ff9640 1 -- 172.21.15.148:0/565872578 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9804014420 con 0x7f980c13f4a0 2023-10-30T13:24:19.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9808ff9640 1 -- 172.21.15.148:0/565872578 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9804014580 con 0x7f980c13f4a0 2023-10-30T13:24:19.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9808ff9640 1 -- 172.21.15.148:0/565872578 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9804014930 con 0x7f980c13f4a0 2023-10-30T13:24:19.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c1584b0 con 0x7f980c13f4a0 2023-10-30T13:24:19.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c134c30 con 0x7f980c13f4a0 2023-10-30T13:24:19.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c04e170 con 0x7f980c13f4a0 2023-10-30T13:24:19.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9808ff9640 1 -- 172.21.15.148:0/565872578 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9804014a90 con 0x7f980c13f4a0 2023-10-30T13:24:19.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.252+0000 7f9808ff9640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97f80664a0 0x7f97f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:19.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.256+0000 7f9808ff9640 1 -- 172.21.15.148:0/565872578 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f98040849b0 con 0x7f980c13f4a0 2023-10-30T13:24:19.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.256+0000 7f980bfff640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97f80664a0 0x7f97f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:19.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.256+0000 7f980bfff640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97f80664a0 0x7f97f8068960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0af000 tx=0x7f97fc0b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:19.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.260+0000 7f9808ff9640 1 -- 172.21.15.148:0/565872578 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9804051ba0 con 0x7f980c13f4a0 2023-10-30T13:24:19.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f980c1350e0 con 0x7f980c13f4a0 2023-10-30T13:24:19.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9808ff9640 1 -- 172.21.15.148:0/565872578 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f98040551c0 con 0x7f980c13f4a0 2023-10-30T13:24:19.436 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:19.437 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:19.437 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 69 2023-10-30T13:24:19.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97f80664a0 msgr2=0x7f97f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:19.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97f80664a0 0x7f97f8068960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0af000 tx=0x7f97fc0b6040 comp rx=0 tx=0).stop 2023-10-30T13:24:19.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c13f4a0 msgr2=0x7f980c135430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:19.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c13f4a0 0x7f980c135430 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9804007e90 tx=0x7f980400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 shutdown_connections 2023-10-30T13:24:19.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97f80664a0 0x7f97f8068960 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f980c13f4a0 0x7f980c135430 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f980c0c6f60 0x7f980c14ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 --2- 172.21.15.148:0/565872578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f980c0c6b80 0x7f980c14fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:19.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 >> 172.21.15.148:0/565872578 conn(0x7f980c0c7910 msgr2=0x7f980c0c0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:19.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 shutdown_connections 2023-10-30T13:24:19.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:19.440+0000 7f9812691640 1 -- 172.21.15.148:0/565872578 wait complete. 2023-10-30T13:24:19.457 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:24:21.459 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:21.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.608+0000 7fc77a33e640 1 Processor -- start 2023-10-30T13:24:21.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.608+0000 7fc77a33e640 1 -- start start 2023-10-30T13:24:21.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc77a33e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc7740bfec0 0x7fc7740c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:21.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc77a33e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc774071290 con 0x7fc77406a720 2023-10-30T13:24:21.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc77a33e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc774071570 con 0x7fc774159e60 2023-10-30T13:24:21.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc77a33e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc774071850 con 0x7fc7740bfec0 2023-10-30T13:24:21.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc7788db640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc77406a720 0x7fc774160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54874/0 (socket says 172.21.15.148:54874) 2023-10-30T13:24:21.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc7788db640 1 -- 172.21.15.148:0/2768975187 learned_addr learned my addr 172.21.15.148:0/2768975187 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:24:21.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc7737fe640 1 --2- 172.21.15.148:0/2768975187 >> v2:172.21.15.150:3300/0 conn(0x7fc7740bfec0 0x7fc7740c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:21.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc772ffd640 1 -- 172.21.15.148:0/2768975187 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230349245 0 0) 0x7fc774071290 con 0x7fc77406a720 2023-10-30T13:24:21.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc772ffd640 1 -- 172.21.15.148:0/2768975187 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc75c003530 con 0x7fc77406a720 2023-10-30T13:24:21.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc772ffd640 1 -- 172.21.15.148:0/2768975187 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 864858371 0 0) 0x7fc75c003530 con 0x7fc77406a720 2023-10-30T13:24:21.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc7737fe640 1 -- 172.21.15.148:0/2768975187 >> v1:172.21.15.150:6789/0 conn(0x7fc774159e60 legacy=0x7fc77406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:21.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc7737fe640 1 -- 172.21.15.148:0/2768975187 >> v1:172.21.15.148:6789/0 conn(0x7fc77406a720 legacy=0x7fc774160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:21.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.612+0000 7fc7737fe640 1 -- 172.21.15.148:0/2768975187 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc774071b30 con 0x7fc7740bfec0 2023-10-30T13:24:21.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc7737fe640 1 --2- 172.21.15.148:0/2768975187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7740bfec0 0x7fc7740c1a40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc76800a960 tx=0x7fc76801ff60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b085b30371434bc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:21.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc772ffd640 1 -- 172.21.15.148:0/2768975187 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c001a00 con 0x7fc7740bfec0 2023-10-30T13:24:21.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc772ffd640 1 -- 172.21.15.148:0/2768975187 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7680a9af0 con 0x7fc7740bfec0 2023-10-30T13:24:21.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc772ffd640 1 -- 172.21.15.148:0/2768975187 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680a9e60 con 0x7fc7740bfec0 2023-10-30T13:24:21.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc77a33e640 1 -- 172.21.15.148:0/2768975187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7740bfec0 msgr2=0x7fc7740c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:21.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc77a33e640 1 --2- 172.21.15.148:0/2768975187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7740bfec0 0x7fc7740c1a40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc76800a960 tx=0x7fc76801ff60 comp rx=0 tx=0).stop 2023-10-30T13:24:21.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc77a33e640 1 -- 172.21.15.148:0/2768975187 shutdown_connections 2023-10-30T13:24:21.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc77a33e640 1 --2- 172.21.15.148:0/2768975187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7740bfec0 0x7fc7740c1a40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc77a33e640 1 -- 172.21.15.148:0/2768975187 >> 172.21.15.148:0/2768975187 conn(0x7fc7740c7910 msgr2=0x7fc7740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:21.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc77a33e640 1 -- 172.21.15.148:0/2768975187 shutdown_connections 2023-10-30T13:24:21.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc77a33e640 1 -- 172.21.15.148:0/2768975187 wait complete. 2023-10-30T13:24:21.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.616+0000 7fc77a33e640 1 Processor -- start 2023-10-30T13:24:21.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 -- start start 2023-10-30T13:24:21.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc77406a720 0x7fc774154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:21.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7740bfec0 0x7fc77414d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:21.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc774159e60 0x7fc774155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:21.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc774155260 con 0x7fc774159e60 2023-10-30T13:24:21.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7740c2200 con 0x7fc77406a720 2023-10-30T13:24:21.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77414e320 con 0x7fc7740bfec0 2023-10-30T13:24:21.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7788db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc774159e60 0x7fc774155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:21.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7788db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc774159e60 0x7fc774155680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52266/0 (socket says 172.21.15.148:52266) 2023-10-30T13:24:21.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7788db640 1 -- 172.21.15.148:0/2250256955 learned_addr learned my addr 172.21.15.148:0/2250256955 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:21.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc773fff640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc77406a720 0x7fc774154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:21.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7737fe640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7740bfec0 0x7fc77414d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:21.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7788db640 1 -- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7740bfec0 msgr2=0x7fc77414d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:21.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7788db640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7740bfec0 0x7fc77414d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7788db640 1 -- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc77406a720 msgr2=0x7fc774154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:21.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7788db640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc77406a720 0x7fc774154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7788db640 1 -- 172.21.15.148:0/2250256955 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76800a5c0 con 0x7fc774159e60 2023-10-30T13:24:21.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc7788db640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc774159e60 0x7fc774155680 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc76c0018a0 tx=0x7fc76c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:21.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc770ff9640 1 -- 172.21.15.148:0/2250256955 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c012400 con 0x7fc774159e60 2023-10-30T13:24:21.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc770ff9640 1 -- 172.21.15.148:0/2250256955 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76c012560 con 0x7fc774159e60 2023-10-30T13:24:21.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc770ff9640 1 -- 172.21.15.148:0/2250256955 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c012910 con 0x7fc774159e60 2023-10-30T13:24:21.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc774159060 con 0x7fc774159e60 2023-10-30T13:24:21.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77414e040 con 0x7fc774159e60 2023-10-30T13:24:21.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.620+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7741553a0 con 0x7fc774159e60 2023-10-30T13:24:21.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.624+0000 7fc770ff9640 1 -- 172.21.15.148:0/2250256955 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc76c012a70 con 0x7fc774159e60 2023-10-30T13:24:21.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.624+0000 7fc770ff9640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7600664a0 0x7fc760068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:21.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.624+0000 7fc770ff9640 1 -- 172.21.15.148:0/2250256955 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fc76c084950 con 0x7fc774159e60 2023-10-30T13:24:21.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.624+0000 7fc773fff640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7600664a0 0x7fc760068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:21.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.624+0000 7fc773fff640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7600664a0 0x7fc760068960 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc764007fd0 tx=0x7fc764009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:21.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.628+0000 7fc770ff9640 1 -- 172.21.15.148:0/2250256955 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc76c051a90 con 0x7fc774159e60 2023-10-30T13:24:21.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.808+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc77404e170 con 0x7fc774159e60 2023-10-30T13:24:21.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.808+0000 7fc770ff9640 1 -- 172.21.15.148:0/2250256955 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fc76c0550b0 con 0x7fc774159e60 2023-10-30T13:24:21.807 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:21.807 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:21.808 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 69 2023-10-30T13:24:21.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7600664a0 msgr2=0x7fc760068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:21.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7600664a0 0x7fc760068960 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc764007fd0 tx=0x7fc764009040 comp rx=0 tx=0).stop 2023-10-30T13:24:21.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc774159e60 msgr2=0x7fc774155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:21.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc774159e60 0x7fc774155680 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc76c0018a0 tx=0x7fc76c00a430 comp rx=0 tx=0).stop 2023-10-30T13:24:21.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 shutdown_connections 2023-10-30T13:24:21.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7600664a0 0x7fc760068960 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc774159e60 0x7fc774155680 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7740bfec0 0x7fc77414d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 --2- 172.21.15.148:0/2250256955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc77406a720 0x7fc774154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 >> 172.21.15.148:0/2250256955 conn(0x7fc7740c7910 msgr2=0x7fc7740ca3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:21.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 shutdown_connections 2023-10-30T13:24:21.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:21.812+0000 7fc77a33e640 1 -- 172.21.15.148:0/2250256955 wait complete. 2023-10-30T13:24:21.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:24:23.829 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:23.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.980+0000 7f2343417640 1 Processor -- start 2023-10-30T13:24:23.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.980+0000 7f2343417640 1 -- start start 2023-10-30T13:24:23.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.980+0000 7f2343417640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f233c065a80 0x7f233c0cb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.980+0000 7f2343417640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233c1560a0 con 0x7f233c065f10 2023-10-30T13:24:23.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.980+0000 7f2343417640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233c156380 con 0x7f233c14fd50 2023-10-30T13:24:23.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.980+0000 7f2343417640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233c156660 con 0x7f233c065a80 2023-10-30T13:24:23.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.980+0000 7f23409b2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f233c065a80 0x7f233c0cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f23419b4640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f233c065f10 0x7f233c071250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36582/0 (socket says 172.21.15.148:36582) 2023-10-30T13:24:23.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f23419b4640 1 -- 172.21.15.148:0/1163529095 learned_addr learned my addr 172.21.15.148:0/1163529095 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:24:23.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f23409b2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f233c065a80 0x7f233c0cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57648/0 (socket says 172.21.15.148:57648) 2023-10-30T13:24:23.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f23409b2640 1 -- 172.21.15.148:0/1163529095 >> v1:172.21.15.150:6789/0 conn(0x7f233c14fd50 legacy=0x7f233c0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f23409b2640 1 -- 172.21.15.148:0/1163529095 >> v1:172.21.15.148:6789/0 conn(0x7f233c065f10 legacy=0x7f233c071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f23409b2640 1 -- 172.21.15.148:0/1163529095 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c156940 con 0x7f233c065a80 2023-10-30T13:24:23.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f23409b2640 1 --2- 172.21.15.148:0/1163529095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 0x7f233c0cb550 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f232c0097c0 tx=0x7f232c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1dbc2255de31990 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:23.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f2333fff640 1 -- 172.21.15.148:0/1163529095 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c0ac620 con 0x7f233c065a80 2023-10-30T13:24:23.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f2333fff640 1 -- 172.21.15.148:0/1163529095 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c0ac780 con 0x7f233c065a80 2023-10-30T13:24:23.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.984+0000 7f2333fff640 1 -- 172.21.15.148:0/1163529095 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c0acb10 con 0x7f233c065a80 2023-10-30T13:24:23.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 -- 172.21.15.148:0/1163529095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 msgr2=0x7f233c0cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 --2- 172.21.15.148:0/1163529095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 0x7f233c0cb550 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f232c0097c0 tx=0x7f232c01c420 comp rx=0 tx=0).stop 2023-10-30T13:24:23.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 -- 172.21.15.148:0/1163529095 shutdown_connections 2023-10-30T13:24:23.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 --2- 172.21.15.148:0/1163529095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 0x7f233c0cb550 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 -- 172.21.15.148:0/1163529095 >> 172.21.15.148:0/1163529095 conn(0x7f233c06c6e0 msgr2=0x7f233c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:23.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 -- 172.21.15.148:0/1163529095 shutdown_connections 2023-10-30T13:24:23.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 -- 172.21.15.148:0/1163529095 wait complete. 2023-10-30T13:24:23.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 Processor -- start 2023-10-30T13:24:23.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 -- start start 2023-10-30T13:24:23.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 0x7f233c140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f233c065f10 0x7f233c1541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f233c14fd50 0x7f233c154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c153d30 con 0x7f233c065a80 2023-10-30T13:24:23.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c146840 con 0x7f233c065f10 2023-10-30T13:24:23.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f2343417640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233c14ebd0 con 0x7f233c14fd50 2023-10-30T13:24:23.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f23411b3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 0x7f233c140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f23411b3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 0x7f233c140e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57652/0 (socket says 172.21.15.148:57652) 2023-10-30T13:24:23.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f23411b3640 1 -- 172.21.15.148:0/905235589 learned_addr learned my addr 172.21.15.148:0/905235589 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:23.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f23409b2640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f233c065f10 0x7f233c1541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.988+0000 7f23419b4640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f233c14fd50 0x7f233c154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f23411b3640 1 -- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f233c14fd50 msgr2=0x7f233c154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f23411b3640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f233c14fd50 0x7f233c154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f23411b3640 1 -- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f233c065f10 msgr2=0x7f233c1541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f23411b3640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f233c065f10 0x7f233c1541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f23411b3640 1 -- 172.21.15.148:0/905235589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c009420 con 0x7f233c065a80 2023-10-30T13:24:23.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f23411b3640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 0x7f233c140e10 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f233800c500 tx=0x7f233800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:23.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f2331ffb640 1 -- 172.21.15.148:0/905235589 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2338005220 con 0x7f233c065a80 2023-10-30T13:24:23.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f2331ffb640 1 -- 172.21.15.148:0/905235589 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2338005380 con 0x7f233c065a80 2023-10-30T13:24:23.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233c14eeb0 con 0x7f233c065a80 2023-10-30T13:24:23.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f2331ffb640 1 -- 172.21.15.148:0/905235589 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2338007270 con 0x7f233c065a80 2023-10-30T13:24:23.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233c14f470 con 0x7f233c065a80 2023-10-30T13:24:23.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233c04e170 con 0x7f233c065a80 2023-10-30T13:24:23.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f2331ffb640 1 -- 172.21.15.148:0/905235589 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23380073d0 con 0x7f233c065a80 2023-10-30T13:24:23.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f2331ffb640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23200664d0 0x7f2320068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f2331ffb640 1 -- 172.21.15.148:0/905235589 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7f2338083990 con 0x7f233c065a80 2023-10-30T13:24:23.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.992+0000 7f23409b2640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23200664d0 0x7f2320068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.996+0000 7f23409b2640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23200664d0 0x7f2320068990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f232c0ae000 tx=0x7f232c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:23.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:23.996+0000 7f2331ffb640 1 -- 172.21.15.148:0/905235589 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2338050b90 con 0x7f233c065a80 2023-10-30T13:24:24.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.180+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f233c0cc1b0 con 0x7f233c065a80 2023-10-30T13:24:24.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.180+0000 7f2331ffb640 1 -- 172.21.15.148:0/905235589 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f23380541b0 con 0x7f233c065a80 2023-10-30T13:24:24.177 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:24.178 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:24.178 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 69 2023-10-30T13:24:24.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.180+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23200664d0 msgr2=0x7f2320068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:24.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.180+0000 7f2343417640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23200664d0 0x7f2320068990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f232c0ae000 tx=0x7f232c008040 comp rx=0 tx=0).stop 2023-10-30T13:24:24.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.180+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 msgr2=0x7f233c140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:24.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.180+0000 7f2343417640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 0x7f233c140e10 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f233800c500 tx=0x7f233800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:24:24.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.184+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 shutdown_connections 2023-10-30T13:24:24.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.184+0000 7f2343417640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23200664d0 0x7f2320068990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:24.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.184+0000 7f2343417640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f233c14fd50 0x7f233c154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:24.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.184+0000 7f2343417640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f233c065f10 0x7f233c1541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:24.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.184+0000 7f2343417640 1 --2- 172.21.15.148:0/905235589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f233c065a80 0x7f233c140e10 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:24.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.184+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 >> 172.21.15.148:0/905235589 conn(0x7f233c06c6e0 msgr2=0x7f233c071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:24.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.184+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 shutdown_connections 2023-10-30T13:24:24.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:24.184+0000 7f2343417640 1 -- 172.21.15.148:0/905235589 wait complete. 2023-10-30T13:24:24.199 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:24:25.502 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:25.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:26.201 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:26.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.352+0000 7fd684a91640 1 Processor -- start 2023-10-30T13:24:26.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.352+0000 7fd684a91640 1 -- start start 2023-10-30T13:24:26.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd684a91640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd680153bf0 0x7fd680071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:26.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd684a91640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6800c5f60 0x7fd6800cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:26.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd684a91640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6800700b0 con 0x7fd6800c6fe0 2023-10-30T13:24:26.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd684a91640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6800c6810 con 0x7fd6800c5f60 2023-10-30T13:24:26.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd684a91640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd680159900 con 0x7fd680153bf0 2023-10-30T13:24:26.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd67e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd680153bf0 0x7fd680071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:26.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd67ed9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6800c5f60 0x7fd6800cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:26.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd67e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd680153bf0 0x7fd680071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:53208/0 (socket says 172.21.15.148:53208) 2023-10-30T13:24:26.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd67e59c640 1 -- 172.21.15.148:0/468765937 learned_addr learned my addr 172.21.15.148:0/468765937 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:26.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd677fff640 1 --1- 172.21.15.148:0/468765937 >> v1:172.21.15.150:6789/0 conn(0x7fd6800c6fe0 0x7fd6800c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:55740/0 (socket says 172.21.15.148:55740) 2023-10-30T13:24:26.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd67ed9d640 1 -- 172.21.15.148:0/468765937 >> v2:172.21.15.150:3300/0 conn(0x7fd680153bf0 msgr2=0x7fd680071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:26.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd67ed9d640 1 --2- 172.21.15.148:0/468765937 >> v2:172.21.15.150:3300/0 conn(0x7fd680153bf0 0x7fd680071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd67ed9d640 1 -- 172.21.15.148:0/468765937 >> v1:172.21.15.150:6789/0 conn(0x7fd6800c6fe0 legacy=0x7fd6800c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:26.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.356+0000 7fd67ed9d640 1 -- 172.21.15.148:0/468765937 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6800c6530 con 0x7fd6800c5f60 2023-10-30T13:24:26.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd67ed9d640 1 --2- 172.21.15.148:0/468765937 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6800c5f60 0x7fd6800cb570 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd6800c9150 tx=0x7fd67801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241fcf5eb84ff52b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:26.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd67dd9b640 1 -- 172.21.15.148:0/468765937 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6780b0440 con 0x7fd6800c5f60 2023-10-30T13:24:26.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd67dd9b640 1 -- 172.21.15.148:0/468765937 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6780b05a0 con 0x7fd6800c5f60 2023-10-30T13:24:26.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd67dd9b640 1 -- 172.21.15.148:0/468765937 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6780b0930 con 0x7fd6800c5f60 2023-10-30T13:24:26.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 -- 172.21.15.148:0/468765937 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6800c5f60 msgr2=0x7fd6800cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:26.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 --2- 172.21.15.148:0/468765937 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6800c5f60 0x7fd6800cb570 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd6800c9150 tx=0x7fd67801f0f0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 -- 172.21.15.148:0/468765937 shutdown_connections 2023-10-30T13:24:26.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 --2- 172.21.15.148:0/468765937 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6800c5f60 0x7fd6800cb570 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 --2- 172.21.15.148:0/468765937 >> v2:172.21.15.150:3300/0 conn(0x7fd680153bf0 0x7fd680071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 -- 172.21.15.148:0/468765937 >> 172.21.15.148:0/468765937 conn(0x7fd68006c6e0 msgr2=0x7fd680069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:26.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 -- 172.21.15.148:0/468765937 shutdown_connections 2023-10-30T13:24:26.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 -- 172.21.15.148:0/468765937 wait complete. 2023-10-30T13:24:26.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 Processor -- start 2023-10-30T13:24:26.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.360+0000 7fd684a91640 1 -- start start 2023-10-30T13:24:26.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd684a91640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6800c5f60 0x7fd6801ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:26.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd684a91640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6800c6fe0 0x7fd6800c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:26.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd684a91640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd680153bf0 0x7fd68014ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:26.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd684a91640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68014c5f0 con 0x7fd680153bf0 2023-10-30T13:24:26.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd684a91640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68014d6f0 con 0x7fd6800c6fe0 2023-10-30T13:24:26.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd684a91640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd680154040 con 0x7fd6800c5f60 2023-10-30T13:24:26.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd680153bf0 0x7fd68014ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:26.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd680153bf0 0x7fd68014ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57674/0 (socket says 172.21.15.148:57674) 2023-10-30T13:24:26.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67ed9d640 1 -- 172.21.15.148:0/1695815517 learned_addr learned my addr 172.21.15.148:0/1695815517 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:26.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd677fff640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6800c6fe0 0x7fd6800c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:26.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67e59c640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6800c5f60 0x7fd6801ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:26.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67ed9d640 1 -- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6800c5f60 msgr2=0x7fd6801ee7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:26.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67ed9d640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6800c5f60 0x7fd6801ee7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67ed9d640 1 -- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6800c6fe0 msgr2=0x7fd6800c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:26.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67ed9d640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6800c6fe0 0x7fd6800c4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67ed9d640 1 -- 172.21.15.148:0/1695815517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678009a70 con 0x7fd680153bf0 2023-10-30T13:24:26.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd67ed9d640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd680153bf0 0x7fd68014ba30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd67801bd00 tx=0x7fd67801be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:26.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd676ffd640 1 -- 172.21.15.148:0/1695815517 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6780b0440 con 0x7fd680153bf0 2023-10-30T13:24:26.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd676ffd640 1 -- 172.21.15.148:0/1695815517 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6780bc860 con 0x7fd680153bf0 2023-10-30T13:24:26.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68014c820 con 0x7fd680153bf0 2023-10-30T13:24:26.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd676ffd640 1 -- 172.21.15.148:0/1695815517 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6780bcc10 con 0x7fd680153bf0 2023-10-30T13:24:26.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68014cdc0 con 0x7fd680153bf0 2023-10-30T13:24:26.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68014d860 con 0x7fd680153bf0 2023-10-30T13:24:26.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.364+0000 7fd676ffd640 1 -- 172.21.15.148:0/1695815517 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6780c7460 con 0x7fd680153bf0 2023-10-30T13:24:26.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.368+0000 7fd676ffd640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6600664a0 0x7fd660068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:26.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.368+0000 7fd676ffd640 1 -- 172.21.15.148:0/1695815517 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fd678122eb0 con 0x7fd680153bf0 2023-10-30T13:24:26.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.368+0000 7fd67e59c640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6600664a0 0x7fd660068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:26.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.368+0000 7fd67e59c640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6600664a0 0x7fd660068960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd670009a40 tx=0x7fd670008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:26.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.372+0000 7fd676ffd640 1 -- 172.21.15.148:0/1695815517 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6780f3a90 con 0x7fd680153bf0 2023-10-30T13:24:26.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.560+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd68004e170 con 0x7fd680153bf0 2023-10-30T13:24:26.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.560+0000 7fd676ffd640 1 -- 172.21.15.148:0/1695815517 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fd6780b5020 con 0x7fd680153bf0 2023-10-30T13:24:26.558 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:26.559 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:26.559 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 69 2023-10-30T13:24:26.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6600664a0 msgr2=0x7fd660068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:26.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6600664a0 0x7fd660068960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd670009a40 tx=0x7fd670008040 comp rx=0 tx=0).stop 2023-10-30T13:24:26.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd680153bf0 msgr2=0x7fd68014ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:26.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd680153bf0 0x7fd68014ba30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd67801bd00 tx=0x7fd67801be90 comp rx=0 tx=0).stop 2023-10-30T13:24:26.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 shutdown_connections 2023-10-30T13:24:26.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6600664a0 0x7fd660068960 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd680153bf0 0x7fd68014ba30 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6800c6fe0 0x7fd6800c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 --2- 172.21.15.148:0/1695815517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6800c5f60 0x7fd6801ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:26.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 >> 172.21.15.148:0/1695815517 conn(0x7fd68006c6e0 msgr2=0x7fd68006e140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:26.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 shutdown_connections 2023-10-30T13:24:26.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:26.564+0000 7fd684a91640 1 -- 172.21.15.148:0/1695815517 wait complete. 2023-10-30T13:24:26.580 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:24:28.581 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:28.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.732+0000 7fe912e68640 1 Processor -- start 2023-10-30T13:24:28.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.732+0000 7fe912e68640 1 -- start start 2023-10-30T13:24:28.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe912e68640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe90c1538a0 0x7fe90c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe912e68640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0cf160 0x7fe90c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe912e68640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90c0cb800 con 0x7fe90c0ca0e0 2023-10-30T13:24:28.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe912e68640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90c15dc30 con 0x7fe90c1538a0 2023-10-30T13:24:28.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe912e68640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0cb330 con 0x7fe90c0cf160 2023-10-30T13:24:28.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe910c04640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe90c1538a0 0x7fe90c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe903fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe90c0ca0e0 0x7fe90c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:55748/0 (socket says 172.21.15.148:55748) 2023-10-30T13:24:28.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe903fff640 1 -- 172.21.15.148:0/1410240282 learned_addr learned my addr 172.21.15.148:0/1410240282 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:24:28.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe911405640 1 --2- 172.21.15.148:0/1410240282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0cf160 0x7fe90c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe9037fe640 1 -- 172.21.15.148:0/1410240282 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790294544 0 0) 0x7fe90c0cb800 con 0x7fe90c0ca0e0 2023-10-30T13:24:28.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe9037fe640 1 -- 172.21.15.148:0/1410240282 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f4003530 con 0x7fe90c0ca0e0 2023-10-30T13:24:28.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe910c04640 1 -- 172.21.15.148:0/1410240282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0cf160 msgr2=0x7fe90c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:28.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe910c04640 1 --2- 172.21.15.148:0/1410240282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0cf160 0x7fe90c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe910c04640 1 -- 172.21.15.148:0/1410240282 >> v1:172.21.15.150:6789/0 conn(0x7fe90c0ca0e0 legacy=0x7fe90c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.736+0000 7fe910c04640 1 -- 172.21.15.148:0/1410240282 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c0cad70 con 0x7fe90c1538a0 2023-10-30T13:24:28.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe910c04640 1 --2- 172.21.15.148:0/1410240282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe90c1538a0 0x7fe90c15d6f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe9080097c0 tx=0x7fe90801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbcf61f2e9147714 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:28.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe9037fe640 1 -- 172.21.15.148:0/1410240282 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9080a8970 con 0x7fe90c1538a0 2023-10-30T13:24:28.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe9037fe640 1 -- 172.21.15.148:0/1410240282 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9080a8ad0 con 0x7fe90c1538a0 2023-10-30T13:24:28.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe9037fe640 1 -- 172.21.15.148:0/1410240282 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9080a8e60 con 0x7fe90c1538a0 2023-10-30T13:24:28.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe912e68640 1 -- 172.21.15.148:0/1410240282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe90c1538a0 msgr2=0x7fe90c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe912e68640 1 --2- 172.21.15.148:0/1410240282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe90c1538a0 0x7fe90c15d6f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe9080097c0 tx=0x7fe90801ec70 comp rx=0 tx=0).stop 2023-10-30T13:24:28.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe912e68640 1 -- 172.21.15.148:0/1410240282 shutdown_connections 2023-10-30T13:24:28.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe912e68640 1 --2- 172.21.15.148:0/1410240282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0cf160 0x7fe90c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe912e68640 1 --2- 172.21.15.148:0/1410240282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe90c1538a0 0x7fe90c15d6f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe912e68640 1 -- 172.21.15.148:0/1410240282 >> 172.21.15.148:0/1410240282 conn(0x7fe90c06bfe0 msgr2=0x7fe90c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:28.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe912e68640 1 -- 172.21.15.148:0/1410240282 shutdown_connections 2023-10-30T13:24:28.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.740+0000 7fe912e68640 1 -- 172.21.15.148:0/1410240282 wait complete. 2023-10-30T13:24:28.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 Processor -- start 2023-10-30T13:24:28.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 -- start start 2023-10-30T13:24:28.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0ca0e0 0x7fe90c1f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe90c0cf160 0x7fe90c14ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe90c1538a0 0x7fe90c14d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0b3380 con 0x7fe90c1538a0 2023-10-30T13:24:28.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c15b270 con 0x7fe90c0cf160 2023-10-30T13:24:28.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0b3620 con 0x7fe90c0ca0e0 2023-10-30T13:24:28.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe911405640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe90c1538a0 0x7fe90c14d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe911405640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe90c1538a0 0x7fe90c14d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57678/0 (socket says 172.21.15.148:57678) 2023-10-30T13:24:28.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe911405640 1 -- 172.21.15.148:0/2403877280 learned_addr learned my addr 172.21.15.148:0/2403877280 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:28.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe903fff640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe90c0cf160 0x7fe90c14ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe910c04640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0ca0e0 0x7fe90c1f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe911405640 1 -- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0ca0e0 msgr2=0x7fe90c1f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe911405640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0ca0e0 0x7fe90c1f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe911405640 1 -- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe90c0cf160 msgr2=0x7fe90c14ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe911405640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe90c0cf160 0x7fe90c14ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe911405640 1 -- 172.21.15.148:0/2403877280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe908009420 con 0x7fe90c1538a0 2023-10-30T13:24:28.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe911405640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe90c1538a0 0x7fe90c14d200 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe90400efc0 tx=0x7fe90400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:28.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe9017fa640 1 -- 172.21.15.148:0/2403877280 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe904009b10 con 0x7fe90c1538a0 2023-10-30T13:24:28.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe9017fa640 1 -- 172.21.15.148:0/2403877280 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90400f040 con 0x7fe90c1538a0 2023-10-30T13:24:28.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe9017fa640 1 -- 172.21.15.148:0/2403877280 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9040146c0 con 0x7fe90c1538a0 2023-10-30T13:24:28.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c0b4970 con 0x7fe90c1538a0 2023-10-30T13:24:28.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c1f3200 con 0x7fe90c1538a0 2023-10-30T13:24:28.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.744+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c0b2a50 con 0x7fe90c1538a0 2023-10-30T13:24:28.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.748+0000 7fe9017fa640 1 -- 172.21.15.148:0/2403877280 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe904014820 con 0x7fe90c1538a0 2023-10-30T13:24:28.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.748+0000 7fe9017fa640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8f80664a0 0x7fe8f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.748+0000 7fe9017fa640 1 -- 172.21.15.148:0/2403877280 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6370+0+0 (secure 0 0 0) 0x7fe904084b80 con 0x7fe90c1538a0 2023-10-30T13:24:28.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.748+0000 7fe910c04640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8f80664a0 0x7fe8f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.748+0000 7fe910c04640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8f80664a0 0x7fe8f8068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe9080b2000 tx=0x7fe908008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:28.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.752+0000 7fe9017fa640 1 -- 172.21.15.148:0/2403877280 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe904051d70 con 0x7fe90c1538a0 2023-10-30T13:24:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe90c04e170 con 0x7fe90c1538a0 2023-10-30T13:24:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe9017fa640 1 -- 172.21.15.148:0/2403877280 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fe904084e60 con 0x7fe90c1538a0 2023-10-30T13:24:28.933 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:28.933 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":5955,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:23:52.910931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5643},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5643":{"gid":5643,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":44,"addr":"172.21.15.148:6835/3194894584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3194894584},{"type":"v1","addr":"172.21.15.148:6835","nonce":3194894584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:28.935 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 69 2023-10-30T13:24:28.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8f80664a0 msgr2=0x7fe8f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8f80664a0 0x7fe8f8068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe9080b2000 tx=0x7fe908008040 comp rx=0 tx=0).stop 2023-10-30T13:24:28.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe90c1538a0 msgr2=0x7fe90c14d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe90c1538a0 0x7fe90c14d200 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe90400efc0 tx=0x7fe90400c490 comp rx=0 tx=0).stop 2023-10-30T13:24:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 shutdown_connections 2023-10-30T13:24:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8f80664a0 0x7fe8f8068960 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe90c1538a0 0x7fe90c14d200 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe90c0cf160 0x7fe90c14ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 --2- 172.21.15.148:0/2403877280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe90c0ca0e0 0x7fe90c1f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 >> 172.21.15.148:0/2403877280 conn(0x7fe90c06bfe0 msgr2=0x7fe90c1e2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 shutdown_connections 2023-10-30T13:24:28.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:28.936+0000 7fe912e68640 1 -- 172.21.15.148:0/2403877280 wait complete. 2023-10-30T13:24:28.954 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:24:30.955 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.104+0000 7f255d53e640 1 Processor -- start 2023-10-30T13:24:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.104+0000 7f255d53e640 1 -- start start 2023-10-30T13:24:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f255d53e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25581538a0 0x7f255815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f255d53e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f25580cf160 0x7f25580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f255d53e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25580cb800 con 0x7f25580ca0e0 2023-10-30T13:24:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f255d53e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255815dc30 con 0x7f25580cf160 2023-10-30T13:24:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f255d53e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25580cb330 con 0x7f25581538a0 2023-10-30T13:24:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f254e7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f25580ca0e0 0x7f255815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36584/0 (socket says 172.21.15.148:36584) 2023-10-30T13:24:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f254e7fc640 1 -- 172.21.15.148:0/811704279 learned_addr learned my addr 172.21.15.148:0/811704279 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:24:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25577fe640 1 --2- 172.21.15.148:0/811704279 >> v2:172.21.15.150:3300/0 conn(0x7f25580cf160 0x7f25580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f2556ffd640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25581538a0 0x7f255815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550483165 0 0) 0x7f25580cb800 con 0x7f25580ca0e0 2023-10-30T13:24:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2544003530 con 0x7f25580ca0e0 2023-10-30T13:24:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2450485168 0 0) 0x7f2544003530 con 0x7f25580ca0e0 2023-10-30T13:24:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25581538a0 msgr2=0x7f255815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25581538a0 0x7f255815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 >> v2:172.21.15.150:3300/0 conn(0x7f25580cf160 msgr2=0x7f25580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 --2- 172.21.15.148:0/811704279 >> v2:172.21.15.150:3300/0 conn(0x7f25580cf160 0x7f25580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580cad70 con 0x7f25580ca0e0 2023-10-30T13:24:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f2548001680 con 0x7f25580ca0e0 2023-10-30T13:24:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 >> v1:172.21.15.148:6789/0 conn(0x7f25580ca0e0 legacy=0x7f255815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2544003980 0x7f2544003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25440047d0 0x7f2544004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25581538a0 0x7f2544005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255815dc30 con 0x7f25440047d0 2023-10-30T13:24:31.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25580cb330 con 0x7f2544003980 2023-10-30T13:24:31.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254400b990 con 0x7f25581538a0 2023-10-30T13:24:31.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f254e7fc640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25440047d0 0x7f2544004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f2556ffd640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2544003980 0x7f2544003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f25577fe640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25581538a0 0x7f2544005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f254e7fc640 1 -- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25581538a0 msgr2=0x7f2544005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f254e7fc640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25581538a0 0x7f2544005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f254e7fc640 1 -- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2544003980 msgr2=0x7f2544003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:31.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f254e7fc640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2544003980 0x7f2544003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.108+0000 7f254e7fc640 1 -- 172.21.15.148:0/811704279 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2544003530 con 0x7f25440047d0 2023-10-30T13:24:31.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f254e7fc640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25440047d0 0x7f2544004c50 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f254800c240 tx=0x7f254801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc605f72c4b63ea9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:31.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25480058e0 con 0x7f25440047d0 2023-10-30T13:24:31.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2548005a40 con 0x7f25440047d0 2023-10-30T13:24:31.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f25567fc640 1 -- 172.21.15.148:0/811704279 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548005e10 con 0x7f25440047d0 2023-10-30T13:24:31.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f255d53e640 1 -- 172.21.15.148:0/811704279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25440047d0 msgr2=0x7f2544004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f255d53e640 1 --2- 172.21.15.148:0/811704279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25440047d0 0x7f2544004c50 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f254800c240 tx=0x7f254801cc60 comp rx=0 tx=0).stop 2023-10-30T13:24:31.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f25577fe640 1 -- 172.21.15.148:0/811704279 reap_dead start 2023-10-30T13:24:31.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f255d53e640 1 -- 172.21.15.148:0/811704279 shutdown_connections 2023-10-30T13:24:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f255d53e640 1 -- 172.21.15.148:0/811704279 >> 172.21.15.148:0/811704279 conn(0x7f255806bfe0 msgr2=0x7f255806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f255d53e640 1 -- 172.21.15.148:0/811704279 shutdown_connections 2023-10-30T13:24:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f255d53e640 1 -- 172.21.15.148:0/811704279 wait complete. 2023-10-30T13:24:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f255d53e640 1 Processor -- start 2023-10-30T13:24:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.112+0000 7f255d53e640 1 -- start start 2023-10-30T13:24:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f255d53e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25440047d0 0x7f25580cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f255d53e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25580ce4b0 0x7f2558147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f255d53e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25581593d0 0x7f25580b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f255d53e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25580cd840 con 0x7f25581593d0 2023-10-30T13:24:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f255d53e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255815cc50 con 0x7f25440047d0 2023-10-30T13:24:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f255d53e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25580b1960 con 0x7f25580ce4b0 2023-10-30T13:24:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f25577fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25581593d0 0x7f25580b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f25577fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25581593d0 0x7f25580b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57700/0 (socket says 172.21.15.148:57700) 2023-10-30T13:24:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f25577fe640 1 -- 172.21.15.148:0/3993729719 learned_addr learned my addr 172.21.15.148:0/3993729719 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f2556ffd640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25440047d0 0x7f25580cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f254e7fc640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25580ce4b0 0x7f2558147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f25577fe640 1 -- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25580ce4b0 msgr2=0x7f2558147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f25577fe640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25580ce4b0 0x7f2558147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f25577fe640 1 -- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25440047d0 msgr2=0x7f25580cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f25577fe640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25440047d0 0x7f25580cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f25577fe640 1 -- 172.21.15.148:0/3993729719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580ce1e0 con 0x7f25581593d0 2023-10-30T13:24:31.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f2556ffd640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25440047d0 0x7f25580cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:31.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f25577fe640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25581593d0 0x7f25580b31a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f25580cfd10 tx=0x7f255000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:31.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f254ffff640 1 -- 172.21.15.148:0/3993729719 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255000e040 con 0x7f25581593d0 2023-10-30T13:24:31.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f254ffff640 1 -- 172.21.15.148:0/3993729719 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255000f040 con 0x7f25581593d0 2023-10-30T13:24:31.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f254ffff640 1 -- 172.21.15.148:0/3993729719 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550015430 con 0x7f25581593d0 2023-10-30T13:24:31.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25580b1bb0 con 0x7f25581593d0 2023-10-30T13:24:31.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25580ceb70 con 0x7f25581593d0 2023-10-30T13:24:31.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.116+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25580ce9d0 con 0x7f25581593d0 2023-10-30T13:24:31.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.120+0000 7f254ffff640 1 -- 172.21.15.148:0/3993729719 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25500157a0 con 0x7f25581593d0 2023-10-30T13:24:31.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.120+0000 7f254ffff640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25380664d0 0x7f2538068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.120+0000 7f254ffff640 1 -- 172.21.15.148:0/3993729719 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2550084a90 con 0x7f25581593d0 2023-10-30T13:24:31.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.120+0000 7f2556ffd640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25380664d0 0x7f2538068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.120+0000 7f2556ffd640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25380664d0 0x7f2538068990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2540009a80 tx=0x7f2540008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:31.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.124+0000 7f254ffff640 1 -- 172.21.15.148:0/3993729719 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25580ce9d0 con 0x7f25581593d0 2023-10-30T13:24:31.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.304+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25580ce9d0 con 0x7f25581593d0 2023-10-30T13:24:31.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.304+0000 7f254ffff640 1 -- 172.21.15.148:0/3993729719 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f25580ce9d0 con 0x7f25581593d0 2023-10-30T13:24:31.303 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:31.303 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:29.547377+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:31.303 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 70 2023-10-30T13:24:31.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25380664d0 msgr2=0x7f2538068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25380664d0 0x7f2538068990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2540009a80 tx=0x7f2540008040 comp rx=0 tx=0).stop 2023-10-30T13:24:31.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25581593d0 msgr2=0x7f25580b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25581593d0 0x7f25580b31a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f25580cfd10 tx=0x7f255000f580 comp rx=0 tx=0).stop 2023-10-30T13:24:31.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 shutdown_connections 2023-10-30T13:24:31.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f25380664d0 0x7f2538068990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25581593d0 0x7f25580b31a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25580ce4b0 0x7f2558147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 --2- 172.21.15.148:0/3993729719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25440047d0 0x7f25580cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 >> 172.21.15.148:0/3993729719 conn(0x7f255806bfe0 msgr2=0x7f2558152550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:31.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 shutdown_connections 2023-10-30T13:24:31.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.308+0000 7f255d53e640 1 -- 172.21.15.148:0/3993729719 wait complete. 2023-10-30T13:24:31.324 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:24:31.325 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:24:31.325 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:31.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86ec05640 1 Processor -- start 2023-10-30T13:24:31.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86ec05640 1 -- start start 2023-10-30T13:24:31.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86ec05640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c05d0 0x7fc86813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86ec05640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc8680c6f60 0x7fc86813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86ec05640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8680c0a50 con 0x7fc8680c6b80 2023-10-30T13:24:31.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86ec05640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86806f6e0 con 0x7fc8680c6f60 2023-10-30T13:24:31.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86ec05640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86806f9c0 con 0x7fc8680c05d0 2023-10-30T13:24:31.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86d1a2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc8680c6f60 0x7fc86813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86d1a2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc8680c6f60 0x7fc86813f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57704/0 (socket says 172.21.15.148:57704) 2023-10-30T13:24:31.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86d1a2640 1 -- 172.21.15.148:0/3223127347 learned_addr learned my addr 172.21.15.148:0/3223127347 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:31.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc85ffff640 1 --2- 172.21.15.148:0/3223127347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c05d0 0x7fc86813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86d1a2640 1 -- 172.21.15.148:0/3223127347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c05d0 msgr2=0x7fc86813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:31.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86d1a2640 1 --2- 172.21.15.148:0/3223127347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c05d0 0x7fc86813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86d1a2640 1 -- 172.21.15.148:0/3223127347 >> v1:172.21.15.150:6789/0 conn(0x7fc8680c6b80 legacy=0x7fc8680bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.468+0000 7fc86d1a2640 1 -- 172.21.15.148:0/3223127347 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86806fca0 con 0x7fc8680c6f60 2023-10-30T13:24:31.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86d1a2640 1 --2- 172.21.15.148:0/3223127347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c6f60 0x7fc86813f2d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc860009e50 tx=0x7fc86001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea4963ed477755c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:31.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc85f7fe640 1 -- 172.21.15.148:0/3223127347 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8600a57f0 con 0x7fc8680c6f60 2023-10-30T13:24:31.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc85f7fe640 1 -- 172.21.15.148:0/3223127347 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8600a5950 con 0x7fc8680c6f60 2023-10-30T13:24:31.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc85f7fe640 1 -- 172.21.15.148:0/3223127347 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8600a5ce0 con 0x7fc8680c6f60 2023-10-30T13:24:31.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 -- 172.21.15.148:0/3223127347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c6f60 msgr2=0x7fc86813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 --2- 172.21.15.148:0/3223127347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c6f60 0x7fc86813f2d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc860009e50 tx=0x7fc86001ca10 comp rx=0 tx=0).stop 2023-10-30T13:24:31.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 -- 172.21.15.148:0/3223127347 shutdown_connections 2023-10-30T13:24:31.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 --2- 172.21.15.148:0/3223127347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c6f60 0x7fc86813f2d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 --2- 172.21.15.148:0/3223127347 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c05d0 0x7fc86813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 -- 172.21.15.148:0/3223127347 >> 172.21.15.148:0/3223127347 conn(0x7fc8680c7910 msgr2=0x7fc8680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:31.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 -- 172.21.15.148:0/3223127347 shutdown_connections 2023-10-30T13:24:31.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 -- 172.21.15.148:0/3223127347 wait complete. 2023-10-30T13:24:31.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 Processor -- start 2023-10-30T13:24:31.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.472+0000 7fc86ec05640 1 -- start start 2023-10-30T13:24:31.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86ec05640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c05d0 0x7fc8681501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86ec05640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c6b80 0x7fc868150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86ec05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680c6f60 0x7fc8680c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86ec05640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8680c92d0 con 0x7fc8680c05d0 2023-10-30T13:24:31.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86ec05640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc868160df0 con 0x7fc8680c6f60 2023-10-30T13:24:31.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86ec05640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8681eecb0 con 0x7fc8680c6b80 2023-10-30T13:24:31.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86c9a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c05d0 0x7fc8681501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86c9a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c05d0 0x7fc8681501c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57718/0 (socket says 172.21.15.148:57718) 2023-10-30T13:24:31.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86c9a1640 1 -- 172.21.15.148:0/908964659 learned_addr learned my addr 172.21.15.148:0/908964659 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:31.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc85ffff640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c6b80 0x7fc868150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86d1a2640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680c6f60 0x7fc8680c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86c9a1640 1 -- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c6b80 msgr2=0x7fc868150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86c9a1640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c6b80 0x7fc868150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86c9a1640 1 -- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680c6f60 msgr2=0x7fc8680c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86c9a1640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680c6f60 0x7fc8680c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86c9a1640 1 -- 172.21.15.148:0/908964659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc860009b30 con 0x7fc8680c05d0 2023-10-30T13:24:31.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86d1a2640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680c6f60 0x7fc8680c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:31.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86c9a1640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c05d0 0x7fc8681501c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc86400c500 tx=0x7fc86400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:31.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc85d7fa640 1 -- 172.21.15.148:0/908964659 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864005220 con 0x7fc8680c05d0 2023-10-30T13:24:31.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc85d7fa640 1 -- 172.21.15.148:0/908964659 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc864005380 con 0x7fc8680c05d0 2023-10-30T13:24:31.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc85d7fa640 1 -- 172.21.15.148:0/908964659 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864007270 con 0x7fc8680c05d0 2023-10-30T13:24:31.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc868134130 con 0x7fc8680c05d0 2023-10-30T13:24:31.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8681343a0 con 0x7fc8680c05d0 2023-10-30T13:24:31.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8680c1180 con 0x7fc8680c05d0 2023-10-30T13:24:31.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.476+0000 7fc85d7fa640 1 -- 172.21.15.148:0/908964659 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8640073d0 con 0x7fc8680c05d0 2023-10-30T13:24:31.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.480+0000 7fc85d7fa640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc854066520 0x7fc8540689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:31.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.480+0000 7fc85d7fa640 1 -- 172.21.15.148:0/908964659 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fc864083490 con 0x7fc8680c05d0 2023-10-30T13:24:31.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.480+0000 7fc85ffff640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc854066520 0x7fc8540689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:31.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.480+0000 7fc85ffff640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc854066520 0x7fc8540689e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc8680c20f0 tx=0x7fc858008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:31.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.484+0000 7fc85d7fa640 1 -- 172.21.15.148:0/908964659 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc864050610 con 0x7fc8680c05d0 2023-10-30T13:24:31.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.660+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc86804e170 con 0x7fc8680c05d0 2023-10-30T13:24:31.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.660+0000 7fc85d7fa640 1 -- 172.21.15.148:0/908964659 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5716 (secure 0 0 0) 0x7fc864053c30 con 0x7fc8680c05d0 2023-10-30T13:24:31.659 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:31.659 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:29.547377+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:31.660 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 70 2023-10-30T13:24:31.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc854066520 msgr2=0x7fc8540689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc854066520 0x7fc8540689e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc8680c20f0 tx=0x7fc858008040 comp rx=0 tx=0).stop 2023-10-30T13:24:31.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c05d0 msgr2=0x7fc8681501c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:31.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c05d0 0x7fc8681501c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc86400c500 tx=0x7fc86400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 shutdown_connections 2023-10-30T13:24:31.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc854066520 0x7fc8540689e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680c6f60 0x7fc8680c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc8680c6b80 0x7fc868150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 --2- 172.21.15.148:0/908964659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8680c05d0 0x7fc8681501c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:31.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 >> 172.21.15.148:0/908964659 conn(0x7fc8680c7910 msgr2=0x7fc8681352c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:31.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 shutdown_connections 2023-10-30T13:24:31.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:31.664+0000 7fc86ec05640 1 -- 172.21.15.148:0/908964659 wait complete. 2023-10-30T13:24:31.680 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 70, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 70, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5955": { "addr": "172.21.15.148:6837/1559416001", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 1559416001, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 1559416001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5955, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:24:29.547377+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5955 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:24:33.682 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:33.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.836+0000 7fbdf5d0f640 1 Processor -- start 2023-10-30T13:24:33.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.836+0000 7fbdf5d0f640 1 -- start start 2023-10-30T13:24:33.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.836+0000 7fbdf5d0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf0153d70 0x7fbdf006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:33.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.836+0000 7fbdf5d0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbdf00c8490 0x7fbdf00c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:33.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.836+0000 7fbdf5d0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf006afa0 con 0x7fbdf006e470 2023-10-30T13:24:33.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.836+0000 7fbdf5d0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf006d780 con 0x7fbdf00c8490 2023-10-30T13:24:33.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.836+0000 7fbdf5d0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0154150 con 0x7fbdf0153d70 2023-10-30T13:24:33.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdef7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf0153d70 0x7fbdf006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:33.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdeffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbdf00c8490 0x7fbdf00c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:33.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdeffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbdf00c8490 0x7fbdf00c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:57334/0 (socket says 172.21.15.148:57334) 2023-10-30T13:24:33.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdeffff640 1 -- 172.21.15.148:0/1925371863 learned_addr learned my addr 172.21.15.148:0/1925371863 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:33.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdeeffd640 1 --1- 172.21.15.148:0/1925371863 >> v1:172.21.15.150:6789/0 conn(0x7fbdf006e470 0x7fbdf00cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:51734/0 (socket says 172.21.15.148:51734) 2023-10-30T13:24:33.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdef7fe640 1 -- 172.21.15.148:0/1925371863 >> v2:172.21.15.150:3300/0 conn(0x7fbdf00c8490 msgr2=0x7fbdf00c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:33.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdef7fe640 1 --2- 172.21.15.148:0/1925371863 >> v2:172.21.15.150:3300/0 conn(0x7fbdf00c8490 0x7fbdf00c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:33.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdef7fe640 1 -- 172.21.15.148:0/1925371863 >> v1:172.21.15.150:6789/0 conn(0x7fbdf006e470 legacy=0x7fbdf00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:33.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdee7fc640 1 -- 172.21.15.148:0/1925371863 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294652210 0 0) 0x7fbdf006afa0 con 0x7fbdf006e470 2023-10-30T13:24:33.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdef7fe640 1 -- 172.21.15.148:0/1925371863 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf006e030 con 0x7fbdf0153d70 2023-10-30T13:24:33.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.840+0000 7fbdeffff640 1 --2- 172.21.15.148:0/1925371863 >> v2:172.21.15.150:3300/0 conn(0x7fbdf00c8490 0x7fbdf00c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:24:33.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdef7fe640 1 --2- 172.21.15.148:0/1925371863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf0153d70 0x7fbdf006ca80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbde00097c0 tx=0x7fbde001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1149bec13bee7bd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:33.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdee7fc640 1 -- 172.21.15.148:0/1925371863 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00ab650 con 0x7fbdf0153d70 2023-10-30T13:24:33.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdee7fc640 1 -- 172.21.15.148:0/1925371863 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde00ab7b0 con 0x7fbdf0153d70 2023-10-30T13:24:33.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdee7fc640 1 -- 172.21.15.148:0/1925371863 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00abb40 con 0x7fbdf0153d70 2023-10-30T13:24:33.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/1925371863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf0153d70 msgr2=0x7fbdf006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:33.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 --2- 172.21.15.148:0/1925371863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf0153d70 0x7fbdf006ca80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbde00097c0 tx=0x7fbde001c450 comp rx=0 tx=0).stop 2023-10-30T13:24:33.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/1925371863 shutdown_connections 2023-10-30T13:24:33.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 --2- 172.21.15.148:0/1925371863 >> v2:172.21.15.150:3300/0 conn(0x7fbdf00c8490 0x7fbdf00c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:33.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 --2- 172.21.15.148:0/1925371863 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf0153d70 0x7fbdf006ca80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:33.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/1925371863 >> 172.21.15.148:0/1925371863 conn(0x7fbdf0071ea0 msgr2=0x7fbdf0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:33.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/1925371863 shutdown_connections 2023-10-30T13:24:33.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/1925371863 wait complete. 2023-10-30T13:24:33.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 Processor -- start 2023-10-30T13:24:33.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.844+0000 7fbdf5d0f640 1 -- start start 2023-10-30T13:24:33.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdf5d0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf006e470 0x7fbdf0152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:33.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdf5d0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf00c8490 0x7fbdf01f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:33.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdf5d0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf0153d70 0x7fbdf01f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:33.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdf5d0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00cc670 con 0x7fbdf00c8490 2023-10-30T13:24:33.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdf5d0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0154990 con 0x7fbdf0153d70 2023-10-30T13:24:33.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdf5d0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0155a50 con 0x7fbdf006e470 2023-10-30T13:24:33.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeeffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf00c8490 0x7fbdf01f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:33.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeeffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf00c8490 0x7fbdf01f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36762/0 (socket says 172.21.15.148:36762) 2023-10-30T13:24:33.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeeffd640 1 -- 172.21.15.148:0/807662334 learned_addr learned my addr 172.21.15.148:0/807662334 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:33.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdef7fe640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf006e470 0x7fbdf0152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:33.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeffff640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf0153d70 0x7fbdf01f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:33.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeeffd640 1 -- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf006e470 msgr2=0x7fbdf0152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeeffd640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf006e470 0x7fbdf0152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeeffd640 1 -- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf0153d70 msgr2=0x7fbdf01f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeeffd640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf0153d70 0x7fbdf01f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeeffd640 1 -- 172.21.15.148:0/807662334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8009b30 con 0x7fbdf00c8490 2023-10-30T13:24:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdeeffd640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf00c8490 0x7fbdf01f2be0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fbde40011f0 tx=0x7fbde400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdd7fff640 1 -- 172.21.15.148:0/807662334 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde40074e0 con 0x7fbdf00c8490 2023-10-30T13:24:33.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdd7fff640 1 -- 172.21.15.148:0/807662334 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde4007640 con 0x7fbdf00c8490 2023-10-30T13:24:33.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdd7fff640 1 -- 172.21.15.148:0/807662334 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4004a10 con 0x7fbdf00c8490 2023-10-30T13:24:33.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde0009420 con 0x7fbdf00c8490 2023-10-30T13:24:33.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf0155770 con 0x7fbdf00c8490 2023-10-30T13:24:33.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.848+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf004e170 con 0x7fbdf00c8490 2023-10-30T13:24:33.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.852+0000 7fbdd7fff640 1 -- 172.21.15.148:0/807662334 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbde4004b70 con 0x7fbdf00c8490 2023-10-30T13:24:33.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.852+0000 7fbdd7fff640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbddc0664a0 0x7fbddc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:33.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.852+0000 7fbdd7fff640 1 -- 172.21.15.148:0/807662334 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fbde40834d0 con 0x7fbdf00c8490 2023-10-30T13:24:33.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.852+0000 7fbdef7fe640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbddc0664a0 0x7fbddc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:33.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.852+0000 7fbdef7fe640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbddc0664a0 0x7fbddc068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbde00af000 tx=0x7fbde00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:33.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:33.856+0000 7fbdd7fff640 1 -- 172.21.15.148:0/807662334 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbde40505e0 con 0x7fbdf00c8490 2023-10-30T13:24:34.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdf006c2a0 con 0x7fbdf00c8490 2023-10-30T13:24:34.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdd7fff640 1 -- 172.21.15.148:0/807662334 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5716 (secure 0 0 0) 0x7fbde4018020 con 0x7fbdf00c8490 2023-10-30T13:24:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:34.032 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:29.547377+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:34.033 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 70 2023-10-30T13:24:34.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbddc0664a0 msgr2=0x7fbddc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:34.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbddc0664a0 0x7fbddc068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbde00af000 tx=0x7fbde00a7000 comp rx=0 tx=0).stop 2023-10-30T13:24:34.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf00c8490 msgr2=0x7fbdf01f2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:34.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf00c8490 0x7fbdf01f2be0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fbde40011f0 tx=0x7fbde400c9b0 comp rx=0 tx=0).stop 2023-10-30T13:24:34.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 shutdown_connections 2023-10-30T13:24:34.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbddc0664a0 0x7fbddc068960 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:34.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbdf00c8490 0x7fbdf01f2be0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:34.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf0153d70 0x7fbdf01f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:34.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 --2- 172.21.15.148:0/807662334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbdf006e470 0x7fbdf0152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:34.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 >> 172.21.15.148:0/807662334 conn(0x7fbdf0071ea0 msgr2=0x7fbdf0150190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:34.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 shutdown_connections 2023-10-30T13:24:34.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:34.036+0000 7fbdf5d0f640 1 -- 172.21.15.148:0/807662334 wait complete. 2023-10-30T13:24:34.053 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:36.055 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:36.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.200+0000 7fd677b0f640 1 Processor -- start 2023-10-30T13:24:36.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.200+0000 7fd677b0f640 1 -- start start 2023-10-30T13:24:36.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd677b0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd67015d6f0 0x7fd67015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:36.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd677b0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6700cf160 0x7fd6700cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:36.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd677b0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6700cb800 con 0x7fd6701538a0 2023-10-30T13:24:36.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd677b0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6700cb330 con 0x7fd6700cf160 2023-10-30T13:24:36.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd677b0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6700cad70 con 0x7fd67015d6f0 2023-10-30T13:24:36.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd6758ab640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd6701538a0 0x7fd6700ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39736/0 (socket says 172.21.15.148:39736) 2023-10-30T13:24:36.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd6758ab640 1 -- 172.21.15.148:0/2331362360 learned_addr learned my addr 172.21.15.148:0/2331362360 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:24:36.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd6760ac640 1 --2- 172.21.15.148:0/2331362360 >> v2:172.21.15.148:3300/0 conn(0x7fd6700cf160 0x7fd6700cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:36.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd6750aa640 1 --2- 172.21.15.148:0/2331362360 >> v2:172.21.15.150:3300/0 conn(0x7fd67015d6f0 0x7fd67015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:36.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd6760ac640 1 -- 172.21.15.148:0/2331362360 >> v2:172.21.15.150:3300/0 conn(0x7fd67015d6f0 msgr2=0x7fd67015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:36.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd6760ac640 1 --2- 172.21.15.148:0/2331362360 >> v2:172.21.15.150:3300/0 conn(0x7fd67015d6f0 0x7fd67015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd6760ac640 1 -- 172.21.15.148:0/2331362360 >> v1:172.21.15.148:6789/0 conn(0x7fd6701538a0 legacy=0x7fd6700ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:36.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.204+0000 7fd6760ac640 1 -- 172.21.15.148:0/2331362360 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6700cb050 con 0x7fd6700cf160 2023-10-30T13:24:36.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd6760ac640 1 --2- 172.21.15.148:0/2331362360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6700cf160 0x7fd6700cf590 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd6700b30a0 tx=0x7fd66801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c419969dc77b7baa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:36.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd6748a9640 1 -- 172.21.15.148:0/2331362360 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6680b0440 con 0x7fd6700cf160 2023-10-30T13:24:36.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd6748a9640 1 -- 172.21.15.148:0/2331362360 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6680b05a0 con 0x7fd6700cf160 2023-10-30T13:24:36.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd6748a9640 1 -- 172.21.15.148:0/2331362360 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6680b0930 con 0x7fd6700cf160 2023-10-30T13:24:36.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 -- 172.21.15.148:0/2331362360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6700cf160 msgr2=0x7fd6700cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:36.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 --2- 172.21.15.148:0/2331362360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6700cf160 0x7fd6700cf590 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd6700b30a0 tx=0x7fd66801f0f0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 -- 172.21.15.148:0/2331362360 shutdown_connections 2023-10-30T13:24:36.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 --2- 172.21.15.148:0/2331362360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6700cf160 0x7fd6700cf590 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 --2- 172.21.15.148:0/2331362360 >> v2:172.21.15.150:3300/0 conn(0x7fd67015d6f0 0x7fd67015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 -- 172.21.15.148:0/2331362360 >> 172.21.15.148:0/2331362360 conn(0x7fd67006bfe0 msgr2=0x7fd67006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:36.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 -- 172.21.15.148:0/2331362360 shutdown_connections 2023-10-30T13:24:36.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 -- 172.21.15.148:0/2331362360 wait complete. 2023-10-30T13:24:36.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 Processor -- start 2023-10-30T13:24:36.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.208+0000 7fd677b0f640 1 -- start start 2023-10-30T13:24:36.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd677b0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6700cf160 0x7fd67014dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:36.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd677b0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6701538a0 0x7fd6701591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:36.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd677b0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67015d6f0 0x7fd6701e5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:36.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd677b0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6701e4e60 con 0x7fd6701538a0 2023-10-30T13:24:36.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd677b0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6701596f0 con 0x7fd6700cf160 2023-10-30T13:24:36.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd677b0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd670158480 con 0x7fd67015d6f0 2023-10-30T13:24:36.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6750aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6701538a0 0x7fd6701591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:36.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6750aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6701538a0 0x7fd6701591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36784/0 (socket says 172.21.15.148:36784) 2023-10-30T13:24:36.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6750aa640 1 -- 172.21.15.148:0/1819118823 learned_addr learned my addr 172.21.15.148:0/1819118823 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:36.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6758ab640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6700cf160 0x7fd67014dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:36.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6760ac640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67015d6f0 0x7fd6701e5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:36.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6750aa640 1 -- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67015d6f0 msgr2=0x7fd6701e5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:36.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6750aa640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67015d6f0 0x7fd6701e5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6750aa640 1 -- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6700cf160 msgr2=0x7fd67014dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:36.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6750aa640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6700cf160 0x7fd67014dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6750aa640 1 -- 172.21.15.148:0/1819118823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd668009a70 con 0x7fd6701538a0 2023-10-30T13:24:36.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6758ab640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6700cf160 0x7fd67014dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:36.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6750aa640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6701538a0 0x7fd6701591b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd66000e8a0 tx=0x7fd66000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:36.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6667fc640 1 -- 172.21.15.148:0/1819118823 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660009510 con 0x7fd6701538a0 2023-10-30T13:24:36.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6667fc640 1 -- 172.21.15.148:0/1819118823 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66000c8e0 con 0x7fd6701538a0 2023-10-30T13:24:36.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6667fc640 1 -- 172.21.15.148:0/1819118823 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66000cc50 con 0x7fd6701538a0 2023-10-30T13:24:36.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd670158d20 con 0x7fd6701538a0 2023-10-30T13:24:36.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6701586d0 con 0x7fd6701538a0 2023-10-30T13:24:36.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6700b2a50 con 0x7fd6701538a0 2023-10-30T13:24:36.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.212+0000 7fd6667fc640 1 -- 172.21.15.148:0/1819118823 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd660013470 con 0x7fd6701538a0 2023-10-30T13:24:36.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.216+0000 7fd6667fc640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6540664a0 0x7fd654068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:36.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.216+0000 7fd6667fc640 1 -- 172.21.15.148:0/1819118823 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd660084230 con 0x7fd6701538a0 2023-10-30T13:24:36.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.216+0000 7fd6758ab640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6540664a0 0x7fd654068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:36.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.216+0000 7fd6758ab640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6540664a0 0x7fd654068960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd66c005890 tx=0x7fd66c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:36.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.220+0000 7fd6667fc640 1 -- 172.21.15.148:0/1819118823 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6600513b0 con 0x7fd6701538a0 2023-10-30T13:24:36.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd670158eb0 con 0x7fd6701538a0 2023-10-30T13:24:36.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd6667fc640 1 -- 172.21.15.148:0/1819118823 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5716 (secure 0 0 0) 0x7fd6600549d0 con 0x7fd6701538a0 2023-10-30T13:24:36.397 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:36.397 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:29.547377+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:36.398 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 70 2023-10-30T13:24:36.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6540664a0 msgr2=0x7fd654068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:36.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6540664a0 0x7fd654068960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd66c005890 tx=0x7fd66c009040 comp rx=0 tx=0).stop 2023-10-30T13:24:36.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6701538a0 msgr2=0x7fd6701591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:36.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6701538a0 0x7fd6701591b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd66000e8a0 tx=0x7fd66000eca0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 shutdown_connections 2023-10-30T13:24:36.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6540664a0 0x7fd654068960 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6701538a0 0x7fd6701591b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd67015d6f0 0x7fd6701e5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 --2- 172.21.15.148:0/1819118823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6700cf160 0x7fd67014dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:36.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 >> 172.21.15.148:0/1819118823 conn(0x7fd67006bfe0 msgr2=0x7fd6700c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:36.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.400+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 shutdown_connections 2023-10-30T13:24:36.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:36.404+0000 7fd677b0f640 1 -- 172.21.15.148:0/1819118823 wait complete. 2023-10-30T13:24:36.417 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:38.419 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:38.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f9596640 1 Processor -- start 2023-10-30T13:24:38.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f9596640 1 -- start start 2023-10-30T13:24:38.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f9596640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f20f41538f0 0x7f20f4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:38.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f9596640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f415cd50 0x7f20f40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:38.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f9596640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f415d300 con 0x7f20f40cefa0 2023-10-30T13:24:38.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f9596640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20f415d5e0 con 0x7f20f41538f0 2023-10-30T13:24:38.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f9596640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f415d8c0 con 0x7f20f415cd50 2023-10-30T13:24:38.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f3fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f20f41538f0 0x7f20f4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:38.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f3fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f20f41538f0 0x7f20f4159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36788/0 (socket says 172.21.15.148:36788) 2023-10-30T13:24:38.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f3fff640 1 -- 172.21.15.148:0/1849484730 learned_addr learned my addr 172.21.15.148:0/1849484730 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:38.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f37fe640 1 --2- 172.21.15.148:0/1849484730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f415cd50 0x7f20f40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:38.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f3fff640 1 -- 172.21.15.148:0/1849484730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f415cd50 msgr2=0x7f20f40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:38.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f3fff640 1 --2- 172.21.15.148:0/1849484730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f415cd50 0x7f20f40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:38.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f3fff640 1 -- 172.21.15.148:0/1849484730 >> v1:172.21.15.150:6789/0 conn(0x7f20f40cefa0 legacy=0x7f20f40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:38.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.572+0000 7f20f3fff640 1 -- 172.21.15.148:0/1849484730 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f415dba0 con 0x7f20f41538f0 2023-10-30T13:24:38.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f3fff640 1 --2- 172.21.15.148:0/1849484730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f41538f0 0x7f20f4159fb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009870 tx=0x7f20e401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c69146bbc123d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:38.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f2ffd640 1 -- 172.21.15.148:0/1849484730 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e40a8a30 con 0x7f20f41538f0 2023-10-30T13:24:38.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f2ffd640 1 -- 172.21.15.148:0/1849484730 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e40a8b90 con 0x7f20f41538f0 2023-10-30T13:24:38.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f2ffd640 1 -- 172.21.15.148:0/1849484730 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e40b0440 con 0x7f20f41538f0 2023-10-30T13:24:38.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f9596640 1 -- 172.21.15.148:0/1849484730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f41538f0 msgr2=0x7f20f4159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:38.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f9596640 1 --2- 172.21.15.148:0/1849484730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f41538f0 0x7f20f4159fb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009870 tx=0x7f20e401ed30 comp rx=0 tx=0).stop 2023-10-30T13:24:38.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f9596640 1 -- 172.21.15.148:0/1849484730 shutdown_connections 2023-10-30T13:24:38.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f9596640 1 --2- 172.21.15.148:0/1849484730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f415cd50 0x7f20f40c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:38.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f9596640 1 --2- 172.21.15.148:0/1849484730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f41538f0 0x7f20f4159fb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:38.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f9596640 1 -- 172.21.15.148:0/1849484730 >> 172.21.15.148:0/1849484730 conn(0x7f20f406a640 msgr2=0x7f20f4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:38.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f9596640 1 -- 172.21.15.148:0/1849484730 shutdown_connections 2023-10-30T13:24:38.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.576+0000 7f20f9596640 1 -- 172.21.15.148:0/1849484730 wait complete. 2023-10-30T13:24:38.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 Processor -- start 2023-10-30T13:24:38.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 -- start start 2023-10-30T13:24:38.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20f40cefa0 0x7f20f40c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:38.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f41538f0 0x7f20f415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:38.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f415cd50 0x7f20f415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:38.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f4146120 con 0x7f20f415cd50 2023-10-30T13:24:38.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f4146540 con 0x7f20f40cefa0 2023-10-30T13:24:38.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f4145a60 con 0x7f20f41538f0 2023-10-30T13:24:38.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f8d95640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f415cd50 0x7f20f415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:38.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f8d95640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f415cd50 0x7f20f415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36798/0 (socket says 172.21.15.148:36798) 2023-10-30T13:24:38.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f8d95640 1 -- 172.21.15.148:0/1611452011 learned_addr learned my addr 172.21.15.148:0/1611452011 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:38.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f3fff640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20f40cefa0 0x7f20f40c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:38.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f37fe640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f41538f0 0x7f20f415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:38.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f8d95640 1 -- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f41538f0 msgr2=0x7f20f415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:38.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f8d95640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f41538f0 0x7f20f415ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:38.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f8d95640 1 -- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20f40cefa0 msgr2=0x7f20f40c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:38.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f8d95640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20f40cefa0 0x7f20f40c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:38.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f8d95640 1 -- 172.21.15.148:0/1611452011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e40094d0 con 0x7f20f415cd50 2023-10-30T13:24:38.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f37fe640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f41538f0 0x7f20f415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:38.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f8d95640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f415cd50 0x7f20f415b340 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00c770 tx=0x7f20ec00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:38.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f0ff9640 1 -- 172.21.15.148:0/1611452011 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec003b80 con 0x7f20f415cd50 2023-10-30T13:24:38.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f0ff9640 1 -- 172.21.15.148:0/1611452011 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ec003ce0 con 0x7f20f415cd50 2023-10-30T13:24:38.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f0ff9640 1 -- 172.21.15.148:0/1611452011 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec00aa30 con 0x7f20f415cd50 2023-10-30T13:24:38.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f415c390 con 0x7f20f415cd50 2023-10-30T13:24:38.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f415c5e0 con 0x7f20f415cd50 2023-10-30T13:24:38.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.580+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f4146260 con 0x7f20f415cd50 2023-10-30T13:24:38.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.584+0000 7f20f0ff9640 1 -- 172.21.15.148:0/1611452011 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20ec00ab90 con 0x7f20f415cd50 2023-10-30T13:24:38.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.584+0000 7f20f0ff9640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f20e00664a0 0x7f20e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:38.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.584+0000 7f20f0ff9640 1 -- 172.21.15.148:0/1611452011 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f20ec083ee0 con 0x7f20f415cd50 2023-10-30T13:24:38.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.584+0000 7f20f3fff640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f20e00664a0 0x7f20e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:38.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.584+0000 7f20f3fff640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f20e00664a0 0x7f20e0068960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f20e40b2000 tx=0x7f20e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:38.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.588+0000 7f20f0ff9640 1 -- 172.21.15.148:0/1611452011 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f20f4146260 con 0x7f20f415cd50 2023-10-30T13:24:38.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.772+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20f4146260 con 0x7f20f415cd50 2023-10-30T13:24:38.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.772+0000 7f20f0ff9640 1 -- 172.21.15.148:0/1611452011 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f20f4146260 con 0x7f20f415cd50 2023-10-30T13:24:38.770 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:38.771 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:36.149908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:38.772 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 71 2023-10-30T13:24:38.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f20e00664a0 msgr2=0x7f20e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:38.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f20e00664a0 0x7f20e0068960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f20e40b2000 tx=0x7f20e4009040 comp rx=0 tx=0).stop 2023-10-30T13:24:38.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f415cd50 msgr2=0x7f20f415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:38.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f415cd50 0x7f20f415b340 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00c770 tx=0x7f20ec00cc30 comp rx=0 tx=0).stop 2023-10-30T13:24:38.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 shutdown_connections 2023-10-30T13:24:38.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f20e00664a0 0x7f20e0068960 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:38.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f20f415cd50 0x7f20f415b340 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:38.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f20f41538f0 0x7f20f415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:38.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 --2- 172.21.15.148:0/1611452011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20f40cefa0 0x7f20f40c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:38.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 >> 172.21.15.148:0/1611452011 conn(0x7f20f406a640 msgr2=0x7f20f4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:38.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 shutdown_connections 2023-10-30T13:24:38.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:38.776+0000 7f20f9596640 1 -- 172.21.15.148:0/1611452011 wait complete. 2023-10-30T13:24:38.790 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:40.792 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:40.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa852446640 1 Processor -- start 2023-10-30T13:24:40.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa852446640 1 -- start start 2023-10-30T13:24:40.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa852446640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa84c0c05d0 0x7fa84c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:40.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa852446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c6f60 0x7fa84c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:40.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa852446640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84c0c0a50 con 0x7fa84c0c6b80 2023-10-30T13:24:40.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa852446640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c06f6e0 con 0x7fa84c0c05d0 2023-10-30T13:24:40.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa852446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c06f9c0 con 0x7fa84c0c6f60 2023-10-30T13:24:40.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa8509e3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c6f60 0x7fa84c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:40.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa84b7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa84c0c05d0 0x7fa84c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:40.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa8509e3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c6f60 0x7fa84c13f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:33026/0 (socket says 172.21.15.148:33026) 2023-10-30T13:24:40.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa8509e3640 1 -- 172.21.15.148:0/2583209184 learned_addr learned my addr 172.21.15.148:0/2583209184 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:40.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa84b7fe640 1 -- 172.21.15.148:0/2583209184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c6f60 msgr2=0x7fa84c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:40.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa84b7fe640 1 --2- 172.21.15.148:0/2583209184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c6f60 0x7fa84c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa84b7fe640 1 -- 172.21.15.148:0/2583209184 >> v1:172.21.15.150:6789/0 conn(0x7fa84c0c6b80 legacy=0x7fa84c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:40.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa84b7fe640 1 -- 172.21.15.148:0/2583209184 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c06fca0 con 0x7fa84c0c05d0 2023-10-30T13:24:40.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.944+0000 7fa8509e3640 1 --2- 172.21.15.148:0/2583209184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c6f60 0x7fa84c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:24:40.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa84b7fe640 1 --2- 172.21.15.148:0/2583209184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa84c0c05d0 0x7fa84c13ed90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa8400097c0 tx=0x7fa84001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d91a7c7d4f7a0dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:40.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa84affd640 1 -- 172.21.15.148:0/2583209184 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400ad640 con 0x7fa84c0c05d0 2023-10-30T13:24:40.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa84affd640 1 -- 172.21.15.148:0/2583209184 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8400ad7a0 con 0x7fa84c0c05d0 2023-10-30T13:24:40.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa84affd640 1 -- 172.21.15.148:0/2583209184 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400adb30 con 0x7fa84c0c05d0 2023-10-30T13:24:40.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa852446640 1 -- 172.21.15.148:0/2583209184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa84c0c05d0 msgr2=0x7fa84c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:40.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa852446640 1 --2- 172.21.15.148:0/2583209184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa84c0c05d0 0x7fa84c13ed90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa8400097c0 tx=0x7fa84001c480 comp rx=0 tx=0).stop 2023-10-30T13:24:40.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa852446640 1 -- 172.21.15.148:0/2583209184 shutdown_connections 2023-10-30T13:24:40.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa852446640 1 --2- 172.21.15.148:0/2583209184 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c6f60 0x7fa84c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa852446640 1 --2- 172.21.15.148:0/2583209184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa84c0c05d0 0x7fa84c13ed90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa852446640 1 -- 172.21.15.148:0/2583209184 >> 172.21.15.148:0/2583209184 conn(0x7fa84c0c7910 msgr2=0x7fa84c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:40.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.948+0000 7fa852446640 1 -- 172.21.15.148:0/2583209184 shutdown_connections 2023-10-30T13:24:40.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa852446640 1 -- 172.21.15.148:0/2583209184 wait complete. 2023-10-30T13:24:40.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa852446640 1 Processor -- start 2023-10-30T13:24:40.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa852446640 1 -- start start 2023-10-30T13:24:40.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa852446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c05d0 0x7fa84c14e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:40.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa852446640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa84c0c6b80 0x7fa84c1f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:40.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa852446640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa84c0c6f60 0x7fa84c130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:40.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa852446640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c132c70 con 0x7fa84c0c6f60 2023-10-30T13:24:40.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa852446640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c130f40 con 0x7fa84c0c6b80 2023-10-30T13:24:40.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa852446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c133550 con 0x7fa84c0c05d0 2023-10-30T13:24:40.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa8509e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa84c0c6f60 0x7fa84c130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:40.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa8509e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa84c0c6f60 0x7fa84c130a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36810/0 (socket says 172.21.15.148:36810) 2023-10-30T13:24:40.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa8509e3640 1 -- 172.21.15.148:0/991773082 learned_addr learned my addr 172.21.15.148:0/991773082 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:40.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa84b7fe640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa84c0c6b80 0x7fa84c1f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:40.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa84bfff640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c05d0 0x7fa84c14e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:40.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa8509e3640 1 -- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c05d0 msgr2=0x7fa84c14e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:40.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa8509e3640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c05d0 0x7fa84c14e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa8509e3640 1 -- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa84c0c6b80 msgr2=0x7fa84c1f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:40.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa8509e3640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa84c0c6b80 0x7fa84c1f1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa8509e3640 1 -- 172.21.15.148:0/991773082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84400bd40 con 0x7fa84c0c6f60 2023-10-30T13:24:40.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa84bfff640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c05d0 0x7fa84c14e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:40.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.952+0000 7fa8509e3640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa84c0c6f60 0x7fa84c130a00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa84400e0f0 tx=0x7fa84400e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:40.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa848ff9640 1 -- 172.21.15.148:0/991773082 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84400b310 con 0x7fa84c0c6f60 2023-10-30T13:24:40.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa848ff9640 1 -- 172.21.15.148:0/991773082 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa844010030 con 0x7fa84c0c6f60 2023-10-30T13:24:40.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa848ff9640 1 -- 172.21.15.148:0/991773082 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8440039a0 con 0x7fa84c0c6f60 2023-10-30T13:24:40.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa840009420 con 0x7fa84c0c6f60 2023-10-30T13:24:40.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c130570 con 0x7fa84c0c6f60 2023-10-30T13:24:40.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c130700 con 0x7fa84c0c6f60 2023-10-30T13:24:40.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa848ff9640 1 -- 172.21.15.148:0/991773082 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa844009600 con 0x7fa84c0c6f60 2023-10-30T13:24:40.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa848ff9640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8380664a0 0x7fa838068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:40.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa848ff9640 1 -- 172.21.15.148:0/991773082 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fa844086980 con 0x7fa84c0c6f60 2023-10-30T13:24:40.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.956+0000 7fa84bfff640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8380664a0 0x7fa838068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:40.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.960+0000 7fa84bfff640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8380664a0 0x7fa838068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa83c000f70 tx=0x7fa83c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:40.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:40.960+0000 7fa848ff9640 1 -- 172.21.15.148:0/991773082 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa844053a70 con 0x7fa84c0c6f60 2023-10-30T13:24:41.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.140+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa84c04e170 con 0x7fa84c0c6f60 2023-10-30T13:24:41.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa848ff9640 1 -- 172.21.15.148:0/991773082 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5720 (secure 0 0 0) 0x7fa844057090 con 0x7fa84c0c6f60 2023-10-30T13:24:41.140 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:41.140 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:36.149908+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:41.140 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 71 2023-10-30T13:24:41.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8380664a0 msgr2=0x7fa838068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:41.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8380664a0 0x7fa838068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa83c000f70 tx=0x7fa83c009040 comp rx=0 tx=0).stop 2023-10-30T13:24:41.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa84c0c6f60 msgr2=0x7fa84c130a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:41.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa84c0c6f60 0x7fa84c130a00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa84400e0f0 tx=0x7fa84400e5b0 comp rx=0 tx=0).stop 2023-10-30T13:24:41.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 shutdown_connections 2023-10-30T13:24:41.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8380664a0 0x7fa838068960 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:41.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa84c0c6f60 0x7fa84c130a00 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:41.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa84c0c6b80 0x7fa84c1f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:41.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 --2- 172.21.15.148:0/991773082 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa84c0c05d0 0x7fa84c14e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:41.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 >> 172.21.15.148:0/991773082 conn(0x7fa84c0c7910 msgr2=0x7fa84c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:41.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 shutdown_connections 2023-10-30T13:24:41.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:41.144+0000 7fa852446640 1 -- 172.21.15.148:0/991773082 wait complete. 2023-10-30T13:24:41.160 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:43.161 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:43.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f2293c09640 1 Processor -- start 2023-10-30T13:24:43.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f2293c09640 1 -- start start 2023-10-30T13:24:43.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f2293c09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c1538a0 0x7f228c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:43.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f2293c09640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f228c0cf160 0x7f228c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:43.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f2293c09640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0cb800 con 0x7f228c0ca0e0 2023-10-30T13:24:43.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f2293c09640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228c15dc30 con 0x7f228c0cf160 2023-10-30T13:24:43.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f2293c09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0cb330 con 0x7f228c1538a0 2023-10-30T13:24:43.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f22921a6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f228c0cf160 0x7f228c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:43.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f22921a6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f228c0cf160 0x7f228c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54736/0 (socket says 172.21.15.148:54736) 2023-10-30T13:24:43.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f22921a6640 1 -- 172.21.15.148:0/2045115070 learned_addr learned my addr 172.21.15.148:0/2045115070 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:43.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f22919a5640 1 --2- 172.21.15.148:0/2045115070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c1538a0 0x7f228c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:43.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f22921a6640 1 -- 172.21.15.148:0/2045115070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c1538a0 msgr2=0x7f228c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:43.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f22921a6640 1 --2- 172.21.15.148:0/2045115070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c1538a0 0x7f228c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f22921a6640 1 -- 172.21.15.148:0/2045115070 >> v1:172.21.15.148:6789/0 conn(0x7f228c0ca0e0 legacy=0x7f228c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:43.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.312+0000 7f22921a6640 1 -- 172.21.15.148:0/2045115070 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c0cad70 con 0x7f228c0cf160 2023-10-30T13:24:43.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f22921a6640 1 --2- 172.21.15.148:0/2045115070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0cf160 0x7f228c0cf590 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2284009e50 tx=0x7f228401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351ec53d99d43610 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:43.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f22909a3640 1 -- 172.21.15.148:0/2045115070 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22840af400 con 0x7f228c0cf160 2023-10-30T13:24:43.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f22909a3640 1 -- 172.21.15.148:0/2045115070 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22840ab050 con 0x7f228c0cf160 2023-10-30T13:24:43.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f22909a3640 1 -- 172.21.15.148:0/2045115070 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22840af780 con 0x7f228c0cf160 2023-10-30T13:24:43.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f2293c09640 1 -- 172.21.15.148:0/2045115070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0cf160 msgr2=0x7f228c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:43.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f2293c09640 1 --2- 172.21.15.148:0/2045115070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0cf160 0x7f228c0cf590 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2284009e50 tx=0x7f228401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f2293c09640 1 -- 172.21.15.148:0/2045115070 shutdown_connections 2023-10-30T13:24:43.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f2293c09640 1 --2- 172.21.15.148:0/2045115070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0cf160 0x7f228c0cf590 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f2293c09640 1 --2- 172.21.15.148:0/2045115070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c1538a0 0x7f228c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f2293c09640 1 -- 172.21.15.148:0/2045115070 >> 172.21.15.148:0/2045115070 conn(0x7f228c06bfe0 msgr2=0x7f228c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:43.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f2293c09640 1 -- 172.21.15.148:0/2045115070 shutdown_connections 2023-10-30T13:24:43.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.316+0000 7f2293c09640 1 -- 172.21.15.148:0/2045115070 wait complete. 2023-10-30T13:24:43.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 Processor -- start 2023-10-30T13:24:43.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 -- start start 2023-10-30T13:24:43.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0ca0e0 0x7f228c15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:43.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c0cf160 0x7f228c15b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:43.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f228c1538a0 0x7f228c15b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:43.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c1e3960 con 0x7f228c0ca0e0 2023-10-30T13:24:43.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c1e4750 con 0x7f228c1538a0 2023-10-30T13:24:43.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c1e3ba0 con 0x7f228c0cf160 2023-10-30T13:24:43.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22919a5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0ca0e0 0x7f228c15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:43.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22919a5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0ca0e0 0x7f228c15a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54748/0 (socket says 172.21.15.148:54748) 2023-10-30T13:24:43.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22919a5640 1 -- 172.21.15.148:0/1797904902 learned_addr learned my addr 172.21.15.148:0/1797904902 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:43.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22911a4640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c0cf160 0x7f228c15b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:43.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22921a6640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f228c1538a0 0x7f228c15b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:43.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22919a5640 1 -- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c0cf160 msgr2=0x7f228c15b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:43.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22919a5640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c0cf160 0x7f228c15b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22919a5640 1 -- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f228c1538a0 msgr2=0x7f228c15b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:43.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22919a5640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f228c1538a0 0x7f228c15b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22919a5640 1 -- 172.21.15.148:0/1797904902 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2284009b30 con 0x7f228c0ca0e0 2023-10-30T13:24:43.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22911a4640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c0cf160 0x7f228c15b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:43.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22919a5640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0ca0e0 0x7f228c15a8b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f228800d5f0 tx=0x7f228800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:43.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22827fc640 1 -- 172.21.15.148:0/1797904902 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22880125f0 con 0x7f228c0ca0e0 2023-10-30T13:24:43.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22827fc640 1 -- 172.21.15.148:0/1797904902 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2288012750 con 0x7f228c0ca0e0 2023-10-30T13:24:43.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c1e3430 con 0x7f228c0ca0e0 2023-10-30T13:24:43.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f22827fc640 1 -- 172.21.15.148:0/1797904902 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2288012ae0 con 0x7f228c0ca0e0 2023-10-30T13:24:43.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.320+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c1e3db0 con 0x7f228c0ca0e0 2023-10-30T13:24:43.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.324+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c1e49c0 con 0x7f228c0ca0e0 2023-10-30T13:24:43.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.324+0000 7f22827fc640 1 -- 172.21.15.148:0/1797904902 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2288012c40 con 0x7f228c0ca0e0 2023-10-30T13:24:43.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.324+0000 7f22827fc640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22700664a0 0x7f2270068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:43.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.324+0000 7f22827fc640 1 -- 172.21.15.148:0/1797904902 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f2288084670 con 0x7f228c0ca0e0 2023-10-30T13:24:43.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.324+0000 7f22911a4640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22700664a0 0x7f2270068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:43.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.324+0000 7f22911a4640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22700664a0 0x7f2270068960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f228c15a130 tx=0x7f227c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:43.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.328+0000 7f22827fc640 1 -- 172.21.15.148:0/1797904902 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2288051800 con 0x7f228c0ca0e0 2023-10-30T13:24:43.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.508+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f228c0b2a50 con 0x7f228c0ca0e0 2023-10-30T13:24:43.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.508+0000 7f22827fc640 1 -- 172.21.15.148:0/1797904902 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f2288007380 con 0x7f228c0ca0e0 2023-10-30T13:24:43.506 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:43.506 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:41.724822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:43.507 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 72 2023-10-30T13:24:43.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22700664a0 msgr2=0x7f2270068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:43.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22700664a0 0x7f2270068960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f228c15a130 tx=0x7f227c009040 comp rx=0 tx=0).stop 2023-10-30T13:24:43.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0ca0e0 msgr2=0x7f228c15a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:43.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0ca0e0 0x7f228c15a8b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f228800d5f0 tx=0x7f228800dab0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 shutdown_connections 2023-10-30T13:24:43.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22700664a0 0x7f2270068960 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f228c1538a0 0x7f228c15b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f228c0cf160 0x7f228c15b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 --2- 172.21.15.148:0/1797904902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f228c0ca0e0 0x7f228c15a8b0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:43.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 >> 172.21.15.148:0/1797904902 conn(0x7f228c06bfe0 msgr2=0x7f228c0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:43.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 shutdown_connections 2023-10-30T13:24:43.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:43.512+0000 7f2293c09640 1 -- 172.21.15.148:0/1797904902 wait complete. 2023-10-30T13:24:43.527 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 72, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 72, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5955": { "addr": "172.21.15.148:6837/1559416001", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 1559416001, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 1559416001, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5955, "incarnation": 70, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:24:41.724822+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 5955 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:24:45.529 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:45.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.680+0000 7fef42b72640 1 Processor -- start 2023-10-30T13:24:45.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.680+0000 7fef42b72640 1 -- start start 2023-10-30T13:24:45.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef42b72640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c0c05d0 0x7fef3c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:45.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef42b72640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3c0c0a50 con 0x7fef3c0c6b80 2023-10-30T13:24:45.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef42b72640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3c06f6e0 con 0x7fef3c0c6f60 2023-10-30T13:24:45.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef42b72640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c06f9c0 con 0x7fef3c0c05d0 2023-10-30T13:24:45.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef4090e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fef3c0c6b80 0x7fef3c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46458/0 (socket says 172.21.15.148:46458) 2023-10-30T13:24:45.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef4090e640 1 -- 172.21.15.148:0/2067765237 learned_addr learned my addr 172.21.15.148:0/2067765237 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:24:45.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef33fff640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c0c05d0 0x7fef3c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:45.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303442865 0 0) 0x7fef3c0c0a50 con 0x7fef3c0c6b80 2023-10-30T13:24:45.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef24003530 con 0x7fef3c0c6b80 2023-10-30T13:24:45.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4271896138 0 0) 0x7fef24003530 con 0x7fef3c0c6b80 2023-10-30T13:24:45.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c0c05d0 msgr2=0x7fef3c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:45.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c0c05d0 0x7fef3c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:45.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 >> v1:172.21.15.150:6789/0 conn(0x7fef3c0c6f60 legacy=0x7fef3c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:45.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c06fca0 con 0x7fef3c0c6b80 2023-10-30T13:24:45.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fef38001680 con 0x7fef3c0c6b80 2023-10-30T13:24:45.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 >> v1:172.21.15.148:6789/0 conn(0x7fef3c0c6b80 legacy=0x7fef3c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:45.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:45.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef24004360 0x7fef24004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:45.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c0c05d0 0x7fef24005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:45.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c06f9c0 con 0x7fef240039b0 2023-10-30T13:24:45.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2400b9c0 con 0x7fef24004360 2023-10-30T13:24:45.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2400bb80 con 0x7fef3c0c05d0 2023-10-30T13:24:45.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef4090e640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:45.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef33fff640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef24004360 0x7fef24004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:45.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef4110f640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c0c05d0 0x7fef24005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:45.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef4090e640 1 -- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c0c05d0 msgr2=0x7fef24005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:45.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef4090e640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c0c05d0 0x7fef24005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:45.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef4090e640 1 -- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef24004360 msgr2=0x7fef24004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:45.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef4090e640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef24004360 0x7fef24004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:45.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.684+0000 7fef4090e640 1 -- 172.21.15.148:0/2067765237 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24003530 con 0x7fef240039b0 2023-10-30T13:24:45.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef4090e640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef24003d90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fef3800c210 tx=0x7fef3801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9871e576595d738 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:45.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef38005ae0 con 0x7fef240039b0 2023-10-30T13:24:45.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef38005c60 con 0x7fef240039b0 2023-10-30T13:24:45.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef337fe640 1 -- 172.21.15.148:0/2067765237 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef380a5850 con 0x7fef240039b0 2023-10-30T13:24:45.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef42b72640 1 -- 172.21.15.148:0/2067765237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 msgr2=0x7fef24003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:45.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef42b72640 1 --2- 172.21.15.148:0/2067765237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef24003d90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fef3800c210 tx=0x7fef3801cc60 comp rx=0 tx=0).stop 2023-10-30T13:24:45.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef4110f640 1 -- 172.21.15.148:0/2067765237 reap_dead start 2023-10-30T13:24:45.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef42b72640 1 -- 172.21.15.148:0/2067765237 shutdown_connections 2023-10-30T13:24:45.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef42b72640 1 -- 172.21.15.148:0/2067765237 >> 172.21.15.148:0/2067765237 conn(0x7fef3c0c7910 msgr2=0x7fef3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:45.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef42b72640 1 -- 172.21.15.148:0/2067765237 shutdown_connections 2023-10-30T13:24:45.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.688+0000 7fef42b72640 1 -- 172.21.15.148:0/2067765237 wait complete. 2023-10-30T13:24:45.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 Processor -- start 2023-10-30T13:24:45.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 -- start start 2023-10-30T13:24:45.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef3c0ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:45.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c0c1ca0 0x7fef3c13d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:45.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c1563b0 0x7fef3c154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:45.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c15f320 con 0x7fef240039b0 2023-10-30T13:24:45.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c154810 con 0x7fef3c0c1ca0 2023-10-30T13:24:45.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c156a50 con 0x7fef3c1563b0 2023-10-30T13:24:45.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4090e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef3c0ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:45.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4090e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef3c0ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54756/0 (socket says 172.21.15.148:54756) 2023-10-30T13:24:45.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4090e640 1 -- 172.21.15.148:0/3325592604 learned_addr learned my addr 172.21.15.148:0/3325592604 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:45.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef33fff640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c0c1ca0 0x7fef3c13d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:45.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4110f640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c1563b0 0x7fef3c154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:45.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4090e640 1 -- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c1563b0 msgr2=0x7fef3c154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:45.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4090e640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c1563b0 0x7fef3c154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:45.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4090e640 1 -- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c0c1ca0 msgr2=0x7fef3c13d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:45.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4090e640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c0c1ca0 0x7fef3c13d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:45.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4090e640 1 -- 172.21.15.148:0/3325592604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c0aef00 con 0x7fef240039b0 2023-10-30T13:24:45.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4110f640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c1563b0 0x7fef3c154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:45.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef4090e640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef3c0ae8c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fef3801ca90 tx=0x7fef24005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:45.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef317fa640 1 -- 172.21.15.148:0/3325592604 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef380b4070 con 0x7fef240039b0 2023-10-30T13:24:45.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef317fa640 1 -- 172.21.15.148:0/3325592604 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef38005dd0 con 0x7fef240039b0 2023-10-30T13:24:45.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3c0af1e0 con 0x7fef240039b0 2023-10-30T13:24:45.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef317fa640 1 -- 172.21.15.148:0/3325592604 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef38001660 con 0x7fef240039b0 2023-10-30T13:24:45.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3c0af7a0 con 0x7fef240039b0 2023-10-30T13:24:45.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.692+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3c0af990 con 0x7fef240039b0 2023-10-30T13:24:45.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.696+0000 7fef317fa640 1 -- 172.21.15.148:0/3325592604 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fef380a5850 con 0x7fef240039b0 2023-10-30T13:24:45.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.696+0000 7fef317fa640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fef280664d0 0x7fef28068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:45.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.696+0000 7fef317fa640 1 -- 172.21.15.148:0/3325592604 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fef3811dc90 con 0x7fef240039b0 2023-10-30T13:24:45.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.696+0000 7fef33fff640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fef280664d0 0x7fef28068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:45.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.696+0000 7fef33fff640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fef280664d0 0x7fef28068990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fef3c161f30 tx=0x7fef2c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:45.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.700+0000 7fef317fa640 1 -- 172.21.15.148:0/3325592604 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fef380ead90 con 0x7fef240039b0 2023-10-30T13:24:45.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.880+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef3c04e170 con 0x7fef240039b0 2023-10-30T13:24:45.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef317fa640 1 -- 172.21.15.148:0/3325592604 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fef380ee3b0 con 0x7fef240039b0 2023-10-30T13:24:45.879 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:45.880 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:41.724822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:45.880 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 72 2023-10-30T13:24:45.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fef280664d0 msgr2=0x7fef28068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:45.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fef280664d0 0x7fef28068990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fef3c161f30 tx=0x7fef2c009190 comp rx=0 tx=0).stop 2023-10-30T13:24:45.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 msgr2=0x7fef3c0ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:45.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef3c0ae8c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fef3801ca90 tx=0x7fef24005800 comp rx=0 tx=0).stop 2023-10-30T13:24:45.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 shutdown_connections 2023-10-30T13:24:45.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fef280664d0 0x7fef28068990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:45.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fef3c1563b0 0x7fef3c154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:45.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef3c0c1ca0 0x7fef3c13d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:45.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 --2- 172.21.15.148:0/3325592604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef240039b0 0x7fef3c0ae8c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:45.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 >> 172.21.15.148:0/3325592604 conn(0x7fef3c0c7910 msgr2=0x7fef3c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:45.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 shutdown_connections 2023-10-30T13:24:45.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:45.884+0000 7fef42b72640 1 -- 172.21.15.148:0/3325592604 wait complete. 2023-10-30T13:24:45.900 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:47.901 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.052+0000 7f869b1d0640 1 Processor -- start 2023-10-30T13:24:48.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.052+0000 7f869b1d0640 1 -- start start 2023-10-30T13:24:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f869b1d0640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f86940c3640 0x7f869415a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f869b1d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c6c20 0x7f8694156590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f869b1d0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86940c3b30 con 0x7f86940c6840 2023-10-30T13:24:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f869b1d0640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86940c2aa0 con 0x7f86940c3640 2023-10-30T13:24:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f869b1d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86940c3df0 con 0x7f86940c6c20 2023-10-30T13:24:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f86999cd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f86940c3640 0x7f869415a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f86999cd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f86940c3640 0x7f869415a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54762/0 (socket says 172.21.15.148:54762) 2023-10-30T13:24:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f86999cd640 1 -- 172.21.15.148:0/49624482 learned_addr learned my addr 172.21.15.148:0/49624482 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f869a9cf640 1 --2- 172.21.15.148:0/49624482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c6c20 0x7f8694156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f86999cd640 1 -- 172.21.15.148:0/49624482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c6c20 msgr2=0x7f8694156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f86999cd640 1 --2- 172.21.15.148:0/49624482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c6c20 0x7f8694156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f86999cd640 1 -- 172.21.15.148:0/49624482 >> v1:172.21.15.150:6789/0 conn(0x7f86940c6840 legacy=0x7f8694153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.056+0000 7f86999cd640 1 -- 172.21.15.148:0/49624482 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8694159ba0 con 0x7f86940c3640 2023-10-30T13:24:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f86999cd640 1 --2- 172.21.15.148:0/49624482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c3640 0x7f869415a680 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8684009870 tx=0x7f868401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b967499972879e3d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f86991cc640 1 -- 172.21.15.148:0/49624482 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86840ab650 con 0x7f86940c3640 2023-10-30T13:24:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f86991cc640 1 -- 172.21.15.148:0/49624482 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86840ab7b0 con 0x7f86940c3640 2023-10-30T13:24:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f86991cc640 1 -- 172.21.15.148:0/49624482 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86840abb40 con 0x7f86940c3640 2023-10-30T13:24:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 -- 172.21.15.148:0/49624482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c3640 msgr2=0x7f869415a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 --2- 172.21.15.148:0/49624482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c3640 0x7f869415a680 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8684009870 tx=0x7f868401c540 comp rx=0 tx=0).stop 2023-10-30T13:24:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 -- 172.21.15.148:0/49624482 shutdown_connections 2023-10-30T13:24:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 --2- 172.21.15.148:0/49624482 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c6c20 0x7f8694156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 --2- 172.21.15.148:0/49624482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c3640 0x7f869415a680 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 -- 172.21.15.148:0/49624482 >> 172.21.15.148:0/49624482 conn(0x7f86940c75d0 msgr2=0x7f86940c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 -- 172.21.15.148:0/49624482 shutdown_connections 2023-10-30T13:24:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 -- 172.21.15.148:0/49624482 wait complete. 2023-10-30T13:24:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 Processor -- start 2023-10-30T13:24:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.060+0000 7f869b1d0640 1 -- start start 2023-10-30T13:24:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869b1d0640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c3640 0x7f8694070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869b1d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86940c6840 0x7f8694144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869b1d0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c6c20 0x7f8694155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869b1d0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86941587c0 con 0x7f86940c6c20 2023-10-30T13:24:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869b1d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86941543c0 con 0x7f86940c6840 2023-10-30T13:24:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869b1d0640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86941548b0 con 0x7f86940c3640 2023-10-30T13:24:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a9cf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c6c20 0x7f8694155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a9cf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c6c20 0x7f8694155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54770/0 (socket says 172.21.15.148:54770) 2023-10-30T13:24:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a9cf640 1 -- 172.21.15.148:0/2002195689 learned_addr learned my addr 172.21.15.148:0/2002195689 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a1ce640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c3640 0x7f8694070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f86999cd640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86940c6840 0x7f8694144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a9cf640 1 -- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c3640 msgr2=0x7f8694070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a9cf640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c3640 0x7f8694070a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a9cf640 1 -- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86940c6840 msgr2=0x7f8694144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a9cf640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86940c6840 0x7f8694144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a9cf640 1 -- 172.21.15.148:0/2002195689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86840094d0 con 0x7f86940c6c20 2023-10-30T13:24:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f86999cd640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86940c6840 0x7f8694144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869a9cf640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c6c20 0x7f8694155e00 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f868c007e60 tx=0x7f868c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f868affd640 1 -- 172.21.15.148:0/2002195689 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c014420 con 0x7f86940c6c20 2023-10-30T13:24:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f868affd640 1 -- 172.21.15.148:0/2002195689 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c0145a0 con 0x7f86940c6c20 2023-10-30T13:24:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86941545f0 con 0x7f86940c6c20 2023-10-30T13:24:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f868affd640 1 -- 172.21.15.148:0/2002195689 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c014950 con 0x7f86940c6c20 2023-10-30T13:24:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8694158200 con 0x7f86940c6c20 2023-10-30T13:24:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.064+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86940c1f30 con 0x7f86940c6c20 2023-10-30T13:24:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.068+0000 7f868affd640 1 -- 172.21.15.148:0/2002195689 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f868c014ab0 con 0x7f86940c6c20 2023-10-30T13:24:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.068+0000 7f868affd640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86780664d0 0x7f8678068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.068+0000 7f868affd640 1 -- 172.21.15.148:0/2002195689 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f868c084ab0 con 0x7f86940c6c20 2023-10-30T13:24:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.068+0000 7f869a1ce640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86780664d0 0x7f8678068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.068+0000 7f869a1ce640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86780664d0 0x7f8678068990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8690000fc0 tx=0x7f8690009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.072+0000 7f868affd640 1 -- 172.21.15.148:0/2002195689 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f86940c1f30 con 0x7f86940c6c20 2023-10-30T13:24:48.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.248+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86940c1f30 con 0x7f86940c6c20 2023-10-30T13:24:48.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f868affd640 1 -- 172.21.15.148:0/2002195689 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f86940c1f30 con 0x7f86940c6c20 2023-10-30T13:24:48.248 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:48.248 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:46.794165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:48.248 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 73 2023-10-30T13:24:48.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86780664d0 msgr2=0x7f8678068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:48.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86780664d0 0x7f8678068990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8690000fc0 tx=0x7f8690009040 comp rx=0 tx=0).stop 2023-10-30T13:24:48.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c6c20 msgr2=0x7f8694155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:48.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c6c20 0x7f8694155e00 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f868c007e60 tx=0x7f868c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 shutdown_connections 2023-10-30T13:24:48.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86780664d0 0x7f8678068990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86940c6c20 0x7f8694155e00 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86940c6840 0x7f8694144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 --2- 172.21.15.148:0/2002195689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86940c3640 0x7f8694070a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:48.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 >> 172.21.15.148:0/2002195689 conn(0x7f86940c75d0 msgr2=0x7f869406d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:48.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 shutdown_connections 2023-10-30T13:24:48.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:48.252+0000 7f869b1d0640 1 -- 172.21.15.148:0/2002195689 wait complete. 2023-10-30T13:24:48.268 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:50.270 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:50.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.424+0000 7f955ee79640 1 Processor -- start 2023-10-30T13:24:50.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.424+0000 7f955ee79640 1 -- start start 2023-10-30T13:24:50.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f955ee79640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f95580c05d0 0x7f955813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:50.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f955ee79640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95580c0a50 con 0x7f95580c6b80 2023-10-30T13:24:50.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f955ee79640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955806f6e0 con 0x7f95580c6f60 2023-10-30T13:24:50.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f955ee79640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955806f9c0 con 0x7f95580c05d0 2023-10-30T13:24:50.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f955cc15640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f95580c6b80 0x7f95580bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46474/0 (socket says 172.21.15.148:46474) 2023-10-30T13:24:50.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f955cc15640 1 -- 172.21.15.148:0/2518364281 learned_addr learned my addr 172.21.15.148:0/2518364281 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:24:50.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f9557fff640 1 --2- 172.21.15.148:0/2518364281 >> v2:172.21.15.148:3300/0 conn(0x7f95580c05d0 0x7f955813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:50.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f9557fff640 1 -- 172.21.15.148:0/2518364281 >> v1:172.21.15.150:6789/0 conn(0x7f95580c6f60 legacy=0x7f95581566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:50.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f9557fff640 1 -- 172.21.15.148:0/2518364281 >> v1:172.21.15.148:6789/0 conn(0x7f95580c6b80 legacy=0x7f95580bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:50.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.428+0000 7f9557fff640 1 -- 172.21.15.148:0/2518364281 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955806fca0 con 0x7f95580c05d0 2023-10-30T13:24:50.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f9557fff640 1 --2- 172.21.15.148:0/2518364281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 0x7f955813ed90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f95440097c0 tx=0x7f954401e690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a347ace6e1fbe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:50.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f95577fe640 1 -- 172.21.15.148:0/2518364281 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95440ac610 con 0x7f95580c05d0 2023-10-30T13:24:50.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f95577fe640 1 -- 172.21.15.148:0/2518364281 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95440a8040 con 0x7f95580c05d0 2023-10-30T13:24:50.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f95577fe640 1 -- 172.21.15.148:0/2518364281 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95440ac990 con 0x7f95580c05d0 2023-10-30T13:24:50.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f955ee79640 1 -- 172.21.15.148:0/2518364281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 msgr2=0x7f955813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:50.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f955ee79640 1 --2- 172.21.15.148:0/2518364281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 0x7f955813ed90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f95440097c0 tx=0x7f954401e690 comp rx=0 tx=0).stop 2023-10-30T13:24:50.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f955ee79640 1 -- 172.21.15.148:0/2518364281 shutdown_connections 2023-10-30T13:24:50.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f955ee79640 1 --2- 172.21.15.148:0/2518364281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 0x7f955813ed90 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:50.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f955ee79640 1 -- 172.21.15.148:0/2518364281 >> 172.21.15.148:0/2518364281 conn(0x7f95580c7910 msgr2=0x7f95580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:50.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f955ee79640 1 -- 172.21.15.148:0/2518364281 shutdown_connections 2023-10-30T13:24:50.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f955ee79640 1 -- 172.21.15.148:0/2518364281 wait complete. 2023-10-30T13:24:50.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f955ee79640 1 Processor -- start 2023-10-30T13:24:50.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.432+0000 7f955ee79640 1 -- start start 2023-10-30T13:24:50.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955ee79640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 0x7f95581f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:50.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955ee79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95580c6b80 0x7f955813c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:50.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955ee79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95580c6f60 0x7f95581591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:50.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955ee79640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95580c2a30 con 0x7f95580c05d0 2023-10-30T13:24:50.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955ee79640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95581f3790 con 0x7f95580c6b80 2023-10-30T13:24:50.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955ee79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9558139d30 con 0x7f95580c6f60 2023-10-30T13:24:50.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955cc15640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 0x7f95581f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:50.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955cc15640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 0x7f95581f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54794/0 (socket says 172.21.15.148:54794) 2023-10-30T13:24:50.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955cc15640 1 -- 172.21.15.148:0/893001586 learned_addr learned my addr 172.21.15.148:0/893001586 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:50.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f9557fff640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95580c6b80 0x7f955813c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:50.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955d416640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95580c6f60 0x7f95581591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:50.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955cc15640 1 -- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95580c6f60 msgr2=0x7f95581591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:50.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955cc15640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95580c6f60 0x7f95581591f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:50.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955cc15640 1 -- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95580c6b80 msgr2=0x7f955813c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:50.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955cc15640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95580c6b80 0x7f955813c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:50.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955cc15640 1 -- 172.21.15.148:0/893001586 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9544009420 con 0x7f95580c05d0 2023-10-30T13:24:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955cc15640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 0x7f95581f3250 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f954c00c560 tx=0x7f954c00ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f95557fa640 1 -- 172.21.15.148:0/893001586 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c005cf0 con 0x7f95580c05d0 2023-10-30T13:24:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f95557fa640 1 -- 172.21.15.148:0/893001586 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954c005e50 con 0x7f95580c05d0 2023-10-30T13:24:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955813d620 con 0x7f95580c05d0 2023-10-30T13:24:50.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f95557fa640 1 -- 172.21.15.148:0/893001586 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c007270 con 0x7f95580c05d0 2023-10-30T13:24:50.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955813cb30 con 0x7f95580c05d0 2023-10-30T13:24:50.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955804e170 con 0x7f95580c05d0 2023-10-30T13:24:50.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.436+0000 7f95557fa640 1 -- 172.21.15.148:0/893001586 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f954c0073d0 con 0x7f95580c05d0 2023-10-30T13:24:50.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.440+0000 7f95557fa640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f953c0664a0 0x7f953c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:50.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.440+0000 7f95557fa640 1 -- 172.21.15.148:0/893001586 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f954c083730 con 0x7f95580c05d0 2023-10-30T13:24:50.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.440+0000 7f9557fff640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f953c0664a0 0x7f953c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:50.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.440+0000 7f9557fff640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f953c0664a0 0x7f953c068960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f95440b0000 tx=0x7f95440a9000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:50.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.444+0000 7f95557fa640 1 -- 172.21.15.148:0/893001586 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f954c050840 con 0x7f95580c05d0 2023-10-30T13:24:50.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.624+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9558139410 con 0x7f95580c05d0 2023-10-30T13:24:50.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.624+0000 7f95557fa640 1 -- 172.21.15.148:0/893001586 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f954c017090 con 0x7f95580c05d0 2023-10-30T13:24:50.623 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:50.623 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:46.794165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:50.624 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 73 2023-10-30T13:24:50.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f953c0664a0 msgr2=0x7f953c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:50.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f953c0664a0 0x7f953c068960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f95440b0000 tx=0x7f95440a9000 comp rx=0 tx=0).stop 2023-10-30T13:24:50.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 msgr2=0x7f95581f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:50.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 0x7f95581f3250 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f954c00c560 tx=0x7f954c00ca20 comp rx=0 tx=0).stop 2023-10-30T13:24:50.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 shutdown_connections 2023-10-30T13:24:50.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f953c0664a0 0x7f953c068960 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:50.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95580c6f60 0x7f95581591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:50.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95580c6b80 0x7f955813c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:50.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 --2- 172.21.15.148:0/893001586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95580c05d0 0x7f95581f3250 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:50.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 >> 172.21.15.148:0/893001586 conn(0x7f95580c7910 msgr2=0x7f955806aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:50.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 shutdown_connections 2023-10-30T13:24:50.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:50.628+0000 7f955ee79640 1 -- 172.21.15.148:0/893001586 wait complete. 2023-10-30T13:24:50.643 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:52.645 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:52.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.892+0000 7f19650d6640 1 Processor -- start 2023-10-30T13:24:52.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.892+0000 7f19650d6640 1 -- start start 2023-10-30T13:24:52.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f19650d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1960152ec0 0x7f1960159790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:52.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f19650d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f19600c8430 0x7f19600715a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:52.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f19650d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f1960069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:52.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f19650d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196006fd10 con 0x7f19600c8430 2023-10-30T13:24:52.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f19650d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1960070b90 con 0x7f1960152ec0 2023-10-30T13:24:52.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f19650d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19601532a0 con 0x7f1960071dc0 2023-10-30T13:24:52.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f195e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f19600c8430 0x7f19600715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:52.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f195e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f19600c8430 0x7f19600715a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38122/0 (socket says 172.21.15.148:38122) 2023-10-30T13:24:52.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f195e59c640 1 -- 172.21.15.148:0/2570655156 learned_addr learned my addr 172.21.15.148:0/2570655156 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:52.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f195f59e640 1 --2- 172.21.15.148:0/2570655156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f1960069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:52.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f195f59e640 1 -- 172.21.15.148:0/2570655156 >> v2:172.21.15.150:3300/0 conn(0x7f1960152ec0 msgr2=0x7f1960159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:52.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f195f59e640 1 --2- 172.21.15.148:0/2570655156 >> v2:172.21.15.150:3300/0 conn(0x7f1960152ec0 0x7f1960159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:52.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f195f59e640 1 -- 172.21.15.148:0/2570655156 >> v2:172.21.15.148:3300/0 conn(0x7f19600c8430 msgr2=0x7f19600715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:52.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f195f59e640 1 --2- 172.21.15.148:0/2570655156 >> v2:172.21.15.148:3300/0 conn(0x7f19600c8430 0x7f19600715a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:52.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.896+0000 7f195f59e640 1 -- 172.21.15.148:0/2570655156 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19600c7280 con 0x7f1960071dc0 2023-10-30T13:24:52.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.900+0000 7f195f59e640 1 --2- 172.21.15.148:0/2570655156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f1960069e90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f196006c750 tx=0x7f194801c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bd01a6e87af9437 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:52.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.900+0000 7f195dd9b640 1 -- 172.21.15.148:0/2570655156 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19480ab600 con 0x7f1960071dc0 2023-10-30T13:24:52.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.900+0000 7f195dd9b640 1 -- 172.21.15.148:0/2570655156 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19480a7050 con 0x7f1960071dc0 2023-10-30T13:24:52.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f195dd9b640 1 -- 172.21.15.148:0/2570655156 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19480ab980 con 0x7f1960071dc0 2023-10-30T13:24:52.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f19650d6640 1 -- 172.21.15.148:0/2570655156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 msgr2=0x7f1960069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:52.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f19650d6640 1 --2- 172.21.15.148:0/2570655156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f1960069e90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f196006c750 tx=0x7f194801c430 comp rx=0 tx=0).stop 2023-10-30T13:24:52.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f19650d6640 1 -- 172.21.15.148:0/2570655156 shutdown_connections 2023-10-30T13:24:52.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f19650d6640 1 --2- 172.21.15.148:0/2570655156 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f1960069e90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:52.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f19650d6640 1 --2- 172.21.15.148:0/2570655156 >> v2:172.21.15.148:3300/0 conn(0x7f19600c8430 0x7f19600715a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:52.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f19650d6640 1 --2- 172.21.15.148:0/2570655156 >> v2:172.21.15.150:3300/0 conn(0x7f1960152ec0 0x7f1960159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:52.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f19650d6640 1 -- 172.21.15.148:0/2570655156 >> 172.21.15.148:0/2570655156 conn(0x7f19600c7910 msgr2=0x7f19600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:52.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f19650d6640 1 -- 172.21.15.148:0/2570655156 shutdown_connections 2023-10-30T13:24:52.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.904+0000 7f19650d6640 1 -- 172.21.15.148:0/2570655156 wait complete. 2023-10-30T13:24:52.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.908+0000 7f19650d6640 1 Processor -- start 2023-10-30T13:24:52.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.908+0000 7f19650d6640 1 -- start start 2023-10-30T13:24:52.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.908+0000 7f19650d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f19600c0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:52.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.908+0000 7f19650d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19600c8430 0x7f1960151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:52.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.908+0000 7f19650d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1960152ec0 0x7f19601517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:52.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.908+0000 7f19650d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196013f620 con 0x7f1960152ec0 2023-10-30T13:24:52.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.908+0000 7f19650d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1960150e40 con 0x7f19600c8430 2023-10-30T13:24:52.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.908+0000 7f19650d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1960160ff0 con 0x7f1960071dc0 2023-10-30T13:24:52.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19600c8430 0x7f1960151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:52.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195ed9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f19600c0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:52.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195ed9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f19600c0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:50806/0 (socket says 172.21.15.148:50806) 2023-10-30T13:24:52.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195ed9d640 1 -- 172.21.15.148:0/3963503590 learned_addr learned my addr 172.21.15.148:0/3963503590 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:52.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195f59e640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1960152ec0 0x7f19601517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:52.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195ed9d640 1 -- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19600c8430 msgr2=0x7f1960151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:52.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195ed9d640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19600c8430 0x7f1960151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:52.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195ed9d640 1 -- 172.21.15.148:0/3963503590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1960152ec0 msgr2=0x7f19601517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:52.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195ed9d640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1960152ec0 0x7f19601517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:52.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195ed9d640 1 -- 172.21.15.148:0/3963503590 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1948009420 con 0x7f1960071dc0 2023-10-30T13:24:52.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195e59c640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19600c8430 0x7f1960151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:24:52.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f195ed9d640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f19600c0e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f195800ba50 tx=0x7f195800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:52.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f19577fe640 1 -- 172.21.15.148:0/3963503590 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195800cbe0 con 0x7f1960071dc0 2023-10-30T13:24:52.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f19577fe640 1 -- 172.21.15.148:0/3963503590 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195800cd40 con 0x7f1960071dc0 2023-10-30T13:24:52.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f19577fe640 1 -- 172.21.15.148:0/3963503590 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958014680 con 0x7f1960071dc0 2023-10-30T13:24:52.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196013f830 con 0x7f1960071dc0 2023-10-30T13:24:52.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.912+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196013fdf0 con 0x7f1960071dc0 2023-10-30T13:24:52.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.916+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1944005260 con 0x7f1960071dc0 2023-10-30T13:24:52.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.916+0000 7f19577fe640 1 -- 172.21.15.148:0/3963503590 <== mon.2 v2:172.21.15.150:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1958007370 con 0x7f1960071dc0 2023-10-30T13:24:52.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.920+0000 7f19577fe640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1940066520 0x7f19400689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:52.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.920+0000 7f19577fe640 1 -- 172.21.15.148:0/3963503590 <== mon.2 v2:172.21.15.150:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f1958083f50 con 0x7f1960071dc0 2023-10-30T13:24:52.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.920+0000 7f195e59c640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1940066520 0x7f19400689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:52.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.920+0000 7f195e59c640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1940066520 0x7f19400689e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f19600b0db0 tx=0x7f1950009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:52.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:52.932+0000 7f19577fe640 1 -- 172.21.15.148:0/3963503590 <== mon.2 v2:172.21.15.150:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19580b5090 con 0x7f1960071dc0 2023-10-30T13:24:53.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.208+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1944005510 con 0x7f1960071dc0 2023-10-30T13:24:53.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.208+0000 7f19577fe640 1 -- 172.21.15.148:0/3963503590 <== mon.2 v2:172.21.15.150:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f1958019090 con 0x7f1960071dc0 2023-10-30T13:24:53.205 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:53.206 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:46.794165+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:53.206 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 73 2023-10-30T13:24:53.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1940066520 msgr2=0x7f19400689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:53.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1940066520 0x7f19400689e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f19600b0db0 tx=0x7f1950009240 comp rx=0 tx=0).stop 2023-10-30T13:24:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 msgr2=0x7f19600c0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f19600c0e90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f195800ba50 tx=0x7f195800bf10 comp rx=0 tx=0).stop 2023-10-30T13:24:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 shutdown_connections 2023-10-30T13:24:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1940066520 0x7f19400689e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:53.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1960152ec0 0x7f19601517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19600c8430 0x7f1960151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 --2- 172.21.15.148:0/3963503590 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1960071dc0 0x7f19600c0e90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 >> 172.21.15.148:0/3963503590 conn(0x7f19600c7910 msgr2=0x7f1960141f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 shutdown_connections 2023-10-30T13:24:53.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:53.212+0000 7f19650d6640 1 -- 172.21.15.148:0/3963503590 wait complete. 2023-10-30T13:24:53.227 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:55.228 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:55.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfecd6640 1 Processor -- start 2023-10-30T13:24:55.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfecd6640 1 -- start start 2023-10-30T13:24:55.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfecd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c05d0 0x7fadf813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:55.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfecd6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fadf80c6f60 0x7fadf813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:55.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfecd6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf80c0a50 con 0x7fadf80c6b80 2023-10-30T13:24:55.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfecd6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf806f6e0 con 0x7fadf80c6f60 2023-10-30T13:24:55.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfecd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf806f9c0 con 0x7fadf80c05d0 2023-10-30T13:24:55.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfd273640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fadf80c6f60 0x7fadf813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfd273640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fadf80c6f60 0x7fadf813f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38130/0 (socket says 172.21.15.148:38130) 2023-10-30T13:24:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfd273640 1 -- 172.21.15.148:0/656889944 learned_addr learned my addr 172.21.15.148:0/656889944 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadeffff640 1 --2- 172.21.15.148:0/656889944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c05d0 0x7fadf813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfd273640 1 -- 172.21.15.148:0/656889944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c05d0 msgr2=0x7fadf813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfd273640 1 --2- 172.21.15.148:0/656889944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c05d0 0x7fadf813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfd273640 1 -- 172.21.15.148:0/656889944 >> v1:172.21.15.148:6789/0 conn(0x7fadf80c6b80 legacy=0x7fadf80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:55.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadfd273640 1 -- 172.21.15.148:0/656889944 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf806fca0 con 0x7fadf80c6f60 2023-10-30T13:24:55.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.380+0000 7fadeffff640 1 --2- 172.21.15.148:0/656889944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c05d0 0x7fadf813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:24:55.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadfd273640 1 --2- 172.21.15.148:0/656889944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c6f60 0x7fadf813f2d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009e50 tx=0x7fadf001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815f76673f313f05 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:55.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadef7fe640 1 -- 172.21.15.148:0/656889944 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf00a57f0 con 0x7fadf80c6f60 2023-10-30T13:24:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadef7fe640 1 -- 172.21.15.148:0/656889944 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf00a5950 con 0x7fadf80c6f60 2023-10-30T13:24:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadef7fe640 1 -- 172.21.15.148:0/656889944 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf00a5ce0 con 0x7fadf80c6f60 2023-10-30T13:24:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadfecd6640 1 -- 172.21.15.148:0/656889944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c6f60 msgr2=0x7fadf813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadfecd6640 1 --2- 172.21.15.148:0/656889944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c6f60 0x7fadf813f2d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009e50 tx=0x7fadf001ca10 comp rx=0 tx=0).stop 2023-10-30T13:24:55.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadfecd6640 1 -- 172.21.15.148:0/656889944 shutdown_connections 2023-10-30T13:24:55.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadfecd6640 1 --2- 172.21.15.148:0/656889944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c6f60 0x7fadf813f2d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadfecd6640 1 --2- 172.21.15.148:0/656889944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c05d0 0x7fadf813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadfecd6640 1 -- 172.21.15.148:0/656889944 >> 172.21.15.148:0/656889944 conn(0x7fadf80c7910 msgr2=0x7fadf80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:55.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadfecd6640 1 -- 172.21.15.148:0/656889944 shutdown_connections 2023-10-30T13:24:55.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.384+0000 7fadfecd6640 1 -- 172.21.15.148:0/656889944 wait complete. 2023-10-30T13:24:55.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 Processor -- start 2023-10-30T13:24:55.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 -- start start 2023-10-30T13:24:55.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c05d0 0x7fadf81f6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:55.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c6b80 0x7fadf806a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:55.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c6f60 0x7fadf80c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:55.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf813ce90 con 0x7fadf80c05d0 2023-10-30T13:24:55.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf81f6550 con 0x7fadf80c6f60 2023-10-30T13:24:55.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf813e0c0 con 0x7fadf80c6b80 2023-10-30T13:24:55.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfca72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c05d0 0x7fadf81f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:55.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfca72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c05d0 0x7fadf81f6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38134/0 (socket says 172.21.15.148:38134) 2023-10-30T13:24:55.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfca72640 1 -- 172.21.15.148:0/583227980 learned_addr learned my addr 172.21.15.148:0/583227980 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:24:55.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfd273640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c6f60 0x7fadf80c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:55.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadeffff640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c6b80 0x7fadf806a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:55.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfca72640 1 -- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c6b80 msgr2=0x7fadf806a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:55.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfca72640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c6b80 0x7fadf806a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfca72640 1 -- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c6f60 msgr2=0x7fadf80c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:55.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfca72640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c6f60 0x7fadf80c9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfca72640 1 -- 172.21.15.148:0/583227980 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf0009b30 con 0x7fadf80c05d0 2023-10-30T13:24:55.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadeffff640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c6b80 0x7fadf806a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:24:55.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfca72640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c05d0 0x7fadf81f6010 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fadf400c530 tx=0x7fadf400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:55.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7faded7fa640 1 -- 172.21.15.148:0/583227980 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf4005220 con 0x7fadf80c05d0 2023-10-30T13:24:55.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7faded7fa640 1 -- 172.21.15.148:0/583227980 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf4005380 con 0x7fadf80c05d0 2023-10-30T13:24:55.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf8157580 con 0x7fadf80c05d0 2023-10-30T13:24:55.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7faded7fa640 1 -- 172.21.15.148:0/583227980 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf4007270 con 0x7fadf80c05d0 2023-10-30T13:24:55.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf813d4b0 con 0x7fadf80c05d0 2023-10-30T13:24:55.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.388+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf804e170 con 0x7fadf80c05d0 2023-10-30T13:24:55.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.392+0000 7faded7fa640 1 -- 172.21.15.148:0/583227980 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fadf40073d0 con 0x7fadf80c05d0 2023-10-30T13:24:55.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.392+0000 7faded7fa640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664a0 0x7fade4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:55.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.392+0000 7faded7fa640 1 -- 172.21.15.148:0/583227980 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fadf4083a00 con 0x7fadf80c05d0 2023-10-30T13:24:55.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.392+0000 7fadeffff640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664a0 0x7fade4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:55.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.392+0000 7fadeffff640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664a0 0x7fade4068960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fadf814f380 tx=0x7fade8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:55.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.396+0000 7faded7fa640 1 -- 172.21.15.148:0/583227980 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fadf4050b80 con 0x7fadf80c05d0 2023-10-30T13:24:55.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.576+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadf80705c0 con 0x7fadf80c05d0 2023-10-30T13:24:55.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.576+0000 7faded7fa640 1 -- 172.21.15.148:0/583227980 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fadf40541a0 con 0x7fadf80c05d0 2023-10-30T13:24:55.575 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:24:55.575 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:55.575 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 74 2023-10-30T13:24:55.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664a0 msgr2=0x7fade4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:55.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664a0 0x7fade4068960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fadf814f380 tx=0x7fade8008040 comp rx=0 tx=0).stop 2023-10-30T13:24:55.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c05d0 msgr2=0x7fadf81f6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:55.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c05d0 0x7fadf81f6010 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fadf400c530 tx=0x7fadf400c9f0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 shutdown_connections 2023-10-30T13:24:55.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664a0 0x7fade4068960 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c6f60 0x7fadf80c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf80c6b80 0x7fadf806a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 --2- 172.21.15.148:0/583227980 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf80c05d0 0x7fadf81f6010 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:55.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 >> 172.21.15.148:0/583227980 conn(0x7fadf80c7910 msgr2=0x7fadf8152c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:55.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 shutdown_connections 2023-10-30T13:24:55.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:24:55.580+0000 7fadfecd6640 1 -- 172.21.15.148:0/583227980 wait complete. 2023-10-30T13:24:55.587 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:55.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:55.595 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=5643 2023-10-30T13:24:55.595 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.a 2023-10-30T13:25:00.683 INFO:tasks.ceph.mds.c.smithi148.stderr:2023-10-30T13:25:00.687+0000 7f2848bdd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:25:01.596 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:25:01.597 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:25:01.597 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:25:01.601 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:25:01.601 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:01.690 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:25:01.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb606a79640 1 Processor -- start 2023-10-30T13:25:01.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb606a79640 1 -- start start 2023-10-30T13:25:01.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb606a79640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb6000c05d0 0x7fb60013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb606a79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6f60 0x7fb60013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb606a79640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6000c0a50 con 0x7fb6000c6b80 2023-10-30T13:25:01.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb606a79640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60006f6e0 con 0x7fb6000c05d0 2023-10-30T13:25:01.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb606a79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60006f9c0 con 0x7fb6000c6f60 2023-10-30T13:25:01.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb5fffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb6000c05d0 0x7fb60013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb5fffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb6000c05d0 0x7fb60013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38148/0 (socket says 172.21.15.148:38148) 2023-10-30T13:25:01.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb5fffff640 1 -- 172.21.15.148:0/4107655526 learned_addr learned my addr 172.21.15.148:0/4107655526 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:25:01.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb605016640 1 --2- 172.21.15.148:0/4107655526 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6f60 0x7fb60013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb5fffff640 1 -- 172.21.15.148:0/4107655526 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6f60 msgr2=0x7fb60013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:01.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb5fffff640 1 --2- 172.21.15.148:0/4107655526 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6f60 0x7fb60013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb5fffff640 1 -- 172.21.15.148:0/4107655526 >> v1:172.21.15.148:6789/0 conn(0x7fb6000c6b80 legacy=0x7fb6000bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb5fffff640 1 -- 172.21.15.148:0/4107655526 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60006fca0 con 0x7fb6000c05d0 2023-10-30T13:25:01.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.747+0000 7fb5ff7fe640 1 -- 172.21.15.148:0/4107655526 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964237317 0 0) 0x7fb6000c0a50 con 0x7fb6000c6b80 2023-10-30T13:25:01.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb5fffff640 1 --2- 172.21.15.148:0/4107655526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 0x7fb60013ed90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009870 tx=0x7fb5ec01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76958b06cd9c470 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:01.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb5ff7fe640 1 -- 172.21.15.148:0/4107655526 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec0ab650 con 0x7fb6000c05d0 2023-10-30T13:25:01.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb5ff7fe640 1 -- 172.21.15.148:0/4107655526 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ec0ab7b0 con 0x7fb6000c05d0 2023-10-30T13:25:01.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb5ff7fe640 1 -- 172.21.15.148:0/4107655526 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec0abb40 con 0x7fb6000c05d0 2023-10-30T13:25:01.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb606a79640 1 -- 172.21.15.148:0/4107655526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 msgr2=0x7fb60013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb606a79640 1 --2- 172.21.15.148:0/4107655526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 0x7fb60013ed90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009870 tx=0x7fb5ec01c540 comp rx=0 tx=0).stop 2023-10-30T13:25:01.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb606a79640 1 -- 172.21.15.148:0/4107655526 shutdown_connections 2023-10-30T13:25:01.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb606a79640 1 --2- 172.21.15.148:0/4107655526 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 0x7fb60013ed90 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb606a79640 1 --2- 172.21.15.148:0/4107655526 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6f60 0x7fb60013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb606a79640 1 -- 172.21.15.148:0/4107655526 >> 172.21.15.148:0/4107655526 conn(0x7fb6000c7910 msgr2=0x7fb6000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:01.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb606a79640 1 -- 172.21.15.148:0/4107655526 shutdown_connections 2023-10-30T13:25:01.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.751+0000 7fb606a79640 1 -- 172.21.15.148:0/4107655526 wait complete. 2023-10-30T13:25:01.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 Processor -- start 2023-10-30T13:25:01.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 -- start start 2023-10-30T13:25:01.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 0x7fb6001f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6b80 0x7fb600133960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6000c6f60 0x7fb6000ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb600133ea0 con 0x7fb6000c05d0 2023-10-30T13:25:01.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6001f3510 con 0x7fb6000c6f60 2023-10-30T13:25:01.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6001f37f0 con 0x7fb6000c6b80 2023-10-30T13:25:01.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb604815640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 0x7fb6001f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb604815640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 0x7fb6001f2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38156/0 (socket says 172.21.15.148:38156) 2023-10-30T13:25:01.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb604815640 1 -- 172.21.15.148:0/262368431 learned_addr learned my addr 172.21.15.148:0/262368431 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:25:01.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb605016640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6000c6f60 0x7fb6000ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb5fffff640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6b80 0x7fb600133960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb604815640 1 -- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6b80 msgr2=0x7fb600133960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb604815640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6b80 0x7fb600133960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb604815640 1 -- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6000c6f60 msgr2=0x7fb6000ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb604815640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6000c6f60 0x7fb6000ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb604815640 1 -- 172.21.15.148:0/262368431 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec0094d0 con 0x7fb6000c05d0 2023-10-30T13:25:01.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb604815640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 0x7fb6001f2ed0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c4d0 tx=0x7fb5f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:01.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb5fd7fa640 1 -- 172.21.15.148:0/262368431 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4005480 con 0x7fb6000c05d0 2023-10-30T13:25:01.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb5fd7fa640 1 -- 172.21.15.148:0/262368431 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f4003600 con 0x7fb6000c05d0 2023-10-30T13:25:01.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb5fd7fa640 1 -- 172.21.15.148:0/262368431 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f40039b0 con 0x7fb6000c05d0 2023-10-30T13:25:01.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6001f3ad0 con 0x7fb6000c05d0 2023-10-30T13:25:01.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.755+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6000c9b10 con 0x7fb6000c05d0 2023-10-30T13:25:01.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.759+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60006b0c0 con 0x7fb6000c05d0 2023-10-30T13:25:01.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.759+0000 7fb5fd7fa640 1 -- 172.21.15.148:0/262368431 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5f4003b10 con 0x7fb6000c05d0 2023-10-30T13:25:01.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.759+0000 7fb5fd7fa640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5e40664a0 0x7fb5e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:01.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.759+0000 7fb5fd7fa640 1 -- 172.21.15.148:0/262368431 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fb5f4083820 con 0x7fb6000c05d0 2023-10-30T13:25:01.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.759+0000 7fb5fffff640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5e40664a0 0x7fb5e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:01.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.759+0000 7fb5fffff640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5e40664a0 0x7fb5e4068960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0af000 tx=0x7fb5ec0b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:01.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.763+0000 7fb5fd7fa640 1 -- 172.21.15.148:0/262368431 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5f40509f0 con 0x7fb6000c05d0 2023-10-30T13:25:01.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.947+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb60004e170 con 0x7fb6000c05d0 2023-10-30T13:25:01.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.947+0000 7fb5fd7fa640 1 -- 172.21.15.148:0/262368431 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fb5f4017020 con 0x7fb6000c05d0 2023-10-30T13:25:01.945 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:25:01.946 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:01.946 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 74 2023-10-30T13:25:01.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5e40664a0 msgr2=0x7fb5e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5e40664a0 0x7fb5e4068960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0af000 tx=0x7fb5ec0b6040 comp rx=0 tx=0).stop 2023-10-30T13:25:01.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 msgr2=0x7fb6001f2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:01.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 0x7fb6001f2ed0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c4d0 tx=0x7fb5f400c990 comp rx=0 tx=0).stop 2023-10-30T13:25:01.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 shutdown_connections 2023-10-30T13:25:01.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5e40664a0 0x7fb5e4068960 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6000c6f60 0x7fb6000ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6000c6b80 0x7fb600133960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 --2- 172.21.15.148:0/262368431 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb6000c05d0 0x7fb6001f2ed0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:01.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 >> 172.21.15.148:0/262368431 conn(0x7fb6000c7910 msgr2=0x7fb6000c1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:01.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 shutdown_connections 2023-10-30T13:25:01.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:01.951+0000 7fb606a79640 1 -- 172.21.15.148:0/262368431 wait complete. 2023-10-30T13:25:01.966 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:25:03.967 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:25:04.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15c291640 1 Processor -- start 2023-10-30T13:25:04.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15c291640 1 -- start start 2023-10-30T13:25:04.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15c291640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff1541538a0 0x7ff15415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:04.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15c291640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540cf160 0x7ff1540cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:04.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15c291640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540cb800 con 0x7ff1540ca0e0 2023-10-30T13:25:04.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15c291640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15415dc30 con 0x7ff1541538a0 2023-10-30T13:25:04.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15c291640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540cb330 con 0x7ff1540cf160 2023-10-30T13:25:04.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15a02d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff1541538a0 0x7ff15415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:04.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15a02d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff1541538a0 0x7ff15415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33412/0 (socket says 172.21.15.148:33412) 2023-10-30T13:25:04.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15a02d640 1 -- 172.21.15.148:0/779608419 learned_addr learned my addr 172.21.15.148:0/779608419 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:25:04.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.115+0000 7ff15a82e640 1 --2- 172.21.15.148:0/779608419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540cf160 0x7ff1540cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:04.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15a02d640 1 -- 172.21.15.148:0/779608419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540cf160 msgr2=0x7ff1540cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:04.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15a02d640 1 --2- 172.21.15.148:0/779608419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540cf160 0x7ff1540cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15a02d640 1 -- 172.21.15.148:0/779608419 >> v1:172.21.15.150:6789/0 conn(0x7ff1540ca0e0 legacy=0x7ff15415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:04.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15a02d640 1 -- 172.21.15.148:0/779608419 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540cad70 con 0x7ff1541538a0 2023-10-30T13:25:04.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15a02d640 1 --2- 172.21.15.148:0/779608419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1541538a0 0x7ff15415d6f0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff150009870 tx=0x7ff15001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26f1388cb1e395b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:04.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15902b640 1 -- 172.21.15.148:0/779608419 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1500a8a30 con 0x7ff1541538a0 2023-10-30T13:25:04.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15902b640 1 -- 172.21.15.148:0/779608419 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1500a8b90 con 0x7ff1541538a0 2023-10-30T13:25:04.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15902b640 1 -- 172.21.15.148:0/779608419 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1500b0440 con 0x7ff1541538a0 2023-10-30T13:25:04.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15c291640 1 -- 172.21.15.148:0/779608419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1541538a0 msgr2=0x7ff15415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:04.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15c291640 1 --2- 172.21.15.148:0/779608419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1541538a0 0x7ff15415d6f0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff150009870 tx=0x7ff15001ed30 comp rx=0 tx=0).stop 2023-10-30T13:25:04.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15c291640 1 -- 172.21.15.148:0/779608419 shutdown_connections 2023-10-30T13:25:04.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15c291640 1 --2- 172.21.15.148:0/779608419 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1540cf160 0x7ff1540cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15c291640 1 --2- 172.21.15.148:0/779608419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1541538a0 0x7ff15415d6f0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.119+0000 7ff15c291640 1 -- 172.21.15.148:0/779608419 >> 172.21.15.148:0/779608419 conn(0x7ff15406bfe0 msgr2=0x7ff15406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:04.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 -- 172.21.15.148:0/779608419 shutdown_connections 2023-10-30T13:25:04.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 -- 172.21.15.148:0/779608419 wait complete. 2023-10-30T13:25:04.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 Processor -- start 2023-10-30T13:25:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 -- start start 2023-10-30T13:25:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ca0e0 0x7ff154157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1540cf160 0x7ff1541519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1541538a0 0x7ff1541e38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541f22b0 con 0x7ff1540ca0e0 2023-10-30T13:25:04.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541587d0 con 0x7ff1540cf160 2023-10-30T13:25:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1541f2500 con 0x7ff1541538a0 2023-10-30T13:25:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a02d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ca0e0 0x7ff154157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a02d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ca0e0 0x7ff154157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33418/0 (socket says 172.21.15.148:33418) 2023-10-30T13:25:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a02d640 1 -- 172.21.15.148:0/3992158709 learned_addr learned my addr 172.21.15.148:0/3992158709 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:25:04.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15982c640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1540cf160 0x7ff1541519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a82e640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1541538a0 0x7ff1541e38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a02d640 1 -- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1541538a0 msgr2=0x7ff1541e38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a02d640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1541538a0 0x7ff1541e38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a02d640 1 -- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1540cf160 msgr2=0x7ff1541519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:04.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a02d640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1540cf160 0x7ff1541519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a02d640 1 -- 172.21.15.148:0/3992158709 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1500094d0 con 0x7ff1540ca0e0 2023-10-30T13:25:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15982c640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1540cf160 0x7ff1541519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15a02d640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ca0e0 0x7ff154157eb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff1500b3040 tx=0x7ff15001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff14affd640 1 -- 172.21.15.148:0/3992158709 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1500be070 con 0x7ff1540ca0e0 2023-10-30T13:25:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff14affd640 1 -- 172.21.15.148:0/3992158709 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1500ba400 con 0x7ff1540ca0e0 2023-10-30T13:25:04.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1541f27e0 con 0x7ff1540ca0e0 2023-10-30T13:25:04.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff14affd640 1 -- 172.21.15.148:0/3992158709 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1500ba7d0 con 0x7ff1540ca0e0 2023-10-30T13:25:04.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.123+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1541f2da0 con 0x7ff1540ca0e0 2023-10-30T13:25:04.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.127+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1540b2a50 con 0x7ff1540ca0e0 2023-10-30T13:25:04.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.127+0000 7ff14affd640 1 -- 172.21.15.148:0/3992158709 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1500c3410 con 0x7ff1540ca0e0 2023-10-30T13:25:04.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.127+0000 7ff14affd640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff1380664a0 0x7ff138068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:04.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.127+0000 7ff14affd640 1 -- 172.21.15.148:0/3992158709 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7ff150121c80 con 0x7ff1540ca0e0 2023-10-30T13:25:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.127+0000 7ff15982c640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff1380664a0 0x7ff138068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.127+0000 7ff15982c640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff1380664a0 0x7ff138068960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff154149770 tx=0x7ff144009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:04.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.131+0000 7ff14affd640 1 -- 172.21.15.148:0/3992158709 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1500eeea0 con 0x7ff1540ca0e0 2023-10-30T13:25:04.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.327+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff15404e170 con 0x7ff1540ca0e0 2023-10-30T13:25:04.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff14affd640 1 -- 172.21.15.148:0/3992158709 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+6511 (secure 0 0 0) 0x7ff1500f24c0 con 0x7ff1540ca0e0 2023-10-30T13:25:04.327 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:25:04.327 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:25:04.328 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 75 2023-10-30T13:25:04.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff1380664a0 msgr2=0x7ff138068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:04.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff1380664a0 0x7ff138068960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff154149770 tx=0x7ff144009040 comp rx=0 tx=0).stop 2023-10-30T13:25:04.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ca0e0 msgr2=0x7ff154157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:04.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ca0e0 0x7ff154157eb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff1500b3040 tx=0x7ff15001bad0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 shutdown_connections 2023-10-30T13:25:04.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff1380664a0 0x7ff138068960 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff1541538a0 0x7ff1541e38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1540cf160 0x7ff1541519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 --2- 172.21.15.148:0/3992158709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ca0e0 0x7ff154157eb0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:04.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 >> 172.21.15.148:0/3992158709 conn(0x7ff15406bfe0 msgr2=0x7ff15414fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:04.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 shutdown_connections 2023-10-30T13:25:04.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:25:04.331+0000 7ff15c291640 1 -- 172.21.15.148:0/3992158709 wait complete. 2023-10-30T13:25:04.356 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:25:04.357 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-10-30T13:25:25.641 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:25.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:55.709 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:55.713 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:14.442 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 0% (97/12071) Updating files: 1% (121/12071) Updating files: 2% (242/12071) Updating files: 2% (301/12071) Updating files: 3% (363/12071) Updating files: 4% (483/12071) Updating files: 5% (604/12071) Updating files: 5% (615/12071) Updating files: 6% (725/12071) Updating files: 6% (731/12071) Updating files: 7% (845/12071) Updating files: 7% (850/12071) Updating files: 7% (937/12071) Updating files: 8% (966/12071) Updating files: 8% (968/12071) Updating files: 8% (996/12071) Updating files: 9% (1087/12071) Updating files: 9% (1092/12071) Updating files: 10% (1208/12071) Updating files: 11% (1328/12071) Updating files: 11% (1338/12071) Updating files: 11% (1381/12071) Updating files: 12% (1449/12071) Updating files: 12% (1494/12071) Updating files: 13% (1570/12071) Updating files: 13% (1624/12071) Updating files: 14% (1690/12071) Updating files: 15% (1811/12071) Updating files: 15% (1821/12071) Updating files: 16% (1932/12071) Updating files: 17% (2053/12071) Updating files: 17% (2125/12071) Updating files: 18% (2173/12071) Updating files: 18% (2279/12071) Updating files: 19% (2294/12071) Updating files: 20% (2415/12071) Updating files: 20% (2473/12071) Updating files: 21% (2535/12071) Updating files: 21% (2586/12071) Updating files: 22% (2656/12071) Updating files: 22% (2664/12071) Updating files: 22% (2690/12071) Updating files: 22% (2733/12071) Updating files: 23% (2777/12071) Updating files: 24% (2898/12071) Updating files: 24% (2949/12071) Updating files: 24% (2964/12071) Updating files: 25% (3018/12071) Updating files: 26% (3139/12071) Updating files: 26% (3225/12071) Updating files: 27% (3260/12071) Updating files: 28% (3380/12071) Updating files: 28% (3451/12071) Updating files: 29% (3501/12071) Updating files: 29% (3509/12071) Updating files: 29% (3523/12071) Updating files: 30% (3622/12071) Updating files: 30% (3637/12071) Updating files: 30% (3705/12071) Updating files: 31% (3743/12071) Updating files: 31% (3799/12071) Updating files: 32% (3863/12071) Updating files: 33% (3984/12071) Updating files: 33% (4071/12071) Updating files: 34% (4105/12071) Updating files: 35% (4225/12071) Updating files: 35% (4266/12071) Updating files: 36% (4346/12071) Updating files: 37% (4467/12071) Updating files: 38% (4587/12071) Updating files: 38% (4630/12071) Updating files: 39% (4708/12071) Updating files: 39% (4719/12071) Updating files: 39% (4768/12071) Updating files: 40% (4829/12071) Updating files: 41% (4950/12071) Updating files: 42% (5070/12071) Updating files: 43% (5191/12071) Updating files: 43% (5217/12071) Updating files: 43% (5242/12071) Updating files: 43% (5247/12071) Updating files: 44% (5312/12071) Updating files: 44% (5319/12071) Updating files: 44% (5389/12071) Updating files: 45% (5432/12071) Updating files: 46% (5553/12071) Updating files: 47% (5674/12071) Updating files: 47% (5678/12071) Updating files: 48% (5795/12071) Updating files: 49% (5915/12071) Updating files: 49% (5955/12071) Updating files: 49% (6006/12071) Updating files: 50% (6036/12071) Updating files: 50% (6046/12071) Updating files: 51% (6157/12071) Updating files: 51% (6243/12071) Updating files: 52% (6277/12071) Updating files: 52% (6299/12071) Updating files: 53% (6398/12071) Updating files: 54% (6519/12071) Updating files: 54% (6544/12071) Updating files: 55% (6640/12071) Updating files: 55% (6744/12071) Updating files: 56% (6760/12071) Updating files: 57% (6881/12071) Updating files: 58% (7002/12071) Updating files: 58% (7006/12071) Updating files: 59% (7122/12071) Updating files: 59% (7198/12071) Updating files: 60% (7243/12071) Updating files: 61% (7364/12071) Updating files: 61% (7403/12071) Updating files: 61% (7470/12071) Updating files: 62% (7485/12071) Updating files: 62% (7557/12071) Updating files: 63% (7605/12071) Updating files: 63% (7713/12071) Updating files: 64% (7726/12071) Updating files: 64% (7787/12071) Updating files: 65% (7847/12071) Updating files: 65% (7934/12071) Updating files: 66% (7967/12071) Updating files: 66% (8020/12071) Updating files: 67% (8088/12071) Updating files: 67% (8159/12071) Updating files: 68% (8209/12071) Updating files: 69% (8329/12071) Updating files: 69% (8332/12071) Updating files: 69% (8397/12071) Updating files: 70% (8450/12071) Updating files: 71% (8571/12071) Updating files: 71% (8573/12071) Updating files: 71% (8601/12071) Updating files: 72% (8692/12071) Updating files: 72% (8728/12071) Updating files: 72% (8767/12071) Updating files: 73% (8812/12071) Updating files: 73% (8840/12071) Updating files: 73% (8931/12071) Updating files: 74% (8933/12071) Updating files: 74% (8945/12071) Updating files: 74% (9002/12071) Updating files: 74% (9047/12071) Updating files: 75% (9054/12071) Updating files: 75% (9105/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 77% (9314/12071) Updating files: 77% (9394/12071) Updating files: 78% (9416/12071) Updating files: 79% (9537/12071) Updating files: 79% (9572/12071) Updating files: 80% (9657/12071) Updating files: 80% (9702/12071) Updating files: 80% (9741/12071) Updating files: 80% (9751/12071) Updating files: 81% (9778/12071) Updating files: 81% (9791/12071) Updating files: 81% (9867/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 83% (10047/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 85% (10347/12071) Updating files: 86% (10382/12071) Updating files: 87% (10502/12071) Updating files: 87% (10620/12071) Updating files: 88% (10623/12071) Updating files: 88% (10731/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 90% (10885/12071) Updating files: 91% (10985/12071) Updating files: 92% (11106/12071) Updating files: 92% (11115/12071) Updating files: 92% (11195/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 95% (11468/12071) Updating files: 95% (11476/12071) Updating files: 95% (11493/12071) Updating files: 95% (11554/12071) Updating files: 96% (11589/12071) Updating files: 96% (11659/12071) Updating files: 96% (11700/12071) Updating files: 97% (11709/12071) Updating files: 97% (11738/12071) Updating files: 98% (11830/12071) Updating files: 98% (11937/12071) Updating files: 99% (11951/12071) Updating files: 99% (11998/12071) Updating files: 99% (12011/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T13:26:14.502 INFO:tasks.workunit.client.0.smithi148.stderr:+ trap - EXIT 2023-10-30T13:26:14.503 INFO:tasks.workunit.client.0.smithi148.stderr:+ cd ceph 2023-10-30T13:26:14.503 INFO:tasks.workunit.client.0.smithi148.stderr:+ seq 1 90 2023-10-30T13:26:14.504 INFO:tasks.workunit.client.0.smithi148.stderr:+ versions=1 2023-10-30T13:26:14.504 INFO:tasks.workunit.client.0.smithi148.stderr:2 2023-10-30T13:26:14.504 INFO:tasks.workunit.client.0.smithi148.stderr:3 2023-10-30T13:26:14.505 INFO:tasks.workunit.client.0.smithi148.stderr:4 2023-10-30T13:26:14.505 INFO:tasks.workunit.client.0.smithi148.stderr:5 2023-10-30T13:26:14.505 INFO:tasks.workunit.client.0.smithi148.stderr:6 2023-10-30T13:26:14.505 INFO:tasks.workunit.client.0.smithi148.stderr:7 2023-10-30T13:26:14.505 INFO:tasks.workunit.client.0.smithi148.stderr:8 2023-10-30T13:26:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:9 2023-10-30T13:26:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:10 2023-10-30T13:26:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:11 2023-10-30T13:26:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:12 2023-10-30T13:26:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:13 2023-10-30T13:26:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:14 2023-10-30T13:26:14.507 INFO:tasks.workunit.client.0.smithi148.stderr:15 2023-10-30T13:26:14.507 INFO:tasks.workunit.client.0.smithi148.stderr:16 2023-10-30T13:26:14.507 INFO:tasks.workunit.client.0.smithi148.stderr:17 2023-10-30T13:26:14.507 INFO:tasks.workunit.client.0.smithi148.stderr:18 2023-10-30T13:26:14.507 INFO:tasks.workunit.client.0.smithi148.stderr:19 2023-10-30T13:26:14.508 INFO:tasks.workunit.client.0.smithi148.stderr:20 2023-10-30T13:26:14.508 INFO:tasks.workunit.client.0.smithi148.stderr:21 2023-10-30T13:26:14.508 INFO:tasks.workunit.client.0.smithi148.stderr:22 2023-10-30T13:26:14.508 INFO:tasks.workunit.client.0.smithi148.stderr:23 2023-10-30T13:26:14.508 INFO:tasks.workunit.client.0.smithi148.stderr:24 2023-10-30T13:26:14.508 INFO:tasks.workunit.client.0.smithi148.stderr:25 2023-10-30T13:26:14.509 INFO:tasks.workunit.client.0.smithi148.stderr:26 2023-10-30T13:26:14.509 INFO:tasks.workunit.client.0.smithi148.stderr:27 2023-10-30T13:26:14.509 INFO:tasks.workunit.client.0.smithi148.stderr:28 2023-10-30T13:26:14.509 INFO:tasks.workunit.client.0.smithi148.stderr:29 2023-10-30T13:26:14.509 INFO:tasks.workunit.client.0.smithi148.stderr:30 2023-10-30T13:26:14.509 INFO:tasks.workunit.client.0.smithi148.stderr:31 2023-10-30T13:26:14.510 INFO:tasks.workunit.client.0.smithi148.stderr:32 2023-10-30T13:26:14.510 INFO:tasks.workunit.client.0.smithi148.stderr:33 2023-10-30T13:26:14.510 INFO:tasks.workunit.client.0.smithi148.stderr:34 2023-10-30T13:26:14.510 INFO:tasks.workunit.client.0.smithi148.stderr:35 2023-10-30T13:26:14.510 INFO:tasks.workunit.client.0.smithi148.stderr:36 2023-10-30T13:26:14.511 INFO:tasks.workunit.client.0.smithi148.stderr:37 2023-10-30T13:26:14.511 INFO:tasks.workunit.client.0.smithi148.stderr:38 2023-10-30T13:26:14.511 INFO:tasks.workunit.client.0.smithi148.stderr:39 2023-10-30T13:26:14.511 INFO:tasks.workunit.client.0.smithi148.stderr:40 2023-10-30T13:26:14.511 INFO:tasks.workunit.client.0.smithi148.stderr:41 2023-10-30T13:26:14.511 INFO:tasks.workunit.client.0.smithi148.stderr:42 2023-10-30T13:26:14.512 INFO:tasks.workunit.client.0.smithi148.stderr:43 2023-10-30T13:26:14.512 INFO:tasks.workunit.client.0.smithi148.stderr:44 2023-10-30T13:26:14.512 INFO:tasks.workunit.client.0.smithi148.stderr:45 2023-10-30T13:26:14.512 INFO:tasks.workunit.client.0.smithi148.stderr:46 2023-10-30T13:26:14.512 INFO:tasks.workunit.client.0.smithi148.stderr:47 2023-10-30T13:26:14.513 INFO:tasks.workunit.client.0.smithi148.stderr:48 2023-10-30T13:26:14.513 INFO:tasks.workunit.client.0.smithi148.stderr:49 2023-10-30T13:26:14.513 INFO:tasks.workunit.client.0.smithi148.stderr:50 2023-10-30T13:26:14.513 INFO:tasks.workunit.client.0.smithi148.stderr:51 2023-10-30T13:26:14.513 INFO:tasks.workunit.client.0.smithi148.stderr:52 2023-10-30T13:26:14.513 INFO:tasks.workunit.client.0.smithi148.stderr:53 2023-10-30T13:26:14.514 INFO:tasks.workunit.client.0.smithi148.stderr:54 2023-10-30T13:26:14.514 INFO:tasks.workunit.client.0.smithi148.stderr:55 2023-10-30T13:26:14.514 INFO:tasks.workunit.client.0.smithi148.stderr:56 2023-10-30T13:26:14.514 INFO:tasks.workunit.client.0.smithi148.stderr:57 2023-10-30T13:26:14.514 INFO:tasks.workunit.client.0.smithi148.stderr:58 2023-10-30T13:26:14.515 INFO:tasks.workunit.client.0.smithi148.stderr:59 2023-10-30T13:26:14.515 INFO:tasks.workunit.client.0.smithi148.stderr:60 2023-10-30T13:26:14.515 INFO:tasks.workunit.client.0.smithi148.stderr:61 2023-10-30T13:26:14.515 INFO:tasks.workunit.client.0.smithi148.stderr:62 2023-10-30T13:26:14.515 INFO:tasks.workunit.client.0.smithi148.stderr:63 2023-10-30T13:26:14.515 INFO:tasks.workunit.client.0.smithi148.stderr:64 2023-10-30T13:26:14.516 INFO:tasks.workunit.client.0.smithi148.stderr:65 2023-10-30T13:26:14.516 INFO:tasks.workunit.client.0.smithi148.stderr:66 2023-10-30T13:26:14.516 INFO:tasks.workunit.client.0.smithi148.stderr:67 2023-10-30T13:26:14.516 INFO:tasks.workunit.client.0.smithi148.stderr:68 2023-10-30T13:26:14.516 INFO:tasks.workunit.client.0.smithi148.stderr:69 2023-10-30T13:26:14.516 INFO:tasks.workunit.client.0.smithi148.stderr:70 2023-10-30T13:26:14.517 INFO:tasks.workunit.client.0.smithi148.stderr:71 2023-10-30T13:26:14.517 INFO:tasks.workunit.client.0.smithi148.stderr:72 2023-10-30T13:26:14.517 INFO:tasks.workunit.client.0.smithi148.stderr:73 2023-10-30T13:26:14.517 INFO:tasks.workunit.client.0.smithi148.stderr:74 2023-10-30T13:26:14.517 INFO:tasks.workunit.client.0.smithi148.stderr:75 2023-10-30T13:26:14.517 INFO:tasks.workunit.client.0.smithi148.stderr:76 2023-10-30T13:26:14.518 INFO:tasks.workunit.client.0.smithi148.stderr:77 2023-10-30T13:26:14.518 INFO:tasks.workunit.client.0.smithi148.stderr:78 2023-10-30T13:26:14.518 INFO:tasks.workunit.client.0.smithi148.stderr:79 2023-10-30T13:26:14.518 INFO:tasks.workunit.client.0.smithi148.stderr:80 2023-10-30T13:26:14.518 INFO:tasks.workunit.client.0.smithi148.stderr:81 2023-10-30T13:26:14.519 INFO:tasks.workunit.client.0.smithi148.stderr:82 2023-10-30T13:26:14.519 INFO:tasks.workunit.client.0.smithi148.stderr:83 2023-10-30T13:26:14.519 INFO:tasks.workunit.client.0.smithi148.stderr:84 2023-10-30T13:26:14.519 INFO:tasks.workunit.client.0.smithi148.stderr:85 2023-10-30T13:26:14.519 INFO:tasks.workunit.client.0.smithi148.stderr:86 2023-10-30T13:26:14.519 INFO:tasks.workunit.client.0.smithi148.stderr:87 2023-10-30T13:26:14.520 INFO:tasks.workunit.client.0.smithi148.stderr:88 2023-10-30T13:26:14.520 INFO:tasks.workunit.client.0.smithi148.stderr:89 2023-10-30T13:26:14.520 INFO:tasks.workunit.client.0.smithi148.stderr:90 2023-10-30T13:26:14.520 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 1 -eq 48 ] 2023-10-30T13:26:14.520 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.1 2023-10-30T13:26:14.520 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.1 2023-10-30T13:26:14.521 INFO:tasks.workunit.client.0.smithi148.stdout:v0.1 2023-10-30T13:26:14.521 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.1 2023-10-30T13:26:25.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:25.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:28.357 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:28.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82edc67640 1 Processor -- start 2023-10-30T13:26:28.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82edc67640 1 -- start start 2023-10-30T13:26:28.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82edc67640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f82e80c6840 0x7f82e815a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82edc67640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82e80c6c20 0x7f82e8156590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82edc67640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e80c3b30 con 0x7f82e80c3640 2023-10-30T13:26:28.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82edc67640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e80c2aa0 con 0x7f82e80c6c20 2023-10-30T13:26:28.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82edc67640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e80c3df0 con 0x7f82e80c6840 2023-10-30T13:26:28.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82ed466640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82e80c6c20 0x7f82e8156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82ed466640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82e80c6c20 0x7f82e8156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38898/0 (socket says 172.21.15.148:38898) 2023-10-30T13:26:28.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82ed466640 1 -- 172.21.15.148:0/1911523416 learned_addr learned my addr 172.21.15.148:0/1911523416 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:28.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82ecc65640 1 --2- 172.21.15.148:0/1911523416 >> v2:172.21.15.150:3300/0 conn(0x7f82e80c6840 0x7f82e815a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82ed466640 1 -- 172.21.15.148:0/1911523416 >> v2:172.21.15.150:3300/0 conn(0x7f82e80c6840 msgr2=0x7f82e815a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:28.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82ed466640 1 --2- 172.21.15.148:0/1911523416 >> v2:172.21.15.150:3300/0 conn(0x7f82e80c6840 0x7f82e815a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82ed466640 1 -- 172.21.15.148:0/1911523416 >> v1:172.21.15.150:6789/0 conn(0x7f82e80c3640 legacy=0x7f82e8153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82ed466640 1 -- 172.21.15.148:0/1911523416 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e8159ba0 con 0x7f82e80c6c20 2023-10-30T13:26:28.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.501+0000 7f82ecc65640 1 --2- 172.21.15.148:0/1911523416 >> v2:172.21.15.150:3300/0 conn(0x7f82e80c6840 0x7f82e815a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:28.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82ed466640 1 --2- 172.21.15.148:0/1911523416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c6c20 0x7f82e8156590 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f82e806b360 tx=0x7f82e001ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae43e7d31f44bbef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:28.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82e77fe640 1 -- 172.21.15.148:0/1911523416 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00a78c0 con 0x7f82e80c6c20 2023-10-30T13:26:28.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82e77fe640 1 -- 172.21.15.148:0/1911523416 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e00a7a20 con 0x7f82e80c6c20 2023-10-30T13:26:28.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82e77fe640 1 -- 172.21.15.148:0/1911523416 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00a7db0 con 0x7f82e80c6c20 2023-10-30T13:26:28.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 -- 172.21.15.148:0/1911523416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c6c20 msgr2=0x7f82e8156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 --2- 172.21.15.148:0/1911523416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c6c20 0x7f82e8156590 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f82e806b360 tx=0x7f82e001ebe0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 -- 172.21.15.148:0/1911523416 shutdown_connections 2023-10-30T13:26:28.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 --2- 172.21.15.148:0/1911523416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c6c20 0x7f82e8156590 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 --2- 172.21.15.148:0/1911523416 >> v2:172.21.15.150:3300/0 conn(0x7f82e80c6840 0x7f82e815a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 -- 172.21.15.148:0/1911523416 >> 172.21.15.148:0/1911523416 conn(0x7f82e80c75d0 msgr2=0x7f82e80c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:28.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 -- 172.21.15.148:0/1911523416 shutdown_connections 2023-10-30T13:26:28.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 -- 172.21.15.148:0/1911523416 wait complete. 2023-10-30T13:26:28.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 Processor -- start 2023-10-30T13:26:28.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.505+0000 7f82edc67640 1 -- start start 2023-10-30T13:26:28.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82edc67640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c3640 0x7f82e80c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82edc67640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e80c6840 0x7f82e8157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82edc67640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82e80c6c20 0x7f82e81f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82edc67640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e814f010 con 0x7f82e80c3640 2023-10-30T13:26:28.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82edc67640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e81503c0 con 0x7f82e80c6840 2023-10-30T13:26:28.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82edc67640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80c4ff0 con 0x7f82e80c6c20 2023-10-30T13:26:28.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ecc65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c3640 0x7f82e80c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ecc65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c3640 0x7f82e80c4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38908/0 (socket says 172.21.15.148:38908) 2023-10-30T13:26:28.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ecc65640 1 -- 172.21.15.148:0/255163780 learned_addr learned my addr 172.21.15.148:0/255163780 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:28.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82e7fff640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e80c6840 0x7f82e8157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ed466640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82e80c6c20 0x7f82e81f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ecc65640 1 -- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82e80c6c20 msgr2=0x7f82e81f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ecc65640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82e80c6c20 0x7f82e81f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ecc65640 1 -- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e80c6840 msgr2=0x7f82e8157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ecc65640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e80c6840 0x7f82e8157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ecc65640 1 -- 172.21.15.148:0/255163780 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e0009b30 con 0x7f82e80c3640 2023-10-30T13:26:28.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82e7fff640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e80c6840 0x7f82e8157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:28.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82ecc65640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c3640 0x7f82e80c4540 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00e7e0 tx=0x7f82dc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:28.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82e57fa640 1 -- 172.21.15.148:0/255163780 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0059a0 con 0x7f82e80c3640 2023-10-30T13:26:28.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82e57fa640 1 -- 172.21.15.148:0/255163780 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc005b00 con 0x7f82e80c3640 2023-10-30T13:26:28.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e8158750 con 0x7f82e80c3640 2023-10-30T13:26:28.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82e57fa640 1 -- 172.21.15.148:0/255163780 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc005e90 con 0x7f82e80c3640 2023-10-30T13:26:28.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e814fd70 con 0x7f82e80c3640 2023-10-30T13:26:28.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e804e170 con 0x7f82e80c3640 2023-10-30T13:26:28.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.509+0000 7f82e57fa640 1 -- 172.21.15.148:0/255163780 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82dc004030 con 0x7f82e80c3640 2023-10-30T13:26:28.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.513+0000 7f82e57fa640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82cc0664a0 0x7f82cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.513+0000 7f82e57fa640 1 -- 172.21.15.148:0/255163780 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f82dc084350 con 0x7f82e80c3640 2023-10-30T13:26:28.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.513+0000 7f82e7fff640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82cc0664a0 0x7f82cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.513+0000 7f82e7fff640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82cc0664a0 0x7f82cc068960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f82e81599f0 tx=0x7f82d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:28.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.517+0000 7f82e57fa640 1 -- 172.21.15.148:0/255163780 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82dc0513b0 con 0x7f82e80c3640 2023-10-30T13:26:28.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.721+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82e814af10 con 0x7f82e80c3640 2023-10-30T13:26:28.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82e57fa640 1 -- 172.21.15.148:0/255163780 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f82dc0549d0 con 0x7f82e80c3640 2023-10-30T13:26:28.719 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:28.720 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:28.720 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 75 2023-10-30T13:26:28.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82cc0664a0 msgr2=0x7f82cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82cc0664a0 0x7f82cc068960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f82e81599f0 tx=0x7f82d4009040 comp rx=0 tx=0).stop 2023-10-30T13:26:28.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c3640 msgr2=0x7f82e80c4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c3640 0x7f82e80c4540 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00e7e0 tx=0x7f82dc00eca0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 shutdown_connections 2023-10-30T13:26:28.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f82cc0664a0 0x7f82cc068960 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82e80c6c20 0x7f82e81f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e80c6840 0x7f82e8157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 --2- 172.21.15.148:0/255163780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82e80c3640 0x7f82e80c4540 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 >> 172.21.15.148:0/255163780 conn(0x7f82e80c75d0 msgr2=0x7f82e81516d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:28.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 shutdown_connections 2023-10-30T13:26:28.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.725+0000 7f82edc67640 1 -- 172.21.15.148:0/255163780 wait complete. 2023-10-30T13:26:28.740 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:26:28.740 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:26:28.741 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:26:28.764 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:26:28.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:28.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cf617640 1 Processor -- start 2023-10-30T13:26:28.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cf617640 1 -- start start 2023-10-30T13:26:28.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cf617640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa6c806f7a0 0x7fa6c806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cf617640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6c8071110 0x7fa6c8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cf617640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c8066cf0 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cf617640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c80700c0 con 0x7fa6c806f7a0 2023-10-30T13:26:28.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cf617640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c80c70d0 con 0x7fa6c8071110 2023-10-30T13:26:28.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cd3b3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa6c806f7a0 0x7fa6c806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cd3b3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa6c806f7a0 0x7fa6c806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38916/0 (socket says 172.21.15.148:38916) 2023-10-30T13:26:28.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cd3b3640 1 -- 172.21.15.148:0/2551696253 learned_addr learned my addr 172.21.15.148:0/2551696253 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:28.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cdbb4640 1 --2- 172.21.15.148:0/2551696253 >> v2:172.21.15.150:3300/0 conn(0x7fa6c8071110 0x7fa6c8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cd3b3640 1 -- 172.21.15.148:0/2551696253 >> v2:172.21.15.150:3300/0 conn(0x7fa6c8071110 msgr2=0x7fa6c8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:28.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cd3b3640 1 --2- 172.21.15.148:0/2551696253 >> v2:172.21.15.150:3300/0 conn(0x7fa6c8071110 0x7fa6c8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cd3b3640 1 -- 172.21.15.148:0/2551696253 >> v1:172.21.15.150:6789/0 conn(0x7fa6c80c6bd0 legacy=0x7fa6c8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cd3b3640 1 -- 172.21.15.148:0/2551696253 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c80c7390 con 0x7fa6c806f7a0 2023-10-30T13:26:28.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.949+0000 7fa6cdbb4640 1 --2- 172.21.15.148:0/2551696253 >> v2:172.21.15.150:3300/0 conn(0x7fa6c8071110 0x7fa6c8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:28.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6cd3b3640 1 --2- 172.21.15.148:0/2551696253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c806f7a0 0x7fa6c806fb80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa6c806cdd0 tx=0x7fa6c401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19fb51159d240eaa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:28.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6bffff640 1 -- 172.21.15.148:0/2551696253 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40a89f0 con 0x7fa6c806f7a0 2023-10-30T13:26:28.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6bffff640 1 -- 172.21.15.148:0/2551696253 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c40a8b50 con 0x7fa6c806f7a0 2023-10-30T13:26:28.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6bffff640 1 -- 172.21.15.148:0/2551696253 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40b0440 con 0x7fa6c806f7a0 2023-10-30T13:26:28.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6cf617640 1 -- 172.21.15.148:0/2551696253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c806f7a0 msgr2=0x7fa6c806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6cf617640 1 --2- 172.21.15.148:0/2551696253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c806f7a0 0x7fa6c806fb80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa6c806cdd0 tx=0x7fa6c401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6cf617640 1 -- 172.21.15.148:0/2551696253 shutdown_connections 2023-10-30T13:26:28.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6cf617640 1 --2- 172.21.15.148:0/2551696253 >> v2:172.21.15.150:3300/0 conn(0x7fa6c8071110 0x7fa6c8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6cf617640 1 --2- 172.21.15.148:0/2551696253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c806f7a0 0x7fa6c806fb80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6cf617640 1 -- 172.21.15.148:0/2551696253 >> 172.21.15.148:0/2551696253 conn(0x7fa6c80c7910 msgr2=0x7fa6c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:28.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6cf617640 1 -- 172.21.15.148:0/2551696253 shutdown_connections 2023-10-30T13:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.953+0000 7fa6cf617640 1 -- 172.21.15.148:0/2551696253 wait complete. 2023-10-30T13:26:28.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 Processor -- start 2023-10-30T13:26:28.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 -- start start 2023-10-30T13:26:28.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c806f7a0 0x7fa6c8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c8071110 0x7fa6c80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c80c6bd0 0x7fa6c80adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c8155d00 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80aeb00 con 0x7fa6c8071110 2023-10-30T13:26:28.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80aede0 con 0x7fa6c806f7a0 2023-10-30T13:26:28.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cdbb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c80c6bd0 0x7fa6c80adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cdbb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c80c6bd0 0x7fa6c80adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38918/0 (socket says 172.21.15.148:38918) 2023-10-30T13:26:28.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cdbb4640 1 -- 172.21.15.148:0/172368809 learned_addr learned my addr 172.21.15.148:0/172368809 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:28.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6ccbb2640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c8071110 0x7fa6c80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cd3b3640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c806f7a0 0x7fa6c8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cdbb4640 1 -- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c806f7a0 msgr2=0x7fa6c8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cdbb4640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c806f7a0 0x7fa6c8157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cdbb4640 1 -- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c8071110 msgr2=0x7fa6c80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:28.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cdbb4640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c8071110 0x7fa6c80bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:28.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cdbb4640 1 -- 172.21.15.148:0/172368809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c40094d0 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6ccbb2640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c8071110 0x7fa6c80bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:28.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cdbb4640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c80c6bd0 0x7fa6c80adcc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0007df0 tx=0x7fa6c000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:28.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6bdffb640 1 -- 172.21.15.148:0/172368809 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c00039a0 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6bdffb640 1 -- 172.21.15.148:0/172368809 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c000f040 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6bdffb640 1 -- 172.21.15.148:0/172368809 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0003d40 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c80af0c0 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.957+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c80c9730 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.961+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c804e170 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.961+0000 7fa6bdffb640 1 -- 172.21.15.148:0/172368809 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6c0004030 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.961+0000 7fa6bdffb640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6ac0664a0 0x7fa6ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:28.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.961+0000 7fa6bdffb640 1 -- 172.21.15.148:0/172368809 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fa6c00844d0 con 0x7fa6c80c6bd0 2023-10-30T13:26:28.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.961+0000 7fa6cd3b3640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6ac0664a0 0x7fa6ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:28.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.961+0000 7fa6cd3b3640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6ac0664a0 0x7fa6ac068960 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40b2000 tx=0x7fa6c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:28.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:28.965+0000 7fa6bdffb640 1 -- 172.21.15.148:0/172368809 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6c00515c0 con 0x7fa6c80c6bd0 2023-10-30T13:26:29.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.145+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6c80c7e20 con 0x7fa6c80c6bd0 2023-10-30T13:26:29.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.145+0000 7fa6bdffb640 1 -- 172.21.15.148:0/172368809 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fa6c0054be0 con 0x7fa6c80c6bd0 2023-10-30T13:26:29.143 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:29.143 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:29.143 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 75 2023-10-30T13:26:29.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6ac0664a0 msgr2=0x7fa6ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:29.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6ac0664a0 0x7fa6ac068960 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40b2000 tx=0x7fa6c4008040 comp rx=0 tx=0).stop 2023-10-30T13:26:29.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c80c6bd0 msgr2=0x7fa6c80adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:29.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c80c6bd0 0x7fa6c80adcc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0007df0 tx=0x7fa6c000d4a0 comp rx=0 tx=0).stop 2023-10-30T13:26:29.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 shutdown_connections 2023-10-30T13:26:29.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6ac0664a0 0x7fa6ac068960 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:29.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c80c6bd0 0x7fa6c80adcc0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:29.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6c8071110 0x7fa6c80bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:29.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 --2- 172.21.15.148:0/172368809 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa6c806f7a0 0x7fa6c8157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:29.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 >> 172.21.15.148:0/172368809 conn(0x7fa6c80c7910 msgr2=0x7fa6c80b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:29.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 shutdown_connections 2023-10-30T13:26:29.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:29.149+0000 7fa6cf617640 1 -- 172.21.15.148:0/172368809 wait complete. 2023-10-30T13:26:29.163 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:26:31.164 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:31.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09f09c640 1 Processor -- start 2023-10-30T13:26:31.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09f09c640 1 -- start start 2023-10-30T13:26:31.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09f09c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 0x7fd0980bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:31.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09f09c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd09813f4a0 0x7fd0980c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:31.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09f09c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0980c0a50 con 0x7fd0980c6f60 2023-10-30T13:26:31.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09f09c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09806f6e0 con 0x7fd09813f4a0 2023-10-30T13:26:31.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09f09c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09806f9c0 con 0x7fd0980c6b80 2023-10-30T13:26:31.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09d639640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd09813f4a0 0x7fd0980c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:31.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09d639640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd09813f4a0 0x7fd0980c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38920/0 (socket says 172.21.15.148:38920) 2023-10-30T13:26:31.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09d639640 1 -- 172.21.15.148:0/3568852603 learned_addr learned my addr 172.21.15.148:0/3568852603 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:31.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.317+0000 7fd09ce38640 1 --2- 172.21.15.148:0/3568852603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 0x7fd0980bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:31.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09d639640 1 -- 172.21.15.148:0/3568852603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 msgr2=0x7fd0980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:31.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09d639640 1 --2- 172.21.15.148:0/3568852603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 0x7fd0980bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09d639640 1 -- 172.21.15.148:0/3568852603 >> v1:172.21.15.150:6789/0 conn(0x7fd0980c6f60 legacy=0x7fd09813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:31.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09d639640 1 -- 172.21.15.148:0/3568852603 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09806fca0 con 0x7fd09813f4a0 2023-10-30T13:26:31.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09d639640 1 --2- 172.21.15.148:0/3568852603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 0x7fd0980c0400 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd09000d150 tx=0x7fd09001ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d1722f73fcc85a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:31.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd0977fe640 1 -- 172.21.15.148:0/3568852603 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09000b2d0 con 0x7fd09813f4a0 2023-10-30T13:26:31.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd0977fe640 1 -- 172.21.15.148:0/3568852603 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09000b430 con 0x7fd09813f4a0 2023-10-30T13:26:31.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd0977fe640 1 -- 172.21.15.148:0/3568852603 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0900a7c10 con 0x7fd09813f4a0 2023-10-30T13:26:31.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09f09c640 1 -- 172.21.15.148:0/3568852603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 msgr2=0x7fd0980c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:31.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09f09c640 1 --2- 172.21.15.148:0/3568852603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 0x7fd0980c0400 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd09000d150 tx=0x7fd09001ec10 comp rx=0 tx=0).stop 2023-10-30T13:26:31.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09f09c640 1 -- 172.21.15.148:0/3568852603 shutdown_connections 2023-10-30T13:26:31.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09f09c640 1 --2- 172.21.15.148:0/3568852603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 0x7fd0980c0400 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09f09c640 1 --2- 172.21.15.148:0/3568852603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 0x7fd0980bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.321+0000 7fd09f09c640 1 -- 172.21.15.148:0/3568852603 >> 172.21.15.148:0/3568852603 conn(0x7fd0980c7910 msgr2=0x7fd0980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:31.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 -- 172.21.15.148:0/3568852603 shutdown_connections 2023-10-30T13:26:31.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 -- 172.21.15.148:0/3568852603 wait complete. 2023-10-30T13:26:31.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 Processor -- start 2023-10-30T13:26:31.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 -- start start 2023-10-30T13:26:31.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 0x7fd098137980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:31.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0980c6f60 0x7fd0981f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:31.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 0x7fd09806b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:31.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09815ec30 con 0x7fd09813f4a0 2023-10-30T13:26:31.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0981f2e40 con 0x7fd0980c6f60 2023-10-30T13:26:31.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09f09c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0981f3400 con 0x7fd0980c6b80 2023-10-30T13:26:31.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09d639640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 0x7fd09806b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:31.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09d639640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 0x7fd09806b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38926/0 (socket says 172.21.15.148:38926) 2023-10-30T13:26:31.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09d639640 1 -- 172.21.15.148:0/1384567454 learned_addr learned my addr 172.21.15.148:0/1384567454 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:31.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09ce38640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 0x7fd098137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:31.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd097fff640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0980c6f60 0x7fd0981f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:31.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09d639640 1 -- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 msgr2=0x7fd098137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:31.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09d639640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 0x7fd098137980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09d639640 1 -- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0980c6f60 msgr2=0x7fd0981f5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:31.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09d639640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0980c6f60 0x7fd0981f5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09d639640 1 -- 172.21.15.148:0/1384567454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09000bd40 con 0x7fd09813f4a0 2023-10-30T13:26:31.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09ce38640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 0x7fd098137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:31.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.325+0000 7fd09d639640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 0x7fd09806b5b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd0900b1040 tx=0x7fd09001b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:31.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd0957fa640 1 -- 172.21.15.148:0/1384567454 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090009a10 con 0x7fd09813f4a0 2023-10-30T13:26:31.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd0957fa640 1 -- 172.21.15.148:0/1384567454 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd090009bd0 con 0x7fd09813f4a0 2023-10-30T13:26:31.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd0957fa640 1 -- 172.21.15.148:0/1384567454 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09000b720 con 0x7fd09813f4a0 2023-10-30T13:26:31.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd098137ec0 con 0x7fd09813f4a0 2023-10-30T13:26:31.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09815fbc0 con 0x7fd09813f4a0 2023-10-30T13:26:31.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09804e170 con 0x7fd09813f4a0 2023-10-30T13:26:31.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd0957fa640 1 -- 172.21.15.148:0/1384567454 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0900c3460 con 0x7fd09813f4a0 2023-10-30T13:26:31.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd0957fa640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd07c0664a0 0x7fd07c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:31.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd0957fa640 1 -- 172.21.15.148:0/1384567454 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fd09011fa70 con 0x7fd09813f4a0 2023-10-30T13:26:31.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.329+0000 7fd09ce38640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd07c0664a0 0x7fd07c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:31.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.333+0000 7fd09ce38640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd07c0664a0 0x7fd07c068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd08c007a70 tx=0x7fd08c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:31.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.333+0000 7fd0957fa640 1 -- 172.21.15.148:0/1384567454 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0900ecbf0 con 0x7fd09813f4a0 2023-10-30T13:26:31.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.513+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0981f3540 con 0x7fd09813f4a0 2023-10-30T13:26:31.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd0957fa640 1 -- 172.21.15.148:0/1384567454 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fd0900f0210 con 0x7fd09813f4a0 2023-10-30T13:26:31.511 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:31.511 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:31.512 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 75 2023-10-30T13:26:31.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd07c0664a0 msgr2=0x7fd07c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:31.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd07c0664a0 0x7fd07c068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd08c007a70 tx=0x7fd08c008040 comp rx=0 tx=0).stop 2023-10-30T13:26:31.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 msgr2=0x7fd09806b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:31.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 0x7fd09806b5b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd0900b1040 tx=0x7fd09001b9b0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 shutdown_connections 2023-10-30T13:26:31.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd07c0664a0 0x7fd07c068960 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd09813f4a0 0x7fd09806b5b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0980c6f60 0x7fd0981f5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 --2- 172.21.15.148:0/1384567454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd0980c6b80 0x7fd098137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:31.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 >> 172.21.15.148:0/1384567454 conn(0x7fd0980c7910 msgr2=0x7fd0980c5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:31.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 shutdown_connections 2023-10-30T13:26:31.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:31.517+0000 7fd09f09c640 1 -- 172.21.15.148:0/1384567454 wait complete. 2023-10-30T13:26:31.532 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:26:33.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:33.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f927d33b640 1 Processor -- start 2023-10-30T13:26:33.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f927d33b640 1 -- start start 2023-10-30T13:26:33.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f927d33b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92781538f0 0x7f9278159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:33.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f927d33b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f927815cd50 0x7f92780c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:33.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f927d33b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927815d300 con 0x7f92780cefa0 2023-10-30T13:26:33.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f927d33b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927815d5e0 con 0x7f92781538f0 2023-10-30T13:26:33.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f927d33b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927815d8c0 con 0x7f927815cd50 2023-10-30T13:26:33.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f9277fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92781538f0 0x7f9278159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:33.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f9277fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92781538f0 0x7f9278159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45762/0 (socket says 172.21.15.148:45762) 2023-10-30T13:26:33.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f9277fff640 1 -- 172.21.15.148:0/1647211401 learned_addr learned my addr 172.21.15.148:0/1647211401 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:33.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f92777fe640 1 --2- 172.21.15.148:0/1647211401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f927815cd50 0x7f92780c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:33.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f9277fff640 1 -- 172.21.15.148:0/1647211401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f927815cd50 msgr2=0x7f92780c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:33.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f9277fff640 1 --2- 172.21.15.148:0/1647211401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f927815cd50 0x7f92780c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:33.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f9277fff640 1 -- 172.21.15.148:0/1647211401 >> v1:172.21.15.150:6789/0 conn(0x7f92780cefa0 legacy=0x7f92780cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:33.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.689+0000 7f9277fff640 1 -- 172.21.15.148:0/1647211401 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927815dba0 con 0x7f92781538f0 2023-10-30T13:26:33.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f9277fff640 1 --2- 172.21.15.148:0/1647211401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92781538f0 0x7f9278159fb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9268009870 tx=0x7f926801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12126fcf288b9306 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:33.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f9276ffd640 1 -- 172.21.15.148:0/1647211401 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92680a8a30 con 0x7f92781538f0 2023-10-30T13:26:33.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f9276ffd640 1 -- 172.21.15.148:0/1647211401 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92680a8b90 con 0x7f92781538f0 2023-10-30T13:26:33.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f9276ffd640 1 -- 172.21.15.148:0/1647211401 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92680b0440 con 0x7f92781538f0 2023-10-30T13:26:33.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f927d33b640 1 -- 172.21.15.148:0/1647211401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92781538f0 msgr2=0x7f9278159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:33.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f927d33b640 1 --2- 172.21.15.148:0/1647211401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92781538f0 0x7f9278159fb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9268009870 tx=0x7f926801ed30 comp rx=0 tx=0).stop 2023-10-30T13:26:33.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f927d33b640 1 -- 172.21.15.148:0/1647211401 shutdown_connections 2023-10-30T13:26:33.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f927d33b640 1 --2- 172.21.15.148:0/1647211401 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f927815cd50 0x7f92780c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f927d33b640 1 --2- 172.21.15.148:0/1647211401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92781538f0 0x7f9278159fb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f927d33b640 1 -- 172.21.15.148:0/1647211401 >> 172.21.15.148:0/1647211401 conn(0x7f927806a640 msgr2=0x7f9278065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f927d33b640 1 -- 172.21.15.148:0/1647211401 shutdown_connections 2023-10-30T13:26:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.693+0000 7f927d33b640 1 -- 172.21.15.148:0/1647211401 wait complete. 2023-10-30T13:26:33.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 Processor -- start 2023-10-30T13:26:33.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 -- start start 2023-10-30T13:26:33.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92780cefa0 0x7f92780b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:33.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92781538f0 0x7f92781e3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:33.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f927815cd50 0x7f92781f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:33.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92780b48f0 con 0x7f927815cd50 2023-10-30T13:26:33.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9278148180 con 0x7f92780cefa0 2023-10-30T13:26:33.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92781f2780 con 0x7f92781538f0 2023-10-30T13:26:33.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927cb3a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f927815cd50 0x7f92781f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:33.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927cb3a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f927815cd50 0x7f92781f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45770/0 (socket says 172.21.15.148:45770) 2023-10-30T13:26:33.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927cb3a640 1 -- 172.21.15.148:0/1712537827 learned_addr learned my addr 172.21.15.148:0/1712537827 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:33.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f92777fe640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92781538f0 0x7f92781e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:33.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f9277fff640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92780cefa0 0x7f92780b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:33.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927cb3a640 1 -- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92781538f0 msgr2=0x7f92781e3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:33.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927cb3a640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92781538f0 0x7f92781e3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:33.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927cb3a640 1 -- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92780cefa0 msgr2=0x7f92780b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:33.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927cb3a640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92780cefa0 0x7f92780b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:33.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927cb3a640 1 -- 172.21.15.148:0/1712537827 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680094d0 con 0x7f927815cd50 2023-10-30T13:26:33.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927cb3a640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f927815cd50 0x7f92781f2c10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f927000c7a0 tx=0x7f927000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:33.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f9274ff9640 1 -- 172.21.15.148:0/1712537827 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9270003b80 con 0x7f927815cd50 2023-10-30T13:26:33.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f9274ff9640 1 -- 172.21.15.148:0/1712537827 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9270003ce0 con 0x7f927815cd50 2023-10-30T13:26:33.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f9274ff9640 1 -- 172.21.15.148:0/1712537827 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927000aa90 con 0x7f927815cd50 2023-10-30T13:26:33.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92781483b0 con 0x7f927815cd50 2023-10-30T13:26:33.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92781e5510 con 0x7f927815cd50 2023-10-30T13:26:33.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.697+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92781f2930 con 0x7f927815cd50 2023-10-30T13:26:33.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.701+0000 7f9274ff9640 1 -- 172.21.15.148:0/1712537827 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f927000abf0 con 0x7f927815cd50 2023-10-30T13:26:33.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.701+0000 7f9274ff9640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92640664d0 0x7f9264068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:33.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.701+0000 7f9274ff9640 1 -- 172.21.15.148:0/1712537827 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f9270083f40 con 0x7f927815cd50 2023-10-30T13:26:33.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.701+0000 7f9277fff640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92640664d0 0x7f9264068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:33.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.701+0000 7f9277fff640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92640664d0 0x7f9264068990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f92680b2000 tx=0x7f9268009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:33.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.705+0000 7f9274ff9640 1 -- 172.21.15.148:0/1712537827 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92781f2930 con 0x7f927815cd50 2023-10-30T13:26:33.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.889+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92781f28f0 con 0x7f927815cd50 2023-10-30T13:26:33.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.889+0000 7f9274ff9640 1 -- 172.21.15.148:0/1712537827 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f92700510e0 con 0x7f927815cd50 2023-10-30T13:26:33.886 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:33.887 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:33.888 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 75 2023-10-30T13:26:33.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92640664d0 msgr2=0x7f9264068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:33.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92640664d0 0x7f9264068990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f92680b2000 tx=0x7f9268009040 comp rx=0 tx=0).stop 2023-10-30T13:26:33.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f927815cd50 msgr2=0x7f92781f2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:33.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f927815cd50 0x7f92781f2c10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f927000c7a0 tx=0x7f927000cc60 comp rx=0 tx=0).stop 2023-10-30T13:26:33.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 shutdown_connections 2023-10-30T13:26:33.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92640664d0 0x7f9264068990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:33.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f927815cd50 0x7f92781f2c10 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:33.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92781538f0 0x7f92781e3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:33.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 --2- 172.21.15.148:0/1712537827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92780cefa0 0x7f92780b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:33.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 >> 172.21.15.148:0/1712537827 conn(0x7f927806a640 msgr2=0x7f92781489b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:33.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 shutdown_connections 2023-10-30T13:26:33.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:33.893+0000 7f927d33b640 1 -- 172.21.15.148:0/1712537827 wait complete. 2023-10-30T13:26:33.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:26:35.908 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:36.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab9a31d640 1 Processor -- start 2023-10-30T13:26:36.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab9a31d640 1 -- start start 2023-10-30T13:26:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab9a31d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab9406f7a0 0x7fab9406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab9a31d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab940c6bd0 0x7fab94069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab9a31d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9406a3d0 con 0x7fab94071110 2023-10-30T13:26:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab9a31d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab940700c0 con 0x7fab9406f7a0 2023-10-30T13:26:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab9a31d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab94066cf0 con 0x7fab940c6bd0 2023-10-30T13:26:36.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab93fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab9406f7a0 0x7fab9406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:36.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab93fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab9406f7a0 0x7fab9406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45778/0 (socket says 172.21.15.148:45778) 2023-10-30T13:26:36.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab93fff640 1 -- 172.21.15.148:0/2239385433 learned_addr learned my addr 172.21.15.148:0/2239385433 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:36.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab937fe640 1 --2- 172.21.15.148:0/2239385433 >> v2:172.21.15.150:3300/0 conn(0x7fab940c6bd0 0x7fab94069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:36.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab93fff640 1 -- 172.21.15.148:0/2239385433 >> v2:172.21.15.150:3300/0 conn(0x7fab940c6bd0 msgr2=0x7fab94069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:36.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab93fff640 1 --2- 172.21.15.148:0/2239385433 >> v2:172.21.15.150:3300/0 conn(0x7fab940c6bd0 0x7fab94069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab93fff640 1 -- 172.21.15.148:0/2239385433 >> v1:172.21.15.150:6789/0 conn(0x7fab94071110 legacy=0x7fab94158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:36.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.061+0000 7fab93fff640 1 -- 172.21.15.148:0/2239385433 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab940c70d0 con 0x7fab9406f7a0 2023-10-30T13:26:36.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab93fff640 1 --2- 172.21.15.148:0/2239385433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab9406f7a0 0x7fab9406fb80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fab9406f1c0 tx=0x7fab8401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07dc8ecc22fcbad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:36.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab92ffd640 1 -- 172.21.15.148:0/2239385433 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab840a89b0 con 0x7fab9406f7a0 2023-10-30T13:26:36.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab92ffd640 1 -- 172.21.15.148:0/2239385433 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab840a8b10 con 0x7fab9406f7a0 2023-10-30T13:26:36.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab92ffd640 1 -- 172.21.15.148:0/2239385433 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab840a8ea0 con 0x7fab9406f7a0 2023-10-30T13:26:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab9a31d640 1 -- 172.21.15.148:0/2239385433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab9406f7a0 msgr2=0x7fab9406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab9a31d640 1 --2- 172.21.15.148:0/2239385433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab9406f7a0 0x7fab9406fb80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fab9406f1c0 tx=0x7fab8401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab9a31d640 1 -- 172.21.15.148:0/2239385433 shutdown_connections 2023-10-30T13:26:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab9a31d640 1 --2- 172.21.15.148:0/2239385433 >> v2:172.21.15.150:3300/0 conn(0x7fab940c6bd0 0x7fab94069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab9a31d640 1 --2- 172.21.15.148:0/2239385433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab9406f7a0 0x7fab9406fb80 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab9a31d640 1 -- 172.21.15.148:0/2239385433 >> 172.21.15.148:0/2239385433 conn(0x7fab940c7910 msgr2=0x7fab940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:36.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab9a31d640 1 -- 172.21.15.148:0/2239385433 shutdown_connections 2023-10-30T13:26:36.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.065+0000 7fab9a31d640 1 -- 172.21.15.148:0/2239385433 wait complete. 2023-10-30T13:26:36.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 Processor -- start 2023-10-30T13:26:36.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 -- start start 2023-10-30T13:26:36.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9406f7a0 0x7fab9414af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:36.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab94071110 0x7fab9414ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:36.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab940c6bd0 0x7fab94150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:36.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9414fb30 con 0x7fab94071110 2023-10-30T13:26:36.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9414ef40 con 0x7fab9406f7a0 2023-10-30T13:26:36.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab941efd00 con 0x7fab940c6bd0 2023-10-30T13:26:36.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab937fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab94071110 0x7fab9414ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:36.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab937fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab94071110 0x7fab9414ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45790/0 (socket says 172.21.15.148:45790) 2023-10-30T13:26:36.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab937fe640 1 -- 172.21.15.148:0/760237189 learned_addr learned my addr 172.21.15.148:0/760237189 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:36.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab93fff640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9406f7a0 0x7fab9414af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:36.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab988ba640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab940c6bd0 0x7fab94150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:36.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab937fe640 1 -- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab940c6bd0 msgr2=0x7fab94150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:36.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab937fe640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab940c6bd0 0x7fab94150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab937fe640 1 -- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9406f7a0 msgr2=0x7fab9414af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:36.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab937fe640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9406f7a0 0x7fab9414af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab937fe640 1 -- 172.21.15.148:0/760237189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab840094d0 con 0x7fab94071110 2023-10-30T13:26:36.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab937fe640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab94071110 0x7fab9414ff50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fab8800d600 tx=0x7fab8800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:36.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab90ff9640 1 -- 172.21.15.148:0/760237189 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88004240 con 0x7fab94071110 2023-10-30T13:26:36.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab90ff9640 1 -- 172.21.15.148:0/760237189 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab880043a0 con 0x7fab94071110 2023-10-30T13:26:36.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9414f190 con 0x7fab94071110 2023-10-30T13:26:36.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab90ff9640 1 -- 172.21.15.148:0/760237189 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88003840 con 0x7fab94071110 2023-10-30T13:26:36.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.069+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9414f750 con 0x7fab94071110 2023-10-30T13:26:36.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.073+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9404e170 con 0x7fab94071110 2023-10-30T13:26:36.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.073+0000 7fab90ff9640 1 -- 172.21.15.148:0/760237189 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab880039a0 con 0x7fab94071110 2023-10-30T13:26:36.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.073+0000 7fab90ff9640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab800664a0 0x7fab80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:36.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.073+0000 7fab90ff9640 1 -- 172.21.15.148:0/760237189 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7fab88084240 con 0x7fab94071110 2023-10-30T13:26:36.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.073+0000 7fab93fff640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab800664a0 0x7fab80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:36.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.073+0000 7fab93fff640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab800664a0 0x7fab80068960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fab840b2000 tx=0x7fab84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:36.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.077+0000 7fab90ff9640 1 -- 172.21.15.148:0/760237189 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab88051470 con 0x7fab94071110 2023-10-30T13:26:36.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.257+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab940c3650 con 0x7fab94071110 2023-10-30T13:26:36.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.257+0000 7fab90ff9640 1 -- 172.21.15.148:0/760237189 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fab88054a90 con 0x7fab94071110 2023-10-30T13:26:36.253 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:36.254 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:36.254 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 75 2023-10-30T13:26:36.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab800664a0 msgr2=0x7fab80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:36.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab800664a0 0x7fab80068960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fab840b2000 tx=0x7fab84009040 comp rx=0 tx=0).stop 2023-10-30T13:26:36.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab94071110 msgr2=0x7fab9414ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:36.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab94071110 0x7fab9414ff50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fab8800d600 tx=0x7fab8800dac0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 shutdown_connections 2023-10-30T13:26:36.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab800664a0 0x7fab80068960 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab94071110 0x7fab9414ff50 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab940c6bd0 0x7fab94150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 --2- 172.21.15.148:0/760237189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab9406f7a0 0x7fab9414af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:36.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 >> 172.21.15.148:0/760237189 conn(0x7fab940c7910 msgr2=0x7fab940c7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:36.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 shutdown_connections 2023-10-30T13:26:36.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:36.261+0000 7fab9a31d640 1 -- 172.21.15.148:0/760237189 wait complete. 2023-10-30T13:26:36.275 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:26:38.277 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1420b1d640 1 Processor -- start 2023-10-30T13:26:38.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1420b1d640 1 -- start start 2023-10-30T13:26:38.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1420b1d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f141c06f7a0 0x7f141c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:38.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1420b1d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141c066cf0 con 0x7f141c071110 2023-10-30T13:26:38.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1420b1d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141c0c70d0 con 0x7f141c0c6bd0 2023-10-30T13:26:38.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1420b1d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f141c0700c0 con 0x7f141c06f7a0 2023-10-30T13:26:38.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f141ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f141c071110 0x7f141c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40224/0 (socket says 172.21.15.148:40224) 2023-10-30T13:26:38.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f141ad9d640 1 -- 172.21.15.148:0/1605349765 learned_addr learned my addr 172.21.15.148:0/1605349765 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:26:38.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f141a59c640 1 --2- 172.21.15.148:0/1605349765 >> v2:172.21.15.150:3300/0 conn(0x7f141c06f7a0 0x7f141c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:38.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127287793 0 0) 0x7f141c066cf0 con 0x7f141c071110 2023-10-30T13:26:38.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1408003530 con 0x7f141c071110 2023-10-30T13:26:38.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 282484266 0 0) 0x7f1408003530 con 0x7f141c071110 2023-10-30T13:26:38.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 >> v2:172.21.15.150:3300/0 conn(0x7f141c06f7a0 msgr2=0x7f141c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:38.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1419d9b640 1 --2- 172.21.15.148:0/1605349765 >> v2:172.21.15.150:3300/0 conn(0x7f141c06f7a0 0x7f141c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 >> v1:172.21.15.150:6789/0 conn(0x7f141c0c6bd0 legacy=0x7f141c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:38.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c0c7390 con 0x7f141c071110 2023-10-30T13:26:38.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.429+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f1414001a00 con 0x7f141c071110 2023-10-30T13:26:38.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 >> v1:172.21.15.148:6789/0 conn(0x7f141c071110 legacy=0x7f141c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:38.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1408003980 0x7f1408003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:38.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14080047d0 0x7f1408004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:38.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f140800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:38.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c0700c0 con 0x7f1408005190 2023-10-30T13:26:38.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140800c040 con 0x7f1408003980 2023-10-30T13:26:38.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140800c200 con 0x7f14080047d0 2023-10-30T13:26:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f141ad9d640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f140800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f141a59c640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1408003980 0x7f1408003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1411d9b640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14080047d0 0x7f1408004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f141ad9d640 1 -- 172.21.15.148:0/1605349765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14080047d0 msgr2=0x7f1408004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f141ad9d640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14080047d0 0x7f1408004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f141ad9d640 1 -- 172.21.15.148:0/1605349765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1408003980 msgr2=0x7f1408003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:38.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f141ad9d640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1408003980 0x7f1408003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f141ad9d640 1 -- 172.21.15.148:0/1605349765 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1408003530 con 0x7f1408005190 2023-10-30T13:26:38.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1411d9b640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f14080047d0 0x7f1408004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f141ad9d640 1 -- 172.21.15.148:0/1605349765 reap_dead start 2023-10-30T13:26:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f141ad9d640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f140800b990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f141400a830 tx=0x7f141401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b273a5bac33f5d4b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140072d0 con 0x7f1408005190 2023-10-30T13:26:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14140a8070 con 0x7f1408005190 2023-10-30T13:26:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.433+0000 7f1419d9b640 1 -- 172.21.15.148:0/1605349765 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140076b0 con 0x7f1408005190 2023-10-30T13:26:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 -- 172.21.15.148:0/1605349765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 msgr2=0x7f140800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f140800b990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f141400a830 tx=0x7f141401d340 comp rx=0 tx=0).stop 2023-10-30T13:26:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 -- 172.21.15.148:0/1605349765 shutdown_connections 2023-10-30T13:26:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 --2- 172.21.15.148:0/1605349765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f140800b990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 -- 172.21.15.148:0/1605349765 >> 172.21.15.148:0/1605349765 conn(0x7f141c0c7910 msgr2=0x7f141c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 -- 172.21.15.148:0/1605349765 shutdown_connections 2023-10-30T13:26:38.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 -- 172.21.15.148:0/1605349765 wait complete. 2023-10-30T13:26:38.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 Processor -- start 2023-10-30T13:26:38.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 -- start start 2023-10-30T13:26:38.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f141c1ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:38.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f141c0c9540 0x7f141c15b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:38.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c156940 0x7f141c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:38.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c156520 con 0x7f1408005190 2023-10-30T13:26:38.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c1efde0 con 0x7f141c156940 2023-10-30T13:26:38.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f1420b1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141c1580e0 con 0x7f141c0c9540 2023-10-30T13:26:38.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f141a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f141c1ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:38.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f141a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f141c1ef8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45804/0 (socket says 172.21.15.148:45804) 2023-10-30T13:26:38.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.437+0000 7f141a59c640 1 -- 172.21.15.148:0/1154048861 learned_addr learned my addr 172.21.15.148:0/1154048861 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:38.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f141ad9d640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c156940 0x7f141c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:38.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f1411d9b640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f141c0c9540 0x7f141c15b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:38.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f141a59c640 1 -- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f141c0c9540 msgr2=0x7f141c15b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:38.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f141a59c640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f141c0c9540 0x7f141c15b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f141a59c640 1 -- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c156940 msgr2=0x7f141c156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:38.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f141a59c640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c156940 0x7f141c156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f141a59c640 1 -- 172.21.15.148:0/1154048861 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c155600 con 0x7f1408005190 2023-10-30T13:26:38.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f141a59c640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f141c1ef8a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f140c011960 tx=0x7f140c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:38.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f14137fe640 1 -- 172.21.15.148:0/1154048861 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c00faf0 con 0x7f1408005190 2023-10-30T13:26:38.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f14137fe640 1 -- 172.21.15.148:0/1154048861 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140c00fc50 con 0x7f1408005190 2023-10-30T13:26:38.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141c1558a0 con 0x7f1408005190 2023-10-30T13:26:38.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f14137fe640 1 -- 172.21.15.148:0/1154048861 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c014440 con 0x7f1408005190 2023-10-30T13:26:38.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141c155e60 con 0x7f1408005190 2023-10-30T13:26:38.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f141c04e170 con 0x7f1408005190 2023-10-30T13:26:38.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f14137fe640 1 -- 172.21.15.148:0/1154048861 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f140c0145a0 con 0x7f1408005190 2023-10-30T13:26:38.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f14137fe640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13fc0664d0 0x7f13fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:38.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f14137fe640 1 -- 172.21.15.148:0/1154048861 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f140c0849c0 con 0x7f1408005190 2023-10-30T13:26:38.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.441+0000 7f1411d9b640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13fc0664d0 0x7f13fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:38.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.445+0000 7f1411d9b640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13fc0664d0 0x7f13fc068990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1404000c10 tx=0x7f1404009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:38.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.445+0000 7f14137fe640 1 -- 172.21.15.148:0/1154048861 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f140c051aa0 con 0x7f1408005190 2023-10-30T13:26:38.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.625+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f141c156790 con 0x7f1408005190 2023-10-30T13:26:38.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.625+0000 7f14137fe640 1 -- 172.21.15.148:0/1154048861 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f140c022090 con 0x7f1408005190 2023-10-30T13:26:38.622 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:38.622 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:38.622 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 75 2023-10-30T13:26:38.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13fc0664d0 msgr2=0x7f13fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:38.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13fc0664d0 0x7f13fc068990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1404000c10 tx=0x7f1404009040 comp rx=0 tx=0).stop 2023-10-30T13:26:38.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 msgr2=0x7f141c1ef8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:38.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f141c1ef8a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f140c011960 tx=0x7f140c011e20 comp rx=0 tx=0).stop 2023-10-30T13:26:38.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 shutdown_connections 2023-10-30T13:26:38.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13fc0664d0 0x7f13fc068990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f141c156940 0x7f141c156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f141c0c9540 0x7f141c15b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 --2- 172.21.15.148:0/1154048861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1408005190 0x7f141c1ef8a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:38.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 >> 172.21.15.148:0/1154048861 conn(0x7f141c0c7910 msgr2=0x7f141c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:38.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 shutdown_connections 2023-10-30T13:26:38.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:38.629+0000 7f1420b1d640 1 -- 172.21.15.148:0/1154048861 wait complete. 2023-10-30T13:26:38.643 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:26:40.645 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:40.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a7a9c5640 1 Processor -- start 2023-10-30T13:26:40.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a7a9c5640 1 -- start start 2023-10-30T13:26:40.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a7a9c5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4a7406e470 0x7f4a7406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:40.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a7a9c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 0x7f4a740c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:40.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a7a9c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7406afa0 con 0x7f4a74153d70 2023-10-30T13:26:40.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a7a9c5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a7406d780 con 0x7f4a7406e470 2023-10-30T13:26:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a7a9c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a74154150 con 0x7f4a740c8490 2023-10-30T13:26:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a737fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4a7406e470 0x7f4a7406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a737fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4a7406e470 0x7f4a7406ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45812/0 (socket says 172.21.15.148:45812) 2023-10-30T13:26:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.797+0000 7f4a737fe640 1 -- 172.21.15.148:0/1372658409 learned_addr learned my addr 172.21.15.148:0/1372658409 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a78f62640 1 --2- 172.21.15.148:0/1372658409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 0x7f4a740c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:40.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a737fe640 1 -- 172.21.15.148:0/1372658409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 msgr2=0x7f4a740c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:40.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a737fe640 1 --2- 172.21.15.148:0/1372658409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 0x7f4a740c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a737fe640 1 -- 172.21.15.148:0/1372658409 >> v1:172.21.15.150:6789/0 conn(0x7f4a74153d70 legacy=0x7f4a740cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:40.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a737fe640 1 -- 172.21.15.148:0/1372658409 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7406e030 con 0x7f4a7406e470 2023-10-30T13:26:40.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a737fe640 1 --2- 172.21.15.148:0/1372658409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a7406e470 0x7f4a7406ca80 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a6801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60abf1e9ee801566 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:40.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a72ffd640 1 -- 172.21.15.148:0/1372658409 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a680aa670 con 0x7f4a7406e470 2023-10-30T13:26:40.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a72ffd640 1 -- 172.21.15.148:0/1372658409 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a680aa7d0 con 0x7f4a7406e470 2023-10-30T13:26:40.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a72ffd640 1 -- 172.21.15.148:0/1372658409 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a680aab60 con 0x7f4a7406e470 2023-10-30T13:26:40.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/1372658409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a7406e470 msgr2=0x7f4a7406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:40.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.801+0000 7f4a7a9c5640 1 --2- 172.21.15.148:0/1372658409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a7406e470 0x7f4a7406ca80 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a6801c510 comp rx=0 tx=0).stop 2023-10-30T13:26:40.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/1372658409 shutdown_connections 2023-10-30T13:26:40.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 --2- 172.21.15.148:0/1372658409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 0x7f4a740c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 --2- 172.21.15.148:0/1372658409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a7406e470 0x7f4a7406ca80 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/1372658409 >> 172.21.15.148:0/1372658409 conn(0x7f4a74071ea0 msgr2=0x7f4a74065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:40.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/1372658409 shutdown_connections 2023-10-30T13:26:40.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/1372658409 wait complete. 2023-10-30T13:26:40.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 Processor -- start 2023-10-30T13:26:40.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 -- start start 2023-10-30T13:26:40.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a7406e470 0x7f4a741f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:40.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 0x7f4a74152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:40.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a74153d70 0x7f4a7413f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:40.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a74150930 con 0x7f4a74153d70 2023-10-30T13:26:40.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a741f60f0 con 0x7f4a7406e470 2023-10-30T13:26:40.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a7a9c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7414fe90 con 0x7f4a740c8490 2023-10-30T13:26:40.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a78f62640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a74153d70 0x7f4a7413f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:40.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a78f62640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a74153d70 0x7f4a7413f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45820/0 (socket says 172.21.15.148:45820) 2023-10-30T13:26:40.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a78f62640 1 -- 172.21.15.148:0/133330527 learned_addr learned my addr 172.21.15.148:0/133330527 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:40.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a73fff640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a7406e470 0x7f4a741f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:40.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a737fe640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 0x7f4a74152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:40.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a78f62640 1 -- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 msgr2=0x7f4a74152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:40.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a78f62640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 0x7f4a74152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a78f62640 1 -- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a7406e470 msgr2=0x7f4a741f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:40.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a78f62640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a7406e470 0x7f4a741f5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.805+0000 7f4a78f62640 1 -- 172.21.15.148:0/133330527 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a680094d0 con 0x7f4a74153d70 2023-10-30T13:26:40.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a78f62640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a74153d70 0x7f4a7413f730 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c007e70 tx=0x7f4a6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:40.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a70ff9640 1 -- 172.21.15.148:0/133330527 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c014620 con 0x7f4a74153d70 2023-10-30T13:26:40.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a70ff9640 1 -- 172.21.15.148:0/133330527 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c0147a0 con 0x7f4a74153d70 2023-10-30T13:26:40.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7414f600 con 0x7f4a74153d70 2023-10-30T13:26:40.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a70ff9640 1 -- 172.21.15.148:0/133330527 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c014b50 con 0x7f4a74153d70 2023-10-30T13:26:40.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7414ab40 con 0x7f4a74153d70 2023-10-30T13:26:40.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a74150a70 con 0x7f4a74153d70 2023-10-30T13:26:40.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a70ff9640 1 -- 172.21.15.148:0/133330527 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4a6c014cb0 con 0x7f4a74153d70 2023-10-30T13:26:40.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a70ff9640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a600664a0 0x7f4a60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:40.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a70ff9640 1 -- 172.21.15.148:0/133330527 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f4a6c084a70 con 0x7f4a74153d70 2023-10-30T13:26:40.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.809+0000 7f4a73fff640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a600664a0 0x7f4a60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:40.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.813+0000 7f4a73fff640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a600664a0 0x7f4a60068960 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4a64007fd0 tx=0x7f4a64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:40.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.813+0000 7f4a70ff9640 1 -- 172.21.15.148:0/133330527 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4a74150a70 con 0x7f4a74153d70 2023-10-30T13:26:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.993+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a74150a70 con 0x7f4a74153d70 2023-10-30T13:26:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a70ff9640 1 -- 172.21.15.148:0/133330527 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f4a74150a70 con 0x7f4a74153d70 2023-10-30T13:26:40.991 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:40.992 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:40.992 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 75 2023-10-30T13:26:40.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a600664a0 msgr2=0x7f4a60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:40.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a600664a0 0x7f4a60068960 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4a64007fd0 tx=0x7f4a64009040 comp rx=0 tx=0).stop 2023-10-30T13:26:40.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a74153d70 msgr2=0x7f4a7413f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:40.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a74153d70 0x7f4a7413f730 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c007e70 tx=0x7f4a6c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 shutdown_connections 2023-10-30T13:26:40.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4a600664a0 0x7f4a60068960 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a74153d70 0x7f4a7413f730 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4a740c8490 0x7f4a74152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 --2- 172.21.15.148:0/133330527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a7406e470 0x7f4a741f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:40.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 >> 172.21.15.148:0/133330527 conn(0x7f4a74071ea0 msgr2=0x7f4a7414e330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:40.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 shutdown_connections 2023-10-30T13:26:40.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:40.997+0000 7f4a7a9c5640 1 -- 172.21.15.148:0/133330527 wait complete. 2023-10-30T13:26:41.012 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:26:43.014 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:43.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.165+0000 7f0bd6cea640 1 Processor -- start 2023-10-30T13:26:43.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.165+0000 7f0bd6cea640 1 -- start start 2023-10-30T13:26:43.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bd6cea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0bd00c5f60 0x7f0bd0071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bd6cea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00700b0 con 0x7f0bd00c6fe0 2023-10-30T13:26:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bd6cea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00c6530 con 0x7f0bd0153bf0 2023-10-30T13:26:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bd6cea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd00c6810 con 0x7f0bd00c5f60 2023-10-30T13:26:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc7fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0bd00c6fe0 0x7f0bd00cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59326/0 (socket says 172.21.15.148:59326) 2023-10-30T13:26:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc7fff640 1 -- 172.21.15.148:0/3817626403 learned_addr learned my addr 172.21.15.148:0/3817626403 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:26:43.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bd5287640 1 --2- 172.21.15.148:0/3817626403 >> v2:172.21.15.150:3300/0 conn(0x7f0bd00c5f60 0x7f0bd0071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764448734 0 0) 0x7f0bd00700b0 con 0x7f0bd00c6fe0 2023-10-30T13:26:43.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb8003530 con 0x7f0bd00c6fe0 2023-10-30T13:26:43.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376611974 0 0) 0x7f0bb8003530 con 0x7f0bd00c6fe0 2023-10-30T13:26:43.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 >> v2:172.21.15.150:3300/0 conn(0x7f0bd00c5f60 msgr2=0x7f0bd0071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:43.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 --2- 172.21.15.148:0/3817626403 >> v2:172.21.15.150:3300/0 conn(0x7f0bd00c5f60 0x7f0bd0071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 >> v1:172.21.15.150:6789/0 conn(0x7f0bd0153bf0 legacy=0x7f0bd00c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:43.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd0159b10 con 0x7f0bd00c6fe0 2023-10-30T13:26:43.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f0bc0001680 con 0x7f0bd00c6fe0 2023-10-30T13:26:43.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 >> v1:172.21.15.148:6789/0 conn(0x7f0bd00c6fe0 legacy=0x7f0bd00cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:43.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bb8003980 0x7f0bb8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bb80047d0 0x7f0bb8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bb8005190 0x7f0bb800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd00c6810 con 0x7f0bb80047d0 2023-10-30T13:26:43.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb800c040 con 0x7f0bb8005190 2023-10-30T13:26:43.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb800c200 con 0x7f0bb8003980 2023-10-30T13:26:43.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc7fff640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bb80047d0 0x7f0bb8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bd5287640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bb8005190 0x7f0bb800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bd4a86640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bb8003980 0x7f0bb8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc7fff640 1 -- 172.21.15.148:0/3817626403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bb8003980 msgr2=0x7f0bb8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:43.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc7fff640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bb8003980 0x7f0bb8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc7fff640 1 -- 172.21.15.148:0/3817626403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bb8005190 msgr2=0x7f0bb800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc7fff640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bb8005190 0x7f0bb800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bc7fff640 1 -- 172.21.15.148:0/3817626403 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8003530 con 0x7f0bb80047d0 2023-10-30T13:26:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bd5287640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bb8005190 0x7f0bb800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.169+0000 7f0bd5287640 1 -- 172.21.15.148:0/3817626403 reap_dead start 2023-10-30T13:26:43.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bc7fff640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bb80047d0 0x7f0bb8004c50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000c1c0 tx=0x7f0bc001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e160e71fbc4e377 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:43.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0005ae0 con 0x7f0bb80047d0 2023-10-30T13:26:43.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc0005c40 con 0x7f0bb80047d0 2023-10-30T13:26:43.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bc77fe640 1 -- 172.21.15.148:0/3817626403 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc00a5b20 con 0x7f0bb80047d0 2023-10-30T13:26:43.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bd6cea640 1 -- 172.21.15.148:0/3817626403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bb80047d0 msgr2=0x7f0bb8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:43.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bd6cea640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bb80047d0 0x7f0bb8004c50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000c1c0 tx=0x7f0bc001cbe0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bd6cea640 1 -- 172.21.15.148:0/3817626403 shutdown_connections 2023-10-30T13:26:43.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bd6cea640 1 --2- 172.21.15.148:0/3817626403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bb80047d0 0x7f0bb8004c50 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bd6cea640 1 -- 172.21.15.148:0/3817626403 >> 172.21.15.148:0/3817626403 conn(0x7f0bd006c6e0 msgr2=0x7f0bd0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bd6cea640 1 -- 172.21.15.148:0/3817626403 shutdown_connections 2023-10-30T13:26:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bd6cea640 1 -- 172.21.15.148:0/3817626403 wait complete. 2023-10-30T13:26:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bd6cea640 1 Processor -- start 2023-10-30T13:26:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.173+0000 7f0bd6cea640 1 -- start start 2023-10-30T13:26:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd6cea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bb80047d0 0x7f0bd01528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd6cea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bd0156c60 0x7f0bd0149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd6cea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bd01497c0 0x7f0bd0154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd6cea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd01522c0 con 0x7f0bd01497c0 2023-10-30T13:26:43.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd6cea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd014fca0 con 0x7f0bd0156c60 2023-10-30T13:26:43.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd6cea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0156830 con 0x7f0bb80047d0 2023-10-30T13:26:43.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd5287640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bd01497c0 0x7f0bd0154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd5287640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bd01497c0 0x7f0bd0154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44698/0 (socket says 172.21.15.148:44698) 2023-10-30T13:26:43.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd5287640 1 -- 172.21.15.148:0/508222366 learned_addr learned my addr 172.21.15.148:0/508222366 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:43.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd4a86640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bb80047d0 0x7f0bd01528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bc7fff640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bd0156c60 0x7f0bd0149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd5287640 1 -- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bb80047d0 msgr2=0x7f0bd01528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:43.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd5287640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bb80047d0 0x7f0bd01528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd5287640 1 -- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bd0156c60 msgr2=0x7f0bd0149280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:43.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd5287640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bd0156c60 0x7f0bd0149280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd5287640 1 -- 172.21.15.148:0/508222366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd0149c40 con 0x7f0bd01497c0 2023-10-30T13:26:43.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd5287640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bd01497c0 0x7f0bd0154300 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00c9100 tx=0x7f0bc8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:43.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bc57fa640 1 -- 172.21.15.148:0/508222366 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc800e050 con 0x7f0bd01497c0 2023-10-30T13:26:43.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bc57fa640 1 -- 172.21.15.148:0/508222366 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc800f040 con 0x7f0bd01497c0 2023-10-30T13:26:43.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bc57fa640 1 -- 172.21.15.148:0/508222366 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc80039a0 con 0x7f0bd01497c0 2023-10-30T13:26:43.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd0071440 con 0x7f0bd01497c0 2023-10-30T13:26:43.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd014e750 con 0x7f0bd01497c0 2023-10-30T13:26:43.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.177+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd0152400 con 0x7f0bd01497c0 2023-10-30T13:26:43.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.181+0000 7f0bc57fa640 1 -- 172.21.15.148:0/508222366 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0bc8015440 con 0x7f0bd01497c0 2023-10-30T13:26:43.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.181+0000 7f0bc57fa640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bbc0664a0 0x7f0bbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.181+0000 7f0bc57fa640 1 -- 172.21.15.148:0/508222366 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6456+0+0 (secure 0 0 0) 0x7f0bc80847a0 con 0x7f0bd01497c0 2023-10-30T13:26:43.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.181+0000 7f0bd4a86640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bbc0664a0 0x7f0bbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.181+0000 7f0bd4a86640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bbc0664a0 0x7f0bbc068960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0bd01e6aa0 tx=0x7f0bcc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:43.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.185+0000 7f0bc57fa640 1 -- 172.21.15.148:0/508222366 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0bd0152400 con 0x7f0bd01497c0 2023-10-30T13:26:43.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bd0152400 con 0x7f0bd01497c0 2023-10-30T13:26:43.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bc57fa640 1 -- 172.21.15.148:0/508222366 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f0bd0152400 con 0x7f0bd01497c0 2023-10-30T13:26:43.361 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:43.361 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:24:53.095655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_5955":{"gid":5955,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":18,"addr":"172.21.15.148:6837/1559416001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1559416001},{"type":"v1","addr":"172.21.15.148:6837","nonce":1559416001}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:43.361 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 75 2023-10-30T13:26:43.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bbc0664a0 msgr2=0x7f0bbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bbc0664a0 0x7f0bbc068960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0bd01e6aa0 tx=0x7f0bcc009040 comp rx=0 tx=0).stop 2023-10-30T13:26:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bd01497c0 msgr2=0x7f0bd0154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bd01497c0 0x7f0bd0154300 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00c9100 tx=0x7f0bc8010590 comp rx=0 tx=0).stop 2023-10-30T13:26:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 shutdown_connections 2023-10-30T13:26:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bbc0664a0 0x7f0bbc068960 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bd01497c0 0x7f0bd0154300 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bd0156c60 0x7f0bd0149280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 --2- 172.21.15.148:0/508222366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0bb80047d0 0x7f0bd01528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 >> 172.21.15.148:0/508222366 conn(0x7f0bd006c6e0 msgr2=0x7f0bd00c94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:43.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 shutdown_connections 2023-10-30T13:26:43.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:43.365+0000 7f0bd6cea640 1 -- 172.21.15.148:0/508222366 wait complete. 2023-10-30T13:26:43.382 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:26:45.383 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:45.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3ffd3640 1 Processor -- start 2023-10-30T13:26:45.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3ffd3640 1 -- start start 2023-10-30T13:26:45.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3ffd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c3815d6f0 0x7f4c3815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3ffd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4c380cf160 0x7f4c380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3ffd3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c380cb800 con 0x7f4c381538a0 2023-10-30T13:26:45.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3ffd3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c380cb330 con 0x7f4c380cf160 2023-10-30T13:26:45.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3ffd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c380cad70 con 0x7f4c3815d6f0 2023-10-30T13:26:45.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3dd6f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4c381538a0 0x7f4c380ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59330/0 (socket says 172.21.15.148:59330) 2023-10-30T13:26:45.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3dd6f640 1 -- 172.21.15.148:0/3311068952 learned_addr learned my addr 172.21.15.148:0/3311068952 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:26:45.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3e570640 1 --2- 172.21.15.148:0/3311068952 >> v2:172.21.15.150:3300/0 conn(0x7f4c380cf160 0x7f4c380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3d56e640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c3815d6f0 0x7f4c3815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310026720 0 0) 0x7f4c380cb800 con 0x7f4c381538a0 2023-10-30T13:26:45.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c24003530 con 0x7f4c381538a0 2023-10-30T13:26:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4162904890 0 0) 0x7f4c24003530 con 0x7f4c381538a0 2023-10-30T13:26:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c3815d6f0 msgr2=0x7f4c3815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c3815d6f0 0x7f4c3815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 >> v2:172.21.15.150:3300/0 conn(0x7f4c380cf160 msgr2=0x7f4c380cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 --2- 172.21.15.148:0/3311068952 >> v2:172.21.15.150:3300/0 conn(0x7f4c380cf160 0x7f4c380cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c380cb050 con 0x7f4c381538a0 2023-10-30T13:26:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f4c34001680 con 0x7f4c381538a0 2023-10-30T13:26:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 >> v1:172.21.15.148:6789/0 conn(0x7f4c381538a0 legacy=0x7f4c380ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c3815d6f0 0x7f4c24003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c24003f00 0x7f4c240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c24004d10 0x7f4c24005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c380cb330 con 0x7f4c24003f00 2023-10-30T13:26:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c380cad70 con 0x7f4c24004d10 2023-10-30T13:26:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2400b9c0 con 0x7f4c3815d6f0 2023-10-30T13:26:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3d56e640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c24003f00 0x7f4c240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3e570640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c24004d10 0x7f4c24005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.541+0000 7f4c3dd6f640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c3815d6f0 0x7f4c24003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3d56e640 1 -- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c3815d6f0 msgr2=0x7f4c24003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3d56e640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c3815d6f0 0x7f4c24003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3d56e640 1 -- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c24004d10 msgr2=0x7f4c24005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3d56e640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c24004d10 0x7f4c24005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3d56e640 1 -- 172.21.15.148:0/3311068952 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c24003530 con 0x7f4c24003f00 2023-10-30T13:26:45.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3dd6f640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c3815d6f0 0x7f4c24003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:45.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3d56e640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c24003f00 0x7f4c240047d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4c28009ae0 tx=0x7f4c2801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb0788fd6b5e1f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:45.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c280a5640 con 0x7f4c24003f00 2023-10-30T13:26:45.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c280a57a0 con 0x7f4c24003f00 2023-10-30T13:26:45.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3cd6d640 1 -- 172.21.15.148:0/3311068952 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c280a5b50 con 0x7f4c24003f00 2023-10-30T13:26:45.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/3311068952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c24003f00 msgr2=0x7f4c240047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:45.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3ffd3640 1 --2- 172.21.15.148:0/3311068952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c24003f00 0x7f4c240047d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4c28009ae0 tx=0x7f4c2801ca50 comp rx=0 tx=0).stop 2023-10-30T13:26:45.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3e570640 1 -- 172.21.15.148:0/3311068952 reap_dead start 2023-10-30T13:26:45.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/3311068952 shutdown_connections 2023-10-30T13:26:45.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/3311068952 >> 172.21.15.148:0/3311068952 conn(0x7f4c3806bfe0 msgr2=0x7f4c3806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:45.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.545+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/3311068952 shutdown_connections 2023-10-30T13:26:45.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/3311068952 wait complete. 2023-10-30T13:26:45.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3ffd3640 1 Processor -- start 2023-10-30T13:26:45.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3ffd3640 1 -- start start 2023-10-30T13:26:45.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3ffd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c24003f00 0x7f4c3814c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3ffd3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c381e5760 0x7f4c3814dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3ffd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c38150630 0x7f4c380c5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3ffd3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c381f3450 con 0x7f4c381e5760 2023-10-30T13:26:45.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3ffd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c381eef30 con 0x7f4c38150630 2023-10-30T13:26:45.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3ffd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c381eff00 con 0x7f4c24003f00 2023-10-30T13:26:45.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3d56e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c381e5760 0x7f4c3814dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3d56e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c381e5760 0x7f4c3814dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44716/0 (socket says 172.21.15.148:44716) 2023-10-30T13:26:45.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3d56e640 1 -- 172.21.15.148:0/2140383548 learned_addr learned my addr 172.21.15.148:0/2140383548 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:45.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3dd6f640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c24003f00 0x7f4c3814c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.549+0000 7f4c3e570640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c38150630 0x7f4c380c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3d56e640 1 -- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c24003f00 msgr2=0x7f4c3814c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:45.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3d56e640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c24003f00 0x7f4c3814c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3d56e640 1 -- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c38150630 msgr2=0x7f4c380c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:45.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3d56e640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c38150630 0x7f4c380c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3d56e640 1 -- 172.21.15.148:0/2140383548 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c381f3170 con 0x7f4c381e5760 2023-10-30T13:26:45.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3d56e640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c381e5760 0x7f4c3814dd60 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4c28009ae0 tx=0x7f4c24005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:45.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c2e7fc640 1 -- 172.21.15.148:0/2140383548 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c28004230 con 0x7f4c381e5760 2023-10-30T13:26:45.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c2e7fc640 1 -- 172.21.15.148:0/2140383548 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c28004390 con 0x7f4c381e5760 2023-10-30T13:26:45.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c2e7fc640 1 -- 172.21.15.148:0/2140383548 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c280ad850 con 0x7f4c381e5760 2023-10-30T13:26:45.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c381ef140 con 0x7f4c381e5760 2023-10-30T13:26:45.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c381ef700 con 0x7f4c381e5760 2023-10-30T13:26:45.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3804e170 con 0x7f4c381e5760 2023-10-30T13:26:45.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c2e7fc640 1 -- 172.21.15.148:0/2140383548 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4c280b4040 con 0x7f4c381e5760 2023-10-30T13:26:45.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c2e7fc640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c1c0664a0 0x7f4c1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c2e7fc640 1 -- 172.21.15.148:0/2140383548 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f4c2811e830 con 0x7f4c381e5760 2023-10-30T13:26:45.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.553+0000 7f4c3dd6f640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c1c0664a0 0x7f4c1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.557+0000 7f4c3dd6f640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c1c0664a0 0x7f4c1c068960 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4c34001060 tx=0x7f4c34007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:45.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.557+0000 7f4c2e7fc640 1 -- 172.21.15.148:0/2140383548 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4c280ebab0 con 0x7f4c381e5760 2023-10-30T13:26:45.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.741+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c38159b40 con 0x7f4c381e5760 2023-10-30T13:26:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.741+0000 7f4c2e7fc640 1 -- 172.21.15.148:0/2140383548 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f4c280ef0d0 con 0x7f4c381e5760 2023-10-30T13:26:45.738 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:45.738 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:26:44.595470+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 76 2023-10-30T13:26:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c1c0664a0 msgr2=0x7f4c1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c1c0664a0 0x7f4c1c068960 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4c34001060 tx=0x7f4c34007040 comp rx=0 tx=0).stop 2023-10-30T13:26:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c381e5760 msgr2=0x7f4c3814dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c381e5760 0x7f4c3814dd60 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4c28009ae0 tx=0x7f4c24005800 comp rx=0 tx=0).stop 2023-10-30T13:26:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 shutdown_connections 2023-10-30T13:26:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c1c0664a0 0x7f4c1c068960 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c381e5760 0x7f4c3814dd60 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c38150630 0x7f4c380c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 --2- 172.21.15.148:0/2140383548 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c24003f00 0x7f4c3814c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 >> 172.21.15.148:0/2140383548 conn(0x7f4c3806bfe0 msgr2=0x7f4c38065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 shutdown_connections 2023-10-30T13:26:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.745+0000 7f4c3ffd3640 1 -- 172.21.15.148:0/2140383548 wait complete. 2023-10-30T13:26:45.759 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:26:45.761 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:26:45.761 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:45.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa39572640 1 Processor -- start 2023-10-30T13:26:45.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa39572640 1 -- start start 2023-10-30T13:26:45.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa39572640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffa340c90d0 0x7ffa340cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa39572640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffa340cf4d0 0x7ffa3415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa39572640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa3415ce30 con 0x7ffa340c8cf0 2023-10-30T13:26:45.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa39572640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa34159880 con 0x7ffa340c90d0 2023-10-30T13:26:45.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa39572640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa34159ae0 con 0x7ffa340cf4d0 2023-10-30T13:26:45.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa327fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffa340c90d0 0x7ffa340cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa327fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffa340c90d0 0x7ffa340cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44718/0 (socket says 172.21.15.148:44718) 2023-10-30T13:26:45.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa327fc640 1 -- 172.21.15.148:0/2486161356 learned_addr learned my addr 172.21.15.148:0/2486161356 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:45.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa337fe640 1 --2- 172.21.15.148:0/2486161356 >> v2:172.21.15.150:3300/0 conn(0x7ffa340cf4d0 0x7ffa3415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa327fc640 1 -- 172.21.15.148:0/2486161356 >> v2:172.21.15.150:3300/0 conn(0x7ffa340cf4d0 msgr2=0x7ffa3415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:45.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa327fc640 1 --2- 172.21.15.148:0/2486161356 >> v2:172.21.15.150:3300/0 conn(0x7ffa340cf4d0 0x7ffa3415d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa327fc640 1 -- 172.21.15.148:0/2486161356 >> v1:172.21.15.150:6789/0 conn(0x7ffa340c8cf0 legacy=0x7ffa340c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:45.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.905+0000 7ffa327fc640 1 -- 172.21.15.148:0/2486161356 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa34159dc0 con 0x7ffa340c90d0 2023-10-30T13:26:45.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa327fc640 1 --2- 172.21.15.148:0/2486161356 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c90d0 0x7ffa340cef90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7ffa340c60c0 tx=0x7ffa2801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77310d368df277c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:45.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa31ffb640 1 -- 172.21.15.148:0/2486161356 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa280ac640 con 0x7ffa340c90d0 2023-10-30T13:26:45.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa31ffb640 1 -- 172.21.15.148:0/2486161356 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa280ac7a0 con 0x7ffa340c90d0 2023-10-30T13:26:45.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa31ffb640 1 -- 172.21.15.148:0/2486161356 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa280acb30 con 0x7ffa340c90d0 2023-10-30T13:26:45.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 -- 172.21.15.148:0/2486161356 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c90d0 msgr2=0x7ffa340cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:45.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 --2- 172.21.15.148:0/2486161356 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c90d0 0x7ffa340cef90 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7ffa340c60c0 tx=0x7ffa2801c4d0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 -- 172.21.15.148:0/2486161356 shutdown_connections 2023-10-30T13:26:45.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 --2- 172.21.15.148:0/2486161356 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c90d0 0x7ffa340cef90 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 --2- 172.21.15.148:0/2486161356 >> v2:172.21.15.150:3300/0 conn(0x7ffa340cf4d0 0x7ffa3415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 -- 172.21.15.148:0/2486161356 >> 172.21.15.148:0/2486161356 conn(0x7ffa3406be40 msgr2=0x7ffa3406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:45.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 -- 172.21.15.148:0/2486161356 shutdown_connections 2023-10-30T13:26:45.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 -- 172.21.15.148:0/2486161356 wait complete. 2023-10-30T13:26:45.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 Processor -- start 2023-10-30T13:26:45.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.909+0000 7ffa39572640 1 -- start start 2023-10-30T13:26:45.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa39572640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c8cf0 0x7ffa341f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa39572640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c90d0 0x7ffa340c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa39572640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa340cf4d0 0x7ffa341ebad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa39572640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa341f20f0 con 0x7ffa340c8cf0 2023-10-30T13:26:45.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa39572640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa341f65b0 con 0x7ffa340cf4d0 2023-10-30T13:26:45.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa39572640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa341f2300 con 0x7ffa340c90d0 2023-10-30T13:26:45.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa32ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c8cf0 0x7ffa341f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa32ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c8cf0 0x7ffa341f6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44726/0 (socket says 172.21.15.148:44726) 2023-10-30T13:26:45.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa32ffd640 1 -- 172.21.15.148:0/4022596545 learned_addr learned my addr 172.21.15.148:0/4022596545 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:45.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa337fe640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa340cf4d0 0x7ffa341ebad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa327fc640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c90d0 0x7ffa340c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa32ffd640 1 -- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c90d0 msgr2=0x7ffa340c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:45.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa32ffd640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c90d0 0x7ffa340c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa32ffd640 1 -- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa340cf4d0 msgr2=0x7ffa341ebad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:45.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa32ffd640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa340cf4d0 0x7ffa341ebad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:45.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa32ffd640 1 -- 172.21.15.148:0/4022596545 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa280094d0 con 0x7ffa340c8cf0 2023-10-30T13:26:45.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa32ffd640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c8cf0 0x7ffa341f6070 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400c3c0 tx=0x7ffa2400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:45.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa1b7fe640 1 -- 172.21.15.148:0/4022596545 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa240052d0 con 0x7ffa340c8cf0 2023-10-30T13:26:45.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa1b7fe640 1 -- 172.21.15.148:0/4022596545 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24005430 con 0x7ffa340c8cf0 2023-10-30T13:26:45.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa1b7fe640 1 -- 172.21.15.148:0/4022596545 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24003840 con 0x7ffa340c8cf0 2023-10-30T13:26:45.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa341f25a0 con 0x7ffa340c8cf0 2023-10-30T13:26:45.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa341f2b60 con 0x7ffa340c8cf0 2023-10-30T13:26:45.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3404e170 con 0x7ffa340c8cf0 2023-10-30T13:26:45.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.913+0000 7ffa1b7fe640 1 -- 172.21.15.148:0/4022596545 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffa240039a0 con 0x7ffa340c8cf0 2023-10-30T13:26:45.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.917+0000 7ffa1b7fe640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa200664a0 0x7ffa20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:45.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.917+0000 7ffa1b7fe640 1 -- 172.21.15.148:0/4022596545 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7ffa24083790 con 0x7ffa340c8cf0 2023-10-30T13:26:45.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.917+0000 7ffa327fc640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa200664a0 0x7ffa20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:45.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.917+0000 7ffa327fc640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa200664a0 0x7ffa20068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ffa280ae000 tx=0x7ffa28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:45.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:45.921+0000 7ffa1b7fe640 1 -- 172.21.15.148:0/4022596545 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffa24050850 con 0x7ffa340c8cf0 2023-10-30T13:26:46.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa34157290 con 0x7ffa340c8cf0 2023-10-30T13:26:46.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa1b7fe640 1 -- 172.21.15.148:0/4022596545 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5716 (secure 0 0 0) 0x7ffa24053e70 con 0x7ffa340c8cf0 2023-10-30T13:26:46.096 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:46.096 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:26:44.595470+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:46.097 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 76 2023-10-30T13:26:46.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa200664a0 msgr2=0x7ffa20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:46.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa200664a0 0x7ffa20068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ffa280ae000 tx=0x7ffa28008040 comp rx=0 tx=0).stop 2023-10-30T13:26:46.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c8cf0 msgr2=0x7ffa341f6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:46.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c8cf0 0x7ffa341f6070 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400c3c0 tx=0x7ffa2400c880 comp rx=0 tx=0).stop 2023-10-30T13:26:46.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 shutdown_connections 2023-10-30T13:26:46.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa200664a0 0x7ffa20068960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:46.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa340cf4d0 0x7ffa341ebad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:46.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c90d0 0x7ffa340c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:46.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 --2- 172.21.15.148:0/4022596545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c8cf0 0x7ffa341f6070 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:46.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 >> 172.21.15.148:0/4022596545 conn(0x7ffa3406be40 msgr2=0x7ffa340c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:46.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.101+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 shutdown_connections 2023-10-30T13:26:46.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:46.105+0000 7ffa39572640 1 -- 172.21.15.148:0/4022596545 wait complete. 2023-10-30T13:26:46.119 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 76, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 76, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6099": { "addr": "172.21.15.148:6835/2733100982", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 2733100982, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 2733100982, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6099, "incarnation": 76, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:26:44.595470+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6099 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:26:48.121 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:48.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f21796d6640 1 Processor -- start 2023-10-30T13:26:48.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f21796d6640 1 -- start start 2023-10-30T13:26:48.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f21796d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f21740c6b80 0x7f21740bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:48.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f21796d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f21740c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:48.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f21796d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21740c0a50 con 0x7f21740c8430 2023-10-30T13:26:48.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f21796d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217406f6e0 con 0x7f21740c6b80 2023-10-30T13:26:48.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f21796d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217406f9c0 con 0x7f21740c6f60 2023-10-30T13:26:48.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f216a7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f21740c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:48.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f2172ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f21740c6b80 0x7f21740bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:48.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f216a7fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f21740c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:33442/0 (socket says 172.21.15.148:33442) 2023-10-30T13:26:48.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f216a7fc640 1 -- 172.21.15.148:0/3858918284 learned_addr learned my addr 172.21.15.148:0/3858918284 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:48.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f216a7fc640 1 -- 172.21.15.148:0/3858918284 >> v2:172.21.15.150:3300/0 conn(0x7f21740c6b80 msgr2=0x7f21740bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:48.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f216a7fc640 1 --2- 172.21.15.148:0/3858918284 >> v2:172.21.15.150:3300/0 conn(0x7f21740c6b80 0x7f21740bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:48.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f216a7fc640 1 -- 172.21.15.148:0/3858918284 >> v1:172.21.15.150:6789/0 conn(0x7f21740c8430 legacy=0x7f217413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:48.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f216a7fc640 1 -- 172.21.15.148:0/3858918284 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217406fca0 con 0x7f21740c6f60 2023-10-30T13:26:48.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.273+0000 7f2172ffd640 1 --2- 172.21.15.148:0/3858918284 >> v2:172.21.15.150:3300/0 conn(0x7f21740c6b80 0x7f21740bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:26:48.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f216a7fc640 1 --2- 172.21.15.148:0/3858918284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f21740c0400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f215c009e50 tx=0x7f215c01eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f690918fc04a92ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:48.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21727fc640 1 -- 172.21.15.148:0/3858918284 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c0a7970 con 0x7f21740c6f60 2023-10-30T13:26:48.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21727fc640 1 -- 172.21.15.148:0/3858918284 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215c0a7ad0 con 0x7f21740c6f60 2023-10-30T13:26:48.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21727fc640 1 -- 172.21.15.148:0/3858918284 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c0a7e60 con 0x7f21740c6f60 2023-10-30T13:26:48.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21796d6640 1 -- 172.21.15.148:0/3858918284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 msgr2=0x7f21740c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:48.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21796d6640 1 --2- 172.21.15.148:0/3858918284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f21740c0400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f215c009e50 tx=0x7f215c01eb90 comp rx=0 tx=0).stop 2023-10-30T13:26:48.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21796d6640 1 -- 172.21.15.148:0/3858918284 shutdown_connections 2023-10-30T13:26:48.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21796d6640 1 --2- 172.21.15.148:0/3858918284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f21740c0400 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:48.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21796d6640 1 --2- 172.21.15.148:0/3858918284 >> v2:172.21.15.150:3300/0 conn(0x7f21740c6b80 0x7f21740bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:48.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21796d6640 1 -- 172.21.15.148:0/3858918284 >> 172.21.15.148:0/3858918284 conn(0x7f21740c7910 msgr2=0x7f21740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:48.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21796d6640 1 -- 172.21.15.148:0/3858918284 shutdown_connections 2023-10-30T13:26:48.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.277+0000 7f21796d6640 1 -- 172.21.15.148:0/3858918284 wait complete. 2023-10-30T13:26:48.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21796d6640 1 Processor -- start 2023-10-30T13:26:48.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21796d6640 1 -- start start 2023-10-30T13:26:48.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21796d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21740c6b80 0x7f21741f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:48.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21796d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f217413a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:48.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21796d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f21740c8430 0x7f21740c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:48.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21796d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21741f1ed0 con 0x7f21740c8430 2023-10-30T13:26:48.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21796d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2174132860 con 0x7f21740c6b80 2023-10-30T13:26:48.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21796d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2174132b00 con 0x7f21740c6f60 2023-10-30T13:26:48.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21737fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f21740c8430 0x7f21740c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:48.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21737fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f21740c8430 0x7f21740c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44738/0 (socket says 172.21.15.148:44738) 2023-10-30T13:26:48.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21737fe640 1 -- 172.21.15.148:0/3758249958 learned_addr learned my addr 172.21.15.148:0/3758249958 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:48.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f216a7fc640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f217413a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:48.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f2172ffd640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21740c6b80 0x7f21741f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:48.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21737fe640 1 -- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 msgr2=0x7f217413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:48.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21737fe640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f217413a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:48.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21737fe640 1 -- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21740c6b80 msgr2=0x7f21741f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:48.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21737fe640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21740c6b80 0x7f21741f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:48.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21737fe640 1 -- 172.21.15.148:0/3758249958 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2164009420 con 0x7f21740c8430 2023-10-30T13:26:48.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21737fe640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f21740c8430 0x7f21740c9b90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c340 tx=0x7f216c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:48.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f216bfff640 1 -- 172.21.15.148:0/3758249958 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c005220 con 0x7f21740c8430 2023-10-30T13:26:48.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f216bfff640 1 -- 172.21.15.148:0/3758249958 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c005380 con 0x7f21740c8430 2023-10-30T13:26:48.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f216bfff640 1 -- 172.21.15.148:0/3758249958 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c007270 con 0x7f21740c8430 2023-10-30T13:26:48.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.281+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c009b30 con 0x7f21740c8430 2023-10-30T13:26:48.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.285+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21741330c0 con 0x7f21740c8430 2023-10-30T13:26:48.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.285+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217404e170 con 0x7f21740c8430 2023-10-30T13:26:48.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.285+0000 7f216bfff640 1 -- 172.21.15.148:0/3758249958 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f216c0073d0 con 0x7f21740c8430 2023-10-30T13:26:48.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.285+0000 7f216bfff640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21540664a0 0x7f2154068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:48.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.285+0000 7f216bfff640 1 -- 172.21.15.148:0/3758249958 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f216c083690 con 0x7f21740c8430 2023-10-30T13:26:48.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.285+0000 7f2172ffd640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21540664a0 0x7f2154068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:48.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.289+0000 7f2172ffd640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21540664a0 0x7f2154068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2164009cf0 tx=0x7f2164009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:48.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.289+0000 7f216bfff640 1 -- 172.21.15.148:0/3758249958 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f216c050750 con 0x7f21740c8430 2023-10-30T13:26:48.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.469+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21740c1770 con 0x7f21740c8430 2023-10-30T13:26:48.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.469+0000 7f216bfff640 1 -- 172.21.15.148:0/3758249958 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f216c053d70 con 0x7f21740c8430 2023-10-30T13:26:48.467 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:48.467 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:26:44.595470+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:48.467 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 76 2023-10-30T13:26:48.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21540664a0 msgr2=0x7f2154068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21540664a0 0x7f2154068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2164009cf0 tx=0x7f2164009190 comp rx=0 tx=0).stop 2023-10-30T13:26:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f21740c8430 msgr2=0x7f21740c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f21740c8430 0x7f21740c9b90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c340 tx=0x7f216c00c800 comp rx=0 tx=0).stop 2023-10-30T13:26:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 shutdown_connections 2023-10-30T13:26:48.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21540664a0 0x7f2154068960 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f21740c8430 0x7f21740c9b90 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f21740c6f60 0x7f217413a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 --2- 172.21.15.148:0/3758249958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21740c6b80 0x7f21741f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 >> 172.21.15.148:0/3758249958 conn(0x7f21740c7910 msgr2=0x7f217413ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:48.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 shutdown_connections 2023-10-30T13:26:48.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:48.473+0000 7f21796d6640 1 -- 172.21.15.148:0/3758249958 wait complete. 2023-10-30T13:26:48.488 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:50.489 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:50.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdab9165640 1 Processor -- start 2023-10-30T13:26:50.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdab9165640 1 -- start start 2023-10-30T13:26:50.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdab9165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab41538a0 0x7fdab415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:50.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdab9165640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdab40ca0e0 0x7fdab415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:50.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdab9165640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab40cb800 con 0x7fdab40cf160 2023-10-30T13:26:50.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdab9165640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab415dc30 con 0x7fdab40ca0e0 2023-10-30T13:26:50.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdab9165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab40cb330 con 0x7fdab41538a0 2023-10-30T13:26:50.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdaabfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdab40ca0e0 0x7fdab415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:50.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdaabfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdab40ca0e0 0x7fdab415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44750/0 (socket says 172.21.15.148:44750) 2023-10-30T13:26:50.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdab359e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fdab40cf160 0x7fdab40cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59332/0 (socket says 172.21.15.148:59332) 2023-10-30T13:26:50.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdaabfff640 1 -- 172.21.15.148:0/1968513948 learned_addr learned my addr 172.21.15.148:0/1968513948 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:50.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdab2d9d640 1 --2- 172.21.15.148:0/1968513948 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab41538a0 0x7fdab415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:50.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdaabfff640 1 -- 172.21.15.148:0/1968513948 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab41538a0 msgr2=0x7fdab415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:50.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdaabfff640 1 --2- 172.21.15.148:0/1968513948 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab41538a0 0x7fdab415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdaabfff640 1 -- 172.21.15.148:0/1968513948 >> v1:172.21.15.148:6789/0 conn(0x7fdab40cf160 legacy=0x7fdab40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:50.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.641+0000 7fdaabfff640 1 -- 172.21.15.148:0/1968513948 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab40cad70 con 0x7fdab40ca0e0 2023-10-30T13:26:50.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdaabfff640 1 --2- 172.21.15.148:0/1968513948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40ca0e0 0x7fdab415a090 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009870 tx=0x7fda9c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7f99cf349a70f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab259c640 1 -- 172.21.15.148:0/1968513948 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c0a8a30 con 0x7fdab40ca0e0 2023-10-30T13:26:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab259c640 1 -- 172.21.15.148:0/1968513948 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c0a8b90 con 0x7fdab40ca0e0 2023-10-30T13:26:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab259c640 1 -- 172.21.15.148:0/1968513948 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c0b0440 con 0x7fdab40ca0e0 2023-10-30T13:26:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab9165640 1 -- 172.21.15.148:0/1968513948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40ca0e0 msgr2=0x7fdab415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab9165640 1 --2- 172.21.15.148:0/1968513948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40ca0e0 0x7fdab415a090 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009870 tx=0x7fda9c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:26:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab9165640 1 -- 172.21.15.148:0/1968513948 shutdown_connections 2023-10-30T13:26:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab9165640 1 --2- 172.21.15.148:0/1968513948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40ca0e0 0x7fdab415a090 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab9165640 1 --2- 172.21.15.148:0/1968513948 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab41538a0 0x7fdab415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab9165640 1 -- 172.21.15.148:0/1968513948 >> 172.21.15.148:0/1968513948 conn(0x7fdab406bfe0 msgr2=0x7fdab406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab9165640 1 -- 172.21.15.148:0/1968513948 shutdown_connections 2023-10-30T13:26:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.649+0000 7fdab9165640 1 -- 172.21.15.148:0/1968513948 wait complete. 2023-10-30T13:26:50.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 Processor -- start 2023-10-30T13:26:50.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 -- start start 2023-10-30T13:26:50.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab40ca0e0 0x7fdab41f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:50.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40cf160 0x7fdab41e3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:50.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab41538a0 0x7fdab41e4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:50.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab41e5500 con 0x7fdab40cf160 2023-10-30T13:26:50.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab40b4060 con 0x7fdab41538a0 2023-10-30T13:26:50.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab41e3220 con 0x7fdab40ca0e0 2023-10-30T13:26:50.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaabfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40cf160 0x7fdab41e3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:50.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaabfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40cf160 0x7fdab41e3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44752/0 (socket says 172.21.15.148:44752) 2023-10-30T13:26:50.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaabfff640 1 -- 172.21.15.148:0/1266751331 learned_addr learned my addr 172.21.15.148:0/1266751331 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:50.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab2d9d640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab40ca0e0 0x7fdab41f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:50.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab359e640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab41538a0 0x7fdab41e4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:50.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaabfff640 1 -- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab40ca0e0 msgr2=0x7fdab41f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:50.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaabfff640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab40ca0e0 0x7fdab41f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaabfff640 1 -- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab41538a0 msgr2=0x7fdab41e4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:50.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaabfff640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab41538a0 0x7fdab41e4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaabfff640 1 -- 172.21.15.148:0/1266751331 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c0094d0 con 0x7fdab40cf160 2023-10-30T13:26:50.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaabfff640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40cf160 0x7fdab41e3db0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0b3040 tx=0x7fda9c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:50.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaab7fe640 1 -- 172.21.15.148:0/1266751331 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c0be070 con 0x7fdab40cf160 2023-10-30T13:26:50.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaab7fe640 1 -- 172.21.15.148:0/1266751331 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c0ba400 con 0x7fdab40cf160 2023-10-30T13:26:50.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab40b3dc0 con 0x7fdab40cf160 2023-10-30T13:26:50.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdaab7fe640 1 -- 172.21.15.148:0/1266751331 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9c0ba7d0 con 0x7fdab40cf160 2023-10-30T13:26:50.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.653+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab41e2f40 con 0x7fdab40cf160 2023-10-30T13:26:50.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.657+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab40b2a50 con 0x7fdab40cf160 2023-10-30T13:26:50.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.657+0000 7fdaab7fe640 1 -- 172.21.15.148:0/1266751331 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda9c0c3410 con 0x7fdab40cf160 2023-10-30T13:26:50.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.657+0000 7fdaab7fe640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda940664a0 0x7fda94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:50.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.657+0000 7fdaab7fe640 1 -- 172.21.15.148:0/1266751331 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fda9c121e70 con 0x7fdab40cf160 2023-10-30T13:26:50.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.657+0000 7fdab2d9d640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda940664a0 0x7fda94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:50.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.661+0000 7fdab2d9d640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda940664a0 0x7fda94068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400b930 tx=0x7fdaa4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:50.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.665+0000 7fdaab7fe640 1 -- 172.21.15.148:0/1266751331 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda9c0eef10 con 0x7fdab40cf160 2023-10-30T13:26:50.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.857+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdab404e170 con 0x7fdab40cf160 2023-10-30T13:26:50.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.857+0000 7fdaab7fe640 1 -- 172.21.15.148:0/1266751331 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5716 (secure 0 0 0) 0x7fda9c0f2530 con 0x7fdab40cf160 2023-10-30T13:26:50.854 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:50.855 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:26:44.595470+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:50.855 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 76 2023-10-30T13:26:50.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda940664a0 msgr2=0x7fda94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:50.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda940664a0 0x7fda94068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400b930 tx=0x7fdaa4008040 comp rx=0 tx=0).stop 2023-10-30T13:26:50.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40cf160 msgr2=0x7fdab41e3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:50.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40cf160 0x7fdab41e3db0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0b3040 tx=0x7fda9c01bad0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 shutdown_connections 2023-10-30T13:26:50.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fda940664a0 0x7fda94068960 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdab40cf160 0x7fdab41e3db0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab41538a0 0x7fdab41e4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 --2- 172.21.15.148:0/1266751331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdab40ca0e0 0x7fdab41f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:50.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 >> 172.21.15.148:0/1266751331 conn(0x7fdab406bfe0 msgr2=0x7fdab4151580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:50.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 shutdown_connections 2023-10-30T13:26:50.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:50.861+0000 7fdab9165640 1 -- 172.21.15.148:0/1266751331 wait complete. 2023-10-30T13:26:50.877 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:52.878 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:53.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47f972640 1 Processor -- start 2023-10-30T13:26:53.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47f972640 1 -- start start 2023-10-30T13:26:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47f972640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4780c6f60 0x7fe4781566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47f972640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4780c0a50 con 0x7fe4780c05d0 2023-10-30T13:26:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47f972640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47806f6e0 con 0x7fe4780c6b80 2023-10-30T13:26:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47f972640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47806f9c0 con 0x7fe4780c6f60 2023-10-30T13:26:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47df0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4780c6f60 0x7fe4781566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47df0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4780c6f60 0x7fe4781566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46464/0 (socket says 172.21.15.148:46464) 2023-10-30T13:26:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47df0f640 1 -- 172.21.15.148:0/3481096024 learned_addr learned my addr 172.21.15.148:0/3481096024 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47df0f640 1 -- 172.21.15.148:0/3481096024 >> v1:172.21.15.150:6789/0 conn(0x7fe4780c6b80 legacy=0x7fe4780bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47df0f640 1 -- 172.21.15.148:0/3481096024 >> v1:172.21.15.148:6789/0 conn(0x7fe4780c05d0 legacy=0x7fe47813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.025+0000 7fe47df0f640 1 -- 172.21.15.148:0/3481096024 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47806fca0 con 0x7fe4780c6f60 2023-10-30T13:26:53.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe47df0f640 1 --2- 172.21.15.148:0/3481096024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c6f60 0x7fe4781566a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe470009e50 tx=0x7fe47001ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7457faa4a6e9417 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:53.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe46ffff640 1 -- 172.21.15.148:0/3481096024 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4700a7900 con 0x7fe4780c6f60 2023-10-30T13:26:53.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe46ffff640 1 -- 172.21.15.148:0/3481096024 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4700a7a60 con 0x7fe4780c6f60 2023-10-30T13:26:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe46ffff640 1 -- 172.21.15.148:0/3481096024 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4700a7df0 con 0x7fe4780c6f60 2023-10-30T13:26:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe47f972640 1 -- 172.21.15.148:0/3481096024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c6f60 msgr2=0x7fe4781566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe47f972640 1 --2- 172.21.15.148:0/3481096024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c6f60 0x7fe4781566a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe470009e50 tx=0x7fe47001ec20 comp rx=0 tx=0).stop 2023-10-30T13:26:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe47f972640 1 -- 172.21.15.148:0/3481096024 shutdown_connections 2023-10-30T13:26:53.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe47f972640 1 --2- 172.21.15.148:0/3481096024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c6f60 0x7fe4781566a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:53.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe47f972640 1 -- 172.21.15.148:0/3481096024 >> 172.21.15.148:0/3481096024 conn(0x7fe4780c7910 msgr2=0x7fe4780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:53.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe47f972640 1 -- 172.21.15.148:0/3481096024 shutdown_connections 2023-10-30T13:26:53.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.029+0000 7fe47f972640 1 -- 172.21.15.148:0/3481096024 wait complete. 2023-10-30T13:26:53.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 Processor -- start 2023-10-30T13:26:53.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 -- start start 2023-10-30T13:26:53.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c05d0 0x7fe478157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:53.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4780c6b80 0x7fe47813ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:53.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4780c6f60 0x7fe47814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:53.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47814be30 con 0x7fe4780c05d0 2023-10-30T13:26:53.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47813e840 con 0x7fe4780c6b80 2023-10-30T13:26:53.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe478159290 con 0x7fe4780c6f60 2023-10-30T13:26:53.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47d70e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c05d0 0x7fe478157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:53.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47d70e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c05d0 0x7fe478157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46480/0 (socket says 172.21.15.148:46480) 2023-10-30T13:26:53.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47d70e640 1 -- 172.21.15.148:0/489399960 learned_addr learned my addr 172.21.15.148:0/489399960 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:53.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47df0f640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4780c6f60 0x7fe47814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47cf0d640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4780c6b80 0x7fe47813ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47d70e640 1 -- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4780c6f60 msgr2=0x7fe47814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47d70e640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4780c6f60 0x7fe47814c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47d70e640 1 -- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4780c6b80 msgr2=0x7fe47813ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:53.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47d70e640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4780c6b80 0x7fe47813ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47d70e640 1 -- 172.21.15.148:0/489399960 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470009b30 con 0x7fe4780c05d0 2023-10-30T13:26:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47d70e640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c05d0 0x7fe478157750 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe47400c500 tx=0x7fe47400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe46dffb640 1 -- 172.21.15.148:0/489399960 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474005220 con 0x7fe4780c05d0 2023-10-30T13:26:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe46dffb640 1 -- 172.21.15.148:0/489399960 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe474005380 con 0x7fe4780c05d0 2023-10-30T13:26:53.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe46dffb640 1 -- 172.21.15.148:0/489399960 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474007270 con 0x7fe4780c05d0 2023-10-30T13:26:53.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe478159550 con 0x7fe4780c05d0 2023-10-30T13:26:53.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4780ca760 con 0x7fe4780c05d0 2023-10-30T13:26:53.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.033+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47804e170 con 0x7fe4780c05d0 2023-10-30T13:26:53.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.037+0000 7fe46dffb640 1 -- 172.21.15.148:0/489399960 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe4740073d0 con 0x7fe4780c05d0 2023-10-30T13:26:53.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.037+0000 7fe46dffb640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe45c0664a0 0x7fe45c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:53.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.037+0000 7fe46dffb640 1 -- 172.21.15.148:0/489399960 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fe474083b30 con 0x7fe4780c05d0 2023-10-30T13:26:53.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.037+0000 7fe47cf0d640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe45c0664a0 0x7fe45c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:53.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.037+0000 7fe47cf0d640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe45c0664a0 0x7fe45c068960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe4780c0510 tx=0x7fe468009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:53.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.041+0000 7fe46dffb640 1 -- 172.21.15.148:0/489399960 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe474050be0 con 0x7fe4780c05d0 2023-10-30T13:26:53.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4780c7560 con 0x7fe4780c05d0 2023-10-30T13:26:53.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe46dffb640 1 -- 172.21.15.148:0/489399960 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5716 (secure 0 0 0) 0x7fe474054200 con 0x7fe4780c05d0 2023-10-30T13:26:53.217 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:53.217 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:26:44.595470+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:53.218 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 76 2023-10-30T13:26:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe45c0664a0 msgr2=0x7fe45c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe45c0664a0 0x7fe45c068960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe4780c0510 tx=0x7fe468009040 comp rx=0 tx=0).stop 2023-10-30T13:26:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c05d0 msgr2=0x7fe478157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:53.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c05d0 0x7fe478157750 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe47400c500 tx=0x7fe47400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:26:53.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 shutdown_connections 2023-10-30T13:26:53.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe45c0664a0 0x7fe45c068960 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:53.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4780c6f60 0x7fe47814c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:53.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4780c6b80 0x7fe47813ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:53.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 --2- 172.21.15.148:0/489399960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4780c05d0 0x7fe478157750 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:53.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 >> 172.21.15.148:0/489399960 conn(0x7fe4780c7910 msgr2=0x7fe4780c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:53.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.221+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 shutdown_connections 2023-10-30T13:26:53.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:53.225+0000 7fe47f972640 1 -- 172.21.15.148:0/489399960 wait complete. 2023-10-30T13:26:53.238 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:55.238 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:55.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7553c9e640 1 Processor -- start 2023-10-30T13:26:55.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7553c9e640 1 -- start start 2023-10-30T13:26:55.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7553c9e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f754c0c6bd0 0x7f754c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:55.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7553c9e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:55.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7553c9e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c066cf0 con 0x7f754c06f7a0 2023-10-30T13:26:55.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7553c9e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754c0700c0 con 0x7f754c0c6bd0 2023-10-30T13:26:55.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7553c9e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c0c70d0 con 0x7f754c071110 2023-10-30T13:26:55.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7551239640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f754c0c6bd0 0x7f754c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:55.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7551239640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f754c0c6bd0 0x7f754c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46496/0 (socket says 172.21.15.148:46496) 2023-10-30T13:26:55.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7551239640 1 -- 172.21.15.148:0/1580512955 learned_addr learned my addr 172.21.15.148:0/1580512955 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:55.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f755223b640 1 --2- 172.21.15.148:0/1580512955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:55.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7551239640 1 -- 172.21.15.148:0/1580512955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 msgr2=0x7f754c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:55.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7551239640 1 --2- 172.21.15.148:0/1580512955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7551239640 1 -- 172.21.15.148:0/1580512955 >> v1:172.21.15.150:6789/0 conn(0x7f754c06f7a0 legacy=0x7f754c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:55.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f7551239640 1 -- 172.21.15.148:0/1580512955 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c0c7390 con 0x7f754c0c6bd0 2023-10-30T13:26:55.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.392+0000 7f755223b640 1 --2- 172.21.15.148:0/1580512955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:55.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7551239640 1 --2- 172.21.15.148:0/1580512955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 0x7f754c06fb80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f753c009870 tx=0x7f753c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e6b4708b08236d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:55.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7550a38640 1 -- 172.21.15.148:0/1580512955 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c0a8a30 con 0x7f754c0c6bd0 2023-10-30T13:26:55.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7550a38640 1 -- 172.21.15.148:0/1580512955 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c0a8b90 con 0x7f754c0c6bd0 2023-10-30T13:26:55.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7550a38640 1 -- 172.21.15.148:0/1580512955 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c0b0440 con 0x7f754c0c6bd0 2023-10-30T13:26:55.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7553c9e640 1 -- 172.21.15.148:0/1580512955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 msgr2=0x7f754c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:55.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7553c9e640 1 --2- 172.21.15.148:0/1580512955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 0x7f754c06fb80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f753c009870 tx=0x7f753c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:26:55.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7553c9e640 1 -- 172.21.15.148:0/1580512955 shutdown_connections 2023-10-30T13:26:55.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7553c9e640 1 --2- 172.21.15.148:0/1580512955 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7553c9e640 1 --2- 172.21.15.148:0/1580512955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 0x7f754c06fb80 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7553c9e640 1 -- 172.21.15.148:0/1580512955 >> 172.21.15.148:0/1580512955 conn(0x7f754c0c7910 msgr2=0x7f754c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:55.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7553c9e640 1 -- 172.21.15.148:0/1580512955 shutdown_connections 2023-10-30T13:26:55.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.396+0000 7f7553c9e640 1 -- 172.21.15.148:0/1580512955 wait complete. 2023-10-30T13:26:55.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 Processor -- start 2023-10-30T13:26:55.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 -- start start 2023-10-30T13:26:55.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f754c06f7a0 0x7f754c156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:55.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:55.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 0x7f754c153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:55.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1ef5d0 con 0x7f754c0c6bd0 2023-10-30T13:26:55.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1531f0 con 0x7f754c06f7a0 2023-10-30T13:26:55.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1efac0 con 0x7f754c071110 2023-10-30T13:26:55.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f755223b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 0x7f754c153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:55.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f755223b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 0x7f754c153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46508/0 (socket says 172.21.15.148:46508) 2023-10-30T13:26:55.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f755223b640 1 -- 172.21.15.148:0/2329994870 learned_addr learned my addr 172.21.15.148:0/2329994870 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:55.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7551a3a640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f754c06f7a0 0x7f754c156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:55.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7551239640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:55.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f755223b640 1 -- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 msgr2=0x7f754c153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:55.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f755223b640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f755223b640 1 -- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f754c06f7a0 msgr2=0x7f754c156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:55.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f755223b640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f754c06f7a0 0x7f754c156da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f755223b640 1 -- 172.21.15.148:0/2329994870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c0094d0 con 0x7f754c0c6bd0 2023-10-30T13:26:55.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7551239640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:55.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f755223b640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 0x7f754c153bc0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7544007ef0 tx=0x7f754400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:55.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f75427fc640 1 -- 172.21.15.148:0/2329994870 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544003ba0 con 0x7f754c0c6bd0 2023-10-30T13:26:55.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f75427fc640 1 -- 172.21.15.148:0/2329994870 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754400f040 con 0x7f754c0c6bd0 2023-10-30T13:26:55.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c152ec0 con 0x7f754c0c6bd0 2023-10-30T13:26:55.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f75427fc640 1 -- 172.21.15.148:0/2329994870 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754400de80 con 0x7f754c0c6bd0 2023-10-30T13:26:55.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c1ef7e0 con 0x7f754c0c6bd0 2023-10-30T13:26:55.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.400+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754c153360 con 0x7f754c0c6bd0 2023-10-30T13:26:55.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.404+0000 7f75427fc640 1 -- 172.21.15.148:0/2329994870 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7544004030 con 0x7f754c0c6bd0 2023-10-30T13:26:55.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.404+0000 7f75427fc640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75300664a0 0x7f7530068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:55.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.404+0000 7f75427fc640 1 -- 172.21.15.148:0/2329994870 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f7544084440 con 0x7f754c0c6bd0 2023-10-30T13:26:55.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.404+0000 7f7551a3a640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75300664a0 0x7f7530068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:55.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.404+0000 7f7551a3a640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75300664a0 0x7f7530068960 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7548007fd0 tx=0x7f7548009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:55.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.408+0000 7f75427fc640 1 -- 172.21.15.148:0/2329994870 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7544051620 con 0x7f754c0c6bd0 2023-10-30T13:26:55.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f754c04e170 con 0x7f754c0c6bd0 2023-10-30T13:26:55.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f75427fc640 1 -- 172.21.15.148:0/2329994870 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f7544054c40 con 0x7f754c0c6bd0 2023-10-30T13:26:55.585 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:55.585 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:26:52.914120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:55.585 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 77 2023-10-30T13:26:55.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75300664a0 msgr2=0x7f7530068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:55.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75300664a0 0x7f7530068960 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7548007fd0 tx=0x7f7548009040 comp rx=0 tx=0).stop 2023-10-30T13:26:55.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 msgr2=0x7f754c153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:55.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 0x7f754c153bc0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7544007ef0 tx=0x7f754400d4a0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 shutdown_connections 2023-10-30T13:26:55.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75300664a0 0x7f7530068960 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f754c0c6bd0 0x7f754c153bc0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f754c071110 0x7f754c153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 --2- 172.21.15.148:0/2329994870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f754c06f7a0 0x7f754c156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:55.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 >> 172.21.15.148:0/2329994870 conn(0x7f754c0c7910 msgr2=0x7f754c0c5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:55.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 shutdown_connections 2023-10-30T13:26:55.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:55.588+0000 7f7553c9e640 1 -- 172.21.15.148:0/2329994870 wait complete. 2023-10-30T13:26:55.606 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:26:55.819 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:55.824 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:57.607 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:26:57.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.760+0000 7f4be773e640 1 Processor -- start 2023-10-30T13:26:57.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.760+0000 7f4be773e640 1 -- start start 2023-10-30T13:26:57.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.760+0000 7f4be773e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be015d6f0 0x7f4be015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:57.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be773e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4be00cf160 0x7f4be00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:57.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be773e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be00cb800 con 0x7f4be01538a0 2023-10-30T13:26:57.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be773e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be00cb330 con 0x7f4be00cf160 2023-10-30T13:26:57.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be773e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00cad70 con 0x7f4be015d6f0 2023-10-30T13:26:57.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be4cd9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be015d6f0 0x7f4be015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:57.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be4cd9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be015d6f0 0x7f4be015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:34224/0 (socket says 172.21.15.148:34224) 2023-10-30T13:26:57.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be4cd9640 1 -- 172.21.15.148:0/3133026028 learned_addr learned my addr 172.21.15.148:0/3133026028 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:57.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be5cdb640 1 --2- 172.21.15.148:0/3133026028 >> v2:172.21.15.150:3300/0 conn(0x7f4be00cf160 0x7f4be00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:57.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4bd7fff640 1 -- 172.21.15.148:0/3133026028 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852711247 0 0) 0x7f4be00cb800 con 0x7f4be01538a0 2023-10-30T13:26:57.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4bd7fff640 1 -- 172.21.15.148:0/3133026028 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bcc003530 con 0x7f4be01538a0 2023-10-30T13:26:57.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be4cd9640 1 -- 172.21.15.148:0/3133026028 >> v2:172.21.15.150:3300/0 conn(0x7f4be00cf160 msgr2=0x7f4be00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:57.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be4cd9640 1 --2- 172.21.15.148:0/3133026028 >> v2:172.21.15.150:3300/0 conn(0x7f4be00cf160 0x7f4be00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:57.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be4cd9640 1 -- 172.21.15.148:0/3133026028 >> v1:172.21.15.150:6789/0 conn(0x7f4be01538a0 legacy=0x7f4be00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:57.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.764+0000 7f4be4cd9640 1 -- 172.21.15.148:0/3133026028 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be00cb050 con 0x7f4be015d6f0 2023-10-30T13:26:57.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be4cd9640 1 --2- 172.21.15.148:0/3133026028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be015d6f0 0x7f4be015a090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009870 tx=0x7f4bd001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae8e665f538f3659 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:57.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4bd7fff640 1 -- 172.21.15.148:0/3133026028 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00a8a30 con 0x7f4be015d6f0 2023-10-30T13:26:57.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4bd7fff640 1 -- 172.21.15.148:0/3133026028 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd00a8b90 con 0x7f4be015d6f0 2023-10-30T13:26:57.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4bd7fff640 1 -- 172.21.15.148:0/3133026028 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00b0440 con 0x7f4be015d6f0 2023-10-30T13:26:57.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 -- 172.21.15.148:0/3133026028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be015d6f0 msgr2=0x7f4be015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 --2- 172.21.15.148:0/3133026028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be015d6f0 0x7f4be015a090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009870 tx=0x7f4bd001ed30 comp rx=0 tx=0).stop 2023-10-30T13:26:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 -- 172.21.15.148:0/3133026028 shutdown_connections 2023-10-30T13:26:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 --2- 172.21.15.148:0/3133026028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be015d6f0 0x7f4be015a090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 --2- 172.21.15.148:0/3133026028 >> v2:172.21.15.150:3300/0 conn(0x7f4be00cf160 0x7f4be00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 -- 172.21.15.148:0/3133026028 >> 172.21.15.148:0/3133026028 conn(0x7f4be006bfe0 msgr2=0x7f4be006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:57.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 -- 172.21.15.148:0/3133026028 shutdown_connections 2023-10-30T13:26:57.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 -- 172.21.15.148:0/3133026028 wait complete. 2023-10-30T13:26:57.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 Processor -- start 2023-10-30T13:26:57.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 -- start start 2023-10-30T13:26:57.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4be00cf160 0x7f4be0159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:57.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be01538a0 0x7f4be01e3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be015d6f0 0x7f4be00ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be0157d10 con 0x7f4be00cf160 2023-10-30T13:26:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be01e3580 con 0x7f4be015d6f0 2023-10-30T13:26:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.768+0000 7f4be773e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be00c6dc0 con 0x7f4be01538a0 2023-10-30T13:26:57.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be54da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4be00cf160 0x7f4be0159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:57.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be54da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4be00cf160 0x7f4be0159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46510/0 (socket says 172.21.15.148:46510) 2023-10-30T13:26:57.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be54da640 1 -- 172.21.15.148:0/2288249550 learned_addr learned my addr 172.21.15.148:0/2288249550 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:26:57.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be4cd9640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be01538a0 0x7f4be01e3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:57.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be5cdb640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be015d6f0 0x7f4be00ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:57.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be54da640 1 -- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be01538a0 msgr2=0x7f4be01e3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:57.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be54da640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be01538a0 0x7f4be01e3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be54da640 1 -- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be015d6f0 msgr2=0x7f4be00ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be54da640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be015d6f0 0x7f4be00ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be54da640 1 -- 172.21.15.148:0/2288249550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd00094d0 con 0x7f4be00cf160 2023-10-30T13:26:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be54da640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4be00cf160 0x7f4be0159140 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00c4c0 tx=0x7f4bdc00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:57.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4bd5ffb640 1 -- 172.21.15.148:0/2288249550 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc0052d0 con 0x7f4be00cf160 2023-10-30T13:26:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4bd5ffb640 1 -- 172.21.15.148:0/2288249550 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bdc005430 con 0x7f4be00cf160 2023-10-30T13:26:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4bd5ffb640 1 -- 172.21.15.148:0/2288249550 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc012870 con 0x7f4be00cf160 2023-10-30T13:26:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be01e2fc0 con 0x7f4be00cf160 2023-10-30T13:26:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be01576c0 con 0x7f4be00cf160 2023-10-30T13:26:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be00b2a50 con 0x7f4be00cf160 2023-10-30T13:26:57.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.772+0000 7f4bd5ffb640 1 -- 172.21.15.148:0/2288249550 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4bdc0129d0 con 0x7f4be00cf160 2023-10-30T13:26:57.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.776+0000 7f4bd5ffb640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664a0 0x7f4bc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:57.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.776+0000 7f4bd5ffb640 1 -- 172.21.15.148:0/2288249550 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f4bdc083d30 con 0x7f4be00cf160 2023-10-30T13:26:57.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.776+0000 7f4be4cd9640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664a0 0x7f4bc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:57.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.776+0000 7f4be4cd9640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664a0 0x7f4bc4068960 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00b2000 tx=0x7f4bd00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:57.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.780+0000 7f4bd5ffb640 1 -- 172.21.15.148:0/2288249550 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4bdc050e80 con 0x7f4be00cf160 2023-10-30T13:26:57.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.960+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4be0157520 con 0x7f4be00cf160 2023-10-30T13:26:57.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.960+0000 7f4bd5ffb640 1 -- 172.21.15.148:0/2288249550 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f4be0157520 con 0x7f4be00cf160 2023-10-30T13:26:57.958 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:26:57.958 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:26:52.914120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:26:57.958 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 77 2023-10-30T13:26:57.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664a0 msgr2=0x7f4bc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:57.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664a0 0x7f4bc4068960 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00b2000 tx=0x7f4bd00aa070 comp rx=0 tx=0).stop 2023-10-30T13:26:57.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4be00cf160 msgr2=0x7f4be0159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:57.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4be00cf160 0x7f4be0159140 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00c4c0 tx=0x7f4bdc00c980 comp rx=0 tx=0).stop 2023-10-30T13:26:57.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 shutdown_connections 2023-10-30T13:26:57.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4bc40664a0 0x7f4bc4068960 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:57.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be015d6f0 0x7f4be00ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:57.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4be01538a0 0x7f4be01e3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:57.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 --2- 172.21.15.148:0/2288249550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4be00cf160 0x7f4be0159140 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:57.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 >> 172.21.15.148:0/2288249550 conn(0x7f4be006bfe0 msgr2=0x7f4be01510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:57.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 shutdown_connections 2023-10-30T13:26:57.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:26:57.964+0000 7f4be773e640 1 -- 172.21.15.148:0/2288249550 wait complete. 2023-10-30T13:26:57.979 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 77, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 77, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6099": { "addr": "172.21.15.148:6835/2733100982", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 2733100982, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 2733100982, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6099, "incarnation": 76, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:26:52.914120+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6099 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:26:59.980 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:00.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.132+0000 7f0488fc5640 1 Processor -- start 2023-10-30T13:27:00.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.132+0000 7f0488fc5640 1 -- start start 2023-10-30T13:27:00.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0488fc5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f048406f7a0 0x7f048406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:00.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0488fc5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0484071110 0x7f0484158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:00.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0488fc5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0484066cf0 con 0x7f04840c6bd0 2023-10-30T13:27:00.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0488fc5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04840700c0 con 0x7f0484071110 2023-10-30T13:27:00.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0488fc5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04840c70d0 con 0x7f048406f7a0 2023-10-30T13:27:00.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f047bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f04840c6bd0 0x7f0484069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:38974/0 (socket says 172.21.15.148:38974) 2023-10-30T13:27:00.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f047bfff640 1 -- 172.21.15.148:0/2788051505 learned_addr learned my addr 172.21.15.148:0/2788051505 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:27:00.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0482d9d640 1 --2- 172.21.15.148:0/2788051505 >> v2:172.21.15.150:3300/0 conn(0x7f0484071110 0x7f0484158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:00.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f048259c640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f048406f7a0 0x7f048406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:00.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039284821 0 0) 0x7f0484066cf0 con 0x7f04840c6bd0 2023-10-30T13:27:00.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0470003530 con 0x7f04840c6bd0 2023-10-30T13:27:00.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351024460 0 0) 0x7f0470003530 con 0x7f04840c6bd0 2023-10-30T13:27:00.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f048406f7a0 msgr2=0x7f048406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:00.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f048406f7a0 0x7f048406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 >> v2:172.21.15.150:3300/0 conn(0x7f0484071110 msgr2=0x7f0484158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:00.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 --2- 172.21.15.148:0/2788051505 >> v2:172.21.15.150:3300/0 conn(0x7f0484071110 0x7f0484158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04840c7390 con 0x7f04840c6bd0 2023-10-30T13:27:00.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f046c001680 con 0x7f04840c6bd0 2023-10-30T13:27:00.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f048259c640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f048406f7a0 0x7f048406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:27:00.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 >> v1:172.21.15.148:6789/0 conn(0x7f04840c6bd0 legacy=0x7f0484069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:00.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f048406f7a0 0x7f0470003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:00.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0470003ec0 0x7f04700047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:00.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0470004d10 0x7f0470005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:00.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04840700c0 con 0x7f0470004d10 2023-10-30T13:27:00.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04840c70d0 con 0x7f0470003ec0 2023-10-30T13:27:00.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047000b990 con 0x7f048406f7a0 2023-10-30T13:27:00.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0482d9d640 1 --2- 172.21.15.148:0/2788051505 >> v2:172.21.15.150:3300/0 conn(0x7f0484071110 0x7f0484158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:27:00.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0482d9d640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0470004d10 0x7f0470005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:00.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f048259c640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f048406f7a0 0x7f0470003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:00.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f047bfff640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0470003ec0 0x7f04700047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:00.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0482d9d640 1 -- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f048406f7a0 msgr2=0x7f0470003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:00.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0482d9d640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f048406f7a0 0x7f0470003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0482d9d640 1 -- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0470003ec0 msgr2=0x7f04700047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:00.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0482d9d640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0470003ec0 0x7f04700047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.136+0000 7f0482d9d640 1 -- 172.21.15.148:0/2788051505 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0474009420 con 0x7f0470004d10 2023-10-30T13:27:00.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0482d9d640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0470004d10 0x7f0470005190 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f047c00e380 tx=0x7f047c01ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b53d15b0a5ddc0d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:00.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c0a5e30 con 0x7f0470004d10 2023-10-30T13:27:00.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c00f040 con 0x7f0470004d10 2023-10-30T13:27:00.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0481d9b640 1 -- 172.21.15.148:0/2788051505 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c003ba0 con 0x7f0470004d10 2023-10-30T13:27:00.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0488fc5640 1 -- 172.21.15.148:0/2788051505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0470004d10 msgr2=0x7f0470005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:00.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0488fc5640 1 --2- 172.21.15.148:0/2788051505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0470004d10 0x7f0470005190 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f047c00e380 tx=0x7f047c01ce20 comp rx=0 tx=0).stop 2023-10-30T13:27:00.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0482d9d640 1 -- 172.21.15.148:0/2788051505 reap_dead start 2023-10-30T13:27:00.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0488fc5640 1 -- 172.21.15.148:0/2788051505 shutdown_connections 2023-10-30T13:27:00.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0488fc5640 1 -- 172.21.15.148:0/2788051505 >> 172.21.15.148:0/2788051505 conn(0x7f04840c7910 msgr2=0x7f04840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:00.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0488fc5640 1 -- 172.21.15.148:0/2788051505 shutdown_connections 2023-10-30T13:27:00.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0488fc5640 1 -- 172.21.15.148:0/2788051505 wait complete. 2023-10-30T13:27:00.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.140+0000 7f0488fc5640 1 Processor -- start 2023-10-30T13:27:00.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 -- start start 2023-10-30T13:27:00.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0470004d10 0x7f0484154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:00.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04840ca800 0x7f048414d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:00.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048414d970 0x7f04841523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:00.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04841ed6b0 con 0x7f048414d970 2023-10-30T13:27:00.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048414ddf0 con 0x7f0470004d10 2023-10-30T13:27:00.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04841ec850 con 0x7f04840ca800 2023-10-30T13:27:00.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0482d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048414d970 0x7f04841523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:00.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0482d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048414d970 0x7f04841523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46538/0 (socket says 172.21.15.148:46538) 2023-10-30T13:27:00.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0482d9d640 1 -- 172.21.15.148:0/4176546919 learned_addr learned my addr 172.21.15.148:0/4176546919 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:00.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f048259c640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0470004d10 0x7f0484154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:00.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f047bfff640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04840ca800 0x7f048414d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:00.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0482d9d640 1 -- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04840ca800 msgr2=0x7f048414d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:00.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0482d9d640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04840ca800 0x7f048414d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0482d9d640 1 -- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0470004d10 msgr2=0x7f0484154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:00.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0482d9d640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0470004d10 0x7f0484154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0482d9d640 1 -- 172.21.15.148:0/4176546919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04841ecb30 con 0x7f048414d970 2023-10-30T13:27:00.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0482d9d640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048414d970 0x7f04841523b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f047c0af040 tx=0x7f047c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:00.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f047affd640 1 -- 172.21.15.148:0/4176546919 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c003ba0 con 0x7f048414d970 2023-10-30T13:27:00.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f047affd640 1 -- 172.21.15.148:0/4176546919 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c003d00 con 0x7f048414d970 2023-10-30T13:27:00.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04841ece10 con 0x7f048414d970 2023-10-30T13:27:00.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f047affd640 1 -- 172.21.15.148:0/4176546919 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c004240 con 0x7f048414d970 2023-10-30T13:27:00.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04841ed3d0 con 0x7f048414d970 2023-10-30T13:27:00.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.144+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048404e170 con 0x7f048414d970 2023-10-30T13:27:00.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.148+0000 7f047affd640 1 -- 172.21.15.148:0/4176546919 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f047c0043a0 con 0x7f048414d970 2023-10-30T13:27:00.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.148+0000 7f047affd640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f04640664a0 0x7f0464068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:00.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.148+0000 7f047affd640 1 -- 172.21.15.148:0/4176546919 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f047c11f1d0 con 0x7f048414d970 2023-10-30T13:27:00.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.148+0000 7f048259c640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f04640664a0 0x7f0464068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:00.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.148+0000 7f048259c640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f04640664a0 0x7f0464068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0474009cb0 tx=0x7f0474008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:00.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.152+0000 7f047affd640 1 -- 172.21.15.148:0/4176546919 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f047c0ec300 con 0x7f048414d970 2023-10-30T13:27:00.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.328+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f048406aa00 con 0x7f048414d970 2023-10-30T13:27:00.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f047affd640 1 -- 172.21.15.148:0/4176546919 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f047c009dc0 con 0x7f048414d970 2023-10-30T13:27:00.328 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:00.328 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:26:58.781868+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:00.329 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 78 2023-10-30T13:27:00.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f04640664a0 msgr2=0x7f0464068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:00.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f04640664a0 0x7f0464068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0474009cb0 tx=0x7f0474008040 comp rx=0 tx=0).stop 2023-10-30T13:27:00.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048414d970 msgr2=0x7f04841523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:00.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048414d970 0x7f04841523b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f047c0af040 tx=0x7f047c009750 comp rx=0 tx=0).stop 2023-10-30T13:27:00.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 shutdown_connections 2023-10-30T13:27:00.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f04640664a0 0x7f0464068960 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048414d970 0x7f04841523b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04840ca800 0x7f048414d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 --2- 172.21.15.148:0/4176546919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0470004d10 0x7f0484154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:00.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 >> 172.21.15.148:0/4176546919 conn(0x7f04840c7910 msgr2=0x7f04840c5710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:00.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 shutdown_connections 2023-10-30T13:27:00.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:00.332+0000 7f0488fc5640 1 -- 172.21.15.148:0/4176546919 wait complete. 2023-10-30T13:27:00.349 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:27:02.349 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.500+0000 7f04172d6640 1 Processor -- start 2023-10-30T13:27:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.500+0000 7f04172d6640 1 -- start start 2023-10-30T13:27:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.500+0000 7f04172d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f04101580e0 0x7f0410070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.500+0000 7f04172d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f04100c6540 0x7f041015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.500+0000 7f04172d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0410069e90 0x7f041006a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.500+0000 7f04172d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04100c5810 con 0x7f04100c6540 2023-10-30T13:27:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.500+0000 7f04172d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04100c6990 con 0x7f04101580e0 2023-10-30T13:27:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.500+0000 7f04172d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04100c6bb0 con 0x7f0410069e90 2023-10-30T13:27:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0414871640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f04100c6540 0x7f041015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0414871640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f04100c6540 0x7f041015e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43130/0 (socket says 172.21.15.148:43130) 2023-10-30T13:27:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0414871640 1 -- 172.21.15.148:0/2236579663 learned_addr learned my addr 172.21.15.148:0/2236579663 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0415072640 1 --2- 172.21.15.148:0/2236579663 >> v2:172.21.15.150:3300/0 conn(0x7f04101580e0 0x7f0410070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0415873640 1 --2- 172.21.15.148:0/2236579663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0410069e90 0x7f041006a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0414871640 1 -- 172.21.15.148:0/2236579663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0410069e90 msgr2=0x7f041006a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0414871640 1 --2- 172.21.15.148:0/2236579663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0410069e90 0x7f041006a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0414871640 1 -- 172.21.15.148:0/2236579663 >> v2:172.21.15.150:3300/0 conn(0x7f04101580e0 msgr2=0x7f0410070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0414871640 1 --2- 172.21.15.148:0/2236579663 >> v2:172.21.15.150:3300/0 conn(0x7f04101580e0 0x7f0410070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0414871640 1 -- 172.21.15.148:0/2236579663 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04100c6e70 con 0x7f04100c6540 2023-10-30T13:27:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0415873640 1 --2- 172.21.15.148:0/2236579663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0410069e90 0x7f041006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:27:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0414871640 1 --2- 172.21.15.148:0/2236579663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04100c6540 0x7f041015e700 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f041006eea0 tx=0x7f040001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e363b718c18831f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0407fff640 1 -- 172.21.15.148:0/2236579663 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04000ac630 con 0x7f04100c6540 2023-10-30T13:27:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0407fff640 1 -- 172.21.15.148:0/2236579663 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04000ac790 con 0x7f04100c6540 2023-10-30T13:27:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.504+0000 7f0407fff640 1 -- 172.21.15.148:0/2236579663 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04000acb20 con 0x7f04100c6540 2023-10-30T13:27:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 -- 172.21.15.148:0/2236579663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04100c6540 msgr2=0x7f041015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 --2- 172.21.15.148:0/2236579663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04100c6540 0x7f041015e700 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f041006eea0 tx=0x7f040001c500 comp rx=0 tx=0).stop 2023-10-30T13:27:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 -- 172.21.15.148:0/2236579663 shutdown_connections 2023-10-30T13:27:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 --2- 172.21.15.148:0/2236579663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0410069e90 0x7f041006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 --2- 172.21.15.148:0/2236579663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f04100c6540 0x7f041015e700 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 --2- 172.21.15.148:0/2236579663 >> v2:172.21.15.150:3300/0 conn(0x7f04101580e0 0x7f0410070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 -- 172.21.15.148:0/2236579663 >> 172.21.15.148:0/2236579663 conn(0x7f04100c7910 msgr2=0x7f04100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 -- 172.21.15.148:0/2236579663 shutdown_connections 2023-10-30T13:27:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 -- 172.21.15.148:0/2236579663 wait complete. 2023-10-30T13:27:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 Processor -- start 2023-10-30T13:27:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 -- start start 2023-10-30T13:27:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0410069e90 0x7f0410151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04100c6540 0x7f04100c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04101580e0 0x7f041014ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0410152a40 con 0x7f0410069e90 2023-10-30T13:27:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0410150910 con 0x7f04100c6540 2023-10-30T13:27:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f04172d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04101526a0 con 0x7f04101580e0 2023-10-30T13:27:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0415072640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0410069e90 0x7f0410151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0415072640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0410069e90 0x7f0410151620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43132/0 (socket says 172.21.15.148:43132) 2023-10-30T13:27:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0415072640 1 -- 172.21.15.148:0/3455640215 learned_addr learned my addr 172.21.15.148:0/3455640215 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0414871640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04100c6540 0x7f04100c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0415873640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04101580e0 0x7f041014ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:02.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0415072640 1 -- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04101580e0 msgr2=0x7f041014ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0415072640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04101580e0 0x7f041014ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0415072640 1 -- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04100c6540 msgr2=0x7f04100c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0415072640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04100c6540 0x7f04100c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.508+0000 7f0415072640 1 -- 172.21.15.148:0/3455640215 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04000094d0 con 0x7f0410069e90 2023-10-30T13:27:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f0415072640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0410069e90 0x7f0410151620 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f040c00e8a0 tx=0x7f040c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:02.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f0405ffb640 1 -- 172.21.15.148:0/3455640215 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c009510 con 0x7f0410069e90 2023-10-30T13:27:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f0405ffb640 1 -- 172.21.15.148:0/3455640215 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c00c940 con 0x7f0410069e90 2023-10-30T13:27:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f0405ffb640 1 -- 172.21.15.148:0/3455640215 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c00ccb0 con 0x7f0410069e90 2023-10-30T13:27:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0410152ee0 con 0x7f0410069e90 2023-10-30T13:27:02.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04101f7610 con 0x7f0410069e90 2023-10-30T13:27:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0410152cb0 con 0x7f0410069e90 2023-10-30T13:27:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f0405ffb640 1 -- 172.21.15.148:0/3455640215 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f040c013470 con 0x7f0410069e90 2023-10-30T13:27:02.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f0405ffb640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03f40664d0 0x7f03f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f0405ffb640 1 -- 172.21.15.148:0/3455640215 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f040c0842c0 con 0x7f0410069e90 2023-10-30T13:27:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.512+0000 7f0414871640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03f40664d0 0x7f03f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.516+0000 7f0414871640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03f40664d0 0x7f03f4068990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f04000ae000 tx=0x7f0400008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:02.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.516+0000 7f0405ffb640 1 -- 172.21.15.148:0/3455640215 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f040c0513f0 con 0x7f0410069e90 2023-10-30T13:27:02.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.696+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f041004e170 con 0x7f0410069e90 2023-10-30T13:27:02.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f0405ffb640 1 -- 172.21.15.148:0/3455640215 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f040c054a10 con 0x7f0410069e90 2023-10-30T13:27:02.695 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:02.696 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:26:58.781868+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:02.696 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 78 2023-10-30T13:27:02.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03f40664d0 msgr2=0x7f03f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:02.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03f40664d0 0x7f03f4068990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f04000ae000 tx=0x7f0400008040 comp rx=0 tx=0).stop 2023-10-30T13:27:02.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0410069e90 msgr2=0x7f0410151620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:02.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0410069e90 0x7f0410151620 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f040c00e8a0 tx=0x7f040c00eca0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 shutdown_connections 2023-10-30T13:27:02.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03f40664d0 0x7f03f4068990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f04101580e0 0x7f041014ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04100c6540 0x7f04100c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 --2- 172.21.15.148:0/3455640215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0410069e90 0x7f0410151620 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:02.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 >> 172.21.15.148:0/3455640215 conn(0x7f04100c7910 msgr2=0x7f0410153500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:02.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 shutdown_connections 2023-10-30T13:27:02.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:02.700+0000 7f04172d6640 1 -- 172.21.15.148:0/3455640215 wait complete. 2023-10-30T13:27:02.717 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:27:04.718 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:04.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c35417640 1 Processor -- start 2023-10-30T13:27:04.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c35417640 1 -- start start 2023-10-30T13:27:04.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c35417640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7c301538a0 0x7f7c3015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c35417640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c300cb800 con 0x7f7c300ca0e0 2023-10-30T13:27:04.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c35417640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3015dc30 con 0x7f7c300cf160 2023-10-30T13:27:04.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c35417640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c300cb330 con 0x7f7c301538a0 2023-10-30T13:27:04.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c27fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7c300ca0e0 0x7f7c3015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34568/0 (socket says 172.21.15.148:34568) 2023-10-30T13:27:04.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c27fff640 1 -- 172.21.15.148:0/2757428047 learned_addr learned my addr 172.21.15.148:0/2757428047 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:27:04.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2effd640 1 --2- 172.21.15.148:0/2757428047 >> v2:172.21.15.150:3300/0 conn(0x7f7c301538a0 0x7f7c3015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850947056 0 0) 0x7f7c300cb800 con 0x7f7c300ca0e0 2023-10-30T13:27:04.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c1c003530 con 0x7f7c300ca0e0 2023-10-30T13:27:04.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733063483 0 0) 0x7f7c280014a0 con 0x7f7c300cf160 2023-10-30T13:27:04.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c300cb800 con 0x7f7c300cf160 2023-10-30T13:27:04.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2170501610 0 0) 0x7f7c1c003530 con 0x7f7c300ca0e0 2023-10-30T13:27:04.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 >> v2:172.21.15.150:3300/0 conn(0x7f7c301538a0 msgr2=0x7f7c3015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:04.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 --2- 172.21.15.148:0/2757428047 >> v2:172.21.15.150:3300/0 conn(0x7f7c301538a0 0x7f7c3015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 >> v1:172.21.15.150:6789/0 conn(0x7f7c300cf160 legacy=0x7f7c30164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c300cad70 con 0x7f7c300ca0e0 2023-10-30T13:27:04.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f7c18001680 con 0x7f7c300ca0e0 2023-10-30T13:27:04.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 >> v1:172.21.15.148:6789/0 conn(0x7f7c300ca0e0 legacy=0x7f7c3015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c1c003b10 0x7f7c1c003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c1c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c1c005190 0x7f7c1c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.868+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c300cb330 con 0x7f7c1c0047d0 2023-10-30T13:27:04.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c00bf80 con 0x7f7c1c005190 2023-10-30T13:27:04.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c00c140 con 0x7f7c1c003b10 2023-10-30T13:27:04.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c27fff640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c1c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c2f7fe640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c1c005190 0x7f7c1c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c2effd640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c1c003b10 0x7f7c1c003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c27fff640 1 -- 172.21.15.148:0/2757428047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c1c003b10 msgr2=0x7f7c1c003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:04.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c27fff640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c1c003b10 0x7f7c1c003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c27fff640 1 -- 172.21.15.148:0/2757428047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c1c005190 msgr2=0x7f7c1c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:04.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c27fff640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c1c005190 0x7f7c1c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c27fff640 1 -- 172.21.15.148:0/2757428047 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c003530 con 0x7f7c1c0047d0 2023-10-30T13:27:04.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c2f7fe640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c1c005190 0x7f7c1c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:27:04.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c2f7fe640 1 -- 172.21.15.148:0/2757428047 reap_dead start 2023-10-30T13:27:04.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c27fff640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c1c004c50 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c160 tx=0x7f7c1801f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0a3aa72cfa1f50 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:04.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c180a8d00 con 0x7f7c1c0047d0 2023-10-30T13:27:04.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c180a8e60 con 0x7f7c1c0047d0 2023-10-30T13:27:04.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.872+0000 7f7c2e7fc640 1 -- 172.21.15.148:0/2757428047 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c18005ae0 con 0x7f7c1c0047d0 2023-10-30T13:27:04.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 -- 172.21.15.148:0/2757428047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 msgr2=0x7f7c1c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c1c004c50 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c160 tx=0x7f7c1801f2d0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 -- 172.21.15.148:0/2757428047 shutdown_connections 2023-10-30T13:27:04.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 --2- 172.21.15.148:0/2757428047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c1c004c50 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 -- 172.21.15.148:0/2757428047 >> 172.21.15.148:0/2757428047 conn(0x7f7c3006bfe0 msgr2=0x7f7c3006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:04.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 -- 172.21.15.148:0/2757428047 shutdown_connections 2023-10-30T13:27:04.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 -- 172.21.15.148:0/2757428047 wait complete. 2023-10-30T13:27:04.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 Processor -- start 2023-10-30T13:27:04.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 -- start start 2023-10-30T13:27:04.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c30158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c30148720 0x7f7c30149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c30145170 0x7f7c301455f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c30145b30 con 0x7f7c1c0047d0 2023-10-30T13:27:04.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c301440f0 con 0x7f7c30148720 2023-10-30T13:27:04.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c35417640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c301443b0 con 0x7f7c30145170 2023-10-30T13:27:04.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c30158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c30158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43142/0 (socket says 172.21.15.148:43142) 2023-10-30T13:27:04.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2effd640 1 -- 172.21.15.148:0/2897866610 learned_addr learned my addr 172.21.15.148:0/2897866610 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:04.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2f7fe640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c30145170 0x7f7c301455f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c27fff640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c30148720 0x7f7c30149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2effd640 1 -- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c30145170 msgr2=0x7f7c301455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2effd640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c30145170 0x7f7c301455f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2effd640 1 -- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c30148720 msgr2=0x7f7c30149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2effd640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c30148720 0x7f7c30149610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2effd640 1 -- 172.21.15.148:0/2897866610 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c30144690 con 0x7f7c1c0047d0 2023-10-30T13:27:04.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c27fff640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c30148720 0x7f7c30149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:27:04.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.876+0000 7f7c2effd640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c30158810 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f7c20011500 tx=0x7f7c200119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:04.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c277fe640 1 -- 172.21.15.148:0/2897866610 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20015870 con 0x7f7c1c0047d0 2023-10-30T13:27:04.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c277fe640 1 -- 172.21.15.148:0/2897866610 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c200159d0 con 0x7f7c1c0047d0 2023-10-30T13:27:04.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c30144970 con 0x7f7c1c0047d0 2023-10-30T13:27:04.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c277fe640 1 -- 172.21.15.148:0/2897866610 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20015d40 con 0x7f7c1c0047d0 2023-10-30T13:27:04.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c301593f0 con 0x7f7c1c0047d0 2023-10-30T13:27:04.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3004e170 con 0x7f7c1c0047d0 2023-10-30T13:27:04.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c277fe640 1 -- 172.21.15.148:0/2897866610 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7c20013400 con 0x7f7c1c0047d0 2023-10-30T13:27:04.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c277fe640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7c100664a0 0x7f7c10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c277fe640 1 -- 172.21.15.148:0/2897866610 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f7c20084650 con 0x7f7c1c0047d0 2023-10-30T13:27:04.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.880+0000 7f7c27fff640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7c100664a0 0x7f7c10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.884+0000 7f7c27fff640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7c100664a0 0x7f7c10068960 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c160 tx=0x7f7c18001990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:04.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:04.884+0000 7f7c277fe640 1 -- 172.21.15.148:0/2897866610 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7c20051670 con 0x7f7c1c0047d0 2023-10-30T13:27:05.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.064+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c300b1dd0 con 0x7f7c1c0047d0 2023-10-30T13:27:05.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.064+0000 7f7c277fe640 1 -- 172.21.15.148:0/2897866610 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f7c20054c90 con 0x7f7c1c0047d0 2023-10-30T13:27:05.063 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:05.063 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:03.955656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:05.064 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 79 2023-10-30T13:27:05.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7c100664a0 msgr2=0x7f7c10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7c100664a0 0x7f7c10068960 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c160 tx=0x7f7c18001990 comp rx=0 tx=0).stop 2023-10-30T13:27:05.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 msgr2=0x7f7c30158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c30158810 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f7c20011500 tx=0x7f7c200119c0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 shutdown_connections 2023-10-30T13:27:05.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7c100664a0 0x7f7c10068960 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7c30145170 0x7f7c301455f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c30148720 0x7f7c30149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 --2- 172.21.15.148:0/2897866610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c1c0047d0 0x7f7c30158810 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 >> 172.21.15.148:0/2897866610 conn(0x7f7c3006bfe0 msgr2=0x7f7c30142980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:05.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 shutdown_connections 2023-10-30T13:27:05.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:05.068+0000 7f7c35417640 1 -- 172.21.15.148:0/2897866610 wait complete. 2023-10-30T13:27:05.084 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:27:07.085 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:07.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.236+0000 7fc4190c9640 1 Processor -- start 2023-10-30T13:27:07.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.236+0000 7fc4190c9640 1 -- start start 2023-10-30T13:27:07.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.236+0000 7fc4190c9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc4140c6b80 0x7fc4140bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.236+0000 7fc4190c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc41413f4a0 0x7fc4140c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.236+0000 7fc4190c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4140c0a50 con 0x7fc4140c6f60 2023-10-30T13:27:07.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.236+0000 7fc4190c9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41406f6e0 con 0x7fc4140c6b80 2023-10-30T13:27:07.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.236+0000 7fc4190c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41406f9c0 con 0x7fc41413f4a0 2023-10-30T13:27:07.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc412d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc4140c6b80 0x7fc4140bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc412d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc4140c6b80 0x7fc4140bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43156/0 (socket says 172.21.15.148:43156) 2023-10-30T13:27:07.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc412d9d640 1 -- 172.21.15.148:0/2444223602 learned_addr learned my addr 172.21.15.148:0/2444223602 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:07.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc41359e640 1 --2- 172.21.15.148:0/2444223602 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc41413f4a0 0x7fc4140c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc412d9d640 1 -- 172.21.15.148:0/2444223602 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc41413f4a0 msgr2=0x7fc4140c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:07.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc412d9d640 1 --2- 172.21.15.148:0/2444223602 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc41413f4a0 0x7fc4140c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc412d9d640 1 -- 172.21.15.148:0/2444223602 >> v1:172.21.15.150:6789/0 conn(0x7fc4140c6f60 legacy=0x7fc41413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc412d9d640 1 -- 172.21.15.148:0/2444223602 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41406fca0 con 0x7fc4140c6b80 2023-10-30T13:27:07.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc412d9d640 1 --2- 172.21.15.148:0/2444223602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6b80 0x7fc4140bfec0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc404009870 tx=0x7fc40401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f266a9ce9b0141 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:07.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc41259c640 1 -- 172.21.15.148:0/2444223602 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4040ab650 con 0x7fc4140c6b80 2023-10-30T13:27:07.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc41259c640 1 -- 172.21.15.148:0/2444223602 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4040ab7b0 con 0x7fc4140c6b80 2023-10-30T13:27:07.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.240+0000 7fc41259c640 1 -- 172.21.15.148:0/2444223602 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4040abb40 con 0x7fc4140c6b80 2023-10-30T13:27:07.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 -- 172.21.15.148:0/2444223602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6b80 msgr2=0x7fc4140bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 --2- 172.21.15.148:0/2444223602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6b80 0x7fc4140bfec0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc404009870 tx=0x7fc40401c540 comp rx=0 tx=0).stop 2023-10-30T13:27:07.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 -- 172.21.15.148:0/2444223602 shutdown_connections 2023-10-30T13:27:07.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 --2- 172.21.15.148:0/2444223602 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc41413f4a0 0x7fc4140c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 --2- 172.21.15.148:0/2444223602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6b80 0x7fc4140bfec0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 -- 172.21.15.148:0/2444223602 >> 172.21.15.148:0/2444223602 conn(0x7fc4140c7910 msgr2=0x7fc4140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:07.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 -- 172.21.15.148:0/2444223602 shutdown_connections 2023-10-30T13:27:07.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 -- 172.21.15.148:0/2444223602 wait complete. 2023-10-30T13:27:07.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 Processor -- start 2023-10-30T13:27:07.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 -- start start 2023-10-30T13:27:07.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4140c6b80 0x7fc414158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6f60 0x7fc414154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc41413f4a0 0x7fc414154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc414133410 con 0x7fc4140c6f60 2023-10-30T13:27:07.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4141563f0 con 0x7fc41413f4a0 2023-10-30T13:27:07.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc4190c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4141f3480 con 0x7fc4140c6b80 2023-10-30T13:27:07.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc40a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6f60 0x7fc414154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc40a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6f60 0x7fc414154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43162/0 (socket says 172.21.15.148:43162) 2023-10-30T13:27:07.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc40a59c640 1 -- 172.21.15.148:0/3805948182 learned_addr learned my addr 172.21.15.148:0/3805948182 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:07.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc412d9d640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4140c6b80 0x7fc414158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.244+0000 7fc41359e640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc41413f4a0 0x7fc414154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40a59c640 1 -- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4140c6b80 msgr2=0x7fc414158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40a59c640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4140c6b80 0x7fc414158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40a59c640 1 -- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc41413f4a0 msgr2=0x7fc414154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40a59c640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc41413f4a0 0x7fc414154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40a59c640 1 -- 172.21.15.148:0/3805948182 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4040094d0 con 0x7fc4140c6f60 2023-10-30T13:27:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc41359e640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc41413f4a0 0x7fc414154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:27:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40a59c640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6f60 0x7fc414154290 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc00c500 tx=0x7fc3fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40bfff640 1 -- 172.21.15.148:0/3805948182 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc005220 con 0x7fc4140c6f60 2023-10-30T13:27:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40bfff640 1 -- 172.21.15.148:0/3805948182 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc005380 con 0x7fc4140c6f60 2023-10-30T13:27:07.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4141f3760 con 0x7fc4140c6f60 2023-10-30T13:27:07.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40bfff640 1 -- 172.21.15.148:0/3805948182 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc007270 con 0x7fc4140c6f60 2023-10-30T13:27:07.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4141f3d20 con 0x7fc4140c6f60 2023-10-30T13:27:07.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41404e170 con 0x7fc4140c6f60 2023-10-30T13:27:07.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40bfff640 1 -- 172.21.15.148:0/3805948182 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3fc0073d0 con 0x7fc4140c6f60 2023-10-30T13:27:07.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40bfff640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3f40664a0 0x7fc3f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.248+0000 7fc40bfff640 1 -- 172.21.15.148:0/3805948182 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fc3fc083530 con 0x7fc4140c6f60 2023-10-30T13:27:07.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.252+0000 7fc412d9d640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3f40664a0 0x7fc3f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.252+0000 7fc412d9d640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3f40664a0 0x7fc3f4068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc4040af000 tx=0x7fc4040b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:07.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.252+0000 7fc40bfff640 1 -- 172.21.15.148:0/3805948182 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3fc050690 con 0x7fc4140c6f60 2023-10-30T13:27:07.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4141595e0 con 0x7fc4140c6f60 2023-10-30T13:27:07.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc40bfff640 1 -- 172.21.15.148:0/3805948182 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5723 (secure 0 0 0) 0x7fc3fc053cb0 con 0x7fc4140c6f60 2023-10-30T13:27:07.432 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:07.432 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:03.955656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:07.433 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 79 2023-10-30T13:27:07.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3f40664a0 msgr2=0x7fc3f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3f40664a0 0x7fc3f4068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc4040af000 tx=0x7fc4040b6040 comp rx=0 tx=0).stop 2023-10-30T13:27:07.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6f60 msgr2=0x7fc414154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6f60 0x7fc414154290 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc00c500 tx=0x7fc3fc00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 shutdown_connections 2023-10-30T13:27:07.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3f40664a0 0x7fc3f4068960 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc4140c6f60 0x7fc414154290 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc41413f4a0 0x7fc414154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 --2- 172.21.15.148:0/3805948182 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc4140c6b80 0x7fc414158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 >> 172.21.15.148:0/3805948182 conn(0x7fc4140c7910 msgr2=0x7fc4140bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:07.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 shutdown_connections 2023-10-30T13:27:07.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:07.436+0000 7fc4190c9640 1 -- 172.21.15.148:0/3805948182 wait complete. 2023-10-30T13:27:07.453 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:27:09.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:09.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f819c26b640 1 Processor -- start 2023-10-30T13:27:09.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f819c26b640 1 -- start start 2023-10-30T13:27:09.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f819c26b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81941538a0 0x7f819415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:09.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f819c26b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81940cf160 0x7f81940cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:09.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f819c26b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81940cb800 con 0x7f81940ca0e0 2023-10-30T13:27:09.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f819c26b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819415dc30 con 0x7f81940cf160 2023-10-30T13:27:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f819c26b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81940cb330 con 0x7f81941538a0 2023-10-30T13:27:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199806640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f81940ca0e0 0x7f819415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34574/0 (socket says 172.21.15.148:34574) 2023-10-30T13:27:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199806640 1 -- 172.21.15.148:0/1629844236 learned_addr learned my addr 172.21.15.148:0/1629844236 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:27:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f819a007640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81941538a0 0x7f819415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:09.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f819a808640 1 --2- 172.21.15.148:0/1629844236 >> v2:172.21.15.150:3300/0 conn(0x7f81940cf160 0x7f81940cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:09.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219728718 0 0) 0x7f81940cb800 con 0x7f81940ca0e0 2023-10-30T13:27:09.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8180003530 con 0x7f81940ca0e0 2023-10-30T13:27:09.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3695690693 0 0) 0x7f8180003530 con 0x7f81940ca0e0 2023-10-30T13:27:09.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81941538a0 msgr2=0x7f819415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:09.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81941538a0 0x7f819415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 >> v2:172.21.15.150:3300/0 conn(0x7f81940cf160 msgr2=0x7f81940cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:09.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 --2- 172.21.15.148:0/1629844236 >> v2:172.21.15.150:3300/0 conn(0x7f81940cf160 0x7f81940cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81940cad70 con 0x7f81940ca0e0 2023-10-30T13:27:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f8184001680 con 0x7f81940ca0e0 2023-10-30T13:27:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 >> v1:172.21.15.148:6789/0 conn(0x7f81940ca0e0 legacy=0x7f819415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8180003980 0x7f8180003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81800047d0 0x7f8180004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81941538a0 0x7f8180005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:09.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819415dc30 con 0x7f8180003980 2023-10-30T13:27:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81940cb330 con 0x7f81800047d0 2023-10-30T13:27:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.608+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818000b990 con 0x7f81941538a0 2023-10-30T13:27:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f819a007640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8180003980 0x7f8180003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f8199806640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81800047d0 0x7f8180004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:09.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f819a808640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81941538a0 0x7f8180005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f819a007640 1 -- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81941538a0 msgr2=0x7f8180005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f819a007640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81941538a0 0x7f8180005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f819a007640 1 -- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81800047d0 msgr2=0x7f8180004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f819a007640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81800047d0 0x7f8180004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f819a007640 1 -- 172.21.15.148:0/1629844236 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8180003530 con 0x7f8180003980 2023-10-30T13:27:09.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f819a007640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8180003980 0x7f8180003da0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8190009ae0 tx=0x7f819001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5853c8e4e806a265 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:09.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81900a6480 con 0x7f8180003980 2023-10-30T13:27:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81900a65e0 con 0x7f8180003980 2023-10-30T13:27:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.612+0000 7f8199005640 1 -- 172.21.15.148:0/1629844236 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81900a6990 con 0x7f8180003980 2023-10-30T13:27:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 -- 172.21.15.148:0/1629844236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8180003980 msgr2=0x7f8180003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 --2- 172.21.15.148:0/1629844236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8180003980 0x7f8180003da0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8190009ae0 tx=0x7f819001ca70 comp rx=0 tx=0).stop 2023-10-30T13:27:09.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 -- 172.21.15.148:0/1629844236 reap_dead start 2023-10-30T13:27:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 -- 172.21.15.148:0/1629844236 shutdown_connections 2023-10-30T13:27:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 -- 172.21.15.148:0/1629844236 >> 172.21.15.148:0/1629844236 conn(0x7f819406bfe0 msgr2=0x7f819406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 -- 172.21.15.148:0/1629844236 shutdown_connections 2023-10-30T13:27:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 -- 172.21.15.148:0/1629844236 wait complete. 2023-10-30T13:27:09.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 Processor -- start 2023-10-30T13:27:09.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 -- start start 2023-10-30T13:27:09.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8180003980 0x7f8194147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:09.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8194156ec0 0x7f81941573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:09.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81940cebf0 0x7f819415cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:09.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81941f01d0 con 0x7f81940cebf0 2023-10-30T13:27:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81940b3250 con 0x7f8180003980 2023-10-30T13:27:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819c26b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81940cd9d0 con 0x7f8194156ec0 2023-10-30T13:27:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81940cebf0 0x7f819415cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81940cebf0 0x7f819415cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43174/0 (socket says 172.21.15.148:43174) 2023-10-30T13:27:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 -- 172.21.15.148:0/1104761263 learned_addr learned my addr 172.21.15.148:0/1104761263 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:09.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f8199806640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8194156ec0 0x7f81941573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a007640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8180003980 0x7f8194147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 -- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8194156ec0 msgr2=0x7f81941573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8194156ec0 0x7f81941573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 -- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8180003980 msgr2=0x7f8194147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:09.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8180003980 0x7f8194147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 -- 172.21.15.148:0/1104761263 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81941f06c0 con 0x7f81940cebf0 2023-10-30T13:27:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.616+0000 7f819a808640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81940cebf0 0x7f819415cf50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f819406bd30 tx=0x7f818c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f818affd640 1 -- 172.21.15.148:0/1104761263 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c00e070 con 0x7f81940cebf0 2023-10-30T13:27:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f818affd640 1 -- 172.21.15.148:0/1104761263 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c011e40 con 0x7f81940cebf0 2023-10-30T13:27:09.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f818affd640 1 -- 172.21.15.148:0/1104761263 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c0039a0 con 0x7f81940cebf0 2023-10-30T13:27:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81940cd0e0 con 0x7f81940cebf0 2023-10-30T13:27:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81940cd380 con 0x7f81940cebf0 2023-10-30T13:27:09.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81940cdb80 con 0x7f81940cebf0 2023-10-30T13:27:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f818affd640 1 -- 172.21.15.148:0/1104761263 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f818c003b00 con 0x7f81940cebf0 2023-10-30T13:27:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f818affd640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81780664d0 0x7f8178068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:09.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f818affd640 1 -- 172.21.15.148:0/1104761263 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f818c084900 con 0x7f81940cebf0 2023-10-30T13:27:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.620+0000 7f819a007640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81780664d0 0x7f8178068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.624+0000 7f819a007640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81780664d0 0x7f8178068990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f81900ad000 tx=0x7f81900b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:09.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.624+0000 7f818affd640 1 -- 172.21.15.148:0/1104761263 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f81940cdb80 con 0x7f81940cebf0 2023-10-30T13:27:09.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81940cdb40 con 0x7f81940cebf0 2023-10-30T13:27:09.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f818affd640 1 -- 172.21.15.148:0/1104761263 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f818c055560 con 0x7f81940cebf0 2023-10-30T13:27:09.804 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:09.805 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:03.955656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:09.805 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 79 2023-10-30T13:27:09.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81780664d0 msgr2=0x7f8178068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:09.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81780664d0 0x7f8178068990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f81900ad000 tx=0x7f81900b2040 comp rx=0 tx=0).stop 2023-10-30T13:27:09.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81940cebf0 msgr2=0x7f819415cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:09.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81940cebf0 0x7f819415cf50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f819406bd30 tx=0x7f818c0115a0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 shutdown_connections 2023-10-30T13:27:09.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f81780664d0 0x7f8178068990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f81940cebf0 0x7f819415cf50 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8194156ec0 0x7f81941573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 --2- 172.21.15.148:0/1104761263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8180003980 0x7f8194147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 >> 172.21.15.148:0/1104761263 conn(0x7f819406bfe0 msgr2=0x7f8194152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:09.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 shutdown_connections 2023-10-30T13:27:09.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:09.808+0000 7f819c26b640 1 -- 172.21.15.148:0/1104761263 wait complete. 2023-10-30T13:27:09.827 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6099": { "addr": "172.21.15.148:6835/2733100982", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 2733100982, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 2733100982, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6099, "incarnation": 76, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:clientreplay", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:27:03.955656+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6099 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:27:11.828 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:11.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd78186b640 1 Processor -- start 2023-10-30T13:27:11.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd78186b640 1 -- start start 2023-10-30T13:27:11.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd78186b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd77c0cf160 0x7fd77c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:11.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd78186b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c0cb800 con 0x7fd77c15d6f0 2023-10-30T13:27:11.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd78186b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77c0cb330 con 0x7fd77c1538a0 2023-10-30T13:27:11.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd78186b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77c0cad70 con 0x7fd77c0cf160 2023-10-30T13:27:11.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd77b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd77c0cf160 0x7fd77c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:11.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd77b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd77c0cf160 0x7fd77c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57532/0 (socket says 172.21.15.148:57532) 2023-10-30T13:27:11.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd77b7fe640 1 -- 172.21.15.148:0/2192690563 learned_addr learned my addr 172.21.15.148:0/2192690563 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:11.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd77b7fe640 1 -- 172.21.15.148:0/2192690563 >> v1:172.21.15.150:6789/0 conn(0x7fd77c1538a0 legacy=0x7fd77c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:11.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd77b7fe640 1 -- 172.21.15.148:0/2192690563 >> v1:172.21.15.148:6789/0 conn(0x7fd77c15d6f0 legacy=0x7fd77c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:11.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd77b7fe640 1 -- 172.21.15.148:0/2192690563 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77c0cb050 con 0x7fd77c0cf160 2023-10-30T13:27:11.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.980+0000 7fd77a7fc640 1 -- 172.21.15.148:0/2192690563 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066399248 0 0) 0x7fd77c0cb800 con 0x7fd77c15d6f0 2023-10-30T13:27:11.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd77b7fe640 1 --2- 172.21.15.148:0/2192690563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c0cf160 0x7fd77c0cf590 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd774009e50 tx=0x7fd77401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cfdeaf430f533 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:11.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd77a7fc640 1 -- 172.21.15.148:0/2192690563 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740af400 con 0x7fd77c0cf160 2023-10-30T13:27:11.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd77a7fc640 1 -- 172.21.15.148:0/2192690563 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7740ab050 con 0x7fd77c0cf160 2023-10-30T13:27:11.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd77a7fc640 1 -- 172.21.15.148:0/2192690563 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740af780 con 0x7fd77c0cf160 2023-10-30T13:27:11.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd78186b640 1 -- 172.21.15.148:0/2192690563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c0cf160 msgr2=0x7fd77c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:11.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd78186b640 1 --2- 172.21.15.148:0/2192690563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c0cf160 0x7fd77c0cf590 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd774009e50 tx=0x7fd77401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:27:11.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd78186b640 1 -- 172.21.15.148:0/2192690563 shutdown_connections 2023-10-30T13:27:11.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd78186b640 1 --2- 172.21.15.148:0/2192690563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c0cf160 0x7fd77c0cf590 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:11.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd78186b640 1 -- 172.21.15.148:0/2192690563 >> 172.21.15.148:0/2192690563 conn(0x7fd77c06bfe0 msgr2=0x7fd77c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:11.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd78186b640 1 -- 172.21.15.148:0/2192690563 shutdown_connections 2023-10-30T13:27:11.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.984+0000 7fd78186b640 1 -- 172.21.15.148:0/2192690563 wait complete. 2023-10-30T13:27:11.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 Processor -- start 2023-10-30T13:27:11.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 -- start start 2023-10-30T13:27:11.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77c0cf160 0x7fd77c149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:11.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd77c1538a0 0x7fd77c0c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:11.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c15d6f0 0x7fd77c1f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:11.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c14dce0 con 0x7fd77c15d6f0 2023-10-30T13:27:11.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c14e070 con 0x7fd77c0cf160 2023-10-30T13:27:11.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c0c6ad0 con 0x7fd77c1538a0 2023-10-30T13:27:11.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c15d6f0 0x7fd77c1f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:11.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c15d6f0 0x7fd77c1f2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57534/0 (socket says 172.21.15.148:57534) 2023-10-30T13:27:11.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77b7fe640 1 -- 172.21.15.148:0/2137914409 learned_addr learned my addr 172.21.15.148:0/2137914409 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:11.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77affd640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77c0cf160 0x7fd77c149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:11.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd7727fc640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd77c1538a0 0x7fd77c0c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:11.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77b7fe640 1 -- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd77c1538a0 msgr2=0x7fd77c0c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:11.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77b7fe640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd77c1538a0 0x7fd77c0c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:11.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77b7fe640 1 -- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77c0cf160 msgr2=0x7fd77c149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:11.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77b7fe640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77c0cf160 0x7fd77c149160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:11.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77b7fe640 1 -- 172.21.15.148:0/2137914409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd774009b30 con 0x7fd77c15d6f0 2023-10-30T13:27:11.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd7727fc640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd77c1538a0 0x7fd77c0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:27:11.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd77b7fe640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c15d6f0 0x7fd77c1f2420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd77401ce20 tx=0x7fd77401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:11.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd778ff9640 1 -- 172.21.15.148:0/2137914409 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7740afd90 con 0x7fd77c15d6f0 2023-10-30T13:27:11.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd778ff9640 1 -- 172.21.15.148:0/2137914409 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd774009510 con 0x7fd77c15d6f0 2023-10-30T13:27:11.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77c0c6f60 con 0x7fd77c15d6f0 2023-10-30T13:27:11.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd778ff9640 1 -- 172.21.15.148:0/2137914409 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd774003be0 con 0x7fd77c15d6f0 2023-10-30T13:27:11.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.988+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c0c67a0 con 0x7fd77c15d6f0 2023-10-30T13:27:11.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.992+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77c04e170 con 0x7fd77c15d6f0 2023-10-30T13:27:11.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.992+0000 7fd778ff9640 1 -- 172.21.15.148:0/2137914409 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd7740b6070 con 0x7fd77c15d6f0 2023-10-30T13:27:11.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.992+0000 7fd778ff9640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd75c0664a0 0x7fd75c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:11.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.992+0000 7fd778ff9640 1 -- 172.21.15.148:0/2137914409 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fd7741220f0 con 0x7fd77c15d6f0 2023-10-30T13:27:11.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.992+0000 7fd77affd640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd75c0664a0 0x7fd75c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:11.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.992+0000 7fd77affd640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd75c0664a0 0x7fd75c068960 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd764007fd0 tx=0x7fd764009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:11.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:11.996+0000 7fd778ff9640 1 -- 172.21.15.148:0/2137914409 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd7740ef360 con 0x7fd77c15d6f0 2023-10-30T13:27:12.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd77c0c6c10 con 0x7fd77c15d6f0 2023-10-30T13:27:12.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd778ff9640 1 -- 172.21.15.148:0/2137914409 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fd7740f2980 con 0x7fd77c15d6f0 2023-10-30T13:27:12.173 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:12.173 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:12.173 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 80 2023-10-30T13:27:12.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd75c0664a0 msgr2=0x7fd75c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:12.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd75c0664a0 0x7fd75c068960 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd764007fd0 tx=0x7fd764009040 comp rx=0 tx=0).stop 2023-10-30T13:27:12.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c15d6f0 msgr2=0x7fd77c1f2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:12.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c15d6f0 0x7fd77c1f2420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd77401ce20 tx=0x7fd77401bf90 comp rx=0 tx=0).stop 2023-10-30T13:27:12.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 shutdown_connections 2023-10-30T13:27:12.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd75c0664a0 0x7fd75c068960 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd77c15d6f0 0x7fd77c1f2420 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd77c1538a0 0x7fd77c0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 --2- 172.21.15.148:0/2137914409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77c0cf160 0x7fd77c149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 >> 172.21.15.148:0/2137914409 conn(0x7fd77c06bfe0 msgr2=0x7fd77c151e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 shutdown_connections 2023-10-30T13:27:12.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:12.176+0000 7fd78186b640 1 -- 172.21.15.148:0/2137914409 wait complete. 2023-10-30T13:27:12.194 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5955 2023-10-30T13:27:12.194 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.c 2023-10-30T13:27:17.650 INFO:tasks.ceph.mds.a.smithi148.stderr:2023-10-30T13:27:17.652+0000 7f94c150f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:27:19.196 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:27:19.196 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:27:19.197 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:27:19.201 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:27:19.201 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:19.296 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:27:19.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f333e640 1 Processor -- start 2023-10-30T13:27:19.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f333e640 1 -- start start 2023-10-30T13:27:19.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f333e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec1538a0 0x7f06ec15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:19.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f333e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06ec0cf160 0x7f06ec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:19.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f333e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec0cb800 con 0x7f06ec0ca0e0 2023-10-30T13:27:19.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f333e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec15dc30 con 0x7f06ec0cf160 2023-10-30T13:27:19.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f333e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec0cb330 con 0x7f06ec1538a0 2023-10-30T13:27:19.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f08d9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f06ec0ca0e0 0x7f06ec15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53774/0 (socket says 172.21.15.148:53774) 2023-10-30T13:27:19.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f08d9640 1 -- 172.21.15.148:0/2645907554 learned_addr learned my addr 172.21.15.148:0/2645907554 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:27:19.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f18db640 1 --2- 172.21.15.148:0/2645907554 >> v2:172.21.15.150:3300/0 conn(0x7f06ec0cf160 0x7f06ec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:19.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f10da640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec1538a0 0x7f06ec15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:19.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979679080 0 0) 0x7f06ec0cb800 con 0x7f06ec0ca0e0 2023-10-30T13:27:19.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d8003530 con 0x7f06ec0ca0e0 2023-10-30T13:27:19.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1004117513 0 0) 0x7f06d8003530 con 0x7f06ec0ca0e0 2023-10-30T13:27:19.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec1538a0 msgr2=0x7f06ec15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:19.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec1538a0 0x7f06ec15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 >> v2:172.21.15.150:3300/0 conn(0x7f06ec0cf160 msgr2=0x7f06ec0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:19.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 --2- 172.21.15.148:0/2645907554 >> v2:172.21.15.150:3300/0 conn(0x7f06ec0cf160 0x7f06ec0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec0cad70 con 0x7f06ec0ca0e0 2023-10-30T13:27:19.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f06dc001680 con 0x7f06ec0ca0e0 2023-10-30T13:27:19.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 >> v1:172.21.15.148:6789/0 conn(0x7f06ec0ca0e0 legacy=0x7f06ec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:19.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec1538a0 0x7f06d8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:19.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06d80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:19.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d8004d10 0x7f06d8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:19.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec15dc30 con 0x7f06d8003f00 2023-10-30T13:27:19.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec0cb330 con 0x7f06d8004d10 2023-10-30T13:27:19.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d800b9c0 con 0x7f06ec1538a0 2023-10-30T13:27:19.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f08d9640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06d80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:19.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f10da640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec1538a0 0x7f06d8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:19.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.356+0000 7f06f18db640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d8004d10 0x7f06d8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:19.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f08d9640 1 -- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec1538a0 msgr2=0x7f06d8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:19.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f08d9640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec1538a0 0x7f06d8003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f08d9640 1 -- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d8004d10 msgr2=0x7f06d8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:19.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f08d9640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d8004d10 0x7f06d8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f08d9640 1 -- 172.21.15.148:0/2645907554 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d8003530 con 0x7f06d8003f00 2023-10-30T13:27:19.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f08d9640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06d80047d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00c240 tx=0x7f06dc01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b91645f5e644009 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:19.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc0a5960 con 0x7f06d8003f00 2023-10-30T13:27:19.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06dc0a5ac0 con 0x7f06d8003f00 2023-10-30T13:27:19.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06e3fff640 1 -- 172.21.15.148:0/2645907554 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc0058e0 con 0x7f06d8003f00 2023-10-30T13:27:19.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f333e640 1 -- 172.21.15.148:0/2645907554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 msgr2=0x7f06d80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:19.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f333e640 1 --2- 172.21.15.148:0/2645907554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06d80047d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00c240 tx=0x7f06dc01cc60 comp rx=0 tx=0).stop 2023-10-30T13:27:19.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f18db640 1 -- 172.21.15.148:0/2645907554 reap_dead start 2023-10-30T13:27:19.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f333e640 1 -- 172.21.15.148:0/2645907554 shutdown_connections 2023-10-30T13:27:19.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.360+0000 7f06f333e640 1 -- 172.21.15.148:0/2645907554 >> 172.21.15.148:0/2645907554 conn(0x7f06ec06bfe0 msgr2=0x7f06ec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:19.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 -- 172.21.15.148:0/2645907554 shutdown_connections 2023-10-30T13:27:19.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 -- 172.21.15.148:0/2645907554 wait complete. 2023-10-30T13:27:19.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 Processor -- start 2023-10-30T13:27:19.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 -- start start 2023-10-30T13:27:19.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06ec0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:19.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06ec1e57f0 0x7f06ec1e3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:19.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec15d690 0x7f06ec1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:19.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1e38c0 con 0x7f06d8003f00 2023-10-30T13:27:19.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec158660 con 0x7f06ec1e57f0 2023-10-30T13:27:19.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1e3590 con 0x7f06ec15d690 2023-10-30T13:27:19.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f10da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06ec0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:19.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f10da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06ec0ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57568/0 (socket says 172.21.15.148:57568) 2023-10-30T13:27:19.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f10da640 1 -- 172.21.15.148:0/3230593759 learned_addr learned my addr 172.21.15.148:0/3230593759 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:19.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f08d9640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06ec1e57f0 0x7f06ec1e3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:19.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f18db640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec15d690 0x7f06ec1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:19.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f10da640 1 -- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec15d690 msgr2=0x7f06ec1579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:19.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f10da640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec15d690 0x7f06ec1579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f10da640 1 -- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06ec1e57f0 msgr2=0x7f06ec1e3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:19.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f10da640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06ec1e57f0 0x7f06ec1e3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f10da640 1 -- 172.21.15.148:0/3230593759 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec1583a0 con 0x7f06d8003f00 2023-10-30T13:27:19.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f10da640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06ec0ccfb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f06e80119d0 tx=0x7f06e8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:19.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06e1ffb640 1 -- 172.21.15.148:0/3230593759 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e800fb50 con 0x7f06d8003f00 2023-10-30T13:27:19.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06e1ffb640 1 -- 172.21.15.148:0/3230593759 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e800fcb0 con 0x7f06d8003f00 2023-10-30T13:27:19.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06e1ffb640 1 -- 172.21.15.148:0/3230593759 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8014440 con 0x7f06d8003f00 2023-10-30T13:27:19.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.364+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec1590c0 con 0x7f06d8003f00 2023-10-30T13:27:19.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.368+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec1593a0 con 0x7f06d8003f00 2023-10-30T13:27:19.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.368+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec159810 con 0x7f06d8003f00 2023-10-30T13:27:19.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.368+0000 7f06e1ffb640 1 -- 172.21.15.148:0/3230593759 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06e80145a0 con 0x7f06d8003f00 2023-10-30T13:27:19.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.368+0000 7f06e1ffb640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06d00664a0 0x7f06d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:19.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.368+0000 7f06e1ffb640 1 -- 172.21.15.148:0/3230593759 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f06e8084b20 con 0x7f06d8003f00 2023-10-30T13:27:19.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.368+0000 7f06f08d9640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06d00664a0 0x7f06d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:19.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.372+0000 7f06f08d9640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06d00664a0 0x7f06d0068960 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0aa000 tx=0x7f06dc001a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:19.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.372+0000 7f06e1ffb640 1 -- 172.21.15.148:0/3230593759 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06e8051bf0 con 0x7f06d8003f00 2023-10-30T13:27:19.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06ec04e170 con 0x7f06d8003f00 2023-10-30T13:27:19.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06e1ffb640 1 -- 172.21.15.148:0/3230593759 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f06e8022070 con 0x7f06d8003f00 2023-10-30T13:27:19.553 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:19.553 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:19.555 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 80 2023-10-30T13:27:19.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06d00664a0 msgr2=0x7f06d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:19.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06d00664a0 0x7f06d0068960 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0aa000 tx=0x7f06dc001a80 comp rx=0 tx=0).stop 2023-10-30T13:27:19.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 msgr2=0x7f06ec0ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:19.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06ec0ccfb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f06e80119d0 tx=0x7f06e8011e90 comp rx=0 tx=0).stop 2023-10-30T13:27:19.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 shutdown_connections 2023-10-30T13:27:19.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06d00664a0 0x7f06d0068960 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06ec15d690 0x7f06ec1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06ec1e57f0 0x7f06ec1e3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 --2- 172.21.15.148:0/3230593759 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06d8003f00 0x7f06ec0ccfb0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:19.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 >> 172.21.15.148:0/3230593759 conn(0x7f06ec06bfe0 msgr2=0x7f06ec152c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:19.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 shutdown_connections 2023-10-30T13:27:19.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:19.556+0000 7f06f333e640 1 -- 172.21.15.148:0/3230593759 wait complete. 2023-10-30T13:27:19.573 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:27:21.575 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:21.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff6e06640 1 Processor -- start 2023-10-30T13:27:21.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff6e06640 1 -- start start 2023-10-30T13:27:21.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff6e06640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbff01538f0 0x7fbff0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:21.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff6e06640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbff015cd50 0x7fbff00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:21.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff6e06640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff015d300 con 0x7fbff00cefa0 2023-10-30T13:27:21.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff6e06640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff015d5e0 con 0x7fbff01538f0 2023-10-30T13:27:21.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff6e06640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff015d8c0 con 0x7fbff015cd50 2023-10-30T13:27:21.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff5e04640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbff01538f0 0x7fbff0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff5e04640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbff01538f0 0x7fbff0159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57580/0 (socket says 172.21.15.148:57580) 2023-10-30T13:27:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff5e04640 1 -- 172.21.15.148:0/496232501 learned_addr learned my addr 172.21.15.148:0/496232501 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff5603640 1 --2- 172.21.15.148:0/496232501 >> v2:172.21.15.150:3300/0 conn(0x7fbff015cd50 0x7fbff00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff5e04640 1 -- 172.21.15.148:0/496232501 >> v2:172.21.15.150:3300/0 conn(0x7fbff015cd50 msgr2=0x7fbff00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:21.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff5e04640 1 --2- 172.21.15.148:0/496232501 >> v2:172.21.15.150:3300/0 conn(0x7fbff015cd50 0x7fbff00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff5e04640 1 -- 172.21.15.148:0/496232501 >> v1:172.21.15.150:6789/0 conn(0x7fbff00cefa0 legacy=0x7fbff00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:21.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.724+0000 7fbff5e04640 1 -- 172.21.15.148:0/496232501 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff015dba0 con 0x7fbff01538f0 2023-10-30T13:27:21.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff5e04640 1 --2- 172.21.15.148:0/496232501 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff01538f0 0x7fbff0159fb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fbff0072e90 tx=0x7fbfec01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561c5017612c8c38 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:21.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff4e02640 1 -- 172.21.15.148:0/496232501 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0a89f0 con 0x7fbff01538f0 2023-10-30T13:27:21.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff4e02640 1 -- 172.21.15.148:0/496232501 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec0a8b50 con 0x7fbff01538f0 2023-10-30T13:27:21.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff4e02640 1 -- 172.21.15.148:0/496232501 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0b0440 con 0x7fbff01538f0 2023-10-30T13:27:21.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff6e06640 1 -- 172.21.15.148:0/496232501 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff01538f0 msgr2=0x7fbff0159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:21.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff6e06640 1 --2- 172.21.15.148:0/496232501 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff01538f0 0x7fbff0159fb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fbff0072e90 tx=0x7fbfec01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff6e06640 1 -- 172.21.15.148:0/496232501 shutdown_connections 2023-10-30T13:27:21.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff6e06640 1 --2- 172.21.15.148:0/496232501 >> v2:172.21.15.150:3300/0 conn(0x7fbff015cd50 0x7fbff00c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff6e06640 1 --2- 172.21.15.148:0/496232501 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff01538f0 0x7fbff0159fb0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff6e06640 1 -- 172.21.15.148:0/496232501 >> 172.21.15.148:0/496232501 conn(0x7fbff006a640 msgr2=0x7fbff0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:21.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff6e06640 1 -- 172.21.15.148:0/496232501 shutdown_connections 2023-10-30T13:27:21.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.728+0000 7fbff6e06640 1 -- 172.21.15.148:0/496232501 wait complete. 2023-10-30T13:27:21.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 Processor -- start 2023-10-30T13:27:21.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 -- start start 2023-10-30T13:27:21.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff00cefa0 0x7fbff01573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:21.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbff01538f0 0x7fbff0145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:21.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbff015cd50 0x7fbff01f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:21.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff0152ad0 con 0x7fbff00cefa0 2023-10-30T13:27:21.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff0146490 con 0x7fbff015cd50 2023-10-30T13:27:21.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff0157ff0 con 0x7fbff01538f0 2023-10-30T13:27:21.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5e04640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff00cefa0 0x7fbff01573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:21.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5e04640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff00cefa0 0x7fbff01573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57586/0 (socket says 172.21.15.148:57586) 2023-10-30T13:27:21.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5e04640 1 -- 172.21.15.148:0/3890084790 learned_addr learned my addr 172.21.15.148:0/3890084790 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:21.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5603640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbff01538f0 0x7fbff0145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:21.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6605640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbff015cd50 0x7fbff01f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:21.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5e04640 1 -- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbff01538f0 msgr2=0x7fbff0145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:21.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5e04640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbff01538f0 0x7fbff0145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5e04640 1 -- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbff015cd50 msgr2=0x7fbff01f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:21.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5e04640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbff015cd50 0x7fbff01f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5e04640 1 -- 172.21.15.148:0/3890084790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec0094d0 con 0x7fbff00cefa0 2023-10-30T13:27:21.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5603640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbff01538f0 0x7fbff0145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:27:21.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff5e04640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff00cefa0 0x7fbff01573a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbfec01b900 tx=0x7fbfec01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:21.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbfe27fc640 1 -- 172.21.15.148:0/3890084790 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0ba600 con 0x7fbff00cefa0 2023-10-30T13:27:21.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbfe27fc640 1 -- 172.21.15.148:0/3890084790 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec0ba760 con 0x7fbff00cefa0 2023-10-30T13:27:21.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbfe27fc640 1 -- 172.21.15.148:0/3890084790 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0bab10 con 0x7fbff00cefa0 2023-10-30T13:27:21.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff0157a30 con 0x7fbff00cefa0 2023-10-30T13:27:21.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff0152480 con 0x7fbff00cefa0 2023-10-30T13:27:21.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.732+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff004e170 con 0x7fbff00cefa0 2023-10-30T13:27:21.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.736+0000 7fbfe27fc640 1 -- 172.21.15.148:0/3890084790 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbfec0bac70 con 0x7fbff00cefa0 2023-10-30T13:27:21.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.736+0000 7fbfe27fc640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbfd40664d0 0x7fbfd4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:21.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.736+0000 7fbfe27fc640 1 -- 172.21.15.148:0/3890084790 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fbfec122890 con 0x7fbff00cefa0 2023-10-30T13:27:21.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.736+0000 7fbff5603640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbfd40664d0 0x7fbfd4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:21.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.736+0000 7fbff5603640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbfd40664d0 0x7fbfd4068990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbff00b1790 tx=0x7fbfe4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:21.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.740+0000 7fbfe27fc640 1 -- 172.21.15.148:0/3890084790 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbfec0ef8a0 con 0x7fbff00cefa0 2023-10-30T13:27:21.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.952+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbff0150ed0 con 0x7fbff00cefa0 2023-10-30T13:27:21.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.952+0000 7fbfe27fc640 1 -- 172.21.15.148:0/3890084790 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fbfec0b5090 con 0x7fbff00cefa0 2023-10-30T13:27:21.951 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:21.952 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6249,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:21.954 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 81 2023-10-30T13:27:21.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbfd40664d0 msgr2=0x7fbfd4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:21.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbfd40664d0 0x7fbfd4068990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbff00b1790 tx=0x7fbfe4008040 comp rx=0 tx=0).stop 2023-10-30T13:27:21.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff00cefa0 msgr2=0x7fbff01573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:21.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff00cefa0 0x7fbff01573a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbfec01b900 tx=0x7fbfec01ba90 comp rx=0 tx=0).stop 2023-10-30T13:27:21.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 shutdown_connections 2023-10-30T13:27:21.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbfd40664d0 0x7fbfd4068990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbff015cd50 0x7fbff01f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbff01538f0 0x7fbff0145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 --2- 172.21.15.148:0/3890084790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbff00cefa0 0x7fbff01573a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:21.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 >> 172.21.15.148:0/3890084790 conn(0x7fbff006a640 msgr2=0x7fbff0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:21.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 shutdown_connections 2023-10-30T13:27:21.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:21.956+0000 7fbff6e06640 1 -- 172.21.15.148:0/3890084790 wait complete. 2023-10-30T13:27:21.971 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:27:21.971 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2023-10-30T13:27:25.869 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:25.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:55.934 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:55.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:57.972 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:58.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9aed046640 1 Processor -- start 2023-10-30T13:27:58.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9aed046640 1 -- start start 2023-10-30T13:27:58.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9aed046640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae815d6f0 0x7f9ae815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9aed046640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9ae80cf160 0x7f9ae80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9aed046640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae80cb800 con 0x7f9ae81538a0 2023-10-30T13:27:58.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9aed046640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae80cb330 con 0x7f9ae80cf160 2023-10-30T13:27:58.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9aed046640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae80cad70 con 0x7f9ae815d6f0 2023-10-30T13:27:58.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9ae6d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9ae81538a0 0x7f9ae80ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49940/0 (socket says 172.21.15.148:49940) 2023-10-30T13:27:58.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9ae6d9d640 1 -- 172.21.15.148:0/287080384 learned_addr learned my addr 172.21.15.148:0/287080384 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:27:58.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9ae759e640 1 --2- 172.21.15.148:0/287080384 >> v2:172.21.15.148:3300/0 conn(0x7f9ae80cf160 0x7f9ae80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9adffff640 1 --2- 172.21.15.148:0/287080384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae815d6f0 0x7f9ae815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9ae659c640 1 -- 172.21.15.148:0/287080384 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562995027 0 0) 0x7f9ae80cb800 con 0x7f9ae81538a0 2023-10-30T13:27:58.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9ae659c640 1 -- 172.21.15.148:0/287080384 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad4003530 con 0x7f9ae81538a0 2023-10-30T13:27:58.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9ae759e640 1 -- 172.21.15.148:0/287080384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae815d6f0 msgr2=0x7f9ae815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:58.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9ae759e640 1 --2- 172.21.15.148:0/287080384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae815d6f0 0x7f9ae815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9ae759e640 1 -- 172.21.15.148:0/287080384 >> v1:172.21.15.148:6789/0 conn(0x7f9ae81538a0 legacy=0x7f9ae80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.115+0000 7f9ae759e640 1 -- 172.21.15.148:0/287080384 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae80cb050 con 0x7f9ae80cf160 2023-10-30T13:27:58.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9ae759e640 1 --2- 172.21.15.148:0/287080384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae80cf160 0x7f9ae80cf590 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0009d90 tx=0x7f9ae001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87820f06b94e15c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:58.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9ae659c640 1 -- 172.21.15.148:0/287080384 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae00af600 con 0x7f9ae80cf160 2023-10-30T13:27:58.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9ae659c640 1 -- 172.21.15.148:0/287080384 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae00ab050 con 0x7f9ae80cf160 2023-10-30T13:27:58.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9ae659c640 1 -- 172.21.15.148:0/287080384 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae00af980 con 0x7f9ae80cf160 2023-10-30T13:27:58.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9aed046640 1 -- 172.21.15.148:0/287080384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae80cf160 msgr2=0x7f9ae80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9aed046640 1 --2- 172.21.15.148:0/287080384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae80cf160 0x7f9ae80cf590 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0009d90 tx=0x7f9ae001f130 comp rx=0 tx=0).stop 2023-10-30T13:27:58.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9aed046640 1 -- 172.21.15.148:0/287080384 shutdown_connections 2023-10-30T13:27:58.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9aed046640 1 --2- 172.21.15.148:0/287080384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae80cf160 0x7f9ae80cf590 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9aed046640 1 --2- 172.21.15.148:0/287080384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae815d6f0 0x7f9ae815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9aed046640 1 -- 172.21.15.148:0/287080384 >> 172.21.15.148:0/287080384 conn(0x7f9ae806bfe0 msgr2=0x7f9ae806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:58.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9aed046640 1 -- 172.21.15.148:0/287080384 shutdown_connections 2023-10-30T13:27:58.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.119+0000 7f9aed046640 1 -- 172.21.15.148:0/287080384 wait complete. 2023-10-30T13:27:58.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 Processor -- start 2023-10-30T13:27:58.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 -- start start 2023-10-30T13:27:58.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae80cf160 0x7f9ae81e4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae81538a0 0x7f9ae8065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae815d6f0 0x7f9ae80c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae80c7800 con 0x7f9ae815d6f0 2023-10-30T13:27:58.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae81e55f0 con 0x7f9ae81538a0 2023-10-30T13:27:58.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae81f24a0 con 0x7f9ae80cf160 2023-10-30T13:27:58.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae815d6f0 0x7f9ae80c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae815d6f0 0x7f9ae80c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44192/0 (socket says 172.21.15.148:44192) 2023-10-30T13:27:58.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae759e640 1 -- 172.21.15.148:0/3204347629 learned_addr learned my addr 172.21.15.148:0/3204347629 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:58.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae6d9d640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae80cf160 0x7f9ae81e4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9adffff640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae81538a0 0x7f9ae8065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae759e640 1 -- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae80cf160 msgr2=0x7f9ae81e4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae759e640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae80cf160 0x7f9ae81e4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae759e640 1 -- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae81538a0 msgr2=0x7f9ae8065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae759e640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae81538a0 0x7f9ae8065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae759e640 1 -- 172.21.15.148:0/3204347629 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae0009a70 con 0x7f9ae815d6f0 2023-10-30T13:27:58.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9ae759e640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae815d6f0 0x7f9ae80c7c50 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00b2040 tx=0x7f9ae001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:58.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9adf7fe640 1 -- 172.21.15.148:0/3204347629 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae0009450 con 0x7f9ae815d6f0 2023-10-30T13:27:58.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9adf7fe640 1 -- 172.21.15.148:0/3204347629 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae00b9410 con 0x7f9ae815d6f0 2023-10-30T13:27:58.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae81f2750 con 0x7f9ae815d6f0 2023-10-30T13:27:58.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9adf7fe640 1 -- 172.21.15.148:0/3204347629 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae00b97c0 con 0x7f9ae815d6f0 2023-10-30T13:27:58.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.123+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae81f2f90 con 0x7f9ae815d6f0 2023-10-30T13:27:58.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.127+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae80b2a50 con 0x7f9ae815d6f0 2023-10-30T13:27:58.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.127+0000 7f9adf7fe640 1 -- 172.21.15.148:0/3204347629 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ae00b6050 con 0x7f9ae815d6f0 2023-10-30T13:27:58.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.127+0000 7f9adf7fe640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ac80664d0 0x7f9ac8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.127+0000 7f9adf7fe640 1 -- 172.21.15.148:0/3204347629 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f9ae01227d0 con 0x7f9ae815d6f0 2023-10-30T13:27:58.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.127+0000 7f9ae6d9d640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ac80664d0 0x7f9ac8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.127+0000 7f9ae6d9d640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ac80664d0 0x7f9ac8068990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80011f0 tx=0x7f9ad8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:58.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.131+0000 7f9adf7fe640 1 -- 172.21.15.148:0/3204347629 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ae00ef900 con 0x7f9ae815d6f0 2023-10-30T13:27:58.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.335+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ae80c7940 con 0x7f9ae815d6f0 2023-10-30T13:27:58.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9adf7fe640 1 -- 172.21.15.148:0/3204347629 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f9ae00f2f20 con 0x7f9ae815d6f0 2023-10-30T13:27:58.336 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:58.336 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6249,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:58.337 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 81 2023-10-30T13:27:58.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ac80664d0 msgr2=0x7f9ac8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ac80664d0 0x7f9ac8068990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80011f0 tx=0x7f9ad8009040 comp rx=0 tx=0).stop 2023-10-30T13:27:58.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae815d6f0 msgr2=0x7f9ae80c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae815d6f0 0x7f9ae80c7c50 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00b2040 tx=0x7f9ae001bed0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 shutdown_connections 2023-10-30T13:27:58.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9ac80664d0 0x7f9ac8068990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ae815d6f0 0x7f9ae80c7c50 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae81538a0 0x7f9ae8065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 --2- 172.21.15.148:0/3204347629 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ae80cf160 0x7f9ae81e4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 >> 172.21.15.148:0/3204347629 conn(0x7f9ae806bfe0 msgr2=0x7f9ae80ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:58.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 shutdown_connections 2023-10-30T13:27:58.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.339+0000 7f9aed046640 1 -- 172.21.15.148:0/3204347629 wait complete. 2023-10-30T13:27:58.359 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:27:58.360 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:27:58.360 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:58.380 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:27:58.380 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:58.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.567+0000 7f9084a09640 1 Processor -- start 2023-10-30T13:27:58.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.567+0000 7f9084a09640 1 -- start start 2023-10-30T13:27:58.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.567+0000 7f9084a09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f90800c6f60 0x7f90801566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.567+0000 7f9084a09640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90800c0a50 con 0x7f90800c6b80 2023-10-30T13:27:58.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.567+0000 7f9084a09640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908006f6e0 con 0x7f90800c05d0 2023-10-30T13:27:58.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.567+0000 7f9084a09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908006f9c0 con 0x7f90800c6f60 2023-10-30T13:27:58.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.567+0000 7f907e59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f90800c6b80 0x7f90800bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49954/0 (socket says 172.21.15.148:49954) 2023-10-30T13:27:58.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.567+0000 7f907e59c640 1 -- 172.21.15.148:0/3070033573 learned_addr learned my addr 172.21.15.148:0/3070033573 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:27:58.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.567+0000 7f907ed9d640 1 --2- 172.21.15.148:0/3070033573 >> v2:172.21.15.150:3300/0 conn(0x7f90800c6f60 0x7f90801566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167343563 0 0) 0x7f90800c0a50 con 0x7f90800c6b80 2023-10-30T13:27:58.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9068005740 con 0x7f90800c6b80 2023-10-30T13:27:58.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2157603658 0 0) 0x7f9068005740 con 0x7f90800c6b80 2023-10-30T13:27:58.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 >> v2:172.21.15.150:3300/0 conn(0x7f90800c6f60 msgr2=0x7f90801566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:58.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 --2- 172.21.15.148:0/3070033573 >> v2:172.21.15.150:3300/0 conn(0x7f90800c6f60 0x7f90801566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 >> v1:172.21.15.150:6789/0 conn(0x7f90800c05d0 legacy=0x7f908013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908006fca0 con 0x7f90800c6b80 2023-10-30T13:27:58.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f9070001680 con 0x7f90800c6b80 2023-10-30T13:27:58.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 >> v1:172.21.15.148:6789/0 conn(0x7f90800c6b80 legacy=0x7f90800bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9068005b90 0x7f9068005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90680069e0 0x7f9068006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90680073a0 0x7f906800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908006f9c0 con 0x7f9068005b90 2023-10-30T13:27:58.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906800e240 con 0x7f90680073a0 2023-10-30T13:27:58.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906800e400 con 0x7f90680069e0 2023-10-30T13:27:58.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907e59c640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9068005b90 0x7f9068005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907ed9d640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90680073a0 0x7f906800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907dd9b640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90680069e0 0x7f9068006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907e59c640 1 -- 172.21.15.148:0/3070033573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90680069e0 msgr2=0x7f9068006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:58.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907e59c640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90680069e0 0x7f9068006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907e59c640 1 -- 172.21.15.148:0/3070033573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90680073a0 msgr2=0x7f906800dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:58.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907e59c640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90680073a0 0x7f906800dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907e59c640 1 -- 172.21.15.148:0/3070033573 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9068005740 con 0x7f9068005b90 2023-10-30T13:27:58.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.571+0000 7f907ed9d640 1 -- 172.21.15.148:0/3070033573 reap_dead start 2023-10-30T13:27:58.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f907e59c640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9068005b90 0x7f9068005fb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f907000c240 tx=0x7f907001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8d6e7260674602 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:58.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070005ae0 con 0x7f9068005b90 2023-10-30T13:27:58.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9070005c40 con 0x7f9068005b90 2023-10-30T13:27:58.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f907d59a640 1 -- 172.21.15.148:0/3070033573 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90700a5a30 con 0x7f9068005b90 2023-10-30T13:27:58.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 -- 172.21.15.148:0/3070033573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9068005b90 msgr2=0x7f9068005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9068005b90 0x7f9068005fb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f907000c240 tx=0x7f907001cc00 comp rx=0 tx=0).stop 2023-10-30T13:27:58.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 -- 172.21.15.148:0/3070033573 shutdown_connections 2023-10-30T13:27:58.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 --2- 172.21.15.148:0/3070033573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9068005b90 0x7f9068005fb0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 -- 172.21.15.148:0/3070033573 >> 172.21.15.148:0/3070033573 conn(0x7f90800c7910 msgr2=0x7f90800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:58.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 -- 172.21.15.148:0/3070033573 shutdown_connections 2023-10-30T13:27:58.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 -- 172.21.15.148:0/3070033573 wait complete. 2023-10-30T13:27:58.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 Processor -- start 2023-10-30T13:27:58.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 -- start start 2023-10-30T13:27:58.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9068005b90 0x7f90801ee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f908013bd40 0x7f90800c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9080137900 0x7f908013ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908013d930 con 0x7f9080137900 2023-10-30T13:27:58.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90801edf60 con 0x7f9068005b90 2023-10-30T13:27:58.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.575+0000 7f9084a09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90800c8f10 con 0x7f908013bd40 2023-10-30T13:27:58.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9080137900 0x7f908013ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9080137900 0x7f908013ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44208/0 (socket says 172.21.15.148:44208) 2023-10-30T13:27:58.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907ed9d640 1 -- 172.21.15.148:0/543780921 learned_addr learned my addr 172.21.15.148:0/543780921 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:27:58.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907dd9b640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f908013bd40 0x7f90800c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907e59c640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9068005b90 0x7f90801ee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907ed9d640 1 -- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f908013bd40 msgr2=0x7f90800c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907ed9d640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f908013bd40 0x7f90800c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907ed9d640 1 -- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9068005b90 msgr2=0x7f90801ee3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907ed9d640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9068005b90 0x7f90801ee3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907ed9d640 1 -- 172.21.15.148:0/543780921 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908013c310 con 0x7f9080137900 2023-10-30T13:27:58.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f907ed9d640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9080137900 0x7f908013ac70 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9080159b90 tx=0x7f9078007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:58.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f906effd640 1 -- 172.21.15.148:0/543780921 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9078018070 con 0x7f9080137900 2023-10-30T13:27:58.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f906effd640 1 -- 172.21.15.148:0/543780921 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9078007e90 con 0x7f9080137900 2023-10-30T13:27:58.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908013c5f0 con 0x7f9080137900 2023-10-30T13:27:58.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f906effd640 1 -- 172.21.15.148:0/543780921 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90780039a0 con 0x7f9080137900 2023-10-30T13:27:58.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90801edbf0 con 0x7f9080137900 2023-10-30T13:27:58.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908004e170 con 0x7f9080137900 2023-10-30T13:27:58.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.579+0000 7f906effd640 1 -- 172.21.15.148:0/543780921 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9078003b00 con 0x7f9080137900 2023-10-30T13:27:58.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.583+0000 7f906effd640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90600664d0 0x7f9060068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:58.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.583+0000 7f906effd640 1 -- 172.21.15.148:0/543780921 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f9078088380 con 0x7f9080137900 2023-10-30T13:27:58.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.583+0000 7f907e59c640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90600664d0 0x7f9060068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:58.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.583+0000 7f907e59c640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90600664d0 0x7f9060068990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f90700ab000 tx=0x7f90700b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:58.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.583+0000 7f906effd640 1 -- 172.21.15.148:0/543780921 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9078055420 con 0x7f9080137900 2023-10-30T13:27:58.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90801ee0a0 con 0x7f9080137900 2023-10-30T13:27:58.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f906effd640 1 -- 172.21.15.148:0/543780921 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f9078058a40 con 0x7f9080137900 2023-10-30T13:27:58.769 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:27:58.769 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6249,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:58.769 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 81 2023-10-30T13:27:58.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90600664d0 msgr2=0x7f9060068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90600664d0 0x7f9060068990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f90700ab000 tx=0x7f90700b1040 comp rx=0 tx=0).stop 2023-10-30T13:27:58.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9080137900 msgr2=0x7f908013ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:58.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9080137900 0x7f908013ac70 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9080159b90 tx=0x7f9078007700 comp rx=0 tx=0).stop 2023-10-30T13:27:58.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 shutdown_connections 2023-10-30T13:27:58.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90600664d0 0x7f9060068990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9080137900 0x7f908013ac70 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f908013bd40 0x7f90800c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 --2- 172.21.15.148:0/543780921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9068005b90 0x7f90801ee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:58.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 >> 172.21.15.148:0/543780921 conn(0x7f90800c7910 msgr2=0x7f90800b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:58.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 shutdown_connections 2023-10-30T13:27:58.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:27:58.771+0000 7f9084a09640 1 -- 172.21.15.148:0/543780921 wait complete. 2023-10-30T13:27:58.790 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:28:00.791 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:00.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74ef8d640 1 Processor -- start 2023-10-30T13:28:00.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74ef8d640 1 -- start start 2023-10-30T13:28:00.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74ef8d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe74815a410 0x7fe74806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:00.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74ef8d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe74806cb80 0x7fe74806d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:00.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74ef8d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74806d550 con 0x7fe748069e90 2023-10-30T13:28:00.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74ef8d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7480c8570 con 0x7fe74806cb80 2023-10-30T13:28:00.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74ef8d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74815a850 con 0x7fe74815a410 2023-10-30T13:28:00.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74cd29640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe748069e90 0x7fe7480c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49970/0 (socket says 172.21.15.148:49970) 2023-10-30T13:28:00.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74cd29640 1 -- 172.21.15.148:0/2735723966 learned_addr learned my addr 172.21.15.148:0/2735723966 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:28:00.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe747fff640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe74815a410 0x7fe74806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:00.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74d52a640 1 --2- 172.21.15.148:0/2735723966 >> v2:172.21.15.150:3300/0 conn(0x7fe74806cb80 0x7fe74806d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:00.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892804538 0 0) 0x7fe74806d550 con 0x7fe748069e90 2023-10-30T13:28:00.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe738003530 con 0x7fe748069e90 2023-10-30T13:28:00.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1537901999 0 0) 0x7fe738003530 con 0x7fe748069e90 2023-10-30T13:28:00.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe74815a410 msgr2=0x7fe74806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:00.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe74815a410 0x7fe74806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:00.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 >> v2:172.21.15.150:3300/0 conn(0x7fe74806cb80 msgr2=0x7fe74806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:00.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 --2- 172.21.15.148:0/2735723966 >> v2:172.21.15.150:3300/0 conn(0x7fe74806cb80 0x7fe74806d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:00.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480c6c10 con 0x7fe748069e90 2023-10-30T13:28:00.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fe73c001680 con 0x7fe748069e90 2023-10-30T13:28:00.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 >> v1:172.21.15.148:6789/0 conn(0x7fe748069e90 legacy=0x7fe7480c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:00.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe738003980 0x7fe738003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:00.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7380047d0 0x7fe738004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe74815a410 0x7fe738005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7480c8570 con 0x7fe738003980 2023-10-30T13:28:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74815a850 con 0x7fe7380047d0 2023-10-30T13:28:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73800b990 con 0x7fe74815a410 2023-10-30T13:28:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74cd29640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe738003980 0x7fe738003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74d52a640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe74815a410 0x7fe738005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe747fff640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7380047d0 0x7fe738004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74cd29640 1 -- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe74815a410 msgr2=0x7fe738005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74cd29640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe74815a410 0x7fe738005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:00.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74cd29640 1 -- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7380047d0 msgr2=0x7fe738004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:00.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74cd29640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7380047d0 0x7fe738004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:00.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.943+0000 7fe74cd29640 1 -- 172.21.15.148:0/2735723966 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738003530 con 0x7fe738003980 2023-10-30T13:28:00.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe74cd29640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe738003980 0x7fe738003da0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00c240 tx=0x7fe73c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b3f93a0a917e49 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:00.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c0058e0 con 0x7fe738003980 2023-10-30T13:28:00.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c005a40 con 0x7fe738003980 2023-10-30T13:28:00.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe7477fe640 1 -- 172.21.15.148:0/2735723966 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c005e10 con 0x7fe738003980 2023-10-30T13:28:00.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe74ef8d640 1 -- 172.21.15.148:0/2735723966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe738003980 msgr2=0x7fe738003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe74ef8d640 1 --2- 172.21.15.148:0/2735723966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe738003980 0x7fe738003da0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00c240 tx=0x7fe73c01cc60 comp rx=0 tx=0).stop 2023-10-30T13:28:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe74d52a640 1 -- 172.21.15.148:0/2735723966 reap_dead start 2023-10-30T13:28:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe74ef8d640 1 -- 172.21.15.148:0/2735723966 shutdown_connections 2023-10-30T13:28:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe74ef8d640 1 -- 172.21.15.148:0/2735723966 >> 172.21.15.148:0/2735723966 conn(0x7fe74806dfd0 msgr2=0x7fe74806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe74ef8d640 1 -- 172.21.15.148:0/2735723966 shutdown_connections 2023-10-30T13:28:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.947+0000 7fe74ef8d640 1 -- 172.21.15.148:0/2735723966 wait complete. 2023-10-30T13:28:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 Processor -- start 2023-10-30T13:28:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 -- start start 2023-10-30T13:28:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe738003980 0x7fe7481f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:00.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe74815ba10 0x7fe7481537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:00.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe748156200 0x7fe748152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:00.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7481f5f60 con 0x7fe74815ba10 2023-10-30T13:28:00.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7481f1bb0 con 0x7fe738003980 2023-10-30T13:28:00.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74815b460 con 0x7fe748156200 2023-10-30T13:28:00.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe747fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe74815ba10 0x7fe7481537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:00.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe747fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe74815ba10 0x7fe7481537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44234/0 (socket says 172.21.15.148:44234) 2023-10-30T13:28:00.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe747fff640 1 -- 172.21.15.148:0/3188382929 learned_addr learned my addr 172.21.15.148:0/3188382929 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:00.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74cd29640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe738003980 0x7fe7481f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:00.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74d52a640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe748156200 0x7fe748152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:00.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe747fff640 1 -- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe748156200 msgr2=0x7fe748152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:00.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe747fff640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe748156200 0x7fe748152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:00.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe747fff640 1 -- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe738003980 msgr2=0x7fe7481f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:00.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe747fff640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe738003980 0x7fe7481f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:00.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe747fff640 1 -- 172.21.15.148:0/3188382929 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7481f1e20 con 0x7fe74815ba10 2023-10-30T13:28:00.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe747fff640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe74815ba10 0x7fe7481537c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fe734011590 tx=0x7fe734011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:00.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe7457fa640 1 -- 172.21.15.148:0/3188382929 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734015930 con 0x7fe74815ba10 2023-10-30T13:28:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe7457fa640 1 -- 172.21.15.148:0/3188382929 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe734015a90 con 0x7fe74815ba10 2023-10-30T13:28:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe7457fa640 1 -- 172.21.15.148:0/3188382929 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734015e20 con 0x7fe74815ba10 2023-10-30T13:28:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7481f2100 con 0x7fe74815ba10 2023-10-30T13:28:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.951+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7481f2900 con 0x7fe74815ba10 2023-10-30T13:28:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.955+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74804e170 con 0x7fe74815ba10 2023-10-30T13:28:00.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.955+0000 7fe7457fa640 1 -- 172.21.15.148:0/3188382929 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe73400f460 con 0x7fe74815ba10 2023-10-30T13:28:00.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.955+0000 7fe7457fa640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe72c0664a0 0x7fe72c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.955+0000 7fe7457fa640 1 -- 172.21.15.148:0/3188382929 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fe734084a50 con 0x7fe74815ba10 2023-10-30T13:28:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.955+0000 7fe74cd29640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe72c0664a0 0x7fe72c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.959+0000 7fe74cd29640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe72c0664a0 0x7fe72c068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0aa000 tx=0x7fe73c001a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:00.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:00.959+0000 7fe7457fa640 1 -- 172.21.15.148:0/3188382929 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe734051a60 con 0x7fe74815ba10 2023-10-30T13:28:01.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.139+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe74806ef60 con 0x7fe74815ba10 2023-10-30T13:28:01.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.139+0000 7fe7457fa640 1 -- 172.21.15.148:0/3188382929 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fe734055080 con 0x7fe74815ba10 2023-10-30T13:28:01.139 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:01.140 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6249,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:01.140 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 81 2023-10-30T13:28:01.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe72c0664a0 msgr2=0x7fe72c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:01.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe72c0664a0 0x7fe72c068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe73c0aa000 tx=0x7fe73c001a80 comp rx=0 tx=0).stop 2023-10-30T13:28:01.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe74815ba10 msgr2=0x7fe7481537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:01.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe74815ba10 0x7fe7481537c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fe734011590 tx=0x7fe734011a50 comp rx=0 tx=0).stop 2023-10-30T13:28:01.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 shutdown_connections 2023-10-30T13:28:01.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe72c0664a0 0x7fe72c068960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:01.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe74815ba10 0x7fe7481537c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:01.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe748156200 0x7fe748152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:01.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 --2- 172.21.15.148:0/3188382929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe738003980 0x7fe7481f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:01.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 >> 172.21.15.148:0/3188382929 conn(0x7fe74806dfd0 msgr2=0x7fe7480cb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:01.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 shutdown_connections 2023-10-30T13:28:01.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:01.143+0000 7fe74ef8d640 1 -- 172.21.15.148:0/3188382929 wait complete. 2023-10-30T13:28:01.160 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:28:03.162 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:03.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc379068640 1 Processor -- start 2023-10-30T13:28:03.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc379068640 1 -- start start 2023-10-30T13:28:03.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc379068640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3740cf160 0x7fc3740cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:03.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc379068640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3740cb800 con 0x7fc3741538a0 2023-10-30T13:28:03.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc379068640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3740cb330 con 0x7fc37415d6f0 2023-10-30T13:28:03.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc379068640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740cad70 con 0x7fc3740cf160 2023-10-30T13:28:03.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc372d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc3741538a0 0x7fc3740ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58682/0 (socket says 172.21.15.148:58682) 2023-10-30T13:28:03.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc372d9d640 1 -- 172.21.15.148:0/3529438445 learned_addr learned my addr 172.21.15.148:0/3529438445 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:28:03.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc37359e640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3740cf160 0x7fc3740cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:03.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233749539 0 0) 0x7fc3740cb800 con 0x7fc3741538a0 2023-10-30T13:28:03.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c003530 con 0x7fc3741538a0 2023-10-30T13:28:03.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 412696504 0 0) 0x7fc35c003530 con 0x7fc3741538a0 2023-10-30T13:28:03.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3740cf160 msgr2=0x7fc3740cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3740cf160 0x7fc3740cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 >> v1:172.21.15.150:6789/0 conn(0x7fc37415d6f0 legacy=0x7fc37415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740cb050 con 0x7fc3741538a0 2023-10-30T13:28:03.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fc364001680 con 0x7fc3741538a0 2023-10-30T13:28:03.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 >> v1:172.21.15.148:6789/0 conn(0x7fc3741538a0 legacy=0x7fc3740ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:03.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3740cf160 0x7fc35c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:03.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c003f30 0x7fc35c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:03.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc35c004d10 0x7fc35c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:03.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740cad70 con 0x7fc35c004d10 2023-10-30T13:28:03.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c00bb00 con 0x7fc35c003f30 2023-10-30T13:28:03.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c00bcc0 con 0x7fc3740cf160 2023-10-30T13:28:03.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc37359e640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc35c004d10 0x7fc35c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:03.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc37259c640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c003f30 0x7fc35c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc372d9d640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3740cf160 0x7fc35c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc37359e640 1 -- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3740cf160 msgr2=0x7fc35c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc37359e640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3740cf160 0x7fc35c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:03.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc37359e640 1 -- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c003f30 msgr2=0x7fc35c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc37359e640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c003f30 0x7fc35c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc37359e640 1 -- 172.21.15.148:0/3529438445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c003530 con 0x7fc35c004d10 2023-10-30T13:28:03.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.315+0000 7fc372d9d640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3740cf160 0x7fc35c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:28:03.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc37359e640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc35c004d10 0x7fc35c005190 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009fc0 tx=0x7fc36c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5fba055bdcad0a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:03.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c0a5c70 con 0x7fc35c004d10 2023-10-30T13:28:03.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c00f040 con 0x7fc35c004d10 2023-10-30T13:28:03.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc371d9b640 1 -- 172.21.15.148:0/3529438445 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c0ad440 con 0x7fc35c004d10 2023-10-30T13:28:03.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc379068640 1 -- 172.21.15.148:0/3529438445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc35c004d10 msgr2=0x7fc35c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:03.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc379068640 1 --2- 172.21.15.148:0/3529438445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc35c004d10 0x7fc35c005190 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009fc0 tx=0x7fc36c01cc60 comp rx=0 tx=0).stop 2023-10-30T13:28:03.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc37359e640 1 -- 172.21.15.148:0/3529438445 reap_dead start 2023-10-30T13:28:03.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc379068640 1 -- 172.21.15.148:0/3529438445 shutdown_connections 2023-10-30T13:28:03.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc379068640 1 -- 172.21.15.148:0/3529438445 >> 172.21.15.148:0/3529438445 conn(0x7fc37406bfe0 msgr2=0x7fc37406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:03.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc379068640 1 -- 172.21.15.148:0/3529438445 shutdown_connections 2023-10-30T13:28:03.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.319+0000 7fc379068640 1 -- 172.21.15.148:0/3529438445 wait complete. 2023-10-30T13:28:03.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 Processor -- start 2023-10-30T13:28:03.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 -- start start 2023-10-30T13:28:03.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc35c004d10 0x7fc3741f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:03.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3740b3110 0x7fc374157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:03.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3740b1960 0x7fc3741440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:03.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740b4b90 con 0x7fc3740b3110 2023-10-30T13:28:03.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3741f21b0 con 0x7fc3740b1960 2023-10-30T13:28:03.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740b4df0 con 0x7fc35c004d10 2023-10-30T13:28:03.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3740b3110 0x7fc374157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:03.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3740b3110 0x7fc374157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54662/0 (socket says 172.21.15.148:54662) 2023-10-30T13:28:03.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37259c640 1 -- 172.21.15.148:0/670180064 learned_addr learned my addr 172.21.15.148:0/670180064 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:03.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc372d9d640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc35c004d10 0x7fc3741f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:03.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37359e640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3740b1960 0x7fc3741440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:03.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37259c640 1 -- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc35c004d10 msgr2=0x7fc3741f2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:03.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37259c640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc35c004d10 0x7fc3741f2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:03.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37259c640 1 -- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3740b1960 msgr2=0x7fc3741440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:03.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37259c640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3740b1960 0x7fc3741440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:03.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37259c640 1 -- 172.21.15.148:0/670180064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3741f2450 con 0x7fc3740b3110 2023-10-30T13:28:03.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc37259c640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3740b3110 0x7fc374157520 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e580 tx=0x7fc36800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:03.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc35b7fe640 1 -- 172.21.15.148:0/670180064 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3680052e0 con 0x7fc3740b3110 2023-10-30T13:28:03.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc35b7fe640 1 -- 172.21.15.148:0/670180064 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc368005440 con 0x7fc3740b3110 2023-10-30T13:28:03.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc35b7fe640 1 -- 172.21.15.148:0/670180064 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc368003840 con 0x7fc3740b3110 2023-10-30T13:28:03.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3741f2730 con 0x7fc3740b3110 2023-10-30T13:28:03.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.323+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3740c50f0 con 0x7fc3740b3110 2023-10-30T13:28:03.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.327+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37404e170 con 0x7fc3740b3110 2023-10-30T13:28:03.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.327+0000 7fc35b7fe640 1 -- 172.21.15.148:0/670180064 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3680039a0 con 0x7fc3740b3110 2023-10-30T13:28:03.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.327+0000 7fc35b7fe640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3600664a0 0x7fc360068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:03.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.327+0000 7fc35b7fe640 1 -- 172.21.15.148:0/670180064 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7fc3680840d0 con 0x7fc3740b3110 2023-10-30T13:28:03.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.327+0000 7fc372d9d640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3600664a0 0x7fc360068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:03.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.327+0000 7fc372d9d640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3600664a0 0x7fc360068960 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc364001210 tx=0x7fc364009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:03.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.331+0000 7fc35b7fe640 1 -- 172.21.15.148:0/670180064 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc368051230 con 0x7fc3740b3110 2023-10-30T13:28:03.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.511+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3740b3bb0 con 0x7fc3740b3110 2023-10-30T13:28:03.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.511+0000 7fc35b7fe640 1 -- 172.21.15.148:0/670180064 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fc368058980 con 0x7fc3740b3110 2023-10-30T13:28:03.510 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:03.511 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6249,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:03.511 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 81 2023-10-30T13:28:03.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3600664a0 msgr2=0x7fc360068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:03.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3600664a0 0x7fc360068960 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc364001210 tx=0x7fc364009040 comp rx=0 tx=0).stop 2023-10-30T13:28:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3740b3110 msgr2=0x7fc374157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3740b3110 0x7fc374157520 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e580 tx=0x7fc36800ea40 comp rx=0 tx=0).stop 2023-10-30T13:28:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 shutdown_connections 2023-10-30T13:28:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3600664a0 0x7fc360068960 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:03.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3740b3110 0x7fc374157520 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:03.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3740b1960 0x7fc3741440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:03.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 --2- 172.21.15.148:0/670180064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc35c004d10 0x7fc3741f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:03.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 >> 172.21.15.148:0/670180064 conn(0x7fc37406bfe0 msgr2=0x7fc374159f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:03.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 shutdown_connections 2023-10-30T13:28:03.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:03.515+0000 7fc379068640 1 -- 172.21.15.148:0/670180064 wait complete. 2023-10-30T13:28:03.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:28:05.532 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:05.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.683+0000 7f2382b1d640 1 Processor -- start 2023-10-30T13:28:05.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.683+0000 7f2382b1d640 1 -- start start 2023-10-30T13:28:05.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f2382b1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c0c93d0 0x7f237c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:05.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f2382b1d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c06afa0 con 0x7f237c153d70 2023-10-30T13:28:05.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f2382b1d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c154150 con 0x7f237c06e470 2023-10-30T13:28:05.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f2382b1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c06d780 con 0x7f237c0c93d0 2023-10-30T13:28:05.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f23808b9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f237c153d70 0x7f237c0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58688/0 (socket says 172.21.15.148:58688) 2023-10-30T13:28:05.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f23808b9640 1 -- 172.21.15.148:0/4006129498 learned_addr learned my addr 172.21.15.148:0/4006129498 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:28:05.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f23810ba640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c0c93d0 0x7f237c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:05.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721127607 0 0) 0x7f237c06afa0 con 0x7f237c153d70 2023-10-30T13:28:05.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f236c003530 con 0x7f237c153d70 2023-10-30T13:28:05.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288716442 0 0) 0x7f236c003530 con 0x7f237c153d70 2023-10-30T13:28:05.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c0c93d0 msgr2=0x7f237c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:05.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c0c93d0 0x7f237c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 >> v1:172.21.15.150:6789/0 conn(0x7f237c06e470 legacy=0x7f237c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:05.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c06e030 con 0x7f237c153d70 2023-10-30T13:28:05.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f2370001680 con 0x7f237c153d70 2023-10-30T13:28:05.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 >> v1:172.21.15.148:6789/0 conn(0x7f237c153d70 legacy=0x7f237c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:05.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c0c93d0 0x7f236c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:05.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f236c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:05.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f236c004d10 0x7f236c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:05.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c06d780 con 0x7f236c003f30 2023-10-30T13:28:05.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c00ba70 con 0x7f236c004d10 2023-10-30T13:28:05.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c00bc30 con 0x7f237c0c93d0 2023-10-30T13:28:05.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237bfff640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f236c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:05.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f23808b9640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c0c93d0 0x7f236c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:05.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f23810ba640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f236c004d10 0x7f236c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:05.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237bfff640 1 -- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c0c93d0 msgr2=0x7f236c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:05.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237bfff640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c0c93d0 0x7f236c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237bfff640 1 -- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f236c004d10 msgr2=0x7f236c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:05.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237bfff640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f236c004d10 0x7f236c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f237bfff640 1 -- 172.21.15.148:0/4006129498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c003530 con 0x7f236c003f30 2023-10-30T13:28:05.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.687+0000 7f23808b9640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c0c93d0 0x7f236c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:05.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f237bfff640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f236c0047d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2368001780 tx=0x7f236801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e98e05a13192f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:05.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23680052d0 con 0x7f236c003f30 2023-10-30T13:28:05.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23680a7050 con 0x7f236c003f30 2023-10-30T13:28:05.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f237b7fe640 1 -- 172.21.15.148:0/4006129498 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368003600 con 0x7f236c003f30 2023-10-30T13:28:05.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f2382b1d640 1 -- 172.21.15.148:0/4006129498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 msgr2=0x7f236c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:05.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f2382b1d640 1 --2- 172.21.15.148:0/4006129498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f236c0047d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2368001780 tx=0x7f236801c400 comp rx=0 tx=0).stop 2023-10-30T13:28:05.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f23810ba640 1 -- 172.21.15.148:0/4006129498 reap_dead start 2023-10-30T13:28:05.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f2382b1d640 1 -- 172.21.15.148:0/4006129498 shutdown_connections 2023-10-30T13:28:05.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f2382b1d640 1 -- 172.21.15.148:0/4006129498 >> 172.21.15.148:0/4006129498 conn(0x7f237c071ea0 msgr2=0x7f237c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:05.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f2382b1d640 1 -- 172.21.15.148:0/4006129498 shutdown_connections 2023-10-30T13:28:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f2382b1d640 1 -- 172.21.15.148:0/4006129498 wait complete. 2023-10-30T13:28:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f2382b1d640 1 Processor -- start 2023-10-30T13:28:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.691+0000 7f2382b1d640 1 -- start start 2023-10-30T13:28:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f2382b1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f237c0c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:05.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f2382b1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c1555d0 0x7f237c1f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f2382b1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f237c1f2f10 0x7f237c149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f2382b1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c154330 con 0x7f236c003f30 2023-10-30T13:28:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f2382b1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1e1fd0 con 0x7f237c1f2f10 2023-10-30T13:28:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f2382b1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1e22b0 con 0x7f237c1555d0 2023-10-30T13:28:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23808b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f237c0c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:05.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23808b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f237c0c77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54672/0 (socket says 172.21.15.148:54672) 2023-10-30T13:28:05.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23808b9640 1 -- 172.21.15.148:0/1770601892 learned_addr learned my addr 172.21.15.148:0/1770601892 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:05.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23810ba640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f237c1f2f10 0x7f237c149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:05.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f237bfff640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c1555d0 0x7f237c1f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:05.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23808b9640 1 -- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c1555d0 msgr2=0x7f237c1f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:05.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23808b9640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c1555d0 0x7f237c1f29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23808b9640 1 -- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f237c1f2f10 msgr2=0x7f237c149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:05.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23808b9640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f237c1f2f10 0x7f237c149890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23808b9640 1 -- 172.21.15.148:0/1770601892 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c1dfb50 con 0x7f236c003f30 2023-10-30T13:28:05.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23810ba640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f237c1f2f10 0x7f237c149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:28:05.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23808b9640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f237c0c77d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2370011810 tx=0x7f2370011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:05.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23797fa640 1 -- 172.21.15.148:0/1770601892 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237000fb00 con 0x7f236c003f30 2023-10-30T13:28:05.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23797fa640 1 -- 172.21.15.148:0/1770601892 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237000fc60 con 0x7f236c003f30 2023-10-30T13:28:05.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c154540 con 0x7f236c003f30 2023-10-30T13:28:05.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23797fa640 1 -- 172.21.15.148:0/1770601892 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370015410 con 0x7f236c003f30 2023-10-30T13:28:05.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c1e2740 con 0x7f236c003f30 2023-10-30T13:28:05.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c04e170 con 0x7f236c003f30 2023-10-30T13:28:05.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.695+0000 7f23797fa640 1 -- 172.21.15.148:0/1770601892 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2370015570 con 0x7f236c003f30 2023-10-30T13:28:05.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.699+0000 7f23797fa640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23600664a0 0x7f2360068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:05.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.699+0000 7f23797fa640 1 -- 172.21.15.148:0/1770601892 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f23700874d0 con 0x7f236c003f30 2023-10-30T13:28:05.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.699+0000 7f237bfff640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23600664a0 0x7f2360068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:05.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.699+0000 7f237bfff640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23600664a0 0x7f2360068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2368004000 tx=0x7f23680011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:05.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.703+0000 7f23797fa640 1 -- 172.21.15.148:0/1770601892 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2370054600 con 0x7f236c003f30 2023-10-30T13:28:05.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f237c1549f0 con 0x7f236c003f30 2023-10-30T13:28:05.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f23797fa640 1 -- 172.21.15.148:0/1770601892 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f2370057c20 con 0x7f236c003f30 2023-10-30T13:28:05.881 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:05.881 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6249,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:05.882 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 81 2023-10-30T13:28:05.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23600664a0 msgr2=0x7f2360068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:05.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23600664a0 0x7f2360068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2368004000 tx=0x7f23680011a0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 msgr2=0x7f237c0c77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:05.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f237c0c77d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2370011810 tx=0x7f2370011cd0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 shutdown_connections 2023-10-30T13:28:05.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23600664a0 0x7f2360068960 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f237c1f2f10 0x7f237c149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f237c1555d0 0x7f237c1f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 --2- 172.21.15.148:0/1770601892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f236c003f30 0x7f237c0c77d0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:05.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 >> 172.21.15.148:0/1770601892 conn(0x7f237c071ea0 msgr2=0x7f237c152f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:05.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 shutdown_connections 2023-10-30T13:28:05.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:05.883+0000 7f2382b1d640 1 -- 172.21.15.148:0/1770601892 wait complete. 2023-10-30T13:28:05.902 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:28:07.902 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:08.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.050+0000 7f183153e640 1 Processor -- start 2023-10-30T13:28:08.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f183153e640 1 -- start start 2023-10-30T13:28:08.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f183153e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f182c0c6b80 0x7f182c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:08.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f183153e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f182c13f4a0 0x7f182c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:08.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f183153e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182c0c0a50 con 0x7f182c0c6f60 2023-10-30T13:28:08.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f183153e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182c06f6e0 con 0x7f182c0c6b80 2023-10-30T13:28:08.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f183153e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182c06f9c0 con 0x7f182c13f4a0 2023-10-30T13:28:08.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f182affd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f182c0c6b80 0x7f182c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:08.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f182affd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f182c0c6b80 0x7f182c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54682/0 (socket says 172.21.15.148:54682) 2023-10-30T13:28:08.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f182affd640 1 -- 172.21.15.148:0/858671846 learned_addr learned my addr 172.21.15.148:0/858671846 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:08.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f182b7fe640 1 --2- 172.21.15.148:0/858671846 >> v2:172.21.15.150:3300/0 conn(0x7f182c13f4a0 0x7f182c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:08.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f182affd640 1 -- 172.21.15.148:0/858671846 >> v2:172.21.15.150:3300/0 conn(0x7f182c13f4a0 msgr2=0x7f182c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:08.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f182affd640 1 --2- 172.21.15.148:0/858671846 >> v2:172.21.15.150:3300/0 conn(0x7f182c13f4a0 0x7f182c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f182affd640 1 -- 172.21.15.148:0/858671846 >> v1:172.21.15.150:6789/0 conn(0x7f182c0c6f60 legacy=0x7f182c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:08.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.054+0000 7f182affd640 1 -- 172.21.15.148:0/858671846 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182c06fca0 con 0x7f182c0c6b80 2023-10-30T13:28:08.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f182affd640 1 --2- 172.21.15.148:0/858671846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c0c6b80 0x7f182c0bfec0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f182c0c8980 tx=0x7f181c01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8128d2e639baf883 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:08.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f182a7fc640 1 -- 172.21.15.148:0/858671846 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c0ac630 con 0x7f182c0c6b80 2023-10-30T13:28:08.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f182a7fc640 1 -- 172.21.15.148:0/858671846 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c0ac790 con 0x7f182c0c6b80 2023-10-30T13:28:08.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f182a7fc640 1 -- 172.21.15.148:0/858671846 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c0acb20 con 0x7f182c0c6b80 2023-10-30T13:28:08.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 -- 172.21.15.148:0/858671846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c0c6b80 msgr2=0x7f182c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:08.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 --2- 172.21.15.148:0/858671846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c0c6b80 0x7f182c0bfec0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f182c0c8980 tx=0x7f181c01c4c0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 -- 172.21.15.148:0/858671846 shutdown_connections 2023-10-30T13:28:08.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 --2- 172.21.15.148:0/858671846 >> v2:172.21.15.150:3300/0 conn(0x7f182c13f4a0 0x7f182c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 --2- 172.21.15.148:0/858671846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c0c6b80 0x7f182c0bfec0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 -- 172.21.15.148:0/858671846 >> 172.21.15.148:0/858671846 conn(0x7f182c0c7910 msgr2=0x7f182c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:08.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 -- 172.21.15.148:0/858671846 shutdown_connections 2023-10-30T13:28:08.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 -- 172.21.15.148:0/858671846 wait complete. 2023-10-30T13:28:08.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 Processor -- start 2023-10-30T13:28:08.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.058+0000 7f183153e640 1 -- start start 2023-10-30T13:28:08.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f183153e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f182c0c6b80 0x7f182c06a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:08.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f183153e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f182c0c6f60 0x7f182c157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:08.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f183153e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c13f4a0 0x7f182c157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:08.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f183153e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c153890 con 0x7f182c13f4a0 2023-10-30T13:28:08.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f183153e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c13d8a0 con 0x7f182c0c6b80 2023-10-30T13:28:08.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f183153e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182c1333d0 con 0x7f182c0c6f60 2023-10-30T13:28:08.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c13f4a0 0x7f182c157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:08.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c13f4a0 0x7f182c157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54698/0 (socket says 172.21.15.148:54698) 2023-10-30T13:28:08.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182b7fe640 1 -- 172.21.15.148:0/1122176430 learned_addr learned my addr 172.21.15.148:0/1122176430 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:08.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182affd640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f182c0c6b80 0x7f182c06a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:08.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f1823fff640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f182c0c6f60 0x7f182c157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:08.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182b7fe640 1 -- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f182c0c6f60 msgr2=0x7f182c157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:08.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182b7fe640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f182c0c6f60 0x7f182c157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182b7fe640 1 -- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f182c0c6b80 msgr2=0x7f182c06a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:08.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182b7fe640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f182c0c6b80 0x7f182c06a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182b7fe640 1 -- 172.21.15.148:0/1122176430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c0094d0 con 0x7f182c13f4a0 2023-10-30T13:28:08.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182affd640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f182c0c6b80 0x7f182c06a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:08.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f182b7fe640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c13f4a0 0x7f182c157cd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1824007e60 tx=0x7f182400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:08.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f18237fe640 1 -- 172.21.15.148:0/1122176430 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824014420 con 0x7f182c13f4a0 2023-10-30T13:28:08.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f18237fe640 1 -- 172.21.15.148:0/1122176430 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18240145a0 con 0x7f182c13f4a0 2023-10-30T13:28:08.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f18237fe640 1 -- 172.21.15.148:0/1122176430 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824014950 con 0x7f182c13f4a0 2023-10-30T13:28:08.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182c156020 con 0x7f182c13f4a0 2023-10-30T13:28:08.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182c1562a0 con 0x7f182c13f4a0 2023-10-30T13:28:08.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.062+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182c04e170 con 0x7f182c13f4a0 2023-10-30T13:28:08.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.066+0000 7f18237fe640 1 -- 172.21.15.148:0/1122176430 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1824014ab0 con 0x7f182c13f4a0 2023-10-30T13:28:08.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.066+0000 7f18237fe640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f180c0664a0 0x7f180c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:08.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.066+0000 7f18237fe640 1 -- 172.21.15.148:0/1122176430 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f1824084b80 con 0x7f182c13f4a0 2023-10-30T13:28:08.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.066+0000 7f182affd640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f180c0664a0 0x7f180c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:08.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.066+0000 7f182affd640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f180c0664a0 0x7f180c068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f181c0ae000 tx=0x7f181c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:08.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.070+0000 7f18237fe640 1 -- 172.21.15.148:0/1122176430 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1824051c30 con 0x7f182c13f4a0 2023-10-30T13:28:08.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.246+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f182c1598d0 con 0x7f182c13f4a0 2023-10-30T13:28:08.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f18237fe640 1 -- 172.21.15.148:0/1122176430 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f1824055250 con 0x7f182c13f4a0 2023-10-30T13:28:08.248 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:08.249 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6249,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:08.249 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 81 2023-10-30T13:28:08.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f180c0664a0 msgr2=0x7f180c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:08.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f180c0664a0 0x7f180c068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f181c0ae000 tx=0x7f181c008040 comp rx=0 tx=0).stop 2023-10-30T13:28:08.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c13f4a0 msgr2=0x7f182c157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:08.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c13f4a0 0x7f182c157cd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1824007e60 tx=0x7f182400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 shutdown_connections 2023-10-30T13:28:08.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f180c0664a0 0x7f180c068960 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f182c13f4a0 0x7f182c157cd0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f182c0c6f60 0x7f182c157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 --2- 172.21.15.148:0/1122176430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f182c0c6b80 0x7f182c06a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:08.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 >> 172.21.15.148:0/1122176430 conn(0x7f182c0c7910 msgr2=0x7f182c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:08.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 shutdown_connections 2023-10-30T13:28:08.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:08.250+0000 7f183153e640 1 -- 172.21.15.148:0/1122176430 wait complete. 2023-10-30T13:28:08.269 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:28:10.270 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:10.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8644d80640 1 Processor -- start 2023-10-30T13:28:10.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8644d80640 1 -- start start 2023-10-30T13:28:10.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8644d80640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f864015d6f0 0x7f864015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:10.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8644d80640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 0x7f86400cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:10.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8644d80640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86400cb800 con 0x7f86401538a0 2023-10-30T13:28:10.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8644d80640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86400cb330 con 0x7f864015d6f0 2023-10-30T13:28:10.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8644d80640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86400cad70 con 0x7f86400cf160 2023-10-30T13:28:10.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f863e59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f86401538a0 0x7f86400ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58702/0 (socket says 172.21.15.148:58702) 2023-10-30T13:28:10.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f863e59c640 1 -- 172.21.15.148:0/1406027206 learned_addr learned my addr 172.21.15.148:0/1406027206 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:28:10.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8635d9b640 1 --2- 172.21.15.148:0/1406027206 >> v2:172.21.15.148:3300/0 conn(0x7f864015d6f0 0x7f864015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:10.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f863ed9d640 1 --2- 172.21.15.148:0/1406027206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 0x7f86400cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:10.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8635d9b640 1 -- 172.21.15.148:0/1406027206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 msgr2=0x7f86400cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:10.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8635d9b640 1 --2- 172.21.15.148:0/1406027206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 0x7f86400cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:10.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8635d9b640 1 -- 172.21.15.148:0/1406027206 >> v1:172.21.15.148:6789/0 conn(0x7f86401538a0 legacy=0x7f86400ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:10.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.422+0000 7f8635d9b640 1 -- 172.21.15.148:0/1406027206 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400cb050 con 0x7f864015d6f0 2023-10-30T13:28:10.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f8635d9b640 1 --2- 172.21.15.148:0/1406027206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f864015d6f0 0x7f864015a090 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f86300097c0 tx=0x7f863001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ffc27c0a766e97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:10.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f863dd9b640 1 -- 172.21.15.148:0/1406027206 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86300a8970 con 0x7f864015d6f0 2023-10-30T13:28:10.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f863dd9b640 1 -- 172.21.15.148:0/1406027206 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86300a8ad0 con 0x7f864015d6f0 2023-10-30T13:28:10.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f863dd9b640 1 -- 172.21.15.148:0/1406027206 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86300a8e60 con 0x7f864015d6f0 2023-10-30T13:28:10.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f8644d80640 1 -- 172.21.15.148:0/1406027206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f864015d6f0 msgr2=0x7f864015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:10.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f8644d80640 1 --2- 172.21.15.148:0/1406027206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f864015d6f0 0x7f864015a090 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f86300097c0 tx=0x7f863001ec70 comp rx=0 tx=0).stop 2023-10-30T13:28:10.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f8644d80640 1 -- 172.21.15.148:0/1406027206 shutdown_connections 2023-10-30T13:28:10.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f8644d80640 1 --2- 172.21.15.148:0/1406027206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f864015d6f0 0x7f864015a090 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:10.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f8644d80640 1 --2- 172.21.15.148:0/1406027206 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 0x7f86400cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:10.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f8644d80640 1 -- 172.21.15.148:0/1406027206 >> 172.21.15.148:0/1406027206 conn(0x7f864006bfe0 msgr2=0x7f864006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:10.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f8644d80640 1 -- 172.21.15.148:0/1406027206 shutdown_connections 2023-10-30T13:28:10.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.426+0000 7f8644d80640 1 -- 172.21.15.148:0/1406027206 wait complete. 2023-10-30T13:28:10.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8644d80640 1 Processor -- start 2023-10-30T13:28:10.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8644d80640 1 -- start start 2023-10-30T13:28:10.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8644d80640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 0x7f8640159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:10.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8644d80640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86401538a0 0x7f8640148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:10.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8644d80640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f864015d6f0 0x7f86400c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:10.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8644d80640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8640158f60 con 0x7f86401538a0 2023-10-30T13:28:10.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8644d80640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86401e4660 con 0x7f864015d6f0 2023-10-30T13:28:10.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8644d80640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8640158a20 con 0x7f86400cf160 2023-10-30T13:28:10.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8635d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86401538a0 0x7f8640148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:10.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8635d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86401538a0 0x7f8640148a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54720/0 (socket says 172.21.15.148:54720) 2023-10-30T13:28:10.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8635d9b640 1 -- 172.21.15.148:0/1295222075 learned_addr learned my addr 172.21.15.148:0/1295222075 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:10.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f863e59c640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 0x7f8640159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:10.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f863ed9d640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f864015d6f0 0x7f86400c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:10.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8635d9b640 1 -- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 msgr2=0x7f8640159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:10.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8635d9b640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 0x7f8640159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:10.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8635d9b640 1 -- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f864015d6f0 msgr2=0x7f86400c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:10.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8635d9b640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f864015d6f0 0x7f86400c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:10.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8635d9b640 1 -- 172.21.15.148:0/1295222075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630009420 con 0x7f86401538a0 2023-10-30T13:28:10.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f863e59c640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 0x7f8640159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:10.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f8635d9b640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86401538a0 0x7f8640148a20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f863001c8a0 tx=0x7f863001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:10.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f86377fe640 1 -- 172.21.15.148:0/1295222075 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86300b9630 con 0x7f86401538a0 2023-10-30T13:28:10.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f86377fe640 1 -- 172.21.15.148:0/1295222075 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86300b97b0 con 0x7f86401538a0 2023-10-30T13:28:10.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.430+0000 7f86377fe640 1 -- 172.21.15.148:0/1295222075 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86300b9b60 con 0x7f86401538a0 2023-10-30T13:28:10.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.434+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86400c6950 con 0x7f86401538a0 2023-10-30T13:28:10.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.434+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8640158460 con 0x7f86401538a0 2023-10-30T13:28:10.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.434+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86400b2a50 con 0x7f86401538a0 2023-10-30T13:28:10.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.434+0000 7f86377fe640 1 -- 172.21.15.148:0/1295222075 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f86300b7040 con 0x7f86401538a0 2023-10-30T13:28:10.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.434+0000 7f86377fe640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86200664a0 0x7f8620068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:10.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.434+0000 7f86377fe640 1 -- 172.21.15.148:0/1295222075 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f8630122130 con 0x7f86401538a0 2023-10-30T13:28:10.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.434+0000 7f863e59c640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86200664a0 0x7f8620068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:10.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.438+0000 7f863e59c640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86200664a0 0x7f8620068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8628005890 tx=0x7f8628009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:10.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.438+0000 7f86377fe640 1 -- 172.21.15.148:0/1295222075 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f86300ef1d0 con 0x7f86401538a0 2023-10-30T13:28:10.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.618+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f864004e170 con 0x7f86401538a0 2023-10-30T13:28:10.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.618+0000 7f86377fe640 1 -- 172.21.15.148:0/1295222075 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f86300f27f0 con 0x7f86401538a0 2023-10-30T13:28:10.619 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:10.619 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6249,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:10.619 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 81 2023-10-30T13:28:10.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86200664a0 msgr2=0x7f8620068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:10.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86200664a0 0x7f8620068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8628005890 tx=0x7f8628009040 comp rx=0 tx=0).stop 2023-10-30T13:28:10.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86401538a0 msgr2=0x7f8640148a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:10.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86401538a0 0x7f8640148a20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f863001c8a0 tx=0x7f863001ba10 comp rx=0 tx=0).stop 2023-10-30T13:28:10.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 shutdown_connections 2023-10-30T13:28:10.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f86200664a0 0x7f8620068960 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:10.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86401538a0 0x7f8640148a20 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:10.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f864015d6f0 0x7f86400c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:10.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 --2- 172.21.15.148:0/1295222075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f86400cf160 0x7f8640159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:10.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 >> 172.21.15.148:0/1295222075 conn(0x7f864006bfe0 msgr2=0x7f86400cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:10.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 shutdown_connections 2023-10-30T13:28:10.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:10.622+0000 7f8644d80640 1 -- 172.21.15.148:0/1295222075 wait complete. 2023-10-30T13:28:10.640 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:28:12.641 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:12.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.790+0000 7f7bcaa17640 1 Processor -- start 2023-10-30T13:28:12.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.790+0000 7f7bcaa17640 1 -- start start 2023-10-30T13:28:12.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bcaa17640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7bc40c6f60 0x7f7bc41566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:12.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bcaa17640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc40c0a50 con 0x7f7bc40c05d0 2023-10-30T13:28:12.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bcaa17640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc406f6e0 con 0x7f7bc40c6b80 2023-10-30T13:28:12.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bcaa17640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc406f9c0 con 0x7f7bc40c6f60 2023-10-30T13:28:12.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bc8fb4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7bc40c6f60 0x7f7bc41566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:12.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bc8fb4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7bc40c6f60 0x7f7bc41566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33844/0 (socket says 172.21.15.148:33844) 2023-10-30T13:28:12.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bc8fb4640 1 -- 172.21.15.148:0/684597894 learned_addr learned my addr 172.21.15.148:0/684597894 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:12.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bc8fb4640 1 -- 172.21.15.148:0/684597894 >> v1:172.21.15.150:6789/0 conn(0x7f7bc40c6b80 legacy=0x7f7bc40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:12.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bc8fb4640 1 -- 172.21.15.148:0/684597894 >> v1:172.21.15.148:6789/0 conn(0x7f7bc40c05d0 legacy=0x7f7bc413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:12.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.794+0000 7f7bc8fb4640 1 -- 172.21.15.148:0/684597894 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc406fca0 con 0x7f7bc40c6f60 2023-10-30T13:28:12.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bc8fb4640 1 --2- 172.21.15.148:0/684597894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6f60 0x7f7bc41566a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc009e50 tx=0x7f7bbc01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64057ac0b767abfc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:12.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bc2ffd640 1 -- 172.21.15.148:0/684597894 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc0a57f0 con 0x7f7bc40c6f60 2023-10-30T13:28:12.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bc2ffd640 1 -- 172.21.15.148:0/684597894 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc0a5950 con 0x7f7bc40c6f60 2023-10-30T13:28:12.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bc2ffd640 1 -- 172.21.15.148:0/684597894 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc0a5ce0 con 0x7f7bc40c6f60 2023-10-30T13:28:12.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bcaa17640 1 -- 172.21.15.148:0/684597894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6f60 msgr2=0x7f7bc41566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:12.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bcaa17640 1 --2- 172.21.15.148:0/684597894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6f60 0x7f7bc41566a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc009e50 tx=0x7f7bbc01ca10 comp rx=0 tx=0).stop 2023-10-30T13:28:12.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bcaa17640 1 -- 172.21.15.148:0/684597894 shutdown_connections 2023-10-30T13:28:12.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bcaa17640 1 --2- 172.21.15.148:0/684597894 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6f60 0x7f7bc41566a0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:12.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bcaa17640 1 -- 172.21.15.148:0/684597894 >> 172.21.15.148:0/684597894 conn(0x7f7bc40c7910 msgr2=0x7f7bc40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:12.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bcaa17640 1 -- 172.21.15.148:0/684597894 shutdown_connections 2023-10-30T13:28:12.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bcaa17640 1 -- 172.21.15.148:0/684597894 wait complete. 2023-10-30T13:28:12.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bcaa17640 1 Processor -- start 2023-10-30T13:28:12.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.798+0000 7f7bcaa17640 1 -- start start 2023-10-30T13:28:12.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bcaa17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bc40c05d0 0x7f7bc40c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:12.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bcaa17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6b80 0x7f7bc40ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:12.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bcaa17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bc40c6f60 0x7f7bc41392f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:12.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bcaa17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc41faa10 con 0x7f7bc40c6b80 2023-10-30T13:28:12.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bcaa17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc41facb0 con 0x7f7bc40c6f60 2023-10-30T13:28:12.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bcaa17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc41fa740 con 0x7f7bc40c05d0 2023-10-30T13:28:12.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6b80 0x7f7bc40ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:12.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6b80 0x7f7bc40ca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33854/0 (socket says 172.21.15.148:33854) 2023-10-30T13:28:12.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc37fe640 1 -- 172.21.15.148:0/3155648421 learned_addr learned my addr 172.21.15.148:0/3155648421 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:12.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc8fb4640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bc40c6f60 0x7f7bc41392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:12.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc3fff640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bc40c05d0 0x7f7bc40c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:12.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc37fe640 1 -- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bc40c05d0 msgr2=0x7f7bc40c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:12.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc37fe640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bc40c05d0 0x7f7bc40c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:12.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc37fe640 1 -- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bc40c6f60 msgr2=0x7f7bc41392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:12.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc37fe640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bc40c6f60 0x7f7bc41392f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:12.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc37fe640 1 -- 172.21.15.148:0/3155648421 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc009b30 con 0x7f7bc40c6b80 2023-10-30T13:28:12.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc37fe640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6b80 0x7f7bc40ca2d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800c4d0 tx=0x7f7bb800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:12.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc0ff9640 1 -- 172.21.15.148:0/3155648421 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb80052d0 con 0x7f7bc40c6b80 2023-10-30T13:28:12.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc0ff9640 1 -- 172.21.15.148:0/3155648421 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb8005430 con 0x7f7bc40c6b80 2023-10-30T13:28:12.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc0ff9640 1 -- 172.21.15.148:0/3155648421 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8012830 con 0x7f7bc40c6b80 2023-10-30T13:28:12.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc41f6a00 con 0x7f7bc40c6b80 2023-10-30T13:28:12.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc41f6fc0 con 0x7f7bc40c6b80 2023-10-30T13:28:12.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc404e170 con 0x7f7bc40c6b80 2023-10-30T13:28:12.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.802+0000 7f7bc0ff9640 1 -- 172.21.15.148:0/3155648421 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7bb8012990 con 0x7f7bc40c6b80 2023-10-30T13:28:12.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.806+0000 7f7bc0ff9640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bb00664a0 0x7f7bb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:12.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.806+0000 7f7bc0ff9640 1 -- 172.21.15.148:0/3155648421 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6542+0+0 (secure 0 0 0) 0x7f7bb8083df0 con 0x7f7bc40c6b80 2023-10-30T13:28:12.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.806+0000 7f7bc3fff640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bb00664a0 0x7f7bb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:12.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.806+0000 7f7bc3fff640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bb00664a0 0x7f7bb0068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4000f70 tx=0x7f7bb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:12.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.810+0000 7f7bc0ff9640 1 -- 172.21.15.148:0/3155648421 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7bb8050e90 con 0x7f7bc40c6b80 2023-10-30T13:28:12.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.990+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bc406bc00 con 0x7f7bc40c6b80 2023-10-30T13:28:12.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.990+0000 7f7bc0ff9640 1 -- 172.21.15.148:0/3155648421 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f7bb80544b0 con 0x7f7bc40c6b80 2023-10-30T13:28:12.992 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:12.992 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6249,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:27:10.731972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6099},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6099":{"gid":6099,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":35,"addr":"172.21.15.148:6835/2733100982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":2733100982},{"type":"v1","addr":"172.21.15.148:6835","nonce":2733100982}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:12.992 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 81 2023-10-30T13:28:12.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bb00664a0 msgr2=0x7f7bb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:12.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bb00664a0 0x7f7bb0068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4000f70 tx=0x7f7bb4009040 comp rx=0 tx=0).stop 2023-10-30T13:28:12.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6b80 msgr2=0x7f7bc40ca2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:12.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6b80 0x7f7bc40ca2d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800c4d0 tx=0x7f7bb800c990 comp rx=0 tx=0).stop 2023-10-30T13:28:12.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 shutdown_connections 2023-10-30T13:28:12.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7bb00664a0 0x7f7bb0068960 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:12.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7bc40c6b80 0x7f7bc40ca2d0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:12.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7bc40c6f60 0x7f7bc41392f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:12.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 --2- 172.21.15.148:0/3155648421 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7bc40c05d0 0x7f7bc40c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:12.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 >> 172.21.15.148:0/3155648421 conn(0x7f7bc40c7910 msgr2=0x7f7bc41532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:12.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 shutdown_connections 2023-10-30T13:28:12.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:12.994+0000 7f7bcaa17640 1 -- 172.21.15.148:0/3155648421 wait complete. 2023-10-30T13:28:13.013 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:28:15.014 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:15.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a977640 1 Processor -- start 2023-10-30T13:28:15.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a977640 1 -- start start 2023-10-30T13:28:15.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a977640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8c140cefa0 0x7f8c140cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a977640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1415a6c0 con 0x7f8c141538f0 2023-10-30T13:28:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a977640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c140ca6a0 con 0x7f8c1415cd50 2023-10-30T13:28:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a977640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c140cf8f0 con 0x7f8c140cefa0 2023-10-30T13:28:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a176640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8c140cefa0 0x7f8c140cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a176640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8c140cefa0 0x7f8c140cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33870/0 (socket says 172.21.15.148:33870) 2023-10-30T13:28:15.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a176640 1 -- 172.21.15.148:0/3742140593 learned_addr learned my addr 172.21.15.148:0/3742140593 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a176640 1 -- 172.21.15.148:0/3742140593 >> v1:172.21.15.150:6789/0 conn(0x7f8c1415cd50 legacy=0x7f8c140c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a176640 1 -- 172.21.15.148:0/3742140593 >> v1:172.21.15.148:6789/0 conn(0x7f8c141538f0 legacy=0x7f8c14159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.166+0000 7f8c1a176640 1 -- 172.21.15.148:0/3742140593 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1415d300 con 0x7f8c140cefa0 2023-10-30T13:28:15.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c1a176640 1 --2- 172.21.15.148:0/3742140593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c140cefa0 0x7f8c140cf3b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c009e50 tx=0x7f8c0c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6561bb88d138bf31 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:15.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c18973640 1 -- 172.21.15.148:0/3742140593 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0af400 con 0x7f8c140cefa0 2023-10-30T13:28:15.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c18973640 1 -- 172.21.15.148:0/3742140593 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0c0ab050 con 0x7f8c140cefa0 2023-10-30T13:28:15.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c18973640 1 -- 172.21.15.148:0/3742140593 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0af780 con 0x7f8c140cefa0 2023-10-30T13:28:15.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c1a977640 1 -- 172.21.15.148:0/3742140593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c140cefa0 msgr2=0x7f8c140cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c1a977640 1 --2- 172.21.15.148:0/3742140593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c140cefa0 0x7f8c140cf3b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c009e50 tx=0x7f8c0c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c1a977640 1 -- 172.21.15.148:0/3742140593 shutdown_connections 2023-10-30T13:28:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c1a977640 1 --2- 172.21.15.148:0/3742140593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c140cefa0 0x7f8c140cf3b0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c1a977640 1 -- 172.21.15.148:0/3742140593 >> 172.21.15.148:0/3742140593 conn(0x7f8c1406a640 msgr2=0x7f8c14065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:15.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c1a977640 1 -- 172.21.15.148:0/3742140593 shutdown_connections 2023-10-30T13:28:15.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.170+0000 7f8c1a977640 1 -- 172.21.15.148:0/3742140593 wait complete. 2023-10-30T13:28:15.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 Processor -- start 2023-10-30T13:28:15.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 -- start start 2023-10-30T13:28:15.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c140cefa0 0x7f8c141efca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c141538f0 0x7f8c140b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c1415cd50 0x7f8c140b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141459a0 con 0x7f8c141538f0 2023-10-30T13:28:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c140b47a0 con 0x7f8c140cefa0 2023-10-30T13:28:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c141ef810 con 0x7f8c1415cd50 2023-10-30T13:28:15.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19174640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c141538f0 0x7f8c140b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19174640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c141538f0 0x7f8c140b3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33872/0 (socket says 172.21.15.148:33872) 2023-10-30T13:28:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19174640 1 -- 172.21.15.148:0/4212647931 learned_addr learned my addr 172.21.15.148:0/4212647931 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19975640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c140cefa0 0x7f8c141efca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a176640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c1415cd50 0x7f8c140b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19174640 1 -- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c1415cd50 msgr2=0x7f8c140b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19174640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c1415cd50 0x7f8c140b4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19174640 1 -- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c140cefa0 msgr2=0x7f8c141efca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19174640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c140cefa0 0x7f8c141efca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19174640 1 -- 172.21.15.148:0/4212647931 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c009b30 con 0x7f8c141538f0 2023-10-30T13:28:15.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c19174640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c141538f0 0x7f8c140b3050 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c3c0 tx=0x7f8c0400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c0a7fc640 1 -- 172.21.15.148:0/4212647931 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040052d0 con 0x7f8c141538f0 2023-10-30T13:28:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c0a7fc640 1 -- 172.21.15.148:0/4212647931 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04005430 con 0x7f8c141538f0 2023-10-30T13:28:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c0a7fc640 1 -- 172.21.15.148:0/4212647931 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04003840 con 0x7f8c141538f0 2023-10-30T13:28:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c140b49c0 con 0x7f8c141538f0 2023-10-30T13:28:15.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c140b4f80 con 0x7f8c141538f0 2023-10-30T13:28:15.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.174+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c141ef980 con 0x7f8c141538f0 2023-10-30T13:28:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.178+0000 7f8c0a7fc640 1 -- 172.21.15.148:0/4212647931 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c040039a0 con 0x7f8c141538f0 2023-10-30T13:28:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.178+0000 7f8c0a7fc640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bf80664a0 0x7f8bf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.178+0000 7f8c0a7fc640 1 -- 172.21.15.148:0/4212647931 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f8c04083760 con 0x7f8c141538f0 2023-10-30T13:28:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.178+0000 7f8c19975640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bf80664a0 0x7f8bf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.178+0000 7f8c19975640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bf80664a0 0x7f8bf8068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8c10007fd0 tx=0x7f8c10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:15.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.182+0000 7f8c0a7fc640 1 -- 172.21.15.148:0/4212647931 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c04050850 con 0x7f8c141538f0 2023-10-30T13:28:15.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.362+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c1404e170 con 0x7f8c141538f0 2023-10-30T13:28:15.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.362+0000 7f8c0a7fc640 1 -- 172.21.15.148:0/4212647931 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f8c04053e70 con 0x7f8c141538f0 2023-10-30T13:28:15.363 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:15.363 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:14.633448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:15.363 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 82 2023-10-30T13:28:15.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bf80664a0 msgr2=0x7f8bf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bf80664a0 0x7f8bf8068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8c10007fd0 tx=0x7f8c10009040 comp rx=0 tx=0).stop 2023-10-30T13:28:15.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c141538f0 msgr2=0x7f8c140b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c141538f0 0x7f8c140b3050 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c3c0 tx=0x7f8c0400c880 comp rx=0 tx=0).stop 2023-10-30T13:28:15.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 shutdown_connections 2023-10-30T13:28:15.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8bf80664a0 0x7f8bf8068960 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c141538f0 0x7f8c140b3050 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8c1415cd50 0x7f8c140b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 --2- 172.21.15.148:0/4212647931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c140cefa0 0x7f8c141efca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 >> 172.21.15.148:0/4212647931 conn(0x7f8c1406a640 msgr2=0x7f8c140cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:15.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 shutdown_connections 2023-10-30T13:28:15.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.366+0000 7f8c1a977640 1 -- 172.21.15.148:0/4212647931 wait complete. 2023-10-30T13:28:15.383 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:28:15.383 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:28:15.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:15.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.522+0000 7f724ab9d640 1 Processor -- start 2023-10-30T13:28:15.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.522+0000 7f724ab9d640 1 -- start start 2023-10-30T13:28:15.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.522+0000 7f724ab9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f72440c5f60 0x7f7244071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.522+0000 7f724ab9d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72440700b0 con 0x7f72440c6fe0 2023-10-30T13:28:15.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f724ab9d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72440c6530 con 0x7f7244153bf0 2023-10-30T13:28:15.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f724ab9d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72440c6810 con 0x7f72440c5f60 2023-10-30T13:28:15.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f724913a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f72440c5f60 0x7f7244071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f724913a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f72440c5f60 0x7f7244071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33880/0 (socket says 172.21.15.148:33880) 2023-10-30T13:28:15.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f724913a640 1 -- 172.21.15.148:0/2236291334 learned_addr learned my addr 172.21.15.148:0/2236291334 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:15.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f724913a640 1 -- 172.21.15.148:0/2236291334 >> v1:172.21.15.150:6789/0 conn(0x7f7244153bf0 legacy=0x7f72440c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f724913a640 1 -- 172.21.15.148:0/2236291334 >> v1:172.21.15.148:6789/0 conn(0x7f72440c6fe0 legacy=0x7f72440cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f724913a640 1 -- 172.21.15.148:0/2236291334 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7244159b10 con 0x7f72440c5f60 2023-10-30T13:28:15.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f724913a640 1 --2- 172.21.15.148:0/2236291334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 0x7f7244071260 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f723c009e50 tx=0x7f723c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa94d9ef9a90a28c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:15.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f723b7fe640 1 -- 172.21.15.148:0/2236291334 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723c0af400 con 0x7f72440c5f60 2023-10-30T13:28:15.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f723b7fe640 1 -- 172.21.15.148:0/2236291334 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723c0ab050 con 0x7f72440c5f60 2023-10-30T13:28:15.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.526+0000 7f723b7fe640 1 -- 172.21.15.148:0/2236291334 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723c0af780 con 0x7f72440c5f60 2023-10-30T13:28:15.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 -- 172.21.15.148:0/2236291334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 msgr2=0x7f7244071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 --2- 172.21.15.148:0/2236291334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 0x7f7244071260 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f723c009e50 tx=0x7f723c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 -- 172.21.15.148:0/2236291334 shutdown_connections 2023-10-30T13:28:15.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 --2- 172.21.15.148:0/2236291334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 0x7f7244071260 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 -- 172.21.15.148:0/2236291334 >> 172.21.15.148:0/2236291334 conn(0x7f724406c6e0 msgr2=0x7f7244069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:15.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 -- 172.21.15.148:0/2236291334 shutdown_connections 2023-10-30T13:28:15.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 -- 172.21.15.148:0/2236291334 wait complete. 2023-10-30T13:28:15.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 Processor -- start 2023-10-30T13:28:15.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 -- start start 2023-10-30T13:28:15.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 0x7f72441564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72440c6fe0 0x7f72440c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7244153bf0 0x7f7244157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7244157310 con 0x7f72440c5f60 2023-10-30T13:28:15.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7244156a30 con 0x7f7244153bf0 2023-10-30T13:28:15.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724ab9d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7244157770 con 0x7f72440c6fe0 2023-10-30T13:28:15.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f7248939640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 0x7f72441564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f7248939640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 0x7f72441564f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33888/0 (socket says 172.21.15.148:33888) 2023-10-30T13:28:15.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f7248939640 1 -- 172.21.15.148:0/4137362130 learned_addr learned my addr 172.21.15.148:0/4137362130 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:15.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f724913a640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7244153bf0 0x7f7244157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.530+0000 7f723bfff640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72440c6fe0 0x7f72440c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f7248939640 1 -- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72440c6fe0 msgr2=0x7f72440c4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f7248939640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72440c6fe0 0x7f72440c4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f7248939640 1 -- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7244153bf0 msgr2=0x7f7244157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f7248939640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7244153bf0 0x7f7244157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f7248939640 1 -- 172.21.15.148:0/4137362130 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c009b30 con 0x7f72440c5f60 2023-10-30T13:28:15.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f723bfff640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72440c6fe0 0x7f72440c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:15.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f7248939640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 0x7f72441564f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f724000c530 tx=0x7f724000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:15.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f72397fa640 1 -- 172.21.15.148:0/4137362130 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240005220 con 0x7f72440c5f60 2023-10-30T13:28:15.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f72397fa640 1 -- 172.21.15.148:0/4137362130 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7240005380 con 0x7f72440c5f60 2023-10-30T13:28:15.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724414d060 con 0x7f72440c5f60 2023-10-30T13:28:15.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f72397fa640 1 -- 172.21.15.148:0/4137362130 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240007270 con 0x7f72440c5f60 2023-10-30T13:28:15.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7244157050 con 0x7f72440c5f60 2023-10-30T13:28:15.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724404e170 con 0x7f72440c5f60 2023-10-30T13:28:15.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f72397fa640 1 -- 172.21.15.148:0/4137362130 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f72400073d0 con 0x7f72440c5f60 2023-10-30T13:28:15.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f72397fa640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72300664a0 0x7f7230068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:15.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f72397fa640 1 -- 172.21.15.148:0/4137362130 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f7240083ab0 con 0x7f72440c5f60 2023-10-30T13:28:15.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.534+0000 7f723bfff640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72300664a0 0x7f7230068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:15.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.538+0000 7f723bfff640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72300664a0 0x7f7230068960 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f72441f01a0 tx=0x7f7234009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:15.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.538+0000 7f72397fa640 1 -- 172.21.15.148:0/4137362130 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7240050b80 con 0x7f72440c5f60 2023-10-30T13:28:15.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f724406fe90 con 0x7f72440c5f60 2023-10-30T13:28:15.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f72397fa640 1 -- 172.21.15.148:0/4137362130 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f72400541a0 con 0x7f72440c5f60 2023-10-30T13:28:15.721 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:15.721 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:14.633448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:15.722 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 82 2023-10-30T13:28:15.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72300664a0 msgr2=0x7f7230068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72300664a0 0x7f7230068960 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f72441f01a0 tx=0x7f7234009040 comp rx=0 tx=0).stop 2023-10-30T13:28:15.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 msgr2=0x7f72441564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:15.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 0x7f72441564f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f724000c530 tx=0x7f724000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 shutdown_connections 2023-10-30T13:28:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72300664a0 0x7f7230068960 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7244153bf0 0x7f7244157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72440c6fe0 0x7f72440c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 --2- 172.21.15.148:0/4137362130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72440c5f60 0x7f72441564f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:15.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 >> 172.21.15.148:0/4137362130 conn(0x7f724406c6e0 msgr2=0x7f7244152b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:15.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 shutdown_connections 2023-10-30T13:28:15.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:15.722+0000 7f724ab9d640 1 -- 172.21.15.148:0/4137362130 wait complete. 2023-10-30T13:28:15.742 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 82, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 82, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6249": { "addr": "172.21.15.148:6837/2740075781", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 2740075781, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 2740075781, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6249, "incarnation": 82, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:28:14.633448+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6249 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:28:17.743 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:17.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f2224a17640 1 Processor -- start 2023-10-30T13:28:17.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f2224a17640 1 -- start start 2023-10-30T13:28:17.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f2224a17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22201538a0 0x7f222015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:17.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f2224a17640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22200cb800 con 0x7f22200ca0e0 2023-10-30T13:28:17.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f2224a17640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222015dc30 con 0x7f22200cf160 2023-10-30T13:28:17.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f2224a17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22200cb330 con 0x7f22201538a0 2023-10-30T13:28:17.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221dd9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f22200ca0e0 0x7f222015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41864/0 (socket says 172.21.15.148:41864) 2023-10-30T13:28:17.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221dd9b640 1 -- 172.21.15.148:0/2005907074 learned_addr learned my addr 172.21.15.148:0/2005907074 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:28:17.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221e59c640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22201538a0 0x7f222015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:17.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361827208 0 0) 0x7f22200cb800 con 0x7f22200ca0e0 2023-10-30T13:28:17.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2208003530 con 0x7f22200ca0e0 2023-10-30T13:28:17.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2525926741 0 0) 0x7f2208003530 con 0x7f22200ca0e0 2023-10-30T13:28:17.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22201538a0 msgr2=0x7f222015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:17.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22201538a0 0x7f222015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:17.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 >> v1:172.21.15.150:6789/0 conn(0x7f22200cf160 legacy=0x7f2220164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:17.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22200cad70 con 0x7f22200ca0e0 2023-10-30T13:28:17.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f2214001680 con 0x7f22200ca0e0 2023-10-30T13:28:17.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 >> v1:172.21.15.148:6789/0 conn(0x7f22200ca0e0 legacy=0x7f222015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:17.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22080039b0 0x7f2208003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:17.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2208004360 0x7f2208004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:17.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22201538a0 0x7f2208005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:17.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22200cb330 con 0x7f2208004360 2023-10-30T13:28:17.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220800b990 con 0x7f22080039b0 2023-10-30T13:28:17.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220800bb50 con 0x7f22201538a0 2023-10-30T13:28:17.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221dd9b640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2208004360 0x7f2208004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:17.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221e59c640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22080039b0 0x7f2208003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:17.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221ed9d640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22201538a0 0x7f2208005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:17.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221dd9b640 1 -- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22201538a0 msgr2=0x7f2208005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:17.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221dd9b640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22201538a0 0x7f2208005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:17.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221dd9b640 1 -- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22080039b0 msgr2=0x7f2208003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:17.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221dd9b640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22080039b0 0x7f2208003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:17.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221dd9b640 1 -- 172.21.15.148:0/2005907074 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2208003530 con 0x7f2208004360 2023-10-30T13:28:17.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.894+0000 7f221ed9d640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22201538a0 0x7f2208005090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:17.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f221dd9b640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2208004360 0x7f2208004830 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f221400c1c0 tx=0x7f221401f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73552959eb01f03 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:17.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214005ae0 con 0x7f2208004360 2023-10-30T13:28:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22140ab070 con 0x7f2208004360 2023-10-30T13:28:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f221d59a640 1 -- 172.21.15.148:0/2005907074 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214005e80 con 0x7f2208004360 2023-10-30T13:28:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f2224a17640 1 -- 172.21.15.148:0/2005907074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2208004360 msgr2=0x7f2208004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f2224a17640 1 --2- 172.21.15.148:0/2005907074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2208004360 0x7f2208004830 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f221400c1c0 tx=0x7f221401f390 comp rx=0 tx=0).stop 2023-10-30T13:28:17.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f221ed9d640 1 -- 172.21.15.148:0/2005907074 reap_dead start 2023-10-30T13:28:17.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f2224a17640 1 -- 172.21.15.148:0/2005907074 shutdown_connections 2023-10-30T13:28:17.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f2224a17640 1 -- 172.21.15.148:0/2005907074 >> 172.21.15.148:0/2005907074 conn(0x7f222006bfe0 msgr2=0x7f222006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:17.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f2224a17640 1 -- 172.21.15.148:0/2005907074 shutdown_connections 2023-10-30T13:28:17.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.898+0000 7f2224a17640 1 -- 172.21.15.148:0/2005907074 wait complete. 2023-10-30T13:28:17.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f2224a17640 1 Processor -- start 2023-10-30T13:28:17.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f2224a17640 1 -- start start 2023-10-30T13:28:17.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f2224a17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2208004360 0x7f22200b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:17.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f2224a17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22201592c0 0x7f22201e6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:17.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f2224a17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2220157520 0x7f22200b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:17.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f2224a17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22201f2e50 con 0x7f22201592c0 2023-10-30T13:28:17.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f2224a17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22200b51c0 con 0x7f2220157520 2023-10-30T13:28:17.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f2224a17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22200cebc0 con 0x7f2208004360 2023-10-30T13:28:17.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22201592c0 0x7f22201e6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:17.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22201592c0 0x7f22201e6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33916/0 (socket says 172.21.15.148:33916) 2023-10-30T13:28:17.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221dd9b640 1 -- 172.21.15.148:0/3026737386 learned_addr learned my addr 172.21.15.148:0/3026737386 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:17.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221e59c640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2208004360 0x7f22200b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:17.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221ed9d640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2220157520 0x7f22200b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:17.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221dd9b640 1 -- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2208004360 msgr2=0x7f22200b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:17.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221dd9b640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2208004360 0x7f22200b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:17.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221dd9b640 1 -- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2220157520 msgr2=0x7f22200b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:17.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221dd9b640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2220157520 0x7f22200b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:17.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221dd9b640 1 -- 172.21.15.148:0/3026737386 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22200b44d0 con 0x7f22201592c0 2023-10-30T13:28:17.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f221dd9b640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22201592c0 0x7f22201e6480 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f221401f1c0 tx=0x7f2208005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:17.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f220effd640 1 -- 172.21.15.148:0/3026737386 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214001640 con 0x7f22201592c0 2023-10-30T13:28:17.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f220effd640 1 -- 172.21.15.148:0/3026737386 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2214007070 con 0x7f22201592c0 2023-10-30T13:28:17.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22200ce890 con 0x7f22201592c0 2023-10-30T13:28:17.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.902+0000 7f220effd640 1 -- 172.21.15.148:0/3026737386 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214007400 con 0x7f22201592c0 2023-10-30T13:28:17.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.906+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22201f3040 con 0x7f22201592c0 2023-10-30T13:28:17.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.906+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22200ced70 con 0x7f22201592c0 2023-10-30T13:28:17.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.906+0000 7f220effd640 1 -- 172.21.15.148:0/3026737386 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2214007560 con 0x7f22201592c0 2023-10-30T13:28:17.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.906+0000 7f220effd640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22000664a0 0x7f2200068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:17.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.906+0000 7f220effd640 1 -- 172.21.15.148:0/3026737386 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f22140b7040 con 0x7f22201592c0 2023-10-30T13:28:17.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.906+0000 7f221e59c640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22000664a0 0x7f2200068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:17.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.906+0000 7f221e59c640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22000664a0 0x7f2200068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2210009be0 tx=0x7f2210009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:17.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:17.910+0000 7f220effd640 1 -- 172.21.15.148:0/3026737386 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f22200ced70 con 0x7f22201592c0 2023-10-30T13:28:18.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.102+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f222004e170 con 0x7f22201592c0 2023-10-30T13:28:18.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.102+0000 7f220effd640 1 -- 172.21.15.148:0/3026737386 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f22140ee6d0 con 0x7f22201592c0 2023-10-30T13:28:18.102 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:18.102 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:14.633448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:18.102 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 82 2023-10-30T13:28:18.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22000664a0 msgr2=0x7f2200068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:18.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22000664a0 0x7f2200068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2210009be0 tx=0x7f2210009190 comp rx=0 tx=0).stop 2023-10-30T13:28:18.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22201592c0 msgr2=0x7f22201e6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:18.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22201592c0 0x7f22201e6480 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f221401f1c0 tx=0x7f2208005800 comp rx=0 tx=0).stop 2023-10-30T13:28:18.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 shutdown_connections 2023-10-30T13:28:18.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22000664a0 0x7f2200068960 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:18.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22201592c0 0x7f22201e6480 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:18.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2220157520 0x7f22200b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:18.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 --2- 172.21.15.148:0/3026737386 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2208004360 0x7f22200b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:18.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 >> 172.21.15.148:0/3026737386 conn(0x7f222006bfe0 msgr2=0x7f22201512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:18.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 shutdown_connections 2023-10-30T13:28:18.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:18.106+0000 7f2224a17640 1 -- 172.21.15.148:0/3026737386 wait complete. 2023-10-30T13:28:18.123 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:28:20.124 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:20.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87cd79d640 1 Processor -- start 2023-10-30T13:28:20.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87cd79d640 1 -- start start 2023-10-30T13:28:20.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87cd79d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f87c815d6f0 0x7f87c815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:20.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87cd79d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c80cf160 0x7f87c80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:20.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87cd79d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c80cb800 con 0x7f87c81538a0 2023-10-30T13:28:20.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87cd79d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c80cb330 con 0x7f87c815d6f0 2023-10-30T13:28:20.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87cd79d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c80cad70 con 0x7f87c80cf160 2023-10-30T13:28:20.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87be7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f87c815d6f0 0x7f87c815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:20.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87c77fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c80cf160 0x7f87c80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:20.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87be7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f87c815d6f0 0x7f87c815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:56218/0 (socket says 172.21.15.148:56218) 2023-10-30T13:28:20.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87be7fc640 1 -- 172.21.15.148:0/3004087383 learned_addr learned my addr 172.21.15.148:0/3004087383 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:20.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87be7fc640 1 -- 172.21.15.148:0/3004087383 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c80cf160 msgr2=0x7f87c80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:20.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87be7fc640 1 --2- 172.21.15.148:0/3004087383 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c80cf160 0x7f87c80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87be7fc640 1 -- 172.21.15.148:0/3004087383 >> v1:172.21.15.150:6789/0 conn(0x7f87c81538a0 legacy=0x7f87c80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:20.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87be7fc640 1 -- 172.21.15.148:0/3004087383 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c80cb050 con 0x7f87c815d6f0 2023-10-30T13:28:20.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.274+0000 7f87c77fe640 1 --2- 172.21.15.148:0/3004087383 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c80cf160 0x7f87c80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:28:20.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87be7fc640 1 --2- 172.21.15.148:0/3004087383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87c815d6f0 0x7f87c815a090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f87b0009e50 tx=0x7f87b001f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2950fcb462e25a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:20.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87c67fc640 1 -- 172.21.15.148:0/3004087383 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b00af400 con 0x7f87c815d6f0 2023-10-30T13:28:20.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87c67fc640 1 -- 172.21.15.148:0/3004087383 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b00ab050 con 0x7f87c815d6f0 2023-10-30T13:28:20.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87c67fc640 1 -- 172.21.15.148:0/3004087383 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b00af780 con 0x7f87c815d6f0 2023-10-30T13:28:20.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87cd79d640 1 -- 172.21.15.148:0/3004087383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87c815d6f0 msgr2=0x7f87c815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:20.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87cd79d640 1 --2- 172.21.15.148:0/3004087383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87c815d6f0 0x7f87c815a090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f87b0009e50 tx=0x7f87b001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87cd79d640 1 -- 172.21.15.148:0/3004087383 shutdown_connections 2023-10-30T13:28:20.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87cd79d640 1 --2- 172.21.15.148:0/3004087383 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c80cf160 0x7f87c80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87cd79d640 1 --2- 172.21.15.148:0/3004087383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87c815d6f0 0x7f87c815a090 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87cd79d640 1 -- 172.21.15.148:0/3004087383 >> 172.21.15.148:0/3004087383 conn(0x7f87c806bfe0 msgr2=0x7f87c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:20.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87cd79d640 1 -- 172.21.15.148:0/3004087383 shutdown_connections 2023-10-30T13:28:20.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.278+0000 7f87cd79d640 1 -- 172.21.15.148:0/3004087383 wait complete. 2023-10-30T13:28:20.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 Processor -- start 2023-10-30T13:28:20.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 -- start start 2023-10-30T13:28:20.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87c80cf160 0x7f87c80c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:20.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c81538a0 0x7f87c81e38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:20.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87c815d6f0 0x7f87c81e3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:20.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c81f00e0 con 0x7f87c815d6f0 2023-10-30T13:28:20.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c80c9580 con 0x7f87c80cf160 2023-10-30T13:28:20.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c81efb80 con 0x7f87c81538a0 2023-10-30T13:28:20.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87c815d6f0 0x7f87c81e3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:20.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87c815d6f0 0x7f87c81e3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33922/0 (socket says 172.21.15.148:33922) 2023-10-30T13:28:20.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c77fe640 1 -- 172.21.15.148:0/1975961238 learned_addr learned my addr 172.21.15.148:0/1975961238 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:20.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c6ffd640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87c80cf160 0x7f87c80c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:20.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87be7fc640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c81538a0 0x7f87c81e38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:20.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c77fe640 1 -- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c81538a0 msgr2=0x7f87c81e38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:20.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c77fe640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c81538a0 0x7f87c81e38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c77fe640 1 -- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87c80cf160 msgr2=0x7f87c80c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:20.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c77fe640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87c80cf160 0x7f87c80c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c77fe640 1 -- 172.21.15.148:0/1975961238 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c0009420 con 0x7f87c815d6f0 2023-10-30T13:28:20.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87c77fe640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87c815d6f0 0x7f87c81e3de0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f87c000e930 tx=0x7f87c000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:20.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87bffff640 1 -- 172.21.15.148:0/1975961238 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c0009670 con 0x7f87c815d6f0 2023-10-30T13:28:20.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87bffff640 1 -- 172.21.15.148:0/1975961238 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c0004020 con 0x7f87c815d6f0 2023-10-30T13:28:20.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87bffff640 1 -- 172.21.15.148:0/1975961238 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c00043b0 con 0x7f87c815d6f0 2023-10-30T13:28:20.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b0009b30 con 0x7f87c815d6f0 2023-10-30T13:28:20.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.282+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c81efe20 con 0x7f87c815d6f0 2023-10-30T13:28:20.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.286+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c80b2a50 con 0x7f87c815d6f0 2023-10-30T13:28:20.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.286+0000 7f87bffff640 1 -- 172.21.15.148:0/1975961238 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87c000c940 con 0x7f87c815d6f0 2023-10-30T13:28:20.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.286+0000 7f87bffff640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87a80664a0 0x7f87a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:20.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.286+0000 7f87bffff640 1 -- 172.21.15.148:0/1975961238 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f87c0084260 con 0x7f87c815d6f0 2023-10-30T13:28:20.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.286+0000 7f87c6ffd640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87a80664a0 0x7f87a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:20.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.286+0000 7f87c6ffd640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87a80664a0 0x7f87a8068960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f87b8000f70 tx=0x7f87b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:20.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.290+0000 7f87bffff640 1 -- 172.21.15.148:0/1975961238 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87c0051190 con 0x7f87c815d6f0 2023-10-30T13:28:20.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.470+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87c804e170 con 0x7f87c815d6f0 2023-10-30T13:28:20.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.470+0000 7f87bffff640 1 -- 172.21.15.148:0/1975961238 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f87c00547b0 con 0x7f87c815d6f0 2023-10-30T13:28:20.470 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:20.470 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:14.633448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:20.470 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 82 2023-10-30T13:28:20.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.470+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87a80664a0 msgr2=0x7f87a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:20.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.470+0000 7f87cd79d640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87a80664a0 0x7f87a8068960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f87b8000f70 tx=0x7f87b8009040 comp rx=0 tx=0).stop 2023-10-30T13:28:20.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.470+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87c815d6f0 msgr2=0x7f87c81e3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:20.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.470+0000 7f87cd79d640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87c815d6f0 0x7f87c81e3de0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f87c000e930 tx=0x7f87c000edf0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.474+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 shutdown_connections 2023-10-30T13:28:20.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.474+0000 7f87cd79d640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87a80664a0 0x7f87a8068960 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.474+0000 7f87cd79d640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87c815d6f0 0x7f87c81e3de0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.474+0000 7f87cd79d640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87c81538a0 0x7f87c81e38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.474+0000 7f87cd79d640 1 --2- 172.21.15.148:0/1975961238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87c80cf160 0x7f87c80c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:20.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.474+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 >> 172.21.15.148:0/1975961238 conn(0x7f87c806bfe0 msgr2=0x7f87c8152790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:20.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.474+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 shutdown_connections 2023-10-30T13:28:20.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:20.474+0000 7f87cd79d640 1 -- 172.21.15.148:0/1975961238 wait complete. 2023-10-30T13:28:20.491 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:28:22.492 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:22.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.646+0000 7f82b0b1d640 1 Processor -- start 2023-10-30T13:28:22.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.646+0000 7f82b0b1d640 1 -- start start 2023-10-30T13:28:22.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.646+0000 7f82b0b1d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f82ac0c6b80 0x7f82ac0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:22.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.646+0000 7f82b0b1d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82ac13f4a0 0x7f82ac0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:22.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.646+0000 7f82b0b1d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac0c0a50 con 0x7f82ac0c6f60 2023-10-30T13:28:22.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.646+0000 7f82b0b1d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac06f6e0 con 0x7f82ac13f4a0 2023-10-30T13:28:22.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.646+0000 7f82b0b1d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac06f9c0 con 0x7f82ac0c6b80 2023-10-30T13:28:22.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.650+0000 7f82aad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82ac13f4a0 0x7f82ac0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:22.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.650+0000 7f82aad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f82ac13f4a0 0x7f82ac0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45076/0 (socket says 172.21.15.148:45076) 2023-10-30T13:28:22.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.650+0000 7f82aad9d640 1 -- 172.21.15.148:0/3586533256 learned_addr learned my addr 172.21.15.148:0/3586533256 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:22.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.650+0000 7f82aa59c640 1 --2- 172.21.15.148:0/3586533256 >> v2:172.21.15.150:3300/0 conn(0x7f82ac0c6b80 0x7f82ac0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:22.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.650+0000 7f82aad9d640 1 -- 172.21.15.148:0/3586533256 >> v2:172.21.15.150:3300/0 conn(0x7f82ac0c6b80 msgr2=0x7f82ac0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:22.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.650+0000 7f82aad9d640 1 --2- 172.21.15.148:0/3586533256 >> v2:172.21.15.150:3300/0 conn(0x7f82ac0c6b80 0x7f82ac0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.650+0000 7f82aad9d640 1 -- 172.21.15.148:0/3586533256 >> v1:172.21.15.150:6789/0 conn(0x7f82ac0c6f60 legacy=0x7f82ac13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:22.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.650+0000 7f82aad9d640 1 -- 172.21.15.148:0/3586533256 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac06fca0 con 0x7f82ac13f4a0 2023-10-30T13:28:22.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82aad9d640 1 --2- 172.21.15.148:0/3586533256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 0x7f82ac0c0400 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f82ac06f1b0 tx=0x7f82a401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f84e5053d9c767c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:22.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82a959a640 1 -- 172.21.15.148:0/3586533256 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a40a57a0 con 0x7f82ac13f4a0 2023-10-30T13:28:22.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82a959a640 1 -- 172.21.15.148:0/3586533256 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a40a5900 con 0x7f82ac13f4a0 2023-10-30T13:28:22.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82a959a640 1 -- 172.21.15.148:0/3586533256 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a40a5c90 con 0x7f82ac13f4a0 2023-10-30T13:28:22.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82b0b1d640 1 -- 172.21.15.148:0/3586533256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 msgr2=0x7f82ac0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:22.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82b0b1d640 1 --2- 172.21.15.148:0/3586533256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 0x7f82ac0c0400 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f82ac06f1b0 tx=0x7f82a401c9d0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82b0b1d640 1 -- 172.21.15.148:0/3586533256 shutdown_connections 2023-10-30T13:28:22.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82b0b1d640 1 --2- 172.21.15.148:0/3586533256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 0x7f82ac0c0400 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82b0b1d640 1 --2- 172.21.15.148:0/3586533256 >> v2:172.21.15.150:3300/0 conn(0x7f82ac0c6b80 0x7f82ac0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82b0b1d640 1 -- 172.21.15.148:0/3586533256 >> 172.21.15.148:0/3586533256 conn(0x7f82ac0c7910 msgr2=0x7f82ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:22.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82b0b1d640 1 -- 172.21.15.148:0/3586533256 shutdown_connections 2023-10-30T13:28:22.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.654+0000 7f82b0b1d640 1 -- 172.21.15.148:0/3586533256 wait complete. 2023-10-30T13:28:22.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82b0b1d640 1 Processor -- start 2023-10-30T13:28:22.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82b0b1d640 1 -- start start 2023-10-30T13:28:22.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82b0b1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82ac0c6b80 0x7f82ac131760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:22.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82b0b1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac0c6f60 0x7f82ac0c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:22.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82b0b1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 0x7f82ac1325e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:22.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82b0b1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac1f3b00 con 0x7f82ac13f4a0 2023-10-30T13:28:22.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82b0b1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac131ca0 con 0x7f82ac0c6f60 2023-10-30T13:28:22.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82b0b1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0ca4f0 con 0x7f82ac0c6b80 2023-10-30T13:28:22.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 0x7f82ac1325e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:22.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 0x7f82ac1325e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45086/0 (socket says 172.21.15.148:45086) 2023-10-30T13:28:22.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aad9d640 1 -- 172.21.15.148:0/2568960873 learned_addr learned my addr 172.21.15.148:0/2568960873 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:22.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82a9d9b640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac0c6f60 0x7f82ac0c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:22.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aa59c640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82ac0c6b80 0x7f82ac131760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:22.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aad9d640 1 -- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82ac0c6b80 msgr2=0x7f82ac131760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:22.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aad9d640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82ac0c6b80 0x7f82ac131760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aad9d640 1 -- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac0c6f60 msgr2=0x7f82ac0c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:22.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aad9d640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac0c6f60 0x7f82ac0c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aad9d640 1 -- 172.21.15.148:0/2568960873 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a4009b30 con 0x7f82ac13f4a0 2023-10-30T13:28:22.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.658+0000 7f82aad9d640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 0x7f82ac1325e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f82a40af040 tx=0x7f82a4019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:22.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.662+0000 7f829affd640 1 -- 172.21.15.148:0/2568960873 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a40b8880 con 0x7f82ac13f4a0 2023-10-30T13:28:22.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.662+0000 7f829affd640 1 -- 172.21.15.148:0/2568960873 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a40b8a00 con 0x7f82ac13f4a0 2023-10-30T13:28:22.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.662+0000 7f829affd640 1 -- 172.21.15.148:0/2568960873 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a40b8db0 con 0x7f82ac13f4a0 2023-10-30T13:28:22.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.662+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac131fc0 con 0x7f82ac13f4a0 2023-10-30T13:28:22.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.662+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac06a5d0 con 0x7f82ac13f4a0 2023-10-30T13:28:22.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.662+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac132300 con 0x7f82ac13f4a0 2023-10-30T13:28:22.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.662+0000 7f829affd640 1 -- 172.21.15.148:0/2568960873 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82a40c1410 con 0x7f82ac13f4a0 2023-10-30T13:28:22.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.662+0000 7f829affd640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f828c0664a0 0x7f828c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:22.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.662+0000 7f829affd640 1 -- 172.21.15.148:0/2568960873 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f82a411fa50 con 0x7f82ac13f4a0 2023-10-30T13:28:22.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.666+0000 7f82aa59c640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f828c0664a0 0x7f828c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:22.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.666+0000 7f82aa59c640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f828c0664a0 0x7f828c068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f829c009b60 tx=0x7f829c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:22.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.666+0000 7f829affd640 1 -- 172.21.15.148:0/2568960873 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82ac132300 con 0x7f82ac13f4a0 2023-10-30T13:28:22.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.874+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82ac1322e0 con 0x7f82ac13f4a0 2023-10-30T13:28:22.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.874+0000 7f829affd640 1 -- 172.21.15.148:0/2568960873 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f82a40ecd90 con 0x7f82ac13f4a0 2023-10-30T13:28:22.874 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:22.874 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:14.633448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:22.875 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 82 2023-10-30T13:28:22.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f828c0664a0 msgr2=0x7f828c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:22.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f828c0664a0 0x7f828c068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f829c009b60 tx=0x7f829c009190 comp rx=0 tx=0).stop 2023-10-30T13:28:22.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 msgr2=0x7f82ac1325e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:22.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 0x7f82ac1325e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f82a40af040 tx=0x7f82a4019770 comp rx=0 tx=0).stop 2023-10-30T13:28:22.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 shutdown_connections 2023-10-30T13:28:22.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f828c0664a0 0x7f828c068960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f82ac13f4a0 0x7f82ac1325e0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac0c6f60 0x7f82ac0c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 --2- 172.21.15.148:0/2568960873 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f82ac0c6b80 0x7f82ac131760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:22.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 >> 172.21.15.148:0/2568960873 conn(0x7f82ac0c7910 msgr2=0x7f82ac0c11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:22.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 shutdown_connections 2023-10-30T13:28:22.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:22.878+0000 7f82b0b1d640 1 -- 172.21.15.148:0/2568960873 wait complete. 2023-10-30T13:28:22.896 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:28:24.897 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:25.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.046+0000 7f2761b9d640 1 Processor -- start 2023-10-30T13:28:25.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.046+0000 7f2761b9d640 1 -- start start 2023-10-30T13:28:25.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f2761b9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f275c0bfec0 0x7f275c0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:25.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f2761b9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c06a720 0x7f275c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:25.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f2761b9d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275c071290 con 0x7f275c159e60 2023-10-30T13:28:25.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f2761b9d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f275c071570 con 0x7f275c0bfec0 2023-10-30T13:28:25.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f2761b9d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275c071850 con 0x7f275c06a720 2023-10-30T13:28:25.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f275c159e60 0x7f275c06f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47300/0 (socket says 172.21.15.148:47300) 2023-10-30T13:28:25.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275b7fe640 1 -- 172.21.15.148:0/3914282507 learned_addr learned my addr 172.21.15.148:0/3914282507 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:28:25.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f2753fff640 1 --2- 172.21.15.148:0/3914282507 >> v2:172.21.15.150:3300/0 conn(0x7f275c0bfec0 0x7f275c0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:25.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275bfff640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c06a720 0x7f275c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:25.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943199668 0 0) 0x7f275c071290 con 0x7f275c159e60 2023-10-30T13:28:25.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2748003530 con 0x7f275c159e60 2023-10-30T13:28:25.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3326603232 0 0) 0x7f2748003530 con 0x7f275c159e60 2023-10-30T13:28:25.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c06a720 msgr2=0x7f275c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:25.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c06a720 0x7f275c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 >> v2:172.21.15.150:3300/0 conn(0x7f275c0bfec0 msgr2=0x7f275c0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:25.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 --2- 172.21.15.148:0/3914282507 >> v2:172.21.15.150:3300/0 conn(0x7f275c0bfec0 0x7f275c0c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c071b30 con 0x7f275c159e60 2023-10-30T13:28:25.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f274c001680 con 0x7f275c159e60 2023-10-30T13:28:25.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 >> v1:172.21.15.148:6789/0 conn(0x7f275c159e60 legacy=0x7f275c06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:25.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c06a720 0x7f2748003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:25.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2748003f00 0x7f27480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:25.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f2748005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:25.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c071570 con 0x7f2748004d10 2023-10-30T13:28:25.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c071850 con 0x7f2748003f00 2023-10-30T13:28:25.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274800dbd0 con 0x7f275c06a720 2023-10-30T13:28:25.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275bfff640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f2748005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:25.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275b7fe640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c06a720 0x7f2748003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:25.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f2753fff640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2748003f00 0x7f27480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:25.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275bfff640 1 -- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c06a720 msgr2=0x7f2748003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:25.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275bfff640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c06a720 0x7f2748003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275bfff640 1 -- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2748003f00 msgr2=0x7f27480047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:25.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275bfff640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2748003f00 0x7f27480047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275bfff640 1 -- 172.21.15.148:0/3914282507 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2748003530 con 0x7f2748004d10 2023-10-30T13:28:25.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.050+0000 7f275b7fe640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c06a720 0x7f2748003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:25.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f275bfff640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f2748005190 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2754009fc0 tx=0x7f275401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbd3ee29be41dca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:25.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27540a5c70 con 0x7f2748004d10 2023-10-30T13:28:25.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275400f040 con 0x7f2748004d10 2023-10-30T13:28:25.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f275affd640 1 -- 172.21.15.148:0/3914282507 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27540ad440 con 0x7f2748004d10 2023-10-30T13:28:25.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f2761b9d640 1 -- 172.21.15.148:0/3914282507 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 msgr2=0x7f2748005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:25.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f2761b9d640 1 --2- 172.21.15.148:0/3914282507 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f2748005190 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2754009fc0 tx=0x7f275401cc60 comp rx=0 tx=0).stop 2023-10-30T13:28:25.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f275bfff640 1 -- 172.21.15.148:0/3914282507 reap_dead start 2023-10-30T13:28:25.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f2761b9d640 1 -- 172.21.15.148:0/3914282507 shutdown_connections 2023-10-30T13:28:25.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f2761b9d640 1 -- 172.21.15.148:0/3914282507 >> 172.21.15.148:0/3914282507 conn(0x7f275c0c7910 msgr2=0x7f275c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:25.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f2761b9d640 1 -- 172.21.15.148:0/3914282507 shutdown_connections 2023-10-30T13:28:25.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.054+0000 7f2761b9d640 1 -- 172.21.15.148:0/3914282507 wait complete. 2023-10-30T13:28:25.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 Processor -- start 2023-10-30T13:28:25.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 -- start start 2023-10-30T13:28:25.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f275c14f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:25.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f275c0c1940 0x7f275c1f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:25.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c0c6550 0x7f275c0c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:25.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c0c41b0 con 0x7f2748004d10 2023-10-30T13:28:25.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c0c4820 con 0x7f275c0c1940 2023-10-30T13:28:25.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275c0c2f30 con 0x7f275c0c6550 2023-10-30T13:28:25.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f275c14f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:25.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f275c14f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45106/0 (socket says 172.21.15.148:45106) 2023-10-30T13:28:25.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275b7fe640 1 -- 172.21.15.148:0/1138247016 learned_addr learned my addr 172.21.15.148:0/1138247016 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:25.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275bfff640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c0c6550 0x7f275c0c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:25.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2753fff640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f275c0c1940 0x7f275c1f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:25.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275b7fe640 1 -- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c0c6550 msgr2=0x7f275c0c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:25.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275b7fe640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c0c6550 0x7f275c0c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275b7fe640 1 -- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f275c0c1940 msgr2=0x7f275c1f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:25.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275b7fe640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f275c0c1940 0x7f275c1f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275b7fe640 1 -- 172.21.15.148:0/1138247016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c0ca840 con 0x7f2748004d10 2023-10-30T13:28:25.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f275b7fe640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f275c14f430 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f274c00e950 tx=0x7f274c00ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:25.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2758ff9640 1 -- 172.21.15.148:0/1138247016 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c004ac0 con 0x7f2748004d10 2023-10-30T13:28:25.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2758ff9640 1 -- 172.21.15.148:0/1138247016 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c004c20 con 0x7f2748004d10 2023-10-30T13:28:25.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275c1e6dd0 con 0x7f2748004d10 2023-10-30T13:28:25.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2758ff9640 1 -- 172.21.15.148:0/1138247016 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c005030 con 0x7f2748004d10 2023-10-30T13:28:25.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f275c1e7350 con 0x7f2748004d10 2023-10-30T13:28:25.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.058+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275c04e170 con 0x7f2748004d10 2023-10-30T13:28:25.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.062+0000 7f2758ff9640 1 -- 172.21.15.148:0/1138247016 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f274c005190 con 0x7f2748004d10 2023-10-30T13:28:25.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.062+0000 7f2758ff9640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f273c0664d0 0x7f273c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:25.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.062+0000 7f2758ff9640 1 -- 172.21.15.148:0/1138247016 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f274c083b70 con 0x7f2748004d10 2023-10-30T13:28:25.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.062+0000 7f2753fff640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f273c0664d0 0x7f273c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:25.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.062+0000 7f2753fff640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f273c0664d0 0x7f273c068990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f275c0c8270 tx=0x7f2744008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:25.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.066+0000 7f2758ff9640 1 -- 172.21.15.148:0/1138247016 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f274c050bd0 con 0x7f2748004d10 2023-10-30T13:28:25.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.246+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f275c0c1f10 con 0x7f2748004d10 2023-10-30T13:28:25.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.246+0000 7f2758ff9640 1 -- 172.21.15.148:0/1138247016 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f274c019030 con 0x7f2748004d10 2023-10-30T13:28:25.247 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:25.248 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:24.457905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:25.248 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 83 2023-10-30T13:28:25.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f273c0664d0 msgr2=0x7f273c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:25.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f273c0664d0 0x7f273c068990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f275c0c8270 tx=0x7f2744008040 comp rx=0 tx=0).stop 2023-10-30T13:28:25.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 msgr2=0x7f275c14f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:25.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f275c14f430 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f274c00e950 tx=0x7f274c00ee10 comp rx=0 tx=0).stop 2023-10-30T13:28:25.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 shutdown_connections 2023-10-30T13:28:25.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f273c0664d0 0x7f273c068990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f275c0c6550 0x7f275c0c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f275c0c1940 0x7f275c1f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 --2- 172.21.15.148:0/1138247016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2748004d10 0x7f275c14f430 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:25.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 >> 172.21.15.148:0/1138247016 conn(0x7f275c0c7910 msgr2=0x7f275c0c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:25.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 shutdown_connections 2023-10-30T13:28:25.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:25.250+0000 7f2761b9d640 1 -- 172.21.15.148:0/1138247016 wait complete. 2023-10-30T13:28:25.268 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:28:25.998 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:28:26.002 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:28:27.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:27.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb21e365640 1 Processor -- start 2023-10-30T13:28:27.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb21e365640 1 -- start start 2023-10-30T13:28:27.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb21e365640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2181539a0 0x7fb21815a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:27.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb21e365640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21815a8c0 con 0x7fb218069e90 2023-10-30T13:28:27.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb21e365640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218153d80 con 0x7fb218070a00 2023-10-30T13:28:27.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb21e365640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21806fbb0 con 0x7fb2181539a0 2023-10-30T13:28:27.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb21c902640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb218069e90 0x7fb21806a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47308/0 (socket says 172.21.15.148:47308) 2023-10-30T13:28:27.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb21c902640 1 -- 172.21.15.148:0/3106634727 learned_addr learned my addr 172.21.15.148:0/3106634727 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:28:27.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb217fff640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2181539a0 0x7fb21815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:27.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940000447 0 0) 0x7fb21815a8c0 con 0x7fb218069e90 2023-10-30T13:28:27.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb200003530 con 0x7fb218069e90 2023-10-30T13:28:27.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2287658096 0 0) 0x7fb200003530 con 0x7fb218069e90 2023-10-30T13:28:27.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2181539a0 msgr2=0x7fb21815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:27.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2181539a0 0x7fb21815a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:27.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 >> v1:172.21.15.150:6789/0 conn(0x7fb218070a00 legacy=0x7fb218070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:27.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2180c5c50 con 0x7fb218069e90 2023-10-30T13:28:27.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fb210001a00 con 0x7fb218069e90 2023-10-30T13:28:27.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 >> v1:172.21.15.148:6789/0 conn(0x7fb218069e90 legacy=0x7fb21806a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:27.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2181539a0 0x7fb2000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:27.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb200003ef0 0x7fb2000047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:27.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb200004d10 0x7fb200005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:27.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21806fbb0 con 0x7fb200003ef0 2023-10-30T13:28:27.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20000baa0 con 0x7fb200004d10 2023-10-30T13:28:27.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20000bc60 con 0x7fb2181539a0 2023-10-30T13:28:27.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.418+0000 7fb2177fe640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb200003ef0 0x7fb2000047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:27.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb217fff640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2181539a0 0x7fb2000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:27.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb21c902640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb200004d10 0x7fb200005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:27.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb2177fe640 1 -- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2181539a0 msgr2=0x7fb2000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:27.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb2177fe640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2181539a0 0x7fb2000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:27.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb2177fe640 1 -- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb200004d10 msgr2=0x7fb200005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:27.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb2177fe640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb200004d10 0x7fb200005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:27.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb2177fe640 1 -- 172.21.15.148:0/3106634727 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb200003530 con 0x7fb200003ef0 2023-10-30T13:28:27.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb2177fe640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb200003ef0 0x7fb2000047d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb20c001860 tx=0x7fb20c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e89ddb7606b109 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:27.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c0a8870 con 0x7fb200003ef0 2023-10-30T13:28:27.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c0a89d0 con 0x7fb200003ef0 2023-10-30T13:28:27.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb216ffd640 1 -- 172.21.15.148:0/3106634727 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c0a8dc0 con 0x7fb200003ef0 2023-10-30T13:28:27.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb21e365640 1 -- 172.21.15.148:0/3106634727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb200003ef0 msgr2=0x7fb2000047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:27.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb21e365640 1 --2- 172.21.15.148:0/3106634727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb200003ef0 0x7fb2000047d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb20c001860 tx=0x7fb20c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:28:27.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.422+0000 7fb21c902640 1 -- 172.21.15.148:0/3106634727 reap_dead start 2023-10-30T13:28:27.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 -- 172.21.15.148:0/3106634727 shutdown_connections 2023-10-30T13:28:27.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 -- 172.21.15.148:0/3106634727 >> 172.21.15.148:0/3106634727 conn(0x7fb2180c7910 msgr2=0x7fb2180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:27.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 -- 172.21.15.148:0/3106634727 shutdown_connections 2023-10-30T13:28:27.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 -- 172.21.15.148:0/3106634727 wait complete. 2023-10-30T13:28:27.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 Processor -- start 2023-10-30T13:28:27.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 -- start start 2023-10-30T13:28:27.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb200003ef0 0x7fb2180c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:27.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2181541f0 0x7fb2181f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:27.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2181548d0 0x7fb2181434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:27.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb218155820 con 0x7fb2181548d0 2023-10-30T13:28:27.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb218155180 con 0x7fb2181541f0 2023-10-30T13:28:27.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21e365640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21815ac40 con 0x7fb200003ef0 2023-10-30T13:28:27.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21c902640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2181548d0 0x7fb2181434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:27.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21c902640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2181548d0 0x7fb2181434d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45128/0 (socket says 172.21.15.148:45128) 2023-10-30T13:28:27.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21c902640 1 -- 172.21.15.148:0/1224705897 learned_addr learned my addr 172.21.15.148:0/1224705897 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:27.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb217fff640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb200003ef0 0x7fb2180c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:27.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb2177fe640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2181541f0 0x7fb2181f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:27.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21c902640 1 -- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb200003ef0 msgr2=0x7fb2180c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:27.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21c902640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb200003ef0 0x7fb2180c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:27.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21c902640 1 -- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2181541f0 msgr2=0x7fb2181f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:27.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21c902640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2181541f0 0x7fb2181f2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:27.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21c902640 1 -- 172.21.15.148:0/1224705897 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21815af20 con 0x7fb2181548d0 2023-10-30T13:28:27.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.426+0000 7fb21c902640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2181548d0 0x7fb2181434d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb21806aea0 tx=0x7fb21000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:27.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb214ff9640 1 -- 172.21.15.148:0/1224705897 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21000ec10 con 0x7fb2181548d0 2023-10-30T13:28:27.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb214ff9640 1 -- 172.21.15.148:0/1224705897 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21000ed90 con 0x7fb2181548d0 2023-10-30T13:28:27.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb214ff9640 1 -- 172.21.15.148:0/1224705897 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210013660 con 0x7fb2181548d0 2023-10-30T13:28:27.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb218155440 con 0x7fb2181548d0 2023-10-30T13:28:27.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21814d910 con 0x7fb2181548d0 2023-10-30T13:28:27.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb218155960 con 0x7fb2181548d0 2023-10-30T13:28:27.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb214ff9640 1 -- 172.21.15.148:0/1224705897 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb2100137c0 con 0x7fb2181548d0 2023-10-30T13:28:27.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb214ff9640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2040664a0 0x7fb204068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:27.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb214ff9640 1 -- 172.21.15.148:0/1224705897 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fb210084650 con 0x7fb2181548d0 2023-10-30T13:28:27.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.430+0000 7fb217fff640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2040664a0 0x7fb204068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:27.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.434+0000 7fb217fff640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2040664a0 0x7fb204068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb208009b20 tx=0x7fb208008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:27.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.434+0000 7fb214ff9640 1 -- 172.21.15.148:0/1224705897 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb218155960 con 0x7fb2181548d0 2023-10-30T13:28:27.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.614+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb218155960 con 0x7fb2181548d0 2023-10-30T13:28:27.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb214ff9640 1 -- 172.21.15.148:0/1224705897 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5720 (secure 0 0 0) 0x7fb218155960 con 0x7fb2181548d0 2023-10-30T13:28:27.616 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:27.616 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:24.457905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:27.616 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 83 2023-10-30T13:28:27.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2040664a0 msgr2=0x7fb204068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:27.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2040664a0 0x7fb204068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb208009b20 tx=0x7fb208008040 comp rx=0 tx=0).stop 2023-10-30T13:28:27.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2181548d0 msgr2=0x7fb2181434d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:27.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2181548d0 0x7fb2181434d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb21806aea0 tx=0x7fb21000e530 comp rx=0 tx=0).stop 2023-10-30T13:28:27.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 shutdown_connections 2023-10-30T13:28:27.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2040664a0 0x7fb204068960 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:27.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2181548d0 0x7fb2181434d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:27.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2181541f0 0x7fb2181f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:27.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 --2- 172.21.15.148:0/1224705897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb200003ef0 0x7fb2180c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:27.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 >> 172.21.15.148:0/1224705897 conn(0x7fb2180c7910 msgr2=0x7fb21814eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:27.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 shutdown_connections 2023-10-30T13:28:27.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:27.618+0000 7fb21e365640 1 -- 172.21.15.148:0/1224705897 wait complete. 2023-10-30T13:28:27.637 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 83, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 83, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6249": { "addr": "172.21.15.148:6837/2740075781", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 2740075781, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 2740075781, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6249, "incarnation": 82, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:28:24.457905+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6249 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:28:29.638 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:29.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cd0e46640 1 Processor -- start 2023-10-30T13:28:29.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cd0e46640 1 -- start start 2023-10-30T13:28:29.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cd0e46640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3ccc153bf0 0x7f3ccc071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:29.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cd0e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 0x7f3ccc0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:29.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cd0e46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ccc0c79e0 0x7f3ccc0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:29.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cd0e46640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ccc0c6810 con 0x7f3ccc153bf0 2023-10-30T13:28:29.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cd0e46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ccc159900 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cd0e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0700b0 con 0x7f3ccc0c6fe0 2023-10-30T13:28:29.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cca59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3ccc153bf0 0x7f3ccc071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:29.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cca59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3ccc153bf0 0x7f3ccc071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45136/0 (socket says 172.21.15.148:45136) 2023-10-30T13:28:29.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cca59c640 1 -- 172.21.15.148:0/3395265282 learned_addr learned my addr 172.21.15.148:0/3395265282 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:29.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.790+0000 7f3cc1d9b640 1 --2- 172.21.15.148:0/3395265282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 0x7f3ccc0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:29.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3ccad9d640 1 --2- 172.21.15.148:0/3395265282 >> v2:172.21.15.150:3300/0 conn(0x7f3ccc0c79e0 0x7f3ccc0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:29.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3cca59c640 1 -- 172.21.15.148:0/3395265282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 msgr2=0x7f3ccc0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3cca59c640 1 --2- 172.21.15.148:0/3395265282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 0x7f3ccc0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3cca59c640 1 -- 172.21.15.148:0/3395265282 >> v2:172.21.15.150:3300/0 conn(0x7f3ccc0c79e0 msgr2=0x7f3ccc0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3cca59c640 1 --2- 172.21.15.148:0/3395265282 >> v2:172.21.15.150:3300/0 conn(0x7f3ccc0c79e0 0x7f3ccc0cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3cca59c640 1 -- 172.21.15.148:0/3395265282 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc0c6530 con 0x7f3ccc153bf0 2023-10-30T13:28:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3cca59c640 1 --2- 172.21.15.148:0/3395265282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc153bf0 0x7f3ccc071260 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0c7400 tx=0x7f3cbc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f473ded9f818da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3cc9d9b640 1 -- 172.21.15.148:0/3395265282 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0a89b0 con 0x7f3ccc153bf0 2023-10-30T13:28:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3cc9d9b640 1 -- 172.21.15.148:0/3395265282 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cbc0a8b10 con 0x7f3ccc153bf0 2023-10-30T13:28:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.794+0000 7f3cc9d9b640 1 -- 172.21.15.148:0/3395265282 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0a8ea0 con 0x7f3ccc153bf0 2023-10-30T13:28:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 -- 172.21.15.148:0/3395265282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc153bf0 msgr2=0x7f3ccc071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/3395265282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc153bf0 0x7f3ccc071260 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0c7400 tx=0x7f3cbc01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 -- 172.21.15.148:0/3395265282 shutdown_connections 2023-10-30T13:28:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/3395265282 >> v2:172.21.15.150:3300/0 conn(0x7f3ccc0c79e0 0x7f3ccc0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/3395265282 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 0x7f3ccc0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/3395265282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc153bf0 0x7f3ccc071260 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 -- 172.21.15.148:0/3395265282 >> 172.21.15.148:0/3395265282 conn(0x7f3ccc06c6e0 msgr2=0x7f3ccc069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 -- 172.21.15.148:0/3395265282 shutdown_connections 2023-10-30T13:28:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 -- 172.21.15.148:0/3395265282 wait complete. 2023-10-30T13:28:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 Processor -- start 2023-10-30T13:28:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 -- start start 2023-10-30T13:28:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 0x7f3ccc154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc0c79e0 0x7f3ccc155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ccc153bf0 0x7f3ccc156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0c9840 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0cadd0 con 0x7f3ccc153bf0 2023-10-30T13:28:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cd0e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0c9580 con 0x7f3ccc0c6fe0 2023-10-30T13:28:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cc1d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc0c79e0 0x7f3ccc155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cc1d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc0c79e0 0x7f3ccc155c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45142/0 (socket says 172.21.15.148:45142) 2023-10-30T13:28:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cc1d9b640 1 -- 172.21.15.148:0/2215262118 learned_addr learned my addr 172.21.15.148:0/2215262118 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cca59c640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 0x7f3ccc154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3ccad9d640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ccc153bf0 0x7f3ccc156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cc1d9b640 1 -- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 msgr2=0x7f3ccc154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.798+0000 7f3cc1d9b640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 0x7f3ccc154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc1d9b640 1 -- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ccc153bf0 msgr2=0x7f3ccc156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc1d9b640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ccc153bf0 0x7f3ccc156160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc1d9b640 1 -- 172.21.15.148:0/2215262118 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc0094d0 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cca59c640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 0x7f3ccc154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc1d9b640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc0c79e0 0x7f3ccc155c20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400d830 tx=0x7f3cb400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc37fe640 1 -- 172.21.15.148:0/2215262118 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb40096c0 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc37fe640 1 -- 172.21.15.148:0/2215262118 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb4009820 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ccc0ca530 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc37fe640 1 -- 172.21.15.148:0/2215262118 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4009bd0 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc0caaf0 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc155940 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc37fe640 1 -- 172.21.15.148:0/2215262118 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3cb4009d30 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc37fe640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3cac0664a0 0x7f3cac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.802+0000 7f3cc37fe640 1 -- 172.21.15.148:0/2215262118 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f3cb4084a10 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.806+0000 7f3cca59c640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3cac0664a0 0x7f3cac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.806+0000 7f3cca59c640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3cac0664a0 0x7f3cac068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0b2000 tx=0x7f3cbc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.810+0000 7f3cc37fe640 1 -- 172.21.15.148:0/2215262118 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3cb4051a50 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ccc04e170 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cc37fe640 1 -- 172.21.15.148:0/2215262118 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f3cb400b440 con 0x7f3ccc0c79e0 2023-10-30T13:28:29.989 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:29.990 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:24.457905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:29.990 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 83 2023-10-30T13:28:29.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3cac0664a0 msgr2=0x7f3cac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:29.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3cac0664a0 0x7f3cac068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc0b2000 tx=0x7f3cbc008040 comp rx=0 tx=0).stop 2023-10-30T13:28:29.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc0c79e0 msgr2=0x7f3ccc155c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:29.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc0c79e0 0x7f3ccc155c20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400d830 tx=0x7f3cb400dcf0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 shutdown_connections 2023-10-30T13:28:29.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3cac0664a0 0x7f3cac068960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ccc0c79e0 0x7f3ccc155c20 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ccc153bf0 0x7f3ccc156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 --2- 172.21.15.148:0/2215262118 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ccc0c6fe0 0x7f3ccc154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:29.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 >> 172.21.15.148:0/2215262118 conn(0x7f3ccc06c6e0 msgr2=0x7f3ccc151aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:29.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 shutdown_connections 2023-10-30T13:28:29.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:29.990+0000 7f3cd0e46640 1 -- 172.21.15.148:0/2215262118 wait complete. 2023-10-30T13:28:30.010 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:28:32.011 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:32.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.162+0000 7fcc81209640 1 Processor -- start 2023-10-30T13:28:32.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.162+0000 7fcc81209640 1 -- start start 2023-10-30T13:28:32.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.162+0000 7fcc81209640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcc7c06f7a0 0x7fcc7c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:32.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.162+0000 7fcc81209640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c071110 0x7fcc7c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:32.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.162+0000 7fcc81209640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c066cf0 con 0x7fcc7c0c6bd0 2023-10-30T13:28:32.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.162+0000 7fcc81209640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c0700c0 con 0x7fcc7c06f7a0 2023-10-30T13:28:32.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.162+0000 7fcc81209640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0c70d0 con 0x7fcc7c071110 2023-10-30T13:28:32.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.162+0000 7fcc73fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fcc7c0c6bd0 0x7fcc7c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42490/0 (socket says 172.21.15.148:42490) 2023-10-30T13:28:32.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.162+0000 7fcc73fff640 1 -- 172.21.15.148:0/3687337786 learned_addr learned my addr 172.21.15.148:0/3687337786 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:28:32.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7ad9d640 1 --2- 172.21.15.148:0/3687337786 >> v2:172.21.15.150:3300/0 conn(0x7fcc7c06f7a0 0x7fcc7c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:32.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7b59e640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c071110 0x7fcc7c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:32.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085287481 0 0) 0x7fcc7c066cf0 con 0x7fcc7c0c6bd0 2023-10-30T13:28:32.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc68003530 con 0x7fcc7c0c6bd0 2023-10-30T13:28:32.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44148250 0 0) 0x7fcc68003530 con 0x7fcc7c0c6bd0 2023-10-30T13:28:32.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c071110 msgr2=0x7fcc7c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:32.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c071110 0x7fcc7c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 >> v2:172.21.15.150:3300/0 conn(0x7fcc7c06f7a0 msgr2=0x7fcc7c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:32.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 --2- 172.21.15.148:0/3687337786 >> v2:172.21.15.150:3300/0 conn(0x7fcc7c06f7a0 0x7fcc7c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c0c7390 con 0x7fcc7c0c6bd0 2023-10-30T13:28:32.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fcc64001680 con 0x7fcc7c0c6bd0 2023-10-30T13:28:32.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 >> v1:172.21.15.148:6789/0 conn(0x7fcc7c0c6bd0 legacy=0x7fcc7c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:32.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc68003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:32.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c071110 0x7fcc680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:32.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc68004d10 0x7fcc68005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:32.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0700c0 con 0x7fcc68003980 2023-10-30T13:28:32.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0c70d0 con 0x7fcc68004d10 2023-10-30T13:28:32.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6800b990 con 0x7fcc7c071110 2023-10-30T13:28:32.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7ad9d640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc68003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:32.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc73fff640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c071110 0x7fcc680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:32.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7b59e640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc68004d10 0x7fcc68005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:32.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7ad9d640 1 -- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c071110 msgr2=0x7fcc680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:32.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7ad9d640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c071110 0x7fcc680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7ad9d640 1 -- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc68004d10 msgr2=0x7fcc68005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:32.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7ad9d640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc68004d10 0x7fcc68005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.166+0000 7fcc7ad9d640 1 -- 172.21.15.148:0/3687337786 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc68003530 con 0x7fcc68003980 2023-10-30T13:28:32.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc7ad9d640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc68003da0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009ae0 tx=0x7fcc6c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e405bea65a3bca03 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:32.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0a5570 con 0x7fcc68003980 2023-10-30T13:28:32.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c0a56d0 con 0x7fcc68003980 2023-10-30T13:28:32.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc7a59c640 1 -- 172.21.15.148:0/3687337786 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0a5a80 con 0x7fcc68003980 2023-10-30T13:28:32.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc81209640 1 -- 172.21.15.148:0/3687337786 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 msgr2=0x7fcc68003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:32.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc81209640 1 --2- 172.21.15.148:0/3687337786 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc68003da0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009ae0 tx=0x7fcc6c01ca90 comp rx=0 tx=0).stop 2023-10-30T13:28:32.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc7b59e640 1 -- 172.21.15.148:0/3687337786 reap_dead start 2023-10-30T13:28:32.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc81209640 1 -- 172.21.15.148:0/3687337786 shutdown_connections 2023-10-30T13:28:32.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc81209640 1 -- 172.21.15.148:0/3687337786 >> 172.21.15.148:0/3687337786 conn(0x7fcc7c0c7910 msgr2=0x7fcc7c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:32.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc81209640 1 -- 172.21.15.148:0/3687337786 shutdown_connections 2023-10-30T13:28:32.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc81209640 1 -- 172.21.15.148:0/3687337786 wait complete. 2023-10-30T13:28:32.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc81209640 1 Processor -- start 2023-10-30T13:28:32.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.170+0000 7fcc81209640 1 -- start start 2023-10-30T13:28:32.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc81209640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc7c157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:32.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc81209640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc7c151ce0 0x7fcc7c1554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:32.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc81209640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c152eb0 0x7fcc7c153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:32.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc81209640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c152290 con 0x7fcc68003980 2023-10-30T13:28:32.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc81209640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c154ab0 con 0x7fcc7c151ce0 2023-10-30T13:28:32.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc81209640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c154d90 con 0x7fcc7c152eb0 2023-10-30T13:28:32.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc7c157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:32.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc7c157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54306/0 (socket says 172.21.15.148:54306) 2023-10-30T13:28:32.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7ad9d640 1 -- 172.21.15.148:0/1691375101 learned_addr learned my addr 172.21.15.148:0/1691375101 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:32.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc73fff640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc7c151ce0 0x7fcc7c1554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:32.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7b59e640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c152eb0 0x7fcc7c153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:32.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7ad9d640 1 -- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c152eb0 msgr2=0x7fcc7c153b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:32.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7ad9d640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c152eb0 0x7fcc7c153b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7ad9d640 1 -- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc7c151ce0 msgr2=0x7fcc7c1554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:32.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7ad9d640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc7c151ce0 0x7fcc7c1554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7ad9d640 1 -- 172.21.15.148:0/1691375101 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c155070 con 0x7fcc68003980 2023-10-30T13:28:32.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc7ad9d640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc7c157410 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009ae0 tx=0x7fcc68005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:32.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc737fe640 1 -- 172.21.15.148:0/1691375101 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c004220 con 0x7fcc68003980 2023-10-30T13:28:32.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc737fe640 1 -- 172.21.15.148:0/1691375101 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c004380 con 0x7fcc68003980 2023-10-30T13:28:32.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc737fe640 1 -- 172.21.15.148:0/1691375101 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c003820 con 0x7fcc68003980 2023-10-30T13:28:32.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c156c10 con 0x7fcc68003980 2023-10-30T13:28:32.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c0c6660 con 0x7fcc68003980 2023-10-30T13:28:32.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c04e170 con 0x7fcc68003980 2023-10-30T13:28:32.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.174+0000 7fcc737fe640 1 -- 172.21.15.148:0/1691375101 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcc6c0b4040 con 0x7fcc68003980 2023-10-30T13:28:32.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.178+0000 7fcc737fe640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc5c0664a0 0x7fcc5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:32.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.178+0000 7fcc737fe640 1 -- 172.21.15.148:0/1691375101 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fcc6c11ea90 con 0x7fcc68003980 2023-10-30T13:28:32.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.178+0000 7fcc73fff640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc5c0664a0 0x7fcc5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:32.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.178+0000 7fcc73fff640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc5c0664a0 0x7fcc5c068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcc64000c10 tx=0x7fcc64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:32.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.178+0000 7fcc737fe640 1 -- 172.21.15.148:0/1691375101 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcc6c0ebb80 con 0x7fcc68003980 2023-10-30T13:28:32.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.362+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc7c152290 con 0x7fcc68003980 2023-10-30T13:28:32.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.362+0000 7fcc737fe640 1 -- 172.21.15.148:0/1691375101 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fcc6c0044f0 con 0x7fcc68003980 2023-10-30T13:28:32.363 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:32.363 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:29.741451+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:32.363 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 84 2023-10-30T13:28:32.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc5c0664a0 msgr2=0x7fcc5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:32.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc5c0664a0 0x7fcc5c068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fcc64000c10 tx=0x7fcc64009040 comp rx=0 tx=0).stop 2023-10-30T13:28:32.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 msgr2=0x7fcc7c157410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:32.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc7c157410 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009ae0 tx=0x7fcc68005800 comp rx=0 tx=0).stop 2023-10-30T13:28:32.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 shutdown_connections 2023-10-30T13:28:32.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc5c0664a0 0x7fcc5c068960 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc7c152eb0 0x7fcc7c153b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc7c151ce0 0x7fcc7c1554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 --2- 172.21.15.148:0/1691375101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc68003980 0x7fcc7c157410 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:32.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 >> 172.21.15.148:0/1691375101 conn(0x7fcc7c0c7910 msgr2=0x7fcc7c0ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:32.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 shutdown_connections 2023-10-30T13:28:32.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:32.366+0000 7fcc81209640 1 -- 172.21.15.148:0/1691375101 wait complete. 2023-10-30T13:28:32.383 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:28:34.384 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:34.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1028965640 1 Processor -- start 2023-10-30T13:28:34.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1028965640 1 -- start start 2023-10-30T13:28:34.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1028965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 0x7f102415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:34.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1028965640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f10240cf160 0x7f10240cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:34.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1028965640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10240cb800 con 0x7f10241538a0 2023-10-30T13:28:34.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1028965640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10240cb330 con 0x7f10240cf160 2023-10-30T13:28:34.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1028965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10240cad70 con 0x7f102415d6f0 2023-10-30T13:28:34.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1022d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f10240cf160 0x7f10240cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:34.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1022d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f10240cf160 0x7f10240cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54322/0 (socket says 172.21.15.148:54322) 2023-10-30T13:28:34.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1022d9d640 1 -- 172.21.15.148:0/1254598755 learned_addr learned my addr 172.21.15.148:0/1254598755 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:34.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f101bfff640 1 --2- 172.21.15.148:0/1254598755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 0x7f102415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:34.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1022d9d640 1 -- 172.21.15.148:0/1254598755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 msgr2=0x7f102415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:34.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1022d9d640 1 --2- 172.21.15.148:0/1254598755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 0x7f102415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1022d9d640 1 -- 172.21.15.148:0/1254598755 >> v1:172.21.15.150:6789/0 conn(0x7f10241538a0 legacy=0x7f10240ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:34.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.534+0000 7f1022d9d640 1 -- 172.21.15.148:0/1254598755 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10240cb050 con 0x7f10240cf160 2023-10-30T13:28:34.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1022d9d640 1 --2- 172.21.15.148:0/1254598755 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10240cf160 0x7f10240cf590 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f101c009e50 tx=0x7f101c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12256dc8663583f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:34.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1021d9b640 1 -- 172.21.15.148:0/1254598755 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c0af400 con 0x7f10240cf160 2023-10-30T13:28:34.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1021d9b640 1 -- 172.21.15.148:0/1254598755 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c0ab050 con 0x7f10240cf160 2023-10-30T13:28:34.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1021d9b640 1 -- 172.21.15.148:0/1254598755 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c0af780 con 0x7f10240cf160 2023-10-30T13:28:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1028965640 1 -- 172.21.15.148:0/1254598755 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10240cf160 msgr2=0x7f10240cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1028965640 1 --2- 172.21.15.148:0/1254598755 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10240cf160 0x7f10240cf590 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f101c009e50 tx=0x7f101c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1028965640 1 -- 172.21.15.148:0/1254598755 shutdown_connections 2023-10-30T13:28:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1028965640 1 --2- 172.21.15.148:0/1254598755 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10240cf160 0x7f10240cf590 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1028965640 1 --2- 172.21.15.148:0/1254598755 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 0x7f102415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1028965640 1 -- 172.21.15.148:0/1254598755 >> 172.21.15.148:0/1254598755 conn(0x7f102406bfe0 msgr2=0x7f102406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1028965640 1 -- 172.21.15.148:0/1254598755 shutdown_connections 2023-10-30T13:28:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.538+0000 7f1028965640 1 -- 172.21.15.148:0/1254598755 wait complete. 2023-10-30T13:28:34.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 Processor -- start 2023-10-30T13:28:34.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 -- start start 2023-10-30T13:28:34.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10240cf160 0x7f102414d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:34.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10241538a0 0x7f10241e4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:34.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 0x7f10241e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:34.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102414d730 con 0x7f10241538a0 2023-10-30T13:28:34.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10241e41e0 con 0x7f10240cf160 2023-10-30T13:28:34.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102414ec30 con 0x7f102415d6f0 2023-10-30T13:28:34.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10241538a0 0x7f10241e4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:34.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10241538a0 0x7f10241e4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54324/0 (socket says 172.21.15.148:54324) 2023-10-30T13:28:34.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101bfff640 1 -- 172.21.15.148:0/1973082313 learned_addr learned my addr 172.21.15.148:0/1973082313 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:34.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f102259c640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10240cf160 0x7f102414d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:34.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1022d9d640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 0x7f10241e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:34.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101bfff640 1 -- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 msgr2=0x7f10241e4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:34.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101bfff640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 0x7f10241e4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101bfff640 1 -- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10240cf160 msgr2=0x7f102414d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:34.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101bfff640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10240cf160 0x7f102414d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101bfff640 1 -- 172.21.15.148:0/1973082313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c009b30 con 0x7f10241538a0 2023-10-30T13:28:34.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1022d9d640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 0x7f10241e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:34.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101bfff640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10241538a0 0x7f10241e4670 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f100c00d5f0 tx=0x7f100c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:34.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101affd640 1 -- 172.21.15.148:0/1973082313 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c0125f0 con 0x7f10241538a0 2023-10-30T13:28:34.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101affd640 1 -- 172.21.15.148:0/1973082313 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c012750 con 0x7f10241538a0 2023-10-30T13:28:34.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f101affd640 1 -- 172.21.15.148:0/1973082313 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c012b00 con 0x7f10241538a0 2023-10-30T13:28:34.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102414ef10 con 0x7f10241538a0 2023-10-30T13:28:34.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.542+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10241e3e70 con 0x7f10241538a0 2023-10-30T13:28:34.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.546+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10241e4030 con 0x7f10241538a0 2023-10-30T13:28:34.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.546+0000 7f101affd640 1 -- 172.21.15.148:0/1973082313 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f100c012c60 con 0x7f10241538a0 2023-10-30T13:28:34.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.546+0000 7f101affd640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10040664a0 0x7f1004068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:34.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.546+0000 7f101affd640 1 -- 172.21.15.148:0/1973082313 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f100c084750 con 0x7f10241538a0 2023-10-30T13:28:34.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.546+0000 7f102259c640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10040664a0 0x7f1004068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:34.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.546+0000 7f102259c640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10040664a0 0x7f1004068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1014007fd0 tx=0x7f1014009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:34.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.550+0000 7f101affd640 1 -- 172.21.15.148:0/1973082313 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f100c051790 con 0x7f10241538a0 2023-10-30T13:28:34.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.734+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10240b2a50 con 0x7f10241538a0 2023-10-30T13:28:34.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.734+0000 7f101affd640 1 -- 172.21.15.148:0/1973082313 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f100c054db0 con 0x7f10241538a0 2023-10-30T13:28:34.734 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:34.734 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:29.741451+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:34.735 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 84 2023-10-30T13:28:34.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10040664a0 msgr2=0x7f1004068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:34.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10040664a0 0x7f1004068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1014007fd0 tx=0x7f1014009040 comp rx=0 tx=0).stop 2023-10-30T13:28:34.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10241538a0 msgr2=0x7f10241e4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10241538a0 0x7f10241e4670 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f100c00d5f0 tx=0x7f100c00dab0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 shutdown_connections 2023-10-30T13:28:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f10040664a0 0x7f1004068960 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10241538a0 0x7f10241e4670 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f102415d6f0 0x7f10241e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 --2- 172.21.15.148:0/1973082313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10240cf160 0x7f102414d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:34.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 >> 172.21.15.148:0/1973082313 conn(0x7f102406bfe0 msgr2=0x7f10240ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:34.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 shutdown_connections 2023-10-30T13:28:34.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:34.738+0000 7f1028965640 1 -- 172.21.15.148:0/1973082313 wait complete. 2023-10-30T13:28:34.755 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:28:36.756 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:36.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.906+0000 7fefe6ac9640 1 Processor -- start 2023-10-30T13:28:36.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.906+0000 7fefe6ac9640 1 -- start start 2023-10-30T13:28:36.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefe6ac9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fefe00c6bd0 0x7fefe006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:36.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefe6ac9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fefe0071110 0x7fefe0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:36.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefe6ac9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe0066cf0 con 0x7fefe006f7a0 2023-10-30T13:28:36.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefe6ac9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe00700c0 con 0x7fefe00c6bd0 2023-10-30T13:28:36.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefe6ac9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe00c70d0 con 0x7fefe0071110 2023-10-30T13:28:36.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefd7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fefe00c6bd0 0x7fefe006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:36.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefd7fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fefe00c6bd0 0x7fefe006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54338/0 (socket says 172.21.15.148:54338) 2023-10-30T13:28:36.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefd7fff640 1 -- 172.21.15.148:0/1601856772 learned_addr learned my addr 172.21.15.148:0/1601856772 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:36.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefe5066640 1 --2- 172.21.15.148:0/1601856772 >> v2:172.21.15.150:3300/0 conn(0x7fefe0071110 0x7fefe0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:36.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefd7fff640 1 -- 172.21.15.148:0/1601856772 >> v2:172.21.15.150:3300/0 conn(0x7fefe0071110 msgr2=0x7fefe0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:36.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefd7fff640 1 --2- 172.21.15.148:0/1601856772 >> v2:172.21.15.150:3300/0 conn(0x7fefe0071110 0x7fefe0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:36.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefd7fff640 1 -- 172.21.15.148:0/1601856772 >> v1:172.21.15.150:6789/0 conn(0x7fefe006f7a0 legacy=0x7fefe0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:36.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.910+0000 7fefd7fff640 1 -- 172.21.15.148:0/1601856772 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe00c7390 con 0x7fefe00c6bd0 2023-10-30T13:28:36.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefd7fff640 1 --2- 172.21.15.148:0/1601856772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe00c6bd0 0x7fefe006fb80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fefe006cdd0 tx=0x7fefd001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2202ede95497a689 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:36.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefd77fe640 1 -- 172.21.15.148:0/1601856772 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00a89f0 con 0x7fefe00c6bd0 2023-10-30T13:28:36.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefd77fe640 1 -- 172.21.15.148:0/1601856772 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd00a8b50 con 0x7fefe00c6bd0 2023-10-30T13:28:36.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefd77fe640 1 -- 172.21.15.148:0/1601856772 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00b0440 con 0x7fefe00c6bd0 2023-10-30T13:28:36.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 -- 172.21.15.148:0/1601856772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe00c6bd0 msgr2=0x7fefe006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:36.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 --2- 172.21.15.148:0/1601856772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe00c6bd0 0x7fefe006fb80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fefe006cdd0 tx=0x7fefd001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:28:36.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 -- 172.21.15.148:0/1601856772 shutdown_connections 2023-10-30T13:28:36.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 --2- 172.21.15.148:0/1601856772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe00c6bd0 0x7fefe006fb80 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:36.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 --2- 172.21.15.148:0/1601856772 >> v2:172.21.15.150:3300/0 conn(0x7fefe0071110 0x7fefe0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:36.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 -- 172.21.15.148:0/1601856772 >> 172.21.15.148:0/1601856772 conn(0x7fefe00c7910 msgr2=0x7fefe00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:36.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 -- 172.21.15.148:0/1601856772 shutdown_connections 2023-10-30T13:28:36.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 -- 172.21.15.148:0/1601856772 wait complete. 2023-10-30T13:28:36.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 Processor -- start 2023-10-30T13:28:36.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.914+0000 7fefe6ac9640 1 -- start start 2023-10-30T13:28:36.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe6ac9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fefe006f7a0 0x7fefe01de330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:36.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe6ac9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe0071110 0x7fefe01df110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:36.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe6ac9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe00c6bd0 0x7fefe01df650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:36.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe6ac9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe01eec70 con 0x7fefe0071110 2023-10-30T13:28:36.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe6ac9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe01e4070 con 0x7fefe00c6bd0 2023-10-30T13:28:36.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe6ac9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe01e4310 con 0x7fefe006f7a0 2023-10-30T13:28:36.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe0071110 0x7fefe01df110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:36.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe0071110 0x7fefe01df110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54340/0 (socket says 172.21.15.148:54340) 2023-10-30T13:28:36.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd7fff640 1 -- 172.21.15.148:0/3209385742 learned_addr learned my addr 172.21.15.148:0/3209385742 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:36.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe5066640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe00c6bd0 0x7fefe01df650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:36.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe4865640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fefe006f7a0 0x7fefe01de330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:36.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd7fff640 1 -- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fefe006f7a0 msgr2=0x7fefe01de330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:36.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd7fff640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fefe006f7a0 0x7fefe01de330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:36.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd7fff640 1 -- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe00c6bd0 msgr2=0x7fefe01df650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:36.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd7fff640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe00c6bd0 0x7fefe01df650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:36.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd7fff640 1 -- 172.21.15.148:0/3209385742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd00094d0 con 0x7fefe0071110 2023-10-30T13:28:36.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd7fff640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe0071110 0x7fefe01df110 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fefd00b3040 tx=0x7fefd001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:36.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd57fa640 1 -- 172.21.15.148:0/3209385742 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00b9620 con 0x7fefe0071110 2023-10-30T13:28:36.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd57fa640 1 -- 172.21.15.148:0/3209385742 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd00b9780 con 0x7fefe0071110 2023-10-30T13:28:36.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd57fa640 1 -- 172.21.15.148:0/3209385742 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00b9b30 con 0x7fefe0071110 2023-10-30T13:28:36.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe01e45f0 con 0x7fefe0071110 2023-10-30T13:28:36.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe00c44e0 con 0x7fefe0071110 2023-10-30T13:28:36.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe004e170 con 0x7fefe0071110 2023-10-30T13:28:36.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.918+0000 7fefd57fa640 1 -- 172.21.15.148:0/3209385742 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fefd00b7070 con 0x7fefe0071110 2023-10-30T13:28:36.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.922+0000 7fefd57fa640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fefcc0664a0 0x7fefcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:36.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.922+0000 7fefd57fa640 1 -- 172.21.15.148:0/3209385742 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fefd0122190 con 0x7fefe0071110 2023-10-30T13:28:36.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.922+0000 7fefe4865640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fefcc0664a0 0x7fefcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:36.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.922+0000 7fefe4865640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fefcc0664a0 0x7fefcc068960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fefdc000f70 tx=0x7fefdc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:36.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:36.926+0000 7fefd57fa640 1 -- 172.21.15.148:0/3209385742 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fefd00ef250 con 0x7fefe0071110 2023-10-30T13:28:37.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefe01eedb0 con 0x7fefe0071110 2023-10-30T13:28:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefd57fa640 1 -- 172.21.15.148:0/3209385742 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5723 (secure 0 0 0) 0x7fefd00f2870 con 0x7fefe0071110 2023-10-30T13:28:37.104 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:37.105 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:35.764746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 85 2023-10-30T13:28:37.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fefcc0664a0 msgr2=0x7fefcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:37.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fefcc0664a0 0x7fefcc068960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fefdc000f70 tx=0x7fefdc009040 comp rx=0 tx=0).stop 2023-10-30T13:28:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe0071110 msgr2=0x7fefe01df110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe0071110 0x7fefe01df110 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fefd00b3040 tx=0x7fefd001ba90 comp rx=0 tx=0).stop 2023-10-30T13:28:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 shutdown_connections 2023-10-30T13:28:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fefcc0664a0 0x7fefcc068960 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefe0071110 0x7fefe01df110 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe00c6bd0 0x7fefe01df650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:37.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 --2- 172.21.15.148:0/3209385742 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fefe006f7a0 0x7fefe01de330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:37.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 >> 172.21.15.148:0/3209385742 conn(0x7fefe00c7910 msgr2=0x7fefe00c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:37.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 shutdown_connections 2023-10-30T13:28:37.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:37.106+0000 7fefe6ac9640 1 -- 172.21.15.148:0/3209385742 wait complete. 2023-10-30T13:28:37.125 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:28:39.126 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:39.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa3b565640 1 Processor -- start 2023-10-30T13:28:39.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa3b565640 1 -- start start 2023-10-30T13:28:39.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa3b565640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa34150560 0x7ffa3406afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:39.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa3b565640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffa340c68b0 0x7ffa3406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:39.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa3b565640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffa34070650 0x7ffa3406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:39.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa3b565640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa34150940 con 0x7ffa340c68b0 2023-10-30T13:28:39.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa3b565640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa341573f0 con 0x7ffa34070650 2023-10-30T13:28:39.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa3b565640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa34156590 con 0x7ffa34150560 2023-10-30T13:28:39.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa38b00640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffa340c68b0 0x7ffa3406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:39.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa38b00640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffa340c68b0 0x7ffa3406b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54356/0 (socket says 172.21.15.148:54356) 2023-10-30T13:28:39.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa38b00640 1 -- 172.21.15.148:0/405784946 learned_addr learned my addr 172.21.15.148:0/405784946 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:39.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa39301640 1 --2- 172.21.15.148:0/405784946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa34150560 0x7ffa3406afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:39.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.274+0000 7ffa39b02640 1 --2- 172.21.15.148:0/405784946 >> v2:172.21.15.150:3300/0 conn(0x7ffa34070650 0x7ffa3406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:39.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa38b00640 1 -- 172.21.15.148:0/405784946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa34150560 msgr2=0x7ffa3406afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:39.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa38b00640 1 --2- 172.21.15.148:0/405784946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa34150560 0x7ffa3406afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa38b00640 1 -- 172.21.15.148:0/405784946 >> v2:172.21.15.150:3300/0 conn(0x7ffa34070650 msgr2=0x7ffa3406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:39.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa38b00640 1 --2- 172.21.15.148:0/405784946 >> v2:172.21.15.150:3300/0 conn(0x7ffa34070650 0x7ffa3406c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa38b00640 1 -- 172.21.15.148:0/405784946 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa34156870 con 0x7ffa340c68b0 2023-10-30T13:28:39.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa39b02640 1 --2- 172.21.15.148:0/405784946 >> v2:172.21.15.150:3300/0 conn(0x7ffa34070650 0x7ffa3406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:39.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa38b00640 1 --2- 172.21.15.148:0/405784946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c68b0 0x7ffa3406b4e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ffa340b1680 tx=0x7ffa2401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1920d92712d5bb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:39.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa2bfff640 1 -- 172.21.15.148:0/405784946 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa240aa5f0 con 0x7ffa340c68b0 2023-10-30T13:28:39.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa2bfff640 1 -- 172.21.15.148:0/405784946 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa240aa750 con 0x7ffa340c68b0 2023-10-30T13:28:39.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa2bfff640 1 -- 172.21.15.148:0/405784946 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa240aaae0 con 0x7ffa340c68b0 2023-10-30T13:28:39.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa3b565640 1 -- 172.21.15.148:0/405784946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c68b0 msgr2=0x7ffa3406b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:39.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.278+0000 7ffa3b565640 1 --2- 172.21.15.148:0/405784946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c68b0 0x7ffa3406b4e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ffa340b1680 tx=0x7ffa2401c4d0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 -- 172.21.15.148:0/405784946 shutdown_connections 2023-10-30T13:28:39.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 --2- 172.21.15.148:0/405784946 >> v2:172.21.15.150:3300/0 conn(0x7ffa34070650 0x7ffa3406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 --2- 172.21.15.148:0/405784946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa340c68b0 0x7ffa3406b4e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 --2- 172.21.15.148:0/405784946 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa34150560 0x7ffa3406afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 -- 172.21.15.148:0/405784946 >> 172.21.15.148:0/405784946 conn(0x7ffa340cbeb0 msgr2=0x7ffa340cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:39.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 -- 172.21.15.148:0/405784946 shutdown_connections 2023-10-30T13:28:39.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 -- 172.21.15.148:0/405784946 wait complete. 2023-10-30T13:28:39.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 Processor -- start 2023-10-30T13:28:39.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 -- start start 2023-10-30T13:28:39.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa34070650 0x7ffa34159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:39.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c68b0 0x7ffa3406a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:39.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa34150560 0x7ffa341f5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:39.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa341423c0 con 0x7ffa34070650 2023-10-30T13:28:39.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa341597b0 con 0x7ffa34150560 2023-10-30T13:28:39.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa3b565640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa34141ab0 con 0x7ffa340c68b0 2023-10-30T13:28:39.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39301640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa34070650 0x7ffa34159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:39.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39301640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa34070650 0x7ffa34159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54362/0 (socket says 172.21.15.148:54362) 2023-10-30T13:28:39.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39301640 1 -- 172.21.15.148:0/709941410 learned_addr learned my addr 172.21.15.148:0/709941410 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:39.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa38b00640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c68b0 0x7ffa3406a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:39.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39b02640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa34150560 0x7ffa341f5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:39.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39301640 1 -- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c68b0 msgr2=0x7ffa3406a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:39.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39301640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c68b0 0x7ffa3406a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39301640 1 -- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa34150560 msgr2=0x7ffa341f5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:39.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39301640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa34150560 0x7ffa341f5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39301640 1 -- 172.21.15.148:0/709941410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa240094d0 con 0x7ffa34070650 2023-10-30T13:28:39.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.282+0000 7ffa39b02640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa34150560 0x7ffa341f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:39.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa39301640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa34070650 0x7ffa34159270 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000e870 tx=0x7ffa3000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:39.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa29ffb640 1 -- 172.21.15.148:0/709941410 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30009510 con 0x7ffa34070650 2023-10-30T13:28:39.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa29ffb640 1 -- 172.21.15.148:0/709941410 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3000eec0 con 0x7ffa34070650 2023-10-30T13:28:39.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa29ffb640 1 -- 172.21.15.148:0/709941410 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3000cae0 con 0x7ffa34070650 2023-10-30T13:28:39.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa34155c00 con 0x7ffa34070650 2023-10-30T13:28:39.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa34156160 con 0x7ffa34070650 2023-10-30T13:28:39.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa340c6d40 con 0x7ffa34070650 2023-10-30T13:28:39.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa29ffb640 1 -- 172.21.15.148:0/709941410 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffa3000cc40 con 0x7ffa34070650 2023-10-30T13:28:39.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa29ffb640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa180664a0 0x7ffa18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:39.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa29ffb640 1 -- 172.21.15.148:0/709941410 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7ffa300841a0 con 0x7ffa34070650 2023-10-30T13:28:39.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.286+0000 7ffa38b00640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa180664a0 0x7ffa18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:39.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.290+0000 7ffa38b00640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa180664a0 0x7ffa18068960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ffa240ae000 tx=0x7ffa240b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:39.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.290+0000 7ffa29ffb640 1 -- 172.21.15.148:0/709941410 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffa300513c0 con 0x7ffa34070650 2023-10-30T13:28:39.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.470+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa34142630 con 0x7ffa34070650 2023-10-30T13:28:39.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.470+0000 7ffa29ffb640 1 -- 172.21.15.148:0/709941410 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5723 (secure 0 0 0) 0x7ffa30059e60 con 0x7ffa34070650 2023-10-30T13:28:39.471 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:39.472 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:35.764746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:39.472 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 85 2023-10-30T13:28:39.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa180664a0 msgr2=0x7ffa18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:39.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa180664a0 0x7ffa18068960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ffa240ae000 tx=0x7ffa240b5040 comp rx=0 tx=0).stop 2023-10-30T13:28:39.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa34070650 msgr2=0x7ffa34159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:39.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa34070650 0x7ffa34159270 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000e870 tx=0x7ffa3000ec70 comp rx=0 tx=0).stop 2023-10-30T13:28:39.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 shutdown_connections 2023-10-30T13:28:39.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffa180664a0 0x7ffa18068960 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa34150560 0x7ffa341f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffa340c68b0 0x7ffa3406a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 --2- 172.21.15.148:0/709941410 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffa34070650 0x7ffa34159270 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:39.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 >> 172.21.15.148:0/709941410 conn(0x7ffa340cbeb0 msgr2=0x7ffa340c5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:39.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 shutdown_connections 2023-10-30T13:28:39.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:39.474+0000 7ffa3b565640 1 -- 172.21.15.148:0/709941410 wait complete. 2023-10-30T13:28:39.493 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 85, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 85, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6249": { "addr": "172.21.15.148:6837/2740075781", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 2740075781, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 2740075781, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6249, "incarnation": 82, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:28:35.764746+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6249 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:28:41.493 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:41.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fba50f640 1 Processor -- start 2023-10-30T13:28:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fba50f640 1 -- start start 2023-10-30T13:28:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fba50f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9fb40c05d0 0x7f9fb413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fba50f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 0x7f9fb413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fba50f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb40c0a50 con 0x7f9fb40c6b80 2023-10-30T13:28:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fba50f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb406f6e0 con 0x7f9fb40c05d0 2023-10-30T13:28:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fba50f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb406f9c0 con 0x7f9fb40c6f60 2023-10-30T13:28:41.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fb3fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9fb40c6b80 0x7f9fb40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42492/0 (socket says 172.21.15.148:42492) 2023-10-30T13:28:41.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fb3fff640 1 -- 172.21.15.148:0/2997165689 learned_addr learned my addr 172.21.15.148:0/2997165689 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:28:41.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.642+0000 7f9fb8aac640 1 --2- 172.21.15.148:0/2997165689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 0x7f9fb413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:41.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb37fe640 1 --2- 172.21.15.148:0/2997165689 >> v2:172.21.15.150:3300/0 conn(0x7f9fb40c05d0 0x7f9fb413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:41.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb2ffd640 1 -- 172.21.15.148:0/2997165689 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386174947 0 0) 0x7f9fb40c0a50 con 0x7f9fb40c6b80 2023-10-30T13:28:41.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb2ffd640 1 -- 172.21.15.148:0/2997165689 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f9c003530 con 0x7f9fb40c6b80 2023-10-30T13:28:41.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb8aac640 1 -- 172.21.15.148:0/2997165689 >> v2:172.21.15.150:3300/0 conn(0x7f9fb40c05d0 msgr2=0x7f9fb413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:41.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb2ffd640 1 -- 172.21.15.148:0/2997165689 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3578895856 0 0) 0x7f9f9c003530 con 0x7f9fb40c6b80 2023-10-30T13:28:41.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb8aac640 1 --2- 172.21.15.148:0/2997165689 >> v2:172.21.15.150:3300/0 conn(0x7f9fb40c05d0 0x7f9fb413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb8aac640 1 -- 172.21.15.148:0/2997165689 >> v1:172.21.15.148:6789/0 conn(0x7f9fb40c6b80 legacy=0x7f9fb40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:41.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb8aac640 1 -- 172.21.15.148:0/2997165689 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb406fca0 con 0x7f9fb40c6f60 2023-10-30T13:28:41.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb37fe640 1 --2- 172.21.15.148:0/2997165689 >> v2:172.21.15.150:3300/0 conn(0x7f9fb40c05d0 0x7f9fb413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:28:41.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb8aac640 1 --2- 172.21.15.148:0/2997165689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 0x7f9fb413f2d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009d90 tx=0x7f9fac01c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e4f5dcc5ff3308b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:41.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.646+0000 7f9fb2ffd640 1 -- 172.21.15.148:0/2997165689 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4001680 con 0x7f9fb40c6f60 2023-10-30T13:28:41.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fb2ffd640 1 -- 172.21.15.148:0/2997165689 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fac0a5580 con 0x7f9fb40c6f60 2023-10-30T13:28:41.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fb2ffd640 1 -- 172.21.15.148:0/2997165689 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac0a58f0 con 0x7f9fb40c6f60 2023-10-30T13:28:41.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 -- 172.21.15.148:0/2997165689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 msgr2=0x7f9fb413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:41.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 --2- 172.21.15.148:0/2997165689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 0x7f9fb413f2d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009d90 tx=0x7f9fac01c870 comp rx=0 tx=0).stop 2023-10-30T13:28:41.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 -- 172.21.15.148:0/2997165689 shutdown_connections 2023-10-30T13:28:41.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 --2- 172.21.15.148:0/2997165689 >> v2:172.21.15.150:3300/0 conn(0x7f9fb40c05d0 0x7f9fb413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 --2- 172.21.15.148:0/2997165689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 0x7f9fb413f2d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 -- 172.21.15.148:0/2997165689 >> 172.21.15.148:0/2997165689 conn(0x7f9fb40c7910 msgr2=0x7f9fb40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:41.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 -- 172.21.15.148:0/2997165689 shutdown_connections 2023-10-30T13:28:41.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 -- 172.21.15.148:0/2997165689 wait complete. 2023-10-30T13:28:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 Processor -- start 2023-10-30T13:28:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 -- start start 2023-10-30T13:28:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb40c05d0 0x7f9fb406bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9fb40c6b80 0x7f9fb40c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 0x7f9fb4135a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb4158db0 con 0x7f9fb40c6b80 2023-10-30T13:28:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb41355f0 con 0x7f9fb40c05d0 2023-10-30T13:28:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fba50f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb41f2db0 con 0x7f9fb40c6f60 2023-10-30T13:28:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fb37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9fb40c6b80 0x7f9fb40c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fb37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9fb40c6b80 0x7f9fb40c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54366/0 (socket says 172.21.15.148:54366) 2023-10-30T13:28:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fb37fe640 1 -- 172.21.15.148:0/1138744535 learned_addr learned my addr 172.21.15.148:0/1138744535 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fb8aac640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 0x7f9fb4135a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.650+0000 7f9fb3fff640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb40c05d0 0x7f9fb406bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb37fe640 1 -- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 msgr2=0x7f9fb4135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb37fe640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 0x7f9fb4135a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb37fe640 1 -- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb40c05d0 msgr2=0x7f9fb406bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb37fe640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb40c05d0 0x7f9fb406bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb37fe640 1 -- 172.21.15.148:0/1138744535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa8009420 con 0x7f9fb40c6b80 2023-10-30T13:28:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb37fe640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9fb40c6b80 0x7f9fb40c8c10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800e9d0 tx=0x7f9fa800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb0ff9640 1 -- 172.21.15.148:0/1138744535 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa800ca60 con 0x7f9fb40c6b80 2023-10-30T13:28:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb0ff9640 1 -- 172.21.15.148:0/1138744535 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa800cbc0 con 0x7f9fb40c6b80 2023-10-30T13:28:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb0ff9640 1 -- 172.21.15.148:0/1138744535 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8013470 con 0x7f9fb40c6b80 2023-10-30T13:28:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fac009a70 con 0x7f9fb40c6b80 2023-10-30T13:28:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb4158510 con 0x7f9fb40c6b80 2023-10-30T13:28:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb404e170 con 0x7f9fb40c6b80 2023-10-30T13:28:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.654+0000 7f9fb0ff9640 1 -- 172.21.15.148:0/1138744535 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9fa80135d0 con 0x7f9fb40c6b80 2023-10-30T13:28:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.658+0000 7f9fb0ff9640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9fa00664a0 0x7f9fa0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.658+0000 7f9fb0ff9640 1 -- 172.21.15.148:0/1138744535 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f9fa8084450 con 0x7f9fb40c6b80 2023-10-30T13:28:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.658+0000 7f9fb3fff640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9fa00664a0 0x7f9fa0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.658+0000 7f9fb3fff640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9fa00664a0 0x7f9fa0068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4006b60 tx=0x7f9fa4001dd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.662+0000 7f9fb0ff9640 1 -- 172.21.15.148:0/1138744535 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9fa80515d0 con 0x7f9fb40c6b80 2023-10-30T13:28:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.946+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fb40c1d30 con 0x7f9fb40c6b80 2023-10-30T13:28:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.946+0000 7f9fb0ff9640 1 -- 172.21.15.148:0/1138744535 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f9fa8054bf0 con 0x7f9fb40c6b80 2023-10-30T13:28:41.945 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:41.945 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:35.764746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 85 2023-10-30T13:28:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.946+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9fa00664a0 msgr2=0x7f9fa0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.946+0000 7f9fba50f640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9fa00664a0 0x7f9fa0068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4006b60 tx=0x7f9fa4001dd0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.946+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9fb40c6b80 msgr2=0x7f9fb40c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.946+0000 7f9fba50f640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9fb40c6b80 0x7f9fb40c8c10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800e9d0 tx=0x7f9fa800ee90 comp rx=0 tx=0).stop 2023-10-30T13:28:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.950+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 shutdown_connections 2023-10-30T13:28:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.950+0000 7f9fba50f640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9fa00664a0 0x7f9fa0068960 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.950+0000 7f9fba50f640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9fb40c6b80 0x7f9fb40c8c10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.950+0000 7f9fba50f640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9fb40c6f60 0x7f9fb4135a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.950+0000 7f9fba50f640 1 --2- 172.21.15.148:0/1138744535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb40c05d0 0x7f9fb406bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.950+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 >> 172.21.15.148:0/1138744535 conn(0x7f9fb40c7910 msgr2=0x7f9fb40bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.950+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 shutdown_connections 2023-10-30T13:28:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:41.950+0000 7f9fba50f640 1 -- 172.21.15.148:0/1138744535 wait complete. 2023-10-30T13:28:41.974 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:28:43.976 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4ba083a640 1 Processor -- start 2023-10-30T13:28:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4ba083a640 1 -- start start 2023-10-30T13:28:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4ba083a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4b9c1538f0 0x7f4b9c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4ba083a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c0ca6a0 con 0x7f4b9c0cefa0 2023-10-30T13:28:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4ba083a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c15d300 con 0x7f4b9c15cd50 2023-10-30T13:28:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4ba083a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c15d5e0 con 0x7f4b9c1538f0 2023-10-30T13:28:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4b9b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4b9c1538f0 0x7f4b9c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4b9b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4b9c1538f0 0x7f4b9c159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47874/0 (socket says 172.21.15.148:47874) 2023-10-30T13:28:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4b9b7fe640 1 -- 172.21.15.148:0/3664517904 learned_addr learned my addr 172.21.15.148:0/3664517904 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4b9b7fe640 1 -- 172.21.15.148:0/3664517904 >> v1:172.21.15.150:6789/0 conn(0x7f4b9c15cd50 legacy=0x7f4b9c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4b9b7fe640 1 -- 172.21.15.148:0/3664517904 >> v1:172.21.15.148:6789/0 conn(0x7f4b9c0cefa0 legacy=0x7f4b9c1640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.126+0000 7f4b9b7fe640 1 -- 172.21.15.148:0/3664517904 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c15d8c0 con 0x7f4b9c1538f0 2023-10-30T13:28:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4b9b7fe640 1 --2- 172.21.15.148:0/3664517904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 0x7f4b9c159fb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009870 tx=0x7f4b8c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3b6f45b31f6b42 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4b9a7fc640 1 -- 172.21.15.148:0/3664517904 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c0a8a30 con 0x7f4b9c1538f0 2023-10-30T13:28:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4b9a7fc640 1 -- 172.21.15.148:0/3664517904 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8c0a8b90 con 0x7f4b9c1538f0 2023-10-30T13:28:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4b9a7fc640 1 -- 172.21.15.148:0/3664517904 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c0b0440 con 0x7f4b9c1538f0 2023-10-30T13:28:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4ba083a640 1 -- 172.21.15.148:0/3664517904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 msgr2=0x7f4b9c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4ba083a640 1 --2- 172.21.15.148:0/3664517904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 0x7f4b9c159fb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009870 tx=0x7f4b8c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:28:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4ba083a640 1 -- 172.21.15.148:0/3664517904 shutdown_connections 2023-10-30T13:28:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4ba083a640 1 --2- 172.21.15.148:0/3664517904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 0x7f4b9c159fb0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4ba083a640 1 -- 172.21.15.148:0/3664517904 >> 172.21.15.148:0/3664517904 conn(0x7f4b9c06a640 msgr2=0x7f4b9c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4ba083a640 1 -- 172.21.15.148:0/3664517904 shutdown_connections 2023-10-30T13:28:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.130+0000 7f4ba083a640 1 -- 172.21.15.148:0/3664517904 wait complete. 2023-10-30T13:28:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 Processor -- start 2023-10-30T13:28:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 -- start start 2023-10-30T13:28:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b9c0cefa0 0x7f4b9c147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 0x7f4b9c1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b9c15cd50 0x7f4b9c152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c157540 con 0x7f4b9c1538f0 2023-10-30T13:28:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c157820 con 0x7f4b9c15cd50 2023-10-30T13:28:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c157b00 con 0x7f4b9c0cefa0 2023-10-30T13:28:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 0x7f4b9c1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 0x7f4b9c1585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47876/0 (socket says 172.21.15.148:47876) 2023-10-30T13:28:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9affd640 1 -- 172.21.15.148:0/4078876409 learned_addr learned my addr 172.21.15.148:0/4078876409 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9bfff640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b9c15cd50 0x7f4b9c152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9b7fe640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b9c0cefa0 0x7f4b9c147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9affd640 1 -- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b9c0cefa0 msgr2=0x7f4b9c147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9affd640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b9c0cefa0 0x7f4b9c147d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9affd640 1 -- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b9c15cd50 msgr2=0x7f4b9c152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9affd640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b9c15cd50 0x7f4b9c152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9affd640 1 -- 172.21.15.148:0/4078876409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c0094d0 con 0x7f4b9c1538f0 2023-10-30T13:28:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9bfff640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b9c15cd50 0x7f4b9c152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9b7fe640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b9c0cefa0 0x7f4b9c147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b9affd640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 0x7f4b9c1585e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c3c0 tx=0x7f4b9000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b83fff640 1 -- 172.21.15.148:0/4078876409 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b900052d0 con 0x7f4b9c1538f0 2023-10-30T13:28:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b83fff640 1 -- 172.21.15.148:0/4078876409 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b90005430 con 0x7f4b9c1538f0 2023-10-30T13:28:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c157de0 con 0x7f4b9c1538f0 2023-10-30T13:28:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4b83fff640 1 -- 172.21.15.148:0/4078876409 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90003840 con 0x7f4b9c1538f0 2023-10-30T13:28:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9c0c56c0 con 0x7f4b9c1538f0 2023-10-30T13:28:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.134+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c158300 con 0x7f4b9c1538f0 2023-10-30T13:28:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.138+0000 7f4b83fff640 1 -- 172.21.15.148:0/4078876409 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b900039a0 con 0x7f4b9c1538f0 2023-10-30T13:28:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.138+0000 7f4b83fff640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b880664d0 0x7f4b88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.138+0000 7f4b83fff640 1 -- 172.21.15.148:0/4078876409 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f4b90083810 con 0x7f4b9c1538f0 2023-10-30T13:28:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.138+0000 7f4b9b7fe640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b880664d0 0x7f4b88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.138+0000 7f4b9b7fe640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b880664d0 0x7f4b88068990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0b2000 tx=0x7f4b8c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.142+0000 7f4b83fff640 1 -- 172.21.15.148:0/4078876409 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b9c158300 con 0x7f4b9c1538f0 2023-10-30T13:28:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b9c158300 con 0x7f4b9c1538f0 2023-10-30T13:28:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4b83fff640 1 -- 172.21.15.148:0/4078876409 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f4b9c158300 con 0x7f4b9c1538f0 2023-10-30T13:28:44.322 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:44.322 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 86 2023-10-30T13:28:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b880664d0 msgr2=0x7f4b88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b880664d0 0x7f4b88068990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0b2000 tx=0x7f4b8c008040 comp rx=0 tx=0).stop 2023-10-30T13:28:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 msgr2=0x7f4b9c1585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 0x7f4b9c1585e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c3c0 tx=0x7f4b9000c880 comp rx=0 tx=0).stop 2023-10-30T13:28:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 shutdown_connections 2023-10-30T13:28:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4b880664d0 0x7f4b88068990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b9c1538f0 0x7f4b9c1585e0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b9c15cd50 0x7f4b9c152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 --2- 172.21.15.148:0/4078876409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4b9c0cefa0 0x7f4b9c147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 >> 172.21.15.148:0/4078876409 conn(0x7f4b9c06a640 msgr2=0x7f4b9c0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 shutdown_connections 2023-10-30T13:28:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:44.322+0000 7f4ba083a640 1 -- 172.21.15.148:0/4078876409 wait complete. 2023-10-30T13:28:44.343 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=6099 2023-10-30T13:28:44.343 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.a 2023-10-30T13:28:47.344 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:28:47.344 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:28:47.345 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:28:47.349 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:28:47.350 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:47.443 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:28:47.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb561a17640 1 Processor -- start 2023-10-30T13:28:47.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb561a17640 1 -- start start 2023-10-30T13:28:47.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb561a17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c1538a0 0x7fb55c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:47.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb561a17640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb55c0cf160 0x7fb55c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:47.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb561a17640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c0cb800 con 0x7fb55c0ca0e0 2023-10-30T13:28:47.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb561a17640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55c15dc30 con 0x7fb55c0cf160 2023-10-30T13:28:47.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb561a17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c0cb330 con 0x7fb55c1538a0 2023-10-30T13:28:47.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb55b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb55c0cf160 0x7fb55c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:47.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb55b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb55c0cf160 0x7fb55c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47902/0 (socket says 172.21.15.148:47902) 2023-10-30T13:28:47.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb55b7fe640 1 -- 172.21.15.148:0/2598492964 learned_addr learned my addr 172.21.15.148:0/2598492964 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:47.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb55affd640 1 --2- 172.21.15.148:0/2598492964 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c1538a0 0x7fb55c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:47.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb55b7fe640 1 -- 172.21.15.148:0/2598492964 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c1538a0 msgr2=0x7fb55c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:47.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb55b7fe640 1 --2- 172.21.15.148:0/2598492964 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c1538a0 0x7fb55c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb55b7fe640 1 -- 172.21.15.148:0/2598492964 >> v1:172.21.15.150:6789/0 conn(0x7fb55c0ca0e0 legacy=0x7fb55c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:47.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.502+0000 7fb55b7fe640 1 -- 172.21.15.148:0/2598492964 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c0cad70 con 0x7fb55c0cf160 2023-10-30T13:28:47.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb55b7fe640 1 --2- 172.21.15.148:0/2598492964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 0x7fb55c0cf590 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb554009e50 tx=0x7fb55401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e063aab3dd2c677 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:47.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb559ffb640 1 -- 172.21.15.148:0/2598492964 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5540af400 con 0x7fb55c0cf160 2023-10-30T13:28:47.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb559ffb640 1 -- 172.21.15.148:0/2598492964 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5540ab050 con 0x7fb55c0cf160 2023-10-30T13:28:47.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb559ffb640 1 -- 172.21.15.148:0/2598492964 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5540af780 con 0x7fb55c0cf160 2023-10-30T13:28:47.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb561a17640 1 -- 172.21.15.148:0/2598492964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 msgr2=0x7fb55c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb561a17640 1 --2- 172.21.15.148:0/2598492964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 0x7fb55c0cf590 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb554009e50 tx=0x7fb55401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb561a17640 1 -- 172.21.15.148:0/2598492964 shutdown_connections 2023-10-30T13:28:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb561a17640 1 --2- 172.21.15.148:0/2598492964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 0x7fb55c0cf590 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb561a17640 1 --2- 172.21.15.148:0/2598492964 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c1538a0 0x7fb55c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb561a17640 1 -- 172.21.15.148:0/2598492964 >> 172.21.15.148:0/2598492964 conn(0x7fb55c06bfe0 msgr2=0x7fb55c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb561a17640 1 -- 172.21.15.148:0/2598492964 shutdown_connections 2023-10-30T13:28:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb561a17640 1 -- 172.21.15.148:0/2598492964 wait complete. 2023-10-30T13:28:47.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.506+0000 7fb561a17640 1 Processor -- start 2023-10-30T13:28:47.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 -- start start 2023-10-30T13:28:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c0ca0e0 0x7fb55c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 0x7fb55c15c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55c1538a0 0x7fb55c144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c1442e0 con 0x7fb55c0cf160 2023-10-30T13:28:47.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c143fe0 con 0x7fb55c1538a0 2023-10-30T13:28:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c14c570 con 0x7fb55c0ca0e0 2023-10-30T13:28:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 0x7fb55c15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 0x7fb55c15c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47906/0 (socket says 172.21.15.148:47906) 2023-10-30T13:28:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55a7fc640 1 -- 172.21.15.148:0/418891858 learned_addr learned my addr 172.21.15.148:0/418891858 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:47.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55affd640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c0ca0e0 0x7fb55c0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:47.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55b7fe640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55c1538a0 0x7fb55c144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:47.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55a7fc640 1 -- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c0ca0e0 msgr2=0x7fb55c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:47.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55a7fc640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c0ca0e0 0x7fb55c0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55a7fc640 1 -- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55c1538a0 msgr2=0x7fb55c144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:47.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55a7fc640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55c1538a0 0x7fb55c144770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55a7fc640 1 -- 172.21.15.148:0/418891858 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb554009b30 con 0x7fb55c0cf160 2023-10-30T13:28:47.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55affd640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c0ca0e0 0x7fb55c0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:28:47.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb55a7fc640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 0x7fb55c15c920 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb55000c530 tx=0x7fb55000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:47.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb543fff640 1 -- 172.21.15.148:0/418891858 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550005220 con 0x7fb55c0cf160 2023-10-30T13:28:47.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb543fff640 1 -- 172.21.15.148:0/418891858 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb550005380 con 0x7fb55c0cf160 2023-10-30T13:28:47.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55c14c850 con 0x7fb55c0cf160 2023-10-30T13:28:47.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb543fff640 1 -- 172.21.15.148:0/418891858 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550007270 con 0x7fb55c0cf160 2023-10-30T13:28:47.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c14ce10 con 0x7fb55c0cf160 2023-10-30T13:28:47.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.510+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c0b2a50 con 0x7fb55c0cf160 2023-10-30T13:28:47.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.514+0000 7fb543fff640 1 -- 172.21.15.148:0/418891858 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5500073d0 con 0x7fb55c0cf160 2023-10-30T13:28:47.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.514+0000 7fb543fff640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5480664a0 0x7fb548068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:47.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.514+0000 7fb543fff640 1 -- 172.21.15.148:0/418891858 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fb550083a90 con 0x7fb55c0cf160 2023-10-30T13:28:47.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.514+0000 7fb55affd640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5480664a0 0x7fb548068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:47.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.514+0000 7fb55affd640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5480664a0 0x7fb548068960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00b9f0 tx=0x7fb54c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:47.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.518+0000 7fb543fff640 1 -- 172.21.15.148:0/418891858 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb550050b60 con 0x7fb55c0cf160 2023-10-30T13:28:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb55c04e170 con 0x7fb55c0cf160 2023-10-30T13:28:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb543fff640 1 -- 172.21.15.148:0/418891858 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fb550054180 con 0x7fb55c0cf160 2023-10-30T13:28:47.697 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:47.697 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 86 2023-10-30T13:28:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5480664a0 msgr2=0x7fb548068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5480664a0 0x7fb548068960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00b9f0 tx=0x7fb54c008040 comp rx=0 tx=0).stop 2023-10-30T13:28:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 msgr2=0x7fb55c15c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 0x7fb55c15c920 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb55000c530 tx=0x7fb55000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 shutdown_connections 2023-10-30T13:28:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5480664a0 0x7fb548068960 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb55c0cf160 0x7fb55c15c920 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55c1538a0 0x7fb55c144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 --2- 172.21.15.148:0/418891858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55c0ca0e0 0x7fb55c0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 >> 172.21.15.148:0/418891858 conn(0x7fb55c06bfe0 msgr2=0x7fb55c151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 shutdown_connections 2023-10-30T13:28:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:47.698+0000 7fb561a17640 1 -- 172.21.15.148:0/418891858 wait complete. 2023-10-30T13:28:47.721 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:28:49.722 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:28:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.893+0000 7f7fbc89a640 1 Processor -- start 2023-10-30T13:28:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.893+0000 7f7fbc89a640 1 -- start start 2023-10-30T13:28:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.893+0000 7f7fbc89a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 0x7f7fb806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.893+0000 7f7fbc89a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7fb80c6bd0 0x7f7fb8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:49.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.893+0000 7f7fbc89a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb806a3d0 con 0x7f7fb8071110 2023-10-30T13:28:49.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.893+0000 7f7fbc89a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb80700c0 con 0x7f7fb80c6bd0 2023-10-30T13:28:49.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.893+0000 7f7fbc89a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8066cf0 con 0x7f7fb806f7a0 2023-10-30T13:28:49.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.897+0000 7f7faffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7fb80c6bd0 0x7f7fb8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:49.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.897+0000 7f7faffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7fb80c6bd0 0x7f7fb8069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47914/0 (socket says 172.21.15.148:47914) 2023-10-30T13:28:49.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.897+0000 7f7faffff640 1 -- 172.21.15.148:0/1890840357 learned_addr learned my addr 172.21.15.148:0/1890840357 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:49.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.897+0000 7f7fb659c640 1 --2- 172.21.15.148:0/1890840357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 0x7f7fb806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:49.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.897+0000 7f7faffff640 1 -- 172.21.15.148:0/1890840357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 msgr2=0x7f7fb806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:28:49.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.897+0000 7f7faffff640 1 --2- 172.21.15.148:0/1890840357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 0x7f7fb806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:49.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.897+0000 7f7faffff640 1 -- 172.21.15.148:0/1890840357 >> v1:172.21.15.150:6789/0 conn(0x7f7fb8071110 legacy=0x7f7fb8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:49.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.897+0000 7f7faffff640 1 -- 172.21.15.148:0/1890840357 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb80c70d0 con 0x7f7fb80c6bd0 2023-10-30T13:28:49.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7faffff640 1 --2- 172.21.15.148:0/1890840357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb80c6bd0 0x7f7fb8069e90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009870 tx=0x7f7fa001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e4d8d11d1d583c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:49.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fb5d9b640 1 -- 172.21.15.148:0/1890840357 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa00a8a30 con 0x7f7fb80c6bd0 2023-10-30T13:28:49.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fb5d9b640 1 -- 172.21.15.148:0/1890840357 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa00a8b90 con 0x7f7fb80c6bd0 2023-10-30T13:28:49.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fb5d9b640 1 -- 172.21.15.148:0/1890840357 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa00b0440 con 0x7f7fb80c6bd0 2023-10-30T13:28:49.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1890840357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb80c6bd0 msgr2=0x7f7fb8069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:49.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 --2- 172.21.15.148:0/1890840357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb80c6bd0 0x7f7fb8069e90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009870 tx=0x7f7fa001ed30 comp rx=0 tx=0).stop 2023-10-30T13:28:49.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1890840357 shutdown_connections 2023-10-30T13:28:49.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 --2- 172.21.15.148:0/1890840357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb80c6bd0 0x7f7fb8069e90 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:49.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 --2- 172.21.15.148:0/1890840357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 0x7f7fb806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:49.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1890840357 >> 172.21.15.148:0/1890840357 conn(0x7f7fb80c7910 msgr2=0x7f7fb80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:49.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1890840357 shutdown_connections 2023-10-30T13:28:49.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1890840357 wait complete. 2023-10-30T13:28:49.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 Processor -- start 2023-10-30T13:28:49.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.901+0000 7f7fbc89a640 1 -- start start 2023-10-30T13:28:49.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fbc89a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 0x7f7fb814ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:49.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fbc89a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb8071110 0x7f7fb80ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:49.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fbc89a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fb80c6bd0 0x7f7fb80aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:49.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fbc89a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80aeb50 con 0x7f7fb8071110 2023-10-30T13:28:49.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fbc89a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80b0280 con 0x7f7fb80c6bd0 2023-10-30T13:28:49.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fbc89a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb80c1a00 con 0x7f7fb806f7a0 2023-10-30T13:28:49.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb8071110 0x7f7fb80ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:49.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb8071110 0x7f7fb80ca620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47928/0 (socket says 172.21.15.148:47928) 2023-10-30T13:28:49.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faffff640 1 -- 172.21.15.148:0/1365067829 learned_addr learned my addr 172.21.15.148:0/1365067829 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:28:49.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fb659c640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 0x7f7fb814ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:49.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fb6d9d640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fb80c6bd0 0x7f7fb80aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:49.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faffff640 1 -- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 msgr2=0x7f7fb814ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:49.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faffff640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 0x7f7fb814ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:49.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faffff640 1 -- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fb80c6bd0 msgr2=0x7f7fb80aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:49.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faffff640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fb80c6bd0 0x7f7fb80aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:49.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faffff640 1 -- 172.21.15.148:0/1365067829 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa00094d0 con 0x7f7fb8071110 2023-10-30T13:28:49.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faffff640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb8071110 0x7f7fb80ca620 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7fa00b3040 tx=0x7f7fa001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:49.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faeffd640 1 -- 172.21.15.148:0/1365067829 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa00b0dd0 con 0x7f7fb8071110 2023-10-30T13:28:49.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faeffd640 1 -- 172.21.15.148:0/1365067829 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa00bb460 con 0x7f7fb8071110 2023-10-30T13:28:49.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb80c1ce0 con 0x7f7fb8071110 2023-10-30T13:28:49.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faeffd640 1 -- 172.21.15.148:0/1365067829 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa00bb810 con 0x7f7fb8071110 2023-10-30T13:28:49.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb80ae7e0 con 0x7f7fb8071110 2023-10-30T13:28:49.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb80b0e50 con 0x7f7fb8071110 2023-10-30T13:28:49.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.905+0000 7f7faeffd640 1 -- 172.21.15.148:0/1365067829 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fa00b83f0 con 0x7f7fb8071110 2023-10-30T13:28:49.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.909+0000 7f7faeffd640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f980664a0 0x7f7f98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:28:49.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.909+0000 7f7faeffd640 1 -- 172.21.15.148:0/1365067829 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f7fa0121a20 con 0x7f7fb8071110 2023-10-30T13:28:49.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.909+0000 7f7fb659c640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f980664a0 0x7f7f98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:28:49.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.909+0000 7f7fb659c640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f980664a0 0x7f7f98068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009a50 tx=0x7f7fa8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:28:49.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:49.913+0000 7f7faeffd640 1 -- 172.21.15.148:0/1365067829 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7fa00eeaf0 con 0x7f7fb8071110 2023-10-30T13:28:50.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fb804e170 con 0x7f7fb8071110 2023-10-30T13:28:50.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7faeffd640 1 -- 172.21.15.148:0/1365067829 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f7fa00b5090 con 0x7f7fb8071110 2023-10-30T13:28:50.097 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:28:50.097 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:28:50.097 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 87 2023-10-30T13:28:50.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f980664a0 msgr2=0x7f7f98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:50.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f980664a0 0x7f7f98068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009a50 tx=0x7f7fa8008040 comp rx=0 tx=0).stop 2023-10-30T13:28:50.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb8071110 msgr2=0x7f7fb80ca620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:28:50.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb8071110 0x7f7fb80ca620 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7fa00b3040 tx=0x7f7fa001bad0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 shutdown_connections 2023-10-30T13:28:50.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f980664a0 0x7f7f98068960 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7fb8071110 0x7f7fb80ca620 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fb80c6bd0 0x7f7fb80aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 --2- 172.21.15.148:0/1365067829 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7fb806f7a0 0x7f7fb814ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:28:50.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 >> 172.21.15.148:0/1365067829 conn(0x7f7fb80c7910 msgr2=0x7f7fb8157f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:28:50.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 shutdown_connections 2023-10-30T13:28:50.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:28:50.097+0000 7f7fbc89a640 1 -- 172.21.15.148:0/1365067829 wait complete. 2023-10-30T13:28:50.117 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:28:50.117 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-10-30T13:28:50.670 INFO:tasks.ceph.mds.c.smithi148.stderr:2023-10-30T13:28:50.673+0000 7f58fd397640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:28:56.381 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:28:56.385 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:29:26.447 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:29:26.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:29:33.119 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25f26b640 1 Processor -- start 2023-10-30T13:29:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25f26b640 1 -- start start 2023-10-30T13:29:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25f26b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe2580c6b80 0x7fe2580bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25f26b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe25813f4a0 0x7fe2580c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25f26b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2580c0a50 con 0x7fe2580c6f60 2023-10-30T13:29:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25f26b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25806f6e0 con 0x7fe2580c6b80 2023-10-30T13:29:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25f26b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25806f9c0 con 0x7fe25813f4a0 2023-10-30T13:29:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25d007640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe2580c6b80 0x7fe2580bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25d007640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe2580c6b80 0x7fe2580bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40272/0 (socket says 172.21.15.148:40272) 2023-10-30T13:29:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25d007640 1 -- 172.21.15.148:0/3339401194 learned_addr learned my addr 172.21.15.148:0/3339401194 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25d808640 1 --2- 172.21.15.148:0/3339401194 >> v2:172.21.15.150:3300/0 conn(0x7fe25813f4a0 0x7fe2580c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25d007640 1 -- 172.21.15.148:0/3339401194 >> v2:172.21.15.150:3300/0 conn(0x7fe25813f4a0 msgr2=0x7fe2580c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25d007640 1 --2- 172.21.15.148:0/3339401194 >> v2:172.21.15.150:3300/0 conn(0x7fe25813f4a0 0x7fe2580c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25d007640 1 -- 172.21.15.148:0/3339401194 >> v1:172.21.15.150:6789/0 conn(0x7fe2580c6f60 legacy=0x7fe25813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.261+0000 7fe25d007640 1 -- 172.21.15.148:0/3339401194 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25806fca0 con 0x7fe2580c6b80 2023-10-30T13:29:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe25d007640 1 --2- 172.21.15.148:0/3339401194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2580c6b80 0x7fe2580bfec0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe2580c8980 tx=0x7fe25401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502c11dd39e299b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe24ffff640 1 -- 172.21.15.148:0/3339401194 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2540ac630 con 0x7fe2580c6b80 2023-10-30T13:29:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe24ffff640 1 -- 172.21.15.148:0/3339401194 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2540ac790 con 0x7fe2580c6b80 2023-10-30T13:29:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe24ffff640 1 -- 172.21.15.148:0/3339401194 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2540acb20 con 0x7fe2580c6b80 2023-10-30T13:29:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe25f26b640 1 -- 172.21.15.148:0/3339401194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2580c6b80 msgr2=0x7fe2580bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe25f26b640 1 --2- 172.21.15.148:0/3339401194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2580c6b80 0x7fe2580bfec0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe2580c8980 tx=0x7fe25401c4c0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe25f26b640 1 -- 172.21.15.148:0/3339401194 shutdown_connections 2023-10-30T13:29:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe25f26b640 1 --2- 172.21.15.148:0/3339401194 >> v2:172.21.15.150:3300/0 conn(0x7fe25813f4a0 0x7fe2580c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe25f26b640 1 --2- 172.21.15.148:0/3339401194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2580c6b80 0x7fe2580bfec0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe25f26b640 1 -- 172.21.15.148:0/3339401194 >> 172.21.15.148:0/3339401194 conn(0x7fe2580c7910 msgr2=0x7fe2580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe25f26b640 1 -- 172.21.15.148:0/3339401194 shutdown_connections 2023-10-30T13:29:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.265+0000 7fe25f26b640 1 -- 172.21.15.148:0/3339401194 wait complete. 2023-10-30T13:29:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25f26b640 1 Processor -- start 2023-10-30T13:29:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25f26b640 1 -- start start 2023-10-30T13:29:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25f26b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2580c6b80 0x7fe2581546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25f26b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2580c6f60 0x7fe258155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25f26b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25813f4a0 0x7fe258153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25f26b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe258154ee0 con 0x7fe25813f4a0 2023-10-30T13:29:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25f26b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2581f1a40 con 0x7fe2580c6b80 2023-10-30T13:29:33.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25f26b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2581f1c50 con 0x7fe2580c6f60 2023-10-30T13:29:33.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d808640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25813f4a0 0x7fe258153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d808640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25813f4a0 0x7fe258153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40280/0 (socket says 172.21.15.148:40280) 2023-10-30T13:29:33.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d808640 1 -- 172.21.15.148:0/1807346739 learned_addr learned my addr 172.21.15.148:0/1807346739 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:33.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25c806640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2580c6f60 0x7fe258155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d007640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2580c6b80 0x7fe2581546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d808640 1 -- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2580c6f60 msgr2=0x7fe258155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d808640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2580c6f60 0x7fe258155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d808640 1 -- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2580c6b80 msgr2=0x7fe2581546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d808640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2580c6b80 0x7fe2581546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d808640 1 -- 172.21.15.148:0/1807346739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2540094d0 con 0x7fe25813f4a0 2023-10-30T13:29:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d007640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2580c6b80 0x7fe2581546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25d808640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25813f4a0 0x7fe258153090 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe250007e60 tx=0x7fe25000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe24dffb640 1 -- 172.21.15.148:0/1807346739 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250014420 con 0x7fe25813f4a0 2023-10-30T13:29:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe24dffb640 1 -- 172.21.15.148:0/1807346739 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2500145a0 con 0x7fe25813f4a0 2023-10-30T13:29:33.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe24dffb640 1 -- 172.21.15.148:0/1807346739 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250014950 con 0x7fe25813f4a0 2023-10-30T13:29:33.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.269+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2581f1f30 con 0x7fe25813f4a0 2023-10-30T13:29:33.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.273+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2581f24f0 con 0x7fe25813f4a0 2023-10-30T13:29:33.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.273+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25804e170 con 0x7fe25813f4a0 2023-10-30T13:29:33.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.273+0000 7fe24dffb640 1 -- 172.21.15.148:0/1807346739 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe250014ab0 con 0x7fe25813f4a0 2023-10-30T13:29:33.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.273+0000 7fe24dffb640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe23c0664a0 0x7fe23c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.273+0000 7fe24dffb640 1 -- 172.21.15.148:0/1807346739 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fe250084bd0 con 0x7fe25813f4a0 2023-10-30T13:29:33.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.273+0000 7fe25d007640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe23c0664a0 0x7fe23c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.273+0000 7fe25d007640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe23c0664a0 0x7fe23c068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe2540ae000 tx=0x7fe254008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:33.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.277+0000 7fe24dffb640 1 -- 172.21.15.148:0/1807346739 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe250051b80 con 0x7fe25813f4a0 2023-10-30T13:29:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.457+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe258154c00 con 0x7fe25813f4a0 2023-10-30T13:29:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.457+0000 7fe24dffb640 1 -- 172.21.15.148:0/1807346739 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fe2500551a0 con 0x7fe25813f4a0 2023-10-30T13:29:33.454 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:33.454 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 87 2023-10-30T13:29:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe23c0664a0 msgr2=0x7fe23c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe23c0664a0 0x7fe23c068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe2540ae000 tx=0x7fe254008040 comp rx=0 tx=0).stop 2023-10-30T13:29:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25813f4a0 msgr2=0x7fe258153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25813f4a0 0x7fe258153090 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe250007e60 tx=0x7fe25000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 shutdown_connections 2023-10-30T13:29:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe23c0664a0 0x7fe23c068960 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe25813f4a0 0x7fe258153090 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2580c6f60 0x7fe258155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 --2- 172.21.15.148:0/1807346739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2580c6b80 0x7fe2581546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 >> 172.21.15.148:0/1807346739 conn(0x7fe2580c7910 msgr2=0x7fe2580c1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 shutdown_connections 2023-10-30T13:29:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.461+0000 7fe25f26b640 1 -- 172.21.15.148:0/1807346739 wait complete. 2023-10-30T13:29:33.475 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:29:33.475 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:29:33.475 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:29:33.495 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:29:33.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:33.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a4646640 1 Processor -- start 2023-10-30T13:29:33.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a4646640 1 -- start start 2023-10-30T13:29:33.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a4646640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb89c15d6f0 0x7fb89c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a4646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c0cf160 0x7fb89c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a4646640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c0cb800 con 0x7fb89c1538a0 2023-10-30T13:29:33.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a4646640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89c0cb330 con 0x7fb89c15d6f0 2023-10-30T13:29:33.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a4646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c0cad70 con 0x7fb89c0cf160 2023-10-30T13:29:33.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a1be1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb89c15d6f0 0x7fb89c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a1be1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb89c15d6f0 0x7fb89c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40282/0 (socket says 172.21.15.148:40282) 2023-10-30T13:29:33.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a1be1640 1 -- 172.21.15.148:0/3246758068 learned_addr learned my addr 172.21.15.148:0/3246758068 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:33.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a23e2640 1 --1- 172.21.15.148:0/3246758068 >> v1:172.21.15.148:6789/0 conn(0x7fb89c1538a0 0x7fb89c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:35506/0 (socket says 172.21.15.148:35506) 2023-10-30T13:29:33.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a2be3640 1 --2- 172.21.15.148:0/3246758068 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c0cf160 0x7fb89c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a1be1640 1 -- 172.21.15.148:0/3246758068 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c0cf160 msgr2=0x7fb89c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:33.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a1be1640 1 --2- 172.21.15.148:0/3246758068 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c0cf160 0x7fb89c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a1be1640 1 -- 172.21.15.148:0/3246758068 >> v1:172.21.15.148:6789/0 conn(0x7fb89c1538a0 legacy=0x7fb89c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a1be1640 1 -- 172.21.15.148:0/3246758068 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c0cb050 con 0x7fb89c15d6f0 2023-10-30T13:29:33.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.681+0000 7fb8a13e0640 1 -- 172.21.15.148:0/3246758068 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442850089 0 0) 0x7fb89c0cb800 con 0x7fb89c1538a0 2023-10-30T13:29:33.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a1be1640 1 --2- 172.21.15.148:0/3246758068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c15d6f0 0x7fb89c15a090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009870 tx=0x7fb88c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01f4d106d7e7a1e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:33.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a13e0640 1 -- 172.21.15.148:0/3246758068 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0a8a30 con 0x7fb89c15d6f0 2023-10-30T13:29:33.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a13e0640 1 -- 172.21.15.148:0/3246758068 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c0a8b90 con 0x7fb89c15d6f0 2023-10-30T13:29:33.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a13e0640 1 -- 172.21.15.148:0/3246758068 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0b0440 con 0x7fb89c15d6f0 2023-10-30T13:29:33.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a4646640 1 -- 172.21.15.148:0/3246758068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c15d6f0 msgr2=0x7fb89c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a4646640 1 --2- 172.21.15.148:0/3246758068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c15d6f0 0x7fb89c15a090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009870 tx=0x7fb88c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:29:33.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a4646640 1 -- 172.21.15.148:0/3246758068 shutdown_connections 2023-10-30T13:29:33.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a4646640 1 --2- 172.21.15.148:0/3246758068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c15d6f0 0x7fb89c15a090 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a4646640 1 --2- 172.21.15.148:0/3246758068 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c0cf160 0x7fb89c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a4646640 1 -- 172.21.15.148:0/3246758068 >> 172.21.15.148:0/3246758068 conn(0x7fb89c06bfe0 msgr2=0x7fb89c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:33.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a4646640 1 -- 172.21.15.148:0/3246758068 shutdown_connections 2023-10-30T13:29:33.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.685+0000 7fb8a4646640 1 -- 172.21.15.148:0/3246758068 wait complete. 2023-10-30T13:29:33.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 Processor -- start 2023-10-30T13:29:33.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 -- start start 2023-10-30T13:29:33.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0cf160 0x7fb89c0cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c1538a0 0x7fb89c14cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c15d6f0 0x7fb89c14d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c158a50 con 0x7fb89c1538a0 2023-10-30T13:29:33.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c0ce000 con 0x7fb89c0cf160 2023-10-30T13:29:33.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c157700 con 0x7fb89c15d6f0 2023-10-30T13:29:33.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a1be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c1538a0 0x7fb89c14cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a1be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c1538a0 0x7fb89c14cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40292/0 (socket says 172.21.15.148:40292) 2023-10-30T13:29:33.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a1be1640 1 -- 172.21.15.148:0/3851828973 learned_addr learned my addr 172.21.15.148:0/3851828973 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:33.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a2be3640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c15d6f0 0x7fb89c14d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a23e2640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0cf160 0x7fb89c0cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a1be1640 1 -- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c15d6f0 msgr2=0x7fb89c14d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a1be1640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c15d6f0 0x7fb89c14d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a1be1640 1 -- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0cf160 msgr2=0x7fb89c0cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a1be1640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0cf160 0x7fb89c0cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a1be1640 1 -- 172.21.15.148:0/3851828973 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c0094d0 con 0x7fb89c1538a0 2023-10-30T13:29:33.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a1be1640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c1538a0 0x7fb89c14cd70 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb88c01d3e0 tx=0x7fb88c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb892ffd640 1 -- 172.21.15.148:0/3851828973 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0ba600 con 0x7fb89c1538a0 2023-10-30T13:29:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb892ffd640 1 -- 172.21.15.148:0/3851828973 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c0ba780 con 0x7fb89c1538a0 2023-10-30T13:29:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89c0cd480 con 0x7fb89c1538a0 2023-10-30T13:29:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb892ffd640 1 -- 172.21.15.148:0/3851828973 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0bab30 con 0x7fb89c1538a0 2023-10-30T13:29:33.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89c0cda40 con 0x7fb89c1538a0 2023-10-30T13:29:33.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.689+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89c0b2a50 con 0x7fb89c1538a0 2023-10-30T13:29:33.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.693+0000 7fb892ffd640 1 -- 172.21.15.148:0/3851828973 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb88c0bac90 con 0x7fb89c1538a0 2023-10-30T13:29:33.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.693+0000 7fb892ffd640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:33.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.693+0000 7fb892ffd640 1 -- 172.21.15.148:0/3851828973 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fb88c122880 con 0x7fb89c1538a0 2023-10-30T13:29:33.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.693+0000 7fb8a23e2640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:33.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.693+0000 7fb8a23e2640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb898000f70 tx=0x7fb898007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:33.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.697+0000 7fb892ffd640 1 -- 172.21.15.148:0/3851828973 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb88c0ef950 con 0x7fb89c1538a0 2023-10-30T13:29:33.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb89c157560 con 0x7fb89c1538a0 2023-10-30T13:29:33.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb892ffd640 1 -- 172.21.15.148:0/3851828973 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fb89c157560 con 0x7fb89c1538a0 2023-10-30T13:29:33.877 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:33.878 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:33.878 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 87 2023-10-30T13:29:33.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 msgr2=0x7fb880068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb898000f70 tx=0x7fb898007b60 comp rx=0 tx=0).stop 2023-10-30T13:29:33.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c1538a0 msgr2=0x7fb89c14cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:33.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c1538a0 0x7fb89c14cd70 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb88c01d3e0 tx=0x7fb88c01bad0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 shutdown_connections 2023-10-30T13:29:33.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c1538a0 0x7fb89c14cd70 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c15d6f0 0x7fb89c14d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 --2- 172.21.15.148:0/3851828973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0cf160 0x7fb89c0cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:33.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 >> 172.21.15.148:0/3851828973 conn(0x7fb89c06bfe0 msgr2=0x7fb89c0ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:33.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 shutdown_connections 2023-10-30T13:29:33.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:33.881+0000 7fb8a4646640 1 -- 172.21.15.148:0/3851828973 wait complete. 2023-10-30T13:29:33.898 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:29:35.899 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:36.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf9217640 1 Processor -- start 2023-10-30T13:29:36.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf9217640 1 -- start start 2023-10-30T13:29:36.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf9217640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8cf41538f0 0x7f8cf4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:36.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf9217640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:36.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf9217640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf40ca6a0 con 0x7f8cf415cd50 2023-10-30T13:29:36.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf9217640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf415d300 con 0x7f8cf41538f0 2023-10-30T13:29:36.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf9217640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf415d5e0 con 0x7f8cf40cefa0 2023-10-30T13:29:36.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf8a16640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:36.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8cf41538f0 0x7f8cf4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf8a16640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:56516/0 (socket says 172.21.15.148:56516) 2023-10-30T13:29:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf8a16640 1 -- 172.21.15.148:0/1070307929 learned_addr learned my addr 172.21.15.148:0/1070307929 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf37fe640 1 --1- 172.21.15.148:0/1070307929 >> v1:172.21.15.150:6789/0 conn(0x7f8cf415cd50 0x7f8cf40c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:54812/0 (socket says 172.21.15.148:54812) 2023-10-30T13:29:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf8a16640 1 -- 172.21.15.148:0/1070307929 >> v2:172.21.15.150:3300/0 conn(0x7f8cf41538f0 msgr2=0x7f8cf4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf8a16640 1 --2- 172.21.15.148:0/1070307929 >> v2:172.21.15.150:3300/0 conn(0x7f8cf41538f0 0x7f8cf4159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf8a16640 1 -- 172.21.15.148:0/1070307929 >> v1:172.21.15.150:6789/0 conn(0x7f8cf415cd50 legacy=0x7f8cf40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:36.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.052+0000 7f8cf8a16640 1 -- 172.21.15.148:0/1070307929 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf415d8c0 con 0x7f8cf40cefa0 2023-10-30T13:29:36.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf8a16640 1 --2- 172.21.15.148:0/1070307929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40cf3d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009e50 tx=0x7f8cec01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c83dc1817c0429d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:36.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf2ffd640 1 -- 172.21.15.148:0/1070307929 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0af400 con 0x7f8cf40cefa0 2023-10-30T13:29:36.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf2ffd640 1 -- 172.21.15.148:0/1070307929 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cec0ab050 con 0x7f8cf40cefa0 2023-10-30T13:29:36.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf2ffd640 1 -- 172.21.15.148:0/1070307929 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0af780 con 0x7f8cf40cefa0 2023-10-30T13:29:36.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf9217640 1 -- 172.21.15.148:0/1070307929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 msgr2=0x7f8cf40cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:36.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf9217640 1 --2- 172.21.15.148:0/1070307929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40cf3d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009e50 tx=0x7f8cec01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf9217640 1 -- 172.21.15.148:0/1070307929 shutdown_connections 2023-10-30T13:29:36.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf9217640 1 --2- 172.21.15.148:0/1070307929 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40cf3d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf9217640 1 --2- 172.21.15.148:0/1070307929 >> v2:172.21.15.150:3300/0 conn(0x7f8cf41538f0 0x7f8cf4159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf9217640 1 -- 172.21.15.148:0/1070307929 >> 172.21.15.148:0/1070307929 conn(0x7f8cf406a640 msgr2=0x7f8cf4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:36.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf9217640 1 -- 172.21.15.148:0/1070307929 shutdown_connections 2023-10-30T13:29:36.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.056+0000 7f8cf9217640 1 -- 172.21.15.148:0/1070307929 wait complete. 2023-10-30T13:29:36.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 Processor -- start 2023-10-30T13:29:36.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 -- start start 2023-10-30T13:29:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf41538f0 0x7f8cf4151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cf415cd50 0x7f8cf4157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf415cb90 con 0x7f8cf415cd50 2023-10-30T13:29:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40b4900 con 0x7f8cf41538f0 2023-10-30T13:29:36.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40b4040 con 0x7f8cf40cefa0 2023-10-30T13:29:36.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf8a16640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cf415cd50 0x7f8cf4157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:36.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf8a16640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cf415cd50 0x7f8cf4157c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40294/0 (socket says 172.21.15.148:40294) 2023-10-30T13:29:36.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf8a16640 1 -- 172.21.15.148:0/3831837605 learned_addr learned my addr 172.21.15.148:0/3831837605 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:36.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf3fff640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:36.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf37fe640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf41538f0 0x7f8cf4151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:36.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf8a16640 1 -- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 msgr2=0x7f8cf40b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:36.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf8a16640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40b4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf8a16640 1 -- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf41538f0 msgr2=0x7f8cf4151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:36.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf8a16640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf41538f0 0x7f8cf4151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf8a16640 1 -- 172.21.15.148:0/3831837605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec009b30 con 0x7f8cf415cd50 2023-10-30T13:29:36.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf37fe640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf41538f0 0x7f8cf4151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:36.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf8a16640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cf415cd50 0x7f8cf4157c30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0b2040 tx=0x7f8cec01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:36.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf0ff9640 1 -- 172.21.15.148:0/3831837605 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0afd30 con 0x7f8cf415cd50 2023-10-30T13:29:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf0ff9640 1 -- 172.21.15.148:0/3831837605 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cec0afe90 con 0x7f8cf415cd50 2023-10-30T13:29:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf0ff9640 1 -- 172.21.15.148:0/3831837605 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0b8670 con 0x7f8cf415cd50 2023-10-30T13:29:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf41486b0 con 0x7f8cf415cd50 2023-10-30T13:29:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf40b4590 con 0x7f8cf415cd50 2023-10-30T13:29:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.060+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf404e170 con 0x7f8cf415cd50 2023-10-30T13:29:36.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.064+0000 7f8cf0ff9640 1 -- 172.21.15.148:0/3831837605 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8cec0b6040 con 0x7f8cf415cd50 2023-10-30T13:29:36.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.064+0000 7f8cf0ff9640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ce00664a0 0x7f8ce0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:36.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.064+0000 7f8cf0ff9640 1 -- 172.21.15.148:0/3831837605 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f8cec121f20 con 0x7f8cf415cd50 2023-10-30T13:29:36.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.064+0000 7f8cf3fff640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ce00664a0 0x7f8ce0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:36.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.064+0000 7f8cf3fff640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ce00664a0 0x7f8ce0068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009b60 tx=0x7f8ce4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:36.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.068+0000 7f8cf0ff9640 1 -- 172.21.15.148:0/3831837605 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8cec0ef030 con 0x7f8cf415cd50 2023-10-30T13:29:36.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cf40b4a40 con 0x7f8cf415cd50 2023-10-30T13:29:36.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf0ff9640 1 -- 172.21.15.148:0/3831837605 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f8cec0b4020 con 0x7f8cf415cd50 2023-10-30T13:29:36.245 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:36.246 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:36.247 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 87 2023-10-30T13:29:36.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ce00664a0 msgr2=0x7f8ce0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:36.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ce00664a0 0x7f8ce0068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009b60 tx=0x7f8ce4009190 comp rx=0 tx=0).stop 2023-10-30T13:29:36.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cf415cd50 msgr2=0x7f8cf4157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:36.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cf415cd50 0x7f8cf4157c30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0b2040 tx=0x7f8cec01bf90 comp rx=0 tx=0).stop 2023-10-30T13:29:36.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 shutdown_connections 2023-10-30T13:29:36.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ce00664a0 0x7f8ce0068960 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8cf415cd50 0x7f8cf4157c30 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cf41538f0 0x7f8cf4151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 --2- 172.21.15.148:0/3831837605 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8cf40cefa0 0x7f8cf40b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:36.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 >> 172.21.15.148:0/3831837605 conn(0x7f8cf406a640 msgr2=0x7f8cf40c5200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:36.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 shutdown_connections 2023-10-30T13:29:36.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:36.248+0000 7f8cf9217640 1 -- 172.21.15.148:0/3831837605 wait complete. 2023-10-30T13:29:36.266 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:29:38.267 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:38.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.420+0000 7f937cc46640 1 Processor -- start 2023-10-30T13:29:38.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.420+0000 7f937cc46640 1 -- start start 2023-10-30T13:29:38.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937cc46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93780c6b80 0x7f93780bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:38.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937cc46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f937813f4a0 0x7f93780c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:38.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937cc46640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93780c0a50 con 0x7f93780c6f60 2023-10-30T13:29:38.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937cc46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937806f6e0 con 0x7f937813f4a0 2023-10-30T13:29:38.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937cc46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937806f9c0 con 0x7f93780c6b80 2023-10-30T13:29:38.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f9375d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f93780c6f60 0x7f937813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:35510/0 (socket says 172.21.15.148:35510) 2023-10-30T13:29:38.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f9375d9b640 1 -- 172.21.15.148:0/1396850010 learned_addr learned my addr 172.21.15.148:0/1396850010 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:29:38.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937659c640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93780c6b80 0x7f93780bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f9376d9d640 1 --2- 172.21.15.148:0/1396850010 >> v2:172.21.15.150:3300/0 conn(0x7f937813f4a0 0x7f93780c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981108775 0 0) 0x7f93780c0a50 con 0x7f93780c6f60 2023-10-30T13:29:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9360003530 con 0x7f93780c6f60 2023-10-30T13:29:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 335023029 0 0) 0x7f9360003530 con 0x7f93780c6f60 2023-10-30T13:29:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93780c6b80 msgr2=0x7f93780bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:38.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93780c6b80 0x7f93780bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 >> v2:172.21.15.150:3300/0 conn(0x7f937813f4a0 msgr2=0x7f93780c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:38.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 --2- 172.21.15.148:0/1396850010 >> v2:172.21.15.150:3300/0 conn(0x7f937813f4a0 0x7f93780c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937806fca0 con 0x7f93780c6f60 2023-10-30T13:29:38.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f936c001680 con 0x7f93780c6f60 2023-10-30T13:29:38.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 >> v1:172.21.15.148:6789/0 conn(0x7f93780c6f60 legacy=0x7f937813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:38.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f9360003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:38.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93780c6b80 0x7f93600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:38.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9360004d10 0x7f9360005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:38.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937806f6e0 con 0x7f9360003980 2023-10-30T13:29:38.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937806f9c0 con 0x7f9360004d10 2023-10-30T13:29:38.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936000b990 con 0x7f93780c6b80 2023-10-30T13:29:38.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937659c640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f9360003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:38.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f9376d9d640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9360004d10 0x7f9360005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:38.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f9375d9b640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93780c6b80 0x7f93600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:38.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937659c640 1 -- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93780c6b80 msgr2=0x7f93600047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:38.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937659c640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93780c6b80 0x7f93600047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937659c640 1 -- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9360004d10 msgr2=0x7f9360005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:38.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937659c640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9360004d10 0x7f9360005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f937659c640 1 -- 172.21.15.148:0/1396850010 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9360003530 con 0x7f9360003980 2023-10-30T13:29:38.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.424+0000 7f9375d9b640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93780c6b80 0x7f93600047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:38.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937659c640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f9360003da0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9368009ae0 tx=0x7f936801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad285a7db41f481 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:38.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93680a5530 con 0x7f9360003980 2023-10-30T13:29:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93680a5690 con 0x7f9360003980 2023-10-30T13:29:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937559a640 1 -- 172.21.15.148:0/1396850010 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93680a5a40 con 0x7f9360003980 2023-10-30T13:29:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937cc46640 1 -- 172.21.15.148:0/1396850010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 msgr2=0x7f9360003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937cc46640 1 --2- 172.21.15.148:0/1396850010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f9360003da0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9368009ae0 tx=0x7f936801ca50 comp rx=0 tx=0).stop 2023-10-30T13:29:38.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f9376d9d640 1 -- 172.21.15.148:0/1396850010 reap_dead start 2023-10-30T13:29:38.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937cc46640 1 -- 172.21.15.148:0/1396850010 shutdown_connections 2023-10-30T13:29:38.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937cc46640 1 -- 172.21.15.148:0/1396850010 >> 172.21.15.148:0/1396850010 conn(0x7f93780c7910 msgr2=0x7f93780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:38.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937cc46640 1 -- 172.21.15.148:0/1396850010 shutdown_connections 2023-10-30T13:29:38.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.428+0000 7f937cc46640 1 -- 172.21.15.148:0/1396850010 wait complete. 2023-10-30T13:29:38.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 Processor -- start 2023-10-30T13:29:38.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 -- start start 2023-10-30T13:29:38.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f937815de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:38.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f937815fb30 0x7f93780c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:38.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f937815e380 0x7f93780ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:38.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780ca940 con 0x7f9360003980 2023-10-30T13:29:38.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780cab30 con 0x7f937815e380 2023-10-30T13:29:38.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93781f0b10 con 0x7f937815fb30 2023-10-30T13:29:38.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f937815de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:38.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f937815de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40314/0 (socket says 172.21.15.148:40314) 2023-10-30T13:29:38.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937659c640 1 -- 172.21.15.148:0/2905491142 learned_addr learned my addr 172.21.15.148:0/2905491142 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f9376d9d640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f937815e380 0x7f93780ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f9375d9b640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f937815fb30 0x7f93780c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937659c640 1 -- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f937815fb30 msgr2=0x7f93780c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937659c640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f937815fb30 0x7f93780c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937659c640 1 -- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f937815e380 msgr2=0x7f93780ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937659c640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f937815e380 0x7f93780ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937659c640 1 -- 172.21.15.148:0/2905491142 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93781f0db0 con 0x7f9360003980 2023-10-30T13:29:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937659c640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f937815de40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f93680b0040 tx=0x7f9360005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f9366ffd640 1 -- 172.21.15.148:0/2905491142 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936800f000 con 0x7f9360003980 2023-10-30T13:29:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f9366ffd640 1 -- 172.21.15.148:0/2905491142 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9368004020 con 0x7f9360003980 2023-10-30T13:29:38.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937815d360 con 0x7f9360003980 2023-10-30T13:29:38.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f9366ffd640 1 -- 172.21.15.148:0/2905491142 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93680043d0 con 0x7f9360003980 2023-10-30T13:29:38.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.432+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937815d920 con 0x7f9360003980 2023-10-30T13:29:38.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.436+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937804e170 con 0x7f9360003980 2023-10-30T13:29:38.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.436+0000 7f9366ffd640 1 -- 172.21.15.148:0/2905491142 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f93680b4040 con 0x7f9360003980 2023-10-30T13:29:38.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.436+0000 7f9366ffd640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93580664a0 0x7f9358068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:38.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.436+0000 7f9366ffd640 1 -- 172.21.15.148:0/2905491142 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f936811d940 con 0x7f9360003980 2023-10-30T13:29:38.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.436+0000 7f9375d9b640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93580664a0 0x7f9358068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:38.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.436+0000 7f9375d9b640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93580664a0 0x7f9358068960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9378070630 tx=0x7f936c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:38.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.440+0000 7f9366ffd640 1 -- 172.21.15.148:0/2905491142 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93680eaa20 con 0x7f9360003980 2023-10-30T13:29:38.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9378157570 con 0x7f9360003980 2023-10-30T13:29:38.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f9366ffd640 1 -- 172.21.15.148:0/2905491142 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f93680ee040 con 0x7f9360003980 2023-10-30T13:29:38.617 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:38.617 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:38.618 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 87 2023-10-30T13:29:38.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93580664a0 msgr2=0x7f9358068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:38.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93580664a0 0x7f9358068960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9378070630 tx=0x7f936c009040 comp rx=0 tx=0).stop 2023-10-30T13:29:38.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 msgr2=0x7f937815de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:38.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f937815de40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f93680b0040 tx=0x7f9360005800 comp rx=0 tx=0).stop 2023-10-30T13:29:38.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 shutdown_connections 2023-10-30T13:29:38.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93580664a0 0x7f9358068960 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f937815e380 0x7f93780ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f937815fb30 0x7f93780c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 --2- 172.21.15.148:0/2905491142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9360003980 0x7f937815de40 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:38.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 >> 172.21.15.148:0/2905491142 conn(0x7f93780c7910 msgr2=0x7f937806b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:38.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 shutdown_connections 2023-10-30T13:29:38.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:38.620+0000 7f937cc46640 1 -- 172.21.15.148:0/2905491142 wait complete. 2023-10-30T13:29:38.638 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:29:40.639 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:40.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe8a0bb640 1 Processor -- start 2023-10-30T13:29:40.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe8a0bb640 1 -- start start 2023-10-30T13:29:40.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe8a0bb640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbe8415d6f0 0x7fbe8415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:40.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe8a0bb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 0x7fbe840cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:40.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe8a0bb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe840cb800 con 0x7fbe841538a0 2023-10-30T13:29:40.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe8a0bb640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe840cb330 con 0x7fbe8415d6f0 2023-10-30T13:29:40.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe8a0bb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840cad70 con 0x7fbe840cf160 2023-10-30T13:29:40.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe82ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbe8415d6f0 0x7fbe8415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:40.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe82ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbe8415d6f0 0x7fbe8415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40324/0 (socket says 172.21.15.148:40324) 2023-10-30T13:29:40.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe82ffd640 1 -- 172.21.15.148:0/2776154048 learned_addr learned my addr 172.21.15.148:0/2776154048 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:40.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe83fff640 1 --2- 172.21.15.148:0/2776154048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 0x7fbe840cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:40.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe82ffd640 1 -- 172.21.15.148:0/2776154048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 msgr2=0x7fbe840cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:40.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe82ffd640 1 --2- 172.21.15.148:0/2776154048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 0x7fbe840cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe82ffd640 1 -- 172.21.15.148:0/2776154048 >> v1:172.21.15.150:6789/0 conn(0x7fbe841538a0 legacy=0x7fbe840ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:40.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.792+0000 7fbe82ffd640 1 -- 172.21.15.148:0/2776154048 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe840cb050 con 0x7fbe8415d6f0 2023-10-30T13:29:40.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe82ffd640 1 --2- 172.21.15.148:0/2776154048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 0x7fbe8415a090 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009870 tx=0x7fbe7801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966bde1a2dd82900 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:40.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe827fc640 1 -- 172.21.15.148:0/2776154048 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe780a8a30 con 0x7fbe8415d6f0 2023-10-30T13:29:40.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe827fc640 1 -- 172.21.15.148:0/2776154048 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe780a8b90 con 0x7fbe8415d6f0 2023-10-30T13:29:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe827fc640 1 -- 172.21.15.148:0/2776154048 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe780b0440 con 0x7fbe8415d6f0 2023-10-30T13:29:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/2776154048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 msgr2=0x7fbe8415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe8a0bb640 1 --2- 172.21.15.148:0/2776154048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 0x7fbe8415a090 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fbe78009870 tx=0x7fbe7801ed30 comp rx=0 tx=0).stop 2023-10-30T13:29:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/2776154048 shutdown_connections 2023-10-30T13:29:40.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe8a0bb640 1 --2- 172.21.15.148:0/2776154048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 0x7fbe840cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe8a0bb640 1 --2- 172.21.15.148:0/2776154048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 0x7fbe8415a090 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/2776154048 >> 172.21.15.148:0/2776154048 conn(0x7fbe8406bfe0 msgr2=0x7fbe8406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:40.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/2776154048 shutdown_connections 2023-10-30T13:29:40.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.796+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/2776154048 wait complete. 2023-10-30T13:29:40.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 Processor -- start 2023-10-30T13:29:40.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 -- start start 2023-10-30T13:29:40.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 0x7fbe84158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:40.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe841538a0 0x7fbe841ef430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:40.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 0x7fbe841ef970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:40.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe841e3760 con 0x7fbe8415d6f0 2023-10-30T13:29:40.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe841e5580 con 0x7fbe841538a0 2023-10-30T13:29:40.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe841e2ae0 con 0x7fbe840cf160 2023-10-30T13:29:40.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe83fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 0x7fbe841ef970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:40.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe83fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 0x7fbe841ef970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40336/0 (socket says 172.21.15.148:40336) 2023-10-30T13:29:40.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe83fff640 1 -- 172.21.15.148:0/3367267054 learned_addr learned my addr 172.21.15.148:0/3367267054 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:40.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe837fe640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 0x7fbe84158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:40.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe82ffd640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe841538a0 0x7fbe841ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:40.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe83fff640 1 -- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 msgr2=0x7fbe84158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:40.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe83fff640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 0x7fbe84158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe83fff640 1 -- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe841538a0 msgr2=0x7fbe841ef430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:40.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe83fff640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe841538a0 0x7fbe841ef430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe83fff640 1 -- 172.21.15.148:0/3367267054 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe780094d0 con 0x7fbe8415d6f0 2023-10-30T13:29:40.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe83fff640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 0x7fbe841ef970 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c007d90 tx=0x7fbe7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:40.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe6bfff640 1 -- 172.21.15.148:0/3367267054 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c014420 con 0x7fbe8415d6f0 2023-10-30T13:29:40.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe6bfff640 1 -- 172.21.15.148:0/3367267054 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c0145a0 con 0x7fbe8415d6f0 2023-10-30T13:29:40.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe6bfff640 1 -- 172.21.15.148:0/3367267054 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c014950 con 0x7fbe8415d6f0 2023-10-30T13:29:40.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe841e2dc0 con 0x7fbe8415d6f0 2023-10-30T13:29:40.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.800+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe841e3380 con 0x7fbe8415d6f0 2023-10-30T13:29:40.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.804+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe840b2a50 con 0x7fbe8415d6f0 2023-10-30T13:29:40.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.804+0000 7fbe6bfff640 1 -- 172.21.15.148:0/3367267054 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe7c014ab0 con 0x7fbe8415d6f0 2023-10-30T13:29:40.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.804+0000 7fbe6bfff640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbe700664a0 0x7fbe70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:40.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.804+0000 7fbe6bfff640 1 -- 172.21.15.148:0/3367267054 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fbe7c084b80 con 0x7fbe8415d6f0 2023-10-30T13:29:40.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.804+0000 7fbe837fe640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbe700664a0 0x7fbe70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:40.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.804+0000 7fbe837fe640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbe700664a0 0x7fbe70068960 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbe74007fd0 tx=0x7fbe74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:40.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.808+0000 7fbe6bfff640 1 -- 172.21.15.148:0/3367267054 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe7c051b60 con 0x7fbe8415d6f0 2023-10-30T13:29:40.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.988+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe8404e170 con 0x7fbe8415d6f0 2023-10-30T13:29:40.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.988+0000 7fbe6bfff640 1 -- 172.21.15.148:0/3367267054 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fbe7c084e60 con 0x7fbe8415d6f0 2023-10-30T13:29:40.988 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:40.988 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:40.988 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 87 2023-10-30T13:29:40.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbe700664a0 msgr2=0x7fbe70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:40.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbe700664a0 0x7fbe70068960 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbe74007fd0 tx=0x7fbe74009040 comp rx=0 tx=0).stop 2023-10-30T13:29:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 msgr2=0x7fbe841ef970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 0x7fbe841ef970 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c007d90 tx=0x7fbe7c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 shutdown_connections 2023-10-30T13:29:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbe700664a0 0x7fbe70068960 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe8415d6f0 0x7fbe841ef970 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe841538a0 0x7fbe841ef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 --2- 172.21.15.148:0/3367267054 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbe840cf160 0x7fbe84158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 >> 172.21.15.148:0/3367267054 conn(0x7fbe8406bfe0 msgr2=0x7fbe840c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 shutdown_connections 2023-10-30T13:29:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:40.992+0000 7fbe8a0bb640 1 -- 172.21.15.148:0/3367267054 wait complete. 2023-10-30T13:29:41.008 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:29:43.009 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:43.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.160+0000 7fa416fb8640 1 Processor -- start 2023-10-30T13:29:43.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.160+0000 7fa416fb8640 1 -- start start 2023-10-30T13:29:43.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa416fb8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 0x7fa4100c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa416fb8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa41006a720 0x7fa41006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa416fb8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa410071290 con 0x7fa410159e60 2023-10-30T13:29:43.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa416fb8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa410071570 con 0x7fa41006a720 2023-10-30T13:29:43.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa416fb8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa410071850 con 0x7fa4100bfec0 2023-10-30T13:29:43.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa407fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 0x7fa4100c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa415555640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa41006a720 0x7fa41006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa414d54640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa410159e60 0x7fa41006f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:39802/0 (socket says 172.21.15.148:39802) 2023-10-30T13:29:43.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa414d54640 1 -- 172.21.15.148:0/1317630204 learned_addr learned my addr 172.21.15.148:0/1317630204 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:29:43.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa415555640 1 -- 172.21.15.148:0/1317630204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 msgr2=0x7fa4100c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:43.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa415555640 1 --2- 172.21.15.148:0/1317630204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 0x7fa4100c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa415555640 1 -- 172.21.15.148:0/1317630204 >> v1:172.21.15.150:6789/0 conn(0x7fa410159e60 legacy=0x7fa41006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.164+0000 7fa415555640 1 -- 172.21.15.148:0/1317630204 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa410071b30 con 0x7fa41006a720 2023-10-30T13:29:43.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa415555640 1 --2- 172.21.15.148:0/1317630204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa41006a720 0x7fa41006abb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa408009d90 tx=0x7fa40801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=471cc6ae47fdc7d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:43.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa4077fe640 1 -- 172.21.15.148:0/1317630204 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4080af600 con 0x7fa41006a720 2023-10-30T13:29:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa4077fe640 1 -- 172.21.15.148:0/1317630204 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4080ab050 con 0x7fa41006a720 2023-10-30T13:29:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa4077fe640 1 -- 172.21.15.148:0/1317630204 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4080af980 con 0x7fa41006a720 2023-10-30T13:29:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa416fb8640 1 -- 172.21.15.148:0/1317630204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa41006a720 msgr2=0x7fa41006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa416fb8640 1 --2- 172.21.15.148:0/1317630204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa41006a720 0x7fa41006abb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa408009d90 tx=0x7fa40801f130 comp rx=0 tx=0).stop 2023-10-30T13:29:43.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa416fb8640 1 -- 172.21.15.148:0/1317630204 shutdown_connections 2023-10-30T13:29:43.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa416fb8640 1 --2- 172.21.15.148:0/1317630204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 0x7fa4100c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa416fb8640 1 --2- 172.21.15.148:0/1317630204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa41006a720 0x7fa41006abb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa416fb8640 1 -- 172.21.15.148:0/1317630204 >> 172.21.15.148:0/1317630204 conn(0x7fa4100c7910 msgr2=0x7fa4100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:43.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa416fb8640 1 -- 172.21.15.148:0/1317630204 shutdown_connections 2023-10-30T13:29:43.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa416fb8640 1 -- 172.21.15.148:0/1317630204 wait complete. 2023-10-30T13:29:43.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.168+0000 7fa416fb8640 1 Processor -- start 2023-10-30T13:29:43.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 -- start start 2023-10-30T13:29:43.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa41006a720 0x7fa41014fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 0x7fa4100aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa410159e60 0x7fa4100c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa410157a70 con 0x7fa410159e60 2023-10-30T13:29:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4100afc40 con 0x7fa41006a720 2023-10-30T13:29:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa410156ef0 con 0x7fa4100bfec0 2023-10-30T13:29:43.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa415555640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa410159e60 0x7fa4100c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa415555640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa410159e60 0x7fa4100c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57994/0 (socket says 172.21.15.148:57994) 2023-10-30T13:29:43.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa415555640 1 -- 172.21.15.148:0/833740316 learned_addr learned my addr 172.21.15.148:0/833740316 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:43.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa414d54640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa41006a720 0x7fa41014fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa407fff640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 0x7fa4100aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa415555640 1 -- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 msgr2=0x7fa4100aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa415555640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 0x7fa4100aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa415555640 1 -- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa41006a720 msgr2=0x7fa41014fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa415555640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa41006a720 0x7fa41014fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa415555640 1 -- 172.21.15.148:0/833740316 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa408009a70 con 0x7fa410159e60 2023-10-30T13:29:43.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa415555640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa410159e60 0x7fa4100c8e70 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa4080b2040 tx=0x7fa40801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:43.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa4057fa640 1 -- 172.21.15.148:0/833740316 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408009450 con 0x7fa410159e60 2023-10-30T13:29:43.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa4057fa640 1 -- 172.21.15.148:0/833740316 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4080b9410 con 0x7fa410159e60 2023-10-30T13:29:43.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa4057fa640 1 -- 172.21.15.148:0/833740316 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4080b97c0 con 0x7fa410159e60 2023-10-30T13:29:43.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4101571d0 con 0x7fa410159e60 2023-10-30T13:29:43.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa410157790 con 0x7fa410159e60 2023-10-30T13:29:43.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.172+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41004e170 con 0x7fa410159e60 2023-10-30T13:29:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.176+0000 7fa4057fa640 1 -- 172.21.15.148:0/833740316 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4080b6050 con 0x7fa410159e60 2023-10-30T13:29:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.176+0000 7fa4057fa640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3fc0664a0 0x7fa3fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.176+0000 7fa4057fa640 1 -- 172.21.15.148:0/833740316 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7fa4081226e0 con 0x7fa410159e60 2023-10-30T13:29:43.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.176+0000 7fa414d54640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3fc0664a0 0x7fa3fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:43.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.176+0000 7fa414d54640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3fc0664a0 0x7fa3fc068960 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa40c005890 tx=0x7fa40c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:43.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.180+0000 7fa4057fa640 1 -- 172.21.15.148:0/833740316 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4080ef740 con 0x7fa410159e60 2023-10-30T13:29:43.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.360+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4100ad8d0 con 0x7fa410159e60 2023-10-30T13:29:43.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.360+0000 7fa4057fa640 1 -- 172.21.15.148:0/833740316 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fa4080b4090 con 0x7fa410159e60 2023-10-30T13:29:43.359 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:43.360 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:43.360 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 87 2023-10-30T13:29:43.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3fc0664a0 msgr2=0x7fa3fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3fc0664a0 0x7fa3fc068960 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa40c005890 tx=0x7fa40c009040 comp rx=0 tx=0).stop 2023-10-30T13:29:43.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa410159e60 msgr2=0x7fa4100c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:43.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa410159e60 0x7fa4100c8e70 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa4080b2040 tx=0x7fa40801bed0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 shutdown_connections 2023-10-30T13:29:43.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3fc0664a0 0x7fa3fc068960 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa410159e60 0x7fa4100c8e70 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa4100bfec0 0x7fa4100aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 --2- 172.21.15.148:0/833740316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa41006a720 0x7fa41014fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 >> 172.21.15.148:0/833740316 conn(0x7fa4100c7910 msgr2=0x7fa4100c9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 shutdown_connections 2023-10-30T13:29:43.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:43.364+0000 7fa416fb8640 1 -- 172.21.15.148:0/833740316 wait complete. 2023-10-30T13:29:43.380 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:29:45.382 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:45.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.532+0000 7f1da3c6b640 1 Processor -- start 2023-10-30T13:29:45.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.532+0000 7f1da3c6b640 1 -- start start 2023-10-30T13:29:45.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da3c6b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1d9c0c05d0 0x7f1d9c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:45.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da3c6b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d9c0c6f60 0x7f1d9c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:45.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da3c6b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c0c0a50 con 0x7f1d9c0c6b80 2023-10-30T13:29:45.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da3c6b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c06f6e0 con 0x7f1d9c0c05d0 2023-10-30T13:29:45.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da3c6b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c06f9c0 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da1206640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1d9c0c05d0 0x7f1d9c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:45.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da1206640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1d9c0c05d0 0x7f1d9c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58004/0 (socket says 172.21.15.148:58004) 2023-10-30T13:29:45.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da1206640 1 -- 172.21.15.148:0/1711157097 learned_addr learned my addr 172.21.15.148:0/1711157097 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:45.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da2208640 1 --2- 172.21.15.148:0/1711157097 >> v2:172.21.15.150:3300/0 conn(0x7f1d9c0c6f60 0x7f1d9c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:45.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da1206640 1 -- 172.21.15.148:0/1711157097 >> v2:172.21.15.150:3300/0 conn(0x7f1d9c0c6f60 msgr2=0x7f1d9c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:45.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da1206640 1 --2- 172.21.15.148:0/1711157097 >> v2:172.21.15.150:3300/0 conn(0x7f1d9c0c6f60 0x7f1d9c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:45.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da1206640 1 -- 172.21.15.148:0/1711157097 >> v1:172.21.15.150:6789/0 conn(0x7f1d9c0c6b80 legacy=0x7f1d9c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:45.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.536+0000 7f1da1206640 1 -- 172.21.15.148:0/1711157097 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c06fca0 con 0x7f1d9c0c05d0 2023-10-30T13:29:45.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da1206640 1 --2- 172.21.15.148:0/1711157097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c05d0 0x7f1d9c13ed90 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0c89b0 tx=0x7f1d8c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bf7c92e419f900 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:45.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da0a05640 1 -- 172.21.15.148:0/1711157097 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0ab610 con 0x7f1d9c0c05d0 2023-10-30T13:29:45.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da0a05640 1 -- 172.21.15.148:0/1711157097 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8c0ab770 con 0x7f1d9c0c05d0 2023-10-30T13:29:45.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da0a05640 1 -- 172.21.15.148:0/1711157097 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0abb00 con 0x7f1d9c0c05d0 2023-10-30T13:29:45.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1711157097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c05d0 msgr2=0x7f1d9c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:45.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 --2- 172.21.15.148:0/1711157097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c05d0 0x7f1d9c13ed90 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0c89b0 tx=0x7f1d8c01c500 comp rx=0 tx=0).stop 2023-10-30T13:29:45.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1711157097 shutdown_connections 2023-10-30T13:29:45.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 --2- 172.21.15.148:0/1711157097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c05d0 0x7f1d9c13ed90 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:45.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 --2- 172.21.15.148:0/1711157097 >> v2:172.21.15.150:3300/0 conn(0x7f1d9c0c6f60 0x7f1d9c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:45.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1711157097 >> 172.21.15.148:0/1711157097 conn(0x7f1d9c0c7910 msgr2=0x7f1d9c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:45.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1711157097 shutdown_connections 2023-10-30T13:29:45.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1711157097 wait complete. 2023-10-30T13:29:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 Processor -- start 2023-10-30T13:29:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.540+0000 7f1da3c6b640 1 -- start start 2023-10-30T13:29:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da3c6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c0c05d0 0x7f1d9c1f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da3c6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d9c0c6b80 0x7f1d9c0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:45.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da3c6b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c6f60 0x7f1d9c1f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da3c6b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c1f2670 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da3c6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c15f9c0 con 0x7f1d9c0c05d0 2023-10-30T13:29:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da3c6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c15fef0 con 0x7f1d9c0c6b80 2023-10-30T13:29:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da2208640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c6f60 0x7f1d9c1f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:45.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da2208640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c6f60 0x7f1d9c1f5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58008/0 (socket says 172.21.15.148:58008) 2023-10-30T13:29:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da2208640 1 -- 172.21.15.148:0/1326171269 learned_addr learned my addr 172.21.15.148:0/1326171269 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da1a07640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c0c05d0 0x7f1d9c1f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da1206640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d9c0c6b80 0x7f1d9c0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da2208640 1 -- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d9c0c6b80 msgr2=0x7f1d9c0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:45.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da2208640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d9c0c6b80 0x7f1d9c0c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da2208640 1 -- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c0c05d0 msgr2=0x7f1d9c1f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da2208640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c0c05d0 0x7f1d9c1f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da2208640 1 -- 172.21.15.148:0/1326171269 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c0094d0 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da2208640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c6f60 0x7f1d9c1f5450 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400efc0 tx=0x7f1d9400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1d927fc640 1 -- 172.21.15.148:0/1326171269 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94009ba0 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1d927fc640 1 -- 172.21.15.148:0/1326171269 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9400f040 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c15fc10 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1d927fc640 1 -- 172.21.15.148:0/1326171269 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d940146c0 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c1f2290 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9c04e170 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.544+0000 7f1d927fc640 1 -- 172.21.15.148:0/1326171269 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1d94014820 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.548+0000 7f1d927fc640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d800664a0 0x7f1d80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:45.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.548+0000 7f1d927fc640 1 -- 172.21.15.148:0/1326171269 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f1d94084f30 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.548+0000 7f1da1a07640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d800664a0 0x7f1d80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:45.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.548+0000 7f1da1a07640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d800664a0 0x7f1d80068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1d98007fd0 tx=0x7f1d98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:45.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.552+0000 7f1d927fc640 1 -- 172.21.15.148:0/1326171269 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1d94052020 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.736+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d9c0c10f0 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.736+0000 7f1d927fc640 1 -- 172.21.15.148:0/1326171269 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f1d94018020 con 0x7f1d9c0c6f60 2023-10-30T13:29:45.733 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:45.734 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:45.734 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 87 2023-10-30T13:29:45.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d800664a0 msgr2=0x7f1d80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:45.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d800664a0 0x7f1d80068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1d98007fd0 tx=0x7f1d98009040 comp rx=0 tx=0).stop 2023-10-30T13:29:45.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c6f60 msgr2=0x7f1d9c1f5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:45.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c6f60 0x7f1d9c1f5450 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400efc0 tx=0x7f1d9400c490 comp rx=0 tx=0).stop 2023-10-30T13:29:45.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 shutdown_connections 2023-10-30T13:29:45.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d800664a0 0x7f1d80068960 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:45.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d9c0c6f60 0x7f1d9c1f5450 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:45.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d9c0c6b80 0x7f1d9c0c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:45.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 --2- 172.21.15.148:0/1326171269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c0c05d0 0x7f1d9c1f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:45.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 >> 172.21.15.148:0/1326171269 conn(0x7f1d9c0c7910 msgr2=0x7f1d9c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:45.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 shutdown_connections 2023-10-30T13:29:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:45.740+0000 7f1da3c6b640 1 -- 172.21.15.148:0/1326171269 wait complete. 2023-10-30T13:29:45.755 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:29:47.757 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:47.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.904+0000 7f9429bd3640 1 Processor -- start 2023-10-30T13:29:47.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.904+0000 7f9429bd3640 1 -- start start 2023-10-30T13:29:47.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9429bd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f942406f7a0 0x7f942406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:47.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9429bd3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f94240c6bd0 0x7f9424069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:47.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9429bd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9424071110 0x7f9424158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:47.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9429bd3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94240700c0 con 0x7f94240c6bd0 2023-10-30T13:29:47.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9429bd3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f942406a3d0 con 0x7f9424071110 2023-10-30T13:29:47.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9429bd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9424066cf0 con 0x7f942406f7a0 2023-10-30T13:29:47.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f941bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f94240c6bd0 0x7f9424069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:47.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f941bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f94240c6bd0 0x7f9424069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58016/0 (socket says 172.21.15.148:58016) 2023-10-30T13:29:47.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f941bfff640 1 -- 172.21.15.148:0/1760175060 learned_addr learned my addr 172.21.15.148:0/1760175060 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:47.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9423fff640 1 --2- 172.21.15.148:0/1760175060 >> v2:172.21.15.150:3300/0 conn(0x7f9424071110 0x7f9424158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:47.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f94237fe640 1 --2- 172.21.15.148:0/1760175060 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f942406f7a0 0x7f942406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:47.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f941bfff640 1 -- 172.21.15.148:0/1760175060 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f942406f7a0 msgr2=0x7f942406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:47.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f941bfff640 1 --2- 172.21.15.148:0/1760175060 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f942406f7a0 0x7f942406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:47.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f941bfff640 1 -- 172.21.15.148:0/1760175060 >> v2:172.21.15.150:3300/0 conn(0x7f9424071110 msgr2=0x7f9424158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:47.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f941bfff640 1 --2- 172.21.15.148:0/1760175060 >> v2:172.21.15.150:3300/0 conn(0x7f9424071110 0x7f9424158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:47.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f941bfff640 1 -- 172.21.15.148:0/1760175060 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94240c70d0 con 0x7f94240c6bd0 2023-10-30T13:29:47.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f941bfff640 1 --2- 172.21.15.148:0/1760175060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94240c6bd0 0x7f9424069e90 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f942406c7d0 tx=0x7f940c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec77180f4bf890 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:47.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9422ffd640 1 -- 172.21.15.148:0/1760175060 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c0a89b0 con 0x7f94240c6bd0 2023-10-30T13:29:47.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9422ffd640 1 -- 172.21.15.148:0/1760175060 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c0a8b10 con 0x7f94240c6bd0 2023-10-30T13:29:47.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.908+0000 7f9422ffd640 1 -- 172.21.15.148:0/1760175060 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c0a8ea0 con 0x7f94240c6bd0 2023-10-30T13:29:47.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 -- 172.21.15.148:0/1760175060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94240c6bd0 msgr2=0x7f9424069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:47.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1760175060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94240c6bd0 0x7f9424069e90 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f942406c7d0 tx=0x7f940c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:29:47.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 -- 172.21.15.148:0/1760175060 shutdown_connections 2023-10-30T13:29:47.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1760175060 >> v2:172.21.15.150:3300/0 conn(0x7f9424071110 0x7f9424158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:47.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1760175060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f94240c6bd0 0x7f9424069e90 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:47.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1760175060 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f942406f7a0 0x7f942406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:47.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 -- 172.21.15.148:0/1760175060 >> 172.21.15.148:0/1760175060 conn(0x7f94240c7910 msgr2=0x7f94240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:47.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 -- 172.21.15.148:0/1760175060 shutdown_connections 2023-10-30T13:29:47.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 -- 172.21.15.148:0/1760175060 wait complete. 2023-10-30T13:29:47.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 Processor -- start 2023-10-30T13:29:47.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 -- start start 2023-10-30T13:29:47.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942406f7a0 0x7f94240c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:47.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9424071110 0x7f94240c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:47.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94240c6bd0 0x7f94240add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:47.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94240c63a0 con 0x7f9424071110 2023-10-30T13:29:47.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942414f600 con 0x7f942406f7a0 2023-10-30T13:29:47.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9429bd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94240ad8d0 con 0x7f94240c6bd0 2023-10-30T13:29:47.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f941bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9424071110 0x7f94240c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:47.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f941bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9424071110 0x7f94240c5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58020/0 (socket says 172.21.15.148:58020) 2023-10-30T13:29:47.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f941bfff640 1 -- 172.21.15.148:0/1615954456 learned_addr learned my addr 172.21.15.148:0/1615954456 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:47.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f94237fe640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942406f7a0 0x7f94240c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:47.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f9423fff640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94240c6bd0 0x7f94240add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:47.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f941bfff640 1 -- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94240c6bd0 msgr2=0x7f94240add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:47.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f941bfff640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94240c6bd0 0x7f94240add60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:47.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.912+0000 7f941bfff640 1 -- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942406f7a0 msgr2=0x7f94240c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:47.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f941bfff640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942406f7a0 0x7f94240c3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:47.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f941bfff640 1 -- 172.21.15.148:0/1615954456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c0094d0 con 0x7f9424071110 2023-10-30T13:29:47.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f941bfff640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9424071110 0x7f94240c5740 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f940c01d3a0 tx=0x7f940c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:47.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f9420ff9640 1 -- 172.21.15.148:0/1615954456 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c0ba600 con 0x7f9424071110 2023-10-30T13:29:47.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f9420ff9640 1 -- 172.21.15.148:0/1615954456 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c0ba760 con 0x7f9424071110 2023-10-30T13:29:47.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94240af6f0 con 0x7f9424071110 2023-10-30T13:29:47.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f9420ff9640 1 -- 172.21.15.148:0/1615954456 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c0bab10 con 0x7f9424071110 2023-10-30T13:29:47.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942406bcd0 con 0x7f9424071110 2023-10-30T13:29:47.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f942406bec0 con 0x7f9424071110 2023-10-30T13:29:47.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f9420ff9640 1 -- 172.21.15.148:0/1615954456 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f940c0bac70 con 0x7f9424071110 2023-10-30T13:29:47.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f9420ff9640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94040664f0 0x7f94040689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:47.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f9420ff9640 1 -- 172.21.15.148:0/1615954456 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6628+0+0 (secure 0 0 0) 0x7f940c122590 con 0x7f9424071110 2023-10-30T13:29:47.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.916+0000 7f94237fe640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94040664f0 0x7f94040689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:47.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.920+0000 7f94237fe640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94040664f0 0x7f94040689b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f941400b990 tx=0x7f9414008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:47.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:47.920+0000 7f9420ff9640 1 -- 172.21.15.148:0/1615954456 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f942406bec0 con 0x7f9424071110 2023-10-30T13:29:48.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94240ada80 con 0x7f9424071110 2023-10-30T13:29:48.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9420ff9640 1 -- 172.21.15.148:0/1615954456 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f94240ada80 con 0x7f9424071110 2023-10-30T13:29:48.101 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:48.101 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6408,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:28:42.945044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6249":{"gid":6249,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":24,"addr":"172.21.15.148:6837/2740075781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2740075781},{"type":"v1","addr":"172.21.15.148:6837","nonce":2740075781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:48.102 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 87 2023-10-30T13:29:48.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94040664f0 msgr2=0x7f94040689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:48.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94040664f0 0x7f94040689b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f941400b990 tx=0x7f9414008040 comp rx=0 tx=0).stop 2023-10-30T13:29:48.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9424071110 msgr2=0x7f94240c5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:48.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9424071110 0x7f94240c5740 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f940c01d3a0 tx=0x7f940c01ba90 comp rx=0 tx=0).stop 2023-10-30T13:29:48.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 shutdown_connections 2023-10-30T13:29:48.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f94040664f0 0x7f94040689b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:48.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9424071110 0x7f94240c5740 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:48.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f94240c6bd0 0x7f94240add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:48.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 --2- 172.21.15.148:0/1615954456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942406f7a0 0x7f94240c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:48.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 >> 172.21.15.148:0/1615954456 conn(0x7f94240c7910 msgr2=0x7f9424150be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:48.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 shutdown_connections 2023-10-30T13:29:48.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:48.104+0000 7f9429bd3640 1 -- 172.21.15.148:0/1615954456 wait complete. 2023-10-30T13:29:48.122 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:29:50.124 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:50.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.276+0000 7f90e4a05640 1 Processor -- start 2023-10-30T13:29:50.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.276+0000 7f90e4a05640 1 -- start start 2023-10-30T13:29:50.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90e4a05640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f90e00c6b80 0x7f90e00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90e4a05640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 0x7f90e00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90e4a05640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f90e00c8430 0x7f90e013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90e4a05640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e00c0a50 con 0x7f90e00c6b80 2023-10-30T13:29:50.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90e4a05640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e006f6e0 con 0x7f90e00c8430 2023-10-30T13:29:50.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90e4a05640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e006f9c0 con 0x7f90e00c6f60 2023-10-30T13:29:50.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90de59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f90e00c6b80 0x7f90e00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90de59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f90e00c6b80 0x7f90e00bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58028/0 (socket says 172.21.15.148:58028) 2023-10-30T13:29:50.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90de59c640 1 -- 172.21.15.148:0/1933942502 learned_addr learned my addr 172.21.15.148:0/1933942502 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:50.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90ddd9b640 1 --2- 172.21.15.148:0/1933942502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 0x7f90e00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90ded9d640 1 --2- 172.21.15.148:0/1933942502 >> v2:172.21.15.150:3300/0 conn(0x7f90e00c8430 0x7f90e013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90de59c640 1 -- 172.21.15.148:0/1933942502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 msgr2=0x7f90e00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:50.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90de59c640 1 --2- 172.21.15.148:0/1933942502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 0x7f90e00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90de59c640 1 -- 172.21.15.148:0/1933942502 >> v2:172.21.15.150:3300/0 conn(0x7f90e00c8430 msgr2=0x7f90e013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:50.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90de59c640 1 --2- 172.21.15.148:0/1933942502 >> v2:172.21.15.150:3300/0 conn(0x7f90e00c8430 0x7f90e013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90de59c640 1 -- 172.21.15.148:0/1933942502 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e006fca0 con 0x7f90e00c6b80 2023-10-30T13:29:50.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.280+0000 7f90de59c640 1 --2- 172.21.15.148:0/1933942502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 0x7f90e00bfec0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f90e00b0860 tx=0x7f90d001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8dfaf65c76d64d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:50.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90dd59a640 1 -- 172.21.15.148:0/1933942502 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d00ab600 con 0x7f90e00c6b80 2023-10-30T13:29:50.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90dd59a640 1 -- 172.21.15.148:0/1933942502 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d00a7050 con 0x7f90e00c6b80 2023-10-30T13:29:50.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90dd59a640 1 -- 172.21.15.148:0/1933942502 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d00ab980 con 0x7f90e00c6b80 2023-10-30T13:29:50.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 -- 172.21.15.148:0/1933942502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 msgr2=0x7f90e00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 --2- 172.21.15.148:0/1933942502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 0x7f90e00bfec0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f90e00b0860 tx=0x7f90d001c470 comp rx=0 tx=0).stop 2023-10-30T13:29:50.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 -- 172.21.15.148:0/1933942502 shutdown_connections 2023-10-30T13:29:50.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 --2- 172.21.15.148:0/1933942502 >> v2:172.21.15.150:3300/0 conn(0x7f90e00c8430 0x7f90e013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 --2- 172.21.15.148:0/1933942502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 0x7f90e00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 --2- 172.21.15.148:0/1933942502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 0x7f90e00bfec0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 -- 172.21.15.148:0/1933942502 >> 172.21.15.148:0/1933942502 conn(0x7f90e00c7910 msgr2=0x7f90e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:50.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 -- 172.21.15.148:0/1933942502 shutdown_connections 2023-10-30T13:29:50.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 -- 172.21.15.148:0/1933942502 wait complete. 2023-10-30T13:29:50.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 Processor -- start 2023-10-30T13:29:50.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 -- start start 2023-10-30T13:29:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 0x7f90e01341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 0x7f90e0153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e00c8430 0x7f90e0153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e013e280 con 0x7f90e00c6b80 2023-10-30T13:29:50.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e0134710 con 0x7f90e00c8430 2023-10-30T13:29:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.284+0000 7f90e4a05640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e0132900 con 0x7f90e00c6f60 2023-10-30T13:29:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90de59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 0x7f90e01341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90de59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 0x7f90e01341d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58040/0 (socket says 172.21.15.148:58040) 2023-10-30T13:29:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90de59c640 1 -- 172.21.15.148:0/136499918 learned_addr learned my addr 172.21.15.148:0/136499918 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:50.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90ddd9b640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 0x7f90e0153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90ded9d640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e00c8430 0x7f90e0153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90de59c640 1 -- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 msgr2=0x7f90e0153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90de59c640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 0x7f90e0153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90de59c640 1 -- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e00c8430 msgr2=0x7f90e0153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90de59c640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e00c8430 0x7f90e0153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90de59c640 1 -- 172.21.15.148:0/136499918 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d00094d0 con 0x7f90e00c6b80 2023-10-30T13:29:50.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90de59c640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 0x7f90e01341d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f90d001ab20 tx=0x7f90d0019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:50.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90ceffd640 1 -- 172.21.15.148:0/136499918 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d00b35d0 con 0x7f90e00c6b80 2023-10-30T13:29:50.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90ceffd640 1 -- 172.21.15.148:0/136499918 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d00b3730 con 0x7f90e00c6b80 2023-10-30T13:29:50.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90ceffd640 1 -- 172.21.15.148:0/136499918 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d00b3aa0 con 0x7f90e00c6b80 2023-10-30T13:29:50.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e013e7b0 con 0x7f90e00c6b80 2023-10-30T13:29:50.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e01521b0 con 0x7f90e00c6b80 2023-10-30T13:29:50.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e004e170 con 0x7f90e00c6b80 2023-10-30T13:29:50.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90ceffd640 1 -- 172.21.15.148:0/136499918 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90d00b2070 con 0x7f90e00c6b80 2023-10-30T13:29:50.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.288+0000 7f90ceffd640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90c00664a0 0x7f90c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.292+0000 7f90ceffd640 1 -- 172.21.15.148:0/136499918 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f90d011e360 con 0x7f90e00c6b80 2023-10-30T13:29:50.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.292+0000 7f90ddd9b640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90c00664a0 0x7f90c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.292+0000 7f90ddd9b640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90c00664a0 0x7f90c0068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f90e0070830 tx=0x7f90d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:50.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.292+0000 7f90ceffd640 1 -- 172.21.15.148:0/136499918 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f90d00eb460 con 0x7f90e00c6b80 2023-10-30T13:29:50.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.472+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90e00c8150 con 0x7f90e00c6b80 2023-10-30T13:29:50.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.472+0000 7f90ceffd640 1 -- 172.21.15.148:0/136499918 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5713 (secure 0 0 0) 0x7f90d00eea80 con 0x7f90e00c6b80 2023-10-30T13:29:50.471 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:50.471 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:29:49.670639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:50.472 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 88 2023-10-30T13:29:50.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90c00664a0 msgr2=0x7f90c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90c00664a0 0x7f90c0068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f90e0070830 tx=0x7f90d4008040 comp rx=0 tx=0).stop 2023-10-30T13:29:50.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 msgr2=0x7f90e01341d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 0x7f90e01341d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f90d001ab20 tx=0x7f90d0019210 comp rx=0 tx=0).stop 2023-10-30T13:29:50.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 shutdown_connections 2023-10-30T13:29:50.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90c00664a0 0x7f90c0068960 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e00c8430 0x7f90e0153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90e00c6f60 0x7f90e0153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 --2- 172.21.15.148:0/136499918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f90e00c6b80 0x7f90e01341d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 >> 172.21.15.148:0/136499918 conn(0x7f90e00c7910 msgr2=0x7f90e00c18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:50.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 shutdown_connections 2023-10-30T13:29:50.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.476+0000 7f90e4a05640 1 -- 172.21.15.148:0/136499918 wait complete. 2023-10-30T13:29:50.491 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:29:50.492 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:29:50.492 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:50.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdefe4640 1 Processor -- start 2023-10-30T13:29:50.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdefe4640 1 -- start start 2023-10-30T13:29:50.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdefe4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fafd81538f0 0x7fafd8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdefe4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fafd815cd50 0x7fafd80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdefe4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd815d300 con 0x7fafd80cefa0 2023-10-30T13:29:50.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdefe4640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd815d5e0 con 0x7fafd815cd50 2023-10-30T13:29:50.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdefe4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd815d8c0 con 0x7fafd81538f0 2023-10-30T13:29:50.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdd7e1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fafd815cd50 0x7fafd80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafde7e3640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fafd80cefa0 0x7fafd80cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39964/0 (socket says 172.21.15.148:39964) 2023-10-30T13:29:50.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafde7e3640 1 -- 172.21.15.148:0/3042242542 learned_addr learned my addr 172.21.15.148:0/3042242542 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:29:50.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafddfe2640 1 --2- 172.21.15.148:0/3042242542 >> v2:172.21.15.150:3300/0 conn(0x7fafd81538f0 0x7fafd8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdd7e1640 1 -- 172.21.15.148:0/3042242542 >> v2:172.21.15.150:3300/0 conn(0x7fafd81538f0 msgr2=0x7fafd8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:50.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdd7e1640 1 --2- 172.21.15.148:0/3042242542 >> v2:172.21.15.150:3300/0 conn(0x7fafd81538f0 0x7fafd8159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdd7e1640 1 -- 172.21.15.148:0/3042242542 >> v1:172.21.15.148:6789/0 conn(0x7fafd80cefa0 legacy=0x7fafd80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdd7e1640 1 -- 172.21.15.148:0/3042242542 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd815dba0 con 0x7fafd815cd50 2023-10-30T13:29:50.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.636+0000 7fafdcfe0640 1 -- 172.21.15.148:0/3042242542 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777592057 0 0) 0x7fafd815d300 con 0x7fafd80cefa0 2023-10-30T13:29:50.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdd7e1640 1 --2- 172.21.15.148:0/3042242542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd815cd50 0x7fafd80c9f90 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fafd80c9cb0 tx=0x7fafc801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6b0634912009d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:50.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdcfe0640 1 -- 172.21.15.148:0/3042242542 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc80a8950 con 0x7fafd815cd50 2023-10-30T13:29:50.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdcfe0640 1 -- 172.21.15.148:0/3042242542 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc80a8ab0 con 0x7fafd815cd50 2023-10-30T13:29:50.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdcfe0640 1 -- 172.21.15.148:0/3042242542 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc80a8e40 con 0x7fafd815cd50 2023-10-30T13:29:50.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdefe4640 1 -- 172.21.15.148:0/3042242542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd815cd50 msgr2=0x7fafd80c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdefe4640 1 --2- 172.21.15.148:0/3042242542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd815cd50 0x7fafd80c9f90 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fafd80c9cb0 tx=0x7fafc801ec50 comp rx=0 tx=0).stop 2023-10-30T13:29:50.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdefe4640 1 -- 172.21.15.148:0/3042242542 shutdown_connections 2023-10-30T13:29:50.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdefe4640 1 --2- 172.21.15.148:0/3042242542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd815cd50 0x7fafd80c9f90 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdefe4640 1 --2- 172.21.15.148:0/3042242542 >> v2:172.21.15.150:3300/0 conn(0x7fafd81538f0 0x7fafd8159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdefe4640 1 -- 172.21.15.148:0/3042242542 >> 172.21.15.148:0/3042242542 conn(0x7fafd806a640 msgr2=0x7fafd8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:50.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdefe4640 1 -- 172.21.15.148:0/3042242542 shutdown_connections 2023-10-30T13:29:50.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.640+0000 7fafdefe4640 1 -- 172.21.15.148:0/3042242542 wait complete. 2023-10-30T13:29:50.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 Processor -- start 2023-10-30T13:29:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 -- start start 2023-10-30T13:29:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd80cefa0 0x7fafd815b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fafd81538f0 0x7fafd80b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd815cd50 0x7fafd8144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd815bdd0 con 0x7fafd80cefa0 2023-10-30T13:29:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd81f2cd0 con 0x7fafd815cd50 2023-10-30T13:29:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd81f20c0 con 0x7fafd81538f0 2023-10-30T13:29:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafddfe2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd80cefa0 0x7fafd815b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafddfe2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd80cefa0 0x7fafd815b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58046/0 (socket says 172.21.15.148:58046) 2023-10-30T13:29:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafddfe2640 1 -- 172.21.15.148:0/2310870738 learned_addr learned my addr 172.21.15.148:0/2310870738 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafde7e3640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd815cd50 0x7fafd8144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdd7e1640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fafd81538f0 0x7fafd80b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafddfe2640 1 -- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fafd81538f0 msgr2=0x7fafd80b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafddfe2640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fafd81538f0 0x7fafd80b4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafddfe2640 1 -- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd815cd50 msgr2=0x7fafd8144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafddfe2640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd815cd50 0x7fafd8144810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafddfe2640 1 -- 172.21.15.148:0/2310870738 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc8009420 con 0x7fafd80cefa0 2023-10-30T13:29:50.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdd7e1640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fafd81538f0 0x7fafd80b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:29:50.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafddfe2640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd80cefa0 0x7fafd815b7b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fafd400d830 tx=0x7fafd400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:50.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafce7fc640 1 -- 172.21.15.148:0/2310870738 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd400b6c0 con 0x7fafd80cefa0 2023-10-30T13:29:50.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafce7fc640 1 -- 172.21.15.148:0/2310870738 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd400b860 con 0x7fafd80cefa0 2023-10-30T13:29:50.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd81f23a0 con 0x7fafd80cefa0 2023-10-30T13:29:50.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafce7fc640 1 -- 172.21.15.148:0/2310870738 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd400bc10 con 0x7fafd80cefa0 2023-10-30T13:29:50.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd81f2960 con 0x7fafd80cefa0 2023-10-30T13:29:50.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.644+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd804e170 con 0x7fafd80cefa0 2023-10-30T13:29:50.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.648+0000 7fafce7fc640 1 -- 172.21.15.148:0/2310870738 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fafd401e460 con 0x7fafd80cefa0 2023-10-30T13:29:50.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.648+0000 7fafce7fc640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fafbc0664d0 0x7fafbc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:50.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.648+0000 7fafce7fc640 1 -- 172.21.15.148:0/2310870738 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fafd4084a20 con 0x7fafd80cefa0 2023-10-30T13:29:50.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.648+0000 7fafdd7e1640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fafbc0664d0 0x7fafbc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:50.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.648+0000 7fafdd7e1640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fafbc0664d0 0x7fafbc068990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fafc80b2000 tx=0x7fafc80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:50.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.652+0000 7fafce7fc640 1 -- 172.21.15.148:0/2310870738 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fafd4051aa0 con 0x7fafd80cefa0 2023-10-30T13:29:50.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.832+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafd80b2290 con 0x7fafd80cefa0 2023-10-30T13:29:50.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafce7fc640 1 -- 172.21.15.148:0/2310870738 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5713 (secure 0 0 0) 0x7fafd40550c0 con 0x7fafd80cefa0 2023-10-30T13:29:50.832 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:50.832 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:29:49.670639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:50.832 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 88 2023-10-30T13:29:50.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fafbc0664d0 msgr2=0x7fafbc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fafbc0664d0 0x7fafbc068990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fafc80b2000 tx=0x7fafc80aa070 comp rx=0 tx=0).stop 2023-10-30T13:29:50.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd80cefa0 msgr2=0x7fafd815b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:50.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd80cefa0 0x7fafd815b7b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fafd400d830 tx=0x7fafd400dcf0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 shutdown_connections 2023-10-30T13:29:50.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fafbc0664d0 0x7fafbc068990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd815cd50 0x7fafd8144810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fafd81538f0 0x7fafd80b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 --2- 172.21.15.148:0/2310870738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fafd80cefa0 0x7fafd815b7b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:50.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 >> 172.21.15.148:0/2310870738 conn(0x7fafd806a640 msgr2=0x7fafd80c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:50.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 shutdown_connections 2023-10-30T13:29:50.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:50.836+0000 7fafdefe4640 1 -- 172.21.15.148:0/2310870738 wait complete. 2023-10-30T13:29:50.853 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 88, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 88, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6408": { "addr": "172.21.15.148:6835/172087710", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 172087710, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 172087710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6408, "incarnation": 88, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:29:49.670639+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6408 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:29:52.854 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:53.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.004+0000 7f30afbf5640 1 Processor -- start 2023-10-30T13:29:53.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.004+0000 7f30afbf5640 1 -- start start 2023-10-30T13:29:53.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30afbf5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f30a80c6f60 0x7f30a81566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:53.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30afbf5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a80c0a50 con 0x7f30a80c05d0 2023-10-30T13:29:53.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30afbf5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a806f6e0 con 0x7f30a80c6b80 2023-10-30T13:29:53.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30afbf5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a806f9c0 con 0x7f30a80c6f60 2023-10-30T13:29:53.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad190640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f30a80c05d0 0x7f30a813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39282/0 (socket says 172.21.15.148:39282) 2023-10-30T13:29:53.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad190640 1 -- 172.21.15.148:0/2136229392 learned_addr learned my addr 172.21.15.148:0/2136229392 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:29:53.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ae192640 1 --2- 172.21.15.148:0/2136229392 >> v2:172.21.15.150:3300/0 conn(0x7f30a80c6f60 0x7f30a81566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:53.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852824496 0 0) 0x7f30a80c0a50 con 0x7f30a80c05d0 2023-10-30T13:29:53.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3094003530 con 0x7f30a80c05d0 2023-10-30T13:29:53.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394010636 0 0) 0x7f3094003530 con 0x7f30a80c05d0 2023-10-30T13:29:53.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 >> v2:172.21.15.150:3300/0 conn(0x7f30a80c6f60 msgr2=0x7f30a81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:53.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 --2- 172.21.15.148:0/2136229392 >> v2:172.21.15.150:3300/0 conn(0x7f30a80c6f60 0x7f30a81566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 >> v1:172.21.15.150:6789/0 conn(0x7f30a80c6b80 legacy=0x7f30a80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:53.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a806fca0 con 0x7f30a80c05d0 2023-10-30T13:29:53.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f3098001680 con 0x7f30a80c05d0 2023-10-30T13:29:53.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 >> v1:172.21.15.148:6789/0 conn(0x7f30a80c05d0 legacy=0x7f30a813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:53.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f3094003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:53.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30940047d0 0x7f3094004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:53.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3094005190 0x7f309400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:53.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a806f9c0 con 0x7f3094003980 2023-10-30T13:29:53.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309400c030 con 0x7f30940047d0 2023-10-30T13:29:53.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309400c1f0 con 0x7f3094005190 2023-10-30T13:29:53.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad991640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f3094003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:53.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad190640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30940047d0 0x7f3094004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:53.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ae192640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3094005190 0x7f309400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:53.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad991640 1 -- 172.21.15.148:0/2136229392 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3094005190 msgr2=0x7f309400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:53.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad991640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3094005190 0x7f309400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad991640 1 -- 172.21.15.148:0/2136229392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30940047d0 msgr2=0x7f3094004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:53.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad991640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30940047d0 0x7f3094004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad991640 1 -- 172.21.15.148:0/2136229392 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094003530 con 0x7f3094003980 2023-10-30T13:29:53.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ae192640 1 -- 172.21.15.148:0/2136229392 reap_dead start 2023-10-30T13:29:53.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.008+0000 7f30ad190640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30940047d0 0x7f3094004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:53.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30ad991640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f3094003da0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f30a4001780 tx=0x7f30a401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb152e728afe382 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:53.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a40a4ea0 con 0x7f3094003980 2023-10-30T13:29:53.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a40a7050 con 0x7f3094003980 2023-10-30T13:29:53.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30ac98f640 1 -- 172.21.15.148:0/2136229392 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4003600 con 0x7f3094003980 2023-10-30T13:29:53.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30afbf5640 1 -- 172.21.15.148:0/2136229392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 msgr2=0x7f3094003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:53.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30afbf5640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f3094003da0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f30a4001780 tx=0x7f30a401c3e0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30afbf5640 1 -- 172.21.15.148:0/2136229392 shutdown_connections 2023-10-30T13:29:53.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30afbf5640 1 --2- 172.21.15.148:0/2136229392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f3094003da0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30afbf5640 1 -- 172.21.15.148:0/2136229392 >> 172.21.15.148:0/2136229392 conn(0x7f30a80c7910 msgr2=0x7f30a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:53.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30afbf5640 1 -- 172.21.15.148:0/2136229392 shutdown_connections 2023-10-30T13:29:53.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.012+0000 7f30afbf5640 1 -- 172.21.15.148:0/2136229392 wait complete. 2023-10-30T13:29:53.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 Processor -- start 2023-10-30T13:29:53.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 -- start start 2023-10-30T13:29:53.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f30a8155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:53.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a80c9210 0x7f30a80ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:53.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a80ca640 0x7f30a813da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:53.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a80caca0 con 0x7f3094003980 2023-10-30T13:29:53.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a80caac0 con 0x7f30a80ca640 2023-10-30T13:29:53.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a8154f10 con 0x7f30a80c9210 2023-10-30T13:29:53.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad991640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f30a8155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:53.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad991640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f30a8155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47652/0 (socket says 172.21.15.148:47652) 2023-10-30T13:29:53.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad991640 1 -- 172.21.15.148:0/3630493141 learned_addr learned my addr 172.21.15.148:0/3630493141 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:53.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad190640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a80c9210 0x7f30a80ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:53.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ae192640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a80ca640 0x7f30a813da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:53.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad991640 1 -- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a80c9210 msgr2=0x7f30a80ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:53.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad991640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a80c9210 0x7f30a80ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad991640 1 -- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a80ca640 msgr2=0x7f30a813da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:53.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad991640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a80ca640 0x7f30a813da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad991640 1 -- 172.21.15.148:0/3630493141 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a813c9c0 con 0x7f3094003980 2023-10-30T13:29:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30ad991640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f30a8155400 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f30a4005040 tx=0x7f3094005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f309e7fc640 1 -- 172.21.15.148:0/3630493141 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4007440 con 0x7f3094003980 2023-10-30T13:29:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f309e7fc640 1 -- 172.21.15.148:0/3630493141 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a40075a0 con 0x7f3094003980 2023-10-30T13:29:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a813cca0 con 0x7f3094003980 2023-10-30T13:29:53.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f309e7fc640 1 -- 172.21.15.148:0/3630493141 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a40069e0 con 0x7f3094003980 2023-10-30T13:29:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a813d260 con 0x7f3094003980 2023-10-30T13:29:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.016+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a804e170 con 0x7f3094003980 2023-10-30T13:29:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.020+0000 7f309e7fc640 1 -- 172.21.15.148:0/3630493141 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f30a4006b40 con 0x7f3094003980 2023-10-30T13:29:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.020+0000 7f309e7fc640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f308c066490 0x7f308c068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.020+0000 7f309e7fc640 1 -- 172.21.15.148:0/3630493141 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f30a411e430 con 0x7f3094003980 2023-10-30T13:29:53.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.020+0000 7f30ad190640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f308c066490 0x7f308c068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:53.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.020+0000 7f30ad190640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f308c066490 0x7f308c068950 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f30a8138660 tx=0x7f3098009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:53.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.024+0000 7f309e7fc640 1 -- 172.21.15.148:0/3630493141 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f30a40eb4b0 con 0x7f3094003980 2023-10-30T13:29:53.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30a80705c0 con 0x7f3094003980 2023-10-30T13:29:53.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f309e7fc640 1 -- 172.21.15.148:0/3630493141 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5713 (secure 0 0 0) 0x7f30a40eead0 con 0x7f3094003980 2023-10-30T13:29:53.200 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:53.200 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:29:49.670639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:53.201 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 88 2023-10-30T13:29:53.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f308c066490 msgr2=0x7f308c068950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:53.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f308c066490 0x7f308c068950 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f30a8138660 tx=0x7f3098009040 comp rx=0 tx=0).stop 2023-10-30T13:29:53.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 msgr2=0x7f30a8155400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:53.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f30a8155400 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f30a4005040 tx=0x7f3094005800 comp rx=0 tx=0).stop 2023-10-30T13:29:53.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 shutdown_connections 2023-10-30T13:29:53.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f308c066490 0x7f308c068950 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30a80ca640 0x7f30a813da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30a80c9210 0x7f30a80ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 --2- 172.21.15.148:0/3630493141 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3094003980 0x7f30a8155400 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:53.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 >> 172.21.15.148:0/3630493141 conn(0x7f30a80c7910 msgr2=0x7f30a8139170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:53.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 shutdown_connections 2023-10-30T13:29:53.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:53.204+0000 7f30afbf5640 1 -- 172.21.15.148:0/3630493141 wait complete. 2023-10-30T13:29:53.221 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:55.222 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:55.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f7107180640 1 Processor -- start 2023-10-30T13:29:55.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f7107180640 1 -- start start 2023-10-30T13:29:55.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f7107180640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f710015d6f0 0x7f710015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:55.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f7107180640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 0x7f71000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:55.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f7107180640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71000cb800 con 0x7f71001538a0 2023-10-30T13:29:55.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f7107180640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71000cb330 con 0x7f710015d6f0 2023-10-30T13:29:55.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f7107180640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71000cad70 con 0x7f71000cf160 2023-10-30T13:29:55.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f70fffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f710015d6f0 0x7f710015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:55.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f7104f1c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f71001538a0 0x7f71000ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39284/0 (socket says 172.21.15.148:39284) 2023-10-30T13:29:55.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f7104f1c640 1 -- 172.21.15.148:0/3169094333 learned_addr learned my addr 172.21.15.148:0/3169094333 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:29:55.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f710571d640 1 --2- 172.21.15.148:0/3169094333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 0x7f71000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:55.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f70fffff640 1 -- 172.21.15.148:0/3169094333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 msgr2=0x7f71000cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:29:55.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f70fffff640 1 --2- 172.21.15.148:0/3169094333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 0x7f71000cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f70fffff640 1 -- 172.21.15.148:0/3169094333 >> v1:172.21.15.148:6789/0 conn(0x7f71001538a0 legacy=0x7f71000ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:55.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f70ff7fe640 1 -- 172.21.15.148:0/3169094333 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000546743 0 0) 0x7f71000cb800 con 0x7f71001538a0 2023-10-30T13:29:55.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.372+0000 7f70fffff640 1 -- 172.21.15.148:0/3169094333 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71000cb050 con 0x7f710015d6f0 2023-10-30T13:29:55.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f70fffff640 1 --2- 172.21.15.148:0/3169094333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 0x7f710015a090 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0097c0 tx=0x7f70ec01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=944ca6c59781a940 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:55.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f70ff7fe640 1 -- 172.21.15.148:0/3169094333 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec0a8970 con 0x7f710015d6f0 2023-10-30T13:29:55.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f70ff7fe640 1 -- 172.21.15.148:0/3169094333 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec0a8ad0 con 0x7f710015d6f0 2023-10-30T13:29:55.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f70ff7fe640 1 -- 172.21.15.148:0/3169094333 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec0a8e60 con 0x7f710015d6f0 2023-10-30T13:29:55.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f7107180640 1 -- 172.21.15.148:0/3169094333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 msgr2=0x7f710015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:55.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f7107180640 1 --2- 172.21.15.148:0/3169094333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 0x7f710015a090 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0097c0 tx=0x7f70ec01ec70 comp rx=0 tx=0).stop 2023-10-30T13:29:55.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f7107180640 1 -- 172.21.15.148:0/3169094333 shutdown_connections 2023-10-30T13:29:55.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f7107180640 1 --2- 172.21.15.148:0/3169094333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 0x7f710015a090 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f7107180640 1 --2- 172.21.15.148:0/3169094333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 0x7f71000cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f7107180640 1 -- 172.21.15.148:0/3169094333 >> 172.21.15.148:0/3169094333 conn(0x7f710006bfe0 msgr2=0x7f710006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:55.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f7107180640 1 -- 172.21.15.148:0/3169094333 shutdown_connections 2023-10-30T13:29:55.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.376+0000 7f7107180640 1 -- 172.21.15.148:0/3169094333 wait complete. 2023-10-30T13:29:55.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 Processor -- start 2023-10-30T13:29:55.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 -- start start 2023-10-30T13:29:55.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 0x7f7100158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71001538a0 0x7f7100158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 0x7f71001f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710014f630 con 0x7f710015d6f0 2023-10-30T13:29:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71001e2b40 con 0x7f71001538a0 2023-10-30T13:29:55.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7100157ec0 con 0x7f71000cf160 2023-10-30T13:29:55.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f710571d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 0x7f71001f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:55.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f710571d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 0x7f71001f2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47670/0 (socket says 172.21.15.148:47670) 2023-10-30T13:29:55.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f710571d640 1 -- 172.21.15.148:0/1042936290 learned_addr learned my addr 172.21.15.148:0/1042936290 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:55.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7104f1c640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 0x7f7100158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:55.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f70fffff640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71001538a0 0x7f7100158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:55.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f710571d640 1 -- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 msgr2=0x7f7100158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:55.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f710571d640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 0x7f7100158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f710571d640 1 -- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71001538a0 msgr2=0x7f7100158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:55.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f710571d640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71001538a0 0x7f7100158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f710571d640 1 -- 172.21.15.148:0/1042936290 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec009420 con 0x7f710015d6f0 2023-10-30T13:29:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f710571d640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 0x7f71001f2a70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f70f8007dc0 tx=0x7f70f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f70fd7fa640 1 -- 172.21.15.148:0/1042936290 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8014420 con 0x7f710015d6f0 2023-10-30T13:29:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f70fd7fa640 1 -- 172.21.15.148:0/1042936290 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f80145a0 con 0x7f710015d6f0 2023-10-30T13:29:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710014e770 con 0x7f710015d6f0 2023-10-30T13:29:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f70fd7fa640 1 -- 172.21.15.148:0/1042936290 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8014950 con 0x7f710015d6f0 2023-10-30T13:29:55.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.380+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710014ec90 con 0x7f710015d6f0 2023-10-30T13:29:55.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.384+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71000b2a50 con 0x7f710015d6f0 2023-10-30T13:29:55.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.384+0000 7f70fd7fa640 1 -- 172.21.15.148:0/1042936290 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70f8014ab0 con 0x7f710015d6f0 2023-10-30T13:29:55.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.384+0000 7f70fd7fa640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70e40664d0 0x7f70e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:55.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.384+0000 7f70fd7fa640 1 -- 172.21.15.148:0/1042936290 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f70f8084a10 con 0x7f710015d6f0 2023-10-30T13:29:55.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.384+0000 7f7104f1c640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70e40664d0 0x7f70e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:55.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.384+0000 7f7104f1c640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70e40664d0 0x7f70e4068990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f70f4007fd0 tx=0x7f70f4007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:55.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.388+0000 7f70fd7fa640 1 -- 172.21.15.148:0/1042936290 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70f8051b50 con 0x7f710015d6f0 2023-10-30T13:29:55.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.580+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7100158070 con 0x7f710015d6f0 2023-10-30T13:29:55.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.580+0000 7f70fd7fa640 1 -- 172.21.15.148:0/1042936290 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5713 (secure 0 0 0) 0x7f7100158070 con 0x7f710015d6f0 2023-10-30T13:29:55.579 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:55.579 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:29:49.670639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:55.579 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 88 2023-10-30T13:29:55.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70e40664d0 msgr2=0x7f70e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:55.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70e40664d0 0x7f70e4068990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f70f4007fd0 tx=0x7f70f4007d60 comp rx=0 tx=0).stop 2023-10-30T13:29:55.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 msgr2=0x7f71001f2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:55.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 0x7f71001f2a70 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f70f8007dc0 tx=0x7f70f800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 shutdown_connections 2023-10-30T13:29:55.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70e40664d0 0x7f70e4068990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f710015d6f0 0x7f71001f2a70 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71001538a0 0x7f7100158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 --2- 172.21.15.148:0/1042936290 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f71000cf160 0x7f7100158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:55.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 >> 172.21.15.148:0/1042936290 conn(0x7f710006bfe0 msgr2=0x7f710006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:55.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 shutdown_connections 2023-10-30T13:29:55.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:55.584+0000 7f7107180640 1 -- 172.21.15.148:0/1042936290 wait complete. 2023-10-30T13:29:55.599 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:29:57.601 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:29:57.720 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:29:57.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:29:57.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.804+0000 7fa67c11d640 1 Processor -- start 2023-10-30T13:29:57.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.804+0000 7fa67c11d640 1 -- start start 2023-10-30T13:29:57.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa67c11d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa67406f7a0 0x7fa67406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:57.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa67c11d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa674066cf0 con 0x7fa674071110 2023-10-30T13:29:57.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa67c11d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6740c70d0 con 0x7fa6740c6bd0 2023-10-30T13:29:57.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa67c11d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6740700c0 con 0x7fa67406f7a0 2023-10-30T13:29:57.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa67a6ba640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa674071110 0x7fa674158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:39292/0 (socket says 172.21.15.148:39292) 2023-10-30T13:29:57.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa67a6ba640 1 -- 172.21.15.148:0/1902569770 learned_addr learned my addr 172.21.15.148:0/1902569770 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:29:57.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa679eb9640 1 --2- 172.21.15.148:0/1902569770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa67406f7a0 0x7fa67406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:57.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa678eb7640 1 -- 172.21.15.148:0/1902569770 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63677769 0 0) 0x7fa674066cf0 con 0x7fa674071110 2023-10-30T13:29:57.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa678eb7640 1 -- 172.21.15.148:0/1902569770 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa660003530 con 0x7fa674071110 2023-10-30T13:29:57.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa678eb7640 1 -- 172.21.15.148:0/1902569770 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116354045 0 0) 0x7fa6740c70d0 con 0x7fa6740c6bd0 2023-10-30T13:29:57.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa678eb7640 1 -- 172.21.15.148:0/1902569770 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa674066cf0 con 0x7fa6740c6bd0 2023-10-30T13:29:57.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa679eb9640 1 -- 172.21.15.148:0/1902569770 >> v1:172.21.15.150:6789/0 conn(0x7fa6740c6bd0 legacy=0x7fa674069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:57.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa679eb9640 1 -- 172.21.15.148:0/1902569770 >> v1:172.21.15.148:6789/0 conn(0x7fa674071110 legacy=0x7fa674158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:57.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.808+0000 7fa679eb9640 1 -- 172.21.15.148:0/1902569770 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6740c7390 con 0x7fa67406f7a0 2023-10-30T13:29:57.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa679eb9640 1 --2- 172.21.15.148:0/1902569770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa67406f7a0 0x7fa67406fb80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa6700097f0 tx=0x7fa67001ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0aca22588e36577 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:57.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa678eb7640 1 -- 172.21.15.148:0/1902569770 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700a89b0 con 0x7fa67406f7a0 2023-10-30T13:29:57.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa678eb7640 1 -- 172.21.15.148:0/1902569770 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6700a8b10 con 0x7fa67406f7a0 2023-10-30T13:29:57.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa678eb7640 1 -- 172.21.15.148:0/1902569770 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700a8ea0 con 0x7fa67406f7a0 2023-10-30T13:29:57.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa67c11d640 1 -- 172.21.15.148:0/1902569770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa67406f7a0 msgr2=0x7fa67406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:57.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa67c11d640 1 --2- 172.21.15.148:0/1902569770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa67406f7a0 0x7fa67406fb80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa6700097f0 tx=0x7fa67001ecb0 comp rx=0 tx=0).stop 2023-10-30T13:29:57.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa67c11d640 1 -- 172.21.15.148:0/1902569770 shutdown_connections 2023-10-30T13:29:57.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa67c11d640 1 --2- 172.21.15.148:0/1902569770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa67406f7a0 0x7fa67406fb80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:57.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa67c11d640 1 -- 172.21.15.148:0/1902569770 >> 172.21.15.148:0/1902569770 conn(0x7fa6740c7910 msgr2=0x7fa6740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:57.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa67c11d640 1 -- 172.21.15.148:0/1902569770 shutdown_connections 2023-10-30T13:29:57.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa67c11d640 1 -- 172.21.15.148:0/1902569770 wait complete. 2023-10-30T13:29:57.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.812+0000 7fa67c11d640 1 Processor -- start 2023-10-30T13:29:57.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 -- start start 2023-10-30T13:29:57.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa67406f7a0 0x7fa67415b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:57.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa674071110 0x7fa67414ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:57.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6740c6bd0 0x7fa674155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:57.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741ec8c0 con 0x7fa67406f7a0 2023-10-30T13:29:57.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa674156930 con 0x7fa6740c6bd0 2023-10-30T13:29:57.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741ecb80 con 0x7fa674071110 2023-10-30T13:29:57.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa679eb9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa67406f7a0 0x7fa67415b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:57.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa679eb9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa67406f7a0 0x7fa67415b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47678/0 (socket says 172.21.15.148:47678) 2023-10-30T13:29:57.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa679eb9640 1 -- 172.21.15.148:0/1557570813 learned_addr learned my addr 172.21.15.148:0/1557570813 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:29:57.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67a6ba640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6740c6bd0 0x7fa674155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:57.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa6796b8640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa674071110 0x7fa67414ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:57.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa679eb9640 1 -- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa674071110 msgr2=0x7fa67414ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:57.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa679eb9640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa674071110 0x7fa67414ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:57.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa679eb9640 1 -- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6740c6bd0 msgr2=0x7fa674155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa679eb9640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6740c6bd0 0x7fa674155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa679eb9640 1 -- 172.21.15.148:0/1557570813 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa670009420 con 0x7fa67406f7a0 2023-10-30T13:29:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa6796b8640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa674071110 0x7fa67414ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:29:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa679eb9640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa67406f7a0 0x7fa67415b7d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa67001b8c0 tx=0x7fa67001ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa66a7fc640 1 -- 172.21.15.148:0/1557570813 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700b0d60 con 0x7fa67406f7a0 2023-10-30T13:29:57.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa66a7fc640 1 -- 172.21.15.148:0/1557570813 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6700b9430 con 0x7fa67406f7a0 2023-10-30T13:29:57.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6741ece60 con 0x7fa67406f7a0 2023-10-30T13:29:57.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa66a7fc640 1 -- 172.21.15.148:0/1557570813 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700b97e0 con 0x7fa67406f7a0 2023-10-30T13:29:57.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6741ed420 con 0x7fa67406f7a0 2023-10-30T13:29:57.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.816+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67404e170 con 0x7fa67406f7a0 2023-10-30T13:29:57.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.820+0000 7fa66a7fc640 1 -- 172.21.15.148:0/1557570813 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6700b7040 con 0x7fa67406f7a0 2023-10-30T13:29:57.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.820+0000 7fa66a7fc640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6580664a0 0x7fa658068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:29:57.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.820+0000 7fa66a7fc640 1 -- 172.21.15.148:0/1557570813 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fa670121cd0 con 0x7fa67406f7a0 2023-10-30T13:29:57.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.820+0000 7fa6796b8640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6580664a0 0x7fa658068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:29:57.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.820+0000 7fa6796b8640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6580664a0 0x7fa658068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa674150260 tx=0x7fa664009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:29:57.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:57.824+0000 7fa66a7fc640 1 -- 172.21.15.148:0/1557570813 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6700eec30 con 0x7fa67406f7a0 2023-10-30T13:29:58.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.052+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa674070600 con 0x7fa67406f7a0 2023-10-30T13:29:58.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.052+0000 7fa66a7fc640 1 -- 172.21.15.148:0/1557570813 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5713 (secure 0 0 0) 0x7fa6700b5070 con 0x7fa67406f7a0 2023-10-30T13:29:58.051 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:29:58.051 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:29:49.670639+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:29:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 88 2023-10-30T13:29:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6580664a0 msgr2=0x7fa658068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:58.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6580664a0 0x7fa658068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa674150260 tx=0x7fa664009040 comp rx=0 tx=0).stop 2023-10-30T13:29:58.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa67406f7a0 msgr2=0x7fa67415b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:29:58.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa67406f7a0 0x7fa67415b7d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa67001b8c0 tx=0x7fa67001ba50 comp rx=0 tx=0).stop 2023-10-30T13:29:58.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 shutdown_connections 2023-10-30T13:29:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa6580664a0 0x7fa658068960 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6740c6bd0 0x7fa674155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa674071110 0x7fa67414ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 --2- 172.21.15.148:0/1557570813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa67406f7a0 0x7fa67415b7d0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:29:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 >> 172.21.15.148:0/1557570813 conn(0x7fa6740c7910 msgr2=0x7fa6740c61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:29:58.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 shutdown_connections 2023-10-30T13:29:58.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:29:58.056+0000 7fa67c11d640 1 -- 172.21.15.148:0/1557570813 wait complete. 2023-10-30T13:29:58.072 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:00.073 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:00.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1f069e640 1 Processor -- start 2023-10-30T13:30:00.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1f069e640 1 -- start start 2023-10-30T13:30:00.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1f069e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa1e81538a0 0x7fa1e815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:00.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1f069e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 0x7fa1e815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:00.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1f069e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e80cb800 con 0x7fa1e80cf160 2023-10-30T13:30:00.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1f069e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e815dc30 con 0x7fa1e81538a0 2023-10-30T13:30:00.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1f069e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80cb330 con 0x7fa1e80ca0e0 2023-10-30T13:30:00.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1ee43a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa1e81538a0 0x7fa1e815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:00.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1ee43a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa1e81538a0 0x7fa1e815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47690/0 (socket says 172.21.15.148:47690) 2023-10-30T13:30:00.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1ee43a640 1 -- 172.21.15.148:0/1474339505 learned_addr learned my addr 172.21.15.148:0/1474339505 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:00.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1edc39640 1 --2- 172.21.15.148:0/1474339505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 0x7fa1e815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:00.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1ee43a640 1 -- 172.21.15.148:0/1474339505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 msgr2=0x7fa1e815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:00.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1ee43a640 1 --2- 172.21.15.148:0/1474339505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 0x7fa1e815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1ee43a640 1 -- 172.21.15.148:0/1474339505 >> v1:172.21.15.150:6789/0 conn(0x7fa1e80cf160 legacy=0x7fa1e80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:00.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.216+0000 7fa1ee43a640 1 -- 172.21.15.148:0/1474339505 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80cad70 con 0x7fa1e81538a0 2023-10-30T13:30:00.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1ee43a640 1 --2- 172.21.15.148:0/1474339505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e81538a0 0x7fa1e815d6f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009870 tx=0x7fa1e401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d231f474528aea6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:00.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1ed438640 1 -- 172.21.15.148:0/1474339505 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40a89b0 con 0x7fa1e81538a0 2023-10-30T13:30:00.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1ed438640 1 -- 172.21.15.148:0/1474339505 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40a8b10 con 0x7fa1e81538a0 2023-10-30T13:30:00.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1ed438640 1 -- 172.21.15.148:0/1474339505 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40a8ea0 con 0x7fa1e81538a0 2023-10-30T13:30:00.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1f069e640 1 -- 172.21.15.148:0/1474339505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e81538a0 msgr2=0x7fa1e815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:00.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1f069e640 1 --2- 172.21.15.148:0/1474339505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e81538a0 0x7fa1e815d6f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009870 tx=0x7fa1e401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1f069e640 1 -- 172.21.15.148:0/1474339505 shutdown_connections 2023-10-30T13:30:00.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1f069e640 1 --2- 172.21.15.148:0/1474339505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 0x7fa1e815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1f069e640 1 --2- 172.21.15.148:0/1474339505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e81538a0 0x7fa1e815d6f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1f069e640 1 -- 172.21.15.148:0/1474339505 >> 172.21.15.148:0/1474339505 conn(0x7fa1e806bfe0 msgr2=0x7fa1e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:00.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1f069e640 1 -- 172.21.15.148:0/1474339505 shutdown_connections 2023-10-30T13:30:00.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.220+0000 7fa1f069e640 1 -- 172.21.15.148:0/1474339505 wait complete. 2023-10-30T13:30:00.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 Processor -- start 2023-10-30T13:30:00.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 -- start start 2023-10-30T13:30:00.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 0x7fa1e81e2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:00.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e80cf160 0x7fa1e8158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:00.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e81538a0 0x7fa1e81ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:00.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81e3810 con 0x7fa1e80cf160 2023-10-30T13:30:00.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81f3100 con 0x7fa1e81538a0 2023-10-30T13:30:00.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81f3590 con 0x7fa1e80ca0e0 2023-10-30T13:30:00.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1edc39640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e80cf160 0x7fa1e8158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:00.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1edc39640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e80cf160 0x7fa1e8158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47700/0 (socket says 172.21.15.148:47700) 2023-10-30T13:30:00.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1edc39640 1 -- 172.21.15.148:0/2934130478 learned_addr learned my addr 172.21.15.148:0/2934130478 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:00.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1eec3b640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e81538a0 0x7fa1e81ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:00.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1ee43a640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 0x7fa1e81e2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:00.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1edc39640 1 -- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 msgr2=0x7fa1e81e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:00.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1edc39640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 0x7fa1e81e2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1edc39640 1 -- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e81538a0 msgr2=0x7fa1e81ef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:00.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1edc39640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e81538a0 0x7fa1e81ef130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1edc39640 1 -- 172.21.15.148:0/2934130478 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e40094d0 con 0x7fa1e80cf160 2023-10-30T13:30:00.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1edc39640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e80cf160 0x7fa1e8158a90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800d830 tx=0x7fa1d800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:00.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1deffd640 1 -- 172.21.15.148:0/2934130478 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d800b6c0 con 0x7fa1e80cf160 2023-10-30T13:30:00.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1deffd640 1 -- 172.21.15.148:0/2934130478 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d800b820 con 0x7fa1e80cf160 2023-10-30T13:30:00.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1deffd640 1 -- 172.21.15.148:0/2934130478 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d800bb90 con 0x7fa1e80cf160 2023-10-30T13:30:00.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e81585e0 con 0x7fa1e80cf160 2023-10-30T13:30:00.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e81f27d0 con 0x7fa1e80cf160 2023-10-30T13:30:00.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.224+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e80b2a50 con 0x7fa1e80cf160 2023-10-30T13:30:00.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.228+0000 7fa1deffd640 1 -- 172.21.15.148:0/2934130478 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1d801e460 con 0x7fa1e80cf160 2023-10-30T13:30:00.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.228+0000 7fa1deffd640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1cc0664a0 0x7fa1cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:00.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.228+0000 7fa1deffd640 1 -- 172.21.15.148:0/2934130478 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fa1d80848c0 con 0x7fa1e80cf160 2023-10-30T13:30:00.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.228+0000 7fa1ee43a640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1cc0664a0 0x7fa1cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:00.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.228+0000 7fa1ee43a640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1cc0664a0 0x7fa1cc068960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40b2000 tx=0x7fa1e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:00.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.232+0000 7fa1deffd640 1 -- 172.21.15.148:0/2934130478 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1d80519e0 con 0x7fa1e80cf160 2023-10-30T13:30:00.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1e804e170 con 0x7fa1e80cf160 2023-10-30T13:30:00.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1deffd640 1 -- 172.21.15.148:0/2934130478 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fa1d8055000 con 0x7fa1e80cf160 2023-10-30T13:30:00.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:30:00.409 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:29:59.219971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:00.410 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 89 2023-10-30T13:30:00.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1cc0664a0 msgr2=0x7fa1cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:00.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1cc0664a0 0x7fa1cc068960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40b2000 tx=0x7fa1e4009040 comp rx=0 tx=0).stop 2023-10-30T13:30:00.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e80cf160 msgr2=0x7fa1e8158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:00.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e80cf160 0x7fa1e8158a90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800d830 tx=0x7fa1d800dcf0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 shutdown_connections 2023-10-30T13:30:00.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa1cc0664a0 0x7fa1cc068960 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1e80cf160 0x7fa1e8158a90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1e81538a0 0x7fa1e81ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 --2- 172.21.15.148:0/2934130478 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa1e80ca0e0 0x7fa1e81e2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:00.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 >> 172.21.15.148:0/2934130478 conn(0x7fa1e806bfe0 msgr2=0x7fa1e80c4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:00.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 shutdown_connections 2023-10-30T13:30:00.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:00.412+0000 7fa1f069e640 1 -- 172.21.15.148:0/2934130478 wait complete. 2023-10-30T13:30:00.429 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:02.431 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:02.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd16a09c640 1 Processor -- start 2023-10-30T13:30:02.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd16a09c640 1 -- start start 2023-10-30T13:30:02.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd16a09c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd1640c6bd0 0x7fd16406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:02.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd16a09c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 0x7fd164158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:02.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd16a09c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd164066cf0 con 0x7fd16406f7a0 2023-10-30T13:30:02.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd16a09c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1640700c0 con 0x7fd1640c6bd0 2023-10-30T13:30:02.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd16a09c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640c70d0 con 0x7fd164071110 2023-10-30T13:30:02.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd15bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd1640c6bd0 0x7fd16406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:02.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd15bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd1640c6bd0 0x7fd16406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39344/0 (socket says 172.21.15.148:39344) 2023-10-30T13:30:02.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd15bfff640 1 -- 172.21.15.148:0/1673369353 learned_addr learned my addr 172.21.15.148:0/1673369353 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:02.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd163fff640 1 --2- 172.21.15.148:0/1673369353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 0x7fd164158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:02.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd15bfff640 1 -- 172.21.15.148:0/1673369353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 msgr2=0x7fd164158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:02.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd15bfff640 1 --2- 172.21.15.148:0/1673369353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 0x7fd164158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd15bfff640 1 -- 172.21.15.148:0/1673369353 >> v1:172.21.15.150:6789/0 conn(0x7fd16406f7a0 legacy=0x7fd164069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:02.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.584+0000 7fd15bfff640 1 -- 172.21.15.148:0/1673369353 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640c7390 con 0x7fd1640c6bd0 2023-10-30T13:30:02.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd15bfff640 1 --2- 172.21.15.148:0/1673369353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd1640c6bd0 0x7fd16406fb80 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009870 tx=0x7fd14c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ebf166a8be959c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:02.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd162ffd640 1 -- 172.21.15.148:0/1673369353 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0a89b0 con 0x7fd1640c6bd0 2023-10-30T13:30:02.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd162ffd640 1 -- 172.21.15.148:0/1673369353 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c0a8b10 con 0x7fd1640c6bd0 2023-10-30T13:30:02.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd162ffd640 1 -- 172.21.15.148:0/1673369353 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0a8ea0 con 0x7fd1640c6bd0 2023-10-30T13:30:02.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd16a09c640 1 -- 172.21.15.148:0/1673369353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd1640c6bd0 msgr2=0x7fd16406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:02.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd16a09c640 1 --2- 172.21.15.148:0/1673369353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd1640c6bd0 0x7fd16406fb80 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009870 tx=0x7fd14c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd16a09c640 1 -- 172.21.15.148:0/1673369353 shutdown_connections 2023-10-30T13:30:02.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd16a09c640 1 --2- 172.21.15.148:0/1673369353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 0x7fd164158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd16a09c640 1 --2- 172.21.15.148:0/1673369353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd1640c6bd0 0x7fd16406fb80 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd16a09c640 1 -- 172.21.15.148:0/1673369353 >> 172.21.15.148:0/1673369353 conn(0x7fd1640c7910 msgr2=0x7fd1640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:02.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd16a09c640 1 -- 172.21.15.148:0/1673369353 shutdown_connections 2023-10-30T13:30:02.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.588+0000 7fd16a09c640 1 -- 172.21.15.148:0/1673369353 wait complete. 2023-10-30T13:30:02.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 Processor -- start 2023-10-30T13:30:02.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 -- start start 2023-10-30T13:30:02.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16406f7a0 0x7fd16414f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:02.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 0x7fd16414fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:02.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1640c6bd0 0x7fd1641f0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:02.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640c1f90 con 0x7fd16406f7a0 2023-10-30T13:30:02.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640aec90 con 0x7fd1640c6bd0 2023-10-30T13:30:02.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640c1cb0 con 0x7fd164071110 2023-10-30T13:30:02.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd1637fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16406f7a0 0x7fd16414f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:02.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd1637fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16406f7a0 0x7fd16414f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39358/0 (socket says 172.21.15.148:39358) 2023-10-30T13:30:02.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd1637fe640 1 -- 172.21.15.148:0/3941968312 learned_addr learned my addr 172.21.15.148:0/3941968312 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:02.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd15bfff640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 0x7fd16414fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:02.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd163fff640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1640c6bd0 0x7fd1641f0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:02.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd1637fe640 1 -- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 msgr2=0x7fd16414fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:02.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd1637fe640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 0x7fd16414fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd1637fe640 1 -- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1640c6bd0 msgr2=0x7fd1641f0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:02.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd1637fe640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1640c6bd0 0x7fd1641f0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd1637fe640 1 -- 172.21.15.148:0/3941968312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c0094d0 con 0x7fd16406f7a0 2023-10-30T13:30:02.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd1637fe640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16406f7a0 0x7fd16414f770 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd15400b410 tx=0x7fd15400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:02.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd160ff9640 1 -- 172.21.15.148:0/3941968312 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154004020 con 0x7fd16406f7a0 2023-10-30T13:30:02.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd160ff9640 1 -- 172.21.15.148:0/3941968312 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd154004180 con 0x7fd16406f7a0 2023-10-30T13:30:02.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd160ff9640 1 -- 172.21.15.148:0/3941968312 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154011630 con 0x7fd16406f7a0 2023-10-30T13:30:02.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1640c14c0 con 0x7fd16406f7a0 2023-10-30T13:30:02.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1640c17a0 con 0x7fd16406f7a0 2023-10-30T13:30:02.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.592+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1640af170 con 0x7fd16406f7a0 2023-10-30T13:30:02.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.596+0000 7fd160ff9640 1 -- 172.21.15.148:0/3941968312 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd154011790 con 0x7fd16406f7a0 2023-10-30T13:30:02.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.596+0000 7fd160ff9640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1440664d0 0x7fd144068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:02.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.596+0000 7fd160ff9640 1 -- 172.21.15.148:0/3941968312 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fd154083ff0 con 0x7fd16406f7a0 2023-10-30T13:30:02.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.596+0000 7fd15bfff640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1440664d0 0x7fd144068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:02.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.596+0000 7fd15bfff640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1440664d0 0x7fd144068990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0b2000 tx=0x7fd14c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:02.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.600+0000 7fd160ff9640 1 -- 172.21.15.148:0/3941968312 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd154051040 con 0x7fd16406f7a0 2023-10-30T13:30:02.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd16404e170 con 0x7fd16406f7a0 2023-10-30T13:30:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd160ff9640 1 -- 172.21.15.148:0/3941968312 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fd154054660 con 0x7fd16406f7a0 2023-10-30T13:30:02.777 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:30:02.777 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:29:59.219971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 89 2023-10-30T13:30:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1440664d0 msgr2=0x7fd144068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1440664d0 0x7fd144068990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0b2000 tx=0x7fd14c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:30:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16406f7a0 msgr2=0x7fd16414f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16406f7a0 0x7fd16414f770 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd15400b410 tx=0x7fd15400b8d0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 shutdown_connections 2023-10-30T13:30:02.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd1440664d0 0x7fd144068990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1640c6bd0 0x7fd1641f0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd164071110 0x7fd16414fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 --2- 172.21.15.148:0/3941968312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd16406f7a0 0x7fd16414f770 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:02.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 >> 172.21.15.148:0/3941968312 conn(0x7fd1640c7910 msgr2=0x7fd1640ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:02.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 shutdown_connections 2023-10-30T13:30:02.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:02.780+0000 7fd16a09c640 1 -- 172.21.15.148:0/3941968312 wait complete. 2023-10-30T13:30:02.798 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6408": { "addr": "172.21.15.148:6835/172087710", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 172087710, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 172087710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6408, "incarnation": 88, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:29:59.219971+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6408 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:30:04.799 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:04.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7fade5c9c640 1 Processor -- start 2023-10-30T13:30:04.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7fade5c9c640 1 -- start start 2023-10-30T13:30:04.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7fade5c9c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fade00c6b80 0x7fade00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:04.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7fade5c9c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fade00c6f60 0x7fade00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:04.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7fade5c9c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c8430 0x7fade013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:04.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7fade5c9c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade00c0a50 con 0x7fade00c6f60 2023-10-30T13:30:04.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7fade5c9c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade006f6e0 con 0x7fade00c6b80 2023-10-30T13:30:04.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7fade5c9c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade006f9c0 con 0x7fade00c8430 2023-10-30T13:30:04.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddeffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fade00c6f60 0x7fade00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:04.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddeffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fade00c6f60 0x7fade00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39360/0 (socket says 172.21.15.148:39360) 2023-10-30T13:30:04.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddeffd640 1 -- 172.21.15.148:0/103839495 learned_addr learned my addr 172.21.15.148:0/103839495 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:04.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddf7fe640 1 --2- 172.21.15.148:0/103839495 >> v2:172.21.15.150:3300/0 conn(0x7fade00c6b80 0x7fade00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:04.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddffff640 1 --2- 172.21.15.148:0/103839495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c8430 0x7fade013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:04.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddeffd640 1 -- 172.21.15.148:0/103839495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c8430 msgr2=0x7fade013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:04.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddeffd640 1 --2- 172.21.15.148:0/103839495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c8430 0x7fade013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:04.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddeffd640 1 -- 172.21.15.148:0/103839495 >> v2:172.21.15.150:3300/0 conn(0x7fade00c6b80 msgr2=0x7fade00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:04.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddeffd640 1 --2- 172.21.15.148:0/103839495 >> v2:172.21.15.150:3300/0 conn(0x7fade00c6b80 0x7fade00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:04.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.952+0000 7faddeffd640 1 -- 172.21.15.148:0/103839495 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade006fca0 con 0x7fade00c6f60 2023-10-30T13:30:04.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7faddeffd640 1 --2- 172.21.15.148:0/103839495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c6f60 0x7fade00c0400 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fade006f1b0 tx=0x7fadd401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4ed95a88048259 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:04.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fadde7fc640 1 -- 172.21.15.148:0/103839495 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd40a7b70 con 0x7fade00c6f60 2023-10-30T13:30:04.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fadde7fc640 1 -- 172.21.15.148:0/103839495 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd40a7cd0 con 0x7fade00c6f60 2023-10-30T13:30:04.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fadde7fc640 1 -- 172.21.15.148:0/103839495 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd40af660 con 0x7fade00c6f60 2023-10-30T13:30:04.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fade5c9c640 1 -- 172.21.15.148:0/103839495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c6f60 msgr2=0x7fade00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fade5c9c640 1 --2- 172.21.15.148:0/103839495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c6f60 0x7fade00c0400 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fade006f1b0 tx=0x7fadd401ebd0 comp rx=0 tx=0).stop 2023-10-30T13:30:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fade5c9c640 1 -- 172.21.15.148:0/103839495 shutdown_connections 2023-10-30T13:30:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fade5c9c640 1 --2- 172.21.15.148:0/103839495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c8430 0x7fade013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fade5c9c640 1 --2- 172.21.15.148:0/103839495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c6f60 0x7fade00c0400 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fade5c9c640 1 --2- 172.21.15.148:0/103839495 >> v2:172.21.15.150:3300/0 conn(0x7fade00c6b80 0x7fade00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fade5c9c640 1 -- 172.21.15.148:0/103839495 >> 172.21.15.148:0/103839495 conn(0x7fade00c7910 msgr2=0x7fade00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:04.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fade5c9c640 1 -- 172.21.15.148:0/103839495 shutdown_connections 2023-10-30T13:30:04.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.956+0000 7fade5c9c640 1 -- 172.21.15.148:0/103839495 wait complete. 2023-10-30T13:30:04.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 Processor -- start 2023-10-30T13:30:04.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 -- start start 2023-10-30T13:30:04.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fade00c6b80 0x7fade013c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:04.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c6f60 0x7fade013d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:04.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c8430 0x7fade015dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:04.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade0157de0 con 0x7fade00c8430 2023-10-30T13:30:04.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade0139020 con 0x7fade00c6b80 2023-10-30T13:30:04.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade0157950 con 0x7fade00c6f60 2023-10-30T13:30:04.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c8430 0x7fade015dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:04.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c8430 0x7fade015dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39368/0 (socket says 172.21.15.148:39368) 2023-10-30T13:30:04.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddffff640 1 -- 172.21.15.148:0/2691173581 learned_addr learned my addr 172.21.15.148:0/2691173581 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:04.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddf7fe640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fade00c6b80 0x7fade013c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:04.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddeffd640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c6f60 0x7fade013d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:04.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddffff640 1 -- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c6f60 msgr2=0x7fade013d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:04.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddffff640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c6f60 0x7fade013d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:04.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddffff640 1 -- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fade00c6b80 msgr2=0x7fade013c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:04.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddffff640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fade00c6b80 0x7fade013c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:04.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddffff640 1 -- 172.21.15.148:0/2691173581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd400b770 con 0x7fade00c8430 2023-10-30T13:30:04.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddf7fe640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fade00c6b80 0x7fade013c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:04.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7faddffff640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c8430 0x7fade015dff0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fadd8007e60 tx=0x7fadd800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:04.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fadc7fff640 1 -- 172.21.15.148:0/2691173581 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd8014420 con 0x7fade00c8430 2023-10-30T13:30:04.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fadc7fff640 1 -- 172.21.15.148:0/2691173581 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd80145a0 con 0x7fade00c8430 2023-10-30T13:30:04.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fadc7fff640 1 -- 172.21.15.148:0/2691173581 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd8014950 con 0x7fade00c8430 2023-10-30T13:30:04.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade01591e0 con 0x7fade00c8430 2023-10-30T13:30:04.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.960+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade01597a0 con 0x7fade00c8430 2023-10-30T13:30:04.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.964+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade0157ac0 con 0x7fade00c8430 2023-10-30T13:30:04.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.964+0000 7fadc7fff640 1 -- 172.21.15.148:0/2691173581 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fadd8014ab0 con 0x7fade00c8430 2023-10-30T13:30:04.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.964+0000 7fadc7fff640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadcc0664a0 0x7fadcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:04.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.964+0000 7fadc7fff640 1 -- 172.21.15.148:0/2691173581 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fadd8084bb0 con 0x7fade00c8430 2023-10-30T13:30:04.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.964+0000 7faddf7fe640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadcc0664a0 0x7fadcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:04.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.964+0000 7faddf7fe640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadcc0664a0 0x7fadcc068960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fadd0004660 tx=0x7fadd0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:04.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:04.968+0000 7fadc7fff640 1 -- 172.21.15.148:0/2691173581 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fadd8051ba0 con 0x7fade00c8430 2023-10-30T13:30:05.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fade004e170 con 0x7fade00c8430 2023-10-30T13:30:05.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fadc7fff640 1 -- 172.21.15.148:0/2691173581 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fadd80551c0 con 0x7fade00c8430 2023-10-30T13:30:05.145 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:30:05.145 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:29:59.219971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:05.145 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 89 2023-10-30T13:30:05.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadcc0664a0 msgr2=0x7fadcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadcc0664a0 0x7fadcc068960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fadd0004660 tx=0x7fadd0008040 comp rx=0 tx=0).stop 2023-10-30T13:30:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c8430 msgr2=0x7fade015dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c8430 0x7fade015dff0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fadd8007e60 tx=0x7fadd800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:30:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 shutdown_connections 2023-10-30T13:30:05.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadcc0664a0 0x7fadcc068960 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fade00c8430 0x7fade015dff0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fade00c6f60 0x7fade013d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 --2- 172.21.15.148:0/2691173581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fade00c6b80 0x7fade013c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 >> 172.21.15.148:0/2691173581 conn(0x7fade00c7910 msgr2=0x7fade0134aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 shutdown_connections 2023-10-30T13:30:05.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:05.148+0000 7fade5c9c640 1 -- 172.21.15.148:0/2691173581 wait complete. 2023-10-30T13:30:05.166 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:07.167 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:07.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.316+0000 7f65560c9640 1 Processor -- start 2023-10-30T13:30:07.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.316+0000 7f65560c9640 1 -- start start 2023-10-30T13:30:07.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f65560c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f655006f7a0 0x7f655006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:07.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f65560c9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6550071110 0x7f6550158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:07.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f65560c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6550066cf0 con 0x7f65500c6bd0 2023-10-30T13:30:07.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f65560c9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65500700c0 con 0x7f6550071110 2023-10-30T13:30:07.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f65560c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65500c70d0 con 0x7f655006f7a0 2023-10-30T13:30:07.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f654ffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6550071110 0x7f6550158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:07.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f654ffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6550071110 0x7f6550158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39378/0 (socket says 172.21.15.148:39378) 2023-10-30T13:30:07.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f654ffff640 1 -- 172.21.15.148:0/193082399 learned_addr learned my addr 172.21.15.148:0/193082399 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:07.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f654f7fe640 1 --2- 172.21.15.148:0/193082399 >> v2:172.21.15.150:3300/0 conn(0x7f655006f7a0 0x7f655006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:07.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f654ffff640 1 -- 172.21.15.148:0/193082399 >> v2:172.21.15.150:3300/0 conn(0x7f655006f7a0 msgr2=0x7f655006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:07.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f654ffff640 1 --2- 172.21.15.148:0/193082399 >> v2:172.21.15.150:3300/0 conn(0x7f655006f7a0 0x7f655006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f654ffff640 1 -- 172.21.15.148:0/193082399 >> v1:172.21.15.150:6789/0 conn(0x7f65500c6bd0 legacy=0x7f6550069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:07.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.320+0000 7f654ffff640 1 -- 172.21.15.148:0/193082399 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65500c7390 con 0x7f6550071110 2023-10-30T13:30:07.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f654ffff640 1 --2- 172.21.15.148:0/193082399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 0x7f6550158110 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f655006c830 tx=0x7f654801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abdf11ebd4a9afb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:07.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f654effd640 1 -- 172.21.15.148:0/193082399 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65480af400 con 0x7f6550071110 2023-10-30T13:30:07.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f654effd640 1 -- 172.21.15.148:0/193082399 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65480ab050 con 0x7f6550071110 2023-10-30T13:30:07.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f654effd640 1 -- 172.21.15.148:0/193082399 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65480af780 con 0x7f6550071110 2023-10-30T13:30:07.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 -- 172.21.15.148:0/193082399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 msgr2=0x7f6550158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:07.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 --2- 172.21.15.148:0/193082399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 0x7f6550158110 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f655006c830 tx=0x7f654801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 -- 172.21.15.148:0/193082399 shutdown_connections 2023-10-30T13:30:07.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 --2- 172.21.15.148:0/193082399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 0x7f6550158110 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 --2- 172.21.15.148:0/193082399 >> v2:172.21.15.150:3300/0 conn(0x7f655006f7a0 0x7f655006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 -- 172.21.15.148:0/193082399 >> 172.21.15.148:0/193082399 conn(0x7f65500c7910 msgr2=0x7f65500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:07.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 -- 172.21.15.148:0/193082399 shutdown_connections 2023-10-30T13:30:07.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 -- 172.21.15.148:0/193082399 wait complete. 2023-10-30T13:30:07.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 Processor -- start 2023-10-30T13:30:07.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.324+0000 7f65560c9640 1 -- start start 2023-10-30T13:30:07.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f65560c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f655006f7a0 0x7f65501575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:07.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f65560c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 0x7f6550152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:07.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f65560c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65500c6bd0 0x7f655014efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:07.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f65560c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655006c560 con 0x7f6550071110 2023-10-30T13:30:07.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f65560c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65500c20b0 con 0x7f655006f7a0 2023-10-30T13:30:07.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f65560c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65500c9340 con 0x7f65500c6bd0 2023-10-30T13:30:07.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f6547fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 0x7f6550152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:07.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f6547fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 0x7f6550152620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39386/0 (socket says 172.21.15.148:39386) 2023-10-30T13:30:07.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f6547fff640 1 -- 172.21.15.148:0/1563605297 learned_addr learned my addr 172.21.15.148:0/1563605297 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:07.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f654f7fe640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f655006f7a0 0x7f65501575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:07.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f654ffff640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65500c6bd0 0x7f655014efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:07.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f6547fff640 1 -- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65500c6bd0 msgr2=0x7f655014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:07.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f6547fff640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65500c6bd0 0x7f655014efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f6547fff640 1 -- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f655006f7a0 msgr2=0x7f65501575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:07.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f6547fff640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f655006f7a0 0x7f65501575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f6547fff640 1 -- 172.21.15.148:0/1563605297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6548009b30 con 0x7f6550071110 2023-10-30T13:30:07.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f6547fff640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 0x7f6550152620 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f653800c4d0 tx=0x7f653800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:07.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f654cff9640 1 -- 172.21.15.148:0/1563605297 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65380052d0 con 0x7f6550071110 2023-10-30T13:30:07.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f654cff9640 1 -- 172.21.15.148:0/1563605297 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6538005430 con 0x7f6550071110 2023-10-30T13:30:07.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6550154b40 con 0x7f6550071110 2023-10-30T13:30:07.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f654cff9640 1 -- 172.21.15.148:0/1563605297 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538012830 con 0x7f6550071110 2023-10-30T13:30:07.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65500c9060 con 0x7f6550071110 2023-10-30T13:30:07.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6550070330 con 0x7f6550071110 2023-10-30T13:30:07.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.328+0000 7f654cff9640 1 -- 172.21.15.148:0/1563605297 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6538012990 con 0x7f6550071110 2023-10-30T13:30:07.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.332+0000 7f654cff9640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65300664a0 0x7f6530068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:07.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.332+0000 7f654cff9640 1 -- 172.21.15.148:0/1563605297 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f6538083d60 con 0x7f6550071110 2023-10-30T13:30:07.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.332+0000 7f654f7fe640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65300664a0 0x7f6530068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:07.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.332+0000 7f654f7fe640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65300664a0 0x7f6530068960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6540009ac0 tx=0x7f6540009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:07.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.336+0000 7f654cff9640 1 -- 172.21.15.148:0/1563605297 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6550070330 con 0x7f6550071110 2023-10-30T13:30:07.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6550070330 con 0x7f6550071110 2023-10-30T13:30:07.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f654cff9640 1 -- 172.21.15.148:0/1563605297 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5714 (secure 0 0 0) 0x7f6550070330 con 0x7f6550071110 2023-10-30T13:30:07.512 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:30:07.512 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:04.765166+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:07.513 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 90 2023-10-30T13:30:07.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65300664a0 msgr2=0x7f6530068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:07.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65300664a0 0x7f6530068960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6540009ac0 tx=0x7f6540009190 comp rx=0 tx=0).stop 2023-10-30T13:30:07.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 msgr2=0x7f6550152620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:07.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 0x7f6550152620 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f653800c4d0 tx=0x7f653800c990 comp rx=0 tx=0).stop 2023-10-30T13:30:07.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 shutdown_connections 2023-10-30T13:30:07.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65300664a0 0x7f6530068960 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6550071110 0x7f6550152620 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65500c6bd0 0x7f655014efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 --2- 172.21.15.148:0/1563605297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f655006f7a0 0x7f65501575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:07.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 >> 172.21.15.148:0/1563605297 conn(0x7f65500c7910 msgr2=0x7f65500c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:07.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 shutdown_connections 2023-10-30T13:30:07.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:07.516+0000 7f65560c9640 1 -- 172.21.15.148:0/1563605297 wait complete. 2023-10-30T13:30:07.533 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:09.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:09.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.684+0000 7feae29aa640 1 Processor -- start 2023-10-30T13:30:09.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.684+0000 7feae29aa640 1 -- start start 2023-10-30T13:30:09.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feae29aa640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feadc0c6b80 0x7feadc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:09.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feae29aa640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feadc0c0a50 con 0x7feadc0c6f60 2023-10-30T13:30:09.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feae29aa640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feadc06f6e0 con 0x7feadc13f4a0 2023-10-30T13:30:09.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feae29aa640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feadc06f9c0 con 0x7feadc0c6b80 2023-10-30T13:30:09.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadb7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7feadc0c6f60 0x7feadc13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44482/0 (socket says 172.21.15.148:44482) 2023-10-30T13:30:09.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadb7fe640 1 -- 172.21.15.148:0/4079615341 learned_addr learned my addr 172.21.15.148:0/4079615341 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:30:09.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadbfff640 1 --2- 172.21.15.148:0/4079615341 >> v2:172.21.15.150:3300/0 conn(0x7feadc0c6b80 0x7feadc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:09.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426263574 0 0) 0x7feadc0c0a50 con 0x7feadc0c6f60 2023-10-30T13:30:09.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac4003530 con 0x7feadc0c6f60 2023-10-30T13:30:09.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1512754481 0 0) 0x7feac4003530 con 0x7feadc0c6f60 2023-10-30T13:30:09.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 >> v2:172.21.15.150:3300/0 conn(0x7feadc0c6b80 msgr2=0x7feadc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:09.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 --2- 172.21.15.148:0/4079615341 >> v2:172.21.15.150:3300/0 conn(0x7feadc0c6b80 0x7feadc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 >> v1:172.21.15.150:6789/0 conn(0x7feadc13f4a0 legacy=0x7feadc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:09.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feadc06fca0 con 0x7feadc0c6f60 2023-10-30T13:30:09.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fead0001680 con 0x7feadc0c6f60 2023-10-30T13:30:09.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 >> v1:172.21.15.148:6789/0 conn(0x7feadc0c6f60 legacy=0x7feadc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:09.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feac4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:09.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feac40047d0 0x7feac4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:09.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feac4005190 0x7feac400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:09.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc06f9c0 con 0x7feac4003980 2023-10-30T13:30:09.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac400c030 con 0x7feac4005190 2023-10-30T13:30:09.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac400e400 con 0x7feac40047d0 2023-10-30T13:30:09.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadbfff640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feac4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:09.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadb7fe640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feac40047d0 0x7feac4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:09.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feae0f47640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feac4005190 0x7feac400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:09.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadbfff640 1 -- 172.21.15.148:0/4079615341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feac40047d0 msgr2=0x7feac4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:09.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadbfff640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feac40047d0 0x7feac4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadbfff640 1 -- 172.21.15.148:0/4079615341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feac4005190 msgr2=0x7feac400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:09.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadbfff640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feac4005190 0x7feac400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feae0f47640 1 -- 172.21.15.148:0/4079615341 reap_dead start 2023-10-30T13:30:09.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.688+0000 7feadbfff640 1 -- 172.21.15.148:0/4079615341 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac4003530 con 0x7feac4003980 2023-10-30T13:30:09.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feadbfff640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feac4003da0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7feacc009ae0 tx=0x7feacc01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45dc699ffbc4b682 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:09.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc0a5530 con 0x7feac4003980 2023-10-30T13:30:09.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc0a5690 con 0x7feac4003980 2023-10-30T13:30:09.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feadaffd640 1 -- 172.21.15.148:0/4079615341 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc0a5a40 con 0x7feac4003980 2023-10-30T13:30:09.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feae29aa640 1 -- 172.21.15.148:0/4079615341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 msgr2=0x7feac4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:09.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feae29aa640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feac4003da0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7feacc009ae0 tx=0x7feacc01ca50 comp rx=0 tx=0).stop 2023-10-30T13:30:09.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feae29aa640 1 -- 172.21.15.148:0/4079615341 shutdown_connections 2023-10-30T13:30:09.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feae29aa640 1 --2- 172.21.15.148:0/4079615341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feac4003da0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feae29aa640 1 -- 172.21.15.148:0/4079615341 >> 172.21.15.148:0/4079615341 conn(0x7feadc0c7910 msgr2=0x7feadc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:09.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feae29aa640 1 -- 172.21.15.148:0/4079615341 shutdown_connections 2023-10-30T13:30:09.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.692+0000 7feae29aa640 1 -- 172.21.15.148:0/4079615341 wait complete. 2023-10-30T13:30:09.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 Processor -- start 2023-10-30T13:30:09.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 -- start start 2023-10-30T13:30:09.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feadc159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:09.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feadc157790 0x7feadc155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:09.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feadc1501c0 0x7feadc150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:09.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc1f1cb0 con 0x7feac4003980 2023-10-30T13:30:09.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc157bf0 con 0x7feadc1501c0 2023-10-30T13:30:09.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feadc13c8c0 con 0x7feadc157790 2023-10-30T13:30:09.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feadc159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:09.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feadc159380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39398/0 (socket says 172.21.15.148:39398) 2023-10-30T13:30:09.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadbfff640 1 -- 172.21.15.148:0/2575915297 learned_addr learned my addr 172.21.15.148:0/2575915297 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:09.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadb7fe640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feadc157790 0x7feadc155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:09.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae0f47640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feadc1501c0 0x7feadc150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:09.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadbfff640 1 -- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feadc157790 msgr2=0x7feadc155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:09.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadbfff640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feadc157790 0x7feadc155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadbfff640 1 -- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feadc1501c0 msgr2=0x7feadc150640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:09.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadbfff640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feadc1501c0 0x7feadc150640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadbfff640 1 -- 172.21.15.148:0/2575915297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feadc154ec0 con 0x7feac4003980 2023-10-30T13:30:09.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feadbfff640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feadc159380 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7feacc0b0040 tx=0x7feacc0197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:09.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7fead8ff9640 1 -- 172.21.15.148:0/2575915297 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc0a5530 con 0x7feac4003980 2023-10-30T13:30:09.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7fead8ff9640 1 -- 172.21.15.148:0/2575915297 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc004240 con 0x7feac4003980 2023-10-30T13:30:09.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7fead8ff9640 1 -- 172.21.15.148:0/2575915297 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc003620 con 0x7feac4003980 2023-10-30T13:30:09.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feadc1f21c0 con 0x7feac4003980 2023-10-30T13:30:09.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feadc1f1ee0 con 0x7feac4003980 2023-10-30T13:30:09.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.696+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feadc04e170 con 0x7feac4003980 2023-10-30T13:30:09.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.700+0000 7fead8ff9640 1 -- 172.21.15.148:0/2575915297 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feacc0b4070 con 0x7feac4003980 2023-10-30T13:30:09.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.700+0000 7fead8ff9640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feac80664a0 0x7feac8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:09.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.700+0000 7fead8ff9640 1 -- 172.21.15.148:0/2575915297 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7feacc11e0e0 con 0x7feac4003980 2023-10-30T13:30:09.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.700+0000 7feadb7fe640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feac80664a0 0x7feac8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:09.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.700+0000 7feadb7fe640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feac80664a0 0x7feac8068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fead0001ca0 tx=0x7fead0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:09.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.704+0000 7fead8ff9640 1 -- 172.21.15.148:0/2575915297 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feacc0eb040 con 0x7feac4003980 2023-10-30T13:30:09.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feadc138470 con 0x7feac4003980 2023-10-30T13:30:09.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7fead8ff9640 1 -- 172.21.15.148:0/2575915297 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5714 (secure 0 0 0) 0x7feacc0b2020 con 0x7feac4003980 2023-10-30T13:30:09.884 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:30:09.884 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:04.765166+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:09.885 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 90 2023-10-30T13:30:09.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feac80664a0 msgr2=0x7feac8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:09.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feac80664a0 0x7feac8068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fead0001ca0 tx=0x7fead0009040 comp rx=0 tx=0).stop 2023-10-30T13:30:09.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 msgr2=0x7feadc159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:09.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feadc159380 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7feacc0b0040 tx=0x7feacc0197f0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 shutdown_connections 2023-10-30T13:30:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feac80664a0 0x7feac8068960 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feadc1501c0 0x7feadc150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feadc157790 0x7feadc155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 --2- 172.21.15.148:0/2575915297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feac4003980 0x7feadc159380 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:09.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 >> 172.21.15.148:0/2575915297 conn(0x7feadc0c7910 msgr2=0x7feadc0c12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:09.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 shutdown_connections 2023-10-30T13:30:09.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:09.888+0000 7feae29aa640 1 -- 172.21.15.148:0/2575915297 wait complete. 2023-10-30T13:30:09.905 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:30:11.906 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:12.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.056+0000 7f19d72c9640 1 Processor -- start 2023-10-30T13:30:12.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.056+0000 7f19d72c9640 1 -- start start 2023-10-30T13:30:12.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d72c9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f19d006f7a0 0x7f19d006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:12.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d72c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d0066cf0 con 0x7f19d00c6bd0 2023-10-30T13:30:12.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d72c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d00c70d0 con 0x7f19d0071110 2023-10-30T13:30:12.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d72c9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d00700c0 con 0x7f19d006f7a0 2023-10-30T13:30:12.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d5065640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f19d006f7a0 0x7f19d006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:12.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d4864640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f19d00c6bd0 0x7f19d0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34592/0 (socket says 172.21.15.148:34592) 2023-10-30T13:30:12.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d4864640 1 -- 172.21.15.148:0/954896815 learned_addr learned my addr 172.21.15.148:0/954896815 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:30:12.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d5065640 1 -- 172.21.15.148:0/954896815 >> v1:172.21.15.150:6789/0 conn(0x7f19d0071110 legacy=0x7f19d0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:12.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d5065640 1 -- 172.21.15.148:0/954896815 >> v1:172.21.15.148:6789/0 conn(0x7f19d00c6bd0 legacy=0x7f19d0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:12.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19d5065640 1 -- 172.21.15.148:0/954896815 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d00c7390 con 0x7f19d006f7a0 2023-10-30T13:30:12.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.060+0000 7f19c7fff640 1 -- 172.21.15.148:0/954896815 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228722923 0 0) 0x7f19d0066cf0 con 0x7f19d00c6bd0 2023-10-30T13:30:12.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d5065640 1 --2- 172.21.15.148:0/954896815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d006f7a0 0x7f19d006fb80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0097c0 tx=0x7f19cc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8375cd0d48f84d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:12.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19c7fff640 1 -- 172.21.15.148:0/954896815 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0a8970 con 0x7f19d006f7a0 2023-10-30T13:30:12.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19c7fff640 1 -- 172.21.15.148:0/954896815 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc0a8ad0 con 0x7f19d006f7a0 2023-10-30T13:30:12.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19c7fff640 1 -- 172.21.15.148:0/954896815 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0a8e60 con 0x7f19d006f7a0 2023-10-30T13:30:12.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d72c9640 1 -- 172.21.15.148:0/954896815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d006f7a0 msgr2=0x7f19d006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:12.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d72c9640 1 --2- 172.21.15.148:0/954896815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d006f7a0 0x7f19d006fb80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0097c0 tx=0x7f19cc01ec70 comp rx=0 tx=0).stop 2023-10-30T13:30:12.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d72c9640 1 -- 172.21.15.148:0/954896815 shutdown_connections 2023-10-30T13:30:12.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d72c9640 1 --2- 172.21.15.148:0/954896815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d006f7a0 0x7f19d006fb80 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:12.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d72c9640 1 -- 172.21.15.148:0/954896815 >> 172.21.15.148:0/954896815 conn(0x7f19d00c7910 msgr2=0x7f19d00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:12.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d72c9640 1 -- 172.21.15.148:0/954896815 shutdown_connections 2023-10-30T13:30:12.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d72c9640 1 -- 172.21.15.148:0/954896815 wait complete. 2023-10-30T13:30:12.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d72c9640 1 Processor -- start 2023-10-30T13:30:12.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.064+0000 7f19d72c9640 1 -- start start 2023-10-30T13:30:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d72c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d006f7a0 0x7f19d014f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d72c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d0071110 0x7f19d0155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d72c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d00c6bd0 0x7f19d00c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d72c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d0154b30 con 0x7f19d00c6bd0 2023-10-30T13:30:12.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d72c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d0154710 con 0x7f19d006f7a0 2023-10-30T13:30:12.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d72c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d0152950 con 0x7f19d0071110 2023-10-30T13:30:12.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5866640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d00c6bd0 0x7f19d00c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:12.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5866640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d00c6bd0 0x7f19d00c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34228/0 (socket says 172.21.15.148:34228) 2023-10-30T13:30:12.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5866640 1 -- 172.21.15.148:0/2028568479 learned_addr learned my addr 172.21.15.148:0/2028568479 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:12.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5065640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d006f7a0 0x7f19d014f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:12.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d4864640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d0071110 0x7f19d0155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:12.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5866640 1 -- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d0071110 msgr2=0x7f19d0155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:12.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5866640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d0071110 0x7f19d0155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:12.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5866640 1 -- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d006f7a0 msgr2=0x7f19d014f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:12.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5866640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d006f7a0 0x7f19d014f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:12.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5866640 1 -- 172.21.15.148:0/2028568479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc009420 con 0x7f19d00c6bd0 2023-10-30T13:30:12.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d4864640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d0071110 0x7f19d0155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:12.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d5866640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d00c6bd0 0x7f19d00c3690 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f19c800c730 tx=0x7f19c800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:12.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19c5ffb640 1 -- 172.21.15.148:0/2028568479 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c8003b80 con 0x7f19d00c6bd0 2023-10-30T13:30:12.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19c5ffb640 1 -- 172.21.15.148:0/2028568479 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c8003ce0 con 0x7f19d00c6bd0 2023-10-30T13:30:12.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d0154da0 con 0x7f19d00c6bd0 2023-10-30T13:30:12.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19c5ffb640 1 -- 172.21.15.148:0/2028568479 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c800aa30 con 0x7f19d00c6bd0 2023-10-30T13:30:12.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d0152ed0 con 0x7f19d00c6bd0 2023-10-30T13:30:12.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d01530c0 con 0x7f19d00c6bd0 2023-10-30T13:30:12.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.068+0000 7f19c5ffb640 1 -- 172.21.15.148:0/2028568479 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19c800ab90 con 0x7f19d00c6bd0 2023-10-30T13:30:12.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.072+0000 7f19c5ffb640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19b40664a0 0x7f19b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:12.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.072+0000 7f19c5ffb640 1 -- 172.21.15.148:0/2028568479 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f19c8083f90 con 0x7f19d00c6bd0 2023-10-30T13:30:12.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.072+0000 7f19d5065640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19b40664a0 0x7f19b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:12.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.072+0000 7f19d5065640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19b40664a0 0x7f19b4068960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0b2000 tx=0x7f19cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:12.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.076+0000 7f19c5ffb640 1 -- 172.21.15.148:0/2028568479 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19c80b5090 con 0x7f19d00c6bd0 2023-10-30T13:30:12.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.256+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19d004e170 con 0x7f19d00c6bd0 2023-10-30T13:30:12.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.256+0000 7f19c5ffb640 1 -- 172.21.15.148:0/2028568479 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5714 (secure 0 0 0) 0x7f19c8054ab0 con 0x7f19d00c6bd0 2023-10-30T13:30:12.255 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:30:12.255 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:12.255 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 91 2023-10-30T13:30:12.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19b40664a0 msgr2=0x7f19b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:12.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19b40664a0 0x7f19b4068960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0b2000 tx=0x7f19cc008040 comp rx=0 tx=0).stop 2023-10-30T13:30:12.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d00c6bd0 msgr2=0x7f19d00c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:12.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d00c6bd0 0x7f19d00c3690 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f19c800c730 tx=0x7f19c800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:30:12.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 shutdown_connections 2023-10-30T13:30:12.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19b40664a0 0x7f19b4068960 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:12.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f19d00c6bd0 0x7f19d00c3690 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:12.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f19d0071110 0x7f19d0155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:12.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 --2- 172.21.15.148:0/2028568479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d006f7a0 0x7f19d014f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:12.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 >> 172.21.15.148:0/2028568479 conn(0x7f19d00c7910 msgr2=0x7f19d00b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:12.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 shutdown_connections 2023-10-30T13:30:12.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:12.260+0000 7f19d72c9640 1 -- 172.21.15.148:0/2028568479 wait complete. 2023-10-30T13:30:12.276 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6249 2023-10-30T13:30:12.276 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.c 2023-10-30T13:30:17.277 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:30:17.277 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:30:17.278 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:30:17.281 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:30:17.282 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:17.373 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:30:17.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815fc79640 1 Processor -- start 2023-10-30T13:30:17.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815fc79640 1 -- start start 2023-10-30T13:30:17.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815fc79640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8158071110 0x7f815806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:17.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815fc79640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8158066cf0 con 0x7f815806f7a0 2023-10-30T13:30:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815fc79640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81580c70d0 con 0x7f81580c6bd0 2023-10-30T13:30:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815fc79640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81580700c0 con 0x7f8158071110 2023-10-30T13:30:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815e216640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8158071110 0x7f815806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815da15640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f815806f7a0 0x7f8158069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34600/0 (socket says 172.21.15.148:34600) 2023-10-30T13:30:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815da15640 1 -- 172.21.15.148:0/904462595 learned_addr learned my addr 172.21.15.148:0/904462595 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:30:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815e216640 1 -- 172.21.15.148:0/904462595 >> v1:172.21.15.150:6789/0 conn(0x7f81580c6bd0 legacy=0x7f8158158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815e216640 1 -- 172.21.15.148:0/904462595 >> v1:172.21.15.148:6789/0 conn(0x7f815806f7a0 legacy=0x7f8158069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:17.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.436+0000 7f815e216640 1 -- 172.21.15.148:0/904462595 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81580c7390 con 0x7f8158071110 2023-10-30T13:30:17.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.440+0000 7f815e216640 1 --2- 172.21.15.148:0/904462595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 0x7f815806fb80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8150009d90 tx=0x7f815001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd298ba1708081e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:17.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.440+0000 7f815ca13640 1 -- 172.21.15.148:0/904462595 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81500a58d0 con 0x7f8158071110 2023-10-30T13:30:17.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.440+0000 7f815ca13640 1 -- 172.21.15.148:0/904462595 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81500a5a30 con 0x7f8158071110 2023-10-30T13:30:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.440+0000 7f815ca13640 1 -- 172.21.15.148:0/904462595 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81500a5dc0 con 0x7f8158071110 2023-10-30T13:30:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 -- 172.21.15.148:0/904462595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 msgr2=0x7f815806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 --2- 172.21.15.148:0/904462595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 0x7f815806fb80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8150009d90 tx=0x7f815001c870 comp rx=0 tx=0).stop 2023-10-30T13:30:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 -- 172.21.15.148:0/904462595 shutdown_connections 2023-10-30T13:30:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 --2- 172.21.15.148:0/904462595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 0x7f815806fb80 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 -- 172.21.15.148:0/904462595 >> 172.21.15.148:0/904462595 conn(0x7f81580c7910 msgr2=0x7f81580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 -- 172.21.15.148:0/904462595 shutdown_connections 2023-10-30T13:30:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 -- 172.21.15.148:0/904462595 wait complete. 2023-10-30T13:30:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 Processor -- start 2023-10-30T13:30:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 -- start start 2023-10-30T13:30:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f815806f7a0 0x7f8158150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 0x7f81581eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:17.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81580c6bd0 0x7f81581ef510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:17.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81581f3140 con 0x7f8158071110 2023-10-30T13:30:17.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81580ae920 con 0x7f815806f7a0 2023-10-30T13:30:17.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815fc79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81581f24e0 con 0x7f81580c6bd0 2023-10-30T13:30:17.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815d214640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 0x7f81581eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:17.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815d214640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 0x7f81581eefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34262/0 (socket says 172.21.15.148:34262) 2023-10-30T13:30:17.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815d214640 1 -- 172.21.15.148:0/3715800066 learned_addr learned my addr 172.21.15.148:0/3715800066 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:17.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815e216640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81580c6bd0 0x7f81581ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:17.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815da15640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f815806f7a0 0x7f8158150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:17.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815d214640 1 -- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81580c6bd0 msgr2=0x7f81581ef510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:17.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815d214640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81580c6bd0 0x7f81581ef510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:17.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815d214640 1 -- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f815806f7a0 msgr2=0x7f8158150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:17.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815d214640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f815806f7a0 0x7f8158150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:17.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815d214640 1 -- 172.21.15.148:0/3715800066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8150009a70 con 0x7f8158071110 2023-10-30T13:30:17.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.444+0000 7f815da15640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f815806f7a0 0x7f8158150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:17.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f815d214640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 0x7f81581eefd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f814800c500 tx=0x7f814800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:17.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f814e7fc640 1 -- 172.21.15.148:0/3715800066 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81480052d0 con 0x7f8158071110 2023-10-30T13:30:17.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f814e7fc640 1 -- 172.21.15.148:0/3715800066 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8148005430 con 0x7f8158071110 2023-10-30T13:30:17.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f814e7fc640 1 -- 172.21.15.148:0/3715800066 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148012830 con 0x7f8158071110 2023-10-30T13:30:17.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81581f27a0 con 0x7f8158071110 2023-10-30T13:30:17.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81581f2d60 con 0x7f8158071110 2023-10-30T13:30:17.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81581f2ef0 con 0x7f8158071110 2023-10-30T13:30:17.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f814e7fc640 1 -- 172.21.15.148:0/3715800066 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8148012990 con 0x7f8158071110 2023-10-30T13:30:17.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f814e7fc640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f813c0664a0 0x7f813c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:17.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f814e7fc640 1 -- 172.21.15.148:0/3715800066 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f8148083e20 con 0x7f8158071110 2023-10-30T13:30:17.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.448+0000 7f815da15640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f813c0664a0 0x7f813c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:17.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.452+0000 7f815da15640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f813c0664a0 0x7f813c068960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8154000f70 tx=0x7f815400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:17.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.456+0000 7f814e7fc640 1 -- 172.21.15.148:0/3715800066 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8148050d90 con 0x7f8158071110 2023-10-30T13:30:17.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.644+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f815804e170 con 0x7f8158071110 2023-10-30T13:30:17.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.644+0000 7f814e7fc640 1 -- 172.21.15.148:0/3715800066 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5714 (secure 0 0 0) 0x7f81480543b0 con 0x7f8158071110 2023-10-30T13:30:17.641 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:30:17.642 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:17.642 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 91 2023-10-30T13:30:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.644+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f813c0664a0 msgr2=0x7f813c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.644+0000 7f815fc79640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f813c0664a0 0x7f813c068960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8154000f70 tx=0x7f815400b040 comp rx=0 tx=0).stop 2023-10-30T13:30:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.644+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 msgr2=0x7f81581eefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:17.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.644+0000 7f815fc79640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 0x7f81581eefd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f814800c500 tx=0x7f814800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:30:17.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.648+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 shutdown_connections 2023-10-30T13:30:17.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.648+0000 7f815fc79640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f813c0664a0 0x7f813c068960 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:17.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.648+0000 7f815fc79640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8158071110 0x7f81581eefd0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:17.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.648+0000 7f815fc79640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f81580c6bd0 0x7f81581ef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:17.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.648+0000 7f815fc79640 1 --2- 172.21.15.148:0/3715800066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f815806f7a0 0x7f8158150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:17.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.648+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 >> 172.21.15.148:0/3715800066 conn(0x7f81580c7910 msgr2=0x7f81580ca0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:17.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.648+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 shutdown_connections 2023-10-30T13:30:17.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:17.648+0000 7f815fc79640 1 -- 172.21.15.148:0/3715800066 wait complete. 2023-10-30T13:30:17.663 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:30:18.709 INFO:tasks.ceph.mds.a.smithi148.stderr:2023-10-30T13:30:18.712+0000 7f7c9b631640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:30:19.663 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:30:19.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76feea640 1 Processor -- start 2023-10-30T13:30:19.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76feea640 1 -- start start 2023-10-30T13:30:19.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76feea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76815a410 0x7fa76806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:19.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76feea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa76806cb80 0x7fa76806d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:19.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76feea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76806d550 con 0x7fa768069e90 2023-10-30T13:30:19.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76feea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7680c8570 con 0x7fa76806cb80 2023-10-30T13:30:19.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76feea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76815a850 con 0x7fa76815a410 2023-10-30T13:30:19.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76e487640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa76806cb80 0x7fa76806d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:19.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76e487640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa76806cb80 0x7fa76806d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34264/0 (socket says 172.21.15.148:34264) 2023-10-30T13:30:19.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76e487640 1 -- 172.21.15.148:0/1655667586 learned_addr learned my addr 172.21.15.148:0/1655667586 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:19.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76d485640 1 --2- 172.21.15.148:0/1655667586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76815a410 0x7fa76806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:19.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76e487640 1 -- 172.21.15.148:0/1655667586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76815a410 msgr2=0x7fa76806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:30:19.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76e487640 1 --2- 172.21.15.148:0/1655667586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76815a410 0x7fa76806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:19.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76e487640 1 -- 172.21.15.148:0/1655667586 >> v1:172.21.15.150:6789/0 conn(0x7fa768069e90 legacy=0x7fa7680c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:19.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.808+0000 7fa76e487640 1 -- 172.21.15.148:0/1655667586 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7680c6c10 con 0x7fa76806cb80 2023-10-30T13:30:19.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76e487640 1 --2- 172.21.15.148:0/1655667586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 0x7fa76806d010 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa760009e50 tx=0x7fa76001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3572210bd9e76b97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:19.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76cc84640 1 -- 172.21.15.148:0/1655667586 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7600a56e0 con 0x7fa76806cb80 2023-10-30T13:30:19.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76cc84640 1 -- 172.21.15.148:0/1655667586 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7600a5840 con 0x7fa76806cb80 2023-10-30T13:30:19.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76cc84640 1 -- 172.21.15.148:0/1655667586 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7600a5bd0 con 0x7fa76806cb80 2023-10-30T13:30:19.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76feea640 1 -- 172.21.15.148:0/1655667586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 msgr2=0x7fa76806d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:19.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76feea640 1 --2- 172.21.15.148:0/1655667586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 0x7fa76806d010 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa760009e50 tx=0x7fa76001ca10 comp rx=0 tx=0).stop 2023-10-30T13:30:19.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76feea640 1 -- 172.21.15.148:0/1655667586 shutdown_connections 2023-10-30T13:30:19.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76feea640 1 --2- 172.21.15.148:0/1655667586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 0x7fa76806d010 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:19.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76feea640 1 --2- 172.21.15.148:0/1655667586 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76815a410 0x7fa76806a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:19.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76feea640 1 -- 172.21.15.148:0/1655667586 >> 172.21.15.148:0/1655667586 conn(0x7fa76806dfd0 msgr2=0x7fa76806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:19.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76feea640 1 -- 172.21.15.148:0/1655667586 shutdown_connections 2023-10-30T13:30:19.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.812+0000 7fa76feea640 1 -- 172.21.15.148:0/1655667586 wait complete. 2023-10-30T13:30:19.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 Processor -- start 2023-10-30T13:30:19.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 -- start start 2023-10-30T13:30:19.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa768069e90 0x7fa76814b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:19.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 0x7fa7681f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:19.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815a410 0x7fa768149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:19.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76815b190 con 0x7fa76806cb80 2023-10-30T13:30:19.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680c61b0 con 0x7fa76815a410 2023-10-30T13:30:19.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7681f2a30 con 0x7fa768069e90 2023-10-30T13:30:19.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76d485640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 0x7fa7681f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:19.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76d485640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 0x7fa7681f4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34280/0 (socket says 172.21.15.148:34280) 2023-10-30T13:30:19.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76d485640 1 -- 172.21.15.148:0/3303737662 learned_addr learned my addr 172.21.15.148:0/3303737662 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:30:19.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76e487640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815a410 0x7fa768149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:19.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76dc86640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa768069e90 0x7fa76814b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:19.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76d485640 1 -- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa768069e90 msgr2=0x7fa76814b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:19.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76d485640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa768069e90 0x7fa76814b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:19.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76d485640 1 -- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815a410 msgr2=0x7fa768149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:19.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76d485640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815a410 0x7fa768149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:19.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76d485640 1 -- 172.21.15.148:0/3303737662 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa760009b30 con 0x7fa76806cb80 2023-10-30T13:30:19.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76dc86640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa768069e90 0x7fa76814b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:30:19.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76e487640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815a410 0x7fa768149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:30:19.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76d485640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 0x7fa7681f4d20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa75800d5f0 tx=0x7fa75800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:19.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa75e7fc640 1 -- 172.21.15.148:0/3303737662 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7580125f0 con 0x7fa76806cb80 2023-10-30T13:30:19.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa75e7fc640 1 -- 172.21.15.148:0/3303737662 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa758012750 con 0x7fa76806cb80 2023-10-30T13:30:19.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa75e7fc640 1 -- 172.21.15.148:0/3303737662 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758012b00 con 0x7fa76806cb80 2023-10-30T13:30:19.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7681f5810 con 0x7fa76806cb80 2023-10-30T13:30:19.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76814b110 con 0x7fa76806cb80 2023-10-30T13:30:19.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.816+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76804e170 con 0x7fa76806cb80 2023-10-30T13:30:19.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.820+0000 7fa75e7fc640 1 -- 172.21.15.148:0/3303737662 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa758012c60 con 0x7fa76806cb80 2023-10-30T13:30:19.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.820+0000 7fa75e7fc640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa74c0664a0 0x7fa74c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:30:19.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.820+0000 7fa75e7fc640 1 -- 172.21.15.148:0/3303737662 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fa758084880 con 0x7fa76806cb80 2023-10-30T13:30:19.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.820+0000 7fa76dc86640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa74c0664a0 0x7fa74c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:30:19.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.820+0000 7fa76dc86640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa74c0664a0 0x7fa74c068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa764000f70 tx=0x7fa764009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:30:19.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:19.824+0000 7fa75e7fc640 1 -- 172.21.15.148:0/3303737662 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa758051870 con 0x7fa76806cb80 2023-10-30T13:30:20.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.044+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa76806ef60 con 0x7fa76806cb80 2023-10-30T13:30:20.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa75e7fc640 1 -- 172.21.15.148:0/3303737662 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fa758054e90 con 0x7fa76806cb80 2023-10-30T13:30:20.044 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:30:20.044 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6546,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:30:20.044 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 92 2023-10-30T13:30:20.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa74c0664a0 msgr2=0x7fa74c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:20.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa74c0664a0 0x7fa74c068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa764000f70 tx=0x7fa764009040 comp rx=0 tx=0).stop 2023-10-30T13:30:20.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 msgr2=0x7fa7681f4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:30:20.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 0x7fa7681f4d20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa75800d5f0 tx=0x7fa75800dab0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 shutdown_connections 2023-10-30T13:30:20.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa74c0664a0 0x7fa74c068960 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76806cb80 0x7fa7681f4d20 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815a410 0x7fa768149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 --2- 172.21.15.148:0/3303737662 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa768069e90 0x7fa76814b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:30:20.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 >> 172.21.15.148:0/3303737662 conn(0x7fa76806dfd0 msgr2=0x7fa76814c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:30:20.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 shutdown_connections 2023-10-30T13:30:20.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:30:20.048+0000 7fa76feea640 1 -- 172.21.15.148:0/3303737662 wait complete. 2023-10-30T13:30:20.065 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:30:20.065 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2023-10-30T13:30:27.785 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:30:27.789 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:30:59.234 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:30:59.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:31:29.289 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:31:29.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:31:41.009 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 1% (139/12861) Updating files: 1% (173/12861) Updating files: 2% (258/12861) Updating files: 2% (326/12861) Updating files: 3% (386/12861) Updating files: 3% (508/12861) Updating files: 4% (515/12861) Updating files: 5% (644/12861) Updating files: 5% (684/12861) Updating files: 5% (702/12861) Updating files: 5% (714/12861) Updating files: 6% (772/12861) Updating files: 6% (836/12861) Updating files: 6% (840/12861) Updating files: 7% (901/12861) Updating files: 7% (934/12861) Updating files: 7% (936/12861) Updating files: 7% (970/12861) Updating files: 7% (979/12861) Updating files: 7% (987/12861) Updating files: 8% (1029/12861) Updating files: 8% (1072/12861) Updating files: 9% (1158/12861) Updating files: 9% (1179/12861) Updating files: 9% (1196/12861) Updating files: 9% (1280/12861) Updating files: 10% (1287/12861) Updating files: 10% (1378/12861) Updating files: 11% (1415/12861) Updating files: 11% (1429/12861) Updating files: 11% (1490/12861) Updating files: 12% (1544/12861) Updating files: 12% (1588/12861) Updating files: 12% (1663/12861) Updating files: 13% (1672/12861) Updating files: 13% (1726/12861) Updating files: 13% (1757/12861) Updating files: 13% (1792/12861) Updating files: 14% (1801/12861) Updating files: 14% (1929/12861) Updating files: 15% (1930/12861) Updating files: 15% (2057/12861) Updating files: 16% (2058/12861) Updating files: 16% (2105/12861) Updating files: 16% (2133/12861) Updating files: 17% (2187/12861) Updating files: 17% (2209/12861) Updating files: 18% (2315/12861) Updating files: 18% (2335/12861) Updating files: 18% (2400/12861) Updating files: 19% (2444/12861) Updating files: 19% (2447/12861) Updating files: 19% (2501/12861) Updating files: 20% (2573/12861) Updating files: 20% (2587/12861) Updating files: 20% (2591/12861) Updating files: 20% (2682/12861) Updating files: 20% (2690/12861) Updating files: 21% (2701/12861) Updating files: 21% (2761/12861) Updating files: 21% (2779/12861) Updating files: 21% (2807/12861) Updating files: 22% (2830/12861) Updating files: 22% (2931/12861) Updating files: 22% (2958/12861) Updating files: 23% (2959/12861) Updating files: 23% (3053/12861) Updating files: 23% (3078/12861) Updating files: 24% (3087/12861) Updating files: 24% (3108/12861) Updating files: 24% (3154/12861) Updating files: 25% (3216/12861) Updating files: 25% (3253/12861) Updating files: 25% (3306/12861) Updating files: 26% (3344/12861) Updating files: 26% (3383/12861) Updating files: 26% (3394/12861) Updating files: 26% (3408/12861) Updating files: 26% (3418/12861) Updating files: 27% (3473/12861) Updating files: 27% (3474/12861) Updating files: 27% (3479/12861) Updating files: 27% (3537/12861) Updating files: 28% (3602/12861) Updating files: 28% (3676/12861) Updating files: 28% (3714/12861) Updating files: 29% (3730/12861) Updating files: 29% (3732/12861) Updating files: 29% (3850/12861) Updating files: 30% (3859/12861) Updating files: 30% (3862/12861) Updating files: 30% (3865/12861) Updating files: 31% (3987/12861) Updating files: 31% (4007/12861) Updating files: 31% (4026/12861) Updating files: 31% (4072/12861) Updating files: 32% (4116/12861) Updating files: 32% (4162/12861) Updating files: 33% (4245/12861) Updating files: 33% (4257/12861) Updating files: 33% (4266/12861) Updating files: 33% (4334/12861) Updating files: 34% (4373/12861) Updating files: 34% (4388/12861) Updating files: 34% (4398/12861) Updating files: 34% (4408/12861) Updating files: 34% (4471/12861) Updating files: 35% (4502/12861) Updating files: 35% (4555/12861) Updating files: 35% (4619/12861) Updating files: 36% (4630/12861) Updating files: 36% (4671/12861) Updating files: 37% (4759/12861) Updating files: 37% (4802/12861) Updating files: 37% (4842/12861) Updating files: 38% (4888/12861) Updating files: 39% (5016/12861) Updating files: 39% (5071/12861) Updating files: 39% (5112/12861) Updating files: 40% (5145/12861) Updating files: 40% (5148/12861) Updating files: 40% (5209/12861) Updating files: 41% (5274/12861) Updating files: 41% (5381/12861) Updating files: 42% (5402/12861) Updating files: 42% (5433/12861) Updating files: 43% (5531/12861) Updating files: 43% (5621/12861) Updating files: 44% (5659/12861) Updating files: 44% (5721/12861) Updating files: 45% (5788/12861) Updating files: 45% (5800/12861) Updating files: 45% (5820/12861) Updating files: 45% (5843/12861) Updating files: 45% (5862/12861) Updating files: 45% (5913/12861) Updating files: 46% (5917/12861) Updating files: 47% (6045/12861) Updating files: 47% (6114/12861) Updating files: 47% (6162/12861) Updating files: 48% (6174/12861) Updating files: 48% (6227/12861) Updating files: 48% (6279/12861) Updating files: 49% (6302/12861) Updating files: 49% (6308/12861) Updating files: 49% (6428/12861) Updating files: 50% (6431/12861) Updating files: 50% (6453/12861) Updating files: 50% (6517/12861) Updating files: 51% (6560/12861) Updating files: 51% (6616/12861) Updating files: 52% (6688/12861) Updating files: 52% (6713/12861) Updating files: 53% (6817/12861) Updating files: 53% (6908/12861) Updating files: 53% (6940/12861) Updating files: 54% (6945/12861) Updating files: 54% (6963/12861) Updating files: 54% (7019/12861) Updating files: 55% (7074/12861) Updating files: 55% (7194/12861) Updating files: 56% (7203/12861) Updating files: 57% (7331/12861) Updating files: 57% (7388/12861) Updating files: 57% (7446/12861) Updating files: 58% (7460/12861) Updating files: 58% (7525/12861) Updating files: 58% (7563/12861) Updating files: 59% (7588/12861) Updating files: 59% (7712/12861) Updating files: 59% (7715/12861) Updating files: 60% (7717/12861) Updating files: 61% (7846/12861) Updating files: 61% (7944/12861) Updating files: 62% (7974/12861) Updating files: 62% (7987/12861) Updating files: 62% (8022/12861) Updating files: 62% (8053/12861) Updating files: 62% (8086/12861) Updating files: 63% (8103/12861) Updating files: 63% (8149/12861) Updating files: 63% (8216/12861) Updating files: 64% (8232/12861) Updating files: 64% (8255/12861) Updating files: 64% (8270/12861) Updating files: 65% (8360/12861) Updating files: 65% (8372/12861) Updating files: 65% (8398/12861) Updating files: 65% (8418/12861) Updating files: 66% (8489/12861) Updating files: 66% (8583/12861) Updating files: 67% (8617/12861) Updating files: 67% (8638/12861) Updating files: 67% (8717/12861) Updating files: 67% (8726/12861) Updating files: 68% (8746/12861) Updating files: 68% (8836/12861) Updating files: 69% (8875/12861) Updating files: 69% (8886/12861) Updating files: 69% (8888/12861) Updating files: 69% (8967/12861) Updating files: 70% (9003/12861) Updating files: 70% (9036/12861) Updating files: 70% (9129/12861) Updating files: 71% (9132/12861) Updating files: 71% (9244/12861) Updating files: 72% (9260/12861) Updating files: 72% (9313/12861) Updating files: 72% (9318/12861) Updating files: 72% (9372/12861) Updating files: 73% (9389/12861) Updating files: 73% (9420/12861) Updating files: 73% (9455/12861) Updating files: 73% (9476/12861) Updating files: 74% (9518/12861) Updating files: 74% (9528/12861) Updating files: 75% (9646/12861) Updating files: 75% (9656/12861) Updating files: 75% (9706/12861) Updating files: 76% (9775/12861) Updating files: 76% (9859/12861) Updating files: 76% (9869/12861) Updating files: 77% (9903/12861) Updating files: 77% (10010/12861) Updating files: 78% (10032/12861) Updating files: 78% (10124/12861) Updating files: 79% (10161/12861) Updating files: 79% (10178/12861) Updating files: 79% (10219/12861) Updating files: 79% (10287/12861) Updating files: 80% (10289/12861) Updating files: 81% (10418/12861) Updating files: 81% (10500/12861) Updating files: 82% (10547/12861) Updating files: 82% (10640/12861) Updating files: 83% (10675/12861) Updating files: 83% (10683/12861) Updating files: 84% (10804/12861) Updating files: 84% (10822/12861) Updating files: 84% (10872/12861) Updating files: 85% (10932/12861) Updating files: 85% (11024/12861) Updating files: 86% (11061/12861) Updating files: 86% (11090/12861) Updating files: 86% (11121/12861) Updating files: 86% (11184/12861) Updating files: 87% (11190/12861) Updating files: 87% (11254/12861) Updating files: 88% (11318/12861) Updating files: 88% (11358/12861) Updating files: 88% (11371/12861) Updating files: 88% (11441/12861) Updating files: 89% (11447/12861) Updating files: 89% (11502/12861) Updating files: 90% (11575/12861) Updating files: 90% (11655/12861) Updating files: 90% (11693/12861) Updating files: 91% (11704/12861) Updating files: 91% (11764/12861) Updating files: 91% (11807/12861) Updating files: 92% (11833/12861) Updating files: 92% (11861/12861) Updating files: 92% (11913/12861) Updating files: 93% (11961/12861) Updating files: 93% (12085/12861) Updating files: 94% (12090/12861) Updating files: 94% (12214/12861) Updating files: 95% (12218/12861) Updating files: 95% (12339/12861) Updating files: 96% (12347/12861) Updating files: 96% (12380/12861) Updating files: 96% (12422/12861) Updating files: 96% (12466/12861) Updating files: 97% (12476/12861) Updating files: 97% (12548/12861) Updating files: 98% (12604/12861) Updating files: 98% (12651/12861) Updating files: 98% (12670/12861) Updating files: 99% (12733/12861) Updating files: 99% (12851/12861) Updating files: 100% (12861/12861) Updating files: 100% (12861/12861), done. 2023-10-30T13:31:41.866 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2023-10-30T13:31:41.875 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.1 2023-10-30T13:31:42.960 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 2 -eq 48 ] 2023-10-30T13:31:42.961 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.2 2023-10-30T13:31:42.961 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.2 2023-10-30T13:31:42.961 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.2 2023-10-30T13:31:42.961 INFO:tasks.workunit.client.0.smithi148.stdout:v0.2 2023-10-30T13:31:44.066 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.210+0000 7fc90913e640 1 Processor -- start 2023-10-30T13:31:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.210+0000 7fc90913e640 1 -- start start 2023-10-30T13:31:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90913e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9040c6b80 0x7fc9040bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90913e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9040c0a50 con 0x7fc90413f4a0 2023-10-30T13:31:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90913e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90406f6e0 con 0x7fc9040c6f60 2023-10-30T13:31:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90913e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90406f9c0 con 0x7fc9040c6b80 2023-10-30T13:31:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90359e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc90413f4a0 0x7fc9041566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41336/0 (socket says 172.21.15.148:41336) 2023-10-30T13:31:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90359e640 1 -- 172.21.15.148:0/3700023442 learned_addr learned my addr 172.21.15.148:0/3700023442 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:31:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc902d9d640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9040c6b80 0x7fc9040bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746614051 0 0) 0x7fc9040c0a50 con 0x7fc90413f4a0 2023-10-30T13:31:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec005740 con 0x7fc90413f4a0 2023-10-30T13:31:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1992482363 0 0) 0x7fc8ec005740 con 0x7fc90413f4a0 2023-10-30T13:31:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9040c6b80 msgr2=0x7fc9040bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9040c6b80 0x7fc9040bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 >> v1:172.21.15.150:6789/0 conn(0x7fc9040c6f60 legacy=0x7fc90413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90406fca0 con 0x7fc90413f4a0 2023-10-30T13:31:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fc8fc001a00 con 0x7fc90413f4a0 2023-10-30T13:31:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 >> v1:172.21.15.148:6789/0 conn(0x7fc90413f4a0 legacy=0x7fc9041566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9040c6b80 0x7fc8ec005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc8ec0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec006f20 0x7fc8ec0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90406f9c0 con 0x7fc8ec006140 2023-10-30T13:31:44.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec00dc80 con 0x7fc8ec006f20 2023-10-30T13:31:44.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec00de40 con 0x7fc9040c6b80 2023-10-30T13:31:44.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90259c640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc8ec0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc902d9d640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9040c6b80 0x7fc8ec005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90359e640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec006f20 0x7fc8ec0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90259c640 1 -- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9040c6b80 msgr2=0x7fc8ec005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:44.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90259c640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9040c6b80 0x7fc8ec005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90259c640 1 -- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec006f20 msgr2=0x7fc8ec0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:44.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90259c640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec006f20 0x7fc8ec0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc90259c640 1 -- 172.21.15.148:0/3700023442 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec005740 con 0x7fc8ec006140 2023-10-30T13:31:44.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.214+0000 7fc902d9d640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9040c6b80 0x7fc8ec005bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:44.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc90259c640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc8ec0069e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8001750 tx=0x7fc8f801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1e57c585af9fec0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:44.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80052d0 con 0x7fc8ec006140 2023-10-30T13:31:44.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f80a7050 con 0x7fc8ec006140 2023-10-30T13:31:44.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc901d9b640 1 -- 172.21.15.148:0/3700023442 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8003600 con 0x7fc8ec006140 2023-10-30T13:31:44.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc90913e640 1 -- 172.21.15.148:0/3700023442 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 msgr2=0x7fc8ec0069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc90913e640 1 --2- 172.21.15.148:0/3700023442 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc8ec0069e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8001750 tx=0x7fc8f801c3c0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc90359e640 1 -- 172.21.15.148:0/3700023442 reap_dead start 2023-10-30T13:31:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc90913e640 1 -- 172.21.15.148:0/3700023442 shutdown_connections 2023-10-30T13:31:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc90913e640 1 -- 172.21.15.148:0/3700023442 >> 172.21.15.148:0/3700023442 conn(0x7fc9040c7910 msgr2=0x7fc9040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc90913e640 1 -- 172.21.15.148:0/3700023442 shutdown_connections 2023-10-30T13:31:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.218+0000 7fc90913e640 1 -- 172.21.15.148:0/3700023442 wait complete. 2023-10-30T13:31:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 Processor -- start 2023-10-30T13:31:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 -- start start 2023-10-30T13:31:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc904158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc904136a10 0x7fc90413e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc90413e640 0x7fc9040c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9041ebb40 con 0x7fc8ec006140 2023-10-30T13:31:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040ca730 con 0x7fc90413e640 2023-10-30T13:31:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040c8e40 con 0x7fc904136a10 2023-10-30T13:31:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc902d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc904158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc902d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc904158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34492/0 (socket says 172.21.15.148:34492) 2023-10-30T13:31:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc902d9d640 1 -- 172.21.15.148:0/3623891554 learned_addr learned my addr 172.21.15.148:0/3623891554 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90359e640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc90413e640 0x7fc9040c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90259c640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc904136a10 0x7fc90413e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc902d9d640 1 -- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc904136a10 msgr2=0x7fc90413e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc902d9d640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc904136a10 0x7fc90413e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc902d9d640 1 -- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc90413e640 msgr2=0x7fc9040c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc902d9d640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc90413e640 0x7fc9040c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc902d9d640 1 -- 172.21.15.148:0/3623891554 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90413c8e0 con 0x7fc8ec006140 2023-10-30T13:31:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc902d9d640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc904158c50 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009fd0 tx=0x7fc8f4011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc8eb7fe640 1 -- 172.21.15.148:0/3623891554 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4004220 con 0x7fc8ec006140 2023-10-30T13:31:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc8eb7fe640 1 -- 172.21.15.148:0/3623891554 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f4004380 con 0x7fc8ec006140 2023-10-30T13:31:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9041eb5a0 con 0x7fc8ec006140 2023-10-30T13:31:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc8eb7fe640 1 -- 172.21.15.148:0/3623891554 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f400f9e0 con 0x7fc8ec006140 2023-10-30T13:31:44.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9040ca940 con 0x7fc8ec006140 2023-10-30T13:31:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.222+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90404e170 con 0x7fc8ec006140 2023-10-30T13:31:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.226+0000 7fc8eb7fe640 1 -- 172.21.15.148:0/3623891554 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8f400fb40 con 0x7fc8ec006140 2023-10-30T13:31:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.226+0000 7fc8eb7fe640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8f00664a0 0x7fc8f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.226+0000 7fc8eb7fe640 1 -- 172.21.15.148:0/3623891554 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fc8f4084a10 con 0x7fc8ec006140 2023-10-30T13:31:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.226+0000 7fc90259c640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8f00664a0 0x7fc8f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.226+0000 7fc90259c640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8f00664a0 0x7fc8f0068960 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc9040c7380 tx=0x7fc8f80011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.230+0000 7fc8eb7fe640 1 -- 172.21.15.148:0/3623891554 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc8f4051970 con 0x7fc8ec006140 2023-10-30T13:31:44.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9040c8c10 con 0x7fc8ec006140 2023-10-30T13:31:44.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc8eb7fe640 1 -- 172.21.15.148:0/3623891554 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fc8f4022020 con 0x7fc8ec006140 2023-10-30T13:31:44.448 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:31:44.448 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6546,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:44.449 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 92 2023-10-30T13:31:44.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8f00664a0 msgr2=0x7fc8f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8f00664a0 0x7fc8f0068960 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc9040c7380 tx=0x7fc8f80011a0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 msgr2=0x7fc904158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc904158c50 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009fd0 tx=0x7fc8f4011b30 comp rx=0 tx=0).stop 2023-10-30T13:31:44.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 shutdown_connections 2023-10-30T13:31:44.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc8f00664a0 0x7fc8f0068960 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc90413e640 0x7fc9040c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc904136a10 0x7fc90413e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 --2- 172.21.15.148:0/3623891554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc8ec006140 0x7fc904158c50 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 >> 172.21.15.148:0/3623891554 conn(0x7fc9040c7910 msgr2=0x7fc9040c7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:44.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 shutdown_connections 2023-10-30T13:31:44.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.450+0000 7fc90913e640 1 -- 172.21.15.148:0/3623891554 wait complete. 2023-10-30T13:31:44.473 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:31:44.474 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:31:44.474 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:31:44.494 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:31:44.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:44.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe51d4d6640 1 Processor -- start 2023-10-30T13:31:44.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe51d4d6640 1 -- start start 2023-10-30T13:31:44.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe51d4d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe5181538a0 0x7fe51815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe51d4d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 0x7fe5180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe51d4d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5180cb800 con 0x7fe5180ca0e0 2023-10-30T13:31:44.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe51d4d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe51815dc30 con 0x7fe5181538a0 2023-10-30T13:31:44.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe51d4d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5180cb330 con 0x7fe5180cf160 2023-10-30T13:31:44.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe516ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe5181538a0 0x7fe51815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe516ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe5181538a0 0x7fe51815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34502/0 (socket says 172.21.15.148:34502) 2023-10-30T13:31:44.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe516ffd640 1 -- 172.21.15.148:0/275376250 learned_addr learned my addr 172.21.15.148:0/275376250 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:44.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe5177fe640 1 --2- 172.21.15.148:0/275376250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 0x7fe5180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe516ffd640 1 -- 172.21.15.148:0/275376250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 msgr2=0x7fe5180cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:44.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe516ffd640 1 --2- 172.21.15.148:0/275376250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 0x7fe5180cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe516ffd640 1 -- 172.21.15.148:0/275376250 >> v1:172.21.15.150:6789/0 conn(0x7fe5180ca0e0 legacy=0x7fe51815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.678+0000 7fe516ffd640 1 -- 172.21.15.148:0/275376250 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5180cad70 con 0x7fe5181538a0 2023-10-30T13:31:44.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe516ffd640 1 --2- 172.21.15.148:0/275376250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 0x7fe51815d6f0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe508009870 tx=0x7fe50801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec000a003d46c38 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:44.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe5167fc640 1 -- 172.21.15.148:0/275376250 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080a8a30 con 0x7fe5181538a0 2023-10-30T13:31:44.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe5167fc640 1 -- 172.21.15.148:0/275376250 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5080a8b90 con 0x7fe5181538a0 2023-10-30T13:31:44.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe5167fc640 1 -- 172.21.15.148:0/275376250 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080b0440 con 0x7fe5181538a0 2023-10-30T13:31:44.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe51d4d6640 1 -- 172.21.15.148:0/275376250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 msgr2=0x7fe51815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe51d4d6640 1 --2- 172.21.15.148:0/275376250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 0x7fe51815d6f0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe508009870 tx=0x7fe50801ed30 comp rx=0 tx=0).stop 2023-10-30T13:31:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe51d4d6640 1 -- 172.21.15.148:0/275376250 shutdown_connections 2023-10-30T13:31:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe51d4d6640 1 --2- 172.21.15.148:0/275376250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 0x7fe5180cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe51d4d6640 1 --2- 172.21.15.148:0/275376250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 0x7fe51815d6f0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe51d4d6640 1 -- 172.21.15.148:0/275376250 >> 172.21.15.148:0/275376250 conn(0x7fe51806bfe0 msgr2=0x7fe51806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:44.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe51d4d6640 1 -- 172.21.15.148:0/275376250 shutdown_connections 2023-10-30T13:31:44.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe51d4d6640 1 -- 172.21.15.148:0/275376250 wait complete. 2023-10-30T13:31:44.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.682+0000 7fe51d4d6640 1 Processor -- start 2023-10-30T13:31:44.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 -- start start 2023-10-30T13:31:44.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5180ca0e0 0x7fe5181e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 0x7fe51815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 0x7fe51815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51814d4f0 con 0x7fe5181538a0 2023-10-30T13:31:44.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5181e4d40 con 0x7fe5180ca0e0 2023-10-30T13:31:44.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51815b2a0 con 0x7fe5180cf160 2023-10-30T13:31:44.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe5177fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 0x7fe51815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe516ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5180ca0e0 0x7fe5181e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe5177fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 0x7fe51815bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34504/0 (socket says 172.21.15.148:34504) 2023-10-30T13:31:44.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe5177fe640 1 -- 172.21.15.148:0/2755738713 learned_addr learned my addr 172.21.15.148:0/2755738713 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:44.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe50e7fc640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 0x7fe51815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe5177fe640 1 -- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 msgr2=0x7fe51815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe5177fe640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 0x7fe51815b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe5177fe640 1 -- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5180ca0e0 msgr2=0x7fe5181e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe5177fe640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5180ca0e0 0x7fe5181e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe5177fe640 1 -- 172.21.15.148:0/2755738713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5080094d0 con 0x7fe5181538a0 2023-10-30T13:31:44.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe5177fe640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 0x7fe51815bb70 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe51000d580 tx=0x7fe51000da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:44.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe50ffff640 1 -- 172.21.15.148:0/2755738713 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5100125f0 con 0x7fe5181538a0 2023-10-30T13:31:44.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe50ffff640 1 -- 172.21.15.148:0/2755738713 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe510012750 con 0x7fe5181538a0 2023-10-30T13:31:44.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5180cef10 con 0x7fe5181538a0 2023-10-30T13:31:44.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe50ffff640 1 -- 172.21.15.148:0/2755738713 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510012b00 con 0x7fe5181538a0 2023-10-30T13:31:44.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5180c50b0 con 0x7fe5181538a0 2023-10-30T13:31:44.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.686+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5180b2a50 con 0x7fe5181538a0 2023-10-30T13:31:44.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.690+0000 7fe50ffff640 1 -- 172.21.15.148:0/2755738713 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe510012c60 con 0x7fe5181538a0 2023-10-30T13:31:44.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.690+0000 7fe50ffff640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4f80664a0 0x7fe4f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:44.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.690+0000 7fe50ffff640 1 -- 172.21.15.148:0/2755738713 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fe510084620 con 0x7fe5181538a0 2023-10-30T13:31:44.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.690+0000 7fe516ffd640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4f80664a0 0x7fe4f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:44.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.690+0000 7fe516ffd640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4f80664a0 0x7fe4f8068960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe5080b2000 tx=0x7fe508009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:44.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.694+0000 7fe50ffff640 1 -- 172.21.15.148:0/2755738713 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5100516d0 con 0x7fe5181538a0 2023-10-30T13:31:44.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5181e4eb0 con 0x7fe5181538a0 2023-10-30T13:31:44.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe50ffff640 1 -- 172.21.15.148:0/2755738713 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fe51005b070 con 0x7fe5181538a0 2023-10-30T13:31:44.877 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:31:44.877 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6546,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:44.877 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 92 2023-10-30T13:31:44.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4f80664a0 msgr2=0x7fe4f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4f80664a0 0x7fe4f8068960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe5080b2000 tx=0x7fe508009040 comp rx=0 tx=0).stop 2023-10-30T13:31:44.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 msgr2=0x7fe51815bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:44.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 0x7fe51815bb70 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe51000d580 tx=0x7fe51000da40 comp rx=0 tx=0).stop 2023-10-30T13:31:44.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 shutdown_connections 2023-10-30T13:31:44.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4f80664a0 0x7fe4f8068960 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5181538a0 0x7fe51815bb70 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe5180cf160 0x7fe51815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 --2- 172.21.15.148:0/2755738713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5180ca0e0 0x7fe5181e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:44.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 >> 172.21.15.148:0/2755738713 conn(0x7fe51806bfe0 msgr2=0x7fe518065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:44.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.878+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 shutdown_connections 2023-10-30T13:31:44.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:44.882+0000 7fe51d4d6640 1 -- 172.21.15.148:0/2755738713 wait complete. 2023-10-30T13:31:44.898 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:31:46.899 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:47.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb467d3640 1 Processor -- start 2023-10-30T13:31:47.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb467d3640 1 -- start start 2023-10-30T13:31:47.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb467d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7feb4015d6f0 0x7feb4015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb467d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feb400cf160 0x7feb400cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb467d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb400cb800 con 0x7feb401538a0 2023-10-30T13:31:47.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb467d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb400cb330 con 0x7feb4015d6f0 2023-10-30T13:31:47.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb467d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb400cad70 con 0x7feb400cf160 2023-10-30T13:31:47.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb3f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7feb4015d6f0 0x7feb4015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb3ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7feb401538a0 0x7feb400ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41346/0 (socket says 172.21.15.148:41346) 2023-10-30T13:31:47.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb3ffff640 1 -- 172.21.15.148:0/1143952588 learned_addr learned my addr 172.21.15.148:0/1143952588 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:31:47.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb44d70640 1 --2- 172.21.15.148:0/1143952588 >> v2:172.21.15.150:3300/0 conn(0x7feb400cf160 0x7feb400cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb3f7fe640 1 -- 172.21.15.148:0/1143952588 >> v2:172.21.15.150:3300/0 conn(0x7feb400cf160 msgr2=0x7feb400cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:47.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb3f7fe640 1 --2- 172.21.15.148:0/1143952588 >> v2:172.21.15.150:3300/0 conn(0x7feb400cf160 0x7feb400cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb3f7fe640 1 -- 172.21.15.148:0/1143952588 >> v1:172.21.15.148:6789/0 conn(0x7feb401538a0 legacy=0x7feb400ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb3f7fe640 1 -- 172.21.15.148:0/1143952588 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb400cb050 con 0x7feb4015d6f0 2023-10-30T13:31:47.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.050+0000 7feb3effd640 1 -- 172.21.15.148:0/1143952588 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830876355 0 0) 0x7feb400cb800 con 0x7feb401538a0 2023-10-30T13:31:47.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb3f7fe640 1 --2- 172.21.15.148:0/1143952588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb4015d6f0 0x7feb4015a090 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7feb400b5330 tx=0x7feb3401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c496176ba1b4f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:47.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb3effd640 1 -- 172.21.15.148:0/1143952588 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb340a8950 con 0x7feb4015d6f0 2023-10-30T13:31:47.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb3effd640 1 -- 172.21.15.148:0/1143952588 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb340a8ab0 con 0x7feb4015d6f0 2023-10-30T13:31:47.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb3effd640 1 -- 172.21.15.148:0/1143952588 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb340a8e40 con 0x7feb4015d6f0 2023-10-30T13:31:47.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb467d3640 1 -- 172.21.15.148:0/1143952588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb4015d6f0 msgr2=0x7feb4015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb467d3640 1 --2- 172.21.15.148:0/1143952588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb4015d6f0 0x7feb4015a090 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7feb400b5330 tx=0x7feb3401ec50 comp rx=0 tx=0).stop 2023-10-30T13:31:47.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb467d3640 1 -- 172.21.15.148:0/1143952588 shutdown_connections 2023-10-30T13:31:47.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb467d3640 1 --2- 172.21.15.148:0/1143952588 >> v2:172.21.15.150:3300/0 conn(0x7feb400cf160 0x7feb400cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb467d3640 1 --2- 172.21.15.148:0/1143952588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb4015d6f0 0x7feb4015a090 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb467d3640 1 -- 172.21.15.148:0/1143952588 >> 172.21.15.148:0/1143952588 conn(0x7feb4006bfe0 msgr2=0x7feb4006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:47.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb467d3640 1 -- 172.21.15.148:0/1143952588 shutdown_connections 2023-10-30T13:31:47.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.054+0000 7feb467d3640 1 -- 172.21.15.148:0/1143952588 wait complete. 2023-10-30T13:31:47.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 Processor -- start 2023-10-30T13:31:47.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 -- start start 2023-10-30T13:31:47.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb400cf160 0x7feb400cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb401538a0 0x7feb40157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb4015d6f0 0x7feb40157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400cef00 con 0x7feb400cf160 2023-10-30T13:31:47.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb400cec20 con 0x7feb401538a0 2023-10-30T13:31:47.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb400b3b30 con 0x7feb4015d6f0 2023-10-30T13:31:47.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb400cf160 0x7feb400cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb400cf160 0x7feb400cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34530/0 (socket says 172.21.15.148:34530) 2023-10-30T13:31:47.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3ffff640 1 -- 172.21.15.148:0/2214343849 learned_addr learned my addr 172.21.15.148:0/2214343849 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:47.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb44d70640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb4015d6f0 0x7feb40157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3f7fe640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb401538a0 0x7feb40157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3ffff640 1 -- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb4015d6f0 msgr2=0x7feb40157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3ffff640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb4015d6f0 0x7feb40157e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3ffff640 1 -- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb401538a0 msgr2=0x7feb40157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3ffff640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb401538a0 0x7feb40157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3ffff640 1 -- 172.21.15.148:0/2214343849 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb34009420 con 0x7feb400cf160 2023-10-30T13:31:47.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3ffff640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb400cf160 0x7feb400cc3e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7feb300011f0 tx=0x7feb3000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:47.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3cff9640 1 -- 172.21.15.148:0/2214343849 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb300074e0 con 0x7feb400cf160 2023-10-30T13:31:47.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3cff9640 1 -- 172.21.15.148:0/2214343849 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb30007640 con 0x7feb400cf160 2023-10-30T13:31:47.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb3cff9640 1 -- 172.21.15.148:0/2214343849 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30004a10 con 0x7feb400cf160 2023-10-30T13:31:47.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb400b3e10 con 0x7feb400cf160 2023-10-30T13:31:47.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb40159400 con 0x7feb400cf160 2023-10-30T13:31:47.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.058+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb400b2a50 con 0x7feb400cf160 2023-10-30T13:31:47.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.062+0000 7feb3cff9640 1 -- 172.21.15.148:0/2214343849 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feb30004b70 con 0x7feb400cf160 2023-10-30T13:31:47.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.062+0000 7feb3cff9640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feb2c0664a0 0x7feb2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:47.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.062+0000 7feb3cff9640 1 -- 172.21.15.148:0/2214343849 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7feb30083580 con 0x7feb400cf160 2023-10-30T13:31:47.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.062+0000 7feb3f7fe640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feb2c0664a0 0x7feb2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:47.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.062+0000 7feb3f7fe640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feb2c0664a0 0x7feb2c068960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feb340b2000 tx=0x7feb34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:47.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.066+0000 7feb3cff9640 1 -- 172.21.15.148:0/2214343849 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feb30050600 con 0x7feb400cf160 2023-10-30T13:31:47.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb4004e170 con 0x7feb400cf160 2023-10-30T13:31:47.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb3cff9640 1 -- 172.21.15.148:0/2214343849 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6508 (secure 0 0 0) 0x7feb30018090 con 0x7feb400cf160 2023-10-30T13:31:47.245 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:31:47.245 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6546,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:47.246 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 92 2023-10-30T13:31:47.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feb2c0664a0 msgr2=0x7feb2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb467d3640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feb2c0664a0 0x7feb2c068960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feb340b2000 tx=0x7feb34008040 comp rx=0 tx=0).stop 2023-10-30T13:31:47.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb400cf160 msgr2=0x7feb400cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:47.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb467d3640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb400cf160 0x7feb400cc3e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7feb300011f0 tx=0x7feb3000c9b0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 shutdown_connections 2023-10-30T13:31:47.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb467d3640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7feb2c0664a0 0x7feb2c068960 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb467d3640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7feb4015d6f0 0x7feb40157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.246+0000 7feb467d3640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb401538a0 0x7feb40157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.250+0000 7feb467d3640 1 --2- 172.21.15.148:0/2214343849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feb400cf160 0x7feb400cc3e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:47.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.250+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 >> 172.21.15.148:0/2214343849 conn(0x7feb4006bfe0 msgr2=0x7feb400c56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:47.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.250+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 shutdown_connections 2023-10-30T13:31:47.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:47.250+0000 7feb467d3640 1 -- 172.21.15.148:0/2214343849 wait complete. 2023-10-30T13:31:47.266 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:31:49.267 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:49.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f529a89e640 1 Processor -- start 2023-10-30T13:31:49.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f529a89e640 1 -- start start 2023-10-30T13:31:49.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f529a89e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f52940c05d0 0x7f529413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:49.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f529a89e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f52940c6f60 0x7f529413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:49.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f529a89e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52940c0a50 con 0x7f52940c6b80 2023-10-30T13:31:49.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f529a89e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529406f6e0 con 0x7f52940c6f60 2023-10-30T13:31:49.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f529a89e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529406f9c0 con 0x7f52940c05d0 2023-10-30T13:31:49.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f5293fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f52940c6b80 0x7f52940bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41356/0 (socket says 172.21.15.148:41356) 2023-10-30T13:31:49.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f5293fff640 1 -- 172.21.15.148:0/3757638490 learned_addr learned my addr 172.21.15.148:0/3757638490 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:31:49.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f5298e3b640 1 --2- 172.21.15.148:0/3757638490 >> v2:172.21.15.148:3300/0 conn(0x7f52940c6f60 0x7f529413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:49.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f52937fe640 1 --2- 172.21.15.148:0/3757638490 >> v2:172.21.15.150:3300/0 conn(0x7f52940c05d0 0x7f529413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:49.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f5292ffd640 1 -- 172.21.15.148:0/3757638490 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673935996 0 0) 0x7f52940c0a50 con 0x7f52940c6b80 2023-10-30T13:31:49.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f5292ffd640 1 -- 172.21.15.148:0/3757638490 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f527c003530 con 0x7f52940c6b80 2023-10-30T13:31:49.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f5298e3b640 1 -- 172.21.15.148:0/3757638490 >> v2:172.21.15.150:3300/0 conn(0x7f52940c05d0 msgr2=0x7f529413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:49.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.418+0000 7f5298e3b640 1 --2- 172.21.15.148:0/3757638490 >> v2:172.21.15.150:3300/0 conn(0x7f52940c05d0 0x7f529413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f5298e3b640 1 -- 172.21.15.148:0/3757638490 >> v1:172.21.15.148:6789/0 conn(0x7f52940c6b80 legacy=0x7f52940bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:49.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f5298e3b640 1 -- 172.21.15.148:0/3757638490 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529406fca0 con 0x7f52940c6f60 2023-10-30T13:31:49.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f5298e3b640 1 --2- 172.21.15.148:0/3757638490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 0x7f529413f2d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f529406c7e0 tx=0x7f528c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2677ce586d64f07 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:49.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f5292ffd640 1 -- 172.21.15.148:0/3757638490 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c0a58e0 con 0x7f52940c6f60 2023-10-30T13:31:49.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f5292ffd640 1 -- 172.21.15.148:0/3757638490 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c0a5a40 con 0x7f52940c6f60 2023-10-30T13:31:49.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f5292ffd640 1 -- 172.21.15.148:0/3757638490 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c0a5dd0 con 0x7f52940c6f60 2023-10-30T13:31:49.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f529a89e640 1 -- 172.21.15.148:0/3757638490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 msgr2=0x7f529413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:49.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f529a89e640 1 --2- 172.21.15.148:0/3757638490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 0x7f529413f2d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f529406c7e0 tx=0x7f528c01c910 comp rx=0 tx=0).stop 2023-10-30T13:31:49.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f529a89e640 1 -- 172.21.15.148:0/3757638490 shutdown_connections 2023-10-30T13:31:49.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f529a89e640 1 --2- 172.21.15.148:0/3757638490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 0x7f529413f2d0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f529a89e640 1 --2- 172.21.15.148:0/3757638490 >> v2:172.21.15.150:3300/0 conn(0x7f52940c05d0 0x7f529413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f529a89e640 1 -- 172.21.15.148:0/3757638490 >> 172.21.15.148:0/3757638490 conn(0x7f52940c7910 msgr2=0x7f52940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:49.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f529a89e640 1 -- 172.21.15.148:0/3757638490 shutdown_connections 2023-10-30T13:31:49.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.422+0000 7f529a89e640 1 -- 172.21.15.148:0/3757638490 wait complete. 2023-10-30T13:31:49.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f529a89e640 1 Processor -- start 2023-10-30T13:31:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f529a89e640 1 -- start start 2023-10-30T13:31:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f529a89e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52940c05d0 0x7f52941550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f529a89e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52940c6b80 0x7f52940c1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f529a89e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 0x7f52941ebbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:49.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f529a89e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52941eae90 con 0x7f52940c6f60 2023-10-30T13:31:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f529a89e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52941eb730 con 0x7f52940c6b80 2023-10-30T13:31:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f529a89e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52941f58b0 con 0x7f52940c05d0 2023-10-30T13:31:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5298e3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 0x7f52941ebbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5298e3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 0x7f52941ebbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34560/0 (socket says 172.21.15.148:34560) 2023-10-30T13:31:49.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5298e3b640 1 -- 172.21.15.148:0/3473017070 learned_addr learned my addr 172.21.15.148:0/3473017070 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5293fff640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52940c05d0 0x7f52941550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f52937fe640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52940c6b80 0x7f52940c1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5298e3b640 1 -- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52940c05d0 msgr2=0x7f52941550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5298e3b640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52940c05d0 0x7f52941550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5298e3b640 1 -- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52940c6b80 msgr2=0x7f52940c1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:49.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5298e3b640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52940c6b80 0x7f52940c1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5298e3b640 1 -- 172.21.15.148:0/3473017070 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c009a70 con 0x7f52940c6f60 2023-10-30T13:31:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5298e3b640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 0x7f52941ebbc0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f528c019520 tx=0x7f528c0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5290ff9640 1 -- 172.21.15.148:0/3473017070 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c0ad8d0 con 0x7f52940c6f60 2023-10-30T13:31:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5290ff9640 1 -- 172.21.15.148:0/3473017070 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c0ada30 con 0x7f52940c6f60 2023-10-30T13:31:49.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f5290ff9640 1 -- 172.21.15.148:0/3473017070 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c0adde0 con 0x7f52940c6f60 2023-10-30T13:31:49.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.426+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52941f5b90 con 0x7f52940c6f60 2023-10-30T13:31:49.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.430+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52941eb3c0 con 0x7f52940c6f60 2023-10-30T13:31:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.430+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52941eb8a0 con 0x7f52940c6f60 2023-10-30T13:31:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.430+0000 7f5290ff9640 1 -- 172.21.15.148:0/3473017070 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f528c0b4050 con 0x7f52940c6f60 2023-10-30T13:31:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.430+0000 7f5290ff9640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f52800664d0 0x7f5280068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.430+0000 7f5290ff9640 1 -- 172.21.15.148:0/3473017070 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f528c11ee10 con 0x7f52940c6f60 2023-10-30T13:31:49.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.430+0000 7f5293fff640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f52800664d0 0x7f5280068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:49.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.434+0000 7f5293fff640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f52800664d0 0x7f5280068990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5284005890 tx=0x7f5284009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:49.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.434+0000 7f5290ff9640 1 -- 172.21.15.148:0/3473017070 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f528c122030 con 0x7f52940c6f60 2023-10-30T13:31:49.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.614+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f529404e170 con 0x7f52940c6f60 2023-10-30T13:31:49.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.614+0000 7f5290ff9640 1 -- 172.21.15.148:0/3473017070 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f528c0ebe00 con 0x7f52940c6f60 2023-10-30T13:31:49.614 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:31:49.614 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6546,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:49.614 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 92 2023-10-30T13:31:49.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f52800664d0 msgr2=0x7f5280068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:49.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f52800664d0 0x7f5280068990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5284005890 tx=0x7f5284009040 comp rx=0 tx=0).stop 2023-10-30T13:31:49.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 msgr2=0x7f52941ebbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:49.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 0x7f52941ebbc0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f528c019520 tx=0x7f528c0196b0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 shutdown_connections 2023-10-30T13:31:49.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f52800664d0 0x7f5280068990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f52940c6f60 0x7f52941ebbc0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52940c6b80 0x7f52940c1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 --2- 172.21.15.148:0/3473017070 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f52940c05d0 0x7f52941550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:49.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 >> 172.21.15.148:0/3473017070 conn(0x7f52940c7910 msgr2=0x7f52940bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:49.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 shutdown_connections 2023-10-30T13:31:49.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:49.618+0000 7f529a89e640 1 -- 172.21.15.148:0/3473017070 wait complete. 2023-10-30T13:31:49.635 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:31:51.637 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:51.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.786+0000 7fb460765640 1 Processor -- start 2023-10-30T13:31:51.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.786+0000 7fb460765640 1 -- start start 2023-10-30T13:31:51.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb460765640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb458153bf0 0x7fb458071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:51.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb460765640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4580c6fe0 0x7fb4580c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:51.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb460765640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4580700b0 con 0x7fb4580c79e0 2023-10-30T13:31:51.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb460765640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4580c6810 con 0x7fb458153bf0 2023-10-30T13:31:51.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb460765640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb458159900 con 0x7fb4580c6fe0 2023-10-30T13:31:51.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45e501640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb458153bf0 0x7fb458071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:51.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45ed02640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb4580c79e0 0x7fb4580cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41362/0 (socket says 172.21.15.148:41362) 2023-10-30T13:31:51.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45ed02640 1 -- 172.21.15.148:0/2094458918 learned_addr learned my addr 172.21.15.148:0/2094458918 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:31:51.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45dd00640 1 --2- 172.21.15.148:0/2094458918 >> v2:172.21.15.150:3300/0 conn(0x7fb4580c6fe0 0x7fb4580c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:51.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45e501640 1 -- 172.21.15.148:0/2094458918 >> v2:172.21.15.150:3300/0 conn(0x7fb4580c6fe0 msgr2=0x7fb4580c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:51.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45e501640 1 --2- 172.21.15.148:0/2094458918 >> v2:172.21.15.150:3300/0 conn(0x7fb4580c6fe0 0x7fb4580c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:51.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45e501640 1 -- 172.21.15.148:0/2094458918 >> v1:172.21.15.148:6789/0 conn(0x7fb4580c79e0 legacy=0x7fb4580cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:51.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45e501640 1 -- 172.21.15.148:0/2094458918 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4580c6530 con 0x7fb458153bf0 2023-10-30T13:31:51.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45dd00640 1 --2- 172.21.15.148:0/2094458918 >> v2:172.21.15.150:3300/0 conn(0x7fb4580c6fe0 0x7fb4580c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:51.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.790+0000 7fb45d4ff640 1 -- 172.21.15.148:0/2094458918 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382423406 0 0) 0x7fb4580700b0 con 0x7fb4580c79e0 2023-10-30T13:31:51.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb45e501640 1 --2- 172.21.15.148:0/2094458918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb458153bf0 0x7fb458071260 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb4580723d0 tx=0x7fb45401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbc82204517b1ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:51.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb45d4ff640 1 -- 172.21.15.148:0/2094458918 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4540a8950 con 0x7fb458153bf0 2023-10-30T13:31:51.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb45d4ff640 1 -- 172.21.15.148:0/2094458918 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4540a8ab0 con 0x7fb458153bf0 2023-10-30T13:31:51.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb45d4ff640 1 -- 172.21.15.148:0/2094458918 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4540a8e40 con 0x7fb458153bf0 2023-10-30T13:31:51.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 -- 172.21.15.148:0/2094458918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb458153bf0 msgr2=0x7fb458071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:51.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 --2- 172.21.15.148:0/2094458918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb458153bf0 0x7fb458071260 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb4580723d0 tx=0x7fb45401ec50 comp rx=0 tx=0).stop 2023-10-30T13:31:51.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 -- 172.21.15.148:0/2094458918 shutdown_connections 2023-10-30T13:31:51.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 --2- 172.21.15.148:0/2094458918 >> v2:172.21.15.150:3300/0 conn(0x7fb4580c6fe0 0x7fb4580c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:51.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 --2- 172.21.15.148:0/2094458918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb458153bf0 0x7fb458071260 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:51.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 -- 172.21.15.148:0/2094458918 >> 172.21.15.148:0/2094458918 conn(0x7fb45806c6e0 msgr2=0x7fb458069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:51.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 -- 172.21.15.148:0/2094458918 shutdown_connections 2023-10-30T13:31:51.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 -- 172.21.15.148:0/2094458918 wait complete. 2023-10-30T13:31:51.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 Processor -- start 2023-10-30T13:31:51.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 -- start start 2023-10-30T13:31:51.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.794+0000 7fb460765640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4580c6fe0 0x7fb4581ef9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:51.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb460765640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4580c79e0 0x7fb4580c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:51.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb460765640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb458153bf0 0x7fb4580c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:51.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb460765640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4580ca4f0 con 0x7fb4580c79e0 2023-10-30T13:31:51.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb460765640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4580c3150 con 0x7fb458153bf0 2023-10-30T13:31:51.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb460765640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb458157110 con 0x7fb4580c6fe0 2023-10-30T13:31:51.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45dd00640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4580c79e0 0x7fb4580c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:51.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45dd00640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4580c79e0 0x7fb4580c38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34580/0 (socket says 172.21.15.148:34580) 2023-10-30T13:31:51.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45dd00640 1 -- 172.21.15.148:0/7725487 learned_addr learned my addr 172.21.15.148:0/7725487 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:51.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45e501640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4580c6fe0 0x7fb4581ef9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:51.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45ed02640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb458153bf0 0x7fb4580c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:51.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45dd00640 1 -- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4580c6fe0 msgr2=0x7fb4581ef9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45dd00640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4580c6fe0 0x7fb4581ef9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45dd00640 1 -- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb458153bf0 msgr2=0x7fb4580c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45dd00640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb458153bf0 0x7fb4580c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45dd00640 1 -- 172.21.15.148:0/7725487 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb454009420 con 0x7fb4580c79e0 2023-10-30T13:31:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb45dd00640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4580c79e0 0x7fb4580c38c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fb44800e840 tx=0x7fb44800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:51.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb44effd640 1 -- 172.21.15.148:0/7725487 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44800c8d0 con 0x7fb4580c79e0 2023-10-30T13:31:51.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb44effd640 1 -- 172.21.15.148:0/7725487 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44800ca30 con 0x7fb4580c79e0 2023-10-30T13:31:51.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb44effd640 1 -- 172.21.15.148:0/7725487 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44800cdc0 con 0x7fb4580c79e0 2023-10-30T13:31:51.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4580ca740 con 0x7fb4580c79e0 2023-10-30T13:31:51.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4580cad00 con 0x7fb4580c79e0 2023-10-30T13:31:51.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45804e170 con 0x7fb4580c79e0 2023-10-30T13:31:51.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.798+0000 7fb44effd640 1 -- 172.21.15.148:0/7725487 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb448013410 con 0x7fb4580c79e0 2023-10-30T13:31:51.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.802+0000 7fb44effd640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb43c0664a0 0x7fb43c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:51.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.802+0000 7fb44effd640 1 -- 172.21.15.148:0/7725487 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fb448084140 con 0x7fb4580c79e0 2023-10-30T13:31:51.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.802+0000 7fb45e501640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb43c0664a0 0x7fb43c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:51.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.802+0000 7fb45e501640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb43c0664a0 0x7fb43c068960 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb4540b2000 tx=0x7fb4540aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:51.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.806+0000 7fb44effd640 1 -- 172.21.15.148:0/7725487 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb448051130 con 0x7fb4580c79e0 2023-10-30T13:31:51.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.986+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb45806fe90 con 0x7fb4580c79e0 2023-10-30T13:31:51.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.986+0000 7fb44effd640 1 -- 172.21.15.148:0/7725487 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fb448054750 con 0x7fb4580c79e0 2023-10-30T13:31:51.987 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:31:51.987 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6546,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:51.987 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 92 2023-10-30T13:31:51.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb43c0664a0 msgr2=0x7fb43c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:51.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb43c0664a0 0x7fb43c068960 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb4540b2000 tx=0x7fb4540aa070 comp rx=0 tx=0).stop 2023-10-30T13:31:51.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4580c79e0 msgr2=0x7fb4580c38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:51.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4580c79e0 0x7fb4580c38c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fb44800e840 tx=0x7fb44800ed00 comp rx=0 tx=0).stop 2023-10-30T13:31:51.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 shutdown_connections 2023-10-30T13:31:51.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb43c0664a0 0x7fb43c068960 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:51.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4580c79e0 0x7fb4580c38c0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:51.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb458153bf0 0x7fb4580c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:51.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 --2- 172.21.15.148:0/7725487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4580c6fe0 0x7fb4581ef9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:51.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 >> 172.21.15.148:0/7725487 conn(0x7fb45806c6e0 msgr2=0x7fb4581575d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:51.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 shutdown_connections 2023-10-30T13:31:51.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:51.990+0000 7fb460765640 1 -- 172.21.15.148:0/7725487 wait complete. 2023-10-30T13:31:52.007 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:31:54.008 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:54.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.158+0000 7fdb4413e640 1 Processor -- start 2023-10-30T13:31:54.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.158+0000 7fdb4413e640 1 -- start start 2023-10-30T13:31:54.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.158+0000 7fdb4413e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdb3c1580e0 0x7fdb3c070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.158+0000 7fdb4413e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0c5810 con 0x7fdb3c15e700 2023-10-30T13:31:54.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.158+0000 7fdb4413e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0c6990 con 0x7fdb3c0c6540 2023-10-30T13:31:54.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.158+0000 7fdb4413e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c06acb0 con 0x7fdb3c1580e0 2023-10-30T13:31:54.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.158+0000 7fdb41eda640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdb3c1580e0 0x7fdb3c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.158+0000 7fdb41eda640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdb3c1580e0 0x7fdb3c070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36568/0 (socket says 172.21.15.148:36568) 2023-10-30T13:31:54.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.158+0000 7fdb41eda640 1 -- 172.21.15.148:0/2429251237 learned_addr learned my addr 172.21.15.148:0/2429251237 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:54.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb41eda640 1 -- 172.21.15.148:0/2429251237 >> v1:172.21.15.150:6789/0 conn(0x7fdb3c0c6540 legacy=0x7fdb3c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb41eda640 1 -- 172.21.15.148:0/2429251237 >> v1:172.21.15.148:6789/0 conn(0x7fdb3c15e700 legacy=0x7fdb3c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb41eda640 1 -- 172.21.15.148:0/2429251237 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c15eb90 con 0x7fdb3c1580e0 2023-10-30T13:31:54.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb40ed8640 1 -- 172.21.15.148:0/2429251237 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883980883 0 0) 0x7fdb3c0c5810 con 0x7fdb3c15e700 2023-10-30T13:31:54.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb41eda640 1 --2- 172.21.15.148:0/2429251237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1580e0 0x7fdb3c070740 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009870 tx=0x7fdb3801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d14ca3d2bb90a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:54.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb40ed8640 1 -- 172.21.15.148:0/2429251237 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb380a9400 con 0x7fdb3c1580e0 2023-10-30T13:31:54.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb40ed8640 1 -- 172.21.15.148:0/2429251237 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb380a9560 con 0x7fdb3c1580e0 2023-10-30T13:31:54.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb40ed8640 1 -- 172.21.15.148:0/2429251237 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb380a98f0 con 0x7fdb3c1580e0 2023-10-30T13:31:54.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb4413e640 1 -- 172.21.15.148:0/2429251237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1580e0 msgr2=0x7fdb3c070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.162+0000 7fdb4413e640 1 --2- 172.21.15.148:0/2429251237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1580e0 0x7fdb3c070740 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009870 tx=0x7fdb3801c540 comp rx=0 tx=0).stop 2023-10-30T13:31:54.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 -- 172.21.15.148:0/2429251237 shutdown_connections 2023-10-30T13:31:54.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 --2- 172.21.15.148:0/2429251237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c1580e0 0x7fdb3c070740 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 -- 172.21.15.148:0/2429251237 >> 172.21.15.148:0/2429251237 conn(0x7fdb3c0c7910 msgr2=0x7fdb3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:54.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 -- 172.21.15.148:0/2429251237 shutdown_connections 2023-10-30T13:31:54.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 -- 172.21.15.148:0/2429251237 wait complete. 2023-10-30T13:31:54.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 Processor -- start 2023-10-30T13:31:54.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 -- start start 2023-10-30T13:31:54.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb3c0c6540 0x7fdb3c14e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3c1580e0 0x7fdb3c149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c15e700 0x7fdb3c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0c2280 con 0x7fdb3c15e700 2023-10-30T13:31:54.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c14c5c0 con 0x7fdb3c1580e0 2023-10-30T13:31:54.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb4413e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c1f7070 con 0x7fdb3c0c6540 2023-10-30T13:31:54.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb426db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c15e700 0x7fdb3c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb426db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c15e700 0x7fdb3c069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36574/0 (socket says 172.21.15.148:36574) 2023-10-30T13:31:54.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb426db640 1 -- 172.21.15.148:0/3556544622 learned_addr learned my addr 172.21.15.148:0/3556544622 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:54.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb41eda640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb3c0c6540 0x7fdb3c14e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb416d9640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3c1580e0 0x7fdb3c149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb426db640 1 -- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb3c0c6540 msgr2=0x7fdb3c14e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb426db640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb3c0c6540 0x7fdb3c14e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb426db640 1 -- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3c1580e0 msgr2=0x7fdb3c149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb426db640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3c1580e0 0x7fdb3c149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb426db640 1 -- 172.21.15.148:0/3556544622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb380094d0 con 0x7fdb3c15e700 2023-10-30T13:31:54.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb416d9640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3c1580e0 0x7fdb3c149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:54.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.166+0000 7fdb426db640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c15e700 0x7fdb3c069e90 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdb34002240 tx=0x7fdb3400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:54.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb327fc640 1 -- 172.21.15.148:0/3556544622 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34003db0 con 0x7fdb3c15e700 2023-10-30T13:31:54.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb327fc640 1 -- 172.21.15.148:0/3556544622 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3400ce30 con 0x7fdb3c15e700 2023-10-30T13:31:54.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c14c840 con 0x7fdb3c15e700 2023-10-30T13:31:54.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb327fc640 1 -- 172.21.15.148:0/3556544622 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34005270 con 0x7fdb3c15e700 2023-10-30T13:31:54.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c1f7330 con 0x7fdb3c15e700 2023-10-30T13:31:54.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c04e170 con 0x7fdb3c15e700 2023-10-30T13:31:54.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb327fc640 1 -- 172.21.15.148:0/3556544622 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdb340053d0 con 0x7fdb3c15e700 2023-10-30T13:31:54.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb327fc640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdb200664a0 0x7fdb20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:54.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb327fc640 1 -- 172.21.15.148:0/3556544622 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fdb340839e0 con 0x7fdb3c15e700 2023-10-30T13:31:54.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.170+0000 7fdb41eda640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdb200664a0 0x7fdb20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:54.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.174+0000 7fdb41eda640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdb200664a0 0x7fdb20068960 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdb380ae000 tx=0x7fdb380b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:54.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.174+0000 7fdb327fc640 1 -- 172.21.15.148:0/3556544622 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb34050a80 con 0x7fdb3c15e700 2023-10-30T13:31:54.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.354+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb3c1693f0 con 0x7fdb3c15e700 2023-10-30T13:31:54.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.354+0000 7fdb327fc640 1 -- 172.21.15.148:0/3556544622 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fdb34017020 con 0x7fdb3c15e700 2023-10-30T13:31:54.354 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:31:54.354 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6546,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:54.356 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 92 2023-10-30T13:31:54.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdb200664a0 msgr2=0x7fdb20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdb200664a0 0x7fdb20068960 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdb380ae000 tx=0x7fdb380b3040 comp rx=0 tx=0).stop 2023-10-30T13:31:54.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c15e700 msgr2=0x7fdb3c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:54.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c15e700 0x7fdb3c069e90 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdb34002240 tx=0x7fdb3400cbe0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 shutdown_connections 2023-10-30T13:31:54.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdb200664a0 0x7fdb20068960 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb3c15e700 0x7fdb3c069e90 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3c1580e0 0x7fdb3c149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 --2- 172.21.15.148:0/3556544622 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb3c0c6540 0x7fdb3c14e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:54.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 >> 172.21.15.148:0/3556544622 conn(0x7fdb3c0c7910 msgr2=0x7fdb3c0c17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:54.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 shutdown_connections 2023-10-30T13:31:54.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:54.358+0000 7fdb4413e640 1 -- 172.21.15.148:0/3556544622 wait complete. 2023-10-30T13:31:54.375 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:31:56.377 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:56.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.526+0000 7fec8a89c640 1 Processor -- start 2023-10-30T13:31:56.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec8a89c640 1 -- start start 2023-10-30T13:31:56.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec8a89c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fec8406f7a0 0x7fec8406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec8a89c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec840c6bd0 0x7fec84069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec8a89c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec84071110 0x7fec84158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec8a89c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec840700c0 con 0x7fec8406f7a0 2023-10-30T13:31:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec8a89c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8406a3d0 con 0x7fec840c6bd0 2023-10-30T13:31:56.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec8a89c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec84066cf0 con 0x7fec84071110 2023-10-30T13:31:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec83fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fec8406f7a0 0x7fec8406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec83fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fec8406f7a0 0x7fec8406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36578/0 (socket says 172.21.15.148:36578) 2023-10-30T13:31:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec83fff640 1 -- 172.21.15.148:0/1167587199 learned_addr learned my addr 172.21.15.148:0/1167587199 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec837fe640 1 --2- 172.21.15.148:0/1167587199 >> v2:172.21.15.150:3300/0 conn(0x7fec840c6bd0 0x7fec84069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec88e39640 1 --2- 172.21.15.148:0/1167587199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec84071110 0x7fec84158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec83fff640 1 -- 172.21.15.148:0/1167587199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec84071110 msgr2=0x7fec84158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec83fff640 1 --2- 172.21.15.148:0/1167587199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec84071110 0x7fec84158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec83fff640 1 -- 172.21.15.148:0/1167587199 >> v2:172.21.15.150:3300/0 conn(0x7fec840c6bd0 msgr2=0x7fec84069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec83fff640 1 --2- 172.21.15.148:0/1167587199 >> v2:172.21.15.150:3300/0 conn(0x7fec840c6bd0 0x7fec84069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec83fff640 1 -- 172.21.15.148:0/1167587199 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec840c70d0 con 0x7fec8406f7a0 2023-10-30T13:31:56.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.530+0000 7fec837fe640 1 --2- 172.21.15.148:0/1167587199 >> v2:172.21.15.150:3300/0 conn(0x7fec840c6bd0 0x7fec84069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec83fff640 1 --2- 172.21.15.148:0/1167587199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec8406f7a0 0x7fec8406fb80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fec8406f1c0 tx=0x7fec7401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa377b87986c10e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec82ffd640 1 -- 172.21.15.148:0/1167587199 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec740a89b0 con 0x7fec8406f7a0 2023-10-30T13:31:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec82ffd640 1 -- 172.21.15.148:0/1167587199 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec740a8b10 con 0x7fec8406f7a0 2023-10-30T13:31:56.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec82ffd640 1 -- 172.21.15.148:0/1167587199 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec740a8ea0 con 0x7fec8406f7a0 2023-10-30T13:31:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 -- 172.21.15.148:0/1167587199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec8406f7a0 msgr2=0x7fec8406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1167587199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec8406f7a0 0x7fec8406fb80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fec8406f1c0 tx=0x7fec7401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 -- 172.21.15.148:0/1167587199 shutdown_connections 2023-10-30T13:31:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1167587199 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec84071110 0x7fec84158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1167587199 >> v2:172.21.15.150:3300/0 conn(0x7fec840c6bd0 0x7fec84069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1167587199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec8406f7a0 0x7fec8406fb80 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 -- 172.21.15.148:0/1167587199 >> 172.21.15.148:0/1167587199 conn(0x7fec840c7910 msgr2=0x7fec840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:56.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 -- 172.21.15.148:0/1167587199 shutdown_connections 2023-10-30T13:31:56.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 -- 172.21.15.148:0/1167587199 wait complete. 2023-10-30T13:31:56.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.534+0000 7fec8a89c640 1 Processor -- start 2023-10-30T13:31:56.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 -- start start 2023-10-30T13:31:56.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec8406f7a0 0x7fec84150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec84071110 0x7fec841ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec840c6bd0 0x7fec8414c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8414ffd0 con 0x7fec84071110 2023-10-30T13:31:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840c39e0 con 0x7fec8406f7a0 2023-10-30T13:31:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8414f0e0 con 0x7fec840c6bd0 2023-10-30T13:31:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec837fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec84071110 0x7fec841ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec837fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec84071110 0x7fec841ee4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36588/0 (socket says 172.21.15.148:36588) 2023-10-30T13:31:56.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec837fe640 1 -- 172.21.15.148:0/1912857057 learned_addr learned my addr 172.21.15.148:0/1912857057 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec83fff640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec8406f7a0 0x7fec84150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec88e39640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec840c6bd0 0x7fec8414c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec837fe640 1 -- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec840c6bd0 msgr2=0x7fec8414c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec837fe640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec840c6bd0 0x7fec8414c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec837fe640 1 -- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec8406f7a0 msgr2=0x7fec84150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec837fe640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec8406f7a0 0x7fec84150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec837fe640 1 -- 172.21.15.148:0/1912857057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740094d0 con 0x7fec84071110 2023-10-30T13:31:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec88e39640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec840c6bd0 0x7fec8414c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec837fe640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec84071110 0x7fec841ee4e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fec7800d7d0 tx=0x7fec7800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec80ff9640 1 -- 172.21.15.148:0/1912857057 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec780096a0 con 0x7fec84071110 2023-10-30T13:31:56.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec80ff9640 1 -- 172.21.15.148:0/1912857057 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec78009800 con 0x7fec84071110 2023-10-30T13:31:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8414f3c0 con 0x7fec84071110 2023-10-30T13:31:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec80ff9640 1 -- 172.21.15.148:0/1912857057 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78009bb0 con 0x7fec84071110 2023-10-30T13:31:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8414f980 con 0x7fec84071110 2023-10-30T13:31:56.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.538+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec840b11e0 con 0x7fec84071110 2023-10-30T13:31:56.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.542+0000 7fec80ff9640 1 -- 172.21.15.148:0/1912857057 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fec78009d10 con 0x7fec84071110 2023-10-30T13:31:56.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.542+0000 7fec80ff9640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec700664a0 0x7fec70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.542+0000 7fec80ff9640 1 -- 172.21.15.148:0/1912857057 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fec78084a30 con 0x7fec84071110 2023-10-30T13:31:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.542+0000 7fec83fff640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec700664a0 0x7fec70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.542+0000 7fec83fff640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec700664a0 0x7fec70068960 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fec740b2000 tx=0x7fec74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:56.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.546+0000 7fec80ff9640 1 -- 172.21.15.148:0/1912857057 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fec78051990 con 0x7fec84071110 2023-10-30T13:31:56.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec8404e170 con 0x7fec84071110 2023-10-30T13:31:56.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec80ff9640 1 -- 172.21.15.148:0/1912857057 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6508 (secure 0 0 0) 0x7fec78054fb0 con 0x7fec84071110 2023-10-30T13:31:56.724 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:31:56.724 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6546,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:56.724 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 92 2023-10-30T13:31:56.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec700664a0 msgr2=0x7fec70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec700664a0 0x7fec70068960 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fec740b2000 tx=0x7fec74009040 comp rx=0 tx=0).stop 2023-10-30T13:31:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec84071110 msgr2=0x7fec841ee4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec84071110 0x7fec841ee4e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fec7800d7d0 tx=0x7fec7800dc90 comp rx=0 tx=0).stop 2023-10-30T13:31:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 shutdown_connections 2023-10-30T13:31:56.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fec700664a0 0x7fec70068960 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fec84071110 0x7fec841ee4e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fec840c6bd0 0x7fec8414c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 --2- 172.21.15.148:0/1912857057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec8406f7a0 0x7fec84150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 >> 172.21.15.148:0/1912857057 conn(0x7fec840c7910 msgr2=0x7fec84156d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:56.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 shutdown_connections 2023-10-30T13:31:56.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:56.726+0000 7fec8a89c640 1 -- 172.21.15.148:0/1912857057 wait complete. 2023-10-30T13:31:56.745 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:31:58.746 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:31:58.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ecb09e640 1 Processor -- start 2023-10-30T13:31:58.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ecb09e640 1 -- start start 2023-10-30T13:31:58.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ecb09e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7ec41538a0 0x7f7ec415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:58.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ecb09e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec40cb800 con 0x7f7ec40cf160 2023-10-30T13:31:58.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ecb09e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec415dc30 con 0x7f7ec40ca0e0 2023-10-30T13:31:58.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ecb09e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec40cb330 con 0x7f7ec41538a0 2023-10-30T13:31:58.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ec963b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7ec40cf160 0x7f7ec4164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43592/0 (socket says 172.21.15.148:43592) 2023-10-30T13:31:58.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ec963b640 1 -- 172.21.15.148:0/1877539510 learned_addr learned my addr 172.21.15.148:0/1877539510 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:31:58.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ec8e3a640 1 --2- 172.21.15.148:0/1877539510 >> v2:172.21.15.150:3300/0 conn(0x7f7ec41538a0 0x7f7ec415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:58.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771400320 0 0) 0x7f7ec40cb800 con 0x7f7ec40cf160 2023-10-30T13:31:58.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.898+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb4003530 con 0x7f7ec40cf160 2023-10-30T13:31:58.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1202349386 0 0) 0x7f7eb4003530 con 0x7f7ec40cf160 2023-10-30T13:31:58.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 >> v2:172.21.15.150:3300/0 conn(0x7f7ec41538a0 msgr2=0x7f7ec415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:58.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 --2- 172.21.15.148:0/1877539510 >> v2:172.21.15.150:3300/0 conn(0x7f7ec41538a0 0x7f7ec415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:58.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 >> v1:172.21.15.150:6789/0 conn(0x7f7ec40ca0e0 legacy=0x7f7ec415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:58.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec40cad70 con 0x7f7ec40cf160 2023-10-30T13:31:58.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f7ebc001a00 con 0x7f7ec40cf160 2023-10-30T13:31:58.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec8e3a640 1 --2- 172.21.15.148:0/1877539510 >> v2:172.21.15.150:3300/0 conn(0x7f7ec41538a0 0x7f7ec415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:31:58.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 >> v1:172.21.15.148:6789/0 conn(0x7f7ec40cf160 legacy=0x7f7ec4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:58.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb4003980 0x7f7eb4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:58.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40047d0 0x7f7eb4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:58.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7eb400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:58.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec40cb330 con 0x7f7eb4005190 2023-10-30T13:31:58.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb400c040 con 0x7f7eb4003980 2023-10-30T13:31:58.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb400c200 con 0x7f7eb40047d0 2023-10-30T13:31:58.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec963b640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7eb400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:58.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec8e3a640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb4003980 0x7f7eb4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:58.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec963b640 1 -- 172.21.15.148:0/1877539510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40047d0 msgr2=0x7f7eb4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:58.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec963b640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7eb40047d0 0x7f7eb4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:58.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec963b640 1 -- 172.21.15.148:0/1877539510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb4003980 msgr2=0x7f7eb4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:31:58.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec963b640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb4003980 0x7f7eb4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:58.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec963b640 1 -- 172.21.15.148:0/1877539510 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4003530 con 0x7f7eb4005190 2023-10-30T13:31:58.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.902+0000 7f7ec8e3a640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb4003980 0x7f7eb4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:31:58.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ec963b640 1 -- 172.21.15.148:0/1877539510 reap_dead start 2023-10-30T13:31:58.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ec963b640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7eb400b990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00a820 tx=0x7f7ebc01d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2c4cf6b033c4ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:58.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0072d0 con 0x7f7eb4005190 2023-10-30T13:31:58.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc0a8070 con 0x7f7eb4005190 2023-10-30T13:31:58.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ec37fe640 1 -- 172.21.15.148:0/1877539510 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0076b0 con 0x7f7eb4005190 2023-10-30T13:31:58.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ecb09e640 1 -- 172.21.15.148:0/1877539510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 msgr2=0x7f7eb400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:58.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ecb09e640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7eb400b990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00a820 tx=0x7f7ebc01d320 comp rx=0 tx=0).stop 2023-10-30T13:31:58.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ecb09e640 1 -- 172.21.15.148:0/1877539510 shutdown_connections 2023-10-30T13:31:58.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ecb09e640 1 --2- 172.21.15.148:0/1877539510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7eb400b990 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:58.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ecb09e640 1 -- 172.21.15.148:0/1877539510 >> 172.21.15.148:0/1877539510 conn(0x7f7ec406bfe0 msgr2=0x7f7ec406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:58.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ecb09e640 1 -- 172.21.15.148:0/1877539510 shutdown_connections 2023-10-30T13:31:58.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ecb09e640 1 -- 172.21.15.148:0/1877539510 wait complete. 2023-10-30T13:31:58.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ecb09e640 1 Processor -- start 2023-10-30T13:31:58.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.906+0000 7f7ecb09e640 1 -- start start 2023-10-30T13:31:58.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ecb09e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7ec415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:58.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ecb09e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ec41495a0 0x7f7ec415bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:58.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ecb09e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec41efaf0 0x7f7ec4152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:58.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ecb09e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec40c71c0 con 0x7f7eb4005190 2023-10-30T13:31:58.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ecb09e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec41ef660 con 0x7f7ec41efaf0 2023-10-30T13:31:58.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ecb09e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec4157ac0 con 0x7f7ec41495a0 2023-10-30T13:31:58.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec8e3a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7ec415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:58.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec8e3a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7ec415cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36596/0 (socket says 172.21.15.148:36596) 2023-10-30T13:31:58.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec8e3a640 1 -- 172.21.15.148:0/259069944 learned_addr learned my addr 172.21.15.148:0/259069944 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:31:58.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec963b640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec41efaf0 0x7f7ec4152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:58.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec3fff640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ec41495a0 0x7f7ec415bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:58.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec8e3a640 1 -- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ec41495a0 msgr2=0x7f7ec415bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:58.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec8e3a640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ec41495a0 0x7f7ec415bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:58.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec8e3a640 1 -- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec41efaf0 msgr2=0x7f7ec4152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:58.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec8e3a640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec41efaf0 0x7f7ec4152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:58.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec8e3a640 1 -- 172.21.15.148:0/259069944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec41ef050 con 0x7f7eb4005190 2023-10-30T13:31:58.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec963b640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec41efaf0 0x7f7ec4152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:31:58.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec3fff640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ec41495a0 0x7f7ec415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:31:58.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec8e3a640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7ec415cb30 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8004660 tx=0x7f7eb8011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:58.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec17fa640 1 -- 172.21.15.148:0/259069944 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb800fbd0 con 0x7f7eb4005190 2023-10-30T13:31:58.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec17fa640 1 -- 172.21.15.148:0/259069944 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb800fd30 con 0x7f7eb4005190 2023-10-30T13:31:58.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec17fa640 1 -- 172.21.15.148:0/259069944 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8015660 con 0x7f7eb4005190 2023-10-30T13:31:58.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec4157500 con 0x7f7eb4005190 2023-10-30T13:31:58.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec41ef2f0 con 0x7f7eb4005190 2023-10-30T13:31:58.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec41eeeb0 con 0x7f7eb4005190 2023-10-30T13:31:58.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec17fa640 1 -- 172.21.15.148:0/259069944 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7eb80157c0 con 0x7f7eb4005190 2023-10-30T13:31:58.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.910+0000 7f7ec17fa640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ea80664a0 0x7f7ea8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:31:58.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.914+0000 7f7ec17fa640 1 -- 172.21.15.148:0/259069944 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f7eb8084bb0 con 0x7f7eb4005190 2023-10-30T13:31:58.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.914+0000 7f7ec3fff640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ea80664a0 0x7f7ea8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:31:58.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.914+0000 7f7ec3fff640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ea80664a0 0x7f7ea8068960 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40b36a0 tx=0x7f7eb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:31:58.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:58.918+0000 7f7ec17fa640 1 -- 172.21.15.148:0/259069944 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ec41eeeb0 con 0x7f7eb4005190 2023-10-30T13:31:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.094+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ec41eeeb0 con 0x7f7eb4005190 2023-10-30T13:31:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ec17fa640 1 -- 172.21.15.148:0/259069944 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6508 (secure 0 0 0) 0x7f7ec41eeeb0 con 0x7f7eb4005190 2023-10-30T13:31:59.095 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:31:59.095 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6546,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:30:10.789637+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6408},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6408":{"gid":6408,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":23,"addr":"172.21.15.148:6835/172087710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":172087710},{"type":"v1","addr":"172.21.15.148:6835","nonce":172087710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:31:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 92 2023-10-30T13:31:59.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ea80664a0 msgr2=0x7f7ea8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:59.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ea80664a0 0x7f7ea8068960 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40b36a0 tx=0x7f7eb0009040 comp rx=0 tx=0).stop 2023-10-30T13:31:59.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 msgr2=0x7f7ec415cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:31:59.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7ec415cb30 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8004660 tx=0x7f7eb8011d60 comp rx=0 tx=0).stop 2023-10-30T13:31:59.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 shutdown_connections 2023-10-30T13:31:59.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ea80664a0 0x7f7ea8068960 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:59.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ec41efaf0 0x7f7ec4152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:59.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7ec41495a0 0x7f7ec415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:59.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 --2- 172.21.15.148:0/259069944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7eb4005190 0x7f7ec415cb30 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:31:59.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 >> 172.21.15.148:0/259069944 conn(0x7f7ec406bfe0 msgr2=0x7f7ec40b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:31:59.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 shutdown_connections 2023-10-30T13:31:59.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:31:59.098+0000 7f7ecb09e640 1 -- 172.21.15.148:0/259069944 wait complete. 2023-10-30T13:31:59.116 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:31:59.345 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:31:59.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:32:01.117 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:01.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9f372640 1 Processor -- start 2023-10-30T13:32:01.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9f372640 1 -- start start 2023-10-30T13:32:01.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9f372640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0a98070740 0x7f0a980c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9f372640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0a9815e700 0x7f0a9806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9f372640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a980c5810 con 0x7f0a981580e0 2023-10-30T13:32:01.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9f372640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9815eb70 con 0x7f0a9815e700 2023-10-30T13:32:01.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9f372640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980c6bb0 con 0x7f0a98070740 2023-10-30T13:32:01.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9d10e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0a981580e0 0x7f0a98069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43598/0 (socket says 172.21.15.148:43598) 2023-10-30T13:32:01.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9d90f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0a9815e700 0x7f0a9806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.266+0000 7f0a9d10e640 1 -- 172.21.15.148:0/415006636 learned_addr learned my addr 172.21.15.148:0/415006636 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:01.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a9c90d640 1 --2- 172.21.15.148:0/415006636 >> v2:172.21.15.150:3300/0 conn(0x7f0a98070740 0x7f0a980c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a8ffff640 1 -- 172.21.15.148:0/415006636 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829392122 0 0) 0x7f0a980c5810 con 0x7f0a981580e0 2023-10-30T13:32:01.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a8ffff640 1 -- 172.21.15.148:0/415006636 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a84003530 con 0x7f0a981580e0 2023-10-30T13:32:01.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a9d90f640 1 -- 172.21.15.148:0/415006636 >> v2:172.21.15.150:3300/0 conn(0x7f0a98070740 msgr2=0x7f0a980c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:01.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a9d90f640 1 --2- 172.21.15.148:0/415006636 >> v2:172.21.15.150:3300/0 conn(0x7f0a98070740 0x7f0a980c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a9d90f640 1 -- 172.21.15.148:0/415006636 >> v1:172.21.15.148:6789/0 conn(0x7f0a981580e0 legacy=0x7f0a98069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a9d90f640 1 -- 172.21.15.148:0/415006636 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a980c6e70 con 0x7f0a9815e700 2023-10-30T13:32:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a9d90f640 1 --2- 172.21.15.148:0/415006636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 0x7f0a9806a5a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0a9806b1a0 tx=0x7f0a9001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93e7a6c84b7b411 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a8ffff640 1 -- 172.21.15.148:0/415006636 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900a58c0 con 0x7f0a9815e700 2023-10-30T13:32:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a8ffff640 1 -- 172.21.15.148:0/415006636 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a900a5a20 con 0x7f0a9815e700 2023-10-30T13:32:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a8ffff640 1 -- 172.21.15.148:0/415006636 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900a5db0 con 0x7f0a9815e700 2023-10-30T13:32:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a9f372640 1 -- 172.21.15.148:0/415006636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 msgr2=0x7f0a9806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.270+0000 7f0a9f372640 1 --2- 172.21.15.148:0/415006636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 0x7f0a9806a5a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0a9806b1a0 tx=0x7f0a9001c910 comp rx=0 tx=0).stop 2023-10-30T13:32:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 -- 172.21.15.148:0/415006636 shutdown_connections 2023-10-30T13:32:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 --2- 172.21.15.148:0/415006636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 0x7f0a9806a5a0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 --2- 172.21.15.148:0/415006636 >> v2:172.21.15.150:3300/0 conn(0x7f0a98070740 0x7f0a980c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 -- 172.21.15.148:0/415006636 >> 172.21.15.148:0/415006636 conn(0x7f0a980c7910 msgr2=0x7f0a980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 -- 172.21.15.148:0/415006636 shutdown_connections 2023-10-30T13:32:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 -- 172.21.15.148:0/415006636 wait complete. 2023-10-30T13:32:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 Processor -- start 2023-10-30T13:32:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 -- start start 2023-10-30T13:32:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a98070740 0x7f0a9806b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a981580e0 0x7f0a9806b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 0x7f0a981499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a981487e0 con 0x7f0a9815e700 2023-10-30T13:32:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a981f6840 con 0x7f0a981580e0 2023-10-30T13:32:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9f372640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98144a20 con 0x7f0a98070740 2023-10-30T13:32:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9d90f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 0x7f0a981499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9d90f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 0x7f0a981499c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36616/0 (socket says 172.21.15.148:36616) 2023-10-30T13:32:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9d90f640 1 -- 172.21.15.148:0/2426292027 learned_addr learned my addr 172.21.15.148:0/2426292027 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9c90d640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a981580e0 0x7f0a9806b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9d10e640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a98070740 0x7f0a9806b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9d90f640 1 -- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a98070740 msgr2=0x7f0a9806b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9d90f640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a98070740 0x7f0a9806b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9d90f640 1 -- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a981580e0 msgr2=0x7f0a9806b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9d90f640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a981580e0 0x7f0a9806b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.274+0000 7f0a9d90f640 1 -- 172.21.15.148:0/2426292027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a90009a70 con 0x7f0a9815e700 2023-10-30T13:32:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.278+0000 7f0a9d90f640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 0x7f0a981499c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0a90019520 tx=0x7f0a900196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.278+0000 7f0a8dffb640 1 -- 172.21.15.148:0/2426292027 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900b9860 con 0x7f0a9815e700 2023-10-30T13:32:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.278+0000 7f0a8dffb640 1 -- 172.21.15.148:0/2426292027 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a900b99c0 con 0x7f0a9815e700 2023-10-30T13:32:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.278+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a981f9f30 con 0x7f0a9815e700 2023-10-30T13:32:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.278+0000 7f0a8dffb640 1 -- 172.21.15.148:0/2426292027 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900b9d70 con 0x7f0a9815e700 2023-10-30T13:32:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.278+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a981f61f0 con 0x7f0a9815e700 2023-10-30T13:32:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.278+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a981f69b0 con 0x7f0a9815e700 2023-10-30T13:32:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.278+0000 7f0a8dffb640 1 -- 172.21.15.148:0/2426292027 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0a900c4460 con 0x7f0a9815e700 2023-10-30T13:32:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.278+0000 7f0a8dffb640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a7c0664a0 0x7f0a7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.282+0000 7f0a8dffb640 1 -- 172.21.15.148:0/2426292027 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f0a9011f4f0 con 0x7f0a9815e700 2023-10-30T13:32:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.282+0000 7f0a9d10e640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a7c0664a0 0x7f0a7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.282+0000 7f0a9d10e640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a7c0664a0 0x7f0a7c068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0a940011f0 tx=0x7f0a94007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.282+0000 7f0a8dffb640 1 -- 172.21.15.148:0/2426292027 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a900ec560 con 0x7f0a9815e700 2023-10-30T13:32:01.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a9804e170 con 0x7f0a9815e700 2023-10-30T13:32:01.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a8dffb640 1 -- 172.21.15.148:0/2426292027 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f0a900efb80 con 0x7f0a9815e700 2023-10-30T13:32:01.464 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:01.464 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:31:59.711354+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:01.464 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 93 2023-10-30T13:32:01.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a7c0664a0 msgr2=0x7f0a7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a7c0664a0 0x7f0a7c068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0a940011f0 tx=0x7f0a94007d40 comp rx=0 tx=0).stop 2023-10-30T13:32:01.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 msgr2=0x7f0a981499c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 0x7f0a981499c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0a90019520 tx=0x7f0a900196b0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 shutdown_connections 2023-10-30T13:32:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a7c0664a0 0x7f0a7c068960 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a9815e700 0x7f0a981499c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a981580e0 0x7f0a9806b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 --2- 172.21.15.148:0/2426292027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0a98070740 0x7f0a9806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 >> 172.21.15.148:0/2426292027 conn(0x7f0a980c7910 msgr2=0x7f0a98147090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:01.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 shutdown_connections 2023-10-30T13:32:01.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.466+0000 7f0a9f372640 1 -- 172.21.15.148:0/2426292027 wait complete. 2023-10-30T13:32:01.485 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:32:01.486 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:32:01.486 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:01.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f15a113e640 1 Processor -- start 2023-10-30T13:32:01.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f15a113e640 1 -- start start 2023-10-30T13:32:01.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f15a113e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f159c06f7a0 0x7f159c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f15a113e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f159c0c6bd0 0x7f159c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f15a113e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 0x7f159c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f15a113e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159c0700c0 con 0x7f159c06f7a0 2023-10-30T13:32:01.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f15a113e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159c06a3d0 con 0x7f159c0c6bd0 2023-10-30T13:32:01.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f15a113e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c066cf0 con 0x7f159c071110 2023-10-30T13:32:01.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159ad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f159c06f7a0 0x7f159c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159ad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f159c06f7a0 0x7f159c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36630/0 (socket says 172.21.15.148:36630) 2023-10-30T13:32:01.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159ad9d640 1 -- 172.21.15.148:0/2147499110 learned_addr learned my addr 172.21.15.148:0/2147499110 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:01.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159a59c640 1 --2- 172.21.15.148:0/2147499110 >> v2:172.21.15.150:3300/0 conn(0x7f159c0c6bd0 0x7f159c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159b59e640 1 --2- 172.21.15.148:0/2147499110 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 0x7f159c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159ad9d640 1 -- 172.21.15.148:0/2147499110 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 msgr2=0x7f159c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:01.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159ad9d640 1 --2- 172.21.15.148:0/2147499110 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 0x7f159c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159ad9d640 1 -- 172.21.15.148:0/2147499110 >> v2:172.21.15.150:3300/0 conn(0x7f159c0c6bd0 msgr2=0x7f159c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:01.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159ad9d640 1 --2- 172.21.15.148:0/2147499110 >> v2:172.21.15.150:3300/0 conn(0x7f159c0c6bd0 0x7f159c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.626+0000 7f159ad9d640 1 -- 172.21.15.148:0/2147499110 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c0c70d0 con 0x7f159c06f7a0 2023-10-30T13:32:01.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f159ad9d640 1 --2- 172.21.15.148:0/2147499110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c06f7a0 0x7f159c06fb80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f159c06f1c0 tx=0x7f158c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439449756a7566bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:01.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f1599d9b640 1 -- 172.21.15.148:0/2147499110 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c0af400 con 0x7f159c06f7a0 2023-10-30T13:32:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f1599d9b640 1 -- 172.21.15.148:0/2147499110 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158c0ab050 con 0x7f159c06f7a0 2023-10-30T13:32:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f1599d9b640 1 -- 172.21.15.148:0/2147499110 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c0af780 con 0x7f159c06f7a0 2023-10-30T13:32:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f15a113e640 1 -- 172.21.15.148:0/2147499110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c06f7a0 msgr2=0x7f159c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f15a113e640 1 --2- 172.21.15.148:0/2147499110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c06f7a0 0x7f159c06fb80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f159c06f1c0 tx=0x7f158c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f15a113e640 1 -- 172.21.15.148:0/2147499110 shutdown_connections 2023-10-30T13:32:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f15a113e640 1 --2- 172.21.15.148:0/2147499110 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 0x7f159c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f15a113e640 1 --2- 172.21.15.148:0/2147499110 >> v2:172.21.15.150:3300/0 conn(0x7f159c0c6bd0 0x7f159c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f15a113e640 1 --2- 172.21.15.148:0/2147499110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c06f7a0 0x7f159c06fb80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f15a113e640 1 -- 172.21.15.148:0/2147499110 >> 172.21.15.148:0/2147499110 conn(0x7f159c0c7910 msgr2=0x7f159c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f15a113e640 1 -- 172.21.15.148:0/2147499110 shutdown_connections 2023-10-30T13:32:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.630+0000 7f15a113e640 1 -- 172.21.15.148:0/2147499110 wait complete. 2023-10-30T13:32:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 Processor -- start 2023-10-30T13:32:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 -- start start 2023-10-30T13:32:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f159c06f7a0 0x7f159c14e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 0x7f159c14f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c0c6bd0 0x7f159c14c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c156640 con 0x7f159c0c6bd0 2023-10-30T13:32:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c14f290 con 0x7f159c06f7a0 2023-10-30T13:32:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c1563c0 con 0x7f159c071110 2023-10-30T13:32:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159b59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c0c6bd0 0x7f159c14c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159b59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c0c6bd0 0x7f159c14c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36646/0 (socket says 172.21.15.148:36646) 2023-10-30T13:32:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159b59e640 1 -- 172.21.15.148:0/1269199926 learned_addr learned my addr 172.21.15.148:0/1269199926 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159ad9d640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f159c06f7a0 0x7f159c14e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159a59c640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 0x7f159c14f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159b59e640 1 -- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 msgr2=0x7f159c14f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159b59e640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 0x7f159c14f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159b59e640 1 -- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f159c06f7a0 msgr2=0x7f159c14e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159b59e640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f159c06f7a0 0x7f159c14e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159b59e640 1 -- 172.21.15.148:0/1269199926 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c009b30 con 0x7f159c0c6bd0 2023-10-30T13:32:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f159b59e640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c0c6bd0 0x7f159c14c820 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f159400d600 tx=0x7f159400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15837fe640 1 -- 172.21.15.148:0/1269199926 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594014620 con 0x7f159c0c6bd0 2023-10-30T13:32:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15837fe640 1 -- 172.21.15.148:0/1269199926 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15940147a0 con 0x7f159c0c6bd0 2023-10-30T13:32:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159c1ef8f0 con 0x7f159c0c6bd0 2023-10-30T13:32:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15837fe640 1 -- 172.21.15.148:0/1269199926 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594014b50 con 0x7f159c0c6bd0 2023-10-30T13:32:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.634+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159c14ef20 con 0x7f159c0c6bd0 2023-10-30T13:32:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.638+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159c04e170 con 0x7f159c0c6bd0 2023-10-30T13:32:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.638+0000 7f15837fe640 1 -- 172.21.15.148:0/1269199926 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1594014cb0 con 0x7f159c0c6bd0 2023-10-30T13:32:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.638+0000 7f15837fe640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15880664a0 0x7f1588068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.638+0000 7f15837fe640 1 -- 172.21.15.148:0/1269199926 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f1594084920 con 0x7f159c0c6bd0 2023-10-30T13:32:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.638+0000 7f159ad9d640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15880664a0 0x7f1588068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.638+0000 7f159ad9d640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15880664a0 0x7f1588068960 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f158c0b1000 tx=0x7f158c0aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.642+0000 7f15837fe640 1 -- 172.21.15.148:0/1269199926 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1594051900 con 0x7f159c0c6bd0 2023-10-30T13:32:01.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.826+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f159c14f3d0 con 0x7f159c0c6bd0 2023-10-30T13:32:01.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.826+0000 7f15837fe640 1 -- 172.21.15.148:0/1269199926 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f1594018090 con 0x7f159c0c6bd0 2023-10-30T13:32:01.826 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:01.826 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:31:59.711354+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:01.827 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 93 2023-10-30T13:32:01.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15880664a0 msgr2=0x7f1588068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15880664a0 0x7f1588068960 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f158c0b1000 tx=0x7f158c0aa000 comp rx=0 tx=0).stop 2023-10-30T13:32:01.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c0c6bd0 msgr2=0x7f159c14c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:01.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c0c6bd0 0x7f159c14c820 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f159400d600 tx=0x7f159400dac0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 shutdown_connections 2023-10-30T13:32:01.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15880664a0 0x7f1588068960 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f159c0c6bd0 0x7f159c14c820 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f159c071110 0x7f159c14f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 --2- 172.21.15.148:0/1269199926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f159c06f7a0 0x7f159c14e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:01.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 >> 172.21.15.148:0/1269199926 conn(0x7f159c0c7910 msgr2=0x7f159c156cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:01.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 shutdown_connections 2023-10-30T13:32:01.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:01.830+0000 7f15a113e640 1 -- 172.21.15.148:0/1269199926 wait complete. 2023-10-30T13:32:01.847 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6546": { "addr": "172.21.15.148:6837/3232618434", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3232618434, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3232618434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6546, "incarnation": 93, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:31:59.711354+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6546 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:32:03.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:03.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:03.998+0000 7fb00a994640 1 Processor -- start 2023-10-30T13:32:03.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:03.998+0000 7fb00a994640 1 -- start start 2023-10-30T13:32:04.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb00a994640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb004159e60 0x7fb00406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:04.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb00a994640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb0040bfec0 0x7fb0040c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:04.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb00a994640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00406fa80 con 0x7fb00406a720 2023-10-30T13:32:04.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb00a994640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb004071290 con 0x7fb0040bfec0 2023-10-30T13:32:04.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb00a994640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb004071570 con 0x7fb004159e60 2023-10-30T13:32:04.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb0037fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb0040bfec0 0x7fb0040c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:04.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb008f31640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb00406a720 0x7fb004155d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48570/0 (socket says 172.21.15.148:48570) 2023-10-30T13:32:04.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb008f31640 1 -- 172.21.15.148:0/986261042 learned_addr learned my addr 172.21.15.148:0/986261042 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:04.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb003fff640 1 --2- 172.21.15.148:0/986261042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb004159e60 0x7fb00406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:04.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb0037fe640 1 -- 172.21.15.148:0/986261042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb004159e60 msgr2=0x7fb00406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:04.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb0037fe640 1 --2- 172.21.15.148:0/986261042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb004159e60 0x7fb00406f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb0037fe640 1 -- 172.21.15.148:0/986261042 >> v1:172.21.15.148:6789/0 conn(0x7fb00406a720 legacy=0x7fb004155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:04.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb0037fe640 1 -- 172.21.15.148:0/986261042 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb004071850 con 0x7fb0040bfec0 2023-10-30T13:32:04.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.002+0000 7fb002ffd640 1 -- 172.21.15.148:0/986261042 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430784344 0 0) 0x7fb00406fa80 con 0x7fb00406a720 2023-10-30T13:32:04.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb0037fe640 1 --2- 172.21.15.148:0/986261042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0040bfec0 0x7fb0040c1a40 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7faff80097c0 tx=0x7faff801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247afdcdfb6e7044 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:04.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb002ffd640 1 -- 172.21.15.148:0/986261042 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff80a8970 con 0x7fb0040bfec0 2023-10-30T13:32:04.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb002ffd640 1 -- 172.21.15.148:0/986261042 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff80a8ad0 con 0x7fb0040bfec0 2023-10-30T13:32:04.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb002ffd640 1 -- 172.21.15.148:0/986261042 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff80a8e60 con 0x7fb0040bfec0 2023-10-30T13:32:04.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 -- 172.21.15.148:0/986261042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0040bfec0 msgr2=0x7fb0040c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:04.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 --2- 172.21.15.148:0/986261042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0040bfec0 0x7fb0040c1a40 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7faff80097c0 tx=0x7faff801ec70 comp rx=0 tx=0).stop 2023-10-30T13:32:04.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 -- 172.21.15.148:0/986261042 shutdown_connections 2023-10-30T13:32:04.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 --2- 172.21.15.148:0/986261042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0040bfec0 0x7fb0040c1a40 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 --2- 172.21.15.148:0/986261042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb004159e60 0x7fb00406f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 -- 172.21.15.148:0/986261042 >> 172.21.15.148:0/986261042 conn(0x7fb0040c7910 msgr2=0x7fb0040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:04.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 -- 172.21.15.148:0/986261042 shutdown_connections 2023-10-30T13:32:04.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 -- 172.21.15.148:0/986261042 wait complete. 2023-10-30T13:32:04.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 Processor -- start 2023-10-30T13:32:04.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.006+0000 7fb00a994640 1 -- start start 2023-10-30T13:32:04.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb00a994640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb00406a720 0x7fb0041f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:04.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb00a994640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0040bfec0 0x7fb0041f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:04.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb00a994640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb004159e60 0x7fb00414c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:04.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb00a994640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0041f3060 con 0x7fb004159e60 2023-10-30T13:32:04.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb00a994640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb004158700 con 0x7fb0040bfec0 2023-10-30T13:32:04.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb00a994640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0041589e0 con 0x7fb00406a720 2023-10-30T13:32:04.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb008f31640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb004159e60 0x7fb00414c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:04.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb008f31640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb004159e60 0x7fb00414c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53418/0 (socket says 172.21.15.148:53418) 2023-10-30T13:32:04.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb008f31640 1 -- 172.21.15.148:0/3502100423 learned_addr learned my addr 172.21.15.148:0/3502100423 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:04.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb003fff640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb00406a720 0x7fb0041f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:04.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb0037fe640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0040bfec0 0x7fb0041f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:04.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb008f31640 1 -- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb00406a720 msgr2=0x7fb0041f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:04.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb008f31640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb00406a720 0x7fb0041f2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb008f31640 1 -- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0040bfec0 msgr2=0x7fb0041f2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:04.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb008f31640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0040bfec0 0x7fb0041f2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb008f31640 1 -- 172.21.15.148:0/3502100423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff8009420 con 0x7fb004159e60 2023-10-30T13:32:04.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb008f31640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb004159e60 0x7fb00414c660 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7faffc001570 tx=0x7faffc00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:04.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb000ff9640 1 -- 172.21.15.148:0/3502100423 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc00a640 con 0x7fb004159e60 2023-10-30T13:32:04.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb000ff9640 1 -- 172.21.15.148:0/3502100423 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faffc00a7c0 con 0x7fb004159e60 2023-10-30T13:32:04.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb000ff9640 1 -- 172.21.15.148:0/3502100423 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc00ab70 con 0x7fb004159e60 2023-10-30T13:32:04.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb004158cc0 con 0x7fb004159e60 2023-10-30T13:32:04.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb004155720 con 0x7fb004159e60 2023-10-30T13:32:04.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00404e170 con 0x7fb004159e60 2023-10-30T13:32:04.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.010+0000 7fb000ff9640 1 -- 172.21.15.148:0/3502100423 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faffc00acd0 con 0x7fb004159e60 2023-10-30T13:32:04.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.014+0000 7fb000ff9640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7faff00664a0 0x7faff0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:04.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.014+0000 7fb000ff9640 1 -- 172.21.15.148:0/3502100423 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7faffc0839a0 con 0x7fb004159e60 2023-10-30T13:32:04.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.014+0000 7fb003fff640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7faff00664a0 0x7faff0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:04.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.014+0000 7fb003fff640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7faff00664a0 0x7faff0068960 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7faff400b930 tx=0x7faff4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:04.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.018+0000 7fb000ff9640 1 -- 172.21.15.148:0/3502100423 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faffc0508d0 con 0x7fb004159e60 2023-10-30T13:32:04.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0041f31a0 con 0x7fb004159e60 2023-10-30T13:32:04.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb000ff9640 1 -- 172.21.15.148:0/3502100423 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5716 (secure 0 0 0) 0x7faffc018070 con 0x7fb004159e60 2023-10-30T13:32:04.208 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:04.209 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:31:59.711354+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:04.209 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 93 2023-10-30T13:32:04.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7faff00664a0 msgr2=0x7faff0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:04.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7faff00664a0 0x7faff0068960 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7faff400b930 tx=0x7faff4008040 comp rx=0 tx=0).stop 2023-10-30T13:32:04.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb004159e60 msgr2=0x7fb00414c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:04.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb004159e60 0x7fb00414c660 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7faffc001570 tx=0x7faffc00cca0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 shutdown_connections 2023-10-30T13:32:04.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7faff00664a0 0x7faff0068960 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb004159e60 0x7fb00414c660 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0040bfec0 0x7fb0041f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 --2- 172.21.15.148:0/3502100423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb00406a720 0x7fb0041f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.210+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 >> 172.21.15.148:0/3502100423 conn(0x7fb0040c7910 msgr2=0x7fb00414fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.214+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 shutdown_connections 2023-10-30T13:32:04.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:04.214+0000 7fb00a994640 1 -- 172.21.15.148:0/3502100423 wait complete. 2023-10-30T13:32:04.230 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:06.231 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:06.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.378+0000 7fe2aff65640 1 Processor -- start 2023-10-30T13:32:06.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.378+0000 7fe2aff65640 1 -- start start 2023-10-30T13:32:06.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.378+0000 7fe2aff65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe2a815d6f0 0x7fe2a815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:06.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.378+0000 7fe2aff65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a80cb800 con 0x7fe2a80cf160 2023-10-30T13:32:06.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.378+0000 7fe2aff65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a80cb330 con 0x7fe2a81538a0 2023-10-30T13:32:06.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.378+0000 7fe2aff65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2a80cad70 con 0x7fe2a815d6f0 2023-10-30T13:32:06.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.378+0000 7fe2ae502640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe2a80cf160 0x7fe2a8164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48576/0 (socket says 172.21.15.148:48576) 2023-10-30T13:32:06.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.378+0000 7fe2ae502640 1 -- 172.21.15.148:0/1749912867 learned_addr learned my addr 172.21.15.148:0/1749912867 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:06.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.378+0000 7fe2ad500640 1 --2- 172.21.15.148:0/1749912867 >> v2:172.21.15.150:3300/0 conn(0x7fe2a815d6f0 0x7fe2a815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:06.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.382+0000 7fe2accff640 1 -- 172.21.15.148:0/1749912867 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245037809 0 0) 0x7fe2a80cb800 con 0x7fe2a80cf160 2023-10-30T13:32:06.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.382+0000 7fe2accff640 1 -- 172.21.15.148:0/1749912867 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe294003530 con 0x7fe2a80cf160 2023-10-30T13:32:06.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.382+0000 7fe2accff640 1 -- 172.21.15.148:0/1749912867 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2239152345 0 0) 0x7fe294003530 con 0x7fe2a80cf160 2023-10-30T13:32:06.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.382+0000 7fe2ad500640 1 -- 172.21.15.148:0/1749912867 >> v1:172.21.15.150:6789/0 conn(0x7fe2a81538a0 legacy=0x7fe2a80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:06.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.382+0000 7fe2ad500640 1 -- 172.21.15.148:0/1749912867 >> v1:172.21.15.148:6789/0 conn(0x7fe2a80cf160 legacy=0x7fe2a8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:06.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.382+0000 7fe2ad500640 1 -- 172.21.15.148:0/1749912867 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a80cb050 con 0x7fe2a815d6f0 2023-10-30T13:32:06.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2ad500640 1 --2- 172.21.15.148:0/1749912867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a815d6f0 0x7fe2a815a090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe2980097c0 tx=0x7fe29801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79aafebbabc99b64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:06.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2accff640 1 -- 172.21.15.148:0/1749912867 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0001a00 con 0x7fe2a815d6f0 2023-10-30T13:32:06.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2accff640 1 -- 172.21.15.148:0/1749912867 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2980a8610 con 0x7fe2a815d6f0 2023-10-30T13:32:06.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2accff640 1 -- 172.21.15.148:0/1749912867 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2980a89a0 con 0x7fe2a815d6f0 2023-10-30T13:32:06.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2aff65640 1 -- 172.21.15.148:0/1749912867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a815d6f0 msgr2=0x7fe2a815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:06.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2aff65640 1 --2- 172.21.15.148:0/1749912867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a815d6f0 0x7fe2a815a090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe2980097c0 tx=0x7fe29801eb90 comp rx=0 tx=0).stop 2023-10-30T13:32:06.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2aff65640 1 -- 172.21.15.148:0/1749912867 shutdown_connections 2023-10-30T13:32:06.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2aff65640 1 --2- 172.21.15.148:0/1749912867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a815d6f0 0x7fe2a815a090 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:06.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2aff65640 1 -- 172.21.15.148:0/1749912867 >> 172.21.15.148:0/1749912867 conn(0x7fe2a806bfe0 msgr2=0x7fe2a806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:06.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2aff65640 1 -- 172.21.15.148:0/1749912867 shutdown_connections 2023-10-30T13:32:06.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.386+0000 7fe2aff65640 1 -- 172.21.15.148:0/1749912867 wait complete. 2023-10-30T13:32:06.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2aff65640 1 Processor -- start 2023-10-30T13:32:06.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2aff65640 1 -- start start 2023-10-30T13:32:06.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2aff65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2a80cf160 0x7fe2a8157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:06.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2aff65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a81538a0 0x7fe2a81e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:06.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2aff65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2a815d6f0 0x7fe2a814e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:06.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2aff65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a81e38c0 con 0x7fe2a80cf160 2023-10-30T13:32:06.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2aff65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a81e54c0 con 0x7fe2a81538a0 2023-10-30T13:32:06.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2aff65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a8159600 con 0x7fe2a815d6f0 2023-10-30T13:32:06.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2add01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2a80cf160 0x7fe2a8157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:06.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2add01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2a80cf160 0x7fe2a8157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53432/0 (socket says 172.21.15.148:53432) 2023-10-30T13:32:06.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2add01640 1 -- 172.21.15.148:0/893253044 learned_addr learned my addr 172.21.15.148:0/893253044 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:06.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2ad500640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a81538a0 0x7fe2a81e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:06.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2ae502640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2a815d6f0 0x7fe2a814e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:06.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2add01640 1 -- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2a815d6f0 msgr2=0x7fe2a814e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:06.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2add01640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2a815d6f0 0x7fe2a814e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:06.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2add01640 1 -- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a81538a0 msgr2=0x7fe2a81e2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:06.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2add01640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a81538a0 0x7fe2a81e2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:06.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.390+0000 7fe2add01640 1 -- 172.21.15.148:0/893253044 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe298009420 con 0x7fe2a80cf160 2023-10-30T13:32:06.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.394+0000 7fe2add01640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2a80cf160 0x7fe2a8157560 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c500 tx=0x7fe2a400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:06.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.394+0000 7fe29e7fc640 1 -- 172.21.15.148:0/893253044 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4005220 con 0x7fe2a80cf160 2023-10-30T13:32:06.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.394+0000 7fe29e7fc640 1 -- 172.21.15.148:0/893253044 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a4005380 con 0x7fe2a80cf160 2023-10-30T13:32:06.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.394+0000 7fe29e7fc640 1 -- 172.21.15.148:0/893253044 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4007270 con 0x7fe2a80cf160 2023-10-30T13:32:06.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.394+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a81e5710 con 0x7fe2a80cf160 2023-10-30T13:32:06.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.394+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a8157aa0 con 0x7fe2a80cf160 2023-10-30T13:32:06.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.394+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a81e3a00 con 0x7fe2a80cf160 2023-10-30T13:32:06.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.394+0000 7fe29e7fc640 1 -- 172.21.15.148:0/893253044 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe2a40073d0 con 0x7fe2a80cf160 2023-10-30T13:32:06.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.398+0000 7fe29e7fc640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe28c0664a0 0x7fe28c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:06.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.398+0000 7fe29e7fc640 1 -- 172.21.15.148:0/893253044 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fe2a4083720 con 0x7fe2a80cf160 2023-10-30T13:32:06.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.398+0000 7fe2ad500640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe28c0664a0 0x7fe28c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:06.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.398+0000 7fe2ad500640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe28c0664a0 0x7fe28c068960 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe2980b2000 tx=0x7fe298008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:06.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.402+0000 7fe29e7fc640 1 -- 172.21.15.148:0/893253044 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe2a81e3a00 con 0x7fe2a80cf160 2023-10-30T13:32:06.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2a81e3a00 con 0x7fe2a80cf160 2023-10-30T13:32:06.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe29e7fc640 1 -- 172.21.15.148:0/893253044 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5716 (secure 0 0 0) 0x7fe2a40506f0 con 0x7fe2a80cf160 2023-10-30T13:32:06.596 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:06.596 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:31:59.711354+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:06.597 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 93 2023-10-30T13:32:06.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe28c0664a0 msgr2=0x7fe28c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:06.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe28c0664a0 0x7fe28c068960 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe2980b2000 tx=0x7fe298008040 comp rx=0 tx=0).stop 2023-10-30T13:32:06.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2a80cf160 msgr2=0x7fe2a8157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:06.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2a80cf160 0x7fe2a8157560 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c500 tx=0x7fe2a400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:32:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 shutdown_connections 2023-10-30T13:32:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe28c0664a0 0x7fe28c068960 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe2a815d6f0 0x7fe2a814e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a81538a0 0x7fe2a81e2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 --2- 172.21.15.148:0/893253044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe2a80cf160 0x7fe2a8157560 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:06.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 >> 172.21.15.148:0/893253044 conn(0x7fe2a806bfe0 msgr2=0x7fe2a815b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:06.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 shutdown_connections 2023-10-30T13:32:06.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:06.598+0000 7fe2aff65640 1 -- 172.21.15.148:0/893253044 wait complete. 2023-10-30T13:32:06.617 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:08.618 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:08.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.770+0000 7f3560a16640 1 Processor -- start 2023-10-30T13:32:08.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.770+0000 7f3560a16640 1 -- start start 2023-10-30T13:32:08.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.770+0000 7f3560a16640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f355c1538f0 0x7f355c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:08.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.770+0000 7f3560a16640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0ca6a0 con 0x7f355c15cd50 2023-10-30T13:32:08.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.770+0000 7f3560a16640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c15d300 con 0x7f355c0cefa0 2023-10-30T13:32:08.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.770+0000 7f3560a16640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c15d5e0 con 0x7f355c1538f0 2023-10-30T13:32:08.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.770+0000 7f355ad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f355c1538f0 0x7f355c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:08.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.774+0000 7f355ad9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f355c1538f0 0x7f355c159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53440/0 (socket says 172.21.15.148:53440) 2023-10-30T13:32:08.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.774+0000 7f355ad9d640 1 -- 172.21.15.148:0/3788617534 learned_addr learned my addr 172.21.15.148:0/3788617534 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:08.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.774+0000 7f355ad9d640 1 -- 172.21.15.148:0/3788617534 >> v1:172.21.15.150:6789/0 conn(0x7f355c0cefa0 legacy=0x7f355c1640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:08.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.774+0000 7f355ad9d640 1 -- 172.21.15.148:0/3788617534 >> v1:172.21.15.148:6789/0 conn(0x7f355c15cd50 legacy=0x7f355c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:08.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.774+0000 7f355ad9d640 1 -- 172.21.15.148:0/3788617534 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c15d8c0 con 0x7f355c1538f0 2023-10-30T13:32:08.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f355ad9d640 1 --2- 172.21.15.148:0/3788617534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c1538f0 0x7f355c159fb0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f354c009870 tx=0x7f354c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d7dbf6adf6bfa0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:08.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3559d9b640 1 -- 172.21.15.148:0/3788617534 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0a8a30 con 0x7f355c1538f0 2023-10-30T13:32:08.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3559d9b640 1 -- 172.21.15.148:0/3788617534 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c0a8b90 con 0x7f355c1538f0 2023-10-30T13:32:08.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3559d9b640 1 -- 172.21.15.148:0/3788617534 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c0b0440 con 0x7f355c1538f0 2023-10-30T13:32:08.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3560a16640 1 -- 172.21.15.148:0/3788617534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c1538f0 msgr2=0x7f355c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:08.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3560a16640 1 --2- 172.21.15.148:0/3788617534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c1538f0 0x7f355c159fb0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f354c009870 tx=0x7f354c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:32:08.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3560a16640 1 -- 172.21.15.148:0/3788617534 shutdown_connections 2023-10-30T13:32:08.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3560a16640 1 --2- 172.21.15.148:0/3788617534 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c1538f0 0x7f355c159fb0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:08.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3560a16640 1 -- 172.21.15.148:0/3788617534 >> 172.21.15.148:0/3788617534 conn(0x7f355c06a640 msgr2=0x7f355c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:08.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3560a16640 1 -- 172.21.15.148:0/3788617534 shutdown_connections 2023-10-30T13:32:08.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.778+0000 7f3560a16640 1 -- 172.21.15.148:0/3788617534 wait complete. 2023-10-30T13:32:08.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f3560a16640 1 Processor -- start 2023-10-30T13:32:08.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f3560a16640 1 -- start start 2023-10-30T13:32:08.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f3560a16640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f355c0cefa0 0x7f355c1f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:08.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f3560a16640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f355c1538f0 0x7f355c0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:08.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f3560a16640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c15cd50 0x7f355c14df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:08.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f3560a16640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c15be10 con 0x7f355c15cd50 2023-10-30T13:32:08.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f3560a16640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c1f3700 con 0x7f355c1538f0 2023-10-30T13:32:08.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f3560a16640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c1ef350 con 0x7f355c0cefa0 2023-10-30T13:32:08.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355b59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c15cd50 0x7f355c14df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:08.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355b59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c15cd50 0x7f355c14df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53446/0 (socket says 172.21.15.148:53446) 2023-10-30T13:32:08.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355b59e640 1 -- 172.21.15.148:0/2400333573 learned_addr learned my addr 172.21.15.148:0/2400333573 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:08.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355ad9d640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f355c0cefa0 0x7f355c1f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:08.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355a59c640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f355c1538f0 0x7f355c0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:08.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355b59e640 1 -- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f355c0cefa0 msgr2=0x7f355c1f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:08.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355b59e640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f355c0cefa0 0x7f355c1f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:08.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355b59e640 1 -- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f355c1538f0 msgr2=0x7f355c0c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:08.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355b59e640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f355c1538f0 0x7f355c0c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:08.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355b59e640 1 -- 172.21.15.148:0/2400333573 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c0094d0 con 0x7f355c15cd50 2023-10-30T13:32:08.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.782+0000 7f355b59e640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c15cd50 0x7f355c14df10 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f355400c770 tx=0x7f355400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:08.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.786+0000 7f35437fe640 1 -- 172.21.15.148:0/2400333573 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554003b80 con 0x7f355c15cd50 2023-10-30T13:32:08.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.786+0000 7f35437fe640 1 -- 172.21.15.148:0/2400333573 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3554003ce0 con 0x7f355c15cd50 2023-10-30T13:32:08.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.786+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c1f0170 con 0x7f355c15cd50 2023-10-30T13:32:08.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.786+0000 7f35437fe640 1 -- 172.21.15.148:0/2400333573 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355400aa30 con 0x7f355c15cd50 2023-10-30T13:32:08.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.786+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c1ef8d0 con 0x7f355c15cd50 2023-10-30T13:32:08.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.786+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c14dc30 con 0x7f355c15cd50 2023-10-30T13:32:08.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.786+0000 7f35437fe640 1 -- 172.21.15.148:0/2400333573 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f355400ab90 con 0x7f355c15cd50 2023-10-30T13:32:08.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.786+0000 7f35437fe640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f35480664a0 0x7f3548068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:08.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.786+0000 7f35437fe640 1 -- 172.21.15.148:0/2400333573 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f3554084000 con 0x7f355c15cd50 2023-10-30T13:32:08.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.790+0000 7f355ad9d640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f35480664a0 0x7f3548068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:08.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.790+0000 7f355ad9d640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f35480664a0 0x7f3548068960 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f354c0b2000 tx=0x7f354c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:08.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.794+0000 7f35437fe640 1 -- 172.21.15.148:0/2400333573 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3554051060 con 0x7f355c15cd50 2023-10-30T13:32:08.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.990+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f355c04e170 con 0x7f355c15cd50 2023-10-30T13:32:08.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f35437fe640 1 -- 172.21.15.148:0/2400333573 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f3554054680 con 0x7f355c15cd50 2023-10-30T13:32:08.991 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:08.991 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:31:59.711354+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:08.991 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 93 2023-10-30T13:32:08.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f35480664a0 msgr2=0x7f3548068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f35480664a0 0x7f3548068960 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f354c0b2000 tx=0x7f354c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:32:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c15cd50 msgr2=0x7f355c14df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c15cd50 0x7f355c14df10 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f355400c770 tx=0x7f355400cc30 comp rx=0 tx=0).stop 2023-10-30T13:32:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 shutdown_connections 2023-10-30T13:32:08.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f35480664a0 0x7f3548068960 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355c15cd50 0x7f355c14df10 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f355c1538f0 0x7f355c0c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 --2- 172.21.15.148:0/2400333573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f355c0cefa0 0x7f355c1f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 >> 172.21.15.148:0/2400333573 conn(0x7f355c06a640 msgr2=0x7f355c0cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:08.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 shutdown_connections 2023-10-30T13:32:08.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:08.994+0000 7f3560a16640 1 -- 172.21.15.148:0/2400333573 wait complete. 2023-10-30T13:32:09.013 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:11.014 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:11.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.154+0000 7fb655380640 1 Processor -- start 2023-10-30T13:32:11.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.154+0000 7fb655380640 1 -- start start 2023-10-30T13:32:11.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.154+0000 7fb655380640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6501539a0 0x7fb65015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:11.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.154+0000 7fb655380640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65015a8c0 con 0x7fb650069e90 2023-10-30T13:32:11.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb655380640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb650153d80 con 0x7fb650070a00 2023-10-30T13:32:11.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb655380640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65006fbb0 con 0x7fb6501539a0 2023-10-30T13:32:11.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64f7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb650069e90 0x7fb65006a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48588/0 (socket says 172.21.15.148:48588) 2023-10-30T13:32:11.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64f7fe640 1 -- 172.21.15.148:0/2881191939 learned_addr learned my addr 172.21.15.148:0/2881191939 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:11.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64effd640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6501539a0 0x7fb65015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:11.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661445656 0 0) 0x7fb65015a8c0 con 0x7fb650069e90 2023-10-30T13:32:11.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb63c003530 con 0x7fb650069e90 2023-10-30T13:32:11.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291139149 0 0) 0x7fb650153d80 con 0x7fb650070a00 2023-10-30T13:32:11.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb65015a8c0 con 0x7fb650070a00 2023-10-30T13:32:11.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 775466280 0 0) 0x7fb63c003530 con 0x7fb650069e90 2023-10-30T13:32:11.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6501539a0 msgr2=0x7fb65015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:11.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6501539a0 0x7fb65015a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 >> v1:172.21.15.150:6789/0 conn(0x7fb650070a00 legacy=0x7fb650070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:11.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500c5c50 con 0x7fb650069e90 2023-10-30T13:32:11.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fb648001a00 con 0x7fb650069e90 2023-10-30T13:32:11.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64effd640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6501539a0 0x7fb65015a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:32:11.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 >> v1:172.21.15.148:6789/0 conn(0x7fb650069e90 legacy=0x7fb65006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:11.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb63c003b40 0x7fb63c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:11.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb63c0047d0 0x7fb63c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:11.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6501539a0 0x7fb63c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:11.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65006fbb0 con 0x7fb63c0047d0 2023-10-30T13:32:11.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c00ba70 con 0x7fb63c003b40 2023-10-30T13:32:11.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c00bc30 con 0x7fb6501539a0 2023-10-30T13:32:11.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb647fff640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb63c0047d0 0x7fb63c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:11.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64effd640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb63c003b40 0x7fb63c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:11.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb64f7fe640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6501539a0 0x7fb63c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:11.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb647fff640 1 -- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6501539a0 msgr2=0x7fb63c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:11.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb647fff640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb6501539a0 0x7fb63c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb647fff640 1 -- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb63c003b40 msgr2=0x7fb63c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:11.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb647fff640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb63c003b40 0x7fb63c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.158+0000 7fb647fff640 1 -- 172.21.15.148:0/2881191939 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb640009420 con 0x7fb63c0047d0 2023-10-30T13:32:11.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb647fff640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb63c0047d0 0x7fb63c004c50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb638007fd0 tx=0x7fb63801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78eadcef231e3e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:11.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6380052d0 con 0x7fb63c0047d0 2023-10-30T13:32:11.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6380a7050 con 0x7fb63c0047d0 2023-10-30T13:32:11.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb64e7fc640 1 -- 172.21.15.148:0/2881191939 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb638003600 con 0x7fb63c0047d0 2023-10-30T13:32:11.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb655380640 1 -- 172.21.15.148:0/2881191939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb63c0047d0 msgr2=0x7fb63c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:11.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb655380640 1 --2- 172.21.15.148:0/2881191939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb63c0047d0 0x7fb63c004c50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb638007fd0 tx=0x7fb63801c4f0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb64f7fe640 1 -- 172.21.15.148:0/2881191939 reap_dead start 2023-10-30T13:32:11.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb655380640 1 -- 172.21.15.148:0/2881191939 shutdown_connections 2023-10-30T13:32:11.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb655380640 1 -- 172.21.15.148:0/2881191939 >> 172.21.15.148:0/2881191939 conn(0x7fb6500c7910 msgr2=0x7fb6500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:11.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb655380640 1 -- 172.21.15.148:0/2881191939 shutdown_connections 2023-10-30T13:32:11.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb655380640 1 -- 172.21.15.148:0/2881191939 wait complete. 2023-10-30T13:32:11.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb655380640 1 Processor -- start 2023-10-30T13:32:11.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.162+0000 7fb655380640 1 -- start start 2023-10-30T13:32:11.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb655380640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb63c0047d0 0x7fb6500c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:11.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb655380640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6500c22b0 0x7fb650159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:11.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb655380640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb65014d430 0x7fb6501f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:11.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb655380640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500c4c60 con 0x7fb65014d430 2023-10-30T13:32:11.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb655380640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6500c4290 con 0x7fb6500c22b0 2023-10-30T13:32:11.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb655380640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6501f2bd0 con 0x7fb63c0047d0 2023-10-30T13:32:11.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb65014d430 0x7fb6501f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:11.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb65014d430 0x7fb6501f1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53460/0 (socket says 172.21.15.148:53460) 2023-10-30T13:32:11.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64f7fe640 1 -- 172.21.15.148:0/3999939287 learned_addr learned my addr 172.21.15.148:0/3999939287 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:11.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64effd640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb63c0047d0 0x7fb6500c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:11.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb647fff640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6500c22b0 0x7fb650159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:11.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64f7fe640 1 -- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb63c0047d0 msgr2=0x7fb6500c4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:11.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64f7fe640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb63c0047d0 0x7fb6500c4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64f7fe640 1 -- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6500c22b0 msgr2=0x7fb650159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:11.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64f7fe640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6500c22b0 0x7fb650159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64f7fe640 1 -- 172.21.15.148:0/3999939287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6501f2610 con 0x7fb65014d430 2023-10-30T13:32:11.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64effd640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb63c0047d0 0x7fb6500c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:32:11.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb64f7fe640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb65014d430 0x7fb6501f1c80 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb63c004750 tx=0x7fb648007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:11.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb6477fe640 1 -- 172.21.15.148:0/3999939287 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800ab70 con 0x7fb65014d430 2023-10-30T13:32:11.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb6477fe640 1 -- 172.21.15.148:0/3999939287 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64800acf0 con 0x7fb65014d430 2023-10-30T13:32:11.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb6477fe640 1 -- 172.21.15.148:0/3999939287 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648013630 con 0x7fb65014d430 2023-10-30T13:32:11.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6501f28f0 con 0x7fb65014d430 2023-10-30T13:32:11.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6500c3f20 con 0x7fb65014d430 2023-10-30T13:32:11.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6500c4440 con 0x7fb65014d430 2023-10-30T13:32:11.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.166+0000 7fb6477fe640 1 -- 172.21.15.148:0/3999939287 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb648013790 con 0x7fb65014d430 2023-10-30T13:32:11.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.170+0000 7fb6477fe640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6300664a0 0x7fb630068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:11.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.170+0000 7fb6477fe640 1 -- 172.21.15.148:0/3999939287 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fb6480849e0 con 0x7fb65014d430 2023-10-30T13:32:11.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.170+0000 7fb64effd640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6300664a0 0x7fb630068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:11.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.170+0000 7fb64effd640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6300664a0 0x7fb630068960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb640009c90 tx=0x7fb640008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:11.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.174+0000 7fb6477fe640 1 -- 172.21.15.148:0/3999939287 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb6500c4440 con 0x7fb65014d430 2023-10-30T13:32:11.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.350+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6500c4440 con 0x7fb65014d430 2023-10-30T13:32:11.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb6477fe640 1 -- 172.21.15.148:0/3999939287 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5720 (secure 0 0 0) 0x7fb6500c4440 con 0x7fb65014d430 2023-10-30T13:32:11.351 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:11.351 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:10.292361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:11.351 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 94 2023-10-30T13:32:11.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6300664a0 msgr2=0x7fb630068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:11.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6300664a0 0x7fb630068960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb640009c90 tx=0x7fb640008040 comp rx=0 tx=0).stop 2023-10-30T13:32:11.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb65014d430 msgr2=0x7fb6501f1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:11.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb65014d430 0x7fb6501f1c80 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb63c004750 tx=0x7fb648007bd0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 shutdown_connections 2023-10-30T13:32:11.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb6300664a0 0x7fb630068960 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb65014d430 0x7fb6501f1c80 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6500c22b0 0x7fb650159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 --2- 172.21.15.148:0/3999939287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb63c0047d0 0x7fb6500c4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:11.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 >> 172.21.15.148:0/3999939287 conn(0x7fb6500c7910 msgr2=0x7fb650164b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:11.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 shutdown_connections 2023-10-30T13:32:11.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:11.354+0000 7fb655380640 1 -- 172.21.15.148:0/3999939287 wait complete. 2023-10-30T13:32:11.371 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:13.372 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:13.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e5bd0f640 1 Processor -- start 2023-10-30T13:32:13.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e5bd0f640 1 -- start start 2023-10-30T13:32:13.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e5bd0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3e541538a0 0x7f3e5415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:13.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e5bd0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e540cb800 con 0x7f3e540ca0e0 2023-10-30T13:32:13.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e5bd0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5415dc30 con 0x7f3e540cf160 2023-10-30T13:32:13.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e5bd0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e540cb330 con 0x7f3e541538a0 2023-10-30T13:32:13.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e59aab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3e541538a0 0x7f3e5415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:13.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e592aa640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3e540ca0e0 0x7f3e5415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40586/0 (socket says 172.21.15.148:40586) 2023-10-30T13:32:13.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e592aa640 1 -- 172.21.15.148:0/3461968807 learned_addr learned my addr 172.21.15.148:0/3461968807 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:13.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e59aab640 1 -- 172.21.15.148:0/3461968807 >> v1:172.21.15.150:6789/0 conn(0x7f3e540cf160 legacy=0x7f3e54164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:13.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e59aab640 1 -- 172.21.15.148:0/3461968807 >> v1:172.21.15.148:6789/0 conn(0x7f3e540ca0e0 legacy=0x7f3e5415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:13.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e59aab640 1 -- 172.21.15.148:0/3461968807 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e540cad70 con 0x7f3e541538a0 2023-10-30T13:32:13.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.522+0000 7f3e58aa9640 1 -- 172.21.15.148:0/3461968807 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136559339 0 0) 0x7f3e540cb800 con 0x7f3e540ca0e0 2023-10-30T13:32:13.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e59aab640 1 --2- 172.21.15.148:0/3461968807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e541538a0 0x7f3e5415d6f0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3e500097c0 tx=0x7f3e5001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd1a5137c1614b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:13.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e58aa9640 1 -- 172.21.15.148:0/3461968807 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e500a8970 con 0x7f3e541538a0 2023-10-30T13:32:13.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e58aa9640 1 -- 172.21.15.148:0/3461968807 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e500a8ad0 con 0x7f3e541538a0 2023-10-30T13:32:13.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e58aa9640 1 -- 172.21.15.148:0/3461968807 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e500a8e60 con 0x7f3e541538a0 2023-10-30T13:32:13.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/3461968807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e541538a0 msgr2=0x7f3e5415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:13.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e5bd0f640 1 --2- 172.21.15.148:0/3461968807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e541538a0 0x7f3e5415d6f0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3e500097c0 tx=0x7f3e5001ec70 comp rx=0 tx=0).stop 2023-10-30T13:32:13.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/3461968807 shutdown_connections 2023-10-30T13:32:13.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e5bd0f640 1 --2- 172.21.15.148:0/3461968807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e541538a0 0x7f3e5415d6f0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:13.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/3461968807 >> 172.21.15.148:0/3461968807 conn(0x7f3e5406bfe0 msgr2=0x7f3e5406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:13.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/3461968807 shutdown_connections 2023-10-30T13:32:13.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.526+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/3461968807 wait complete. 2023-10-30T13:32:13.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 Processor -- start 2023-10-30T13:32:13.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 -- start start 2023-10-30T13:32:13.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e540ca0e0 0x7f3e540b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:13.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e540cf160 0x7f3e54148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:13.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e541538a0 0x7f3e541448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:13.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e541e5480 con 0x7f3e540ca0e0 2023-10-30T13:32:13.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540b4d10 con 0x7f3e540cf160 2023-10-30T13:32:13.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54144010 con 0x7f3e541538a0 2023-10-30T13:32:13.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e59aab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e540ca0e0 0x7f3e540b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:13.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e59aab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e540ca0e0 0x7f3e540b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45208/0 (socket says 172.21.15.148:45208) 2023-10-30T13:32:13.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e59aab640 1 -- 172.21.15.148:0/4121636151 learned_addr learned my addr 172.21.15.148:0/4121636151 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:13.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e592aa640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e540cf160 0x7f3e54148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:13.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5a2ac640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e541538a0 0x7f3e541448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:13.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e59aab640 1 -- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e541538a0 msgr2=0x7f3e541448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:13.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e59aab640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e541538a0 0x7f3e541448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:13.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e59aab640 1 -- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e540cf160 msgr2=0x7f3e54148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:13.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e59aab640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e540cf160 0x7f3e54148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:13.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e59aab640 1 -- 172.21.15.148:0/4121636151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e50009420 con 0x7f3e540ca0e0 2023-10-30T13:32:13.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e59aab640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e540ca0e0 0x7f3e540b47d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3e5001b880 tx=0x7f3e5001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:13.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e4a7fc640 1 -- 172.21.15.148:0/4121636151 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e500a8970 con 0x7f3e540ca0e0 2023-10-30T13:32:13.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e4a7fc640 1 -- 172.21.15.148:0/4121636151 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e500b9410 con 0x7f3e540ca0e0 2023-10-30T13:32:13.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e4a7fc640 1 -- 172.21.15.148:0/4121636151 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e500b97a0 con 0x7f3e540ca0e0 2023-10-30T13:32:13.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e54144420 con 0x7f3e540ca0e0 2023-10-30T13:32:13.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.530+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e540b4490 con 0x7f3e540ca0e0 2023-10-30T13:32:13.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.534+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5404e170 con 0x7f3e540ca0e0 2023-10-30T13:32:13.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.534+0000 7f3e4a7fc640 1 -- 172.21.15.148:0/4121636151 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e500b7040 con 0x7f3e540ca0e0 2023-10-30T13:32:13.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.534+0000 7f3e4a7fc640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e380664a0 0x7f3e38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:13.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.534+0000 7f3e4a7fc640 1 -- 172.21.15.148:0/4121636151 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f3e50121f70 con 0x7f3e540ca0e0 2023-10-30T13:32:13.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.534+0000 7f3e592aa640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e380664a0 0x7f3e38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:13.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.534+0000 7f3e592aa640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e380664a0 0x7f3e38068960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3e54151060 tx=0x7f3e44007d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:13.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.538+0000 7f3e4a7fc640 1 -- 172.21.15.148:0/4121636151 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e500eefc0 con 0x7f3e540ca0e0 2023-10-30T13:32:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.718+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e540666a0 con 0x7f3e540ca0e0 2023-10-30T13:32:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e4a7fc640 1 -- 172.21.15.148:0/4121636151 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f3e500b5020 con 0x7f3e540ca0e0 2023-10-30T13:32:13.719 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:13.719 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:10.292361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 94 2023-10-30T13:32:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e380664a0 msgr2=0x7f3e38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:13.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e380664a0 0x7f3e38068960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3e54151060 tx=0x7f3e44007d10 comp rx=0 tx=0).stop 2023-10-30T13:32:13.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e540ca0e0 msgr2=0x7f3e540b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:13.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e540ca0e0 0x7f3e540b47d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3e5001b880 tx=0x7f3e5001ba10 comp rx=0 tx=0).stop 2023-10-30T13:32:13.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 shutdown_connections 2023-10-30T13:32:13.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e380664a0 0x7f3e38068960 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:13.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e541538a0 0x7f3e541448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:13.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e540cf160 0x7f3e54148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:13.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 --2- 172.21.15.148:0/4121636151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e540ca0e0 0x7f3e540b47d0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:13.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 >> 172.21.15.148:0/4121636151 conn(0x7f3e5406bfe0 msgr2=0x7f3e54142980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:13.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 shutdown_connections 2023-10-30T13:32:13.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:13.722+0000 7f3e5bd0f640 1 -- 172.21.15.148:0/4121636151 wait complete. 2023-10-30T13:32:13.740 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6546": { "addr": "172.21.15.148:6837/3232618434", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3232618434, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3232618434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6546, "incarnation": 93, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 30 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:32:10.292361+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6546 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:32:15.741 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:15.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd9362640 1 Processor -- start 2023-10-30T13:32:15.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd9362640 1 -- start start 2023-10-30T13:32:15.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd9362640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd415cd50 0x7f2cd40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:15.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd9362640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2cd40cefa0 0x7f2cd40cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:15.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd9362640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd415a6c0 con 0x7f2cd41538f0 2023-10-30T13:32:15.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd9362640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd415d300 con 0x7f2cd40cefa0 2023-10-30T13:32:15.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd9362640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd415d5e0 con 0x7f2cd415cd50 2023-10-30T13:32:15.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd3fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2cd41538f0 0x7f2cd4159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40596/0 (socket says 172.21.15.148:40596) 2023-10-30T13:32:15.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd3fff640 1 -- 172.21.15.148:0/959725204 learned_addr learned my addr 172.21.15.148:0/959725204 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:15.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd37fe640 1 --2- 172.21.15.148:0/959725204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd415cd50 0x7f2cd40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:15.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd8b61640 1 --2- 172.21.15.148:0/959725204 >> v2:172.21.15.150:3300/0 conn(0x7f2cd40cefa0 0x7f2cd40cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:15.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd2ffd640 1 -- 172.21.15.148:0/959725204 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755638477 0 0) 0x7f2cd415a6c0 con 0x7f2cd41538f0 2023-10-30T13:32:15.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd2ffd640 1 -- 172.21.15.148:0/959725204 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cbc003530 con 0x7f2cd41538f0 2023-10-30T13:32:15.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd37fe640 1 -- 172.21.15.148:0/959725204 >> v2:172.21.15.150:3300/0 conn(0x7f2cd40cefa0 msgr2=0x7f2cd40cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:15.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd37fe640 1 --2- 172.21.15.148:0/959725204 >> v2:172.21.15.150:3300/0 conn(0x7f2cd40cefa0 0x7f2cd40cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:15.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd37fe640 1 -- 172.21.15.148:0/959725204 >> v1:172.21.15.148:6789/0 conn(0x7f2cd41538f0 legacy=0x7f2cd4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:15.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd2ffd640 1 -- 172.21.15.148:0/959725204 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756259410 0 0) 0x7f2cbc003530 con 0x7f2cd41538f0 2023-10-30T13:32:15.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd37fe640 1 -- 172.21.15.148:0/959725204 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd415d8c0 con 0x7f2cd415cd50 2023-10-30T13:32:15.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.890+0000 7f2cd8b61640 1 --2- 172.21.15.148:0/959725204 >> v2:172.21.15.150:3300/0 conn(0x7f2cd40cefa0 0x7f2cd40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:32:15.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd37fe640 1 --2- 172.21.15.148:0/959725204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd415cd50 0x7f2cd40c9f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2cc80097c0 tx=0x7f2cc801eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23a49c7109a3b386 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:15.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd2ffd640 1 -- 172.21.15.148:0/959725204 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4001680 con 0x7f2cd415cd50 2023-10-30T13:32:15.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd2ffd640 1 -- 172.21.15.148:0/959725204 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc80a85d0 con 0x7f2cd415cd50 2023-10-30T13:32:15.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd2ffd640 1 -- 172.21.15.148:0/959725204 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc80a8940 con 0x7f2cd415cd50 2023-10-30T13:32:15.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd9362640 1 -- 172.21.15.148:0/959725204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd415cd50 msgr2=0x7f2cd40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:15.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd9362640 1 --2- 172.21.15.148:0/959725204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd415cd50 0x7f2cd40c9f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2cc80097c0 tx=0x7f2cc801eb90 comp rx=0 tx=0).stop 2023-10-30T13:32:15.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd9362640 1 -- 172.21.15.148:0/959725204 shutdown_connections 2023-10-30T13:32:15.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd9362640 1 --2- 172.21.15.148:0/959725204 >> v2:172.21.15.150:3300/0 conn(0x7f2cd40cefa0 0x7f2cd40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:15.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd9362640 1 --2- 172.21.15.148:0/959725204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd415cd50 0x7f2cd40c9f90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:15.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd9362640 1 -- 172.21.15.148:0/959725204 >> 172.21.15.148:0/959725204 conn(0x7f2cd406a640 msgr2=0x7f2cd4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:15.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd9362640 1 -- 172.21.15.148:0/959725204 shutdown_connections 2023-10-30T13:32:15.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.894+0000 7f2cd9362640 1 -- 172.21.15.148:0/959725204 wait complete. 2023-10-30T13:32:15.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 Processor -- start 2023-10-30T13:32:15.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 -- start start 2023-10-30T13:32:15.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd40cefa0 0x7f2cd4158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:15.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2cd41538f0 0x7f2cd41eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:15.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd415cd50 0x7f2cd41ef410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:15.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd4158e70 con 0x7f2cd41538f0 2023-10-30T13:32:15.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd415b150 con 0x7f2cd415cd50 2023-10-30T13:32:15.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd41efcb0 con 0x7f2cd40cefa0 2023-10-30T13:32:15.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2cd41538f0 0x7f2cd41eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:15.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2cd41538f0 0x7f2cd41eeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45220/0 (socket says 172.21.15.148:45220) 2023-10-30T13:32:15.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd37fe640 1 -- 172.21.15.148:0/2548346042 learned_addr learned my addr 172.21.15.148:0/2548346042 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:15.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd3fff640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd40cefa0 0x7f2cd4158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:15.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd8b61640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd415cd50 0x7f2cd41ef410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:15.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd37fe640 1 -- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd40cefa0 msgr2=0x7f2cd4158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:15.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd37fe640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd40cefa0 0x7f2cd4158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:15.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd37fe640 1 -- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd415cd50 msgr2=0x7f2cd41ef410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:15.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd37fe640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd415cd50 0x7f2cd41ef410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:15.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd37fe640 1 -- 172.21.15.148:0/2548346042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc009a70 con 0x7f2cd41538f0 2023-10-30T13:32:15.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd37fe640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2cd41538f0 0x7f2cd41eeed0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2cc801f430 tx=0x7f2cc801b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:15.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd0ff9640 1 -- 172.21.15.148:0/2548346042 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc80c1070 con 0x7f2cd41538f0 2023-10-30T13:32:15.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd0ff9640 1 -- 172.21.15.148:0/2548346042 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc80a99e0 con 0x7f2cd41538f0 2023-10-30T13:32:15.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc8009420 con 0x7f2cd41538f0 2023-10-30T13:32:15.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd0ff9640 1 -- 172.21.15.148:0/2548346042 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc80a9d50 con 0x7f2cd41538f0 2023-10-30T13:32:15.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.898+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd4158130 con 0x7f2cd41538f0 2023-10-30T13:32:15.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.902+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd404e170 con 0x7f2cd41538f0 2023-10-30T13:32:15.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.902+0000 7f2cd0ff9640 1 -- 172.21.15.148:0/2548346042 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2cc8003620 con 0x7f2cd41538f0 2023-10-30T13:32:15.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.902+0000 7f2cd0ff9640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2cc00664d0 0x7f2cc0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:15.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.902+0000 7f2cd0ff9640 1 -- 172.21.15.148:0/2548346042 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f2cc8121ae0 con 0x7f2cd41538f0 2023-10-30T13:32:15.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.902+0000 7f2cd3fff640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2cc00664d0 0x7f2cc0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:15.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.902+0000 7f2cd3fff640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2cc00664d0 0x7f2cc0068990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4007a10 tx=0x7f2cc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:15.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:15.906+0000 7f2cd0ff9640 1 -- 172.21.15.148:0/2548346042 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2cc80eeb20 con 0x7f2cd41538f0 2023-10-30T13:32:16.085 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.086+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cd4149950 con 0x7f2cd41538f0 2023-10-30T13:32:16.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.086+0000 7f2cd0ff9640 1 -- 172.21.15.148:0/2548346042 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5720 (secure 0 0 0) 0x7f2cc80b5090 con 0x7f2cd41538f0 2023-10-30T13:32:16.086 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:16.086 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:10.292361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:16.087 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 94 2023-10-30T13:32:16.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2cc00664d0 msgr2=0x7f2cc0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:16.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2cc00664d0 0x7f2cc0068990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4007a10 tx=0x7f2cc4009040 comp rx=0 tx=0).stop 2023-10-30T13:32:16.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2cd41538f0 msgr2=0x7f2cd41eeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:16.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2cd41538f0 0x7f2cd41eeed0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2cc801f430 tx=0x7f2cc801b930 comp rx=0 tx=0).stop 2023-10-30T13:32:16.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 shutdown_connections 2023-10-30T13:32:16.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2cc00664d0 0x7f2cc0068990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:16.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2cd41538f0 0x7f2cd41eeed0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:16.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd415cd50 0x7f2cd41ef410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:16.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 --2- 172.21.15.148:0/2548346042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2cd40cefa0 0x7f2cd4158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:16.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 >> 172.21.15.148:0/2548346042 conn(0x7f2cd406a640 msgr2=0x7f2cd40c4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:16.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 shutdown_connections 2023-10-30T13:32:16.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:16.090+0000 7f2cd9362640 1 -- 172.21.15.148:0/2548346042 wait complete. 2023-10-30T13:32:16.107 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:18.107 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:18.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54eebf5640 1 Processor -- start 2023-10-30T13:32:18.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54eebf5640 1 -- start start 2023-10-30T13:32:18.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54eebf5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e81538a0 0x7f54e815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:18.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54eebf5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f54e80ca0e0 0x7f54e815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:18.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54eebf5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e80cb800 con 0x7f54e80cf160 2023-10-30T13:32:18.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54eebf5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e815dc30 con 0x7f54e80ca0e0 2023-10-30T13:32:18.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54eebf5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80cb330 con 0x7f54e81538a0 2023-10-30T13:32:18.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54ec991640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e81538a0 0x7f54e815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:18.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54e7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f54e80ca0e0 0x7f54e815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:18.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54ec991640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e81538a0 0x7f54e815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:60858/0 (socket says 172.21.15.148:60858) 2023-10-30T13:32:18.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54e7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f54e80ca0e0 0x7f54e815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:43512/0 (socket says 172.21.15.148:43512) 2023-10-30T13:32:18.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54ec991640 1 -- 172.21.15.148:0/2408988670 learned_addr learned my addr 172.21.15.148:0/2408988670 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:18.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54ec991640 1 -- 172.21.15.148:0/2408988670 >> v2:172.21.15.150:3300/0 conn(0x7f54e80ca0e0 msgr2=0x7f54e815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:18.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54ec991640 1 --2- 172.21.15.148:0/2408988670 >> v2:172.21.15.150:3300/0 conn(0x7f54e80ca0e0 0x7f54e815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:18.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54ec991640 1 -- 172.21.15.148:0/2408988670 >> v1:172.21.15.150:6789/0 conn(0x7f54e80cf160 legacy=0x7f54e80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:18.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54ec991640 1 -- 172.21.15.148:0/2408988670 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e80cad70 con 0x7f54e81538a0 2023-10-30T13:32:18.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.258+0000 7f54e7fff640 1 --2- 172.21.15.148:0/2408988670 >> v2:172.21.15.150:3300/0 conn(0x7f54e80ca0e0 0x7f54e815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:32:18.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54ec991640 1 --2- 172.21.15.148:0/2408988670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e81538a0 0x7f54e815d6f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009870 tx=0x7f54dc01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cda8d097d58d7762 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:18.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54e77fe640 1 -- 172.21.15.148:0/2408988670 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc0a8a30 con 0x7f54e81538a0 2023-10-30T13:32:18.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54e77fe640 1 -- 172.21.15.148:0/2408988670 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54dc0a8b90 con 0x7f54e81538a0 2023-10-30T13:32:18.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54e77fe640 1 -- 172.21.15.148:0/2408988670 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc0b0440 con 0x7f54e81538a0 2023-10-30T13:32:18.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54eebf5640 1 -- 172.21.15.148:0/2408988670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e81538a0 msgr2=0x7f54e815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:18.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54eebf5640 1 --2- 172.21.15.148:0/2408988670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e81538a0 0x7f54e815d6f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009870 tx=0x7f54dc01ed30 comp rx=0 tx=0).stop 2023-10-30T13:32:18.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54eebf5640 1 -- 172.21.15.148:0/2408988670 shutdown_connections 2023-10-30T13:32:18.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54eebf5640 1 --2- 172.21.15.148:0/2408988670 >> v2:172.21.15.150:3300/0 conn(0x7f54e80ca0e0 0x7f54e815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:18.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54eebf5640 1 --2- 172.21.15.148:0/2408988670 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e81538a0 0x7f54e815d6f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:18.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54eebf5640 1 -- 172.21.15.148:0/2408988670 >> 172.21.15.148:0/2408988670 conn(0x7f54e806bfe0 msgr2=0x7f54e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:18.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54eebf5640 1 -- 172.21.15.148:0/2408988670 shutdown_connections 2023-10-30T13:32:18.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.262+0000 7f54eebf5640 1 -- 172.21.15.148:0/2408988670 wait complete. 2023-10-30T13:32:18.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54eebf5640 1 Processor -- start 2023-10-30T13:32:18.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54eebf5640 1 -- start start 2023-10-30T13:32:18.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54eebf5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e80ca0e0 0x7f54e80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:18.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54eebf5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e80cf160 0x7f54e815b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:18.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54eebf5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54e81538a0 0x7f54e81475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:18.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54eebf5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8158920 con 0x7f54e80cf160 2023-10-30T13:32:18.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54eebf5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8157d90 con 0x7f54e81538a0 2023-10-30T13:32:18.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54eebf5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e81594e0 con 0x7f54e80ca0e0 2023-10-30T13:32:18.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e80cf160 0x7f54e815b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:18.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e80cf160 0x7f54e815b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45224/0 (socket says 172.21.15.148:45224) 2023-10-30T13:32:18.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e7fff640 1 -- 172.21.15.148:0/1392265039 learned_addr learned my addr 172.21.15.148:0/1392265039 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:18.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54ec991640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e80ca0e0 0x7f54e80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:18.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54ed192640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54e81538a0 0x7f54e81475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:18.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e7fff640 1 -- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e80ca0e0 msgr2=0x7f54e80b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:18.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e7fff640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e80ca0e0 0x7f54e80b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:18.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e7fff640 1 -- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54e81538a0 msgr2=0x7f54e81475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:18.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e7fff640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54e81538a0 0x7f54e81475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:18.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e7fff640 1 -- 172.21.15.148:0/1392265039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d4009420 con 0x7f54e80cf160 2023-10-30T13:32:18.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e7fff640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e80cf160 0x7f54e815b210 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f54d400ebe0 tx=0x7f54d400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:18.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e57fa640 1 -- 172.21.15.148:0/1392265039 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d400eee0 con 0x7f54e80cf160 2023-10-30T13:32:18.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e57fa640 1 -- 172.21.15.148:0/1392265039 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d400cc30 con 0x7f54e80cf160 2023-10-30T13:32:18.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc0094d0 con 0x7f54e80cf160 2023-10-30T13:32:18.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.266+0000 7f54e57fa640 1 -- 172.21.15.148:0/1392265039 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4013470 con 0x7f54e80cf160 2023-10-30T13:32:18.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.270+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e81582d0 con 0x7f54e80cf160 2023-10-30T13:32:18.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.270+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e80b2a50 con 0x7f54e80cf160 2023-10-30T13:32:18.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.270+0000 7f54e57fa640 1 -- 172.21.15.148:0/1392265039 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f54d40135d0 con 0x7f54e80cf160 2023-10-30T13:32:18.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.270+0000 7f54e57fa640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54cc0664d0 0x7f54cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:18.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.270+0000 7f54e57fa640 1 -- 172.21.15.148:0/1392265039 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f54d40845a0 con 0x7f54e80cf160 2023-10-30T13:32:18.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.270+0000 7f54ec991640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54cc0664d0 0x7f54cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:18.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.274+0000 7f54ec991640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54cc0664d0 0x7f54cc068990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0b2000 tx=0x7f54dc0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:18.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.274+0000 7f54e57fa640 1 -- 172.21.15.148:0/1392265039 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f54d4051580 con 0x7f54e80cf160 2023-10-30T13:32:18.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.458+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54e804e170 con 0x7f54e80cf160 2023-10-30T13:32:18.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.458+0000 7f54e57fa640 1 -- 172.21.15.148:0/1392265039 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f54d4054ba0 con 0x7f54e80cf160 2023-10-30T13:32:18.459 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:18.459 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:15.897043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:18.459 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 95 2023-10-30T13:32:18.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54cc0664d0 msgr2=0x7f54cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:18.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54cc0664d0 0x7f54cc068990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0b2000 tx=0x7f54dc0aa070 comp rx=0 tx=0).stop 2023-10-30T13:32:18.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e80cf160 msgr2=0x7f54e815b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:18.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e80cf160 0x7f54e815b210 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f54d400ebe0 tx=0x7f54d400c520 comp rx=0 tx=0).stop 2023-10-30T13:32:18.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 shutdown_connections 2023-10-30T13:32:18.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f54cc0664d0 0x7f54cc068990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:18.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54e80cf160 0x7f54e815b210 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:18.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54e81538a0 0x7f54e81475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:18.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 --2- 172.21.15.148:0/1392265039 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f54e80ca0e0 0x7f54e80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:18.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 >> 172.21.15.148:0/1392265039 conn(0x7f54e806bfe0 msgr2=0x7f54e814fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:18.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 shutdown_connections 2023-10-30T13:32:18.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:18.462+0000 7f54eebf5640 1 -- 172.21.15.148:0/1392265039 wait complete. 2023-10-30T13:32:18.479 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:20.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:20.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.629+0000 7f068cb9d640 1 Processor -- start 2023-10-30T13:32:20.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.629+0000 7f068cb9d640 1 -- start start 2023-10-30T13:32:20.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068cb9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06880c6b80 0x7f06880bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:20.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068cb9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f068813f4a0 0x7f06880c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:20.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068cb9d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06880c0a50 con 0x7f06880c6f60 2023-10-30T13:32:20.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068cb9d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068806f6e0 con 0x7f06880c6b80 2023-10-30T13:32:20.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068cb9d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068806f9c0 con 0x7f068813f4a0 2023-10-30T13:32:20.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f067dd9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f06880c6f60 0x7f068813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40604/0 (socket says 172.21.15.148:40604) 2023-10-30T13:32:20.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f067dd9b640 1 -- 172.21.15.148:0/3380053497 learned_addr learned my addr 172.21.15.148:0/3380053497 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:20.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068659c640 1 --2- 172.21.15.148:0/3380053497 >> v2:172.21.15.150:3300/0 conn(0x7f06880c6b80 0x7f06880bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:20.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f0686d9d640 1 --2- 172.21.15.148:0/3380053497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f068813f4a0 0x7f06880c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:20.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f0685d9b640 1 -- 172.21.15.148:0/3380053497 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17143868 0 0) 0x7f06880c0a50 con 0x7f06880c6f60 2023-10-30T13:32:20.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f0685d9b640 1 -- 172.21.15.148:0/3380053497 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0674003530 con 0x7f06880c6f60 2023-10-30T13:32:20.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f0685d9b640 1 -- 172.21.15.148:0/3380053497 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531576140 0 0) 0x7f0674003530 con 0x7f06880c6f60 2023-10-30T13:32:20.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068659c640 1 -- 172.21.15.148:0/3380053497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f068813f4a0 msgr2=0x7f06880c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:20.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068659c640 1 --2- 172.21.15.148:0/3380053497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f068813f4a0 0x7f06880c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068659c640 1 -- 172.21.15.148:0/3380053497 >> v1:172.21.15.148:6789/0 conn(0x7f06880c6f60 legacy=0x7f068813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:20.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f068659c640 1 -- 172.21.15.148:0/3380053497 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068806fca0 con 0x7f06880c6b80 2023-10-30T13:32:20.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.633+0000 7f0686d9d640 1 --2- 172.21.15.148:0/3380053497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f068813f4a0 0x7f06880c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:32:20.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068659c640 1 --2- 172.21.15.148:0/3380053497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06880c6b80 0x7f06880bfec0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f06780097c0 tx=0x7f067801c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f3f011fe5cb6d9b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:20.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f0685d9b640 1 -- 172.21.15.148:0/3380053497 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0670001a00 con 0x7f06880c6b80 2023-10-30T13:32:20.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f0685d9b640 1 -- 172.21.15.148:0/3380053497 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06780a7050 con 0x7f06880c6b80 2023-10-30T13:32:20.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f0685d9b640 1 -- 172.21.15.148:0/3380053497 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06780ab600 con 0x7f06880c6b80 2023-10-30T13:32:20.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068cb9d640 1 -- 172.21.15.148:0/3380053497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06880c6b80 msgr2=0x7f06880bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:20.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068cb9d640 1 --2- 172.21.15.148:0/3380053497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06880c6b80 0x7f06880bfec0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f06780097c0 tx=0x7f067801c3a0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068cb9d640 1 -- 172.21.15.148:0/3380053497 shutdown_connections 2023-10-30T13:32:20.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068cb9d640 1 --2- 172.21.15.148:0/3380053497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f068813f4a0 0x7f06880c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068cb9d640 1 --2- 172.21.15.148:0/3380053497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06880c6b80 0x7f06880bfec0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068cb9d640 1 -- 172.21.15.148:0/3380053497 >> 172.21.15.148:0/3380053497 conn(0x7f06880c7910 msgr2=0x7f06880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:20.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068cb9d640 1 -- 172.21.15.148:0/3380053497 shutdown_connections 2023-10-30T13:32:20.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068cb9d640 1 -- 172.21.15.148:0/3380053497 wait complete. 2023-10-30T13:32:20.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.637+0000 7f068cb9d640 1 Processor -- start 2023-10-30T13:32:20.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 -- start start 2023-10-30T13:32:20.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06880c6b80 0x7f0688135480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:20.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06880c6f60 0x7f0688135d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:20.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f068813f4a0 0x7f0688158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:20.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06881558c0 con 0x7f068813f4a0 2023-10-30T13:32:20.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06881359c0 con 0x7f06880c6b80 2023-10-30T13:32:20.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0688154350 con 0x7f06880c6f60 2023-10-30T13:32:20.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f0686d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f068813f4a0 0x7f0688158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:20.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f0686d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f068813f4a0 0x7f0688158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45230/0 (socket says 172.21.15.148:45230) 2023-10-30T13:32:20.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f0686d9d640 1 -- 172.21.15.148:0/2846436394 learned_addr learned my addr 172.21.15.148:0/2846436394 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:20.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068659c640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06880c6b80 0x7f0688135480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:20.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f067dd9b640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06880c6f60 0x7f0688135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:20.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f0686d9d640 1 -- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06880c6f60 msgr2=0x7f0688135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:20.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f0686d9d640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06880c6f60 0x7f0688135d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f0686d9d640 1 -- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06880c6b80 msgr2=0x7f0688135480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:20.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f0686d9d640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06880c6b80 0x7f0688135480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f0686d9d640 1 -- 172.21.15.148:0/2846436394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0680009420 con 0x7f068813f4a0 2023-10-30T13:32:20.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f067dd9b640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06880c6f60 0x7f0688135d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:32:20.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f0686d9d640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f068813f4a0 0x7f0688158c50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f068000e930 tx=0x7f068000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:20.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f067f7fe640 1 -- 172.21.15.148:0/2846436394 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0680009680 con 0x7f068813f4a0 2023-10-30T13:32:20.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f067f7fe640 1 -- 172.21.15.148:0/2846436394 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0680004020 con 0x7f068813f4a0 2023-10-30T13:32:20.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f067f7fe640 1 -- 172.21.15.148:0/2846436394 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06800043f0 con 0x7f068813f4a0 2023-10-30T13:32:20.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0678009420 con 0x7f068813f4a0 2023-10-30T13:32:20.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0688155bf0 con 0x7f068813f4a0 2023-10-30T13:32:20.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.641+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06880c7d90 con 0x7f068813f4a0 2023-10-30T13:32:20.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.645+0000 7f067f7fe640 1 -- 172.21.15.148:0/2846436394 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f068000c940 con 0x7f068813f4a0 2023-10-30T13:32:20.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.645+0000 7f067f7fe640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06680664d0 0x7f0668068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:20.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.645+0000 7f067f7fe640 1 -- 172.21.15.148:0/2846436394 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f0680083ff0 con 0x7f068813f4a0 2023-10-30T13:32:20.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.645+0000 7f068659c640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06680664d0 0x7f0668068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:20.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.645+0000 7f068659c640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06680664d0 0x7f0668068990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f06780ad000 tx=0x7f06780a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:20.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.649+0000 7f067f7fe640 1 -- 172.21.15.148:0/2846436394 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06800b5090 con 0x7f068813f4a0 2023-10-30T13:32:20.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0688155bf0 con 0x7f068813f4a0 2023-10-30T13:32:20.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f067f7fe640 1 -- 172.21.15.148:0/2846436394 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f0688155bf0 con 0x7f068813f4a0 2023-10-30T13:32:20.828 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:20.829 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:15.897043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:20.829 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 95 2023-10-30T13:32:20.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06680664d0 msgr2=0x7f0668068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:20.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06680664d0 0x7f0668068990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f06780ad000 tx=0x7f06780a6000 comp rx=0 tx=0).stop 2023-10-30T13:32:20.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f068813f4a0 msgr2=0x7f0688158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:20.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f068813f4a0 0x7f0688158c50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f068000e930 tx=0x7f068000edf0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 shutdown_connections 2023-10-30T13:32:20.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06680664d0 0x7f0668068990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f068813f4a0 0x7f0688158c50 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06880c6f60 0x7f0688135d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 --2- 172.21.15.148:0/2846436394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06880c6b80 0x7f0688135480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:20.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 >> 172.21.15.148:0/2846436394 conn(0x7f06880c7910 msgr2=0x7f06880c0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:20.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 shutdown_connections 2023-10-30T13:32:20.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:20.829+0000 7f068cb9d640 1 -- 172.21.15.148:0/2846436394 wait complete. 2023-10-30T13:32:20.849 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:22.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1c291640 1 Processor -- start 2023-10-30T13:32:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1c291640 1 -- start start 2023-10-30T13:32:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1c291640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad1406f7a0 0x7fad1406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1c291640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fad140c6bd0 0x7fad14069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1c291640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1406a3d0 con 0x7fad14071110 2023-10-30T13:32:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1c291640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad140700c0 con 0x7fad140c6bd0 2023-10-30T13:32:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1c291640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad14066cf0 con 0x7fad1406f7a0 2023-10-30T13:32:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1982c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fad140c6bd0 0x7fad14069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1982c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fad140c6bd0 0x7fad14069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59504/0 (socket says 172.21.15.148:59504) 2023-10-30T13:32:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1982c640 1 -- 172.21.15.148:0/344816575 learned_addr learned my addr 172.21.15.148:0/344816575 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:22.997+0000 7fad1a02d640 1 --2- 172.21.15.148:0/344816575 >> v2:172.21.15.150:3300/0 conn(0x7fad1406f7a0 0x7fad1406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1982c640 1 -- 172.21.15.148:0/344816575 >> v2:172.21.15.150:3300/0 conn(0x7fad1406f7a0 msgr2=0x7fad1406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1982c640 1 --2- 172.21.15.148:0/344816575 >> v2:172.21.15.150:3300/0 conn(0x7fad1406f7a0 0x7fad1406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1982c640 1 -- 172.21.15.148:0/344816575 >> v1:172.21.15.150:6789/0 conn(0x7fad14071110 legacy=0x7fad14158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1982c640 1 -- 172.21.15.148:0/344816575 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad140c70d0 con 0x7fad140c6bd0 2023-10-30T13:32:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1982c640 1 --2- 172.21.15.148:0/344816575 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad140c6bd0 0x7fad14069e90 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fad1406c830 tx=0x7fad0401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d8069505d27bee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1902b640 1 -- 172.21.15.148:0/344816575 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad040a89f0 con 0x7fad140c6bd0 2023-10-30T13:32:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1902b640 1 -- 172.21.15.148:0/344816575 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad040a8b50 con 0x7fad140c6bd0 2023-10-30T13:32:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1902b640 1 -- 172.21.15.148:0/344816575 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad040b0440 con 0x7fad140c6bd0 2023-10-30T13:32:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1c291640 1 -- 172.21.15.148:0/344816575 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad140c6bd0 msgr2=0x7fad14069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.001+0000 7fad1c291640 1 --2- 172.21.15.148:0/344816575 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad140c6bd0 0x7fad14069e90 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fad1406c830 tx=0x7fad0401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:32:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 -- 172.21.15.148:0/344816575 shutdown_connections 2023-10-30T13:32:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 --2- 172.21.15.148:0/344816575 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad140c6bd0 0x7fad14069e90 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 --2- 172.21.15.148:0/344816575 >> v2:172.21.15.150:3300/0 conn(0x7fad1406f7a0 0x7fad1406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 -- 172.21.15.148:0/344816575 >> 172.21.15.148:0/344816575 conn(0x7fad140c7910 msgr2=0x7fad140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 -- 172.21.15.148:0/344816575 shutdown_connections 2023-10-30T13:32:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 -- 172.21.15.148:0/344816575 wait complete. 2023-10-30T13:32:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 Processor -- start 2023-10-30T13:32:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 -- start start 2023-10-30T13:32:23.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad1406f7a0 0x7fad140bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:23.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad14071110 0x7fad141f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:23.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad140c6bd0 0x7fad1414ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:23.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad140ca710 con 0x7fad14071110 2023-10-30T13:32:23.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad140ca9d0 con 0x7fad1406f7a0 2023-10-30T13:32:23.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1c291640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad141f2cf0 con 0x7fad140c6bd0 2023-10-30T13:32:23.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1982c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad14071110 0x7fad141f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:23.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1982c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad14071110 0x7fad141f2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59516/0 (socket says 172.21.15.148:59516) 2023-10-30T13:32:23.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1982c640 1 -- 172.21.15.148:0/3591856098 learned_addr learned my addr 172.21.15.148:0/3591856098 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:23.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1a02d640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad1406f7a0 0x7fad140bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:23.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1a82e640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad140c6bd0 0x7fad1414ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:23.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1982c640 1 -- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad140c6bd0 msgr2=0x7fad1414ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:23.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1982c640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad140c6bd0 0x7fad1414ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1982c640 1 -- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad1406f7a0 msgr2=0x7fad140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1982c640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad1406f7a0 0x7fad140bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1982c640 1 -- 172.21.15.148:0/3591856098 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad040094d0 con 0x7fad14071110 2023-10-30T13:32:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1a82e640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad140c6bd0 0x7fad1414ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:32:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.005+0000 7fad1982c640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad14071110 0x7fad141f2420 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fad040b3040 tx=0x7fad0401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:23.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad0affd640 1 -- 172.21.15.148:0/3591856098 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad040b9620 con 0x7fad14071110 2023-10-30T13:32:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad0affd640 1 -- 172.21.15.148:0/3591856098 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad040b9780 con 0x7fad14071110 2023-10-30T13:32:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad141f2fd0 con 0x7fad14071110 2023-10-30T13:32:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad0affd640 1 -- 172.21.15.148:0/3591856098 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad040b9b30 con 0x7fad14071110 2023-10-30T13:32:23.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad140c1780 con 0x7fad14071110 2023-10-30T13:32:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1404e170 con 0x7fad14071110 2023-10-30T13:32:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad0affd640 1 -- 172.21.15.148:0/3591856098 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad040b7070 con 0x7fad14071110 2023-10-30T13:32:23.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad0affd640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7facf80664a0 0x7facf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:23.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad0affd640 1 -- 172.21.15.148:0/3591856098 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fad041222a0 con 0x7fad14071110 2023-10-30T13:32:23.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.009+0000 7fad1a02d640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7facf80664a0 0x7facf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:23.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.013+0000 7fad1a02d640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7facf80664a0 0x7facf8068960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fad10009a50 tx=0x7fad10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:23.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.013+0000 7fad0affd640 1 -- 172.21.15.148:0/3591856098 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad040ef1d0 con 0x7fad14071110 2023-10-30T13:32:23.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.193+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad140c8400 con 0x7fad14071110 2023-10-30T13:32:23.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.193+0000 7fad0affd640 1 -- 172.21.15.148:0/3591856098 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5717 (secure 0 0 0) 0x7fad040f27f0 con 0x7fad14071110 2023-10-30T13:32:23.190 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:23.190 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:15.897043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:23.190 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 95 2023-10-30T13:32:23.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7facf80664a0 msgr2=0x7facf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:23.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7facf80664a0 0x7facf8068960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fad10009a50 tx=0x7fad10008040 comp rx=0 tx=0).stop 2023-10-30T13:32:23.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad14071110 msgr2=0x7fad141f2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:23.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad14071110 0x7fad141f2420 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fad040b3040 tx=0x7fad0401ba90 comp rx=0 tx=0).stop 2023-10-30T13:32:23.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 shutdown_connections 2023-10-30T13:32:23.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7facf80664a0 0x7facf8068960 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:23.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fad14071110 0x7fad141f2420 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:23.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fad140c6bd0 0x7fad1414ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:23.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 --2- 172.21.15.148:0/3591856098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad1406f7a0 0x7fad140bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:23.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 >> 172.21.15.148:0/3591856098 conn(0x7fad140c7910 msgr2=0x7fad140c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:23.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 shutdown_connections 2023-10-30T13:32:23.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:23.197+0000 7fad1c291640 1 -- 172.21.15.148:0/3591856098 wait complete. 2023-10-30T13:32:23.210 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:25.211 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:25.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419c8d6640 1 Processor -- start 2023-10-30T13:32:25.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419c8d6640 1 -- start start 2023-10-30T13:32:25.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419c8d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981538a0 0x7f419815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:25.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419c8d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41980cf160 0x7f41980cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:25.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419c8d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41980cb800 con 0x7f41980ca0e0 2023-10-30T13:32:25.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419c8d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419815dc30 con 0x7f41980cf160 2023-10-30T13:32:25.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419c8d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41980cb330 con 0x7f41981538a0 2023-10-30T13:32:25.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4195d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f41980ca0e0 0x7f419815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44810/0 (socket says 172.21.15.148:44810) 2023-10-30T13:32:25.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4195d9b640 1 -- 172.21.15.148:0/2368519532 learned_addr learned my addr 172.21.15.148:0/2368519532 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:25.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4196d9d640 1 --2- 172.21.15.148:0/2368519532 >> v2:172.21.15.150:3300/0 conn(0x7f41980cf160 0x7f41980cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:25.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419659c640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981538a0 0x7f419815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:25.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487934251 0 0) 0x7f41980cb800 con 0x7f41980ca0e0 2023-10-30T13:32:25.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4180003530 con 0x7f41980ca0e0 2023-10-30T13:32:25.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231189256 0 0) 0x7f4180003530 con 0x7f41980ca0e0 2023-10-30T13:32:25.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981538a0 msgr2=0x7f419815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:25.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981538a0 0x7f419815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 >> v2:172.21.15.150:3300/0 conn(0x7f41980cf160 msgr2=0x7f41980cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:25.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 --2- 172.21.15.148:0/2368519532 >> v2:172.21.15.150:3300/0 conn(0x7f41980cf160 0x7f41980cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41980cad70 con 0x7f41980ca0e0 2023-10-30T13:32:25.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f418c001680 con 0x7f41980ca0e0 2023-10-30T13:32:25.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 >> v1:172.21.15.148:6789/0 conn(0x7f41980ca0e0 legacy=0x7f419815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:25.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981538a0 0x7f4180003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:25.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:25.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4180004d10 0x7f4180005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:25.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419815dc30 con 0x7f4180003f00 2023-10-30T13:32:25.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41980cb330 con 0x7f4180004d10 2023-10-30T13:32:25.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418000b9c0 con 0x7f41981538a0 2023-10-30T13:32:25.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4195d9b640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:25.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f419659c640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981538a0 0x7f4180003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:25.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4196d9d640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4180004d10 0x7f4180005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:25.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4195d9b640 1 -- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981538a0 msgr2=0x7f4180003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:25.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4195d9b640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981538a0 0x7f4180003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4195d9b640 1 -- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4180004d10 msgr2=0x7f4180005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:25.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4195d9b640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4180004d10 0x7f4180005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.365+0000 7f4195d9b640 1 -- 172.21.15.148:0/2368519532 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4180003530 con 0x7f4180003f00 2023-10-30T13:32:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f4195d9b640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41800047d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c290 tx=0x7f418c01cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1e37a87539bef7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c0a5950 con 0x7f4180003f00 2023-10-30T13:32:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c0a5ab0 con 0x7f4180003f00 2023-10-30T13:32:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f419559a640 1 -- 172.21.15.148:0/2368519532 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c0a5e60 con 0x7f4180003f00 2023-10-30T13:32:25.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f419c8d6640 1 -- 172.21.15.148:0/2368519532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 msgr2=0x7f41800047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:25.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f419c8d6640 1 --2- 172.21.15.148:0/2368519532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41800047d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c290 tx=0x7f418c01cce0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f4196d9d640 1 -- 172.21.15.148:0/2368519532 reap_dead start 2023-10-30T13:32:25.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f419c8d6640 1 -- 172.21.15.148:0/2368519532 shutdown_connections 2023-10-30T13:32:25.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f419c8d6640 1 -- 172.21.15.148:0/2368519532 >> 172.21.15.148:0/2368519532 conn(0x7f419806bfe0 msgr2=0x7f419806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:25.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f419c8d6640 1 -- 172.21.15.148:0/2368519532 shutdown_connections 2023-10-30T13:32:25.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.369+0000 7f419c8d6640 1 -- 172.21.15.148:0/2368519532 wait complete. 2023-10-30T13:32:25.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 Processor -- start 2023-10-30T13:32:25.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 -- start start 2023-10-30T13:32:25.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41981f00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:25.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981e3fd0 0x7f419815b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:25.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41981f3450 0x7f4198149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:25.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4198159430 con 0x7f4180003f00 2023-10-30T13:32:25.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41981efc30 con 0x7f41981f3450 2023-10-30T13:32:25.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419815ac80 con 0x7f41981e3fd0 2023-10-30T13:32:25.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41981f00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:25.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41981f00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59534/0 (socket says 172.21.15.148:59534) 2023-10-30T13:32:25.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419659c640 1 -- 172.21.15.148:0/1697437150 learned_addr learned my addr 172.21.15.148:0/1697437150 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:25.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f4196d9d640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41981f3450 0x7f4198149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:25.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f4195d9b640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981e3fd0 0x7f419815b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:25.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419659c640 1 -- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981e3fd0 msgr2=0x7f419815b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:25.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419659c640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981e3fd0 0x7f419815b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419659c640 1 -- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41981f3450 msgr2=0x7f4198149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:25.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419659c640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41981f3450 0x7f4198149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419659c640 1 -- 172.21.15.148:0/1697437150 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419815af60 con 0x7f4180003f00 2023-10-30T13:32:25.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419659c640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41981f00c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f41880119d0 tx=0x7f4188011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:25.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f4186ffd640 1 -- 172.21.15.148:0/1697437150 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418800faf0 con 0x7f4180003f00 2023-10-30T13:32:25.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f4186ffd640 1 -- 172.21.15.148:0/1697437150 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418800fc50 con 0x7f4180003f00 2023-10-30T13:32:25.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f4186ffd640 1 -- 172.21.15.148:0/1697437150 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188014440 con 0x7f4180003f00 2023-10-30T13:32:25.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419815b240 con 0x7f4180003f00 2023-10-30T13:32:25.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.373+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41981ef8c0 con 0x7f4180003f00 2023-10-30T13:32:25.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.377+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41981efa80 con 0x7f4180003f00 2023-10-30T13:32:25.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.377+0000 7f4186ffd640 1 -- 172.21.15.148:0/1697437150 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41880145a0 con 0x7f4180003f00 2023-10-30T13:32:25.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.377+0000 7f4186ffd640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41780664d0 0x7f4178068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:25.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.377+0000 7f4186ffd640 1 -- 172.21.15.148:0/1697437150 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f4188084b60 con 0x7f4180003f00 2023-10-30T13:32:25.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.377+0000 7f4195d9b640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41780664d0 0x7f4178068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:25.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.377+0000 7f4195d9b640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41780664d0 0x7f4178068990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f418c0aa000 tx=0x7f418c001a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:25.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.381+0000 7f4186ffd640 1 -- 172.21.15.148:0/1697437150 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4188051b50 con 0x7f4180003f00 2023-10-30T13:32:25.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.561+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f419804e170 con 0x7f4180003f00 2023-10-30T13:32:25.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.561+0000 7f4186ffd640 1 -- 172.21.15.148:0/1697437150 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f4188022070 con 0x7f4180003f00 2023-10-30T13:32:25.558 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:25.559 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:22.810872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:clientreplay","state_seq":34,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:25.559 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 96 2023-10-30T13:32:25.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41780664d0 msgr2=0x7f4178068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:25.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41780664d0 0x7f4178068990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f418c0aa000 tx=0x7f418c001a10 comp rx=0 tx=0).stop 2023-10-30T13:32:25.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 msgr2=0x7f41981f00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:25.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41981f00c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f41880119d0 tx=0x7f4188011e90 comp rx=0 tx=0).stop 2023-10-30T13:32:25.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 shutdown_connections 2023-10-30T13:32:25.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41780664d0 0x7f4178068990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41981f3450 0x7f4198149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41981e3fd0 0x7f419815b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 --2- 172.21.15.148:0/1697437150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4180003f00 0x7f41981f00c0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:25.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 >> 172.21.15.148:0/1697437150 conn(0x7f419806bfe0 msgr2=0x7f41980b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:25.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 shutdown_connections 2023-10-30T13:32:25.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:25.565+0000 7f419c8d6640 1 -- 172.21.15.148:0/1697437150 wait complete. 2023-10-30T13:32:25.580 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 96, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 96, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6546": { "addr": "172.21.15.148:6837/3232618434", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3232618434, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3232618434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6546, "incarnation": 93, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 34 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:32:22.810872+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6546 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:32:27.582 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:27.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.733+0000 7fecd73a1640 1 Processor -- start 2023-10-30T13:32:27.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.733+0000 7fecd73a1640 1 -- start start 2023-10-30T13:32:27.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd73a1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fecd00c6b80 0x7fecd00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:27.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd73a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 0x7fecd00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:27.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd73a1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fecd00c8430 0x7fecd013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:27.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd73a1640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd00c0a50 con 0x7fecd00c6b80 2023-10-30T13:32:27.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd73a1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd006f6e0 con 0x7fecd00c8430 2023-10-30T13:32:27.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd73a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd006f9c0 con 0x7fecd00c6f60 2023-10-30T13:32:27.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd513d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fecd00c6b80 0x7fecd00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:27.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd513d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fecd00c6b80 0x7fecd00bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59542/0 (socket says 172.21.15.148:59542) 2023-10-30T13:32:27.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd513d640 1 -- 172.21.15.148:0/4281662681 learned_addr learned my addr 172.21.15.148:0/4281662681 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:27.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd593e640 1 --2- 172.21.15.148:0/4281662681 >> v2:172.21.15.150:3300/0 conn(0x7fecd00c8430 0x7fecd013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:27.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd493c640 1 --2- 172.21.15.148:0/4281662681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 0x7fecd00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:27.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd513d640 1 -- 172.21.15.148:0/4281662681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 msgr2=0x7fecd00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:27.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd513d640 1 --2- 172.21.15.148:0/4281662681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 0x7fecd00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd513d640 1 -- 172.21.15.148:0/4281662681 >> v2:172.21.15.150:3300/0 conn(0x7fecd00c8430 msgr2=0x7fecd013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:27.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd513d640 1 --2- 172.21.15.148:0/4281662681 >> v2:172.21.15.150:3300/0 conn(0x7fecd00c8430 0x7fecd013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd513d640 1 -- 172.21.15.148:0/4281662681 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd006fca0 con 0x7fecd00c6b80 2023-10-30T13:32:27.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd493c640 1 --2- 172.21.15.148:0/4281662681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 0x7fecd00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:32:27.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd593e640 1 --2- 172.21.15.148:0/4281662681 >> v2:172.21.15.150:3300/0 conn(0x7fecd00c8430 0x7fecd013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:32:27.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.737+0000 7fecd513d640 1 --2- 172.21.15.148:0/4281662681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 0x7fecd00bfec0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fecd00b0860 tx=0x7feccc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639883bdd581bbae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:27.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecc7fff640 1 -- 172.21.15.148:0/4281662681 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc0ab600 con 0x7fecd00c6b80 2023-10-30T13:32:27.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecc7fff640 1 -- 172.21.15.148:0/4281662681 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feccc0a7050 con 0x7fecd00c6b80 2023-10-30T13:32:27.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecc7fff640 1 -- 172.21.15.148:0/4281662681 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc0ab980 con 0x7fecd00c6b80 2023-10-30T13:32:27.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 -- 172.21.15.148:0/4281662681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 msgr2=0x7fecd00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:27.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 --2- 172.21.15.148:0/4281662681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 0x7fecd00bfec0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fecd00b0860 tx=0x7feccc01c470 comp rx=0 tx=0).stop 2023-10-30T13:32:27.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 -- 172.21.15.148:0/4281662681 shutdown_connections 2023-10-30T13:32:27.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 --2- 172.21.15.148:0/4281662681 >> v2:172.21.15.150:3300/0 conn(0x7fecd00c8430 0x7fecd013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 --2- 172.21.15.148:0/4281662681 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 0x7fecd00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 --2- 172.21.15.148:0/4281662681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 0x7fecd00bfec0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 -- 172.21.15.148:0/4281662681 >> 172.21.15.148:0/4281662681 conn(0x7fecd00c7910 msgr2=0x7fecd00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:27.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 -- 172.21.15.148:0/4281662681 shutdown_connections 2023-10-30T13:32:27.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 -- 172.21.15.148:0/4281662681 wait complete. 2023-10-30T13:32:27.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 Processor -- start 2023-10-30T13:32:27.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 -- start start 2023-10-30T13:32:27.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 0x7fecd015f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:27.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 0x7fecd013e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:27.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd00c8430 0x7fecd013a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:27.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd015f710 con 0x7fecd00c6b80 2023-10-30T13:32:27.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd013a5b0 con 0x7fecd00c8430 2023-10-30T13:32:27.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd73a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd013d5b0 con 0x7fecd00c6f60 2023-10-30T13:32:27.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd513d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 0x7fecd015f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:27.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd513d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 0x7fecd015f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59550/0 (socket says 172.21.15.148:59550) 2023-10-30T13:32:27.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.741+0000 7fecd513d640 1 -- 172.21.15.148:0/683366924 learned_addr learned my addr 172.21.15.148:0/683366924 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:27.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd493c640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 0x7fecd013e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:27.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd593e640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd00c8430 0x7fecd013a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:27.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd513d640 1 -- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 msgr2=0x7fecd013e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:27.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd513d640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 0x7fecd013e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd513d640 1 -- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd00c8430 msgr2=0x7fecd013a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:27.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd513d640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd00c8430 0x7fecd013a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd513d640 1 -- 172.21.15.148:0/683366924 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc0094d0 con 0x7fecd00c6b80 2023-10-30T13:32:27.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd513d640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 0x7fecd015f1d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7feccc01ab20 tx=0x7feccc019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:27.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecc5ffb640 1 -- 172.21.15.148:0/683366924 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc0b4420 con 0x7fecd00c6b80 2023-10-30T13:32:27.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecc5ffb640 1 -- 172.21.15.148:0/683366924 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feccc0b4580 con 0x7fecd00c6b80 2023-10-30T13:32:27.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecc5ffb640 1 -- 172.21.15.148:0/683366924 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc0b48f0 con 0x7fecd00c6b80 2023-10-30T13:32:27.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd013c9a0 con 0x7fecd00c6b80 2023-10-30T13:32:27.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd013cf60 con 0x7fecd00c6b80 2023-10-30T13:32:27.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd004e170 con 0x7fecd00c6b80 2023-10-30T13:32:27.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecc5ffb640 1 -- 172.21.15.148:0/683366924 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feccc0b2070 con 0x7fecd00c6b80 2023-10-30T13:32:27.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.745+0000 7fecc5ffb640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecb40664a0 0x7fecb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:27.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.749+0000 7fecc5ffb640 1 -- 172.21.15.148:0/683366924 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7feccc11d8e0 con 0x7fecd00c6b80 2023-10-30T13:32:27.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.749+0000 7fecd493c640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecb40664a0 0x7fecb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:27.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.749+0000 7fecd493c640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecb40664a0 0x7fecb4068960 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fecd0138580 tx=0x7fecc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:27.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.753+0000 7fecc5ffb640 1 -- 172.21.15.148:0/683366924 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feccc0ea930 con 0x7fecd00c6b80 2023-10-30T13:32:27.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.933+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecd0160660 con 0x7fecd00c6b80 2023-10-30T13:32:27.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.933+0000 7fecc5ffb640 1 -- 172.21.15.148:0/683366924 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5723 (secure 0 0 0) 0x7feccc0b0090 con 0x7fecd00c6b80 2023-10-30T13:32:27.929 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:27.929 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:22.810872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:clientreplay","state_seq":34,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:27.930 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 96 2023-10-30T13:32:27.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecb40664a0 msgr2=0x7fecb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:27.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecb40664a0 0x7fecb4068960 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fecd0138580 tx=0x7fecc0008040 comp rx=0 tx=0).stop 2023-10-30T13:32:27.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 msgr2=0x7fecd015f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:27.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 0x7fecd015f1d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7feccc01ab20 tx=0x7feccc019210 comp rx=0 tx=0).stop 2023-10-30T13:32:27.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 shutdown_connections 2023-10-30T13:32:27.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fecb40664a0 0x7fecb4068960 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecd00c8430 0x7fecd013a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fecd00c6f60 0x7fecd013e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 --2- 172.21.15.148:0/683366924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fecd00c6b80 0x7fecd015f1d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:27.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 >> 172.21.15.148:0/683366924 conn(0x7fecd00c7910 msgr2=0x7fecd00c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:27.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 shutdown_connections 2023-10-30T13:32:27.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:27.937+0000 7fecd73a1640 1 -- 172.21.15.148:0/683366924 wait complete. 2023-10-30T13:32:27.950 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:29.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:32:29.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:32:29.950 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.101+0000 7f2d26e46640 1 Processor -- start 2023-10-30T13:32:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.101+0000 7f2d26e46640 1 -- start start 2023-10-30T13:32:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d26e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6b80 0x7f2d200bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d26e46640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2d2013f4a0 0x7f2d200c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d26e46640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d200c0a50 con 0x7f2d200c6f60 2023-10-30T13:32:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d26e46640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2006f6e0 con 0x7f2d2013f4a0 2023-10-30T13:32:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d26e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2006f9c0 con 0x7f2d200c6b80 2023-10-30T13:32:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d253e3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2d2013f4a0 0x7f2d200c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d17fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2d200c6f60 0x7f2d2013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44812/0 (socket says 172.21.15.148:44812) 2023-10-30T13:32:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d17fff640 1 -- 172.21.15.148:0/1032736166 learned_addr learned my addr 172.21.15.148:0/1032736166 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d24be2640 1 --2- 172.21.15.148:0/1032736166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6b80 0x7f2d200bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d253e3640 1 -- 172.21.15.148:0/1032736166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6b80 msgr2=0x7f2d200bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d253e3640 1 --2- 172.21.15.148:0/1032736166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6b80 0x7f2d200bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d253e3640 1 -- 172.21.15.148:0/1032736166 >> v1:172.21.15.148:6789/0 conn(0x7f2d200c6f60 legacy=0x7f2d2013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d253e3640 1 -- 172.21.15.148:0/1032736166 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2006fca0 con 0x7f2d2013f4a0 2023-10-30T13:32:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.105+0000 7f2d177fe640 1 -- 172.21.15.148:0/1032736166 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587208179 0 0) 0x7f2d200c0a50 con 0x7f2d200c6f60 2023-10-30T13:32:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d253e3640 1 --2- 172.21.15.148:0/1032736166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d2013f4a0 0x7f2d200c0400 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2d1800bfa0 tx=0x7f2d1801eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a7441b9c34694b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d177fe640 1 -- 172.21.15.148:0/1032736166 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1800b2d0 con 0x7f2d2013f4a0 2023-10-30T13:32:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d177fe640 1 -- 172.21.15.148:0/1032736166 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d180a7930 con 0x7f2d2013f4a0 2023-10-30T13:32:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d177fe640 1 -- 172.21.15.148:0/1032736166 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d180a7cc0 con 0x7f2d2013f4a0 2023-10-30T13:32:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 -- 172.21.15.148:0/1032736166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d2013f4a0 msgr2=0x7f2d200c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 --2- 172.21.15.148:0/1032736166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d2013f4a0 0x7f2d200c0400 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2d1800bfa0 tx=0x7f2d1801eb50 comp rx=0 tx=0).stop 2023-10-30T13:32:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 -- 172.21.15.148:0/1032736166 shutdown_connections 2023-10-30T13:32:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 --2- 172.21.15.148:0/1032736166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d2013f4a0 0x7f2d200c0400 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 --2- 172.21.15.148:0/1032736166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6b80 0x7f2d200bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 -- 172.21.15.148:0/1032736166 >> 172.21.15.148:0/1032736166 conn(0x7f2d200c7910 msgr2=0x7f2d200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 -- 172.21.15.148:0/1032736166 shutdown_connections 2023-10-30T13:32:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 -- 172.21.15.148:0/1032736166 wait complete. 2023-10-30T13:32:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 Processor -- start 2023-10-30T13:32:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.109+0000 7f2d26e46640 1 -- start start 2023-10-30T13:32:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d26e46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d200c6b80 0x7f2d20158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d26e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6f60 0x7f2d20154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d26e46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d2013f4a0 0x7f2d20155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d26e46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d20155d10 con 0x7f2d200c6b80 2023-10-30T13:32:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d26e46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2013df10 con 0x7f2d2013f4a0 2023-10-30T13:32:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d26e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d201546b0 con 0x7f2d200c6f60 2023-10-30T13:32:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d24be2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d200c6b80 0x7f2d20158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d24be2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d200c6b80 0x7f2d20158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59576/0 (socket says 172.21.15.148:59576) 2023-10-30T13:32:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d24be2640 1 -- 172.21.15.148:0/4169731806 learned_addr learned my addr 172.21.15.148:0/4169731806 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d17fff640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6f60 0x7f2d20154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d253e3640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d2013f4a0 0x7f2d20155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d24be2640 1 -- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6f60 msgr2=0x7f2d20154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d24be2640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6f60 0x7f2d20154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d24be2640 1 -- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d2013f4a0 msgr2=0x7f2d20155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d24be2640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d2013f4a0 0x7f2d20155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d24be2640 1 -- 172.21.15.148:0/4169731806 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1800bc80 con 0x7f2d200c6b80 2023-10-30T13:32:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d24be2640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d200c6b80 0x7f2d20158c50 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00d800 tx=0x7f2d1c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d157fa640 1 -- 172.21.15.148:0/4169731806 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00b6c0 con 0x7f2d200c6b80 2023-10-30T13:32:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d157fa640 1 -- 172.21.15.148:0/4169731806 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c00b820 con 0x7f2d200c6b80 2023-10-30T13:32:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d157fa640 1 -- 172.21.15.148:0/4169731806 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c00bb90 con 0x7f2d200c6b80 2023-10-30T13:32:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2013e160 con 0x7f2d200c6b80 2023-10-30T13:32:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d201f2b70 con 0x7f2d200c6b80 2023-10-30T13:32:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d201f2d90 con 0x7f2d200c6b80 2023-10-30T13:32:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.113+0000 7f2d157fa640 1 -- 172.21.15.148:0/4169731806 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2d1c00bcf0 con 0x7f2d200c6b80 2023-10-30T13:32:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.117+0000 7f2d157fa640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d0c0664a0 0x7f2d0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.117+0000 7f2d157fa640 1 -- 172.21.15.148:0/4169731806 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f2d1c084900 con 0x7f2d200c6b80 2023-10-30T13:32:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.117+0000 7f2d17fff640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d0c0664a0 0x7f2d0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.117+0000 7f2d17fff640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d0c0664a0 0x7f2d0c068960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2d2006a4b0 tx=0x7f2d10007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.121+0000 7f2d157fa640 1 -- 172.21.15.148:0/4169731806 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2d1c0518c0 con 0x7f2d200c6b80 2023-10-30T13:32:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.297+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d201544a0 con 0x7f2d200c6b80 2023-10-30T13:32:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.297+0000 7f2d157fa640 1 -- 172.21.15.148:0/4169731806 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5723 (secure 0 0 0) 0x7f2d1c054ee0 con 0x7f2d200c6b80 2023-10-30T13:32:30.294 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:30.294 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:22.810872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:clientreplay","state_seq":34,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:30.295 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 96 2023-10-30T13:32:30.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d0c0664a0 msgr2=0x7f2d0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:30.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d0c0664a0 0x7f2d0c068960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2d2006a4b0 tx=0x7f2d10007b60 comp rx=0 tx=0).stop 2023-10-30T13:32:30.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d200c6b80 msgr2=0x7f2d20158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:30.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d200c6b80 0x7f2d20158c50 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00d800 tx=0x7f2d1c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 shutdown_connections 2023-10-30T13:32:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d0c0664a0 0x7f2d0c068960 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d2013f4a0 0x7f2d20155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2d200c6f60 0x7f2d20154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 --2- 172.21.15.148:0/4169731806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d200c6b80 0x7f2d20158c50 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 >> 172.21.15.148:0/4169731806 conn(0x7f2d200c7910 msgr2=0x7f2d200c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:30.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 shutdown_connections 2023-10-30T13:32:30.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:30.301+0000 7f2d26e46640 1 -- 172.21.15.148:0/4169731806 wait complete. 2023-10-30T13:32:30.314 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:32:32.315 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:32.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.469+0000 7f13bb8c9640 1 Processor -- start 2023-10-30T13:32:32.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.469+0000 7f13bb8c9640 1 -- start start 2023-10-30T13:32:32.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.469+0000 7f13bb8c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f13b41539a0 0x7f13b415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:32.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.469+0000 7f13bb8c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b415a8c0 con 0x7f13b4069e90 2023-10-30T13:32:32.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.469+0000 7f13bb8c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b4153d80 con 0x7f13b4070a00 2023-10-30T13:32:32.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.469+0000 7f13bb8c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b406fbb0 con 0x7f13b41539a0 2023-10-30T13:32:32.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.469+0000 7f13b9e66640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f13b4069e90 0x7f13b406a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:35138/0 (socket says 172.21.15.148:35138) 2023-10-30T13:32:32.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.469+0000 7f13b9e66640 1 -- 172.21.15.148:0/1466406286 learned_addr learned my addr 172.21.15.148:0/1466406286 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:32.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.469+0000 7f13b9665640 1 --2- 172.21.15.148:0/1466406286 >> v2:172.21.15.150:3300/0 conn(0x7f13b41539a0 0x7f13b415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103515824 0 0) 0x7f13b415a8c0 con 0x7f13b4069e90 2023-10-30T13:32:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a0003530 con 0x7f13b4069e90 2023-10-30T13:32:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632075936 0 0) 0x7f13a0003530 con 0x7f13b4069e90 2023-10-30T13:32:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 >> v2:172.21.15.150:3300/0 conn(0x7f13b41539a0 msgr2=0x7f13b415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 --2- 172.21.15.148:0/1466406286 >> v2:172.21.15.150:3300/0 conn(0x7f13b41539a0 0x7f13b415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 >> v1:172.21.15.150:6789/0 conn(0x7f13b4070a00 legacy=0x7f13b4070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:32.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b40c5c50 con 0x7f13b4069e90 2023-10-30T13:32:32.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f13ac001a00 con 0x7f13b4069e90 2023-10-30T13:32:32.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 >> v1:172.21.15.148:6789/0 conn(0x7f13b4069e90 legacy=0x7f13b406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:32.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13a0003980 0x7f13a0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:32.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13a0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:32.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0005190 0x7f13a000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:32.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b406fbb0 con 0x7f13a00047d0 2023-10-30T13:32:32.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a000c130 con 0x7f13a0005190 2023-10-30T13:32:32.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a000c2f0 con 0x7f13a0003980 2023-10-30T13:32:32.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b8e64640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13a0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:32.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b9665640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13a0003980 0x7f13a0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:32.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b9e66640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0005190 0x7f13a000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:32.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b8e64640 1 -- 172.21.15.148:0/1466406286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13a0003980 msgr2=0x7f13a0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:32.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b8e64640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13a0003980 0x7f13a0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b8e64640 1 -- 172.21.15.148:0/1466406286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0005190 msgr2=0x7f13a000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:32.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b8e64640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0005190 0x7f13a000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b8e64640 1 -- 172.21.15.148:0/1466406286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0003530 con 0x7f13a00047d0 2023-10-30T13:32:32.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b9e66640 1 -- 172.21.15.148:0/1466406286 reap_dead start 2023-10-30T13:32:32.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.473+0000 7f13b8e64640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13a0004c50 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f13a4001780 tx=0x7f13a401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc91b7ad933ade0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:32.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a40052d0 con 0x7f13a00047d0 2023-10-30T13:32:32.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a40a7050 con 0x7f13a00047d0 2023-10-30T13:32:32.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13abfff640 1 -- 172.21.15.148:0/1466406286 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a4003600 con 0x7f13a00047d0 2023-10-30T13:32:32.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 -- 172.21.15.148:0/1466406286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 msgr2=0x7f13a0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:32.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13a0004c50 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f13a4001780 tx=0x7f13a401c3b0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 -- 172.21.15.148:0/1466406286 shutdown_connections 2023-10-30T13:32:32.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 --2- 172.21.15.148:0/1466406286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13a0004c50 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 -- 172.21.15.148:0/1466406286 >> 172.21.15.148:0/1466406286 conn(0x7f13b40c7910 msgr2=0x7f13b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:32.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 -- 172.21.15.148:0/1466406286 shutdown_connections 2023-10-30T13:32:32.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 -- 172.21.15.148:0/1466406286 wait complete. 2023-10-30T13:32:32.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 Processor -- start 2023-10-30T13:32:32.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 -- start start 2023-10-30T13:32:32.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13b41541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:32.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b4156d40 0x7f13b40c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:32.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13b4157920 0x7f13b4157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:32.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4154aa0 con 0x7f13a00047d0 2023-10-30T13:32:32.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b41f3630 con 0x7f13b4156d40 2023-10-30T13:32:32.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13bb8c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b41f2a80 con 0x7f13b4157920 2023-10-30T13:32:32.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.477+0000 7f13b9665640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13b41541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:32.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b9665640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13b41541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36156/0 (socket says 172.21.15.148:36156) 2023-10-30T13:32:32.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b9665640 1 -- 172.21.15.148:0/982082920 learned_addr learned my addr 172.21.15.148:0/982082920 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:32.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b8e64640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b4156d40 0x7f13b40c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:32.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b9e66640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13b4157920 0x7f13b4157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:32.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b9665640 1 -- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13b4157920 msgr2=0x7f13b4157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:32.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b9665640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13b4157920 0x7f13b4157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b9665640 1 -- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b4156d40 msgr2=0x7f13b40c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:32.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b9665640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b4156d40 0x7f13b40c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b9665640 1 -- 172.21.15.148:0/982082920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b41582e0 con 0x7f13a00047d0 2023-10-30T13:32:32.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13b9665640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13b41541f0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f13b0011500 tx=0x7f13b00119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:32.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13a9ffb640 1 -- 172.21.15.148:0/982082920 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0015870 con 0x7f13a00047d0 2023-10-30T13:32:32.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13a9ffb640 1 -- 172.21.15.148:0/982082920 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b00159d0 con 0x7f13a00047d0 2023-10-30T13:32:32.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13a9ffb640 1 -- 172.21.15.148:0/982082920 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0015d40 con 0x7f13a00047d0 2023-10-30T13:32:32.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b41f2d00 con 0x7f13a00047d0 2023-10-30T13:32:32.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b41f32c0 con 0x7f13a00047d0 2023-10-30T13:32:32.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b4157770 con 0x7f13a00047d0 2023-10-30T13:32:32.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.481+0000 7f13a9ffb640 1 -- 172.21.15.148:0/982082920 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13b0013400 con 0x7f13a00047d0 2023-10-30T13:32:32.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.485+0000 7f13a9ffb640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13980664a0 0x7f1398068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:32.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.485+0000 7f13a9ffb640 1 -- 172.21.15.148:0/982082920 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f13b0084c10 con 0x7f13a00047d0 2023-10-30T13:32:32.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.485+0000 7f13b8e64640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13980664a0 0x7f1398068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:32.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.485+0000 7f13b8e64640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13980664a0 0x7f1398068960 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f13a4001750 tx=0x7f13a40011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:32.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.485+0000 7f13a9ffb640 1 -- 172.21.15.148:0/982082920 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13b0051b50 con 0x7f13a00047d0 2023-10-30T13:32:32.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.665+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13b404e170 con 0x7f13a00047d0 2023-10-30T13:32:32.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13a9ffb640 1 -- 172.21.15.148:0/982082920 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5717 (secure 0 0 0) 0x7f13b0055170 con 0x7f13a00047d0 2023-10-30T13:32:32.663 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:32.663 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:32.663 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 97 2023-10-30T13:32:32.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13980664a0 msgr2=0x7f1398068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:32.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13980664a0 0x7f1398068960 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f13a4001750 tx=0x7f13a40011a0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 msgr2=0x7f13b41541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:32.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13b41541f0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f13b0011500 tx=0x7f13b00119c0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 shutdown_connections 2023-10-30T13:32:32.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13980664a0 0x7f1398068960 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13b4157920 0x7f13b4157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b4156d40 0x7f13b40c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 --2- 172.21.15.148:0/982082920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13a00047d0 0x7f13b41541f0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:32.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 >> 172.21.15.148:0/982082920 conn(0x7f13b40c7910 msgr2=0x7f13b41e92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:32.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 shutdown_connections 2023-10-30T13:32:32.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:32.669+0000 7f13bb8c9640 1 -- 172.21.15.148:0/982082920 wait complete. 2023-10-30T13:32:32.683 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=6408 2023-10-30T13:32:32.683 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2023-10-30T13:32:36.626 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 33% (75/222) Updating files: 34% (76/222) Updating files: 35% (78/222) Updating files: 36% (80/222) Updating files: 37% (83/222) Updating files: 38% (85/222) Updating files: 39% (87/222) Updating files: 40% (89/222) Updating files: 41% (92/222) Updating files: 42% (94/222) Updating files: 43% (96/222) Updating files: 44% (98/222) Updating files: 45% (100/222) Updating files: 46% (103/222) Updating files: 47% (105/222) Updating files: 48% (107/222) Updating files: 49% (109/222) Updating files: 50% (111/222) Updating files: 51% (114/222) Updating files: 51% (115/222) Updating files: 52% (116/222) Updating files: 53% (118/222) Updating files: 54% (120/222) Updating files: 55% (123/222) Updating files: 56% (125/222) Updating files: 57% (127/222) Updating files: 58% (129/222) Updating files: 59% (131/222) Updating files: 60% (134/222) Updating files: 61% (136/222) Updating files: 62% (138/222) Updating files: 63% (140/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2023-10-30T13:32:36.781 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 2b5f76f48e9 readme 2023-10-30T13:32:36.786 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.2 2023-10-30T13:32:36.803 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 3 -eq 48 ] 2023-10-30T13:32:36.803 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.3 2023-10-30T13:32:36.803 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.3 2023-10-30T13:32:36.804 INFO:tasks.workunit.client.0.smithi148.stdout:v0.3 2023-10-30T13:32:36.804 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.3 2023-10-30T13:32:38.716 INFO:tasks.ceph.mds.c.smithi148.stderr:2023-10-30T13:32:38.721+0000 7f5970895640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:32:40.685 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:32:40.685 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:32:40.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:32:40.689 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:32:40.689 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:32:40.777 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:32:40.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f888b69e640 1 Processor -- start 2023-10-30T13:32:40.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f888b69e640 1 -- start start 2023-10-30T13:32:40.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f888b69e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f88840c6b80 0x7f88840bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:40.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f888b69e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f88840c6f60 0x7f88840c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:40.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f888b69e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88840c0a50 con 0x7f88840c8430 2023-10-30T13:32:40.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f888b69e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888406f6e0 con 0x7f88840c6b80 2023-10-30T13:32:40.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f888b69e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888406f9c0 con 0x7f88840c6f60 2023-10-30T13:32:40.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f888943a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f88840c6b80 0x7f88840bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:40.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f8889c3b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f88840c8430 0x7f888413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:35166/0 (socket says 172.21.15.148:35166) 2023-10-30T13:32:40.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f888943a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f88840c6b80 0x7f88840bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36172/0 (socket says 172.21.15.148:36172) 2023-10-30T13:32:40.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f8889c3b640 1 -- 172.21.15.148:0/2288529803 learned_addr learned my addr 172.21.15.148:0/2288529803 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:32:40.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.841+0000 7f8888c39640 1 --2- 172.21.15.148:0/2288529803 >> v2:172.21.15.150:3300/0 conn(0x7f88840c6f60 0x7f88840c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:40.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f888943a640 1 -- 172.21.15.148:0/2288529803 >> v2:172.21.15.150:3300/0 conn(0x7f88840c6f60 msgr2=0x7f88840c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:32:40.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f888943a640 1 --2- 172.21.15.148:0/2288529803 >> v2:172.21.15.150:3300/0 conn(0x7f88840c6f60 0x7f88840c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:40.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f888943a640 1 -- 172.21.15.148:0/2288529803 >> v1:172.21.15.148:6789/0 conn(0x7f88840c8430 legacy=0x7f888413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:40.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f888943a640 1 -- 172.21.15.148:0/2288529803 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888406fca0 con 0x7f88840c6b80 2023-10-30T13:32:40.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f887bfff640 1 -- 172.21.15.148:0/2288529803 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880632768 0 0) 0x7f88840c0a50 con 0x7f88840c8430 2023-10-30T13:32:40.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f8888c39640 1 --2- 172.21.15.148:0/2288529803 >> v2:172.21.15.150:3300/0 conn(0x7f88840c6f60 0x7f88840c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:32:40.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f888943a640 1 --2- 172.21.15.148:0/2288529803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c6b80 0x7f88840bfec0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f8884159b90 tx=0x7f888001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ba42e3c6136bb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:40.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f887bfff640 1 -- 172.21.15.148:0/2288529803 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800ab600 con 0x7f88840c6b80 2023-10-30T13:32:40.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f887bfff640 1 -- 172.21.15.148:0/2288529803 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88800a7050 con 0x7f88840c6b80 2023-10-30T13:32:40.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.845+0000 7f887bfff640 1 -- 172.21.15.148:0/2288529803 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800ab980 con 0x7f88840c6b80 2023-10-30T13:32:40.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 -- 172.21.15.148:0/2288529803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c6b80 msgr2=0x7f88840bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:40.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 --2- 172.21.15.148:0/2288529803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c6b80 0x7f88840bfec0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f8884159b90 tx=0x7f888001c460 comp rx=0 tx=0).stop 2023-10-30T13:32:40.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 -- 172.21.15.148:0/2288529803 shutdown_connections 2023-10-30T13:32:40.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 --2- 172.21.15.148:0/2288529803 >> v2:172.21.15.150:3300/0 conn(0x7f88840c6f60 0x7f88840c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:40.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 --2- 172.21.15.148:0/2288529803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c6b80 0x7f88840bfec0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:40.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 -- 172.21.15.148:0/2288529803 >> 172.21.15.148:0/2288529803 conn(0x7f88840c7910 msgr2=0x7f88840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:40.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 -- 172.21.15.148:0/2288529803 shutdown_connections 2023-10-30T13:32:40.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 -- 172.21.15.148:0/2288529803 wait complete. 2023-10-30T13:32:40.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 Processor -- start 2023-10-30T13:32:40.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.849+0000 7f888b69e640 1 -- start start 2023-10-30T13:32:40.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888b69e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88840c6b80 0x7f8884138620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:40.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888b69e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6f60 0x7f8884138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:40.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888b69e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c8430 0x7f888413d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:40.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888b69e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8884159950 con 0x7f88840c8430 2023-10-30T13:32:40.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888b69e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888413caa0 con 0x7f88840c6b80 2023-10-30T13:32:40.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888b69e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88841f2310 con 0x7f88840c6f60 2023-10-30T13:32:40.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8889c3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c8430 0x7f888413d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:40.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8889c3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c8430 0x7f888413d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36182/0 (socket says 172.21.15.148:36182) 2023-10-30T13:32:40.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8889c3b640 1 -- 172.21.15.148:0/1320957968 learned_addr learned my addr 172.21.15.148:0/1320957968 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:32:40.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8888c39640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6f60 0x7f8884138b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:40.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888943a640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88840c6b80 0x7f8884138620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:40.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8889c3b640 1 -- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6f60 msgr2=0x7f8884138b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:40.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8889c3b640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6f60 0x7f8884138b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8889c3b640 1 -- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88840c6b80 msgr2=0x7f8884138620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8889c3b640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88840c6b80 0x7f8884138620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8889c3b640 1 -- 172.21.15.148:0/1320957968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8880009420 con 0x7f88840c8430 2023-10-30T13:32:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8889c3b640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c8430 0x7f888413d880 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f887c00c780 tx=0x7f887c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8879ffb640 1 -- 172.21.15.148:0/1320957968 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c00ce90 con 0x7f88840c8430 2023-10-30T13:32:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8879ffb640 1 -- 172.21.15.148:0/1320957968 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c00a640 con 0x7f88840c8430 2023-10-30T13:32:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888413cd40 con 0x7f88840c8430 2023-10-30T13:32:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f8879ffb640 1 -- 172.21.15.148:0/1320957968 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c00a9d0 con 0x7f88840c8430 2023-10-30T13:32:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8884153e20 con 0x7f88840c8430 2023-10-30T13:32:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.853+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888404e170 con 0x7f88840c8430 2023-10-30T13:32:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.857+0000 7f8879ffb640 1 -- 172.21.15.148:0/1320957968 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f887c00ab30 con 0x7f88840c8430 2023-10-30T13:32:40.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.857+0000 7f8879ffb640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664a0 0x7f8868068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:32:40.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.857+0000 7f8879ffb640 1 -- 172.21.15.148:0/1320957968 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f887c083a40 con 0x7f88840c8430 2023-10-30T13:32:40.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.857+0000 7f888943a640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664a0 0x7f8868068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:32:40.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.857+0000 7f888943a640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664a0 0x7f8868068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f88800ad000 tx=0x7f8880008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:32:40.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:40.861+0000 7f8879ffb640 1 -- 172.21.15.148:0/1320957968 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f887c085e60 con 0x7f88840c8430 2023-10-30T13:32:41.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.081+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8884154070 con 0x7f88840c8430 2023-10-30T13:32:41.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.081+0000 7f8879ffb640 1 -- 172.21.15.148:0/1320957968 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f887c018020 con 0x7f88840c8430 2023-10-30T13:32:41.078 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:32:41.078 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:32:41.079 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 98 2023-10-30T13:32:41.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664a0 msgr2=0x7f8868068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:41.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664a0 0x7f8868068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f88800ad000 tx=0x7f8880008040 comp rx=0 tx=0).stop 2023-10-30T13:32:41.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c8430 msgr2=0x7f888413d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:32:41.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c8430 0x7f888413d880 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f887c00c780 tx=0x7f887c00cc40 comp rx=0 tx=0).stop 2023-10-30T13:32:41.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 shutdown_connections 2023-10-30T13:32:41.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664a0 0x7f8868068960 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:41.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88840c8430 0x7f888413d880 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6f60 0x7f8884138b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 --2- 172.21.15.148:0/1320957968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88840c6b80 0x7f8884138620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:32:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 >> 172.21.15.148:0/1320957968 conn(0x7f88840c7910 msgr2=0x7f88840b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:32:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 shutdown_connections 2023-10-30T13:32:41.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:32:41.085+0000 7f888b69e640 1 -- 172.21.15.148:0/1320957968 wait complete. 2023-10-30T13:32:41.100 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:32:41.101 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 57 secs before thrashing 2023-10-30T13:32:43.642 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 36% (109/295) Updating files: 37% (110/295) Updating files: 38% (113/295) Updating files: 39% (116/295) Updating files: 40% (118/295) Updating files: 41% (121/295) Updating files: 42% (124/295) Updating files: 43% (127/295) Updating files: 44% (130/295) Updating files: 45% (133/295) Updating files: 46% (136/295) Updating files: 47% (139/295) Updating files: 48% (142/295) Updating files: 49% (145/295) Updating files: 50% (148/295) Updating files: 51% (151/295) Updating files: 52% (154/295) Updating files: 53% (157/295) Updating files: 54% (160/295) Updating files: 55% (163/295) Updating files: 56% (166/295) Updating files: 56% (168/295) Updating files: 57% (169/295) Updating files: 58% (172/295) Updating files: 59% (175/295) Updating files: 60% (177/295) Updating files: 61% (180/295) Updating files: 62% (183/295) Updating files: 63% (186/295) Updating files: 64% (189/295) Updating files: 65% (192/295) Updating files: 66% (195/295) Updating files: 67% (198/295) Updating files: 68% (201/295) Updating files: 69% (204/295) Updating files: 70% (207/295) Updating files: 71% (210/295) Updating files: 72% (213/295) Updating files: 73% (216/295) Updating files: 74% (219/295) Updating files: 75% (222/295) Updating files: 76% (225/295) Updating files: 77% (228/295) Updating files: 78% (231/295) Updating files: 79% (234/295) Updating files: 80% (236/295) Updating files: 81% (239/295) Updating files: 82% (242/295) Updating files: 83% (245/295) Updating files: 84% (248/295) Updating files: 85% (251/295) Updating files: 86% (254/295) Updating files: 87% (257/295) Updating files: 88% (260/295) Updating files: 89% (263/295) Updating files: 90% (266/295) Updating files: 91% (269/295) Updating files: 92% (272/295) Updating files: 93% (275/295) Updating files: 94% (278/295) Updating files: 95% (281/295) Updating files: 96% (284/295) Updating files: 97% (287/295) Updating files: 98% (290/295) Updating files: 99% (293/295) Updating files: 100% (295/295) Updating files: 100% (295/295), done. 2023-10-30T13:32:43.657 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2023-10-30T13:32:43.666 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.3 2023-10-30T13:32:43.832 INFO:tasks.workunit.client.0.smithi148.stdout:v0.4 2023-10-30T13:32:43.832 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 4 -eq 48 ] 2023-10-30T13:32:43.833 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.4 2023-10-30T13:32:43.833 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.4 2023-10-30T13:32:43.833 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.4 2023-10-30T13:32:47.496 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 42% (104/247) Updating files: 43% (107/247) Updating files: 44% (109/247) Updating files: 45% (112/247) Updating files: 46% (114/247) Updating files: 47% (117/247) Updating files: 48% (119/247) Updating files: 49% (122/247) Updating files: 50% (124/247) Updating files: 51% (126/247) Updating files: 52% (129/247) Updating files: 53% (131/247) Updating files: 54% (134/247) Updating files: 55% (136/247) Updating files: 56% (139/247) Updating files: 57% (141/247) Updating files: 58% (144/247) Updating files: 59% (146/247) Updating files: 60% (149/247) Updating files: 61% (151/247) Updating files: 62% (154/247) Updating files: 63% (156/247) Updating files: 64% (159/247) Updating files: 65% (161/247) Updating files: 66% (164/247) Updating files: 67% (166/247) Updating files: 68% (168/247) Updating files: 69% (171/247) Updating files: 70% (173/247) Updating files: 71% (176/247) Updating files: 72% (178/247) Updating files: 73% (181/247) Updating files: 74% (183/247) Updating files: 75% (186/247) Updating files: 76% (188/247) Updating files: 77% (191/247) Updating files: 78% (193/247) Updating files: 79% (196/247) Updating files: 80% (198/247) Updating files: 81% (201/247) Updating files: 82% (203/247) Updating files: 83% (206/247) Updating files: 84% (208/247) Updating files: 85% (210/247) Updating files: 85% (211/247) Updating files: 86% (213/247) Updating files: 87% (215/247) Updating files: 88% (218/247) Updating files: 89% (220/247) Updating files: 90% (223/247) Updating files: 91% (225/247) Updating files: 92% (228/247) Updating files: 93% (230/247) Updating files: 94% (233/247) Updating files: 95% (235/247) Updating files: 96% (238/247) Updating files: 97% (240/247) Updating files: 98% (243/247) Updating files: 99% (245/247) Updating files: 100% (247/247) Updating files: 100% (247/247), done. 2023-10-30T13:32:47.522 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2023-10-30T13:32:47.528 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.4 2023-10-30T13:32:47.800 INFO:tasks.workunit.client.0.smithi148.stdout:v0.5 2023-10-30T13:32:47.800 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 5 -eq 48 ] 2023-10-30T13:32:47.801 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.5 2023-10-30T13:32:47.801 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.5 2023-10-30T13:32:47.801 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.5 2023-10-30T13:32:49.566 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 8% (22/245) Updating files: 9% (23/245) Updating files: 10% (25/245) Updating files: 11% (27/245) Updating files: 12% (30/245) Updating files: 13% (32/245) Updating files: 14% (35/245) Updating files: 15% (37/245) Updating files: 16% (40/245) Updating files: 17% (42/245) Updating files: 18% (45/245) Updating files: 19% (47/245) Updating files: 20% (49/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-30T13:32:49.590 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2023-10-30T13:32:49.596 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.5 2023-10-30T13:32:50.815 INFO:tasks.workunit.client.0.smithi148.stdout:v0.6 2023-10-30T13:32:50.815 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 6 -eq 48 ] 2023-10-30T13:32:50.816 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.6 2023-10-30T13:32:50.816 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.6 2023-10-30T13:32:50.816 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.6 2023-10-30T13:32:59.461 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:32:59.465 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:08.816 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 16% (37/227) Updating files: 17% (39/227) Updating files: 18% (41/227) Updating files: 19% (44/227) Updating files: 20% (46/227) Updating files: 21% (48/227) Updating files: 22% (50/227) Updating files: 23% (53/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 27% (63/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-10-30T13:33:08.848 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at a4c752df713 v0.6 2023-10-30T13:33:08.854 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.6 2023-10-30T13:33:09.165 INFO:tasks.workunit.client.0.smithi148.stdout:v0.7 2023-10-30T13:33:09.166 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 7 -eq 48 ] 2023-10-30T13:33:09.166 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.7 2023-10-30T13:33:09.166 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.7 2023-10-30T13:33:09.166 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.7 2023-10-30T13:33:11.887 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 26% (54/201) Updating files: 27% (55/201) Updating files: 28% (57/201) Updating files: 29% (59/201) Updating files: 30% (61/201) Updating files: 31% (63/201) Updating files: 32% (65/201) Updating files: 33% (67/201) Updating files: 34% (69/201) Updating files: 35% (71/201) Updating files: 36% (73/201) Updating files: 37% (75/201) Updating files: 38% (77/201) Updating files: 39% (79/201) Updating files: 40% (81/201) Updating files: 41% (83/201) Updating files: 42% (85/201) Updating files: 43% (87/201) Updating files: 44% (89/201) Updating files: 45% (91/201) Updating files: 46% (93/201) Updating files: 47% (95/201) Updating files: 48% (97/201) Updating files: 49% (99/201) Updating files: 50% (101/201) Updating files: 51% (103/201) Updating files: 52% (105/201) Updating files: 53% (107/201) Updating files: 54% (109/201) Updating files: 55% (111/201) Updating files: 56% (113/201) Updating files: 57% (115/201) Updating files: 58% (117/201) Updating files: 59% (119/201) Updating files: 60% (121/201) Updating files: 61% (123/201) Updating files: 62% (125/201) Updating files: 63% (127/201) Updating files: 64% (129/201) Updating files: 65% (131/201) Updating files: 66% (133/201) Updating files: 67% (135/201) Updating files: 68% (137/201) Updating files: 69% (139/201) Updating files: 70% (141/201) Updating files: 71% (143/201) Updating files: 72% (145/201) Updating files: 73% (147/201) Updating files: 74% (149/201) Updating files: 75% (151/201) Updating files: 76% (153/201) Updating files: 77% (155/201) Updating files: 78% (157/201) Updating files: 79% (159/201) Updating files: 80% (161/201) Updating files: 81% (163/201) Updating files: 82% (165/201) Updating files: 83% (167/201) Updating files: 84% (169/201) Updating files: 85% (171/201) Updating files: 86% (173/201) Updating files: 87% (175/201) Updating files: 88% (177/201) Updating files: 89% (179/201) Updating files: 90% (181/201) Updating files: 91% (183/201) Updating files: 92% (185/201) Updating files: 93% (187/201) Updating files: 94% (189/201) Updating files: 95% (191/201) Updating files: 96% (193/201) Updating files: 97% (195/201) Updating files: 98% (197/201) Updating files: 99% (199/201) Updating files: 100% (201/201) Updating files: 100% (201/201), done. 2023-10-30T13:33:11.916 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2023-10-30T13:33:11.921 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.7 2023-10-30T13:33:12.793 INFO:tasks.workunit.client.0.smithi148.stdout:v0.8 2023-10-30T13:33:12.793 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 8 -eq 48 ] 2023-10-30T13:33:12.794 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.8 2023-10-30T13:33:12.794 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.8 2023-10-30T13:33:12.794 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.8 2023-10-30T13:33:18.198 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 66% (149/225) Updating files: 67% (151/225) Updating files: 68% (153/225) Updating files: 69% (156/225) Updating files: 70% (158/225) Updating files: 71% (160/225) Updating files: 72% (162/225) Updating files: 73% (165/225) Updating files: 74% (167/225) Updating files: 75% (169/225) Updating files: 76% (171/225) Updating files: 77% (174/225) Updating files: 78% (176/225) Updating files: 79% (178/225) Updating files: 80% (180/225) Updating files: 81% (183/225) Updating files: 82% (185/225) Updating files: 83% (187/225) Updating files: 84% (189/225) Updating files: 85% (192/225) Updating files: 86% (194/225) Updating files: 87% (196/225) Updating files: 88% (198/225) Updating files: 89% (201/225) Updating files: 90% (203/225) Updating files: 91% (205/225) Updating files: 92% (207/225) Updating files: 93% (210/225) Updating files: 94% (212/225) Updating files: 95% (214/225) Updating files: 96% (216/225) Updating files: 97% (219/225) Updating files: 98% (221/225) Updating files: 99% (223/225) Updating files: 100% (225/225) Updating files: 100% (225/225), done. 2023-10-30T13:33:18.475 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2023-10-30T13:33:18.481 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.8 2023-10-30T13:33:18.498 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 9 -eq 48 ] 2023-10-30T13:33:18.498 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.9 2023-10-30T13:33:18.498 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.9 2023-10-30T13:33:18.499 INFO:tasks.workunit.client.0.smithi148.stdout:v0.9 2023-10-30T13:33:18.499 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.9 2023-10-30T13:33:23.665 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 17% (34/193) Updating files: 18% (35/193) Updating files: 19% (37/193) Updating files: 20% (39/193) Updating files: 21% (41/193) Updating files: 22% (43/193) Updating files: 23% (45/193) Updating files: 24% (47/193) Updating files: 25% (49/193) Updating files: 26% (51/193) Updating files: 27% (53/193) Updating files: 28% (55/193) Updating files: 29% (56/193) Updating files: 30% (58/193) Updating files: 31% (60/193) Updating files: 31% (61/193) Updating files: 32% (62/193) Updating files: 33% (64/193) Updating files: 34% (66/193) Updating files: 35% (68/193) Updating files: 36% (70/193) Updating files: 37% (72/193) Updating files: 38% (74/193) Updating files: 39% (76/193) Updating files: 40% (78/193) Updating files: 41% (80/193) Updating files: 42% (82/193) Updating files: 43% (83/193) Updating files: 44% (85/193) Updating files: 45% (87/193) Updating files: 46% (89/193) Updating files: 47% (91/193) Updating files: 48% (93/193) Updating files: 49% (95/193) Updating files: 50% (97/193) Updating files: 51% (99/193) Updating files: 52% (101/193) Updating files: 53% (103/193) Updating files: 54% (105/193) Updating files: 55% (107/193) Updating files: 56% (109/193) Updating files: 57% (111/193) Updating files: 58% (112/193) Updating files: 59% (114/193) Updating files: 60% (116/193) Updating files: 61% (118/193) Updating files: 62% (120/193) Updating files: 63% (122/193) Updating files: 64% (124/193) Updating files: 65% (126/193) Updating files: 66% (128/193) Updating files: 67% (130/193) Updating files: 68% (132/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 76% (148/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 89% (172/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2023-10-30T13:33:23.696 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2023-10-30T13:33:23.702 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.9 2023-10-30T13:33:23.750 INFO:tasks.workunit.client.0.smithi148.stdout:v0.10 2023-10-30T13:33:23.750 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 10 -eq 48 ] 2023-10-30T13:33:23.750 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.10 2023-10-30T13:33:23.751 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.10 2023-10-30T13:33:23.751 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.10 2023-10-30T13:33:27.505 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 14% (24/164) Updating files: 15% (25/164) Updating files: 16% (27/164) Updating files: 17% (28/164) Updating files: 18% (30/164) Updating files: 19% (32/164) Updating files: 20% (33/164) Updating files: 21% (35/164) Updating files: 22% (37/164) Updating files: 23% (38/164) Updating files: 24% (40/164) Updating files: 25% (41/164) Updating files: 26% (43/164) Updating files: 27% (45/164) Updating files: 28% (46/164) Updating files: 29% (48/164) Updating files: 30% (50/164) Updating files: 31% (51/164) Updating files: 31% (52/164) Updating files: 32% (53/164) Updating files: 33% (55/164) Updating files: 34% (56/164) Updating files: 35% (58/164) Updating files: 36% (60/164) Updating files: 37% (61/164) Updating files: 38% (63/164) Updating files: 39% (64/164) Updating files: 40% (66/164) Updating files: 41% (68/164) Updating files: 42% (69/164) Updating files: 43% (71/164) Updating files: 44% (73/164) Updating files: 45% (74/164) Updating files: 46% (76/164) Updating files: 47% (78/164) Updating files: 48% (79/164) Updating files: 49% (81/164) Updating files: 50% (82/164) Updating files: 51% (84/164) Updating files: 52% (86/164) Updating files: 53% (87/164) Updating files: 54% (89/164) Updating files: 55% (91/164) Updating files: 56% (92/164) Updating files: 57% (94/164) Updating files: 58% (96/164) Updating files: 59% (97/164) Updating files: 60% (99/164) Updating files: 61% (101/164) Updating files: 62% (102/164) Updating files: 63% (104/164) Updating files: 64% (105/164) Updating files: 65% (107/164) Updating files: 66% (109/164) Updating files: 67% (110/164) Updating files: 68% (112/164) Updating files: 69% (114/164) Updating files: 70% (115/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2023-10-30T13:33:27.534 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 007313fe844 v0.10 2023-10-30T13:33:27.540 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.10 2023-10-30T13:33:27.661 INFO:tasks.workunit.client.0.smithi148.stdout:v0.11 2023-10-30T13:33:27.661 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 11 -eq 48 ] 2023-10-30T13:33:27.661 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.11 2023-10-30T13:33:27.661 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.11 2023-10-30T13:33:27.662 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.11 2023-10-30T13:33:28.980 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 95% (66/69) Updating files: 97% (67/69) Updating files: 98% (68/69) Updating files: 100% (69/69) Updating files: 100% (69/69), done. 2023-10-30T13:33:29.011 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2023-10-30T13:33:29.015 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.11 2023-10-30T13:33:29.517 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:29.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:29.787 INFO:tasks.workunit.client.0.smithi148.stdout:v0.12 2023-10-30T13:33:29.787 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 12 -eq 48 ] 2023-10-30T13:33:29.787 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.12 2023-10-30T13:33:29.788 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.12 2023-10-30T13:33:29.788 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.12 2023-10-30T13:33:30.733 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at f524150822a v0.12 2023-10-30T13:33:30.737 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.12 2023-10-30T13:33:30.857 INFO:tasks.workunit.client.0.smithi148.stdout:v0.13 2023-10-30T13:33:30.858 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 13 -eq 48 ] 2023-10-30T13:33:30.858 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.13 2023-10-30T13:33:30.858 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.13 2023-10-30T13:33:30.858 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.13 2023-10-30T13:33:34.121 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 22% (30/133) Updating files: 23% (31/133) Updating files: 24% (32/133) Updating files: 25% (34/133) Updating files: 26% (35/133) Updating files: 27% (36/133) Updating files: 28% (38/133) Updating files: 29% (39/133) Updating files: 30% (40/133) Updating files: 31% (42/133) Updating files: 32% (43/133) Updating files: 33% (44/133) Updating files: 34% (46/133) Updating files: 35% (47/133) Updating files: 36% (48/133) Updating files: 37% (50/133) Updating files: 38% (51/133) Updating files: 39% (52/133) Updating files: 40% (54/133) Updating files: 41% (55/133) Updating files: 42% (56/133) Updating files: 43% (58/133) Updating files: 44% (59/133) Updating files: 45% (60/133) Updating files: 46% (62/133) Updating files: 47% (63/133) Updating files: 48% (64/133) Updating files: 49% (66/133) Updating files: 50% (67/133) Updating files: 51% (68/133) Updating files: 52% (70/133) Updating files: 53% (71/133) Updating files: 54% (72/133) Updating files: 55% (74/133) Updating files: 56% (75/133) Updating files: 57% (76/133) Updating files: 58% (78/133) Updating files: 59% (79/133) Updating files: 60% (80/133) Updating files: 61% (82/133) Updating files: 62% (83/133) Updating files: 63% (84/133) Updating files: 64% (86/133) Updating files: 65% (87/133) Updating files: 66% (88/133) Updating files: 67% (90/133) Updating files: 68% (91/133) Updating files: 69% (92/133) Updating files: 70% (94/133) Updating files: 71% (95/133) Updating files: 72% (96/133) Updating files: 73% (98/133) Updating files: 74% (99/133) Updating files: 75% (100/133) Updating files: 76% (102/133) Updating files: 77% (103/133) Updating files: 78% (104/133) Updating files: 79% (106/133) Updating files: 80% (107/133) Updating files: 81% (108/133) Updating files: 82% (110/133) Updating files: 83% (111/133) Updating files: 84% (112/133) Updating files: 85% (114/133) Updating files: 86% (115/133) Updating files: 87% (116/133) Updating files: 88% (118/133) Updating files: 89% (119/133) Updating files: 90% (120/133) Updating files: 91% (122/133) Updating files: 92% (123/133) Updating files: 93% (124/133) Updating files: 94% (126/133) Updating files: 95% (127/133) Updating files: 96% (128/133) Updating files: 97% (130/133) Updating files: 98% (131/133) Updating files: 99% (132/133) Updating files: 100% (133/133) Updating files: 100% (133/133), done. 2023-10-30T13:33:34.151 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 2b32297267d v0.13 2023-10-30T13:33:34.156 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.13 2023-10-30T13:33:34.243 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 14 -eq 48 ] 2023-10-30T13:33:34.243 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.14 2023-10-30T13:33:34.243 INFO:tasks.workunit.client.0.smithi148.stdout:v0.14 2023-10-30T13:33:34.244 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.14 2023-10-30T13:33:34.244 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.14 2023-10-30T13:33:38.102 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:38.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dd8dc640 1 Processor -- start 2023-10-30T13:33:38.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dd8dc640 1 -- start start 2023-10-30T13:33:38.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dd8dc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f41d81538f0 0x7f41d8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dd8dc640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d80ca6a0 con 0x7f41d815cd50 2023-10-30T13:33:38.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dd8dc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d815d300 con 0x7f41d80cefa0 2023-10-30T13:33:38.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dd8dc640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d815d5e0 con 0x7f41d81538f0 2023-10-30T13:33:38.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dc8da640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f41d81538f0 0x7f41d8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41d7fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f41d815cd50 0x7f41d80c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40626/0 (socket says 172.21.15.148:40626) 2023-10-30T13:33:38.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41d7fff640 1 -- 172.21.15.148:0/724051934 learned_addr learned my addr 172.21.15.148:0/724051934 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:33:38.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dc8da640 1 -- 172.21.15.148:0/724051934 >> v1:172.21.15.150:6789/0 conn(0x7f41d80cefa0 legacy=0x7f41d81640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dc8da640 1 -- 172.21.15.148:0/724051934 >> v1:172.21.15.148:6789/0 conn(0x7f41d815cd50 legacy=0x7f41d80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41dc8da640 1 -- 172.21.15.148:0/724051934 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d815d8c0 con 0x7f41d81538f0 2023-10-30T13:33:38.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.256+0000 7f41d77fe640 1 -- 172.21.15.148:0/724051934 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540469203 0 0) 0x7f41d80ca6a0 con 0x7f41d815cd50 2023-10-30T13:33:38.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41dc8da640 1 --2- 172.21.15.148:0/724051934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d81538f0 0x7f41d8159fb0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097c0 tx=0x7f41cc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17f92bc56fa475f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:38.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41d77fe640 1 -- 172.21.15.148:0/724051934 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0a8970 con 0x7f41d81538f0 2023-10-30T13:33:38.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41d77fe640 1 -- 172.21.15.148:0/724051934 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc0a8ad0 con 0x7f41d81538f0 2023-10-30T13:33:38.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41d77fe640 1 -- 172.21.15.148:0/724051934 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0a8e60 con 0x7f41d81538f0 2023-10-30T13:33:38.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41dd8dc640 1 -- 172.21.15.148:0/724051934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d81538f0 msgr2=0x7f41d8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41dd8dc640 1 --2- 172.21.15.148:0/724051934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d81538f0 0x7f41d8159fb0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097c0 tx=0x7f41cc01ec70 comp rx=0 tx=0).stop 2023-10-30T13:33:38.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41dd8dc640 1 -- 172.21.15.148:0/724051934 shutdown_connections 2023-10-30T13:33:38.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41dd8dc640 1 --2- 172.21.15.148:0/724051934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d81538f0 0x7f41d8159fb0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41dd8dc640 1 -- 172.21.15.148:0/724051934 >> 172.21.15.148:0/724051934 conn(0x7f41d806a640 msgr2=0x7f41d8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:38.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41dd8dc640 1 -- 172.21.15.148:0/724051934 shutdown_connections 2023-10-30T13:33:38.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41dd8dc640 1 -- 172.21.15.148:0/724051934 wait complete. 2023-10-30T13:33:38.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.260+0000 7f41dd8dc640 1 Processor -- start 2023-10-30T13:33:38.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 -- start start 2023-10-30T13:33:38.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d80cefa0 0x7f41d815b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41d81538f0 0x7f41d80b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41d815cd50 0x7f41d8157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8151a30 con 0x7f41d80cefa0 2023-10-30T13:33:38.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d815c9d0 con 0x7f41d815cd50 2023-10-30T13:33:38.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8150a20 con 0x7f41d81538f0 2023-10-30T13:33:38.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dc8da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d80cefa0 0x7f41d815b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dc8da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d80cefa0 0x7f41d815b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34828/0 (socket says 172.21.15.148:34828) 2023-10-30T13:33:38.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dc8da640 1 -- 172.21.15.148:0/2309745438 learned_addr learned my addr 172.21.15.148:0/2309745438 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:38.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41d7fff640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41d81538f0 0x7f41d80b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd0db640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41d815cd50 0x7f41d8157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dc8da640 1 -- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41d81538f0 msgr2=0x7f41d80b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dc8da640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41d81538f0 0x7f41d80b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dc8da640 1 -- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41d815cd50 msgr2=0x7f41d8157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dc8da640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41d815cd50 0x7f41d8157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dc8da640 1 -- 172.21.15.148:0/2309745438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc009420 con 0x7f41d80cefa0 2023-10-30T13:33:38.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd0db640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41d815cd50 0x7f41d8157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:33:38.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dc8da640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d80cefa0 0x7f41d815b330 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097c0 tx=0x7f41cc01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:38.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41d57fa640 1 -- 172.21.15.148:0/2309745438 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0b9630 con 0x7f41d80cefa0 2023-10-30T13:33:38.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41d57fa640 1 -- 172.21.15.148:0/2309745438 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc0b97b0 con 0x7f41d80cefa0 2023-10-30T13:33:38.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d81529e0 con 0x7f41d80cefa0 2023-10-30T13:33:38.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41d57fa640 1 -- 172.21.15.148:0/2309745438 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0b9b60 con 0x7f41d80cefa0 2023-10-30T13:33:38.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d8150f50 con 0x7f41d80cefa0 2023-10-30T13:33:38.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.264+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d804e170 con 0x7f41d80cefa0 2023-10-30T13:33:38.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.268+0000 7f41d57fa640 1 -- 172.21.15.148:0/2309745438 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41cc0b7040 con 0x7f41d80cefa0 2023-10-30T13:33:38.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.268+0000 7f41d57fa640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41bc0664a0 0x7f41bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.268+0000 7f41d57fa640 1 -- 172.21.15.148:0/2309745438 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f41cc122300 con 0x7f41d80cefa0 2023-10-30T13:33:38.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.268+0000 7f41d7fff640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41bc0664a0 0x7f41bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.268+0000 7f41d7fff640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41bc0664a0 0x7f41bc068960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f41d8149350 tx=0x7f41c4007d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:38.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.272+0000 7f41d57fa640 1 -- 172.21.15.148:0/2309745438 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41cc0ef470 con 0x7f41d80cefa0 2023-10-30T13:33:38.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.456+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41d8157180 con 0x7f41d80cefa0 2023-10-30T13:33:38.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.456+0000 7f41d57fa640 1 -- 172.21.15.148:0/2309745438 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f41cc0f2a90 con 0x7f41d80cefa0 2023-10-30T13:33:38.452 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:38.452 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:38.453 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 98 2023-10-30T13:33:38.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41bc0664a0 msgr2=0x7f41bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41bc0664a0 0x7f41bc068960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f41d8149350 tx=0x7f41c4007d10 comp rx=0 tx=0).stop 2023-10-30T13:33:38.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d80cefa0 msgr2=0x7f41d815b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d80cefa0 0x7f41d815b330 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097c0 tx=0x7f41cc01ba10 comp rx=0 tx=0).stop 2023-10-30T13:33:38.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 shutdown_connections 2023-10-30T13:33:38.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41bc0664a0 0x7f41bc068960 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41d815cd50 0x7f41d8157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41d81538f0 0x7f41d80b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 --2- 172.21.15.148:0/2309745438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d80cefa0 0x7f41d815b330 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 >> 172.21.15.148:0/2309745438 conn(0x7f41d806a640 msgr2=0x7f41d80caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:38.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 shutdown_connections 2023-10-30T13:33:38.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.460+0000 7f41dd8dc640 1 -- 172.21.15.148:0/2309745438 wait complete. 2023-10-30T13:33:38.473 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:33:38.473 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:33:38.473 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:33:38.495 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:33:38.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:38.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.676+0000 7f4e6349e640 1 Processor -- start 2023-10-30T13:33:38.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.676+0000 7f4e6349e640 1 -- start start 2023-10-30T13:33:38.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e6349e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e5c0c6f60 0x7f4e5c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e6349e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c0c0a50 con 0x7f4e5c0c6b80 2023-10-30T13:33:38.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e6349e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c06f6e0 con 0x7f4e5c0c05d0 2023-10-30T13:33:38.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e6349e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c06f9c0 con 0x7f4e5c0c6f60 2023-10-30T13:33:38.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e6123a640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4e5c0c6b80 0x7f4e5c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40640/0 (socket says 172.21.15.148:40640) 2023-10-30T13:33:38.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e6123a640 1 -- 172.21.15.148:0/1135009075 learned_addr learned my addr 172.21.15.148:0/1135009075 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:33:38.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e61a3b640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e5c0c6f60 0x7f4e5c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761550802 0 0) 0x7f4e5c0c0a50 con 0x7f4e5c0c6b80 2023-10-30T13:33:38.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e48003530 con 0x7f4e5c0c6b80 2023-10-30T13:33:38.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2794137300 0 0) 0x7f4e48003530 con 0x7f4e5c0c6b80 2023-10-30T13:33:38.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e5c0c6f60 msgr2=0x7f4e5c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:38.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e5c0c6f60 0x7f4e5c1566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 >> v1:172.21.15.150:6789/0 conn(0x7f4e5c0c05d0 legacy=0x7f4e5c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c06fca0 con 0x7f4e5c0c6b80 2023-10-30T13:33:38.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f4e58001680 con 0x7f4e5c0c6b80 2023-10-30T13:33:38.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 >> v1:172.21.15.148:6789/0 conn(0x7f4e5c0c6b80 legacy=0x7f4e5c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e5c0c6f60 0x7f4e480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48003f30 0x7f4e480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e48004d10 0x7f4e48005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c06f9c0 con 0x7f4e48003f30 2023-10-30T13:33:38.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4800ba70 con 0x7f4e48004d10 2023-10-30T13:33:38.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4800de40 con 0x7f4e5c0c6f60 2023-10-30T13:33:38.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e60a39640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48003f30 0x7f4e480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e6123a640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e5c0c6f60 0x7f4e480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e61a3b640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e48004d10 0x7f4e48005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e60a39640 1 -- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e5c0c6f60 msgr2=0x7f4e480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:38.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e60a39640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e5c0c6f60 0x7f4e480039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e60a39640 1 -- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e48004d10 msgr2=0x7f4e48005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:38.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e60a39640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e48004d10 0x7f4e48005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e60a39640 1 -- 172.21.15.148:0/1135009075 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e48003530 con 0x7f4e48003f30 2023-10-30T13:33:38.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.680+0000 7f4e61a3b640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e48004d10 0x7f4e48005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e60a39640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48003f30 0x7f4e480047d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c001750 tx=0x7f4e4c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1b3a9798de8db3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c0052d0 con 0x7f4e48003f30 2023-10-30T13:33:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4c0a7050 con 0x7f4e48003f30 2023-10-30T13:33:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e53fff640 1 -- 172.21.15.148:0/1135009075 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c003600 con 0x7f4e48003f30 2023-10-30T13:33:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e6349e640 1 -- 172.21.15.148:0/1135009075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48003f30 msgr2=0x7f4e480047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e6349e640 1 --2- 172.21.15.148:0/1135009075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48003f30 0x7f4e480047d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c001750 tx=0x7f4e4c01c3c0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e61a3b640 1 -- 172.21.15.148:0/1135009075 reap_dead start 2023-10-30T13:33:38.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e6349e640 1 -- 172.21.15.148:0/1135009075 shutdown_connections 2023-10-30T13:33:38.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e6349e640 1 -- 172.21.15.148:0/1135009075 >> 172.21.15.148:0/1135009075 conn(0x7f4e5c0c7910 msgr2=0x7f4e5c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:38.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e6349e640 1 -- 172.21.15.148:0/1135009075 shutdown_connections 2023-10-30T13:33:38.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.684+0000 7f4e6349e640 1 -- 172.21.15.148:0/1135009075 wait complete. 2023-10-30T13:33:38.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 Processor -- start 2023-10-30T13:33:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 -- start start 2023-10-30T13:33:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e48003f30 0x7f4e5c0c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e5c13d8d0 0x7f4e5c14e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e5c14ed80 0x7f4e5c134710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c13e3a0 con 0x7f4e5c14ed80 2023-10-30T13:33:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c13e660 con 0x7f4e5c13d8d0 2023-10-30T13:33:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c14f9d0 con 0x7f4e48003f30 2023-10-30T13:33:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e61a3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e5c14ed80 0x7f4e5c134710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e61a3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e5c14ed80 0x7f4e5c134710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34848/0 (socket says 172.21.15.148:34848) 2023-10-30T13:33:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e61a3b640 1 -- 172.21.15.148:0/2788402266 learned_addr learned my addr 172.21.15.148:0/2788402266 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6123a640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e48003f30 0x7f4e5c0c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e60a39640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e5c13d8d0 0x7f4e5c14e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e61a3b640 1 -- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e48003f30 msgr2=0x7f4e5c0c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e61a3b640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e48003f30 0x7f4e5c0c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e61a3b640 1 -- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e5c13d8d0 msgr2=0x7f4e5c14e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e61a3b640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e5c13d8d0 0x7f4e5c14e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e61a3b640 1 -- 172.21.15.148:0/2788402266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c14fcb0 con 0x7f4e5c14ed80 2023-10-30T13:33:38.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6123a640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e48003f30 0x7f4e5c0c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:38.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e61a3b640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e5c14ed80 0x7f4e5c134710 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0b1190 tx=0x7f4e54010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:38.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e51ffb640 1 -- 172.21.15.148:0/2788402266 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5400e050 con 0x7f4e5c14ed80 2023-10-30T13:33:38.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e51ffb640 1 -- 172.21.15.148:0/2788402266 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5400f040 con 0x7f4e5c14ed80 2023-10-30T13:33:38.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e51ffb640 1 -- 172.21.15.148:0/2788402266 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e54003be0 con 0x7f4e5c14ed80 2023-10-30T13:33:38.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c14ff90 con 0x7f4e5c14ed80 2023-10-30T13:33:38.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5c0c6440 con 0x7f4e5c14ed80 2023-10-30T13:33:38.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.688+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5c14f830 con 0x7f4e5c14ed80 2023-10-30T13:33:38.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.692+0000 7f4e51ffb640 1 -- 172.21.15.148:0/2788402266 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e54003d40 con 0x7f4e5c14ed80 2023-10-30T13:33:38.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.692+0000 7f4e51ffb640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e400664d0 0x7f4e40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:38.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.692+0000 7f4e51ffb640 1 -- 172.21.15.148:0/2788402266 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f4e54084970 con 0x7f4e5c14ed80 2023-10-30T13:33:38.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.692+0000 7f4e6123a640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e400664d0 0x7f4e40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:38.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.692+0000 7f4e6123a640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e400664d0 0x7f4e40068990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4e58001060 tx=0x7f4e58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:38.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.696+0000 7f4e51ffb640 1 -- 172.21.15.148:0/2788402266 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e5c14f830 con 0x7f4e5c14ed80 2023-10-30T13:33:38.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.876+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e5c14f830 con 0x7f4e5c14ed80 2023-10-30T13:33:38.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e51ffb640 1 -- 172.21.15.148:0/2788402266 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f4e5c14f830 con 0x7f4e5c14ed80 2023-10-30T13:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:38.874 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:38.875 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 98 2023-10-30T13:33:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e400664d0 msgr2=0x7f4e40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e400664d0 0x7f4e40068990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4e58001060 tx=0x7f4e58009040 comp rx=0 tx=0).stop 2023-10-30T13:33:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e5c14ed80 msgr2=0x7f4e5c134710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:38.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e5c14ed80 0x7f4e5c134710 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0b1190 tx=0x7f4e54010590 comp rx=0 tx=0).stop 2023-10-30T13:33:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 shutdown_connections 2023-10-30T13:33:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4e400664d0 0x7f4e40068990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e5c14ed80 0x7f4e5c134710 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e5c13d8d0 0x7f4e5c14e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 --2- 172.21.15.148:0/2788402266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4e48003f30 0x7f4e5c0c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:38.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 >> 172.21.15.148:0/2788402266 conn(0x7f4e5c0c7910 msgr2=0x7f4e5c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:38.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 shutdown_connections 2023-10-30T13:33:38.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:38.880+0000 7f4e6349e640 1 -- 172.21.15.148:0/2788402266 wait complete. 2023-10-30T13:33:38.895 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:33:40.896 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:41.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3006fc4640 1 Processor -- start 2023-10-30T13:33:41.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3006fc4640 1 -- start start 2023-10-30T13:33:41.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3006fc4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f300006f7a0 0x7f300006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:41.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3006fc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3000066cf0 con 0x7f30000c6bd0 2023-10-30T13:33:41.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3006fc4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30000c70d0 con 0x7f3000071110 2023-10-30T13:33:41.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3006fc4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30000700c0 con 0x7f300006f7a0 2023-10-30T13:33:41.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2ffffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f30000c6bd0 0x7f3000069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40652/0 (socket says 172.21.15.148:40652) 2023-10-30T13:33:41.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2ffffff640 1 -- 172.21.15.148:0/1080194460 learned_addr learned my addr 172.21.15.148:0/1080194460 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:33:41.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3004d60640 1 --2- 172.21.15.148:0/1080194460 >> v2:172.21.15.150:3300/0 conn(0x7f300006f7a0 0x7f300006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:41.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496858971 0 0) 0x7f3000066cf0 con 0x7f30000c6bd0 2023-10-30T13:33:41.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff0003530 con 0x7f30000c6bd0 2023-10-30T13:33:41.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1359916532 0 0) 0x7f2ff0003530 con 0x7f30000c6bd0 2023-10-30T13:33:41.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 >> v2:172.21.15.150:3300/0 conn(0x7f300006f7a0 msgr2=0x7f300006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:41.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 --2- 172.21.15.148:0/1080194460 >> v2:172.21.15.150:3300/0 conn(0x7f300006f7a0 0x7f300006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 >> v1:172.21.15.150:6789/0 conn(0x7f3000071110 legacy=0x7f3000158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:41.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30000c7390 con 0x7f30000c6bd0 2023-10-30T13:33:41.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f2fec001680 con 0x7f30000c6bd0 2023-10-30T13:33:41.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 >> v1:172.21.15.148:6789/0 conn(0x7f30000c6bd0 legacy=0x7f3000069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:41.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ff0003980 0x7f2ff0003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:41.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ff0004380 0x7f2ff0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:41.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ff0005050 0x7f2ff000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:41.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30000700c0 con 0x7f2ff0003980 2023-10-30T13:33:41.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff000bf00 con 0x7f2ff0005050 2023-10-30T13:33:41.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff000c0c0 con 0x7f2ff0004380 2023-10-30T13:33:41.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3004d60640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ff0003980 0x7f2ff0003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:41.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3005561640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ff0005050 0x7f2ff000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:41.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f2ffffff640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ff0004380 0x7f2ff0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:41.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3004d60640 1 -- 172.21.15.148:0/1080194460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ff0004380 msgr2=0x7f2ff0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:41.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3004d60640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ff0004380 0x7f2ff0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3004d60640 1 -- 172.21.15.148:0/1080194460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ff0005050 msgr2=0x7f2ff000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:41.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3004d60640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ff0005050 0x7f2ff000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3004d60640 1 -- 172.21.15.148:0/1080194460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0003530 con 0x7f2ff0003980 2023-10-30T13:33:41.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.052+0000 7f3005561640 1 -- 172.21.15.148:0/1080194460 reap_dead start 2023-10-30T13:33:41.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f3004d60640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ff0003980 0x7f2ff0003d60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009ae0 tx=0x7f2ff401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8f2786c29f5739 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:41.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40a5640 con 0x7f2ff0003980 2023-10-30T13:33:41.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff40a57a0 con 0x7f2ff0003980 2023-10-30T13:33:41.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f2fff7fe640 1 -- 172.21.15.148:0/1080194460 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40a5b50 con 0x7f2ff0003980 2023-10-30T13:33:41.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f3006fc4640 1 -- 172.21.15.148:0/1080194460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ff0003980 msgr2=0x7f2ff0003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:41.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f3006fc4640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ff0003980 0x7f2ff0003d60 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009ae0 tx=0x7f2ff401ca50 comp rx=0 tx=0).stop 2023-10-30T13:33:41.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f3006fc4640 1 -- 172.21.15.148:0/1080194460 shutdown_connections 2023-10-30T13:33:41.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f3006fc4640 1 --2- 172.21.15.148:0/1080194460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ff0003980 0x7f2ff0003d60 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f3006fc4640 1 -- 172.21.15.148:0/1080194460 >> 172.21.15.148:0/1080194460 conn(0x7f30000c7910 msgr2=0x7f30000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:41.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f3006fc4640 1 -- 172.21.15.148:0/1080194460 shutdown_connections 2023-10-30T13:33:41.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.056+0000 7f3006fc4640 1 -- 172.21.15.148:0/1080194460 wait complete. 2023-10-30T13:33:41.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 Processor -- start 2023-10-30T13:33:41.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 -- start start 2023-10-30T13:33:41.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ff0003980 0x7f30000aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:41.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3000158a50 0x7f30000c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:41.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30000c07a0 0x7f30000c0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:41.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30000afbf0 con 0x7f3000158a50 2023-10-30T13:33:41.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30000af4a0 con 0x7f2ff0003980 2023-10-30T13:33:41.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30000c1170 con 0x7f30000c07a0 2023-10-30T13:33:41.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3000158a50 0x7f30000c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:41.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3000158a50 0x7f30000c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34856/0 (socket says 172.21.15.148:34856) 2023-10-30T13:33:41.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffffff640 1 -- 172.21.15.148:0/3842627128 learned_addr learned my addr 172.21.15.148:0/3842627128 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:41.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3005561640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30000c07a0 0x7f30000c0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:41.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3004d60640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ff0003980 0x7f30000aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:41.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffffff640 1 -- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30000c07a0 msgr2=0x7f30000c0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:41.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffffff640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30000c07a0 0x7f30000c0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffffff640 1 -- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ff0003980 msgr2=0x7f30000aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:41.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffffff640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ff0003980 0x7f30000aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffffff640 1 -- 172.21.15.148:0/3842627128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30001f2140 con 0x7f3000158a50 2023-10-30T13:33:41.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffffff640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3000158a50 0x7f30000c4090 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00e7a0 tx=0x7f2fec00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:41.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffd7fa640 1 -- 172.21.15.148:0/3842627128 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec00eeb0 con 0x7f3000158a50 2023-10-30T13:33:41.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffd7fa640 1 -- 172.21.15.148:0/3842627128 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec00c8d0 con 0x7f3000158a50 2023-10-30T13:33:41.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f2ffd7fa640 1 -- 172.21.15.148:0/3842627128 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec00cc80 con 0x7f3000158a50 2023-10-30T13:33:41.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30000ae8d0 con 0x7f3000158a50 2023-10-30T13:33:41.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.060+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30000aee50 con 0x7f3000158a50 2023-10-30T13:33:41.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.064+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300004e170 con 0x7f3000158a50 2023-10-30T13:33:41.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.064+0000 7f2ffd7fa640 1 -- 172.21.15.148:0/3842627128 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2fec01d460 con 0x7f3000158a50 2023-10-30T13:33:41.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.064+0000 7f2ffd7fa640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fe40664a0 0x7f2fe4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:41.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.064+0000 7f2ffd7fa640 1 -- 172.21.15.148:0/3842627128 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f2fec084360 con 0x7f3000158a50 2023-10-30T13:33:41.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.064+0000 7f3004d60640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fe40664a0 0x7f2fe4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:41.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.064+0000 7f3004d60640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fe40664a0 0x7f2fe4068960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40af000 tx=0x7f2ff4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:41.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.068+0000 7f2ffd7fa640 1 -- 172.21.15.148:0/3842627128 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2fec054db0 con 0x7f3000158a50 2023-10-30T13:33:41.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.248+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30000af5e0 con 0x7f3000158a50 2023-10-30T13:33:41.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.248+0000 7f2ffd7fa640 1 -- 172.21.15.148:0/3842627128 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f2fec054940 con 0x7f3000158a50 2023-10-30T13:33:41.244 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:41.245 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:41.245 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 98 2023-10-30T13:33:41.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fe40664a0 msgr2=0x7f2fe4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:41.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fe40664a0 0x7f2fe4068960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40af000 tx=0x7f2ff4008040 comp rx=0 tx=0).stop 2023-10-30T13:33:41.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3000158a50 msgr2=0x7f30000c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:41.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3000158a50 0x7f30000c4090 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00e7a0 tx=0x7f2fec00ec60 comp rx=0 tx=0).stop 2023-10-30T13:33:41.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 shutdown_connections 2023-10-30T13:33:41.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fe40664a0 0x7f2fe4068960 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3000158a50 0x7f30000c4090 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f30000c07a0 0x7f30000c0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 --2- 172.21.15.148:0/3842627128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ff0003980 0x7f30000aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:41.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 >> 172.21.15.148:0/3842627128 conn(0x7f30000c7910 msgr2=0x7f30000c62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:41.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 shutdown_connections 2023-10-30T13:33:41.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:41.252+0000 7f3006fc4640 1 -- 172.21.15.148:0/3842627128 wait complete. 2023-10-30T13:33:41.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:33:43.266 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:43.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f845e046640 1 Processor -- start 2023-10-30T13:33:43.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f845e046640 1 -- start start 2023-10-30T13:33:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f845e046640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f84580cf160 0x7f84580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f845e046640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84580cb800 con 0x7f845815d6f0 2023-10-30T13:33:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f845e046640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84580cb330 con 0x7f84581538a0 2023-10-30T13:33:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f845e046640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84580cad70 con 0x7f84580cf160 2023-10-30T13:33:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f8457fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f84580cf160 0x7f84580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f8457fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f84580cf160 0x7f84580cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33282/0 (socket says 172.21.15.148:33282) 2023-10-30T13:33:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f8457fff640 1 -- 172.21.15.148:0/2567429512 learned_addr learned my addr 172.21.15.148:0/2567429512 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f8457fff640 1 -- 172.21.15.148:0/2567429512 >> v1:172.21.15.150:6789/0 conn(0x7f84581538a0 legacy=0x7f84580ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f8457fff640 1 -- 172.21.15.148:0/2567429512 >> v1:172.21.15.148:6789/0 conn(0x7f845815d6f0 legacy=0x7f845815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.420+0000 7f8457fff640 1 -- 172.21.15.148:0/2567429512 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84580cb050 con 0x7f84580cf160 2023-10-30T13:33:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f8457fff640 1 --2- 172.21.15.148:0/2567429512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84580cf160 0x7f84580cf590 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8450009e50 tx=0x7f845001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b586c68d709294cb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f84567fc640 1 -- 172.21.15.148:0/2567429512 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84500af400 con 0x7f84580cf160 2023-10-30T13:33:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f84567fc640 1 -- 172.21.15.148:0/2567429512 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84500ab050 con 0x7f84580cf160 2023-10-30T13:33:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f84567fc640 1 -- 172.21.15.148:0/2567429512 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84500af780 con 0x7f84580cf160 2023-10-30T13:33:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f845e046640 1 -- 172.21.15.148:0/2567429512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84580cf160 msgr2=0x7f84580cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f845e046640 1 --2- 172.21.15.148:0/2567429512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84580cf160 0x7f84580cf590 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8450009e50 tx=0x7f845001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:33:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f845e046640 1 -- 172.21.15.148:0/2567429512 shutdown_connections 2023-10-30T13:33:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f845e046640 1 --2- 172.21.15.148:0/2567429512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84580cf160 0x7f84580cf590 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f845e046640 1 -- 172.21.15.148:0/2567429512 >> 172.21.15.148:0/2567429512 conn(0x7f845806bfe0 msgr2=0x7f845806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f845e046640 1 -- 172.21.15.148:0/2567429512 shutdown_connections 2023-10-30T13:33:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.424+0000 7f845e046640 1 -- 172.21.15.148:0/2567429512 wait complete. 2023-10-30T13:33:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f845e046640 1 Processor -- start 2023-10-30T13:33:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f845e046640 1 -- start start 2023-10-30T13:33:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f845e046640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84580cf160 0x7f84580b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f845e046640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84581538a0 0x7f845814e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f845e046640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845815d6f0 0x7f84581f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f845e046640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8458158e30 con 0x7f845815d6f0 2023-10-30T13:33:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f845e046640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8458158320 con 0x7f84581538a0 2023-10-30T13:33:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f845e046640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8458159710 con 0x7f84580cf160 2023-10-30T13:33:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8457fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845815d6f0 0x7f84581f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8457fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845815d6f0 0x7f84581f2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33294/0 (socket says 172.21.15.148:33294) 2023-10-30T13:33:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8457fff640 1 -- 172.21.15.148:0/154068444 learned_addr learned my addr 172.21.15.148:0/154068444 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8456ffd640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84581538a0 0x7f845814e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f84577fe640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84580cf160 0x7f84580b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8457fff640 1 -- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84580cf160 msgr2=0x7f84580b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8457fff640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84580cf160 0x7f84580b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8457fff640 1 -- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84581538a0 msgr2=0x7f845814e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8457fff640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84581538a0 0x7f845814e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8457fff640 1 -- 172.21.15.148:0/154068444 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450009b30 con 0x7f845815d6f0 2023-10-30T13:33:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f8457fff640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845815d6f0 0x7f84581f2d80 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f845001be00 tx=0x7f845001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f843ffff640 1 -- 172.21.15.148:0/154068444 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84500afd30 con 0x7f845815d6f0 2023-10-30T13:33:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f843ffff640 1 -- 172.21.15.148:0/154068444 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84500afe90 con 0x7f845815d6f0 2023-10-30T13:33:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845814e110 con 0x7f845815d6f0 2023-10-30T13:33:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.428+0000 7f843ffff640 1 -- 172.21.15.148:0/154068444 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84500ba680 con 0x7f845815d6f0 2023-10-30T13:33:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.432+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8458158b50 con 0x7f845815d6f0 2023-10-30T13:33:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.432+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8458157de0 con 0x7f845815d6f0 2023-10-30T13:33:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.432+0000 7f843ffff640 1 -- 172.21.15.148:0/154068444 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84500b6040 con 0x7f845815d6f0 2023-10-30T13:33:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.432+0000 7f843ffff640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84440664a0 0x7f8444068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.432+0000 7f843ffff640 1 -- 172.21.15.148:0/154068444 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f8450122240 con 0x7f845815d6f0 2023-10-30T13:33:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.432+0000 7f84577fe640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84440664a0 0x7f8444068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.432+0000 7f84577fe640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84440664a0 0x7f8444068960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8448000f70 tx=0x7f8448009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.436+0000 7f843ffff640 1 -- 172.21.15.148:0/154068444 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8458157de0 con 0x7f845815d6f0 2023-10-30T13:33:43.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.616+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8458158460 con 0x7f845815d6f0 2023-10-30T13:33:43.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.616+0000 7f843ffff640 1 -- 172.21.15.148:0/154068444 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f8458158460 con 0x7f845815d6f0 2023-10-30T13:33:43.614 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:43.614 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:43.616 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 98 2023-10-30T13:33:43.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84440664a0 msgr2=0x7f8444068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:43.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84440664a0 0x7f8444068960 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8448000f70 tx=0x7f8448009040 comp rx=0 tx=0).stop 2023-10-30T13:33:43.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845815d6f0 msgr2=0x7f84581f2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:43.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845815d6f0 0x7f84581f2d80 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f845001be00 tx=0x7f845001bf90 comp rx=0 tx=0).stop 2023-10-30T13:33:43.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 shutdown_connections 2023-10-30T13:33:43.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f84440664a0 0x7f8444068960 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:43.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f845815d6f0 0x7f84581f2d80 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:43.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84581538a0 0x7f845814e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:43.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 --2- 172.21.15.148:0/154068444 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f84580cf160 0x7f84580b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:43.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 >> 172.21.15.148:0/154068444 conn(0x7f845806bfe0 msgr2=0x7f84580cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:43.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 shutdown_connections 2023-10-30T13:33:43.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:43.620+0000 7f845e046640 1 -- 172.21.15.148:0/154068444 wait complete. 2023-10-30T13:33:43.634 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:33:45.635 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:45.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdb0613e640 1 Processor -- start 2023-10-30T13:33:45.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdb0613e640 1 -- start start 2023-10-30T13:33:45.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdb0613e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdb0006f7a0 0x7fdb0006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:45.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdb0613e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb000c6bd0 0x7fdb00069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdb0613e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0006a3d0 con 0x7fdb00071110 2023-10-30T13:33:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdb0613e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb000700c0 con 0x7fdb0006f7a0 2023-10-30T13:33:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdb0613e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb00066cf0 con 0x7fdb000c6bd0 2023-10-30T13:33:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdaff7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdb0006f7a0 0x7fdb0006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:45.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdaff7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdb0006f7a0 0x7fdb0006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33296/0 (socket says 172.21.15.148:33296) 2023-10-30T13:33:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdaff7fe640 1 -- 172.21.15.148:0/4289623085 learned_addr learned my addr 172.21.15.148:0/4289623085 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdafeffd640 1 --2- 172.21.15.148:0/4289623085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb000c6bd0 0x7fdb00069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdaff7fe640 1 -- 172.21.15.148:0/4289623085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb000c6bd0 msgr2=0x7fdb00069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdaff7fe640 1 --2- 172.21.15.148:0/4289623085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb000c6bd0 0x7fdb00069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdaff7fe640 1 -- 172.21.15.148:0/4289623085 >> v1:172.21.15.148:6789/0 conn(0x7fdb00071110 legacy=0x7fdb00158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:45.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.792+0000 7fdaff7fe640 1 -- 172.21.15.148:0/4289623085 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb000c70d0 con 0x7fdb0006f7a0 2023-10-30T13:33:45.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.796+0000 7fdaff7fe640 1 --2- 172.21.15.148:0/4289623085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 0x7fdb0006fb80 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0009870 tx=0x7fdaf001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc659f47160dc97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:45.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.796+0000 7fdafe7fc640 1 -- 172.21.15.148:0/4289623085 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf00a8a30 con 0x7fdb0006f7a0 2023-10-30T13:33:45.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.796+0000 7fdafe7fc640 1 -- 172.21.15.148:0/4289623085 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf00a8b90 con 0x7fdb0006f7a0 2023-10-30T13:33:45.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.796+0000 7fdafe7fc640 1 -- 172.21.15.148:0/4289623085 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf00b0440 con 0x7fdb0006f7a0 2023-10-30T13:33:45.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 -- 172.21.15.148:0/4289623085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 msgr2=0x7fdb0006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:45.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 --2- 172.21.15.148:0/4289623085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 0x7fdb0006fb80 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0009870 tx=0x7fdaf001ed30 comp rx=0 tx=0).stop 2023-10-30T13:33:45.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 -- 172.21.15.148:0/4289623085 shutdown_connections 2023-10-30T13:33:45.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 --2- 172.21.15.148:0/4289623085 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb000c6bd0 0x7fdb00069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 --2- 172.21.15.148:0/4289623085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 0x7fdb0006fb80 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 -- 172.21.15.148:0/4289623085 >> 172.21.15.148:0/4289623085 conn(0x7fdb000c7910 msgr2=0x7fdb000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:45.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 -- 172.21.15.148:0/4289623085 shutdown_connections 2023-10-30T13:33:45.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 -- 172.21.15.148:0/4289623085 wait complete. 2023-10-30T13:33:45.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 Processor -- start 2023-10-30T13:33:45.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 -- start start 2023-10-30T13:33:45.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 0x7fdb001f0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:45.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb00071110 0x7fdb00146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:45.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb000c6bd0 0x7fdb0014d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:45.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0014cef0 con 0x7fdb0006f7a0 2023-10-30T13:33:45.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0014aa90 con 0x7fdb000c6bd0 2023-10-30T13:33:45.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdb0613e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0014c8c0 con 0x7fdb00071110 2023-10-30T13:33:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdaff7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 0x7fdb001f0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdaff7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 0x7fdb001f0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33302/0 (socket says 172.21.15.148:33302) 2023-10-30T13:33:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdaff7fe640 1 -- 172.21.15.148:0/2082761578 learned_addr learned my addr 172.21.15.148:0/2082761578 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdafeffd640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb00071110 0x7fdb00146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdafffff640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb000c6bd0 0x7fdb0014d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdaff7fe640 1 -- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb00071110 msgr2=0x7fdb00146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdaff7fe640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb00071110 0x7fdb00146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdaff7fe640 1 -- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb000c6bd0 msgr2=0x7fdb0014d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdaff7fe640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb000c6bd0 0x7fdb0014d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdaff7fe640 1 -- 172.21.15.148:0/2082761578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf00094d0 con 0x7fdb0006f7a0 2023-10-30T13:33:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdafeffd640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb00071110 0x7fdb00146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:33:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.800+0000 7fdafffff640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb000c6bd0 0x7fdb0014d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:33:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdaff7fe640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 0x7fdb001f0050 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00b3040 tx=0x7fdaf001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdae7fff640 1 -- 172.21.15.148:0/2082761578 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf00be070 con 0x7fdb0006f7a0 2023-10-30T13:33:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdae7fff640 1 -- 172.21.15.148:0/2082761578 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf00ba400 con 0x7fdb0006f7a0 2023-10-30T13:33:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb001f0870 con 0x7fdb0006f7a0 2023-10-30T13:33:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdae7fff640 1 -- 172.21.15.148:0/2082761578 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf00ba7d0 con 0x7fdb0006f7a0 2023-10-30T13:33:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb001f0e30 con 0x7fdb0006f7a0 2023-10-30T13:33:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0014d030 con 0x7fdb0006f7a0 2023-10-30T13:33:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdae7fff640 1 -- 172.21.15.148:0/2082761578 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdaf00c3410 con 0x7fdb0006f7a0 2023-10-30T13:33:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdae7fff640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdaec0664a0 0x7fdaec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdae7fff640 1 -- 172.21.15.148:0/2082761578 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fdaf0121e90 con 0x7fdb0006f7a0 2023-10-30T13:33:45.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.804+0000 7fdafeffd640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdaec0664a0 0x7fdaec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:45.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.808+0000 7fdafeffd640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdaec0664a0 0x7fdaec068960 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdb001e7470 tx=0x7fdaf4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:45.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.808+0000 7fdae7fff640 1 -- 172.21.15.148:0/2082761578 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb0014d030 con 0x7fdb0006f7a0 2023-10-30T13:33:45.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.992+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb0014d030 con 0x7fdb0006f7a0 2023-10-30T13:33:45.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.992+0000 7fdae7fff640 1 -- 172.21.15.148:0/2082761578 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fdb0014d030 con 0x7fdb0006f7a0 2023-10-30T13:33:45.990 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:45.990 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:45.990 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 98 2023-10-30T13:33:45.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdaec0664a0 msgr2=0x7fdaec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:45.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdaec0664a0 0x7fdaec068960 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdb001e7470 tx=0x7fdaf4008040 comp rx=0 tx=0).stop 2023-10-30T13:33:45.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 msgr2=0x7fdb001f0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:45.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 0x7fdb001f0050 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00b3040 tx=0x7fdaf001bad0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 shutdown_connections 2023-10-30T13:33:45.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdaec0664a0 0x7fdaec068960 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb000c6bd0 0x7fdb0014d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdb00071110 0x7fdb00146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 --2- 172.21.15.148:0/2082761578 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb0006f7a0 0x7fdb001f0050 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:45.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 >> 172.21.15.148:0/2082761578 conn(0x7fdb000c7910 msgr2=0x7fdb000c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:45.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 shutdown_connections 2023-10-30T13:33:45.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:45.996+0000 7fdb0613e640 1 -- 172.21.15.148:0/2082761578 wait complete. 2023-10-30T13:33:46.011 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:33:48.012 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:48.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.164+0000 7efc0f417640 1 Processor -- start 2023-10-30T13:33:48.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.164+0000 7efc0f417640 1 -- start start 2023-10-30T13:33:48.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.164+0000 7efc0f417640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc0806f7a0 0x7efc0806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:48.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.164+0000 7efc0f417640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc08158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:48.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.164+0000 7efc0f417640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc08066cf0 con 0x7efc080c6bd0 2023-10-30T13:33:48.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.164+0000 7efc0f417640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc080700c0 con 0x7efc0806f7a0 2023-10-30T13:33:48.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.164+0000 7efc0f417640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080c70d0 con 0x7efc08071110 2023-10-30T13:33:48.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.168+0000 7efc0d9b4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc08158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:48.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.168+0000 7efc0d1b3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc0806f7a0 0x7efc0806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:48.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.168+0000 7efc0d9b4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc08158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:39930/0 (socket says 172.21.15.148:39930) 2023-10-30T13:33:48.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.168+0000 7efc0d9b4640 1 -- 172.21.15.148:0/1408776089 learned_addr learned my addr 172.21.15.148:0/1408776089 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:48.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.168+0000 7efc0d9b4640 1 -- 172.21.15.148:0/1408776089 >> v2:172.21.15.150:3300/0 conn(0x7efc0806f7a0 msgr2=0x7efc0806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:48.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.168+0000 7efc0d9b4640 1 --2- 172.21.15.148:0/1408776089 >> v2:172.21.15.150:3300/0 conn(0x7efc0806f7a0 0x7efc0806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.168+0000 7efc0d9b4640 1 -- 172.21.15.148:0/1408776089 >> v1:172.21.15.150:6789/0 conn(0x7efc080c6bd0 legacy=0x7efc08069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:48.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.168+0000 7efc0d9b4640 1 -- 172.21.15.148:0/1408776089 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080c7390 con 0x7efc08071110 2023-10-30T13:33:48.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.168+0000 7efc0d1b3640 1 --2- 172.21.15.148:0/1408776089 >> v2:172.21.15.150:3300/0 conn(0x7efc0806f7a0 0x7efc0806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:33:48.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0d9b4640 1 --2- 172.21.15.148:0/1408776089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc08158110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efc00009e50 tx=0x7efc0001f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5368bf0cb60c3cea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:48.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efbfffff640 1 -- 172.21.15.148:0/1408776089 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000af400 con 0x7efc08071110 2023-10-30T13:33:48.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efbfffff640 1 -- 172.21.15.148:0/1408776089 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc000ab050 con 0x7efc08071110 2023-10-30T13:33:48.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efbfffff640 1 -- 172.21.15.148:0/1408776089 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000af780 con 0x7efc08071110 2023-10-30T13:33:48.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 -- 172.21.15.148:0/1408776089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 msgr2=0x7efc08158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:48.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 --2- 172.21.15.148:0/1408776089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc08158110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efc00009e50 tx=0x7efc0001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 -- 172.21.15.148:0/1408776089 shutdown_connections 2023-10-30T13:33:48.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 --2- 172.21.15.148:0/1408776089 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc08158110 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 --2- 172.21.15.148:0/1408776089 >> v2:172.21.15.150:3300/0 conn(0x7efc0806f7a0 0x7efc0806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 -- 172.21.15.148:0/1408776089 >> 172.21.15.148:0/1408776089 conn(0x7efc080c7910 msgr2=0x7efc080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:48.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 -- 172.21.15.148:0/1408776089 shutdown_connections 2023-10-30T13:33:48.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 -- 172.21.15.148:0/1408776089 wait complete. 2023-10-30T13:33:48.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 Processor -- start 2023-10-30T13:33:48.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.172+0000 7efc0f417640 1 -- start start 2023-10-30T13:33:48.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0f417640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc0806f7a0 0x7efc08153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:48.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0f417640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc0814ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:48.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0f417640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc080c6bd0 0x7efc08154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:48.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0f417640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080c2060 con 0x7efc080c6bd0 2023-10-30T13:33:48.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0f417640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080c40a0 con 0x7efc0806f7a0 2023-10-30T13:33:48.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0f417640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080c4590 con 0x7efc08071110 2023-10-30T13:33:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d9b4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc080c6bd0 0x7efc08154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d1b3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc0806f7a0 0x7efc08153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d9b4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc080c6bd0 0x7efc08154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33312/0 (socket says 172.21.15.148:33312) 2023-10-30T13:33:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d9b4640 1 -- 172.21.15.148:0/3924676144 learned_addr learned my addr 172.21.15.148:0/3924676144 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0c9b2640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc0814ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:48.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d9b4640 1 -- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 msgr2=0x7efc0814ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:48.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d9b4640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc0814ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d9b4640 1 -- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc0806f7a0 msgr2=0x7efc08153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:48.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d9b4640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc0806f7a0 0x7efc08153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d9b4640 1 -- 172.21.15.148:0/3924676144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc04009420 con 0x7efc080c6bd0 2023-10-30T13:33:48.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0d9b4640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc080c6bd0 0x7efc08154630 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7efc0001ce20 tx=0x7efc0001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:48.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efbfdffb640 1 -- 172.21.15.148:0/3924676144 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000afd30 con 0x7efc080c6bd0 2023-10-30T13:33:48.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efbfdffb640 1 -- 172.21.15.148:0/3924676144 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc000afe90 con 0x7efc080c6bd0 2023-10-30T13:33:48.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efbfdffb640 1 -- 172.21.15.148:0/3924676144 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000ba660 con 0x7efc080c6bd0 2023-10-30T13:33:48.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc00009b30 con 0x7efc080c6bd0 2023-10-30T13:33:48.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc08157e00 con 0x7efc080c6bd0 2023-10-30T13:33:48.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc080c21a0 con 0x7efc080c6bd0 2023-10-30T13:33:48.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.176+0000 7efbfdffb640 1 -- 172.21.15.148:0/3924676144 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efc000b6040 con 0x7efc080c6bd0 2023-10-30T13:33:48.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.180+0000 7efbfdffb640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efbec0664a0 0x7efbec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:48.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.180+0000 7efbfdffb640 1 -- 172.21.15.148:0/3924676144 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7efc00122540 con 0x7efc080c6bd0 2023-10-30T13:33:48.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.180+0000 7efc0d1b3640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efbec0664a0 0x7efbec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:48.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.180+0000 7efc0d1b3640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efbec0664a0 0x7efbec068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7efc04009c50 tx=0x7efc04009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:48.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.184+0000 7efbfdffb640 1 -- 172.21.15.148:0/3924676144 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efc080c21a0 con 0x7efc080c6bd0 2023-10-30T13:33:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.360+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc080c21a0 con 0x7efc080c6bd0 2023-10-30T13:33:48.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efbfdffb640 1 -- 172.21.15.148:0/3924676144 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6511 (secure 0 0 0) 0x7efc080c21a0 con 0x7efc080c6bd0 2023-10-30T13:33:48.358 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:48.359 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 98 2023-10-30T13:33:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efbec0664a0 msgr2=0x7efbec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efbec0664a0 0x7efbec068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7efc04009c50 tx=0x7efc04009190 comp rx=0 tx=0).stop 2023-10-30T13:33:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc080c6bd0 msgr2=0x7efc08154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc080c6bd0 0x7efc08154630 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7efc0001ce20 tx=0x7efc0001bf90 comp rx=0 tx=0).stop 2023-10-30T13:33:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 shutdown_connections 2023-10-30T13:33:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efbec0664a0 0x7efbec068960 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc080c6bd0 0x7efc08154630 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc08071110 0x7efc0814ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 --2- 172.21.15.148:0/3924676144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc0806f7a0 0x7efc08153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 >> 172.21.15.148:0/3924676144 conn(0x7efc080c7910 msgr2=0x7efc080c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 shutdown_connections 2023-10-30T13:33:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:48.364+0000 7efc0f417640 1 -- 172.21.15.148:0/3924676144 wait complete. 2023-10-30T13:33:48.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:33:50.380 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:50.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92b1bd3640 1 Processor -- start 2023-10-30T13:33:50.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92b1bd3640 1 -- start start 2023-10-30T13:33:50.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92b1bd3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92ac0c6f60 0x7f92ac1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:50.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92b1bd3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac0c0a50 con 0x7f92ac0c6b80 2023-10-30T13:33:50.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92b1bd3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac06f6e0 con 0x7f92ac0c05d0 2023-10-30T13:33:50.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92b1bd3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92ac06f9c0 con 0x7f92ac0c6f60 2023-10-30T13:33:50.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92abfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92ac0c6f60 0x7f92ac1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:50.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92abfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92ac0c6f60 0x7f92ac1566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33318/0 (socket says 172.21.15.148:33318) 2023-10-30T13:33:50.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92abfff640 1 -- 172.21.15.148:0/1262732464 learned_addr learned my addr 172.21.15.148:0/1262732464 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92abfff640 1 -- 172.21.15.148:0/1262732464 >> v1:172.21.15.150:6789/0 conn(0x7f92ac0c05d0 legacy=0x7f92ac13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92abfff640 1 -- 172.21.15.148:0/1262732464 >> v1:172.21.15.148:6789/0 conn(0x7f92ac0c6b80 legacy=0x7f92ac0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.528+0000 7f92abfff640 1 -- 172.21.15.148:0/1262732464 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac06fca0 con 0x7f92ac0c6f60 2023-10-30T13:33:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92abfff640 1 --2- 172.21.15.148:0/1262732464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6f60 0x7f92ac1566a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f92a4009e50 tx=0x7f92a401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f71d532172fe88 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92aa7fc640 1 -- 172.21.15.148:0/1262732464 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a40a57f0 con 0x7f92ac0c6f60 2023-10-30T13:33:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92aa7fc640 1 -- 172.21.15.148:0/1262732464 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a40a5950 con 0x7f92ac0c6f60 2023-10-30T13:33:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92aa7fc640 1 -- 172.21.15.148:0/1262732464 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a40a5ce0 con 0x7f92ac0c6f60 2023-10-30T13:33:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1262732464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6f60 msgr2=0x7f92ac1566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92b1bd3640 1 --2- 172.21.15.148:0/1262732464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6f60 0x7f92ac1566a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f92a4009e50 tx=0x7f92a401ca10 comp rx=0 tx=0).stop 2023-10-30T13:33:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1262732464 shutdown_connections 2023-10-30T13:33:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92b1bd3640 1 --2- 172.21.15.148:0/1262732464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6f60 0x7f92ac1566a0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1262732464 >> 172.21.15.148:0/1262732464 conn(0x7f92ac0c7910 msgr2=0x7f92ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1262732464 shutdown_connections 2023-10-30T13:33:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.532+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1262732464 wait complete. 2023-10-30T13:33:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 Processor -- start 2023-10-30T13:33:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 -- start start 2023-10-30T13:33:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92ac0c05d0 0x7f92ac154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6b80 0x7f92ac154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92ac0c6f60 0x7f92ac150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac13d330 con 0x7f92ac0c6b80 2023-10-30T13:33:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac1f0da0 con 0x7f92ac0c05d0 2023-10-30T13:33:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac0c1c60 con 0x7f92ac0c6f60 2023-10-30T13:33:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92aaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6b80 0x7f92ac154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92aaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6b80 0x7f92ac154a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33328/0 (socket says 172.21.15.148:33328) 2023-10-30T13:33:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92aaffd640 1 -- 172.21.15.148:0/1564170580 learned_addr learned my addr 172.21.15.148:0/1564170580 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92ab7fe640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92ac0c05d0 0x7f92ac154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92abfff640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92ac0c6f60 0x7f92ac150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92aaffd640 1 -- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92ac0c6f60 msgr2=0x7f92ac150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92aaffd640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92ac0c6f60 0x7f92ac150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92aaffd640 1 -- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92ac0c05d0 msgr2=0x7f92ac154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92aaffd640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92ac0c05d0 0x7f92ac154550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92aaffd640 1 -- 172.21.15.148:0/1564170580 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a4009b30 con 0x7f92ac0c6b80 2023-10-30T13:33:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92aaffd640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6b80 0x7f92ac154a90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f92a000c500 tx=0x7f92a000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f9293fff640 1 -- 172.21.15.148:0/1564170580 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a00052d0 con 0x7f92ac0c6b80 2023-10-30T13:33:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f9293fff640 1 -- 172.21.15.148:0/1564170580 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a0005430 con 0x7f92ac0c6b80 2023-10-30T13:33:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f9293fff640 1 -- 172.21.15.148:0/1564170580 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a0012830 con 0x7f92ac0c6b80 2023-10-30T13:33:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac13c720 con 0x7f92ac0c6b80 2023-10-30T13:33:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.536+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac13cce0 con 0x7f92ac0c6b80 2023-10-30T13:33:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.540+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac13d4e0 con 0x7f92ac0c6b80 2023-10-30T13:33:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.540+0000 7f9293fff640 1 -- 172.21.15.148:0/1564170580 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f92a0012990 con 0x7f92ac0c6b80 2023-10-30T13:33:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.540+0000 7f9293fff640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92980664a0 0x7f9298068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.540+0000 7f9293fff640 1 -- 172.21.15.148:0/1564170580 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f92a0083cf0 con 0x7f92ac0c6b80 2023-10-30T13:33:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.540+0000 7f92ab7fe640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92980664a0 0x7f9298068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.540+0000 7f92ab7fe640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92980664a0 0x7f9298068960 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f929c007fd0 tx=0x7f929c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.544+0000 7f9293fff640 1 -- 172.21.15.148:0/1564170580 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92a0050d70 con 0x7f92ac0c6b80 2023-10-30T13:33:50.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.724+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92ac04e170 con 0x7f92ac0c6b80 2023-10-30T13:33:50.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.724+0000 7f9293fff640 1 -- 172.21.15.148:0/1564170580 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6511 (secure 0 0 0) 0x7f92a0054390 con 0x7f92ac0c6b80 2023-10-30T13:33:50.720 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:50.721 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:50.721 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 98 2023-10-30T13:33:50.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92980664a0 msgr2=0x7f9298068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:50.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92980664a0 0x7f9298068960 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f929c007fd0 tx=0x7f929c009040 comp rx=0 tx=0).stop 2023-10-30T13:33:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6b80 msgr2=0x7f92ac154a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6b80 0x7f92ac154a90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f92a000c500 tx=0x7f92a000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:33:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 shutdown_connections 2023-10-30T13:33:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92980664a0 0x7f9298068960 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:50.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92ac0c6b80 0x7f92ac154a90 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92ac0c6f60 0x7f92ac150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 --2- 172.21.15.148:0/1564170580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92ac0c05d0 0x7f92ac154550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 >> 172.21.15.148:0/1564170580 conn(0x7f92ac0c7910 msgr2=0x7f92ac1532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 shutdown_connections 2023-10-30T13:33:50.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:50.728+0000 7f92b1bd3640 1 -- 172.21.15.148:0/1564170580 wait complete. 2023-10-30T13:33:50.741 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:33:52.742 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:52.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd1c8d6640 1 Processor -- start 2023-10-30T13:33:52.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd1c8d6640 1 -- start start 2023-10-30T13:33:52.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd1c8d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbd1806f7a0 0x7fbd1806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd1c8d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbd180c6bd0 0x7fbd18069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd1c8d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1806a3d0 con 0x7fbd18071110 2023-10-30T13:33:52.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd1c8d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd180700c0 con 0x7fbd180c6bd0 2023-10-30T13:33:52.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd1c8d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd18066cf0 con 0x7fbd1806f7a0 2023-10-30T13:33:52.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd0ffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbd180c6bd0 0x7fbd18069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd0ffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbd180c6bd0 0x7fbd18069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35916/0 (socket says 172.21.15.148:35916) 2023-10-30T13:33:52.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.896+0000 7fbd0ffff640 1 -- 172.21.15.148:0/811271684 learned_addr learned my addr 172.21.15.148:0/811271684 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:52.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd1659c640 1 --2- 172.21.15.148:0/811271684 >> v2:172.21.15.150:3300/0 conn(0x7fbd1806f7a0 0x7fbd1806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd0ffff640 1 -- 172.21.15.148:0/811271684 >> v2:172.21.15.150:3300/0 conn(0x7fbd1806f7a0 msgr2=0x7fbd1806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:52.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd0ffff640 1 --2- 172.21.15.148:0/811271684 >> v2:172.21.15.150:3300/0 conn(0x7fbd1806f7a0 0x7fbd1806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd0ffff640 1 -- 172.21.15.148:0/811271684 >> v1:172.21.15.148:6789/0 conn(0x7fbd18071110 legacy=0x7fbd18158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd0ffff640 1 -- 172.21.15.148:0/811271684 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd180c70d0 con 0x7fbd180c6bd0 2023-10-30T13:33:52.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd1659c640 1 --2- 172.21.15.148:0/811271684 >> v2:172.21.15.150:3300/0 conn(0x7fbd1806f7a0 0x7fbd1806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:52.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd0ffff640 1 --2- 172.21.15.148:0/811271684 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 0x7fbd18069e90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fbd1806c830 tx=0x7fbd0001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a179fdaa1cb6f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:52.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd15d9b640 1 -- 172.21.15.148:0/811271684 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd000a89f0 con 0x7fbd180c6bd0 2023-10-30T13:33:52.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd15d9b640 1 -- 172.21.15.148:0/811271684 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd000a8b50 con 0x7fbd180c6bd0 2023-10-30T13:33:52.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd15d9b640 1 -- 172.21.15.148:0/811271684 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd000b0440 con 0x7fbd180c6bd0 2023-10-30T13:33:52.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/811271684 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 msgr2=0x7fbd18069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.900+0000 7fbd1c8d6640 1 --2- 172.21.15.148:0/811271684 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 0x7fbd18069e90 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fbd1806c830 tx=0x7fbd0001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/811271684 shutdown_connections 2023-10-30T13:33:52.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 --2- 172.21.15.148:0/811271684 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 0x7fbd18069e90 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 --2- 172.21.15.148:0/811271684 >> v2:172.21.15.150:3300/0 conn(0x7fbd1806f7a0 0x7fbd1806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/811271684 >> 172.21.15.148:0/811271684 conn(0x7fbd180c7910 msgr2=0x7fbd180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:52.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/811271684 shutdown_connections 2023-10-30T13:33:52.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/811271684 wait complete. 2023-10-30T13:33:52.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 Processor -- start 2023-10-30T13:33:52.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 -- start start 2023-10-30T13:33:52.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd1806f7a0 0x7fbd180af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd18071110 0x7fbd181eee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 0x7fbd180ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd181495a0 con 0x7fbd180c6bd0 2023-10-30T13:33:52.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd181f2380 con 0x7fbd18071110 2023-10-30T13:33:52.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1c8d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd180ae420 con 0x7fbd1806f7a0 2023-10-30T13:33:52.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 0x7fbd180ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 0x7fbd180ca390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35920/0 (socket says 172.21.15.148:35920) 2023-10-30T13:33:52.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd16d9d640 1 -- 172.21.15.148:0/855773475 learned_addr learned my addr 172.21.15.148:0/855773475 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:52.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd0ffff640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd18071110 0x7fbd181eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1659c640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd1806f7a0 0x7fbd180af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd16d9d640 1 -- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd1806f7a0 msgr2=0x7fbd180af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd16d9d640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd1806f7a0 0x7fbd180af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd16d9d640 1 -- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd18071110 msgr2=0x7fbd181eee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:52.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd16d9d640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd18071110 0x7fbd181eee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:52.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd16d9d640 1 -- 172.21.15.148:0/855773475 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd000094d0 con 0x7fbd180c6bd0 2023-10-30T13:33:52.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd1659c640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd1806f7a0 0x7fbd180af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:52.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.904+0000 7fbd16d9d640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 0x7fbd180ca390 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000c730 tx=0x7fbd1000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:52.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd0effd640 1 -- 172.21.15.148:0/855773475 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10003b80 con 0x7fbd180c6bd0 2023-10-30T13:33:52.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd0effd640 1 -- 172.21.15.148:0/855773475 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd10003ce0 con 0x7fbd180c6bd0 2023-10-30T13:33:52.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd0effd640 1 -- 172.21.15.148:0/855773475 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1000aa30 con 0x7fbd180c6bd0 2023-10-30T13:33:52.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd180ae670 con 0x7fbd180c6bd0 2023-10-30T13:33:52.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd181f25b0 con 0x7fbd180c6bd0 2023-10-30T13:33:52.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd1804e170 con 0x7fbd180c6bd0 2023-10-30T13:33:52.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd0effd640 1 -- 172.21.15.148:0/855773475 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd1000ab90 con 0x7fbd180c6bd0 2023-10-30T13:33:52.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd0effd640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcf80664a0 0x7fbcf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:52.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd0effd640 1 -- 172.21.15.148:0/855773475 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6800+0+0 (secure 0 0 0) 0x7fbd10083fd0 con 0x7fbd180c6bd0 2023-10-30T13:33:52.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.908+0000 7fbd1659c640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcf80664a0 0x7fbcf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:52.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.912+0000 7fbd1659c640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcf80664a0 0x7fbcf8068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800b990 tx=0x7fbd08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:52.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:52.912+0000 7fbd0effd640 1 -- 172.21.15.148:0/855773475 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd10051020 con 0x7fbd180c6bd0 2023-10-30T13:33:53.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.096+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd180c8400 con 0x7fbd180c6bd0 2023-10-30T13:33:53.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.096+0000 7fbd0effd640 1 -- 172.21.15.148:0/855773475 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6511 (secure 0 0 0) 0x7fbd10054640 con 0x7fbd180c6bd0 2023-10-30T13:33:53.094 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:53.094 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:32:29.984268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6546":{"gid":6546,"name":"c","rank":0,"incarnation":93,"state":"up:active","state_seq":36,"addr":"172.21.15.148:6837/3232618434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3232618434},{"type":"v1","addr":"172.21.15.148:6837","nonce":3232618434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:53.095 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 98 2023-10-30T13:33:53.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcf80664a0 msgr2=0x7fbcf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:53.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcf80664a0 0x7fbcf8068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800b990 tx=0x7fbd08008040 comp rx=0 tx=0).stop 2023-10-30T13:33:53.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 msgr2=0x7fbd180ca390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:53.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 0x7fbd180ca390 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000c730 tx=0x7fbd1000cbf0 comp rx=0 tx=0).stop 2023-10-30T13:33:53.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 shutdown_connections 2023-10-30T13:33:53.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcf80664a0 0x7fbcf8068960 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:53.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd180c6bd0 0x7fbd180ca390 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:53.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd18071110 0x7fbd181eee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:53.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 --2- 172.21.15.148:0/855773475 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbd1806f7a0 0x7fbd180af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:53.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 >> 172.21.15.148:0/855773475 conn(0x7fbd180c7910 msgr2=0x7fbd180c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:53.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 shutdown_connections 2023-10-30T13:33:53.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:53.100+0000 7fbd1c8d6640 1 -- 172.21.15.148:0/855773475 wait complete. 2023-10-30T13:33:53.114 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:33:55.115 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:55.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38759e640 1 Processor -- start 2023-10-30T13:33:55.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38759e640 1 -- start start 2023-10-30T13:33:55.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38759e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb3881538f0 0x7fb388159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38759e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb3880cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38759e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3880ca6a0 con 0x7fb38815cd50 2023-10-30T13:33:55.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38759e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38815d300 con 0x7fb3881538f0 2023-10-30T13:33:55.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38759e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38815d5e0 con 0x7fb3880cefa0 2023-10-30T13:33:55.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb386d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb3880cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb3881538f0 0x7fb388159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb386d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb3880cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:36662/0 (socket says 172.21.15.148:36662) 2023-10-30T13:33:55.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb386d9d640 1 -- 172.21.15.148:0/867964298 learned_addr learned my addr 172.21.15.148:0/867964298 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:55.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38659c640 1 -- 172.21.15.148:0/867964298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 msgr2=0x7fb3880cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:55.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38659c640 1 --2- 172.21.15.148:0/867964298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb3880cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38659c640 1 -- 172.21.15.148:0/867964298 >> v1:172.21.15.150:6789/0 conn(0x7fb38815cd50 legacy=0x7fb3880c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb38659c640 1 -- 172.21.15.148:0/867964298 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38815d8c0 con 0x7fb3881538f0 2023-10-30T13:33:55.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.268+0000 7fb385d9b640 1 -- 172.21.15.148:0/867964298 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777807033 0 0) 0x7fb3880ca6a0 con 0x7fb38815cd50 2023-10-30T13:33:55.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb38659c640 1 --2- 172.21.15.148:0/867964298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3881538f0 0x7fb388159fb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb3700097c0 tx=0x7fb37001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8abd44c6f4ece29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:55.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb385d9b640 1 -- 172.21.15.148:0/867964298 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3700a8970 con 0x7fb3881538f0 2023-10-30T13:33:55.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb385d9b640 1 -- 172.21.15.148:0/867964298 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3700a8ad0 con 0x7fb3881538f0 2023-10-30T13:33:55.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb385d9b640 1 -- 172.21.15.148:0/867964298 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3700a8e60 con 0x7fb3881538f0 2023-10-30T13:33:55.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb38759e640 1 -- 172.21.15.148:0/867964298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3881538f0 msgr2=0x7fb388159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb38759e640 1 --2- 172.21.15.148:0/867964298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3881538f0 0x7fb388159fb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb3700097c0 tx=0x7fb37001ec70 comp rx=0 tx=0).stop 2023-10-30T13:33:55.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb38759e640 1 -- 172.21.15.148:0/867964298 shutdown_connections 2023-10-30T13:33:55.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb38759e640 1 --2- 172.21.15.148:0/867964298 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb3880cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb38759e640 1 --2- 172.21.15.148:0/867964298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3881538f0 0x7fb388159fb0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb38759e640 1 -- 172.21.15.148:0/867964298 >> 172.21.15.148:0/867964298 conn(0x7fb38806a640 msgr2=0x7fb388065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:55.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb38759e640 1 -- 172.21.15.148:0/867964298 shutdown_connections 2023-10-30T13:33:55.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.272+0000 7fb38759e640 1 -- 172.21.15.148:0/867964298 wait complete. 2023-10-30T13:33:55.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 Processor -- start 2023-10-30T13:33:55.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 -- start start 2023-10-30T13:33:55.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb38814f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3881538f0 0x7fb3881f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb38815cd50 0x7fb388145d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38815a920 con 0x7fb3881538f0 2023-10-30T13:33:55.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb388152980 con 0x7fb38815cd50 2023-10-30T13:33:55.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb388146250 con 0x7fb3880cefa0 2023-10-30T13:33:55.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3881538f0 0x7fb3881f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3881538f0 0x7fb3881f2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35924/0 (socket says 172.21.15.148:35924) 2023-10-30T13:33:55.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37dd9b640 1 -- 172.21.15.148:0/1144319284 learned_addr learned my addr 172.21.15.148:0/1144319284 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:55.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb386d9d640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb38815cd50 0x7fb388145d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38659c640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb38814f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37dd9b640 1 -- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 msgr2=0x7fb38814f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37dd9b640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb38814f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37dd9b640 1 -- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb38815cd50 msgr2=0x7fb388145d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37dd9b640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb38815cd50 0x7fb388145d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37dd9b640 1 -- 172.21.15.148:0/1144319284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb370009420 con 0x7fb3881538f0 2023-10-30T13:33:55.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38659c640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb38814f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:33:55.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37dd9b640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3881538f0 0x7fb3881f2310 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb378001ec0 tx=0x7fb37800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:55.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37f7fe640 1 -- 172.21.15.148:0/1144319284 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3780054e0 con 0x7fb3881538f0 2023-10-30T13:33:55.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37f7fe640 1 -- 172.21.15.148:0/1144319284 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb378003600 con 0x7fb3881538f0 2023-10-30T13:33:55.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb37f7fe640 1 -- 172.21.15.148:0/1144319284 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3780039b0 con 0x7fb3881538f0 2023-10-30T13:33:55.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38815ab50 con 0x7fb3881538f0 2023-10-30T13:33:55.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.276+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb388159370 con 0x7fb3881538f0 2023-10-30T13:33:55.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.280+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb388152af0 con 0x7fb3881538f0 2023-10-30T13:33:55.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.280+0000 7fb37f7fe640 1 -- 172.21.15.148:0/1144319284 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb378003b10 con 0x7fb3881538f0 2023-10-30T13:33:55.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.280+0000 7fb37f7fe640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3680664d0 0x7fb368068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.280+0000 7fb37f7fe640 1 -- 172.21.15.148:0/1144319284 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fb378083940 con 0x7fb3881538f0 2023-10-30T13:33:55.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.280+0000 7fb38659c640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3680664d0 0x7fb368068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.280+0000 7fb38659c640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3680664d0 0x7fb368068990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb3700b2000 tx=0x7fb3700aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:55.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.284+0000 7fb37f7fe640 1 -- 172.21.15.148:0/1144319284 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3780508a0 con 0x7fb3881538f0 2023-10-30T13:33:55.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.464+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb38804e170 con 0x7fb3881538f0 2023-10-30T13:33:55.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.464+0000 7fb37f7fe640 1 -- 172.21.15.148:0/1144319284 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5716 (secure 0 0 0) 0x7fb378017070 con 0x7fb3881538f0 2023-10-30T13:33:55.461 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:55.461 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:33:54.741421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:55.461 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 99 2023-10-30T13:33:55.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3680664d0 msgr2=0x7fb368068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3680664d0 0x7fb368068990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb3700b2000 tx=0x7fb3700aa070 comp rx=0 tx=0).stop 2023-10-30T13:33:55.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3881538f0 msgr2=0x7fb3881f2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3881538f0 0x7fb3881f2310 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb378001ec0 tx=0x7fb37800c8e0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 shutdown_connections 2023-10-30T13:33:55.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb3680664d0 0x7fb368068990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3881538f0 0x7fb3881f2310 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb38815cd50 0x7fb388145d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 --2- 172.21.15.148:0/1144319284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb3880cefa0 0x7fb38814f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 >> 172.21.15.148:0/1144319284 conn(0x7fb38806a640 msgr2=0x7fb3880caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:55.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 shutdown_connections 2023-10-30T13:33:55.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.468+0000 7fb38759e640 1 -- 172.21.15.148:0/1144319284 wait complete. 2023-10-30T13:33:55.482 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:33:55.482 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:33:55.482 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:55.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.624+0000 7f5555091640 1 Processor -- start 2023-10-30T13:33:55.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.624+0000 7f5555091640 1 -- start start 2023-10-30T13:33:55.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f5555091640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5550071110 0x7f555006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f5555091640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5550066cf0 con 0x7f555006f7a0 2023-10-30T13:33:55.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f5555091640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55500c70d0 con 0x7f55500c6bd0 2023-10-30T13:33:55.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f5555091640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55500700c0 con 0x7f5550071110 2023-10-30T13:33:55.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554ed9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f555006f7a0 0x7f5550069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41546/0 (socket says 172.21.15.148:41546) 2023-10-30T13:33:55.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554ed9d640 1 -- 172.21.15.148:0/4021805824 learned_addr learned my addr 172.21.15.148:0/4021805824 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:33:55.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554f59e640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5550071110 0x7f555006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392163160 0 0) 0x7f5550066cf0 con 0x7f555006f7a0 2023-10-30T13:33:55.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f553c003530 con 0x7f555006f7a0 2023-10-30T13:33:55.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759316905 0 0) 0x7f55500c70d0 con 0x7f55500c6bd0 2023-10-30T13:33:55.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5550066cf0 con 0x7f55500c6bd0 2023-10-30T13:33:55.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251013020 0 0) 0x7f553c003530 con 0x7f555006f7a0 2023-10-30T13:33:55.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5550071110 msgr2=0x7f555006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:55.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5550071110 0x7f555006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 >> v1:172.21.15.150:6789/0 conn(0x7f55500c6bd0 legacy=0x7f5550158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55500c7390 con 0x7f555006f7a0 2023-10-30T13:33:55.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f5538001680 con 0x7f555006f7a0 2023-10-30T13:33:55.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 >> v1:172.21.15.148:6789/0 conn(0x7f555006f7a0 legacy=0x7f5550069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f553c003b40 0x7f553c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f553c0047d0 0x7f553c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5550071110 0x7f553c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55500700c0 con 0x7f553c0047d0 2023-10-30T13:33:55.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c00ba40 con 0x7f553c003b40 2023-10-30T13:33:55.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c00bbd0 con 0x7f5550071110 2023-10-30T13:33:55.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554659c640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f553c0047d0 0x7f553c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554ed9d640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f553c003b40 0x7f553c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554f59e640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5550071110 0x7f553c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554659c640 1 -- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5550071110 msgr2=0x7f553c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:55.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554659c640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5550071110 0x7f553c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554659c640 1 -- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f553c003b40 msgr2=0x7f553c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:55.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554659c640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f553c003b40 0x7f553c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.628+0000 7f554659c640 1 -- 172.21.15.148:0/4021805824 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c003530 con 0x7f553c0047d0 2023-10-30T13:33:55.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f554659c640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f553c0047d0 0x7f553c004c50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f55500c7000 tx=0x7f554001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65097a1b8e3b0e55 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:55.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5538007ea0 con 0x7f553c0047d0 2023-10-30T13:33:55.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55400052f0 con 0x7f553c0047d0 2023-10-30T13:33:55.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f554e59c640 1 -- 172.21.15.148:0/4021805824 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540003600 con 0x7f553c0047d0 2023-10-30T13:33:55.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f5555091640 1 -- 172.21.15.148:0/4021805824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f553c0047d0 msgr2=0x7f553c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f5555091640 1 --2- 172.21.15.148:0/4021805824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f553c0047d0 0x7f553c004c50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f55500c7000 tx=0x7f554001c4e0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f554f59e640 1 -- 172.21.15.148:0/4021805824 reap_dead start 2023-10-30T13:33:55.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f5555091640 1 -- 172.21.15.148:0/4021805824 shutdown_connections 2023-10-30T13:33:55.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f5555091640 1 -- 172.21.15.148:0/4021805824 >> 172.21.15.148:0/4021805824 conn(0x7f55500c7910 msgr2=0x7f55500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:55.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f5555091640 1 -- 172.21.15.148:0/4021805824 shutdown_connections 2023-10-30T13:33:55.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.632+0000 7f5555091640 1 -- 172.21.15.148:0/4021805824 wait complete. 2023-10-30T13:33:55.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 Processor -- start 2023-10-30T13:33:55.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 -- start start 2023-10-30T13:33:55.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f553c0047d0 0x7f55500af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55500c0dc0 0x7f5550149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f555014a130 0x7f55501f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55500ae7c0 con 0x7f555014a130 2023-10-30T13:33:55.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55500aed80 con 0x7f55500c0dc0 2023-10-30T13:33:55.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55500ad8d0 con 0x7f553c0047d0 2023-10-30T13:33:55.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554f59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f555014a130 0x7f55501f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554f59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f555014a130 0x7f55501f1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35944/0 (socket says 172.21.15.148:35944) 2023-10-30T13:33:55.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554f59e640 1 -- 172.21.15.148:0/2443240579 learned_addr learned my addr 172.21.15.148:0/2443240579 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554659c640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55500c0dc0 0x7f5550149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554ed9d640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f553c0047d0 0x7f55500af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554f59e640 1 -- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f553c0047d0 msgr2=0x7f55500af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554f59e640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f553c0047d0 0x7f55500af210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554f59e640 1 -- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55500c0dc0 msgr2=0x7f5550149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554f59e640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55500c0dc0 0x7f5550149bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554f59e640 1 -- 172.21.15.148:0/2443240579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55500adb70 con 0x7f555014a130 2023-10-30T13:33:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f554f59e640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f555014a130 0x7f55501f1e40 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f55500c6fb0 tx=0x7f554800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5547fff640 1 -- 172.21.15.148:0/2443240579 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554800e040 con 0x7f555014a130 2023-10-30T13:33:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5547fff640 1 -- 172.21.15.148:0/2443240579 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554800f040 con 0x7f555014a130 2023-10-30T13:33:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5547fff640 1 -- 172.21.15.148:0/2443240579 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5548015670 con 0x7f555014a130 2023-10-30T13:33:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55501ef3f0 con 0x7f555014a130 2023-10-30T13:33:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55500aea10 con 0x7f555014a130 2023-10-30T13:33:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.636+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55501ef8a0 con 0x7f555014a130 2023-10-30T13:33:55.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.640+0000 7f5547fff640 1 -- 172.21.15.148:0/2443240579 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f55480157d0 con 0x7f555014a130 2023-10-30T13:33:55.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.640+0000 7f5547fff640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f55300664a0 0x7f5530068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:55.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.640+0000 7f5547fff640 1 -- 172.21.15.148:0/2443240579 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f5548084bc0 con 0x7f555014a130 2023-10-30T13:33:55.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.640+0000 7f554ed9d640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f55300664a0 0x7f5530068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:55.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.640+0000 7f554ed9d640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f55300664a0 0x7f5530068960 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5538001060 tx=0x7f5538009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:55.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.644+0000 7f5547fff640 1 -- 172.21.15.148:0/2443240579 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5548051a70 con 0x7f555014a130 2023-10-30T13:33:55.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.828+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55500aef30 con 0x7f555014a130 2023-10-30T13:33:55.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.828+0000 7f5547fff640 1 -- 172.21.15.148:0/2443240579 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f55500aef30 con 0x7f555014a130 2023-10-30T13:33:55.824 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:55.824 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:33:54.741421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:55.824 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 99 2023-10-30T13:33:55.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.828+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f55300664a0 msgr2=0x7f5530068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.828+0000 7f5555091640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f55300664a0 0x7f5530068960 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5538001060 tx=0x7f5538009040 comp rx=0 tx=0).stop 2023-10-30T13:33:55.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.828+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f555014a130 msgr2=0x7f55501f1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:55.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.828+0000 7f5555091640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f555014a130 0x7f55501f1e40 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f55500c6fb0 tx=0x7f554800f580 comp rx=0 tx=0).stop 2023-10-30T13:33:55.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.832+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 shutdown_connections 2023-10-30T13:33:55.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.832+0000 7f5555091640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f55300664a0 0x7f5530068960 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.832+0000 7f5555091640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f555014a130 0x7f55501f1e40 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.832+0000 7f5555091640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55500c0dc0 0x7f5550149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.832+0000 7f5555091640 1 --2- 172.21.15.148:0/2443240579 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f553c0047d0 0x7f55500af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:55.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.832+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 >> 172.21.15.148:0/2443240579 conn(0x7f55500c7910 msgr2=0x7f55500c8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:55.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.832+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 shutdown_connections 2023-10-30T13:33:55.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:55.832+0000 7f5555091640 1 -- 172.21.15.148:0/2443240579 wait complete. 2023-10-30T13:33:55.845 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 99, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6729": { "addr": "172.21.15.148:6835/1386276781", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1386276781, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1386276781, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6729, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:33:54.741421+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6729 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:33:57.846 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:33:57.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e8d86b640 1 Processor -- start 2023-10-30T13:33:57.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e8d86b640 1 -- start start 2023-10-30T13:33:57.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e8d86b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3e881538a0 0x7f3e8815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:57.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e8d86b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880cf160 0x7f3e880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:57.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e8d86b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e880cb800 con 0x7f3e880ca0e0 2023-10-30T13:33:57.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e8d86b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8815dc30 con 0x7f3e881538a0 2023-10-30T13:33:57.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e8d86b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e880cb330 con 0x7f3e880cf160 2023-10-30T13:33:57.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e86ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3e881538a0 0x7f3e8815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:57.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e86ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3e881538a0 0x7f3e8815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35956/0 (socket says 172.21.15.148:35956) 2023-10-30T13:33:57.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e86ffd640 1 -- 172.21.15.148:0/680397368 learned_addr learned my addr 172.21.15.148:0/680397368 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:57.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e877fe640 1 --2- 172.21.15.148:0/680397368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880cf160 0x7f3e880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:57.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e86ffd640 1 -- 172.21.15.148:0/680397368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880cf160 msgr2=0x7f3e880cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:33:57.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e86ffd640 1 --2- 172.21.15.148:0/680397368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880cf160 0x7f3e880cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:57.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e86ffd640 1 -- 172.21.15.148:0/680397368 >> v1:172.21.15.148:6789/0 conn(0x7f3e880ca0e0 legacy=0x7f3e8815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:57.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.000+0000 7f3e86ffd640 1 -- 172.21.15.148:0/680397368 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e880cad70 con 0x7f3e881538a0 2023-10-30T13:33:58.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e86ffd640 1 --2- 172.21.15.148:0/680397368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 0x7f3e8815d6f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009870 tx=0x7f3e7001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d580bee6995ba5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:58.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e867fc640 1 -- 172.21.15.148:0/680397368 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e700a8a30 con 0x7f3e881538a0 2023-10-30T13:33:58.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e867fc640 1 -- 172.21.15.148:0/680397368 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e700a8b90 con 0x7f3e881538a0 2023-10-30T13:33:58.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e867fc640 1 -- 172.21.15.148:0/680397368 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e700b0440 con 0x7f3e881538a0 2023-10-30T13:33:58.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e8d86b640 1 -- 172.21.15.148:0/680397368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 msgr2=0x7f3e8815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:58.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e8d86b640 1 --2- 172.21.15.148:0/680397368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 0x7f3e8815d6f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009870 tx=0x7f3e7001ed30 comp rx=0 tx=0).stop 2023-10-30T13:33:58.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e8d86b640 1 -- 172.21.15.148:0/680397368 shutdown_connections 2023-10-30T13:33:58.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e8d86b640 1 --2- 172.21.15.148:0/680397368 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880cf160 0x7f3e880cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:58.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e8d86b640 1 --2- 172.21.15.148:0/680397368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 0x7f3e8815d6f0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:58.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e8d86b640 1 -- 172.21.15.148:0/680397368 >> 172.21.15.148:0/680397368 conn(0x7f3e8806bfe0 msgr2=0x7f3e8806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:58.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e8d86b640 1 -- 172.21.15.148:0/680397368 shutdown_connections 2023-10-30T13:33:58.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.004+0000 7f3e8d86b640 1 -- 172.21.15.148:0/680397368 wait complete. 2023-10-30T13:33:58.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 Processor -- start 2023-10-30T13:33:58.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 -- start start 2023-10-30T13:33:58.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880ca0e0 0x7f3e881e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:58.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e880cf160 0x7f3e8815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:58.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 0x7f3e8815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:58.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8814d4f0 con 0x7f3e881538a0 2023-10-30T13:33:58.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e881e4d40 con 0x7f3e880cf160 2023-10-30T13:33:58.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8815b2a0 con 0x7f3e880ca0e0 2023-10-30T13:33:58.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e877fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 0x7f3e8815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:58.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e877fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 0x7f3e8815bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35968/0 (socket says 172.21.15.148:35968) 2023-10-30T13:33:58.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e877fe640 1 -- 172.21.15.148:0/1910869785 learned_addr learned my addr 172.21.15.148:0/1910869785 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:33:58.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e86ffd640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880ca0e0 0x7f3e881e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:58.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e7e7fc640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e880cf160 0x7f3e8815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e877fe640 1 -- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880ca0e0 msgr2=0x7f3e881e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e877fe640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880ca0e0 0x7f3e881e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e877fe640 1 -- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e880cf160 msgr2=0x7f3e8815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e877fe640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e880cf160 0x7f3e8815b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:58.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e877fe640 1 -- 172.21.15.148:0/1910869785 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e700094d0 con 0x7f3e881538a0 2023-10-30T13:33:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e877fe640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 0x7f3e8815bb70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3e80007d90 tx=0x7f3e8000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e84ff9640 1 -- 172.21.15.148:0/1910869785 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80014420 con 0x7f3e881538a0 2023-10-30T13:33:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e84ff9640 1 -- 172.21.15.148:0/1910869785 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e800145a0 con 0x7f3e881538a0 2023-10-30T13:33:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e84ff9640 1 -- 172.21.15.148:0/1910869785 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e80014950 con 0x7f3e881538a0 2023-10-30T13:33:58.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e880cef10 con 0x7f3e881538a0 2023-10-30T13:33:58.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.008+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e880c50b0 con 0x7f3e881538a0 2023-10-30T13:33:58.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.012+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e880b2a50 con 0x7f3e881538a0 2023-10-30T13:33:58.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.012+0000 7f3e84ff9640 1 -- 172.21.15.148:0/1910869785 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e80014ab0 con 0x7f3e881538a0 2023-10-30T13:33:58.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.012+0000 7f3e84ff9640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e680664a0 0x7f3e68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:33:58.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.012+0000 7f3e84ff9640 1 -- 172.21.15.148:0/1910869785 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f3e80084c60 con 0x7f3e881538a0 2023-10-30T13:33:58.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.012+0000 7f3e86ffd640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e680664a0 0x7f3e68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:33:58.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.012+0000 7f3e86ffd640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e680664a0 0x7f3e68068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3e700b2000 tx=0x7f3e70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:33:58.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.016+0000 7f3e84ff9640 1 -- 172.21.15.148:0/1910869785 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e80051ba0 con 0x7f3e881538a0 2023-10-30T13:33:58.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.196+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e881e4eb0 con 0x7f3e881538a0 2023-10-30T13:33:58.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e84ff9640 1 -- 172.21.15.148:0/1910869785 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f3e800551c0 con 0x7f3e881538a0 2023-10-30T13:33:58.194 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:33:58.194 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:33:54.741421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:33:58.195 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 99 2023-10-30T13:33:58.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e680664a0 msgr2=0x7f3e68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:58.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e680664a0 0x7f3e68068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3e700b2000 tx=0x7f3e70009040 comp rx=0 tx=0).stop 2023-10-30T13:33:58.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 msgr2=0x7f3e8815bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:33:58.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 0x7f3e8815bb70 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3e80007d90 tx=0x7f3e8000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:33:58.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 shutdown_connections 2023-10-30T13:33:58.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e680664a0 0x7f3e68068960 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:58.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e881538a0 0x7f3e8815bb70 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:58.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e880cf160 0x7f3e8815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:58.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 --2- 172.21.15.148:0/1910869785 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e880ca0e0 0x7f3e881e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:33:58.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 >> 172.21.15.148:0/1910869785 conn(0x7f3e8806bfe0 msgr2=0x7f3e88065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:33:58.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 shutdown_connections 2023-10-30T13:33:58.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:33:58.200+0000 7f3e8d86b640 1 -- 172.21.15.148:0/1910869785 wait complete. 2023-10-30T13:33:58.215 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:33:59.892 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:33:59.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:00.216 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:00.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.372+0000 7f2c02b3e640 1 Processor -- start 2023-10-30T13:34:00.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.372+0000 7f2c02b3e640 1 -- start start 2023-10-30T13:34:00.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c02b3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2bfc1538a0 0x7f2bfc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:00.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c02b3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 0x7f2bfc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:00.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c02b3e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc0cb800 con 0x7f2bfc0cf160 2023-10-30T13:34:00.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c02b3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bfc15dc30 con 0x7f2bfc1538a0 2023-10-30T13:34:00.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c02b3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0cb330 con 0x7f2bfc0ca0e0 2023-10-30T13:34:00.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c010db640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2bfc0cf160 0x7f2bfc0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41562/0 (socket says 172.21.15.148:41562) 2023-10-30T13:34:00.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c010db640 1 -- 172.21.15.148:0/2451769694 learned_addr learned my addr 172.21.15.148:0/2451769694 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:34:00.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2bfbfff640 1 --2- 172.21.15.148:0/2451769694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 0x7f2bfc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:00.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c008da640 1 --2- 172.21.15.148:0/2451769694 >> v2:172.21.15.150:3300/0 conn(0x7f2bfc1538a0 0x7f2bfc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:00.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2bfb7fe640 1 -- 172.21.15.148:0/2451769694 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262681546 0 0) 0x7f2bfc0cb800 con 0x7f2bfc0cf160 2023-10-30T13:34:00.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2bfb7fe640 1 -- 172.21.15.148:0/2451769694 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bec003530 con 0x7f2bfc0cf160 2023-10-30T13:34:00.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c008da640 1 -- 172.21.15.148:0/2451769694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 msgr2=0x7f2bfc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:00.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c008da640 1 --2- 172.21.15.148:0/2451769694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 0x7f2bfc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:00.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c008da640 1 -- 172.21.15.148:0/2451769694 >> v1:172.21.15.148:6789/0 conn(0x7f2bfc0cf160 legacy=0x7f2bfc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:00.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2bfb7fe640 1 -- 172.21.15.148:0/2451769694 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43726225 0 0) 0x7f2bec003530 con 0x7f2bfc0cf160 2023-10-30T13:34:00.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2c008da640 1 -- 172.21.15.148:0/2451769694 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc0cad70 con 0x7f2bfc1538a0 2023-10-30T13:34:00.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.376+0000 7f2bfbfff640 1 --2- 172.21.15.148:0/2451769694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 0x7f2bfc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:34:00.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2c008da640 1 --2- 172.21.15.148:0/2451769694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bfc1538a0 0x7f2bfc15d6f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00097c0 tx=0x7f2bf001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed00deee745841af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:00.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2bfb7fe640 1 -- 172.21.15.148:0/2451769694 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4001a00 con 0x7f2bfc1538a0 2023-10-30T13:34:00.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2bfb7fe640 1 -- 172.21.15.148:0/2451769694 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf00a85d0 con 0x7f2bfc1538a0 2023-10-30T13:34:00.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2bfb7fe640 1 -- 172.21.15.148:0/2451769694 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf00a8940 con 0x7f2bfc1538a0 2023-10-30T13:34:00.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2c02b3e640 1 -- 172.21.15.148:0/2451769694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bfc1538a0 msgr2=0x7f2bfc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:00.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2c02b3e640 1 --2- 172.21.15.148:0/2451769694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bfc1538a0 0x7f2bfc15d6f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00097c0 tx=0x7f2bf001eb90 comp rx=0 tx=0).stop 2023-10-30T13:34:00.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2c02b3e640 1 -- 172.21.15.148:0/2451769694 shutdown_connections 2023-10-30T13:34:00.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2c02b3e640 1 --2- 172.21.15.148:0/2451769694 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 0x7f2bfc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:00.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2c02b3e640 1 --2- 172.21.15.148:0/2451769694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bfc1538a0 0x7f2bfc15d6f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:00.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2c02b3e640 1 -- 172.21.15.148:0/2451769694 >> 172.21.15.148:0/2451769694 conn(0x7f2bfc06bfe0 msgr2=0x7f2bfc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:00.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2c02b3e640 1 -- 172.21.15.148:0/2451769694 shutdown_connections 2023-10-30T13:34:00.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.380+0000 7f2c02b3e640 1 -- 172.21.15.148:0/2451769694 wait complete. 2023-10-30T13:34:00.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 Processor -- start 2023-10-30T13:34:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 -- start start 2023-10-30T13:34:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 0x7f2bfc0b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bfc0cf160 0x7f2bfc144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bfc1538a0 0x7f2bfc1452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc145800 con 0x7f2bfc1538a0 2023-10-30T13:34:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0ce820 con 0x7f2bfc0cf160 2023-10-30T13:34:00.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0ceae0 con 0x7f2bfc0ca0e0 2023-10-30T13:34:00.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c010db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bfc1538a0 0x7f2bfc1452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:00.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c008da640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 0x7f2bfc0b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:00.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c010db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bfc1538a0 0x7f2bfc1452c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35970/0 (socket says 172.21.15.148:35970) 2023-10-30T13:34:00.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c010db640 1 -- 172.21.15.148:0/1107294460 learned_addr learned my addr 172.21.15.148:0/1107294460 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:00.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2bfbfff640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bfc0cf160 0x7f2bfc144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:00.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c010db640 1 -- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 msgr2=0x7f2bfc0b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:00.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c010db640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 0x7f2bfc0b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:00.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c010db640 1 -- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bfc0cf160 msgr2=0x7f2bfc144d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:00.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c010db640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bfc0cf160 0x7f2bfc144d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:00.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c010db640 1 -- 172.21.15.148:0/1107294460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be8009420 con 0x7f2bfc1538a0 2023-10-30T13:34:00.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c010db640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bfc1538a0 0x7f2bfc1452c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400cca0 tx=0x7f2bf400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2bf97fa640 1 -- 172.21.15.148:0/1107294460 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4012400 con 0x7f2bfc1538a0 2023-10-30T13:34:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2bf97fa640 1 -- 172.21.15.148:0/1107294460 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf4012560 con 0x7f2bfc1538a0 2023-10-30T13:34:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf0009420 con 0x7f2bfc1538a0 2023-10-30T13:34:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2bf97fa640 1 -- 172.21.15.148:0/1107294460 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4012910 con 0x7f2bfc1538a0 2023-10-30T13:34:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc0b3e80 con 0x7f2bfc1538a0 2023-10-30T13:34:00.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.384+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bfc0b2a50 con 0x7f2bfc1538a0 2023-10-30T13:34:00.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.388+0000 7f2bf97fa640 1 -- 172.21.15.148:0/1107294460 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2bf4012a70 con 0x7f2bfc1538a0 2023-10-30T13:34:00.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.388+0000 7f2bf97fa640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2be00664a0 0x7f2be0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:00.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.388+0000 7f2bf97fa640 1 -- 172.21.15.148:0/1107294460 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f2bf4084970 con 0x7f2bfc1538a0 2023-10-30T13:34:00.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.388+0000 7f2c008da640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2be00664a0 0x7f2be0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:00.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.392+0000 7f2c008da640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2be00664a0 0x7f2be0068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00b2000 tx=0x7f2bf00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:00.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.392+0000 7f2bf97fa640 1 -- 172.21.15.148:0/1107294460 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2bf4051a90 con 0x7f2bfc1538a0 2023-10-30T13:34:00.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.584+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bfc04e170 con 0x7f2bfc1538a0 2023-10-30T13:34:00.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2bf97fa640 1 -- 172.21.15.148:0/1107294460 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f2bf40550b0 con 0x7f2bfc1538a0 2023-10-30T13:34:00.582 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:00.582 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:33:54.741421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:00.582 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 99 2023-10-30T13:34:00.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2be00664a0 msgr2=0x7f2be0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:00.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2be00664a0 0x7f2be0068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00b2000 tx=0x7f2bf00aa070 comp rx=0 tx=0).stop 2023-10-30T13:34:00.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bfc1538a0 msgr2=0x7f2bfc1452c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:00.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bfc1538a0 0x7f2bfc1452c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400cca0 tx=0x7f2bf400a430 comp rx=0 tx=0).stop 2023-10-30T13:34:00.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 shutdown_connections 2023-10-30T13:34:00.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2be00664a0 0x7f2be0068960 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:00.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bfc1538a0 0x7f2bfc1452c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:00.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bfc0cf160 0x7f2bfc144d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:00.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 --2- 172.21.15.148:0/1107294460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bfc0ca0e0 0x7f2bfc0b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:00.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 >> 172.21.15.148:0/1107294460 conn(0x7f2bfc06bfe0 msgr2=0x7f2bfc1526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:00.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 shutdown_connections 2023-10-30T13:34:00.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:00.588+0000 7f2c02b3e640 1 -- 172.21.15.148:0/1107294460 wait complete. 2023-10-30T13:34:00.604 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:02.605 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f214353e640 1 Processor -- start 2023-10-30T13:34:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f214353e640 1 -- start start 2023-10-30T13:34:02.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f214353e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f213c1538a0 0x7f213c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:02.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f214353e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f213c0ca0e0 0x7f213c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:02.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f214353e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213c0cb800 con 0x7f213c0cf160 2023-10-30T13:34:02.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f214353e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213c15dc30 con 0x7f213c0ca0e0 2023-10-30T13:34:02.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f214353e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213c0cb330 con 0x7f213c1538a0 2023-10-30T13:34:02.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f2140ad9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f213c0ca0e0 0x7f213c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:02.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f2140ad9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f213c0ca0e0 0x7f213c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48990/0 (socket says 172.21.15.148:48990) 2023-10-30T13:34:02.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f2140ad9640 1 -- 172.21.15.148:0/4242985401 learned_addr learned my addr 172.21.15.148:0/4242985401 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:02.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f21412da640 1 --2- 172.21.15.148:0/4242985401 >> v2:172.21.15.150:3300/0 conn(0x7f213c1538a0 0x7f213c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:02.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f2140ad9640 1 -- 172.21.15.148:0/4242985401 >> v2:172.21.15.150:3300/0 conn(0x7f213c1538a0 msgr2=0x7f213c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:02.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f2140ad9640 1 --2- 172.21.15.148:0/4242985401 >> v2:172.21.15.150:3300/0 conn(0x7f213c1538a0 0x7f213c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f2140ad9640 1 -- 172.21.15.148:0/4242985401 >> v1:172.21.15.148:6789/0 conn(0x7f213c0cf160 legacy=0x7f213c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:02.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.756+0000 7f2140ad9640 1 -- 172.21.15.148:0/4242985401 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c0cad70 con 0x7f213c0ca0e0 2023-10-30T13:34:02.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f2140ad9640 1 --2- 172.21.15.148:0/4242985401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c0ca0e0 0x7f213c15a090 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f213c0cfd10 tx=0x7f212c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f8b73773a71941 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:02.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f2133fff640 1 -- 172.21.15.148:0/4242985401 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0a89f0 con 0x7f213c0ca0e0 2023-10-30T13:34:02.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f2133fff640 1 -- 172.21.15.148:0/4242985401 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c0a8b50 con 0x7f213c0ca0e0 2023-10-30T13:34:02.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f2133fff640 1 -- 172.21.15.148:0/4242985401 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c0b0440 con 0x7f213c0ca0e0 2023-10-30T13:34:02.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f214353e640 1 -- 172.21.15.148:0/4242985401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c0ca0e0 msgr2=0x7f213c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:02.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f214353e640 1 --2- 172.21.15.148:0/4242985401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c0ca0e0 0x7f213c15a090 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f213c0cfd10 tx=0x7f212c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f214353e640 1 -- 172.21.15.148:0/4242985401 shutdown_connections 2023-10-30T13:34:02.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f214353e640 1 --2- 172.21.15.148:0/4242985401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c0ca0e0 0x7f213c15a090 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f214353e640 1 --2- 172.21.15.148:0/4242985401 >> v2:172.21.15.150:3300/0 conn(0x7f213c1538a0 0x7f213c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f214353e640 1 -- 172.21.15.148:0/4242985401 >> 172.21.15.148:0/4242985401 conn(0x7f213c06bfe0 msgr2=0x7f213c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f214353e640 1 -- 172.21.15.148:0/4242985401 shutdown_connections 2023-10-30T13:34:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.760+0000 7f214353e640 1 -- 172.21.15.148:0/4242985401 wait complete. 2023-10-30T13:34:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 Processor -- start 2023-10-30T13:34:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 -- start start 2023-10-30T13:34:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f213c0ca0e0 0x7f213c140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f213c0cf160 0x7f213c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c1538a0 0x7f213c0cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c1e4380 con 0x7f213c1538a0 2023-10-30T13:34:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c158620 con 0x7f213c0cf160 2023-10-30T13:34:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213c159690 con 0x7f213c0ca0e0 2023-10-30T13:34:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2141adb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c1538a0 0x7f213c0cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2141adb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c1538a0 0x7f213c0cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48992/0 (socket says 172.21.15.148:48992) 2023-10-30T13:34:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2141adb640 1 -- 172.21.15.148:0/3559126499 learned_addr learned my addr 172.21.15.148:0/3559126499 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f21412da640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f213c0ca0e0 0x7f213c140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2140ad9640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f213c0cf160 0x7f213c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2141adb640 1 -- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f213c0ca0e0 msgr2=0x7f213c140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2141adb640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f213c0ca0e0 0x7f213c140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2141adb640 1 -- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f213c0cf160 msgr2=0x7f213c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2141adb640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f213c0cf160 0x7f213c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2141adb640 1 -- 172.21.15.148:0/3559126499 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c0094d0 con 0x7f213c1538a0 2023-10-30T13:34:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f21412da640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f213c0ca0e0 0x7f213c140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2141adb640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c1538a0 0x7f213c0cd4f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f213400c6f0 tx=0x7f213400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2131ffb640 1 -- 172.21.15.148:0/3559126499 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134003980 con 0x7f213c1538a0 2023-10-30T13:34:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2131ffb640 1 -- 172.21.15.148:0/3559126499 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2134003ae0 con 0x7f213c1538a0 2023-10-30T13:34:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f2131ffb640 1 -- 172.21.15.148:0/3559126499 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134003e90 con 0x7f213c1538a0 2023-10-30T13:34:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213c1e51e0 con 0x7f213c1538a0 2023-10-30T13:34:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.764+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213c1e5610 con 0x7f213c1538a0 2023-10-30T13:34:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.768+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213c0b2a50 con 0x7f213c1538a0 2023-10-30T13:34:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.768+0000 7f2131ffb640 1 -- 172.21.15.148:0/3559126499 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f213400a850 con 0x7f213c1538a0 2023-10-30T13:34:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.768+0000 7f2131ffb640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21200664d0 0x7f2120068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.768+0000 7f2131ffb640 1 -- 172.21.15.148:0/3559126499 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f2134083f70 con 0x7f213c1538a0 2023-10-30T13:34:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.772+0000 7f21412da640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21200664d0 0x7f2120068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.772+0000 7f21412da640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21200664d0 0x7f2120068990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f213c0cf070 tx=0x7f2138008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.776+0000 7f2131ffb640 1 -- 172.21.15.148:0/3559126499 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f21340b5090 con 0x7f213c1538a0 2023-10-30T13:34:02.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.992+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f213c158790 con 0x7f213c1538a0 2023-10-30T13:34:02.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.992+0000 7f2131ffb640 1 -- 172.21.15.148:0/3559126499 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f2134054a70 con 0x7f213c1538a0 2023-10-30T13:34:02.989 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:02.989 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:33:54.741421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:02.989 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 99 2023-10-30T13:34:02.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21200664d0 msgr2=0x7f2120068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21200664d0 0x7f2120068990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f213c0cf070 tx=0x7f2138008040 comp rx=0 tx=0).stop 2023-10-30T13:34:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c1538a0 msgr2=0x7f213c0cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c1538a0 0x7f213c0cd4f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f213400c6f0 tx=0x7f213400cbb0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 shutdown_connections 2023-10-30T13:34:02.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21200664d0 0x7f2120068990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f213c1538a0 0x7f213c0cd4f0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f213c0cf160 0x7f213c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 --2- 172.21.15.148:0/3559126499 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f213c0ca0e0 0x7f213c140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:02.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 >> 172.21.15.148:0/3559126499 conn(0x7f213c06bfe0 msgr2=0x7f213c0c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:02.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 shutdown_connections 2023-10-30T13:34:02.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:02.996+0000 7f214353e640 1 -- 172.21.15.148:0/3559126499 wait complete. 2023-10-30T13:34:03.011 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:05.013 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:05.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e79c9f640 1 Processor -- start 2023-10-30T13:34:05.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e79c9f640 1 -- start start 2023-10-30T13:34:05.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e79c9f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9e740cefa0 0x7f9e740cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:05.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e79c9f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7415a6c0 con 0x7f9e741538f0 2023-10-30T13:34:05.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e79c9f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740ca6a0 con 0x7f9e7415cd50 2023-10-30T13:34:05.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e79c9f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e740cf8f0 con 0x7f9e740cefa0 2023-10-30T13:34:05.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e78c9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9e741538f0 0x7f9e74159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55638/0 (socket says 172.21.15.148:55638) 2023-10-30T13:34:05.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e78c9d640 1 -- 172.21.15.148:0/3929666167 learned_addr learned my addr 172.21.15.148:0/3929666167 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:34:05.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e7949e640 1 --2- 172.21.15.148:0/3929666167 >> v2:172.21.15.150:3300/0 conn(0x7f9e740cefa0 0x7f9e740cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:05.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243194494 0 0) 0x7f9e7415a6c0 con 0x7f9e741538f0 2023-10-30T13:34:05.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e64003530 con 0x7f9e741538f0 2023-10-30T13:34:05.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4246798238 0 0) 0x7f9e64003530 con 0x7f9e741538f0 2023-10-30T13:34:05.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 >> v2:172.21.15.150:3300/0 conn(0x7f9e740cefa0 msgr2=0x7f9e740cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:05.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e737fe640 1 --2- 172.21.15.148:0/3929666167 >> v2:172.21.15.150:3300/0 conn(0x7f9e740cefa0 0x7f9e740cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 >> v1:172.21.15.150:6789/0 conn(0x7f9e7415cd50 legacy=0x7f9e740c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:05.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7415d300 con 0x7f9e741538f0 2023-10-30T13:34:05.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.164+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f9e68001680 con 0x7f9e741538f0 2023-10-30T13:34:05.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 >> v1:172.21.15.148:6789/0 conn(0x7f9e741538f0 legacy=0x7f9e74159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:05.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e737fe640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e64003980 0x7f9e64003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:05.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e737fe640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e64004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:05.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e737fe640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e64005190 0x7f9e6400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:05.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740cf8f0 con 0x7f9e640047d0 2023-10-30T13:34:05.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6400c040 con 0x7f9e64003980 2023-10-30T13:34:05.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6400c200 con 0x7f9e64005190 2023-10-30T13:34:05.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e73fff640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e64004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:05.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e78c9d640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e64003980 0x7f9e64003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:05.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e7949e640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e64005190 0x7f9e6400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:05.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e73fff640 1 -- 172.21.15.148:0/3929666167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e64005190 msgr2=0x7f9e6400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:05.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e73fff640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e64005190 0x7f9e6400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e73fff640 1 -- 172.21.15.148:0/3929666167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e64003980 msgr2=0x7f9e64003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:05.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e73fff640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e64003980 0x7f9e64003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e73fff640 1 -- 172.21.15.148:0/3929666167 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64003530 con 0x7f9e640047d0 2023-10-30T13:34:05.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e78c9d640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e64003980 0x7f9e64003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:05.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e7949e640 1 -- 172.21.15.148:0/3929666167 reap_dead start 2023-10-30T13:34:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e73fff640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e64004c50 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9e60001780 tx=0x7f9e6001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af1f8ed5c55439 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.168+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600052d0 con 0x7f9e640047d0 2023-10-30T13:34:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e600a7050 con 0x7f9e640047d0 2023-10-30T13:34:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e737fe640 1 -- 172.21.15.148:0/3929666167 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60003600 con 0x7f9e640047d0 2023-10-30T13:34:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 -- 172.21.15.148:0/3929666167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 msgr2=0x7f9e64004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e64004c50 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9e60001780 tx=0x7f9e6001c3e0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 -- 172.21.15.148:0/3929666167 shutdown_connections 2023-10-30T13:34:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 --2- 172.21.15.148:0/3929666167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e64004c50 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 -- 172.21.15.148:0/3929666167 >> 172.21.15.148:0/3929666167 conn(0x7f9e7406a640 msgr2=0x7f9e74065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 -- 172.21.15.148:0/3929666167 shutdown_connections 2023-10-30T13:34:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 -- 172.21.15.148:0/3929666167 wait complete. 2023-10-30T13:34:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 Processor -- start 2023-10-30T13:34:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 -- start start 2023-10-30T13:34:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e740b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e74157600 0x7f9e74159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7413e880 0x7f9e7413ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740b3e90 con 0x7f9e640047d0 2023-10-30T13:34:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740c5510 con 0x7f9e7413e880 2023-10-30T13:34:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.172+0000 7f9e79c9f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740c4f50 con 0x7f9e74157600 2023-10-30T13:34:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e78c9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e740b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e78c9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e740b3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49020/0 (socket says 172.21.15.148:49020) 2023-10-30T13:34:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e78c9d640 1 -- 172.21.15.148:0/1050951190 learned_addr learned my addr 172.21.15.148:0/1050951190 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e73fff640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e74157600 0x7f9e74159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e7949e640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7413e880 0x7f9e7413ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e78c9d640 1 -- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e74157600 msgr2=0x7f9e74159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e78c9d640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e74157600 0x7f9e74159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e78c9d640 1 -- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7413e880 msgr2=0x7f9e7413ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e78c9d640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7413e880 0x7f9e7413ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e78c9d640 1 -- 172.21.15.148:0/1050951190 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740c5230 con 0x7f9e640047d0 2023-10-30T13:34:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e7949e640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7413e880 0x7f9e7413ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e78c9d640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e740b3950 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800e7b0 tx=0x7f9e6800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e717fa640 1 -- 172.21.15.148:0/1050951190 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800c6c0 con 0x7f9e640047d0 2023-10-30T13:34:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e717fa640 1 -- 172.21.15.148:0/1050951190 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6800c820 con 0x7f9e640047d0 2023-10-30T13:34:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e740b4ac0 con 0x7f9e640047d0 2023-10-30T13:34:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e717fa640 1 -- 172.21.15.148:0/1050951190 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800cb90 con 0x7f9e640047d0 2023-10-30T13:34:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e741597d0 con 0x7f9e640047d0 2023-10-30T13:34:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e740b4f70 con 0x7f9e640047d0 2023-10-30T13:34:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.176+0000 7f9e717fa640 1 -- 172.21.15.148:0/1050951190 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e6800ccf0 con 0x7f9e640047d0 2023-10-30T13:34:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.180+0000 7f9e717fa640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9e580664a0 0x7f9e58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.180+0000 7f9e717fa640 1 -- 172.21.15.148:0/1050951190 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f9e68084290 con 0x7f9e640047d0 2023-10-30T13:34:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.180+0000 7f9e73fff640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9e580664a0 0x7f9e58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.180+0000 7f9e73fff640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9e580664a0 0x7f9e58068960 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9e60001750 tx=0x7f9e600011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.184+0000 7f9e717fa640 1 -- 172.21.15.148:0/1050951190 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e740b4f70 con 0x7f9e640047d0 2023-10-30T13:34:05.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.392+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e740b4f70 con 0x7f9e640047d0 2023-10-30T13:34:05.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e717fa640 1 -- 172.21.15.148:0/1050951190 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5716 (secure 0 0 0) 0x7f9e740b4f70 con 0x7f9e640047d0 2023-10-30T13:34:05.390 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:05.390 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:33:54.741421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:05.390 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 99 2023-10-30T13:34:05.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9e580664a0 msgr2=0x7f9e58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:05.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9e580664a0 0x7f9e58068960 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9e60001750 tx=0x7f9e600011a0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 msgr2=0x7f9e740b3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:05.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e740b3950 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800e7b0 tx=0x7f9e6800ec70 comp rx=0 tx=0).stop 2023-10-30T13:34:05.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 shutdown_connections 2023-10-30T13:34:05.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9e580664a0 0x7f9e58068960 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7413e880 0x7f9e7413ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9e74157600 0x7f9e74159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 --2- 172.21.15.148:0/1050951190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e640047d0 0x7f9e740b3950 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:05.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 >> 172.21.15.148:0/1050951190 conn(0x7f9e7406a640 msgr2=0x7f9e740c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:05.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 shutdown_connections 2023-10-30T13:34:05.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:05.396+0000 7f9e79c9f640 1 -- 172.21.15.148:0/1050951190 wait complete. 2023-10-30T13:34:05.411 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:07.412 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:07.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1fa1503640 1 Processor -- start 2023-10-30T13:34:07.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1fa1503640 1 -- start start 2023-10-30T13:34:07.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1fa1503640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15cd50 0x7f1f9c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:07.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1fa1503640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9c15a6c0 con 0x7f1f9c1538f0 2023-10-30T13:34:07.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1fa1503640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9c15d300 con 0x7f1f9c0cefa0 2023-10-30T13:34:07.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1fa1503640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c15d5e0 con 0x7f1f9c15cd50 2023-10-30T13:34:07.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f1f9c1538f0 0x7f1f9c159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55644/0 (socket says 172.21.15.148:55644) 2023-10-30T13:34:07.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9bfff640 1 -- 172.21.15.148:0/964795112 learned_addr learned my addr 172.21.15.148:0/964795112 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:34:07.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9b7fe640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15cd50 0x7f1f9c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:07.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458465234 0 0) 0x7f1f9c15a6c0 con 0x7f1f9c1538f0 2023-10-30T13:34:07.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f84003530 con 0x7f1f9c1538f0 2023-10-30T13:34:07.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2935742555 0 0) 0x7f1f84003530 con 0x7f1f9c1538f0 2023-10-30T13:34:07.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15cd50 msgr2=0x7f1f9c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:07.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15cd50 0x7f1f9c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:07.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 >> v1:172.21.15.150:6789/0 conn(0x7f1f9c0cefa0 legacy=0x7f1f9c1640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:07.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c15d8c0 con 0x7f1f9c1538f0 2023-10-30T13:34:07.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f1f8c001680 con 0x7f1f9c1538f0 2023-10-30T13:34:07.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 >> v1:172.21.15.148:6789/0 conn(0x7f1f9c1538f0 legacy=0x7f1f9c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:07.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f840039b0 0x7f1f84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f84004360 0x7f1f84004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15cd50 0x7f1f84005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c15d5e0 con 0x7f1f84004360 2023-10-30T13:34:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8400b9c0 con 0x7f1f840039b0 2023-10-30T13:34:07.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8400bb80 con 0x7f1f9c15cd50 2023-10-30T13:34:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9b7fe640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f84004360 0x7f1f84004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9bfff640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f840039b0 0x7f1f84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1fa0d02640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15cd50 0x7f1f84005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9b7fe640 1 -- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15cd50 msgr2=0x7f1f84005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9b7fe640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15cd50 0x7f1f84005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:07.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9b7fe640 1 -- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f840039b0 msgr2=0x7f1f84003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:07.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9b7fe640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f840039b0 0x7f1f84003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:07.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9b7fe640 1 -- 172.21.15.148:0/964795112 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84003530 con 0x7f1f84004360 2023-10-30T13:34:07.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.568+0000 7f1f9bfff640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f840039b0 0x7f1f84003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1f9b7fe640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f84004360 0x7f1f84004830 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f1f90009ae0 tx=0x7f1f9001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9332cf1f90b4b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f900a5640 con 0x7f1f84004360 2023-10-30T13:34:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f900a57a0 con 0x7f1f84004360 2023-10-30T13:34:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1f9affd640 1 -- 172.21.15.148:0/964795112 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f900a5b50 con 0x7f1f84004360 2023-10-30T13:34:07.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1fa1503640 1 -- 172.21.15.148:0/964795112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f84004360 msgr2=0x7f1f84004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1fa1503640 1 --2- 172.21.15.148:0/964795112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f84004360 0x7f1f84004830 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f1f90009ae0 tx=0x7f1f9001ca50 comp rx=0 tx=0).stop 2023-10-30T13:34:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1fa0d02640 1 -- 172.21.15.148:0/964795112 reap_dead start 2023-10-30T13:34:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1fa1503640 1 -- 172.21.15.148:0/964795112 shutdown_connections 2023-10-30T13:34:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1fa1503640 1 -- 172.21.15.148:0/964795112 >> 172.21.15.148:0/964795112 conn(0x7f1f9c06a640 msgr2=0x7f1f9c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1fa1503640 1 -- 172.21.15.148:0/964795112 shutdown_connections 2023-10-30T13:34:07.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.572+0000 7f1fa1503640 1 -- 172.21.15.148:0/964795112 wait complete. 2023-10-30T13:34:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 Processor -- start 2023-10-30T13:34:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 -- start start 2023-10-30T13:34:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f84004360 0x7f1f9c15be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15c7c0 0x7f1f9c1546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:07.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f9c14f380 0x7f1f9c0b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c154430 con 0x7f1f9c14f380 2023-10-30T13:34:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c15b9f0 con 0x7f1f84004360 2023-10-30T13:34:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c15ab00 con 0x7f1f9c15c7c0 2023-10-30T13:34:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa0d02640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f9c14f380 0x7f1f9c0b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa0d02640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f9c14f380 0x7f1f9c0b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49044/0 (socket says 172.21.15.148:49044) 2023-10-30T13:34:07.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa0d02640 1 -- 172.21.15.148:0/3601172375 learned_addr learned my addr 172.21.15.148:0/3601172375 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:07.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1f9b7fe640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15c7c0 0x7f1f9c1546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:07.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1f9bfff640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f84004360 0x7f1f9c15be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:07.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa0d02640 1 -- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15c7c0 msgr2=0x7f1f9c1546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:07.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa0d02640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15c7c0 0x7f1f9c1546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:07.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa0d02640 1 -- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f84004360 msgr2=0x7f1f9c15be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa0d02640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f84004360 0x7f1f9c15be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa0d02640 1 -- 172.21.15.148:0/3601172375 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c15ade0 con 0x7f1f9c14f380 2023-10-30T13:34:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa0d02640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f9c14f380 0x7f1f9c0b3a30 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c066500 tx=0x7f1f9400eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1f98ff9640 1 -- 172.21.15.148:0/3601172375 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94003980 con 0x7f1f9c14f380 2023-10-30T13:34:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1f98ff9640 1 -- 172.21.15.148:0/3601172375 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f94003ae0 con 0x7f1f9c14f380 2023-10-30T13:34:07.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9c15b0c0 con 0x7f1f9c14f380 2023-10-30T13:34:07.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1f98ff9640 1 -- 172.21.15.148:0/3601172375 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94003e90 con 0x7f1f9c14f380 2023-10-30T13:34:07.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.576+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9c15b680 con 0x7f1f9c14f380 2023-10-30T13:34:07.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.580+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9c04e170 con 0x7f1f9c14f380 2023-10-30T13:34:07.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.580+0000 7f1f98ff9640 1 -- 172.21.15.148:0/3601172375 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1f940050d0 con 0x7f1f9c14f380 2023-10-30T13:34:07.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.580+0000 7f1f98ff9640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f880664d0 0x7f1f88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:07.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.580+0000 7f1f98ff9640 1 -- 172.21.15.148:0/3601172375 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f1f94084240 con 0x7f1f9c14f380 2023-10-30T13:34:07.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.580+0000 7f1f9bfff640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f880664d0 0x7f1f88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:07.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.580+0000 7f1f9bfff640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f880664d0 0x7f1f88068990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c000c10 tx=0x7f1f8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:07.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.584+0000 7f1f98ff9640 1 -- 172.21.15.148:0/3601172375 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1f94054c10 con 0x7f1f9c14f380 2023-10-30T13:34:07.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.764+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f9c0b2290 con 0x7f1f9c14f380 2023-10-30T13:34:07.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1f98ff9640 1 -- 172.21.15.148:0/3601172375 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5722 (secure 0 0 0) 0x7f1f940547a0 con 0x7f1f9c14f380 2023-10-30T13:34:07.762 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:07.762 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:06.216068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:07.762 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 100 2023-10-30T13:34:07.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f880664d0 msgr2=0x7f1f88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:07.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f880664d0 0x7f1f88068990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c000c10 tx=0x7f1f8c009040 comp rx=0 tx=0).stop 2023-10-30T13:34:07.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f9c14f380 msgr2=0x7f1f9c0b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:07.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f9c14f380 0x7f1f9c0b3a30 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c066500 tx=0x7f1f9400eb70 comp rx=0 tx=0).stop 2023-10-30T13:34:07.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 shutdown_connections 2023-10-30T13:34:07.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f880664d0 0x7f1f88068990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:07.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f9c14f380 0x7f1f9c0b3a30 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:07.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f9c15c7c0 0x7f1f9c1546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:07.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 --2- 172.21.15.148:0/3601172375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f84004360 0x7f1f9c15be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:07.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 >> 172.21.15.148:0/3601172375 conn(0x7f1f9c06a640 msgr2=0x7f1f9c0cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:07.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 shutdown_connections 2023-10-30T13:34:07.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:07.768+0000 7f1fa1503640 1 -- 172.21.15.148:0/3601172375 wait complete. 2023-10-30T13:34:07.783 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 100, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6729": { "addr": "172.21.15.148:6835/1386276781", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1386276781, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1386276781, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6729, "incarnation": 99, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:34:06.216068+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6729 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:34:09.784 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:09.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.932+0000 7efda0a05640 1 Processor -- start 2023-10-30T13:34:09.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.932+0000 7efda0a05640 1 -- start start 2023-10-30T13:34:09.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efda0a05640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efd9c1538a0 0x7efd9c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:09.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efda0a05640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efd9c0ca0e0 0x7efd9c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:09.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efda0a05640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0cf160 0x7efd9c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:09.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efda0a05640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9c0cb800 con 0x7efd9c1538a0 2023-10-30T13:34:09.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efda0a05640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9c15dc30 con 0x7efd9c0ca0e0 2023-10-30T13:34:09.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efda0a05640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0cb330 con 0x7efd9c0cf160 2023-10-30T13:34:09.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9a59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efd9c1538a0 0x7efd9c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:09.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9a59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efd9c1538a0 0x7efd9c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49054/0 (socket says 172.21.15.148:49054) 2023-10-30T13:34:09.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9a59c640 1 -- 172.21.15.148:0/3583485571 learned_addr learned my addr 172.21.15.148:0/3583485571 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:09.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd99d9b640 1 --2- 172.21.15.148:0/3583485571 >> v2:172.21.15.150:3300/0 conn(0x7efd9c0ca0e0 0x7efd9c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:09.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9ad9d640 1 --2- 172.21.15.148:0/3583485571 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0cf160 0x7efd9c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:09.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9a59c640 1 -- 172.21.15.148:0/3583485571 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0cf160 msgr2=0x7efd9c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:09.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9a59c640 1 --2- 172.21.15.148:0/3583485571 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0cf160 0x7efd9c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:09.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9a59c640 1 -- 172.21.15.148:0/3583485571 >> v2:172.21.15.150:3300/0 conn(0x7efd9c0ca0e0 msgr2=0x7efd9c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:09.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9a59c640 1 --2- 172.21.15.148:0/3583485571 >> v2:172.21.15.150:3300/0 conn(0x7efd9c0ca0e0 0x7efd9c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:09.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9a59c640 1 -- 172.21.15.148:0/3583485571 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c0cad70 con 0x7efd9c1538a0 2023-10-30T13:34:09.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.936+0000 7efd9ad9d640 1 --2- 172.21.15.148:0/3583485571 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0cf160 0x7efd9c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:34:09.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efd9a59c640 1 --2- 172.21.15.148:0/3583485571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c1538a0 0x7efd9c15d6f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0b30a0 tx=0x7efd8c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d9109ada41eb18 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:09.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efd9959a640 1 -- 172.21.15.148:0/3583485571 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c0a89b0 con 0x7efd9c1538a0 2023-10-30T13:34:09.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efd9959a640 1 -- 172.21.15.148:0/3583485571 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c0a8b10 con 0x7efd9c1538a0 2023-10-30T13:34:09.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efd9959a640 1 -- 172.21.15.148:0/3583485571 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c0a8ea0 con 0x7efd9c1538a0 2023-10-30T13:34:09.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 -- 172.21.15.148:0/3583485571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c1538a0 msgr2=0x7efd9c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:09.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 --2- 172.21.15.148:0/3583485571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c1538a0 0x7efd9c15d6f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0b30a0 tx=0x7efd8c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:34:09.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 -- 172.21.15.148:0/3583485571 shutdown_connections 2023-10-30T13:34:09.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 --2- 172.21.15.148:0/3583485571 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0cf160 0x7efd9c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:09.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 --2- 172.21.15.148:0/3583485571 >> v2:172.21.15.150:3300/0 conn(0x7efd9c0ca0e0 0x7efd9c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:09.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 --2- 172.21.15.148:0/3583485571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c1538a0 0x7efd9c15d6f0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:09.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 -- 172.21.15.148:0/3583485571 >> 172.21.15.148:0/3583485571 conn(0x7efd9c06bfe0 msgr2=0x7efd9c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:09.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 -- 172.21.15.148:0/3583485571 shutdown_connections 2023-10-30T13:34:09.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 -- 172.21.15.148:0/3583485571 wait complete. 2023-10-30T13:34:09.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.940+0000 7efda0a05640 1 Processor -- start 2023-10-30T13:34:09.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 -- start start 2023-10-30T13:34:09.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0ca0e0 0x7efd9c0c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:09.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c0cf160 0x7efd9c157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:09.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd9c1538a0 0x7efd9c152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:09.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c158d90 con 0x7efd9c0cf160 2023-10-30T13:34:09.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0c5aa0 con 0x7efd9c1538a0 2023-10-30T13:34:09.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0c4b10 con 0x7efd9c0ca0e0 2023-10-30T13:34:09.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd99d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c0cf160 0x7efd9c157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:09.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd99d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c0cf160 0x7efd9c157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49060/0 (socket says 172.21.15.148:49060) 2023-10-30T13:34:09.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd99d9b640 1 -- 172.21.15.148:0/2507123525 learned_addr learned my addr 172.21.15.148:0/2507123525 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:09.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd9a59c640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0ca0e0 0x7efd9c0c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:09.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd9ad9d640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd9c1538a0 0x7efd9c152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:09.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd99d9b640 1 -- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0ca0e0 msgr2=0x7efd9c0c5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:09.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd99d9b640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0ca0e0 0x7efd9c0c5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:09.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd99d9b640 1 -- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd9c1538a0 msgr2=0x7efd9c152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:09.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd99d9b640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd9c1538a0 0x7efd9c152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:09.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd99d9b640 1 -- 172.21.15.148:0/2507123525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c0094d0 con 0x7efd9c0cf160 2023-10-30T13:34:09.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd99d9b640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c0cf160 0x7efd9c157560 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7efd9000d830 tx=0x7efd9000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:09.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd8affd640 1 -- 172.21.15.148:0/2507123525 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd900096a0 con 0x7efd9c0cf160 2023-10-30T13:34:09.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd8affd640 1 -- 172.21.15.148:0/2507123525 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90009800 con 0x7efd9c0cf160 2023-10-30T13:34:09.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efd8affd640 1 -- 172.21.15.148:0/2507123525 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90009b70 con 0x7efd9c0cf160 2023-10-30T13:34:09.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9c0cdb50 con 0x7efd9c0cf160 2023-10-30T13:34:09.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c1594f0 con 0x7efd9c0cf160 2023-10-30T13:34:09.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.944+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9c0b2a50 con 0x7efd9c0cf160 2023-10-30T13:34:09.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.948+0000 7efd8affd640 1 -- 172.21.15.148:0/2507123525 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd90014410 con 0x7efd9c0cf160 2023-10-30T13:34:09.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.948+0000 7efd8affd640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd7c0664a0 0x7efd7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:09.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.948+0000 7efd8affd640 1 -- 172.21.15.148:0/2507123525 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7efd90084b50 con 0x7efd9c0cf160 2023-10-30T13:34:09.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.948+0000 7efd9a59c640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd7c0664a0 0x7efd7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:09.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.948+0000 7efd9a59c640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd7c0664a0 0x7efd7c068960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0b2000 tx=0x7efd8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:09.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:09.952+0000 7efd8affd640 1 -- 172.21.15.148:0/2507123525 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd90051ba0 con 0x7efd9c0cf160 2023-10-30T13:34:10.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd9c04e170 con 0x7efd9c0cf160 2023-10-30T13:34:10.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efd8affd640 1 -- 172.21.15.148:0/2507123525 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5722 (secure 0 0 0) 0x7efd900551c0 con 0x7efd9c0cf160 2023-10-30T13:34:10.131 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:10.131 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:06.216068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:10.132 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 100 2023-10-30T13:34:10.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd7c0664a0 msgr2=0x7efd7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:10.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd7c0664a0 0x7efd7c068960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0b2000 tx=0x7efd8c009040 comp rx=0 tx=0).stop 2023-10-30T13:34:10.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c0cf160 msgr2=0x7efd9c157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:10.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c0cf160 0x7efd9c157560 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7efd9000d830 tx=0x7efd9000dcf0 comp rx=0 tx=0).stop 2023-10-30T13:34:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 shutdown_connections 2023-10-30T13:34:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efd7c0664a0 0x7efd7c068960 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efd9c0cf160 0x7efd9c157560 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd9c1538a0 0x7efd9c152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 --2- 172.21.15.148:0/2507123525 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efd9c0ca0e0 0x7efd9c0c5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:10.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 >> 172.21.15.148:0/2507123525 conn(0x7efd9c06bfe0 msgr2=0x7efd9c150fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:10.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 shutdown_connections 2023-10-30T13:34:10.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:10.136+0000 7efda0a05640 1 -- 172.21.15.148:0/2507123525 wait complete. 2023-10-30T13:34:10.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:12.153 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:12.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f59848bb640 1 Processor -- start 2023-10-30T13:34:12.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f59848bb640 1 -- start start 2023-10-30T13:34:12.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f59848bb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f598015cd50 0x7f59800c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:12.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f59848bb640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598015a6c0 con 0x7f59801538f0 2023-10-30T13:34:12.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f59848bb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598015d300 con 0x7f59800cefa0 2023-10-30T13:34:12.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f59848bb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598015d5e0 con 0x7f598015cd50 2023-10-30T13:34:12.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597f7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f59801538f0 0x7f5980159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58926/0 (socket says 172.21.15.148:58926) 2023-10-30T13:34:12.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597f7fe640 1 -- 172.21.15.148:0/3090260871 learned_addr learned my addr 172.21.15.148:0/3090260871 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:34:12.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597effd640 1 --2- 172.21.15.148:0/3090260871 >> v2:172.21.15.150:3300/0 conn(0x7f598015cd50 0x7f59800c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:12.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679369949 0 0) 0x7f598015a6c0 con 0x7f59801538f0 2023-10-30T13:34:12.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968003530 con 0x7f59801538f0 2023-10-30T13:34:12.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 237846973 0 0) 0x7f5968003530 con 0x7f59801538f0 2023-10-30T13:34:12.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 >> v2:172.21.15.150:3300/0 conn(0x7f598015cd50 msgr2=0x7f59800c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:12.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 --2- 172.21.15.148:0/3090260871 >> v2:172.21.15.150:3300/0 conn(0x7f598015cd50 0x7f59800c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 >> v1:172.21.15.150:6789/0 conn(0x7f59800cefa0 legacy=0x7f59801640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:12.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598015d8c0 con 0x7f59801538f0 2023-10-30T13:34:12.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f5970001680 con 0x7f59801538f0 2023-10-30T13:34:12.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 >> v1:172.21.15.148:6789/0 conn(0x7f59801538f0 legacy=0x7f5980159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:12.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5968003980 0x7f5968003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:12.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59680047d0 0x7f5968004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:12.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5968005190 0x7f596800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:12.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598015d5e0 con 0x7f5968003980 2023-10-30T13:34:12.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596800c040 con 0x7f59680047d0 2023-10-30T13:34:12.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596800c200 con 0x7f5968005190 2023-10-30T13:34:12.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597f7fe640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5968003980 0x7f5968003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:12.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597effd640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59680047d0 0x7f5968004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:12.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597ffff640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5968005190 0x7f596800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:12.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597f7fe640 1 -- 172.21.15.148:0/3090260871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5968005190 msgr2=0x7f596800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:12.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597f7fe640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5968005190 0x7f596800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597f7fe640 1 -- 172.21.15.148:0/3090260871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59680047d0 msgr2=0x7f5968004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:12.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597f7fe640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59680047d0 0x7f5968004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597f7fe640 1 -- 172.21.15.148:0/3090260871 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968003530 con 0x7f5968003980 2023-10-30T13:34:12.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597effd640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59680047d0 0x7f5968004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:12.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.308+0000 7f597ffff640 1 -- 172.21.15.148:0/3090260871 reap_dead start 2023-10-30T13:34:12.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f597f7fe640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5968003980 0x7f5968003da0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f597000c240 tx=0x7f597001f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729b366d8db981f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:12.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970005ae0 con 0x7f5968003980 2023-10-30T13:34:12.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59700ab070 con 0x7f5968003980 2023-10-30T13:34:12.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f597e7fc640 1 -- 172.21.15.148:0/3090260871 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970007070 con 0x7f5968003980 2023-10-30T13:34:12.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f59848bb640 1 -- 172.21.15.148:0/3090260871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5968003980 msgr2=0x7f5968003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:12.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f59848bb640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5968003980 0x7f5968003da0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f597000c240 tx=0x7f597001f3b0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f59848bb640 1 -- 172.21.15.148:0/3090260871 shutdown_connections 2023-10-30T13:34:12.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f59848bb640 1 --2- 172.21.15.148:0/3090260871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5968003980 0x7f5968003da0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f59848bb640 1 -- 172.21.15.148:0/3090260871 >> 172.21.15.148:0/3090260871 conn(0x7f598006a640 msgr2=0x7f5980065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:12.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f59848bb640 1 -- 172.21.15.148:0/3090260871 shutdown_connections 2023-10-30T13:34:12.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.312+0000 7f59848bb640 1 -- 172.21.15.148:0/3090260871 wait complete. 2023-10-30T13:34:12.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 Processor -- start 2023-10-30T13:34:12.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 -- start start 2023-10-30T13:34:12.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5968003980 0x7f59801f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:12.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5980151e70 0x7f598015bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:12.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f598015c8a0 0x7f59801524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:12.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5980151460 con 0x7f598015c8a0 2023-10-30T13:34:12.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5980151180 con 0x7f5980151e70 2023-10-30T13:34:12.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5980148330 con 0x7f5968003980 2023-10-30T13:34:12.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f598015c8a0 0x7f59801524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:12.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f598015c8a0 0x7f59801524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33210/0 (socket says 172.21.15.148:33210) 2023-10-30T13:34:12.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597ffff640 1 -- 172.21.15.148:0/2716898027 learned_addr learned my addr 172.21.15.148:0/2716898027 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:12.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597f7fe640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5968003980 0x7f59801f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:12.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597effd640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5980151e70 0x7f598015bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:12.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597ffff640 1 -- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5968003980 msgr2=0x7f59801f2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:12.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597ffff640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5968003980 0x7f59801f2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597ffff640 1 -- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5980151e70 msgr2=0x7f598015bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:12.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597ffff640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5980151e70 0x7f598015bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597ffff640 1 -- 172.21.15.148:0/2716898027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59801f2700 con 0x7f598015c8a0 2023-10-30T13:34:12.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f597ffff640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f598015c8a0 0x7f59801524b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f59800b1ae0 tx=0x7f597800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:12.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f5967fff640 1 -- 172.21.15.148:0/2716898027 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978003980 con 0x7f598015c8a0 2023-10-30T13:34:12.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f5967fff640 1 -- 172.21.15.148:0/2716898027 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5978003b00 con 0x7f598015c8a0 2023-10-30T13:34:12.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f5967fff640 1 -- 172.21.15.148:0/2716898027 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59780050d0 con 0x7f598015c8a0 2023-10-30T13:34:12.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59801f29e0 con 0x7f598015c8a0 2023-10-30T13:34:12.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.316+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59800b50b0 con 0x7f598015c8a0 2023-10-30T13:34:12.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.320+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59801f2560 con 0x7f598015c8a0 2023-10-30T13:34:12.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.320+0000 7f5967fff640 1 -- 172.21.15.148:0/2716898027 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5978005230 con 0x7f598015c8a0 2023-10-30T13:34:12.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.320+0000 7f5967fff640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f596c0664a0 0x7f596c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:12.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.320+0000 7f5967fff640 1 -- 172.21.15.148:0/2716898027 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f5978083fb0 con 0x7f598015c8a0 2023-10-30T13:34:12.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.320+0000 7f597f7fe640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f596c0664a0 0x7f596c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:12.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.320+0000 7f597f7fe640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f596c0664a0 0x7f596c068960 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f59700af000 tx=0x7f59700aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:12.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.324+0000 7f5967fff640 1 -- 172.21.15.148:0/2716898027 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f59801f2560 con 0x7f598015c8a0 2023-10-30T13:34:12.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.504+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5980159a70 con 0x7f598015c8a0 2023-10-30T13:34:12.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.504+0000 7f5967fff640 1 -- 172.21.15.148:0/2716898027 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f5980159a70 con 0x7f598015c8a0 2023-10-30T13:34:12.500 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:12.501 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:11.775244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:12.501 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 101 2023-10-30T13:34:12.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f596c0664a0 msgr2=0x7f596c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:12.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f596c0664a0 0x7f596c068960 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f59700af000 tx=0x7f59700aa000 comp rx=0 tx=0).stop 2023-10-30T13:34:12.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f598015c8a0 msgr2=0x7f59801524b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:12.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f598015c8a0 0x7f59801524b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f59800b1ae0 tx=0x7f597800ec10 comp rx=0 tx=0).stop 2023-10-30T13:34:12.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 shutdown_connections 2023-10-30T13:34:12.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f596c0664a0 0x7f596c068960 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f598015c8a0 0x7f59801524b0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5980151e70 0x7f598015bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 --2- 172.21.15.148:0/2716898027 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5968003980 0x7f59801f2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:12.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 >> 172.21.15.148:0/2716898027 conn(0x7f598006a640 msgr2=0x7f59800c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:12.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 shutdown_connections 2023-10-30T13:34:12.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:12.508+0000 7f59848bb640 1 -- 172.21.15.148:0/2716898027 wait complete. 2023-10-30T13:34:12.522 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:14.523 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:14.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.676+0000 7f1548b2a640 1 Processor -- start 2023-10-30T13:34:14.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.676+0000 7f1548b2a640 1 -- start start 2023-10-30T13:34:14.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.676+0000 7f1548b2a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f15441539a0 0x7f154415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.676+0000 7f1548b2a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 0x7f1544070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.676+0000 7f1548b2a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154415a8c0 con 0x7f1544069e90 2023-10-30T13:34:14.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.676+0000 7f1548b2a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154406fbb0 con 0x7f15441539a0 2023-10-30T13:34:14.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.676+0000 7f1548b2a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1544071380 con 0x7f1544070a00 2023-10-30T13:34:14.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f154259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f15441539a0 0x7f154415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f154259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f15441539a0 0x7f154415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33226/0 (socket says 172.21.15.148:33226) 2023-10-30T13:34:14.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f154259c640 1 -- 172.21.15.148:0/210633254 learned_addr learned my addr 172.21.15.148:0/210633254 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:14.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f1539d9b640 1 --2- 172.21.15.148:0/210633254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 0x7f1544070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f154259c640 1 -- 172.21.15.148:0/210633254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 msgr2=0x7f1544070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:14.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f154259c640 1 --2- 172.21.15.148:0/210633254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 0x7f1544070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f154259c640 1 -- 172.21.15.148:0/210633254 >> v1:172.21.15.148:6789/0 conn(0x7f1544069e90 legacy=0x7f154406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f154259c640 1 -- 172.21.15.148:0/210633254 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1544153d80 con 0x7f15441539a0 2023-10-30T13:34:14.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f154259c640 1 --2- 172.21.15.148:0/210633254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 0x7f154415a380 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f1534009870 tx=0x7f153401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b5f48b98bf7d80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:14.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f1541d9b640 1 -- 172.21.15.148:0/210633254 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15340ab680 con 0x7f15441539a0 2023-10-30T13:34:14.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f1541d9b640 1 -- 172.21.15.148:0/210633254 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15340ab7e0 con 0x7f15441539a0 2023-10-30T13:34:14.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.680+0000 7f1541d9b640 1 -- 172.21.15.148:0/210633254 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15340abb70 con 0x7f15441539a0 2023-10-30T13:34:14.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 -- 172.21.15.148:0/210633254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 msgr2=0x7f154415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 --2- 172.21.15.148:0/210633254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 0x7f154415a380 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f1534009870 tx=0x7f153401c480 comp rx=0 tx=0).stop 2023-10-30T13:34:14.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 -- 172.21.15.148:0/210633254 shutdown_connections 2023-10-30T13:34:14.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 --2- 172.21.15.148:0/210633254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 0x7f1544070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 --2- 172.21.15.148:0/210633254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 0x7f154415a380 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 -- 172.21.15.148:0/210633254 >> 172.21.15.148:0/210633254 conn(0x7f15440c7910 msgr2=0x7f15440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:14.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 -- 172.21.15.148:0/210633254 shutdown_connections 2023-10-30T13:34:14.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 -- 172.21.15.148:0/210633254 wait complete. 2023-10-30T13:34:14.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 Processor -- start 2023-10-30T13:34:14.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 -- start start 2023-10-30T13:34:14.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544069e90 0x7f1544144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 0x7f15440c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 0x7f15440c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15440c8c80 con 0x7f15441539a0 2023-10-30T13:34:14.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15440c9ee0 con 0x7f1544069e90 2023-10-30T13:34:14.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1548b2a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15440ca1a0 con 0x7f1544070a00 2023-10-30T13:34:14.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1542d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 0x7f15440c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1542d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 0x7f15440c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33236/0 (socket says 172.21.15.148:33236) 2023-10-30T13:34:14.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1542d9d640 1 -- 172.21.15.148:0/4194487340 learned_addr learned my addr 172.21.15.148:0/4194487340 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:14.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f1539d9b640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 0x7f15440c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.684+0000 7f154259c640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544069e90 0x7f1544144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f1542d9d640 1 -- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 msgr2=0x7f15440c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f1542d9d640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 0x7f15440c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f1542d9d640 1 -- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544069e90 msgr2=0x7f1544144c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f1542d9d640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544069e90 0x7f1544144c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f1542d9d640 1 -- 172.21.15.148:0/4194487340 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15340094d0 con 0x7f15441539a0 2023-10-30T13:34:14.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f1542d9d640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 0x7f15440c95e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c430 tx=0x7f153c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:14.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f153b7fe640 1 -- 172.21.15.148:0/4194487340 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c005220 con 0x7f15441539a0 2023-10-30T13:34:14.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f153b7fe640 1 -- 172.21.15.148:0/4194487340 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c005380 con 0x7f15441539a0 2023-10-30T13:34:14.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f153b7fe640 1 -- 172.21.15.148:0/4194487340 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c007270 con 0x7f15441539a0 2023-10-30T13:34:14.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15440ca480 con 0x7f15441539a0 2023-10-30T13:34:14.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15440caa40 con 0x7f15441539a0 2023-10-30T13:34:14.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15440c8dc0 con 0x7f15441539a0 2023-10-30T13:34:14.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f153b7fe640 1 -- 172.21.15.148:0/4194487340 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f153c0073d0 con 0x7f15441539a0 2023-10-30T13:34:14.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f153b7fe640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15240664a0 0x7f1524068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:14.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.688+0000 7f153b7fe640 1 -- 172.21.15.148:0/4194487340 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f153c083910 con 0x7f15441539a0 2023-10-30T13:34:14.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.692+0000 7f154259c640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15240664a0 0x7f1524068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:14.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.692+0000 7f154259c640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15240664a0 0x7f1524068960 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f15340af000 tx=0x7f15340a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:14.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.692+0000 7f153b7fe640 1 -- 172.21.15.148:0/4194487340 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15440c8dc0 con 0x7f15441539a0 2023-10-30T13:34:14.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15440c8dc0 con 0x7f15441539a0 2023-10-30T13:34:14.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f153b7fe640 1 -- 172.21.15.148:0/4194487340 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f15440c8dc0 con 0x7f15441539a0 2023-10-30T13:34:14.871 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:14.871 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:11.775244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:14.871 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 101 2023-10-30T13:34:14.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15240664a0 msgr2=0x7f1524068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15240664a0 0x7f1524068960 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f15340af000 tx=0x7f15340a7000 comp rx=0 tx=0).stop 2023-10-30T13:34:14.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 msgr2=0x7f15440c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:14.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 0x7f15440c95e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c430 tx=0x7f153c00c8f0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 shutdown_connections 2023-10-30T13:34:14.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15240664a0 0x7f1524068960 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15441539a0 0x7f15440c95e0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1544070a00 0x7f15440c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 --2- 172.21.15.148:0/4194487340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1544069e90 0x7f1544144c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:14.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 >> 172.21.15.148:0/4194487340 conn(0x7f15440c7910 msgr2=0x7f1544152b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:14.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 shutdown_connections 2023-10-30T13:34:14.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:14.876+0000 7f1548b2a640 1 -- 172.21.15.148:0/4194487340 wait complete. 2023-10-30T13:34:14.892 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:16.893 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:17.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c9e46640 1 Processor -- start 2023-10-30T13:34:17.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c9e46640 1 -- start start 2023-10-30T13:34:17.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c9e46640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff7c406f7a0 0x7ff7c406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:17.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c9e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c40c6bd0 0x7ff7c4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:17.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c9e46640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c406a3d0 con 0x7ff7c4071110 2023-10-30T13:34:17.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c9e46640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c40700c0 con 0x7ff7c406f7a0 2023-10-30T13:34:17.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c9e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c4066cf0 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c37fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff7c406f7a0 0x7ff7c406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:17.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c37fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff7c406f7a0 0x7ff7c406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33240/0 (socket says 172.21.15.148:33240) 2023-10-30T13:34:17.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c37fe640 1 -- 172.21.15.148:0/2404872571 learned_addr learned my addr 172.21.15.148:0/2404872571 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:17.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c2ffd640 1 --2- 172.21.15.148:0/2404872571 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c40c6bd0 0x7ff7c4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:17.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c37fe640 1 -- 172.21.15.148:0/2404872571 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c40c6bd0 msgr2=0x7ff7c4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:17.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c37fe640 1 --2- 172.21.15.148:0/2404872571 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c40c6bd0 0x7ff7c4069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c37fe640 1 -- 172.21.15.148:0/2404872571 >> v1:172.21.15.148:6789/0 conn(0x7ff7c4071110 legacy=0x7ff7c4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:17.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.048+0000 7ff7c37fe640 1 -- 172.21.15.148:0/2404872571 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c40c70d0 con 0x7ff7c406f7a0 2023-10-30T13:34:17.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c37fe640 1 --2- 172.21.15.148:0/2404872571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c406f7a0 0x7ff7c406fb80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009870 tx=0x7ff7b401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba367c86357b9e94 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:17.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c27fc640 1 -- 172.21.15.148:0/2404872571 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b40a8a30 con 0x7ff7c406f7a0 2023-10-30T13:34:17.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c27fc640 1 -- 172.21.15.148:0/2404872571 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b40a8b90 con 0x7ff7c406f7a0 2023-10-30T13:34:17.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c27fc640 1 -- 172.21.15.148:0/2404872571 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b40b0440 con 0x7ff7c406f7a0 2023-10-30T13:34:17.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c9e46640 1 -- 172.21.15.148:0/2404872571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c406f7a0 msgr2=0x7ff7c406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:17.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c9e46640 1 --2- 172.21.15.148:0/2404872571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c406f7a0 0x7ff7c406fb80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009870 tx=0x7ff7b401ed30 comp rx=0 tx=0).stop 2023-10-30T13:34:17.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c9e46640 1 -- 172.21.15.148:0/2404872571 shutdown_connections 2023-10-30T13:34:17.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c9e46640 1 --2- 172.21.15.148:0/2404872571 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c40c6bd0 0x7ff7c4069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c9e46640 1 --2- 172.21.15.148:0/2404872571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c406f7a0 0x7ff7c406fb80 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c9e46640 1 -- 172.21.15.148:0/2404872571 >> 172.21.15.148:0/2404872571 conn(0x7ff7c40c7910 msgr2=0x7ff7c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:17.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c9e46640 1 -- 172.21.15.148:0/2404872571 shutdown_connections 2023-10-30T13:34:17.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.052+0000 7ff7c9e46640 1 -- 172.21.15.148:0/2404872571 wait complete. 2023-10-30T13:34:17.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 Processor -- start 2023-10-30T13:34:17.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 -- start start 2023-10-30T13:34:17.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c406f7a0 0x7ff7c415b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:17.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4071110 0x7ff7c40af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:17.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c40c6bd0 0x7ff7c40c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:17.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40afaf0 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40ae450 con 0x7ff7c4071110 2023-10-30T13:34:17.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c415bad0 con 0x7ff7c406f7a0 2023-10-30T13:34:17.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c40c6bd0 0x7ff7c40c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:17.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c40c6bd0 0x7ff7c40c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33248/0 (socket says 172.21.15.148:33248) 2023-10-30T13:34:17.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c3fff640 1 -- 172.21.15.148:0/579774230 learned_addr learned my addr 172.21.15.148:0/579774230 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:17.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c2ffd640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4071110 0x7ff7c40af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:17.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c37fe640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c406f7a0 0x7ff7c415b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:17.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c3fff640 1 -- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c406f7a0 msgr2=0x7ff7c415b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:17.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c3fff640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c406f7a0 0x7ff7c415b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c3fff640 1 -- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4071110 msgr2=0x7ff7c40af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:17.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c3fff640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4071110 0x7ff7c40af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c3fff640 1 -- 172.21.15.148:0/579774230 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b40094d0 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c2ffd640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4071110 0x7ff7c40af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:34:17.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c3fff640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c40c6bd0 0x7ff7c40c60b0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00c730 tx=0x7ff7bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:17.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7abfff640 1 -- 172.21.15.148:0/579774230 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc003b80 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7abfff640 1 -- 172.21.15.148:0/579774230 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7bc003ce0 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c40ad8d0 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7abfff640 1 -- 172.21.15.148:0/579774230 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc00aa30 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.056+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c40ade90 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.060+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c41591c0 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.060+0000 7ff7abfff640 1 -- 172.21.15.148:0/579774230 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff7bc00ab90 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.060+0000 7ff7abfff640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff7b00664a0 0x7ff7b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:17.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.060+0000 7ff7abfff640 1 -- 172.21.15.148:0/579774230 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7ff7bc084150 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.060+0000 7ff7c37fe640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff7b00664a0 0x7ff7b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:17.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.060+0000 7ff7c37fe640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff7b00664a0 0x7ff7b0068960 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40b2000 tx=0x7ff7b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:17.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.064+0000 7ff7abfff640 1 -- 172.21.15.148:0/579774230 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff7c41591c0 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7c41591c0 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7abfff640 1 -- 172.21.15.148:0/579774230 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5719 (secure 0 0 0) 0x7ff7c41591c0 con 0x7ff7c40c6bd0 2023-10-30T13:34:17.238 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:17.239 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:11.775244+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:17.239 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 101 2023-10-30T13:34:17.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff7b00664a0 msgr2=0x7ff7b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:17.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff7b00664a0 0x7ff7b0068960 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff7b40b2000 tx=0x7ff7b4008040 comp rx=0 tx=0).stop 2023-10-30T13:34:17.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c40c6bd0 msgr2=0x7ff7c40c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:17.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c40c6bd0 0x7ff7c40c60b0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00c730 tx=0x7ff7bc00cbf0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 shutdown_connections 2023-10-30T13:34:17.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff7b00664a0 0x7ff7b0068960 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7c40c6bd0 0x7ff7c40c60b0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7c4071110 0x7ff7c40af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 --2- 172.21.15.148:0/579774230 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff7c406f7a0 0x7ff7c415b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:17.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 >> 172.21.15.148:0/579774230 conn(0x7ff7c40c7910 msgr2=0x7ff7c40c7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:17.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 shutdown_connections 2023-10-30T13:34:17.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:17.244+0000 7ff7c9e46640 1 -- 172.21.15.148:0/579774230 wait complete. 2023-10-30T13:34:17.260 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:34:19.262 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:19.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f12a79d3640 1 Processor -- start 2023-10-30T13:34:19.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f12a79d3640 1 -- start start 2023-10-30T13:34:19.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f12a79d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12a014fd50 0x7f12a00c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f12a79d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a01560a0 con 0x7f12a0071960 2023-10-30T13:34:19.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f12a79d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a0156380 con 0x7f12a00cb550 2023-10-30T13:34:19.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f12a79d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a0156660 con 0x7f12a014fd50 2023-10-30T13:34:19.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f12a5f70640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f12a0071960 0x7f12a0160710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58944/0 (socket says 172.21.15.148:58944) 2023-10-30T13:34:19.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f12a5f70640 1 -- 172.21.15.148:0/3667781864 learned_addr learned my addr 172.21.15.148:0/3667781864 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:34:19.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f12a576f640 1 --2- 172.21.15.148:0/3667781864 >> v2:172.21.15.150:3300/0 conn(0x7f12a014fd50 0x7f12a00c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562666144 0 0) 0x7f12a01560a0 con 0x7f12a0071960 2023-10-30T13:34:19.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.416+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f128c003530 con 0x7f12a0071960 2023-10-30T13:34:19.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204174951 0 0) 0x7f12a0156380 con 0x7f12a00cb550 2023-10-30T13:34:19.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a01560a0 con 0x7f12a00cb550 2023-10-30T13:34:19.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1868869869 0 0) 0x7f128c003530 con 0x7f12a0071960 2023-10-30T13:34:19.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 >> v2:172.21.15.150:3300/0 conn(0x7f12a014fd50 msgr2=0x7f12a00c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:19.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 --2- 172.21.15.148:0/3667781864 >> v2:172.21.15.150:3300/0 conn(0x7f12a014fd50 0x7f12a00c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 >> v1:172.21.15.150:6789/0 conn(0x7f12a00cb550 legacy=0x7f12a0071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0156940 con 0x7f12a0071960 2023-10-30T13:34:19.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f1298001a00 con 0x7f12a0071960 2023-10-30T13:34:19.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 >> v1:172.21.15.148:6789/0 conn(0x7f12a0071960 legacy=0x7f12a0160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f128c003b60 0x7f128c003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f128c0047d0 0x7f128c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f128c0051e0 0x7f128c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0156660 con 0x7f128c0051e0 2023-10-30T13:34:19.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128c00c110 con 0x7f128c003b60 2023-10-30T13:34:19.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128c00c2d0 con 0x7f128c0047d0 2023-10-30T13:34:19.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a5f70640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f128c0051e0 0x7f128c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a576f640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f128c003b60 0x7f128c003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a4f6e640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f128c0047d0 0x7f128c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a5f70640 1 -- 172.21.15.148:0/3667781864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f128c0047d0 msgr2=0x7f128c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:19.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a5f70640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f128c0047d0 0x7f128c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a5f70640 1 -- 172.21.15.148:0/3667781864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f128c003b60 msgr2=0x7f128c003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:19.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a5f70640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f128c003b60 0x7f128c003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a5f70640 1 -- 172.21.15.148:0/3667781864 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c003530 con 0x7f128c0051e0 2023-10-30T13:34:19.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a5f70640 1 -- 172.21.15.148:0/3667781864 reap_dead start 2023-10-30T13:34:19.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.420+0000 7f12a5f70640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f128c0051e0 0x7f128c00b990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f129800a700 tx=0x7f129801d1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3722625a886ec2f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:19.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12980ad630 con 0x7f128c0051e0 2023-10-30T13:34:19.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12980a8070 con 0x7f128c0051e0 2023-10-30T13:34:19.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f1297fff640 1 -- 172.21.15.148:0/3667781864 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12980ad9d0 con 0x7f128c0051e0 2023-10-30T13:34:19.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 -- 172.21.15.148:0/3667781864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f128c0051e0 msgr2=0x7f128c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f128c0051e0 0x7f128c00b990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f129800a700 tx=0x7f129801d1e0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 -- 172.21.15.148:0/3667781864 shutdown_connections 2023-10-30T13:34:19.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 --2- 172.21.15.148:0/3667781864 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f128c0051e0 0x7f128c00b990 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 -- 172.21.15.148:0/3667781864 >> 172.21.15.148:0/3667781864 conn(0x7f12a006c6e0 msgr2=0x7f12a0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:19.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 -- 172.21.15.148:0/3667781864 shutdown_connections 2023-10-30T13:34:19.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 -- 172.21.15.148:0/3667781864 wait complete. 2023-10-30T13:34:19.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 Processor -- start 2023-10-30T13:34:19.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 -- start start 2023-10-30T13:34:19.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f128c0051e0 0x7f12a0152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a01f2b40 0x7f12a0143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12a0157a30 0x7f12a00c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0159ad0 con 0x7f12a0157a30 2023-10-30T13:34:19.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0157ee0 con 0x7f12a01f2b40 2023-10-30T13:34:19.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a79d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0152230 con 0x7f128c0051e0 2023-10-30T13:34:19.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a5f70640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12a0157a30 0x7f12a00c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a5f70640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12a0157a30 0x7f12a00c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33272/0 (socket says 172.21.15.148:33272) 2023-10-30T13:34:19.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.424+0000 7f12a5f70640 1 -- 172.21.15.148:0/2241923472 learned_addr learned my addr 172.21.15.148:0/2241923472 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:19.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a4f6e640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a01f2b40 0x7f12a0143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a576f640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f128c0051e0 0x7f12a0152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a5f70640 1 -- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f128c0051e0 msgr2=0x7f12a0152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a5f70640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f128c0051e0 0x7f12a0152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a5f70640 1 -- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a01f2b40 msgr2=0x7f12a0143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a5f70640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a01f2b40 0x7f12a0143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a5f70640 1 -- 172.21.15.148:0/2241923472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0152510 con 0x7f12a0157a30 2023-10-30T13:34:19.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a5f70640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12a0157a30 0x7f12a00c9f00 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f129800a700 tx=0x7f128c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:19.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f1295ffb640 1 -- 172.21.15.148:0/2241923472 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298001a00 con 0x7f12a0157a30 2023-10-30T13:34:19.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f1295ffb640 1 -- 172.21.15.148:0/2241923472 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12980b3040 con 0x7f12a0157a30 2023-10-30T13:34:19.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f1295ffb640 1 -- 172.21.15.148:0/2241923472 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12980ad630 con 0x7f12a0157a30 2023-10-30T13:34:19.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a0156f50 con 0x7f12a0157a30 2023-10-30T13:34:19.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a0157510 con 0x7f12a0157a30 2023-10-30T13:34:19.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a0157700 con 0x7f12a0157a30 2023-10-30T13:34:19.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.428+0000 7f1295ffb640 1 -- 172.21.15.148:0/2241923472 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12980aa030 con 0x7f12a0157a30 2023-10-30T13:34:19.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.432+0000 7f1295ffb640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12840664d0 0x7f1284068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.432+0000 7f1295ffb640 1 -- 172.21.15.148:0/2241923472 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f129811ddc0 con 0x7f12a0157a30 2023-10-30T13:34:19.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.432+0000 7f12a576f640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12840664d0 0x7f1284068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.432+0000 7f12a576f640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12840664d0 0x7f1284068990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f129c00ab60 tx=0x7f129c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:19.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.436+0000 7f1295ffb640 1 -- 172.21.15.148:0/2241923472 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12980eadb0 con 0x7f12a0157a30 2023-10-30T13:34:19.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.616+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12a004e170 con 0x7f12a0157a30 2023-10-30T13:34:19.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.616+0000 7f1295ffb640 1 -- 172.21.15.148:0/2241923472 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f12980ee3d0 con 0x7f12a0157a30 2023-10-30T13:34:19.613 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:19.613 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:19.613 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 102 2023-10-30T13:34:19.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12840664d0 msgr2=0x7f1284068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12840664d0 0x7f1284068990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f129c00ab60 tx=0x7f129c008040 comp rx=0 tx=0).stop 2023-10-30T13:34:19.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12a0157a30 msgr2=0x7f12a00c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12a0157a30 0x7f12a00c9f00 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f129800a700 tx=0x7f128c005800 comp rx=0 tx=0).stop 2023-10-30T13:34:19.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 shutdown_connections 2023-10-30T13:34:19.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12840664d0 0x7f1284068990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12a0157a30 0x7f12a00c9f00 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a01f2b40 0x7f12a0143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 --2- 172.21.15.148:0/2241923472 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f128c0051e0 0x7f12a0152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 >> 172.21.15.148:0/2241923472 conn(0x7f12a006c6e0 msgr2=0x7f12a00c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:19.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 shutdown_connections 2023-10-30T13:34:19.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.620+0000 7f12a79d3640 1 -- 172.21.15.148:0/2241923472 wait complete. 2023-10-30T13:34:19.634 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6546 2023-10-30T13:34:19.634 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2023-10-30T13:34:19.634 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:34:19.635 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:34:19.635 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:34:19.639 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:34:19.639 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:34:19.725 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:34:19.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0ab1180640 1 Processor -- start 2023-10-30T13:34:19.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0ab1180640 1 -- start start 2023-10-30T13:34:19.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0ab1180640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 0x7f0aac06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0ab1180640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0aac0c8490 0x7f0aac0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0ab1180640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aac06afa0 con 0x7f0aac153d70 2023-10-30T13:34:19.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0ab1180640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac06d780 con 0x7f0aac0c8490 2023-10-30T13:34:19.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0ab1180640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac154150 con 0x7f0aac06e470 2023-10-30T13:34:19.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0aab59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0aac0c8490 0x7f0aac0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0aab59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0aac0c8490 0x7f0aac0c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33314/0 (socket says 172.21.15.148:33314) 2023-10-30T13:34:19.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0aab59e640 1 -- 172.21.15.148:0/1276667796 learned_addr learned my addr 172.21.15.148:0/1276667796 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:19.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0aaa59c640 1 --2- 172.21.15.148:0/1276667796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 0x7f0aac06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0aab59e640 1 -- 172.21.15.148:0/1276667796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 msgr2=0x7f0aac06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:34:19.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0aab59e640 1 --2- 172.21.15.148:0/1276667796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 0x7f0aac06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0aab59e640 1 -- 172.21.15.148:0/1276667796 >> v1:172.21.15.150:6789/0 conn(0x7f0aac153d70 legacy=0x7f0aac0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.788+0000 7f0aab59e640 1 -- 172.21.15.148:0/1276667796 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac06e030 con 0x7f0aac0c8490 2023-10-30T13:34:19.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0aab59e640 1 --2- 172.21.15.148:0/1276667796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 0x7f0aac0c8cc0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009e50 tx=0x7f0aa401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18887266e9fc2197 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:19.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0aa9d9b640 1 -- 172.21.15.148:0/1276667796 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa40a58a0 con 0x7f0aac0c8490 2023-10-30T13:34:19.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0aa9d9b640 1 -- 172.21.15.148:0/1276667796 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa40a5a00 con 0x7f0aac0c8490 2023-10-30T13:34:19.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0aa9d9b640 1 -- 172.21.15.148:0/1276667796 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa40a5d90 con 0x7f0aac0c8490 2023-10-30T13:34:19.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0ab1180640 1 -- 172.21.15.148:0/1276667796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 msgr2=0x7f0aac0c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0ab1180640 1 --2- 172.21.15.148:0/1276667796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 0x7f0aac0c8cc0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009e50 tx=0x7f0aa401c9e0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0ab1180640 1 -- 172.21.15.148:0/1276667796 shutdown_connections 2023-10-30T13:34:19.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0ab1180640 1 --2- 172.21.15.148:0/1276667796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 0x7f0aac0c8cc0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0ab1180640 1 --2- 172.21.15.148:0/1276667796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 0x7f0aac06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0ab1180640 1 -- 172.21.15.148:0/1276667796 >> 172.21.15.148:0/1276667796 conn(0x7f0aac071ea0 msgr2=0x7f0aac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:19.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0ab1180640 1 -- 172.21.15.148:0/1276667796 shutdown_connections 2023-10-30T13:34:19.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.792+0000 7f0ab1180640 1 -- 172.21.15.148:0/1276667796 wait complete. 2023-10-30T13:34:19.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 Processor -- start 2023-10-30T13:34:19.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 -- start start 2023-10-30T13:34:19.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 0x7f0aac0caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 0x7f0aac0b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aac153d70 0x7f0aac0b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0ae9a0 con 0x7f0aac0c8490 2023-10-30T13:34:19.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0ae5f0 con 0x7f0aac153d70 2023-10-30T13:34:19.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0c8020 con 0x7f0aac06e470 2023-10-30T13:34:19.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaa59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 0x7f0aac0b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaa59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 0x7f0aac0b0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33316/0 (socket says 172.21.15.148:33316) 2023-10-30T13:34:19.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaa59c640 1 -- 172.21.15.148:0/3338427286 learned_addr learned my addr 172.21.15.148:0/3338427286 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:34:19.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aab59e640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aac153d70 0x7f0aac0b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaad9d640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 0x7f0aac0caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaa59c640 1 -- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 msgr2=0x7f0aac0caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaa59c640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 0x7f0aac0caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaa59c640 1 -- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aac153d70 msgr2=0x7f0aac0b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaa59c640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aac153d70 0x7f0aac0b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaa59c640 1 -- 172.21.15.148:0/3338427286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4009b30 con 0x7f0aac0c8490 2023-10-30T13:34:19.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0aaa59c640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 0x7f0aac0b0f70 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000d5f0 tx=0x7f0aa000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:19.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0a937fe640 1 -- 172.21.15.148:0/3338427286 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa00125f0 con 0x7f0aac0c8490 2023-10-30T13:34:19.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0a937fe640 1 -- 172.21.15.148:0/3338427286 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa0012750 con 0x7f0aac0c8490 2023-10-30T13:34:19.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0a937fe640 1 -- 172.21.15.148:0/3338427286 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0012ae0 con 0x7f0aac0c8490 2023-10-30T13:34:19.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aac0ada70 con 0x7f0aac0c8490 2023-10-30T13:34:19.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.796+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac0ae030 con 0x7f0aac0c8490 2023-10-30T13:34:19.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.800+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aac1eb720 con 0x7f0aac0c8490 2023-10-30T13:34:19.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.800+0000 7f0a937fe640 1 -- 172.21.15.148:0/3338427286 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0aa0012c40 con 0x7f0aac0c8490 2023-10-30T13:34:19.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.800+0000 7f0a937fe640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a980664a0 0x7f0a98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:34:19.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.800+0000 7f0a937fe640 1 -- 172.21.15.148:0/3338427286 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f0aa0084800 con 0x7f0aac0c8490 2023-10-30T13:34:19.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.800+0000 7f0aaad9d640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a980664a0 0x7f0a98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:34:19.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.800+0000 7f0aaad9d640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a980664a0 0x7f0a98068960 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c000f70 tx=0x7f0a9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:34:19.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.804+0000 7f0a937fe640 1 -- 172.21.15.148:0/3338427286 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0aa00517d0 con 0x7f0aac0c8490 2023-10-30T13:34:19.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.988+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0aac04e170 con 0x7f0aac0c8490 2023-10-30T13:34:19.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0a937fe640 1 -- 172.21.15.148:0/3338427286 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7f0aa0007380 con 0x7f0aac0c8490 2023-10-30T13:34:19.986 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:34:19.986 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:34:19.987 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:34:19.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a980664a0 msgr2=0x7f0a98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a980664a0 0x7f0a98068960 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c000f70 tx=0x7f0a9c009040 comp rx=0 tx=0).stop 2023-10-30T13:34:19.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 msgr2=0x7f0aac0b0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:34:19.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 0x7f0aac0b0f70 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000d5f0 tx=0x7f0aa000dab0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 shutdown_connections 2023-10-30T13:34:19.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0a980664a0 0x7f0a98068960 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aac0c8490 0x7f0aac0b0f70 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aac153d70 0x7f0aac0b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 --2- 172.21.15.148:0/3338427286 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aac06e470 0x7f0aac0caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:34:19.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 >> 172.21.15.148:0/3338427286 conn(0x7f0aac071ea0 msgr2=0x7f0aac14fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:34:19.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 shutdown_connections 2023-10-30T13:34:19.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:34:19.992+0000 7f0ab1180640 1 -- 172.21.15.148:0/3338427286 wait complete. 2023-10-30T13:34:20.007 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:34:20.008 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-10-30T13:34:24.955 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 52% (79/151) Updating files: 53% (81/151) Updating files: 54% (82/151) Updating files: 55% (84/151) Updating files: 56% (85/151) Updating files: 57% (87/151) Updating files: 58% (88/151) Updating files: 59% (90/151) Updating files: 60% (91/151) Updating files: 61% (93/151) Updating files: 62% (94/151) Updating files: 63% (96/151) Updating files: 64% (97/151) Updating files: 65% (99/151) Updating files: 66% (100/151) Updating files: 67% (102/151) Updating files: 68% (103/151) Updating files: 69% (105/151) Updating files: 70% (106/151) Updating files: 71% (108/151) Updating files: 72% (109/151) Updating files: 73% (111/151) Updating files: 74% (112/151) Updating files: 75% (114/151) Updating files: 76% (115/151) Updating files: 77% (117/151) Updating files: 78% (118/151) Updating files: 79% (120/151) Updating files: 80% (121/151) Updating files: 81% (123/151) Updating files: 82% (124/151) Updating files: 83% (126/151) Updating files: 84% (127/151) Updating files: 85% (129/151) Updating files: 86% (130/151) Updating files: 87% (132/151) Updating files: 88% (133/151) Updating files: 89% (135/151) Updating files: 90% (136/151) Updating files: 91% (138/151) Updating files: 92% (139/151) Updating files: 93% (141/151) Updating files: 94% (142/151) Updating files: 95% (144/151) Updating files: 96% (145/151) Updating files: 97% (147/151) Updating files: 98% (148/151) Updating files: 99% (150/151) Updating files: 100% (151/151) Updating files: 100% (151/151), done. 2023-10-30T13:34:25.344 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2023-10-30T13:34:25.348 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.14 2023-10-30T13:34:25.365 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 15 -eq 48 ] 2023-10-30T13:34:25.365 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.15 2023-10-30T13:34:25.365 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.15 2023-10-30T13:34:25.366 INFO:tasks.workunit.client.0.smithi148.stdout:v0.15 2023-10-30T13:34:25.366 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.15 2023-10-30T13:34:26.819 INFO:tasks.ceph.mds.a.smithi148.stderr:2023-10-30T13:34:26.824+0000 7f011fc1e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:34:29.946 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:29.950 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:34:30.386 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 59% (67/112) Updating files: 60% (68/112) Updating files: 61% (69/112) Updating files: 62% (70/112) Updating files: 63% (71/112) Updating files: 64% (72/112) Updating files: 65% (73/112) Updating files: 66% (74/112) Updating files: 67% (76/112) Updating files: 68% (77/112) Updating files: 69% (78/112) Updating files: 70% (79/112) Updating files: 71% (80/112) Updating files: 72% (81/112) Updating files: 73% (82/112) Updating files: 74% (83/112) Updating files: 75% (84/112) Updating files: 76% (86/112) Updating files: 77% (87/112) Updating files: 78% (88/112) Updating files: 79% (89/112) Updating files: 80% (90/112) Updating files: 81% (91/112) Updating files: 82% (92/112) Updating files: 83% (93/112) Updating files: 84% (95/112) Updating files: 85% (96/112) Updating files: 86% (97/112) Updating files: 87% (98/112) Updating files: 88% (99/112) Updating files: 89% (100/112) Updating files: 90% (101/112) Updating files: 91% (102/112) Updating files: 92% (104/112) Updating files: 93% (105/112) Updating files: 94% (106/112) Updating files: 95% (107/112) Updating files: 96% (108/112) Updating files: 97% (109/112) Updating files: 98% (110/112) Updating files: 99% (111/112) Updating files: 100% (112/112) Updating files: 100% (112/112), done. 2023-10-30T13:34:30.409 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at ab3c421eb95 v0.15 2023-10-30T13:34:30.413 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.15 2023-10-30T13:34:30.494 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 16 -eq 48 ] 2023-10-30T13:34:30.494 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.16 2023-10-30T13:34:30.494 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.16 2023-10-30T13:34:30.495 INFO:tasks.workunit.client.0.smithi148.stdout:v0.16 2023-10-30T13:34:30.495 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.16 2023-10-30T13:34:32.876 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 51% (55/107) Updating files: 52% (56/107) Updating files: 53% (57/107) Updating files: 54% (58/107) Updating files: 55% (59/107) Updating files: 56% (60/107) Updating files: 57% (61/107) Updating files: 58% (63/107) Updating files: 59% (64/107) Updating files: 60% (65/107) Updating files: 61% (66/107) Updating files: 62% (67/107) Updating files: 63% (68/107) Updating files: 64% (69/107) Updating files: 65% (70/107) Updating files: 66% (71/107) Updating files: 67% (72/107) Updating files: 68% (73/107) Updating files: 69% (74/107) Updating files: 70% (75/107) Updating files: 71% (76/107) Updating files: 72% (78/107) Updating files: 73% (79/107) Updating files: 74% (80/107) Updating files: 75% (81/107) Updating files: 76% (82/107) Updating files: 77% (83/107) Updating files: 78% (84/107) Updating files: 79% (85/107) Updating files: 80% (86/107) Updating files: 81% (87/107) Updating files: 82% (88/107) Updating files: 83% (89/107) Updating files: 84% (90/107) Updating files: 85% (91/107) Updating files: 86% (93/107) Updating files: 87% (94/107) Updating files: 88% (95/107) Updating files: 89% (96/107) Updating files: 90% (97/107) Updating files: 91% (98/107) Updating files: 92% (99/107) Updating files: 93% (100/107) Updating files: 94% (101/107) Updating files: 95% (102/107) Updating files: 96% (103/107) Updating files: 97% (104/107) Updating files: 98% (105/107) Updating files: 99% (106/107) Updating files: 100% (107/107) Updating files: 100% (107/107), done. 2023-10-30T13:34:32.905 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 38dbaa5175c v0.16 2023-10-30T13:34:32.909 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.16 2023-10-30T13:34:32.945 INFO:tasks.workunit.client.0.smithi148.stdout:v0.17 2023-10-30T13:34:32.946 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 17 -eq 48 ] 2023-10-30T13:34:32.946 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.17 2023-10-30T13:34:32.946 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.17 2023-10-30T13:34:32.946 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.17 2023-10-30T13:34:34.879 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 64% (140/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-30T13:34:35.038 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at dac3a8c35fb v0.17 2023-10-30T13:34:35.041 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.17 2023-10-30T13:34:35.798 INFO:tasks.workunit.client.0.smithi148.stdout:v0.18 2023-10-30T13:34:35.798 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 18 -eq 48 ] 2023-10-30T13:34:35.799 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.18 2023-10-30T13:34:35.799 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.18 2023-10-30T13:34:35.799 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.18 2023-10-30T13:34:42.843 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 11% (89/779) Updating files: 12% (94/779) Updating files: 13% (102/779) Updating files: 14% (110/779) Updating files: 15% (117/779) Updating files: 16% (125/779) Updating files: 17% (133/779) Updating files: 18% (141/779) Updating files: 19% (149/779) Updating files: 20% (156/779) Updating files: 21% (164/779) Updating files: 22% (172/779) Updating files: 23% (180/779) Updating files: 24% (187/779) Updating files: 25% (195/779) Updating files: 26% (203/779) Updating files: 27% (211/779) Updating files: 28% (219/779) Updating files: 28% (224/779) Updating files: 29% (226/779) Updating files: 30% (234/779) Updating files: 31% (242/779) Updating files: 32% (250/779) Updating files: 33% (258/779) Updating files: 34% (265/779) Updating files: 35% (273/779) Updating files: 36% (281/779) Updating files: 37% (289/779) Updating files: 38% (297/779) Updating files: 39% (304/779) Updating files: 40% (312/779) Updating files: 41% (320/779) Updating files: 42% (328/779) Updating files: 43% (335/779) Updating files: 44% (343/779) Updating files: 45% (351/779) Updating files: 46% (359/779) Updating files: 47% (367/779) Updating files: 48% (374/779) Updating files: 48% (381/779) Updating files: 49% (382/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 52% (406/779) Updating files: 53% (413/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 58% (453/779) Updating files: 59% (460/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 63% (492/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 75% (585/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 78% (608/779) Updating files: 79% (616/779) Updating files: 80% (624/779) Updating files: 81% (631/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 84% (655/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 91% (709/779) Updating files: 92% (717/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 99% (772/779) Updating files: 99% (776/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2023-10-30T13:34:42.873 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 2f10137986a v0.18 2023-10-30T13:34:42.877 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.18 2023-10-30T13:34:42.980 INFO:tasks.workunit.client.0.smithi148.stdout:v0.19 2023-10-30T13:34:42.980 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 19 -eq 48 ] 2023-10-30T13:34:42.981 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.19 2023-10-30T13:34:42.981 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.19 2023-10-30T13:34:42.981 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.19 2023-10-30T13:34:45.500 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 23% (52/217) Updating files: 24% (53/217) Updating files: 25% (55/217) Updating files: 26% (57/217) Updating files: 27% (59/217) Updating files: 28% (61/217) Updating files: 29% (63/217) Updating files: 30% (66/217) Updating files: 31% (68/217) Updating files: 32% (70/217) Updating files: 33% (72/217) Updating files: 34% (74/217) Updating files: 35% (76/217) Updating files: 36% (79/217) Updating files: 37% (81/217) Updating files: 38% (83/217) Updating files: 39% (85/217) Updating files: 40% (87/217) Updating files: 41% (89/217) Updating files: 42% (92/217) Updating files: 43% (94/217) Updating files: 44% (96/217) Updating files: 45% (98/217) Updating files: 46% (100/217) Updating files: 47% (102/217) Updating files: 48% (105/217) Updating files: 49% (107/217) Updating files: 50% (109/217) Updating files: 51% (111/217) Updating files: 52% (113/217) Updating files: 53% (116/217) Updating files: 54% (118/217) Updating files: 55% (120/217) Updating files: 56% (122/217) Updating files: 57% (124/217) Updating files: 58% (126/217) Updating files: 59% (129/217) Updating files: 60% (131/217) Updating files: 61% (133/217) Updating files: 62% (135/217) Updating files: 63% (137/217) Updating files: 64% (139/217) Updating files: 65% (142/217) Updating files: 66% (144/217) Updating files: 67% (146/217) Updating files: 68% (148/217) Updating files: 69% (150/217) Updating files: 70% (152/217) Updating files: 71% (155/217) Updating files: 72% (157/217) Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 83% (182/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-30T13:34:45.533 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 98f5be53f85 v0.19 2023-10-30T13:34:45.537 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.19 2023-10-30T13:34:45.816 INFO:tasks.workunit.client.0.smithi148.stdout:v0.20 2023-10-30T13:34:45.817 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 20 -eq 48 ] 2023-10-30T13:34:45.817 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.20 2023-10-30T13:34:45.817 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.20 2023-10-30T13:34:45.817 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.20 2023-10-30T13:34:58.937 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 30% (75/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 77% (190/245) Updating files: 78% (192/245) Updating files: 78% (193/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-30T13:34:58.965 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2023-10-30T13:34:58.968 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.20 2023-10-30T13:34:59.051 INFO:tasks.workunit.client.0.smithi148.stdout:v0.21 2023-10-30T13:34:59.051 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 21 -eq 48 ] 2023-10-30T13:34:59.052 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.21 2023-10-30T13:34:59.052 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.21 2023-10-30T13:34:59.052 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.21 2023-10-30T13:35:00.003 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:00.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:20.256 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 8% (46/537) Updating files: 9% (49/537) Updating files: 10% (54/537) Updating files: 11% (60/537) Updating files: 12% (65/537) Updating files: 13% (70/537) Updating files: 14% (76/537) Updating files: 15% (81/537) Updating files: 16% (86/537) Updating files: 17% (92/537) Updating files: 18% (97/537) Updating files: 19% (103/537) Updating files: 20% (108/537) Updating files: 21% (113/537) Updating files: 22% (119/537) Updating files: 23% (124/537) Updating files: 24% (129/537) Updating files: 25% (135/537) Updating files: 26% (140/537) Updating files: 27% (145/537) Updating files: 28% (151/537) Updating files: 29% (156/537) Updating files: 30% (162/537) Updating files: 31% (167/537) Updating files: 32% (172/537) Updating files: 33% (178/537) Updating files: 34% (183/537) Updating files: 35% (188/537) Updating files: 36% (194/537) Updating files: 37% (199/537) Updating files: 38% (205/537) Updating files: 39% (210/537) Updating files: 40% (215/537) Updating files: 41% (221/537) Updating files: 42% (226/537) Updating files: 43% (231/537) Updating files: 43% (235/537) Updating files: 44% (237/537) Updating files: 45% (242/537) Updating files: 46% (248/537) Updating files: 47% (253/537) Updating files: 48% (258/537) Updating files: 49% (264/537) Updating files: 49% (268/537) Updating files: 50% (269/537) Updating files: 51% (274/537) Updating files: 52% (280/537) Updating files: 52% (282/537) Updating files: 53% (285/537) Updating files: 54% (290/537) Updating files: 55% (296/537) Updating files: 56% (301/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 63% (339/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 68% (367/537) Updating files: 69% (371/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 77% (418/537) Updating files: 78% (419/537) Updating files: 79% (425/537) Updating files: 80% (430/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 84% (452/537) Updating files: 85% (457/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 91% (489/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 97% (521/537) Updating files: 97% (524/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2023-10-30T13:35:20.693 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 090436f56ac v0.21 2023-10-30T13:35:20.699 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.21 2023-10-30T13:35:20.719 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 22 -eq 48 ] 2023-10-30T13:35:20.720 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.22 2023-10-30T13:35:20.720 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.22 2023-10-30T13:35:20.720 INFO:tasks.workunit.client.0.smithi148.stdout:v0.22 2023-10-30T13:35:20.721 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.22 2023-10-30T13:35:24.151 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 71% (146/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2023-10-30T13:35:24.180 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 8a7c95f60ad v0.22 2023-10-30T13:35:24.185 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.22 2023-10-30T13:35:24.240 INFO:tasks.workunit.client.0.smithi148.stdout:v0.23 2023-10-30T13:35:24.240 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 23 -eq 48 ] 2023-10-30T13:35:24.240 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.23 2023-10-30T13:35:24.241 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.23 2023-10-30T13:35:24.241 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.23 2023-10-30T13:35:27.008 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:27.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdfe29c640 1 Processor -- start 2023-10-30T13:35:27.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdfe29c640 1 -- start start 2023-10-30T13:35:27.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdfe29c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efdf80c6b80 0x7efdf80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdfe29c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efdf813f4a0 0x7efdf80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdfe29c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf80c0a50 con 0x7efdf80c6f60 2023-10-30T13:35:27.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdfe29c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf806f6e0 con 0x7efdf813f4a0 2023-10-30T13:35:27.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdfe29c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf806f9c0 con 0x7efdf80c6b80 2023-10-30T13:35:27.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf77fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efdf80c6f60 0x7efdf813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43074/0 (socket says 172.21.15.148:43074) 2023-10-30T13:35:27.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf77fe640 1 -- 172.21.15.148:0/2958021986 learned_addr learned my addr 172.21.15.148:0/2958021986 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:35:27.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdfc839640 1 --2- 172.21.15.148:0/2958021986 >> v2:172.21.15.150:3300/0 conn(0x7efdf813f4a0 0x7efdf80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf7fff640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efdf80c6b80 0x7efdf80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106176081 0 0) 0x7efdf80c0a50 con 0x7efdf80c6f60 2023-10-30T13:35:27.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efde0003530 con 0x7efdf80c6f60 2023-10-30T13:35:27.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1582353268 0 0) 0x7efde0003530 con 0x7efdf80c6f60 2023-10-30T13:35:27.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efdf80c6b80 msgr2=0x7efdf80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:27.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efdf80c6b80 0x7efdf80bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 >> v2:172.21.15.150:3300/0 conn(0x7efdf813f4a0 msgr2=0x7efdf80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:27.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 --2- 172.21.15.148:0/2958021986 >> v2:172.21.15.150:3300/0 conn(0x7efdf813f4a0 0x7efdf80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf806fca0 con 0x7efdf80c6f60 2023-10-30T13:35:27.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7efdec001680 con 0x7efdf80c6f60 2023-10-30T13:35:27.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 >> v1:172.21.15.148:6789/0 conn(0x7efdf80c6f60 legacy=0x7efdf813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efde0003980 0x7efde0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efde00047d0 0x7efde0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efdf80c6b80 0x7efde0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf806f6e0 con 0x7efde00047d0 2023-10-30T13:35:27.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf806f9c0 con 0x7efde0003980 2023-10-30T13:35:27.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde000b990 con 0x7efdf80c6b80 2023-10-30T13:35:27.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf77fe640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efde00047d0 0x7efde0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf7fff640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efde0003980 0x7efde0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdfc839640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efdf80c6b80 0x7efde0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf77fe640 1 -- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efdf80c6b80 msgr2=0x7efde0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:27.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf77fe640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efdf80c6b80 0x7efde0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf77fe640 1 -- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efde0003980 msgr2=0x7efde0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:27.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf77fe640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efde0003980 0x7efde0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf77fe640 1 -- 172.21.15.148:0/2958021986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde0003530 con 0x7efde00047d0 2023-10-30T13:35:27.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.161+0000 7efdf77fe640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efde00047d0 0x7efde0004c50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7efdec00c210 tx=0x7efdec01f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98238b9f2abb6235 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:27.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec0a8cb0 con 0x7efde00047d0 2023-10-30T13:35:27.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec0a8e10 con 0x7efde00047d0 2023-10-30T13:35:27.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdf6ffd640 1 -- 172.21.15.148:0/2958021986 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec005b00 con 0x7efde00047d0 2023-10-30T13:35:27.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 -- 172.21.15.148:0/2958021986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efde00047d0 msgr2=0x7efde0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 --2- 172.21.15.148:0/2958021986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efde00047d0 0x7efde0004c50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7efdec00c210 tx=0x7efdec01f390 comp rx=0 tx=0).stop 2023-10-30T13:35:27.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfc839640 1 -- 172.21.15.148:0/2958021986 reap_dead start 2023-10-30T13:35:27.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 -- 172.21.15.148:0/2958021986 shutdown_connections 2023-10-30T13:35:27.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 -- 172.21.15.148:0/2958021986 >> 172.21.15.148:0/2958021986 conn(0x7efdf80c7910 msgr2=0x7efdf80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:27.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 -- 172.21.15.148:0/2958021986 shutdown_connections 2023-10-30T13:35:27.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 -- 172.21.15.148:0/2958021986 wait complete. 2023-10-30T13:35:27.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 Processor -- start 2023-10-30T13:35:27.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 -- start start 2023-10-30T13:35:27.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde00047d0 0x7efdf8135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdf8136d20 0x7efdf81dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdf81e01c0 0x7efdf80c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf81ef680 con 0x7efdf8136d20 2023-10-30T13:35:27.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf8153bd0 con 0x7efdf81e01c0 2023-10-30T13:35:27.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdfe29c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf80cab40 con 0x7efde00047d0 2023-10-30T13:35:27.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdf77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdf8136d20 0x7efdf81dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdf77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdf8136d20 0x7efdf81dfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60184/0 (socket says 172.21.15.148:60184) 2023-10-30T13:35:27.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.165+0000 7efdf77fe640 1 -- 172.21.15.148:0/505665498 learned_addr learned my addr 172.21.15.148:0/505665498 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:27.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdfc839640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdf81e01c0 0x7efdf80c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf7fff640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde00047d0 0x7efdf8135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf77fe640 1 -- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde00047d0 msgr2=0x7efdf8135ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf77fe640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde00047d0 0x7efdf8135ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf77fe640 1 -- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdf81e01c0 msgr2=0x7efdf80c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf77fe640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdf81e01c0 0x7efdf80c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf77fe640 1 -- 172.21.15.148:0/505665498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf81eea00 con 0x7efdf8136d20 2023-10-30T13:35:27.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdfc839640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdf81e01c0 0x7efdf80c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:27.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf77fe640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdf8136d20 0x7efdf81dfc80 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7efdec00c210 tx=0x7efde0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:27.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf4ff9640 1 -- 172.21.15.148:0/505665498 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec0049f0 con 0x7efdf8136d20 2023-10-30T13:35:27.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf4ff9640 1 -- 172.21.15.148:0/505665498 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec0b6070 con 0x7efdf8136d20 2023-10-30T13:35:27.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf4ff9640 1 -- 172.21.15.148:0/505665498 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec004d70 con 0x7efdf8136d20 2023-10-30T13:35:27.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf81eece0 con 0x7efdf8136d20 2023-10-30T13:35:27.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf81ef2a0 con 0x7efdf8136d20 2023-10-30T13:35:27.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf81ef7c0 con 0x7efdf8136d20 2023-10-30T13:35:27.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.169+0000 7efdf4ff9640 1 -- 172.21.15.148:0/505665498 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efdec0070d0 con 0x7efdf8136d20 2023-10-30T13:35:27.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.173+0000 7efdf4ff9640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efde40664d0 0x7efde4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.173+0000 7efdf4ff9640 1 -- 172.21.15.148:0/505665498 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7efdec0b8070 con 0x7efdf8136d20 2023-10-30T13:35:27.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.173+0000 7efdf7fff640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efde40664d0 0x7efde4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.173+0000 7efdf7fff640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efde40664d0 0x7efde4068990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efde80098d0 tx=0x7efde8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:27.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.177+0000 7efdf4ff9640 1 -- 172.21.15.148:0/505665498 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efdf81ef7c0 con 0x7efdf8136d20 2023-10-30T13:35:27.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.357+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdf81ef7c0 con 0x7efdf8136d20 2023-10-30T13:35:27.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.357+0000 7efdf4ff9640 1 -- 172.21.15.148:0/505665498 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7efdf81ef7c0 con 0x7efdf8136d20 2023-10-30T13:35:27.356 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:27.357 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:27.357 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:35:27.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efde40664d0 msgr2=0x7efde4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efde40664d0 0x7efde4068990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efde80098d0 tx=0x7efde8008040 comp rx=0 tx=0).stop 2023-10-30T13:35:27.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdf8136d20 msgr2=0x7efdf81dfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdf8136d20 0x7efdf81dfc80 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7efdec00c210 tx=0x7efde0005800 comp rx=0 tx=0).stop 2023-10-30T13:35:27.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 shutdown_connections 2023-10-30T13:35:27.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efde40664d0 0x7efde4068990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdf8136d20 0x7efdf81dfc80 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdf81e01c0 0x7efdf80c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 --2- 172.21.15.148:0/505665498 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efde00047d0 0x7efdf8135ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 >> 172.21.15.148:0/505665498 conn(0x7efdf80c7910 msgr2=0x7efdf80c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:27.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 shutdown_connections 2023-10-30T13:35:27.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.361+0000 7efdfe29c640 1 -- 172.21.15.148:0/505665498 wait complete. 2023-10-30T13:35:27.378 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:35:27.378 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:35:27.378 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:35:27.401 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:35:27.402 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:27.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.585+0000 7f2fce0c9640 1 Processor -- start 2023-10-30T13:35:27.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.585+0000 7f2fce0c9640 1 -- start start 2023-10-30T13:35:27.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fce0c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2fc80c05d0 0x7f2fc813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fce0c9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2fc80c6f60 0x7f2fc813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fce0c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc80c0a50 con 0x7f2fc80c6b80 2023-10-30T13:35:27.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fce0c9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc806f6e0 con 0x7f2fc80c6f60 2023-10-30T13:35:27.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fce0c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc806f9c0 con 0x7f2fc80c05d0 2023-10-30T13:35:27.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc77fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2fc80c6b80 0x7f2fc80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43082/0 (socket says 172.21.15.148:43082) 2023-10-30T13:35:27.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc77fe640 1 -- 172.21.15.148:0/2461234840 learned_addr learned my addr 172.21.15.148:0/2461234840 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:35:27.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc7fff640 1 --2- 172.21.15.148:0/2461234840 >> v2:172.21.15.148:3300/0 conn(0x7f2fc80c6f60 0x7f2fc813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fbffff640 1 --2- 172.21.15.148:0/2461234840 >> v2:172.21.15.150:3300/0 conn(0x7f2fc80c05d0 0x7f2fc813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc6ffd640 1 -- 172.21.15.148:0/2461234840 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41893424 0 0) 0x7f2fc80c0a50 con 0x7f2fc80c6b80 2023-10-30T13:35:27.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc6ffd640 1 -- 172.21.15.148:0/2461234840 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb4005740 con 0x7f2fc80c6b80 2023-10-30T13:35:27.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc7fff640 1 -- 172.21.15.148:0/2461234840 >> v2:172.21.15.150:3300/0 conn(0x7f2fc80c05d0 msgr2=0x7f2fc813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:27.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc7fff640 1 --2- 172.21.15.148:0/2461234840 >> v2:172.21.15.150:3300/0 conn(0x7f2fc80c05d0 0x7f2fc813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc7fff640 1 -- 172.21.15.148:0/2461234840 >> v1:172.21.15.148:6789/0 conn(0x7f2fc80c6b80 legacy=0x7f2fc80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc7fff640 1 -- 172.21.15.148:0/2461234840 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc806fca0 con 0x7f2fc80c6f60 2023-10-30T13:35:27.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.589+0000 7f2fc7fff640 1 --2- 172.21.15.148:0/2461234840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c6f60 0x7f2fc813f2d0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2fc806c7e0 tx=0x7f2fc001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abd62a8443b2bb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:27.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fc6ffd640 1 -- 172.21.15.148:0/2461234840 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00a58e0 con 0x7f2fc80c6f60 2023-10-30T13:35:27.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fc6ffd640 1 -- 172.21.15.148:0/2461234840 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc00a5a40 con 0x7f2fc80c6f60 2023-10-30T13:35:27.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fc6ffd640 1 -- 172.21.15.148:0/2461234840 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00a5dd0 con 0x7f2fc80c6f60 2023-10-30T13:35:27.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2461234840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c6f60 msgr2=0x7f2fc813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 --2- 172.21.15.148:0/2461234840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c6f60 0x7f2fc813f2d0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2fc806c7e0 tx=0x7f2fc001c910 comp rx=0 tx=0).stop 2023-10-30T13:35:27.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2461234840 shutdown_connections 2023-10-30T13:35:27.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 --2- 172.21.15.148:0/2461234840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c6f60 0x7f2fc813f2d0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 --2- 172.21.15.148:0/2461234840 >> v2:172.21.15.150:3300/0 conn(0x7f2fc80c05d0 0x7f2fc813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2461234840 >> 172.21.15.148:0/2461234840 conn(0x7f2fc80c7910 msgr2=0x7f2fc80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:27.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2461234840 shutdown_connections 2023-10-30T13:35:27.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2461234840 wait complete. 2023-10-30T13:35:27.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 Processor -- start 2023-10-30T13:35:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 -- start start 2023-10-30T13:35:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c05d0 0x7f2fc8150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fc80c6b80 0x7f2fc814b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2fc80c6f60 0x7f2fc8133190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc8150190 con 0x7f2fc80c05d0 2023-10-30T13:35:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc814fe60 con 0x7f2fc80c6b80 2023-10-30T13:35:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fce0c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc814d710 con 0x7f2fc80c6f60 2023-10-30T13:35:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fc77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c05d0 0x7f2fc8150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fc77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c05d0 0x7f2fc8150620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60206/0 (socket says 172.21.15.148:60206) 2023-10-30T13:35:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.593+0000 7f2fc77fe640 1 -- 172.21.15.148:0/2151155747 learned_addr learned my addr 172.21.15.148:0/2151155747 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fbffff640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fc80c6b80 0x7f2fc814b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc7fff640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2fc80c6f60 0x7f2fc8133190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc77fe640 1 -- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2fc80c6f60 msgr2=0x7f2fc8133190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc77fe640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2fc80c6f60 0x7f2fc8133190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc77fe640 1 -- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fc80c6b80 msgr2=0x7f2fc814b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc77fe640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fc80c6b80 0x7f2fc814b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc77fe640 1 -- 172.21.15.148:0/2151155747 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc0009a70 con 0x7f2fc80c05d0 2023-10-30T13:35:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fbffff640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fc80c6b80 0x7f2fc814b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc77fe640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c05d0 0x7f2fc8150620 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800c450 tx=0x7f2fb800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc4ff9640 1 -- 172.21.15.148:0/2151155747 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8005cf0 con 0x7f2fc80c05d0 2023-10-30T13:35:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc4ff9640 1 -- 172.21.15.148:0/2151155747 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8005e50 con 0x7f2fc80c05d0 2023-10-30T13:35:27.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc814d9f0 con 0x7f2fc80c05d0 2023-10-30T13:35:27.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc4ff9640 1 -- 172.21.15.148:0/2151155747 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8007290 con 0x7f2fc80c05d0 2023-10-30T13:35:27.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc80b0a70 con 0x7f2fc80c05d0 2023-10-30T13:35:27.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc804e170 con 0x7f2fc80c05d0 2023-10-30T13:35:27.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.597+0000 7f2fc4ff9640 1 -- 172.21.15.148:0/2151155747 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2fb80073f0 con 0x7f2fc80c05d0 2023-10-30T13:35:27.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.601+0000 7f2fc4ff9640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fa80664a0 0x7f2fa8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:27.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.601+0000 7f2fc4ff9640 1 -- 172.21.15.148:0/2151155747 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f2fb80839d0 con 0x7f2fc80c05d0 2023-10-30T13:35:27.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.601+0000 7f2fbffff640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fa80664a0 0x7f2fa8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:27.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.601+0000 7f2fbffff640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fa80664a0 0x7f2fa8068960 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80cade0 tx=0x7f2fb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:27.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.605+0000 7f2fc4ff9640 1 -- 172.21.15.148:0/2151155747 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2fb8050880 con 0x7f2fc80c05d0 2023-10-30T13:35:27.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.785+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fc814ffd0 con 0x7f2fc80c05d0 2023-10-30T13:35:27.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.785+0000 7f2fc4ff9640 1 -- 172.21.15.148:0/2151155747 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7f2fb8017070 con 0x7f2fc80c05d0 2023-10-30T13:35:27.780 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:27.781 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:27.781 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:35:27.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fa80664a0 msgr2=0x7f2fa8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fa80664a0 0x7f2fa8068960 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80cade0 tx=0x7f2fb0008040 comp rx=0 tx=0).stop 2023-10-30T13:35:27.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c05d0 msgr2=0x7f2fc8150620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:27.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c05d0 0x7f2fc8150620 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800c450 tx=0x7f2fb800c910 comp rx=0 tx=0).stop 2023-10-30T13:35:27.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 shutdown_connections 2023-10-30T13:35:27.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2fa80664a0 0x7f2fa8068960 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2fc80c6f60 0x7f2fc8133190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fc80c6b80 0x7f2fc814b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 --2- 172.21.15.148:0/2151155747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2fc80c05d0 0x7f2fc8150620 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 >> 172.21.15.148:0/2151155747 conn(0x7f2fc80c7910 msgr2=0x7f2fc80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 shutdown_connections 2023-10-30T13:35:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:27.789+0000 7f2fce0c9640 1 -- 172.21.15.148:0/2151155747 wait complete. 2023-10-30T13:35:27.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:35:29.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.957+0000 7f027a6d6640 1 Processor -- start 2023-10-30T13:35:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.957+0000 7f027a6d6640 1 -- start start 2023-10-30T13:35:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.957+0000 7f027a6d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f02740c6b80 0x7f02740bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.957+0000 7f027a6d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02740c6f60 0x7f02740c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.957+0000 7f027a6d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02740c0a50 con 0x7f02740c8430 2023-10-30T13:35:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.957+0000 7f027a6d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027406f6e0 con 0x7f02740c6b80 2023-10-30T13:35:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.957+0000 7f027a6d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027406f9c0 con 0x7f02740c6f60 2023-10-30T13:35:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0273fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f02740c6b80 0x7f02740bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0273fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f02740c6b80 0x7f02740bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60210/0 (socket says 172.21.15.148:60210) 2023-10-30T13:35:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0273fff640 1 -- 172.21.15.148:0/1949169819 learned_addr learned my addr 172.21.15.148:0/1949169819 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f02737fe640 1 --2- 172.21.15.148:0/1949169819 >> v2:172.21.15.150:3300/0 conn(0x7f02740c6f60 0x7f02740c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0273fff640 1 -- 172.21.15.148:0/1949169819 >> v2:172.21.15.150:3300/0 conn(0x7f02740c6f60 msgr2=0x7f02740c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0273fff640 1 --2- 172.21.15.148:0/1949169819 >> v2:172.21.15.150:3300/0 conn(0x7f02740c6f60 0x7f02740c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0273fff640 1 -- 172.21.15.148:0/1949169819 >> v1:172.21.15.150:6789/0 conn(0x7f02740c8430 legacy=0x7f027413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0273fff640 1 -- 172.21.15.148:0/1949169819 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027406fca0 con 0x7f02740c6b80 2023-10-30T13:35:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0273fff640 1 --2- 172.21.15.148:0/1949169819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6b80 0x7f02740bfec0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f0274159b90 tx=0x7f026401eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d647af439f6ba04c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0272ffd640 1 -- 172.21.15.148:0/1949169819 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02640a7b70 con 0x7f02740c6b80 2023-10-30T13:35:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0272ffd640 1 -- 172.21.15.148:0/1949169819 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02640a7cd0 con 0x7f02740c6b80 2023-10-30T13:35:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.961+0000 7f0272ffd640 1 -- 172.21.15.148:0/1949169819 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02640af660 con 0x7f02740c6b80 2023-10-30T13:35:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 -- 172.21.15.148:0/1949169819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6b80 msgr2=0x7f02740bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 --2- 172.21.15.148:0/1949169819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6b80 0x7f02740bfec0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f0274159b90 tx=0x7f026401eb90 comp rx=0 tx=0).stop 2023-10-30T13:35:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 -- 172.21.15.148:0/1949169819 shutdown_connections 2023-10-30T13:35:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 --2- 172.21.15.148:0/1949169819 >> v2:172.21.15.150:3300/0 conn(0x7f02740c6f60 0x7f02740c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 --2- 172.21.15.148:0/1949169819 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6b80 0x7f02740bfec0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 -- 172.21.15.148:0/1949169819 >> 172.21.15.148:0/1949169819 conn(0x7f02740c7910 msgr2=0x7f02740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 -- 172.21.15.148:0/1949169819 shutdown_connections 2023-10-30T13:35:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 -- 172.21.15.148:0/1949169819 wait complete. 2023-10-30T13:35:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 Processor -- start 2023-10-30T13:35:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 -- start start 2023-10-30T13:35:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02740c6b80 0x7f027413a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6f60 0x7f027413ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02740c8430 0x7f02740c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02740b0030 con 0x7f02740c6f60 2023-10-30T13:35:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027412f300 con 0x7f02740c8430 2023-10-30T13:35:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f027a6d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027413a440 con 0x7f02740c6b80 2023-10-30T13:35:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f02737fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6f60 0x7f027413ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f02737fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6f60 0x7f027413ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60222/0 (socket says 172.21.15.148:60222) 2023-10-30T13:35:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f02737fe640 1 -- 172.21.15.148:0/1641213874 learned_addr learned my addr 172.21.15.148:0/1641213874 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f0278c73640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02740c8430 0x7f02740c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.965+0000 7f0273fff640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02740c6b80 0x7f027413a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f02737fe640 1 -- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02740c6b80 msgr2=0x7f027413a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f02737fe640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02740c6b80 0x7f027413a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f02737fe640 1 -- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02740c8430 msgr2=0x7f02740c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f02737fe640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02740c8430 0x7f02740c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f02737fe640 1 -- 172.21.15.148:0/1641213874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026400b770 con 0x7f02740c6f60 2023-10-30T13:35:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f02737fe640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6f60 0x7f027413ae10 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f026800d830 tx=0x7f026800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f0270ff9640 1 -- 172.21.15.148:0/1641213874 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026800b660 con 0x7f02740c6f60 2023-10-30T13:35:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f0270ff9640 1 -- 172.21.15.148:0/1641213874 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026800b7c0 con 0x7f02740c6f60 2023-10-30T13:35:29.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f0270ff9640 1 -- 172.21.15.148:0/1641213874 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026800bb30 con 0x7f02740c6f60 2023-10-30T13:35:29.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027413cfc0 con 0x7f02740c6f60 2023-10-30T13:35:29.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027413e6b0 con 0x7f02740c6f60 2023-10-30T13:35:29.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027413a5f0 con 0x7f02740c6f60 2023-10-30T13:35:29.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f0270ff9640 1 -- 172.21.15.148:0/1641213874 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f026801e460 con 0x7f02740c6f60 2023-10-30T13:35:29.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f0270ff9640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02600664a0 0x7f0260068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:29.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.969+0000 7f0270ff9640 1 -- 172.21.15.148:0/1641213874 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f0268084870 con 0x7f02740c6f60 2023-10-30T13:35:29.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.973+0000 7f0273fff640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02600664a0 0x7f0260068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:29.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.973+0000 7f0273fff640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02600664a0 0x7f0260068960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f02640b1000 tx=0x7f02640a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:29.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:29.973+0000 7f0270ff9640 1 -- 172.21.15.148:0/1641213874 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f027413a5f0 con 0x7f02740c6f60 2023-10-30T13:35:30.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f027413a5b0 con 0x7f02740c6f60 2023-10-30T13:35:30.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f0270ff9640 1 -- 172.21.15.148:0/1641213874 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7f0268051990 con 0x7f02740c6f60 2023-10-30T13:35:30.151 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:30.152 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:30.152 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:35:30.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02600664a0 msgr2=0x7f0260068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02600664a0 0x7f0260068960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f02640b1000 tx=0x7f02640a9070 comp rx=0 tx=0).stop 2023-10-30T13:35:30.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6f60 msgr2=0x7f027413ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:30.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6f60 0x7f027413ae10 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f026800d830 tx=0x7f026800dcf0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 shutdown_connections 2023-10-30T13:35:30.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02600664a0 0x7f0260068960 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02740c6f60 0x7f027413ae10 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02740c8430 0x7f02740c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 --2- 172.21.15.148:0/1641213874 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02740c6b80 0x7f027413a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:30.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 >> 172.21.15.148:0/1641213874 conn(0x7f02740c7910 msgr2=0x7f02740c11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:30.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 shutdown_connections 2023-10-30T13:35:30.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:30.157+0000 7f027a6d6640 1 -- 172.21.15.148:0/1641213874 wait complete. 2023-10-30T13:35:30.172 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:35:30.361 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:30.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:35:32.258 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:32.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093fb0f640 1 Processor -- start 2023-10-30T13:35:32.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093fb0f640 1 -- start start 2023-10-30T13:35:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093fb0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09380b1ea0 0x7f093806a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093fb0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093806ade0 con 0x7f09380cc0b0 2023-10-30T13:35:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093fb0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09381598f0 con 0x7f0938153c30 2023-10-30T13:35:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093fb0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09381596e0 con 0x7f09380b1ea0 2023-10-30T13:35:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093d0aa640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f09380cc0b0 0x7f09380cabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47956/0 (socket says 172.21.15.148:47956) 2023-10-30T13:35:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093d0aa640 1 -- 172.21.15.148:0/1396525373 learned_addr learned my addr 172.21.15.148:0/1396525373 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:35:32.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093e0ac640 1 --2- 172.21.15.148:0/1396525373 >> v2:172.21.15.150:3300/0 conn(0x7f09380b1ea0 0x7f093806a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:32.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733736950 0 0) 0x7f09280012a0 con 0x7f09380cc0b0 2023-10-30T13:35:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0924003530 con 0x7f09380cc0b0 2023-10-30T13:35:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3009865201 0 0) 0x7f0924003530 con 0x7f09380cc0b0 2023-10-30T13:35:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 >> v2:172.21.15.150:3300/0 conn(0x7f09380b1ea0 msgr2=0x7f093806a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 --2- 172.21.15.148:0/1396525373 >> v2:172.21.15.150:3300/0 conn(0x7f09380b1ea0 0x7f093806a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 >> v1:172.21.15.150:6789/0 conn(0x7f0938153c30 legacy=0x7f09380cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938159b90 con 0x7f09380cc0b0 2023-10-30T13:35:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f0928001990 con 0x7f09380cc0b0 2023-10-30T13:35:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 >> v1:172.21.15.148:6789/0 conn(0x7f09380cc0b0 legacy=0x7f09380cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0924003980 0x7f0924003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f0924004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0924005190 0x7f092400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09381596e0 con 0x7f09240047d0 2023-10-30T13:35:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092400c040 con 0x7f0924005190 2023-10-30T13:35:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092400c200 con 0x7f0924003980 2023-10-30T13:35:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.413+0000 7f093d0aa640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f0924004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093d8ab640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0924003980 0x7f0924003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093e0ac640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0924005190 0x7f092400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093d0aa640 1 -- 172.21.15.148:0/1396525373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0924003980 msgr2=0x7f0924003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093d0aa640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0924003980 0x7f0924003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093d0aa640 1 -- 172.21.15.148:0/1396525373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0924005190 msgr2=0x7f092400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093d0aa640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0924005190 0x7f092400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093d0aa640 1 -- 172.21.15.148:0/1396525373 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924003530 con 0x7f09240047d0 2023-10-30T13:35:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093e0ac640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0924005190 0x7f092400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093e0ac640 1 -- 172.21.15.148:0/1396525373 reap_dead start 2023-10-30T13:35:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093d0aa640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f0924004c50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f092800c190 tx=0x7f092801cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c239b3aa2ba144db server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09280a5ae0 con 0x7f09240047d0 2023-10-30T13:35:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09280a5c60 con 0x7f09240047d0 2023-10-30T13:35:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093c8a9640 1 -- 172.21.15.148:0/1396525373 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09280072b0 con 0x7f09240047d0 2023-10-30T13:35:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093fb0f640 1 -- 172.21.15.148:0/1396525373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 msgr2=0x7f0924004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:32.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.417+0000 7f093fb0f640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f0924004c50 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f092800c190 tx=0x7f092801cbe0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 -- 172.21.15.148:0/1396525373 shutdown_connections 2023-10-30T13:35:32.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 --2- 172.21.15.148:0/1396525373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f0924004c50 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 -- 172.21.15.148:0/1396525373 >> 172.21.15.148:0/1396525373 conn(0x7f093806bf40 msgr2=0x7f093806c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:32.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 -- 172.21.15.148:0/1396525373 shutdown_connections 2023-10-30T13:35:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 -- 172.21.15.148:0/1396525373 wait complete. 2023-10-30T13:35:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 Processor -- start 2023-10-30T13:35:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 -- start start 2023-10-30T13:35:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f09380cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0938158440 0x7f09380cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:32.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09380ce4a0 0x7f093815bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:32.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09380cc980 con 0x7f09240047d0 2023-10-30T13:35:32.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09380cdad0 con 0x7f09380ce4a0 2023-10-30T13:35:32.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093fb0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09380c8610 con 0x7f0938158440 2023-10-30T13:35:32.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d8ab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f09380cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:32.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d8ab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f09380cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53870/0 (socket says 172.21.15.148:53870) 2023-10-30T13:35:32.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d8ab640 1 -- 172.21.15.148:0/2426585015 learned_addr learned my addr 172.21.15.148:0/2426585015 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:32.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093e0ac640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09380ce4a0 0x7f093815bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:32.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d0aa640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0938158440 0x7f09380cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:32.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d8ab640 1 -- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0938158440 msgr2=0x7f09380cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:32.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d8ab640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0938158440 0x7f09380cdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d8ab640 1 -- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09380ce4a0 msgr2=0x7f093815bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:32.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d8ab640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09380ce4a0 0x7f093815bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d8ab640 1 -- 172.21.15.148:0/2426585015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09380ce920 con 0x7f09240047d0 2023-10-30T13:35:32.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.421+0000 7f093d8ab640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f09380cd0e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f093400f610 tx=0x7f093400fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:32.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f092e7fc640 1 -- 172.21.15.148:0/2426585015 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09340052e0 con 0x7f09240047d0 2023-10-30T13:35:32.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f092e7fc640 1 -- 172.21.15.148:0/2426585015 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0934005440 con 0x7f09240047d0 2023-10-30T13:35:32.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f092e7fc640 1 -- 172.21.15.148:0/2426585015 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934003820 con 0x7f09240047d0 2023-10-30T13:35:32.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09380c7a90 con 0x7f09240047d0 2023-10-30T13:35:32.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09380c8050 con 0x7f09240047d0 2023-10-30T13:35:32.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093804e170 con 0x7f09240047d0 2023-10-30T13:35:32.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f092e7fc640 1 -- 172.21.15.148:0/2426585015 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0934003980 con 0x7f09240047d0 2023-10-30T13:35:32.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f092e7fc640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f091c0664a0 0x7f091c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:32.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f092e7fc640 1 -- 172.21.15.148:0/2426585015 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f09340842e0 con 0x7f09240047d0 2023-10-30T13:35:32.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.425+0000 7f093d0aa640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f091c0664a0 0x7f091c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:32.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.429+0000 7f093d0aa640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f091c0664a0 0x7f091c068960 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f09280ab000 tx=0x7f09280a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:32.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.429+0000 7f092e7fc640 1 -- 172.21.15.148:0/2426585015 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0934051190 con 0x7f09240047d0 2023-10-30T13:35:32.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.609+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09380c9440 con 0x7f09240047d0 2023-10-30T13:35:32.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.609+0000 7f092e7fc640 1 -- 172.21.15.148:0/2426585015 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7f09340547b0 con 0x7f09240047d0 2023-10-30T13:35:32.606 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:32.606 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:32.606 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:35:32.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f091c0664a0 msgr2=0x7f091c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:32.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f091c0664a0 0x7f091c068960 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f09280ab000 tx=0x7f09280a7070 comp rx=0 tx=0).stop 2023-10-30T13:35:32.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 msgr2=0x7f09380cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:32.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f09380cd0e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f093400f610 tx=0x7f093400fad0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 shutdown_connections 2023-10-30T13:35:32.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f091c0664a0 0x7f091c068960 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09380ce4a0 0x7f093815bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0938158440 0x7f09380cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 --2- 172.21.15.148:0/2426585015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09240047d0 0x7f09380cd0e0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 >> 172.21.15.148:0/2426585015 conn(0x7f093806bf40 msgr2=0x7f0938151b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 shutdown_connections 2023-10-30T13:35:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:32.613+0000 7f093fb0f640 1 -- 172.21.15.148:0/2426585015 wait complete. 2023-10-30T13:35:32.627 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:35:34.628 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:34.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.785+0000 7fe8bf291640 1 Processor -- start 2023-10-30T13:35:34.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.785+0000 7fe8bf291640 1 -- start start 2023-10-30T13:35:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.785+0000 7fe8bf291640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe8b81538a0 0x7fe8b815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.785+0000 7fe8bf291640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe8b80ca0e0 0x7fe8b815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.785+0000 7fe8bf291640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b80cb800 con 0x7fe8b80cf160 2023-10-30T13:35:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.785+0000 7fe8bf291640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b815dc30 con 0x7fe8b81538a0 2023-10-30T13:35:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.785+0000 7fe8bf291640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b80cb330 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.789+0000 7fe8bd02d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe8b81538a0 0x7fe8b815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.789+0000 7fe8bd02d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe8b81538a0 0x7fe8b815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53884/0 (socket says 172.21.15.148:53884) 2023-10-30T13:35:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.789+0000 7fe8bd02d640 1 -- 172.21.15.148:0/2882958662 learned_addr learned my addr 172.21.15.148:0/2882958662 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.789+0000 7fe8bc82c640 1 --2- 172.21.15.148:0/2882958662 >> v2:172.21.15.150:3300/0 conn(0x7fe8b80ca0e0 0x7fe8b815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.789+0000 7fe8bd02d640 1 -- 172.21.15.148:0/2882958662 >> v2:172.21.15.150:3300/0 conn(0x7fe8b80ca0e0 msgr2=0x7fe8b815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.789+0000 7fe8bd02d640 1 --2- 172.21.15.148:0/2882958662 >> v2:172.21.15.150:3300/0 conn(0x7fe8b80ca0e0 0x7fe8b815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.789+0000 7fe8bd02d640 1 -- 172.21.15.148:0/2882958662 >> v1:172.21.15.150:6789/0 conn(0x7fe8b80cf160 legacy=0x7fe8b80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.789+0000 7fe8bd02d640 1 -- 172.21.15.148:0/2882958662 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b80cad70 con 0x7fe8b81538a0 2023-10-30T13:35:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.793+0000 7fe8bd02d640 1 --2- 172.21.15.148:0/2882958662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b81538a0 0x7fe8b815d6f0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80b30a0 tx=0x7fe8b401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647d399b20aea720 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.793+0000 7fe8affff640 1 -- 172.21.15.148:0/2882958662 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b40a89f0 con 0x7fe8b81538a0 2023-10-30T13:35:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.793+0000 7fe8affff640 1 -- 172.21.15.148:0/2882958662 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b40a8b50 con 0x7fe8b81538a0 2023-10-30T13:35:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.793+0000 7fe8affff640 1 -- 172.21.15.148:0/2882958662 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b40b0440 con 0x7fe8b81538a0 2023-10-30T13:35:34.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.793+0000 7fe8bf291640 1 -- 172.21.15.148:0/2882958662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b81538a0 msgr2=0x7fe8b815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.793+0000 7fe8bf291640 1 --2- 172.21.15.148:0/2882958662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b81538a0 0x7fe8b815d6f0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80b30a0 tx=0x7fe8b401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 -- 172.21.15.148:0/2882958662 shutdown_connections 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 --2- 172.21.15.148:0/2882958662 >> v2:172.21.15.150:3300/0 conn(0x7fe8b80ca0e0 0x7fe8b815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 --2- 172.21.15.148:0/2882958662 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b81538a0 0x7fe8b815d6f0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 -- 172.21.15.148:0/2882958662 >> 172.21.15.148:0/2882958662 conn(0x7fe8b806bfe0 msgr2=0x7fe8b806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:34.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 -- 172.21.15.148:0/2882958662 shutdown_connections 2023-10-30T13:35:34.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 -- 172.21.15.148:0/2882958662 wait complete. 2023-10-30T13:35:34.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 Processor -- start 2023-10-30T13:35:34.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 -- start start 2023-10-30T13:35:34.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b80ca0e0 0x7fe8b8158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8b80cf160 0x7fe8b80c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8b81538a0 0x7fe8b81ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b81e4db0 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b81ef520 con 0x7fe8b80cf160 2023-10-30T13:35:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bf291640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b81f2960 con 0x7fe8b81538a0 2023-10-30T13:35:34.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bd02d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b80ca0e0 0x7fe8b8158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bd02d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b80ca0e0 0x7fe8b8158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53888/0 (socket says 172.21.15.148:53888) 2023-10-30T13:35:34.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bd02d640 1 -- 172.21.15.148:0/304599001 learned_addr learned my addr 172.21.15.148:0/304599001 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:34.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bd82e640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8b81538a0 0x7fe8b81ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bc82c640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8b80cf160 0x7fe8b80c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bd02d640 1 -- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8b81538a0 msgr2=0x7fe8b81ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bd02d640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8b81538a0 0x7fe8b81ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bd02d640 1 -- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8b80cf160 msgr2=0x7fe8b80c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bd02d640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8b80cf160 0x7fe8b80c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.797+0000 7fe8bd02d640 1 -- 172.21.15.148:0/304599001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b40094d0 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.801+0000 7fe8bd02d640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b80ca0e0 0x7fe8b8158440 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe8b401b900 tx=0x7fe8b401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.801+0000 7fe8adffb640 1 -- 172.21.15.148:0/304599001 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b40ba600 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.801+0000 7fe8adffb640 1 -- 172.21.15.148:0/304599001 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b40ba760 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.801+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b81f2c20 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.801+0000 7fe8adffb640 1 -- 172.21.15.148:0/304599001 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b40bab10 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.801+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b81f31e0 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.801+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b80b2a50 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.801+0000 7fe8adffb640 1 -- 172.21.15.148:0/304599001 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8b40bac70 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.805+0000 7fe8adffb640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe89c0664d0 0x7fe89c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:34.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.805+0000 7fe8adffb640 1 -- 172.21.15.148:0/304599001 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fe8b41226a0 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.805+0000 7fe8bc82c640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe89c0664d0 0x7fe89c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:34.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.805+0000 7fe8bc82c640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe89c0664d0 0x7fe89c068990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80cf0d0 tx=0x7fe8a8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:34.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.809+0000 7fe8adffb640 1 -- 172.21.15.148:0/304599001 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8b40ef730 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.989+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8b8159850 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.989+0000 7fe8adffb640 1 -- 172.21.15.148:0/304599001 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7fe8b40f2d50 con 0x7fe8b80ca0e0 2023-10-30T13:35:34.986 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:34.987 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:34.989 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:35:34.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe89c0664d0 msgr2=0x7fe89c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe89c0664d0 0x7fe89c068990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80cf0d0 tx=0x7fe8a8009190 comp rx=0 tx=0).stop 2023-10-30T13:35:34.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b80ca0e0 msgr2=0x7fe8b8158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:34.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b80ca0e0 0x7fe8b8158440 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe8b401b900 tx=0x7fe8b401ba90 comp rx=0 tx=0).stop 2023-10-30T13:35:34.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 shutdown_connections 2023-10-30T13:35:34.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe89c0664d0 0x7fe89c068990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe8b81538a0 0x7fe8b81ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8b80cf160 0x7fe8b80c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 --2- 172.21.15.148:0/304599001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8b80ca0e0 0x7fe8b8158440 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:34.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 >> 172.21.15.148:0/304599001 conn(0x7fe8b806bfe0 msgr2=0x7fe8b8151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:34.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 shutdown_connections 2023-10-30T13:35:34.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:34.993+0000 7fe8bf291640 1 -- 172.21.15.148:0/304599001 wait complete. 2023-10-30T13:35:35.007 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:35:37.007 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:37.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3ae33e640 1 Processor -- start 2023-10-30T13:35:37.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3ae33e640 1 -- start start 2023-10-30T13:35:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3ae33e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc3a815d6f0 0x7fc3a815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3ae33e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a80cb800 con 0x7fc3a81538a0 2023-10-30T13:35:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3ae33e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a80cb330 con 0x7fc3a80cf160 2023-10-30T13:35:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3ae33e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a80cad70 con 0x7fc3a815d6f0 2023-10-30T13:35:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3a7fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc3a81538a0 0x7fc3a80ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47970/0 (socket says 172.21.15.148:47970) 2023-10-30T13:35:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3a7fff640 1 -- 172.21.15.148:0/338137749 learned_addr learned my addr 172.21.15.148:0/338137749 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:35:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3a77fe640 1 --2- 172.21.15.148:0/338137749 >> v2:172.21.15.148:3300/0 conn(0x7fc3a815d6f0 0x7fc3a815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3a6ffd640 1 -- 172.21.15.148:0/338137749 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295899635 0 0) 0x7fc3a80cb800 con 0x7fc3a81538a0 2023-10-30T13:35:37.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3a6ffd640 1 -- 172.21.15.148:0/338137749 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc390003530 con 0x7fc3a81538a0 2023-10-30T13:35:37.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3a77fe640 1 -- 172.21.15.148:0/338137749 >> v1:172.21.15.150:6789/0 conn(0x7fc3a80cf160 legacy=0x7fc3a8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3a77fe640 1 -- 172.21.15.148:0/338137749 >> v1:172.21.15.148:6789/0 conn(0x7fc3a81538a0 legacy=0x7fc3a80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.161+0000 7fc3a77fe640 1 -- 172.21.15.148:0/338137749 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80cb050 con 0x7fc3a815d6f0 2023-10-30T13:35:37.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3a77fe640 1 --2- 172.21.15.148:0/338137749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a815d6f0 0x7fc3a815a090 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0097c0 tx=0x7fc39c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9358ceb889768d84 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:37.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3a6ffd640 1 -- 172.21.15.148:0/338137749 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0a8970 con 0x7fc3a815d6f0 2023-10-30T13:35:37.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3a6ffd640 1 -- 172.21.15.148:0/338137749 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c0a8ad0 con 0x7fc3a815d6f0 2023-10-30T13:35:37.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3a6ffd640 1 -- 172.21.15.148:0/338137749 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0a8e60 con 0x7fc3a815d6f0 2023-10-30T13:35:37.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3ae33e640 1 -- 172.21.15.148:0/338137749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a815d6f0 msgr2=0x7fc3a815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3ae33e640 1 --2- 172.21.15.148:0/338137749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a815d6f0 0x7fc3a815a090 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0097c0 tx=0x7fc39c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:35:37.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3ae33e640 1 -- 172.21.15.148:0/338137749 shutdown_connections 2023-10-30T13:35:37.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3ae33e640 1 --2- 172.21.15.148:0/338137749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a815d6f0 0x7fc3a815a090 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3ae33e640 1 -- 172.21.15.148:0/338137749 >> 172.21.15.148:0/338137749 conn(0x7fc3a806bfe0 msgr2=0x7fc3a806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:37.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3ae33e640 1 -- 172.21.15.148:0/338137749 shutdown_connections 2023-10-30T13:35:37.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.165+0000 7fc3ae33e640 1 -- 172.21.15.148:0/338137749 wait complete. 2023-10-30T13:35:37.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 Processor -- start 2023-10-30T13:35:37.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 -- start start 2023-10-30T13:35:37.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3a80cf160 0x7fc3a80b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a81538a0 0x7fc3a8147830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3a815d6f0 0x7fc3a81463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80b36f0 con 0x7fc3a81538a0 2023-10-30T13:35:37.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80b51b0 con 0x7fc3a80cf160 2023-10-30T13:35:37.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80b4140 con 0x7fc3a815d6f0 2023-10-30T13:35:37.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a81538a0 0x7fc3a8147830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a81538a0 0x7fc3a8147830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53898/0 (socket says 172.21.15.148:53898) 2023-10-30T13:35:37.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a77fe640 1 -- 172.21.15.148:0/51202363 learned_addr learned my addr 172.21.15.148:0/51202363 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:37.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ac8db640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3a815d6f0 0x7fc3a81463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a7fff640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3a80cf160 0x7fc3a80b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a77fe640 1 -- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3a815d6f0 msgr2=0x7fc3a81463e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a77fe640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3a815d6f0 0x7fc3a81463e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a77fe640 1 -- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3a80cf160 msgr2=0x7fc3a80b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a77fe640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3a80cf160 0x7fc3a80b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a77fe640 1 -- 172.21.15.148:0/51202363 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c009420 con 0x7fc3a81538a0 2023-10-30T13:35:37.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ac8db640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3a815d6f0 0x7fc3a81463e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:37.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a77fe640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a81538a0 0x7fc3a8147830 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fc39c01b880 tx=0x7fc39c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:37.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a4ff9640 1 -- 172.21.15.148:0/51202363 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0b9630 con 0x7fc3a81538a0 2023-10-30T13:35:37.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a4ff9640 1 -- 172.21.15.148:0/51202363 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c0b97b0 con 0x7fc3a81538a0 2023-10-30T13:35:37.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3a4ff9640 1 -- 172.21.15.148:0/51202363 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0b9b60 con 0x7fc3a81538a0 2023-10-30T13:35:37.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a80b4ef0 con 0x7fc3a81538a0 2023-10-30T13:35:37.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.169+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a80ce780 con 0x7fc3a81538a0 2023-10-30T13:35:37.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.173+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a80670f0 con 0x7fc3a81538a0 2023-10-30T13:35:37.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.173+0000 7fc3a4ff9640 1 -- 172.21.15.148:0/51202363 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc39c0b7040 con 0x7fc3a81538a0 2023-10-30T13:35:37.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.173+0000 7fc3a4ff9640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3940664d0 0x7fc394068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:37.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.173+0000 7fc3a4ff9640 1 -- 172.21.15.148:0/51202363 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7fc39c1222b0 con 0x7fc3a81538a0 2023-10-30T13:35:37.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.173+0000 7fc3a7fff640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3940664d0 0x7fc394068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:37.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.173+0000 7fc3a7fff640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3940664d0 0x7fc394068990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc3980011f0 tx=0x7fc398009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:37.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.177+0000 7fc3a4ff9640 1 -- 172.21.15.148:0/51202363 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc39c0ef1f0 con 0x7fc3a81538a0 2023-10-30T13:35:37.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.357+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3a804e170 con 0x7fc3a81538a0 2023-10-30T13:35:37.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.357+0000 7fc3a4ff9640 1 -- 172.21.15.148:0/51202363 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7fc39c0f2810 con 0x7fc3a81538a0 2023-10-30T13:35:37.352 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:37.353 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:37.353 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:35:37.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3940664d0 msgr2=0x7fc394068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3940664d0 0x7fc394068990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc3980011f0 tx=0x7fc398009040 comp rx=0 tx=0).stop 2023-10-30T13:35:37.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a81538a0 msgr2=0x7fc3a8147830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:37.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a81538a0 0x7fc3a8147830 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fc39c01b880 tx=0x7fc39c01ba10 comp rx=0 tx=0).stop 2023-10-30T13:35:37.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 shutdown_connections 2023-10-30T13:35:37.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc3940664d0 0x7fc394068990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc3a81538a0 0x7fc3a8147830 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc3a815d6f0 0x7fc3a81463e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 --2- 172.21.15.148:0/51202363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3a80cf160 0x7fc3a80b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 >> 172.21.15.148:0/51202363 conn(0x7fc3a806bfe0 msgr2=0x7fc3a806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 shutdown_connections 2023-10-30T13:35:37.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:37.361+0000 7fc3ae33e640 1 -- 172.21.15.148:0/51202363 wait complete. 2023-10-30T13:35:37.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:35:39.374 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:39.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e9b01640 1 Processor -- start 2023-10-30T13:35:39.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e9b01640 1 -- start start 2023-10-30T13:35:39.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e9b01640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f50e415d6f0 0x7f50e415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e9b01640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f50e40cf160 0x7f50e40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e9b01640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e40cb800 con 0x7f50e41538a0 2023-10-30T13:35:39.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e9b01640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e40cb330 con 0x7f50e415d6f0 2023-10-30T13:35:39.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e9b01640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e40cad70 con 0x7f50e40cf160 2023-10-30T13:35:39.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e2ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f50e415d6f0 0x7f50e415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e2ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f50e415d6f0 0x7f50e415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53904/0 (socket says 172.21.15.148:53904) 2023-10-30T13:35:39.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e2ffd640 1 -- 172.21.15.148:0/1142862279 learned_addr learned my addr 172.21.15.148:0/1142862279 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:39.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e3fff640 1 --2- 172.21.15.148:0/1142862279 >> v2:172.21.15.150:3300/0 conn(0x7f50e40cf160 0x7f50e40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e2ffd640 1 -- 172.21.15.148:0/1142862279 >> v2:172.21.15.150:3300/0 conn(0x7f50e40cf160 msgr2=0x7f50e40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:39.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e2ffd640 1 --2- 172.21.15.148:0/1142862279 >> v2:172.21.15.150:3300/0 conn(0x7f50e40cf160 0x7f50e40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e2ffd640 1 -- 172.21.15.148:0/1142862279 >> v1:172.21.15.150:6789/0 conn(0x7f50e41538a0 legacy=0x7f50e40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e2ffd640 1 -- 172.21.15.148:0/1142862279 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e40cb050 con 0x7f50e415d6f0 2023-10-30T13:35:39.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.529+0000 7f50e3fff640 1 --2- 172.21.15.148:0/1142862279 >> v2:172.21.15.150:3300/0 conn(0x7f50e40cf160 0x7f50e40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:39.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e2ffd640 1 --2- 172.21.15.148:0/1142862279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e415d6f0 0x7f50e415a090 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f50e40b5330 tx=0x7f50d801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7164d2fb5f1d6bc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:39.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e27fc640 1 -- 172.21.15.148:0/1142862279 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d80a89f0 con 0x7f50e415d6f0 2023-10-30T13:35:39.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e27fc640 1 -- 172.21.15.148:0/1142862279 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d80a8b50 con 0x7f50e415d6f0 2023-10-30T13:35:39.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e27fc640 1 -- 172.21.15.148:0/1142862279 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d80b0440 con 0x7f50e415d6f0 2023-10-30T13:35:39.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e9b01640 1 -- 172.21.15.148:0/1142862279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e415d6f0 msgr2=0x7f50e415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e9b01640 1 --2- 172.21.15.148:0/1142862279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e415d6f0 0x7f50e415a090 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f50e40b5330 tx=0x7f50d801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e9b01640 1 -- 172.21.15.148:0/1142862279 shutdown_connections 2023-10-30T13:35:39.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e9b01640 1 --2- 172.21.15.148:0/1142862279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e415d6f0 0x7f50e415a090 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e9b01640 1 --2- 172.21.15.148:0/1142862279 >> v2:172.21.15.150:3300/0 conn(0x7f50e40cf160 0x7f50e40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e9b01640 1 -- 172.21.15.148:0/1142862279 >> 172.21.15.148:0/1142862279 conn(0x7f50e406bfe0 msgr2=0x7f50e406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:39.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e9b01640 1 -- 172.21.15.148:0/1142862279 shutdown_connections 2023-10-30T13:35:39.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.533+0000 7f50e9b01640 1 -- 172.21.15.148:0/1142862279 wait complete. 2023-10-30T13:35:39.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 Processor -- start 2023-10-30T13:35:39.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 -- start start 2023-10-30T13:35:39.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e40cf160 0x7f50e4158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e41538a0 0x7f50e40c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50e415d6f0 0x7f50e40b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4159030 con 0x7f50e41538a0 2023-10-30T13:35:39.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4158d50 con 0x7f50e40cf160 2023-10-30T13:35:39.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4157690 con 0x7f50e415d6f0 2023-10-30T13:35:39.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e41538a0 0x7f50e40c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e41538a0 0x7f50e40c7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53908/0 (socket says 172.21.15.148:53908) 2023-10-30T13:35:39.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e2ffd640 1 -- 172.21.15.148:0/3753580362 learned_addr learned my addr 172.21.15.148:0/3753580362 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:39.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e37fe640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e40cf160 0x7f50e4158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e3fff640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50e415d6f0 0x7f50e40b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e2ffd640 1 -- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50e415d6f0 msgr2=0x7f50e40b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e2ffd640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50e415d6f0 0x7f50e40b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e2ffd640 1 -- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e40cf160 msgr2=0x7f50e4158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e2ffd640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e40cf160 0x7f50e4158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e2ffd640 1 -- 172.21.15.148:0/3753580362 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d80094d0 con 0x7f50e41538a0 2023-10-30T13:35:39.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e2ffd640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e41538a0 0x7f50e40c7410 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f50d801b900 tx=0x7f50d801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:39.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50cbfff640 1 -- 172.21.15.148:0/3753580362 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d80ba600 con 0x7f50e41538a0 2023-10-30T13:35:39.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50cbfff640 1 -- 172.21.15.148:0/3753580362 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d80ba760 con 0x7f50e41538a0 2023-10-30T13:35:39.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50cbfff640 1 -- 172.21.15.148:0/3753580362 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d80bab10 con 0x7f50e41538a0 2023-10-30T13:35:39.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e4157970 con 0x7f50e41538a0 2023-10-30T13:35:39.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.537+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e4157f30 con 0x7f50e41538a0 2023-10-30T13:35:39.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.541+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e40b2a50 con 0x7f50e41538a0 2023-10-30T13:35:39.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.541+0000 7f50cbfff640 1 -- 172.21.15.148:0/3753580362 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f50d80bac70 con 0x7f50e41538a0 2023-10-30T13:35:39.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.541+0000 7f50cbfff640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50d00664a0 0x7f50d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:39.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.541+0000 7f50cbfff640 1 -- 172.21.15.148:0/3753580362 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f50d8122710 con 0x7f50e41538a0 2023-10-30T13:35:39.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.541+0000 7f50e37fe640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50d00664a0 0x7f50d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:39.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.541+0000 7f50e37fe640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50d00664a0 0x7f50d0068960 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f50d4007fd0 tx=0x7f50d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:39.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.545+0000 7f50cbfff640 1 -- 172.21.15.148:0/3753580362 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f50d80ef830 con 0x7f50e41538a0 2023-10-30T13:35:39.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50e40670f0 con 0x7f50e41538a0 2023-10-30T13:35:39.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50cbfff640 1 -- 172.21.15.148:0/3753580362 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7f50d80f2e50 con 0x7f50e41538a0 2023-10-30T13:35:39.720 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:39.720 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:39.721 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:35:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50d00664a0 msgr2=0x7f50d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50d00664a0 0x7f50d0068960 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f50d4007fd0 tx=0x7f50d4009040 comp rx=0 tx=0).stop 2023-10-30T13:35:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e41538a0 msgr2=0x7f50e40c7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e41538a0 0x7f50e40c7410 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f50d801b900 tx=0x7f50d801ba90 comp rx=0 tx=0).stop 2023-10-30T13:35:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 shutdown_connections 2023-10-30T13:35:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f50d00664a0 0x7f50d0068960 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e41538a0 0x7f50e40c7410 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f50e415d6f0 0x7f50e40b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 --2- 172.21.15.148:0/3753580362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e40cf160 0x7f50e4158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 >> 172.21.15.148:0/3753580362 conn(0x7f50e406bfe0 msgr2=0x7f50e41486b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 shutdown_connections 2023-10-30T13:35:39.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:39.725+0000 7f50e9b01640 1 -- 172.21.15.148:0/3753580362 wait complete. 2023-10-30T13:35:39.741 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:35:41.742 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:41.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f347dba1640 1 Processor -- start 2023-10-30T13:35:41.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f347dba1640 1 -- start start 2023-10-30T13:35:41.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f347dba1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34780bfec0 0x7f34780c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:41.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f347dba1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f347806a720 0x7f347806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:41.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f347dba1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3478071290 con 0x7f3478159e60 2023-10-30T13:35:41.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f347dba1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3478071570 con 0x7f34780bfec0 2023-10-30T13:35:41.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f347dba1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3478071850 con 0x7f347806a720 2023-10-30T13:35:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f3476ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34780bfec0 0x7f34780c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f3476ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34780bfec0 0x7f34780c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:50002/0 (socket says 172.21.15.148:50002) 2023-10-30T13:35:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f3476ffd640 1 -- 172.21.15.148:0/4075156331 learned_addr learned my addr 172.21.15.148:0/4075156331 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.897+0000 7f3477fff640 1 --2- 172.21.15.148:0/4075156331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f347806a720 0x7f347806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:41.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.901+0000 7f3476ffd640 1 -- 172.21.15.148:0/4075156331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f347806a720 msgr2=0x7f347806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:41.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.901+0000 7f3476ffd640 1 --2- 172.21.15.148:0/4075156331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f347806a720 0x7f347806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:41.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.901+0000 7f3476ffd640 1 -- 172.21.15.148:0/4075156331 >> v1:172.21.15.150:6789/0 conn(0x7f3478159e60 legacy=0x7f347806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:41.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.901+0000 7f3476ffd640 1 -- 172.21.15.148:0/4075156331 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478071b30 con 0x7f34780bfec0 2023-10-30T13:35:41.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.901+0000 7f3476ffd640 1 --2- 172.21.15.148:0/4075156331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34780bfec0 0x7f34780c1a40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f346c00bb10 tx=0x7f346c021400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b30fc93ed7008e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:41.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.901+0000 7f34767fc640 1 -- 172.21.15.148:0/4075156331 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c0b1600 con 0x7f34780bfec0 2023-10-30T13:35:41.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.901+0000 7f34767fc640 1 -- 172.21.15.148:0/4075156331 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c0ad050 con 0x7f34780bfec0 2023-10-30T13:35:41.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.901+0000 7f34767fc640 1 -- 172.21.15.148:0/4075156331 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c0b1980 con 0x7f34780bfec0 2023-10-30T13:35:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 -- 172.21.15.148:0/4075156331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34780bfec0 msgr2=0x7f34780c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 --2- 172.21.15.148:0/4075156331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34780bfec0 0x7f34780c1a40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f346c00bb10 tx=0x7f346c021400 comp rx=0 tx=0).stop 2023-10-30T13:35:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 -- 172.21.15.148:0/4075156331 shutdown_connections 2023-10-30T13:35:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 --2- 172.21.15.148:0/4075156331 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f347806a720 0x7f347806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:41.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 --2- 172.21.15.148:0/4075156331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34780bfec0 0x7f34780c1a40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 -- 172.21.15.148:0/4075156331 >> 172.21.15.148:0/4075156331 conn(0x7f34780c7910 msgr2=0x7f34780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 -- 172.21.15.148:0/4075156331 shutdown_connections 2023-10-30T13:35:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 -- 172.21.15.148:0/4075156331 wait complete. 2023-10-30T13:35:41.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 Processor -- start 2023-10-30T13:35:41.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 -- start start 2023-10-30T13:35:41.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f347806a720 0x7f34780c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:41.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34780bfec0 0x7f34780c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:41.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3478159e60 0x7f34781f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:41.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c1940 con 0x7f347806a720 2023-10-30T13:35:41.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c1be0 con 0x7f34780bfec0 2023-10-30T13:35:41.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f347dba1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34780c1ec0 con 0x7f3478159e60 2023-10-30T13:35:41.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f34777fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f347806a720 0x7f34780c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:41.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f34777fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f347806a720 0x7f34780c40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56128/0 (socket says 172.21.15.148:56128) 2023-10-30T13:35:41.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f34777fe640 1 -- 172.21.15.148:0/3130952102 learned_addr learned my addr 172.21.15.148:0/3130952102 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:41.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f3476ffd640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34780bfec0 0x7f34780c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:41.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f3477fff640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3478159e60 0x7f34781f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:41.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f34777fe640 1 -- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3478159e60 msgr2=0x7f34781f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:41.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.905+0000 7f34777fe640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3478159e60 0x7f34781f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:41.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f34777fe640 1 -- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34780bfec0 msgr2=0x7f34780c45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:41.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f34777fe640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34780bfec0 0x7f34780c45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:41.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f34777fe640 1 -- 172.21.15.148:0/3130952102 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c00b770 con 0x7f347806a720 2023-10-30T13:35:41.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f34777fe640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f347806a720 0x7f34780c40a0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f346800c3c0 tx=0x7f346800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:41.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f345ffff640 1 -- 172.21.15.148:0/3130952102 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34680052d0 con 0x7f347806a720 2023-10-30T13:35:41.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f345ffff640 1 -- 172.21.15.148:0/3130952102 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3468005430 con 0x7f347806a720 2023-10-30T13:35:41.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34780c21a0 con 0x7f347806a720 2023-10-30T13:35:41.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f345ffff640 1 -- 172.21.15.148:0/3130952102 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468003840 con 0x7f347806a720 2023-10-30T13:35:41.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34780c2760 con 0x7f347806a720 2023-10-30T13:35:41.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347804e170 con 0x7f347806a720 2023-10-30T13:35:41.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f345ffff640 1 -- 172.21.15.148:0/3130952102 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34680039a0 con 0x7f347806a720 2023-10-30T13:35:41.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f345ffff640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34640664a0 0x7f3464068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:41.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f345ffff640 1 -- 172.21.15.148:0/3130952102 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f3468083970 con 0x7f347806a720 2023-10-30T13:35:41.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.909+0000 7f3476ffd640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34640664a0 0x7f3464068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:41.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.913+0000 7f3476ffd640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34640664a0 0x7f3464068960 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f346c0b3000 tx=0x7f346c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:41.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:41.913+0000 7f345ffff640 1 -- 172.21.15.148:0/3130952102 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3468050820 con 0x7f347806a720 2023-10-30T13:35:42.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3478157830 con 0x7f347806a720 2023-10-30T13:35:42.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f345ffff640 1 -- 172.21.15.148:0/3130952102 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7f3468017090 con 0x7f347806a720 2023-10-30T13:35:42.092 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:42.092 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:42.093 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:35:42.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34640664a0 msgr2=0x7f3464068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:42.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34640664a0 0x7f3464068960 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f346c0b3000 tx=0x7f346c008040 comp rx=0 tx=0).stop 2023-10-30T13:35:42.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f347806a720 msgr2=0x7f34780c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:42.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f347806a720 0x7f34780c40a0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f346800c3c0 tx=0x7f346800c880 comp rx=0 tx=0).stop 2023-10-30T13:35:42.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 shutdown_connections 2023-10-30T13:35:42.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f34640664a0 0x7f3464068960 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:42.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3478159e60 0x7f34781f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:42.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34780bfec0 0x7f34780c45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:42.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 --2- 172.21.15.148:0/3130952102 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f347806a720 0x7f34780c40a0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:42.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 >> 172.21.15.148:0/3130952102 conn(0x7f34780c7910 msgr2=0x7f34780ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:42.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 shutdown_connections 2023-10-30T13:35:42.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:42.097+0000 7f347dba1640 1 -- 172.21.15.148:0/3130952102 wait complete. 2023-10-30T13:35:42.113 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:35:44.114 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f6574d0f640 1 Processor -- start 2023-10-30T13:35:44.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f6574d0f640 1 -- start start 2023-10-30T13:35:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f6574d0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 0x7f657015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f6574d0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f657006a270 0x7f65700c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f6574d0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65700c6c10 con 0x7f657006cb80 2023-10-30T13:35:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f6574d0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65700c8570 con 0x7f657006a270 2023-10-30T13:35:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f6574d0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65700c6ef0 con 0x7f6570069e90 2023-10-30T13:35:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f656dd9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f657006a270 0x7f65700c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f656dd9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f657006a270 0x7f65700c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:50022/0 (socket says 172.21.15.148:50022) 2023-10-30T13:35:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f656dd9b640 1 -- 172.21.15.148:0/3309106604 learned_addr learned my addr 172.21.15.148:0/3309106604 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f656e59c640 1 --2- 172.21.15.148:0/3309106604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 0x7f657015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f656dd9b640 1 -- 172.21.15.148:0/3309106604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 msgr2=0x7f657015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:44.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f656dd9b640 1 --2- 172.21.15.148:0/3309106604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 0x7f657015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f656dd9b640 1 -- 172.21.15.148:0/3309106604 >> v1:172.21.15.150:6789/0 conn(0x7f657006cb80 legacy=0x7f657006d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:44.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.269+0000 7f656dd9b640 1 -- 172.21.15.148:0/3309106604 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65700c71d0 con 0x7f657006a270 2023-10-30T13:35:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f656dd9b640 1 --2- 172.21.15.148:0/3309106604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f657006a270 0x7f65700c9cb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6564009870 tx=0x7f656401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51d1514ec31e485c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f656d59a640 1 -- 172.21.15.148:0/3309106604 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65640a7070 con 0x7f657006a270 2023-10-30T13:35:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f656d59a640 1 -- 172.21.15.148:0/3309106604 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65640ab430 con 0x7f657006a270 2023-10-30T13:35:44.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f656d59a640 1 -- 172.21.15.148:0/3309106604 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65640ab7c0 con 0x7f657006a270 2023-10-30T13:35:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f6574d0f640 1 -- 172.21.15.148:0/3309106604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f657006a270 msgr2=0x7f65700c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f6574d0f640 1 --2- 172.21.15.148:0/3309106604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f657006a270 0x7f65700c9cb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6564009870 tx=0x7f656401c540 comp rx=0 tx=0).stop 2023-10-30T13:35:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f6574d0f640 1 -- 172.21.15.148:0/3309106604 shutdown_connections 2023-10-30T13:35:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f6574d0f640 1 --2- 172.21.15.148:0/3309106604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f657006a270 0x7f65700c9cb0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f6574d0f640 1 --2- 172.21.15.148:0/3309106604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 0x7f657015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f6574d0f640 1 -- 172.21.15.148:0/3309106604 >> 172.21.15.148:0/3309106604 conn(0x7f657006dfd0 msgr2=0x7f657006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f6574d0f640 1 -- 172.21.15.148:0/3309106604 shutdown_connections 2023-10-30T13:35:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.273+0000 7f6574d0f640 1 -- 172.21.15.148:0/3309106604 wait complete. 2023-10-30T13:35:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 Processor -- start 2023-10-30T13:35:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 -- start start 2023-10-30T13:35:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 0x7f6570150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f657006a270 0x7f65701f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657006cb80 0x7f65701556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6570072310 con 0x7f657006cb80 2023-10-30T13:35:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657014a4d0 con 0x7f657006a270 2023-10-30T13:35:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65701f2680 con 0x7f6570069e90 2023-10-30T13:35:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657006cb80 0x7f65701556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657006cb80 0x7f65701556a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56130/0 (socket says 172.21.15.148:56130) 2023-10-30T13:35:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656e59c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 0x7f6570150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656ed9d640 1 -- 172.21.15.148:0/1440340788 learned_addr learned my addr 172.21.15.148:0/1440340788 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656dd9b640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f657006a270 0x7f65701f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656ed9d640 1 -- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 msgr2=0x7f6570150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656ed9d640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 0x7f6570150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656ed9d640 1 -- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f657006a270 msgr2=0x7f65701f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656ed9d640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f657006a270 0x7f65701f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656ed9d640 1 -- 172.21.15.148:0/1440340788 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65640094d0 con 0x7f657006cb80 2023-10-30T13:35:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f656ed9d640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657006cb80 0x7f65701556a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f656800c730 tx=0x7f656800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f655effd640 1 -- 172.21.15.148:0/1440340788 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568003b80 con 0x7f657006cb80 2023-10-30T13:35:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f655effd640 1 -- 172.21.15.148:0/1440340788 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6568003ce0 con 0x7f657006cb80 2023-10-30T13:35:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f655effd640 1 -- 172.21.15.148:0/1440340788 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656800aa30 con 0x7f657006cb80 2023-10-30T13:35:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6570155be0 con 0x7f657006cb80 2023-10-30T13:35:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.277+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657014bf70 con 0x7f657006cb80 2023-10-30T13:35:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.281+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657006c640 con 0x7f657006cb80 2023-10-30T13:35:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.281+0000 7f655effd640 1 -- 172.21.15.148:0/1440340788 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f656800ab90 con 0x7f657006cb80 2023-10-30T13:35:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.281+0000 7f655effd640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65500664a0 0x7f6550068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.281+0000 7f655effd640 1 -- 172.21.15.148:0/1440340788 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6886+0+0 (secure 0 0 0) 0x7f6568083f90 con 0x7f657006cb80 2023-10-30T13:35:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.281+0000 7f656e59c640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65500664a0 0x7f6550068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.281+0000 7f656e59c640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65500664a0 0x7f6550068960 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6560009b60 tx=0x7f6560009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.285+0000 7f655effd640 1 -- 172.21.15.148:0/1440340788 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f657006c640 con 0x7f657006cb80 2023-10-30T13:35:44.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.465+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f657006c640 con 0x7f657006cb80 2023-10-30T13:35:44.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.465+0000 7f655effd640 1 -- 172.21.15.148:0/1440340788 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6514 (secure 0 0 0) 0x7f657006c640 con 0x7f657006cb80 2023-10-30T13:35:44.462 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:44.462 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":6870,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:34:16.997554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":99,"state":"up:active","state_seq":27,"addr":"172.21.15.148:6835/1386276781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1386276781},{"type":"v1","addr":"172.21.15.148:6835","nonce":1386276781}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:44.463 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 103 2023-10-30T13:35:44.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65500664a0 msgr2=0x7f6550068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:44.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65500664a0 0x7f6550068960 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6560009b60 tx=0x7f6560009190 comp rx=0 tx=0).stop 2023-10-30T13:35:44.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657006cb80 msgr2=0x7f65701556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:44.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657006cb80 0x7f65701556a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f656800c730 tx=0x7f656800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 shutdown_connections 2023-10-30T13:35:44.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65500664a0 0x7f6550068960 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f657006cb80 0x7f65701556a0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f657006a270 0x7f65701f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 --2- 172.21.15.148:0/1440340788 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6570069e90 0x7f6570150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:44.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 >> 172.21.15.148:0/1440340788 conn(0x7f657006dfd0 msgr2=0x7f657006afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:44.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 shutdown_connections 2023-10-30T13:35:44.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:44.469+0000 7f6574d0f640 1 -- 172.21.15.148:0/1440340788 wait complete. 2023-10-30T13:35:44.481 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:35:46.483 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:46.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.625+0000 7f25218c9640 1 Processor -- start 2023-10-30T13:35:46.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.625+0000 7f25218c9640 1 -- start start 2023-10-30T13:35:46.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25218c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f251c1505b0 0x7f251c0c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25218c9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f251c0c3140 0x7f251c06cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25218c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c06d0e0 0x7f251c1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25218c9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251c0c2f30 con 0x7f251c0c3140 2023-10-30T13:35:46.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25218c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251c0c6a10 con 0x7f251c1505b0 2023-10-30T13:35:46.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25218c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251c150990 con 0x7f251c06d0e0 2023-10-30T13:35:46.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25127fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f251c0c3140 0x7f251c06cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25127fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f251c0c3140 0x7f251c06cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56134/0 (socket says 172.21.15.148:56134) 2023-10-30T13:35:46.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25127fc640 1 -- 172.21.15.148:0/1114587861 learned_addr learned my addr 172.21.15.148:0/1114587861 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:46.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f251affd640 1 --2- 172.21.15.148:0/1114587861 >> v2:172.21.15.150:3300/0 conn(0x7f251c1505b0 0x7f251c0c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f251b7fe640 1 --2- 172.21.15.148:0/1114587861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c06d0e0 0x7f251c1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25127fc640 1 -- 172.21.15.148:0/1114587861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c06d0e0 msgr2=0x7f251c1568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:46.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25127fc640 1 --2- 172.21.15.148:0/1114587861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c06d0e0 0x7f251c1568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25127fc640 1 -- 172.21.15.148:0/1114587861 >> v2:172.21.15.150:3300/0 conn(0x7f251c1505b0 msgr2=0x7f251c0c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:46.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25127fc640 1 --2- 172.21.15.148:0/1114587861 >> v2:172.21.15.150:3300/0 conn(0x7f251c1505b0 0x7f251c0c29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f25127fc640 1 -- 172.21.15.148:0/1114587861 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c0c7220 con 0x7f251c0c3140 2023-10-30T13:35:46.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.629+0000 7f251b7fe640 1 --2- 172.21.15.148:0/1114587861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c06d0e0 0x7f251c1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:46.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25127fc640 1 --2- 172.21.15.148:0/1114587861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c0c3140 0x7f251c06cba0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f251c06df40 tx=0x7f250401ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21feb3d729a7a76 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:46.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f251a7fc640 1 -- 172.21.15.148:0/1114587861 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25040a7980 con 0x7f251c0c3140 2023-10-30T13:35:46.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f251a7fc640 1 -- 172.21.15.148:0/1114587861 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25040a7ae0 con 0x7f251c0c3140 2023-10-30T13:35:46.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f251a7fc640 1 -- 172.21.15.148:0/1114587861 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25040a7e70 con 0x7f251c0c3140 2023-10-30T13:35:46.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 -- 172.21.15.148:0/1114587861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c0c3140 msgr2=0x7f251c06cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:46.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 --2- 172.21.15.148:0/1114587861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c0c3140 0x7f251c06cba0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f251c06df40 tx=0x7f250401ebe0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 -- 172.21.15.148:0/1114587861 shutdown_connections 2023-10-30T13:35:46.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 --2- 172.21.15.148:0/1114587861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c06d0e0 0x7f251c1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 --2- 172.21.15.148:0/1114587861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c0c3140 0x7f251c06cba0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 --2- 172.21.15.148:0/1114587861 >> v2:172.21.15.150:3300/0 conn(0x7f251c1505b0 0x7f251c0c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 -- 172.21.15.148:0/1114587861 >> 172.21.15.148:0/1114587861 conn(0x7f251c0cac50 msgr2=0x7f251c0cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:46.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 -- 172.21.15.148:0/1114587861 shutdown_connections 2023-10-30T13:35:46.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 -- 172.21.15.148:0/1114587861 wait complete. 2023-10-30T13:35:46.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 Processor -- start 2023-10-30T13:35:46.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.633+0000 7f25218c9640 1 -- start start 2023-10-30T13:35:46.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25218c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f251c06d0e0 0x7f251c158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25218c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c0c3140 0x7f251c071830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25218c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c1505b0 0x7f251c1f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25218c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c159b90 con 0x7f251c1505b0 2023-10-30T13:35:46.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25218c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c157420 con 0x7f251c06d0e0 2023-10-30T13:35:46.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25218c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251c1576e0 con 0x7f251c0c3140 2023-10-30T13:35:46.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c1505b0 0x7f251c1f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c1505b0 0x7f251c1f4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56138/0 (socket says 172.21.15.148:56138) 2023-10-30T13:35:46.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251b7fe640 1 -- 172.21.15.148:0/3347415798 learned_addr learned my addr 172.21.15.148:0/3347415798 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:46.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251affd640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f251c06d0e0 0x7f251c158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25127fc640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c0c3140 0x7f251c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251b7fe640 1 -- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c0c3140 msgr2=0x7f251c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:46.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251b7fe640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c0c3140 0x7f251c071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251b7fe640 1 -- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f251c06d0e0 msgr2=0x7f251c158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:46.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251b7fe640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f251c06d0e0 0x7f251c158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251b7fe640 1 -- 172.21.15.148:0/3347415798 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504009b30 con 0x7f251c1505b0 2023-10-30T13:35:46.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251affd640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f251c06d0e0 0x7f251c158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:46.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f251b7fe640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c1505b0 0x7f251c1f4fb0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f251400d5d0 tx=0x7f251400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:46.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f2518ff9640 1 -- 172.21.15.148:0/3347415798 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514004030 con 0x7f251c1505b0 2023-10-30T13:35:46.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f2518ff9640 1 -- 172.21.15.148:0/3347415798 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2514004190 con 0x7f251c1505b0 2023-10-30T13:35:46.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251c1590b0 con 0x7f251c1505b0 2023-10-30T13:35:46.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f2518ff9640 1 -- 172.21.15.148:0/3347415798 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514003620 con 0x7f251c1505b0 2023-10-30T13:35:46.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251c159670 con 0x7f251c1505b0 2023-10-30T13:35:46.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251c158ef0 con 0x7f251c1505b0 2023-10-30T13:35:46.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.637+0000 7f2518ff9640 1 -- 172.21.15.148:0/3347415798 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2514003780 con 0x7f251c1505b0 2023-10-30T13:35:46.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.641+0000 7f2518ff9640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24fc0664a0 0x7f24fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.641+0000 7f2518ff9640 1 -- 172.21.15.148:0/3347415798 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f2514084990 con 0x7f251c1505b0 2023-10-30T13:35:46.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.641+0000 7f251affd640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24fc0664a0 0x7f24fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.641+0000 7f251affd640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24fc0664a0 0x7f24fc068960 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f250c009bb0 tx=0x7f250c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:46.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.645+0000 7f2518ff9640 1 -- 172.21.15.148:0/3347415798 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2514051910 con 0x7f251c1505b0 2023-10-30T13:35:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.821+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f251c159860 con 0x7f251c1505b0 2023-10-30T13:35:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.821+0000 7f2518ff9640 1 -- 172.21.15.148:0/3347415798 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f2514054f30 con 0x7f251c1505b0 2023-10-30T13:35:46.818 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:46.819 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:35:44.780053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 104 2023-10-30T13:35:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24fc0664a0 msgr2=0x7f24fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24fc0664a0 0x7f24fc068960 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f250c009bb0 tx=0x7f250c008040 comp rx=0 tx=0).stop 2023-10-30T13:35:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c1505b0 msgr2=0x7f251c1f4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c1505b0 0x7f251c1f4fb0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f251400d5d0 tx=0x7f251400da90 comp rx=0 tx=0).stop 2023-10-30T13:35:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 shutdown_connections 2023-10-30T13:35:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24fc0664a0 0x7f24fc068960 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f251c1505b0 0x7f251c1f4fb0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f251c0c3140 0x7f251c071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 --2- 172.21.15.148:0/3347415798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f251c06d0e0 0x7f251c158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 >> 172.21.15.148:0/3347415798 conn(0x7f251c0cac50 msgr2=0x7f251c14de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 shutdown_connections 2023-10-30T13:35:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.825+0000 7f25218c9640 1 -- 172.21.15.148:0/3347415798 wait complete. 2023-10-30T13:35:46.839 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:35:46.839 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:35:46.840 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:46.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.985+0000 7f26bf017640 1 Processor -- start 2023-10-30T13:35:46.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.985+0000 7f26bf017640 1 -- start start 2023-10-30T13:35:46.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26bf017640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f26b806f7a0 0x7f26b806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26bf017640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f26b80c6bd0 0x7f26b8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26bf017640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b806a3d0 con 0x7f26b8071110 2023-10-30T13:35:46.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26bf017640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b80700c0 con 0x7f26b80c6bd0 2023-10-30T13:35:46.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26bf017640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b8066cf0 con 0x7f26b806f7a0 2023-10-30T13:35:46.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26affff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f26b80c6bd0 0x7f26b8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26affff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f26b80c6bd0 0x7f26b8069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56150/0 (socket says 172.21.15.148:56150) 2023-10-30T13:35:46.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26affff640 1 -- 172.21.15.148:0/2168801657 learned_addr learned my addr 172.21.15.148:0/2168801657 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:46.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26bd5b4640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f26b8071110 0x7f26b8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59000/0 (socket says 172.21.15.148:59000) 2023-10-30T13:35:46.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26bcdb3640 1 --2- 172.21.15.148:0/2168801657 >> v2:172.21.15.150:3300/0 conn(0x7f26b806f7a0 0x7f26b806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26affff640 1 -- 172.21.15.148:0/2168801657 >> v2:172.21.15.150:3300/0 conn(0x7f26b806f7a0 msgr2=0x7f26b806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:46.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26affff640 1 --2- 172.21.15.148:0/2168801657 >> v2:172.21.15.150:3300/0 conn(0x7f26b806f7a0 0x7f26b806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26affff640 1 -- 172.21.15.148:0/2168801657 >> v1:172.21.15.148:6789/0 conn(0x7f26b8071110 legacy=0x7f26b8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:46.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26af7fe640 1 -- 172.21.15.148:0/2168801657 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844259107 0 0) 0x7f26b806a3d0 con 0x7f26b8071110 2023-10-30T13:35:46.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.989+0000 7f26affff640 1 -- 172.21.15.148:0/2168801657 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b80c70d0 con 0x7f26b80c6bd0 2023-10-30T13:35:46.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26affff640 1 --2- 172.21.15.148:0/2168801657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b80c6bd0 0x7f26b8069e90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f26b806c830 tx=0x7f26a801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e037493f1a8d5d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:46.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26af7fe640 1 -- 172.21.15.148:0/2168801657 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a80a89f0 con 0x7f26b80c6bd0 2023-10-30T13:35:46.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26af7fe640 1 -- 172.21.15.148:0/2168801657 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a80a8b50 con 0x7f26b80c6bd0 2023-10-30T13:35:46.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26af7fe640 1 -- 172.21.15.148:0/2168801657 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a80b0440 con 0x7f26b80c6bd0 2023-10-30T13:35:46.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 -- 172.21.15.148:0/2168801657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b80c6bd0 msgr2=0x7f26b8069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:46.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 --2- 172.21.15.148:0/2168801657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b80c6bd0 0x7f26b8069e90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f26b806c830 tx=0x7f26a801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 -- 172.21.15.148:0/2168801657 shutdown_connections 2023-10-30T13:35:46.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 --2- 172.21.15.148:0/2168801657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b80c6bd0 0x7f26b8069e90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 --2- 172.21.15.148:0/2168801657 >> v2:172.21.15.150:3300/0 conn(0x7f26b806f7a0 0x7f26b806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 -- 172.21.15.148:0/2168801657 >> 172.21.15.148:0/2168801657 conn(0x7f26b80c7910 msgr2=0x7f26b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:46.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 -- 172.21.15.148:0/2168801657 shutdown_connections 2023-10-30T13:35:46.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 -- 172.21.15.148:0/2168801657 wait complete. 2023-10-30T13:35:46.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 Processor -- start 2023-10-30T13:35:46.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 -- start start 2023-10-30T13:35:46.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b806f7a0 0x7f26b8157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26b8071110 0x7f26b80c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26b80c6bd0 0x7f26b80ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b81efed0 con 0x7f26b806f7a0 2023-10-30T13:35:46.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b80c37c0 con 0x7f26b8071110 2023-10-30T13:35:46.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.993+0000 7f26bf017640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8155550 con 0x7f26b80c6bd0 2023-10-30T13:35:46.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bcdb3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b806f7a0 0x7f26b8157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bcdb3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b806f7a0 0x7f26b8157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56160/0 (socket says 172.21.15.148:56160) 2023-10-30T13:35:46.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bcdb3640 1 -- 172.21.15.148:0/1031924119 learned_addr learned my addr 172.21.15.148:0/1031924119 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:46.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bd5b4640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26b80c6bd0 0x7f26b80ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26affff640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26b8071110 0x7f26b80c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bcdb3640 1 -- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26b80c6bd0 msgr2=0x7f26b80ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:46.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bcdb3640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26b80c6bd0 0x7f26b80ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bcdb3640 1 -- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26b8071110 msgr2=0x7f26b80c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:46.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bcdb3640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26b8071110 0x7f26b80c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:46.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bcdb3640 1 -- 172.21.15.148:0/1031924119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a80094d0 con 0x7f26b806f7a0 2023-10-30T13:35:46.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bcdb3640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b806f7a0 0x7f26b8157c70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f26b400d5f0 tx=0x7f26b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:46.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26ad7fa640 1 -- 172.21.15.148:0/1031924119 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b40125f0 con 0x7f26b806f7a0 2023-10-30T13:35:46.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26ad7fa640 1 -- 172.21.15.148:0/1031924119 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b4012750 con 0x7f26b806f7a0 2023-10-30T13:35:46.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b80c39f0 con 0x7f26b806f7a0 2023-10-30T13:35:46.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26ad7fa640 1 -- 172.21.15.148:0/1031924119 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b4012ae0 con 0x7f26b806f7a0 2023-10-30T13:35:46.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b80ada70 con 0x7f26b806f7a0 2023-10-30T13:35:46.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b804e170 con 0x7f26b806f7a0 2023-10-30T13:35:46.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26ad7fa640 1 -- 172.21.15.148:0/1031924119 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f26b4012c40 con 0x7f26b806f7a0 2023-10-30T13:35:46.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26ad7fa640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26a40664d0 0x7f26a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:46.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:46.997+0000 7f26ad7fa640 1 -- 172.21.15.148:0/1031924119 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f26b4084870 con 0x7f26b806f7a0 2023-10-30T13:35:46.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.001+0000 7f26affff640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26a40664d0 0x7f26a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:46.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.001+0000 7f26affff640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26a40664d0 0x7f26a4068990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f26a80b2000 tx=0x7f26a80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:46.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.001+0000 7f26ad7fa640 1 -- 172.21.15.148:0/1031924119 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f26b4051760 con 0x7f26b806f7a0 2023-10-30T13:35:47.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.189+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26b8070600 con 0x7f26b806f7a0 2023-10-30T13:35:47.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.189+0000 7f26ad7fa640 1 -- 172.21.15.148:0/1031924119 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f26b4007380 con 0x7f26b806f7a0 2023-10-30T13:35:47.185 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:47.186 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:35:44.780053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:47.186 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 104 2023-10-30T13:35:47.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26a40664d0 msgr2=0x7f26a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:47.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26a40664d0 0x7f26a4068990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f26a80b2000 tx=0x7f26a80aa070 comp rx=0 tx=0).stop 2023-10-30T13:35:47.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b806f7a0 msgr2=0x7f26b8157c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:47.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b806f7a0 0x7f26b8157c70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f26b400d5f0 tx=0x7f26b400dab0 comp rx=0 tx=0).stop 2023-10-30T13:35:47.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 shutdown_connections 2023-10-30T13:35:47.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26a40664d0 0x7f26a4068990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:47.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f26b80c6bd0 0x7f26b80ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:47.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26b8071110 0x7f26b80c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:47.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 --2- 172.21.15.148:0/1031924119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26b806f7a0 0x7f26b8157c70 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:47.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 >> 172.21.15.148:0/1031924119 conn(0x7f26b80c7910 msgr2=0x7f26b80ca250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:47.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 shutdown_connections 2023-10-30T13:35:47.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:47.193+0000 7f26bf017640 1 -- 172.21.15.148:0/1031924119 wait complete. 2023-10-30T13:35:47.207 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 104, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 104, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6870": { "addr": "172.21.15.148:6837/3144126860", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3144126860, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3144126860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6870, "incarnation": 104, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:35:44.780053+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6870 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:35:49.209 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:49.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.361+0000 7f939e49e640 1 Processor -- start 2023-10-30T13:35:49.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.361+0000 7f939e49e640 1 -- start start 2023-10-30T13:35:49.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f939e49e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9398153bf0 0x7f9398071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:49.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f939e49e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93980700b0 con 0x7f93980c5f60 2023-10-30T13:35:49.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f939e49e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93980c6530 con 0x7f93980c6fe0 2023-10-30T13:35:49.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f939e49e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93980c6810 con 0x7f9398153bf0 2023-10-30T13:35:49.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f9397fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9398153bf0 0x7f9398071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:49.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f9397fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9398153bf0 0x7f9398071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56176/0 (socket says 172.21.15.148:56176) 2023-10-30T13:35:49.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f9397fff640 1 -- 172.21.15.148:0/3643021737 learned_addr learned my addr 172.21.15.148:0/3643021737 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:49.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f939ca3b640 1 --1- 172.21.15.148:0/3643021737 >> v1:172.21.15.148:6789/0 conn(0x7f93980c5f60 0x7f93980cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59016/0 (socket says 172.21.15.148:59016) 2023-10-30T13:35:49.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f9397fff640 1 -- 172.21.15.148:0/3643021737 >> v1:172.21.15.150:6789/0 conn(0x7f93980c6fe0 legacy=0x7f93980c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:49.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f9397fff640 1 -- 172.21.15.148:0/3643021737 >> v1:172.21.15.148:6789/0 conn(0x7f93980c5f60 legacy=0x7f93980cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:49.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f9397fff640 1 -- 172.21.15.148:0/3643021737 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398159b10 con 0x7f9398153bf0 2023-10-30T13:35:49.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.365+0000 7f9396ffd640 1 -- 172.21.15.148:0/3643021737 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842062210 0 0) 0x7f93980700b0 con 0x7f93980c5f60 2023-10-30T13:35:49.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f9397fff640 1 --2- 172.21.15.148:0/3643021737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 0x7f9398071260 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f9388009870 tx=0x7f938801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc46103e46b7211 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:49.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f9396ffd640 1 -- 172.21.15.148:0/3643021737 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93880a8a30 con 0x7f9398153bf0 2023-10-30T13:35:49.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f9396ffd640 1 -- 172.21.15.148:0/3643021737 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93880a8b90 con 0x7f9398153bf0 2023-10-30T13:35:49.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f9396ffd640 1 -- 172.21.15.148:0/3643021737 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93880b0440 con 0x7f9398153bf0 2023-10-30T13:35:49.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f939e49e640 1 -- 172.21.15.148:0/3643021737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 msgr2=0x7f9398071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:49.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f939e49e640 1 --2- 172.21.15.148:0/3643021737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 0x7f9398071260 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f9388009870 tx=0x7f938801ed30 comp rx=0 tx=0).stop 2023-10-30T13:35:49.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f939e49e640 1 -- 172.21.15.148:0/3643021737 shutdown_connections 2023-10-30T13:35:49.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f939e49e640 1 --2- 172.21.15.148:0/3643021737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 0x7f9398071260 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:49.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f939e49e640 1 -- 172.21.15.148:0/3643021737 >> 172.21.15.148:0/3643021737 conn(0x7f939806c6e0 msgr2=0x7f9398069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:49.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f939e49e640 1 -- 172.21.15.148:0/3643021737 shutdown_connections 2023-10-30T13:35:49.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f939e49e640 1 -- 172.21.15.148:0/3643021737 wait complete. 2023-10-30T13:35:49.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f939e49e640 1 Processor -- start 2023-10-30T13:35:49.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.369+0000 7f939e49e640 1 -- start start 2023-10-30T13:35:49.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939e49e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93980c5f60 0x7f9398156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:49.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939e49e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93980c6fe0 0x7f9398158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:49.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939e49e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 0x7f939814c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:49.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939e49e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93981576e0 con 0x7f9398153bf0 2023-10-30T13:35:49.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939e49e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93981579c0 con 0x7f93980c6fe0 2023-10-30T13:35:49.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939e49e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9398157ca0 con 0x7f93980c5f60 2023-10-30T13:35:49.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939ca3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 0x7f939814c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:49.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939ca3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 0x7f939814c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56178/0 (socket says 172.21.15.148:56178) 2023-10-30T13:35:49.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939ca3b640 1 -- 172.21.15.148:0/2276670576 learned_addr learned my addr 172.21.15.148:0/2276670576 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f9397fff640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93980c5f60 0x7f9398156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f93977fe640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93980c6fe0 0x7f9398158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939ca3b640 1 -- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93980c5f60 msgr2=0x7f9398156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939ca3b640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93980c5f60 0x7f9398156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:49.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939ca3b640 1 -- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93980c6fe0 msgr2=0x7f9398158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939ca3b640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93980c6fe0 0x7f9398158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939ca3b640 1 -- 172.21.15.148:0/2276670576 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93880094d0 con 0x7f9398153bf0 2023-10-30T13:35:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f93977fe640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93980c6fe0 0x7f9398158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:35:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939ca3b640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 0x7f939814c2d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f9390001570 tx=0x7f939000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f9394ff9640 1 -- 172.21.15.148:0/2276670576 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939000a640 con 0x7f9398153bf0 2023-10-30T13:35:49.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f9394ff9640 1 -- 172.21.15.148:0/2276670576 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939000a7c0 con 0x7f9398153bf0 2023-10-30T13:35:49.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9398157f80 con 0x7f9398153bf0 2023-10-30T13:35:49.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f9394ff9640 1 -- 172.21.15.148:0/2276670576 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939000ab70 con 0x7f9398153bf0 2023-10-30T13:35:49.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939814fde0 con 0x7f9398153bf0 2023-10-30T13:35:49.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93981584a0 con 0x7f9398153bf0 2023-10-30T13:35:49.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.373+0000 7f9394ff9640 1 -- 172.21.15.148:0/2276670576 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f939000acd0 con 0x7f9398153bf0 2023-10-30T13:35:49.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.377+0000 7f9394ff9640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93840664a0 0x7f9384068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:49.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.377+0000 7f9394ff9640 1 -- 172.21.15.148:0/2276670576 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f9390083930 con 0x7f9398153bf0 2023-10-30T13:35:49.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.377+0000 7f9397fff640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93840664a0 0x7f9384068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:49.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.377+0000 7f9397fff640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93840664a0 0x7f9384068960 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f93880b2000 tx=0x7f9388009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:49.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.381+0000 7f9394ff9640 1 -- 172.21.15.148:0/2276670576 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93981584a0 con 0x7f9398153bf0 2023-10-30T13:35:49.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.561+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93981584a0 con 0x7f9398153bf0 2023-10-30T13:35:49.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.561+0000 7f9394ff9640 1 -- 172.21.15.148:0/2276670576 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f93981584a0 con 0x7f9398153bf0 2023-10-30T13:35:49.556 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:49.557 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:35:44.780053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:49.557 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 104 2023-10-30T13:35:49.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.561+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93840664a0 msgr2=0x7f9384068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:49.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.561+0000 7f939e49e640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93840664a0 0x7f9384068960 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f93880b2000 tx=0x7f9388009040 comp rx=0 tx=0).stop 2023-10-30T13:35:49.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.561+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 msgr2=0x7f939814c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:49.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.561+0000 7f939e49e640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 0x7f939814c2d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f9390001570 tx=0x7f939000cca0 comp rx=0 tx=0).stop 2023-10-30T13:35:49.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.565+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 shutdown_connections 2023-10-30T13:35:49.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.565+0000 7f939e49e640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93840664a0 0x7f9384068960 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:49.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.565+0000 7f939e49e640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9398153bf0 0x7f939814c2d0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:49.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.565+0000 7f939e49e640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93980c6fe0 0x7f9398158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:49.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.565+0000 7f939e49e640 1 --2- 172.21.15.148:0/2276670576 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93980c5f60 0x7f9398156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:49.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.565+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 >> 172.21.15.148:0/2276670576 conn(0x7f939806c6e0 msgr2=0x7f93980c7450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:49.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.565+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 shutdown_connections 2023-10-30T13:35:49.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:49.565+0000 7f939e49e640 1 -- 172.21.15.148:0/2276670576 wait complete. 2023-10-30T13:35:49.577 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:51.578 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:51.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.732+0000 7f70ab59e640 1 Processor -- start 2023-10-30T13:35:51.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.732+0000 7f70ab59e640 1 -- start start 2023-10-30T13:35:51.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.732+0000 7f70ab59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70ac1538f0 0x7f70ac159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:51.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.732+0000 7f70ab59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac0cefa0 0x7f70ac0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:51.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.732+0000 7f70ab59e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac0ca6a0 con 0x7f70ac15cd50 2023-10-30T13:35:51.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.732+0000 7f70ab59e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ac15d300 con 0x7f70ac1538f0 2023-10-30T13:35:51.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.732+0000 7f70ab59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac15d5e0 con 0x7f70ac0cefa0 2023-10-30T13:35:51.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.732+0000 7f70aa59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70ac1538f0 0x7f70ac159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:51.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.736+0000 7f70a3fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f70ac15cd50 0x7f70ac0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:37906/0 (socket says 172.21.15.148:37906) 2023-10-30T13:35:51.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.736+0000 7f70aad9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac0cefa0 0x7f70ac0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:51.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.736+0000 7f70a3fff640 1 -- 172.21.15.148:0/2599562665 learned_addr learned my addr 172.21.15.148:0/2599562665 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:35:51.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.736+0000 7f70aad9d640 1 -- 172.21.15.148:0/2599562665 >> v2:172.21.15.150:3300/0 conn(0x7f70ac1538f0 msgr2=0x7f70ac159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:51.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.736+0000 7f70aad9d640 1 --2- 172.21.15.148:0/2599562665 >> v2:172.21.15.150:3300/0 conn(0x7f70ac1538f0 0x7f70ac159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.736+0000 7f70aad9d640 1 -- 172.21.15.148:0/2599562665 >> v1:172.21.15.150:6789/0 conn(0x7f70ac15cd50 legacy=0x7f70ac0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:51.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.736+0000 7f70aad9d640 1 -- 172.21.15.148:0/2599562665 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac15d8c0 con 0x7f70ac0cefa0 2023-10-30T13:35:51.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.736+0000 7f70aa59c640 1 --2- 172.21.15.148:0/2599562665 >> v2:172.21.15.150:3300/0 conn(0x7f70ac1538f0 0x7f70ac159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:35:51.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70aad9d640 1 --2- 172.21.15.148:0/2599562665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac0cefa0 0x7f70ac0cf3d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009d90 tx=0x7f70a401f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dedd5391adc1534c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:51.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70a9d9b640 1 -- 172.21.15.148:0/2599562665 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a40af600 con 0x7f70ac0cefa0 2023-10-30T13:35:51.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70a9d9b640 1 -- 172.21.15.148:0/2599562665 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a40ab050 con 0x7f70ac0cefa0 2023-10-30T13:35:51.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70a9d9b640 1 -- 172.21.15.148:0/2599562665 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a40af980 con 0x7f70ac0cefa0 2023-10-30T13:35:51.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70ab59e640 1 -- 172.21.15.148:0/2599562665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac0cefa0 msgr2=0x7f70ac0cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:51.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70ab59e640 1 --2- 172.21.15.148:0/2599562665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac0cefa0 0x7f70ac0cf3d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009d90 tx=0x7f70a401f130 comp rx=0 tx=0).stop 2023-10-30T13:35:51.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70ab59e640 1 -- 172.21.15.148:0/2599562665 shutdown_connections 2023-10-30T13:35:51.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70ab59e640 1 --2- 172.21.15.148:0/2599562665 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac0cefa0 0x7f70ac0cf3d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70ab59e640 1 --2- 172.21.15.148:0/2599562665 >> v2:172.21.15.150:3300/0 conn(0x7f70ac1538f0 0x7f70ac159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70ab59e640 1 -- 172.21.15.148:0/2599562665 >> 172.21.15.148:0/2599562665 conn(0x7f70ac06a640 msgr2=0x7f70ac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:51.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70ab59e640 1 -- 172.21.15.148:0/2599562665 shutdown_connections 2023-10-30T13:35:51.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.740+0000 7f70ab59e640 1 -- 172.21.15.148:0/2599562665 wait complete. 2023-10-30T13:35:51.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70ab59e640 1 Processor -- start 2023-10-30T13:35:51.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70ab59e640 1 -- start start 2023-10-30T13:35:51.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70ab59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70ac0cefa0 0x7f70ac15b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:51.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70ab59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac1538f0 0x7f70ac152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:51.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70ab59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70ac15cd50 0x7f70ac152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:51.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70ab59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac1f0520 con 0x7f70ac15cd50 2023-10-30T13:35:51.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70ab59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac151ce0 con 0x7f70ac0cefa0 2023-10-30T13:35:51.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70ab59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac148980 con 0x7f70ac1538f0 2023-10-30T13:35:51.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70aad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70ac15cd50 0x7f70ac152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:51.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70aad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70ac15cd50 0x7f70ac152650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56180/0 (socket says 172.21.15.148:56180) 2023-10-30T13:35:51.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70aad9d640 1 -- 172.21.15.148:0/1593101483 learned_addr learned my addr 172.21.15.148:0/1593101483 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:51.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70a3fff640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac1538f0 0x7f70ac152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:51.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70aa59c640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70ac0cefa0 0x7f70ac15b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:51.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70aad9d640 1 -- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac1538f0 msgr2=0x7f70ac152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:51.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70aad9d640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac1538f0 0x7f70ac152110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70aad9d640 1 -- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70ac0cefa0 msgr2=0x7f70ac15b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:51.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70aad9d640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70ac0cefa0 0x7f70ac15b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.744+0000 7f70aad9d640 1 -- 172.21.15.148:0/1593101483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709c009420 con 0x7f70ac15cd50 2023-10-30T13:35:51.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.748+0000 7f70aad9d640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70ac15cd50 0x7f70ac152650 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009d90 tx=0x7f70a401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:51.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.748+0000 7f70a2ffd640 1 -- 172.21.15.148:0/1593101483 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4007000 con 0x7f70ac15cd50 2023-10-30T13:35:51.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.748+0000 7f70a2ffd640 1 -- 172.21.15.148:0/1593101483 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a4009450 con 0x7f70ac15cd50 2023-10-30T13:35:51.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.748+0000 7f70a2ffd640 1 -- 172.21.15.148:0/1593101483 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a40ba420 con 0x7f70ac15cd50 2023-10-30T13:35:51.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.748+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a4009a70 con 0x7f70ac15cd50 2023-10-30T13:35:51.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.748+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ac151a00 con 0x7f70ac15cd50 2023-10-30T13:35:51.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.748+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ac04e170 con 0x7f70ac15cd50 2023-10-30T13:35:51.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.748+0000 7f70a2ffd640 1 -- 172.21.15.148:0/1593101483 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70a40ba5a0 con 0x7f70ac15cd50 2023-10-30T13:35:51.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.752+0000 7f70a2ffd640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f708c0664a0 0x7f708c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:51.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.752+0000 7f70a2ffd640 1 -- 172.21.15.148:0/1593101483 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f70a4122b60 con 0x7f70ac15cd50 2023-10-30T13:35:51.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.752+0000 7f70aa59c640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f708c0664a0 0x7f708c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:51.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.752+0000 7f70aa59c640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f708c0664a0 0x7f708c068960 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f709c009cf0 tx=0x7f709c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:51.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.756+0000 7f70a2ffd640 1 -- 172.21.15.148:0/1593101483 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70a40efc20 con 0x7f70ac15cd50 2023-10-30T13:35:51.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.948+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70ac06c580 con 0x7f70ac15cd50 2023-10-30T13:35:51.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.948+0000 7f70a2ffd640 1 -- 172.21.15.148:0/1593101483 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f70a40be090 con 0x7f70ac15cd50 2023-10-30T13:35:51.944 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:51.945 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:35:44.780053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:51.945 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 104 2023-10-30T13:35:51.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.948+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f708c0664a0 msgr2=0x7f708c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:51.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.948+0000 7f70ab59e640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f708c0664a0 0x7f708c068960 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f709c009cf0 tx=0x7f709c009190 comp rx=0 tx=0).stop 2023-10-30T13:35:51.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.948+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70ac15cd50 msgr2=0x7f70ac152650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:51.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.948+0000 7f70ab59e640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70ac15cd50 0x7f70ac152650 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009d90 tx=0x7f70a401bed0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.952+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 shutdown_connections 2023-10-30T13:35:51.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.952+0000 7f70ab59e640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f708c0664a0 0x7f708c068960 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.952+0000 7f70ab59e640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f70ac15cd50 0x7f70ac152650 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.952+0000 7f70ab59e640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70ac1538f0 0x7f70ac152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.952+0000 7f70ab59e640 1 --2- 172.21.15.148:0/1593101483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70ac0cefa0 0x7f70ac15b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:51.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.952+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 >> 172.21.15.148:0/1593101483 conn(0x7f70ac06a640 msgr2=0x7f70ac148be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:51.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.952+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 shutdown_connections 2023-10-30T13:35:51.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:51.952+0000 7f70ab59e640 1 -- 172.21.15.148:0/1593101483 wait complete. 2023-10-30T13:35:51.967 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:53.968 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:54.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.120+0000 7f2213046640 1 Processor -- start 2023-10-30T13:35:54.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2213046640 1 -- start start 2023-10-30T13:35:54.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2213046640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c06f7a0 0x7f220c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:54.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2213046640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f220c0c6bd0 0x7f220c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:54.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2213046640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220c06a3d0 con 0x7f220c071110 2023-10-30T13:35:54.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2213046640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220c0700c0 con 0x7f220c0c6bd0 2023-10-30T13:35:54.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2213046640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c066cf0 con 0x7f220c06f7a0 2023-10-30T13:35:54.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f22115e3640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f220c071110 0x7f220c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46026/0 (socket says 172.21.15.148:46026) 2023-10-30T13:35:54.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f22115e3640 1 -- 172.21.15.148:0/2259982791 learned_addr learned my addr 172.21.15.148:0/2259982791 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:35:54.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2210de2640 1 --2- 172.21.15.148:0/2259982791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c06f7a0 0x7f220c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:54.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f220bfff640 1 --2- 172.21.15.148:0/2259982791 >> v2:172.21.15.150:3300/0 conn(0x7f220c0c6bd0 0x7f220c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:54.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f220b7fe640 1 -- 172.21.15.148:0/2259982791 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064727113 0 0) 0x7f220c06a3d0 con 0x7f220c071110 2023-10-30T13:35:54.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f220b7fe640 1 -- 172.21.15.148:0/2259982791 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21fc003530 con 0x7f220c071110 2023-10-30T13:35:54.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2210de2640 1 -- 172.21.15.148:0/2259982791 >> v2:172.21.15.150:3300/0 conn(0x7f220c0c6bd0 msgr2=0x7f220c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:54.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2210de2640 1 --2- 172.21.15.148:0/2259982791 >> v2:172.21.15.150:3300/0 conn(0x7f220c0c6bd0 0x7f220c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:54.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2210de2640 1 -- 172.21.15.148:0/2259982791 >> v1:172.21.15.148:6789/0 conn(0x7f220c071110 legacy=0x7f220c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:54.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f220b7fe640 1 -- 172.21.15.148:0/2259982791 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1261744302 0 0) 0x7f21fc003530 con 0x7f220c071110 2023-10-30T13:35:54.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.124+0000 7f2210de2640 1 -- 172.21.15.148:0/2259982791 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c0c70d0 con 0x7f220c06f7a0 2023-10-30T13:35:54.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f2210de2640 1 --2- 172.21.15.148:0/2259982791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c06f7a0 0x7f220c06fb80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f22000097c0 tx=0x7f220001eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aba8702750171d8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:54.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f220b7fe640 1 -- 172.21.15.148:0/2259982791 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204001a00 con 0x7f220c06f7a0 2023-10-30T13:35:54.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f220b7fe640 1 -- 172.21.15.148:0/2259982791 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22000a85d0 con 0x7f220c06f7a0 2023-10-30T13:35:54.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f220b7fe640 1 -- 172.21.15.148:0/2259982791 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000a8940 con 0x7f220c06f7a0 2023-10-30T13:35:54.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f2213046640 1 -- 172.21.15.148:0/2259982791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c06f7a0 msgr2=0x7f220c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:54.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f2213046640 1 --2- 172.21.15.148:0/2259982791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c06f7a0 0x7f220c06fb80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f22000097c0 tx=0x7f220001eb90 comp rx=0 tx=0).stop 2023-10-30T13:35:54.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f2213046640 1 -- 172.21.15.148:0/2259982791 shutdown_connections 2023-10-30T13:35:54.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f2213046640 1 --2- 172.21.15.148:0/2259982791 >> v2:172.21.15.150:3300/0 conn(0x7f220c0c6bd0 0x7f220c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:54.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f2213046640 1 --2- 172.21.15.148:0/2259982791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c06f7a0 0x7f220c06fb80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:54.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f2213046640 1 -- 172.21.15.148:0/2259982791 >> 172.21.15.148:0/2259982791 conn(0x7f220c0c7910 msgr2=0x7f220c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:54.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f2213046640 1 -- 172.21.15.148:0/2259982791 shutdown_connections 2023-10-30T13:35:54.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.128+0000 7f2213046640 1 -- 172.21.15.148:0/2259982791 wait complete. 2023-10-30T13:35:54.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 Processor -- start 2023-10-30T13:35:54.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 -- start start 2023-10-30T13:35:54.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f220c06f7a0 0x7f220c0ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:54.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f220c071110 0x7f220c0c07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:54.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c0c6bd0 0x7f220c0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:54.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0c1170 con 0x7f220c06f7a0 2023-10-30T13:35:54.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0c1660 con 0x7f220c071110 2023-10-30T13:35:54.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0c0360 con 0x7f220c0c6bd0 2023-10-30T13:35:54.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2210de2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f220c06f7a0 0x7f220c0ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:54.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2210de2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f220c06f7a0 0x7f220c0ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46426/0 (socket says 172.21.15.148:46426) 2023-10-30T13:35:54.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2210de2640 1 -- 172.21.15.148:0/1416368950 learned_addr learned my addr 172.21.15.148:0/1416368950 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:54.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f220bfff640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f220c071110 0x7f220c0c07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:54.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f22115e3640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c0c6bd0 0x7f220c0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:54.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2210de2640 1 -- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c0c6bd0 msgr2=0x7f220c0ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:54.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2210de2640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c0c6bd0 0x7f220c0ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:54.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2210de2640 1 -- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f220c071110 msgr2=0x7f220c0c07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:54.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2210de2640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f220c071110 0x7f220c0c07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:54.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2210de2640 1 -- 172.21.15.148:0/1416368950 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2200009420 con 0x7f220c06f7a0 2023-10-30T13:35:54.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2210de2640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f220c06f7a0 0x7f220c0ae950 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2200000c10 tx=0x7f220001b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:54.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f22097fa640 1 -- 172.21.15.148:0/1416368950 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000a8c10 con 0x7f220c06f7a0 2023-10-30T13:35:54.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f22097fa640 1 -- 172.21.15.148:0/1416368950 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22000a8d70 con 0x7f220c06f7a0 2023-10-30T13:35:54.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f22097fa640 1 -- 172.21.15.148:0/1416368950 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200003820 con 0x7f220c06f7a0 2023-10-30T13:35:54.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220c0c0d30 con 0x7f220c06f7a0 2023-10-30T13:35:54.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.132+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c0c1380 con 0x7f220c06f7a0 2023-10-30T13:35:54.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.136+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c04e170 con 0x7f220c06f7a0 2023-10-30T13:35:54.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.136+0000 7f22097fa640 1 -- 172.21.15.148:0/1416368950 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22000b7040 con 0x7f220c06f7a0 2023-10-30T13:35:54.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.136+0000 7f22097fa640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21f00664d0 0x7f21f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:54.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.136+0000 7f22097fa640 1 -- 172.21.15.148:0/1416368950 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f2200121770 con 0x7f220c06f7a0 2023-10-30T13:35:54.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.136+0000 7f220bfff640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21f00664d0 0x7f21f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:54.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.136+0000 7f220bfff640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21f00664d0 0x7f21f0068990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f220c072150 tx=0x7f21f8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:54.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.140+0000 7f22097fa640 1 -- 172.21.15.148:0/1416368950 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f22000ee830 con 0x7f220c06f7a0 2023-10-30T13:35:54.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.328+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f220c0c4990 con 0x7f220c06f7a0 2023-10-30T13:35:54.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f22097fa640 1 -- 172.21.15.148:0/1416368950 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f22000f1e50 con 0x7f220c06f7a0 2023-10-30T13:35:54.327 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:54.327 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:35:44.780053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:54.328 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 104 2023-10-30T13:35:54.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21f00664d0 msgr2=0x7f21f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:54.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21f00664d0 0x7f21f0068990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f220c072150 tx=0x7f21f8009190 comp rx=0 tx=0).stop 2023-10-30T13:35:54.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f220c06f7a0 msgr2=0x7f220c0ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:54.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f220c06f7a0 0x7f220c0ae950 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2200000c10 tx=0x7f220001b930 comp rx=0 tx=0).stop 2023-10-30T13:35:54.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 shutdown_connections 2023-10-30T13:35:54.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f21f00664d0 0x7f21f0068990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:54.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f220c0c6bd0 0x7f220c0ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:54.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f220c071110 0x7f220c0c07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:54.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 --2- 172.21.15.148:0/1416368950 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f220c06f7a0 0x7f220c0ae950 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:54.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 >> 172.21.15.148:0/1416368950 conn(0x7f220c0c7910 msgr2=0x7f220c0c3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:54.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 shutdown_connections 2023-10-30T13:35:54.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:54.332+0000 7f2213046640 1 -- 172.21.15.148:0/1416368950 wait complete. 2023-10-30T13:35:54.348 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:56.350 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:56.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.496+0000 7f8f81e91640 1 Processor -- start 2023-10-30T13:35:56.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.496+0000 7f8f81e91640 1 -- start start 2023-10-30T13:35:56.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f81e91640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8f7c0c6b80 0x7f8f7c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:56.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f81e91640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 0x7f8f7c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:56.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f81e91640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c0c0a50 con 0x7f8f7c0c8430 2023-10-30T13:35:56.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f81e91640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7c06f6e0 con 0x7f8f7c0c6b80 2023-10-30T13:35:56.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f81e91640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c06f9c0 con 0x7f8f7c0c6f60 2023-10-30T13:35:56.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f7b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8f7c0c6b80 0x7f8f7c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:56.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f7b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8f7c0c6b80 0x7f8f7c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46440/0 (socket says 172.21.15.148:46440) 2023-10-30T13:35:56.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f7b7fe640 1 -- 172.21.15.148:0/3933510682 learned_addr learned my addr 172.21.15.148:0/3933510682 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:56.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f7affd640 1 --2- 172.21.15.148:0/3933510682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 0x7f8f7c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:56.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f7b7fe640 1 -- 172.21.15.148:0/3933510682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 msgr2=0x7f8f7c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:56.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f7b7fe640 1 --2- 172.21.15.148:0/3933510682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 0x7f8f7c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f7b7fe640 1 -- 172.21.15.148:0/3933510682 >> v1:172.21.15.150:6789/0 conn(0x7f8f7c0c8430 legacy=0x7f8f7c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:56.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.500+0000 7f8f7b7fe640 1 -- 172.21.15.148:0/3933510682 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c06fca0 con 0x7f8f7c0c6b80 2023-10-30T13:35:56.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f7b7fe640 1 --2- 172.21.15.148:0/3933510682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c6b80 0x7f8f7c0bfec0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009870 tx=0x7f8f6c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c38f1486c0289c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:56.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f7a7fc640 1 -- 172.21.15.148:0/3933510682 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c0ac620 con 0x7f8f7c0c6b80 2023-10-30T13:35:56.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f7a7fc640 1 -- 172.21.15.148:0/3933510682 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c0ac780 con 0x7f8f7c0c6b80 2023-10-30T13:35:56.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f7a7fc640 1 -- 172.21.15.148:0/3933510682 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c0acb10 con 0x7f8f7c0c6b80 2023-10-30T13:35:56.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 -- 172.21.15.148:0/3933510682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c6b80 msgr2=0x7f8f7c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:56.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 --2- 172.21.15.148:0/3933510682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c6b80 0x7f8f7c0bfec0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009870 tx=0x7f8f6c01c4b0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 -- 172.21.15.148:0/3933510682 shutdown_connections 2023-10-30T13:35:56.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 --2- 172.21.15.148:0/3933510682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 0x7f8f7c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 --2- 172.21.15.148:0/3933510682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c6b80 0x7f8f7c0bfec0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 -- 172.21.15.148:0/3933510682 >> 172.21.15.148:0/3933510682 conn(0x7f8f7c0c7910 msgr2=0x7f8f7c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:56.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 -- 172.21.15.148:0/3933510682 shutdown_connections 2023-10-30T13:35:56.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 -- 172.21.15.148:0/3933510682 wait complete. 2023-10-30T13:35:56.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 Processor -- start 2023-10-30T13:35:56.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.504+0000 7f8f81e91640 1 -- start start 2023-10-30T13:35:56.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f81e91640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f7c0c6b80 0x7f8f7c14c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:56.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f81e91640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 0x7f8f7c0afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:56.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f81e91640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c8430 0x7f8f7c14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:56.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f81e91640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c159310 con 0x7f8f7c0c8430 2023-10-30T13:35:56.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f81e91640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c159a10 con 0x7f8f7c0c6b80 2023-10-30T13:35:56.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f81e91640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0c93d0 con 0x7f8f7c0c6f60 2023-10-30T13:35:56.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c8430 0x7f8f7c14e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:56.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c8430 0x7f8f7c14e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46444/0 (socket says 172.21.15.148:46444) 2023-10-30T13:35:56.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7bfff640 1 -- 172.21.15.148:0/4120068379 learned_addr learned my addr 172.21.15.148:0/4120068379 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:56.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7affd640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 0x7f8f7c0afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:56.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7b7fe640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f7c0c6b80 0x7f8f7c14c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:56.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7bfff640 1 -- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 msgr2=0x7f8f7c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:56.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7bfff640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 0x7f8f7c0afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7bfff640 1 -- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f7c0c6b80 msgr2=0x7f8f7c14c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:56.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7bfff640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f7c0c6b80 0x7f8f7c14c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7bfff640 1 -- 172.21.15.148:0/4120068379 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c0094d0 con 0x7f8f7c0c8430 2023-10-30T13:35:56.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f7bfff640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c8430 0x7f8f7c14e680 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400c760 tx=0x7f8f7400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:56.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f63fff640 1 -- 172.21.15.148:0/4120068379 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74003b80 con 0x7f8f7c0c8430 2023-10-30T13:35:56.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f63fff640 1 -- 172.21.15.148:0/4120068379 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74003ce0 con 0x7f8f7c0c8430 2023-10-30T13:35:56.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f63fff640 1 -- 172.21.15.148:0/4120068379 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7400aa30 con 0x7f8f7c0c8430 2023-10-30T13:35:56.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c14ac70 con 0x7f8f7c0c8430 2023-10-30T13:35:56.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c14a4e0 con 0x7f8f7c0c8430 2023-10-30T13:35:56.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c1575c0 con 0x7f8f7c0c8430 2023-10-30T13:35:56.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.508+0000 7f8f63fff640 1 -- 172.21.15.148:0/4120068379 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f7400ab90 con 0x7f8f7c0c8430 2023-10-30T13:35:56.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.512+0000 7f8f63fff640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f680664a0 0x7f8f68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:56.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.512+0000 7f8f63fff640 1 -- 172.21.15.148:0/4120068379 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f8f74083ff0 con 0x7f8f7c0c8430 2023-10-30T13:35:56.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.512+0000 7f8f7b7fe640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f680664a0 0x7f8f68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:56.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.512+0000 7f8f7b7fe640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f680664a0 0x7f8f68068960 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0ae000 tx=0x7f8f6c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:56.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.516+0000 7f8f63fff640 1 -- 172.21.15.148:0/4120068379 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f74050f90 con 0x7f8f7c0c8430 2023-10-30T13:35:56.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.704+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f7c04e170 con 0x7f8f7c0c8430 2023-10-30T13:35:56.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.704+0000 7f8f63fff640 1 -- 172.21.15.148:0/4120068379 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f8f740545b0 con 0x7f8f7c0c8430 2023-10-30T13:35:56.701 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:56.701 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:35:44.780053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:56.701 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 104 2023-10-30T13:35:56.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.704+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f680664a0 msgr2=0x7f8f68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:56.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f680664a0 0x7f8f68068960 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0ae000 tx=0x7f8f6c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:35:56.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c8430 msgr2=0x7f8f7c14e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:56.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c8430 0x7f8f7c14e680 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400c760 tx=0x7f8f7400cc20 comp rx=0 tx=0).stop 2023-10-30T13:35:56.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 shutdown_connections 2023-10-30T13:35:56.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f680664a0 0x7f8f68068960 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f7c0c8430 0x7f8f7c14e680 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f7c0c6f60 0x7f8f7c0afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 --2- 172.21.15.148:0/4120068379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f7c0c6b80 0x7f8f7c14c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:56.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 >> 172.21.15.148:0/4120068379 conn(0x7f8f7c0c7910 msgr2=0x7f8f7c0c19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:56.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 shutdown_connections 2023-10-30T13:35:56.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:56.708+0000 7f8f81e91640 1 -- 172.21.15.148:0/4120068379 wait complete. 2023-10-30T13:35:56.723 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:35:58.724 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:35:58.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f0705067640 1 Processor -- start 2023-10-30T13:35:58.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f0705067640 1 -- start start 2023-10-30T13:35:58.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f0705067640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f070015cd50 0x7f07000c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:58.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f0705067640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070015a6c0 con 0x7f07000cefa0 2023-10-30T13:35:58.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f0705067640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070015d300 con 0x7f07001538f0 2023-10-30T13:35:58.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f0705067640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f070015d5e0 con 0x7f070015cd50 2023-10-30T13:35:58.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f0704866640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f07000cefa0 0x7f07001640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46030/0 (socket says 172.21.15.148:46030) 2023-10-30T13:35:58.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f0704866640 1 -- 172.21.15.148:0/621137451 learned_addr learned my addr 172.21.15.148:0/621137451 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:35:58.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06ff7fe640 1 --2- 172.21.15.148:0/621137451 >> v2:172.21.15.150:3300/0 conn(0x7f070015cd50 0x7f07000c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:58.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959129917 0 0) 0x7f070015a6c0 con 0x7f07000cefa0 2023-10-30T13:35:58.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e8003530 con 0x7f07000cefa0 2023-10-30T13:35:58.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3028381787 0 0) 0x7f06e8003530 con 0x7f07000cefa0 2023-10-30T13:35:58.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 >> v2:172.21.15.150:3300/0 conn(0x7f070015cd50 msgr2=0x7f07000c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:58.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 --2- 172.21.15.148:0/621137451 >> v2:172.21.15.150:3300/0 conn(0x7f070015cd50 0x7f07000c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:58.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 >> v1:172.21.15.150:6789/0 conn(0x7f07001538f0 legacy=0x7f0700159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:58.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070015d8c0 con 0x7f07000cefa0 2023-10-30T13:35:58.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f06f8001a00 con 0x7f07000cefa0 2023-10-30T13:35:58.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 >> v1:172.21.15.148:6789/0 conn(0x7f07000cefa0 legacy=0x7f07001640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:58.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.876+0000 7f06feffd640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f06e8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:58.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06feffd640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e80047d0 0x7f06e8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:58.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06feffd640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e8005190 0x7f06e800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:58.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070015d5e0 con 0x7f06e8003980 2023-10-30T13:35:58.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e800c040 con 0x7f06e80047d0 2023-10-30T13:35:58.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e800c200 con 0x7f06e8005190 2023-10-30T13:35:58.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06fffff640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f06e8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:58.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f0704866640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e8005190 0x7f06e800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:58.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06ff7fe640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e80047d0 0x7f06e8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:58.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06fffff640 1 -- 172.21.15.148:0/621137451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e8005190 msgr2=0x7f06e800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:58.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06fffff640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f06e8005190 0x7f06e800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:58.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06fffff640 1 -- 172.21.15.148:0/621137451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e80047d0 msgr2=0x7f06e8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:35:58.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06fffff640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e80047d0 0x7f06e8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:58.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06fffff640 1 -- 172.21.15.148:0/621137451 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e8003530 con 0x7f06e8003980 2023-10-30T13:35:58.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f0704866640 1 -- 172.21.15.148:0/621137451 reap_dead start 2023-10-30T13:35:58.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06fffff640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f06e8003da0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f06f0001780 tx=0x7f06f001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a4e0ddf4bc5d24 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:58.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f00a4ea0 con 0x7f06e8003980 2023-10-30T13:35:58.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f00a7050 con 0x7f06e8003980 2023-10-30T13:35:58.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.880+0000 7f06feffd640 1 -- 172.21.15.148:0/621137451 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0003600 con 0x7f06e8003980 2023-10-30T13:35:58.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 -- 172.21.15.148:0/621137451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 msgr2=0x7f06e8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:58.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f06e8003da0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f06f0001780 tx=0x7f06f001c3e0 comp rx=0 tx=0).stop 2023-10-30T13:35:58.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 -- 172.21.15.148:0/621137451 shutdown_connections 2023-10-30T13:35:58.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 --2- 172.21.15.148:0/621137451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f06e8003da0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:58.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 -- 172.21.15.148:0/621137451 >> 172.21.15.148:0/621137451 conn(0x7f070006a640 msgr2=0x7f0700065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:58.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 -- 172.21.15.148:0/621137451 shutdown_connections 2023-10-30T13:35:58.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 -- 172.21.15.148:0/621137451 wait complete. 2023-10-30T13:35:58.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 Processor -- start 2023-10-30T13:35:58.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 -- start start 2023-10-30T13:35:58.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f07001f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:58.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0700154b90 0x7f07001efba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:58.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07001582d0 0x7f070015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:58.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07001553f0 con 0x7f06e8003980 2023-10-30T13:35:58.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07001f00e0 con 0x7f0700154b90 2023-10-30T13:35:58.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0705067640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07001540f0 con 0x7f07001582d0 2023-10-30T13:35:58.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f06fffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f07001f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:58.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f06fffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f07001f2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46462/0 (socket says 172.21.15.148:46462) 2023-10-30T13:35:58.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f06fffff640 1 -- 172.21.15.148:0/2206757400 learned_addr learned my addr 172.21.15.148:0/2206757400 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:35:58.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f06ff7fe640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0700154b90 0x7f07001efba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:58.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f0704866640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07001582d0 0x7f070015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:58.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f06fffff640 1 -- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07001582d0 msgr2=0x7f070015b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:58.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f06fffff640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07001582d0 0x7f070015b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:58.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f06fffff640 1 -- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0700154b90 msgr2=0x7f07001efba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:58.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f06fffff640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0700154b90 0x7f07001efba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:58.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.884+0000 7f06fffff640 1 -- 172.21.15.148:0/2206757400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700154ff0 con 0x7f06e8003980 2023-10-30T13:35:58.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f06fffff640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f07001f2690 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f06f001c210 tx=0x7f06e8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:58.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f06fcff9640 1 -- 172.21.15.148:0/2206757400 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0003e40 con 0x7f06e8003980 2023-10-30T13:35:58.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f06fcff9640 1 -- 172.21.15.148:0/2206757400 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f0007a40 con 0x7f06e8003980 2023-10-30T13:35:58.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f06fcff9640 1 -- 172.21.15.148:0/2206757400 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0007270 con 0x7f06e8003980 2023-10-30T13:35:58.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0700156940 con 0x7f06e8003980 2023-10-30T13:35:58.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0700154390 con 0x7f06e8003980 2023-10-30T13:35:58.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070004e170 con 0x7f06e8003980 2023-10-30T13:35:58.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f06fcff9640 1 -- 172.21.15.148:0/2206757400 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06f00073d0 con 0x7f06e8003980 2023-10-30T13:35:58.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f06fcff9640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664d0 0x7f06ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:35:58.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f06fcff9640 1 -- 172.21.15.148:0/2206757400 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f06f011e730 con 0x7f06e8003980 2023-10-30T13:35:58.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.888+0000 7f06ff7fe640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664d0 0x7f06ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:35:58.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.892+0000 7f06ff7fe640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664d0 0x7f06ec068990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0700065fc0 tx=0x7f06f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:35:58.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:58.892+0000 7f06fcff9640 1 -- 172.21.15.148:0/2206757400 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06f00eb6b0 con 0x7f06e8003980 2023-10-30T13:35:59.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.076+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f070014d750 con 0x7f06e8003980 2023-10-30T13:35:59.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f06fcff9640 1 -- 172.21.15.148:0/2206757400 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f06f00eecd0 con 0x7f06e8003980 2023-10-30T13:35:59.075 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:35:59.075 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:35:56.719949+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:35:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 105 2023-10-30T13:35:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664d0 msgr2=0x7f06ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664d0 0x7f06ec068990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0700065fc0 tx=0x7f06f4008040 comp rx=0 tx=0).stop 2023-10-30T13:35:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 msgr2=0x7f07001f2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:35:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f07001f2690 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f06f001c210 tx=0x7f06e8005800 comp rx=0 tx=0).stop 2023-10-30T13:35:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 shutdown_connections 2023-10-30T13:35:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664d0 0x7f06ec068990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07001582d0 0x7f070015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0700154b90 0x7f07001efba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 --2- 172.21.15.148:0/2206757400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06e8003980 0x7f07001f2690 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:35:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 >> 172.21.15.148:0/2206757400 conn(0x7f070006a640 msgr2=0x7f07000cf540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:35:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 shutdown_connections 2023-10-30T13:35:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:35:59.080+0000 7f0705067640 1 -- 172.21.15.148:0/2206757400 wait complete. 2023-10-30T13:35:59.096 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 105, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 105, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6870": { "addr": "172.21.15.148:6837/3144126860", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3144126860, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3144126860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6870, "incarnation": 104, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:35:56.719949+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6870 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:36:00.443 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:00.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:01.097 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:01.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.248+0000 7fe4c9446640 1 Processor -- start 2023-10-30T13:36:01.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.248+0000 7fe4c9446640 1 -- start start 2023-10-30T13:36:01.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c9446640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe4c41538a0 0x7fe4c415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:01.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c9446640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4c40cf160 0x7fe4c40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:01.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c9446640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c40cb800 con 0x7fe4c40ca0e0 2023-10-30T13:36:01.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c9446640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c415dc30 con 0x7fe4c40cf160 2023-10-30T13:36:01.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c9446640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c40cb330 con 0x7fe4c41538a0 2023-10-30T13:36:01.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c37fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4c40cf160 0x7fe4c40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:01.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c37fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4c40cf160 0x7fe4c40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46470/0 (socket says 172.21.15.148:46470) 2023-10-30T13:36:01.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c37fe640 1 -- 172.21.15.148:0/3504906382 learned_addr learned my addr 172.21.15.148:0/3504906382 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:36:01.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c2ffd640 1 --2- 172.21.15.148:0/3504906382 >> v2:172.21.15.150:3300/0 conn(0x7fe4c41538a0 0x7fe4c415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:01.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c37fe640 1 -- 172.21.15.148:0/3504906382 >> v2:172.21.15.150:3300/0 conn(0x7fe4c41538a0 msgr2=0x7fe4c415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:01.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c37fe640 1 --2- 172.21.15.148:0/3504906382 >> v2:172.21.15.150:3300/0 conn(0x7fe4c41538a0 0x7fe4c415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c37fe640 1 -- 172.21.15.148:0/3504906382 >> v1:172.21.15.150:6789/0 conn(0x7fe4c40ca0e0 legacy=0x7fe4c415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:01.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.252+0000 7fe4c37fe640 1 -- 172.21.15.148:0/3504906382 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c40cad70 con 0x7fe4c40cf160 2023-10-30T13:36:01.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c37fe640 1 --2- 172.21.15.148:0/3504906382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c40cf160 0x7fe4c40cf590 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40cfd10 tx=0x7fe4bc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33820eec713c0dff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:01.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c27fc640 1 -- 172.21.15.148:0/3504906382 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0a89f0 con 0x7fe4c40cf160 2023-10-30T13:36:01.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c27fc640 1 -- 172.21.15.148:0/3504906382 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc0a8b50 con 0x7fe4c40cf160 2023-10-30T13:36:01.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c27fc640 1 -- 172.21.15.148:0/3504906382 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0b0440 con 0x7fe4c40cf160 2023-10-30T13:36:01.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 -- 172.21.15.148:0/3504906382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c40cf160 msgr2=0x7fe4c40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:01.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 --2- 172.21.15.148:0/3504906382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c40cf160 0x7fe4c40cf590 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40cfd10 tx=0x7fe4bc01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 -- 172.21.15.148:0/3504906382 shutdown_connections 2023-10-30T13:36:01.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 --2- 172.21.15.148:0/3504906382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c40cf160 0x7fe4c40cf590 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 --2- 172.21.15.148:0/3504906382 >> v2:172.21.15.150:3300/0 conn(0x7fe4c41538a0 0x7fe4c415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 -- 172.21.15.148:0/3504906382 >> 172.21.15.148:0/3504906382 conn(0x7fe4c406bfe0 msgr2=0x7fe4c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:01.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 -- 172.21.15.148:0/3504906382 shutdown_connections 2023-10-30T13:36:01.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 -- 172.21.15.148:0/3504906382 wait complete. 2023-10-30T13:36:01.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 Processor -- start 2023-10-30T13:36:01.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.256+0000 7fe4c9446640 1 -- start start 2023-10-30T13:36:01.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c9446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4c40ca0e0 0x7fe4c4143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:01.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c9446640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c40cf160 0x7fe4c41f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:01.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c9446640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c41538a0 0x7fe4c40c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:01.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c9446640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c4144510 con 0x7fe4c41538a0 2023-10-30T13:36:01.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c9446640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c41f3800 con 0x7fe4c40cf160 2023-10-30T13:36:01.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c9446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c40b34d0 con 0x7fe4c40ca0e0 2023-10-30T13:36:01.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c41538a0 0x7fe4c40c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:01.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c41538a0 0x7fe4c40c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46472/0 (socket says 172.21.15.148:46472) 2023-10-30T13:36:01.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c37fe640 1 -- 172.21.15.148:0/3223171689 learned_addr learned my addr 172.21.15.148:0/3223171689 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:36:01.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4ba7fc640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c40cf160 0x7fe4c41f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:01.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c2ffd640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4c40ca0e0 0x7fe4c4143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:01.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c37fe640 1 -- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4c40ca0e0 msgr2=0x7fe4c4143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:01.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c37fe640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4c40ca0e0 0x7fe4c4143fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c37fe640 1 -- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c40cf160 msgr2=0x7fe4c41f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:01.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c37fe640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c40cf160 0x7fe4c41f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c37fe640 1 -- 172.21.15.148:0/3223171689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc0094d0 con 0x7fe4c41538a0 2023-10-30T13:36:01.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c2ffd640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4c40ca0e0 0x7fe4c4143fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:36:01.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4ba7fc640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c40cf160 0x7fe4c41f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:36:01.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c37fe640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c41538a0 0x7fe4c40c4c80 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0b3040 tx=0x7fe4bc01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:01.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4bbfff640 1 -- 172.21.15.148:0/3223171689 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0b0d50 con 0x7fe4c41538a0 2023-10-30T13:36:01.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4bbfff640 1 -- 172.21.15.148:0/3223171689 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc0bb460 con 0x7fe4c41538a0 2023-10-30T13:36:01.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4bbfff640 1 -- 172.21.15.148:0/3223171689 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0bb810 con 0x7fe4c41538a0 2023-10-30T13:36:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c41ef340 con 0x7fe4c41538a0 2023-10-30T13:36:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c41ef8c0 con 0x7fe4c41538a0 2023-10-30T13:36:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c40b2a50 con 0x7fe4c41538a0 2023-10-30T13:36:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.260+0000 7fe4bbfff640 1 -- 172.21.15.148:0/3223171689 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe4bc0b83f0 con 0x7fe4c41538a0 2023-10-30T13:36:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.264+0000 7fe4bbfff640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4a40664a0 0x7fe4a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.264+0000 7fe4bbfff640 1 -- 172.21.15.148:0/3223171689 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fe4bc121d00 con 0x7fe4c41538a0 2023-10-30T13:36:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.264+0000 7fe4c2ffd640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4a40664a0 0x7fe4a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.264+0000 7fe4c2ffd640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4a40664a0 0x7fe4a4068960 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009840 tx=0x7fe4b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:01.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.268+0000 7fe4bbfff640 1 -- 172.21.15.148:0/3223171689 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe4bc0eec00 con 0x7fe4c41538a0 2023-10-30T13:36:01.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4c404e170 con 0x7fe4c41538a0 2023-10-30T13:36:01.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4bbfff640 1 -- 172.21.15.148:0/3223171689 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fe4bc0b5020 con 0x7fe4c41538a0 2023-10-30T13:36:01.444 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:36:01.445 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:35:56.719949+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:01.445 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 105 2023-10-30T13:36:01.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4a40664a0 msgr2=0x7fe4a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:01.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4a40664a0 0x7fe4a4068960 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009840 tx=0x7fe4b4008040 comp rx=0 tx=0).stop 2023-10-30T13:36:01.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c41538a0 msgr2=0x7fe4c40c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:01.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c41538a0 0x7fe4c40c4c80 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0b3040 tx=0x7fe4bc01ba90 comp rx=0 tx=0).stop 2023-10-30T13:36:01.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 shutdown_connections 2023-10-30T13:36:01.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe4a40664a0 0x7fe4a4068960 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4c41538a0 0x7fe4c40c4c80 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c40cf160 0x7fe4c41f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 --2- 172.21.15.148:0/3223171689 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe4c40ca0e0 0x7fe4c4143fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:01.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 >> 172.21.15.148:0/3223171689 conn(0x7fe4c406bfe0 msgr2=0x7fe4c41513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:01.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 shutdown_connections 2023-10-30T13:36:01.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:01.448+0000 7fe4c9446640 1 -- 172.21.15.148:0/3223171689 wait complete. 2023-10-30T13:36:01.465 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:36:03.467 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:03.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59cd5a1640 1 Processor -- start 2023-10-30T13:36:03.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59cd5a1640 1 -- start start 2023-10-30T13:36:03.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59cd5a1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f59c80c90d0 0x7f59c80cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:03.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59cd5a1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c815ce30 con 0x7f59c80cf4d0 2023-10-30T13:36:03.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59cd5a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c8159880 con 0x7f59c80c8cf0 2023-10-30T13:36:03.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59cd5a1640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c8159ae0 con 0x7f59c80c90d0 2023-10-30T13:36:03.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59be7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f59c80c90d0 0x7f59c80cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:03.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59c77fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f59c80cf4d0 0x7f59c815d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54172/0 (socket says 172.21.15.148:54172) 2023-10-30T13:36:03.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59c77fe640 1 -- 172.21.15.148:0/2632863750 learned_addr learned my addr 172.21.15.148:0/2632863750 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:36:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59be7fc640 1 -- 172.21.15.148:0/2632863750 >> v1:172.21.15.150:6789/0 conn(0x7f59c80c8cf0 legacy=0x7f59c80c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.620+0000 7f59be7fc640 1 -- 172.21.15.148:0/2632863750 >> v1:172.21.15.148:6789/0 conn(0x7f59c80cf4d0 legacy=0x7f59c815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59be7fc640 1 -- 172.21.15.148:0/2632863750 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c8159dc0 con 0x7f59c80c90d0 2023-10-30T13:36:03.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59c67fc640 1 -- 172.21.15.148:0/2632863750 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905193282 0 0) 0x7f59c815ce30 con 0x7f59c80cf4d0 2023-10-30T13:36:03.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59be7fc640 1 --2- 172.21.15.148:0/2632863750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 0x7f59c80cef90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f59b00097c0 tx=0x7f59b001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655ccc4b495f1ee5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:03.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59c67fc640 1 -- 172.21.15.148:0/2632863750 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b00ad660 con 0x7f59c80c90d0 2023-10-30T13:36:03.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59c67fc640 1 -- 172.21.15.148:0/2632863750 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b00ad7c0 con 0x7f59c80c90d0 2023-10-30T13:36:03.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59c67fc640 1 -- 172.21.15.148:0/2632863750 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b00adb50 con 0x7f59c80c90d0 2023-10-30T13:36:03.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59cd5a1640 1 -- 172.21.15.148:0/2632863750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 msgr2=0x7f59c80cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:03.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59cd5a1640 1 --2- 172.21.15.148:0/2632863750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 0x7f59c80cef90 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f59b00097c0 tx=0x7f59b001c450 comp rx=0 tx=0).stop 2023-10-30T13:36:03.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59cd5a1640 1 -- 172.21.15.148:0/2632863750 shutdown_connections 2023-10-30T13:36:03.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59cd5a1640 1 --2- 172.21.15.148:0/2632863750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 0x7f59c80cef90 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:03.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59cd5a1640 1 -- 172.21.15.148:0/2632863750 >> 172.21.15.148:0/2632863750 conn(0x7f59c806be40 msgr2=0x7f59c806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:03.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59cd5a1640 1 -- 172.21.15.148:0/2632863750 shutdown_connections 2023-10-30T13:36:03.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.624+0000 7f59cd5a1640 1 -- 172.21.15.148:0/2632863750 wait complete. 2023-10-30T13:36:03.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59cd5a1640 1 Processor -- start 2023-10-30T13:36:03.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59cd5a1640 1 -- start start 2023-10-30T13:36:03.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59cd5a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c8cf0 0x7f59c815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:03.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59cd5a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 0x7f59c81f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:03.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59cd5a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80cf4d0 0x7f59c81f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:03.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59cd5a1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c81f1c20 con 0x7f59c80c90d0 2023-10-30T13:36:03.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59cd5a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c81f52a0 con 0x7f59c80cf4d0 2023-10-30T13:36:03.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59cd5a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c81f5580 con 0x7f59c80c8cf0 2023-10-30T13:36:03.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59be7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 0x7f59c81f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:03.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59be7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 0x7f59c81f2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47196/0 (socket says 172.21.15.148:47196) 2023-10-30T13:36:03.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59be7fc640 1 -- 172.21.15.148:0/1820115769 learned_addr learned my addr 172.21.15.148:0/1820115769 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:36:03.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59c6ffd640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c8cf0 0x7f59c815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:03.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59c77fe640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80cf4d0 0x7f59c81f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:03.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59be7fc640 1 -- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c8cf0 msgr2=0x7f59c815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:03.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59be7fc640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c8cf0 0x7f59c815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:03.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59be7fc640 1 -- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80cf4d0 msgr2=0x7f59c81f2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:03.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59be7fc640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80cf4d0 0x7f59c81f2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:03.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59be7fc640 1 -- 172.21.15.148:0/1820115769 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b0009420 con 0x7f59c80c90d0 2023-10-30T13:36:03.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59c77fe640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80cf4d0 0x7f59c81f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.628+0000 7f59be7fc640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 0x7f59c81f2030 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f59b001a080 tx=0x7f59b00191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59bffff640 1 -- 172.21.15.148:0/1820115769 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b0004220 con 0x7f59c80c90d0 2023-10-30T13:36:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59bffff640 1 -- 172.21.15.148:0/1820115769 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b0004380 con 0x7f59c80c90d0 2023-10-30T13:36:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59bffff640 1 -- 172.21.15.148:0/1820115769 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b0003820 con 0x7f59c80c90d0 2023-10-30T13:36:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c81f5860 con 0x7f59c80c90d0 2023-10-30T13:36:03.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c81f5e20 con 0x7f59c80c90d0 2023-10-30T13:36:03.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c80b1b30 con 0x7f59c80c90d0 2023-10-30T13:36:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59bffff640 1 -- 172.21.15.148:0/1820115769 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f59b00b4040 con 0x7f59c80c90d0 2023-10-30T13:36:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59bffff640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f59a80664d0 0x7f59a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59bffff640 1 -- 172.21.15.148:0/1820115769 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f59b011db00 con 0x7f59c80c90d0 2023-10-30T13:36:03.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.632+0000 7f59c6ffd640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f59a80664d0 0x7f59a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:03.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.636+0000 7f59c6ffd640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f59a80664d0 0x7f59a8068990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f59b8007fd0 tx=0x7f59b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:03.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.636+0000 7f59bffff640 1 -- 172.21.15.148:0/1820115769 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f59b00eaa90 con 0x7f59c80c90d0 2023-10-30T13:36:03.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59c804e170 con 0x7f59c80c90d0 2023-10-30T13:36:03.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59bffff640 1 -- 172.21.15.148:0/1820115769 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f59b00b2090 con 0x7f59c80c90d0 2023-10-30T13:36:03.832 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:36:03.832 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:01.956608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:03.832 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 106 2023-10-30T13:36:03.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f59a80664d0 msgr2=0x7f59a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:03.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f59a80664d0 0x7f59a8068990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f59b8007fd0 tx=0x7f59b8009040 comp rx=0 tx=0).stop 2023-10-30T13:36:03.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 msgr2=0x7f59c81f2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:03.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 0x7f59c81f2030 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f59b001a080 tx=0x7f59b00191f0 comp rx=0 tx=0).stop 2023-10-30T13:36:03.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 shutdown_connections 2023-10-30T13:36:03.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f59a80664d0 0x7f59a8068990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:03.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59c80c90d0 0x7f59c81f2030 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:03.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80cf4d0 0x7f59c81f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:03.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 --2- 172.21.15.148:0/1820115769 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f59c80c8cf0 0x7f59c815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:03.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 >> 172.21.15.148:0/1820115769 conn(0x7f59c806be40 msgr2=0x7f59c8158880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:03.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 shutdown_connections 2023-10-30T13:36:03.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:03.836+0000 7f59cd5a1640 1 -- 172.21.15.148:0/1820115769 wait complete. 2023-10-30T13:36:03.855 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:36:05.856 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:06.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154f148640 1 Processor -- start 2023-10-30T13:36:06.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154f148640 1 -- start start 2023-10-30T13:36:06.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154f148640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f15481539a0 0x7f15480717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:06.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154f148640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1548070450 0x7f15480c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:06.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154f148640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 0x7f154806e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:06.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154f148640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1548071fe0 con 0x7f1548070450 2023-10-30T13:36:06.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154f148640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15480722c0 con 0x7f15481539a0 2023-10-30T13:36:06.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154f148640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154815cb00 con 0x7f154806ded0 2023-10-30T13:36:06.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154d945640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1548070450 0x7f15480c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:06.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154d945640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1548070450 0x7f15480c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47202/0 (socket says 172.21.15.148:47202) 2023-10-30T13:36:06.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154d945640 1 -- 172.21.15.148:0/1130461908 learned_addr learned my addr 172.21.15.148:0/1130461908 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:36:06.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154e146640 1 --2- 172.21.15.148:0/1130461908 >> v2:172.21.15.150:3300/0 conn(0x7f15481539a0 0x7f15480717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:06.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154e947640 1 --2- 172.21.15.148:0/1130461908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 0x7f154806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:06.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154d945640 1 -- 172.21.15.148:0/1130461908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 msgr2=0x7f154806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:06.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154d945640 1 --2- 172.21.15.148:0/1130461908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 0x7f154806e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154d945640 1 -- 172.21.15.148:0/1130461908 >> v2:172.21.15.150:3300/0 conn(0x7f15481539a0 msgr2=0x7f15480717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:06.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154d945640 1 --2- 172.21.15.148:0/1130461908 >> v2:172.21.15.150:3300/0 conn(0x7f15481539a0 0x7f15480717c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154d945640 1 -- 172.21.15.148:0/1130461908 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154815cde0 con 0x7f1548070450 2023-10-30T13:36:06.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.008+0000 7f154e947640 1 --2- 172.21.15.148:0/1130461908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 0x7f154806e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:06.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154d945640 1 --2- 172.21.15.148:0/1130461908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1548070450 0x7f15480c65f0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f15480c7c00 tx=0x7f153801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20c83e01f1d26b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:06.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154d144640 1 -- 172.21.15.148:0/1130461908 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15380ab600 con 0x7f1548070450 2023-10-30T13:36:06.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154d144640 1 -- 172.21.15.148:0/1130461908 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15380a7050 con 0x7f1548070450 2023-10-30T13:36:06.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154d144640 1 -- 172.21.15.148:0/1130461908 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15380ab980 con 0x7f1548070450 2023-10-30T13:36:06.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154f148640 1 -- 172.21.15.148:0/1130461908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1548070450 msgr2=0x7f15480c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:06.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154f148640 1 --2- 172.21.15.148:0/1130461908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1548070450 0x7f15480c65f0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f15480c7c00 tx=0x7f153801c470 comp rx=0 tx=0).stop 2023-10-30T13:36:06.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154f148640 1 -- 172.21.15.148:0/1130461908 shutdown_connections 2023-10-30T13:36:06.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154f148640 1 --2- 172.21.15.148:0/1130461908 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 0x7f154806e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154f148640 1 --2- 172.21.15.148:0/1130461908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1548070450 0x7f15480c65f0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154f148640 1 --2- 172.21.15.148:0/1130461908 >> v2:172.21.15.150:3300/0 conn(0x7f15481539a0 0x7f15480717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154f148640 1 -- 172.21.15.148:0/1130461908 >> 172.21.15.148:0/1130461908 conn(0x7f15480cbe40 msgr2=0x7f15480cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:06.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154f148640 1 -- 172.21.15.148:0/1130461908 shutdown_connections 2023-10-30T13:36:06.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.012+0000 7f154f148640 1 -- 172.21.15.148:0/1130461908 wait complete. 2023-10-30T13:36:06.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 Processor -- start 2023-10-30T13:36:06.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 -- start start 2023-10-30T13:36:06.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 0x7f1548147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:06.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548070450 0x7f15481f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:06.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15481539a0 0x7f1548142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:06.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15480cb000 con 0x7f15481539a0 2023-10-30T13:36:06.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15480c9250 con 0x7f1548070450 2023-10-30T13:36:06.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1548144ab0 con 0x7f154806ded0 2023-10-30T13:36:06.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e947640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15481539a0 0x7f1548142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:06.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e947640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15481539a0 0x7f1548142a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47216/0 (socket says 172.21.15.148:47216) 2023-10-30T13:36:06.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e947640 1 -- 172.21.15.148:0/1385293395 learned_addr learned my addr 172.21.15.148:0/1385293395 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:36:06.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e146640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 0x7f1548147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:06.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154d945640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548070450 0x7f15481f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:06.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e947640 1 -- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 msgr2=0x7f1548147330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:06.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e947640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 0x7f1548147330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e947640 1 -- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548070450 msgr2=0x7f15481f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:06.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e947640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548070450 0x7f15481f2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e947640 1 -- 172.21.15.148:0/1385293395 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15380094d0 con 0x7f15481539a0 2023-10-30T13:36:06.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154e947640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15481539a0 0x7f1548142a90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1540007e10 tx=0x7f154000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:06.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f153effd640 1 -- 172.21.15.148:0/1385293395 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1540003ba0 con 0x7f15481539a0 2023-10-30T13:36:06.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f153effd640 1 -- 172.21.15.148:0/1385293395 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154000f040 con 0x7f15481539a0 2023-10-30T13:36:06.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1548144580 con 0x7f15481539a0 2023-10-30T13:36:06.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f153effd640 1 -- 172.21.15.148:0/1385293395 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154000de80 con 0x7f15481539a0 2023-10-30T13:36:06.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.016+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15481447f0 con 0x7f15481539a0 2023-10-30T13:36:06.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.020+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15480cb140 con 0x7f15481539a0 2023-10-30T13:36:06.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.020+0000 7f153effd640 1 -- 172.21.15.148:0/1385293395 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1540004030 con 0x7f15481539a0 2023-10-30T13:36:06.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.020+0000 7f153effd640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f152c0664a0 0x7f152c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:06.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.020+0000 7f153effd640 1 -- 172.21.15.148:0/1385293395 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f1540084640 con 0x7f15481539a0 2023-10-30T13:36:06.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.020+0000 7f154e146640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f152c0664a0 0x7f152c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:06.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.024+0000 7f154e146640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f152c0664a0 0x7f152c068960 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1544009a50 tx=0x7f1544008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:06.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.024+0000 7f153effd640 1 -- 172.21.15.148:0/1385293395 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15480cb140 con 0x7f15481539a0 2023-10-30T13:36:06.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.216+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15480cb140 con 0x7f15481539a0 2023-10-30T13:36:06.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.216+0000 7f153effd640 1 -- 172.21.15.148:0/1385293395 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f15480cb140 con 0x7f15481539a0 2023-10-30T13:36:06.213 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:36:06.214 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:01.956608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:06.214 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 106 2023-10-30T13:36:06.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f152c0664a0 msgr2=0x7f152c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:06.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f152c0664a0 0x7f152c068960 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1544009a50 tx=0x7f1544008040 comp rx=0 tx=0).stop 2023-10-30T13:36:06.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15481539a0 msgr2=0x7f1548142a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:06.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15481539a0 0x7f1548142a90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1540007e10 tx=0x7f154000d4a0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 shutdown_connections 2023-10-30T13:36:06.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f152c0664a0 0x7f152c068960 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15481539a0 0x7f1548142a90 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548070450 0x7f15481f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 --2- 172.21.15.148:0/1385293395 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f154806ded0 0x7f1548147330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:06.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 >> 172.21.15.148:0/1385293395 conn(0x7f15480cbe40 msgr2=0x7f15481573a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:06.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 shutdown_connections 2023-10-30T13:36:06.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:06.220+0000 7f154f148640 1 -- 172.21.15.148:0/1385293395 wait complete. 2023-10-30T13:36:06.234 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:36:08.235 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:08.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d750e640 1 Processor -- start 2023-10-30T13:36:08.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d750e640 1 -- start start 2023-10-30T13:36:08.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d750e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01538a0 0x7fa0d015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:08.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d750e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d00cb800 con 0x7fa0d00cf160 2023-10-30T13:36:08.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d750e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d015dc30 con 0x7fa0d00ca0e0 2023-10-30T13:36:08.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d750e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d00cb330 con 0x7fa0d01538a0 2023-10-30T13:36:08.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d5aab640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa0d00cf160 0x7fa0d0164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54186/0 (socket says 172.21.15.148:54186) 2023-10-30T13:36:08.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d5aab640 1 -- 172.21.15.148:0/2288232858 learned_addr learned my addr 172.21.15.148:0/2288232858 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:36:08.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d52aa640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01538a0 0x7fa0d015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:08.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885204245 0 0) 0x7fa0d00cb800 con 0x7fa0d00cf160 2023-10-30T13:36:08.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0bc003530 con 0x7fa0d00cf160 2023-10-30T13:36:08.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2401391996 0 0) 0x7fa0bc003530 con 0x7fa0d00cf160 2023-10-30T13:36:08.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01538a0 msgr2=0x7fa0d015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:08.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01538a0 0x7fa0d015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:08.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 >> v1:172.21.15.150:6789/0 conn(0x7fa0d00ca0e0 legacy=0x7fa0d015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:08.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d00cad70 con 0x7fa0d00cf160 2023-10-30T13:36:08.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fa0c8001a00 con 0x7fa0d00cf160 2023-10-30T13:36:08.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 >> v1:172.21.15.148:6789/0 conn(0x7fa0d00cf160 legacy=0x7fa0d0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:08.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0039b0 0x7fa0bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:08.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0bc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:08.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01538a0 0x7fa0bc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:08.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d00cb330 con 0x7fa0bc0047d0 2023-10-30T13:36:08.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc00bad0 con 0x7fa0bc0039b0 2023-10-30T13:36:08.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc00bc90 con 0x7fa0d01538a0 2023-10-30T13:36:08.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d4aa9640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0bc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:08.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d52aa640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0039b0 0x7fa0bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:08.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d5aab640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01538a0 0x7fa0bc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:08.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d4aa9640 1 -- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01538a0 msgr2=0x7fa0bc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:08.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d4aa9640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01538a0 0x7fa0bc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:08.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d4aa9640 1 -- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0039b0 msgr2=0x7fa0bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:08.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d4aa9640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0039b0 0x7fa0bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:08.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d4aa9640 1 -- 172.21.15.148:0/2288232858 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc003530 con 0x7fa0bc0047d0 2023-10-30T13:36:08.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.388+0000 7fa0d5aab640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01538a0 0x7fa0bc005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:08.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0d4aa9640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0bc004c50 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00015b0 tx=0x7fa0c001c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a40b262b225dfa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:08.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00052d0 con 0x7fa0bc0047d0 2023-10-30T13:36:08.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c00a7050 con 0x7fa0bc0047d0 2023-10-30T13:36:08.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0c7fff640 1 -- 172.21.15.148:0/2288232858 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0003600 con 0x7fa0bc0047d0 2023-10-30T13:36:08.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0d750e640 1 -- 172.21.15.148:0/2288232858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 msgr2=0x7fa0bc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:08.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0d750e640 1 --2- 172.21.15.148:0/2288232858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0bc004c50 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00015b0 tx=0x7fa0c001c200 comp rx=0 tx=0).stop 2023-10-30T13:36:08.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0d5aab640 1 -- 172.21.15.148:0/2288232858 reap_dead start 2023-10-30T13:36:08.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0d750e640 1 -- 172.21.15.148:0/2288232858 shutdown_connections 2023-10-30T13:36:08.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0d750e640 1 -- 172.21.15.148:0/2288232858 >> 172.21.15.148:0/2288232858 conn(0x7fa0d006bfe0 msgr2=0x7fa0d006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:08.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0d750e640 1 -- 172.21.15.148:0/2288232858 shutdown_connections 2023-10-30T13:36:08.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.392+0000 7fa0d750e640 1 -- 172.21.15.148:0/2288232858 wait complete. 2023-10-30T13:36:08.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 Processor -- start 2023-10-30T13:36:08.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 -- start start 2023-10-30T13:36:08.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0047d0 0x7fa0d01e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:08.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01e3720 0x7fa0d01e45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:08.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0d00ce6d0 0x7fa0d00b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:08.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d00b4a40 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d01e5910 con 0x7fa0bc0047d0 2023-10-30T13:36:08.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d00cef70 con 0x7fa0d01e3720 2023-10-30T13:36:08.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d5aab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0d00ce6d0 0x7fa0d00b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:08.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d5aab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0d00ce6d0 0x7fa0d00b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47240/0 (socket says 172.21.15.148:47240) 2023-10-30T13:36:08.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d5aab640 1 -- 172.21.15.148:0/2894544257 learned_addr learned my addr 172.21.15.148:0/2894544257 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:36:08.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d4aa9640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01e3720 0x7fa0d01e45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:08.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d52aa640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0047d0 0x7fa0d01e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:08.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d5aab640 1 -- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01e3720 msgr2=0x7fa0d01e45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:08.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d5aab640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01e3720 0x7fa0d01e45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:08.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d5aab640 1 -- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0047d0 msgr2=0x7fa0d01e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:08.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d5aab640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0047d0 0x7fa0d01e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:08.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d5aab640 1 -- 172.21.15.148:0/2894544257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d01e5670 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d52aa640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0047d0 0x7fa0d01e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:08.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d4aa9640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01e3720 0x7fa0d01e45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:36:08.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d5aab640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0d00ce6d0 0x7fa0d00b3210 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa0d006b0e0 tx=0x7fa0c800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:08.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0c5ffb640 1 -- 172.21.15.148:0/2894544257 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c800eb30 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0c5ffb640 1 -- 172.21.15.148:0/2894544257 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c800ec90 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0c5ffb640 1 -- 172.21.15.148:0/2894544257 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8013420 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d00cec90 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.396+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d0157cd0 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.400+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d0157850 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.400+0000 7fa0c5ffb640 1 -- 172.21.15.148:0/2894544257 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0c8013580 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.400+0000 7fa0c5ffb640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 0x7fa0b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:08.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.400+0000 7fa0c5ffb640 1 -- 172.21.15.148:0/2894544257 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fa0c8084750 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.400+0000 7fa0d52aa640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 0x7fa0b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:08.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.400+0000 7fa0d52aa640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 0x7fa0b4068990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009be0 tx=0x7fa0cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:08.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.404+0000 7fa0c5ffb640 1 -- 172.21.15.148:0/2894544257 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0d0157850 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0d0157850 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0c5ffb640 1 -- 172.21.15.148:0/2894544257 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5720 (secure 0 0 0) 0x7fa0d0157850 con 0x7fa0d00ce6d0 2023-10-30T13:36:08.580 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:36:08.580 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:08.581 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 107 2023-10-30T13:36:08.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 msgr2=0x7fa0b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:08.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 0x7fa0b4068990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009be0 tx=0x7fa0cc008040 comp rx=0 tx=0).stop 2023-10-30T13:36:08.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0d00ce6d0 msgr2=0x7fa0d00b3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:08.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0d00ce6d0 0x7fa0d00b3210 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa0d006b0e0 tx=0x7fa0c800e530 comp rx=0 tx=0).stop 2023-10-30T13:36:08.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 shutdown_connections 2023-10-30T13:36:08.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 0x7fa0b4068990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:08.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0d00ce6d0 0x7fa0d00b3210 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:08.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d01e3720 0x7fa0d01e45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:08.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 --2- 172.21.15.148:0/2894544257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc0047d0 0x7fa0d01e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:08.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 >> 172.21.15.148:0/2894544257 conn(0x7fa0d006bfe0 msgr2=0x7fa0d006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:08.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 shutdown_connections 2023-10-30T13:36:08.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:08.584+0000 7fa0d750e640 1 -- 172.21.15.148:0/2894544257 wait complete. 2023-10-30T13:36:08.602 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=6729 2023-10-30T13:36:08.603 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2023-10-30T13:36:15.603 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:36:15.604 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:36:15.604 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:36:15.608 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:36:15.608 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:36:15.695 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:36:15.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1ca25ab640 1 Processor -- start 2023-10-30T13:36:15.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1ca25ab640 1 -- start start 2023-10-30T13:36:15.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1ca25ab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1c9c1538a0 0x7f1c9c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:15.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1ca25ab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1c9c0ca0e0 0x7f1c9c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:15.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1ca25ab640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9c0cb800 con 0x7f1c9c0cf160 2023-10-30T13:36:15.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1ca25ab640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9c15dc30 con 0x7f1c9c0ca0e0 2023-10-30T13:36:15.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1ca25ab640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9c0cb330 con 0x7f1c9c1538a0 2023-10-30T13:36:15.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1c9b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1c9c0ca0e0 0x7f1c9c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:15.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1c9b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1c9c0ca0e0 0x7f1c9c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53776/0 (socket says 172.21.15.148:53776) 2023-10-30T13:36:15.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1c9b7fe640 1 -- 172.21.15.148:0/2929679169 learned_addr learned my addr 172.21.15.148:0/2929679169 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:36:15.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.755+0000 7f1c9bfff640 1 --2- 172.21.15.148:0/2929679169 >> v2:172.21.15.150:3300/0 conn(0x7f1c9c1538a0 0x7f1c9c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:15.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.759+0000 7f1c9b7fe640 1 -- 172.21.15.148:0/2929679169 >> v2:172.21.15.150:3300/0 conn(0x7f1c9c1538a0 msgr2=0x7f1c9c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:36:15.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.759+0000 7f1c9b7fe640 1 --2- 172.21.15.148:0/2929679169 >> v2:172.21.15.150:3300/0 conn(0x7f1c9c1538a0 0x7f1c9c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:15.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.759+0000 7f1c9b7fe640 1 -- 172.21.15.148:0/2929679169 >> v1:172.21.15.150:6789/0 conn(0x7f1c9c0cf160 legacy=0x7f1c9c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:15.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.759+0000 7f1c9b7fe640 1 -- 172.21.15.148:0/2929679169 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c0cad70 con 0x7f1c9c0ca0e0 2023-10-30T13:36:15.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.759+0000 7f1c9b7fe640 1 --2- 172.21.15.148:0/2929679169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c0ca0e0 0x7f1c9c15a090 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0cfd10 tx=0x7f1c9001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba3ebce7f109909 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:15.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.759+0000 7f1c9affd640 1 -- 172.21.15.148:0/2929679169 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c900a89f0 con 0x7f1c9c0ca0e0 2023-10-30T13:36:15.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.759+0000 7f1c9affd640 1 -- 172.21.15.148:0/2929679169 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c900a8b50 con 0x7f1c9c0ca0e0 2023-10-30T13:36:15.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.759+0000 7f1c9affd640 1 -- 172.21.15.148:0/2929679169 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c900b0440 con 0x7f1c9c0ca0e0 2023-10-30T13:36:15.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2929679169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c0ca0e0 msgr2=0x7f1c9c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:15.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 --2- 172.21.15.148:0/2929679169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c0ca0e0 0x7f1c9c15a090 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0cfd10 tx=0x7f1c9001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:36:15.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2929679169 shutdown_connections 2023-10-30T13:36:15.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 --2- 172.21.15.148:0/2929679169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c0ca0e0 0x7f1c9c15a090 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:15.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 --2- 172.21.15.148:0/2929679169 >> v2:172.21.15.150:3300/0 conn(0x7f1c9c1538a0 0x7f1c9c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:15.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2929679169 >> 172.21.15.148:0/2929679169 conn(0x7f1c9c06bfe0 msgr2=0x7f1c9c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:15.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2929679169 shutdown_connections 2023-10-30T13:36:15.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2929679169 wait complete. 2023-10-30T13:36:15.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 Processor -- start 2023-10-30T13:36:15.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.763+0000 7f1ca25ab640 1 -- start start 2023-10-30T13:36:15.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca25ab640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c9c0ca0e0 0x7f1c9c140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:15.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca25ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c9c0cf160 0x7f1c9c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:15.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca25ab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c1538a0 0x7f1c9c0cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:15.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca25ab640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c1e4380 con 0x7f1c9c1538a0 2023-10-30T13:36:15.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca25ab640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c158620 con 0x7f1c9c0cf160 2023-10-30T13:36:15.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca25ab640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c159690 con 0x7f1c9c0ca0e0 2023-10-30T13:36:15.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca0b48640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c1538a0 0x7f1c9c0cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:15.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca0b48640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c1538a0 0x7f1c9c0cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53780/0 (socket says 172.21.15.148:53780) 2023-10-30T13:36:15.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca0b48640 1 -- 172.21.15.148:0/2380837081 learned_addr learned my addr 172.21.15.148:0/2380837081 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:36:15.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1c9bfff640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c9c0ca0e0 0x7f1c9c140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:15.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1c9b7fe640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c9c0cf160 0x7f1c9c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:15.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca0b48640 1 -- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c9c0ca0e0 msgr2=0x7f1c9c140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:15.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca0b48640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c9c0ca0e0 0x7f1c9c140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:15.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca0b48640 1 -- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c9c0cf160 msgr2=0x7f1c9c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:15.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca0b48640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c9c0cf160 0x7f1c9c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:15.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca0b48640 1 -- 172.21.15.148:0/2380837081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c900094d0 con 0x7f1c9c1538a0 2023-10-30T13:36:15.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca0b48640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c1538a0 0x7f1c9c0cd4f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400c6f0 tx=0x7f1c9400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:15.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1c98ff9640 1 -- 172.21.15.148:0/2380837081 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94003980 con 0x7f1c9c1538a0 2023-10-30T13:36:15.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1c98ff9640 1 -- 172.21.15.148:0/2380837081 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c94003ae0 con 0x7f1c9c1538a0 2023-10-30T13:36:15.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9c1e51e0 con 0x7f1c9c1538a0 2023-10-30T13:36:15.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1c98ff9640 1 -- 172.21.15.148:0/2380837081 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94003e90 con 0x7f1c9c1538a0 2023-10-30T13:36:15.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9c1e5610 con 0x7f1c9c1538a0 2023-10-30T13:36:15.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.767+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9c0b2a50 con 0x7f1c9c1538a0 2023-10-30T13:36:15.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.771+0000 7f1c98ff9640 1 -- 172.21.15.148:0/2380837081 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c9400a850 con 0x7f1c9c1538a0 2023-10-30T13:36:15.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.771+0000 7f1c98ff9640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c880664d0 0x7f1c88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:36:15.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.771+0000 7f1c98ff9640 1 -- 172.21.15.148:0/2380837081 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f1c94083fb0 con 0x7f1c9c1538a0 2023-10-30T13:36:15.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.771+0000 7f1c9bfff640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c880664d0 0x7f1c88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:36:15.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.771+0000 7f1c9bfff640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c880664d0 0x7f1c88068990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0cf070 tx=0x7f1c8c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:36:15.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:15.775+0000 7f1c98ff9640 1 -- 172.21.15.148:0/2380837081 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c94050fe0 con 0x7f1c9c1538a0 2023-10-30T13:36:15.868 INFO:tasks.ceph.mds.c.smithi148.stderr:2023-10-30T13:36:15.871+0000 7feadc30f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:36:15.983 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 52% (86/163) Updating files: 53% (87/163) Updating files: 54% (89/163) Updating files: 55% (90/163) Updating files: 56% (92/163) Updating files: 57% (93/163) Updating files: 57% (94/163) Updating files: 58% (95/163) Updating files: 59% (97/163) Updating files: 60% (98/163) Updating files: 61% (100/163) Updating files: 62% (102/163) Updating files: 63% (103/163) Updating files: 64% (105/163) Updating files: 65% (106/163) Updating files: 66% (108/163) Updating files: 67% (110/163) Updating files: 68% (111/163) Updating files: 69% (113/163) Updating files: 70% (115/163) Updating files: 71% (116/163) Updating files: 72% (118/163) Updating files: 73% (119/163) Updating files: 74% (121/163) Updating files: 75% (123/163) Updating files: 76% (124/163) Updating files: 77% (126/163) Updating files: 78% (128/163) Updating files: 79% (129/163) Updating files: 80% (131/163) Updating files: 81% (133/163) Updating files: 82% (134/163) Updating files: 83% (136/163) Updating files: 84% (137/163) Updating files: 85% (139/163) Updating files: 86% (141/163) Updating files: 87% (142/163) Updating files: 88% (144/163) Updating files: 89% (146/163) Updating files: 90% (147/163) Updating files: 91% (149/163) Updating files: 92% (150/163) Updating files: 93% (152/163) Updating files: 94% (154/163) Updating files: 95% (155/163) Updating files: 96% (157/163) Updating files: 97% (159/163) Updating files: 98% (160/163) Updating files: 99% (162/163) Updating files: 100% (163/163) Updating files: 100% (163/163), done. 2023-10-30T13:36:16.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.015+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c9c158790 con 0x7f1c9c1538a0 2023-10-30T13:36:16.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.015+0000 7f1c98ff9640 1 -- 172.21.15.148:0/2380837081 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f1c94054600 con 0x7f1c9c1538a0 2023-10-30T13:36:16.013 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:36:16.013 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:36:16.014 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:36:16.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c880664d0 msgr2=0x7f1c88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:16.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c880664d0 0x7f1c88068990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c0cf070 tx=0x7f1c8c009190 comp rx=0 tx=0).stop 2023-10-30T13:36:16.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c1538a0 msgr2=0x7f1c9c0cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:36:16.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c1538a0 0x7f1c9c0cd4f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400c6f0 tx=0x7f1c9400cbb0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 shutdown_connections 2023-10-30T13:36:16.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c880664d0 0x7f1c88068990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c1538a0 0x7f1c9c0cd4f0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c9c0cf160 0x7f1c9c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 --2- 172.21.15.148:0/2380837081 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c9c0ca0e0 0x7f1c9c140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:36:16.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 >> 172.21.15.148:0/2380837081 conn(0x7f1c9c06bfe0 msgr2=0x7f1c9c0c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:36:16.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 shutdown_connections 2023-10-30T13:36:16.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:36:16.019+0000 7f1ca25ab640 1 -- 172.21.15.148:0/2380837081 wait complete. 2023-10-30T13:36:16.039 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:36:16.039 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 59 secs before thrashing 2023-10-30T13:36:16.070 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 5d1d8d0c460 v0.23 2023-10-30T13:36:16.075 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.23 2023-10-30T13:36:16.090 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 24 -eq 48 ] 2023-10-30T13:36:16.090 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.24 2023-10-30T13:36:16.091 INFO:tasks.workunit.client.0.smithi148.stdout:v0.24 2023-10-30T13:36:16.091 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.24 2023-10-30T13:36:16.091 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.24 2023-10-30T13:36:23.713 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 6% (12/185) Updating files: 7% (13/185) Updating files: 8% (15/185) Updating files: 9% (17/185) Updating files: 10% (19/185) Updating files: 11% (21/185) Updating files: 12% (23/185) Updating files: 13% (25/185) Updating files: 14% (26/185) Updating files: 15% (28/185) Updating files: 16% (30/185) Updating files: 17% (32/185) Updating files: 18% (34/185) Updating files: 19% (36/185) Updating files: 20% (37/185) Updating files: 21% (39/185) Updating files: 22% (41/185) Updating files: 23% (43/185) Updating files: 24% (45/185) Updating files: 25% (47/185) Updating files: 26% (49/185) Updating files: 27% (50/185) Updating files: 28% (52/185) Updating files: 29% (54/185) Updating files: 30% (56/185) Updating files: 31% (58/185) Updating files: 32% (60/185) Updating files: 33% (62/185) Updating files: 34% (63/185) Updating files: 35% (65/185) Updating files: 36% (67/185) Updating files: 37% (69/185) Updating files: 38% (71/185) Updating files: 39% (73/185) Updating files: 40% (74/185) Updating files: 41% (76/185) Updating files: 42% (78/185) Updating files: 43% (80/185) Updating files: 44% (82/185) Updating files: 45% (84/185) Updating files: 46% (86/185) Updating files: 47% (87/185) Updating files: 48% (89/185) Updating files: 49% (91/185) Updating files: 50% (93/185) Updating files: 51% (95/185) Updating files: 52% (97/185) Updating files: 53% (99/185) Updating files: 54% (100/185) Updating files: 55% (102/185) Updating files: 56% (104/185) Updating files: 57% (106/185) Updating files: 58% (108/185) Updating files: 59% (110/185) Updating files: 60% (111/185) Updating files: 61% (113/185) Updating files: 62% (115/185) Updating files: 63% (117/185) Updating files: 64% (119/185) Updating files: 65% (121/185) Updating files: 66% (123/185) Updating files: 67% (124/185) Updating files: 68% (126/185) Updating files: 69% (128/185) Updating files: 70% (130/185) Updating files: 71% (132/185) Updating files: 72% (134/185) Updating files: 73% (136/185) Updating files: 74% (137/185) Updating files: 75% (139/185) Updating files: 76% (141/185) Updating files: 77% (143/185) Updating files: 78% (145/185) Updating files: 79% (147/185) Updating files: 80% (148/185) Updating files: 81% (150/185) Updating files: 82% (152/185) Updating files: 83% (154/185) Updating files: 84% (156/185) Updating files: 85% (158/185) Updating files: 86% (160/185) Updating files: 87% (161/185) Updating files: 88% (163/185) Updating files: 89% (165/185) Updating files: 90% (167/185) Updating files: 91% (169/185) Updating files: 92% (171/185) Updating files: 93% (173/185) Updating files: 94% (174/185) Updating files: 95% (176/185) Updating files: 96% (178/185) Updating files: 97% (180/185) Updating files: 98% (182/185) Updating files: 99% (184/185) Updating files: 100% (185/185) Updating files: 100% (185/185), done. 2023-10-30T13:36:23.743 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 180a4176035 v0.24 2023-10-30T13:36:23.747 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.24 2023-10-30T13:36:23.800 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 25 -eq 48 ] 2023-10-30T13:36:23.800 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.25 2023-10-30T13:36:23.800 INFO:tasks.workunit.client.0.smithi148.stdout:v0.25 2023-10-30T13:36:23.801 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.25 2023-10-30T13:36:23.801 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.25 2023-10-30T13:36:30.500 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:30.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:36:30.758 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 9% (60/622) Updating files: 10% (63/622) Updating files: 11% (69/622) Updating files: 12% (75/622) Updating files: 13% (81/622) Updating files: 14% (88/622) Updating files: 15% (94/622) Updating files: 16% (100/622) Updating files: 17% (106/622) Updating files: 18% (112/622) Updating files: 18% (116/622) Updating files: 19% (119/622) Updating files: 20% (125/622) Updating files: 21% (131/622) Updating files: 22% (137/622) Updating files: 23% (144/622) Updating files: 24% (150/622) Updating files: 25% (156/622) Updating files: 25% (158/622) Updating files: 26% (162/622) Updating files: 27% (168/622) Updating files: 28% (175/622) Updating files: 29% (181/622) Updating files: 30% (187/622) Updating files: 31% (193/622) Updating files: 32% (200/622) Updating files: 33% (206/622) Updating files: 34% (212/622) Updating files: 35% (218/622) Updating files: 36% (224/622) Updating files: 37% (231/622) Updating files: 38% (237/622) Updating files: 39% (243/622) Updating files: 40% (249/622) Updating files: 41% (256/622) Updating files: 42% (262/622) Updating files: 43% (268/622) Updating files: 43% (271/622) Updating files: 44% (274/622) Updating files: 45% (280/622) Updating files: 46% (287/622) Updating files: 47% (293/622) Updating files: 48% (299/622) Updating files: 49% (305/622) Updating files: 50% (311/622) Updating files: 51% (318/622) Updating files: 52% (324/622) Updating files: 53% (330/622) Updating files: 54% (336/622) Updating files: 55% (343/622) Updating files: 56% (349/622) Updating files: 57% (355/622) Updating files: 58% (361/622) Updating files: 59% (367/622) Updating files: 60% (374/622) Updating files: 61% (380/622) Updating files: 62% (386/622) Updating files: 62% (391/622) Updating files: 63% (392/622) Updating files: 64% (399/622) Updating files: 65% (405/622) Updating files: 66% (411/622) Updating files: 67% (417/622) Updating files: 68% (423/622) Updating files: 69% (430/622) Updating files: 70% (436/622) Updating files: 71% (442/622) Updating files: 72% (448/622) Updating files: 72% (454/622) Updating files: 73% (455/622) Updating files: 74% (461/622) Updating files: 75% (467/622) Updating files: 76% (473/622) Updating files: 77% (479/622) Updating files: 78% (486/622) Updating files: 79% (492/622) Updating files: 80% (498/622) Updating files: 81% (504/622) Updating files: 82% (511/622) Updating files: 83% (517/622) Updating files: 84% (523/622) Updating files: 85% (529/622) Updating files: 86% (535/622) Updating files: 87% (542/622) Updating files: 88% (548/622) Updating files: 89% (554/622) Updating files: 90% (560/622) Updating files: 91% (567/622) Updating files: 92% (573/622) Updating files: 93% (579/622) Updating files: 94% (585/622) Updating files: 95% (591/622) Updating files: 96% (598/622) Updating files: 97% (604/622) Updating files: 98% (610/622) Updating files: 99% (616/622) Updating files: 100% (622/622) Updating files: 100% (622/622), done. 2023-10-30T13:36:30.776 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 3fa121bf0a8 v0.25 2023-10-30T13:36:30.781 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.25 2023-10-30T13:36:30.913 INFO:tasks.workunit.client.0.smithi148.stdout:v0.26 2023-10-30T13:36:30.914 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 26 -eq 48 ] 2023-10-30T13:36:30.914 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.26 2023-10-30T13:36:30.914 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.26 2023-10-30T13:36:30.914 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.26 2023-10-30T13:36:34.064 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 87% (209/239) Updating files: 88% (211/239) Updating files: 89% (213/239) Updating files: 90% (216/239) Updating files: 91% (218/239) Updating files: 92% (220/239) Updating files: 93% (223/239) Updating files: 94% (225/239) Updating files: 94% (226/239) Updating files: 95% (228/239) Updating files: 96% (230/239) Updating files: 97% (232/239) Updating files: 98% (235/239) Updating files: 99% (237/239) Updating files: 100% (239/239) Updating files: 100% (239/239), done. 2023-10-30T13:36:34.099 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 9981ff90968 v0.26 2023-10-30T13:36:34.104 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.26 2023-10-30T13:36:34.157 INFO:tasks.workunit.client.0.smithi148.stdout:v0.27 2023-10-30T13:36:34.157 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 27 -eq 48 ] 2023-10-30T13:36:34.158 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.27 2023-10-30T13:36:34.158 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.27 2023-10-30T13:36:34.158 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.27 2023-10-30T13:36:49.575 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 9% (24/253) Updating files: 10% (26/253) Updating files: 11% (28/253) Updating files: 12% (31/253) Updating files: 13% (33/253) Updating files: 14% (36/253) Updating files: 15% (38/253) Updating files: 16% (41/253) Updating files: 17% (44/253) Updating files: 18% (46/253) Updating files: 19% (49/253) Updating files: 20% (51/253) Updating files: 21% (54/253) Updating files: 22% (56/253) Updating files: 23% (59/253) Updating files: 24% (61/253) Updating files: 25% (64/253) Updating files: 26% (66/253) Updating files: 27% (69/253) Updating files: 28% (71/253) Updating files: 29% (74/253) Updating files: 30% (76/253) Updating files: 31% (79/253) Updating files: 32% (81/253) Updating files: 33% (84/253) Updating files: 34% (87/253) Updating files: 35% (89/253) Updating files: 36% (92/253) Updating files: 37% (94/253) Updating files: 38% (97/253) Updating files: 39% (99/253) Updating files: 40% (102/253) Updating files: 41% (104/253) Updating files: 42% (107/253) Updating files: 43% (109/253) Updating files: 44% (112/253) Updating files: 45% (114/253) Updating files: 46% (117/253) Updating files: 47% (119/253) Updating files: 48% (122/253) Updating files: 49% (124/253) Updating files: 50% (127/253) Updating files: 51% (130/253) Updating files: 52% (132/253) Updating files: 53% (135/253) Updating files: 54% (137/253) Updating files: 55% (140/253) Updating files: 56% (142/253) Updating files: 57% (145/253) Updating files: 58% (147/253) Updating files: 59% (150/253) Updating files: 60% (152/253) Updating files: 61% (155/253) Updating files: 62% (157/253) Updating files: 63% (160/253) Updating files: 64% (162/253) Updating files: 65% (165/253) Updating files: 66% (167/253) Updating files: 67% (170/253) Updating files: 68% (173/253) Updating files: 69% (175/253) Updating files: 70% (178/253) Updating files: 71% (180/253) Updating files: 72% (183/253) Updating files: 73% (185/253) Updating files: 74% (188/253) Updating files: 75% (190/253) Updating files: 76% (193/253) Updating files: 77% (195/253) Updating files: 78% (198/253) Updating files: 78% (199/253) Updating files: 79% (200/253) Updating files: 80% (203/253) Updating files: 81% (205/253) Updating files: 82% (208/253) Updating files: 83% (210/253) Updating files: 84% (213/253) Updating files: 85% (216/253) Updating files: 86% (218/253) Updating files: 87% (221/253) Updating files: 88% (223/253) Updating files: 89% (226/253) Updating files: 90% (228/253) Updating files: 91% (231/253) Updating files: 92% (233/253) Updating files: 93% (236/253) Updating files: 94% (238/253) Updating files: 95% (241/253) Updating files: 96% (243/253) Updating files: 97% (246/253) Updating files: 98% (248/253) Updating files: 99% (251/253) Updating files: 100% (253/253) Updating files: 100% (253/253), done. 2023-10-30T13:36:49.607 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 793034c62c8 v0.27 2023-10-30T13:36:49.611 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.27 2023-10-30T13:36:49.813 INFO:tasks.workunit.client.0.smithi148.stdout:v0.28 2023-10-30T13:36:49.813 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 28 -eq 48 ] 2023-10-30T13:36:49.813 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.28 2023-10-30T13:36:49.814 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.28 2023-10-30T13:36:49.814 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.28 2023-10-30T13:36:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 24% (56/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 59% (135/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-10-30T13:36:54.132 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 071881d7e55 v0.28 2023-10-30T13:36:54.138 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.28 2023-10-30T13:36:54.241 INFO:tasks.workunit.client.0.smithi148.stdout:v0.29 2023-10-30T13:36:54.241 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 29 -eq 48 ] 2023-10-30T13:36:54.241 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.29 2023-10-30T13:36:54.241 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.29 2023-10-30T13:36:54.242 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.29 2023-10-30T13:36:55.645 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 74% (118/159) Updating files: 75% (120/159) Updating files: 76% (121/159) Updating files: 77% (123/159) Updating files: 78% (125/159) Updating files: 79% (126/159) Updating files: 80% (128/159) Updating files: 81% (129/159) Updating files: 82% (131/159) Updating files: 83% (132/159) Updating files: 84% (134/159) Updating files: 85% (136/159) Updating files: 86% (137/159) Updating files: 87% (139/159) Updating files: 88% (140/159) Updating files: 89% (142/159) Updating files: 90% (144/159) Updating files: 91% (145/159) Updating files: 92% (147/159) Updating files: 93% (148/159) Updating files: 94% (150/159) Updating files: 95% (152/159) Updating files: 96% (153/159) Updating files: 97% (155/159) Updating files: 98% (156/159) Updating files: 99% (158/159) Updating files: 100% (159/159) Updating files: 100% (159/159), done. 2023-10-30T13:36:56.583 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 8e69c39f699 v0.29 2023-10-30T13:36:56.588 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.29 2023-10-30T13:36:56.660 INFO:tasks.workunit.client.0.smithi148.stdout:v0.30 2023-10-30T13:36:56.660 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 30 -eq 48 ] 2023-10-30T13:36:56.660 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.30 2023-10-30T13:36:56.661 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.30 2023-10-30T13:36:56.661 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.30 2023-10-30T13:37:00.565 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:00.569 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:15.040 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:15.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.189+0000 7f9172a09640 1 Processor -- start 2023-10-30T13:37:15.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.189+0000 7f9172a09640 1 -- start start 2023-10-30T13:37:15.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f9172a09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c06f7a0 0x7f916c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f9172a09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f916c071110 0x7f916c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f9172a09640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c066cf0 con 0x7f916c0c6bd0 2023-10-30T13:37:15.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f9172a09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c0700c0 con 0x7f916c071110 2023-10-30T13:37:15.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f9172a09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0c70d0 con 0x7f916c06f7a0 2023-10-30T13:37:15.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f9170fa6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f916c071110 0x7f916c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f916bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c06f7a0 0x7f916c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f916b7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f916c0c6bd0 0x7f916c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:41172/0 (socket says 172.21.15.148:41172) 2023-10-30T13:37:15.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f9170fa6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f916c071110 0x7f916c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:56388/0 (socket says 172.21.15.148:56388) 2023-10-30T13:37:15.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f916b7fe640 1 -- 172.21.15.148:0/1900018254 learned_addr learned my addr 172.21.15.148:0/1900018254 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:15.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f916affd640 1 -- 172.21.15.148:0/1900018254 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305740351 0 0) 0x7f916c066cf0 con 0x7f916c0c6bd0 2023-10-30T13:37:15.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f916affd640 1 -- 172.21.15.148:0/1900018254 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9154003530 con 0x7f916c0c6bd0 2023-10-30T13:37:15.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f916bfff640 1 -- 172.21.15.148:0/1900018254 >> v2:172.21.15.150:3300/0 conn(0x7f916c071110 msgr2=0x7f916c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:15.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f916bfff640 1 --2- 172.21.15.148:0/1900018254 >> v2:172.21.15.150:3300/0 conn(0x7f916c071110 0x7f916c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f916bfff640 1 -- 172.21.15.148:0/1900018254 >> v1:172.21.15.150:6789/0 conn(0x7f916c0c6bd0 legacy=0x7f916c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f916bfff640 1 -- 172.21.15.148:0/1900018254 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0c7390 con 0x7f916c06f7a0 2023-10-30T13:37:15.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.193+0000 7f9170fa6640 1 --2- 172.21.15.148:0/1900018254 >> v2:172.21.15.150:3300/0 conn(0x7f916c071110 0x7f916c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:37:15.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f916bfff640 1 --2- 172.21.15.148:0/1900018254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c06f7a0 0x7f916c06fb80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f915c0097c0 tx=0x7f915c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daf6d05097f51b82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:15.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f916affd640 1 -- 172.21.15.148:0/1900018254 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c0a8970 con 0x7f916c06f7a0 2023-10-30T13:37:15.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f916affd640 1 -- 172.21.15.148:0/1900018254 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c0a8ad0 con 0x7f916c06f7a0 2023-10-30T13:37:15.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f916affd640 1 -- 172.21.15.148:0/1900018254 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c0a8e60 con 0x7f916c06f7a0 2023-10-30T13:37:15.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f9172a09640 1 -- 172.21.15.148:0/1900018254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c06f7a0 msgr2=0x7f916c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f9172a09640 1 --2- 172.21.15.148:0/1900018254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c06f7a0 0x7f916c06fb80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f915c0097c0 tx=0x7f915c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:37:15.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f9172a09640 1 -- 172.21.15.148:0/1900018254 shutdown_connections 2023-10-30T13:37:15.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f9172a09640 1 --2- 172.21.15.148:0/1900018254 >> v2:172.21.15.150:3300/0 conn(0x7f916c071110 0x7f916c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f9172a09640 1 --2- 172.21.15.148:0/1900018254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c06f7a0 0x7f916c06fb80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f9172a09640 1 -- 172.21.15.148:0/1900018254 >> 172.21.15.148:0/1900018254 conn(0x7f916c0c7910 msgr2=0x7f916c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:15.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f9172a09640 1 -- 172.21.15.148:0/1900018254 shutdown_connections 2023-10-30T13:37:15.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.197+0000 7f9172a09640 1 -- 172.21.15.148:0/1900018254 wait complete. 2023-10-30T13:37:15.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 Processor -- start 2023-10-30T13:37:15.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 -- start start 2023-10-30T13:37:15.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c06f7a0 0x7f916c156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c071110 0x7f916c14e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f916c0c6bd0 0x7f916c14ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c14d7a0 con 0x7f916c06f7a0 2023-10-30T13:37:15.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c1571f0 con 0x7f916c0c6bd0 2023-10-30T13:37:15.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c06ad30 con 0x7f916c071110 2023-10-30T13:37:15.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c06f7a0 0x7f916c156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c06f7a0 0x7f916c156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45466/0 (socket says 172.21.15.148:45466) 2023-10-30T13:37:15.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916bfff640 1 -- 172.21.15.148:0/2651088231 learned_addr learned my addr 172.21.15.148:0/2651088231 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:15.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9170fa6640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f916c0c6bd0 0x7f916c14ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916b7fe640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c071110 0x7f916c14e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916bfff640 1 -- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c071110 msgr2=0x7f916c14e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916bfff640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c071110 0x7f916c14e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916bfff640 1 -- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f916c0c6bd0 msgr2=0x7f916c14ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916bfff640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f916c0c6bd0 0x7f916c14ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916bfff640 1 -- 172.21.15.148:0/2651088231 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9164009a70 con 0x7f916c06f7a0 2023-10-30T13:37:15.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f916bfff640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c06f7a0 0x7f916c156cb0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f915c0b3040 tx=0x7f915c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:15.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9168ff9640 1 -- 172.21.15.148:0/2651088231 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c0b0d60 con 0x7f916c06f7a0 2023-10-30T13:37:15.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9168ff9640 1 -- 172.21.15.148:0/2651088231 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c0b9430 con 0x7f916c06f7a0 2023-10-30T13:37:15.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9168ff9640 1 -- 172.21.15.148:0/2651088231 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c0b97e0 con 0x7f916c06f7a0 2023-10-30T13:37:15.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915c009420 con 0x7f916c06f7a0 2023-10-30T13:37:15.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c1ef5f0 con 0x7f916c06f7a0 2023-10-30T13:37:15.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.201+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916c14e340 con 0x7f916c06f7a0 2023-10-30T13:37:15.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.205+0000 7f9168ff9640 1 -- 172.21.15.148:0/2651088231 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f915c0b7040 con 0x7f916c06f7a0 2023-10-30T13:37:15.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.205+0000 7f9168ff9640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91580664a0 0x7f9158068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.205+0000 7f9168ff9640 1 -- 172.21.15.148:0/2651088231 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f915c121d80 con 0x7f916c06f7a0 2023-10-30T13:37:15.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.205+0000 7f916b7fe640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91580664a0 0x7f9158068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.205+0000 7f916b7fe640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91580664a0 0x7f9158068960 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f916c072380 tx=0x7f9160009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:15.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.209+0000 7f9168ff9640 1 -- 172.21.15.148:0/2651088231 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f915c0eed00 con 0x7f916c06f7a0 2023-10-30T13:37:15.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f916c04e170 con 0x7f916c06f7a0 2023-10-30T13:37:15.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9168ff9640 1 -- 172.21.15.148:0/2651088231 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f915c0b5020 con 0x7f916c06f7a0 2023-10-30T13:37:15.397 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:15.398 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:15.398 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:15.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91580664a0 msgr2=0x7f9158068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91580664a0 0x7f9158068960 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f916c072380 tx=0x7f9160009040 comp rx=0 tx=0).stop 2023-10-30T13:37:15.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c06f7a0 msgr2=0x7f916c156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c06f7a0 0x7f916c156cb0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f915c0b3040 tx=0x7f915c01ba10 comp rx=0 tx=0).stop 2023-10-30T13:37:15.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 shutdown_connections 2023-10-30T13:37:15.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f91580664a0 0x7f9158068960 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f916c0c6bd0 0x7f916c14ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f916c071110 0x7f916c14e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 --2- 172.21.15.148:0/2651088231 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f916c06f7a0 0x7f916c156cb0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 >> 172.21.15.148:0/2651088231 conn(0x7f916c0c7910 msgr2=0x7f916c06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:15.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 shutdown_connections 2023-10-30T13:37:15.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.401+0000 7f9172a09640 1 -- 172.21.15.148:0/2651088231 wait complete. 2023-10-30T13:37:15.419 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:37:15.419 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:37:15.419 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:37:15.443 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:37:15.443 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:15.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7e1c54640 1 Processor -- start 2023-10-30T13:37:15.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7e1c54640 1 -- start start 2023-10-30T13:37:15.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7e1c54640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc7dc0c6bd0 0x7fc7dc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7e1c54640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc7dc071110 0x7fc7dc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7e1c54640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc066cf0 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7e1c54640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7dc0700c0 con 0x7fc7dc071110 2023-10-30T13:37:15.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7e1c54640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7dc0c70d0 con 0x7fc7dc0c6bd0 2023-10-30T13:37:15.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7dbfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc7dc071110 0x7fc7dc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7dbfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc7dc071110 0x7fc7dc158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45478/0 (socket says 172.21.15.148:45478) 2023-10-30T13:37:15.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7dbfff640 1 -- 172.21.15.148:0/168021628 learned_addr learned my addr 172.21.15.148:0/168021628 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:15.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7daffd640 1 --2- 172.21.15.148:0/168021628 >> v2:172.21.15.150:3300/0 conn(0x7fc7dc0c6bd0 0x7fc7dc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7dbfff640 1 -- 172.21.15.148:0/168021628 >> v2:172.21.15.150:3300/0 conn(0x7fc7dc0c6bd0 msgr2=0x7fc7dc06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:15.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7dbfff640 1 --2- 172.21.15.148:0/168021628 >> v2:172.21.15.150:3300/0 conn(0x7fc7dc0c6bd0 0x7fc7dc06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7dbfff640 1 -- 172.21.15.148:0/168021628 >> v1:172.21.15.150:6789/0 conn(0x7fc7dc06f7a0 legacy=0x7fc7dc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.633+0000 7fc7dbfff640 1 -- 172.21.15.148:0/168021628 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0c7390 con 0x7fc7dc071110 2023-10-30T13:37:15.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7dbfff640 1 --2- 172.21.15.148:0/168021628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc071110 0x7fc7dc158110 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc06f1c0 tx=0x7fc7d401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa9eaf0a85799ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:15.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7da7fc640 1 -- 172.21.15.148:0/168021628 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40af400 con 0x7fc7dc071110 2023-10-30T13:37:15.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7da7fc640 1 -- 172.21.15.148:0/168021628 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d40ab050 con 0x7fc7dc071110 2023-10-30T13:37:15.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7da7fc640 1 -- 172.21.15.148:0/168021628 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40af780 con 0x7fc7dc071110 2023-10-30T13:37:15.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7e1c54640 1 -- 172.21.15.148:0/168021628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc071110 msgr2=0x7fc7dc158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7e1c54640 1 --2- 172.21.15.148:0/168021628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc071110 0x7fc7dc158110 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc06f1c0 tx=0x7fc7d401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7e1c54640 1 -- 172.21.15.148:0/168021628 shutdown_connections 2023-10-30T13:37:15.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7e1c54640 1 --2- 172.21.15.148:0/168021628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc071110 0x7fc7dc158110 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7e1c54640 1 --2- 172.21.15.148:0/168021628 >> v2:172.21.15.150:3300/0 conn(0x7fc7dc0c6bd0 0x7fc7dc06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7e1c54640 1 -- 172.21.15.148:0/168021628 >> 172.21.15.148:0/168021628 conn(0x7fc7dc0c7910 msgr2=0x7fc7dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7e1c54640 1 -- 172.21.15.148:0/168021628 shutdown_connections 2023-10-30T13:37:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.637+0000 7fc7e1c54640 1 -- 172.21.15.148:0/168021628 wait complete. 2023-10-30T13:37:15.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 Processor -- start 2023-10-30T13:37:15.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 -- start start 2023-10-30T13:37:15.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc06f7a0 0x7fc7dc1e2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7dc071110 0x7fc7dc1eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7dc0c6bd0 0x7fc7dc1ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc1e36e0 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc0c0670 con 0x7fc7dc071110 2023-10-30T13:37:15.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc0c0950 con 0x7fc7dc0c6bd0 2023-10-30T13:37:15.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7db7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc06f7a0 0x7fc7dc1e2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7db7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc06f7a0 0x7fc7dc1e2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45488/0 (socket says 172.21.15.148:45488) 2023-10-30T13:37:15.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7db7fe640 1 -- 172.21.15.148:0/3827244297 learned_addr learned my addr 172.21.15.148:0/3827244297 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:15.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7daffd640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7dc071110 0x7fc7dc1eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7dbfff640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7dc0c6bd0 0x7fc7dc1ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7db7fe640 1 -- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7dc0c6bd0 msgr2=0x7fc7dc1ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7db7fe640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7dc0c6bd0 0x7fc7dc1ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7db7fe640 1 -- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7dc071110 msgr2=0x7fc7dc1eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7db7fe640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7dc071110 0x7fc7dc1eeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7db7fe640 1 -- 172.21.15.148:0/3827244297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4009b30 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7db7fe640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc06f7a0 0x7fc7dc1e2690 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00c3c0 tx=0x7fc7cc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:15.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7c3fff640 1 -- 172.21.15.148:0/3827244297 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0052d0 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7c3fff640 1 -- 172.21.15.148:0/3827244297 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc005430 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7c3fff640 1 -- 172.21.15.148:0/3827244297 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc003840 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc0c0c30 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc0c11f0 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.641+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc0c1410 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.645+0000 7fc7c3fff640 1 -- 172.21.15.148:0/3827244297 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7cc0039a0 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.645+0000 7fc7c3fff640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7c80664d0 0x7fc7c8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:15.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.645+0000 7fc7c3fff640 1 -- 172.21.15.148:0/3827244297 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fc7cc0838f0 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.645+0000 7fc7daffd640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7c80664d0 0x7fc7c8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:15.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.645+0000 7fc7daffd640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7c80664d0 0x7fc7c8068990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0c7000 tx=0x7fc7d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:15.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.649+0000 7fc7c3fff640 1 -- 172.21.15.148:0/3827244297 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7cc0507e0 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.833+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7dc04e170 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.833+0000 7fc7c3fff640 1 -- 172.21.15.148:0/3827244297 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7fc7cc053e00 con 0x7fc7dc06f7a0 2023-10-30T13:37:15.831 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:15.831 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:15.831 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:15.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7c80664d0 msgr2=0x7fc7c8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7c80664d0 0x7fc7c8068990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0c7000 tx=0x7fc7d0008040 comp rx=0 tx=0).stop 2023-10-30T13:37:15.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc06f7a0 msgr2=0x7fc7dc1e2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:15.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc06f7a0 0x7fc7dc1e2690 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00c3c0 tx=0x7fc7cc00c880 comp rx=0 tx=0).stop 2023-10-30T13:37:15.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 shutdown_connections 2023-10-30T13:37:15.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc7c80664d0 0x7fc7c8068990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc7dc0c6bd0 0x7fc7dc1ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7dc071110 0x7fc7dc1eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 --2- 172.21.15.148:0/3827244297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7dc06f7a0 0x7fc7dc1e2690 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:15.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 >> 172.21.15.148:0/3827244297 conn(0x7fc7dc0c7910 msgr2=0x7fc7dc0c8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:15.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 shutdown_connections 2023-10-30T13:37:15.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:15.837+0000 7fc7e1c54640 1 -- 172.21.15.148:0/3827244297 wait complete. 2023-10-30T13:37:15.853 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:37:17.853 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:17.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10b1605640 1 Processor -- start 2023-10-30T13:37:17.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10b1605640 1 -- start start 2023-10-30T13:37:17.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10b1605640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10ac1538a0 0x7f10ac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:17.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10b1605640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 0x7f10ac15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:18.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10b1605640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ac0cb800 con 0x7f10ac0cf160 2023-10-30T13:37:18.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10b1605640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ac15dc30 con 0x7f10ac1538a0 2023-10-30T13:37:18.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10b1605640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0cb330 con 0x7f10ac0ca0e0 2023-10-30T13:37:18.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10ab7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f10ac0cf160 0x7f10ac0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47826/0 (socket says 172.21.15.148:47826) 2023-10-30T13:37:18.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10ab7fe640 1 -- 172.21.15.148:0/2750983163 learned_addr learned my addr 172.21.15.148:0/2750983163 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:18.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10aaffd640 1 --2- 172.21.15.148:0/2750983163 >> v2:172.21.15.150:3300/0 conn(0x7f10ac1538a0 0x7f10ac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:18.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10a27fc640 1 --2- 172.21.15.148:0/2750983163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 0x7f10ac15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:18.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10aa7fc640 1 -- 172.21.15.148:0/2750983163 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291041039 0 0) 0x7f10ac0cb800 con 0x7f10ac0cf160 2023-10-30T13:37:18.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10aa7fc640 1 -- 172.21.15.148:0/2750983163 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1098003530 con 0x7f10ac0cf160 2023-10-30T13:37:18.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10aa7fc640 1 -- 172.21.15.148:0/2750983163 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348556726 0 0) 0x7f1098003530 con 0x7f10ac0cf160 2023-10-30T13:37:18.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10aaffd640 1 -- 172.21.15.148:0/2750983163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 msgr2=0x7f10ac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:18.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10aaffd640 1 --2- 172.21.15.148:0/2750983163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 0x7f10ac15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:18.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10aaffd640 1 -- 172.21.15.148:0/2750983163 >> v1:172.21.15.148:6789/0 conn(0x7f10ac0cf160 legacy=0x7f10ac0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:18.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10aaffd640 1 -- 172.21.15.148:0/2750983163 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac0cad70 con 0x7f10ac1538a0 2023-10-30T13:37:18.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.001+0000 7f10a27fc640 1 --2- 172.21.15.148:0/2750983163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 0x7f10ac15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:37:18.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.005+0000 7f10aaffd640 1 --2- 172.21.15.148:0/2750983163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 0x7f10ac15d6f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f109c0097c0 tx=0x7f109c01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37a8fbd60fb80a40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:18.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.005+0000 7f10aa7fc640 1 -- 172.21.15.148:0/2750983163 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4001680 con 0x7f10ac1538a0 2023-10-30T13:37:18.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.005+0000 7f10aa7fc640 1 -- 172.21.15.148:0/2750983163 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c0a85d0 con 0x7f10ac1538a0 2023-10-30T13:37:18.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.005+0000 7f10aa7fc640 1 -- 172.21.15.148:0/2750983163 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c0a8940 con 0x7f10ac1538a0 2023-10-30T13:37:18.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.005+0000 7f10b1605640 1 -- 172.21.15.148:0/2750983163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 msgr2=0x7f10ac15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:18.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.005+0000 7f10b1605640 1 --2- 172.21.15.148:0/2750983163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 0x7f10ac15d6f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f109c0097c0 tx=0x7f109c01eb90 comp rx=0 tx=0).stop 2023-10-30T13:37:18.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 -- 172.21.15.148:0/2750983163 shutdown_connections 2023-10-30T13:37:18.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 --2- 172.21.15.148:0/2750983163 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 0x7f10ac15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:18.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 --2- 172.21.15.148:0/2750983163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 0x7f10ac15d6f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:18.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 -- 172.21.15.148:0/2750983163 >> 172.21.15.148:0/2750983163 conn(0x7f10ac06bfe0 msgr2=0x7f10ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:18.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 -- 172.21.15.148:0/2750983163 shutdown_connections 2023-10-30T13:37:18.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 -- 172.21.15.148:0/2750983163 wait complete. 2023-10-30T13:37:18.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 Processor -- start 2023-10-30T13:37:18.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 -- start start 2023-10-30T13:37:18.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 0x7f10ac0b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:18.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10ac0cf160 0x7f10ac144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:18.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 0x7f10ac1452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:18.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac145800 con 0x7f10ac0cf160 2023-10-30T13:37:18.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0ce820 con 0x7f10ac1538a0 2023-10-30T13:37:18.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10b1605640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0ceae0 con 0x7f10ac0ca0e0 2023-10-30T13:37:18.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10a27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10ac0cf160 0x7f10ac144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:18.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10a27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10ac0cf160 0x7f10ac144d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45496/0 (socket says 172.21.15.148:45496) 2023-10-30T13:37:18.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10a27fc640 1 -- 172.21.15.148:0/1450997561 learned_addr learned my addr 172.21.15.148:0/1450997561 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:18.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10aaffd640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 0x7f10ac0b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:18.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10ab7fe640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 0x7f10ac1452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:18.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10a27fc640 1 -- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 msgr2=0x7f10ac0b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:18.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10a27fc640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 0x7f10ac0b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10a27fc640 1 -- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 msgr2=0x7f10ac1452c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10a27fc640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 0x7f10ac1452c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10a27fc640 1 -- 172.21.15.148:0/1450997561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1094009a70 con 0x7f10ac0cf160 2023-10-30T13:37:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10ab7fe640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 0x7f10ac1452c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.009+0000 7f10a27fc640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10ac0cf160 0x7f10ac144d80 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f109400efc0 tx=0x7f109400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10a3fff640 1 -- 172.21.15.148:0/1450997561 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094014680 con 0x7f10ac0cf160 2023-10-30T13:37:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10a3fff640 1 -- 172.21.15.148:0/1450997561 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109400f040 con 0x7f10ac0cf160 2023-10-30T13:37:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109c009420 con 0x7f10ac0cf160 2023-10-30T13:37:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10a3fff640 1 -- 172.21.15.148:0/1450997561 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094014a00 con 0x7f10ac0cf160 2023-10-30T13:37:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ac0b3e80 con 0x7f10ac0cf160 2023-10-30T13:37:18.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ac0b2a50 con 0x7f10ac0cf160 2023-10-30T13:37:18.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10a3fff640 1 -- 172.21.15.148:0/1450997561 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1094014b60 con 0x7f10ac0cf160 2023-10-30T13:37:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10a3fff640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f108c0664a0 0x7f108c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10a3fff640 1 -- 172.21.15.148:0/1450997561 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f10940851d0 con 0x7f10ac0cf160 2023-10-30T13:37:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.013+0000 7f10aaffd640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f108c0664a0 0x7f108c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.017+0000 7f10aaffd640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f108c0664a0 0x7f108c068960 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f109c0b2000 tx=0x7f109c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.017+0000 7f10a3fff640 1 -- 172.21.15.148:0/1450997561 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10940521f0 con 0x7f10ac0cf160 2023-10-30T13:37:18.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.197+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10ac04e170 con 0x7f10ac0cf160 2023-10-30T13:37:18.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.197+0000 7f10a3fff640 1 -- 172.21.15.148:0/1450997561 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f1094055810 con 0x7f10ac0cf160 2023-10-30T13:37:18.194 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:18.195 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:18.195 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:18.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f108c0664a0 msgr2=0x7f108c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:18.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f108c0664a0 0x7f108c068960 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f109c0b2000 tx=0x7f109c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:37:18.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10ac0cf160 msgr2=0x7f10ac144d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:18.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10ac0cf160 0x7f10ac144d80 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f109400efc0 tx=0x7f109400c490 comp rx=0 tx=0).stop 2023-10-30T13:37:18.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 shutdown_connections 2023-10-30T13:37:18.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f108c0664a0 0x7f108c068960 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:18.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f10ac0cf160 0x7f10ac144d80 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:18.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1538a0 0x7f10ac1452c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:18.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 --2- 172.21.15.148:0/1450997561 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f10ac0ca0e0 0x7f10ac0b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:18.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 >> 172.21.15.148:0/1450997561 conn(0x7f10ac06bfe0 msgr2=0x7f10ac1526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:18.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 shutdown_connections 2023-10-30T13:37:18.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:18.201+0000 7f10b1605640 1 -- 172.21.15.148:0/1450997561 wait complete. 2023-10-30T13:37:18.216 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:37:20.217 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:20.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.369+0000 7f9da6846640 1 Processor -- start 2023-10-30T13:37:20.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.369+0000 7f9da6846640 1 -- start start 2023-10-30T13:37:20.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.369+0000 7f9da6846640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9da0070a00 0x7f9da0070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.369+0000 7f9da6846640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da00713b0 con 0x7f9da01539a0 2023-10-30T13:37:20.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.369+0000 7f9da6846640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da0153d80 con 0x7f9da0069e90 2023-10-30T13:37:20.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.369+0000 7f9da6846640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da006fbb0 con 0x7f9da0070a00 2023-10-30T13:37:20.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.369+0000 7f9d9ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9da01539a0 0x7f9da015a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47828/0 (socket says 172.21.15.148:47828) 2023-10-30T13:37:20.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.369+0000 7f9d9ffff640 1 -- 172.21.15.148:0/1008365541 learned_addr learned my addr 172.21.15.148:0/1008365541 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:20.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.369+0000 7f9d9f7fe640 1 --2- 172.21.15.148:0/1008365541 >> v2:172.21.15.150:3300/0 conn(0x7f9da0070a00 0x7f9da0070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651859802 0 0) 0x7f9da00713b0 con 0x7f9da01539a0 2023-10-30T13:37:20.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d88003530 con 0x7f9da01539a0 2023-10-30T13:37:20.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 149720941 0 0) 0x7f9d88003530 con 0x7f9da01539a0 2023-10-30T13:37:20.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 >> v2:172.21.15.150:3300/0 conn(0x7f9da0070a00 msgr2=0x7f9da0070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:20.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 --2- 172.21.15.148:0/1008365541 >> v2:172.21.15.150:3300/0 conn(0x7f9da0070a00 0x7f9da0070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 >> v1:172.21.15.150:6789/0 conn(0x7f9da0069e90 legacy=0x7f9da006a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da00c5c50 con 0x7f9da01539a0 2023-10-30T13:37:20.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f9d90001680 con 0x7f9da01539a0 2023-10-30T13:37:20.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 >> v1:172.21.15.148:6789/0 conn(0x7f9da01539a0 legacy=0x7f9da015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d88003980 0x7f9d88003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d880047d0 0x7f9d88004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d88005190 0x7f9d8800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da006fbb0 con 0x7f9d880047d0 2023-10-30T13:37:20.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8800c130 con 0x7f9d88003980 2023-10-30T13:37:20.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8800c2f0 con 0x7f9d88005190 2023-10-30T13:37:20.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9f7fe640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d880047d0 0x7f9d88004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9ffff640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d88003980 0x7f9d88003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9da4de3640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d88005190 0x7f9d8800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9f7fe640 1 -- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d88005190 msgr2=0x7f9d8800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:20.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9f7fe640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d88005190 0x7f9d8800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9f7fe640 1 -- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d88003980 msgr2=0x7f9d88003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:20.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9f7fe640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d88003980 0x7f9d88003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9f7fe640 1 -- 172.21.15.148:0/1008365541 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d88003530 con 0x7f9d880047d0 2023-10-30T13:37:20.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9da4de3640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9d88005190 0x7f9d8800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:20.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9d9ffff640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d88003980 0x7f9d88003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:20.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.373+0000 7f9da4de3640 1 -- 172.21.15.148:0/1008365541 reap_dead start 2023-10-30T13:37:20.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9d9f7fe640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d880047d0 0x7f9d88004c50 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009ae0 tx=0x7f9d9401ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2c9bfc64b932f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:20.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d940a6510 con 0x7f9d880047d0 2023-10-30T13:37:20.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d940a6670 con 0x7f9d880047d0 2023-10-30T13:37:20.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9d9effd640 1 -- 172.21.15.148:0/1008365541 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d940a6a20 con 0x7f9d880047d0 2023-10-30T13:37:20.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 -- 172.21.15.148:0/1008365541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d880047d0 msgr2=0x7f9d88004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d880047d0 0x7f9d88004c50 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009ae0 tx=0x7f9d9401ca20 comp rx=0 tx=0).stop 2023-10-30T13:37:20.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 -- 172.21.15.148:0/1008365541 shutdown_connections 2023-10-30T13:37:20.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 --2- 172.21.15.148:0/1008365541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9d880047d0 0x7f9d88004c50 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 -- 172.21.15.148:0/1008365541 >> 172.21.15.148:0/1008365541 conn(0x7f9da00c7910 msgr2=0x7f9da00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:20.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 -- 172.21.15.148:0/1008365541 shutdown_connections 2023-10-30T13:37:20.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 -- 172.21.15.148:0/1008365541 wait complete. 2023-10-30T13:37:20.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 Processor -- start 2023-10-30T13:37:20.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 -- start start 2023-10-30T13:37:20.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880047d0 0x7f9da00ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9da01e71a0 0x7f9da01f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9da01e75e0 0x7f9da01f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da0157a70 con 0x7f9da01e71a0 2023-10-30T13:37:20.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da0157540 con 0x7f9d880047d0 2023-10-30T13:37:20.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.377+0000 7f9da6846640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da0157790 con 0x7f9da01e75e0 2023-10-30T13:37:20.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9da01e71a0 0x7f9da01f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9da01e71a0 0x7f9da01f5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45514/0 (socket says 172.21.15.148:45514) 2023-10-30T13:37:20.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9f7fe640 1 -- 172.21.15.148:0/542293497 learned_addr learned my addr 172.21.15.148:0/542293497 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:20.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9ffff640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880047d0 0x7f9da00ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9da4de3640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9da01e75e0 0x7f9da01f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9f7fe640 1 -- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9da01e75e0 msgr2=0x7f9da01f6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9f7fe640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9da01e75e0 0x7f9da01f6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9f7fe640 1 -- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880047d0 msgr2=0x7f9da00ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9f7fe640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880047d0 0x7f9da00ca730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9f7fe640 1 -- 172.21.15.148:0/542293497 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da00c97c0 con 0x7f9da01e71a0 2023-10-30T13:37:20.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9f7fe640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9da01e71a0 0x7f9da01f5cc0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f9d940b0040 tx=0x7f9d88005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:20.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9cff9640 1 -- 172.21.15.148:0/542293497 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d940a69e0 con 0x7f9da01e71a0 2023-10-30T13:37:20.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9cff9640 1 -- 172.21.15.148:0/542293497 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d940a6b40 con 0x7f9da01e71a0 2023-10-30T13:37:20.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9cff9640 1 -- 172.21.15.148:0/542293497 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94006040 con 0x7f9da01e71a0 2023-10-30T13:37:20.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da00c9aa0 con 0x7f9da01e71a0 2023-10-30T13:37:20.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da015dd70 con 0x7f9da01e71a0 2023-10-30T13:37:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da00c9620 con 0x7f9da01e71a0 2023-10-30T13:37:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.381+0000 7f9d9cff9640 1 -- 172.21.15.148:0/542293497 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9d940061a0 con 0x7f9da01e71a0 2023-10-30T13:37:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.385+0000 7f9d9cff9640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9d8c0664d0 0x7f9d8c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.385+0000 7f9d9cff9640 1 -- 172.21.15.148:0/542293497 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f9d940aa070 con 0x7f9da01e71a0 2023-10-30T13:37:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.385+0000 7f9d9ffff640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9d8c0664d0 0x7f9d8c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:20.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.385+0000 7f9d9ffff640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9d8c0664d0 0x7f9d8c068990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9d900017b0 tx=0x7f9d90007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:20.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.389+0000 7f9d9cff9640 1 -- 172.21.15.148:0/542293497 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9da00c9620 con 0x7f9da01e71a0 2023-10-30T13:37:20.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.565+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9da00c9620 con 0x7f9da01e71a0 2023-10-30T13:37:20.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.565+0000 7f9d9cff9640 1 -- 172.21.15.148:0/542293497 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f9da00c9620 con 0x7f9da01e71a0 2023-10-30T13:37:20.564 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:20.564 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:20.564 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:20.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9d8c0664d0 msgr2=0x7f9d8c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9d8c0664d0 0x7f9d8c068990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9d900017b0 tx=0x7f9d90007040 comp rx=0 tx=0).stop 2023-10-30T13:37:20.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9da01e71a0 msgr2=0x7f9da01f5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:20.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9da01e71a0 0x7f9da01f5cc0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f9d940b0040 tx=0x7f9d88005800 comp rx=0 tx=0).stop 2023-10-30T13:37:20.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 shutdown_connections 2023-10-30T13:37:20.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9d8c0664d0 0x7f9d8c068990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9da01e71a0 0x7f9da01f5cc0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9da01e75e0 0x7f9da01f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 --2- 172.21.15.148:0/542293497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d880047d0 0x7f9da00ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:20.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 >> 172.21.15.148:0/542293497 conn(0x7f9da00c7910 msgr2=0x7f9da006b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:20.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 shutdown_connections 2023-10-30T13:37:20.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:20.569+0000 7f9da6846640 1 -- 172.21.15.148:0/542293497 wait complete. 2023-10-30T13:37:20.585 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:37:22.586 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:22.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.732+0000 7fde402f7640 1 Processor -- start 2023-10-30T13:37:22.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.732+0000 7fde402f7640 1 -- start start 2023-10-30T13:37:22.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde402f7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fde380c6b80 0x7fde380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde402f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde3813f4a0 0x7fde380c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde402f7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde380c0a50 con 0x7fde380c6f60 2023-10-30T13:37:22.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde402f7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde3806f6e0 con 0x7fde380c6b80 2023-10-30T13:37:22.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde402f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3806f9c0 con 0x7fde3813f4a0 2023-10-30T13:37:22.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d892640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fde380c6f60 0x7fde3813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46774/0 (socket says 172.21.15.148:46774) 2023-10-30T13:37:22.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d892640 1 -- 172.21.15.148:0/457537704 learned_addr learned my addr 172.21.15.148:0/457537704 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:22.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3e894640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde3813f4a0 0x7fde380c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3e093640 1 --2- 172.21.15.148:0/457537704 >> v2:172.21.15.150:3300/0 conn(0x7fde380c6b80 0x7fde380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881139840 0 0) 0x7fde380c0a50 con 0x7fde380c6f60 2023-10-30T13:37:22.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde24003530 con 0x7fde380c6f60 2023-10-30T13:37:22.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3432687106 0 0) 0x7fde24003530 con 0x7fde380c6f60 2023-10-30T13:37:22.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde3813f4a0 msgr2=0x7fde380c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:22.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde3813f4a0 0x7fde380c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 >> v2:172.21.15.150:3300/0 conn(0x7fde380c6b80 msgr2=0x7fde380bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:22.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 --2- 172.21.15.148:0/457537704 >> v2:172.21.15.150:3300/0 conn(0x7fde380c6b80 0x7fde380bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3806fca0 con 0x7fde380c6f60 2023-10-30T13:37:22.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fde28001680 con 0x7fde380c6f60 2023-10-30T13:37:22.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 >> v1:172.21.15.148:6789/0 conn(0x7fde380c6f60 legacy=0x7fde3813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde24003980 0x7fde24003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde3813f4a0 0x7fde240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde24004d10 0x7fde24005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3806f6e0 con 0x7fde24004d10 2023-10-30T13:37:22.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3806f9c0 con 0x7fde24003980 2023-10-30T13:37:22.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2400b990 con 0x7fde3813f4a0 2023-10-30T13:37:22.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3e894640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde24004d10 0x7fde24005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3e093640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde24003980 0x7fde24003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3d892640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde3813f4a0 0x7fde240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3e894640 1 -- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde3813f4a0 msgr2=0x7fde240047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:22.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3e894640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde3813f4a0 0x7fde240047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3e894640 1 -- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde24003980 msgr2=0x7fde24003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:22.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3e894640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde24003980 0x7fde24003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.736+0000 7fde3e894640 1 -- 172.21.15.148:0/457537704 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde24003530 con 0x7fde24004d10 2023-10-30T13:37:22.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde3e894640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde24004d10 0x7fde24005190 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fde30009fa0 tx=0x7fde3001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261399f78974d266 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:22.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde300a5cf0 con 0x7fde24004d10 2023-10-30T13:37:22.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3000f040 con 0x7fde24004d10 2023-10-30T13:37:22.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde3d091640 1 -- 172.21.15.148:0/457537704 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde300ad660 con 0x7fde24004d10 2023-10-30T13:37:22.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde402f7640 1 -- 172.21.15.148:0/457537704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde24004d10 msgr2=0x7fde24005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde402f7640 1 --2- 172.21.15.148:0/457537704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde24004d10 0x7fde24005190 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fde30009fa0 tx=0x7fde3001cc20 comp rx=0 tx=0).stop 2023-10-30T13:37:22.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde3e894640 1 -- 172.21.15.148:0/457537704 reap_dead start 2023-10-30T13:37:22.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde402f7640 1 -- 172.21.15.148:0/457537704 shutdown_connections 2023-10-30T13:37:22.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde402f7640 1 -- 172.21.15.148:0/457537704 >> 172.21.15.148:0/457537704 conn(0x7fde380c7910 msgr2=0x7fde380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:22.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde402f7640 1 -- 172.21.15.148:0/457537704 shutdown_connections 2023-10-30T13:37:22.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.740+0000 7fde402f7640 1 -- 172.21.15.148:0/457537704 wait complete. 2023-10-30T13:37:22.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 Processor -- start 2023-10-30T13:37:22.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 -- start start 2023-10-30T13:37:22.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde24004d10 0x7fde381386a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde380b0740 0x7fde380c11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde380c3420 0x7fde380c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde381f1150 con 0x7fde380c3420 2023-10-30T13:37:22.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde381f0280 con 0x7fde24004d10 2023-10-30T13:37:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde380c3de0 con 0x7fde380b0740 2023-10-30T13:37:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e894640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde380c3420 0x7fde380c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e894640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde380c3420 0x7fde380c38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59362/0 (socket says 172.21.15.148:59362) 2023-10-30T13:37:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e894640 1 -- 172.21.15.148:0/1007747793 learned_addr learned my addr 172.21.15.148:0/1007747793 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e093640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde24004d10 0x7fde381386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3d892640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde380b0740 0x7fde380c11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e894640 1 -- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde380b0740 msgr2=0x7fde380c11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e894640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde380b0740 0x7fde380c11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e894640 1 -- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde24004d10 msgr2=0x7fde381386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e894640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde24004d10 0x7fde381386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e894640 1 -- 172.21.15.148:0/1007747793 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde381f04d0 con 0x7fde380c3420 2023-10-30T13:37:22.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde3e894640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde380c3420 0x7fde380c38a0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fde30009fa0 tx=0x7fde24005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:22.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde2effd640 1 -- 172.21.15.148:0/1007747793 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde300ada80 con 0x7fde380c3420 2023-10-30T13:37:22.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde2effd640 1 -- 172.21.15.148:0/1007747793 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde300adc20 con 0x7fde380c3420 2023-10-30T13:37:22.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde2effd640 1 -- 172.21.15.148:0/1007747793 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde300b9450 con 0x7fde380c3420 2023-10-30T13:37:22.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde381f07b0 con 0x7fde380c3420 2023-10-30T13:37:22.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde381f0d70 con 0x7fde380c3420 2023-10-30T13:37:22.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.744+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3804e170 con 0x7fde380c3420 2023-10-30T13:37:22.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.748+0000 7fde2effd640 1 -- 172.21.15.148:0/1007747793 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde300b95b0 con 0x7fde380c3420 2023-10-30T13:37:22.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.748+0000 7fde2effd640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde1c0664a0 0x7fde1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:22.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.748+0000 7fde2effd640 1 -- 172.21.15.148:0/1007747793 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fde3011e9b0 con 0x7fde380c3420 2023-10-30T13:37:22.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.748+0000 7fde3e093640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde1c0664a0 0x7fde1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:22.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.748+0000 7fde3e093640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde1c0664a0 0x7fde1c068960 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fde34009b20 tx=0x7fde34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:22.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.752+0000 7fde2effd640 1 -- 172.21.15.148:0/1007747793 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde300eba70 con 0x7fde380c3420 2023-10-30T13:37:22.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.932+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde380c53b0 con 0x7fde380c3420 2023-10-30T13:37:22.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde2effd640 1 -- 172.21.15.148:0/1007747793 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7fde300bf090 con 0x7fde380c3420 2023-10-30T13:37:22.932 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:22.933 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:22.933 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:22.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde1c0664a0 msgr2=0x7fde1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde1c0664a0 0x7fde1c068960 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fde34009b20 tx=0x7fde34008040 comp rx=0 tx=0).stop 2023-10-30T13:37:22.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde380c3420 msgr2=0x7fde380c38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:22.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde380c3420 0x7fde380c38a0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fde30009fa0 tx=0x7fde24005800 comp rx=0 tx=0).stop 2023-10-30T13:37:22.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 shutdown_connections 2023-10-30T13:37:22.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fde1c0664a0 0x7fde1c068960 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fde380c3420 0x7fde380c38a0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fde380b0740 0x7fde380c11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 --2- 172.21.15.148:0/1007747793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde24004d10 0x7fde381386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:22.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 >> 172.21.15.148:0/1007747793 conn(0x7fde380c7910 msgr2=0x7fde3813ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:22.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 shutdown_connections 2023-10-30T13:37:22.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:22.936+0000 7fde402f7640 1 -- 172.21.15.148:0/1007747793 wait complete. 2023-10-30T13:37:22.953 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:37:24.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:25.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.104+0000 7fb55cac9640 1 Processor -- start 2023-10-30T13:37:25.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.104+0000 7fb55cac9640 1 -- start start 2023-10-30T13:37:25.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb55cac9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb55806f7a0 0x7fb55806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:25.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb55cac9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb558066cf0 con 0x7fb5580c6bd0 2023-10-30T13:37:25.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb55cac9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5580c70d0 con 0x7fb558071110 2023-10-30T13:37:25.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb55cac9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580700c0 con 0x7fb55806f7a0 2023-10-30T13:37:25.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb54dd9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb5580c6bd0 0x7fb558069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46790/0 (socket says 172.21.15.148:46790) 2023-10-30T13:37:25.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb54dd9b640 1 -- 172.21.15.148:0/2978845820 learned_addr learned my addr 172.21.15.148:0/2978845820 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:25.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb55659c640 1 --2- 172.21.15.148:0/2978845820 >> v2:172.21.15.150:3300/0 conn(0x7fb55806f7a0 0x7fb55806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:25.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb555d9b640 1 -- 172.21.15.148:0/2978845820 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367749056 0 0) 0x7fb558066cf0 con 0x7fb5580c6bd0 2023-10-30T13:37:25.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb555d9b640 1 -- 172.21.15.148:0/2978845820 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb544003530 con 0x7fb5580c6bd0 2023-10-30T13:37:25.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb555d9b640 1 -- 172.21.15.148:0/2978845820 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3978654192 0 0) 0x7fb544003530 con 0x7fb5580c6bd0 2023-10-30T13:37:25.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb55659c640 1 -- 172.21.15.148:0/2978845820 >> v1:172.21.15.150:6789/0 conn(0x7fb558071110 legacy=0x7fb558158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:25.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb55659c640 1 -- 172.21.15.148:0/2978845820 >> v1:172.21.15.148:6789/0 conn(0x7fb5580c6bd0 legacy=0x7fb558069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:25.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.108+0000 7fb55659c640 1 -- 172.21.15.148:0/2978845820 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580c7390 con 0x7fb55806f7a0 2023-10-30T13:37:25.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb55659c640 1 --2- 172.21.15.148:0/2978845820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55806f7a0 0x7fb55806fb80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb5480097c0 tx=0x7fb54801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ce51f65994ca4c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:25.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb555d9b640 1 -- 172.21.15.148:0/2978845820 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540001680 con 0x7fb55806f7a0 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb555d9b640 1 -- 172.21.15.148:0/2978845820 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5480a8610 con 0x7fb55806f7a0 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb555d9b640 1 -- 172.21.15.148:0/2978845820 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480a8980 con 0x7fb55806f7a0 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb55cac9640 1 -- 172.21.15.148:0/2978845820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55806f7a0 msgr2=0x7fb55806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb55cac9640 1 --2- 172.21.15.148:0/2978845820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55806f7a0 0x7fb55806fb80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb5480097c0 tx=0x7fb54801eb90 comp rx=0 tx=0).stop 2023-10-30T13:37:25.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb55cac9640 1 -- 172.21.15.148:0/2978845820 shutdown_connections 2023-10-30T13:37:25.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb55cac9640 1 --2- 172.21.15.148:0/2978845820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55806f7a0 0x7fb55806fb80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb55cac9640 1 -- 172.21.15.148:0/2978845820 >> 172.21.15.148:0/2978845820 conn(0x7fb5580c7910 msgr2=0x7fb5580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:25.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb55cac9640 1 -- 172.21.15.148:0/2978845820 shutdown_connections 2023-10-30T13:37:25.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.112+0000 7fb55cac9640 1 -- 172.21.15.148:0/2978845820 wait complete. 2023-10-30T13:37:25.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 Processor -- start 2023-10-30T13:37:25.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 -- start start 2023-10-30T13:37:25.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55806f7a0 0x7fb55814c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:25.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb558071110 0x7fb5581e2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:25.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5580c6bd0 0x7fb5581ef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:25.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb558154bd0 con 0x7fb558071110 2023-10-30T13:37:25.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5581ef210 con 0x7fb5580c6bd0 2023-10-30T13:37:25.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb558154e20 con 0x7fb55806f7a0 2023-10-30T13:37:25.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb558071110 0x7fb5581e2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:25.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb558071110 0x7fb5581e2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59368/0 (socket says 172.21.15.148:59368) 2023-10-30T13:37:25.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54dd9b640 1 -- 172.21.15.148:0/694951464 learned_addr learned my addr 172.21.15.148:0/694951464 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:25.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55659c640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55806f7a0 0x7fb55814c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:25.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb556d9d640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5580c6bd0 0x7fb5581ef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:25.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54dd9b640 1 -- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55806f7a0 msgr2=0x7fb55814c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:25.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54dd9b640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55806f7a0 0x7fb55814c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54dd9b640 1 -- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5580c6bd0 msgr2=0x7fb5581ef6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:25.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54dd9b640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5580c6bd0 0x7fb5581ef6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54dd9b640 1 -- 172.21.15.148:0/694951464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548009420 con 0x7fb558071110 2023-10-30T13:37:25.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55659c640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55806f7a0 0x7fb55814c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:25.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54dd9b640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb558071110 0x7fb5581e2250 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c750 tx=0x7fb54000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:25.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54f7fe640 1 -- 172.21.15.148:0/694951464 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540005ae0 con 0x7fb558071110 2023-10-30T13:37:25.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54f7fe640 1 -- 172.21.15.148:0/694951464 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540005c40 con 0x7fb558071110 2023-10-30T13:37:25.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb558155100 con 0x7fb558071110 2023-10-30T13:37:25.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb54f7fe640 1 -- 172.21.15.148:0/694951464 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400072b0 con 0x7fb558071110 2023-10-30T13:37:25.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5580c7e00 con 0x7fb558071110 2023-10-30T13:37:25.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.116+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5581ef380 con 0x7fb558071110 2023-10-30T13:37:25.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.120+0000 7fb54f7fe640 1 -- 172.21.15.148:0/694951464 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb540007410 con 0x7fb558071110 2023-10-30T13:37:25.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.120+0000 7fb54f7fe640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5380664d0 0x7fb538068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:25.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.120+0000 7fb54f7fe640 1 -- 172.21.15.148:0/694951464 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fb540083910 con 0x7fb558071110 2023-10-30T13:37:25.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.120+0000 7fb55659c640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5380664d0 0x7fb538068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:25.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.120+0000 7fb55659c640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5380664d0 0x7fb538068990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb5480b2000 tx=0x7fb5480aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:25.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.124+0000 7fb54f7fe640 1 -- 172.21.15.148:0/694951464 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5400509e0 con 0x7fb558071110 2023-10-30T13:37:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb55804e170 con 0x7fb558071110 2023-10-30T13:37:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb54f7fe640 1 -- 172.21.15.148:0/694951464 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7fb540017090 con 0x7fb558071110 2023-10-30T13:37:25.301 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:25.301 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5380664d0 msgr2=0x7fb538068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5380664d0 0x7fb538068990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb5480b2000 tx=0x7fb5480aa070 comp rx=0 tx=0).stop 2023-10-30T13:37:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb558071110 msgr2=0x7fb5581e2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb558071110 0x7fb5581e2250 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c750 tx=0x7fb54000cc10 comp rx=0 tx=0).stop 2023-10-30T13:37:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 shutdown_connections 2023-10-30T13:37:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb5380664d0 0x7fb538068990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb558071110 0x7fb5581e2250 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5580c6bd0 0x7fb5581ef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 --2- 172.21.15.148:0/694951464 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb55806f7a0 0x7fb55814c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 >> 172.21.15.148:0/694951464 conn(0x7fb5580c7910 msgr2=0x7fb5580c3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 shutdown_connections 2023-10-30T13:37:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:25.304+0000 7fb55cac9640 1 -- 172.21.15.148:0/694951464 wait complete. 2023-10-30T13:37:25.323 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:37:27.324 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:27.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0cbf80640 1 Processor -- start 2023-10-30T13:37:27.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0cbf80640 1 -- start start 2023-10-30T13:37:27.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0cbf80640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff0c41538a0 0x7ff0c415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0cbf80640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 0x7ff0c40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0cbf80640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40cb800 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0cbf80640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c415dc30 con 0x7ff0c41538a0 2023-10-30T13:37:27.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0cbf80640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40cb330 con 0x7ff0c40cf160 2023-10-30T13:37:27.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0c9d1c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff0c41538a0 0x7ff0c415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0c9d1c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff0c41538a0 0x7ff0c415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59378/0 (socket says 172.21.15.148:59378) 2023-10-30T13:37:27.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0c9d1c640 1 -- 172.21.15.148:0/1441264875 learned_addr learned my addr 172.21.15.148:0/1441264875 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:27.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0ca51d640 1 --2- 172.21.15.148:0/1441264875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 0x7ff0c40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0c9d1c640 1 -- 172.21.15.148:0/1441264875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 msgr2=0x7ff0c40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:27.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0c9d1c640 1 --2- 172.21.15.148:0/1441264875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 0x7ff0c40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0c9d1c640 1 -- 172.21.15.148:0/1441264875 >> v1:172.21.15.148:6789/0 conn(0x7ff0c40ca0e0 legacy=0x7ff0c415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.476+0000 7ff0c9d1c640 1 -- 172.21.15.148:0/1441264875 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40cad70 con 0x7ff0c41538a0 2023-10-30T13:37:27.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0c9d1c640 1 --2- 172.21.15.148:0/1441264875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c41538a0 0x7ff0c415d6f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009870 tx=0x7ff0c001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e515c8fc5e28051 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:27.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0c8d1a640 1 -- 172.21.15.148:0/1441264875 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00a8a30 con 0x7ff0c41538a0 2023-10-30T13:37:27.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0c8d1a640 1 -- 172.21.15.148:0/1441264875 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c00a8b90 con 0x7ff0c41538a0 2023-10-30T13:37:27.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0c8d1a640 1 -- 172.21.15.148:0/1441264875 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00b0440 con 0x7ff0c41538a0 2023-10-30T13:37:27.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0cbf80640 1 -- 172.21.15.148:0/1441264875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c41538a0 msgr2=0x7ff0c415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0cbf80640 1 --2- 172.21.15.148:0/1441264875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c41538a0 0x7ff0c415d6f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009870 tx=0x7ff0c001ed30 comp rx=0 tx=0).stop 2023-10-30T13:37:27.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0cbf80640 1 -- 172.21.15.148:0/1441264875 shutdown_connections 2023-10-30T13:37:27.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0cbf80640 1 --2- 172.21.15.148:0/1441264875 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 0x7ff0c40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0cbf80640 1 --2- 172.21.15.148:0/1441264875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c41538a0 0x7ff0c415d6f0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0cbf80640 1 -- 172.21.15.148:0/1441264875 >> 172.21.15.148:0/1441264875 conn(0x7ff0c406bfe0 msgr2=0x7ff0c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:27.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0cbf80640 1 -- 172.21.15.148:0/1441264875 shutdown_connections 2023-10-30T13:37:27.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.480+0000 7ff0cbf80640 1 -- 172.21.15.148:0/1441264875 wait complete. 2023-10-30T13:37:27.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 Processor -- start 2023-10-30T13:37:27.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 -- start start 2023-10-30T13:37:27.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c40ca0e0 0x7ff0c41e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 0x7ff0c415b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c41538a0 0x7ff0c415bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c414d4f0 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c41e4d40 con 0x7ff0c41538a0 2023-10-30T13:37:27.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c415b2a0 con 0x7ff0c40cf160 2023-10-30T13:37:27.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c9d1c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c40ca0e0 0x7ff0c41e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c9d1c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c40ca0e0 0x7ff0c41e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59386/0 (socket says 172.21.15.148:59386) 2023-10-30T13:37:27.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c9d1c640 1 -- 172.21.15.148:0/503893485 learned_addr learned my addr 172.21.15.148:0/503893485 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:27.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c951b640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 0x7ff0c415b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0ca51d640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c41538a0 0x7ff0c415bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c9d1c640 1 -- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 msgr2=0x7ff0c415b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c9d1c640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 0x7ff0c415b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c9d1c640 1 -- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c41538a0 msgr2=0x7ff0c415bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c9d1c640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c41538a0 0x7ff0c415bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c9d1c640 1 -- 172.21.15.148:0/503893485 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c00094d0 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c951b640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 0x7ff0c415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:37:27.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0ca51d640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c41538a0 0x7ff0c415bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:37:27.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0c9d1c640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c40ca0e0 0x7ff0c41e4090 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00b3040 tx=0x7ff0c001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:27.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0ba7fc640 1 -- 172.21.15.148:0/503893485 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00ba600 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0ba7fc640 1 -- 172.21.15.148:0/503893485 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c00ba780 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0ba7fc640 1 -- 172.21.15.148:0/503893485 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00bab30 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c40cef10 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c40c50b0 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.484+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c40b2a50 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.488+0000 7ff0ba7fc640 1 -- 172.21.15.148:0/503893485 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff0c00bac90 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.488+0000 7ff0ba7fc640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff0a80664a0 0x7ff0a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:27.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.488+0000 7ff0ba7fc640 1 -- 172.21.15.148:0/503893485 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7ff0c0122930 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.488+0000 7ff0c951b640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff0a80664a0 0x7ff0a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:27.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.488+0000 7ff0c951b640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff0a80664a0 0x7ff0a8068960 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40c4660 tx=0x7ff0b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:27.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.492+0000 7ff0ba7fc640 1 -- 172.21.15.148:0/503893485 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff0c00ef940 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0c4151940 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0ba7fc640 1 -- 172.21.15.148:0/503893485 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7ff0c00f2f60 con 0x7ff0c40ca0e0 2023-10-30T13:37:27.670 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:27.670 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:27.670 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:27.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff0a80664a0 msgr2=0x7ff0a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff0a80664a0 0x7ff0a8068960 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40c4660 tx=0x7ff0b4009040 comp rx=0 tx=0).stop 2023-10-30T13:37:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c40ca0e0 msgr2=0x7ff0c41e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c40ca0e0 0x7ff0c41e4090 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00b3040 tx=0x7ff0c001bad0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 shutdown_connections 2023-10-30T13:37:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff0a80664a0 0x7ff0a8068960 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0c41538a0 0x7ff0c415bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff0c40cf160 0x7ff0c415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 --2- 172.21.15.148:0/503893485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0c40ca0e0 0x7ff0c41e4090 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 >> 172.21.15.148:0/503893485 conn(0x7ff0c406bfe0 msgr2=0x7ff0c4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 shutdown_connections 2023-10-30T13:37:27.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:27.672+0000 7ff0cbf80640 1 -- 172.21.15.148:0/503893485 wait complete. 2023-10-30T13:37:27.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:37:29.692 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:29.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.844+0000 7fcea7809640 1 Processor -- start 2023-10-30T13:37:29.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.844+0000 7fcea7809640 1 -- start start 2023-10-30T13:37:29.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea7809640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcea00c05d0 0x7fcea013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea7809640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcea00c6f60 0x7fcea013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea7809640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea00c0a50 con 0x7fcea00c6b80 2023-10-30T13:37:29.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea7809640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea006f6e0 con 0x7fcea00c6f60 2023-10-30T13:37:29.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea7809640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea006f9c0 con 0x7fcea00c05d0 2023-10-30T13:37:29.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea55a5640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fcea00c6b80 0x7fcea00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46802/0 (socket says 172.21.15.148:46802) 2023-10-30T13:37:29.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea55a5640 1 -- 172.21.15.148:0/71203193 learned_addr learned my addr 172.21.15.148:0/71203193 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:29.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea5da6640 1 --2- 172.21.15.148:0/71203193 >> v2:172.21.15.148:3300/0 conn(0x7fcea00c6f60 0x7fcea013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea4da4640 1 --2- 172.21.15.148:0/71203193 >> v2:172.21.15.150:3300/0 conn(0x7fcea00c05d0 0x7fcea013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea5da6640 1 -- 172.21.15.148:0/71203193 >> v2:172.21.15.150:3300/0 conn(0x7fcea00c05d0 msgr2=0x7fcea013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:29.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea5da6640 1 --2- 172.21.15.148:0/71203193 >> v2:172.21.15.150:3300/0 conn(0x7fcea00c05d0 0x7fcea013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea5da6640 1 -- 172.21.15.148:0/71203193 >> v1:172.21.15.148:6789/0 conn(0x7fcea00c6b80 legacy=0x7fcea00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea5da6640 1 -- 172.21.15.148:0/71203193 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea006fca0 con 0x7fcea00c6f60 2023-10-30T13:37:29.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fcea5da6640 1 --2- 172.21.15.148:0/71203193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c6f60 0x7fcea013f2d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fcea006c7e0 tx=0x7fce9801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1da8ace3e5a340 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:29.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fce97fff640 1 -- 172.21.15.148:0/71203193 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce980a58e0 con 0x7fcea00c6f60 2023-10-30T13:37:29.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fce97fff640 1 -- 172.21.15.148:0/71203193 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce980a5a40 con 0x7fcea00c6f60 2023-10-30T13:37:29.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.848+0000 7fce97fff640 1 -- 172.21.15.148:0/71203193 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce980a5dd0 con 0x7fcea00c6f60 2023-10-30T13:37:29.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 -- 172.21.15.148:0/71203193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c6f60 msgr2=0x7fcea013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 --2- 172.21.15.148:0/71203193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c6f60 0x7fcea013f2d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fcea006c7e0 tx=0x7fce9801c910 comp rx=0 tx=0).stop 2023-10-30T13:37:29.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 -- 172.21.15.148:0/71203193 shutdown_connections 2023-10-30T13:37:29.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 --2- 172.21.15.148:0/71203193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c6f60 0x7fcea013f2d0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 --2- 172.21.15.148:0/71203193 >> v2:172.21.15.150:3300/0 conn(0x7fcea00c05d0 0x7fcea013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 -- 172.21.15.148:0/71203193 >> 172.21.15.148:0/71203193 conn(0x7fcea00c7910 msgr2=0x7fcea00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:29.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 -- 172.21.15.148:0/71203193 shutdown_connections 2023-10-30T13:37:29.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 -- 172.21.15.148:0/71203193 wait complete. 2023-10-30T13:37:29.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 Processor -- start 2023-10-30T13:37:29.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 -- start start 2023-10-30T13:37:29.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c05d0 0x7fcea0162070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcea00c6b80 0x7fcea00c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcea00c6f60 0x7fcea01501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea012ffc0 con 0x7fcea00c05d0 2023-10-30T13:37:29.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea00ca1a0 con 0x7fcea00c6b80 2023-10-30T13:37:29.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea7809640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea012fd20 con 0x7fcea00c6f60 2023-10-30T13:37:29.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea55a5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c05d0 0x7fcea0162070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea55a5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c05d0 0x7fcea0162070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59404/0 (socket says 172.21.15.148:59404) 2023-10-30T13:37:29.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea55a5640 1 -- 172.21.15.148:0/1739576708 learned_addr learned my addr 172.21.15.148:0/1739576708 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:29.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea5da6640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcea00c6f60 0x7fcea01501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.852+0000 7fcea4da4640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcea00c6b80 0x7fcea00c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea55a5640 1 -- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcea00c6f60 msgr2=0x7fcea01501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea55a5640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcea00c6f60 0x7fcea01501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea55a5640 1 -- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcea00c6b80 msgr2=0x7fcea00c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:29.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea55a5640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcea00c6b80 0x7fcea00c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:29.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea55a5640 1 -- 172.21.15.148:0/1739576708 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce98009a70 con 0x7fcea00c05d0 2023-10-30T13:37:29.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea4da4640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcea00c6b80 0x7fcea00c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:29.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea55a5640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c05d0 0x7fcea0162070 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c450 tx=0x7fce9c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:29.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fce95ffb640 1 -- 172.21.15.148:0/1739576708 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c005cf0 con 0x7fcea00c05d0 2023-10-30T13:37:29.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fce95ffb640 1 -- 172.21.15.148:0/1739576708 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c005e50 con 0x7fcea00c05d0 2023-10-30T13:37:29.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fce95ffb640 1 -- 172.21.15.148:0/1739576708 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c007290 con 0x7fcea00c05d0 2023-10-30T13:37:29.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea01631c0 con 0x7fcea00c05d0 2023-10-30T13:37:29.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea01341e0 con 0x7fcea00c05d0 2023-10-30T13:37:29.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea004e170 con 0x7fcea00c05d0 2023-10-30T13:37:29.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fce95ffb640 1 -- 172.21.15.148:0/1739576708 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fce9c0073f0 con 0x7fcea00c05d0 2023-10-30T13:37:29.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fce95ffb640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce840664a0 0x7fce84068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:29.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fce95ffb640 1 -- 172.21.15.148:0/1739576708 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fce9c0837d0 con 0x7fcea00c05d0 2023-10-30T13:37:29.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.856+0000 7fcea4da4640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce840664a0 0x7fce84068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:29.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.860+0000 7fcea4da4640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce840664a0 0x7fce84068960 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcea006a800 tx=0x7fce90008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:29.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:29.864+0000 7fce95ffb640 1 -- 172.21.15.148:0/1739576708 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fce9c0507f0 con 0x7fcea00c05d0 2023-10-30T13:37:30.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.044+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcea00c8190 con 0x7fcea00c05d0 2023-10-30T13:37:30.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.044+0000 7fce95ffb640 1 -- 172.21.15.148:0/1739576708 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7fce9c017020 con 0x7fcea00c05d0 2023-10-30T13:37:30.044 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:30.044 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:30.045 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:30.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce840664a0 msgr2=0x7fce84068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:30.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce840664a0 0x7fce84068960 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcea006a800 tx=0x7fce90008040 comp rx=0 tx=0).stop 2023-10-30T13:37:30.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c05d0 msgr2=0x7fcea0162070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:30.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c05d0 0x7fcea0162070 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c450 tx=0x7fce9c00c910 comp rx=0 tx=0).stop 2023-10-30T13:37:30.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 shutdown_connections 2023-10-30T13:37:30.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce840664a0 0x7fce84068960 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:30.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcea00c6f60 0x7fcea01501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:30.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcea00c6b80 0x7fcea00c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:30.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 --2- 172.21.15.148:0/1739576708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcea00c05d0 0x7fcea0162070 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:30.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 >> 172.21.15.148:0/1739576708 conn(0x7fcea00c7910 msgr2=0x7fcea00c54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:30.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 shutdown_connections 2023-10-30T13:37:30.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:30.048+0000 7fcea7809640 1 -- 172.21.15.148:0/1739576708 wait complete. 2023-10-30T13:37:30.065 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:37:30.630 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:30.634 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:37:32.066 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:32.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd624df5640 1 Processor -- start 2023-10-30T13:37:32.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd624df5640 1 -- start start 2023-10-30T13:37:32.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd624df5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c05d0 0x7fd62013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:32.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd624df5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6200c6f60 0x7fd62013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:32.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd624df5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6200c0a50 con 0x7fd6200c6b80 2023-10-30T13:37:32.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd624df5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd62006f6e0 con 0x7fd6200c6f60 2023-10-30T13:37:32.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd624df5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62006f9c0 con 0x7fd6200c05d0 2023-10-30T13:37:32.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd61ed9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6200c6f60 0x7fd62013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:32.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd61ed9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6200c6f60 0x7fd62013f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39146/0 (socket says 172.21.15.148:39146) 2023-10-30T13:37:32.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd61ed9d640 1 -- 172.21.15.148:0/1551508153 learned_addr learned my addr 172.21.15.148:0/1551508153 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:32.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd615d9b640 1 --2- 172.21.15.148:0/1551508153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c05d0 0x7fd62013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:32.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd61ed9d640 1 -- 172.21.15.148:0/1551508153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c05d0 msgr2=0x7fd62013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:32.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd61ed9d640 1 --2- 172.21.15.148:0/1551508153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c05d0 0x7fd62013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd61ed9d640 1 -- 172.21.15.148:0/1551508153 >> v1:172.21.15.150:6789/0 conn(0x7fd6200c6b80 legacy=0x7fd6200bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:32.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.216+0000 7fd61ed9d640 1 -- 172.21.15.148:0/1551508153 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62006fca0 con 0x7fd6200c6f60 2023-10-30T13:37:32.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd61ed9d640 1 --2- 172.21.15.148:0/1551508153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 0x7fd62013f2d0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd618009e50 tx=0x7fd61801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c89f6c5192a2d95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:32.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd61dd9b640 1 -- 172.21.15.148:0/1551508153 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180a57f0 con 0x7fd6200c6f60 2023-10-30T13:37:32.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd61dd9b640 1 -- 172.21.15.148:0/1551508153 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6180a5950 con 0x7fd6200c6f60 2023-10-30T13:37:32.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd61dd9b640 1 -- 172.21.15.148:0/1551508153 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180a5ce0 con 0x7fd6200c6f60 2023-10-30T13:37:32.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd624df5640 1 -- 172.21.15.148:0/1551508153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 msgr2=0x7fd62013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd624df5640 1 --2- 172.21.15.148:0/1551508153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 0x7fd62013f2d0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd618009e50 tx=0x7fd61801ca10 comp rx=0 tx=0).stop 2023-10-30T13:37:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd624df5640 1 -- 172.21.15.148:0/1551508153 shutdown_connections 2023-10-30T13:37:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd624df5640 1 --2- 172.21.15.148:0/1551508153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 0x7fd62013f2d0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd624df5640 1 --2- 172.21.15.148:0/1551508153 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c05d0 0x7fd62013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd624df5640 1 -- 172.21.15.148:0/1551508153 >> 172.21.15.148:0/1551508153 conn(0x7fd6200c7910 msgr2=0x7fd6200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd624df5640 1 -- 172.21.15.148:0/1551508153 shutdown_connections 2023-10-30T13:37:32.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.220+0000 7fd624df5640 1 -- 172.21.15.148:0/1551508153 wait complete. 2023-10-30T13:37:32.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd624df5640 1 Processor -- start 2023-10-30T13:37:32.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd624df5640 1 -- start start 2023-10-30T13:37:32.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd624df5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200c05d0 0x7fd620162f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:32.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd624df5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c6b80 0x7fd6201389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:32.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd624df5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 0x7fd6201373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:32.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd624df5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd620161f60 con 0x7fd6200c6f60 2023-10-30T13:37:32.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd624df5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62013d650 con 0x7fd6200c05d0 2023-10-30T13:37:32.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd624df5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd620163b00 con 0x7fd6200c6b80 2023-10-30T13:37:32.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 0x7fd6201373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:32.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 0x7fd6201373b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39148/0 (socket says 172.21.15.148:39148) 2023-10-30T13:37:32.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61ed9d640 1 -- 172.21.15.148:0/3321187943 learned_addr learned my addr 172.21.15.148:0/3321187943 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:32.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61e59c640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200c05d0 0x7fd620162f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:32.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd615d9b640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c6b80 0x7fd6201389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61ed9d640 1 -- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c6b80 msgr2=0x7fd6201389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61ed9d640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c6b80 0x7fd6201389a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61ed9d640 1 -- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200c05d0 msgr2=0x7fd620162f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61ed9d640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200c05d0 0x7fd620162f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61ed9d640 1 -- 172.21.15.148:0/3321187943 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd618009b30 con 0x7fd6200c6f60 2023-10-30T13:37:32.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd615d9b640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c6b80 0x7fd6201389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:32.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.224+0000 7fd61ed9d640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 0x7fd6201373b0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd61801a640 tx=0x7fd6180197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:32.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd6177fe640 1 -- 172.21.15.148:0/3321187943 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180b8880 con 0x7fd6200c6f60 2023-10-30T13:37:32.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd6177fe640 1 -- 172.21.15.148:0/3321187943 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6180b89e0 con 0x7fd6200c6f60 2023-10-30T13:37:32.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6201621b0 con 0x7fd6200c6f60 2023-10-30T13:37:32.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd6177fe640 1 -- 172.21.15.148:0/3321187943 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180b8d90 con 0x7fd6200c6f60 2023-10-30T13:37:32.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd620162770 con 0x7fd6200c6f60 2023-10-30T13:37:32.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62004e170 con 0x7fd6200c6f60 2023-10-30T13:37:32.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd6177fe640 1 -- 172.21.15.148:0/3321187943 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6180c3460 con 0x7fd6200c6f60 2023-10-30T13:37:32.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd6177fe640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6000664a0 0x7fd600068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:32.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd6177fe640 1 -- 172.21.15.148:0/3321187943 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7fd61811fc60 con 0x7fd6200c6f60 2023-10-30T13:37:32.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.228+0000 7fd61e59c640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6000664a0 0x7fd600068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:32.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.232+0000 7fd61e59c640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6000664a0 0x7fd600068960 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd610007fd0 tx=0x7fd610009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:32.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.232+0000 7fd6177fe640 1 -- 172.21.15.148:0/3321187943 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6180ecc70 con 0x7fd6200c6f60 2023-10-30T13:37:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd620154070 con 0x7fd6200c6f60 2023-10-30T13:37:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd6177fe640 1 -- 172.21.15.148:0/3321187943 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7fd6180f0290 con 0x7fd6200c6f60 2023-10-30T13:37:32.410 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:32.410 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:32.410 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6000664a0 msgr2=0x7fd600068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6000664a0 0x7fd600068960 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd610007fd0 tx=0x7fd610009040 comp rx=0 tx=0).stop 2023-10-30T13:37:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 msgr2=0x7fd6201373b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 0x7fd6201373b0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd61801a640 tx=0x7fd6180197b0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 shutdown_connections 2023-10-30T13:37:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6000664a0 0x7fd600068960 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6200c6f60 0x7fd6201373b0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd6200c6b80 0x7fd6201389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 --2- 172.21.15.148:0/3321187943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200c05d0 0x7fd620162f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 >> 172.21.15.148:0/3321187943 conn(0x7fd6200c7910 msgr2=0x7fd6200aefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 shutdown_connections 2023-10-30T13:37:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:32.412+0000 7fd624df5640 1 -- 172.21.15.148:0/3321187943 wait complete. 2023-10-30T13:37:32.431 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:37:34.432 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:34.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.584+0000 7f540edd3640 1 Processor -- start 2023-10-30T13:37:34.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.584+0000 7f540edd3640 1 -- start start 2023-10-30T13:37:34.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540edd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5408070a00 0x7f5408070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:34.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540edd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5408069e90 0x7f540806a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:34.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540edd3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54080713b0 con 0x7f54081539a0 2023-10-30T13:37:34.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540edd3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540806fbb0 con 0x7f5408070a00 2023-10-30T13:37:34.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540edd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5408153d80 con 0x7f5408069e90 2023-10-30T13:37:34.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540cb6f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f54081539a0 0x7f540815a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49492/0 (socket says 172.21.15.148:49492) 2023-10-30T13:37:34.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540cb6f640 1 -- 172.21.15.148:0/2544295825 learned_addr learned my addr 172.21.15.148:0/2544295825 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f5407fff640 1 --2- 172.21.15.148:0/2544295825 >> v2:172.21.15.150:3300/0 conn(0x7f5408070a00 0x7f5408070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540d370640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5408069e90 0x7f540806a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85448030 0 0) 0x7f54080713b0 con 0x7f54081539a0 2023-10-30T13:37:34.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f8003530 con 0x7f54081539a0 2023-10-30T13:37:34.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1431602090 0 0) 0x7f53f8003530 con 0x7f54081539a0 2023-10-30T13:37:34.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5408069e90 msgr2=0x7f540806a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:34.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5408069e90 0x7f540806a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 >> v2:172.21.15.150:3300/0 conn(0x7f5408070a00 msgr2=0x7f5408070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:34.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 --2- 172.21.15.148:0/2544295825 >> v2:172.21.15.150:3300/0 conn(0x7f5408070a00 0x7f5408070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54080c5c50 con 0x7f54081539a0 2023-10-30T13:37:34.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f53fc001680 con 0x7f54081539a0 2023-10-30T13:37:34.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 >> v1:172.21.15.148:6789/0 conn(0x7f54081539a0 legacy=0x7f540815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:34.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5408069e90 0x7f53f8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:34.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53f8003f00 0x7f53f80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:34.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53f8004d10 0x7f53f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:34.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540806fbb0 con 0x7f53f8003f00 2023-10-30T13:37:34.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5408153d80 con 0x7f53f8004d10 2023-10-30T13:37:34.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f800b9e0 con 0x7f5408069e90 2023-10-30T13:37:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f5407fff640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53f8003f00 0x7f53f80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540cb6f640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5408069e90 0x7f53f8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540d370640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53f8004d10 0x7f53f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f5407fff640 1 -- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5408069e90 msgr2=0x7f53f8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f5407fff640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5408069e90 0x7f53f8003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f5407fff640 1 -- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53f8004d10 msgr2=0x7f53f8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:34.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f5407fff640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53f8004d10 0x7f53f8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f5407fff640 1 -- 172.21.15.148:0/2544295825 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f8003530 con 0x7f53f8003f00 2023-10-30T13:37:34.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.588+0000 7f540cb6f640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5408069e90 0x7f53f8003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f5407fff640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53f8003f00 0x7f53f80047d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009b10 tx=0x7f53f401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0256f0952a75b3e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40a5630 con 0x7f53f8003f00 2023-10-30T13:37:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f40a5790 con 0x7f53f8003f00 2023-10-30T13:37:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f54077fe640 1 -- 172.21.15.148:0/2544295825 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40a5b40 con 0x7f53f8003f00 2023-10-30T13:37:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f540edd3640 1 -- 172.21.15.148:0/2544295825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53f8003f00 msgr2=0x7f53f80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f540edd3640 1 --2- 172.21.15.148:0/2544295825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53f8003f00 0x7f53f80047d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009b10 tx=0x7f53f401ca60 comp rx=0 tx=0).stop 2023-10-30T13:37:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f540d370640 1 -- 172.21.15.148:0/2544295825 reap_dead start 2023-10-30T13:37:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f540edd3640 1 -- 172.21.15.148:0/2544295825 shutdown_connections 2023-10-30T13:37:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f540edd3640 1 -- 172.21.15.148:0/2544295825 >> 172.21.15.148:0/2544295825 conn(0x7f54080c7910 msgr2=0x7f54080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f540edd3640 1 -- 172.21.15.148:0/2544295825 shutdown_connections 2023-10-30T13:37:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.592+0000 7f540edd3640 1 -- 172.21.15.148:0/2544295825 wait complete. 2023-10-30T13:37:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 Processor -- start 2023-10-30T13:37:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 -- start start 2023-10-30T13:37:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53f8003f00 0x7f5408144d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:34.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5408159250 0x7f54081f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:34.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408159690 0x7f5408140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:34.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54081620e0 con 0x7f5408159250 2023-10-30T13:37:34.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54080c1a90 con 0x7f5408159690 2023-10-30T13:37:34.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5408161bf0 con 0x7f53f8003f00 2023-10-30T13:37:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f5407fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5408159250 0x7f54081f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f5407fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5408159250 0x7f54081f5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39160/0 (socket says 172.21.15.148:39160) 2023-10-30T13:37:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f5407fff640 1 -- 172.21.15.148:0/64700173 learned_addr learned my addr 172.21.15.148:0/64700173 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540cb6f640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53f8003f00 0x7f5408144d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540d370640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408159690 0x7f5408140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f5407fff640 1 -- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53f8003f00 msgr2=0x7f5408144d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:34.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f5407fff640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53f8003f00 0x7f5408144d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f5407fff640 1 -- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408159690 msgr2=0x7f5408140510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:34.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f5407fff640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408159690 0x7f5408140510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f5407fff640 1 -- 172.21.15.148:0/64700173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54081457d0 con 0x7f5408159250 2023-10-30T13:37:34.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f5407fff640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5408159250 0x7f54081f5a00 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009b10 tx=0x7f53f8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:34.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f54057fa640 1 -- 172.21.15.148:0/64700173 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40a5630 con 0x7f5408159250 2023-10-30T13:37:34.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f54057fa640 1 -- 172.21.15.148:0/64700173 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f4004020 con 0x7f5408159250 2023-10-30T13:37:34.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5408150a50 con 0x7f5408159250 2023-10-30T13:37:34.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f54057fa640 1 -- 172.21.15.148:0/64700173 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40043d0 con 0x7f5408159250 2023-10-30T13:37:34.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5408161e00 con 0x7f5408159250 2023-10-30T13:37:34.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.596+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5408159010 con 0x7f5408159250 2023-10-30T13:37:34.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.600+0000 7f54057fa640 1 -- 172.21.15.148:0/64700173 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f53f40b4040 con 0x7f5408159250 2023-10-30T13:37:34.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.600+0000 7f54057fa640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53ec0664a0 0x7f53ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:34.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.600+0000 7f54057fa640 1 -- 172.21.15.148:0/64700173 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6972+0+0 (secure 0 0 0) 0x7f53f411e520 con 0x7f5408159250 2023-10-30T13:37:34.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.600+0000 7f540cb6f640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53ec0664a0 0x7f53ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:34.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.600+0000 7f540cb6f640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53ec0664a0 0x7f53ec068960 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f53fc001060 tx=0x7f53fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:34.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.604+0000 7f54057fa640 1 -- 172.21.15.148:0/64700173 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53f40eb670 con 0x7f5408159250 2023-10-30T13:37:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f540804e170 con 0x7f5408159250 2023-10-30T13:37:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f54057fa640 1 -- 172.21.15.148:0/64700173 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f53f40eec90 con 0x7f5408159250 2023-10-30T13:37:34.782 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:34.783 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7005,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:36:07.108351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6870},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6870":{"gid":6870,"name":"c","rank":0,"incarnation":104,"state":"up:active","state_seq":30,"addr":"172.21.15.148:6837/3144126860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3144126860},{"type":"v1","addr":"172.21.15.148:6837","nonce":3144126860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 108 2023-10-30T13:37:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53ec0664a0 msgr2=0x7f53ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53ec0664a0 0x7f53ec068960 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f53fc001060 tx=0x7f53fc009040 comp rx=0 tx=0).stop 2023-10-30T13:37:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5408159250 msgr2=0x7f54081f5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5408159250 0x7f54081f5a00 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009b10 tx=0x7f53f8005800 comp rx=0 tx=0).stop 2023-10-30T13:37:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 shutdown_connections 2023-10-30T13:37:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53ec0664a0 0x7f53ec068960 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5408159250 0x7f54081f5a00 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408159690 0x7f5408140510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 --2- 172.21.15.148:0/64700173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53f8003f00 0x7f5408144d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 >> 172.21.15.148:0/64700173 conn(0x7f54080c7910 msgr2=0x7f540814e260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 shutdown_connections 2023-10-30T13:37:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:34.784+0000 7f540edd3640 1 -- 172.21.15.148:0/64700173 wait complete. 2023-10-30T13:37:34.803 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:37:36.804 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:36.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076f10f640 1 Processor -- start 2023-10-30T13:37:36.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076f10f640 1 -- start start 2023-10-30T13:37:36.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076f10f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f07680c7df0 0x7f07680c8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:36.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076f10f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0768153f80 con 0x7f0768153ba0 2023-10-30T13:37:36.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076f10f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07680b0e60 con 0x7f07680c74b0 2023-10-30T13:37:36.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076f10f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07680c1f50 con 0x7f07680c7df0 2023-10-30T13:37:36.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076ceab640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0768153ba0 0x7f076815a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49494/0 (socket says 172.21.15.148:49494) 2023-10-30T13:37:36.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076ceab640 1 -- 172.21.15.148:0/2181225573 learned_addr learned my addr 172.21.15.148:0/2181225573 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:36.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076d6ac640 1 --2- 172.21.15.148:0/2181225573 >> v2:172.21.15.150:3300/0 conn(0x7f07680c7df0 0x7f07680c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:36.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615626600 0 0) 0x7f0768153f80 con 0x7f0768153ba0 2023-10-30T13:37:36.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0758003530 con 0x7f0768153ba0 2023-10-30T13:37:36.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 275732118 0 0) 0x7f0758003530 con 0x7f0768153ba0 2023-10-30T13:37:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 >> v2:172.21.15.150:3300/0 conn(0x7f07680c7df0 msgr2=0x7f07680c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 --2- 172.21.15.148:0/2181225573 >> v2:172.21.15.150:3300/0 conn(0x7f07680c7df0 0x7f07680c8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 >> v1:172.21.15.150:6789/0 conn(0x7f07680c74b0 legacy=0x7f0768069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07680b1140 con 0x7f0768153ba0 2023-10-30T13:37:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f075c001680 con 0x7f0768153ba0 2023-10-30T13:37:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 >> v1:172.21.15.148:6789/0 conn(0x7f0768153ba0 legacy=0x7f076815a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0758003980 0x7f0758003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07580047d0 0x7f0758004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0758005190 0x7f075800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07680c1f50 con 0x7f0758005190 2023-10-30T13:37:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075800c130 con 0x7f0758003980 2023-10-30T13:37:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075800c2f0 con 0x7f07580047d0 2023-10-30T13:37:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076d6ac640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0758005190 0x7f075800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076ceab640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0758003980 0x7f0758003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f0767fff640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07580047d0 0x7f0758004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076d6ac640 1 -- 172.21.15.148:0/2181225573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07580047d0 msgr2=0x7f0758004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076d6ac640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07580047d0 0x7f0758004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076d6ac640 1 -- 172.21.15.148:0/2181225573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0758003980 msgr2=0x7f0758003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076d6ac640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0758003980 0x7f0758003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:36.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.956+0000 7f076d6ac640 1 -- 172.21.15.148:0/2181225573 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0758003530 con 0x7f0758005190 2023-10-30T13:37:36.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f076d6ac640 1 -- 172.21.15.148:0/2181225573 reap_dead start 2023-10-30T13:37:36.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f076d6ac640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0758005190 0x7f075800b990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f0760009fc0 tx=0x7f076001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c94fe97016400e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:36.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07600a5bb0 con 0x7f0758005190 2023-10-30T13:37:36.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076000f040 con 0x7f0758005190 2023-10-30T13:37:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f07677fe640 1 -- 172.21.15.148:0/2181225573 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07600039a0 con 0x7f0758005190 2023-10-30T13:37:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f076f10f640 1 -- 172.21.15.148:0/2181225573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0758005190 msgr2=0x7f075800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f076f10f640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0758005190 0x7f075800b990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f0760009fc0 tx=0x7f076001cca0 comp rx=0 tx=0).stop 2023-10-30T13:37:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f076f10f640 1 -- 172.21.15.148:0/2181225573 shutdown_connections 2023-10-30T13:37:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f076f10f640 1 --2- 172.21.15.148:0/2181225573 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0758005190 0x7f075800b990 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f076f10f640 1 -- 172.21.15.148:0/2181225573 >> 172.21.15.148:0/2181225573 conn(0x7f076806d770 msgr2=0x7f076806db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f076f10f640 1 -- 172.21.15.148:0/2181225573 shutdown_connections 2023-10-30T13:37:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.960+0000 7f076f10f640 1 -- 172.21.15.148:0/2181225573 wait complete. 2023-10-30T13:37:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 Processor -- start 2023-10-30T13:37:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 -- start start 2023-10-30T13:37:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0758005190 0x7f076814e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07680c4ff0 0x7f0768146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0768147190 0x7f0768154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076806abf0 con 0x7f07680c4ff0 2023-10-30T13:37:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07681475f0 con 0x7f0758005190 2023-10-30T13:37:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0768155ed0 con 0x7f0768147190 2023-10-30T13:37:36.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f0767fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07680c4ff0 0x7f0768146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:36.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f0767fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07680c4ff0 0x7f0768146c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39182/0 (socket says 172.21.15.148:39182) 2023-10-30T13:37:36.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f0767fff640 1 -- 172.21.15.148:0/1191966011 learned_addr learned my addr 172.21.15.148:0/1191966011 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:36.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076ceab640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0758005190 0x7f076814e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:36.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076d6ac640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0768147190 0x7f0768154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:36.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f0767fff640 1 -- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0768147190 msgr2=0x7f0768154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:36.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f0767fff640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0768147190 0x7f0768154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:36.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f0767fff640 1 -- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0758005190 msgr2=0x7f076814e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:36.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f0767fff640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0758005190 0x7f076814e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:36.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f0767fff640 1 -- 172.21.15.148:0/1191966011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0768156170 con 0x7f07680c4ff0 2023-10-30T13:37:36.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f0767fff640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07680c4ff0 0x7f0768146c50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f075400e440 tx=0x7f075400e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:36.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f07657fa640 1 -- 172.21.15.148:0/1191966011 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07540052e0 con 0x7f07680c4ff0 2023-10-30T13:37:36.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f07657fa640 1 -- 172.21.15.148:0/1191966011 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0754005440 con 0x7f07680c4ff0 2023-10-30T13:37:36.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076806ae40 con 0x7f07680c4ff0 2023-10-30T13:37:36.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f07657fa640 1 -- 172.21.15.148:0/1191966011 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754003840 con 0x7f07680c4ff0 2023-10-30T13:37:36.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.964+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07681582e0 con 0x7f07680c4ff0 2023-10-30T13:37:36.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.968+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0768074250 con 0x7f07680c4ff0 2023-10-30T13:37:36.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.968+0000 7f07657fa640 1 -- 172.21.15.148:0/1191966011 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07540039a0 con 0x7f07680c4ff0 2023-10-30T13:37:36.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.968+0000 7f07657fa640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f074c066470 0x7f074c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:36.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.968+0000 7f07657fa640 1 -- 172.21.15.148:0/1191966011 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f0754083c60 con 0x7f07680c4ff0 2023-10-30T13:37:36.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.968+0000 7f076ceab640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f074c066470 0x7f074c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:36.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.968+0000 7f076ceab640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f074c066470 0x7f074c068930 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f075c004880 tx=0x7f075c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:36.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:36.972+0000 7f07657fa640 1 -- 172.21.15.148:0/1191966011 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0754050b80 con 0x7f07680c4ff0 2023-10-30T13:37:37.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.156+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0768154150 con 0x7f07680c4ff0 2023-10-30T13:37:37.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.156+0000 7f07657fa640 1 -- 172.21.15.148:0/1191966011 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f0754019020 con 0x7f07680c4ff0 2023-10-30T13:37:37.155 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:37.155 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:34.820690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:37.155 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 109 2023-10-30T13:37:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f074c066470 msgr2=0x7f074c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f074c066470 0x7f074c068930 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f075c004880 tx=0x7f075c009040 comp rx=0 tx=0).stop 2023-10-30T13:37:37.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07680c4ff0 msgr2=0x7f0768146c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07680c4ff0 0x7f0768146c50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f075400e440 tx=0x7f075400e900 comp rx=0 tx=0).stop 2023-10-30T13:37:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 shutdown_connections 2023-10-30T13:37:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f074c066470 0x7f074c068930 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07680c4ff0 0x7f0768146c50 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0768147190 0x7f0768154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 --2- 172.21.15.148:0/1191966011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0758005190 0x7f076814e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 >> 172.21.15.148:0/1191966011 conn(0x7f076806d770 msgr2=0x7f07681590a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:37.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 shutdown_connections 2023-10-30T13:37:37.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.160+0000 7f076f10f640 1 -- 172.21.15.148:0/1191966011 wait complete. 2023-10-30T13:37:37.176 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:37:37.176 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:37:37.177 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:37.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.316+0000 7fcf25ba7640 1 Processor -- start 2023-10-30T13:37:37.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.316+0000 7fcf25ba7640 1 -- start start 2023-10-30T13:37:37.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf25ba7640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcf201538f0 0x7fcf20159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:37.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf25ba7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 0x7fcf200c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:37.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf25ba7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf200cefa0 0x7fcf200cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:37.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf25ba7640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2015d300 con 0x7fcf201538f0 2023-10-30T13:37:37.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf25ba7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2015d5e0 con 0x7fcf200cefa0 2023-10-30T13:37:37.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf25ba7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2015d8c0 con 0x7fcf2015cd50 2023-10-30T13:37:37.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf24ba5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcf201538f0 0x7fcf20159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:37.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf24ba5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcf201538f0 0x7fcf20159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39192/0 (socket says 172.21.15.148:39192) 2023-10-30T13:37:37.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf24ba5640 1 -- 172.21.15.148:0/61147796 learned_addr learned my addr 172.21.15.148:0/61147796 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:37.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf1ffff640 1 --2- 172.21.15.148:0/61147796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 0x7fcf200c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:37.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf253a6640 1 --2- 172.21.15.148:0/61147796 >> v2:172.21.15.150:3300/0 conn(0x7fcf200cefa0 0x7fcf200cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:37.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf24ba5640 1 -- 172.21.15.148:0/61147796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 msgr2=0x7fcf200c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:37.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf24ba5640 1 --2- 172.21.15.148:0/61147796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 0x7fcf200c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf24ba5640 1 -- 172.21.15.148:0/61147796 >> v2:172.21.15.150:3300/0 conn(0x7fcf200cefa0 msgr2=0x7fcf200cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:37.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf24ba5640 1 --2- 172.21.15.148:0/61147796 >> v2:172.21.15.150:3300/0 conn(0x7fcf200cefa0 0x7fcf200cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf24ba5640 1 -- 172.21.15.148:0/61147796 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2015dba0 con 0x7fcf201538f0 2023-10-30T13:37:37.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.320+0000 7fcf1ffff640 1 --2- 172.21.15.148:0/61147796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 0x7fcf200c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:37:37.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf24ba5640 1 --2- 172.21.15.148:0/61147796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf201538f0 0x7fcf20159fb0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fcf200b1ae0 tx=0x7fcf1401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed06f1d511f07f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:37.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf1f7fe640 1 -- 172.21.15.148:0/61147796 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf140a89b0 con 0x7fcf201538f0 2023-10-30T13:37:37.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf1f7fe640 1 -- 172.21.15.148:0/61147796 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf140a8b10 con 0x7fcf201538f0 2023-10-30T13:37:37.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf1f7fe640 1 -- 172.21.15.148:0/61147796 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf140a8ea0 con 0x7fcf201538f0 2023-10-30T13:37:37.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 -- 172.21.15.148:0/61147796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf201538f0 msgr2=0x7fcf20159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:37.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/61147796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf201538f0 0x7fcf20159fb0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fcf200b1ae0 tx=0x7fcf1401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 -- 172.21.15.148:0/61147796 shutdown_connections 2023-10-30T13:37:37.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/61147796 >> v2:172.21.15.150:3300/0 conn(0x7fcf200cefa0 0x7fcf200cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/61147796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 0x7fcf200c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/61147796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf201538f0 0x7fcf20159fb0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 -- 172.21.15.148:0/61147796 >> 172.21.15.148:0/61147796 conn(0x7fcf2006a640 msgr2=0x7fcf20065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:37.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 -- 172.21.15.148:0/61147796 shutdown_connections 2023-10-30T13:37:37.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 -- 172.21.15.148:0/61147796 wait complete. 2023-10-30T13:37:37.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 Processor -- start 2023-10-30T13:37:37.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.324+0000 7fcf25ba7640 1 -- start start 2023-10-30T13:37:37.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf25ba7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf200cefa0 0x7fcf20151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:37.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf25ba7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf201538f0 0x7fcf201529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:37.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf25ba7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 0x7fcf200b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:37.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf25ba7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2015ae50 con 0x7fcf200cefa0 2023-10-30T13:37:37.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf25ba7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf201ef1e0 con 0x7fcf201538f0 2023-10-30T13:37:37.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf25ba7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf201f01b0 con 0x7fcf2015cd50 2023-10-30T13:37:37.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf24ba5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf200cefa0 0x7fcf20151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:37.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf24ba5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf200cefa0 0x7fcf20151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39194/0 (socket says 172.21.15.148:39194) 2023-10-30T13:37:37.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf24ba5640 1 -- 172.21.15.148:0/2786678430 learned_addr learned my addr 172.21.15.148:0/2786678430 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:37.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf253a6640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 0x7fcf200b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:37.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf1ffff640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf201538f0 0x7fcf201529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:37.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf24ba5640 1 -- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 msgr2=0x7fcf200b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:37.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf24ba5640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 0x7fcf200b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf24ba5640 1 -- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf201538f0 msgr2=0x7fcf201529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:37.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf24ba5640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf201538f0 0x7fcf201529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf24ba5640 1 -- 172.21.15.148:0/2786678430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf140094d0 con 0x7fcf200cefa0 2023-10-30T13:37:37.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf24ba5640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf200cefa0 0x7fcf20151e60 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fcf1401d3a0 tx=0x7fcf1401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:37.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf1d7fa640 1 -- 172.21.15.148:0/2786678430 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf140ba600 con 0x7fcf200cefa0 2023-10-30T13:37:37.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf1d7fa640 1 -- 172.21.15.148:0/2786678430 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf140ba760 con 0x7fcf200cefa0 2023-10-30T13:37:37.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf1d7fa640 1 -- 172.21.15.148:0/2786678430 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf140bab10 con 0x7fcf200cefa0 2023-10-30T13:37:37.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf201523a0 con 0x7fcf200cefa0 2023-10-30T13:37:37.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf20151a30 con 0x7fcf200cefa0 2023-10-30T13:37:37.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2004e170 con 0x7fcf200cefa0 2023-10-30T13:37:37.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.328+0000 7fcf1d7fa640 1 -- 172.21.15.148:0/2786678430 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcf140bac70 con 0x7fcf200cefa0 2023-10-30T13:37:37.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.332+0000 7fcf1d7fa640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf040664a0 0x7fcf04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:37.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.332+0000 7fcf1d7fa640 1 -- 172.21.15.148:0/2786678430 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fcf141229d0 con 0x7fcf200cefa0 2023-10-30T13:37:37.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.332+0000 7fcf1ffff640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf040664a0 0x7fcf04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:37.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.332+0000 7fcf1ffff640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf040664a0 0x7fcf04068960 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c006fd0 tx=0x7fcf0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:37.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.332+0000 7fcf1d7fa640 1 -- 172.21.15.148:0/2786678430 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcf140ef8f0 con 0x7fcf200cefa0 2023-10-30T13:37:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf200b5220 con 0x7fcf200cefa0 2023-10-30T13:37:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf1d7fa640 1 -- 172.21.15.148:0/2786678430 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5719 (secure 0 0 0) 0x7fcf140f2f10 con 0x7fcf200cefa0 2023-10-30T13:37:37.513 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:37.514 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:34.820690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:37.514 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 109 2023-10-30T13:37:37.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf040664a0 msgr2=0x7fcf04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:37.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf040664a0 0x7fcf04068960 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c006fd0 tx=0x7fcf0c008040 comp rx=0 tx=0).stop 2023-10-30T13:37:37.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf200cefa0 msgr2=0x7fcf20151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:37.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf200cefa0 0x7fcf20151e60 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fcf1401d3a0 tx=0x7fcf1401ba90 comp rx=0 tx=0).stop 2023-10-30T13:37:37.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 shutdown_connections 2023-10-30T13:37:37.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcf040664a0 0x7fcf04068960 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcf2015cd50 0x7fcf200b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf201538f0 0x7fcf201529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 --2- 172.21.15.148:0/2786678430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcf200cefa0 0x7fcf20151e60 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:37.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 >> 172.21.15.148:0/2786678430 conn(0x7fcf2006a640 msgr2=0x7fcf20149290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:37.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 shutdown_connections 2023-10-30T13:37:37.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:37.516+0000 7fcf25ba7640 1 -- 172.21.15.148:0/2786678430 wait complete. 2023-10-30T13:37:37.535 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 109, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 109, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7005": { "addr": "172.21.15.148:6835/1079809213", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1079809213, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1079809213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7005, "incarnation": 109, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:37:34.820690+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7005 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:37:39.537 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:39.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b20c32640 1 Processor -- start 2023-10-30T13:37:39.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b20c32640 1 -- start start 2023-10-30T13:37:39.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b20c32640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c071110 0x7f3b1c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:39.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b20c32640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1c066cf0 con 0x7f3b1c06f7a0 2023-10-30T13:37:39.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b20c32640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1c0c70d0 con 0x7f3b1c0c6bd0 2023-10-30T13:37:39.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b20c32640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0700c0 con 0x7f3b1c071110 2023-10-30T13:37:39.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b1a59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3b1c06f7a0 0x7f3b1c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49508/0 (socket says 172.21.15.148:49508) 2023-10-30T13:37:39.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b1a59c640 1 -- 172.21.15.148:0/2994761142 learned_addr learned my addr 172.21.15.148:0/2994761142 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:39.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b1ad9d640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c071110 0x7f3b1c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:39.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358339435 0 0) 0x7f3b1c066cf0 con 0x7f3b1c06f7a0 2023-10-30T13:37:39.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.688+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b08003530 con 0x7f3b1c06f7a0 2023-10-30T13:37:39.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126670008 0 0) 0x7f3b08003530 con 0x7f3b1c06f7a0 2023-10-30T13:37:39.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c071110 msgr2=0x7f3b1c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:39.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c071110 0x7f3b1c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 >> v1:172.21.15.150:6789/0 conn(0x7f3b1c0c6bd0 legacy=0x7f3b1c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:39.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c0c7390 con 0x7f3b1c06f7a0 2023-10-30T13:37:39.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f3b0c001680 con 0x7f3b1c06f7a0 2023-10-30T13:37:39.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 >> v1:172.21.15.148:6789/0 conn(0x7f3b1c06f7a0 legacy=0x7f3b1c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:39.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b080039b0 0x7f3b08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:39.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c071110 0x7f3b080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:39.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004d10 0x7f3b08005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:39.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0700c0 con 0x7f3b080039b0 2023-10-30T13:37:39.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0800b990 con 0x7f3b08004d10 2023-10-30T13:37:39.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0800bb50 con 0x7f3b1c071110 2023-10-30T13:37:39.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b1a59c640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b080039b0 0x7f3b08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:39.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b1ad9d640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004d10 0x7f3b08005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:39.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b13fff640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c071110 0x7f3b080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:39.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b1a59c640 1 -- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c071110 msgr2=0x7f3b080047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:39.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b1a59c640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c071110 0x7f3b080047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b1a59c640 1 -- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004d10 msgr2=0x7f3b08005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:39.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b1a59c640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004d10 0x7f3b08005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b1a59c640 1 -- 172.21.15.148:0/2994761142 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b08003530 con 0x7f3b080039b0 2023-10-30T13:37:39.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b1ad9d640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004d10 0x7f3b08005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:39.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.692+0000 7f3b1a59c640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b080039b0 0x7f3b08003d90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00c210 tx=0x7f3b0c01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5984edb4e8b52e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:39.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c005ae0 con 0x7f3b080039b0 2023-10-30T13:37:39.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0c005c60 con 0x7f3b080039b0 2023-10-30T13:37:39.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b19d9b640 1 -- 172.21.15.148:0/2994761142 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c0a58a0 con 0x7f3b080039b0 2023-10-30T13:37:39.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 -- 172.21.15.148:0/2994761142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b080039b0 msgr2=0x7f3b08003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:39.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 --2- 172.21.15.148:0/2994761142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b080039b0 0x7f3b08003d90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00c210 tx=0x7f3b0c01cba0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b1ad9d640 1 -- 172.21.15.148:0/2994761142 reap_dead start 2023-10-30T13:37:39.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 -- 172.21.15.148:0/2994761142 shutdown_connections 2023-10-30T13:37:39.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 -- 172.21.15.148:0/2994761142 >> 172.21.15.148:0/2994761142 conn(0x7f3b1c0c7910 msgr2=0x7f3b1c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:39.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 -- 172.21.15.148:0/2994761142 shutdown_connections 2023-10-30T13:37:39.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 -- 172.21.15.148:0/2994761142 wait complete. 2023-10-30T13:37:39.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 Processor -- start 2023-10-30T13:37:39.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 -- start start 2023-10-30T13:37:39.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b080039b0 0x7f3b1c158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:39.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c1560c0 0x7f3b1c14caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:39.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b1c156610 0x7f3b1c0c15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:39.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c1ef550 con 0x7f3b1c156610 2023-10-30T13:37:39.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c158650 con 0x7f3b080039b0 2023-10-30T13:37:39.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.696+0000 7f3b20c32640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0c9150 con 0x7f3b1c1560c0 2023-10-30T13:37:39.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b1c156610 0x7f3b1c0c15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:39.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b1c156610 0x7f3b1c0c15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39210/0 (socket says 172.21.15.148:39210) 2023-10-30T13:37:39.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1ad9d640 1 -- 172.21.15.148:0/3443022661 learned_addr learned my addr 172.21.15.148:0/3443022661 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:39.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b13fff640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c1560c0 0x7f3b1c14caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:39.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1a59c640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b080039b0 0x7f3b1c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:39.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1ad9d640 1 -- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c1560c0 msgr2=0x7f3b1c14caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:39.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1ad9d640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c1560c0 0x7f3b1c14caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1ad9d640 1 -- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b080039b0 msgr2=0x7f3b1c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:39.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1ad9d640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b080039b0 0x7f3b1c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1ad9d640 1 -- 172.21.15.148:0/3443022661 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c1eebf0 con 0x7f3b1c156610 2023-10-30T13:37:39.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1a59c640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b080039b0 0x7f3b1c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:39.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b13fff640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c1560c0 0x7f3b1c14caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:39.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b1ad9d640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b1c156610 0x7f3b1c0c15e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c06f0d0 tx=0x7f3b14010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:39.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b12ffd640 1 -- 172.21.15.148:0/3443022661 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1400e050 con 0x7f3b1c156610 2023-10-30T13:37:39.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b12ffd640 1 -- 172.21.15.148:0/3443022661 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1400f040 con 0x7f3b1c156610 2023-10-30T13:37:39.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b12ffd640 1 -- 172.21.15.148:0/3443022661 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b140039a0 con 0x7f3b1c156610 2023-10-30T13:37:39.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c1eee90 con 0x7f3b1c156610 2023-10-30T13:37:39.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c14d030 con 0x7f3b1c156610 2023-10-30T13:37:39.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c04e170 con 0x7f3b1c156610 2023-10-30T13:37:39.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b12ffd640 1 -- 172.21.15.148:0/3443022661 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b14015440 con 0x7f3b1c156610 2023-10-30T13:37:39.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.700+0000 7f3b12ffd640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3afc0664a0 0x7f3afc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:39.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.704+0000 7f3b12ffd640 1 -- 172.21.15.148:0/3443022661 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f3b14084da0 con 0x7f3b1c156610 2023-10-30T13:37:39.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.704+0000 7f3b1a59c640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3afc0664a0 0x7f3afc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:39.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.704+0000 7f3b1a59c640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3afc0664a0 0x7f3afc068960 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0ab000 tx=0x7f3b0c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:39.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.708+0000 7f3b12ffd640 1 -- 172.21.15.148:0/3443022661 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b14051cc0 con 0x7f3b1c156610 2023-10-30T13:37:39.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.948+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b1c073e10 con 0x7f3b1c156610 2023-10-30T13:37:39.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.952+0000 7f3b12ffd640 1 -- 172.21.15.148:0/3443022661 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f3b140552e0 con 0x7f3b1c156610 2023-10-30T13:37:39.952 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:39.952 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:34.820690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:39.953 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 109 2023-10-30T13:37:39.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3afc0664a0 msgr2=0x7f3afc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:39.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3afc0664a0 0x7f3afc068960 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0ab000 tx=0x7f3b0c0a7070 comp rx=0 tx=0).stop 2023-10-30T13:37:39.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b1c156610 msgr2=0x7f3b1c0c15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:39.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b1c156610 0x7f3b1c0c15e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c06f0d0 tx=0x7f3b14010590 comp rx=0 tx=0).stop 2023-10-30T13:37:39.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 shutdown_connections 2023-10-30T13:37:39.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3afc0664a0 0x7f3afc068960 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b1c156610 0x7f3b1c0c15e0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b1c1560c0 0x7f3b1c14caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 --2- 172.21.15.148:0/3443022661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b080039b0 0x7f3b1c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:39.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 >> 172.21.15.148:0/3443022661 conn(0x7f3b1c0c7910 msgr2=0x7f3b1c0c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:39.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 shutdown_connections 2023-10-30T13:37:39.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:39.956+0000 7f3b20c32640 1 -- 172.21.15.148:0/3443022661 wait complete. 2023-10-30T13:37:39.975 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:41.976 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.120+0000 7f403a49c640 1 Processor -- start 2023-10-30T13:37:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f403a49c640 1 -- start start 2023-10-30T13:37:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f403a49c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f40341538a0 0x7f403415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f403a49c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40340ca0e0 0x7f403415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f403a49c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340cb800 con 0x7f40340cf160 2023-10-30T13:37:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f403a49c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403415dc30 con 0x7f40341538a0 2023-10-30T13:37:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f403a49c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40340cb330 con 0x7f40340ca0e0 2023-10-30T13:37:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4038a39640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f40340cf160 0x7f40340cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44626/0 (socket says 172.21.15.148:44626) 2023-10-30T13:37:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4038a39640 1 -- 172.21.15.148:0/284792637 learned_addr learned my addr 172.21.15.148:0/284792637 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4033fff640 1 --2- 172.21.15.148:0/284792637 >> v2:172.21.15.150:3300/0 conn(0x7f40341538a0 0x7f403415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f40337fe640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40340ca0e0 0x7f403415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396495005 0 0) 0x7f40340cb800 con 0x7f40340cf160 2023-10-30T13:37:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401c003530 con 0x7f40340cf160 2023-10-30T13:37:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1404121619 0 0) 0x7f401c003530 con 0x7f40340cf160 2023-10-30T13:37:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40340ca0e0 msgr2=0x7f403415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40340ca0e0 0x7f403415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 >> v2:172.21.15.150:3300/0 conn(0x7f40341538a0 msgr2=0x7f403415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 --2- 172.21.15.148:0/284792637 >> v2:172.21.15.150:3300/0 conn(0x7f40341538a0 0x7f403415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340cad70 con 0x7f40340cf160 2023-10-30T13:37:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f402c001a00 con 0x7f40340cf160 2023-10-30T13:37:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 >> v1:172.21.15.148:6789/0 conn(0x7f40340cf160 legacy=0x7f40340cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f401c003980 0x7f401c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f401c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40340ca0e0 0x7f401c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403415dc30 con 0x7f401c0047d0 2023-10-30T13:37:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40340cb330 con 0x7f401c003980 2023-10-30T13:37:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c00b9c0 con 0x7f40340ca0e0 2023-10-30T13:37:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f40337fe640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f401c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4033fff640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f401c003980 0x7f401c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4038a39640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40340ca0e0 0x7f401c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f40337fe640 1 -- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40340ca0e0 msgr2=0x7f401c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f40337fe640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40340ca0e0 0x7f401c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f40337fe640 1 -- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f401c003980 msgr2=0x7f401c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f40337fe640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f401c003980 0x7f401c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f40337fe640 1 -- 172.21.15.148:0/284792637 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c003530 con 0x7f401c0047d0 2023-10-30T13:37:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.124+0000 7f4038a39640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40340ca0e0 0x7f401c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f40337fe640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f401c004c50 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f4028009ae0 tx=0x7f402801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb294829d9dfbb06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280a5560 con 0x7f401c0047d0 2023-10-30T13:37:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40280a56c0 con 0x7f401c0047d0 2023-10-30T13:37:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f4032ffd640 1 -- 172.21.15.148:0/284792637 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280a5a70 con 0x7f401c0047d0 2023-10-30T13:37:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f403a49c640 1 -- 172.21.15.148:0/284792637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 msgr2=0x7f401c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f403a49c640 1 --2- 172.21.15.148:0/284792637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f401c004c50 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f4028009ae0 tx=0x7f402801ca50 comp rx=0 tx=0).stop 2023-10-30T13:37:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f4038a39640 1 -- 172.21.15.148:0/284792637 reap_dead start 2023-10-30T13:37:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f403a49c640 1 -- 172.21.15.148:0/284792637 shutdown_connections 2023-10-30T13:37:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f403a49c640 1 -- 172.21.15.148:0/284792637 >> 172.21.15.148:0/284792637 conn(0x7f403406bfe0 msgr2=0x7f403406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f403a49c640 1 -- 172.21.15.148:0/284792637 shutdown_connections 2023-10-30T13:37:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.128+0000 7f403a49c640 1 -- 172.21.15.148:0/284792637 wait complete. 2023-10-30T13:37:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f403a49c640 1 Processor -- start 2023-10-30T13:37:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f403a49c640 1 -- start start 2023-10-30T13:37:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f403a49c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f40341583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f403a49c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4034157b30 0x7f40341e30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f403a49c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40341e35e0 0x7f4034145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f403a49c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40341e2c80 con 0x7f401c0047d0 2023-10-30T13:37:42.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f403a49c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40340c5a80 con 0x7f40341e35e0 2023-10-30T13:37:42.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f403a49c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40341e3a60 con 0x7f4034157b30 2023-10-30T13:37:42.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4033fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f40341583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:42.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4033fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f40341583f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35994/0 (socket says 172.21.15.148:35994) 2023-10-30T13:37:42.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4033fff640 1 -- 172.21.15.148:0/1466956657 learned_addr learned my addr 172.21.15.148:0/1466956657 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:42.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4038a39640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40341e35e0 0x7f4034145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:42.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f40337fe640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4034157b30 0x7f40341e30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:42.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4033fff640 1 -- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4034157b30 msgr2=0x7f40341e30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:42.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4033fff640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4034157b30 0x7f40341e30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4033fff640 1 -- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40341e35e0 msgr2=0x7f4034145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4033fff640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40341e35e0 0x7f4034145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4033fff640 1 -- 172.21.15.148:0/1466956657 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40341f2b40 con 0x7f401c0047d0 2023-10-30T13:37:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4038a39640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40341e35e0 0x7f4034145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:42.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4033fff640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f40341583f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4024011aa0 tx=0x7f4024011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.132+0000 7f4030ff9640 1 -- 172.21.15.148:0/1466956657 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402400fb20 con 0x7f401c0047d0 2023-10-30T13:37:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.136+0000 7f4030ff9640 1 -- 172.21.15.148:0/1466956657 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402400fc80 con 0x7f401c0047d0 2023-10-30T13:37:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.136+0000 7f4030ff9640 1 -- 172.21.15.148:0/1466956657 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024014440 con 0x7f401c0047d0 2023-10-30T13:37:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.136+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40341f2e20 con 0x7f401c0047d0 2023-10-30T13:37:42.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.136+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40341f33e0 con 0x7f401c0047d0 2023-10-30T13:37:42.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.136+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403404e170 con 0x7f401c0047d0 2023-10-30T13:37:42.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.136+0000 7f4030ff9640 1 -- 172.21.15.148:0/1466956657 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40240145c0 con 0x7f401c0047d0 2023-10-30T13:37:42.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.136+0000 7f4030ff9640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40200664a0 0x7f4020068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:42.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.136+0000 7f4030ff9640 1 -- 172.21.15.148:0/1466956657 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f4024084cf0 con 0x7f401c0047d0 2023-10-30T13:37:42.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.136+0000 7f40337fe640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40200664a0 0x7f4020068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:42.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.140+0000 7f40337fe640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40200664a0 0x7f4020068960 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f40280af000 tx=0x7f40280046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:42.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.140+0000 7f4030ff9640 1 -- 172.21.15.148:0/1466956657 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4024051b80 con 0x7f401c0047d0 2023-10-30T13:37:42.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.332+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40340b1dd0 con 0x7f401c0047d0 2023-10-30T13:37:42.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.332+0000 7f4030ff9640 1 -- 172.21.15.148:0/1466956657 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f4024022020 con 0x7f401c0047d0 2023-10-30T13:37:42.331 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:42.331 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:34.820690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:42.332 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 109 2023-10-30T13:37:42.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40200664a0 msgr2=0x7f4020068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:42.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40200664a0 0x7f4020068960 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f40280af000 tx=0x7f40280046d0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 msgr2=0x7f40341583f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:42.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f40341583f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4024011aa0 tx=0x7f4024011f60 comp rx=0 tx=0).stop 2023-10-30T13:37:42.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 shutdown_connections 2023-10-30T13:37:42.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40200664a0 0x7f4020068960 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40341e35e0 0x7f4034145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4034157b30 0x7f40341e30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 --2- 172.21.15.148:0/1466956657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f401c0047d0 0x7f40341583f0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:42.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 >> 172.21.15.148:0/1466956657 conn(0x7f403406bfe0 msgr2=0x7f40341526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:42.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 shutdown_connections 2023-10-30T13:37:42.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:42.336+0000 7f403a49c640 1 -- 172.21.15.148:0/1466956657 wait complete. 2023-10-30T13:37:42.353 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:44.354 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:44.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.512+0000 7fa76c86b640 1 Processor -- start 2023-10-30T13:37:44.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa76c86b640 1 -- start start 2023-10-30T13:37:44.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa76c86b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa76815d6f0 0x7fa76815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:44.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa76c86b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7680cb800 con 0x7fa7680cf160 2023-10-30T13:37:44.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa76c86b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7680cb330 con 0x7fa7681538a0 2023-10-30T13:37:44.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa76c86b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7680cad70 con 0x7fa76815d6f0 2023-10-30T13:37:44.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa766d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa7680cf160 0x7fa768164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44628/0 (socket says 172.21.15.148:44628) 2023-10-30T13:37:44.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa766d9d640 1 -- 172.21.15.148:0/326035443 learned_addr learned my addr 172.21.15.148:0/326035443 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:44.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa75ffff640 1 --2- 172.21.15.148:0/326035443 >> v2:172.21.15.150:3300/0 conn(0x7fa76815d6f0 0x7fa76815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:44.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa765d9b640 1 -- 172.21.15.148:0/326035443 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476896611 0 0) 0x7fa7680cb800 con 0x7fa7680cf160 2023-10-30T13:37:44.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa765d9b640 1 -- 172.21.15.148:0/326035443 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa754003530 con 0x7fa7680cf160 2023-10-30T13:37:44.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa765d9b640 1 -- 172.21.15.148:0/326035443 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809321670 0 0) 0x7fa754003530 con 0x7fa7680cf160 2023-10-30T13:37:44.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa75ffff640 1 -- 172.21.15.148:0/326035443 >> v1:172.21.15.150:6789/0 conn(0x7fa7681538a0 legacy=0x7fa7680ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:44.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa75ffff640 1 -- 172.21.15.148:0/326035443 >> v1:172.21.15.148:6789/0 conn(0x7fa7680cf160 legacy=0x7fa768164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:44.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.516+0000 7fa75ffff640 1 -- 172.21.15.148:0/326035443 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7680cb050 con 0x7fa76815d6f0 2023-10-30T13:37:44.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa75ffff640 1 --2- 172.21.15.148:0/326035443 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815d6f0 0x7fa76815a090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa7500097c0 tx=0x7fa75001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=698437e828d5958a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:44.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa765d9b640 1 -- 172.21.15.148:0/326035443 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760001a00 con 0x7fa76815d6f0 2023-10-30T13:37:44.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa765d9b640 1 -- 172.21.15.148:0/326035443 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7500a8610 con 0x7fa76815d6f0 2023-10-30T13:37:44.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa765d9b640 1 -- 172.21.15.148:0/326035443 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7500a8980 con 0x7fa76815d6f0 2023-10-30T13:37:44.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa76c86b640 1 -- 172.21.15.148:0/326035443 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815d6f0 msgr2=0x7fa76815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:44.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa76c86b640 1 --2- 172.21.15.148:0/326035443 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815d6f0 0x7fa76815a090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa7500097c0 tx=0x7fa75001eb90 comp rx=0 tx=0).stop 2023-10-30T13:37:44.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa76c86b640 1 -- 172.21.15.148:0/326035443 shutdown_connections 2023-10-30T13:37:44.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa76c86b640 1 --2- 172.21.15.148:0/326035443 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815d6f0 0x7fa76815a090 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:44.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa76c86b640 1 -- 172.21.15.148:0/326035443 >> 172.21.15.148:0/326035443 conn(0x7fa76806bfe0 msgr2=0x7fa76806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:44.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa76c86b640 1 -- 172.21.15.148:0/326035443 shutdown_connections 2023-10-30T13:37:44.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.520+0000 7fa76c86b640 1 -- 172.21.15.148:0/326035443 wait complete. 2023-10-30T13:37:44.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 Processor -- start 2023-10-30T13:37:44.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 -- start start 2023-10-30T13:37:44.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7680cf160 0x7fa7680cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7681538a0 0x7fa7680cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815d6f0 0x7fa76814cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680cd2b0 con 0x7fa7680cf160 2023-10-30T13:37:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76814e880 con 0x7fa76815d6f0 2023-10-30T13:37:44.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa768158ab0 con 0x7fa7681538a0 2023-10-30T13:37:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7680cf160 0x7fa7680cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7680cf160 0x7fa7680cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36008/0 (socket says 172.21.15.148:36008) 2023-10-30T13:37:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76659c640 1 -- 172.21.15.148:0/2295572719 learned_addr learned my addr 172.21.15.148:0/2295572719 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa766d9d640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815d6f0 0x7fa76814cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:44.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa75ffff640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7681538a0 0x7fa7680cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76659c640 1 -- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7681538a0 msgr2=0x7fa7680cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76659c640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7681538a0 0x7fa7680cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76659c640 1 -- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815d6f0 msgr2=0x7fa76814cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76659c640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815d6f0 0x7fa76814cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:44.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76659c640 1 -- 172.21.15.148:0/2295572719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa750009420 con 0x7fa7680cf160 2023-10-30T13:37:44.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76659c640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7680cf160 0x7fa7680cd450 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fa75800c500 tx=0x7fa75800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:44.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa75effd640 1 -- 172.21.15.148:0/2295572719 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758005220 con 0x7fa7680cf160 2023-10-30T13:37:44.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa75effd640 1 -- 172.21.15.148:0/2295572719 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa758005380 con 0x7fa7680cf160 2023-10-30T13:37:44.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa75effd640 1 -- 172.21.15.148:0/2295572719 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758007270 con 0x7fa7680cf160 2023-10-30T13:37:44.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa768157f00 con 0x7fa7680cf160 2023-10-30T13:37:44.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.524+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa768158180 con 0x7fa7680cf160 2023-10-30T13:37:44.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.528+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76804e170 con 0x7fa7680cf160 2023-10-30T13:37:44.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.528+0000 7fa75effd640 1 -- 172.21.15.148:0/2295572719 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7580073d0 con 0x7fa7680cf160 2023-10-30T13:37:44.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.528+0000 7fa75effd640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7480664a0 0x7fa748068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:44.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.528+0000 7fa75effd640 1 -- 172.21.15.148:0/2295572719 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fa758083a10 con 0x7fa7680cf160 2023-10-30T13:37:44.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.528+0000 7fa75ffff640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7480664a0 0x7fa748068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:44.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.528+0000 7fa75ffff640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7480664a0 0x7fa748068960 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa7500b2000 tx=0x7fa7500aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:44.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.532+0000 7fa75effd640 1 -- 172.21.15.148:0/2295572719 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7580508b0 con 0x7fa7680cf160 2023-10-30T13:37:44.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.740+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa76815c8c0 con 0x7fa7680cf160 2023-10-30T13:37:44.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.740+0000 7fa75effd640 1 -- 172.21.15.148:0/2295572719 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5719 (secure 0 0 0) 0x7fa758053ed0 con 0x7fa7680cf160 2023-10-30T13:37:44.741 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:44.741 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:34.820690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 109 2023-10-30T13:37:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7480664a0 msgr2=0x7fa748068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7480664a0 0x7fa748068960 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa7500b2000 tx=0x7fa7500aa070 comp rx=0 tx=0).stop 2023-10-30T13:37:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7680cf160 msgr2=0x7fa7680cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7680cf160 0x7fa7680cd450 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fa75800c500 tx=0x7fa75800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:37:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 shutdown_connections 2023-10-30T13:37:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7480664a0 0x7fa748068960 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76815d6f0 0x7fa76814cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7681538a0 0x7fa7680cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 --2- 172.21.15.148:0/2295572719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7680cf160 0x7fa7680cd450 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 >> 172.21.15.148:0/2295572719 conn(0x7fa76806bfe0 msgr2=0x7fa76815b640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 shutdown_connections 2023-10-30T13:37:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:44.744+0000 7fa76c86b640 1 -- 172.21.15.148:0/2295572719 wait complete. 2023-10-30T13:37:44.763 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:46.765 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:46.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.908+0000 7efe71c46640 1 Processor -- start 2023-10-30T13:37:46.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.908+0000 7efe71c46640 1 -- start start 2023-10-30T13:37:46.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe71c46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c06f7a0 0x7efe6c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:46.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe71c46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efe6c0c6bd0 0x7efe6c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:46.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe71c46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c06a3d0 con 0x7efe6c071110 2023-10-30T13:37:46.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe71c46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6c0700c0 con 0x7efe6c0c6bd0 2023-10-30T13:37:46.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe71c46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c066cf0 con 0x7efe6c06f7a0 2023-10-30T13:37:46.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe6affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efe6c0c6bd0 0x7efe6c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:46.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe6bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efe6c071110 0x7efe6c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:37350/0 (socket says 172.21.15.148:37350) 2023-10-30T13:37:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe6affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efe6c0c6bd0 0x7efe6c069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:41930/0 (socket says 172.21.15.148:41930) 2023-10-30T13:37:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe6bfff640 1 -- 172.21.15.148:0/4240752573 learned_addr learned my addr 172.21.15.148:0/4240752573 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe6b7fe640 1 --2- 172.21.15.148:0/4240752573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c06f7a0 0x7efe6c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe6affd640 1 -- 172.21.15.148:0/4240752573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c06f7a0 msgr2=0x7efe6c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:46.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe6affd640 1 --2- 172.21.15.148:0/4240752573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c06f7a0 0x7efe6c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:46.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe6affd640 1 -- 172.21.15.148:0/4240752573 >> v1:172.21.15.150:6789/0 conn(0x7efe6c071110 legacy=0x7efe6c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:46.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.912+0000 7efe6affd640 1 -- 172.21.15.148:0/4240752573 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c0c70d0 con 0x7efe6c0c6bd0 2023-10-30T13:37:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe6affd640 1 --2- 172.21.15.148:0/4240752573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0c6bd0 0x7efe6c069e90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efe600097c0 tx=0x7efe6001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ddaccfa1826a26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe6a7fc640 1 -- 172.21.15.148:0/4240752573 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe600a8970 con 0x7efe6c0c6bd0 2023-10-30T13:37:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe6a7fc640 1 -- 172.21.15.148:0/4240752573 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe600a8ad0 con 0x7efe6c0c6bd0 2023-10-30T13:37:46.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe6a7fc640 1 -- 172.21.15.148:0/4240752573 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe600a8e60 con 0x7efe6c0c6bd0 2023-10-30T13:37:46.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe71c46640 1 -- 172.21.15.148:0/4240752573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0c6bd0 msgr2=0x7efe6c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:46.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe71c46640 1 --2- 172.21.15.148:0/4240752573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0c6bd0 0x7efe6c069e90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7efe600097c0 tx=0x7efe6001ec70 comp rx=0 tx=0).stop 2023-10-30T13:37:46.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe71c46640 1 -- 172.21.15.148:0/4240752573 shutdown_connections 2023-10-30T13:37:46.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe71c46640 1 --2- 172.21.15.148:0/4240752573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c0c6bd0 0x7efe6c069e90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:46.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe71c46640 1 --2- 172.21.15.148:0/4240752573 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c06f7a0 0x7efe6c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:46.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe71c46640 1 -- 172.21.15.148:0/4240752573 >> 172.21.15.148:0/4240752573 conn(0x7efe6c0c7910 msgr2=0x7efe6c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:46.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe71c46640 1 -- 172.21.15.148:0/4240752573 shutdown_connections 2023-10-30T13:37:46.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.916+0000 7efe71c46640 1 -- 172.21.15.148:0/4240752573 wait complete. 2023-10-30T13:37:46.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 Processor -- start 2023-10-30T13:37:46.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 -- start start 2023-10-30T13:37:46.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c06f7a0 0x7efe6c1ed6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:46.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c071110 0x7efe6c15b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:46.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0c6bd0 0x7efe6c145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:46.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c1ed2d0 con 0x7efe6c071110 2023-10-30T13:37:46.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c14d5d0 con 0x7efe6c06f7a0 2023-10-30T13:37:46.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c0c6340 con 0x7efe6c0c6bd0 2023-10-30T13:37:46.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c071110 0x7efe6c15b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:46.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c071110 0x7efe6c15b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36018/0 (socket says 172.21.15.148:36018) 2023-10-30T13:37:46.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6affd640 1 -- 172.21.15.148:0/4039465338 learned_addr learned my addr 172.21.15.148:0/4039465338 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:46.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6b7fe640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c06f7a0 0x7efe6c1ed6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:46.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6bfff640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0c6bd0 0x7efe6c145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:46.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6affd640 1 -- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0c6bd0 msgr2=0x7efe6c145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:46.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6affd640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0c6bd0 0x7efe6c145dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:46.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6affd640 1 -- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c06f7a0 msgr2=0x7efe6c1ed6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:46.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6affd640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c06f7a0 0x7efe6c1ed6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:46.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6affd640 1 -- 172.21.15.148:0/4039465338 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe60009420 con 0x7efe6c071110 2023-10-30T13:37:46.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe6affd640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c071110 0x7efe6c15b3b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7efe6001b880 tx=0x7efe6001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:46.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe53fff640 1 -- 172.21.15.148:0/4039465338 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe600b1dc0 con 0x7efe6c071110 2023-10-30T13:37:46.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe53fff640 1 -- 172.21.15.148:0/4039465338 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe600b9410 con 0x7efe6c071110 2023-10-30T13:37:46.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe53fff640 1 -- 172.21.15.148:0/4039465338 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe600b97c0 con 0x7efe6c071110 2023-10-30T13:37:46.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c154d10 con 0x7efe6c071110 2023-10-30T13:37:46.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6c15a9c0 con 0x7efe6c071110 2023-10-30T13:37:46.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.920+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6c1ed410 con 0x7efe6c071110 2023-10-30T13:37:46.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.924+0000 7efe53fff640 1 -- 172.21.15.148:0/4039465338 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe600b7040 con 0x7efe6c071110 2023-10-30T13:37:46.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.924+0000 7efe53fff640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe580664d0 0x7efe58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:46.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.924+0000 7efe53fff640 1 -- 172.21.15.148:0/4039465338 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7efe60121be0 con 0x7efe6c071110 2023-10-30T13:37:46.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.924+0000 7efe6b7fe640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe580664d0 0x7efe58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:46.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.924+0000 7efe6b7fe640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe580664d0 0x7efe58068990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009ac0 tx=0x7efe5c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:46.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:46.928+0000 7efe53fff640 1 -- 172.21.15.148:0/4039465338 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe6c1ed410 con 0x7efe6c071110 2023-10-30T13:37:47.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.108+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe6c1ed410 con 0x7efe6c071110 2023-10-30T13:37:47.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.108+0000 7efe53fff640 1 -- 172.21.15.148:0/4039465338 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5723 (secure 0 0 0) 0x7efe6c1ed410 con 0x7efe6c071110 2023-10-30T13:37:47.108 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:47.109 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:46.137362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:47.109 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 110 2023-10-30T13:37:47.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe580664d0 msgr2=0x7efe58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:47.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe580664d0 0x7efe58068990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009ac0 tx=0x7efe5c009190 comp rx=0 tx=0).stop 2023-10-30T13:37:47.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c071110 msgr2=0x7efe6c15b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:47.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c071110 0x7efe6c15b3b0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7efe6001b880 tx=0x7efe6001ba10 comp rx=0 tx=0).stop 2023-10-30T13:37:47.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 shutdown_connections 2023-10-30T13:37:47.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe580664d0 0x7efe58068990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:47.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe6c071110 0x7efe6c15b3b0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:47.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe6c0c6bd0 0x7efe6c145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:47.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 --2- 172.21.15.148:0/4039465338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe6c06f7a0 0x7efe6c1ed6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:47.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 >> 172.21.15.148:0/4039465338 conn(0x7efe6c0c7910 msgr2=0x7efe6c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:47.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 shutdown_connections 2023-10-30T13:37:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:47.112+0000 7efe71c46640 1 -- 172.21.15.148:0/4039465338 wait complete. 2023-10-30T13:37:47.129 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:49.131 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:49.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.279+0000 7f3c6e246640 1 Processor -- start 2023-10-30T13:37:49.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.279+0000 7f3c6e246640 1 -- start start 2023-10-30T13:37:49.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c6e246640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c680c05d0 0x7f3c6813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:49.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c6e246640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c680c0a50 con 0x7f3c680c6b80 2023-10-30T13:37:49.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c6e246640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6806f6e0 con 0x7f3c680c6f60 2023-10-30T13:37:49.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c6e246640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6806f9c0 con 0x7f3c680c05d0 2023-10-30T13:37:49.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c677fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3c680c6b80 0x7f3c680bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44640/0 (socket says 172.21.15.148:44640) 2023-10-30T13:37:49.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c677fe640 1 -- 172.21.15.148:0/2808379506 learned_addr learned my addr 172.21.15.148:0/2808379506 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:49.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c66ffd640 1 --2- 172.21.15.148:0/2808379506 >> v2:172.21.15.150:3300/0 conn(0x7f3c680c05d0 0x7f3c6813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:49.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786028839 0 0) 0x7f3c680c0a50 con 0x7f3c680c6b80 2023-10-30T13:37:49.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c50003530 con 0x7f3c680c6b80 2023-10-30T13:37:49.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 949382303 0 0) 0x7f3c50003530 con 0x7f3c680c6b80 2023-10-30T13:37:49.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 >> v2:172.21.15.150:3300/0 conn(0x7f3c680c05d0 msgr2=0x7f3c6813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:49.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 --2- 172.21.15.148:0/2808379506 >> v2:172.21.15.150:3300/0 conn(0x7f3c680c05d0 0x7f3c6813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 >> v1:172.21.15.150:6789/0 conn(0x7f3c680c6f60 legacy=0x7f3c681566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:49.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6806fca0 con 0x7f3c680c6b80 2023-10-30T13:37:49.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f3c58001680 con 0x7f3c680c6b80 2023-10-30T13:37:49.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 >> v1:172.21.15.148:6789/0 conn(0x7f3c680c6b80 legacy=0x7f3c680bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:49.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c50003980 0x7f3c50003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:49.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c500047d0 0x7f3c50004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:49.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c50005190 0x7f3c5000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:49.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6806f9c0 con 0x7f3c50003980 2023-10-30T13:37:49.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5000c030 con 0x7f3c50005190 2023-10-30T13:37:49.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5000c1f0 con 0x7f3c500047d0 2023-10-30T13:37:49.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c677fe640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c50003980 0x7f3c50003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:49.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c66ffd640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c500047d0 0x7f3c50004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:49.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c67fff640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c50005190 0x7f3c5000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:49.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c677fe640 1 -- 172.21.15.148:0/2808379506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c500047d0 msgr2=0x7f3c50004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:49.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c677fe640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c500047d0 0x7f3c50004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c677fe640 1 -- 172.21.15.148:0/2808379506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c50005190 msgr2=0x7f3c5000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:49.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c677fe640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c50005190 0x7f3c5000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c677fe640 1 -- 172.21.15.148:0/2808379506 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c50003530 con 0x7f3c50003980 2023-10-30T13:37:49.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.283+0000 7f3c67fff640 1 -- 172.21.15.148:0/2808379506 reap_dead start 2023-10-30T13:37:49.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c677fe640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c50003980 0x7f3c50003da0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800c240 tx=0x7f3c5801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c197f303b74a6094 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:49.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58005ae0 con 0x7f3c50003980 2023-10-30T13:37:49.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c58005c60 con 0x7f3c50003980 2023-10-30T13:37:49.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c667fc640 1 -- 172.21.15.148:0/2808379506 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c580a5830 con 0x7f3c50003980 2023-10-30T13:37:49.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c6e246640 1 -- 172.21.15.148:0/2808379506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c50003980 msgr2=0x7f3c50003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:49.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c6e246640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c50003980 0x7f3c50003da0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800c240 tx=0x7f3c5801cc40 comp rx=0 tx=0).stop 2023-10-30T13:37:49.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c6e246640 1 -- 172.21.15.148:0/2808379506 shutdown_connections 2023-10-30T13:37:49.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c6e246640 1 --2- 172.21.15.148:0/2808379506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c50003980 0x7f3c50003da0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c6e246640 1 -- 172.21.15.148:0/2808379506 >> 172.21.15.148:0/2808379506 conn(0x7f3c680c7910 msgr2=0x7f3c680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:49.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c6e246640 1 -- 172.21.15.148:0/2808379506 shutdown_connections 2023-10-30T13:37:49.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.287+0000 7f3c6e246640 1 -- 172.21.15.148:0/2808379506 wait complete. 2023-10-30T13:37:49.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 Processor -- start 2023-10-30T13:37:49.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 -- start start 2023-10-30T13:37:49.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c50003980 0x7f3c68155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:49.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c681534a0 0x7f3c68154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:49.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c68154e90 0x7f3c6813d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:49.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c68156560 con 0x7f3c681534a0 2023-10-30T13:37:49.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c680ae250 con 0x7f3c50003980 2023-10-30T13:37:49.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c680af870 con 0x7f3c68154e90 2023-10-30T13:37:49.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c66ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c681534a0 0x7f3c68154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:49.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c66ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c681534a0 0x7f3c68154950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36032/0 (socket says 172.21.15.148:36032) 2023-10-30T13:37:49.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c66ffd640 1 -- 172.21.15.148:0/53824439 learned_addr learned my addr 172.21.15.148:0/53824439 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:49.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c677fe640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c50003980 0x7f3c68155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:49.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c67fff640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c68154e90 0x7f3c6813d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c66ffd640 1 -- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c68154e90 msgr2=0x7f3c6813d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c66ffd640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c68154e90 0x7f3c6813d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c66ffd640 1 -- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c50003980 msgr2=0x7f3c68155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c66ffd640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c50003980 0x7f3c68155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c66ffd640 1 -- 172.21.15.148:0/53824439 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c68155340 con 0x7f3c681534a0 2023-10-30T13:37:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c66ffd640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c681534a0 0x7f3c68154950 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c011530 tx=0x7f3c5c0119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c4ffff640 1 -- 172.21.15.148:0/53824439 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c015870 con 0x7f3c681534a0 2023-10-30T13:37:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c4ffff640 1 -- 172.21.15.148:0/53824439 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c0159d0 con 0x7f3c681534a0 2023-10-30T13:37:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c4ffff640 1 -- 172.21.15.148:0/53824439 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c015d40 con 0x7f3c681534a0 2023-10-30T13:37:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6813cc40 con 0x7f3c681534a0 2023-10-30T13:37:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.291+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c680aef40 con 0x7f3c681534a0 2023-10-30T13:37:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.295+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c68072560 con 0x7f3c681534a0 2023-10-30T13:37:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.295+0000 7f3c4ffff640 1 -- 172.21.15.148:0/53824439 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c5c013400 con 0x7f3c681534a0 2023-10-30T13:37:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.295+0000 7f3c4ffff640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c540664d0 0x7f3c54068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.295+0000 7f3c4ffff640 1 -- 172.21.15.148:0/53824439 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f3c5c084850 con 0x7f3c681534a0 2023-10-30T13:37:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.295+0000 7f3c677fe640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c540664d0 0x7f3c54068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.295+0000 7f3c677fe640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c540664d0 0x7f3c54068990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3c580ab000 tx=0x7f3c580b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.299+0000 7f3c4ffff640 1 -- 172.21.15.148:0/53824439 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c5c051650 con 0x7f3c681534a0 2023-10-30T13:37:49.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c6804e170 con 0x7f3c681534a0 2023-10-30T13:37:49.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c4ffff640 1 -- 172.21.15.148:0/53824439 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f3c5c054c70 con 0x7f3c681534a0 2023-10-30T13:37:49.483 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:49.483 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:46.137362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:49.484 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 110 2023-10-30T13:37:49.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c540664d0 msgr2=0x7f3c54068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:49.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c540664d0 0x7f3c54068990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3c580ab000 tx=0x7f3c580b1040 comp rx=0 tx=0).stop 2023-10-30T13:37:49.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c681534a0 msgr2=0x7f3c68154950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:49.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c681534a0 0x7f3c68154950 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c011530 tx=0x7f3c5c0119f0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 shutdown_connections 2023-10-30T13:37:49.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c540664d0 0x7f3c54068990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c681534a0 0x7f3c68154950 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c68154e90 0x7f3c6813d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 --2- 172.21.15.148:0/53824439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c50003980 0x7f3c68155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:49.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 >> 172.21.15.148:0/53824439 conn(0x7f3c680c7910 msgr2=0x7f3c680c1250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:49.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 shutdown_connections 2023-10-30T13:37:49.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:49.483+0000 7f3c6e246640 1 -- 172.21.15.148:0/53824439 wait complete. 2023-10-30T13:37:49.504 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 110, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 110, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7005": { "addr": "172.21.15.148:6835/1079809213", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1079809213, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1079809213, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7005, "incarnation": 109, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:37:46.137362+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7005 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:37:51.505 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:51.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ee3b5640 1 Processor -- start 2023-10-30T13:37:51.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ee3b5640 1 -- start start 2023-10-30T13:37:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ee3b5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f97e8153890 0x7f97e8159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ee3b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 0x7f97e80c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ee3b5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e80c54d0 con 0x7f97e8069e90 2023-10-30T13:37:51.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ee3b5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e80c1cd0 con 0x7f97e8153890 2023-10-30T13:37:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ee3b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e80c5a90 con 0x7f97e806ab20 2023-10-30T13:37:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ed3b3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f97e8153890 0x7f97e8159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ed3b3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f97e8153890 0x7f97e8159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36048/0 (socket says 172.21.15.148:36048) 2023-10-30T13:37:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ed3b3640 1 -- 172.21.15.148:0/4282661259 learned_addr learned my addr 172.21.15.148:0/4282661259 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:51.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ecbb2640 1 --2- 172.21.15.148:0/4282661259 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 0x7f97e80c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ed3b3640 1 -- 172.21.15.148:0/4282661259 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 msgr2=0x7f97e80c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ed3b3640 1 --2- 172.21.15.148:0/4282661259 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 0x7f97e80c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ed3b3640 1 -- 172.21.15.148:0/4282661259 >> v1:172.21.15.150:6789/0 conn(0x7f97e8069e90 legacy=0x7f97e815cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:51.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.655+0000 7f97ed3b3640 1 -- 172.21.15.148:0/4282661259 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e8159b70 con 0x7f97e8153890 2023-10-30T13:37:51.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97ed3b3640 1 --2- 172.21.15.148:0/4282661259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8153890 0x7f97e8159630 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009860 tx=0x7f97e401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56481d4dd81546b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:51.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97dffff640 1 -- 172.21.15.148:0/4282661259 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40aa680 con 0x7f97e8153890 2023-10-30T13:37:51.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97dffff640 1 -- 172.21.15.148:0/4282661259 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e40aa7e0 con 0x7f97e8153890 2023-10-30T13:37:51.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97dffff640 1 -- 172.21.15.148:0/4282661259 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40aab70 con 0x7f97e8153890 2023-10-30T13:37:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97ee3b5640 1 -- 172.21.15.148:0/4282661259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8153890 msgr2=0x7f97e8159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97ee3b5640 1 --2- 172.21.15.148:0/4282661259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8153890 0x7f97e8159630 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009860 tx=0x7f97e401c520 comp rx=0 tx=0).stop 2023-10-30T13:37:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97ee3b5640 1 -- 172.21.15.148:0/4282661259 shutdown_connections 2023-10-30T13:37:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97ee3b5640 1 --2- 172.21.15.148:0/4282661259 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 0x7f97e80c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:51.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97ee3b5640 1 --2- 172.21.15.148:0/4282661259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8153890 0x7f97e8159630 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:51.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97ee3b5640 1 -- 172.21.15.148:0/4282661259 >> 172.21.15.148:0/4282661259 conn(0x7f97e80c75d0 msgr2=0x7f97e80c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:51.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97ee3b5640 1 -- 172.21.15.148:0/4282661259 shutdown_connections 2023-10-30T13:37:51.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.659+0000 7f97ee3b5640 1 -- 172.21.15.148:0/4282661259 wait complete. 2023-10-30T13:37:51.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 Processor -- start 2023-10-30T13:37:51.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 -- start start 2023-10-30T13:37:51.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8069e90 0x7f97e81514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:51.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 0x7f97e80c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:51.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97e8153890 0x7f97e80c2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:51.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e8152e60 con 0x7f97e8069e90 2023-10-30T13:37:51.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e814c3b0 con 0x7f97e8153890 2023-10-30T13:37:51.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e814ced0 con 0x7f97e806ab20 2023-10-30T13:37:51.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ed3b3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8069e90 0x7f97e81514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:51.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ed3b3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8069e90 0x7f97e81514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36056/0 (socket says 172.21.15.148:36056) 2023-10-30T13:37:51.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ed3b3640 1 -- 172.21.15.148:0/2988476428 learned_addr learned my addr 172.21.15.148:0/2988476428 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:51.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ecbb2640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 0x7f97e80c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:51.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97edbb4640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97e8153890 0x7f97e80c2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:51.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ed3b3640 1 -- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 msgr2=0x7f97e80c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:51.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ed3b3640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 0x7f97e80c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:51.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ed3b3640 1 -- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97e8153890 msgr2=0x7f97e80c2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:51.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ed3b3640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97e8153890 0x7f97e80c2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:51.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ed3b3640 1 -- 172.21.15.148:0/2988476428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009470 con 0x7f97e8069e90 2023-10-30T13:37:51.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97edbb4640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97e8153890 0x7f97e80c2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:51.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ed3b3640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8069e90 0x7f97e81514e0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f97e40ac040 tx=0x7f97e401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:51.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ddffb640 1 -- 172.21.15.148:0/2988476428 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40a98f0 con 0x7f97e8069e90 2023-10-30T13:37:51.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ddffb640 1 -- 172.21.15.148:0/2988476428 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e40a9a70 con 0x7f97e8069e90 2023-10-30T13:37:51.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ddffb640 1 -- 172.21.15.148:0/2988476428 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40a9e20 con 0x7f97e8069e90 2023-10-30T13:37:51.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e815aff0 con 0x7f97e8069e90 2023-10-30T13:37:51.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.663+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e8152b80 con 0x7f97e8069e90 2023-10-30T13:37:51.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.667+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e804e170 con 0x7f97e8069e90 2023-10-30T13:37:51.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.667+0000 7f97ddffb640 1 -- 172.21.15.148:0/2988476428 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97e40b33f0 con 0x7f97e8069e90 2023-10-30T13:37:51.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.667+0000 7f97ddffb640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97cc0664a0 0x7f97cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:51.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.667+0000 7f97ddffb640 1 -- 172.21.15.148:0/2988476428 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f97e411ef30 con 0x7f97e8069e90 2023-10-30T13:37:51.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.667+0000 7f97ecbb2640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97cc0664a0 0x7f97cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:51.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.667+0000 7f97ecbb2640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97cc0664a0 0x7f97cc068960 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f97e80ca080 tx=0x7f97d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:51.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.671+0000 7f97ddffb640 1 -- 172.21.15.148:0/2988476428 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97e40ebe60 con 0x7f97e8069e90 2023-10-30T13:37:51.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.855+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97e8164da0 con 0x7f97e8069e90 2023-10-30T13:37:51.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ddffb640 1 -- 172.21.15.148:0/2988476428 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f97e40c3070 con 0x7f97e8069e90 2023-10-30T13:37:51.857 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:51.857 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:46.137362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:51.858 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 110 2023-10-30T13:37:51.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97cc0664a0 msgr2=0x7f97cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:51.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97cc0664a0 0x7f97cc068960 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f97e80ca080 tx=0x7f97d8008040 comp rx=0 tx=0).stop 2023-10-30T13:37:51.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8069e90 msgr2=0x7f97e81514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:51.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8069e90 0x7f97e81514e0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f97e40ac040 tx=0x7f97e401ad60 comp rx=0 tx=0).stop 2023-10-30T13:37:51.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 shutdown_connections 2023-10-30T13:37:51.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97cc0664a0 0x7f97cc068960 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:51.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97e8153890 0x7f97e80c2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:51.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97e806ab20 0x7f97e80c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:51.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 --2- 172.21.15.148:0/2988476428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97e8069e90 0x7f97e81514e0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:51.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 >> 172.21.15.148:0/2988476428 conn(0x7f97e80c75d0 msgr2=0x7f97e8157cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:51.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 shutdown_connections 2023-10-30T13:37:51.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:51.859+0000 7f97ee3b5640 1 -- 172.21.15.148:0/2988476428 wait complete. 2023-10-30T13:37:51.878 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:53.879 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:54.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed77c5640 1 Processor -- start 2023-10-30T13:37:54.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed77c5640 1 -- start start 2023-10-30T13:37:54.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed77c5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2ed00c6bd0 0x7f2ed006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:54.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed77c5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed0066cf0 con 0x7f2ed006f7a0 2023-10-30T13:37:54.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed77c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed00c70d0 con 0x7f2ed0071110 2023-10-30T13:37:54.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed77c5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed00700c0 con 0x7f2ed00c6bd0 2023-10-30T13:37:54.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed4d60640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2ed00c6bd0 0x7f2ed006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:54.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed4d60640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2ed00c6bd0 0x7f2ed006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42086/0 (socket says 172.21.15.148:42086) 2023-10-30T13:37:54.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed4d60640 1 -- 172.21.15.148:0/2421175960 learned_addr learned my addr 172.21.15.148:0/2421175960 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:54.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed4d60640 1 -- 172.21.15.148:0/2421175960 >> v1:172.21.15.150:6789/0 conn(0x7f2ed0071110 legacy=0x7f2ed0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:54.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed4d60640 1 -- 172.21.15.148:0/2421175960 >> v1:172.21.15.148:6789/0 conn(0x7f2ed006f7a0 legacy=0x7f2ed0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:54.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.031+0000 7f2ed4d60640 1 -- 172.21.15.148:0/2421175960 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed00c7390 con 0x7f2ed00c6bd0 2023-10-30T13:37:54.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ed4d60640 1 --2- 172.21.15.148:0/2421175960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed00c6bd0 0x7f2ed006fb80 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009870 tx=0x7f2ec001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5ad810f8d48e45 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:54.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ec7fff640 1 -- 172.21.15.148:0/2421175960 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00a8a30 con 0x7f2ed00c6bd0 2023-10-30T13:37:54.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ec7fff640 1 -- 172.21.15.148:0/2421175960 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec00a8b90 con 0x7f2ed00c6bd0 2023-10-30T13:37:54.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ec7fff640 1 -- 172.21.15.148:0/2421175960 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00b0440 con 0x7f2ed00c6bd0 2023-10-30T13:37:54.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2421175960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed00c6bd0 msgr2=0x7f2ed006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:54.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ed77c5640 1 --2- 172.21.15.148:0/2421175960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed00c6bd0 0x7f2ed006fb80 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009870 tx=0x7f2ec001ed30 comp rx=0 tx=0).stop 2023-10-30T13:37:54.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2421175960 shutdown_connections 2023-10-30T13:37:54.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ed77c5640 1 --2- 172.21.15.148:0/2421175960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed00c6bd0 0x7f2ed006fb80 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:54.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2421175960 >> 172.21.15.148:0/2421175960 conn(0x7f2ed00c7910 msgr2=0x7f2ed00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:54.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2421175960 shutdown_connections 2023-10-30T13:37:54.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.035+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2421175960 wait complete. 2023-10-30T13:37:54.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 Processor -- start 2023-10-30T13:37:54.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 -- start start 2023-10-30T13:37:54.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ed006f7a0 0x7f2ed01ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:54.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed0071110 0x7f2ed0153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:54.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ed00c6bd0 0x7f2ed014fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:54.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed01ef520 con 0x7f2ed0071110 2023-10-30T13:37:54.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed006a650 con 0x7f2ed00c6bd0 2023-10-30T13:37:54.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed00af480 con 0x7f2ed006f7a0 2023-10-30T13:37:54.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed4d60640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed0071110 0x7f2ed0153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:54.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed4d60640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed0071110 0x7f2ed0153710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42102/0 (socket says 172.21.15.148:42102) 2023-10-30T13:37:54.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed4d60640 1 -- 172.21.15.148:0/2405763170 learned_addr learned my addr 172.21.15.148:0/2405763170 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:54.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed5561640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ed006f7a0 0x7f2ed01ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:54.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed5d62640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ed00c6bd0 0x7f2ed014fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:54.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed4d60640 1 -- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ed006f7a0 msgr2=0x7f2ed01ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:54.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed4d60640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ed006f7a0 0x7f2ed01ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:54.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed4d60640 1 -- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ed00c6bd0 msgr2=0x7f2ed014fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:54.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed4d60640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ed00c6bd0 0x7f2ed014fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:54.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed4d60640 1 -- 172.21.15.148:0/2405763170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec00094d0 con 0x7f2ed0071110 2023-10-30T13:37:54.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed4d60640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed0071110 0x7f2ed0153710 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00b3040 tx=0x7f2ec001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:54.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ec5ffb640 1 -- 172.21.15.148:0/2405763170 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00b9620 con 0x7f2ed0071110 2023-10-30T13:37:54.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ec5ffb640 1 -- 172.21.15.148:0/2405763170 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec00b97c0 con 0x7f2ed0071110 2023-10-30T13:37:54.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed00af760 con 0x7f2ed0071110 2023-10-30T13:37:54.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ec5ffb640 1 -- 172.21.15.148:0/2405763170 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00b9b70 con 0x7f2ed0071110 2023-10-30T13:37:54.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.039+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed01ef1b0 con 0x7f2ed0071110 2023-10-30T13:37:54.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.043+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed01ef690 con 0x7f2ed0071110 2023-10-30T13:37:54.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.043+0000 7f2ec5ffb640 1 -- 172.21.15.148:0/2405763170 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ec00b7070 con 0x7f2ed0071110 2023-10-30T13:37:54.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.043+0000 7f2ec5ffb640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2eb4066470 0x7f2eb4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:54.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.043+0000 7f2ec5ffb640 1 -- 172.21.15.148:0/2405763170 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f2ec0122300 con 0x7f2ed0071110 2023-10-30T13:37:54.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.043+0000 7f2ed5561640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2eb4066470 0x7f2eb4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:54.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.043+0000 7f2ed5561640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2eb4066470 0x7f2eb4068930 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc007fd0 tx=0x7f2ecc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:54.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.047+0000 7f2ec5ffb640 1 -- 172.21.15.148:0/2405763170 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ec00ef230 con 0x7f2ed0071110 2023-10-30T13:37:54.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.227+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ed004e170 con 0x7f2ed0071110 2023-10-30T13:37:54.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.227+0000 7f2ec5ffb640 1 -- 172.21.15.148:0/2405763170 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f2ec00f2850 con 0x7f2ed0071110 2023-10-30T13:37:54.228 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:54.229 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:51.844410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:54.229 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 111 2023-10-30T13:37:54.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2eb4066470 msgr2=0x7f2eb4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:54.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2eb4066470 0x7f2eb4068930 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc007fd0 tx=0x7f2ecc009040 comp rx=0 tx=0).stop 2023-10-30T13:37:54.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed0071110 msgr2=0x7f2ed0153710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:54.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed0071110 0x7f2ed0153710 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00b3040 tx=0x7f2ec001bad0 comp rx=0 tx=0).stop 2023-10-30T13:37:54.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 shutdown_connections 2023-10-30T13:37:54.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2eb4066470 0x7f2eb4068930 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:54.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ed0071110 0x7f2ed0153710 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:54.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ed00c6bd0 0x7f2ed014fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:54.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 --2- 172.21.15.148:0/2405763170 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ed006f7a0 0x7f2ed01ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:54.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 >> 172.21.15.148:0/2405763170 conn(0x7f2ed00c7910 msgr2=0x7f2ed0150ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:54.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 shutdown_connections 2023-10-30T13:37:54.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:54.231+0000 7f2ed77c5640 1 -- 172.21.15.148:0/2405763170 wait complete. 2023-10-30T13:37:54.249 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:56.250 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:56.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fca0579d640 1 Processor -- start 2023-10-30T13:37:56.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fca0579d640 1 -- start start 2023-10-30T13:37:56.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fca0579d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00153d70 0x7fca0006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:56.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fca0579d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0006afa0 con 0x7fca000c8490 2023-10-30T13:37:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fca0579d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca00154150 con 0x7fca0006e470 2023-10-30T13:37:56.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fca0579d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0006d780 con 0x7fca00153d70 2023-10-30T13:37:56.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9ff7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fca000c8490 0x7fca000c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51244/0 (socket says 172.21.15.148:51244) 2023-10-30T13:37:56.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9ff7fe640 1 -- 172.21.15.148:0/744031583 learned_addr learned my addr 172.21.15.148:0/744031583 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:56.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9feffd640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00153d70 0x7fca0006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:56.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746771642 0 0) 0x7fca0006afa0 con 0x7fca000c8490 2023-10-30T13:37:56.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e8003530 con 0x7fca000c8490 2023-10-30T13:37:56.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17408156 0 0) 0x7fca00154150 con 0x7fca0006e470 2023-10-30T13:37:56.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0006afa0 con 0x7fca0006e470 2023-10-30T13:37:56.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 660196532 0 0) 0x7fc9e8003530 con 0x7fca000c8490 2023-10-30T13:37:56.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00153d70 msgr2=0x7fca0006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:56.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00153d70 0x7fca0006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 >> v1:172.21.15.150:6789/0 conn(0x7fca0006e470 legacy=0x7fca000cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:56.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0006e030 con 0x7fca000c8490 2023-10-30T13:37:56.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fc9f8001a00 con 0x7fca000c8490 2023-10-30T13:37:56.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 >> v1:172.21.15.148:6789/0 conn(0x7fca000c8490 legacy=0x7fca000c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:56.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e8003b40 0x7fc9e8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:56.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e80047d0 0x7fc9e8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:56.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00153d70 0x7fc9e8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:56.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0006d780 con 0x7fc9e80047d0 2023-10-30T13:37:56.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e800ba40 con 0x7fc9e8003b40 2023-10-30T13:37:56.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e800bbd0 con 0x7fca00153d70 2023-10-30T13:37:56.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fe7fc640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e80047d0 0x7fc9e8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:56.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9ff7fe640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00153d70 0x7fc9e8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:56.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9feffd640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e8003b40 0x7fc9e8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:56.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fe7fc640 1 -- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00153d70 msgr2=0x7fc9e8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:56.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fe7fc640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00153d70 0x7fc9e8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fe7fc640 1 -- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e8003b40 msgr2=0x7fc9e8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:56.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fe7fc640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e8003b40 0x7fc9e8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9fe7fc640 1 -- 172.21.15.148:0/744031583 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e8003530 con 0x7fc9e80047d0 2023-10-30T13:37:56.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.399+0000 7fc9feffd640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e8003b40 0x7fc9e8003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:56.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fc9fe7fc640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e80047d0 0x7fc9e8004c50 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40018d0 tx=0x7fc9f401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8407b2a21add882 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:56.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40052d0 con 0x7fc9e80047d0 2023-10-30T13:37:56.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f4005450 con 0x7fc9e80047d0 2023-10-30T13:37:56.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fc9fdffb640 1 -- 172.21.15.148:0/744031583 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40a88d0 con 0x7fc9e80047d0 2023-10-30T13:37:56.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fca0579d640 1 -- 172.21.15.148:0/744031583 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e80047d0 msgr2=0x7fc9e8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:56.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fca0579d640 1 --2- 172.21.15.148:0/744031583 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9e80047d0 0x7fc9e8004c50 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40018d0 tx=0x7fc9f401ec70 comp rx=0 tx=0).stop 2023-10-30T13:37:56.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fc9ff7fe640 1 -- 172.21.15.148:0/744031583 reap_dead start 2023-10-30T13:37:56.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fca0579d640 1 -- 172.21.15.148:0/744031583 shutdown_connections 2023-10-30T13:37:56.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fca0579d640 1 -- 172.21.15.148:0/744031583 >> 172.21.15.148:0/744031583 conn(0x7fca00071ea0 msgr2=0x7fca00065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:56.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fca0579d640 1 -- 172.21.15.148:0/744031583 shutdown_connections 2023-10-30T13:37:56.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.403+0000 7fca0579d640 1 -- 172.21.15.148:0/744031583 wait complete. 2023-10-30T13:37:56.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 Processor -- start 2023-10-30T13:37:56.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 -- start start 2023-10-30T13:37:56.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e80047d0 0x7fca0014c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:56.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00150570 0x7fca0014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:56.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00155330 0x7fca000cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:56.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca001f2eb0 con 0x7fca00155330 2023-10-30T13:37:56.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca001f25a0 con 0x7fc9e80047d0 2023-10-30T13:37:56.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0014cc30 con 0x7fca00150570 2023-10-30T13:37:56.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9ff7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00155330 0x7fca000cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:56.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9ff7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00155330 0x7fca000cbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42112/0 (socket says 172.21.15.148:42112) 2023-10-30T13:37:56.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9ff7fe640 1 -- 172.21.15.148:0/2615568516 learned_addr learned my addr 172.21.15.148:0/2615568516 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:56.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9fe7fc640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00150570 0x7fca0014a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:56.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9feffd640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e80047d0 0x7fca0014c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:56.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9ff7fe640 1 -- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00150570 msgr2=0x7fca0014a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:56.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9ff7fe640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00150570 0x7fca0014a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9ff7fe640 1 -- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e80047d0 msgr2=0x7fca0014c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:56.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9ff7fe640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e80047d0 0x7fca0014c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9ff7fe640 1 -- 172.21.15.148:0/2615568516 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca00150b20 con 0x7fca00155330 2023-10-30T13:37:56.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9ff7fe640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00155330 0x7fca000cbcc0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fca000cdcd0 tx=0x7fc9f8007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:56.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9e77fe640 1 -- 172.21.15.148:0/2615568516 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f800abf0 con 0x7fca00155330 2023-10-30T13:37:56.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9e77fe640 1 -- 172.21.15.148:0/2615568516 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f800ad70 con 0x7fca00155330 2023-10-30T13:37:56.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca00150de0 con 0x7fca00155330 2023-10-30T13:37:56.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fc9e77fe640 1 -- 172.21.15.148:0/2615568516 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f8013630 con 0x7fca00155330 2023-10-30T13:37:56.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.407+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca001f2ad0 con 0x7fca00155330 2023-10-30T13:37:56.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.411+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca001f2ff0 con 0x7fca00155330 2023-10-30T13:37:56.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.411+0000 7fc9e77fe640 1 -- 172.21.15.148:0/2615568516 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9f8013790 con 0x7fca00155330 2023-10-30T13:37:56.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.411+0000 7fc9e77fe640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9ec0664a0 0x7fc9ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:56.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.411+0000 7fc9e77fe640 1 -- 172.21.15.148:0/2615568516 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fc9f8084eb0 con 0x7fca00155330 2023-10-30T13:37:56.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.411+0000 7fc9feffd640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9ec0664a0 0x7fc9ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:56.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.411+0000 7fc9feffd640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9ec0664a0 0x7fc9ec068960 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009950 tx=0x7fc9f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:56.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.415+0000 7fc9e77fe640 1 -- 172.21.15.148:0/2615568516 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fca001f2ff0 con 0x7fca00155330 2023-10-30T13:37:56.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.599+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca001f2ff0 con 0x7fca00155330 2023-10-30T13:37:56.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.599+0000 7fc9e77fe640 1 -- 172.21.15.148:0/2615568516 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5720 (secure 0 0 0) 0x7fca001f2ff0 con 0x7fca00155330 2023-10-30T13:37:56.600 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:56.600 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:51.844410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:56.600 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 111 2023-10-30T13:37:56.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9ec0664a0 msgr2=0x7fc9ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:56.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9ec0664a0 0x7fc9ec068960 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009950 tx=0x7fc9f0008040 comp rx=0 tx=0).stop 2023-10-30T13:37:56.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00155330 msgr2=0x7fca000cbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:56.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00155330 0x7fca000cbcc0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fca000cdcd0 tx=0x7fc9f8007bd0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 shutdown_connections 2023-10-30T13:37:56.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9ec0664a0 0x7fc9ec068960 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00155330 0x7fca000cbcc0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca00150570 0x7fca0014a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 --2- 172.21.15.148:0/2615568516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e80047d0 0x7fca0014c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:56.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 >> 172.21.15.148:0/2615568516 conn(0x7fca00071ea0 msgr2=0x7fca001515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:56.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 shutdown_connections 2023-10-30T13:37:56.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:56.603+0000 7fca0579d640 1 -- 172.21.15.148:0/2615568516 wait complete. 2023-10-30T13:37:56.620 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:37:58.621 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:37:58.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca18a9c640 1 Processor -- start 2023-10-30T13:37:58.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca18a9c640 1 -- start start 2023-10-30T13:37:58.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca18a9c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca1415a410 0x7fca1406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:58.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca18a9c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fca1406cb80 0x7fca1406cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:58.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca18a9c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca1406d530 con 0x7fca14069e90 2023-10-30T13:37:58.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca18a9c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca140c8570 con 0x7fca1406cb80 2023-10-30T13:37:58.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca18a9c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1415a850 con 0x7fca1415a410 2023-10-30T13:37:58.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca1259c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fca14069e90 0x7fca140c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51252/0 (socket says 172.21.15.148:51252) 2023-10-30T13:37:58.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca1259c640 1 -- 172.21.15.148:0/670081415 learned_addr learned my addr 172.21.15.148:0/670081415 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:37:58.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca12d9d640 1 --2- 172.21.15.148:0/670081415 >> v2:172.21.15.148:3300/0 conn(0x7fca1406cb80 0x7fca1406cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:58.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca11d9b640 1 --2- 172.21.15.148:0/670081415 >> v2:172.21.15.150:3300/0 conn(0x7fca1415a410 0x7fca1406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:58.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca12d9d640 1 -- 172.21.15.148:0/670081415 >> v2:172.21.15.150:3300/0 conn(0x7fca1415a410 msgr2=0x7fca1406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:37:58.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca12d9d640 1 --2- 172.21.15.148:0/670081415 >> v2:172.21.15.150:3300/0 conn(0x7fca1415a410 0x7fca1406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca12d9d640 1 -- 172.21.15.148:0/670081415 >> v1:172.21.15.148:6789/0 conn(0x7fca14069e90 legacy=0x7fca140c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:58.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca12d9d640 1 -- 172.21.15.148:0/670081415 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140c6c10 con 0x7fca1406cb80 2023-10-30T13:37:58.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.779+0000 7fca11d9b640 1 --2- 172.21.15.148:0/670081415 >> v2:172.21.15.150:3300/0 conn(0x7fca1415a410 0x7fca1406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:58.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca12d9d640 1 --2- 172.21.15.148:0/670081415 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 0x7fca1406cff0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fca1406feb0 tx=0x7fca0c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f1e7777f617778 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:58.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca1159a640 1 -- 172.21.15.148:0/670081415 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0a57d0 con 0x7fca1406cb80 2023-10-30T13:37:58.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca1159a640 1 -- 172.21.15.148:0/670081415 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c0a5930 con 0x7fca1406cb80 2023-10-30T13:37:58.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca1159a640 1 -- 172.21.15.148:0/670081415 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0a5cc0 con 0x7fca1406cb80 2023-10-30T13:37:58.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca18a9c640 1 -- 172.21.15.148:0/670081415 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 msgr2=0x7fca1406cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:58.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca18a9c640 1 --2- 172.21.15.148:0/670081415 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 0x7fca1406cff0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fca1406feb0 tx=0x7fca0c01c910 comp rx=0 tx=0).stop 2023-10-30T13:37:58.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca18a9c640 1 -- 172.21.15.148:0/670081415 shutdown_connections 2023-10-30T13:37:58.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca18a9c640 1 --2- 172.21.15.148:0/670081415 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 0x7fca1406cff0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca18a9c640 1 --2- 172.21.15.148:0/670081415 >> v2:172.21.15.150:3300/0 conn(0x7fca1415a410 0x7fca1406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca18a9c640 1 -- 172.21.15.148:0/670081415 >> 172.21.15.148:0/670081415 conn(0x7fca1406dfd0 msgr2=0x7fca1406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:58.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca18a9c640 1 -- 172.21.15.148:0/670081415 shutdown_connections 2023-10-30T13:37:58.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.783+0000 7fca18a9c640 1 -- 172.21.15.148:0/670081415 wait complete. 2023-10-30T13:37:58.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 Processor -- start 2023-10-30T13:37:58.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 -- start start 2023-10-30T13:37:58.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca14069e90 0x7fca1414b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:58.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 0x7fca141f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:58.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1415a410 0x7fca141f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:58.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1414b540 con 0x7fca1406cb80 2023-10-30T13:37:58.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca140ca1e0 con 0x7fca1415a410 2023-10-30T13:37:58.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca141569b0 con 0x7fca14069e90 2023-10-30T13:37:58.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca11d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 0x7fca141f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:58.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca11d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 0x7fca141f52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42140/0 (socket says 172.21.15.148:42140) 2023-10-30T13:37:58.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca11d9b640 1 -- 172.21.15.148:0/245207985 learned_addr learned my addr 172.21.15.148:0/245207985 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:37:58.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca1259c640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca14069e90 0x7fca1414b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:58.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca12d9d640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1415a410 0x7fca141f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:58.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca11d9b640 1 -- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca14069e90 msgr2=0x7fca1414b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:58.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca11d9b640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca14069e90 0x7fca1414b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca11d9b640 1 -- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1415a410 msgr2=0x7fca141f1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:58.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca11d9b640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1415a410 0x7fca141f1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca11d9b640 1 -- 172.21.15.148:0/245207985 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c009a70 con 0x7fca1406cb80 2023-10-30T13:37:58.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca12d9d640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1415a410 0x7fca141f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:37:58.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca11d9b640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 0x7fca141f52d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fca0800d830 tx=0x7fca0800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:58.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca02ffd640 1 -- 172.21.15.148:0/245207985 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca080096c0 con 0x7fca1406cb80 2023-10-30T13:37:58.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca02ffd640 1 -- 172.21.15.148:0/245207985 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca08009820 con 0x7fca1406cb80 2023-10-30T13:37:58.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca02ffd640 1 -- 172.21.15.148:0/245207985 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08009bd0 con 0x7fca1406cb80 2023-10-30T13:37:58.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca141f5ba0 con 0x7fca1406cb80 2023-10-30T13:37:58.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.787+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca14150c40 con 0x7fca1406cb80 2023-10-30T13:37:58.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.791+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1404e170 con 0x7fca1406cb80 2023-10-30T13:37:58.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.791+0000 7fca02ffd640 1 -- 172.21.15.148:0/245207985 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fca08014410 con 0x7fca1406cb80 2023-10-30T13:37:58.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.791+0000 7fca02ffd640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9f40664d0 0x7fc9f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:37:58.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.791+0000 7fca02ffd640 1 -- 172.21.15.148:0/245207985 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fca08084d50 con 0x7fca1406cb80 2023-10-30T13:37:58.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.791+0000 7fca1259c640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9f40664d0 0x7fc9f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:37:58.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.791+0000 7fca1259c640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9f40664d0 0x7fc9f4068990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fca04005890 tx=0x7fca04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:37:58.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.795+0000 7fca02ffd640 1 -- 172.21.15.148:0/245207985 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fca08051b50 con 0x7fca1406cb80 2023-10-30T13:37:58.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.987+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca1414f5d0 con 0x7fca1406cb80 2023-10-30T13:37:58.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.987+0000 7fca02ffd640 1 -- 172.21.15.148:0/245207985 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5720 (secure 0 0 0) 0x7fca08055170 con 0x7fca1406cb80 2023-10-30T13:37:58.987 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:37:58.988 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:37:58.989 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 112 2023-10-30T13:37:58.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9f40664d0 msgr2=0x7fc9f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:58.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9f40664d0 0x7fc9f4068990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fca04005890 tx=0x7fca04009040 comp rx=0 tx=0).stop 2023-10-30T13:37:58.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 msgr2=0x7fca141f52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:37:58.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 0x7fca141f52d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fca0800d830 tx=0x7fca0800dcf0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 shutdown_connections 2023-10-30T13:37:58.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9f40664d0 0x7fc9f4068990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca1406cb80 0x7fca141f52d0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca1415a410 0x7fca141f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 --2- 172.21.15.148:0/245207985 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca14069e90 0x7fca1414b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:37:58.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 >> 172.21.15.148:0/245207985 conn(0x7fca1406dfd0 msgr2=0x7fca140ca450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:37:58.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 shutdown_connections 2023-10-30T13:37:58.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:37:58.991+0000 7fca18a9c640 1 -- 172.21.15.148:0/245207985 wait complete. 2023-10-30T13:37:59.008 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6870 2023-10-30T13:37:59.008 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.c 2023-10-30T13:38:00.686 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:00.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:04.009 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:38:04.010 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:38:04.010 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:38:04.015 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:38:04.016 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:04.101 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:38:04.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.159+0000 7f4651765640 1 Processor -- start 2023-10-30T13:38:04.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.159+0000 7f4651765640 1 -- start start 2023-10-30T13:38:04.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f4651765640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f464c1538a0 0x7f464c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:04.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f4651765640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c0cf160 0x7f464c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:04.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f4651765640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c0cb800 con 0x7f464c0ca0e0 2023-10-30T13:38:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f4651765640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c15dc30 con 0x7f464c1538a0 2023-10-30T13:38:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f4651765640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c0cb330 con 0x7f464c0cf160 2023-10-30T13:38:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f464a7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f464c0ca0e0 0x7f464c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36128/0 (socket says 172.21.15.148:36128) 2023-10-30T13:38:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f464a7fc640 1 -- 172.21.15.148:0/373916580 learned_addr learned my addr 172.21.15.148:0/373916580 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:38:04.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f464affd640 1 --2- 172.21.15.148:0/373916580 >> v2:172.21.15.150:3300/0 conn(0x7f464c1538a0 0x7f464c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f464b7fe640 1 --2- 172.21.15.148:0/373916580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c0cf160 0x7f464c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f4649ffb640 1 -- 172.21.15.148:0/373916580 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511843744 0 0) 0x7f464c0cb800 con 0x7f464c0ca0e0 2023-10-30T13:38:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f4649ffb640 1 -- 172.21.15.148:0/373916580 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4634003530 con 0x7f464c0ca0e0 2023-10-30T13:38:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f464affd640 1 -- 172.21.15.148:0/373916580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c0cf160 msgr2=0x7f464c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f464affd640 1 --2- 172.21.15.148:0/373916580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c0cf160 0x7f464c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:04.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f464affd640 1 -- 172.21.15.148:0/373916580 >> v1:172.21.15.148:6789/0 conn(0x7f464c0ca0e0 legacy=0x7f464c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:04.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f4649ffb640 1 -- 172.21.15.148:0/373916580 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3264429508 0 0) 0x7f4634003530 con 0x7f464c0ca0e0 2023-10-30T13:38:04.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.163+0000 7f464affd640 1 -- 172.21.15.148:0/373916580 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c0cad70 con 0x7f464c1538a0 2023-10-30T13:38:04.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f464affd640 1 --2- 172.21.15.148:0/373916580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c1538a0 0x7f464c15d6f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f463c0097c0 tx=0x7f463c01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=426d0637fdb3beec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4649ffb640 1 -- 172.21.15.148:0/373916580 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640001680 con 0x7f464c1538a0 2023-10-30T13:38:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4649ffb640 1 -- 172.21.15.148:0/373916580 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c0a85d0 con 0x7f464c1538a0 2023-10-30T13:38:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4649ffb640 1 -- 172.21.15.148:0/373916580 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0a8960 con 0x7f464c1538a0 2023-10-30T13:38:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4651765640 1 -- 172.21.15.148:0/373916580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c1538a0 msgr2=0x7f464c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:04.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4651765640 1 --2- 172.21.15.148:0/373916580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c1538a0 0x7f464c15d6f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f463c0097c0 tx=0x7f463c01eb90 comp rx=0 tx=0).stop 2023-10-30T13:38:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4651765640 1 -- 172.21.15.148:0/373916580 shutdown_connections 2023-10-30T13:38:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4651765640 1 --2- 172.21.15.148:0/373916580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c0cf160 0x7f464c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4651765640 1 --2- 172.21.15.148:0/373916580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c1538a0 0x7f464c15d6f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4651765640 1 -- 172.21.15.148:0/373916580 >> 172.21.15.148:0/373916580 conn(0x7f464c06bfe0 msgr2=0x7f464c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:04.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4651765640 1 -- 172.21.15.148:0/373916580 shutdown_connections 2023-10-30T13:38:04.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.167+0000 7f4651765640 1 -- 172.21.15.148:0/373916580 wait complete. 2023-10-30T13:38:04.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 Processor -- start 2023-10-30T13:38:04.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 -- start start 2023-10-30T13:38:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0ca0e0 0x7f464c1f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f464c0cf160 0x7f464c158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c1538a0 0x7f464c15d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c15b050 con 0x7f464c0cf160 2023-10-30T13:38:04.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c1f0680 con 0x7f464c0ca0e0 2023-10-30T13:38:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c1ef810 con 0x7f464c1538a0 2023-10-30T13:38:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f464c0cf160 0x7f464c158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f464c0cf160 0x7f464c158f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42474/0 (socket says 172.21.15.148:42474) 2023-10-30T13:38:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464a7fc640 1 -- 172.21.15.148:0/797431512 learned_addr learned my addr 172.21.15.148:0/797431512 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:38:04.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464b7fe640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c1538a0 0x7f464c15d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464affd640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0ca0e0 0x7f464c1f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464a7fc640 1 -- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c1538a0 msgr2=0x7f464c15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464a7fc640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c1538a0 0x7f464c15d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464a7fc640 1 -- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0ca0e0 msgr2=0x7f464c1f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464a7fc640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0ca0e0 0x7f464c1f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:04.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464a7fc640 1 -- 172.21.15.148:0/797431512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c009420 con 0x7f464c0cf160 2023-10-30T13:38:04.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f464a7fc640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f464c0cf160 0x7f464c158f30 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f464000c790 tx=0x7f464000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:04.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f46337fe640 1 -- 172.21.15.148:0/797431512 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640005ae0 con 0x7f464c0cf160 2023-10-30T13:38:04.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f46337fe640 1 -- 172.21.15.148:0/797431512 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4640005c40 con 0x7f464c0cf160 2023-10-30T13:38:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464c1f3cd0 con 0x7f464c0cf160 2023-10-30T13:38:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f46337fe640 1 -- 172.21.15.148:0/797431512 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640007070 con 0x7f464c0cf160 2023-10-30T13:38:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.171+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c1efa00 con 0x7f464c0cf160 2023-10-30T13:38:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.175+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c0b2a50 con 0x7f464c0cf160 2023-10-30T13:38:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.175+0000 7f46337fe640 1 -- 172.21.15.148:0/797431512 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46400071d0 con 0x7f464c0cf160 2023-10-30T13:38:04.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.175+0000 7f46337fe640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46380664d0 0x7f4638068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:04.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.175+0000 7f46337fe640 1 -- 172.21.15.148:0/797431512 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f4640083ee0 con 0x7f464c0cf160 2023-10-30T13:38:04.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.175+0000 7f464affd640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46380664d0 0x7f4638068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:04.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.179+0000 7f464affd640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46380664d0 0x7f4638068990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b2000 tx=0x7f463c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:04.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.183+0000 7f46337fe640 1 -- 172.21.15.148:0/797431512 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4640050eb0 con 0x7f464c0cf160 2023-10-30T13:38:04.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.399+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f464c14d770 con 0x7f464c0cf160 2023-10-30T13:38:04.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f46337fe640 1 -- 172.21.15.148:0/797431512 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f464c14d770 con 0x7f464c0cf160 2023-10-30T13:38:04.401 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:38:04.401 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:04.401 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 112 2023-10-30T13:38:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46380664d0 msgr2=0x7f4638068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46380664d0 0x7f4638068990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b2000 tx=0x7f463c008040 comp rx=0 tx=0).stop 2023-10-30T13:38:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f464c0cf160 msgr2=0x7f464c158f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:04.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f464c0cf160 0x7f464c158f30 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f464000c790 tx=0x7f464000cc50 comp rx=0 tx=0).stop 2023-10-30T13:38:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 shutdown_connections 2023-10-30T13:38:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46380664d0 0x7f4638068990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f464c0cf160 0x7f464c158f30 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f464c1538a0 0x7f464c15d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:04.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 --2- 172.21.15.148:0/797431512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f464c0ca0e0 0x7f464c1f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 >> 172.21.15.148:0/797431512 conn(0x7f464c06bfe0 msgr2=0x7f464c152810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 shutdown_connections 2023-10-30T13:38:04.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:04.403+0000 7f4651765640 1 -- 172.21.15.148:0/797431512 wait complete. 2023-10-30T13:38:04.422 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:38:05.497 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 16% (48/285) Updating files: 17% (49/285) Updating files: 18% (52/285) Updating files: 19% (55/285) Updating files: 20% (57/285) Updating files: 21% (60/285) Updating files: 22% (63/285) Updating files: 23% (66/285) Updating files: 24% (69/285) Updating files: 25% (72/285) Updating files: 26% (75/285) Updating files: 27% (77/285) Updating files: 28% (80/285) Updating files: 29% (83/285) Updating files: 30% (86/285) Updating files: 31% (89/285) Updating files: 32% (92/285) Updating files: 33% (95/285) Updating files: 34% (97/285) Updating files: 35% (100/285) Updating files: 36% (103/285) Updating files: 37% (106/285) Updating files: 38% (109/285) Updating files: 39% (112/285) Updating files: 40% (114/285) Updating files: 41% (117/285) Updating files: 42% (120/285) Updating files: 43% (123/285) Updating files: 44% (126/285) Updating files: 45% (129/285) Updating files: 46% (132/285) Updating files: 47% (134/285) Updating files: 48% (137/285) Updating files: 49% (140/285) Updating files: 50% (143/285) Updating files: 51% (146/285) Updating files: 52% (149/285) Updating files: 53% (152/285) Updating files: 54% (154/285) Updating files: 55% (157/285) Updating files: 56% (160/285) Updating files: 57% (163/285) Updating files: 58% (166/285) Updating files: 59% (169/285) Updating files: 60% (171/285) Updating files: 61% (174/285) Updating files: 62% (177/285) Updating files: 63% (180/285) Updating files: 64% (183/285) Updating files: 65% (186/285) Updating files: 66% (189/285) Updating files: 67% (191/285) Updating files: 67% (193/285) Updating files: 68% (194/285) Updating files: 69% (197/285) Updating files: 70% (200/285) Updating files: 71% (203/285) Updating files: 72% (206/285) Updating files: 73% (209/285) Updating files: 74% (211/285) Updating files: 75% (214/285) Updating files: 76% (217/285) Updating files: 77% (220/285) Updating files: 78% (223/285) Updating files: 79% (226/285) Updating files: 80% (228/285) Updating files: 81% (231/285) Updating files: 82% (234/285) Updating files: 83% (237/285) Updating files: 84% (240/285) Updating files: 85% (243/285) Updating files: 85% (244/285) Updating files: 86% (246/285) Updating files: 87% (248/285) Updating files: 88% (251/285) Updating files: 89% (254/285) Updating files: 90% (257/285) Updating files: 91% (260/285) Updating files: 92% (263/285) Updating files: 93% (266/285) Updating files: 94% (268/285) Updating files: 95% (271/285) Updating files: 96% (274/285) Updating files: 97% (277/285) Updating files: 98% (280/285) Updating files: 99% (283/285) Updating files: 100% (285/285) Updating files: 100% (285/285), done. 2023-10-30T13:38:05.769 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 64b1b2c70f0 v0.30 2023-10-30T13:38:05.774 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.30 2023-10-30T13:38:05.794 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 31 -eq 48 ] 2023-10-30T13:38:05.794 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.31 2023-10-30T13:38:05.794 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.31 2023-10-30T13:38:05.794 INFO:tasks.workunit.client.0.smithi148.stdout:v0.31 2023-10-30T13:38:05.795 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.31 2023-10-30T13:38:05.820 INFO:tasks.ceph.mds.a.smithi148.stderr:2023-10-30T13:38:05.819+0000 7f7944541640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:38:06.423 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:38:06.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.607+0000 7fdbfdb3b640 1 Processor -- start 2023-10-30T13:38:06.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfdb3b640 1 -- start start 2023-10-30T13:38:06.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfdb3b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdbf81538f0 0x7fdbf8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:06.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfdb3b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdbf815cd50 0x7fdbf80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:06.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfdb3b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 0x7fdbf80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:06.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfdb3b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf815d300 con 0x7fdbf81538f0 2023-10-30T13:38:06.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfdb3b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf815d5e0 con 0x7fdbf815cd50 2023-10-30T13:38:06.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfdb3b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf815d8c0 con 0x7fdbf80cefa0 2023-10-30T13:38:06.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfcb39640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdbf81538f0 0x7fdbf8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:06.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfcb39640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdbf81538f0 0x7fdbf8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42476/0 (socket says 172.21.15.148:42476) 2023-10-30T13:38:06.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfcb39640 1 -- 172.21.15.148:0/498940739 learned_addr learned my addr 172.21.15.148:0/498940739 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:38:06.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfd33a640 1 --2- 172.21.15.148:0/498940739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 0x7fdbf80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:06.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbf7fff640 1 --2- 172.21.15.148:0/498940739 >> v2:172.21.15.150:3300/0 conn(0x7fdbf815cd50 0x7fdbf80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:06.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfcb39640 1 -- 172.21.15.148:0/498940739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 msgr2=0x7fdbf80cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:06.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfcb39640 1 --2- 172.21.15.148:0/498940739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 0x7fdbf80cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfcb39640 1 -- 172.21.15.148:0/498940739 >> v2:172.21.15.150:3300/0 conn(0x7fdbf815cd50 msgr2=0x7fdbf80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:38:06.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfcb39640 1 --2- 172.21.15.148:0/498940739 >> v2:172.21.15.150:3300/0 conn(0x7fdbf815cd50 0x7fdbf80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.611+0000 7fdbfcb39640 1 -- 172.21.15.148:0/498940739 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf815dba0 con 0x7fdbf81538f0 2023-10-30T13:38:06.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfcb39640 1 --2- 172.21.15.148:0/498940739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 0x7fdbf8159fb0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8072e90 tx=0x7fdbec01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82aadbbf3412ca0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:06.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbf77fe640 1 -- 172.21.15.148:0/498940739 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0a89b0 con 0x7fdbf81538f0 2023-10-30T13:38:06.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbf77fe640 1 -- 172.21.15.148:0/498940739 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec0a8b10 con 0x7fdbf81538f0 2023-10-30T13:38:06.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbf77fe640 1 -- 172.21.15.148:0/498940739 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0a8ea0 con 0x7fdbf81538f0 2023-10-30T13:38:06.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/498940739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 msgr2=0x7fdbf8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:06.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/498940739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 0x7fdbf8159fb0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8072e90 tx=0x7fdbec01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/498940739 shutdown_connections 2023-10-30T13:38:06.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/498940739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 0x7fdbf80cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/498940739 >> v2:172.21.15.150:3300/0 conn(0x7fdbf815cd50 0x7fdbf80c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/498940739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 0x7fdbf8159fb0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/498940739 >> 172.21.15.148:0/498940739 conn(0x7fdbf806a640 msgr2=0x7fdbf8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:06.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/498940739 shutdown_connections 2023-10-30T13:38:06.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.615+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/498940739 wait complete. 2023-10-30T13:38:06.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 Processor -- start 2023-10-30T13:38:06.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 -- start start 2023-10-30T13:38:06.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 0x7fdbf81f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:06.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 0x7fdbf80c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:06.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf815cd50 0x7fdbf8147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:06.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8152a70 con 0x7fdbf81538f0 2023-10-30T13:38:06.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf81522c0 con 0x7fdbf815cd50 2023-10-30T13:38:06.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf815c230 con 0x7fdbf80cefa0 2023-10-30T13:38:06.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 0x7fdbf80c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:06.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 0x7fdbf80c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42480/0 (socket says 172.21.15.148:42480) 2023-10-30T13:38:06.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf7fff640 1 -- 172.21.15.148:0/102810966 learned_addr learned my addr 172.21.15.148:0/102810966 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:38:06.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfcb39640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 0x7fdbf81f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:06.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfd33a640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf815cd50 0x7fdbf8147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:06.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf7fff640 1 -- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 msgr2=0x7fdbf81f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:06.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf7fff640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 0x7fdbf81f2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf7fff640 1 -- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf815cd50 msgr2=0x7fdbf8147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:06.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf7fff640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf815cd50 0x7fdbf8147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf7fff640 1 -- 172.21.15.148:0/102810966 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec0094d0 con 0x7fdbf81538f0 2023-10-30T13:38:06.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfcb39640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 0x7fdbf81f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:38:06.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf7fff640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 0x7fdbf80c8e10 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400e870 tx=0x7fdbe400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:06.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf57fa640 1 -- 172.21.15.148:0/102810966 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4009510 con 0x7fdbf81538f0 2023-10-30T13:38:06.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf57fa640 1 -- 172.21.15.148:0/102810966 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe400eec0 con 0x7fdbf81538f0 2023-10-30T13:38:06.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf8159220 con 0x7fdbf81538f0 2023-10-30T13:38:06.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbf57fa640 1 -- 172.21.15.148:0/102810966 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe400cb40 con 0x7fdbf81538f0 2023-10-30T13:38:06.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf8152790 con 0x7fdbf81538f0 2023-10-30T13:38:06.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.619+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf804e170 con 0x7fdbf81538f0 2023-10-30T13:38:06.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.623+0000 7fdbf57fa640 1 -- 172.21.15.148:0/102810966 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdbe400ccc0 con 0x7fdbf81538f0 2023-10-30T13:38:06.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.623+0000 7fdbf57fa640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdbdc0664a0 0x7fdbdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:38:06.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.623+0000 7fdbf57fa640 1 -- 172.21.15.148:0/102810966 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fdbe4084420 con 0x7fdbf81538f0 2023-10-30T13:38:06.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.623+0000 7fdbfcb39640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdbdc0664a0 0x7fdbdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:38:06.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.623+0000 7fdbfcb39640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdbdc0664a0 0x7fdbdc068960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0b2000 tx=0x7fdbec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:38:06.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.627+0000 7fdbf57fa640 1 -- 172.21.15.148:0/102810966 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdbe40513e0 con 0x7fdbf81538f0 2023-10-30T13:38:06.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbf80b2290 con 0x7fdbf81538f0 2023-10-30T13:38:06.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbf57fa640 1 -- 172.21.15.148:0/102810966 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6512 (secure 0 0 0) 0x7fdbe4054a00 con 0x7fdbf81538f0 2023-10-30T13:38:06.810 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:38:06.810 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7149,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:38:06.810 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 113 2023-10-30T13:38:06.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdbdc0664a0 msgr2=0x7fdbdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdbdc0664a0 0x7fdbdc068960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0b2000 tx=0x7fdbec008040 comp rx=0 tx=0).stop 2023-10-30T13:38:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 msgr2=0x7fdbf80c8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:38:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 0x7fdbf80c8e10 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400e870 tx=0x7fdbe400ec70 comp rx=0 tx=0).stop 2023-10-30T13:38:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 shutdown_connections 2023-10-30T13:38:06.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdbdc0664a0 0x7fdbdc068960 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf81538f0 0x7fdbf80c8e10 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf815cd50 0x7fdbf8147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 --2- 172.21.15.148:0/102810966 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbf80cefa0 0x7fdbf81f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:38:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 >> 172.21.15.148:0/102810966 conn(0x7fdbf806a640 msgr2=0x7fdbf8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:38:06.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 shutdown_connections 2023-10-30T13:38:06.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:38:06.811+0000 7fdbfdb3b640 1 -- 172.21.15.148:0/102810966 wait complete. 2023-10-30T13:38:06.830 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:38:06.831 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 94 secs before thrashing 2023-10-30T13:38:14.094 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 19% (76/398) Updating files: 20% (80/398) Updating files: 21% (84/398) Updating files: 22% (88/398) Updating files: 23% (92/398) Updating files: 24% (96/398) Updating files: 25% (100/398) Updating files: 26% (104/398) Updating files: 27% (108/398) Updating files: 28% (112/398) Updating files: 29% (116/398) Updating files: 30% (120/398) Updating files: 31% (124/398) Updating files: 32% (128/398) Updating files: 32% (130/398) Updating files: 33% (132/398) Updating files: 34% (136/398) Updating files: 35% (140/398) Updating files: 36% (144/398) Updating files: 37% (148/398) Updating files: 38% (152/398) Updating files: 39% (156/398) Updating files: 40% (160/398) Updating files: 41% (164/398) Updating files: 42% (168/398) Updating files: 43% (172/398) Updating files: 44% (176/398) Updating files: 45% (180/398) Updating files: 46% (184/398) Updating files: 47% (188/398) Updating files: 48% (192/398) Updating files: 49% (196/398) Updating files: 50% (199/398) Updating files: 51% (203/398) Updating files: 51% (206/398) Updating files: 52% (207/398) Updating files: 53% (211/398) Updating files: 54% (215/398) Updating files: 54% (218/398) Updating files: 55% (219/398) Updating files: 56% (223/398) Updating files: 57% (227/398) Updating files: 58% (231/398) Updating files: 59% (235/398) Updating files: 60% (239/398) Updating files: 61% (243/398) Updating files: 62% (247/398) Updating files: 63% (251/398) Updating files: 64% (255/398) Updating files: 65% (259/398) Updating files: 66% (263/398) Updating files: 66% (264/398) Updating files: 67% (267/398) Updating files: 68% (271/398) Updating files: 69% (275/398) Updating files: 70% (279/398) Updating files: 71% (283/398) Updating files: 72% (287/398) Updating files: 73% (291/398) Updating files: 74% (295/398) Updating files: 75% (299/398) Updating files: 76% (303/398) Updating files: 77% (307/398) Updating files: 78% (311/398) Updating files: 79% (315/398) Updating files: 80% (319/398) Updating files: 81% (323/398) Updating files: 82% (327/398) Updating files: 83% (331/398) Updating files: 84% (335/398) Updating files: 85% (339/398) Updating files: 86% (343/398) Updating files: 87% (347/398) Updating files: 88% (351/398) Updating files: 89% (355/398) Updating files: 90% (359/398) Updating files: 91% (363/398) Updating files: 92% (367/398) Updating files: 93% (371/398) Updating files: 93% (374/398) Updating files: 94% (375/398) Updating files: 95% (379/398) Updating files: 96% (383/398) Updating files: 97% (387/398) Updating files: 98% (391/398) Updating files: 99% (395/398) Updating files: 100% (398/398) Updating files: 100% (398/398), done. 2023-10-30T13:38:14.112 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 9019c6ce640 v0.31 2023-10-30T13:38:14.116 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.31 2023-10-30T13:38:14.183 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 32 -eq 48 ] 2023-10-30T13:38:14.183 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.32 2023-10-30T13:38:14.183 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.32 2023-10-30T13:38:14.184 INFO:tasks.workunit.client.0.smithi148.stdout:v0.32 2023-10-30T13:38:14.184 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.32 2023-10-30T13:38:18.123 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 69% (136/195) Updating files: 70% (137/195) Updating files: 71% (139/195) Updating files: 72% (141/195) Updating files: 73% (143/195) Updating files: 74% (145/195) Updating files: 75% (147/195) Updating files: 76% (149/195) Updating files: 77% (151/195) Updating files: 78% (153/195) Updating files: 79% (155/195) Updating files: 80% (156/195) Updating files: 81% (158/195) Updating files: 82% (160/195) Updating files: 83% (162/195) Updating files: 84% (164/195) Updating files: 85% (166/195) Updating files: 86% (168/195) Updating files: 87% (170/195) Updating files: 88% (172/195) Updating files: 89% (174/195) Updating files: 90% (176/195) Updating files: 91% (178/195) Updating files: 92% (180/195) Updating files: 93% (182/195) Updating files: 94% (184/195) Updating files: 95% (186/195) Updating files: 96% (188/195) Updating files: 97% (190/195) Updating files: 98% (192/195) Updating files: 99% (194/195) Updating files: 100% (195/195) Updating files: 100% (195/195), done. 2023-10-30T13:38:18.178 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at c08d08baa6a v0.32 2023-10-30T13:38:18.183 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.32 2023-10-30T13:38:18.868 INFO:tasks.workunit.client.0.smithi148.stdout:v0.33 2023-10-30T13:38:18.869 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 33 -eq 48 ] 2023-10-30T13:38:18.869 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.33 2023-10-30T13:38:18.869 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.33 2023-10-30T13:38:18.869 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.33 2023-10-30T13:38:20.806 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 98% (157/160) Updating files: 99% (159/160) Updating files: 100% (160/160) Updating files: 100% (160/160), done. 2023-10-30T13:38:20.839 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 340b4348bf6 v0.33 2023-10-30T13:38:20.843 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.33 2023-10-30T13:38:20.878 INFO:tasks.workunit.client.0.smithi148.stdout:v0.34 2023-10-30T13:38:20.879 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 34 -eq 48 ] 2023-10-30T13:38:20.879 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.34 2023-10-30T13:38:20.879 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.34 2023-10-30T13:38:20.879 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.34 2023-10-30T13:38:22.183 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 2f039eeeb74 v0.34 2023-10-30T13:38:22.186 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.34 2023-10-30T13:38:22.212 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 35 -eq 48 ] 2023-10-30T13:38:22.212 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.35 2023-10-30T13:38:22.212 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.35 2023-10-30T13:38:22.212 INFO:tasks.workunit.client.0.smithi148.stdout:v0.35 2023-10-30T13:38:22.213 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.35 2023-10-30T13:38:30.758 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:30.762 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:38:41.211 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 37% (125/334) Updating files: 38% (127/334) Updating files: 38% (129/334) Updating files: 39% (131/334) Updating files: 40% (134/334) Updating files: 41% (137/334) Updating files: 42% (141/334) Updating files: 43% (144/334) Updating files: 44% (147/334) Updating files: 45% (151/334) Updating files: 46% (154/334) Updating files: 47% (157/334) Updating files: 48% (161/334) Updating files: 49% (164/334) Updating files: 50% (167/334) Updating files: 51% (171/334) Updating files: 52% (174/334) Updating files: 53% (178/334) Updating files: 54% (181/334) Updating files: 55% (184/334) Updating files: 56% (188/334) Updating files: 57% (191/334) Updating files: 58% (194/334) Updating files: 59% (198/334) Updating files: 60% (201/334) Updating files: 61% (204/334) Updating files: 62% (208/334) Updating files: 63% (211/334) Updating files: 64% (214/334) Updating files: 64% (216/334) Updating files: 65% (218/334) Updating files: 66% (221/334) Updating files: 67% (224/334) Updating files: 68% (228/334) Updating files: 69% (231/334) Updating files: 70% (234/334) Updating files: 71% (238/334) Updating files: 72% (241/334) Updating files: 73% (244/334) Updating files: 74% (248/334) Updating files: 75% (251/334) Updating files: 76% (254/334) Updating files: 77% (258/334) Updating files: 78% (261/334) Updating files: 79% (264/334) Updating files: 80% (268/334) Updating files: 81% (271/334) Updating files: 82% (274/334) Updating files: 83% (278/334) Updating files: 84% (281/334) Updating files: 85% (284/334) Updating files: 86% (288/334) Updating files: 87% (291/334) Updating files: 87% (292/334) Updating files: 88% (294/334) Updating files: 89% (298/334) Updating files: 90% (301/334) Updating files: 91% (304/334) Updating files: 92% (308/334) Updating files: 93% (311/334) Updating files: 94% (314/334) Updating files: 95% (318/334) Updating files: 96% (321/334) Updating files: 97% (324/334) Updating files: 98% (328/334) Updating files: 99% (331/334) Updating files: 100% (334/334) Updating files: 100% (334/334), done. 2023-10-30T13:38:41.246 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 0afda378124 v0.35 2023-10-30T13:38:41.250 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.35 2023-10-30T13:38:41.345 INFO:tasks.workunit.client.0.smithi148.stdout:v0.36 2023-10-30T13:38:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 36 -eq 48 ] 2023-10-30T13:38:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.36 2023-10-30T13:38:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.36 2023-10-30T13:38:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.36 2023-10-30T13:38:46.232 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 24% (59/244) Updating files: 25% (61/244) Updating files: 26% (64/244) Updating files: 27% (66/244) Updating files: 28% (69/244) Updating files: 29% (71/244) Updating files: 30% (74/244) Updating files: 31% (76/244) Updating files: 32% (79/244) Updating files: 33% (81/244) Updating files: 34% (83/244) Updating files: 35% (86/244) Updating files: 36% (88/244) Updating files: 37% (91/244) Updating files: 38% (93/244) Updating files: 39% (96/244) Updating files: 40% (98/244) Updating files: 41% (101/244) Updating files: 42% (103/244) Updating files: 43% (105/244) Updating files: 44% (108/244) Updating files: 45% (110/244) Updating files: 46% (113/244) Updating files: 46% (114/244) Updating files: 47% (115/244) Updating files: 48% (118/244) Updating files: 49% (120/244) Updating files: 50% (122/244) Updating files: 51% (125/244) Updating files: 52% (127/244) Updating files: 53% (130/244) Updating files: 54% (132/244) Updating files: 55% (135/244) Updating files: 56% (137/244) Updating files: 57% (140/244) Updating files: 58% (142/244) Updating files: 59% (144/244) Updating files: 60% (147/244) Updating files: 61% (149/244) Updating files: 62% (152/244) Updating files: 63% (154/244) Updating files: 64% (157/244) Updating files: 65% (159/244) Updating files: 66% (162/244) Updating files: 67% (164/244) Updating files: 68% (166/244) Updating files: 69% (169/244) Updating files: 70% (171/244) Updating files: 71% (174/244) Updating files: 72% (176/244) Updating files: 73% (179/244) Updating files: 74% (181/244) Updating files: 75% (183/244) Updating files: 75% (184/244) Updating files: 76% (186/244) Updating files: 77% (188/244) Updating files: 78% (191/244) Updating files: 79% (193/244) Updating files: 80% (196/244) Updating files: 81% (198/244) Updating files: 82% (201/244) Updating files: 83% (203/244) Updating files: 84% (205/244) Updating files: 85% (208/244) Updating files: 86% (210/244) Updating files: 87% (213/244) Updating files: 88% (215/244) Updating files: 89% (218/244) Updating files: 90% (220/244) Updating files: 91% (223/244) Updating files: 92% (225/244) Updating files: 93% (227/244) Updating files: 94% (230/244) Updating files: 95% (232/244) Updating files: 96% (235/244) Updating files: 97% (237/244) Updating files: 98% (240/244) Updating files: 99% (242/244) Updating files: 100% (244/244) Updating files: 100% (244/244), done. 2023-10-30T13:38:46.256 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 877cacbbdbb v0.36 2023-10-30T13:38:46.258 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.36 2023-10-30T13:38:46.385 INFO:tasks.workunit.client.0.smithi148.stdout:v0.37 2023-10-30T13:38:46.386 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 37 -eq 48 ] 2023-10-30T13:38:46.386 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.37 2023-10-30T13:38:46.386 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.37 2023-10-30T13:38:46.386 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.37 2023-10-30T13:38:49.034 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 74% (135/181) Updating files: 75% (136/181) Updating files: 76% (138/181) Updating files: 77% (140/181) Updating files: 78% (142/181) Updating files: 79% (143/181) Updating files: 80% (145/181) Updating files: 81% (147/181) Updating files: 82% (149/181) Updating files: 83% (151/181) Updating files: 84% (153/181) Updating files: 85% (154/181) Updating files: 86% (156/181) Updating files: 87% (158/181) Updating files: 88% (160/181) Updating files: 89% (162/181) Updating files: 90% (163/181) Updating files: 91% (165/181) Updating files: 92% (167/181) Updating files: 93% (169/181) Updating files: 94% (171/181) Updating files: 95% (172/181) Updating files: 96% (174/181) Updating files: 97% (176/181) Updating files: 98% (178/181) Updating files: 99% (180/181) Updating files: 100% (181/181) Updating files: 100% (181/181), done. 2023-10-30T13:38:49.067 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at a6f3bbb744a v0.37 2023-10-30T13:38:49.070 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.37 2023-10-30T13:38:49.333 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 38 -eq 48 ] 2023-10-30T13:38:49.333 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.38 2023-10-30T13:38:49.333 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.38 2023-10-30T13:38:49.334 INFO:tasks.workunit.client.0.smithi148.stdout:v0.38 2023-10-30T13:38:49.334 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.38 2023-10-30T13:38:50.817 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 83% (143/171) Updating files: 84% (144/171) Updating files: 85% (146/171) Updating files: 86% (148/171) Updating files: 87% (149/171) Updating files: 88% (151/171) Updating files: 89% (153/171) Updating files: 90% (154/171) Updating files: 91% (156/171) Updating files: 92% (158/171) Updating files: 93% (160/171) Updating files: 94% (161/171) Updating files: 95% (163/171) Updating files: 96% (165/171) Updating files: 97% (166/171) Updating files: 98% (168/171) Updating files: 99% (170/171) Updating files: 100% (171/171) Updating files: 100% (171/171), done. 2023-10-30T13:38:50.855 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at b600ec2ac7c v0.38 2023-10-30T13:38:50.858 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.38 2023-10-30T13:38:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:v0.39 2023-10-30T13:38:51.134 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 39 -eq 48 ] 2023-10-30T13:38:51.134 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.39 2023-10-30T13:38:51.135 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.39 2023-10-30T13:38:51.135 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.39 2023-10-30T13:39:00.975 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:00.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:09.584 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 23% (61/258) Updating files: 24% (62/258) Updating files: 25% (65/258) Updating files: 26% (68/258) Updating files: 27% (70/258) Updating files: 28% (73/258) Updating files: 29% (75/258) Updating files: 30% (78/258) Updating files: 30% (79/258) Updating files: 31% (80/258) Updating files: 32% (83/258) Updating files: 33% (86/258) Updating files: 34% (88/258) Updating files: 35% (91/258) Updating files: 36% (93/258) Updating files: 37% (96/258) Updating files: 38% (99/258) Updating files: 39% (101/258) Updating files: 40% (104/258) Updating files: 41% (106/258) Updating files: 41% (108/258) Updating files: 42% (109/258) Updating files: 43% (111/258) Updating files: 44% (114/258) Updating files: 45% (117/258) Updating files: 46% (119/258) Updating files: 47% (122/258) Updating files: 48% (124/258) Updating files: 49% (127/258) Updating files: 49% (128/258) Updating files: 50% (129/258) Updating files: 51% (132/258) Updating files: 52% (135/258) Updating files: 53% (137/258) Updating files: 53% (139/258) Updating files: 54% (140/258) Updating files: 55% (142/258) Updating files: 56% (145/258) Updating files: 57% (148/258) Updating files: 58% (150/258) Updating files: 59% (153/258) Updating files: 60% (155/258) Updating files: 61% (158/258) Updating files: 62% (160/258) Updating files: 63% (163/258) Updating files: 64% (166/258) Updating files: 65% (168/258) Updating files: 66% (171/258) Updating files: 67% (173/258) Updating files: 68% (176/258) Updating files: 69% (179/258) Updating files: 70% (181/258) Updating files: 71% (184/258) Updating files: 72% (186/258) Updating files: 73% (189/258) Updating files: 74% (191/258) Updating files: 75% (194/258) Updating files: 76% (197/258) Updating files: 77% (199/258) Updating files: 78% (202/258) Updating files: 79% (204/258) Updating files: 80% (207/258) Updating files: 81% (209/258) Updating files: 82% (212/258) Updating files: 83% (215/258) Updating files: 84% (217/258) Updating files: 84% (219/258) Updating files: 85% (220/258) Updating files: 86% (222/258) Updating files: 87% (225/258) Updating files: 88% (228/258) Updating files: 89% (230/258) Updating files: 90% (233/258) Updating files: 91% (235/258) Updating files: 92% (238/258) Updating files: 93% (240/258) Updating files: 94% (243/258) Updating files: 95% (246/258) Updating files: 96% (248/258) Updating files: 97% (251/258) Updating files: 98% (253/258) Updating files: 99% (256/258) Updating files: 100% (258/258) Updating files: 100% (258/258), done. 2023-10-30T13:39:09.625 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 321ecdaba2c v0.39 2023-10-30T13:39:09.629 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.39 2023-10-30T13:39:09.752 INFO:tasks.workunit.client.0.smithi148.stdout:v0.40 2023-10-30T13:39:09.752 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 40 -eq 48 ] 2023-10-30T13:39:09.752 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.40 2023-10-30T13:39:09.752 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.40 2023-10-30T13:39:09.752 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.40 2023-10-30T13:39:13.612 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 11% (25/226) Updating files: 12% (28/226) Updating files: 13% (30/226) Updating files: 14% (32/226) Updating files: 15% (34/226) Updating files: 16% (37/226) Updating files: 17% (39/226) Updating files: 18% (41/226) Updating files: 19% (43/226) Updating files: 20% (46/226) Updating files: 21% (48/226) Updating files: 22% (50/226) Updating files: 23% (52/226) Updating files: 24% (55/226) Updating files: 25% (57/226) Updating files: 26% (59/226) Updating files: 27% (62/226) Updating files: 28% (64/226) Updating files: 29% (66/226) Updating files: 30% (68/226) Updating files: 31% (71/226) Updating files: 32% (73/226) Updating files: 33% (75/226) Updating files: 34% (77/226) Updating files: 34% (78/226) Updating files: 35% (80/226) Updating files: 36% (82/226) Updating files: 37% (84/226) Updating files: 38% (86/226) Updating files: 39% (89/226) Updating files: 40% (91/226) Updating files: 41% (93/226) Updating files: 42% (95/226) Updating files: 43% (98/226) Updating files: 44% (100/226) Updating files: 45% (102/226) Updating files: 46% (104/226) Updating files: 47% (107/226) Updating files: 48% (109/226) Updating files: 49% (111/226) Updating files: 50% (113/226) Updating files: 51% (116/226) Updating files: 52% (118/226) Updating files: 53% (120/226) Updating files: 54% (123/226) Updating files: 55% (125/226) Updating files: 56% (127/226) Updating files: 57% (129/226) Updating files: 58% (132/226) Updating files: 59% (134/226) Updating files: 60% (136/226) Updating files: 61% (138/226) Updating files: 62% (141/226) Updating files: 63% (143/226) Updating files: 64% (145/226) Updating files: 65% (147/226) Updating files: 66% (150/226) Updating files: 67% (152/226) Updating files: 68% (154/226) Updating files: 69% (156/226) Updating files: 70% (159/226) Updating files: 71% (161/226) Updating files: 72% (163/226) Updating files: 73% (165/226) Updating files: 74% (168/226) Updating files: 75% (170/226) Updating files: 76% (172/226) Updating files: 77% (175/226) Updating files: 78% (177/226) Updating files: 79% (179/226) Updating files: 80% (181/226) Updating files: 81% (184/226) Updating files: 82% (186/226) Updating files: 83% (188/226) Updating files: 84% (190/226) Updating files: 85% (193/226) Updating files: 86% (195/226) Updating files: 87% (197/226) Updating files: 88% (199/226) Updating files: 89% (202/226) Updating files: 90% (204/226) Updating files: 91% (206/226) Updating files: 92% (208/226) Updating files: 93% (211/226) Updating files: 94% (213/226) Updating files: 95% (215/226) Updating files: 96% (217/226) Updating files: 97% (220/226) Updating files: 98% (222/226) Updating files: 99% (224/226) Updating files: 100% (226/226) Updating files: 100% (226/226), done. 2023-10-30T13:39:13.647 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 7eea40ea37f v0.40 2023-10-30T13:39:13.650 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.40 2023-10-30T13:39:13.791 INFO:tasks.workunit.client.0.smithi148.stdout:v0.41 2023-10-30T13:39:13.791 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 41 -eq 48 ] 2023-10-30T13:39:13.792 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.41 2023-10-30T13:39:13.792 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.41 2023-10-30T13:39:13.792 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.41 2023-10-30T13:39:31.048 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:31.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:39:33.468 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 89% (78/87) Updating files: 90% (79/87) Updating files: 91% (80/87) Updating files: 93% (81/87) Updating files: 94% (82/87) Updating files: 95% (83/87) Updating files: 96% (84/87) Updating files: 97% (85/87) Updating files: 98% (86/87) Updating files: 100% (87/87) Updating files: 100% (87/87), done. 2023-10-30T13:39:34.362 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at c1345f7136a v0.41 2023-10-30T13:39:34.364 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.41 2023-10-30T13:39:34.398 INFO:tasks.workunit.client.0.smithi148.stdout:v0.42 2023-10-30T13:39:34.399 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 42 -eq 48 ] 2023-10-30T13:39:34.399 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.42 2023-10-30T13:39:34.399 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.42 2023-10-30T13:39:34.399 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.42 2023-10-30T13:39:37.989 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 23% (58/250) Updating files: 24% (60/250) Updating files: 25% (63/250) Updating files: 26% (65/250) Updating files: 27% (68/250) Updating files: 28% (70/250) Updating files: 29% (73/250) Updating files: 30% (75/250) Updating files: 31% (78/250) Updating files: 32% (80/250) Updating files: 33% (83/250) Updating files: 34% (85/250) Updating files: 35% (88/250) Updating files: 36% (90/250) Updating files: 37% (93/250) Updating files: 38% (95/250) Updating files: 39% (98/250) Updating files: 40% (100/250) Updating files: 41% (103/250) Updating files: 42% (105/250) Updating files: 43% (108/250) Updating files: 44% (110/250) Updating files: 45% (113/250) Updating files: 46% (115/250) Updating files: 47% (118/250) Updating files: 48% (120/250) Updating files: 49% (123/250) Updating files: 50% (125/250) Updating files: 51% (128/250) Updating files: 52% (130/250) Updating files: 53% (133/250) Updating files: 54% (135/250) Updating files: 55% (138/250) Updating files: 56% (140/250) Updating files: 57% (143/250) Updating files: 58% (145/250) Updating files: 59% (148/250) Updating files: 60% (150/250) Updating files: 61% (153/250) Updating files: 62% (155/250) Updating files: 63% (158/250) Updating files: 64% (160/250) Updating files: 65% (163/250) Updating files: 66% (165/250) Updating files: 67% (168/250) Updating files: 68% (170/250) Updating files: 69% (173/250) Updating files: 70% (175/250) Updating files: 71% (178/250) Updating files: 72% (180/250) Updating files: 73% (183/250) Updating files: 74% (185/250) Updating files: 75% (188/250) Updating files: 76% (190/250) Updating files: 77% (193/250) Updating files: 78% (195/250) Updating files: 79% (198/250) Updating files: 80% (200/250) Updating files: 81% (203/250) Updating files: 82% (205/250) Updating files: 83% (208/250) Updating files: 84% (210/250) Updating files: 85% (213/250) Updating files: 86% (215/250) Updating files: 87% (218/250) Updating files: 88% (220/250) Updating files: 89% (223/250) Updating files: 90% (225/250) Updating files: 91% (228/250) Updating files: 92% (230/250) Updating files: 93% (233/250) Updating files: 94% (235/250) Updating files: 95% (238/250) Updating files: 96% (240/250) Updating files: 97% (243/250) Updating files: 98% (245/250) Updating files: 99% (248/250) Updating files: 100% (250/250) Updating files: 100% (250/250), done. 2023-10-30T13:39:38.032 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at b205c64c687 v0.42 2023-10-30T13:39:38.035 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.42 2023-10-30T13:39:38.124 INFO:tasks.workunit.client.0.smithi148.stdout:v0.43 2023-10-30T13:39:38.124 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 43 -eq 48 ] 2023-10-30T13:39:38.124 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.43 2023-10-30T13:39:38.124 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.43 2023-10-30T13:39:38.125 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.43 2023-10-30T13:39:40.831 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:40.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.976+0000 7fbef2017640 1 Processor -- start 2023-10-30T13:39:40.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.976+0000 7fbef2017640 1 -- start start 2023-10-30T13:39:40.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.976+0000 7fbef2017640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbeec0c6b80 0x7fbeec0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:40.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.976+0000 7fbef2017640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbeec13f4a0 0x7fbeec0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:40.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.976+0000 7fbef2017640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeec0c0a50 con 0x7fbeec0c6f60 2023-10-30T13:39:40.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.976+0000 7fbef2017640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeec06f6e0 con 0x7fbeec0c6b80 2023-10-30T13:39:40.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.976+0000 7fbef2017640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeec06f9c0 con 0x7fbeec13f4a0 2023-10-30T13:39:40.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeeb7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbeec0c6b80 0x7fbeec0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:40.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeeb7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fbeec0c6b80 0x7fbeec0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55572/0 (socket says 172.21.15.148:55572) 2023-10-30T13:39:40.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeeb7fe640 1 -- 172.21.15.148:0/3150088443 learned_addr learned my addr 172.21.15.148:0/3150088443 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:40.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeeaffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbeec0c6f60 0x7fbeec13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37096/0 (socket says 172.21.15.148:37096) 2023-10-30T13:39:40.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeebfff640 1 --2- 172.21.15.148:0/3150088443 >> v2:172.21.15.150:3300/0 conn(0x7fbeec13f4a0 0x7fbeec0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:40.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeeb7fe640 1 -- 172.21.15.148:0/3150088443 >> v2:172.21.15.150:3300/0 conn(0x7fbeec13f4a0 msgr2=0x7fbeec0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:40.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeeb7fe640 1 --2- 172.21.15.148:0/3150088443 >> v2:172.21.15.150:3300/0 conn(0x7fbeec13f4a0 0x7fbeec0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:40.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeeb7fe640 1 -- 172.21.15.148:0/3150088443 >> v1:172.21.15.148:6789/0 conn(0x7fbeec0c6f60 legacy=0x7fbeec13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:40.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeeb7fe640 1 -- 172.21.15.148:0/3150088443 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec06fca0 con 0x7fbeec0c6b80 2023-10-30T13:39:40.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeeb7fe640 1 --2- 172.21.15.148:0/3150088443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 0x7fbeec0bfec0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fbeec0c8980 tx=0x7fbedc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae6c662de3b6575 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:40.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeea7fc640 1 -- 172.21.15.148:0/3150088443 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0ab610 con 0x7fbeec0c6b80 2023-10-30T13:39:40.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeea7fc640 1 -- 172.21.15.148:0/3150088443 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbedc0ab770 con 0x7fbeec0c6b80 2023-10-30T13:39:40.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.980+0000 7fbeea7fc640 1 -- 172.21.15.148:0/3150088443 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0abb00 con 0x7fbeec0c6b80 2023-10-30T13:39:40.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 -- 172.21.15.148:0/3150088443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 msgr2=0x7fbeec0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:40.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 --2- 172.21.15.148:0/3150088443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 0x7fbeec0bfec0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fbeec0c8980 tx=0x7fbedc01c500 comp rx=0 tx=0).stop 2023-10-30T13:39:40.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 -- 172.21.15.148:0/3150088443 shutdown_connections 2023-10-30T13:39:40.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 --2- 172.21.15.148:0/3150088443 >> v2:172.21.15.150:3300/0 conn(0x7fbeec13f4a0 0x7fbeec0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:40.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 --2- 172.21.15.148:0/3150088443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 0x7fbeec0bfec0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:40.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 -- 172.21.15.148:0/3150088443 >> 172.21.15.148:0/3150088443 conn(0x7fbeec0c7910 msgr2=0x7fbeec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:40.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 -- 172.21.15.148:0/3150088443 shutdown_connections 2023-10-30T13:39:40.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 -- 172.21.15.148:0/3150088443 wait complete. 2023-10-30T13:39:40.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 Processor -- start 2023-10-30T13:39:40.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 -- start start 2023-10-30T13:39:40.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 0x7fbeec0ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:40.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbeec0c6f60 0x7fbeec138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:40.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbeec13f4a0 0x7fbeec1392c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:40.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec1555c0 con 0x7fbeec0c6b80 2023-10-30T13:39:40.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec1f5440 con 0x7fbeec13f4a0 2023-10-30T13:39:40.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbef2017640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec1f5720 con 0x7fbeec0c6f60 2023-10-30T13:39:40.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbeeb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 0x7fbeec0ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:40.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbeeb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 0x7fbeec0ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55586/0 (socket says 172.21.15.148:55586) 2023-10-30T13:39:40.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbeeb7fe640 1 -- 172.21.15.148:0/3493157692 learned_addr learned my addr 172.21.15.148:0/3493157692 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:40.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.984+0000 7fbeeaffd640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbeec0c6f60 0x7fbeec138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:40.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbeebfff640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbeec13f4a0 0x7fbeec1392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:40.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbeeb7fe640 1 -- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbeec0c6f60 msgr2=0x7fbeec138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:40.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbeeb7fe640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbeec0c6f60 0x7fbeec138d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:40.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbeeb7fe640 1 -- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbeec13f4a0 msgr2=0x7fbeec1392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:40.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbeeb7fe640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbeec13f4a0 0x7fbeec1392c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:40.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbeeb7fe640 1 -- 172.21.15.148:0/3493157692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbedc0094d0 con 0x7fbeec0c6b80 2023-10-30T13:39:40.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbeeb7fe640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 0x7fbeec0ca570 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fbedc019110 tx=0x7fbedc01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbed3fff640 1 -- 172.21.15.148:0/3493157692 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0b4800 con 0x7fbeec0c6b80 2023-10-30T13:39:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbed3fff640 1 -- 172.21.15.148:0/3493157692 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbedc0b4960 con 0x7fbeec0c6b80 2023-10-30T13:39:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeec1f5a00 con 0x7fbeec0c6b80 2023-10-30T13:39:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbed3fff640 1 -- 172.21.15.148:0/3493157692 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0b4d10 con 0x7fbeec0c6b80 2023-10-30T13:39:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeec1f5fc0 con 0x7fbeec0c6b80 2023-10-30T13:39:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeec1544a0 con 0x7fbeec0c6b80 2023-10-30T13:39:40.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbed3fff640 1 -- 172.21.15.148:0/3493157692 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbedc0b2410 con 0x7fbeec0c6b80 2023-10-30T13:39:40.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbed3fff640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbed80664a0 0x7fbed8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:40.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbed3fff640 1 -- 172.21.15.148:0/3493157692 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fbedc11e650 con 0x7fbeec0c6b80 2023-10-30T13:39:40.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.988+0000 7fbeeaffd640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbed80664a0 0x7fbed8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:40.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.992+0000 7fbeeaffd640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbed80664a0 0x7fbed8068960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbeec134ce0 tx=0x7fbee0007c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:40.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:40.992+0000 7fbed3fff640 1 -- 172.21.15.148:0/3493157692 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbeec1544a0 con 0x7fbeec0c6b80 2023-10-30T13:39:41.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.172+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbeec1544a0 con 0x7fbeec0c6b80 2023-10-30T13:39:41.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbed3fff640 1 -- 172.21.15.148:0/3493157692 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6512 (secure 0 0 0) 0x7fbeec1544a0 con 0x7fbeec0c6b80 2023-10-30T13:39:41.174 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:41.174 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7149,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:41.175 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 113 2023-10-30T13:39:41.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbed80664a0 msgr2=0x7fbed8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbed80664a0 0x7fbed8068960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbeec134ce0 tx=0x7fbee0007c90 comp rx=0 tx=0).stop 2023-10-30T13:39:41.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 msgr2=0x7fbeec0ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 0x7fbeec0ca570 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fbedc019110 tx=0x7fbedc01ad40 comp rx=0 tx=0).stop 2023-10-30T13:39:41.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 shutdown_connections 2023-10-30T13:39:41.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbed80664a0 0x7fbed8068960 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbeec13f4a0 0x7fbeec1392c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbeec0c6f60 0x7fbeec138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 --2- 172.21.15.148:0/3493157692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbeec0c6b80 0x7fbeec0ca570 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 >> 172.21.15.148:0/3493157692 conn(0x7fbeec0c7910 msgr2=0x7fbeec069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:41.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 shutdown_connections 2023-10-30T13:39:41.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.176+0000 7fbef2017640 1 -- 172.21.15.148:0/3493157692 wait complete. 2023-10-30T13:39:41.195 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:39:41.195 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:39:41.195 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:39:41.223 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:39:41.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:41.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89ca79640 1 Processor -- start 2023-10-30T13:39:41.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89ca79640 1 -- start start 2023-10-30T13:39:41.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89ca79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c6bd0 0x7fb89806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89ca79640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb898071110 0x7fb898158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89ca79640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb898066cf0 con 0x7fb89806f7a0 2023-10-30T13:39:41.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89ca79640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8980700c0 con 0x7fb898071110 2023-10-30T13:39:41.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89ca79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980c70d0 con 0x7fb8980c6bd0 2023-10-30T13:39:41.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb89806f7a0 0x7fb898069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37112/0 (socket says 172.21.15.148:37112) 2023-10-30T13:39:41.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89659c640 1 -- 172.21.15.148:0/3062562292 learned_addr learned my addr 172.21.15.148:0/3062562292 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:39:41.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb896d9d640 1 --2- 172.21.15.148:0/3062562292 >> v2:172.21.15.150:3300/0 conn(0x7fb898071110 0x7fb898158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb88dd9b640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c6bd0 0x7fb89806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724371371 0 0) 0x7fb898066cf0 con 0x7fb89806f7a0 2023-10-30T13:39:41.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb884003530 con 0x7fb89806f7a0 2023-10-30T13:39:41.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751346188 0 0) 0x7fb884003530 con 0x7fb89806f7a0 2023-10-30T13:39:41.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c6bd0 msgr2=0x7fb89806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:41.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c6bd0 0x7fb89806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 >> v2:172.21.15.150:3300/0 conn(0x7fb898071110 msgr2=0x7fb898158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:41.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 --2- 172.21.15.148:0/3062562292 >> v2:172.21.15.150:3300/0 conn(0x7fb898071110 0x7fb898158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980c7390 con 0x7fb89806f7a0 2023-10-30T13:39:41.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fb880001680 con 0x7fb89806f7a0 2023-10-30T13:39:41.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 >> v1:172.21.15.148:6789/0 conn(0x7fb89806f7a0 legacy=0x7fb898069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c6bd0 0x7fb884003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb8840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb884004d10 0x7fb884005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980700c0 con 0x7fb884003f00 2023-10-30T13:39:41.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980c70d0 con 0x7fb884004d10 2023-10-30T13:39:41.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88400b9c0 con 0x7fb8980c6bd0 2023-10-30T13:39:41.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb896d9d640 1 --2- 172.21.15.148:0/3062562292 >> v2:172.21.15.150:3300/0 conn(0x7fb898071110 0x7fb898158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:39:41.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb88dd9b640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb8840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb89659c640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c6bd0 0x7fb884003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb88dd9b640 1 -- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c6bd0 msgr2=0x7fb884003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:41.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb896d9d640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb884004d10 0x7fb884005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb88dd9b640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c6bd0 0x7fb884003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb88dd9b640 1 -- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb884004d10 msgr2=0x7fb884005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:41.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb88dd9b640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb884004d10 0x7fb884005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb88dd9b640 1 -- 172.21.15.148:0/3062562292 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb884003530 con 0x7fb884003f00 2023-10-30T13:39:41.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.404+0000 7fb896d9d640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb884004d10 0x7fb884005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:39:41.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb88dd9b640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb8840047d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb888009ae0 tx=0x7fb88801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9629471d87550c99 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:41.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8880a5640 con 0x7fb884003f00 2023-10-30T13:39:41.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8880a57a0 con 0x7fb884003f00 2023-10-30T13:39:41.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb895d9b640 1 -- 172.21.15.148:0/3062562292 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8880a5b50 con 0x7fb884003f00 2023-10-30T13:39:41.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb89ca79640 1 -- 172.21.15.148:0/3062562292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 msgr2=0x7fb8840047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb89ca79640 1 --2- 172.21.15.148:0/3062562292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb8840047d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb888009ae0 tx=0x7fb88801ca50 comp rx=0 tx=0).stop 2023-10-30T13:39:41.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb896d9d640 1 -- 172.21.15.148:0/3062562292 reap_dead start 2023-10-30T13:39:41.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb89ca79640 1 -- 172.21.15.148:0/3062562292 shutdown_connections 2023-10-30T13:39:41.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb89ca79640 1 -- 172.21.15.148:0/3062562292 >> 172.21.15.148:0/3062562292 conn(0x7fb8980c7910 msgr2=0x7fb8980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:41.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb89ca79640 1 -- 172.21.15.148:0/3062562292 shutdown_connections 2023-10-30T13:39:41.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.408+0000 7fb89ca79640 1 -- 172.21.15.148:0/3062562292 wait complete. 2023-10-30T13:39:41.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 Processor -- start 2023-10-30T13:39:41.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 -- start start 2023-10-30T13:39:41.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb89815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8981f0970 0x7fb8981521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c2bf0 0x7fb8981549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980c2790 con 0x7fb884003f00 2023-10-30T13:39:41.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981efc40 con 0x7fb8981f0970 2023-10-30T13:39:41.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981f04e0 con 0x7fb8980c2bf0 2023-10-30T13:39:41.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb89815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb89815b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55612/0 (socket says 172.21.15.148:55612) 2023-10-30T13:39:41.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89659c640 1 -- 172.21.15.148:0/166131254 learned_addr learned my addr 172.21.15.148:0/166131254 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:41.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb88dd9b640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8981f0970 0x7fb8981521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb896d9d640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c2bf0 0x7fb8981549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89659c640 1 -- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c2bf0 msgr2=0x7fb8981549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89659c640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c2bf0 0x7fb8981549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89659c640 1 -- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8981f0970 msgr2=0x7fb8981521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89659c640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8981f0970 0x7fb8981521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89659c640 1 -- 172.21.15.148:0/166131254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980c5e80 con 0x7fb884003f00 2023-10-30T13:39:41.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89659c640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb89815b770 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fb88000e820 tx=0x7fb88000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:41.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb88f7fe640 1 -- 172.21.15.148:0/166131254 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88000c850 con 0x7fb884003f00 2023-10-30T13:39:41.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb88f7fe640 1 -- 172.21.15.148:0/166131254 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88000c9d0 con 0x7fb884003f00 2023-10-30T13:39:41.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8980c6120 con 0x7fb884003f00 2023-10-30T13:39:41.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb88f7fe640 1 -- 172.21.15.148:0/166131254 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88000cd80 con 0x7fb884003f00 2023-10-30T13:39:41.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.412+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8981f0170 con 0x7fb884003f00 2023-10-30T13:39:41.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.416+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89804e170 con 0x7fb884003f00 2023-10-30T13:39:41.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.416+0000 7fb88f7fe640 1 -- 172.21.15.148:0/166131254 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb88001d460 con 0x7fb884003f00 2023-10-30T13:39:41.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.416+0000 7fb88f7fe640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8780664a0 0x7fb878068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:41.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.416+0000 7fb88f7fe640 1 -- 172.21.15.148:0/166131254 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7fb880084440 con 0x7fb884003f00 2023-10-30T13:39:41.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.416+0000 7fb88dd9b640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8780664a0 0x7fb878068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:41.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.416+0000 7fb88dd9b640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8780664a0 0x7fb878068960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb8880af000 tx=0x7fb8880046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:41.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.420+0000 7fb88f7fe640 1 -- 172.21.15.148:0/166131254 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8800512d0 con 0x7fb884003f00 2023-10-30T13:39:41.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.604+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb898152710 con 0x7fb884003f00 2023-10-30T13:39:41.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.604+0000 7fb88f7fe640 1 -- 172.21.15.148:0/166131254 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6512 (secure 0 0 0) 0x7fb880001600 con 0x7fb884003f00 2023-10-30T13:39:41.604 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:41.604 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7149,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:41.605 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 113 2023-10-30T13:39:41.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8780664a0 msgr2=0x7fb878068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8780664a0 0x7fb878068960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb8880af000 tx=0x7fb8880046d0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 msgr2=0x7fb89815b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:41.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb89815b770 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fb88000e820 tx=0x7fb88000ece0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 shutdown_connections 2023-10-30T13:39:41.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8780664a0 0x7fb878068960 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8980c2bf0 0x7fb8981549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8981f0970 0x7fb8981521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 --2- 172.21.15.148:0/166131254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb884003f00 0x7fb89815b770 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 >> 172.21.15.148:0/166131254 conn(0x7fb8980c7910 msgr2=0x7fb8980c3580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 shutdown_connections 2023-10-30T13:39:41.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:41.608+0000 7fb89ca79640 1 -- 172.21.15.148:0/166131254 wait complete. 2023-10-30T13:39:41.625 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:43.626 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:43.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f465a817640 1 Processor -- start 2023-10-30T13:39:43.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f465a817640 1 -- start start 2023-10-30T13:39:43.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f465a817640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f465415d6f0 0x7f465415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f465a817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46540cf160 0x7f46540cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f465a817640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46540cb800 con 0x7f46541538a0 2023-10-30T13:39:43.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f465a817640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46540cb330 con 0x7f465415d6f0 2023-10-30T13:39:43.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f465a817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46540cad70 con 0x7f46540cf160 2023-10-30T13:39:43.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f46537fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f465415d6f0 0x7f465415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:43.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f46537fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f465415d6f0 0x7f465415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58876/0 (socket says 172.21.15.148:58876) 2023-10-30T13:39:43.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f46537fe640 1 -- 172.21.15.148:0/2146731791 learned_addr learned my addr 172.21.15.148:0/2146731791 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:43.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f4658db4640 1 --2- 172.21.15.148:0/2146731791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46540cf160 0x7f46540cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:43.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f46537fe640 1 -- 172.21.15.148:0/2146731791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46540cf160 msgr2=0x7f46540cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:43.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f46537fe640 1 --2- 172.21.15.148:0/2146731791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46540cf160 0x7f46540cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f46537fe640 1 -- 172.21.15.148:0/2146731791 >> v1:172.21.15.148:6789/0 conn(0x7f46541538a0 legacy=0x7f46540ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:43.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.776+0000 7f46537fe640 1 -- 172.21.15.148:0/2146731791 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46540cb050 con 0x7f465415d6f0 2023-10-30T13:39:43.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f46537fe640 1 --2- 172.21.15.148:0/2146731791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f465415d6f0 0x7f465415a090 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4648009870 tx=0x7f464801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920bb61e35966005 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:43.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f4652ffd640 1 -- 172.21.15.148:0/2146731791 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46480a8a30 con 0x7f465415d6f0 2023-10-30T13:39:43.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f4652ffd640 1 -- 172.21.15.148:0/2146731791 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46480a8b90 con 0x7f465415d6f0 2023-10-30T13:39:43.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f4652ffd640 1 -- 172.21.15.148:0/2146731791 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46480b0440 con 0x7f465415d6f0 2023-10-30T13:39:43.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f465a817640 1 -- 172.21.15.148:0/2146731791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f465415d6f0 msgr2=0x7f465415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:43.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f465a817640 1 --2- 172.21.15.148:0/2146731791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f465415d6f0 0x7f465415a090 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4648009870 tx=0x7f464801ed30 comp rx=0 tx=0).stop 2023-10-30T13:39:43.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f465a817640 1 -- 172.21.15.148:0/2146731791 shutdown_connections 2023-10-30T13:39:43.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f465a817640 1 --2- 172.21.15.148:0/2146731791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f465415d6f0 0x7f465415a090 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f465a817640 1 --2- 172.21.15.148:0/2146731791 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46540cf160 0x7f46540cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f465a817640 1 -- 172.21.15.148:0/2146731791 >> 172.21.15.148:0/2146731791 conn(0x7f465406bfe0 msgr2=0x7f465406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:43.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f465a817640 1 -- 172.21.15.148:0/2146731791 shutdown_connections 2023-10-30T13:39:43.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.780+0000 7f465a817640 1 -- 172.21.15.148:0/2146731791 wait complete. 2023-10-30T13:39:43.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 Processor -- start 2023-10-30T13:39:43.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 -- start start 2023-10-30T13:39:43.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46540cf160 0x7f4654159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46541538a0 0x7f4654148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f465415d6f0 0x7f46540c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4654158f60 con 0x7f46540cf160 2023-10-30T13:39:43.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46541e4660 con 0x7f465415d6f0 2023-10-30T13:39:43.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4654158a20 con 0x7f46541538a0 2023-10-30T13:39:43.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4653fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46540cf160 0x7f4654159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:43.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4653fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46540cf160 0x7f4654159350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58892/0 (socket says 172.21.15.148:58892) 2023-10-30T13:39:43.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4653fff640 1 -- 172.21.15.148:0/1515137042 learned_addr learned my addr 172.21.15.148:0/1515137042 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:43.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4658db4640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f465415d6f0 0x7f46540c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:43.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f46537fe640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46541538a0 0x7f4654148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:43.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4653fff640 1 -- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46541538a0 msgr2=0x7f4654148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:43.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4653fff640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46541538a0 0x7f4654148a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4653fff640 1 -- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f465415d6f0 msgr2=0x7f46540c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:43.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4653fff640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f465415d6f0 0x7f46540c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4653fff640 1 -- 172.21.15.148:0/1515137042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46480094d0 con 0x7f46540cf160 2023-10-30T13:39:43.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4653fff640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46540cf160 0x7f4654159350 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f464400c530 tx=0x7f464400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:43.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4650ff9640 1 -- 172.21.15.148:0/1515137042 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644005220 con 0x7f46540cf160 2023-10-30T13:39:43.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4650ff9640 1 -- 172.21.15.148:0/1515137042 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4644005380 con 0x7f46540cf160 2023-10-30T13:39:43.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46540c6950 con 0x7f46540cf160 2023-10-30T13:39:43.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f4650ff9640 1 -- 172.21.15.148:0/1515137042 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4644007270 con 0x7f46540cf160 2023-10-30T13:39:43.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.784+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4654158460 con 0x7f46540cf160 2023-10-30T13:39:43.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.788+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46540b2a50 con 0x7f46540cf160 2023-10-30T13:39:43.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.788+0000 7f4650ff9640 1 -- 172.21.15.148:0/1515137042 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46440073d0 con 0x7f46540cf160 2023-10-30T13:39:43.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.788+0000 7f4650ff9640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46400664a0 0x7f4640068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:43.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.788+0000 7f4650ff9640 1 -- 172.21.15.148:0/1515137042 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f4644083ce0 con 0x7f46540cf160 2023-10-30T13:39:43.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.788+0000 7f46537fe640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46400664a0 0x7f4640068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:43.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.792+0000 7f46537fe640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46400664a0 0x7f4640068960 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f46480b2000 tx=0x7f4648008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:43.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.792+0000 7f4650ff9640 1 -- 172.21.15.148:0/1515137042 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4644050c00 con 0x7f46540cf160 2023-10-30T13:39:43.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f465404e170 con 0x7f46540cf160 2023-10-30T13:39:43.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f4650ff9640 1 -- 172.21.15.148:0/1515137042 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f4644054220 con 0x7f46540cf160 2023-10-30T13:39:43.975 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:43.975 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7149,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:43.976 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 113 2023-10-30T13:39:43.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46400664a0 msgr2=0x7f4640068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:43.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46400664a0 0x7f4640068960 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f46480b2000 tx=0x7f4648008040 comp rx=0 tx=0).stop 2023-10-30T13:39:43.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46540cf160 msgr2=0x7f4654159350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:43.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46540cf160 0x7f4654159350 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f464400c530 tx=0x7f464400c9f0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 shutdown_connections 2023-10-30T13:39:43.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f46400664a0 0x7f4640068960 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f465415d6f0 0x7f46540c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f46541538a0 0x7f4654148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 --2- 172.21.15.148:0/1515137042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f46540cf160 0x7f4654159350 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:43.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 >> 172.21.15.148:0/1515137042 conn(0x7f465406bfe0 msgr2=0x7f46540cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:43.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 shutdown_connections 2023-10-30T13:39:43.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:43.976+0000 7f465a817640 1 -- 172.21.15.148:0/1515137042 wait complete. 2023-10-30T13:39:43.996 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:45.999 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:46.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.148+0000 7ff92e5c4640 1 Processor -- start 2023-10-30T13:39:46.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.148+0000 7ff92e5c4640 1 -- start start 2023-10-30T13:39:46.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92e5c4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 0x7ff928070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:46.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92e5c4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff92815e700 0x7ff92806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:46.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92e5c4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9280c5810 con 0x7ff9280c6540 2023-10-30T13:39:46.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92e5c4640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92815eb90 con 0x7ff92815e700 2023-10-30T13:39:46.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92e5c4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280c6960 con 0x7ff9281580e0 2023-10-30T13:39:46.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92cb61640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff92815e700 0x7ff92806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:46.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92cb61640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff92815e700 0x7ff92806a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58900/0 (socket says 172.21.15.148:58900) 2023-10-30T13:39:46.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92cb61640 1 -- 172.21.15.148:0/3009263174 learned_addr learned my addr 172.21.15.148:0/3009263174 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:46.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff927fff640 1 --2- 172.21.15.148:0/3009263174 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 0x7ff928070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:46.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92cb61640 1 -- 172.21.15.148:0/3009263174 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 msgr2=0x7ff928070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:46.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92cb61640 1 --2- 172.21.15.148:0/3009263174 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 0x7ff928070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92cb61640 1 -- 172.21.15.148:0/3009263174 >> v1:172.21.15.150:6789/0 conn(0x7ff9280c6540 legacy=0x7ff928069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:46.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.152+0000 7ff92cb61640 1 -- 172.21.15.148:0/3009263174 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9280c6bb0 con 0x7ff92815e700 2023-10-30T13:39:46.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92cb61640 1 --2- 172.21.15.148:0/3009263174 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff92815e700 0x7ff92806a5a0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7ff920009e50 tx=0x7ff92001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63444b6ba4577da4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:46.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff926ffd640 1 -- 172.21.15.148:0/3009263174 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9200a58a0 con 0x7ff92815e700 2023-10-30T13:39:46.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff926ffd640 1 -- 172.21.15.148:0/3009263174 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9200a5a00 con 0x7ff92815e700 2023-10-30T13:39:46.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff926ffd640 1 -- 172.21.15.148:0/3009263174 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9200a5d90 con 0x7ff92815e700 2023-10-30T13:39:46.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 -- 172.21.15.148:0/3009263174 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff92815e700 msgr2=0x7ff92806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:46.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 --2- 172.21.15.148:0/3009263174 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff92815e700 0x7ff92806a5a0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7ff920009e50 tx=0x7ff92001c9e0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 -- 172.21.15.148:0/3009263174 shutdown_connections 2023-10-30T13:39:46.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 --2- 172.21.15.148:0/3009263174 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff92815e700 0x7ff92806a5a0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 --2- 172.21.15.148:0/3009263174 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 0x7ff928070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 -- 172.21.15.148:0/3009263174 >> 172.21.15.148:0/3009263174 conn(0x7ff9280c7910 msgr2=0x7ff9280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:46.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 -- 172.21.15.148:0/3009263174 shutdown_connections 2023-10-30T13:39:46.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 -- 172.21.15.148:0/3009263174 wait complete. 2023-10-30T13:39:46.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 Processor -- start 2023-10-30T13:39:46.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.156+0000 7ff92e5c4640 1 -- start start 2023-10-30T13:39:46.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92e5c4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9280c6540 0x7ff9280c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:46.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92e5c4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 0x7ff9281491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:46.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92e5c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff92815e700 0x7ff92815f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:46.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92e5c4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280c50a0 con 0x7ff9280c6540 2023-10-30T13:39:46.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92e5c4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92815ef20 con 0x7ff92815e700 2023-10-30T13:39:46.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92e5c4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9280b0cf0 con 0x7ff9281580e0 2023-10-30T13:39:46.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff927fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9280c6540 0x7ff9280c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:46.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff927fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9280c6540 0x7ff9280c49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58910/0 (socket says 172.21.15.148:58910) 2023-10-30T13:39:46.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff927fff640 1 -- 172.21.15.148:0/2346457621 learned_addr learned my addr 172.21.15.148:0/2346457621 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:46.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92cb61640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff92815e700 0x7ff92815f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:46.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff9277fe640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 0x7ff9281491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:46.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff927fff640 1 -- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 msgr2=0x7ff9281491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:46.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff927fff640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 0x7ff9281491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff927fff640 1 -- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff92815e700 msgr2=0x7ff92815f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:46.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff927fff640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff92815e700 0x7ff92815f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff927fff640 1 -- 172.21.15.148:0/2346457621 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff920009b30 con 0x7ff9280c6540 2023-10-30T13:39:46.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff927fff640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9280c6540 0x7ff9280c49f0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff91800d800 tx=0x7ff91800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:46.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff924ff9640 1 -- 172.21.15.148:0/2346457621 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91800b6c0 con 0x7ff9280c6540 2023-10-30T13:39:46.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff924ff9640 1 -- 172.21.15.148:0/2346457621 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91800b820 con 0x7ff9280c6540 2023-10-30T13:39:46.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff928071020 con 0x7ff9280c6540 2023-10-30T13:39:46.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff924ff9640 1 -- 172.21.15.148:0/2346457621 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91800bb90 con 0x7ff9280c6540 2023-10-30T13:39:46.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9280715e0 con 0x7ff9280c6540 2023-10-30T13:39:46.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92804e170 con 0x7ff9280c6540 2023-10-30T13:39:46.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.160+0000 7ff924ff9640 1 -- 172.21.15.148:0/2346457621 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff91800bcf0 con 0x7ff9280c6540 2023-10-30T13:39:46.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.164+0000 7ff924ff9640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff9140664a0 0x7ff914068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:46.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.164+0000 7ff924ff9640 1 -- 172.21.15.148:0/2346457621 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7ff918084ad0 con 0x7ff9280c6540 2023-10-30T13:39:46.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.164+0000 7ff9277fe640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff9140664a0 0x7ff914068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:46.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.164+0000 7ff9277fe640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff9140664a0 0x7ff914068960 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff9280c47b0 tx=0x7ff91c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:46.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.168+0000 7ff924ff9640 1 -- 172.21.15.148:0/2346457621 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9180518d0 con 0x7ff9280c6540 2023-10-30T13:39:46.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9281683e0 con 0x7ff9280c6540 2023-10-30T13:39:46.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff924ff9640 1 -- 172.21.15.148:0/2346457621 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6512 (secure 0 0 0) 0x7ff918054ef0 con 0x7ff9280c6540 2023-10-30T13:39:46.351 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:46.352 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7149,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:46.352 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 113 2023-10-30T13:39:46.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff9140664a0 msgr2=0x7ff914068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:46.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff9140664a0 0x7ff914068960 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff9280c47b0 tx=0x7ff91c009040 comp rx=0 tx=0).stop 2023-10-30T13:39:46.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9280c6540 msgr2=0x7ff9280c49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:46.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9280c6540 0x7ff9280c49f0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff91800d800 tx=0x7ff91800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 shutdown_connections 2023-10-30T13:39:46.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff9140664a0 0x7ff914068960 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff92815e700 0x7ff92815f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff9281580e0 0x7ff9281491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 --2- 172.21.15.148:0/2346457621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9280c6540 0x7ff9280c49f0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:46.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.352+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 >> 172.21.15.148:0/2346457621 conn(0x7ff9280c7910 msgr2=0x7ff9281567f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:46.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.356+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 shutdown_connections 2023-10-30T13:39:46.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:46.356+0000 7ff92e5c4640 1 -- 172.21.15.148:0/2346457621 wait complete. 2023-10-30T13:39:46.373 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:48.373 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.520+0000 7f0710054640 1 Processor -- start 2023-10-30T13:39:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.520+0000 7f0710054640 1 -- start start 2023-10-30T13:39:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.520+0000 7f0710054640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f07080c6540 0x7f070815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.520+0000 7f0710054640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07080c5810 con 0x7f0708070740 2023-10-30T13:39:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f0710054640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07080c69b0 con 0x7f07081580e0 2023-10-30T13:39:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f0710054640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f070806acb0 con 0x7f07080c6540 2023-10-30T13:39:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070e5f1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f07080c6540 0x7f070815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070e5f1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f07080c6540 0x7f070815e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58926/0 (socket says 172.21.15.148:58926) 2023-10-30T13:39:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070e5f1640 1 -- 172.21.15.148:0/728267398 learned_addr learned my addr 172.21.15.148:0/728267398 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070e5f1640 1 -- 172.21.15.148:0/728267398 >> v1:172.21.15.150:6789/0 conn(0x7f07081580e0 legacy=0x7f0708069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070e5f1640 1 -- 172.21.15.148:0/728267398 >> v1:172.21.15.148:6789/0 conn(0x7f0708070740 legacy=0x7f070806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070e5f1640 1 -- 172.21.15.148:0/728267398 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07080c6bb0 con 0x7f07080c6540 2023-10-30T13:39:48.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070e5f1640 1 --2- 172.21.15.148:0/728267398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 0x7f070815e700 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f0700009e50 tx=0x7f070001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395ff0602dbd9b8c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:48.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070cdee640 1 -- 172.21.15.148:0/728267398 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07000a57f0 con 0x7f07080c6540 2023-10-30T13:39:48.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070cdee640 1 -- 172.21.15.148:0/728267398 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07000a5950 con 0x7f07080c6540 2023-10-30T13:39:48.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.524+0000 7f070cdee640 1 -- 172.21.15.148:0/728267398 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07000a5ce0 con 0x7f07080c6540 2023-10-30T13:39:48.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 -- 172.21.15.148:0/728267398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 msgr2=0x7f070815e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:48.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 --2- 172.21.15.148:0/728267398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 0x7f070815e700 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f0700009e50 tx=0x7f070001ca10 comp rx=0 tx=0).stop 2023-10-30T13:39:48.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 -- 172.21.15.148:0/728267398 shutdown_connections 2023-10-30T13:39:48.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 --2- 172.21.15.148:0/728267398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 0x7f070815e700 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:48.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 -- 172.21.15.148:0/728267398 >> 172.21.15.148:0/728267398 conn(0x7f07080c7910 msgr2=0x7f07080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:48.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 -- 172.21.15.148:0/728267398 shutdown_connections 2023-10-30T13:39:48.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 -- 172.21.15.148:0/728267398 wait complete. 2023-10-30T13:39:48.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 Processor -- start 2023-10-30T13:39:48.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 -- start start 2023-10-30T13:39:48.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0708070740 0x7f0708156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:48.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 0x7f07081f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:48.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07081580e0 0x7f070814d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:48.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07081f8d70 con 0x7f07080c6540 2023-10-30T13:39:48.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07081f4610 con 0x7f07081580e0 2023-10-30T13:39:48.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f0710054640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07081f8160 con 0x7f0708070740 2023-10-30T13:39:48.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070d5ef640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 0x7f07081f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:48.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070d5ef640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 0x7f07081f40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58940/0 (socket says 172.21.15.148:58940) 2023-10-30T13:39:48.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070d5ef640 1 -- 172.21.15.148:0/2955574366 learned_addr learned my addr 172.21.15.148:0/2955574366 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:48.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070e5f1640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07081580e0 0x7f070814d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:48.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070ddf0640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0708070740 0x7f0708156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:48.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070d5ef640 1 -- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0708070740 msgr2=0x7f0708156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:48.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070d5ef640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0708070740 0x7f0708156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:48.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070d5ef640 1 -- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07081580e0 msgr2=0x7f070814d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:48.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070d5ef640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07081580e0 0x7f070814d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:48.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.528+0000 7f070d5ef640 1 -- 172.21.15.148:0/2955574366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700009b30 con 0x7f07080c6540 2023-10-30T13:39:48.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f070d5ef640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 0x7f07081f40d0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f06f800c500 tx=0x7f06f800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:48.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f06fe7fc640 1 -- 172.21.15.148:0/2955574366 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8005220 con 0x7f07080c6540 2023-10-30T13:39:48.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f06fe7fc640 1 -- 172.21.15.148:0/2955574366 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f8005380 con 0x7f07080c6540 2023-10-30T13:39:48.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07081f8440 con 0x7f07080c6540 2023-10-30T13:39:48.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f06fe7fc640 1 -- 172.21.15.148:0/2955574366 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8007270 con 0x7f07080c6540 2023-10-30T13:39:48.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07081f8a00 con 0x7f07080c6540 2023-10-30T13:39:48.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07081f8bc0 con 0x7f07080c6540 2023-10-30T13:39:48.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f06fe7fc640 1 -- 172.21.15.148:0/2955574366 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06f80073d0 con 0x7f07080c6540 2023-10-30T13:39:48.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f06fe7fc640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664a0 0x7f06ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:48.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.532+0000 7f06fe7fc640 1 -- 172.21.15.148:0/2955574366 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f06f8083d50 con 0x7f07080c6540 2023-10-30T13:39:48.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.536+0000 7f070ddf0640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664a0 0x7f06ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:48.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.536+0000 7f070ddf0640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664a0 0x7f06ec068960 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0704000f70 tx=0x7f0704009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:48.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.536+0000 7f06fe7fc640 1 -- 172.21.15.148:0/2955574366 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06f8050bf0 con 0x7f07080c6540 2023-10-30T13:39:48.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07081f8f00 con 0x7f07080c6540 2023-10-30T13:39:48.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f06fe7fc640 1 -- 172.21.15.148:0/2955574366 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f06f8054210 con 0x7f07080c6540 2023-10-30T13:39:48.719 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:48.720 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7149,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:48.720 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 113 2023-10-30T13:39:48.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664a0 msgr2=0x7f06ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:48.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664a0 0x7f06ec068960 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0704000f70 tx=0x7f0704009040 comp rx=0 tx=0).stop 2023-10-30T13:39:48.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 msgr2=0x7f07081f40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:48.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 0x7f07081f40d0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f06f800c500 tx=0x7f06f800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:39:48.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 shutdown_connections 2023-10-30T13:39:48.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06ec0664a0 0x7f06ec068960 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:48.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07080c6540 0x7f07081f40d0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:48.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07081580e0 0x7f070814d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:48.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 --2- 172.21.15.148:0/2955574366 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0708070740 0x7f0708156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:48.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 >> 172.21.15.148:0/2955574366 conn(0x7f07080c7910 msgr2=0x7f070814e320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:48.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 shutdown_connections 2023-10-30T13:39:48.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:48.720+0000 7f0710054640 1 -- 172.21.15.148:0/2955574366 wait complete. 2023-10-30T13:39:48.741 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:50.742 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:50.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c10172640 1 Processor -- start 2023-10-30T13:39:50.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c10172640 1 -- start start 2023-10-30T13:39:50.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c10172640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3c08070740 0x7f3c080c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:50.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c10172640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c0815e700 0x7f3c0806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:50.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c10172640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c080c5810 con 0x7f3c081580e0 2023-10-30T13:39:50.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c10172640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0815eb90 con 0x7f3c08070740 2023-10-30T13:39:50.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c10172640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c080c6bb0 con 0x7f3c0815e700 2023-10-30T13:39:50.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c0df0e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3c081580e0 0x7f3c08069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44036/0 (socket says 172.21.15.148:44036) 2023-10-30T13:39:50.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c0df0e640 1 -- 172.21.15.148:0/1077497393 learned_addr learned my addr 172.21.15.148:0/1077497393 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:39:50.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c0d70d640 1 --2- 172.21.15.148:0/1077497393 >> v2:172.21.15.148:3300/0 conn(0x7f3c08070740 0x7f3c080c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:50.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c0e70f640 1 --2- 172.21.15.148:0/1077497393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c0815e700 0x7f3c0806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:50.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c0d70d640 1 -- 172.21.15.148:0/1077497393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c0815e700 msgr2=0x7f3c0806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:50.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c0d70d640 1 --2- 172.21.15.148:0/1077497393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c0815e700 0x7f3c0806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:50.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c0d70d640 1 -- 172.21.15.148:0/1077497393 >> v1:172.21.15.148:6789/0 conn(0x7f3c081580e0 legacy=0x7f3c08069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:50.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.892+0000 7f3c0d70d640 1 -- 172.21.15.148:0/1077497393 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c080c6e70 con 0x7f3c08070740 2023-10-30T13:39:50.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c0d70d640 1 --2- 172.21.15.148:0/1077497393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c08070740 0x7f3c080c6540 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80097c0 tx=0x7f3bf801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7143b07be0e23445 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:50.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c0cf0c640 1 -- 172.21.15.148:0/1077497393 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80ab600 con 0x7f3c08070740 2023-10-30T13:39:50.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c0cf0c640 1 -- 172.21.15.148:0/1077497393 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf80a7050 con 0x7f3c08070740 2023-10-30T13:39:50.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c0cf0c640 1 -- 172.21.15.148:0/1077497393 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80ab980 con 0x7f3c08070740 2023-10-30T13:39:50.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c10172640 1 -- 172.21.15.148:0/1077497393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c08070740 msgr2=0x7f3c080c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:50.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c10172640 1 --2- 172.21.15.148:0/1077497393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c08070740 0x7f3c080c6540 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80097c0 tx=0x7f3bf801c480 comp rx=0 tx=0).stop 2023-10-30T13:39:50.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c10172640 1 -- 172.21.15.148:0/1077497393 shutdown_connections 2023-10-30T13:39:50.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c10172640 1 --2- 172.21.15.148:0/1077497393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c08070740 0x7f3c080c6540 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:50.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c10172640 1 --2- 172.21.15.148:0/1077497393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c0815e700 0x7f3c0806a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:50.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c10172640 1 -- 172.21.15.148:0/1077497393 >> 172.21.15.148:0/1077497393 conn(0x7f3c080c7910 msgr2=0x7f3c080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:50.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c10172640 1 -- 172.21.15.148:0/1077497393 shutdown_connections 2023-10-30T13:39:50.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.896+0000 7f3c10172640 1 -- 172.21.15.148:0/1077497393 wait complete. 2023-10-30T13:39:50.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 Processor -- start 2023-10-30T13:39:50.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 -- start start 2023-10-30T13:39:50.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c08070740 0x7f3c08161700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:50.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c081580e0 0x7f3c081f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:50.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c0815e700 0x7f3c080c0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:50.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c08160d20 con 0x7f3c0815e700 2023-10-30T13:39:50.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c08153e30 con 0x7f3c081580e0 2023-10-30T13:39:50.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c081536d0 con 0x7f3c08070740 2023-10-30T13:39:50.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0df0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c08070740 0x7f3c08161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:50.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0e70f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c0815e700 0x7f3c080c0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:50.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0e70f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c0815e700 0x7f3c080c0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58948/0 (socket says 172.21.15.148:58948) 2023-10-30T13:39:50.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0e70f640 1 -- 172.21.15.148:0/1005008409 learned_addr learned my addr 172.21.15.148:0/1005008409 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:50.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0df0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c08070740 0x7f3c08161700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:41618/0 (socket says 172.21.15.148:41618) 2023-10-30T13:39:50.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0d70d640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c081580e0 0x7f3c081f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:50.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0e70f640 1 -- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c08070740 msgr2=0x7f3c08161700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:50.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0e70f640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c08070740 0x7f3c08161700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:50.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0e70f640 1 -- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c081580e0 msgr2=0x7f3c081f6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:50.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0e70f640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c081580e0 0x7f3c081f6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:50.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0e70f640 1 -- 172.21.15.148:0/1005008409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf8009420 con 0x7f3c0815e700 2023-10-30T13:39:50.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c0e70f640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c0815e700 0x7f3c080c0a50 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3c00007e90 tx=0x7f3c0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:50.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3bfe7fc640 1 -- 172.21.15.148:0/1005008409 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c00014420 con 0x7f3c0815e700 2023-10-30T13:39:50.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3bfe7fc640 1 -- 172.21.15.148:0/1005008409 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c00014580 con 0x7f3c0815e700 2023-10-30T13:39:50.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3bfe7fc640 1 -- 172.21.15.148:0/1005008409 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c00014930 con 0x7f3c0815e700 2023-10-30T13:39:50.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c08152b50 con 0x7f3c0815e700 2023-10-30T13:39:50.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.900+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c08153110 con 0x7f3c0815e700 2023-10-30T13:39:50.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.904+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0804e170 con 0x7f3c0815e700 2023-10-30T13:39:50.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.904+0000 7f3bfe7fc640 1 -- 172.21.15.148:0/1005008409 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c00014a90 con 0x7f3c0815e700 2023-10-30T13:39:50.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.904+0000 7f3bfe7fc640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3bec0664a0 0x7f3bec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:50.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.904+0000 7f3bfe7fc640 1 -- 172.21.15.148:0/1005008409 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f3c00084b60 con 0x7f3c0815e700 2023-10-30T13:39:50.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.904+0000 7f3c0df0e640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3bec0664a0 0x7f3bec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:50.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.904+0000 7f3c0df0e640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3bec0664a0 0x7f3bec068960 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3c04000f70 tx=0x7f3c04007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:50.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:50.908+0000 7f3bfe7fc640 1 -- 172.21.15.148:0/1005008409 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c00051b30 con 0x7f3c0815e700 2023-10-30T13:39:51.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.092+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c08153f70 con 0x7f3c0815e700 2023-10-30T13:39:51.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.092+0000 7f3bfe7fc640 1 -- 172.21.15.148:0/1005008409 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f3c00055150 con 0x7f3c0815e700 2023-10-30T13:39:51.093 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:51.094 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7149,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:51.094 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 113 2023-10-30T13:39:51.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3bec0664a0 msgr2=0x7f3bec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:51.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3bec0664a0 0x7f3bec068960 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3c04000f70 tx=0x7f3c04007e90 comp rx=0 tx=0).stop 2023-10-30T13:39:51.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c0815e700 msgr2=0x7f3c080c0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:51.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c0815e700 0x7f3c080c0a50 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3c00007e90 tx=0x7f3c0000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:39:51.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 shutdown_connections 2023-10-30T13:39:51.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3bec0664a0 0x7f3bec068960 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:51.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c0815e700 0x7f3c080c0a50 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:51.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c081580e0 0x7f3c081f6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:51.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 --2- 172.21.15.148:0/1005008409 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c08070740 0x7f3c08161700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:51.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 >> 172.21.15.148:0/1005008409 conn(0x7f3c080c7910 msgr2=0x7f3c0814aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:51.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 shutdown_connections 2023-10-30T13:39:51.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:51.096+0000 7f3c10172640 1 -- 172.21.15.148:0/1005008409 wait complete. 2023-10-30T13:39:51.114 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:53.116 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:53.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.263+0000 7f78c3101640 1 Processor -- start 2023-10-30T13:39:53.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.263+0000 7f78c3101640 1 -- start start 2023-10-30T13:39:53.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.263+0000 7f78c3101640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f78bc0c6540 0x7f78bc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:53.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.263+0000 7f78c3101640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78bc0c5810 con 0x7f78bc070740 2023-10-30T13:39:53.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.263+0000 7f78c3101640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78bc0c69b0 con 0x7f78bc1580e0 2023-10-30T13:39:53.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.263+0000 7f78c3101640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78bc06acb0 con 0x7f78bc0c6540 2023-10-30T13:39:53.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.263+0000 7f78c169e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f78bc0c6540 0x7f78bc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:53.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.263+0000 7f78c169e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f78bc0c6540 0x7f78bc15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47860/0 (socket says 172.21.15.148:47860) 2023-10-30T13:39:53.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.263+0000 7f78c169e640 1 -- 172.21.15.148:0/3342465761 learned_addr learned my addr 172.21.15.148:0/3342465761 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:53.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78c169e640 1 -- 172.21.15.148:0/3342465761 >> v1:172.21.15.150:6789/0 conn(0x7f78bc1580e0 legacy=0x7f78bc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:53.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78c169e640 1 -- 172.21.15.148:0/3342465761 >> v1:172.21.15.148:6789/0 conn(0x7f78bc070740 legacy=0x7f78bc06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:53.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78c169e640 1 -- 172.21.15.148:0/3342465761 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc0c6bb0 con 0x7f78bc0c6540 2023-10-30T13:39:53.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78c169e640 1 --2- 172.21.15.148:0/3342465761 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 0x7f78bc15e700 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009e50 tx=0x7f78b401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43cf5e34b764aed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:53.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78b37fe640 1 -- 172.21.15.148:0/3342465761 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40a57a0 con 0x7f78bc0c6540 2023-10-30T13:39:53.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78b37fe640 1 -- 172.21.15.148:0/3342465761 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b40a5900 con 0x7f78bc0c6540 2023-10-30T13:39:53.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78b37fe640 1 -- 172.21.15.148:0/3342465761 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40a5c90 con 0x7f78bc0c6540 2023-10-30T13:39:53.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78c3101640 1 -- 172.21.15.148:0/3342465761 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 msgr2=0x7f78bc15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:53.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78c3101640 1 --2- 172.21.15.148:0/3342465761 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 0x7f78bc15e700 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009e50 tx=0x7f78b401c9d0 comp rx=0 tx=0).stop 2023-10-30T13:39:53.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78c3101640 1 -- 172.21.15.148:0/3342465761 shutdown_connections 2023-10-30T13:39:53.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78c3101640 1 --2- 172.21.15.148:0/3342465761 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 0x7f78bc15e700 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:53.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.267+0000 7f78c3101640 1 -- 172.21.15.148:0/3342465761 >> 172.21.15.148:0/3342465761 conn(0x7f78bc0c7910 msgr2=0x7f78bc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:53.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 -- 172.21.15.148:0/3342465761 shutdown_connections 2023-10-30T13:39:53.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 -- 172.21.15.148:0/3342465761 wait complete. 2023-10-30T13:39:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 Processor -- start 2023-10-30T13:39:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 -- start start 2023-10-30T13:39:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78bc070740 0x7f78bc1f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:53.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 0x7f78bc15ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78bc1580e0 0x7f78bc0c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc1f9fa0 con 0x7f78bc0c6540 2023-10-30T13:39:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc1f6fb0 con 0x7f78bc1580e0 2023-10-30T13:39:53.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c3101640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc15f380 con 0x7f78bc070740 2023-10-30T13:39:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 0x7f78bc15ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 0x7f78bc15ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47872/0 (socket says 172.21.15.148:47872) 2023-10-30T13:39:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b3fff640 1 -- 172.21.15.148:0/2671148909 learned_addr learned my addr 172.21.15.148:0/2671148909 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c169e640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78bc1580e0 0x7f78bc0c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:53.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c0e9d640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78bc070740 0x7f78bc1f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:53.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b3fff640 1 -- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78bc070740 msgr2=0x7f78bc1f6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:53.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b3fff640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78bc070740 0x7f78bc1f6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:53.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b3fff640 1 -- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78bc1580e0 msgr2=0x7f78bc0c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:53.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b3fff640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78bc1580e0 0x7f78bc0c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:53.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b3fff640 1 -- 172.21.15.148:0/2671148909 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4009b30 con 0x7f78bc0c6540 2023-10-30T13:39:53.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78c169e640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78bc1580e0 0x7f78bc0c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:53.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b3fff640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 0x7f78bc15ee40 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00c530 tx=0x7f78ac00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:53.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b17fa640 1 -- 172.21.15.148:0/2671148909 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac005220 con 0x7f78bc0c6540 2023-10-30T13:39:53.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.271+0000 7f78b17fa640 1 -- 172.21.15.148:0/2671148909 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac005380 con 0x7f78bc0c6540 2023-10-30T13:39:53.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.275+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78bc070f30 con 0x7f78bc0c6540 2023-10-30T13:39:53.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.275+0000 7f78b17fa640 1 -- 172.21.15.148:0/2671148909 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac007270 con 0x7f78bc0c6540 2023-10-30T13:39:53.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.275+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc15f8f0 con 0x7f78bc0c6540 2023-10-30T13:39:53.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.275+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78bc1fa0e0 con 0x7f78bc0c6540 2023-10-30T13:39:53.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.275+0000 7f78b17fa640 1 -- 172.21.15.148:0/2671148909 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78ac0073d0 con 0x7f78bc0c6540 2023-10-30T13:39:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.275+0000 7f78b17fa640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78a80664d0 0x7f78a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.275+0000 7f78b17fa640 1 -- 172.21.15.148:0/2671148909 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7058+0+0 (secure 0 0 0) 0x7f78ac083c70 con 0x7f78bc0c6540 2023-10-30T13:39:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.275+0000 7f78c0e9d640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78a80664d0 0x7f78a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:53.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.279+0000 7f78c0e9d640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78a80664d0 0x7f78a8068990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f78b800b810 tx=0x7f78b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:53.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.279+0000 7f78b17fa640 1 -- 172.21.15.148:0/2671148909 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78bc1fa0e0 con 0x7f78bc0c6540 2023-10-30T13:39:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78bc1fa0e0 con 0x7f78bc0c6540 2023-10-30T13:39:53.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78b17fa640 1 -- 172.21.15.148:0/2671148909 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f78bc1fa0e0 con 0x7f78bc0c6540 2023-10-30T13:39:53.459 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:53.460 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7149,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:37:57.759969+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7005":{"gid":7005,"name":"a","rank":0,"incarnation":109,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1079809213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1079809213},{"type":"v1","addr":"172.21.15.148:6835","nonce":1079809213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:53.461 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 113 2023-10-30T13:39:53.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78a80664d0 msgr2=0x7f78a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:53.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78a80664d0 0x7f78a8068990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f78b800b810 tx=0x7f78b8008040 comp rx=0 tx=0).stop 2023-10-30T13:39:53.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 msgr2=0x7f78bc15ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:53.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 0x7f78bc15ee40 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f78ac00c530 tx=0x7f78ac00c9f0 comp rx=0 tx=0).stop 2023-10-30T13:39:53.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 shutdown_connections 2023-10-30T13:39:53.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f78a80664d0 0x7f78a8068990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:53.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78bc0c6540 0x7f78bc15ee40 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:53.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78bc1580e0 0x7f78bc0c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:53.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 --2- 172.21.15.148:0/2671148909 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f78bc070740 0x7f78bc1f6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:53.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 >> 172.21.15.148:0/2671148909 conn(0x7f78bc0c7910 msgr2=0x7f78bc152730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:53.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 shutdown_connections 2023-10-30T13:39:53.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:53.459+0000 7f78c3101640 1 -- 172.21.15.148:0/2671148909 wait complete. 2023-10-30T13:39:53.481 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:39:55.483 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:55.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5f31d640 1 Processor -- start 2023-10-30T13:39:55.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5f31d640 1 -- start start 2023-10-30T13:39:55.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5f31d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7f581538a0 0x7f7f5815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5f31d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580ca0e0 0x7f7f5815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5f31d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7f580cf160 0x7f7f580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5f31d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f580cb800 con 0x7f7f581538a0 2023-10-30T13:39:55.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5f31d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5815dc30 con 0x7f7f580cf160 2023-10-30T13:39:55.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5f31d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f580cb330 con 0x7f7f580ca0e0 2023-10-30T13:39:55.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d0b9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7f581538a0 0x7f7f5815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d0b9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7f581538a0 0x7f7f5815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47888/0 (socket says 172.21.15.148:47888) 2023-10-30T13:39:55.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d0b9640 1 -- 172.21.15.148:0/538209074 learned_addr learned my addr 172.21.15.148:0/538209074 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:55.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d8ba640 1 --2- 172.21.15.148:0/538209074 >> v2:172.21.15.150:3300/0 conn(0x7f7f580cf160 0x7f7f580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5c8b8640 1 --2- 172.21.15.148:0/538209074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580ca0e0 0x7f7f5815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d0b9640 1 -- 172.21.15.148:0/538209074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580ca0e0 msgr2=0x7f7f5815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:55.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d0b9640 1 --2- 172.21.15.148:0/538209074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580ca0e0 0x7f7f5815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d0b9640 1 -- 172.21.15.148:0/538209074 >> v2:172.21.15.150:3300/0 conn(0x7f7f580cf160 msgr2=0x7f7f580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:55.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d0b9640 1 --2- 172.21.15.148:0/538209074 >> v2:172.21.15.150:3300/0 conn(0x7f7f580cf160 0x7f7f580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d0b9640 1 -- 172.21.15.148:0/538209074 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f580cad70 con 0x7f7f581538a0 2023-10-30T13:39:55.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.631+0000 7f7f5d8ba640 1 --2- 172.21.15.148:0/538209074 >> v2:172.21.15.150:3300/0 conn(0x7f7f580cf160 0x7f7f580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5d0b9640 1 --2- 172.21.15.148:0/538209074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f581538a0 0x7f7f5815d6f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7f5806cb00 tx=0x7f7f5401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c89786c185ec0fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f4ffff640 1 -- 172.21.15.148:0/538209074 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540a89b0 con 0x7f7f581538a0 2023-10-30T13:39:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f4ffff640 1 -- 172.21.15.148:0/538209074 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f540a8b10 con 0x7f7f581538a0 2023-10-30T13:39:55.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f4ffff640 1 -- 172.21.15.148:0/538209074 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540a8ea0 con 0x7f7f581538a0 2023-10-30T13:39:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5f31d640 1 -- 172.21.15.148:0/538209074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f581538a0 msgr2=0x7f7f5815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/538209074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f581538a0 0x7f7f5815d6f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7f5806cb00 tx=0x7f7f5401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5f31d640 1 -- 172.21.15.148:0/538209074 shutdown_connections 2023-10-30T13:39:55.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/538209074 >> v2:172.21.15.150:3300/0 conn(0x7f7f580cf160 0x7f7f580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/538209074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580ca0e0 0x7f7f5815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/538209074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f581538a0 0x7f7f5815d6f0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5f31d640 1 -- 172.21.15.148:0/538209074 >> 172.21.15.148:0/538209074 conn(0x7f7f5806bfe0 msgr2=0x7f7f5806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5f31d640 1 -- 172.21.15.148:0/538209074 shutdown_connections 2023-10-30T13:39:55.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.635+0000 7f7f5f31d640 1 -- 172.21.15.148:0/538209074 wait complete. 2023-10-30T13:39:55.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 Processor -- start 2023-10-30T13:39:55.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 -- start start 2023-10-30T13:39:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f580ca0e0 0x7f7f581e2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580cf160 0x7f7f5814d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f581538a0 0x7f7f581e3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f581e3b00 con 0x7f7f580ca0e0 2023-10-30T13:39:55.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5814d730 con 0x7f7f581538a0 2023-10-30T13:39:55.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f581e3540 con 0x7f7f580cf160 2023-10-30T13:39:55.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d0b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f580ca0e0 0x7f7f581e2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d0b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f580ca0e0 0x7f7f581e2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47904/0 (socket says 172.21.15.148:47904) 2023-10-30T13:39:55.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d0b9640 1 -- 172.21.15.148:0/931966353 learned_addr learned my addr 172.21.15.148:0/931966353 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:55.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d8ba640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f581538a0 0x7f7f581e3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5c8b8640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580cf160 0x7f7f5814d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d0b9640 1 -- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580cf160 msgr2=0x7f7f5814d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d0b9640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580cf160 0x7f7f5814d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d0b9640 1 -- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f581538a0 msgr2=0x7f7f581e3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d0b9640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f581538a0 0x7f7f581e3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d0b9640 1 -- 172.21.15.148:0/931966353 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f540094d0 con 0x7f7f580ca0e0 2023-10-30T13:39:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5d0b9640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f580ca0e0 0x7f7f581e2ed0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f7f5401d3a0 tx=0x7f7f5401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f4dffb640 1 -- 172.21.15.148:0/931966353 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540ba600 con 0x7f7f580ca0e0 2023-10-30T13:39:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f4dffb640 1 -- 172.21.15.148:0/931966353 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f540ba760 con 0x7f7f580ca0e0 2023-10-30T13:39:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f581e3820 con 0x7f7f580ca0e0 2023-10-30T13:39:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f4dffb640 1 -- 172.21.15.148:0/931966353 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540bab10 con 0x7f7f580ca0e0 2023-10-30T13:39:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f581f2db0 con 0x7f7f580ca0e0 2023-10-30T13:39:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.639+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f580b2a50 con 0x7f7f580ca0e0 2023-10-30T13:39:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.643+0000 7f7f4dffb640 1 -- 172.21.15.148:0/931966353 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f540bac70 con 0x7f7f580ca0e0 2023-10-30T13:39:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.643+0000 7f7f4dffb640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f3c0664a0 0x7f7f3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.643+0000 7f7f4dffb640 1 -- 172.21.15.148:0/931966353 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f7f54122a90 con 0x7f7f580ca0e0 2023-10-30T13:39:55.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.643+0000 7f7f5c8b8640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f3c0664a0 0x7f7f3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.643+0000 7f7f5c8b8640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f3c0664a0 0x7f7f3c068960 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7f580c5bb0 tx=0x7f7f48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:55.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.647+0000 7f7f4dffb640 1 -- 172.21.15.148:0/931966353 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f540ef9f0 con 0x7f7f580ca0e0 2023-10-30T13:39:55.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.827+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f581f2930 con 0x7f7f580ca0e0 2023-10-30T13:39:55.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.827+0000 7f7f4dffb640 1 -- 172.21.15.148:0/931966353 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5716 (secure 0 0 0) 0x7f7f581f2930 con 0x7f7f580ca0e0 2023-10-30T13:39:55.828 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:55.828 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:39:54.856186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:55.828 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 114 2023-10-30T13:39:55.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f3c0664a0 msgr2=0x7f7f3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:55.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f3c0664a0 0x7f7f3c068960 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7f580c5bb0 tx=0x7f7f48008040 comp rx=0 tx=0).stop 2023-10-30T13:39:55.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f580ca0e0 msgr2=0x7f7f581e2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:55.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f580ca0e0 0x7f7f581e2ed0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f7f5401d3a0 tx=0x7f7f5401ba90 comp rx=0 tx=0).stop 2023-10-30T13:39:55.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 shutdown_connections 2023-10-30T13:39:55.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f3c0664a0 0x7f7f3c068960 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f581538a0 0x7f7f581e3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f580cf160 0x7f7f5814d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 --2- 172.21.15.148:0/931966353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f580ca0e0 0x7f7f581e2ed0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 >> 172.21.15.148:0/931966353 conn(0x7f7f5806bfe0 msgr2=0x7f7f58151d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:55.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 shutdown_connections 2023-10-30T13:39:55.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.831+0000 7f7f5f31d640 1 -- 172.21.15.148:0/931966353 wait complete. 2023-10-30T13:39:55.849 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:39:55.849 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:39:55.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:55.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.987+0000 7f5b4ccd6640 1 Processor -- start 2023-10-30T13:39:55.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.987+0000 7f5b4ccd6640 1 -- start start 2023-10-30T13:39:55.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b4ccd6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b480c6b80 0x7f5b480bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b4ccd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 0x7f5b480c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b4ccd6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b480c8430 0x7f5b4813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:55.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b4ccd6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b480c0a50 con 0x7f5b480c8430 2023-10-30T13:39:55.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b4ccd6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b4806f6e0 con 0x7f5b480c6b80 2023-10-30T13:39:55.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b4ccd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4806f9c0 con 0x7f5b480c6f60 2023-10-30T13:39:55.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b46d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b480c8430 0x7f5b4813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b46d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b480c8430 0x7f5b4813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47908/0 (socket says 172.21.15.148:47908) 2023-10-30T13:39:55.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b46d9d640 1 -- 172.21.15.148:0/1808437783 learned_addr learned my addr 172.21.15.148:0/1808437783 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:55.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b3ffff640 1 --2- 172.21.15.148:0/1808437783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 0x7f5b480c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b4659c640 1 --2- 172.21.15.148:0/1808437783 >> v2:172.21.15.150:3300/0 conn(0x7f5b480c6b80 0x7f5b480bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:55.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b46d9d640 1 -- 172.21.15.148:0/1808437783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 msgr2=0x7f5b480c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:55.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b46d9d640 1 --2- 172.21.15.148:0/1808437783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 0x7f5b480c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b46d9d640 1 -- 172.21.15.148:0/1808437783 >> v2:172.21.15.150:3300/0 conn(0x7f5b480c6b80 msgr2=0x7f5b480bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:55.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b46d9d640 1 --2- 172.21.15.148:0/1808437783 >> v2:172.21.15.150:3300/0 conn(0x7f5b480c6b80 0x7f5b480bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b46d9d640 1 -- 172.21.15.148:0/1808437783 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4806fca0 con 0x7f5b480c8430 2023-10-30T13:39:55.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b46d9d640 1 --2- 172.21.15.148:0/1808437783 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c8430 0x7f5b4813ed90 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f5b4806f1b0 tx=0x7f5b4001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d74915f2a4da81 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:55.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b45d9b640 1 -- 172.21.15.148:0/1808437783 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b400a56d0 con 0x7f5b480c8430 2023-10-30T13:39:55.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b45d9b640 1 -- 172.21.15.148:0/1808437783 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b400a5830 con 0x7f5b480c8430 2023-10-30T13:39:55.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.991+0000 7f5b45d9b640 1 -- 172.21.15.148:0/1808437783 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b400a5bc0 con 0x7f5b480c8430 2023-10-30T13:39:55.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/1808437783 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c8430 msgr2=0x7f5b4813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:55.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/1808437783 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c8430 0x7f5b4813ed90 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f5b4806f1b0 tx=0x7f5b4001c940 comp rx=0 tx=0).stop 2023-10-30T13:39:55.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/1808437783 shutdown_connections 2023-10-30T13:39:55.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/1808437783 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c8430 0x7f5b4813ed90 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/1808437783 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 0x7f5b480c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/1808437783 >> v2:172.21.15.150:3300/0 conn(0x7f5b480c6b80 0x7f5b480bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:55.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/1808437783 >> 172.21.15.148:0/1808437783 conn(0x7f5b480c7910 msgr2=0x7f5b480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:55.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/1808437783 shutdown_connections 2023-10-30T13:39:55.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/1808437783 wait complete. 2023-10-30T13:39:55.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 Processor -- start 2023-10-30T13:39:55.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 -- start start 2023-10-30T13:39:56.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c6b80 0x7f5b48150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:56.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 0x7f5b481398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:56.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480c8430 0x7f5b481f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:56.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b48155d90 con 0x7f5b480c6b80 2023-10-30T13:39:56.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b48151430 con 0x7f5b480c8430 2023-10-30T13:39:56.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4ccd6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b48155430 con 0x7f5b480c6f60 2023-10-30T13:39:56.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c6b80 0x7f5b48150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:56.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c6b80 0x7f5b48150ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47914/0 (socket says 172.21.15.148:47914) 2023-10-30T13:39:56.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4659c640 1 -- 172.21.15.148:0/2394255446 learned_addr learned my addr 172.21.15.148:0/2394255446 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:56.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b3ffff640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 0x7f5b481398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:56.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b46d9d640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480c8430 0x7f5b481f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:56.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4659c640 1 -- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 msgr2=0x7f5b481398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:56.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4659c640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 0x7f5b481398b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:56.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4659c640 1 -- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480c8430 msgr2=0x7f5b481f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:56.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4659c640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480c8430 0x7f5b481f5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:56.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b4659c640 1 -- 172.21.15.148:0/2394255446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b40009b30 con 0x7f5b480c6b80 2023-10-30T13:39:56.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.995+0000 7f5b46d9d640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480c8430 0x7f5b481f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:39:56.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b4659c640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c6b80 0x7f5b48150ef0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800d800 tx=0x7f5b3800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:56.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b3effd640 1 -- 172.21.15.148:0/2394255446 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b380096a0 con 0x7f5b480c6b80 2023-10-30T13:39:56.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b3effd640 1 -- 172.21.15.148:0/2394255446 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b38009800 con 0x7f5b480c6b80 2023-10-30T13:39:56.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b481f2ef0 con 0x7f5b480c6b80 2023-10-30T13:39:56.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b3effd640 1 -- 172.21.15.148:0/2394255446 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38009b70 con 0x7f5b480c6b80 2023-10-30T13:39:56.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b481559b0 con 0x7f5b480c6b80 2023-10-30T13:39:56.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4804e170 con 0x7f5b480c6b80 2023-10-30T13:39:56.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b3effd640 1 -- 172.21.15.148:0/2394255446 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b38009cd0 con 0x7f5b480c6b80 2023-10-30T13:39:56.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b3effd640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b280664d0 0x7f5b28068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:56.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b3effd640 1 -- 172.21.15.148:0/2394255446 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f5b38084bb0 con 0x7f5b480c6b80 2023-10-30T13:39:56.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:55.999+0000 7f5b3ffff640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b280664d0 0x7f5b28068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:56.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.003+0000 7f5b3ffff640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b280664d0 0x7f5b28068990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5b48073ec0 tx=0x7f5b30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:56.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.003+0000 7f5b3effd640 1 -- 172.21.15.148:0/2394255446 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b380519f0 con 0x7f5b480c6b80 2023-10-30T13:39:56.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.187+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b480c5a60 con 0x7f5b480c6b80 2023-10-30T13:39:56.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.187+0000 7f5b3effd640 1 -- 172.21.15.148:0/2394255446 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5716 (secure 0 0 0) 0x7f5b38055010 con 0x7f5b480c6b80 2023-10-30T13:39:56.188 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:56.188 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:39:54.856186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:56.189 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 114 2023-10-30T13:39:56.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b280664d0 msgr2=0x7f5b28068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:56.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b280664d0 0x7f5b28068990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5b48073ec0 tx=0x7f5b30008040 comp rx=0 tx=0).stop 2023-10-30T13:39:56.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c6b80 msgr2=0x7f5b48150ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:56.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c6b80 0x7f5b48150ef0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800d800 tx=0x7f5b3800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:39:56.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 shutdown_connections 2023-10-30T13:39:56.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b280664d0 0x7f5b28068990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:56.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480c8430 0x7f5b481f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:56.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b480c6f60 0x7f5b481398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:56.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 --2- 172.21.15.148:0/2394255446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b480c6b80 0x7f5b48150ef0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:56.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 >> 172.21.15.148:0/2394255446 conn(0x7f5b480c7910 msgr2=0x7f5b480bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:56.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 shutdown_connections 2023-10-30T13:39:56.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:56.191+0000 7f5b4ccd6640 1 -- 172.21.15.148:0/2394255446 wait complete. 2023-10-30T13:39:56.210 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 114, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7149": { "addr": "172.21.15.148:6837/500606168", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 500606168, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 500606168, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7149, "incarnation": 114, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:39:54.856186+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7149 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:39:58.211 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:39:58.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.359+0000 7f80062c9640 1 Processor -- start 2023-10-30T13:39:58.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.359+0000 7f80062c9640 1 -- start start 2023-10-30T13:39:58.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f80062c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8000153d70 0x7f800006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:58.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f80062c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f80000c8490 0x7f80000c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:58.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f80062c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800006afa0 con 0x7f800006e470 2023-10-30T13:39:58.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f80062c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f800006d780 con 0x7f80000c8490 2023-10-30T13:39:58.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f80062c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8000154150 con 0x7f8000153d70 2023-10-30T13:39:58.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f8004866640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f80000c8490 0x7f80000c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:58.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f7ffffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8000153d70 0x7f800006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:58.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f8004866640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f80000c8490 0x7f80000c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:46906/0 (socket says 172.21.15.148:46906) 2023-10-30T13:39:58.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f8004866640 1 -- 172.21.15.148:0/3624734446 learned_addr learned my addr 172.21.15.148:0/3624734446 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:58.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f7ffffff640 1 -- 172.21.15.148:0/3624734446 >> v2:172.21.15.150:3300/0 conn(0x7f80000c8490 msgr2=0x7f80000c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:39:58.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f7ffffff640 1 --2- 172.21.15.148:0/3624734446 >> v2:172.21.15.150:3300/0 conn(0x7f80000c8490 0x7f80000c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:58.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f7ffffff640 1 -- 172.21.15.148:0/3624734446 >> v1:172.21.15.150:6789/0 conn(0x7f800006e470 legacy=0x7f80000cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:58.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.363+0000 7f7ffffff640 1 -- 172.21.15.148:0/3624734446 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800006e030 con 0x7f8000153d70 2023-10-30T13:39:58.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f7ffffff640 1 --2- 172.21.15.148:0/3624734446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8000153d70 0x7f800006ca80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00097c0 tx=0x7f7ff001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f787d8fa1150b80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:58.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f7ffeffd640 1 -- 172.21.15.148:0/3624734446 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00ab650 con 0x7f8000153d70 2023-10-30T13:39:58.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f7ffeffd640 1 -- 172.21.15.148:0/3624734446 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff00ab7b0 con 0x7f8000153d70 2023-10-30T13:39:58.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f7ffeffd640 1 -- 172.21.15.148:0/3624734446 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00abb40 con 0x7f8000153d70 2023-10-30T13:39:58.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 -- 172.21.15.148:0/3624734446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8000153d70 msgr2=0x7f800006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:58.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 --2- 172.21.15.148:0/3624734446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8000153d70 0x7f800006ca80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00097c0 tx=0x7f7ff001c450 comp rx=0 tx=0).stop 2023-10-30T13:39:58.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 -- 172.21.15.148:0/3624734446 shutdown_connections 2023-10-30T13:39:58.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 --2- 172.21.15.148:0/3624734446 >> v2:172.21.15.150:3300/0 conn(0x7f80000c8490 0x7f80000c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:58.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 --2- 172.21.15.148:0/3624734446 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8000153d70 0x7f800006ca80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:58.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 -- 172.21.15.148:0/3624734446 >> 172.21.15.148:0/3624734446 conn(0x7f8000071ea0 msgr2=0x7f8000065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:58.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 -- 172.21.15.148:0/3624734446 shutdown_connections 2023-10-30T13:39:58.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 -- 172.21.15.148:0/3624734446 wait complete. 2023-10-30T13:39:58.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 Processor -- start 2023-10-30T13:39:58.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.367+0000 7f80062c9640 1 -- start start 2023-10-30T13:39:58.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f80062c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f800006e470 0x7f8000152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:58.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f80062c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80000c8490 0x7f800015b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:58.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f80062c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8000153d70 0x7f80001f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:58.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f80062c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800015c4a0 con 0x7f800006e470 2023-10-30T13:39:58.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f80062c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800015c780 con 0x7f8000153d70 2023-10-30T13:39:58.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f80062c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800015ca60 con 0x7f80000c8490 2023-10-30T13:39:58.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f800006e470 0x7f8000152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:58.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f800006e470 0x7f8000152420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47922/0 (socket says 172.21.15.148:47922) 2023-10-30T13:39:58.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffffff640 1 -- 172.21.15.148:0/1876903734 learned_addr learned my addr 172.21.15.148:0/1876903734 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:39:58.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7fff7fe640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80000c8490 0x7f800015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:58.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f8004866640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8000153d70 0x7f80001f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:58.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffffff640 1 -- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80000c8490 msgr2=0x7f800015b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:58.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffffff640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80000c8490 0x7f800015b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:58.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffffff640 1 -- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8000153d70 msgr2=0x7f80001f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:58.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffffff640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8000153d70 0x7f80001f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:58.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffffff640 1 -- 172.21.15.148:0/1876903734 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff0009420 con 0x7f800006e470 2023-10-30T13:39:58.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffffff640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f800006e470 0x7f8000152420 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00ad040 tx=0x7f7ff001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:58.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffcff9640 1 -- 172.21.15.148:0/1876903734 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00bf610 con 0x7f800006e470 2023-10-30T13:39:58.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffcff9640 1 -- 172.21.15.148:0/1876903734 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff00bf770 con 0x7f800006e470 2023-10-30T13:39:58.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800015cd40 con 0x7f800006e470 2023-10-30T13:39:58.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffcff9640 1 -- 172.21.15.148:0/1876903734 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00bfae0 con 0x7f800006e470 2023-10-30T13:39:58.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8000152960 con 0x7f800006e470 2023-10-30T13:39:58.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800004e170 con 0x7f800006e470 2023-10-30T13:39:58.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.371+0000 7f7ffcff9640 1 -- 172.21.15.148:0/1876903734 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7ff00bfc40 con 0x7f800006e470 2023-10-30T13:39:58.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.375+0000 7f7ffcff9640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fec0664a0 0x7f7fec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:39:58.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.375+0000 7f7ffcff9640 1 -- 172.21.15.148:0/1876903734 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f7ff011f360 con 0x7f800006e470 2023-10-30T13:39:58.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.375+0000 7f7fff7fe640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fec0664a0 0x7f7fec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:39:58.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.375+0000 7f7fff7fe640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fec0664a0 0x7f7fec068960 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f800015a870 tx=0x7f7ff4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:39:58.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.379+0000 7f7ffcff9640 1 -- 172.21.15.148:0/1876903734 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ff00ec2b0 con 0x7f800006e470 2023-10-30T13:39:58.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.555+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80001f5bb0 con 0x7f800006e470 2023-10-30T13:39:58.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.555+0000 7f7ffcff9640 1 -- 172.21.15.148:0/1876903734 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5716 (secure 0 0 0) 0x7f7ff00ef8d0 con 0x7f800006e470 2023-10-30T13:39:58.557 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:39:58.557 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:39:54.856186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:39:58.557 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 114 2023-10-30T13:39:58.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fec0664a0 msgr2=0x7f7fec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:58.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fec0664a0 0x7f7fec068960 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f800015a870 tx=0x7f7ff4009040 comp rx=0 tx=0).stop 2023-10-30T13:39:58.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f800006e470 msgr2=0x7f8000152420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:39:58.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f800006e470 0x7f8000152420 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7ff00ad040 tx=0x7f7ff001ac90 comp rx=0 tx=0).stop 2023-10-30T13:39:58.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 shutdown_connections 2023-10-30T13:39:58.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7fec0664a0 0x7f7fec068960 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:58.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8000153d70 0x7f80001f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:58.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f80000c8490 0x7f800015b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:58.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 --2- 172.21.15.148:0/1876903734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f800006e470 0x7f8000152420 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:39:58.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 >> 172.21.15.148:0/1876903734 conn(0x7f8000071ea0 msgr2=0x7f8000151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:39:58.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 shutdown_connections 2023-10-30T13:39:58.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:39:58.559+0000 7f80062c9640 1 -- 172.21.15.148:0/1876903734 wait complete. 2023-10-30T13:39:58.578 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:00.579 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:00.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.735+0000 7f2259fc5640 1 Processor -- start 2023-10-30T13:40:00.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.735+0000 7f2259fc5640 1 -- start start 2023-10-30T13:40:00.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f2259fc5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f22541538a0 0x7f225415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:00.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f2259fc5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22540cb800 con 0x7f22540cf160 2023-10-30T13:40:00.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f2259fc5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225415dc30 con 0x7f22540ca0e0 2023-10-30T13:40:00.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f2259fc5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22540cb330 con 0x7f22541538a0 2023-10-30T13:40:00.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f22537fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f22541538a0 0x7f225415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:00.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f22537fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f22541538a0 0x7f225415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47936/0 (socket says 172.21.15.148:47936) 2023-10-30T13:40:00.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f22537fe640 1 -- 172.21.15.148:0/3119817274 learned_addr learned my addr 172.21.15.148:0/3119817274 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:00.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f22537fe640 1 -- 172.21.15.148:0/3119817274 >> v1:172.21.15.150:6789/0 conn(0x7f22540ca0e0 legacy=0x7f225415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:00.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f22537fe640 1 -- 172.21.15.148:0/3119817274 >> v1:172.21.15.148:6789/0 conn(0x7f22540cf160 legacy=0x7f2254164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:00.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.739+0000 7f22537fe640 1 -- 172.21.15.148:0/3119817274 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540cad70 con 0x7f22541538a0 2023-10-30T13:40:00.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f22537fe640 1 --2- 172.21.15.148:0/3119817274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22541538a0 0x7f225415d6f0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2244009870 tx=0x7f224401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ddc8cd8fdd62f32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:00.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f22527fc640 1 -- 172.21.15.148:0/3119817274 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22440a8a30 con 0x7f22541538a0 2023-10-30T13:40:00.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f22527fc640 1 -- 172.21.15.148:0/3119817274 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22440a8b90 con 0x7f22541538a0 2023-10-30T13:40:00.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f22527fc640 1 -- 172.21.15.148:0/3119817274 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22440b0440 con 0x7f22541538a0 2023-10-30T13:40:00.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f2259fc5640 1 -- 172.21.15.148:0/3119817274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22541538a0 msgr2=0x7f225415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f2259fc5640 1 --2- 172.21.15.148:0/3119817274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22541538a0 0x7f225415d6f0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f2244009870 tx=0x7f224401ed30 comp rx=0 tx=0).stop 2023-10-30T13:40:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f2259fc5640 1 -- 172.21.15.148:0/3119817274 shutdown_connections 2023-10-30T13:40:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f2259fc5640 1 --2- 172.21.15.148:0/3119817274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22541538a0 0x7f225415d6f0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f2259fc5640 1 -- 172.21.15.148:0/3119817274 >> 172.21.15.148:0/3119817274 conn(0x7f225406bfe0 msgr2=0x7f225406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:00.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f2259fc5640 1 -- 172.21.15.148:0/3119817274 shutdown_connections 2023-10-30T13:40:00.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f2259fc5640 1 -- 172.21.15.148:0/3119817274 wait complete. 2023-10-30T13:40:00.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f2259fc5640 1 Processor -- start 2023-10-30T13:40:00.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.743+0000 7f2259fc5640 1 -- start start 2023-10-30T13:40:00.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2259fc5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22540ca0e0 0x7f22541f00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:00.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2259fc5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22540cf160 0x7f2254152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:00.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2259fc5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22541538a0 0x7f2254140b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:00.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2259fc5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2254145000 con 0x7f22540ca0e0 2023-10-30T13:40:00.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2259fc5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22540cedd0 con 0x7f22541538a0 2023-10-30T13:40:00.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2259fc5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22541f0600 con 0x7f22540cf160 2023-10-30T13:40:00.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f22537fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22540ca0e0 0x7f22541f00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:00.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f22537fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22540ca0e0 0x7f22541f00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47944/0 (socket says 172.21.15.148:47944) 2023-10-30T13:40:00.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f22537fe640 1 -- 172.21.15.148:0/474943940 learned_addr learned my addr 172.21.15.148:0/474943940 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:00.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2253fff640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22541538a0 0x7f2254140b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:00.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2252ffd640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22540cf160 0x7f2254152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:00.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f22537fe640 1 -- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22540cf160 msgr2=0x7f2254152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:00.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f22537fe640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22540cf160 0x7f2254152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:00.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f22537fe640 1 -- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22541538a0 msgr2=0x7f2254140b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:00.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f22537fe640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22541538a0 0x7f2254140b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:00.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f22537fe640 1 -- 172.21.15.148:0/474943940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22440094d0 con 0x7f22540ca0e0 2023-10-30T13:40:00.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f22537fe640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22540ca0e0 0x7f22541f00c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f224401c960 tx=0x7f224401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:00.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f223bfff640 1 -- 172.21.15.148:0/474943940 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22440ba600 con 0x7f22540ca0e0 2023-10-30T13:40:00.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f223bfff640 1 -- 172.21.15.148:0/474943940 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22440ba7a0 con 0x7f22540ca0e0 2023-10-30T13:40:00.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f223bfff640 1 -- 172.21.15.148:0/474943940 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22440bab50 con 0x7f22540ca0e0 2023-10-30T13:40:00.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22540ce570 con 0x7f22540ca0e0 2023-10-30T13:40:00.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22540ceaf0 con 0x7f22540ca0e0 2023-10-30T13:40:00.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225404e170 con 0x7f22540ca0e0 2023-10-30T13:40:00.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.747+0000 7f223bfff640 1 -- 172.21.15.148:0/474943940 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22440bacb0 con 0x7f22540ca0e0 2023-10-30T13:40:00.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.751+0000 7f223bfff640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22400664a0 0x7f2240068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:00.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.751+0000 7f223bfff640 1 -- 172.21.15.148:0/474943940 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f2244122ac0 con 0x7f22540ca0e0 2023-10-30T13:40:00.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.751+0000 7f2252ffd640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22400664a0 0x7f2240068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:00.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.751+0000 7f2252ffd640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22400664a0 0x7f2240068960 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f22541572c0 tx=0x7f2248009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:00.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.759+0000 7f223bfff640 1 -- 172.21.15.148:0/474943940 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f22440ef900 con 0x7f22540ca0e0 2023-10-30T13:40:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.947+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2254159b40 con 0x7f22540ca0e0 2023-10-30T13:40:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.947+0000 7f223bfff640 1 -- 172.21.15.148:0/474943940 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5716 (secure 0 0 0) 0x7f22440f2f20 con 0x7f22540ca0e0 2023-10-30T13:40:00.950 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:00.950 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:39:54.856186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:00.950 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 114 2023-10-30T13:40:00.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22400664a0 msgr2=0x7f2240068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:00.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22400664a0 0x7f2240068960 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f22541572c0 tx=0x7f2248009040 comp rx=0 tx=0).stop 2023-10-30T13:40:00.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22540ca0e0 msgr2=0x7f22541f00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:00.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22540ca0e0 0x7f22541f00c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f224401c960 tx=0x7f224401bad0 comp rx=0 tx=0).stop 2023-10-30T13:40:00.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 shutdown_connections 2023-10-30T13:40:00.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f22400664a0 0x7f2240068960 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:00.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22541538a0 0x7f2254140b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:00.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f22540cf160 0x7f2254152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:00.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 --2- 172.21.15.148:0/474943940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f22540ca0e0 0x7f22541f00c0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:00.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 >> 172.21.15.148:0/474943940 conn(0x7f225406bfe0 msgr2=0x7f2254142e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:00.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 shutdown_connections 2023-10-30T13:40:00.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:00.951+0000 7f2259fc5640 1 -- 172.21.15.148:0/474943940 wait complete. 2023-10-30T13:40:00.971 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:01.103 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:01.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:02.973 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:03.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8e409640 1 Processor -- start 2023-10-30T13:40:03.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8e409640 1 -- start start 2023-10-30T13:40:03.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8e409640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcd881538f0 0x7fcd88159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:03.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8e409640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcd880cefa0 0x7fcd880cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:03.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8e409640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd880ca6a0 con 0x7fcd8815cd50 2023-10-30T13:40:03.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8e409640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd880cf8f0 con 0x7fcd881538f0 2023-10-30T13:40:03.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8e409640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8815d300 con 0x7fcd880cefa0 2023-10-30T13:40:03.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8d407640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcd881538f0 0x7fcd88159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:03.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8d407640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcd881538f0 0x7fcd88159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45428/0 (socket says 172.21.15.148:45428) 2023-10-30T13:40:03.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8d407640 1 -- 172.21.15.148:0/492854680 learned_addr learned my addr 172.21.15.148:0/492854680 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:03.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8dc08640 1 --2- 172.21.15.148:0/492854680 >> v2:172.21.15.150:3300/0 conn(0x7fcd880cefa0 0x7fcd880cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:03.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8d407640 1 -- 172.21.15.148:0/492854680 >> v2:172.21.15.150:3300/0 conn(0x7fcd880cefa0 msgr2=0x7fcd880cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:03.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8d407640 1 --2- 172.21.15.148:0/492854680 >> v2:172.21.15.150:3300/0 conn(0x7fcd880cefa0 0x7fcd880cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8d407640 1 -- 172.21.15.148:0/492854680 >> v1:172.21.15.148:6789/0 conn(0x7fcd8815cd50 legacy=0x7fcd880c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:03.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.127+0000 7fcd8d407640 1 -- 172.21.15.148:0/492854680 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8815d5e0 con 0x7fcd881538f0 2023-10-30T13:40:03.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd8d407640 1 --2- 172.21.15.148:0/492854680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd881538f0 0x7fcd88159fb0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fcd88066a40 tx=0x7fcd8401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9b196bf8db6107 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:03.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd7ffff640 1 -- 172.21.15.148:0/492854680 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd840a89f0 con 0x7fcd881538f0 2023-10-30T13:40:03.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd7ffff640 1 -- 172.21.15.148:0/492854680 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd840a8b50 con 0x7fcd881538f0 2023-10-30T13:40:03.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd7ffff640 1 -- 172.21.15.148:0/492854680 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd840b0440 con 0x7fcd881538f0 2023-10-30T13:40:03.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd8e409640 1 -- 172.21.15.148:0/492854680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd881538f0 msgr2=0x7fcd88159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:03.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd8e409640 1 --2- 172.21.15.148:0/492854680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd881538f0 0x7fcd88159fb0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fcd88066a40 tx=0x7fcd8401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd8e409640 1 -- 172.21.15.148:0/492854680 shutdown_connections 2023-10-30T13:40:03.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd8e409640 1 --2- 172.21.15.148:0/492854680 >> v2:172.21.15.150:3300/0 conn(0x7fcd880cefa0 0x7fcd880cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd8e409640 1 --2- 172.21.15.148:0/492854680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd881538f0 0x7fcd88159fb0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd8e409640 1 -- 172.21.15.148:0/492854680 >> 172.21.15.148:0/492854680 conn(0x7fcd8806a640 msgr2=0x7fcd88065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:03.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd8e409640 1 -- 172.21.15.148:0/492854680 shutdown_connections 2023-10-30T13:40:03.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.131+0000 7fcd8e409640 1 -- 172.21.15.148:0/492854680 wait complete. 2023-10-30T13:40:03.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 Processor -- start 2023-10-30T13:40:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 -- start start 2023-10-30T13:40:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd880cefa0 0x7fcd8815bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd881538f0 0x7fcd88146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:03.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd8815cd50 0x7fcd88141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:03.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd881488c0 con 0x7fcd8815cd50 2023-10-30T13:40:03.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd88143860 con 0x7fcd880cefa0 2023-10-30T13:40:03.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd88143430 con 0x7fcd881538f0 2023-10-30T13:40:03.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8dc08640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd8815cd50 0x7fcd88141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:03.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8dc08640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd8815cd50 0x7fcd88141a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45436/0 (socket says 172.21.15.148:45436) 2023-10-30T13:40:03.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8dc08640 1 -- 172.21.15.148:0/1022808021 learned_addr learned my addr 172.21.15.148:0/1022808021 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:03.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8d407640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd880cefa0 0x7fcd8815bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:03.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8cc06640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd881538f0 0x7fcd88146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:03.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8dc08640 1 -- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd881538f0 msgr2=0x7fcd88146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:03.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8dc08640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd881538f0 0x7fcd88146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8dc08640 1 -- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd880cefa0 msgr2=0x7fcd8815bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:03.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8dc08640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd880cefa0 0x7fcd8815bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8dc08640 1 -- 172.21.15.148:0/1022808021 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd840094d0 con 0x7fcd8815cd50 2023-10-30T13:40:03.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8dc08640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd8815cd50 0x7fcd88141a00 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fcd80007e40 tx=0x7fcd8000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:03.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd7dffb640 1 -- 172.21.15.148:0/1022808021 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd80014620 con 0x7fcd8815cd50 2023-10-30T13:40:03.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd7dffb640 1 -- 172.21.15.148:0/1022808021 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8000ee90 con 0x7fcd8815cd50 2023-10-30T13:40:03.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd881479d0 con 0x7fcd8815cd50 2023-10-30T13:40:03.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd7dffb640 1 -- 172.21.15.148:0/1022808021 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd800149e0 con 0x7fcd8815cd50 2023-10-30T13:40:03.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.135+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd88147f90 con 0x7fcd8815cd50 2023-10-30T13:40:03.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.139+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8804e170 con 0x7fcd8815cd50 2023-10-30T13:40:03.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.139+0000 7fcd7dffb640 1 -- 172.21.15.148:0/1022808021 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd80014b40 con 0x7fcd8815cd50 2023-10-30T13:40:03.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.139+0000 7fcd7dffb640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd6c0664a0 0x7fcd6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:03.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.139+0000 7fcd7dffb640 1 -- 172.21.15.148:0/1022808021 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7fcd80084e50 con 0x7fcd8815cd50 2023-10-30T13:40:03.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.139+0000 7fcd8d407640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd6c0664a0 0x7fcd6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:03.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.139+0000 7fcd8d407640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd6c0664a0 0x7fcd6c068960 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcd840b2000 tx=0x7fcd84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:03.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.143+0000 7fcd7dffb640 1 -- 172.21.15.148:0/1022808021 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd80051ca0 con 0x7fcd8815cd50 2023-10-30T13:40:03.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.335+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd88148a00 con 0x7fcd8815cd50 2023-10-30T13:40:03.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.335+0000 7fcd7dffb640 1 -- 172.21.15.148:0/1022808021 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5716 (secure 0 0 0) 0x7fcd800552c0 con 0x7fcd8815cd50 2023-10-30T13:40:03.338 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:03.338 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:39:54.856186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:03.338 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 114 2023-10-30T13:40:03.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd6c0664a0 msgr2=0x7fcd6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:03.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd6c0664a0 0x7fcd6c068960 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcd840b2000 tx=0x7fcd84009040 comp rx=0 tx=0).stop 2023-10-30T13:40:03.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd8815cd50 msgr2=0x7fcd88141a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:03.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd8815cd50 0x7fcd88141a00 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fcd80007e40 tx=0x7fcd8000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 shutdown_connections 2023-10-30T13:40:03.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcd6c0664a0 0x7fcd6c068960 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcd8815cd50 0x7fcd88141a00 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcd881538f0 0x7fcd88146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 --2- 172.21.15.148:0/1022808021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd880cefa0 0x7fcd8815bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:03.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 >> 172.21.15.148:0/1022808021 conn(0x7fcd8806a640 msgr2=0x7fcd88065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:03.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 shutdown_connections 2023-10-30T13:40:03.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:03.339+0000 7fcd8e409640 1 -- 172.21.15.148:0/1022808021 wait complete. 2023-10-30T13:40:03.360 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:05.361 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.503+0000 7f960b2c9640 1 Processor -- start 2023-10-30T13:40:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f960b2c9640 1 -- start start 2023-10-30T13:40:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f960b2c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96041538a0 0x7f960415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:05.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f960b2c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96040cb800 con 0x7f96040cf160 2023-10-30T13:40:05.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f960b2c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960415dc30 con 0x7f96040ca0e0 2023-10-30T13:40:05.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f960b2c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96040cb330 con 0x7f96041538a0 2023-10-30T13:40:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609866640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f96040cf160 0x7f9604164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36160/0 (socket says 172.21.15.148:36160) 2023-10-30T13:40:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609866640 1 -- 172.21.15.148:0/1540315014 learned_addr learned my addr 172.21.15.148:0/1540315014 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:40:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609065640 1 --2- 172.21.15.148:0/1540315014 >> v2:172.21.15.150:3300/0 conn(0x7f96041538a0 0x7f960415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452737886 0 0) 0x7f96040cb800 con 0x7f96040cf160 2023-10-30T13:40:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f0003530 con 0x7f96040cf160 2023-10-30T13:40:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380247015 0 0) 0x7f95f4001120 con 0x7f96040ca0e0 2023-10-30T13:40:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96040cb800 con 0x7f96040ca0e0 2023-10-30T13:40:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 180303254 0 0) 0x7f95f0003530 con 0x7f96040cf160 2023-10-30T13:40:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 >> v2:172.21.15.150:3300/0 conn(0x7f96041538a0 msgr2=0x7f960415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 --2- 172.21.15.148:0/1540315014 >> v2:172.21.15.150:3300/0 conn(0x7f96041538a0 0x7f960415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 >> v1:172.21.15.150:6789/0 conn(0x7f96040ca0e0 legacy=0x7f960415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96040cad70 con 0x7f96040cf160 2023-10-30T13:40:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f95fc001a00 con 0x7f96040cf160 2023-10-30T13:40:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 >> v1:172.21.15.148:6789/0 conn(0x7f96040cf160 legacy=0x7f9604164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95f0003b60 0x7f95f0003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f00047d0 0x7f95f0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f0005190 0x7f95f000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96040cb330 con 0x7f95f0003b60 2023-10-30T13:40:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f000bfb0 con 0x7f95f0005190 2023-10-30T13:40:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f000c170 con 0x7f95f00047d0 2023-10-30T13:40:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609065640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95f0003b60 0x7f95f0003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609866640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f0005190 0x7f95f000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609065640 1 -- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f00047d0 msgr2=0x7f95f0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9608864640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f00047d0 0x7f95f0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609065640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f00047d0 0x7f95f0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609065640 1 -- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f0005190 msgr2=0x7f95f000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609065640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f0005190 0x7f95f000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609065640 1 -- 172.21.15.148:0/1540315014 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f0003530 con 0x7f95f0003b60 2023-10-30T13:40:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9608864640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f00047d0 0x7f95f0004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:40:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609866640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f0005190 0x7f95f000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.507+0000 7f9609866640 1 -- 172.21.15.148:0/1540315014 reap_dead start 2023-10-30T13:40:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f9609065640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95f0003b60 0x7f95f0003f60 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f9600009ae0 tx=0x7f960001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420d1ca6cd74c5fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96000a5640 con 0x7f95f0003b60 2023-10-30T13:40:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f4001d80 con 0x7f95f0003b60 2023-10-30T13:40:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f95fbfff640 1 -- 172.21.15.148:0/1540315014 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96000a59e0 con 0x7f95f0003b60 2023-10-30T13:40:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f960b2c9640 1 -- 172.21.15.148:0/1540315014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95f0003b60 msgr2=0x7f95f0003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f960b2c9640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95f0003b60 0x7f95f0003f60 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f9600009ae0 tx=0x7f960001ca50 comp rx=0 tx=0).stop 2023-10-30T13:40:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f960b2c9640 1 -- 172.21.15.148:0/1540315014 shutdown_connections 2023-10-30T13:40:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f960b2c9640 1 --2- 172.21.15.148:0/1540315014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f95f0003b60 0x7f95f0003f60 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f960b2c9640 1 -- 172.21.15.148:0/1540315014 >> 172.21.15.148:0/1540315014 conn(0x7f960406bfe0 msgr2=0x7f960406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f960b2c9640 1 -- 172.21.15.148:0/1540315014 shutdown_connections 2023-10-30T13:40:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.511+0000 7f960b2c9640 1 -- 172.21.15.148:0/1540315014 wait complete. 2023-10-30T13:40:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.515+0000 7f960b2c9640 1 Processor -- start 2023-10-30T13:40:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.515+0000 7f960b2c9640 1 -- start start 2023-10-30T13:40:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.515+0000 7f960b2c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f0003b60 0x7f96040ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.515+0000 7f960b2c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96041509d0 0x7f96041efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.515+0000 7f960b2c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9604152910 0x7f9604140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.515+0000 7f960b2c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96041ef690 con 0x7f9604152910 2023-10-30T13:40:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.515+0000 7f960b2c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96041e58f0 con 0x7f96041509d0 2023-10-30T13:40:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.515+0000 7f960b2c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96041e5050 con 0x7f95f0003b60 2023-10-30T13:40:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609866640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9604152910 0x7f9604140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609866640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9604152910 0x7f9604140480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45442/0 (socket says 172.21.15.148:45442) 2023-10-30T13:40:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609866640 1 -- 172.21.15.148:0/4131987198 learned_addr learned my addr 172.21.15.148:0/4131987198 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609065640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f0003b60 0x7f96040ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9608864640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96041509d0 0x7f96041efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609866640 1 -- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f0003b60 msgr2=0x7f96040ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609866640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f0003b60 0x7f96040ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609866640 1 -- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96041509d0 msgr2=0x7f96041efb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609866640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96041509d0 0x7f96041efb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609866640 1 -- 172.21.15.148:0/4131987198 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96041e5330 con 0x7f9604152910 2023-10-30T13:40:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f9609866640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9604152910 0x7f9604140480 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f9604072ce0 tx=0x7f95fc007ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:05.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f95f9ffb640 1 -- 172.21.15.148:0/4131987198 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc00e9c0 con 0x7f9604152910 2023-10-30T13:40:05.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f95f9ffb640 1 -- 172.21.15.148:0/4131987198 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95fc00eb20 con 0x7f9604152910 2023-10-30T13:40:05.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f95f9ffb640 1 -- 172.21.15.148:0/4131987198 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc00ee90 con 0x7f9604152910 2023-10-30T13:40:05.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96041e5610 con 0x7f9604152910 2023-10-30T13:40:05.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96041f0060 con 0x7f9604152910 2023-10-30T13:40:05.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960404e170 con 0x7f9604152910 2023-10-30T13:40:05.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.519+0000 7f95f9ffb640 1 -- 172.21.15.148:0/4131987198 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95fc01d460 con 0x7f9604152910 2023-10-30T13:40:05.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.523+0000 7f95f9ffb640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f95e80664a0 0x7f95e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:05.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.523+0000 7f95f9ffb640 1 -- 172.21.15.148:0/4131987198 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f95fc0845f0 con 0x7f9604152910 2023-10-30T13:40:05.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.523+0000 7f9609065640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f95e80664a0 0x7f95e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:05.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.523+0000 7f9609065640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f95e80664a0 0x7f95e8068960 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f96000af000 tx=0x7f96000046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:05.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.527+0000 7f95f9ffb640 1 -- 172.21.15.148:0/4131987198 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95fc0514e0 con 0x7f9604152910 2023-10-30T13:40:05.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.711+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96040ca9b0 con 0x7f9604152910 2023-10-30T13:40:05.713 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.711+0000 7f95f9ffb640 1 -- 172.21.15.148:0/4131987198 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5716 (secure 0 0 0) 0x7f95fc054b00 con 0x7f9604152910 2023-10-30T13:40:05.713 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:05.714 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:39:54.856186+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:05.714 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 114 2023-10-30T13:40:05.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f95e80664a0 msgr2=0x7f95e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:05.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f95e80664a0 0x7f95e8068960 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f96000af000 tx=0x7f96000046d0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9604152910 msgr2=0x7f9604140480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:05.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9604152910 0x7f9604140480 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f9604072ce0 tx=0x7f95fc007ba0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 shutdown_connections 2023-10-30T13:40:05.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f95e80664a0 0x7f95e8068960 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9604152910 0x7f9604140480 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96041509d0 0x7f96041efb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 --2- 172.21.15.148:0/4131987198 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f95f0003b60 0x7f96040ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:05.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 >> 172.21.15.148:0/4131987198 conn(0x7f960406bfe0 msgr2=0x7f960406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:05.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 shutdown_connections 2023-10-30T13:40:05.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:05.715+0000 7f960b2c9640 1 -- 172.21.15.148:0/4131987198 wait complete. 2023-10-30T13:40:05.734 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:07.737 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:07.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.887+0000 7f799a371640 1 Processor -- start 2023-10-30T13:40:07.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.887+0000 7f799a371640 1 -- start start 2023-10-30T13:40:07.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.887+0000 7f799a371640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79941538f0 0x7f7994159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:07.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f799a371640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f79940cefa0 0x7f79940cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:07.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f799a371640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79940ca6a0 con 0x7f799415cd50 2023-10-30T13:40:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f799a371640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799415d300 con 0x7f79940cefa0 2023-10-30T13:40:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f799a371640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799415d5e0 con 0x7f79941538f0 2023-10-30T13:40:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f7999b70640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f79940cefa0 0x7f79940cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f7999b70640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f79940cefa0 0x7f79940cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45456/0 (socket says 172.21.15.148:45456) 2023-10-30T13:40:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f7999b70640 1 -- 172.21.15.148:0/3504390856 learned_addr learned my addr 172.21.15.148:0/3504390856 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:07.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f799936f640 1 --2- 172.21.15.148:0/3504390856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79941538f0 0x7f7994159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f7999b70640 1 -- 172.21.15.148:0/3504390856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79941538f0 msgr2=0x7f7994159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f7999b70640 1 --2- 172.21.15.148:0/3504390856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79941538f0 0x7f7994159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f7999b70640 1 -- 172.21.15.148:0/3504390856 >> v1:172.21.15.150:6789/0 conn(0x7f799415cd50 legacy=0x7f79940c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f7999b70640 1 -- 172.21.15.148:0/3504390856 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799415d8c0 con 0x7f79940cefa0 2023-10-30T13:40:07.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f799936f640 1 --2- 172.21.15.148:0/3504390856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79941538f0 0x7f7994159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:07.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f7999b70640 1 --2- 172.21.15.148:0/3504390856 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79940cefa0 0x7f79940cf3f0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f798c009e50 tx=0x7f798c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36bd209c9a5c3e2f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:07.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f798bfff640 1 -- 172.21.15.148:0/3504390856 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c0af400 con 0x7f79940cefa0 2023-10-30T13:40:07.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f798bfff640 1 -- 172.21.15.148:0/3504390856 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c0ab050 con 0x7f79940cefa0 2023-10-30T13:40:07.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.891+0000 7f798bfff640 1 -- 172.21.15.148:0/3504390856 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c0af780 con 0x7f79940cefa0 2023-10-30T13:40:07.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 -- 172.21.15.148:0/3504390856 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79940cefa0 msgr2=0x7f79940cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 --2- 172.21.15.148:0/3504390856 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79940cefa0 0x7f79940cf3f0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f798c009e50 tx=0x7f798c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:40:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 -- 172.21.15.148:0/3504390856 shutdown_connections 2023-10-30T13:40:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 --2- 172.21.15.148:0/3504390856 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79940cefa0 0x7f79940cf3f0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 --2- 172.21.15.148:0/3504390856 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79941538f0 0x7f7994159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:07.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 -- 172.21.15.148:0/3504390856 >> 172.21.15.148:0/3504390856 conn(0x7f799406a640 msgr2=0x7f7994065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:07.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 -- 172.21.15.148:0/3504390856 shutdown_connections 2023-10-30T13:40:07.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 -- 172.21.15.148:0/3504390856 wait complete. 2023-10-30T13:40:07.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 Processor -- start 2023-10-30T13:40:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 -- start start 2023-10-30T13:40:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79940cefa0 0x7f79940b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79941538f0 0x7f79941e4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:07.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f799415cd50 0x7f79941458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79940b43a0 con 0x7f79941538f0 2023-10-30T13:40:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79941e6ce0 con 0x7f79940cefa0 2023-10-30T13:40:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799a371640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79941e6740 con 0x7f799415cd50 2023-10-30T13:40:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f7998b6e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79941538f0 0x7f79941e4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f7998b6e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79941538f0 0x7f79941e4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45464/0 (socket says 172.21.15.148:45464) 2023-10-30T13:40:07.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f7998b6e640 1 -- 172.21.15.148:0/3520393828 learned_addr learned my addr 172.21.15.148:0/3520393828 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f799936f640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79940cefa0 0x7f79940b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.895+0000 7f7999b70640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f799415cd50 0x7f79941458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7998b6e640 1 -- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f799415cd50 msgr2=0x7f79941458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7998b6e640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f799415cd50 0x7f79941458a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:07.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7998b6e640 1 -- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79940cefa0 msgr2=0x7f79940b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7998b6e640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79940cefa0 0x7f79940b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7998b6e640 1 -- 172.21.15.148:0/3520393828 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c009b30 con 0x7f79941538f0 2023-10-30T13:40:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7999b70640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f799415cd50 0x7f79941458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:07.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7998b6e640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79941538f0 0x7f79941e4a30 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f798400c3c0 tx=0x7f798400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:07.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7989ffb640 1 -- 172.21.15.148:0/3520393828 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79840052d0 con 0x7f79941538f0 2023-10-30T13:40:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7989ffb640 1 -- 172.21.15.148:0/3520393828 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7984005430 con 0x7f79941538f0 2023-10-30T13:40:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7989ffb640 1 -- 172.21.15.148:0/3520393828 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7984003840 con 0x7f79941538f0 2023-10-30T13:40:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79941e6a00 con 0x7f79941538f0 2023-10-30T13:40:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79940b4e80 con 0x7f79941538f0 2023-10-30T13:40:07.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799404e170 con 0x7f79941538f0 2023-10-30T13:40:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7989ffb640 1 -- 172.21.15.148:0/3520393828 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79840039a0 con 0x7f79941538f0 2023-10-30T13:40:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7989ffb640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79780664a0 0x7f7978068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f7989ffb640 1 -- 172.21.15.148:0/3520393828 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f7984083960 con 0x7f79941538f0 2023-10-30T13:40:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.899+0000 7f799936f640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79780664a0 0x7f7978068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:07.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.903+0000 7f799936f640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79780664a0 0x7f7978068960 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f799000c900 tx=0x7f7990008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:07.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:07.903+0000 7f7989ffb640 1 -- 172.21.15.148:0/3520393828 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7984050840 con 0x7f79941538f0 2023-10-30T13:40:08.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79940b2290 con 0x7f79941538f0 2023-10-30T13:40:08.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f7989ffb640 1 -- 172.21.15.148:0/3520393828 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f7984053e60 con 0x7f79941538f0 2023-10-30T13:40:08.091 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:08.092 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:05.541365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:08.092 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 115 2023-10-30T13:40:08.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79780664a0 msgr2=0x7f7978068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:08.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79780664a0 0x7f7978068960 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f799000c900 tx=0x7f7990008040 comp rx=0 tx=0).stop 2023-10-30T13:40:08.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79941538f0 msgr2=0x7f79941e4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:08.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79941538f0 0x7f79941e4a30 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f798400c3c0 tx=0x7f798400c880 comp rx=0 tx=0).stop 2023-10-30T13:40:08.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 shutdown_connections 2023-10-30T13:40:08.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79780664a0 0x7f7978068960 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:08.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79941538f0 0x7f79941e4a30 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:08.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f799415cd50 0x7f79941458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:08.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 --2- 172.21.15.148:0/3520393828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79940cefa0 0x7f79940b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:08.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 >> 172.21.15.148:0/3520393828 conn(0x7f799406a640 msgr2=0x7f7994158f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:08.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 shutdown_connections 2023-10-30T13:40:08.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:08.091+0000 7f799a371640 1 -- 172.21.15.148:0/3520393828 wait complete. 2023-10-30T13:40:08.112 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 115, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 115, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7149": { "addr": "172.21.15.148:6837/500606168", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 500606168, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 500606168, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7149, "incarnation": 114, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:40:05.541365+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7149 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:40:10.114 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:10.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b6d09f640 1 Processor -- start 2023-10-30T13:40:10.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b6d09f640 1 -- start start 2023-10-30T13:40:10.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b6d09f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b681538f0 0x7f3b68159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:10.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b6d09f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3b680cefa0 0x7f3b680cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:10.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b6d09f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b680ca6a0 con 0x7f3b6815cd50 2023-10-30T13:40:10.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b6d09f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6815d300 con 0x7f3b680cefa0 2023-10-30T13:40:10.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b6d09f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6815d5e0 con 0x7f3b681538f0 2023-10-30T13:40:10.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b67fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b681538f0 0x7f3b68159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:10.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b6c89e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3b680cefa0 0x7f3b680cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:10.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b67fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b681538f0 0x7f3b68159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:49354/0 (socket says 172.21.15.148:49354) 2023-10-30T13:40:10.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b67fff640 1 -- 172.21.15.148:0/2924270177 learned_addr learned my addr 172.21.15.148:0/2924270177 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:10.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b67fff640 1 -- 172.21.15.148:0/2924270177 >> v2:172.21.15.150:3300/0 conn(0x7f3b680cefa0 msgr2=0x7f3b680cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:10.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b67fff640 1 --2- 172.21.15.148:0/2924270177 >> v2:172.21.15.150:3300/0 conn(0x7f3b680cefa0 0x7f3b680cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b67fff640 1 -- 172.21.15.148:0/2924270177 >> v1:172.21.15.150:6789/0 conn(0x7f3b6815cd50 legacy=0x7f3b680c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:10.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b67fff640 1 -- 172.21.15.148:0/2924270177 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6815d8c0 con 0x7f3b681538f0 2023-10-30T13:40:10.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.263+0000 7f3b6c89e640 1 --2- 172.21.15.148:0/2924270177 >> v2:172.21.15.150:3300/0 conn(0x7f3b680cefa0 0x7f3b680cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:40:10.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b67fff640 1 --2- 172.21.15.148:0/2924270177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b681538f0 0x7f3b68159fb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b5801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=962c295481ec9582 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:10.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b66ffd640 1 -- 172.21.15.148:0/2924270177 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580a8a30 con 0x7f3b681538f0 2023-10-30T13:40:10.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b66ffd640 1 -- 172.21.15.148:0/2924270177 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b580a8b90 con 0x7f3b681538f0 2023-10-30T13:40:10.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b66ffd640 1 -- 172.21.15.148:0/2924270177 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580b0440 con 0x7f3b681538f0 2023-10-30T13:40:10.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b6d09f640 1 -- 172.21.15.148:0/2924270177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b681538f0 msgr2=0x7f3b68159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:10.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b6d09f640 1 --2- 172.21.15.148:0/2924270177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b681538f0 0x7f3b68159fb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b5801ed30 comp rx=0 tx=0).stop 2023-10-30T13:40:10.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b6d09f640 1 -- 172.21.15.148:0/2924270177 shutdown_connections 2023-10-30T13:40:10.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b6d09f640 1 --2- 172.21.15.148:0/2924270177 >> v2:172.21.15.150:3300/0 conn(0x7f3b680cefa0 0x7f3b680cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b6d09f640 1 --2- 172.21.15.148:0/2924270177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b681538f0 0x7f3b68159fb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b6d09f640 1 -- 172.21.15.148:0/2924270177 >> 172.21.15.148:0/2924270177 conn(0x7f3b6806a640 msgr2=0x7f3b68065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:10.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b6d09f640 1 -- 172.21.15.148:0/2924270177 shutdown_connections 2023-10-30T13:40:10.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.267+0000 7f3b6d09f640 1 -- 172.21.15.148:0/2924270177 wait complete. 2023-10-30T13:40:10.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 Processor -- start 2023-10-30T13:40:10.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 -- start start 2023-10-30T13:40:10.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b680cefa0 0x7f3b681ef690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:10.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b681538f0 0x7f3b68147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:10.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b6815cd50 0x7f3b680b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:10.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6813d4e0 con 0x7f3b680cefa0 2023-10-30T13:40:10.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6813d750 con 0x7f3b681538f0 2023-10-30T13:40:10.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6813da30 con 0x7f3b6815cd50 2023-10-30T13:40:10.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b67fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b680cefa0 0x7f3b681ef690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:10.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b67fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b680cefa0 0x7f3b681ef690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45474/0 (socket says 172.21.15.148:45474) 2023-10-30T13:40:10.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b67fff640 1 -- 172.21.15.148:0/1500956495 learned_addr learned my addr 172.21.15.148:0/1500956495 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:10.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6c89e640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b6815cd50 0x7f3b680b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:10.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b677fe640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b681538f0 0x7f3b68147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:10.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b67fff640 1 -- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b6815cd50 msgr2=0x7f3b680b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:10.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b67fff640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b6815cd50 0x7f3b680b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b67fff640 1 -- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b681538f0 msgr2=0x7f3b68147bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:10.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b67fff640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b681538f0 0x7f3b68147bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b67fff640 1 -- 172.21.15.148:0/1500956495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b60009a70 con 0x7f3b680cefa0 2023-10-30T13:40:10.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b67fff640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b680cefa0 0x7f3b681ef690 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3b580b3040 tx=0x7f3b5801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:10.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b64ff9640 1 -- 172.21.15.148:0/1500956495 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580b0dd0 con 0x7f3b680cefa0 2023-10-30T13:40:10.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b64ff9640 1 -- 172.21.15.148:0/1500956495 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b580bb460 con 0x7f3b680cefa0 2023-10-30T13:40:10.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b580094d0 con 0x7f3b680cefa0 2023-10-30T13:40:10.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b64ff9640 1 -- 172.21.15.148:0/1500956495 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580bb810 con 0x7f3b680cefa0 2023-10-30T13:40:10.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.271+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6813dff0 con 0x7f3b680cefa0 2023-10-30T13:40:10.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.275+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6815aae0 con 0x7f3b680cefa0 2023-10-30T13:40:10.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.275+0000 7f3b64ff9640 1 -- 172.21.15.148:0/1500956495 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b580bb970 con 0x7f3b680cefa0 2023-10-30T13:40:10.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.275+0000 7f3b64ff9640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3b540664a0 0x7f3b54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:10.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.275+0000 7f3b64ff9640 1 -- 172.21.15.148:0/1500956495 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f3b58121c60 con 0x7f3b680cefa0 2023-10-30T13:40:10.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.275+0000 7f3b677fe640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3b540664a0 0x7f3b54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:10.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.275+0000 7f3b677fe640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3b540664a0 0x7f3b54068960 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3b6815aab0 tx=0x7f3b5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:10.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.279+0000 7f3b64ff9640 1 -- 172.21.15.148:0/1500956495 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b580eead0 con 0x7f3b680cefa0 2023-10-30T13:40:10.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.463+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b6804e170 con 0x7f3b680cefa0 2023-10-30T13:40:10.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.463+0000 7f3b64ff9640 1 -- 172.21.15.148:0/1500956495 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f3b580f20f0 con 0x7f3b680cefa0 2023-10-30T13:40:10.465 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:10.465 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:05.541365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:10.465 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 115 2023-10-30T13:40:10.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3b540664a0 msgr2=0x7f3b54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:10.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3b540664a0 0x7f3b54068960 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3b6815aab0 tx=0x7f3b5c009040 comp rx=0 tx=0).stop 2023-10-30T13:40:10.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b680cefa0 msgr2=0x7f3b681ef690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:10.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b680cefa0 0x7f3b681ef690 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3b580b3040 tx=0x7f3b5801bad0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 shutdown_connections 2023-10-30T13:40:10.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3b540664a0 0x7f3b54068960 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3b6815cd50 0x7f3b680b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b681538f0 0x7f3b68147bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 --2- 172.21.15.148:0/1500956495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3b680cefa0 0x7f3b681ef690 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:10.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 >> 172.21.15.148:0/1500956495 conn(0x7f3b6806a640 msgr2=0x7f3b68065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:10.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 shutdown_connections 2023-10-30T13:40:10.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:10.467+0000 7f3b6d09f640 1 -- 172.21.15.148:0/1500956495 wait complete. 2023-10-30T13:40:10.486 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:12.487 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:12.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348dcdc640 1 Processor -- start 2023-10-30T13:40:12.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348dcdc640 1 -- start start 2023-10-30T13:40:12.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348dcdc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34881538f0 0x7f3488159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:12.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348dcdc640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34880ca6a0 con 0x7f348815cd50 2023-10-30T13:40:12.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348dcdc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348815d300 con 0x7f34880cefa0 2023-10-30T13:40:12.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348dcdc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348815d5e0 con 0x7f34881538f0 2023-10-30T13:40:12.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f3487fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f348815cd50 0x7f34880c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48720/0 (socket says 172.21.15.148:48720) 2023-10-30T13:40:12.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f3487fff640 1 -- 172.21.15.148:0/450236451 learned_addr learned my addr 172.21.15.148:0/450236451 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:40:12.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348ccda640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34881538f0 0x7f3488159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:12.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823586425 0 0) 0x7f34880ca6a0 con 0x7f348815cd50 2023-10-30T13:40:12.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3478003530 con 0x7f348815cd50 2023-10-30T13:40:12.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3248369583 0 0) 0x7f3478003530 con 0x7f348815cd50 2023-10-30T13:40:12.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34881538f0 msgr2=0x7f3488159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:12.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34881538f0 0x7f3488159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 >> v1:172.21.15.150:6789/0 conn(0x7f34880cefa0 legacy=0x7f34881640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:12.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348815d8c0 con 0x7f348815cd50 2023-10-30T13:40:12.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f3474001680 con 0x7f348815cd50 2023-10-30T13:40:12.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 >> v1:172.21.15.148:6789/0 conn(0x7f348815cd50 legacy=0x7f34880c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:12.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34881538f0 0x7f34780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:12.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3478003ef0 0x7f34780047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:12.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f3478005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:12.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348815d5e0 con 0x7f3478004d10 2023-10-30T13:40:12.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347800b9f0 con 0x7f3478003ef0 2023-10-30T13:40:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347800bbb0 con 0x7f34881538f0 2023-10-30T13:40:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348d4db640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f3478005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f3487fff640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3478003ef0 0x7f34780047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348ccda640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34881538f0 0x7f34780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348d4db640 1 -- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34881538f0 msgr2=0x7f34780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:12.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348d4db640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f34881538f0 0x7f34780039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348d4db640 1 -- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3478003ef0 msgr2=0x7f34780047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:12.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348d4db640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3478003ef0 0x7f34780047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.635+0000 7f348d4db640 1 -- 172.21.15.148:0/450236451 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478003530 con 0x7f3478004d10 2023-10-30T13:40:12.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f348d4db640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f3478005190 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f3480001f60 tx=0x7f348001c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d20a0799d8f3dfc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:12.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34800a5750 con 0x7f3478004d10 2023-10-30T13:40:12.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34800a58b0 con 0x7f3478004d10 2023-10-30T13:40:12.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f34877fe640 1 -- 172.21.15.148:0/450236451 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34800a5ca0 con 0x7f3478004d10 2023-10-30T13:40:12.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f348dcdc640 1 -- 172.21.15.148:0/450236451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 msgr2=0x7f3478005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:12.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f348dcdc640 1 --2- 172.21.15.148:0/450236451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f3478005190 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f3480001f60 tx=0x7f348001c860 comp rx=0 tx=0).stop 2023-10-30T13:40:12.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f348d4db640 1 -- 172.21.15.148:0/450236451 reap_dead start 2023-10-30T13:40:12.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f348dcdc640 1 -- 172.21.15.148:0/450236451 shutdown_connections 2023-10-30T13:40:12.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f348dcdc640 1 -- 172.21.15.148:0/450236451 >> 172.21.15.148:0/450236451 conn(0x7f348806a640 msgr2=0x7f3488065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:12.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f348dcdc640 1 -- 172.21.15.148:0/450236451 shutdown_connections 2023-10-30T13:40:12.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.639+0000 7f348dcdc640 1 -- 172.21.15.148:0/450236451 wait complete. 2023-10-30T13:40:12.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 Processor -- start 2023-10-30T13:40:12.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 -- start start 2023-10-30T13:40:12.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f34880ce760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:12.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3488065c10 0x7f348815b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:12.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f348815b620 0x7f34881f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:12.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3488159790 con 0x7f3478004d10 2023-10-30T13:40:12.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348815ac50 con 0x7f348815b620 2023-10-30T13:40:12.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3488152920 con 0x7f3488065c10 2023-10-30T13:40:12.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348ccda640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f34880ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:12.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348ccda640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f34880ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35460/0 (socket says 172.21.15.148:35460) 2023-10-30T13:40:12.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348ccda640 1 -- 172.21.15.148:0/1811996635 learned_addr learned my addr 172.21.15.148:0/1811996635 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:12.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f3487fff640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3488065c10 0x7f348815b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:12.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348d4db640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f348815b620 0x7f34881f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:12.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348ccda640 1 -- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3488065c10 msgr2=0x7f348815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:12.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348ccda640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3488065c10 0x7f348815b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348ccda640 1 -- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f348815b620 msgr2=0x7f34881f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:12.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348ccda640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f348815b620 0x7f34881f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348ccda640 1 -- 172.21.15.148:0/1811996635 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34881f0180 con 0x7f3478004d10 2023-10-30T13:40:12.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348ccda640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f34880ce760 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f347c011610 tx=0x7f347c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:12.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f34857fa640 1 -- 172.21.15.148:0/1811996635 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c00f8f0 con 0x7f3478004d10 2023-10-30T13:40:12.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f34857fa640 1 -- 172.21.15.148:0/1811996635 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347c00fa50 con 0x7f3478004d10 2023-10-30T13:40:12.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348815a920 con 0x7f3478004d10 2023-10-30T13:40:12.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f34857fa640 1 -- 172.21.15.148:0/1811996635 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347c00fdc0 con 0x7f3478004d10 2023-10-30T13:40:12.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.643+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3488150ba0 con 0x7f3478004d10 2023-10-30T13:40:12.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.647+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348804e170 con 0x7f3478004d10 2023-10-30T13:40:12.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.647+0000 7f34857fa640 1 -- 172.21.15.148:0/1811996635 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f347c015460 con 0x7f3478004d10 2023-10-30T13:40:12.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.647+0000 7f34857fa640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f346c0664d0 0x7f346c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:12.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.647+0000 7f34857fa640 1 -- 172.21.15.148:0/1811996635 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f347c0848b0 con 0x7f3478004d10 2023-10-30T13:40:12.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.647+0000 7f3487fff640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f346c0664d0 0x7f346c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:12.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.647+0000 7f3487fff640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f346c0664d0 0x7f346c068990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3488066010 tx=0x7f3474009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:12.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.651+0000 7f34857fa640 1 -- 172.21.15.148:0/1811996635 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f347c051780 con 0x7f3478004d10 2023-10-30T13:40:12.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.831+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f348814d980 con 0x7f3478004d10 2023-10-30T13:40:12.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.831+0000 7f34857fa640 1 -- 172.21.15.148:0/1811996635 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5717 (secure 0 0 0) 0x7f347c054da0 con 0x7f3478004d10 2023-10-30T13:40:12.832 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:12.833 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:10.890277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:12.833 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 116 2023-10-30T13:40:12.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f346c0664d0 msgr2=0x7f346c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:12.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f346c0664d0 0x7f346c068990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3488066010 tx=0x7f3474009040 comp rx=0 tx=0).stop 2023-10-30T13:40:12.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 msgr2=0x7f34880ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:12.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f34880ce760 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f347c011610 tx=0x7f347c011ad0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 shutdown_connections 2023-10-30T13:40:12.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f346c0664d0 0x7f346c068990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f348815b620 0x7f34881f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3488065c10 0x7f348815b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 --2- 172.21.15.148:0/1811996635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3478004d10 0x7f34880ce760 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:12.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 >> 172.21.15.148:0/1811996635 conn(0x7f348806a640 msgr2=0x7f34880caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:12.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 shutdown_connections 2023-10-30T13:40:12.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:12.835+0000 7f348dcdc640 1 -- 172.21.15.148:0/1811996635 wait complete. 2023-10-30T13:40:12.853 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:14.853 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:15.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.011+0000 7f9667291640 1 Processor -- start 2023-10-30T13:40:15.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.011+0000 7f9667291640 1 -- start start 2023-10-30T13:40:15.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.011+0000 7f9667291640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f96600c6bd0 0x7f966006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:15.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f9667291640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9660071110 0x7f9660158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:15.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f9667291640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9660066cf0 con 0x7f966006f7a0 2023-10-30T13:40:15.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f9667291640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96600700c0 con 0x7f9660071110 2023-10-30T13:40:15.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f9667291640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96600c70d0 con 0x7f96600c6bd0 2023-10-30T13:40:15.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f966582e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9660071110 0x7f9660158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:15.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f966582e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9660071110 0x7f9660158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35464/0 (socket says 172.21.15.148:35464) 2023-10-30T13:40:15.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f966582e640 1 -- 172.21.15.148:0/3561262051 learned_addr learned my addr 172.21.15.148:0/3561262051 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:15.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f966482c640 1 --2- 172.21.15.148:0/3561262051 >> v2:172.21.15.150:3300/0 conn(0x7f96600c6bd0 0x7f966006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:15.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f966582e640 1 -- 172.21.15.148:0/3561262051 >> v2:172.21.15.150:3300/0 conn(0x7f96600c6bd0 msgr2=0x7f966006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:15.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f966582e640 1 --2- 172.21.15.148:0/3561262051 >> v2:172.21.15.150:3300/0 conn(0x7f96600c6bd0 0x7f966006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f966582e640 1 -- 172.21.15.148:0/3561262051 >> v1:172.21.15.150:6789/0 conn(0x7f966006f7a0 legacy=0x7f9660069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:15.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.015+0000 7f966582e640 1 -- 172.21.15.148:0/3561262051 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96600c7390 con 0x7f9660071110 2023-10-30T13:40:15.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f966582e640 1 --2- 172.21.15.148:0/3561262051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9660071110 0x7f9660158110 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f966006f1c0 tx=0x7f965801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da057322cf81a13e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:15.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9657fff640 1 -- 172.21.15.148:0/3561262051 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96580af400 con 0x7f9660071110 2023-10-30T13:40:15.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9657fff640 1 -- 172.21.15.148:0/3561262051 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96580ab050 con 0x7f9660071110 2023-10-30T13:40:15.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9657fff640 1 -- 172.21.15.148:0/3561262051 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96580af780 con 0x7f9660071110 2023-10-30T13:40:15.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 -- 172.21.15.148:0/3561262051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9660071110 msgr2=0x7f9660158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:15.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 --2- 172.21.15.148:0/3561262051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9660071110 0x7f9660158110 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f966006f1c0 tx=0x7f965801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 -- 172.21.15.148:0/3561262051 shutdown_connections 2023-10-30T13:40:15.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 --2- 172.21.15.148:0/3561262051 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9660071110 0x7f9660158110 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 --2- 172.21.15.148:0/3561262051 >> v2:172.21.15.150:3300/0 conn(0x7f96600c6bd0 0x7f966006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 -- 172.21.15.148:0/3561262051 >> 172.21.15.148:0/3561262051 conn(0x7f96600c7910 msgr2=0x7f96600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:15.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 -- 172.21.15.148:0/3561262051 shutdown_connections 2023-10-30T13:40:15.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 -- 172.21.15.148:0/3561262051 wait complete. 2023-10-30T13:40:15.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 Processor -- start 2023-10-30T13:40:15.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.019+0000 7f9667291640 1 -- start start 2023-10-30T13:40:15.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9667291640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f966006f7a0 0x7f96600c1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:15.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9667291640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9660071110 0x7f96601f2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:15.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9667291640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96600c6bd0 0x7f96601f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:15.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9667291640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9660155170 con 0x7f966006f7a0 2023-10-30T13:40:15.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9667291640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9660154e90 con 0x7f96600c6bd0 2023-10-30T13:40:15.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9667291640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96601529e0 con 0x7f9660071110 2023-10-30T13:40:15.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966502d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f966006f7a0 0x7f96600c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:15.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966502d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f966006f7a0 0x7f96600c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35474/0 (socket says 172.21.15.148:35474) 2023-10-30T13:40:15.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966502d640 1 -- 172.21.15.148:0/1032932639 learned_addr learned my addr 172.21.15.148:0/1032932639 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:15.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966482c640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9660071110 0x7f96601f2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:15.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966582e640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96600c6bd0 0x7f96601f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:15.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966502d640 1 -- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9660071110 msgr2=0x7f96601f2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:15.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966502d640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9660071110 0x7f96601f2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966502d640 1 -- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96600c6bd0 msgr2=0x7f96601f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:15.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966502d640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96600c6bd0 0x7f96601f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966502d640 1 -- 172.21.15.148:0/1032932639 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9658009b30 con 0x7f966006f7a0 2023-10-30T13:40:15.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966482c640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9660071110 0x7f96601f2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:15.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f966502d640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f966006f7a0 0x7f96600c1710 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f965c00c3c0 tx=0x7f965c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:15.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9655ffb640 1 -- 172.21.15.148:0/1032932639 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c0052d0 con 0x7f966006f7a0 2023-10-30T13:40:15.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9655ffb640 1 -- 172.21.15.148:0/1032932639 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965c005430 con 0x7f966006f7a0 2023-10-30T13:40:15.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9660152cc0 con 0x7f966006f7a0 2023-10-30T13:40:15.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9655ffb640 1 -- 172.21.15.148:0/1032932639 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c003840 con 0x7f966006f7a0 2023-10-30T13:40:15.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9660153280 con 0x7f966006f7a0 2023-10-30T13:40:15.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966004e170 con 0x7f966006f7a0 2023-10-30T13:40:15.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.023+0000 7f9655ffb640 1 -- 172.21.15.148:0/1032932639 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f965c0039a0 con 0x7f966006f7a0 2023-10-30T13:40:15.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.027+0000 7f9655ffb640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96440664d0 0x7f9644068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:15.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.027+0000 7f9655ffb640 1 -- 172.21.15.148:0/1032932639 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f965c0839a0 con 0x7f966006f7a0 2023-10-30T13:40:15.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.027+0000 7f966482c640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96440664d0 0x7f9644068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:15.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.027+0000 7f966482c640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96440664d0 0x7f9644068990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f96600c7f10 tx=0x7f9650008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:15.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.031+0000 7f9655ffb640 1 -- 172.21.15.148:0/1032932639 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f965c0507e0 con 0x7f966006f7a0 2023-10-30T13:40:15.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.211+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96600c4300 con 0x7f966006f7a0 2023-10-30T13:40:15.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.211+0000 7f9655ffb640 1 -- 172.21.15.148:0/1032932639 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5717 (secure 0 0 0) 0x7f965c053e00 con 0x7f966006f7a0 2023-10-30T13:40:15.213 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:15.213 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:10.890277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:15.214 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 116 2023-10-30T13:40:15.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96440664d0 msgr2=0x7f9644068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:15.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96440664d0 0x7f9644068990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f96600c7f10 tx=0x7f9650008040 comp rx=0 tx=0).stop 2023-10-30T13:40:15.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f966006f7a0 msgr2=0x7f96600c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:15.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f966006f7a0 0x7f96600c1710 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f965c00c3c0 tx=0x7f965c00c880 comp rx=0 tx=0).stop 2023-10-30T13:40:15.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 shutdown_connections 2023-10-30T13:40:15.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f96440664d0 0x7f9644068990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96600c6bd0 0x7f96601f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9660071110 0x7f96601f2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 --2- 172.21.15.148:0/1032932639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f966006f7a0 0x7f96600c1710 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:15.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 >> 172.21.15.148:0/1032932639 conn(0x7f96600c7910 msgr2=0x7f96600b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:15.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 shutdown_connections 2023-10-30T13:40:15.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:15.215+0000 7f9667291640 1 -- 172.21.15.148:0/1032932639 wait complete. 2023-10-30T13:40:15.235 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:17.237 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:17.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.375+0000 7f405bec9640 1 Processor -- start 2023-10-30T13:40:17.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.375+0000 7f405bec9640 1 -- start start 2023-10-30T13:40:17.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405bec9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f405406f7a0 0x7f405406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:17.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405bec9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4054071110 0x7f4054158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:17.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405bec9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4054066cf0 con 0x7f40540c6bd0 2023-10-30T13:40:17.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405bec9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40540700c0 con 0x7f4054071110 2023-10-30T13:40:17.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405bec9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40540c70d0 con 0x7f405406f7a0 2023-10-30T13:40:17.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405a466640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4054071110 0x7f4054158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:17.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405a466640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4054071110 0x7f4054158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35484/0 (socket says 172.21.15.148:35484) 2023-10-30T13:40:17.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405a466640 1 -- 172.21.15.148:0/2165437946 learned_addr learned my addr 172.21.15.148:0/2165437946 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:17.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f4059c65640 1 --2- 172.21.15.148:0/2165437946 >> v2:172.21.15.150:3300/0 conn(0x7f405406f7a0 0x7f405406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:17.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405a466640 1 -- 172.21.15.148:0/2165437946 >> v2:172.21.15.150:3300/0 conn(0x7f405406f7a0 msgr2=0x7f405406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:17.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405a466640 1 --2- 172.21.15.148:0/2165437946 >> v2:172.21.15.150:3300/0 conn(0x7f405406f7a0 0x7f405406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405a466640 1 -- 172.21.15.148:0/2165437946 >> v1:172.21.15.150:6789/0 conn(0x7f40540c6bd0 legacy=0x7f4054069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:17.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.379+0000 7f405a466640 1 -- 172.21.15.148:0/2165437946 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40540c7390 con 0x7f4054071110 2023-10-30T13:40:17.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405a466640 1 --2- 172.21.15.148:0/2165437946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 0x7f4054158110 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f405406c830 tx=0x7f404c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4096066df0a92cff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:17.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f4058c63640 1 -- 172.21.15.148:0/2165437946 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c0af400 con 0x7f4054071110 2023-10-30T13:40:17.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f4058c63640 1 -- 172.21.15.148:0/2165437946 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c0ab050 con 0x7f4054071110 2023-10-30T13:40:17.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f4058c63640 1 -- 172.21.15.148:0/2165437946 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c0af780 con 0x7f4054071110 2023-10-30T13:40:17.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 -- 172.21.15.148:0/2165437946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 msgr2=0x7f4054158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:17.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 --2- 172.21.15.148:0/2165437946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 0x7f4054158110 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f405406c830 tx=0x7f404c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 -- 172.21.15.148:0/2165437946 shutdown_connections 2023-10-30T13:40:17.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 --2- 172.21.15.148:0/2165437946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 0x7f4054158110 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 --2- 172.21.15.148:0/2165437946 >> v2:172.21.15.150:3300/0 conn(0x7f405406f7a0 0x7f405406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 -- 172.21.15.148:0/2165437946 >> 172.21.15.148:0/2165437946 conn(0x7f40540c7910 msgr2=0x7f40540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:17.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 -- 172.21.15.148:0/2165437946 shutdown_connections 2023-10-30T13:40:17.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 -- 172.21.15.148:0/2165437946 wait complete. 2023-10-30T13:40:17.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 Processor -- start 2023-10-30T13:40:17.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.383+0000 7f405bec9640 1 -- start start 2023-10-30T13:40:17.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405bec9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f405406f7a0 0x7f40541575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:17.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405bec9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 0x7f4054152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:17.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405bec9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40540c6bd0 0x7f405414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:17.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405bec9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405406c560 con 0x7f4054071110 2023-10-30T13:40:17.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405bec9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40540c20b0 con 0x7f405406f7a0 2023-10-30T13:40:17.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405bec9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40540c9340 con 0x7f40540c6bd0 2023-10-30T13:40:17.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059464640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 0x7f4054152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:17.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059464640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 0x7f4054152620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35490/0 (socket says 172.21.15.148:35490) 2023-10-30T13:40:17.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059464640 1 -- 172.21.15.148:0/3465748696 learned_addr learned my addr 172.21.15.148:0/3465748696 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:17.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059c65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f405406f7a0 0x7f40541575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:17.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405a466640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40540c6bd0 0x7f405414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:17.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059464640 1 -- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40540c6bd0 msgr2=0x7f405414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:17.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059464640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40540c6bd0 0x7f405414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059464640 1 -- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f405406f7a0 msgr2=0x7f40541575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:17.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059464640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f405406f7a0 0x7f40541575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059464640 1 -- 172.21.15.148:0/3465748696 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c009b30 con 0x7f4054071110 2023-10-30T13:40:17.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405a466640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40540c6bd0 0x7f405414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:17.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f4059464640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 0x7f4054152620 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f404400c4d0 tx=0x7f404400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:17.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f404a7fc640 1 -- 172.21.15.148:0/3465748696 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40440052d0 con 0x7f4054071110 2023-10-30T13:40:17.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f404a7fc640 1 -- 172.21.15.148:0/3465748696 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4044005430 con 0x7f4054071110 2023-10-30T13:40:17.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f404a7fc640 1 -- 172.21.15.148:0/3465748696 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4044012830 con 0x7f4054071110 2023-10-30T13:40:17.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4054154b40 con 0x7f4054071110 2023-10-30T13:40:17.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40540c9060 con 0x7f4054071110 2023-10-30T13:40:17.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4054070330 con 0x7f4054071110 2023-10-30T13:40:17.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.387+0000 7f404a7fc640 1 -- 172.21.15.148:0/3465748696 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4044012990 con 0x7f4054071110 2023-10-30T13:40:17.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.391+0000 7f404a7fc640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40380664a0 0x7f4038068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:17.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.391+0000 7f404a7fc640 1 -- 172.21.15.148:0/3465748696 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f4044083f10 con 0x7f4054071110 2023-10-30T13:40:17.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.391+0000 7f4059c65640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40380664a0 0x7f4038068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:17.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.391+0000 7f4059c65640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40380664a0 0x7f4038068960 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f40500079c0 tx=0x7f4050008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:17.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.395+0000 7f404a7fc640 1 -- 172.21.15.148:0/3465748696 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4054070330 con 0x7f4054071110 2023-10-30T13:40:17.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4054070330 con 0x7f4054071110 2023-10-30T13:40:17.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f404a7fc640 1 -- 172.21.15.148:0/3465748696 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5717 (secure 0 0 0) 0x7f4054070330 con 0x7f4054071110 2023-10-30T13:40:17.579 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:17.579 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:10.890277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:17.580 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 116 2023-10-30T13:40:17.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40380664a0 msgr2=0x7f4038068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:17.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40380664a0 0x7f4038068960 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f40500079c0 tx=0x7f4050008040 comp rx=0 tx=0).stop 2023-10-30T13:40:17.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 msgr2=0x7f4054152620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:17.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 0x7f4054152620 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f404400c4d0 tx=0x7f404400c990 comp rx=0 tx=0).stop 2023-10-30T13:40:17.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 shutdown_connections 2023-10-30T13:40:17.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40380664a0 0x7f4038068960 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4054071110 0x7f4054152620 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f40540c6bd0 0x7f405414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 --2- 172.21.15.148:0/3465748696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f405406f7a0 0x7f40541575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:17.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 >> 172.21.15.148:0/3465748696 conn(0x7f40540c7910 msgr2=0x7f40540c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:17.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 shutdown_connections 2023-10-30T13:40:17.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:17.579+0000 7f405bec9640 1 -- 172.21.15.148:0/3465748696 wait complete. 2023-10-30T13:40:17.600 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:19.600 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:19.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.747+0000 7f55b17d3640 1 Processor -- start 2023-10-30T13:40:19.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55b17d3640 1 -- start start 2023-10-30T13:40:19.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55b17d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f55ac153bf0 0x7f55ac071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:19.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55b17d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55ac0c6fe0 0x7f55ac0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:19.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55b17d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c79e0 0x7f55ac0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:19.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55b17d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ac0c6810 con 0x7f55ac153bf0 2023-10-30T13:40:19.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55b17d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ac159900 con 0x7f55ac0c6fe0 2023-10-30T13:40:19.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55b17d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0700b0 con 0x7f55ac0c79e0 2023-10-30T13:40:19.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55aaffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f55ac153bf0 0x7f55ac071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:19.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55aaffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f55ac153bf0 0x7f55ac071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35504/0 (socket says 172.21.15.148:35504) 2023-10-30T13:40:19.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55aaffd640 1 -- 172.21.15.148:0/2929162758 learned_addr learned my addr 172.21.15.148:0/2929162758 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:19.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55ab7fe640 1 --2- 172.21.15.148:0/2929162758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c79e0 0x7f55ac0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:19.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55a27fc640 1 --2- 172.21.15.148:0/2929162758 >> v2:172.21.15.150:3300/0 conn(0x7f55ac0c6fe0 0x7f55ac0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:19.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55aaffd640 1 -- 172.21.15.148:0/2929162758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c79e0 msgr2=0x7f55ac0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:19.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55aaffd640 1 --2- 172.21.15.148:0/2929162758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c79e0 0x7f55ac0cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55aaffd640 1 -- 172.21.15.148:0/2929162758 >> v2:172.21.15.150:3300/0 conn(0x7f55ac0c6fe0 msgr2=0x7f55ac0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:19.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55aaffd640 1 --2- 172.21.15.148:0/2929162758 >> v2:172.21.15.150:3300/0 conn(0x7f55ac0c6fe0 0x7f55ac0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.751+0000 7f55aaffd640 1 -- 172.21.15.148:0/2929162758 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac0c6530 con 0x7f55ac153bf0 2023-10-30T13:40:19.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55aaffd640 1 --2- 172.21.15.148:0/2929162758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 0x7f55ac071260 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0723d0 tx=0x7f559c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09b3c9f1ce83904 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:19.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55aa7fc640 1 -- 172.21.15.148:0/2929162758 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c0a89b0 con 0x7f55ac153bf0 2023-10-30T13:40:19.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55aa7fc640 1 -- 172.21.15.148:0/2929162758 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c0a8b10 con 0x7f55ac153bf0 2023-10-30T13:40:19.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55aa7fc640 1 -- 172.21.15.148:0/2929162758 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c0a8ea0 con 0x7f55ac153bf0 2023-10-30T13:40:19.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 -- 172.21.15.148:0/2929162758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 msgr2=0x7f55ac071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:19.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 --2- 172.21.15.148:0/2929162758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 0x7f55ac071260 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0723d0 tx=0x7f559c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 -- 172.21.15.148:0/2929162758 shutdown_connections 2023-10-30T13:40:19.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 --2- 172.21.15.148:0/2929162758 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c79e0 0x7f55ac0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 --2- 172.21.15.148:0/2929162758 >> v2:172.21.15.150:3300/0 conn(0x7f55ac0c6fe0 0x7f55ac0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 --2- 172.21.15.148:0/2929162758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 0x7f55ac071260 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 -- 172.21.15.148:0/2929162758 >> 172.21.15.148:0/2929162758 conn(0x7f55ac06c6e0 msgr2=0x7f55ac069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:19.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 -- 172.21.15.148:0/2929162758 shutdown_connections 2023-10-30T13:40:19.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 -- 172.21.15.148:0/2929162758 wait complete. 2023-10-30T13:40:19.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.755+0000 7f55b17d3640 1 Processor -- start 2023-10-30T13:40:19.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 -- start start 2023-10-30T13:40:19.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c6fe0 0x7f55ac152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:19.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55ac0c79e0 0x7f55ac155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:19.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 0x7f55ac155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:19.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac14ca20 con 0x7f55ac153bf0 2023-10-30T13:40:19.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0c7640 con 0x7f55ac0c79e0 2023-10-30T13:40:19.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac14bea0 con 0x7f55ac0c6fe0 2023-10-30T13:40:19.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55ab7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 0x7f55ac155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:19.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55ab7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 0x7f55ac155cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35518/0 (socket says 172.21.15.148:35518) 2023-10-30T13:40:19.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55ab7fe640 1 -- 172.21.15.148:0/448015770 learned_addr learned my addr 172.21.15.148:0/448015770 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:19.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55a27fc640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55ac0c79e0 0x7f55ac155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:19.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55aaffd640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c6fe0 0x7f55ac152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:19.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55ab7fe640 1 -- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c6fe0 msgr2=0x7f55ac152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:19.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55ab7fe640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c6fe0 0x7f55ac152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55ab7fe640 1 -- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55ac0c79e0 msgr2=0x7f55ac155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:19.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55ab7fe640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55ac0c79e0 0x7f55ac155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55ab7fe640 1 -- 172.21.15.148:0/448015770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c0094d0 con 0x7f55ac153bf0 2023-10-30T13:40:19.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55a27fc640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55ac0c79e0 0x7f55ac155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:19.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55ab7fe640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 0x7f55ac155cd0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f55a400d800 tx=0x7f55a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:19.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55a3fff640 1 -- 172.21.15.148:0/448015770 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a400b6c0 con 0x7f55ac153bf0 2023-10-30T13:40:19.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55a3fff640 1 -- 172.21.15.148:0/448015770 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a400b820 con 0x7f55ac153bf0 2023-10-30T13:40:19.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55a3fff640 1 -- 172.21.15.148:0/448015770 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a400bbd0 con 0x7f55ac153bf0 2023-10-30T13:40:19.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ac14c180 con 0x7f55ac153bf0 2023-10-30T13:40:19.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ac14c740 con 0x7f55ac153bf0 2023-10-30T13:40:19.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.759+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ac04e170 con 0x7f55ac153bf0 2023-10-30T13:40:19.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.763+0000 7f55a3fff640 1 -- 172.21.15.148:0/448015770 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f55a400bd30 con 0x7f55ac153bf0 2023-10-30T13:40:19.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.763+0000 7f55a3fff640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f558c0664d0 0x7f558c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:19.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.763+0000 7f55a3fff640 1 -- 172.21.15.148:0/448015770 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f55a4084940 con 0x7f55ac153bf0 2023-10-30T13:40:19.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.763+0000 7f55aaffd640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f558c0664d0 0x7f558c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:19.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.763+0000 7f55aaffd640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f558c0664d0 0x7f558c068990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f559c0b2000 tx=0x7f559c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:19.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.767+0000 7f55a3fff640 1 -- 172.21.15.148:0/448015770 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f55a4051820 con 0x7f55ac153bf0 2023-10-30T13:40:19.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55ac06fe90 con 0x7f55ac153bf0 2023-10-30T13:40:19.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55a3fff640 1 -- 172.21.15.148:0/448015770 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f55a4054e40 con 0x7f55ac153bf0 2023-10-30T13:40:19.948 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:19.948 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:17.012249+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:clientreplay","state_seq":36,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:19.948 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 117 2023-10-30T13:40:19.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f558c0664d0 msgr2=0x7f558c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:19.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f558c0664d0 0x7f558c068990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f559c0b2000 tx=0x7f559c008040 comp rx=0 tx=0).stop 2023-10-30T13:40:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 msgr2=0x7f55ac155cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 0x7f55ac155cd0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f55a400d800 tx=0x7f55a400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 shutdown_connections 2023-10-30T13:40:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f558c0664d0 0x7f558c068990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55ac153bf0 0x7f55ac155cd0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55ac0c79e0 0x7f55ac155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 --2- 172.21.15.148:0/448015770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f55ac0c6fe0 0x7f55ac152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 >> 172.21.15.148:0/448015770 conn(0x7f55ac06c6e0 msgr2=0x7f55ac151be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 shutdown_connections 2023-10-30T13:40:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:19.947+0000 7f55b17d3640 1 -- 172.21.15.148:0/448015770 wait complete. 2023-10-30T13:40:19.969 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 117, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 117, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7149": { "addr": "172.21.15.148:6837/500606168", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 500606168, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 500606168, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7149, "incarnation": 114, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:40:17.012249+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7149 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:40:21.970 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:22.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6077a79640 1 Processor -- start 2023-10-30T13:40:22.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6077a79640 1 -- start start 2023-10-30T13:40:22.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6077a79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 0x7f607006d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:22.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6077a79640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f60700c2570 0x7f60700cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:22.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6077a79640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f60700c29d0 0x7f60700cb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:22.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6077a79640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60700c68b0 con 0x7f60700c2570 2023-10-30T13:40:22.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6077a79640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6070150d00 con 0x7f60700c29d0 2023-10-30T13:40:22.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6077a79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607006d960 con 0x7f6070150920 2023-10-30T13:40:22.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6075014640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f60700c2570 0x7f60700cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:22.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6075014640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f60700c2570 0x7f60700cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45858/0 (socket says 172.21.15.148:45858) 2023-10-30T13:40:22.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6075014640 1 -- 172.21.15.148:0/1662594886 learned_addr learned my addr 172.21.15.148:0/1662594886 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:22.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6076016640 1 --2- 172.21.15.148:0/1662594886 >> v2:172.21.15.150:3300/0 conn(0x7f60700c29d0 0x7f60700cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:22.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.119+0000 7f6075815640 1 --2- 172.21.15.148:0/1662594886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 0x7f607006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:22.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6075014640 1 -- 172.21.15.148:0/1662594886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 msgr2=0x7f607006d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:22.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6075014640 1 --2- 172.21.15.148:0/1662594886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 0x7f607006d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6075014640 1 -- 172.21.15.148:0/1662594886 >> v2:172.21.15.150:3300/0 conn(0x7f60700c29d0 msgr2=0x7f60700cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:22.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6075014640 1 --2- 172.21.15.148:0/1662594886 >> v2:172.21.15.150:3300/0 conn(0x7f60700c29d0 0x7f60700cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6075014640 1 -- 172.21.15.148:0/1662594886 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60700c2e90 con 0x7f60700c2570 2023-10-30T13:40:22.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6076016640 1 --2- 172.21.15.148:0/1662594886 >> v2:172.21.15.150:3300/0 conn(0x7f60700c29d0 0x7f60700cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:22.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6075014640 1 --2- 172.21.15.148:0/1662594886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 0x7f60700cb0d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f60700cbca0 tx=0x7f606001e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7715e4f4207ecced server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:22.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6074813640 1 -- 172.21.15.148:0/1662594886 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60600ab400 con 0x7f60700c2570 2023-10-30T13:40:22.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6074813640 1 -- 172.21.15.148:0/1662594886 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60600ab560 con 0x7f60700c2570 2023-10-30T13:40:22.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6074813640 1 -- 172.21.15.148:0/1662594886 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60600ab8f0 con 0x7f60700c2570 2023-10-30T13:40:22.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6077a79640 1 -- 172.21.15.148:0/1662594886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 msgr2=0x7f60700cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:22.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6077a79640 1 --2- 172.21.15.148:0/1662594886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 0x7f60700cb0d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f60700cbca0 tx=0x7f606001e680 comp rx=0 tx=0).stop 2023-10-30T13:40:22.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6077a79640 1 -- 172.21.15.148:0/1662594886 shutdown_connections 2023-10-30T13:40:22.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6077a79640 1 --2- 172.21.15.148:0/1662594886 >> v2:172.21.15.150:3300/0 conn(0x7f60700c29d0 0x7f60700cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6077a79640 1 --2- 172.21.15.148:0/1662594886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 0x7f60700cb0d0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6077a79640 1 --2- 172.21.15.148:0/1662594886 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 0x7f607006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.123+0000 7f6077a79640 1 -- 172.21.15.148:0/1662594886 >> 172.21.15.148:0/1662594886 conn(0x7f607006dfd0 msgr2=0x7f607006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:22.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 -- 172.21.15.148:0/1662594886 shutdown_connections 2023-10-30T13:40:22.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 -- 172.21.15.148:0/1662594886 wait complete. 2023-10-30T13:40:22.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 Processor -- start 2023-10-30T13:40:22.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 -- start start 2023-10-30T13:40:22.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 0x7f6070154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:22.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700c29d0 0x7f6070155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:22.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 0x7f6070155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:22.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6070153f10 con 0x7f60700c2570 2023-10-30T13:40:22.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607006ba40 con 0x7f60700c29d0 2023-10-30T13:40:22.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607014ca70 con 0x7f6070150920 2023-10-30T13:40:22.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075815640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 0x7f6070154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:22.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075815640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 0x7f6070154420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45872/0 (socket says 172.21.15.148:45872) 2023-10-30T13:40:22.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075815640 1 -- 172.21.15.148:0/1754608356 learned_addr learned my addr 172.21.15.148:0/1754608356 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:22.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6076016640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 0x7f6070155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:22.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075014640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700c29d0 0x7f6070155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:22.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075815640 1 -- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 msgr2=0x7f6070155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:22.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075815640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 0x7f6070155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075815640 1 -- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700c29d0 msgr2=0x7f6070155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:22.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075815640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700c29d0 0x7f6070155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075815640 1 -- 172.21.15.148:0/1754608356 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60600094d0 con 0x7f60700c2570 2023-10-30T13:40:22.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075014640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700c29d0 0x7f6070155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:22.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6075815640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 0x7f6070154420 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f606c00e870 tx=0x7f606c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:22.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f60667fc640 1 -- 172.21.15.148:0/1754608356 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c009510 con 0x7f60700c2570 2023-10-30T13:40:22.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f60667fc640 1 -- 172.21.15.148:0/1754608356 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606c00eec0 con 0x7f60700c2570 2023-10-30T13:40:22.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f60667fc640 1 -- 172.21.15.148:0/1754608356 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c00cb40 con 0x7f60700c2570 2023-10-30T13:40:22.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.127+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607006bce0 con 0x7f60700c2570 2023-10-30T13:40:22.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.131+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6070154e00 con 0x7f60700c2570 2023-10-30T13:40:22.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.131+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607004e170 con 0x7f60700c2570 2023-10-30T13:40:22.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.131+0000 7f60667fc640 1 -- 172.21.15.148:0/1754608356 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f606c00cca0 con 0x7f60700c2570 2023-10-30T13:40:22.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.131+0000 7f60667fc640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60540664a0 0x7f6054068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:22.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.131+0000 7f60667fc640 1 -- 172.21.15.148:0/1754608356 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f606c084680 con 0x7f60700c2570 2023-10-30T13:40:22.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.131+0000 7f6075014640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60540664a0 0x7f6054068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:22.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.135+0000 7f6075014640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60540664a0 0x7f6054068960 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f60600b0000 tx=0x7f60600b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:22.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.135+0000 7f60667fc640 1 -- 172.21.15.148:0/1754608356 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f606c051430 con 0x7f60700c2570 2023-10-30T13:40:22.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6070073e10 con 0x7f60700c2570 2023-10-30T13:40:22.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f60667fc640 1 -- 172.21.15.148:0/1754608356 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f606c054a50 con 0x7f60700c2570 2023-10-30T13:40:22.319 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:22.320 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:17.012249+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:clientreplay","state_seq":36,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:22.320 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 117 2023-10-30T13:40:22.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60540664a0 msgr2=0x7f6054068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:22.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60540664a0 0x7f6054068960 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f60600b0000 tx=0x7f60600b5040 comp rx=0 tx=0).stop 2023-10-30T13:40:22.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 msgr2=0x7f6070154420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:22.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 0x7f6070154420 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f606c00e870 tx=0x7f606c00ec70 comp rx=0 tx=0).stop 2023-10-30T13:40:22.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 shutdown_connections 2023-10-30T13:40:22.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f60540664a0 0x7f6054068960 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6070150920 0x7f6070155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700c29d0 0x7f6070155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 --2- 172.21.15.148:0/1754608356 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60700c2570 0x7f6070154420 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:22.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 >> 172.21.15.148:0/1754608356 conn(0x7f607006dfd0 msgr2=0x7f6070070650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:22.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 shutdown_connections 2023-10-30T13:40:22.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:22.319+0000 7f6077a79640 1 -- 172.21.15.148:0/1754608356 wait complete. 2023-10-30T13:40:22.340 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:24.341 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:24.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f701aa640 1 Processor -- start 2023-10-30T13:40:24.480 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f701aa640 1 -- start start 2023-10-30T13:40:24.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f701aa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c6b80 0x7f0f680bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:24.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f701aa640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0f680c6f60 0x7f0f680c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:24.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f701aa640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f680c0a50 con 0x7f0f680c8430 2023-10-30T13:40:24.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f701aa640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6806f6e0 con 0x7f0f680c6f60 2023-10-30T13:40:24.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f701aa640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6806f9c0 con 0x7f0f680c6b80 2023-10-30T13:40:24.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f6d745640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0f680c6f60 0x7f0f680c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:24.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f6df46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c6b80 0x7f0f680bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:24.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f6df46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c6b80 0x7f0f680bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:33056/0 (socket says 172.21.15.148:33056) 2023-10-30T13:40:24.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f6df46640 1 -- 172.21.15.148:0/276147041 learned_addr learned my addr 172.21.15.148:0/276147041 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:24.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f6df46640 1 -- 172.21.15.148:0/276147041 >> v2:172.21.15.150:3300/0 conn(0x7f0f680c6f60 msgr2=0x7f0f680c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:24.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f6df46640 1 --2- 172.21.15.148:0/276147041 >> v2:172.21.15.150:3300/0 conn(0x7f0f680c6f60 0x7f0f680c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:24.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f6df46640 1 -- 172.21.15.148:0/276147041 >> v1:172.21.15.150:6789/0 conn(0x7f0f680c8430 legacy=0x7f0f6813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:24.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.479+0000 7f0f6df46640 1 -- 172.21.15.148:0/276147041 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6806fca0 con 0x7f0f680c6b80 2023-10-30T13:40:24.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f6df46640 1 --2- 172.21.15.148:0/276147041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c6b80 0x7f0f680bfec0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009870 tx=0x7f0f6401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c61ae09a04b71636 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:24.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f6cf44640 1 -- 172.21.15.148:0/276147041 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f640ab650 con 0x7f0f680c6b80 2023-10-30T13:40:24.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f6cf44640 1 -- 172.21.15.148:0/276147041 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f640ab7b0 con 0x7f0f680c6b80 2023-10-30T13:40:24.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f6cf44640 1 -- 172.21.15.148:0/276147041 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f640abb40 con 0x7f0f680c6b80 2023-10-30T13:40:24.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f701aa640 1 -- 172.21.15.148:0/276147041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c6b80 msgr2=0x7f0f680bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:24.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f701aa640 1 --2- 172.21.15.148:0/276147041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c6b80 0x7f0f680bfec0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009870 tx=0x7f0f6401c540 comp rx=0 tx=0).stop 2023-10-30T13:40:24.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f701aa640 1 -- 172.21.15.148:0/276147041 shutdown_connections 2023-10-30T13:40:24.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f701aa640 1 --2- 172.21.15.148:0/276147041 >> v2:172.21.15.150:3300/0 conn(0x7f0f680c6f60 0x7f0f680c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:24.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f701aa640 1 --2- 172.21.15.148:0/276147041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c6b80 0x7f0f680bfec0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:24.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f701aa640 1 -- 172.21.15.148:0/276147041 >> 172.21.15.148:0/276147041 conn(0x7f0f680c7910 msgr2=0x7f0f680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:24.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f701aa640 1 -- 172.21.15.148:0/276147041 shutdown_connections 2023-10-30T13:40:24.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.483+0000 7f0f701aa640 1 -- 172.21.15.148:0/276147041 wait complete. 2023-10-30T13:40:24.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f701aa640 1 Processor -- start 2023-10-30T13:40:24.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f701aa640 1 -- start start 2023-10-30T13:40:24.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f701aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0f680c6b80 0x7f0f6813a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:24.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f701aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f680c6f60 0x7f0f68132d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:24.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f701aa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c8430 0x7f0f681589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:24.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f701aa640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6813d860 con 0x7f0f680c6b80 2023-10-30T13:40:24.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f701aa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6813d5c0 con 0x7f0f680c6f60 2023-10-30T13:40:24.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f701aa640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f681f2050 con 0x7f0f680c8430 2023-10-30T13:40:24.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6df46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0f680c6b80 0x7f0f6813a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:24.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6df46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0f680c6b80 0x7f0f6813a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45882/0 (socket says 172.21.15.148:45882) 2023-10-30T13:40:24.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6df46640 1 -- 172.21.15.148:0/2860355519 learned_addr learned my addr 172.21.15.148:0/2860355519 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:24.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6e747640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c8430 0x7f0f681589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:24.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6d745640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f680c6f60 0x7f0f68132d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:24.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6df46640 1 -- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c8430 msgr2=0x7f0f681589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:24.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6df46640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c8430 0x7f0f681589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:24.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6df46640 1 -- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f680c6f60 msgr2=0x7f0f68132d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:24.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6df46640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f680c6f60 0x7f0f68132d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:24.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6df46640 1 -- 172.21.15.148:0/2860355519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640094d0 con 0x7f0f680c6b80 2023-10-30T13:40:24.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f6df46640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0f680c6b80 0x7f0f6813a9a0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f0f640ad040 tx=0x7f0f6401ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:24.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f5e7fc640 1 -- 172.21.15.148:0/2860355519 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f640b4800 con 0x7f0f680c6b80 2023-10-30T13:40:24.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f5e7fc640 1 -- 172.21.15.148:0/2860355519 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f640b4960 con 0x7f0f680c6b80 2023-10-30T13:40:24.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f681f1290 con 0x7f0f680c6b80 2023-10-30T13:40:24.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.487+0000 7f0f5e7fc640 1 -- 172.21.15.148:0/2860355519 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f640b4d10 con 0x7f0f680c6b80 2023-10-30T13:40:24.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.491+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f681f1850 con 0x7f0f680c6b80 2023-10-30T13:40:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.491+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6804e170 con 0x7f0f680c6b80 2023-10-30T13:40:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.491+0000 7f0f5e7fc640 1 -- 172.21.15.148:0/2860355519 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f640aa430 con 0x7f0f680c6b80 2023-10-30T13:40:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.491+0000 7f0f5e7fc640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f4c0664a0 0x7f0f4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.491+0000 7f0f5e7fc640 1 -- 172.21.15.148:0/2860355519 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f0f6411f160 con 0x7f0f680c6b80 2023-10-30T13:40:24.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.491+0000 7f0f6d745640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f4c0664a0 0x7f0f4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:24.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.491+0000 7f0f6d745640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f4c0664a0 0x7f0f4c068960 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0f681540b0 tx=0x7f0f58009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:24.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.495+0000 7f0f5e7fc640 1 -- 172.21.15.148:0/2860355519 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f640ec050 con 0x7f0f680c6b80 2023-10-30T13:40:24.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.675+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f680c7e00 con 0x7f0f680c6b80 2023-10-30T13:40:24.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.675+0000 7f0f5e7fc640 1 -- 172.21.15.148:0/2860355519 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f0f640ef670 con 0x7f0f680c6b80 2023-10-30T13:40:24.676 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:24.676 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:17.012249+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:clientreplay","state_seq":36,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:24.677 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 117 2023-10-30T13:40:24.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f4c0664a0 msgr2=0x7f0f4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:24.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f4c0664a0 0x7f0f4c068960 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0f681540b0 tx=0x7f0f58009190 comp rx=0 tx=0).stop 2023-10-30T13:40:24.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0f680c6b80 msgr2=0x7f0f6813a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:24.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0f680c6b80 0x7f0f6813a9a0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f0f640ad040 tx=0x7f0f6401ad80 comp rx=0 tx=0).stop 2023-10-30T13:40:24.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 shutdown_connections 2023-10-30T13:40:24.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f4c0664a0 0x7f0f4c068960 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:24.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0f680c8430 0x7f0f681589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:24.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f680c6f60 0x7f0f68132d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:24.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 --2- 172.21.15.148:0/2860355519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0f680c6b80 0x7f0f6813a9a0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:24.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 >> 172.21.15.148:0/2860355519 conn(0x7f0f680c7910 msgr2=0x7f0f680bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:24.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 shutdown_connections 2023-10-30T13:40:24.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:24.679+0000 7f0f701aa640 1 -- 172.21.15.148:0/2860355519 wait complete. 2023-10-30T13:40:24.697 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:40:26.699 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:26.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f302a10e640 1 Processor -- start 2023-10-30T13:40:26.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f302a10e640 1 -- start start 2023-10-30T13:40:26.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f302a10e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f30240c6b80 0x7f30240bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:26.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f302a10e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30240c0a50 con 0x7f302413f4a0 2023-10-30T13:40:26.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f302a10e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302406f6e0 con 0x7f30240c6f60 2023-10-30T13:40:26.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f302a10e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302406f9c0 con 0x7f30240c6b80 2023-10-30T13:40:26.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f3023fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f302413f4a0 0x7f30241566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52756/0 (socket says 172.21.15.148:52756) 2023-10-30T13:40:26.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f3023fff640 1 -- 172.21.15.148:0/2852361968 learned_addr learned my addr 172.21.15.148:0/2852361968 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:40:26.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f30237fe640 1 --2- 172.21.15.148:0/2852361968 >> v2:172.21.15.150:3300/0 conn(0x7f30240c6b80 0x7f30240bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:26.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520489795 0 0) 0x7f30240c0a50 con 0x7f302413f4a0 2023-10-30T13:40:26.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.847+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c003530 con 0x7f302413f4a0 2023-10-30T13:40:26.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3761734043 0 0) 0x7f300c003530 con 0x7f302413f4a0 2023-10-30T13:40:26.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 >> v2:172.21.15.150:3300/0 conn(0x7f30240c6b80 msgr2=0x7f30240bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:26.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 --2- 172.21.15.148:0/2852361968 >> v2:172.21.15.150:3300/0 conn(0x7f30240c6b80 0x7f30240bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:26.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 >> v1:172.21.15.150:6789/0 conn(0x7f30240c6f60 legacy=0x7f302413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:26.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302406fca0 con 0x7f302413f4a0 2023-10-30T13:40:26.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f301c001a00 con 0x7f302413f4a0 2023-10-30T13:40:26.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 >> v1:172.21.15.148:6789/0 conn(0x7f302413f4a0 legacy=0x7f30241566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:26.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c003980 0x7f300c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:26.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300c0047d0 0x7f300c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:26.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f300c005190 0x7f300c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:26.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302406f9c0 con 0x7f300c005190 2023-10-30T13:40:26.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c00c030 con 0x7f300c0047d0 2023-10-30T13:40:26.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300c00c1f0 con 0x7f300c003980 2023-10-30T13:40:26.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f3023fff640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f300c005190 0x7f300c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:26.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f3022ffd640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300c0047d0 0x7f300c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:26.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f30237fe640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c003980 0x7f300c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:26.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f3023fff640 1 -- 172.21.15.148:0/2852361968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c003980 msgr2=0x7f300c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:26.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f3023fff640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c003980 0x7f300c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:26.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f3023fff640 1 -- 172.21.15.148:0/2852361968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300c0047d0 msgr2=0x7f300c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:26.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f3023fff640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300c0047d0 0x7f300c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:26.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f3023fff640 1 -- 172.21.15.148:0/2852361968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c003530 con 0x7f300c005190 2023-10-30T13:40:26.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.851+0000 7f3023fff640 1 -- 172.21.15.148:0/2852361968 reap_dead start 2023-10-30T13:40:26.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f3023fff640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f300c005190 0x7f300c00b990 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f301c00a410 tx=0x7f301c01f550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834b9519a1277817 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:26.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c0072d0 con 0x7f300c005190 2023-10-30T13:40:26.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c007470 con 0x7f300c005190 2023-10-30T13:40:26.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f30227fc640 1 -- 172.21.15.148:0/2852361968 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c007820 con 0x7f300c005190 2023-10-30T13:40:26.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 -- 172.21.15.148:0/2852361968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f300c005190 msgr2=0x7f300c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:26.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f300c005190 0x7f300c00b990 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f301c00a410 tx=0x7f301c01f550 comp rx=0 tx=0).stop 2023-10-30T13:40:26.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 -- 172.21.15.148:0/2852361968 shutdown_connections 2023-10-30T13:40:26.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 --2- 172.21.15.148:0/2852361968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f300c005190 0x7f300c00b990 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:26.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 -- 172.21.15.148:0/2852361968 >> 172.21.15.148:0/2852361968 conn(0x7f30240c7910 msgr2=0x7f30240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:26.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 -- 172.21.15.148:0/2852361968 shutdown_connections 2023-10-30T13:40:26.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 -- 172.21.15.148:0/2852361968 wait complete. 2023-10-30T13:40:26.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 Processor -- start 2023-10-30T13:40:26.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 -- start start 2023-10-30T13:40:26.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c005190 0x7f302413bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:26.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30240705c0 0x7f3024138590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:26.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3024139310 0x7f3024138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:26.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302413ad50 con 0x7f3024139310 2023-10-30T13:40:26.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3024139010 con 0x7f30240705c0 2023-10-30T13:40:26.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.855+0000 7f302a10e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302413a800 con 0x7f300c005190 2023-10-30T13:40:26.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3023fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3024139310 0x7f3024138ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:26.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3023fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3024139310 0x7f3024138ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45904/0 (socket says 172.21.15.148:45904) 2023-10-30T13:40:26.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3023fff640 1 -- 172.21.15.148:0/854035737 learned_addr learned my addr 172.21.15.148:0/854035737 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:26.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3022ffd640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30240705c0 0x7f3024138590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:26.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f30237fe640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c005190 0x7f302413bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:26.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3023fff640 1 -- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c005190 msgr2=0x7f302413bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:26.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3023fff640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c005190 0x7f302413bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:26.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3023fff640 1 -- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30240705c0 msgr2=0x7f3024138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:26.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3023fff640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30240705c0 0x7f3024138590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:26.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3023fff640 1 -- 172.21.15.148:0/854035737 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302413aa70 con 0x7f3024139310 2023-10-30T13:40:26.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f30237fe640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c005190 0x7f302413bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:26.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f3023fff640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3024139310 0x7f3024138ad0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f301c00a800 tx=0x7f300c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:26.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f300bfff640 1 -- 172.21.15.148:0/854035737 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c004d70 con 0x7f3024139310 2023-10-30T13:40:26.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f300bfff640 1 -- 172.21.15.148:0/854035737 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c00f1d0 con 0x7f3024139310 2023-10-30T13:40:26.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f300bfff640 1 -- 172.21.15.148:0/854035737 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c0077e0 con 0x7f3024139310 2023-10-30T13:40:26.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302413b050 con 0x7f3024139310 2023-10-30T13:40:26.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302413b610 con 0x7f3024139310 2023-10-30T13:40:26.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302413aeb0 con 0x7f3024139310 2023-10-30T13:40:26.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f300bfff640 1 -- 172.21.15.148:0/854035737 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f301c007940 con 0x7f3024139310 2023-10-30T13:40:26.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.859+0000 7f300bfff640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30100664d0 0x7f3010068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:26.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.863+0000 7f300bfff640 1 -- 172.21.15.148:0/854035737 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f301c0aa070 con 0x7f3024139310 2023-10-30T13:40:26.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.863+0000 7f30237fe640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30100664d0 0x7f3010068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:26.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.863+0000 7f30237fe640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30100664d0 0x7f3010068990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3014009ac0 tx=0x7f3014008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:26.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:26.867+0000 7f300bfff640 1 -- 172.21.15.148:0/854035737 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f302413aeb0 con 0x7f3024139310 2023-10-30T13:40:27.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.047+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f302413aeb0 con 0x7f3024139310 2023-10-30T13:40:27.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.047+0000 7f300bfff640 1 -- 172.21.15.148:0/854035737 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5717 (secure 0 0 0) 0x7f302413aeb0 con 0x7f3024139310 2023-10-30T13:40:27.050 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:27.050 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:27.050 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 118 2023-10-30T13:40:27.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30100664d0 msgr2=0x7f3010068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:27.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30100664d0 0x7f3010068990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3014009ac0 tx=0x7f3014008040 comp rx=0 tx=0).stop 2023-10-30T13:40:27.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3024139310 msgr2=0x7f3024138ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:27.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3024139310 0x7f3024138ad0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f301c00a800 tx=0x7f300c005800 comp rx=0 tx=0).stop 2023-10-30T13:40:27.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 shutdown_connections 2023-10-30T13:40:27.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30100664d0 0x7f3010068990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:27.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3024139310 0x7f3024138ad0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:27.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30240705c0 0x7f3024138590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:27.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 --2- 172.21.15.148:0/854035737 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f300c005190 0x7f302413bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:27.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 >> 172.21.15.148:0/854035737 conn(0x7f30240c7910 msgr2=0x7f302406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:27.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 shutdown_connections 2023-10-30T13:40:27.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:27.051+0000 7f302a10e640 1 -- 172.21.15.148:0/854035737 wait complete. 2023-10-30T13:40:27.071 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=7005 2023-10-30T13:40:27.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2023-10-30T13:40:31.160 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:31.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:40:32.072 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:40:32.073 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:40:32.074 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:40:32.077 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:40:32.078 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:32.178 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:40:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.258+0000 7f0fafac9640 1 Processor -- start 2023-10-30T13:40:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.258+0000 7f0fafac9640 1 -- start start 2023-10-30T13:40:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fafac9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0fa81538a0 0x7f0fa815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fafac9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0fa80ca0e0 0x7f0fa815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fafac9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa80cb800 con 0x7f0fa80cf160 2023-10-30T13:40:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fafac9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa815dc30 con 0x7f0fa81538a0 2023-10-30T13:40:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fafac9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa80cb330 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fad865640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0fa81538a0 0x7f0fa815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fad865640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0fa81538a0 0x7f0fa815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59800/0 (socket says 172.21.15.148:59800) 2023-10-30T13:40:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fad865640 1 -- 172.21.15.148:0/3055309975 learned_addr learned my addr 172.21.15.148:0/3055309975 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fad064640 1 --2- 172.21.15.148:0/3055309975 >> v2:172.21.15.150:3300/0 conn(0x7f0fa80ca0e0 0x7f0fa815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fad865640 1 -- 172.21.15.148:0/3055309975 >> v2:172.21.15.150:3300/0 conn(0x7f0fa80ca0e0 msgr2=0x7f0fa815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fad865640 1 --2- 172.21.15.148:0/3055309975 >> v2:172.21.15.150:3300/0 conn(0x7f0fa80ca0e0 0x7f0fa815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fad865640 1 -- 172.21.15.148:0/3055309975 >> v1:172.21.15.150:6789/0 conn(0x7f0fa80cf160 legacy=0x7f0fa80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.262+0000 7f0fad865640 1 -- 172.21.15.148:0/3055309975 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa80cad70 con 0x7f0fa81538a0 2023-10-30T13:40:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fad865640 1 --2- 172.21.15.148:0/3055309975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa81538a0 0x7f0fa815d6f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80b30a0 tx=0x7f0fa401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771888b4fb0de5ba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fac863640 1 -- 172.21.15.148:0/3055309975 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40a89f0 con 0x7f0fa81538a0 2023-10-30T13:40:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fac863640 1 -- 172.21.15.148:0/3055309975 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa40a8b50 con 0x7f0fa81538a0 2023-10-30T13:40:32.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fac863640 1 -- 172.21.15.148:0/3055309975 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40b0440 con 0x7f0fa81538a0 2023-10-30T13:40:32.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 -- 172.21.15.148:0/3055309975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa81538a0 msgr2=0x7f0fa815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:32.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 --2- 172.21.15.148:0/3055309975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa81538a0 0x7f0fa815d6f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80b30a0 tx=0x7f0fa401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 -- 172.21.15.148:0/3055309975 shutdown_connections 2023-10-30T13:40:32.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 --2- 172.21.15.148:0/3055309975 >> v2:172.21.15.150:3300/0 conn(0x7f0fa80ca0e0 0x7f0fa815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 --2- 172.21.15.148:0/3055309975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa81538a0 0x7f0fa815d6f0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 -- 172.21.15.148:0/3055309975 >> 172.21.15.148:0/3055309975 conn(0x7f0fa806bfe0 msgr2=0x7f0fa806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 -- 172.21.15.148:0/3055309975 shutdown_connections 2023-10-30T13:40:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 -- 172.21.15.148:0/3055309975 wait complete. 2023-10-30T13:40:32.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 Processor -- start 2023-10-30T13:40:32.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.266+0000 7f0fafac9640 1 -- start start 2023-10-30T13:40:32.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fafac9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa80ca0e0 0x7f0fa80c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:32.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fafac9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fa80cf160 0x7f0fa814ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:32.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fafac9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fa81538a0 0x7f0fa81ef210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:32.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fafac9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa8151a80 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fafac9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa81f3330 con 0x7f0fa81538a0 2023-10-30T13:40:32.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fafac9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa80c9430 con 0x7f0fa80cf160 2023-10-30T13:40:32.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad865640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa80ca0e0 0x7f0fa80c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:32.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad865640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa80ca0e0 0x7f0fa80c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59802/0 (socket says 172.21.15.148:59802) 2023-10-30T13:40:32.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad865640 1 -- 172.21.15.148:0/2240153216 learned_addr learned my addr 172.21.15.148:0/2240153216 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:32.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fae066640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fa81538a0 0x7f0fa81ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:32.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad064640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fa80cf160 0x7f0fa814ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:32.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad865640 1 -- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fa80cf160 msgr2=0x7f0fa814ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:32.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad865640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fa80cf160 0x7f0fa814ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad865640 1 -- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fa81538a0 msgr2=0x7f0fa81ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:32.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad865640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fa81538a0 0x7f0fa81ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad865640 1 -- 172.21.15.148:0/2240153216 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa40094d0 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fae066640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fa81538a0 0x7f0fa81ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:40:32.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad064640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fa80cf160 0x7f0fa814ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:40:32.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fad865640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa80ca0e0 0x7f0fa80c8bf0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40b3040 tx=0x7f0fa401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:32.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0f9e7fc640 1 -- 172.21.15.148:0/2240153216 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40ba600 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0f9e7fc640 1 -- 172.21.15.148:0/2240153216 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa40ba760 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa81f2720 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0f9e7fc640 1 -- 172.21.15.148:0/2240153216 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40bab10 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa81f2ce0 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa80b2a50 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.270+0000 7f0f9e7fc640 1 -- 172.21.15.148:0/2240153216 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0fa40bac70 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.274+0000 7f0f9e7fc640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f8c0664a0 0x7f0f8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:32.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.274+0000 7f0f9e7fc640 1 -- 172.21.15.148:0/2240153216 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f0fa4122a30 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.274+0000 7f0fad064640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f8c0664a0 0x7f0f8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:32.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.274+0000 7f0fad064640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f8c0664a0 0x7f0f8c068960 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80c5040 tx=0x7f0f98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:32.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.278+0000 7f0f9e7fc640 1 -- 172.21.15.148:0/2240153216 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0fa40ef900 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.462+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fa804e170 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.462+0000 7f0f9e7fc640 1 -- 172.21.15.148:0/2240153216 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5717 (secure 0 0 0) 0x7f0fa40f2f20 con 0x7f0fa80ca0e0 2023-10-30T13:40:32.466 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:32.466 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:32.466 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 118 2023-10-30T13:40:32.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f8c0664a0 msgr2=0x7f0f8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:32.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f8c0664a0 0x7f0f8c068960 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80c5040 tx=0x7f0f98008040 comp rx=0 tx=0).stop 2023-10-30T13:40:32.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa80ca0e0 msgr2=0x7f0fa80c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:32.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa80ca0e0 0x7f0fa80c8bf0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40b3040 tx=0x7f0fa401ba90 comp rx=0 tx=0).stop 2023-10-30T13:40:32.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 shutdown_connections 2023-10-30T13:40:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0f8c0664a0 0x7f0f8c068960 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fa81538a0 0x7f0fa81ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0fa80cf160 0x7f0fa814ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 --2- 172.21.15.148:0/2240153216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fa80ca0e0 0x7f0fa80c8bf0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 >> 172.21.15.148:0/2240153216 conn(0x7f0fa806bfe0 msgr2=0x7f0fa80c4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:32.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 shutdown_connections 2023-10-30T13:40:32.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:32.466+0000 7f0fafac9640 1 -- 172.21.15.148:0/2240153216 wait complete. 2023-10-30T13:40:32.490 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:40:33.014 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 26% (29/111) Updating files: 27% (30/111) Updating files: 28% (32/111) Updating files: 29% (33/111) Updating files: 30% (34/111) Updating files: 31% (35/111) Updating files: 32% (36/111) Updating files: 33% (37/111) Updating files: 34% (38/111) Updating files: 35% (39/111) Updating files: 36% (40/111) Updating files: 37% (42/111) Updating files: 38% (43/111) Updating files: 39% (44/111) Updating files: 40% (45/111) Updating files: 41% (46/111) Updating files: 42% (47/111) Updating files: 43% (48/111) Updating files: 44% (49/111) Updating files: 45% (50/111) Updating files: 46% (52/111) Updating files: 47% (53/111) Updating files: 48% (54/111) Updating files: 49% (55/111) Updating files: 50% (56/111) Updating files: 51% (57/111) Updating files: 52% (58/111) Updating files: 53% (59/111) Updating files: 54% (60/111) Updating files: 55% (62/111) Updating files: 56% (63/111) Updating files: 57% (64/111) Updating files: 58% (65/111) Updating files: 59% (66/111) Updating files: 60% (67/111) Updating files: 61% (68/111) Updating files: 62% (69/111) Updating files: 63% (70/111) Updating files: 64% (72/111) Updating files: 65% (73/111) Updating files: 66% (74/111) Updating files: 67% (75/111) Updating files: 68% (76/111) Updating files: 69% (77/111) Updating files: 70% (78/111) Updating files: 71% (79/111) Updating files: 72% (80/111) Updating files: 73% (82/111) Updating files: 74% (83/111) Updating files: 75% (84/111) Updating files: 76% (85/111) Updating files: 77% (86/111) Updating files: 78% (87/111) Updating files: 79% (88/111) Updating files: 80% (89/111) Updating files: 81% (90/111) Updating files: 82% (92/111) Updating files: 83% (93/111) Updating files: 84% (94/111) Updating files: 85% (95/111) Updating files: 86% (96/111) Updating files: 87% (97/111) Updating files: 88% (98/111) Updating files: 89% (99/111) Updating files: 90% (100/111) Updating files: 91% (102/111) Updating files: 92% (103/111) Updating files: 93% (104/111) Updating files: 94% (105/111) Updating files: 95% (106/111) Updating files: 96% (107/111) Updating files: 97% (108/111) Updating files: 98% (109/111) Updating files: 99% (110/111) Updating files: 100% (111/111) Updating files: 100% (111/111), done. 2023-10-30T13:40:33.181 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 9fa8781c014 v0.43 2023-10-30T13:40:33.184 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.43 2023-10-30T13:40:33.214 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 44 -eq 48 ] 2023-10-30T13:40:33.214 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.44 2023-10-30T13:40:33.215 INFO:tasks.workunit.client.0.smithi148.stdout:v0.44 2023-10-30T13:40:33.215 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.44 2023-10-30T13:40:33.215 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.44 2023-10-30T13:40:34.492 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:40:34.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574e565640 1 Processor -- start 2023-10-30T13:40:34.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574e565640 1 -- start start 2023-10-30T13:40:34.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574e565640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5748153d70 0x7f574806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:34.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574e565640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f57480c8490 0x7f57480c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:34.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574e565640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574806afa0 con 0x7f574806e470 2023-10-30T13:40:34.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574e565640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574806d780 con 0x7f57480c8490 2023-10-30T13:40:34.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574e565640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5748154150 con 0x7f5748153d70 2023-10-30T13:40:34.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574cb02640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f57480c8490 0x7f57480c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:34.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f57477fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f574806e470 0x7f57480cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54786/0 (socket says 172.21.15.148:54786) 2023-10-30T13:40:34.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f57477fe640 1 -- 172.21.15.148:0/1356586212 learned_addr learned my addr 172.21.15.148:0/1356586212 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:40:34.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f5747fff640 1 --2- 172.21.15.148:0/1356586212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5748153d70 0x7f574806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:34.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574cb02640 1 -- 172.21.15.148:0/1356586212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5748153d70 msgr2=0x7f574806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:40:34.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574cb02640 1 --2- 172.21.15.148:0/1356586212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5748153d70 0x7f574806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574cb02640 1 -- 172.21.15.148:0/1356586212 >> v1:172.21.15.148:6789/0 conn(0x7f574806e470 legacy=0x7f57480cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:34.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f574cb02640 1 -- 172.21.15.148:0/1356586212 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574806e030 con 0x7f57480c8490 2023-10-30T13:40:34.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f5746ffd640 1 -- 172.21.15.148:0/1356586212 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270964497 0 0) 0x7f574806afa0 con 0x7f574806e470 2023-10-30T13:40:34.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.634+0000 7f5747fff640 1 --2- 172.21.15.148:0/1356586212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5748153d70 0x7f574806ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:40:34.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.638+0000 7f574cb02640 1 --2- 172.21.15.148:0/1356586212 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57480c8490 0x7f57480c8cc0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5740009d90 tx=0x7f574001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97de34629b3e367 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:34.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.638+0000 7f5746ffd640 1 -- 172.21.15.148:0/1356586212 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57400a59e0 con 0x7f57480c8490 2023-10-30T13:40:34.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.638+0000 7f5746ffd640 1 -- 172.21.15.148:0/1356586212 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57400a5b40 con 0x7f57480c8490 2023-10-30T13:40:34.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.638+0000 7f5746ffd640 1 -- 172.21.15.148:0/1356586212 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57400ad440 con 0x7f57480c8490 2023-10-30T13:40:34.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 -- 172.21.15.148:0/1356586212 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57480c8490 msgr2=0x7f57480c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:34.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 --2- 172.21.15.148:0/1356586212 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57480c8490 0x7f57480c8cc0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5740009d90 tx=0x7f574001c920 comp rx=0 tx=0).stop 2023-10-30T13:40:34.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 -- 172.21.15.148:0/1356586212 shutdown_connections 2023-10-30T13:40:34.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 --2- 172.21.15.148:0/1356586212 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f57480c8490 0x7f57480c8cc0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 --2- 172.21.15.148:0/1356586212 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5748153d70 0x7f574806ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 -- 172.21.15.148:0/1356586212 >> 172.21.15.148:0/1356586212 conn(0x7f5748071ea0 msgr2=0x7f5748065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:34.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 -- 172.21.15.148:0/1356586212 shutdown_connections 2023-10-30T13:40:34.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 -- 172.21.15.148:0/1356586212 wait complete. 2023-10-30T13:40:34.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 Processor -- start 2023-10-30T13:40:34.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 -- start start 2023-10-30T13:40:34.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574806e470 0x7f5748147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:34.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57480c8490 0x7f5748141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:34.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.642+0000 7f574e565640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5748153d70 0x7f5748141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:34.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574e565640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57481f5d00 con 0x7f5748153d70 2023-10-30T13:40:34.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574e565640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5748143a80 con 0x7f57480c8490 2023-10-30T13:40:34.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574e565640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5748143150 con 0x7f574806e470 2023-10-30T13:40:34.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574cb02640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5748153d70 0x7f5748141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:34.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574cb02640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5748153d70 0x7f5748141dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59820/0 (socket says 172.21.15.148:59820) 2023-10-30T13:40:34.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574cb02640 1 -- 172.21.15.148:0/2190300191 learned_addr learned my addr 172.21.15.148:0/2190300191 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:40:34.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f5747fff640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574806e470 0x7f5748147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:34.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f57477fe640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57480c8490 0x7f5748141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:34.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574cb02640 1 -- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574806e470 msgr2=0x7f5748147680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:34.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574cb02640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574806e470 0x7f5748147680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574cb02640 1 -- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57480c8490 msgr2=0x7f5748141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:34.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574cb02640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57480c8490 0x7f5748141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574cb02640 1 -- 172.21.15.148:0/2190300191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5740009a70 con 0x7f5748153d70 2023-10-30T13:40:34.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574cb02640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5748153d70 0x7f5748141dc0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f574001afd0 tx=0x7f57400196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:34.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f5744ff9640 1 -- 172.21.15.148:0/2190300191 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57400ade40 con 0x7f5748153d70 2023-10-30T13:40:34.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f5744ff9640 1 -- 172.21.15.148:0/2190300191 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5740009450 con 0x7f5748153d70 2023-10-30T13:40:34.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f5744ff9640 1 -- 172.21.15.148:0/2190300191 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57400b7630 con 0x7f5748153d70 2023-10-30T13:40:34.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5748143430 con 0x7f5748153d70 2023-10-30T13:40:34.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574815d9d0 con 0x7f5748153d70 2023-10-30T13:40:34.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.646+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574804e170 con 0x7f5748153d70 2023-10-30T13:40:34.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.650+0000 7f5744ff9640 1 -- 172.21.15.148:0/2190300191 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57400b4050 con 0x7f5748153d70 2023-10-30T13:40:34.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.650+0000 7f5744ff9640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57340664a0 0x7f5734068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:40:34.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.650+0000 7f5744ff9640 1 -- 172.21.15.148:0/2190300191 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f574011f6c0 con 0x7f5748153d70 2023-10-30T13:40:34.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.650+0000 7f5747fff640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57340664a0 0x7f5734068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:40:34.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.650+0000 7f5747fff640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57340664a0 0x7f5734068960 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5738006e90 tx=0x7f5738008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:40:34.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.654+0000 7f5744ff9640 1 -- 172.21.15.148:0/2190300191 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57400ec520 con 0x7f5748153d70 2023-10-30T13:40:34.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.838+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f574814de00 con 0x7f5748153d70 2023-10-30T13:40:34.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.838+0000 7f5744ff9640 1 -- 172.21.15.148:0/2190300191 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f57400b2090 con 0x7f5748153d70 2023-10-30T13:40:34.841 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:40:34.841 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:40:34.841 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 119 2023-10-30T13:40:34.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57340664a0 msgr2=0x7f5734068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:34.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57340664a0 0x7f5734068960 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5738006e90 tx=0x7f5738008040 comp rx=0 tx=0).stop 2023-10-30T13:40:34.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5748153d70 msgr2=0x7f5748141dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:40:34.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5748153d70 0x7f5748141dc0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f574001afd0 tx=0x7f57400196c0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 shutdown_connections 2023-10-30T13:40:34.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f57340664a0 0x7f5734068960 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5748153d70 0x7f5748141dc0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57480c8490 0x7f5748141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 --2- 172.21.15.148:0/2190300191 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f574806e470 0x7f5748147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:40:34.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 >> 172.21.15.148:0/2190300191 conn(0x7f5748071ea0 msgr2=0x7f574806f210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:40:34.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 shutdown_connections 2023-10-30T13:40:34.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:40:34.842+0000 7f574e565640 1 -- 172.21.15.148:0/2190300191 wait complete. 2023-10-30T13:40:34.861 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:40:34.862 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-10-30T13:40:35.932 INFO:tasks.ceph.mds.c.smithi148.stderr:2023-10-30T13:40:35.934+0000 7fc27a6b7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:40:39.824 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 21% (40/186) Updating files: 22% (41/186) Updating files: 23% (43/186) Updating files: 24% (45/186) Updating files: 25% (47/186) Updating files: 26% (49/186) Updating files: 27% (51/186) Updating files: 28% (53/186) Updating files: 29% (54/186) Updating files: 30% (56/186) Updating files: 31% (58/186) Updating files: 32% (60/186) Updating files: 33% (62/186) Updating files: 34% (64/186) Updating files: 35% (66/186) Updating files: 36% (67/186) Updating files: 36% (68/186) Updating files: 37% (69/186) Updating files: 38% (71/186) Updating files: 39% (73/186) Updating files: 40% (75/186) Updating files: 41% (77/186) Updating files: 42% (79/186) Updating files: 43% (80/186) Updating files: 44% (82/186) Updating files: 45% (84/186) Updating files: 46% (86/186) Updating files: 47% (88/186) Updating files: 48% (90/186) Updating files: 49% (92/186) Updating files: 50% (93/186) Updating files: 51% (95/186) Updating files: 52% (97/186) Updating files: 53% (99/186) Updating files: 54% (101/186) Updating files: 55% (103/186) Updating files: 56% (105/186) Updating files: 57% (107/186) Updating files: 58% (108/186) Updating files: 59% (110/186) Updating files: 60% (112/186) Updating files: 61% (114/186) Updating files: 62% (116/186) Updating files: 63% (118/186) Updating files: 64% (120/186) Updating files: 65% (121/186) Updating files: 66% (123/186) Updating files: 67% (125/186) Updating files: 68% (127/186) Updating files: 69% (129/186) Updating files: 70% (131/186) Updating files: 71% (133/186) Updating files: 72% (134/186) Updating files: 73% (136/186) Updating files: 74% (138/186) Updating files: 75% (140/186) Updating files: 76% (142/186) Updating files: 77% (144/186) Updating files: 78% (146/186) Updating files: 79% (147/186) Updating files: 80% (149/186) Updating files: 81% (151/186) Updating files: 82% (153/186) Updating files: 83% (155/186) Updating files: 84% (157/186) Updating files: 85% (159/186) Updating files: 86% (160/186) Updating files: 87% (162/186) Updating files: 88% (164/186) Updating files: 89% (166/186) Updating files: 90% (168/186) Updating files: 91% (170/186) Updating files: 92% (172/186) Updating files: 93% (173/186) Updating files: 94% (175/186) Updating files: 95% (177/186) Updating files: 96% (179/186) Updating files: 97% (181/186) Updating files: 98% (183/186) Updating files: 99% (185/186) Updating files: 100% (186/186) Updating files: 100% (186/186), done. 2023-10-30T13:40:39.863 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at cdd5298d276 v0.44 2023-10-30T13:40:39.867 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.44 2023-10-30T13:40:39.977 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 45 -eq 48 ] 2023-10-30T13:40:39.977 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.45 2023-10-30T13:40:39.978 INFO:tasks.workunit.client.0.smithi148.stdout:v0.45 2023-10-30T13:40:39.978 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.45 2023-10-30T13:40:39.978 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.45 2023-10-30T13:40:42.120 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 31% (48/154) Updating files: 32% (50/154) Updating files: 33% (51/154) Updating files: 34% (53/154) Updating files: 35% (54/154) Updating files: 36% (56/154) Updating files: 37% (57/154) Updating files: 38% (59/154) Updating files: 39% (61/154) Updating files: 40% (62/154) Updating files: 41% (64/154) Updating files: 42% (65/154) Updating files: 43% (67/154) Updating files: 44% (68/154) Updating files: 45% (70/154) Updating files: 46% (71/154) Updating files: 47% (73/154) Updating files: 48% (74/154) Updating files: 49% (76/154) Updating files: 50% (77/154) Updating files: 51% (79/154) Updating files: 52% (81/154) Updating files: 53% (82/154) Updating files: 54% (84/154) Updating files: 55% (85/154) Updating files: 56% (87/154) Updating files: 57% (88/154) Updating files: 58% (90/154) Updating files: 59% (91/154) Updating files: 60% (93/154) Updating files: 61% (94/154) Updating files: 62% (96/154) Updating files: 63% (98/154) Updating files: 64% (99/154) Updating files: 65% (101/154) Updating files: 66% (102/154) Updating files: 67% (104/154) Updating files: 68% (105/154) Updating files: 69% (107/154) Updating files: 70% (108/154) Updating files: 71% (110/154) Updating files: 72% (111/154) Updating files: 73% (113/154) Updating files: 74% (114/154) Updating files: 75% (116/154) Updating files: 76% (118/154) Updating files: 77% (119/154) Updating files: 78% (121/154) Updating files: 79% (122/154) Updating files: 80% (124/154) Updating files: 81% (125/154) Updating files: 82% (127/154) Updating files: 83% (128/154) Updating files: 84% (130/154) Updating files: 85% (131/154) Updating files: 86% (133/154) Updating files: 87% (134/154) Updating files: 88% (136/154) Updating files: 89% (138/154) Updating files: 90% (139/154) Updating files: 91% (141/154) Updating files: 92% (142/154) Updating files: 93% (144/154) Updating files: 94% (145/154) Updating files: 95% (147/154) Updating files: 96% (148/154) Updating files: 97% (150/154) Updating files: 98% (151/154) Updating files: 99% (153/154) Updating files: 100% (154/154) Updating files: 100% (154/154), done. 2023-10-30T13:40:42.778 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 0aea1cb1df5 v0.45 2023-10-30T13:40:42.781 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.45 2023-10-30T13:40:43.743 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 46 -eq 48 ] 2023-10-30T13:40:43.743 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.46 2023-10-30T13:40:43.743 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.46 2023-10-30T13:40:43.744 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.46 2023-10-30T13:40:43.744 INFO:tasks.workunit.client.0.smithi148.stdout:v0.46 2023-10-30T13:40:47.055 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 26% (82/309) Updating files: 27% (84/309) Updating files: 28% (87/309) Updating files: 29% (90/309) Updating files: 30% (93/309) Updating files: 31% (96/309) Updating files: 32% (99/309) Updating files: 33% (102/309) Updating files: 34% (106/309) Updating files: 35% (109/309) Updating files: 36% (112/309) Updating files: 37% (115/309) Updating files: 38% (118/309) Updating files: 39% (121/309) Updating files: 40% (124/309) Updating files: 41% (127/309) Updating files: 42% (130/309) Updating files: 43% (133/309) Updating files: 43% (134/309) Updating files: 44% (136/309) Updating files: 45% (140/309) Updating files: 46% (143/309) Updating files: 47% (146/309) Updating files: 48% (149/309) Updating files: 49% (152/309) Updating files: 50% (155/309) Updating files: 51% (158/309) Updating files: 52% (161/309) Updating files: 53% (164/309) Updating files: 54% (167/309) Updating files: 55% (170/309) Updating files: 56% (174/309) Updating files: 57% (177/309) Updating files: 58% (180/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2023-10-30T13:40:47.102 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at cb7f1c9c752 v0.46 2023-10-30T13:40:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.46 2023-10-30T13:40:48.128 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 47 -eq 48 ] 2023-10-30T13:40:48.128 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.47 2023-10-30T13:40:48.129 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.47 2023-10-30T13:40:48.129 INFO:tasks.workunit.client.0.smithi148.stdout:v0.47 2023-10-30T13:40:48.129 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.47 2023-10-30T13:41:01.222 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 36% (79/219) Updating files: 37% (82/219) Updating files: 38% (84/219) Updating files: 39% (86/219) Updating files: 40% (88/219) Updating files: 41% (90/219) Updating files: 42% (92/219) Updating files: 43% (95/219) Updating files: 44% (97/219) Updating files: 45% (99/219) Updating files: 46% (101/219) Updating files: 47% (103/219) Updating files: 48% (106/219) Updating files: 49% (108/219) Updating files: 50% (110/219) Updating files: 51% (112/219) Updating files: 52% (114/219) Updating files: 53% (117/219) Updating files: 54% (119/219) Updating files: 55% (121/219) Updating files: 56% (123/219) Updating files: 57% (125/219) Updating files: 58% (128/219) Updating files: 59% (130/219) Updating files: 60% (132/219) Updating files: 61% (134/219) Updating files: 62% (136/219) Updating files: 63% (138/219) Updating files: 64% (141/219) Updating files: 65% (143/219) Updating files: 66% (145/219) Updating files: 67% (147/219) Updating files: 68% (149/219) Updating files: 69% (152/219) Updating files: 70% (154/219) Updating files: 71% (156/219) Updating files: 72% (158/219) Updating files: 73% (160/219) Updating files: 74% (163/219) Updating files: 75% (165/219) Updating files: 76% (167/219) Updating files: 77% (169/219) Updating files: 78% (171/219) Updating files: 79% (174/219) Updating files: 80% (176/219) Updating files: 81% (178/219) Updating files: 82% (180/219) Updating files: 83% (182/219) Updating files: 84% (184/219) Updating files: 85% (187/219) Updating files: 86% (189/219) Updating files: 87% (191/219) Updating files: 88% (193/219) Updating files: 89% (195/219) Updating files: 90% (198/219) Updating files: 91% (200/219) Updating files: 92% (202/219) Updating files: 93% (204/219) Updating files: 94% (206/219) Updating files: 95% (209/219) Updating files: 96% (211/219) Updating files: 97% (213/219) Updating files: 98% (215/219) Updating files: 99% (217/219) Updating files: 100% (219/219) Updating files: 100% (219/219), done. 2023-10-30T13:41:01.222 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:01.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:01.508 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at ad663d58c89 v0.47 2023-10-30T13:41:01.511 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.47 2023-10-30T13:41:01.545 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 48 -eq 48 ] 2023-10-30T13:41:01.545 INFO:tasks.workunit.client.0.smithi148.stderr:+ continue 2023-10-30T13:41:01.546 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 49 -eq 48 ] 2023-10-30T13:41:01.546 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.49 2023-10-30T13:41:01.546 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.49 2023-10-30T13:41:01.546 INFO:tasks.workunit.client.0.smithi148.stdout:v0.49 2023-10-30T13:41:01.547 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.49 2023-10-30T13:41:07.370 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 38% (185/481) Updating files: 39% (188/481) Updating files: 40% (193/481) Updating files: 41% (198/481) Updating files: 42% (203/481) Updating files: 43% (207/481) Updating files: 44% (212/481) Updating files: 45% (217/481) Updating files: 46% (222/481) Updating files: 47% (227/481) Updating files: 48% (231/481) Updating files: 49% (236/481) Updating files: 50% (241/481) Updating files: 51% (246/481) Updating files: 52% (251/481) Updating files: 53% (255/481) Updating files: 54% (260/481) Updating files: 55% (265/481) Updating files: 56% (270/481) Updating files: 57% (275/481) Updating files: 58% (279/481) Updating files: 59% (284/481) Updating files: 60% (289/481) Updating files: 61% (294/481) Updating files: 62% (299/481) Updating files: 63% (304/481) Updating files: 64% (308/481) Updating files: 65% (313/481) Updating files: 66% (318/481) Updating files: 67% (323/481) Updating files: 68% (328/481) Updating files: 69% (332/481) Updating files: 70% (337/481) Updating files: 71% (342/481) Updating files: 72% (347/481) Updating files: 73% (352/481) Updating files: 74% (356/481) Updating files: 75% (361/481) Updating files: 76% (366/481) Updating files: 77% (371/481) Updating files: 78% (376/481) Updating files: 79% (380/481) Updating files: 80% (385/481) Updating files: 81% (390/481) Updating files: 82% (395/481) Updating files: 83% (400/481) Updating files: 84% (405/481) Updating files: 85% (409/481) Updating files: 86% (414/481) Updating files: 87% (419/481) Updating files: 88% (424/481) Updating files: 88% (426/481) Updating files: 89% (429/481) Updating files: 90% (433/481) Updating files: 91% (438/481) Updating files: 92% (443/481) Updating files: 93% (448/481) Updating files: 94% (453/481) Updating files: 95% (457/481) Updating files: 96% (462/481) Updating files: 97% (467/481) Updating files: 98% (472/481) Updating files: 99% (477/481) Updating files: 100% (481/481) Updating files: 100% (481/481), done. 2023-10-30T13:41:07.983 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at ca6265d0f4d v0.49 2023-10-30T13:41:07.987 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.49 2023-10-30T13:41:08.473 INFO:tasks.workunit.client.0.smithi148.stdout:v0.50 2023-10-30T13:41:08.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 50 -eq 48 ] 2023-10-30T13:41:08.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.50 2023-10-30T13:41:08.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.50 2023-10-30T13:41:08.474 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.50 2023-10-30T13:41:18.353 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 81% (111/137) Updating files: 82% (113/137) Updating files: 83% (114/137) Updating files: 84% (116/137) Updating files: 85% (117/137) Updating files: 86% (118/137) Updating files: 87% (120/137) Updating files: 88% (121/137) Updating files: 89% (122/137) Updating files: 90% (124/137) Updating files: 91% (125/137) Updating files: 92% (127/137) Updating files: 93% (128/137) Updating files: 94% (129/137) Updating files: 95% (131/137) Updating files: 96% (132/137) Updating files: 97% (133/137) Updating files: 98% (135/137) Updating files: 99% (136/137) Updating files: 100% (137/137) Updating files: 100% (137/137), done. 2023-10-30T13:41:18.389 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at a4428bd0e19 v0.50 2023-10-30T13:41:18.392 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.50 2023-10-30T13:41:18.666 INFO:tasks.workunit.client.0.smithi148.stdout:v0.51 2023-10-30T13:41:18.667 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 51 -eq 48 ] 2023-10-30T13:41:18.667 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.51 2023-10-30T13:41:18.667 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.51 2023-10-30T13:41:18.668 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.51 2023-10-30T13:41:23.757 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 10% (11/105) Updating files: 11% (12/105) Updating files: 12% (13/105) Updating files: 13% (14/105) Updating files: 14% (15/105) Updating files: 15% (16/105) Updating files: 16% (17/105) Updating files: 17% (18/105) Updating files: 18% (19/105) Updating files: 19% (20/105) Updating files: 20% (21/105) Updating files: 21% (23/105) Updating files: 22% (24/105) Updating files: 23% (25/105) Updating files: 24% (26/105) Updating files: 25% (27/105) Updating files: 26% (28/105) Updating files: 27% (29/105) Updating files: 28% (30/105) Updating files: 29% (31/105) Updating files: 30% (32/105) Updating files: 31% (33/105) Updating files: 32% (34/105) Updating files: 33% (35/105) Updating files: 34% (36/105) Updating files: 35% (37/105) Updating files: 36% (38/105) Updating files: 37% (39/105) Updating files: 38% (40/105) Updating files: 39% (41/105) Updating files: 40% (42/105) Updating files: 41% (44/105) Updating files: 42% (45/105) Updating files: 43% (46/105) Updating files: 44% (47/105) Updating files: 45% (48/105) Updating files: 46% (49/105) Updating files: 47% (50/105) Updating files: 48% (51/105) Updating files: 49% (52/105) Updating files: 50% (53/105) Updating files: 51% (54/105) Updating files: 52% (55/105) Updating files: 53% (56/105) Updating files: 54% (57/105) Updating files: 55% (58/105) Updating files: 56% (59/105) Updating files: 57% (60/105) Updating files: 58% (61/105) Updating files: 59% (62/105) Updating files: 60% (63/105) Updating files: 61% (65/105) Updating files: 62% (66/105) Updating files: 63% (67/105) Updating files: 64% (68/105) Updating files: 65% (69/105) Updating files: 66% (70/105) Updating files: 67% (71/105) Updating files: 68% (72/105) Updating files: 69% (73/105) Updating files: 70% (74/105) Updating files: 71% (75/105) Updating files: 72% (76/105) Updating files: 73% (77/105) Updating files: 74% (78/105) Updating files: 75% (79/105) Updating files: 76% (80/105) Updating files: 77% (81/105) Updating files: 78% (82/105) Updating files: 79% (83/105) Updating files: 80% (84/105) Updating files: 81% (86/105) Updating files: 82% (87/105) Updating files: 83% (88/105) Updating files: 84% (89/105) Updating files: 85% (90/105) Updating files: 86% (91/105) Updating files: 87% (92/105) Updating files: 88% (93/105) Updating files: 89% (94/105) Updating files: 90% (95/105) Updating files: 91% (96/105) Updating files: 92% (97/105) Updating files: 93% (98/105) Updating files: 94% (99/105) Updating files: 95% (100/105) Updating files: 96% (101/105) Updating files: 97% (102/105) Updating files: 98% (103/105) Updating files: 99% (104/105) Updating files: 100% (105/105) Updating files: 100% (105/105), done. 2023-10-30T13:41:23.807 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at c03ca95d235 v0.51 2023-10-30T13:41:23.810 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.51 2023-10-30T13:41:23.907 INFO:tasks.workunit.client.0.smithi148.stdout:v0.52 2023-10-30T13:41:23.908 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 52 -eq 48 ] 2023-10-30T13:41:23.908 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.52 2023-10-30T13:41:23.908 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.52 2023-10-30T13:41:23.908 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.52 2023-10-30T13:41:31.433 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:31.438 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:41:38.032 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 65% (120/183) Updating files: 66% (121/183) Updating files: 67% (123/183) Updating files: 68% (125/183) Updating files: 69% (127/183) Updating files: 70% (129/183) Updating files: 71% (130/183) Updating files: 72% (132/183) Updating files: 73% (134/183) Updating files: 74% (136/183) Updating files: 75% (138/183) Updating files: 76% (140/183) Updating files: 77% (141/183) Updating files: 77% (142/183) Updating files: 78% (143/183) Updating files: 79% (145/183) Updating files: 80% (147/183) Updating files: 81% (149/183) Updating files: 82% (151/183) Updating files: 83% (152/183) Updating files: 84% (154/183) Updating files: 85% (156/183) Updating files: 86% (158/183) Updating files: 87% (160/183) Updating files: 88% (162/183) Updating files: 89% (163/183) Updating files: 90% (165/183) Updating files: 91% (167/183) Updating files: 92% (169/183) Updating files: 93% (171/183) Updating files: 94% (173/183) Updating files: 95% (174/183) Updating files: 96% (176/183) Updating files: 97% (178/183) Updating files: 98% (180/183) Updating files: 99% (182/183) Updating files: 100% (183/183) Updating files: 100% (183/183), done. 2023-10-30T13:41:38.138 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at e48859474c4 v0.52 2023-10-30T13:41:38.142 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.52 2023-10-30T13:41:38.417 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 53 -eq 48 ] 2023-10-30T13:41:38.418 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.53 2023-10-30T13:41:38.418 INFO:tasks.workunit.client.0.smithi148.stdout:v0.53 2023-10-30T13:41:38.419 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.53 2023-10-30T13:41:38.419 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.53 2023-10-30T13:41:43.221 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 30% (117/386) Updating files: 31% (120/386) Updating files: 32% (124/386) Updating files: 33% (128/386) Updating files: 34% (132/386) Updating files: 35% (136/386) Updating files: 36% (139/386) Updating files: 37% (143/386) Updating files: 38% (147/386) Updating files: 39% (151/386) Updating files: 40% (155/386) Updating files: 41% (159/386) Updating files: 42% (163/386) Updating files: 43% (166/386) Updating files: 44% (170/386) Updating files: 45% (174/386) Updating files: 46% (178/386) Updating files: 47% (182/386) Updating files: 48% (186/386) Updating files: 49% (190/386) Updating files: 50% (193/386) Updating files: 50% (195/386) Updating files: 51% (197/386) Updating files: 52% (201/386) Updating files: 53% (205/386) Updating files: 54% (209/386) Updating files: 55% (213/386) Updating files: 56% (217/386) Updating files: 57% (221/386) Updating files: 58% (224/386) Updating files: 59% (228/386) Updating files: 60% (232/386) Updating files: 61% (236/386) Updating files: 62% (240/386) Updating files: 63% (244/386) Updating files: 64% (248/386) Updating files: 65% (251/386) Updating files: 66% (255/386) Updating files: 67% (259/386) Updating files: 68% (263/386) Updating files: 69% (267/386) Updating files: 70% (271/386) Updating files: 71% (275/386) Updating files: 72% (278/386) Updating files: 73% (282/386) Updating files: 74% (286/386) Updating files: 75% (290/386) Updating files: 76% (294/386) Updating files: 77% (298/386) Updating files: 78% (302/386) Updating files: 79% (305/386) Updating files: 80% (309/386) Updating files: 81% (313/386) Updating files: 82% (317/386) Updating files: 83% (321/386) Updating files: 83% (324/386) Updating files: 84% (325/386) Updating files: 85% (329/386) Updating files: 86% (332/386) Updating files: 87% (336/386) Updating files: 88% (340/386) Updating files: 89% (344/386) Updating files: 90% (348/386) Updating files: 91% (352/386) Updating files: 92% (356/386) Updating files: 93% (359/386) Updating files: 94% (363/386) Updating files: 95% (367/386) Updating files: 96% (371/386) Updating files: 96% (373/386) Updating files: 97% (375/386) Updating files: 98% (379/386) Updating files: 99% (383/386) Updating files: 100% (386/386) Updating files: 100% (386/386), done. 2023-10-30T13:41:43.847 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 2528b5ee105 v0.43 2023-10-30T13:41:43.854 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.53 2023-10-30T13:41:44.117 INFO:tasks.workunit.client.0.smithi148.stdout:v0.54 2023-10-30T13:41:44.118 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 54 -eq 48 ] 2023-10-30T13:41:44.118 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.54 2023-10-30T13:41:44.119 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.54 2023-10-30T13:41:44.119 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.54 2023-10-30T13:41:54.533 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 2% (5/212) Updating files: 3% (7/212) Updating files: 4% (9/212) Updating files: 5% (11/212) Updating files: 6% (13/212) Updating files: 7% (15/212) Updating files: 8% (17/212) Updating files: 9% (20/212) Updating files: 10% (22/212) Updating files: 11% (24/212) Updating files: 12% (26/212) Updating files: 13% (28/212) Updating files: 14% (30/212) Updating files: 15% (32/212) Updating files: 16% (34/212) Updating files: 17% (37/212) Updating files: 18% (39/212) Updating files: 19% (41/212) Updating files: 20% (43/212) Updating files: 21% (45/212) Updating files: 22% (47/212) Updating files: 23% (49/212) Updating files: 24% (51/212) Updating files: 25% (53/212) Updating files: 26% (56/212) Updating files: 27% (58/212) Updating files: 28% (60/212) Updating files: 29% (62/212) Updating files: 30% (64/212) Updating files: 31% (66/212) Updating files: 32% (68/212) Updating files: 33% (70/212) Updating files: 34% (73/212) Updating files: 35% (75/212) Updating files: 36% (77/212) Updating files: 37% (79/212) Updating files: 38% (81/212) Updating files: 39% (83/212) Updating files: 40% (85/212) Updating files: 41% (87/212) Updating files: 42% (90/212) Updating files: 43% (92/212) Updating files: 44% (94/212) Updating files: 45% (96/212) Updating files: 46% (98/212) Updating files: 47% (100/212) Updating files: 48% (102/212) Updating files: 49% (104/212) Updating files: 50% (106/212) Updating files: 51% (109/212) Updating files: 52% (111/212) Updating files: 53% (113/212) Updating files: 54% (115/212) Updating files: 55% (117/212) Updating files: 56% (119/212) Updating files: 57% (121/212) Updating files: 58% (123/212) Updating files: 59% (126/212) Updating files: 60% (128/212) Updating files: 61% (130/212) Updating files: 62% (132/212) Updating files: 63% (134/212) Updating files: 64% (136/212) Updating files: 65% (138/212) Updating files: 66% (140/212) Updating files: 67% (143/212) Updating files: 68% (145/212) Updating files: 69% (147/212) Updating files: 70% (149/212) Updating files: 71% (151/212) Updating files: 72% (153/212) Updating files: 73% (155/212) Updating files: 74% (157/212) Updating files: 75% (159/212) Updating files: 76% (162/212) Updating files: 77% (164/212) Updating files: 78% (166/212) Updating files: 79% (168/212) Updating files: 80% (170/212) Updating files: 81% (172/212) Updating files: 82% (174/212) Updating files: 83% (176/212) Updating files: 84% (179/212) Updating files: 85% (181/212) Updating files: 86% (183/212) Updating files: 87% (185/212) Updating files: 88% (187/212) Updating files: 89% (189/212) Updating files: 90% (191/212) Updating files: 91% (193/212) Updating files: 92% (196/212) Updating files: 93% (198/212) Updating files: 94% (200/212) Updating files: 95% (202/212) Updating files: 96% (204/212) Updating files: 97% (206/212) Updating files: 98% (208/212) Updating files: 99% (210/212) Updating files: 100% (212/212) Updating files: 100% (212/212), done. 2023-10-30T13:41:54.571 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 60b84b095b1 v0.54 2023-10-30T13:41:54.576 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.54 2023-10-30T13:41:55.035 INFO:tasks.workunit.client.0.smithi148.stdout:v0.55 2023-10-30T13:41:55.035 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 55 -eq 48 ] 2023-10-30T13:41:55.035 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.55 2023-10-30T13:41:55.036 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.55 2023-10-30T13:41:55.036 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.55 2023-10-30T13:42:01.488 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:01.493 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:11.069 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 8% (46/540) Updating files: 9% (49/540) Updating files: 10% (54/540) Updating files: 11% (60/540) Updating files: 12% (65/540) Updating files: 13% (71/540) Updating files: 14% (76/540) Updating files: 15% (81/540) Updating files: 15% (85/540) Updating files: 16% (87/540) Updating files: 17% (92/540) Updating files: 18% (98/540) Updating files: 19% (103/540) Updating files: 20% (108/540) Updating files: 21% (114/540) Updating files: 22% (119/540) Updating files: 23% (125/540) Updating files: 24% (130/540) Updating files: 25% (135/540) Updating files: 26% (141/540) Updating files: 27% (146/540) Updating files: 28% (152/540) Updating files: 29% (157/540) Updating files: 30% (162/540) Updating files: 31% (168/540) Updating files: 31% (171/540) Updating files: 32% (173/540) Updating files: 33% (179/540) Updating files: 34% (184/540) Updating files: 34% (186/540) Updating files: 35% (189/540) Updating files: 36% (195/540) Updating files: 37% (200/540) Updating files: 38% (206/540) Updating files: 39% (211/540) Updating files: 40% (216/540) Updating files: 41% (222/540) Updating files: 42% (227/540) Updating files: 43% (233/540) Updating files: 44% (238/540) Updating files: 45% (243/540) Updating files: 46% (249/540) Updating files: 47% (254/540) Updating files: 48% (260/540) Updating files: 49% (265/540) Updating files: 49% (266/540) Updating files: 50% (270/540) Updating files: 51% (276/540) Updating files: 52% (281/540) Updating files: 53% (287/540) Updating files: 54% (292/540) Updating files: 55% (297/540) Updating files: 56% (303/540) Updating files: 57% (308/540) Updating files: 58% (314/540) Updating files: 58% (315/540) Updating files: 59% (319/540) Updating files: 60% (324/540) Updating files: 61% (330/540) Updating files: 62% (335/540) Updating files: 63% (341/540) Updating files: 64% (346/540) Updating files: 65% (351/540) Updating files: 65% (356/540) Updating files: 66% (357/540) Updating files: 67% (362/540) Updating files: 68% (368/540) Updating files: 69% (373/540) Updating files: 70% (378/540) Updating files: 71% (384/540) Updating files: 72% (389/540) Updating files: 72% (394/540) Updating files: 73% (395/540) Updating files: 74% (400/540) Updating files: 75% (405/540) Updating files: 76% (411/540) Updating files: 77% (416/540) Updating files: 78% (422/540) Updating files: 79% (427/540) Updating files: 80% (432/540) Updating files: 81% (438/540) Updating files: 82% (443/540) Updating files: 83% (449/540) Updating files: 84% (454/540) Updating files: 85% (459/540) Updating files: 86% (465/540) Updating files: 87% (470/540) Updating files: 88% (476/540) Updating files: 89% (481/540) Updating files: 90% (486/540) Updating files: 91% (492/540) Updating files: 92% (497/540) Updating files: 93% (503/540) Updating files: 94% (508/540) Updating files: 95% (513/540) Updating files: 95% (516/540) Updating files: 96% (519/540) Updating files: 97% (524/540) Updating files: 98% (530/540) Updating files: 99% (535/540) Updating files: 100% (540/540) Updating files: 100% (540/540), done. 2023-10-30T13:42:11.099 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 690f8175606 v0.55 2023-10-30T13:42:11.106 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.55 2023-10-30T13:42:11.246 INFO:tasks.workunit.client.0.smithi148.stdout:v0.56 2023-10-30T13:42:11.246 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 56 -eq 48 ] 2023-10-30T13:42:11.246 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.56 2023-10-30T13:42:11.246 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.56 2023-10-30T13:42:11.247 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.56 2023-10-30T13:42:13.748 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 34% (84/246) Updating files: 35% (87/246) Updating files: 36% (89/246) Updating files: 37% (92/246) Updating files: 38% (94/246) Updating files: 39% (96/246) Updating files: 40% (99/246) Updating files: 41% (101/246) Updating files: 42% (104/246) Updating files: 43% (106/246) Updating files: 44% (109/246) Updating files: 45% (111/246) Updating files: 46% (114/246) Updating files: 47% (116/246) Updating files: 48% (119/246) Updating files: 49% (121/246) Updating files: 50% (123/246) Updating files: 51% (126/246) Updating files: 52% (128/246) Updating files: 53% (131/246) Updating files: 54% (133/246) Updating files: 55% (136/246) Updating files: 56% (138/246) Updating files: 57% (141/246) Updating files: 58% (143/246) Updating files: 59% (146/246) Updating files: 60% (148/246) Updating files: 61% (151/246) Updating files: 62% (153/246) Updating files: 63% (155/246) Updating files: 64% (158/246) Updating files: 65% (160/246) Updating files: 66% (163/246) Updating files: 67% (165/246) Updating files: 68% (168/246) Updating files: 69% (170/246) Updating files: 70% (173/246) Updating files: 71% (175/246) Updating files: 72% (178/246) Updating files: 73% (180/246) Updating files: 74% (183/246) Updating files: 75% (185/246) Updating files: 76% (187/246) Updating files: 77% (190/246) Updating files: 78% (192/246) Updating files: 79% (195/246) Updating files: 80% (197/246) Updating files: 81% (200/246) Updating files: 82% (202/246) Updating files: 83% (205/246) Updating files: 84% (207/246) Updating files: 85% (210/246) Updating files: 86% (212/246) Updating files: 87% (215/246) Updating files: 88% (217/246) Updating files: 89% (219/246) Updating files: 90% (222/246) Updating files: 91% (224/246) Updating files: 91% (226/246) Updating files: 92% (227/246) Updating files: 93% (229/246) Updating files: 94% (232/246) Updating files: 95% (234/246) Updating files: 96% (237/246) Updating files: 97% (239/246) Updating files: 98% (242/246) Updating files: 99% (244/246) Updating files: 100% (246/246) Updating files: 100% (246/246), done. 2023-10-30T13:42:14.037 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 1a32f0a0b42 v0.56 2023-10-30T13:42:14.043 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.56 2023-10-30T13:42:14.331 INFO:tasks.workunit.client.0.smithi148.stdout:v0.57 2023-10-30T13:42:14.331 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 57 -eq 48 ] 2023-10-30T13:42:14.331 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.57 2023-10-30T13:42:14.331 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.57 2023-10-30T13:42:14.332 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.57 2023-10-30T13:42:16.865 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:17.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f9067b1d640 1 Processor -- start 2023-10-30T13:42:17.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f9067b1d640 1 -- start start 2023-10-30T13:42:17.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f9067b1d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f906006f7a0 0x7f906006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f9067b1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9060071110 0x7f9060158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f9067b1d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9060066cf0 con 0x7f90600c6bd0 2023-10-30T13:42:17.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f9067b1d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90600700c0 con 0x7f906006f7a0 2023-10-30T13:42:17.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f9067b1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90600c70d0 con 0x7f9060071110 2023-10-30T13:42:17.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f90650b8640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f90600c6bd0 0x7f9060069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51710/0 (socket says 172.21.15.148:51710) 2023-10-30T13:42:17.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f90658b9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f906006f7a0 0x7f906006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f90650b8640 1 -- 172.21.15.148:0/323607770 learned_addr learned my addr 172.21.15.148:0/323607770 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:17.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.076+0000 7f90660ba640 1 --2- 172.21.15.148:0/323607770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9060071110 0x7f9060158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90648b7640 1 -- 172.21.15.148:0/323607770 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353424994 0 0) 0x7f9060066cf0 con 0x7f90600c6bd0 2023-10-30T13:42:17.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90648b7640 1 -- 172.21.15.148:0/323607770 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c003530 con 0x7f90600c6bd0 2023-10-30T13:42:17.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90660ba640 1 -- 172.21.15.148:0/323607770 >> v2:172.21.15.150:3300/0 conn(0x7f906006f7a0 msgr2=0x7f906006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:17.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90660ba640 1 --2- 172.21.15.148:0/323607770 >> v2:172.21.15.150:3300/0 conn(0x7f906006f7a0 0x7f906006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90660ba640 1 -- 172.21.15.148:0/323607770 >> v1:172.21.15.148:6789/0 conn(0x7f90600c6bd0 legacy=0x7f9060069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90660ba640 1 -- 172.21.15.148:0/323607770 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90600c7390 con 0x7f9060071110 2023-10-30T13:42:17.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90658b9640 1 --2- 172.21.15.148:0/323607770 >> v2:172.21.15.150:3300/0 conn(0x7f906006f7a0 0x7f906006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:17.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90660ba640 1 --2- 172.21.15.148:0/323607770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9060071110 0x7f9060158110 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9058009d90 tx=0x7f905801f100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf7d248483dfc11f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:17.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90648b7640 1 -- 172.21.15.148:0/323607770 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90580b0440 con 0x7f9060071110 2023-10-30T13:42:17.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90648b7640 1 -- 172.21.15.148:0/323607770 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90580b05a0 con 0x7f9060071110 2023-10-30T13:42:17.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.080+0000 7f90648b7640 1 -- 172.21.15.148:0/323607770 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90580b0910 con 0x7f9060071110 2023-10-30T13:42:17.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 -- 172.21.15.148:0/323607770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9060071110 msgr2=0x7f9060158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 --2- 172.21.15.148:0/323607770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9060071110 0x7f9060158110 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9058009d90 tx=0x7f905801f100 comp rx=0 tx=0).stop 2023-10-30T13:42:17.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 -- 172.21.15.148:0/323607770 shutdown_connections 2023-10-30T13:42:17.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 --2- 172.21.15.148:0/323607770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9060071110 0x7f9060158110 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 --2- 172.21.15.148:0/323607770 >> v2:172.21.15.150:3300/0 conn(0x7f906006f7a0 0x7f906006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 -- 172.21.15.148:0/323607770 >> 172.21.15.148:0/323607770 conn(0x7f90600c7910 msgr2=0x7f90600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:17.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 -- 172.21.15.148:0/323607770 shutdown_connections 2023-10-30T13:42:17.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 -- 172.21.15.148:0/323607770 wait complete. 2023-10-30T13:42:17.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 Processor -- start 2023-10-30T13:42:17.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 -- start start 2023-10-30T13:42:17.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f906006f7a0 0x7f9060157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9060071110 0x7f906014ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90600c6bd0 0x7f906014f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90601e6780 con 0x7f9060071110 2023-10-30T13:42:17.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90600c37b0 con 0x7f906006f7a0 2023-10-30T13:42:17.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f9067b1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90600afb30 con 0x7f90600c6bd0 2023-10-30T13:42:17.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f90650b8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9060071110 0x7f906014ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f90650b8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9060071110 0x7f906014ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40938/0 (socket says 172.21.15.148:40938) 2023-10-30T13:42:17.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f90650b8640 1 -- 172.21.15.148:0/3730590768 learned_addr learned my addr 172.21.15.148:0/3730590768 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:17.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f90658b9640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f906006f7a0 0x7f9060157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.084+0000 7f90660ba640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90600c6bd0 0x7f906014f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90650b8640 1 -- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90600c6bd0 msgr2=0x7f906014f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90650b8640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90600c6bd0 0x7f906014f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90650b8640 1 -- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f906006f7a0 msgr2=0x7f9060157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90650b8640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f906006f7a0 0x7f9060157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90650b8640 1 -- 172.21.15.148:0/3730590768 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c009420 con 0x7f9060071110 2023-10-30T13:42:17.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90650b8640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9060071110 0x7f906014ef00 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f905000c4d0 tx=0x7f905000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:17.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90567fc640 1 -- 172.21.15.148:0/3730590768 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90500074e0 con 0x7f9060071110 2023-10-30T13:42:17.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90567fc640 1 -- 172.21.15.148:0/3730590768 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9050007640 con 0x7f9060071110 2023-10-30T13:42:17.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90567fc640 1 -- 172.21.15.148:0/3730590768 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9050004a30 con 0x7f9060071110 2023-10-30T13:42:17.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9058009a70 con 0x7f9060071110 2023-10-30T13:42:17.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90601576e0 con 0x7f9060071110 2023-10-30T13:42:17.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9060070330 con 0x7f9060071110 2023-10-30T13:42:17.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.088+0000 7f90567fc640 1 -- 172.21.15.148:0/3730590768 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9050004b90 con 0x7f9060071110 2023-10-30T13:42:17.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.092+0000 7f90567fc640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90440664a0 0x7f9044068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.092+0000 7f90567fc640 1 -- 172.21.15.148:0/3730590768 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f90500834e0 con 0x7f9060071110 2023-10-30T13:42:17.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.092+0000 7f90658b9640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90440664a0 0x7f9044068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.092+0000 7f90658b9640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90440664a0 0x7f9044068960 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f905c009c50 tx=0x7f905c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:17.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.096+0000 7f90567fc640 1 -- 172.21.15.148:0/3730590768 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9060070330 con 0x7f9060071110 2023-10-30T13:42:17.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.320+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90600c3960 con 0x7f9060071110 2023-10-30T13:42:17.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.320+0000 7f90567fc640 1 -- 172.21.15.148:0/3730590768 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f90600c3960 con 0x7f9060071110 2023-10-30T13:42:17.321 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:17.321 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:17.322 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 119 2023-10-30T13:42:17.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90440664a0 msgr2=0x7f9044068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90440664a0 0x7f9044068960 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f905c009c50 tx=0x7f905c009190 comp rx=0 tx=0).stop 2023-10-30T13:42:17.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9060071110 msgr2=0x7f906014ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9060071110 0x7f906014ef00 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f905000c4d0 tx=0x7f905000c990 comp rx=0 tx=0).stop 2023-10-30T13:42:17.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 shutdown_connections 2023-10-30T13:42:17.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f90440664a0 0x7f9044068960 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9060071110 0x7f906014ef00 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f90600c6bd0 0x7f906014f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 --2- 172.21.15.148:0/3730590768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f906006f7a0 0x7f9060157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 >> 172.21.15.148:0/3730590768 conn(0x7f90600c7910 msgr2=0x7f90600c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:17.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 shutdown_connections 2023-10-30T13:42:17.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.324+0000 7f9067b1d640 1 -- 172.21.15.148:0/3730590768 wait complete. 2023-10-30T13:42:17.343 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:42:17.343 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:42:17.344 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:42:17.389 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:42:17.390 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:17.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28cab1d640 1 Processor -- start 2023-10-30T13:42:17.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28cab1d640 1 -- start start 2023-10-30T13:42:17.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28cab1d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f28c41538a0 0x7f28c415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28cab1d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c40cb800 con 0x7f28c40cf160 2023-10-30T13:42:17.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28cab1d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c415dc30 con 0x7f28c40ca0e0 2023-10-30T13:42:17.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28cab1d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c40cb330 con 0x7f28c41538a0 2023-10-30T13:42:17.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28c90ba640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f28c40cf160 0x7f28c4164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51712/0 (socket says 172.21.15.148:51712) 2023-10-30T13:42:17.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28c90ba640 1 -- 172.21.15.148:0/2518197853 learned_addr learned my addr 172.21.15.148:0/2518197853 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:17.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28c88b9640 1 --2- 172.21.15.148:0/2518197853 >> v2:172.21.15.150:3300/0 conn(0x7f28c41538a0 0x7f28c415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28c37fe640 1 -- 172.21.15.148:0/2518197853 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868631165 0 0) 0x7f28c40cb800 con 0x7f28c40cf160 2023-10-30T13:42:17.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28c37fe640 1 -- 172.21.15.148:0/2518197853 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b4003530 con 0x7f28c40cf160 2023-10-30T13:42:17.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28c88b9640 1 -- 172.21.15.148:0/2518197853 >> v1:172.21.15.150:6789/0 conn(0x7f28c40ca0e0 legacy=0x7f28c415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28c88b9640 1 -- 172.21.15.148:0/2518197853 >> v1:172.21.15.148:6789/0 conn(0x7f28c40cf160 legacy=0x7f28c4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28c37fe640 1 -- 172.21.15.148:0/2518197853 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2091993209 0 0) 0x7f28b4003530 con 0x7f28c40cf160 2023-10-30T13:42:17.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.576+0000 7f28c88b9640 1 -- 172.21.15.148:0/2518197853 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c40cad70 con 0x7f28c41538a0 2023-10-30T13:42:17.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28c88b9640 1 --2- 172.21.15.148:0/2518197853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c41538a0 0x7f28c415d6f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f28b80097c0 tx=0x7f28b801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d2ddb4de67aae21 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:17.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28c37fe640 1 -- 172.21.15.148:0/2518197853 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc001a00 con 0x7f28c41538a0 2023-10-30T13:42:17.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28c37fe640 1 -- 172.21.15.148:0/2518197853 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b80a8610 con 0x7f28c41538a0 2023-10-30T13:42:17.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28c37fe640 1 -- 172.21.15.148:0/2518197853 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b80a89a0 con 0x7f28c41538a0 2023-10-30T13:42:17.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28cab1d640 1 -- 172.21.15.148:0/2518197853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c41538a0 msgr2=0x7f28c415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28cab1d640 1 --2- 172.21.15.148:0/2518197853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c41538a0 0x7f28c415d6f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f28b80097c0 tx=0x7f28b801eb90 comp rx=0 tx=0).stop 2023-10-30T13:42:17.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28cab1d640 1 -- 172.21.15.148:0/2518197853 shutdown_connections 2023-10-30T13:42:17.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28cab1d640 1 --2- 172.21.15.148:0/2518197853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c41538a0 0x7f28c415d6f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28cab1d640 1 -- 172.21.15.148:0/2518197853 >> 172.21.15.148:0/2518197853 conn(0x7f28c406bfe0 msgr2=0x7f28c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:17.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28cab1d640 1 -- 172.21.15.148:0/2518197853 shutdown_connections 2023-10-30T13:42:17.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.580+0000 7f28cab1d640 1 -- 172.21.15.148:0/2518197853 wait complete. 2023-10-30T13:42:17.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28cab1d640 1 Processor -- start 2023-10-30T13:42:17.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28cab1d640 1 -- start start 2023-10-30T13:42:17.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28cab1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c40ca0e0 0x7f28c4151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28cab1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c40cf160 0x7f28c4157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28cab1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c41538a0 0x7f28c4157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28cab1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c41effe0 con 0x7f28c40cf160 2023-10-30T13:42:17.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28cab1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c41ef3f0 con 0x7f28c40ca0e0 2023-10-30T13:42:17.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28cab1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c40b4960 con 0x7f28c41538a0 2023-10-30T13:42:17.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c40cf160 0x7f28c4157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c40cf160 0x7f28c4157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40946/0 (socket says 172.21.15.148:40946) 2023-10-30T13:42:17.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c3fff640 1 -- 172.21.15.148:0/1718180105 learned_addr learned my addr 172.21.15.148:0/1718180105 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:17.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c88b9640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c40ca0e0 0x7f28c4151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c90ba640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c41538a0 0x7f28c4157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c3fff640 1 -- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c41538a0 msgr2=0x7f28c4157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c3fff640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c41538a0 0x7f28c4157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c3fff640 1 -- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c40ca0e0 msgr2=0x7f28c4151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c3fff640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c40ca0e0 0x7f28c4151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c3fff640 1 -- 172.21.15.148:0/1718180105 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b8009420 con 0x7f28c40cf160 2023-10-30T13:42:17.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c88b9640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c40ca0e0 0x7f28c4151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:17.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.584+0000 7f28c3fff640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c40cf160 0x7f28c4157520 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f28b000c3c0 tx=0x7f28b000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:17.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28c17fa640 1 -- 172.21.15.148:0/1718180105 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b00052d0 con 0x7f28c40cf160 2023-10-30T13:42:17.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28c17fa640 1 -- 172.21.15.148:0/1718180105 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b0005430 con 0x7f28c40cf160 2023-10-30T13:42:17.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28c17fa640 1 -- 172.21.15.148:0/1718180105 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b0003840 con 0x7f28c40cf160 2023-10-30T13:42:17.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c41ef6b0 con 0x7f28c40cf160 2023-10-30T13:42:17.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c41efc70 con 0x7f28c40cf160 2023-10-30T13:42:17.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c41f0190 con 0x7f28c40cf160 2023-10-30T13:42:17.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28c17fa640 1 -- 172.21.15.148:0/1718180105 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f28b00039a0 con 0x7f28c40cf160 2023-10-30T13:42:17.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28c17fa640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28a80664d0 0x7f28a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:17.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28c17fa640 1 -- 172.21.15.148:0/1718180105 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f28b0083770 con 0x7f28c40cf160 2023-10-30T13:42:17.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.588+0000 7f28c88b9640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28a80664d0 0x7f28a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:17.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.592+0000 7f28c88b9640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28a80664d0 0x7f28a8068990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f28b80b2000 tx=0x7f28b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:17.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.592+0000 7f28c17fa640 1 -- 172.21.15.148:0/1718180105 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f28c41f0190 con 0x7f28c40cf160 2023-10-30T13:42:17.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.772+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28c415dc30 con 0x7f28c40cf160 2023-10-30T13:42:17.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28c17fa640 1 -- 172.21.15.148:0/1718180105 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f28c415dc30 con 0x7f28c40cf160 2023-10-30T13:42:17.775 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:17.775 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:17.775 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 119 2023-10-30T13:42:17.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28a80664d0 msgr2=0x7f28a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28a80664d0 0x7f28a8068990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f28b80b2000 tx=0x7f28b8008040 comp rx=0 tx=0).stop 2023-10-30T13:42:17.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c40cf160 msgr2=0x7f28c4157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:17.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c40cf160 0x7f28c4157520 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f28b000c3c0 tx=0x7f28b000c880 comp rx=0 tx=0).stop 2023-10-30T13:42:17.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 shutdown_connections 2023-10-30T13:42:17.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28a80664d0 0x7f28a8068990 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c40cf160 0x7f28c4157520 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c41538a0 0x7f28c4157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 --2- 172.21.15.148:0/1718180105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c40ca0e0 0x7f28c4151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:17.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 >> 172.21.15.148:0/1718180105 conn(0x7f28c406bfe0 msgr2=0x7f28c4145f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:17.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 shutdown_connections 2023-10-30T13:42:17.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:17.776+0000 7f28cab1d640 1 -- 172.21.15.148:0/1718180105 wait complete. 2023-10-30T13:42:17.796 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:42:19.798 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:19.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12e5371640 1 Processor -- start 2023-10-30T13:42:19.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12e5371640 1 -- start start 2023-10-30T13:42:19.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12e5371640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f12e01538f0 0x7f12e0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12e5371640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12e015cd50 0x7f12e00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12e5371640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e015d300 con 0x7f12e00cefa0 2023-10-30T13:42:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12e5371640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e015d5e0 con 0x7f12e01538f0 2023-10-30T13:42:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12e5371640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e015d8c0 con 0x7f12e015cd50 2023-10-30T13:42:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12dffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f12e01538f0 0x7f12e0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12dffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f12e01538f0 0x7f12e0159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40950/0 (socket says 172.21.15.148:40950) 2023-10-30T13:42:19.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12dffff640 1 -- 172.21.15.148:0/1864131594 learned_addr learned my addr 172.21.15.148:0/1864131594 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12df7fe640 1 --2- 172.21.15.148:0/1864131594 >> v2:172.21.15.150:3300/0 conn(0x7f12e015cd50 0x7f12e00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12dffff640 1 -- 172.21.15.148:0/1864131594 >> v2:172.21.15.150:3300/0 conn(0x7f12e015cd50 msgr2=0x7f12e00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12dffff640 1 --2- 172.21.15.148:0/1864131594 >> v2:172.21.15.150:3300/0 conn(0x7f12e015cd50 0x7f12e00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12dffff640 1 -- 172.21.15.148:0/1864131594 >> v1:172.21.15.150:6789/0 conn(0x7f12e00cefa0 legacy=0x7f12e00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:19.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.948+0000 7f12dffff640 1 -- 172.21.15.148:0/1864131594 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e015dba0 con 0x7f12e01538f0 2023-10-30T13:42:19.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12dffff640 1 --2- 172.21.15.148:0/1864131594 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e01538f0 0x7f12e0159fb0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f12e0072e90 tx=0x7f12d001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae2cc1a5402f994 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:19.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12deffd640 1 -- 172.21.15.148:0/1864131594 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d00a89f0 con 0x7f12e01538f0 2023-10-30T13:42:19.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12deffd640 1 -- 172.21.15.148:0/1864131594 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d00a8b50 con 0x7f12e01538f0 2023-10-30T13:42:19.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12deffd640 1 -- 172.21.15.148:0/1864131594 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d00b0440 con 0x7f12e01538f0 2023-10-30T13:42:19.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12e5371640 1 -- 172.21.15.148:0/1864131594 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e01538f0 msgr2=0x7f12e0159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:19.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12e5371640 1 --2- 172.21.15.148:0/1864131594 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e01538f0 0x7f12e0159fb0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f12e0072e90 tx=0x7f12d001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12e5371640 1 -- 172.21.15.148:0/1864131594 shutdown_connections 2023-10-30T13:42:19.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12e5371640 1 --2- 172.21.15.148:0/1864131594 >> v2:172.21.15.150:3300/0 conn(0x7f12e015cd50 0x7f12e00c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12e5371640 1 --2- 172.21.15.148:0/1864131594 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e01538f0 0x7f12e0159fb0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12e5371640 1 -- 172.21.15.148:0/1864131594 >> 172.21.15.148:0/1864131594 conn(0x7f12e006a640 msgr2=0x7f12e0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:19.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.952+0000 7f12e5371640 1 -- 172.21.15.148:0/1864131594 shutdown_connections 2023-10-30T13:42:19.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e5371640 1 -- 172.21.15.148:0/1864131594 wait complete. 2023-10-30T13:42:19.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e5371640 1 Processor -- start 2023-10-30T13:42:19.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e5371640 1 -- start start 2023-10-30T13:42:19.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e5371640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12e00cefa0 0x7f12e01480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e5371640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e01538f0 0x7f12e01485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e5371640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e015cd50 0x7f12e014daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e5371640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e015af00 con 0x7f12e015cd50 2023-10-30T13:42:19.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e5371640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e015b4c0 con 0x7f12e01538f0 2023-10-30T13:42:19.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e5371640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e015ab20 con 0x7f12e00cefa0 2023-10-30T13:42:19.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e4b70640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e015cd50 0x7f12e014daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e4b70640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e015cd50 0x7f12e014daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40966/0 (socket says 172.21.15.148:40966) 2023-10-30T13:42:19.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e4b70640 1 -- 172.21.15.148:0/2549278334 learned_addr learned my addr 172.21.15.148:0/2549278334 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:19.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12dffff640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12e00cefa0 0x7f12e01480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12df7fe640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e01538f0 0x7f12e01485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e4b70640 1 -- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12e00cefa0 msgr2=0x7f12e01480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:19.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e4b70640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12e00cefa0 0x7f12e01480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e4b70640 1 -- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e01538f0 msgr2=0x7f12e01485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:19.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e4b70640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e01538f0 0x7f12e01485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:19.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e4b70640 1 -- 172.21.15.148:0/2549278334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d00094d0 con 0x7f12e015cd50 2023-10-30T13:42:19.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12df7fe640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e01538f0 0x7f12e01485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:19.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.956+0000 7f12e4b70640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e015cd50 0x7f12e014daa0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f12d800c7a0 tx=0x7f12d800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:19.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12dcff9640 1 -- 172.21.15.148:0/2549278334 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d8003b80 con 0x7f12e015cd50 2023-10-30T13:42:19.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12dcff9640 1 -- 172.21.15.148:0/2549278334 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d8003ce0 con 0x7f12e015cd50 2023-10-30T13:42:19.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12dcff9640 1 -- 172.21.15.148:0/2549278334 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d800aa90 con 0x7f12e015cd50 2023-10-30T13:42:19.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e014fa30 con 0x7f12e015cd50 2023-10-30T13:42:19.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e015b150 con 0x7f12e015cd50 2023-10-30T13:42:19.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e015b670 con 0x7f12e015cd50 2023-10-30T13:42:19.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12dcff9640 1 -- 172.21.15.148:0/2549278334 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12d800abf0 con 0x7f12e015cd50 2023-10-30T13:42:19.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12dcff9640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12cc0664a0 0x7f12cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:19.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12dcff9640 1 -- 172.21.15.148:0/2549278334 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f12d8084170 con 0x7f12e015cd50 2023-10-30T13:42:19.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.960+0000 7f12dffff640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12cc0664a0 0x7f12cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:19.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.964+0000 7f12dffff640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12cc0664a0 0x7f12cc068960 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f12d00b2000 tx=0x7f12d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:19.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:19.964+0000 7f12dcff9640 1 -- 172.21.15.148:0/2549278334 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12e015b670 con 0x7f12e015cd50 2023-10-30T13:42:20.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.144+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12e015b630 con 0x7f12e015cd50 2023-10-30T13:42:20.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12dcff9640 1 -- 172.21.15.148:0/2549278334 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f12d80510e0 con 0x7f12e015cd50 2023-10-30T13:42:20.146 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:20.147 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:20.147 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 119 2023-10-30T13:42:20.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12cc0664a0 msgr2=0x7f12cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:20.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12cc0664a0 0x7f12cc068960 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f12d00b2000 tx=0x7f12d0009040 comp rx=0 tx=0).stop 2023-10-30T13:42:20.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e015cd50 msgr2=0x7f12e014daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:20.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e015cd50 0x7f12e014daa0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f12d800c7a0 tx=0x7f12d800cc60 comp rx=0 tx=0).stop 2023-10-30T13:42:20.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 shutdown_connections 2023-10-30T13:42:20.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f12cc0664a0 0x7f12cc068960 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f12e015cd50 0x7f12e014daa0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12e01538f0 0x7f12e01485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 --2- 172.21.15.148:0/2549278334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f12e00cefa0 0x7f12e01480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:20.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 >> 172.21.15.148:0/2549278334 conn(0x7f12e006a640 msgr2=0x7f12e00caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:20.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 shutdown_connections 2023-10-30T13:42:20.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:20.148+0000 7f12e5371640 1 -- 172.21.15.148:0/2549278334 wait complete. 2023-10-30T13:42:20.168 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:42:22.169 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:22.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.316+0000 7f4c9abc5640 1 Processor -- start 2023-10-30T13:42:22.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.316+0000 7f4c9abc5640 1 -- start start 2023-10-30T13:42:22.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c9abc5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f4c94070740 0x7f4c940c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c9abc5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 0x7f4c9406a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c9abc5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c940c5810 con 0x7f4c941580e0 2023-10-30T13:42:22.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c9abc5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9415eb90 con 0x7f4c94070740 2023-10-30T13:42:22.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c9abc5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c940c6bb0 con 0x7f4c9415e700 2023-10-30T13:42:22.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c98961640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4c941580e0 0x7f4c94069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37338/0 (socket says 172.21.15.148:37338) 2023-10-30T13:42:22.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c98961640 1 -- 172.21.15.148:0/3937595215 learned_addr learned my addr 172.21.15.148:0/3937595215 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:22.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c93fff640 1 --2- 172.21.15.148:0/3937595215 >> v2:172.21.15.148:3300/0 conn(0x7f4c94070740 0x7f4c940c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c99162640 1 --2- 172.21.15.148:0/3937595215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 0x7f4c9406a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c93fff640 1 -- 172.21.15.148:0/3937595215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 msgr2=0x7f4c9406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:22.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c93fff640 1 --2- 172.21.15.148:0/3937595215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 0x7f4c9406a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c93fff640 1 -- 172.21.15.148:0/3937595215 >> v1:172.21.15.148:6789/0 conn(0x7f4c941580e0 legacy=0x7f4c94069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.320+0000 7f4c93fff640 1 -- 172.21.15.148:0/3937595215 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c940c6e70 con 0x7f4c94070740 2023-10-30T13:42:22.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c93fff640 1 --2- 172.21.15.148:0/3937595215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 0x7f4c940c6540 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f4c800097c0 tx=0x7f4c8001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d457c8e9cd333bf1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:22.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c937fe640 1 -- 172.21.15.148:0/3937595215 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c800ab600 con 0x7f4c94070740 2023-10-30T13:42:22.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c937fe640 1 -- 172.21.15.148:0/3937595215 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c800a7050 con 0x7f4c94070740 2023-10-30T13:42:22.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c937fe640 1 -- 172.21.15.148:0/3937595215 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c800ab980 con 0x7f4c94070740 2023-10-30T13:42:22.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 -- 172.21.15.148:0/3937595215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 msgr2=0x7f4c940c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 --2- 172.21.15.148:0/3937595215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 0x7f4c940c6540 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f4c800097c0 tx=0x7f4c8001c480 comp rx=0 tx=0).stop 2023-10-30T13:42:22.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 -- 172.21.15.148:0/3937595215 shutdown_connections 2023-10-30T13:42:22.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 --2- 172.21.15.148:0/3937595215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 0x7f4c940c6540 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 --2- 172.21.15.148:0/3937595215 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 0x7f4c9406a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 -- 172.21.15.148:0/3937595215 >> 172.21.15.148:0/3937595215 conn(0x7f4c940c7910 msgr2=0x7f4c940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:22.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 -- 172.21.15.148:0/3937595215 shutdown_connections 2023-10-30T13:42:22.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 -- 172.21.15.148:0/3937595215 wait complete. 2023-10-30T13:42:22.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 Processor -- start 2023-10-30T13:42:22.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.324+0000 7f4c9abc5640 1 -- start start 2023-10-30T13:42:22.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c9abc5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 0x7f4c94161700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c9abc5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c941580e0 0x7f4c941f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c9abc5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 0x7f4c940c0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c9abc5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c94160d20 con 0x7f4c94070740 2023-10-30T13:42:22.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c9abc5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c94153e30 con 0x7f4c941580e0 2023-10-30T13:42:22.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c9abc5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941536d0 con 0x7f4c9415e700 2023-10-30T13:42:22.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c98961640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 0x7f4c94161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c98961640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 0x7f4c94161700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43286/0 (socket says 172.21.15.148:43286) 2023-10-30T13:42:22.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c98961640 1 -- 172.21.15.148:0/1749037121 learned_addr learned my addr 172.21.15.148:0/1749037121 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:22.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c99162640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 0x7f4c940c0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c93fff640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c941580e0 0x7f4c941f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c98961640 1 -- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 msgr2=0x7f4c940c0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c98961640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 0x7f4c940c0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c98961640 1 -- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c941580e0 msgr2=0x7f4c941f6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c98961640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c941580e0 0x7f4c941f6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c98961640 1 -- 172.21.15.148:0/1749037121 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c80009420 con 0x7f4c94070740 2023-10-30T13:42:22.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c98961640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 0x7f4c94161700 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800c530 tx=0x7f4c8800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:22.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c917fa640 1 -- 172.21.15.148:0/1749037121 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88005cf0 con 0x7f4c94070740 2023-10-30T13:42:22.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c917fa640 1 -- 172.21.15.148:0/1749037121 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c88005e50 con 0x7f4c94070740 2023-10-30T13:42:22.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c94152b50 con 0x7f4c94070740 2023-10-30T13:42:22.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c917fa640 1 -- 172.21.15.148:0/1749037121 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88007270 con 0x7f4c94070740 2023-10-30T13:42:22.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c94153110 con 0x7f4c94070740 2023-10-30T13:42:22.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.328+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9404e170 con 0x7f4c94070740 2023-10-30T13:42:22.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.332+0000 7f4c917fa640 1 -- 172.21.15.148:0/1749037121 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4c880073d0 con 0x7f4c94070740 2023-10-30T13:42:22.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.332+0000 7f4c917fa640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c780664a0 0x7f4c78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:22.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.332+0000 7f4c917fa640 1 -- 172.21.15.148:0/1749037121 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f4c88083870 con 0x7f4c94070740 2023-10-30T13:42:22.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.332+0000 7f4c93fff640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c780664a0 0x7f4c78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:22.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.332+0000 7f4c93fff640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c780664a0 0x7f4c78068960 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4c800ad000 tx=0x7f4c80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:22.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.336+0000 7f4c917fa640 1 -- 172.21.15.148:0/1749037121 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4c880507f0 con 0x7f4c94070740 2023-10-30T13:42:22.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.516+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c9406aed0 con 0x7f4c94070740 2023-10-30T13:42:22.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.516+0000 7f4c917fa640 1 -- 172.21.15.148:0/1749037121 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f4c88053e10 con 0x7f4c94070740 2023-10-30T13:42:22.517 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:22.518 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:22.518 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 119 2023-10-30T13:42:22.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c780664a0 msgr2=0x7f4c78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c780664a0 0x7f4c78068960 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4c800ad000 tx=0x7f4c80008040 comp rx=0 tx=0).stop 2023-10-30T13:42:22.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 msgr2=0x7f4c94161700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:22.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 0x7f4c94161700 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800c530 tx=0x7f4c8800c9f0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 shutdown_connections 2023-10-30T13:42:22.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f4c780664a0 0x7f4c78068960 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4c9415e700 0x7f4c940c0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c941580e0 0x7f4c941f6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 --2- 172.21.15.148:0/1749037121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c94070740 0x7f4c94161700 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:22.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 >> 172.21.15.148:0/1749037121 conn(0x7f4c940c7910 msgr2=0x7f4c9414aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:22.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 shutdown_connections 2023-10-30T13:42:22.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:22.520+0000 7f4c9abc5640 1 -- 172.21.15.148:0/1749037121 wait complete. 2023-10-30T13:42:22.538 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:42:24.539 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:24.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b9512a640 1 Processor -- start 2023-10-30T13:42:24.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b9512a640 1 -- start start 2023-10-30T13:42:24.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b9512a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b901538a0 0x7f0b9015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b9512a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0b900ca0e0 0x7f0b9015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b9512a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b900cb800 con 0x7f0b900cf160 2023-10-30T13:42:24.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b9512a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9015dc30 con 0x7f0b900ca0e0 2023-10-30T13:42:24.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b9512a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b900cb330 con 0x7f0b901538a0 2023-10-30T13:42:24.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b8659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0b900ca0e0 0x7f0b9015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b8659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0b900ca0e0 0x7f0b9015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43296/0 (socket says 172.21.15.148:43296) 2023-10-30T13:42:24.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b8659c640 1 -- 172.21.15.148:0/3285468921 learned_addr learned my addr 172.21.15.148:0/3285468921 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:24.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b8ed9d640 1 --2- 172.21.15.148:0/3285468921 >> v2:172.21.15.150:3300/0 conn(0x7f0b901538a0 0x7f0b9015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b8659c640 1 -- 172.21.15.148:0/3285468921 >> v2:172.21.15.150:3300/0 conn(0x7f0b901538a0 msgr2=0x7f0b9015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:24.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b8659c640 1 --2- 172.21.15.148:0/3285468921 >> v2:172.21.15.150:3300/0 conn(0x7f0b901538a0 0x7f0b9015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b8659c640 1 -- 172.21.15.148:0/3285468921 >> v1:172.21.15.150:6789/0 conn(0x7f0b900cf160 legacy=0x7f0b900cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.688+0000 7f0b8659c640 1 -- 172.21.15.148:0/3285468921 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b900cad70 con 0x7f0b900ca0e0 2023-10-30T13:42:24.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b8659c640 1 --2- 172.21.15.148:0/3285468921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 0x7f0b9015a090 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f0b900cfd10 tx=0x7f0b8001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fd25c5bd819157 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:24.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b8e59c640 1 -- 172.21.15.148:0/3285468921 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b800a89f0 con 0x7f0b900ca0e0 2023-10-30T13:42:24.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b8e59c640 1 -- 172.21.15.148:0/3285468921 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b800a8b50 con 0x7f0b900ca0e0 2023-10-30T13:42:24.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b8e59c640 1 -- 172.21.15.148:0/3285468921 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b800b0440 con 0x7f0b900ca0e0 2023-10-30T13:42:24.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b9512a640 1 -- 172.21.15.148:0/3285468921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 msgr2=0x7f0b9015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b9512a640 1 --2- 172.21.15.148:0/3285468921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 0x7f0b9015a090 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f0b900cfd10 tx=0x7f0b8001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b9512a640 1 -- 172.21.15.148:0/3285468921 shutdown_connections 2023-10-30T13:42:24.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b9512a640 1 --2- 172.21.15.148:0/3285468921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 0x7f0b9015a090 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b9512a640 1 --2- 172.21.15.148:0/3285468921 >> v2:172.21.15.150:3300/0 conn(0x7f0b901538a0 0x7f0b9015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b9512a640 1 -- 172.21.15.148:0/3285468921 >> 172.21.15.148:0/3285468921 conn(0x7f0b9006bfe0 msgr2=0x7f0b9006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:24.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b9512a640 1 -- 172.21.15.148:0/3285468921 shutdown_connections 2023-10-30T13:42:24.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.692+0000 7f0b9512a640 1 -- 172.21.15.148:0/3285468921 wait complete. 2023-10-30T13:42:24.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 Processor -- start 2023-10-30T13:42:24.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 -- start start 2023-10-30T13:42:24.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 0x7f0b901ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b900cf160 0x7f0b900cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b901538a0 0x7f0b901585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b900b3110 con 0x7f0b900ca0e0 2023-10-30T13:42:24.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b901efab0 con 0x7f0b901538a0 2023-10-30T13:42:24.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b900c4420 con 0x7f0b900cf160 2023-10-30T13:42:24.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 0x7f0b901ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 0x7f0b901ef570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43304/0 (socket says 172.21.15.148:43304) 2023-10-30T13:42:24.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8ed9d640 1 -- 172.21.15.148:0/76232796 learned_addr learned my addr 172.21.15.148:0/76232796 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:24.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8f59e640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b901538a0 0x7f0b901585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8659c640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b900cf160 0x7f0b900cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8ed9d640 1 -- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b900cf160 msgr2=0x7f0b900cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8ed9d640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b900cf160 0x7f0b900cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8ed9d640 1 -- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b901538a0 msgr2=0x7f0b901585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8ed9d640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b901538a0 0x7f0b901585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8ed9d640 1 -- 172.21.15.148:0/76232796 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b800094d0 con 0x7f0b900ca0e0 2023-10-30T13:42:24.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8f59e640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b901538a0 0x7f0b901585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:24.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b8ed9d640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 0x7f0b901ef570 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800d5f0 tx=0x7f0b7800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:24.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b87fff640 1 -- 172.21.15.148:0/76232796 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b780125f0 con 0x7f0b900ca0e0 2023-10-30T13:42:24.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b87fff640 1 -- 172.21.15.148:0/76232796 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b78012750 con 0x7f0b900ca0e0 2023-10-30T13:42:24.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b87fff640 1 -- 172.21.15.148:0/76232796 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78012ae0 con 0x7f0b900ca0e0 2023-10-30T13:42:24.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b901efd20 con 0x7f0b900ca0e0 2023-10-30T13:42:24.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.696+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b901f02c0 con 0x7f0b900ca0e0 2023-10-30T13:42:24.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.700+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b900b2a50 con 0x7f0b900ca0e0 2023-10-30T13:42:24.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.700+0000 7f0b87fff640 1 -- 172.21.15.148:0/76232796 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b78012c40 con 0x7f0b900ca0e0 2023-10-30T13:42:24.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.700+0000 7f0b87fff640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b700664d0 0x7f0b70068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:24.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.700+0000 7f0b87fff640 1 -- 172.21.15.148:0/76232796 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f0b78084960 con 0x7f0b900ca0e0 2023-10-30T13:42:24.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.700+0000 7f0b8659c640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b700664d0 0x7f0b70068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:24.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.700+0000 7f0b8659c640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b700664d0 0x7f0b70068990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0b800b2000 tx=0x7f0b80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:24.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.704+0000 7f0b87fff640 1 -- 172.21.15.148:0/76232796 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b780517a0 con 0x7f0b900ca0e0 2023-10-30T13:42:24.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.884+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b90159860 con 0x7f0b900ca0e0 2023-10-30T13:42:24.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.884+0000 7f0b87fff640 1 -- 172.21.15.148:0/76232796 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f0b78007380 con 0x7f0b900ca0e0 2023-10-30T13:42:24.885 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:24.886 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:24.887 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 119 2023-10-30T13:42:24.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b700664d0 msgr2=0x7f0b70068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b700664d0 0x7f0b70068990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0b800b2000 tx=0x7f0b80008040 comp rx=0 tx=0).stop 2023-10-30T13:42:24.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 msgr2=0x7f0b901ef570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:24.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 0x7f0b901ef570 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800d5f0 tx=0x7f0b7800dab0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 shutdown_connections 2023-10-30T13:42:24.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0b700664d0 0x7f0b70068990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b901538a0 0x7f0b901585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b900cf160 0x7f0b900cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 --2- 172.21.15.148:0/76232796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b900ca0e0 0x7f0b901ef570 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:24.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 >> 172.21.15.148:0/76232796 conn(0x7f0b9006bfe0 msgr2=0x7f0b90148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:24.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 shutdown_connections 2023-10-30T13:42:24.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:24.888+0000 7f0b9512a640 1 -- 172.21.15.148:0/76232796 wait complete. 2023-10-30T13:42:24.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:42:26.960 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:27.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.112+0000 7f0865809640 1 Processor -- start 2023-10-30T13:42:27.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.112+0000 7f0865809640 1 -- start start 2023-10-30T13:42:27.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.112+0000 7f0865809640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f086006f7a0 0x7f086006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:27.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.112+0000 7f0865809640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0860066cf0 con 0x7f0860071110 2023-10-30T13:42:27.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.112+0000 7f0865809640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08600c70d0 con 0x7f08600c6bd0 2023-10-30T13:42:27.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.112+0000 7f0865809640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08600700c0 con 0x7f086006f7a0 2023-10-30T13:42:27.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.112+0000 7f085effd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f086006f7a0 0x7f086006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:27.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.112+0000 7f085effd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f086006f7a0 0x7f086006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43310/0 (socket says 172.21.15.148:43310) 2023-10-30T13:42:27.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.112+0000 7f085effd640 1 -- 172.21.15.148:0/2654356906 learned_addr learned my addr 172.21.15.148:0/2654356906 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:27.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.116+0000 7f085effd640 1 -- 172.21.15.148:0/2654356906 >> v1:172.21.15.150:6789/0 conn(0x7f08600c6bd0 legacy=0x7f0860069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.116+0000 7f085effd640 1 -- 172.21.15.148:0/2654356906 >> v1:172.21.15.148:6789/0 conn(0x7f0860071110 legacy=0x7f0860158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.116+0000 7f085effd640 1 -- 172.21.15.148:0/2654356906 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600c7390 con 0x7f086006f7a0 2023-10-30T13:42:27.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.116+0000 7f085effd640 1 --2- 172.21.15.148:0/2654356906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086006f7a0 0x7f086006fb80 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0850009870 tx=0x7f085001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d346580a121b06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:27.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.116+0000 7f085dffb640 1 -- 172.21.15.148:0/2654356906 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500a8a30 con 0x7f086006f7a0 2023-10-30T13:42:27.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.116+0000 7f085dffb640 1 -- 172.21.15.148:0/2654356906 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08500a8b90 con 0x7f086006f7a0 2023-10-30T13:42:27.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.116+0000 7f085dffb640 1 -- 172.21.15.148:0/2654356906 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500b0440 con 0x7f086006f7a0 2023-10-30T13:42:27.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.116+0000 7f0865809640 1 -- 172.21.15.148:0/2654356906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086006f7a0 msgr2=0x7f086006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.116+0000 7f0865809640 1 --2- 172.21.15.148:0/2654356906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086006f7a0 0x7f086006fb80 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0850009870 tx=0x7f085001ed30 comp rx=0 tx=0).stop 2023-10-30T13:42:27.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 -- 172.21.15.148:0/2654356906 shutdown_connections 2023-10-30T13:42:27.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 --2- 172.21.15.148:0/2654356906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f086006f7a0 0x7f086006fb80 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 -- 172.21.15.148:0/2654356906 >> 172.21.15.148:0/2654356906 conn(0x7f08600c7910 msgr2=0x7f08600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:27.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 -- 172.21.15.148:0/2654356906 shutdown_connections 2023-10-30T13:42:27.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 -- 172.21.15.148:0/2654356906 wait complete. 2023-10-30T13:42:27.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 Processor -- start 2023-10-30T13:42:27.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 -- start start 2023-10-30T13:42:27.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f086006f7a0 0x7f08601464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:27.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0860071110 0x7f08601e2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:27.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08600c6bd0 0x7f0860140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:27.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08601f2620 con 0x7f08600c6bd0 2023-10-30T13:42:27.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0860153ee0 con 0x7f086006f7a0 2023-10-30T13:42:27.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0865809640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0860154150 con 0x7f0860071110 2023-10-30T13:42:27.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08600c6bd0 0x7f0860140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:27.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08600c6bd0 0x7f0860140af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43322/0 (socket says 172.21.15.148:43322) 2023-10-30T13:42:27.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085f7fe640 1 -- 172.21.15.148:0/501161658 learned_addr learned my addr 172.21.15.148:0/501161658 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:27.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085e7fc640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0860071110 0x7f08601e2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:27.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085effd640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f086006f7a0 0x7f08601464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:27.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085f7fe640 1 -- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0860071110 msgr2=0x7f08601e2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085f7fe640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0860071110 0x7f08601e2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085f7fe640 1 -- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f086006f7a0 msgr2=0x7f08601464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085f7fe640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f086006f7a0 0x7f08601464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085f7fe640 1 -- 172.21.15.148:0/501161658 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500094d0 con 0x7f08600c6bd0 2023-10-30T13:42:27.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f085f7fe640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08600c6bd0 0x7f0860140af0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f085800c730 tx=0x7f085800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:27.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0847fff640 1 -- 172.21.15.148:0/501161658 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0858003b80 con 0x7f08600c6bd0 2023-10-30T13:42:27.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.120+0000 7f0847fff640 1 -- 172.21.15.148:0/501161658 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0858003ce0 con 0x7f08600c6bd0 2023-10-30T13:42:27.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.124+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0860154430 con 0x7f08600c6bd0 2023-10-30T13:42:27.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.124+0000 7f0847fff640 1 -- 172.21.15.148:0/501161658 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085800aa30 con 0x7f08600c6bd0 2023-10-30T13:42:27.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.124+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08601e3570 con 0x7f08600c6bd0 2023-10-30T13:42:27.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.124+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08601f2790 con 0x7f08600c6bd0 2023-10-30T13:42:27.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.124+0000 7f0847fff640 1 -- 172.21.15.148:0/501161658 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f085800ab90 con 0x7f08600c6bd0 2023-10-30T13:42:27.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.124+0000 7f0847fff640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f084c0664a0 0x7f084c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:27.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.124+0000 7f0847fff640 1 -- 172.21.15.148:0/501161658 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f0858084370 con 0x7f08600c6bd0 2023-10-30T13:42:27.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.124+0000 7f085effd640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f084c0664a0 0x7f084c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:27.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.128+0000 7f085effd640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f084c0664a0 0x7f084c068960 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f08500b2000 tx=0x7f0850008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:27.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.128+0000 7f0847fff640 1 -- 172.21.15.148:0/501161658 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0858051120 con 0x7f08600c6bd0 2023-10-30T13:42:27.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.312+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f086004e170 con 0x7f08600c6bd0 2023-10-30T13:42:27.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.312+0000 7f0847fff640 1 -- 172.21.15.148:0/501161658 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f0858054740 con 0x7f08600c6bd0 2023-10-30T13:42:27.314 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:27.314 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:27.314 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 119 2023-10-30T13:42:27.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f084c0664a0 msgr2=0x7f084c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f084c0664a0 0x7f084c068960 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f08500b2000 tx=0x7f0850008040 comp rx=0 tx=0).stop 2023-10-30T13:42:27.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08600c6bd0 msgr2=0x7f0860140af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:27.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08600c6bd0 0x7f0860140af0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f085800c730 tx=0x7f085800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 shutdown_connections 2023-10-30T13:42:27.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f084c0664a0 0x7f084c068960 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08600c6bd0 0x7f0860140af0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0860071110 0x7f08601e2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 --2- 172.21.15.148:0/501161658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f086006f7a0 0x7f08601464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:27.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 >> 172.21.15.148:0/501161658 conn(0x7f08600c7910 msgr2=0x7f08600c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:27.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 shutdown_connections 2023-10-30T13:42:27.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:27.316+0000 7f0865809640 1 -- 172.21.15.148:0/501161658 wait complete. 2023-10-30T13:42:27.336 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:42:29.337 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:29.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.484+0000 7f888be91640 1 Processor -- start 2023-10-30T13:42:29.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.484+0000 7f888be91640 1 -- start start 2023-10-30T13:42:29.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f888be91640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f888406f7a0 0x7f888406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f888be91640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8884071110 0x7f8884158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f888be91640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8884066cf0 con 0x7f88840c6bd0 2023-10-30T13:42:29.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f888be91640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88840700c0 con 0x7f888406f7a0 2023-10-30T13:42:29.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f888be91640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88840c70d0 con 0x7f8884071110 2023-10-30T13:42:29.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f8889c2d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f888406f7a0 0x7f888406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f8889c2d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f888406f7a0 0x7f888406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43332/0 (socket says 172.21.15.148:43332) 2023-10-30T13:42:29.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f8889c2d640 1 -- 172.21.15.148:0/1317114234 learned_addr learned my addr 172.21.15.148:0/1317114234 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:29.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f888a42e640 1 --2- 172.21.15.148:0/1317114234 >> v2:172.21.15.150:3300/0 conn(0x7f8884071110 0x7f8884158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f8889c2d640 1 -- 172.21.15.148:0/1317114234 >> v2:172.21.15.150:3300/0 conn(0x7f8884071110 msgr2=0x7f8884158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:29.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f8889c2d640 1 --2- 172.21.15.148:0/1317114234 >> v2:172.21.15.150:3300/0 conn(0x7f8884071110 0x7f8884158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f8889c2d640 1 -- 172.21.15.148:0/1317114234 >> v1:172.21.15.148:6789/0 conn(0x7f88840c6bd0 legacy=0x7f8884069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.488+0000 7f8889c2d640 1 -- 172.21.15.148:0/1317114234 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88840c7390 con 0x7f888406f7a0 2023-10-30T13:42:29.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f8889c2d640 1 --2- 172.21.15.148:0/1317114234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f888406f7a0 0x7f888406fb80 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f888406cdd0 tx=0x7f888001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5011b4c7b4a1b0e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:29.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f8888c2b640 1 -- 172.21.15.148:0/1317114234 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800a89f0 con 0x7f888406f7a0 2023-10-30T13:42:29.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f8888c2b640 1 -- 172.21.15.148:0/1317114234 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88800a8b50 con 0x7f888406f7a0 2023-10-30T13:42:29.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f8888c2b640 1 -- 172.21.15.148:0/1317114234 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800b0440 con 0x7f888406f7a0 2023-10-30T13:42:29.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 -- 172.21.15.148:0/1317114234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f888406f7a0 msgr2=0x7f888406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 --2- 172.21.15.148:0/1317114234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f888406f7a0 0x7f888406fb80 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f888406cdd0 tx=0x7f888001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 -- 172.21.15.148:0/1317114234 shutdown_connections 2023-10-30T13:42:29.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 --2- 172.21.15.148:0/1317114234 >> v2:172.21.15.150:3300/0 conn(0x7f8884071110 0x7f8884158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 --2- 172.21.15.148:0/1317114234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f888406f7a0 0x7f888406fb80 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 -- 172.21.15.148:0/1317114234 >> 172.21.15.148:0/1317114234 conn(0x7f88840c7910 msgr2=0x7f88840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:29.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 -- 172.21.15.148:0/1317114234 shutdown_connections 2023-10-30T13:42:29.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 -- 172.21.15.148:0/1317114234 wait complete. 2023-10-30T13:42:29.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 Processor -- start 2023-10-30T13:42:29.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.492+0000 7f888be91640 1 -- start start 2023-10-30T13:42:29.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888be91640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888406f7a0 0x7f88841528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888be91640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8884071110 0x7f88841f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888be91640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6bd0 0x7f88841efa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888be91640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88841f3470 con 0x7f8884071110 2023-10-30T13:42:29.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888be91640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88841eff70 con 0x7f888406f7a0 2023-10-30T13:42:29.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888be91640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88841ef5a0 con 0x7f88840c6bd0 2023-10-30T13:42:29.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888942c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8884071110 0x7f88841f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888942c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8884071110 0x7f88841f2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43334/0 (socket says 172.21.15.148:43334) 2023-10-30T13:42:29.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888942c640 1 -- 172.21.15.148:0/845311990 learned_addr learned my addr 172.21.15.148:0/845311990 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:29.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f8889c2d640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888406f7a0 0x7f88841528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888a42e640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6bd0 0x7f88841efa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888942c640 1 -- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6bd0 msgr2=0x7f88841efa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888942c640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6bd0 0x7f88841efa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888942c640 1 -- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888406f7a0 msgr2=0x7f88841528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888942c640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888406f7a0 0x7f88841528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888942c640 1 -- 172.21.15.148:0/845311990 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88800094d0 con 0x7f8884071110 2023-10-30T13:42:29.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888942c640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8884071110 0x7f88841f2f30 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f887400c4d0 tx=0x7f887400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f887a7fc640 1 -- 172.21.15.148:0/845311990 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88740052d0 con 0x7f8884071110 2023-10-30T13:42:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f887a7fc640 1 -- 172.21.15.148:0/845311990 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8874005430 con 0x7f8884071110 2023-10-30T13:42:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88841eefb0 con 0x7f8884071110 2023-10-30T13:42:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f887a7fc640 1 -- 172.21.15.148:0/845311990 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874003800 con 0x7f8884071110 2023-10-30T13:42:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88840c1a60 con 0x7f8884071110 2023-10-30T13:42:29.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888406acf0 con 0x7f8884071110 2023-10-30T13:42:29.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.496+0000 7f887a7fc640 1 -- 172.21.15.148:0/845311990 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8874003960 con 0x7f8884071110 2023-10-30T13:42:29.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.500+0000 7f887a7fc640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664d0 0x7f8868068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:29.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.500+0000 7f887a7fc640 1 -- 172.21.15.148:0/845311990 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7144+0+0 (secure 0 0 0) 0x7f88740838b0 con 0x7f8884071110 2023-10-30T13:42:29.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.500+0000 7f8889c2d640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664d0 0x7f8868068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:29.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.500+0000 7f8889c2d640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664d0 0x7f8868068990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f88800b2000 tx=0x7f8880009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:29.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.504+0000 7f887a7fc640 1 -- 172.21.15.148:0/845311990 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88740507b0 con 0x7f8884071110 2023-10-30T13:42:29.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.684+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f888404e170 con 0x7f8884071110 2023-10-30T13:42:29.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.684+0000 7f887a7fc640 1 -- 172.21.15.148:0/845311990 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+6512 (secure 0 0 0) 0x7f8874017090 con 0x7f8884071110 2023-10-30T13:42:29.686 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:29.686 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":119}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:40:26.130895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7149},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7149":{"gid":7149,"name":"c","rank":0,"incarnation":114,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6837/500606168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":500606168},{"type":"v1","addr":"172.21.15.148:6837","nonce":500606168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:29.686 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 119 2023-10-30T13:42:29.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664d0 msgr2=0x7f8868068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664d0 0x7f8868068990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f88800b2000 tx=0x7f8880009040 comp rx=0 tx=0).stop 2023-10-30T13:42:29.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8884071110 msgr2=0x7f88841f2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:29.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8884071110 0x7f88841f2f30 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f887400c4d0 tx=0x7f887400c990 comp rx=0 tx=0).stop 2023-10-30T13:42:29.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 shutdown_connections 2023-10-30T13:42:29.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f88680664d0 0x7f8868068990 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8884071110 0x7f88841f2f30 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88840c6bd0 0x7f88841efa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 --2- 172.21.15.148:0/845311990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888406f7a0 0x7f88841528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:29.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 >> 172.21.15.148:0/845311990 conn(0x7f88840c7910 msgr2=0x7f888406a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:29.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 shutdown_connections 2023-10-30T13:42:29.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:29.688+0000 7f888be91640 1 -- 172.21.15.148:0/845311990 wait complete. 2023-10-30T13:42:29.707 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:42:31.544 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:31.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:42:31.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:31.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.855+0000 7f93a4e17640 1 Processor -- start 2023-10-30T13:42:31.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.855+0000 7f93a4e17640 1 -- start start 2023-10-30T13:42:31.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.855+0000 7f93a4e17640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f93a00c6b80 0x7f93a00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.855+0000 7f93a4e17640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a00c0a50 con 0x7f93a00c6f60 2023-10-30T13:42:31.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.855+0000 7f93a4e17640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a006f6e0 con 0x7f93a013f4a0 2023-10-30T13:42:31.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.855+0000 7f93a4e17640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a006f9c0 con 0x7f93a00c6b80 2023-10-30T13:42:31.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.855+0000 7f939e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f93a00c6b80 0x7f93a00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.855+0000 7f939e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f93a00c6b80 0x7f93a00bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49476/0 (socket says 172.21.15.148:49476) 2023-10-30T13:42:31.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.855+0000 7f939e59c640 1 -- 172.21.15.148:0/1728949426 learned_addr learned my addr 172.21.15.148:0/1728949426 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:31.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.859+0000 7f939e59c640 1 -- 172.21.15.148:0/1728949426 >> v1:172.21.15.150:6789/0 conn(0x7f93a013f4a0 legacy=0x7f93a01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.859+0000 7f939e59c640 1 -- 172.21.15.148:0/1728949426 >> v1:172.21.15.148:6789/0 conn(0x7f93a00c6f60 legacy=0x7f93a013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.859+0000 7f939e59c640 1 -- 172.21.15.148:0/1728949426 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a006fca0 con 0x7f93a00c6b80 2023-10-30T13:42:31.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.859+0000 7f939e59c640 1 --2- 172.21.15.148:0/1728949426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6b80 0x7f93a00bfec0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f9390009870 tx=0x7f939001e750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10bb0468897ba43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:31.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.859+0000 7f939dd9b640 1 -- 172.21.15.148:0/1728949426 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93900ab630 con 0x7f93a00c6b80 2023-10-30T13:42:31.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.859+0000 7f939dd9b640 1 -- 172.21.15.148:0/1728949426 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93900ab790 con 0x7f93a00c6b80 2023-10-30T13:42:31.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.859+0000 7f939dd9b640 1 -- 172.21.15.148:0/1728949426 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93900abb20 con 0x7f93a00c6b80 2023-10-30T13:42:31.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.859+0000 7f93a4e17640 1 -- 172.21.15.148:0/1728949426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6b80 msgr2=0x7f93a00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.859+0000 7f93a4e17640 1 --2- 172.21.15.148:0/1728949426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6b80 0x7f93a00bfec0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f9390009870 tx=0x7f939001e750 comp rx=0 tx=0).stop 2023-10-30T13:42:31.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 -- 172.21.15.148:0/1728949426 shutdown_connections 2023-10-30T13:42:31.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 --2- 172.21.15.148:0/1728949426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6b80 0x7f93a00bfec0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 -- 172.21.15.148:0/1728949426 >> 172.21.15.148:0/1728949426 conn(0x7f93a00c7910 msgr2=0x7f93a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:31.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 -- 172.21.15.148:0/1728949426 shutdown_connections 2023-10-30T13:42:31.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 -- 172.21.15.148:0/1728949426 wait complete. 2023-10-30T13:42:31.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 Processor -- start 2023-10-30T13:42:31.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 -- start start 2023-10-30T13:42:31.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a00c6b80 0x7f93a00c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6f60 0x7f93a01f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93a013f4a0 0x7f93a01544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a01f22b0 con 0x7f93a00c6f60 2023-10-30T13:42:31.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a01f3210 con 0x7f93a013f4a0 2023-10-30T13:42:31.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f93a4e17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00c4a90 con 0x7f93a00c6b80 2023-10-30T13:42:31.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f9395d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6f60 0x7f93a01f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f9395d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6f60 0x7f93a01f2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49490/0 (socket says 172.21.15.148:49490) 2023-10-30T13:42:31.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f9395d9b640 1 -- 172.21.15.148:0/1150180517 learned_addr learned my addr 172.21.15.148:0/1150180517 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:31.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f939e59c640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a00c6b80 0x7f93a00c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f939ed9d640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93a013f4a0 0x7f93a01544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.863+0000 7f9395d9b640 1 -- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a00c6b80 msgr2=0x7f93a00c2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f9395d9b640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a00c6b80 0x7f93a00c2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f9395d9b640 1 -- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93a013f4a0 msgr2=0x7f93a01544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:31.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f9395d9b640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93a013f4a0 0x7f93a01544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:31.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f9395d9b640 1 -- 172.21.15.148:0/1150180517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93900094d0 con 0x7f93a00c6f60 2023-10-30T13:42:31.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f939e59c640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a00c6b80 0x7f93a00c2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:31.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f9395d9b640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6f60 0x7f93a01f2cd0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f938800c500 tx=0x7f938800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:31.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f93977fe640 1 -- 172.21.15.148:0/1150180517 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9388005220 con 0x7f93a00c6f60 2023-10-30T13:42:31.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f93977fe640 1 -- 172.21.15.148:0/1150180517 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9388005380 con 0x7f93a00c6f60 2023-10-30T13:42:31.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a00c4d70 con 0x7f93a00c6f60 2023-10-30T13:42:31.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f93977fe640 1 -- 172.21.15.148:0/1150180517 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9388007270 con 0x7f93a00c6f60 2023-10-30T13:42:31.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a01f24d0 con 0x7f93a00c6f60 2023-10-30T13:42:31.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a00c5290 con 0x7f93a00c6f60 2023-10-30T13:42:31.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f93977fe640 1 -- 172.21.15.148:0/1150180517 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f93880073d0 con 0x7f93a00c6f60 2023-10-30T13:42:31.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f93977fe640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93800664a0 0x7f9380068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:31.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f93977fe640 1 -- 172.21.15.148:0/1150180517 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f9388083dc0 con 0x7f93a00c6f60 2023-10-30T13:42:31.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.867+0000 7f939e59c640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93800664a0 0x7f9380068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:31.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.871+0000 7f939e59c640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93800664a0 0x7f9380068960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f93900b0000 tx=0x7f93900a9000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:31.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:31.871+0000 7f93977fe640 1 -- 172.21.15.148:0/1150180517 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9388050bc0 con 0x7f93a00c6f60 2023-10-30T13:42:32.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.051+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93a004e170 con 0x7f93a00c6f60 2023-10-30T13:42:32.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.051+0000 7f93977fe640 1 -- 172.21.15.148:0/1150180517 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f93880541e0 con 0x7f93a00c6f60 2023-10-30T13:42:32.054 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:32.054 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:29.894115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:32.055 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 120 2023-10-30T13:42:32.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93800664a0 msgr2=0x7f9380068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93800664a0 0x7f9380068960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f93900b0000 tx=0x7f93900a9000 comp rx=0 tx=0).stop 2023-10-30T13:42:32.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6f60 msgr2=0x7f93a01f2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6f60 0x7f93a01f2cd0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f938800c500 tx=0x7f938800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 shutdown_connections 2023-10-30T13:42:32.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f93800664a0 0x7f9380068960 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f93a00c6f60 0x7f93a01f2cd0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93a013f4a0 0x7f93a01544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 --2- 172.21.15.148:0/1150180517 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f93a00c6b80 0x7f93a00c2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 >> 172.21.15.148:0/1150180517 conn(0x7f93a00c7910 msgr2=0x7f93a0138ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:32.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 shutdown_connections 2023-10-30T13:42:32.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.055+0000 7f93a4e17640 1 -- 172.21.15.148:0/1150180517 wait complete. 2023-10-30T13:42:32.075 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:42:32.075 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:42:32.075 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:32.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.211+0000 7f61a5b0e640 1 Processor -- start 2023-10-30T13:42:32.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.211+0000 7f61a5b0e640 1 -- start start 2023-10-30T13:42:32.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f61a5b0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a00c6b80 0x7f61a00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f61a5b0e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f61a00c6f60 0x7f61a00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f61a5b0e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a00c0a50 con 0x7f61a00c8430 2023-10-30T13:42:32.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f61a5b0e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a006f6e0 con 0x7f61a00c6f60 2023-10-30T13:42:32.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f61a5b0e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a006f9c0 con 0x7f61a00c6b80 2023-10-30T13:42:32.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f61a00c8430 0x7f61a013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55768/0 (socket says 172.21.15.148:55768) 2023-10-30T13:42:32.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619ffff640 1 -- 172.21.15.148:0/3823291734 learned_addr learned my addr 172.21.15.148:0/3823291734 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:32.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619effd640 1 --2- 172.21.15.148:0/3823291734 >> v2:172.21.15.150:3300/0 conn(0x7f61a00c6f60 0x7f61a00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619f7fe640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a00c6b80 0x7f61a00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540997392 0 0) 0x7f61a00c0a50 con 0x7f61a00c8430 2023-10-30T13:42:32.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188003530 con 0x7f61a00c8430 2023-10-30T13:42:32.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 338487605 0 0) 0x7f6188003530 con 0x7f61a00c8430 2023-10-30T13:42:32.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a00c6b80 msgr2=0x7f61a00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a00c6b80 0x7f61a00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 >> v2:172.21.15.150:3300/0 conn(0x7f61a00c6f60 msgr2=0x7f61a00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 --2- 172.21.15.148:0/3823291734 >> v2:172.21.15.150:3300/0 conn(0x7f61a00c6f60 0x7f61a00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a006fca0 con 0x7f61a00c8430 2023-10-30T13:42:32.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f6198001a00 con 0x7f61a00c8430 2023-10-30T13:42:32.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 >> v1:172.21.15.148:6789/0 conn(0x7f61a00c8430 legacy=0x7f61a013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6188003980 0x7f6188003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61880047d0 0x7f6188004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a00c6b80 0x7f6188005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a006f6e0 con 0x7f6188003980 2023-10-30T13:42:32.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a006f9c0 con 0x7f61880047d0 2023-10-30T13:42:32.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618800b990 con 0x7f61a00c6b80 2023-10-30T13:42:32.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619f7fe640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6188003980 0x7f6188003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619effd640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61880047d0 0x7f6188004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619ffff640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a00c6b80 0x7f6188005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619f7fe640 1 -- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a00c6b80 msgr2=0x7f6188005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:32.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619f7fe640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a00c6b80 0x7f6188005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619f7fe640 1 -- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61880047d0 msgr2=0x7f6188004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:32.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619f7fe640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61880047d0 0x7f6188004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.215+0000 7f619f7fe640 1 -- 172.21.15.148:0/3823291734 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6188003530 con 0x7f6188003980 2023-10-30T13:42:32.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f619f7fe640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6188003980 0x7f6188003da0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f6190009ae0 tx=0x7f619001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e85d81eb04a720d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:32.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61900a6460 con 0x7f6188003980 2023-10-30T13:42:32.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61900a65c0 con 0x7f6188003980 2023-10-30T13:42:32.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f619e7fc640 1 -- 172.21.15.148:0/3823291734 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61900a6970 con 0x7f6188003980 2023-10-30T13:42:32.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f61a5b0e640 1 -- 172.21.15.148:0/3823291734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6188003980 msgr2=0x7f6188003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f61a5b0e640 1 --2- 172.21.15.148:0/3823291734 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6188003980 0x7f6188003da0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f6190009ae0 tx=0x7f619001ca50 comp rx=0 tx=0).stop 2023-10-30T13:42:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f619ffff640 1 -- 172.21.15.148:0/3823291734 reap_dead start 2023-10-30T13:42:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f61a5b0e640 1 -- 172.21.15.148:0/3823291734 shutdown_connections 2023-10-30T13:42:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f61a5b0e640 1 -- 172.21.15.148:0/3823291734 >> 172.21.15.148:0/3823291734 conn(0x7f61a00c7910 msgr2=0x7f61a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f61a5b0e640 1 -- 172.21.15.148:0/3823291734 shutdown_connections 2023-10-30T13:42:32.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.219+0000 7f61a5b0e640 1 -- 172.21.15.148:0/3823291734 wait complete. 2023-10-30T13:42:32.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 Processor -- start 2023-10-30T13:42:32.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 -- start start 2023-10-30T13:42:32.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6188003980 0x7f61a0153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61a00ca920 0x7f61a0153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a0154500 0x7f61a0154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00ca5b0 con 0x7f61a00ca920 2023-10-30T13:42:32.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a0154ec0 con 0x7f6188003980 2023-10-30T13:42:32.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00c96c0 con 0x7f61a0154500 2023-10-30T13:42:32.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61a00ca920 0x7f61a0153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61a00ca920 0x7f61a0153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49508/0 (socket says 172.21.15.148:49508) 2023-10-30T13:42:32.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619effd640 1 -- 172.21.15.148:0/960812977 learned_addr learned my addr 172.21.15.148:0/960812977 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:32.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619f7fe640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6188003980 0x7f61a0153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619ffff640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a0154500 0x7f61a0154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619effd640 1 -- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a0154500 msgr2=0x7f61a0154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619effd640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a0154500 0x7f61a0154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619effd640 1 -- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6188003980 msgr2=0x7f61a0153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619effd640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6188003980 0x7f61a0153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619effd640 1 -- 172.21.15.148:0/960812977 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a00c99a0 con 0x7f61a00ca920 2023-10-30T13:42:32.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619f7fe640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6188003980 0x7f61a0153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:32.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f619effd640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61a00ca920 0x7f61a0153660 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f6194011600 tx=0x7f6194011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:32.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f6187fff640 1 -- 172.21.15.148:0/960812977 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194004220 con 0x7f61a00ca920 2023-10-30T13:42:32.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f6187fff640 1 -- 172.21.15.148:0/960812977 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6194004380 con 0x7f61a00ca920 2023-10-30T13:42:32.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f6187fff640 1 -- 172.21.15.148:0/960812977 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619400f960 con 0x7f61a00ca920 2023-10-30T13:42:32.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a00c9c80 con 0x7f61a00ca920 2023-10-30T13:42:32.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a00ca240 con 0x7f61a00ca920 2023-10-30T13:42:32.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.223+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a004e170 con 0x7f61a00ca920 2023-10-30T13:42:32.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.227+0000 7f6187fff640 1 -- 172.21.15.148:0/960812977 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f619400fac0 con 0x7f61a00ca920 2023-10-30T13:42:32.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.227+0000 7f6187fff640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f618c066490 0x7f618c068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:32.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.227+0000 7f6187fff640 1 -- 172.21.15.148:0/960812977 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f6194084670 con 0x7f61a00ca920 2023-10-30T13:42:32.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.227+0000 7f619f7fe640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f618c066490 0x7f618c068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:32.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.227+0000 7f619f7fe640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f618c066490 0x7f618c068950 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f61900ad000 tx=0x7f61900b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:32.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.231+0000 7f6187fff640 1 -- 172.21.15.148:0/960812977 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6194051460 con 0x7f61a00ca920 2023-10-30T13:42:32.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61a00b08b0 con 0x7f61a00ca920 2023-10-30T13:42:32.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f6187fff640 1 -- 172.21.15.148:0/960812977 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f61940044f0 con 0x7f61a00ca920 2023-10-30T13:42:32.412 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:32.412 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:29.894115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 120 2023-10-30T13:42:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f618c066490 msgr2=0x7f618c068950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f618c066490 0x7f618c068950 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f61900ad000 tx=0x7f61900b2040 comp rx=0 tx=0).stop 2023-10-30T13:42:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61a00ca920 msgr2=0x7f61a0153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61a00ca920 0x7f61a0153660 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f6194011600 tx=0x7f6194011ac0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 shutdown_connections 2023-10-30T13:42:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f618c066490 0x7f618c068950 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61a00ca920 0x7f61a0153660 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61a0154500 0x7f61a0154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 --2- 172.21.15.148:0/960812977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6188003980 0x7f61a0153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 >> 172.21.15.148:0/960812977 conn(0x7f61a00c7910 msgr2=0x7f61a0159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.411+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 shutdown_connections 2023-10-30T13:42:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:32.415+0000 7f61a5b0e640 1 -- 172.21.15.148:0/960812977 wait complete. 2023-10-30T13:42:32.434 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 120, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 120, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7305": { "addr": "172.21.15.148:6835/3829165476", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 3829165476, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 3829165476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7305, "incarnation": 120, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:29.894115+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7305 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:34.435 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:34.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.583+0000 7f53ee11d640 1 Processor -- start 2023-10-30T13:42:34.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.583+0000 7f53ee11d640 1 -- start start 2023-10-30T13:42:34.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.583+0000 7f53ee11d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 0x7f53e80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.583+0000 7f53ee11d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f53e80c6f60 0x7f53e80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.583+0000 7f53ee11d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e80c0a50 con 0x7f53e80c8430 2023-10-30T13:42:34.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.583+0000 7f53ee11d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e806f6e0 con 0x7f53e80c6f60 2023-10-30T13:42:34.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.583+0000 7f53ee11d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e806f9c0 con 0x7f53e80c6b80 2023-10-30T13:42:34.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e6ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f53e80c6f60 0x7f53e80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e6ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f53e80c6f60 0x7f53e80c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49520/0 (socket says 172.21.15.148:49520) 2023-10-30T13:42:34.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e6ffd640 1 -- 172.21.15.148:0/1926404287 learned_addr learned my addr 172.21.15.148:0/1926404287 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e77fe640 1 --2- 172.21.15.148:0/1926404287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 0x7f53e80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e6ffd640 1 -- 172.21.15.148:0/1926404287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 msgr2=0x7f53e80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e6ffd640 1 --2- 172.21.15.148:0/1926404287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 0x7f53e80bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e6ffd640 1 -- 172.21.15.148:0/1926404287 >> v1:172.21.15.150:6789/0 conn(0x7f53e80c8430 legacy=0x7f53e813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:34.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e6ffd640 1 -- 172.21.15.148:0/1926404287 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e806fca0 con 0x7f53e80c6f60 2023-10-30T13:42:34.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e6ffd640 1 --2- 172.21.15.148:0/1926404287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c6f60 0x7f53e80c0400 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f53dc009870 tx=0x7f53dc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfbcabb2f8a614f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:34.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e67fc640 1 -- 172.21.15.148:0/1926404287 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0ac630 con 0x7f53e80c6f60 2023-10-30T13:42:34.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e67fc640 1 -- 172.21.15.148:0/1926404287 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53dc0ac790 con 0x7f53e80c6f60 2023-10-30T13:42:34.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.587+0000 7f53e67fc640 1 -- 172.21.15.148:0/1926404287 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0acb20 con 0x7f53e80c6f60 2023-10-30T13:42:34.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 -- 172.21.15.148:0/1926404287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c6f60 msgr2=0x7f53e80c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 --2- 172.21.15.148:0/1926404287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c6f60 0x7f53e80c0400 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f53dc009870 tx=0x7f53dc01c500 comp rx=0 tx=0).stop 2023-10-30T13:42:34.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 -- 172.21.15.148:0/1926404287 shutdown_connections 2023-10-30T13:42:34.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 --2- 172.21.15.148:0/1926404287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c6f60 0x7f53e80c0400 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 --2- 172.21.15.148:0/1926404287 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 0x7f53e80bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 -- 172.21.15.148:0/1926404287 >> 172.21.15.148:0/1926404287 conn(0x7f53e80c7910 msgr2=0x7f53e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 -- 172.21.15.148:0/1926404287 shutdown_connections 2023-10-30T13:42:34.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 -- 172.21.15.148:0/1926404287 wait complete. 2023-10-30T13:42:34.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 Processor -- start 2023-10-30T13:42:34.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 -- start start 2023-10-30T13:42:34.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 0x7f53e8157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80c6f60 0x7f53e8159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c8430 0x7f53e81f5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e80b0bf0 con 0x7f53e80c8430 2023-10-30T13:42:34.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e8138e20 con 0x7f53e80c6f60 2023-10-30T13:42:34.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53ee11d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e81390c0 con 0x7f53e80c6b80 2023-10-30T13:42:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c8430 0x7f53e81f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c8430 0x7f53e81f5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49536/0 (socket says 172.21.15.148:49536) 2023-10-30T13:42:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53e7fff640 1 -- 172.21.15.148:0/3484181473 learned_addr learned my addr 172.21.15.148:0/3484181473 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53e77fe640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 0x7f53e8157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.591+0000 7f53e6ffd640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80c6f60 0x7f53e8159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53e7fff640 1 -- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 msgr2=0x7f53e8157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53e7fff640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 0x7f53e8157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53e7fff640 1 -- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80c6f60 msgr2=0x7f53e8159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53e7fff640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80c6f60 0x7f53e8159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53e7fff640 1 -- 172.21.15.148:0/3484181473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc0094d0 con 0x7f53e80c8430 2023-10-30T13:42:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53e6ffd640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80c6f60 0x7f53e8159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53e7fff640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c8430 0x7f53e81f5780 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f53e000b970 tx=0x7f53e000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53cffff640 1 -- 172.21.15.148:0/3484181473 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e000cb40 con 0x7f53e80c8430 2023-10-30T13:42:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53cffff640 1 -- 172.21.15.148:0/3484181473 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e000cca0 con 0x7f53e80c8430 2023-10-30T13:42:34.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53cffff640 1 -- 172.21.15.148:0/3484181473 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0014680 con 0x7f53e80c8430 2023-10-30T13:42:34.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e80ca220 con 0x7f53e80c8430 2023-10-30T13:42:34.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e8157cd0 con 0x7f53e80c8430 2023-10-30T13:42:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e80ca080 con 0x7f53e80c8430 2023-10-30T13:42:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53cffff640 1 -- 172.21.15.148:0/3484181473 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f53e001d3b0 con 0x7f53e80c8430 2023-10-30T13:42:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53cffff640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53d40664d0 0x7f53d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.595+0000 7f53cffff640 1 -- 172.21.15.148:0/3484181473 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f53e0084180 con 0x7f53e80c8430 2023-10-30T13:42:34.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.599+0000 7f53e77fe640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53d40664d0 0x7f53d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:34.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.599+0000 7f53e77fe640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53d40664d0 0x7f53d4068990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f53d8007a60 tx=0x7f53d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:34.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.599+0000 7f53cffff640 1 -- 172.21.15.148:0/3484181473 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53e80ca080 con 0x7f53e80c8430 2023-10-30T13:42:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53e80ca080 con 0x7f53e80c8430 2023-10-30T13:42:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53cffff640 1 -- 172.21.15.148:0/3484181473 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f53e80ca080 con 0x7f53e80c8430 2023-10-30T13:42:34.783 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:34.784 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:29.894115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 120 2023-10-30T13:42:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53d40664d0 msgr2=0x7f53d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53d40664d0 0x7f53d4068990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f53d8007a60 tx=0x7f53d8008040 comp rx=0 tx=0).stop 2023-10-30T13:42:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c8430 msgr2=0x7f53e81f5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c8430 0x7f53e81f5780 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f53e000b970 tx=0x7f53e000be30 comp rx=0 tx=0).stop 2023-10-30T13:42:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 shutdown_connections 2023-10-30T13:42:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f53d40664d0 0x7f53d4068990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f53e80c8430 0x7f53e81f5780 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80c6f60 0x7f53e8159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 --2- 172.21.15.148:0/3484181473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f53e80c6b80 0x7f53e8157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 >> 172.21.15.148:0/3484181473 conn(0x7f53e80c7910 msgr2=0x7f53e80c5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 shutdown_connections 2023-10-30T13:42:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:34.783+0000 7f53ee11d640 1 -- 172.21.15.148:0/3484181473 wait complete. 2023-10-30T13:42:34.805 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:36.806 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:36.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8d1d3640 1 Processor -- start 2023-10-30T13:42:36.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8d1d3640 1 -- start start 2023-10-30T13:42:36.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8d1d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc8815d6f0 0x7fcc8815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8d1d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcc880cf160 0x7fcc880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8d1d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc880cb800 con 0x7fcc881538a0 2023-10-30T13:42:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8d1d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc880cb330 con 0x7fcc880cf160 2023-10-30T13:42:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8d1d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc880cad70 con 0x7fcc8815d6f0 2023-10-30T13:42:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8759e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcc880cf160 0x7fcc880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8759e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcc880cf160 0x7fcc880cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49546/0 (socket says 172.21.15.148:49546) 2023-10-30T13:42:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8759e640 1 -- 172.21.15.148:0/2115541995 learned_addr learned my addr 172.21.15.148:0/2115541995 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:36.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.951+0000 7fcc8659c640 1 --2- 172.21.15.148:0/2115541995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc8815d6f0 0x7fcc8815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.955+0000 7fcc8759e640 1 -- 172.21.15.148:0/2115541995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc8815d6f0 msgr2=0x7fcc8815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:36.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.955+0000 7fcc8759e640 1 --2- 172.21.15.148:0/2115541995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc8815d6f0 0x7fcc8815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.955+0000 7fcc8759e640 1 -- 172.21.15.148:0/2115541995 >> v1:172.21.15.150:6789/0 conn(0x7fcc881538a0 legacy=0x7fcc880ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.955+0000 7fcc8759e640 1 -- 172.21.15.148:0/2115541995 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc880cb050 con 0x7fcc880cf160 2023-10-30T13:42:36.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc8759e640 1 --2- 172.21.15.148:0/2115541995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 0x7fcc880cf590 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009e50 tx=0x7fcc8001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5d7ea5ded65141 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc85d9b640 1 -- 172.21.15.148:0/2115541995 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc800af400 con 0x7fcc880cf160 2023-10-30T13:42:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc85d9b640 1 -- 172.21.15.148:0/2115541995 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc800ab050 con 0x7fcc880cf160 2023-10-30T13:42:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc85d9b640 1 -- 172.21.15.148:0/2115541995 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc800af780 con 0x7fcc880cf160 2023-10-30T13:42:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2115541995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 msgr2=0x7fcc880cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc8d1d3640 1 --2- 172.21.15.148:0/2115541995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 0x7fcc880cf590 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009e50 tx=0x7fcc8001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2115541995 shutdown_connections 2023-10-30T13:42:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc8d1d3640 1 --2- 172.21.15.148:0/2115541995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 0x7fcc880cf590 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc8d1d3640 1 --2- 172.21.15.148:0/2115541995 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc8815d6f0 0x7fcc8815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2115541995 >> 172.21.15.148:0/2115541995 conn(0x7fcc8806bfe0 msgr2=0x7fcc8806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2115541995 shutdown_connections 2023-10-30T13:42:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.959+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2115541995 wait complete. 2023-10-30T13:42:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 Processor -- start 2023-10-30T13:42:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 -- start start 2023-10-30T13:42:36.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 0x7fcc881e49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc881538a0 0x7fcc880b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc8815d6f0 0x7fcc881f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc881e3180 con 0x7fcc880cf160 2023-10-30T13:42:36.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc881e2a10 con 0x7fcc8815d6f0 2023-10-30T13:42:36.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc881e2cf0 con 0x7fcc881538a0 2023-10-30T13:42:36.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc86d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 0x7fcc881e49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc86d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 0x7fcc881e49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49558/0 (socket says 172.21.15.148:49558) 2023-10-30T13:42:36.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc86d9d640 1 -- 172.21.15.148:0/2658343656 learned_addr learned my addr 172.21.15.148:0/2658343656 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:36.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8759e640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc8815d6f0 0x7fcc881f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8659c640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc881538a0 0x7fcc880b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc86d9d640 1 -- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc881538a0 msgr2=0x7fcc880b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc86d9d640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc881538a0 0x7fcc880b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc86d9d640 1 -- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc8815d6f0 msgr2=0x7fcc881f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc86d9d640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc8815d6f0 0x7fcc881f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc86d9d640 1 -- 172.21.15.148:0/2658343656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc80009b30 con 0x7fcc880cf160 2023-10-30T13:42:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc86d9d640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 0x7fcc881e49e0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c500 tx=0x7fcc7800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc6f7fe640 1 -- 172.21.15.148:0/2658343656 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78005220 con 0x7fcc880cf160 2023-10-30T13:42:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc6f7fe640 1 -- 172.21.15.148:0/2658343656 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78005380 con 0x7fcc880cf160 2023-10-30T13:42:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc6f7fe640 1 -- 172.21.15.148:0/2658343656 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78007270 con 0x7fcc880cf160 2023-10-30T13:42:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc880b4a90 con 0x7fcc880cf160 2023-10-30T13:42:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.963+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc880b4d30 con 0x7fcc880cf160 2023-10-30T13:42:36.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.967+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc880b2a50 con 0x7fcc880cf160 2023-10-30T13:42:36.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.967+0000 7fcc6f7fe640 1 -- 172.21.15.148:0/2658343656 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcc780073d0 con 0x7fcc880cf160 2023-10-30T13:42:36.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.967+0000 7fcc6f7fe640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc740664a0 0x7fcc74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:36.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.967+0000 7fcc6f7fe640 1 -- 172.21.15.148:0/2658343656 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7fcc78083bf0 con 0x7fcc880cf160 2023-10-30T13:42:36.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.971+0000 7fcc8659c640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc740664a0 0x7fcc74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:36.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.971+0000 7fcc8659c640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc740664a0 0x7fcc74068960 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fcc880c5da0 tx=0x7fcc7c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:36.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:36.971+0000 7fcc6f7fe640 1 -- 172.21.15.148:0/2658343656 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcc78050b40 con 0x7fcc880cf160 2023-10-30T13:42:37.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.247+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc8804e170 con 0x7fcc880cf160 2023-10-30T13:42:37.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.247+0000 7fcc6f7fe640 1 -- 172.21.15.148:0/2658343656 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5719 (secure 0 0 0) 0x7fcc78054160 con 0x7fcc880cf160 2023-10-30T13:42:37.248 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:37.248 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:29.894115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:37.249 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 120 2023-10-30T13:42:37.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.247+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc740664a0 msgr2=0x7fcc74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:37.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.247+0000 7fcc8d1d3640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc740664a0 0x7fcc74068960 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fcc880c5da0 tx=0x7fcc7c009190 comp rx=0 tx=0).stop 2023-10-30T13:42:37.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.247+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 msgr2=0x7fcc881e49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:37.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.247+0000 7fcc8d1d3640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 0x7fcc881e49e0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c500 tx=0x7fcc7800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.251+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 shutdown_connections 2023-10-30T13:42:37.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.251+0000 7fcc8d1d3640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcc740664a0 0x7fcc74068960 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.251+0000 7fcc8d1d3640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc8815d6f0 0x7fcc881f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.251+0000 7fcc8d1d3640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcc881538a0 0x7fcc880b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.251+0000 7fcc8d1d3640 1 --2- 172.21.15.148:0/2658343656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcc880cf160 0x7fcc881e49e0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:37.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.251+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 >> 172.21.15.148:0/2658343656 conn(0x7fcc8806bfe0 msgr2=0x7fcc88151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:37.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.251+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 shutdown_connections 2023-10-30T13:42:37.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:37.251+0000 7fcc8d1d3640 1 -- 172.21.15.148:0/2658343656 wait complete. 2023-10-30T13:42:37.270 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:39.271 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:39.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febdb72a640 1 Processor -- start 2023-10-30T13:42:39.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febdb72a640 1 -- start start 2023-10-30T13:42:39.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febdb72a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 0x7febd40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:39.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febdb72a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7febd413f4a0 0x7febd40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:39.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febdb72a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd40c0a50 con 0x7febd40c6f60 2023-10-30T13:42:39.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febdb72a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd406f6e0 con 0x7febd413f4a0 2023-10-30T13:42:39.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febdb72a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd406f9c0 con 0x7febd40c6b80 2023-10-30T13:42:39.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febd9cc7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7febd413f4a0 0x7febd40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:39.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febd94c6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 0x7febd40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:39.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febd8cc5640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7febd40c6f60 0x7febd413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:40236/0 (socket says 172.21.15.148:40236) 2023-10-30T13:42:39.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febd8cc5640 1 -- 172.21.15.148:0/1923496739 learned_addr learned my addr 172.21.15.148:0/1923496739 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:39.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febd94c6640 1 -- 172.21.15.148:0/1923496739 >> v2:172.21.15.150:3300/0 conn(0x7febd413f4a0 msgr2=0x7febd40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:39.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febd94c6640 1 --2- 172.21.15.148:0/1923496739 >> v2:172.21.15.150:3300/0 conn(0x7febd413f4a0 0x7febd40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febd94c6640 1 -- 172.21.15.148:0/1923496739 >> v1:172.21.15.150:6789/0 conn(0x7febd40c6f60 legacy=0x7febd413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:39.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.419+0000 7febd94c6640 1 -- 172.21.15.148:0/1923496739 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd406fca0 con 0x7febd40c6b80 2023-10-30T13:42:39.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.423+0000 7febd94c6640 1 --2- 172.21.15.148:0/1923496739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 0x7febd40bfec0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7febd00097c0 tx=0x7febd001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8643c0837b4d05a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:39.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febcbfff640 1 -- 172.21.15.148:0/1923496739 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd00ad640 con 0x7febd40c6b80 2023-10-30T13:42:39.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febcbfff640 1 -- 172.21.15.148:0/1923496739 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd00ad7a0 con 0x7febd40c6b80 2023-10-30T13:42:39.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febcbfff640 1 -- 172.21.15.148:0/1923496739 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd00adb30 con 0x7febd40c6b80 2023-10-30T13:42:39.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 -- 172.21.15.148:0/1923496739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 msgr2=0x7febd40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:39.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 --2- 172.21.15.148:0/1923496739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 0x7febd40bfec0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7febd00097c0 tx=0x7febd001c480 comp rx=0 tx=0).stop 2023-10-30T13:42:39.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 -- 172.21.15.148:0/1923496739 shutdown_connections 2023-10-30T13:42:39.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 --2- 172.21.15.148:0/1923496739 >> v2:172.21.15.150:3300/0 conn(0x7febd413f4a0 0x7febd40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 --2- 172.21.15.148:0/1923496739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 0x7febd40bfec0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 -- 172.21.15.148:0/1923496739 >> 172.21.15.148:0/1923496739 conn(0x7febd40c7910 msgr2=0x7febd40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:39.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 -- 172.21.15.148:0/1923496739 shutdown_connections 2023-10-30T13:42:39.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 -- 172.21.15.148:0/1923496739 wait complete. 2023-10-30T13:42:39.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 Processor -- start 2023-10-30T13:42:39.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 -- start start 2023-10-30T13:42:39.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 0x7febd413e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:39.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd40c6f60 0x7febd4150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:39.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febd413f4a0 0x7febd4150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:39.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd4157ef0 con 0x7febd413f4a0 2023-10-30T13:42:39.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd41357e0 con 0x7febd40c6f60 2023-10-30T13:42:39.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.427+0000 7febdb72a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd4135220 con 0x7febd40c6b80 2023-10-30T13:42:39.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd9cc7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febd413f4a0 0x7febd4150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:39.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd9cc7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febd413f4a0 0x7febd4150550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49570/0 (socket says 172.21.15.148:49570) 2023-10-30T13:42:39.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd9cc7640 1 -- 172.21.15.148:0/1887876299 learned_addr learned my addr 172.21.15.148:0/1887876299 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:39.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd8cc5640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd40c6f60 0x7febd4150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:39.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd94c6640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 0x7febd413e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:39.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd9cc7640 1 -- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 msgr2=0x7febd413e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:39.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd9cc7640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 0x7febd413e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd9cc7640 1 -- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd40c6f60 msgr2=0x7febd4150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:39.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd9cc7640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd40c6f60 0x7febd4150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd9cc7640 1 -- 172.21.15.148:0/1887876299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd0009420 con 0x7febd413f4a0 2023-10-30T13:42:39.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febd9cc7640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febd413f4a0 0x7febd4150550 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7febcc00efc0 tx=0x7febcc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:39.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febc9ffb640 1 -- 172.21.15.148:0/1887876299 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febcc009b20 con 0x7febd413f4a0 2023-10-30T13:42:39.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febc9ffb640 1 -- 172.21.15.148:0/1887876299 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febcc00f040 con 0x7febd413f4a0 2023-10-30T13:42:39.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febc9ffb640 1 -- 172.21.15.148:0/1887876299 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febcc0146c0 con 0x7febd413f4a0 2023-10-30T13:42:39.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd4135500 con 0x7febd413f4a0 2023-10-30T13:42:39.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd4157b80 con 0x7febd413f4a0 2023-10-30T13:42:39.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.431+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd41580a0 con 0x7febd413f4a0 2023-10-30T13:42:39.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.435+0000 7febc9ffb640 1 -- 172.21.15.148:0/1887876299 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7febcc014820 con 0x7febd413f4a0 2023-10-30T13:42:39.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.435+0000 7febc9ffb640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7febb80664d0 0x7febb8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:39.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.435+0000 7febc9ffb640 1 -- 172.21.15.148:0/1887876299 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7febcc085220 con 0x7febd413f4a0 2023-10-30T13:42:39.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.435+0000 7febd94c6640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7febb80664d0 0x7febb8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:39.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.435+0000 7febd94c6640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7febb80664d0 0x7febb8068990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7febd00af000 tx=0x7febd0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:39.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.439+0000 7febc9ffb640 1 -- 172.21.15.148:0/1887876299 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7febd41580a0 con 0x7febd413f4a0 2023-10-30T13:42:39.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.627+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febd40705c0 con 0x7febd413f4a0 2023-10-30T13:42:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.627+0000 7febc9ffb640 1 -- 172.21.15.148:0/1887876299 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5719 (secure 0 0 0) 0x7febcc0520a0 con 0x7febd413f4a0 2023-10-30T13:42:39.628 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:39.628 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:29.894115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:39.629 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 120 2023-10-30T13:42:39.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7febb80664d0 msgr2=0x7febb8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:39.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7febb80664d0 0x7febb8068990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7febd00af000 tx=0x7febd0008040 comp rx=0 tx=0).stop 2023-10-30T13:42:39.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febd413f4a0 msgr2=0x7febd4150550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:39.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febd413f4a0 0x7febd4150550 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7febcc00efc0 tx=0x7febcc00c490 comp rx=0 tx=0).stop 2023-10-30T13:42:39.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 shutdown_connections 2023-10-30T13:42:39.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7febb80664d0 0x7febb8068990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febd413f4a0 0x7febd4150550 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febd40c6f60 0x7febd4150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 --2- 172.21.15.148:0/1887876299 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7febd40c6b80 0x7febd413e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:39.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 >> 172.21.15.148:0/1887876299 conn(0x7febd40c7910 msgr2=0x7febd41532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:39.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 shutdown_connections 2023-10-30T13:42:39.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:39.631+0000 7febdb72a640 1 -- 172.21.15.148:0/1887876299 wait complete. 2023-10-30T13:42:39.649 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:41.651 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:41.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.791+0000 7f5e9cf93640 1 Processor -- start 2023-10-30T13:42:41.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.791+0000 7f5e9cf93640 1 -- start start 2023-10-30T13:42:41.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e9cf93640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e98153870 0x7f5e980708c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e9cf93640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5e9815d7c0 0x7f5e980c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e9cf93640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e98070e00 con 0x7f5e98159ff0 2023-10-30T13:42:41.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e9cf93640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9806be50 con 0x7f5e9815d7c0 2023-10-30T13:42:41.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e9cf93640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e980715f0 con 0x7f5e98153870 2023-10-30T13:42:41.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e97fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5e98159ff0 0x7f5e98071860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55784/0 (socket says 172.21.15.148:55784) 2023-10-30T13:42:41.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e977fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e98153870 0x7f5e980708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e97fff640 1 -- 172.21.15.148:0/1342766034 learned_addr learned my addr 172.21.15.148:0/1342766034 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:41.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e96ffd640 1 --2- 172.21.15.148:0/1342766034 >> v2:172.21.15.150:3300/0 conn(0x7f5e9815d7c0 0x7f5e980c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e967fc640 1 -- 172.21.15.148:0/1342766034 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869103941 0 0) 0x7f5e98070e00 con 0x7f5e98159ff0 2023-10-30T13:42:41.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e967fc640 1 -- 172.21.15.148:0/1342766034 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e80003530 con 0x7f5e98159ff0 2023-10-30T13:42:41.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e977fe640 1 -- 172.21.15.148:0/1342766034 >> v2:172.21.15.150:3300/0 conn(0x7f5e9815d7c0 msgr2=0x7f5e980c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:41.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e977fe640 1 --2- 172.21.15.148:0/1342766034 >> v2:172.21.15.150:3300/0 conn(0x7f5e9815d7c0 0x7f5e980c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e977fe640 1 -- 172.21.15.148:0/1342766034 >> v1:172.21.15.148:6789/0 conn(0x7f5e98159ff0 legacy=0x7f5e98071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e977fe640 1 -- 172.21.15.148:0/1342766034 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e980c6b30 con 0x7f5e98153870 2023-10-30T13:42:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.795+0000 7f5e96ffd640 1 --2- 172.21.15.148:0/1342766034 >> v2:172.21.15.150:3300/0 conn(0x7f5e9815d7c0 0x7f5e980c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:42:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e977fe640 1 --2- 172.21.15.148:0/1342766034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e98153870 0x7f5e980708c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5e880097c0 tx=0x7f5e8801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1379b9caeb7da4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e967fc640 1 -- 172.21.15.148:0/1342766034 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880a8970 con 0x7f5e98153870 2023-10-30T13:42:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e967fc640 1 -- 172.21.15.148:0/1342766034 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e880a8ad0 con 0x7f5e98153870 2023-10-30T13:42:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e967fc640 1 -- 172.21.15.148:0/1342766034 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880a8e60 con 0x7f5e98153870 2023-10-30T13:42:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 -- 172.21.15.148:0/1342766034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e98153870 msgr2=0x7f5e980708c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 --2- 172.21.15.148:0/1342766034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e98153870 0x7f5e980708c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5e880097c0 tx=0x7f5e8801ec70 comp rx=0 tx=0).stop 2023-10-30T13:42:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 -- 172.21.15.148:0/1342766034 shutdown_connections 2023-10-30T13:42:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 --2- 172.21.15.148:0/1342766034 >> v2:172.21.15.150:3300/0 conn(0x7f5e9815d7c0 0x7f5e980c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 --2- 172.21.15.148:0/1342766034 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e98153870 0x7f5e980708c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 -- 172.21.15.148:0/1342766034 >> 172.21.15.148:0/1342766034 conn(0x7f5e9806e1b0 msgr2=0x7f5e9806e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 -- 172.21.15.148:0/1342766034 shutdown_connections 2023-10-30T13:42:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 -- 172.21.15.148:0/1342766034 wait complete. 2023-10-30T13:42:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 Processor -- start 2023-10-30T13:42:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.799+0000 7f5e9cf93640 1 -- start start 2023-10-30T13:42:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e9cf93640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e98153870 0x7f5e9815bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e9cf93640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e98159ff0 0x7f5e981eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e9cf93640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e9815d7c0 0x7f5e981584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e9cf93640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e981ef1d0 con 0x7f5e98153870 2023-10-30T13:42:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e9cf93640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9815a710 con 0x7f5e98159ff0 2023-10-30T13:42:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e9cf93640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e980cb8c0 con 0x7f5e9815d7c0 2023-10-30T13:42:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e977fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e98153870 0x7f5e9815bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e977fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e98153870 0x7f5e9815bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49578/0 (socket says 172.21.15.148:49578) 2023-10-30T13:42:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e977fe640 1 -- 172.21.15.148:0/996174041 learned_addr learned my addr 172.21.15.148:0/996174041 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e96ffd640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e98159ff0 0x7f5e981eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e97fff640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e9815d7c0 0x7f5e981584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e977fe640 1 -- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e9815d7c0 msgr2=0x7f5e981584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e977fe640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e9815d7c0 0x7f5e981584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e977fe640 1 -- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e98159ff0 msgr2=0x7f5e981eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e977fe640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e98159ff0 0x7f5e981eec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e977fe640 1 -- 172.21.15.148:0/996174041 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c009420 con 0x7f5e98153870 2023-10-30T13:42:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e96ffd640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e98159ff0 0x7f5e981eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e977fe640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e98153870 0x7f5e9815bc70 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5e880b3040 tx=0x7f5e8801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e7ffff640 1 -- 172.21.15.148:0/996174041 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880b0d60 con 0x7f5e98153870 2023-10-30T13:42:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e7ffff640 1 -- 172.21.15.148:0/996174041 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e880b9410 con 0x7f5e98153870 2023-10-30T13:42:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e7ffff640 1 -- 172.21.15.148:0/996174041 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880b97c0 con 0x7f5e98153870 2023-10-30T13:42:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e88009420 con 0x7f5e98153870 2023-10-30T13:42:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9815ac40 con 0x7f5e98153870 2023-10-30T13:42:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9804e170 con 0x7f5e98153870 2023-10-30T13:42:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.803+0000 7f5e7ffff640 1 -- 172.21.15.148:0/996174041 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5e880b7040 con 0x7f5e98153870 2023-10-30T13:42:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.807+0000 7f5e7ffff640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e840664a0 0x7f5e84068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.807+0000 7f5e7ffff640 1 -- 172.21.15.148:0/996174041 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f5e881220a0 con 0x7f5e98153870 2023-10-30T13:42:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.807+0000 7f5e96ffd640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e840664a0 0x7f5e84068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.807+0000 7f5e96ffd640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e840664a0 0x7f5e84068960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5e980c2450 tx=0x7f5e8c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.811+0000 7f5e7ffff640 1 -- 172.21.15.148:0/996174041 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5e880eee90 con 0x7f5e98153870 2023-10-30T13:42:41.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e9806e980 con 0x7f5e98153870 2023-10-30T13:42:41.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e7ffff640 1 -- 172.21.15.148:0/996174041 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f5e880b5020 con 0x7f5e98153870 2023-10-30T13:42:41.992 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:41.992 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:39.785575+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:41.993 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 121 2023-10-30T13:42:41.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e840664a0 msgr2=0x7f5e84068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e840664a0 0x7f5e84068960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5e980c2450 tx=0x7f5e8c008040 comp rx=0 tx=0).stop 2023-10-30T13:42:41.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e98153870 msgr2=0x7f5e9815bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:41.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e98153870 0x7f5e9815bc70 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5e880b3040 tx=0x7f5e8801ba10 comp rx=0 tx=0).stop 2023-10-30T13:42:41.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 shutdown_connections 2023-10-30T13:42:41.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5e840664a0 0x7f5e84068960 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5e9815d7c0 0x7f5e981584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e98159ff0 0x7f5e981eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 --2- 172.21.15.148:0/996174041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5e98153870 0x7f5e9815bc70 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:41.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.991+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 >> 172.21.15.148:0/996174041 conn(0x7f5e9806e1b0 msgr2=0x7f5e98152b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:41.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.995+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 shutdown_connections 2023-10-30T13:42:41.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:41.995+0000 7f5e9cf93640 1 -- 172.21.15.148:0/996174041 wait complete. 2023-10-30T13:42:42.013 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:44.014 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:44.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.163+0000 7f87ae49e640 1 Processor -- start 2023-10-30T13:42:44.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.163+0000 7f87ae49e640 1 -- start start 2023-10-30T13:42:44.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.163+0000 7f87ae49e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a8069e90 0x7f87a806a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.163+0000 7f87ae49e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a8153d80 con 0x7f87a8070a00 2023-10-30T13:42:44.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.163+0000 7f87ae49e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a80c5c50 con 0x7f87a81539a0 2023-10-30T13:42:44.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.163+0000 7f87ae49e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a806fbb0 con 0x7f87a8069e90 2023-10-30T13:42:44.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.163+0000 7f87a77fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f87a8070a00 0x7f87a8070e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59726/0 (socket says 172.21.15.148:59726) 2023-10-30T13:42:44.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.163+0000 7f87a77fe640 1 -- 172.21.15.148:0/3760717846 learned_addr learned my addr 172.21.15.148:0/3760717846 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.163+0000 7f87aca3b640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a8069e90 0x7f87a806a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577860036 0 0) 0x7f87a8153d80 con 0x7f87a8070a00 2023-10-30T13:42:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8790003530 con 0x7f87a8070a00 2023-10-30T13:42:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 757453136 0 0) 0x7f8790003530 con 0x7f87a8070a00 2023-10-30T13:42:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a8069e90 msgr2=0x7f87a806a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a8069e90 0x7f87a806a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 >> v1:172.21.15.150:6789/0 conn(0x7f87a81539a0 legacy=0x7f87a815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a80c5f30 con 0x7f87a8070a00 2023-10-30T13:42:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f879c001680 con 0x7f87a8070a00 2023-10-30T13:42:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 >> v1:172.21.15.148:6789/0 conn(0x7f87a8070a00 legacy=0x7f87a8070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900039b0 0x7f8790003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790004360 0x7f8790004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a8069e90 0x7f8790005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a806fbb0 con 0x7f87900039b0 2023-10-30T13:42:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879000ba40 con 0x7f8790004360 2023-10-30T13:42:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879000bc00 con 0x7f87a8069e90 2023-10-30T13:42:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a7fff640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900039b0 0x7f8790003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a77fe640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790004360 0x7f8790004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87aca3b640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a8069e90 0x7f8790005130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a7fff640 1 -- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a8069e90 msgr2=0x7f8790005130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a7fff640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a8069e90 0x7f8790005130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a7fff640 1 -- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790004360 msgr2=0x7f8790004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a7fff640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790004360 0x7f8790004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a7fff640 1 -- 172.21.15.148:0/3760717846 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8790003530 con 0x7f87900039b0 2023-10-30T13:42:44.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.167+0000 7f87a7fff640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900039b0 0x7f8790003d90 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f8798001860 tx=0x7f879801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36cc3b4e59630bb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:44.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980ab5f0 con 0x7f87900039b0 2023-10-30T13:42:44.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87980ab750 con 0x7f87900039b0 2023-10-30T13:42:44.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87a6ffd640 1 -- 172.21.15.148:0/3760717846 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980abb00 con 0x7f87900039b0 2023-10-30T13:42:44.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 -- 172.21.15.148:0/3760717846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900039b0 msgr2=0x7f8790003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 --2- 172.21.15.148:0/3760717846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900039b0 0x7f8790003d90 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f8798001860 tx=0x7f879801c4d0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87aca3b640 1 -- 172.21.15.148:0/3760717846 reap_dead start 2023-10-30T13:42:44.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 -- 172.21.15.148:0/3760717846 shutdown_connections 2023-10-30T13:42:44.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 -- 172.21.15.148:0/3760717846 >> 172.21.15.148:0/3760717846 conn(0x7f87a80c7910 msgr2=0x7f87a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:44.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 -- 172.21.15.148:0/3760717846 shutdown_connections 2023-10-30T13:42:44.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 -- 172.21.15.148:0/3760717846 wait complete. 2023-10-30T13:42:44.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 Processor -- start 2023-10-30T13:42:44.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 -- start start 2023-10-30T13:42:44.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87900039b0 0x7f87a80c2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87a8156830 0x7f87a80c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a80c4520 0x7f87a80c2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a80c1310 con 0x7f87a8156830 2023-10-30T13:42:44.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a80c3840 con 0x7f87900039b0 2023-10-30T13:42:44.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87ae49e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a80c4f60 con 0x7f87a80c4520 2023-10-30T13:42:44.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.171+0000 7f87a77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87a8156830 0x7f87a80c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87a8156830 0x7f87a80c3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32872/0 (socket says 172.21.15.148:32872) 2023-10-30T13:42:44.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a77fe640 1 -- 172.21.15.148:0/3054160052 learned_addr learned my addr 172.21.15.148:0/3054160052 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:44.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a7fff640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87900039b0 0x7f87a80c2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87aca3b640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a80c4520 0x7f87a80c2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a77fe640 1 -- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a80c4520 msgr2=0x7f87a80c2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a77fe640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a80c4520 0x7f87a80c2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a77fe640 1 -- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87900039b0 msgr2=0x7f87a80c2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a77fe640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87900039b0 0x7f87a80c2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a77fe640 1 -- 172.21.15.148:0/3054160052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a80c5240 con 0x7f87a8156830 2023-10-30T13:42:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a77fe640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87a8156830 0x7f87a80c3fe0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f879c001ec0 tx=0x7f879c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a4ff9640 1 -- 172.21.15.148:0/3054160052 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c00ee00 con 0x7f87a8156830 2023-10-30T13:42:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a4ff9640 1 -- 172.21.15.148:0/3054160052 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879c00c8d0 con 0x7f87a8156830 2023-10-30T13:42:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a80c0d50 con 0x7f87a8156830 2023-10-30T13:42:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a4ff9640 1 -- 172.21.15.148:0/3054160052 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879c00cc80 con 0x7f87a8156830 2023-10-30T13:42:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a8156c70 con 0x7f87a8156830 2023-10-30T13:42:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a80c3c80 con 0x7f87a8156830 2023-10-30T13:42:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a4ff9640 1 -- 172.21.15.148:0/3054160052 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f879c01d460 con 0x7f87a8156830 2023-10-30T13:42:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a4ff9640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87940664a0 0x7f8794068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.175+0000 7f87a4ff9640 1 -- 172.21.15.148:0/3054160052 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f879c084320 con 0x7f87a8156830 2023-10-30T13:42:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.179+0000 7f87a7fff640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87940664a0 0x7f8794068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.179+0000 7f87a7fff640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87940664a0 0x7f8794068960 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f87980af000 tx=0x7f87980a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.179+0000 7f87a4ff9640 1 -- 172.21.15.148:0/3054160052 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87a80c3c80 con 0x7f87a8156830 2023-10-30T13:42:44.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87a80c3c80 con 0x7f87a8156830 2023-10-30T13:42:44.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87a4ff9640 1 -- 172.21.15.148:0/3054160052 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f87a80c3c80 con 0x7f87a8156830 2023-10-30T13:42:44.363 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:44.363 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:39.785575+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:44.364 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 121 2023-10-30T13:42:44.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87940664a0 msgr2=0x7f8794068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87940664a0 0x7f8794068960 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f87980af000 tx=0x7f87980a7000 comp rx=0 tx=0).stop 2023-10-30T13:42:44.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87a8156830 msgr2=0x7f87a80c3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:44.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87a8156830 0x7f87a80c3fe0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f879c001ec0 tx=0x7f879c00ec10 comp rx=0 tx=0).stop 2023-10-30T13:42:44.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 shutdown_connections 2023-10-30T13:42:44.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87940664a0 0x7f8794068960 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87a8156830 0x7f87a80c3fe0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87a80c4520 0x7f87a80c2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 --2- 172.21.15.148:0/3054160052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87900039b0 0x7f87a80c2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:44.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 >> 172.21.15.148:0/3054160052 conn(0x7f87a80c7910 msgr2=0x7f87a8070e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:44.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 shutdown_connections 2023-10-30T13:42:44.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:44.363+0000 7f87ae49e640 1 -- 172.21.15.148:0/3054160052 wait complete. 2023-10-30T13:42:44.384 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 121, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7305": { "addr": "172.21.15.148:6835/3829165476", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 3829165476, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 3829165476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7305, "incarnation": 120, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:39.785575+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7305 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:46.385 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:46.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd546c17640 1 Processor -- start 2023-10-30T13:42:46.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd546c17640 1 -- start start 2023-10-30T13:42:46.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd546c17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5400cf160 0x7fd5400cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd546c17640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5400cb800 con 0x7fd54015d6f0 2023-10-30T13:42:46.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd546c17640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5400cb330 con 0x7fd5401538a0 2023-10-30T13:42:46.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd546c17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5400cad70 con 0x7fd5400cf160 2023-10-30T13:42:46.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd54015d6f0 0x7fd54015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59736/0 (socket says 172.21.15.148:59736) 2023-10-30T13:42:46.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53ffff640 1 -- 172.21.15.148:0/4137670835 learned_addr learned my addr 172.21.15.148:0/4137670835 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:46.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd5451b4640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5400cf160 0x7fd5400cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728628700 0 0) 0x7fd5400cb800 con 0x7fd54015d6f0 2023-10-30T13:42:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd530003530 con 0x7fd54015d6f0 2023-10-30T13:42:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 524368780 0 0) 0x7fd530003530 con 0x7fd54015d6f0 2023-10-30T13:42:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5400cf160 msgr2=0x7fd5400cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5400cf160 0x7fd5400cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 >> v1:172.21.15.150:6789/0 conn(0x7fd5401538a0 legacy=0x7fd5400ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5400cb050 con 0x7fd54015d6f0 2023-10-30T13:42:46.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fd52c001680 con 0x7fd54015d6f0 2023-10-30T13:42:46.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 >> v1:172.21.15.148:6789/0 conn(0x7fd54015d6f0 legacy=0x7fd54015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5300039b0 0x7fd530003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5400cf160 0x7fd5300047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd530004d10 0x7fd530005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5400cad70 con 0x7fd530004d10 2023-10-30T13:42:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53000bad0 con 0x7fd5300039b0 2023-10-30T13:42:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53000bc90 con 0x7fd5400cf160 2023-10-30T13:42:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd5451b4640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd530004d10 0x7fd530005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd53ffff640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5400cf160 0x7fd5300047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd5449b3640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5300039b0 0x7fd530003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd5451b4640 1 -- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5400cf160 msgr2=0x7fd5300047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd5451b4640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd5400cf160 0x7fd5300047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd5451b4640 1 -- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5300039b0 msgr2=0x7fd530003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:46.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd5451b4640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5300039b0 0x7fd530003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.543+0000 7fd5451b4640 1 -- 172.21.15.148:0/4137670835 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530003530 con 0x7fd530004d10 2023-10-30T13:42:46.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd5451b4640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd530004d10 0x7fd530005190 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fd538009f90 tx=0x7fd53801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe27deefdeb0ca0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:46.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5380a5c30 con 0x7fd530004d10 2023-10-30T13:42:46.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53800f040 con 0x7fd530004d10 2023-10-30T13:42:46.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd53f7fe640 1 -- 172.21.15.148:0/4137670835 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5380ad440 con 0x7fd530004d10 2023-10-30T13:42:46.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd546c17640 1 -- 172.21.15.148:0/4137670835 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd530004d10 msgr2=0x7fd530005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd546c17640 1 --2- 172.21.15.148:0/4137670835 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd530004d10 0x7fd530005190 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fd538009f90 tx=0x7fd53801cc20 comp rx=0 tx=0).stop 2023-10-30T13:42:46.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd5451b4640 1 -- 172.21.15.148:0/4137670835 reap_dead start 2023-10-30T13:42:46.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd546c17640 1 -- 172.21.15.148:0/4137670835 shutdown_connections 2023-10-30T13:42:46.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd546c17640 1 -- 172.21.15.148:0/4137670835 >> 172.21.15.148:0/4137670835 conn(0x7fd54006bfe0 msgr2=0x7fd54006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:46.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd546c17640 1 -- 172.21.15.148:0/4137670835 shutdown_connections 2023-10-30T13:42:46.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.547+0000 7fd546c17640 1 -- 172.21.15.148:0/4137670835 wait complete. 2023-10-30T13:42:46.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 Processor -- start 2023-10-30T13:42:46.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 -- start start 2023-10-30T13:42:46.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd530004d10 0x7fd5401e2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd54015aab0 0x7fd54015aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5401f2a40 0x7fd5401f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5401f03a0 con 0x7fd54015aab0 2023-10-30T13:42:46.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54014d900 con 0x7fd5401f2a40 2023-10-30T13:42:46.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5401ef4d0 con 0x7fd530004d10 2023-10-30T13:42:46.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd54015aab0 0x7fd54015aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd54015aab0 0x7fd54015aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32888/0 (socket says 172.21.15.148:32888) 2023-10-30T13:42:46.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53ffff640 1 -- 172.21.15.148:0/2422211049 learned_addr learned my addr 172.21.15.148:0/2422211049 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:46.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd5449b3640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd530004d10 0x7fd5401e2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd5451b4640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5401f2a40 0x7fd5401f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53ffff640 1 -- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd530004d10 msgr2=0x7fd5401e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53ffff640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd530004d10 0x7fd5401e2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53ffff640 1 -- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5401f2a40 msgr2=0x7fd5401f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53ffff640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5401f2a40 0x7fd5401f2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53ffff640 1 -- 172.21.15.148:0/2422211049 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54015b430 con 0x7fd54015aab0 2023-10-30T13:42:46.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53ffff640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd54015aab0 0x7fd54015aef0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e7b0 tx=0x7fd52c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:46.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53d7fa640 1 -- 172.21.15.148:0/2422211049 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c00c8d0 con 0x7fd54015aab0 2023-10-30T13:42:46.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53d7fa640 1 -- 172.21.15.148:0/2422211049 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c00ca50 con 0x7fd54015aab0 2023-10-30T13:42:46.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd53d7fa640 1 -- 172.21.15.148:0/2422211049 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c00ce00 con 0x7fd54015aab0 2023-10-30T13:42:46.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5401ef790 con 0x7fd54015aab0 2023-10-30T13:42:46.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.551+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5401efd50 con 0x7fd54015aab0 2023-10-30T13:42:46.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.555+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5400cc7b0 con 0x7fd54015aab0 2023-10-30T13:42:46.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.555+0000 7fd53d7fa640 1 -- 172.21.15.148:0/2422211049 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd52c01d460 con 0x7fd54015aab0 2023-10-30T13:42:46.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.555+0000 7fd53d7fa640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd5240664a0 0x7fd524068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:46.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.555+0000 7fd53d7fa640 1 -- 172.21.15.148:0/2422211049 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7fd52c0845d0 con 0x7fd54015aab0 2023-10-30T13:42:46.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.555+0000 7fd5449b3640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd5240664a0 0x7fd524068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:46.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.555+0000 7fd5449b3640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd5240664a0 0x7fd524068960 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd534005510 tx=0x7fd534009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:46.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.559+0000 7fd53d7fa640 1 -- 172.21.15.148:0/2422211049 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd52c0513c0 con 0x7fd54015aab0 2023-10-30T13:42:46.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.739+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd54004e170 con 0x7fd54015aab0 2023-10-30T13:42:46.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd53d7fa640 1 -- 172.21.15.148:0/2422211049 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5720 (secure 0 0 0) 0x7fd52c0549e0 con 0x7fd54015aab0 2023-10-30T13:42:46.742 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:46.743 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:45.072515+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:46.746 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 122 2023-10-30T13:42:46.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd5240664a0 msgr2=0x7fd524068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd5240664a0 0x7fd524068960 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd534005510 tx=0x7fd534009040 comp rx=0 tx=0).stop 2023-10-30T13:42:46.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd54015aab0 msgr2=0x7fd54015aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:46.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd54015aab0 0x7fd54015aef0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e7b0 tx=0x7fd52c00ec70 comp rx=0 tx=0).stop 2023-10-30T13:42:46.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 shutdown_connections 2023-10-30T13:42:46.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd5240664a0 0x7fd524068960 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd54015aab0 0x7fd54015aef0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5401f2a40 0x7fd5401f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 --2- 172.21.15.148:0/2422211049 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd530004d10 0x7fd5401e2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:46.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 >> 172.21.15.148:0/2422211049 conn(0x7fd54006bfe0 msgr2=0x7fd5400cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:46.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 shutdown_connections 2023-10-30T13:42:46.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:46.743+0000 7fd546c17640 1 -- 172.21.15.148:0/2422211049 wait complete. 2023-10-30T13:42:46.763 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:48.764 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:48.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f68518d6640 1 Processor -- start 2023-10-30T13:42:48.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f68518d6640 1 -- start start 2023-10-30T13:42:48.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f68518d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c1538a0 0x7f684c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f68518d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f684c0ca0e0 0x7f684c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f68518d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684c0cb800 con 0x7f684c0cf160 2023-10-30T13:42:48.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f68518d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684c15dc30 con 0x7f684c0ca0e0 2023-10-30T13:42:48.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f68518d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0cb330 con 0x7f684c1538a0 2023-10-30T13:42:48.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f684c0cf160 0x7f684c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59746/0 (socket says 172.21.15.148:59746) 2023-10-30T13:42:48.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684b7fe640 1 -- 172.21.15.148:0/3986092430 learned_addr learned my addr 172.21.15.148:0/3986092430 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:48.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f6843fff640 1 --2- 172.21.15.148:0/3986092430 >> v2:172.21.15.150:3300/0 conn(0x7f684c0ca0e0 0x7f684c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684affd640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c1538a0 0x7f684c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236911712 0 0) 0x7f684c0cb800 con 0x7f684c0cf160 2023-10-30T13:42:48.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6838003530 con 0x7f684c0cf160 2023-10-30T13:42:48.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1383124321 0 0) 0x7f6838003530 con 0x7f684c0cf160 2023-10-30T13:42:48.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c1538a0 msgr2=0x7f684c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:48.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c1538a0 0x7f684c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 >> v2:172.21.15.150:3300/0 conn(0x7f684c0ca0e0 msgr2=0x7f684c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:48.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 --2- 172.21.15.148:0/3986092430 >> v2:172.21.15.150:3300/0 conn(0x7f684c0ca0e0 0x7f684c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c0cad70 con 0x7f684c0cf160 2023-10-30T13:42:48.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f6844001a00 con 0x7f684c0cf160 2023-10-30T13:42:48.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 >> v1:172.21.15.148:6789/0 conn(0x7f684c0cf160 legacy=0x7f684c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f6838003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68380047d0 0x7f6838004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c1538a0 0x7f6838005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c15dc30 con 0x7f6838003980 2023-10-30T13:42:48.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0cb330 con 0x7f68380047d0 2023-10-30T13:42:48.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683800b990 con 0x7f684c1538a0 2023-10-30T13:42:48.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684affd640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f6838003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f6843fff640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68380047d0 0x7f6838004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684b7fe640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c1538a0 0x7f6838005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684affd640 1 -- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c1538a0 msgr2=0x7f6838005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:48.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684affd640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c1538a0 0x7f6838005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684affd640 1 -- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68380047d0 msgr2=0x7f6838004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:48.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684affd640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68380047d0 0x7f6838004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.915+0000 7f684affd640 1 -- 172.21.15.148:0/3986092430 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6838003530 con 0x7f6838003980 2023-10-30T13:42:48.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f684affd640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f6838003da0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f683c009ae0 tx=0x7f683c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409dc81ceb03b69d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:48.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0a6460 con 0x7f6838003980 2023-10-30T13:42:48.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c0a65c0 con 0x7f6838003980 2023-10-30T13:42:48.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f684a7fc640 1 -- 172.21.15.148:0/3986092430 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0a6970 con 0x7f6838003980 2023-10-30T13:42:48.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f68518d6640 1 -- 172.21.15.148:0/3986092430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 msgr2=0x7f6838003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f68518d6640 1 --2- 172.21.15.148:0/3986092430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f6838003da0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f683c009ae0 tx=0x7f683c01ca50 comp rx=0 tx=0).stop 2023-10-30T13:42:48.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f684b7fe640 1 -- 172.21.15.148:0/3986092430 reap_dead start 2023-10-30T13:42:48.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f68518d6640 1 -- 172.21.15.148:0/3986092430 shutdown_connections 2023-10-30T13:42:48.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f68518d6640 1 -- 172.21.15.148:0/3986092430 >> 172.21.15.148:0/3986092430 conn(0x7f684c06bfe0 msgr2=0x7f684c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:48.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f68518d6640 1 -- 172.21.15.148:0/3986092430 shutdown_connections 2023-10-30T13:42:48.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.919+0000 7f68518d6640 1 -- 172.21.15.148:0/3986092430 wait complete. 2023-10-30T13:42:48.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 Processor -- start 2023-10-30T13:42:48.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 -- start start 2023-10-30T13:42:48.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f684c1efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c158a60 0x7f684c144390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f684c144ea0 0x7f684c0b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c159000 con 0x7f6838003980 2023-10-30T13:42:48.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c1592c0 con 0x7f684c144ea0 2023-10-30T13:42:48.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c1595a0 con 0x7f684c158a60 2023-10-30T13:42:48.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f684c1efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f684c1efcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32906/0 (socket says 172.21.15.148:32906) 2023-10-30T13:42:48.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684affd640 1 -- 172.21.15.148:0/1770579552 learned_addr learned my addr 172.21.15.148:0/1770579552 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:48.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f6843fff640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c158a60 0x7f684c144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684b7fe640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f684c144ea0 0x7f684c0b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684affd640 1 -- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c158a60 msgr2=0x7f684c144390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684affd640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c158a60 0x7f684c144390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684affd640 1 -- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f684c144ea0 msgr2=0x7f684c0b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:48.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684affd640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f684c144ea0 0x7f684c0b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:48.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684affd640 1 -- 172.21.15.148:0/1770579552 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c1f27f0 con 0x7f6838003980 2023-10-30T13:42:48.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684b7fe640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f684c144ea0 0x7f684c0b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:42:48.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f684affd640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f684c1efcd0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f683c009ae0 tx=0x7f6838005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:48.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f6848ff9640 1 -- 172.21.15.148:0/1770579552 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0a7440 con 0x7f6838003980 2023-10-30T13:42:48.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f6848ff9640 1 -- 172.21.15.148:0/1770579552 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c0a75a0 con 0x7f6838003980 2023-10-30T13:42:48.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684c159000 con 0x7f6838003980 2023-10-30T13:42:48.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f6848ff9640 1 -- 172.21.15.148:0/1770579552 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0a7930 con 0x7f6838003980 2023-10-30T13:42:48.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.923+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684c1f2db0 con 0x7f6838003980 2023-10-30T13:42:48.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.927+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684c0c4070 con 0x7f6838003980 2023-10-30T13:42:48.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.927+0000 7f6848ff9640 1 -- 172.21.15.148:0/1770579552 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f683c0a7a90 con 0x7f6838003980 2023-10-30T13:42:48.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.927+0000 7f6848ff9640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f682c066470 0x7f682c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:48.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.927+0000 7f6848ff9640 1 -- 172.21.15.148:0/1770579552 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f683c11e5c0 con 0x7f6838003980 2023-10-30T13:42:48.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.927+0000 7f6843fff640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f682c066470 0x7f682c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:48.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.927+0000 7f6843fff640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f682c066470 0x7f682c068930 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f684c1572c0 tx=0x7f6834008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:48.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:48.931+0000 7f6848ff9640 1 -- 172.21.15.148:0/1770579552 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f683c0eb3b0 con 0x7f6838003980 2023-10-30T13:42:49.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.115+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f684c04e170 con 0x7f6838003980 2023-10-30T13:42:49.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.115+0000 7f6848ff9640 1 -- 172.21.15.148:0/1770579552 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f683c0ee9d0 con 0x7f6838003980 2023-10-30T13:42:49.117 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:49.118 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:45.072515+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:49.118 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 122 2023-10-30T13:42:49.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f682c066470 msgr2=0x7f682c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:49.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f682c066470 0x7f682c068930 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f684c1572c0 tx=0x7f6834008040 comp rx=0 tx=0).stop 2023-10-30T13:42:49.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 msgr2=0x7f684c1efcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:49.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f684c1efcd0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f683c009ae0 tx=0x7f6838005800 comp rx=0 tx=0).stop 2023-10-30T13:42:49.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 shutdown_connections 2023-10-30T13:42:49.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f682c066470 0x7f682c068930 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f684c144ea0 0x7f684c0b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f684c158a60 0x7f684c144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 --2- 172.21.15.148:0/1770579552 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6838003980 0x7f684c1efcd0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:49.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 >> 172.21.15.148:0/1770579552 conn(0x7f684c06bfe0 msgr2=0x7f684c151860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:49.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 shutdown_connections 2023-10-30T13:42:49.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:49.119+0000 7f68518d6640 1 -- 172.21.15.148:0/1770579552 wait complete. 2023-10-30T13:42:49.138 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:51.140 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:51.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc9572640 1 Processor -- start 2023-10-30T13:42:51.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc9572640 1 -- start start 2023-10-30T13:42:51.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc9572640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc415d6f0 0x7fdbc415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc9572640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdbc40cf160 0x7fdbc40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc9572640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc40cb800 con 0x7fdbc41538a0 2023-10-30T13:42:51.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc9572640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc40cb330 con 0x7fdbc40cf160 2023-10-30T13:42:51.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc9572640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40cad70 con 0x7fdbc415d6f0 2023-10-30T13:42:51.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc37fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdbc40cf160 0x7fdbc40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc2ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fdbc41538a0 0x7fdbc40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59750/0 (socket says 172.21.15.148:59750) 2023-10-30T13:42:51.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc2ffd640 1 -- 172.21.15.148:0/3706854733 learned_addr learned my addr 172.21.15.148:0/3706854733 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:51.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbbbfff640 1 --2- 172.21.15.148:0/3706854733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc415d6f0 0x7fdbc415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc37fe640 1 -- 172.21.15.148:0/3706854733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc415d6f0 msgr2=0x7fdbc415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:51.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc37fe640 1 --2- 172.21.15.148:0/3706854733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc415d6f0 0x7fdbc415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc37fe640 1 -- 172.21.15.148:0/3706854733 >> v1:172.21.15.148:6789/0 conn(0x7fdbc41538a0 legacy=0x7fdbc40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.283+0000 7fdbc37fe640 1 -- 172.21.15.148:0/3706854733 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40cb050 con 0x7fdbc40cf160 2023-10-30T13:42:51.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc37fe640 1 --2- 172.21.15.148:0/3706854733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 0x7fdbc40cf590 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009d90 tx=0x7fdbbc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db02a47fa8f47e24 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:51.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc27fc640 1 -- 172.21.15.148:0/3706854733 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc0af600 con 0x7fdbc40cf160 2023-10-30T13:42:51.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc27fc640 1 -- 172.21.15.148:0/3706854733 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc0ab050 con 0x7fdbc40cf160 2023-10-30T13:42:51.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc27fc640 1 -- 172.21.15.148:0/3706854733 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc0af980 con 0x7fdbc40cf160 2023-10-30T13:42:51.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc9572640 1 -- 172.21.15.148:0/3706854733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 msgr2=0x7fdbc40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc9572640 1 --2- 172.21.15.148:0/3706854733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 0x7fdbc40cf590 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009d90 tx=0x7fdbbc01f130 comp rx=0 tx=0).stop 2023-10-30T13:42:51.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc9572640 1 -- 172.21.15.148:0/3706854733 shutdown_connections 2023-10-30T13:42:51.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc9572640 1 --2- 172.21.15.148:0/3706854733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 0x7fdbc40cf590 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc9572640 1 --2- 172.21.15.148:0/3706854733 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc415d6f0 0x7fdbc415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc9572640 1 -- 172.21.15.148:0/3706854733 >> 172.21.15.148:0/3706854733 conn(0x7fdbc406bfe0 msgr2=0x7fdbc406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:51.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc9572640 1 -- 172.21.15.148:0/3706854733 shutdown_connections 2023-10-30T13:42:51.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.287+0000 7fdbc9572640 1 -- 172.21.15.148:0/3706854733 wait complete. 2023-10-30T13:42:51.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 Processor -- start 2023-10-30T13:42:51.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 -- start start 2023-10-30T13:42:51.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 0x7fdbc41e49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc41538a0 0x7fdbc40b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbc415d6f0 0x7fdbc41f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc41e3180 con 0x7fdbc40cf160 2023-10-30T13:42:51.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc41e2a10 con 0x7fdbc415d6f0 2023-10-30T13:42:51.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc41e2cf0 con 0x7fdbc41538a0 2023-10-30T13:42:51.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 0x7fdbc41e49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 0x7fdbc41e49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32914/0 (socket says 172.21.15.148:32914) 2023-10-30T13:42:51.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc2ffd640 1 -- 172.21.15.148:0/3273987074 learned_addr learned my addr 172.21.15.148:0/3273987074 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:51.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbbbfff640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc41538a0 0x7fdbc40b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc37fe640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbc415d6f0 0x7fdbc41f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc2ffd640 1 -- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc41538a0 msgr2=0x7fdbc40b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc2ffd640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc41538a0 0x7fdbc40b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc2ffd640 1 -- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbc415d6f0 msgr2=0x7fdbc41f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc2ffd640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbc415d6f0 0x7fdbc41f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc2ffd640 1 -- 172.21.15.148:0/3273987074 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc009a70 con 0x7fdbc40cf160 2023-10-30T13:42:51.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc2ffd640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 0x7fdbc41e49e0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400c500 tx=0x7fdbb400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:51.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbbb7fe640 1 -- 172.21.15.148:0/3273987074 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4005cf0 con 0x7fdbc40cf160 2023-10-30T13:42:51.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbbb7fe640 1 -- 172.21.15.148:0/3273987074 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb4005e50 con 0x7fdbc40cf160 2023-10-30T13:42:51.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc40b4a90 con 0x7fdbc40cf160 2023-10-30T13:42:51.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbbb7fe640 1 -- 172.21.15.148:0/3273987074 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40072f0 con 0x7fdbc40cf160 2023-10-30T13:42:51.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.291+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc40b4d30 con 0x7fdbc40cf160 2023-10-30T13:42:51.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.295+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc40b2a50 con 0x7fdbc40cf160 2023-10-30T13:42:51.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.295+0000 7fdbbb7fe640 1 -- 172.21.15.148:0/3273987074 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdbb4007450 con 0x7fdbc40cf160 2023-10-30T13:42:51.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.295+0000 7fdbbb7fe640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdba40664a0 0x7fdba4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:51.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.295+0000 7fdbbb7fe640 1 -- 172.21.15.148:0/3273987074 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7fdbb40839d0 con 0x7fdbc40cf160 2023-10-30T13:42:51.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.295+0000 7fdbbbfff640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdba40664a0 0x7fdba4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:51.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.295+0000 7fdbbbfff640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdba40664a0 0x7fdba4068960 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40c5da0 tx=0x7fdbac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:51.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.299+0000 7fdbbb7fe640 1 -- 172.21.15.148:0/3273987074 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdbb4050870 con 0x7fdbc40cf160 2023-10-30T13:42:51.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.511+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbc404e170 con 0x7fdbc40cf160 2023-10-30T13:42:51.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbbb7fe640 1 -- 172.21.15.148:0/3273987074 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5720 (secure 0 0 0) 0x7fdbb4017090 con 0x7fdbc40cf160 2023-10-30T13:42:51.514 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:51.515 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:45.072515+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:51.515 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 122 2023-10-30T13:42:51.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdba40664a0 msgr2=0x7fdba4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdba40664a0 0x7fdba4068960 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40c5da0 tx=0x7fdbac008040 comp rx=0 tx=0).stop 2023-10-30T13:42:51.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 msgr2=0x7fdbc41e49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:51.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 0x7fdbc41e49e0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400c500 tx=0x7fdbb400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 shutdown_connections 2023-10-30T13:42:51.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdba40664a0 0x7fdba4068960 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbc415d6f0 0x7fdbc41f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdbc41538a0 0x7fdbc40b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 --2- 172.21.15.148:0/3273987074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbc40cf160 0x7fdbc41e49e0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:51.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 >> 172.21.15.148:0/3273987074 conn(0x7fdbc406bfe0 msgr2=0x7fdbc4151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:51.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 shutdown_connections 2023-10-30T13:42:51.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:51.515+0000 7fdbc9572640 1 -- 172.21.15.148:0/3273987074 wait complete. 2023-10-30T13:42:51.543 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:53.544 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2daae54640 1 Processor -- start 2023-10-30T13:42:53.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2daae54640 1 -- start start 2023-10-30T13:42:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2daae54640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2da40c05d0 0x7f2da413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2daae54640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c6f60 0x7f2da413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2daae54640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da40c0a50 con 0x7f2da40c6b80 2023-10-30T13:42:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2daae54640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da406f6e0 con 0x7f2da40c05d0 2023-10-30T13:42:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2daae54640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da406f9c0 con 0x7f2da40c6f60 2023-10-30T13:42:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2da8bf0640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2da40c6b80 0x7f2da40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44576/0 (socket says 172.21.15.148:44576) 2023-10-30T13:42:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2da8bf0640 1 -- 172.21.15.148:0/4205617811 learned_addr learned my addr 172.21.15.148:0/4205617811 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2d9bfff640 1 --2- 172.21.15.148:0/4205617811 >> v2:172.21.15.148:3300/0 conn(0x7f2da40c05d0 0x7f2da413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:53.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2da93f1640 1 --2- 172.21.15.148:0/4205617811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c6f60 0x7f2da413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:53.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2d9bfff640 1 -- 172.21.15.148:0/4205617811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c6f60 msgr2=0x7f2da413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:53.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2d9bfff640 1 --2- 172.21.15.148:0/4205617811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c6f60 0x7f2da413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2d9bfff640 1 -- 172.21.15.148:0/4205617811 >> v1:172.21.15.148:6789/0 conn(0x7f2da40c6b80 legacy=0x7f2da40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:53.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.691+0000 7f2d9bfff640 1 -- 172.21.15.148:0/4205617811 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da406fca0 con 0x7f2da40c05d0 2023-10-30T13:42:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2d9bfff640 1 --2- 172.21.15.148:0/4205617811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c05d0 0x7f2da413ed90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2d940097c0 tx=0x7f2d9401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fd640f6f738646 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2d9b7fe640 1 -- 172.21.15.148:0/4205617811 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d940ad640 con 0x7f2da40c05d0 2023-10-30T13:42:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2d9b7fe640 1 -- 172.21.15.148:0/4205617811 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d940ad7a0 con 0x7f2da40c05d0 2023-10-30T13:42:53.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2d9b7fe640 1 -- 172.21.15.148:0/4205617811 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d940adb30 con 0x7f2da40c05d0 2023-10-30T13:42:53.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2daae54640 1 -- 172.21.15.148:0/4205617811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c05d0 msgr2=0x7f2da413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:53.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2daae54640 1 --2- 172.21.15.148:0/4205617811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c05d0 0x7f2da413ed90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f2d940097c0 tx=0x7f2d9401c480 comp rx=0 tx=0).stop 2023-10-30T13:42:53.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2daae54640 1 -- 172.21.15.148:0/4205617811 shutdown_connections 2023-10-30T13:42:53.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2daae54640 1 --2- 172.21.15.148:0/4205617811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c05d0 0x7f2da413ed90 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2daae54640 1 --2- 172.21.15.148:0/4205617811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c6f60 0x7f2da413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.699 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.695+0000 7f2daae54640 1 -- 172.21.15.148:0/4205617811 >> 172.21.15.148:0/4205617811 conn(0x7f2da40c7910 msgr2=0x7f2da40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:53.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 -- 172.21.15.148:0/4205617811 shutdown_connections 2023-10-30T13:42:53.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 -- 172.21.15.148:0/4205617811 wait complete. 2023-10-30T13:42:53.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 Processor -- start 2023-10-30T13:42:53.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 -- start start 2023-10-30T13:42:53.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c05d0 0x7f2da40aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:53.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2da40c6b80 0x7f2da4135210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c6f60 0x7f2da4135750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da41f2ce0 con 0x7f2da40c6f60 2023-10-30T13:42:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da41f2740 con 0x7f2da40c6b80 2023-10-30T13:42:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2daae54640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da41f5ed0 con 0x7f2da40c05d0 2023-10-30T13:42:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da93f1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c6f60 0x7f2da4135750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:53.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da93f1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c6f60 0x7f2da4135750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40486/0 (socket says 172.21.15.148:40486) 2023-10-30T13:42:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da93f1640 1 -- 172.21.15.148:0/2533134591 learned_addr learned my addr 172.21.15.148:0/2533134591 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da8bf0640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c05d0 0x7f2da40aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2d9bfff640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2da40c6b80 0x7f2da4135210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da93f1640 1 -- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c05d0 msgr2=0x7f2da40aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da93f1640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c05d0 0x7f2da40aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da93f1640 1 -- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2da40c6b80 msgr2=0x7f2da4135210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da93f1640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2da40c6b80 0x7f2da4135210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da93f1640 1 -- 172.21.15.148:0/2533134591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d94009420 con 0x7f2da40c6f60 2023-10-30T13:42:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da8bf0640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c05d0 0x7f2da40aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.699+0000 7f2da93f1640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c6f60 0x7f2da4135750 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c007e60 tx=0x7f2d9c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2d997fa640 1 -- 172.21.15.148:0/2533134591 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c014420 con 0x7f2da40c6f60 2023-10-30T13:42:53.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2d997fa640 1 -- 172.21.15.148:0/2533134591 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c0145a0 con 0x7f2da40c6f60 2023-10-30T13:42:53.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da41f2a00 con 0x7f2da40c6f60 2023-10-30T13:42:53.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2d997fa640 1 -- 172.21.15.148:0/2533134591 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c014950 con 0x7f2da40c6f60 2023-10-30T13:42:53.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da40af340 con 0x7f2da40c6f60 2023-10-30T13:42:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da4134b50 con 0x7f2da40c6f60 2023-10-30T13:42:53.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2d997fa640 1 -- 172.21.15.148:0/2533134591 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2d9c014ab0 con 0x7f2da40c6f60 2023-10-30T13:42:53.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2d997fa640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d900664a0 0x7f2d90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:53.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2d997fa640 1 -- 172.21.15.148:0/2533134591 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f2d9c084e30 con 0x7f2da40c6f60 2023-10-30T13:42:53.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.703+0000 7f2da8bf0640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d900664a0 0x7f2d90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:53.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.707+0000 7f2da8bf0640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d900664a0 0x7f2d90068960 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2da0000f70 tx=0x7f2da0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:53.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.707+0000 7f2d997fa640 1 -- 172.21.15.148:0/2533134591 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2da4134b50 con 0x7f2da40c6f60 2023-10-30T13:42:53.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.887+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2da4134b50 con 0x7f2da40c6f60 2023-10-30T13:42:53.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2d997fa640 1 -- 172.21.15.148:0/2533134591 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f2da4134b50 con 0x7f2da40c6f60 2023-10-30T13:42:53.890 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:53.890 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:51.580494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:53.891 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 123 2023-10-30T13:42:53.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d900664a0 msgr2=0x7f2d90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:53.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d900664a0 0x7f2d90068960 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2da0000f70 tx=0x7f2da0009040 comp rx=0 tx=0).stop 2023-10-30T13:42:53.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c6f60 msgr2=0x7f2da4135750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:53.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c6f60 0x7f2da4135750 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c007e60 tx=0x7f2d9c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 shutdown_connections 2023-10-30T13:42:53.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2d900664a0 0x7f2d90068960 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2da40c6f60 0x7f2da4135750 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2da40c6b80 0x7f2da4135210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 --2- 172.21.15.148:0/2533134591 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2da40c05d0 0x7f2da40aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:53.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 >> 172.21.15.148:0/2533134591 conn(0x7f2da40c7910 msgr2=0x7f2da40c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:53.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 shutdown_connections 2023-10-30T13:42:53.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:53.891+0000 7f2daae54640 1 -- 172.21.15.148:0/2533134591 wait complete. 2023-10-30T13:42:53.911 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:42:55.913 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:56.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69cd609640 1 Processor -- start 2023-10-30T13:42:56.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69cd609640 1 -- start start 2023-10-30T13:42:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69cd609640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c80cf160 0x7f69c80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69cd609640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c80cb800 con 0x7f69c815d6f0 2023-10-30T13:42:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69cd609640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c80cb330 con 0x7f69c81538a0 2023-10-30T13:42:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69cd609640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c80cad70 con 0x7f69c80cf160 2023-10-30T13:42:56.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69be7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f69c815d6f0 0x7f69c815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44578/0 (socket says 172.21.15.148:44578) 2023-10-30T13:42:56.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69be7fc640 1 -- 172.21.15.148:0/2293695799 learned_addr learned my addr 172.21.15.148:0/2293695799 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:56.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69c77fe640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c80cf160 0x7f69c80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415401935 0 0) 0x7f69c80cb800 con 0x7f69c815d6f0 2023-10-30T13:42:56.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.059+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b4003530 con 0x7f69c815d6f0 2023-10-30T13:42:56.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1530018509 0 0) 0x7f69b4003530 con 0x7f69c815d6f0 2023-10-30T13:42:56.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c80cf160 msgr2=0x7f69c80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:56.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c80cf160 0x7f69c80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 >> v1:172.21.15.150:6789/0 conn(0x7f69c81538a0 legacy=0x7f69c80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c80cb050 con 0x7f69c815d6f0 2023-10-30T13:42:56.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f69b8001680 con 0x7f69c815d6f0 2023-10-30T13:42:56.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 >> v1:172.21.15.148:6789/0 conn(0x7f69c815d6f0 legacy=0x7f69c815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c80cf160 0x7f69b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69b4003f30 0x7f69b40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b4004d10 0x7f69b4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c80cad70 con 0x7f69b4003f30 2023-10-30T13:42:56.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b400bb00 con 0x7f69b4004d10 2023-10-30T13:42:56.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b400bcc0 con 0x7f69c80cf160 2023-10-30T13:42:56.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69be7fc640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69b4003f30 0x7f69b40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c6ffd640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c80cf160 0x7f69b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69c77fe640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b4004d10 0x7f69b4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69be7fc640 1 -- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c80cf160 msgr2=0x7f69b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:56.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69be7fc640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c80cf160 0x7f69b40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69be7fc640 1 -- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b4004d10 msgr2=0x7f69b4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:56.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69be7fc640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b4004d10 0x7f69b4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.063+0000 7f69be7fc640 1 -- 172.21.15.148:0/2293695799 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b4003530 con 0x7f69b4003f30 2023-10-30T13:42:56.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69be7fc640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69b4003f30 0x7f69b40047d0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f69b800c240 tx=0x7f69b801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d289a09f4a4a93 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:56.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80a5b40 con 0x7f69b4003f30 2023-10-30T13:42:56.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b80a5ca0 con 0x7f69b4003f30 2023-10-30T13:42:56.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69c67fc640 1 -- 172.21.15.148:0/2293695799 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8005b20 con 0x7f69b4003f30 2023-10-30T13:42:56.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 -- 172.21.15.148:0/2293695799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69b4003f30 msgr2=0x7f69b40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 --2- 172.21.15.148:0/2293695799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69b4003f30 0x7f69b40047d0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f69b800c240 tx=0x7f69b801cc40 comp rx=0 tx=0).stop 2023-10-30T13:42:56.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69c77fe640 1 -- 172.21.15.148:0/2293695799 reap_dead start 2023-10-30T13:42:56.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 -- 172.21.15.148:0/2293695799 shutdown_connections 2023-10-30T13:42:56.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 -- 172.21.15.148:0/2293695799 >> 172.21.15.148:0/2293695799 conn(0x7f69c806bfe0 msgr2=0x7f69c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:56.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 -- 172.21.15.148:0/2293695799 shutdown_connections 2023-10-30T13:42:56.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 -- 172.21.15.148:0/2293695799 wait complete. 2023-10-30T13:42:56.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 Processor -- start 2023-10-30T13:42:56.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 -- start start 2023-10-30T13:42:56.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b4003f30 0x7f69c80ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.067+0000 7f69cd609640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69c80ccfb0 0x7f69c81eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69cd609640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c81eef40 0x7f69c81ef3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69cd609640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c814d790 con 0x7f69c80ccfb0 2023-10-30T13:42:56.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69cd609640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c81ef900 con 0x7f69b4003f30 2023-10-30T13:42:56.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69cd609640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c80ce1e0 con 0x7f69c81eef40 2023-10-30T13:42:56.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69be7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69c80ccfb0 0x7f69c81eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69be7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69c80ccfb0 0x7f69c81eea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40508/0 (socket says 172.21.15.148:40508) 2023-10-30T13:42:56.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69be7fc640 1 -- 172.21.15.148:0/377842440 learned_addr learned my addr 172.21.15.148:0/377842440 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:56.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69c6ffd640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b4003f30 0x7f69c80ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69c77fe640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c81eef40 0x7f69c81ef3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69be7fc640 1 -- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c81eef40 msgr2=0x7f69c81ef3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69be7fc640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c81eef40 0x7f69c81ef3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69be7fc640 1 -- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b4003f30 msgr2=0x7f69c80ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69be7fc640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b4003f30 0x7f69c80ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69be7fc640 1 -- 172.21.15.148:0/377842440 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8158110 con 0x7f69c80ccfb0 2023-10-30T13:42:56.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69be7fc640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69c80ccfb0 0x7f69c81eea00 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f69b801c6b0 tx=0x7f69b4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:56.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69bffff640 1 -- 172.21.15.148:0/377842440 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80b4070 con 0x7f69c80ccfb0 2023-10-30T13:42:56.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69bffff640 1 -- 172.21.15.148:0/377842440 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b8004a00 con 0x7f69c80ccfb0 2023-10-30T13:42:56.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c80cd8b0 con 0x7f69c80ccfb0 2023-10-30T13:42:56.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69bffff640 1 -- 172.21.15.148:0/377842440 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8004d90 con 0x7f69c80ccfb0 2023-10-30T13:42:56.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c80cde70 con 0x7f69c80ccfb0 2023-10-30T13:42:56.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c804e170 con 0x7f69c80ccfb0 2023-10-30T13:42:56.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69bffff640 1 -- 172.21.15.148:0/377842440 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69b8007070 con 0x7f69c80ccfb0 2023-10-30T13:42:56.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.071+0000 7f69bffff640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69a80664a0 0x7f69a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:56.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.075+0000 7f69bffff640 1 -- 172.21.15.148:0/377842440 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f69b811e590 con 0x7f69c80ccfb0 2023-10-30T13:42:56.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.075+0000 7f69c6ffd640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69a80664a0 0x7f69a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:56.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.075+0000 7f69c6ffd640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69a80664a0 0x7f69a8068960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f69b0000c10 tx=0x7f69b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:56.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.079+0000 7f69bffff640 1 -- 172.21.15.148:0/377842440 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69b80eb2f0 con 0x7f69c80ccfb0 2023-10-30T13:42:56.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.259+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69c80b1dd0 con 0x7f69c80ccfb0 2023-10-30T13:42:56.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.259+0000 7f69bffff640 1 -- 172.21.15.148:0/377842440 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f69b80ee910 con 0x7f69c80ccfb0 2023-10-30T13:42:56.260 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:56.260 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:51.580494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:56.261 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 123 2023-10-30T13:42:56.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.259+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69a80664a0 msgr2=0x7f69a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.259+0000 7f69cd609640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69a80664a0 0x7f69a8068960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f69b0000c10 tx=0x7f69b0009040 comp rx=0 tx=0).stop 2023-10-30T13:42:56.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.259+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69c80ccfb0 msgr2=0x7f69c81eea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:56.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.259+0000 7f69cd609640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69c80ccfb0 0x7f69c81eea00 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f69b801c6b0 tx=0x7f69b4005800 comp rx=0 tx=0).stop 2023-10-30T13:42:56.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.263+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 shutdown_connections 2023-10-30T13:42:56.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.263+0000 7f69cd609640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f69a80664a0 0x7f69a8068960 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.263+0000 7f69cd609640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69c80ccfb0 0x7f69c81eea00 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.263+0000 7f69cd609640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f69c81eef40 0x7f69c81ef3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.263+0000 7f69cd609640 1 --2- 172.21.15.148:0/377842440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69b4003f30 0x7f69c80ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:56.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.263+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 >> 172.21.15.148:0/377842440 conn(0x7f69c806bfe0 msgr2=0x7f69c8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:56.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.263+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 shutdown_connections 2023-10-30T13:42:56.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:56.263+0000 7f69cd609640 1 -- 172.21.15.148:0/377842440 wait complete. 2023-10-30T13:42:56.281 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 123, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 123, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7305": { "addr": "172.21.15.148:6835/3829165476", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 3829165476, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 3829165476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7305, "incarnation": 120, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:clientreplay", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:42:51.580494+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7305 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:42:58.281 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:42:58.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee9f3f8640 1 Processor -- start 2023-10-30T13:42:58.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee9f3f8640 1 -- start start 2023-10-30T13:42:58.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee9f3f8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c05d0 0x7fee9813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:58.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee9f3f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee980c0a50 con 0x7fee980c6b80 2023-10-30T13:42:58.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee9f3f8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9806f6e0 con 0x7fee980c6f60 2023-10-30T13:42:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee9f3f8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9806f9c0 con 0x7fee980c05d0 2023-10-30T13:42:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee9d194640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fee980c6b80 0x7fee980bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44592/0 (socket says 172.21.15.148:44592) 2023-10-30T13:42:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee9d194640 1 -- 172.21.15.148:0/1367285065 learned_addr learned my addr 172.21.15.148:0/1367285065 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:42:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee9c993640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c05d0 0x7fee9813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519353406 0 0) 0x7fee980c0a50 con 0x7fee980c6b80 2023-10-30T13:42:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee84005740 con 0x7fee980c6b80 2023-10-30T13:42:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781609662 0 0) 0x7fee84005740 con 0x7fee980c6b80 2023-10-30T13:42:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c05d0 msgr2=0x7fee9813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee8ffff640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c05d0 0x7fee9813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 >> v1:172.21.15.150:6789/0 conn(0x7fee980c6f60 legacy=0x7fee981566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9806fca0 con 0x7fee980c6b80 2023-10-30T13:42:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fee94001680 con 0x7fee980c6b80 2023-10-30T13:42:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.423+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 >> v1:172.21.15.148:6789/0 conn(0x7fee980c6b80 legacy=0x7fee980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee8ffff640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee84005bc0 0x7fee84005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee8ffff640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee84006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee8ffff640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c05d0 0x7fee840073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9806f9c0 con 0x7fee840069e0 2023-10-30T13:42:58.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8400dc80 con 0x7fee84005bc0 2023-10-30T13:42:58.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8400de40 con 0x7fee980c05d0 2023-10-30T13:42:58.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9c993640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee84006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:58.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9d194640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee84005bc0 0x7fee84005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:58.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9d995640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c05d0 0x7fee840073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9c993640 1 -- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c05d0 msgr2=0x7fee840073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9c993640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c05d0 0x7fee840073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9c993640 1 -- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee84005bc0 msgr2=0x7fee84005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:42:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9c993640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee84005bc0 0x7fee84005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9c993640 1 -- 172.21.15.148:0/1367285065 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee84005740 con 0x7fee840069e0 2023-10-30T13:42:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9d194640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee84005bc0 0x7fee84005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:42:58.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee9c993640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee84006e60 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fee88009ab0 tx=0x7fee8801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da8445e8da27baa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:58.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee880a5600 con 0x7fee840069e0 2023-10-30T13:42:58.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee880a5760 con 0x7fee840069e0 2023-10-30T13:42:58.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.427+0000 7fee8ffff640 1 -- 172.21.15.148:0/1367285065 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee880a5b10 con 0x7fee840069e0 2023-10-30T13:42:58.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 -- 172.21.15.148:0/1367285065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 msgr2=0x7fee84006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:58.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 --2- 172.21.15.148:0/1367285065 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee84006e60 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fee88009ab0 tx=0x7fee8801ca10 comp rx=0 tx=0).stop 2023-10-30T13:42:58.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9d995640 1 -- 172.21.15.148:0/1367285065 reap_dead start 2023-10-30T13:42:58.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 -- 172.21.15.148:0/1367285065 shutdown_connections 2023-10-30T13:42:58.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 -- 172.21.15.148:0/1367285065 >> 172.21.15.148:0/1367285065 conn(0x7fee980c7910 msgr2=0x7fee980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:58.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 -- 172.21.15.148:0/1367285065 shutdown_connections 2023-10-30T13:42:58.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 -- 172.21.15.148:0/1367285065 wait complete. 2023-10-30T13:42:58.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 Processor -- start 2023-10-30T13:42:58.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 -- start start 2023-10-30T13:42:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee9814f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c4e20 0x7fee981544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee981549e0 0x7fee981e49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee980c2f10 con 0x7fee840069e0 2023-10-30T13:42:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9814f630 con 0x7fee981549e0 2023-10-30T13:42:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9f3f8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee98154e70 con 0x7fee980c4e20 2023-10-30T13:42:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9d194640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee9814f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9d194640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee9814f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40534/0 (socket says 172.21.15.148:40534) 2023-10-30T13:42:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9d194640 1 -- 172.21.15.148:0/2826467789 learned_addr learned my addr 172.21.15.148:0/2826467789 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:42:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9c993640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c4e20 0x7fee981544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.431+0000 7fee9d995640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee981549e0 0x7fee981e49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee9d194640 1 -- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c4e20 msgr2=0x7fee981544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee9d194640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c4e20 0x7fee981544a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee9d194640 1 -- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee981549e0 msgr2=0x7fee981e49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee9d194640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee981549e0 0x7fee981e49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee9d194640 1 -- 172.21.15.148:0/2826467789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee980c2340 con 0x7fee840069e0 2023-10-30T13:42:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee9d194640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee9814f0f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fee94011970 tx=0x7fee94011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee8dffb640 1 -- 172.21.15.148:0/2826467789 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9400fc20 con 0x7fee840069e0 2023-10-30T13:42:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee8dffb640 1 -- 172.21.15.148:0/2826467789 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9400fd80 con 0x7fee840069e0 2023-10-30T13:42:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee8dffb640 1 -- 172.21.15.148:0/2826467789 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94015610 con 0x7fee840069e0 2023-10-30T13:42:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee980c25e0 con 0x7fee840069e0 2023-10-30T13:42:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee980c2ba0 con 0x7fee840069e0 2023-10-30T13:42:58.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee980c2d60 con 0x7fee840069e0 2023-10-30T13:42:58.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee8dffb640 1 -- 172.21.15.148:0/2826467789 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee94015770 con 0x7fee840069e0 2023-10-30T13:42:58.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee8dffb640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee7c0664d0 0x7fee7c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:42:58.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.435+0000 7fee8dffb640 1 -- 172.21.15.148:0/2826467789 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7fee94087880 con 0x7fee840069e0 2023-10-30T13:42:58.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.439+0000 7fee9c993640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee7c0664d0 0x7fee7c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:42:58.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.439+0000 7fee9c993640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee7c0664d0 0x7fee7c068990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fee880af000 tx=0x7fee880046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:42:58.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.443+0000 7fee8dffb640 1 -- 172.21.15.148:0/2826467789 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee94054670 con 0x7fee840069e0 2023-10-30T13:42:58.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.651+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee9804e170 con 0x7fee840069e0 2023-10-30T13:42:58.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.651+0000 7fee8dffb640 1 -- 172.21.15.148:0/2826467789 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+5726 (secure 0 0 0) 0x7fee94057c90 con 0x7fee840069e0 2023-10-30T13:42:58.652 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:42:58.652 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:51.580494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:42:58.653 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 123 2023-10-30T13:42:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee7c0664d0 msgr2=0x7fee7c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee7c0664d0 0x7fee7c068990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fee880af000 tx=0x7fee880046d0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 msgr2=0x7fee9814f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:42:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee9814f0f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fee94011970 tx=0x7fee94011e30 comp rx=0 tx=0).stop 2023-10-30T13:42:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 shutdown_connections 2023-10-30T13:42:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fee7c0664d0 0x7fee7c068990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee981549e0 0x7fee981e49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee980c4e20 0x7fee981544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 --2- 172.21.15.148:0/2826467789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee840069e0 0x7fee9814f0f0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:42:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 >> 172.21.15.148:0/2826467789 conn(0x7fee980c7910 msgr2=0x7fee98139cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:42:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 shutdown_connections 2023-10-30T13:42:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:42:58.655+0000 7fee9f3f8640 1 -- 172.21.15.148:0/2826467789 wait complete. 2023-10-30T13:42:58.674 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:43:00.674 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:00.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68daa68640 1 Processor -- start 2023-10-30T13:43:00.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68daa68640 1 -- start start 2023-10-30T13:43:00.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68daa68640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d41538a0 0x7f68d415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:00.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68daa68640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f68d40cf160 0x7f68d40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:00.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68daa68640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d40cb800 con 0x7f68d40ca0e0 2023-10-30T13:43:00.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68daa68640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d415dc30 con 0x7f68d40cf160 2023-10-30T13:43:00.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68daa68640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d40cb330 con 0x7f68d41538a0 2023-10-30T13:43:00.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68d9005640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f68d40cf160 0x7f68d40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:00.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68d9005640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f68d40cf160 0x7f68d40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40540/0 (socket says 172.21.15.148:40540) 2023-10-30T13:43:00.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68d9005640 1 -- 172.21.15.148:0/3934278514 learned_addr learned my addr 172.21.15.148:0/3934278514 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:43:00.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68d8804640 1 --2- 172.21.15.148:0/3934278514 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d41538a0 0x7f68d415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:00.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68d9005640 1 -- 172.21.15.148:0/3934278514 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d41538a0 msgr2=0x7f68d415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:00.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68d9005640 1 --2- 172.21.15.148:0/3934278514 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d41538a0 0x7f68d415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:00.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68d9005640 1 -- 172.21.15.148:0/3934278514 >> v1:172.21.15.150:6789/0 conn(0x7f68d40ca0e0 legacy=0x7f68d415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:00.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.815+0000 7f68d9005640 1 -- 172.21.15.148:0/3934278514 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d40cad70 con 0x7f68d40cf160 2023-10-30T13:43:00.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68d9005640 1 --2- 172.21.15.148:0/3934278514 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 0x7f68d40cf590 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009e50 tx=0x7f68cc01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9671005fcfdda19c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:00.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68d37fe640 1 -- 172.21.15.148:0/3934278514 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0af400 con 0x7f68d40cf160 2023-10-30T13:43:00.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68d37fe640 1 -- 172.21.15.148:0/3934278514 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc0ab050 con 0x7f68d40cf160 2023-10-30T13:43:00.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68d37fe640 1 -- 172.21.15.148:0/3934278514 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0af780 con 0x7f68d40cf160 2023-10-30T13:43:00.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68daa68640 1 -- 172.21.15.148:0/3934278514 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 msgr2=0x7f68d40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:00.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68daa68640 1 --2- 172.21.15.148:0/3934278514 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 0x7f68d40cf590 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009e50 tx=0x7f68cc01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:43:00.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68daa68640 1 -- 172.21.15.148:0/3934278514 shutdown_connections 2023-10-30T13:43:00.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68daa68640 1 --2- 172.21.15.148:0/3934278514 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 0x7f68d40cf590 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:00.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68daa68640 1 --2- 172.21.15.148:0/3934278514 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d41538a0 0x7f68d415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:00.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68daa68640 1 -- 172.21.15.148:0/3934278514 >> 172.21.15.148:0/3934278514 conn(0x7f68d406bfe0 msgr2=0x7f68d406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:00.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68daa68640 1 -- 172.21.15.148:0/3934278514 shutdown_connections 2023-10-30T13:43:00.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.819+0000 7f68daa68640 1 -- 172.21.15.148:0/3934278514 wait complete. 2023-10-30T13:43:00.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 Processor -- start 2023-10-30T13:43:00.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 -- start start 2023-10-30T13:43:00.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d40ca0e0 0x7f68d40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:00.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 0x7f68d415c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:00.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68d41538a0 0x7f68d4144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:00.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d41442e0 con 0x7f68d40cf160 2023-10-30T13:43:00.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4143fe0 con 0x7f68d41538a0 2023-10-30T13:43:00.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d414c570 con 0x7f68d40ca0e0 2023-10-30T13:43:00.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 0x7f68d415c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:00.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 0x7f68d415c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:40542/0 (socket says 172.21.15.148:40542) 2023-10-30T13:43:00.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d3fff640 1 -- 172.21.15.148:0/185287880 learned_addr learned my addr 172.21.15.148:0/185287880 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:43:00.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d8804640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d40ca0e0 0x7f68d40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:00.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d9005640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68d41538a0 0x7f68d4144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:00.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d3fff640 1 -- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d40ca0e0 msgr2=0x7f68d40ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:00.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d3fff640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d40ca0e0 0x7f68d40ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:00.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d3fff640 1 -- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68d41538a0 msgr2=0x7f68d4144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:00.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d3fff640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68d41538a0 0x7f68d4144770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:00.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d3fff640 1 -- 172.21.15.148:0/185287880 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc009b30 con 0x7f68d40cf160 2023-10-30T13:43:00.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d8804640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d40ca0e0 0x7f68d40ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:00.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d3fff640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 0x7f68d415c920 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f68c000c530 tx=0x7f68c000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:00.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d17fa640 1 -- 172.21.15.148:0/185287880 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0005220 con 0x7f68d40cf160 2023-10-30T13:43:00.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d17fa640 1 -- 172.21.15.148:0/185287880 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c0005380 con 0x7f68d40cf160 2023-10-30T13:43:00.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68d17fa640 1 -- 172.21.15.148:0/185287880 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0007270 con 0x7f68d40cf160 2023-10-30T13:43:00.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d414c850 con 0x7f68d40cf160 2023-10-30T13:43:00.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.823+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d414ce10 con 0x7f68d40cf160 2023-10-30T13:43:00.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.827+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d40b2a50 con 0x7f68d40cf160 2023-10-30T13:43:00.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.827+0000 7f68d17fa640 1 -- 172.21.15.148:0/185287880 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f68c00073d0 con 0x7f68d40cf160 2023-10-30T13:43:00.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.827+0000 7f68d17fa640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f68b80664a0 0x7f68b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:00.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.827+0000 7f68d17fa640 1 -- 172.21.15.148:0/185287880 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f68c0083d80 con 0x7f68d40cf160 2023-10-30T13:43:00.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.827+0000 7f68d8804640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f68b80664a0 0x7f68b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:00.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.831+0000 7f68d8804640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f68b80664a0 0x7f68b8068960 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f68c800b9f0 tx=0x7f68c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:00.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:00.831+0000 7f68d17fa640 1 -- 172.21.15.148:0/185287880 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f68c0050c00 con 0x7f68d40cf160 2023-10-30T13:43:01.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68d414cfd0 con 0x7f68d40cf160 2023-10-30T13:43:01.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68d17fa640 1 -- 172.21.15.148:0/185287880 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f68d414cfd0 con 0x7f68d40cf160 2023-10-30T13:43:01.011 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:43:01.012 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:01.012 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 124 2023-10-30T13:43:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f68b80664a0 msgr2=0x7f68b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f68b80664a0 0x7f68b8068960 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f68c800b9f0 tx=0x7f68c8008040 comp rx=0 tx=0).stop 2023-10-30T13:43:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 msgr2=0x7f68d415c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:01.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 0x7f68d415c920 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f68c000c530 tx=0x7f68c000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 shutdown_connections 2023-10-30T13:43:01.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f68b80664a0 0x7f68b8068960 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f68d40cf160 0x7f68d415c920 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68d41538a0 0x7f68d4144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 --2- 172.21.15.148:0/185287880 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68d40ca0e0 0x7f68d40ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:01.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 >> 172.21.15.148:0/185287880 conn(0x7f68d406bfe0 msgr2=0x7f68d4151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:01.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 shutdown_connections 2023-10-30T13:43:01.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:01.011+0000 7f68daa68640 1 -- 172.21.15.148:0/185287880 wait complete. 2023-10-30T13:43:01.033 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7149 2023-10-30T13:43:01.034 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.c 2023-10-30T13:43:01.618 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:01.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:04.000 INFO:tasks.ceph.mds.a.smithi148.stderr:2023-10-30T13:43:03.999+0000 7f9cbb36b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:43:06.262 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 34% (106/304) Updating files: 35% (107/304) Updating files: 36% (110/304) Updating files: 37% (113/304) Updating files: 38% (116/304) Updating files: 39% (119/304) Updating files: 40% (122/304) Updating files: 41% (125/304) Updating files: 42% (128/304) Updating files: 43% (131/304) Updating files: 44% (134/304) Updating files: 45% (137/304) Updating files: 46% (140/304) Updating files: 47% (143/304) Updating files: 48% (146/304) Updating files: 49% (149/304) Updating files: 50% (152/304) Updating files: 51% (156/304) Updating files: 52% (159/304) Updating files: 53% (162/304) Updating files: 54% (165/304) Updating files: 55% (168/304) Updating files: 56% (171/304) Updating files: 57% (174/304) Updating files: 58% (177/304) Updating files: 59% (180/304) Updating files: 60% (183/304) Updating files: 61% (186/304) Updating files: 62% (189/304) Updating files: 63% (192/304) Updating files: 64% (195/304) Updating files: 65% (198/304) Updating files: 66% (201/304) Updating files: 67% (204/304) Updating files: 68% (207/304) Updating files: 69% (210/304) Updating files: 70% (213/304) Updating files: 71% (216/304) Updating files: 72% (219/304) Updating files: 73% (222/304) Updating files: 74% (225/304) Updating files: 75% (228/304) Updating files: 76% (232/304) Updating files: 76% (233/304) Updating files: 77% (235/304) Updating files: 78% (238/304) Updating files: 79% (241/304) Updating files: 80% (244/304) Updating files: 81% (247/304) Updating files: 82% (250/304) Updating files: 82% (252/304) Updating files: 83% (253/304) Updating files: 84% (256/304) Updating files: 85% (259/304) Updating files: 86% (262/304) Updating files: 87% (265/304) Updating files: 88% (268/304) Updating files: 89% (271/304) Updating files: 90% (274/304) Updating files: 91% (277/304) Updating files: 92% (280/304) Updating files: 93% (283/304) Updating files: 94% (286/304) Updating files: 95% (289/304) Updating files: 96% (292/304) Updating files: 97% (295/304) Updating files: 98% (298/304) Updating files: 99% (301/304) Updating files: 100% (304/304) Updating files: 100% (304/304), done. 2023-10-30T13:43:07.034 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:43:07.035 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:43:07.035 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:43:07.039 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:43:07.039 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:07.132 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:43:07.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.187+0000 7f1a31aea640 1 Processor -- start 2023-10-30T13:43:07.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.187+0000 7f1a31aea640 1 -- start start 2023-10-30T13:43:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.187+0000 7f1a31aea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1a2c153bf0 0x7f1a2c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.187+0000 7f1a31aea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c6fe0 0x7f1a2c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:07.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.187+0000 7f1a31aea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0700b0 con 0x7f1a2c0c79e0 2023-10-30T13:43:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a31aea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c0c6810 con 0x7f1a2c153bf0 2023-10-30T13:43:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a31aea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c159900 con 0x7f1a2c0c6fe0 2023-10-30T13:43:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a2b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1a2c153bf0 0x7f1a2c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a2b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1a2c153bf0 0x7f1a2c071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56244/0 (socket says 172.21.15.148:56244) 2023-10-30T13:43:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a2b7fe640 1 -- 172.21.15.148:0/3293871627 learned_addr learned my addr 172.21.15.148:0/3293871627 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:43:07.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a23fff640 1 --2- 172.21.15.148:0/3293871627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c6fe0 0x7f1a2c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:07.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a2b7fe640 1 -- 172.21.15.148:0/3293871627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c6fe0 msgr2=0x7f1a2c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:07.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a2b7fe640 1 --2- 172.21.15.148:0/3293871627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c6fe0 0x7f1a2c0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:07.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a2b7fe640 1 -- 172.21.15.148:0/3293871627 >> v1:172.21.15.150:6789/0 conn(0x7f1a2c0c79e0 legacy=0x7f1a2c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:07.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a2b7fe640 1 -- 172.21.15.148:0/3293871627 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c0c6530 con 0x7f1a2c153bf0 2023-10-30T13:43:07.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.191+0000 7f1a2b7fe640 1 --2- 172.21.15.148:0/3293871627 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 0x7f1a2c071260 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009870 tx=0x7f1a1c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87549346e5d23de2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:07.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a2affd640 1 -- 172.21.15.148:0/3293871627 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c0a8a30 con 0x7f1a2c153bf0 2023-10-30T13:43:07.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a2affd640 1 -- 172.21.15.148:0/3293871627 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c0a8b90 con 0x7f1a2c153bf0 2023-10-30T13:43:07.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a2affd640 1 -- 172.21.15.148:0/3293871627 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c0b0440 con 0x7f1a2c153bf0 2023-10-30T13:43:07.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 -- 172.21.15.148:0/3293871627 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 msgr2=0x7f1a2c071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:07.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 --2- 172.21.15.148:0/3293871627 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 0x7f1a2c071260 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009870 tx=0x7f1a1c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:43:07.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 -- 172.21.15.148:0/3293871627 shutdown_connections 2023-10-30T13:43:07.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 --2- 172.21.15.148:0/3293871627 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c6fe0 0x7f1a2c0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:07.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 --2- 172.21.15.148:0/3293871627 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 0x7f1a2c071260 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:07.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 -- 172.21.15.148:0/3293871627 >> 172.21.15.148:0/3293871627 conn(0x7f1a2c06c6e0 msgr2=0x7f1a2c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:07.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 -- 172.21.15.148:0/3293871627 shutdown_connections 2023-10-30T13:43:07.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 -- 172.21.15.148:0/3293871627 wait complete. 2023-10-30T13:43:07.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 Processor -- start 2023-10-30T13:43:07.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 -- start start 2023-10-30T13:43:07.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a2c0c6fe0 0x7f1a2c1502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:07.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c79e0 0x7f1a2c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:07.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 0x7f1a2c1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:07.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c1ef1f0 con 0x7f1a2c153bf0 2023-10-30T13:43:07.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c1543f0 con 0x7f1a2c0c6fe0 2023-10-30T13:43:07.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a31aea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c1546d0 con 0x7f1a2c0c79e0 2023-10-30T13:43:07.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 0x7f1a2c1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:07.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 0x7f1a2c1556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56254/0 (socket says 172.21.15.148:56254) 2023-10-30T13:43:07.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a2bfff640 1 -- 172.21.15.148:0/3600803441 learned_addr learned my addr 172.21.15.148:0/3600803441 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:43:07.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a2b7fe640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a2c0c6fe0 0x7f1a2c1502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:07.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.195+0000 7f1a23fff640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c79e0 0x7f1a2c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:07.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a2bfff640 1 -- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c79e0 msgr2=0x7f1a2c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:07.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a2bfff640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c79e0 0x7f1a2c1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:07.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a2bfff640 1 -- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a2c0c6fe0 msgr2=0x7f1a2c1502d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:07.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a2bfff640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a2c0c6fe0 0x7f1a2c1502d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:07.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a2bfff640 1 -- 172.21.15.148:0/3600803441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c0094d0 con 0x7f1a2c153bf0 2023-10-30T13:43:07.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a23fff640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c79e0 0x7f1a2c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:07.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a2bfff640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 0x7f1a2c1556f0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c770 tx=0x7f1a2400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:07.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a28ff9640 1 -- 172.21.15.148:0/3600803441 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24003b80 con 0x7f1a2c153bf0 2023-10-30T13:43:07.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a28ff9640 1 -- 172.21.15.148:0/3600803441 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a24003ce0 con 0x7f1a2c153bf0 2023-10-30T13:43:07.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a28ff9640 1 -- 172.21.15.148:0/3600803441 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2400aa90 con 0x7f1a2c153bf0 2023-10-30T13:43:07.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c1549b0 con 0x7f1a2c153bf0 2023-10-30T13:43:07.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c0c52a0 con 0x7f1a2c153bf0 2023-10-30T13:43:07.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c04e170 con 0x7f1a2c153bf0 2023-10-30T13:43:07.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a28ff9640 1 -- 172.21.15.148:0/3600803441 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a2400abf0 con 0x7f1a2c153bf0 2023-10-30T13:43:07.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a28ff9640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1a0c0664a0 0x7f1a0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:07.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.199+0000 7f1a28ff9640 1 -- 172.21.15.148:0/3600803441 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f1a240842b0 con 0x7f1a2c153bf0 2023-10-30T13:43:07.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.203+0000 7f1a2b7fe640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1a0c0664a0 0x7f1a0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:07.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.203+0000 7f1a2b7fe640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1a0c0664a0 0x7f1a0c068960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0b2000 tx=0x7f1a1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:07.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.207+0000 7f1a28ff9640 1 -- 172.21.15.148:0/3600803441 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a240510a0 con 0x7f1a2c153bf0 2023-10-30T13:43:07.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.387+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a2c06fe90 con 0x7f1a2c153bf0 2023-10-30T13:43:07.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.387+0000 7f1a28ff9640 1 -- 172.21.15.148:0/3600803441 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f1a240546c0 con 0x7f1a2c153bf0 2023-10-30T13:43:07.389 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:43:07.389 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:07.389 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 124 2023-10-30T13:43:07.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1a0c0664a0 msgr2=0x7f1a0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:07.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1a0c0664a0 0x7f1a0c068960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0b2000 tx=0x7f1a1c009040 comp rx=0 tx=0).stop 2023-10-30T13:43:07.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 msgr2=0x7f1a2c1556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:07.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 0x7f1a2c1556f0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c770 tx=0x7f1a2400cc30 comp rx=0 tx=0).stop 2023-10-30T13:43:07.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 shutdown_connections 2023-10-30T13:43:07.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1a0c0664a0 0x7f1a0c068960 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:07.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1a2c153bf0 0x7f1a2c1556f0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:07.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1a2c0c79e0 0x7f1a2c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:07.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 --2- 172.21.15.148:0/3600803441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a2c0c6fe0 0x7f1a2c1502d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:07.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 >> 172.21.15.148:0/3600803441 conn(0x7f1a2c06c6e0 msgr2=0x7f1a2c0c9e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:07.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 shutdown_connections 2023-10-30T13:43:07.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:07.391+0000 7f1a31aea640 1 -- 172.21.15.148:0/3600803441 wait complete. 2023-10-30T13:43:07.410 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:43:07.484 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 9a7a9d06c06 v0.57 2023-10-30T13:43:07.491 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.57 2023-10-30T13:43:07.516 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 58 -eq 48 ] 2023-10-30T13:43:07.516 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.58 2023-10-30T13:43:07.517 INFO:tasks.workunit.client.0.smithi148.stdout:v0.58 2023-10-30T13:43:07.517 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.58 2023-10-30T13:43:07.517 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.58 2023-10-30T13:43:09.411 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:43:09.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d2d846640 1 Processor -- start 2023-10-30T13:43:09.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d2d846640 1 -- start start 2023-10-30T13:43:09.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d2d846640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d280c6b80 0x7f1d280bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:09.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d2d846640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d2813f4a0 0x7f1d280c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:09.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d2d846640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d280c0a50 con 0x7f1d280c6f60 2023-10-30T13:43:09.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d2d846640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d2806f6e0 con 0x7f1d2813f4a0 2023-10-30T13:43:09.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d2d846640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2806f9c0 con 0x7f1d280c6b80 2023-10-30T13:43:09.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d1ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f1d280c6f60 0x7f1d2813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44666/0 (socket says 172.21.15.148:44666) 2023-10-30T13:43:09.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d1ffff640 1 -- 172.21.15.148:0/2629645041 learned_addr learned my addr 172.21.15.148:0/2629645041 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:43:09.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d26ffd640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d280c6b80 0x7f1d280bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d277fe640 1 --2- 172.21.15.148:0/2629645041 >> v2:172.21.15.150:3300/0 conn(0x7f1d2813f4a0 0x7f1d280c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390973494 0 0) 0x7f1d280c0a50 con 0x7f1d280c6f60 2023-10-30T13:43:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d14003530 con 0x7f1d280c6f60 2023-10-30T13:43:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3925738027 0 0) 0x7f1d14003530 con 0x7f1d280c6f60 2023-10-30T13:43:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d280c6b80 msgr2=0x7f1d280bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:09.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d267fc640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d280c6b80 0x7f1d280bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 >> v2:172.21.15.150:3300/0 conn(0x7f1d2813f4a0 msgr2=0x7f1d280c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.551+0000 7f1d267fc640 1 --2- 172.21.15.148:0/2629645041 >> v2:172.21.15.150:3300/0 conn(0x7f1d2813f4a0 0x7f1d280c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2806fca0 con 0x7f1d280c6f60 2023-10-30T13:43:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f1d10001680 con 0x7f1d280c6f60 2023-10-30T13:43:09.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 >> v1:172.21.15.148:6789/0 conn(0x7f1d280c6f60 legacy=0x7f1d2813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d280c6b80 0x7f1d14003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d14003f00 0x7f1d140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d14005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2806f6e0 con 0x7f1d14004d10 2023-10-30T13:43:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2806f9c0 con 0x7f1d14003f00 2023-10-30T13:43:09.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1400b990 con 0x7f1d280c6b80 2023-10-30T13:43:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d277fe640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d14005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d1ffff640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d14003f00 0x7f1d140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d26ffd640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d280c6b80 0x7f1d14003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d277fe640 1 -- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d280c6b80 msgr2=0x7f1d14003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:09.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d277fe640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d280c6b80 0x7f1d14003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d277fe640 1 -- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d14003f00 msgr2=0x7f1d140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:43:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d277fe640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d14003f00 0x7f1d140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d277fe640 1 -- 172.21.15.148:0/2629645041 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d14003530 con 0x7f1d14004d10 2023-10-30T13:43:09.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d26ffd640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d280c6b80 0x7f1d14003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:43:09.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d277fe640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d14005190 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f1d2806f4f0 tx=0x7f1d2001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9f6e8067a261aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:09.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d200a5c50 con 0x7f1d14004d10 2023-10-30T13:43:09.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.555+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2000f040 con 0x7f1d14004d10 2023-10-30T13:43:09.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d267fc640 1 -- 172.21.15.148:0/2629645041 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d200ad440 con 0x7f1d14004d10 2023-10-30T13:43:09.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 -- 172.21.15.148:0/2629645041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 msgr2=0x7f1d14005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:09.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 --2- 172.21.15.148:0/2629645041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d14005190 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f1d2806f4f0 tx=0x7f1d2001cbe0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d277fe640 1 -- 172.21.15.148:0/2629645041 reap_dead start 2023-10-30T13:43:09.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 -- 172.21.15.148:0/2629645041 shutdown_connections 2023-10-30T13:43:09.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 -- 172.21.15.148:0/2629645041 >> 172.21.15.148:0/2629645041 conn(0x7f1d280c7910 msgr2=0x7f1d280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:09.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 -- 172.21.15.148:0/2629645041 shutdown_connections 2023-10-30T13:43:09.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 -- 172.21.15.148:0/2629645041 wait complete. 2023-10-30T13:43:09.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 Processor -- start 2023-10-30T13:43:09.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 -- start start 2023-10-30T13:43:09.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d280c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:09.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d28136ff0 0x7f1d281ed760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:09.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d281edca0 0x7f1d280c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:09.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d281ead30 con 0x7f1d14004d10 2023-10-30T13:43:09.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d28150a90 con 0x7f1d28136ff0 2023-10-30T13:43:09.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d2d846640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d280c52e0 con 0x7f1d281edca0 2023-10-30T13:43:09.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d26ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d280c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:09.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d26ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d280c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56272/0 (socket says 172.21.15.148:56272) 2023-10-30T13:43:09.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.559+0000 7f1d26ffd640 1 -- 172.21.15.148:0/298091790 learned_addr learned my addr 172.21.15.148:0/298091790 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:43:09.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d277fe640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d281edca0 0x7f1d280c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:09.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d1ffff640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d28136ff0 0x7f1d281ed760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:09.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d26ffd640 1 -- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d281edca0 msgr2=0x7f1d280c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:09.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d26ffd640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d281edca0 0x7f1d280c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d26ffd640 1 -- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d28136ff0 msgr2=0x7f1d281ed760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:09.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d26ffd640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d28136ff0 0x7f1d281ed760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d26ffd640 1 -- 172.21.15.148:0/298091790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d281378b0 con 0x7f1d14004d10 2023-10-30T13:43:09.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d26ffd640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d280c4da0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f1d180115f0 tx=0x7f1d18011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:09.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d24ff9640 1 -- 172.21.15.148:0/298091790 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18004220 con 0x7f1d14004d10 2023-10-30T13:43:09.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d24ff9640 1 -- 172.21.15.148:0/298091790 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d18004380 con 0x7f1d14004d10 2023-10-30T13:43:09.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d24ff9640 1 -- 172.21.15.148:0/298091790 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1800f8f0 con 0x7f1d14004d10 2023-10-30T13:43:09.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d281eb150 con 0x7f1d14004d10 2023-10-30T13:43:09.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d281ea390 con 0x7f1d14004d10 2023-10-30T13:43:09.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2804e170 con 0x7f1d14004d10 2023-10-30T13:43:09.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d24ff9640 1 -- 172.21.15.148:0/298091790 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1d1800fa50 con 0x7f1d14004d10 2023-10-30T13:43:09.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d24ff9640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d080664a0 0x7f1d08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:43:09.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.563+0000 7f1d24ff9640 1 -- 172.21.15.148:0/298091790 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f1d18084770 con 0x7f1d14004d10 2023-10-30T13:43:09.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.567+0000 7f1d1ffff640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d080664a0 0x7f1d08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:43:09.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.567+0000 7f1d1ffff640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d080664a0 0x7f1d08068960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1d281e7e20 tx=0x7f1d10007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:43:09.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.567+0000 7f1d24ff9640 1 -- 172.21.15.148:0/298091790 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1d18051560 con 0x7f1d14004d10 2023-10-30T13:43:09.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.747+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d2813f1a0 con 0x7f1d14004d10 2023-10-30T13:43:09.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.747+0000 7f1d24ff9640 1 -- 172.21.15.148:0/298091790 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f1d18054b80 con 0x7f1d14004d10 2023-10-30T13:43:09.748 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:43:09.749 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:43:09.749 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 125 2023-10-30T13:43:09.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d080664a0 msgr2=0x7f1d08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:09.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d080664a0 0x7f1d08068960 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1d281e7e20 tx=0x7f1d10007900 comp rx=0 tx=0).stop 2023-10-30T13:43:09.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 msgr2=0x7f1d280c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:43:09.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d280c4da0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f1d180115f0 tx=0x7f1d18011ab0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 shutdown_connections 2023-10-30T13:43:09.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1d080664a0 0x7f1d08068960 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1d281edca0 0x7f1d280c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d28136ff0 0x7f1d281ed760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 --2- 172.21.15.148:0/298091790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1d14004d10 0x7f1d280c4da0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:43:09.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 >> 172.21.15.148:0/298091790 conn(0x7f1d280c7910 msgr2=0x7f1d280c14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:43:09.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 shutdown_connections 2023-10-30T13:43:09.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:43:09.751+0000 7f1d2d846640 1 -- 172.21.15.148:0/298091790 wait complete. 2023-10-30T13:43:09.769 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:43:09.769 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2023-10-30T13:43:16.601 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 16% (43/268) Updating files: 17% (46/268) Updating files: 18% (49/268) Updating files: 19% (51/268) Updating files: 20% (54/268) Updating files: 21% (57/268) Updating files: 22% (59/268) Updating files: 23% (62/268) Updating files: 24% (65/268) Updating files: 25% (67/268) Updating files: 26% (70/268) Updating files: 27% (73/268) Updating files: 28% (76/268) Updating files: 29% (78/268) Updating files: 30% (81/268) Updating files: 31% (84/268) Updating files: 31% (85/268) Updating files: 32% (86/268) Updating files: 33% (89/268) Updating files: 34% (92/268) Updating files: 35% (94/268) Updating files: 36% (97/268) Updating files: 37% (100/268) Updating files: 38% (102/268) Updating files: 39% (105/268) Updating files: 40% (108/268) Updating files: 41% (110/268) Updating files: 42% (113/268) Updating files: 43% (116/268) Updating files: 44% (118/268) Updating files: 45% (121/268) Updating files: 46% (124/268) Updating files: 47% (126/268) Updating files: 48% (129/268) Updating files: 49% (132/268) Updating files: 50% (134/268) Updating files: 51% (137/268) Updating files: 52% (140/268) Updating files: 53% (143/268) Updating files: 54% (145/268) Updating files: 55% (148/268) Updating files: 56% (151/268) Updating files: 57% (153/268) Updating files: 58% (156/268) Updating files: 59% (159/268) Updating files: 60% (161/268) Updating files: 61% (164/268) Updating files: 62% (167/268) Updating files: 63% (169/268) Updating files: 64% (172/268) Updating files: 65% (175/268) Updating files: 66% (177/268) Updating files: 67% (180/268) Updating files: 68% (183/268) Updating files: 69% (185/268) Updating files: 70% (188/268) Updating files: 71% (191/268) Updating files: 71% (192/268) Updating files: 72% (193/268) Updating files: 73% (196/268) Updating files: 74% (199/268) Updating files: 75% (201/268) Updating files: 76% (204/268) Updating files: 77% (207/268) Updating files: 78% (210/268) Updating files: 79% (212/268) Updating files: 80% (215/268) Updating files: 81% (218/268) Updating files: 82% (220/268) Updating files: 83% (223/268) Updating files: 84% (226/268) Updating files: 85% (228/268) Updating files: 86% (231/268) Updating files: 87% (234/268) Updating files: 88% (236/268) Updating files: 89% (239/268) Updating files: 90% (242/268) Updating files: 91% (244/268) Updating files: 92% (247/268) Updating files: 93% (250/268) Updating files: 94% (252/268) Updating files: 95% (255/268) Updating files: 96% (258/268) Updating files: 97% (260/268) Updating files: 98% (263/268) Updating files: 99% (266/268) Updating files: 100% (268/268) Updating files: 100% (268/268), done. 2023-10-30T13:43:17.829 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at ba3f91e7504 v0.58 2023-10-30T13:43:17.836 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.58 2023-10-30T13:43:17.873 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 59 -eq 48 ] 2023-10-30T13:43:17.873 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.59 2023-10-30T13:43:17.873 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.59 2023-10-30T13:43:17.874 INFO:tasks.workunit.client.0.smithi148.stdout:v0.59 2023-10-30T13:43:17.874 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.59 2023-10-30T13:43:25.670 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 49% (104/211) Updating files: 50% (106/211) Updating files: 51% (108/211) Updating files: 52% (110/211) Updating files: 53% (112/211) Updating files: 54% (114/211) Updating files: 55% (117/211) Updating files: 56% (119/211) Updating files: 57% (121/211) Updating files: 58% (123/211) Updating files: 59% (125/211) Updating files: 60% (127/211) Updating files: 61% (129/211) Updating files: 62% (131/211) Updating files: 63% (133/211) Updating files: 64% (136/211) Updating files: 65% (138/211) Updating files: 66% (140/211) Updating files: 67% (142/211) Updating files: 68% (144/211) Updating files: 69% (146/211) Updating files: 70% (148/211) Updating files: 71% (150/211) Updating files: 72% (152/211) Updating files: 73% (155/211) Updating files: 74% (157/211) Updating files: 75% (159/211) Updating files: 76% (161/211) Updating files: 77% (163/211) Updating files: 78% (165/211) Updating files: 79% (167/211) Updating files: 80% (169/211) Updating files: 81% (171/211) Updating files: 82% (174/211) Updating files: 83% (176/211) Updating files: 84% (178/211) Updating files: 85% (180/211) Updating files: 86% (182/211) Updating files: 87% (184/211) Updating files: 88% (186/211) Updating files: 89% (188/211) Updating files: 90% (190/211) Updating files: 91% (193/211) Updating files: 92% (195/211) Updating files: 93% (197/211) Updating files: 94% (199/211) Updating files: 95% (201/211) Updating files: 96% (203/211) Updating files: 97% (205/211) Updating files: 97% (206/211) Updating files: 98% (207/211) Updating files: 99% (209/211) Updating files: 100% (211/211) Updating files: 100% (211/211), done. 2023-10-30T13:43:25.933 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at cbae6a435c6 v0.59 2023-10-30T13:43:25.939 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.59 2023-10-30T13:43:25.989 INFO:tasks.workunit.client.0.smithi148.stdout:v0.60 2023-10-30T13:43:25.989 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 60 -eq 48 ] 2023-10-30T13:43:25.990 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.60 2023-10-30T13:43:25.990 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.60 2023-10-30T13:43:25.990 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.60 2023-10-30T13:43:28.919 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 62% (160/256) Updating files: 63% (162/256) Updating files: 64% (164/256) Updating files: 65% (167/256) Updating files: 66% (169/256) Updating files: 67% (172/256) Updating files: 68% (175/256) Updating files: 69% (177/256) Updating files: 70% (180/256) Updating files: 71% (182/256) Updating files: 72% (185/256) Updating files: 73% (187/256) Updating files: 74% (190/256) Updating files: 75% (192/256) Updating files: 76% (195/256) Updating files: 77% (198/256) Updating files: 77% (199/256) Updating files: 78% (200/256) Updating files: 79% (203/256) Updating files: 80% (205/256) Updating files: 81% (208/256) Updating files: 82% (210/256) Updating files: 83% (213/256) Updating files: 84% (216/256) Updating files: 85% (218/256) Updating files: 86% (221/256) Updating files: 87% (223/256) Updating files: 88% (226/256) Updating files: 89% (228/256) Updating files: 90% (231/256) Updating files: 91% (233/256) Updating files: 92% (236/256) Updating files: 93% (239/256) Updating files: 94% (241/256) Updating files: 95% (244/256) Updating files: 96% (246/256) Updating files: 97% (249/256) Updating files: 98% (251/256) Updating files: 99% (254/256) Updating files: 100% (256/256) Updating files: 100% (256/256), done. 2023-10-30T13:43:28.958 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at f26f7a39021 v0.60 2023-10-30T13:43:28.966 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.60 2023-10-30T13:43:29.228 INFO:tasks.workunit.client.0.smithi148.stdout:v0.61 2023-10-30T13:43:29.228 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 61 -eq 48 ] 2023-10-30T13:43:29.228 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.61 2023-10-30T13:43:29.229 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.61 2023-10-30T13:43:29.229 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.61 2023-10-30T13:43:31.676 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:31.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:43:51.130 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 24% (91/376) Updating files: 25% (94/376) Updating files: 26% (98/376) Updating files: 27% (102/376) Updating files: 28% (106/376) Updating files: 29% (110/376) Updating files: 30% (113/376) Updating files: 31% (117/376) Updating files: 32% (121/376) Updating files: 33% (125/376) Updating files: 34% (128/376) Updating files: 35% (132/376) Updating files: 36% (136/376) Updating files: 37% (140/376) Updating files: 38% (143/376) Updating files: 39% (147/376) Updating files: 40% (151/376) Updating files: 40% (154/376) Updating files: 41% (155/376) Updating files: 42% (158/376) Updating files: 43% (162/376) Updating files: 44% (166/376) Updating files: 45% (170/376) Updating files: 46% (173/376) Updating files: 47% (177/376) Updating files: 48% (181/376) Updating files: 49% (185/376) Updating files: 49% (186/376) Updating files: 50% (188/376) Updating files: 51% (192/376) Updating files: 52% (196/376) Updating files: 53% (200/376) Updating files: 54% (204/376) Updating files: 55% (207/376) Updating files: 56% (211/376) Updating files: 57% (215/376) Updating files: 58% (219/376) Updating files: 59% (222/376) Updating files: 60% (226/376) Updating files: 61% (230/376) Updating files: 62% (234/376) Updating files: 63% (237/376) Updating files: 64% (241/376) Updating files: 65% (245/376) Updating files: 66% (249/376) Updating files: 67% (252/376) Updating files: 68% (256/376) Updating files: 69% (260/376) Updating files: 70% (264/376) Updating files: 71% (267/376) Updating files: 72% (271/376) Updating files: 73% (275/376) Updating files: 74% (279/376) Updating files: 75% (282/376) Updating files: 76% (286/376) Updating files: 77% (290/376) Updating files: 78% (294/376) Updating files: 79% (298/376) Updating files: 79% (300/376) Updating files: 80% (301/376) Updating files: 80% (303/376) Updating files: 81% (305/376) Updating files: 82% (309/376) Updating files: 83% (313/376) Updating files: 84% (316/376) Updating files: 85% (320/376) Updating files: 86% (324/376) Updating files: 87% (328/376) Updating files: 88% (331/376) Updating files: 89% (335/376) Updating files: 90% (339/376) Updating files: 91% (343/376) Updating files: 92% (346/376) Updating files: 93% (350/376) Updating files: 94% (354/376) Updating files: 95% (358/376) Updating files: 96% (361/376) Updating files: 97% (365/376) Updating files: 98% (369/376) Updating files: 99% (373/376) Updating files: 100% (376/376) Updating files: 100% (376/376), done. 2023-10-30T13:43:51.686 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 237f3f1e8d8 v0.61 2023-10-30T13:43:51.695 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.61 2023-10-30T13:43:52.069 INFO:tasks.workunit.client.0.smithi148.stdout:v0.62 2023-10-30T13:43:52.070 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 62 -eq 48 ] 2023-10-30T13:43:52.070 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.62 2023-10-30T13:43:52.070 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.62 2023-10-30T13:43:52.070 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.62 2023-10-30T13:43:53.015 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 9382379cc6b v0.62 2023-10-30T13:43:53.018 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.62 2023-10-30T13:43:53.835 INFO:tasks.workunit.client.0.smithi148.stdout:v0.63 2023-10-30T13:43:53.836 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 63 -eq 48 ] 2023-10-30T13:43:53.836 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.63 2023-10-30T13:43:53.836 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.63 2023-10-30T13:43:53.836 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.63 2023-10-30T13:44:01.732 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:01.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:01.876 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 79% (116/146) Updating files: 80% (117/146) Updating files: 81% (119/146) Updating files: 82% (120/146) Updating files: 83% (122/146) Updating files: 84% (123/146) Updating files: 85% (125/146) Updating files: 86% (126/146) Updating files: 87% (128/146) Updating files: 88% (129/146) Updating files: 89% (130/146) Updating files: 89% (131/146) Updating files: 90% (132/146) Updating files: 91% (133/146) Updating files: 92% (135/146) Updating files: 93% (136/146) Updating files: 94% (138/146) Updating files: 95% (139/146) Updating files: 96% (141/146) Updating files: 97% (142/146) Updating files: 98% (144/146) Updating files: 99% (145/146) Updating files: 100% (146/146) Updating files: 100% (146/146), done. 2023-10-30T13:44:02.430 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 054e96cf79e v0.63 2023-10-30T13:44:02.438 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.63 2023-10-30T13:44:02.954 INFO:tasks.workunit.client.0.smithi148.stdout:v0.64 2023-10-30T13:44:02.954 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 64 -eq 48 ] 2023-10-30T13:44:02.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.64 2023-10-30T13:44:02.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.64 2023-10-30T13:44:02.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.64 2023-10-30T13:44:15.410 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 3% (5/143) Updating files: 4% (6/143) Updating files: 5% (8/143) Updating files: 6% (9/143) Updating files: 7% (11/143) Updating files: 8% (12/143) Updating files: 9% (13/143) Updating files: 10% (15/143) Updating files: 11% (16/143) Updating files: 12% (18/143) Updating files: 13% (19/143) Updating files: 14% (21/143) Updating files: 15% (22/143) Updating files: 16% (23/143) Updating files: 17% (25/143) Updating files: 18% (26/143) Updating files: 19% (28/143) Updating files: 20% (29/143) Updating files: 21% (31/143) Updating files: 22% (32/143) Updating files: 23% (33/143) Updating files: 24% (35/143) Updating files: 25% (36/143) Updating files: 26% (38/143) Updating files: 27% (39/143) Updating files: 28% (41/143) Updating files: 29% (42/143) Updating files: 30% (43/143) Updating files: 31% (45/143) Updating files: 32% (46/143) Updating files: 33% (48/143) Updating files: 34% (49/143) Updating files: 35% (51/143) Updating files: 36% (52/143) Updating files: 37% (53/143) Updating files: 38% (55/143) Updating files: 39% (56/143) Updating files: 40% (58/143) Updating files: 41% (59/143) Updating files: 42% (61/143) Updating files: 43% (62/143) Updating files: 44% (63/143) Updating files: 45% (65/143) Updating files: 46% (66/143) Updating files: 47% (68/143) Updating files: 48% (69/143) Updating files: 49% (71/143) Updating files: 50% (72/143) Updating files: 51% (73/143) Updating files: 52% (75/143) Updating files: 53% (76/143) Updating files: 54% (78/143) Updating files: 55% (79/143) Updating files: 56% (81/143) Updating files: 57% (82/143) Updating files: 58% (83/143) Updating files: 59% (85/143) Updating files: 60% (86/143) Updating files: 61% (88/143) Updating files: 62% (89/143) Updating files: 63% (91/143) Updating files: 64% (92/143) Updating files: 65% (93/143) Updating files: 66% (95/143) Updating files: 67% (96/143) Updating files: 68% (98/143) Updating files: 69% (99/143) Updating files: 70% (101/143) Updating files: 71% (102/143) Updating files: 72% (103/143) Updating files: 73% (105/143) Updating files: 74% (106/143) Updating files: 75% (108/143) Updating files: 76% (109/143) Updating files: 77% (111/143) Updating files: 78% (112/143) Updating files: 79% (113/143) Updating files: 80% (115/143) Updating files: 81% (116/143) Updating files: 82% (118/143) Updating files: 83% (119/143) Updating files: 84% (121/143) Updating files: 85% (122/143) Updating files: 86% (123/143) Updating files: 87% (125/143) Updating files: 88% (126/143) Updating files: 89% (128/143) Updating files: 90% (129/143) Updating files: 91% (131/143) Updating files: 92% (132/143) Updating files: 93% (133/143) Updating files: 94% (135/143) Updating files: 95% (136/143) Updating files: 96% (138/143) Updating files: 97% (139/143) Updating files: 98% (141/143) Updating files: 99% (142/143) Updating files: 100% (143/143) Updating files: 100% (143/143), done. 2023-10-30T13:44:15.459 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 42e06c12db6 v0.64 2023-10-30T13:44:15.471 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.64 2023-10-30T13:44:15.584 INFO:tasks.workunit.client.0.smithi148.stdout:v0.65 2023-10-30T13:44:15.584 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 65 -eq 48 ] 2023-10-30T13:44:15.585 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.65 2023-10-30T13:44:15.585 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.65 2023-10-30T13:44:15.585 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.65 2023-10-30T13:44:20.770 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:20.835 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 45% (93/203) Updating files: 46% (94/203) Updating files: 47% (96/203) Updating files: 48% (98/203) Updating files: 49% (100/203) Updating files: 50% (102/203) Updating files: 51% (104/203) Updating files: 52% (106/203) Updating files: 53% (108/203) Updating files: 54% (110/203) Updating files: 55% (112/203) Updating files: 56% (114/203) Updating files: 57% (116/203) Updating files: 58% (118/203) Updating files: 59% (120/203) Updating files: 60% (122/203) Updating files: 61% (124/203) Updating files: 62% (126/203) Updating files: 63% (128/203) Updating files: 64% (130/203) Updating files: 65% (132/203) Updating files: 65% (133/203) Updating files: 66% (134/203) Updating files: 67% (137/203) Updating files: 68% (139/203) Updating files: 69% (141/203) Updating files: 70% (143/203) Updating files: 71% (145/203) Updating files: 72% (147/203) Updating files: 73% (149/203) Updating files: 74% (151/203) Updating files: 75% (153/203) Updating files: 76% (155/203) Updating files: 77% (157/203) Updating files: 78% (159/203) Updating files: 79% (161/203) Updating files: 80% (163/203) Updating files: 81% (165/203) Updating files: 82% (167/203) Updating files: 83% (169/203) Updating files: 84% (171/203) Updating files: 85% (173/203) Updating files: 86% (175/203) Updating files: 87% (177/203) Updating files: 88% (179/203) Updating files: 89% (181/203) Updating files: 90% (183/203) Updating files: 91% (185/203) Updating files: 92% (187/203) Updating files: 93% (189/203) Updating files: 94% (191/203) Updating files: 95% (193/203) Updating files: 96% (195/203) Updating files: 97% (197/203) Updating files: 98% (199/203) Updating files: 99% (201/203) Updating files: 100% (203/203) Updating files: 100% (203/203), done. 2023-10-30T13:44:20.885 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at c2d517ef966 v0.65 2023-10-30T13:44:20.890 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.65 2023-10-30T13:44:20.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2b4dc640 1 Processor -- start 2023-10-30T13:44:20.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2b4dc640 1 -- start start 2023-10-30T13:44:20.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2b4dc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efe2415cd50 0x7efe240c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:20.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2b4dc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe240cefa0 0x7efe240cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:20.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2b4dc640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2415a6c0 con 0x7efe241538f0 2023-10-30T13:44:20.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2b4dc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2415d300 con 0x7efe2415cd50 2023-10-30T13:44:20.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2b4dc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2415d5e0 con 0x7efe240cefa0 2023-10-30T13:44:20.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2a4da640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efe241538f0 0x7efe24159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44276/0 (socket says 172.21.15.148:44276) 2023-10-30T13:44:20.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2a4da640 1 -- 172.21.15.148:0/1075782035 learned_addr learned my addr 172.21.15.148:0/1075782035 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:20.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe29cd9640 1 --2- 172.21.15.148:0/1075782035 >> v2:172.21.15.150:3300/0 conn(0x7efe2415cd50 0x7efe240c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:20.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2acdb640 1 --2- 172.21.15.148:0/1075782035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe240cefa0 0x7efe240cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:20.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe294d8640 1 -- 172.21.15.148:0/1075782035 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667075282 0 0) 0x7efe2415a6c0 con 0x7efe241538f0 2023-10-30T13:44:20.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe294d8640 1 -- 172.21.15.148:0/1075782035 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe10003530 con 0x7efe241538f0 2023-10-30T13:44:20.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe294d8640 1 -- 172.21.15.148:0/1075782035 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3268379496 0 0) 0x7efe10003530 con 0x7efe241538f0 2023-10-30T13:44:20.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe29cd9640 1 -- 172.21.15.148:0/1075782035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe240cefa0 msgr2=0x7efe240cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:20.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe29cd9640 1 --2- 172.21.15.148:0/1075782035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe240cefa0 0x7efe240cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:20.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe29cd9640 1 -- 172.21.15.148:0/1075782035 >> v1:172.21.15.148:6789/0 conn(0x7efe241538f0 legacy=0x7efe24159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:20.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe29cd9640 1 -- 172.21.15.148:0/1075782035 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2415d8c0 con 0x7efe2415cd50 2023-10-30T13:44:20.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.929+0000 7efe2acdb640 1 --2- 172.21.15.148:0/1075782035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe240cefa0 0x7efe240cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:44:20.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe29cd9640 1 --2- 172.21.15.148:0/1075782035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe2415cd50 0x7efe240c9f90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efe140097c0 tx=0x7efe1401eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aebb2efe4b2e6a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:20.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe294d8640 1 -- 172.21.15.148:0/1075782035 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20001680 con 0x7efe2415cd50 2023-10-30T13:44:20.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe294d8640 1 -- 172.21.15.148:0/1075782035 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe140a85d0 con 0x7efe2415cd50 2023-10-30T13:44:20.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe294d8640 1 -- 172.21.15.148:0/1075782035 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe140a8940 con 0x7efe2415cd50 2023-10-30T13:44:20.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 -- 172.21.15.148:0/1075782035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe2415cd50 msgr2=0x7efe240c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:20.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 --2- 172.21.15.148:0/1075782035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe2415cd50 0x7efe240c9f90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efe140097c0 tx=0x7efe1401eb90 comp rx=0 tx=0).stop 2023-10-30T13:44:20.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 -- 172.21.15.148:0/1075782035 shutdown_connections 2023-10-30T13:44:20.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 --2- 172.21.15.148:0/1075782035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe2415cd50 0x7efe240c9f90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:20.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 --2- 172.21.15.148:0/1075782035 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe240cefa0 0x7efe240cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:20.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 -- 172.21.15.148:0/1075782035 >> 172.21.15.148:0/1075782035 conn(0x7efe2406a640 msgr2=0x7efe24065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:20.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 -- 172.21.15.148:0/1075782035 shutdown_connections 2023-10-30T13:44:20.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 -- 172.21.15.148:0/1075782035 wait complete. 2023-10-30T13:44:20.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 Processor -- start 2023-10-30T13:44:20.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.937+0000 7efe2b4dc640 1 -- start start 2023-10-30T13:44:20.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2b4dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe240cefa0 0x7efe241f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:20.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2b4dc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe241538f0 0x7efe2413fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:20.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2b4dc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe2415cd50 0x7efe241510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:20.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2b4dc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe240b2e10 con 0x7efe2415cd50 2023-10-30T13:44:20.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2b4dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe240b3c70 con 0x7efe240cefa0 2023-10-30T13:44:20.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2b4dc640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe240b4100 con 0x7efe241538f0 2023-10-30T13:44:20.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2acdb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe2415cd50 0x7efe241510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:20.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2acdb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe2415cd50 0x7efe241510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48456/0 (socket says 172.21.15.148:48456) 2023-10-30T13:44:20.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2acdb640 1 -- 172.21.15.148:0/2976090674 learned_addr learned my addr 172.21.15.148:0/2976090674 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:20.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2a4da640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe240cefa0 0x7efe241f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:20.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe29cd9640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe241538f0 0x7efe2413fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:20.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2acdb640 1 -- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe241538f0 msgr2=0x7efe2413fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:20.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2acdb640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe241538f0 0x7efe2413fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:20.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2acdb640 1 -- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe240cefa0 msgr2=0x7efe241f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:20.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2acdb640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe240cefa0 0x7efe241f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:20.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2acdb640 1 -- 172.21.15.148:0/2976090674 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c009a70 con 0x7efe2415cd50 2023-10-30T13:44:20.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2acdb640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe2415cd50 0x7efe241510e0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7efe1c009e60 tx=0x7efe1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:20.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe1affd640 1 -- 172.21.15.148:0/2976090674 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c009cd0 con 0x7efe2415cd50 2023-10-30T13:44:20.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe1affd640 1 -- 172.21.15.148:0/2976090674 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c00f040 con 0x7efe2415cd50 2023-10-30T13:44:20.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe1affd640 1 -- 172.21.15.148:0/2976090674 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c0039a0 con 0x7efe2415cd50 2023-10-30T13:44:20.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe14009420 con 0x7efe2415cd50 2023-10-30T13:44:20.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe240b3340 con 0x7efe2415cd50 2023-10-30T13:44:20.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.941+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2404e170 con 0x7efe2415cd50 2023-10-30T13:44:20.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.945+0000 7efe1affd640 1 -- 172.21.15.148:0/2976090674 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe1c014480 con 0x7efe2415cd50 2023-10-30T13:44:20.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.945+0000 7efe1affd640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe080664a0 0x7efe08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:20.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.945+0000 7efe1affd640 1 -- 172.21.15.148:0/2976090674 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7efe1c084f00 con 0x7efe2415cd50 2023-10-30T13:44:20.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.945+0000 7efe2a4da640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe080664a0 0x7efe08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:20.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.945+0000 7efe2a4da640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe080664a0 0x7efe08068960 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7efe20006b60 tx=0x7efe20001dd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:20.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:20.953+0000 7efe1affd640 1 -- 172.21.15.148:0/2976090674 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe1c051e30 con 0x7efe2415cd50 2023-10-30T13:44:20.954 INFO:tasks.workunit.client.0.smithi148.stdout:v0.66 2023-10-30T13:44:20.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 66 -eq 48 ] 2023-10-30T13:44:20.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.66 2023-10-30T13:44:20.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.66 2023-10-30T13:44:20.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.66 2023-10-30T13:44:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.209+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe240b3db0 con 0x7efe2415cd50 2023-10-30T13:44:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.209+0000 7efe1affd640 1 -- 172.21.15.148:0/2976090674 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+6515 (secure 0 0 0) 0x7efe1c055450 con 0x7efe2415cd50 2023-10-30T13:44:21.209 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:21.210 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 125 2023-10-30T13:44:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe080664a0 msgr2=0x7efe08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe080664a0 0x7efe08068960 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7efe20006b60 tx=0x7efe20001dd0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe2415cd50 msgr2=0x7efe241510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe2415cd50 0x7efe241510e0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7efe1c009e60 tx=0x7efe1c00c490 comp rx=0 tx=0).stop 2023-10-30T13:44:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 shutdown_connections 2023-10-30T13:44:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efe080664a0 0x7efe08068960 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe2415cd50 0x7efe241510e0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efe241538f0 0x7efe2413fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 --2- 172.21.15.148:0/2976090674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe240cefa0 0x7efe241f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 >> 172.21.15.148:0/2976090674 conn(0x7efe2406a640 msgr2=0x7efe240c48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 shutdown_connections 2023-10-30T13:44:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.213+0000 7efe2b4dc640 1 -- 172.21.15.148:0/2976090674 wait complete. 2023-10-30T13:44:21.231 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2023-10-30T13:44:21.231 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:44:21.232 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:44:21.264 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:44:21.265 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:21.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dcb6d6640 1 Processor -- start 2023-10-30T13:44:21.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dcb6d6640 1 -- start start 2023-10-30T13:44:21.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dcb6d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2dc41538a0 0x7f2dc415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:21.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dcb6d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc40cb800 con 0x7f2dc40ca0e0 2023-10-30T13:44:21.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dcb6d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc415dc30 con 0x7f2dc40cf160 2023-10-30T13:44:21.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dcb6d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc40cb330 con 0x7f2dc41538a0 2023-10-30T13:44:21.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc8c71640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2dc40ca0e0 0x7f2dc415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44284/0 (socket says 172.21.15.148:44284) 2023-10-30T13:44:21.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc8c71640 1 -- 172.21.15.148:0/4114871622 learned_addr learned my addr 172.21.15.148:0/4114871622 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:21.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc9472640 1 --2- 172.21.15.148:0/4114871622 >> v2:172.21.15.150:3300/0 conn(0x7f2dc41538a0 0x7f2dc415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:21.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900764924 0 0) 0x7f2dc40cb800 con 0x7f2dc40ca0e0 2023-10-30T13:44:21.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0003530 con 0x7f2dc40ca0e0 2023-10-30T13:44:21.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3147317859 0 0) 0x7f2db0003530 con 0x7f2dc40ca0e0 2023-10-30T13:44:21.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 >> v2:172.21.15.150:3300/0 conn(0x7f2dc41538a0 msgr2=0x7f2dc415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:21.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 --2- 172.21.15.148:0/4114871622 >> v2:172.21.15.150:3300/0 conn(0x7f2dc41538a0 0x7f2dc415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 >> v1:172.21.15.150:6789/0 conn(0x7f2dc40cf160 legacy=0x7f2dc4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:21.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc40cad70 con 0x7f2dc40ca0e0 2023-10-30T13:44:21.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f2db4001680 con 0x7f2dc40ca0e0 2023-10-30T13:44:21.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 >> v1:172.21.15.148:6789/0 conn(0x7f2dc40ca0e0 legacy=0x7f2dc415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:21.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2db0003980 0x7f2db0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:21.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db00047d0 0x7f2db0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:21.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2db000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:21.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc40cb330 con 0x7f2db0005190 2023-10-30T13:44:21.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db000c040 con 0x7f2db00047d0 2023-10-30T13:44:21.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db000c200 con 0x7f2db0003980 2023-10-30T13:44:21.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc9c73640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2db000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:21.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc8c71640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db00047d0 0x7f2db0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:21.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc9472640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2db0003980 0x7f2db0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:21.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc9c73640 1 -- 172.21.15.148:0/4114871622 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2db0003980 msgr2=0x7f2db0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:21.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc9c73640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2db0003980 0x7f2db0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc9c73640 1 -- 172.21.15.148:0/4114871622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db00047d0 msgr2=0x7f2db0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:21.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc9c73640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db00047d0 0x7f2db0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc9c73640 1 -- 172.21.15.148:0/4114871622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0003530 con 0x7f2db0005190 2023-10-30T13:44:21.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.445+0000 7f2dc9472640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2db0003980 0x7f2db0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:21.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dc9c73640 1 -- 172.21.15.148:0/4114871622 reap_dead start 2023-10-30T13:44:21.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dc9c73640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2db000b990 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc001cd0 tx=0x7f2dbc01c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99d2a96143ce389 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:21.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc003b80 con 0x7f2db0005190 2023-10-30T13:44:21.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc003d20 con 0x7f2db0005190 2023-10-30T13:44:21.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dbbfff640 1 -- 172.21.15.148:0/4114871622 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc005310 con 0x7f2db0005190 2023-10-30T13:44:21.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/4114871622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 msgr2=0x7f2db000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:21.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dcb6d6640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2db000b990 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc001cd0 tx=0x7f2dbc01c5c0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/4114871622 shutdown_connections 2023-10-30T13:44:21.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dcb6d6640 1 --2- 172.21.15.148:0/4114871622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2db000b990 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/4114871622 >> 172.21.15.148:0/4114871622 conn(0x7f2dc406bfe0 msgr2=0x7f2dc406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:21.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/4114871622 shutdown_connections 2023-10-30T13:44:21.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.449+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/4114871622 wait complete. 2023-10-30T13:44:21.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 Processor -- start 2023-10-30T13:44:21.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 -- start start 2023-10-30T13:44:21.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2dc41510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:21.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2dc41525a0 0x7f2dc40b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:21.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dc41efe20 0x7f2dc41459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:21.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc415d340 con 0x7f2db0005190 2023-10-30T13:44:21.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc41532a0 con 0x7f2dc41efe20 2023-10-30T13:44:21.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc4152b10 con 0x7f2dc41525a0 2023-10-30T13:44:21.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9472640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2dc41510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:21.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9472640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2dc41510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48470/0 (socket says 172.21.15.148:48470) 2023-10-30T13:44:21.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9472640 1 -- 172.21.15.148:0/3003103723 learned_addr learned my addr 172.21.15.148:0/3003103723 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:21.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9c73640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dc41efe20 0x7f2dc41459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:21.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc8c71640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2dc41525a0 0x7f2dc40b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:21.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9472640 1 -- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2dc41525a0 msgr2=0x7f2dc40b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:21.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9472640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2dc41525a0 0x7f2dc40b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9472640 1 -- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dc41efe20 msgr2=0x7f2dc41459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:21.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9472640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dc41efe20 0x7f2dc41459d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9472640 1 -- 172.21.15.148:0/3003103723 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4152d00 con 0x7f2db0005190 2023-10-30T13:44:21.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9c73640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dc41efe20 0x7f2dc41459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:21.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dc9472640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2dc41510d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009fd0 tx=0x7f2dc0011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:21.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2db9ffb640 1 -- 172.21.15.148:0/3003103723 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0004220 con 0x7f2db0005190 2023-10-30T13:44:21.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2db9ffb640 1 -- 172.21.15.148:0/3003103723 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc0004380 con 0x7f2db0005190 2023-10-30T13:44:21.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc4152fc0 con 0x7f2db0005190 2023-10-30T13:44:21.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2db9ffb640 1 -- 172.21.15.148:0/3003103723 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0015890 con 0x7f2db0005190 2023-10-30T13:44:21.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.453+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc41ef620 con 0x7f2db0005190 2023-10-30T13:44:21.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.457+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc41ef130 con 0x7f2db0005190 2023-10-30T13:44:21.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.457+0000 7f2db9ffb640 1 -- 172.21.15.148:0/3003103723 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2dc00159f0 con 0x7f2db0005190 2023-10-30T13:44:21.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.457+0000 7f2db9ffb640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2da80664a0 0x7f2da8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:21.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.457+0000 7f2db9ffb640 1 -- 172.21.15.148:0/3003103723 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f2dc0084960 con 0x7f2db0005190 2023-10-30T13:44:21.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.457+0000 7f2dc8c71640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2da80664a0 0x7f2da8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:21.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.457+0000 7f2dc8c71640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2da80664a0 0x7f2da8068960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2db4001a00 tx=0x7f2db4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:21.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.461+0000 7f2db9ffb640 1 -- 172.21.15.148:0/3003103723 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2dc0051750 con 0x7f2db0005190 2023-10-30T13:44:21.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dc404e170 con 0x7f2db0005190 2023-10-30T13:44:21.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2db9ffb640 1 -- 172.21.15.148:0/3003103723 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f2dc0054d70 con 0x7f2db0005190 2023-10-30T13:44:21.639 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:21.639 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:21.639 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 125 2023-10-30T13:44:21.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2da80664a0 msgr2=0x7f2da8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:21.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2da80664a0 0x7f2da8068960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2db4001a00 tx=0x7f2db4009040 comp rx=0 tx=0).stop 2023-10-30T13:44:21.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 msgr2=0x7f2dc41510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:21.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2dc41510d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009fd0 tx=0x7f2dc0011a60 comp rx=0 tx=0).stop 2023-10-30T13:44:21.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 shutdown_connections 2023-10-30T13:44:21.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2da80664a0 0x7f2da8068960 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dc41efe20 0x7f2dc41459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2dc41525a0 0x7f2dc40b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 --2- 172.21.15.148:0/3003103723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2db0005190 0x7f2dc41510d0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:21.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 >> 172.21.15.148:0/3003103723 conn(0x7f2dc406bfe0 msgr2=0x7f2dc4157ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:21.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 shutdown_connections 2023-10-30T13:44:21.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:21.641+0000 7f2dcb6d6640 1 -- 172.21.15.148:0/3003103723 wait complete. 2023-10-30T13:44:21.660 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:44:23.661 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:23.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ffa646640 1 Processor -- start 2023-10-30T13:44:23.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ffa646640 1 -- start start 2023-10-30T13:44:23.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ffa646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070650 0x7f5ff4071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ffa646640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5ff4070c90 0x7f5ff40720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ffa646640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff40c2f20 con 0x7f5ff4156f80 2023-10-30T13:44:23.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ffa646640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff4157360 con 0x7f5ff4070c90 2023-10-30T13:44:23.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ffa646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff40c85c0 con 0x7f5ff4070650 2023-10-30T13:44:23.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ff8be3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5ff4070c90 0x7f5ff40720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ff8be3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5ff4070c90 0x7f5ff40720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35624/0 (socket says 172.21.15.148:35624) 2023-10-30T13:44:23.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ff8be3640 1 -- 172.21.15.148:0/2116708505 learned_addr learned my addr 172.21.15.148:0/2116708505 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:23.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ff37fe640 1 --2- 172.21.15.148:0/2116708505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070650 0x7f5ff4071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ff8be3640 1 -- 172.21.15.148:0/2116708505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070650 msgr2=0x7f5ff4071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:23.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ff8be3640 1 --2- 172.21.15.148:0/2116708505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070650 0x7f5ff4071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ff8be3640 1 -- 172.21.15.148:0/2116708505 >> v1:172.21.15.150:6789/0 conn(0x7f5ff4156f80 legacy=0x7f5ff4159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.813+0000 7f5ff8be3640 1 -- 172.21.15.148:0/2116708505 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff4071230 con 0x7f5ff4070c90 2023-10-30T13:44:23.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ff8be3640 1 --2- 172.21.15.148:0/2116708505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4070c90 0x7f5ff40720f0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009e50 tx=0x7f5fec01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f9458514a2e6a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:23.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ff2ffd640 1 -- 172.21.15.148:0/2116708505 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec0a5870 con 0x7f5ff4070c90 2023-10-30T13:44:23.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ff2ffd640 1 -- 172.21.15.148:0/2116708505 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec0a59d0 con 0x7f5ff4070c90 2023-10-30T13:44:23.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ff2ffd640 1 -- 172.21.15.148:0/2116708505 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec0a5d60 con 0x7f5ff4070c90 2023-10-30T13:44:23.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ffa646640 1 -- 172.21.15.148:0/2116708505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4070c90 msgr2=0x7f5ff40720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ffa646640 1 --2- 172.21.15.148:0/2116708505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4070c90 0x7f5ff40720f0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009e50 tx=0x7f5fec01ca10 comp rx=0 tx=0).stop 2023-10-30T13:44:23.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ffa646640 1 -- 172.21.15.148:0/2116708505 shutdown_connections 2023-10-30T13:44:23.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ffa646640 1 --2- 172.21.15.148:0/2116708505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4070c90 0x7f5ff40720f0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ffa646640 1 --2- 172.21.15.148:0/2116708505 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070650 0x7f5ff4071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ffa646640 1 -- 172.21.15.148:0/2116708505 >> 172.21.15.148:0/2116708505 conn(0x7f5ff40cbeb0 msgr2=0x7f5ff40cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:23.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ffa646640 1 -- 172.21.15.148:0/2116708505 shutdown_connections 2023-10-30T13:44:23.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.817+0000 7f5ffa646640 1 -- 172.21.15.148:0/2116708505 wait complete. 2023-10-30T13:44:23.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 Processor -- start 2023-10-30T13:44:23.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 -- start start 2023-10-30T13:44:23.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ff4070650 0x7f5ff4152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070c90 0x7f5ff41f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4156f80 0x7f5ff414bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff4158ba0 con 0x7f5ff4156f80 2023-10-30T13:44:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff41f5c90 con 0x7f5ff4070650 2023-10-30T13:44:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff41547e0 con 0x7f5ff4070c90 2023-10-30T13:44:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff8be3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4156f80 0x7f5ff414bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff8be3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4156f80 0x7f5ff414bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35636/0 (socket says 172.21.15.148:35636) 2023-10-30T13:44:23.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff8be3640 1 -- 172.21.15.148:0/863892862 learned_addr learned my addr 172.21.15.148:0/863892862 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:23.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff37fe640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070c90 0x7f5ff41f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff3fff640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ff4070650 0x7f5ff4152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff8be3640 1 -- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070c90 msgr2=0x7f5ff41f6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff8be3640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070c90 0x7f5ff41f6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff8be3640 1 -- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ff4070650 msgr2=0x7f5ff4152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:23.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff8be3640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ff4070650 0x7f5ff4152e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:23.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff8be3640 1 -- 172.21.15.148:0/863892862 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec009b30 con 0x7f5ff4156f80 2023-10-30T13:44:23.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff3fff640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ff4070650 0x7f5ff4152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:23.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff8be3640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4156f80 0x7f5ff414bce0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5fec01a640 tx=0x7f5fec0197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:23.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff0ff9640 1 -- 172.21.15.148:0/863892862 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec0bd070 con 0x7f5ff4156f80 2023-10-30T13:44:23.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff0ff9640 1 -- 172.21.15.148:0/863892862 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec0b8470 con 0x7f5ff4156f80 2023-10-30T13:44:23.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff41533b0 con 0x7f5ff4156f80 2023-10-30T13:44:23.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ff0ff9640 1 -- 172.21.15.148:0/863892862 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec0b8820 con 0x7f5ff4156f80 2023-10-30T13:44:23.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.821+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff4155dd0 con 0x7f5ff4156f80 2023-10-30T13:44:23.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.825+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff404e170 con 0x7f5ff4156f80 2023-10-30T13:44:23.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.825+0000 7f5ff0ff9640 1 -- 172.21.15.148:0/863892862 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5fec0b8980 con 0x7f5ff4156f80 2023-10-30T13:44:23.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.825+0000 7f5ff0ff9640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5fe00664a0 0x7f5fe0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:23.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.825+0000 7f5ff0ff9640 1 -- 172.21.15.148:0/863892862 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f5fec11f250 con 0x7f5ff4156f80 2023-10-30T13:44:23.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.825+0000 7f5ff3fff640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5fe00664a0 0x7f5fe0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:23.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.825+0000 7f5ff3fff640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5fe00664a0 0x7f5fe0068960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4000f70 tx=0x7f5fe4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:23.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:23.829+0000 7f5ff0ff9640 1 -- 172.21.15.148:0/863892862 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5fec0ebfb0 con 0x7f5ff4156f80 2023-10-30T13:44:24.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.009+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ff41f5dd0 con 0x7f5ff4156f80 2023-10-30T13:44:24.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.009+0000 7f5ff0ff9640 1 -- 172.21.15.148:0/863892862 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f5fec0ef5d0 con 0x7f5ff4156f80 2023-10-30T13:44:24.009 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:24.010 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:24.010 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 125 2023-10-30T13:44:24.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5fe00664a0 msgr2=0x7f5fe0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5fe00664a0 0x7f5fe0068960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4000f70 tx=0x7f5fe4009040 comp rx=0 tx=0).stop 2023-10-30T13:44:24.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4156f80 msgr2=0x7f5ff414bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:24.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4156f80 0x7f5ff414bce0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5fec01a640 tx=0x7f5fec0197b0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 shutdown_connections 2023-10-30T13:44:24.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5fe00664a0 0x7f5fe0068960 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ff4156f80 0x7f5ff414bce0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ff4070c90 0x7f5ff41f6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 --2- 172.21.15.148:0/863892862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ff4070650 0x7f5ff4152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:24.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 >> 172.21.15.148:0/863892862 conn(0x7f5ff40cbeb0 msgr2=0x7f5ff40c1740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:24.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 shutdown_connections 2023-10-30T13:44:24.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:24.013+0000 7f5ffa646640 1 -- 172.21.15.148:0/863892862 wait complete. 2023-10-30T13:44:24.031 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:44:26.032 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:26.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.193+0000 7f28cdad6640 1 Processor -- start 2023-10-30T13:44:26.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.193+0000 7f28cdad6640 1 -- start start 2023-10-30T13:44:26.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.193+0000 7f28cdad6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f28c806f7a0 0x7f28c806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.193+0000 7f28cdad6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f28c80c6bd0 0x7f28c8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.193+0000 7f28cdad6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c806a3d0 con 0x7f28c8071110 2023-10-30T13:44:26.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.193+0000 7f28cdad6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c80700c0 con 0x7f28c806f7a0 2023-10-30T13:44:26.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.193+0000 7f28cdad6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c8066cf0 con 0x7f28c80c6bd0 2023-10-30T13:44:26.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c77fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f28c806f7a0 0x7f28c806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c77fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f28c806f7a0 0x7f28c806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35638/0 (socket says 172.21.15.148:35638) 2023-10-30T13:44:26.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c7fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f28c8071110 0x7f28c8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58740/0 (socket says 172.21.15.148:58740) 2023-10-30T13:44:26.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c77fe640 1 -- 172.21.15.148:0/3113630342 learned_addr learned my addr 172.21.15.148:0/3113630342 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:26.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c6ffd640 1 --2- 172.21.15.148:0/3113630342 >> v2:172.21.15.150:3300/0 conn(0x7f28c80c6bd0 0x7f28c8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c77fe640 1 -- 172.21.15.148:0/3113630342 >> v2:172.21.15.150:3300/0 conn(0x7f28c80c6bd0 msgr2=0x7f28c8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:26.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c77fe640 1 --2- 172.21.15.148:0/3113630342 >> v2:172.21.15.150:3300/0 conn(0x7f28c80c6bd0 0x7f28c8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c77fe640 1 -- 172.21.15.148:0/3113630342 >> v1:172.21.15.148:6789/0 conn(0x7f28c8071110 legacy=0x7f28c8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c77fe640 1 -- 172.21.15.148:0/3113630342 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c80c70d0 con 0x7f28c806f7a0 2023-10-30T13:44:26.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c67fc640 1 -- 172.21.15.148:0/3113630342 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824284087 0 0) 0x7f28c806a3d0 con 0x7f28c8071110 2023-10-30T13:44:26.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.197+0000 7f28c77fe640 1 --2- 172.21.15.148:0/3113630342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 0x7f28c806fb80 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f28c806f1c0 tx=0x7f28b801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7787c41b869a854d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:26.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28c67fc640 1 -- 172.21.15.148:0/3113630342 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b80a89f0 con 0x7f28c806f7a0 2023-10-30T13:44:26.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28c67fc640 1 -- 172.21.15.148:0/3113630342 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b80a8b50 con 0x7f28c806f7a0 2023-10-30T13:44:26.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28c67fc640 1 -- 172.21.15.148:0/3113630342 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b80b0440 con 0x7f28c806f7a0 2023-10-30T13:44:26.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 -- 172.21.15.148:0/3113630342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 msgr2=0x7f28c806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 --2- 172.21.15.148:0/3113630342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 0x7f28c806fb80 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f28c806f1c0 tx=0x7f28b801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 -- 172.21.15.148:0/3113630342 shutdown_connections 2023-10-30T13:44:26.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 --2- 172.21.15.148:0/3113630342 >> v2:172.21.15.150:3300/0 conn(0x7f28c80c6bd0 0x7f28c8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 --2- 172.21.15.148:0/3113630342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 0x7f28c806fb80 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 -- 172.21.15.148:0/3113630342 >> 172.21.15.148:0/3113630342 conn(0x7f28c80c7910 msgr2=0x7f28c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:26.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 -- 172.21.15.148:0/3113630342 shutdown_connections 2023-10-30T13:44:26.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 -- 172.21.15.148:0/3113630342 wait complete. 2023-10-30T13:44:26.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 Processor -- start 2023-10-30T13:44:26.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 -- start start 2023-10-30T13:44:26.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 0x7f28c814be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c8071110 0x7f28c8156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c80c6bd0 0x7f28c8156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c81f01e0 con 0x7f28c806f7a0 2023-10-30T13:44:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c814b1e0 con 0x7f28c8071110 2023-10-30T13:44:26.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28cdad6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c814a5d0 con 0x7f28c80c6bd0 2023-10-30T13:44:26.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28c77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 0x7f28c814be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28c77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 0x7f28c814be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35646/0 (socket says 172.21.15.148:35646) 2023-10-30T13:44:26.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.201+0000 7f28c77fe640 1 -- 172.21.15.148:0/1542535192 learned_addr learned my addr 172.21.15.148:0/1542535192 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:26.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28c6ffd640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c8071110 0x7f28c8156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28c7fff640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c80c6bd0 0x7f28c8156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28c77fe640 1 -- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c80c6bd0 msgr2=0x7f28c8156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28c77fe640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c80c6bd0 0x7f28c8156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28c77fe640 1 -- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c8071110 msgr2=0x7f28c8156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28c77fe640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c8071110 0x7f28c8156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28c77fe640 1 -- 172.21.15.148:0/1542535192 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b80094d0 con 0x7f28c806f7a0 2023-10-30T13:44:26.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28c77fe640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 0x7f28c814be20 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f28b80b3040 tx=0x7f28b801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:26.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28affff640 1 -- 172.21.15.148:0/1542535192 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b80ba600 con 0x7f28c806f7a0 2023-10-30T13:44:26.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28affff640 1 -- 172.21.15.148:0/1542535192 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b80ba760 con 0x7f28c806f7a0 2023-10-30T13:44:26.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c814a8b0 con 0x7f28c806f7a0 2023-10-30T13:44:26.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28affff640 1 -- 172.21.15.148:0/1542535192 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b80bab10 con 0x7f28c806f7a0 2023-10-30T13:44:26.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c814ae70 con 0x7f28c806f7a0 2023-10-30T13:44:26.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c804e170 con 0x7f28c806f7a0 2023-10-30T13:44:26.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28affff640 1 -- 172.21.15.148:0/1542535192 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f28b80bac70 con 0x7f28c806f7a0 2023-10-30T13:44:26.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28affff640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28b40664a0 0x7f28b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:26.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.205+0000 7f28affff640 1 -- 172.21.15.148:0/1542535192 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f28b8122a70 con 0x7f28c806f7a0 2023-10-30T13:44:26.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.209+0000 7f28c6ffd640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28b40664a0 0x7f28b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.209+0000 7f28c6ffd640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28b40664a0 0x7f28b4068960 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f28c8155e80 tx=0x7f28bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:26.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.209+0000 7f28affff640 1 -- 172.21.15.148:0/1542535192 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f28b80ef980 con 0x7f28c806f7a0 2023-10-30T13:44:26.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28c806a170 con 0x7f28c806f7a0 2023-10-30T13:44:26.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28affff640 1 -- 172.21.15.148:0/1542535192 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f28b80b5090 con 0x7f28c806f7a0 2023-10-30T13:44:26.391 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:26.391 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:26.392 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 125 2023-10-30T13:44:26.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28b40664a0 msgr2=0x7f28b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28b40664a0 0x7f28b4068960 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f28c8155e80 tx=0x7f28bc008040 comp rx=0 tx=0).stop 2023-10-30T13:44:26.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 msgr2=0x7f28c814be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:26.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 0x7f28c814be20 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f28b80b3040 tx=0x7f28b801ba90 comp rx=0 tx=0).stop 2023-10-30T13:44:26.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 shutdown_connections 2023-10-30T13:44:26.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f28b40664a0 0x7f28b4068960 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f28c80c6bd0 0x7f28c8156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28c8071110 0x7f28c8156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 --2- 172.21.15.148:0/1542535192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28c806f7a0 0x7f28c814be20 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:26.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 >> 172.21.15.148:0/1542535192 conn(0x7f28c80c7910 msgr2=0x7f28c80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:26.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 shutdown_connections 2023-10-30T13:44:26.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:26.393+0000 7f28cdad6640 1 -- 172.21.15.148:0/1542535192 wait complete. 2023-10-30T13:44:26.413 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:44:28.415 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:28.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a71aa640 1 Processor -- start 2023-10-30T13:44:28.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a71aa640 1 -- start start 2023-10-30T13:44:28.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a71aa640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47a01538a0 0x7f47a015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a71aa640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f47a00cf160 0x7f47a00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a71aa640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a00cb800 con 0x7f47a00ca0e0 2023-10-30T13:44:28.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a71aa640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a015dc30 con 0x7f47a00cf160 2023-10-30T13:44:28.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a71aa640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a00cb330 con 0x7f47a01538a0 2023-10-30T13:44:28.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a5747640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f47a00cf160 0x7f47a00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a5747640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f47a00cf160 0x7f47a00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35652/0 (socket says 172.21.15.148:35652) 2023-10-30T13:44:28.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a5747640 1 -- 172.21.15.148:0/4146689557 learned_addr learned my addr 172.21.15.148:0/4146689557 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:28.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a4f46640 1 --2- 172.21.15.148:0/4146689557 >> v2:172.21.15.150:3300/0 conn(0x7f47a01538a0 0x7f47a015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a5747640 1 -- 172.21.15.148:0/4146689557 >> v2:172.21.15.150:3300/0 conn(0x7f47a01538a0 msgr2=0x7f47a015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:28.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a5747640 1 --2- 172.21.15.148:0/4146689557 >> v2:172.21.15.150:3300/0 conn(0x7f47a01538a0 0x7f47a015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a5747640 1 -- 172.21.15.148:0/4146689557 >> v1:172.21.15.150:6789/0 conn(0x7f47a00ca0e0 legacy=0x7f47a015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:28.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.565+0000 7f47a5747640 1 -- 172.21.15.148:0/4146689557 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a00cad70 con 0x7f47a00cf160 2023-10-30T13:44:28.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f47a5747640 1 --2- 172.21.15.148:0/4146689557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 0x7f47a00cf590 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f47a00cfd10 tx=0x7f479801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12db14c60ef5a83a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:28.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f479f7fe640 1 -- 172.21.15.148:0/4146689557 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980af400 con 0x7f47a00cf160 2023-10-30T13:44:28.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f479f7fe640 1 -- 172.21.15.148:0/4146689557 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47980ab050 con 0x7f47a00cf160 2023-10-30T13:44:28.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f479f7fe640 1 -- 172.21.15.148:0/4146689557 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980af780 con 0x7f47a00cf160 2023-10-30T13:44:28.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f47a71aa640 1 -- 172.21.15.148:0/4146689557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 msgr2=0x7f47a00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:28.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f47a71aa640 1 --2- 172.21.15.148:0/4146689557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 0x7f47a00cf590 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f47a00cfd10 tx=0x7f479801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f47a71aa640 1 -- 172.21.15.148:0/4146689557 shutdown_connections 2023-10-30T13:44:28.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f47a71aa640 1 --2- 172.21.15.148:0/4146689557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 0x7f47a00cf590 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f47a71aa640 1 --2- 172.21.15.148:0/4146689557 >> v2:172.21.15.150:3300/0 conn(0x7f47a01538a0 0x7f47a015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f47a71aa640 1 -- 172.21.15.148:0/4146689557 >> 172.21.15.148:0/4146689557 conn(0x7f47a006bfe0 msgr2=0x7f47a006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:28.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f47a71aa640 1 -- 172.21.15.148:0/4146689557 shutdown_connections 2023-10-30T13:44:28.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.569+0000 7f47a71aa640 1 -- 172.21.15.148:0/4146689557 wait complete. 2023-10-30T13:44:28.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 Processor -- start 2023-10-30T13:44:28.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 -- start start 2023-10-30T13:44:28.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00ca0e0 0x7f47a01480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 0x7f47a0158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a01538a0 0x7f47a01f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01e5330 con 0x7f47a00cf160 2023-10-30T13:44:28.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00b4f30 con 0x7f47a00ca0e0 2023-10-30T13:44:28.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01e4d70 con 0x7f47a01538a0 2023-10-30T13:44:28.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 0x7f47a0158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 0x7f47a0158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35654/0 (socket says 172.21.15.148:35654) 2023-10-30T13:44:28.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a4f46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00ca0e0 0x7f47a01480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479ffff640 1 -- 172.21.15.148:0/1487134008 learned_addr learned my addr 172.21.15.148:0/1487134008 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:28.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a5747640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a01538a0 0x7f47a01f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479ffff640 1 -- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a01538a0 msgr2=0x7f47a01f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:28.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479ffff640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a01538a0 0x7f47a01f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479ffff640 1 -- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00ca0e0 msgr2=0x7f47a01480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:28.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479ffff640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00ca0e0 0x7f47a01480f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479ffff640 1 -- 172.21.15.148:0/1487134008 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798009b30 con 0x7f47a00cf160 2023-10-30T13:44:28.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479ffff640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 0x7f47a0158f80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f478c00c3c0 tx=0x7f478c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:28.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479d7fa640 1 -- 172.21.15.148:0/1487134008 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c0052d0 con 0x7f47a00cf160 2023-10-30T13:44:28.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479d7fa640 1 -- 172.21.15.148:0/1487134008 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478c005430 con 0x7f47a00cf160 2023-10-30T13:44:28.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a01e5050 con 0x7f47a00cf160 2023-10-30T13:44:28.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f479d7fa640 1 -- 172.21.15.148:0/1487134008 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c003840 con 0x7f47a00cf160 2023-10-30T13:44:28.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.573+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a00b4bc0 con 0x7f47a00cf160 2023-10-30T13:44:28.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.577+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a00b4d80 con 0x7f47a00cf160 2023-10-30T13:44:28.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.577+0000 7f479d7fa640 1 -- 172.21.15.148:0/1487134008 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f478c0039a0 con 0x7f47a00cf160 2023-10-30T13:44:28.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.577+0000 7f479d7fa640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47840664a0 0x7f4784068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:28.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.577+0000 7f479d7fa640 1 -- 172.21.15.148:0/1487134008 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f478c083940 con 0x7f47a00cf160 2023-10-30T13:44:28.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.577+0000 7f47a4f46640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47840664a0 0x7f4784068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:28.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.577+0000 7f47a4f46640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47840664a0 0x7f4784068960 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4794007a60 tx=0x7f4794008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:28.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.581+0000 7f479d7fa640 1 -- 172.21.15.148:0/1487134008 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f478c0507f0 con 0x7f47a00cf160 2023-10-30T13:44:28.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.761+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47a00b2a50 con 0x7f47a00cf160 2023-10-30T13:44:28.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.761+0000 7f479d7fa640 1 -- 172.21.15.148:0/1487134008 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f478c017090 con 0x7f47a00cf160 2023-10-30T13:44:28.760 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:28.760 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:28.761 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 125 2023-10-30T13:44:28.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47840664a0 msgr2=0x7f4784068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:28.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47840664a0 0x7f4784068960 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4794007a60 tx=0x7f4794008040 comp rx=0 tx=0).stop 2023-10-30T13:44:28.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 msgr2=0x7f47a0158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:28.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 0x7f47a0158f80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f478c00c3c0 tx=0x7f478c00c880 comp rx=0 tx=0).stop 2023-10-30T13:44:28.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 shutdown_connections 2023-10-30T13:44:28.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f47840664a0 0x7f4784068960 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47a00cf160 0x7f47a0158f80 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f47a01538a0 0x7f47a01f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.763 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 --2- 172.21.15.148:0/1487134008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a00ca0e0 0x7f47a01480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:28.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 >> 172.21.15.148:0/1487134008 conn(0x7f47a006bfe0 msgr2=0x7f47a006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:28.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 shutdown_connections 2023-10-30T13:44:28.764 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:28.765+0000 7f47a71aa640 1 -- 172.21.15.148:0/1487134008 wait complete. 2023-10-30T13:44:28.781 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:44:30.782 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:30.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4bcf3e640 1 Processor -- start 2023-10-30T13:44:30.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4bcf3e640 1 -- start start 2023-10-30T13:44:30.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4bcf3e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff4b81538a0 0x7ff4b815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:30.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4bcf3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 0x7ff4b80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:30.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4bcf3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b80cb800 con 0x7ff4b80ca0e0 2023-10-30T13:44:30.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4bcf3e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b815dc30 con 0x7ff4b81538a0 2023-10-30T13:44:30.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4bcf3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b80cb330 con 0x7ff4b80cf160 2023-10-30T13:44:30.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4b659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff4b81538a0 0x7ff4b815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:30.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4b659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff4b81538a0 0x7ff4b815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35658/0 (socket says 172.21.15.148:35658) 2023-10-30T13:44:30.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4b659c640 1 -- 172.21.15.148:0/1466801965 learned_addr learned my addr 172.21.15.148:0/1466801965 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:30.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4b6d9d640 1 --2- 172.21.15.148:0/1466801965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 0x7ff4b80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:30.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4b659c640 1 -- 172.21.15.148:0/1466801965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 msgr2=0x7ff4b80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:30.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4b659c640 1 --2- 172.21.15.148:0/1466801965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 0x7ff4b80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:30.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4b659c640 1 -- 172.21.15.148:0/1466801965 >> v1:172.21.15.150:6789/0 conn(0x7ff4b80ca0e0 legacy=0x7ff4b815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:30.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.933+0000 7ff4b659c640 1 -- 172.21.15.148:0/1466801965 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b80cad70 con 0x7ff4b81538a0 2023-10-30T13:44:30.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4b659c640 1 --2- 172.21.15.148:0/1466801965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 0x7ff4b815d6f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009e50 tx=0x7ff4a801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f78073f447cdf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:30.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4b559a640 1 -- 172.21.15.148:0/1466801965 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a80af400 con 0x7ff4b81538a0 2023-10-30T13:44:30.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4b559a640 1 -- 172.21.15.148:0/1466801965 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a80ab050 con 0x7ff4b81538a0 2023-10-30T13:44:30.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4b559a640 1 -- 172.21.15.148:0/1466801965 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a80af780 con 0x7ff4b81538a0 2023-10-30T13:44:30.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/1466801965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 msgr2=0x7ff4b815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4bcf3e640 1 --2- 172.21.15.148:0/1466801965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 0x7ff4b815d6f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009e50 tx=0x7ff4a801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:44:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/1466801965 shutdown_connections 2023-10-30T13:44:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4bcf3e640 1 --2- 172.21.15.148:0/1466801965 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 0x7ff4b80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4bcf3e640 1 --2- 172.21.15.148:0/1466801965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 0x7ff4b815d6f0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/1466801965 >> 172.21.15.148:0/1466801965 conn(0x7ff4b806bfe0 msgr2=0x7ff4b806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:30.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/1466801965 shutdown_connections 2023-10-30T13:44:30.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.937+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/1466801965 wait complete. 2023-10-30T13:44:30.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4bcf3e640 1 Processor -- start 2023-10-30T13:44:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4bcf3e640 1 -- start start 2023-10-30T13:44:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4bcf3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80ca0e0 0x7ff4b815d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4bcf3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 0x7ff4b815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4bcf3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 0x7ff4b814cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4bcf3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b815bc50 con 0x7ff4b81538a0 2023-10-30T13:44:30.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4bcf3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b81efb00 con 0x7ff4b80ca0e0 2023-10-30T13:44:30.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4bcf3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b81ef240 con 0x7ff4b80cf160 2023-10-30T13:44:30.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 0x7ff4b814cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:30.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 0x7ff4b814cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35660/0 (socket says 172.21.15.148:35660) 2023-10-30T13:44:30.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b6d9d640 1 -- 172.21.15.148:0/152247139 learned_addr learned my addr 172.21.15.148:0/152247139 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:30.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b5d9b640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 0x7ff4b815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:30.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b659c640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80ca0e0 0x7ff4b815d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:30.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b6d9d640 1 -- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 msgr2=0x7ff4b815a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:30.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b6d9d640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 0x7ff4b815a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:30.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b6d9d640 1 -- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80ca0e0 msgr2=0x7ff4b815d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:30.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b6d9d640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80ca0e0 0x7ff4b815d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:30.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b6d9d640 1 -- 172.21.15.148:0/152247139 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a8009b30 con 0x7ff4b81538a0 2023-10-30T13:44:30.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4b6d9d640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 0x7ff4b814cd50 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000d580 tx=0x7ff4b000da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:30.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4a6ffd640 1 -- 172.21.15.148:0/152247139 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b00125f0 con 0x7ff4b81538a0 2023-10-30T13:44:30.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4a6ffd640 1 -- 172.21.15.148:0/152247139 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b0012750 con 0x7ff4b81538a0 2023-10-30T13:44:30.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b81e4a90 con 0x7ff4b81538a0 2023-10-30T13:44:30.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.941+0000 7ff4a6ffd640 1 -- 172.21.15.148:0/152247139 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0012b00 con 0x7ff4b81538a0 2023-10-30T13:44:30.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.945+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b81ef790 con 0x7ff4b81538a0 2023-10-30T13:44:30.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.945+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b80b2a50 con 0x7ff4b81538a0 2023-10-30T13:44:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.945+0000 7ff4a6ffd640 1 -- 172.21.15.148:0/152247139 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4b0012c60 con 0x7ff4b81538a0 2023-10-30T13:44:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.945+0000 7ff4a6ffd640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4980664a0 0x7ff498068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.945+0000 7ff4a6ffd640 1 -- 172.21.15.148:0/152247139 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7ff4b0084890 con 0x7ff4b81538a0 2023-10-30T13:44:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.945+0000 7ff4b659c640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4980664a0 0x7ff498068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.945+0000 7ff4b659c640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4980664a0 0x7ff498068960 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80b1000 tx=0x7ff4a80aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:30.949+0000 7ff4a6ffd640 1 -- 172.21.15.148:0/152247139 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4b0051740 con 0x7ff4b81538a0 2023-10-30T13:44:31.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.133+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4b804e170 con 0x7ff4b81538a0 2023-10-30T13:44:31.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.133+0000 7ff4a6ffd640 1 -- 172.21.15.148:0/152247139 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+6515 (secure 0 0 0) 0x7ff4b0054d60 con 0x7ff4b81538a0 2023-10-30T13:44:31.133 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:31.133 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:31.133 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 125 2023-10-30T13:44:31.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4980664a0 msgr2=0x7ff498068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:31.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4980664a0 0x7ff498068960 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80b1000 tx=0x7ff4a80aa000 comp rx=0 tx=0).stop 2023-10-30T13:44:31.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 msgr2=0x7ff4b814cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:31.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 0x7ff4b814cd50 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000d580 tx=0x7ff4b000da40 comp rx=0 tx=0).stop 2023-10-30T13:44:31.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 shutdown_connections 2023-10-30T13:44:31.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4980664a0 0x7ff498068960 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4b81538a0 0x7ff4b814cd50 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4b80cf160 0x7ff4b815a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 --2- 172.21.15.148:0/152247139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80ca0e0 0x7ff4b815d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:31.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 >> 172.21.15.148:0/152247139 conn(0x7ff4b806bfe0 msgr2=0x7ff4b8151cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:31.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 shutdown_connections 2023-10-30T13:44:31.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:31.137+0000 7ff4bcf3e640 1 -- 172.21.15.148:0/152247139 wait complete. 2023-10-30T13:44:31.154 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:44:31.788 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:31.792 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:44:33.155 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:33.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0378eea640 1 Processor -- start 2023-10-30T13:44:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0378eea640 1 -- start start 2023-10-30T13:44:33.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0378eea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374070a00 0x7f0374070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0378eea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0374069e90 0x7f037406a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0378eea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0374071380 con 0x7f03741539a0 2023-10-30T13:44:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0378eea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037406fbb0 con 0x7f0374069e90 2023-10-30T13:44:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0378eea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0374153d80 con 0x7f0374070a00 2023-10-30T13:44:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037259c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f03741539a0 0x7f037415a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36572/0 (socket says 172.21.15.148:36572) 2023-10-30T13:44:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037259c640 1 -- 172.21.15.148:0/2425391509 learned_addr learned my addr 172.21.15.148:0/2425391509 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0371d9b640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374070a00 0x7f0374070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0372d9d640 1 --2- 172.21.15.148:0/2425391509 >> v2:172.21.15.150:3300/0 conn(0x7f0374069e90 0x7f037406a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469112116 0 0) 0x7f0374071380 con 0x7f03741539a0 2023-10-30T13:44:33.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c003530 con 0x7f03741539a0 2023-10-30T13:44:33.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 414143398 0 0) 0x7f035c003530 con 0x7f03741539a0 2023-10-30T13:44:33.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374070a00 msgr2=0x7f0374070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:33.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374070a00 0x7f0374070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 >> v2:172.21.15.150:3300/0 conn(0x7f0374069e90 msgr2=0x7f037406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:33.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 --2- 172.21.15.148:0/2425391509 >> v2:172.21.15.150:3300/0 conn(0x7f0374069e90 0x7f037406a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740c5c50 con 0x7f03741539a0 2023-10-30T13:44:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f0364001680 con 0x7f03741539a0 2023-10-30T13:44:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 >> v1:172.21.15.148:6789/0 conn(0x7f03741539a0 legacy=0x7f037415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f035c003980 0x7f035c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f035c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374070a00 0x7f035c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037406fbb0 con 0x7f035c0047d0 2023-10-30T13:44:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0374153d80 con 0x7f035c003980 2023-10-30T13:44:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035c00b990 con 0x7f0374070a00 2023-10-30T13:44:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.305+0000 7f0371d9b640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f035c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0372d9d640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374070a00 0x7f035c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f037259c640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f035c003980 0x7f035c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0371d9b640 1 -- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374070a00 msgr2=0x7f035c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0371d9b640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374070a00 0x7f035c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0371d9b640 1 -- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f035c003980 msgr2=0x7f035c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:33.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0371d9b640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f035c003980 0x7f035c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0371d9b640 1 -- 172.21.15.148:0/2425391509 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c003530 con 0x7f035c0047d0 2023-10-30T13:44:33.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f037259c640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f035c003980 0x7f035c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:33.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0371d9b640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f035c004c50 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f0368009b10 tx=0x7f036801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9437b279d87068d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:33.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03680a6430 con 0x7f035c0047d0 2023-10-30T13:44:33.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03680a6590 con 0x7f035c0047d0 2023-10-30T13:44:33.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f037159a640 1 -- 172.21.15.148:0/2425391509 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03680a6940 con 0x7f035c0047d0 2023-10-30T13:44:33.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0378eea640 1 -- 172.21.15.148:0/2425391509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 msgr2=0x7f035c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0378eea640 1 --2- 172.21.15.148:0/2425391509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f035c004c50 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f0368009b10 tx=0x7f036801ca20 comp rx=0 tx=0).stop 2023-10-30T13:44:33.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.309+0000 7f0372d9d640 1 -- 172.21.15.148:0/2425391509 reap_dead start 2023-10-30T13:44:33.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 -- 172.21.15.148:0/2425391509 shutdown_connections 2023-10-30T13:44:33.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 -- 172.21.15.148:0/2425391509 >> 172.21.15.148:0/2425391509 conn(0x7f03740c7910 msgr2=0x7f03740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:33.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 -- 172.21.15.148:0/2425391509 shutdown_connections 2023-10-30T13:44:33.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 -- 172.21.15.148:0/2425391509 wait complete. 2023-10-30T13:44:33.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 Processor -- start 2023-10-30T13:44:33.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 -- start start 2023-10-30T13:44:33.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f037415ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374150ad0 0x7f037415f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03740c8d10 0x7f03740c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03740c96d0 con 0x7f035c0047d0 2023-10-30T13:44:33.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03740ca6a0 con 0x7f03740c8d10 2023-10-30T13:44:33.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0378eea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03740c0970 con 0x7f0374150ad0 2023-10-30T13:44:33.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f037259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f037415ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f037259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f037415ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60858/0 (socket says 172.21.15.148:60858) 2023-10-30T13:44:33.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f037259c640 1 -- 172.21.15.148:0/2780047838 learned_addr learned my addr 172.21.15.148:0/2780047838 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:33.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0371d9b640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374150ad0 0x7f037415f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0372d9d640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03740c8d10 0x7f03740c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f037259c640 1 -- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374150ad0 msgr2=0x7f037415f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f037259c640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374150ad0 0x7f037415f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f037259c640 1 -- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03740c8d10 msgr2=0x7f03740c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f037259c640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03740c8d10 0x7f03740c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f037259c640 1 -- 172.21.15.148:0/2780047838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740c0c50 con 0x7f035c0047d0 2023-10-30T13:44:33.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.313+0000 7f0372d9d640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03740c8d10 0x7f03740c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:33.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f037259c640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f037415ec20 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f0364001000 tx=0x7f036400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:33.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0362ffd640 1 -- 172.21.15.148:0/2780047838 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036400c840 con 0x7f035c0047d0 2023-10-30T13:44:33.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0362ffd640 1 -- 172.21.15.148:0/2780047838 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036400c9a0 con 0x7f035c0047d0 2023-10-30T13:44:33.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0362ffd640 1 -- 172.21.15.148:0/2780047838 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036400cd10 con 0x7f035c0047d0 2023-10-30T13:44:33.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03740c0f30 con 0x7f035c0047d0 2023-10-30T13:44:33.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03740ca960 con 0x7f035c0047d0 2023-10-30T13:44:33.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03740c1a60 con 0x7f035c0047d0 2023-10-30T13:44:33.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0362ffd640 1 -- 172.21.15.148:0/2780047838 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f036401d460 con 0x7f035c0047d0 2023-10-30T13:44:33.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0362ffd640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03540664a0 0x7f0354068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:33.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0362ffd640 1 -- 172.21.15.148:0/2780047838 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7230+0+0 (secure 0 0 0) 0x7f03640844b0 con 0x7f035c0047d0 2023-10-30T13:44:33.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.317+0000 7f0371d9b640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03540664a0 0x7f0354068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:33.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.321+0000 7f0371d9b640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03540664a0 0x7f0354068960 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f03680ad000 tx=0x7f03680b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:33.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.321+0000 7f0362ffd640 1 -- 172.21.15.148:0/2780047838 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03740c1a60 con 0x7f035c0047d0 2023-10-30T13:44:33.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.501+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03740c1a60 con 0x7f035c0047d0 2023-10-30T13:44:33.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.501+0000 7f0362ffd640 1 -- 172.21.15.148:0/2780047838 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f03740c1a60 con 0x7f035c0047d0 2023-10-30T13:44:33.500 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:33.501 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:42:58.759857+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":120,"state":"up:active","state_seq":39,"addr":"172.21.15.148:6835/3829165476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":3829165476},{"type":"v1","addr":"172.21.15.148:6835","nonce":3829165476}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:33.502 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 125 2023-10-30T13:44:33.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03540664a0 msgr2=0x7f0354068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03540664a0 0x7f0354068960 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f03680ad000 tx=0x7f03680b3040 comp rx=0 tx=0).stop 2023-10-30T13:44:33.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 msgr2=0x7f037415ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:33.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f037415ec20 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f0364001000 tx=0x7f036400ec80 comp rx=0 tx=0).stop 2023-10-30T13:44:33.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 shutdown_connections 2023-10-30T13:44:33.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03540664a0 0x7f0354068960 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03740c8d10 0x7f03740c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0374150ad0 0x7f037415f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 --2- 172.21.15.148:0/2780047838 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0047d0 0x7f037415ec20 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:33.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 >> 172.21.15.148:0/2780047838 conn(0x7f03740c7910 msgr2=0x7f037406b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:33.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 shutdown_connections 2023-10-30T13:44:33.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:33.505+0000 7f0378eea640 1 -- 172.21.15.148:0/2780047838 wait complete. 2023-10-30T13:44:33.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:44:35.522 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:35.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.673+0000 7f8571a46640 1 Processor -- start 2023-10-30T13:44:35.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f8571a46640 1 -- start start 2023-10-30T13:44:35.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f8571a46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c1538a0 0x7f856c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:35.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f8571a46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f856c0cf160 0x7f856c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:35.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f8571a46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c0cb800 con 0x7f856c0ca0e0 2023-10-30T13:44:35.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f8571a46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c15dc30 con 0x7f856c0cf160 2023-10-30T13:44:35.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f8571a46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0cb330 con 0x7f856c1538a0 2023-10-30T13:44:35.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f856affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c1538a0 0x7f856c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:35.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f856affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c1538a0 0x7f856c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:40482/0 (socket says 172.21.15.148:40482) 2023-10-30T13:44:35.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f856affd640 1 -- 172.21.15.148:0/2321685607 learned_addr learned my addr 172.21.15.148:0/2321685607 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:35.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f856b7fe640 1 --2- 172.21.15.148:0/2321685607 >> v2:172.21.15.150:3300/0 conn(0x7f856c0cf160 0x7f856c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:35.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f856affd640 1 -- 172.21.15.148:0/2321685607 >> v2:172.21.15.150:3300/0 conn(0x7f856c0cf160 msgr2=0x7f856c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:35.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f856affd640 1 --2- 172.21.15.148:0/2321685607 >> v2:172.21.15.150:3300/0 conn(0x7f856c0cf160 0x7f856c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f856affd640 1 -- 172.21.15.148:0/2321685607 >> v1:172.21.15.150:6789/0 conn(0x7f856c0ca0e0 legacy=0x7f856c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:35.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.677+0000 7f856affd640 1 -- 172.21.15.148:0/2321685607 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0cad70 con 0x7f856c1538a0 2023-10-30T13:44:35.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f856affd640 1 --2- 172.21.15.148:0/2321685607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c1538a0 0x7f856c15d6f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f855c009870 tx=0x7f855c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=355dd75e7256989 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:35.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f856a7fc640 1 -- 172.21.15.148:0/2321685607 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c0a8a30 con 0x7f856c1538a0 2023-10-30T13:44:35.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f856a7fc640 1 -- 172.21.15.148:0/2321685607 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855c0a8b90 con 0x7f856c1538a0 2023-10-30T13:44:35.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f856a7fc640 1 -- 172.21.15.148:0/2321685607 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c0b0440 con 0x7f856c1538a0 2023-10-30T13:44:35.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f8571a46640 1 -- 172.21.15.148:0/2321685607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c1538a0 msgr2=0x7f856c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:35.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f8571a46640 1 --2- 172.21.15.148:0/2321685607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c1538a0 0x7f856c15d6f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f855c009870 tx=0x7f855c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:44:35.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f8571a46640 1 -- 172.21.15.148:0/2321685607 shutdown_connections 2023-10-30T13:44:35.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f8571a46640 1 --2- 172.21.15.148:0/2321685607 >> v2:172.21.15.150:3300/0 conn(0x7f856c0cf160 0x7f856c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f8571a46640 1 --2- 172.21.15.148:0/2321685607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c1538a0 0x7f856c15d6f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f8571a46640 1 -- 172.21.15.148:0/2321685607 >> 172.21.15.148:0/2321685607 conn(0x7f856c06bfe0 msgr2=0x7f856c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f8571a46640 1 -- 172.21.15.148:0/2321685607 shutdown_connections 2023-10-30T13:44:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.681+0000 7f8571a46640 1 -- 172.21.15.148:0/2321685607 wait complete. 2023-10-30T13:44:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 Processor -- start 2023-10-30T13:44:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 -- start start 2023-10-30T13:44:35.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c0ca0e0 0x7f856c1f3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:35.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f856c0cf160 0x7f856c1e5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:35.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f856c1538a0 0x7f856c1efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:35.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c158d00 con 0x7f856c0cf160 2023-10-30T13:44:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c158fe0 con 0x7f856c1538a0 2023-10-30T13:44:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c1592c0 con 0x7f856c0ca0e0 2023-10-30T13:44:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8563fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f856c0cf160 0x7f856c1e5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f856affd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c0ca0e0 0x7f856c1f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8563fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f856c0cf160 0x7f856c1e5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60870/0 (socket says 172.21.15.148:60870) 2023-10-30T13:44:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8563fff640 1 -- 172.21.15.148:0/3294952417 learned_addr learned my addr 172.21.15.148:0/3294952417 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f856b7fe640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f856c1538a0 0x7f856c1efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8563fff640 1 -- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c0ca0e0 msgr2=0x7f856c1f3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8563fff640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c0ca0e0 0x7f856c1f3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8563fff640 1 -- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f856c1538a0 msgr2=0x7f856c1efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:35.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8563fff640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f856c1538a0 0x7f856c1efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8563fff640 1 -- 172.21.15.148:0/3294952417 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855c0094d0 con 0x7f856c0cf160 2023-10-30T13:44:35.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8563fff640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f856c0cf160 0x7f856c1e5f80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f855400c500 tx=0x7f855400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:35.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8568ff9640 1 -- 172.21.15.148:0/3294952417 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554005220 con 0x7f856c0cf160 2023-10-30T13:44:35.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8568ff9640 1 -- 172.21.15.148:0/3294952417 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8554005380 con 0x7f856c0cf160 2023-10-30T13:44:35.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8568ff9640 1 -- 172.21.15.148:0/3294952417 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554007270 con 0x7f856c0cf160 2023-10-30T13:44:35.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c1ef5a0 con 0x7f856c0cf160 2023-10-30T13:44:35.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c158a90 con 0x7f856c0cf160 2023-10-30T13:44:35.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.685+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c0b2a50 con 0x7f856c0cf160 2023-10-30T13:44:35.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.689+0000 7f8568ff9640 1 -- 172.21.15.148:0/3294952417 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85540073d0 con 0x7f856c0cf160 2023-10-30T13:44:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.689+0000 7f8568ff9640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f854c0664a0 0x7f854c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.689+0000 7f8568ff9640 1 -- 172.21.15.148:0/3294952417 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f8554083d60 con 0x7f856c0cf160 2023-10-30T13:44:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.689+0000 7f856affd640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f854c0664a0 0x7f854c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.689+0000 7f856affd640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f854c0664a0 0x7f854c068960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f855c0b2000 tx=0x7f855c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.693+0000 7f8568ff9640 1 -- 172.21.15.148:0/3294952417 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8554050bb0 con 0x7f856c0cf160 2023-10-30T13:44:35.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f856c1efac0 con 0x7f856c0cf160 2023-10-30T13:44:35.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8568ff9640 1 -- 172.21.15.148:0/3294952417 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f856c1efac0 con 0x7f856c0cf160 2023-10-30T13:44:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:35.875 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:34.947993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 126 2023-10-30T13:44:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f854c0664a0 msgr2=0x7f854c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f854c0664a0 0x7f854c068960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f855c0b2000 tx=0x7f855c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:44:35.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f856c0cf160 msgr2=0x7f856c1e5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:35.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f856c0cf160 0x7f856c1e5f80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f855400c500 tx=0x7f855400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 shutdown_connections 2023-10-30T13:44:35.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f854c0664a0 0x7f854c068960 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f856c0cf160 0x7f856c1e5f80 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f856c1538a0 0x7f856c1efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 --2- 172.21.15.148:0/3294952417 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f856c0ca0e0 0x7f856c1f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:35.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 >> 172.21.15.148:0/3294952417 conn(0x7f856c06bfe0 msgr2=0x7f856c14dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:35.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 shutdown_connections 2023-10-30T13:44:35.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:35.877+0000 7f8571a46640 1 -- 172.21.15.148:0/3294952417 wait complete. 2023-10-30T13:44:35.896 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:44:35.896 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:44:35.896 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:36.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f73794640 1 Processor -- start 2023-10-30T13:44:36.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f73794640 1 -- start start 2023-10-30T13:44:36.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f73794640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c0cf160 0x7f8f6c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f73794640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c0cb800 con 0x7f8f6c15d6f0 2023-10-30T13:44:36.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f73794640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c0cb330 con 0x7f8f6c1538a0 2023-10-30T13:44:36.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f73794640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0cad70 con 0x7f8f6c0cf160 2023-10-30T13:44:36.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f70d2f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8f6c15d6f0 0x7f8f6c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36580/0 (socket says 172.21.15.148:36580) 2023-10-30T13:44:36.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f70d2f640 1 -- 172.21.15.148:0/3369414877 learned_addr learned my addr 172.21.15.148:0/3369414877 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:36.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f71d31640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c0cf160 0x7f8f6c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046550717 0 0) 0x7f8f6c0cb800 con 0x7f8f6c15d6f0 2023-10-30T13:44:36.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f58003530 con 0x7f8f6c15d6f0 2023-10-30T13:44:36.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452786900 0 0) 0x7f8f6c0cb330 con 0x7f8f6c1538a0 2023-10-30T13:44:36.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c0cb800 con 0x7f8f6c1538a0 2023-10-30T13:44:36.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2308649096 0 0) 0x7f8f58003530 con 0x7f8f6c15d6f0 2023-10-30T13:44:36.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c0cf160 msgr2=0x7f8f6c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:36.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c0cf160 0x7f8f6c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 >> v1:172.21.15.150:6789/0 conn(0x7f8f6c1538a0 legacy=0x7f8f6c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c0cb050 con 0x7f8f6c15d6f0 2023-10-30T13:44:36.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f8f5c001680 con 0x7f8f6c15d6f0 2023-10-30T13:44:36.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 >> v1:172.21.15.148:6789/0 conn(0x7f8f6c15d6f0 legacy=0x7f8f6c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c0cf160 0x7f8f58003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f580040a0 0x7f8f580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.037+0000 7f8f63fff640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f58004d10 0x7f8f58005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0cad70 con 0x7f8f58004d10 2023-10-30T13:44:36.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5800ba70 con 0x7f8f580040a0 2023-10-30T13:44:36.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5800bc30 con 0x7f8f6c0cf160 2023-10-30T13:44:36.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f71d31640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f58004d10 0x7f8f58005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f70d2f640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f580040a0 0x7f8f580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f71530640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c0cf160 0x7f8f58003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f71d31640 1 -- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c0cf160 msgr2=0x7f8f58003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:36.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f71d31640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c0cf160 0x7f8f58003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f71d31640 1 -- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f580040a0 msgr2=0x7f8f580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:36.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f71d31640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f580040a0 0x7f8f580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f71d31640 1 -- 172.21.15.148:0/3369414877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f58003530 con 0x7f8f58004d10 2023-10-30T13:44:36.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f71d31640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f58004d10 0x7f8f58005190 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f8f64009fc0 tx=0x7f8f6401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d3a93a28caa323 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:36.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f640a5c70 con 0x7f8f58004d10 2023-10-30T13:44:36.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6400f040 con 0x7f8f58004d10 2023-10-30T13:44:36.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.041+0000 7f8f63fff640 1 -- 172.21.15.148:0/3369414877 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f640ad440 con 0x7f8f58004d10 2023-10-30T13:44:36.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 -- 172.21.15.148:0/3369414877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f58004d10 msgr2=0x7f8f58005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 --2- 172.21.15.148:0/3369414877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f58004d10 0x7f8f58005190 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f8f64009fc0 tx=0x7f8f6401cc60 comp rx=0 tx=0).stop 2023-10-30T13:44:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f71d31640 1 -- 172.21.15.148:0/3369414877 reap_dead start 2023-10-30T13:44:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 -- 172.21.15.148:0/3369414877 shutdown_connections 2023-10-30T13:44:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 -- 172.21.15.148:0/3369414877 >> 172.21.15.148:0/3369414877 conn(0x7f8f6c06bfe0 msgr2=0x7f8f6c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 -- 172.21.15.148:0/3369414877 shutdown_connections 2023-10-30T13:44:36.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 -- 172.21.15.148:0/3369414877 wait complete. 2023-10-30T13:44:36.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 Processor -- start 2023-10-30T13:44:36.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 -- start start 2023-10-30T13:44:36.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f6c0c6730 0x7f8f6c1eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0c6b10 0x7f8f6c156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14c790 0x7f8f6c14cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c1547f0 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c14d150 con 0x7f8f6c0c6730 2023-10-30T13:44:36.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f73794640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c156620 con 0x7f8f6c14c790 2023-10-30T13:44:36.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f70d2f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0c6b10 0x7f8f6c156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f70d2f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0c6b10 0x7f8f6c156ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60880/0 (socket says 172.21.15.148:60880) 2023-10-30T13:44:36.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f70d2f640 1 -- 172.21.15.148:0/3697756320 learned_addr learned my addr 172.21.15.148:0/3697756320 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:36.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f71530640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f6c0c6730 0x7f8f6c1eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.045+0000 7f8f71d31640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14c790 0x7f8f6c14cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f70d2f640 1 -- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14c790 msgr2=0x7f8f6c14cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f70d2f640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14c790 0x7f8f6c14cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f70d2f640 1 -- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f6c0c6730 msgr2=0x7f8f6c1eef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f70d2f640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f6c0c6730 0x7f8f6c1eef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f70d2f640 1 -- 172.21.15.148:0/3697756320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c154020 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f70d2f640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0c6b10 0x7f8f6c156ab0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c007370 tx=0x7f8f5c004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:36.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f61ffb640 1 -- 172.21.15.148:0/3697756320 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c00f960 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f61ffb640 1 -- 172.21.15.148:0/3697756320 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c00fb00 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c155730 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f61ffb640 1 -- 172.21.15.148:0/3697756320 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c00feb0 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c155cf0 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c04e170 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f61ffb640 1 -- 172.21.15.148:0/3697756320 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f5c01d460 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f61ffb640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f500664d0 0x7f8f50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:36.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f61ffb640 1 -- 172.21.15.148:0/3697756320 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f8f5c0843e0 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.049+0000 7f8f71530640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f500664d0 0x7f8f50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.053+0000 7f8f71530640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f500664d0 0x7f8f50068990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8f68007af0 tx=0x7f8f68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:36.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.053+0000 7f8f61ffb640 1 -- 172.21.15.148:0/3697756320 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f5c0512b0 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.237+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f6c156760 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.237+0000 7f8f61ffb640 1 -- 172.21.15.148:0/3697756320 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5719 (secure 0 0 0) 0x7f8f5c0548d0 con 0x7f8f6c0c6b10 2023-10-30T13:44:36.237 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:36.237 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:34.947993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:36.238 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 126 2023-10-30T13:44:36.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f500664d0 msgr2=0x7f8f50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f500664d0 0x7f8f50068990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8f68007af0 tx=0x7f8f68009040 comp rx=0 tx=0).stop 2023-10-30T13:44:36.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0c6b10 msgr2=0x7f8f6c156ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:36.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0c6b10 0x7f8f6c156ab0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c007370 tx=0x7f8f5c004f90 comp rx=0 tx=0).stop 2023-10-30T13:44:36.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 shutdown_connections 2023-10-30T13:44:36.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f500664d0 0x7f8f50068990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0c6b10 0x7f8f6c156ab0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14c790 0x7f8f6c14cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 --2- 172.21.15.148:0/3697756320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f6c0c6730 0x7f8f6c1eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:36.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 >> 172.21.15.148:0/3697756320 conn(0x7f8f6c06bfe0 msgr2=0x7f8f6c0c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:36.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 shutdown_connections 2023-10-30T13:44:36.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:36.241+0000 7f8f73794640 1 -- 172.21.15.148:0/3697756320 wait complete. 2023-10-30T13:44:36.259 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 126, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 126, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7464": { "addr": "172.21.15.148:6837/3703944646", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3703944646, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3703944646, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7464, "incarnation": 126, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:44:34.947993+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7464 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:44:38.260 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:38.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.409+0000 7fca01f65640 1 Processor -- start 2023-10-30T13:44:38.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.409+0000 7fca01f65640 1 -- start start 2023-10-30T13:44:38.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.409+0000 7fca01f65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc9fc070a00 0x7fc9fc070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.409+0000 7fca01f65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc0713b0 con 0x7fc9fc069e90 2023-10-30T13:44:38.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.409+0000 7fca01f65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9fc153d80 con 0x7fc9fc1539a0 2023-10-30T13:44:38.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.409+0000 7fca01f65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9fc06fbb0 con 0x7fc9fc070a00 2023-10-30T13:44:38.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.409+0000 7fc9fbfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc9fc069e90 0x7fc9fc06a2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36588/0 (socket says 172.21.15.148:36588) 2023-10-30T13:44:38.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.409+0000 7fc9fbfff640 1 -- 172.21.15.148:0/1929093043 learned_addr learned my addr 172.21.15.148:0/1929093043 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:38.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.409+0000 7fc9f3fff640 1 --2- 172.21.15.148:0/1929093043 >> v2:172.21.15.150:3300/0 conn(0x7fc9fc070a00 0x7fc9fc070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9faffd640 1 -- 172.21.15.148:0/1929093043 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318544788 0 0) 0x7fc9fc0713b0 con 0x7fc9fc069e90 2023-10-30T13:44:38.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9faffd640 1 -- 172.21.15.148:0/1929093043 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e8003530 con 0x7fc9fc069e90 2023-10-30T13:44:38.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9f3fff640 1 -- 172.21.15.148:0/1929093043 >> v1:172.21.15.150:6789/0 conn(0x7fc9fc1539a0 legacy=0x7fc9fc15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9faffd640 1 -- 172.21.15.148:0/1929093043 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165389855 0 0) 0x7fc9e8003530 con 0x7fc9fc069e90 2023-10-30T13:44:38.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9f3fff640 1 -- 172.21.15.148:0/1929093043 >> v1:172.21.15.148:6789/0 conn(0x7fc9fc069e90 legacy=0x7fc9fc06a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9f3fff640 1 -- 172.21.15.148:0/1929093043 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc0c5c50 con 0x7fc9fc070a00 2023-10-30T13:44:38.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9f3fff640 1 --2- 172.21.15.148:0/1929093043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9fc070a00 0x7fc9fc070e70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40097c0 tx=0x7fc9e401c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a5f15f1c7b78aa8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:38.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9faffd640 1 -- 172.21.15.148:0/1929093043 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4001a00 con 0x7fc9fc070a00 2023-10-30T13:44:38.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9faffd640 1 -- 172.21.15.148:0/1929093043 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e40a7050 con 0x7fc9fc070a00 2023-10-30T13:44:38.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.413+0000 7fc9faffd640 1 -- 172.21.15.148:0/1929093043 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e40ab600 con 0x7fc9fc070a00 2023-10-30T13:44:38.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 -- 172.21.15.148:0/1929093043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9fc070a00 msgr2=0x7fc9fc070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 --2- 172.21.15.148:0/1929093043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9fc070a00 0x7fc9fc070e70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40097c0 tx=0x7fc9e401c2e0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 -- 172.21.15.148:0/1929093043 shutdown_connections 2023-10-30T13:44:38.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 --2- 172.21.15.148:0/1929093043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9fc070a00 0x7fc9fc070e70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 -- 172.21.15.148:0/1929093043 >> 172.21.15.148:0/1929093043 conn(0x7fc9fc0c7910 msgr2=0x7fc9fc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:38.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 -- 172.21.15.148:0/1929093043 shutdown_connections 2023-10-30T13:44:38.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 -- 172.21.15.148:0/1929093043 wait complete. 2023-10-30T13:44:38.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 Processor -- start 2023-10-30T13:44:38.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 -- start start 2023-10-30T13:44:38.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9fc069e90 0x7fc9fc1f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9fc070a00 0x7fc9fc14e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9fc1539a0 0x7fc9fc0c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc148d50 con 0x7fc9fc070a00 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1487f0 con 0x7fc9fc1539a0 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fca01f65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc148a70 con 0x7fc9fc069e90 2023-10-30T13:44:38.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fc9f3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9fc070a00 0x7fc9fc14e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fc9f3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9fc070a00 0x7fc9fc14e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60886/0 (socket says 172.21.15.148:60886) 2023-10-30T13:44:38.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fc9f3fff640 1 -- 172.21.15.148:0/71717637 learned_addr learned my addr 172.21.15.148:0/71717637 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:38.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fc9fbfff640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9fc1539a0 0x7fc9fc0c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.417+0000 7fc9fb7fe640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9fc069e90 0x7fc9fc1f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f3fff640 1 -- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9fc069e90 msgr2=0x7fc9fc1f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f3fff640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9fc069e90 0x7fc9fc1f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f3fff640 1 -- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9fc1539a0 msgr2=0x7fc9fc0c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f3fff640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9fc1539a0 0x7fc9fc0c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f3fff640 1 -- 172.21.15.148:0/71717637 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e4009420 con 0x7fc9fc070a00 2023-10-30T13:44:38.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f3fff640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9fc070a00 0x7fc9fc14e040 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4000c10 tx=0x7fc9e4019080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:38.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f8ff9640 1 -- 172.21.15.148:0/71717637 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4003b30 con 0x7fc9fc070a00 2023-10-30T13:44:38.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f8ff9640 1 -- 172.21.15.148:0/71717637 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e4003c90 con 0x7fc9fc070a00 2023-10-30T13:44:38.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f8ff9640 1 -- 172.21.15.148:0/71717637 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e40ab400 con 0x7fc9fc070a00 2023-10-30T13:44:38.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9fc1f5de0 con 0x7fc9fc070a00 2023-10-30T13:44:38.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9fc1f63a0 con 0x7fc9fc070a00 2023-10-30T13:44:38.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9fc1f5c40 con 0x7fc9fc070a00 2023-10-30T13:44:38.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f8ff9640 1 -- 172.21.15.148:0/71717637 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9e40b2040 con 0x7fc9fc070a00 2023-10-30T13:44:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f8ff9640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9dc0664d0 0x7fc9dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9f8ff9640 1 -- 172.21.15.148:0/71717637 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fc9e411e4b0 con 0x7fc9fc070a00 2023-10-30T13:44:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.421+0000 7fc9fb7fe640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9dc0664d0 0x7fc9dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:38.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.425+0000 7fc9fb7fe640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9dc0664d0 0x7fc9dc068990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec000f70 tx=0x7fc9ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:38.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.425+0000 7fc9f8ff9640 1 -- 172.21.15.148:0/71717637 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9fc1f5c40 con 0x7fc9fc070a00 2023-10-30T13:44:38.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.605+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9fc1f5c40 con 0x7fc9fc070a00 2023-10-30T13:44:38.604 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.605+0000 7fc9f8ff9640 1 -- 172.21.15.148:0/71717637 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5719 (secure 0 0 0) 0x7fc9fc1f5c40 con 0x7fc9fc070a00 2023-10-30T13:44:38.604 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:38.605 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:34.947993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:38.605 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 126 2023-10-30T13:44:38.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9dc0664d0 msgr2=0x7fc9dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9dc0664d0 0x7fc9dc068990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec000f70 tx=0x7fc9ec009040 comp rx=0 tx=0).stop 2023-10-30T13:44:38.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9fc070a00 msgr2=0x7fc9fc14e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:38.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9fc070a00 0x7fc9fc14e040 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4000c10 tx=0x7fc9e4019080 comp rx=0 tx=0).stop 2023-10-30T13:44:38.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 shutdown_connections 2023-10-30T13:44:38.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9dc0664d0 0x7fc9dc068990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9fc070a00 0x7fc9fc14e040 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9fc1539a0 0x7fc9fc0c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 --2- 172.21.15.148:0/71717637 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc9fc069e90 0x7fc9fc1f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:38.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 >> 172.21.15.148:0/71717637 conn(0x7fc9fc0c7910 msgr2=0x7fc9fc1507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:38.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 shutdown_connections 2023-10-30T13:44:38.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:38.609+0000 7fca01f65640 1 -- 172.21.15.148:0/71717637 wait complete. 2023-10-30T13:44:38.626 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:40.627 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:40.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.765+0000 7fe3bb46b640 1 Processor -- start 2023-10-30T13:44:40.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.765+0000 7fe3bb46b640 1 -- start start 2023-10-30T13:44:40.766 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3bb46b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe3b406f7a0 0x7fe3b406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3bb46b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4071110 0x7fe3b4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3bb46b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b4066cf0 con 0x7fe3b40c6bd0 2023-10-30T13:44:40.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3bb46b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b40700c0 con 0x7fe3b406f7a0 2023-10-30T13:44:40.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3bb46b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b40c70d0 con 0x7fe3b4071110 2023-10-30T13:44:40.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3b8a06640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe3b40c6bd0 0x7fe3b4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36598/0 (socket says 172.21.15.148:36598) 2023-10-30T13:44:40.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3b8a06640 1 -- 172.21.15.148:0/4069733759 learned_addr learned my addr 172.21.15.148:0/4069733759 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:40.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3b9a08640 1 --2- 172.21.15.148:0/4069733759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4071110 0x7fe3b4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3b9207640 1 --2- 172.21.15.148:0/4069733759 >> v2:172.21.15.150:3300/0 conn(0x7fe3b406f7a0 0x7fe3b406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3abfff640 1 -- 172.21.15.148:0/4069733759 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248564084 0 0) 0x7fe3b4066cf0 con 0x7fe3b40c6bd0 2023-10-30T13:44:40.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3abfff640 1 -- 172.21.15.148:0/4069733759 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a0003530 con 0x7fe3b40c6bd0 2023-10-30T13:44:40.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3abfff640 1 -- 172.21.15.148:0/4069733759 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570251947 0 0) 0x7fe3a0003530 con 0x7fe3b40c6bd0 2023-10-30T13:44:40.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3b9a08640 1 -- 172.21.15.148:0/4069733759 >> v2:172.21.15.150:3300/0 conn(0x7fe3b406f7a0 msgr2=0x7fe3b406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:40.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3b9a08640 1 --2- 172.21.15.148:0/4069733759 >> v2:172.21.15.150:3300/0 conn(0x7fe3b406f7a0 0x7fe3b406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3b9a08640 1 -- 172.21.15.148:0/4069733759 >> v1:172.21.15.148:6789/0 conn(0x7fe3b40c6bd0 legacy=0x7fe3b4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.769+0000 7fe3b9a08640 1 -- 172.21.15.148:0/4069733759 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b40c7390 con 0x7fe3b4071110 2023-10-30T13:44:40.771 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3b9a08640 1 --2- 172.21.15.148:0/4069733759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4071110 0x7fe3b4158110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac009d90 tx=0x7fe3ac01f060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1996930172f5f5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:40.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3abfff640 1 -- 172.21.15.148:0/4069733759 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4001680 con 0x7fe3b4071110 2023-10-30T13:44:40.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3abfff640 1 -- 172.21.15.148:0/4069733759 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ac0a8bb0 con 0x7fe3b4071110 2023-10-30T13:44:40.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3abfff640 1 -- 172.21.15.148:0/4069733759 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac0b0440 con 0x7fe3b4071110 2023-10-30T13:44:40.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4069733759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4071110 msgr2=0x7fe3b4158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.772 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3bb46b640 1 --2- 172.21.15.148:0/4069733759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4071110 0x7fe3b4158110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac009d90 tx=0x7fe3ac01f060 comp rx=0 tx=0).stop 2023-10-30T13:44:40.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4069733759 shutdown_connections 2023-10-30T13:44:40.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3bb46b640 1 --2- 172.21.15.148:0/4069733759 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b4071110 0x7fe3b4158110 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3bb46b640 1 --2- 172.21.15.148:0/4069733759 >> v2:172.21.15.150:3300/0 conn(0x7fe3b406f7a0 0x7fe3b406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4069733759 >> 172.21.15.148:0/4069733759 conn(0x7fe3b40c7910 msgr2=0x7fe3b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:40.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4069733759 shutdown_connections 2023-10-30T13:44:40.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.773+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4069733759 wait complete. 2023-10-30T13:44:40.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 Processor -- start 2023-10-30T13:44:40.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 -- start start 2023-10-30T13:44:40.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b406f7a0 0x7fe3b41541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3b4071110 0x7fe3b4153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b40c6bd0 0x7fe3b40c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b4157320 con 0x7fe3b4071110 2023-10-30T13:44:40.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b414f830 con 0x7fe3b406f7a0 2023-10-30T13:44:40.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b4154e60 con 0x7fe3b40c6bd0 2023-10-30T13:44:40.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b8a06640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3b4071110 0x7fe3b4153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b8a06640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3b4071110 0x7fe3b4153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60890/0 (socket says 172.21.15.148:60890) 2023-10-30T13:44:40.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b8a06640 1 -- 172.21.15.148:0/4092116350 learned_addr learned my addr 172.21.15.148:0/4092116350 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:40.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b9a08640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b40c6bd0 0x7fe3b40c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b9207640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b406f7a0 0x7fe3b41541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b8a06640 1 -- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b40c6bd0 msgr2=0x7fe3b40c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b8a06640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b40c6bd0 0x7fe3b40c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b8a06640 1 -- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b406f7a0 msgr2=0x7fe3b41541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b8a06640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b406f7a0 0x7fe3b41541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b8a06640 1 -- 172.21.15.148:0/4092116350 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac009a70 con 0x7fe3b4071110 2023-10-30T13:44:40.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b9a08640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b40c6bd0 0x7fe3b40c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:40.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3b8a06640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3b4071110 0x7fe3b4153090 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400c750 tx=0x7fe3a400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:40.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3a9ffb640 1 -- 172.21.15.148:0/4092116350 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4005af0 con 0x7fe3b4071110 2023-10-30T13:44:40.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3a9ffb640 1 -- 172.21.15.148:0/4092116350 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a4005c50 con 0x7fe3b4071110 2023-10-30T13:44:40.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3a9ffb640 1 -- 172.21.15.148:0/4092116350 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4007070 con 0x7fe3b4071110 2023-10-30T13:44:40.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b414f570 con 0x7fe3b4071110 2023-10-30T13:44:40.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b4154af0 con 0x7fe3b4071110 2023-10-30T13:44:40.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.777+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b4154950 con 0x7fe3b4071110 2023-10-30T13:44:40.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.781+0000 7fe3a9ffb640 1 -- 172.21.15.148:0/4092116350 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe3a40071d0 con 0x7fe3b4071110 2023-10-30T13:44:40.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.781+0000 7fe3a9ffb640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe3980664a0 0x7fe398068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:40.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.781+0000 7fe3a9ffb640 1 -- 172.21.15.148:0/4092116350 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fe3a4084290 con 0x7fe3b4071110 2023-10-30T13:44:40.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.781+0000 7fe3b9207640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe3980664a0 0x7fe398068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:40.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.781+0000 7fe3b9207640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe3980664a0 0x7fe398068960 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009a50 tx=0x7fe3b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:40.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.785+0000 7fe3a9ffb640 1 -- 172.21.15.148:0/4092116350 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe3b4154950 con 0x7fe3b4071110 2023-10-30T13:44:40.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3b4158630 con 0x7fe3b4071110 2023-10-30T13:44:40.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3a9ffb640 1 -- 172.21.15.148:0/4092116350 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5719 (secure 0 0 0) 0x7fe3b4158630 con 0x7fe3b4071110 2023-10-30T13:44:40.963 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:40.964 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:34.947993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:40.964 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 126 2023-10-30T13:44:40.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe3980664a0 msgr2=0x7fe398068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe3980664a0 0x7fe398068960 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009a50 tx=0x7fe3b0008040 comp rx=0 tx=0).stop 2023-10-30T13:44:40.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3b4071110 msgr2=0x7fe3b4153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3b4071110 0x7fe3b4153090 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400c750 tx=0x7fe3a400cc10 comp rx=0 tx=0).stop 2023-10-30T13:44:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 shutdown_connections 2023-10-30T13:44:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe3980664a0 0x7fe398068960 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe3b4071110 0x7fe3b4153090 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe3b40c6bd0 0x7fe3b40c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 --2- 172.21.15.148:0/4092116350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3b406f7a0 0x7fe3b41541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:40.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 >> 172.21.15.148:0/4092116350 conn(0x7fe3b40c7910 msgr2=0x7fe3b40c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:40.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 shutdown_connections 2023-10-30T13:44:40.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:40.965+0000 7fe3bb46b640 1 -- 172.21.15.148:0/4092116350 wait complete. 2023-10-30T13:44:40.984 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:42.985 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:43.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.137+0000 7faddc13e640 1 Processor -- start 2023-10-30T13:44:43.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.137+0000 7faddc13e640 1 -- start start 2023-10-30T13:44:43.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7faddc13e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fadd41580e0 0x7fadd4070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7faddc13e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd40c5810 con 0x7fadd415e700 2023-10-30T13:44:43.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7faddc13e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd40c6990 con 0x7fadd40c6540 2023-10-30T13:44:43.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7faddc13e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd406acb0 con 0x7fadd41580e0 2023-10-30T13:44:43.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadda6db640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fadd415e700 0x7fadd406a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55232/0 (socket says 172.21.15.148:55232) 2023-10-30T13:44:43.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadda6db640 1 -- 172.21.15.148:0/2036015577 learned_addr learned my addr 172.21.15.148:0/2036015577 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:43.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd9eda640 1 --2- 172.21.15.148:0/2036015577 >> v2:172.21.15.150:3300/0 conn(0x7fadd41580e0 0x7fadd4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665376486 0 0) 0x7fadd40c5810 con 0x7fadd415e700 2023-10-30T13:44:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc0003530 con 0x7fadd415e700 2023-10-30T13:44:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43246680 0 0) 0x7fadc0003530 con 0x7fadd415e700 2023-10-30T13:44:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 >> v2:172.21.15.150:3300/0 conn(0x7fadd41580e0 msgr2=0x7fadd4070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 --2- 172.21.15.148:0/2036015577 >> v2:172.21.15.150:3300/0 conn(0x7fadd41580e0 0x7fadd4070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 >> v1:172.21.15.150:6789/0 conn(0x7fadd40c6540 legacy=0x7fadd4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd415eb90 con 0x7fadd415e700 2023-10-30T13:44:43.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fadcc001a00 con 0x7fadd415e700 2023-10-30T13:44:43.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 >> v1:172.21.15.148:6789/0 conn(0x7fadd415e700 legacy=0x7fadd406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc0003980 0x7fadc0003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc0004380 0x7fadc0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadc0005050 0x7fadc000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc000bf80 con 0x7fadc0004380 2023-10-30T13:44:43.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc000c0f0 con 0x7fadc0003980 2023-10-30T13:44:43.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc000c2b0 con 0x7fadc0005050 2023-10-30T13:44:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd96d9640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc0004380 0x7fadc0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd9eda640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc0003980 0x7fadc0003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadda6db640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadc0005050 0x7fadc000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd96d9640 1 -- 172.21.15.148:0/2036015577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadc0005050 msgr2=0x7fadc000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd96d9640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadc0005050 0x7fadc000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd96d9640 1 -- 172.21.15.148:0/2036015577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc0003980 msgr2=0x7fadc0003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:43.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd96d9640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc0003980 0x7fadc0003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadd96d9640 1 -- 172.21.15.148:0/2036015577 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc0003530 con 0x7fadc0004380 2023-10-30T13:44:43.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.141+0000 7fadda6db640 1 -- 172.21.15.148:0/2036015577 reap_dead start 2023-10-30T13:44:43.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7fadd96d9640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc0004380 0x7fadc0004850 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fadc4001860 tx=0x7fadc401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c59ef9b179ce8b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:43.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc40a8850 con 0x7fadc0004380 2023-10-30T13:44:43.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc40a89b0 con 0x7fadc0004380 2023-10-30T13:44:43.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7fadd8ed8640 1 -- 172.21.15.148:0/2036015577 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc40a8de0 con 0x7fadc0004380 2023-10-30T13:44:43.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7faddc13e640 1 -- 172.21.15.148:0/2036015577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc0004380 msgr2=0x7fadc0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7faddc13e640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc0004380 0x7fadc0004850 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fadc4001860 tx=0x7fadc401ec30 comp rx=0 tx=0).stop 2023-10-30T13:44:43.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7faddc13e640 1 -- 172.21.15.148:0/2036015577 shutdown_connections 2023-10-30T13:44:43.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7faddc13e640 1 --2- 172.21.15.148:0/2036015577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadc0004380 0x7fadc0004850 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7faddc13e640 1 -- 172.21.15.148:0/2036015577 >> 172.21.15.148:0/2036015577 conn(0x7fadd40c7910 msgr2=0x7fadd40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:43.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7faddc13e640 1 -- 172.21.15.148:0/2036015577 shutdown_connections 2023-10-30T13:44:43.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.145+0000 7faddc13e640 1 -- 172.21.15.148:0/2036015577 wait complete. 2023-10-30T13:44:43.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 Processor -- start 2023-10-30T13:44:43.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 -- start start 2023-10-30T13:44:43.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadc0004380 0x7fadd41f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadd41609a0 0x7fadd415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd415cfd0 0x7fadd415f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd41f6590 con 0x7fadd41609a0 2023-10-30T13:44:43.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd41f74b0 con 0x7fadd415cfd0 2023-10-30T13:44:43.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40c4780 con 0x7fadc0004380 2023-10-30T13:44:43.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd96d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadd41609a0 0x7fadd415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd96d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadd41609a0 0x7fadd415ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49634/0 (socket says 172.21.15.148:49634) 2023-10-30T13:44:43.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd96d9640 1 -- 172.21.15.148:0/2340392715 learned_addr learned my addr 172.21.15.148:0/2340392715 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:43.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd9eda640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadc0004380 0x7fadd41f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadda6db640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd415cfd0 0x7fadd415f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd96d9640 1 -- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadc0004380 msgr2=0x7fadd41f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd96d9640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadc0004380 0x7fadd41f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd96d9640 1 -- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd415cfd0 msgr2=0x7fadd415f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd96d9640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd415cfd0 0x7fadd415f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd96d9640 1 -- 172.21.15.148:0/2340392715 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd40c4a60 con 0x7fadd41609a0 2023-10-30T13:44:43.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadda6db640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd415cfd0 0x7fadd415f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:43.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadd96d9640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadd41609a0 0x7fadd415ca90 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fadc401f4d0 tx=0x7fadc0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:43.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadca7fc640 1 -- 172.21.15.148:0/2340392715 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc40069e0 con 0x7fadd41609a0 2023-10-30T13:44:43.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadca7fc640 1 -- 172.21.15.148:0/2340392715 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc4006b40 con 0x7fadd41609a0 2023-10-30T13:44:43.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7fadca7fc640 1 -- 172.21.15.148:0/2340392715 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc40a87d0 con 0x7fadd41609a0 2023-10-30T13:44:43.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd40c4d40 con 0x7fadd41609a0 2023-10-30T13:44:43.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.149+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd41f6a50 con 0x7fadd41609a0 2023-10-30T13:44:43.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.153+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd41f6be0 con 0x7fadd41609a0 2023-10-30T13:44:43.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.153+0000 7fadca7fc640 1 -- 172.21.15.148:0/2340392715 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fadc40b7040 con 0x7fadd41609a0 2023-10-30T13:44:43.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.153+0000 7fadca7fc640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadb80664a0 0x7fadb8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:43.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.153+0000 7fadca7fc640 1 -- 172.21.15.148:0/2340392715 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fadc4120ab0 con 0x7fadd41609a0 2023-10-30T13:44:43.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.153+0000 7fadd9eda640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadb80664a0 0x7fadb8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:43.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.153+0000 7fadd9eda640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadb80664a0 0x7fadb8068960 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009c60 tx=0x7fadd0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:43.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.161+0000 7fadca7fc640 1 -- 172.21.15.148:0/2340392715 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fadc40ed8f0 con 0x7fadd41609a0 2023-10-30T13:44:43.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.353+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadd40c6b80 con 0x7fadd41609a0 2023-10-30T13:44:43.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7fadca7fc640 1 -- 172.21.15.148:0/2340392715 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5719 (secure 0 0 0) 0x7fadc40f0f10 con 0x7fadd41609a0 2023-10-30T13:44:43.354 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:43.355 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:34.947993+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:43.355 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 126 2023-10-30T13:44:43.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadb80664a0 msgr2=0x7fadb8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadb80664a0 0x7fadb8068960 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009c60 tx=0x7fadd0009190 comp rx=0 tx=0).stop 2023-10-30T13:44:43.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadd41609a0 msgr2=0x7fadd415ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:43.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadd41609a0 0x7fadd415ca90 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fadc401f4d0 tx=0x7fadc0005800 comp rx=0 tx=0).stop 2023-10-30T13:44:43.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 shutdown_connections 2023-10-30T13:44:43.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fadb80664a0 0x7fadb8068960 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadd41609a0 0x7fadd415ca90 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd415cfd0 0x7fadd415f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 --2- 172.21.15.148:0/2340392715 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadc0004380 0x7fadd41f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:43.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 >> 172.21.15.148:0/2340392715 conn(0x7fadd40c7910 msgr2=0x7fadd414ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:43.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 shutdown_connections 2023-10-30T13:44:43.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:43.357+0000 7faddc13e640 1 -- 172.21.15.148:0/2340392715 wait complete. 2023-10-30T13:44:43.377 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:45.378 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:45.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.517+0000 7f3108232640 1 Processor -- start 2023-10-30T13:44:45.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.517+0000 7f3108232640 1 -- start start 2023-10-30T13:44:45.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f3108232640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f31000c6540 0x7f310015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f3108232640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31000c5810 con 0x7f31001580e0 2023-10-30T13:44:45.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f3108232640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31000c69b0 con 0x7f3100070740 2023-10-30T13:44:45.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f3108232640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310006acb0 con 0x7f31000c6540 2023-10-30T13:44:45.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f31067cf640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f31000c6540 0x7f310015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f31067cf640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f31000c6540 0x7f310015e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49648/0 (socket says 172.21.15.148:49648) 2023-10-30T13:44:45.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f31067cf640 1 -- 172.21.15.148:0/1102219814 learned_addr learned my addr 172.21.15.148:0/1102219814 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:45.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f31067cf640 1 -- 172.21.15.148:0/1102219814 >> v1:172.21.15.150:6789/0 conn(0x7f3100070740 legacy=0x7f310006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f3104fcc640 1 -- 172.21.15.148:0/1102219814 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735223619 0 0) 0x7f31000c5810 con 0x7f31001580e0 2023-10-30T13:44:45.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f31067cf640 1 -- 172.21.15.148:0/1102219814 >> v1:172.21.15.148:6789/0 conn(0x7f31001580e0 legacy=0x7f3100069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.521+0000 7f31067cf640 1 -- 172.21.15.148:0/1102219814 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31000c6bb0 con 0x7f31000c6540 2023-10-30T13:44:45.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f31067cf640 1 --2- 172.21.15.148:0/1102219814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 0x7f310015e700 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009e50 tx=0x7f30f801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fdc9b4bb95fd95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:45.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3104fcc640 1 -- 172.21.15.148:0/1102219814 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f80a57f0 con 0x7f31000c6540 2023-10-30T13:44:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3104fcc640 1 -- 172.21.15.148:0/1102219814 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f80a5950 con 0x7f31000c6540 2023-10-30T13:44:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3104fcc640 1 -- 172.21.15.148:0/1102219814 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f80a5ce0 con 0x7f31000c6540 2023-10-30T13:44:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 -- 172.21.15.148:0/1102219814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 msgr2=0x7f310015e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 --2- 172.21.15.148:0/1102219814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 0x7f310015e700 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009e50 tx=0x7f30f801ca10 comp rx=0 tx=0).stop 2023-10-30T13:44:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 -- 172.21.15.148:0/1102219814 shutdown_connections 2023-10-30T13:44:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 --2- 172.21.15.148:0/1102219814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 0x7f310015e700 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 -- 172.21.15.148:0/1102219814 >> 172.21.15.148:0/1102219814 conn(0x7f31000c7910 msgr2=0x7f31000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:45.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 -- 172.21.15.148:0/1102219814 shutdown_connections 2023-10-30T13:44:45.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 -- 172.21.15.148:0/1102219814 wait complete. 2023-10-30T13:44:45.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 Processor -- start 2023-10-30T13:44:45.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 -- start start 2023-10-30T13:44:45.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.525+0000 7f3108232640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3100070740 0x7f3100145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f3108232640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 0x7f3100148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f3108232640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31001580e0 0x7f3100148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f3108232640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310013f4c0 con 0x7f31000c6540 2023-10-30T13:44:45.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f3108232640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31001475b0 con 0x7f31001580e0 2023-10-30T13:44:45.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f3108232640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3100147890 con 0x7f3100070740 2023-10-30T13:44:45.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31057cd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 0x7f3100148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31057cd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 0x7f3100148650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49664/0 (socket says 172.21.15.148:49664) 2023-10-30T13:44:45.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31057cd640 1 -- 172.21.15.148:0/858480025 learned_addr learned my addr 172.21.15.148:0/858480025 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:45.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31067cf640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31001580e0 0x7f3100148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f3105fce640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3100070740 0x7f3100145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31057cd640 1 -- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3100070740 msgr2=0x7f3100145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31057cd640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3100070740 0x7f3100145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31057cd640 1 -- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31001580e0 msgr2=0x7f3100148b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31057cd640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31001580e0 0x7f3100148b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31057cd640 1 -- 172.21.15.148:0/858480025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f8009b30 con 0x7f31000c6540 2023-10-30T13:44:45.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f31057cd640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 0x7f3100148650 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c500 tx=0x7f30f000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:45.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f30f67fc640 1 -- 172.21.15.148:0/858480025 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0005220 con 0x7f31000c6540 2023-10-30T13:44:45.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f30f67fc640 1 -- 172.21.15.148:0/858480025 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f0005380 con 0x7f31000c6540 2023-10-30T13:44:45.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f30f67fc640 1 -- 172.21.15.148:0/858480025 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0007270 con 0x7f31000c6540 2023-10-30T13:44:45.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3100147b70 con 0x7f31000c6540 2023-10-30T13:44:45.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3100148130 con 0x7f31000c6540 2023-10-30T13:44:45.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310004e170 con 0x7f31000c6540 2023-10-30T13:44:45.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f30f67fc640 1 -- 172.21.15.148:0/858480025 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f30f00073d0 con 0x7f31000c6540 2023-10-30T13:44:45.532 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.529+0000 7f30f67fc640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30e40664a0 0x7f30e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:45.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.533+0000 7f30f67fc640 1 -- 172.21.15.148:0/858480025 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f30f0083dd0 con 0x7f31000c6540 2023-10-30T13:44:45.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.533+0000 7f3105fce640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30e40664a0 0x7f30e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:45.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.533+0000 7f3105fce640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30e40664a0 0x7f30e4068960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f30fc000f70 tx=0x7f30fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:45.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.537+0000 7f30f67fc640 1 -- 172.21.15.148:0/858480025 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f30f0050b70 con 0x7f31000c6540 2023-10-30T13:44:45.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.713+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31000c5a30 con 0x7f31000c6540 2023-10-30T13:44:45.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.713+0000 7f30f67fc640 1 -- 172.21.15.148:0/858480025 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f30f0054190 con 0x7f31000c6540 2023-10-30T13:44:45.713 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:45.713 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:44.806238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:45.713 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 127 2023-10-30T13:44:45.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30e40664a0 msgr2=0x7f30e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30e40664a0 0x7f30e4068960 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f30fc000f70 tx=0x7f30fc009040 comp rx=0 tx=0).stop 2023-10-30T13:44:45.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 msgr2=0x7f3100148650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:45.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 0x7f3100148650 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c500 tx=0x7f30f000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 shutdown_connections 2023-10-30T13:44:45.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f30e40664a0 0x7f30e4068960 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31000c6540 0x7f3100148650 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31001580e0 0x7f3100148b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 --2- 172.21.15.148:0/858480025 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3100070740 0x7f3100145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:45.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 >> 172.21.15.148:0/858480025 conn(0x7f31000c7910 msgr2=0x7f31000c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:45.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 shutdown_connections 2023-10-30T13:44:45.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:45.717+0000 7f3108232640 1 -- 172.21.15.148:0/858480025 wait complete. 2023-10-30T13:44:45.733 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:47.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:47.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61cd780640 1 Processor -- start 2023-10-30T13:44:47.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61cd780640 1 -- start start 2023-10-30T13:44:47.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61cd780640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f61c815a410 0x7f61c806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:47.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61cd780640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c806cb80 0x7f61c806d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:47.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61cd780640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c806d550 con 0x7f61c8069e90 2023-10-30T13:44:47.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61cd780640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c80c8570 con 0x7f61c815a410 2023-10-30T13:44:47.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61cd780640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c815a850 con 0x7f61c806cb80 2023-10-30T13:44:47.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c67fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f61c815a410 0x7f61c806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:47.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c6ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f61c8069e90 0x7f61c80c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55258/0 (socket says 172.21.15.148:55258) 2023-10-30T13:44:47.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c6ffd640 1 -- 172.21.15.148:0/2625887641 learned_addr learned my addr 172.21.15.148:0/2625887641 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:47.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c77fe640 1 --2- 172.21.15.148:0/2625887641 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c806cb80 0x7f61c806d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:47.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c67fc640 1 -- 172.21.15.148:0/2625887641 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c806cb80 msgr2=0x7f61c806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:47.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c67fc640 1 --2- 172.21.15.148:0/2625887641 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c806cb80 0x7f61c806d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:47.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c67fc640 1 -- 172.21.15.148:0/2625887641 >> v1:172.21.15.148:6789/0 conn(0x7f61c8069e90 legacy=0x7f61c80c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:47.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c67fc640 1 -- 172.21.15.148:0/2625887641 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c80c6c10 con 0x7f61c815a410 2023-10-30T13:44:47.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c5ffb640 1 -- 172.21.15.148:0/2625887641 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131045474 0 0) 0x7f61c806d550 con 0x7f61c8069e90 2023-10-30T13:44:47.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.885+0000 7f61c77fe640 1 --2- 172.21.15.148:0/2625887641 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c806cb80 0x7f61c806d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:47.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61c67fc640 1 --2- 172.21.15.148:0/2625887641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c815a410 0x7f61c806a270 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0097c0 tx=0x7f61bc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b140b15b5777d9f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:47.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61c5ffb640 1 -- 172.21.15.148:0/2625887641 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0ab600 con 0x7f61c815a410 2023-10-30T13:44:47.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61c5ffb640 1 -- 172.21.15.148:0/2625887641 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc0a7050 con 0x7f61c815a410 2023-10-30T13:44:47.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61c5ffb640 1 -- 172.21.15.148:0/2625887641 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0ab980 con 0x7f61c815a410 2023-10-30T13:44:47.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61cd780640 1 -- 172.21.15.148:0/2625887641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c815a410 msgr2=0x7f61c806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:47.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61cd780640 1 --2- 172.21.15.148:0/2625887641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c815a410 0x7f61c806a270 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0097c0 tx=0x7f61bc01c480 comp rx=0 tx=0).stop 2023-10-30T13:44:47.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61cd780640 1 -- 172.21.15.148:0/2625887641 shutdown_connections 2023-10-30T13:44:47.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61cd780640 1 --2- 172.21.15.148:0/2625887641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c815a410 0x7f61c806a270 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:47.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61cd780640 1 --2- 172.21.15.148:0/2625887641 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c806cb80 0x7f61c806d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:47.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61cd780640 1 -- 172.21.15.148:0/2625887641 >> 172.21.15.148:0/2625887641 conn(0x7f61c806dfd0 msgr2=0x7f61c806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:47.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61cd780640 1 -- 172.21.15.148:0/2625887641 shutdown_connections 2023-10-30T13:44:47.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.889+0000 7f61cd780640 1 -- 172.21.15.148:0/2625887641 wait complete. 2023-10-30T13:44:47.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 Processor -- start 2023-10-30T13:44:47.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 -- start start 2023-10-30T13:44:47.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c8069e90 0x7f61c814c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:47.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c806cb80 0x7f61c81f4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:47.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61c815a410 0x7f61c8148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:47.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c814c490 con 0x7f61c806cb80 2023-10-30T13:44:47.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c815ad40 con 0x7f61c815a410 2023-10-30T13:44:47.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c81f5840 con 0x7f61c8069e90 2023-10-30T13:44:47.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c67fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c806cb80 0x7f61c81f4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:47.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c67fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c806cb80 0x7f61c81f4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49684/0 (socket says 172.21.15.148:49684) 2023-10-30T13:44:47.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c67fc640 1 -- 172.21.15.148:0/3204322226 learned_addr learned my addr 172.21.15.148:0/3204322226 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:47.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c6ffd640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c8069e90 0x7f61c814c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:47.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c77fe640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61c815a410 0x7f61c8148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:47.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c67fc640 1 -- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c8069e90 msgr2=0x7f61c814c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:47.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c67fc640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c8069e90 0x7f61c814c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:47.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c67fc640 1 -- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61c815a410 msgr2=0x7f61c8148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:47.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c67fc640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61c815a410 0x7f61c8148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:47.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c67fc640 1 -- 172.21.15.148:0/3204322226 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc009420 con 0x7f61c806cb80 2023-10-30T13:44:47.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61c67fc640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c806cb80 0x7f61c81f4ee0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f61bc01a0b0 tx=0x7f61bc019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:47.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61af7fe640 1 -- 172.21.15.148:0/3204322226 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0b4610 con 0x7f61c806cb80 2023-10-30T13:44:47.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61af7fe640 1 -- 172.21.15.148:0/3204322226 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc0abe90 con 0x7f61c806cb80 2023-10-30T13:44:47.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61af7fe640 1 -- 172.21.15.148:0/3204322226 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0b49d0 con 0x7f61c806cb80 2023-10-30T13:44:47.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c815aa60 con 0x7f61c806cb80 2023-10-30T13:44:47.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.893+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c814c120 con 0x7f61c806cb80 2023-10-30T13:44:47.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.897+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c814c600 con 0x7f61c806cb80 2023-10-30T13:44:47.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.897+0000 7f61af7fe640 1 -- 172.21.15.148:0/3204322226 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61bc0b2040 con 0x7f61c806cb80 2023-10-30T13:44:47.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.897+0000 7f61af7fe640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61b40664d0 0x7f61b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:47.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.897+0000 7f61af7fe640 1 -- 172.21.15.148:0/3204322226 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f61bc11e450 con 0x7f61c806cb80 2023-10-30T13:44:47.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.897+0000 7f61c6ffd640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61b40664d0 0x7f61b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:47.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.897+0000 7f61c6ffd640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61b40664d0 0x7f61b4068990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f61b8007fd0 tx=0x7f61b8007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:47.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:47.901+0000 7f61af7fe640 1 -- 172.21.15.148:0/3204322226 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61bc0eb270 con 0x7f61c806cb80 2023-10-30T13:44:48.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.085+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61c804e170 con 0x7f61c806cb80 2023-10-30T13:44:48.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.085+0000 7f61af7fe640 1 -- 172.21.15.148:0/3204322226 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f61bc0b0020 con 0x7f61c806cb80 2023-10-30T13:44:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:48.084 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:44.806238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:48.084 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 127 2023-10-30T13:44:48.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61b40664d0 msgr2=0x7f61b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61b40664d0 0x7f61b4068990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f61b8007fd0 tx=0x7f61b8007d60 comp rx=0 tx=0).stop 2023-10-30T13:44:48.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c806cb80 msgr2=0x7f61c81f4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:48.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c806cb80 0x7f61c81f4ee0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f61bc01a0b0 tx=0x7f61bc019220 comp rx=0 tx=0).stop 2023-10-30T13:44:48.086 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 shutdown_connections 2023-10-30T13:44:48.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61b40664d0 0x7f61b4068990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61c806cb80 0x7f61c81f4ee0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61c815a410 0x7f61c8148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 --2- 172.21.15.148:0/3204322226 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f61c8069e90 0x7f61c814c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:48.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 >> 172.21.15.148:0/3204322226 conn(0x7f61c806dfd0 msgr2=0x7f61c80c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:48.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 shutdown_connections 2023-10-30T13:44:48.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:48.089+0000 7f61cd780640 1 -- 172.21.15.148:0/3204322226 wait complete. 2023-10-30T13:44:48.105 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 127, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 127, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7464": { "addr": "172.21.15.148:6837/3703944646", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3703944646, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3703944646, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7464, "incarnation": 126, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:44:44.806238+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7464 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:44:50.106 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.252+0000 7f8f72e09640 1 Processor -- start 2023-10-30T13:44:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.252+0000 7f8f72e09640 1 -- start start 2023-10-30T13:44:50.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f72e09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8f6c0c29e0 0x7f8f6c0c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f72e09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c065e10 0x7f8f6c0c3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f72e09640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c151eb0 con 0x7f8f6c151ad0 2023-10-30T13:44:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f72e09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6c1587d0 con 0x7f8f6c0c29e0 2023-10-30T13:44:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f72e09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0c5a40 con 0x7f8f6c065e10 2023-10-30T13:44:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f70ba5640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8f6c151ad0 0x7f8f6c1580c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55270/0 (socket says 172.21.15.148:55270) 2023-10-30T13:44:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f70ba5640 1 -- 172.21.15.148:0/3357553763 learned_addr learned my addr 172.21.15.148:0/3357553763 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f63fff640 1 --2- 172.21.15.148:0/3357553763 >> v2:172.21.15.150:3300/0 conn(0x7f8f6c0c29e0 0x7f8f6c0c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f713a6640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c065e10 0x7f8f6c0c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073794484 0 0) 0x7f8f68001120 con 0x7f8f6c151ad0 2023-10-30T13:44:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f54003530 con 0x7f8f6c151ad0 2023-10-30T13:44:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4233269929 0 0) 0x7f8f54003530 con 0x7f8f6c151ad0 2023-10-30T13:44:50.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c065e10 msgr2=0x7f8f6c0c3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c065e10 0x7f8f6c0c3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 >> v2:172.21.15.150:3300/0 conn(0x7f8f6c0c29e0 msgr2=0x7f8f6c0c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 --2- 172.21.15.148:0/3357553763 >> v2:172.21.15.150:3300/0 conn(0x7f8f6c0c29e0 0x7f8f6c0c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c157900 con 0x7f8f6c151ad0 2023-10-30T13:44:50.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f8f68001810 con 0x7f8f6c151ad0 2023-10-30T13:44:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 >> v1:172.21.15.148:6789/0 conn(0x7f8f6c151ad0 legacy=0x7f8f6c1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c065e10 0x7f8f54003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54003f00 0x7f8f540047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f54004d10 0x7f8f54005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.259 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0c5a40 con 0x7f8f54004d10 2023-10-30T13:44:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5400b990 con 0x7f8f54003f00 2023-10-30T13:44:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5400bb00 con 0x7f8f6c065e10 2023-10-30T13:44:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f70ba5640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c065e10 0x7f8f54003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f713a6640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f54004d10 0x7f8f54005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f63fff640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54003f00 0x7f8f540047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f713a6640 1 -- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c065e10 msgr2=0x7f8f54003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:50.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f713a6640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c065e10 0x7f8f54003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f713a6640 1 -- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54003f00 msgr2=0x7f8f540047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:50.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f713a6640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54003f00 0x7f8f540047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.256+0000 7f8f713a6640 1 -- 172.21.15.148:0/3357553763 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54003530 con 0x7f8f54004d10 2023-10-30T13:44:50.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f713a6640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f54004d10 0x7f8f54005190 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c066260 tx=0x7f8f6401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cddd71f7d370585 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:50.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f640a5c10 con 0x7f8f54004d10 2023-10-30T13:44:50.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6400f040 con 0x7f8f54004d10 2023-10-30T13:44:50.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f637fe640 1 -- 172.21.15.148:0/3357553763 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f640ad440 con 0x7f8f54004d10 2023-10-30T13:44:50.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f72e09640 1 -- 172.21.15.148:0/3357553763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f54004d10 msgr2=0x7f8f54005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f72e09640 1 --2- 172.21.15.148:0/3357553763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f54004d10 0x7f8f54005190 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c066260 tx=0x7f8f6401cc00 comp rx=0 tx=0).stop 2023-10-30T13:44:50.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f713a6640 1 -- 172.21.15.148:0/3357553763 reap_dead start 2023-10-30T13:44:50.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f72e09640 1 -- 172.21.15.148:0/3357553763 shutdown_connections 2023-10-30T13:44:50.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f72e09640 1 -- 172.21.15.148:0/3357553763 >> 172.21.15.148:0/3357553763 conn(0x7f8f6c06c6e0 msgr2=0x7f8f6c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:50.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f72e09640 1 -- 172.21.15.148:0/3357553763 shutdown_connections 2023-10-30T13:44:50.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f72e09640 1 -- 172.21.15.148:0/3357553763 wait complete. 2023-10-30T13:44:50.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.260+0000 7f8f72e09640 1 Processor -- start 2023-10-30T13:44:50.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 -- start start 2023-10-30T13:44:50.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54004d10 0x7f8f6c154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14d180 0x7f8f6c0ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0cab70 0x7f8f6c149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0c6870 con 0x7f8f6c0cab70 2023-10-30T13:44:50.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c1efa80 con 0x7f8f54004d10 2023-10-30T13:44:50.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0c6b10 con 0x7f8f6c14d180 2023-10-30T13:44:50.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f713a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0cab70 0x7f8f6c149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f713a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0cab70 0x7f8f6c149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49710/0 (socket says 172.21.15.148:49710) 2023-10-30T13:44:50.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f713a6640 1 -- 172.21.15.148:0/1314142457 learned_addr learned my addr 172.21.15.148:0/1314142457 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:50.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f70ba5640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54004d10 0x7f8f6c154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f63fff640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14d180 0x7f8f6c0ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f713a6640 1 -- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14d180 msgr2=0x7f8f6c0ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f713a6640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14d180 0x7f8f6c0ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f713a6640 1 -- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54004d10 msgr2=0x7f8f6c154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f713a6640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54004d10 0x7f8f6c154600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f713a6640 1 -- 172.21.15.148:0/1314142457 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c0c6df0 con 0x7f8f6c0cab70 2023-10-30T13:44:50.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f70ba5640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54004d10 0x7f8f6c154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:50.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f713a6640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0cab70 0x7f8f6c149a10 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f8f64002240 tx=0x7f8f64009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f617fa640 1 -- 172.21.15.148:0/1314142457 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f640039a0 con 0x7f8f6c0cab70 2023-10-30T13:44:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f617fa640 1 -- 172.21.15.148:0/1314142457 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f64003b00 con 0x7f8f6c0cab70 2023-10-30T13:44:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c0c70d0 con 0x7f8f6c0cab70 2023-10-30T13:44:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f617fa640 1 -- 172.21.15.148:0/1314142457 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64003e90 con 0x7f8f6c0cab70 2023-10-30T13:44:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c154b40 con 0x7f8f6c0cab70 2023-10-30T13:44:50.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.264+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c14f9e0 con 0x7f8f6c0cab70 2023-10-30T13:44:50.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.268+0000 7f8f617fa640 1 -- 172.21.15.148:0/1314142457 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f640b4040 con 0x7f8f6c0cab70 2023-10-30T13:44:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.268+0000 7f8f617fa640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f580664a0 0x7f8f58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.268+0000 7f8f617fa640 1 -- 172.21.15.148:0/1314142457 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f8f6411e360 con 0x7f8f6c0cab70 2023-10-30T13:44:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.268+0000 7f8f70ba5640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f580664a0 0x7f8f58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.268+0000 7f8f70ba5640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f580664a0 0x7f8f58068960 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8f68001060 tx=0x7f8f68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:50.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.272+0000 7f8f617fa640 1 -- 172.21.15.148:0/1314142457 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f640eb1a0 con 0x7f8f6c0cab70 2023-10-30T13:44:50.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f6c04e170 con 0x7f8f6c0cab70 2023-10-30T13:44:50.455 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f617fa640 1 -- 172.21.15.148:0/1314142457 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f8f640a5e60 con 0x7f8f6c0cab70 2023-10-30T13:44:50.456 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:50.456 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:44.806238+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:50.456 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 127 2023-10-30T13:44:50.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f580664a0 msgr2=0x7f8f58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f580664a0 0x7f8f58068960 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8f68001060 tx=0x7f8f68009040 comp rx=0 tx=0).stop 2023-10-30T13:44:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0cab70 msgr2=0x7f8f6c149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0cab70 0x7f8f6c149a10 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f8f64002240 tx=0x7f8f64009750 comp rx=0 tx=0).stop 2023-10-30T13:44:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 shutdown_connections 2023-10-30T13:44:50.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8f580664a0 0x7f8f58068960 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8f6c0cab70 0x7f8f6c149a10 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8f6c14d180 0x7f8f6c0ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 --2- 172.21.15.148:0/1314142457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f54004d10 0x7f8f6c154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.456+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 >> 172.21.15.148:0/1314142457 conn(0x7f8f6c06c6e0 msgr2=0x7f8f6c150990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.460+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 shutdown_connections 2023-10-30T13:44:50.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:50.460+0000 7f8f72e09640 1 -- 172.21.15.148:0/1314142457 wait complete. 2023-10-30T13:44:50.476 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:52.477 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:52.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.624+0000 7f7a933c5640 1 Processor -- start 2023-10-30T13:44:52.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.624+0000 7f7a933c5640 1 -- start start 2023-10-30T13:44:52.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a933c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7a8c0bfec0 0x7f7a8c0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a933c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a933c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8c071290 con 0x7f7a8c159e60 2023-10-30T13:44:52.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a933c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8c071570 con 0x7f7a8c0bfec0 2023-10-30T13:44:52.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a933c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c071850 con 0x7f7a8c06a720 2023-10-30T13:44:52.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a91962640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a90960640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7a8c0bfec0 0x7f7a8c0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a91962640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:42844/0 (socket says 172.21.15.148:42844) 2023-10-30T13:44:52.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a91962640 1 -- 172.21.15.148:0/325294861 learned_addr learned my addr 172.21.15.148:0/325294861 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:52.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a91962640 1 -- 172.21.15.148:0/325294861 >> v2:172.21.15.150:3300/0 conn(0x7f7a8c0bfec0 msgr2=0x7f7a8c0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:52.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a91962640 1 --2- 172.21.15.148:0/325294861 >> v2:172.21.15.150:3300/0 conn(0x7f7a8c0bfec0 0x7f7a8c0c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a91962640 1 -- 172.21.15.148:0/325294861 >> v1:172.21.15.150:6789/0 conn(0x7f7a8c159e60 legacy=0x7f7a8c06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.628+0000 7f7a91962640 1 -- 172.21.15.148:0/325294861 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c071b30 con 0x7f7a8c06a720 2023-10-30T13:44:52.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a91962640 1 --2- 172.21.15.148:0/325294861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c06abb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009e50 tx=0x7f7a8401f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=323a7368edf4ce2c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:52.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a83fff640 1 -- 172.21.15.148:0/325294861 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a840af400 con 0x7f7a8c06a720 2023-10-30T13:44:52.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a83fff640 1 -- 172.21.15.148:0/325294861 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a840ab050 con 0x7f7a8c06a720 2023-10-30T13:44:52.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a83fff640 1 -- 172.21.15.148:0/325294861 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a840af780 con 0x7f7a8c06a720 2023-10-30T13:44:52.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 -- 172.21.15.148:0/325294861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 msgr2=0x7f7a8c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 --2- 172.21.15.148:0/325294861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c06abb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009e50 tx=0x7f7a8401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 -- 172.21.15.148:0/325294861 shutdown_connections 2023-10-30T13:44:52.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 --2- 172.21.15.148:0/325294861 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c06abb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 --2- 172.21.15.148:0/325294861 >> v2:172.21.15.150:3300/0 conn(0x7f7a8c0bfec0 0x7f7a8c0c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 -- 172.21.15.148:0/325294861 >> 172.21.15.148:0/325294861 conn(0x7f7a8c0c7910 msgr2=0x7f7a8c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:52.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 -- 172.21.15.148:0/325294861 shutdown_connections 2023-10-30T13:44:52.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 -- 172.21.15.148:0/325294861 wait complete. 2023-10-30T13:44:52.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 Processor -- start 2023-10-30T13:44:52.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.632+0000 7f7a933c5640 1 -- start start 2023-10-30T13:44:52.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a933c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c14cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a933c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8c0bfec0 0x7f7a8c0c1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a933c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a8c159e60 0x7f7a8c14a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a933c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c0c21f0 con 0x7f7a8c159e60 2023-10-30T13:44:52.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a933c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c14b300 con 0x7f7a8c0bfec0 2023-10-30T13:44:52.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a933c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c1ed440 con 0x7f7a8c06a720 2023-10-30T13:44:52.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91962640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a8c159e60 0x7f7a8c14a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91962640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a8c159e60 0x7f7a8c14a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35076/0 (socket says 172.21.15.148:35076) 2023-10-30T13:44:52.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91962640 1 -- 172.21.15.148:0/1547609048 learned_addr learned my addr 172.21.15.148:0/1547609048 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:52.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91161640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c14cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a90960640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8c0bfec0 0x7f7a8c0c1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91962640 1 -- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 msgr2=0x7f7a8c14cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91962640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c14cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91962640 1 -- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8c0bfec0 msgr2=0x7f7a8c0c1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91962640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8c0bfec0 0x7f7a8c0c1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91962640 1 -- 172.21.15.148:0/1547609048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a84009b30 con 0x7f7a8c159e60 2023-10-30T13:44:52.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a91962640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a8c159e60 0x7f7a8c14a620 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f7a8401ce20 tx=0x7f7a8401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:52.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a81ffb640 1 -- 172.21.15.148:0/1547609048 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a840afd30 con 0x7f7a8c159e60 2023-10-30T13:44:52.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a81ffb640 1 -- 172.21.15.148:0/1547609048 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a840afe90 con 0x7f7a8c159e60 2023-10-30T13:44:52.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8c1ed970 con 0x7f7a8c159e60 2023-10-30T13:44:52.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a81ffb640 1 -- 172.21.15.148:0/1547609048 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a840ba680 con 0x7f7a8c159e60 2023-10-30T13:44:52.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8c14b510 con 0x7f7a8c159e60 2023-10-30T13:44:52.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8c0c19d0 con 0x7f7a8c159e60 2023-10-30T13:44:52.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.636+0000 7f7a81ffb640 1 -- 172.21.15.148:0/1547609048 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a840b6040 con 0x7f7a8c159e60 2023-10-30T13:44:52.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.640+0000 7f7a81ffb640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7a700664d0 0x7f7a70068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:52.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.640+0000 7f7a81ffb640 1 -- 172.21.15.148:0/1547609048 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f7a841225e0 con 0x7f7a8c159e60 2023-10-30T13:44:52.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.640+0000 7f7a91161640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7a700664d0 0x7f7a70068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:52.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.640+0000 7f7a91161640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7a700664d0 0x7f7a70068990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000f70 tx=0x7f7a8800c4e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:52.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.644+0000 7f7a81ffb640 1 -- 172.21.15.148:0/1547609048 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a8c0c19d0 con 0x7f7a8c159e60 2023-10-30T13:44:52.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.824+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a8c0c19d0 con 0x7f7a8c159e60 2023-10-30T13:44:52.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a81ffb640 1 -- 172.21.15.148:0/1547609048 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f7a8c0c19d0 con 0x7f7a8c159e60 2023-10-30T13:44:52.826 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:52.826 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:50.063395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:52.826 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 128 2023-10-30T13:44:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7a700664d0 msgr2=0x7f7a70068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7a700664d0 0x7f7a70068990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000f70 tx=0x7f7a8800c4e0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a8c159e60 msgr2=0x7f7a8c14a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a8c159e60 0x7f7a8c14a620 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f7a8401ce20 tx=0x7f7a8401bf90 comp rx=0 tx=0).stop 2023-10-30T13:44:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 shutdown_connections 2023-10-30T13:44:52.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7a700664d0 0x7f7a70068990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a8c159e60 0x7f7a8c14a620 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8c0bfec0 0x7f7a8c0c1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 --2- 172.21.15.148:0/1547609048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7a8c06a720 0x7f7a8c14cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:52.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 >> 172.21.15.148:0/1547609048 conn(0x7f7a8c0c7910 msgr2=0x7f7a8c073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:52.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 shutdown_connections 2023-10-30T13:44:52.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:52.828+0000 7f7a933c5640 1 -- 172.21.15.148:0/1547609048 wait complete. 2023-10-30T13:44:52.847 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:54.848 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:54.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:54.996+0000 7fb81d10e640 1 Processor -- start 2023-10-30T13:44:54.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:54.996+0000 7fb81d10e640 1 -- start start 2023-10-30T13:44:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb81d10e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb818159e60 0x7fb8180c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:54.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb81d10e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb818069e90 0x7fb81806a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb81d10e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb818071290 con 0x7fb8180bfec0 2023-10-30T13:44:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb81d10e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb818071570 con 0x7fb818159e60 2023-10-30T13:44:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb81d10e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb818071850 con 0x7fb818069e90 2023-10-30T13:44:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb80e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb818159e60 0x7fb8180c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb80e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb818159e60 0x7fb8180c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35080/0 (socket says 172.21.15.148:35080) 2023-10-30T13:44:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb80e59c640 1 -- 172.21.15.148:0/788272336 learned_addr learned my addr 172.21.15.148:0/788272336 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb81759e640 1 --2- 172.21.15.148:0/788272336 >> v2:172.21.15.150:3300/0 conn(0x7fb818069e90 0x7fb81806a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb80e59c640 1 -- 172.21.15.148:0/788272336 >> v2:172.21.15.150:3300/0 conn(0x7fb818069e90 msgr2=0x7fb81806a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb80e59c640 1 --2- 172.21.15.148:0/788272336 >> v2:172.21.15.150:3300/0 conn(0x7fb818069e90 0x7fb81806a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb80e59c640 1 -- 172.21.15.148:0/788272336 >> v1:172.21.15.150:6789/0 conn(0x7fb8180bfec0 legacy=0x7fb81806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.000+0000 7fb80e59c640 1 -- 172.21.15.148:0/788272336 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb818071b30 con 0x7fb818159e60 2023-10-30T13:44:55.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb80e59c640 1 --2- 172.21.15.148:0/788272336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818159e60 0x7fb8180c1a40 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb818071e10 tx=0x7fb8000213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0bf29f149bc2b55 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:55.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81659c640 1 -- 172.21.15.148:0/788272336 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8000b1600 con 0x7fb818159e60 2023-10-30T13:44:55.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81659c640 1 -- 172.21.15.148:0/788272336 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8000ad050 con 0x7fb818159e60 2023-10-30T13:44:55.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81659c640 1 -- 172.21.15.148:0/788272336 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8000b1980 con 0x7fb818159e60 2023-10-30T13:44:55.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81d10e640 1 -- 172.21.15.148:0/788272336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818159e60 msgr2=0x7fb8180c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81d10e640 1 --2- 172.21.15.148:0/788272336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818159e60 0x7fb8180c1a40 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fb818071e10 tx=0x7fb8000213c0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81d10e640 1 -- 172.21.15.148:0/788272336 shutdown_connections 2023-10-30T13:44:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81d10e640 1 --2- 172.21.15.148:0/788272336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818159e60 0x7fb8180c1a40 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81d10e640 1 --2- 172.21.15.148:0/788272336 >> v2:172.21.15.150:3300/0 conn(0x7fb818069e90 0x7fb81806a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81d10e640 1 -- 172.21.15.148:0/788272336 >> 172.21.15.148:0/788272336 conn(0x7fb8180c7910 msgr2=0x7fb8180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81d10e640 1 -- 172.21.15.148:0/788272336 shutdown_connections 2023-10-30T13:44:55.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81d10e640 1 -- 172.21.15.148:0/788272336 wait complete. 2023-10-30T13:44:55.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.004+0000 7fb81d10e640 1 Processor -- start 2023-10-30T13:44:55.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 -- start start 2023-10-30T13:44:55.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818069e90 0x7fb8180c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8180bfec0 0x7fb8180c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb818159e60 0x7fb81814c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8180c6520 con 0x7fb818069e90 2023-10-30T13:44:55.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81814d5e0 con 0x7fb818159e60 2023-10-30T13:44:55.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8180c6790 con 0x7fb8180bfec0 2023-10-30T13:44:55.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb816d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818069e90 0x7fb8180c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb816d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818069e90 0x7fb8180c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35086/0 (socket says 172.21.15.148:35086) 2023-10-30T13:44:55.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb816d9d640 1 -- 172.21.15.148:0/1953138202 learned_addr learned my addr 172.21.15.148:0/1953138202 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:55.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb80e59c640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8180bfec0 0x7fb8180c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81759e640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb818159e60 0x7fb81814c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb816d9d640 1 -- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8180bfec0 msgr2=0x7fb8180c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb816d9d640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8180bfec0 0x7fb8180c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb816d9d640 1 -- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb818159e60 msgr2=0x7fb81814c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb816d9d640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb818159e60 0x7fb81814c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb816d9d640 1 -- 172.21.15.148:0/1953138202 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80000b770 con 0x7fb818069e90 2023-10-30T13:44:55.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb816d9d640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818069e90 0x7fb8180c5650 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c7f0 tx=0x7fb80800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:55.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb80ffff640 1 -- 172.21.15.148:0/1953138202 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808003b80 con 0x7fb818069e90 2023-10-30T13:44:55.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb80ffff640 1 -- 172.21.15.148:0/1953138202 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb808003ce0 con 0x7fb818069e90 2023-10-30T13:44:55.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb80ffff640 1 -- 172.21.15.148:0/1953138202 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80800aa90 con 0x7fb818069e90 2023-10-30T13:44:55.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8181eed60 con 0x7fb818069e90 2023-10-30T13:44:55.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8181ef320 con 0x7fb818069e90 2023-10-30T13:44:55.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.008+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81804e170 con 0x7fb818069e90 2023-10-30T13:44:55.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.012+0000 7fb80ffff640 1 -- 172.21.15.148:0/1953138202 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb80800abf0 con 0x7fb818069e90 2023-10-30T13:44:55.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.012+0000 7fb80ffff640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f80664a0 0x7fb7f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:55.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.012+0000 7fb80ffff640 1 -- 172.21.15.148:0/1953138202 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fb808084410 con 0x7fb818069e90 2023-10-30T13:44:55.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.012+0000 7fb80e59c640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f80664a0 0x7fb7f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:55.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.012+0000 7fb80e59c640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f80664a0 0x7fb7f8068960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb8000b3000 tx=0x7fb800008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:55.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.016+0000 7fb80ffff640 1 -- 172.21.15.148:0/1953138202 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb808051240 con 0x7fb818069e90 2023-10-30T13:44:55.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.196+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb81806bef0 con 0x7fb818069e90 2023-10-30T13:44:55.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb80ffff640 1 -- 172.21.15.148:0/1953138202 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5720 (secure 0 0 0) 0x7fb808054860 con 0x7fb818069e90 2023-10-30T13:44:55.198 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:55.198 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:50.063395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:55.199 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 128 2023-10-30T13:44:55.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f80664a0 msgr2=0x7fb7f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f80664a0 0x7fb7f8068960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb8000b3000 tx=0x7fb800008040 comp rx=0 tx=0).stop 2023-10-30T13:44:55.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818069e90 msgr2=0x7fb8180c5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:55.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818069e90 0x7fb8180c5650 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c7f0 tx=0x7fb80800ccb0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 shutdown_connections 2023-10-30T13:44:55.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f80664a0 0x7fb7f8068960 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb818159e60 0x7fb81814c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb8180bfec0 0x7fb8180c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 --2- 172.21.15.148:0/1953138202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb818069e90 0x7fb8180c5650 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:55.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 >> 172.21.15.148:0/1953138202 conn(0x7fb8180c7910 msgr2=0x7fb81814f830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:55.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 shutdown_connections 2023-10-30T13:44:55.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:55.200+0000 7fb81d10e640 1 -- 172.21.15.148:0/1953138202 wait complete. 2023-10-30T13:44:55.219 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:57.220 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:57.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61d009640 1 Processor -- start 2023-10-30T13:44:57.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61d009640 1 -- start start 2023-10-30T13:44:57.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61d009640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa61806f7a0 0x7fa61806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61d009640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6180c6bd0 0x7fa618069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61d009640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa618071110 0x7fa618158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61d009640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6180700c0 con 0x7fa618071110 2023-10-30T13:44:57.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61d009640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61806a3d0 con 0x7fa6180c6bd0 2023-10-30T13:44:57.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61d009640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa618066cf0 con 0x7fa61806f7a0 2023-10-30T13:44:57.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61759e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa618071110 0x7fa618158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61759e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa618071110 0x7fa618158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35102/0 (socket says 172.21.15.148:35102) 2023-10-30T13:44:57.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61759e640 1 -- 172.21.15.148:0/1290058430 learned_addr learned my addr 172.21.15.148:0/1290058430 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:57.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa616d9d640 1 --2- 172.21.15.148:0/1290058430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa61806f7a0 0x7fa61806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa60e59c640 1 --2- 172.21.15.148:0/1290058430 >> v2:172.21.15.150:3300/0 conn(0x7fa6180c6bd0 0x7fa618069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61759e640 1 -- 172.21.15.148:0/1290058430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa61806f7a0 msgr2=0x7fa61806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:57.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61759e640 1 --2- 172.21.15.148:0/1290058430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa61806f7a0 0x7fa61806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61759e640 1 -- 172.21.15.148:0/1290058430 >> v2:172.21.15.150:3300/0 conn(0x7fa6180c6bd0 msgr2=0x7fa618069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:57.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61759e640 1 --2- 172.21.15.148:0/1290058430 >> v2:172.21.15.150:3300/0 conn(0x7fa6180c6bd0 0x7fa618069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.360+0000 7fa61759e640 1 -- 172.21.15.148:0/1290058430 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6180c70d0 con 0x7fa618071110 2023-10-30T13:44:57.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61759e640 1 --2- 172.21.15.148:0/1290058430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa618071110 0x7fa618158110 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fa6180b1210 tx=0x7fa61001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5d20e604c497a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61659c640 1 -- 172.21.15.148:0/1290058430 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6100af400 con 0x7fa618071110 2023-10-30T13:44:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61659c640 1 -- 172.21.15.148:0/1290058430 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6100ab050 con 0x7fa618071110 2023-10-30T13:44:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61659c640 1 -- 172.21.15.148:0/1290058430 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6100af780 con 0x7fa618071110 2023-10-30T13:44:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61d009640 1 -- 172.21.15.148:0/1290058430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa618071110 msgr2=0x7fa618158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61d009640 1 --2- 172.21.15.148:0/1290058430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa618071110 0x7fa618158110 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fa6180b1210 tx=0x7fa61001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61d009640 1 -- 172.21.15.148:0/1290058430 shutdown_connections 2023-10-30T13:44:57.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61d009640 1 --2- 172.21.15.148:0/1290058430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa618071110 0x7fa618158110 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61d009640 1 --2- 172.21.15.148:0/1290058430 >> v2:172.21.15.150:3300/0 conn(0x7fa6180c6bd0 0x7fa618069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61d009640 1 --2- 172.21.15.148:0/1290058430 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa61806f7a0 0x7fa61806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61d009640 1 -- 172.21.15.148:0/1290058430 >> 172.21.15.148:0/1290058430 conn(0x7fa6180c7910 msgr2=0x7fa6180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:57.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61d009640 1 -- 172.21.15.148:0/1290058430 shutdown_connections 2023-10-30T13:44:57.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.364+0000 7fa61d009640 1 -- 172.21.15.148:0/1290058430 wait complete. 2023-10-30T13:44:57.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 Processor -- start 2023-10-30T13:44:57.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 -- start start 2023-10-30T13:44:57.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61806f7a0 0x7fa6180c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa618071110 0x7fa61815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c6bd0 0x7fa6181497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61814ab20 con 0x7fa61806f7a0 2023-10-30T13:44:57.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61815a740 con 0x7fa6180c6bd0 2023-10-30T13:44:57.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6181588a0 con 0x7fa618071110 2023-10-30T13:44:57.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa616d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61806f7a0 0x7fa6180c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa616d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61806f7a0 0x7fa6180c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35106/0 (socket says 172.21.15.148:35106) 2023-10-30T13:44:57.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa616d9d640 1 -- 172.21.15.148:0/251119845 learned_addr learned my addr 172.21.15.148:0/251119845 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:57.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa60e59c640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa618071110 0x7fa61815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61759e640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c6bd0 0x7fa6181497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa616d9d640 1 -- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa618071110 msgr2=0x7fa61815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:57.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa616d9d640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa618071110 0x7fa61815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa616d9d640 1 -- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c6bd0 msgr2=0x7fa6181497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa616d9d640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c6bd0 0x7fa6181497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa616d9d640 1 -- 172.21.15.148:0/251119845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610009b30 con 0x7fa61806f7a0 2023-10-30T13:44:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa616d9d640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61806f7a0 0x7fa6180c33e0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fa60000e8a0 tx=0x7fa60000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa60ffff640 1 -- 172.21.15.148:0/251119845 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60000ee90 con 0x7fa61806f7a0 2023-10-30T13:44:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa60ffff640 1 -- 172.21.15.148:0/251119845 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60000c8e0 con 0x7fa61806f7a0 2023-10-30T13:44:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa60ffff640 1 -- 172.21.15.148:0/251119845 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60000cc90 con 0x7fa61806f7a0 2023-10-30T13:44:57.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa618158b20 con 0x7fa61806f7a0 2023-10-30T13:44:57.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.368+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6181590e0 con 0x7fa61806f7a0 2023-10-30T13:44:57.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.372+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61804e170 con 0x7fa61806f7a0 2023-10-30T13:44:57.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.372+0000 7fa60ffff640 1 -- 172.21.15.148:0/251119845 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa600013470 con 0x7fa61806f7a0 2023-10-30T13:44:57.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.372+0000 7fa60ffff640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa5f80664a0 0x7fa5f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:57.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.372+0000 7fa60ffff640 1 -- 172.21.15.148:0/251119845 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fa600084460 con 0x7fa61806f7a0 2023-10-30T13:44:57.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.372+0000 7fa60e59c640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa5f80664a0 0x7fa5f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:57.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.372+0000 7fa60e59c640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa5f80664a0 0x7fa5f8068960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa61806feb0 tx=0x7fa608009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:57.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.376+0000 7fa60ffff640 1 -- 172.21.15.148:0/251119845 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6000512a0 con 0x7fa61806f7a0 2023-10-30T13:44:57.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.556+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6180c7d90 con 0x7fa61806f7a0 2023-10-30T13:44:57.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.556+0000 7fa60ffff640 1 -- 172.21.15.148:0/251119845 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5720 (secure 0 0 0) 0x7fa6000548c0 con 0x7fa61806f7a0 2023-10-30T13:44:57.558 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:57.558 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:50.063395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:57.558 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 128 2023-10-30T13:44:57.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa5f80664a0 msgr2=0x7fa5f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:57.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa5f80664a0 0x7fa5f8068960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa61806feb0 tx=0x7fa608009190 comp rx=0 tx=0).stop 2023-10-30T13:44:57.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61806f7a0 msgr2=0x7fa6180c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:57.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61806f7a0 0x7fa6180c33e0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fa60000e8a0 tx=0x7fa60000ed60 comp rx=0 tx=0).stop 2023-10-30T13:44:57.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 shutdown_connections 2023-10-30T13:44:57.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa5f80664a0 0x7fa5f8068960 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6180c6bd0 0x7fa6181497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa618071110 0x7fa61815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 --2- 172.21.15.148:0/251119845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61806f7a0 0x7fa6180c33e0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:57.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 >> 172.21.15.148:0/251119845 conn(0x7fa6180c7910 msgr2=0x7fa6180c4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:57.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 shutdown_connections 2023-10-30T13:44:57.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:57.560+0000 7fa61d009640 1 -- 172.21.15.148:0/251119845 wait complete. 2023-10-30T13:44:57.578 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:44:59.579 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:44:59.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f346110f640 1 Processor -- start 2023-10-30T13:44:59.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f346110f640 1 -- start start 2023-10-30T13:44:59.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f346110f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f345c0c5f60 0x7f345c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:59.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f346110f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c0700b0 con 0x7f345c153bf0 2023-10-30T13:44:59.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f346110f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c0c6530 con 0x7f345c0c6fe0 2023-10-30T13:44:59.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f346110f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345c0c6810 con 0x7f345c0c5f60 2023-10-30T13:44:59.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f345ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f345c153bf0 0x7f345c0c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53200/0 (socket says 172.21.15.148:53200) 2023-10-30T13:44:59.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f345ad9d640 1 -- 172.21.15.148:0/886199697 learned_addr learned my addr 172.21.15.148:0/886199697 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:44:59.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f345b59e640 1 --2- 172.21.15.148:0/886199697 >> v2:172.21.15.150:3300/0 conn(0x7f345c0c5f60 0x7f345c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:59.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418098726 0 0) 0x7f345c0700b0 con 0x7f345c153bf0 2023-10-30T13:44:59.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.728+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3448003530 con 0x7f345c153bf0 2023-10-30T13:44:59.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345449442 0 0) 0x7f3448003530 con 0x7f345c153bf0 2023-10-30T13:44:59.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 >> v2:172.21.15.150:3300/0 conn(0x7f345c0c5f60 msgr2=0x7f345c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:59.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 --2- 172.21.15.148:0/886199697 >> v2:172.21.15.150:3300/0 conn(0x7f345c0c5f60 0x7f345c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 >> v1:172.21.15.150:6789/0 conn(0x7f345c0c6fe0 legacy=0x7f345c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:59.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c159b10 con 0x7f345c153bf0 2023-10-30T13:44:59.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f344c001680 con 0x7f345c153bf0 2023-10-30T13:44:59.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 >> v1:172.21.15.148:6789/0 conn(0x7f345c153bf0 legacy=0x7f345c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:59.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3448003980 0x7f3448003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:59.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f3448004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:59.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3448005190 0x7f344800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:59.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0c6810 con 0x7f34480047d0 2023-10-30T13:44:59.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344800c040 con 0x7f3448005190 2023-10-30T13:44:59.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344800c200 con 0x7f3448003980 2023-10-30T13:44:59.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345259c640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f3448004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:59.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345b59e640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3448005190 0x7f344800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:59.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345ad9d640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3448003980 0x7f3448003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:59.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345259c640 1 -- 172.21.15.148:0/886199697 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3448003980 msgr2=0x7f3448003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:59.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345259c640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3448003980 0x7f3448003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345259c640 1 -- 172.21.15.148:0/886199697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3448005190 msgr2=0x7f344800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:44:59.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345259c640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3448005190 0x7f344800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345259c640 1 -- 172.21.15.148:0/886199697 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3448003530 con 0x7f34480047d0 2023-10-30T13:44:59.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345b59e640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3448005190 0x7f344800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:44:59.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345b59e640 1 -- 172.21.15.148:0/886199697 reap_dead start 2023-10-30T13:44:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345259c640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f3448004c50 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3444001780 tx=0x7f344401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9b5253ac082bd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34440052d0 con 0x7f34480047d0 2023-10-30T13:44:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34440a7050 con 0x7f34480047d0 2023-10-30T13:44:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.732+0000 7f345a59c640 1 -- 172.21.15.148:0/886199697 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444003600 con 0x7f34480047d0 2023-10-30T13:44:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 -- 172.21.15.148:0/886199697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 msgr2=0x7f3448004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f3448004c50 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3444001780 tx=0x7f344401c3e0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 -- 172.21.15.148:0/886199697 shutdown_connections 2023-10-30T13:44:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 --2- 172.21.15.148:0/886199697 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f3448004c50 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 -- 172.21.15.148:0/886199697 >> 172.21.15.148:0/886199697 conn(0x7f345c06c6e0 msgr2=0x7f345c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:59.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 -- 172.21.15.148:0/886199697 shutdown_connections 2023-10-30T13:44:59.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 -- 172.21.15.148:0/886199697 wait complete. 2023-10-30T13:44:59.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 Processor -- start 2023-10-30T13:44:59.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 -- start start 2023-10-30T13:44:59.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f345c14cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:59.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f345c0c4be0 0x7f345c0c4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:59.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f345c14bb30 0x7f345c14bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:59.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c14b710 con 0x7f34480047d0 2023-10-30T13:44:59.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c14b3c0 con 0x7f345c0c4be0 2023-10-30T13:44:59.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f346110f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c14c4f0 con 0x7f345c14bb30 2023-10-30T13:44:59.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f345ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f345c14cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:59.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f345ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f345c14cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35114/0 (socket says 172.21.15.148:35114) 2023-10-30T13:44:59.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.736+0000 7f345ad9d640 1 -- 172.21.15.148:0/101348104 learned_addr learned my addr 172.21.15.148:0/101348104 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:44:59.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f345b59e640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f345c14bb30 0x7f345c14bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:59.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f345259c640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f345c0c4be0 0x7f345c0c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:59.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f345ad9d640 1 -- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f345c14bb30 msgr2=0x7f345c14bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f345ad9d640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f345c14bb30 0x7f345c14bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f345ad9d640 1 -- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f345c0c4be0 msgr2=0x7f345c0c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f345ad9d640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f345c0c4be0 0x7f345c0c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f345ad9d640 1 -- 172.21.15.148:0/101348104 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c1f32a0 con 0x7f34480047d0 2023-10-30T13:44:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f345ad9d640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f345c14cca0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f344c00e7e0 tx=0x7f344c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f3453fff640 1 -- 172.21.15.148:0/101348104 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c00ee90 con 0x7f34480047d0 2023-10-30T13:44:59.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f3453fff640 1 -- 172.21.15.148:0/101348104 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c00c8d0 con 0x7f34480047d0 2023-10-30T13:44:59.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f3453fff640 1 -- 172.21.15.148:0/101348104 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c00cc80 con 0x7f34480047d0 2023-10-30T13:44:59.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345c1e5590 con 0x7f34480047d0 2023-10-30T13:44:59.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345c1e5b10 con 0x7f34480047d0 2023-10-30T13:44:59.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345c04e170 con 0x7f34480047d0 2023-10-30T13:44:59.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f3453fff640 1 -- 172.21.15.148:0/101348104 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f344c01d460 con 0x7f34480047d0 2023-10-30T13:44:59.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f3453fff640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f343c0664a0 0x7f343c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:44:59.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.740+0000 7f3453fff640 1 -- 172.21.15.148:0/101348104 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f344c084280 con 0x7f34480047d0 2023-10-30T13:44:59.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.744+0000 7f345259c640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f343c0664a0 0x7f343c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:44:59.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.744+0000 7f345259c640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f343c0664a0 0x7f343c068960 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3444001750 tx=0x7f34440011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:44:59.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.744+0000 7f3453fff640 1 -- 172.21.15.148:0/101348104 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f344c051150 con 0x7f34480047d0 2023-10-30T13:44:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f345c06fe90 con 0x7f34480047d0 2023-10-30T13:44:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f3453fff640 1 -- 172.21.15.148:0/101348104 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f344c054770 con 0x7f34480047d0 2023-10-30T13:44:59.928 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:44:59.928 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:57.121005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:clientreplay","state_seq":30,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:44:59.930 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 129 2023-10-30T13:44:59.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f343c0664a0 msgr2=0x7f343c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:59.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f343c0664a0 0x7f343c068960 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3444001750 tx=0x7f34440011a0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 msgr2=0x7f345c14cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:44:59.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f345c14cca0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f344c00e7e0 tx=0x7f344c00eca0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 shutdown_connections 2023-10-30T13:44:59.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f343c0664a0 0x7f343c068960 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f345c14bb30 0x7f345c14bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f345c0c4be0 0x7f345c0c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 --2- 172.21.15.148:0/101348104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f34480047d0 0x7f345c14cca0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:44:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 >> 172.21.15.148:0/101348104 conn(0x7f345c06c6e0 msgr2=0x7f345c151d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:44:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 shutdown_connections 2023-10-30T13:44:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:44:59.928+0000 7f346110f640 1 -- 172.21.15.148:0/101348104 wait complete. 2023-10-30T13:44:59.949 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 129, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 129, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7464": { "addr": "172.21.15.148:6837/3703944646", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3703944646, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3703944646, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7464, "incarnation": 126, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 30 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:44:57.121005+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7464 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:45:01.950 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:02.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41ff33e640 1 Processor -- start 2023-10-30T13:45:02.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41ff33e640 1 -- start start 2023-10-30T13:45:02.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41ff33e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41f81580e0 0x7f41f8070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41ff33e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f80c5810 con 0x7f41f80c6540 2023-10-30T13:45:02.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41ff33e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f80c6990 con 0x7f41f815e700 2023-10-30T13:45:02.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41ff33e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f806acb0 con 0x7f41f81580e0 2023-10-30T13:45:02.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41fc8d9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f41f80c6540 0x7f41f8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40384/0 (socket says 172.21.15.148:40384) 2023-10-30T13:45:02.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41fc8d9640 1 -- 172.21.15.148:0/1307930644 learned_addr learned my addr 172.21.15.148:0/1307930644 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:45:02.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41fd0da640 1 --2- 172.21.15.148:0/1307930644 >> v2:172.21.15.150:3300/0 conn(0x7f41f81580e0 0x7f41f8070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383491745 0 0) 0x7f41f80c5810 con 0x7f41f80c6540 2023-10-30T13:45:02.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e4003530 con 0x7f41f80c6540 2023-10-30T13:45:02.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3720931309 0 0) 0x7f41e4003530 con 0x7f41f80c6540 2023-10-30T13:45:02.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 >> v2:172.21.15.150:3300/0 conn(0x7f41f81580e0 msgr2=0x7f41f8070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:02.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 --2- 172.21.15.148:0/1307930644 >> v2:172.21.15.150:3300/0 conn(0x7f41f81580e0 0x7f41f8070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 >> v1:172.21.15.150:6789/0 conn(0x7f41f815e700 legacy=0x7f41f806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f815eb90 con 0x7f41f80c6540 2023-10-30T13:45:02.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f41e8001680 con 0x7f41f80c6540 2023-10-30T13:45:02.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 >> v1:172.21.15.148:6789/0 conn(0x7f41f80c6540 legacy=0x7f41f8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41e4003980 0x7f41e4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e40047d0 0x7f41e4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e4005190 0x7f41e400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e400c000 con 0x7f41e4003980 2023-10-30T13:45:02.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e400c170 con 0x7f41e4005190 2023-10-30T13:45:02.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e400c330 con 0x7f41e40047d0 2023-10-30T13:45:02.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.100+0000 7f41fd0da640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41e4003980 0x7f41e4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fd8db640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e4005190 0x7f41e400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fc8d9640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e40047d0 0x7f41e4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fd0da640 1 -- 172.21.15.148:0/1307930644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e40047d0 msgr2=0x7f41e4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:02.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fd0da640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e40047d0 0x7f41e4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fd0da640 1 -- 172.21.15.148:0/1307930644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e4005190 msgr2=0x7f41e400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:02.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fd0da640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e4005190 0x7f41e400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fd0da640 1 -- 172.21.15.148:0/1307930644 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e4003530 con 0x7f41e4003980 2023-10-30T13:45:02.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fd8db640 1 -- 172.21.15.148:0/1307930644 reap_dead start 2023-10-30T13:45:02.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fc8d9640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41e40047d0 0x7f41e4004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:45:02.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41fd0da640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41e4003980 0x7f41e4003da0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009ae0 tx=0x7f41f401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e103fa1214213826 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:02.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f40a5530 con 0x7f41e4003980 2023-10-30T13:45:02.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f40a5690 con 0x7f41e4003980 2023-10-30T13:45:02.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41effff640 1 -- 172.21.15.148:0/1307930644 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f40a5a80 con 0x7f41e4003980 2023-10-30T13:45:02.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41ff33e640 1 -- 172.21.15.148:0/1307930644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41e4003980 msgr2=0x7f41e4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41ff33e640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41e4003980 0x7f41e4003da0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009ae0 tx=0x7f41f401ca50 comp rx=0 tx=0).stop 2023-10-30T13:45:02.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41ff33e640 1 -- 172.21.15.148:0/1307930644 shutdown_connections 2023-10-30T13:45:02.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41ff33e640 1 --2- 172.21.15.148:0/1307930644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41e4003980 0x7f41e4003da0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.104+0000 7f41ff33e640 1 -- 172.21.15.148:0/1307930644 >> 172.21.15.148:0/1307930644 conn(0x7f41f80c7910 msgr2=0x7f41f80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:02.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 -- 172.21.15.148:0/1307930644 shutdown_connections 2023-10-30T13:45:02.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 -- 172.21.15.148:0/1307930644 wait complete. 2023-10-30T13:45:02.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 Processor -- start 2023-10-30T13:45:02.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 -- start start 2023-10-30T13:45:02.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e4003980 0x7f41f81f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41f814f020 0x7f41f814c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41f815ee40 0x7f41f8154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f81f7c20 con 0x7f41f814f020 2023-10-30T13:45:02.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f8151dd0 con 0x7f41e4003980 2023-10-30T13:45:02.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f814cb60 con 0x7f41f815ee40 2023-10-30T13:45:02.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fc8d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41f814f020 0x7f41f814c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fc8d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41f814f020 0x7f41f814c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57026/0 (socket says 172.21.15.148:57026) 2023-10-30T13:45:02.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fc8d9640 1 -- 172.21.15.148:0/241214128 learned_addr learned my addr 172.21.15.148:0/241214128 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:02.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fd0da640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e4003980 0x7f41f81f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fd8db640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41f815ee40 0x7f41f8154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fc8d9640 1 -- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41f815ee40 msgr2=0x7f41f8154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fc8d9640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41f815ee40 0x7f41f8154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fc8d9640 1 -- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e4003980 msgr2=0x7f41f81f76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fc8d9640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e4003980 0x7f41f81f76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fc8d9640 1 -- 172.21.15.148:0/241214128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f8151810 con 0x7f41f814f020 2023-10-30T13:45:02.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41fc8d9640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41f814f020 0x7f41f814c620 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f41e800d7a0 tx=0x7f41e800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:02.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41edffb640 1 -- 172.21.15.148:0/241214128 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e80074e0 con 0x7f41f814f020 2023-10-30T13:45:02.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41edffb640 1 -- 172.21.15.148:0/241214128 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e8007640 con 0x7f41f814f020 2023-10-30T13:45:02.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f8151af0 con 0x7f41f814f020 2023-10-30T13:45:02.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.108+0000 7f41edffb640 1 -- 172.21.15.148:0/241214128 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e80049f0 con 0x7f41f814f020 2023-10-30T13:45:02.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.112+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f81f9bf0 con 0x7f41f814f020 2023-10-30T13:45:02.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.112+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f804e170 con 0x7f41f814f020 2023-10-30T13:45:02.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.112+0000 7f41edffb640 1 -- 172.21.15.148:0/241214128 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41e8004b50 con 0x7f41f814f020 2023-10-30T13:45:02.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.112+0000 7f41edffb640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41dc0664d0 0x7f41dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:02.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.112+0000 7f41edffb640 1 -- 172.21.15.148:0/241214128 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f41e80840a0 con 0x7f41f814f020 2023-10-30T13:45:02.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.112+0000 7f41fd0da640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41dc0664d0 0x7f41dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:02.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.112+0000 7f41fd0da640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41dc0664d0 0x7f41dc068990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f41f40af000 tx=0x7f41f40a7040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:02.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.116+0000 7f41edffb640 1 -- 172.21.15.148:0/241214128 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41e8050ee0 con 0x7f41f814f020 2023-10-30T13:45:02.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.296+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41f814f600 con 0x7f41f814f020 2023-10-30T13:45:02.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.296+0000 7f41edffb640 1 -- 172.21.15.148:0/241214128 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5726 (secure 0 0 0) 0x7f41e8019090 con 0x7f41f814f020 2023-10-30T13:45:02.297 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:02.297 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:44:57.121005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:clientreplay","state_seq":30,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:02.297 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 129 2023-10-30T13:45:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41dc0664d0 msgr2=0x7f41dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41dc0664d0 0x7f41dc068990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f41f40af000 tx=0x7f41f40a7040 comp rx=0 tx=0).stop 2023-10-30T13:45:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41f814f020 msgr2=0x7f41f814c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:02.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41f814f020 0x7f41f814c620 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f41e800d7a0 tx=0x7f41e800dc60 comp rx=0 tx=0).stop 2023-10-30T13:45:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 shutdown_connections 2023-10-30T13:45:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f41dc0664d0 0x7f41dc068990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41f814f020 0x7f41f814c620 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f41f815ee40 0x7f41f8154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 --2- 172.21.15.148:0/241214128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41e4003980 0x7f41f81f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:02.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 >> 172.21.15.148:0/241214128 conn(0x7f41f80c7910 msgr2=0x7f41f8157350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:02.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 shutdown_connections 2023-10-30T13:45:02.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:02.300+0000 7f41ff33e640 1 -- 172.21.15.148:0/241214128 wait complete. 2023-10-30T13:45:02.318 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:45:02.562 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:02.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:04.319 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f674d180640 1 Processor -- start 2023-10-30T13:45:04.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f674d180640 1 -- start start 2023-10-30T13:45:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f674d180640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f67481538a0 0x7f674815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f674d180640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67480cf160 0x7f67480cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f674d180640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67480cb800 con 0x7f67480ca0e0 2023-10-30T13:45:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f674d180640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f674815dc30 con 0x7f67481538a0 2023-10-30T13:45:04.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f674d180640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67480cb330 con 0x7f67480cf160 2023-10-30T13:45:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f6746d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f67481538a0 0x7f674815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f6746d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f67481538a0 0x7f674815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57032/0 (socket says 172.21.15.148:57032) 2023-10-30T13:45:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f6746d9d640 1 -- 172.21.15.148:0/3522757842 learned_addr learned my addr 172.21.15.148:0/3522757842 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f674759e640 1 --2- 172.21.15.148:0/3522757842 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67480cf160 0x7f67480cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f6746d9d640 1 -- 172.21.15.148:0/3522757842 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67480cf160 msgr2=0x7f67480cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:04.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f6746d9d640 1 --2- 172.21.15.148:0/3522757842 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67480cf160 0x7f67480cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f6746d9d640 1 -- 172.21.15.148:0/3522757842 >> v1:172.21.15.150:6789/0 conn(0x7f67480ca0e0 legacy=0x7f674815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.464+0000 7f6746d9d640 1 -- 172.21.15.148:0/3522757842 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67480cad70 con 0x7f67481538a0 2023-10-30T13:45:04.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f6746d9d640 1 --2- 172.21.15.148:0/3522757842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67481538a0 0x7f674815d6f0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f6738009870 tx=0x7f673801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61410e8fe7434ffb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f6745d9b640 1 -- 172.21.15.148:0/3522757842 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67380a8a30 con 0x7f67481538a0 2023-10-30T13:45:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f6745d9b640 1 -- 172.21.15.148:0/3522757842 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67380a8b90 con 0x7f67481538a0 2023-10-30T13:45:04.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f6745d9b640 1 -- 172.21.15.148:0/3522757842 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67380b0440 con 0x7f67481538a0 2023-10-30T13:45:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f674d180640 1 -- 172.21.15.148:0/3522757842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67481538a0 msgr2=0x7f674815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:04.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f674d180640 1 --2- 172.21.15.148:0/3522757842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67481538a0 0x7f674815d6f0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f6738009870 tx=0x7f673801ed30 comp rx=0 tx=0).stop 2023-10-30T13:45:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f674d180640 1 -- 172.21.15.148:0/3522757842 shutdown_connections 2023-10-30T13:45:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f674d180640 1 --2- 172.21.15.148:0/3522757842 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67480cf160 0x7f67480cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f674d180640 1 --2- 172.21.15.148:0/3522757842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67481538a0 0x7f674815d6f0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f674d180640 1 -- 172.21.15.148:0/3522757842 >> 172.21.15.148:0/3522757842 conn(0x7f674806bfe0 msgr2=0x7f674806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:04.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f674d180640 1 -- 172.21.15.148:0/3522757842 shutdown_connections 2023-10-30T13:45:04.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.468+0000 7f674d180640 1 -- 172.21.15.148:0/3522757842 wait complete. 2023-10-30T13:45:04.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 Processor -- start 2023-10-30T13:45:04.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 -- start start 2023-10-30T13:45:04.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67480ca0e0 0x7f67481e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:04.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67480cf160 0x7f674815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:04.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67481538a0 0x7f674815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:04.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674814d4f0 con 0x7f67480ca0e0 2023-10-30T13:45:04.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67481e4d40 con 0x7f67480cf160 2023-10-30T13:45:04.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674815b2a0 con 0x7f67481538a0 2023-10-30T13:45:04.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f6746d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67480ca0e0 0x7f67481e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:04.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f6746d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67480ca0e0 0x7f67481e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57046/0 (socket says 172.21.15.148:57046) 2023-10-30T13:45:04.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f6746d9d640 1 -- 172.21.15.148:0/4048887416 learned_addr learned my addr 172.21.15.148:0/4048887416 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:04.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674659c640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67480cf160 0x7f674815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:04.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674759e640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67481538a0 0x7f674815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:04.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f6746d9d640 1 -- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67481538a0 msgr2=0x7f674815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:04.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f6746d9d640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67481538a0 0x7f674815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f6746d9d640 1 -- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67480cf160 msgr2=0x7f674815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:04.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f6746d9d640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67480cf160 0x7f674815b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f6746d9d640 1 -- 172.21.15.148:0/4048887416 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67380094d0 con 0x7f67480ca0e0 2023-10-30T13:45:04.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674759e640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67481538a0 0x7f674815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:04.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f6746d9d640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67480ca0e0 0x7f67481e4090 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f67380b3040 tx=0x7f673801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:04.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f672f7fe640 1 -- 172.21.15.148:0/4048887416 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67380ba600 con 0x7f67480ca0e0 2023-10-30T13:45:04.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f672f7fe640 1 -- 172.21.15.148:0/4048887416 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67380ba780 con 0x7f67480ca0e0 2023-10-30T13:45:04.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f672f7fe640 1 -- 172.21.15.148:0/4048887416 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67380bab30 con 0x7f67480ca0e0 2023-10-30T13:45:04.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67480cef10 con 0x7f67480ca0e0 2023-10-30T13:45:04.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67480c50b0 con 0x7f67480ca0e0 2023-10-30T13:45:04.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.472+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67480b2a50 con 0x7f67480ca0e0 2023-10-30T13:45:04.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.476+0000 7f672f7fe640 1 -- 172.21.15.148:0/4048887416 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67380bac90 con 0x7f67480ca0e0 2023-10-30T13:45:04.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.476+0000 7f672f7fe640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67340664a0 0x7f6734068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:04.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.476+0000 7f672f7fe640 1 -- 172.21.15.148:0/4048887416 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f67381229e0 con 0x7f67480ca0e0 2023-10-30T13:45:04.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.476+0000 7f674659c640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67340664a0 0x7f6734068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:04.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.476+0000 7f674659c640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67340664a0 0x7f6734068960 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f67480c4660 tx=0x7f673c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:04.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.480+0000 7f672f7fe640 1 -- 172.21.15.148:0/4048887416 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67380ef770 con 0x7f67480ca0e0 2023-10-30T13:45:04.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.656+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6748151940 con 0x7f67480ca0e0 2023-10-30T13:45:04.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f672f7fe640 1 -- 172.21.15.148:0/4048887416 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f67380b5090 con 0x7f67480ca0e0 2023-10-30T13:45:04.658 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:04.658 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:04.134933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 130 2023-10-30T13:45:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67340664a0 msgr2=0x7f6734068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67340664a0 0x7f6734068960 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f67480c4660 tx=0x7f673c009040 comp rx=0 tx=0).stop 2023-10-30T13:45:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67480ca0e0 msgr2=0x7f67481e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67480ca0e0 0x7f67481e4090 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f67380b3040 tx=0x7f673801bad0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 shutdown_connections 2023-10-30T13:45:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f67340664a0 0x7f6734068960 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f67481538a0 0x7f674815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67480cf160 0x7f674815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 --2- 172.21.15.148:0/4048887416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67480ca0e0 0x7f67481e4090 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 >> 172.21.15.148:0/4048887416 conn(0x7f674806bfe0 msgr2=0x7f6748065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 shutdown_connections 2023-10-30T13:45:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:04.660+0000 7f674d180640 1 -- 172.21.15.148:0/4048887416 wait complete. 2023-10-30T13:45:04.678 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=7305 2023-10-30T13:45:04.679 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2023-10-30T13:45:05.679 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:45:05.680 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:45:05.680 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:45:05.684 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:45:05.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:05.780 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:45:05.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f086a780640 1 Processor -- start 2023-10-30T13:45:05.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f086a780640 1 -- start start 2023-10-30T13:45:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f086a780640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f08641538a0 0x7f086415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f086a780640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 0x7f086415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f086a780640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f08640cf160 0x7f08640cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f086a780640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08640cb800 con 0x7f08640cf160 2023-10-30T13:45:05.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f086a780640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086415dc30 con 0x7f08641538a0 2023-10-30T13:45:05.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f086a780640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08640cb330 con 0x7f08640ca0e0 2023-10-30T13:45:05.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f0868d1d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f08640cf160 0x7f08640cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f08637fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 0x7f086415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f0868d1d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f08640cf160 0x7f08640cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57064/0 (socket says 172.21.15.148:57064) 2023-10-30T13:45:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f0863fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f08641538a0 0x7f086415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f0868d1d640 1 -- 172.21.15.148:0/3856946563 learned_addr learned my addr 172.21.15.148:0/3856946563 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f0868d1d640 1 -- 172.21.15.148:0/3856946563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 msgr2=0x7f086415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:05.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f0868d1d640 1 --2- 172.21.15.148:0/3856946563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 0x7f086415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f0868d1d640 1 -- 172.21.15.148:0/3856946563 >> v2:172.21.15.150:3300/0 conn(0x7f08641538a0 msgr2=0x7f086415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:05.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f0868d1d640 1 --2- 172.21.15.148:0/3856946563 >> v2:172.21.15.150:3300/0 conn(0x7f08641538a0 0x7f086415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.840+0000 7f0868d1d640 1 -- 172.21.15.148:0/3856946563 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08640cad70 con 0x7f08640cf160 2023-10-30T13:45:05.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f0868d1d640 1 --2- 172.21.15.148:0/3856946563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08640cf160 0x7f08640cf590 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f08640cfd10 tx=0x7f085c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13be6663236e9785 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:05.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f0862ffd640 1 -- 172.21.15.148:0/3856946563 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c0af400 con 0x7f08640cf160 2023-10-30T13:45:05.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f0862ffd640 1 -- 172.21.15.148:0/3856946563 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085c0ab050 con 0x7f08640cf160 2023-10-30T13:45:05.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f0862ffd640 1 -- 172.21.15.148:0/3856946563 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c0af780 con 0x7f08640cf160 2023-10-30T13:45:05.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f086a780640 1 -- 172.21.15.148:0/3856946563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08640cf160 msgr2=0x7f08640cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f086a780640 1 --2- 172.21.15.148:0/3856946563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08640cf160 0x7f08640cf590 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f08640cfd10 tx=0x7f085c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f086a780640 1 -- 172.21.15.148:0/3856946563 shutdown_connections 2023-10-30T13:45:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f086a780640 1 --2- 172.21.15.148:0/3856946563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08640cf160 0x7f08640cf590 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f086a780640 1 --2- 172.21.15.148:0/3856946563 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 0x7f086415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f086a780640 1 --2- 172.21.15.148:0/3856946563 >> v2:172.21.15.150:3300/0 conn(0x7f08641538a0 0x7f086415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f086a780640 1 -- 172.21.15.148:0/3856946563 >> 172.21.15.148:0/3856946563 conn(0x7f086406bfe0 msgr2=0x7f086406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f086a780640 1 -- 172.21.15.148:0/3856946563 shutdown_connections 2023-10-30T13:45:05.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.844+0000 7f086a780640 1 -- 172.21.15.148:0/3856946563 wait complete. 2023-10-30T13:45:05.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 Processor -- start 2023-10-30T13:45:05.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 -- start start 2023-10-30T13:45:05.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 0x7f08641f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08640cf160 0x7f08641f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08641538a0 0x7f08640b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086415aac0 con 0x7f08641538a0 2023-10-30T13:45:05.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08640cd9c0 con 0x7f08640cf160 2023-10-30T13:45:05.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08641f35f0 con 0x7f08640ca0e0 2023-10-30T13:45:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0868d1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08641538a0 0x7f08640b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0868d1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08641538a0 0x7f08640b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57076/0 (socket says 172.21.15.148:57076) 2023-10-30T13:45:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0868d1d640 1 -- 172.21.15.148:0/2486410381 learned_addr learned my addr 172.21.15.148:0/2486410381 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f08637fe640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08640cf160 0x7f08641f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0863fff640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 0x7f08641f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0868d1d640 1 -- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 msgr2=0x7f08641f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0868d1d640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 0x7f08641f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0868d1d640 1 -- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08640cf160 msgr2=0x7f08641f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:05.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0868d1d640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08640cf160 0x7f08641f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0868d1d640 1 -- 172.21.15.148:0/2486410381 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085c009b30 con 0x7f08641538a0 2023-10-30T13:45:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0868d1d640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08641538a0 0x7f08640b3440 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f085c0b2040 tx=0x7f085c01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0860ff9640 1 -- 172.21.15.148:0/2486410381 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c0afc60 con 0x7f08641538a0 2023-10-30T13:45:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0860ff9640 1 -- 172.21.15.148:0/2486410381 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085c0afde0 con 0x7f08641538a0 2023-10-30T13:45:05.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08640cdc80 con 0x7f08641538a0 2023-10-30T13:45:05.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f0860ff9640 1 -- 172.21.15.148:0/2486410381 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c0ba660 con 0x7f08641538a0 2023-10-30T13:45:05.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.848+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08640ce240 con 0x7f08641538a0 2023-10-30T13:45:05.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.852+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08640b2a50 con 0x7f08641538a0 2023-10-30T13:45:05.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.852+0000 7f0860ff9640 1 -- 172.21.15.148:0/2486410381 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f085c0b6040 con 0x7f08641538a0 2023-10-30T13:45:05.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.852+0000 7f0860ff9640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08500664d0 0x7f0850068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:05.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.852+0000 7f0860ff9640 1 -- 172.21.15.148:0/2486410381 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f085c122560 con 0x7f08641538a0 2023-10-30T13:45:05.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.852+0000 7f0863fff640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08500664d0 0x7f0850068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:05.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.852+0000 7f0863fff640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08500664d0 0x7f0850068990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0854009ac0 tx=0x7f0854009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:05.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:05.856+0000 7f0860ff9640 1 -- 172.21.15.148:0/2486410381 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f085c0ef3a0 con 0x7f08641538a0 2023-10-30T13:45:06.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.040+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08640ce760 con 0x7f08641538a0 2023-10-30T13:45:06.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.040+0000 7f0860ff9640 1 -- 172.21.15.148:0/2486410381 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f08640ce760 con 0x7f08641538a0 2023-10-30T13:45:06.041 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:06.041 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:04.134933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:06.041 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 130 2023-10-30T13:45:06.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08500664d0 msgr2=0x7f0850068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:06.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08500664d0 0x7f0850068990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0854009ac0 tx=0x7f0854009190 comp rx=0 tx=0).stop 2023-10-30T13:45:06.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08641538a0 msgr2=0x7f08640b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:06.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08641538a0 0x7f08640b3440 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f085c0b2040 tx=0x7f085c01bf60 comp rx=0 tx=0).stop 2023-10-30T13:45:06.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 shutdown_connections 2023-10-30T13:45:06.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f08500664d0 0x7f0850068990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:06.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f08641538a0 0x7f08640b3440 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:06.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08640cf160 0x7f08641f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:06.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 --2- 172.21.15.148:0/2486410381 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f08640ca0e0 0x7f08641f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:06.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 >> 172.21.15.148:0/2486410381 conn(0x7f086406bfe0 msgr2=0x7f0864065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:06.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 shutdown_connections 2023-10-30T13:45:06.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:06.044+0000 7f086a780640 1 -- 172.21.15.148:0/2486410381 wait complete. 2023-10-30T13:45:06.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:45:08.063 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:08.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c59365640 1 Processor -- start 2023-10-30T13:45:08.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c59365640 1 -- start start 2023-10-30T13:45:08.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c59365640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540c05d0 0x7f6c5413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:08.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c59365640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c540c0a50 con 0x7f6c540c6b80 2023-10-30T13:45:08.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c59365640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5406f6e0 con 0x7f6c540c6f60 2023-10-30T13:45:08.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c59365640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5406f9c0 con 0x7f6c540c05d0 2023-10-30T13:45:08.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c52ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6c540c6b80 0x7f6c540bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40402/0 (socket says 172.21.15.148:40402) 2023-10-30T13:45:08.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c52ffd640 1 -- 172.21.15.148:0/2830619318 learned_addr learned my addr 172.21.15.148:0/2830619318 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:45:08.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c4a7fc640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540c05d0 0x7f6c5413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:08.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526093952 0 0) 0x7f6c540c0a50 con 0x7f6c540c6b80 2023-10-30T13:45:08.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c40003530 con 0x7f6c540c6b80 2023-10-30T13:45:08.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485063443 0 0) 0x7f6c5406f6e0 con 0x7f6c540c6f60 2023-10-30T13:45:08.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c540c0a50 con 0x7f6c540c6f60 2023-10-30T13:45:08.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 916204944 0 0) 0x7f6c40003530 con 0x7f6c540c6b80 2023-10-30T13:45:08.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540c05d0 msgr2=0x7f6c5413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:08.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540c05d0 0x7f6c5413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:08.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 >> v1:172.21.15.150:6789/0 conn(0x7f6c540c6f60 legacy=0x7f6c541566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:08.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5406fca0 con 0x7f6c540c6b80 2023-10-30T13:45:08.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.212+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f6c44001680 con 0x7f6c540c6b80 2023-10-30T13:45:08.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 >> v1:172.21.15.148:6789/0 conn(0x7f6c540c6b80 legacy=0x7f6c540bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:08.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540c05d0 0x7f6c40003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:08.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c400040f0 0x7f6c400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:08.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c40004d10 0x7f6c40005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:08.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5406f9c0 con 0x7f6c40004d10 2023-10-30T13:45:08.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4000dce0 con 0x7f6c400040f0 2023-10-30T13:45:08.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4000dea0 con 0x7f6c540c05d0 2023-10-30T13:45:08.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c537fe640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c40004d10 0x7f6c40005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:08.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c4a7fc640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c400040f0 0x7f6c400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:08.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c52ffd640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540c05d0 0x7f6c40003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:08.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c537fe640 1 -- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540c05d0 msgr2=0x7f6c40003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:08.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c537fe640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540c05d0 0x7f6c40003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:08.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c537fe640 1 -- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c400040f0 msgr2=0x7f6c400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:08.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c537fe640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c400040f0 0x7f6c400047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:08.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c537fe640 1 -- 172.21.15.148:0/2830619318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c40003530 con 0x7f6c40004d10 2023-10-30T13:45:08.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c537fe640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c40004d10 0x7f6c40005190 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c001fd0 tx=0x7f6c4c01c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306da06898c68cae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:08.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c003b80 con 0x7f6c40004d10 2023-10-30T13:45:08.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c003d00 con 0x7f6c40004d10 2023-10-30T13:45:08.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.216+0000 7f6c527fc640 1 -- 172.21.15.148:0/2830619318 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c005310 con 0x7f6c40004d10 2023-10-30T13:45:08.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 -- 172.21.15.148:0/2830619318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c40004d10 msgr2=0x7f6c40005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:08.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 --2- 172.21.15.148:0/2830619318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c40004d10 0x7f6c40005190 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c001fd0 tx=0x7f6c4c01c820 comp rx=0 tx=0).stop 2023-10-30T13:45:08.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c537fe640 1 -- 172.21.15.148:0/2830619318 reap_dead start 2023-10-30T13:45:08.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 -- 172.21.15.148:0/2830619318 shutdown_connections 2023-10-30T13:45:08.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 -- 172.21.15.148:0/2830619318 >> 172.21.15.148:0/2830619318 conn(0x7f6c540c7910 msgr2=0x7f6c540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:08.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 -- 172.21.15.148:0/2830619318 shutdown_connections 2023-10-30T13:45:08.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 -- 172.21.15.148:0/2830619318 wait complete. 2023-10-30T13:45:08.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 Processor -- start 2023-10-30T13:45:08.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 -- start start 2023-10-30T13:45:08.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c54153a40 0x7f6c540c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:08.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c540c3610 0x7f6c540c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:08.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540ca2d0 0x7f6c541edc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:08.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540c24c0 con 0x7f6c54153a40 2023-10-30T13:45:08.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c541eaae0 con 0x7f6c540c3610 2023-10-30T13:45:08.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c59365640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c54155fc0 con 0x7f6c540ca2d0 2023-10-30T13:45:08.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c52ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c54153a40 0x7f6c540c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:08.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c52ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c54153a40 0x7f6c540c2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:57088/0 (socket says 172.21.15.148:57088) 2023-10-30T13:45:08.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c52ffd640 1 -- 172.21.15.148:0/2647465087 learned_addr learned my addr 172.21.15.148:0/2647465087 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:08.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c4a7fc640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c540c3610 0x7f6c540c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:08.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.220+0000 7f6c537fe640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540ca2d0 0x7f6c541edc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:08.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c52ffd640 1 -- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540ca2d0 msgr2=0x7f6c541edc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:08.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c52ffd640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540ca2d0 0x7f6c541edc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:08.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c52ffd640 1 -- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c540c3610 msgr2=0x7f6c540c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:08.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c52ffd640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c540c3610 0x7f6c540c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:08.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c52ffd640 1 -- 172.21.15.148:0/2647465087 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c541ea800 con 0x7f6c54153a40 2023-10-30T13:45:08.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c52ffd640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c54153a40 0x7f6c540c2ea0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6c44011b80 tx=0x7f6c4400f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:08.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c4bfff640 1 -- 172.21.15.148:0/2647465087 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44011e70 con 0x7f6c54153a40 2023-10-30T13:45:08.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c4bfff640 1 -- 172.21.15.148:0/2647465087 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4400fe60 con 0x7f6c54153a40 2023-10-30T13:45:08.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c54155690 con 0x7f6c54153a40 2023-10-30T13:45:08.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c4bfff640 1 -- 172.21.15.148:0/2647465087 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44015630 con 0x7f6c54153a40 2023-10-30T13:45:08.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c54155c50 con 0x7f6c54153a40 2023-10-30T13:45:08.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5404e170 con 0x7f6c54153a40 2023-10-30T13:45:08.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c4bfff640 1 -- 172.21.15.148:0/2647465087 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c44015790 con 0x7f6c54153a40 2023-10-30T13:45:08.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c4bfff640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c340664a0 0x7f6c34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:08.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c4bfff640 1 -- 172.21.15.148:0/2647465087 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f6c44087ad0 con 0x7f6c54153a40 2023-10-30T13:45:08.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.224+0000 7f6c4a7fc640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c340664a0 0x7f6c34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:08.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.228+0000 7f6c4a7fc640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c340664a0 0x7f6c34068960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6c540c1f90 tx=0x7f6c3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:08.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.228+0000 7f6c4bfff640 1 -- 172.21.15.148:0/2647465087 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c44054860 con 0x7f6c54153a40 2023-10-30T13:45:08.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.408+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c54138f50 con 0x7f6c54153a40 2023-10-30T13:45:08.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.408+0000 7f6c4bfff640 1 -- 172.21.15.148:0/2647465087 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6515 (secure 0 0 0) 0x7f6c44057e80 con 0x7f6c54153a40 2023-10-30T13:45:08.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:08.409 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:04.134933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:08.409 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 131 2023-10-30T13:45:08.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c340664a0 msgr2=0x7f6c34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:08.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c340664a0 0x7f6c34068960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6c540c1f90 tx=0x7f6c3c008040 comp rx=0 tx=0).stop 2023-10-30T13:45:08.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c54153a40 msgr2=0x7f6c540c2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:08.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c54153a40 0x7f6c540c2ea0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6c44011b80 tx=0x7f6c4400f540 comp rx=0 tx=0).stop 2023-10-30T13:45:08.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 shutdown_connections 2023-10-30T13:45:08.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c340664a0 0x7f6c34068960 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:08.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c540ca2d0 0x7f6c541edc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:08.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c540c3610 0x7f6c540c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:08.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 --2- 172.21.15.148:0/2647465087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c54153a40 0x7f6c540c2ea0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:08.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 >> 172.21.15.148:0/2647465087 conn(0x7f6c540c7910 msgr2=0x7f6c54069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:08.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 shutdown_connections 2023-10-30T13:45:08.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:08.412+0000 7f6c59365640 1 -- 172.21.15.148:0/2647465087 wait complete. 2023-10-30T13:45:08.430 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:45:08.430 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2023-10-30T13:45:13.102 INFO:tasks.ceph.mds.c.smithi148.stderr:2023-10-30T13:45:13.104+0000 7f226a61c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:45:15.083 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 73% (112/152) Updating files: 74% (113/152) Updating files: 75% (114/152) Updating files: 76% (116/152) Updating files: 77% (118/152) Updating files: 78% (119/152) Updating files: 79% (121/152) Updating files: 80% (122/152) Updating files: 81% (124/152) Updating files: 82% (125/152) Updating files: 83% (127/152) Updating files: 84% (128/152) Updating files: 85% (130/152) Updating files: 86% (131/152) Updating files: 87% (133/152) Updating files: 88% (134/152) Updating files: 89% (136/152) Updating files: 90% (137/152) Updating files: 91% (139/152) Updating files: 92% (140/152) Updating files: 93% (142/152) Updating files: 94% (143/152) Updating files: 95% (145/152) Updating files: 96% (146/152) Updating files: 97% (148/152) Updating files: 98% (149/152) Updating files: 99% (151/152) Updating files: 100% (152/152) Updating files: 100% (152/152), done. 2023-10-30T13:45:15.804 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at b6b48dbefad v0.66 2023-10-30T13:45:15.808 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.66 2023-10-30T13:45:15.856 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 67 -eq 48 ] 2023-10-30T13:45:15.857 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.67 2023-10-30T13:45:15.857 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.67 2023-10-30T13:45:15.857 INFO:tasks.workunit.client.0.smithi148.stdout:v0.67 2023-10-30T13:45:15.858 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.67 2023-10-30T13:45:27.752 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 39% (140/355) Updating files: 40% (142/355) Updating files: 41% (146/355) Updating files: 42% (150/355) Updating files: 43% (153/355) Updating files: 44% (157/355) Updating files: 45% (160/355) Updating files: 45% (163/355) Updating files: 46% (164/355) Updating files: 47% (167/355) Updating files: 48% (171/355) Updating files: 49% (174/355) Updating files: 50% (178/355) Updating files: 51% (182/355) Updating files: 52% (185/355) Updating files: 53% (189/355) Updating files: 54% (192/355) Updating files: 55% (196/355) Updating files: 56% (199/355) Updating files: 57% (203/355) Updating files: 58% (206/355) Updating files: 59% (210/355) Updating files: 60% (213/355) Updating files: 61% (217/355) Updating files: 62% (221/355) Updating files: 63% (224/355) Updating files: 64% (228/355) Updating files: 65% (231/355) Updating files: 66% (235/355) Updating files: 67% (238/355) Updating files: 68% (242/355) Updating files: 69% (245/355) Updating files: 70% (249/355) Updating files: 71% (253/355) Updating files: 72% (256/355) Updating files: 73% (260/355) Updating files: 74% (263/355) Updating files: 75% (267/355) Updating files: 76% (270/355) Updating files: 77% (274/355) Updating files: 78% (277/355) Updating files: 79% (281/355) Updating files: 79% (282/355) Updating files: 80% (284/355) Updating files: 81% (288/355) Updating files: 82% (292/355) Updating files: 83% (295/355) Updating files: 84% (299/355) Updating files: 85% (302/355) Updating files: 86% (306/355) Updating files: 87% (309/355) Updating files: 88% (313/355) Updating files: 89% (316/355) Updating files: 90% (320/355) Updating files: 91% (324/355) Updating files: 92% (327/355) Updating files: 93% (331/355) Updating files: 93% (333/355) Updating files: 94% (334/355) Updating files: 95% (338/355) Updating files: 96% (341/355) Updating files: 97% (345/355) Updating files: 98% (348/355) Updating files: 99% (352/355) Updating files: 100% (355/355) Updating files: 100% (355/355), done. 2023-10-30T13:45:28.493 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at e3b7bc5bce8 v0.67 2023-10-30T13:45:28.500 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.67 2023-10-30T13:45:28.548 INFO:tasks.workunit.client.0.smithi148.stdout:v0.68 2023-10-30T13:45:28.548 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 68 -eq 48 ] 2023-10-30T13:45:28.548 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.68 2023-10-30T13:45:28.549 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.68 2023-10-30T13:45:28.549 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.68 2023-10-30T13:45:31.431 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:31.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda0b6ea640 1 Processor -- start 2023-10-30T13:45:31.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda0b6ea640 1 -- start start 2023-10-30T13:45:31.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda0b6ea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fda040cf160 0x7fda040cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda0b6ea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040cb800 con 0x7fda041538a0 2023-10-30T13:45:31.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda0b6ea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040cb330 con 0x7fda0415d6f0 2023-10-30T13:45:31.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda0b6ea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda040cad70 con 0x7fda040cf160 2023-10-30T13:45:31.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda09c87640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fda040cf160 0x7fda040cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda09c87640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fda040cf160 0x7fda040cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45448/0 (socket says 172.21.15.148:45448) 2023-10-30T13:45:31.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda09c87640 1 -- 172.21.15.148:0/3288711200 learned_addr learned my addr 172.21.15.148:0/3288711200 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:31.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda09486640 1 --1- 172.21.15.148:0/3288711200 >> v1:172.21.15.148:6789/0 conn(0x7fda041538a0 0x7fda040ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48294/0 (socket says 172.21.15.148:48294) 2023-10-30T13:45:31.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda09c87640 1 -- 172.21.15.148:0/3288711200 >> v1:172.21.15.150:6789/0 conn(0x7fda0415d6f0 legacy=0x7fda0415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda09c87640 1 -- 172.21.15.148:0/3288711200 >> v1:172.21.15.148:6789/0 conn(0x7fda041538a0 legacy=0x7fda040ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fda09c87640 1 -- 172.21.15.148:0/3288711200 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda040cb050 con 0x7fda040cf160 2023-10-30T13:45:31.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.571+0000 7fd9fbfff640 1 -- 172.21.15.148:0/3288711200 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792749967 0 0) 0x7fda040cb800 con 0x7fda041538a0 2023-10-30T13:45:31.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fda09c87640 1 --2- 172.21.15.148:0/3288711200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda040cf160 0x7fda040cf590 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009e50 tx=0x7fd9fc01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53f018f62064d35 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:31.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fd9fbfff640 1 -- 172.21.15.148:0/3288711200 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc0af400 con 0x7fda040cf160 2023-10-30T13:45:31.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fd9fbfff640 1 -- 172.21.15.148:0/3288711200 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc0ab050 con 0x7fda040cf160 2023-10-30T13:45:31.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fd9fbfff640 1 -- 172.21.15.148:0/3288711200 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc0af780 con 0x7fda040cf160 2023-10-30T13:45:31.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3288711200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda040cf160 msgr2=0x7fda040cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fda0b6ea640 1 --2- 172.21.15.148:0/3288711200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda040cf160 0x7fda040cf590 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009e50 tx=0x7fd9fc01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3288711200 shutdown_connections 2023-10-30T13:45:31.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fda0b6ea640 1 --2- 172.21.15.148:0/3288711200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda040cf160 0x7fda040cf590 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3288711200 >> 172.21.15.148:0/3288711200 conn(0x7fda0406bfe0 msgr2=0x7fda0406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:31.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3288711200 shutdown_connections 2023-10-30T13:45:31.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3288711200 wait complete. 2023-10-30T13:45:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.575+0000 7fda0b6ea640 1 Processor -- start 2023-10-30T13:45:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 -- start start 2023-10-30T13:45:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda040cf160 0x7fda0415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda041538a0 0x7fda04140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda0415d6f0 0x7fda04141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda040ceb40 con 0x7fda041538a0 2023-10-30T13:45:31.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda04159210 con 0x7fda040cf160 2023-10-30T13:45:31.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda041594f0 con 0x7fda0415d6f0 2023-10-30T13:45:31.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda08c85640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda041538a0 0x7fda04140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda08c85640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda041538a0 0x7fda04140cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:45450/0 (socket says 172.21.15.148:45450) 2023-10-30T13:45:31.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda08c85640 1 -- 172.21.15.148:0/3605520617 learned_addr learned my addr 172.21.15.148:0/3605520617 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:31.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda09486640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda040cf160 0x7fda0415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda09c87640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda0415d6f0 0x7fda04141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda08c85640 1 -- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda0415d6f0 msgr2=0x7fda04141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda08c85640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda0415d6f0 0x7fda04141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda08c85640 1 -- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda040cf160 msgr2=0x7fda0415bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda08c85640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda040cf160 0x7fda0415bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda08c85640 1 -- 172.21.15.148:0/3605520617 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9fc009b30 con 0x7fda041538a0 2023-10-30T13:45:31.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda09c87640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda0415d6f0 0x7fda04141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:31.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda08c85640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda041538a0 0x7fda04140cd0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd9f400c3c0 tx=0x7fd9f400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:31.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fd9f9ffb640 1 -- 172.21.15.148:0/3605520617 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f40052d0 con 0x7fda041538a0 2023-10-30T13:45:31.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fd9f9ffb640 1 -- 172.21.15.148:0/3605520617 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f4005430 con 0x7fda041538a0 2023-10-30T13:45:31.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda040ced70 con 0x7fda041538a0 2023-10-30T13:45:31.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fd9f9ffb640 1 -- 172.21.15.148:0/3605520617 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f4003840 con 0x7fda041538a0 2023-10-30T13:45:31.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda041464e0 con 0x7fda041538a0 2023-10-30T13:45:31.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.579+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda04159660 con 0x7fda041538a0 2023-10-30T13:45:31.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.583+0000 7fd9f9ffb640 1 -- 172.21.15.148:0/3605520617 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd9f40039a0 con 0x7fda041538a0 2023-10-30T13:45:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.583+0000 7fd9f9ffb640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9e80664d0 0x7fd9e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.583+0000 7fd9f9ffb640 1 -- 172.21.15.148:0/3605520617 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fd9f4083880 con 0x7fda041538a0 2023-10-30T13:45:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.583+0000 7fda09486640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9e80664d0 0x7fd9e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.583+0000 7fda09486640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9e80664d0 0x7fd9e8068990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fda00000f70 tx=0x7fda00007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:31.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.587+0000 7fd9f9ffb640 1 -- 172.21.15.148:0/3605520617 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd9f40507f0 con 0x7fda041538a0 2023-10-30T13:45:31.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.771+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda0404e170 con 0x7fda041538a0 2023-10-30T13:45:31.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.771+0000 7fd9f9ffb640 1 -- 172.21.15.148:0/3605520617 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6515 (secure 0 0 0) 0x7fd9f4053e10 con 0x7fda041538a0 2023-10-30T13:45:31.774 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:31.774 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:04.134933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7464},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:31.774 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 131 2023-10-30T13:45:31.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9e80664d0 msgr2=0x7fd9e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9e80664d0 0x7fd9e8068990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fda00000f70 tx=0x7fda00007240 comp rx=0 tx=0).stop 2023-10-30T13:45:31.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda041538a0 msgr2=0x7fda04140cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.776 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda041538a0 0x7fda04140cd0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd9f400c3c0 tx=0x7fd9f400c880 comp rx=0 tx=0).stop 2023-10-30T13:45:31.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 shutdown_connections 2023-10-30T13:45:31.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9e80664d0 0x7fd9e8068990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda041538a0 0x7fda04140cd0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fda0415d6f0 0x7fda04141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 --2- 172.21.15.148:0/3605520617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda040cf160 0x7fda0415bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 >> 172.21.15.148:0/3605520617 conn(0x7fda0406bfe0 msgr2=0x7fda04151b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:31.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 shutdown_connections 2023-10-30T13:45:31.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.775+0000 7fda0b6ea640 1 -- 172.21.15.148:0/3605520617 wait complete. 2023-10-30T13:45:31.795 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 1 -> 2 2023-10-30T13:45:31.795 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2023-10-30T13:45:31.937 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 78% (190/241) Updating files: 79% (191/241) Updating files: 80% (193/241) Updating files: 81% (196/241) Updating files: 82% (198/241) Updating files: 83% (201/241) Updating files: 84% (203/241) Updating files: 85% (205/241) Updating files: 86% (208/241) Updating files: 87% (210/241) Updating files: 87% (211/241) Updating files: 88% (213/241) Updating files: 89% (215/241) Updating files: 90% (217/241) Updating files: 91% (220/241) Updating files: 92% (222/241) Updating files: 93% (225/241) Updating files: 94% (227/241) Updating files: 95% (229/241) Updating files: 96% (232/241) Updating files: 97% (234/241) Updating files: 98% (237/241) Updating files: 99% (239/241) Updating files: 100% (241/241) Updating files: 100% (241/241), done. 2023-10-30T13:45:31.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.939+0000 7fae8f3d3640 1 Processor -- start 2023-10-30T13:45:31.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.939+0000 7fae8f3d3640 1 -- start start 2023-10-30T13:45:31.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8f3d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fae88153c90 0x7fae880be770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8f3d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fae8806b540 0x7fae8806b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8f3d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 0x7fae88159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8f3d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae88070850 con 0x7fae88153c90 2023-10-30T13:45:31.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8f3d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8806e8c0 con 0x7fae8806b540 2023-10-30T13:45:31.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8f3d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae880becb0 con 0x7fae881597a0 2023-10-30T13:45:31.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8d16f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fae88153c90 0x7fae880be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8d16f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fae88153c90 0x7fae880be770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49472/0 (socket says 172.21.15.148:49472) 2023-10-30T13:45:31.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8d16f640 1 -- 172.21.15.148:0/2928059884 learned_addr learned my addr 172.21.15.148:0/2928059884 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:31.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8c96e640 1 --2- 172.21.15.148:0/2928059884 >> v2:172.21.15.150:3300/0 conn(0x7fae8806b540 0x7fae8806b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8d970640 1 --2- 172.21.15.148:0/2928059884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 0x7fae88159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8d16f640 1 -- 172.21.15.148:0/2928059884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 msgr2=0x7fae88159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:31.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8d16f640 1 --2- 172.21.15.148:0/2928059884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 0x7fae88159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8d16f640 1 -- 172.21.15.148:0/2928059884 >> v2:172.21.15.150:3300/0 conn(0x7fae8806b540 msgr2=0x7fae8806b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:31.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8d16f640 1 --2- 172.21.15.148:0/2928059884 >> v2:172.21.15.150:3300/0 conn(0x7fae8806b540 0x7fae8806b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.943+0000 7fae8d16f640 1 -- 172.21.15.148:0/2928059884 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae880be140 con 0x7fae88153c90 2023-10-30T13:45:31.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae8d16f640 1 --2- 172.21.15.148:0/2928059884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae88153c90 0x7fae880be770 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fae8806f970 tx=0x7fae8401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3fd7de6beea2ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae7ffff640 1 -- 172.21.15.148:0/2928059884 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae840a89b0 con 0x7fae88153c90 2023-10-30T13:45:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae7ffff640 1 -- 172.21.15.148:0/2928059884 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae840a8b10 con 0x7fae88153c90 2023-10-30T13:45:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae7ffff640 1 -- 172.21.15.148:0/2928059884 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae840a8ea0 con 0x7fae88153c90 2023-10-30T13:45:31.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae8f3d3640 1 -- 172.21.15.148:0/2928059884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae88153c90 msgr2=0x7fae880be770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/2928059884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae88153c90 0x7fae880be770 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fae8806f970 tx=0x7fae8401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae8f3d3640 1 -- 172.21.15.148:0/2928059884 shutdown_connections 2023-10-30T13:45:31.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/2928059884 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 0x7fae88159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/2928059884 >> v2:172.21.15.150:3300/0 conn(0x7fae8806b540 0x7fae8806b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/2928059884 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae88153c90 0x7fae880be770 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.947+0000 7fae8f3d3640 1 -- 172.21.15.148:0/2928059884 >> 172.21.15.148:0/2928059884 conn(0x7fae880c7fe0 msgr2=0x7fae880c83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:31.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 -- 172.21.15.148:0/2928059884 shutdown_connections 2023-10-30T13:45:31.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 -- 172.21.15.148:0/2928059884 wait complete. 2023-10-30T13:45:31.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 Processor -- start 2023-10-30T13:45:31.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 -- start start 2023-10-30T13:45:31.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8806b540 0x7fae881f2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae88153c90 0x7fae881f0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 0x7fae881f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae881f31b0 con 0x7fae8806b540 2023-10-30T13:45:31.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae881f2ed0 con 0x7fae88153c90 2023-10-30T13:45:31.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8f3d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae880c1a10 con 0x7fae881597a0 2023-10-30T13:45:31.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8d16f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8806b540 0x7fae881f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8d16f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8806b540 0x7fae881f2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49476/0 (socket says 172.21.15.148:49476) 2023-10-30T13:45:31.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8d16f640 1 -- 172.21.15.148:0/1695128357 learned_addr learned my addr 172.21.15.148:0/1695128357 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:31.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8c96e640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae88153c90 0x7fae881f0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.951+0000 7fae8d970640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 0x7fae881f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae8d16f640 1 -- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 msgr2=0x7fae881f0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae8d16f640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 0x7fae881f0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae8d16f640 1 -- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae88153c90 msgr2=0x7fae881f0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:31.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae8d16f640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae88153c90 0x7fae881f0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:31.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae8d16f640 1 -- 172.21.15.148:0/1695128357 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae840094d0 con 0x7fae8806b540 2023-10-30T13:45:31.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae8d16f640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8806b540 0x7fae881f2490 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fae8401d3a0 tx=0x7fae8401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:31.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae7dffb640 1 -- 172.21.15.148:0/1695128357 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae840b0d40 con 0x7fae8806b540 2023-10-30T13:45:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae7dffb640 1 -- 172.21.15.148:0/1695128357 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae840bb460 con 0x7fae8806b540 2023-10-30T13:45:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae880c1cf0 con 0x7fae8806b540 2023-10-30T13:45:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae7dffb640 1 -- 172.21.15.148:0/1695128357 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae840b0ea0 con 0x7fae8806b540 2023-10-30T13:45:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae88157b70 con 0x7fae8806b540 2023-10-30T13:45:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae880c1870 con 0x7fae8806b540 2023-10-30T13:45:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae7dffb640 1 -- 172.21.15.148:0/1695128357 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae840b83f0 con 0x7fae8806b540 2023-10-30T13:45:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.955+0000 7fae7dffb640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae6c0664d0 0x7fae6c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.959+0000 7fae7dffb640 1 -- 172.21.15.148:0/1695128357 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fae84121e60 con 0x7fae8806b540 2023-10-30T13:45:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.959+0000 7fae8c96e640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae6c0664d0 0x7fae6c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.959+0000 7fae8c96e640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae6c0664d0 0x7fae6c068990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fae88152960 tx=0x7fae78009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:31.963+0000 7fae7dffb640 1 -- 172.21.15.148:0/1695128357 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae880c1870 con 0x7fae8806b540 2023-10-30T13:45:31.986 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at b4cf0f2574b v0.68 2023-10-30T13:45:31.995 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.68 2023-10-30T13:45:32.146 INFO:tasks.workunit.client.0.smithi148.stdout:v0.69 2023-10-30T13:45:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 69 -eq 48 ] 2023-10-30T13:45:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.69 2023-10-30T13:45:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.69 2023-10-30T13:45:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.69 2023-10-30T13:45:32.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:32.219+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fae880c1870 con 0x7fae8806b540 2023-10-30T13:45:32.612 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:32.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:45:33.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.135+0000 7fae7dffb640 1 -- 172.21.15.148:0/1695128357 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v132) v1 ==== 105+0+0 (secure 0 0 0) 0x7fae880c1870 con 0x7fae8806b540 2023-10-30T13:45:33.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae6c0664d0 msgr2=0x7fae6c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:33.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae6c0664d0 0x7fae6c068990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fae88152960 tx=0x7fae78009190 comp rx=0 tx=0).stop 2023-10-30T13:45:33.140 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8806b540 msgr2=0x7fae881f2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:33.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8806b540 0x7fae881f2490 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fae8401d3a0 tx=0x7fae8401ba90 comp rx=0 tx=0).stop 2023-10-30T13:45:33.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 shutdown_connections 2023-10-30T13:45:33.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fae6c0664d0 0x7fae6c068990 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fae881597a0 0x7fae881f0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae88153c90 0x7fae881f0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 --2- 172.21.15.148:0/1695128357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fae8806b540 0x7fae881f2490 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 >> 172.21.15.148:0/1695128357 conn(0x7fae880c7fe0 msgr2=0x7fae8806cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:33.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 shutdown_connections 2023-10-30T13:45:33.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.139+0000 7fae8f3d3640 1 -- 172.21.15.148:0/1695128357 wait complete. 2023-10-30T13:45:33.160 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:45:33.160 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:33.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc5273e640 1 Processor -- start 2023-10-30T13:45:33.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc5273e640 1 -- start start 2023-10-30T13:45:33.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc5273e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdc4c1538a0 0x7fdc4c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:33.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc5273e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdc4c0cf160 0x7fdc4c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc5273e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c0cb800 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc5273e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4c15dc30 con 0x7fdc4c0cf160 2023-10-30T13:45:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc5273e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4c0cb330 con 0x7fdc4c1538a0 2023-10-30T13:45:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc50cdb640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdc4c0cf160 0x7fdc4c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc4b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fdc4c0ca0e0 0x7fdc4c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56178/0 (socket says 172.21.15.148:56178) 2023-10-30T13:45:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc4b7fe640 1 -- 172.21.15.148:0/942890808 learned_addr learned my addr 172.21.15.148:0/942890808 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:45:33.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc4bfff640 1 --2- 172.21.15.148:0/942890808 >> v2:172.21.15.150:3300/0 conn(0x7fdc4c1538a0 0x7fdc4c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:33.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc50cdb640 1 -- 172.21.15.148:0/942890808 >> v2:172.21.15.150:3300/0 conn(0x7fdc4c1538a0 msgr2=0x7fdc4c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:33.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc50cdb640 1 --2- 172.21.15.148:0/942890808 >> v2:172.21.15.150:3300/0 conn(0x7fdc4c1538a0 0x7fdc4c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc50cdb640 1 -- 172.21.15.148:0/942890808 >> v1:172.21.15.148:6789/0 conn(0x7fdc4c0ca0e0 legacy=0x7fdc4c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:33.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.299+0000 7fdc50cdb640 1 -- 172.21.15.148:0/942890808 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c0cad70 con 0x7fdc4c0cf160 2023-10-30T13:45:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc50cdb640 1 --2- 172.21.15.148:0/942890808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0cf160 0x7fdc4c0cf590 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0cfd10 tx=0x7fdc4401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef71ccf672ce3e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:33.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc4affd640 1 -- 172.21.15.148:0/942890808 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc440b0440 con 0x7fdc4c0cf160 2023-10-30T13:45:33.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc4affd640 1 -- 172.21.15.148:0/942890808 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc440b05a0 con 0x7fdc4c0cf160 2023-10-30T13:45:33.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc4affd640 1 -- 172.21.15.148:0/942890808 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc440b0930 con 0x7fdc4c0cf160 2023-10-30T13:45:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc5273e640 1 -- 172.21.15.148:0/942890808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0cf160 msgr2=0x7fdc4c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc5273e640 1 --2- 172.21.15.148:0/942890808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0cf160 0x7fdc4c0cf590 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0cfd10 tx=0x7fdc4401f0f0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc5273e640 1 -- 172.21.15.148:0/942890808 shutdown_connections 2023-10-30T13:45:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc5273e640 1 --2- 172.21.15.148:0/942890808 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0cf160 0x7fdc4c0cf590 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc5273e640 1 --2- 172.21.15.148:0/942890808 >> v2:172.21.15.150:3300/0 conn(0x7fdc4c1538a0 0x7fdc4c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc5273e640 1 -- 172.21.15.148:0/942890808 >> 172.21.15.148:0/942890808 conn(0x7fdc4c06bfe0 msgr2=0x7fdc4c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:33.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc5273e640 1 -- 172.21.15.148:0/942890808 shutdown_connections 2023-10-30T13:45:33.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.303+0000 7fdc5273e640 1 -- 172.21.15.148:0/942890808 wait complete. 2023-10-30T13:45:33.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 Processor -- start 2023-10-30T13:45:33.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 -- start start 2023-10-30T13:45:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0ca0e0 0x7fdc4c1480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c0cf160 0x7fdc4c157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc4c1538a0 0x7fdc4c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c1444d0 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c0b4850 con 0x7fdc4c0cf160 2023-10-30T13:45:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c0b4b10 con 0x7fdc4c1538a0 2023-10-30T13:45:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0ca0e0 0x7fdc4c1480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0ca0e0 0x7fdc4c1480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49484/0 (socket says 172.21.15.148:49484) 2023-10-30T13:45:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4bfff640 1 -- 172.21.15.148:0/2089775002 learned_addr learned my addr 172.21.15.148:0/2089775002 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:33.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4b7fe640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c0cf160 0x7fdc4c157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc50cdb640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc4c1538a0 0x7fdc4c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4bfff640 1 -- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc4c1538a0 msgr2=0x7fdc4c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4bfff640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc4c1538a0 0x7fdc4c158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4bfff640 1 -- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c0cf160 msgr2=0x7fdc4c157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4bfff640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c0cf160 0x7fdc4c157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4bfff640 1 -- 172.21.15.148:0/2089775002 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc44009a70 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4b7fe640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c0cf160 0x7fdc4c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:33.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc50cdb640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc4c1538a0 0x7fdc4c158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:33.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc4bfff640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0ca0e0 0x7fdc4c1480f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00d800 tx=0x7fdc3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:33.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc48ff9640 1 -- 172.21.15.148:0/2089775002 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00b6c0 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc48ff9640 1 -- 172.21.15.148:0/2089775002 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c00b820 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c0b4df0 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc48ff9640 1 -- 172.21.15.148:0/2089775002 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00bb90 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c1e49d0 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.307+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4c0b2a50 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.311+0000 7fdc48ff9640 1 -- 172.21.15.148:0/2089775002 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdc3c00bcf0 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.311+0000 7fdc48ff9640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc380664a0 0x7fdc38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:33.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.311+0000 7fdc48ff9640 1 -- 172.21.15.148:0/2089775002 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fdc3c084a60 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.311+0000 7fdc4b7fe640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc380664a0 0x7fdc38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:33.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.311+0000 7fdc4b7fe640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc380664a0 0x7fdc38068960 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdc40007fd0 tx=0x7fdc40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:33.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.315+0000 7fdc48ff9640 1 -- 172.21.15.148:0/2089775002 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdc3c051890 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.511+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc4c159b40 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.511+0000 7fdc48ff9640 1 -- 172.21.15.148:0/2089775002 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6529 (secure 0 0 0) 0x7fdc3c054eb0 con 0x7fdc4c0ca0e0 2023-10-30T13:45:33.512 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:33.512 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:33.143888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:starting","state_seq":1,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:33.513 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 133 2023-10-30T13:45:33.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc380664a0 msgr2=0x7fdc38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:33.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc380664a0 0x7fdc38068960 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdc40007fd0 tx=0x7fdc40009040 comp rx=0 tx=0).stop 2023-10-30T13:45:33.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0ca0e0 msgr2=0x7fdc4c1480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:33.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0ca0e0 0x7fdc4c1480f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00d800 tx=0x7fdc3c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 shutdown_connections 2023-10-30T13:45:33.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc380664a0 0x7fdc38068960 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc4c1538a0 0x7fdc4c158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc4c0cf160 0x7fdc4c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 --2- 172.21.15.148:0/2089775002 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc4c0ca0e0 0x7fdc4c1480f0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:33.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 >> 172.21.15.148:0/2089775002 conn(0x7fdc4c06bfe0 msgr2=0x7fdc4c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:33.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 shutdown_connections 2023-10-30T13:45:33.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:33.515+0000 7fdc5273e640 1 -- 172.21.15.148:0/2089775002 wait complete. 2023-10-30T13:45:33.542 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 133, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 133, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7464": { "addr": "172.21.15.148:6837/3703944646", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 3703944646, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 3703944646, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7464, "incarnation": 126, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 32 }, "gid_7611": { "addr": "172.21.15.148:6835/1228827025", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1228827025, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1228827025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7611, "incarnation": 133, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:starting", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:45:33.143888+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7464, "mds_1": 7611 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 63, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:45:34.963 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 32% (51/155) Updating files: 33% (52/155) Updating files: 34% (53/155) Updating files: 35% (55/155) Updating files: 36% (56/155) Updating files: 37% (58/155) Updating files: 38% (59/155) Updating files: 39% (61/155) Updating files: 40% (62/155) Updating files: 41% (64/155) Updating files: 42% (66/155) Updating files: 43% (67/155) Updating files: 44% (69/155) Updating files: 45% (70/155) Updating files: 46% (72/155) Updating files: 47% (73/155) Updating files: 48% (75/155) Updating files: 49% (76/155) Updating files: 50% (78/155) Updating files: 51% (80/155) Updating files: 52% (81/155) Updating files: 53% (83/155) Updating files: 54% (84/155) Updating files: 55% (86/155) Updating files: 56% (87/155) Updating files: 57% (89/155) Updating files: 58% (90/155) Updating files: 59% (92/155) Updating files: 60% (93/155) Updating files: 61% (95/155) Updating files: 62% (97/155) Updating files: 63% (98/155) Updating files: 64% (100/155) Updating files: 65% (101/155) Updating files: 66% (103/155) Updating files: 67% (104/155) Updating files: 68% (106/155) Updating files: 69% (107/155) Updating files: 70% (109/155) Updating files: 71% (111/155) Updating files: 72% (112/155) Updating files: 73% (114/155) Updating files: 74% (115/155) Updating files: 75% (117/155) Updating files: 76% (118/155) Updating files: 77% (120/155) Updating files: 78% (121/155) Updating files: 79% (123/155) Updating files: 80% (124/155) Updating files: 81% (126/155) Updating files: 82% (128/155) Updating files: 83% (129/155) Updating files: 84% (131/155) Updating files: 85% (132/155) Updating files: 86% (134/155) Updating files: 87% (135/155) Updating files: 88% (137/155) Updating files: 89% (138/155) Updating files: 90% (140/155) Updating files: 91% (142/155) Updating files: 92% (143/155) Updating files: 93% (145/155) Updating files: 94% (146/155) Updating files: 95% (148/155) Updating files: 96% (149/155) Updating files: 97% (151/155) Updating files: 98% (152/155) Updating files: 99% (154/155) Updating files: 100% (155/155) Updating files: 100% (155/155), done. 2023-10-30T13:45:35.009 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 6ca6f2f9f75 v0.69 2023-10-30T13:45:35.015 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.69 2023-10-30T13:45:35.145 INFO:tasks.workunit.client.0.smithi148.stdout:v0.70 2023-10-30T13:45:35.145 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 70 -eq 48 ] 2023-10-30T13:45:35.145 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.70 2023-10-30T13:45:35.146 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.70 2023-10-30T13:45:35.146 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.70 2023-10-30T13:45:35.543 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:35.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.679+0000 7ff277646640 1 Processor -- start 2023-10-30T13:45:35.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.679+0000 7ff277646640 1 -- start start 2023-10-30T13:45:35.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff277646640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff2701539a0 0x7ff27015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff277646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff270070a00 0x7ff270070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff277646640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff270069e90 0x7ff27006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff277646640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27006fbb0 con 0x7ff270069e90 2023-10-30T13:45:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff277646640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27015a8c0 con 0x7ff2701539a0 2023-10-30T13:45:35.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff277646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff270071380 con 0x7ff270070a00 2023-10-30T13:45:35.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff275be3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff270069e90 0x7ff27006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:35.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff275be3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff270069e90 0x7ff27006a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49500/0 (socket says 172.21.15.148:49500) 2023-10-30T13:45:35.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff275be3640 1 -- 172.21.15.148:0/1577287077 learned_addr learned my addr 172.21.15.148:0/1577287077 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:35.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff274be1640 1 --2- 172.21.15.148:0/1577287077 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff270070a00 0x7ff270070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:35.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff2753e2640 1 --2- 172.21.15.148:0/1577287077 >> v2:172.21.15.150:3300/0 conn(0x7ff2701539a0 0x7ff27015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:35.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff275be3640 1 -- 172.21.15.148:0/1577287077 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff270070a00 msgr2=0x7ff270070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:35.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff275be3640 1 --2- 172.21.15.148:0/1577287077 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff270070a00 0x7ff270070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff275be3640 1 -- 172.21.15.148:0/1577287077 >> v2:172.21.15.150:3300/0 conn(0x7ff2701539a0 msgr2=0x7ff27015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:35.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff275be3640 1 --2- 172.21.15.148:0/1577287077 >> v2:172.21.15.150:3300/0 conn(0x7ff2701539a0 0x7ff27015a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff275be3640 1 -- 172.21.15.148:0/1577287077 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff270153d80 con 0x7ff270069e90 2023-10-30T13:45:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.683+0000 7ff275be3640 1 --2- 172.21.15.148:0/1577287077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 0x7ff27006a300 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7ff2700c8950 tx=0x7ff26801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8443254c97b27d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff267fff640 1 -- 172.21.15.148:0/1577287077 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2680a56a0 con 0x7ff270069e90 2023-10-30T13:45:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff267fff640 1 -- 172.21.15.148:0/1577287077 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2680a5800 con 0x7ff270069e90 2023-10-30T13:45:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff267fff640 1 -- 172.21.15.148:0/1577287077 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2680a5b90 con 0x7ff270069e90 2023-10-30T13:45:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 -- 172.21.15.148:0/1577287077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 msgr2=0x7ff27006a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 --2- 172.21.15.148:0/1577287077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 0x7ff27006a300 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7ff2700c8950 tx=0x7ff26801c910 comp rx=0 tx=0).stop 2023-10-30T13:45:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 -- 172.21.15.148:0/1577287077 shutdown_connections 2023-10-30T13:45:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 --2- 172.21.15.148:0/1577287077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 0x7ff27006a300 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 --2- 172.21.15.148:0/1577287077 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff270070a00 0x7ff270070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 --2- 172.21.15.148:0/1577287077 >> v2:172.21.15.150:3300/0 conn(0x7ff2701539a0 0x7ff27015a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 -- 172.21.15.148:0/1577287077 >> 172.21.15.148:0/1577287077 conn(0x7ff2700c7910 msgr2=0x7ff2700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:35.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 -- 172.21.15.148:0/1577287077 shutdown_connections 2023-10-30T13:45:35.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 -- 172.21.15.148:0/1577287077 wait complete. 2023-10-30T13:45:35.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 Processor -- start 2023-10-30T13:45:35.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 -- start start 2023-10-30T13:45:35.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 0x7ff27006b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:35.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff270070a00 0x7ff2700c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:35.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2701539a0 0x7ff2700c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:35.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27015cb80 con 0x7ff270069e90 2023-10-30T13:45:35.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27015b480 con 0x7ff270070a00 2023-10-30T13:45:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.687+0000 7ff277646640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700c3010 con 0x7ff2701539a0 2023-10-30T13:45:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff2753e2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 0x7ff27006b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff2753e2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 0x7ff27006b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49504/0 (socket says 172.21.15.148:49504) 2023-10-30T13:45:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff2753e2640 1 -- 172.21.15.148:0/1501066673 learned_addr learned my addr 172.21.15.148:0/1501066673 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff274be1640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff270070a00 0x7ff2700c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:35.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff275be3640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2701539a0 0x7ff2700c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:35.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff2753e2640 1 -- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2701539a0 msgr2=0x7ff2700c40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:35.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff2753e2640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2701539a0 0x7ff2700c40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff2753e2640 1 -- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff270070a00 msgr2=0x7ff2700c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:35.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff2753e2640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff270070a00 0x7ff2700c3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff2753e2640 1 -- 172.21.15.148:0/1501066673 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268009b30 con 0x7ff270069e90 2023-10-30T13:45:35.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff274be1640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff270070a00 0x7ff2700c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:35.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff2753e2640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 0x7ff27006b780 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00d830 tx=0x7ff26c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:35.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff265ffb640 1 -- 172.21.15.148:0/1501066673 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c00b660 con 0x7ff270069e90 2023-10-30T13:45:35.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff265ffb640 1 -- 172.21.15.148:0/1501066673 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c00b7c0 con 0x7ff270069e90 2023-10-30T13:45:35.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff265ffb640 1 -- 172.21.15.148:0/1501066673 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c00bb30 con 0x7ff270069e90 2023-10-30T13:45:35.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2700b0e30 con 0x7ff270069e90 2023-10-30T13:45:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27015b950 con 0x7ff270069e90 2023-10-30T13:45:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27004e170 con 0x7ff270069e90 2023-10-30T13:45:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.691+0000 7ff265ffb640 1 -- 172.21.15.148:0/1501066673 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff26c01e460 con 0x7ff270069e90 2023-10-30T13:45:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.695+0000 7ff265ffb640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2540664a0 0x7ff254068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.695+0000 7ff265ffb640 1 -- 172.21.15.148:0/1501066673 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7ff26c084bb0 con 0x7ff270069e90 2023-10-30T13:45:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.695+0000 7ff274be1640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2540664a0 0x7ff254068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.695+0000 7ff274be1640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2540664a0 0x7ff254068960 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff260004660 tx=0x7ff260008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.699+0000 7ff265ffb640 1 -- 172.21.15.148:0/1501066673 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff26c051940 con 0x7ff270069e90 2023-10-30T13:45:35.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2700c72f0 con 0x7ff270069e90 2023-10-30T13:45:35.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff265ffb640 1 -- 172.21.15.148:0/1501066673 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6529 (secure 0 0 0) 0x7ff26c054f60 con 0x7ff270069e90 2023-10-30T13:45:35.884 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:35.884 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:33.143888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:starting","state_seq":1,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:35.884 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 133 2023-10-30T13:45:35.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2540664a0 msgr2=0x7ff254068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:35.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2540664a0 0x7ff254068960 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff260004660 tx=0x7ff260008040 comp rx=0 tx=0).stop 2023-10-30T13:45:35.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 msgr2=0x7ff27006b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:35.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 0x7ff27006b780 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00d830 tx=0x7ff26c00dcf0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 shutdown_connections 2023-10-30T13:45:35.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff2540664a0 0x7ff254068960 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff2701539a0 0x7ff2700c40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff270070a00 0x7ff2700c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 --2- 172.21.15.148:0/1501066673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff270069e90 0x7ff27006b780 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:35.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 >> 172.21.15.148:0/1501066673 conn(0x7ff2700c7910 msgr2=0x7ff270073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:35.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 shutdown_connections 2023-10-30T13:45:35.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:35.883+0000 7ff277646640 1 -- 172.21.15.148:0/1501066673 wait complete. 2023-10-30T13:45:35.904 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:45:36.987 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 9% (4/44) Updating files: 11% (5/44) Updating files: 13% (6/44) Updating files: 15% (7/44) Updating files: 18% (8/44) Updating files: 20% (9/44) Updating files: 22% (10/44) Updating files: 25% (11/44) Updating files: 27% (12/44) Updating files: 29% (13/44) Updating files: 31% (14/44) Updating files: 34% (15/44) Updating files: 36% (16/44) Updating files: 38% (17/44) Updating files: 40% (18/44) Updating files: 43% (19/44) Updating files: 45% (20/44) Updating files: 47% (21/44) Updating files: 50% (22/44) Updating files: 52% (23/44) Updating files: 54% (24/44) Updating files: 56% (25/44) Updating files: 59% (26/44) Updating files: 61% (27/44) Updating files: 63% (28/44) Updating files: 65% (29/44) Updating files: 68% (30/44) Updating files: 70% (31/44) Updating files: 72% (32/44) Updating files: 75% (33/44) Updating files: 77% (34/44) Updating files: 79% (35/44) Updating files: 81% (36/44) Updating files: 84% (37/44) Updating files: 86% (38/44) Updating files: 88% (39/44) Updating files: 90% (40/44) Updating files: 93% (41/44) Updating files: 95% (42/44) Updating files: 97% (43/44) Updating files: 100% (44/44) Updating files: 100% (44/44), done. 2023-10-30T13:45:37.047 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at e3bb0656d92 v0.70 2023-10-30T13:45:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.70 2023-10-30T13:45:37.121 INFO:tasks.workunit.client.0.smithi148.stdout:v0.71 2023-10-30T13:45:37.121 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 71 -eq 48 ] 2023-10-30T13:45:37.121 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.71 2023-10-30T13:45:37.121 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.71 2023-10-30T13:45:37.122 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.71 2023-10-30T13:45:37.906 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:38.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.043+0000 7f7f8f59e640 1 Processor -- start 2023-10-30T13:45:38.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8f59e640 1 -- start start 2023-10-30T13:45:38.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8f59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f9015cd50 0x7f7f900c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8f59e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7f900cefa0 0x7f7f900cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8f59e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9015a6c0 con 0x7f7f901538f0 2023-10-30T13:45:38.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8f59e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f9015d300 con 0x7f7f900cefa0 2023-10-30T13:45:38.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8f59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9015d5e0 con 0x7f7f9015cd50 2023-10-30T13:45:38.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8ed9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7f900cefa0 0x7f7f900cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8ed9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7f900cefa0 0x7f7f900cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49520/0 (socket says 172.21.15.148:49520) 2023-10-30T13:45:38.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8ed9d640 1 -- 172.21.15.148:0/3729365595 learned_addr learned my addr 172.21.15.148:0/3729365595 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:38.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8dd9b640 1 --2- 172.21.15.148:0/3729365595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f9015cd50 0x7f7f900c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8ed9d640 1 -- 172.21.15.148:0/3729365595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f9015cd50 msgr2=0x7f7f900c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:38.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8ed9d640 1 --2- 172.21.15.148:0/3729365595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f9015cd50 0x7f7f900c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8ed9d640 1 -- 172.21.15.148:0/3729365595 >> v1:172.21.15.150:6789/0 conn(0x7f7f901538f0 legacy=0x7f7f90159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.047+0000 7f7f8ed9d640 1 -- 172.21.15.148:0/3729365595 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9015d8c0 con 0x7f7f900cefa0 2023-10-30T13:45:38.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8ed9d640 1 --2- 172.21.15.148:0/3729365595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f900cefa0 0x7f7f900cf3f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009e50 tx=0x7f7f8801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b73ade0a16f619 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:38.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8d59a640 1 -- 172.21.15.148:0/3729365595 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f880af400 con 0x7f7f900cefa0 2023-10-30T13:45:38.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8d59a640 1 -- 172.21.15.148:0/3729365595 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f880ab050 con 0x7f7f900cefa0 2023-10-30T13:45:38.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8d59a640 1 -- 172.21.15.148:0/3729365595 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f880af780 con 0x7f7f900cefa0 2023-10-30T13:45:38.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8f59e640 1 -- 172.21.15.148:0/3729365595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f900cefa0 msgr2=0x7f7f900cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8f59e640 1 --2- 172.21.15.148:0/3729365595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f900cefa0 0x7f7f900cf3f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009e50 tx=0x7f7f8801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8f59e640 1 -- 172.21.15.148:0/3729365595 shutdown_connections 2023-10-30T13:45:38.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8f59e640 1 --2- 172.21.15.148:0/3729365595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f900cefa0 0x7f7f900cf3f0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8f59e640 1 --2- 172.21.15.148:0/3729365595 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f9015cd50 0x7f7f900c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8f59e640 1 -- 172.21.15.148:0/3729365595 >> 172.21.15.148:0/3729365595 conn(0x7f7f9006a640 msgr2=0x7f7f90065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:38.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8f59e640 1 -- 172.21.15.148:0/3729365595 shutdown_connections 2023-10-30T13:45:38.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.051+0000 7f7f8f59e640 1 -- 172.21.15.148:0/3729365595 wait complete. 2023-10-30T13:45:38.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 Processor -- start 2023-10-30T13:45:38.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 -- start start 2023-10-30T13:45:38.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f900cefa0 0x7f7f901f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f901538f0 0x7f7f90158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f9015cd50 0x7f7f900b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9015c320 con 0x7f7f9015cd50 2023-10-30T13:45:38.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f900c6480 con 0x7f7f901538f0 2023-10-30T13:45:38.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f901582d0 con 0x7f7f900cefa0 2023-10-30T13:45:38.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f9015cd50 0x7f7f900b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f9015cd50 0x7f7f900b3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49530/0 (socket says 172.21.15.148:49530) 2023-10-30T13:45:38.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8ed9d640 1 -- 172.21.15.148:0/1328798607 learned_addr learned my addr 172.21.15.148:0/1328798607 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:38.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8dd9b640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f901538f0 0x7f7f90158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8e59c640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f900cefa0 0x7f7f901f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8ed9d640 1 -- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f900cefa0 msgr2=0x7f7f901f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8ed9d640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f900cefa0 0x7f7f901f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8ed9d640 1 -- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f901538f0 msgr2=0x7f7f90158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8ed9d640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f901538f0 0x7f7f90158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8ed9d640 1 -- 172.21.15.148:0/1328798607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f88009b30 con 0x7f7f9015cd50 2023-10-30T13:45:38.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8ed9d640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f9015cd50 0x7f7f900b3bc0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f7f8801ce20 tx=0x7f7f8801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:38.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f7effd640 1 -- 172.21.15.148:0/1328798607 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f880afd30 con 0x7f7f9015cd50 2023-10-30T13:45:38.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f7effd640 1 -- 172.21.15.148:0/1328798607 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f880afe90 con 0x7f7f9015cd50 2023-10-30T13:45:38.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f7effd640 1 -- 172.21.15.148:0/1328798607 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f880ba660 con 0x7f7f9015cd50 2023-10-30T13:45:38.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f901f3260 con 0x7f7f9015cd50 2023-10-30T13:45:38.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f90158ca0 con 0x7f7f9015cd50 2023-10-30T13:45:38.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.055+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f90158460 con 0x7f7f9015cd50 2023-10-30T13:45:38.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.059+0000 7f7f7effd640 1 -- 172.21.15.148:0/1328798607 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f880b6040 con 0x7f7f9015cd50 2023-10-30T13:45:38.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.059+0000 7f7f7effd640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f700664a0 0x7f7f70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.059+0000 7f7f7effd640 1 -- 172.21.15.148:0/1328798607 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f7f88122560 con 0x7f7f9015cd50 2023-10-30T13:45:38.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.059+0000 7f7f8e59c640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f700664a0 0x7f7f70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.059+0000 7f7f8e59c640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f700664a0 0x7f7f70068960 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7f80007fd0 tx=0x7f7f80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:38.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.063+0000 7f7f7effd640 1 -- 172.21.15.148:0/1328798607 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f880ef580 con 0x7f7f9015cd50 2023-10-30T13:45:38.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f9004e170 con 0x7f7f9015cd50 2023-10-30T13:45:38.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f7effd640 1 -- 172.21.15.148:0/1328798607 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6528 (secure 0 0 0) 0x7f7f88009510 con 0x7f7f9015cd50 2023-10-30T13:45:38.251 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:38.251 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:38.159276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:38.252 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 134 2023-10-30T13:45:38.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f700664a0 msgr2=0x7f7f70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f700664a0 0x7f7f70068960 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7f80007fd0 tx=0x7f7f80009040 comp rx=0 tx=0).stop 2023-10-30T13:45:38.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f9015cd50 msgr2=0x7f7f900b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f9015cd50 0x7f7f900b3bc0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f7f8801ce20 tx=0x7f7f8801bf90 comp rx=0 tx=0).stop 2023-10-30T13:45:38.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 shutdown_connections 2023-10-30T13:45:38.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f700664a0 0x7f7f70068960 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f9015cd50 0x7f7f900b3bc0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f901538f0 0x7f7f90158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 --2- 172.21.15.148:0/1328798607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f900cefa0 0x7f7f901f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 >> 172.21.15.148:0/1328798607 conn(0x7f7f9006a640 msgr2=0x7f7f900cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:38.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 shutdown_connections 2023-10-30T13:45:38.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.251+0000 7f7f8f59e640 1 -- 172.21.15.148:0/1328798607 wait complete. 2023-10-30T13:45:38.272 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 2 alive and active, now stable! 2023-10-30T13:45:38.272 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:45:38.272 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:45:38.272 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:45:38.304 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:45:38.305 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:38.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.487+0000 7fb7355d3640 1 Processor -- start 2023-10-30T13:45:38.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.487+0000 7fb7355d3640 1 -- start start 2023-10-30T13:45:38.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7355d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 0x7fb73015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7355d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb7300ca0e0 0x7fb73015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7355d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb7300cf160 0x7fb7300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7355d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7300cb800 con 0x7fb7300ca0e0 2023-10-30T13:45:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7355d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73015dc30 con 0x7fb7300cf160 2023-10-30T13:45:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7355d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7300cb330 con 0x7fb7301538a0 2023-10-30T13:45:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7267fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb7300ca0e0 0x7fb73015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7267fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb7300ca0e0 0x7fb73015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49544/0 (socket says 172.21.15.148:49544) 2023-10-30T13:45:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7267fc640 1 -- 172.21.15.148:0/2413585849 learned_addr learned my addr 172.21.15.148:0/2413585849 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb72f7fe640 1 --2- 172.21.15.148:0/2413585849 >> v2:172.21.15.150:3300/0 conn(0x7fb7300cf160 0x7fb7300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb72effd640 1 --2- 172.21.15.148:0/2413585849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 0x7fb73015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7267fc640 1 -- 172.21.15.148:0/2413585849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 msgr2=0x7fb73015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7267fc640 1 --2- 172.21.15.148:0/2413585849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 0x7fb73015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7267fc640 1 -- 172.21.15.148:0/2413585849 >> v2:172.21.15.150:3300/0 conn(0x7fb7300cf160 msgr2=0x7fb7300cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7267fc640 1 --2- 172.21.15.148:0/2413585849 >> v2:172.21.15.150:3300/0 conn(0x7fb7300cf160 0x7fb7300cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb7267fc640 1 -- 172.21.15.148:0/2413585849 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7300cad70 con 0x7fb7300ca0e0 2023-10-30T13:45:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.491+0000 7fb72f7fe640 1 --2- 172.21.15.148:0/2413585849 >> v2:172.21.15.150:3300/0 conn(0x7fb7300cf160 0x7fb7300cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7267fc640 1 --2- 172.21.15.148:0/2413585849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 0x7fb73015a090 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb73006cb00 tx=0x7fb71801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856d8c1a205e21ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb72e7fc640 1 -- 172.21.15.148:0/2413585849 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7180a89b0 con 0x7fb7300ca0e0 2023-10-30T13:45:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb72e7fc640 1 -- 172.21.15.148:0/2413585849 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7180a8b10 con 0x7fb7300ca0e0 2023-10-30T13:45:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb72e7fc640 1 -- 172.21.15.148:0/2413585849 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7180a8ea0 con 0x7fb7300ca0e0 2023-10-30T13:45:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 -- 172.21.15.148:0/2413585849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 msgr2=0x7fb73015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 --2- 172.21.15.148:0/2413585849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 0x7fb73015a090 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb73006cb00 tx=0x7fb71801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 -- 172.21.15.148:0/2413585849 shutdown_connections 2023-10-30T13:45:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 --2- 172.21.15.148:0/2413585849 >> v2:172.21.15.150:3300/0 conn(0x7fb7300cf160 0x7fb7300cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 --2- 172.21.15.148:0/2413585849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 0x7fb73015a090 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 --2- 172.21.15.148:0/2413585849 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 0x7fb73015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 -- 172.21.15.148:0/2413585849 >> 172.21.15.148:0/2413585849 conn(0x7fb73006bfe0 msgr2=0x7fb73006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 -- 172.21.15.148:0/2413585849 shutdown_connections 2023-10-30T13:45:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 -- 172.21.15.148:0/2413585849 wait complete. 2023-10-30T13:45:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 Processor -- start 2023-10-30T13:45:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.495+0000 7fb7355d3640 1 -- start start 2023-10-30T13:45:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7355d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 0x7fb7301469e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7355d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7300cf160 0x7fb730148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7355d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 0x7fb730147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7355d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb730158c20 con 0x7fb7300ca0e0 2023-10-30T13:45:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7355d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7301ef490 con 0x7fb7300cf160 2023-10-30T13:45:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7355d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7300cd710 con 0x7fb7301538a0 2023-10-30T13:45:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 0x7fb7301469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 0x7fb7301469e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49560/0 (socket says 172.21.15.148:49560) 2023-10-30T13:45:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72effd640 1 -- 172.21.15.148:0/3366840275 learned_addr learned my addr 172.21.15.148:0/3366840275 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7267fc640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7300cf160 0x7fb730148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72f7fe640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 0x7fb730147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72effd640 1 -- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 msgr2=0x7fb730147300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72effd640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 0x7fb730147300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72effd640 1 -- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7300cf160 msgr2=0x7fb730148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72effd640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7300cf160 0x7fb730148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72effd640 1 -- 172.21.15.148:0/3366840275 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7180094d0 con 0x7fb7300ca0e0 2023-10-30T13:45:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7267fc640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7300cf160 0x7fb730148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:45:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb72effd640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 0x7fb7301469e0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb720007e40 tx=0x7fb72000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb727fff640 1 -- 172.21.15.148:0/3366840275 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720014620 con 0x7fb7300ca0e0 2023-10-30T13:45:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb727fff640 1 -- 172.21.15.148:0/3366840275 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72000ee90 con 0x7fb7300ca0e0 2023-10-30T13:45:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb727fff640 1 -- 172.21.15.148:0/3366840275 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7200149c0 con 0x7fb7300ca0e0 2023-10-30T13:45:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb730149a10 con 0x7fb7300ca0e0 2023-10-30T13:45:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb730146f20 con 0x7fb7300ca0e0 2023-10-30T13:45:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7300b2a50 con 0x7fb7300ca0e0 2023-10-30T13:45:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.499+0000 7fb727fff640 1 -- 172.21.15.148:0/3366840275 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb720014b20 con 0x7fb7300ca0e0 2023-10-30T13:45:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.503+0000 7fb727fff640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7100664a0 0x7fb710068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.503+0000 7fb727fff640 1 -- 172.21.15.148:0/3366840275 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fb720084f00 con 0x7fb7300ca0e0 2023-10-30T13:45:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.503+0000 7fb7267fc640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7100664a0 0x7fb710068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:38.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.503+0000 7fb7267fc640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7100664a0 0x7fb710068960 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb7180b2000 tx=0x7fb7180aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:38.508 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.507+0000 7fb727fff640 1 -- 172.21.15.148:0/3366840275 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb720051c90 con 0x7fb7300ca0e0 2023-10-30T13:45:38.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.683+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb73004e170 con 0x7fb7300ca0e0 2023-10-30T13:45:38.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.683+0000 7fb727fff640 1 -- 172.21.15.148:0/3366840275 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6528 (secure 0 0 0) 0x7fb7200552b0 con 0x7fb7300ca0e0 2023-10-30T13:45:38.685 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:38.685 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:38.159276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:38.686 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 134 2023-10-30T13:45:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7100664a0 msgr2=0x7fb710068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7100664a0 0x7fb710068960 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb7180b2000 tx=0x7fb7180aa070 comp rx=0 tx=0).stop 2023-10-30T13:45:38.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 msgr2=0x7fb7301469e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 0x7fb7301469e0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb720007e40 tx=0x7fb72000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 shutdown_connections 2023-10-30T13:45:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7100664a0 0x7fb710068960 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb7301538a0 0x7fb730147300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7300cf160 0x7fb730148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 --2- 172.21.15.148:0/3366840275 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb7300ca0e0 0x7fb7301469e0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 >> 172.21.15.148:0/3366840275 conn(0x7fb73006bfe0 msgr2=0x7fb730151770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 shutdown_connections 2023-10-30T13:45:38.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:38.687+0000 7fb7355d3640 1 -- 172.21.15.148:0/3366840275 wait complete. 2023-10-30T13:45:38.707 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:45:40.708 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:40.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071fbd3640 1 Processor -- start 2023-10-30T13:45:40.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071fbd3640 1 -- start start 2023-10-30T13:45:40.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071fbd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0718065f10 0x7f0718071250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:40.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071fbd3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07181560a0 con 0x7f0718065a80 2023-10-30T13:45:40.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071fbd3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0718156380 con 0x7f071814fd50 2023-10-30T13:45:40.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071fbd3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0718156660 con 0x7f0718065f10 2023-10-30T13:45:40.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071d16e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0718065a80 0x7f07180cb550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56184/0 (socket says 172.21.15.148:56184) 2023-10-30T13:45:40.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071d16e640 1 -- 172.21.15.148:0/490310316 learned_addr learned my addr 172.21.15.148:0/490310316 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:45:40.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071e170640 1 --2- 172.21.15.148:0/490310316 >> v2:172.21.15.150:3300/0 conn(0x7f0718065f10 0x7f0718071250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:40.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140342613 0 0) 0x7f07181560a0 con 0x7f0718065a80 2023-10-30T13:45:40.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.855+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0704003530 con 0x7f0718065a80 2023-10-30T13:45:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106043103 0 0) 0x7f0704003530 con 0x7f0718065a80 2023-10-30T13:45:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 >> v2:172.21.15.150:3300/0 conn(0x7f0718065f10 msgr2=0x7f0718071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 --2- 172.21.15.148:0/490310316 >> v2:172.21.15.150:3300/0 conn(0x7f0718065f10 0x7f0718071250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 >> v1:172.21.15.150:6789/0 conn(0x7f071814fd50 legacy=0x7f07180c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718156940 con 0x7f0718065a80 2023-10-30T13:45:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f0708001680 con 0x7f0718065a80 2023-10-30T13:45:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 >> v1:172.21.15.148:6789/0 conn(0x7f0718065a80 legacy=0x7f07180cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704003980 0x7f0704003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0704004380 0x7f0704004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0704005050 0x7f070400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0718156660 con 0x7f0704004380 2023-10-30T13:45:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070400bfb0 con 0x7f0704003980 2023-10-30T13:45:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070400c170 con 0x7f0704005050 2023-10-30T13:45:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071d16e640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0704004380 0x7f0704004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071d96f640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704003980 0x7f0704003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071e170640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0704005050 0x7f070400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071d16e640 1 -- 172.21.15.148:0/490310316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0704005050 msgr2=0x7f070400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071d16e640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0704005050 0x7f070400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071d16e640 1 -- 172.21.15.148:0/490310316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704003980 msgr2=0x7f0704003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071d16e640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704003980 0x7f0704003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071d16e640 1 -- 172.21.15.148:0/490310316 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0704003530 con 0x7f0704004380 2023-10-30T13:45:40.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071e170640 1 -- 172.21.15.148:0/490310316 reap_dead start 2023-10-30T13:45:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.859+0000 7f071d16e640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0704004380 0x7f0704004850 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f070800c190 tx=0x7f070801cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cdea23e45722db server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0708005ae0 con 0x7f0704004380 2023-10-30T13:45:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0708005c60 con 0x7f0704004380 2023-10-30T13:45:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071c96d640 1 -- 172.21.15.148:0/490310316 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07080a5a80 con 0x7f0704004380 2023-10-30T13:45:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 -- 172.21.15.148:0/490310316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0704004380 msgr2=0x7f0704004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0704004380 0x7f0704004850 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f070800c190 tx=0x7f070801cb40 comp rx=0 tx=0).stop 2023-10-30T13:45:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 -- 172.21.15.148:0/490310316 shutdown_connections 2023-10-30T13:45:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 --2- 172.21.15.148:0/490310316 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0704004380 0x7f0704004850 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 -- 172.21.15.148:0/490310316 >> 172.21.15.148:0/490310316 conn(0x7f071806c6e0 msgr2=0x7f0718069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 -- 172.21.15.148:0/490310316 shutdown_connections 2023-10-30T13:45:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 -- 172.21.15.148:0/490310316 wait complete. 2023-10-30T13:45:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 Processor -- start 2023-10-30T13:45:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 -- start start 2023-10-30T13:45:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704004380 0x7f0718152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07180cb480 0x7f071814e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07181487e0 0x7f0718148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07181ebb50 con 0x7f07180cb480 2023-10-30T13:45:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07181534e0 con 0x7f0704004380 2023-10-30T13:45:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.863+0000 7f071fbd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07180c9da0 con 0x7f07181487e0 2023-10-30T13:45:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d16e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07180cb480 0x7f071814e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d16e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07180cb480 0x7f071814e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:49582/0 (socket says 172.21.15.148:49582) 2023-10-30T13:45:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d16e640 1 -- 172.21.15.148:0/1686886877 learned_addr learned my addr 172.21.15.148:0/1686886877 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071e170640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07181487e0 0x7f0718148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d96f640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704004380 0x7f0718152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d16e640 1 -- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07181487e0 msgr2=0x7f0718148c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d16e640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07181487e0 0x7f0718148c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d16e640 1 -- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704004380 msgr2=0x7f0718152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d16e640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704004380 0x7f0718152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d16e640 1 -- 172.21.15.148:0/1686886877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718156f50 con 0x7f07180cb480 2023-10-30T13:45:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071e170640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07181487e0 0x7f0718148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d96f640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704004380 0x7f0718152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071d16e640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07180cb480 0x7f071814e830 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f070801c970 tx=0x7f0704005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f070e7fc640 1 -- 172.21.15.148:0/1686886877 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07080b4070 con 0x7f07180cb480 2023-10-30T13:45:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f070e7fc640 1 -- 172.21.15.148:0/1686886877 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07080a5a50 con 0x7f07180cb480 2023-10-30T13:45:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f070e7fc640 1 -- 172.21.15.148:0/1686886877 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07080a5dc0 con 0x7f07180cb480 2023-10-30T13:45:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07180ca020 con 0x7f07180cb480 2023-10-30T13:45:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07180ca5e0 con 0x7f07180cb480 2023-10-30T13:45:40.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071804e170 con 0x7f07180cb480 2023-10-30T13:45:40.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f070e7fc640 1 -- 172.21.15.148:0/1686886877 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0708007070 con 0x7f07180cb480 2023-10-30T13:45:40.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f070e7fc640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06fc066490 0x7f06fc068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.867+0000 7f070e7fc640 1 -- 172.21.15.148:0/1686886877 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f070811e6f0 con 0x7f07180cb480 2023-10-30T13:45:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.871+0000 7f071d96f640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06fc066490 0x7f06fc068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.871+0000 7f071d96f640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06fc066490 0x7f06fc068950 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0718069010 tx=0x7f0714007cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:40.875+0000 7f070e7fc640 1 -- 172.21.15.148:0/1686886877 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07080eb510 con 0x7f07180cb480 2023-10-30T13:45:41.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.051+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07180c7180 con 0x7f07180cb480 2023-10-30T13:45:41.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.051+0000 7f070e7fc640 1 -- 172.21.15.148:0/1686886877 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6528 (secure 0 0 0) 0x7f07080eeb30 con 0x7f07180cb480 2023-10-30T13:45:41.054 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:41.054 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:38.159276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:41.055 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 134 2023-10-30T13:45:41.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06fc066490 msgr2=0x7f06fc068950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:41.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06fc066490 0x7f06fc068950 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0718069010 tx=0x7f0714007cd0 comp rx=0 tx=0).stop 2023-10-30T13:45:41.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07180cb480 msgr2=0x7f071814e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:41.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07180cb480 0x7f071814e830 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f070801c970 tx=0x7f0704005800 comp rx=0 tx=0).stop 2023-10-30T13:45:41.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 shutdown_connections 2023-10-30T13:45:41.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f06fc066490 0x7f06fc068950 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:41.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07180cb480 0x7f071814e830 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:41.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07181487e0 0x7f0718148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:41.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 --2- 172.21.15.148:0/1686886877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0704004380 0x7f0718152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:41.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 >> 172.21.15.148:0/1686886877 conn(0x7f071806c6e0 msgr2=0x7f07180c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:41.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 shutdown_connections 2023-10-30T13:45:41.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:41.055+0000 7f071fbd3640 1 -- 172.21.15.148:0/1686886877 wait complete. 2023-10-30T13:45:41.075 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:45:43.077 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:43.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f5d0d6640 1 Processor -- start 2023-10-30T13:45:43.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f5d0d6640 1 -- start start 2023-10-30T13:45:43.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f5d0d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1f58153bf0 0x7f1f58071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:43.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f5d0d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1f580c6fe0 0x7f1f580c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:43.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f5d0d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f580c60e0 con 0x7f1f580cb550 2023-10-30T13:45:43.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f5d0d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f5815d870 con 0x7f1f580c6fe0 2023-10-30T13:45:43.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f5d0d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f580c63a0 con 0x7f1f58153bf0 2023-10-30T13:45:43.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f4ffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1f580c6fe0 0x7f1f580c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:43.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f4ffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1f580c6fe0 0x7f1f580c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42218/0 (socket says 172.21.15.148:42218) 2023-10-30T13:45:43.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f4ffff640 1 -- 172.21.15.148:0/4205559805 learned_addr learned my addr 172.21.15.148:0/4205559805 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:43.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f56d9d640 1 --2- 172.21.15.148:0/4205559805 >> v2:172.21.15.150:3300/0 conn(0x7f1f58153bf0 0x7f1f58071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:43.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f4ffff640 1 -- 172.21.15.148:0/4205559805 >> v2:172.21.15.150:3300/0 conn(0x7f1f58153bf0 msgr2=0x7f1f58071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:43.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.226+0000 7f1f4ffff640 1 --2- 172.21.15.148:0/4205559805 >> v2:172.21.15.150:3300/0 conn(0x7f1f58153bf0 0x7f1f58071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f4ffff640 1 -- 172.21.15.148:0/4205559805 >> v1:172.21.15.148:6789/0 conn(0x7f1f580cb550 legacy=0x7f1f58159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:43.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f4ffff640 1 -- 172.21.15.148:0/4205559805 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f580c6680 con 0x7f1f580c6fe0 2023-10-30T13:45:43.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f4ffff640 1 --2- 172.21.15.148:0/4205559805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f580c6fe0 0x7f1f580c7480 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f1f5806abe0 tx=0x7f1f4001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fdcc8a6d0813043 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:43.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5659c640 1 -- 172.21.15.148:0/4205559805 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5815a820 con 0x7f1f580c6fe0 2023-10-30T13:45:43.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5659c640 1 -- 172.21.15.148:0/4205559805 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f400aa430 con 0x7f1f580c6fe0 2023-10-30T13:45:43.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5659c640 1 -- 172.21.15.148:0/4205559805 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f400aa7c0 con 0x7f1f580c6fe0 2023-10-30T13:45:43.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/4205559805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f580c6fe0 msgr2=0x7f1f580c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:43.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5d0d6640 1 --2- 172.21.15.148:0/4205559805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f580c6fe0 0x7f1f580c7480 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f1f5806abe0 tx=0x7f1f4001c4d0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/4205559805 shutdown_connections 2023-10-30T13:45:43.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5d0d6640 1 --2- 172.21.15.148:0/4205559805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f580c6fe0 0x7f1f580c7480 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5d0d6640 1 --2- 172.21.15.148:0/4205559805 >> v2:172.21.15.150:3300/0 conn(0x7f1f58153bf0 0x7f1f58071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/4205559805 >> 172.21.15.148:0/4205559805 conn(0x7f1f5806c6e0 msgr2=0x7f1f58069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:43.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/4205559805 shutdown_connections 2023-10-30T13:45:43.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.230+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/4205559805 wait complete. 2023-10-30T13:45:43.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5d0d6640 1 Processor -- start 2023-10-30T13:45:43.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5d0d6640 1 -- start start 2023-10-30T13:45:43.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5d0d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580c6fe0 0x7f1f5814caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:43.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5d0d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580cb550 0x7f1f581f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:43.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5d0d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f58153bf0 0x7f1f58150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:43.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5d0d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5815c7b0 con 0x7f1f58153bf0 2023-10-30T13:45:43.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5d0d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5815c570 con 0x7f1f580cb550 2023-10-30T13:45:43.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5d0d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5815cce0 con 0x7f1f580c6fe0 2023-10-30T13:45:43.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f58153bf0 0x7f1f58150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:43.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f58153bf0 0x7f1f58150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42234/0 (socket says 172.21.15.148:42234) 2023-10-30T13:45:43.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5759e640 1 -- 172.21.15.148:0/907663028 learned_addr learned my addr 172.21.15.148:0/907663028 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:43.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f4ffff640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580cb550 0x7f1f581f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:43.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f56d9d640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580c6fe0 0x7f1f5814caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:43.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5759e640 1 -- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580c6fe0 msgr2=0x7f1f5814caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:43.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5759e640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580c6fe0 0x7f1f5814caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5759e640 1 -- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580cb550 msgr2=0x7f1f581f3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:43.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5759e640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580cb550 0x7f1f581f3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5759e640 1 -- 172.21.15.148:0/907663028 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f400094d0 con 0x7f1f58153bf0 2023-10-30T13:45:43.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f56d9d640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580c6fe0 0x7f1f5814caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:43.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f4ffff640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580cb550 0x7f1f581f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:43.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5759e640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f58153bf0 0x7f1f58150bd0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c7a0 tx=0x7f1f5000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:43.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f4f7fe640 1 -- 172.21.15.148:0/907663028 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50003b80 con 0x7f1f58153bf0 2023-10-30T13:45:43.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f4f7fe640 1 -- 172.21.15.148:0/907663028 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f50003ce0 con 0x7f1f58153bf0 2023-10-30T13:45:43.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f4f7fe640 1 -- 172.21.15.148:0/907663028 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5000aa90 con 0x7f1f58153bf0 2023-10-30T13:45:43.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.234+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5814c230 con 0x7f1f58153bf0 2023-10-30T13:45:43.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.238+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5814c4d0 con 0x7f1f58153bf0 2023-10-30T13:45:43.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.238+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5804e170 con 0x7f1f58153bf0 2023-10-30T13:45:43.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.238+0000 7f1f4f7fe640 1 -- 172.21.15.148:0/907663028 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1f5000abf0 con 0x7f1f58153bf0 2023-10-30T13:45:43.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.238+0000 7f1f4f7fe640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f380664a0 0x7f1f38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:43.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.238+0000 7f1f4f7fe640 1 -- 172.21.15.148:0/907663028 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f1f50084430 con 0x7f1f58153bf0 2023-10-30T13:45:43.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.238+0000 7f1f56d9d640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f380664a0 0x7f1f38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:43.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.242+0000 7f1f56d9d640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f380664a0 0x7f1f38068960 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009b10 tx=0x7f1f48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:43.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.242+0000 7f1f4f7fe640 1 -- 172.21.15.148:0/907663028 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1f50051140 con 0x7f1f58153bf0 2023-10-30T13:45:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.422+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f58151200 con 0x7f1f58153bf0 2023-10-30T13:45:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.422+0000 7f1f4f7fe640 1 -- 172.21.15.148:0/907663028 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6528 (secure 0 0 0) 0x7f1f50054760 con 0x7f1f58153bf0 2023-10-30T13:45:43.426 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:43.426 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:38.159276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 134 2023-10-30T13:45:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f380664a0 msgr2=0x7f1f38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f380664a0 0x7f1f38068960 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009b10 tx=0x7f1f48008040 comp rx=0 tx=0).stop 2023-10-30T13:45:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f58153bf0 msgr2=0x7f1f58150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f58153bf0 0x7f1f58150bd0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000c7a0 tx=0x7f1f5000cc60 comp rx=0 tx=0).stop 2023-10-30T13:45:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 shutdown_connections 2023-10-30T13:45:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1f380664a0 0x7f1f38068960 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f58153bf0 0x7f1f58150bd0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f580cb550 0x7f1f581f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 --2- 172.21.15.148:0/907663028 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1f580c6fe0 0x7f1f5814caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 >> 172.21.15.148:0/907663028 conn(0x7f1f5806c6e0 msgr2=0x7f1f580c73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 shutdown_connections 2023-10-30T13:45:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:43.426+0000 7f1f5d0d6640 1 -- 172.21.15.148:0/907663028 wait complete. 2023-10-30T13:45:43.447 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:45:45.448 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.594+0000 7ff3fbb0f640 1 Processor -- start 2023-10-30T13:45:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.594+0000 7ff3fbb0f640 1 -- start start 2023-10-30T13:45:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3fbb0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff3f40c6fe0 0x7ff3f4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3fbb0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40700b0 con 0x7ff3f40c5f60 2023-10-30T13:45:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3fbb0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40c6530 con 0x7ff3f4153bf0 2023-10-30T13:45:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3fbb0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f40c6810 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3f90aa640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff3f40c6fe0 0x7ff3f4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3f90aa640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff3f40c6fe0 0x7ff3f4071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42238/0 (socket says 172.21.15.148:42238) 2023-10-30T13:45:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3f90aa640 1 -- 172.21.15.148:0/3316680299 learned_addr learned my addr 172.21.15.148:0/3316680299 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3f90aa640 1 -- 172.21.15.148:0/3316680299 >> v1:172.21.15.150:6789/0 conn(0x7ff3f4153bf0 legacy=0x7ff3f40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3f90aa640 1 -- 172.21.15.148:0/3316680299 >> v1:172.21.15.148:6789/0 conn(0x7ff3f40c5f60 legacy=0x7ff3f40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.598+0000 7ff3f90aa640 1 -- 172.21.15.148:0/3316680299 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f4159b10 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3f90aa640 1 --2- 172.21.15.148:0/3316680299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 0x7ff3f4071260 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009870 tx=0x7ff3e401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50bc8be7521940d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3f88a9640 1 -- 172.21.15.148:0/3316680299 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40a8a30 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3f88a9640 1 -- 172.21.15.148:0/3316680299 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e40a8b90 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3f88a9640 1 -- 172.21.15.148:0/3316680299 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40b0440 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/3316680299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 msgr2=0x7ff3f4071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3fbb0f640 1 --2- 172.21.15.148:0/3316680299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 0x7ff3f4071260 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009870 tx=0x7ff3e401ed30 comp rx=0 tx=0).stop 2023-10-30T13:45:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/3316680299 shutdown_connections 2023-10-30T13:45:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3fbb0f640 1 --2- 172.21.15.148:0/3316680299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 0x7ff3f4071260 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/3316680299 >> 172.21.15.148:0/3316680299 conn(0x7ff3f406c6e0 msgr2=0x7ff3f4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/3316680299 shutdown_connections 2023-10-30T13:45:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/3316680299 wait complete. 2023-10-30T13:45:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3fbb0f640 1 Processor -- start 2023-10-30T13:45:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.602+0000 7ff3fbb0f640 1 -- start start 2023-10-30T13:45:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fbb0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3f40c5f60 0x7ff3f4158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fbb0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 0x7ff3f41ef000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:45.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fbb0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4153bf0 0x7ff3f41ef540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:45.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fbb0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4155110 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fbb0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4156f00 con 0x7ff3f4153bf0 2023-10-30T13:45:45.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fbb0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f414ee50 con 0x7ff3f40c5f60 2023-10-30T13:45:45.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f90aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 0x7ff3f41ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:45.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f90aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 0x7ff3f41ef000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42240/0 (socket says 172.21.15.148:42240) 2023-10-30T13:45:45.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f90aa640 1 -- 172.21.15.148:0/2180682676 learned_addr learned my addr 172.21.15.148:0/2180682676 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:45.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f98ab640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3f40c5f60 0x7ff3f4158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:45.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fa0ac640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4153bf0 0x7ff3f41ef540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:45.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f90aa640 1 -- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3f40c5f60 msgr2=0x7ff3f4158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:45.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f90aa640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3f40c5f60 0x7ff3f4158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:45.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f90aa640 1 -- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4153bf0 msgr2=0x7ff3f41ef540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:45.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f90aa640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4153bf0 0x7ff3f41ef540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:45.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f90aa640 1 -- 172.21.15.148:0/2180682676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e40094d0 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f98ab640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3f40c5f60 0x7ff3f4158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:45.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3f90aa640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 0x7ff3f41ef000 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40b3040 tx=0x7ff3e401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:45.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3ea7fc640 1 -- 172.21.15.148:0/2180682676 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40ba600 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3ea7fc640 1 -- 172.21.15.148:0/2180682676 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e40ba7a0 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f414f130 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3ea7fc640 1 -- 172.21.15.148:0/2180682676 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40bab50 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f4156700 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f4156260 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.606+0000 7ff3ea7fc640 1 -- 172.21.15.148:0/2180682676 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3e40bacb0 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.610+0000 7ff3ea7fc640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3d80664d0 0x7ff3d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:45.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.610+0000 7ff3ea7fc640 1 -- 172.21.15.148:0/2180682676 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7ff3e4122b10 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.610+0000 7ff3f98ab640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3d80664d0 0x7ff3d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:45.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.610+0000 7ff3f98ab640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3d80664d0 0x7ff3d8068990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0000f70 tx=0x7ff3f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:45.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.614+0000 7ff3ea7fc640 1 -- 172.21.15.148:0/2180682676 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3e40ef8a0 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3f4156260 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3ea7fc640 1 -- 172.21.15.148:0/2180682676 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6528 (secure 0 0 0) 0x7ff3e40b5090 con 0x7ff3f40c6fe0 2023-10-30T13:45:45.800 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:45.800 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:38.159276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:45.801 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 134 2023-10-30T13:45:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3d80664d0 msgr2=0x7ff3d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3d80664d0 0x7ff3d8068990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0000f70 tx=0x7ff3f0009040 comp rx=0 tx=0).stop 2023-10-30T13:45:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 msgr2=0x7ff3f41ef000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:45.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 0x7ff3f41ef000 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40b3040 tx=0x7ff3e401bad0 comp rx=0 tx=0).stop 2023-10-30T13:45:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 shutdown_connections 2023-10-30T13:45:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff3d80664d0 0x7ff3d8068990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3f40c6fe0 0x7ff3f41ef000 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f4153bf0 0x7ff3f41ef540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:45.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 --2- 172.21.15.148:0/2180682676 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff3f40c5f60 0x7ff3f4158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 >> 172.21.15.148:0/2180682676 conn(0x7ff3f406c6e0 msgr2=0x7ff3f414d110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 shutdown_connections 2023-10-30T13:45:45.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:45.798+0000 7ff3fbb0f640 1 -- 172.21.15.148:0/2180682676 wait complete. 2023-10-30T13:45:45.821 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:45:47.823 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:47.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f9786c46640 1 Processor -- start 2023-10-30T13:45:47.971 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f9786c46640 1 -- start start 2023-10-30T13:45:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f9786c46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97800c6b80 0x7f97800bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f9786c46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 0x7f97800c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f9786c46640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97800c0a50 con 0x7f97800c8430 2023-10-30T13:45:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f9786c46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978006f6e0 con 0x7f97800c6b80 2023-10-30T13:45:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f9786c46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978006f9c0 con 0x7f97800c6f60 2023-10-30T13:45:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f97851e3640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f97800c8430 0x7f978013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36172/0 (socket says 172.21.15.148:36172) 2023-10-30T13:45:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f97851e3640 1 -- 172.21.15.148:0/2426185931 learned_addr learned my addr 172.21.15.148:0/2426185931 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:45:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f97849e2640 1 --2- 172.21.15.148:0/2426185931 >> v2:172.21.15.150:3300/0 conn(0x7f97800c6b80 0x7f97800bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f977ffff640 1 --2- 172.21.15.148:0/2426185931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 0x7f97800c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f977f7fe640 1 -- 172.21.15.148:0/2426185931 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097463936 0 0) 0x7f97800c0a50 con 0x7f97800c8430 2023-10-30T13:45:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f977f7fe640 1 -- 172.21.15.148:0/2426185931 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9770003530 con 0x7f97800c8430 2023-10-30T13:45:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f977f7fe640 1 -- 172.21.15.148:0/2426185931 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369660276 0 0) 0x7f9770003530 con 0x7f97800c8430 2023-10-30T13:45:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f977ffff640 1 -- 172.21.15.148:0/2426185931 >> v2:172.21.15.150:3300/0 conn(0x7f97800c6b80 msgr2=0x7f97800bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f977ffff640 1 --2- 172.21.15.148:0/2426185931 >> v2:172.21.15.150:3300/0 conn(0x7f97800c6b80 0x7f97800bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f977ffff640 1 -- 172.21.15.148:0/2426185931 >> v1:172.21.15.148:6789/0 conn(0x7f97800c8430 legacy=0x7f978013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.970+0000 7f977ffff640 1 -- 172.21.15.148:0/2426185931 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978006fca0 con 0x7f97800c6f60 2023-10-30T13:45:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f977ffff640 1 --2- 172.21.15.148:0/2426185931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 0x7f97800c0400 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f976c0097c0 tx=0x7f976c01c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58ff7620734c28c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f977f7fe640 1 -- 172.21.15.148:0/2426185931 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9778001a00 con 0x7f97800c6f60 2023-10-30T13:45:47.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f977f7fe640 1 -- 172.21.15.148:0/2426185931 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c0a7050 con 0x7f97800c6f60 2023-10-30T13:45:47.977 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f977f7fe640 1 -- 172.21.15.148:0/2426185931 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c0ab600 con 0x7f97800c6f60 2023-10-30T13:45:47.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f9786c46640 1 -- 172.21.15.148:0/2426185931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 msgr2=0x7f97800c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:47.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f9786c46640 1 --2- 172.21.15.148:0/2426185931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 0x7f97800c0400 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f976c0097c0 tx=0x7f976c01c310 comp rx=0 tx=0).stop 2023-10-30T13:45:47.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f9786c46640 1 -- 172.21.15.148:0/2426185931 shutdown_connections 2023-10-30T13:45:47.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f9786c46640 1 --2- 172.21.15.148:0/2426185931 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 0x7f97800c0400 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:47.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f9786c46640 1 --2- 172.21.15.148:0/2426185931 >> v2:172.21.15.150:3300/0 conn(0x7f97800c6b80 0x7f97800bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:47.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f9786c46640 1 -- 172.21.15.148:0/2426185931 >> 172.21.15.148:0/2426185931 conn(0x7f97800c7910 msgr2=0x7f97800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:47.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f9786c46640 1 -- 172.21.15.148:0/2426185931 shutdown_connections 2023-10-30T13:45:47.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.974+0000 7f9786c46640 1 -- 172.21.15.148:0/2426185931 wait complete. 2023-10-30T13:45:47.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 Processor -- start 2023-10-30T13:45:47.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 -- start start 2023-10-30T13:45:47.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97800c6b80 0x7f9780158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 0x7f9780150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97800c8430 0x7f97801506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97801f02e0 con 0x7f97800c8430 2023-10-30T13:45:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978013d0a0 con 0x7f97800c6b80 2023-10-30T13:45:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9780153eb0 con 0x7f97800c6f60 2023-10-30T13:45:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97851e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97800c8430 0x7f97801506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97851e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97800c8430 0x7f97801506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42254/0 (socket says 172.21.15.148:42254) 2023-10-30T13:45:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97851e3640 1 -- 172.21.15.148:0/66674625 learned_addr learned my addr 172.21.15.148:0/66674625 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f977ffff640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 0x7f9780150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97849e2640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97800c6b80 0x7f9780158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97851e3640 1 -- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 msgr2=0x7f9780150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97851e3640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 0x7f9780150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97851e3640 1 -- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97800c6b80 msgr2=0x7f9780158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97851e3640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97800c6b80 0x7f9780158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97851e3640 1 -- 172.21.15.148:0/66674625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976c009420 con 0x7f97800c8430 2023-10-30T13:45:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97849e2640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97800c6b80 0x7f9780158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f97851e3640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97800c8430 0x7f97801506a0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f977800cc00 tx=0x7f977800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f977d7fa640 1 -- 172.21.15.148:0/66674625 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977800ce10 con 0x7f97800c8430 2023-10-30T13:45:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f977d7fa640 1 -- 172.21.15.148:0/66674625 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9778012400 con 0x7f97800c8430 2023-10-30T13:45:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f977d7fa640 1 -- 172.21.15.148:0/66674625 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97780127b0 con 0x7f97800c8430 2023-10-30T13:45:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97801ef6d0 con 0x7f97800c8430 2023-10-30T13:45:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.978+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97801efc90 con 0x7f97800c8430 2023-10-30T13:45:47.986 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.982+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97801f0450 con 0x7f97800c8430 2023-10-30T13:45:47.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.982+0000 7f977d7fa640 1 -- 172.21.15.148:0/66674625 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9778012910 con 0x7f97800c8430 2023-10-30T13:45:47.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.982+0000 7f977d7fa640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97640664a0 0x7f9764068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:47.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.982+0000 7f977d7fa640 1 -- 172.21.15.148:0/66674625 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7f9778084cc0 con 0x7f97800c8430 2023-10-30T13:45:47.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.982+0000 7f97849e2640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97640664a0 0x7f9764068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:47.987 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.982+0000 7f97849e2640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97640664a0 0x7f9764068960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9774009bb0 tx=0x7f9774008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:47.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:47.986+0000 7f977d7fa640 1 -- 172.21.15.148:0/66674625 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f977808b080 con 0x7f97800c8430 2023-10-30T13:45:48.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.166+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f978004e170 con 0x7f97800c8430 2023-10-30T13:45:48.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.166+0000 7f977d7fa640 1 -- 172.21.15.148:0/66674625 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6528 (secure 0 0 0) 0x7f97780519c0 con 0x7f97800c8430 2023-10-30T13:45:48.170 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:48.170 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:38.159276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:48.170 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 134 2023-10-30T13:45:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97640664a0 msgr2=0x7f9764068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97640664a0 0x7f9764068960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9774009bb0 tx=0x7f9774008040 comp rx=0 tx=0).stop 2023-10-30T13:45:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97800c8430 msgr2=0x7f97801506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97800c8430 0x7f97801506a0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f977800cc00 tx=0x7f977800a430 comp rx=0 tx=0).stop 2023-10-30T13:45:48.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 shutdown_connections 2023-10-30T13:45:48.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f97640664a0 0x7f9764068960 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:48.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97800c8430 0x7f97801506a0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:48.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97800c6f60 0x7f9780150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:48.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 --2- 172.21.15.148:0/66674625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97800c6b80 0x7f9780158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:48.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 >> 172.21.15.148:0/66674625 conn(0x7f97800c7910 msgr2=0x7f97800c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:48.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 shutdown_connections 2023-10-30T13:45:48.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:48.170+0000 7f9786c46640 1 -- 172.21.15.148:0/66674625 wait complete. 2023-10-30T13:45:48.193 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:45:50.195 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:50.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e7a46640 1 Processor -- start 2023-10-30T13:45:50.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e7a46640 1 -- start start 2023-10-30T13:45:50.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e7a46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e01538a0 0x7fa2e015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:50.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e7a46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2e00ca0e0 0x7fa2e015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:50.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e7a46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e00cb800 con 0x7fa2e00cf160 2023-10-30T13:45:50.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e7a46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e015dc30 con 0x7fa2e00ca0e0 2023-10-30T13:45:50.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e7a46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e00cb330 con 0x7fa2e01538a0 2023-10-30T13:45:50.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e4fe1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2e00ca0e0 0x7fa2e015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:50.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e4fe1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2e00ca0e0 0x7fa2e015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:59830/0 (socket says 172.21.15.148:59830) 2023-10-30T13:45:50.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e4fe1640 1 -- 172.21.15.148:0/135005897 learned_addr learned my addr 172.21.15.148:0/135005897 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:50.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e5fe3640 1 --1- 172.21.15.148:0/135005897 >> v1:172.21.15.150:6789/0 conn(0x7fa2e00cf160 0x7fa2e00cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:39040/0 (socket says 172.21.15.148:39040) 2023-10-30T13:45:50.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e57e2640 1 --2- 172.21.15.148:0/135005897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e01538a0 0x7fa2e015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:50.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e4fe1640 1 -- 172.21.15.148:0/135005897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e01538a0 msgr2=0x7fa2e015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:50.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e4fe1640 1 --2- 172.21.15.148:0/135005897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e01538a0 0x7fa2e015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e4fe1640 1 -- 172.21.15.148:0/135005897 >> v1:172.21.15.150:6789/0 conn(0x7fa2e00cf160 legacy=0x7fa2e00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:50.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.342+0000 7fa2e4fe1640 1 -- 172.21.15.148:0/135005897 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e00cad70 con 0x7fa2e00ca0e0 2023-10-30T13:45:50.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2e4fe1640 1 --2- 172.21.15.148:0/135005897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e00ca0e0 0x7fa2e015a090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009870 tx=0x7fa2d001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f241e8a9c126730 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:50.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2d7fff640 1 -- 172.21.15.148:0/135005897 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00a8a30 con 0x7fa2e00ca0e0 2023-10-30T13:45:50.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2d7fff640 1 -- 172.21.15.148:0/135005897 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d00a8b90 con 0x7fa2e00ca0e0 2023-10-30T13:45:50.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2d7fff640 1 -- 172.21.15.148:0/135005897 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00b0440 con 0x7fa2e00ca0e0 2023-10-30T13:45:50.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2e7a46640 1 -- 172.21.15.148:0/135005897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e00ca0e0 msgr2=0x7fa2e015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:50.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2e7a46640 1 --2- 172.21.15.148:0/135005897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e00ca0e0 0x7fa2e015a090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009870 tx=0x7fa2d001ed30 comp rx=0 tx=0).stop 2023-10-30T13:45:50.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2e7a46640 1 -- 172.21.15.148:0/135005897 shutdown_connections 2023-10-30T13:45:50.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2e7a46640 1 --2- 172.21.15.148:0/135005897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e00ca0e0 0x7fa2e015a090 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2e7a46640 1 --2- 172.21.15.148:0/135005897 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e01538a0 0x7fa2e015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2e7a46640 1 -- 172.21.15.148:0/135005897 >> 172.21.15.148:0/135005897 conn(0x7fa2e006bfe0 msgr2=0x7fa2e006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:50.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2e7a46640 1 -- 172.21.15.148:0/135005897 shutdown_connections 2023-10-30T13:45:50.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.346+0000 7fa2e7a46640 1 -- 172.21.15.148:0/135005897 wait complete. 2023-10-30T13:45:50.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 Processor -- start 2023-10-30T13:45:50.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 -- start start 2023-10-30T13:45:50.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e00ca0e0 0x7fa2e0147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:50.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e00cf160 0x7fa2e015c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:50.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2e01538a0 0x7fa2e015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:50.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e01f0c50 con 0x7fa2e01538a0 2023-10-30T13:45:50.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e01436f0 con 0x7fa2e00cf160 2023-10-30T13:45:50.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e015acf0 con 0x7fa2e00ca0e0 2023-10-30T13:45:50.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e5fe3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2e01538a0 0x7fa2e015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:50.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e5fe3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2e01538a0 0x7fa2e015ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42270/0 (socket says 172.21.15.148:42270) 2023-10-30T13:45:50.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e57e2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e00ca0e0 0x7fa2e0147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:50.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e5fe3640 1 -- 172.21.15.148:0/1695876064 learned_addr learned my addr 172.21.15.148:0/1695876064 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:50.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e4fe1640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e00cf160 0x7fa2e015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:50.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e5fe3640 1 -- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e00ca0e0 msgr2=0x7fa2e0147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:50.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e5fe3640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e00ca0e0 0x7fa2e0147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e5fe3640 1 -- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e00cf160 msgr2=0x7fa2e015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:50.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e5fe3640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e00cf160 0x7fa2e015c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e5fe3640 1 -- 172.21.15.148:0/1695876064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d00094d0 con 0x7fa2e01538a0 2023-10-30T13:45:50.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e5fe3640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2e01538a0 0x7fa2e015ca40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800c730 tx=0x7fa2d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:50.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2d5ffb640 1 -- 172.21.15.148:0/1695876064 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d800ce40 con 0x7fa2e01538a0 2023-10-30T13:45:50.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2d5ffb640 1 -- 172.21.15.148:0/1695876064 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d800a5e0 con 0x7fa2e01538a0 2023-10-30T13:45:50.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e015afd0 con 0x7fa2e01538a0 2023-10-30T13:45:50.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2d5ffb640 1 -- 172.21.15.148:0/1695876064 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d800a950 con 0x7fa2e01538a0 2023-10-30T13:45:50.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.350+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e015b590 con 0x7fa2e01538a0 2023-10-30T13:45:50.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.354+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e00b2a50 con 0x7fa2e01538a0 2023-10-30T13:45:50.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.354+0000 7fa2d5ffb640 1 -- 172.21.15.148:0/1695876064 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2d800aab0 con 0x7fa2e01538a0 2023-10-30T13:45:50.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.354+0000 7fa2d5ffb640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa2c40664a0 0x7fa2c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:50.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.354+0000 7fa2d5ffb640 1 -- 172.21.15.148:0/1695876064 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7fa2d8083c40 con 0x7fa2e01538a0 2023-10-30T13:45:50.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.354+0000 7fa2e57e2640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa2c40664a0 0x7fa2c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:50.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.358+0000 7fa2e57e2640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa2c40664a0 0x7fa2c4068960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009ac0 tx=0x7fa2dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:50.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.358+0000 7fa2d5ffb640 1 -- 172.21.15.148:0/1695876064 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2d80509f0 con 0x7fa2e01538a0 2023-10-30T13:45:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2e01f0d90 con 0x7fa2e01538a0 2023-10-30T13:45:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2d5ffb640 1 -- 172.21.15.148:0/1695876064 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6528 (secure 0 0 0) 0x7fa2e01f0d90 con 0x7fa2e01538a0 2023-10-30T13:45:50.543 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:50.544 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:38.159276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 134 2023-10-30T13:45:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa2c40664a0 msgr2=0x7fa2c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa2c40664a0 0x7fa2c4068960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009ac0 tx=0x7fa2dc009190 comp rx=0 tx=0).stop 2023-10-30T13:45:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2e01538a0 msgr2=0x7fa2e015ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2e01538a0 0x7fa2e015ca40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800c730 tx=0x7fa2d800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 shutdown_connections 2023-10-30T13:45:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa2c40664a0 0x7fa2c4068960 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2e01538a0 0x7fa2e015ca40 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e00cf160 0x7fa2e015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 --2- 172.21.15.148:0/1695876064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa2e00ca0e0 0x7fa2e0147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 >> 172.21.15.148:0/1695876064 conn(0x7fa2e006bfe0 msgr2=0x7fa2e00cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 shutdown_connections 2023-10-30T13:45:50.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:50.542+0000 7fa2e7a46640 1 -- 172.21.15.148:0/1695876064 wait complete. 2023-10-30T13:45:50.565 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:45:52.566 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:52.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.710+0000 7effe57a1640 1 Processor -- start 2023-10-30T13:45:52.714 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effe57a1640 1 -- start start 2023-10-30T13:45:52.715 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effe57a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00c6b80 0x7effe00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effe57a1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7effe00c6f60 0x7effe00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effe57a1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe00c0a50 con 0x7effe00c8430 2023-10-30T13:45:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effe57a1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe006f6e0 con 0x7effe00c6f60 2023-10-30T13:45:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effe57a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe006f9c0 con 0x7effe00c6b80 2023-10-30T13:45:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdf7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7effe00c8430 0x7effe013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37256/0 (socket says 172.21.15.148:37256) 2023-10-30T13:45:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdf7fe640 1 -- 172.21.15.148:0/3512819204 learned_addr learned my addr 172.21.15.148:0/3512819204 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:45:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdeffd640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00c6b80 0x7effe00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effd67fc640 1 --2- 172.21.15.148:0/3512819204 >> v2:172.21.15.150:3300/0 conn(0x7effe00c6f60 0x7effe00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:52.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891964459 0 0) 0x7effe00c0a50 con 0x7effe00c8430 2023-10-30T13:45:52.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effcc003530 con 0x7effe00c8430 2023-10-30T13:45:52.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2064686013 0 0) 0x7effcc003530 con 0x7effe00c8430 2023-10-30T13:45:52.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00c6b80 msgr2=0x7effe00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:52.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00c6b80 0x7effe00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 >> v2:172.21.15.150:3300/0 conn(0x7effe00c6f60 msgr2=0x7effe00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:52.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 --2- 172.21.15.148:0/3512819204 >> v2:172.21.15.150:3300/0 conn(0x7effe00c6f60 0x7effe00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe006fca0 con 0x7effe00c8430 2023-10-30T13:45:52.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7effd8001a00 con 0x7effe00c8430 2023-10-30T13:45:52.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effd67fc640 1 --2- 172.21.15.148:0/3512819204 >> v2:172.21.15.150:3300/0 conn(0x7effe00c6f60 0x7effe00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:45:52.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 >> v1:172.21.15.148:6789/0 conn(0x7effe00c8430 legacy=0x7effe013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:52.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effcc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:52.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00c6b80 0x7effcc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:52.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effcc004d10 0x7effcc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:52.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe006f6e0 con 0x7effcc003980 2023-10-30T13:45:52.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe006f9c0 con 0x7effcc004d10 2023-10-30T13:45:52.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc00b990 con 0x7effe00c6b80 2023-10-30T13:45:52.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdeffd640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effcc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:52.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effd67fc640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00c6b80 0x7effcc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:52.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdf7fe640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effcc004d10 0x7effcc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:52.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdeffd640 1 -- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00c6b80 msgr2=0x7effcc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:52.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdeffd640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00c6b80 0x7effcc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdeffd640 1 -- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effcc004d10 msgr2=0x7effcc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:52.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdeffd640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effcc004d10 0x7effcc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.714+0000 7effdeffd640 1 -- 172.21.15.148:0/3512819204 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc003530 con 0x7effcc003980 2023-10-30T13:45:52.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effdeffd640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effcc003da0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7effd0009670 tx=0x7effd001ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faecf249c87dbaea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:52.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00a7770 con 0x7effcc003980 2023-10-30T13:45:52.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd00a78d0 con 0x7effcc003980 2023-10-30T13:45:52.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effde7fc640 1 -- 172.21.15.148:0/3512819204 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00a7c80 con 0x7effcc003980 2023-10-30T13:45:52.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effe57a1640 1 -- 172.21.15.148:0/3512819204 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 msgr2=0x7effcc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:52.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effe57a1640 1 --2- 172.21.15.148:0/3512819204 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effcc003da0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7effd0009670 tx=0x7effd001ec80 comp rx=0 tx=0).stop 2023-10-30T13:45:52.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effdf7fe640 1 -- 172.21.15.148:0/3512819204 reap_dead start 2023-10-30T13:45:52.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effe57a1640 1 -- 172.21.15.148:0/3512819204 shutdown_connections 2023-10-30T13:45:52.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effe57a1640 1 -- 172.21.15.148:0/3512819204 >> 172.21.15.148:0/3512819204 conn(0x7effe00c7910 msgr2=0x7effe00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:52.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effe57a1640 1 -- 172.21.15.148:0/3512819204 shutdown_connections 2023-10-30T13:45:52.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.718+0000 7effe57a1640 1 -- 172.21.15.148:0/3512819204 wait complete. 2023-10-30T13:45:52.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 Processor -- start 2023-10-30T13:45:52.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 -- start start 2023-10-30T13:45:52.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effe01f5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:52.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe0152f80 0x7effe01f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:52.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00ca920 0x7effe00c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:52.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0154480 con 0x7effcc003980 2023-10-30T13:45:52.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0155ee0 con 0x7effe0152f80 2023-10-30T13:45:52.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe01f6400 con 0x7effe00ca920 2023-10-30T13:45:52.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdeffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effe01f5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:52.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdeffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effe01f5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52462/0 (socket says 172.21.15.148:52462) 2023-10-30T13:45:52.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdeffd640 1 -- 172.21.15.148:0/2924088895 learned_addr learned my addr 172.21.15.148:0/2924088895 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:52.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effd67fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe0152f80 0x7effe01f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:52.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdf7fe640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00ca920 0x7effe00c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:52.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdeffd640 1 -- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00ca920 msgr2=0x7effe00c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:52.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdeffd640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00ca920 0x7effe00c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdeffd640 1 -- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe0152f80 msgr2=0x7effe01f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:52.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdeffd640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe0152f80 0x7effe01f5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdeffd640 1 -- 172.21.15.148:0/2924088895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe0156180 con 0x7effcc003980 2023-10-30T13:45:52.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effdeffd640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effe01f5890 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7effd00b2040 tx=0x7effcc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:52.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effd7fff640 1 -- 172.21.15.148:0/2924088895 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00a7770 con 0x7effcc003980 2023-10-30T13:45:52.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effd7fff640 1 -- 172.21.15.148:0/2924088895 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd0004020 con 0x7effcc003980 2023-10-30T13:45:52.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe0156460 con 0x7effcc003980 2023-10-30T13:45:52.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effd7fff640 1 -- 172.21.15.148:0/2924088895 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00043d0 con 0x7effcc003980 2023-10-30T13:45:52.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.722+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe015a560 con 0x7effcc003980 2023-10-30T13:45:52.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.726+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe004e170 con 0x7effcc003980 2023-10-30T13:45:52.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.726+0000 7effd7fff640 1 -- 172.21.15.148:0/2924088895 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7effd0003620 con 0x7effcc003980 2023-10-30T13:45:52.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.726+0000 7effd7fff640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7effc00664d0 0x7effc0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:52.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.726+0000 7effd7fff640 1 -- 172.21.15.148:0/2924088895 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7316+0+0 (secure 0 0 0) 0x7effd011e130 con 0x7effcc003980 2023-10-30T13:45:52.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.726+0000 7effd67fc640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7effc00664d0 0x7effc0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:52.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.726+0000 7effd67fc640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7effc00664d0 0x7effc0068990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7effe00c2060 tx=0x7effc8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:52.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.730+0000 7effd7fff640 1 -- 172.21.15.148:0/2924088895 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7effd00eaf80 con 0x7effcc003980 2023-10-30T13:45:52.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.910+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effe00b08b0 con 0x7effcc003980 2023-10-30T13:45:52.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effd7fff640 1 -- 172.21.15.148:0/2924088895 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+6528 (secure 0 0 0) 0x7effd00b4090 con 0x7effcc003980 2023-10-30T13:45:52.915 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:52.915 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":5453,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:38.159276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7464,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7464":{"gid":7464,"name":"c","rank":0,"incarnation":126,"state":"up:active","state_seq":32,"addr":"172.21.15.148:6837/3703944646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":3703944646},{"type":"v1","addr":"172.21.15.148:6837","nonce":3703944646}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:52.915 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 134 2023-10-30T13:45:52.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7effc00664d0 msgr2=0x7effc0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:52.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7effc00664d0 0x7effc0068990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7effe00c2060 tx=0x7effc8008040 comp rx=0 tx=0).stop 2023-10-30T13:45:52.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 msgr2=0x7effe01f5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:52.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effe01f5890 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7effd00b2040 tx=0x7effcc005800 comp rx=0 tx=0).stop 2023-10-30T13:45:52.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 shutdown_connections 2023-10-30T13:45:52.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7effc00664d0 0x7effc0068990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7effe00ca920 0x7effe00c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe0152f80 0x7effe01f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 --2- 172.21.15.148:0/2924088895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effcc003980 0x7effe01f5890 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:52.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 >> 172.21.15.148:0/2924088895 conn(0x7effe00c7910 msgr2=0x7effe0159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:52.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 shutdown_connections 2023-10-30T13:45:52.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:52.914+0000 7effe57a1640 1 -- 172.21.15.148:0/2924088895 wait complete. 2023-10-30T13:45:52.937 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:45:54.938 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:55.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.086+0000 7fd34f654640 1 Processor -- start 2023-10-30T13:45:55.089 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.086+0000 7fd34f654640 1 -- start start 2023-10-30T13:45:55.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34f654640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd3481538a0 0x7fd34815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34f654640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd3480ca0e0 0x7fd34815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34f654640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3480cb800 con 0x7fd3480cf160 2023-10-30T13:45:55.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34f654640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34815dc30 con 0x7fd3481538a0 2023-10-30T13:45:55.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34f654640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3480cb330 con 0x7fd3480ca0e0 2023-10-30T13:45:55.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34d3f0640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd3481538a0 0x7fd34815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34d3f0640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd3481538a0 0x7fd34815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52476/0 (socket says 172.21.15.148:52476) 2023-10-30T13:45:55.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34d3f0640 1 -- 172.21.15.148:0/3522430880 learned_addr learned my addr 172.21.15.148:0/3522430880 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:55.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34cbef640 1 --2- 172.21.15.148:0/3522430880 >> v2:172.21.15.150:3300/0 conn(0x7fd3480ca0e0 0x7fd34815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34d3f0640 1 -- 172.21.15.148:0/3522430880 >> v2:172.21.15.150:3300/0 conn(0x7fd3480ca0e0 msgr2=0x7fd34815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:55.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34d3f0640 1 --2- 172.21.15.148:0/3522430880 >> v2:172.21.15.150:3300/0 conn(0x7fd3480ca0e0 0x7fd34815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34d3f0640 1 -- 172.21.15.148:0/3522430880 >> v1:172.21.15.150:6789/0 conn(0x7fd3480cf160 legacy=0x7fd3480cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:55.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.090+0000 7fd34d3f0640 1 -- 172.21.15.148:0/3522430880 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480cad70 con 0x7fd3481538a0 2023-10-30T13:45:55.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34d3f0640 1 --2- 172.21.15.148:0/3522430880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3481538a0 0x7fd34815d6f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fd3480b30a0 tx=0x7fd34401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f827077fb674a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:55.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd33ffff640 1 -- 172.21.15.148:0/3522430880 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440a89b0 con 0x7fd3481538a0 2023-10-30T13:45:55.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd33ffff640 1 -- 172.21.15.148:0/3522430880 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3440a8b10 con 0x7fd3481538a0 2023-10-30T13:45:55.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd33ffff640 1 -- 172.21.15.148:0/3522430880 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3440a8ea0 con 0x7fd3481538a0 2023-10-30T13:45:55.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 -- 172.21.15.148:0/3522430880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3481538a0 msgr2=0x7fd34815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 --2- 172.21.15.148:0/3522430880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3481538a0 0x7fd34815d6f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fd3480b30a0 tx=0x7fd34401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 -- 172.21.15.148:0/3522430880 shutdown_connections 2023-10-30T13:45:55.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 --2- 172.21.15.148:0/3522430880 >> v2:172.21.15.150:3300/0 conn(0x7fd3480ca0e0 0x7fd34815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 --2- 172.21.15.148:0/3522430880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3481538a0 0x7fd34815d6f0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 -- 172.21.15.148:0/3522430880 >> 172.21.15.148:0/3522430880 conn(0x7fd34806bfe0 msgr2=0x7fd34806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:55.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 -- 172.21.15.148:0/3522430880 shutdown_connections 2023-10-30T13:45:55.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 -- 172.21.15.148:0/3522430880 wait complete. 2023-10-30T13:45:55.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 Processor -- start 2023-10-30T13:45:55.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.094+0000 7fd34f654640 1 -- start start 2023-10-30T13:45:55.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34f654640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480ca0e0 0x7fd3480b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34f654640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480cf160 0x7fd3481466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34f654640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3481538a0 0x7fd3481498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34f654640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480cda00 con 0x7fd3480cf160 2023-10-30T13:45:55.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34f654640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3481e2ae0 con 0x7fd3480ca0e0 2023-10-30T13:45:55.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34f654640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480b3570 con 0x7fd3481538a0 2023-10-30T13:45:55.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34cbef640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480cf160 0x7fd3481466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34cbef640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480cf160 0x7fd3481466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52492/0 (socket says 172.21.15.148:52492) 2023-10-30T13:45:55.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34cbef640 1 -- 172.21.15.148:0/3035811556 learned_addr learned my addr 172.21.15.148:0/3035811556 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:55.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34d3f0640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480ca0e0 0x7fd3480b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34dbf1640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3481538a0 0x7fd3481498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34cbef640 1 -- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3481538a0 msgr2=0x7fd3481498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34cbef640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3481538a0 0x7fd3481498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34cbef640 1 -- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480ca0e0 msgr2=0x7fd3480b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34cbef640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480ca0e0 0x7fd3480b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34cbef640 1 -- 172.21.15.148:0/3035811556 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3440094d0 con 0x7fd3480cf160 2023-10-30T13:45:55.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34d3f0640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480ca0e0 0x7fd3480b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:55.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34cbef640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480cf160 0x7fd3481466e0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fd33800d800 tx=0x7fd33800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:55.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd33dffb640 1 -- 172.21.15.148:0/3035811556 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33800b6c0 con 0x7fd3480cf160 2023-10-30T13:45:55.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd33dffb640 1 -- 172.21.15.148:0/3035811556 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33800b820 con 0x7fd3480cf160 2023-10-30T13:45:55.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd33dffb640 1 -- 172.21.15.148:0/3035811556 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33800bb90 con 0x7fd3480cf160 2023-10-30T13:45:55.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd348145ff0 con 0x7fd3480cf160 2023-10-30T13:45:55.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3480ccfb0 con 0x7fd3480cf160 2023-10-30T13:45:55.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3480b2a50 con 0x7fd3480cf160 2023-10-30T13:45:55.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.098+0000 7fd33dffb640 1 -- 172.21.15.148:0/3035811556 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd33800bcf0 con 0x7fd3480cf160 2023-10-30T13:45:55.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.102+0000 7fd33dffb640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd32c0664a0 0x7fd32c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.102+0000 7fd33dffb640 1 -- 172.21.15.148:0/3035811556 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7fd338084a50 con 0x7fd3480cf160 2023-10-30T13:45:55.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.102+0000 7fd34d3f0640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd32c0664a0 0x7fd32c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.102+0000 7fd34d3f0640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd32c0664a0 0x7fd32c068960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd3440b2000 tx=0x7fd344008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:55.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.102+0000 7fd33dffb640 1 -- 172.21.15.148:0/3035811556 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd338051820 con 0x7fd3480cf160 2023-10-30T13:45:55.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd34804e170 con 0x7fd3480cf160 2023-10-30T13:45:55.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd33dffb640 1 -- 172.21.15.148:0/3035811556 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5733 (secure 0 0 0) 0x7fd338054e40 con 0x7fd3480cf160 2023-10-30T13:45:55.288 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:55.288 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:54.968923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:55.288 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 135 2023-10-30T13:45:55.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd32c0664a0 msgr2=0x7fd32c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd32c0664a0 0x7fd32c068960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd3440b2000 tx=0x7fd344008040 comp rx=0 tx=0).stop 2023-10-30T13:45:55.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480cf160 msgr2=0x7fd3481466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480cf160 0x7fd3481466e0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fd33800d800 tx=0x7fd33800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 shutdown_connections 2023-10-30T13:45:55.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd32c0664a0 0x7fd32c068960 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480cf160 0x7fd3481466e0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3481538a0 0x7fd3481498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 --2- 172.21.15.148:0/3035811556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480ca0e0 0x7fd3480b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 >> 172.21.15.148:0/3035811556 conn(0x7fd34806bfe0 msgr2=0x7fd3480c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:55.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 shutdown_connections 2023-10-30T13:45:55.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.286+0000 7fd34f654640 1 -- 172.21.15.148:0/3035811556 wait complete. 2023-10-30T13:45:55.309 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:45:55.309 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:45:55.309 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:55.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.442+0000 7f6224832640 1 Processor -- start 2023-10-30T13:45:55.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.442+0000 7f6224832640 1 -- start start 2023-10-30T13:45:55.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f6224832640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f62200c6b80 0x7f62200bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f6224832640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f622013f4a0 0x7f62200c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f6224832640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62200c0a50 con 0x7f62200c6f60 2023-10-30T13:45:55.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f6224832640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622006f6e0 con 0x7f62200c6b80 2023-10-30T13:45:55.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f6224832640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622006f9c0 con 0x7f622013f4a0 2023-10-30T13:45:55.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f621e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f62200c6b80 0x7f62200bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f621e59c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f62200c6b80 0x7f62200bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52494/0 (socket says 172.21.15.148:52494) 2023-10-30T13:45:55.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f621e59c640 1 -- 172.21.15.148:0/1086862878 learned_addr learned my addr 172.21.15.148:0/1086862878 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:55.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f621ed9d640 1 --2- 172.21.15.148:0/1086862878 >> v2:172.21.15.150:3300/0 conn(0x7f622013f4a0 0x7f62200c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f621e59c640 1 -- 172.21.15.148:0/1086862878 >> v2:172.21.15.150:3300/0 conn(0x7f622013f4a0 msgr2=0x7f62200c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:55.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f621e59c640 1 --2- 172.21.15.148:0/1086862878 >> v2:172.21.15.150:3300/0 conn(0x7f622013f4a0 0x7f62200c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f621e59c640 1 -- 172.21.15.148:0/1086862878 >> v1:172.21.15.148:6789/0 conn(0x7f62200c6f60 legacy=0x7f622013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.446+0000 7f621e59c640 1 -- 172.21.15.148:0/1086862878 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622006fca0 con 0x7f62200c6b80 2023-10-30T13:45:55.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f621e59c640 1 --2- 172.21.15.148:0/1086862878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6b80 0x7f62200bfec0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f62200c8980 tx=0x7f621001ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ab99ebbf6107b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:55.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f621dd9b640 1 -- 172.21.15.148:0/1086862878 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62100a7bb0 con 0x7f62200c6b80 2023-10-30T13:45:55.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f621dd9b640 1 -- 172.21.15.148:0/1086862878 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62100a7d10 con 0x7f62200c6b80 2023-10-30T13:45:55.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f621dd9b640 1 -- 172.21.15.148:0/1086862878 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62100af660 con 0x7f62200c6b80 2023-10-30T13:45:55.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 -- 172.21.15.148:0/1086862878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6b80 msgr2=0x7f62200bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 --2- 172.21.15.148:0/1086862878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6b80 0x7f62200bfec0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f62200c8980 tx=0x7f621001ebd0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 -- 172.21.15.148:0/1086862878 shutdown_connections 2023-10-30T13:45:55.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 --2- 172.21.15.148:0/1086862878 >> v2:172.21.15.150:3300/0 conn(0x7f622013f4a0 0x7f62200c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 --2- 172.21.15.148:0/1086862878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6b80 0x7f62200bfec0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 -- 172.21.15.148:0/1086862878 >> 172.21.15.148:0/1086862878 conn(0x7f62200c7910 msgr2=0x7f62200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:55.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 -- 172.21.15.148:0/1086862878 shutdown_connections 2023-10-30T13:45:55.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 -- 172.21.15.148:0/1086862878 wait complete. 2023-10-30T13:45:55.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 Processor -- start 2023-10-30T13:45:55.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.450+0000 7f6224832640 1 -- start start 2023-10-30T13:45:55.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6224832640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62200c6b80 0x7f62201f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6224832640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6f60 0x7f622006a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6224832640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f622013f4a0 0x7f62201386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6224832640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62200caae0 con 0x7f62200c6f60 2023-10-30T13:45:55.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6224832640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62201f5a00 con 0x7f62200c6b80 2023-10-30T13:45:55.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6224832640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62200ca540 con 0x7f622013f4a0 2023-10-30T13:45:55.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6215d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6f60 0x7f622006a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6215d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6f60 0x7f622006a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52508/0 (socket says 172.21.15.148:52508) 2023-10-30T13:45:55.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6215d9b640 1 -- 172.21.15.148:0/1147908574 learned_addr learned my addr 172.21.15.148:0/1147908574 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:55.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f621e59c640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62200c6b80 0x7f62201f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f621ed9d640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f622013f4a0 0x7f62201386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6215d9b640 1 -- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f622013f4a0 msgr2=0x7f62201386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6215d9b640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f622013f4a0 0x7f62201386e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6215d9b640 1 -- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62200c6b80 msgr2=0x7f62201f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6215d9b640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62200c6b80 0x7f62201f1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6215d9b640 1 -- 172.21.15.148:0/1147908574 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621000b770 con 0x7f62200c6f60 2023-10-30T13:45:55.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f621ed9d640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f622013f4a0 0x7f62201386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:55.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6215d9b640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6f60 0x7f622006a8b0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f620800c820 tx=0x7f620800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:55.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f62177fe640 1 -- 172.21.15.148:0/1147908574 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6208003b80 con 0x7f62200c6f60 2023-10-30T13:45:55.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f62177fe640 1 -- 172.21.15.148:0/1147908574 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6208003ce0 con 0x7f62200c6f60 2023-10-30T13:45:55.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f62177fe640 1 -- 172.21.15.148:0/1147908574 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620800aa90 con 0x7f62200c6f60 2023-10-30T13:45:55.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62200ca800 con 0x7f62200c6f60 2023-10-30T13:45:55.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62201f5690 con 0x7f62200c6f60 2023-10-30T13:45:55.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.454+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62201f5850 con 0x7f62200c6f60 2023-10-30T13:45:55.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.458+0000 7f62177fe640 1 -- 172.21.15.148:0/1147908574 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f620800abf0 con 0x7f62200c6f60 2023-10-30T13:45:55.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.458+0000 7f62177fe640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62000664a0 0x7f6200068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:55.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.458+0000 7f62177fe640 1 -- 172.21.15.148:0/1147908574 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f6208084440 con 0x7f62200c6f60 2023-10-30T13:45:55.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.458+0000 7f621e59c640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62000664a0 0x7f6200068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:55.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.458+0000 7f621e59c640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62000664a0 0x7f6200068960 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f62100b1000 tx=0x7f6210008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:55.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.462+0000 7f62177fe640 1 -- 172.21.15.148:0/1147908574 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6208051220 con 0x7f62200c6f60 2023-10-30T13:45:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62201f5bb0 con 0x7f62200c6f60 2023-10-30T13:45:55.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f62177fe640 1 -- 172.21.15.148:0/1147908574 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f62201f5bb0 con 0x7f62200c6f60 2023-10-30T13:45:55.644 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:55.644 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:54.968923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:55.644 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 135 2023-10-30T13:45:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62000664a0 msgr2=0x7f6200068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62000664a0 0x7f6200068960 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f62100b1000 tx=0x7f6210008040 comp rx=0 tx=0).stop 2023-10-30T13:45:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6f60 msgr2=0x7f622006a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6f60 0x7f622006a8b0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f620800c820 tx=0x7f620800cce0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 shutdown_connections 2023-10-30T13:45:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f62000664a0 0x7f6200068960 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f62200c6f60 0x7f622006a8b0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f622013f4a0 0x7f62201386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 --2- 172.21.15.148:0/1147908574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62200c6b80 0x7f62201f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 >> 172.21.15.148:0/1147908574 conn(0x7f62200c7910 msgr2=0x7f62200c12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:55.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 shutdown_connections 2023-10-30T13:45:55.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:55.642+0000 7f6224832640 1 -- 172.21.15.148:0/1147908574 wait complete. 2023-10-30T13:45:55.665 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 135, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 135, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5453": { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 135, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_7611": { "addr": "172.21.15.148:6835/1228827025", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1228827025, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1228827025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7611, "incarnation": 133, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:45:54.968923+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5453, "mds_1": 7611 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:45:57.666 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:45:57.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7190e05640 1 Processor -- start 2023-10-30T13:45:57.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7190e05640 1 -- start start 2023-10-30T13:45:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7190e05640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f718c1538a0 0x7f718c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7190e05640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f718c0ca0e0 0x7f718c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7190e05640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0cf160 0x7f718c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:57.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7190e05640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f718c0cb800 con 0x7f718c0ca0e0 2023-10-30T13:45:57.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7190e05640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f718c15dc30 con 0x7f718c1538a0 2023-10-30T13:45:57.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7190e05640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718c0cb330 con 0x7f718c0cf160 2023-10-30T13:45:57.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7181d9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f718c0ca0e0 0x7f718c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:57.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7181d9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f718c0ca0e0 0x7f718c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52516/0 (socket says 172.21.15.148:52516) 2023-10-30T13:45:57.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7181d9b640 1 -- 172.21.15.148:0/1231951398 learned_addr learned my addr 172.21.15.148:0/1231951398 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:57.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f718a59c640 1 --2- 172.21.15.148:0/1231951398 >> v2:172.21.15.150:3300/0 conn(0x7f718c1538a0 0x7f718c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:57.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f718ad9d640 1 --2- 172.21.15.148:0/1231951398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0cf160 0x7f718c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:57.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7181d9b640 1 -- 172.21.15.148:0/1231951398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0cf160 msgr2=0x7f718c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:57.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7181d9b640 1 --2- 172.21.15.148:0/1231951398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0cf160 0x7f718c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:57.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7181d9b640 1 -- 172.21.15.148:0/1231951398 >> v2:172.21.15.150:3300/0 conn(0x7f718c1538a0 msgr2=0x7f718c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:45:57.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7181d9b640 1 --2- 172.21.15.148:0/1231951398 >> v2:172.21.15.150:3300/0 conn(0x7f718c1538a0 0x7f718c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:57.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.814+0000 7f7181d9b640 1 -- 172.21.15.148:0/1231951398 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c0cad70 con 0x7f718c0ca0e0 2023-10-30T13:45:57.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7181d9b640 1 --2- 172.21.15.148:0/1231951398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0ca0e0 0x7f718c15a090 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f718c0cfd10 tx=0x7f717c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d409271ff9d17cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:57.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7189d9b640 1 -- 172.21.15.148:0/1231951398 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c0a89b0 con 0x7f718c0ca0e0 2023-10-30T13:45:57.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7189d9b640 1 -- 172.21.15.148:0/1231951398 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717c0a8b10 con 0x7f718c0ca0e0 2023-10-30T13:45:57.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7189d9b640 1 -- 172.21.15.148:0/1231951398 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c0a8ea0 con 0x7f718c0ca0e0 2023-10-30T13:45:57.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7190e05640 1 -- 172.21.15.148:0/1231951398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0ca0e0 msgr2=0x7f718c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:57.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7190e05640 1 --2- 172.21.15.148:0/1231951398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0ca0e0 0x7f718c15a090 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f718c0cfd10 tx=0x7f717c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:45:57.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7190e05640 1 -- 172.21.15.148:0/1231951398 shutdown_connections 2023-10-30T13:45:57.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7190e05640 1 --2- 172.21.15.148:0/1231951398 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0cf160 0x7f718c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:57.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7190e05640 1 --2- 172.21.15.148:0/1231951398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0ca0e0 0x7f718c15a090 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:57.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7190e05640 1 --2- 172.21.15.148:0/1231951398 >> v2:172.21.15.150:3300/0 conn(0x7f718c1538a0 0x7f718c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:57.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7190e05640 1 -- 172.21.15.148:0/1231951398 >> 172.21.15.148:0/1231951398 conn(0x7f718c06bfe0 msgr2=0x7f718c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:57.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7190e05640 1 -- 172.21.15.148:0/1231951398 shutdown_connections 2023-10-30T13:45:57.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.818+0000 7f7190e05640 1 -- 172.21.15.148:0/1231951398 wait complete. 2023-10-30T13:45:57.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 Processor -- start 2023-10-30T13:45:57.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 -- start start 2023-10-30T13:45:57.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0ca0e0 0x7f718c0ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:57.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0cf160 0x7f718c0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:57.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718c1538a0 0x7f718c0b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:57.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c0ce9c0 con 0x7f718c0cf160 2023-10-30T13:45:57.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c0ceca0 con 0x7f718c1538a0 2023-10-30T13:45:57.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718c15aa50 con 0x7f718c0ca0e0 2023-10-30T13:45:57.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7181d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0cf160 0x7f718c0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:57.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7181d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0cf160 0x7f718c0b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52520/0 (socket says 172.21.15.148:52520) 2023-10-30T13:45:57.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7181d9b640 1 -- 172.21.15.148:0/3083651460 learned_addr learned my addr 172.21.15.148:0/3083651460 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:45:57.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f718ad9d640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718c1538a0 0x7f718c0b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:57.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f718a59c640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0ca0e0 0x7f718c0ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:57.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7181d9b640 1 -- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0ca0e0 msgr2=0x7f718c0ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:57.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7181d9b640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0ca0e0 0x7f718c0ce2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:57.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7181d9b640 1 -- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718c1538a0 msgr2=0x7f718c0b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:57.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7181d9b640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718c1538a0 0x7f718c0b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:57.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7181d9b640 1 -- 172.21.15.148:0/3083651460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c0094d0 con 0x7f718c0cf160 2023-10-30T13:45:57.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f718a59c640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0ca0e0 0x7f718c0ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:45:57.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7181d9b640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0cf160 0x7f718c0b34d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f717c01d3a0 tx=0x7f717c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:57.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f71837fe640 1 -- 172.21.15.148:0/3083651460 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c0b0d40 con 0x7f718c0cf160 2023-10-30T13:45:57.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f71837fe640 1 -- 172.21.15.148:0/3083651460 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717c0bb460 con 0x7f718c0cf160 2023-10-30T13:45:57.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718c15ad30 con 0x7f718c0cf160 2023-10-30T13:45:57.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f71837fe640 1 -- 172.21.15.148:0/3083651460 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717c0b0ea0 con 0x7f718c0cf160 2023-10-30T13:45:57.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.822+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718c0b3a10 con 0x7f718c0cf160 2023-10-30T13:45:57.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.826+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f718c0b2a50 con 0x7f718c0cf160 2023-10-30T13:45:57.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.826+0000 7f71837fe640 1 -- 172.21.15.148:0/3083651460 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f717c0b83f0 con 0x7f718c0cf160 2023-10-30T13:45:57.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.826+0000 7f71837fe640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f716c0664d0 0x7f716c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:45:57.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.826+0000 7f71837fe640 1 -- 172.21.15.148:0/3083651460 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f717c121ef0 con 0x7f718c0cf160 2023-10-30T13:45:57.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.826+0000 7f718a59c640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f716c0664d0 0x7f716c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:45:57.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.826+0000 7f718a59c640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f716c0664d0 0x7f716c068990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7174009b10 tx=0x7f7174008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:45:57.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:57.830+0000 7f71837fe640 1 -- 172.21.15.148:0/3083651460 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f717c0eec50 con 0x7f718c0cf160 2023-10-30T13:45:58.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.010+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f718c15a8b0 con 0x7f718c0cf160 2023-10-30T13:45:58.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.010+0000 7f71837fe640 1 -- 172.21.15.148:0/3083651460 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f718c15a8b0 con 0x7f718c0cf160 2023-10-30T13:45:58.014 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:45:58.015 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:54.968923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:45:58.015 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 135 2023-10-30T13:45:58.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f716c0664d0 msgr2=0x7f716c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f716c0664d0 0x7f716c068990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7174009b10 tx=0x7f7174008040 comp rx=0 tx=0).stop 2023-10-30T13:45:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0cf160 msgr2=0x7f718c0b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:45:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0cf160 0x7f718c0b34d0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f717c01d3a0 tx=0x7f717c01ba90 comp rx=0 tx=0).stop 2023-10-30T13:45:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 shutdown_connections 2023-10-30T13:45:58.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f716c0664d0 0x7f716c068990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:58.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f718c0cf160 0x7f718c0b34d0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:58.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f718c1538a0 0x7f718c0b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:58.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 --2- 172.21.15.148:0/3083651460 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f718c0ca0e0 0x7f718c0ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:45:58.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 >> 172.21.15.148:0/3083651460 conn(0x7f718c06bfe0 msgr2=0x7f718c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:45:58.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 shutdown_connections 2023-10-30T13:45:58.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:45:58.014+0000 7f7190e05640 1 -- 172.21.15.148:0/3083651460 wait complete. 2023-10-30T13:45:58.035 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:00.036 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:00.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.174+0000 7f1974246640 1 Processor -- start 2023-10-30T13:46:00.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.174+0000 7f1974246640 1 -- start start 2023-10-30T13:46:00.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1974246640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f196c0c6b80 0x7f196c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:00.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1974246640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f196c13f4a0 0x7f196c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:00.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1974246640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196c0c0a50 con 0x7f196c0c6f60 2023-10-30T13:46:00.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1974246640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196c06f6e0 con 0x7f196c0c6b80 2023-10-30T13:46:00.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1974246640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196c06f9c0 con 0x7f196c13f4a0 2023-10-30T13:46:00.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1971fe2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f196c0c6b80 0x7f196c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:00.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1971fe2640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f196c0c6b80 0x7f196c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52532/0 (socket says 172.21.15.148:52532) 2023-10-30T13:46:00.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1971fe2640 1 -- 172.21.15.148:0/3635064785 learned_addr learned my addr 172.21.15.148:0/3635064785 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:00.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f19727e3640 1 --2- 172.21.15.148:0/3635064785 >> v2:172.21.15.150:3300/0 conn(0x7f196c13f4a0 0x7f196c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:00.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1971fe2640 1 -- 172.21.15.148:0/3635064785 >> v2:172.21.15.150:3300/0 conn(0x7f196c13f4a0 msgr2=0x7f196c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:00.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1971fe2640 1 --2- 172.21.15.148:0/3635064785 >> v2:172.21.15.150:3300/0 conn(0x7f196c13f4a0 0x7f196c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1971fe2640 1 -- 172.21.15.148:0/3635064785 >> v1:172.21.15.148:6789/0 conn(0x7f196c0c6f60 legacy=0x7f196c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:00.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.178+0000 7f1971fe2640 1 -- 172.21.15.148:0/3635064785 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196c06fca0 con 0x7f196c0c6b80 2023-10-30T13:46:00.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1971fe2640 1 --2- 172.21.15.148:0/3635064785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6b80 0x7f196c0bfec0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f196c0c8980 tx=0x7f196801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183e3946594bfc66 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:00.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1970fe0640 1 -- 172.21.15.148:0/3635064785 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19680ab610 con 0x7f196c0c6b80 2023-10-30T13:46:00.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1970fe0640 1 -- 172.21.15.148:0/3635064785 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19680ab770 con 0x7f196c0c6b80 2023-10-30T13:46:00.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1970fe0640 1 -- 172.21.15.148:0/3635064785 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19680abb00 con 0x7f196c0c6b80 2023-10-30T13:46:00.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 -- 172.21.15.148:0/3635064785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6b80 msgr2=0x7f196c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:00.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 --2- 172.21.15.148:0/3635064785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6b80 0x7f196c0bfec0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f196c0c8980 tx=0x7f196801c500 comp rx=0 tx=0).stop 2023-10-30T13:46:00.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 -- 172.21.15.148:0/3635064785 shutdown_connections 2023-10-30T13:46:00.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 --2- 172.21.15.148:0/3635064785 >> v2:172.21.15.150:3300/0 conn(0x7f196c13f4a0 0x7f196c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 --2- 172.21.15.148:0/3635064785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6b80 0x7f196c0bfec0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 -- 172.21.15.148:0/3635064785 >> 172.21.15.148:0/3635064785 conn(0x7f196c0c7910 msgr2=0x7f196c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:00.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 -- 172.21.15.148:0/3635064785 shutdown_connections 2023-10-30T13:46:00.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 -- 172.21.15.148:0/3635064785 wait complete. 2023-10-30T13:46:00.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 Processor -- start 2023-10-30T13:46:00.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.182+0000 7f1974246640 1 -- start start 2023-10-30T13:46:00.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1974246640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f196c0c6b80 0x7f196c1f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:00.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1974246640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6f60 0x7f196c13cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:00.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1974246640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f196c13f4a0 0x7f196c13d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:00.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1974246640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c15f070 con 0x7f196c0c6f60 2023-10-30T13:46:00.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1974246640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c15eb40 con 0x7f196c0c6b80 2023-10-30T13:46:00.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1974246640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196c1f5850 con 0x7f196c13f4a0 2023-10-30T13:46:00.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19717e1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6f60 0x7f196c13cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:00.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19717e1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6f60 0x7f196c13cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52538/0 (socket says 172.21.15.148:52538) 2023-10-30T13:46:00.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19717e1640 1 -- 172.21.15.148:0/2593360841 learned_addr learned my addr 172.21.15.148:0/2593360841 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:00.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1971fe2640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f196c0c6b80 0x7f196c1f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:00.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19727e3640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f196c13f4a0 0x7f196c13d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:00.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19717e1640 1 -- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f196c13f4a0 msgr2=0x7f196c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:00.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19717e1640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f196c13f4a0 0x7f196c13d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19717e1640 1 -- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f196c0c6b80 msgr2=0x7f196c1f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:00.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19717e1640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f196c0c6b80 0x7f196c1f5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19717e1640 1 -- 172.21.15.148:0/2593360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19680094d0 con 0x7f196c0c6f60 2023-10-30T13:46:00.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19717e1640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6f60 0x7f196c13cf90 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f195c00c530 tx=0x7f195c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:00.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19627fc640 1 -- 172.21.15.148:0/2593360841 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c005220 con 0x7f196c0c6f60 2023-10-30T13:46:00.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19627fc640 1 -- 172.21.15.148:0/2593360841 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195c005380 con 0x7f196c0c6f60 2023-10-30T13:46:00.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19627fc640 1 -- 172.21.15.148:0/2593360841 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c007270 con 0x7f196c0c6f60 2023-10-30T13:46:00.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196c1f5520 con 0x7f196c0c6f60 2023-10-30T13:46:00.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196c0c8db0 con 0x7f196c0c6f60 2023-10-30T13:46:00.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196c04e170 con 0x7f196c0c6f60 2023-10-30T13:46:00.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.186+0000 7f19627fc640 1 -- 172.21.15.148:0/2593360841 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f195c0073d0 con 0x7f196c0c6f60 2023-10-30T13:46:00.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.190+0000 7f19627fc640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19500664a0 0x7f1950068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:00.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.190+0000 7f19627fc640 1 -- 172.21.15.148:0/2593360841 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f195c083d50 con 0x7f196c0c6f60 2023-10-30T13:46:00.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.190+0000 7f1971fe2640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19500664a0 0x7f1950068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:00.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.190+0000 7f1971fe2640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19500664a0 0x7f1950068960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f19680af000 tx=0x7f19680b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:00.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.194+0000 7f19627fc640 1 -- 172.21.15.148:0/2593360841 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f195c050b20 con 0x7f196c0c6f60 2023-10-30T13:46:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.378+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f196c160bd0 con 0x7f196c0c6f60 2023-10-30T13:46:00.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.378+0000 7f19627fc640 1 -- 172.21.15.148:0/2593360841 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f195c054140 con 0x7f196c0c6f60 2023-10-30T13:46:00.381 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:00.381 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:54.968923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:00.381 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 135 2023-10-30T13:46:00.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19500664a0 msgr2=0x7f1950068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:00.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19500664a0 0x7f1950068960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f19680af000 tx=0x7f19680b6040 comp rx=0 tx=0).stop 2023-10-30T13:46:00.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6f60 msgr2=0x7f196c13cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:00.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6f60 0x7f196c13cf90 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f195c00c530 tx=0x7f195c00c9f0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 shutdown_connections 2023-10-30T13:46:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f19500664a0 0x7f1950068960 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f196c0c6f60 0x7f196c13cf90 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f196c13f4a0 0x7f196c13d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 --2- 172.21.15.148:0/2593360841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f196c0c6b80 0x7f196c1f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 >> 172.21.15.148:0/2593360841 conn(0x7f196c0c7910 msgr2=0x7f196c0c6210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:00.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 shutdown_connections 2023-10-30T13:46:00.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:00.382+0000 7f1974246640 1 -- 172.21.15.148:0/2593360841 wait complete. 2023-10-30T13:46:00.404 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:02.406 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:02.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.550+0000 7f6efd046640 1 Processor -- start 2023-10-30T13:46:02.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.550+0000 7f6efd046640 1 -- start start 2023-10-30T13:46:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.550+0000 7f6efd046640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6ef80c6f60 0x7f6ef81566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.550+0000 7f6efd046640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef80c0a50 con 0x7f6ef80c05d0 2023-10-30T13:46:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.550+0000 7f6efd046640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef806f6e0 con 0x7f6ef80c6b80 2023-10-30T13:46:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.550+0000 7f6efd046640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef806f9c0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef759e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6ef80c6f60 0x7f6ef81566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef759e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6ef80c6f60 0x7f6ef81566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39154/0 (socket says 172.21.15.148:39154) 2023-10-30T13:46:02.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef759e640 1 -- 172.21.15.148:0/2910928585 learned_addr learned my addr 172.21.15.148:0/2910928585 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:02.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef759e640 1 -- 172.21.15.148:0/2910928585 >> v1:172.21.15.150:6789/0 conn(0x7f6ef80c6b80 legacy=0x7f6ef80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:02.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef759e640 1 -- 172.21.15.148:0/2910928585 >> v1:172.21.15.148:6789/0 conn(0x7f6ef80c05d0 legacy=0x7f6ef813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:02.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef759e640 1 -- 172.21.15.148:0/2910928585 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef806fca0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef759e640 1 --2- 172.21.15.148:0/2910928585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 0x7f6ef81566a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009e50 tx=0x7f6ef001ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6329367ac0cda6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:02.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef5d9b640 1 -- 172.21.15.148:0/2910928585 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00a7900 con 0x7f6ef80c6f60 2023-10-30T13:46:02.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef5d9b640 1 -- 172.21.15.148:0/2910928585 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef00a7a60 con 0x7f6ef80c6f60 2023-10-30T13:46:02.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.554+0000 7f6ef5d9b640 1 -- 172.21.15.148:0/2910928585 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00a7df0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.559 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 -- 172.21.15.148:0/2910928585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 msgr2=0x7f6ef81566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:02.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 --2- 172.21.15.148:0/2910928585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 0x7f6ef81566a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009e50 tx=0x7f6ef001ec20 comp rx=0 tx=0).stop 2023-10-30T13:46:02.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 -- 172.21.15.148:0/2910928585 shutdown_connections 2023-10-30T13:46:02.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 --2- 172.21.15.148:0/2910928585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 0x7f6ef81566a0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:02.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 -- 172.21.15.148:0/2910928585 >> 172.21.15.148:0/2910928585 conn(0x7f6ef80c7910 msgr2=0x7f6ef80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:02.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 -- 172.21.15.148:0/2910928585 shutdown_connections 2023-10-30T13:46:02.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 -- 172.21.15.148:0/2910928585 wait complete. 2023-10-30T13:46:02.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 Processor -- start 2023-10-30T13:46:02.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 -- start start 2023-10-30T13:46:02.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ef80c05d0 0x7f6ef8153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:02.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ef80c6b80 0x7f6ef8157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:02.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 0x7f6ef81517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:02.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef80ca8a0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef80c9fb0 con 0x7f6ef80c6b80 2023-10-30T13:46:02.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6efd046640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef8159460 con 0x7f6ef80c05d0 2023-10-30T13:46:02.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6ef759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 0x7f6ef81517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:02.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6ef6d9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ef80c05d0 0x7f6ef8153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:02.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6ef759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 0x7f6ef81517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39162/0 (socket says 172.21.15.148:39162) 2023-10-30T13:46:02.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6ef759e640 1 -- 172.21.15.148:0/190785024 learned_addr learned my addr 172.21.15.148:0/190785024 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:02.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.558+0000 7f6ef659c640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ef80c6b80 0x7f6ef8157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:02.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6ef759e640 1 -- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ef80c05d0 msgr2=0x7f6ef8153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:02.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6ef759e640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ef80c05d0 0x7f6ef8153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:02.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6ef759e640 1 -- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ef80c6b80 msgr2=0x7f6ef8157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:02.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6ef759e640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ef80c6b80 0x7f6ef8157ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:02.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6ef759e640 1 -- 172.21.15.148:0/190785024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef0009b30 con 0x7f6ef80c6f60 2023-10-30T13:46:02.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6ef759e640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 0x7f6ef81517c0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00b1040 tx=0x7f6ef001b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:02.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6edf7fe640 1 -- 172.21.15.148:0/190785024 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00ba670 con 0x7f6ef80c6f60 2023-10-30T13:46:02.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6edf7fe640 1 -- 172.21.15.148:0/190785024 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef00ba7f0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef80caaf0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6edf7fe640 1 -- 172.21.15.148:0/190785024 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00baba0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef80ca200 con 0x7f6ef80c6f60 2023-10-30T13:46:02.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef81577e0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6edf7fe640 1 -- 172.21.15.148:0/190785024 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ef00bad00 con 0x7f6ef80c6f60 2023-10-30T13:46:02.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6edf7fe640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ee40664a0 0x7f6ee4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:02.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.562+0000 7f6edf7fe640 1 -- 172.21.15.148:0/190785024 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f6ef0121bd0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.566+0000 7f6ef6d9d640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ee40664a0 0x7f6ee4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:02.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.566+0000 7f6ef6d9d640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ee40664a0 0x7f6ee4068960 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8007fd0 tx=0x7f6ee8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:02.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.566+0000 7f6edf7fe640 1 -- 172.21.15.148:0/190785024 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ef81577e0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.668 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:02.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:02.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.750+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ef81577e0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.750+0000 7f6edf7fe640 1 -- 172.21.15.148:0/190785024 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f6ef81577e0 con 0x7f6ef80c6f60 2023-10-30T13:46:02.754 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:02.755 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:54.968923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:02.755 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 135 2023-10-30T13:46:02.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ee40664a0 msgr2=0x7f6ee4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:02.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ee40664a0 0x7f6ee4068960 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8007fd0 tx=0x7f6ee8009040 comp rx=0 tx=0).stop 2023-10-30T13:46:02.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 msgr2=0x7f6ef81517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:02.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 0x7f6ef81517c0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00b1040 tx=0x7f6ef001b9c0 comp rx=0 tx=0).stop 2023-10-30T13:46:02.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 shutdown_connections 2023-10-30T13:46:02.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ee40664a0 0x7f6ee4068960 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:02.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ef80c6f60 0x7f6ef81517c0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:02.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ef80c6b80 0x7f6ef8157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:02.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 --2- 172.21.15.148:0/190785024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ef80c05d0 0x7f6ef8153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:02.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 >> 172.21.15.148:0/190785024 conn(0x7f6ef80c7910 msgr2=0x7f6ef80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:02.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 shutdown_connections 2023-10-30T13:46:02.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:02.754+0000 7f6efd046640 1 -- 172.21.15.148:0/190785024 wait complete. 2023-10-30T13:46:02.775 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:04.776 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:04.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.914+0000 7f7d465c5640 1 Processor -- start 2023-10-30T13:46:04.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.914+0000 7f7d465c5640 1 -- start start 2023-10-30T13:46:04.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d465c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d400bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:04.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d465c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7d4013f4a0 0x7f7d400c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:04.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d465c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d400c0a50 con 0x7f7d400c6f60 2023-10-30T13:46:04.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d465c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4006f6e0 con 0x7f7d4013f4a0 2023-10-30T13:46:04.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d465c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4006f9c0 con 0x7f7d400c6b80 2023-10-30T13:46:04.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d400bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:04.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d44b62640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7d4013f4a0 0x7f7d400c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:04.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d400bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:46868/0 (socket says 172.21.15.148:46868) 2023-10-30T13:46:04.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3ffff640 1 -- 172.21.15.148:0/3991144730 learned_addr learned my addr 172.21.15.148:0/3991144730 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:04.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3f7fe640 1 --1- 172.21.15.148:0/3991144730 >> v1:172.21.15.150:6789/0 conn(0x7f7d400c6f60 0x7f7d4013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:47148/0 (socket says 172.21.15.148:47148) 2023-10-30T13:46:04.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3effd640 1 -- 172.21.15.148:0/3991144730 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272252706 0 0) 0x7f7d400c0a50 con 0x7f7d400c6f60 2023-10-30T13:46:04.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3effd640 1 -- 172.21.15.148:0/3991144730 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d28003530 con 0x7f7d400c6f60 2023-10-30T13:46:04.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3ffff640 1 -- 172.21.15.148:0/3991144730 >> v2:172.21.15.150:3300/0 conn(0x7f7d4013f4a0 msgr2=0x7f7d400c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:04.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3ffff640 1 --2- 172.21.15.148:0/3991144730 >> v2:172.21.15.150:3300/0 conn(0x7f7d4013f4a0 0x7f7d400c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:04.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3ffff640 1 -- 172.21.15.148:0/3991144730 >> v1:172.21.15.150:6789/0 conn(0x7f7d400c6f60 legacy=0x7f7d4013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:04.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.918+0000 7f7d3ffff640 1 -- 172.21.15.148:0/3991144730 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4006fca0 con 0x7f7d400c6b80 2023-10-30T13:46:04.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d3ffff640 1 --2- 172.21.15.148:0/3991144730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d400bfec0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009870 tx=0x7f7d3001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca2da90cceb7a2fc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:04.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d3effd640 1 -- 172.21.15.148:0/3991144730 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d300ab650 con 0x7f7d400c6b80 2023-10-30T13:46:04.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d3effd640 1 -- 172.21.15.148:0/3991144730 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d300ab7b0 con 0x7f7d400c6b80 2023-10-30T13:46:04.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d3effd640 1 -- 172.21.15.148:0/3991144730 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d300abb20 con 0x7f7d400c6b80 2023-10-30T13:46:04.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 -- 172.21.15.148:0/3991144730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 msgr2=0x7f7d400bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:04.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 --2- 172.21.15.148:0/3991144730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d400bfec0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009870 tx=0x7f7d3001c540 comp rx=0 tx=0).stop 2023-10-30T13:46:04.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 -- 172.21.15.148:0/3991144730 shutdown_connections 2023-10-30T13:46:04.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 --2- 172.21.15.148:0/3991144730 >> v2:172.21.15.150:3300/0 conn(0x7f7d4013f4a0 0x7f7d400c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:04.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 --2- 172.21.15.148:0/3991144730 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d400bfec0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:04.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 -- 172.21.15.148:0/3991144730 >> 172.21.15.148:0/3991144730 conn(0x7f7d400c7910 msgr2=0x7f7d400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:04.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 -- 172.21.15.148:0/3991144730 shutdown_connections 2023-10-30T13:46:04.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 -- 172.21.15.148:0/3991144730 wait complete. 2023-10-30T13:46:04.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 Processor -- start 2023-10-30T13:46:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.922+0000 7f7d465c5640 1 -- start start 2023-10-30T13:46:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d465c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d401f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d465c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d400c6f60 0x7f7d40155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d465c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4013f4a0 0x7f7d4014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d465c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400c5db0 con 0x7f7d400c6f60 2023-10-30T13:46:04.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d465c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400c6070 con 0x7f7d4013f4a0 2023-10-30T13:46:04.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d465c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40150c80 con 0x7f7d400c6b80 2023-10-30T13:46:04.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d400c6f60 0x7f7d40155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:04.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3ffff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d401f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:04.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d400c6f60 0x7f7d40155e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39178/0 (socket says 172.21.15.148:39178) 2023-10-30T13:46:04.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3f7fe640 1 -- 172.21.15.148:0/654078570 learned_addr learned my addr 172.21.15.148:0/654078570 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:04.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d44b62640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4013f4a0 0x7f7d4014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:04.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3f7fe640 1 -- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 msgr2=0x7f7d401f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:04.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3f7fe640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d401f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:04.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3f7fe640 1 -- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4013f4a0 msgr2=0x7f7d4014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:04.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3f7fe640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4013f4a0 0x7f7d4014ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:04.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3f7fe640 1 -- 172.21.15.148:0/654078570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300094d0 con 0x7f7d400c6f60 2023-10-30T13:46:04.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3f7fe640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d400c6f60 0x7f7d40155e10 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400c400 tx=0x7f7d3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:04.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3cff9640 1 -- 172.21.15.148:0/654078570 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d340074e0 con 0x7f7d400c6f60 2023-10-30T13:46:04.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3cff9640 1 -- 172.21.15.148:0/654078570 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34007640 con 0x7f7d400c6f60 2023-10-30T13:46:04.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d3cff9640 1 -- 172.21.15.148:0/654078570 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34004a30 con 0x7f7d400c6f60 2023-10-30T13:46:04.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d40150f60 con 0x7f7d400c6f60 2023-10-30T13:46:04.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d401f23e0 con 0x7f7d400c6f60 2023-10-30T13:46:04.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.926+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4004e170 con 0x7f7d400c6f60 2023-10-30T13:46:04.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.930+0000 7f7d3cff9640 1 -- 172.21.15.148:0/654078570 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d34004b90 con 0x7f7d400c6f60 2023-10-30T13:46:04.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.930+0000 7f7d3cff9640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d2c0664d0 0x7f7d2c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:04.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.930+0000 7f7d3cff9640 1 -- 172.21.15.148:0/654078570 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f7d34083ef0 con 0x7f7d400c6f60 2023-10-30T13:46:04.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.930+0000 7f7d3ffff640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d2c0664d0 0x7f7d2c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:04.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.930+0000 7f7d3ffff640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d2c0664d0 0x7f7d2c068990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f7d300af000 tx=0x7f7d300b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:04.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:04.934+0000 7f7d3cff9640 1 -- 172.21.15.148:0/654078570 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d34050ce0 con 0x7f7d400c6f60 2023-10-30T13:46:05.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d401f25a0 con 0x7f7d400c6f60 2023-10-30T13:46:05.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d3cff9640 1 -- 172.21.15.148:0/654078570 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5733 (secure 0 0 0) 0x7f7d34054300 con 0x7f7d400c6f60 2023-10-30T13:46:05.119 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:05.120 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:45:54.968923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:05.120 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 135 2023-10-30T13:46:05.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d2c0664d0 msgr2=0x7f7d2c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:05.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d2c0664d0 0x7f7d2c068990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f7d300af000 tx=0x7f7d300b6040 comp rx=0 tx=0).stop 2023-10-30T13:46:05.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d400c6f60 msgr2=0x7f7d40155e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:05.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d400c6f60 0x7f7d40155e10 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400c400 tx=0x7f7d3400c8c0 comp rx=0 tx=0).stop 2023-10-30T13:46:05.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 shutdown_connections 2023-10-30T13:46:05.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d2c0664d0 0x7f7d2c068990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:05.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d400c6f60 0x7f7d40155e10 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:05.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4013f4a0 0x7f7d4014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:05.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 --2- 172.21.15.148:0/654078570 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d400c6b80 0x7f7d401f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:05.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 >> 172.21.15.148:0/654078570 conn(0x7f7d400c7910 msgr2=0x7f7d40153990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:05.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 shutdown_connections 2023-10-30T13:46:05.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:05.118+0000 7f7d465c5640 1 -- 172.21.15.148:0/654078570 wait complete. 2023-10-30T13:46:05.141 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:07.142 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:07.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bde73e640 1 Processor -- start 2023-10-30T13:46:07.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bde73e640 1 -- start start 2023-10-30T13:46:07.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bde73e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 0x7f9bd815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:07.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bde73e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9bd80ca0e0 0x7f9bd815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:07.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bde73e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd80cb800 con 0x7f9bd80cf160 2023-10-30T13:46:07.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bde73e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd815dc30 con 0x7f9bd80ca0e0 2023-10-30T13:46:07.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bde73e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80cb330 con 0x7f9bd81538a0 2023-10-30T13:46:07.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bdccdb640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9bd80cf160 0x7f9bd80cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51638/0 (socket says 172.21.15.148:51638) 2023-10-30T13:46:07.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bdccdb640 1 -- 172.21.15.148:0/2717489776 learned_addr learned my addr 172.21.15.148:0/2717489776 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:46:07.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bd77fe640 1 --2- 172.21.15.148:0/2717489776 >> v2:172.21.15.150:3300/0 conn(0x7f9bd80ca0e0 0x7f9bd815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:07.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bd7fff640 1 --2- 172.21.15.148:0/2717489776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 0x7f9bd815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:07.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bd6ffd640 1 -- 172.21.15.148:0/2717489776 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267251989 0 0) 0x7f9bd80cb800 con 0x7f9bd80cf160 2023-10-30T13:46:07.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bd6ffd640 1 -- 172.21.15.148:0/2717489776 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc0003530 con 0x7f9bd80cf160 2023-10-30T13:46:07.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bd77fe640 1 -- 172.21.15.148:0/2717489776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 msgr2=0x7f9bd815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:07.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bd77fe640 1 --2- 172.21.15.148:0/2717489776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 0x7f9bd815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:07.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bd77fe640 1 -- 172.21.15.148:0/2717489776 >> v1:172.21.15.148:6789/0 conn(0x7f9bd80cf160 legacy=0x7f9bd80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:07.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.290+0000 7f9bd77fe640 1 -- 172.21.15.148:0/2717489776 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd80cad70 con 0x7f9bd80ca0e0 2023-10-30T13:46:07.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bd77fe640 1 --2- 172.21.15.148:0/2717489776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bd80ca0e0 0x7f9bd815a090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0097c0 tx=0x7f9bcc01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb1e11c879cd8a95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:07.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bd6ffd640 1 -- 172.21.15.148:0/2717489776 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc0a8970 con 0x7f9bd80ca0e0 2023-10-30T13:46:07.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bd6ffd640 1 -- 172.21.15.148:0/2717489776 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc0a8ad0 con 0x7f9bd80ca0e0 2023-10-30T13:46:07.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bd6ffd640 1 -- 172.21.15.148:0/2717489776 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc0a8e40 con 0x7f9bd80ca0e0 2023-10-30T13:46:07.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bde73e640 1 -- 172.21.15.148:0/2717489776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bd80ca0e0 msgr2=0x7f9bd815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:07.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bde73e640 1 --2- 172.21.15.148:0/2717489776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bd80ca0e0 0x7f9bd815a090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0097c0 tx=0x7f9bcc01ec70 comp rx=0 tx=0).stop 2023-10-30T13:46:07.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bde73e640 1 -- 172.21.15.148:0/2717489776 shutdown_connections 2023-10-30T13:46:07.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bde73e640 1 --2- 172.21.15.148:0/2717489776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bd80ca0e0 0x7f9bd815a090 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:07.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bde73e640 1 --2- 172.21.15.148:0/2717489776 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 0x7f9bd815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:07.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bde73e640 1 -- 172.21.15.148:0/2717489776 >> 172.21.15.148:0/2717489776 conn(0x7f9bd806bfe0 msgr2=0x7f9bd806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:07.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bde73e640 1 -- 172.21.15.148:0/2717489776 shutdown_connections 2023-10-30T13:46:07.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.294+0000 7f9bde73e640 1 -- 172.21.15.148:0/2717489776 wait complete. 2023-10-30T13:46:07.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bde73e640 1 Processor -- start 2023-10-30T13:46:07.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bde73e640 1 -- start start 2023-10-30T13:46:07.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bde73e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bd80ca0e0 0x7f9bd814e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:07.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bde73e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bd80cf160 0x7f9bd8150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:07.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bde73e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 0x7f9bd815cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:07.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bde73e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80b4ba0 con 0x7f9bd80cf160 2023-10-30T13:46:07.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bde73e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd814ea60 con 0x7f9bd80ca0e0 2023-10-30T13:46:07.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bde73e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd815c6c0 con 0x7f9bd81538a0 2023-10-30T13:46:07.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bd80cf160 0x7f9bd8150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:07.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bd80cf160 0x7f9bd8150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39194/0 (socket says 172.21.15.148:39194) 2023-10-30T13:46:07.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd77fe640 1 -- 172.21.15.148:0/2673569959 learned_addr learned my addr 172.21.15.148:0/2673569959 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:07.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd7fff640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bd80ca0e0 0x7f9bd814e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:07.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bdccdb640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 0x7f9bd815cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:07.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd77fe640 1 -- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 msgr2=0x7f9bd815cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:07.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd77fe640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 0x7f9bd815cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:07.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd77fe640 1 -- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bd80ca0e0 msgr2=0x7f9bd814e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:07.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd77fe640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bd80ca0e0 0x7f9bd814e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:07.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd77fe640 1 -- 172.21.15.148:0/2673569959 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc009420 con 0x7f9bd80cf160 2023-10-30T13:46:07.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd77fe640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bd80cf160 0x7f9bd8150440 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc01c8a0 tx=0x7f9bcc01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:07.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.298+0000 7f9bd4ff9640 1 -- 172.21.15.148:0/2673569959 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc004450 con 0x7f9bd80cf160 2023-10-30T13:46:07.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.302+0000 7f9bd4ff9640 1 -- 172.21.15.148:0/2673569959 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc003620 con 0x7f9bd80cf160 2023-10-30T13:46:07.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.302+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd80b4300 con 0x7f9bd80cf160 2023-10-30T13:46:07.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.302+0000 7f9bd4ff9640 1 -- 172.21.15.148:0/2673569959 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc003990 con 0x7f9bd80cf160 2023-10-30T13:46:07.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.302+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd80b48c0 con 0x7f9bd80cf160 2023-10-30T13:46:07.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.302+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd80b2a50 con 0x7f9bd80cf160 2023-10-30T13:46:07.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.302+0000 7f9bd4ff9640 1 -- 172.21.15.148:0/2673569959 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9bcc0b7040 con 0x7f9bd80cf160 2023-10-30T13:46:07.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.302+0000 7f9bd4ff9640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9bc40664d0 0x7f9bc4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:07.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.302+0000 7f9bd4ff9640 1 -- 172.21.15.148:0/2673569959 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f9bcc121ab0 con 0x7f9bd80cf160 2023-10-30T13:46:07.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.302+0000 7f9bd7fff640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9bc40664d0 0x7f9bc4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:07.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.306+0000 7f9bd7fff640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9bc40664d0 0x7f9bc4068990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009b60 tx=0x7f9bc8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:07.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.306+0000 7f9bd4ff9640 1 -- 172.21.15.148:0/2673569959 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9bcc0ee9c0 con 0x7f9bd80cf160 2023-10-30T13:46:07.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.486+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bd815c870 con 0x7f9bd80cf160 2023-10-30T13:46:07.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bd4ff9640 1 -- 172.21.15.148:0/2673569959 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f9bd815c870 con 0x7f9bd80cf160 2023-10-30T13:46:07.491 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:07.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:05.033315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:resolve","state_seq":347,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:07.491 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 136 2023-10-30T13:46:07.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9bc40664d0 msgr2=0x7f9bc4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:07.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9bc40664d0 0x7f9bc4068990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009b60 tx=0x7f9bc8009190 comp rx=0 tx=0).stop 2023-10-30T13:46:07.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bd80cf160 msgr2=0x7f9bd8150440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:07.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bd80cf160 0x7f9bd8150440 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc01c8a0 tx=0x7f9bcc01ba10 comp rx=0 tx=0).stop 2023-10-30T13:46:07.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 shutdown_connections 2023-10-30T13:46:07.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9bc40664d0 0x7f9bc4068990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:07.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9bd80cf160 0x7f9bd8150440 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:07.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9bd81538a0 0x7f9bd815cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:07.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 --2- 172.21.15.148:0/2673569959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bd80ca0e0 0x7f9bd814e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:07.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 >> 172.21.15.148:0/2673569959 conn(0x7f9bd806bfe0 msgr2=0x7f9bd81524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:07.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 shutdown_connections 2023-10-30T13:46:07.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:07.490+0000 7f9bde73e640 1 -- 172.21.15.148:0/2673569959 wait complete. 2023-10-30T13:46:07.512 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 136, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 136, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5453": { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 135, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:resolve", "state_seq": 347 }, "gid_7611": { "addr": "172.21.15.148:6835/1228827025", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1228827025, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1228827025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7611, "incarnation": 133, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:46:05.033315+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5453, "mds_1": 7611 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:46:09.513 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:09.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.658+0000 7f02fdaad640 1 Processor -- start 2023-10-30T13:46:09.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.658+0000 7f02fdaad640 1 -- start start 2023-10-30T13:46:09.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fdaad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815cd50 0x7f02f80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:09.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fdaad640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f815a6c0 con 0x7f02f81538f0 2023-10-30T13:46:09.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fdaad640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f815d300 con 0x7f02f80cefa0 2023-10-30T13:46:09.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fdaad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f815d5e0 con 0x7f02f815cd50 2023-10-30T13:46:09.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fcaab640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f02f81538f0 0x7f02f8159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51650/0 (socket says 172.21.15.148:51650) 2023-10-30T13:46:09.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fcaab640 1 -- 172.21.15.148:0/3231165136 learned_addr learned my addr 172.21.15.148:0/3231165136 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:46:09.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02effff640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815cd50 0x7f02f80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:09.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543315422 0 0) 0x7f02f815a6c0 con 0x7f02f81538f0 2023-10-30T13:46:09.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e0003530 con 0x7f02f81538f0 2023-10-30T13:46:09.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837867801 0 0) 0x7f02f815d300 con 0x7f02f80cefa0 2023-10-30T13:46:09.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f815a6c0 con 0x7f02f80cefa0 2023-10-30T13:46:09.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4161739971 0 0) 0x7f02e0003530 con 0x7f02f81538f0 2023-10-30T13:46:09.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815cd50 msgr2=0x7f02f80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:09.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815cd50 0x7f02f80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 >> v1:172.21.15.150:6789/0 conn(0x7f02f80cefa0 legacy=0x7f02f81640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f815d8c0 con 0x7f02f81538f0 2023-10-30T13:46:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f02f4001680 con 0x7f02f81538f0 2023-10-30T13:46:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 >> v1:172.21.15.148:6789/0 conn(0x7f02f81538f0 legacy=0x7f02f8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:09.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02e0003b40 0x7f02e0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815cd50 0x7f02e00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f815d5e0 con 0x7f02e0004d10 2023-10-30T13:46:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e000ba70 con 0x7f02e0003b40 2023-10-30T13:46:09.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e000bc30 con 0x7f02f815cd50 2023-10-30T13:46:09.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fd2ac640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:09.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fcaab640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02e0003b40 0x7f02e0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:09.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02effff640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815cd50 0x7f02e00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:09.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fd2ac640 1 -- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815cd50 msgr2=0x7f02e00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:09.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fd2ac640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815cd50 0x7f02e00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fd2ac640 1 -- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02e0003b40 msgr2=0x7f02e0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:09.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fd2ac640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02e0003b40 0x7f02e0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.662+0000 7f02fd2ac640 1 -- 172.21.15.148:0/3231165136 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e0003530 con 0x7f02e0004d10 2023-10-30T13:46:09.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02fd2ac640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02e0005190 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f02f0001fd0 tx=0x7f02f001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366b903d45e7e566 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:09.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f0003b80 con 0x7f02e0004d10 2023-10-30T13:46:09.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f0003d00 con 0x7f02e0004d10 2023-10-30T13:46:09.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02ef7fe640 1 -- 172.21.15.148:0/3231165136 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f0005310 con 0x7f02e0004d10 2023-10-30T13:46:09.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02fdaad640 1 -- 172.21.15.148:0/3231165136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 msgr2=0x7f02e0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:09.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02fdaad640 1 --2- 172.21.15.148:0/3231165136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02e0005190 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f02f0001fd0 tx=0x7f02f001c8e0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02fd2ac640 1 -- 172.21.15.148:0/3231165136 reap_dead start 2023-10-30T13:46:09.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02fdaad640 1 -- 172.21.15.148:0/3231165136 shutdown_connections 2023-10-30T13:46:09.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02fdaad640 1 -- 172.21.15.148:0/3231165136 >> 172.21.15.148:0/3231165136 conn(0x7f02f806a640 msgr2=0x7f02f8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:09.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02fdaad640 1 -- 172.21.15.148:0/3231165136 shutdown_connections 2023-10-30T13:46:09.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.666+0000 7f02fdaad640 1 -- 172.21.15.148:0/3231165136 wait complete. 2023-10-30T13:46:09.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 Processor -- start 2023-10-30T13:46:09.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 -- start start 2023-10-30T13:46:09.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02f81f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:09.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80b3d50 0x7f02f8147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:09.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815c600 0x7f02f8145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:09.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f815c1f0 con 0x7f02e0004d10 2023-10-30T13:46:09.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f81f2800 con 0x7f02f80b3d50 2023-10-30T13:46:09.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f815b970 con 0x7f02f815c600 2023-10-30T13:46:09.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fcaab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02f81f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:09.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fcaab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02f81f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39220/0 (socket says 172.21.15.148:39220) 2023-10-30T13:46:09.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fcaab640 1 -- 172.21.15.148:0/2002813314 learned_addr learned my addr 172.21.15.148:0/2002813314 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:09.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fd2ac640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815c600 0x7f02f8145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:09.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02effff640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80b3d50 0x7f02f8147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:09.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fcaab640 1 -- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815c600 msgr2=0x7f02f8145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:09.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fcaab640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815c600 0x7f02f8145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fcaab640 1 -- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80b3d50 msgr2=0x7f02f8147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:09.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fcaab640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80b3d50 0x7f02f8147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fcaab640 1 -- 172.21.15.148:0/2002813314 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f815bc30 con 0x7f02e0004d10 2023-10-30T13:46:09.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fd2ac640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815c600 0x7f02f8145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:46:09.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fcaab640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02f81f2c90 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f02f40072c0 tx=0x7f02f4007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:09.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02ed7fa640 1 -- 172.21.15.148:0/2002813314 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f400f960 con 0x7f02e0004d10 2023-10-30T13:46:09.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02ed7fa640 1 -- 172.21.15.148:0/2002813314 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f400fb00 con 0x7f02e0004d10 2023-10-30T13:46:09.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02ed7fa640 1 -- 172.21.15.148:0/2002813314 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f400feb0 con 0x7f02e0004d10 2023-10-30T13:46:09.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f815bf10 con 0x7f02e0004d10 2023-10-30T13:46:09.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f815a920 con 0x7f02e0004d10 2023-10-30T13:46:09.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.670+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f804e170 con 0x7f02e0004d10 2023-10-30T13:46:09.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.674+0000 7f02ed7fa640 1 -- 172.21.15.148:0/2002813314 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02f401d460 con 0x7f02e0004d10 2023-10-30T13:46:09.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.674+0000 7f02ed7fa640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02e40664a0 0x7f02e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:09.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.674+0000 7f02ed7fa640 1 -- 172.21.15.148:0/2002813314 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f02f4084800 con 0x7f02e0004d10 2023-10-30T13:46:09.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.674+0000 7f02effff640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02e40664a0 0x7f02e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:09.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.674+0000 7f02effff640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02e40664a0 0x7f02e4068960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f02f814ea80 tx=0x7f02e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:09.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.678+0000 7f02ed7fa640 1 -- 172.21.15.148:0/2002813314 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02f4051540 con 0x7f02e0004d10 2023-10-30T13:46:09.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.862+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02f80b1c10 con 0x7f02e0004d10 2023-10-30T13:46:09.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.862+0000 7f02ed7fa640 1 -- 172.21.15.148:0/2002813314 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5736 (secure 0 0 0) 0x7f02f4054b60 con 0x7f02e0004d10 2023-10-30T13:46:09.866 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:09.867 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:05.033315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:resolve","state_seq":347,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:09.867 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 136 2023-10-30T13:46:09.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02e40664a0 msgr2=0x7f02e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:09.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02e40664a0 0x7f02e4068960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f02f814ea80 tx=0x7f02e8008040 comp rx=0 tx=0).stop 2023-10-30T13:46:09.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 msgr2=0x7f02f81f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:09.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02f81f2c90 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f02f40072c0 tx=0x7f02f4007780 comp rx=0 tx=0).stop 2023-10-30T13:46:09.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 shutdown_connections 2023-10-30T13:46:09.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02e40664a0 0x7f02e4068960 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02f815c600 0x7f02f8145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80b3d50 0x7f02f8147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 --2- 172.21.15.148:0/2002813314 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02e0004d10 0x7f02f81f2c90 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:09.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 >> 172.21.15.148:0/2002813314 conn(0x7f02f806a640 msgr2=0x7f02f80cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:09.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 shutdown_connections 2023-10-30T13:46:09.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:09.866+0000 7f02fdaad640 1 -- 172.21.15.148:0/2002813314 wait complete. 2023-10-30T13:46:09.887 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:11.889 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:12.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.037+0000 7f8796c32640 1 Processor -- start 2023-10-30T13:46:12.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.037+0000 7f8796c32640 1 -- start start 2023-10-30T13:46:12.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f8796c32640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8790069e90 0x7f879006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:12.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f8796c32640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8790153d80 con 0x7f87901539a0 2023-10-30T13:46:12.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f8796c32640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87900c5c50 con 0x7f8790070a00 2023-10-30T13:46:12.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f8796c32640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f879006fbb0 con 0x7f8790069e90 2023-10-30T13:46:12.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87949ce640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f87901539a0 0x7f879015a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:50458/0 (socket says 172.21.15.148:50458) 2023-10-30T13:46:12.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87949ce640 1 -- 172.21.15.148:0/2279798587 learned_addr learned my addr 172.21.15.148:0/2279798587 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:46:12.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87951cf640 1 --2- 172.21.15.148:0/2279798587 >> v2:172.21.15.150:3300/0 conn(0x7f8790069e90 0x7f879006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:12.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958533894 0 0) 0x7f8790153d80 con 0x7f87901539a0 2023-10-30T13:46:12.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8778003530 con 0x7f87901539a0 2023-10-30T13:46:12.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285160962 0 0) 0x7f8778003530 con 0x7f87901539a0 2023-10-30T13:46:12.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 >> v2:172.21.15.150:3300/0 conn(0x7f8790069e90 msgr2=0x7f879006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:12.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 --2- 172.21.15.148:0/2279798587 >> v2:172.21.15.150:3300/0 conn(0x7f8790069e90 0x7f879006a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 >> v1:172.21.15.150:6789/0 conn(0x7f8790070a00 legacy=0x7f8790070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:12.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87900c5f30 con 0x7f87901539a0 2023-10-30T13:46:12.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f878c001680 con 0x7f87901539a0 2023-10-30T13:46:12.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 >> v1:172.21.15.148:6789/0 conn(0x7f87901539a0 legacy=0x7f879015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:12.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8778003980 0x7f8778003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:12.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87780047d0 0x7f8778004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:12.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8778005190 0x7f877800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:12.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879006fbb0 con 0x7f8778005190 2023-10-30T13:46:12.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877800c130 con 0x7f87780047d0 2023-10-30T13:46:12.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877800c2f0 con 0x7f8778003980 2023-10-30T13:46:12.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87951cf640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8778005190 0x7f877800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:12.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f8787fff640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87780047d0 0x7f8778004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:12.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87949ce640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8778003980 0x7f8778003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:12.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87951cf640 1 -- 172.21.15.148:0/2279798587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8778003980 msgr2=0x7f8778003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:12.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87951cf640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8778003980 0x7f8778003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87951cf640 1 -- 172.21.15.148:0/2279798587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87780047d0 msgr2=0x7f8778004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:12.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87951cf640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87780047d0 0x7f8778004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.041+0000 7f87951cf640 1 -- 172.21.15.148:0/2279798587 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8778003530 con 0x7f8778005190 2023-10-30T13:46:12.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f87951cf640 1 -- 172.21.15.148:0/2279798587 reap_dead start 2023-10-30T13:46:12.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f87951cf640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8778005190 0x7f877800b990 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f8788009fc0 tx=0x7f878801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b4aca3e2947e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:12.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87880a5bb0 con 0x7f8778005190 2023-10-30T13:46:12.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878800f040 con 0x7f8778005190 2023-10-30T13:46:12.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f87877fe640 1 -- 172.21.15.148:0/2279798587 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87880039a0 con 0x7f8778005190 2023-10-30T13:46:12.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f8796c32640 1 -- 172.21.15.148:0/2279798587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8778005190 msgr2=0x7f877800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:12.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f8796c32640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8778005190 0x7f877800b990 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f8788009fc0 tx=0x7f878801cca0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f8796c32640 1 -- 172.21.15.148:0/2279798587 shutdown_connections 2023-10-30T13:46:12.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f8796c32640 1 --2- 172.21.15.148:0/2279798587 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8778005190 0x7f877800b990 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f8796c32640 1 -- 172.21.15.148:0/2279798587 >> 172.21.15.148:0/2279798587 conn(0x7f87900c7910 msgr2=0x7f87900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:12.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f8796c32640 1 -- 172.21.15.148:0/2279798587 shutdown_connections 2023-10-30T13:46:12.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f8796c32640 1 -- 172.21.15.148:0/2279798587 wait complete. 2023-10-30T13:46:12.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f8796c32640 1 Processor -- start 2023-10-30T13:46:12.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.045+0000 7f8796c32640 1 -- start start 2023-10-30T13:46:12.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8796c32640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8778005190 0x7f87900c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:12.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8796c32640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87901444c0 0x7f87901501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:12.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8796c32640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900c24e0 0x7f87900c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:12.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8796c32640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8790145b30 con 0x7f87900c24e0 2023-10-30T13:46:12.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8796c32640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87901495d0 con 0x7f8778005190 2023-10-30T13:46:12.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8796c32640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87901f2370 con 0x7f87901444c0 2023-10-30T13:46:12.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87951cf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900c24e0 0x7f87900c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:12.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87951cf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900c24e0 0x7f87900c9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60894/0 (socket says 172.21.15.148:60894) 2023-10-30T13:46:12.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87951cf640 1 -- 172.21.15.148:0/674016249 learned_addr learned my addr 172.21.15.148:0/674016249 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:12.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87949ce640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8778005190 0x7f87900c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:12.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8787fff640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87901444c0 0x7f87901501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:12.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87951cf640 1 -- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87901444c0 msgr2=0x7f87901501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:12.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87951cf640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87901444c0 0x7f87901501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87951cf640 1 -- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8778005190 msgr2=0x7f87900c4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:12.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87951cf640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8778005190 0x7f87900c4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87951cf640 1 -- 172.21.15.148:0/674016249 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87901f2610 con 0x7f87900c24e0 2023-10-30T13:46:12.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87951cf640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900c24e0 0x7f87900c9c60 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f87880987d0 tx=0x7f8788009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:12.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87857fa640 1 -- 172.21.15.148:0/674016249 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87880a5dc0 con 0x7f87900c24e0 2023-10-30T13:46:12.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87857fa640 1 -- 172.21.15.148:0/674016249 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87880039a0 con 0x7f87900c24e0 2023-10-30T13:46:12.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87901f28f0 con 0x7f87900c24e0 2023-10-30T13:46:12.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f87857fa640 1 -- 172.21.15.148:0/674016249 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788003d90 con 0x7f87900c24e0 2023-10-30T13:46:12.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8790149870 con 0x7f87900c24e0 2023-10-30T13:46:12.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.049+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87901f2e10 con 0x7f87900c24e0 2023-10-30T13:46:12.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.053+0000 7f87857fa640 1 -- 172.21.15.148:0/674016249 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87880ae920 con 0x7f87900c24e0 2023-10-30T13:46:12.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.053+0000 7f87857fa640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f877c0664d0 0x7f877c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:12.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.053+0000 7f87857fa640 1 -- 172.21.15.148:0/674016249 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f878811f090 con 0x7f87900c24e0 2023-10-30T13:46:12.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.053+0000 7f87949ce640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f877c0664d0 0x7f877c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:12.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.053+0000 7f87949ce640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f877c0664d0 0x7f877c068990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f878c000c10 tx=0x7f878c001c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:12.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.057+0000 7f87857fa640 1 -- 172.21.15.148:0/674016249 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87901f2e10 con 0x7f87900c24e0 2023-10-30T13:46:12.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.237+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87901f2e10 con 0x7f87900c24e0 2023-10-30T13:46:12.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.237+0000 7f87857fa640 1 -- 172.21.15.148:0/674016249 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f87901f2e10 con 0x7f87900c24e0 2023-10-30T13:46:12.238 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:12.238 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:11.122766+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:reconnect","state_seq":349,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:12.238 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 137 2023-10-30T13:46:12.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f877c0664d0 msgr2=0x7f877c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:12.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f877c0664d0 0x7f877c068990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f878c000c10 tx=0x7f878c001c10 comp rx=0 tx=0).stop 2023-10-30T13:46:12.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900c24e0 msgr2=0x7f87900c9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:12.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900c24e0 0x7f87900c9c60 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f87880987d0 tx=0x7f8788009750 comp rx=0 tx=0).stop 2023-10-30T13:46:12.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 shutdown_connections 2023-10-30T13:46:12.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f877c0664d0 0x7f877c068990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87900c24e0 0x7f87900c9c60 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f87901444c0 0x7f87901501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 --2- 172.21.15.148:0/674016249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8778005190 0x7f87900c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:12.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 >> 172.21.15.148:0/674016249 conn(0x7f87900c7910 msgr2=0x7f879006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:12.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 shutdown_connections 2023-10-30T13:46:12.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:12.241+0000 7f8796c32640 1 -- 172.21.15.148:0/674016249 wait complete. 2023-10-30T13:46:12.259 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:14.261 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:14.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.409+0000 7f0eb6394640 1 Processor -- start 2023-10-30T13:46:14.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.409+0000 7f0eb6394640 1 -- start start 2023-10-30T13:46:14.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eb6394640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0eb006a720 0x7f0eb006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:14.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eb6394640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0071290 con 0x7f0eb00bfec0 2023-10-30T13:46:14.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eb6394640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb0071570 con 0x7f0eb0159e60 2023-10-30T13:46:14.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eb6394640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb0071850 con 0x7f0eb006a720 2023-10-30T13:46:14.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaf7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0eb00bfec0 0x7f0eb00c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:50466/0 (socket says 172.21.15.148:50466) 2023-10-30T13:46:14.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaf7fe640 1 -- 172.21.15.148:0/2818267562 learned_addr learned my addr 172.21.15.148:0/2818267562 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:46:14.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eb4931640 1 --2- 172.21.15.148:0/2818267562 >> v2:172.21.15.150:3300/0 conn(0x7f0eb006a720 0x7f0eb006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:14.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369695099 0 0) 0x7f0eb0071290 con 0x7f0eb00bfec0 2023-10-30T13:46:14.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e98003530 con 0x7f0eb00bfec0 2023-10-30T13:46:14.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2976321170 0 0) 0x7f0e98003530 con 0x7f0eb00bfec0 2023-10-30T13:46:14.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 >> v2:172.21.15.150:3300/0 conn(0x7f0eb006a720 msgr2=0x7f0eb006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:14.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 --2- 172.21.15.148:0/2818267562 >> v2:172.21.15.150:3300/0 conn(0x7f0eb006a720 0x7f0eb006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 >> v1:172.21.15.150:6789/0 conn(0x7f0eb0159e60 legacy=0x7f0eb006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:14.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb0071b30 con 0x7f0eb00bfec0 2023-10-30T13:46:14.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f0ea4001680 con 0x7f0eb00bfec0 2023-10-30T13:46:14.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 >> v1:172.21.15.148:6789/0 conn(0x7f0eb00bfec0 legacy=0x7f0eb00c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:14.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e98003980 0x7f0e98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:14.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e980047d0 0x7f0e98004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:14.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e98005190 0x7f0e9800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:14.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb0071850 con 0x7f0e98003980 2023-10-30T13:46:14.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9800c040 con 0x7f0e980047d0 2023-10-30T13:46:14.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9800c200 con 0x7f0e98005190 2023-10-30T13:46:14.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaffff640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e98003980 0x7f0e98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:14.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eb4931640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e98005190 0x7f0e9800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:14.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaf7fe640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e980047d0 0x7f0e98004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:14.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaffff640 1 -- 172.21.15.148:0/2818267562 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e98005190 msgr2=0x7f0e9800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:14.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaffff640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0e98005190 0x7f0e9800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaffff640 1 -- 172.21.15.148:0/2818267562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e980047d0 msgr2=0x7f0e98004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:14.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaffff640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e980047d0 0x7f0e98004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eaffff640 1 -- 172.21.15.148:0/2818267562 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e98003530 con 0x7f0e98003980 2023-10-30T13:46:14.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.413+0000 7f0eb4931640 1 -- 172.21.15.148:0/2818267562 reap_dead start 2023-10-30T13:46:14.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eaffff640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e98003980 0x7f0e98003da0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0001780 tx=0x7f0ea001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7826beba0ece1341 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:14.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea00052d0 con 0x7f0e98003980 2023-10-30T13:46:14.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea00a7050 con 0x7f0e98003980 2023-10-30T13:46:14.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eaeffd640 1 -- 172.21.15.148:0/2818267562 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0003600 con 0x7f0e98003980 2023-10-30T13:46:14.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eb6394640 1 -- 172.21.15.148:0/2818267562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e98003980 msgr2=0x7f0e98003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:14.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eb6394640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e98003980 0x7f0e98003da0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0001780 tx=0x7f0ea001c400 comp rx=0 tx=0).stop 2023-10-30T13:46:14.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eb6394640 1 -- 172.21.15.148:0/2818267562 shutdown_connections 2023-10-30T13:46:14.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eb6394640 1 --2- 172.21.15.148:0/2818267562 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e98003980 0x7f0e98003da0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eb6394640 1 -- 172.21.15.148:0/2818267562 >> 172.21.15.148:0/2818267562 conn(0x7f0eb00c7910 msgr2=0x7f0eb00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:14.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eb6394640 1 -- 172.21.15.148:0/2818267562 shutdown_connections 2023-10-30T13:46:14.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eb6394640 1 -- 172.21.15.148:0/2818267562 wait complete. 2023-10-30T13:46:14.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.417+0000 7f0eb6394640 1 Processor -- start 2023-10-30T13:46:14.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 -- start start 2023-10-30T13:46:14.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e98003980 0x7f0eb01e1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:14.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0eb0158880 0x7f0eb014d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:14.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0eb01eed40 0x7f0eb00c41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:14.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb00c97a0 con 0x7f0eb0158880 2023-10-30T13:46:14.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb014ddd0 con 0x7f0e98003980 2023-10-30T13:46:14.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb014d8a0 con 0x7f0eb01eed40 2023-10-30T13:46:14.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0eb0158880 0x7f0eb014d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:14.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e98003980 0x7f0eb01e1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:14.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0eb0158880 0x7f0eb014d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60916/0 (socket says 172.21.15.148:60916) 2023-10-30T13:46:14.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaf7fe640 1 -- 172.21.15.148:0/3662769881 learned_addr learned my addr 172.21.15.148:0/3662769881 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:14.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb4931640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0eb01eed40 0x7f0eb00c41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:14.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaf7fe640 1 -- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0eb01eed40 msgr2=0x7f0eb00c41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:14.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaf7fe640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0eb01eed40 0x7f0eb00c41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaf7fe640 1 -- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e98003980 msgr2=0x7f0eb01e1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:14.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaf7fe640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e98003980 0x7f0eb01e1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaf7fe640 1 -- 172.21.15.148:0/3662769881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb01e1a50 con 0x7f0eb0158880 2023-10-30T13:46:14.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eaf7fe640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0eb0158880 0x7f0eb014d360 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40124c0 tx=0x7f0ea4012980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:14.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eacff9640 1 -- 172.21.15.148:0/3662769881 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4014810 con 0x7f0eb0158880 2023-10-30T13:46:14.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eacff9640 1 -- 172.21.15.148:0/3662769881 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea4014970 con 0x7f0eb0158880 2023-10-30T13:46:14.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eacff9640 1 -- 172.21.15.148:0/3662769881 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4014d20 con 0x7f0eb0158880 2023-10-30T13:46:14.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb01e1cf0 con 0x7f0eb0158880 2023-10-30T13:46:14.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb00c9bc0 con 0x7f0eb0158880 2023-10-30T13:46:14.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.421+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb00c9220 con 0x7f0eb0158880 2023-10-30T13:46:14.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.425+0000 7f0eacff9640 1 -- 172.21.15.148:0/3662769881 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ea40153e0 con 0x7f0eb0158880 2023-10-30T13:46:14.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.425+0000 7f0eacff9640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e9c066490 0x7f0e9c068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:14.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.425+0000 7f0eacff9640 1 -- 172.21.15.148:0/3662769881 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f0ea4086dd0 con 0x7f0eb0158880 2023-10-30T13:46:14.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.425+0000 7f0eaffff640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e9c066490 0x7f0e9c068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:14.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.425+0000 7f0eaffff640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e9c066490 0x7f0e9c068950 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0001750 tx=0x7f0ea00011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:14.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.429+0000 7f0eacff9640 1 -- 172.21.15.148:0/3662769881 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0eb00c9220 con 0x7f0eb0158880 2023-10-30T13:46:14.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0eb00c98e0 con 0x7f0eb0158880 2023-10-30T13:46:14.608 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eacff9640 1 -- 172.21.15.148:0/3662769881 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f0eb00c98e0 con 0x7f0eb0158880 2023-10-30T13:46:14.608 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:14.608 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:11.122766+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:reconnect","state_seq":349,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:14.609 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 137 2023-10-30T13:46:14.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e9c066490 msgr2=0x7f0e9c068950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:14.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e9c066490 0x7f0e9c068950 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0001750 tx=0x7f0ea00011a0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0eb0158880 msgr2=0x7f0eb014d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:14.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0eb0158880 0x7f0eb014d360 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40124c0 tx=0x7f0ea4012980 comp rx=0 tx=0).stop 2023-10-30T13:46:14.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 shutdown_connections 2023-10-30T13:46:14.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0e9c066490 0x7f0e9c068950 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0eb0158880 0x7f0eb014d360 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0eb01eed40 0x7f0eb00c41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 --2- 172.21.15.148:0/3662769881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e98003980 0x7f0eb01e1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:14.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 >> 172.21.15.148:0/3662769881 conn(0x7f0eb00c7910 msgr2=0x7f0eb014fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:14.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 shutdown_connections 2023-10-30T13:46:14.612 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:14.609+0000 7f0eb6394640 1 -- 172.21.15.148:0/3662769881 wait complete. 2023-10-30T13:46:14.630 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:16.631 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:16.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c32879640 1 Processor -- start 2023-10-30T13:46:16.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c32879640 1 -- start start 2023-10-30T13:46:16.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c32879640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c2c1538a0 0x7f3c2c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:16.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c32879640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3c2c0ca0e0 0x7f3c2c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:16.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c32879640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2c0cb800 con 0x7f3c2c0cf160 2023-10-30T13:46:16.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c32879640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2c15dc30 con 0x7f3c2c0ca0e0 2023-10-30T13:46:16.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c32879640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2c0cb330 con 0x7f3c2c1538a0 2023-10-30T13:46:16.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c2b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3c2c0ca0e0 0x7f3c2c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:16.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c2b7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3c2c0ca0e0 0x7f3c2c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60928/0 (socket says 172.21.15.148:60928) 2023-10-30T13:46:16.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c2b7fe640 1 -- 172.21.15.148:0/516098971 learned_addr learned my addr 172.21.15.148:0/516098971 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:16.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c2bfff640 1 --2- 172.21.15.148:0/516098971 >> v2:172.21.15.150:3300/0 conn(0x7f3c2c1538a0 0x7f3c2c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:16.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c2b7fe640 1 -- 172.21.15.148:0/516098971 >> v2:172.21.15.150:3300/0 conn(0x7f3c2c1538a0 msgr2=0x7f3c2c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:16.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c2b7fe640 1 --2- 172.21.15.148:0/516098971 >> v2:172.21.15.150:3300/0 conn(0x7f3c2c1538a0 0x7f3c2c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c2b7fe640 1 -- 172.21.15.148:0/516098971 >> v1:172.21.15.150:6789/0 conn(0x7f3c2c0cf160 legacy=0x7f3c2c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:16.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.781+0000 7f3c2b7fe640 1 -- 172.21.15.148:0/516098971 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c0cad70 con 0x7f3c2c0ca0e0 2023-10-30T13:46:16.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c2b7fe640 1 --2- 172.21.15.148:0/516098971 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c0ca0e0 0x7f3c2c15a090 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0cfd10 tx=0x7f3c2001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e765997be0fc35f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:16.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c2affd640 1 -- 172.21.15.148:0/516098971 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c200a89f0 con 0x7f3c2c0ca0e0 2023-10-30T13:46:16.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c2affd640 1 -- 172.21.15.148:0/516098971 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c200a8b50 con 0x7f3c2c0ca0e0 2023-10-30T13:46:16.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c2affd640 1 -- 172.21.15.148:0/516098971 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c200b0440 con 0x7f3c2c0ca0e0 2023-10-30T13:46:16.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c32879640 1 -- 172.21.15.148:0/516098971 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c0ca0e0 msgr2=0x7f3c2c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:16.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c32879640 1 --2- 172.21.15.148:0/516098971 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c0ca0e0 0x7f3c2c15a090 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0cfd10 tx=0x7f3c2001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c32879640 1 -- 172.21.15.148:0/516098971 shutdown_connections 2023-10-30T13:46:16.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c32879640 1 --2- 172.21.15.148:0/516098971 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c0ca0e0 0x7f3c2c15a090 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c32879640 1 --2- 172.21.15.148:0/516098971 >> v2:172.21.15.150:3300/0 conn(0x7f3c2c1538a0 0x7f3c2c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c32879640 1 -- 172.21.15.148:0/516098971 >> 172.21.15.148:0/516098971 conn(0x7f3c2c06bfe0 msgr2=0x7f3c2c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:16.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c32879640 1 -- 172.21.15.148:0/516098971 shutdown_connections 2023-10-30T13:46:16.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.785+0000 7f3c32879640 1 -- 172.21.15.148:0/516098971 wait complete. 2023-10-30T13:46:16.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 Processor -- start 2023-10-30T13:46:16.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 -- start start 2023-10-30T13:46:16.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c2c0ca0e0 0x7f3c2c1e3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:16.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c2c0cf160 0x7f3c2c1e3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:16.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c1538a0 0x7f3c2c157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:16.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c159650 con 0x7f3c2c1538a0 2023-10-30T13:46:16.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c1e2f20 con 0x7f3c2c0cf160 2023-10-30T13:46:16.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c1efc70 con 0x7f3c2c0ca0e0 2023-10-30T13:46:16.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c30e16640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c1538a0 0x7f3c2c157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:16.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c30e16640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c1538a0 0x7f3c2c157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60936/0 (socket says 172.21.15.148:60936) 2023-10-30T13:46:16.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c30e16640 1 -- 172.21.15.148:0/3626038173 learned_addr learned my addr 172.21.15.148:0/3626038173 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:16.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c2b7fe640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c2c0cf160 0x7f3c2c1e3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:16.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c2bfff640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c2c0ca0e0 0x7f3c2c1e3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:16.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c30e16640 1 -- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c2c0ca0e0 msgr2=0x7f3c2c1e3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:16.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c30e16640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c2c0ca0e0 0x7f3c2c1e3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c30e16640 1 -- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c2c0cf160 msgr2=0x7f3c2c1e3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:16.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c30e16640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c2c0cf160 0x7f3c2c1e3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c30e16640 1 -- 172.21.15.148:0/3626038173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c200094d0 con 0x7f3c2c1538a0 2023-10-30T13:46:16.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c30e16640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c1538a0 0x7f3c2c157b40 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400c6c0 tx=0x7f3c2400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:16.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c28ff9640 1 -- 172.21.15.148:0/3626038173 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24003b80 con 0x7f3c2c1538a0 2023-10-30T13:46:16.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c28ff9640 1 -- 172.21.15.148:0/3626038173 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c24003ce0 con 0x7f3c2c1538a0 2023-10-30T13:46:16.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2c0b4aa0 con 0x7f3c2c1538a0 2023-10-30T13:46:16.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c28ff9640 1 -- 172.21.15.148:0/3626038173 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c240052f0 con 0x7f3c2c1538a0 2023-10-30T13:46:16.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.789+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2c158e40 con 0x7f3c2c1538a0 2023-10-30T13:46:16.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.793+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c0b2a50 con 0x7f3c2c1538a0 2023-10-30T13:46:16.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.793+0000 7f3c28ff9640 1 -- 172.21.15.148:0/3626038173 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c24005450 con 0x7f3c2c1538a0 2023-10-30T13:46:16.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.793+0000 7f3c28ff9640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c180664d0 0x7f3c18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:16.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.793+0000 7f3c28ff9640 1 -- 172.21.15.148:0/3626038173 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f3c240841e0 con 0x7f3c2c1538a0 2023-10-30T13:46:16.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.793+0000 7f3c2bfff640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c180664d0 0x7f3c18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:16.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.793+0000 7f3c2bfff640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c180664d0 0x7f3c18068990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009a50 tx=0x7f3c1c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:16.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.797+0000 7f3c28ff9640 1 -- 172.21.15.148:0/3626038173 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c24050fb0 con 0x7f3c2c1538a0 2023-10-30T13:46:16.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c2c1e3060 con 0x7f3c2c1538a0 2023-10-30T13:46:16.975 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c28ff9640 1 -- 172.21.15.148:0/3626038173 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f3c240545d0 con 0x7f3c2c1538a0 2023-10-30T13:46:16.976 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:16.976 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:11.122766+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:reconnect","state_seq":349,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:16.976 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 137 2023-10-30T13:46:16.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c180664d0 msgr2=0x7f3c18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:16.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c180664d0 0x7f3c18068990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009a50 tx=0x7f3c1c008040 comp rx=0 tx=0).stop 2023-10-30T13:46:16.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c1538a0 msgr2=0x7f3c2c157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:16.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c1538a0 0x7f3c2c157b40 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400c6c0 tx=0x7f3c2400cb80 comp rx=0 tx=0).stop 2023-10-30T13:46:16.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 shutdown_connections 2023-10-30T13:46:16.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3c180664d0 0x7f3c18068990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c2c1538a0 0x7f3c2c157b40 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c2c0cf160 0x7f3c2c1e3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 --2- 172.21.15.148:0/3626038173 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3c2c0ca0e0 0x7f3c2c1e3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:16.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 >> 172.21.15.148:0/3626038173 conn(0x7f3c2c06bfe0 msgr2=0x7f3c2c0c45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:16.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 shutdown_connections 2023-10-30T13:46:16.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:16.977+0000 7f3c32879640 1 -- 172.21.15.148:0/3626038173 wait complete. 2023-10-30T13:46:16.998 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:18.999 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:19.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.149+0000 7f709ccea640 1 Processor -- start 2023-10-30T13:46:19.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.149+0000 7f709ccea640 1 -- start start 2023-10-30T13:46:19.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709ccea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70980c05d0 0x7f709813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:19.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709ccea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70980c0a50 con 0x7f70980c6f60 2023-10-30T13:46:19.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709ccea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709806f6e0 con 0x7f70980c6b80 2023-10-30T13:46:19.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709ccea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709806f9c0 con 0x7f70980c05d0 2023-10-30T13:46:19.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f7096d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f70980c6f60 0x7f70981566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:50480/0 (socket says 172.21.15.148:50480) 2023-10-30T13:46:19.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f7096d9d640 1 -- 172.21.15.148:0/3705652177 learned_addr learned my addr 172.21.15.148:0/3705652177 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:46:19.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f7095d9b640 1 --2- 172.21.15.148:0/3705652177 >> v2:172.21.15.150:3300/0 conn(0x7f70980c05d0 0x7f709813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:19.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778393741 0 0) 0x7f70980c0a50 con 0x7f70980c6f60 2023-10-30T13:46:19.153 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7080003530 con 0x7f70980c6f60 2023-10-30T13:46:19.154 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394208263 0 0) 0x7f709806f6e0 con 0x7f70980c6b80 2023-10-30T13:46:19.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70980c0a50 con 0x7f70980c6b80 2023-10-30T13:46:19.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2901340633 0 0) 0x7f7080003530 con 0x7f70980c6f60 2023-10-30T13:46:19.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 >> v2:172.21.15.150:3300/0 conn(0x7f70980c05d0 msgr2=0x7f709813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:19.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 --2- 172.21.15.148:0/3705652177 >> v2:172.21.15.150:3300/0 conn(0x7f70980c05d0 0x7f709813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 >> v1:172.21.15.150:6789/0 conn(0x7f70980c6b80 legacy=0x7f70980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:19.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709806fca0 con 0x7f70980c6f60 2023-10-30T13:46:19.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f7090001a00 con 0x7f70980c6f60 2023-10-30T13:46:19.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 >> v1:172.21.15.148:6789/0 conn(0x7f70980c6f60 legacy=0x7f70981566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:19.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7080003b10 0x7f7080003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:19.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70800047d0 0x7f7080004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:19.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080005190 0x7f708000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:19.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709806f9c0 con 0x7f7080003b10 2023-10-30T13:46:19.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708000e220 con 0x7f7080005190 2023-10-30T13:46:19.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708000e3e0 con 0x7f70800047d0 2023-10-30T13:46:19.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709659c640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7080003b10 0x7f7080003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:19.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f7095d9b640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70800047d0 0x7f7080004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:19.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f7096d9d640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080005190 0x7f708000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:19.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709659c640 1 -- 172.21.15.148:0/3705652177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70800047d0 msgr2=0x7f7080004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:19.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709659c640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70800047d0 0x7f7080004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709659c640 1 -- 172.21.15.148:0/3705652177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080005190 msgr2=0x7f708000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:19.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709659c640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080005190 0x7f708000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f709659c640 1 -- 172.21.15.148:0/3705652177 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080003530 con 0x7f7080003b10 2023-10-30T13:46:19.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f7096d9d640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080005190 0x7f708000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:46:19.159 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.153+0000 7f7096d9d640 1 -- 172.21.15.148:0/3705652177 reap_dead start 2023-10-30T13:46:19.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709659c640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7080003b10 0x7f7080003f10 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f70880018d0 tx=0x7f708801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b974b0fc526fbb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:19.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70880052d0 con 0x7f7080003b10 2023-10-30T13:46:19.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70880a7050 con 0x7f7080003b10 2023-10-30T13:46:19.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709559a640 1 -- 172.21.15.148:0/3705652177 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7088003600 con 0x7f7080003b10 2023-10-30T13:46:19.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709ccea640 1 -- 172.21.15.148:0/3705652177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7080003b10 msgr2=0x7f7080003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:19.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709ccea640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7080003b10 0x7f7080003f10 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f70880018d0 tx=0x7f708801c540 comp rx=0 tx=0).stop 2023-10-30T13:46:19.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709ccea640 1 -- 172.21.15.148:0/3705652177 shutdown_connections 2023-10-30T13:46:19.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709ccea640 1 --2- 172.21.15.148:0/3705652177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7080003b10 0x7f7080003f10 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709ccea640 1 -- 172.21.15.148:0/3705652177 >> 172.21.15.148:0/3705652177 conn(0x7f70980c7910 msgr2=0x7f70980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:19.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709ccea640 1 -- 172.21.15.148:0/3705652177 shutdown_connections 2023-10-30T13:46:19.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.157+0000 7f709ccea640 1 -- 172.21.15.148:0/3705652177 wait complete. 2023-10-30T13:46:19.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 Processor -- start 2023-10-30T13:46:19.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 -- start start 2023-10-30T13:46:19.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080003b10 0x7f70981f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:19.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70980c8e80 0x7f709813b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:19.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7098158f90 0x7f7098159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:19.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70980ca4a0 con 0x7f7098158f90 2023-10-30T13:46:19.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7098136d10 con 0x7f7080003b10 2023-10-30T13:46:19.164 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7098139920 con 0x7f70980c8e80 2023-10-30T13:46:19.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7096d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7098158f90 0x7f7098159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:19.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7096d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7098158f90 0x7f7098159410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60946/0 (socket says 172.21.15.148:60946) 2023-10-30T13:46:19.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7096d9d640 1 -- 172.21.15.148:0/1789181919 learned_addr learned my addr 172.21.15.148:0/1789181919 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:19.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7095d9b640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70980c8e80 0x7f709813b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:19.165 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709659c640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080003b10 0x7f70981f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:19.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7096d9d640 1 -- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70980c8e80 msgr2=0x7f709813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:19.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7096d9d640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70980c8e80 0x7f709813b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7096d9d640 1 -- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080003b10 msgr2=0x7f70981f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:19.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7096d9d640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080003b10 0x7f70981f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7096d9d640 1 -- 172.21.15.148:0/1789181919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70980c9560 con 0x7f7098158f90 2023-10-30T13:46:19.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709659c640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080003b10 0x7f70981f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:46:19.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7096d9d640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7098158f90 0x7f7098159410 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f70980b1190 tx=0x7f7090007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:19.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7086ffd640 1 -- 172.21.15.148:0/1789181919 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709000e960 con 0x7f7098158f90 2023-10-30T13:46:19.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7086ffd640 1 -- 172.21.15.148:0/1789181919 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709000eac0 con 0x7f7098158f90 2023-10-30T13:46:19.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70980c9820 con 0x7f7098158f90 2023-10-30T13:46:19.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f7086ffd640 1 -- 172.21.15.148:0/1789181919 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709000ee70 con 0x7f7098158f90 2023-10-30T13:46:19.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70980c9de0 con 0x7f7098158f90 2023-10-30T13:46:19.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.161+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f709804e170 con 0x7f7098158f90 2023-10-30T13:46:19.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.165+0000 7f7086ffd640 1 -- 172.21.15.148:0/1789181919 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f709001d460 con 0x7f7098158f90 2023-10-30T13:46:19.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.165+0000 7f7086ffd640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70780664d0 0x7f7078068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:19.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.165+0000 7f7086ffd640 1 -- 172.21.15.148:0/1789181919 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f70900846d0 con 0x7f7098158f90 2023-10-30T13:46:19.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.165+0000 7f709659c640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70780664d0 0x7f7078068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:19.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.165+0000 7f709659c640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70780664d0 0x7f7078068990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f70880ac000 tx=0x7f7088009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:19.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.169+0000 7f7086ffd640 1 -- 172.21.15.148:0/1789181919 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7090051540 con 0x7f7098158f90 2023-10-30T13:46:19.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.349+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7098137dc0 con 0x7f7098158f90 2023-10-30T13:46:19.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.349+0000 7f7086ffd640 1 -- 172.21.15.148:0/1789181919 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5735 (secure 0 0 0) 0x7f7090054b60 con 0x7f7098158f90 2023-10-30T13:46:19.350 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:19.351 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:16.172058+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:rejoin","state_seq":351,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:19.351 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 138 2023-10-30T13:46:19.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70780664d0 msgr2=0x7f7078068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:19.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70780664d0 0x7f7078068990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f70880ac000 tx=0x7f7088009040 comp rx=0 tx=0).stop 2023-10-30T13:46:19.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7098158f90 msgr2=0x7f7098159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:19.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7098158f90 0x7f7098159410 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f70980b1190 tx=0x7f7090007b00 comp rx=0 tx=0).stop 2023-10-30T13:46:19.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 shutdown_connections 2023-10-30T13:46:19.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f70780664d0 0x7f7078068990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7098158f90 0x7f7098159410 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f70980c8e80 0x7f709813b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 --2- 172.21.15.148:0/1789181919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7080003b10 0x7f70981f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:19.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 >> 172.21.15.148:0/1789181919 conn(0x7f70980c7910 msgr2=0x7f709815a730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:19.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 shutdown_connections 2023-10-30T13:46:19.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:19.353+0000 7f709ccea640 1 -- 172.21.15.148:0/1789181919 wait complete. 2023-10-30T13:46:19.372 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 138, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 138, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5453": { "addr": "172.21.15.150:6836/116502200", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 116502200, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 116502200, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5453, "incarnation": 135, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 351 }, "gid_7611": { "addr": "172.21.15.148:6835/1228827025", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1228827025, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1228827025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7611, "incarnation": 133, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:46:16.172058+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5453, "mds_1": 7611 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:46:21.374 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.521+0000 7f02ac032640 1 Processor -- start 2023-10-30T13:46:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.521+0000 7f02ac032640 1 -- start start 2023-10-30T13:46:21.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02ac032640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a414fd50 0x7f02a40c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:21.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02ac032640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02a4071960 0x7f02a40cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:21.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02ac032640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a41560a0 con 0x7f02a40cb550 2023-10-30T13:46:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02ac032640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a4156380 con 0x7f02a4071960 2023-10-30T13:46:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02ac032640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4156660 con 0x7f02a414fd50 2023-10-30T13:46:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02a9dce640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a414fd50 0x7f02a40c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02aa5cf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02a4071960 0x7f02a40cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02a9dce640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a414fd50 0x7f02a40c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:60754/0 (socket says 172.21.15.148:60754) 2023-10-30T13:46:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02a9dce640 1 -- 172.21.15.148:0/4058701700 learned_addr learned my addr 172.21.15.148:0/4058701700 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02a95cd640 1 --1- 172.21.15.148:0/4058701700 >> v1:172.21.15.150:6789/0 conn(0x7f02a40cb550 0x7f02a4071250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:53282/0 (socket says 172.21.15.148:53282) 2023-10-30T13:46:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02a9dce640 1 -- 172.21.15.148:0/4058701700 >> v2:172.21.15.150:3300/0 conn(0x7f02a4071960 msgr2=0x7f02a40cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02a9dce640 1 --2- 172.21.15.148:0/4058701700 >> v2:172.21.15.150:3300/0 conn(0x7f02a4071960 0x7f02a40cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02a9dce640 1 -- 172.21.15.148:0/4058701700 >> v1:172.21.15.150:6789/0 conn(0x7f02a40cb550 legacy=0x7f02a4071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.525+0000 7f02a9dce640 1 -- 172.21.15.148:0/4058701700 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a4156940 con 0x7f02a414fd50 2023-10-30T13:46:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02a9dce640 1 --2- 172.21.15.148:0/4058701700 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a414fd50 0x7f02a40c65d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009870 tx=0x7f02a001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73f47f23a1bda9f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02a8dcc640 1 -- 172.21.15.148:0/4058701700 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00ab650 con 0x7f02a414fd50 2023-10-30T13:46:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02a8dcc640 1 -- 172.21.15.148:0/4058701700 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a00ab7b0 con 0x7f02a414fd50 2023-10-30T13:46:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02a8dcc640 1 -- 172.21.15.148:0/4058701700 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00abb40 con 0x7f02a414fd50 2023-10-30T13:46:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02ac032640 1 -- 172.21.15.148:0/4058701700 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a414fd50 msgr2=0x7f02a40c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02ac032640 1 --2- 172.21.15.148:0/4058701700 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a414fd50 0x7f02a40c65d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009870 tx=0x7f02a001c510 comp rx=0 tx=0).stop 2023-10-30T13:46:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02ac032640 1 -- 172.21.15.148:0/4058701700 shutdown_connections 2023-10-30T13:46:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02ac032640 1 --2- 172.21.15.148:0/4058701700 >> v2:172.21.15.150:3300/0 conn(0x7f02a4071960 0x7f02a40cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02ac032640 1 --2- 172.21.15.148:0/4058701700 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a414fd50 0x7f02a40c65d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02ac032640 1 -- 172.21.15.148:0/4058701700 >> 172.21.15.148:0/4058701700 conn(0x7f02a406c6e0 msgr2=0x7f02a4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:21.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02ac032640 1 -- 172.21.15.148:0/4058701700 shutdown_connections 2023-10-30T13:46:21.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.529+0000 7f02ac032640 1 -- 172.21.15.148:0/4058701700 wait complete. 2023-10-30T13:46:21.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 Processor -- start 2023-10-30T13:46:21.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 -- start start 2023-10-30T13:46:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02a4071960 0x7f02a41598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a40cb550 0x7f02a4147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02a414fd50 0x7f02a4148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4158770 con 0x7f02a414fd50 2023-10-30T13:46:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4159e00 con 0x7f02a4071960 2023-10-30T13:46:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a415a290 con 0x7f02a40cb550 2023-10-30T13:46:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02aa5cf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02a414fd50 0x7f02a4148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02aa5cf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02a414fd50 0x7f02a4148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60962/0 (socket says 172.21.15.148:60962) 2023-10-30T13:46:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02aa5cf640 1 -- 172.21.15.148:0/881264048 learned_addr learned my addr 172.21.15.148:0/881264048 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02a9dce640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02a4071960 0x7f02a41598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02a95cd640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a40cb550 0x7f02a4147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:21.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02aa5cf640 1 -- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a40cb550 msgr2=0x7f02a4147bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:21.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02aa5cf640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a40cb550 0x7f02a4147bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02aa5cf640 1 -- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02a4071960 msgr2=0x7f02a41598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:21.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02aa5cf640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02a4071960 0x7f02a41598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02aa5cf640 1 -- 172.21.15.148:0/881264048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a00094d0 con 0x7f02a414fd50 2023-10-30T13:46:21.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02aa5cf640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02a414fd50 0x7f02a4148130 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f029c009db0 tx=0x7f029c0075e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:21.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f029a7fc640 1 -- 172.21.15.148:0/881264048 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c007e70 con 0x7f02a414fd50 2023-10-30T13:46:21.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f029a7fc640 1 -- 172.21.15.148:0/881264048 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c010040 con 0x7f02a414fd50 2023-10-30T13:46:21.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a41f3800 con 0x7f02a414fd50 2023-10-30T13:46:21.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f029a7fc640 1 -- 172.21.15.148:0/881264048 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c003be0 con 0x7f02a414fd50 2023-10-30T13:46:21.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a415a040 con 0x7f02a414fd50 2023-10-30T13:46:21.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.533+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a41f3990 con 0x7f02a414fd50 2023-10-30T13:46:21.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.537+0000 7f029a7fc640 1 -- 172.21.15.148:0/881264048 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f029c003d40 con 0x7f02a414fd50 2023-10-30T13:46:21.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.537+0000 7f029a7fc640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02880664a0 0x7f0288068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:21.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.537+0000 7f029a7fc640 1 -- 172.21.15.148:0/881264048 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f029c084630 con 0x7f02a414fd50 2023-10-30T13:46:21.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.537+0000 7f02a9dce640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02880664a0 0x7f0288068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:21.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.537+0000 7f02a9dce640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02880664a0 0x7f0288068960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f02a00af000 tx=0x7f02a00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:21.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.541+0000 7f029a7fc640 1 -- 172.21.15.148:0/881264048 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f029c051310 con 0x7f02a414fd50 2023-10-30T13:46:21.719 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02a404e170 con 0x7f02a414fd50 2023-10-30T13:46:21.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f029a7fc640 1 -- 172.21.15.148:0/881264048 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5735 (secure 0 0 0) 0x7f029c054930 con 0x7f02a414fd50 2023-10-30T13:46:21.720 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:21.721 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:16.172058+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:rejoin","state_seq":351,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:21.721 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 138 2023-10-30T13:46:21.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02880664a0 msgr2=0x7f0288068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02880664a0 0x7f0288068960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f02a00af000 tx=0x7f02a00a7000 comp rx=0 tx=0).stop 2023-10-30T13:46:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02a414fd50 msgr2=0x7f02a4148130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02a414fd50 0x7f02a4148130 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f029c009db0 tx=0x7f029c0075e0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 shutdown_connections 2023-10-30T13:46:21.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f02880664a0 0x7f0288068960 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02a414fd50 0x7f02a4148130 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f02a40cb550 0x7f02a4147bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 --2- 172.21.15.148:0/881264048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02a4071960 0x7f02a41598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:21.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 >> 172.21.15.148:0/881264048 conn(0x7f02a406c6e0 msgr2=0x7f02a40ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:21.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 shutdown_connections 2023-10-30T13:46:21.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:21.721+0000 7f02ac032640 1 -- 172.21.15.148:0/881264048 wait complete. 2023-10-30T13:46:21.741 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:23.742 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:23.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4f692a640 1 Processor -- start 2023-10-30T13:46:23.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4f692a640 1 -- start start 2023-10-30T13:46:23.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4f692a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4f01538a0 0x7fb4f015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:23.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4f692a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f00ca0e0 0x7fb4f015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:23.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4f692a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f00cb800 con 0x7fb4f00cf160 2023-10-30T13:46:23.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4f692a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f015dc30 con 0x7fb4f01538a0 2023-10-30T13:46:23.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4f692a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00cb330 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4ef7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f00ca0e0 0x7fb4f015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:23.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4ef7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f00ca0e0 0x7fb4f015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:36344/0 (socket says 172.21.15.148:36344) 2023-10-30T13:46:23.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4ef7fe640 1 -- 172.21.15.148:0/800027098 learned_addr learned my addr 172.21.15.148:0/800027098 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:23.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4effff640 1 --2- 172.21.15.148:0/800027098 >> v2:172.21.15.150:3300/0 conn(0x7fb4f01538a0 0x7fb4f015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:23.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4eeffd640 1 -- 172.21.15.148:0/800027098 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168035672 0 0) 0x7fb4f00cb800 con 0x7fb4f00cf160 2023-10-30T13:46:23.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4eeffd640 1 -- 172.21.15.148:0/800027098 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d8003530 con 0x7fb4f00cf160 2023-10-30T13:46:23.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4ef7fe640 1 -- 172.21.15.148:0/800027098 >> v2:172.21.15.150:3300/0 conn(0x7fb4f01538a0 msgr2=0x7fb4f015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:23.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4ef7fe640 1 --2- 172.21.15.148:0/800027098 >> v2:172.21.15.150:3300/0 conn(0x7fb4f01538a0 0x7fb4f015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:23.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4ef7fe640 1 -- 172.21.15.148:0/800027098 >> v1:172.21.15.150:6789/0 conn(0x7fb4f00cf160 legacy=0x7fb4f00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:23.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.893+0000 7fb4ef7fe640 1 -- 172.21.15.148:0/800027098 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f00cad70 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4ef7fe640 1 --2- 172.21.15.148:0/800027098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f00ca0e0 0x7fb4f015a090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4009870 tx=0x7fb4e401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85e214c463fca187 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:23.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4eeffd640 1 -- 172.21.15.148:0/800027098 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e40a8a30 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4eeffd640 1 -- 172.21.15.148:0/800027098 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e40a8b90 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4eeffd640 1 -- 172.21.15.148:0/800027098 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e40b0440 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4f692a640 1 -- 172.21.15.148:0/800027098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f00ca0e0 msgr2=0x7fb4f015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:23.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4f692a640 1 --2- 172.21.15.148:0/800027098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f00ca0e0 0x7fb4f015a090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4009870 tx=0x7fb4e401ed30 comp rx=0 tx=0).stop 2023-10-30T13:46:23.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4f692a640 1 -- 172.21.15.148:0/800027098 shutdown_connections 2023-10-30T13:46:23.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4f692a640 1 --2- 172.21.15.148:0/800027098 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f00ca0e0 0x7fb4f015a090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:23.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4f692a640 1 --2- 172.21.15.148:0/800027098 >> v2:172.21.15.150:3300/0 conn(0x7fb4f01538a0 0x7fb4f015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:23.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4f692a640 1 -- 172.21.15.148:0/800027098 >> 172.21.15.148:0/800027098 conn(0x7fb4f006bfe0 msgr2=0x7fb4f006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:23.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4f692a640 1 -- 172.21.15.148:0/800027098 shutdown_connections 2023-10-30T13:46:23.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.897+0000 7fb4f692a640 1 -- 172.21.15.148:0/800027098 wait complete. 2023-10-30T13:46:23.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 Processor -- start 2023-10-30T13:46:23.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 -- start start 2023-10-30T13:46:23.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4f00ca0e0 0x7fb4f0148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:23.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00cf160 0x7fb4f00b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:23.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f01538a0 0x7fb4f00b3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:23.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00cee30 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.902 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00b4b10 con 0x7fb4f00cf160 2023-10-30T13:46:23.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f01f30b0 con 0x7fb4f01538a0 2023-10-30T13:46:23.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4effff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4f00ca0e0 0x7fb4f0148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:23.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4effff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4f00ca0e0 0x7fb4f0148170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59924/0 (socket says 172.21.15.148:59924) 2023-10-30T13:46:23.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4effff640 1 -- 172.21.15.148:0/1798278934 learned_addr learned my addr 172.21.15.148:0/1798278934 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:23.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f4ec7640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f01538a0 0x7fb4f00b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:23.903 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4ef7fe640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00cf160 0x7fb4f00b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:23.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4effff640 1 -- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f01538a0 msgr2=0x7fb4f00b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:23.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4effff640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f01538a0 0x7fb4f00b3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:23.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4effff640 1 -- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00cf160 msgr2=0x7fb4f00b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:23.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4effff640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00cf160 0x7fb4f00b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:23.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4effff640 1 -- 172.21.15.148:0/1798278934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e40094d0 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4ef7fe640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00cf160 0x7fb4f00b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:46:23.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4effff640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4f00ca0e0 0x7fb4f0148170 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fb4e000e870 tx=0x7fb4e000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:23.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4ecff9640 1 -- 172.21.15.148:0/1798278934 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0009510 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4ecff9640 1 -- 172.21.15.148:0/1798278934 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e000eec0 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f00b4db0 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4ecff9640 1 -- 172.21.15.148:0/1798278934 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e000cb40 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.901+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f01f3390 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.905+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f00b2a50 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.905+0000 7fb4ecff9640 1 -- 172.21.15.148:0/1798278934 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4e000cca0 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.905+0000 7fb4ecff9640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4dc0664a0 0x7fb4dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:23.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.905+0000 7fb4ecff9640 1 -- 172.21.15.148:0/1798278934 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7fb4e0084660 con 0x7fb4f00ca0e0 2023-10-30T13:46:23.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.905+0000 7fb4ef7fe640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4dc0664a0 0x7fb4dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:23.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.905+0000 7fb4ef7fe640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4dc0664a0 0x7fb4dc068960 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb4e40b2000 tx=0x7fb4e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:23.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:23.909+0000 7fb4ecff9640 1 -- 172.21.15.148:0/1798278934 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4e0051450 con 0x7fb4f00ca0e0 2023-10-30T13:46:24.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4f004e170 con 0x7fb4f00ca0e0 2023-10-30T13:46:24.088 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4ecff9640 1 -- 172.21.15.148:0/1798278934 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fb4e0054a70 con 0x7fb4f00ca0e0 2023-10-30T13:46:24.088 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:24.088 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:23.017151+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:clientreplay","state_seq":353,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:24.089 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 139 2023-10-30T13:46:24.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4dc0664a0 msgr2=0x7fb4dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:24.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4dc0664a0 0x7fb4dc068960 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb4e40b2000 tx=0x7fb4e4008040 comp rx=0 tx=0).stop 2023-10-30T13:46:24.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4f00ca0e0 msgr2=0x7fb4f0148170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:24.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4f00ca0e0 0x7fb4f0148170 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fb4e000e870 tx=0x7fb4e000ec70 comp rx=0 tx=0).stop 2023-10-30T13:46:24.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 shutdown_connections 2023-10-30T13:46:24.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4dc0664a0 0x7fb4dc068960 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:24.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4f01538a0 0x7fb4f00b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:24.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00cf160 0x7fb4f00b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:24.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 --2- 172.21.15.148:0/1798278934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4f00ca0e0 0x7fb4f0148170 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:24.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 >> 172.21.15.148:0/1798278934 conn(0x7fb4f006bfe0 msgr2=0x7fb4f01524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:24.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 shutdown_connections 2023-10-30T13:46:24.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:24.089+0000 7fb4f692a640 1 -- 172.21.15.148:0/1798278934 wait complete. 2023-10-30T13:46:24.110 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:26.111 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:26.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c09246640 1 Processor -- start 2023-10-30T13:46:26.260 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c09246640 1 -- start start 2023-10-30T13:46:26.261 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c09246640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c0406e470 0x7f6c0406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:26.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c09246640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6c040c8490 0x7f6c040c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:26.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c09246640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0406afa0 con 0x7f6c04153d70 2023-10-30T13:46:26.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c09246640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0406d780 con 0x7f6c040c8490 2023-10-30T13:46:26.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c09246640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c04154150 con 0x7f6c0406e470 2023-10-30T13:46:26.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c02d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6c04153d70 0x7f6c040cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47288/0 (socket says 172.21.15.148:47288) 2023-10-30T13:46:26.262 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c02d9d640 1 -- 172.21.15.148:0/1722759385 learned_addr learned my addr 172.21.15.148:0/1722759385 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:46:26.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c0359e640 1 --2- 172.21.15.148:0/1722759385 >> v2:172.21.15.148:3300/0 conn(0x7f6c040c8490 0x7f6c040c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:26.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6bfbfff640 1 --2- 172.21.15.148:0/1722759385 >> v2:172.21.15.150:3300/0 conn(0x7f6c0406e470 0x7f6c0406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:26.263 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c0359e640 1 -- 172.21.15.148:0/1722759385 >> v2:172.21.15.150:3300/0 conn(0x7f6c0406e470 msgr2=0x7f6c0406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:26.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c0359e640 1 --2- 172.21.15.148:0/1722759385 >> v2:172.21.15.150:3300/0 conn(0x7f6c0406e470 0x7f6c0406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c0359e640 1 -- 172.21.15.148:0/1722759385 >> v1:172.21.15.148:6789/0 conn(0x7f6c04153d70 legacy=0x7f6c040cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:26.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.261+0000 7f6c0359e640 1 -- 172.21.15.148:0/1722759385 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0406e030 con 0x7f6c040c8490 2023-10-30T13:46:26.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c0359e640 1 --2- 172.21.15.148:0/1722759385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 0x7f6c040c8cc0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f6c04070230 tx=0x7f6bfc01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ae8470fa3ae751 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:26.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c0259c640 1 -- 172.21.15.148:0/1722759385 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc0a59a0 con 0x7f6c040c8490 2023-10-30T13:46:26.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c0259c640 1 -- 172.21.15.148:0/1722759385 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc0a5b00 con 0x7f6c040c8490 2023-10-30T13:46:26.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c0259c640 1 -- 172.21.15.148:0/1722759385 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc0a5e90 con 0x7f6c040c8490 2023-10-30T13:46:26.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c09246640 1 -- 172.21.15.148:0/1722759385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 msgr2=0x7f6c040c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:26.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c09246640 1 --2- 172.21.15.148:0/1722759385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 0x7f6c040c8cc0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f6c04070230 tx=0x7f6bfc01c8e0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c09246640 1 -- 172.21.15.148:0/1722759385 shutdown_connections 2023-10-30T13:46:26.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c09246640 1 --2- 172.21.15.148:0/1722759385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 0x7f6c040c8cc0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c09246640 1 --2- 172.21.15.148:0/1722759385 >> v2:172.21.15.150:3300/0 conn(0x7f6c0406e470 0x7f6c0406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c09246640 1 -- 172.21.15.148:0/1722759385 >> 172.21.15.148:0/1722759385 conn(0x7f6c04071ea0 msgr2=0x7f6c04065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:26.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c09246640 1 -- 172.21.15.148:0/1722759385 shutdown_connections 2023-10-30T13:46:26.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.265+0000 7f6c09246640 1 -- 172.21.15.148:0/1722759385 wait complete. 2023-10-30T13:46:26.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 Processor -- start 2023-10-30T13:46:26.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 -- start start 2023-10-30T13:46:26.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c0406e470 0x7f6c04149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:26.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 0x7f6c04142b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:26.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c04153d70 0x7f6c041484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:26.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c041f2d00 con 0x7f6c040c8490 2023-10-30T13:46:26.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c041f2fe0 con 0x7f6c04153d70 2023-10-30T13:46:26.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c041f32c0 con 0x7f6c0406e470 2023-10-30T13:46:26.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 0x7f6c04142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:26.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 0x7f6c04142b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59948/0 (socket says 172.21.15.148:59948) 2023-10-30T13:46:26.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfbfff640 1 -- 172.21.15.148:0/3031896820 learned_addr learned my addr 172.21.15.148:0/3031896820 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:26.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c0359e640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c04153d70 0x7f6c041484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:26.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c02d9d640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c0406e470 0x7f6c04149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:26.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfbfff640 1 -- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c0406e470 msgr2=0x7f6c04149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:26.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfbfff640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c0406e470 0x7f6c04149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfbfff640 1 -- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c04153d70 msgr2=0x7f6c041484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:26.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfbfff640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c04153d70 0x7f6c041484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfbfff640 1 -- 172.21.15.148:0/3031896820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc009a70 con 0x7f6c040c8490 2023-10-30T13:46:26.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c02d9d640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c0406e470 0x7f6c04149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:46:26.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfbfff640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 0x7f6c04142b30 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00d5f0 tx=0x7f6bec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:26.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfb7fe640 1 -- 172.21.15.148:0/3031896820 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0125f0 con 0x7f6c040c8490 2023-10-30T13:46:26.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfb7fe640 1 -- 172.21.15.148:0/3031896820 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bec012750 con 0x7f6c040c8490 2023-10-30T13:46:26.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6bfb7fe640 1 -- 172.21.15.148:0/3031896820 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec012ae0 con 0x7f6c040c8490 2023-10-30T13:46:26.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c041f35a0 con 0x7f6c040c8490 2023-10-30T13:46:26.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.269+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0414a0f0 con 0x7f6c040c8490 2023-10-30T13:46:26.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.273+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0404e170 con 0x7f6c040c8490 2023-10-30T13:46:26.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.273+0000 7f6bfb7fe640 1 -- 172.21.15.148:0/3031896820 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6bec012c40 con 0x7f6c040c8490 2023-10-30T13:46:26.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.273+0000 7f6bfb7fe640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6be40664a0 0x7f6be4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:26.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.273+0000 7f6bfb7fe640 1 -- 172.21.15.148:0/3031896820 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f6bec084990 con 0x7f6c040c8490 2023-10-30T13:46:26.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.273+0000 7f6c02d9d640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6be40664a0 0x7f6be4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:26.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.273+0000 7f6c02d9d640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6be40664a0 0x7f6be4068960 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4005890 tx=0x7f6bf4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:26.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.277+0000 7f6bfb7fe640 1 -- 172.21.15.148:0/3031896820 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6bec051800 con 0x7f6c040c8490 2023-10-30T13:46:26.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.461+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c0406c610 con 0x7f6c040c8490 2023-10-30T13:46:26.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.461+0000 7f6bfb7fe640 1 -- 172.21.15.148:0/3031896820 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f6bec007380 con 0x7f6c040c8490 2023-10-30T13:46:26.462 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:26.462 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:23.017151+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:clientreplay","state_seq":353,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:26.462 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 139 2023-10-30T13:46:26.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6be40664a0 msgr2=0x7f6be4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:26.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6be40664a0 0x7f6be4068960 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4005890 tx=0x7f6bf4009040 comp rx=0 tx=0).stop 2023-10-30T13:46:26.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 msgr2=0x7f6c04142b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:26.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 0x7f6c04142b30 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00d5f0 tx=0x7f6bec00dab0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 shutdown_connections 2023-10-30T13:46:26.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6be40664a0 0x7f6be4068960 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c040c8490 0x7f6c04142b30 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c04153d70 0x7f6c041484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 --2- 172.21.15.148:0/3031896820 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c0406e470 0x7f6c04149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:26.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 >> 172.21.15.148:0/3031896820 conn(0x7f6c04071ea0 msgr2=0x7f6c040cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:26.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 shutdown_connections 2023-10-30T13:46:26.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:26.465+0000 7f6c09246640 1 -- 172.21.15.148:0/3031896820 wait complete. 2023-10-30T13:46:26.483 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:28.485 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:28.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.637+0000 7f798a4ea640 1 Processor -- start 2023-10-30T13:46:28.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.637+0000 7f798a4ea640 1 -- start start 2023-10-30T13:46:28.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.637+0000 7f798a4ea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f798415d6f0 0x7f798415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:28.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.637+0000 7f798a4ea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79840cb800 con 0x7f79840cf160 2023-10-30T13:46:28.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.637+0000 7f798a4ea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79840cb330 con 0x7f79841538a0 2023-10-30T13:46:28.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.637+0000 7f798a4ea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79840cad70 con 0x7f798415d6f0 2023-10-30T13:46:28.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.637+0000 7f79837fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f798415d6f0 0x7f798415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:28.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.637+0000 7f79837fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f798415d6f0 0x7f798415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59960/0 (socket says 172.21.15.148:59960) 2023-10-30T13:46:28.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.637+0000 7f79837fe640 1 -- 172.21.15.148:0/2027807524 learned_addr learned my addr 172.21.15.148:0/2027807524 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:28.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.641+0000 7f79837fe640 1 -- 172.21.15.148:0/2027807524 >> v1:172.21.15.150:6789/0 conn(0x7f79841538a0 legacy=0x7f79840ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:28.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.641+0000 7f79837fe640 1 -- 172.21.15.148:0/2027807524 >> v1:172.21.15.148:6789/0 conn(0x7f79840cf160 legacy=0x7f7984164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:28.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.641+0000 7f79837fe640 1 -- 172.21.15.148:0/2027807524 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79840cb050 con 0x7f798415d6f0 2023-10-30T13:46:28.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.641+0000 7f79837fe640 1 --2- 172.21.15.148:0/2027807524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 0x7f798415a090 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f7978009870 tx=0x7f797801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e66582608b24400 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:28.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.641+0000 7f7982ffd640 1 -- 172.21.15.148:0/2027807524 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780a8a30 con 0x7f798415d6f0 2023-10-30T13:46:28.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.641+0000 7f7982ffd640 1 -- 172.21.15.148:0/2027807524 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79780a8b90 con 0x7f798415d6f0 2023-10-30T13:46:28.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.641+0000 7f7982ffd640 1 -- 172.21.15.148:0/2027807524 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780b0440 con 0x7f798415d6f0 2023-10-30T13:46:28.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 -- 172.21.15.148:0/2027807524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 msgr2=0x7f798415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:28.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 --2- 172.21.15.148:0/2027807524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 0x7f798415a090 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f7978009870 tx=0x7f797801ed30 comp rx=0 tx=0).stop 2023-10-30T13:46:28.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 -- 172.21.15.148:0/2027807524 shutdown_connections 2023-10-30T13:46:28.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 --2- 172.21.15.148:0/2027807524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 0x7f798415a090 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:28.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 -- 172.21.15.148:0/2027807524 >> 172.21.15.148:0/2027807524 conn(0x7f798406bfe0 msgr2=0x7f798406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:28.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 -- 172.21.15.148:0/2027807524 shutdown_connections 2023-10-30T13:46:28.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 -- 172.21.15.148:0/2027807524 wait complete. 2023-10-30T13:46:28.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 Processor -- start 2023-10-30T13:46:28.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 -- start start 2023-10-30T13:46:28.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79840cf160 0x7f79841e3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:28.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841538a0 0x7f79841592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:28.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 0x7f79840c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:28.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79840c8ce0 con 0x7f798415d6f0 2023-10-30T13:46:28.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79840c8310 con 0x7f79841538a0 2023-10-30T13:46:28.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f798a4ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79840c7060 con 0x7f79840cf160 2023-10-30T13:46:28.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f7988a87640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 0x7f79840c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:28.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f7988a87640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 0x7f79840c87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59962/0 (socket says 172.21.15.148:59962) 2023-10-30T13:46:28.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f7988a87640 1 -- 172.21.15.148:0/2646334055 learned_addr learned my addr 172.21.15.148:0/2646334055 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:28.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.645+0000 7f79837fe640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841538a0 0x7f79841592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:28.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7983fff640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79840cf160 0x7f79841e3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:28.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7988a87640 1 -- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79840cf160 msgr2=0x7f79841e3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:28.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7988a87640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79840cf160 0x7f79841e3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:28.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7988a87640 1 -- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841538a0 msgr2=0x7f79841592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:28.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7988a87640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841538a0 0x7f79841592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:28.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7988a87640 1 -- 172.21.15.148:0/2646334055 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79780094d0 con 0x7f798415d6f0 2023-10-30T13:46:28.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7983fff640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79840cf160 0x7f79841e3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:46:28.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7988a87640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 0x7f79840c87a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f797c00c730 tx=0x7f797c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7980ff9640 1 -- 172.21.15.148:0/2646334055 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c003b80 con 0x7f798415d6f0 2023-10-30T13:46:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7980ff9640 1 -- 172.21.15.148:0/2646334055 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c003ce0 con 0x7f798415d6f0 2023-10-30T13:46:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7980ff9640 1 -- 172.21.15.148:0/2646334055 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c005310 con 0x7f798415d6f0 2023-10-30T13:46:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79840c7d00 con 0x7f798415d6f0 2023-10-30T13:46:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79840c6aa0 con 0x7f798415d6f0 2023-10-30T13:46:28.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798404e170 con 0x7f798415d6f0 2023-10-30T13:46:28.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7980ff9640 1 -- 172.21.15.148:0/2646334055 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f797c005680 con 0x7f798415d6f0 2023-10-30T13:46:28.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7980ff9640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79700664a0 0x7f7970068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:28.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.649+0000 7f7980ff9640 1 -- 172.21.15.148:0/2646334055 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f797c084410 con 0x7f798415d6f0 2023-10-30T13:46:28.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.653+0000 7f7983fff640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79700664a0 0x7f7970068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:28.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.653+0000 7f7983fff640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79700664a0 0x7f7970068960 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7974000f70 tx=0x7f7974009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:28.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.653+0000 7f7980ff9640 1 -- 172.21.15.148:0/2646334055 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f797c051150 con 0x7f798415d6f0 2023-10-30T13:46:28.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.853+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79840c8450 con 0x7f798415d6f0 2023-10-30T13:46:28.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.853+0000 7f7980ff9640 1 -- 172.21.15.148:0/2646334055 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f797c054770 con 0x7f798415d6f0 2023-10-30T13:46:28.854 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:28.855 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:23.017151+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:clientreplay","state_seq":353,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:28.855 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 139 2023-10-30T13:46:28.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79700664a0 msgr2=0x7f7970068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:28.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79700664a0 0x7f7970068960 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7974000f70 tx=0x7f7974009040 comp rx=0 tx=0).stop 2023-10-30T13:46:28.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 msgr2=0x7f79840c87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:28.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 0x7f79840c87a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f797c00c730 tx=0x7f797c00cbf0 comp rx=0 tx=0).stop 2023-10-30T13:46:28.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 shutdown_connections 2023-10-30T13:46:28.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f79700664a0 0x7f7970068960 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:28.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f798415d6f0 0x7f79840c87a0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:28.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79841538a0 0x7f79841592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:28.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 --2- 172.21.15.148:0/2646334055 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f79840cf160 0x7f79841e3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:28.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 >> 172.21.15.148:0/2646334055 conn(0x7f798406bfe0 msgr2=0x7f79840ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:28.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 shutdown_connections 2023-10-30T13:46:28.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:28.857+0000 7f798a4ea640 1 -- 172.21.15.148:0/2646334055 wait complete. 2023-10-30T13:46:28.877 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:46:30.878 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:31.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7f980640 1 Processor -- start 2023-10-30T13:46:31.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7f980640 1 -- start start 2023-10-30T13:46:31.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7f980640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e78156f60 0x7f1e78071b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:31.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7f980640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e7806fd50 0x7f1e78072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:31.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7f980640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e78157340 con 0x7f1e7806afa0 2023-10-30T13:46:31.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7f980640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7806bea0 con 0x7f1e7806fd50 2023-10-30T13:46:31.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7f980640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e780caa80 con 0x7f1e78156f60 2023-10-30T13:46:31.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7df1d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f1e7806afa0 0x7f1e7815ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47296/0 (socket says 172.21.15.148:47296) 2023-10-30T13:46:31.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7df1d640 1 -- 172.21.15.148:0/1393440836 learned_addr learned my addr 172.21.15.148:0/1393440836 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:46:31.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7d71c640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e78156f60 0x7f1e78071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:31.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e7cf1b640 1 --2- 172.21.15.148:0/1393440836 >> v2:172.21.15.150:3300/0 conn(0x7f1e7806fd50 0x7f1e78072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:31.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118648984 0 0) 0x7f1e78157340 con 0x7f1e7806afa0 2023-10-30T13:46:31.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.029+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e64003530 con 0x7f1e7806afa0 2023-10-30T13:46:31.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3763624646 0 0) 0x7f1e64003530 con 0x7f1e7806afa0 2023-10-30T13:46:31.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e78156f60 msgr2=0x7f1e78071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:31.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e78156f60 0x7f1e78071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 >> v2:172.21.15.150:3300/0 conn(0x7f1e7806fd50 msgr2=0x7f1e78072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:31.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 --2- 172.21.15.148:0/1393440836 >> v2:172.21.15.150:3300/0 conn(0x7f1e7806fd50 0x7f1e78072050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.034 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7806ca30 con 0x7f1e7806afa0 2023-10-30T13:46:31.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f1e70001a00 con 0x7f1e7806afa0 2023-10-30T13:46:31.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 >> v1:172.21.15.148:6789/0 conn(0x7f1e7806afa0 legacy=0x7f1e7815ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:31.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e64003980 0x7f1e64003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:31.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e78156f60 0x7f1e640047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:31.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e64004d10 0x7f1e64005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:31.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7806bea0 con 0x7f1e64003980 2023-10-30T13:46:31.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e780caa80 con 0x7f1e64004d10 2023-10-30T13:46:31.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6400b990 con 0x7f1e78156f60 2023-10-30T13:46:31.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e7d71c640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e64003980 0x7f1e64003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:31.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e7cf1b640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e78156f60 0x7f1e640047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:31.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e7df1d640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e64004d10 0x7f1e64005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:31.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e7d71c640 1 -- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e78156f60 msgr2=0x7f1e640047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:31.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e7d71c640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e78156f60 0x7f1e640047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e7d71c640 1 -- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e64004d10 msgr2=0x7f1e64005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:31.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e7d71c640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e64004d10 0x7f1e64005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e7d71c640 1 -- 172.21.15.148:0/1393440836 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e64003530 con 0x7f1e64003980 2023-10-30T13:46:31.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.033+0000 7f1e7d71c640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e64003980 0x7f1e64003da0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009ae0 tx=0x7f1e7401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a451555b9e3334b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:31.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e740a5510 con 0x7f1e64003980 2023-10-30T13:46:31.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e740a5670 con 0x7f1e64003980 2023-10-30T13:46:31.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e6ffff640 1 -- 172.21.15.148:0/1393440836 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e740a5a20 con 0x7f1e64003980 2023-10-30T13:46:31.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 -- 172.21.15.148:0/1393440836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e64003980 msgr2=0x7f1e64003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:31.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 --2- 172.21.15.148:0/1393440836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e64003980 0x7f1e64003da0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009ae0 tx=0x7f1e7401ca30 comp rx=0 tx=0).stop 2023-10-30T13:46:31.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7df1d640 1 -- 172.21.15.148:0/1393440836 reap_dead start 2023-10-30T13:46:31.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 -- 172.21.15.148:0/1393440836 shutdown_connections 2023-10-30T13:46:31.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 -- 172.21.15.148:0/1393440836 >> 172.21.15.148:0/1393440836 conn(0x7f1e78069e90 msgr2=0x7f1e7806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:31.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 -- 172.21.15.148:0/1393440836 shutdown_connections 2023-10-30T13:46:31.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 -- 172.21.15.148:0/1393440836 wait complete. 2023-10-30T13:46:31.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 Processor -- start 2023-10-30T13:46:31.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 -- start start 2023-10-30T13:46:31.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e64003980 0x7f1e7814e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:31.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e780b1b20 0x7f1e781f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:31.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e78160a90 0x7f1e781f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:31.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e781f1c40 con 0x7f1e78160a90 2023-10-30T13:46:31.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e781f6120 con 0x7f1e780b1b20 2023-10-30T13:46:31.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7f980640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e781f52c0 con 0x7f1e64003980 2023-10-30T13:46:31.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.037+0000 7f1e7df1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e78160a90 0x7f1e781f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:31.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7df1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e78160a90 0x7f1e781f2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59976/0 (socket says 172.21.15.148:59976) 2023-10-30T13:46:31.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7df1d640 1 -- 172.21.15.148:0/3390190535 learned_addr learned my addr 172.21.15.148:0/3390190535 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:31.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7d71c640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e64003980 0x7f1e7814e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:31.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7cf1b640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e780b1b20 0x7f1e781f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:31.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7df1d640 1 -- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e64003980 msgr2=0x7f1e7814e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:31.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7df1d640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e64003980 0x7f1e7814e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7df1d640 1 -- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e780b1b20 msgr2=0x7f1e781f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:31.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7df1d640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e780b1b20 0x7f1e781f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7df1d640 1 -- 172.21.15.148:0/3390190535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e781f55a0 con 0x7f1e78160a90 2023-10-30T13:46:31.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7d71c640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e64003980 0x7f1e7814e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:46:31.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7df1d640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e78160a90 0x7f1e781f2590 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f1e7806ab20 tx=0x7f1e70007ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:31.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e6dffb640 1 -- 172.21.15.148:0/3390190535 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7000e9e0 con 0x7f1e78160a90 2023-10-30T13:46:31.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e6dffb640 1 -- 172.21.15.148:0/3390190535 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7000eb60 con 0x7f1e78160a90 2023-10-30T13:46:31.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e781f5880 con 0x7f1e78160a90 2023-10-30T13:46:31.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e6dffb640 1 -- 172.21.15.148:0/3390190535 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70013420 con 0x7f1e78160a90 2023-10-30T13:46:31.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e781f5e40 con 0x7f1e78160a90 2023-10-30T13:46:31.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7804e170 con 0x7f1e78160a90 2023-10-30T13:46:31.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e6dffb640 1 -- 172.21.15.148:0/3390190535 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1e70013580 con 0x7f1e78160a90 2023-10-30T13:46:31.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e6dffb640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e5c0664f0 0x7f1e5c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:31.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.041+0000 7f1e6dffb640 1 -- 172.21.15.148:0/3390190535 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f1e70084740 con 0x7f1e78160a90 2023-10-30T13:46:31.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.045+0000 7f1e7d71c640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e5c0664f0 0x7f1e5c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:31.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.045+0000 7f1e7d71c640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e5c0664f0 0x7f1e5c0689b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1e740af000 tx=0x7f1e740046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:31.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.049+0000 7f1e6dffb640 1 -- 172.21.15.148:0/3390190535 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1e70051590 con 0x7f1e78160a90 2023-10-30T13:46:31.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.229+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e781f1db0 con 0x7f1e78160a90 2023-10-30T13:46:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.229+0000 7f1e6dffb640 1 -- 172.21.15.148:0/3390190535 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5735 (secure 0 0 0) 0x7f1e70054bb0 con 0x7f1e78160a90 2023-10-30T13:46:31.229 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:31.229 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:30.174493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 140 2023-10-30T13:46:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e5c0664f0 msgr2=0x7f1e5c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e5c0664f0 0x7f1e5c0689b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1e740af000 tx=0x7f1e740046d0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e78160a90 msgr2=0x7f1e781f2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e78160a90 0x7f1e781f2590 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f1e7806ab20 tx=0x7f1e70007ae0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 shutdown_connections 2023-10-30T13:46:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e5c0664f0 0x7f1e5c0689b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e78160a90 0x7f1e781f2590 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e780b1b20 0x7f1e781f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 --2- 172.21.15.148:0/3390190535 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1e64003980 0x7f1e7814e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 >> 172.21.15.148:0/3390190535 conn(0x7f1e78069e90 msgr2=0x7f1e780c7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 shutdown_connections 2023-10-30T13:46:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:31.233+0000 7f1e7f980640 1 -- 172.21.15.148:0/3390190535 wait complete. 2023-10-30T13:46:31.250 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=7464 2023-10-30T13:46:31.251 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.c 2023-10-30T13:46:32.252 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:46:32.252 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:46:32.252 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:46:32.256 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:46:32.257 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:32.349 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:46:32.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.405+0000 7f3f9eed6640 1 Processor -- start 2023-10-30T13:46:32.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.405+0000 7f3f9eed6640 1 -- start start 2023-10-30T13:46:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9eed6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3f9806f7a0 0x7f3f9806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9eed6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3f98071110 0x7f3f98158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9eed6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f98066cf0 con 0x7f3f980c6bd0 2023-10-30T13:46:32.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9eed6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f980700c0 con 0x7f3f98071110 2023-10-30T13:46:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9eed6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f980c70d0 con 0x7f3f9806f7a0 2023-10-30T13:46:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9d473640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3f98071110 0x7f3f98158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9d473640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3f98071110 0x7f3f98158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32856/0 (socket says 172.21.15.148:32856) 2023-10-30T13:46:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9d473640 1 -- 172.21.15.148:0/458335110 learned_addr learned my addr 172.21.15.148:0/458335110 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:32.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9cc72640 1 --2- 172.21.15.148:0/458335110 >> v2:172.21.15.150:3300/0 conn(0x7f3f9806f7a0 0x7f3f9806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:32.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9d473640 1 -- 172.21.15.148:0/458335110 >> v2:172.21.15.150:3300/0 conn(0x7f3f9806f7a0 msgr2=0x7f3f9806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:32.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9d473640 1 --2- 172.21.15.148:0/458335110 >> v2:172.21.15.150:3300/0 conn(0x7f3f9806f7a0 0x7f3f9806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9d473640 1 -- 172.21.15.148:0/458335110 >> v1:172.21.15.148:6789/0 conn(0x7f3f980c6bd0 legacy=0x7f3f98069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:32.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f9d473640 1 -- 172.21.15.148:0/458335110 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f980c7390 con 0x7f3f98071110 2023-10-30T13:46:32.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.409+0000 7f3f977fe640 1 -- 172.21.15.148:0/458335110 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334766597 0 0) 0x7f3f98066cf0 con 0x7f3f980c6bd0 2023-10-30T13:46:32.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9d473640 1 --2- 172.21.15.148:0/458335110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f98071110 0x7f3f98158110 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f3f9806c830 tx=0x7f3f9001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308163403b31258d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:32.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f977fe640 1 -- 172.21.15.148:0/458335110 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f900af400 con 0x7f3f98071110 2023-10-30T13:46:32.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f977fe640 1 -- 172.21.15.148:0/458335110 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f900ab050 con 0x7f3f98071110 2023-10-30T13:46:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f977fe640 1 -- 172.21.15.148:0/458335110 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f900af780 con 0x7f3f98071110 2023-10-30T13:46:32.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 -- 172.21.15.148:0/458335110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f98071110 msgr2=0x7f3f98158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 --2- 172.21.15.148:0/458335110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f98071110 0x7f3f98158110 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f3f9806c830 tx=0x7f3f9001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 -- 172.21.15.148:0/458335110 shutdown_connections 2023-10-30T13:46:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 --2- 172.21.15.148:0/458335110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f98071110 0x7f3f98158110 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 --2- 172.21.15.148:0/458335110 >> v2:172.21.15.150:3300/0 conn(0x7f3f9806f7a0 0x7f3f9806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 -- 172.21.15.148:0/458335110 >> 172.21.15.148:0/458335110 conn(0x7f3f980c7910 msgr2=0x7f3f980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 -- 172.21.15.148:0/458335110 shutdown_connections 2023-10-30T13:46:32.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 -- 172.21.15.148:0/458335110 wait complete. 2023-10-30T13:46:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 Processor -- start 2023-10-30T13:46:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.413+0000 7f3f9eed6640 1 -- start start 2023-10-30T13:46:32.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9eed6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9806f7a0 0x7f3f981f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:32.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9eed6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f98071110 0x7f3f98152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:32.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9eed6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f980c6bd0 0x7f3f98154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:32.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9eed6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f981f2e70 con 0x7f3f980c6bd0 2023-10-30T13:46:32.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9eed6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f981e63a0 con 0x7f3f9806f7a0 2023-10-30T13:46:32.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9eed6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f981f2bf0 con 0x7f3f98071110 2023-10-30T13:46:32.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9d473640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f980c6bd0 0x7f3f98154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:32.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9d473640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f980c6bd0 0x7f3f98154010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32858/0 (socket says 172.21.15.148:32858) 2023-10-30T13:46:32.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9d473640 1 -- 172.21.15.148:0/1951617630 learned_addr learned my addr 172.21.15.148:0/1951617630 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:32.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9cc72640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9806f7a0 0x7f3f981f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:32.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f97fff640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f98071110 0x7f3f98152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:32.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9d473640 1 -- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f98071110 msgr2=0x7f3f98152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:32.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9d473640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f98071110 0x7f3f98152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9d473640 1 -- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9806f7a0 msgr2=0x7f3f981f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:32.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9d473640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9806f7a0 0x7f3f981f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9d473640 1 -- 172.21.15.148:0/1951617630 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f90009b30 con 0x7f3f980c6bd0 2023-10-30T13:46:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9d473640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f980c6bd0 0x7f3f98154010 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f3f900b2040 tx=0x7f3f9001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f957fa640 1 -- 172.21.15.148:0/1951617630 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f900afd30 con 0x7f3f980c6bd0 2023-10-30T13:46:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f957fa640 1 -- 172.21.15.148:0/1951617630 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f90009510 con 0x7f3f980c6bd0 2023-10-30T13:46:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f98154db0 con 0x7f3f980c6bd0 2023-10-30T13:46:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f957fa640 1 -- 172.21.15.148:0/1951617630 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f900ba660 con 0x7f3f980c6bd0 2023-10-30T13:46:32.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f980c3ff0 con 0x7f3f980c6bd0 2023-10-30T13:46:32.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f981e6550 con 0x7f3f980c6bd0 2023-10-30T13:46:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.417+0000 7f3f957fa640 1 -- 172.21.15.148:0/1951617630 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f900b6040 con 0x7f3f980c6bd0 2023-10-30T13:46:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.421+0000 7f3f957fa640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3f7c0664a0 0x7f3f7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.421+0000 7f3f957fa640 1 -- 172.21.15.148:0/1951617630 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f3f90122560 con 0x7f3f980c6bd0 2023-10-30T13:46:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.421+0000 7f3f9cc72640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3f7c0664a0 0x7f3f7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:32.422 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.421+0000 7f3f9cc72640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3f7c0664a0 0x7f3f7c068960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c009ac0 tx=0x7f3f8c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:32.423 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.425+0000 7f3f957fa640 1 -- 172.21.15.148:0/1951617630 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f981e6550 con 0x7f3f980c6bd0 2023-10-30T13:46:32.606 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f981e6550 con 0x7f3f980c6bd0 2023-10-30T13:46:32.607 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f957fa640 1 -- 172.21.15.148:0/1951617630 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5735 (secure 0 0 0) 0x7f3f981e6550 con 0x7f3f980c6bd0 2023-10-30T13:46:32.607 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:32.608 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:30.174493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:32.608 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 140 2023-10-30T13:46:32.609 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3f7c0664a0 msgr2=0x7f3f7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3f7c0664a0 0x7f3f7c068960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c009ac0 tx=0x7f3f8c009190 comp rx=0 tx=0).stop 2023-10-30T13:46:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f980c6bd0 msgr2=0x7f3f98154010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f980c6bd0 0x7f3f98154010 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f3f900b2040 tx=0x7f3f9001bf60 comp rx=0 tx=0).stop 2023-10-30T13:46:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 shutdown_connections 2023-10-30T13:46:32.610 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3f7c0664a0 0x7f3f7c068960 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f980c6bd0 0x7f3f98154010 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3f98071110 0x7f3f98152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 --2- 172.21.15.148:0/1951617630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9806f7a0 0x7f3f981f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:32.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 >> 172.21.15.148:0/1951617630 conn(0x7f3f980c7910 msgr2=0x7f3f980c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:32.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 shutdown_connections 2023-10-30T13:46:32.611 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:32.609+0000 7f3f9eed6640 1 -- 172.21.15.148:0/1951617630 wait complete. 2023-10-30T13:46:32.629 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:46:32.725 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:32.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:46:34.629 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:46:34.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.777+0000 7f8ae6479640 1 Processor -- start 2023-10-30T13:46:34.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.777+0000 7f8ae6479640 1 -- start start 2023-10-30T13:46:34.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8ae6479640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8ae006f7a0 0x7f8ae006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:34.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8ae6479640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 0x7f8ae0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:34.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8ae6479640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae006a3d0 con 0x7f8ae0071110 2023-10-30T13:46:34.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8ae6479640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae00700c0 con 0x7f8ae006f7a0 2023-10-30T13:46:34.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8ae6479640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae0066cf0 con 0x7f8ae00c6bd0 2023-10-30T13:46:34.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8adf7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 0x7f8ae0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8adffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8ae006f7a0 0x7f8ae006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8ae4a16640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8ae0071110 0x7f8ae0158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:37554/0 (socket says 172.21.15.148:37554) 2023-10-30T13:46:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8ae4a16640 1 -- 172.21.15.148:0/1122282167 learned_addr learned my addr 172.21.15.148:0/1122282167 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:46:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8adf7fe640 1 -- 172.21.15.148:0/1122282167 >> v2:172.21.15.150:3300/0 conn(0x7f8ae006f7a0 msgr2=0x7f8ae006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:46:34.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8adf7fe640 1 --2- 172.21.15.148:0/1122282167 >> v2:172.21.15.150:3300/0 conn(0x7f8ae006f7a0 0x7f8ae006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8adf7fe640 1 -- 172.21.15.148:0/1122282167 >> v1:172.21.15.150:6789/0 conn(0x7f8ae0071110 legacy=0x7f8ae0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8adf7fe640 1 -- 172.21.15.148:0/1122282167 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae00c70d0 con 0x7f8ae00c6bd0 2023-10-30T13:46:34.782 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.781+0000 7f8adffff640 1 --2- 172.21.15.148:0/1122282167 >> v2:172.21.15.150:3300/0 conn(0x7f8ae006f7a0 0x7f8ae006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:46:34.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8adf7fe640 1 --2- 172.21.15.148:0/1122282167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 0x7f8ae0069e90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40097c0 tx=0x7f8ad401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5806df3ef63b261a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8adeffd640 1 -- 172.21.15.148:0/1122282167 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40a8970 con 0x7f8ae00c6bd0 2023-10-30T13:46:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8adeffd640 1 -- 172.21.15.148:0/1122282167 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad40a8ad0 con 0x7f8ae00c6bd0 2023-10-30T13:46:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8adeffd640 1 -- 172.21.15.148:0/1122282167 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40a8e60 con 0x7f8ae00c6bd0 2023-10-30T13:46:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 -- 172.21.15.148:0/1122282167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 msgr2=0x7f8ae0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:34.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 --2- 172.21.15.148:0/1122282167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 0x7f8ae0069e90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40097c0 tx=0x7f8ad401ec70 comp rx=0 tx=0).stop 2023-10-30T13:46:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 -- 172.21.15.148:0/1122282167 shutdown_connections 2023-10-30T13:46:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 --2- 172.21.15.148:0/1122282167 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 0x7f8ae0069e90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 --2- 172.21.15.148:0/1122282167 >> v2:172.21.15.150:3300/0 conn(0x7f8ae006f7a0 0x7f8ae006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 -- 172.21.15.148:0/1122282167 >> 172.21.15.148:0/1122282167 conn(0x7f8ae00c7910 msgr2=0x7f8ae00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:34.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 -- 172.21.15.148:0/1122282167 shutdown_connections 2023-10-30T13:46:34.786 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 -- 172.21.15.148:0/1122282167 wait complete. 2023-10-30T13:46:34.787 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 Processor -- start 2023-10-30T13:46:34.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.785+0000 7f8ae6479640 1 -- start start 2023-10-30T13:46:34.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae6479640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ae006f7a0 0x7f8ae00ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:34.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae6479640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ae0071110 0x7f8ae00adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:34.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae6479640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 0x7f8ae01f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:34.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae6479640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae006fec0 con 0x7f8ae006f7a0 2023-10-30T13:46:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae6479640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00af880 con 0x7f8ae0071110 2023-10-30T13:46:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae6479640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae014f3d0 con 0x7f8ae00c6bd0 2023-10-30T13:46:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ae006f7a0 0x7f8ae00ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ae006f7a0 0x7f8ae00ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32862/0 (socket says 172.21.15.148:32862) 2023-10-30T13:46:34.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adffff640 1 -- 172.21.15.148:0/119604656 learned_addr learned my addr 172.21.15.148:0/119604656 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:46:34.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adf7fe640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ae0071110 0x7f8ae00adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:34.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae4a16640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 0x7f8ae01f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:34.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adffff640 1 -- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 msgr2=0x7f8ae01f0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:34.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adffff640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 0x7f8ae01f0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:34.790 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adffff640 1 -- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ae0071110 msgr2=0x7f8ae00adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:34.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adffff640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ae0071110 0x7f8ae00adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:34.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adffff640 1 -- 172.21.15.148:0/119604656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad0009420 con 0x7f8ae006f7a0 2023-10-30T13:46:34.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adffff640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ae006f7a0 0x7f8ae00ca670 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000e980 tx=0x7f8ad000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:34.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adcff9640 1 -- 172.21.15.148:0/119604656 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad000ca00 con 0x7f8ae006f7a0 2023-10-30T13:46:34.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adcff9640 1 -- 172.21.15.148:0/119604656 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad000cb60 con 0x7f8ae006f7a0 2023-10-30T13:46:34.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad4009420 con 0x7f8ae006f7a0 2023-10-30T13:46:34.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adcff9640 1 -- 172.21.15.148:0/119604656 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0013470 con 0x7f8ae006f7a0 2023-10-30T13:46:34.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae00afb10 con 0x7f8ae006f7a0 2023-10-30T13:46:34.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae0158220 con 0x7f8ae006f7a0 2023-10-30T13:46:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.789+0000 7f8adcff9640 1 -- 172.21.15.148:0/119604656 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ad00135d0 con 0x7f8ae006f7a0 2023-10-30T13:46:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.793+0000 7f8adcff9640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8acc0664a0 0x7f8acc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:46:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.793+0000 7f8adcff9640 1 -- 172.21.15.148:0/119604656 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f8ad0084730 con 0x7f8ae006f7a0 2023-10-30T13:46:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.793+0000 7f8adf7fe640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8acc0664a0 0x7f8acc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:46:34.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.793+0000 7f8adf7fe640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8acc0664a0 0x7f8acc068960 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40b2000 tx=0x7f8ad4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:46:34.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.797+0000 7f8adcff9640 1 -- 172.21.15.148:0/119604656 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ad0051530 con 0x7f8ae006f7a0 2023-10-30T13:46:34.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ae004e170 con 0x7f8ae006f7a0 2023-10-30T13:46:34.976 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8adcff9640 1 -- 172.21.15.148:0/119604656 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+6530 (secure 0 0 0) 0x7f8ad0054b50 con 0x7f8ae006f7a0 2023-10-30T13:46:34.977 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:46:34.977 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:46:30.174493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:46:34.977 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 141 2023-10-30T13:46:34.978 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8acc0664a0 msgr2=0x7f8acc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:34.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8acc0664a0 0x7f8acc068960 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40b2000 tx=0x7f8ad4008040 comp rx=0 tx=0).stop 2023-10-30T13:46:34.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ae006f7a0 msgr2=0x7f8ae00ca670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:46:34.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ae006f7a0 0x7f8ae00ca670 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000e980 tx=0x7f8ad000ee40 comp rx=0 tx=0).stop 2023-10-30T13:46:34.979 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 shutdown_connections 2023-10-30T13:46:34.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8acc0664a0 0x7f8acc068960 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:34.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8ae00c6bd0 0x7f8ae01f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:34.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ae0071110 0x7f8ae00adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:34.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 --2- 172.21.15.148:0/119604656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8ae006f7a0 0x7f8ae00ca670 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:46:34.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.977+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 >> 172.21.15.148:0/119604656 conn(0x7f8ae00c7910 msgr2=0x7f8ae00c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:46:34.980 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.981+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 shutdown_connections 2023-10-30T13:46:34.981 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:46:34.981+0000 7f8ae6479640 1 -- 172.21.15.148:0/119604656 wait complete. 2023-10-30T13:46:34.999 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:46:34.999 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2023-10-30T13:46:38.645 INFO:tasks.ceph.mds.d.smithi150.stderr:2023-10-30T13:46:38.640+0000 7fba413d5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:46:38.645 INFO:tasks.ceph.mds.d.smithi150.stderr:2023-10-30T13:46:38.640+0000 7fba413d5640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:46:45.076 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 24% (98/405) Updating files: 25% (102/405) Updating files: 26% (106/405) Updating files: 27% (110/405) Updating files: 28% (114/405) Updating files: 29% (118/405) Updating files: 30% (122/405) Updating files: 31% (126/405) Updating files: 32% (130/405) Updating files: 33% (134/405) Updating files: 34% (138/405) Updating files: 35% (142/405) Updating files: 36% (146/405) Updating files: 37% (150/405) Updating files: 37% (152/405) Updating files: 38% (154/405) Updating files: 39% (158/405) Updating files: 40% (162/405) Updating files: 41% (167/405) Updating files: 42% (171/405) Updating files: 43% (175/405) Updating files: 44% (179/405) Updating files: 45% (183/405) Updating files: 46% (187/405) Updating files: 47% (191/405) Updating files: 48% (195/405) Updating files: 49% (199/405) Updating files: 50% (203/405) Updating files: 50% (205/405) Updating files: 51% (207/405) Updating files: 52% (211/405) Updating files: 53% (215/405) Updating files: 54% (219/405) Updating files: 55% (223/405) Updating files: 56% (227/405) Updating files: 57% (231/405) Updating files: 58% (235/405) Updating files: 59% (239/405) Updating files: 60% (243/405) Updating files: 60% (245/405) Updating files: 61% (248/405) Updating files: 62% (252/405) Updating files: 63% (256/405) Updating files: 64% (260/405) Updating files: 65% (264/405) Updating files: 66% (268/405) Updating files: 67% (272/405) Updating files: 68% (276/405) Updating files: 69% (280/405) Updating files: 70% (284/405) Updating files: 71% (288/405) Updating files: 72% (292/405) Updating files: 73% (296/405) Updating files: 74% (300/405) Updating files: 75% (304/405) Updating files: 76% (308/405) Updating files: 77% (312/405) Updating files: 78% (316/405) Updating files: 79% (320/405) Updating files: 80% (324/405) Updating files: 81% (329/405) Updating files: 82% (333/405) Updating files: 83% (337/405) Updating files: 84% (341/405) Updating files: 84% (344/405) Updating files: 85% (345/405) Updating files: 86% (349/405) Updating files: 87% (353/405) Updating files: 88% (357/405) Updating files: 89% (361/405) Updating files: 90% (365/405) Updating files: 90% (368/405) Updating files: 91% (369/405) Updating files: 92% (373/405) Updating files: 93% (377/405) Updating files: 94% (381/405) Updating files: 95% (385/405) Updating files: 96% (389/405) Updating files: 97% (393/405) Updating files: 97% (396/405) Updating files: 98% (397/405) Updating files: 99% (401/405) Updating files: 100% (405/405) Updating files: 100% (405/405), done. 2023-10-30T13:46:45.247 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at e509cb1e69c v0.71 2023-10-30T13:46:45.255 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.71 2023-10-30T13:46:45.767 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 72 -eq 48 ] 2023-10-30T13:46:45.768 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.72 2023-10-30T13:46:45.768 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.72 2023-10-30T13:46:45.768 INFO:tasks.workunit.client.0.smithi148.stdout:v0.72 2023-10-30T13:46:45.769 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.72 2023-10-30T13:47:00.214 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 9% (18/191) Updating files: 10% (20/191) Updating files: 11% (22/191) Updating files: 12% (23/191) Updating files: 13% (25/191) Updating files: 14% (27/191) Updating files: 15% (29/191) Updating files: 16% (31/191) Updating files: 17% (33/191) Updating files: 18% (35/191) Updating files: 19% (37/191) Updating files: 20% (39/191) Updating files: 21% (41/191) Updating files: 22% (43/191) Updating files: 23% (44/191) Updating files: 24% (46/191) Updating files: 25% (48/191) Updating files: 26% (50/191) Updating files: 27% (52/191) Updating files: 28% (54/191) Updating files: 29% (56/191) Updating files: 30% (58/191) Updating files: 31% (60/191) Updating files: 32% (62/191) Updating files: 33% (64/191) Updating files: 34% (65/191) Updating files: 35% (67/191) Updating files: 36% (69/191) Updating files: 37% (71/191) Updating files: 38% (73/191) Updating files: 39% (75/191) Updating files: 40% (77/191) Updating files: 41% (79/191) Updating files: 42% (81/191) Updating files: 43% (83/191) Updating files: 44% (85/191) Updating files: 45% (86/191) Updating files: 46% (88/191) Updating files: 47% (90/191) Updating files: 48% (92/191) Updating files: 49% (94/191) Updating files: 50% (96/191) Updating files: 51% (98/191) Updating files: 51% (99/191) Updating files: 52% (100/191) Updating files: 53% (102/191) Updating files: 54% (104/191) Updating files: 55% (106/191) Updating files: 56% (107/191) Updating files: 57% (109/191) Updating files: 58% (111/191) Updating files: 59% (113/191) Updating files: 60% (115/191) Updating files: 61% (117/191) Updating files: 62% (119/191) Updating files: 63% (121/191) Updating files: 64% (123/191) Updating files: 65% (125/191) Updating files: 66% (127/191) Updating files: 67% (128/191) Updating files: 68% (130/191) Updating files: 69% (132/191) Updating files: 70% (134/191) Updating files: 71% (136/191) Updating files: 72% (138/191) Updating files: 73% (140/191) Updating files: 74% (142/191) Updating files: 75% (144/191) Updating files: 76% (146/191) Updating files: 77% (148/191) Updating files: 78% (149/191) Updating files: 79% (151/191) Updating files: 80% (153/191) Updating files: 81% (155/191) Updating files: 82% (157/191) Updating files: 83% (159/191) Updating files: 84% (161/191) Updating files: 85% (163/191) Updating files: 86% (165/191) Updating files: 86% (166/191) Updating files: 87% (167/191) Updating files: 88% (169/191) Updating files: 89% (170/191) Updating files: 90% (172/191) Updating files: 91% (174/191) Updating files: 92% (176/191) Updating files: 93% (178/191) Updating files: 94% (180/191) Updating files: 95% (182/191) Updating files: 96% (184/191) Updating files: 97% (186/191) Updating files: 98% (188/191) Updating files: 99% (190/191) Updating files: 100% (191/191) Updating files: 100% (191/191), done. 2023-10-30T13:47:00.290 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 5832e2603c7 v0.72 2023-10-30T13:47:00.300 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.72 2023-10-30T13:47:00.442 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 73 -eq 48 ] 2023-10-30T13:47:00.443 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.73 2023-10-30T13:47:00.443 INFO:tasks.workunit.client.0.smithi148.stdout:v0.73 2023-10-30T13:47:00.444 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.73 2023-10-30T13:47:00.444 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.73 2023-10-30T13:47:02.782 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:02.786 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:04.884 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 31% (46/148) Updating files: 32% (48/148) Updating files: 33% (49/148) Updating files: 34% (51/148) Updating files: 35% (52/148) Updating files: 36% (54/148) Updating files: 37% (55/148) Updating files: 38% (57/148) Updating files: 39% (58/148) Updating files: 40% (60/148) Updating files: 41% (61/148) Updating files: 42% (63/148) Updating files: 43% (64/148) Updating files: 44% (66/148) Updating files: 45% (67/148) Updating files: 46% (69/148) Updating files: 47% (70/148) Updating files: 48% (72/148) Updating files: 49% (73/148) Updating files: 50% (74/148) Updating files: 51% (76/148) Updating files: 52% (77/148) Updating files: 53% (79/148) Updating files: 54% (80/148) Updating files: 55% (82/148) Updating files: 56% (83/148) Updating files: 57% (85/148) Updating files: 58% (86/148) Updating files: 59% (88/148) Updating files: 60% (89/148) Updating files: 61% (91/148) Updating files: 62% (92/148) Updating files: 63% (94/148) Updating files: 64% (95/148) Updating files: 65% (97/148) Updating files: 66% (98/148) Updating files: 67% (100/148) Updating files: 68% (101/148) Updating files: 69% (103/148) Updating files: 70% (104/148) Updating files: 71% (106/148) Updating files: 72% (107/148) Updating files: 73% (109/148) Updating files: 74% (110/148) Updating files: 75% (111/148) Updating files: 76% (113/148) Updating files: 77% (114/148) Updating files: 78% (116/148) Updating files: 79% (117/148) Updating files: 80% (119/148) Updating files: 81% (120/148) Updating files: 82% (122/148) Updating files: 83% (123/148) Updating files: 84% (125/148) Updating files: 85% (126/148) Updating files: 86% (128/148) Updating files: 87% (129/148) Updating files: 88% (131/148) Updating files: 89% (132/148) Updating files: 90% (134/148) Updating files: 91% (135/148) Updating files: 92% (137/148) Updating files: 93% (138/148) Updating files: 94% (140/148) Updating files: 95% (141/148) Updating files: 96% (143/148) Updating files: 97% (144/148) Updating files: 98% (146/148) Updating files: 99% (147/148) Updating files: 100% (148/148) Updating files: 100% (148/148), done. 2023-10-30T13:47:04.938 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at d8ad51ee8ab v0.73 2023-10-30T13:47:04.943 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.73 2023-10-30T13:47:05.549 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 74 -eq 48 ] 2023-10-30T13:47:05.549 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.74 2023-10-30T13:47:05.550 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.74 2023-10-30T13:47:05.550 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.74 2023-10-30T13:47:05.550 INFO:tasks.workunit.client.0.smithi148.stdout:v0.74 2023-10-30T13:47:08.223 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 80% (154/192) Updating files: 81% (156/192) Updating files: 82% (158/192) Updating files: 83% (160/192) Updating files: 84% (162/192) Updating files: 85% (164/192) Updating files: 86% (166/192) Updating files: 87% (168/192) Updating files: 88% (169/192) Updating files: 89% (171/192) Updating files: 90% (173/192) Updating files: 91% (175/192) Updating files: 92% (177/192) Updating files: 93% (179/192) Updating files: 94% (181/192) Updating files: 95% (183/192) Updating files: 96% (185/192) Updating files: 97% (187/192) Updating files: 98% (189/192) Updating files: 99% (191/192) Updating files: 100% (192/192) Updating files: 100% (192/192), done. 2023-10-30T13:47:08.298 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at c165483bc72 v0.74 2023-10-30T13:47:08.304 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.74 2023-10-30T13:47:08.522 INFO:tasks.workunit.client.0.smithi148.stdout:v0.75 2023-10-30T13:47:08.522 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 75 -eq 48 ] 2023-10-30T13:47:08.523 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.75 2023-10-30T13:47:08.523 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.75 2023-10-30T13:47:08.523 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.75 2023-10-30T13:47:15.017 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 1% (5/291) Updating files: 2% (6/291) Updating files: 3% (9/291) Updating files: 4% (12/291) Updating files: 5% (15/291) Updating files: 6% (18/291) Updating files: 7% (21/291) Updating files: 8% (24/291) Updating files: 9% (27/291) Updating files: 10% (30/291) Updating files: 11% (33/291) Updating files: 12% (35/291) Updating files: 13% (38/291) Updating files: 14% (41/291) Updating files: 15% (44/291) Updating files: 16% (47/291) Updating files: 17% (50/291) Updating files: 18% (53/291) Updating files: 19% (56/291) Updating files: 20% (59/291) Updating files: 21% (62/291) Updating files: 22% (65/291) Updating files: 23% (67/291) Updating files: 24% (70/291) Updating files: 25% (73/291) Updating files: 26% (76/291) Updating files: 27% (79/291) Updating files: 28% (82/291) Updating files: 29% (85/291) Updating files: 30% (88/291) Updating files: 30% (90/291) Updating files: 31% (91/291) Updating files: 32% (94/291) Updating files: 33% (97/291) Updating files: 34% (99/291) Updating files: 35% (102/291) Updating files: 36% (105/291) Updating files: 37% (108/291) Updating files: 38% (111/291) Updating files: 39% (114/291) Updating files: 40% (117/291) Updating files: 41% (120/291) Updating files: 42% (123/291) Updating files: 43% (126/291) Updating files: 44% (129/291) Updating files: 45% (131/291) Updating files: 46% (134/291) Updating files: 47% (137/291) Updating files: 48% (140/291) Updating files: 49% (143/291) Updating files: 50% (146/291) Updating files: 51% (149/291) Updating files: 52% (152/291) Updating files: 53% (155/291) Updating files: 54% (158/291) Updating files: 55% (161/291) Updating files: 56% (163/291) Updating files: 57% (166/291) Updating files: 58% (169/291) Updating files: 59% (172/291) Updating files: 60% (175/291) Updating files: 61% (178/291) Updating files: 62% (181/291) Updating files: 63% (184/291) Updating files: 64% (187/291) Updating files: 64% (188/291) Updating files: 65% (190/291) Updating files: 66% (193/291) Updating files: 67% (195/291) Updating files: 68% (198/291) Updating files: 69% (201/291) Updating files: 70% (204/291) Updating files: 71% (207/291) Updating files: 72% (210/291) Updating files: 73% (213/291) Updating files: 74% (216/291) Updating files: 75% (219/291) Updating files: 76% (222/291) Updating files: 77% (225/291) Updating files: 78% (227/291) Updating files: 79% (230/291) Updating files: 80% (233/291) Updating files: 81% (236/291) Updating files: 82% (239/291) Updating files: 83% (242/291) Updating files: 84% (245/291) Updating files: 85% (248/291) Updating files: 86% (251/291) Updating files: 87% (254/291) Updating files: 88% (257/291) Updating files: 89% (259/291) Updating files: 90% (262/291) Updating files: 91% (265/291) Updating files: 92% (268/291) Updating files: 93% (271/291) Updating files: 94% (274/291) Updating files: 95% (277/291) Updating files: 96% (280/291) Updating files: 97% (283/291) Updating files: 98% (286/291) Updating files: 98% (287/291) Updating files: 99% (289/291) Updating files: 100% (291/291) Updating files: 100% (291/291), done. 2023-10-30T13:47:15.591 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 946d6036958 v0.75 2023-10-30T13:47:15.596 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.75 2023-10-30T13:47:15.743 INFO:tasks.workunit.client.0.smithi148.stdout:v0.76 2023-10-30T13:47:15.744 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 76 -eq 48 ] 2023-10-30T13:47:15.744 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.76 2023-10-30T13:47:15.744 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.76 2023-10-30T13:47:15.745 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.76 2023-10-30T13:47:20.597 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 38% (65/168) Updating files: 39% (66/168) Updating files: 40% (68/168) Updating files: 41% (69/168) Updating files: 42% (71/168) Updating files: 43% (73/168) Updating files: 44% (74/168) Updating files: 45% (76/168) Updating files: 46% (78/168) Updating files: 47% (79/168) Updating files: 48% (81/168) Updating files: 49% (83/168) Updating files: 50% (84/168) Updating files: 51% (86/168) Updating files: 51% (87/168) Updating files: 52% (88/168) Updating files: 53% (90/168) Updating files: 54% (91/168) Updating files: 55% (93/168) Updating files: 56% (95/168) Updating files: 57% (96/168) Updating files: 58% (98/168) Updating files: 59% (100/168) Updating files: 60% (101/168) Updating files: 61% (103/168) Updating files: 62% (105/168) Updating files: 63% (106/168) Updating files: 64% (108/168) Updating files: 65% (110/168) Updating files: 66% (111/168) Updating files: 67% (113/168) Updating files: 68% (115/168) Updating files: 69% (116/168) Updating files: 70% (118/168) Updating files: 71% (120/168) Updating files: 72% (121/168) Updating files: 73% (123/168) Updating files: 74% (125/168) Updating files: 75% (126/168) Updating files: 76% (128/168) Updating files: 77% (130/168) Updating files: 78% (132/168) Updating files: 79% (133/168) Updating files: 80% (135/168) Updating files: 81% (137/168) Updating files: 82% (138/168) Updating files: 83% (140/168) Updating files: 84% (142/168) Updating files: 85% (143/168) Updating files: 86% (145/168) Updating files: 87% (147/168) Updating files: 88% (148/168) Updating files: 89% (150/168) Updating files: 90% (152/168) Updating files: 91% (153/168) Updating files: 92% (155/168) Updating files: 93% (157/168) Updating files: 94% (158/168) Updating files: 95% (160/168) Updating files: 96% (162/168) Updating files: 97% (163/168) Updating files: 98% (165/168) Updating files: 99% (167/168) Updating files: 100% (168/168) Updating files: 100% (168/168), done. 2023-10-30T13:47:20.665 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 3b990136bfa v0.76 2023-10-30T13:47:20.670 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.76 2023-10-30T13:47:20.958 INFO:tasks.workunit.client.0.smithi148.stdout:v0.77 2023-10-30T13:47:20.958 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 77 -eq 48 ] 2023-10-30T13:47:20.958 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.77 2023-10-30T13:47:20.958 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.77 2023-10-30T13:47:20.959 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.77 2023-10-30T13:47:32.855 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:32.859 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:47:33.475 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 8% (25/298) Updating files: 9% (27/298) Updating files: 10% (30/298) Updating files: 11% (33/298) Updating files: 12% (36/298) Updating files: 13% (39/298) Updating files: 14% (42/298) Updating files: 15% (45/298) Updating files: 16% (48/298) Updating files: 17% (51/298) Updating files: 18% (54/298) Updating files: 19% (57/298) Updating files: 20% (60/298) Updating files: 21% (63/298) Updating files: 22% (66/298) Updating files: 23% (69/298) Updating files: 24% (72/298) Updating files: 25% (75/298) Updating files: 26% (78/298) Updating files: 27% (81/298) Updating files: 28% (84/298) Updating files: 29% (87/298) Updating files: 30% (90/298) Updating files: 31% (93/298) Updating files: 32% (96/298) Updating files: 33% (99/298) Updating files: 34% (102/298) Updating files: 35% (105/298) Updating files: 36% (108/298) Updating files: 37% (111/298) Updating files: 38% (114/298) Updating files: 39% (117/298) Updating files: 39% (119/298) Updating files: 40% (120/298) Updating files: 41% (123/298) Updating files: 42% (126/298) Updating files: 43% (129/298) Updating files: 44% (132/298) Updating files: 45% (135/298) Updating files: 46% (138/298) Updating files: 47% (141/298) Updating files: 48% (144/298) Updating files: 49% (147/298) Updating files: 50% (149/298) Updating files: 51% (152/298) Updating files: 52% (155/298) Updating files: 53% (158/298) Updating files: 54% (161/298) Updating files: 55% (164/298) Updating files: 56% (167/298) Updating files: 57% (170/298) Updating files: 58% (173/298) Updating files: 59% (176/298) Updating files: 60% (179/298) Updating files: 61% (182/298) Updating files: 62% (185/298) Updating files: 63% (188/298) Updating files: 64% (191/298) Updating files: 65% (194/298) Updating files: 66% (197/298) Updating files: 67% (200/298) Updating files: 68% (203/298) Updating files: 69% (206/298) Updating files: 70% (209/298) Updating files: 71% (212/298) Updating files: 72% (215/298) Updating files: 73% (218/298) Updating files: 74% (221/298) Updating files: 75% (224/298) Updating files: 76% (227/298) Updating files: 77% (230/298) Updating files: 78% (233/298) Updating files: 79% (236/298) Updating files: 80% (239/298) Updating files: 81% (242/298) Updating files: 81% (244/298) Updating files: 82% (245/298) Updating files: 83% (248/298) Updating files: 84% (251/298) Updating files: 85% (254/298) Updating files: 86% (257/298) Updating files: 87% (260/298) Updating files: 88% (263/298) Updating files: 89% (266/298) Updating files: 90% (269/298) Updating files: 91% (272/298) Updating files: 92% (275/298) Updating files: 93% (278/298) Updating files: 94% (281/298) Updating files: 95% (284/298) Updating files: 96% (287/298) Updating files: 97% (290/298) Updating files: 98% (293/298) Updating files: 99% (296/298) Updating files: 100% (298/298) Updating files: 100% (298/298), done. 2023-10-30T13:47:33.813 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 1bca9c5c412 v0.77 2023-10-30T13:47:33.819 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.77 2023-10-30T13:47:33.907 INFO:tasks.workunit.client.0.smithi148.stdout:v0.78 2023-10-30T13:47:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 78 -eq 48 ] 2023-10-30T13:47:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.78 2023-10-30T13:47:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.78 2023-10-30T13:47:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.78 2023-10-30T13:47:40.767 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 12% (60/496) Updating files: 13% (65/496) Updating files: 14% (70/496) Updating files: 15% (75/496) Updating files: 15% (76/496) Updating files: 16% (80/496) Updating files: 17% (85/496) Updating files: 18% (90/496) Updating files: 19% (95/496) Updating files: 20% (100/496) Updating files: 21% (105/496) Updating files: 22% (110/496) Updating files: 23% (115/496) Updating files: 24% (120/496) Updating files: 25% (124/496) Updating files: 26% (129/496) Updating files: 27% (134/496) Updating files: 28% (139/496) Updating files: 29% (144/496) Updating files: 30% (149/496) Updating files: 31% (154/496) Updating files: 32% (159/496) Updating files: 33% (164/496) Updating files: 34% (169/496) Updating files: 35% (174/496) Updating files: 35% (177/496) Updating files: 36% (179/496) Updating files: 37% (184/496) Updating files: 38% (189/496) Updating files: 39% (194/496) Updating files: 40% (199/496) Updating files: 41% (204/496) Updating files: 42% (209/496) Updating files: 43% (214/496) Updating files: 44% (219/496) Updating files: 45% (224/496) Updating files: 46% (229/496) Updating files: 47% (234/496) Updating files: 48% (239/496) Updating files: 49% (244/496) Updating files: 50% (248/496) Updating files: 51% (253/496) Updating files: 52% (258/496) Updating files: 53% (263/496) Updating files: 54% (268/496) Updating files: 55% (273/496) Updating files: 56% (278/496) Updating files: 57% (283/496) Updating files: 58% (288/496) Updating files: 59% (293/496) Updating files: 60% (298/496) Updating files: 61% (303/496) Updating files: 62% (308/496) Updating files: 63% (313/496) Updating files: 64% (318/496) Updating files: 65% (323/496) Updating files: 66% (328/496) Updating files: 67% (333/496) Updating files: 68% (338/496) Updating files: 69% (343/496) Updating files: 70% (348/496) Updating files: 71% (353/496) Updating files: 72% (358/496) Updating files: 73% (363/496) Updating files: 73% (367/496) Updating files: 74% (368/496) Updating files: 75% (372/496) Updating files: 76% (377/496) Updating files: 77% (382/496) Updating files: 78% (387/496) Updating files: 79% (392/496) Updating files: 80% (397/496) Updating files: 81% (402/496) Updating files: 82% (407/496) Updating files: 83% (412/496) Updating files: 84% (417/496) Updating files: 85% (422/496) Updating files: 86% (427/496) Updating files: 87% (432/496) Updating files: 88% (437/496) Updating files: 89% (442/496) Updating files: 90% (447/496) Updating files: 91% (452/496) Updating files: 92% (457/496) Updating files: 93% (462/496) Updating files: 94% (467/496) Updating files: 95% (472/496) Updating files: 96% (477/496) Updating files: 97% (482/496) Updating files: 98% (487/496) Updating files: 99% (492/496) Updating files: 100% (496/496) Updating files: 100% (496/496), done. 2023-10-30T13:47:40.829 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at f6c746c314d 0.78 2023-10-30T13:47:40.836 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.78 2023-10-30T13:47:40.954 INFO:tasks.workunit.client.0.smithi148.stdout:v0.79 2023-10-30T13:47:40.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 79 -eq 48 ] 2023-10-30T13:47:40.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.79 2023-10-30T13:47:40.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.79 2023-10-30T13:47:40.955 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.79 2023-10-30T13:47:54.275 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 24% (79/329) Updating files: 25% (83/329) Updating files: 26% (86/329) Updating files: 27% (89/329) Updating files: 28% (93/329) Updating files: 29% (96/329) Updating files: 30% (99/329) Updating files: 31% (102/329) Updating files: 32% (106/329) Updating files: 33% (109/329) Updating files: 34% (112/329) Updating files: 35% (116/329) Updating files: 36% (119/329) Updating files: 37% (122/329) Updating files: 38% (126/329) Updating files: 39% (129/329) Updating files: 40% (132/329) Updating files: 41% (135/329) Updating files: 42% (139/329) Updating files: 43% (142/329) Updating files: 44% (145/329) Updating files: 45% (149/329) Updating files: 46% (152/329) Updating files: 47% (155/329) Updating files: 48% (158/329) Updating files: 49% (162/329) Updating files: 50% (165/329) Updating files: 51% (168/329) Updating files: 52% (172/329) Updating files: 53% (175/329) Updating files: 54% (178/329) Updating files: 55% (181/329) Updating files: 56% (185/329) Updating files: 57% (188/329) Updating files: 58% (191/329) Updating files: 59% (195/329) Updating files: 59% (196/329) Updating files: 60% (198/329) Updating files: 61% (201/329) Updating files: 62% (204/329) Updating files: 63% (208/329) Updating files: 64% (211/329) Updating files: 65% (214/329) Updating files: 66% (218/329) Updating files: 67% (221/329) Updating files: 68% (224/329) Updating files: 69% (228/329) Updating files: 70% (231/329) Updating files: 71% (234/329) Updating files: 72% (237/329) Updating files: 73% (241/329) Updating files: 74% (244/329) Updating files: 75% (247/329) Updating files: 76% (251/329) Updating files: 77% (254/329) Updating files: 78% (257/329) Updating files: 79% (260/329) Updating files: 80% (264/329) Updating files: 81% (267/329) Updating files: 82% (270/329) Updating files: 83% (274/329) Updating files: 84% (277/329) Updating files: 85% (280/329) Updating files: 86% (283/329) Updating files: 87% (287/329) Updating files: 88% (290/329) Updating files: 89% (293/329) Updating files: 90% (297/329) Updating files: 91% (300/329) Updating files: 92% (303/329) Updating files: 93% (306/329) Updating files: 94% (310/329) Updating files: 95% (313/329) Updating files: 96% (316/329) Updating files: 97% (320/329) Updating files: 98% (323/329) Updating files: 99% (326/329) Updating files: 100% (329/329) Updating files: 100% (329/329), done. 2023-10-30T13:47:54.360 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 4c2d73a5095 0.79 2023-10-30T13:47:54.367 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.79 2023-10-30T13:47:54.778 INFO:tasks.workunit.client.0.smithi148.stdout:v0.80 2023-10-30T13:47:54.778 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 80 -eq 48 ] 2023-10-30T13:47:54.778 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.80 2023-10-30T13:47:54.778 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.80 2023-10-30T13:47:54.779 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.80 2023-10-30T13:47:58.000 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:58.166 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.162+0000 7f3e2eebb640 1 Processor -- start 2023-10-30T13:47:58.167 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.162+0000 7f3e2eebb640 1 -- start start 2023-10-30T13:47:58.168 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2eebb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e281538f0 0x7f3e28159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2eebb640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e280ca6a0 con 0x7f3e280cefa0 2023-10-30T13:47:58.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2eebb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2815d300 con 0x7f3e2815cd50 2023-10-30T13:47:58.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2eebb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2815d5e0 con 0x7f3e281538f0 2023-10-30T13:47:58.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2e6ba640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3e280cefa0 0x7f3e281640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36578/0 (socket says 172.21.15.148:36578) 2023-10-30T13:47:58.169 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2e6ba640 1 -- 172.21.15.148:0/1052162170 learned_addr learned my addr 172.21.15.148:0/1052162170 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:47:58.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2deb9640 1 --2- 172.21.15.148:0/1052162170 >> v2:172.21.15.150:3300/0 conn(0x7f3e281538f0 0x7f3e28159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2ceb7640 1 -- 172.21.15.148:0/1052162170 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076853770 0 0) 0x7f3e280ca6a0 con 0x7f3e280cefa0 2023-10-30T13:47:58.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2ceb7640 1 -- 172.21.15.148:0/1052162170 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e14003530 con 0x7f3e280cefa0 2023-10-30T13:47:58.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2deb9640 1 -- 172.21.15.148:0/1052162170 >> v1:172.21.15.150:6789/0 conn(0x7f3e2815cd50 legacy=0x7f3e280c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2deb9640 1 -- 172.21.15.148:0/1052162170 >> v1:172.21.15.148:6789/0 conn(0x7f3e280cefa0 legacy=0x7f3e281640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.166+0000 7f3e2deb9640 1 -- 172.21.15.148:0/1052162170 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2815d8c0 con 0x7f3e281538f0 2023-10-30T13:47:58.172 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2deb9640 1 --2- 172.21.15.148:0/1052162170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e281538f0 0x7f3e28159fb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3e240097c0 tx=0x7f3e2401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ca2604d3e51b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:58.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2ceb7640 1 -- 172.21.15.148:0/1052162170 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e240a8970 con 0x7f3e281538f0 2023-10-30T13:47:58.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2ceb7640 1 -- 172.21.15.148:0/1052162170 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e240a8ad0 con 0x7f3e281538f0 2023-10-30T13:47:58.173 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2ceb7640 1 -- 172.21.15.148:0/1052162170 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e240a8e40 con 0x7f3e281538f0 2023-10-30T13:47:58.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2eebb640 1 -- 172.21.15.148:0/1052162170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e281538f0 msgr2=0x7f3e28159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2eebb640 1 --2- 172.21.15.148:0/1052162170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e281538f0 0x7f3e28159fb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3e240097c0 tx=0x7f3e2401ec70 comp rx=0 tx=0).stop 2023-10-30T13:47:58.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2eebb640 1 -- 172.21.15.148:0/1052162170 shutdown_connections 2023-10-30T13:47:58.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2eebb640 1 --2- 172.21.15.148:0/1052162170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e281538f0 0x7f3e28159fb0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2eebb640 1 -- 172.21.15.148:0/1052162170 >> 172.21.15.148:0/1052162170 conn(0x7f3e2806a640 msgr2=0x7f3e28065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:58.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2eebb640 1 -- 172.21.15.148:0/1052162170 shutdown_connections 2023-10-30T13:47:58.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.170+0000 7f3e2eebb640 1 -- 172.21.15.148:0/1052162170 wait complete. 2023-10-30T13:47:58.177 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 Processor -- start 2023-10-30T13:47:58.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 -- start start 2023-10-30T13:47:58.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e280cefa0 0x7f3e28144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e281538f0 0x7f3e281f0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e2815cd50 0x7f3e280b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.178 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2814dec0 con 0x7f3e281538f0 2023-10-30T13:47:58.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2814ea10 con 0x7f3e280cefa0 2023-10-30T13:47:58.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2814f9c0 con 0x7f3e2815cd50 2023-10-30T13:47:58.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2d6b8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e281538f0 0x7f3e281f0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2d6b8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e281538f0 0x7f3e281f0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42378/0 (socket says 172.21.15.148:42378) 2023-10-30T13:47:58.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2d6b8640 1 -- 172.21.15.148:0/4049726414 learned_addr learned my addr 172.21.15.148:0/4049726414 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:47:58.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2deb9640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e280cefa0 0x7f3e28144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2e6ba640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e2815cd50 0x7f3e280b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2d6b8640 1 -- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e2815cd50 msgr2=0x7f3e280b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2d6b8640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e2815cd50 0x7f3e280b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.180 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2d6b8640 1 -- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e280cefa0 msgr2=0x7f3e28144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2d6b8640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e280cefa0 0x7f3e28144fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2d6b8640 1 -- 172.21.15.148:0/4049726414 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e24009420 con 0x7f3e281538f0 2023-10-30T13:47:58.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2deb9640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e280cefa0 0x7f3e28144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:47:58.181 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2d6b8640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e281538f0 0x7f3e281f0750 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800c3c0 tx=0x7f3e1800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:58.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e1e7fc640 1 -- 172.21.15.148:0/4049726414 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180052d0 con 0x7f3e281538f0 2023-10-30T13:47:58.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e1e7fc640 1 -- 172.21.15.148:0/4049726414 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e18005430 con 0x7f3e281538f0 2023-10-30T13:47:58.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e1e7fc640 1 -- 172.21.15.148:0/4049726414 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18003840 con 0x7f3e281538f0 2023-10-30T13:47:58.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e280b5020 con 0x7f3e281538f0 2023-10-30T13:47:58.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.174+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2814ec00 con 0x7f3e281538f0 2023-10-30T13:47:58.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.178+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2804e170 con 0x7f3e281538f0 2023-10-30T13:47:58.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.178+0000 7f3e1e7fc640 1 -- 172.21.15.148:0/4049726414 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e180039a0 con 0x7f3e281538f0 2023-10-30T13:47:58.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.178+0000 7f3e1e7fc640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e0c0664a0 0x7f3e0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.178+0000 7f3e1e7fc640 1 -- 172.21.15.148:0/4049726414 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f3e180838c0 con 0x7f3e281538f0 2023-10-30T13:47:58.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.178+0000 7f3e2deb9640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e0c0664a0 0x7f3e0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.182+0000 7f3e2deb9640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e0c0664a0 0x7f3e0c068960 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3e240b2000 tx=0x7f3e240aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:58.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.182+0000 7f3e1e7fc640 1 -- 172.21.15.148:0/4049726414 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e180506a0 con 0x7f3e281538f0 2023-10-30T13:47:58.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.430+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e280b5020 con 0x7f3e281538f0 2023-10-30T13:47:58.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.430+0000 7f3e1e7fc640 1 -- 172.21.15.148:0/4049726414 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f3e280b5020 con 0x7f3e281538f0 2023-10-30T13:47:58.436 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:47:58.437 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:47:53.222242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 143 2023-10-30T13:47:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.434+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e0c0664a0 msgr2=0x7f3e0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.434+0000 7f3e2eebb640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e0c0664a0 0x7f3e0c068960 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3e240b2000 tx=0x7f3e240aa070 comp rx=0 tx=0).stop 2023-10-30T13:47:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.434+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e281538f0 msgr2=0x7f3e281f0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.434+0000 7f3e2eebb640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e281538f0 0x7f3e281f0750 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800c3c0 tx=0x7f3e1800c880 comp rx=0 tx=0).stop 2023-10-30T13:47:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.438+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 shutdown_connections 2023-10-30T13:47:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.438+0000 7f3e2eebb640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3e0c0664a0 0x7f3e0c068960 secure :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3e240b2000 tx=0x7f3e240aa070 comp rx=0 tx=0).stop 2023-10-30T13:47:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.438+0000 7f3e2eebb640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e281538f0 0x7f3e281f0750 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.438+0000 7f3e2eebb640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3e2815cd50 0x7f3e280b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.438+0000 7f3e2eebb640 1 --2- 172.21.15.148:0/4049726414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e280cefa0 0x7f3e28144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.438+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 >> 172.21.15.148:0/4049726414 conn(0x7f3e2806a640 msgr2=0x7f3e280caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.438+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 shutdown_connections 2023-10-30T13:47:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.438+0000 7f3e2eebb640 1 -- 172.21.15.148:0/4049726414 wait complete. 2023-10-30T13:47:58.461 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-30T13:47:58.462 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:47:58.462 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:47:58.497 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:47:58.497 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:47:58.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.634+0000 7f5c1f91d640 1 Processor -- start 2023-10-30T13:47:58.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.634+0000 7f5c1f91d640 1 -- start start 2023-10-30T13:47:58.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.634+0000 7f5c1f91d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c1806f7a0 0x7f5c1806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.634+0000 7f5c1f91d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5c18071110 0x7f5c18158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.634+0000 7f5c1f91d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c18066cf0 con 0x7f5c180c6bd0 2023-10-30T13:47:58.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.634+0000 7f5c1f91d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c180700c0 con 0x7f5c18071110 2023-10-30T13:47:58.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.634+0000 7f5c1f91d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180c70d0 con 0x7f5c1806f7a0 2023-10-30T13:47:58.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.634+0000 7f5c1ceb8640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5c180c6bd0 0x7f5c18069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36584/0 (socket says 172.21.15.148:36584) 2023-10-30T13:47:58.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.634+0000 7f5c1ceb8640 1 -- 172.21.15.148:0/3500026878 learned_addr learned my addr 172.21.15.148:0/3500026878 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:47:58.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1deba640 1 --2- 172.21.15.148:0/3500026878 >> v2:172.21.15.150:3300/0 conn(0x7f5c18071110 0x7f5c18158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1d6b9640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c1806f7a0 0x7f5c1806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110904032 0 0) 0x7f5c18066cf0 con 0x7f5c180c6bd0 2023-10-30T13:47:58.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c04003530 con 0x7f5c180c6bd0 2023-10-30T13:47:58.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1534283490 0 0) 0x7f5c04003530 con 0x7f5c180c6bd0 2023-10-30T13:47:58.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c1806f7a0 msgr2=0x7f5c1806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:58.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c1806f7a0 0x7f5c1806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 >> v2:172.21.15.150:3300/0 conn(0x7f5c18071110 msgr2=0x7f5c18158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:58.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 --2- 172.21.15.148:0/3500026878 >> v2:172.21.15.150:3300/0 conn(0x7f5c18071110 0x7f5c18158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c180c7390 con 0x7f5c180c6bd0 2023-10-30T13:47:58.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f5c08001680 con 0x7f5c180c6bd0 2023-10-30T13:47:58.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 >> v1:172.21.15.148:6789/0 conn(0x7f5c180c6bd0 legacy=0x7f5c18069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c04003980 0x7f5c04003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c1806f7a0 0x7f5c040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04004d10 0x7f5c04005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180700c0 con 0x7f5c04004d10 2023-10-30T13:47:58.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180c70d0 con 0x7f5c04003980 2023-10-30T13:47:58.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0400b990 con 0x7f5c1806f7a0 2023-10-30T13:47:58.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1deba640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04004d10 0x7f5c04005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1ceb8640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c1806f7a0 0x7f5c040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1d6b9640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c04003980 0x7f5c04003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1deba640 1 -- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c1806f7a0 msgr2=0x7f5c040047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:58.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1deba640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c1806f7a0 0x7f5c040047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1deba640 1 -- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c04003980 msgr2=0x7f5c04003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:47:58.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1deba640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c04003980 0x7f5c04003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.638+0000 7f5c1deba640 1 -- 172.21.15.148:0/3500026878 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c04003530 con 0x7f5c04004d10 2023-10-30T13:47:58.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1deba640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04004d10 0x7f5c04005190 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f5c10009fc0 tx=0x7f5c1001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c198734679117b4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:58.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c100a5cb0 con 0x7f5c04004d10 2023-10-30T13:47:58.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1000f040 con 0x7f5c04004d10 2023-10-30T13:47:58.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c0ffff640 1 -- 172.21.15.148:0/3500026878 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c100ad680 con 0x7f5c04004d10 2023-10-30T13:47:58.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1f91d640 1 -- 172.21.15.148:0/3500026878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04004d10 msgr2=0x7f5c04005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1f91d640 1 --2- 172.21.15.148:0/3500026878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c04004d10 0x7f5c04005190 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f5c10009fc0 tx=0x7f5c1001cca0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1deba640 1 -- 172.21.15.148:0/3500026878 reap_dead start 2023-10-30T13:47:58.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1f91d640 1 -- 172.21.15.148:0/3500026878 shutdown_connections 2023-10-30T13:47:58.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1f91d640 1 -- 172.21.15.148:0/3500026878 >> 172.21.15.148:0/3500026878 conn(0x7f5c180c7910 msgr2=0x7f5c180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:58.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1f91d640 1 -- 172.21.15.148:0/3500026878 shutdown_connections 2023-10-30T13:47:58.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1f91d640 1 -- 172.21.15.148:0/3500026878 wait complete. 2023-10-30T13:47:58.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1f91d640 1 Processor -- start 2023-10-30T13:47:58.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1f91d640 1 -- start start 2023-10-30T13:47:58.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.642+0000 7f5c1f91d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c04004d10 0x7f5c181f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1f91d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c180b0020 0x7f5c180af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1f91d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c181591b0 0x7f5c18159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1f91d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1815add0 con 0x7f5c180b0020 2023-10-30T13:47:58.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1f91d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1815a9d0 con 0x7f5c04004d10 2023-10-30T13:47:58.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1f91d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1815baf0 con 0x7f5c181591b0 2023-10-30T13:47:58.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1ceb8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c180b0020 0x7f5c180af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1ceb8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c180b0020 0x7f5c180af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42400/0 (socket says 172.21.15.148:42400) 2023-10-30T13:47:58.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1ceb8640 1 -- 172.21.15.148:0/1712883811 learned_addr learned my addr 172.21.15.148:0/1712883811 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:47:58.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1d6b9640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c04004d10 0x7f5c181f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1deba640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c181591b0 0x7f5c18159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1ceb8640 1 -- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c181591b0 msgr2=0x7f5c18159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1ceb8640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c181591b0 0x7f5c18159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1ceb8640 1 -- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c04004d10 msgr2=0x7f5c181f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1ceb8640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c04004d10 0x7f5c181f1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1ceb8640 1 -- 172.21.15.148:0/1712883811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c181f24e0 con 0x7f5c180b0020 2023-10-30T13:47:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1ceb8640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c180b0020 0x7f5c180af120 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800e880 tx=0x7f5c0800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c0dffb640 1 -- 172.21.15.148:0/1712883811 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0800c8d0 con 0x7f5c180b0020 2023-10-30T13:47:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c0dffb640 1 -- 172.21.15.148:0/1712883811 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0800ca30 con 0x7f5c180b0020 2023-10-30T13:47:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c0dffb640 1 -- 172.21.15.148:0/1712883811 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0800cde0 con 0x7f5c180b0020 2023-10-30T13:47:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1815b250 con 0x7f5c180b0020 2023-10-30T13:47:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1815b810 con 0x7f5c180b0020 2023-10-30T13:47:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1804e170 con 0x7f5c180b0020 2023-10-30T13:47:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c0dffb640 1 -- 172.21.15.148:0/1712883811 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5c0801d460 con 0x7f5c180b0020 2023-10-30T13:47:58.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.646+0000 7f5c0dffb640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5bfc0664a0 0x7f5bfc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:47:58.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.650+0000 7f5c0dffb640 1 -- 172.21.15.148:0/1712883811 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f5c08084760 con 0x7f5c180b0020 2023-10-30T13:47:58.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.650+0000 7f5c1d6b9640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5bfc0664a0 0x7f5bfc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:47:58.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.650+0000 7f5c1d6b9640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5bfc0664a0 0x7f5bfc068960 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5c140099b0 tx=0x7f5c14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:47:58.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.650+0000 7f5c0dffb640 1 -- 172.21.15.148:0/1712883811 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5c08051410 con 0x7f5c180b0020 2023-10-30T13:47:58.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.834+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c18066510 con 0x7f5c180b0020 2023-10-30T13:47:58.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.834+0000 7f5c0dffb640 1 -- 172.21.15.148:0/1712883811 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f5c08054a30 con 0x7f5c180b0020 2023-10-30T13:47:58.838 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:47:58.838 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:47:53.222242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:47:58.838 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 143 2023-10-30T13:47:58.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.834+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5bfc0664a0 msgr2=0x7f5bfc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.834+0000 7f5c1f91d640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5bfc0664a0 0x7f5bfc068960 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5c140099b0 tx=0x7f5c14008040 comp rx=0 tx=0).stop 2023-10-30T13:47:58.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.834+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c180b0020 msgr2=0x7f5c180af120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:47:58.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.834+0000 7f5c1f91d640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c180b0020 0x7f5c180af120 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800e880 tx=0x7f5c0800ed40 comp rx=0 tx=0).stop 2023-10-30T13:47:58.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.838+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 shutdown_connections 2023-10-30T13:47:58.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.838+0000 7f5c1f91d640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5bfc0664a0 0x7f5bfc068960 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.838+0000 7f5c1f91d640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5c180b0020 0x7f5c180af120 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.838+0000 7f5c1f91d640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5c181591b0 0x7f5c18159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.838+0000 7f5c1f91d640 1 --2- 172.21.15.148:0/1712883811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c04004d10 0x7f5c181f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:47:58.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.838+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 >> 172.21.15.148:0/1712883811 conn(0x7f5c180c7910 msgr2=0x7f5c180c9020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:47:58.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.838+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 shutdown_connections 2023-10-30T13:47:58.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:47:58.838+0000 7f5c1f91d640 1 -- 172.21.15.148:0/1712883811 wait complete. 2023-10-30T13:47:58.858 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:48:00.859 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:01.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89b532640 1 Processor -- start 2023-10-30T13:48:01.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89b532640 1 -- start start 2023-10-30T13:48:01.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89b532640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 0x7fa894159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:01.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89b532640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa89415cd50 0x7fa8940c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:01.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89b532640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89415d300 con 0x7fa8940cefa0 2023-10-30T13:48:01.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89b532640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa89415d5e0 con 0x7fa89415cd50 2023-10-30T13:48:01.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89b532640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89415d8c0 con 0x7fa8941538f0 2023-10-30T13:48:01.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89ad31640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa8940cefa0 0x7fa8940cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36598/0 (socket says 172.21.15.148:36598) 2023-10-30T13:48:01.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89ad31640 1 -- 172.21.15.148:0/790532835 learned_addr learned my addr 172.21.15.148:0/790532835 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:01.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89a530640 1 --2- 172.21.15.148:0/790532835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 0x7fa894159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:01.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa899d2f640 1 --2- 172.21.15.148:0/790532835 >> v2:172.21.15.150:3300/0 conn(0x7fa89415cd50 0x7fa8940c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:01.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89952e640 1 -- 172.21.15.148:0/790532835 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266980266 0 0) 0x7fa89415d300 con 0x7fa8940cefa0 2023-10-30T13:48:01.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.002+0000 7fa89952e640 1 -- 172.21.15.148:0/790532835 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa880003530 con 0x7fa8940cefa0 2023-10-30T13:48:01.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89a530640 1 -- 172.21.15.148:0/790532835 >> v2:172.21.15.150:3300/0 conn(0x7fa89415cd50 msgr2=0x7fa8940c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:01.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89a530640 1 --2- 172.21.15.148:0/790532835 >> v2:172.21.15.150:3300/0 conn(0x7fa89415cd50 0x7fa8940c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:01.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89a530640 1 -- 172.21.15.148:0/790532835 >> v1:172.21.15.148:6789/0 conn(0x7fa8940cefa0 legacy=0x7fa8940cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:01.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89a530640 1 -- 172.21.15.148:0/790532835 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89415dba0 con 0x7fa8941538f0 2023-10-30T13:48:01.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa899d2f640 1 --2- 172.21.15.148:0/790532835 >> v2:172.21.15.150:3300/0 conn(0x7fa89415cd50 0x7fa8940c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:48:01.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89a530640 1 --2- 172.21.15.148:0/790532835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 0x7fa894159fb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa8900097c0 tx=0x7fa89001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9cc3ad2d72617d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:01.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89952e640 1 -- 172.21.15.148:0/790532835 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8900a8970 con 0x7fa8941538f0 2023-10-30T13:48:01.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89952e640 1 -- 172.21.15.148:0/790532835 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8900a8ad0 con 0x7fa8941538f0 2023-10-30T13:48:01.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89952e640 1 -- 172.21.15.148:0/790532835 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8900a8e60 con 0x7fa8941538f0 2023-10-30T13:48:01.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89b532640 1 -- 172.21.15.148:0/790532835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 msgr2=0x7fa894159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:01.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.006+0000 7fa89b532640 1 --2- 172.21.15.148:0/790532835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 0x7fa894159fb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa8900097c0 tx=0x7fa89001ec70 comp rx=0 tx=0).stop 2023-10-30T13:48:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 -- 172.21.15.148:0/790532835 shutdown_connections 2023-10-30T13:48:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 --2- 172.21.15.148:0/790532835 >> v2:172.21.15.150:3300/0 conn(0x7fa89415cd50 0x7fa8940c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 --2- 172.21.15.148:0/790532835 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 0x7fa894159fb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 -- 172.21.15.148:0/790532835 >> 172.21.15.148:0/790532835 conn(0x7fa89406a640 msgr2=0x7fa894065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 -- 172.21.15.148:0/790532835 shutdown_connections 2023-10-30T13:48:01.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 -- 172.21.15.148:0/790532835 wait complete. 2023-10-30T13:48:01.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 Processor -- start 2023-10-30T13:48:01.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 -- start start 2023-10-30T13:48:01.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8940cefa0 0x7fa89414f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:01.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 0x7fa894152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:01.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89415cd50 0x7fa8941526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:01.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa894148660 con 0x7fa89415cd50 2023-10-30T13:48:01.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940c9260 con 0x7fa8940cefa0 2023-10-30T13:48:01.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89b532640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89414fa00 con 0x7fa8941538f0 2023-10-30T13:48:01.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89ad31640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89415cd50 0x7fa8941526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:01.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89ad31640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89415cd50 0x7fa8941526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42406/0 (socket says 172.21.15.148:42406) 2023-10-30T13:48:01.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89ad31640 1 -- 172.21.15.148:0/347566383 learned_addr learned my addr 172.21.15.148:0/347566383 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:01.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89a530640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8940cefa0 0x7fa89414f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:01.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa899d2f640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 0x7fa894152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:01.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89ad31640 1 -- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 msgr2=0x7fa894152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:01.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89ad31640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 0x7fa894152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:01.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89ad31640 1 -- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8940cefa0 msgr2=0x7fa89414f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:01.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89ad31640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8940cefa0 0x7fa89414f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:01.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89ad31640 1 -- 172.21.15.148:0/347566383 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa884009420 con 0x7fa89415cd50 2023-10-30T13:48:01.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.010+0000 7fa89ad31640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89415cd50 0x7fa8941526b0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00cb70 tx=0x7fa88c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:01.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa88affd640 1 -- 172.21.15.148:0/347566383 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00ace0 con 0x7fa89415cd50 2023-10-30T13:48:01.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa88affd640 1 -- 172.21.15.148:0/347566383 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88c00ae40 con 0x7fa89415cd50 2023-10-30T13:48:01.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa890009420 con 0x7fa89415cd50 2023-10-30T13:48:01.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa88affd640 1 -- 172.21.15.148:0/347566383 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c012620 con 0x7fa89415cd50 2023-10-30T13:48:01.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa894147d50 con 0x7fa89415cd50 2023-10-30T13:48:01.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8941487a0 con 0x7fa89415cd50 2023-10-30T13:48:01.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa88affd640 1 -- 172.21.15.148:0/347566383 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa88c007070 con 0x7fa89415cd50 2023-10-30T13:48:01.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa88affd640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8780664d0 0x7fa878068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:01.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa88affd640 1 -- 172.21.15.148:0/347566383 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7fa88c0844b0 con 0x7fa89415cd50 2023-10-30T13:48:01.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.014+0000 7fa89a530640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8780664d0 0x7fa878068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:01.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.018+0000 7fa89a530640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8780664d0 0x7fa878068990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa8900b2000 tx=0x7fa890008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:01.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.018+0000 7fa88affd640 1 -- 172.21.15.148:0/347566383 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa88c0512a0 con 0x7fa89415cd50 2023-10-30T13:48:01.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.202+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8940b4cb0 con 0x7fa89415cd50 2023-10-30T13:48:01.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.202+0000 7fa88affd640 1 -- 172.21.15.148:0/347566383 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6532 (secure 0 0 0) 0x7fa88c0548c0 con 0x7fa89415cd50 2023-10-30T13:48:01.206 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:01.206 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:47:53.222242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:01.207 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 143 2023-10-30T13:48:01.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8780664d0 msgr2=0x7fa878068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:01.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8780664d0 0x7fa878068990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa8900b2000 tx=0x7fa890008040 comp rx=0 tx=0).stop 2023-10-30T13:48:01.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89415cd50 msgr2=0x7fa8941526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:01.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89415cd50 0x7fa8941526b0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00cb70 tx=0x7fa88c00a430 comp rx=0 tx=0).stop 2023-10-30T13:48:01.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 shutdown_connections 2023-10-30T13:48:01.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa8780664d0 0x7fa878068990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:01.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa89415cd50 0x7fa8941526b0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:01.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa8941538f0 0x7fa894152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:01.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 --2- 172.21.15.148:0/347566383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8940cefa0 0x7fa89414f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:01.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 >> 172.21.15.148:0/347566383 conn(0x7fa89406a640 msgr2=0x7fa8940c5290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:01.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 shutdown_connections 2023-10-30T13:48:01.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:01.206+0000 7fa89b532640 1 -- 172.21.15.148:0/347566383 wait complete. 2023-10-30T13:48:01.227 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:48:02.936 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:02.940 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:03.228 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:03.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.370+0000 7fbbd9dab640 1 Processor -- start 2023-10-30T13:48:03.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.370+0000 7fbbd9dab640 1 -- start start 2023-10-30T13:48:03.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd9dab640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:03.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd9dab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbbd40c6bd0 0x7fbbd4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:03.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd9dab640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd406a3d0 con 0x7fbbd4071110 2023-10-30T13:48:03.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd9dab640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd40700c0 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd9dab640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd4066cf0 con 0x7fbbd406f7a0 2023-10-30T13:48:03.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd37fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:03.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd3fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbbd4071110 0x7fbbd4158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:33062/0 (socket says 172.21.15.148:33062) 2023-10-30T13:48:03.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd3fff640 1 -- 172.21.15.148:0/272465064 learned_addr learned my addr 172.21.15.148:0/272465064 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:03.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd2ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbbd40c6bd0 0x7fbbd4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:03.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd37fe640 1 --2- 172.21.15.148:0/272465064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:38132/0 (socket says 172.21.15.148:38132) 2023-10-30T13:48:03.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd27fc640 1 -- 172.21.15.148:0/272465064 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689300445 0 0) 0x7fbbd406a3d0 con 0x7fbbd4071110 2023-10-30T13:48:03.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd27fc640 1 -- 172.21.15.148:0/272465064 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbbc003530 con 0x7fbbd4071110 2023-10-30T13:48:03.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd37fe640 1 -- 172.21.15.148:0/272465064 >> v2:172.21.15.150:3300/0 conn(0x7fbbd40c6bd0 msgr2=0x7fbbd4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:03.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd37fe640 1 --2- 172.21.15.148:0/272465064 >> v2:172.21.15.150:3300/0 conn(0x7fbbd40c6bd0 0x7fbbd4069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:03.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd37fe640 1 -- 172.21.15.148:0/272465064 >> v1:172.21.15.150:6789/0 conn(0x7fbbd4071110 legacy=0x7fbbd4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:03.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.374+0000 7fbbd37fe640 1 -- 172.21.15.148:0/272465064 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd40c70d0 con 0x7fbbd406f7a0 2023-10-30T13:48:03.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd37fe640 1 --2- 172.21.15.148:0/272465064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd406fb80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40097c0 tx=0x7fbbc401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=746effacf4ed2eaf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:03.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd27fc640 1 -- 172.21.15.148:0/272465064 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc40a8970 con 0x7fbbd406f7a0 2023-10-30T13:48:03.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd27fc640 1 -- 172.21.15.148:0/272465064 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc40a8ad0 con 0x7fbbd406f7a0 2023-10-30T13:48:03.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd27fc640 1 -- 172.21.15.148:0/272465064 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc40a8e60 con 0x7fbbd406f7a0 2023-10-30T13:48:03.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 -- 172.21.15.148:0/272465064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 msgr2=0x7fbbd406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:03.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 --2- 172.21.15.148:0/272465064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd406fb80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40097c0 tx=0x7fbbc401ec70 comp rx=0 tx=0).stop 2023-10-30T13:48:03.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 -- 172.21.15.148:0/272465064 shutdown_connections 2023-10-30T13:48:03.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 --2- 172.21.15.148:0/272465064 >> v2:172.21.15.150:3300/0 conn(0x7fbbd40c6bd0 0x7fbbd4069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:03.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 --2- 172.21.15.148:0/272465064 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd406fb80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:03.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 -- 172.21.15.148:0/272465064 >> 172.21.15.148:0/272465064 conn(0x7fbbd40c7910 msgr2=0x7fbbd40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:03.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 -- 172.21.15.148:0/272465064 shutdown_connections 2023-10-30T13:48:03.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 -- 172.21.15.148:0/272465064 wait complete. 2023-10-30T13:48:03.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 Processor -- start 2023-10-30T13:48:03.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.378+0000 7fbbd9dab640 1 -- start start 2023-10-30T13:48:03.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd9dab640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd40c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:03.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd9dab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbd4071110 0x7fbbd41eee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:03.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd9dab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbd40c6bd0 0x7fbbd41ef3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:03.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd9dab640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd414cf70 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd9dab640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd414d390 con 0x7fbbd4071110 2023-10-30T13:48:03.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd9dab640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd40ada50 con 0x7fbbd406f7a0 2023-10-30T13:48:03.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbd40c6bd0 0x7fbbd41ef3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:03.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd37fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd40c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:03.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbd40c6bd0 0x7fbbd41ef3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36184/0 (socket says 172.21.15.148:36184) 2023-10-30T13:48:03.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd3fff640 1 -- 172.21.15.148:0/1487312266 learned_addr learned my addr 172.21.15.148:0/1487312266 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:03.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd2ffd640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbd4071110 0x7fbbd41eee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:03.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd3fff640 1 -- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 msgr2=0x7fbbd40c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:03.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd3fff640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd40c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd3fff640 1 -- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbd4071110 msgr2=0x7fbbd41eee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd3fff640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbd4071110 0x7fbbd41eee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd3fff640 1 -- 172.21.15.148:0/1487312266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc4009420 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd3fff640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbd40c6bd0 0x7fbbd41ef3b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00cad0 tx=0x7fbbcc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbbbfff640 1 -- 172.21.15.148:0/1487312266 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc00ac30 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbbbfff640 1 -- 172.21.15.148:0/1487312266 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbcc00ad90 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbbbfff640 1 -- 172.21.15.148:0/1487312266 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc012620 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd40add30 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd414c8b0 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd414d0b0 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.382+0000 7fbbbbfff640 1 -- 172.21.15.148:0/1487312266 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbbcc007070 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.386+0000 7fbbbbfff640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbbc00664a0 0x7fbbc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:03.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.386+0000 7fbbbbfff640 1 -- 172.21.15.148:0/1487312266 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7fbbcc084460 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.386+0000 7fbbd37fe640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbbc00664a0 0x7fbbc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:03.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.386+0000 7fbbd37fe640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbbc00664a0 0x7fbbc0068960 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40b2000 tx=0x7fbbc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:03.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.390+0000 7fbbbbfff640 1 -- 172.21.15.148:0/1487312266 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbbd414d0b0 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbd414d0b0 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbbbfff640 1 -- 172.21.15.148:0/1487312266 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6532 (secure 0 0 0) 0x7fbbd414d0b0 con 0x7fbbd40c6bd0 2023-10-30T13:48:03.573 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:03.573 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:47:53.222242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:03.573 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 143 2023-10-30T13:48:03.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbbc00664a0 msgr2=0x7fbbc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:03.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbbc00664a0 0x7fbbc0068960 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40b2000 tx=0x7fbbc4008040 comp rx=0 tx=0).stop 2023-10-30T13:48:03.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbd40c6bd0 msgr2=0x7fbbd41ef3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:03.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbd40c6bd0 0x7fbbd41ef3b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00cad0 tx=0x7fbbcc00cf90 comp rx=0 tx=0).stop 2023-10-30T13:48:03.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 shutdown_connections 2023-10-30T13:48:03.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbbc00664a0 0x7fbbc0068960 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:03.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbd40c6bd0 0x7fbbd41ef3b0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:03.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbd4071110 0x7fbbd41eee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:03.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 --2- 172.21.15.148:0/1487312266 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbbd406f7a0 0x7fbbd40c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:03.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 >> 172.21.15.148:0/1487312266 conn(0x7fbbd40c7910 msgr2=0x7fbbd40c2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:03.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.570+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 shutdown_connections 2023-10-30T13:48:03.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:03.574+0000 7fbbd9dab640 1 -- 172.21.15.148:0/1487312266 wait complete. 2023-10-30T13:48:03.594 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:48:05.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:05.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0c06446640 1 Processor -- start 2023-10-30T13:48:05.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0c06446640 1 -- start start 2023-10-30T13:48:05.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0c06446640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0c001539a0 0x7f0c0015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:05.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0c06446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 0x7f0c0006a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:05.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0c06446640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0015a8c0 con 0x7f0c00070a00 2023-10-30T13:48:05.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0c06446640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0006fbb0 con 0x7f0c001539a0 2023-10-30T13:48:05.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0c06446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c00153d80 con 0x7f0c00069e90 2023-10-30T13:48:05.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0bfffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0c001539a0 0x7f0c0015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:05.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0bfffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0c001539a0 0x7f0c0015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36186/0 (socket says 172.21.15.148:36186) 2023-10-30T13:48:05.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0bfffff640 1 -- 172.21.15.148:0/3803920502 learned_addr learned my addr 172.21.15.148:0/3803920502 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:05.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0bff7fe640 1 --1- 172.21.15.148:0/3803920502 >> v1:172.21.15.148:6789/0 conn(0x7f0c00070a00 0x7f0c00070e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47312/0 (socket says 172.21.15.148:47312) 2023-10-30T13:48:05.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0c049e3640 1 --2- 172.21.15.148:0/3803920502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 0x7f0c0006a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:05.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0bfffff640 1 -- 172.21.15.148:0/3803920502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 msgr2=0x7f0c0006a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:05.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0bfffff640 1 --2- 172.21.15.148:0/3803920502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 0x7f0c0006a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0bfffff640 1 -- 172.21.15.148:0/3803920502 >> v1:172.21.15.148:6789/0 conn(0x7f0c00070a00 legacy=0x7f0c00070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:05.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0bfffff640 1 -- 172.21.15.148:0/3803920502 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000c5c50 con 0x7f0c001539a0 2023-10-30T13:48:05.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.742+0000 7f0bfeffd640 1 -- 172.21.15.148:0/3803920502 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529799168 0 0) 0x7f0c0015a8c0 con 0x7f0c00070a00 2023-10-30T13:48:05.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0bfffff640 1 --2- 172.21.15.148:0/3803920502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 0x7f0c0015a380 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009870 tx=0x7f0bf001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41afe6877fd143a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:05.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0bfeffd640 1 -- 172.21.15.148:0/3803920502 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf00ad640 con 0x7f0c001539a0 2023-10-30T13:48:05.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0bfeffd640 1 -- 172.21.15.148:0/3803920502 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf00ad7a0 con 0x7f0c001539a0 2023-10-30T13:48:05.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0bfeffd640 1 -- 172.21.15.148:0/3803920502 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf00adb30 con 0x7f0c001539a0 2023-10-30T13:48:05.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0c06446640 1 -- 172.21.15.148:0/3803920502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 msgr2=0x7f0c0015a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:05.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0c06446640 1 --2- 172.21.15.148:0/3803920502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 0x7f0c0015a380 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009870 tx=0x7f0bf001c480 comp rx=0 tx=0).stop 2023-10-30T13:48:05.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0c06446640 1 -- 172.21.15.148:0/3803920502 shutdown_connections 2023-10-30T13:48:05.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0c06446640 1 --2- 172.21.15.148:0/3803920502 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 0x7f0c0006a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0c06446640 1 --2- 172.21.15.148:0/3803920502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 0x7f0c0015a380 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0c06446640 1 -- 172.21.15.148:0/3803920502 >> 172.21.15.148:0/3803920502 conn(0x7f0c000c7910 msgr2=0x7f0c000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:05.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0c06446640 1 -- 172.21.15.148:0/3803920502 shutdown_connections 2023-10-30T13:48:05.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.746+0000 7f0c06446640 1 -- 172.21.15.148:0/3803920502 wait complete. 2023-10-30T13:48:05.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c06446640 1 Processor -- start 2023-10-30T13:48:05.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c06446640 1 -- start start 2023-10-30T13:48:05.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c06446640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 0x7f0c000b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:05.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c06446640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c00070a00 0x7f0c000c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:05.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c06446640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 0x7f0c000c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:05.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c06446640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000c4700 con 0x7f0c001539a0 2023-10-30T13:48:05.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c06446640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000c51f0 con 0x7f0c00070a00 2023-10-30T13:48:05.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c06446640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000caca0 con 0x7f0c00069e90 2023-10-30T13:48:05.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c049e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 0x7f0c000c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:05.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c049e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 0x7f0c000c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36196/0 (socket says 172.21.15.148:36196) 2023-10-30T13:48:05.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c049e3640 1 -- 172.21.15.148:0/3153323713 learned_addr learned my addr 172.21.15.148:0/3153323713 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:05.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0bfffff640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 0x7f0c000b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:05.757 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0bff7fe640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c00070a00 0x7f0c000c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:05.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c049e3640 1 -- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 msgr2=0x7f0c000b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:05.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c049e3640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 0x7f0c000b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c049e3640 1 -- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c00070a00 msgr2=0x7f0c000c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:05.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c049e3640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c00070a00 0x7f0c000c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.758 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c049e3640 1 -- 172.21.15.148:0/3153323713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf00094d0 con 0x7f0c001539a0 2023-10-30T13:48:05.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0c049e3640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 0x7f0c000c12a0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8007e40 tx=0x7f0bf800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:05.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.750+0000 7f0bfcff9640 1 -- 172.21.15.148:0/3153323713 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8014620 con 0x7f0c001539a0 2023-10-30T13:48:05.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.754+0000 7f0bfcff9640 1 -- 172.21.15.148:0/3153323713 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf80147a0 con 0x7f0c001539a0 2023-10-30T13:48:05.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.754+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c000c2cb0 con 0x7f0c001539a0 2023-10-30T13:48:05.759 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.754+0000 7f0bfcff9640 1 -- 172.21.15.148:0/3153323713 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8014b50 con 0x7f0c001539a0 2023-10-30T13:48:05.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.754+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c000c4c30 con 0x7f0c001539a0 2023-10-30T13:48:05.760 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.754+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c000c2e40 con 0x7f0c001539a0 2023-10-30T13:48:05.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.754+0000 7f0bfcff9640 1 -- 172.21.15.148:0/3153323713 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0bf8014cb0 con 0x7f0c001539a0 2023-10-30T13:48:05.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.754+0000 7f0bfcff9640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bec0664a0 0x7f0bec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:05.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.754+0000 7f0bfcff9640 1 -- 172.21.15.148:0/3153323713 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f0bf8084dc0 con 0x7f0c001539a0 2023-10-30T13:48:05.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.754+0000 7f0bfffff640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bec0664a0 0x7f0bec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:05.761 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.758+0000 7f0bfffff640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bec0664a0 0x7f0bec068960 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00af000 tx=0x7f0bf0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:05.762 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.758+0000 7f0bfcff9640 1 -- 172.21.15.148:0/3153323713 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0bf8051c40 con 0x7f0c001539a0 2023-10-30T13:48:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.938+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c000c64c0 con 0x7f0c001539a0 2023-10-30T13:48:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.938+0000 7f0bfcff9640 1 -- 172.21.15.148:0/3153323713 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f0bf8055260 con 0x7f0c001539a0 2023-10-30T13:48:05.943 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:05.943 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:47:53.222242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 143 2023-10-30T13:48:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bec0664a0 msgr2=0x7f0bec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bec0664a0 0x7f0bec068960 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00af000 tx=0x7f0bf0008040 comp rx=0 tx=0).stop 2023-10-30T13:48:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 msgr2=0x7f0c000c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 0x7f0c000c12a0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8007e40 tx=0x7f0bf800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 shutdown_connections 2023-10-30T13:48:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0bec0664a0 0x7f0bec068960 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0c001539a0 0x7f0c000c12a0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c00070a00 0x7f0c000c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 --2- 172.21.15.148:0/3153323713 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0c00069e90 0x7f0c000b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:05.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 >> 172.21.15.148:0/3153323713 conn(0x7f0c000c7910 msgr2=0x7f0c0014f170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:05.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 shutdown_connections 2023-10-30T13:48:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:05.942+0000 7f0c06446640 1 -- 172.21.15.148:0/3153323713 wait complete. 2023-10-30T13:48:05.964 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:48:07.965 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:08.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.110+0000 7f2b5b491640 1 Processor -- start 2023-10-30T13:48:08.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b5b491640 1 -- start start 2023-10-30T13:48:08.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b5b491640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2b5406d290 0x7f2b5406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:08.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b5b491640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b540c6c10 con 0x7f2b54069e90 2023-10-30T13:48:08.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b5b491640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b540c6ef0 con 0x7f2b5415a410 2023-10-30T13:48:08.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b5b491640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b540c8570 con 0x7f2b5406d290 2023-10-30T13:48:08.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b5922d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2b54069e90 0x7f2b540c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47324/0 (socket says 172.21.15.148:47324) 2023-10-30T13:48:08.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b5922d640 1 -- 172.21.15.148:0/3287192318 learned_addr learned my addr 172.21.15.148:0/3287192318 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:08.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b59a2e640 1 --2- 172.21.15.148:0/3287192318 >> v2:172.21.15.150:3300/0 conn(0x7f2b5406d290 0x7f2b5406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:08.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776383496 0 0) 0x7f2b540c6c10 con 0x7f2b54069e90 2023-10-30T13:48:08.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b40003530 con 0x7f2b54069e90 2023-10-30T13:48:08.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2982883313 0 0) 0x7f2b40003530 con 0x7f2b54069e90 2023-10-30T13:48:08.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 >> v2:172.21.15.150:3300/0 conn(0x7f2b5406d290 msgr2=0x7f2b5406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:08.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 --2- 172.21.15.148:0/3287192318 >> v2:172.21.15.150:3300/0 conn(0x7f2b5406d290 0x7f2b5406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 >> v1:172.21.15.150:6789/0 conn(0x7f2b5415a410 legacy=0x7f2b5406cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:08.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b540c71d0 con 0x7f2b54069e90 2023-10-30T13:48:08.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f2b50001680 con 0x7f2b54069e90 2023-10-30T13:48:08.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 >> v1:172.21.15.148:6789/0 conn(0x7f2b54069e90 legacy=0x7f2b540c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:08.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b40003980 0x7f2b40003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:08.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b400047d0 0x7f2b40004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:08.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b4000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:08.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b540c8570 con 0x7f2b40005190 2023-10-30T13:48:08.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4000c030 con 0x7f2b40003980 2023-10-30T13:48:08.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4000c1f0 con 0x7f2b400047d0 2023-10-30T13:48:08.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b59a2e640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b4000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:08.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b5922d640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b40003980 0x7f2b40003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:08.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b58a2c640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b400047d0 0x7f2b40004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:08.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b59a2e640 1 -- 172.21.15.148:0/3287192318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b400047d0 msgr2=0x7f2b40004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:08.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b59a2e640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b400047d0 0x7f2b40004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b59a2e640 1 -- 172.21.15.148:0/3287192318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b40003980 msgr2=0x7f2b40003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:08.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b59a2e640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b40003980 0x7f2b40003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b59a2e640 1 -- 172.21.15.148:0/3287192318 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b40003530 con 0x7f2b40005190 2023-10-30T13:48:08.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.114+0000 7f2b58a2c640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b400047d0 0x7f2b40004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:08.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b59a2e640 1 -- 172.21.15.148:0/3287192318 reap_dead start 2023-10-30T13:48:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b59a2e640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b4000b990 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c009fc0 tx=0x7f2b4c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cb25c02d0c8ca4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c0a5cb0 con 0x7f2b40005190 2023-10-30T13:48:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c00f040 con 0x7f2b40005190 2023-10-30T13:48:08.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b4bfff640 1 -- 172.21.15.148:0/3287192318 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c003be0 con 0x7f2b40005190 2023-10-30T13:48:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b5b491640 1 -- 172.21.15.148:0/3287192318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 msgr2=0x7f2b4000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b5b491640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b4000b990 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c009fc0 tx=0x7f2b4c01cca0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b5b491640 1 -- 172.21.15.148:0/3287192318 shutdown_connections 2023-10-30T13:48:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b5b491640 1 --2- 172.21.15.148:0/3287192318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b4000b990 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b5b491640 1 -- 172.21.15.148:0/3287192318 >> 172.21.15.148:0/3287192318 conn(0x7f2b5406dfd0 msgr2=0x7f2b5406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b5b491640 1 -- 172.21.15.148:0/3287192318 shutdown_connections 2023-10-30T13:48:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.118+0000 7f2b5b491640 1 -- 172.21.15.148:0/3287192318 wait complete. 2023-10-30T13:48:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 Processor -- start 2023-10-30T13:48:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 -- start start 2023-10-30T13:48:08.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b541f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:08.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b5406b540 0x7f2b5415b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:08.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54155ae0 0x7f2b54155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:08.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5415b1d0 con 0x7f2b40005190 2023-10-30T13:48:08.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b541556c0 con 0x7f2b54155ae0 2023-10-30T13:48:08.132 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54150490 con 0x7f2b5406b540 2023-10-30T13:48:08.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5922d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b541f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:08.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5922d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b541f2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36218/0 (socket says 172.21.15.148:36218) 2023-10-30T13:48:08.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5922d640 1 -- 172.21.15.148:0/3204398341 learned_addr learned my addr 172.21.15.148:0/3204398341 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:08.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b59a2e640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54155ae0 0x7f2b54155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:08.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b58a2c640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b5406b540 0x7f2b5415b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:08.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5922d640 1 -- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b5406b540 msgr2=0x7f2b5415b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:08.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5922d640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b5406b540 0x7f2b5415b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5922d640 1 -- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54155ae0 msgr2=0x7f2b54155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:08.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5922d640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54155ae0 0x7f2b54155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5922d640 1 -- 172.21.15.148:0/3204398341 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5415a9c0 con 0x7f2b40005190 2023-10-30T13:48:08.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b59a2e640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54155ae0 0x7f2b54155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:08.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5922d640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b541f2770 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f2b500072c0 tx=0x7f2b50007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:08.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b49ffb640 1 -- 172.21.15.148:0/3204398341 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5000f9c0 con 0x7f2b40005190 2023-10-30T13:48:08.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b49ffb640 1 -- 172.21.15.148:0/3204398341 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5000fb40 con 0x7f2b40005190 2023-10-30T13:48:08.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b49ffb640 1 -- 172.21.15.148:0/3204398341 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b500133f0 con 0x7f2b40005190 2023-10-30T13:48:08.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5415dc40 con 0x7f2b40005190 2023-10-30T13:48:08.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.122+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5415ac10 con 0x7f2b40005190 2023-10-30T13:48:08.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.126+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b54155800 con 0x7f2b40005190 2023-10-30T13:48:08.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.126+0000 7f2b49ffb640 1 -- 172.21.15.148:0/3204398341 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2b5001c3b0 con 0x7f2b40005190 2023-10-30T13:48:08.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.126+0000 7f2b49ffb640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b380664a0 0x7f2b38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:08.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.126+0000 7f2b49ffb640 1 -- 172.21.15.148:0/3204398341 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f2b50083e70 con 0x7f2b40005190 2023-10-30T13:48:08.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.126+0000 7f2b58a2c640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b380664a0 0x7f2b38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:08.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.126+0000 7f2b58a2c640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b380664a0 0x7f2b38068960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2b540c6160 tx=0x7f2b44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:08.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.130+0000 7f2b49ffb640 1 -- 172.21.15.148:0/3204398341 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2b54155800 con 0x7f2b40005190 2023-10-30T13:48:08.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b54155800 con 0x7f2b40005190 2023-10-30T13:48:08.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b49ffb640 1 -- 172.21.15.148:0/3204398341 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f2b54155800 con 0x7f2b40005190 2023-10-30T13:48:08.361 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:08.361 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:47:53.222242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:08.362 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 143 2023-10-30T13:48:08.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b380664a0 msgr2=0x7f2b38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:08.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b380664a0 0x7f2b38068960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2b540c6160 tx=0x7f2b44009040 comp rx=0 tx=0).stop 2023-10-30T13:48:08.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 msgr2=0x7f2b541f2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:08.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b541f2770 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f2b500072c0 tx=0x7f2b50007780 comp rx=0 tx=0).stop 2023-10-30T13:48:08.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 shutdown_connections 2023-10-30T13:48:08.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2b380664a0 0x7f2b38068960 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b54155ae0 0x7f2b54155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2b5406b540 0x7f2b5415b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 --2- 172.21.15.148:0/3204398341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b40005190 0x7f2b541f2770 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:08.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 >> 172.21.15.148:0/3204398341 conn(0x7f2b5406dfd0 msgr2=0x7f2b5406e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:08.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 shutdown_connections 2023-10-30T13:48:08.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:08.358+0000 7f2b5b491640 1 -- 172.21.15.148:0/3204398341 wait complete. 2023-10-30T13:48:08.382 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:48:10.383 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:10.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.530+0000 7f4272409640 1 Processor -- start 2023-10-30T13:48:10.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f4272409640 1 -- start start 2023-10-30T13:48:10.536 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f4272409640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1538a0 0x7f426c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:10.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f4272409640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c0cb800 con 0x7f426c0cf160 2023-10-30T13:48:10.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f4272409640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c15dc30 con 0x7f426c0ca0e0 2023-10-30T13:48:10.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f4272409640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0cb330 con 0x7f426c1538a0 2023-10-30T13:48:10.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f42709a6640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f426c0cf160 0x7f426c164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47334/0 (socket says 172.21.15.148:47334) 2023-10-30T13:48:10.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f42709a6640 1 -- 172.21.15.148:0/711387245 learned_addr learned my addr 172.21.15.148:0/711387245 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:10.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f426bfff640 1 --2- 172.21.15.148:0/711387245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1538a0 0x7f426c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:10.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f426affd640 1 -- 172.21.15.148:0/711387245 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561357798 0 0) 0x7f426c0cb800 con 0x7f426c0cf160 2023-10-30T13:48:10.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f426affd640 1 -- 172.21.15.148:0/711387245 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4254003530 con 0x7f426c0cf160 2023-10-30T13:48:10.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f426affd640 1 -- 172.21.15.148:0/711387245 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4110607003 0 0) 0x7f4254003530 con 0x7f426c0cf160 2023-10-30T13:48:10.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f426bfff640 1 -- 172.21.15.148:0/711387245 >> v1:172.21.15.150:6789/0 conn(0x7f426c0ca0e0 legacy=0x7f426c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:10.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f426bfff640 1 -- 172.21.15.148:0/711387245 >> v1:172.21.15.148:6789/0 conn(0x7f426c0cf160 legacy=0x7f426c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:10.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.534+0000 7f426bfff640 1 -- 172.21.15.148:0/711387245 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c0cad70 con 0x7f426c1538a0 2023-10-30T13:48:10.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f426bfff640 1 --2- 172.21.15.148:0/711387245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1538a0 0x7f426c15d6f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f425c0097f0 tx=0x7f425c01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=625c0fb98aaa2542 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:10.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f426affd640 1 -- 172.21.15.148:0/711387245 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264001a00 con 0x7f426c1538a0 2023-10-30T13:48:10.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f426affd640 1 -- 172.21.15.148:0/711387245 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c0a7050 con 0x7f426c1538a0 2023-10-30T13:48:10.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f426affd640 1 -- 172.21.15.148:0/711387245 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c0ab620 con 0x7f426c1538a0 2023-10-30T13:48:10.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f4272409640 1 -- 172.21.15.148:0/711387245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1538a0 msgr2=0x7f426c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:10.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f4272409640 1 --2- 172.21.15.148:0/711387245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1538a0 0x7f426c15d6f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f425c0097f0 tx=0x7f425c01c320 comp rx=0 tx=0).stop 2023-10-30T13:48:10.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f4272409640 1 -- 172.21.15.148:0/711387245 shutdown_connections 2023-10-30T13:48:10.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f4272409640 1 --2- 172.21.15.148:0/711387245 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c1538a0 0x7f426c15d6f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:10.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f4272409640 1 -- 172.21.15.148:0/711387245 >> 172.21.15.148:0/711387245 conn(0x7f426c06bfe0 msgr2=0x7f426c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:10.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f4272409640 1 -- 172.21.15.148:0/711387245 shutdown_connections 2023-10-30T13:48:10.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.538+0000 7f4272409640 1 -- 172.21.15.148:0/711387245 wait complete. 2023-10-30T13:48:10.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 Processor -- start 2023-10-30T13:48:10.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 -- start start 2023-10-30T13:48:10.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426c0ca0e0 0x7f426c1e37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:10.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c0cf160 0x7f426c1e2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:10.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c1538a0 0x7f426c1ef870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:10.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c15b0e0 con 0x7f426c1538a0 2023-10-30T13:48:10.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c1e3cf0 con 0x7f426c0ca0e0 2023-10-30T13:48:10.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c1588c0 con 0x7f426c0cf160 2023-10-30T13:48:10.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f42709a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c1538a0 0x7f426c1ef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:10.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f42709a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c1538a0 0x7f426c1ef870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:36234/0 (socket says 172.21.15.148:36234) 2023-10-30T13:48:10.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f42709a6640 1 -- 172.21.15.148:0/422723726 learned_addr learned my addr 172.21.15.148:0/422723726 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:10.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f426bfff640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426c0ca0e0 0x7f426c1e37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:10.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f426b7fe640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c0cf160 0x7f426c1e2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:10.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f42709a6640 1 -- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c0cf160 msgr2=0x7f426c1e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:10.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f42709a6640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c0cf160 0x7f426c1e2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:10.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f42709a6640 1 -- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426c0ca0e0 msgr2=0x7f426c1e37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:10.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f42709a6640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426c0ca0e0 0x7f426c1e37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:10.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f42709a6640 1 -- 172.21.15.148:0/422723726 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c009420 con 0x7f426c1538a0 2023-10-30T13:48:10.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f42709a6640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c1538a0 0x7f426c1ef870 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f426400c8a0 tx=0x7f426400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:10.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4268ff9640 1 -- 172.21.15.148:0/422723726 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426400a5e0 con 0x7f426c1538a0 2023-10-30T13:48:10.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4268ff9640 1 -- 172.21.15.148:0/422723726 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426400a760 con 0x7f426c1538a0 2023-10-30T13:48:10.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426c159690 con 0x7f426c1538a0 2023-10-30T13:48:10.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4268ff9640 1 -- 172.21.15.148:0/422723726 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426400ab10 con 0x7f426c1538a0 2023-10-30T13:48:10.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c066580 con 0x7f426c1538a0 2023-10-30T13:48:10.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.542+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c04e170 con 0x7f426c1538a0 2023-10-30T13:48:10.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.546+0000 7f4268ff9640 1 -- 172.21.15.148:0/422723726 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f426400ac70 con 0x7f426c1538a0 2023-10-30T13:48:10.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.546+0000 7f4268ff9640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42580664d0 0x7f4258068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:10.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.546+0000 7f4268ff9640 1 -- 172.21.15.148:0/422723726 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7f4264083de0 con 0x7f426c1538a0 2023-10-30T13:48:10.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.546+0000 7f426bfff640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42580664d0 0x7f4258068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:10.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.546+0000 7f426bfff640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42580664d0 0x7f4258068990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f425c0ad000 tx=0x7f425c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:10.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.550+0000 7f4268ff9640 1 -- 172.21.15.148:0/422723726 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4264050be0 con 0x7f426c1538a0 2023-10-30T13:48:10.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f426c0b1dd0 con 0x7f426c1538a0 2023-10-30T13:48:10.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4268ff9640 1 -- 172.21.15.148:0/422723726 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6532 (secure 0 0 0) 0x7f4264018070 con 0x7f426c1538a0 2023-10-30T13:48:10.732 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:10.732 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:47:53.222242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:10.733 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 143 2023-10-30T13:48:10.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42580664d0 msgr2=0x7f4258068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:10.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42580664d0 0x7f4258068990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f425c0ad000 tx=0x7f425c0a6000 comp rx=0 tx=0).stop 2023-10-30T13:48:10.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c1538a0 msgr2=0x7f426c1ef870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:10.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c1538a0 0x7f426c1ef870 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f426400c8a0 tx=0x7f426400cd60 comp rx=0 tx=0).stop 2023-10-30T13:48:10.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 shutdown_connections 2023-10-30T13:48:10.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42580664d0 0x7f4258068990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:10.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f426c1538a0 0x7f426c1ef870 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:10.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f426c0cf160 0x7f426c1e2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:10.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 --2- 172.21.15.148:0/422723726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f426c0ca0e0 0x7f426c1e37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:10.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 >> 172.21.15.148:0/422723726 conn(0x7f426c06bfe0 msgr2=0x7f426c14db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:10.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 shutdown_connections 2023-10-30T13:48:10.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:10.730+0000 7f4272409640 1 -- 172.21.15.148:0/422723726 wait complete. 2023-10-30T13:48:10.754 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:48:12.756 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:12.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb80eb65640 1 Processor -- start 2023-10-30T13:48:12.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb80eb65640 1 -- start start 2023-10-30T13:48:12.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb80eb65640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb80806f7a0 0x7fb80806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:12.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb80eb65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb808071110 0x7fb808158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:12.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb80eb65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb808066cf0 con 0x7fb8080c6bd0 2023-10-30T13:48:12.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb80eb65640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8080700c0 con 0x7fb80806f7a0 2023-10-30T13:48:12.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb80eb65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8080c70d0 con 0x7fb808071110 2023-10-30T13:48:12.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb80c901640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb80806f7a0 0x7fb80806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:12.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb7fffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb8080c6bd0 0x7fb808069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53372/0 (socket says 172.21.15.148:53372) 2023-10-30T13:48:12.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb7fffff640 1 -- 172.21.15.148:0/2900653039 learned_addr learned my addr 172.21.15.148:0/2900653039 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:12.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.902+0000 7fb80d102640 1 --2- 172.21.15.148:0/2900653039 >> v2:172.21.15.150:3300/0 conn(0x7fb808071110 0x7fb808158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:12.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80c901640 1 -- 172.21.15.148:0/2900653039 >> v2:172.21.15.150:3300/0 conn(0x7fb808071110 msgr2=0x7fb808158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:12.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80c901640 1 --2- 172.21.15.148:0/2900653039 >> v2:172.21.15.150:3300/0 conn(0x7fb808071110 0x7fb808158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:12.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80c901640 1 -- 172.21.15.148:0/2900653039 >> v1:172.21.15.148:6789/0 conn(0x7fb8080c6bd0 legacy=0x7fb808069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:12.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80c901640 1 -- 172.21.15.148:0/2900653039 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080c7390 con 0x7fb80806f7a0 2023-10-30T13:48:12.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb7ff7fe640 1 -- 172.21.15.148:0/2900653039 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345965906 0 0) 0x7fb808066cf0 con 0x7fb8080c6bd0 2023-10-30T13:48:12.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80c901640 1 --2- 172.21.15.148:0/2900653039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb80806f7a0 0x7fb80806fb80 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fb80806cdd0 tx=0x7fb80401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c50265432cfded server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:12.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb7ff7fe640 1 -- 172.21.15.148:0/2900653039 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8040a8950 con 0x7fb80806f7a0 2023-10-30T13:48:12.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb7ff7fe640 1 -- 172.21.15.148:0/2900653039 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8040a8ab0 con 0x7fb80806f7a0 2023-10-30T13:48:12.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb7ff7fe640 1 -- 172.21.15.148:0/2900653039 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8040a8e40 con 0x7fb80806f7a0 2023-10-30T13:48:12.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80eb65640 1 -- 172.21.15.148:0/2900653039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb80806f7a0 msgr2=0x7fb80806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:12.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80eb65640 1 --2- 172.21.15.148:0/2900653039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb80806f7a0 0x7fb80806fb80 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fb80806cdd0 tx=0x7fb80401ec50 comp rx=0 tx=0).stop 2023-10-30T13:48:12.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80eb65640 1 -- 172.21.15.148:0/2900653039 shutdown_connections 2023-10-30T13:48:12.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80eb65640 1 --2- 172.21.15.148:0/2900653039 >> v2:172.21.15.150:3300/0 conn(0x7fb808071110 0x7fb808158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:12.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80eb65640 1 --2- 172.21.15.148:0/2900653039 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb80806f7a0 0x7fb80806fb80 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:12.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80eb65640 1 -- 172.21.15.148:0/2900653039 >> 172.21.15.148:0/2900653039 conn(0x7fb8080c7910 msgr2=0x7fb8080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:12.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80eb65640 1 -- 172.21.15.148:0/2900653039 shutdown_connections 2023-10-30T13:48:12.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.906+0000 7fb80eb65640 1 -- 172.21.15.148:0/2900653039 wait complete. 2023-10-30T13:48:12.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80eb65640 1 Processor -- start 2023-10-30T13:48:12.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80eb65640 1 -- start start 2023-10-30T13:48:12.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80eb65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb80806f7a0 0x7fb8081f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:12.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80eb65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808071110 0x7fb8080c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:12.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80eb65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8080c6bd0 0x7fb808157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:12.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80eb65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8081ed450 con 0x7fb8080c6bd0 2023-10-30T13:48:12.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80eb65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8081f08e0 con 0x7fb808071110 2023-10-30T13:48:12.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80eb65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8081ec560 con 0x7fb80806f7a0 2023-10-30T13:48:12.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80d102640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8080c6bd0 0x7fb808157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:12.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80d102640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8080c6bd0 0x7fb808157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51490/0 (socket says 172.21.15.148:51490) 2023-10-30T13:48:12.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80d102640 1 -- 172.21.15.148:0/1140577473 learned_addr learned my addr 172.21.15.148:0/1140577473 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:12.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb7fffff640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808071110 0x7fb8080c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:12.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80c901640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb80806f7a0 0x7fb8081f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:12.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80d102640 1 -- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb80806f7a0 msgr2=0x7fb8081f03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:12.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80d102640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb80806f7a0 0x7fb8081f03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:12.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80d102640 1 -- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808071110 msgr2=0x7fb8080c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:12.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80d102640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808071110 0x7fb8080c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:12.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80d102640 1 -- 172.21.15.148:0/1140577473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb804009420 con 0x7fb8080c6bd0 2023-10-30T13:48:12.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb80d102640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8080c6bd0 0x7fb808157920 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fb800007e60 tx=0x7fb80000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:12.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.910+0000 7fb7fd7fa640 1 -- 172.21.15.148:0/1140577473 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800014420 con 0x7fb8080c6bd0 2023-10-30T13:48:12.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.914+0000 7fb7fd7fa640 1 -- 172.21.15.148:0/1140577473 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8000145a0 con 0x7fb8080c6bd0 2023-10-30T13:48:12.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.914+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8081ec840 con 0x7fb8080c6bd0 2023-10-30T13:48:12.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.914+0000 7fb7fd7fa640 1 -- 172.21.15.148:0/1140577473 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800014950 con 0x7fb8080c6bd0 2023-10-30T13:48:12.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.914+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8081ece00 con 0x7fb8080c6bd0 2023-10-30T13:48:12.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.914+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80804e170 con 0x7fb8080c6bd0 2023-10-30T13:48:12.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.914+0000 7fb7fd7fa640 1 -- 172.21.15.148:0/1140577473 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb800014ab0 con 0x7fb8080c6bd0 2023-10-30T13:48:12.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.914+0000 7fb7fd7fa640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f40664d0 0x7fb7f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:12.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.914+0000 7fb7fd7fa640 1 -- 172.21.15.148:0/1140577473 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7402+0+0 (secure 0 0 0) 0x7fb800084ca0 con 0x7fb8080c6bd0 2023-10-30T13:48:12.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.914+0000 7fb80c901640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f40664d0 0x7fb7f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:12.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.918+0000 7fb80c901640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f40664d0 0x7fb7f4068990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb8040b2000 tx=0x7fb804008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:12.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:12.918+0000 7fb7fd7fa640 1 -- 172.21.15.148:0/1140577473 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb800051b30 con 0x7fb8080c6bd0 2023-10-30T13:48:13.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.098+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb80814cfd0 con 0x7fb8080c6bd0 2023-10-30T13:48:13.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb7fd7fa640 1 -- 172.21.15.148:0/1140577473 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+6532 (secure 0 0 0) 0x7fb800055150 con 0x7fb8080c6bd0 2023-10-30T13:48:13.104 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:13.104 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7812,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:47:53.222242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5453,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5453":{"gid":5453,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":355,"addr":"172.21.15.150:6836/116502200","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":116502200},{"type":"v1","addr":"172.21.15.150:6836","nonce":116502200}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:13.105 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 143 2023-10-30T13:48:13.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f40664d0 msgr2=0x7fb7f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:13.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f40664d0 0x7fb7f4068990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb8040b2000 tx=0x7fb804008040 comp rx=0 tx=0).stop 2023-10-30T13:48:13.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8080c6bd0 msgr2=0x7fb808157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:13.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8080c6bd0 0x7fb808157920 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fb800007e60 tx=0x7fb80000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:48:13.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 shutdown_connections 2023-10-30T13:48:13.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb7f40664d0 0x7fb7f4068990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:13.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8080c6bd0 0x7fb808157920 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:13.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb808071110 0x7fb8080c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:13.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 --2- 172.21.15.148:0/1140577473 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb80806f7a0 0x7fb8081f03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:13.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 >> 172.21.15.148:0/1140577473 conn(0x7fb8080c7910 msgr2=0x7fb8080c8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:13.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 shutdown_connections 2023-10-30T13:48:13.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:13.102+0000 7fb80eb65640 1 -- 172.21.15.148:0/1140577473 wait complete. 2023-10-30T13:48:13.125 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:48:15.126 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:15.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.270+0000 7f1b5e53e640 1 Processor -- start 2023-10-30T13:48:15.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.270+0000 7f1b5e53e640 1 -- start start 2023-10-30T13:48:15.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.270+0000 7f1b5e53e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b580c05d0 0x7f1b5813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.270+0000 7f1b5e53e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b580c6f60 0x7f1b5813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.270+0000 7f1b5e53e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b580c0a50 con 0x7f1b580c6b80 2023-10-30T13:48:15.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.270+0000 7f1b5e53e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b5806f6e0 con 0x7f1b580c6f60 2023-10-30T13:48:15.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.270+0000 7f1b5e53e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5806f9c0 con 0x7f1b580c05d0 2023-10-30T13:48:15.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b57fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f1b580c6b80 0x7f1b580bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53374/0 (socket says 172.21.15.148:53374) 2023-10-30T13:48:15.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b57fff640 1 -- 172.21.15.148:0/3257830213 learned_addr learned my addr 172.21.15.148:0/3257830213 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:15.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b577fe640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b580c05d0 0x7f1b5813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b5cadb640 1 --2- 172.21.15.148:0/3257830213 >> v2:172.21.15.150:3300/0 conn(0x7f1b580c6f60 0x7f1b5813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071095803 0 0) 0x7f1b580c0a50 con 0x7f1b580c6b80 2023-10-30T13:48:15.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b40005740 con 0x7f1b580c6b80 2023-10-30T13:48:15.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3931229474 0 0) 0x7f1b40005740 con 0x7f1b580c6b80 2023-10-30T13:48:15.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b580c05d0 msgr2=0x7f1b5813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:15.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b580c05d0 0x7f1b5813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.279 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 >> v2:172.21.15.150:3300/0 conn(0x7f1b580c6f60 msgr2=0x7f1b5813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:15.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 --2- 172.21.15.148:0/3257830213 >> v2:172.21.15.150:3300/0 conn(0x7f1b580c6f60 0x7f1b5813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5806fca0 con 0x7f1b580c6b80 2023-10-30T13:48:15.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f1b48001680 con 0x7f1b580c6b80 2023-10-30T13:48:15.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 >> v1:172.21.15.148:6789/0 conn(0x7f1b580c6b80 legacy=0x7f1b580bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b40005b90 0x7f1b40005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b400069e0 0x7f1b40006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b580c05d0 0x7f1b400073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5806f6e0 con 0x7f1b400069e0 2023-10-30T13:48:15.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5806f9c0 con 0x7f1b40005b90 2023-10-30T13:48:15.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4000dba0 con 0x7f1b580c05d0 2023-10-30T13:48:15.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b577fe640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b400069e0 0x7f1b40006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b57fff640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b40005b90 0x7f1b40005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b5cadb640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b580c05d0 0x7f1b400073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b577fe640 1 -- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b580c05d0 msgr2=0x7f1b400073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:15.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b577fe640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b580c05d0 0x7f1b400073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b577fe640 1 -- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b40005b90 msgr2=0x7f1b40005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:15.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b577fe640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b40005b90 0x7f1b40005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.274+0000 7f1b577fe640 1 -- 172.21.15.148:0/3257830213 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b40005740 con 0x7f1b400069e0 2023-10-30T13:48:15.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b577fe640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b400069e0 0x7f1b40006e60 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009ae0 tx=0x7f1b4c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5673c04938dc9106 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:15.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0a5560 con 0x7f1b400069e0 2023-10-30T13:48:15.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4c0a56c0 con 0x7f1b400069e0 2023-10-30T13:48:15.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b56ffd640 1 -- 172.21.15.148:0/3257830213 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0a5a70 con 0x7f1b400069e0 2023-10-30T13:48:15.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3257830213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b400069e0 msgr2=0x7f1b40006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b5e53e640 1 --2- 172.21.15.148:0/3257830213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b400069e0 0x7f1b40006e60 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009ae0 tx=0x7f1b4c01ca50 comp rx=0 tx=0).stop 2023-10-30T13:48:15.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b5cadb640 1 -- 172.21.15.148:0/3257830213 reap_dead start 2023-10-30T13:48:15.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3257830213 shutdown_connections 2023-10-30T13:48:15.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3257830213 >> 172.21.15.148:0/3257830213 conn(0x7f1b580c7910 msgr2=0x7f1b580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:15.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3257830213 shutdown_connections 2023-10-30T13:48:15.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3257830213 wait complete. 2023-10-30T13:48:15.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b5e53e640 1 Processor -- start 2023-10-30T13:48:15.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.278+0000 7f1b5e53e640 1 -- start start 2023-10-30T13:48:15.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5e53e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b400069e0 0x7f1b58135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5e53e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b581dd070 0x7f1b580c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5e53e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b581dda20 0x7f1b581de880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5e53e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58136340 con 0x7f1b581dda20 2023-10-30T13:48:15.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5e53e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580ad8d0 con 0x7f1b581dd070 2023-10-30T13:48:15.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5e53e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5813d770 con 0x7f1b400069e0 2023-10-30T13:48:15.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5cadb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b581dda20 0x7f1b581de880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5cadb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b581dda20 0x7f1b581de880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51510/0 (socket says 172.21.15.148:51510) 2023-10-30T13:48:15.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5cadb640 1 -- 172.21.15.148:0/3090057567 learned_addr learned my addr 172.21.15.148:0/3090057567 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:15.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b577fe640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b581dd070 0x7f1b580c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b57fff640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b400069e0 0x7f1b58135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5cadb640 1 -- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b400069e0 msgr2=0x7f1b58135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5cadb640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b400069e0 0x7f1b58135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5cadb640 1 -- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b581dd070 msgr2=0x7f1b580c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5cadb640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b581dd070 0x7f1b580c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5cadb640 1 -- 172.21.15.148:0/3090057567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5813c740 con 0x7f1b581dda20 2023-10-30T13:48:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5cadb640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b581dda20 0x7f1b581de880 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f1b580b1190 tx=0x7f1b5000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:15.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b54ff9640 1 -- 172.21.15.148:0/3090057567 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5000e040 con 0x7f1b581dda20 2023-10-30T13:48:15.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b54ff9640 1 -- 172.21.15.148:0/3090057567 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5000f040 con 0x7f1b581dda20 2023-10-30T13:48:15.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b54ff9640 1 -- 172.21.15.148:0/3090057567 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50015430 con 0x7f1b581dda20 2023-10-30T13:48:15.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5813c9b0 con 0x7f1b581dda20 2023-10-30T13:48:15.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5813cf70 con 0x7f1b581dda20 2023-10-30T13:48:15.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5804e170 con 0x7f1b581dda20 2023-10-30T13:48:15.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.282+0000 7f1b54ff9640 1 -- 172.21.15.148:0/3090057567 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1b50015590 con 0x7f1b581dda20 2023-10-30T13:48:15.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.286+0000 7f1b54ff9640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b440664a0 0x7f1b44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.286+0000 7f1b54ff9640 1 -- 172.21.15.148:0/3090057567 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f1b50084db0 con 0x7f1b581dda20 2023-10-30T13:48:15.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.286+0000 7f1b57fff640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b440664a0 0x7f1b44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.286+0000 7f1b57fff640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b440664a0 0x7f1b44068960 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1b480050c0 tx=0x7f1b48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:15.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.290+0000 7f1b54ff9640 1 -- 172.21.15.148:0/3090057567 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1b50089070 con 0x7f1b581dda20 2023-10-30T13:48:15.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b580c1670 con 0x7f1b581dda20 2023-10-30T13:48:15.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b54ff9640 1 -- 172.21.15.148:0/3090057567 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f1b50051ac0 con 0x7f1b581dda20 2023-10-30T13:48:15.473 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:15.473 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:15.005721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:15.473 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 144 2023-10-30T13:48:15.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b440664a0 msgr2=0x7f1b44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b440664a0 0x7f1b44068960 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1b480050c0 tx=0x7f1b48009040 comp rx=0 tx=0).stop 2023-10-30T13:48:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b581dda20 msgr2=0x7f1b581de880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b581dda20 0x7f1b581de880 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f1b580b1190 tx=0x7f1b5000f580 comp rx=0 tx=0).stop 2023-10-30T13:48:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 shutdown_connections 2023-10-30T13:48:15.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b440664a0 0x7f1b44068960 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b581dda20 0x7f1b581de880 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b581dd070 0x7f1b580c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 --2- 172.21.15.148:0/3090057567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b400069e0 0x7f1b58135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 >> 172.21.15.148:0/3090057567 conn(0x7f1b580c7910 msgr2=0x7f1b58136930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 shutdown_connections 2023-10-30T13:48:15.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.470+0000 7f1b5e53e640 1 -- 172.21.15.148:0/3090057567 wait complete. 2023-10-30T13:48:15.493 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:48:15.493 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:48:15.494 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:15.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4f5c5640 1 Processor -- start 2023-10-30T13:48:15.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4f5c5640 1 -- start start 2023-10-30T13:48:15.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4f5c5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcb481538a0 0x7fcb4815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4f5c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480ca0e0 0x7fcb4815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4f5c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480cb800 con 0x7fcb480cf160 2023-10-30T13:48:15.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4f5c5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4815dc30 con 0x7fcb481538a0 2023-10-30T13:48:15.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4f5c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480cb330 con 0x7fcb480ca0e0 2023-10-30T13:48:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4d361640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcb481538a0 0x7fcb4815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4d361640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fcb481538a0 0x7fcb4815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51520/0 (socket says 172.21.15.148:51520) 2023-10-30T13:48:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4d361640 1 -- 172.21.15.148:0/3400862244 learned_addr learned my addr 172.21.15.148:0/3400862244 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4cb60640 1 --2- 172.21.15.148:0/3400862244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480ca0e0 0x7fcb4815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4d361640 1 -- 172.21.15.148:0/3400862244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480ca0e0 msgr2=0x7fcb4815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:15.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4d361640 1 --2- 172.21.15.148:0/3400862244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480ca0e0 0x7fcb4815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4d361640 1 -- 172.21.15.148:0/3400862244 >> v1:172.21.15.150:6789/0 conn(0x7fcb480cf160 legacy=0x7fcb480cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.630+0000 7fcb4d361640 1 -- 172.21.15.148:0/3400862244 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb480cad70 con 0x7fcb481538a0 2023-10-30T13:48:15.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb4d361640 1 --2- 172.21.15.148:0/3400862244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb481538a0 0x7fcb4815d6f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009870 tx=0x7fcb4401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be53ae73d67899f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:15.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb3ffff640 1 -- 172.21.15.148:0/3400862244 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb440a8a30 con 0x7fcb481538a0 2023-10-30T13:48:15.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb3ffff640 1 -- 172.21.15.148:0/3400862244 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb440a8b90 con 0x7fcb481538a0 2023-10-30T13:48:15.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb3ffff640 1 -- 172.21.15.148:0/3400862244 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb440b0440 con 0x7fcb481538a0 2023-10-30T13:48:15.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/3400862244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb481538a0 msgr2=0x7fcb4815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb4f5c5640 1 --2- 172.21.15.148:0/3400862244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb481538a0 0x7fcb4815d6f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009870 tx=0x7fcb4401ed30 comp rx=0 tx=0).stop 2023-10-30T13:48:15.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/3400862244 shutdown_connections 2023-10-30T13:48:15.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb4f5c5640 1 --2- 172.21.15.148:0/3400862244 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480ca0e0 0x7fcb4815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb4f5c5640 1 --2- 172.21.15.148:0/3400862244 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb481538a0 0x7fcb4815d6f0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/3400862244 >> 172.21.15.148:0/3400862244 conn(0x7fcb4806bfe0 msgr2=0x7fcb4806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:15.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/3400862244 shutdown_connections 2023-10-30T13:48:15.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.634+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/3400862244 wait complete. 2023-10-30T13:48:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 Processor -- start 2023-10-30T13:48:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 -- start start 2023-10-30T13:48:15.642 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb480ca0e0 0x7fcb4815b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480cf160 0x7fcb481e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb481538a0 0x7fcb481e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb481e5580 con 0x7fcb480ca0e0 2023-10-30T13:48:15.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4815bc60 con 0x7fcb481538a0 2023-10-30T13:48:15.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb481e4670 con 0x7fcb480cf160 2023-10-30T13:48:15.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4d361640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb480ca0e0 0x7fcb4815b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4d361640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb480ca0e0 0x7fcb4815b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51526/0 (socket says 172.21.15.148:51526) 2023-10-30T13:48:15.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4d361640 1 -- 172.21.15.148:0/610881519 learned_addr learned my addr 172.21.15.148:0/610881519 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:15.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4cb60640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480cf160 0x7fcb481e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4db62640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb481538a0 0x7fcb481e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4d361640 1 -- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480cf160 msgr2=0x7fcb481e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4d361640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480cf160 0x7fcb481e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4d361640 1 -- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb481538a0 msgr2=0x7fcb481e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4d361640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb481538a0 0x7fcb481e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4d361640 1 -- 172.21.15.148:0/610881519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb440094d0 con 0x7fcb480ca0e0 2023-10-30T13:48:15.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4db62640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb481538a0 0x7fcb481e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:15.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4d361640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb480ca0e0 0x7fcb4815b040 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fcb4401d3e0 tx=0x7fcb4401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:15.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb3dffb640 1 -- 172.21.15.148:0/610881519 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb440b0dd0 con 0x7fcb480ca0e0 2023-10-30T13:48:15.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb3dffb640 1 -- 172.21.15.148:0/610881519 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb440bb460 con 0x7fcb480ca0e0 2023-10-30T13:48:15.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb3dffb640 1 -- 172.21.15.148:0/610881519 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb440bb810 con 0x7fcb480ca0e0 2023-10-30T13:48:15.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb48158840 con 0x7fcb480ca0e0 2023-10-30T13:48:15.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.638+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb48158ae0 con 0x7fcb480ca0e0 2023-10-30T13:48:15.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.642+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb480b2a50 con 0x7fcb480ca0e0 2023-10-30T13:48:15.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.642+0000 7fcb3dffb640 1 -- 172.21.15.148:0/610881519 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb440b83f0 con 0x7fcb480ca0e0 2023-10-30T13:48:15.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.642+0000 7fcb3dffb640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcb2c0664a0 0x7fcb2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:15.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.642+0000 7fcb3dffb640 1 -- 172.21.15.148:0/610881519 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fcb44121d40 con 0x7fcb480ca0e0 2023-10-30T13:48:15.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.642+0000 7fcb4cb60640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcb2c0664a0 0x7fcb2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:15.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.646+0000 7fcb4cb60640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcb2c0664a0 0x7fcb2c068960 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcb481e58f0 tx=0x7fcb38009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:15.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.646+0000 7fcb3dffb640 1 -- 172.21.15.148:0/610881519 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb440eeaf0 con 0x7fcb480ca0e0 2023-10-30T13:48:15.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.826+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb4804e170 con 0x7fcb480ca0e0 2023-10-30T13:48:15.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.826+0000 7fcb3dffb640 1 -- 172.21.15.148:0/610881519 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fcb440f2110 con 0x7fcb480ca0e0 2023-10-30T13:48:15.830 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:15.831 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:15.005721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:15.831 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 144 2023-10-30T13:48:15.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcb2c0664a0 msgr2=0x7fcb2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcb2c0664a0 0x7fcb2c068960 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcb481e58f0 tx=0x7fcb38009190 comp rx=0 tx=0).stop 2023-10-30T13:48:15.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb480ca0e0 msgr2=0x7fcb4815b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:15.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb480ca0e0 0x7fcb4815b040 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fcb4401d3e0 tx=0x7fcb4401bad0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 shutdown_connections 2023-10-30T13:48:15.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fcb2c0664a0 0x7fcb2c068960 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb481538a0 0x7fcb481e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fcb480cf160 0x7fcb481e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 --2- 172.21.15.148:0/610881519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb480ca0e0 0x7fcb4815b040 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:15.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 >> 172.21.15.148:0/610881519 conn(0x7fcb4806bfe0 msgr2=0x7fcb48065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:15.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 shutdown_connections 2023-10-30T13:48:15.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:15.830+0000 7fcb4f5c5640 1 -- 172.21.15.148:0/610881519 wait complete. 2023-10-30T13:48:15.852 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 144, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7611": { "addr": "172.21.15.148:6835/1228827025", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1228827025, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1228827025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7611, "incarnation": 133, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 10 }, "gid_7812": { "addr": "172.21.15.148:6837/1530445724", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 1530445724, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 1530445724, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7812, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:48:15.005721+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7812, "mds_1": 7611 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:48:17.853 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:18.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:17.998+0000 7f976f809640 1 Processor -- start 2023-10-30T13:48:18.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:17.998+0000 7f976f809640 1 -- start start 2023-10-30T13:48:18.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:17.998+0000 7f976f809640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97680c6b80 0x7f97680bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:18.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:17.998+0000 7f976f809640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 0x7f97680c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:18.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:17.998+0000 7f976f809640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97680c0a50 con 0x7f97680c8430 2023-10-30T13:48:18.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:17.998+0000 7f976f809640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f976806f6e0 con 0x7f97680c6b80 2023-10-30T13:48:18.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:17.998+0000 7f976f809640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976806f9c0 con 0x7f97680c6f60 2023-10-30T13:48:18.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:17.998+0000 7f976cda4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 0x7f97680c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:18.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:17.998+0000 7f976d5a5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97680c6b80 0x7f97680bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:18.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f976cda4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 0x7f97680c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:48242/0 (socket says 172.21.15.148:48242) 2023-10-30T13:48:18.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f976cda4640 1 -- 172.21.15.148:0/2604925305 learned_addr learned my addr 172.21.15.148:0/2604925305 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:18.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f976dda6640 1 --1- 172.21.15.148:0/2604925305 >> v1:172.21.15.150:6789/0 conn(0x7f97680c8430 0x7f976813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:45386/0 (socket says 172.21.15.148:45386) 2023-10-30T13:48:18.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f976d5a5640 1 -- 172.21.15.148:0/2604925305 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 msgr2=0x7f97680c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:18.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f976d5a5640 1 --2- 172.21.15.148:0/2604925305 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 0x7f97680c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f976d5a5640 1 -- 172.21.15.148:0/2604925305 >> v1:172.21.15.150:6789/0 conn(0x7f97680c8430 legacy=0x7f976813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:18.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f976d5a5640 1 -- 172.21.15.148:0/2604925305 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976806fca0 con 0x7f97680c6b80 2023-10-30T13:48:18.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f976d5a5640 1 --2- 172.21.15.148:0/2604925305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 0x7f97680bfec0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f97640097c0 tx=0x7f976401c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28ff9f4728660ce0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:18.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f975ffff640 1 -- 172.21.15.148:0/2604925305 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97640ab600 con 0x7f97680c6b80 2023-10-30T13:48:18.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f975ffff640 1 -- 172.21.15.148:0/2604925305 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97640a7050 con 0x7f97680c6b80 2023-10-30T13:48:18.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.002+0000 7f975ffff640 1 -- 172.21.15.148:0/2604925305 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97640ab980 con 0x7f97680c6b80 2023-10-30T13:48:18.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 -- 172.21.15.148:0/2604925305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 msgr2=0x7f97680bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:18.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 --2- 172.21.15.148:0/2604925305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 0x7f97680bfec0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f97640097c0 tx=0x7f976401c3f0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 -- 172.21.15.148:0/2604925305 shutdown_connections 2023-10-30T13:48:18.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 --2- 172.21.15.148:0/2604925305 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 0x7f97680c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 --2- 172.21.15.148:0/2604925305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 0x7f97680bfec0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 -- 172.21.15.148:0/2604925305 >> 172.21.15.148:0/2604925305 conn(0x7f97680c7910 msgr2=0x7f97680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:18.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 -- 172.21.15.148:0/2604925305 shutdown_connections 2023-10-30T13:48:18.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 -- 172.21.15.148:0/2604925305 wait complete. 2023-10-30T13:48:18.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 Processor -- start 2023-10-30T13:48:18.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 -- start start 2023-10-30T13:48:18.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 0x7f9768155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:18.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976d5a5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 0x7f9768155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:18.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.006+0000 7f976f809640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 0x7f97681394a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976f809640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97680c8430 0x7f97681f5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976d5a5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 0x7f9768155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:38734/0 (socket says 172.21.15.148:38734) 2023-10-30T13:48:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976d5a5640 1 -- 172.21.15.148:0/143584324 learned_addr learned my addr 172.21.15.148:0/143584324 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681563d0 con 0x7f97680c8430 2023-10-30T13:48:18.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681582d0 con 0x7f97680c6b80 2023-10-30T13:48:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9768159300 con 0x7f97680c6f60 2023-10-30T13:48:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976dda6640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97680c8430 0x7f97681f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976cda4640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 0x7f97681394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976d5a5640 1 -- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 msgr2=0x7f97681394a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976d5a5640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 0x7f97681394a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976d5a5640 1 -- 172.21.15.148:0/143584324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97680c8430 msgr2=0x7f97681f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:18.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976d5a5640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97680c8430 0x7f97681f5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976d5a5640 1 -- 172.21.15.148:0/143584324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9764009420 con 0x7f97680c6b80 2023-10-30T13:48:18.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976d5a5640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 0x7f9768155940 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f976401a020 tx=0x7f976401ac30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:18.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f975dffb640 1 -- 172.21.15.148:0/143584324 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97640b7650 con 0x7f97680c6b80 2023-10-30T13:48:18.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f975dffb640 1 -- 172.21.15.148:0/143584324 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97640b77d0 con 0x7f97680c6b80 2023-10-30T13:48:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9768155e80 con 0x7f97680c6b80 2023-10-30T13:48:18.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f975dffb640 1 -- 172.21.15.148:0/143584324 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97640b7b80 con 0x7f97680c6b80 2023-10-30T13:48:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9768157930 con 0x7f97680c6b80 2023-10-30T13:48:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.010+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976804e170 con 0x7f97680c6b80 2023-10-30T13:48:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.014+0000 7f975dffb640 1 -- 172.21.15.148:0/143584324 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97640c9430 con 0x7f97680c6b80 2023-10-30T13:48:18.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.014+0000 7f975dffb640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f974c0664a0 0x7f974c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:18.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.014+0000 7f975dffb640 1 -- 172.21.15.148:0/143584324 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f976411f570 con 0x7f97680c6b80 2023-10-30T13:48:18.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.014+0000 7f976cda4640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f974c0664a0 0x7f974c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:18.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.014+0000 7f976cda4640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f974c0664a0 0x7f974c068960 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9768160480 tx=0x7f9758008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:18.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.026+0000 7f975dffb640 1 -- 172.21.15.148:0/143584324 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97640ec420 con 0x7f97680c6b80 2023-10-30T13:48:18.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.206+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9768071b90 con 0x7f97680c6b80 2023-10-30T13:48:18.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.206+0000 7f975dffb640 1 -- 172.21.15.148:0/143584324 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f97640b0090 con 0x7f97680c6b80 2023-10-30T13:48:18.210 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:18.210 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:15.005721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:18.210 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 144 2023-10-30T13:48:18.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f974c0664a0 msgr2=0x7f974c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:18.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f974c0664a0 0x7f974c068960 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9768160480 tx=0x7f9758008040 comp rx=0 tx=0).stop 2023-10-30T13:48:18.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 msgr2=0x7f9768155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:18.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 0x7f9768155940 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f976401a020 tx=0x7f976401ac30 comp rx=0 tx=0).stop 2023-10-30T13:48:18.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 shutdown_connections 2023-10-30T13:48:18.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f974c0664a0 0x7f974c068960 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97680c8430 0x7f97681f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f97680c6f60 0x7f97681394a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 --2- 172.21.15.148:0/143584324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97680c6b80 0x7f9768155940 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:18.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 >> 172.21.15.148:0/143584324 conn(0x7f97680c7910 msgr2=0x7f9768135bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:18.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 shutdown_connections 2023-10-30T13:48:18.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:18.210+0000 7f976f809640 1 -- 172.21.15.148:0/143584324 wait complete. 2023-10-30T13:48:18.233 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:20.234 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:20.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.378+0000 7fdae1032640 1 Processor -- start 2023-10-30T13:48:20.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.378+0000 7fdae1032640 1 -- start start 2023-10-30T13:48:20.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdae1032640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6b80 0x7fdadc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:20.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdae1032640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdadc0c6f60 0x7fdadc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:20.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdae1032640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdadc0c0a50 con 0x7fdadc0c8430 2023-10-30T13:48:20.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdae1032640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdadc06f6e0 con 0x7fdadc0c6f60 2023-10-30T13:48:20.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdae1032640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc06f9c0 con 0x7fdadc0c6b80 2023-10-30T13:48:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdad3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdadc0c6f60 0x7fdadc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdadad9d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6b80 0x7fdadc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdad3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdadc0c6f60 0x7fdadc0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:38740/0 (socket says 172.21.15.148:38740) 2023-10-30T13:48:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdadb59e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fdadc0c8430 0x7fdadc13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:45388/0 (socket says 172.21.15.148:45388) 2023-10-30T13:48:20.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdad3fff640 1 -- 172.21.15.148:0/316150603 learned_addr learned my addr 172.21.15.148:0/316150603 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:20.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdad3fff640 1 -- 172.21.15.148:0/316150603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6b80 msgr2=0x7fdadc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:20.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdad3fff640 1 --2- 172.21.15.148:0/316150603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6b80 0x7fdadc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdad3fff640 1 -- 172.21.15.148:0/316150603 >> v1:172.21.15.150:6789/0 conn(0x7fdadc0c8430 legacy=0x7fdadc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:20.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.382+0000 7fdad3fff640 1 -- 172.21.15.148:0/316150603 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc06fca0 con 0x7fdadc0c6f60 2023-10-30T13:48:20.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdad3fff640 1 --2- 172.21.15.148:0/316150603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdadc0c6f60 0x7fdadc0c0400 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009870 tx=0x7fdac401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc9e5c2b0ef3b5be server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:20.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdada59c640 1 -- 172.21.15.148:0/316150603 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac40ab650 con 0x7fdadc0c6f60 2023-10-30T13:48:20.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdada59c640 1 -- 172.21.15.148:0/316150603 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac40ab7b0 con 0x7fdadc0c6f60 2023-10-30T13:48:20.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdada59c640 1 -- 172.21.15.148:0/316150603 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac40abb40 con 0x7fdadc0c6f60 2023-10-30T13:48:20.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdae1032640 1 -- 172.21.15.148:0/316150603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdadc0c6f60 msgr2=0x7fdadc0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdae1032640 1 --2- 172.21.15.148:0/316150603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdadc0c6f60 0x7fdadc0c0400 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009870 tx=0x7fdac401c540 comp rx=0 tx=0).stop 2023-10-30T13:48:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdae1032640 1 -- 172.21.15.148:0/316150603 shutdown_connections 2023-10-30T13:48:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdae1032640 1 --2- 172.21.15.148:0/316150603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdadc0c6f60 0x7fdadc0c0400 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdae1032640 1 --2- 172.21.15.148:0/316150603 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6b80 0x7fdadc0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdae1032640 1 -- 172.21.15.148:0/316150603 >> 172.21.15.148:0/316150603 conn(0x7fdadc0c7910 msgr2=0x7fdadc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:20.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdae1032640 1 -- 172.21.15.148:0/316150603 shutdown_connections 2023-10-30T13:48:20.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.386+0000 7fdae1032640 1 -- 172.21.15.148:0/316150603 wait complete. 2023-10-30T13:48:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 Processor -- start 2023-10-30T13:48:20.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 -- start start 2023-10-30T13:48:20.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdadc0c6b80 0x7fdadc1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:20.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6f60 0x7fdadc157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:20.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdadc0c8430 0x7fdadc1f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:20.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc12fcb0 con 0x7fdadc0c8430 2023-10-30T13:48:20.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc132340 con 0x7fdadc0c6b80 2023-10-30T13:48:20.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc158470 con 0x7fdadc0c6f60 2023-10-30T13:48:20.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadb59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdadc0c8430 0x7fdadc1f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:20.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadb59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdadc0c8430 0x7fdadc1f3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:51534/0 (socket says 172.21.15.148:51534) 2023-10-30T13:48:20.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadb59e640 1 -- 172.21.15.148:0/3620476650 learned_addr learned my addr 172.21.15.148:0/3620476650 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:20.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadad9d640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdadc0c6b80 0x7fdadc1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:20.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdad3fff640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6f60 0x7fdadc157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:20.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadb59e640 1 -- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6f60 msgr2=0x7fdadc157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:20.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadb59e640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6f60 0x7fdadc157f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadb59e640 1 -- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdadc0c6b80 msgr2=0x7fdadc1579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:20.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadb59e640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdadc0c6b80 0x7fdadc1579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadb59e640 1 -- 172.21.15.148:0/3620476650 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac40094d0 con 0x7fdadc0c8430 2023-10-30T13:48:20.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdadb59e640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdadc0c8430 0x7fdadc1f3290 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fdad400c700 tx=0x7fdad400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:20.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdad37fe640 1 -- 172.21.15.148:0/3620476650 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad400ce10 con 0x7fdadc0c8430 2023-10-30T13:48:20.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdad37fe640 1 -- 172.21.15.148:0/3620476650 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad400a640 con 0x7fdadc0c8430 2023-10-30T13:48:20.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdadc132060 con 0x7fdadc0c8430 2023-10-30T13:48:20.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdad37fe640 1 -- 172.21.15.148:0/3620476650 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad400a9b0 con 0x7fdadc0c8430 2023-10-30T13:48:20.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.390+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdadc1333c0 con 0x7fdadc0c8430 2023-10-30T13:48:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.394+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdadc04e170 con 0x7fdadc0c8430 2023-10-30T13:48:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.394+0000 7fdad37fe640 1 -- 172.21.15.148:0/3620476650 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdad400ab10 con 0x7fdadc0c8430 2023-10-30T13:48:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.394+0000 7fdad37fe640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdabc0664a0 0x7fdabc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:20.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.394+0000 7fdad37fe640 1 -- 172.21.15.148:0/3620476650 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fdad4083bf0 con 0x7fdadc0c8430 2023-10-30T13:48:20.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.394+0000 7fdadad9d640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdabc0664a0 0x7fdabc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:20.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.398+0000 7fdadad9d640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdabc0664a0 0x7fdabc068960 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009b60 tx=0x7fdacc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:20.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.398+0000 7fdad37fe640 1 -- 172.21.15.148:0/3620476650 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdad4050a40 con 0x7fdadc0c8430 2023-10-30T13:48:20.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.594+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdadc1573e0 con 0x7fdadc0c8430 2023-10-30T13:48:20.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.594+0000 7fdad37fe640 1 -- 172.21.15.148:0/3620476650 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fdad4017090 con 0x7fdadc0c8430 2023-10-30T13:48:20.597 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:20.598 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:15.005721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:20.598 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 144 2023-10-30T13:48:20.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.594+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdabc0664a0 msgr2=0x7fdabc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:20.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.594+0000 7fdae1032640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdabc0664a0 0x7fdabc068960 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009b60 tx=0x7fdacc009190 comp rx=0 tx=0).stop 2023-10-30T13:48:20.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.594+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdadc0c8430 msgr2=0x7fdadc1f3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:20.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.594+0000 7fdae1032640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdadc0c8430 0x7fdadc1f3290 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fdad400c700 tx=0x7fdad400cbc0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.598+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 shutdown_connections 2023-10-30T13:48:20.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.598+0000 7fdae1032640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdabc0664a0 0x7fdabc068960 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.598+0000 7fdae1032640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdadc0c8430 0x7fdadc1f3290 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.598+0000 7fdae1032640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdadc0c6f60 0x7fdadc157f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.598+0000 7fdae1032640 1 --2- 172.21.15.148:0/3620476650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdadc0c6b80 0x7fdadc1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.598+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 >> 172.21.15.148:0/3620476650 conn(0x7fdadc0c7910 msgr2=0x7fdadc13ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.598+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 shutdown_connections 2023-10-30T13:48:20.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:20.598+0000 7fdae1032640 1 -- 172.21.15.148:0/3620476650 wait complete. 2023-10-30T13:48:20.619 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:22.621 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:22.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.805+0000 7fc10dd0f640 1 Processor -- start 2023-10-30T13:48:22.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.805+0000 7fc10dd0f640 1 -- start start 2023-10-30T13:48:22.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc10dd0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc1081538a0 0x7fc10815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:22.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc10dd0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc1080ca0e0 0x7fc10815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:22.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc10dd0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080cf160 0x7fc1080cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:22.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc10dd0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1080cb800 con 0x7fc1080ca0e0 2023-10-30T13:48:22.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc10dd0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10815dc30 con 0x7fc1081538a0 2023-10-30T13:48:22.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc10dd0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080cb330 con 0x7fc1080cf160 2023-10-30T13:48:22.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc106ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc1080ca0e0 0x7fc10815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:22.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc106ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc1080ca0e0 0x7fc10815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50040/0 (socket says 172.21.15.148:50040) 2023-10-30T13:48:22.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc106ffd640 1 -- 172.21.15.148:0/970906538 learned_addr learned my addr 172.21.15.148:0/970906538 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:22.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc107fff640 1 --2- 172.21.15.148:0/970906538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080cf160 0x7fc1080cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:22.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc1077fe640 1 --2- 172.21.15.148:0/970906538 >> v2:172.21.15.150:3300/0 conn(0x7fc1081538a0 0x7fc10815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:22.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc106ffd640 1 -- 172.21.15.148:0/970906538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080cf160 msgr2=0x7fc1080cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:22.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc106ffd640 1 --2- 172.21.15.148:0/970906538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080cf160 0x7fc1080cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:22.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc106ffd640 1 -- 172.21.15.148:0/970906538 >> v2:172.21.15.150:3300/0 conn(0x7fc1081538a0 msgr2=0x7fc10815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:22.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc106ffd640 1 --2- 172.21.15.148:0/970906538 >> v2:172.21.15.150:3300/0 conn(0x7fc1081538a0 0x7fc10815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:22.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc106ffd640 1 -- 172.21.15.148:0/970906538 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1080cad70 con 0x7fc1080ca0e0 2023-10-30T13:48:22.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.809+0000 7fc107fff640 1 --2- 172.21.15.148:0/970906538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080cf160 0x7fc1080cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:22.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc106ffd640 1 --2- 172.21.15.148:0/970906538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1080ca0e0 0x7fc10815a090 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fc1080cfd10 tx=0x7fc0fc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f728d270d5807753 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc1067fc640 1 -- 172.21.15.148:0/970906538 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0a89b0 con 0x7fc1080ca0e0 2023-10-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc1067fc640 1 -- 172.21.15.148:0/970906538 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0fc0a8b10 con 0x7fc1080ca0e0 2023-10-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc1067fc640 1 -- 172.21.15.148:0/970906538 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0a8ea0 con 0x7fc1080ca0e0 2023-10-30T13:48:22.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 -- 172.21.15.148:0/970906538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1080ca0e0 msgr2=0x7fc10815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:22.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/970906538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1080ca0e0 0x7fc10815a090 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fc1080cfd10 tx=0x7fc0fc01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:48:22.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 -- 172.21.15.148:0/970906538 shutdown_connections 2023-10-30T13:48:22.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/970906538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080cf160 0x7fc1080cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:22.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/970906538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1080ca0e0 0x7fc10815a090 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:22.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/970906538 >> v2:172.21.15.150:3300/0 conn(0x7fc1081538a0 0x7fc10815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:22.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 -- 172.21.15.148:0/970906538 >> 172.21.15.148:0/970906538 conn(0x7fc10806bfe0 msgr2=0x7fc10806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:22.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 -- 172.21.15.148:0/970906538 shutdown_connections 2023-10-30T13:48:22.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 -- 172.21.15.148:0/970906538 wait complete. 2023-10-30T13:48:22.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 Processor -- start 2023-10-30T13:48:22.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.813+0000 7fc10dd0f640 1 -- start start 2023-10-30T13:48:22.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc10dd0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080ca0e0 0x7fc1080cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:22.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc10dd0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1080cf160 0x7fc108147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:22.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc10dd0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1081538a0 0x7fc1080b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:22.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc10dd0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080b3520 con 0x7fc1081538a0 2023-10-30T13:48:22.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc10dd0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080cd0c0 con 0x7fc1080cf160 2023-10-30T13:48:22.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc10dd0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1081e5d00 con 0x7fc1080ca0e0 2023-10-30T13:48:22.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc107fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1081538a0 0x7fc1080b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:22.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc107fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1081538a0 0x7fc1080b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50044/0 (socket says 172.21.15.148:50044) 2023-10-30T13:48:22.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc107fff640 1 -- 172.21.15.148:0/2003864968 learned_addr learned my addr 172.21.15.148:0/2003864968 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:22.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc1077fe640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080ca0e0 0x7fc1080cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:22.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc106ffd640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1080cf160 0x7fc108147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:22.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc107fff640 1 -- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080ca0e0 msgr2=0x7fc1080cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:22.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc107fff640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080ca0e0 0x7fc1080cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:22.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc107fff640 1 -- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1080cf160 msgr2=0x7fc108147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:22.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc107fff640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1080cf160 0x7fc108147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:22.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc107fff640 1 -- 172.21.15.148:0/2003864968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc0094d0 con 0x7fc1081538a0 2023-10-30T13:48:22.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc1077fe640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080ca0e0 0x7fc1080cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:22.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc107fff640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1081538a0 0x7fc1080b39b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc100007d60 tx=0x7fc10000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:22.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc0effff640 1 -- 172.21.15.148:0/2003864968 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1000039a0 con 0x7fc1081538a0 2023-10-30T13:48:22.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc0effff640 1 -- 172.21.15.148:0/2003864968 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10000f040 con 0x7fc1081538a0 2023-10-30T13:48:22.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc0effff640 1 -- 172.21.15.148:0/2003864968 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc100003d40 con 0x7fc1081538a0 2023-10-30T13:48:22.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1080ceec0 con 0x7fc1081538a0 2023-10-30T13:48:22.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1080b31b0 con 0x7fc1081538a0 2023-10-30T13:48:22.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.817+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1080b2a50 con 0x7fc1081538a0 2023-10-30T13:48:22.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.821+0000 7fc0effff640 1 -- 172.21.15.148:0/2003864968 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc100004030 con 0x7fc1081538a0 2023-10-30T13:48:22.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.821+0000 7fc0effff640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc0f40664a0 0x7fc0f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:22.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.821+0000 7fc0effff640 1 -- 172.21.15.148:0/2003864968 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fc1000847e0 con 0x7fc1081538a0 2023-10-30T13:48:22.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.821+0000 7fc1077fe640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc0f40664a0 0x7fc0f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:22.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.821+0000 7fc1077fe640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc0f40664a0 0x7fc0f4068960 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800b9f0 tx=0x7fc0f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:22.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:22.829+0000 7fc0effff640 1 -- 172.21.15.148:0/2003864968 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc100051570 con 0x7fc1081538a0 2023-10-30T13:48:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1080b36d0 con 0x7fc1081538a0 2023-10-30T13:48:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc0effff640 1 -- 172.21.15.148:0/2003864968 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fc1080b36d0 con 0x7fc1081538a0 2023-10-30T13:48:23.037 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:23.037 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:15.005721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 144 2023-10-30T13:48:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc0f40664a0 msgr2=0x7fc0f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc0f40664a0 0x7fc0f4068960 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800b9f0 tx=0x7fc0f8008040 comp rx=0 tx=0).stop 2023-10-30T13:48:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1081538a0 msgr2=0x7fc1080b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1081538a0 0x7fc1080b39b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc100007d60 tx=0x7fc10000d4a0 comp rx=0 tx=0).stop 2023-10-30T13:48:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 shutdown_connections 2023-10-30T13:48:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc0f40664a0 0x7fc0f4068960 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1081538a0 0x7fc1080b39b0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:23.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1080cf160 0x7fc108147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:23.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 --2- 172.21.15.148:0/2003864968 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc1080ca0e0 0x7fc1080cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:23.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.033+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 >> 172.21.15.148:0/2003864968 conn(0x7fc10806bfe0 msgr2=0x7fc1081525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:23.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.037+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 shutdown_connections 2023-10-30T13:48:23.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:23.037+0000 7fc10dd0f640 1 -- 172.21.15.148:0/2003864968 wait complete. 2023-10-30T13:48:23.062 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:25.063 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:25.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.209+0000 7f15b3bd3640 1 Processor -- start 2023-10-30T13:48:25.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b3bd3640 1 -- start start 2023-10-30T13:48:25.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b3bd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac1538a0 0x7f15ac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:25.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b3bd3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f15ac0cf160 0x7f15ac0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:25.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b3bd3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ac0cb800 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b3bd3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ac15dc30 con 0x7f15ac0cf160 2023-10-30T13:48:25.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b3bd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac0cb330 con 0x7f15ac1538a0 2023-10-30T13:48:25.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b116e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f15ac0ca0e0 0x7f15ac15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48114/0 (socket says 172.21.15.148:48114) 2023-10-30T13:48:25.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b116e640 1 -- 172.21.15.148:0/173961924 learned_addr learned my addr 172.21.15.148:0/173961924 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:25.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b2170640 1 --2- 172.21.15.148:0/173961924 >> v2:172.21.15.148:3300/0 conn(0x7f15ac0cf160 0x7f15ac0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:25.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b196f640 1 --2- 172.21.15.148:0/173961924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac1538a0 0x7f15ac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:25.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b096d640 1 -- 172.21.15.148:0/173961924 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790848890 0 0) 0x7f15ac0cb800 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b096d640 1 -- 172.21.15.148:0/173961924 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1598003530 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b2170640 1 -- 172.21.15.148:0/173961924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac1538a0 msgr2=0x7f15ac15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:25.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b2170640 1 --2- 172.21.15.148:0/173961924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac1538a0 0x7f15ac15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:25.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b2170640 1 -- 172.21.15.148:0/173961924 >> v1:172.21.15.148:6789/0 conn(0x7f15ac0ca0e0 legacy=0x7f15ac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:25.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.213+0000 7f15b2170640 1 -- 172.21.15.148:0/173961924 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac0cad70 con 0x7f15ac0cf160 2023-10-30T13:48:25.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b2170640 1 --2- 172.21.15.148:0/173961924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0cf160 0x7f15ac0cf590 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009d90 tx=0x7f15a401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c01cd8c04cc463 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:25.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b096d640 1 -- 172.21.15.148:0/173961924 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40af600 con 0x7f15ac0cf160 2023-10-30T13:48:25.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b096d640 1 -- 172.21.15.148:0/173961924 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a40ab050 con 0x7f15ac0cf160 2023-10-30T13:48:25.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b096d640 1 -- 172.21.15.148:0/173961924 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40af980 con 0x7f15ac0cf160 2023-10-30T13:48:25.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 -- 172.21.15.148:0/173961924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0cf160 msgr2=0x7f15ac0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:25.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 --2- 172.21.15.148:0/173961924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0cf160 0x7f15ac0cf590 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009d90 tx=0x7f15a401f130 comp rx=0 tx=0).stop 2023-10-30T13:48:25.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 -- 172.21.15.148:0/173961924 shutdown_connections 2023-10-30T13:48:25.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 --2- 172.21.15.148:0/173961924 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0cf160 0x7f15ac0cf590 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:25.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 --2- 172.21.15.148:0/173961924 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac1538a0 0x7f15ac15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:25.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 -- 172.21.15.148:0/173961924 >> 172.21.15.148:0/173961924 conn(0x7f15ac06bfe0 msgr2=0x7f15ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:25.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 -- 172.21.15.148:0/173961924 shutdown_connections 2023-10-30T13:48:25.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 -- 172.21.15.148:0/173961924 wait complete. 2023-10-30T13:48:25.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 Processor -- start 2023-10-30T13:48:25.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.217+0000 7f15b3bd3640 1 -- start start 2023-10-30T13:48:25.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b3bd3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0ca0e0 0x7f15ac1ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:25.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b3bd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac0cf160 0x7f15ac0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:25.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b3bd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15ac1538a0 0x7f15ac0b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:25.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b3bd3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac15b2f0 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b3bd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac143ba0 con 0x7f15ac1538a0 2023-10-30T13:48:25.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b3bd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac1ef310 con 0x7f15ac0cf160 2023-10-30T13:48:25.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b196f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0ca0e0 0x7f15ac1ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:25.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b196f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0ca0e0 0x7f15ac1ef7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50052/0 (socket says 172.21.15.148:50052) 2023-10-30T13:48:25.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b196f640 1 -- 172.21.15.148:0/3888703546 learned_addr learned my addr 172.21.15.148:0/3888703546 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:25.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b116e640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac0cf160 0x7f15ac0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:25.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b2170640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15ac1538a0 0x7f15ac0b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:25.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b196f640 1 -- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac0cf160 msgr2=0x7f15ac0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:25.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b196f640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac0cf160 0x7f15ac0b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:25.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b196f640 1 -- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15ac1538a0 msgr2=0x7f15ac0b4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:25.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b196f640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15ac1538a0 0x7f15ac0b4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:25.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b196f640 1 -- 172.21.15.148:0/3888703546 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a4009a70 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b196f640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0ca0e0 0x7f15ac1ef7a0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f15a800e870 tx=0x7f15a800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:25.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15a27fc640 1 -- 172.21.15.148:0/3888703546 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a8009510 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15a27fc640 1 -- 172.21.15.148:0/3888703546 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a800eec0 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac158fb0 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15a27fc640 1 -- 172.21.15.148:0/3888703546 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a800cb40 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac1eefa0 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.226 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.221+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac0b2a50 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.225+0000 7f15a27fc640 1 -- 172.21.15.148:0/3888703546 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15a800cca0 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.225+0000 7f15a27fc640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15900664d0 0x7f1590068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:25.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.225+0000 7f15a27fc640 1 -- 172.21.15.148:0/3888703546 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f15a8084a00 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.225+0000 7f15b116e640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15900664d0 0x7f1590068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:25.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.225+0000 7f15b116e640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15900664d0 0x7f1590068990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f15ac15acf0 tx=0x7f159c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:25.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.229+0000 7f15a27fc640 1 -- 172.21.15.148:0/3888703546 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15a8051710 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.409+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15ac04e170 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.409+0000 7f15a27fc640 1 -- 172.21.15.148:0/3888703546 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f15a8054d30 con 0x7f15ac0ca0e0 2023-10-30T13:48:25.410 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:25.411 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:23.978101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:resolve","state_seq":29,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:25.411 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 145 2023-10-30T13:48:25.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15900664d0 msgr2=0x7f1590068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:25.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15900664d0 0x7f1590068990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f15ac15acf0 tx=0x7f159c009040 comp rx=0 tx=0).stop 2023-10-30T13:48:25.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0ca0e0 msgr2=0x7f15ac1ef7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:25.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0ca0e0 0x7f15ac1ef7a0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f15a800e870 tx=0x7f15a800ec70 comp rx=0 tx=0).stop 2023-10-30T13:48:25.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 shutdown_connections 2023-10-30T13:48:25.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f15900664d0 0x7f1590068990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:25.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15ac1538a0 0x7f15ac0b4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:25.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f15ac0cf160 0x7f15ac0b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:25.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 --2- 172.21.15.148:0/3888703546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f15ac0ca0e0 0x7f15ac1ef7a0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:25.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 >> 172.21.15.148:0/3888703546 conn(0x7f15ac06bfe0 msgr2=0x7f15ac152410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:25.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 shutdown_connections 2023-10-30T13:48:25.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:25.413+0000 7f15b3bd3640 1 -- 172.21.15.148:0/3888703546 wait complete. 2023-10-30T13:48:25.430 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:27.431 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:27.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27c1817640 1 Processor -- start 2023-10-30T13:48:27.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27c1817640 1 -- start start 2023-10-30T13:48:27.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27c1817640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f27bc0c6b80 0x7f27bc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:27.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27c1817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c6f60 0x7f27bc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:27.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27c1817640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0c0a50 con 0x7f27bc0c8430 2023-10-30T13:48:27.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27c1817640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc06f6e0 con 0x7f27bc0c6b80 2023-10-30T13:48:27.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27c1817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc06f9c0 con 0x7f27bc0c6f60 2023-10-30T13:48:27.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27baffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f27bc0c6b80 0x7f27bc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:27.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27baffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f27bc0c6b80 0x7f27bc0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50058/0 (socket says 172.21.15.148:50058) 2023-10-30T13:48:27.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27baffd640 1 -- 172.21.15.148:0/1090256016 learned_addr learned my addr 172.21.15.148:0/1090256016 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:27.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27b27fc640 1 --2- 172.21.15.148:0/1090256016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c6f60 0x7f27bc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:27.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27baffd640 1 -- 172.21.15.148:0/1090256016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c6f60 msgr2=0x7f27bc0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:27.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27baffd640 1 --2- 172.21.15.148:0/1090256016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c6f60 0x7f27bc0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27baffd640 1 -- 172.21.15.148:0/1090256016 >> v1:172.21.15.150:6789/0 conn(0x7f27bc0c8430 legacy=0x7f27bc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:27.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.581+0000 7f27baffd640 1 -- 172.21.15.148:0/1090256016 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc06fca0 con 0x7f27bc0c6b80 2023-10-30T13:48:27.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27baffd640 1 --2- 172.21.15.148:0/1090256016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6b80 0x7f27bc0bfec0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f27ac009870 tx=0x7f27ac01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c326a0d32d9580b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:27.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27ba7fc640 1 -- 172.21.15.148:0/1090256016 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0ab600 con 0x7f27bc0c6b80 2023-10-30T13:48:27.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27ba7fc640 1 -- 172.21.15.148:0/1090256016 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac0a7050 con 0x7f27bc0c6b80 2023-10-30T13:48:27.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27ba7fc640 1 -- 172.21.15.148:0/1090256016 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0ab980 con 0x7f27bc0c6b80 2023-10-30T13:48:27.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27c1817640 1 -- 172.21.15.148:0/1090256016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6b80 msgr2=0x7f27bc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:27.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27c1817640 1 --2- 172.21.15.148:0/1090256016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6b80 0x7f27bc0bfec0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f27ac009870 tx=0x7f27ac01c470 comp rx=0 tx=0).stop 2023-10-30T13:48:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27c1817640 1 -- 172.21.15.148:0/1090256016 shutdown_connections 2023-10-30T13:48:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27c1817640 1 --2- 172.21.15.148:0/1090256016 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c6f60 0x7f27bc0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27c1817640 1 --2- 172.21.15.148:0/1090256016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6b80 0x7f27bc0bfec0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27c1817640 1 -- 172.21.15.148:0/1090256016 >> 172.21.15.148:0/1090256016 conn(0x7f27bc0c7910 msgr2=0x7f27bc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:27.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27c1817640 1 -- 172.21.15.148:0/1090256016 shutdown_connections 2023-10-30T13:48:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.585+0000 7f27c1817640 1 -- 172.21.15.148:0/1090256016 wait complete. 2023-10-30T13:48:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27c1817640 1 Processor -- start 2023-10-30T13:48:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27c1817640 1 -- start start 2023-10-30T13:48:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27c1817640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0c6b80 0x7f27bc138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:27.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27c1817640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6f60 0x7f27bc13de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27c1817640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c8430 0x7f27bc158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27c1817640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc13c4f0 con 0x7f27bc0c6f60 2023-10-30T13:48:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27c1817640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc154540 con 0x7f27bc0c6b80 2023-10-30T13:48:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27c1817640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc1f9f90 con 0x7f27bc0c8430 2023-10-30T13:48:27.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27b27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6f60 0x7f27bc13de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27b27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6f60 0x7f27bc13de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50074/0 (socket says 172.21.15.148:50074) 2023-10-30T13:48:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27b27fc640 1 -- 172.21.15.148:0/905989242 learned_addr learned my addr 172.21.15.148:0/905989242 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27baffd640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0c6b80 0x7f27bc138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27bb7fe640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c8430 0x7f27bc158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27b27fc640 1 -- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c8430 msgr2=0x7f27bc158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:27.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27b27fc640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c8430 0x7f27bc158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27b27fc640 1 -- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0c6b80 msgr2=0x7f27bc138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:27.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27b27fc640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0c6b80 0x7f27bc138920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27b27fc640 1 -- 172.21.15.148:0/905989242 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ac0094d0 con 0x7f27bc0c6f60 2023-10-30T13:48:27.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.589+0000 7f27b27fc640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6f60 0x7f27bc13de80 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f27a4007ef0 tx=0x7f27a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:27.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27b8ff9640 1 -- 172.21.15.148:0/905989242 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4014620 con 0x7f27bc0c6f60 2023-10-30T13:48:27.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27b8ff9640 1 -- 172.21.15.148:0/905989242 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a4014780 con 0x7f27bc0c6f60 2023-10-30T13:48:27.598 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27b8ff9640 1 -- 172.21.15.148:0/905989242 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4014b30 con 0x7f27bc0c6f60 2023-10-30T13:48:27.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc1547e0 con 0x7f27bc0c6f60 2023-10-30T13:48:27.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc138e60 con 0x7f27bc0c6f60 2023-10-30T13:48:27.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc154a20 con 0x7f27bc0c6f60 2023-10-30T13:48:27.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27b8ff9640 1 -- 172.21.15.148:0/905989242 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f27a4014c90 con 0x7f27bc0c6f60 2023-10-30T13:48:27.599 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27b8ff9640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f279c0664a0 0x7f279c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:27.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27b8ff9640 1 -- 172.21.15.148:0/905989242 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f27a4084f20 con 0x7f27bc0c6f60 2023-10-30T13:48:27.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.593+0000 7f27baffd640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f279c0664a0 0x7f279c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:27.600 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.597+0000 7f27baffd640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f279c0664a0 0x7f279c068960 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0b0860 tx=0x7f27ac0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:27.601 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.597+0000 7f27b8ff9640 1 -- 172.21.15.148:0/905989242 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f27bc154a20 con 0x7f27bc0c6f60 2023-10-30T13:48:27.780 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27bc154a20 con 0x7f27bc0c6f60 2023-10-30T13:48:27.781 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27b8ff9640 1 -- 172.21.15.148:0/905989242 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f27bc154a20 con 0x7f27bc0c6f60 2023-10-30T13:48:27.781 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:27.781 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:23.978101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:resolve","state_seq":29,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:27.782 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 145 2023-10-30T13:48:27.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f279c0664a0 msgr2=0x7f279c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:27.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f279c0664a0 0x7f279c068960 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0b0860 tx=0x7f27ac0a6000 comp rx=0 tx=0).stop 2023-10-30T13:48:27.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6f60 msgr2=0x7f27bc13de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:27.783 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6f60 0x7f27bc13de80 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f27a4007ef0 tx=0x7f27a400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 shutdown_connections 2023-10-30T13:48:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f279c0664a0 0x7f279c068960 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27bc0c6f60 0x7f27bc13de80 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f27bc0c8430 0x7f27bc158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.784 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 --2- 172.21.15.148:0/905989242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0c6b80 0x7f27bc138920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:27.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 >> 172.21.15.148:0/905989242 conn(0x7f27bc0c7910 msgr2=0x7f27bc0b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:27.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 shutdown_connections 2023-10-30T13:48:27.785 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:27.781+0000 7f27c1817640 1 -- 172.21.15.148:0/905989242 wait complete. 2023-10-30T13:48:27.803 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 145, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 145, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7611": { "addr": "172.21.15.148:6835/1228827025", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1228827025, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1228827025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7611, "incarnation": 133, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 10 }, "gid_7812": { "addr": "172.21.15.148:6837/1530445724", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 1530445724, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 1530445724, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7812, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:resolve", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:48:23.978101+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7812, "mds_1": 7611 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:48:29.804 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb2110e640 1 Processor -- start 2023-10-30T13:48:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb2110e640 1 -- start start 2023-10-30T13:48:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb2110e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c1538a0 0x7ffb1c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb2110e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffb1c0ca0e0 0x7ffb1c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb2110e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffb1c0cf160 0x7ffb1c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb2110e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c0cb800 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb2110e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c15dc30 con 0x7ffb1c0cf160 2023-10-30T13:48:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb2110e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c0cb330 con 0x7ffb1c1538a0 2023-10-30T13:48:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffb1c0ca0e0 0x7ffb1c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffb1c0ca0e0 0x7ffb1c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50084/0 (socket says 172.21.15.148:50084) 2023-10-30T13:48:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1259c640 1 -- 172.21.15.148:0/881714030 learned_addr learned my addr 172.21.15.148:0/881714030 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1ad9d640 1 --2- 172.21.15.148:0/881714030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c1538a0 0x7ffb1c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1b59e640 1 --2- 172.21.15.148:0/881714030 >> v2:172.21.15.150:3300/0 conn(0x7ffb1c0cf160 0x7ffb1c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1259c640 1 -- 172.21.15.148:0/881714030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c1538a0 msgr2=0x7ffb1c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1259c640 1 --2- 172.21.15.148:0/881714030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c1538a0 0x7ffb1c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1259c640 1 -- 172.21.15.148:0/881714030 >> v2:172.21.15.150:3300/0 conn(0x7ffb1c0cf160 msgr2=0x7ffb1c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1259c640 1 --2- 172.21.15.148:0/881714030 >> v2:172.21.15.150:3300/0 conn(0x7ffb1c0cf160 0x7ffb1c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.949+0000 7ffb1259c640 1 -- 172.21.15.148:0/881714030 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c0cad70 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb1259c640 1 --2- 172.21.15.148:0/881714030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 0x7ffb1c15a090 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c06cb00 tx=0x7ffb0c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d06d6d643eaf177 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb1a59c640 1 -- 172.21.15.148:0/881714030 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0a89b0 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb1a59c640 1 -- 172.21.15.148:0/881714030 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c0a8b10 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb1a59c640 1 -- 172.21.15.148:0/881714030 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0a8ea0 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb2110e640 1 -- 172.21.15.148:0/881714030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 msgr2=0x7ffb1c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb2110e640 1 --2- 172.21.15.148:0/881714030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 0x7ffb1c15a090 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c06cb00 tx=0x7ffb0c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:48:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb2110e640 1 -- 172.21.15.148:0/881714030 shutdown_connections 2023-10-30T13:48:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb2110e640 1 --2- 172.21.15.148:0/881714030 >> v2:172.21.15.150:3300/0 conn(0x7ffb1c0cf160 0x7ffb1c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb2110e640 1 --2- 172.21.15.148:0/881714030 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 0x7ffb1c15a090 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb2110e640 1 --2- 172.21.15.148:0/881714030 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c1538a0 0x7ffb1c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb2110e640 1 -- 172.21.15.148:0/881714030 >> 172.21.15.148:0/881714030 conn(0x7ffb1c06bfe0 msgr2=0x7ffb1c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb2110e640 1 -- 172.21.15.148:0/881714030 shutdown_connections 2023-10-30T13:48:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.953+0000 7ffb2110e640 1 -- 172.21.15.148:0/881714030 wait complete. 2023-10-30T13:48:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 Processor -- start 2023-10-30T13:48:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 -- start start 2023-10-30T13:48:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 0x7ffb1c15d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0cf160 0x7ffb1c14a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c1538a0 0x7ffb1c1f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c1f2fd0 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c15ad70 con 0x7ffb1c1538a0 2023-10-30T13:48:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c14bab0 con 0x7ffb1c0cf160 2023-10-30T13:48:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 0x7ffb1c15d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 0x7ffb1c15d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:50086/0 (socket says 172.21.15.148:50086) 2023-10-30T13:48:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1ad9d640 1 -- 172.21.15.148:0/404584334 learned_addr learned my addr 172.21.15.148:0/404584334 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1259c640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0cf160 0x7ffb1c14a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1b59e640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c1538a0 0x7ffb1c1f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1ad9d640 1 -- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0cf160 msgr2=0x7ffb1c14a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1ad9d640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0cf160 0x7ffb1c14a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1ad9d640 1 -- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c1538a0 msgr2=0x7ffb1c1f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1ad9d640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c1538a0 0x7ffb1c1f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1ad9d640 1 -- 172.21.15.148:0/404584334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0094d0 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1259c640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0cf160 0x7ffb1c14a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb1ad9d640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 0x7ffb1c15d410 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400d620 tx=0x7ffb0400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb13fff640 1 -- 172.21.15.148:0/404584334 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400b620 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb13fff640 1 -- 172.21.15.148:0/404584334 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0400b780 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c0ce130 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb13fff640 1 -- 172.21.15.148:0/404584334 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400bb30 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.957+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c0ce380 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.961+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c0b2a50 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.961+0000 7ffb13fff640 1 -- 172.21.15.148:0/404584334 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb0400bc90 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.961+0000 7ffb13fff640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffafc0664d0 0x7ffafc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.961+0000 7ffb13fff640 1 -- 172.21.15.148:0/404584334 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7ffb04084a90 con 0x7ffb1c0ca0e0 2023-10-30T13:48:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.961+0000 7ffb1259c640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffafc0664d0 0x7ffafc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.961+0000 7ffb1259c640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffafc0664d0 0x7ffafc068990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0b2000 tx=0x7ffb0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:29.965+0000 7ffb13fff640 1 -- 172.21.15.148:0/404584334 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb04051710 con 0x7ffb1c0ca0e0 2023-10-30T13:48:30.170 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.169+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb1c04e170 con 0x7ffb1c0ca0e0 2023-10-30T13:48:30.171 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.169+0000 7ffb13fff640 1 -- 172.21.15.148:0/404584334 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5741 (secure 0 0 0) 0x7ffb04007380 con 0x7ffb1c0ca0e0 2023-10-30T13:48:30.172 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:30.172 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:29.142939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:30.172 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 146 2023-10-30T13:48:30.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffafc0664d0 msgr2=0x7ffafc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:30.174 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffafc0664d0 0x7ffafc068990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0b2000 tx=0x7ffb0c008040 comp rx=0 tx=0).stop 2023-10-30T13:48:30.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 msgr2=0x7ffb1c15d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:30.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 0x7ffb1c15d410 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400d620 tx=0x7ffb0400dae0 comp rx=0 tx=0).stop 2023-10-30T13:48:30.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 shutdown_connections 2023-10-30T13:48:30.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffafc0664d0 0x7ffafc068990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:30.175 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb1c1538a0 0x7ffb1c1f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:30.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffb1c0cf160 0x7ffb1c14a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:30.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 --2- 172.21.15.148:0/404584334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb1c0ca0e0 0x7ffb1c15d410 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:30.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 >> 172.21.15.148:0/404584334 conn(0x7ffb1c06bfe0 msgr2=0x7ffb1c1483a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:30.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 shutdown_connections 2023-10-30T13:48:30.176 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:30.173+0000 7ffb2110e640 1 -- 172.21.15.148:0/404584334 wait complete. 2023-10-30T13:48:30.199 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:32.200 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:32.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfeca9640 1 Processor -- start 2023-10-30T13:48:32.348 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfeca9640 1 -- start start 2023-10-30T13:48:32.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfeca9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fadf8153d70 0x7fadf806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:32.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfeca9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf806e470 0x7fadf80cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:32.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfeca9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf806afa0 con 0x7fadf80c8490 2023-10-30T13:48:32.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfeca9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf806d780 con 0x7fadf8153d70 2023-10-30T13:48:32.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfeca9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8154150 con 0x7fadf806e470 2023-10-30T13:48:32.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfca45640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fadf8153d70 0x7fadf806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:32.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfca45640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fadf8153d70 0x7fadf806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56818/0 (socket says 172.21.15.148:56818) 2023-10-30T13:48:32.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfd246640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fadf80c8490 0x7fadf80c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56906/0 (socket says 172.21.15.148:56906) 2023-10-30T13:48:32.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfca45640 1 -- 172.21.15.148:0/3034069165 learned_addr learned my addr 172.21.15.148:0/3034069165 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:32.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadeffff640 1 --2- 172.21.15.148:0/3034069165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf806e470 0x7fadf80cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:32.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfca45640 1 -- 172.21.15.148:0/3034069165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf806e470 msgr2=0x7fadf80cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:32.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfca45640 1 --2- 172.21.15.148:0/3034069165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf806e470 0x7fadf80cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:32.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfca45640 1 -- 172.21.15.148:0/3034069165 >> v1:172.21.15.148:6789/0 conn(0x7fadf80c8490 legacy=0x7fadf80c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:32.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.349+0000 7fadfca45640 1 -- 172.21.15.148:0/3034069165 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf806e030 con 0x7fadf8153d70 2023-10-30T13:48:32.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadfca45640 1 --2- 172.21.15.148:0/3034069165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf8153d70 0x7fadf806ca80 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009870 tx=0x7fadf401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca011eb2f73a02 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:32.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadef7fe640 1 -- 172.21.15.148:0/3034069165 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf40aa670 con 0x7fadf8153d70 2023-10-30T13:48:32.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadef7fe640 1 -- 172.21.15.148:0/3034069165 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf40aa7d0 con 0x7fadf8153d70 2023-10-30T13:48:32.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadef7fe640 1 -- 172.21.15.148:0/3034069165 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf40aab60 con 0x7fadf8153d70 2023-10-30T13:48:32.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadfeca9640 1 -- 172.21.15.148:0/3034069165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf8153d70 msgr2=0x7fadf806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:32.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadfeca9640 1 --2- 172.21.15.148:0/3034069165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf8153d70 0x7fadf806ca80 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009870 tx=0x7fadf401c510 comp rx=0 tx=0).stop 2023-10-30T13:48:32.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadfeca9640 1 -- 172.21.15.148:0/3034069165 shutdown_connections 2023-10-30T13:48:32.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadfeca9640 1 --2- 172.21.15.148:0/3034069165 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf806e470 0x7fadf80cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:32.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadfeca9640 1 --2- 172.21.15.148:0/3034069165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf8153d70 0x7fadf806ca80 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:32.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadfeca9640 1 -- 172.21.15.148:0/3034069165 >> 172.21.15.148:0/3034069165 conn(0x7fadf8071ea0 msgr2=0x7fadf8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:32.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadfeca9640 1 -- 172.21.15.148:0/3034069165 shutdown_connections 2023-10-30T13:48:32.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.353+0000 7fadfeca9640 1 -- 172.21.15.148:0/3034069165 wait complete. 2023-10-30T13:48:32.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 Processor -- start 2023-10-30T13:48:32.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 -- start start 2023-10-30T13:48:32.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf806e470 0x7fadf81f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:32.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c8490 0x7fadf8147640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:32.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf8153d70 0x7fadf814ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:32.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80ae4e0 con 0x7fadf806e470 2023-10-30T13:48:32.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf814c840 con 0x7fadf80c8490 2023-10-30T13:48:32.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf81f7360 con 0x7fadf8153d70 2023-10-30T13:48:32.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfca45640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf806e470 0x7fadf81f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:32.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfca45640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf806e470 0x7fadf81f6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56826/0 (socket says 172.21.15.148:56826) 2023-10-30T13:48:32.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfca45640 1 -- 172.21.15.148:0/1439204655 learned_addr learned my addr 172.21.15.148:0/1439204655 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:32.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfd246640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf8153d70 0x7fadf814ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:32.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadeffff640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c8490 0x7fadf8147640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:32.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfca45640 1 -- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf8153d70 msgr2=0x7fadf814ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:32.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfca45640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf8153d70 0x7fadf814ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:32.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfca45640 1 -- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c8490 msgr2=0x7fadf8147640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:32.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfca45640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c8490 0x7fadf8147640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:32.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfca45640 1 -- 172.21.15.148:0/1439204655 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf40094d0 con 0x7fadf806e470 2023-10-30T13:48:32.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadeffff640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c8490 0x7fadf8147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:32.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfca45640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf806e470 0x7fadf81f6450 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fadf40ac040 tx=0x7fadf401ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:32.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7faded7fa640 1 -- 172.21.15.148:0/1439204655 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf40b3800 con 0x7fadf806e470 2023-10-30T13:48:32.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7faded7fa640 1 -- 172.21.15.148:0/1439204655 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf40b3980 con 0x7fadf806e470 2023-10-30T13:48:32.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7faded7fa640 1 -- 172.21.15.148:0/1439204655 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf40b3d30 con 0x7fadf806e470 2023-10-30T13:48:32.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf81f6ae0 con 0x7fadf806e470 2023-10-30T13:48:32.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.357+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf81f7080 con 0x7fadf806e470 2023-10-30T13:48:32.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.361+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf804e170 con 0x7fadf806e470 2023-10-30T13:48:32.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.361+0000 7faded7fa640 1 -- 172.21.15.148:0/1439204655 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fadf40c6440 con 0x7fadf806e470 2023-10-30T13:48:32.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.361+0000 7faded7fa640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664d0 0x7fade4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:32.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.361+0000 7faded7fa640 1 -- 172.21.15.148:0/1439204655 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fadf411f2f0 con 0x7fadf806e470 2023-10-30T13:48:32.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.361+0000 7fadeffff640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664d0 0x7fade4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:32.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.361+0000 7fadeffff640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664d0 0x7fade4068990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fadf81510a0 tx=0x7fade8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:32.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.365+0000 7faded7fa640 1 -- 172.21.15.148:0/1439204655 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fadf40ec110 con 0x7fadf806e470 2023-10-30T13:48:32.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.545+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadf814de00 con 0x7fadf806e470 2023-10-30T13:48:32.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7faded7fa640 1 -- 172.21.15.148:0/1439204655 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fadf40ef730 con 0x7fadf806e470 2023-10-30T13:48:32.548 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:32.549 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:29.142939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:32.549 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 146 2023-10-30T13:48:32.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664d0 msgr2=0x7fade4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:32.550 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664d0 0x7fade4068990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fadf81510a0 tx=0x7fade8008040 comp rx=0 tx=0).stop 2023-10-30T13:48:32.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf806e470 msgr2=0x7fadf81f6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:32.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf806e470 0x7fadf81f6450 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fadf40ac040 tx=0x7fadf401ad50 comp rx=0 tx=0).stop 2023-10-30T13:48:32.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 shutdown_connections 2023-10-30T13:48:32.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fade40664d0 0x7fade4068990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:32.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fadf8153d70 0x7fadf814ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:32.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf80c8490 0x7fadf8147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:32.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 --2- 172.21.15.148:0/1439204655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fadf806e470 0x7fadf81f6450 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:32.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 >> 172.21.15.148:0/1439204655 conn(0x7fadf8071ea0 msgr2=0x7fadf8152210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:32.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 shutdown_connections 2023-10-30T13:48:32.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:32.549+0000 7fadfeca9640 1 -- 172.21.15.148:0/1439204655 wait complete. 2023-10-30T13:48:32.569 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:33.000 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:33.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:48:34.570 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:34.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.717+0000 7f402df1d640 1 Processor -- start 2023-10-30T13:48:34.718 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.717+0000 7f402df1d640 1 -- start start 2023-10-30T13:48:34.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f402df1d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f402806f7a0 0x7f402806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:34.720 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f402df1d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4028071110 0x7f4028158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:34.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f402df1d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4028066cf0 con 0x7f40280c6bd0 2023-10-30T13:48:34.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f402df1d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40280700c0 con 0x7f402806f7a0 2023-10-30T13:48:34.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f402df1d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40280c70d0 con 0x7f4028071110 2023-10-30T13:48:34.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f40277fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f402806f7a0 0x7f402806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:34.721 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f40277fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f402806f7a0 0x7f402806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56838/0 (socket says 172.21.15.148:56838) 2023-10-30T13:48:34.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f40277fe640 1 -- 172.21.15.148:0/3749652577 learned_addr learned my addr 172.21.15.148:0/3749652577 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:34.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f4027fff640 1 --2- 172.21.15.148:0/3749652577 >> v2:172.21.15.150:3300/0 conn(0x7f4028071110 0x7f4028158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:34.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f40277fe640 1 -- 172.21.15.148:0/3749652577 >> v2:172.21.15.150:3300/0 conn(0x7f4028071110 msgr2=0x7f4028158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:34.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f40277fe640 1 --2- 172.21.15.148:0/3749652577 >> v2:172.21.15.150:3300/0 conn(0x7f4028071110 0x7f4028158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.722 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f40277fe640 1 -- 172.21.15.148:0/3749652577 >> v1:172.21.15.150:6789/0 conn(0x7f40280c6bd0 legacy=0x7f4028069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:34.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.721+0000 7f40277fe640 1 -- 172.21.15.148:0/3749652577 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40280c7390 con 0x7f402806f7a0 2023-10-30T13:48:34.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f40277fe640 1 --2- 172.21.15.148:0/3749652577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f402806f7a0 0x7f402806fb80 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f402806cdd0 tx=0x7f401801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a037232c8707eac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:34.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f40267fc640 1 -- 172.21.15.148:0/3749652577 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40180a89f0 con 0x7f402806f7a0 2023-10-30T13:48:34.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f40267fc640 1 -- 172.21.15.148:0/3749652577 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40180a8b50 con 0x7f402806f7a0 2023-10-30T13:48:34.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f40267fc640 1 -- 172.21.15.148:0/3749652577 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40180b0440 con 0x7f402806f7a0 2023-10-30T13:48:34.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 -- 172.21.15.148:0/3749652577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f402806f7a0 msgr2=0x7f402806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:34.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 --2- 172.21.15.148:0/3749652577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f402806f7a0 0x7f402806fb80 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f402806cdd0 tx=0x7f401801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.725 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 -- 172.21.15.148:0/3749652577 shutdown_connections 2023-10-30T13:48:34.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 --2- 172.21.15.148:0/3749652577 >> v2:172.21.15.150:3300/0 conn(0x7f4028071110 0x7f4028158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 --2- 172.21.15.148:0/3749652577 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f402806f7a0 0x7f402806fb80 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 -- 172.21.15.148:0/3749652577 >> 172.21.15.148:0/3749652577 conn(0x7f40280c7910 msgr2=0x7f40280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:34.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 -- 172.21.15.148:0/3749652577 shutdown_connections 2023-10-30T13:48:34.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 -- 172.21.15.148:0/3749652577 wait complete. 2023-10-30T13:48:34.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 Processor -- start 2023-10-30T13:48:34.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.725+0000 7f402df1d640 1 -- start start 2023-10-30T13:48:34.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f402df1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f402806f7a0 0x7f4028157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:34.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f402df1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4028071110 0x7f40280c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:34.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f402df1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f40280c6bd0 0x7f402814f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:34.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f402df1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4028156c50 con 0x7f40280c6bd0 2023-10-30T13:48:34.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f402df1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4028157070 con 0x7f402806f7a0 2023-10-30T13:48:34.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f402df1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4028155cf0 con 0x7f4028071110 2023-10-30T13:48:34.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4027fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f40280c6bd0 0x7f402814f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:34.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4027fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f40280c6bd0 0x7f402814f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56854/0 (socket says 172.21.15.148:56854) 2023-10-30T13:48:34.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4027fff640 1 -- 172.21.15.148:0/3958395456 learned_addr learned my addr 172.21.15.148:0/3958395456 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:34.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f40277fe640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f402806f7a0 0x7f4028157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:34.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4026ffd640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4028071110 0x7f40280c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:34.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4027fff640 1 -- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4028071110 msgr2=0x7f40280c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:34.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4027fff640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4028071110 0x7f40280c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4027fff640 1 -- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f402806f7a0 msgr2=0x7f4028157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:34.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4027fff640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f402806f7a0 0x7f4028157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4027fff640 1 -- 172.21.15.148:0/3958395456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40180094d0 con 0x7f40280c6bd0 2023-10-30T13:48:34.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f4027fff640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f40280c6bd0 0x7f402814f190 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f4020007e60 tx=0x7f402000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:34.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f400ffff640 1 -- 172.21.15.148:0/3958395456 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020014420 con 0x7f40280c6bd0 2023-10-30T13:48:34.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f400ffff640 1 -- 172.21.15.148:0/3958395456 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40200145a0 con 0x7f40280c6bd0 2023-10-30T13:48:34.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4028155fd0 con 0x7f40280c6bd0 2023-10-30T13:48:34.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f400ffff640 1 -- 172.21.15.148:0/3958395456 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020014950 con 0x7f40280c6bd0 2023-10-30T13:48:34.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4028156590 con 0x7f40280c6bd0 2023-10-30T13:48:34.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4028156d90 con 0x7f40280c6bd0 2023-10-30T13:48:34.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.729+0000 7f400ffff640 1 -- 172.21.15.148:0/3958395456 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4020014ab0 con 0x7f40280c6bd0 2023-10-30T13:48:34.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.733+0000 7f400ffff640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40140664a0 0x7f4014068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:34.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.733+0000 7f400ffff640 1 -- 172.21.15.148:0/3958395456 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f4020084d30 con 0x7f40280c6bd0 2023-10-30T13:48:34.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.733+0000 7f40277fe640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40140664a0 0x7f4014068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:34.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.733+0000 7f40277fe640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40140664a0 0x7f4014068960 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f40180b2000 tx=0x7f4018009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:34.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.737+0000 7f400ffff640 1 -- 172.21.15.148:0/3958395456 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4028156d90 con 0x7f40280c6bd0 2023-10-30T13:48:34.914 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.913+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4028156d90 con 0x7f40280c6bd0 2023-10-30T13:48:34.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.913+0000 7f400ffff640 1 -- 172.21.15.148:0/3958395456 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f4028156d90 con 0x7f40280c6bd0 2023-10-30T13:48:34.915 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:34.915 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:29.142939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:34.915 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 146 2023-10-30T13:48:34.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40140664a0 msgr2=0x7f4014068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:34.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40140664a0 0x7f4014068960 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f40180b2000 tx=0x7f4018009040 comp rx=0 tx=0).stop 2023-10-30T13:48:34.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f40280c6bd0 msgr2=0x7f402814f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:34.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f40280c6bd0 0x7f402814f190 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f4020007e60 tx=0x7f402000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 shutdown_connections 2023-10-30T13:48:34.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f40140664a0 0x7f4014068960 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f40280c6bd0 0x7f402814f190 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f4028071110 0x7f40280c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 --2- 172.21.15.148:0/3958395456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f402806f7a0 0x7f4028157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:34.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 >> 172.21.15.148:0/3958395456 conn(0x7f40280c7910 msgr2=0x7f40280c9550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:34.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 shutdown_connections 2023-10-30T13:48:34.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:34.917+0000 7f402df1d640 1 -- 172.21.15.148:0/3958395456 wait complete. 2023-10-30T13:48:34.936 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:36.936 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:37.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.085+0000 7f0b0abd3640 1 Processor -- start 2023-10-30T13:48:37.087 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.085+0000 7f0b0abd3640 1 -- start start 2023-10-30T13:48:37.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b0abd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0406fd40 0x7f0b040710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:37.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b0abd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b040c5ee0 0x7f0b0415acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:37.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b0abd3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b040c6330 con 0x7f0b041537e0 2023-10-30T13:48:37.090 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b0abd3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b041598d0 con 0x7f0b040c5ee0 2023-10-30T13:48:37.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b0abd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b04159b70 con 0x7f0b0406fd40 2023-10-30T13:48:37.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b0896f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0b041537e0 0x7f0b040cb4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56916/0 (socket says 172.21.15.148:56916) 2023-10-30T13:48:37.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b0896f640 1 -- 172.21.15.148:0/1070396487 learned_addr learned my addr 172.21.15.148:0/1070396487 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:37.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afbfff640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0406fd40 0x7f0b040710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:37.091 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b09170640 1 --2- 172.21.15.148:0/1070396487 >> v2:172.21.15.150:3300/0 conn(0x7f0b040c5ee0 0x7f0b0415acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:37.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549894058 0 0) 0x7f0b00001120 con 0x7f0b041537e0 2023-10-30T13:48:37.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aec003530 con 0x7f0b041537e0 2023-10-30T13:48:37.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3901576739 0 0) 0x7f0aec003530 con 0x7f0b041537e0 2023-10-30T13:48:37.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0406fd40 msgr2=0x7f0b040710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:37.092 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0406fd40 0x7f0b040710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 >> v2:172.21.15.150:3300/0 conn(0x7f0b040c5ee0 msgr2=0x7f0b0415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:37.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 --2- 172.21.15.148:0/1070396487 >> v2:172.21.15.150:3300/0 conn(0x7f0b040c5ee0 0x7f0b0415acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b04159e50 con 0x7f0b041537e0 2023-10-30T13:48:37.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f0b00001810 con 0x7f0b041537e0 2023-10-30T13:48:37.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 >> v1:172.21.15.148:6789/0 conn(0x7f0b041537e0 legacy=0x7f0b040cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:37.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0406fd40 0x7f0aec003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:37.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aec003f00 0x7f0aec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:37.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aec004d10 0x7f0aec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:37.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b041598d0 con 0x7f0aec004d10 2023-10-30T13:48:37.094 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b04159b70 con 0x7f0aec003f00 2023-10-30T13:48:37.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec00b990 con 0x7f0b0406fd40 2023-10-30T13:48:37.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b09170640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aec004d10 0x7f0aec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:37.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0afbfff640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aec003f00 0x7f0aec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:37.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b0896f640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0406fd40 0x7f0aec003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:37.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b09170640 1 -- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0406fd40 msgr2=0x7f0aec003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:37.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b09170640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0b0406fd40 0x7f0aec003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b09170640 1 -- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aec003f00 msgr2=0x7f0aec0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:37.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b09170640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0aec003f00 0x7f0aec0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.089+0000 7f0b09170640 1 -- 172.21.15.148:0/1070396487 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aec003530 con 0x7f0aec004d10 2023-10-30T13:48:37.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0b09170640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aec004d10 0x7f0aec005190 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f0afc009fc0 tx=0x7f0afc01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94a718927a2837 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:37.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0a5cb0 con 0x7f0aec004d10 2023-10-30T13:48:37.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc00f040 con 0x7f0aec004d10 2023-10-30T13:48:37.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0afb7fe640 1 -- 172.21.15.148:0/1070396487 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0ad680 con 0x7f0aec004d10 2023-10-30T13:48:37.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0b0abd3640 1 -- 172.21.15.148:0/1070396487 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aec004d10 msgr2=0x7f0aec005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:37.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0b0abd3640 1 --2- 172.21.15.148:0/1070396487 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0aec004d10 0x7f0aec005190 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f0afc009fc0 tx=0x7f0afc01cc40 comp rx=0 tx=0).stop 2023-10-30T13:48:37.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0b09170640 1 -- 172.21.15.148:0/1070396487 reap_dead start 2023-10-30T13:48:37.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0b0abd3640 1 -- 172.21.15.148:0/1070396487 shutdown_connections 2023-10-30T13:48:37.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0b0abd3640 1 -- 172.21.15.148:0/1070396487 >> 172.21.15.148:0/1070396487 conn(0x7f0b0406c5c0 msgr2=0x7f0b04069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:37.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0b0abd3640 1 -- 172.21.15.148:0/1070396487 shutdown_connections 2023-10-30T13:48:37.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.093+0000 7f0b0abd3640 1 -- 172.21.15.148:0/1070396487 wait complete. 2023-10-30T13:48:37.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 Processor -- start 2023-10-30T13:48:37.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 -- start start 2023-10-30T13:48:37.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aec004d10 0x7f0b0414bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:37.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b04158890 0x7f0b040c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:37.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b040c4ff0 0x7f0b0414c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:37.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b040c32e0 con 0x7f0b040c4ff0 2023-10-30T13:48:37.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0414ae70 con 0x7f0b04158890 2023-10-30T13:48:37.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0414ab40 con 0x7f0aec004d10 2023-10-30T13:48:37.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b09170640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b040c4ff0 0x7f0b0414c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b09170640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b040c4ff0 0x7f0b0414c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56872/0 (socket says 172.21.15.148:56872) 2023-10-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b09170640 1 -- 172.21.15.148:0/3608763333 learned_addr learned my addr 172.21.15.148:0/3608763333 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0896f640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aec004d10 0x7f0b0414bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0afbfff640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b04158890 0x7f0b040c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b09170640 1 -- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aec004d10 msgr2=0x7f0b0414bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:37.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b09170640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aec004d10 0x7f0b0414bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b09170640 1 -- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b04158890 msgr2=0x7f0b040c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b09170640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b04158890 0x7f0b040c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b09170640 1 -- 172.21.15.148:0/3608763333 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b041585b0 con 0x7f0b040c4ff0 2023-10-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b09170640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b040c4ff0 0x7f0b0414c9e0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f0afc0b0040 tx=0x7f0afc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:37.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0af97fa640 1 -- 172.21.15.148:0/3608763333 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0039a0 con 0x7f0b040c4ff0 2023-10-30T13:48:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0af97fa640 1 -- 172.21.15.148:0/3608763333 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc003b20 con 0x7f0b040c4ff0 2023-10-30T13:48:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b041ef3c0 con 0x7f0b040c4ff0 2023-10-30T13:48:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0af97fa640 1 -- 172.21.15.148:0/3608763333 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0ae890 con 0x7f0b040c4ff0 2023-10-30T13:48:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0414c300 con 0x7f0b040c4ff0 2023-10-30T13:48:37.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.097+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0404e170 con 0x7f0b040c4ff0 2023-10-30T13:48:37.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.101+0000 7f0af97fa640 1 -- 172.21.15.148:0/3608763333 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0afc0b4040 con 0x7f0b040c4ff0 2023-10-30T13:48:37.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.101+0000 7f0af97fa640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0af00664d0 0x7f0af0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.101+0000 7f0af97fa640 1 -- 172.21.15.148:0/3608763333 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f0afc11f050 con 0x7f0b040c4ff0 2023-10-30T13:48:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.101+0000 7f0b0896f640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0af00664d0 0x7f0af0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.101+0000 7f0b0896f640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0af00664d0 0x7f0af0068990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0b00000c10 tx=0x7f0b00001650 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:37.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.105+0000 7f0af97fa640 1 -- 172.21.15.148:0/3608763333 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0afc0ebf20 con 0x7f0b040c4ff0 2023-10-30T13:48:37.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b0414afb0 con 0x7f0b040c4ff0 2023-10-30T13:48:37.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0af97fa640 1 -- 172.21.15.148:0/3608763333 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f0afc0b2020 con 0x7f0b040c4ff0 2023-10-30T13:48:37.289 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:37.290 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:35.149793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:37.290 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 147 2023-10-30T13:48:37.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0af00664d0 msgr2=0x7f0af0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:37.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0af00664d0 0x7f0af0068990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0b00000c10 tx=0x7f0b00001650 comp rx=0 tx=0).stop 2023-10-30T13:48:37.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b040c4ff0 msgr2=0x7f0b0414c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:37.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b040c4ff0 0x7f0b0414c9e0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f0afc0b0040 tx=0x7f0afc009750 comp rx=0 tx=0).stop 2023-10-30T13:48:37.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 shutdown_connections 2023-10-30T13:48:37.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0af00664d0 0x7f0af0068990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b040c4ff0 0x7f0b0414c9e0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b04158890 0x7f0b040c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 --2- 172.21.15.148:0/3608763333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0aec004d10 0x7f0b0414bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:37.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.289+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 >> 172.21.15.148:0/3608763333 conn(0x7f0b0406c5c0 msgr2=0x7f0b04069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:37.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.293+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 shutdown_connections 2023-10-30T13:48:37.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:37.293+0000 7f0b0abd3640 1 -- 172.21.15.148:0/3608763333 wait complete. 2023-10-30T13:48:37.311 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:39.312 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:39.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.457+0000 7f25a62ea640 1 Processor -- start 2023-10-30T13:48:39.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.457+0000 7f25a62ea640 1 -- start start 2023-10-30T13:48:39.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f25a62ea640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f25a0070a00 0x7f25a0070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:39.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f25a62ea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a00713b0 con 0x7f25a01539a0 2023-10-30T13:48:39.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f25a62ea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a0153d80 con 0x7f25a0069e90 2023-10-30T13:48:39.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f25a62ea640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a006fbb0 con 0x7f25a0070a00 2023-10-30T13:48:39.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f259f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f25a0070a00 0x7f25a0070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:39.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f259f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f25a0070a00 0x7f25a0070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56884/0 (socket says 172.21.15.148:56884) 2023-10-30T13:48:39.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f259f7fe640 1 -- 172.21.15.148:0/1517131677 learned_addr learned my addr 172.21.15.148:0/1517131677 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:39.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f259f7fe640 1 -- 172.21.15.148:0/1517131677 >> v1:172.21.15.150:6789/0 conn(0x7f25a0069e90 legacy=0x7f25a006a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:39.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f259f7fe640 1 -- 172.21.15.148:0/1517131677 >> v1:172.21.15.148:6789/0 conn(0x7f25a01539a0 legacy=0x7f25a015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:39.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.461+0000 7f259f7fe640 1 -- 172.21.15.148:0/1517131677 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a00c5c50 con 0x7f25a0070a00 2023-10-30T13:48:39.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f259f7fe640 1 --2- 172.21.15.148:0/1517131677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0070a00 0x7f25a0070e70 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f2594009870 tx=0x7f259401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cd760e70ce671 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:39.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f259effd640 1 -- 172.21.15.148:0/1517131677 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940ab640 con 0x7f25a0070a00 2023-10-30T13:48:39.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f259effd640 1 -- 172.21.15.148:0/1517131677 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25940a7070 con 0x7f25a0070a00 2023-10-30T13:48:39.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f259effd640 1 -- 172.21.15.148:0/1517131677 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940ab9c0 con 0x7f25a0070a00 2023-10-30T13:48:39.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f25a62ea640 1 -- 172.21.15.148:0/1517131677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0070a00 msgr2=0x7f25a0070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:39.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f25a62ea640 1 --2- 172.21.15.148:0/1517131677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0070a00 0x7f25a0070e70 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f2594009870 tx=0x7f259401c440 comp rx=0 tx=0).stop 2023-10-30T13:48:39.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f25a62ea640 1 -- 172.21.15.148:0/1517131677 shutdown_connections 2023-10-30T13:48:39.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f25a62ea640 1 --2- 172.21.15.148:0/1517131677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0070a00 0x7f25a0070e70 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:39.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f25a62ea640 1 -- 172.21.15.148:0/1517131677 >> 172.21.15.148:0/1517131677 conn(0x7f25a00c7910 msgr2=0x7f25a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:39.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f25a62ea640 1 -- 172.21.15.148:0/1517131677 shutdown_connections 2023-10-30T13:48:39.466 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f25a62ea640 1 -- 172.21.15.148:0/1517131677 wait complete. 2023-10-30T13:48:39.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f25a62ea640 1 Processor -- start 2023-10-30T13:48:39.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.465+0000 7f25a62ea640 1 -- start start 2023-10-30T13:48:39.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a62ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0069e90 0x7f25a00c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:39.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a62ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25a0070a00 0x7f25a00c3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:39.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a62ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a01539a0 0x7f25a014fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:39.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a62ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a0147690 con 0x7f25a0069e90 2023-10-30T13:48:39.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a62ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a006f980 con 0x7f25a01539a0 2023-10-30T13:48:39.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a62ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00b0ba0 con 0x7f25a0070a00 2023-10-30T13:48:39.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0069e90 0x7f25a00c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:39.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0069e90 0x7f25a00c20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56898/0 (socket says 172.21.15.148:56898) 2023-10-30T13:48:39.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259ffff640 1 -- 172.21.15.148:0/596562907 learned_addr learned my addr 172.21.15.148:0/596562907 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:39.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259f7fe640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25a0070a00 0x7f25a00c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:39.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a4887640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a01539a0 0x7f25a014fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:39.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259ffff640 1 -- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25a0070a00 msgr2=0x7f25a00c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:39.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259ffff640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25a0070a00 0x7f25a00c3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:39.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259ffff640 1 -- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a01539a0 msgr2=0x7f25a014fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:39.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259ffff640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a01539a0 0x7f25a014fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:39.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259ffff640 1 -- 172.21.15.148:0/596562907 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25940094d0 con 0x7f25a0069e90 2023-10-30T13:48:39.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259ffff640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0069e90 0x7f25a00c20b0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f259000c3f0 tx=0x7f259000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:39.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259cff9640 1 -- 172.21.15.148:0/596562907 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25900052d0 con 0x7f25a0069e90 2023-10-30T13:48:39.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259cff9640 1 -- 172.21.15.148:0/596562907 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2590005430 con 0x7f25a0069e90 2023-10-30T13:48:39.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259cff9640 1 -- 172.21.15.148:0/596562907 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2590003840 con 0x7f25a0069e90 2023-10-30T13:48:39.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a0146cf0 con 0x7f25a0069e90 2023-10-30T13:48:39.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a01472b0 con 0x7f25a0069e90 2023-10-30T13:48:39.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a004e170 con 0x7f25a0069e90 2023-10-30T13:48:39.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.469+0000 7f259cff9640 1 -- 172.21.15.148:0/596562907 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25900039a0 con 0x7f25a0069e90 2023-10-30T13:48:39.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.473+0000 7f259cff9640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f258c0664d0 0x7f258c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:39.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.473+0000 7f259cff9640 1 -- 172.21.15.148:0/596562907 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f2590083b60 con 0x7f25a0069e90 2023-10-30T13:48:39.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.473+0000 7f259f7fe640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f258c0664d0 0x7f258c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:39.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.473+0000 7f259f7fe640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f258c0664d0 0x7f258c068990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f25940af000 tx=0x7f25940a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:39.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.477+0000 7f259cff9640 1 -- 172.21.15.148:0/596562907 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2590050870 con 0x7f25a0069e90 2023-10-30T13:48:39.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.653+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a01506e0 con 0x7f25a0069e90 2023-10-30T13:48:39.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.653+0000 7f259cff9640 1 -- 172.21.15.148:0/596562907 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f2590053e90 con 0x7f25a0069e90 2023-10-30T13:48:39.655 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:39.655 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:35.149793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 147 2023-10-30T13:48:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f258c0664d0 msgr2=0x7f258c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f258c0664d0 0x7f258c068990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f25940af000 tx=0x7f25940a7000 comp rx=0 tx=0).stop 2023-10-30T13:48:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0069e90 msgr2=0x7f25a00c20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0069e90 0x7f25a00c20b0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f259000c3f0 tx=0x7f259000c8b0 comp rx=0 tx=0).stop 2023-10-30T13:48:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 shutdown_connections 2023-10-30T13:48:39.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f258c0664d0 0x7f258c068990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:39.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a01539a0 0x7f25a014fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:39.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f25a0070a00 0x7f25a00c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:39.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 --2- 172.21.15.148:0/596562907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25a0069e90 0x7f25a00c20b0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:39.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 >> 172.21.15.148:0/596562907 conn(0x7f25a00c7910 msgr2=0x7f25a006bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:39.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 shutdown_connections 2023-10-30T13:48:39.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:39.657+0000 7f25a62ea640 1 -- 172.21.15.148:0/596562907 wait complete. 2023-10-30T13:48:39.676 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 147, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 147, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7611": { "addr": "172.21.15.148:6835/1228827025", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1228827025, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1228827025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7611, "incarnation": 133, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 10 }, "gid_7812": { "addr": "172.21.15.148:6837/1530445724", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 1530445724, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 1530445724, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7812, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:48:35.149793+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7812, "mds_1": 7611 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:48:41.678 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:41.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe863005640 1 Processor -- start 2023-10-30T13:48:41.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe863005640 1 -- start start 2023-10-30T13:48:41.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe863005640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe85c153d70 0x7fe85c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:41.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe863005640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c06e470 0x7fe85c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:41.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe863005640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe85c0c8490 0x7fe85c0cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:41.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe863005640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85c06d780 con 0x7fe85c153d70 2023-10-30T13:48:41.826 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe863005640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85c06afa0 con 0x7fe85c0c8490 2023-10-30T13:48:41.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe863005640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c154150 con 0x7fe85c06e470 2023-10-30T13:48:41.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe860da1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe85c153d70 0x7fe85c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:41.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe860da1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe85c153d70 0x7fe85c06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53624/0 (socket says 172.21.15.148:53624) 2023-10-30T13:48:41.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe860da1640 1 -- 172.21.15.148:0/1286474465 learned_addr learned my addr 172.21.15.148:0/1286474465 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:41.827 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe8615a2640 1 --2- 172.21.15.148:0/1286474465 >> v2:172.21.15.150:3300/0 conn(0x7fe85c0c8490 0x7fe85c0cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:41.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe85bfff640 1 --2- 172.21.15.148:0/1286474465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c06e470 0x7fe85c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:41.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe860da1640 1 -- 172.21.15.148:0/1286474465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c06e470 msgr2=0x7fe85c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:41.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe860da1640 1 --2- 172.21.15.148:0/1286474465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c06e470 0x7fe85c0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:41.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe860da1640 1 -- 172.21.15.148:0/1286474465 >> v2:172.21.15.150:3300/0 conn(0x7fe85c0c8490 msgr2=0x7fe85c0cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:41.828 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe860da1640 1 --2- 172.21.15.148:0/1286474465 >> v2:172.21.15.150:3300/0 conn(0x7fe85c0c8490 0x7fe85c0cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:41.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe860da1640 1 -- 172.21.15.148:0/1286474465 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c06e030 con 0x7fe85c153d70 2023-10-30T13:48:41.829 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.825+0000 7fe8615a2640 1 --2- 172.21.15.148:0/1286474465 >> v2:172.21.15.150:3300/0 conn(0x7fe85c0c8490 0x7fe85c0cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe860da1640 1 --2- 172.21.15.148:0/1286474465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 0x7fe85c06ca80 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe85c0b1c40 tx=0x7fe85001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09de3fb59516226 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe85b7fe640 1 -- 172.21.15.148:0/1286474465 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8500aa5f0 con 0x7fe85c153d70 2023-10-30T13:48:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe85b7fe640 1 -- 172.21.15.148:0/1286474465 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8500aa750 con 0x7fe85c153d70 2023-10-30T13:48:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe85b7fe640 1 -- 172.21.15.148:0/1286474465 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8500aaae0 con 0x7fe85c153d70 2023-10-30T13:48:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe863005640 1 -- 172.21.15.148:0/1286474465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 msgr2=0x7fe85c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe863005640 1 --2- 172.21.15.148:0/1286474465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 0x7fe85c06ca80 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe85c0b1c40 tx=0x7fe85001c4d0 comp rx=0 tx=0).stop 2023-10-30T13:48:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe863005640 1 -- 172.21.15.148:0/1286474465 shutdown_connections 2023-10-30T13:48:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe863005640 1 --2- 172.21.15.148:0/1286474465 >> v2:172.21.15.150:3300/0 conn(0x7fe85c0c8490 0x7fe85c0cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe863005640 1 --2- 172.21.15.148:0/1286474465 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c06e470 0x7fe85c0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe863005640 1 --2- 172.21.15.148:0/1286474465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 0x7fe85c06ca80 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe863005640 1 -- 172.21.15.148:0/1286474465 >> 172.21.15.148:0/1286474465 conn(0x7fe85c071ea0 msgr2=0x7fe85c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe863005640 1 -- 172.21.15.148:0/1286474465 shutdown_connections 2023-10-30T13:48:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.829+0000 7fe863005640 1 -- 172.21.15.148:0/1286474465 wait complete. 2023-10-30T13:48:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 Processor -- start 2023-10-30T13:48:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 -- start start 2023-10-30T13:48:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe85c06e470 0x7fe85c1f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c0c8490 0x7fe85c145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 0x7fe85c1f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c0cc1a0 con 0x7fe85c153d70 2023-10-30T13:48:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c15c880 con 0x7fe85c06e470 2023-10-30T13:48:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c1485a0 con 0x7fe85c0c8490 2023-10-30T13:48:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8615a2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 0x7fe85c1f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8615a2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 0x7fe85c1f5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53632/0 (socket says 172.21.15.148:53632) 2023-10-30T13:48:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8615a2640 1 -- 172.21.15.148:0/93487454 learned_addr learned my addr 172.21.15.148:0/93487454 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe85bfff640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c0c8490 0x7fe85c145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe860da1640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe85c06e470 0x7fe85c1f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8615a2640 1 -- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c0c8490 msgr2=0x7fe85c145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8615a2640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c0c8490 0x7fe85c145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8615a2640 1 -- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe85c06e470 msgr2=0x7fe85c1f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8615a2640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe85c06e470 0x7fe85c1f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8615a2640 1 -- 172.21.15.148:0/93487454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8500094d0 con 0x7fe85c153d70 2023-10-30T13:48:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe860da1640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe85c06e470 0x7fe85c1f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8615a2640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 0x7fe85c1f5bd0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fe854009fa0 tx=0x7fe85400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8597fa640 1 -- 172.21.15.148:0/93487454 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854003ba0 con 0x7fe85c153d70 2023-10-30T13:48:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8597fa640 1 -- 172.21.15.148:0/93487454 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85400f040 con 0x7fe85c153d70 2023-10-30T13:48:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c1f2660 con 0x7fe85c153d70 2023-10-30T13:48:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe8597fa640 1 -- 172.21.15.148:0/93487454 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854004030 con 0x7fe85c153d70 2023-10-30T13:48:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c148860 con 0x7fe85c153d70 2023-10-30T13:48:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.833+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85c04e170 con 0x7fe85c153d70 2023-10-30T13:48:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.837+0000 7fe8597fa640 1 -- 172.21.15.148:0/93487454 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe854004190 con 0x7fe85c153d70 2023-10-30T13:48:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.837+0000 7fe8597fa640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8400664d0 0x7fe840068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.837+0000 7fe8597fa640 1 -- 172.21.15.148:0/93487454 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fe8540848e0 con 0x7fe85c153d70 2023-10-30T13:48:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.837+0000 7fe860da1640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8400664d0 0x7fe840068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.837+0000 7fe860da1640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8400664d0 0x7fe840068990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe8500ae000 tx=0x7fe8500a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:41.841+0000 7fe8597fa640 1 -- 172.21.15.148:0/93487454 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8540515f0 con 0x7fe85c153d70 2023-10-30T13:48:42.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.021+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe85c06dd20 con 0x7fe85c153d70 2023-10-30T13:48:42.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.021+0000 7fe8597fa640 1 -- 172.21.15.148:0/93487454 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+5738 (secure 0 0 0) 0x7fe85401a090 con 0x7fe85c153d70 2023-10-30T13:48:42.023 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:42.024 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:35.149793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:42.024 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 147 2023-10-30T13:48:42.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8400664d0 msgr2=0x7fe840068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:42.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8400664d0 0x7fe840068990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe8500ae000 tx=0x7fe8500a7000 comp rx=0 tx=0).stop 2023-10-30T13:48:42.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 msgr2=0x7fe85c1f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:42.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 0x7fe85c1f5bd0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fe854009fa0 tx=0x7fe85400d560 comp rx=0 tx=0).stop 2023-10-30T13:48:42.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 shutdown_connections 2023-10-30T13:48:42.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe8400664d0 0x7fe840068990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:42.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe85c153d70 0x7fe85c1f5bd0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:42.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe85c0c8490 0x7fe85c145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:42.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 --2- 172.21.15.148:0/93487454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe85c06e470 0x7fe85c1f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:42.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 >> 172.21.15.148:0/93487454 conn(0x7fe85c071ea0 msgr2=0x7fe85c14faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:42.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 shutdown_connections 2023-10-30T13:48:42.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:42.025+0000 7fe863005640 1 -- 172.21.15.148:0/93487454 wait complete. 2023-10-30T13:48:42.044 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:44.045 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:44.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f877ae46640 1 Processor -- start 2023-10-30T13:48:44.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f877ae46640 1 -- start start 2023-10-30T13:48:44.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f877ae46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f877415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:44.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f877ae46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f87740cf160 0x7f87740cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:44.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f877ae46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87740cb800 con 0x7f87741538a0 2023-10-30T13:48:44.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f877ae46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87740cb330 con 0x7f87740cf160 2023-10-30T13:48:44.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f877ae46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87740cad70 con 0x7f877415d6f0 2023-10-30T13:48:44.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f876bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f877415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:44.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f8778be2640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f87741538a0 0x7f87740ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:51532/0 (socket says 172.21.15.148:51532) 2023-10-30T13:48:44.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f876bfff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f877415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:59260/0 (socket says 172.21.15.148:59260) 2023-10-30T13:48:44.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f87793e3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f87740cf160 0x7f87740cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:44.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f8778be2640 1 -- 172.21.15.148:0/106397088 learned_addr learned my addr 172.21.15.148:0/106397088 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:44.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f876b7fe640 1 -- 172.21.15.148:0/106397088 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268472484 0 0) 0x7f87740cb800 con 0x7f87741538a0 2023-10-30T13:48:44.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f876b7fe640 1 -- 172.21.15.148:0/106397088 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f875c003530 con 0x7f87741538a0 2023-10-30T13:48:44.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f876bfff640 1 -- 172.21.15.148:0/106397088 >> v2:172.21.15.150:3300/0 conn(0x7f87740cf160 msgr2=0x7f87740cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:44.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f876bfff640 1 --2- 172.21.15.148:0/106397088 >> v2:172.21.15.150:3300/0 conn(0x7f87740cf160 0x7f87740cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:44.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f876bfff640 1 -- 172.21.15.148:0/106397088 >> v1:172.21.15.150:6789/0 conn(0x7f87741538a0 legacy=0x7f87740ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:44.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.197+0000 7f876bfff640 1 -- 172.21.15.148:0/106397088 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87740cb050 con 0x7f877415d6f0 2023-10-30T13:48:44.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f876bfff640 1 --2- 172.21.15.148:0/106397088 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f877415a090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f87640097c0 tx=0x7f876401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb75af7748d2610e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:44.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f876b7fe640 1 -- 172.21.15.148:0/106397088 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640a8970 con 0x7f877415d6f0 2023-10-30T13:48:44.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f876b7fe640 1 -- 172.21.15.148:0/106397088 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87640a8ad0 con 0x7f877415d6f0 2023-10-30T13:48:44.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f876b7fe640 1 -- 172.21.15.148:0/106397088 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640a8e40 con 0x7f877415d6f0 2023-10-30T13:48:44.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f877ae46640 1 -- 172.21.15.148:0/106397088 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 msgr2=0x7f877415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:44.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f877ae46640 1 --2- 172.21.15.148:0/106397088 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f877415a090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f87640097c0 tx=0x7f876401ec70 comp rx=0 tx=0).stop 2023-10-30T13:48:44.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f877ae46640 1 -- 172.21.15.148:0/106397088 shutdown_connections 2023-10-30T13:48:44.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f877ae46640 1 --2- 172.21.15.148:0/106397088 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f877415a090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:44.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f877ae46640 1 --2- 172.21.15.148:0/106397088 >> v2:172.21.15.150:3300/0 conn(0x7f87740cf160 0x7f87740cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:44.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f877ae46640 1 -- 172.21.15.148:0/106397088 >> 172.21.15.148:0/106397088 conn(0x7f877406bfe0 msgr2=0x7f877406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:44.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f877ae46640 1 -- 172.21.15.148:0/106397088 shutdown_connections 2023-10-30T13:48:44.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.201+0000 7f877ae46640 1 -- 172.21.15.148:0/106397088 wait complete. 2023-10-30T13:48:44.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 Processor -- start 2023-10-30T13:48:44.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 -- start start 2023-10-30T13:48:44.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87740cf160 0x7f87740b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:44.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87741538a0 0x7f87740c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f8774150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87740b3a20 con 0x7f87741538a0 2023-10-30T13:48:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8774150760 con 0x7f87740cf160 2023-10-30T13:48:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87741f2e70 con 0x7f877415d6f0 2023-10-30T13:48:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f876bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87741538a0 0x7f87740c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:44.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f876bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87741538a0 0x7f87740c7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53654/0 (socket says 172.21.15.148:53654) 2023-10-30T13:48:44.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f876bfff640 1 -- 172.21.15.148:0/436471721 learned_addr learned my addr 172.21.15.148:0/436471721 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:44.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f8778be2640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87740cf160 0x7f87740b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:44.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f87793e3640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f8774150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:44.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f876bfff640 1 -- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 msgr2=0x7f8774150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f876bfff640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f8774150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f876bfff640 1 -- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87740cf160 msgr2=0x7f87740b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f876bfff640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87740cf160 0x7f87740b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f876bfff640 1 -- 172.21.15.148:0/436471721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8764009420 con 0x7f87741538a0 2023-10-30T13:48:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f8778be2640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87740cf160 0x7f87740b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f876bfff640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87741538a0 0x7f87740c7310 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f876401d320 tx=0x7f876401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f87697fa640 1 -- 172.21.15.148:0/436471721 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640044d0 con 0x7f87741538a0 2023-10-30T13:48:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f87697fa640 1 -- 172.21.15.148:0/436471721 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8764003620 con 0x7f87741538a0 2023-10-30T13:48:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87741f3150 con 0x7f87741538a0 2023-10-30T13:48:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f87697fa640 1 -- 172.21.15.148:0/436471721 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764003990 con 0x7f87741538a0 2023-10-30T13:48:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.205+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87740b4450 con 0x7f87741538a0 2023-10-30T13:48:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.209+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87740b2a50 con 0x7f87741538a0 2023-10-30T13:48:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.209+0000 7f87697fa640 1 -- 172.21.15.148:0/436471721 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8764003af0 con 0x7f87741538a0 2023-10-30T13:48:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.209+0000 7f87697fa640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87600664a0 0x7f8760068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.209+0000 7f87697fa640 1 -- 172.21.15.148:0/436471721 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f87641225f0 con 0x7f87741538a0 2023-10-30T13:48:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.209+0000 7f8778be2640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87600664a0 0x7f8760068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.213+0000 7f8778be2640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87600664a0 0x7f8760068960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8770005890 tx=0x7f8770009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.213+0000 7f87697fa640 1 -- 172.21.15.148:0/436471721 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87640ef530 con 0x7f87741538a0 2023-10-30T13:48:44.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f877404e170 con 0x7f87741538a0 2023-10-30T13:48:44.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f87697fa640 1 -- 172.21.15.148:0/436471721 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5744 (secure 0 0 0) 0x7f87640f2b50 con 0x7f87741538a0 2023-10-30T13:48:44.393 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:44.394 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:42.607756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:44.394 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 148 2023-10-30T13:48:44.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87600664a0 msgr2=0x7f8760068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:44.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87600664a0 0x7f8760068960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8770005890 tx=0x7f8770009040 comp rx=0 tx=0).stop 2023-10-30T13:48:44.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87741538a0 msgr2=0x7f87740c7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:44.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87741538a0 0x7f87740c7310 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f876401d320 tx=0x7f876401ba10 comp rx=0 tx=0).stop 2023-10-30T13:48:44.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 shutdown_connections 2023-10-30T13:48:44.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f87600664a0 0x7f8760068960 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:44.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f87741538a0 0x7f87740c7310 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:44.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f877415d6f0 0x7f8774150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:44.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 --2- 172.21.15.148:0/436471721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87740cf160 0x7f87740b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:44.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 >> 172.21.15.148:0/436471721 conn(0x7f877406bfe0 msgr2=0x7f877415aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:44.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.393+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 shutdown_connections 2023-10-30T13:48:44.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:44.397+0000 7f877ae46640 1 -- 172.21.15.148:0/436471721 wait complete. 2023-10-30T13:48:44.414 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:46.415 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:46.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b95891640 1 Processor -- start 2023-10-30T13:48:46.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b95891640 1 -- start start 2023-10-30T13:48:46.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b95891640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b901538a0 0x7f5b9015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:46.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b95891640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b900ca0e0 0x7f5b9015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:46.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b95891640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b900cb800 con 0x7f5b900cf160 2023-10-30T13:48:46.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b95891640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b9015dc30 con 0x7f5b900ca0e0 2023-10-30T13:48:46.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b95891640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b900cb330 con 0x7f5b901538a0 2023-10-30T13:48:46.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b8e7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b900ca0e0 0x7f5b9015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:46.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b8e7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b900ca0e0 0x7f5b9015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53658/0 (socket says 172.21.15.148:53658) 2023-10-30T13:48:46.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b8e7fc640 1 -- 172.21.15.148:0/985550419 learned_addr learned my addr 172.21.15.148:0/985550419 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:46.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b8e7fc640 1 -- 172.21.15.148:0/985550419 >> v2:172.21.15.150:3300/0 conn(0x7f5b901538a0 msgr2=0x7f5b9015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:46.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b8effd640 1 --2- 172.21.15.148:0/985550419 >> v2:172.21.15.150:3300/0 conn(0x7f5b901538a0 0x7f5b9015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:46.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b8e7fc640 1 --2- 172.21.15.148:0/985550419 >> v2:172.21.15.150:3300/0 conn(0x7f5b901538a0 0x7f5b9015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b8e7fc640 1 -- 172.21.15.148:0/985550419 >> v1:172.21.15.150:6789/0 conn(0x7f5b900cf160 legacy=0x7f5b900cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:46.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b8e7fc640 1 -- 172.21.15.148:0/985550419 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b900cad70 con 0x7f5b900ca0e0 2023-10-30T13:48:46.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.565+0000 7f5b8effd640 1 --2- 172.21.15.148:0/985550419 >> v2:172.21.15.150:3300/0 conn(0x7f5b901538a0 0x7f5b9015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:48:46.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b8e7fc640 1 --2- 172.21.15.148:0/985550419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900ca0e0 0x7f5b9015a090 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f5b900cfd10 tx=0x7f5b8401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9a54da920c2e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:46.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b8dffb640 1 -- 172.21.15.148:0/985550419 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b840a89f0 con 0x7f5b900ca0e0 2023-10-30T13:48:46.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b8dffb640 1 -- 172.21.15.148:0/985550419 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b840a8b50 con 0x7f5b900ca0e0 2023-10-30T13:48:46.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b8dffb640 1 -- 172.21.15.148:0/985550419 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b840b0440 con 0x7f5b900ca0e0 2023-10-30T13:48:46.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b95891640 1 -- 172.21.15.148:0/985550419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900ca0e0 msgr2=0x7f5b9015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:46.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b95891640 1 --2- 172.21.15.148:0/985550419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900ca0e0 0x7f5b9015a090 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f5b900cfd10 tx=0x7f5b8401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b95891640 1 -- 172.21.15.148:0/985550419 shutdown_connections 2023-10-30T13:48:46.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b95891640 1 --2- 172.21.15.148:0/985550419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900ca0e0 0x7f5b9015a090 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b95891640 1 --2- 172.21.15.148:0/985550419 >> v2:172.21.15.150:3300/0 conn(0x7f5b901538a0 0x7f5b9015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b95891640 1 -- 172.21.15.148:0/985550419 >> 172.21.15.148:0/985550419 conn(0x7f5b9006bfe0 msgr2=0x7f5b9006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:46.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b95891640 1 -- 172.21.15.148:0/985550419 shutdown_connections 2023-10-30T13:48:46.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b95891640 1 -- 172.21.15.148:0/985550419 wait complete. 2023-10-30T13:48:46.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.569+0000 7f5b95891640 1 Processor -- start 2023-10-30T13:48:46.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 -- start start 2023-10-30T13:48:46.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b900ca0e0 0x7f5b901e4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:46.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900cf160 0x7f5b901e49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:46.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b901538a0 0x7f5b900cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:46.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b90157f30 con 0x7f5b900cf160 2023-10-30T13:48:46.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b901e2dc0 con 0x7f5b901538a0 2023-10-30T13:48:46.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b900b4c20 con 0x7f5b900ca0e0 2023-10-30T13:48:46.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900cf160 0x7f5b901e49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:46.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900cf160 0x7f5b901e49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53672/0 (socket says 172.21.15.148:53672) 2023-10-30T13:48:46.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8e7fc640 1 -- 172.21.15.148:0/2383531589 learned_addr learned my addr 172.21.15.148:0/2383531589 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:46.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8effd640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b900ca0e0 0x7f5b901e4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:46.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8f7fe640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b901538a0 0x7f5b900cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:46.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8e7fc640 1 -- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b900ca0e0 msgr2=0x7f5b901e4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:46.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8e7fc640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b900ca0e0 0x7f5b901e4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8e7fc640 1 -- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b901538a0 msgr2=0x7f5b900cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:46.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8e7fc640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b901538a0 0x7f5b900cd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8e7fc640 1 -- 172.21.15.148:0/2383531589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b840094d0 con 0x7f5b900cf160 2023-10-30T13:48:46.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8f7fe640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b901538a0 0x7f5b900cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:46.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b8e7fc640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900cf160 0x7f5b901e49b0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f5b840b3040 tx=0x7f5b8401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:46.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b77fff640 1 -- 172.21.15.148:0/2383531589 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b840ba600 con 0x7f5b900cf160 2023-10-30T13:48:46.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b77fff640 1 -- 172.21.15.148:0/2383531589 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b840ba780 con 0x7f5b900cf160 2023-10-30T13:48:46.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b77fff640 1 -- 172.21.15.148:0/2383531589 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b840bab30 con 0x7f5b900cf160 2023-10-30T13:48:46.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b900b4f00 con 0x7f5b900cf160 2023-10-30T13:48:46.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b90158420 con 0x7f5b900cf160 2023-10-30T13:48:46.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.573+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b900b2a50 con 0x7f5b900cf160 2023-10-30T13:48:46.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.577+0000 7f5b77fff640 1 -- 172.21.15.148:0/2383531589 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b840bac90 con 0x7f5b900cf160 2023-10-30T13:48:46.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.577+0000 7f5b77fff640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b7c0664d0 0x7f5b7c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:46.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.577+0000 7f5b77fff640 1 -- 172.21.15.148:0/2383531589 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f5b84122b80 con 0x7f5b900cf160 2023-10-30T13:48:46.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.577+0000 7f5b8effd640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b7c0664d0 0x7f5b7c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:46.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.577+0000 7f5b8effd640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b7c0664d0 0x7f5b7c068990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5b80006fd0 tx=0x7f5b80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:46.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.581+0000 7f5b77fff640 1 -- 172.21.15.148:0/2383531589 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b840ef920 con 0x7f5b900cf160 2023-10-30T13:48:46.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.769+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b9004e170 con 0x7f5b900cf160 2023-10-30T13:48:46.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.769+0000 7f5b77fff640 1 -- 172.21.15.148:0/2383531589 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5744 (secure 0 0 0) 0x7f5b840f2f40 con 0x7f5b900cf160 2023-10-30T13:48:46.771 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:46.771 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:42.607756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:46.771 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 148 2023-10-30T13:48:46.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b7c0664d0 msgr2=0x7f5b7c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:46.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b7c0664d0 0x7f5b7c068990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5b80006fd0 tx=0x7f5b80008040 comp rx=0 tx=0).stop 2023-10-30T13:48:46.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900cf160 msgr2=0x7f5b901e49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:46.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900cf160 0x7f5b901e49b0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f5b840b3040 tx=0x7f5b8401ba90 comp rx=0 tx=0).stop 2023-10-30T13:48:46.773 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 shutdown_connections 2023-10-30T13:48:46.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b7c0664d0 0x7f5b7c068990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b900cf160 0x7f5b901e49b0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b901538a0 0x7f5b900cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 --2- 172.21.15.148:0/2383531589 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b900ca0e0 0x7f5b901e4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:46.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 >> 172.21.15.148:0/2383531589 conn(0x7f5b9006bfe0 msgr2=0x7f5b900cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:46.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 shutdown_connections 2023-10-30T13:48:46.775 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:46.773+0000 7f5b95891640 1 -- 172.21.15.148:0/2383531589 wait complete. 2023-10-30T13:48:46.792 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:48.793 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:48.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.941+0000 7f9ed34c9640 1 Processor -- start 2023-10-30T13:48:48.943 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.941+0000 7f9ed34c9640 1 -- start start 2023-10-30T13:48:48.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed34c9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9ecc06f7a0 0x7f9ecc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:48.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed34c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc066cf0 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed34c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc0c70d0 con 0x7f9ecc071110 2023-10-30T13:48:48.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed34c9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ecc0700c0 con 0x7f9ecc06f7a0 2023-10-30T13:48:48.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed1265640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9ecc06f7a0 0x7f9ecc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:48.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed1265640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9ecc06f7a0 0x7f9ecc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53678/0 (socket says 172.21.15.148:53678) 2023-10-30T13:48:48.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed1265640 1 -- 172.21.15.148:0/784278330 learned_addr learned my addr 172.21.15.148:0/784278330 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:48.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed0a64640 1 --1- 172.21.15.148:0/784278330 >> v1:172.21.15.148:6789/0 conn(0x7f9ecc0c6bd0 0x7f9ecc069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36792/0 (socket says 172.21.15.148:36792) 2023-10-30T13:48:48.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ec3fff640 1 -- 172.21.15.148:0/784278330 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960995909 0 0) 0x7f9ecc066cf0 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ec3fff640 1 -- 172.21.15.148:0/784278330 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eb8003530 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed1265640 1 -- 172.21.15.148:0/784278330 >> v1:172.21.15.150:6789/0 conn(0x7f9ecc071110 legacy=0x7f9ecc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:48.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed1265640 1 -- 172.21.15.148:0/784278330 >> v1:172.21.15.148:6789/0 conn(0x7f9ecc0c6bd0 legacy=0x7f9ecc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:48.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.945+0000 7f9ed1265640 1 -- 172.21.15.148:0/784278330 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc0c7390 con 0x7f9ecc06f7a0 2023-10-30T13:48:48.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ed1265640 1 --2- 172.21.15.148:0/784278330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc06f7a0 0x7f9ecc06fb80 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009870 tx=0x7f9ec801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1ded0e5cf54e22 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:48.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ec3fff640 1 -- 172.21.15.148:0/784278330 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec80a8a30 con 0x7f9ecc06f7a0 2023-10-30T13:48:48.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ec3fff640 1 -- 172.21.15.148:0/784278330 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec80a8b90 con 0x7f9ecc06f7a0 2023-10-30T13:48:48.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ec3fff640 1 -- 172.21.15.148:0/784278330 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec80b0440 con 0x7f9ecc06f7a0 2023-10-30T13:48:48.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ed34c9640 1 -- 172.21.15.148:0/784278330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc06f7a0 msgr2=0x7f9ecc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:48.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ed34c9640 1 --2- 172.21.15.148:0/784278330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc06f7a0 0x7f9ecc06fb80 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009870 tx=0x7f9ec801ed30 comp rx=0 tx=0).stop 2023-10-30T13:48:48.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ed34c9640 1 -- 172.21.15.148:0/784278330 shutdown_connections 2023-10-30T13:48:48.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ed34c9640 1 --2- 172.21.15.148:0/784278330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc06f7a0 0x7f9ecc06fb80 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:48.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ed34c9640 1 -- 172.21.15.148:0/784278330 >> 172.21.15.148:0/784278330 conn(0x7f9ecc0c7910 msgr2=0x7f9ecc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:48.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ed34c9640 1 -- 172.21.15.148:0/784278330 shutdown_connections 2023-10-30T13:48:48.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ed34c9640 1 -- 172.21.15.148:0/784278330 wait complete. 2023-10-30T13:48:48.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.949+0000 7f9ed34c9640 1 Processor -- start 2023-10-30T13:48:48.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 -- start start 2023-10-30T13:48:48.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ecc06f7a0 0x7f9ecc154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:48.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ecc071110 0x7f9ecc0c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:48.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc0c6bd0 0x7f9ecc153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:48.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc1ed4f0 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc153630 con 0x7f9ecc071110 2023-10-30T13:48:48.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc152a20 con 0x7f9ecc06f7a0 2023-10-30T13:48:48.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1a66640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc0c6bd0 0x7f9ecc153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:48.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1a66640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc0c6bd0 0x7f9ecc153ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53694/0 (socket says 172.21.15.148:53694) 2023-10-30T13:48:48.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1a66640 1 -- 172.21.15.148:0/3643422176 learned_addr learned my addr 172.21.15.148:0/3643422176 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:48.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed0a64640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ecc071110 0x7f9ecc0c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:48.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1265640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ecc06f7a0 0x7f9ecc154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:48.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1a66640 1 -- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ecc06f7a0 msgr2=0x7f9ecc154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:48.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1a66640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ecc06f7a0 0x7f9ecc154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:48.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1a66640 1 -- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ecc071110 msgr2=0x7f9ecc0c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:48.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1a66640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ecc071110 0x7f9ecc0c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:48.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1a66640 1 -- 172.21.15.148:0/3643422176 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80094d0 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed1a66640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc0c6bd0 0x7f9ecc153ac0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400c730 tx=0x7f9ec400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:48.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ec1ffb640 1 -- 172.21.15.148:0/3643422176 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4003b80 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ec1ffb640 1 -- 172.21.15.148:0/3643422176 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec4003ce0 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ec1ffb640 1 -- 172.21.15.148:0/3643422176 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec400aa30 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc152d00 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc1532c0 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.953+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc1537e0 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.957+0000 7f9ec1ffb640 1 -- 172.21.15.148:0/3643422176 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ec400ab90 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.957+0000 7f9ec1ffb640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9eb00664a0 0x7f9eb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:48.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.957+0000 7f9ec1ffb640 1 -- 172.21.15.148:0/3643422176 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f9ec40841e0 con 0x7f9ecc0c6bd0 2023-10-30T13:48:48.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.957+0000 7f9ed1265640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9eb00664a0 0x7f9eb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:48.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.957+0000 7f9ed1265640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9eb00664a0 0x7f9eb0068960 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80b2000 tx=0x7f9ec80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:48.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:48.961+0000 7f9ec1ffb640 1 -- 172.21.15.148:0/3643422176 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ecc1537e0 con 0x7f9ecc0c6bd0 2023-10-30T13:48:49.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.141+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ecc153450 con 0x7f9ecc0c6bd0 2023-10-30T13:48:49.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.141+0000 7f9ec1ffb640 1 -- 172.21.15.148:0/3643422176 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5744 (secure 0 0 0) 0x7f9ec4051030 con 0x7f9ecc0c6bd0 2023-10-30T13:48:49.142 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:49.142 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:42.607756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:49.143 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 148 2023-10-30T13:48:49.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9eb00664a0 msgr2=0x7f9eb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:49.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9eb00664a0 0x7f9eb0068960 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80b2000 tx=0x7f9ec80aa070 comp rx=0 tx=0).stop 2023-10-30T13:48:49.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc0c6bd0 msgr2=0x7f9ecc153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:49.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc0c6bd0 0x7f9ecc153ac0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400c730 tx=0x7f9ec400cbf0 comp rx=0 tx=0).stop 2023-10-30T13:48:49.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 shutdown_connections 2023-10-30T13:48:49.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f9eb00664a0 0x7f9eb0068960 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:49.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9ecc0c6bd0 0x7f9ecc153ac0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:49.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ecc071110 0x7f9ecc0c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:49.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 --2- 172.21.15.148:0/3643422176 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f9ecc06f7a0 0x7f9ecc154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:49.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 >> 172.21.15.148:0/3643422176 conn(0x7f9ecc0c7910 msgr2=0x7f9ecc14fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:49.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 shutdown_connections 2023-10-30T13:48:49.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:49.145+0000 7f9ed34c9640 1 -- 172.21.15.148:0/3643422176 wait complete. 2023-10-30T13:48:49.164 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:48:51.165 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:51.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab3353e640 1 Processor -- start 2023-10-30T13:48:51.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab3353e640 1 -- start start 2023-10-30T13:48:51.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab3353e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c1538a0 0x7fab2c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:51.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab3353e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0cf160 0x7fab2c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:51.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab3353e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2c0cb800 con 0x7fab2c0ca0e0 2023-10-30T13:48:51.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab3353e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2c15dc30 con 0x7fab2c0cf160 2023-10-30T13:48:51.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab3353e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0cb330 con 0x7fab2c1538a0 2023-10-30T13:48:51.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab31adb640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0cf160 0x7fab2c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:51.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab31adb640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0cf160 0x7fab2c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53696/0 (socket says 172.21.15.148:53696) 2023-10-30T13:48:51.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab31adb640 1 -- 172.21.15.148:0/2471509438 learned_addr learned my addr 172.21.15.148:0/2471509438 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:51.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab312da640 1 --2- 172.21.15.148:0/2471509438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c1538a0 0x7fab2c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:51.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab31adb640 1 -- 172.21.15.148:0/2471509438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c1538a0 msgr2=0x7fab2c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:51.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab31adb640 1 --2- 172.21.15.148:0/2471509438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c1538a0 0x7fab2c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab31adb640 1 -- 172.21.15.148:0/2471509438 >> v1:172.21.15.148:6789/0 conn(0x7fab2c0ca0e0 legacy=0x7fab2c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:51.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab31adb640 1 -- 172.21.15.148:0/2471509438 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c0cad70 con 0x7fab2c0cf160 2023-10-30T13:48:51.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.305+0000 7fab23fff640 1 -- 172.21.15.148:0/2471509438 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934630976 0 0) 0x7fab2c0cb800 con 0x7fab2c0ca0e0 2023-10-30T13:48:51.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab31adb640 1 --2- 172.21.15.148:0/2471509438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c0cf160 0x7fab2c0cf590 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fab24009e50 tx=0x7fab2401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb70fdb7008a880 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:51.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab23fff640 1 -- 172.21.15.148:0/2471509438 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab240af400 con 0x7fab2c0cf160 2023-10-30T13:48:51.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab23fff640 1 -- 172.21.15.148:0/2471509438 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab240ab050 con 0x7fab2c0cf160 2023-10-30T13:48:51.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab23fff640 1 -- 172.21.15.148:0/2471509438 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab240af780 con 0x7fab2c0cf160 2023-10-30T13:48:51.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab3353e640 1 -- 172.21.15.148:0/2471509438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c0cf160 msgr2=0x7fab2c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:51.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab3353e640 1 --2- 172.21.15.148:0/2471509438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c0cf160 0x7fab2c0cf590 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fab24009e50 tx=0x7fab2401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab3353e640 1 -- 172.21.15.148:0/2471509438 shutdown_connections 2023-10-30T13:48:51.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab3353e640 1 --2- 172.21.15.148:0/2471509438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c0cf160 0x7fab2c0cf590 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab3353e640 1 --2- 172.21.15.148:0/2471509438 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c1538a0 0x7fab2c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab3353e640 1 -- 172.21.15.148:0/2471509438 >> 172.21.15.148:0/2471509438 conn(0x7fab2c06bfe0 msgr2=0x7fab2c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:51.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab3353e640 1 -- 172.21.15.148:0/2471509438 shutdown_connections 2023-10-30T13:48:51.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.309+0000 7fab3353e640 1 -- 172.21.15.148:0/2471509438 wait complete. 2023-10-30T13:48:51.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 Processor -- start 2023-10-30T13:48:51.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 -- start start 2023-10-30T13:48:51.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c0ca0e0 0x7fab2c0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:51.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c0cf160 0x7fab2c15bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:51.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c1538a0 0x7fab2c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:51.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c15b840 con 0x7fab2c1538a0 2023-10-30T13:48:51.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c15a950 con 0x7fab2c0ca0e0 2023-10-30T13:48:51.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c15ac30 con 0x7fab2c0cf160 2023-10-30T13:48:51.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab31adb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c1538a0 0x7fab2c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:51.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab31adb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c1538a0 0x7fab2c15c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53708/0 (socket says 172.21.15.148:53708) 2023-10-30T13:48:51.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab31adb640 1 -- 172.21.15.148:0/2497980727 learned_addr learned my addr 172.21.15.148:0/2497980727 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:51.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab312da640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c0ca0e0 0x7fab2c0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:51.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab30ad9640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c0cf160 0x7fab2c15bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:51.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab31adb640 1 -- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c0cf160 msgr2=0x7fab2c15bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:51.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab31adb640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c0cf160 0x7fab2c15bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab31adb640 1 -- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c0ca0e0 msgr2=0x7fab2c0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:51.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab31adb640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c0ca0e0 0x7fab2c0b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.316 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab31adb640 1 -- 172.21.15.148:0/2497980727 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab24009b30 con 0x7fab2c1538a0 2023-10-30T13:48:51.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab31adb640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c1538a0 0x7fab2c15c2a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fab2401ce20 tx=0x7fab2401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:51.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab21ffb640 1 -- 172.21.15.148:0/2497980727 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab240afc80 con 0x7fab2c1538a0 2023-10-30T13:48:51.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab21ffb640 1 -- 172.21.15.148:0/2497980727 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab240afde0 con 0x7fab2c1538a0 2023-10-30T13:48:51.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c15af10 con 0x7fab2c1538a0 2023-10-30T13:48:51.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab21ffb640 1 -- 172.21.15.148:0/2497980727 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab240ba660 con 0x7fab2c1538a0 2023-10-30T13:48:51.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c15b4d0 con 0x7fab2c1538a0 2023-10-30T13:48:51.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.313+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2c0b2a50 con 0x7fab2c1538a0 2023-10-30T13:48:51.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.317+0000 7fab21ffb640 1 -- 172.21.15.148:0/2497980727 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab240b6040 con 0x7fab2c1538a0 2023-10-30T13:48:51.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.317+0000 7fab21ffb640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab100664a0 0x7fab10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:51.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.317+0000 7fab21ffb640 1 -- 172.21.15.148:0/2497980727 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fab24122730 con 0x7fab2c1538a0 2023-10-30T13:48:51.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.317+0000 7fab312da640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab100664a0 0x7fab10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:51.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.317+0000 7fab312da640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab100664a0 0x7fab10068960 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fab28009ac0 tx=0x7fab28009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.321+0000 7fab21ffb640 1 -- 172.21.15.148:0/2497980727 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab240ef580 con 0x7fab2c1538a0 2023-10-30T13:48:51.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.501+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab2c04e170 con 0x7fab2c1538a0 2023-10-30T13:48:51.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.501+0000 7fab21ffb640 1 -- 172.21.15.148:0/2497980727 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5744 (secure 0 0 0) 0x7fab24009510 con 0x7fab2c1538a0 2023-10-30T13:48:51.503 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:51.503 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:42.607756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:51.504 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 148 2023-10-30T13:48:51.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab100664a0 msgr2=0x7fab10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:51.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab100664a0 0x7fab10068960 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fab28009ac0 tx=0x7fab28009190 comp rx=0 tx=0).stop 2023-10-30T13:48:51.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c1538a0 msgr2=0x7fab2c15c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:51.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c1538a0 0x7fab2c15c2a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fab2401ce20 tx=0x7fab2401bf90 comp rx=0 tx=0).stop 2023-10-30T13:48:51.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 shutdown_connections 2023-10-30T13:48:51.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fab100664a0 0x7fab10068960 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c1538a0 0x7fab2c15c2a0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.506 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fab2c0cf160 0x7fab2c15bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 --2- 172.21.15.148:0/2497980727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c0ca0e0 0x7fab2c0b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:51.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 >> 172.21.15.148:0/2497980727 conn(0x7fab2c06bfe0 msgr2=0x7fab2c1523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:51.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 shutdown_connections 2023-10-30T13:48:51.507 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:51.505+0000 7fab3353e640 1 -- 172.21.15.148:0/2497980727 wait complete. 2023-10-30T13:48:51.524 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 148, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 148, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7611": { "addr": "172.21.15.148:6835/1228827025", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1228827025, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1228827025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 7611, "incarnation": 133, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 10 }, "gid_7812": { "addr": "172.21.15.148:6837/1530445724", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 1530445724, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 1530445724, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7812, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:48:42.607756+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7812, "mds_1": 7611 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:48:53.525 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:53.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.673+0000 7fbcd2bc5640 1 Processor -- start 2023-10-30T13:48:53.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.673+0000 7fbcd2bc5640 1 -- start start 2023-10-30T13:48:53.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.673+0000 7fbcd2bc5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbccc0c6b80 0x7fbccc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:53.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.673+0000 7fbcd2bc5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbccc0c0a50 con 0x7fbccc13f4a0 2023-10-30T13:48:53.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.673+0000 7fbcd2bc5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbccc06f6e0 con 0x7fbccc0c6f60 2023-10-30T13:48:53.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.673+0000 7fbcd2bc5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc06f9c0 con 0x7fbccc0c6b80 2023-10-30T13:48:53.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.673+0000 7fbcd1162640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbccc13f4a0 0x7fbccc1566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45544/0 (socket says 172.21.15.148:45544) 2023-10-30T13:48:53.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.673+0000 7fbcd1162640 1 -- 172.21.15.148:0/1583686243 learned_addr learned my addr 172.21.15.148:0/1583686243 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:48:53.676 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcd0961640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbccc0c6b80 0x7fbccc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:53.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333136971 0 0) 0x7fbccc0c0a50 con 0x7fbccc13f4a0 2023-10-30T13:48:53.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb4003530 con 0x7fbccc13f4a0 2023-10-30T13:48:53.678 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2024215194 0 0) 0x7fbcb4003530 con 0x7fbccc13f4a0 2023-10-30T13:48:53.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbccc0c6b80 msgr2=0x7fbccc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:53.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbccc0c6b80 0x7fbccc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 >> v1:172.21.15.150:6789/0 conn(0x7fbccc0c6f60 legacy=0x7fbccc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:53.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc06fca0 con 0x7fbccc13f4a0 2023-10-30T13:48:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fbcc4001a00 con 0x7fbccc13f4a0 2023-10-30T13:48:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 >> v1:172.21.15.148:6789/0 conn(0x7fbccc13f4a0 legacy=0x7fbccc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb40039b0 0x7fbcb4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb4004310 0x7fbcb40047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:53.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcb4004ff0 0x7fbcb400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc06f9c0 con 0x7fbcb4004ff0 2023-10-30T13:48:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb400bf00 con 0x7fbcb4004310 2023-10-30T13:48:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb400c0c0 con 0x7fbcb40039b0 2023-10-30T13:48:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcd0961640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbccc0c6b80 0x7fbccc0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:48:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcd1162640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcb4004ff0 0x7fbcb400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:53.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcc3fff640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb4004310 0x7fbcb40047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcd0961640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb40039b0 0x7fbcb4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcd1162640 1 -- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb40039b0 msgr2=0x7fbcb4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcd1162640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb40039b0 0x7fbcb4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcd1162640 1 -- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb4004310 msgr2=0x7fbcb40047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcd1162640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb4004310 0x7fbcb40047f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.677+0000 7fbcd1162640 1 -- 172.21.15.148:0/1583686243 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb4003530 con 0x7fbcb4004ff0 2023-10-30T13:48:53.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd1162640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcb4004ff0 0x7fbcb400b990 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fbccc06f4f0 tx=0x7fbcc401f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56fa91b55d35fdf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:53.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc40ac670 con 0x7fbcb4004ff0 2023-10-30T13:48:53.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc40ac7f0 con 0x7fbcb4004ff0 2023-10-30T13:48:53.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcc37fe640 1 -- 172.21.15.148:0/1583686243 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc40acbe0 con 0x7fbcb4004ff0 2023-10-30T13:48:53.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/1583686243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcb4004ff0 msgr2=0x7fbcb400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:53.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd2bc5640 1 --2- 172.21.15.148:0/1583686243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcb4004ff0 0x7fbcb400b990 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fbccc06f4f0 tx=0x7fbcc401f9c0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd1162640 1 -- 172.21.15.148:0/1583686243 reap_dead start 2023-10-30T13:48:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/1583686243 shutdown_connections 2023-10-30T13:48:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/1583686243 >> 172.21.15.148:0/1583686243 conn(0x7fbccc0c7910 msgr2=0x7fbccc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/1583686243 shutdown_connections 2023-10-30T13:48:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/1583686243 wait complete. 2023-10-30T13:48:53.687 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd2bc5640 1 Processor -- start 2023-10-30T13:48:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.681+0000 7fbcd2bc5640 1 -- start start 2023-10-30T13:48:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd2bc5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb4004ff0 0x7fbccc1eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd2bc5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc157500 0x7fbccc155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd2bc5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc155530 0x7fbccc0af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:53.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd2bc5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0ae150 con 0x7fbccc157500 2023-10-30T13:48:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd2bc5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc1eea00 con 0x7fbccc155530 2023-10-30T13:48:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd2bc5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc154fc0 con 0x7fbcb4004ff0 2023-10-30T13:48:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc157500 0x7fbccc155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc157500 0x7fbccc155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39246/0 (socket says 172.21.15.148:39246) 2023-10-30T13:48:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc3fff640 1 -- 172.21.15.148:0/830316058 learned_addr learned my addr 172.21.15.148:0/830316058 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:53.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd0961640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb4004ff0 0x7fbccc1eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd1162640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc155530 0x7fbccc0af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc3fff640 1 -- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb4004ff0 msgr2=0x7fbccc1eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc3fff640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb4004ff0 0x7fbccc1eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc3fff640 1 -- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc155530 msgr2=0x7fbccc0af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:53.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc3fff640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc155530 0x7fbccc0af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc3fff640 1 -- 172.21.15.148:0/830316058 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc155280 con 0x7fbccc157500 2023-10-30T13:48:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd0961640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb4004ff0 0x7fbccc1eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:48:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc3fff640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc157500 0x7fbccc155a80 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc00f820 tx=0x7fbcbc00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc17fa640 1 -- 172.21.15.148:0/830316058 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc00fed0 con 0x7fbccc157500 2023-10-30T13:48:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc17fa640 1 -- 172.21.15.148:0/830316058 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcbc0052e0 con 0x7fbccc157500 2023-10-30T13:48:53.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc17fa640 1 -- 172.21.15.148:0/830316058 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc00d8f0 con 0x7fbccc157500 2023-10-30T13:48:53.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc1ee3b0 con 0x7fbccc157500 2023-10-30T13:48:53.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc0c1e90 con 0x7fbccc157500 2023-10-30T13:48:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc04e170 con 0x7fbccc157500 2023-10-30T13:48:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.685+0000 7fbcc17fa640 1 -- 172.21.15.148:0/830316058 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbcbc00da50 con 0x7fbccc157500 2023-10-30T13:48:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.689+0000 7fbcc17fa640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcb80664a0 0x7fbcb8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:53.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.689+0000 7fbcc17fa640 1 -- 172.21.15.148:0/830316058 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fbcbc084520 con 0x7fbccc157500 2023-10-30T13:48:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.689+0000 7fbcd0961640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcb80664a0 0x7fbcb8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.689+0000 7fbcd0961640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcb80664a0 0x7fbcb8068960 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fbccc06f1b0 tx=0x7fbcc8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:53.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.693+0000 7fbcc17fa640 1 -- 172.21.15.148:0/830316058 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbcbc0512b0 con 0x7fbccc157500 2023-10-30T13:48:53.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.873+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbccc0adcf0 con 0x7fbccc157500 2023-10-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.873+0000 7fbcc17fa640 1 -- 172.21.15.148:0/830316058 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+5738 (secure 0 0 0) 0x7fbcbc0548d0 con 0x7fbccc157500 2023-10-30T13:48:53.874 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:53.875 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:51.064345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:53.875 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 149 2023-10-30T13:48:53.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcb80664a0 msgr2=0x7fbcb8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:53.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcb80664a0 0x7fbcb8068960 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fbccc06f1b0 tx=0x7fbcc8008040 comp rx=0 tx=0).stop 2023-10-30T13:48:53.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc157500 msgr2=0x7fbccc155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:53.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc157500 0x7fbccc155a80 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc00f820 tx=0x7fbcbc00fce0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 shutdown_connections 2023-10-30T13:48:53.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcb80664a0 0x7fbcb8068960 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbccc157500 0x7fbccc155a80 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc155530 0x7fbccc0af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 --2- 172.21.15.148:0/830316058 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcb4004ff0 0x7fbccc1eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:53.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 >> 172.21.15.148:0/830316058 conn(0x7fbccc0c7910 msgr2=0x7fbccc1391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:53.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 shutdown_connections 2023-10-30T13:48:53.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:53.877+0000 7fbcd2bc5640 1 -- 172.21.15.148:0/830316058 wait complete. 2023-10-30T13:48:53.895 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=5453 2023-10-30T13:48:53.896 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2023-10-30T13:48:57.897 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:48:57.897 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:48:57.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:48:57.900 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:48:57.901 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:48:57.976 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2023-10-30T13:48:58.041 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865f165640 1 Processor -- start 2023-10-30T13:48:58.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865f165640 1 -- start start 2023-10-30T13:48:58.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865f165640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f865806f7a0 0x7f865806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:58.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865f165640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8658071110 0x7f8658158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:58.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865f165640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8658066cf0 con 0x7f86580c6bd0 2023-10-30T13:48:58.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865f165640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86580700c0 con 0x7f8658071110 2023-10-30T13:48:58.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865f165640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86580c70d0 con 0x7f865806f7a0 2023-10-30T13:48:58.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865d702640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8658071110 0x7f8658158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:58.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865d702640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8658071110 0x7f8658158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39252/0 (socket says 172.21.15.148:39252) 2023-10-30T13:48:58.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.045+0000 7f865d702640 1 -- 172.21.15.148:0/3875553034 learned_addr learned my addr 172.21.15.148:0/3875553034 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:58.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.041+0000 7f865cf01640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f865806f7a0 0x7f865806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:58.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.045+0000 7f865d702640 1 -- 172.21.15.148:0/3875553034 >> v2:172.21.15.150:3300/0 conn(0x7f865806f7a0 msgr2=0x7f865806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:48:58.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.045+0000 7f865d702640 1 --2- 172.21.15.148:0/3875553034 >> v2:172.21.15.150:3300/0 conn(0x7f865806f7a0 0x7f865806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.045+0000 7f865d702640 1 -- 172.21.15.148:0/3875553034 >> v1:172.21.15.148:6789/0 conn(0x7f86580c6bd0 legacy=0x7f8658069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:58.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.045+0000 7f865d702640 1 -- 172.21.15.148:0/3875553034 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580c7390 con 0x7f8658071110 2023-10-30T13:48:58.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.045+0000 7f865d702640 1 --2- 172.21.15.148:0/3875553034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8658071110 0x7f8658158110 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f865806c830 tx=0x7f865001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8faad97af4af2d77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:58.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.045+0000 7f86577fe640 1 -- 172.21.15.148:0/3875553034 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86500af400 con 0x7f8658071110 2023-10-30T13:48:58.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.045+0000 7f86577fe640 1 -- 172.21.15.148:0/3875553034 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86500ab050 con 0x7f8658071110 2023-10-30T13:48:58.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.045+0000 7f86577fe640 1 -- 172.21.15.148:0/3875553034 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86500af780 con 0x7f8658071110 2023-10-30T13:48:58.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 -- 172.21.15.148:0/3875553034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8658071110 msgr2=0x7f8658158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:58.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 --2- 172.21.15.148:0/3875553034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8658071110 0x7f8658158110 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f865806c830 tx=0x7f865001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 -- 172.21.15.148:0/3875553034 shutdown_connections 2023-10-30T13:48:58.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 --2- 172.21.15.148:0/3875553034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8658071110 0x7f8658158110 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 --2- 172.21.15.148:0/3875553034 >> v2:172.21.15.150:3300/0 conn(0x7f865806f7a0 0x7f865806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 -- 172.21.15.148:0/3875553034 >> 172.21.15.148:0/3875553034 conn(0x7f86580c7910 msgr2=0x7f86580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:58.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 -- 172.21.15.148:0/3875553034 shutdown_connections 2023-10-30T13:48:58.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 -- 172.21.15.148:0/3875553034 wait complete. 2023-10-30T13:48:58.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 Processor -- start 2023-10-30T13:48:58.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 -- start start 2023-10-30T13:48:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f865806f7a0 0x7f86581de210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8658071110 0x7f86581de750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86580c6bd0 0x7f86581ee8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8658157c30 con 0x7f86580c6bd0 2023-10-30T13:48:58.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86580c40a0 con 0x7f8658071110 2023-10-30T13:48:58.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865f165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865814d6c0 con 0x7f865806f7a0 2023-10-30T13:48:58.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865d702640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86580c6bd0 0x7f86581ee8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:58.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865d702640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86580c6bd0 0x7f86581ee8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39268/0 (socket says 172.21.15.148:39268) 2023-10-30T13:48:58.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865d702640 1 -- 172.21.15.148:0/1300148312 learned_addr learned my addr 172.21.15.148:0/1300148312 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:48:58.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f865cf01640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f865806f7a0 0x7f86581de210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.049+0000 7f8657fff640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8658071110 0x7f86581de750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f865d702640 1 -- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f865806f7a0 msgr2=0x7f86581de210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f865d702640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f865806f7a0 0x7f86581de210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f865d702640 1 -- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8658071110 msgr2=0x7f86581de750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:58.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f865d702640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8658071110 0x7f86581de750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f865d702640 1 -- 172.21.15.148:0/1300148312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8650009b30 con 0x7f86580c6bd0 2023-10-30T13:48:58.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f865d702640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86580c6bd0 0x7f86581ee8d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f86500b2040 tx=0x7f865001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:58.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f86557fa640 1 -- 172.21.15.148:0/1300148312 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86500afcb0 con 0x7f86580c6bd0 2023-10-30T13:48:58.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8658157e80 con 0x7f86580c6bd0 2023-10-30T13:48:58.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86581e3ff0 con 0x7f86580c6bd0 2023-10-30T13:48:58.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f86557fa640 1 -- 172.21.15.148:0/1300148312 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8650009510 con 0x7f86580c6bd0 2023-10-30T13:48:58.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f86557fa640 1 -- 172.21.15.148:0/1300148312 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86500c6660 con 0x7f86580c6bd0 2023-10-30T13:48:58.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865804e170 con 0x7f86580c6bd0 2023-10-30T13:48:58.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.053+0000 7f86557fa640 1 -- 172.21.15.148:0/1300148312 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f86500b6040 con 0x7f86580c6bd0 2023-10-30T13:48:58.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.057+0000 7f86557fa640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f863c0664a0 0x7f863c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:48:58.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.057+0000 7f86557fa640 1 -- 172.21.15.148:0/1300148312 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f8650122570 con 0x7f86580c6bd0 2023-10-30T13:48:58.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.057+0000 7f865cf01640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f863c0664a0 0x7f863c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:48:58.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.057+0000 7f865cf01640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f863c0664a0 0x7f863c068960 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ac0 tx=0x7f864c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:48:58.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.061+0000 7f86557fa640 1 -- 172.21.15.148:0/1300148312 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f86500ef340 con 0x7f86580c6bd0 2023-10-30T13:48:58.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.245+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86580c7f90 con 0x7f86580c6bd0 2023-10-30T13:48:58.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f86557fa640 1 -- 172.21.15.148:0/1300148312 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6533 (secure 0 0 0) 0x7f86500f2960 con 0x7f86580c6bd0 2023-10-30T13:48:58.248 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:48:58.248 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:48:51.064345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:48:58.248 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 150 2023-10-30T13:48:58.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f863c0664a0 msgr2=0x7f863c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:58.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f863c0664a0 0x7f863c068960 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f864c009ac0 tx=0x7f864c009190 comp rx=0 tx=0).stop 2023-10-30T13:48:58.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86580c6bd0 msgr2=0x7f86581ee8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:48:58.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86580c6bd0 0x7f86581ee8d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f86500b2040 tx=0x7f865001bf60 comp rx=0 tx=0).stop 2023-10-30T13:48:58.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 shutdown_connections 2023-10-30T13:48:58.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f863c0664a0 0x7f863c068960 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86580c6bd0 0x7f86581ee8d0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8658071110 0x7f86581de750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 --2- 172.21.15.148:0/1300148312 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f865806f7a0 0x7f86581de210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:48:58.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 >> 172.21.15.148:0/1300148312 conn(0x7f86580c7910 msgr2=0x7f86580c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:48:58.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 shutdown_connections 2023-10-30T13:48:58.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:48:58.249+0000 7f865f165640 1 -- 172.21.15.148:0/1300148312 wait complete. 2023-10-30T13:48:58.268 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:48:58.268 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before thrashing 2023-10-30T13:48:59.012 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 60% (106/176) Updating files: 61% (108/176) Updating files: 62% (110/176) Updating files: 63% (111/176) Updating files: 64% (113/176) Updating files: 65% (115/176) Updating files: 66% (117/176) Updating files: 67% (118/176) Updating files: 68% (120/176) Updating files: 69% (122/176) Updating files: 70% (124/176) Updating files: 71% (125/176) Updating files: 72% (127/176) Updating files: 73% (129/176) Updating files: 74% (131/176) Updating files: 75% (132/176) Updating files: 75% (133/176) Updating files: 76% (134/176) Updating files: 76% (135/176) Updating files: 77% (136/176) Updating files: 78% (138/176) Updating files: 79% (140/176) Updating files: 80% (141/176) Updating files: 81% (143/176) Updating files: 82% (145/176) Updating files: 83% (147/176) Updating files: 84% (148/176) Updating files: 85% (150/176) Updating files: 86% (152/176) Updating files: 87% (154/176) Updating files: 88% (155/176) Updating files: 89% (157/176) Updating files: 90% (159/176) Updating files: 91% (161/176) Updating files: 92% (162/176) Updating files: 93% (164/176) Updating files: 94% (166/176) Updating files: 95% (168/176) Updating files: 96% (169/176) Updating files: 97% (171/176) Updating files: 98% (173/176) Updating files: 99% (175/176) Updating files: 100% (176/176) Updating files: 100% (176/176), done. 2023-10-30T13:48:59.049 INFO:tasks.ceph.mds.c.smithi148.stderr:2023-10-30T13:48:59.049+0000 7f2b9caac640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:48:59.049 INFO:tasks.ceph.mds.c.smithi148.stderr:2023-10-30T13:48:59.049+0000 7f2b9caac640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:49:00.031 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at b78644e7dee 0.80 2023-10-30T13:49:00.037 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.80 2023-10-30T13:49:00.072 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 81 -eq 48 ] 2023-10-30T13:49:00.072 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.81 2023-10-30T13:49:00.073 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.81 2023-10-30T13:49:00.073 INFO:tasks.workunit.client.0.smithi148.stdout:v0.81 2023-10-30T13:49:00.073 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.81 2023-10-30T13:49:02.269 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:02.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c5349a640 1 Processor -- start 2023-10-30T13:49:02.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c5349a640 1 -- start start 2023-10-30T13:49:02.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c5349a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 0x7f1c4c070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c5349a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1c4c0c6540 0x7f1c4c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c5349a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1c4c069e90 0x7f1c4c06a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c5349a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c4c0c5810 con 0x7f1c4c0c6540 2023-10-30T13:49:02.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c5349a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c4c0c6960 con 0x7f1c4c069e90 2023-10-30T13:49:02.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c5349a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c0c6bb0 con 0x7f1c4c1580e0 2023-10-30T13:49:02.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c50a35640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1c4c0c6540 0x7f1c4c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c50a35640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1c4c0c6540 0x7f1c4c15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37868/0 (socket says 172.21.15.148:37868) 2023-10-30T13:49:02.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c50a35640 1 -- 172.21.15.148:0/2994503168 learned_addr learned my addr 172.21.15.148:0/2994503168 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:02.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c51236640 1 --2- 172.21.15.148:0/2994503168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 0x7f1c4c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c51a37640 1 --2- 172.21.15.148:0/2994503168 >> v2:172.21.15.150:3300/0 conn(0x7f1c4c069e90 0x7f1c4c06a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c50a35640 1 -- 172.21.15.148:0/2994503168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 msgr2=0x7f1c4c070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:02.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c50a35640 1 --2- 172.21.15.148:0/2994503168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 0x7f1c4c070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c50a35640 1 -- 172.21.15.148:0/2994503168 >> v2:172.21.15.150:3300/0 conn(0x7f1c4c069e90 msgr2=0x7f1c4c06a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:02.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c50a35640 1 --2- 172.21.15.148:0/2994503168 >> v2:172.21.15.150:3300/0 conn(0x7f1c4c069e90 0x7f1c4c06a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c50a35640 1 -- 172.21.15.148:0/2994503168 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c0c6e70 con 0x7f1c4c0c6540 2023-10-30T13:49:02.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.428+0000 7f1c51a37640 1 --2- 172.21.15.148:0/2994503168 >> v2:172.21.15.150:3300/0 conn(0x7f1c4c069e90 0x7f1c4c06a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:02.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c50a35640 1 --2- 172.21.15.148:0/2994503168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c0c6540 0x7f1c4c15e700 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c0c54b0 tx=0x7f1c3c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915a856bd67335dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:02.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c43fff640 1 -- 172.21.15.148:0/2994503168 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c0ac630 con 0x7f1c4c0c6540 2023-10-30T13:49:02.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c43fff640 1 -- 172.21.15.148:0/2994503168 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c0ac790 con 0x7f1c4c0c6540 2023-10-30T13:49:02.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c43fff640 1 -- 172.21.15.148:0/2994503168 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c0acb20 con 0x7f1c4c0c6540 2023-10-30T13:49:02.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c5349a640 1 -- 172.21.15.148:0/2994503168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c0c6540 msgr2=0x7f1c4c15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:02.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c5349a640 1 --2- 172.21.15.148:0/2994503168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c0c6540 0x7f1c4c15e700 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c0c54b0 tx=0x7f1c3c01c500 comp rx=0 tx=0).stop 2023-10-30T13:49:02.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c5349a640 1 -- 172.21.15.148:0/2994503168 shutdown_connections 2023-10-30T13:49:02.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c5349a640 1 --2- 172.21.15.148:0/2994503168 >> v2:172.21.15.150:3300/0 conn(0x7f1c4c069e90 0x7f1c4c06a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c5349a640 1 --2- 172.21.15.148:0/2994503168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c0c6540 0x7f1c4c15e700 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c5349a640 1 --2- 172.21.15.148:0/2994503168 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 0x7f1c4c070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c5349a640 1 -- 172.21.15.148:0/2994503168 >> 172.21.15.148:0/2994503168 conn(0x7f1c4c0c7910 msgr2=0x7f1c4c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:02.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c5349a640 1 -- 172.21.15.148:0/2994503168 shutdown_connections 2023-10-30T13:49:02.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.432+0000 7f1c5349a640 1 -- 172.21.15.148:0/2994503168 wait complete. 2023-10-30T13:49:02.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 Processor -- start 2023-10-30T13:49:02.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 -- start start 2023-10-30T13:49:02.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c069e90 0x7f1c4c1fa490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c4c0c6540 0x7f1c4c0c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 0x7f1c4c149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c0c4640 con 0x7f1c4c069e90 2023-10-30T13:49:02.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c0c2260 con 0x7f1c4c0c6540 2023-10-30T13:49:02.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c1fa9d0 con 0x7f1c4c1580e0 2023-10-30T13:49:02.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51236640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c069e90 0x7f1c4c1fa490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51236640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c069e90 0x7f1c4c1fa490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37884/0 (socket says 172.21.15.148:37884) 2023-10-30T13:49:02.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51236640 1 -- 172.21.15.148:0/1678152088 learned_addr learned my addr 172.21.15.148:0/1678152088 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:02.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51a37640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 0x7f1c4c149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c50a35640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c4c0c6540 0x7f1c4c0c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51236640 1 -- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 msgr2=0x7f1c4c149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:02.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51236640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 0x7f1c4c149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51236640 1 -- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c4c0c6540 msgr2=0x7f1c4c0c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:02.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51236640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c4c0c6540 0x7f1c4c0c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51236640 1 -- 172.21.15.148:0/1678152088 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c0094d0 con 0x7f1c4c069e90 2023-10-30T13:49:02.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c51236640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c069e90 0x7f1c4c1fa490 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800e8a0 tx=0x7f1c4800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:02.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c41ffb640 1 -- 172.21.15.148:0/1678152088 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48009510 con 0x7f1c4c069e90 2023-10-30T13:49:02.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c41ffb640 1 -- 172.21.15.148:0/1678152088 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4800c940 con 0x7f1c4c069e90 2023-10-30T13:49:02.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c41ffb640 1 -- 172.21.15.148:0/1678152088 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4800ccb0 con 0x7f1c4c069e90 2023-10-30T13:49:02.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4c0c2490 con 0x7f1c4c069e90 2023-10-30T13:49:02.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4c160c50 con 0x7f1c4c069e90 2023-10-30T13:49:02.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.436+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4c04e170 con 0x7f1c4c069e90 2023-10-30T13:49:02.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.440+0000 7f1c41ffb640 1 -- 172.21.15.148:0/1678152088 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c48013470 con 0x7f1c4c069e90 2023-10-30T13:49:02.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.440+0000 7f1c41ffb640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c300664a0 0x7f1c30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.440+0000 7f1c41ffb640 1 -- 172.21.15.148:0/1678152088 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f1c480846c0 con 0x7f1c4c069e90 2023-10-30T13:49:02.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.440+0000 7f1c50a35640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c300664a0 0x7f1c30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.440+0000 7f1c50a35640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c300664a0 0x7f1c30068960 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0ae000 tx=0x7f1c3c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:02.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.444+0000 7f1c41ffb640 1 -- 172.21.15.148:0/1678152088 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c480513d0 con 0x7f1c4c069e90 2023-10-30T13:49:02.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.624+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c4c0c5d10 con 0x7f1c4c069e90 2023-10-30T13:49:02.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.624+0000 7f1c41ffb640 1 -- 172.21.15.148:0/1678152088 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f1c480549f0 con 0x7f1c4c069e90 2023-10-30T13:49:02.627 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:02.628 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:02.069202+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:02.628 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 151 2023-10-30T13:49:02.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c300664a0 msgr2=0x7f1c30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:02.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c300664a0 0x7f1c30068960 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c0ae000 tx=0x7f1c3c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:49:02.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c069e90 msgr2=0x7f1c4c1fa490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:02.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c069e90 0x7f1c4c1fa490 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800e8a0 tx=0x7f1c4800eca0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 shutdown_connections 2023-10-30T13:49:02.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1c300664a0 0x7f1c30068960 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1c4c1580e0 0x7f1c4c149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c4c0c6540 0x7f1c4c0c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 --2- 172.21.15.148:0/1678152088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c4c069e90 0x7f1c4c1fa490 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 >> 172.21.15.148:0/1678152088 conn(0x7f1c4c0c7910 msgr2=0x7f1c4c154540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:02.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 shutdown_connections 2023-10-30T13:49:02.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.628+0000 7f1c5349a640 1 -- 172.21.15.148:0/1678152088 wait complete. 2023-10-30T13:49:02.648 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-10-30T13:49:02.648 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:49:02.648 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:49:02.676 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:49:02.677 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:02.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.856+0000 7fee1bf0f640 1 Processor -- start 2023-10-30T13:49:02.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.856+0000 7fee1bf0f640 1 -- start start 2023-10-30T13:49:02.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.856+0000 7fee1bf0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee14151800 0x7fee1406e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.856+0000 7fee1bf0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c1ee0 0x7fee140c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.856+0000 7fee1bf0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee140c9230 0x7fee140cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.856+0000 7fee1bf0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1406dc70 con 0x7fee14151800 2023-10-30T13:49:02.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee1bf0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1406e090 con 0x7fee140c9230 2023-10-30T13:49:02.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee1bf0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee140c8a50 con 0x7fee140c1ee0 2023-10-30T13:49:02.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee19cab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee14151800 0x7fee1406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee19cab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fee14151800 0x7fee1406e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37896/0 (socket says 172.21.15.148:37896) 2023-10-30T13:49:02.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee19cab640 1 -- 172.21.15.148:0/372583486 learned_addr learned my addr 172.21.15.148:0/372583486 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:02.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee194aa640 1 --2- 172.21.15.148:0/372583486 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c1ee0 0x7fee140c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee1a4ac640 1 --2- 172.21.15.148:0/372583486 >> v2:172.21.15.150:3300/0 conn(0x7fee140c9230 0x7fee140cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee19cab640 1 -- 172.21.15.148:0/372583486 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c1ee0 msgr2=0x7fee140c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:02.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee19cab640 1 --2- 172.21.15.148:0/372583486 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c1ee0 0x7fee140c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee19cab640 1 -- 172.21.15.148:0/372583486 >> v2:172.21.15.150:3300/0 conn(0x7fee140c9230 msgr2=0x7fee140cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:02.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee19cab640 1 --2- 172.21.15.148:0/372583486 >> v2:172.21.15.150:3300/0 conn(0x7fee140c9230 0x7fee140cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee19cab640 1 -- 172.21.15.148:0/372583486 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee140c3b10 con 0x7fee14151800 2023-10-30T13:49:02.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee19cab640 1 --2- 172.21.15.148:0/372583486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee14151800 0x7fee1406e470 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fee140c7480 tx=0x7fee1001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f10341a4592f54 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:02.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee18ca9640 1 -- 172.21.15.148:0/372583486 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100aa5f0 con 0x7fee14151800 2023-10-30T13:49:02.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee18ca9640 1 -- 172.21.15.148:0/372583486 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee100aa750 con 0x7fee14151800 2023-10-30T13:49:02.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.860+0000 7fee18ca9640 1 -- 172.21.15.148:0/372583486 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100aaae0 con 0x7fee14151800 2023-10-30T13:49:02.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 -- 172.21.15.148:0/372583486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee14151800 msgr2=0x7fee1406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:02.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/372583486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee14151800 0x7fee1406e470 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fee140c7480 tx=0x7fee1001c4d0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 -- 172.21.15.148:0/372583486 shutdown_connections 2023-10-30T13:49:02.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/372583486 >> v2:172.21.15.150:3300/0 conn(0x7fee140c9230 0x7fee140cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/372583486 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c1ee0 0x7fee140c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/372583486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee14151800 0x7fee1406e470 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 -- 172.21.15.148:0/372583486 >> 172.21.15.148:0/372583486 conn(0x7fee14071ea0 msgr2=0x7fee14065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:02.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 -- 172.21.15.148:0/372583486 shutdown_connections 2023-10-30T13:49:02.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 -- 172.21.15.148:0/372583486 wait complete. 2023-10-30T13:49:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 Processor -- start 2023-10-30T13:49:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 -- start start 2023-10-30T13:49:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee140c1ee0 0x7fee14149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c9230 0x7fee141f08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee14151800 0x7fee141f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee141f0490 con 0x7fee140c1ee0 2023-10-30T13:49:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee14154710 con 0x7fee14151800 2023-10-30T13:49:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1bf0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee141549f0 con 0x7fee140c9230 2023-10-30T13:49:02.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee19cab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee140c1ee0 0x7fee14149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee19cab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee140c1ee0 0x7fee14149a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37904/0 (socket says 172.21.15.148:37904) 2023-10-30T13:49:02.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee19cab640 1 -- 172.21.15.148:0/1603548581 learned_addr learned my addr 172.21.15.148:0/1603548581 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:02.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee1a4ac640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee14151800 0x7fee141f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee194aa640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c9230 0x7fee141f08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee19cab640 1 -- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c9230 msgr2=0x7fee141f08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:02.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee19cab640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c9230 0x7fee141f08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee19cab640 1 -- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee14151800 msgr2=0x7fee141f0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:02.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee19cab640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee14151800 0x7fee141f0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:02.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee19cab640 1 -- 172.21.15.148:0/1603548581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee100094d0 con 0x7fee140c1ee0 2023-10-30T13:49:02.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.864+0000 7fee19cab640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee140c1ee0 0x7fee14149a00 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fee1001ab80 tx=0x7fee1001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:02.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee0a7fc640 1 -- 172.21.15.148:0/1603548581 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100b3800 con 0x7fee140c1ee0 2023-10-30T13:49:02.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee0a7fc640 1 -- 172.21.15.148:0/1603548581 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee100b3980 con 0x7fee140c1ee0 2023-10-30T13:49:02.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee0a7fc640 1 -- 172.21.15.148:0/1603548581 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee100b3d30 con 0x7fee140c1ee0 2023-10-30T13:49:02.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee14154cd0 con 0x7fee140c1ee0 2023-10-30T13:49:02.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee14155290 con 0x7fee140c1ee0 2023-10-30T13:49:02.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1404e170 con 0x7fee140c1ee0 2023-10-30T13:49:02.875 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee0a7fc640 1 -- 172.21.15.148:0/1603548581 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee100b2380 con 0x7fee140c1ee0 2023-10-30T13:49:02.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee0a7fc640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fedf80664a0 0x7fedf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:02.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee0a7fc640 1 -- 172.21.15.148:0/1603548581 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fee100a7070 con 0x7fee140c1ee0 2023-10-30T13:49:02.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.868+0000 7fee194aa640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fedf80664a0 0x7fedf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:02.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.872+0000 7fee194aa640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fedf80664a0 0x7fedf8068960 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fee141edf50 tx=0x7fee04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:02.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:02.872+0000 7fee0a7fc640 1 -- 172.21.15.148:0/1603548581 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee100eb8f0 con 0x7fee140c1ee0 2023-10-30T13:49:03.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee140c6de0 con 0x7fee140c1ee0 2023-10-30T13:49:03.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee0a7fc640 1 -- 172.21.15.148:0/1603548581 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6534 (secure 0 0 0) 0x7fee100c3090 con 0x7fee140c1ee0 2023-10-30T13:49:03.057 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:03.057 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:02.069202+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:03.058 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 151 2023-10-30T13:49:03.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fedf80664a0 msgr2=0x7fedf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:03.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fedf80664a0 0x7fedf8068960 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fee141edf50 tx=0x7fee04008040 comp rx=0 tx=0).stop 2023-10-30T13:49:03.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee140c1ee0 msgr2=0x7fee14149a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:03.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee140c1ee0 0x7fee14149a00 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fee1001ab80 tx=0x7fee1001ad10 comp rx=0 tx=0).stop 2023-10-30T13:49:03.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 shutdown_connections 2023-10-30T13:49:03.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fedf80664a0 0x7fedf8068960 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:03.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee14151800 0x7fee141f0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:03.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fee140c9230 0x7fee141f08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:03.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 --2- 172.21.15.148:0/1603548581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fee140c1ee0 0x7fee14149a00 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:03.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 >> 172.21.15.148:0/1603548581 conn(0x7fee14071ea0 msgr2=0x7fee14147ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:03.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 shutdown_connections 2023-10-30T13:49:03.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:03.056+0000 7fee1bf0f640 1 -- 172.21.15.148:0/1603548581 wait complete. 2023-10-30T13:49:03.078 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:03.368 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:03.372 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:05.079 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:05.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa7884d6640 1 Processor -- start 2023-10-30T13:49:05.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa7884d6640 1 -- start start 2023-10-30T13:49:05.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa7884d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa78015d6f0 0x7fa78015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:05.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa7884d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7800cf160 0x7fa7800cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:05.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa7884d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7800cb800 con 0x7fa7801538a0 2023-10-30T13:49:05.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa7884d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7800cb330 con 0x7fa78015d6f0 2023-10-30T13:49:05.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa7884d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800cad70 con 0x7fa7800cf160 2023-10-30T13:49:05.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa786272640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa7801538a0 0x7fa7800ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59640/0 (socket says 172.21.15.148:59640) 2023-10-30T13:49:05.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa786272640 1 -- 172.21.15.148:0/2956531262 learned_addr learned my addr 172.21.15.148:0/2956531262 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:05.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa786a73640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7800cf160 0x7fa7800cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:05.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785a71640 1 --2- 172.21.15.148:0/2956531262 >> v2:172.21.15.150:3300/0 conn(0x7fa78015d6f0 0x7fa78015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:05.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130781730 0 0) 0x7fa7800cb800 con 0x7fa7801538a0 2023-10-30T13:49:05.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c003530 con 0x7fa7801538a0 2023-10-30T13:49:05.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3100425371 0 0) 0x7fa76c003530 con 0x7fa7801538a0 2023-10-30T13:49:05.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7800cf160 msgr2=0x7fa7800cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:05.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7800cf160 0x7fa7800cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 >> v2:172.21.15.150:3300/0 conn(0x7fa78015d6f0 msgr2=0x7fa78015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:05.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 --2- 172.21.15.148:0/2956531262 >> v2:172.21.15.150:3300/0 conn(0x7fa78015d6f0 0x7fa78015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800cb050 con 0x7fa7801538a0 2023-10-30T13:49:05.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fa77c001680 con 0x7fa7801538a0 2023-10-30T13:49:05.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 >> v1:172.21.15.148:6789/0 conn(0x7fa7801538a0 legacy=0x7fa7800ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:05.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76c003980 0x7fa76c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:05.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76c0047d0 0x7fa76c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:05.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7800cf160 0x7fa76c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:05.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800cb330 con 0x7fa76c0047d0 2023-10-30T13:49:05.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800cad70 con 0x7fa76c003980 2023-10-30T13:49:05.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c00b990 con 0x7fa7800cf160 2023-10-30T13:49:05.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa785a71640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76c0047d0 0x7fa76c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:05.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.228+0000 7fa786a73640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7800cf160 0x7fa76c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:05.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa786272640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76c003980 0x7fa76c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:05.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa785a71640 1 -- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7800cf160 msgr2=0x7fa76c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:05.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa785a71640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7800cf160 0x7fa76c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa785a71640 1 -- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76c003980 msgr2=0x7fa76c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:05.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa785a71640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa76c003980 0x7fa76c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa785a71640 1 -- 172.21.15.148:0/2956531262 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c003530 con 0x7fa76c0047d0 2023-10-30T13:49:05.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa785a71640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76c0047d0 0x7fa76c004c50 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fa770009ae0 tx=0x7fa77001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10353e43ee69e41 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:05.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7700a55a0 con 0x7fa76c0047d0 2023-10-30T13:49:05.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7700a5700 con 0x7fa76c0047d0 2023-10-30T13:49:05.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa785270640 1 -- 172.21.15.148:0/2956531262 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7700a5ab0 con 0x7fa76c0047d0 2023-10-30T13:49:05.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa7884d6640 1 -- 172.21.15.148:0/2956531262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76c0047d0 msgr2=0x7fa76c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:05.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa7884d6640 1 --2- 172.21.15.148:0/2956531262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa76c0047d0 0x7fa76c004c50 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fa770009ae0 tx=0x7fa77001ca90 comp rx=0 tx=0).stop 2023-10-30T13:49:05.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.232+0000 7fa786a73640 1 -- 172.21.15.148:0/2956531262 reap_dead start 2023-10-30T13:49:05.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 -- 172.21.15.148:0/2956531262 shutdown_connections 2023-10-30T13:49:05.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 -- 172.21.15.148:0/2956531262 >> 172.21.15.148:0/2956531262 conn(0x7fa78006bfe0 msgr2=0x7fa78006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:05.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 -- 172.21.15.148:0/2956531262 shutdown_connections 2023-10-30T13:49:05.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 -- 172.21.15.148:0/2956531262 wait complete. 2023-10-30T13:49:05.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 Processor -- start 2023-10-30T13:49:05.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 -- start start 2023-10-30T13:49:05.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0047d0 0x7fa78015bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:05.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa78015c260 0x7fa780158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:05.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa780159060 0x7fa7801f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:05.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800cd0b0 con 0x7fa78015c260 2023-10-30T13:49:05.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78015b2c0 con 0x7fa780159060 2023-10-30T13:49:05.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa7884d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7801efd80 con 0x7fa76c0047d0 2023-10-30T13:49:05.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa785a71640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa78015c260 0x7fa780158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:05.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa785a71640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa78015c260 0x7fa780158b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37928/0 (socket says 172.21.15.148:37928) 2023-10-30T13:49:05.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa785a71640 1 -- 172.21.15.148:0/3251206372 learned_addr learned my addr 172.21.15.148:0/3251206372 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:05.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa786272640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0047d0 0x7fa78015bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:05.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa786a73640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa780159060 0x7fa7801f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:05.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa785a71640 1 -- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0047d0 msgr2=0x7fa78015bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:05.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa785a71640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0047d0 0x7fa78015bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa785a71640 1 -- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa780159060 msgr2=0x7fa7801f2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:05.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa785a71640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa780159060 0x7fa7801f2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa785a71640 1 -- 172.21.15.148:0/3251206372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78015a920 con 0x7fa78015c260 2023-10-30T13:49:05.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa786a73640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa780159060 0x7fa7801f2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:05.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.236+0000 7fa785a71640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa78015c260 0x7fa780158b20 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fa77001c8c0 tx=0x7fa76c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:05.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa776ffd640 1 -- 172.21.15.148:0/3251206372 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa770004220 con 0x7fa78015c260 2023-10-30T13:49:05.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa776ffd640 1 -- 172.21.15.148:0/3251206372 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa770004380 con 0x7fa78015c260 2023-10-30T13:49:05.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa776ffd640 1 -- 172.21.15.148:0/3251206372 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7700ad850 con 0x7fa78015c260 2023-10-30T13:49:05.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78015c6a0 con 0x7fa78015c260 2023-10-30T13:49:05.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78015b510 con 0x7fa78015c260 2023-10-30T13:49:05.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78015b700 con 0x7fa78015c260 2023-10-30T13:49:05.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa776ffd640 1 -- 172.21.15.148:0/3251206372 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7700b4040 con 0x7fa78015c260 2023-10-30T13:49:05.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa776ffd640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7640664d0 0x7fa764068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:05.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa776ffd640 1 -- 172.21.15.148:0/3251206372 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7fa77011eb50 con 0x7fa78015c260 2023-10-30T13:49:05.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.240+0000 7fa786272640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7640664d0 0x7fa764068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:05.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.244+0000 7fa786272640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7640664d0 0x7fa764068990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fa77c005b60 tx=0x7fa77c007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:05.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.244+0000 7fa776ffd640 1 -- 172.21.15.148:0/3251206372 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7700eba30 con 0x7fa78015c260 2023-10-30T13:49:05.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.428+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa78004e170 con 0x7fa78015c260 2023-10-30T13:49:05.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.428+0000 7fa776ffd640 1 -- 172.21.15.148:0/3251206372 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6534 (secure 0 0 0) 0x7fa7700ef050 con 0x7fa78015c260 2023-10-30T13:49:05.430 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:05.430 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:02.069202+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:05.430 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 151 2023-10-30T13:49:05.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7640664d0 msgr2=0x7fa764068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:05.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7640664d0 0x7fa764068990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fa77c005b60 tx=0x7fa77c007040 comp rx=0 tx=0).stop 2023-10-30T13:49:05.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa78015c260 msgr2=0x7fa780158b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:05.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa78015c260 0x7fa780158b20 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fa77001c8c0 tx=0x7fa76c005800 comp rx=0 tx=0).stop 2023-10-30T13:49:05.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 shutdown_connections 2023-10-30T13:49:05.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7640664d0 0x7fa764068990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa78015c260 0x7fa780158b20 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa780159060 0x7fa7801f2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 --2- 172.21.15.148:0/3251206372 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa76c0047d0 0x7fa78015bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:05.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 >> 172.21.15.148:0/3251206372 conn(0x7fa78006bfe0 msgr2=0x7fa780151870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:05.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 shutdown_connections 2023-10-30T13:49:05.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:05.432+0000 7fa7884d6640 1 -- 172.21.15.148:0/3251206372 wait complete. 2023-10-30T13:49:05.451 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:07.452 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:07.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f762113e640 1 Processor -- start 2023-10-30T13:49:07.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f762113e640 1 -- start start 2023-10-30T13:49:07.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f762113e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f761c0cb550 0x7f761c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f762113e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f761c071250 0x7f761c071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f762113e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c1560a0 con 0x7f761c14fd50 2023-10-30T13:49:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f762113e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761c156380 con 0x7f761c071250 2023-10-30T13:49:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f762113e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761c156660 con 0x7f761c0cb550 2023-10-30T13:49:07.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f761ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f761c14fd50 0x7f761c0c65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59650/0 (socket says 172.21.15.148:59650) 2023-10-30T13:49:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f761ad9d640 1 -- 172.21.15.148:0/3193813984 learned_addr learned my addr 172.21.15.148:0/3193813984 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.616+0000 7f761b59e640 1 --2- 172.21.15.148:0/3193813984 >> v2:172.21.15.148:3300/0 conn(0x7f761c071250 0x7f761c071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f761a59c640 1 --2- 172.21.15.148:0/3193813984 >> v2:172.21.15.150:3300/0 conn(0x7f761c0cb550 0x7f761c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f7619d9b640 1 -- 172.21.15.148:0/3193813984 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969896830 0 0) 0x7f761c1560a0 con 0x7f761c14fd50 2023-10-30T13:49:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f7619d9b640 1 -- 172.21.15.148:0/3193813984 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7604005740 con 0x7f761c14fd50 2023-10-30T13:49:07.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f761b59e640 1 -- 172.21.15.148:0/3193813984 >> v2:172.21.15.150:3300/0 conn(0x7f761c0cb550 msgr2=0x7f761c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:07.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f761b59e640 1 --2- 172.21.15.148:0/3193813984 >> v2:172.21.15.150:3300/0 conn(0x7f761c0cb550 0x7f761c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f761b59e640 1 -- 172.21.15.148:0/3193813984 >> v1:172.21.15.148:6789/0 conn(0x7f761c14fd50 legacy=0x7f761c0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:07.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f761b59e640 1 -- 172.21.15.148:0/3193813984 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c156940 con 0x7f761c071250 2023-10-30T13:49:07.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f761b59e640 1 --2- 172.21.15.148:0/3193813984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 0x7f761c071650 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f761c06ac30 tx=0x7f761401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd93ef5ec05ce08 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:07.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f7619d9b640 1 -- 172.21.15.148:0/3193813984 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76140a5910 con 0x7f761c071250 2023-10-30T13:49:07.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f7619d9b640 1 -- 172.21.15.148:0/3193813984 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76140a5a70 con 0x7f761c071250 2023-10-30T13:49:07.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f7619d9b640 1 -- 172.21.15.148:0/3193813984 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76140a5e00 con 0x7f761c071250 2023-10-30T13:49:07.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f762113e640 1 -- 172.21.15.148:0/3193813984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 msgr2=0x7f761c071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:07.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.620+0000 7f762113e640 1 --2- 172.21.15.148:0/3193813984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 0x7f761c071650 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f761c06ac30 tx=0x7f761401c910 comp rx=0 tx=0).stop 2023-10-30T13:49:07.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 -- 172.21.15.148:0/3193813984 shutdown_connections 2023-10-30T13:49:07.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 --2- 172.21.15.148:0/3193813984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 0x7f761c071650 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 --2- 172.21.15.148:0/3193813984 >> v2:172.21.15.150:3300/0 conn(0x7f761c0cb550 0x7f761c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 -- 172.21.15.148:0/3193813984 >> 172.21.15.148:0/3193813984 conn(0x7f761c06c6e0 msgr2=0x7f761c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:07.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 -- 172.21.15.148:0/3193813984 shutdown_connections 2023-10-30T13:49:07.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 -- 172.21.15.148:0/3193813984 wait complete. 2023-10-30T13:49:07.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 Processor -- start 2023-10-30T13:49:07.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 -- start start 2023-10-30T13:49:07.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 0x7f761c13a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:07.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f761c0cb550 0x7f761c15fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:07.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761c14fd50 0x7f761c14ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:07.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c1f2b00 con 0x7f761c071250 2023-10-30T13:49:07.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c13a850 con 0x7f761c14fd50 2023-10-30T13:49:07.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f762113e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761c1f1ee0 con 0x7f761c0cb550 2023-10-30T13:49:07.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 0x7f761c13a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 0x7f761c13a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37946/0 (socket says 172.21.15.148:37946) 2023-10-30T13:49:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761ad9d640 1 -- 172.21.15.148:0/437215687 learned_addr learned my addr 172.21.15.148:0/437215687 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761b59e640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761c14fd50 0x7f761c14ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761a59c640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f761c0cb550 0x7f761c15fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:07.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761ad9d640 1 -- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f761c0cb550 msgr2=0x7f761c15fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761ad9d640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f761c0cb550 0x7f761c15fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761ad9d640 1 -- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761c14fd50 msgr2=0x7f761c14ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761ad9d640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761c14fd50 0x7f761c14ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.624+0000 7f761ad9d640 1 -- 172.21.15.148:0/437215687 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7614009a70 con 0x7f761c071250 2023-10-30T13:49:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f761ad9d640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 0x7f761c13a310 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f760c00c420 tx=0x7f760c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:07.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f76037fe640 1 -- 172.21.15.148:0/437215687 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760c005cf0 con 0x7f761c071250 2023-10-30T13:49:07.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f76037fe640 1 -- 172.21.15.148:0/437215687 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760c005e50 con 0x7f761c071250 2023-10-30T13:49:07.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f76037fe640 1 -- 172.21.15.148:0/437215687 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760c0072d0 con 0x7f761c071250 2023-10-30T13:49:07.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c14f380 con 0x7f761c071250 2023-10-30T13:49:07.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c1f2440 con 0x7f761c071250 2023-10-30T13:49:07.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c04e170 con 0x7f761c071250 2023-10-30T13:49:07.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f76037fe640 1 -- 172.21.15.148:0/437215687 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f760c007430 con 0x7f761c071250 2023-10-30T13:49:07.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f76037fe640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f76080664a0 0x7f7608068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:07.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f76037fe640 1 -- 172.21.15.148:0/437215687 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f760c083a80 con 0x7f761c071250 2023-10-30T13:49:07.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.628+0000 7f761a59c640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f76080664a0 0x7f7608068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:07.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.632+0000 7f761a59c640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f76080664a0 0x7f7608068960 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f761c0c3230 tx=0x7f7610008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:07.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.632+0000 7f76037fe640 1 -- 172.21.15.148:0/437215687 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f760c0508c0 con 0x7f761c071250 2023-10-30T13:49:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f761c0c7180 con 0x7f761c071250 2023-10-30T13:49:07.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f76037fe640 1 -- 172.21.15.148:0/437215687 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f760c017090 con 0x7f761c071250 2023-10-30T13:49:07.817 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:07.817 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:02.069202+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:07.817 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 151 2023-10-30T13:49:07.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f76080664a0 msgr2=0x7f7608068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:07.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f76080664a0 0x7f7608068960 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f761c0c3230 tx=0x7f7610008040 comp rx=0 tx=0).stop 2023-10-30T13:49:07.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 msgr2=0x7f761c13a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:07.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 0x7f761c13a310 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f760c00c420 tx=0x7f760c00c8e0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 shutdown_connections 2023-10-30T13:49:07.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f76080664a0 0x7f7608068960 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f761c14fd50 0x7f761c14ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f761c0cb550 0x7f761c15fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 --2- 172.21.15.148:0/437215687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f761c071250 0x7f761c13a310 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:07.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 >> 172.21.15.148:0/437215687 conn(0x7f761c06c6e0 msgr2=0x7f761c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:07.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 shutdown_connections 2023-10-30T13:49:07.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:07.816+0000 7f762113e640 1 -- 172.21.15.148:0/437215687 wait complete. 2023-10-30T13:49:07.839 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:09.840 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:09.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebf5d3640 1 Processor -- start 2023-10-30T13:49:09.988 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebf5d3640 1 -- start start 2023-10-30T13:49:09.989 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebf5d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2eb806f7a0 0x7f2eb806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:09.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebf5d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8066cf0 con 0x7f2eb80c6bd0 2023-10-30T13:49:09.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebf5d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb80c70d0 con 0x7f2eb8071110 2023-10-30T13:49:09.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebf5d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb80700c0 con 0x7f2eb806f7a0 2023-10-30T13:49:09.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebcb6e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2eb80c6bd0 0x7f2eb8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:59662/0 (socket says 172.21.15.148:59662) 2023-10-30T13:49:09.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebcb6e640 1 -- 172.21.15.148:0/67070789 learned_addr learned my addr 172.21.15.148:0/67070789 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:09.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebd36f640 1 --2- 172.21.15.148:0/67070789 >> v2:172.21.15.150:3300/0 conn(0x7f2eb806f7a0 0x7f2eb806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:09.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218569689 0 0) 0x7f2eb8066cf0 con 0x7f2eb80c6bd0 2023-10-30T13:49:09.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea4003530 con 0x7f2eb80c6bd0 2023-10-30T13:49:09.992 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670327673 0 0) 0x7f2ea4003530 con 0x7f2eb80c6bd0 2023-10-30T13:49:09.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 >> v2:172.21.15.150:3300/0 conn(0x7f2eb806f7a0 msgr2=0x7f2eb806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:09.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 --2- 172.21.15.148:0/67070789 >> v2:172.21.15.150:3300/0 conn(0x7f2eb806f7a0 0x7f2eb806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:09.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 >> v1:172.21.15.150:6789/0 conn(0x7f2eb8071110 legacy=0x7f2eb8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:09.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb80c7390 con 0x7f2eb80c6bd0 2023-10-30T13:49:09.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f2ea8001680 con 0x7f2eb80c6bd0 2023-10-30T13:49:09.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 >> v1:172.21.15.148:6789/0 conn(0x7f2eb80c6bd0 legacy=0x7f2eb8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:09.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ea4003980 0x7f2ea4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:09.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ea40047d0 0x7f2ea4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:09.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea4005190 0x7f2ea400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:09.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80700c0 con 0x7f2ea40047d0 2023-10-30T13:49:09.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea400c040 con 0x7f2ea4005190 2023-10-30T13:49:09.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea400c200 con 0x7f2ea4003980 2023-10-30T13:49:09.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebcb6e640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ea40047d0 0x7f2ea4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:09.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebd36f640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ea4003980 0x7f2ea4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:09.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebdb70640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea4005190 0x7f2ea400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:09.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebcb6e640 1 -- 172.21.15.148:0/67070789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ea4003980 msgr2=0x7f2ea4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:09.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebcb6e640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2ea4003980 0x7f2ea4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:09.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebcb6e640 1 -- 172.21.15.148:0/67070789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea4005190 msgr2=0x7f2ea400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:09.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebcb6e640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea4005190 0x7f2ea400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:09.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebcb6e640 1 -- 172.21.15.148:0/67070789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea4003530 con 0x7f2ea40047d0 2023-10-30T13:49:09.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.988+0000 7f2ebdb70640 1 -- 172.21.15.148:0/67070789 reap_dead start 2023-10-30T13:49:09.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2ebcb6e640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ea40047d0 0x7f2ea4004c50 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c210 tx=0x7f2ea801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5c1dbc73d8d247 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:09.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea80a5b20 con 0x7f2ea40047d0 2023-10-30T13:49:09.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea80a5c80 con 0x7f2ea40047d0 2023-10-30T13:49:10.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2eaffff640 1 -- 172.21.15.148:0/67070789 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8005b20 con 0x7f2ea40047d0 2023-10-30T13:49:10.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/67070789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ea40047d0 msgr2=0x7f2ea4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:10.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2ebf5d3640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ea40047d0 0x7f2ea4004c50 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c210 tx=0x7f2ea801cc20 comp rx=0 tx=0).stop 2023-10-30T13:49:10.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/67070789 shutdown_connections 2023-10-30T13:49:10.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2ebf5d3640 1 --2- 172.21.15.148:0/67070789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ea40047d0 0x7f2ea4004c50 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:10.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/67070789 >> 172.21.15.148:0/67070789 conn(0x7f2eb80c7910 msgr2=0x7f2eb80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:10.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/67070789 shutdown_connections 2023-10-30T13:49:10.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.992+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/67070789 wait complete. 2023-10-30T13:49:10.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 Processor -- start 2023-10-30T13:49:10.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 -- start start 2023-10-30T13:49:10.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea40047d0 0x7f2eb81eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:10.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2eb8151a40 0x7f2eb81ee580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:10.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2eb8152aa0 0x7f2eb814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:10.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81ee0f0 con 0x7f2eb8152aa0 2023-10-30T13:49:10.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81eddf0 con 0x7f2ea40047d0 2023-10-30T13:49:10.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb8151f30 con 0x7f2eb8151a40 2023-10-30T13:49:10.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebdb70640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2eb8152aa0 0x7f2eb814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:10.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebdb70640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2eb8152aa0 0x7f2eb814df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37950/0 (socket says 172.21.15.148:37950) 2023-10-30T13:49:10.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebdb70640 1 -- 172.21.15.148:0/3481091289 learned_addr learned my addr 172.21.15.148:0/3481091289 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:10.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebcb6e640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2eb8151a40 0x7f2eb81ee580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:10.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebd36f640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea40047d0 0x7f2eb81eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:10.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebdb70640 1 -- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2eb8151a40 msgr2=0x7f2eb81ee580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:10.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebdb70640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2eb8151a40 0x7f2eb81ee580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:10.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebdb70640 1 -- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea40047d0 msgr2=0x7f2eb81eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:10.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebdb70640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea40047d0 0x7f2eb81eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:10.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebdb70640 1 -- 172.21.15.148:0/3481091289 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8152210 con 0x7f2eb8152aa0 2023-10-30T13:49:10.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebdb70640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2eb8152aa0 0x7f2eb814df80 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80c6fb0 tx=0x7f2eb000e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:10.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2eadffb640 1 -- 172.21.15.148:0/3481091289 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0006e30 con 0x7f2eb8152aa0 2023-10-30T13:49:10.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2eadffb640 1 -- 172.21.15.148:0/3481091289 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0003980 con 0x7f2eb8152aa0 2023-10-30T13:49:10.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb81524f0 con 0x7f2eb8152aa0 2023-10-30T13:49:10.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2eadffb640 1 -- 172.21.15.148:0/3481091289 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0003d30 con 0x7f2eb8152aa0 2023-10-30T13:49:10.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:09.996+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb81527d0 con 0x7f2eb8152aa0 2023-10-30T13:49:10.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.000+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb804e170 con 0x7f2eb8152aa0 2023-10-30T13:49:10.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.000+0000 7f2eadffb640 1 -- 172.21.15.148:0/3481091289 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2eb0005070 con 0x7f2eb8152aa0 2023-10-30T13:49:10.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.000+0000 7f2eadffb640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e9c0664a0 0x7f2e9c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:10.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.000+0000 7f2eadffb640 1 -- 172.21.15.148:0/3481091289 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f2eb0083ad0 con 0x7f2eb8152aa0 2023-10-30T13:49:10.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.000+0000 7f2ebd36f640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e9c0664a0 0x7f2e9c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:10.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.004+0000 7f2ebd36f640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e9c0664a0 0x7f2e9c068960 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8069010 tx=0x7f2eb4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:10.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.004+0000 7f2eadffb640 1 -- 172.21.15.148:0/3481091289 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2eb0050880 con 0x7f2eb8152aa0 2023-10-30T13:49:10.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2eb80c4240 con 0x7f2eb8152aa0 2023-10-30T13:49:10.189 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2eadffb640 1 -- 172.21.15.148:0/3481091289 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f2eb0019020 con 0x7f2eb8152aa0 2023-10-30T13:49:10.189 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:10.189 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:02.069202+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:10.189 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 151 2023-10-30T13:49:10.190 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e9c0664a0 msgr2=0x7f2e9c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:10.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e9c0664a0 0x7f2e9c068960 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8069010 tx=0x7f2eb4009190 comp rx=0 tx=0).stop 2023-10-30T13:49:10.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2eb8152aa0 msgr2=0x7f2eb814df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:10.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2eb8152aa0 0x7f2eb814df80 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80c6fb0 tx=0x7f2eb000e920 comp rx=0 tx=0).stop 2023-10-30T13:49:10.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 shutdown_connections 2023-10-30T13:49:10.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e9c0664a0 0x7f2e9c068960 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:10.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2eb8152aa0 0x7f2eb814df80 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:10.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2eb8151a40 0x7f2eb81ee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:10.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 --2- 172.21.15.148:0/3481091289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea40047d0 0x7f2eb81eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:10.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 >> 172.21.15.148:0/3481091289 conn(0x7f2eb80c7910 msgr2=0x7f2eb80c95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:10.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 shutdown_connections 2023-10-30T13:49:10.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:10.188+0000 7f2ebf5d3640 1 -- 172.21.15.148:0/3481091289 wait complete. 2023-10-30T13:49:10.210 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:12.211 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:12.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.356+0000 7f6a9eed6640 1 Processor -- start 2023-10-30T13:49:12.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9eed6640 1 -- start start 2023-10-30T13:49:12.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9eed6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 0x7f6a980bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:12.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9eed6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a9813f4a0 0x7f6a980c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:12.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9eed6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a980c0a50 con 0x7f6a980c6f60 2023-10-30T13:49:12.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9eed6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9806f6e0 con 0x7f6a9813f4a0 2023-10-30T13:49:12.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9eed6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9806f9c0 con 0x7f6a980c6b80 2023-10-30T13:49:12.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9d473640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a9813f4a0 0x7f6a980c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:12.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9cc72640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 0x7f6a980bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:12.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9d473640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a9813f4a0 0x7f6a980c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:44390/0 (socket says 172.21.15.148:44390) 2023-10-30T13:49:12.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9d473640 1 -- 172.21.15.148:0/877122925 learned_addr learned my addr 172.21.15.148:0/877122925 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:12.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9cc72640 1 -- 172.21.15.148:0/877122925 >> v2:172.21.15.150:3300/0 conn(0x7f6a9813f4a0 msgr2=0x7f6a980c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:12.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9cc72640 1 --2- 172.21.15.148:0/877122925 >> v2:172.21.15.150:3300/0 conn(0x7f6a9813f4a0 0x7f6a980c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:12.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9cc72640 1 -- 172.21.15.148:0/877122925 >> v1:172.21.15.150:6789/0 conn(0x7f6a980c6f60 legacy=0x7f6a9813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:12.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9cc72640 1 -- 172.21.15.148:0/877122925 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9806fca0 con 0x7f6a980c6b80 2023-10-30T13:49:12.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.360+0000 7f6a9d473640 1 --2- 172.21.15.148:0/877122925 >> v2:172.21.15.150:3300/0 conn(0x7f6a9813f4a0 0x7f6a980c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:49:12.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a9cc72640 1 --2- 172.21.15.148:0/877122925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 0x7f6a980bfec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6a940097c0 tx=0x7f6a9401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c27b0711ac315789 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:12.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a8f7fe640 1 -- 172.21.15.148:0/877122925 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a940ad640 con 0x7f6a980c6b80 2023-10-30T13:49:12.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a8f7fe640 1 -- 172.21.15.148:0/877122925 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a940ad7a0 con 0x7f6a980c6b80 2023-10-30T13:49:12.365 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a8f7fe640 1 -- 172.21.15.148:0/877122925 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a940adb30 con 0x7f6a980c6b80 2023-10-30T13:49:12.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a9eed6640 1 -- 172.21.15.148:0/877122925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 msgr2=0x7f6a980bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:12.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a9eed6640 1 --2- 172.21.15.148:0/877122925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 0x7f6a980bfec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6a940097c0 tx=0x7f6a9401c480 comp rx=0 tx=0).stop 2023-10-30T13:49:12.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a9eed6640 1 -- 172.21.15.148:0/877122925 shutdown_connections 2023-10-30T13:49:12.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a9eed6640 1 --2- 172.21.15.148:0/877122925 >> v2:172.21.15.150:3300/0 conn(0x7f6a9813f4a0 0x7f6a980c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:12.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a9eed6640 1 --2- 172.21.15.148:0/877122925 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 0x7f6a980bfec0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:12.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a9eed6640 1 -- 172.21.15.148:0/877122925 >> 172.21.15.148:0/877122925 conn(0x7f6a980c7910 msgr2=0x7f6a980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:12.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a9eed6640 1 -- 172.21.15.148:0/877122925 shutdown_connections 2023-10-30T13:49:12.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.364+0000 7f6a9eed6640 1 -- 172.21.15.148:0/877122925 wait complete. 2023-10-30T13:49:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 Processor -- start 2023-10-30T13:49:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 -- start start 2023-10-30T13:49:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 0x7f6a981f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a980c6f60 0x7f6a9806a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:12.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a9813f4a0 0x7f6a98152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:12.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a981649b0 con 0x7f6a980c6f60 2023-10-30T13:49:12.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a98157df0 con 0x7f6a9813f4a0 2023-10-30T13:49:12.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a981641e0 con 0x7f6a980c6b80 2023-10-30T13:49:12.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a980c6f60 0x7f6a9806a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:12.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a980c6f60 0x7f6a9806a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44688/0 (socket says 172.21.15.148:44688) 2023-10-30T13:49:12.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8ffff640 1 -- 172.21.15.148:0/3344913720 learned_addr learned my addr 172.21.15.148:0/3344913720 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:12.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9cc72640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 0x7f6a981f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:12.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9d473640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a9813f4a0 0x7f6a98152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:12.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8ffff640 1 -- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 msgr2=0x7f6a981f6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:12.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8ffff640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 0x7f6a981f6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:12.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8ffff640 1 -- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a9813f4a0 msgr2=0x7f6a98152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:12.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8ffff640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a9813f4a0 0x7f6a98152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:12.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8ffff640 1 -- 172.21.15.148:0/3344913720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a90009b30 con 0x7f6a980c6f60 2023-10-30T13:49:12.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8ffff640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a980c6f60 0x7f6a9806a690 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800c4d0 tx=0x7f6a8800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:12.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8d7fa640 1 -- 172.21.15.148:0/3344913720 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a880052d0 con 0x7f6a980c6f60 2023-10-30T13:49:12.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8d7fa640 1 -- 172.21.15.148:0/3344913720 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a88005430 con 0x7f6a980c6f60 2023-10-30T13:49:12.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a8d7fa640 1 -- 172.21.15.148:0/3344913720 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a88003800 con 0x7f6a980c6f60 2023-10-30T13:49:12.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a94009420 con 0x7f6a980c6f60 2023-10-30T13:49:12.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a981643f0 con 0x7f6a980c6f60 2023-10-30T13:49:12.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.368+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9804e170 con 0x7f6a980c6f60 2023-10-30T13:49:12.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.372+0000 7f6a8d7fa640 1 -- 172.21.15.148:0/3344913720 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6a88003960 con 0x7f6a980c6f60 2023-10-30T13:49:12.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.372+0000 7f6a8d7fa640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6a840664a0 0x7f6a84068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:12.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.372+0000 7f6a8d7fa640 1 -- 172.21.15.148:0/3344913720 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f6a88083970 con 0x7f6a980c6f60 2023-10-30T13:49:12.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.372+0000 7f6a9cc72640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6a840664a0 0x7f6a84068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:12.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.372+0000 7f6a9cc72640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6a840664a0 0x7f6a84068960 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6a940af000 tx=0x7f6a94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:12.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.376+0000 7f6a8d7fa640 1 -- 172.21.15.148:0/3344913720 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6a880507b0 con 0x7f6a980c6f60 2023-10-30T13:49:12.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.556+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a98073cc0 con 0x7f6a980c6f60 2023-10-30T13:49:12.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.556+0000 7f6a8d7fa640 1 -- 172.21.15.148:0/3344913720 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f6a88053dd0 con 0x7f6a980c6f60 2023-10-30T13:49:12.558 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:12.558 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:02.069202+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:12.559 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 151 2023-10-30T13:49:12.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6a840664a0 msgr2=0x7f6a84068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:12.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6a840664a0 0x7f6a84068960 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6a940af000 tx=0x7f6a94008040 comp rx=0 tx=0).stop 2023-10-30T13:49:12.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a980c6f60 msgr2=0x7f6a9806a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:12.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a980c6f60 0x7f6a9806a690 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f6a8800c4d0 tx=0x7f6a8800c990 comp rx=0 tx=0).stop 2023-10-30T13:49:12.560 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 shutdown_connections 2023-10-30T13:49:12.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6a840664a0 0x7f6a84068960 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:12.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a980c6f60 0x7f6a9806a690 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:12.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a9813f4a0 0x7f6a98152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:12.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 --2- 172.21.15.148:0/3344913720 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6a980c6b80 0x7f6a981f6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:12.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 >> 172.21.15.148:0/3344913720 conn(0x7f6a980c7910 msgr2=0x7f6a980c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:12.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 shutdown_connections 2023-10-30T13:49:12.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:12.560+0000 7f6a9eed6640 1 -- 172.21.15.148:0/3344913720 wait complete. 2023-10-30T13:49:12.579 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:14.581 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:14.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d2969c640 1 Processor -- start 2023-10-30T13:49:14.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d2969c640 1 -- start start 2023-10-30T13:49:14.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d2969c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0d240c6bd0 0x7f0d2406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:14.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d2969c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d24066cf0 con 0x7f0d2406f7a0 2023-10-30T13:49:14.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d2969c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240c70d0 con 0x7f0d24071110 2023-10-30T13:49:14.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d2969c640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d240700c0 con 0x7f0d240c6bd0 2023-10-30T13:49:14.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d22ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0d2406f7a0 0x7f0d24069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:57626/0 (socket says 172.21.15.148:57626) 2023-10-30T13:49:14.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d22ffd640 1 -- 172.21.15.148:0/3358770919 learned_addr learned my addr 172.21.15.148:0/3358770919 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:14.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d1a7fc640 1 --2- 172.21.15.148:0/3358770919 >> v2:172.21.15.148:3300/0 conn(0x7f0d240c6bd0 0x7f0d2406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:14.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d227fc640 1 -- 172.21.15.148:0/3358770919 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794614402 0 0) 0x7f0d24066cf0 con 0x7f0d2406f7a0 2023-10-30T13:49:14.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d227fc640 1 -- 172.21.15.148:0/3358770919 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d10003530 con 0x7f0d2406f7a0 2023-10-30T13:49:14.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d1a7fc640 1 -- 172.21.15.148:0/3358770919 >> v1:172.21.15.150:6789/0 conn(0x7f0d24071110 legacy=0x7f0d24158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:14.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d1a7fc640 1 -- 172.21.15.148:0/3358770919 >> v1:172.21.15.148:6789/0 conn(0x7f0d2406f7a0 legacy=0x7f0d24069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:14.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.728+0000 7f0d1a7fc640 1 -- 172.21.15.148:0/3358770919 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d240c7390 con 0x7f0d240c6bd0 2023-10-30T13:49:14.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d1a7fc640 1 --2- 172.21.15.148:0/3358770919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 0x7f0d2406fb80 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f0d140097c0 tx=0x7f0d1401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11411023c81169c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:14.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d227fc640 1 -- 172.21.15.148:0/3358770919 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d140a8970 con 0x7f0d240c6bd0 2023-10-30T13:49:14.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d227fc640 1 -- 172.21.15.148:0/3358770919 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d140a8ad0 con 0x7f0d240c6bd0 2023-10-30T13:49:14.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d227fc640 1 -- 172.21.15.148:0/3358770919 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d140a8e60 con 0x7f0d240c6bd0 2023-10-30T13:49:14.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d2969c640 1 -- 172.21.15.148:0/3358770919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 msgr2=0x7f0d2406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:14.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d2969c640 1 --2- 172.21.15.148:0/3358770919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 0x7f0d2406fb80 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f0d140097c0 tx=0x7f0d1401ec70 comp rx=0 tx=0).stop 2023-10-30T13:49:14.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d2969c640 1 -- 172.21.15.148:0/3358770919 shutdown_connections 2023-10-30T13:49:14.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d2969c640 1 --2- 172.21.15.148:0/3358770919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 0x7f0d2406fb80 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:14.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d2969c640 1 -- 172.21.15.148:0/3358770919 >> 172.21.15.148:0/3358770919 conn(0x7f0d240c7910 msgr2=0x7f0d240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:14.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d2969c640 1 -- 172.21.15.148:0/3358770919 shutdown_connections 2023-10-30T13:49:14.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.732+0000 7f0d2969c640 1 -- 172.21.15.148:0/3358770919 wait complete. 2023-10-30T13:49:14.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 Processor -- start 2023-10-30T13:49:14.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 -- start start 2023-10-30T13:49:14.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d2406f7a0 0x7f0d2414edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:14.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d24071110 0x7f0d24154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:14.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 0x7f0d24154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:14.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d241e6890 con 0x7f0d240c6bd0 2023-10-30T13:49:14.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2414f300 con 0x7f0d24071110 2023-10-30T13:49:14.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d241e61a0 con 0x7f0d2406f7a0 2023-10-30T13:49:14.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d237fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 0x7f0d24154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:14.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d237fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 0x7f0d24154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44712/0 (socket says 172.21.15.148:44712) 2023-10-30T13:49:14.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d237fe640 1 -- 172.21.15.148:0/786592567 learned_addr learned my addr 172.21.15.148:0/786592567 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:14.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d1a7fc640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d24071110 0x7f0d24154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:14.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d22ffd640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d2406f7a0 0x7f0d2414edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:14.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d237fe640 1 -- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d2406f7a0 msgr2=0x7f0d2414edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:14.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d237fe640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d2406f7a0 0x7f0d2414edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:14.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d237fe640 1 -- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d24071110 msgr2=0x7f0d24154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:14.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d237fe640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d24071110 0x7f0d24154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:14.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d237fe640 1 -- 172.21.15.148:0/786592567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d14009420 con 0x7f0d240c6bd0 2023-10-30T13:49:14.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d237fe640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 0x7f0d24154ad0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00c760 tx=0x7f0d1c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:14.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d1bfff640 1 -- 172.21.15.148:0/786592567 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c003b80 con 0x7f0d240c6bd0 2023-10-30T13:49:14.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d1bfff640 1 -- 172.21.15.148:0/786592567 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1c003ce0 con 0x7f0d240c6bd0 2023-10-30T13:49:14.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d240c7cf0 con 0x7f0d240c6bd0 2023-10-30T13:49:14.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d1bfff640 1 -- 172.21.15.148:0/786592567 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c00aa30 con 0x7f0d240c6bd0 2023-10-30T13:49:14.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.736+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d241ef380 con 0x7f0d240c6bd0 2023-10-30T13:49:14.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.740+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2404e170 con 0x7f0d240c6bd0 2023-10-30T13:49:14.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.740+0000 7f0d1bfff640 1 -- 172.21.15.148:0/786592567 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0d1c00ab90 con 0x7f0d240c6bd0 2023-10-30T13:49:14.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.740+0000 7f0d1bfff640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0d040664d0 0x7f0d04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:14.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.740+0000 7f0d1bfff640 1 -- 172.21.15.148:0/786592567 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f0d1c084220 con 0x7f0d240c6bd0 2023-10-30T13:49:14.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.740+0000 7f0d22ffd640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0d040664d0 0x7f0d04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:14.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.740+0000 7f0d22ffd640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0d040664d0 0x7f0d04068990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0011f0 tx=0x7f0d0c007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:14.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.744+0000 7f0d1bfff640 1 -- 172.21.15.148:0/786592567 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0d1c050fe0 con 0x7f0d240c6bd0 2023-10-30T13:49:14.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.924+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d2406fea0 con 0x7f0d240c6bd0 2023-10-30T13:49:14.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.924+0000 7f0d1bfff640 1 -- 172.21.15.148:0/786592567 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f0d1c054600 con 0x7f0d240c6bd0 2023-10-30T13:49:14.926 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:14.926 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:02.069202+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:14.927 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 151 2023-10-30T13:49:14.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0d040664d0 msgr2=0x7f0d04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:14.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0d040664d0 0x7f0d04068990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0011f0 tx=0x7f0d0c007d40 comp rx=0 tx=0).stop 2023-10-30T13:49:14.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 msgr2=0x7f0d24154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:14.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 0x7f0d24154ad0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00c760 tx=0x7f0d1c00cc20 comp rx=0 tx=0).stop 2023-10-30T13:49:14.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 shutdown_connections 2023-10-30T13:49:14.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f0d040664d0 0x7f0d04068990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:14.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d240c6bd0 0x7f0d24154ad0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:14.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d24071110 0x7f0d24154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:14.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 --2- 172.21.15.148:0/786592567 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f0d2406f7a0 0x7f0d2414edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:14.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 >> 172.21.15.148:0/786592567 conn(0x7f0d240c7910 msgr2=0x7f0d240c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:14.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 shutdown_connections 2023-10-30T13:49:14.930 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:14.928+0000 7f0d2969c640 1 -- 172.21.15.148:0/786592567 wait complete. 2023-10-30T13:49:14.947 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:16.949 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:17.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39cb1d3640 1 Processor -- start 2023-10-30T13:49:17.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39cb1d3640 1 -- start start 2023-10-30T13:49:17.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39cb1d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39c415d6f0 0x7f39c415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:17.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39cb1d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f39c40cf160 0x7f39c40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:17.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39cb1d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c40cb800 con 0x7f39c41538a0 2023-10-30T13:49:17.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39cb1d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c40cb330 con 0x7f39c40cf160 2023-10-30T13:49:17.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39cb1d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c40cad70 con 0x7f39c415d6f0 2023-10-30T13:49:17.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39c8f6f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f39c41538a0 0x7f39c40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:57630/0 (socket says 172.21.15.148:57630) 2023-10-30T13:49:17.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39c8f6f640 1 -- 172.21.15.148:0/1607645004 learned_addr learned my addr 172.21.15.148:0/1607645004 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:17.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39c9770640 1 --2- 172.21.15.148:0/1607645004 >> v2:172.21.15.148:3300/0 conn(0x7f39c40cf160 0x7f39c40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:17.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39bbfff640 1 --2- 172.21.15.148:0/1607645004 >> v2:172.21.15.150:3300/0 conn(0x7f39c415d6f0 0x7f39c415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:17.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39bb7fe640 1 -- 172.21.15.148:0/1607645004 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749521648 0 0) 0x7f39c40cb800 con 0x7f39c41538a0 2023-10-30T13:49:17.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39bb7fe640 1 -- 172.21.15.148:0/1607645004 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ac003530 con 0x7f39c41538a0 2023-10-30T13:49:17.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39c9770640 1 -- 172.21.15.148:0/1607645004 >> v2:172.21.15.150:3300/0 conn(0x7f39c415d6f0 msgr2=0x7f39c415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:17.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39c9770640 1 --2- 172.21.15.148:0/1607645004 >> v2:172.21.15.150:3300/0 conn(0x7f39c415d6f0 0x7f39c415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39c9770640 1 -- 172.21.15.148:0/1607645004 >> v1:172.21.15.148:6789/0 conn(0x7f39c41538a0 legacy=0x7f39c40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:17.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.100+0000 7f39c9770640 1 -- 172.21.15.148:0/1607645004 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c40cb050 con 0x7f39c40cf160 2023-10-30T13:49:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39c9770640 1 --2- 172.21.15.148:0/1607645004 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c40cf160 0x7f39c40cf590 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f39c40b30a0 tx=0x7f39bc01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f3326483184f64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39bb7fe640 1 -- 172.21.15.148:0/1607645004 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0b0440 con 0x7f39c40cf160 2023-10-30T13:49:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39bb7fe640 1 -- 172.21.15.148:0/1607645004 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc0b05a0 con 0x7f39c40cf160 2023-10-30T13:49:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39bb7fe640 1 -- 172.21.15.148:0/1607645004 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0b0930 con 0x7f39c40cf160 2023-10-30T13:49:17.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39cb1d3640 1 -- 172.21.15.148:0/1607645004 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c40cf160 msgr2=0x7f39c40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39cb1d3640 1 --2- 172.21.15.148:0/1607645004 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c40cf160 0x7f39c40cf590 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f39c40b30a0 tx=0x7f39bc01f0f0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39cb1d3640 1 -- 172.21.15.148:0/1607645004 shutdown_connections 2023-10-30T13:49:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39cb1d3640 1 --2- 172.21.15.148:0/1607645004 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c40cf160 0x7f39c40cf590 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39cb1d3640 1 --2- 172.21.15.148:0/1607645004 >> v2:172.21.15.150:3300/0 conn(0x7f39c415d6f0 0x7f39c415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39cb1d3640 1 -- 172.21.15.148:0/1607645004 >> 172.21.15.148:0/1607645004 conn(0x7f39c406bfe0 msgr2=0x7f39c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:17.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39cb1d3640 1 -- 172.21.15.148:0/1607645004 shutdown_connections 2023-10-30T13:49:17.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.104+0000 7f39cb1d3640 1 -- 172.21.15.148:0/1607645004 wait complete. 2023-10-30T13:49:17.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 Processor -- start 2023-10-30T13:49:17.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 -- start start 2023-10-30T13:49:17.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40cf160 0x7f39c41efeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:17.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c41538a0 0x7f39c4157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:17.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c415d6f0 0x7f39c40b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:17.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c41e4e50 con 0x7f39c41538a0 2023-10-30T13:49:17.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c41f03f0 con 0x7f39c415d6f0 2023-10-30T13:49:17.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c4159000 con 0x7f39c40cf160 2023-10-30T13:49:17.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39bbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c41538a0 0x7f39c4157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:17.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39bbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c41538a0 0x7f39c4157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44742/0 (socket says 172.21.15.148:44742) 2023-10-30T13:49:17.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39bbfff640 1 -- 172.21.15.148:0/3054167539 learned_addr learned my addr 172.21.15.148:0/3054167539 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:17.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39c9770640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c415d6f0 0x7f39c40b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:17.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39c8f6f640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40cf160 0x7f39c41efeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:17.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39bbfff640 1 -- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40cf160 msgr2=0x7f39c41efeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:17.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39bbfff640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40cf160 0x7f39c41efeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39bbfff640 1 -- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c415d6f0 msgr2=0x7f39c40b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:17.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39bbfff640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c415d6f0 0x7f39c40b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39bbfff640 1 -- 172.21.15.148:0/3054167539 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc009a70 con 0x7f39c41538a0 2023-10-30T13:49:17.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39bbfff640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c41538a0 0x7f39c4157520 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f39b400e870 tx=0x7f39b400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:17.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39b97fa640 1 -- 172.21.15.148:0/3054167539 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4009510 con 0x7f39c41538a0 2023-10-30T13:49:17.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39b97fa640 1 -- 172.21.15.148:0/3054167539 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b400eec0 con 0x7f39c41538a0 2023-10-30T13:49:17.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39b97fa640 1 -- 172.21.15.148:0/3054167539 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b400cb40 con 0x7f39c41538a0 2023-10-30T13:49:17.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c4157a60 con 0x7f39c41538a0 2023-10-30T13:49:17.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.108+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c41592c0 con 0x7f39c41538a0 2023-10-30T13:49:17.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.112+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c40b2a50 con 0x7f39c41538a0 2023-10-30T13:49:17.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.112+0000 7f39b97fa640 1 -- 172.21.15.148:0/3054167539 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39b400cca0 con 0x7f39c41538a0 2023-10-30T13:49:17.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.112+0000 7f39b97fa640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39b00664a0 0x7f39b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:17.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.112+0000 7f39b97fa640 1 -- 172.21.15.148:0/3054167539 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f39b4084800 con 0x7f39c41538a0 2023-10-30T13:49:17.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.112+0000 7f39c8f6f640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39b00664a0 0x7f39b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:17.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.112+0000 7f39c8f6f640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39b00664a0 0x7f39b0068960 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f39c00011f0 tx=0x7f39c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:17.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.116+0000 7f39b97fa640 1 -- 172.21.15.148:0/3054167539 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39b4051480 con 0x7f39c41538a0 2023-10-30T13:49:17.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.296+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39c404e170 con 0x7f39c41538a0 2023-10-30T13:49:17.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39b97fa640 1 -- 172.21.15.148:0/3054167539 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f39b4054aa0 con 0x7f39c41538a0 2023-10-30T13:49:17.300 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:17.300 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:02.069202+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:17.301 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 151 2023-10-30T13:49:17.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39b00664a0 msgr2=0x7f39b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:17.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39b00664a0 0x7f39b0068960 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f39c00011f0 tx=0x7f39c0009040 comp rx=0 tx=0).stop 2023-10-30T13:49:17.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c41538a0 msgr2=0x7f39c4157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:17.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c41538a0 0x7f39c4157520 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f39b400e870 tx=0x7f39b400ec70 comp rx=0 tx=0).stop 2023-10-30T13:49:17.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 shutdown_connections 2023-10-30T13:49:17.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f39b00664a0 0x7f39b0068960 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39c41538a0 0x7f39c4157520 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39c415d6f0 0x7f39c40b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 --2- 172.21.15.148:0/3054167539 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f39c40cf160 0x7f39c41efeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:17.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 >> 172.21.15.148:0/3054167539 conn(0x7f39c406bfe0 msgr2=0x7f39c415aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:17.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 shutdown_connections 2023-10-30T13:49:17.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:17.300+0000 7f39cb1d3640 1 -- 172.21.15.148:0/3054167539 wait complete. 2023-10-30T13:49:17.322 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:19.323 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:19.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.468+0000 7f92d531d640 1 Processor -- start 2023-10-30T13:49:19.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.468+0000 7f92d531d640 1 -- start start 2023-10-30T13:49:19.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92d531d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92d00c6bd0 0x7f92d006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:19.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92d531d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d0071110 0x7f92d0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:19.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92d531d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d0066cf0 con 0x7f92d006f7a0 2023-10-30T13:49:19.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92d531d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d00700c0 con 0x7f92d00c6bd0 2023-10-30T13:49:19.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92d531d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d00c70d0 con 0x7f92d0071110 2023-10-30T13:49:19.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92c67fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92d00c6bd0 0x7f92d006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:19.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92c67fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f92d00c6bd0 0x7f92d006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44744/0 (socket says 172.21.15.148:44744) 2023-10-30T13:49:19.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92c67fc640 1 -- 172.21.15.148:0/2715735179 learned_addr learned my addr 172.21.15.148:0/2715735179 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:19.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92cf7fe640 1 --2- 172.21.15.148:0/2715735179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d0071110 0x7f92d0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:19.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92c67fc640 1 -- 172.21.15.148:0/2715735179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d0071110 msgr2=0x7f92d0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:19.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92c67fc640 1 --2- 172.21.15.148:0/2715735179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d0071110 0x7f92d0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:19.475 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92c67fc640 1 -- 172.21.15.148:0/2715735179 >> v1:172.21.15.150:6789/0 conn(0x7f92d006f7a0 legacy=0x7f92d0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:19.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92c67fc640 1 -- 172.21.15.148:0/2715735179 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00c7390 con 0x7f92d00c6bd0 2023-10-30T13:49:19.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.472+0000 7f92cf7fe640 1 --2- 172.21.15.148:0/2715735179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d0071110 0x7f92d0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:19.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92c67fc640 1 --2- 172.21.15.148:0/2715735179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 0x7f92d006fb80 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009870 tx=0x7f92c001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9c5d945316ccec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:19.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92ce7fc640 1 -- 172.21.15.148:0/2715735179 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c00a8a30 con 0x7f92d00c6bd0 2023-10-30T13:49:19.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92ce7fc640 1 -- 172.21.15.148:0/2715735179 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c00a8b90 con 0x7f92d00c6bd0 2023-10-30T13:49:19.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92ce7fc640 1 -- 172.21.15.148:0/2715735179 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c00b0440 con 0x7f92d00c6bd0 2023-10-30T13:49:19.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 -- 172.21.15.148:0/2715735179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 msgr2=0x7f92d006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:19.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 --2- 172.21.15.148:0/2715735179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 0x7f92d006fb80 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009870 tx=0x7f92c001ed30 comp rx=0 tx=0).stop 2023-10-30T13:49:19.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 -- 172.21.15.148:0/2715735179 shutdown_connections 2023-10-30T13:49:19.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 --2- 172.21.15.148:0/2715735179 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d0071110 0x7f92d0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:19.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 --2- 172.21.15.148:0/2715735179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 0x7f92d006fb80 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:19.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 -- 172.21.15.148:0/2715735179 >> 172.21.15.148:0/2715735179 conn(0x7f92d00c7910 msgr2=0x7f92d00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:19.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 -- 172.21.15.148:0/2715735179 shutdown_connections 2023-10-30T13:49:19.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 -- 172.21.15.148:0/2715735179 wait complete. 2023-10-30T13:49:19.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 Processor -- start 2023-10-30T13:49:19.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.476+0000 7f92d531d640 1 -- start start 2023-10-30T13:49:19.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92d531d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d006f7a0 0x7f92d01efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:19.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92d531d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92d0071110 0x7f92d01ec630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:19.481 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92d531d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 0x7f92d01ecb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:19.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92d531d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d00c4420 con 0x7f92d00c6bd0 2023-10-30T13:49:19.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92d531d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d00aa970 con 0x7f92d0071110 2023-10-30T13:49:19.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92d531d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d00aac50 con 0x7f92d006f7a0 2023-10-30T13:49:19.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92cf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 0x7f92d01ecb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:19.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92cf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 0x7f92d01ecb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:44760/0 (socket says 172.21.15.148:44760) 2023-10-30T13:49:19.482 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92cf7fe640 1 -- 172.21.15.148:0/1000010181 learned_addr learned my addr 172.21.15.148:0/1000010181 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:19.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92c67fc640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92d0071110 0x7f92d01ec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:19.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92ceffd640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d006f7a0 0x7f92d01efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:19.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92cf7fe640 1 -- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d006f7a0 msgr2=0x7f92d01efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:19.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92cf7fe640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d006f7a0 0x7f92d01efe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:19.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92cf7fe640 1 -- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92d0071110 msgr2=0x7f92d01ec630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:19.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92cf7fe640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92d0071110 0x7f92d01ec630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:19.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92cf7fe640 1 -- 172.21.15.148:0/1000010181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00094d0 con 0x7f92d00c6bd0 2023-10-30T13:49:19.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92c67fc640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92d0071110 0x7f92d01ec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:19.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92cf7fe640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 0x7f92d01ecb70 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f92c8007f20 tx=0x7f92c800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:19.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92c7fff640 1 -- 172.21.15.148:0/1000010181 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c800dee0 con 0x7f92d00c6bd0 2023-10-30T13:49:19.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92c7fff640 1 -- 172.21.15.148:0/1000010181 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c800f040 con 0x7f92d00c6bd0 2023-10-30T13:49:19.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d00ab0e0 con 0x7f92d00c6bd0 2023-10-30T13:49:19.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92c7fff640 1 -- 172.21.15.148:0/1000010181 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8003de0 con 0x7f92d00c6bd0 2023-10-30T13:49:19.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d01f0a70 con 0x7f92d00c6bd0 2023-10-30T13:49:19.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d0070610 con 0x7f92d00c6bd0 2023-10-30T13:49:19.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.480+0000 7f92c7fff640 1 -- 172.21.15.148:0/1000010181 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f92c8004030 con 0x7f92d00c6bd0 2023-10-30T13:49:19.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.484+0000 7f92c7fff640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92b00664a0 0x7f92b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:19.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.484+0000 7f92c7fff640 1 -- 172.21.15.148:0/1000010181 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7488+0+0 (secure 0 0 0) 0x7f92c8084940 con 0x7f92d00c6bd0 2023-10-30T13:49:19.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.484+0000 7f92ceffd640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92b00664a0 0x7f92b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:19.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.484+0000 7f92ceffd640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92b00664a0 0x7f92b0068960 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f92b8000f70 tx=0x7f92b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:19.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.488+0000 7f92c7fff640 1 -- 172.21.15.148:0/1000010181 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92d0070610 con 0x7f92d00c6bd0 2023-10-30T13:49:19.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92d0070610 con 0x7f92d00c6bd0 2023-10-30T13:49:19.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92c7fff640 1 -- 172.21.15.148:0/1000010181 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f92d0070610 con 0x7f92d00c6bd0 2023-10-30T13:49:19.669 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:19.669 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:02.069202+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":7611},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7611":{"gid":7611,"name":"a","rank":1,"incarnation":133,"state":"up:active","state_seq":10,"addr":"172.21.15.148:6835/1228827025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1228827025},{"type":"v1","addr":"172.21.15.148:6835","nonce":1228827025}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:19.669 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 151 2023-10-30T13:49:19.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92b00664a0 msgr2=0x7f92b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:19.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92b00664a0 0x7f92b0068960 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f92b8000f70 tx=0x7f92b8009040 comp rx=0 tx=0).stop 2023-10-30T13:49:19.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 msgr2=0x7f92d01ecb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:19.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 0x7f92d01ecb70 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f92c8007f20 tx=0x7f92c800d560 comp rx=0 tx=0).stop 2023-10-30T13:49:19.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 shutdown_connections 2023-10-30T13:49:19.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92b00664a0 0x7f92b0068960 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:19.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92d00c6bd0 0x7f92d01ecb70 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:19.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92d0071110 0x7f92d01ec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:19.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 --2- 172.21.15.148:0/1000010181 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92d006f7a0 0x7f92d01efe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:19.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 >> 172.21.15.148:0/1000010181 conn(0x7f92d00c7910 msgr2=0x7f92d0157410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:19.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 shutdown_connections 2023-10-30T13:49:19.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:19.668+0000 7f92d531d640 1 -- 172.21.15.148:0/1000010181 wait complete. 2023-10-30T13:49:19.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-30T13:49:21.691 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:21.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.840+0000 7f180b1a1640 1 Processor -- start 2023-10-30T13:49:21.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.840+0000 7f180b1a1640 1 -- start start 2023-10-30T13:49:21.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f180b1a1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f18040c6b80 0x7f18040bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f180b1a1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18040c0a50 con 0x7f18040c6f60 2023-10-30T13:49:21.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f180b1a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180406f6e0 con 0x7f180413f4a0 2023-10-30T13:49:21.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f180b1a1640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180406f9c0 con 0x7f18040c6b80 2023-10-30T13:49:21.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f1808f3d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f18040c6b80 0x7f18040bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f1808f3d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f18040c6b80 0x7f18040bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34478/0 (socket says 172.21.15.148:34478) 2023-10-30T13:49:21.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f1808f3d640 1 -- 172.21.15.148:0/947229168 learned_addr learned my addr 172.21.15.148:0/947229168 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:21.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f1808f3d640 1 -- 172.21.15.148:0/947229168 >> v1:172.21.15.150:6789/0 conn(0x7f180413f4a0 legacy=0x7f18041566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f1808f3d640 1 -- 172.21.15.148:0/947229168 >> v1:172.21.15.148:6789/0 conn(0x7f18040c6f60 legacy=0x7f180413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.844+0000 7f1808f3d640 1 -- 172.21.15.148:0/947229168 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180406fca0 con 0x7f18040c6b80 2023-10-30T13:49:21.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f1808f3d640 1 --2- 172.21.15.148:0/947229168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18040c6b80 0x7f18040bfec0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f1800009870 tx=0x7f180001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b073b2cdb5f39a16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:21.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f17fb7fe640 1 -- 172.21.15.148:0/947229168 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18000ab650 con 0x7f18040c6b80 2023-10-30T13:49:21.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f17fb7fe640 1 -- 172.21.15.148:0/947229168 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18000ab7b0 con 0x7f18040c6b80 2023-10-30T13:49:21.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f17fb7fe640 1 -- 172.21.15.148:0/947229168 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18000abb40 con 0x7f18040c6b80 2023-10-30T13:49:21.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 -- 172.21.15.148:0/947229168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18040c6b80 msgr2=0x7f18040bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 --2- 172.21.15.148:0/947229168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18040c6b80 0x7f18040bfec0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f1800009870 tx=0x7f180001c540 comp rx=0 tx=0).stop 2023-10-30T13:49:21.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 -- 172.21.15.148:0/947229168 shutdown_connections 2023-10-30T13:49:21.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 --2- 172.21.15.148:0/947229168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18040c6b80 0x7f18040bfec0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 -- 172.21.15.148:0/947229168 >> 172.21.15.148:0/947229168 conn(0x7f18040c7910 msgr2=0x7f18040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:21.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 -- 172.21.15.148:0/947229168 shutdown_connections 2023-10-30T13:49:21.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 -- 172.21.15.148:0/947229168 wait complete. 2023-10-30T13:49:21.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 Processor -- start 2023-10-30T13:49:21.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 -- start start 2023-10-30T13:49:21.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18040c6b80 0x7f180406a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18040c6f60 0x7f180413cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f180413f4a0 0x7f1804132600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18041f4590 con 0x7f180413f4a0 2023-10-30T13:49:21.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180412e1c0 con 0x7f18040c6b80 2023-10-30T13:49:21.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.848+0000 7f180b1a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180413cb30 con 0x7f18040c6f60 2023-10-30T13:49:21.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180973e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f180413f4a0 0x7f1804132600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180973e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f180413f4a0 0x7f1804132600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34480/0 (socket says 172.21.15.148:34480) 2023-10-30T13:49:21.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180973e640 1 -- 172.21.15.148:0/2984864838 learned_addr learned my addr 172.21.15.148:0/2984864838 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:21.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f1808f3d640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18040c6b80 0x7f180406a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f17fbfff640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18040c6f60 0x7f180413cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180973e640 1 -- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18040c6f60 msgr2=0x7f180413cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180973e640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18040c6f60 0x7f180413cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180973e640 1 -- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18040c6b80 msgr2=0x7f180406a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:21.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180973e640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18040c6b80 0x7f180406a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:21.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180973e640 1 -- 172.21.15.148:0/2984864838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18000094d0 con 0x7f180413f4a0 2023-10-30T13:49:21.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f1808f3d640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18040c6b80 0x7f180406a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:21.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180973e640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f180413f4a0 0x7f1804132600 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00c760 tx=0x7f17fc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:21.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f17f97fa640 1 -- 172.21.15.148:0/2984864838 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc003b80 con 0x7f180413f4a0 2023-10-30T13:49:21.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f17f97fa640 1 -- 172.21.15.148:0/2984864838 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17fc003ce0 con 0x7f180413f4a0 2023-10-30T13:49:21.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f17f97fa640 1 -- 172.21.15.148:0/2984864838 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc00aa30 con 0x7f180413f4a0 2023-10-30T13:49:21.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18041f4a80 con 0x7f180413f4a0 2023-10-30T13:49:21.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1804158210 con 0x7f180413f4a0 2023-10-30T13:49:21.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180413cce0 con 0x7f180413f4a0 2023-10-30T13:49:21.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f17f97fa640 1 -- 172.21.15.148:0/2984864838 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f17fc00ab90 con 0x7f180413f4a0 2023-10-30T13:49:21.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.852+0000 7f17f97fa640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17f00664a0 0x7f17f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:21.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.856+0000 7f17f97fa640 1 -- 172.21.15.148:0/2984864838 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f17fc0843c0 con 0x7f180413f4a0 2023-10-30T13:49:21.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.856+0000 7f1808f3d640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17f00664a0 0x7f17f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:21.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.856+0000 7f1808f3d640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17f00664a0 0x7f17f0068960 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f18000af000 tx=0x7f18000a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:21.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:21.860+0000 7f17f97fa640 1 -- 172.21.15.148:0/2984864838 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f180413cce0 con 0x7f180413f4a0 2023-10-30T13:49:22.042 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.040+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f180413cca0 con 0x7f180413f4a0 2023-10-30T13:49:22.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.040+0000 7f17f97fa640 1 -- 172.21.15.148:0/2984864838 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f17fc051050 con 0x7f180413f4a0 2023-10-30T13:49:22.043 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:22.044 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:20.023100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:22.044 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 152 2023-10-30T13:49:22.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17f00664a0 msgr2=0x7f17f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:22.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17f00664a0 0x7f17f0068960 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f18000af000 tx=0x7f18000a7000 comp rx=0 tx=0).stop 2023-10-30T13:49:22.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f180413f4a0 msgr2=0x7f1804132600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:22.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f180413f4a0 0x7f1804132600 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00c760 tx=0x7f17fc00cc20 comp rx=0 tx=0).stop 2023-10-30T13:49:22.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 shutdown_connections 2023-10-30T13:49:22.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17f00664a0 0x7f17f0068960 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f180413f4a0 0x7f1804132600 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f18040c6f60 0x7f180413cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 --2- 172.21.15.148:0/2984864838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18040c6b80 0x7f180406a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 >> 172.21.15.148:0/2984864838 conn(0x7f18040c7910 msgr2=0x7f18040c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:22.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 shutdown_connections 2023-10-30T13:49:22.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.044+0000 7f180b1a1640 1 -- 172.21.15.148:0/2984864838 wait complete. 2023-10-30T13:49:22.065 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-30T13:49:22.065 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:49:22.065 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:22.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd4a06a9640 1 Processor -- start 2023-10-30T13:49:22.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd4a06a9640 1 -- start start 2023-10-30T13:49:22.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd4a06a9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4980cf160 0x7fd4980cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:22.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd4a06a9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980cb800 con 0x7fd4981538a0 2023-10-30T13:49:22.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd4a06a9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4980cb330 con 0x7fd49815d6f0 2023-10-30T13:49:22.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd4a06a9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980cad70 con 0x7fd4980cf160 2023-10-30T13:49:22.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49e445640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd4981538a0 0x7fd4980ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36584/0 (socket says 172.21.15.148:36584) 2023-10-30T13:49:22.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49e445640 1 -- 172.21.15.148:0/377961021 learned_addr learned my addr 172.21.15.148:0/377961021 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:22.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49ec46640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4980cf160 0x7fd4980cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:22.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388129181 0 0) 0x7fd4980cb800 con 0x7fd4981538a0 2023-10-30T13:49:22.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd484003530 con 0x7fd4981538a0 2023-10-30T13:49:22.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632173791 0 0) 0x7fd484003530 con 0x7fd4981538a0 2023-10-30T13:49:22.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4980cf160 msgr2=0x7fd4980cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:22.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4980cf160 0x7fd4980cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 >> v1:172.21.15.150:6789/0 conn(0x7fd49815d6f0 legacy=0x7fd49815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:22.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4980cb050 con 0x7fd4981538a0 2023-10-30T13:49:22.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fd494001680 con 0x7fd4981538a0 2023-10-30T13:49:22.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 >> v1:172.21.15.148:6789/0 conn(0x7fd4981538a0 legacy=0x7fd4980ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:22.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840039b0 0x7fd484003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:22.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4840047d0 0x7fd484004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:22.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4980cf160 0x7fd484005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:22.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980cad70 con 0x7fd4840047d0 2023-10-30T13:49:22.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48400bad0 con 0x7fd4840039b0 2023-10-30T13:49:22.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48400bc90 con 0x7fd4980cf160 2023-10-30T13:49:22.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49dc44640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4840047d0 0x7fd484004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:22.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.204+0000 7fd49ec46640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4980cf160 0x7fd484005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:22.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49e445640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840039b0 0x7fd484003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:22.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49dc44640 1 -- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4980cf160 msgr2=0x7fd484005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:22.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49dc44640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4980cf160 0x7fd484005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49dc44640 1 -- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840039b0 msgr2=0x7fd484003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:22.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49dc44640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840039b0 0x7fd484003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49dc44640 1 -- 172.21.15.148:0/377961021 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd484003530 con 0x7fd4840047d0 2023-10-30T13:49:22.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49ec46640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd4980cf160 0x7fd484005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:22.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49dc44640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4840047d0 0x7fd484004c50 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fd488001750 tx=0x7fd48801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76ecd9a99413db0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:22.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4880052d0 con 0x7fd4840047d0 2023-10-30T13:49:22.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4880a7050 con 0x7fd4840047d0 2023-10-30T13:49:22.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49d443640 1 -- 172.21.15.148:0/377961021 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488003600 con 0x7fd4840047d0 2023-10-30T13:49:22.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd4a06a9640 1 -- 172.21.15.148:0/377961021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4840047d0 msgr2=0x7fd484004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:22.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd4a06a9640 1 --2- 172.21.15.148:0/377961021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4840047d0 0x7fd484004c50 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fd488001750 tx=0x7fd48801c3a0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.208+0000 7fd49ec46640 1 -- 172.21.15.148:0/377961021 reap_dead start 2023-10-30T13:49:22.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 -- 172.21.15.148:0/377961021 shutdown_connections 2023-10-30T13:49:22.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 -- 172.21.15.148:0/377961021 >> 172.21.15.148:0/377961021 conn(0x7fd49806bfe0 msgr2=0x7fd49806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:22.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 -- 172.21.15.148:0/377961021 shutdown_connections 2023-10-30T13:49:22.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 -- 172.21.15.148:0/377961021 wait complete. 2023-10-30T13:49:22.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 Processor -- start 2023-10-30T13:49:22.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 -- start start 2023-10-30T13:49:22.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840047d0 0x7fd49814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:22.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4981e3940 0x7fd4981e3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:22.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd49814db50 0x7fd49814dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:22.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4981e4f50 con 0x7fd4981e3940 2023-10-30T13:49:22.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4981e34f0 con 0x7fd4840047d0 2023-10-30T13:49:22.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd4a06a9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4981e6ac0 con 0x7fd49814db50 2023-10-30T13:49:22.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49dc44640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4981e3940 0x7fd4981e3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:22.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49dc44640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4981e3940 0x7fd4981e3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34508/0 (socket says 172.21.15.148:34508) 2023-10-30T13:49:22.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49dc44640 1 -- 172.21.15.148:0/1617148572 learned_addr learned my addr 172.21.15.148:0/1617148572 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:22.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49e445640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840047d0 0x7fd49814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:22.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49ec46640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd49814db50 0x7fd49814dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:22.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49dc44640 1 -- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd49814db50 msgr2=0x7fd49814dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:22.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49dc44640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd49814db50 0x7fd49814dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49dc44640 1 -- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840047d0 msgr2=0x7fd49814fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:22.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49dc44640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840047d0 0x7fd49814fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.212+0000 7fd49dc44640 1 -- 172.21.15.148:0/1617148572 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4981e42e0 con 0x7fd4981e3940 2023-10-30T13:49:22.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd49dc44640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4981e3940 0x7fd4981e3da0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fd488005040 tx=0x7fd484005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:22.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd48effd640 1 -- 172.21.15.148:0/1617148572 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488007a40 con 0x7fd4981e3940 2023-10-30T13:49:22.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd48effd640 1 -- 172.21.15.148:0/1617148572 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd488007680 con 0x7fd4981e3940 2023-10-30T13:49:22.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd498150350 con 0x7fd4981e3940 2023-10-30T13:49:22.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd48effd640 1 -- 172.21.15.148:0/1617148572 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488006a00 con 0x7fd4981e3940 2023-10-30T13:49:22.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4980c9a10 con 0x7fd4981e3940 2023-10-30T13:49:22.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4981e3660 con 0x7fd4981e3940 2023-10-30T13:49:22.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd48effd640 1 -- 172.21.15.148:0/1617148572 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd488006b60 con 0x7fd4981e3940 2023-10-30T13:49:22.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd48effd640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd47c0664d0 0x7fd47c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:22.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd48effd640 1 -- 172.21.15.148:0/1617148572 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fd48811ebc0 con 0x7fd4981e3940 2023-10-30T13:49:22.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.216+0000 7fd49e445640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd47c0664d0 0x7fd47c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:22.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.220+0000 7fd49e445640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd47c0664d0 0x7fd47c068990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd494001210 tx=0x7fd494007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:22.225 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.220+0000 7fd48effd640 1 -- 172.21.15.148:0/1617148572 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd4981e3660 con 0x7fd4981e3940 2023-10-30T13:49:22.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4981e3660 con 0x7fd4981e3940 2023-10-30T13:49:22.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd48effd640 1 -- 172.21.15.148:0/1617148572 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fd4981e3660 con 0x7fd4981e3940 2023-10-30T13:49:22.406 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:22.406 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:20.023100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:22.406 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 152 2023-10-30T13:49:22.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd47c0664d0 msgr2=0x7fd47c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:22.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd47c0664d0 0x7fd47c068990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd494001210 tx=0x7fd494007040 comp rx=0 tx=0).stop 2023-10-30T13:49:22.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4981e3940 msgr2=0x7fd4981e3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:22.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4981e3940 0x7fd4981e3da0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fd488005040 tx=0x7fd484005800 comp rx=0 tx=0).stop 2023-10-30T13:49:22.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 shutdown_connections 2023-10-30T13:49:22.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd47c0664d0 0x7fd47c068990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4981e3940 0x7fd4981e3da0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd49814db50 0x7fd49814dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 --2- 172.21.15.148:0/1617148572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4840047d0 0x7fd49814fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:22.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.404+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 >> 172.21.15.148:0/1617148572 conn(0x7fd49806bfe0 msgr2=0x7fd4980c49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:22.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.408+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 shutdown_connections 2023-10-30T13:49:22.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:22.408+0000 7fd4a06a9640 1 -- 172.21.15.148:0/1617148572 wait complete. 2023-10-30T13:49:22.428 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 152, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 152, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7812": { "addr": "172.21.15.148:6837/1530445724", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 1530445724, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 1530445724, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7812, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 38 }, "gid_8041": { "addr": "172.21.15.150:6836/1947781173", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1947781173, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 1947781173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8041, "incarnation": 152, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:49:20.023100+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7812, "mds_1": 8041 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:49:24.429 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:24.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7b01163640 1 Processor -- start 2023-10-30T13:49:24.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7b01163640 1 -- start start 2023-10-30T13:49:24.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7b01163640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7afc15cd50 0x7f7afc0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7b01163640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc0cefa0 0x7f7afc0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7b01163640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc15a6c0 con 0x7f7afc1538f0 2023-10-30T13:49:24.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7b01163640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7afc15d300 con 0x7f7afc15cd50 2023-10-30T13:49:24.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7b01163640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc15d5e0 con 0x7f7afc0cefa0 2023-10-30T13:49:24.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7afb7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7afc15cd50 0x7f7afc0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7b00962640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc0cefa0 0x7f7afc0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7afb7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7afc15cd50 0x7f7afc0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:51082/0 (socket says 172.21.15.148:51082) 2023-10-30T13:49:24.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7afb7fe640 1 -- 172.21.15.148:0/3665866604 learned_addr learned my addr 172.21.15.148:0/3665866604 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:24.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7afbfff640 1 --1- 172.21.15.148:0/3665866604 >> v1:172.21.15.150:6789/0 conn(0x7f7afc1538f0 0x7f7afc159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:43246/0 (socket says 172.21.15.148:43246) 2023-10-30T13:49:24.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7afb7fe640 1 -- 172.21.15.148:0/3665866604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc0cefa0 msgr2=0x7f7afc0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:24.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7afb7fe640 1 --2- 172.21.15.148:0/3665866604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc0cefa0 0x7f7afc0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7afb7fe640 1 -- 172.21.15.148:0/3665866604 >> v1:172.21.15.150:6789/0 conn(0x7f7afc1538f0 legacy=0x7f7afc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7afb7fe640 1 -- 172.21.15.148:0/3665866604 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc15d8c0 con 0x7f7afc15cd50 2023-10-30T13:49:24.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.580+0000 7f7b00962640 1 --2- 172.21.15.148:0/3665866604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc0cefa0 0x7f7afc0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:49:24.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7afb7fe640 1 --2- 172.21.15.148:0/3665866604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7afc15cd50 0x7f7afc0c9f90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009870 tx=0x7f7af001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13a6c1f6037548eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:24.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7afaffd640 1 -- 172.21.15.148:0/3665866604 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af00a8a30 con 0x7f7afc15cd50 2023-10-30T13:49:24.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7afaffd640 1 -- 172.21.15.148:0/3665866604 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af00a8b90 con 0x7f7afc15cd50 2023-10-30T13:49:24.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7afaffd640 1 -- 172.21.15.148:0/3665866604 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af00b0440 con 0x7f7afc15cd50 2023-10-30T13:49:24.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7b01163640 1 -- 172.21.15.148:0/3665866604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7afc15cd50 msgr2=0x7f7afc0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7b01163640 1 --2- 172.21.15.148:0/3665866604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7afc15cd50 0x7f7afc0c9f90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009870 tx=0x7f7af001ed30 comp rx=0 tx=0).stop 2023-10-30T13:49:24.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7b01163640 1 -- 172.21.15.148:0/3665866604 shutdown_connections 2023-10-30T13:49:24.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7b01163640 1 --2- 172.21.15.148:0/3665866604 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc0cefa0 0x7f7afc0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7b01163640 1 --2- 172.21.15.148:0/3665866604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7afc15cd50 0x7f7afc0c9f90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7b01163640 1 -- 172.21.15.148:0/3665866604 >> 172.21.15.148:0/3665866604 conn(0x7f7afc06a640 msgr2=0x7f7afc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:24.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7b01163640 1 -- 172.21.15.148:0/3665866604 shutdown_connections 2023-10-30T13:49:24.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.584+0000 7f7b01163640 1 -- 172.21.15.148:0/3665866604 wait complete. 2023-10-30T13:49:24.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 Processor -- start 2023-10-30T13:49:24.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 -- start start 2023-10-30T13:49:24.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7afc0cefa0 0x7f7afc1f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7afc1538f0 0x7f7afc0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc15cd50 0x7f7afc1527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc152520 con 0x7f7afc0cefa0 2023-10-30T13:49:24.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0ce000 con 0x7f7afc1538f0 2023-10-30T13:49:24.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc1f3620 con 0x7f7afc15cd50 2023-10-30T13:49:24.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7afc0cefa0 0x7f7afc1f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afbfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7afc0cefa0 0x7f7afc1f30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34510/0 (socket says 172.21.15.148:34510) 2023-10-30T13:49:24.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afbfff640 1 -- 172.21.15.148:0/2994858588 learned_addr learned my addr 172.21.15.148:0/2994858588 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:24.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afb7fe640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7afc1538f0 0x7f7afc0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b00962640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc15cd50 0x7f7afc1527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afbfff640 1 -- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc15cd50 msgr2=0x7f7afc1527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afbfff640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc15cd50 0x7f7afc1527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afbfff640 1 -- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7afc1538f0 msgr2=0x7f7afc0ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afbfff640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7afc1538f0 0x7f7afc0ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afbfff640 1 -- 172.21.15.148:0/2994858588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af4009a70 con 0x7f7afc0cefa0 2023-10-30T13:49:24.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7afbfff640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7afc0cefa0 0x7f7afc1f30e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c490 tx=0x7f7aec00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:24.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7af8ff9640 1 -- 172.21.15.148:0/2994858588 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec007480 con 0x7f7afc0cefa0 2023-10-30T13:49:24.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7af8ff9640 1 -- 172.21.15.148:0/2994858588 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec0075e0 con 0x7f7afc0cefa0 2023-10-30T13:49:24.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af00094d0 con 0x7f7afc0cefa0 2023-10-30T13:49:24.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7af8ff9640 1 -- 172.21.15.148:0/2994858588 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec004a10 con 0x7f7afc0cefa0 2023-10-30T13:49:24.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.588+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc0b44d0 con 0x7f7afc0cefa0 2023-10-30T13:49:24.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.592+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc04e170 con 0x7f7afc0cefa0 2023-10-30T13:49:24.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.592+0000 7f7af8ff9640 1 -- 172.21.15.148:0/2994858588 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7aec004b70 con 0x7f7afc0cefa0 2023-10-30T13:49:24.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.592+0000 7f7af8ff9640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ae80664a0 0x7f7ae8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:24.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.592+0000 7f7af8ff9640 1 -- 172.21.15.148:0/2994858588 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f7aec083aa0 con 0x7f7afc0cefa0 2023-10-30T13:49:24.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.592+0000 7f7afb7fe640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ae80664a0 0x7f7ae8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:24.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.592+0000 7f7afb7fe640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ae80664a0 0x7f7ae8068960 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f7af00b2000 tx=0x7f7af0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:24.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.596+0000 7f7af8ff9640 1 -- 172.21.15.148:0/2994858588 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7aec0507f0 con 0x7f7afc0cefa0 2023-10-30T13:49:24.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.776+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7afc14d7d0 con 0x7f7afc0cefa0 2023-10-30T13:49:24.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.776+0000 7f7af8ff9640 1 -- 172.21.15.148:0/2994858588 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f7aec018090 con 0x7f7afc0cefa0 2023-10-30T13:49:24.798 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:24.798 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:20.023100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:24.799 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 152 2023-10-30T13:49:24.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ae80664a0 msgr2=0x7f7ae8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ae80664a0 0x7f7ae8068960 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f7af00b2000 tx=0x7f7af0009040 comp rx=0 tx=0).stop 2023-10-30T13:49:24.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7afc0cefa0 msgr2=0x7f7afc1f30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:24.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7afc0cefa0 0x7f7afc1f30e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00c490 tx=0x7f7aec00c950 comp rx=0 tx=0).stop 2023-10-30T13:49:24.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 shutdown_connections 2023-10-30T13:49:24.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7ae80664a0 0x7f7ae8068960 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7afc15cd50 0x7f7afc1527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7afc1538f0 0x7f7afc0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 --2- 172.21.15.148:0/2994858588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7afc0cefa0 0x7f7afc1f30e0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:24.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 >> 172.21.15.148:0/2994858588 conn(0x7f7afc06a640 msgr2=0x7f7afc0c4850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:24.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 shutdown_connections 2023-10-30T13:49:24.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:24.780+0000 7f7b01163640 1 -- 172.21.15.148:0/2994858588 wait complete. 2023-10-30T13:49:24.803 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:26.804 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.944+0000 7f922c479640 1 Processor -- start 2023-10-30T13:49:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.944+0000 7f922c479640 1 -- start start 2023-10-30T13:49:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922c479640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 0x7f922406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922c479640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9224071110 0x7f9224158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922c479640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9224066cf0 con 0x7f92240c6bd0 2023-10-30T13:49:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922c479640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92240700c0 con 0x7f9224071110 2023-10-30T13:49:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922c479640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92240c70d0 con 0x7f922406f7a0 2023-10-30T13:49:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922a215640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 0x7f922406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f9229a14640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f92240c6bd0 0x7f9224069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:43256/0 (socket says 172.21.15.148:43256) 2023-10-30T13:49:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922aa16640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9224071110 0x7f9224158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f9229a14640 1 -- 172.21.15.148:0/3803962626 learned_addr learned my addr 172.21.15.148:0/3803962626 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f9229213640 1 -- 172.21.15.148:0/3803962626 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018451304 0 0) 0x7f9224066cf0 con 0x7f92240c6bd0 2023-10-30T13:49:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f9229213640 1 -- 172.21.15.148:0/3803962626 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9210003530 con 0x7f92240c6bd0 2023-10-30T13:49:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922a215640 1 -- 172.21.15.148:0/3803962626 >> v2:172.21.15.150:3300/0 conn(0x7f9224071110 msgr2=0x7f9224158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922a215640 1 --2- 172.21.15.148:0/3803962626 >> v2:172.21.15.150:3300/0 conn(0x7f9224071110 0x7f9224158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922a215640 1 -- 172.21.15.148:0/3803962626 >> v1:172.21.15.150:6789/0 conn(0x7f92240c6bd0 legacy=0x7f9224069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922a215640 1 -- 172.21.15.148:0/3803962626 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92240c7390 con 0x7f922406f7a0 2023-10-30T13:49:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.948+0000 7f922aa16640 1 --2- 172.21.15.148:0/3803962626 >> v2:172.21.15.150:3300/0 conn(0x7f9224071110 0x7f9224158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:49:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922a215640 1 --2- 172.21.15.148:0/3803962626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 0x7f922406fb80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f92200097c0 tx=0x7f922001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=920c93d6a74be0db server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f9229213640 1 -- 172.21.15.148:0/3803962626 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92200a8970 con 0x7f922406f7a0 2023-10-30T13:49:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f9229213640 1 -- 172.21.15.148:0/3803962626 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92200a8ad0 con 0x7f922406f7a0 2023-10-30T13:49:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f9229213640 1 -- 172.21.15.148:0/3803962626 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92200a8e40 con 0x7f922406f7a0 2023-10-30T13:49:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922c479640 1 -- 172.21.15.148:0/3803962626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 msgr2=0x7f922406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922c479640 1 --2- 172.21.15.148:0/3803962626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 0x7f922406fb80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f92200097c0 tx=0x7f922001ec70 comp rx=0 tx=0).stop 2023-10-30T13:49:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922c479640 1 -- 172.21.15.148:0/3803962626 shutdown_connections 2023-10-30T13:49:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922c479640 1 --2- 172.21.15.148:0/3803962626 >> v2:172.21.15.150:3300/0 conn(0x7f9224071110 0x7f9224158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922c479640 1 --2- 172.21.15.148:0/3803962626 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 0x7f922406fb80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922c479640 1 -- 172.21.15.148:0/3803962626 >> 172.21.15.148:0/3803962626 conn(0x7f92240c7910 msgr2=0x7f92240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922c479640 1 -- 172.21.15.148:0/3803962626 shutdown_connections 2023-10-30T13:49:26.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922c479640 1 -- 172.21.15.148:0/3803962626 wait complete. 2023-10-30T13:49:26.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.952+0000 7f922c479640 1 Processor -- start 2023-10-30T13:49:26.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 -- start start 2023-10-30T13:49:26.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 0x7f922415b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071110 0x7f92241565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92240c6bd0 0x7f9224156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92241eec20 con 0x7f92240c6bd0 2023-10-30T13:49:26.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92240c1cf0 con 0x7f9224071110 2023-10-30T13:49:26.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922415bbc0 con 0x7f922406f7a0 2023-10-30T13:49:26.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922aa16640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92240c6bd0 0x7f9224156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922aa16640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92240c6bd0 0x7f9224156ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34520/0 (socket says 172.21.15.148:34520) 2023-10-30T13:49:26.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922aa16640 1 -- 172.21.15.148:0/649645686 learned_addr learned my addr 172.21.15.148:0/649645686 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:26.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922a215640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 0x7f922415b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f9229a14640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071110 0x7f92241565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922aa16640 1 -- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 msgr2=0x7f922415b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922aa16640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 0x7f922415b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922aa16640 1 -- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071110 msgr2=0x7f92241565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:26.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922aa16640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071110 0x7f92241565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:26.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922aa16640 1 -- 172.21.15.148:0/649645686 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c009a70 con 0x7f92240c6bd0 2023-10-30T13:49:26.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f9229a14640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071110 0x7f92241565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:26.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922aa16640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92240c6bd0 0x7f9224156ae0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f921c009e60 tx=0x7f921c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:26.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f921affd640 1 -- 172.21.15.148:0/649645686 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c009cd0 con 0x7f92240c6bd0 2023-10-30T13:49:26.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f921affd640 1 -- 172.21.15.148:0/649645686 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c00f040 con 0x7f92240c6bd0 2023-10-30T13:49:26.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f921affd640 1 -- 172.21.15.148:0/649645686 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0039a0 con 0x7f92240c6bd0 2023-10-30T13:49:26.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9220009420 con 0x7f92240c6bd0 2023-10-30T13:49:26.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92240ae910 con 0x7f92240c6bd0 2023-10-30T13:49:26.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.956+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922404e170 con 0x7f92240c6bd0 2023-10-30T13:49:26.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.960+0000 7f921affd640 1 -- 172.21.15.148:0/649645686 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f921c014480 con 0x7f92240c6bd0 2023-10-30T13:49:26.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.960+0000 7f921affd640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92080664a0 0x7f9208068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:26.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.960+0000 7f921affd640 1 -- 172.21.15.148:0/649645686 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f921c0852a0 con 0x7f92240c6bd0 2023-10-30T13:49:26.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.960+0000 7f922a215640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92080664a0 0x7f9208068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:26.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.960+0000 7f922a215640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92080664a0 0x7f9208068960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f92200b2000 tx=0x7f92200aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:26.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:26.964+0000 7f921affd640 1 -- 172.21.15.148:0/649645686 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f921c051ea0 con 0x7f92240c6bd0 2023-10-30T13:49:27.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f922414e230 con 0x7f92240c6bd0 2023-10-30T13:49:27.152 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f921affd640 1 -- 172.21.15.148:0/649645686 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f921c0554c0 con 0x7f92240c6bd0 2023-10-30T13:49:27.153 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:27.153 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:25.985527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:resolve","state_seq":8,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:27.153 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 153 2023-10-30T13:49:27.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92080664a0 msgr2=0x7f9208068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:27.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92080664a0 0x7f9208068960 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f92200b2000 tx=0x7f92200aa070 comp rx=0 tx=0).stop 2023-10-30T13:49:27.155 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92240c6bd0 msgr2=0x7f9224156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:27.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92240c6bd0 0x7f9224156ae0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f921c009e60 tx=0x7f921c00c490 comp rx=0 tx=0).stop 2023-10-30T13:49:27.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 shutdown_connections 2023-10-30T13:49:27.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f92080664a0 0x7f9208068960 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:27.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f92240c6bd0 0x7f9224156ae0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:27.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071110 0x7f92241565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:27.156 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 --2- 172.21.15.148:0/649645686 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f922406f7a0 0x7f922415b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:27.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 >> 172.21.15.148:0/649645686 conn(0x7f92240c7910 msgr2=0x7f92240ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:27.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 shutdown_connections 2023-10-30T13:49:27.157 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:27.152+0000 7f922c479640 1 -- 172.21.15.148:0/649645686 wait complete. 2023-10-30T13:49:27.174 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:29.176 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:29.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.320+0000 7f8ebeb65640 1 Processor -- start 2023-10-30T13:49:29.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.320+0000 7f8ebeb65640 1 -- start start 2023-10-30T13:49:29.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebeb65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb80c6b80 0x7f8eb80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:29.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebeb65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8eb813f4a0 0x7f8eb80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:29.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebeb65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb80c0a50 con 0x7f8eb80c6f60 2023-10-30T13:49:29.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebeb65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb806f6e0 con 0x7f8eb813f4a0 2023-10-30T13:49:29.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebeb65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb806f9c0 con 0x7f8eb80c6b80 2023-10-30T13:49:29.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebc901640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb80c6b80 0x7f8eb80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:29.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebd102640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8eb813f4a0 0x7f8eb80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:29.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebd102640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8eb813f4a0 0x7f8eb80c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:51108/0 (socket says 172.21.15.148:51108) 2023-10-30T13:49:29.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8eaffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8eb80c6f60 0x7f8eb813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:43264/0 (socket says 172.21.15.148:43264) 2023-10-30T13:49:29.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebd102640 1 -- 172.21.15.148:0/3772750997 learned_addr learned my addr 172.21.15.148:0/3772750997 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:29.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebc901640 1 -- 172.21.15.148:0/3772750997 >> v2:172.21.15.150:3300/0 conn(0x7f8eb813f4a0 msgr2=0x7f8eb80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:29.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebc901640 1 --2- 172.21.15.148:0/3772750997 >> v2:172.21.15.150:3300/0 conn(0x7f8eb813f4a0 0x7f8eb80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebc901640 1 -- 172.21.15.148:0/3772750997 >> v1:172.21.15.150:6789/0 conn(0x7f8eb80c6f60 legacy=0x7f8eb813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:29.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebc901640 1 -- 172.21.15.148:0/3772750997 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb806fca0 con 0x7f8eb80c6b80 2023-10-30T13:49:29.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.324+0000 7f8ebd102640 1 --2- 172.21.15.148:0/3772750997 >> v2:172.21.15.150:3300/0 conn(0x7f8eb813f4a0 0x7f8eb80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:49:29.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebc901640 1 --2- 172.21.15.148:0/3772750997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb80c6b80 0x7f8eb80bfec0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40097c0 tx=0x7f8eb401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bbdac7d141a8013 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:29.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8eaf7fe640 1 -- 172.21.15.148:0/3772750997 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40ad640 con 0x7f8eb80c6b80 2023-10-30T13:49:29.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8eaf7fe640 1 -- 172.21.15.148:0/3772750997 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb40ad7a0 con 0x7f8eb80c6b80 2023-10-30T13:49:29.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8eaf7fe640 1 -- 172.21.15.148:0/3772750997 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40adb30 con 0x7f8eb80c6b80 2023-10-30T13:49:29.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 -- 172.21.15.148:0/3772750997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb80c6b80 msgr2=0x7f8eb80bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:29.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 --2- 172.21.15.148:0/3772750997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb80c6b80 0x7f8eb80bfec0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40097c0 tx=0x7f8eb401c480 comp rx=0 tx=0).stop 2023-10-30T13:49:29.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 -- 172.21.15.148:0/3772750997 shutdown_connections 2023-10-30T13:49:29.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 --2- 172.21.15.148:0/3772750997 >> v2:172.21.15.150:3300/0 conn(0x7f8eb813f4a0 0x7f8eb80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 --2- 172.21.15.148:0/3772750997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb80c6b80 0x7f8eb80bfec0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 -- 172.21.15.148:0/3772750997 >> 172.21.15.148:0/3772750997 conn(0x7f8eb80c7910 msgr2=0x7f8eb80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:29.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 -- 172.21.15.148:0/3772750997 shutdown_connections 2023-10-30T13:49:29.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 -- 172.21.15.148:0/3772750997 wait complete. 2023-10-30T13:49:29.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 Processor -- start 2023-10-30T13:49:29.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.328+0000 7f8ebeb65640 1 -- start start 2023-10-30T13:49:29.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebeb65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8eb80c6b80 0x7f8eb8134960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:29.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebeb65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80c6f60 0x7f8eb8135420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:29.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebeb65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb813f4a0 0x7f8eb8135960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:29.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebeb65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb8158bc0 con 0x7f8eb80c6b80 2023-10-30T13:49:29.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebeb65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb8158e60 con 0x7f8eb80c6f60 2023-10-30T13:49:29.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebeb65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb8155190 con 0x7f8eb813f4a0 2023-10-30T13:49:29.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebc901640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8eb80c6b80 0x7f8eb8134960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:29.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebc901640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8eb80c6b80 0x7f8eb8134960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34528/0 (socket says 172.21.15.148:34528) 2023-10-30T13:49:29.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebc901640 1 -- 172.21.15.148:0/4093822692 learned_addr learned my addr 172.21.15.148:0/4093822692 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:29.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8eaffff640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80c6f60 0x7f8eb8135420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:29.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebd102640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb813f4a0 0x7f8eb8135960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:29.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebc901640 1 -- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb813f4a0 msgr2=0x7f8eb8135960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:29.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebc901640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb813f4a0 0x7f8eb8135960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebc901640 1 -- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80c6f60 msgr2=0x7f8eb8135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:29.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebc901640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80c6f60 0x7f8eb8135420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebc901640 1 -- 172.21.15.148:0/4093822692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb0009b30 con 0x7f8eb80c6b80 2023-10-30T13:49:29.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebc901640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8eb80c6b80 0x7f8eb8134960 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40b0040 tx=0x7f8eb4019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:29.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ead7fa640 1 -- 172.21.15.148:0/4093822692 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40b6820 con 0x7f8eb80c6b80 2023-10-30T13:49:29.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ead7fa640 1 -- 172.21.15.148:0/4093822692 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb40a6040 con 0x7f8eb80c6b80 2023-10-30T13:49:29.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ead7fa640 1 -- 172.21.15.148:0/4093822692 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40b6be0 con 0x7f8eb80c6b80 2023-10-30T13:49:29.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb4009420 con 0x7f8eb80c6b80 2023-10-30T13:49:29.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb8155750 con 0x7f8eb80c6b80 2023-10-30T13:49:29.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb80705c0 con 0x7f8eb80c6b80 2023-10-30T13:49:29.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.332+0000 7f8ead7fa640 1 -- 172.21.15.148:0/4093822692 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8eb40b4040 con 0x7f8eb80c6b80 2023-10-30T13:49:29.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.336+0000 7f8ead7fa640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ea40664d0 0x7f8ea4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:29.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.336+0000 7f8ead7fa640 1 -- 172.21.15.148:0/4093822692 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f8eb411e8e0 con 0x7f8eb80c6b80 2023-10-30T13:49:29.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.336+0000 7f8eaffff640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ea40664d0 0x7f8ea4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:29.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.336+0000 7f8eaffff640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ea40664d0 0x7f8ea4068990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8eb80c5560 tx=0x7f8ea8007e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:29.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.340+0000 7f8ead7fa640 1 -- 172.21.15.148:0/4093822692 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8eb40eb620 con 0x7f8eb80c6b80 2023-10-30T13:49:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.520+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8eb804e170 con 0x7f8eb80c6b80 2023-10-30T13:49:29.524 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ead7fa640 1 -- 172.21.15.148:0/4093822692 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f8eb40b2020 con 0x7f8eb80c6b80 2023-10-30T13:49:29.524 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:29.524 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:25.985527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:resolve","state_seq":8,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:29.525 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 153 2023-10-30T13:49:29.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ea40664d0 msgr2=0x7f8ea4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:29.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ea40664d0 0x7f8ea4068990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8eb80c5560 tx=0x7f8ea8007e50 comp rx=0 tx=0).stop 2023-10-30T13:49:29.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8eb80c6b80 msgr2=0x7f8eb8134960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:29.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8eb80c6b80 0x7f8eb8134960 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40b0040 tx=0x7f8eb4019220 comp rx=0 tx=0).stop 2023-10-30T13:49:29.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 shutdown_connections 2023-10-30T13:49:29.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8ea40664d0 0x7f8ea4068990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8eb813f4a0 0x7f8eb8135960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8eb80c6f60 0x7f8eb8135420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 --2- 172.21.15.148:0/4093822692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8eb80c6b80 0x7f8eb8134960 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:29.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 >> 172.21.15.148:0/4093822692 conn(0x7f8eb80c7910 msgr2=0x7f8eb81532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:29.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 shutdown_connections 2023-10-30T13:49:29.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:29.524+0000 7f8ebeb65640 1 -- 172.21.15.148:0/4093822692 wait complete. 2023-10-30T13:49:29.545 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:31.546 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:31.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.691+0000 7f2c7b71d640 1 Processor -- start 2023-10-30T13:49:31.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.691+0000 7f2c7b71d640 1 -- start start 2023-10-30T13:49:31.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.691+0000 7f2c7b71d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7406f7a0 0x7f2c7406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.691+0000 7f2c7b71d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c74066cf0 con 0x7f2c74071110 2023-10-30T13:49:31.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.691+0000 7f2c7b71d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c740c70d0 con 0x7f2c740c6bd0 2023-10-30T13:49:31.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.691+0000 7f2c7b71d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c740700c0 con 0x7f2c7406f7a0 2023-10-30T13:49:31.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.691+0000 7f2c79cba640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2c74071110 0x7f2c74158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36590/0 (socket says 172.21.15.148:36590) 2023-10-30T13:49:31.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.691+0000 7f2c79cba640 1 -- 172.21.15.148:0/3841986848 learned_addr learned my addr 172.21.15.148:0/3841986848 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:31.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.691+0000 7f2c794b9640 1 --2- 172.21.15.148:0/3841986848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7406f7a0 0x7f2c7406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.695+0000 7f2c6bfff640 1 -- 172.21.15.148:0/3841986848 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300359979 0 0) 0x7f2c74066cf0 con 0x7f2c74071110 2023-10-30T13:49:31.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.695+0000 7f2c6bfff640 1 -- 172.21.15.148:0/3841986848 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c60003530 con 0x7f2c74071110 2023-10-30T13:49:31.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.695+0000 7f2c6bfff640 1 -- 172.21.15.148:0/3841986848 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798456035 0 0) 0x7f2c740c70d0 con 0x7f2c740c6bd0 2023-10-30T13:49:31.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.695+0000 7f2c6bfff640 1 -- 172.21.15.148:0/3841986848 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c74066cf0 con 0x7f2c740c6bd0 2023-10-30T13:49:31.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.695+0000 7f2c6bfff640 1 -- 172.21.15.148:0/3841986848 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2853630119 0 0) 0x7f2c60003530 con 0x7f2c74071110 2023-10-30T13:49:31.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.695+0000 7f2c794b9640 1 -- 172.21.15.148:0/3841986848 >> v1:172.21.15.150:6789/0 conn(0x7f2c740c6bd0 legacy=0x7f2c74069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.695+0000 7f2c794b9640 1 -- 172.21.15.148:0/3841986848 >> v1:172.21.15.148:6789/0 conn(0x7f2c74071110 legacy=0x7f2c74158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.695+0000 7f2c794b9640 1 -- 172.21.15.148:0/3841986848 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c740c7390 con 0x7f2c7406f7a0 2023-10-30T13:49:31.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c794b9640 1 --2- 172.21.15.148:0/3841986848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7406f7a0 0x7f2c7406fb80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2c700097f0 tx=0x7f2c7001ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92639774c74d0886 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:31.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c6bfff640 1 -- 172.21.15.148:0/3841986848 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c001a00 con 0x7f2c7406f7a0 2023-10-30T13:49:31.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c6bfff640 1 -- 172.21.15.148:0/3841986848 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c700a8650 con 0x7f2c7406f7a0 2023-10-30T13:49:31.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c6bfff640 1 -- 172.21.15.148:0/3841986848 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c700a89c0 con 0x7f2c7406f7a0 2023-10-30T13:49:31.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3841986848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7406f7a0 msgr2=0x7f2c7406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c7b71d640 1 --2- 172.21.15.148:0/3841986848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7406f7a0 0x7f2c7406fb80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2c700097f0 tx=0x7f2c7001ebd0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3841986848 shutdown_connections 2023-10-30T13:49:31.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c7b71d640 1 --2- 172.21.15.148:0/3841986848 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c7406f7a0 0x7f2c7406fb80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3841986848 >> 172.21.15.148:0/3841986848 conn(0x7f2c740c7910 msgr2=0x7f2c740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:31.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3841986848 shutdown_connections 2023-10-30T13:49:31.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3841986848 wait complete. 2023-10-30T13:49:31.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c7b71d640 1 Processor -- start 2023-10-30T13:49:31.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.699+0000 7f2c7b71d640 1 -- start start 2023-10-30T13:49:31.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c7b71d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c7406f7a0 0x7f2c741f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c7b71d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c74071110 0x7f2c741eec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c7b71d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740c6bd0 0x7f2c7414d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c7b71d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c741ef1c0 con 0x7f2c7406f7a0 2023-10-30T13:49:31.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c7b71d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c741f28e0 con 0x7f2c74071110 2023-10-30T13:49:31.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c7b71d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7414dc40 con 0x7f2c740c6bd0 2023-10-30T13:49:31.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c794b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c7406f7a0 0x7f2c741f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c794b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c7406f7a0 0x7f2c741f2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:34540/0 (socket says 172.21.15.148:34540) 2023-10-30T13:49:31.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c794b9640 1 -- 172.21.15.148:0/3427527638 learned_addr learned my addr 172.21.15.148:0/3427527638 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:31.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c78cb8640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c74071110 0x7f2c741eec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c79cba640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740c6bd0 0x7f2c7414d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c794b9640 1 -- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740c6bd0 msgr2=0x7f2c7414d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c794b9640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740c6bd0 0x7f2c7414d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c794b9640 1 -- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c74071110 msgr2=0x7f2c741eec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c794b9640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c74071110 0x7f2c741eec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c794b9640 1 -- 172.21.15.148:0/3427527638 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c70009420 con 0x7f2c7406f7a0 2023-10-30T13:49:31.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c79cba640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740c6bd0 0x7f2c7414d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:49:31.708 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c794b9640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c7406f7a0 0x7f2c741f2d70 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f2c700b3040 tx=0x7f2c7001b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:31.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c69ffb640 1 -- 172.21.15.148:0/3427527638 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c700a8bd0 con 0x7f2c7406f7a0 2023-10-30T13:49:31.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c69ffb640 1 -- 172.21.15.148:0/3427527638 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c700a8d30 con 0x7f2c7406f7a0 2023-10-30T13:49:31.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c69ffb640 1 -- 172.21.15.148:0/3427527638 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c700a8ea0 con 0x7f2c7406f7a0 2023-10-30T13:49:31.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c740c1ea0 con 0x7f2c7406f7a0 2023-10-30T13:49:31.709 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c740c1bc0 con 0x7f2c7406f7a0 2023-10-30T13:49:31.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7404e170 con 0x7f2c7406f7a0 2023-10-30T13:49:31.710 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.703+0000 7f2c69ffb640 1 -- 172.21.15.148:0/3427527638 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c700b7040 con 0x7f2c7406f7a0 2023-10-30T13:49:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.707+0000 7f2c69ffb640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2c580664a0 0x7f2c58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.707+0000 7f2c69ffb640 1 -- 172.21.15.148:0/3427527638 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f2c70120e30 con 0x7f2c7406f7a0 2023-10-30T13:49:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.707+0000 7f2c78cb8640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2c580664a0 0x7f2c58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:31.711 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.707+0000 7f2c78cb8640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2c580664a0 0x7f2c58068960 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2c7415bd60 tx=0x7f2c64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:31.712 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.711+0000 7f2c69ffb640 1 -- 172.21.15.148:0/3427527638 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c700edac0 con 0x7f2c7406f7a0 2023-10-30T13:49:31.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c740c8400 con 0x7f2c7406f7a0 2023-10-30T13:49:31.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c69ffb640 1 -- 172.21.15.148:0/3427527638 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f2c700f10e0 con 0x7f2c7406f7a0 2023-10-30T13:49:31.894 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:31.894 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:25.985527+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:resolve","state_seq":8,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:31.895 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 153 2023-10-30T13:49:31.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2c580664a0 msgr2=0x7f2c58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2c580664a0 0x7f2c58068960 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2c7415bd60 tx=0x7f2c64009040 comp rx=0 tx=0).stop 2023-10-30T13:49:31.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c7406f7a0 msgr2=0x7f2c741f2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:31.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c7406f7a0 0x7f2c741f2d70 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f2c700b3040 tx=0x7f2c7001b970 comp rx=0 tx=0).stop 2023-10-30T13:49:31.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 shutdown_connections 2023-10-30T13:49:31.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2c580664a0 0x7f2c58068960 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c740c6bd0 0x7f2c7414d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c74071110 0x7f2c741eec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 --2- 172.21.15.148:0/3427527638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c7406f7a0 0x7f2c741f2d70 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:31.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 >> 172.21.15.148:0/3427527638 conn(0x7f2c740c7910 msgr2=0x7f2c740c8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:31.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 shutdown_connections 2023-10-30T13:49:31.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:31.891+0000 7f2c7b71d640 1 -- 172.21.15.148:0/3427527638 wait complete. 2023-10-30T13:49:31.915 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:33.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:33.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:49:33.916 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:34.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc197ad6640 1 Processor -- start 2023-10-30T13:49:34.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc197ad6640 1 -- start start 2023-10-30T13:49:34.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc197ad6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc1900c6bd0 0x7fc19006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:34.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc197ad6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc190071110 0x7fc190158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:34.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc197ad6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc190066cf0 con 0x7fc19006f7a0 2023-10-30T13:49:34.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc197ad6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1900700c0 con 0x7fc190071110 2023-10-30T13:49:34.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc197ad6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1900c70d0 con 0x7fc1900c6bd0 2023-10-30T13:49:34.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc196073640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc190071110 0x7fc190158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:34.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc196073640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fc190071110 0x7fc190158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56774/0 (socket says 172.21.15.148:56774) 2023-10-30T13:49:34.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc196073640 1 -- 172.21.15.148:0/2714633172 learned_addr learned my addr 172.21.15.148:0/2714633172 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:34.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc195071640 1 --2- 172.21.15.148:0/2714633172 >> v2:172.21.15.150:3300/0 conn(0x7fc1900c6bd0 0x7fc19006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:34.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc196073640 1 -- 172.21.15.148:0/2714633172 >> v2:172.21.15.150:3300/0 conn(0x7fc1900c6bd0 msgr2=0x7fc19006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:34.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc196073640 1 --2- 172.21.15.148:0/2714633172 >> v2:172.21.15.150:3300/0 conn(0x7fc1900c6bd0 0x7fc19006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:34.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc196073640 1 -- 172.21.15.148:0/2714633172 >> v1:172.21.15.150:6789/0 conn(0x7fc19006f7a0 legacy=0x7fc190069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:34.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.063+0000 7fc196073640 1 -- 172.21.15.148:0/2714633172 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1900c7390 con 0x7fc190071110 2023-10-30T13:49:34.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc196073640 1 --2- 172.21.15.148:0/2714633172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc190071110 0x7fc190158110 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fc19006f1c0 tx=0x7fc18801f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8247087d81e2fdf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:34.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc194870640 1 -- 172.21.15.148:0/2714633172 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1880af400 con 0x7fc190071110 2023-10-30T13:49:34.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc194870640 1 -- 172.21.15.148:0/2714633172 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1880ab050 con 0x7fc190071110 2023-10-30T13:49:34.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc194870640 1 -- 172.21.15.148:0/2714633172 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1880af780 con 0x7fc190071110 2023-10-30T13:49:34.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc197ad6640 1 -- 172.21.15.148:0/2714633172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc190071110 msgr2=0x7fc190158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:34.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc197ad6640 1 --2- 172.21.15.148:0/2714633172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc190071110 0x7fc190158110 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fc19006f1c0 tx=0x7fc18801f180 comp rx=0 tx=0).stop 2023-10-30T13:49:34.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc197ad6640 1 -- 172.21.15.148:0/2714633172 shutdown_connections 2023-10-30T13:49:34.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc197ad6640 1 --2- 172.21.15.148:0/2714633172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc190071110 0x7fc190158110 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:34.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc197ad6640 1 --2- 172.21.15.148:0/2714633172 >> v2:172.21.15.150:3300/0 conn(0x7fc1900c6bd0 0x7fc19006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:34.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc197ad6640 1 -- 172.21.15.148:0/2714633172 >> 172.21.15.148:0/2714633172 conn(0x7fc1900c7910 msgr2=0x7fc1900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:34.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc197ad6640 1 -- 172.21.15.148:0/2714633172 shutdown_connections 2023-10-30T13:49:34.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.067+0000 7fc197ad6640 1 -- 172.21.15.148:0/2714633172 wait complete. 2023-10-30T13:49:34.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 Processor -- start 2023-10-30T13:49:34.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 -- start start 2023-10-30T13:49:34.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc19006f7a0 0x7fc1901e38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:34.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc190071110 0x7fc1900c0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:34.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1900c6bd0 0x7fc19014f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:34.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc190153180 con 0x7fc1900c6bd0 2023-10-30T13:49:34.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1901efd00 con 0x7fc19006f7a0 2023-10-30T13:49:34.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900bfec0 con 0x7fc190071110 2023-10-30T13:49:34.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc196073640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1900c6bd0 0x7fc19014f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:34.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc196073640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1900c6bd0 0x7fc19014f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56788/0 (socket says 172.21.15.148:56788) 2023-10-30T13:49:34.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc196073640 1 -- 172.21.15.148:0/603760307 learned_addr learned my addr 172.21.15.148:0/603760307 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:34.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc195872640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc19006f7a0 0x7fc1901e38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:34.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc195071640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc190071110 0x7fc1900c0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:34.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc196073640 1 -- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc190071110 msgr2=0x7fc1900c0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:34.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc196073640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc190071110 0x7fc1900c0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:34.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc196073640 1 -- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc19006f7a0 msgr2=0x7fc1901e38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:34.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc196073640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc19006f7a0 0x7fc1901e38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:34.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc196073640 1 -- 172.21.15.148:0/603760307 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc188009b30 con 0x7fc1900c6bd0 2023-10-30T13:49:34.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc196073640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1900c6bd0 0x7fc19014f670 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc18801cdb0 tx=0x7fc18801bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:34.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc1867fc640 1 -- 172.21.15.148:0/603760307 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1880afde0 con 0x7fc1900c6bd0 2023-10-30T13:49:34.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc1867fc640 1 -- 172.21.15.148:0/603760307 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc188009510 con 0x7fc1900c6bd0 2023-10-30T13:49:34.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc1867fc640 1 -- 172.21.15.148:0/603760307 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1880ba660 con 0x7fc1900c6bd0 2023-10-30T13:49:34.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1900c0160 con 0x7fc1900c6bd0 2023-10-30T13:49:34.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.071+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1900c0720 con 0x7fc1900c6bd0 2023-10-30T13:49:34.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.075+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1900c8720 con 0x7fc1900c6bd0 2023-10-30T13:49:34.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.075+0000 7fc1867fc640 1 -- 172.21.15.148:0/603760307 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc1880b6040 con 0x7fc1900c6bd0 2023-10-30T13:49:34.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.075+0000 7fc1867fc640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1740664a0 0x7fc174068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:34.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.075+0000 7fc1867fc640 1 -- 172.21.15.148:0/603760307 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fc188122bd0 con 0x7fc1900c6bd0 2023-10-30T13:49:34.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.075+0000 7fc195872640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1740664a0 0x7fc174068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:34.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.075+0000 7fc195872640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1740664a0 0x7fc174068960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0098e0 tx=0x7fc18c009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:34.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.079+0000 7fc1867fc640 1 -- 172.21.15.148:0/603760307 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc1880ef9a0 con 0x7fc1900c6bd0 2023-10-30T13:49:34.264 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc19004e170 con 0x7fc1900c6bd0 2023-10-30T13:49:34.265 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc1867fc640 1 -- 172.21.15.148:0/603760307 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fc1880f2fc0 con 0x7fc1900c6bd0 2023-10-30T13:49:34.265 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:34.266 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:31.223482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:34.266 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 154 2023-10-30T13:49:34.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1740664a0 msgr2=0x7fc174068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:34.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1740664a0 0x7fc174068960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0098e0 tx=0x7fc18c009060 comp rx=0 tx=0).stop 2023-10-30T13:49:34.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1900c6bd0 msgr2=0x7fc19014f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:34.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1900c6bd0 0x7fc19014f670 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fc18801cdb0 tx=0x7fc18801bf20 comp rx=0 tx=0).stop 2023-10-30T13:49:34.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 shutdown_connections 2023-10-30T13:49:34.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc1740664a0 0x7fc174068960 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:34.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1900c6bd0 0x7fc19014f670 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:34.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc190071110 0x7fc1900c0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:34.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 --2- 172.21.15.148:0/603760307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc19006f7a0 0x7fc1901e38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:34.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 >> 172.21.15.148:0/603760307 conn(0x7fc1900c7910 msgr2=0x7fc1900c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:34.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 shutdown_connections 2023-10-30T13:49:34.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:34.263+0000 7fc197ad6640 1 -- 172.21.15.148:0/603760307 wait complete. 2023-10-30T13:49:34.287 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7812": { "addr": "172.21.15.148:6837/1530445724", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6836", "nonce": 1530445724, "type": "v2" }, { "addr": "172.21.15.148:6837", "nonce": 1530445724, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 7812, "incarnation": 144, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 38 }, "gid_8041": { "addr": "172.21.15.150:6836/1947781173", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1947781173, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 1947781173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8041, "incarnation": 152, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:reconnect", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:49:31.223482+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 7812, "mds_1": 8041 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:49:36.288 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:36.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd69b009640 1 Processor -- start 2023-10-30T13:49:36.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd69b009640 1 -- start start 2023-10-30T13:49:36.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd69b009640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6941539a0 0x7fd69415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd69b009640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd694069e90 0x7fd69406a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd69b009640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69415a8c0 con 0x7fd694070a00 2023-10-30T13:49:36.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd69b009640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69406fbb0 con 0x7fd6941539a0 2023-10-30T13:49:36.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd69b009640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd694153d80 con 0x7fd694069e90 2023-10-30T13:49:36.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd698da5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd6941539a0 0x7fd69415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd693fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd694070a00 0x7fd694070e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:57306/0 (socket says 172.21.15.148:57306) 2023-10-30T13:49:36.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd693fff640 1 -- 172.21.15.148:0/1485131591 learned_addr learned my addr 172.21.15.148:0/1485131591 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:36.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd6995a6640 1 --2- 172.21.15.148:0/1485131591 >> v2:172.21.15.150:3300/0 conn(0x7fd694069e90 0x7fd69406a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd6937fe640 1 -- 172.21.15.148:0/1485131591 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554143454 0 0) 0x7fd69415a8c0 con 0x7fd694070a00 2023-10-30T13:49:36.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd6937fe640 1 -- 172.21.15.148:0/1485131591 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd684003530 con 0x7fd694070a00 2023-10-30T13:49:36.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd698da5640 1 -- 172.21.15.148:0/1485131591 >> v2:172.21.15.150:3300/0 conn(0x7fd694069e90 msgr2=0x7fd69406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:36.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd698da5640 1 --2- 172.21.15.148:0/1485131591 >> v2:172.21.15.150:3300/0 conn(0x7fd694069e90 0x7fd69406a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd698da5640 1 -- 172.21.15.148:0/1485131591 >> v1:172.21.15.148:6789/0 conn(0x7fd694070a00 legacy=0x7fd694070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.435+0000 7fd698da5640 1 -- 172.21.15.148:0/1485131591 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940c5c50 con 0x7fd6941539a0 2023-10-30T13:49:36.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd698da5640 1 --2- 172.21.15.148:0/1485131591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 0x7fd69415a380 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fd6940c8900 tx=0x7fd68801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676bfe1b2840f722 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:36.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd6937fe640 1 -- 172.21.15.148:0/1485131591 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6880ab600 con 0x7fd6941539a0 2023-10-30T13:49:36.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd6937fe640 1 -- 172.21.15.148:0/1485131591 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6880a7050 con 0x7fd6941539a0 2023-10-30T13:49:36.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd6937fe640 1 -- 172.21.15.148:0/1485131591 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6880ab980 con 0x7fd6941539a0 2023-10-30T13:49:36.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd69b009640 1 -- 172.21.15.148:0/1485131591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 msgr2=0x7fd69415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd69b009640 1 --2- 172.21.15.148:0/1485131591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 0x7fd69415a380 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fd6940c8900 tx=0x7fd68801c3a0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd69b009640 1 -- 172.21.15.148:0/1485131591 shutdown_connections 2023-10-30T13:49:36.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd69b009640 1 --2- 172.21.15.148:0/1485131591 >> v2:172.21.15.150:3300/0 conn(0x7fd694069e90 0x7fd69406a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd69b009640 1 --2- 172.21.15.148:0/1485131591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 0x7fd69415a380 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd69b009640 1 -- 172.21.15.148:0/1485131591 >> 172.21.15.148:0/1485131591 conn(0x7fd6940c7910 msgr2=0x7fd6940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:36.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd69b009640 1 -- 172.21.15.148:0/1485131591 shutdown_connections 2023-10-30T13:49:36.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.439+0000 7fd69b009640 1 -- 172.21.15.148:0/1485131591 wait complete. 2023-10-30T13:49:36.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 Processor -- start 2023-10-30T13:49:36.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 -- start start 2023-10-30T13:49:36.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694069e90 0x7fd694146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd694070a00 0x7fd694147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 0x7fd69415a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6941476b0 con 0x7fd6941539a0 2023-10-30T13:49:36.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6941f2200 con 0x7fd694069e90 2023-10-30T13:49:36.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6941f24e0 con 0x7fd694070a00 2023-10-30T13:49:36.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6995a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 0x7fd69415a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6995a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 0x7fd69415a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56800/0 (socket says 172.21.15.148:56800) 2023-10-30T13:49:36.449 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6995a6640 1 -- 172.21.15.148:0/1591506146 learned_addr learned my addr 172.21.15.148:0/1591506146 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:36.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd698da5640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694069e90 0x7fd694146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd693fff640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd694070a00 0x7fd694147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6995a6640 1 -- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd694070a00 msgr2=0x7fd694147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6995a6640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd694070a00 0x7fd694147070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6995a6640 1 -- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694069e90 msgr2=0x7fd694146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6995a6640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694069e90 0x7fd694146560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6995a6640 1 -- 172.21.15.148:0/1591506146 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688009420 con 0x7fd6941539a0 2023-10-30T13:49:36.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd698da5640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694069e90 0x7fd694146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:36.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6995a6640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 0x7fd69415a2a0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00efc0 tx=0x7fd68c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:36.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6917fa640 1 -- 172.21.15.148:0/1591506146 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c009ba0 con 0x7fd6941539a0 2023-10-30T13:49:36.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6917fa640 1 -- 172.21.15.148:0/1591506146 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c00f040 con 0x7fd6941539a0 2023-10-30T13:49:36.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6941f27c0 con 0x7fd6941539a0 2023-10-30T13:49:36.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd6917fa640 1 -- 172.21.15.148:0/1591506146 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c0146c0 con 0x7fd6941539a0 2023-10-30T13:49:36.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6941f1ef0 con 0x7fd6941539a0 2023-10-30T13:49:36.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.443+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69404e170 con 0x7fd6941539a0 2023-10-30T13:49:36.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.447+0000 7fd6917fa640 1 -- 172.21.15.148:0/1591506146 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd68c014820 con 0x7fd6941539a0 2023-10-30T13:49:36.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.447+0000 7fd6917fa640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6780664d0 0x7fd678068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:36.453 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.447+0000 7fd6917fa640 1 -- 172.21.15.148:0/1591506146 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fd68c085120 con 0x7fd6941539a0 2023-10-30T13:49:36.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.447+0000 7fd698da5640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6780664d0 0x7fd678068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:36.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.447+0000 7fd698da5640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6780664d0 0x7fd678068990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fd6880ad000 tx=0x7fd688008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:36.454 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.451+0000 7fd6917fa640 1 -- 172.21.15.148:0/1591506146 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd68c051db0 con 0x7fd6941539a0 2023-10-30T13:49:36.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.631+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6940c67a0 con 0x7fd6941539a0 2023-10-30T13:49:36.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd6917fa640 1 -- 172.21.15.148:0/1591506146 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5741 (secure 0 0 0) 0x7fd68c018090 con 0x7fd6941539a0 2023-10-30T13:49:36.636 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:36.636 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:31.223482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:36.638 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 154 2023-10-30T13:49:36.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6780664d0 msgr2=0x7fd678068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6780664d0 0x7fd678068990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fd6880ad000 tx=0x7fd688008040 comp rx=0 tx=0).stop 2023-10-30T13:49:36.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 msgr2=0x7fd69415a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:36.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 0x7fd69415a2a0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00efc0 tx=0x7fd68c00c490 comp rx=0 tx=0).stop 2023-10-30T13:49:36.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 shutdown_connections 2023-10-30T13:49:36.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd6780664d0 0x7fd678068990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6941539a0 0x7fd69415a2a0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd694070a00 0x7fd694147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 --2- 172.21.15.148:0/1591506146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694069e90 0x7fd694146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:36.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 >> 172.21.15.148:0/1591506146 conn(0x7fd6940c7910 msgr2=0x7fd6941523f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:36.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 shutdown_connections 2023-10-30T13:49:36.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:36.635+0000 7fd69b009640 1 -- 172.21.15.148:0/1591506146 wait complete. 2023-10-30T13:49:36.657 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:38.658 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:38.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.803+0000 7f6c421d3640 1 Processor -- start 2023-10-30T13:49:38.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.803+0000 7f6c421d3640 1 -- start start 2023-10-30T13:49:38.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c421d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c070740 0x7f6c3c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c421d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6c3c15e700 0x7f6c3c06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c421d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0c5810 con 0x7f6c3c1580e0 2023-10-30T13:49:38.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c421d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c070b60 con 0x7f6c3c15e700 2023-10-30T13:49:38.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c421d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c15eb90 con 0x7f6c3c070740 2023-10-30T13:49:38.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c3bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6c3c15e700 0x7f6c3c06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c3bfff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6c3c15e700 0x7f6c3c06a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56802/0 (socket says 172.21.15.148:56802) 2023-10-30T13:49:38.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c3b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6c3c1580e0 0x7f6c3c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:57322/0 (socket says 172.21.15.148:57322) 2023-10-30T13:49:38.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c3bfff640 1 -- 172.21.15.148:0/3872265422 learned_addr learned my addr 172.21.15.148:0/3872265422 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:38.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c33fff640 1 --2- 172.21.15.148:0/3872265422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c070740 0x7f6c3c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c3bfff640 1 -- 172.21.15.148:0/3872265422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c070740 msgr2=0x7f6c3c0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:38.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c3bfff640 1 --2- 172.21.15.148:0/3872265422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c070740 0x7f6c3c0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c3bfff640 1 -- 172.21.15.148:0/3872265422 >> v1:172.21.15.148:6789/0 conn(0x7f6c3c1580e0 legacy=0x7f6c3c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c3bfff640 1 -- 172.21.15.148:0/3872265422 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c0c6bb0 con 0x7f6c3c15e700 2023-10-30T13:49:38.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.807+0000 7f6c3affd640 1 -- 172.21.15.148:0/3872265422 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834052458 0 0) 0x7f6c3c0c5810 con 0x7f6c3c1580e0 2023-10-30T13:49:38.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c3bfff640 1 --2- 172.21.15.148:0/3872265422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c15e700 0x7f6c3c06a5a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009e50 tx=0x7f6c3401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cadc57716520d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:38.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c3affd640 1 -- 172.21.15.148:0/3872265422 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340a58a0 con 0x7f6c3c15e700 2023-10-30T13:49:38.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c3affd640 1 -- 172.21.15.148:0/3872265422 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c340a5a00 con 0x7f6c3c15e700 2023-10-30T13:49:38.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c3affd640 1 -- 172.21.15.148:0/3872265422 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340a5d90 con 0x7f6c3c15e700 2023-10-30T13:49:38.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 -- 172.21.15.148:0/3872265422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c15e700 msgr2=0x7f6c3c06a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 --2- 172.21.15.148:0/3872265422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c15e700 0x7f6c3c06a5a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009e50 tx=0x7f6c3401c9e0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 -- 172.21.15.148:0/3872265422 shutdown_connections 2023-10-30T13:49:38.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 --2- 172.21.15.148:0/3872265422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c15e700 0x7f6c3c06a5a0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 --2- 172.21.15.148:0/3872265422 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c070740 0x7f6c3c0c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 -- 172.21.15.148:0/3872265422 >> 172.21.15.148:0/3872265422 conn(0x7f6c3c0c7910 msgr2=0x7f6c3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:38.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 -- 172.21.15.148:0/3872265422 shutdown_connections 2023-10-30T13:49:38.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 -- 172.21.15.148:0/3872265422 wait complete. 2023-10-30T13:49:38.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 Processor -- start 2023-10-30T13:49:38.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.811+0000 7f6c421d3640 1 -- start start 2023-10-30T13:49:38.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c421d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3c070740 0x7f6c3c1f9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c421d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c1580e0 0x7f6c3c06bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c421d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c15e700 0x7f6c3c14ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c421d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c15efc0 con 0x7f6c3c1580e0 2023-10-30T13:49:38.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c421d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c15f2a0 con 0x7f6c3c070740 2023-10-30T13:49:38.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c421d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c15f580 con 0x7f6c3c15e700 2023-10-30T13:49:38.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c33fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c1580e0 0x7f6c3c06bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c33fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c1580e0 0x7f6c3c06bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56806/0 (socket says 172.21.15.148:56806) 2023-10-30T13:49:38.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c33fff640 1 -- 172.21.15.148:0/1032760264 learned_addr learned my addr 172.21.15.148:0/1032760264 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:38.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c3b7fe640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3c070740 0x7f6c3c1f9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c3bfff640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c15e700 0x7f6c3c14ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c33fff640 1 -- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c15e700 msgr2=0x7f6c3c14ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c33fff640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c15e700 0x7f6c3c14ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c33fff640 1 -- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3c070740 msgr2=0x7f6c3c1f9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:38.821 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c33fff640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3c070740 0x7f6c3c1f9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:38.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c33fff640 1 -- 172.21.15.148:0/1032760264 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c34009b30 con 0x7f6c3c1580e0 2023-10-30T13:49:38.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c3b7fe640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3c070740 0x7f6c3c1f9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:38.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c33fff640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c1580e0 0x7f6c3c06bbb0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400d800 tx=0x7f6c2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:38.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c38ff9640 1 -- 172.21.15.148:0/1032760264 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2400b6c0 con 0x7f6c3c1580e0 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c38ff9640 1 -- 172.21.15.148:0/1032760264 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2400b820 con 0x7f6c3c1580e0 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c38ff9640 1 -- 172.21.15.148:0/1032760264 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2400bb90 con 0x7f6c3c1580e0 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c15c830 con 0x7f6c3c1580e0 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c15cdf0 con 0x7f6c3c1580e0 2023-10-30T13:49:38.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.815+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c15cfb0 con 0x7f6c3c1580e0 2023-10-30T13:49:38.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.819+0000 7f6c38ff9640 1 -- 172.21.15.148:0/1032760264 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c2400bcf0 con 0x7f6c3c1580e0 2023-10-30T13:49:38.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.819+0000 7f6c38ff9640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c1c0664a0 0x7f6c1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:38.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.819+0000 7f6c38ff9640 1 -- 172.21.15.148:0/1032760264 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f6c24084c90 con 0x7f6c3c1580e0 2023-10-30T13:49:38.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.819+0000 7f6c3b7fe640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c1c0664a0 0x7f6c1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:38.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.819+0000 7f6c3b7fe640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c1c0664a0 0x7f6c1c068960 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c007fd0 tx=0x7f6c2c007c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:38.825 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:38.823+0000 7f6c38ff9640 1 -- 172.21.15.148:0/1032760264 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c24051890 con 0x7f6c3c1580e0 2023-10-30T13:49:39.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c3c04e170 con 0x7f6c3c1580e0 2023-10-30T13:49:39.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c38ff9640 1 -- 172.21.15.148:0/1032760264 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f6c24011340 con 0x7f6c3c1580e0 2023-10-30T13:49:39.005 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:39.005 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:37.212781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:39.005 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 155 2023-10-30T13:49:39.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c1c0664a0 msgr2=0x7f6c1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:39.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c1c0664a0 0x7f6c1c068960 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c007fd0 tx=0x7f6c2c007c20 comp rx=0 tx=0).stop 2023-10-30T13:49:39.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c1580e0 msgr2=0x7f6c3c06bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:39.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c1580e0 0x7f6c3c06bbb0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400d800 tx=0x7f6c2400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 shutdown_connections 2023-10-30T13:49:39.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6c1c0664a0 0x7f6c1c068960 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c3c1580e0 0x7f6c3c06bbb0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6c3c15e700 0x7f6c3c14ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 --2- 172.21.15.148:0/1032760264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3c070740 0x7f6c3c1f9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:39.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 >> 172.21.15.148:0/1032760264 conn(0x7f6c3c0c7910 msgr2=0x7f6c3c0c1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:39.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 shutdown_connections 2023-10-30T13:49:39.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:39.003+0000 7f6c421d3640 1 -- 172.21.15.148:0/1032760264 wait complete. 2023-10-30T13:49:39.027 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:41.027 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:41.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.175+0000 7f7f753d3640 1 Processor -- start 2023-10-30T13:49:41.179 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.175+0000 7f7f753d3640 1 -- start start 2023-10-30T13:49:41.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f753d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f7015d630 0x7f7f700be7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f753d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7f7006d7a0 0x7f7f7006dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f753d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f70159920 con 0x7f7f7006fc00 2023-10-30T13:49:41.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f753d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7015a4c0 con 0x7f7f7006d7a0 2023-10-30T13:49:41.182 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f753d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f700bece0 con 0x7f7f7015d630 2023-10-30T13:49:41.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f667fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7f7006fc00 0x7f7f70159db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:57324/0 (socket says 172.21.15.148:57324) 2023-10-30T13:49:41.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f667fc640 1 -- 172.21.15.148:0/1602780225 learned_addr learned my addr 172.21.15.148:0/1602780225 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:41.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6f7fe640 1 --2- 172.21.15.148:0/1602780225 >> v2:172.21.15.150:3300/0 conn(0x7f7f7006d7a0 0x7f7f7006dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6effd640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f7015d630 0x7f7f700be7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319090649 0 0) 0x7f7f70159920 con 0x7f7f7006fc00 2023-10-30T13:49:41.183 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f5c003530 con 0x7f7f7006fc00 2023-10-30T13:49:41.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3179712523 0 0) 0x7f7f5c003530 con 0x7f7f7006fc00 2023-10-30T13:49:41.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f7015d630 msgr2=0x7f7f700be7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:41.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f7015d630 0x7f7f700be7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 >> v2:172.21.15.150:3300/0 conn(0x7f7f7006d7a0 msgr2=0x7f7f7006dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:41.184 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 --2- 172.21.15.148:0/1602780225 >> v2:172.21.15.150:3300/0 conn(0x7f7f7006d7a0 0x7f7f7006dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f700c5d80 con 0x7f7f7006fc00 2023-10-30T13:49:41.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f7f58001680 con 0x7f7f7006fc00 2023-10-30T13:49:41.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 >> v1:172.21.15.148:6789/0 conn(0x7f7f7006fc00 legacy=0x7f7f70159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f5c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.185 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f7015d630 0x7f7f5c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c004d10 0x7f7f5c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7015a4c0 con 0x7f7f5c003980 2023-10-30T13:49:41.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f700bece0 con 0x7f7f5c004d10 2023-10-30T13:49:41.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c00b990 con 0x7f7f7015d630 2023-10-30T13:49:41.186 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6effd640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f5c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6f7fe640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c004d10 0x7f7f5c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f667fc640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f7015d630 0x7f7f5c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6effd640 1 -- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f7015d630 msgr2=0x7f7f5c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:41.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6effd640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f7015d630 0x7f7f5c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.187 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6effd640 1 -- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c004d10 msgr2=0x7f7f5c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:41.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6effd640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c004d10 0x7f7f5c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6effd640 1 -- 172.21.15.148:0/1602780225 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c003530 con 0x7f7f5c003980 2023-10-30T13:49:41.188 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.179+0000 7f7f6f7fe640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c004d10 0x7f7f5c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:41.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f6effd640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f5c003da0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009ae0 tx=0x7f7f6001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92b20523804ac8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:41.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f600a5530 con 0x7f7f5c003980 2023-10-30T13:49:41.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f600a5690 con 0x7f7f5c003980 2023-10-30T13:49:41.191 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f6e7fc640 1 -- 172.21.15.148:0/1602780225 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f600a5a40 con 0x7f7f5c003980 2023-10-30T13:49:41.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f753d3640 1 -- 172.21.15.148:0/1602780225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 msgr2=0x7f7f5c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f753d3640 1 --2- 172.21.15.148:0/1602780225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f5c003da0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009ae0 tx=0x7f7f6001ca50 comp rx=0 tx=0).stop 2023-10-30T13:49:41.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f6f7fe640 1 -- 172.21.15.148:0/1602780225 reap_dead start 2023-10-30T13:49:41.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f753d3640 1 -- 172.21.15.148:0/1602780225 shutdown_connections 2023-10-30T13:49:41.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f753d3640 1 -- 172.21.15.148:0/1602780225 >> 172.21.15.148:0/1602780225 conn(0x7f7f7006bb70 msgr2=0x7f7f7006bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:41.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f753d3640 1 -- 172.21.15.148:0/1602780225 shutdown_connections 2023-10-30T13:49:41.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f753d3640 1 -- 172.21.15.148:0/1602780225 wait complete. 2023-10-30T13:49:41.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f753d3640 1 Processor -- start 2023-10-30T13:49:41.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.183+0000 7f7f753d3640 1 -- start start 2023-10-30T13:49:41.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f753d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f700cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f753d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f70157220 0x7f7f7015bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f753d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f701f3500 0x7f7f70149780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f753d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7014b980 con 0x7f7f5c003980 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f753d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7014b550 con 0x7f7f70157220 2023-10-30T13:49:41.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f753d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f700cc1b0 con 0x7f7f701f3500 2023-10-30T13:49:41.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f700cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f700cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56824/0 (socket says 172.21.15.148:56824) 2023-10-30T13:49:41.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6effd640 1 -- 172.21.15.148:0/4262482360 learned_addr learned my addr 172.21.15.148:0/4262482360 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:41.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f667fc640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f70157220 0x7f7f7015bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6f7fe640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f701f3500 0x7f7f70149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6effd640 1 -- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f701f3500 msgr2=0x7f7f70149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6effd640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f701f3500 0x7f7f70149780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6effd640 1 -- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f70157220 msgr2=0x7f7f7015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6effd640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f70157220 0x7f7f7015bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6effd640 1 -- 172.21.15.148:0/4262482360 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f700c9280 con 0x7f7f5c003980 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f667fc640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f70157220 0x7f7f7015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f6effd640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f700cbb70 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009ae0 tx=0x7f7f5c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f67fff640 1 -- 172.21.15.148:0/4262482360 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60004220 con 0x7f7f5c003980 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f67fff640 1 -- 172.21.15.148:0/4262482360 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f60004380 con 0x7f7f5c003980 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f67fff640 1 -- 172.21.15.148:0/4262482360 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60003820 con 0x7f7f5c003980 2023-10-30T13:49:41.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f700c9560 con 0x7f7f5c003980 2023-10-30T13:49:41.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f700c9b20 con 0x7f7f5c003980 2023-10-30T13:49:41.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.187+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7004e170 con 0x7f7f5c003980 2023-10-30T13:49:41.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.191+0000 7f7f67fff640 1 -- 172.21.15.148:0/4262482360 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f600b4040 con 0x7f7f5c003980 2023-10-30T13:49:41.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.191+0000 7f7f67fff640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f500664d0 0x7f7f50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:41.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.191+0000 7f7f67fff640 1 -- 172.21.15.148:0/4262482360 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f7f6011e6f0 con 0x7f7f5c003980 2023-10-30T13:49:41.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.191+0000 7f7f667fc640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f500664d0 0x7f7f50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:41.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.191+0000 7f7f667fc640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f500664d0 0x7f7f50068990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7f58001ec0 tx=0x7f7f58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:41.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.195+0000 7f7f67fff640 1 -- 172.21.15.148:0/4262482360 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f600eb560 con 0x7f7f5c003980 2023-10-30T13:49:41.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f700bdfb0 con 0x7f7f5c003980 2023-10-30T13:49:41.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f67fff640 1 -- 172.21.15.148:0/4262482360 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f7f600eeb80 con 0x7f7f5c003980 2023-10-30T13:49:41.381 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:41.381 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:37.212781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:41.382 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 155 2023-10-30T13:49:41.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f500664d0 msgr2=0x7f7f50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f500664d0 0x7f7f50068990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7f58001ec0 tx=0x7f7f58009040 comp rx=0 tx=0).stop 2023-10-30T13:49:41.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 msgr2=0x7f7f700cbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:41.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f700cbb70 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009ae0 tx=0x7f7f5c005800 comp rx=0 tx=0).stop 2023-10-30T13:49:41.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 shutdown_connections 2023-10-30T13:49:41.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7f500664d0 0x7f7f50068990 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7f701f3500 0x7f7f70149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f70157220 0x7f7f7015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 --2- 172.21.15.148:0/4262482360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f5c003980 0x7f7f700cbb70 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:41.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 >> 172.21.15.148:0/4262482360 conn(0x7f7f7006bb70 msgr2=0x7f7f700bc520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:41.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 shutdown_connections 2023-10-30T13:49:41.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:41.379+0000 7f7f753d3640 1 -- 172.21.15.148:0/4262482360 wait complete. 2023-10-30T13:49:41.402 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:43.403 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:43.548 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d1db80640 1 Processor -- start 2023-10-30T13:49:43.549 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d1db80640 1 -- start start 2023-10-30T13:49:43.551 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d1db80640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d18159e60 0x7f6d1806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d1db80640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6d1806a720 0x7f6d1806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d1db80640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1806fa80 con 0x7f6d180bfec0 2023-10-30T13:49:43.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d1db80640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d18071290 con 0x7f6d1806a720 2023-10-30T13:49:43.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d1db80640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18071570 con 0x7f6d18159e60 2023-10-30T13:49:43.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d16ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6d180bfec0 0x7f6d180c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44866/0 (socket says 172.21.15.148:44866) 2023-10-30T13:49:43.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d16ffd640 1 -- 172.21.15.148:0/756921166 learned_addr learned my addr 172.21.15.148:0/756921166 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:43.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d17fff640 1 --2- 172.21.15.148:0/756921166 >> v2:172.21.15.150:3300/0 conn(0x7f6d1806a720 0x7f6d1806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d177fe640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d18159e60 0x7f6d1806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.553 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077442234 0 0) 0x7f6d1806fa80 con 0x7f6d180bfec0 2023-10-30T13:49:43.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d00005740 con 0x7f6d180bfec0 2023-10-30T13:49:43.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735558766 0 0) 0x7f6d00005740 con 0x7f6d180bfec0 2023-10-30T13:49:43.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d18159e60 msgr2=0x7f6d1806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:43.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d18159e60 0x7f6d1806f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.554 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 >> v2:172.21.15.150:3300/0 conn(0x7f6d1806a720 msgr2=0x7f6d1806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:43.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 --2- 172.21.15.148:0/756921166 >> v2:172.21.15.150:3300/0 conn(0x7f6d1806a720 0x7f6d1806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d18071850 con 0x7f6d180bfec0 2023-10-30T13:49:43.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f6d0c001680 con 0x7f6d180bfec0 2023-10-30T13:49:43.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 >> v1:172.21.15.148:6789/0 conn(0x7f6d180bfec0 legacy=0x7f6d180c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d00005b90 0x7f6d00005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d00006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d18159e60 0x7f6d000073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18071290 con 0x7f6d000069e0 2023-10-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18071570 con 0x7f6d00005b90 2023-10-30T13:49:43.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0000dba0 con 0x7f6d18159e60 2023-10-30T13:49:43.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d16ffd640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d00006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d177fe640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d00005b90 0x7f6d00005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d17fff640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d18159e60 0x7f6d000073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d16ffd640 1 -- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d18159e60 msgr2=0x7f6d000073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:43.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d16ffd640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d18159e60 0x7f6d000073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d16ffd640 1 -- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d00005b90 msgr2=0x7f6d00005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:43.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d16ffd640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d00005b90 0x7f6d00005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d16ffd640 1 -- 172.21.15.148:0/756921166 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d00005740 con 0x7f6d000069e0 2023-10-30T13:49:43.558 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.547+0000 7f6d177fe640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d00005b90 0x7f6d00005fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:43.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d16ffd640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d00006e60 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00c240 tx=0x7f6d0c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1db3097933c55eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:43.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c0058e0 con 0x7f6d000069e0 2023-10-30T13:49:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0c005a40 con 0x7f6d000069e0 2023-10-30T13:49:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d167fc640 1 -- 172.21.15.148:0/756921166 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c005e10 con 0x7f6d000069e0 2023-10-30T13:49:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d1db80640 1 -- 172.21.15.148:0/756921166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 msgr2=0x7f6d00006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d1db80640 1 --2- 172.21.15.148:0/756921166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d00006e60 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00c240 tx=0x7f6d0c01cc60 comp rx=0 tx=0).stop 2023-10-30T13:49:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d17fff640 1 -- 172.21.15.148:0/756921166 reap_dead start 2023-10-30T13:49:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d1db80640 1 -- 172.21.15.148:0/756921166 shutdown_connections 2023-10-30T13:49:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d1db80640 1 -- 172.21.15.148:0/756921166 >> 172.21.15.148:0/756921166 conn(0x7f6d180c7910 msgr2=0x7f6d180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d1db80640 1 -- 172.21.15.148:0/756921166 shutdown_connections 2023-10-30T13:49:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.551+0000 7f6d1db80640 1 -- 172.21.15.148:0/756921166 wait complete. 2023-10-30T13:49:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d1db80640 1 Processor -- start 2023-10-30T13:49:43.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d1db80640 1 -- start start 2023-10-30T13:49:43.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d1db80640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d18151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d1db80640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d1814f810 0x7f6d180c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d1db80640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d180c40f0 0x7f6d180c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d1db80640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18150380 con 0x7f6d000069e0 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d1db80640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d180c4890 con 0x7f6d1814f810 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d1db80640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d181eb7c0 con 0x7f6d180c40f0 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d177fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d18151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d177fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d18151030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37214/0 (socket says 172.21.15.148:37214) 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d177fe640 1 -- 172.21.15.148:0/429521831 learned_addr learned my addr 172.21.15.148:0/429521831 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:43.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d16ffd640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d1814f810 0x7f6d180c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d17fff640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d180c40f0 0x7f6d180c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d177fe640 1 -- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d180c40f0 msgr2=0x7f6d180c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d177fe640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d180c40f0 0x7f6d180c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d177fe640 1 -- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d1814f810 msgr2=0x7f6d180c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d177fe640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d1814f810 0x7f6d180c3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d177fe640 1 -- 172.21.15.148:0/429521831 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d180c45b0 con 0x7f6d000069e0 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d177fe640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d18151030 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009160 tx=0x7f6d08011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6cfffff640 1 -- 172.21.15.148:0/429521831 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08004220 con 0x7f6d000069e0 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6cfffff640 1 -- 172.21.15.148:0/429521831 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d08004380 con 0x7f6d000069e0 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6cfffff640 1 -- 172.21.15.148:0/429521831 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08015890 con 0x7f6d000069e0 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.555+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d181eaa00 con 0x7f6d000069e0 2023-10-30T13:49:43.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.559+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d181eafc0 con 0x7f6d000069e0 2023-10-30T13:49:43.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.559+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1804e170 con 0x7f6d000069e0 2023-10-30T13:49:43.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.559+0000 7f6cfffff640 1 -- 172.21.15.148:0/429521831 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6d080159f0 con 0x7f6d000069e0 2023-10-30T13:49:43.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.559+0000 7f6cfffff640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6d04066470 0x7f6d04068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:43.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.559+0000 7f6cfffff640 1 -- 172.21.15.148:0/429521831 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f6d08084a70 con 0x7f6d000069e0 2023-10-30T13:49:43.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.559+0000 7f6d16ffd640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6d04066470 0x7f6d04068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:43.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.563+0000 7f6d16ffd640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6d04066470 0x7f6d04068930 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c0aa000 tx=0x7f6d0c001a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:43.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.563+0000 7f6cfffff640 1 -- 172.21.15.148:0/429521831 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6d08051720 con 0x7f6d000069e0 2023-10-30T13:49:43.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.743+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d1806cb90 con 0x7f6d000069e0 2023-10-30T13:49:43.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.743+0000 7f6cfffff640 1 -- 172.21.15.148:0/429521831 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f6d08022090 con 0x7f6d000069e0 2023-10-30T13:49:43.748 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:43.748 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:37.212781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:43.748 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 155 2023-10-30T13:49:43.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6d04066470 msgr2=0x7f6d04068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6d04066470 0x7f6d04068930 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c0aa000 tx=0x7f6d0c001a80 comp rx=0 tx=0).stop 2023-10-30T13:49:43.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 msgr2=0x7f6d18151030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:43.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d18151030 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009160 tx=0x7f6d08011a60 comp rx=0 tx=0).stop 2023-10-30T13:49:43.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 shutdown_connections 2023-10-30T13:49:43.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6d04066470 0x7f6d04068930 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6d180c40f0 0x7f6d180c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d1814f810 0x7f6d180c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 --2- 172.21.15.148:0/429521831 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d000069e0 0x7f6d18151030 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:43.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 >> 172.21.15.148:0/429521831 conn(0x7f6d180c7910 msgr2=0x7f6d180c9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:43.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 shutdown_connections 2023-10-30T13:49:43.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:43.747+0000 7f6d1db80640 1 -- 172.21.15.148:0/429521831 wait complete. 2023-10-30T13:49:43.769 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:49:45.771 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:45.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2663165640 1 Processor -- start 2023-10-30T13:49:45.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2663165640 1 -- start start 2023-10-30T13:49:45.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2663165640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f265c1538a0 0x7f265c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2663165640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f265c0cf160 0x7f265c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2663165640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c0cb800 con 0x7f265c0ca0e0 2023-10-30T13:49:45.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2663165640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f265c15dc30 con 0x7f265c1538a0 2023-10-30T13:49:45.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2663165640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f265c0cb330 con 0x7f265c0cf160 2023-10-30T13:49:45.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2660f01640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f265c1538a0 0x7f265c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f265bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f265c0ca0e0 0x7f265c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44876/0 (socket says 172.21.15.148:44876) 2023-10-30T13:49:45.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f265bfff640 1 -- 172.21.15.148:0/1213576903 learned_addr learned my addr 172.21.15.148:0/1213576903 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:49:45.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2661702640 1 --2- 172.21.15.148:0/1213576903 >> v2:172.21.15.150:3300/0 conn(0x7f265c0cf160 0x7f265c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.924 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f265b7fe640 1 -- 172.21.15.148:0/1213576903 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506525564 0 0) 0x7f265c0cb800 con 0x7f265c0ca0e0 2023-10-30T13:49:45.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f265b7fe640 1 -- 172.21.15.148:0/1213576903 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2644003530 con 0x7f265c0ca0e0 2023-10-30T13:49:45.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2660f01640 1 -- 172.21.15.148:0/1213576903 >> v2:172.21.15.150:3300/0 conn(0x7f265c0cf160 msgr2=0x7f265c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:45.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2660f01640 1 --2- 172.21.15.148:0/1213576903 >> v2:172.21.15.150:3300/0 conn(0x7f265c0cf160 0x7f265c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2660f01640 1 -- 172.21.15.148:0/1213576903 >> v1:172.21.15.148:6789/0 conn(0x7f265c0ca0e0 legacy=0x7f265c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:45.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.919+0000 7f2660f01640 1 -- 172.21.15.148:0/1213576903 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c0cad70 con 0x7f265c1538a0 2023-10-30T13:49:45.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f2660f01640 1 --2- 172.21.15.148:0/1213576903 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c1538a0 0x7f265c15d6f0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f265c0b5330 tx=0x7f264c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811e5b8edba2b2ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:45.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f265b7fe640 1 -- 172.21.15.148:0/1213576903 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c0a8950 con 0x7f265c1538a0 2023-10-30T13:49:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f265b7fe640 1 -- 172.21.15.148:0/1213576903 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c0a8ab0 con 0x7f265c1538a0 2023-10-30T13:49:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f265b7fe640 1 -- 172.21.15.148:0/1213576903 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c0a8e40 con 0x7f265c1538a0 2023-10-30T13:49:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f2663165640 1 -- 172.21.15.148:0/1213576903 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c1538a0 msgr2=0x7f265c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f2663165640 1 --2- 172.21.15.148:0/1213576903 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c1538a0 0x7f265c15d6f0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f265c0b5330 tx=0x7f264c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:49:45.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f2663165640 1 -- 172.21.15.148:0/1213576903 shutdown_connections 2023-10-30T13:49:45.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f2663165640 1 --2- 172.21.15.148:0/1213576903 >> v2:172.21.15.150:3300/0 conn(0x7f265c0cf160 0x7f265c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f2663165640 1 --2- 172.21.15.148:0/1213576903 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c1538a0 0x7f265c15d6f0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.928 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f2663165640 1 -- 172.21.15.148:0/1213576903 >> 172.21.15.148:0/1213576903 conn(0x7f265c06bfe0 msgr2=0x7f265c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:45.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f2663165640 1 -- 172.21.15.148:0/1213576903 shutdown_connections 2023-10-30T13:49:45.929 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.923+0000 7f2663165640 1 -- 172.21.15.148:0/1213576903 wait complete. 2023-10-30T13:49:45.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 Processor -- start 2023-10-30T13:49:45.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 -- start start 2023-10-30T13:49:45.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c0ca0e0 0x7f265c15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.931 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f265c0cf160 0x7f265c1495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f265c1538a0 0x7f265c146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c149190 con 0x7f265c0ca0e0 2023-10-30T13:49:45.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c148a80 con 0x7f265c0cf160 2023-10-30T13:49:45.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c147780 con 0x7f265c1538a0 2023-10-30T13:49:45.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2660f01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c0ca0e0 0x7f265c15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2660f01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c0ca0e0 0x7f265c15a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:37244/0 (socket says 172.21.15.148:37244) 2023-10-30T13:49:45.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2660f01640 1 -- 172.21.15.148:0/4245352693 learned_addr learned my addr 172.21.15.148:0/4245352693 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:45.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f265bfff640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f265c0cf160 0x7f265c1495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2661702640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f265c1538a0 0x7f265c146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2660f01640 1 -- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f265c1538a0 msgr2=0x7f265c146ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:45.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2660f01640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f265c1538a0 0x7f265c146ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2660f01640 1 -- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f265c0cf160 msgr2=0x7f265c1495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2660f01640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f265c0cf160 0x7f265c1495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2660f01640 1 -- 172.21.15.148:0/4245352693 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c009420 con 0x7f265c0ca0e0 2023-10-30T13:49:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2660f01640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c0ca0e0 0x7f265c15a8b0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f264c01c880 tx=0x7f264c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f26597fa640 1 -- 172.21.15.148:0/4245352693 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c0b0d20 con 0x7f265c0ca0e0 2023-10-30T13:49:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f26597fa640 1 -- 172.21.15.148:0/4245352693 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c0b9430 con 0x7f265c0ca0e0 2023-10-30T13:49:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c1474e0 con 0x7f265c0ca0e0 2023-10-30T13:49:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f26597fa640 1 -- 172.21.15.148:0/4245352693 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c0b0ea0 con 0x7f265c0ca0e0 2023-10-30T13:49:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.927+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c14a6f0 con 0x7f265c0ca0e0 2023-10-30T13:49:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.931+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c0b2a50 con 0x7f265c0ca0e0 2023-10-30T13:49:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.931+0000 7f26597fa640 1 -- 172.21.15.148:0/4245352693 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f264c0b7040 con 0x7f265c0ca0e0 2023-10-30T13:49:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.931+0000 7f26597fa640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26480664d0 0x7f2648068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.931+0000 7f26597fa640 1 -- 172.21.15.148:0/4245352693 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f264c1220d0 con 0x7f265c0ca0e0 2023-10-30T13:49:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.931+0000 7f265bfff640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26480664d0 0x7f2648068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.931+0000 7f265bfff640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26480664d0 0x7f2648068990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f265c0cddd0 tx=0x7f2650009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:45.935+0000 7f26597fa640 1 -- 172.21.15.148:0/4245352693 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f264c0eedf0 con 0x7f265c0ca0e0 2023-10-30T13:49:46.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f265c04e170 con 0x7f265c0ca0e0 2023-10-30T13:49:46.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f26597fa640 1 -- 172.21.15.148:0/4245352693 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f264c0f2410 con 0x7f265c0ca0e0 2023-10-30T13:49:46.122 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:46.122 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:43.210127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:46.122 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 156 2023-10-30T13:49:46.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26480664d0 msgr2=0x7f2648068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:46.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26480664d0 0x7f2648068990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f265c0cddd0 tx=0x7f2650009040 comp rx=0 tx=0).stop 2023-10-30T13:49:46.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c0ca0e0 msgr2=0x7f265c15a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:46.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c0ca0e0 0x7f265c15a8b0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f264c01c880 tx=0x7f264c01b9f0 comp rx=0 tx=0).stop 2023-10-30T13:49:46.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 shutdown_connections 2023-10-30T13:49:46.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f26480664d0 0x7f2648068990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:46.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f265c1538a0 0x7f265c146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:46.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f265c0cf160 0x7f265c1495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:46.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 --2- 172.21.15.148:0/4245352693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f265c0ca0e0 0x7f265c15a8b0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:46.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.119+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 >> 172.21.15.148:0/4245352693 conn(0x7f265c06bfe0 msgr2=0x7f265c1525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:46.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.123+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 shutdown_connections 2023-10-30T13:49:46.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:46.123+0000 7f2663165640 1 -- 172.21.15.148:0/4245352693 wait complete. 2023-10-30T13:49:46.143 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=1, replacing gid=7611 2023-10-30T13:49:46.143 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.a 2023-10-30T13:49:52.144 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-30T13:49:52.144 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T13:49:52.144 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T13:49:52.148 INFO:tasks.ceph.mds.a:Started 2023-10-30T13:49:52.149 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:52.233 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:49:52.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc62ed6640 1 Processor -- start 2023-10-30T13:49:52.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc62ed6640 1 -- start start 2023-10-30T13:49:52.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc62ed6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc5c0c6bd0 0x7efc5c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:52.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc62ed6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c071110 0x7efc5c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:52.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc62ed6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5c066cf0 con 0x7efc5c06f7a0 2023-10-30T13:49:52.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc62ed6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5c0700c0 con 0x7efc5c0c6bd0 2023-10-30T13:49:52.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc62ed6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c0c70d0 con 0x7efc5c071110 2023-10-30T13:49:52.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc5bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc5c0c6bd0 0x7efc5c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:52.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc61473640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c071110 0x7efc5c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:52.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc61473640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c071110 0x7efc5c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:51074/0 (socket says 172.21.15.148:51074) 2023-10-30T13:49:52.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc61473640 1 -- 172.21.15.148:0/746081415 learned_addr learned my addr 172.21.15.148:0/746081415 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:52.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc5bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc5c0c6bd0 0x7efc5c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:54148/0 (socket says 172.21.15.148:54148) 2023-10-30T13:49:52.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc61473640 1 -- 172.21.15.148:0/746081415 >> v2:172.21.15.150:3300/0 conn(0x7efc5c0c6bd0 msgr2=0x7efc5c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:52.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc61473640 1 --2- 172.21.15.148:0/746081415 >> v2:172.21.15.150:3300/0 conn(0x7efc5c0c6bd0 0x7efc5c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc61473640 1 -- 172.21.15.148:0/746081415 >> v1:172.21.15.150:6789/0 conn(0x7efc5c06f7a0 legacy=0x7efc5c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:52.292 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.287+0000 7efc61473640 1 -- 172.21.15.148:0/746081415 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c0c7390 con 0x7efc5c071110 2023-10-30T13:49:52.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc61473640 1 --2- 172.21.15.148:0/746081415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c071110 0x7efc5c158110 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efc54009e50 tx=0x7efc5401f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=762d5dfa60361b60 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:52.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc5b7fe640 1 -- 172.21.15.148:0/746081415 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc540af400 con 0x7efc5c071110 2023-10-30T13:49:52.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc5b7fe640 1 -- 172.21.15.148:0/746081415 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc540ab050 con 0x7efc5c071110 2023-10-30T13:49:52.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc5b7fe640 1 -- 172.21.15.148:0/746081415 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc540af780 con 0x7efc5c071110 2023-10-30T13:49:52.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc62ed6640 1 -- 172.21.15.148:0/746081415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c071110 msgr2=0x7efc5c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:52.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc62ed6640 1 --2- 172.21.15.148:0/746081415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c071110 0x7efc5c158110 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efc54009e50 tx=0x7efc5401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc62ed6640 1 -- 172.21.15.148:0/746081415 shutdown_connections 2023-10-30T13:49:52.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc62ed6640 1 --2- 172.21.15.148:0/746081415 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c071110 0x7efc5c158110 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc62ed6640 1 --2- 172.21.15.148:0/746081415 >> v2:172.21.15.150:3300/0 conn(0x7efc5c0c6bd0 0x7efc5c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc62ed6640 1 -- 172.21.15.148:0/746081415 >> 172.21.15.148:0/746081415 conn(0x7efc5c0c7910 msgr2=0x7efc5c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:52.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc62ed6640 1 -- 172.21.15.148:0/746081415 shutdown_connections 2023-10-30T13:49:52.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.291+0000 7efc62ed6640 1 -- 172.21.15.148:0/746081415 wait complete. 2023-10-30T13:49:52.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 Processor -- start 2023-10-30T13:49:52.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 -- start start 2023-10-30T13:49:52.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5c06f7a0 0x7efc5c153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:52.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc5c071110 0x7efc5c1e2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:52.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c0c6bd0 0x7efc5c1e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:52.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c0c3840 con 0x7efc5c071110 2023-10-30T13:49:52.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c156890 con 0x7efc5c06f7a0 2023-10-30T13:49:52.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c1e22c0 con 0x7efc5c0c6bd0 2023-10-30T13:49:52.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc5bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc5c071110 0x7efc5c1e2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:52.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc5bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc5c071110 0x7efc5c1e2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59314/0 (socket says 172.21.15.148:59314) 2023-10-30T13:49:52.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc5bfff640 1 -- 172.21.15.148:0/3025263459 learned_addr learned my addr 172.21.15.148:0/3025263459 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:52.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc60c72640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5c06f7a0 0x7efc5c153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:52.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc61473640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c0c6bd0 0x7efc5c1e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:52.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc5bfff640 1 -- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c0c6bd0 msgr2=0x7efc5c1e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:52.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc5bfff640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c0c6bd0 0x7efc5c1e2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc5bfff640 1 -- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5c06f7a0 msgr2=0x7efc5c153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:52.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc5bfff640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5c06f7a0 0x7efc5c153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc5bfff640 1 -- 172.21.15.148:0/3025263459 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc54009b30 con 0x7efc5c071110 2023-10-30T13:49:52.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc60c72640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5c06f7a0 0x7efc5c153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:52.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc5bfff640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc5c071110 0x7efc5c1e2750 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7efc4800e930 tx=0x7efc4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:52.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc597fa640 1 -- 172.21.15.148:0/3025263459 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48009690 con 0x7efc5c071110 2023-10-30T13:49:52.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc597fa640 1 -- 172.21.15.148:0/3025263459 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4800ca00 con 0x7efc5c071110 2023-10-30T13:49:52.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc597fa640 1 -- 172.21.15.148:0/3025263459 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4800cd90 con 0x7efc5c071110 2023-10-30T13:49:52.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5c156ae0 con 0x7efc5c071110 2023-10-30T13:49:52.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5c1e1c70 con 0x7efc5c071110 2023-10-30T13:49:52.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.295+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5c04e170 con 0x7efc5c071110 2023-10-30T13:49:52.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.299+0000 7efc597fa640 1 -- 172.21.15.148:0/3025263459 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efc48013470 con 0x7efc5c071110 2023-10-30T13:49:52.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.299+0000 7efc597fa640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc400664d0 0x7efc40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:52.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.299+0000 7efc597fa640 1 -- 172.21.15.148:0/3025263459 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7efc48084b20 con 0x7efc5c071110 2023-10-30T13:49:52.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.299+0000 7efc60c72640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc400664d0 0x7efc40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:52.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.299+0000 7efc60c72640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc400664d0 0x7efc40068990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efc50000f70 tx=0x7efc50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:52.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.303+0000 7efc597fa640 1 -- 172.21.15.148:0/3025263459 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efc48051920 con 0x7efc5c071110 2023-10-30T13:49:52.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc5c1e2400 con 0x7efc5c071110 2023-10-30T13:49:52.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc597fa640 1 -- 172.21.15.148:0/3025263459 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5738 (secure 0 0 0) 0x7efc48054f40 con 0x7efc5c071110 2023-10-30T13:49:52.490 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:52.490 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:43.210127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:52.490 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 156 2023-10-30T13:49:52.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc400664d0 msgr2=0x7efc40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:52.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc400664d0 0x7efc40068990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efc50000f70 tx=0x7efc50009040 comp rx=0 tx=0).stop 2023-10-30T13:49:52.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc5c071110 msgr2=0x7efc5c1e2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:52.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc5c071110 0x7efc5c1e2750 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7efc4800e930 tx=0x7efc4800edf0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 shutdown_connections 2023-10-30T13:49:52.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc400664d0 0x7efc40068990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.492 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc5c071110 0x7efc5c1e2750 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc5c0c6bd0 0x7efc5c1e2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 --2- 172.21.15.148:0/3025263459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc5c06f7a0 0x7efc5c153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:52.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 >> 172.21.15.148:0/3025263459 conn(0x7efc5c0c7910 msgr2=0x7efc5c0c4340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:52.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 shutdown_connections 2023-10-30T13:49:52.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:52.487+0000 7efc62ed6640 1 -- 172.21.15.148:0/3025263459 wait complete. 2023-10-30T13:49:52.511 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-30T13:49:54.512 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:49:54.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.647+0000 7f5b315b8640 1 Processor -- start 2023-10-30T13:49:54.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.647+0000 7f5b315b8640 1 -- start start 2023-10-30T13:49:54.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b315b8640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b2c0c6b80 0x7f5b2c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b315b8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 0x7f5b2c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b315b8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b2c0c8430 0x7f5b2c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b315b8640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b2c0c0a50 con 0x7f5b2c0c6b80 2023-10-30T13:49:54.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b315b8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b2c06f6e0 con 0x7f5b2c0c8430 2023-10-30T13:49:54.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b315b8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c06f9c0 con 0x7f5b2c0c6f60 2023-10-30T13:49:54.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2affd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b2c0c6b80 0x7f5b2c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2affd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5b2c0c6b80 0x7f5b2c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59324/0 (socket says 172.21.15.148:59324) 2023-10-30T13:49:54.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2affd640 1 -- 172.21.15.148:0/26684772 learned_addr learned my addr 172.21.15.148:0/26684772 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:54.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2b7fe640 1 --2- 172.21.15.148:0/26684772 >> v2:172.21.15.150:3300/0 conn(0x7f5b2c0c8430 0x7f5b2c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b23fff640 1 --2- 172.21.15.148:0/26684772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 0x7f5b2c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2affd640 1 -- 172.21.15.148:0/26684772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 msgr2=0x7f5b2c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:54.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2affd640 1 --2- 172.21.15.148:0/26684772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 0x7f5b2c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2affd640 1 -- 172.21.15.148:0/26684772 >> v2:172.21.15.150:3300/0 conn(0x7f5b2c0c8430 msgr2=0x7f5b2c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:49:54.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2affd640 1 --2- 172.21.15.148:0/26684772 >> v2:172.21.15.150:3300/0 conn(0x7f5b2c0c8430 0x7f5b2c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2affd640 1 -- 172.21.15.148:0/26684772 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c06fca0 con 0x7f5b2c0c6b80 2023-10-30T13:49:54.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.651+0000 7f5b2b7fe640 1 --2- 172.21.15.148:0/26684772 >> v2:172.21.15.150:3300/0 conn(0x7f5b2c0c8430 0x7f5b2c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:49:54.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b2affd640 1 --2- 172.21.15.148:0/26684772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c6b80 0x7f5b2c0bfec0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0b0860 tx=0x7f5b1c01e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b396f545a9283bfe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:54.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b2a7fc640 1 -- 172.21.15.148:0/26684772 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0ab620 con 0x7f5b2c0c6b80 2023-10-30T13:49:54.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b2a7fc640 1 -- 172.21.15.148:0/26684772 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c0ab780 con 0x7f5b2c0c6b80 2023-10-30T13:49:54.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b2a7fc640 1 -- 172.21.15.148:0/26684772 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0abb10 con 0x7f5b2c0c6b80 2023-10-30T13:49:54.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 -- 172.21.15.148:0/26684772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c6b80 msgr2=0x7f5b2c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 --2- 172.21.15.148:0/26684772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c6b80 0x7f5b2c0bfec0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0b0860 tx=0x7f5b1c01e680 comp rx=0 tx=0).stop 2023-10-30T13:49:54.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 -- 172.21.15.148:0/26684772 shutdown_connections 2023-10-30T13:49:54.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 --2- 172.21.15.148:0/26684772 >> v2:172.21.15.150:3300/0 conn(0x7f5b2c0c8430 0x7f5b2c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 --2- 172.21.15.148:0/26684772 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 0x7f5b2c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 --2- 172.21.15.148:0/26684772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c6b80 0x7f5b2c0bfec0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 -- 172.21.15.148:0/26684772 >> 172.21.15.148:0/26684772 conn(0x7f5b2c0c7910 msgr2=0x7f5b2c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:54.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 -- 172.21.15.148:0/26684772 shutdown_connections 2023-10-30T13:49:54.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 -- 172.21.15.148:0/26684772 wait complete. 2023-10-30T13:49:54.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 Processor -- start 2023-10-30T13:49:54.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.655+0000 7f5b315b8640 1 -- start start 2023-10-30T13:49:54.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b315b8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0c6b80 0x7f5b2c153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b315b8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 0x7f5b2c139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b315b8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c8430 0x7f5b2c13a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b315b8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c1563d0 con 0x7f5b2c0c8430 2023-10-30T13:49:54.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b315b8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c1f1e90 con 0x7f5b2c0c6b80 2023-10-30T13:49:54.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b315b8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c1f2170 con 0x7f5b2c0c6f60 2023-10-30T13:49:54.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c8430 0x7f5b2c13a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c8430 0x7f5b2c13a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:59338/0 (socket says 172.21.15.148:59338) 2023-10-30T13:49:54.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2b7fe640 1 -- 172.21.15.148:0/1979747599 learned_addr learned my addr 172.21.15.148:0/1979747599 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:49:54.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2affd640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0c6b80 0x7f5b2c153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b23fff640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 0x7f5b2c139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2b7fe640 1 -- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 msgr2=0x7f5b2c139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2b7fe640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 0x7f5b2c139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2b7fe640 1 -- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0c6b80 msgr2=0x7f5b2c153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2b7fe640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0c6b80 0x7f5b2c153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2b7fe640 1 -- 172.21.15.148:0/1979747599 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c0094d0 con 0x7f5b2c0c8430 2023-10-30T13:49:54.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b2b7fe640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c8430 0x7f5b2c13a3f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f5b24007df0 tx=0x7f5b2400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:54.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b237fe640 1 -- 172.21.15.148:0/1979747599 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b240039a0 con 0x7f5b2c0c8430 2023-10-30T13:49:54.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b237fe640 1 -- 172.21.15.148:0/1979747599 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2400f040 con 0x7f5b2c0c8430 2023-10-30T13:49:54.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b237fe640 1 -- 172.21.15.148:0/1979747599 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24003d40 con 0x7f5b2c0c8430 2023-10-30T13:49:54.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2c1f2450 con 0x7f5b2c0c8430 2023-10-30T13:49:54.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c1f2a10 con 0x7f5b2c0c8430 2023-10-30T13:49:54.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2c1f2c30 con 0x7f5b2c0c8430 2023-10-30T13:49:54.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.659+0000 7f5b237fe640 1 -- 172.21.15.148:0/1979747599 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b24004030 con 0x7f5b2c0c8430 2023-10-30T13:49:54.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.663+0000 7f5b237fe640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b0c0664d0 0x7f5b0c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:49:54.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.663+0000 7f5b237fe640 1 -- 172.21.15.148:0/1979747599 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f5b24084880 con 0x7f5b2c0c8430 2023-10-30T13:49:54.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.663+0000 7f5b2affd640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b0c0664d0 0x7f5b0c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:49:54.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.663+0000 7f5b2affd640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b0c0664d0 0x7f5b0c068990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0b0000 tx=0x7f5b1c0a9000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:49:54.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.667+0000 7f5b237fe640 1 -- 172.21.15.148:0/1979747599 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b240515b0 con 0x7f5b2c0c8430 2023-10-30T13:49:54.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.843+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b2c04e170 con 0x7f5b2c0c8430 2023-10-30T13:49:54.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b237fe640 1 -- 172.21.15.148:0/1979747599 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+6533 (secure 0 0 0) 0x7f5b24054bd0 con 0x7f5b2c0c8430 2023-10-30T13:49:54.848 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:49:54.849 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8145,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:43.210127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:49:54.849 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 157 2023-10-30T13:49:54.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b0c0664d0 msgr2=0x7f5b0c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b0c0664d0 0x7f5b0c068990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0b0000 tx=0x7f5b1c0a9000 comp rx=0 tx=0).stop 2023-10-30T13:49:54.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c8430 msgr2=0x7f5b2c13a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:49:54.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c8430 0x7f5b2c13a3f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f5b24007df0 tx=0x7f5b2400d4a0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 shutdown_connections 2023-10-30T13:49:54.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b0c0664d0 0x7f5b0c068990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b2c0c8430 0x7f5b2c13a3f0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5b2c0c6f60 0x7f5b2c139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 --2- 172.21.15.148:0/1979747599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0c6b80 0x7f5b2c153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:49:54.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 >> 172.21.15.148:0/1979747599 conn(0x7f5b2c0c7910 msgr2=0x7f5b2c138590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:49:54.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 shutdown_connections 2023-10-30T13:49:54.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:49:54.847+0000 7f5b315b8640 1 -- 172.21.15.148:0/1979747599 wait complete. 2023-10-30T13:49:54.870 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-30T13:49:54.870 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2023-10-30T13:50:03.498 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:03.502 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:04.942 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 12% (39/309) Updating files: 13% (41/309) Updating files: 14% (44/309) Updating files: 15% (47/309) Updating files: 16% (50/309) Updating files: 17% (53/309) Updating files: 18% (56/309) Updating files: 19% (59/309) Updating files: 20% (62/309) Updating files: 21% (65/309) Updating files: 22% (68/309) Updating files: 23% (72/309) Updating files: 24% (75/309) Updating files: 25% (78/309) Updating files: 26% (81/309) Updating files: 27% (84/309) Updating files: 27% (85/309) Updating files: 28% (87/309) Updating files: 29% (90/309) Updating files: 30% (93/309) Updating files: 31% (96/309) Updating files: 32% (99/309) Updating files: 33% (102/309) Updating files: 34% (106/309) Updating files: 35% (109/309) Updating files: 36% (112/309) Updating files: 37% (115/309) Updating files: 38% (118/309) Updating files: 39% (121/309) Updating files: 40% (124/309) Updating files: 41% (127/309) Updating files: 42% (130/309) Updating files: 42% (132/309) Updating files: 43% (133/309) Updating files: 44% (136/309) Updating files: 45% (140/309) Updating files: 46% (143/309) Updating files: 47% (146/309) Updating files: 48% (149/309) Updating files: 49% (152/309) Updating files: 50% (155/309) Updating files: 51% (158/309) Updating files: 52% (161/309) Updating files: 53% (164/309) Updating files: 54% (167/309) Updating files: 55% (170/309) Updating files: 56% (174/309) Updating files: 57% (177/309) Updating files: 57% (178/309) Updating files: 58% (180/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 77% (241/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2023-10-30T13:50:05.072 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 8de9501df27 0.81 2023-10-30T13:50:05.080 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.81 2023-10-30T13:50:06.333 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 82 -eq 48 ] 2023-10-30T13:50:06.334 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.82 2023-10-30T13:50:06.334 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.82 2023-10-30T13:50:06.334 INFO:tasks.workunit.client.0.smithi148.stdout:v0.82 2023-10-30T13:50:06.335 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.82 2023-10-30T13:50:14.022 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 14% (29/204) Updating files: 15% (31/204) Updating files: 16% (33/204) Updating files: 17% (35/204) Updating files: 18% (37/204) Updating files: 19% (39/204) Updating files: 20% (41/204) Updating files: 21% (43/204) Updating files: 22% (45/204) Updating files: 23% (47/204) Updating files: 24% (49/204) Updating files: 25% (51/204) Updating files: 26% (54/204) Updating files: 27% (56/204) Updating files: 28% (58/204) Updating files: 29% (60/204) Updating files: 30% (62/204) Updating files: 31% (64/204) Updating files: 32% (66/204) Updating files: 33% (68/204) Updating files: 34% (70/204) Updating files: 35% (72/204) Updating files: 36% (74/204) Updating files: 37% (76/204) Updating files: 38% (78/204) Updating files: 39% (80/204) Updating files: 40% (82/204) Updating files: 41% (84/204) Updating files: 42% (86/204) Updating files: 43% (88/204) Updating files: 44% (90/204) Updating files: 45% (92/204) Updating files: 46% (94/204) Updating files: 47% (96/204) Updating files: 48% (98/204) Updating files: 49% (100/204) Updating files: 50% (102/204) Updating files: 51% (105/204) Updating files: 52% (107/204) Updating files: 53% (109/204) Updating files: 54% (111/204) Updating files: 55% (113/204) Updating files: 56% (115/204) Updating files: 57% (117/204) Updating files: 58% (119/204) Updating files: 59% (121/204) Updating files: 60% (123/204) Updating files: 60% (124/204) Updating files: 61% (125/204) Updating files: 62% (127/204) Updating files: 63% (129/204) Updating files: 64% (131/204) Updating files: 65% (133/204) Updating files: 66% (135/204) Updating files: 67% (137/204) Updating files: 68% (139/204) Updating files: 69% (141/204) Updating files: 70% (143/204) Updating files: 71% (145/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2023-10-30T13:50:14.055 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 14085f42ddd 0.82 2023-10-30T13:50:14.061 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.82 2023-10-30T13:50:14.756 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 83 -eq 48 ] 2023-10-30T13:50:14.756 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.83 2023-10-30T13:50:14.757 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.83 2023-10-30T13:50:14.757 INFO:tasks.workunit.client.0.smithi148.stdout:v0.83 2023-10-30T13:50:14.757 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.83 2023-10-30T13:50:18.618 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 5% (13/218) Updating files: 6% (14/218) Updating files: 7% (16/218) Updating files: 8% (18/218) Updating files: 9% (20/218) Updating files: 10% (22/218) Updating files: 11% (24/218) Updating files: 12% (27/218) Updating files: 13% (29/218) Updating files: 14% (31/218) Updating files: 15% (33/218) Updating files: 16% (35/218) Updating files: 17% (38/218) Updating files: 18% (40/218) Updating files: 19% (42/218) Updating files: 20% (44/218) Updating files: 21% (46/218) Updating files: 22% (48/218) Updating files: 23% (51/218) Updating files: 24% (53/218) Updating files: 25% (55/218) Updating files: 26% (57/218) Updating files: 27% (59/218) Updating files: 28% (62/218) Updating files: 29% (64/218) Updating files: 30% (66/218) Updating files: 31% (68/218) Updating files: 32% (70/218) Updating files: 33% (72/218) Updating files: 34% (75/218) Updating files: 35% (77/218) Updating files: 36% (79/218) Updating files: 37% (81/218) Updating files: 38% (83/218) Updating files: 39% (86/218) Updating files: 40% (88/218) Updating files: 41% (90/218) Updating files: 42% (92/218) Updating files: 43% (94/218) Updating files: 44% (96/218) Updating files: 45% (99/218) Updating files: 46% (101/218) Updating files: 47% (103/218) Updating files: 48% (105/218) Updating files: 49% (107/218) Updating files: 50% (109/218) Updating files: 51% (112/218) Updating files: 52% (114/218) Updating files: 53% (116/218) Updating files: 54% (118/218) Updating files: 55% (120/218) Updating files: 56% (123/218) Updating files: 57% (125/218) Updating files: 58% (127/218) Updating files: 59% (129/218) Updating files: 60% (131/218) Updating files: 61% (133/218) Updating files: 62% (136/218) Updating files: 63% (138/218) Updating files: 64% (140/218) Updating files: 65% (142/218) Updating files: 66% (144/218) Updating files: 67% (147/218) Updating files: 68% (149/218) Updating files: 69% (151/218) Updating files: 70% (153/218) Updating files: 71% (155/218) Updating files: 72% (157/218) Updating files: 73% (160/218) Updating files: 74% (162/218) Updating files: 75% (164/218) Updating files: 76% (166/218) Updating files: 77% (168/218) Updating files: 78% (171/218) Updating files: 79% (173/218) Updating files: 80% (175/218) Updating files: 81% (177/218) Updating files: 82% (179/218) Updating files: 83% (181/218) Updating files: 84% (184/218) Updating files: 85% (186/218) Updating files: 86% (188/218) Updating files: 87% (190/218) Updating files: 88% (192/218) Updating files: 89% (195/218) Updating files: 90% (197/218) Updating files: 91% (199/218) Updating files: 92% (201/218) Updating files: 93% (203/218) Updating files: 94% (205/218) Updating files: 95% (208/218) Updating files: 96% (210/218) Updating files: 97% (212/218) Updating files: 98% (214/218) Updating files: 99% (216/218) Updating files: 100% (218/218) Updating files: 100% (218/218), done. 2023-10-30T13:50:18.670 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 78ff1f0a5df 0.83 2023-10-30T13:50:18.675 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.83 2023-10-30T13:50:19.278 INFO:tasks.workunit.client.0.smithi148.stdout:v0.84 2023-10-30T13:50:19.278 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 84 -eq 48 ] 2023-10-30T13:50:19.278 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.84 2023-10-30T13:50:19.279 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.84 2023-10-30T13:50:19.279 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.84 2023-10-30T13:50:28.274 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 9% (34/356) Updating files: 10% (36/356) Updating files: 11% (40/356) Updating files: 12% (43/356) Updating files: 13% (47/356) Updating files: 14% (50/356) Updating files: 15% (54/356) Updating files: 16% (57/356) Updating files: 17% (61/356) Updating files: 18% (65/356) Updating files: 19% (68/356) Updating files: 20% (72/356) Updating files: 21% (75/356) Updating files: 22% (79/356) Updating files: 22% (80/356) Updating files: 23% (82/356) Updating files: 24% (86/356) Updating files: 25% (89/356) Updating files: 26% (93/356) Updating files: 27% (97/356) Updating files: 28% (100/356) Updating files: 29% (104/356) Updating files: 30% (107/356) Updating files: 31% (111/356) Updating files: 32% (114/356) Updating files: 33% (118/356) Updating files: 34% (122/356) Updating files: 35% (125/356) Updating files: 36% (129/356) Updating files: 37% (132/356) Updating files: 38% (136/356) Updating files: 39% (139/356) Updating files: 40% (143/356) Updating files: 40% (144/356) Updating files: 41% (146/356) Updating files: 42% (150/356) Updating files: 43% (154/356) Updating files: 44% (157/356) Updating files: 45% (161/356) Updating files: 46% (164/356) Updating files: 47% (168/356) Updating files: 48% (171/356) Updating files: 49% (175/356) Updating files: 50% (178/356) Updating files: 51% (182/356) Updating files: 52% (186/356) Updating files: 53% (189/356) Updating files: 54% (193/356) Updating files: 55% (196/356) Updating files: 56% (200/356) Updating files: 57% (203/356) Updating files: 58% (207/356) Updating files: 59% (211/356) Updating files: 60% (214/356) Updating files: 61% (218/356) Updating files: 62% (221/356) Updating files: 63% (225/356) Updating files: 64% (228/356) Updating files: 65% (232/356) Updating files: 66% (235/356) Updating files: 67% (239/356) Updating files: 68% (243/356) Updating files: 68% (244/356) Updating files: 69% (246/356) Updating files: 70% (250/356) Updating files: 71% (253/356) Updating files: 72% (257/356) Updating files: 73% (260/356) Updating files: 74% (264/356) Updating files: 75% (267/356) Updating files: 76% (271/356) Updating files: 77% (275/356) Updating files: 78% (278/356) Updating files: 79% (282/356) Updating files: 80% (285/356) Updating files: 81% (289/356) Updating files: 82% (292/356) Updating files: 83% (296/356) Updating files: 84% (300/356) Updating files: 85% (303/356) Updating files: 86% (307/356) Updating files: 87% (310/356) Updating files: 88% (314/356) Updating files: 89% (317/356) Updating files: 90% (321/356) Updating files: 91% (324/356) Updating files: 92% (328/356) Updating files: 93% (332/356) Updating files: 94% (335/356) Updating files: 95% (339/356) Updating files: 96% (342/356) Updating files: 97% (346/356) Updating files: 98% (349/356) Updating files: 99% (353/356) Updating files: 100% (356/356) Updating files: 100% (356/356), done. 2023-10-30T13:50:28.393 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 8336f81c5c7 0.84 2023-10-30T13:50:28.401 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.84 2023-10-30T13:50:28.710 INFO:tasks.workunit.client.0.smithi148.stdout:v0.85 2023-10-30T13:50:28.710 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 85 -eq 48 ] 2023-10-30T13:50:28.711 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.85 2023-10-30T13:50:28.711 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.85 2023-10-30T13:50:28.711 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.85 2023-10-30T13:50:31.697 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 53% (131/243) Updating files: 54% (132/243) Updating files: 55% (134/243) Updating files: 56% (137/243) Updating files: 57% (139/243) Updating files: 58% (141/243) Updating files: 59% (144/243) Updating files: 60% (146/243) Updating files: 61% (149/243) Updating files: 62% (151/243) Updating files: 63% (154/243) Updating files: 64% (156/243) Updating files: 65% (158/243) Updating files: 66% (161/243) Updating files: 67% (163/243) Updating files: 68% (166/243) Updating files: 69% (168/243) Updating files: 70% (171/243) Updating files: 71% (173/243) Updating files: 72% (175/243) Updating files: 73% (178/243) Updating files: 74% (180/243) Updating files: 75% (183/243) Updating files: 76% (185/243) Updating files: 77% (188/243) Updating files: 78% (190/243) Updating files: 79% (192/243) Updating files: 80% (195/243) Updating files: 81% (197/243) Updating files: 82% (200/243) Updating files: 83% (202/243) Updating files: 84% (205/243) Updating files: 85% (207/243) Updating files: 86% (209/243) Updating files: 87% (212/243) Updating files: 88% (214/243) Updating files: 89% (217/243) Updating files: 90% (219/243) Updating files: 91% (222/243) Updating files: 92% (224/243) Updating files: 93% (226/243) Updating files: 93% (228/243) Updating files: 94% (229/243) Updating files: 95% (231/243) Updating files: 96% (234/243) Updating files: 97% (236/243) Updating files: 98% (239/243) Updating files: 99% (241/243) Updating files: 100% (243/243) Updating files: 100% (243/243), done. 2023-10-30T13:50:31.743 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at a0c22842db9 0.85 2023-10-30T13:50:31.750 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.85 2023-10-30T13:50:32.046 INFO:tasks.workunit.client.0.smithi148.stdout:v0.86 2023-10-30T13:50:32.047 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 86 -eq 48 ] 2023-10-30T13:50:32.047 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.86 2023-10-30T13:50:32.047 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.86 2023-10-30T13:50:32.047 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.86 2023-10-30T13:50:33.792 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:33.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:50:39.506 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 12% (56/447) Updating files: 13% (59/447) Updating files: 14% (63/447) Updating files: 15% (68/447) Updating files: 16% (72/447) Updating files: 17% (76/447) Updating files: 18% (81/447) Updating files: 19% (85/447) Updating files: 20% (90/447) Updating files: 21% (94/447) Updating files: 22% (99/447) Updating files: 23% (103/447) Updating files: 24% (108/447) Updating files: 25% (112/447) Updating files: 26% (117/447) Updating files: 27% (121/447) Updating files: 28% (126/447) Updating files: 29% (130/447) Updating files: 30% (135/447) Updating files: 30% (138/447) Updating files: 31% (139/447) Updating files: 32% (144/447) Updating files: 33% (148/447) Updating files: 34% (152/447) Updating files: 35% (157/447) Updating files: 36% (161/447) Updating files: 37% (166/447) Updating files: 38% (170/447) Updating files: 39% (175/447) Updating files: 40% (179/447) Updating files: 41% (184/447) Updating files: 42% (188/447) Updating files: 43% (193/447) Updating files: 44% (197/447) Updating files: 45% (202/447) Updating files: 46% (206/447) Updating files: 47% (211/447) Updating files: 48% (215/447) Updating files: 49% (220/447) Updating files: 50% (224/447) Updating files: 51% (228/447) Updating files: 52% (233/447) Updating files: 53% (237/447) Updating files: 54% (242/447) Updating files: 55% (246/447) Updating files: 56% (251/447) Updating files: 57% (255/447) Updating files: 58% (260/447) Updating files: 59% (264/447) Updating files: 60% (269/447) Updating files: 61% (273/447) Updating files: 62% (278/447) Updating files: 63% (282/447) Updating files: 64% (287/447) Updating files: 65% (291/447) Updating files: 66% (296/447) Updating files: 67% (300/447) Updating files: 68% (304/447) Updating files: 68% (305/447) Updating files: 69% (309/447) Updating files: 70% (313/447) Updating files: 71% (318/447) Updating files: 72% (322/447) Updating files: 73% (327/447) Updating files: 74% (331/447) Updating files: 75% (336/447) Updating files: 76% (340/447) Updating files: 77% (345/447) Updating files: 78% (349/447) Updating files: 79% (354/447) Updating files: 80% (358/447) Updating files: 81% (363/447) Updating files: 82% (367/447) Updating files: 83% (372/447) Updating files: 84% (376/447) Updating files: 85% (380/447) Updating files: 86% (385/447) Updating files: 87% (389/447) Updating files: 88% (394/447) Updating files: 89% (398/447) Updating files: 90% (403/447) Updating files: 90% (405/447) Updating files: 91% (407/447) Updating files: 92% (412/447) Updating files: 93% (416/447) Updating files: 94% (421/447) Updating files: 95% (425/447) Updating files: 96% (430/447) Updating files: 97% (434/447) Updating files: 98% (439/447) Updating files: 99% (443/447) Updating files: 100% (447/447) Updating files: 100% (447/447), done. 2023-10-30T13:50:39.565 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 97dcc0539df 0.86 2023-10-30T13:50:39.574 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.86 2023-10-30T13:50:39.711 INFO:tasks.workunit.client.0.smithi148.stdout:v0.87 2023-10-30T13:50:39.711 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 87 -eq 48 ] 2023-10-30T13:50:39.711 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.87 2023-10-30T13:50:39.712 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.87 2023-10-30T13:50:39.712 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.87 2023-10-30T13:50:40.871 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:41.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.013+0000 7f9285d3e640 1 Processor -- start 2023-10-30T13:50:41.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.013+0000 7f9285d3e640 1 -- start start 2023-10-30T13:50:41.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.013+0000 7f9285d3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f928006f7a0 0x7f928006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.013+0000 7f9285d3e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f9280071110 0x7f9280158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.013+0000 7f9285d3e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9280066cf0 con 0x7f92800c6bd0 2023-10-30T13:50:41.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.013+0000 7f9285d3e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92800700c0 con 0x7f9280071110 2023-10-30T13:50:41.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.013+0000 7f9285d3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92800c70d0 con 0x7f928006f7a0 2023-10-30T13:50:41.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927effd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f92800c6bd0 0x7f9280069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:57170/0 (socket says 172.21.15.148:57170) 2023-10-30T13:50:41.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927effd640 1 -- 172.21.15.148:0/2069490495 learned_addr learned my addr 172.21.15.148:0/2069490495 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:50:41.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927ffff640 1 --2- 172.21.15.148:0/2069490495 >> v2:172.21.15.148:3300/0 conn(0x7f9280071110 0x7f9280158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927f7fe640 1 --2- 172.21.15.148:0/2069490495 >> v2:172.21.15.150:3300/0 conn(0x7f928006f7a0 0x7f928006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927e7fc640 1 -- 172.21.15.148:0/2069490495 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140092052 0 0) 0x7f9280066cf0 con 0x7f92800c6bd0 2023-10-30T13:50:41.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927e7fc640 1 -- 172.21.15.148:0/2069490495 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9268003530 con 0x7f92800c6bd0 2023-10-30T13:50:41.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927ffff640 1 -- 172.21.15.148:0/2069490495 >> v2:172.21.15.150:3300/0 conn(0x7f928006f7a0 msgr2=0x7f928006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:41.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927ffff640 1 --2- 172.21.15.148:0/2069490495 >> v2:172.21.15.150:3300/0 conn(0x7f928006f7a0 0x7f928006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927ffff640 1 -- 172.21.15.148:0/2069490495 >> v1:172.21.15.148:6789/0 conn(0x7f92800c6bd0 legacy=0x7f9280069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927ffff640 1 -- 172.21.15.148:0/2069490495 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92800c7390 con 0x7f9280071110 2023-10-30T13:50:41.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927ffff640 1 --2- 172.21.15.148:0/2069490495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 0x7f9280158110 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f928006c830 tx=0x7f927801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6686f16cb46b58 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:41.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927e7fc640 1 -- 172.21.15.148:0/2069490495 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92780b0440 con 0x7f9280071110 2023-10-30T13:50:41.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927e7fc640 1 -- 172.21.15.148:0/2069490495 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92780b05a0 con 0x7f9280071110 2023-10-30T13:50:41.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.017+0000 7f927e7fc640 1 -- 172.21.15.148:0/2069490495 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92780b0910 con 0x7f9280071110 2023-10-30T13:50:41.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 -- 172.21.15.148:0/2069490495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 msgr2=0x7f9280158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 --2- 172.21.15.148:0/2069490495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 0x7f9280158110 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f928006c830 tx=0x7f927801f0f0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 -- 172.21.15.148:0/2069490495 shutdown_connections 2023-10-30T13:50:41.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 --2- 172.21.15.148:0/2069490495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 0x7f9280158110 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 --2- 172.21.15.148:0/2069490495 >> v2:172.21.15.150:3300/0 conn(0x7f928006f7a0 0x7f928006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 -- 172.21.15.148:0/2069490495 >> 172.21.15.148:0/2069490495 conn(0x7f92800c7910 msgr2=0x7f92800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:41.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 -- 172.21.15.148:0/2069490495 shutdown_connections 2023-10-30T13:50:41.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 -- 172.21.15.148:0/2069490495 wait complete. 2023-10-30T13:50:41.024 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 Processor -- start 2023-10-30T13:50:41.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 -- start start 2023-10-30T13:50:41.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928006f7a0 0x7f9280155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 0x7f928014f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92800c6bd0 0x7f92800af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.025 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92801536c0 con 0x7f9280071110 2023-10-30T13:50:41.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92801ef930 con 0x7f928006f7a0 2023-10-30T13:50:41.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f9285d3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92801e6020 con 0x7f92800c6bd0 2023-10-30T13:50:41.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f927effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 0x7f928014f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f927effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 0x7f928014f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55102/0 (socket says 172.21.15.148:55102) 2023-10-30T13:50:41.026 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.021+0000 7f927effd640 1 -- 172.21.15.148:0/3217850663 learned_addr learned my addr 172.21.15.148:0/3217850663 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:41.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f927f7fe640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928006f7a0 0x7f9280155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f927ffff640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92800c6bd0 0x7f92800af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f927effd640 1 -- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92800c6bd0 msgr2=0x7f92800af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f927effd640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92800c6bd0 0x7f92800af460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.027 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f927effd640 1 -- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928006f7a0 msgr2=0x7f9280155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f927effd640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928006f7a0 0x7f9280155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f927effd640 1 -- 172.21.15.148:0/3217850663 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9278009a70 con 0x7f9280071110 2023-10-30T13:50:41.028 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f927effd640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 0x7f928014f240 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f927400c420 tx=0x7f927400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:41.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f9267fff640 1 -- 172.21.15.148:0/3217850663 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9274005cf0 con 0x7f9280071110 2023-10-30T13:50:41.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f9267fff640 1 -- 172.21.15.148:0/3217850663 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9274005e70 con 0x7f9280071110 2023-10-30T13:50:41.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92801e6300 con 0x7f9280071110 2023-10-30T13:50:41.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f9267fff640 1 -- 172.21.15.148:0/3217850663 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740072f0 con 0x7f9280071110 2023-10-30T13:50:41.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9280155860 con 0x7f9280071110 2023-10-30T13:50:41.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9280158610 con 0x7f9280071110 2023-10-30T13:50:41.030 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f9267fff640 1 -- 172.21.15.148:0/3217850663 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9274007470 con 0x7f9280071110 2023-10-30T13:50:41.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f9267fff640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f926c0664a0 0x7f926c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.025+0000 7f9267fff640 1 -- 172.21.15.148:0/3217850663 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f9274083ba0 con 0x7f9280071110 2023-10-30T13:50:41.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.029+0000 7f927f7fe640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f926c0664a0 0x7f926c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.031 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.029+0000 7f927f7fe640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f926c0664a0 0x7f926c068960 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9270009ac0 tx=0x7f9270009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:41.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.033+0000 7f9267fff640 1 -- 172.21.15.148:0/3217850663 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9274054430 con 0x7f9280071110 2023-10-30T13:50:41.227 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.225+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f928004e170 con 0x7f9280071110 2023-10-30T13:50:41.228 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.225+0000 7f9267fff640 1 -- 172.21.15.148:0/3217850663 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f9274017020 con 0x7f9280071110 2023-10-30T13:50:41.228 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:41.229 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8145,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:59.158472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:41.229 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 158 2023-10-30T13:50:41.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f926c0664a0 msgr2=0x7f926c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f926c0664a0 0x7f926c068960 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9270009ac0 tx=0x7f9270009190 comp rx=0 tx=0).stop 2023-10-30T13:50:41.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 msgr2=0x7f928014f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 0x7f928014f240 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f927400c420 tx=0x7f927400c8e0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 shutdown_connections 2023-10-30T13:50:41.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f926c0664a0 0x7f926c068960 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9280071110 0x7f928014f240 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f92800c6bd0 0x7f92800af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 --2- 172.21.15.148:0/3217850663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f928006f7a0 0x7f9280155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 >> 172.21.15.148:0/3217850663 conn(0x7f92800c7910 msgr2=0x7f92800c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:41.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 shutdown_connections 2023-10-30T13:50:41.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.229+0000 7f9285d3e640 1 -- 172.21.15.148:0/3217850663 wait complete. 2023-10-30T13:50:41.248 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:50:41.249 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:50:41.249 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:50:41.275 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:50:41.275 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:41.456 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbfaa79640 1 Processor -- start 2023-10-30T13:50:41.457 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbfaa79640 1 -- start start 2023-10-30T13:50:41.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbfaa79640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffbf41538a0 0x7ffbf415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbfaa79640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf40cb800 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.458 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbfaa79640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf415dc30 con 0x7ffbf40cf160 2023-10-30T13:50:41.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbfaa79640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf40cb330 con 0x7ffbf41538a0 2023-10-30T13:50:41.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbf8815640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffbf41538a0 0x7ffbf415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbf8815640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ffbf41538a0 0x7ffbf415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55106/0 (socket says 172.21.15.148:55106) 2023-10-30T13:50:41.459 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbf8815640 1 -- 172.21.15.148:0/1999624771 learned_addr learned my addr 172.21.15.148:0/1999624771 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:41.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbf8815640 1 -- 172.21.15.148:0/1999624771 >> v1:172.21.15.150:6789/0 conn(0x7ffbf40cf160 legacy=0x7ffbf4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbf8815640 1 -- 172.21.15.148:0/1999624771 >> v1:172.21.15.148:6789/0 conn(0x7ffbf40ca0e0 legacy=0x7ffbf415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.460 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.457+0000 7ffbf8815640 1 -- 172.21.15.148:0/1999624771 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf40cad70 con 0x7ffbf41538a0 2023-10-30T13:50:41.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbf8815640 1 --2- 172.21.15.148:0/1999624771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf41538a0 0x7ffbf415d6f0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7ffbe8009870 tx=0x7ffbe801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02fa66c4ff262c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:41.462 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbf37fe640 1 -- 172.21.15.148:0/1999624771 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe80a8a30 con 0x7ffbf41538a0 2023-10-30T13:50:41.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbf37fe640 1 -- 172.21.15.148:0/1999624771 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe80a8b90 con 0x7ffbf41538a0 2023-10-30T13:50:41.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbf37fe640 1 -- 172.21.15.148:0/1999624771 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe80b0440 con 0x7ffbf41538a0 2023-10-30T13:50:41.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbfaa79640 1 -- 172.21.15.148:0/1999624771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf41538a0 msgr2=0x7ffbf415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbfaa79640 1 --2- 172.21.15.148:0/1999624771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf41538a0 0x7ffbf415d6f0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7ffbe8009870 tx=0x7ffbe801ed30 comp rx=0 tx=0).stop 2023-10-30T13:50:41.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbfaa79640 1 -- 172.21.15.148:0/1999624771 shutdown_connections 2023-10-30T13:50:41.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbfaa79640 1 --2- 172.21.15.148:0/1999624771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf41538a0 0x7ffbf415d6f0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbfaa79640 1 -- 172.21.15.148:0/1999624771 >> 172.21.15.148:0/1999624771 conn(0x7ffbf406bfe0 msgr2=0x7ffbf406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:41.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbfaa79640 1 -- 172.21.15.148:0/1999624771 shutdown_connections 2023-10-30T13:50:41.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.461+0000 7ffbfaa79640 1 -- 172.21.15.148:0/1999624771 wait complete. 2023-10-30T13:50:41.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 Processor -- start 2023-10-30T13:50:41.465 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 -- start start 2023-10-30T13:50:41.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf40ca0e0 0x7ffbf4157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf40cf160 0x7ffbf40b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbf41538a0 0x7ffbf41e48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf40b1c10 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf41e38d0 con 0x7ffbf40cf160 2023-10-30T13:50:41.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf41e4450 con 0x7ffbf41538a0 2023-10-30T13:50:41.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf8815640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf40ca0e0 0x7ffbf4157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf8815640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf40ca0e0 0x7ffbf4157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:55116/0 (socket says 172.21.15.148:55116) 2023-10-30T13:50:41.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf8815640 1 -- 172.21.15.148:0/2348936022 learned_addr learned my addr 172.21.15.148:0/2348936022 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:41.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf3fff640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf40cf160 0x7ffbf40b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf9016640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbf41538a0 0x7ffbf41e48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf8815640 1 -- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbf41538a0 msgr2=0x7ffbf41e48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf8815640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbf41538a0 0x7ffbf41e48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf8815640 1 -- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf40cf160 msgr2=0x7ffbf40b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf8815640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf40cf160 0x7ffbf40b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf8815640 1 -- 172.21.15.148:0/2348936022 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe80094d0 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf8815640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf40ca0e0 0x7ffbf4157ee0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7ffbe80b3040 tx=0x7ffbe801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:41.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf17fa640 1 -- 172.21.15.148:0/2348936022 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe80be070 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf17fa640 1 -- 172.21.15.148:0/2348936022 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe80ba400 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbf17fa640 1 -- 172.21.15.148:0/2348936022 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe80ba7b0 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf4158420 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.471 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf41e3b20 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.465+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf4067a30 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.472 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.469+0000 7ffbf17fa640 1 -- 172.21.15.148:0/2348936022 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffbe80ba910 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.469+0000 7ffbf17fa640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbd80664d0 0x7ffbd8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:41.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.469+0000 7ffbf17fa640 1 -- 172.21.15.148:0/2348936022 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7ffbe8122340 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.469+0000 7ffbf3fff640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbd80664d0 0x7ffbd8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:41.473 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.469+0000 7ffbf3fff640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbd80664d0 0x7ffbd8068990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ffbf41489d0 tx=0x7ffbe0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:41.474 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.473+0000 7ffbf17fa640 1 -- 172.21.15.148:0/2348936022 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffbe80ef0e0 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.657+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbf404e170 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbf17fa640 1 -- 172.21.15.148:0/2348936022 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6534 (secure 0 0 0) 0x7ffbe80f2700 con 0x7ffbf40ca0e0 2023-10-30T13:50:41.660 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:41.661 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8145,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:59.158472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 158 2023-10-30T13:50:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbd80664d0 msgr2=0x7ffbd8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbd80664d0 0x7ffbd8068990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ffbf41489d0 tx=0x7ffbe0009040 comp rx=0 tx=0).stop 2023-10-30T13:50:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf40ca0e0 msgr2=0x7ffbf4157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf40ca0e0 0x7ffbf4157ee0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7ffbe80b3040 tx=0x7ffbe801bad0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 shutdown_connections 2023-10-30T13:50:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffbd80664d0 0x7ffbd8068990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbf41538a0 0x7ffbf41e48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbf40cf160 0x7ffbf40b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 --2- 172.21.15.148:0/2348936022 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbf40ca0e0 0x7ffbf4157ee0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 >> 172.21.15.148:0/2348936022 conn(0x7ffbf406bfe0 msgr2=0x7ffbf40c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 shutdown_connections 2023-10-30T13:50:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:41.661+0000 7ffbfaa79640 1 -- 172.21.15.148:0/2348936022 wait complete. 2023-10-30T13:50:41.681 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:50:43.683 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:43.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff48af3e640 1 Processor -- start 2023-10-30T13:50:43.834 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff48af3e640 1 -- start start 2023-10-30T13:50:43.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff48af3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff4841538a0 0x7ff48415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:43.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff48af3e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4840cb800 con 0x7ff4840cf160 2023-10-30T13:50:43.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff48af3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48415dc30 con 0x7ff4840ca0e0 2023-10-30T13:50:43.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff48af3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4840cb330 con 0x7ff4841538a0 2023-10-30T13:50:43.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff4894db640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff4840cf160 0x7ff484164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56742/0 (socket says 172.21.15.148:56742) 2023-10-30T13:50:43.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff4894db640 1 -- 172.21.15.148:0/2886690503 learned_addr learned my addr 172.21.15.148:0/2886690503 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:50:43.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff488cda640 1 --2- 172.21.15.148:0/2886690503 >> v2:172.21.15.150:3300/0 conn(0x7ff4841538a0 0x7ff48415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:43.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff4837fe640 1 -- 172.21.15.148:0/2886690503 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639369355 0 0) 0x7ff4840cb800 con 0x7ff4840cf160 2023-10-30T13:50:43.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff4837fe640 1 -- 172.21.15.148:0/2886690503 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff474003530 con 0x7ff4840cf160 2023-10-30T13:50:43.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff488cda640 1 -- 172.21.15.148:0/2886690503 >> v1:172.21.15.150:6789/0 conn(0x7ff4840ca0e0 legacy=0x7ff48415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:43.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff4837fe640 1 -- 172.21.15.148:0/2886690503 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 19465543 0 0) 0x7ff474003530 con 0x7ff4840cf160 2023-10-30T13:50:43.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff488cda640 1 -- 172.21.15.148:0/2886690503 >> v1:172.21.15.148:6789/0 conn(0x7ff4840cf160 legacy=0x7ff484164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:43.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.833+0000 7ff488cda640 1 -- 172.21.15.148:0/2886690503 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4840cad70 con 0x7ff4841538a0 2023-10-30T13:50:43.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff488cda640 1 --2- 172.21.15.148:0/2886690503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4841538a0 0x7ff48415d6f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff4780097c0 tx=0x7ff47801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21475721a035f99e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:43.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff4837fe640 1 -- 172.21.15.148:0/2886690503 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c001a00 con 0x7ff4841538a0 2023-10-30T13:50:43.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff4837fe640 1 -- 172.21.15.148:0/2886690503 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4780a8610 con 0x7ff4841538a0 2023-10-30T13:50:43.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff4837fe640 1 -- 172.21.15.148:0/2886690503 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4780a8980 con 0x7ff4841538a0 2023-10-30T13:50:43.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff48af3e640 1 -- 172.21.15.148:0/2886690503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4841538a0 msgr2=0x7ff48415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:43.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff48af3e640 1 --2- 172.21.15.148:0/2886690503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4841538a0 0x7ff48415d6f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff4780097c0 tx=0x7ff47801eb90 comp rx=0 tx=0).stop 2023-10-30T13:50:43.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff48af3e640 1 -- 172.21.15.148:0/2886690503 shutdown_connections 2023-10-30T13:50:43.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff48af3e640 1 --2- 172.21.15.148:0/2886690503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4841538a0 0x7ff48415d6f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:43.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff48af3e640 1 -- 172.21.15.148:0/2886690503 >> 172.21.15.148:0/2886690503 conn(0x7ff48406bfe0 msgr2=0x7ff48406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:43.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff48af3e640 1 -- 172.21.15.148:0/2886690503 shutdown_connections 2023-10-30T13:50:43.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.837+0000 7ff48af3e640 1 -- 172.21.15.148:0/2886690503 wait complete. 2023-10-30T13:50:43.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff48af3e640 1 Processor -- start 2023-10-30T13:50:43.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff48af3e640 1 -- start start 2023-10-30T13:50:43.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff48af3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4840ca0e0 0x7ff4840b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:43.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff48af3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4840cf160 0x7ff484148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:43.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff48af3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4841538a0 0x7ff4840cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:43.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff48af3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4840ce960 con 0x7ff4841538a0 2023-10-30T13:50:43.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff48af3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff484158420 con 0x7ff4840cf160 2023-10-30T13:50:43.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff48af3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4840b41e0 con 0x7ff4840ca0e0 2023-10-30T13:50:43.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4894db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4841538a0 0x7ff4840cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:43.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4894db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4841538a0 0x7ff4840cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33526/0 (socket says 172.21.15.148:33526) 2023-10-30T13:50:43.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4894db640 1 -- 172.21.15.148:0/39047739 learned_addr learned my addr 172.21.15.148:0/39047739 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:43.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff483fff640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4840cf160 0x7ff484148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:43.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff488cda640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4840ca0e0 0x7ff4840b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:43.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4894db640 1 -- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4840ca0e0 msgr2=0x7ff4840b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:43.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4894db640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4840ca0e0 0x7ff4840b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:43.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4894db640 1 -- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4840cf160 msgr2=0x7ff484148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:43.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4894db640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4840cf160 0x7ff484148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:43.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4894db640 1 -- 172.21.15.148:0/39047739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478009420 con 0x7ff4841538a0 2023-10-30T13:50:43.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff488cda640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4840ca0e0 0x7ff4840b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:43.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4894db640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4841538a0 0x7ff4840cd630 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00cbf0 tx=0x7ff47c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:43.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4817fa640 1 -- 172.21.15.148:0/39047739 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c00ad50 con 0x7ff4841538a0 2023-10-30T13:50:43.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4817fa640 1 -- 172.21.15.148:0/39047739 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47c012400 con 0x7ff4841538a0 2023-10-30T13:50:43.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4840b37e0 con 0x7ff4841538a0 2023-10-30T13:50:43.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.841+0000 7ff4817fa640 1 -- 172.21.15.148:0/39047739 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c0127f0 con 0x7ff4841538a0 2023-10-30T13:50:43.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.845+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48415d890 con 0x7ff4841538a0 2023-10-30T13:50:43.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.845+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48404e170 con 0x7ff4841538a0 2023-10-30T13:50:43.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.845+0000 7ff4817fa640 1 -- 172.21.15.148:0/39047739 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff47c012950 con 0x7ff4841538a0 2023-10-30T13:50:43.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.845+0000 7ff4817fa640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4680664a0 0x7ff468068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:43.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.845+0000 7ff4817fa640 1 -- 172.21.15.148:0/39047739 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7ff47c084c70 con 0x7ff4841538a0 2023-10-30T13:50:43.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.845+0000 7ff488cda640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4680664a0 0x7ff468068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:43.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.845+0000 7ff488cda640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4680664a0 0x7ff468068960 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff4780b2000 tx=0x7ff4780aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:43.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:43.849+0000 7ff4817fa640 1 -- 172.21.15.148:0/39047739 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff47c051900 con 0x7ff4841538a0 2023-10-30T13:50:44.032 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4840b1dd0 con 0x7ff4841538a0 2023-10-30T13:50:44.033 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff4817fa640 1 -- 172.21.15.148:0/39047739 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6534 (secure 0 0 0) 0x7ff47c054f20 con 0x7ff4841538a0 2023-10-30T13:50:44.033 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:44.034 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8145,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:59.158472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:44.034 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 158 2023-10-30T13:50:44.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4680664a0 msgr2=0x7ff468068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:44.035 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4680664a0 0x7ff468068960 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff4780b2000 tx=0x7ff4780aa070 comp rx=0 tx=0).stop 2023-10-30T13:50:44.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4841538a0 msgr2=0x7ff4840cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:44.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4841538a0 0x7ff4840cd630 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00cbf0 tx=0x7ff47c00a430 comp rx=0 tx=0).stop 2023-10-30T13:50:44.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 shutdown_connections 2023-10-30T13:50:44.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4680664a0 0x7ff468068960 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:44.036 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4841538a0 0x7ff4840cd630 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:44.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4840cf160 0x7ff484148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:44.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 --2- 172.21.15.148:0/39047739 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4840ca0e0 0x7ff4840b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:44.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 >> 172.21.15.148:0/39047739 conn(0x7ff48406bfe0 msgr2=0x7ff4841516c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:44.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 shutdown_connections 2023-10-30T13:50:44.037 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:44.033+0000 7ff48af3e640 1 -- 172.21.15.148:0/39047739 wait complete. 2023-10-30T13:50:44.055 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:50:46.056 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:46.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3fa6b640 1 Processor -- start 2023-10-30T13:50:46.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3fa6b640 1 -- start start 2023-10-30T13:50:46.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3fa6b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf3815d6f0 0x7fbf3815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:46.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3fa6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf380cf160 0x7fbf380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:46.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3fa6b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf380cb800 con 0x7fbf381538a0 2023-10-30T13:50:46.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3fa6b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf380cb330 con 0x7fbf3815d6f0 2023-10-30T13:50:46.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3fa6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380cad70 con 0x7fbf380cf160 2023-10-30T13:50:46.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3d006640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf3815d6f0 0x7fbf3815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:46.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3d006640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf3815d6f0 0x7fbf3815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:41306/0 (socket says 172.21.15.148:41306) 2023-10-30T13:50:46.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3e008640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf380cf160 0x7fbf380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:46.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.205+0000 7fbf3d006640 1 -- 172.21.15.148:0/2759372140 learned_addr learned my addr 172.21.15.148:0/2759372140 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:46.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.209+0000 7fbf3d006640 1 -- 172.21.15.148:0/2759372140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf380cf160 msgr2=0x7fbf380cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:46.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.209+0000 7fbf3d006640 1 --2- 172.21.15.148:0/2759372140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf380cf160 0x7fbf380cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:46.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.209+0000 7fbf3d006640 1 -- 172.21.15.148:0/2759372140 >> v1:172.21.15.150:6789/0 conn(0x7fbf381538a0 legacy=0x7fbf380ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:46.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.209+0000 7fbf3d006640 1 -- 172.21.15.148:0/2759372140 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380cb050 con 0x7fbf3815d6f0 2023-10-30T13:50:46.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.209+0000 7fbf3d006640 1 --2- 172.21.15.148:0/2759372140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf3815d6f0 0x7fbf3815a090 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbf28009870 tx=0x7fbf2801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52bc077e2230fff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:46.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.209+0000 7fbf3c805640 1 -- 172.21.15.148:0/2759372140 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf280a8a30 con 0x7fbf3815d6f0 2023-10-30T13:50:46.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.209+0000 7fbf3c805640 1 -- 172.21.15.148:0/2759372140 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf280a8b90 con 0x7fbf3815d6f0 2023-10-30T13:50:46.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.209+0000 7fbf3c805640 1 -- 172.21.15.148:0/2759372140 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf280b0440 con 0x7fbf3815d6f0 2023-10-30T13:50:46.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/2759372140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf3815d6f0 msgr2=0x7fbf3815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:46.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 --2- 172.21.15.148:0/2759372140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf3815d6f0 0x7fbf3815a090 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbf28009870 tx=0x7fbf2801ed30 comp rx=0 tx=0).stop 2023-10-30T13:50:46.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/2759372140 shutdown_connections 2023-10-30T13:50:46.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 --2- 172.21.15.148:0/2759372140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf380cf160 0x7fbf380cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:46.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 --2- 172.21.15.148:0/2759372140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf3815d6f0 0x7fbf3815a090 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:46.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/2759372140 >> 172.21.15.148:0/2759372140 conn(0x7fbf3806bfe0 msgr2=0x7fbf3806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:46.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/2759372140 shutdown_connections 2023-10-30T13:50:46.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/2759372140 wait complete. 2023-10-30T13:50:46.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 Processor -- start 2023-10-30T13:50:46.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 -- start start 2023-10-30T13:50:46.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf380cf160 0x7fbf38159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:46.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf381538a0 0x7fbf380c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:46.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf3815d6f0 0x7fbf381e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:46.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf381e3bf0 con 0x7fbf3815d6f0 2023-10-30T13:50:46.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf381580f0 con 0x7fbf380cf160 2023-10-30T13:50:46.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3fa6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380c59a0 con 0x7fbf381538a0 2023-10-30T13:50:46.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3e008640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf3815d6f0 0x7fbf381e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:46.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3d807640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf380cf160 0x7fbf38159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:46.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3e008640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf3815d6f0 0x7fbf381e2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33540/0 (socket says 172.21.15.148:33540) 2023-10-30T13:50:46.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3e008640 1 -- 172.21.15.148:0/691970399 learned_addr learned my addr 172.21.15.148:0/691970399 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:46.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.213+0000 7fbf3d006640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf381538a0 0x7fbf380c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:46.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3e008640 1 -- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf381538a0 msgr2=0x7fbf380c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:46.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3e008640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf381538a0 0x7fbf380c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:46.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3e008640 1 -- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf380cf160 msgr2=0x7fbf38159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:46.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3e008640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf380cf160 0x7fbf38159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:46.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3e008640 1 -- 172.21.15.148:0/691970399 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf280094d0 con 0x7fbf3815d6f0 2023-10-30T13:50:46.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3e008640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf3815d6f0 0x7fbf381e2a40 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000efc0 tx=0x7fbf3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:46.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf2e7fc640 1 -- 172.21.15.148:0/691970399 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30009b10 con 0x7fbf3815d6f0 2023-10-30T13:50:46.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf2e7fc640 1 -- 172.21.15.148:0/691970399 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3000f040 con 0x7fbf3815d6f0 2023-10-30T13:50:46.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf381584d0 con 0x7fbf3815d6f0 2023-10-30T13:50:46.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf2e7fc640 1 -- 172.21.15.148:0/691970399 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf300146c0 con 0x7fbf3815d6f0 2023-10-30T13:50:46.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf380c4f60 con 0x7fbf3815d6f0 2023-10-30T13:50:46.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf380b2a50 con 0x7fbf3815d6f0 2023-10-30T13:50:46.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf2e7fc640 1 -- 172.21.15.148:0/691970399 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf30014820 con 0x7fbf3815d6f0 2023-10-30T13:50:46.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf2e7fc640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbf1c0664d0 0x7fbf1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:46.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf2e7fc640 1 -- 172.21.15.148:0/691970399 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fbf30085080 con 0x7fbf3815d6f0 2023-10-30T13:50:46.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.217+0000 7fbf3d807640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbf1c0664d0 0x7fbf1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:46.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.221+0000 7fbf3d807640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbf1c0664d0 0x7fbf1c068990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbf34001810 tx=0x7fbf34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.221+0000 7fbf2e7fc640 1 -- 172.21.15.148:0/691970399 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbf300b6090 con 0x7fbf3815d6f0 2023-10-30T13:50:46.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.401+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf3804e170 con 0x7fbf3815d6f0 2023-10-30T13:50:46.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.401+0000 7fbf2e7fc640 1 -- 172.21.15.148:0/691970399 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6534 (secure 0 0 0) 0x7fbf30018070 con 0x7fbf3815d6f0 2023-10-30T13:50:46.404 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:46.404 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8145,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:59.158472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:46.404 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 158 2023-10-30T13:50:46.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbf1c0664d0 msgr2=0x7fbf1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:46.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbf1c0664d0 0x7fbf1c068990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbf34001810 tx=0x7fbf34009040 comp rx=0 tx=0).stop 2023-10-30T13:50:46.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf3815d6f0 msgr2=0x7fbf381e2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:46.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf3815d6f0 0x7fbf381e2a40 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000efc0 tx=0x7fbf3000c490 comp rx=0 tx=0).stop 2023-10-30T13:50:46.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 shutdown_connections 2023-10-30T13:50:46.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbf1c0664d0 0x7fbf1c068990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:46.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf3815d6f0 0x7fbf381e2a40 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:46.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbf381538a0 0x7fbf380c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:46.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 --2- 172.21.15.148:0/691970399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf380cf160 0x7fbf38159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:46.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 >> 172.21.15.148:0/691970399 conn(0x7fbf3806bfe0 msgr2=0x7fbf3814ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:46.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 shutdown_connections 2023-10-30T13:50:46.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:46.405+0000 7fbf3fa6b640 1 -- 172.21.15.148:0/691970399 wait complete. 2023-10-30T13:50:46.424 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:50:48.426 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:48.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.573+0000 7fe04a4ad640 1 Processor -- start 2023-10-30T13:50:48.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe04a4ad640 1 -- start start 2023-10-30T13:50:48.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe04a4ad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe04415cd50 0x7fe0440c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:48.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe04a4ad640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04415a6c0 con 0x7fe0440cefa0 2023-10-30T13:50:48.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe04a4ad640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04415d300 con 0x7fe0441538f0 2023-10-30T13:50:48.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe04a4ad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04415d5e0 con 0x7fe04415cd50 2023-10-30T13:50:48.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe049cac640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe0440cefa0 0x7fe0441640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56748/0 (socket says 172.21.15.148:56748) 2023-10-30T13:50:48.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe049cac640 1 -- 172.21.15.148:0/3595603511 learned_addr learned my addr 172.21.15.148:0/3595603511 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:50:48.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe048caa640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe04415cd50 0x7fe0440c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:48.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180706585 0 0) 0x7fe04415a6c0 con 0x7fe0440cefa0 2023-10-30T13:50:48.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe030003530 con 0x7fe0440cefa0 2023-10-30T13:50:48.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4049346164 0 0) 0x7fe030003530 con 0x7fe0440cefa0 2023-10-30T13:50:48.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe04415cd50 msgr2=0x7fe0440c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:48.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe04415cd50 0x7fe0440c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 >> v1:172.21.15.150:6789/0 conn(0x7fe0441538f0 legacy=0x7fe044159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:48.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04415d8c0 con 0x7fe0440cefa0 2023-10-30T13:50:48.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fe03c001a00 con 0x7fe0440cefa0 2023-10-30T13:50:48.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 >> v1:172.21.15.148:6789/0 conn(0x7fe0440cefa0 legacy=0x7fe0441640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:48.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0300039b0 0x7fe030003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:48.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe04415cd50 0x7fe0300047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:48.583 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe030004d10 0x7fe030005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:48.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04415d5e0 con 0x7fe0300039b0 2023-10-30T13:50:48.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03000bb00 con 0x7fe030004d10 2023-10-30T13:50:48.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03000bcc0 con 0x7fe04415cd50 2023-10-30T13:50:48.584 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe0494ab640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0300039b0 0x7fe030003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:48.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe048caa640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe04415cd50 0x7fe0300047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:48.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe049cac640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe030004d10 0x7fe030005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:48.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe0494ab640 1 -- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe04415cd50 msgr2=0x7fe0300047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:48.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe0494ab640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe04415cd50 0x7fe0300047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe0494ab640 1 -- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe030004d10 msgr2=0x7fe030005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:48.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe0494ab640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe030004d10 0x7fe030005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.577+0000 7fe0494ab640 1 -- 172.21.15.148:0/3595603511 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe030003530 con 0x7fe0300039b0 2023-10-30T13:50:48.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe0494ab640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0300039b0 0x7fe030003dd0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe040001750 tx=0x7fe04001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558e9c9bb55e7774 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:48.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0400052d0 con 0x7fe0300039b0 2023-10-30T13:50:48.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0400a7050 con 0x7fe0300039b0 2023-10-30T13:50:48.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe03bfff640 1 -- 172.21.15.148:0/3595603511 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040003600 con 0x7fe0300039b0 2023-10-30T13:50:48.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3595603511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0300039b0 msgr2=0x7fe030003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:48.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe04a4ad640 1 --2- 172.21.15.148:0/3595603511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0300039b0 0x7fe030003dd0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe040001750 tx=0x7fe04001c3c0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe049cac640 1 -- 172.21.15.148:0/3595603511 reap_dead start 2023-10-30T13:50:48.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3595603511 shutdown_connections 2023-10-30T13:50:48.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3595603511 >> 172.21.15.148:0/3595603511 conn(0x7fe04406a640 msgr2=0x7fe044065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:48.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3595603511 shutdown_connections 2023-10-30T13:50:48.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.581+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3595603511 wait complete. 2023-10-30T13:50:48.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 Processor -- start 2023-10-30T13:50:48.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 -- start start 2023-10-30T13:50:48.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0300039b0 0x7fe044152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:48.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe04415b570 0x7fe0441ef150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:48.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0441ef690 0x7fe0440b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:48.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe044148640 con 0x7fe0441ef690 2023-10-30T13:50:48.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04413ef40 con 0x7fe04415b570 2023-10-30T13:50:48.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0441f2760 con 0x7fe0300039b0 2023-10-30T13:50:48.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe049cac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0441ef690 0x7fe0440b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:48.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe049cac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0441ef690 0x7fe0440b4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33560/0 (socket says 172.21.15.148:33560) 2023-10-30T13:50:48.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe048caa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe04415b570 0x7fe0441ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:48.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe049cac640 1 -- 172.21.15.148:0/3991515997 learned_addr learned my addr 172.21.15.148:0/3991515997 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:48.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe0494ab640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0300039b0 0x7fe044152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:48.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe049cac640 1 -- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0300039b0 msgr2=0x7fe044152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:48.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe049cac640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0300039b0 0x7fe044152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe049cac640 1 -- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe04415b570 msgr2=0x7fe0441ef150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:48.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe049cac640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe04415b570 0x7fe0441ef150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe049cac640 1 -- 172.21.15.148:0/3991515997 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04415ad60 con 0x7fe0441ef690 2023-10-30T13:50:48.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe049cac640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0441ef690 0x7fe0440b4320 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe044066500 tx=0x7fe03c011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:48.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe039ffb640 1 -- 172.21.15.148:0/3991515997 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c00f8c0 con 0x7fe0441ef690 2023-10-30T13:50:48.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe039ffb640 1 -- 172.21.15.148:0/3991515997 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c00fa40 con 0x7fe0441ef690 2023-10-30T13:50:48.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe039ffb640 1 -- 172.21.15.148:0/3991515997 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c00fdf0 con 0x7fe0441ef690 2023-10-30T13:50:48.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0441f29d0 con 0x7fe0441ef690 2023-10-30T13:50:48.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.585+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0441f2f90 con 0x7fe0441ef690 2023-10-30T13:50:48.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.589+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0441f31b0 con 0x7fe0441ef690 2023-10-30T13:50:48.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.589+0000 7fe039ffb640 1 -- 172.21.15.148:0/3991515997 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe03c01f460 con 0x7fe0441ef690 2023-10-30T13:50:48.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.589+0000 7fe039ffb640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe0280664d0 0x7fe028068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:48.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.589+0000 7fe039ffb640 1 -- 172.21.15.148:0/3991515997 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fe03c087540 con 0x7fe0441ef690 2023-10-30T13:50:48.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.589+0000 7fe0494ab640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe0280664d0 0x7fe028068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:48.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.589+0000 7fe0494ab640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe0280664d0 0x7fe028068990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe0300057b0 tx=0x7fe0400011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:48.597 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.593+0000 7fe039ffb640 1 -- 172.21.15.148:0/3991515997 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe03c0541f0 con 0x7fe0441ef690 2023-10-30T13:50:48.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.773+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe04404e170 con 0x7fe0441ef690 2023-10-30T13:50:48.774 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.773+0000 7fe039ffb640 1 -- 172.21.15.148:0/3991515997 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6534 (secure 0 0 0) 0x7fe03c057810 con 0x7fe0441ef690 2023-10-30T13:50:48.775 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:48.775 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8145,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:59.158472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:48.776 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 158 2023-10-30T13:50:48.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe0280664d0 msgr2=0x7fe028068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:48.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe0280664d0 0x7fe028068990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe0300057b0 tx=0x7fe0400011a0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.777 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0441ef690 msgr2=0x7fe0440b4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:48.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0441ef690 0x7fe0440b4320 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe044066500 tx=0x7fe03c011c20 comp rx=0 tx=0).stop 2023-10-30T13:50:48.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 shutdown_connections 2023-10-30T13:50:48.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fe0280664d0 0x7fe028068990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe0441ef690 0x7fe0440b4320 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.778 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe04415b570 0x7fe0441ef150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 --2- 172.21.15.148:0/3991515997 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fe0300039b0 0x7fe044152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:48.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 >> 172.21.15.148:0/3991515997 conn(0x7fe04406a640 msgr2=0x7fe04406bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:48.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 shutdown_connections 2023-10-30T13:50:48.779 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:48.777+0000 7fe04a4ad640 1 -- 172.21.15.148:0/3991515997 wait complete. 2023-10-30T13:50:48.796 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:50:50.797 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:50.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.945+0000 7f760990f640 1 Processor -- start 2023-10-30T13:50:50.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.945+0000 7f760990f640 1 -- start start 2023-10-30T13:50:50.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f760990f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f760415d6f0 0x7f760415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:50.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f760990f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 0x7f76040cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:50.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f760990f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040cb800 con 0x7f76041538a0 2023-10-30T13:50:50.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f760990f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76040cb330 con 0x7f760415d6f0 2023-10-30T13:50:50.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f760990f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76040cad70 con 0x7f76040cf160 2023-10-30T13:50:50.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f75fa7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f760415d6f0 0x7f760415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:50.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f75fa7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f760415d6f0 0x7f760415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33572/0 (socket says 172.21.15.148:33572) 2023-10-30T13:50:50.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f75fa7fc640 1 -- 172.21.15.148:0/3185857580 learned_addr learned my addr 172.21.15.148:0/3185857580 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:50.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f76037fe640 1 --2- 172.21.15.148:0/3185857580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 0x7f76040cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:50.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f75fa7fc640 1 -- 172.21.15.148:0/3185857580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 msgr2=0x7f76040cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:50.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f75fa7fc640 1 --2- 172.21.15.148:0/3185857580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 0x7f76040cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:50.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f75fa7fc640 1 -- 172.21.15.148:0/3185857580 >> v1:172.21.15.148:6789/0 conn(0x7f76041538a0 legacy=0x7f76040ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:50.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f75fa7fc640 1 -- 172.21.15.148:0/3185857580 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76040cb050 con 0x7f760415d6f0 2023-10-30T13:50:50.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f75fa7fc640 1 --2- 172.21.15.148:0/3185857580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 0x7f760415a090 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009870 tx=0x7f75ec01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8595f090579dde3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:50.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.949+0000 7f76027fc640 1 -- 172.21.15.148:0/3185857580 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0a8a30 con 0x7f760415d6f0 2023-10-30T13:50:50.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f76027fc640 1 -- 172.21.15.148:0/3185857580 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec0a8b90 con 0x7f760415d6f0 2023-10-30T13:50:50.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f76027fc640 1 -- 172.21.15.148:0/3185857580 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0b0440 con 0x7f760415d6f0 2023-10-30T13:50:50.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 -- 172.21.15.148:0/3185857580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 msgr2=0x7f760415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:50.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 --2- 172.21.15.148:0/3185857580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 0x7f760415a090 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009870 tx=0x7f75ec01ed30 comp rx=0 tx=0).stop 2023-10-30T13:50:50.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 -- 172.21.15.148:0/3185857580 shutdown_connections 2023-10-30T13:50:50.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 --2- 172.21.15.148:0/3185857580 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 0x7f760415a090 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:50.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 --2- 172.21.15.148:0/3185857580 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 0x7f76040cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:50.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 -- 172.21.15.148:0/3185857580 >> 172.21.15.148:0/3185857580 conn(0x7f760406bfe0 msgr2=0x7f760406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:50.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 -- 172.21.15.148:0/3185857580 shutdown_connections 2023-10-30T13:50:50.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 -- 172.21.15.148:0/3185857580 wait complete. 2023-10-30T13:50:50.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 Processor -- start 2023-10-30T13:50:50.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 -- start start 2023-10-30T13:50:50.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 0x7f7604159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:50.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76041538a0 0x7f7604148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:50.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 0x7f76040c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:50.958 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604158f60 con 0x7f760415d6f0 2023-10-30T13:50:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041e4660 con 0x7f76041538a0 2023-10-30T13:50:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f760990f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7604158a20 con 0x7f76040cf160 2023-10-30T13:50:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f76037fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 0x7f76040c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f76037fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 0x7f76040c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:33584/0 (socket says 172.21.15.148:33584) 2023-10-30T13:50:50.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f76037fe640 1 -- 172.21.15.148:0/3637215254 learned_addr learned my addr 172.21.15.148:0/3637215254 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.953+0000 7f75fa7fc640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76041538a0 0x7f7604148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f7602ffd640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 0x7f7604159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f76037fe640 1 -- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 msgr2=0x7f7604159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f76037fe640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 0x7f7604159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:50.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f76037fe640 1 -- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76041538a0 msgr2=0x7f7604148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:50.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f76037fe640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76041538a0 0x7f7604148a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:50.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f76037fe640 1 -- 172.21.15.148:0/3637215254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec0094d0 con 0x7f760415d6f0 2023-10-30T13:50:50.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f75fa7fc640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76041538a0 0x7f7604148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:50.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f7602ffd640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 0x7f7604159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:50.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f76037fe640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 0x7f76040c6d50 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f75fc007dc0 tx=0x7f75fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:50.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f7600ff9640 1 -- 172.21.15.148:0/3637215254 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc014420 con 0x7f760415d6f0 2023-10-30T13:50:50.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f7600ff9640 1 -- 172.21.15.148:0/3637215254 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75fc0145a0 con 0x7f760415d6f0 2023-10-30T13:50:50.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76040c6950 con 0x7f760415d6f0 2023-10-30T13:50:50.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f7600ff9640 1 -- 172.21.15.148:0/3637215254 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc014950 con 0x7f760415d6f0 2023-10-30T13:50:50.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7604158460 con 0x7f760415d6f0 2023-10-30T13:50:50.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76040b2a50 con 0x7f760415d6f0 2023-10-30T13:50:50.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.957+0000 7f7600ff9640 1 -- 172.21.15.148:0/3637215254 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75fc014ab0 con 0x7f760415d6f0 2023-10-30T13:50:50.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.961+0000 7f7600ff9640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75e40664a0 0x7f75e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:50.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.961+0000 7f7600ff9640 1 -- 172.21.15.148:0/3637215254 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7f75fc084fa0 con 0x7f760415d6f0 2023-10-30T13:50:50.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.961+0000 7f7602ffd640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75e40664a0 0x7f75e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:50.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.961+0000 7f7602ffd640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75e40664a0 0x7f75e4068960 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f75f4000f70 tx=0x7f75f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:50.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:50.965+0000 7f7600ff9640 1 -- 172.21.15.148:0/3637215254 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f75fc051c30 con 0x7f760415d6f0 2023-10-30T13:50:51.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f760404e170 con 0x7f760415d6f0 2023-10-30T13:50:51.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f7600ff9640 1 -- 172.21.15.148:0/3637215254 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f75fc055250 con 0x7f760415d6f0 2023-10-30T13:50:51.146 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:51.146 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8145,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:59.158472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:51.146 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 158 2023-10-30T13:50:51.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75e40664a0 msgr2=0x7f75e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:51.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75e40664a0 0x7f75e4068960 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f75f4000f70 tx=0x7f75f4009040 comp rx=0 tx=0).stop 2023-10-30T13:50:51.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 msgr2=0x7f76040c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:51.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 0x7f76040c6d50 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f75fc007dc0 tx=0x7f75fc00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:50:51.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 shutdown_connections 2023-10-30T13:50:51.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f75e40664a0 0x7f75e4068960 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:51.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f760415d6f0 0x7f76040c6d50 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:51.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76041538a0 0x7f7604148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:51.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 --2- 172.21.15.148:0/3637215254 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f76040cf160 0x7f7604159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:51.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 >> 172.21.15.148:0/3637215254 conn(0x7f760406bfe0 msgr2=0x7f76040cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:51.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 shutdown_connections 2023-10-30T13:50:51.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:51.145+0000 7f760990f640 1 -- 172.21.15.148:0/3637215254 wait complete. 2023-10-30T13:50:51.167 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:50:53.168 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:53.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb95209c640 1 Processor -- start 2023-10-30T13:50:53.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb95209c640 1 -- start start 2023-10-30T13:50:53.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb95209c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb94c0c6b80 0x7fb94c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:53.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb95209c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c0c6f60 0x7fb94c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:53.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb95209c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0c0a50 con 0x7fb94c0c8430 2023-10-30T13:50:53.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb95209c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c06f6e0 con 0x7fb94c0c6b80 2023-10-30T13:50:53.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb95209c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c06f9c0 con 0x7fb94c0c6f60 2023-10-30T13:50:53.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb94c0c8430 0x7fb94c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56260/0 (socket says 172.21.15.148:56260) 2023-10-30T13:50:53.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94bfff640 1 -- 172.21.15.148:0/169960644 learned_addr learned my addr 172.21.15.148:0/169960644 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:50:53.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94b7fe640 1 --2- 172.21.15.148:0/169960644 >> v2:172.21.15.150:3300/0 conn(0x7fb94c0c6b80 0x7fb94c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:53.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94affd640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c0c6f60 0x7fb94c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:53.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415736785 0 0) 0x7fb94c0c0a50 con 0x7fb94c0c8430 2023-10-30T13:50:53.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb934003530 con 0x7fb94c0c8430 2023-10-30T13:50:53.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 921018571 0 0) 0x7fb934003530 con 0x7fb94c0c8430 2023-10-30T13:50:53.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c0c6f60 msgr2=0x7fb94c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:53.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c0c6f60 0x7fb94c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 >> v2:172.21.15.150:3300/0 conn(0x7fb94c0c6b80 msgr2=0x7fb94c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:53.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 --2- 172.21.15.148:0/169960644 >> v2:172.21.15.150:3300/0 conn(0x7fb94c0c6b80 0x7fb94c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c06fca0 con 0x7fb94c0c8430 2023-10-30T13:50:53.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fb944001a00 con 0x7fb94c0c8430 2023-10-30T13:50:53.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 >> v1:172.21.15.148:6789/0 conn(0x7fb94c0c8430 legacy=0x7fb94c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:53.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c0c6f60 0x7fb934003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:53.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934003f00 0x7fb9340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:53.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb934004d10 0x7fb934005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:53.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c06f6e0 con 0x7fb934004d10 2023-10-30T13:50:53.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c06f9c0 con 0x7fb934003f00 2023-10-30T13:50:53.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.317+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93400b9e0 con 0x7fb94c0c6f60 2023-10-30T13:50:53.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94bfff640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb934004d10 0x7fb934005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:53.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94affd640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934003f00 0x7fb9340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:53.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94b7fe640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c0c6f60 0x7fb934003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:53.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94bfff640 1 -- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c0c6f60 msgr2=0x7fb934003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:53.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94bfff640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c0c6f60 0x7fb934003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94bfff640 1 -- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934003f00 msgr2=0x7fb9340047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:53.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94bfff640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934003f00 0x7fb9340047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94bfff640 1 -- 172.21.15.148:0/169960644 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb934003530 con 0x7fb934004d10 2023-10-30T13:50:53.327 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94affd640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934003f00 0x7fb9340047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:53.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94bfff640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb934004d10 0x7fb934005190 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fb94400a890 tx=0x7fb94401fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77422125503c828 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:53.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9440aa6f0 con 0x7fb934004d10 2023-10-30T13:50:53.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9440aa850 con 0x7fb934004d10 2023-10-30T13:50:53.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.321+0000 7fb94a7fc640 1 -- 172.21.15.148:0/169960644 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9440aac60 con 0x7fb934004d10 2023-10-30T13:50:53.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 -- 172.21.15.148:0/169960644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb934004d10 msgr2=0x7fb934005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:53.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 --2- 172.21.15.148:0/169960644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb934004d10 0x7fb934005190 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fb94400a890 tx=0x7fb94401fac0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94bfff640 1 -- 172.21.15.148:0/169960644 reap_dead start 2023-10-30T13:50:53.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 -- 172.21.15.148:0/169960644 shutdown_connections 2023-10-30T13:50:53.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 -- 172.21.15.148:0/169960644 >> 172.21.15.148:0/169960644 conn(0x7fb94c0c7910 msgr2=0x7fb94c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:53.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 -- 172.21.15.148:0/169960644 shutdown_connections 2023-10-30T13:50:53.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 -- 172.21.15.148:0/169960644 wait complete. 2023-10-30T13:50:53.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 Processor -- start 2023-10-30T13:50:53.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 -- start start 2023-10-30T13:50:53.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934004d10 0x7fb94c137530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:53.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c13c750 0x7fb94c1ee530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:53.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb94c1eea70 0x7fb94c1e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:53.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c1e3760 con 0x7fb94c1eea70 2023-10-30T13:50:53.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c153d60 con 0x7fb934004d10 2023-10-30T13:50:53.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb95209c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c1f2a00 con 0x7fb94c13c750 2023-10-30T13:50:53.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb94c1eea70 0x7fb94c1e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:53.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb94c1eea70 0x7fb94c1e0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47276/0 (socket says 172.21.15.148:47276) 2023-10-30T13:50:53.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94bfff640 1 -- 172.21.15.148:0/1713908301 learned_addr learned my addr 172.21.15.148:0/1713908301 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:53.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94b7fe640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934004d10 0x7fb94c137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:53.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94affd640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c13c750 0x7fb94c1ee530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:53.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94bfff640 1 -- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c13c750 msgr2=0x7fb94c1ee530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:53.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94bfff640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c13c750 0x7fb94c1ee530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94bfff640 1 -- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934004d10 msgr2=0x7fb94c137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:53.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94bfff640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934004d10 0x7fb94c137530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.325+0000 7fb94bfff640 1 -- 172.21.15.148:0/1713908301 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c1e3480 con 0x7fb94c1eea70 2023-10-30T13:50:53.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb94bfff640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb94c1eea70 0x7fb94c1e0aa0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fb94400a830 tx=0x7fb934005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:53.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb933fff640 1 -- 172.21.15.148:0/1713908301 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9440aa6f0 con 0x7fb94c1eea70 2023-10-30T13:50:53.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb933fff640 1 -- 172.21.15.148:0/1713908301 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb944001fe0 con 0x7fb94c1eea70 2023-10-30T13:50:53.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb933fff640 1 -- 172.21.15.148:0/1713908301 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9440019c0 con 0x7fb94c1eea70 2023-10-30T13:50:53.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c1f1c40 con 0x7fb94c1eea70 2023-10-30T13:50:53.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c1f2200 con 0x7fb94c1eea70 2023-10-30T13:50:53.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c04e170 con 0x7fb94c1eea70 2023-10-30T13:50:53.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb933fff640 1 -- 172.21.15.148:0/1713908301 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9440be480 con 0x7fb94c1eea70 2023-10-30T13:50:53.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb933fff640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9380664a0 0x7fb938068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:53.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb933fff640 1 -- 172.21.15.148:0/1713908301 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7574+0+0 (secure 0 0 0) 0x7fb9440b9070 con 0x7fb94c1eea70 2023-10-30T13:50:53.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.329+0000 7fb94b7fe640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9380664a0 0x7fb938068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:53.337 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.333+0000 7fb94b7fe640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9380664a0 0x7fb938068960 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009ac0 tx=0x7fb93c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:53.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.333+0000 7fb933fff640 1 -- 172.21.15.148:0/1713908301 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9440eeec0 con 0x7fb94c1eea70 2023-10-30T13:50:53.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.513+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb94c1e4970 con 0x7fb94c1eea70 2023-10-30T13:50:53.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.513+0000 7fb933fff640 1 -- 172.21.15.148:0/1713908301 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+6534 (secure 0 0 0) 0x7fb9440f24e0 con 0x7fb94c1eea70 2023-10-30T13:50:53.515 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:53.515 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8145,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:49:59.158472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7812,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_7812":{"gid":7812,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":38,"addr":"172.21.15.148:6837/1530445724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":1530445724},{"type":"v1","addr":"172.21.15.148:6837","nonce":1530445724}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:53.516 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 158 2023-10-30T13:50:53.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9380664a0 msgr2=0x7fb938068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:53.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9380664a0 0x7fb938068960 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009ac0 tx=0x7fb93c008040 comp rx=0 tx=0).stop 2023-10-30T13:50:53.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb94c1eea70 msgr2=0x7fb94c1e0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:53.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb94c1eea70 0x7fb94c1e0aa0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fb94400a830 tx=0x7fb934005800 comp rx=0 tx=0).stop 2023-10-30T13:50:53.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 shutdown_connections 2023-10-30T13:50:53.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb9380664a0 0x7fb938068960 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb94c1eea70 0x7fb94c1e0aa0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb94c13c750 0x7fb94c1ee530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 --2- 172.21.15.148:0/1713908301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb934004d10 0x7fb94c137530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:53.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 >> 172.21.15.148:0/1713908301 conn(0x7fb94c0c7910 msgr2=0x7fb94c1386a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:53.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 shutdown_connections 2023-10-30T13:50:53.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:53.517+0000 7fb95209c640 1 -- 172.21.15.148:0/1713908301 wait complete. 2023-10-30T13:50:53.537 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:50:55.538 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:55.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb15cd3e640 1 Processor -- start 2023-10-30T13:50:55.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb15cd3e640 1 -- start start 2023-10-30T13:50:55.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb15cd3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1580c05d0 0x7fb15813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:55.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb15cd3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb1580c6f60 0x7fb15813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:55.688 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb15cd3e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1580c0a50 con 0x7fb1580c6b80 2023-10-30T13:50:55.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb15cd3e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15806f6e0 con 0x7fb1580c6f60 2023-10-30T13:50:55.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb15cd3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15806f9c0 con 0x7fb1580c05d0 2023-10-30T13:50:55.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb15659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb1580c6b80 0x7fb1580bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56272/0 (socket says 172.21.15.148:56272) 2023-10-30T13:50:55.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb15659c640 1 -- 172.21.15.148:0/3418743238 learned_addr learned my addr 172.21.15.148:0/3418743238 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:50:55.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb156d9d640 1 --2- 172.21.15.148:0/3418743238 >> v2:172.21.15.150:3300/0 conn(0x7fb1580c6f60 0x7fb15813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:55.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.685+0000 7fb14dd9b640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1580c05d0 0x7fb15813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:55.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235396789 0 0) 0x7fb1580c0a50 con 0x7fb1580c6b80 2023-10-30T13:50:55.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb144003530 con 0x7fb1580c6b80 2023-10-30T13:50:55.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698616148 0 0) 0x7fb144003530 con 0x7fb1580c6b80 2023-10-30T13:50:55.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1580c05d0 msgr2=0x7fb15813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:55.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1580c05d0 0x7fb15813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 >> v2:172.21.15.150:3300/0 conn(0x7fb1580c6f60 msgr2=0x7fb15813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:55.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 --2- 172.21.15.148:0/3418743238 >> v2:172.21.15.150:3300/0 conn(0x7fb1580c6f60 0x7fb15813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15806fca0 con 0x7fb1580c6b80 2023-10-30T13:50:55.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fb140001680 con 0x7fb1580c6b80 2023-10-30T13:50:55.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 >> v1:172.21.15.148:6789/0 conn(0x7fb1580c6b80 legacy=0x7fb1580bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:55.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb144003980 0x7fb144003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:55.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1440047d0 0x7fb144004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:55.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1580c05d0 0x7fb144005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:55.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15806f6e0 con 0x7fb1440047d0 2023-10-30T13:50:55.694 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15806f9c0 con 0x7fb144003980 2023-10-30T13:50:55.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14400b990 con 0x7fb1580c05d0 2023-10-30T13:50:55.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb14dd9b640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1440047d0 0x7fb144004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:55.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb15659c640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb144003980 0x7fb144003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:55.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb156d9d640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1580c05d0 0x7fb144005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:55.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb14dd9b640 1 -- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1580c05d0 msgr2=0x7fb144005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:55.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb14dd9b640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1580c05d0 0x7fb144005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb14dd9b640 1 -- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb144003980 msgr2=0x7fb144003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:55.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb14dd9b640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb144003980 0x7fb144003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb14dd9b640 1 -- 172.21.15.148:0/3418743238 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144003530 con 0x7fb1440047d0 2023-10-30T13:50:55.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb15659c640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb144003980 0x7fb144003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:55.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.689+0000 7fb14dd9b640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1440047d0 0x7fb144004c50 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb148009ae0 tx=0x7fb14801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3831ac35aab272 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:55.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1480a5560 con 0x7fb1440047d0 2023-10-30T13:50:55.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1480a56c0 con 0x7fb1440047d0 2023-10-30T13:50:55.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb155d9b640 1 -- 172.21.15.148:0/3418743238 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1480a5a70 con 0x7fb1440047d0 2023-10-30T13:50:55.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 -- 172.21.15.148:0/3418743238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1440047d0 msgr2=0x7fb144004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:55.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 --2- 172.21.15.148:0/3418743238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1440047d0 0x7fb144004c50 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb148009ae0 tx=0x7fb14801ca50 comp rx=0 tx=0).stop 2023-10-30T13:50:55.700 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb156d9d640 1 -- 172.21.15.148:0/3418743238 reap_dead start 2023-10-30T13:50:55.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 -- 172.21.15.148:0/3418743238 shutdown_connections 2023-10-30T13:50:55.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 -- 172.21.15.148:0/3418743238 >> 172.21.15.148:0/3418743238 conn(0x7fb1580c7910 msgr2=0x7fb1580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:55.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 -- 172.21.15.148:0/3418743238 shutdown_connections 2023-10-30T13:50:55.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 -- 172.21.15.148:0/3418743238 wait complete. 2023-10-30T13:50:55.701 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 Processor -- start 2023-10-30T13:50:55.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 -- start start 2023-10-30T13:50:55.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1440047d0 0x7fb158156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:55.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb15813d900 0x7fb1580c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:55.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1580c50c0 0x7fb158137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:55.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1581547b0 con 0x7fb1580c50c0 2023-10-30T13:50:55.702 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1581572a0 con 0x7fb15813d900 2023-10-30T13:50:55.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.693+0000 7fb15cd3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15814f280 con 0x7fb1440047d0 2023-10-30T13:50:55.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb156d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1580c50c0 0x7fb158137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:55.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb156d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1580c50c0 0x7fb158137bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47292/0 (socket says 172.21.15.148:47292) 2023-10-30T13:50:55.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb156d9d640 1 -- 172.21.15.148:0/2154187607 learned_addr learned my addr 172.21.15.148:0/2154187607 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:55.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb14dd9b640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb15813d900 0x7fb1580c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:55.703 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb15659c640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1440047d0 0x7fb158156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:55.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb156d9d640 1 -- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1440047d0 msgr2=0x7fb158156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:55.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb156d9d640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1440047d0 0x7fb158156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb156d9d640 1 -- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb15813d900 msgr2=0x7fb1580c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:55.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb156d9d640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb15813d900 0x7fb1580c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.704 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb156d9d640 1 -- 172.21.15.148:0/2154187607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15813d0f0 con 0x7fb1580c50c0 2023-10-30T13:50:55.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb15659c640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1440047d0 0x7fb158156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:55.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb156d9d640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1580c50c0 0x7fb158137bf0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fb1580c8980 tx=0x7fb1500115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:55.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb14f7fe640 1 -- 172.21.15.148:0/2154187607 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15000e070 con 0x7fb1580c50c0 2023-10-30T13:50:55.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb14f7fe640 1 -- 172.21.15.148:0/2154187607 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb150011e10 con 0x7fb1580c50c0 2023-10-30T13:50:55.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb14f7fe640 1 -- 172.21.15.148:0/2154187607 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500039a0 con 0x7fb1580c50c0 2023-10-30T13:50:55.705 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1581389b0 con 0x7fb1580c50c0 2023-10-30T13:50:55.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15813d620 con 0x7fb1580c50c0 2023-10-30T13:50:55.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1581548f0 con 0x7fb1580c50c0 2023-10-30T13:50:55.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.697+0000 7fb14f7fe640 1 -- 172.21.15.148:0/2154187607 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb150003b00 con 0x7fb1580c50c0 2023-10-30T13:50:55.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.701+0000 7fb14f7fe640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb1380664a0 0x7fb138068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:55.706 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.701+0000 7fb14f7fe640 1 -- 172.21.15.148:0/2154187607 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fb150084c00 con 0x7fb1580c50c0 2023-10-30T13:50:55.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.701+0000 7fb15659c640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb1380664a0 0x7fb138068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:55.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.701+0000 7fb15659c640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb1380664a0 0x7fb138068960 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb140005320 tx=0x7fb140009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:55.707 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.705+0000 7fb14f7fe640 1 -- 172.21.15.148:0/2154187607 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb1581548f0 con 0x7fb1580c50c0 2023-10-30T13:50:55.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1581548f0 con 0x7fb1580c50c0 2023-10-30T13:50:55.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb14f7fe640 1 -- 172.21.15.148:0/2154187607 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fb1581548f0 con 0x7fb1580c50c0 2023-10-30T13:50:55.885 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:55.885 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:50:55.057633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:55.886 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 159 2023-10-30T13:50:55.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb1380664a0 msgr2=0x7fb138068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:55.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb1380664a0 0x7fb138068960 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb140005320 tx=0x7fb140009040 comp rx=0 tx=0).stop 2023-10-30T13:50:55.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1580c50c0 msgr2=0x7fb158137bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:55.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1580c50c0 0x7fb158137bf0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fb1580c8980 tx=0x7fb1500115a0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 shutdown_connections 2023-10-30T13:50:55.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb1380664a0 0x7fb138068960 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1580c50c0 0x7fb158137bf0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb15813d900 0x7fb1580c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 --2- 172.21.15.148:0/2154187607 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb1440047d0 0x7fb158156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:55.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 >> 172.21.15.148:0/2154187607 conn(0x7fb1580c7910 msgr2=0x7fb1580c5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:55.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 shutdown_connections 2023-10-30T13:50:55.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:55.885+0000 7fb15cd3e640 1 -- 172.21.15.148:0/2154187607 wait complete. 2023-10-30T13:50:55.907 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:50:55.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:50:55.907 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb92631d640 1 Processor -- start 2023-10-30T13:50:56.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb92631d640 1 -- start start 2023-10-30T13:50:56.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb92631d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb92006f7a0 0x7fb92006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:56.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb92631d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb9200c6bd0 0x7fb920069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:56.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb92631d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 0x7fb920158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:56.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb92631d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9200700c0 con 0x7fb9200c6bd0 2023-10-30T13:50:56.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb92631d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92006a3d0 con 0x7fb92006f7a0 2023-10-30T13:50:56.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb92631d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb920066cf0 con 0x7fb920071110 2023-10-30T13:50:56.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb91f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb9200c6bd0 0x7fb920069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:56.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb91f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb9200c6bd0 0x7fb920069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47308/0 (socket says 172.21.15.148:47308) 2023-10-30T13:50:56.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb91f7fe640 1 -- 172.21.15.148:0/757357373 learned_addr learned my addr 172.21.15.148:0/757357373 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:56.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb91ffff640 1 --2- 172.21.15.148:0/757357373 >> v2:172.21.15.150:3300/0 conn(0x7fb92006f7a0 0x7fb92006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:56.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb9248ba640 1 --2- 172.21.15.148:0/757357373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 0x7fb920158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:56.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb91f7fe640 1 -- 172.21.15.148:0/757357373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 msgr2=0x7fb920158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:56.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb91f7fe640 1 --2- 172.21.15.148:0/757357373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 0x7fb920158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb91f7fe640 1 -- 172.21.15.148:0/757357373 >> v2:172.21.15.150:3300/0 conn(0x7fb92006f7a0 msgr2=0x7fb92006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:56.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb91f7fe640 1 --2- 172.21.15.148:0/757357373 >> v2:172.21.15.150:3300/0 conn(0x7fb92006f7a0 0x7fb92006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.045+0000 7fb91f7fe640 1 -- 172.21.15.148:0/757357373 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9200c70d0 con 0x7fb9200c6bd0 2023-10-30T13:50:56.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb91f7fe640 1 --2- 172.21.15.148:0/757357373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9200c6bd0 0x7fb920069e90 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fb92006c830 tx=0x7fb91401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b2d8f840ba9ab0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:56.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb91effd640 1 -- 172.21.15.148:0/757357373 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9140a89b0 con 0x7fb9200c6bd0 2023-10-30T13:50:56.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb91effd640 1 -- 172.21.15.148:0/757357373 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9140a8b10 con 0x7fb9200c6bd0 2023-10-30T13:50:56.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb91effd640 1 -- 172.21.15.148:0/757357373 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9140a8ea0 con 0x7fb9200c6bd0 2023-10-30T13:50:56.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb92631d640 1 -- 172.21.15.148:0/757357373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9200c6bd0 msgr2=0x7fb920069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:56.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb92631d640 1 --2- 172.21.15.148:0/757357373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9200c6bd0 0x7fb920069e90 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fb92006c830 tx=0x7fb91401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb92631d640 1 -- 172.21.15.148:0/757357373 shutdown_connections 2023-10-30T13:50:56.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb92631d640 1 --2- 172.21.15.148:0/757357373 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 0x7fb920158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb92631d640 1 --2- 172.21.15.148:0/757357373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9200c6bd0 0x7fb920069e90 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb92631d640 1 --2- 172.21.15.148:0/757357373 >> v2:172.21.15.150:3300/0 conn(0x7fb92006f7a0 0x7fb92006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb92631d640 1 -- 172.21.15.148:0/757357373 >> 172.21.15.148:0/757357373 conn(0x7fb9200c7910 msgr2=0x7fb9200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:56.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb92631d640 1 -- 172.21.15.148:0/757357373 shutdown_connections 2023-10-30T13:50:56.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.049+0000 7fb92631d640 1 -- 172.21.15.148:0/757357373 wait complete. 2023-10-30T13:50:56.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 Processor -- start 2023-10-30T13:50:56.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 -- start start 2023-10-30T13:50:56.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92006f7a0 0x7fb9201f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:56.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 0x7fb9201ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:56.057 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9200c6bd0 0x7fb920156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:56.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9201f1d50 con 0x7fb92006f7a0 2023-10-30T13:50:56.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9201f2ce0 con 0x7fb9200c6bd0 2023-10-30T13:50:56.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9201eefc0 con 0x7fb920071110 2023-10-30T13:50:56.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92006f7a0 0x7fb9201f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:56.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92006f7a0 0x7fb9201f27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47324/0 (socket says 172.21.15.148:47324) 2023-10-30T13:50:56.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91ffff640 1 -- 172.21.15.148:0/2714364467 learned_addr learned my addr 172.21.15.148:0/2714364467 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:56.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb9248ba640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9200c6bd0 0x7fb920156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:56.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91f7fe640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 0x7fb9201ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:56.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91ffff640 1 -- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 msgr2=0x7fb9201ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:56.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91ffff640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 0x7fb9201ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91ffff640 1 -- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9200c6bd0 msgr2=0x7fb920156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:56.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91ffff640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9200c6bd0 0x7fb920156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91ffff640 1 -- 172.21.15.148:0/2714364467 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140094d0 con 0x7fb92006f7a0 2023-10-30T13:50:56.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb9248ba640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9200c6bd0 0x7fb920156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:50:56.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91ffff640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92006f7a0 0x7fb9201f27a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d800 tx=0x7fb91000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91cff9640 1 -- 172.21.15.148:0/2714364467 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9100096a0 con 0x7fb92006f7a0 2023-10-30T13:50:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91cff9640 1 -- 172.21.15.148:0/2714364467 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910009800 con 0x7fb92006f7a0 2023-10-30T13:50:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9201ef2a0 con 0x7fb92006f7a0 2023-10-30T13:50:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb91cff9640 1 -- 172.21.15.148:0/2714364467 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910009990 con 0x7fb92006f7a0 2023-10-30T13:50:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9201f1fa0 con 0x7fb92006f7a0 2023-10-30T13:50:56.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.053+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92004e170 con 0x7fb92006f7a0 2023-10-30T13:50:56.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.057+0000 7fb91cff9640 1 -- 172.21.15.148:0/2714364467 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb910009af0 con 0x7fb92006f7a0 2023-10-30T13:50:56.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.057+0000 7fb91cff9640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb90c0664a0 0x7fb90c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:56.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.057+0000 7fb91cff9640 1 -- 172.21.15.148:0/2714364467 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fb910084ba0 con 0x7fb92006f7a0 2023-10-30T13:50:56.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.057+0000 7fb91f7fe640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb90c0664a0 0x7fb90c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:56.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.057+0000 7fb91f7fe640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb90c0664a0 0x7fb90c068960 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb9140b2000 tx=0x7fb9140aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:56.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.061+0000 7fb91cff9640 1 -- 172.21.15.148:0/2714364467 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9100517d0 con 0x7fb92006f7a0 2023-10-30T13:50:56.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.245+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9201eece0 con 0x7fb92006f7a0 2023-10-30T13:50:56.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.245+0000 7fb91cff9640 1 -- 172.21.15.148:0/2714364467 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fb910054df0 con 0x7fb92006f7a0 2023-10-30T13:50:56.247 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:56.247 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:50:55.057633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:56.248 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 159 2023-10-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb90c0664a0 msgr2=0x7fb90c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb90c0664a0 0x7fb90c068960 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb9140b2000 tx=0x7fb9140aa070 comp rx=0 tx=0).stop 2023-10-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92006f7a0 msgr2=0x7fb9201f27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92006f7a0 0x7fb9201f27a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d800 tx=0x7fb91000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 shutdown_connections 2023-10-30T13:50:56.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb90c0664a0 0x7fb90c068960 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9200c6bd0 0x7fb920156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb920071110 0x7fb9201ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 --2- 172.21.15.148:0/2714364467 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92006f7a0 0x7fb9201f27a0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:56.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 >> 172.21.15.148:0/2714364467 conn(0x7fb9200c7910 msgr2=0x7fb9200c3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:56.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 shutdown_connections 2023-10-30T13:50:56.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:56.249+0000 7fb92631d640 1 -- 172.21.15.148:0/2714364467 wait complete. 2023-10-30T13:50:56.269 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 159, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 159, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8041": { "addr": "172.21.15.150:6836/1947781173", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1947781173, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 1947781173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8041, "incarnation": 152, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_8145": { "addr": "172.21.15.148:6835/1526234342", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1526234342, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1526234342, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8145, "incarnation": 159, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:50:55.057633+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8145, "mds_1": 8041 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:50:58.270 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:50:58.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff50a4ea640 1 Processor -- start 2023-10-30T13:50:58.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff50a4ea640 1 -- start start 2023-10-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff50a4ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5041538a0 0x7ff50415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff50a4ea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff5040ca0e0 0x7ff50415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:58.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff50a4ea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5040cb800 con 0x7ff5040cf160 2023-10-30T13:50:58.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff50a4ea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50415dc30 con 0x7ff5040ca0e0 2023-10-30T13:50:58.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff50a4ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040cb330 con 0x7ff5041538a0 2023-10-30T13:50:58.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff508a87640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff5040cf160 0x7ff5040cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56278/0 (socket says 172.21.15.148:56278) 2023-10-30T13:50:58.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff508a87640 1 -- 172.21.15.148:0/283552333 learned_addr learned my addr 172.21.15.148:0/283552333 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff5037fe640 1 --2- 172.21.15.148:0/283552333 >> v2:172.21.15.150:3300/0 conn(0x7ff5040ca0e0 0x7ff50415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff503fff640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5041538a0 0x7ff50415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590056693 0 0) 0x7ff5040cb800 con 0x7ff5040cf160 2023-10-30T13:50:58.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ec003530 con 0x7ff5040cf160 2023-10-30T13:50:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074667374 0 0) 0x7ff4ec003530 con 0x7ff5040cf160 2023-10-30T13:50:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5041538a0 msgr2=0x7ff50415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5041538a0 0x7ff50415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 >> v2:172.21.15.150:3300/0 conn(0x7ff5040ca0e0 msgr2=0x7ff50415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:58.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 --2- 172.21.15.148:0/283552333 >> v2:172.21.15.150:3300/0 conn(0x7ff5040ca0e0 0x7ff50415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5040cad70 con 0x7ff5040cf160 2023-10-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7ff4fc001a00 con 0x7ff5040cf160 2023-10-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff5037fe640 1 --2- 172.21.15.148:0/283552333 >> v2:172.21.15.150:3300/0 conn(0x7ff5040ca0e0 0x7ff50415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:50:58.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 >> v1:172.21.15.148:6789/0 conn(0x7ff5040cf160 legacy=0x7ff5040cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5041538a0 0x7ff4ec003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec003f00 0x7ff4ec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec004d10 0x7ff4ec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:58.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50415dc30 con 0x7ff4ec004d10 2023-10-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040cb330 con 0x7ff4ec003f00 2023-10-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec00b9c0 con 0x7ff5041538a0 2023-10-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff508a87640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec004d10 0x7ff4ec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:58.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff503fff640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5041538a0 0x7ff4ec003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff5037fe640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec003f00 0x7ff4ec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff508a87640 1 -- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5041538a0 msgr2=0x7ff4ec003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff508a87640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff5041538a0 0x7ff4ec003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff508a87640 1 -- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec003f00 msgr2=0x7ff4ec0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:50:58.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff508a87640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4ec003f00 0x7ff4ec0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.421+0000 7ff508a87640 1 -- 172.21.15.148:0/283552333 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec003530 con 0x7ff4ec004d10 2023-10-30T13:50:58.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff508a87640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec004d10 0x7ff4ec005190 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00a860 tx=0x7ff4fc01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a2df7eef77de5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:58.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0ad630 con 0x7ff4ec004d10 2023-10-30T13:50:58.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc0a8070 con 0x7ff4ec004d10 2023-10-30T13:50:58.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff502ffd640 1 -- 172.21.15.148:0/283552333 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0ada30 con 0x7ff4ec004d10 2023-10-30T13:50:58.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff50a4ea640 1 -- 172.21.15.148:0/283552333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec004d10 msgr2=0x7ff4ec005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff50a4ea640 1 --2- 172.21.15.148:0/283552333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff4ec004d10 0x7ff4ec005190 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00a860 tx=0x7ff4fc01d360 comp rx=0 tx=0).stop 2023-10-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff508a87640 1 -- 172.21.15.148:0/283552333 reap_dead start 2023-10-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff50a4ea640 1 -- 172.21.15.148:0/283552333 shutdown_connections 2023-10-30T13:50:58.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff50a4ea640 1 -- 172.21.15.148:0/283552333 >> 172.21.15.148:0/283552333 conn(0x7ff50406bfe0 msgr2=0x7ff50406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff50a4ea640 1 -- 172.21.15.148:0/283552333 shutdown_connections 2023-10-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.425+0000 7ff50a4ea640 1 -- 172.21.15.148:0/283552333 wait complete. 2023-10-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 Processor -- start 2023-10-30T13:50:58.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 -- start start 2023-10-30T13:50:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec004d10 0x7ff5040b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff50415d630 0x7ff504145960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504145ea0 0x7ff5040c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:58.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040c6c30 con 0x7ff504145ea0 2023-10-30T13:50:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040c7080 con 0x7ff50415d630 2023-10-30T13:50:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50415cb40 con 0x7ff4ec004d10 2023-10-30T13:50:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff508a87640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504145ea0 0x7ff5040c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:58.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff503fff640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec004d10 0x7ff5040b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff508a87640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504145ea0 0x7ff5040c7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47336/0 (socket says 172.21.15.148:47336) 2023-10-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff5037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff50415d630 0x7ff504145960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff508a87640 1 -- 172.21.15.148:0/2389610647 learned_addr learned my addr 172.21.15.148:0/2389610647 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff508a87640 1 -- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec004d10 msgr2=0x7ff5040b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:58.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff508a87640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec004d10 0x7ff5040b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff508a87640 1 -- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff50415d630 msgr2=0x7ff504145960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:58.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff508a87640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff50415d630 0x7ff504145960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff508a87640 1 -- 172.21.15.148:0/2389610647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5040c6670 con 0x7ff504145ea0 2023-10-30T13:50:58.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff508a87640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504145ea0 0x7ff5040c7510 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00a860 tx=0x7ff4ec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff500ff9640 1 -- 172.21.15.148:0/2389610647 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0017c0 con 0x7ff504145ea0 2023-10-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff500ff9640 1 -- 172.21.15.148:0/2389610647 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc001920 con 0x7ff504145ea0 2023-10-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5040c6950 con 0x7ff504145ea0 2023-10-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff500ff9640 1 -- 172.21.15.148:0/2389610647 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc004d70 con 0x7ff504145ea0 2023-10-30T13:50:58.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.429+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff504146320 con 0x7ff504145ea0 2023-10-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.433+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5040c6e70 con 0x7ff504145ea0 2023-10-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.433+0000 7ff500ff9640 1 -- 172.21.15.148:0/2389610647 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4fc0aa030 con 0x7ff504145ea0 2023-10-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.433+0000 7ff500ff9640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4f00664d0 0x7ff4f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.433+0000 7ff500ff9640 1 -- 172.21.15.148:0/2389610647 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7ff4fc11dd20 con 0x7ff504145ea0 2023-10-30T13:50:58.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.433+0000 7ff503fff640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4f00664d0 0x7ff4f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:50:58.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.433+0000 7ff503fff640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4f00664d0 0x7ff4f0068990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4009be0 tx=0x7ff4f4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:50:58.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.437+0000 7ff500ff9640 1 -- 172.21.15.148:0/2389610647 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4fc0eaaa0 con 0x7ff504145ea0 2023-10-30T13:50:58.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.621+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5040c6e70 con 0x7ff504145ea0 2023-10-30T13:50:58.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.621+0000 7ff500ff9640 1 -- 172.21.15.148:0/2389610647 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5737 (secure 0 0 0) 0x7ff4fc0ee0c0 con 0x7ff504145ea0 2023-10-30T13:50:58.623 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:50:58.623 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:50:55.057633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 159 2023-10-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4f00664d0 msgr2=0x7ff4f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4f00664d0 0x7ff4f0068990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4009be0 tx=0x7ff4f4009190 comp rx=0 tx=0).stop 2023-10-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504145ea0 msgr2=0x7ff5040c7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:50:58.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504145ea0 0x7ff5040c7510 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00a860 tx=0x7ff4ec005800 comp rx=0 tx=0).stop 2023-10-30T13:50:58.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 shutdown_connections 2023-10-30T13:50:58.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ff4f00664d0 0x7ff4f0068990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504145ea0 0x7ff5040c7510 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff50415d630 0x7ff504145960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 --2- 172.21.15.148:0/2389610647 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ff4ec004d10 0x7ff5040b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:50:58.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 >> 172.21.15.148:0/2389610647 conn(0x7ff50406bfe0 msgr2=0x7ff504152450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:50:58.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 shutdown_connections 2023-10-30T13:50:58.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:50:58.625+0000 7ff50a4ea640 1 -- 172.21.15.148:0/2389610647 wait complete. 2023-10-30T13:50:58.644 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:00.646 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:00.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.793+0000 7fbcf7d65640 1 Processor -- start 2023-10-30T13:51:00.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.793+0000 7fbcf7d65640 1 -- start start 2023-10-30T13:51:00.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf7d65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbcf00c6f60 0x7fbcf01566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:00.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf7d65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf00c0a50 con 0x7fbcf00c6b80 2023-10-30T13:51:00.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf7d65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf006f6e0 con 0x7fbcf00c05d0 2023-10-30T13:51:00.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf7d65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf006f9c0 con 0x7fbcf00c6f60 2023-10-30T13:51:00.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf5b01640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbcf00c6b80 0x7fbcf00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:56292/0 (socket says 172.21.15.148:56292) 2023-10-30T13:51:00.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf5b01640 1 -- 172.21.15.148:0/3676747334 learned_addr learned my addr 172.21.15.148:0/3676747334 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:51:00.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf6302640 1 --2- 172.21.15.148:0/3676747334 >> v2:172.21.15.150:3300/0 conn(0x7fbcf00c6f60 0x7fbcf01566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:00.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477265202 0 0) 0x7fbcf00c0a50 con 0x7fbcf00c6b80 2023-10-30T13:51:00.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcdc005740 con 0x7fbcf00c6b80 2023-10-30T13:51:00.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1043795652 0 0) 0x7fbcdc005740 con 0x7fbcf00c6b80 2023-10-30T13:51:00.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 >> v2:172.21.15.150:3300/0 conn(0x7fbcf00c6f60 msgr2=0x7fbcf01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:00.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 --2- 172.21.15.148:0/3676747334 >> v2:172.21.15.150:3300/0 conn(0x7fbcf00c6f60 0x7fbcf01566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:00.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 >> v1:172.21.15.150:6789/0 conn(0x7fbcf00c05d0 legacy=0x7fbcf013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:00.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf006fca0 con 0x7fbcf00c6b80 2023-10-30T13:51:00.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fbcec001680 con 0x7fbcf00c6b80 2023-10-30T13:51:00.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 >> v1:172.21.15.148:6789/0 conn(0x7fbcf00c6b80 legacy=0x7fbcf00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:00.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc005b90 0x7fbcdc005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:00.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0069e0 0x7fbcdc006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:00.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcdc00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:00.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf006f9c0 con 0x7fbcdc0073a0 2023-10-30T13:51:00.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc00e240 con 0x7fbcdc0069e0 2023-10-30T13:51:00.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc00e400 con 0x7fbcdc005b90 2023-10-30T13:51:00.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf6302640 1 --2- 172.21.15.148:0/3676747334 >> v2:172.21.15.150:3300/0 conn(0x7fbcf00c6f60 0x7fbcf01566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:51:00.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf6302640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcdc00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:00.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf5300640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0069e0 0x7fbcdc006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:00.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf5b01640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc005b90 0x7fbcdc005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:00.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf6302640 1 -- 172.21.15.148:0/3676747334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc005b90 msgr2=0x7fbcdc005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:00.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf6302640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcdc005b90 0x7fbcdc005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:00.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf6302640 1 -- 172.21.15.148:0/3676747334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0069e0 msgr2=0x7fbcdc006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:00.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf6302640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0069e0 0x7fbcdc006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:00.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.797+0000 7fbcf6302640 1 -- 172.21.15.148:0/3676747334 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc005740 con 0x7fbcdc0073a0 2023-10-30T13:51:00.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf6302640 1 -- 172.21.15.148:0/3676747334 reap_dead start 2023-10-30T13:51:00.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf6302640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcdc00dba0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fbce800e350 tx=0x7fbce801ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e821870441ebe18 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:00.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce80a5e50 con 0x7fbcdc0073a0 2023-10-30T13:51:00.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce800f040 con 0x7fbcdc0073a0 2023-10-30T13:51:00.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf4aff640 1 -- 172.21.15.148:0/3676747334 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce8003bc0 con 0x7fbcdc0073a0 2023-10-30T13:51:00.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3676747334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 msgr2=0x7fbcdc00dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:00.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf7d65640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcdc00dba0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fbce800e350 tx=0x7fbce801ce40 comp rx=0 tx=0).stop 2023-10-30T13:51:00.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3676747334 shutdown_connections 2023-10-30T13:51:00.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf7d65640 1 --2- 172.21.15.148:0/3676747334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcdc00dba0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:00.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3676747334 >> 172.21.15.148:0/3676747334 conn(0x7fbcf00c7910 msgr2=0x7fbcf00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:00.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3676747334 shutdown_connections 2023-10-30T13:51:00.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.801+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3676747334 wait complete. 2023-10-30T13:51:00.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf7d65640 1 Processor -- start 2023-10-30T13:51:00.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf7d65640 1 -- start start 2023-10-30T13:51:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf7d65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcf01ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf7d65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcf01f1be0 0x7fbcf00c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf7d65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcf0158510 0x7fbcf01589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf7d65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf0158f00 con 0x7fbcdc0073a0 2023-10-30T13:51:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf7d65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf01ef610 con 0x7fbcf01f1be0 2023-10-30T13:51:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf7d65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf01eed70 con 0x7fbcf0158510 2023-10-30T13:51:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5b01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcf01ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5b01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcf01ee670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:47360/0 (socket says 172.21.15.148:47360) 2023-10-30T13:51:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5b01640 1 -- 172.21.15.148:0/3538112877 learned_addr learned my addr 172.21.15.148:0/3538112877 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5300640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcf01f1be0 0x7fbcf00c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf6302640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcf0158510 0x7fbcf01589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:00.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5b01640 1 -- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcf0158510 msgr2=0x7fbcf01589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:00.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5b01640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcf0158510 0x7fbcf01589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:00.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5b01640 1 -- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcf01f1be0 msgr2=0x7fbcf00c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:00.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5b01640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcf01f1be0 0x7fbcf00c0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:00.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5b01640 1 -- 172.21.15.148:0/3538112877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf01ef050 con 0x7fbcdc0073a0 2023-10-30T13:51:00.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf5b01640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcf01ee670 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00e780 tx=0x7fbcec00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:00.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbce67fc640 1 -- 172.21.15.148:0/3538112877 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec00ee90 con 0x7fbcdc0073a0 2023-10-30T13:51:00.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbce67fc640 1 -- 172.21.15.148:0/3538112877 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcec00c8d0 con 0x7fbcdc0073a0 2023-10-30T13:51:00.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbce67fc640 1 -- 172.21.15.148:0/3538112877 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec00cc80 con 0x7fbcdc0073a0 2023-10-30T13:51:00.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.805+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf01ef330 con 0x7fbcdc0073a0 2023-10-30T13:51:00.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.809+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf0157ff0 con 0x7fbcdc0073a0 2023-10-30T13:51:00.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.809+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf004e170 con 0x7fbcdc0073a0 2023-10-30T13:51:00.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.809+0000 7fbce67fc640 1 -- 172.21.15.148:0/3538112877 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbcec01d460 con 0x7fbcdc0073a0 2023-10-30T13:51:00.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.809+0000 7fbce67fc640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcd40664d0 0x7fbcd4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:00.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.813+0000 7fbce67fc640 1 -- 172.21.15.148:0/3538112877 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fbcec0843c0 con 0x7fbcdc0073a0 2023-10-30T13:51:00.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.813+0000 7fbcf5300640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcd40664d0 0x7fbcd4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:00.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.813+0000 7fbcf5300640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcd40664d0 0x7fbcd4068990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbce0001100 tx=0x7fbce0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:00.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:00.817+0000 7fbce67fc640 1 -- 172.21.15.148:0/3538112877 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbcec051150 con 0x7fbcdc0073a0 2023-10-30T13:51:01.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcf00c0240 con 0x7fbcdc0073a0 2023-10-30T13:51:01.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbce67fc640 1 -- 172.21.15.148:0/3538112877 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5737 (secure 0 0 0) 0x7fbcec054770 con 0x7fbcdc0073a0 2023-10-30T13:51:01.010 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:01.010 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:50:55.057633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:01.011 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 159 2023-10-30T13:51:01.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcd40664d0 msgr2=0x7fbcd4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:01.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcd40664d0 0x7fbcd4068990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbce0001100 tx=0x7fbce0009040 comp rx=0 tx=0).stop 2023-10-30T13:51:01.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 msgr2=0x7fbcf01ee670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcf01ee670 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00e780 tx=0x7fbcec00ec40 comp rx=0 tx=0).stop 2023-10-30T13:51:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 shutdown_connections 2023-10-30T13:51:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fbcd40664d0 0x7fbcd4068990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fbcf0158510 0x7fbcf01589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:01.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcf01f1be0 0x7fbcf00c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:01.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 --2- 172.21.15.148:0/3538112877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbcdc0073a0 0x7fbcf01ee670 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:01.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 >> 172.21.15.148:0/3538112877 conn(0x7fbcf00c7910 msgr2=0x7fbcf006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:01.014 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 shutdown_connections 2023-10-30T13:51:01.015 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:01.009+0000 7fbcf7d65640 1 -- 172.21.15.148:0/3538112877 wait complete. 2023-10-30T13:51:01.040 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:03.041 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:03.193 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f4415b01640 1 Processor -- start 2023-10-30T13:51:03.194 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f4415b01640 1 -- start start 2023-10-30T13:51:03.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f4415b01640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f44101538a0 0x7f441015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:03.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f4415b01640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44100cf160 0x7f44100cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:03.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f4415b01640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44100cb800 con 0x7f44100ca0e0 2023-10-30T13:51:03.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f4415b01640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441015dc30 con 0x7f44101538a0 2023-10-30T13:51:03.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f4415b01640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44100cb330 con 0x7f44100cf160 2023-10-30T13:51:03.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f440f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f44101538a0 0x7f441015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:03.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f440f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f44101538a0 0x7f441015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53794/0 (socket says 172.21.15.148:53794) 2023-10-30T13:51:03.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f440f7fe640 1 -- 172.21.15.148:0/3627886766 learned_addr learned my addr 172.21.15.148:0/3627886766 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:03.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f440ffff640 1 --2- 172.21.15.148:0/3627886766 >> v2:172.21.15.150:3300/0 conn(0x7f44100cf160 0x7f44100cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:03.197 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f440f7fe640 1 -- 172.21.15.148:0/3627886766 >> v2:172.21.15.150:3300/0 conn(0x7f44100cf160 msgr2=0x7f44100cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:03.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f440f7fe640 1 --2- 172.21.15.148:0/3627886766 >> v2:172.21.15.150:3300/0 conn(0x7f44100cf160 0x7f44100cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f440f7fe640 1 -- 172.21.15.148:0/3627886766 >> v1:172.21.15.150:6789/0 conn(0x7f44100ca0e0 legacy=0x7f441015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:03.198 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.193+0000 7f440f7fe640 1 -- 172.21.15.148:0/3627886766 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44100cad70 con 0x7f44101538a0 2023-10-30T13:51:03.199 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f440f7fe640 1 --2- 172.21.15.148:0/3627886766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44101538a0 0x7f441015d6f0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f44100b5330 tx=0x7f440001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0a115675cb0637 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:03.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f440effd640 1 -- 172.21.15.148:0/3627886766 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44000a89f0 con 0x7f44101538a0 2023-10-30T13:51:03.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f440effd640 1 -- 172.21.15.148:0/3627886766 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44000a8b50 con 0x7f44101538a0 2023-10-30T13:51:03.200 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f440effd640 1 -- 172.21.15.148:0/3627886766 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44000b0440 con 0x7f44101538a0 2023-10-30T13:51:03.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f4415b01640 1 -- 172.21.15.148:0/3627886766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44101538a0 msgr2=0x7f441015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:03.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f4415b01640 1 --2- 172.21.15.148:0/3627886766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44101538a0 0x7f441015d6f0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f44100b5330 tx=0x7f440001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f4415b01640 1 -- 172.21.15.148:0/3627886766 shutdown_connections 2023-10-30T13:51:03.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f4415b01640 1 --2- 172.21.15.148:0/3627886766 >> v2:172.21.15.150:3300/0 conn(0x7f44100cf160 0x7f44100cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f4415b01640 1 --2- 172.21.15.148:0/3627886766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44101538a0 0x7f441015d6f0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f4415b01640 1 -- 172.21.15.148:0/3627886766 >> 172.21.15.148:0/3627886766 conn(0x7f441006bfe0 msgr2=0x7f441006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:03.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f4415b01640 1 -- 172.21.15.148:0/3627886766 shutdown_connections 2023-10-30T13:51:03.202 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.197+0000 7f4415b01640 1 -- 172.21.15.148:0/3627886766 wait complete. 2023-10-30T13:51:03.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 Processor -- start 2023-10-30T13:51:03.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 -- start start 2023-10-30T13:51:03.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44100ca0e0 0x7f4410158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:03.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44100cf160 0x7f44101e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:03.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44101538a0 0x7f44101e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:03.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441015d9f0 con 0x7f44100ca0e0 2023-10-30T13:51:03.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441015d710 con 0x7f44101538a0 2023-10-30T13:51:03.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44101e3600 con 0x7f44100cf160 2023-10-30T13:51:03.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44100ca0e0 0x7f4410158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:03.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44100ca0e0 0x7f4410158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53808/0 (socket says 172.21.15.148:53808) 2023-10-30T13:51:03.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440f7fe640 1 -- 172.21.15.148:0/2527363161 learned_addr learned my addr 172.21.15.148:0/2527363161 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:03.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440ffff640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44101538a0 0x7f44101e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:03.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4407fff640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44100cf160 0x7f44101e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:03.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440f7fe640 1 -- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44100cf160 msgr2=0x7f44101e40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:03.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440f7fe640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44100cf160 0x7f44101e40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440f7fe640 1 -- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44101538a0 msgr2=0x7f44101e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:03.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440f7fe640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44101538a0 0x7f44101e4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440f7fe640 1 -- 172.21.15.148:0/2527363161 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000094d0 con 0x7f44100ca0e0 2023-10-30T13:51:03.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440f7fe640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44100ca0e0 0x7f4410158810 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f440001b900 tx=0x7f440001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:03.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440cff9640 1 -- 172.21.15.148:0/2527363161 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44000ba600 con 0x7f44100ca0e0 2023-10-30T13:51:03.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440cff9640 1 -- 172.21.15.148:0/2527363161 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44000ba760 con 0x7f44100ca0e0 2023-10-30T13:51:03.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44101e38e0 con 0x7f44100ca0e0 2023-10-30T13:51:03.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f440cff9640 1 -- 172.21.15.148:0/2527363161 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44000bab10 con 0x7f44100ca0e0 2023-10-30T13:51:03.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.201+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44101595d0 con 0x7f44100ca0e0 2023-10-30T13:51:03.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.205+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44101e3f30 con 0x7f44100ca0e0 2023-10-30T13:51:03.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.205+0000 7f440cff9640 1 -- 172.21.15.148:0/2527363161 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f44000bac70 con 0x7f44100ca0e0 2023-10-30T13:51:03.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.205+0000 7f440cff9640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43f00664a0 0x7f43f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:03.211 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.205+0000 7f440cff9640 1 -- 172.21.15.148:0/2527363161 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f4400122c10 con 0x7f44100ca0e0 2023-10-30T13:51:03.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.205+0000 7f4407fff640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43f00664a0 0x7f43f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:03.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.209+0000 7f4407fff640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43f00664a0 0x7f43f0068960 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f44101592c0 tx=0x7f43f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:03.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.209+0000 7f440cff9640 1 -- 172.21.15.148:0/2527363161 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f44000ef850 con 0x7f44100ca0e0 2023-10-30T13:51:03.463 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.461+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44100b2a50 con 0x7f44100ca0e0 2023-10-30T13:51:03.464 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.461+0000 7f440cff9640 1 -- 172.21.15.148:0/2527363161 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f44000f2e70 con 0x7f44100ca0e0 2023-10-30T13:51:03.464 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:03.465 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:50:55.057633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:03.465 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 159 2023-10-30T13:51:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43f00664a0 msgr2=0x7f43f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43f00664a0 0x7f43f0068960 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f44101592c0 tx=0x7f43f8009040 comp rx=0 tx=0).stop 2023-10-30T13:51:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44100ca0e0 msgr2=0x7f4410158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44100ca0e0 0x7f4410158810 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f440001b900 tx=0x7f440001ba90 comp rx=0 tx=0).stop 2023-10-30T13:51:03.467 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 shutdown_connections 2023-10-30T13:51:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f43f00664a0 0x7f43f0068960 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44101538a0 0x7f44101e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f44100cf160 0x7f44101e40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 --2- 172.21.15.148:0/2527363161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44100ca0e0 0x7f4410158810 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 >> 172.21.15.148:0/2527363161 conn(0x7f441006bfe0 msgr2=0x7f4410065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:03.468 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 shutdown_connections 2023-10-30T13:51:03.469 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:03.465+0000 7f4415b01640 1 -- 172.21.15.148:0/2527363161 wait complete. 2023-10-30T13:51:03.488 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:03.945 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:03.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:05.489 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:05.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49fad1d640 1 Processor -- start 2023-10-30T13:51:05.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49fad1d640 1 -- start start 2023-10-30T13:51:05.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49fad1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 0x7f49f406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:05.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49fad1d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f49f406cb80 0x7f49f406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:05.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49fad1d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f406d550 con 0x7f49f4069e90 2023-10-30T13:51:05.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49fad1d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f40c8570 con 0x7f49f406cb80 2023-10-30T13:51:05.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49fad1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f415a850 con 0x7f49f415a410 2023-10-30T13:51:05.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49f8ab9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f49f4069e90 0x7f49f40c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55340/0 (socket says 172.21.15.148:55340) 2023-10-30T13:51:05.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49f8ab9640 1 -- 172.21.15.148:0/3543351690 learned_addr learned my addr 172.21.15.148:0/3543351690 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:51:05.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49f92ba640 1 --2- 172.21.15.148:0/3543351690 >> v2:172.21.15.148:3300/0 conn(0x7f49f406cb80 0x7f49f406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:05.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49ebfff640 1 --2- 172.21.15.148:0/3543351690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 0x7f49f406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:05.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49f92ba640 1 -- 172.21.15.148:0/3543351690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 msgr2=0x7f49f406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49f92ba640 1 --2- 172.21.15.148:0/3543351690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 0x7f49f406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49f92ba640 1 -- 172.21.15.148:0/3543351690 >> v1:172.21.15.148:6789/0 conn(0x7f49f4069e90 legacy=0x7f49f40c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.633+0000 7f49f92ba640 1 -- 172.21.15.148:0/3543351690 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f40c6c10 con 0x7f49f406cb80 2023-10-30T13:51:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49f92ba640 1 --2- 172.21.15.148:0/3543351690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 0x7f49f406d010 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009d90 tx=0x7f49ec01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a47c96554ecc06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49eb7fe640 1 -- 172.21.15.148:0/3543351690 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0a5810 con 0x7f49f406cb80 2023-10-30T13:51:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49eb7fe640 1 -- 172.21.15.148:0/3543351690 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ec0a5970 con 0x7f49f406cb80 2023-10-30T13:51:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49eb7fe640 1 -- 172.21.15.148:0/3543351690 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0a5d00 con 0x7f49f406cb80 2023-10-30T13:51:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49fad1d640 1 -- 172.21.15.148:0/3543351690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 msgr2=0x7f49f406d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49fad1d640 1 --2- 172.21.15.148:0/3543351690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 0x7f49f406d010 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f49ec009d90 tx=0x7f49ec01c950 comp rx=0 tx=0).stop 2023-10-30T13:51:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49fad1d640 1 -- 172.21.15.148:0/3543351690 shutdown_connections 2023-10-30T13:51:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49fad1d640 1 --2- 172.21.15.148:0/3543351690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 0x7f49f406d010 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49fad1d640 1 --2- 172.21.15.148:0/3543351690 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 0x7f49f406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49fad1d640 1 -- 172.21.15.148:0/3543351690 >> 172.21.15.148:0/3543351690 conn(0x7f49f406dfd0 msgr2=0x7f49f406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49fad1d640 1 -- 172.21.15.148:0/3543351690 shutdown_connections 2023-10-30T13:51:05.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.637+0000 7f49fad1d640 1 -- 172.21.15.148:0/3543351690 wait complete. 2023-10-30T13:51:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 Processor -- start 2023-10-30T13:51:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 -- start start 2023-10-30T13:51:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f4069e90 0x7f49f41f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 0x7f49f4071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:05.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 0x7f49f4071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:05.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f4072010 con 0x7f49f406cb80 2023-10-30T13:51:05.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f41f5dd0 con 0x7f49f4069e90 2023-10-30T13:51:05.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f41f1e30 con 0x7f49f415a410 2023-10-30T13:51:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49ebfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 0x7f49f4071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49ebfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 0x7f49f4071120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53820/0 (socket says 172.21.15.148:53820) 2023-10-30T13:51:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49ebfff640 1 -- 172.21.15.148:0/3141208778 learned_addr learned my addr 172.21.15.148:0/3141208778 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49f8ab9640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f4069e90 0x7f49f41f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49f92ba640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 0x7f49f4071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49ebfff640 1 -- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 msgr2=0x7f49f4071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49ebfff640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 0x7f49f4071660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49ebfff640 1 -- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f4069e90 msgr2=0x7f49f41f22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49ebfff640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f4069e90 0x7f49f41f22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49ebfff640 1 -- 172.21.15.148:0/3141208778 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec009a70 con 0x7f49f406cb80 2023-10-30T13:51:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49ebfff640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 0x7f49f4071120 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f49e400d620 tx=0x7f49e400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49e97fa640 1 -- 172.21.15.148:0/3141208778 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e400b620 con 0x7f49f406cb80 2023-10-30T13:51:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49e97fa640 1 -- 172.21.15.148:0/3141208778 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e400b780 con 0x7f49f406cb80 2023-10-30T13:51:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49e97fa640 1 -- 172.21.15.148:0/3141208778 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e400bb10 con 0x7f49f406cb80 2023-10-30T13:51:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f4150a60 con 0x7f49f406cb80 2023-10-30T13:51:05.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f4072260 con 0x7f49f406cb80 2023-10-30T13:51:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.641+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f404e170 con 0x7f49f406cb80 2023-10-30T13:51:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.645+0000 7f49e97fa640 1 -- 172.21.15.148:0/3141208778 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f49e400bc70 con 0x7f49f406cb80 2023-10-30T13:51:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.645+0000 7f49e97fa640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f49e00664d0 0x7f49e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.645+0000 7f49e97fa640 1 -- 172.21.15.148:0/3141208778 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f49e4084b80 con 0x7f49f406cb80 2023-10-30T13:51:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.645+0000 7f49f8ab9640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f49e00664d0 0x7f49e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.645+0000 7f49f8ab9640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f49e00664d0 0x7f49e0068990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f49f0005890 tx=0x7f49f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.649+0000 7f49e97fa640 1 -- 172.21.15.148:0/3141208778 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f49e4051730 con 0x7f49f406cb80 2023-10-30T13:51:05.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.829+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49f40c6350 con 0x7f49f406cb80 2023-10-30T13:51:05.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49e97fa640 1 -- 172.21.15.148:0/3141208778 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5737 (secure 0 0 0) 0x7f49e4084e60 con 0x7f49f406cb80 2023-10-30T13:51:05.833 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:05.833 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:50:55.057633+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:replay","state_seq":1,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:05.833 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 159 2023-10-30T13:51:05.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f49e00664d0 msgr2=0x7f49e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:05.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f49e00664d0 0x7f49e0068990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f49f0005890 tx=0x7f49f0009040 comp rx=0 tx=0).stop 2023-10-30T13:51:05.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 msgr2=0x7f49f4071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:05.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 0x7f49f4071120 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f49e400d620 tx=0x7f49e400dae0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 shutdown_connections 2023-10-30T13:51:05.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f49e00664d0 0x7f49e0068990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f49f406cb80 0x7f49f4071120 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f49f415a410 0x7f49f4071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 --2- 172.21.15.148:0/3141208778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49f4069e90 0x7f49f41f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 >> 172.21.15.148:0/3141208778 conn(0x7f49f406dfd0 msgr2=0x7f49f4158fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 shutdown_connections 2023-10-30T13:51:05.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:05.833+0000 7f49fad1d640 1 -- 172.21.15.148:0/3141208778 wait complete. 2023-10-30T13:51:05.854 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:07.856 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:08.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.000+0000 7fd34de6b640 1 Processor -- start 2023-10-30T13:51:08.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd34de6b640 1 -- start start 2023-10-30T13:51:08.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd34de6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd348159e60 0x7fd34806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:08.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd34de6b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd3480bfec0 0x7fd3480c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:08.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd34de6b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd34806a720 0x7fd34806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:08.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd34de6b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34806fa80 con 0x7fd34806a720 2023-10-30T13:51:08.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd34de6b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd348071290 con 0x7fd3480bfec0 2023-10-30T13:51:08.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd34de6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd348071570 con 0x7fd348159e60 2023-10-30T13:51:08.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd347fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd34806a720 0x7fd34806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:08.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd347fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd34806a720 0x7fd34806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53832/0 (socket says 172.21.15.148:53832) 2023-10-30T13:51:08.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd347fff640 1 -- 172.21.15.148:0/660408830 learned_addr learned my addr 172.21.15.148:0/660408830 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:08.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd3477fe640 1 --2- 172.21.15.148:0/660408830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd348159e60 0x7fd34806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:08.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd346ffd640 1 --2- 172.21.15.148:0/660408830 >> v2:172.21.15.150:3300/0 conn(0x7fd3480bfec0 0x7fd3480c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:08.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd347fff640 1 -- 172.21.15.148:0/660408830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd348159e60 msgr2=0x7fd34806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:08.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd347fff640 1 --2- 172.21.15.148:0/660408830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd348159e60 0x7fd34806f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd347fff640 1 -- 172.21.15.148:0/660408830 >> v2:172.21.15.150:3300/0 conn(0x7fd3480bfec0 msgr2=0x7fd3480c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:08.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd347fff640 1 --2- 172.21.15.148:0/660408830 >> v2:172.21.15.150:3300/0 conn(0x7fd3480bfec0 0x7fd3480c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.009 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd347fff640 1 -- 172.21.15.148:0/660408830 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd348071850 con 0x7fd34806a720 2023-10-30T13:51:08.010 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.004+0000 7fd3477fe640 1 --2- 172.21.15.148:0/660408830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd348159e60 0x7fd34806f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:08.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd347fff640 1 --2- 172.21.15.148:0/660408830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34806a720 0x7fd34806abb0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd34806f230 tx=0x7fd34001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc83c76a8d5804c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:08.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd3467fc640 1 -- 172.21.15.148:0/660408830 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400af400 con 0x7fd34806a720 2023-10-30T13:51:08.011 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd3467fc640 1 -- 172.21.15.148:0/660408830 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3400ab050 con 0x7fd34806a720 2023-10-30T13:51:08.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd3467fc640 1 -- 172.21.15.148:0/660408830 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400af780 con 0x7fd34806a720 2023-10-30T13:51:08.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 -- 172.21.15.148:0/660408830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34806a720 msgr2=0x7fd34806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:08.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 --2- 172.21.15.148:0/660408830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34806a720 0x7fd34806abb0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd34806f230 tx=0x7fd34001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 -- 172.21.15.148:0/660408830 shutdown_connections 2023-10-30T13:51:08.012 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 --2- 172.21.15.148:0/660408830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34806a720 0x7fd34806abb0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 --2- 172.21.15.148:0/660408830 >> v2:172.21.15.150:3300/0 conn(0x7fd3480bfec0 0x7fd3480c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 --2- 172.21.15.148:0/660408830 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd348159e60 0x7fd34806f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 -- 172.21.15.148:0/660408830 >> 172.21.15.148:0/660408830 conn(0x7fd3480c7910 msgr2=0x7fd3480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:08.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 -- 172.21.15.148:0/660408830 shutdown_connections 2023-10-30T13:51:08.013 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 -- 172.21.15.148:0/660408830 wait complete. 2023-10-30T13:51:08.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.008+0000 7fd34de6b640 1 Processor -- start 2023-10-30T13:51:08.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 -- start start 2023-10-30T13:51:08.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd34806a720 0x7fd3481ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:08.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480bfec0 0x7fd34814e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:08.016 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd348159e60 0x7fd348158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:08.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348158a80 con 0x7fd3480bfec0 2023-10-30T13:51:08.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3481efab0 con 0x7fd348159e60 2023-10-30T13:51:08.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3480c6620 con 0x7fd34806a720 2023-10-30T13:51:08.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd346ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480bfec0 0x7fd34814e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:08.017 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd346ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480bfec0 0x7fd34814e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53846/0 (socket says 172.21.15.148:53846) 2023-10-30T13:51:08.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd346ffd640 1 -- 172.21.15.148:0/2087669851 learned_addr learned my addr 172.21.15.148:0/2087669851 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:08.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd3477fe640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd34806a720 0x7fd3481ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:08.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd347fff640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd348159e60 0x7fd348158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:08.018 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd346ffd640 1 -- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd34806a720 msgr2=0x7fd3481ef570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:08.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd346ffd640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd34806a720 0x7fd3481ef570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd346ffd640 1 -- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd348159e60 msgr2=0x7fd348158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:08.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd346ffd640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd348159e60 0x7fd348158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd346ffd640 1 -- 172.21.15.148:0/2087669851 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340009b30 con 0x7fd3480bfec0 2023-10-30T13:51:08.019 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd347fff640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd348159e60 0x7fd348158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:08.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd346ffd640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480bfec0 0x7fd34814e260 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00e8b0 tx=0x7fd33c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:08.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd32ffff640 1 -- 172.21.15.148:0/2087669851 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00c8e0 con 0x7fd3480bfec0 2023-10-30T13:51:08.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd32ffff640 1 -- 172.21.15.148:0/2087669851 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c00ca40 con 0x7fd3480bfec0 2023-10-30T13:51:08.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3481584c0 con 0x7fd3480bfec0 2023-10-30T13:51:08.020 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd32ffff640 1 -- 172.21.15.148:0/2087669851 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00cdd0 con 0x7fd3480bfec0 2023-10-30T13:51:08.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34814d780 con 0x7fd3480bfec0 2023-10-30T13:51:08.021 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.012+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34814d5e0 con 0x7fd3480bfec0 2023-10-30T13:51:08.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.016+0000 7fd32ffff640 1 -- 172.21.15.148:0/2087669851 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd33c013470 con 0x7fd3480bfec0 2023-10-30T13:51:08.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.016+0000 7fd32ffff640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3340664d0 0x7fd334068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:08.022 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.016+0000 7fd32ffff640 1 -- 172.21.15.148:0/2087669851 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fd33c084ab0 con 0x7fd3480bfec0 2023-10-30T13:51:08.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.016+0000 7fd3477fe640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3340664d0 0x7fd334068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:08.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.016+0000 7fd3477fe640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3340664d0 0x7fd334068990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd3380045f0 tx=0x7fd338008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:08.023 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.020+0000 7fd32ffff640 1 -- 172.21.15.148:0/2087669851 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd34814d5e0 con 0x7fd3480bfec0 2023-10-30T13:51:08.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.200+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd34814d5e0 con 0x7fd3480bfec0 2023-10-30T13:51:08.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.200+0000 7fd32ffff640 1 -- 172.21.15.148:0/2087669851 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5739 (secure 0 0 0) 0x7fd34814d5e0 con 0x7fd3480bfec0 2023-10-30T13:51:08.204 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:08.205 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:05.563166+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:resolve","state_seq":20,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:08.205 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 160 2023-10-30T13:51:08.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3340664d0 msgr2=0x7fd334068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:08.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3340664d0 0x7fd334068990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd3380045f0 tx=0x7fd338008040 comp rx=0 tx=0).stop 2023-10-30T13:51:08.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480bfec0 msgr2=0x7fd34814e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:08.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480bfec0 0x7fd34814e260 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00e8b0 tx=0x7fd33c00ed70 comp rx=0 tx=0).stop 2023-10-30T13:51:08.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 shutdown_connections 2023-10-30T13:51:08.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd3340664d0 0x7fd334068990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.207 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3480bfec0 0x7fd34814e260 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd348159e60 0x7fd348158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 --2- 172.21.15.148:0/2087669851 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd34806a720 0x7fd3481ef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:08.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 >> 172.21.15.148:0/2087669851 conn(0x7fd3480c7910 msgr2=0x7fd34814fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:08.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 shutdown_connections 2023-10-30T13:51:08.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:08.204+0000 7fd34de6b640 1 -- 172.21.15.148:0/2087669851 wait complete. 2023-10-30T13:51:08.225 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 160, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 160, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8041": { "addr": "172.21.15.150:6836/1947781173", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1947781173, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 1947781173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8041, "incarnation": 152, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_8145": { "addr": "172.21.15.148:6835/1526234342", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1526234342, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1526234342, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8145, "incarnation": 159, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:resolve", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:51:05.563166+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8145, "mds_1": 8041 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:51:10.227 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:10.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.372+0000 7f6ac359e640 1 Processor -- start 2023-10-30T13:51:10.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.372+0000 7f6ac359e640 1 -- start start 2023-10-30T13:51:10.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.372+0000 7f6ac359e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6290 0x7f6ac415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:10.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.372+0000 7f6ac359e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac415b060 con 0x7f6ac4071b90 2023-10-30T13:51:10.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.372+0000 7f6ac359e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac40c4ab0 con 0x7f6ac40717b0 2023-10-30T13:51:10.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.372+0000 7f6ac359e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4157530 con 0x7f6ac40c6290 2023-10-30T13:51:10.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac1d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6ac4071b90 0x7f6ac415fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55348/0 (socket says 172.21.15.148:55348) 2023-10-30T13:51:10.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac1d9b640 1 -- 172.21.15.148:0/2471350262 learned_addr learned my addr 172.21.15.148:0/2471350262 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:51:10.377 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac2d9d640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6290 0x7f6ac415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:10.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857334671 0 0) 0x7f6ac415b060 con 0x7f6ac4071b90 2023-10-30T13:51:10.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aac003530 con 0x7f6ac4071b90 2023-10-30T13:51:10.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278665750 0 0) 0x7f6ac40c4ab0 con 0x7f6ac40717b0 2023-10-30T13:51:10.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac415b060 con 0x7f6ac40717b0 2023-10-30T13:51:10.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 602574533 0 0) 0x7f6aac003530 con 0x7f6ac4071b90 2023-10-30T13:51:10.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6290 msgr2=0x7f6ac415ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:10.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6290 0x7f6ac415ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 >> v1:172.21.15.150:6789/0 conn(0x7f6ac40717b0 legacy=0x7f6ac41513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:10.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac40c4d90 con 0x7f6ac4071b90 2023-10-30T13:51:10.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f6ab8001680 con 0x7f6ac4071b90 2023-10-30T13:51:10.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 >> v1:172.21.15.148:6789/0 conn(0x7f6ac4071b90 legacy=0x7f6ac415fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:10.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6290 0x7f6aac003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:10.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6aac0040a0 0x7f6aac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:10.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6aac004d10 0x7f6aac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:10.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac00ba40 con 0x7f6aac0040a0 2023-10-30T13:51:10.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac00bbb0 con 0x7f6aac004d10 2023-10-30T13:51:10.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac00bd70 con 0x7f6ac40c6290 2023-10-30T13:51:10.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac1d9b640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6aac0040a0 0x7f6aac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:10.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac2d9d640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6aac004d10 0x7f6aac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:10.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac259c640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6290 0x7f6aac003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:10.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac1d9b640 1 -- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6290 msgr2=0x7f6aac003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:10.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac1d9b640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6290 0x7f6aac003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac1d9b640 1 -- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6aac004d10 msgr2=0x7f6aac005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:10.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac1d9b640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6aac004d10 0x7f6aac005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac1d9b640 1 -- 172.21.15.148:0/2471350262 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac003530 con 0x7f6aac0040a0 2023-10-30T13:51:10.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.376+0000 7f6ac259c640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac40c6290 0x7f6aac003b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:10.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac1d9b640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6aac0040a0 0x7f6aac0047d0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800c160 tx=0x7f6ab801f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439e949831e207e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:10.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8005ae0 con 0x7f6aac0040a0 2023-10-30T13:51:10.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab80ab070 con 0x7f6aac0040a0 2023-10-30T13:51:10.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac159a640 1 -- 172.21.15.148:0/2471350262 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8007070 con 0x7f6aac0040a0 2023-10-30T13:51:10.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac359e640 1 -- 172.21.15.148:0/2471350262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6aac0040a0 msgr2=0x7f6aac0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:10.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac359e640 1 --2- 172.21.15.148:0/2471350262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6aac0040a0 0x7f6aac0047d0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800c160 tx=0x7f6ab801f310 comp rx=0 tx=0).stop 2023-10-30T13:51:10.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac2d9d640 1 -- 172.21.15.148:0/2471350262 reap_dead start 2023-10-30T13:51:10.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac359e640 1 -- 172.21.15.148:0/2471350262 shutdown_connections 2023-10-30T13:51:10.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac359e640 1 -- 172.21.15.148:0/2471350262 >> 172.21.15.148:0/2471350262 conn(0x7f6ac40c75d0 msgr2=0x7f6ac40c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:10.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac359e640 1 -- 172.21.15.148:0/2471350262 shutdown_connections 2023-10-30T13:51:10.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac359e640 1 -- 172.21.15.148:0/2471350262 wait complete. 2023-10-30T13:51:10.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac359e640 1 Processor -- start 2023-10-30T13:51:10.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.380+0000 7f6ac359e640 1 -- start start 2023-10-30T13:51:10.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6aac0040a0 0x7f6ac415a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:10.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac359e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac4151420 0x7f6ac4155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:10.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac359e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac41492d0 0x7f6ac4149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:10.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac359e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4149c90 con 0x7f6ac41492d0 2023-10-30T13:51:10.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac359e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40c9b50 con 0x7f6aac0040a0 2023-10-30T13:51:10.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac359e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40c9e10 con 0x7f6ac4151420 2023-10-30T13:51:10.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac2d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac41492d0 0x7f6ac4149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:10.391 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac2d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac41492d0 0x7f6ac4149750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:53866/0 (socket says 172.21.15.148:53866) 2023-10-30T13:51:10.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac2d9d640 1 -- 172.21.15.148:0/1998120169 learned_addr learned my addr 172.21.15.148:0/1998120169 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:10.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac259c640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6aac0040a0 0x7f6ac415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:10.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac1d9b640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac4151420 0x7f6ac4155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:10.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac2d9d640 1 -- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac4151420 msgr2=0x7f6ac4155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:10.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac2d9d640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac4151420 0x7f6ac4155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac2d9d640 1 -- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6aac0040a0 msgr2=0x7f6ac415a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:10.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac2d9d640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6aac0040a0 0x7f6ac415a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac2d9d640 1 -- 172.21.15.148:0/1998120169 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac4159e70 con 0x7f6ac41492d0 2023-10-30T13:51:10.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac2d9d640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac41492d0 0x7f6ac4149750 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f6ac406f2d0 tx=0x7f6abc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:10.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ab2ffd640 1 -- 172.21.15.148:0/1998120169 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc00e040 con 0x7f6ac41492d0 2023-10-30T13:51:10.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ab2ffd640 1 -- 172.21.15.148:0/1998120169 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc00f040 con 0x7f6ac41492d0 2023-10-30T13:51:10.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac415a150 con 0x7f6ac41492d0 2023-10-30T13:51:10.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ab2ffd640 1 -- 172.21.15.148:0/1998120169 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc015430 con 0x7f6ac41492d0 2023-10-30T13:51:10.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac4155760 con 0x7f6ac41492d0 2023-10-30T13:51:10.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac404e170 con 0x7f6ac41492d0 2023-10-30T13:51:10.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.384+0000 7f6ab2ffd640 1 -- 172.21.15.148:0/1998120169 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6abc015590 con 0x7f6ac41492d0 2023-10-30T13:51:10.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.388+0000 7f6ab2ffd640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa40664d0 0x7f6aa4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:10.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.388+0000 7f6ab2ffd640 1 -- 172.21.15.148:0/1998120169 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f6abc084f90 con 0x7f6ac41492d0 2023-10-30T13:51:10.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.388+0000 7f6ac259c640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa40664d0 0x7f6aa4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:10.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.388+0000 7f6ac259c640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa40664d0 0x7f6aa4068990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4007fd0 tx=0x7f6ab4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:10.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.392+0000 7f6ab2ffd640 1 -- 172.21.15.148:0/1998120169 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6abc051c60 con 0x7f6ac41492d0 2023-10-30T13:51:10.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ac40c3410 con 0x7f6ac41492d0 2023-10-30T13:51:10.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ab2ffd640 1 -- 172.21.15.148:0/1998120169 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f6abc055280 con 0x7f6ac41492d0 2023-10-30T13:51:10.574 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:10.574 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:05.563166+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:resolve","state_seq":20,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:10.574 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 160 2023-10-30T13:51:10.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa40664d0 msgr2=0x7f6aa4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:10.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa40664d0 0x7f6aa4068990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4007fd0 tx=0x7f6ab4009040 comp rx=0 tx=0).stop 2023-10-30T13:51:10.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac41492d0 msgr2=0x7f6ac4149750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:10.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac41492d0 0x7f6ac4149750 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f6ac406f2d0 tx=0x7f6abc00f580 comp rx=0 tx=0).stop 2023-10-30T13:51:10.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 shutdown_connections 2023-10-30T13:51:10.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6aa40664d0 0x7f6aa4068990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ac41492d0 0x7f6ac4149750 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6ac4151420 0x7f6ac4155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 --2- 172.21.15.148:0/1998120169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6aac0040a0 0x7f6ac415a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:10.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 >> 172.21.15.148:0/1998120169 conn(0x7f6ac40c75d0 msgr2=0x7f6ac406ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:10.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 shutdown_connections 2023-10-30T13:51:10.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:10.572+0000 7f6ac359e640 1 -- 172.21.15.148:0/1998120169 wait complete. 2023-10-30T13:51:10.595 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:12.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:12.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.736+0000 7f23cbe17640 1 Processor -- start 2023-10-30T13:51:12.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.736+0000 7f23cbe17640 1 -- start start 2023-10-30T13:51:12.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.736+0000 7f23cbe17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065a80 0x7f23c40cb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.736+0000 7f23cbe17640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23c4065ee0 0x7f23c40cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.736+0000 7f23cbe17640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c41560a0 con 0x7f23c414fd50 2023-10-30T13:51:12.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.736+0000 7f23cbe17640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c4156380 con 0x7f23c4065ee0 2023-10-30T13:51:12.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.736+0000 7f23cbe17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c4156660 con 0x7f23c4065a80 2023-10-30T13:51:12.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23ca3b4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23c4065ee0 0x7f23c40cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23c93b2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065a80 0x7f23c40cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23ca3b4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23c4065ee0 0x7f23c40cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:43506/0 (socket says 172.21.15.148:43506) 2023-10-30T13:51:12.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23c93b2640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065a80 0x7f23c40cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:35806/0 (socket says 172.21.15.148:35806) 2023-10-30T13:51:12.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23ca3b4640 1 -- 172.21.15.148:0/374795306 learned_addr learned my addr 172.21.15.148:0/374795306 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:12.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23ca3b4640 1 -- 172.21.15.148:0/374795306 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065a80 msgr2=0x7f23c40cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:12.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23ca3b4640 1 --2- 172.21.15.148:0/374795306 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065a80 0x7f23c40cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23ca3b4640 1 -- 172.21.15.148:0/374795306 >> v1:172.21.15.150:6789/0 conn(0x7f23c414fd50 legacy=0x7f23c40c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23ca3b4640 1 -- 172.21.15.148:0/374795306 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c4156940 con 0x7f23c4065ee0 2023-10-30T13:51:12.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.740+0000 7f23ca3b4640 1 --2- 172.21.15.148:0/374795306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c4065ee0 0x7f23c40cba90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009e50 tx=0x7f23bc01c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68bf0cd90ace756a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:12.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23c8bb1640 1 -- 172.21.15.148:0/374795306 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc0a5810 con 0x7f23c4065ee0 2023-10-30T13:51:12.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23c8bb1640 1 -- 172.21.15.148:0/374795306 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23bc0a5970 con 0x7f23c4065ee0 2023-10-30T13:51:12.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23c8bb1640 1 -- 172.21.15.148:0/374795306 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc0a5d00 con 0x7f23c4065ee0 2023-10-30T13:51:12.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 -- 172.21.15.148:0/374795306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c4065ee0 msgr2=0x7f23c40cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 --2- 172.21.15.148:0/374795306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c4065ee0 0x7f23c40cba90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009e50 tx=0x7f23bc01c9b0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 -- 172.21.15.148:0/374795306 shutdown_connections 2023-10-30T13:51:12.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 --2- 172.21.15.148:0/374795306 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065a80 0x7f23c40cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 --2- 172.21.15.148:0/374795306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c4065ee0 0x7f23c40cba90 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 -- 172.21.15.148:0/374795306 >> 172.21.15.148:0/374795306 conn(0x7f23c406c6e0 msgr2=0x7f23c4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:12.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 -- 172.21.15.148:0/374795306 shutdown_connections 2023-10-30T13:51:12.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 -- 172.21.15.148:0/374795306 wait complete. 2023-10-30T13:51:12.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 Processor -- start 2023-10-30T13:51:12.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 -- start start 2023-10-30T13:51:12.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c4065a80 0x7f23c41efa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065ee0 0x7f23c41462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c414fd50 0x7f23c40c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41f3eb0 con 0x7f23c414fd50 2023-10-30T13:51:12.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41f3090 con 0x7f23c4065a80 2023-10-30T13:51:12.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23cbe17640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c41f3330 con 0x7f23c4065ee0 2023-10-30T13:51:12.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23ca3b4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c414fd50 0x7f23c40c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23ca3b4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c414fd50 0x7f23c40c83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41006/0 (socket says 172.21.15.148:41006) 2023-10-30T13:51:12.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.744+0000 7f23ca3b4640 1 -- 172.21.15.148:0/936529280 learned_addr learned my addr 172.21.15.148:0/936529280 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:12.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23c93b2640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065ee0 0x7f23c41462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23c9bb3640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c4065a80 0x7f23c41efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ca3b4640 1 -- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065ee0 msgr2=0x7f23c41462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ca3b4640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065ee0 0x7f23c41462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ca3b4640 1 -- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c4065a80 msgr2=0x7f23c41efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ca3b4640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c4065a80 0x7f23c41efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ca3b4640 1 -- 172.21.15.148:0/936529280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc009b30 con 0x7f23c414fd50 2023-10-30T13:51:12.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ca3b4640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c414fd50 0x7f23c40c83f0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f23bc01a640 tx=0x7f23bc0197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:12.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ba7fc640 1 -- 172.21.15.148:0/936529280 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc0b8670 con 0x7f23c414fd50 2023-10-30T13:51:12.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c41f3610 con 0x7f23c414fd50 2023-10-30T13:51:12.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ba7fc640 1 -- 172.21.15.148:0/936529280 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23bc0b87d0 con 0x7f23c414fd50 2023-10-30T13:51:12.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ba7fc640 1 -- 172.21.15.148:0/936529280 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc0b8b40 con 0x7f23c414fd50 2023-10-30T13:51:12.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c41f3bd0 con 0x7f23c414fd50 2023-10-30T13:51:12.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c4158910 con 0x7f23c414fd50 2023-10-30T13:51:12.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ba7fc640 1 -- 172.21.15.148:0/936529280 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23bc0b8ca0 con 0x7f23c414fd50 2023-10-30T13:51:12.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.748+0000 7f23ba7fc640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23a80664a0 0x7f23a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:12.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.752+0000 7f23ba7fc640 1 -- 172.21.15.148:0/936529280 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f23bc11fc90 con 0x7f23c414fd50 2023-10-30T13:51:12.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.752+0000 7f23c9bb3640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23a80664a0 0x7f23a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:12.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.752+0000 7f23c9bb3640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23a80664a0 0x7f23a8068960 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f23c0007fd0 tx=0x7f23c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:12.756 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.756+0000 7f23ba7fc640 1 -- 172.21.15.148:0/936529280 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23c4158910 con 0x7f23c414fd50 2023-10-30T13:51:12.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23c4158910 con 0x7f23c414fd50 2023-10-30T13:51:12.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23ba7fc640 1 -- 172.21.15.148:0/936529280 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f23c4158910 con 0x7f23c414fd50 2023-10-30T13:51:12.938 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:12.938 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:11.153745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:12.939 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 161 2023-10-30T13:51:12.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23a80664a0 msgr2=0x7f23a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23a80664a0 0x7f23a8068960 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f23c0007fd0 tx=0x7f23c0009040 comp rx=0 tx=0).stop 2023-10-30T13:51:12.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c414fd50 msgr2=0x7f23c40c83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:12.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c414fd50 0x7f23c40c83f0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f23bc01a640 tx=0x7f23bc0197b0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 shutdown_connections 2023-10-30T13:51:12.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f23a80664a0 0x7f23a8068960 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23c414fd50 0x7f23c40c83f0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f23c4065ee0 0x7f23c41462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 --2- 172.21.15.148:0/936529280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c4065a80 0x7f23c41efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:12.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 >> 172.21.15.148:0/936529280 conn(0x7f23c406c6e0 msgr2=0x7f23c40cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:12.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 shutdown_connections 2023-10-30T13:51:12.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:12.936+0000 7f23cbe17640 1 -- 172.21.15.148:0/936529280 wait complete. 2023-10-30T13:51:12.959 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:14.960 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:15.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.108+0000 7f45a7994640 1 Processor -- start 2023-10-30T13:51:15.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.108+0000 7f45a7994640 1 -- start start 2023-10-30T13:51:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.108+0000 7f45a7994640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f45a01580e0 0x7f45a0070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:15.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.108+0000 7f45a7994640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 0x7f45a006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.108+0000 7f45a7994640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00c5810 con 0x7f45a00c6540 2023-10-30T13:51:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.108+0000 7f45a7994640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a00c6990 con 0x7f45a01580e0 2023-10-30T13:51:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.108+0000 7f45a7994640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a015eb90 con 0x7f45a015e700 2023-10-30T13:51:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f45a4f2f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f45a00c6540 0x7f45a0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:58832/0 (socket says 172.21.15.148:58832) 2023-10-30T13:51:15.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f45a4f2f640 1 -- 172.21.15.148:0/408547534 learned_addr learned my addr 172.21.15.148:0/408547534 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:51:15.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f45a5730640 1 --2- 172.21.15.148:0/408547534 >> v2:172.21.15.150:3300/0 conn(0x7f45a01580e0 0x7f45a0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:15.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f45a5f31640 1 --2- 172.21.15.148:0/408547534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 0x7f45a006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:15.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f4597fff640 1 -- 172.21.15.148:0/408547534 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957525143 0 0) 0x7f45a00c5810 con 0x7f45a00c6540 2023-10-30T13:51:15.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f4597fff640 1 -- 172.21.15.148:0/408547534 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f458c003530 con 0x7f45a00c6540 2023-10-30T13:51:15.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f45a5f31640 1 -- 172.21.15.148:0/408547534 >> v2:172.21.15.150:3300/0 conn(0x7f45a01580e0 msgr2=0x7f45a0070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:15.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f45a5f31640 1 --2- 172.21.15.148:0/408547534 >> v2:172.21.15.150:3300/0 conn(0x7f45a01580e0 0x7f45a0070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:15.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f4597fff640 1 -- 172.21.15.148:0/408547534 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4065318010 0 0) 0x7f458c003530 con 0x7f45a00c6540 2023-10-30T13:51:15.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f45a5f31640 1 -- 172.21.15.148:0/408547534 >> v1:172.21.15.148:6789/0 conn(0x7f45a00c6540 legacy=0x7f45a0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:15.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f45a5f31640 1 -- 172.21.15.148:0/408547534 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a00c6bb0 con 0x7f45a015e700 2023-10-30T13:51:15.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.112+0000 7f45a5f31640 1 --2- 172.21.15.148:0/408547534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 0x7f45a006a5a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4598009d90 tx=0x7f459801c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6445d058b5cd322e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:15.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f4597fff640 1 -- 172.21.15.148:0/408547534 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590001680 con 0x7f45a015e700 2023-10-30T13:51:15.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f4597fff640 1 -- 172.21.15.148:0/408547534 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45980a5640 con 0x7f45a015e700 2023-10-30T13:51:15.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f4597fff640 1 -- 172.21.15.148:0/408547534 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45980a59d0 con 0x7f45a015e700 2023-10-30T13:51:15.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 -- 172.21.15.148:0/408547534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 msgr2=0x7f45a006a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:15.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 --2- 172.21.15.148:0/408547534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 0x7f45a006a5a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4598009d90 tx=0x7f459801c840 comp rx=0 tx=0).stop 2023-10-30T13:51:15.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 -- 172.21.15.148:0/408547534 shutdown_connections 2023-10-30T13:51:15.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 --2- 172.21.15.148:0/408547534 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 0x7f45a006a5a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:15.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 --2- 172.21.15.148:0/408547534 >> v2:172.21.15.150:3300/0 conn(0x7f45a01580e0 0x7f45a0070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:15.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 -- 172.21.15.148:0/408547534 >> 172.21.15.148:0/408547534 conn(0x7f45a00c7910 msgr2=0x7f45a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:15.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 -- 172.21.15.148:0/408547534 shutdown_connections 2023-10-30T13:51:15.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 -- 172.21.15.148:0/408547534 wait complete. 2023-10-30T13:51:15.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 Processor -- start 2023-10-30T13:51:15.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 -- start start 2023-10-30T13:51:15.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a00c6540 0x7f45a00c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:15.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a01580e0 0x7f45a0140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:15.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 0x7f45a0140560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:15.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00c9470 con 0x7f45a01580e0 2023-10-30T13:51:15.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00c9730 con 0x7f45a00c6540 2023-10-30T13:51:15.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a7994640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a01f5fb0 con 0x7f45a015e700 2023-10-30T13:51:15.123 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a4f2f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a01580e0 0x7f45a0140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:15.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a4f2f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a01580e0 0x7f45a0140020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41016/0 (socket says 172.21.15.148:41016) 2023-10-30T13:51:15.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a4f2f640 1 -- 172.21.15.148:0/509355428 learned_addr learned my addr 172.21.15.148:0/509355428 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:15.124 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.116+0000 7f45a5730640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a00c6540 0x7f45a00c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:15.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a5f31640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 0x7f45a0140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:15.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a4f2f640 1 -- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 msgr2=0x7f45a0140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:15.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a4f2f640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 0x7f45a0140560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:15.125 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a4f2f640 1 -- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a00c6540 msgr2=0x7f45a00c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:15.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a4f2f640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a00c6540 0x7f45a00c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:15.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a4f2f640 1 -- 172.21.15.148:0/509355428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4598009a70 con 0x7f45a01580e0 2023-10-30T13:51:15.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a5730640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a00c6540 0x7f45a00c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:15.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a4f2f640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a01580e0 0x7f45a0140020 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f459000c780 tx=0x7f459000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:15.126 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f4595ffb640 1 -- 172.21.15.148:0/509355428 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590005cf0 con 0x7f45a01580e0 2023-10-30T13:51:15.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f4595ffb640 1 -- 172.21.15.148:0/509355428 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4590005e50 con 0x7f45a01580e0 2023-10-30T13:51:15.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a01f6290 con 0x7f45a01580e0 2023-10-30T13:51:15.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f4595ffb640 1 -- 172.21.15.148:0/509355428 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45900072b0 con 0x7f45a01580e0 2023-10-30T13:51:15.127 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a01f6850 con 0x7f45a01580e0 2023-10-30T13:51:15.128 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a004e170 con 0x7f45a01580e0 2023-10-30T13:51:15.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.120+0000 7f4595ffb640 1 -- 172.21.15.148:0/509355428 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4590007410 con 0x7f45a01580e0 2023-10-30T13:51:15.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.124+0000 7f4595ffb640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45840664a0 0x7f4584068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:15.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.124+0000 7f4595ffb640 1 -- 172.21.15.148:0/509355428 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f45900844f0 con 0x7f45a01580e0 2023-10-30T13:51:15.129 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.124+0000 7f45a5730640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45840664a0 0x7f4584068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:15.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.124+0000 7f45a5730640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45840664a0 0x7f4584068960 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f459c009bb0 tx=0x7f459c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:15.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.128+0000 7f4595ffb640 1 -- 172.21.15.148:0/509355428 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f45900510a0 con 0x7f45a01580e0 2023-10-30T13:51:15.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.308+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45a006aed0 con 0x7f45a01580e0 2023-10-30T13:51:15.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.308+0000 7f4595ffb640 1 -- 172.21.15.148:0/509355428 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f45900546c0 con 0x7f45a01580e0 2023-10-30T13:51:15.311 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:15.311 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:11.153745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:15.312 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 161 2023-10-30T13:51:15.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45840664a0 msgr2=0x7f4584068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:15.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45840664a0 0x7f4584068960 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f459c009bb0 tx=0x7f459c008040 comp rx=0 tx=0).stop 2023-10-30T13:51:15.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a01580e0 msgr2=0x7f45a0140020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:15.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a01580e0 0x7f45a0140020 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f459000c780 tx=0x7f459000cc40 comp rx=0 tx=0).stop 2023-10-30T13:51:15.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 shutdown_connections 2023-10-30T13:51:15.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45840664a0 0x7f4584068960 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:15.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a01580e0 0x7f45a0140020 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:15.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45a015e700 0x7f45a0140560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:15.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 --2- 172.21.15.148:0/509355428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45a00c6540 0x7f45a00c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:15.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 >> 172.21.15.148:0/509355428 conn(0x7f45a00c7910 msgr2=0x7f45a0147980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:15.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 shutdown_connections 2023-10-30T13:51:15.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:15.312+0000 7f45a7994640 1 -- 172.21.15.148:0/509355428 wait complete. 2023-10-30T13:51:15.332 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:17.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:17.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.480+0000 7f6f0a69a640 1 Processor -- start 2023-10-30T13:51:17.483 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.480+0000 7f6f0a69a640 1 -- start start 2023-10-30T13:51:17.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.480+0000 7f6f0a69a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6f04159e60 0x7f6f0406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.480+0000 7f6f0a69a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f040bfec0 0x7f6f040c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f0a69a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0406fa80 con 0x7f6f0406a720 2023-10-30T13:51:17.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f0a69a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f04071290 con 0x7f6f04159e60 2023-10-30T13:51:17.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f0a69a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04071570 con 0x7f6f040bfec0 2023-10-30T13:51:17.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f03fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6f04159e60 0x7f6f0406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f03fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6f04159e60 0x7f6f0406f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41026/0 (socket says 172.21.15.148:41026) 2023-10-30T13:51:17.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f03fff640 1 -- 172.21.15.148:0/2353858513 learned_addr learned my addr 172.21.15.148:0/2353858513 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:17.486 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f037fe640 1 --2- 172.21.15.148:0/2353858513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f040bfec0 0x7f6f040c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f03fff640 1 -- 172.21.15.148:0/2353858513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f040bfec0 msgr2=0x7f6f040c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:17.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f03fff640 1 --2- 172.21.15.148:0/2353858513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f040bfec0 0x7f6f040c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f03fff640 1 -- 172.21.15.148:0/2353858513 >> v1:172.21.15.148:6789/0 conn(0x7f6f0406a720 legacy=0x7f6f04155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.487 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f03fff640 1 -- 172.21.15.148:0/2353858513 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f04071850 con 0x7f6f04159e60 2023-10-30T13:51:17.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.484+0000 7f6f03fff640 1 --2- 172.21.15.148:0/2353858513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f04159e60 0x7f6f0406f540 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4006fd0 tx=0x7f6ef40211c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cf171532af0fe0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:17.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f02ffd640 1 -- 172.21.15.148:0/2353858513 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4004030 con 0x7f6f04159e60 2023-10-30T13:51:17.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f02ffd640 1 -- 172.21.15.148:0/2353858513 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef40ad030 con 0x7f6f04159e60 2023-10-30T13:51:17.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f02ffd640 1 -- 172.21.15.148:0/2353858513 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4004410 con 0x7f6f04159e60 2023-10-30T13:51:17.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 -- 172.21.15.148:0/2353858513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f04159e60 msgr2=0x7f6f0406f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 --2- 172.21.15.148:0/2353858513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f04159e60 0x7f6f0406f540 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4006fd0 tx=0x7f6ef40211c0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 -- 172.21.15.148:0/2353858513 shutdown_connections 2023-10-30T13:51:17.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 --2- 172.21.15.148:0/2353858513 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f040bfec0 0x7f6f040c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 --2- 172.21.15.148:0/2353858513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f04159e60 0x7f6f0406f540 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 -- 172.21.15.148:0/2353858513 >> 172.21.15.148:0/2353858513 conn(0x7f6f040c7910 msgr2=0x7f6f040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:17.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 -- 172.21.15.148:0/2353858513 shutdown_connections 2023-10-30T13:51:17.491 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 -- 172.21.15.148:0/2353858513 wait complete. 2023-10-30T13:51:17.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 Processor -- start 2023-10-30T13:51:17.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 -- start start 2023-10-30T13:51:17.493 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f0406a720 0x7f6f04146b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f040bfec0 0x7f6f041eecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f04159e60 0x7f6f041ef1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04151930 con 0x7f6f0406a720 2023-10-30T13:51:17.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040b0450 con 0x7f6f040bfec0 2023-10-30T13:51:17.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f0a69a640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04159350 con 0x7f6f04159e60 2023-10-30T13:51:17.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f03fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f0406a720 0x7f6f04146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f03fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f0406a720 0x7f6f04146b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41038/0 (socket says 172.21.15.148:41038) 2023-10-30T13:51:17.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.488+0000 7f6f03fff640 1 -- 172.21.15.148:0/3303715257 learned_addr learned my addr 172.21.15.148:0/3303715257 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:17.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f08c37640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f04159e60 0x7f6f041ef1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f037fe640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f040bfec0 0x7f6f041eecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f03fff640 1 -- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f04159e60 msgr2=0x7f6f041ef1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f03fff640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f04159e60 0x7f6f041ef1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f03fff640 1 -- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f040bfec0 msgr2=0x7f6f041eecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f03fff640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f040bfec0 0x7f6f041eecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f03fff640 1 -- 172.21.15.148:0/3303715257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef400b780 con 0x7f6f0406a720 2023-10-30T13:51:17.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f08c37640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f04159e60 0x7f6f041ef1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:51:17.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f03fff640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f0406a720 0x7f6f04146b20 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4006fd0 tx=0x7f6ef401df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:17.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f00ff9640 1 -- 172.21.15.148:0/3303715257 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef40038f0 con 0x7f6f0406a720 2023-10-30T13:51:17.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f00ff9640 1 -- 172.21.15.148:0/3303715257 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef4003a50 con 0x7f6f0406a720 2023-10-30T13:51:17.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f00ff9640 1 -- 172.21.15.148:0/3303715257 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4003de0 con 0x7f6f0406a720 2023-10-30T13:51:17.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f04159630 con 0x7f6f0406a720 2023-10-30T13:51:17.498 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f040afc50 con 0x7f6f0406a720 2023-10-30T13:51:17.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0404e170 con 0x7f6f0406a720 2023-10-30T13:51:17.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f00ff9640 1 -- 172.21.15.148:0/3303715257 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ef40c7460 con 0x7f6f0406a720 2023-10-30T13:51:17.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.492+0000 7f6f00ff9640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ef00664a0 0x7f6ef0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:17.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.496+0000 7f6f00ff9640 1 -- 172.21.15.148:0/3303715257 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f6ef4123150 con 0x7f6f0406a720 2023-10-30T13:51:17.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.496+0000 7f6f037fe640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ef00664a0 0x7f6ef0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:17.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.496+0000 7f6f037fe640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ef00664a0 0x7f6ef0068960 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6f041ec8e0 tx=0x7f6ef8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:17.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.496+0000 7f6f00ff9640 1 -- 172.21.15.148:0/3303715257 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ef40eff60 con 0x7f6f0406a720 2023-10-30T13:51:17.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.676+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f040c6670 con 0x7f6f0406a720 2023-10-30T13:51:17.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.676+0000 7f6f00ff9640 1 -- 172.21.15.148:0/3303715257 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f6ef40f3580 con 0x7f6f0406a720 2023-10-30T13:51:17.680 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:17.681 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:17.140793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:17.681 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 162 2023-10-30T13:51:17.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ef00664a0 msgr2=0x7f6ef0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ef00664a0 0x7f6ef0068960 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6f041ec8e0 tx=0x7f6ef8008040 comp rx=0 tx=0).stop 2023-10-30T13:51:17.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f0406a720 msgr2=0x7f6f04146b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:17.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f0406a720 0x7f6f04146b20 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4006fd0 tx=0x7f6ef401df60 comp rx=0 tx=0).stop 2023-10-30T13:51:17.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 shutdown_connections 2023-10-30T13:51:17.683 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6ef00664a0 0x7f6ef0068960 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6f04159e60 0x7f6f041ef1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f040bfec0 0x7f6f041eecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 --2- 172.21.15.148:0/3303715257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f0406a720 0x7f6f04146b20 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:17.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 >> 172.21.15.148:0/3303715257 conn(0x7f6f040c7910 msgr2=0x7f6f040ca250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:17.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 shutdown_connections 2023-10-30T13:51:17.684 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:17.680+0000 7f6f0a69a640 1 -- 172.21.15.148:0/3303715257 wait complete. 2023-10-30T13:51:17.702 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:19.702 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:19.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.848+0000 7fca0770e640 1 Processor -- start 2023-10-30T13:51:19.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.848+0000 7fca0770e640 1 -- start start 2023-10-30T13:51:19.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.848+0000 7fca0770e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca001580e0 0x7fca00070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.848+0000 7fca0770e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fca0015e700 0x7fca0006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.848+0000 7fca0770e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca000c5810 con 0x7fca000c6540 2023-10-30T13:51:19.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.848+0000 7fca0770e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca000c6990 con 0x7fca0015e700 2023-10-30T13:51:19.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.848+0000 7fca0770e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0015eb90 con 0x7fca001580e0 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.848+0000 7fca05cab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fca0015e700 0x7fca0006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fca05cab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fca0015e700 0x7fca0006a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41040/0 (socket says 172.21.15.148:41040) 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fca05cab640 1 -- 172.21.15.148:0/665882875 learned_addr learned my addr 172.21.15.148:0/665882875 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fca054aa640 1 --2- 172.21.15.148:0/665882875 >> v2:172.21.15.150:3300/0 conn(0x7fca001580e0 0x7fca00070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fca05cab640 1 -- 172.21.15.148:0/665882875 >> v2:172.21.15.150:3300/0 conn(0x7fca001580e0 msgr2=0x7fca00070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:19.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fca05cab640 1 --2- 172.21.15.148:0/665882875 >> v2:172.21.15.150:3300/0 conn(0x7fca001580e0 0x7fca00070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fca05cab640 1 -- 172.21.15.148:0/665882875 >> v1:172.21.15.148:6789/0 conn(0x7fca000c6540 legacy=0x7fca00069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fca05cab640 1 -- 172.21.15.148:0/665882875 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca000c6bb0 con 0x7fca0015e700 2023-10-30T13:51:19.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fca05cab640 1 --2- 172.21.15.148:0/665882875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 0x7fca0006a5a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fca0006eea0 tx=0x7fc9f801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26991bf88192ea39 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:19.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fc9f7fff640 1 -- 172.21.15.148:0/665882875 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f80a5860 con 0x7fca0015e700 2023-10-30T13:51:19.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fc9f7fff640 1 -- 172.21.15.148:0/665882875 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f80a59c0 con 0x7fca0015e700 2023-10-30T13:51:19.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fc9f7fff640 1 -- 172.21.15.148:0/665882875 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f80a5d50 con 0x7fca0015e700 2023-10-30T13:51:19.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.852+0000 7fca0770e640 1 -- 172.21.15.148:0/665882875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 msgr2=0x7fca0006a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 --2- 172.21.15.148:0/665882875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 0x7fca0006a5a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fca0006eea0 tx=0x7fc9f801c9a0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 -- 172.21.15.148:0/665882875 shutdown_connections 2023-10-30T13:51:19.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 --2- 172.21.15.148:0/665882875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 0x7fca0006a5a0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 --2- 172.21.15.148:0/665882875 >> v2:172.21.15.150:3300/0 conn(0x7fca001580e0 0x7fca00070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 -- 172.21.15.148:0/665882875 >> 172.21.15.148:0/665882875 conn(0x7fca000c7910 msgr2=0x7fca000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:19.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 -- 172.21.15.148:0/665882875 shutdown_connections 2023-10-30T13:51:19.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 -- 172.21.15.148:0/665882875 wait complete. 2023-10-30T13:51:19.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 Processor -- start 2023-10-30T13:51:19.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 -- start start 2023-10-30T13:51:19.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca000c6540 0x7fca00153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca001580e0 0x7fca000ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 0x7fca00145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0014b890 con 0x7fca0015e700 2023-10-30T13:51:19.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca001f9c40 con 0x7fca001580e0 2023-10-30T13:51:19.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca0770e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca001f9f20 con 0x7fca000c6540 2023-10-30T13:51:19.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca05cab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 0x7fca00145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca05cab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 0x7fca00145f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41050/0 (socket says 172.21.15.148:41050) 2023-10-30T13:51:19.862 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca05cab640 1 -- 172.21.15.148:0/1042948729 learned_addr learned my addr 172.21.15.148:0/1042948729 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:19.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca054aa640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca000c6540 0x7fca00153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca04ca9640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca001580e0 0x7fca000ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca05cab640 1 -- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca000c6540 msgr2=0x7fca00153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca05cab640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca000c6540 0x7fca00153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca05cab640 1 -- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca001580e0 msgr2=0x7fca000ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:19.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca05cab640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca001580e0 0x7fca000ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:19.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca05cab640 1 -- 172.21.15.148:0/1042948729 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f8009b30 con 0x7fca0015e700 2023-10-30T13:51:19.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fca05cab640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 0x7fca00145f40 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80af040 tx=0x7fc9f8019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:19.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.856+0000 7fc9f5ffb640 1 -- 172.21.15.148:0/1042948729 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f80b8670 con 0x7fca0015e700 2023-10-30T13:51:19.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.860+0000 7fc9f5ffb640 1 -- 172.21.15.148:0/1042948729 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f80b8810 con 0x7fca0015e700 2023-10-30T13:51:19.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.860+0000 7fc9f5ffb640 1 -- 172.21.15.148:0/1042948729 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f80b8bc0 con 0x7fca0015e700 2023-10-30T13:51:19.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.860+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca001fa200 con 0x7fca0015e700 2023-10-30T13:51:19.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.860+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca001fa7c0 con 0x7fca0015e700 2023-10-30T13:51:19.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.860+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca001fa9b0 con 0x7fca0015e700 2023-10-30T13:51:19.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.860+0000 7fc9f5ffb640 1 -- 172.21.15.148:0/1042948729 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9f80b8d20 con 0x7fca0015e700 2023-10-30T13:51:19.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.860+0000 7fc9f5ffb640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9e40664a0 0x7fc9e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:19.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.860+0000 7fc9f5ffb640 1 -- 172.21.15.148:0/1042948729 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fc9f811fc10 con 0x7fca0015e700 2023-10-30T13:51:19.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.860+0000 7fca054aa640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9e40664a0 0x7fc9e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:19.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.864+0000 7fca054aa640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9e40664a0 0x7fc9e4068960 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009af0 tx=0x7fc9fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:19.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:19.864+0000 7fc9f5ffb640 1 -- 172.21.15.148:0/1042948729 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9f80ec850 con 0x7fca0015e700 2023-10-30T13:51:20.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.044+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca0004e170 con 0x7fca0015e700 2023-10-30T13:51:20.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.044+0000 7fc9f5ffb640 1 -- 172.21.15.148:0/1042948729 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5738 (secure 0 0 0) 0x7fc9f80efe70 con 0x7fca0015e700 2023-10-30T13:51:20.047 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:20.048 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:17.140793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:20.048 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 162 2023-10-30T13:51:20.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9e40664a0 msgr2=0x7fc9e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9e40664a0 0x7fc9e4068960 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009af0 tx=0x7fc9fc008040 comp rx=0 tx=0).stop 2023-10-30T13:51:20.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 msgr2=0x7fca00145f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:20.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 0x7fca00145f40 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80af040 tx=0x7fc9f8019740 comp rx=0 tx=0).stop 2023-10-30T13:51:20.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 shutdown_connections 2023-10-30T13:51:20.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc9e40664a0 0x7fc9e4068960 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca0015e700 0x7fca00145f40 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca001580e0 0x7fca000ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 --2- 172.21.15.148:0/1042948729 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fca000c6540 0x7fca00153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:20.051 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 >> 172.21.15.148:0/1042948729 conn(0x7fca000c7910 msgr2=0x7fca00070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:20.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 shutdown_connections 2023-10-30T13:51:20.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:20.048+0000 7fca0770e640 1 -- 172.21.15.148:0/1042948729 wait complete. 2023-10-30T13:51:20.069 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 162, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T13:10:34.058265+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 162, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8041": { "addr": "172.21.15.150:6836/1947781173", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1947781173, "type": "v2" }, { "addr": "172.21.15.150:6836", "nonce": 1947781173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8041, "incarnation": 152, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_8145": { "addr": "172.21.15.148:6835/1526234342", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6833", "nonce": 1526234342, "type": "v2" }, { "addr": "172.21.15.148:6835", "nonce": 1526234342, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8145, "incarnation": 159, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:51:17.140793+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8145, "mds_1": 8041 } } } ], "standbys": [ { "addr": "172.21.15.150:6837/1180930508", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6835", "nonce": 1180930508, "type": "v2" }, { "addr": "172.21.15.150:6837", "nonce": 1180930508, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4241, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.150:6839/3684521078", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6838", "nonce": 3684521078, "type": "v2" }, { "addr": "172.21.15.150:6839", "nonce": 3684521078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.148:6834/650045128", "addrs": { "addrvec": [ { "addr": "172.21.15.148:6832", "nonce": 650045128, "type": "v2" }, { "addr": "172.21.15.148:6834", "nonce": 650045128, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 42, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5400, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:51:22.070 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:22.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.236+0000 7f07ee3a7640 1 Processor -- start 2023-10-30T13:51:22.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ee3a7640 1 -- start start 2023-10-30T13:51:22.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ee3a7640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f07e81538f0 0x7f07e8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ee3a7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f07e80cefa0 0x7f07e80cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ee3a7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e80ca6a0 con 0x7f07e815cd50 2023-10-30T13:51:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ee3a7640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e80cf8f0 con 0x7f07e81538f0 2023-10-30T13:51:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ee3a7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e815d300 con 0x7f07e80cefa0 2023-10-30T13:51:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ed3a5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f07e81538f0 0x7f07e8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ed3a5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f07e81538f0 0x7f07e8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54720/0 (socket says 172.21.15.148:54720) 2023-10-30T13:51:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ed3a5640 1 -- 172.21.15.148:0/2860607445 learned_addr learned my addr 172.21.15.148:0/2860607445 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07edba6640 1 --2- 172.21.15.148:0/2860607445 >> v2:172.21.15.150:3300/0 conn(0x7f07e80cefa0 0x7f07e80cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ed3a5640 1 -- 172.21.15.148:0/2860607445 >> v2:172.21.15.150:3300/0 conn(0x7f07e80cefa0 msgr2=0x7f07e80cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:22.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ed3a5640 1 --2- 172.21.15.148:0/2860607445 >> v2:172.21.15.150:3300/0 conn(0x7f07e80cefa0 0x7f07e80cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ed3a5640 1 -- 172.21.15.148:0/2860607445 >> v1:172.21.15.148:6789/0 conn(0x7f07e815cd50 legacy=0x7f07e80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.240+0000 7f07ed3a5640 1 -- 172.21.15.148:0/2860607445 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e815d5e0 con 0x7f07e81538f0 2023-10-30T13:51:22.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07ed3a5640 1 --2- 172.21.15.148:0/2860607445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e81538f0 0x7f07e8159fb0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f07e8066a40 tx=0x7f07e401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f154cf2e289ff5f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07dffff640 1 -- 172.21.15.148:0/2860607445 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e40a89f0 con 0x7f07e81538f0 2023-10-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07dffff640 1 -- 172.21.15.148:0/2860607445 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e40a8b50 con 0x7f07e81538f0 2023-10-30T13:51:22.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07dffff640 1 -- 172.21.15.148:0/2860607445 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e40b0440 con 0x7f07e81538f0 2023-10-30T13:51:22.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07ee3a7640 1 -- 172.21.15.148:0/2860607445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e81538f0 msgr2=0x7f07e8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07ee3a7640 1 --2- 172.21.15.148:0/2860607445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e81538f0 0x7f07e8159fb0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f07e8066a40 tx=0x7f07e401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07ee3a7640 1 -- 172.21.15.148:0/2860607445 shutdown_connections 2023-10-30T13:51:22.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07ee3a7640 1 --2- 172.21.15.148:0/2860607445 >> v2:172.21.15.150:3300/0 conn(0x7f07e80cefa0 0x7f07e80cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07ee3a7640 1 --2- 172.21.15.148:0/2860607445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e81538f0 0x7f07e8159fb0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07ee3a7640 1 -- 172.21.15.148:0/2860607445 >> 172.21.15.148:0/2860607445 conn(0x7f07e806a640 msgr2=0x7f07e8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:22.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07ee3a7640 1 -- 172.21.15.148:0/2860607445 shutdown_connections 2023-10-30T13:51:22.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.244+0000 7f07ee3a7640 1 -- 172.21.15.148:0/2860607445 wait complete. 2023-10-30T13:51:22.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 Processor -- start 2023-10-30T13:51:22.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 -- start start 2023-10-30T13:51:22.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07e80cefa0 0x7f07e815bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07e81538f0 0x7f07e8146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e815cd50 0x7f07e8141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e81488c0 con 0x7f07e815cd50 2023-10-30T13:51:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8143860 con 0x7f07e80cefa0 2023-10-30T13:51:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8143430 con 0x7f07e81538f0 2023-10-30T13:51:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07edba6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e815cd50 0x7f07e8141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07edba6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e815cd50 0x7f07e8141a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54736/0 (socket says 172.21.15.148:54736) 2023-10-30T13:51:22.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07edba6640 1 -- 172.21.15.148:0/1854349000 learned_addr learned my addr 172.21.15.148:0/1854349000 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:22.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ecba4640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07e81538f0 0x7f07e8146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ed3a5640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07e80cefa0 0x7f07e815bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07edba6640 1 -- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07e81538f0 msgr2=0x7f07e8146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07edba6640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07e81538f0 0x7f07e8146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07edba6640 1 -- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07e80cefa0 msgr2=0x7f07e815bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07edba6640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07e80cefa0 0x7f07e815bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07edba6640 1 -- 172.21.15.148:0/1854349000 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e40094d0 con 0x7f07e815cd50 2023-10-30T13:51:22.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ed3a5640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07e80cefa0 0x7f07e815bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:22.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ecba4640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07e81538f0 0x7f07e8146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:22.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07edba6640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e815cd50 0x7f07e8141a00 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f07e0007e40 tx=0x7f07e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:22.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ddffb640 1 -- 172.21.15.148:0/1854349000 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0014620 con 0x7f07e815cd50 2023-10-30T13:51:22.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ddffb640 1 -- 172.21.15.148:0/1854349000 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e000ee90 con 0x7f07e815cd50 2023-10-30T13:51:22.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ddffb640 1 -- 172.21.15.148:0/1854349000 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e00149e0 con 0x7f07e815cd50 2023-10-30T13:51:22.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e81479d0 con 0x7f07e815cd50 2023-10-30T13:51:22.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e8147f90 con 0x7f07e815cd50 2023-10-30T13:51:22.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.248+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e804e170 con 0x7f07e815cd50 2023-10-30T13:51:22.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.252+0000 7f07ddffb640 1 -- 172.21.15.148:0/1854349000 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07e0014b40 con 0x7f07e815cd50 2023-10-30T13:51:22.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.252+0000 7f07ddffb640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f07cc0664a0 0x7f07cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:22.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.252+0000 7f07ddffb640 1 -- 172.21.15.148:0/1854349000 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f07e00851d0 con 0x7f07e815cd50 2023-10-30T13:51:22.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.252+0000 7f07ed3a5640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f07cc0664a0 0x7f07cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:22.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.252+0000 7f07ed3a5640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f07cc0664a0 0x7f07cc068960 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f07e40b2000 tx=0x7f07e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:22.258 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.256+0000 7f07ddffb640 1 -- 172.21.15.148:0/1854349000 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07e0051d80 con 0x7f07e815cd50 2023-10-30T13:51:22.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.444+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07e8148a00 con 0x7f07e815cd50 2023-10-30T13:51:22.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.444+0000 7f07ddffb640 1 -- 172.21.15.148:0/1854349000 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f07e00553a0 con 0x7f07e815cd50 2023-10-30T13:51:22.448 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:22.448 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:17.140793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:22.448 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 162 2023-10-30T13:51:22.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f07cc0664a0 msgr2=0x7f07cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f07cc0664a0 0x7f07cc068960 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f07e40b2000 tx=0x7f07e4008040 comp rx=0 tx=0).stop 2023-10-30T13:51:22.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e815cd50 msgr2=0x7f07e8141a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:22.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e815cd50 0x7f07e8141a00 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f07e0007e40 tx=0x7f07e000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.450 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 shutdown_connections 2023-10-30T13:51:22.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f07cc0664a0 0x7f07cc068960 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f07e815cd50 0x7f07e8141a00 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f07e81538f0 0x7f07e8146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 --2- 172.21.15.148:0/1854349000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07e80cefa0 0x7f07e815bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:22.451 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 >> 172.21.15.148:0/1854349000 conn(0x7f07e806a640 msgr2=0x7f07e8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:22.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 shutdown_connections 2023-10-30T13:51:22.452 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:22.448+0000 7f07ee3a7640 1 -- 172.21.15.148:0/1854349000 wait complete. 2023-10-30T13:51:22.469 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:24.470 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:24.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb27d046640 1 Processor -- start 2023-10-30T13:51:24.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb27d046640 1 -- start start 2023-10-30T13:51:24.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb27d046640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb2780c6b80 0x7fb2780bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb27d046640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27813f4a0 0x7fb2780c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb27d046640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2780c0a50 con 0x7fb2780c6f60 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb27d046640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb27806f6e0 con 0x7fb2780c6b80 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb27d046640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb27806f9c0 con 0x7fb27813f4a0 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb276d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb2780c6b80 0x7fb2780bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb276d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb2780c6b80 0x7fb2780bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54748/0 (socket says 172.21.15.148:54748) 2023-10-30T13:51:24.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.616+0000 7fb276d9d640 1 -- 172.21.15.148:0/642742017 learned_addr learned my addr 172.21.15.148:0/642742017 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:24.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb27759e640 1 --2- 172.21.15.148:0/642742017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27813f4a0 0x7fb2780c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb276d9d640 1 -- 172.21.15.148:0/642742017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27813f4a0 msgr2=0x7fb2780c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:24.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb276d9d640 1 --2- 172.21.15.148:0/642742017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27813f4a0 0x7fb2780c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb276d9d640 1 -- 172.21.15.148:0/642742017 >> v1:172.21.15.148:6789/0 conn(0x7fb2780c6f60 legacy=0x7fb27813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb276d9d640 1 -- 172.21.15.148:0/642742017 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27806fca0 con 0x7fb2780c6b80 2023-10-30T13:51:24.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb276d9d640 1 --2- 172.21.15.148:0/642742017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2780c6b80 0x7fb2780bfec0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fb268009870 tx=0x7fb26801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad91fe5fe1699bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:24.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb27659c640 1 -- 172.21.15.148:0/642742017 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2680ab650 con 0x7fb2780c6b80 2023-10-30T13:51:24.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb27659c640 1 -- 172.21.15.148:0/642742017 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2680ab7b0 con 0x7fb2780c6b80 2023-10-30T13:51:24.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb27659c640 1 -- 172.21.15.148:0/642742017 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2680abb40 con 0x7fb2780c6b80 2023-10-30T13:51:24.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb27d046640 1 -- 172.21.15.148:0/642742017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2780c6b80 msgr2=0x7fb2780bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.620+0000 7fb27d046640 1 --2- 172.21.15.148:0/642742017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2780c6b80 0x7fb2780bfec0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fb268009870 tx=0x7fb26801c540 comp rx=0 tx=0).stop 2023-10-30T13:51:24.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 -- 172.21.15.148:0/642742017 shutdown_connections 2023-10-30T13:51:24.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 --2- 172.21.15.148:0/642742017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb27813f4a0 0x7fb2780c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 --2- 172.21.15.148:0/642742017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2780c6b80 0x7fb2780bfec0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 -- 172.21.15.148:0/642742017 >> 172.21.15.148:0/642742017 conn(0x7fb2780c7910 msgr2=0x7fb2780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:24.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 -- 172.21.15.148:0/642742017 shutdown_connections 2023-10-30T13:51:24.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 -- 172.21.15.148:0/642742017 wait complete. 2023-10-30T13:51:24.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 Processor -- start 2023-10-30T13:51:24.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 -- start start 2023-10-30T13:51:24.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780c6b80 0x7fb27813c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2780c6f60 0x7fb278139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb27813f4a0 0x7fb27813a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2780ca0c0 con 0x7fb27813f4a0 2023-10-30T13:51:24.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2780c9060 con 0x7fb2780c6b80 2023-10-30T13:51:24.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27d046640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2780c9340 con 0x7fb2780c6f60 2023-10-30T13:51:24.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb27813f4a0 0x7fb27813a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb27813f4a0 0x7fb27813a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54760/0 (socket says 172.21.15.148:54760) 2023-10-30T13:51:24.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27759e640 1 -- 172.21.15.148:0/4232497790 learned_addr learned my addr 172.21.15.148:0/4232497790 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:24.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb276d9d640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780c6b80 0x7fb27813c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb26ffff640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2780c6f60 0x7fb278139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27759e640 1 -- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2780c6f60 msgr2=0x7fb278139bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27759e640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2780c6f60 0x7fb278139bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27759e640 1 -- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780c6b80 msgr2=0x7fb27813c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27759e640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780c6b80 0x7fb27813c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.624+0000 7fb27759e640 1 -- 172.21.15.148:0/4232497790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680094d0 con 0x7fb27813f4a0 2023-10-30T13:51:24.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb27759e640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb27813f4a0 0x7fb27813a130 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fb270007e60 tx=0x7fb27000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:24.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb26f7fe640 1 -- 172.21.15.148:0/4232497790 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270014420 con 0x7fb27813f4a0 2023-10-30T13:51:24.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb26f7fe640 1 -- 172.21.15.148:0/4232497790 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2700145a0 con 0x7fb27813f4a0 2023-10-30T13:51:24.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb26f7fe640 1 -- 172.21.15.148:0/4232497790 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb270014950 con 0x7fb27813f4a0 2023-10-30T13:51:24.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2780ca310 con 0x7fb27813f4a0 2023-10-30T13:51:24.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2781396d0 con 0x7fb27813f4a0 2023-10-30T13:51:24.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27804e170 con 0x7fb27813f4a0 2023-10-30T13:51:24.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb26f7fe640 1 -- 172.21.15.148:0/4232497790 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb270014ab0 con 0x7fb27813f4a0 2023-10-30T13:51:24.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb26f7fe640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2580664a0 0x7fb258068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:24.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.628+0000 7fb26f7fe640 1 -- 172.21.15.148:0/4232497790 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fb270084eb0 con 0x7fb27813f4a0 2023-10-30T13:51:24.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.632+0000 7fb276d9d640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2580664a0 0x7fb258068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:24.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.632+0000 7fb276d9d640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2580664a0 0x7fb258068960 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb2680af000 tx=0x7fb2680b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:24.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.632+0000 7fb26f7fe640 1 -- 172.21.15.148:0/4232497790 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb270051b80 con 0x7fb27813f4a0 2023-10-30T13:51:24.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.816+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb278160010 con 0x7fb27813f4a0 2023-10-30T13:51:24.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.816+0000 7fb26f7fe640 1 -- 172.21.15.148:0/4232497790 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+5744 (secure 0 0 0) 0x7fb2700551a0 con 0x7fb27813f4a0 2023-10-30T13:51:24.820 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:24.820 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:22.640524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:24.821 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 163 2023-10-30T13:51:24.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2580664a0 msgr2=0x7fb258068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.822 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2580664a0 0x7fb258068960 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb2680af000 tx=0x7fb2680b6040 comp rx=0 tx=0).stop 2023-10-30T13:51:24.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb27813f4a0 msgr2=0x7fb27813a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:24.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb27813f4a0 0x7fb27813a130 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fb270007e60 tx=0x7fb27000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 shutdown_connections 2023-10-30T13:51:24.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb2580664a0 0x7fb258068960 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb27813f4a0 0x7fb27813a130 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.823 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb2780c6f60 0x7fb278139bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 --2- 172.21.15.148:0/4232497790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780c6b80 0x7fb27813c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:24.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 >> 172.21.15.148:0/4232497790 conn(0x7fb2780c7910 msgr2=0x7fb278160800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:24.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 shutdown_connections 2023-10-30T13:51:24.824 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:24.820+0000 7fb27d046640 1 -- 172.21.15.148:0/4232497790 wait complete. 2023-10-30T13:51:24.841 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:26.842 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:26.990 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.988+0000 7f7d43f0f640 1 Processor -- start 2023-10-30T13:51:26.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.988+0000 7f7d43f0f640 1 -- start start 2023-10-30T13:51:26.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.988+0000 7f7d43f0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7d3c0c6b80 0x7f7d3c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:26.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.988+0000 7f7d43f0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7d3c0c6f60 0x7f7d3c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:26.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.988+0000 7f7d43f0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c0c0a50 con 0x7f7d3c0c8430 2023-10-30T13:51:26.993 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.988+0000 7f7d43f0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3c06f6e0 con 0x7f7d3c0c6b80 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.988+0000 7f7d43f0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3c06f9c0 con 0x7f7d3c0c6f60 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d41cab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7d3c0c6b80 0x7f7d3c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d41cab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f7d3c0c6b80 0x7f7d3c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54774/0 (socket says 172.21.15.148:54774) 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d41cab640 1 -- 172.21.15.148:0/2035383227 learned_addr learned my addr 172.21.15.148:0/2035383227 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d414aa640 1 --2- 172.21.15.148:0/2035383227 >> v2:172.21.15.150:3300/0 conn(0x7f7d3c0c6f60 0x7f7d3c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:26.994 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d41cab640 1 -- 172.21.15.148:0/2035383227 >> v2:172.21.15.150:3300/0 conn(0x7f7d3c0c6f60 msgr2=0x7f7d3c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:26.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d41cab640 1 --2- 172.21.15.148:0/2035383227 >> v2:172.21.15.150:3300/0 conn(0x7f7d3c0c6f60 0x7f7d3c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:26.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d41cab640 1 -- 172.21.15.148:0/2035383227 >> v1:172.21.15.148:6789/0 conn(0x7f7d3c0c8430 legacy=0x7f7d3c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:26.995 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d41cab640 1 -- 172.21.15.148:0/2035383227 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c06fca0 con 0x7f7d3c0c6b80 2023-10-30T13:51:26.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d41cab640 1 --2- 172.21.15.148:0/2035383227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6b80 0x7f7d3c0bfec0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c159b90 tx=0x7f7d3801e710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b048f52fca9131e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:26.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d40ca9640 1 -- 172.21.15.148:0/2035383227 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d380aa650 con 0x7f7d3c0c6b80 2023-10-30T13:51:26.996 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d40ca9640 1 -- 172.21.15.148:0/2035383227 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d380aa7b0 con 0x7f7d3c0c6b80 2023-10-30T13:51:26.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.992+0000 7f7d40ca9640 1 -- 172.21.15.148:0/2035383227 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d380aab40 con 0x7f7d3c0c6b80 2023-10-30T13:51:26.997 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 -- 172.21.15.148:0/2035383227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6b80 msgr2=0x7f7d3c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 --2- 172.21.15.148:0/2035383227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6b80 0x7f7d3c0bfec0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c159b90 tx=0x7f7d3801e710 comp rx=0 tx=0).stop 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 -- 172.21.15.148:0/2035383227 shutdown_connections 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 --2- 172.21.15.148:0/2035383227 >> v2:172.21.15.150:3300/0 conn(0x7f7d3c0c6f60 0x7f7d3c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:26.998 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 --2- 172.21.15.148:0/2035383227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6b80 0x7f7d3c0bfec0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:26.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 -- 172.21.15.148:0/2035383227 >> 172.21.15.148:0/2035383227 conn(0x7f7d3c0c7910 msgr2=0x7f7d3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:26.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 -- 172.21.15.148:0/2035383227 shutdown_connections 2023-10-30T13:51:26.999 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 -- 172.21.15.148:0/2035383227 wait complete. 2023-10-30T13:51:27.001 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 Processor -- start 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 -- start start 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d3c0c6b80 0x7f7d3c0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6f60 0x7f7d3c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d3c0c8430 0x7f7d3c152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.002 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c157750 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c13e690 con 0x7f7d3c0c6b80 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d43f0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c13e0d0 con 0x7f7d3c0c8430 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d414aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6f60 0x7f7d3c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d414aa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6f60 0x7f7d3c0c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54788/0 (socket says 172.21.15.148:54788) 2023-10-30T13:51:27.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d414aa640 1 -- 172.21.15.148:0/1489541097 learned_addr learned my addr 172.21.15.148:0/1489541097 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d41cab640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d3c0c6b80 0x7f7d3c0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d424ac640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d3c0c8430 0x7f7d3c152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d414aa640 1 -- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d3c0c8430 msgr2=0x7f7d3c152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d414aa640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d3c0c8430 0x7f7d3c152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d414aa640 1 -- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d3c0c6b80 msgr2=0x7f7d3c0c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d414aa640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d3c0c6b80 0x7f7d3c0c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:26.996+0000 7f7d414aa640 1 -- 172.21.15.148:0/1489541097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d380094d0 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d414aa640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6f60 0x7f7d3c0c9150 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00d800 tx=0x7f7d2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:27.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d327fc640 1 -- 172.21.15.148:0/1489541097 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00b6c0 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d327fc640 1 -- 172.21.15.148:0/1489541097 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c00b820 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d327fc640 1 -- 172.21.15.148:0/1489541097 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00bb90 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c13e3b0 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c135b40 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.006 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c04e170 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d327fc640 1 -- 172.21.15.148:0/1489541097 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d2c00bcf0 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d327fc640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d200664a0 0x7f7d20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:27.007 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d327fc640 1 -- 172.21.15.148:0/1489541097 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f7d2c084c70 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.000+0000 7f7d41cab640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d200664a0 0x7f7d20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:27.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.004+0000 7f7d41cab640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d200664a0 0x7f7d20068960 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7d380b1000 tx=0x7f7d38008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:27.008 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.004+0000 7f7d327fc640 1 -- 172.21.15.148:0/1489541097 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d2c051820 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.188+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d3c0c1480 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.192 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d327fc640 1 -- 172.21.15.148:0/1489541097 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+5744 (secure 0 0 0) 0x7f7d2c054e40 con 0x7f7d3c0c6f60 2023-10-30T13:51:27.193 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:27.193 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:22.640524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:27.193 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 163 2023-10-30T13:51:27.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d200664a0 msgr2=0x7f7d20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d200664a0 0x7f7d20068960 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7d380b1000 tx=0x7f7d38008040 comp rx=0 tx=0).stop 2023-10-30T13:51:27.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6f60 msgr2=0x7f7d3c0c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:27.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6f60 0x7f7d3c0c9150 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00d800 tx=0x7f7d2c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 shutdown_connections 2023-10-30T13:51:27.195 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f7d200664a0 0x7f7d20068960 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d3c0c6f60 0x7f7d3c0c9150 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f7d3c0c8430 0x7f7d3c152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 --2- 172.21.15.148:0/1489541097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d3c0c6b80 0x7f7d3c0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:27.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 >> 172.21.15.148:0/1489541097 conn(0x7f7d3c0c7910 msgr2=0x7f7d3c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:27.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 shutdown_connections 2023-10-30T13:51:27.196 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:27.192+0000 7f7d43f0f640 1 -- 172.21.15.148:0/1489541097 wait complete. 2023-10-30T13:51:27.213 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:51:29.215 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:29.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.352+0000 7fd9dde46640 1 Processor -- start 2023-10-30T13:51:29.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.352+0000 7fd9dde46640 1 -- start start 2023-10-30T13:51:29.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9dde46640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd9d815d6f0 0x7fd9d815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9dde46640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d80cb800 con 0x7fd9d80cf160 2023-10-30T13:51:29.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9dde46640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d80cb330 con 0x7fd9d81538a0 2023-10-30T13:51:29.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9dde46640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d80cad70 con 0x7fd9d815d6f0 2023-10-30T13:51:29.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9d6ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd9d815d6f0 0x7fd9d815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9d6ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd9d815d6f0 0x7fd9d815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54796/0 (socket says 172.21.15.148:54796) 2023-10-30T13:51:29.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9d7fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd9d80cf160 0x7fd9d8164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47342/0 (socket says 172.21.15.148:47342) 2023-10-30T13:51:29.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9d7fff640 1 -- 172.21.15.148:0/525222537 learned_addr learned my addr 172.21.15.148:0/525222537 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:51:29.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9d6ffd640 1 -- 172.21.15.148:0/525222537 >> v1:172.21.15.150:6789/0 conn(0x7fd9d81538a0 legacy=0x7fd9d80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9d6ffd640 1 -- 172.21.15.148:0/525222537 >> v1:172.21.15.148:6789/0 conn(0x7fd9d80cf160 legacy=0x7fd9d8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.356+0000 7fd9d6ffd640 1 -- 172.21.15.148:0/525222537 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d80cb050 con 0x7fd9d815d6f0 2023-10-30T13:51:29.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9d6ffd640 1 --2- 172.21.15.148:0/525222537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d815d6f0 0x7fd9d815a090 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0097c0 tx=0x7fd9cc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c47267d1543b770 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:29.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9d67fc640 1 -- 172.21.15.148:0/525222537 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc0a8970 con 0x7fd9d815d6f0 2023-10-30T13:51:29.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9d67fc640 1 -- 172.21.15.148:0/525222537 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc0a8ad0 con 0x7fd9d815d6f0 2023-10-30T13:51:29.361 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9d67fc640 1 -- 172.21.15.148:0/525222537 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc0a8e60 con 0x7fd9d815d6f0 2023-10-30T13:51:29.362 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9dde46640 1 -- 172.21.15.148:0/525222537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d815d6f0 msgr2=0x7fd9d815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9dde46640 1 --2- 172.21.15.148:0/525222537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d815d6f0 0x7fd9d815a090 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0097c0 tx=0x7fd9cc01ec70 comp rx=0 tx=0).stop 2023-10-30T13:51:29.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9dde46640 1 -- 172.21.15.148:0/525222537 shutdown_connections 2023-10-30T13:51:29.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9dde46640 1 --2- 172.21.15.148:0/525222537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d815d6f0 0x7fd9d815a090 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9dde46640 1 -- 172.21.15.148:0/525222537 >> 172.21.15.148:0/525222537 conn(0x7fd9d806bfe0 msgr2=0x7fd9d806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:29.363 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9dde46640 1 -- 172.21.15.148:0/525222537 shutdown_connections 2023-10-30T13:51:29.364 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9dde46640 1 -- 172.21.15.148:0/525222537 wait complete. 2023-10-30T13:51:29.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9dde46640 1 Processor -- start 2023-10-30T13:51:29.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.360+0000 7fd9dde46640 1 -- start start 2023-10-30T13:51:29.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9dde46640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d80cf160 0x7fd9d81e3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9dde46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9d81538a0 0x7fd9d81592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9dde46640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9d815d6f0 0x7fd9d80c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9dde46640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80c8ce0 con 0x7fd9d80cf160 2023-10-30T13:51:29.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9dde46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80c8310 con 0x7fd9d81538a0 2023-10-30T13:51:29.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9dde46640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d80c7060 con 0x7fd9d815d6f0 2023-10-30T13:51:29.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d80cf160 0x7fd9d81e3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d80cf160 0x7fd9d81e3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:54800/0 (socket says 172.21.15.148:54800) 2023-10-30T13:51:29.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d77fe640 1 -- 172.21.15.148:0/1536531530 learned_addr learned my addr 172.21.15.148:0/1536531530 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:29.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d6ffd640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9d81538a0 0x7fd9d81592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d7fff640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9d815d6f0 0x7fd9d80c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d77fe640 1 -- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9d815d6f0 msgr2=0x7fd9d80c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d77fe640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9d815d6f0 0x7fd9d80c87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d77fe640 1 -- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9d81538a0 msgr2=0x7fd9d81592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d77fe640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9d81538a0 0x7fd9d81592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d77fe640 1 -- 172.21.15.148:0/1536531530 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc009420 con 0x7fd9d80cf160 2023-10-30T13:51:29.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9d77fe640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d80cf160 0x7fd9d81e3750 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800c500 tx=0x7fd9c800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:29.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9bffff640 1 -- 172.21.15.148:0/1536531530 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8005220 con 0x7fd9d80cf160 2023-10-30T13:51:29.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9bffff640 1 -- 172.21.15.148:0/1536531530 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c8005380 con 0x7fd9d80cf160 2023-10-30T13:51:29.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9bffff640 1 -- 172.21.15.148:0/1536531530 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8007270 con 0x7fd9d80cf160 2023-10-30T13:51:29.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d80c7d00 con 0x7fd9d80cf160 2023-10-30T13:51:29.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d80c6aa0 con 0x7fd9d80cf160 2023-10-30T13:51:29.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d804e170 con 0x7fd9d80cf160 2023-10-30T13:51:29.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.364+0000 7fd9bffff640 1 -- 172.21.15.148:0/1536531530 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd9c80073d0 con 0x7fd9d80cf160 2023-10-30T13:51:29.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.368+0000 7fd9bffff640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9c40664a0 0x7fd9c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:29.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.368+0000 7fd9bffff640 1 -- 172.21.15.148:0/1536531530 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fd9c8083ee0 con 0x7fd9d80cf160 2023-10-30T13:51:29.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.368+0000 7fd9d6ffd640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9c40664a0 0x7fd9c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:29.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.368+0000 7fd9d6ffd640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9c40664a0 0x7fd9c4068960 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0b2000 tx=0x7fd9cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:29.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.372+0000 7fd9bffff640 1 -- 172.21.15.148:0/1536531530 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd9c8050bb0 con 0x7fd9d80cf160 2023-10-30T13:51:29.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.548+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9d80d0090 con 0x7fd9d80cf160 2023-10-30T13:51:29.552 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9bffff640 1 -- 172.21.15.148:0/1536531530 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5738 (secure 0 0 0) 0x7fd9c80541d0 con 0x7fd9d80cf160 2023-10-30T13:51:29.553 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:29.553 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:28.940082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:29.553 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 164 2023-10-30T13:51:29.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9c40664a0 msgr2=0x7fd9c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9c40664a0 0x7fd9c4068960 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0b2000 tx=0x7fd9cc008040 comp rx=0 tx=0).stop 2023-10-30T13:51:29.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d80cf160 msgr2=0x7fd9d81e3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:29.555 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d80cf160 0x7fd9d81e3750 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800c500 tx=0x7fd9c800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 shutdown_connections 2023-10-30T13:51:29.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9c40664a0 0x7fd9c4068960 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd9d815d6f0 0x7fd9d80c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9d81538a0 0x7fd9d81592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.556 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 --2- 172.21.15.148:0/1536531530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9d80cf160 0x7fd9d81e3750 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:29.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 >> 172.21.15.148:0/1536531530 conn(0x7fd9d806bfe0 msgr2=0x7fd9d80ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:29.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 shutdown_connections 2023-10-30T13:51:29.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:29.552+0000 7fd9dde46640 1 -- 172.21.15.148:0/1536531530 wait complete. 2023-10-30T13:51:29.573 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7812 2023-10-30T13:51:29.574 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.c 2023-10-30T13:51:34.111 INFO:tasks.ceph.mds.a.smithi148.stderr:2023-10-30T13:51:34.108+0000 7f1f78c90640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:51:34.111 INFO:tasks.ceph.mds.a.smithi148.stderr:2023-10-30T13:51:34.108+0000 7f1f78c90640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:51:34.947 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:34.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:51:35.575 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:51:35.575 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:51:35.575 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:51:35.580 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:51:35.580 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:35.670 INFO:tasks.ceph.mds.c.smithi148.stdout:starting mds.c at 2023-10-30T13:51:35.723 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.720+0000 7f88b68c9640 1 Processor -- start 2023-10-30T13:51:35.724 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.720+0000 7f88b68c9640 1 -- start start 2023-10-30T13:51:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88b68c9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f88b01538a0 0x7f88b015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88b68c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f88b00ca0e0 0x7f88b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88b68c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00cb800 con 0x7f88b00cf160 2023-10-30T13:51:35.726 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88b68c9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b015dc30 con 0x7f88b01538a0 2023-10-30T13:51:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88b68c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b00cb330 con 0x7f88b00ca0e0 2023-10-30T13:51:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88affff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f88b01538a0 0x7f88b015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88b4e66640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f88b00cf160 0x7f88b00cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53906/0 (socket says 172.21.15.148:53906) 2023-10-30T13:51:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88b4e66640 1 -- 172.21.15.148:0/3095066459 learned_addr learned my addr 172.21.15.148:0/3095066459 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:51:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88af7fe640 1 --2- 172.21.15.148:0/3095066459 >> v2:172.21.15.150:3300/0 conn(0x7f88b00ca0e0 0x7f88b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:35.727 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88aeffd640 1 -- 172.21.15.148:0/3095066459 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670744498 0 0) 0x7f88b00cb800 con 0x7f88b00cf160 2023-10-30T13:51:35.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88aeffd640 1 -- 172.21.15.148:0/3095066459 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8898003530 con 0x7f88b00cf160 2023-10-30T13:51:35.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88affff640 1 -- 172.21.15.148:0/3095066459 >> v2:172.21.15.150:3300/0 conn(0x7f88b00ca0e0 msgr2=0x7f88b015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:35.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88affff640 1 --2- 172.21.15.148:0/3095066459 >> v2:172.21.15.150:3300/0 conn(0x7f88b00ca0e0 0x7f88b015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:35.728 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88affff640 1 -- 172.21.15.148:0/3095066459 >> v1:172.21.15.148:6789/0 conn(0x7f88b00cf160 legacy=0x7f88b00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:35.729 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.724+0000 7f88affff640 1 -- 172.21.15.148:0/3095066459 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b00cad70 con 0x7f88b01538a0 2023-10-30T13:51:35.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88affff640 1 --2- 172.21.15.148:0/3095066459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 0x7f88b015d6f0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f88b00b30a0 tx=0x7f88a001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92b881aed3e87a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:35.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88aeffd640 1 -- 172.21.15.148:0/3095066459 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a00a8950 con 0x7f88b01538a0 2023-10-30T13:51:35.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88aeffd640 1 -- 172.21.15.148:0/3095066459 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a00a8ab0 con 0x7f88b01538a0 2023-10-30T13:51:35.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88aeffd640 1 -- 172.21.15.148:0/3095066459 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a00a8e40 con 0x7f88b01538a0 2023-10-30T13:51:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 -- 172.21.15.148:0/3095066459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 msgr2=0x7f88b015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 --2- 172.21.15.148:0/3095066459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 0x7f88b015d6f0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f88b00b30a0 tx=0x7f88a001ec50 comp rx=0 tx=0).stop 2023-10-30T13:51:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 -- 172.21.15.148:0/3095066459 shutdown_connections 2023-10-30T13:51:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 --2- 172.21.15.148:0/3095066459 >> v2:172.21.15.150:3300/0 conn(0x7f88b00ca0e0 0x7f88b015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:35.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 --2- 172.21.15.148:0/3095066459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 0x7f88b015d6f0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:35.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 -- 172.21.15.148:0/3095066459 >> 172.21.15.148:0/3095066459 conn(0x7f88b006bfe0 msgr2=0x7f88b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:35.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 -- 172.21.15.148:0/3095066459 shutdown_connections 2023-10-30T13:51:35.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 -- 172.21.15.148:0/3095066459 wait complete. 2023-10-30T13:51:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 Processor -- start 2023-10-30T13:51:35.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 -- start start 2023-10-30T13:51:35.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88b00ca0e0 0x7f88b0158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:35.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88b00cf160 0x7f88b01ef2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:35.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 0x7f88b01ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b01f3400 con 0x7f88b01538a0 2023-10-30T13:51:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b01584e0 con 0x7f88b00ca0e0 2023-10-30T13:51:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b68c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b01f27f0 con 0x7f88b00cf160 2023-10-30T13:51:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.728+0000 7f88b4e66640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 0x7f88b01ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:35.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b4e66640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 0x7f88b01ef820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58050/0 (socket says 172.21.15.148:58050) 2023-10-30T13:51:35.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b4e66640 1 -- 172.21.15.148:0/3604547745 learned_addr learned my addr 172.21.15.148:0/3604547745 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:35.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88affff640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88b00ca0e0 0x7f88b0158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:35.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88af7fe640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88b00cf160 0x7f88b01ef2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:35.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b4e66640 1 -- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88b00cf160 msgr2=0x7f88b01ef2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:35.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b4e66640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88b00cf160 0x7f88b01ef2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:35.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b4e66640 1 -- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88b00ca0e0 msgr2=0x7f88b0158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:35.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b4e66640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88b00ca0e0 0x7f88b0158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:35.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b4e66640 1 -- 172.21.15.148:0/3604547745 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a0009420 con 0x7f88b01538a0 2023-10-30T13:51:35.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88affff640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88b00ca0e0 0x7f88b0158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:35.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b4e66640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 0x7f88b01ef820 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f88a800cad0 tx=0x7f88a800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:35.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88acff9640 1 -- 172.21.15.148:0/3604547745 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a800ac30 con 0x7f88b01538a0 2023-10-30T13:51:35.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88acff9640 1 -- 172.21.15.148:0/3604547745 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a800ad90 con 0x7f88b01538a0 2023-10-30T13:51:35.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b01f2ad0 con 0x7f88b01538a0 2023-10-30T13:51:35.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88acff9640 1 -- 172.21.15.148:0/3604547745 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8012620 con 0x7f88b01538a0 2023-10-30T13:51:35.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b01f3090 con 0x7f88b01538a0 2023-10-30T13:51:35.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b00b2a50 con 0x7f88b01538a0 2023-10-30T13:51:35.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88acff9640 1 -- 172.21.15.148:0/3604547745 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88a8007070 con 0x7f88b01538a0 2023-10-30T13:51:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88acff9640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f889c0664a0 0x7f889c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.732+0000 7f88acff9640 1 -- 172.21.15.148:0/3604547745 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f88a8084610 con 0x7f88b01538a0 2023-10-30T13:51:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.736+0000 7f88affff640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f889c0664a0 0x7f889c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.736+0000 7f88affff640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f889c0664a0 0x7f889c068960 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f88a00b2000 tx=0x7f88a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:35.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.740+0000 7f88acff9640 1 -- 172.21.15.148:0/3604547745 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88a8051270 con 0x7f88b01538a0 2023-10-30T13:51:35.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.920+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b004e170 con 0x7f88b01538a0 2023-10-30T13:51:35.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.920+0000 7f88acff9640 1 -- 172.21.15.148:0/3604547745 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5738 (secure 0 0 0) 0x7f88a8017090 con 0x7f88b01538a0 2023-10-30T13:51:35.923 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:35.924 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:28.940082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:35.924 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 164 2023-10-30T13:51:35.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f889c0664a0 msgr2=0x7f889c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:35.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f889c0664a0 0x7f889c068960 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f88a00b2000 tx=0x7f88a0008040 comp rx=0 tx=0).stop 2023-10-30T13:51:35.925 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 msgr2=0x7f88b01ef820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:35.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 0x7f88b01ef820 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f88a800cad0 tx=0x7f88a800cf90 comp rx=0 tx=0).stop 2023-10-30T13:51:35.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 shutdown_connections 2023-10-30T13:51:35.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f889c0664a0 0x7f889c068960 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:35.926 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88b01538a0 0x7f88b01ef820 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:35.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f88b00cf160 0x7f88b01ef2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:35.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 --2- 172.21.15.148:0/3604547745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88b00ca0e0 0x7f88b0158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:35.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 >> 172.21.15.148:0/3604547745 conn(0x7f88b006bfe0 msgr2=0x7f88b0151900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:35.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 shutdown_connections 2023-10-30T13:51:35.927 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:35.924+0000 7f88b68c9640 1 -- 172.21.15.148:0/3604547745 wait complete. 2023-10-30T13:51:35.944 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:51:37.944 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:51:38.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b6dd0f640 1 Processor -- start 2023-10-30T13:51:38.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b6dd0f640 1 -- start start 2023-10-30T13:51:38.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b6dd0f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6b681538a0 0x7f6b6815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b6dd0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6b680ca0e0 0x7f6b6815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b6dd0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b680cb800 con 0x7f6b680cf160 2023-10-30T13:51:38.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b6dd0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b6815dc30 con 0x7f6b680ca0e0 2023-10-30T13:51:38.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b6dd0f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b680cb330 con 0x7f6b681538a0 2023-10-30T13:51:38.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b66ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6b680ca0e0 0x7f6b6815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b66ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6b680ca0e0 0x7f6b6815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58062/0 (socket says 172.21.15.148:58062) 2023-10-30T13:51:38.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b66ffd640 1 -- 172.21.15.148:0/605945064 learned_addr learned my addr 172.21.15.148:0/605945064 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:38.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.100+0000 7f6b677fe640 1 --2- 172.21.15.148:0/605945064 >> v2:172.21.15.150:3300/0 conn(0x7f6b681538a0 0x7f6b6815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b66ffd640 1 -- 172.21.15.148:0/605945064 >> v2:172.21.15.150:3300/0 conn(0x7f6b681538a0 msgr2=0x7f6b6815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:51:38.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b66ffd640 1 --2- 172.21.15.148:0/605945064 >> v2:172.21.15.150:3300/0 conn(0x7f6b681538a0 0x7f6b6815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b66ffd640 1 -- 172.21.15.148:0/605945064 >> v1:172.21.15.150:6789/0 conn(0x7f6b680cf160 legacy=0x7f6b680cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b66ffd640 1 -- 172.21.15.148:0/605945064 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b680cad70 con 0x7f6b680ca0e0 2023-10-30T13:51:38.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b677fe640 1 --2- 172.21.15.148:0/605945064 >> v2:172.21.15.150:3300/0 conn(0x7f6b681538a0 0x7f6b6815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:38.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b66ffd640 1 --2- 172.21.15.148:0/605945064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680ca0e0 0x7f6b6815a090 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f6b680cfd10 tx=0x7f6b5c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a4d6a70780eadb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:38.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b667fc640 1 -- 172.21.15.148:0/605945064 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c0a89f0 con 0x7f6b680ca0e0 2023-10-30T13:51:38.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b667fc640 1 -- 172.21.15.148:0/605945064 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c0a8b50 con 0x7f6b680ca0e0 2023-10-30T13:51:38.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b667fc640 1 -- 172.21.15.148:0/605945064 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c0b0440 con 0x7f6b680ca0e0 2023-10-30T13:51:38.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/605945064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680ca0e0 msgr2=0x7f6b6815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.104+0000 7f6b6dd0f640 1 --2- 172.21.15.148:0/605945064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680ca0e0 0x7f6b6815a090 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f6b680cfd10 tx=0x7f6b5c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/605945064 shutdown_connections 2023-10-30T13:51:38.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 --2- 172.21.15.148:0/605945064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680ca0e0 0x7f6b6815a090 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 --2- 172.21.15.148:0/605945064 >> v2:172.21.15.150:3300/0 conn(0x7f6b681538a0 0x7f6b6815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/605945064 >> 172.21.15.148:0/605945064 conn(0x7f6b6806bfe0 msgr2=0x7f6b6806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:38.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/605945064 shutdown_connections 2023-10-30T13:51:38.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/605945064 wait complete. 2023-10-30T13:51:38.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 Processor -- start 2023-10-30T13:51:38.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 -- start start 2023-10-30T13:51:38.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b680ca0e0 0x7f6b680b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680cf160 0x7f6b681e3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b681538a0 0x7f6b680cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b680b4450 con 0x7f6b680cf160 2023-10-30T13:51:38.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b680cd2b0 con 0x7f6b680ca0e0 2023-10-30T13:51:38.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b6dd0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b68157860 con 0x7f6b681538a0 2023-10-30T13:51:38.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b66ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680cf160 0x7f6b681e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b66ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680cf160 0x7f6b681e3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:58078/0 (socket says 172.21.15.148:58078) 2023-10-30T13:51:38.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b66ffd640 1 -- 172.21.15.148:0/1107590746 learned_addr learned my addr 172.21.15.148:0/1107590746 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:51:38.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b677fe640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b680ca0e0 0x7f6b680b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b67fff640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b681538a0 0x7f6b680cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b66ffd640 1 -- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b681538a0 msgr2=0x7f6b680cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b66ffd640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b681538a0 0x7f6b680cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b66ffd640 1 -- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b680ca0e0 msgr2=0x7f6b680b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b66ffd640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b680ca0e0 0x7f6b680b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b66ffd640 1 -- 172.21.15.148:0/1107590746 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c0094d0 con 0x7f6b680cf160 2023-10-30T13:51:38.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b677fe640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b680ca0e0 0x7f6b680b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:51:38.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.108+0000 7f6b66ffd640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680cf160 0x7f6b681e3640 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0b3040 tx=0x7f6b5c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:38.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b4ffff640 1 -- 172.21.15.148:0/1107590746 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c0ba600 con 0x7f6b680cf160 2023-10-30T13:51:38.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b4ffff640 1 -- 172.21.15.148:0/1107590746 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c0ba760 con 0x7f6b680cf160 2023-10-30T13:51:38.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b68157b00 con 0x7f6b680cf160 2023-10-30T13:51:38.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b4ffff640 1 -- 172.21.15.148:0/1107590746 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c0bab10 con 0x7f6b680cf160 2023-10-30T13:51:38.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b680b3e00 con 0x7f6b680cf160 2023-10-30T13:51:38.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b680b2a50 con 0x7f6b680cf160 2023-10-30T13:51:38.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b4ffff640 1 -- 172.21.15.148:0/1107590746 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b5c0bac70 con 0x7f6b680cf160 2023-10-30T13:51:38.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b4ffff640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6b540664a0 0x7f6b54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:51:38.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b4ffff640 1 -- 172.21.15.148:0/1107590746 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f6b5c122c90 con 0x7f6b680cf160 2023-10-30T13:51:38.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.112+0000 7f6b677fe640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6b540664a0 0x7f6b54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:51:38.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.116+0000 7f6b677fe640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6b540664a0 0x7f6b54068960 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800b990 tx=0x7f6b58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:51:38.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.116+0000 7f6b4ffff640 1 -- 172.21.15.148:0/1107590746 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b5c0ef8d0 con 0x7f6b680cf160 2023-10-30T13:51:38.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b6804e170 con 0x7f6b680cf160 2023-10-30T13:51:38.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b4ffff640 1 -- 172.21.15.148:0/1107590746 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+6533 (secure 0 0 0) 0x7f6b5c0f2ef0 con 0x7f6b680cf160 2023-10-30T13:51:38.303 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:51:38.303 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2057257086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2057257086},{"type":"v1","addr":"172.21.15.148:6837","nonce":2057257086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:28.940082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:51:38.304 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 165 2023-10-30T13:51:38.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6b540664a0 msgr2=0x7f6b54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6b540664a0 0x7f6b54068960 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800b990 tx=0x7f6b58008040 comp rx=0 tx=0).stop 2023-10-30T13:51:38.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680cf160 msgr2=0x7f6b681e3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:51:38.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680cf160 0x7f6b681e3640 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c0b3040 tx=0x7f6b5c01ba90 comp rx=0 tx=0).stop 2023-10-30T13:51:38.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 shutdown_connections 2023-10-30T13:51:38.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6b540664a0 0x7f6b54068960 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b680cf160 0x7f6b681e3640 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6b681538a0 0x7f6b680cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 --2- 172.21.15.148:0/1107590746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b680ca0e0 0x7f6b680b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:51:38.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.300+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 >> 172.21.15.148:0/1107590746 conn(0x7f6b6806bfe0 msgr2=0x7f6b68148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:51:38.306 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.304+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 shutdown_connections 2023-10-30T13:51:38.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:51:38.304+0000 7f6b6dd0f640 1 -- 172.21.15.148:0/1107590746 wait complete. 2023-10-30T13:51:38.324 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:51:38.324 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 99 secs before thrashing 2023-10-30T13:51:38.900 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 10% (6/59) Updating files: 11% (7/59) Updating files: 13% (8/59) Updating files: 15% (9/59) Updating files: 16% (10/59) Updating files: 18% (11/59) Updating files: 20% (12/59) Updating files: 22% (13/59) Updating files: 23% (14/59) Updating files: 25% (15/59) Updating files: 27% (16/59) Updating files: 28% (17/59) Updating files: 30% (18/59) Updating files: 32% (19/59) Updating files: 33% (20/59) Updating files: 35% (21/59) Updating files: 37% (22/59) Updating files: 38% (23/59) Updating files: 40% (24/59) Updating files: 42% (25/59) Updating files: 44% (26/59) Updating files: 45% (27/59) Updating files: 47% (28/59) Updating files: 49% (29/59) Updating files: 50% (30/59) Updating files: 52% (31/59) Updating files: 54% (32/59) Updating files: 55% (33/59) Updating files: 57% (34/59) Updating files: 59% (35/59) Updating files: 61% (36/59) Updating files: 62% (37/59) Updating files: 64% (38/59) Updating files: 66% (39/59) Updating files: 67% (40/59) Updating files: 69% (41/59) Updating files: 71% (42/59) Updating files: 72% (43/59) Updating files: 74% (44/59) Updating files: 76% (45/59) Updating files: 77% (46/59) Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-10-30T13:51:40.649 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at c51c8f9d80f 0.87 2023-10-30T13:51:40.652 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.87 2023-10-30T13:51:40.693 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 88 -eq 48 ] 2023-10-30T13:51:40.693 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.88 2023-10-30T13:51:40.693 INFO:tasks.workunit.client.0.smithi148.stdout:v0.88 2023-10-30T13:51:40.694 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.88 2023-10-30T13:51:40.694 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.88 2023-10-30T13:51:59.948 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 21% (72/335) Updating files: 22% (74/335) Updating files: 23% (78/335) Updating files: 24% (81/335) Updating files: 25% (84/335) Updating files: 26% (88/335) Updating files: 27% (91/335) Updating files: 28% (94/335) Updating files: 29% (98/335) Updating files: 30% (101/335) Updating files: 31% (104/335) Updating files: 32% (108/335) Updating files: 33% (111/335) Updating files: 34% (114/335) Updating files: 35% (118/335) Updating files: 35% (119/335) Updating files: 36% (121/335) Updating files: 37% (124/335) Updating files: 38% (128/335) Updating files: 39% (131/335) Updating files: 40% (134/335) Updating files: 41% (138/335) Updating files: 42% (141/335) Updating files: 42% (144/335) Updating files: 43% (145/335) Updating files: 44% (148/335) Updating files: 45% (151/335) Updating files: 46% (155/335) Updating files: 47% (158/335) Updating files: 48% (161/335) Updating files: 49% (165/335) Updating files: 50% (168/335) Updating files: 51% (171/335) Updating files: 52% (175/335) Updating files: 53% (178/335) Updating files: 54% (181/335) Updating files: 55% (185/335) Updating files: 56% (188/335) Updating files: 57% (191/335) Updating files: 58% (195/335) Updating files: 59% (198/335) Updating files: 60% (201/335) Updating files: 61% (205/335) Updating files: 62% (208/335) Updating files: 62% (211/335) Updating files: 63% (212/335) Updating files: 64% (215/335) Updating files: 65% (218/335) Updating files: 66% (222/335) Updating files: 67% (225/335) Updating files: 68% (228/335) Updating files: 69% (232/335) Updating files: 70% (235/335) Updating files: 71% (238/335) Updating files: 72% (242/335) Updating files: 73% (245/335) Updating files: 74% (248/335) Updating files: 74% (249/335) Updating files: 75% (252/335) Updating files: 76% (255/335) Updating files: 77% (258/335) Updating files: 78% (262/335) Updating files: 78% (264/335) Updating files: 79% (265/335) Updating files: 80% (268/335) Updating files: 81% (272/335) Updating files: 82% (275/335) Updating files: 83% (279/335) Updating files: 84% (282/335) Updating files: 85% (285/335) Updating files: 86% (289/335) Updating files: 87% (292/335) Updating files: 88% (295/335) Updating files: 89% (299/335) Updating files: 90% (302/335) Updating files: 91% (305/335) Updating files: 92% (309/335) Updating files: 93% (312/335) Updating files: 94% (315/335) Updating files: 95% (319/335) Updating files: 96% (322/335) Updating files: 97% (325/335) Updating files: 98% (329/335) Updating files: 99% (332/335) Updating files: 100% (335/335) Updating files: 100% (335/335), done. 2023-10-30T13:51:59.979 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 4be687bf448 0.88 2023-10-30T13:51:59.986 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.88 2023-10-30T13:52:00.523 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 89 -eq 48 ] 2023-10-30T13:52:00.523 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.89 2023-10-30T13:52:00.523 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.89 2023-10-30T13:52:00.523 INFO:tasks.workunit.client.0.smithi148.stdout:v0.89 2023-10-30T13:52:00.524 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.89 2023-10-30T13:52:05.042 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:05.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:09.552 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 10% (20/197) Updating files: 11% (22/197) Updating files: 12% (24/197) Updating files: 13% (26/197) Updating files: 14% (28/197) Updating files: 15% (30/197) Updating files: 16% (32/197) Updating files: 17% (34/197) Updating files: 18% (36/197) Updating files: 19% (38/197) Updating files: 20% (40/197) Updating files: 21% (42/197) Updating files: 22% (44/197) Updating files: 23% (46/197) Updating files: 24% (48/197) Updating files: 25% (50/197) Updating files: 26% (52/197) Updating files: 27% (54/197) Updating files: 28% (56/197) Updating files: 29% (58/197) Updating files: 30% (60/197) Updating files: 31% (62/197) Updating files: 32% (64/197) Updating files: 33% (66/197) Updating files: 34% (67/197) Updating files: 35% (69/197) Updating files: 36% (71/197) Updating files: 37% (73/197) Updating files: 38% (75/197) Updating files: 39% (77/197) Updating files: 40% (79/197) Updating files: 41% (81/197) Updating files: 42% (83/197) Updating files: 43% (85/197) Updating files: 44% (87/197) Updating files: 45% (89/197) Updating files: 46% (91/197) Updating files: 47% (93/197) Updating files: 48% (95/197) Updating files: 49% (97/197) Updating files: 50% (99/197) Updating files: 51% (101/197) Updating files: 52% (103/197) Updating files: 53% (105/197) Updating files: 54% (107/197) Updating files: 55% (109/197) Updating files: 56% (111/197) Updating files: 57% (113/197) Updating files: 58% (115/197) Updating files: 59% (117/197) Updating files: 60% (119/197) Updating files: 61% (121/197) Updating files: 62% (123/197) Updating files: 63% (125/197) Updating files: 64% (127/197) Updating files: 65% (129/197) Updating files: 66% (131/197) Updating files: 67% (132/197) Updating files: 68% (134/197) Updating files: 69% (136/197) Updating files: 70% (138/197) Updating files: 71% (140/197) Updating files: 72% (142/197) Updating files: 73% (144/197) Updating files: 73% (145/197) Updating files: 74% (146/197) Updating files: 75% (148/197) Updating files: 76% (150/197) Updating files: 77% (152/197) Updating files: 78% (154/197) Updating files: 79% (156/197) Updating files: 80% (158/197) Updating files: 81% (160/197) Updating files: 82% (162/197) Updating files: 83% (164/197) Updating files: 84% (166/197) Updating files: 85% (168/197) Updating files: 86% (170/197) Updating files: 87% (172/197) Updating files: 88% (174/197) Updating files: 89% (176/197) Updating files: 90% (178/197) Updating files: 91% (180/197) Updating files: 92% (182/197) Updating files: 93% (184/197) Updating files: 94% (186/197) Updating files: 95% (188/197) Updating files: 96% (190/197) Updating files: 97% (192/197) Updating files: 98% (194/197) Updating files: 98% (195/197) Updating files: 99% (196/197) Updating files: 100% (197/197) Updating files: 100% (197/197), done. 2023-10-30T13:52:09.595 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 68fdc0f68e6 0.89 2023-10-30T13:52:09.601 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.89 2023-10-30T13:52:10.097 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 90 -eq 48 ] 2023-10-30T13:52:10.097 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.90 2023-10-30T13:52:10.098 INFO:tasks.workunit.client.0.smithi148.stdout:v0.90 2023-10-30T13:52:10.098 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo v0.90 2023-10-30T13:52:10.098 INFO:tasks.workunit.client.0.smithi148.stderr:+ git reset --hard v0.90 2023-10-30T13:52:12.928 INFO:tasks.workunit.client.0.smithi148.stderr:Updating files: 18% (50/265) Updating files: 19% (51/265) Updating files: 20% (53/265) Updating files: 21% (56/265) Updating files: 22% (59/265) Updating files: 23% (61/265) Updating files: 24% (64/265) Updating files: 25% (67/265) Updating files: 26% (69/265) Updating files: 27% (72/265) Updating files: 28% (75/265) Updating files: 29% (77/265) Updating files: 30% (80/265) Updating files: 31% (83/265) Updating files: 32% (85/265) Updating files: 33% (88/265) Updating files: 34% (91/265) Updating files: 35% (93/265) Updating files: 36% (96/265) Updating files: 37% (99/265) Updating files: 38% (101/265) Updating files: 39% (104/265) Updating files: 40% (106/265) Updating files: 41% (109/265) Updating files: 42% (112/265) Updating files: 43% (114/265) Updating files: 44% (117/265) Updating files: 45% (120/265) Updating files: 46% (122/265) Updating files: 47% (125/265) Updating files: 48% (128/265) Updating files: 49% (130/265) Updating files: 50% (133/265) Updating files: 51% (136/265) Updating files: 52% (138/265) Updating files: 53% (141/265) Updating files: 54% (144/265) Updating files: 55% (146/265) Updating files: 56% (149/265) Updating files: 57% (152/265) Updating files: 58% (154/265) Updating files: 59% (157/265) Updating files: 60% (159/265) Updating files: 61% (162/265) Updating files: 62% (165/265) Updating files: 63% (167/265) Updating files: 64% (170/265) Updating files: 65% (173/265) Updating files: 66% (175/265) Updating files: 67% (178/265) Updating files: 68% (181/265) Updating files: 69% (183/265) Updating files: 70% (186/265) Updating files: 71% (189/265) Updating files: 72% (191/265) Updating files: 73% (194/265) Updating files: 74% (197/265) Updating files: 75% (199/265) Updating files: 76% (202/265) Updating files: 77% (205/265) Updating files: 78% (207/265) Updating files: 79% (210/265) Updating files: 80% (212/265) Updating files: 81% (215/265) Updating files: 82% (218/265) Updating files: 83% (220/265) Updating files: 84% (223/265) Updating files: 85% (226/265) Updating files: 86% (228/265) Updating files: 87% (231/265) Updating files: 88% (234/265) Updating files: 89% (236/265) Updating files: 90% (239/265) Updating files: 91% (242/265) Updating files: 92% (244/265) Updating files: 93% (247/265) Updating files: 94% (250/265) Updating files: 95% (252/265) Updating files: 96% (255/265) Updating files: 97% (258/265) Updating files: 98% (260/265) Updating files: 99% (263/265) Updating files: 100% (265/265) Updating files: 100% (265/265), done. 2023-10-30T13:52:12.977 INFO:tasks.workunit.client.0.smithi148.stdout:HEAD is now at 08bd1e1eee3 0.90 2023-10-30T13:52:12.983 INFO:tasks.workunit.client.0.smithi148.stderr:+ mkdir .snap/v0.90 2023-10-30T13:52:13.802 INFO:tasks.workunit.client.0.smithi148.stdout:checking v0.1 2023-10-30T13:52:13.803 INFO:tasks.workunit.client.0.smithi148.stderr:+ [ 1 -eq 48 ] 2023-10-30T13:52:13.803 INFO:tasks.workunit.client.0.smithi148.stderr:+ ver=v0.1 2023-10-30T13:52:13.803 INFO:tasks.workunit.client.0.smithi148.stderr:+ echo checking v0.1 2023-10-30T13:52:13.803 INFO:tasks.workunit.client.0.smithi148.stderr:+ cd .snap/v0.1 2023-10-30T13:52:13.826 INFO:tasks.workunit.client.0.smithi148.stderr:+ git diff --exit-code 2023-10-30T13:52:20.194 INFO:tasks.workunit.client.0.smithi148.stdout:diff --git a/src/mds/Anchor.h b/src/mds/Anchor.h 2023-10-30T13:52:20.194 INFO:tasks.workunit.client.0.smithi148.stdout:index a55a07dd306..e69de29bb2d 100644 2023-10-30T13:52:20.194 INFO:tasks.workunit.client.0.smithi148.stdout:--- a/src/mds/Anchor.h 2023-10-30T13:52:20.194 INFO:tasks.workunit.client.0.smithi148.stdout:+++ b/src/mds/Anchor.h 2023-10-30T13:52:20.194 INFO:tasks.workunit.client.0.smithi148.stdout:@@ -1,108 +0,0 @@ 2023-10-30T13:52:20.195 INFO:tasks.workunit.client.0.smithi148.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2023-10-30T13:52:20.195 INFO:tasks.workunit.client.0.smithi148.stdout:-// vim: ts=8 sw=2 smarttab 2023-10-30T13:52:20.195 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.195 INFO:tasks.workunit.client.0.smithi148.stdout:- * Ceph - scalable distributed file system 2023-10-30T13:52:20.195 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.196 INFO:tasks.workunit.client.0.smithi148.stdout:- * Copyright (C) 2004-2006 Sage Weil 2023-10-30T13:52:20.196 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.196 INFO:tasks.workunit.client.0.smithi148.stdout:- * This is free software; you can redistribute it and/or 2023-10-30T13:52:20.196 INFO:tasks.workunit.client.0.smithi148.stdout:- * modify it under the terms of the GNU Lesser General Public 2023-10-30T13:52:20.196 INFO:tasks.workunit.client.0.smithi148.stdout:- * License version 2.1, as published by the Free Software 2023-10-30T13:52:20.197 INFO:tasks.workunit.client.0.smithi148.stdout:- * Foundation. See file COPYING. 2023-10-30T13:52:20.197 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.197 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.197 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.197 INFO:tasks.workunit.client.0.smithi148.stdout:-#ifndef __ANCHOR_H 2023-10-30T13:52:20.198 INFO:tasks.workunit.client.0.smithi148.stdout:-#define __ANCHOR_H 2023-10-30T13:52:20.198 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.198 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.198 INFO:tasks.workunit.client.0.smithi148.stdout:-using std::string; 2023-10-30T13:52:20.198 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.199 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/types.h" 2023-10-30T13:52:20.199 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "mdstypes.h" 2023-10-30T13:52:20.199 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/buffer.h" 2023-10-30T13:52:20.199 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.199 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.199 INFO:tasks.workunit.client.0.smithi148.stdout:-// anchor ops 2023-10-30T13:52:20.200 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_LOOKUP 1 2023-10-30T13:52:20.200 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_LOOKUP_REPLY -2 2023-10-30T13:52:20.200 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.200 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_CREATE_PREPARE 11 2023-10-30T13:52:20.200 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_CREATE_AGREE -12 2023-10-30T13:52:20.201 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.201 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_DESTROY_PREPARE 21 2023-10-30T13:52:20.201 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_DESTROY_AGREE -22 2023-10-30T13:52:20.201 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.201 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_UPDATE_PREPARE 31 2023-10-30T13:52:20.201 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_UPDATE_AGREE -32 2023-10-30T13:52:20.202 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.202 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_COMMIT 41 2023-10-30T13:52:20.202 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_ACK -42 2023-10-30T13:52:20.202 INFO:tasks.workunit.client.0.smithi148.stdout:-#define ANCHOR_OP_ROLLBACK 43 2023-10-30T13:52:20.202 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.203 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.203 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.203 INFO:tasks.workunit.client.0.smithi148.stdout:-inline const char* get_anchor_opname(int o) { 2023-10-30T13:52:20.203 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (o) { 2023-10-30T13:52:20.203 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_LOOKUP: return "lookup"; 2023-10-30T13:52:20.203 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_LOOKUP_REPLY: return "lookup_reply"; 2023-10-30T13:52:20.204 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.204 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_CREATE_PREPARE: return "create_prepare"; 2023-10-30T13:52:20.204 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_CREATE_AGREE: return "create_agree"; 2023-10-30T13:52:20.204 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_DESTROY_PREPARE: return "destroy_prepare"; 2023-10-30T13:52:20.204 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_DESTROY_AGREE: return "destroy_agree"; 2023-10-30T13:52:20.204 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_UPDATE_PREPARE: return "update_prepare"; 2023-10-30T13:52:20.205 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_UPDATE_AGREE: return "update_agree"; 2023-10-30T13:52:20.205 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.205 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_COMMIT: return "commit"; 2023-10-30T13:52:20.205 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_ACK: return "ack"; 2023-10-30T13:52:20.205 INFO:tasks.workunit.client.0.smithi148.stdout:- case ANCHOR_OP_ROLLBACK: return "rollback"; 2023-10-30T13:52:20.205 INFO:tasks.workunit.client.0.smithi148.stdout:- default: assert(0); return 0; 2023-10-30T13:52:20.206 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.206 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.206 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.206 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.206 INFO:tasks.workunit.client.0.smithi148.stdout:-// identifies a anchor table mutation 2023-10-30T13:52:20.207 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.207 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.207 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.207 INFO:tasks.workunit.client.0.smithi148.stdout:-// anchor type 2023-10-30T13:52:20.207 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.208 INFO:tasks.workunit.client.0.smithi148.stdout:-class Anchor { 2023-10-30T13:52:20.208 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.208 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t ino; // anchored ino 2023-10-30T13:52:20.208 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrag_t dirfrag; // containing dirfrag 2023-10-30T13:52:20.208 INFO:tasks.workunit.client.0.smithi148.stdout:- //string ref_dn; // referring dentry 2023-10-30T13:52:20.209 INFO:tasks.workunit.client.0.smithi148.stdout:- int nref; // reference count 2023-10-30T13:52:20.209 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.209 INFO:tasks.workunit.client.0.smithi148.stdout:- Anchor() {} 2023-10-30T13:52:20.209 INFO:tasks.workunit.client.0.smithi148.stdout:- Anchor(inodeno_t i, dirfrag_t df, 2023-10-30T13:52:20.209 INFO:tasks.workunit.client.0.smithi148.stdout:- //string& rd, 2023-10-30T13:52:20.209 INFO:tasks.workunit.client.0.smithi148.stdout:- int nr=0) : 2023-10-30T13:52:20.210 INFO:tasks.workunit.client.0.smithi148.stdout:- ino(i), dirfrag(df), 2023-10-30T13:52:20.210 INFO:tasks.workunit.client.0.smithi148.stdout:- //ref_dn(rd), 2023-10-30T13:52:20.210 INFO:tasks.workunit.client.0.smithi148.stdout:- nref(nr) { } 2023-10-30T13:52:20.210 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.210 INFO:tasks.workunit.client.0.smithi148.stdout:- void _encode(bufferlist &bl) { 2023-10-30T13:52:20.211 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append((char*)&ino, sizeof(ino)); 2023-10-30T13:52:20.211 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append((char*)&dirfrag, sizeof(dirfrag)); 2023-10-30T13:52:20.211 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append((char*)&nref, sizeof(nref)); 2023-10-30T13:52:20.211 INFO:tasks.workunit.client.0.smithi148.stdout:- //::_encode(ref_dn, bl); 2023-10-30T13:52:20.211 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.211 INFO:tasks.workunit.client.0.smithi148.stdout:- void _decode(bufferlist& bl, int& off) { 2023-10-30T13:52:20.212 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.copy(off, sizeof(ino), (char*)&ino); 2023-10-30T13:52:20.212 INFO:tasks.workunit.client.0.smithi148.stdout:- off += sizeof(ino); 2023-10-30T13:52:20.212 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.copy(off, sizeof(dirfrag), (char*)&dirfrag); 2023-10-30T13:52:20.212 INFO:tasks.workunit.client.0.smithi148.stdout:- off += sizeof(dirfrag); 2023-10-30T13:52:20.212 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.copy(off, sizeof(nref), (char*)&nref); 2023-10-30T13:52:20.213 INFO:tasks.workunit.client.0.smithi148.stdout:- off += sizeof(nref); 2023-10-30T13:52:20.213 INFO:tasks.workunit.client.0.smithi148.stdout:- //::_decode(ref_dn, bl, off); 2023-10-30T13:52:20.213 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.213 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:20.213 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.214 INFO:tasks.workunit.client.0.smithi148.stdout:-inline ostream& operator<<(ostream& out, Anchor& a) 2023-10-30T13:52:20.214 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.214 INFO:tasks.workunit.client.0.smithi148.stdout:- return out << "a(" << a.ino << " " << a.dirfrag << " " << a.nref << ")"; 2023-10-30T13:52:20.214 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.214 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.215 INFO:tasks.workunit.client.0.smithi148.stdout:-#endif 2023-10-30T13:52:20.215 INFO:tasks.workunit.client.0.smithi148.stdout:diff --git a/src/mds/CDentry.cc b/src/mds/CDentry.cc 2023-10-30T13:52:20.215 INFO:tasks.workunit.client.0.smithi148.stdout:index ce1fd04c14a..e69de29bb2d 100644 2023-10-30T13:52:20.215 INFO:tasks.workunit.client.0.smithi148.stdout:--- a/src/mds/CDentry.cc 2023-10-30T13:52:20.215 INFO:tasks.workunit.client.0.smithi148.stdout:+++ b/src/mds/CDentry.cc 2023-10-30T13:52:20.216 INFO:tasks.workunit.client.0.smithi148.stdout:@@ -1,384 +0,0 @@ 2023-10-30T13:52:20.216 INFO:tasks.workunit.client.0.smithi148.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2023-10-30T13:52:20.216 INFO:tasks.workunit.client.0.smithi148.stdout:-// vim: ts=8 sw=2 smarttab 2023-10-30T13:52:20.216 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.216 INFO:tasks.workunit.client.0.smithi148.stdout:- * Ceph - scalable distributed file system 2023-10-30T13:52:20.217 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.217 INFO:tasks.workunit.client.0.smithi148.stdout:- * Copyright (C) 2004-2006 Sage Weil 2023-10-30T13:52:20.217 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.217 INFO:tasks.workunit.client.0.smithi148.stdout:- * This is free software; you can redistribute it and/or 2023-10-30T13:52:20.217 INFO:tasks.workunit.client.0.smithi148.stdout:- * modify it under the terms of the GNU Lesser General Public 2023-10-30T13:52:20.217 INFO:tasks.workunit.client.0.smithi148.stdout:- * License version 2.1, as published by the Free Software 2023-10-30T13:52:20.218 INFO:tasks.workunit.client.0.smithi148.stdout:- * Foundation. See file COPYING. 2023-10-30T13:52:20.218 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.218 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.218 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.218 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.219 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.219 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CDentry.h" 2023-10-30T13:52:20.219 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CInode.h" 2023-10-30T13:52:20.219 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CDir.h" 2023-10-30T13:52:20.219 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "Anchor.h" 2023-10-30T13:52:20.220 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.220 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "MDS.h" 2023-10-30T13:52:20.220 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "MDCache.h" 2023-10-30T13:52:20.220 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "LogSegment.h" 2023-10-30T13:52:20.220 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.221 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "messages/MLock.h" 2023-10-30T13:52:20.221 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.221 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.221 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.221 INFO:tasks.workunit.client.0.smithi148.stdout:-#define dout(x) if (x <= g_conf.debug || x <= g_conf.debug_mds) *_dout << dbeginl << g_clock.now() << " mds" << dir->cache->mds->get_nodeid() << ".cache.den(" << dir->dirfrag() << " " << name << ") " 2023-10-30T13:52:20.222 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.222 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.222 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.222 INFO:tasks.workunit.client.0.smithi148.stdout:-ostream& CDentry::print_db_line_prefix(ostream& out) 2023-10-30T13:52:20.222 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.222 INFO:tasks.workunit.client.0.smithi148.stdout:- return out << g_clock.now() << " mds" << dir->cache->mds->get_nodeid() << ".cache.den(" << dir->ino() << " " << name << ") "; 2023-10-30T13:52:20.223 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.223 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.223 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.223 INFO:tasks.workunit.client.0.smithi148.stdout:-// CDentry 2023-10-30T13:52:20.223 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.224 INFO:tasks.workunit.client.0.smithi148.stdout:-ostream& operator<<(ostream& out, CDentry& dn) 2023-10-30T13:52:20.224 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.224 INFO:tasks.workunit.client.0.smithi148.stdout:- filepath path; 2023-10-30T13:52:20.224 INFO:tasks.workunit.client.0.smithi148.stdout:- dn.make_path(path); 2023-10-30T13:52:20.224 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.224 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "[dentry " << path; 2023-10-30T13:52:20.225 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.225 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn.is_auth()) { 2023-10-30T13:52:20.225 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " auth"; 2023-10-30T13:52:20.225 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn.is_replicated()) 2023-10-30T13:52:20.225 INFO:tasks.workunit.client.0.smithi148.stdout:- out << dn.get_replicas(); 2023-10-30T13:52:20.225 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.226 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " rep@" << dn.authority(); 2023-10-30T13:52:20.226 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "." << dn.get_replica_nonce(); 2023-10-30T13:52:20.226 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn.get_replica_nonce() >= 0); 2023-10-30T13:52:20.226 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.226 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.227 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn.is_null()) out << " NULL"; 2023-10-30T13:52:20.227 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn.is_remote()) { 2023-10-30T13:52:20.227 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " REMOTE("; 2023-10-30T13:52:20.227 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (dn.get_remote_d_type() << 12) { 2023-10-30T13:52:20.227 INFO:tasks.workunit.client.0.smithi148.stdout:- case S_IFREG: out << "reg"; break; 2023-10-30T13:52:20.227 INFO:tasks.workunit.client.0.smithi148.stdout:- case S_IFDIR: out << "dir"; break; 2023-10-30T13:52:20.228 INFO:tasks.workunit.client.0.smithi148.stdout:- case S_IFLNK: out << "lnk"; break; 2023-10-30T13:52:20.228 INFO:tasks.workunit.client.0.smithi148.stdout:- default: assert(0); 2023-10-30T13:52:20.228 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.228 INFO:tasks.workunit.client.0.smithi148.stdout:- out << ")"; 2023-10-30T13:52:20.228 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.229 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.229 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " " << dn.lock; 2023-10-30T13:52:20.229 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.229 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " v=" << dn.get_version(); 2023-10-30T13:52:20.229 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " pv=" << dn.get_projected_version(); 2023-10-30T13:52:20.229 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.230 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " inode=" << dn.get_inode(); 2023-10-30T13:52:20.230 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.230 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn.is_new()) out << " state=new"; 2023-10-30T13:52:20.230 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.230 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn.get_num_ref()) { 2023-10-30T13:52:20.231 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " |"; 2023-10-30T13:52:20.231 INFO:tasks.workunit.client.0.smithi148.stdout:- dn.print_pin_set(out); 2023-10-30T13:52:20.231 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.231 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.231 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " " << &dn; 2023-10-30T13:52:20.231 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "]"; 2023-10-30T13:52:20.232 INFO:tasks.workunit.client.0.smithi148.stdout:- return out; 2023-10-30T13:52:20.232 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.232 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.232 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.232 INFO:tasks.workunit.client.0.smithi148.stdout:-bool operator<(const CDentry& l, const CDentry& r) 2023-10-30T13:52:20.233 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.233 INFO:tasks.workunit.client.0.smithi148.stdout:- if (l.get_dir()->ino() < r.get_dir()->ino()) return true; 2023-10-30T13:52:20.233 INFO:tasks.workunit.client.0.smithi148.stdout:- if (l.get_dir()->ino() == r.get_dir()->ino() && 2023-10-30T13:52:20.233 INFO:tasks.workunit.client.0.smithi148.stdout:- l.get_name() < r.get_name()) return true; 2023-10-30T13:52:20.233 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.234 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.234 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.234 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.234 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::print(ostream& out) 2023-10-30T13:52:20.234 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.234 INFO:tasks.workunit.client.0.smithi148.stdout:- out << *this; 2023-10-30T13:52:20.235 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.235 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.235 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.235 INFO:tasks.workunit.client.0.smithi148.stdout:-inodeno_t CDentry::get_ino() 2023-10-30T13:52:20.235 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.236 INFO:tasks.workunit.client.0.smithi148.stdout:- if (inode) 2023-10-30T13:52:20.236 INFO:tasks.workunit.client.0.smithi148.stdout:- return inode->ino(); 2023-10-30T13:52:20.236 INFO:tasks.workunit.client.0.smithi148.stdout:- return inodeno_t(); 2023-10-30T13:52:20.236 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.236 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.236 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.237 INFO:tasks.workunit.client.0.smithi148.stdout:-pair CDentry::authority() 2023-10-30T13:52:20.237 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.237 INFO:tasks.workunit.client.0.smithi148.stdout:- return dir->authority(); 2023-10-30T13:52:20.237 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.237 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.238 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.238 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::add_waiter(int tag, Context *c) 2023-10-30T13:52:20.238 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.238 INFO:tasks.workunit.client.0.smithi148.stdout:- // wait on the directory? 2023-10-30T13:52:20.238 INFO:tasks.workunit.client.0.smithi148.stdout:- if (tag & (WAIT_UNFREEZE|WAIT_SINGLEAUTH)) { 2023-10-30T13:52:20.239 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->add_waiter(tag, c); 2023-10-30T13:52:20.239 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.239 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.239 INFO:tasks.workunit.client.0.smithi148.stdout:- MDSCacheObject::add_waiter(tag, c); 2023-10-30T13:52:20.239 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.240 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.240 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.240 INFO:tasks.workunit.client.0.smithi148.stdout:-version_t CDentry::pre_dirty(version_t min) 2023-10-30T13:52:20.240 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.240 INFO:tasks.workunit.client.0.smithi148.stdout:- projected_version = dir->pre_dirty(min); 2023-10-30T13:52:20.241 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << " pre_dirty " << *this << dendl; 2023-10-30T13:52:20.241 INFO:tasks.workunit.client.0.smithi148.stdout:- return projected_version; 2023-10-30T13:52:20.241 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.241 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.241 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.241 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::_mark_dirty(LogSegment *ls) 2023-10-30T13:52:20.242 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.242 INFO:tasks.workunit.client.0.smithi148.stdout:- // state+pin 2023-10-30T13:52:20.242 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!state_test(STATE_DIRTY)) { 2023-10-30T13:52:20.242 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_DIRTY); 2023-10-30T13:52:20.242 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->inc_num_dirty(); 2023-10-30T13:52:20.243 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_DIRTY); 2023-10-30T13:52:20.243 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(ls); 2023-10-30T13:52:20.243 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.243 INFO:tasks.workunit.client.0.smithi148.stdout:- if (ls) 2023-10-30T13:52:20.243 INFO:tasks.workunit.client.0.smithi148.stdout:- ls->dirty_dentries.push_back(&xlist_dirty); 2023-10-30T13:52:20.244 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.244 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.244 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::mark_dirty(version_t pv, LogSegment *ls) 2023-10-30T13:52:20.244 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.244 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << " mark_dirty " << *this << dendl; 2023-10-30T13:52:20.244 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.245 INFO:tasks.workunit.client.0.smithi148.stdout:- // i now live in this new dir version 2023-10-30T13:52:20.245 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(pv <= projected_version); 2023-10-30T13:52:20.245 INFO:tasks.workunit.client.0.smithi148.stdout:- version = pv; 2023-10-30T13:52:20.245 INFO:tasks.workunit.client.0.smithi148.stdout:- _mark_dirty(ls); 2023-10-30T13:52:20.245 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.245 INFO:tasks.workunit.client.0.smithi148.stdout:- // mark dir too 2023-10-30T13:52:20.246 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->mark_dirty(pv, ls); 2023-10-30T13:52:20.246 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.246 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.246 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.246 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::mark_clean() 2023-10-30T13:52:20.247 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.247 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << " mark_clean " << *this << dendl; 2023-10-30T13:52:20.247 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_dirty()); 2023-10-30T13:52:20.247 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dir->get_version() == 0 || version <= dir->get_version()); // hmm? 2023-10-30T13:52:20.247 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.247 INFO:tasks.workunit.client.0.smithi148.stdout:- // state+pin 2023-10-30T13:52:20.248 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_DIRTY); 2023-10-30T13:52:20.248 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->dec_num_dirty(); 2023-10-30T13:52:20.248 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_DIRTY); 2023-10-30T13:52:20.248 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.248 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty.remove_myself(); 2023-10-30T13:52:20.249 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.249 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_NEW)) 2023-10-30T13:52:20.249 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_NEW); 2023-10-30T13:52:20.249 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.249 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.250 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::mark_new() 2023-10-30T13:52:20.250 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.250 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << " mark_new " << *this << dendl; 2023-10-30T13:52:20.250 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_NEW); 2023-10-30T13:52:20.250 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.250 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.251 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::make_path_string(string& s) 2023-10-30T13:52:20.251 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.251 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir) { 2023-10-30T13:52:20.251 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->inode->make_path_string(s); 2023-10-30T13:52:20.251 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.252 INFO:tasks.workunit.client.0.smithi148.stdout:- s = "???"; 2023-10-30T13:52:20.252 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.252 INFO:tasks.workunit.client.0.smithi148.stdout:- s += "/"; 2023-10-30T13:52:20.252 INFO:tasks.workunit.client.0.smithi148.stdout:- s += name; 2023-10-30T13:52:20.253 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.253 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.253 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::make_path(filepath& fp) 2023-10-30T13:52:20.253 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.253 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dir); 2023-10-30T13:52:20.254 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->inode->is_base()) 2023-10-30T13:52:20.254 INFO:tasks.workunit.client.0.smithi148.stdout:- fp.set_ino(dir->inode->ino()); // base case 2023-10-30T13:52:20.254 INFO:tasks.workunit.client.0.smithi148.stdout:- else if (dir->inode->get_parent_dn()) 2023-10-30T13:52:20.254 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->inode->get_parent_dn()->make_path(fp); // recurse 2023-10-30T13:52:20.254 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.255 INFO:tasks.workunit.client.0.smithi148.stdout:- fp.set_ino(dir->inode->ino()); // relative but not base? hrm! 2023-10-30T13:52:20.255 INFO:tasks.workunit.client.0.smithi148.stdout:- fp.push_dentry(name); 2023-10-30T13:52:20.255 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.255 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.255 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.255 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::make_path(string& s, inodeno_t tobase) 2023-10-30T13:52:20.256 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.256 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dir); 2023-10-30T13:52:20.256 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.256 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->inode->is_root()) { 2023-10-30T13:52:20.256 INFO:tasks.workunit.client.0.smithi148.stdout:- s += "/"; // make it an absolute path (no matter what) if we hit the root. 2023-10-30T13:52:20.256 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.257 INFO:tasks.workunit.client.0.smithi148.stdout:- else if (dir->inode->get_parent_dn() && 2023-10-30T13:52:20.257 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->inode->ino() != tobase) { 2023-10-30T13:52:20.257 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->inode->get_parent_dn()->make_path(s, tobase); 2023-10-30T13:52:20.257 INFO:tasks.workunit.client.0.smithi148.stdout:- s += "/"; 2023-10-30T13:52:20.257 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.257 INFO:tasks.workunit.client.0.smithi148.stdout:- s += name; 2023-10-30T13:52:20.258 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.258 INFO:tasks.workunit.client.0.smithi148.stdout:-*/ 2023-10-30T13:52:20.258 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.258 INFO:tasks.workunit.client.0.smithi148.stdout:-/** make_anchor_trace 2023-10-30T13:52:20.258 INFO:tasks.workunit.client.0.smithi148.stdout:- * construct an anchor trace for this dentry, as if it were linked to *in. 2023-10-30T13:52:20.258 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.259 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::make_anchor_trace(vector& trace, CInode *in) 2023-10-30T13:52:20.259 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.259 INFO:tasks.workunit.client.0.smithi148.stdout:- // start with parent dir inode 2023-10-30T13:52:20.259 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir) 2023-10-30T13:52:20.259 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->inode->make_anchor_trace(trace); 2023-10-30T13:52:20.260 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.260 INFO:tasks.workunit.client.0.smithi148.stdout:- // add this inode (in my dirfrag) to the end 2023-10-30T13:52:20.260 INFO:tasks.workunit.client.0.smithi148.stdout:- trace.push_back(Anchor(in->ino(), dir->dirfrag())); 2023-10-30T13:52:20.260 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "make_anchor_trace added " << trace.back() << dendl; 2023-10-30T13:52:20.260 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.261 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.261 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.261 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.261 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::link_remote(CInode *in) 2023-10-30T13:52:20.261 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.261 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_remote()); 2023-10-30T13:52:20.262 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(in->ino() == remote_ino); 2023-10-30T13:52:20.262 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.262 INFO:tasks.workunit.client.0.smithi148.stdout:- inode = in; 2023-10-30T13:52:20.262 INFO:tasks.workunit.client.0.smithi148.stdout:- in->add_remote_parent(this); 2023-10-30T13:52:20.262 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.263 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.263 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::unlink_remote() 2023-10-30T13:52:20.263 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.263 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_remote()); 2023-10-30T13:52:20.263 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(inode); 2023-10-30T13:52:20.263 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.264 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->remove_remote_parent(this); 2023-10-30T13:52:20.264 INFO:tasks.workunit.client.0.smithi148.stdout:- inode = 0; 2023-10-30T13:52:20.264 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.264 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.264 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.264 INFO:tasks.workunit.client.0.smithi148.stdout:-CDentryDiscover *CDentry::replicate_to(int who) 2023-10-30T13:52:20.264 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.265 INFO:tasks.workunit.client.0.smithi148.stdout:- int nonce = add_replica(who); 2023-10-30T13:52:20.265 INFO:tasks.workunit.client.0.smithi148.stdout:- return new CDentryDiscover(this, nonce); 2023-10-30T13:52:20.265 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.265 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.265 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.265 INFO:tasks.workunit.client.0.smithi148.stdout:-// ---------------------------- 2023-10-30T13:52:20.266 INFO:tasks.workunit.client.0.smithi148.stdout:-// auth pins 2023-10-30T13:52:20.266 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.266 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CDentry::can_auth_pin() 2023-10-30T13:52:20.266 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.266 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dir); 2023-10-30T13:52:20.266 INFO:tasks.workunit.client.0.smithi148.stdout:- return dir->can_auth_pin(); 2023-10-30T13:52:20.267 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.267 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.267 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::auth_pin() 2023-10-30T13:52:20.267 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.267 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth_pins == 0) 2023-10-30T13:52:20.267 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_AUTHPIN); 2023-10-30T13:52:20.268 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins++; 2023-10-30T13:52:20.268 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.268 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "auth_pin on " << *this 2023-10-30T13:52:20.268 INFO:tasks.workunit.client.0.smithi148.stdout:- << " now " << auth_pins << "+" << nested_auth_pins 2023-10-30T13:52:20.268 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.268 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.269 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->adjust_nested_auth_pins(1); 2023-10-30T13:52:20.269 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.269 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.269 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::auth_unpin() 2023-10-30T13:52:20.269 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.270 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins--; 2023-10-30T13:52:20.270 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth_pins == 0) 2023-10-30T13:52:20.270 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_AUTHPIN); 2023-10-30T13:52:20.270 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.270 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "auth_unpin on " << *this 2023-10-30T13:52:20.270 INFO:tasks.workunit.client.0.smithi148.stdout:- << " now " << auth_pins << "+" << nested_auth_pins 2023-10-30T13:52:20.271 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.271 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(auth_pins >= 0); 2023-10-30T13:52:20.271 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.271 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->adjust_nested_auth_pins(-1); 2023-10-30T13:52:20.271 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.271 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.271 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::adjust_nested_auth_pins(int by) 2023-10-30T13:52:20.272 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.272 INFO:tasks.workunit.client.0.smithi148.stdout:- nested_auth_pins += by; 2023-10-30T13:52:20.272 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.272 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << "adjust_nested_auth_pins by " << by 2023-10-30T13:52:20.272 INFO:tasks.workunit.client.0.smithi148.stdout:- << " now " << auth_pins << "+" << nested_auth_pins 2023-10-30T13:52:20.272 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.273 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nested_auth_pins >= 0); 2023-10-30T13:52:20.273 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.273 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->adjust_nested_auth_pins(by); 2023-10-30T13:52:20.273 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.273 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.273 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CDentry::is_frozen() 2023-10-30T13:52:20.274 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.274 INFO:tasks.workunit.client.0.smithi148.stdout:- return dir->is_frozen(); 2023-10-30T13:52:20.274 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.274 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.274 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.274 INFO:tasks.workunit.client.0.smithi148.stdout:-// ---------------------------- 2023-10-30T13:52:20.275 INFO:tasks.workunit.client.0.smithi148.stdout:-// locking 2023-10-30T13:52:20.275 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.275 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::set_object_info(MDSCacheObjectInfo &info) 2023-10-30T13:52:20.275 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.275 INFO:tasks.workunit.client.0.smithi148.stdout:- info.dirfrag = dir->dirfrag(); 2023-10-30T13:52:20.276 INFO:tasks.workunit.client.0.smithi148.stdout:- info.dname = name; 2023-10-30T13:52:20.276 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.276 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.276 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::encode_lock_state(int type, bufferlist& bl) 2023-10-30T13:52:20.276 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.276 INFO:tasks.workunit.client.0.smithi148.stdout:- // null, ino, or remote_ino? 2023-10-30T13:52:20.276 INFO:tasks.workunit.client.0.smithi148.stdout:- int c; 2023-10-30T13:52:20.277 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_primary()) { 2023-10-30T13:52:20.277 INFO:tasks.workunit.client.0.smithi148.stdout:- c = 1; 2023-10-30T13:52:20.277 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(c, bl); 2023-10-30T13:52:20.277 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(inode->inode.ino, bl); 2023-10-30T13:52:20.277 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.277 INFO:tasks.workunit.client.0.smithi148.stdout:- else if (is_remote()) { 2023-10-30T13:52:20.278 INFO:tasks.workunit.client.0.smithi148.stdout:- c = 2; 2023-10-30T13:52:20.278 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(c, bl); 2023-10-30T13:52:20.278 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(remote_ino, bl); 2023-10-30T13:52:20.278 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.278 INFO:tasks.workunit.client.0.smithi148.stdout:- else if (is_null()) { 2023-10-30T13:52:20.279 INFO:tasks.workunit.client.0.smithi148.stdout:- // encode nothing. 2023-10-30T13:52:20.279 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.279 INFO:tasks.workunit.client.0.smithi148.stdout:- else assert(0); 2023-10-30T13:52:20.279 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.279 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.279 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDentry::decode_lock_state(int type, bufferlist& bl) 2023-10-30T13:52:20.280 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.280 INFO:tasks.workunit.client.0.smithi148.stdout:- if (bl.length() == 0) { 2023-10-30T13:52:20.280 INFO:tasks.workunit.client.0.smithi148.stdout:- // null 2023-10-30T13:52:20.280 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_null()); 2023-10-30T13:52:20.280 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.281 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.281 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.281 INFO:tasks.workunit.client.0.smithi148.stdout:- int off = 0; 2023-10-30T13:52:20.281 INFO:tasks.workunit.client.0.smithi148.stdout:- char c; 2023-10-30T13:52:20.281 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t ino; 2023-10-30T13:52:20.281 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(c, bl, off); 2023-10-30T13:52:20.282 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.282 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (c) { 2023-10-30T13:52:20.282 INFO:tasks.workunit.client.0.smithi148.stdout:- case 1: 2023-10-30T13:52:20.282 INFO:tasks.workunit.client.0.smithi148.stdout:- case 2: 2023-10-30T13:52:20.282 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(ino, bl, off); 2023-10-30T13:52:20.283 INFO:tasks.workunit.client.0.smithi148.stdout:- // newly linked? 2023-10-30T13:52:20.283 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_null() && !is_auth()) { 2023-10-30T13:52:20.283 INFO:tasks.workunit.client.0.smithi148.stdout:- // force trim from cache! 2023-10-30T13:52:20.283 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "decode_lock_state replica dentry null -> non-null, must trim" << dendl; 2023-10-30T13:52:20.283 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(get_num_ref() == 0); 2023-10-30T13:52:20.283 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.283 INFO:tasks.workunit.client.0.smithi148.stdout:- // verify? 2023-10-30T13:52:20.284 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.284 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.284 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.284 INFO:tasks.workunit.client.0.smithi148.stdout:- default: 2023-10-30T13:52:20.284 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(0); 2023-10-30T13:52:20.284 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.285 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.285 INFO:tasks.workunit.client.0.smithi148.stdout:diff --git a/src/mds/CDentry.h b/src/mds/CDentry.h 2023-10-30T13:52:20.285 INFO:tasks.workunit.client.0.smithi148.stdout:index b99ad9ea603..e69de29bb2d 100644 2023-10-30T13:52:20.285 INFO:tasks.workunit.client.0.smithi148.stdout:--- a/src/mds/CDentry.h 2023-10-30T13:52:20.285 INFO:tasks.workunit.client.0.smithi148.stdout:+++ b/src/mds/CDentry.h 2023-10-30T13:52:20.285 INFO:tasks.workunit.client.0.smithi148.stdout:@@ -1,325 +0,0 @@ 2023-10-30T13:52:20.286 INFO:tasks.workunit.client.0.smithi148.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2023-10-30T13:52:20.286 INFO:tasks.workunit.client.0.smithi148.stdout:-// vim: ts=8 sw=2 smarttab 2023-10-30T13:52:20.286 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.286 INFO:tasks.workunit.client.0.smithi148.stdout:- * Ceph - scalable distributed file system 2023-10-30T13:52:20.286 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.287 INFO:tasks.workunit.client.0.smithi148.stdout:- * Copyright (C) 2004-2006 Sage Weil 2023-10-30T13:52:20.287 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.287 INFO:tasks.workunit.client.0.smithi148.stdout:- * This is free software; you can redistribute it and/or 2023-10-30T13:52:20.287 INFO:tasks.workunit.client.0.smithi148.stdout:- * modify it under the terms of the GNU Lesser General Public 2023-10-30T13:52:20.287 INFO:tasks.workunit.client.0.smithi148.stdout:- * License version 2.1, as published by the Free Software 2023-10-30T13:52:20.288 INFO:tasks.workunit.client.0.smithi148.stdout:- * Foundation. See file COPYING. 2023-10-30T13:52:20.288 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.288 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.288 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.288 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.288 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.289 INFO:tasks.workunit.client.0.smithi148.stdout:-#ifndef __CDENTRY_H 2023-10-30T13:52:20.289 INFO:tasks.workunit.client.0.smithi148.stdout:-#define __CDENTRY_H 2023-10-30T13:52:20.289 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.289 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.289 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.289 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.290 INFO:tasks.workunit.client.0.smithi148.stdout:-using namespace std; 2023-10-30T13:52:20.290 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.290 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/types.h" 2023-10-30T13:52:20.290 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/buffer.h" 2023-10-30T13:52:20.290 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/lru.h" 2023-10-30T13:52:20.291 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/xlist.h" 2023-10-30T13:52:20.291 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/filepath.h" 2023-10-30T13:52:20.291 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "mdstypes.h" 2023-10-30T13:52:20.291 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.291 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "SimpleLock.h" 2023-10-30T13:52:20.292 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.292 INFO:tasks.workunit.client.0.smithi148.stdout:-class CInode; 2023-10-30T13:52:20.292 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDir; 2023-10-30T13:52:20.292 INFO:tasks.workunit.client.0.smithi148.stdout:-class MDRequest; 2023-10-30T13:52:20.292 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.292 INFO:tasks.workunit.client.0.smithi148.stdout:-class Message; 2023-10-30T13:52:20.293 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDentryDiscover; 2023-10-30T13:52:20.293 INFO:tasks.workunit.client.0.smithi148.stdout:-class Anchor; 2023-10-30T13:52:20.293 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.293 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDentry; 2023-10-30T13:52:20.293 INFO:tasks.workunit.client.0.smithi148.stdout:-class LogSegment; 2023-10-30T13:52:20.293 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.293 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.294 INFO:tasks.workunit.client.0.smithi148.stdout:-// define an ordering 2023-10-30T13:52:20.294 INFO:tasks.workunit.client.0.smithi148.stdout:-bool operator<(const CDentry& l, const CDentry& r); 2023-10-30T13:52:20.294 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.294 INFO:tasks.workunit.client.0.smithi148.stdout:-// dentry 2023-10-30T13:52:20.294 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDentry : public MDSCacheObject, public LRUObject { 2023-10-30T13:52:20.294 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.295 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- state -- 2023-10-30T13:52:20.295 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_NEW = 1; 2023-10-30T13:52:20.295 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_FRAGMENTING = 2; 2023-10-30T13:52:20.295 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.295 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- pins -- 2023-10-30T13:52:20.295 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_INODEPIN = 1; // linked inode is pinned 2023-10-30T13:52:20.296 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_FRAGMENTING = -2; // containing dir is refragmenting 2023-10-30T13:52:20.296 INFO:tasks.workunit.client.0.smithi148.stdout:- const char *pin_name(int p) { 2023-10-30T13:52:20.296 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (p) { 2023-10-30T13:52:20.296 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_INODEPIN: return "inodepin"; 2023-10-30T13:52:20.296 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_FRAGMENTING: return "fragmenting"; 2023-10-30T13:52:20.296 INFO:tasks.workunit.client.0.smithi148.stdout:- default: return generic_pin_name(p); 2023-10-30T13:52:20.297 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.297 INFO:tasks.workunit.client.0.smithi148.stdout:- }; 2023-10-30T13:52:20.297 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.297 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- wait -- 2023-10-30T13:52:20.297 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_LOCK_OFFSET = 8; 2023-10-30T13:52:20.297 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.298 INFO:tasks.workunit.client.0.smithi148.stdout:- void add_waiter(int tag, Context *c); 2023-10-30T13:52:20.298 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.298 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int EXPORT_NONCE = 1; 2023-10-30T13:52:20.298 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.298 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_lt(const MDSCacheObject *r) const { 2023-10-30T13:52:20.299 INFO:tasks.workunit.client.0.smithi148.stdout:- return *this < *(CDentry*)r; 2023-10-30T13:52:20.299 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.299 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.299 INFO:tasks.workunit.client.0.smithi148.stdout:- protected: 2023-10-30T13:52:20.299 INFO:tasks.workunit.client.0.smithi148.stdout:- string name; 2023-10-30T13:52:20.299 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.300 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t remote_ino; // if remote dentry 2023-10-30T13:52:20.300 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned char remote_d_type; 2023-10-30T13:52:20.300 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.300 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *inode; // linked inode (if any) 2023-10-30T13:52:20.300 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir; // containing dirfrag 2023-10-30T13:52:20.300 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.301 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t version; // dir version when last touched. 2023-10-30T13:52:20.301 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t projected_version; // what it will be when i unlock/commit. 2023-10-30T13:52:20.301 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.301 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist::item xlist_dirty; 2023-10-30T13:52:20.301 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.301 INFO:tasks.workunit.client.0.smithi148.stdout:- off_t dir_offset; 2023-10-30T13:52:20.301 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.302 INFO:tasks.workunit.client.0.smithi148.stdout:- int auth_pins, nested_auth_pins; 2023-10-30T13:52:20.302 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.302 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class Migrator; 2023-10-30T13:52:20.302 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class Locker; 2023-10-30T13:52:20.302 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class Renamer; 2023-10-30T13:52:20.303 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class Server; 2023-10-30T13:52:20.303 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class MDCache; 2023-10-30T13:52:20.303 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class MDS; 2023-10-30T13:52:20.303 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class CInode; 2023-10-30T13:52:20.303 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class C_MDC_XlockRequest; 2023-10-30T13:52:20.303 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.304 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.304 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.304 INFO:tasks.workunit.client.0.smithi148.stdout:- // lock 2023-10-30T13:52:20.304 INFO:tasks.workunit.client.0.smithi148.stdout:- SimpleLock lock; 2023-10-30T13:52:20.304 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.304 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.305 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.305 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.305 INFO:tasks.workunit.client.0.smithi148.stdout:- // cons 2023-10-30T13:52:20.305 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry() : 2023-10-30T13:52:20.305 INFO:tasks.workunit.client.0.smithi148.stdout:- remote_ino(0), remote_d_type(0), 2023-10-30T13:52:20.305 INFO:tasks.workunit.client.0.smithi148.stdout:- inode(0), dir(0), 2023-10-30T13:52:20.306 INFO:tasks.workunit.client.0.smithi148.stdout:- version(0), projected_version(0), 2023-10-30T13:52:20.306 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty(this), 2023-10-30T13:52:20.306 INFO:tasks.workunit.client.0.smithi148.stdout:- dir_offset(0), 2023-10-30T13:52:20.306 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins(0), nested_auth_pins(0), 2023-10-30T13:52:20.306 INFO:tasks.workunit.client.0.smithi148.stdout:- lock(this, LOCK_OTYPE_DN, WAIT_LOCK_OFFSET) { } 2023-10-30T13:52:20.307 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry(const string& n, CInode *in) : 2023-10-30T13:52:20.307 INFO:tasks.workunit.client.0.smithi148.stdout:- name(n), 2023-10-30T13:52:20.307 INFO:tasks.workunit.client.0.smithi148.stdout:- remote_ino(0), remote_d_type(0), 2023-10-30T13:52:20.307 INFO:tasks.workunit.client.0.smithi148.stdout:- inode(in), dir(0), 2023-10-30T13:52:20.307 INFO:tasks.workunit.client.0.smithi148.stdout:- version(0), projected_version(0), 2023-10-30T13:52:20.307 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty(this), 2023-10-30T13:52:20.308 INFO:tasks.workunit.client.0.smithi148.stdout:- dir_offset(0), 2023-10-30T13:52:20.308 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins(0), nested_auth_pins(0), 2023-10-30T13:52:20.308 INFO:tasks.workunit.client.0.smithi148.stdout:- lock(this, LOCK_OTYPE_DN, WAIT_LOCK_OFFSET) { } 2023-10-30T13:52:20.308 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry(const string& n, inodeno_t ino, unsigned char dt, CInode *in=0) : 2023-10-30T13:52:20.308 INFO:tasks.workunit.client.0.smithi148.stdout:- name(n), 2023-10-30T13:52:20.308 INFO:tasks.workunit.client.0.smithi148.stdout:- remote_ino(ino), remote_d_type(dt), 2023-10-30T13:52:20.309 INFO:tasks.workunit.client.0.smithi148.stdout:- inode(in), dir(0), 2023-10-30T13:52:20.309 INFO:tasks.workunit.client.0.smithi148.stdout:- version(0), projected_version(0), 2023-10-30T13:52:20.309 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty(this), 2023-10-30T13:52:20.309 INFO:tasks.workunit.client.0.smithi148.stdout:- dir_offset(0), 2023-10-30T13:52:20.309 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins(0), nested_auth_pins(0), 2023-10-30T13:52:20.309 INFO:tasks.workunit.client.0.smithi148.stdout:- lock(this, LOCK_OTYPE_DN, WAIT_LOCK_OFFSET) { } 2023-10-30T13:52:20.310 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.310 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *get_inode() const { return inode; } 2023-10-30T13:52:20.310 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *get_dir() const { return dir; } 2023-10-30T13:52:20.310 INFO:tasks.workunit.client.0.smithi148.stdout:- const string& get_name() const { return name; } 2023-10-30T13:52:20.310 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t get_ino(); 2023-10-30T13:52:20.310 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.311 INFO:tasks.workunit.client.0.smithi148.stdout:- off_t get_dir_offset() { return dir_offset; } 2023-10-30T13:52:20.311 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_dir_offset(off_t o) { dir_offset = o; } 2023-10-30T13:52:20.311 INFO:tasks.workunit.client.0.smithi148.stdout:- void clear_dir_offset() { dir_offset = 0; } 2023-10-30T13:52:20.311 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.311 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t get_remote_ino() { return remote_ino; } 2023-10-30T13:52:20.311 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned char get_remote_d_type() { return remote_d_type; } 2023-10-30T13:52:20.312 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_remote(inodeno_t ino, unsigned char d_type) { 2023-10-30T13:52:20.312 INFO:tasks.workunit.client.0.smithi148.stdout:- remote_ino = ino; 2023-10-30T13:52:20.312 INFO:tasks.workunit.client.0.smithi148.stdout:- remote_d_type = d_type; 2023-10-30T13:52:20.312 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.312 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.312 INFO:tasks.workunit.client.0.smithi148.stdout:- // ref counts: pin ourselves in the LRU when we're pinned. 2023-10-30T13:52:20.313 INFO:tasks.workunit.client.0.smithi148.stdout:- void first_get() { 2023-10-30T13:52:20.313 INFO:tasks.workunit.client.0.smithi148.stdout:- lru_pin(); 2023-10-30T13:52:20.313 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.313 INFO:tasks.workunit.client.0.smithi148.stdout:- void last_put() { 2023-10-30T13:52:20.313 INFO:tasks.workunit.client.0.smithi148.stdout:- lru_unpin(); 2023-10-30T13:52:20.314 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.314 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.314 INFO:tasks.workunit.client.0.smithi148.stdout:- // auth pins 2023-10-30T13:52:20.314 INFO:tasks.workunit.client.0.smithi148.stdout:- bool can_auth_pin(); 2023-10-30T13:52:20.314 INFO:tasks.workunit.client.0.smithi148.stdout:- void auth_pin(); 2023-10-30T13:52:20.314 INFO:tasks.workunit.client.0.smithi148.stdout:- void auth_unpin(); 2023-10-30T13:52:20.315 INFO:tasks.workunit.client.0.smithi148.stdout:- void adjust_nested_auth_pins(int by); 2023-10-30T13:52:20.315 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_frozen(); 2023-10-30T13:52:20.315 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.315 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.315 INFO:tasks.workunit.client.0.smithi148.stdout:- // dentry type is primary || remote || null 2023-10-30T13:52:20.315 INFO:tasks.workunit.client.0.smithi148.stdout:- // inode ptr is required for primary, optional for remote, undefined for null 2023-10-30T13:52:20.316 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_primary() { return remote_ino == 0 && inode != 0; } 2023-10-30T13:52:20.316 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_remote() { return remote_ino > 0; } 2023-10-30T13:52:20.316 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_null() { return (remote_ino == 0 && inode == 0) ? true:false; } 2023-10-30T13:52:20.316 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.316 INFO:tasks.workunit.client.0.smithi148.stdout:- // remote links 2023-10-30T13:52:20.316 INFO:tasks.workunit.client.0.smithi148.stdout:- void link_remote(CInode *in); 2023-10-30T13:52:20.317 INFO:tasks.workunit.client.0.smithi148.stdout:- void unlink_remote(); 2023-10-30T13:52:20.317 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.317 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.317 INFO:tasks.workunit.client.0.smithi148.stdout:- // copy cons 2023-10-30T13:52:20.317 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry(const CDentry& m); 2023-10-30T13:52:20.317 INFO:tasks.workunit.client.0.smithi148.stdout:- const CDentry& operator= (const CDentry& right); 2023-10-30T13:52:20.318 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.318 INFO:tasks.workunit.client.0.smithi148.stdout:- // misc 2023-10-30T13:52:20.318 INFO:tasks.workunit.client.0.smithi148.stdout:- void make_path_string(string& s); 2023-10-30T13:52:20.318 INFO:tasks.workunit.client.0.smithi148.stdout:- void make_path(filepath& fp); 2023-10-30T13:52:20.318 INFO:tasks.workunit.client.0.smithi148.stdout:- void make_anchor_trace(vector& trace, CInode *in); 2023-10-30T13:52:20.319 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.319 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- version -- 2023-10-30T13:52:20.319 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t get_version() { return version; } 2023-10-30T13:52:20.319 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_version(version_t v) { projected_version = version = v; } 2023-10-30T13:52:20.319 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t get_projected_version() { return projected_version; } 2023-10-30T13:52:20.319 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_projected_version(version_t v) { projected_version = v; } 2023-10-30T13:52:20.319 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.320 INFO:tasks.workunit.client.0.smithi148.stdout:- pair authority(); 2023-10-30T13:52:20.320 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:52:20.323 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t pre_dirty(version_t min=0); 2023-10-30T13:52:20.323 INFO:tasks.workunit.client.0.smithi148.stdout:- void _mark_dirty(LogSegment *ls); 2023-10-30T13:52:20.323 INFO:tasks.workunit.client.0.smithi148.stdout:- void mark_dirty(version_t projected_dirv, LogSegment *ls); 2023-10-30T13:52:20.324 INFO:tasks.workunit.client.0.smithi148.stdout:- void mark_clean(); 2023-10-30T13:52:20.324 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.324 INFO:tasks.workunit.client.0.smithi148.stdout:- void mark_new(); 2023-10-30T13:52:20.324 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_new() { return state_test(STATE_NEW); } 2023-10-30T13:52:20.324 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.324 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- replication 2023-10-30T13:52:20.325 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentryDiscover *replicate_to(int rep); 2023-10-30T13:52:20.325 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.325 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.325 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- exporting 2023-10-30T13:52:20.325 INFO:tasks.workunit.client.0.smithi148.stdout:- // note: this assumes the dentry already exists. 2023-10-30T13:52:20.325 INFO:tasks.workunit.client.0.smithi148.stdout:- // i.e., the name is already extracted... so we just need the other state. 2023-10-30T13:52:20.326 INFO:tasks.workunit.client.0.smithi148.stdout:- void encode_export(bufferlist& bl) { 2023-10-30T13:52:20.326 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(state, bl); 2023-10-30T13:52:20.326 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(version, bl); 2023-10-30T13:52:20.326 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(projected_version, bl); 2023-10-30T13:52:20.326 INFO:tasks.workunit.client.0.smithi148.stdout:- lock._encode(bl); 2023-10-30T13:52:20.327 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(replica_map, bl); 2023-10-30T13:52:20.327 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_TEMPEXPORTING); 2023-10-30T13:52:20.327 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.327 INFO:tasks.workunit.client.0.smithi148.stdout:- void finish_export() { 2023-10-30T13:52:20.327 INFO:tasks.workunit.client.0.smithi148.stdout:- // twiddle 2023-10-30T13:52:20.327 INFO:tasks.workunit.client.0.smithi148.stdout:- clear_replica_map(); 2023-10-30T13:52:20.328 INFO:tasks.workunit.client.0.smithi148.stdout:- replica_nonce = EXPORT_NONCE; 2023-10-30T13:52:20.328 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(CDentry::STATE_AUTH); 2023-10-30T13:52:20.328 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_dirty()) 2023-10-30T13:52:20.328 INFO:tasks.workunit.client.0.smithi148.stdout:- mark_clean(); 2023-10-30T13:52:20.328 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_TEMPEXPORTING); 2023-10-30T13:52:20.329 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.329 INFO:tasks.workunit.client.0.smithi148.stdout:- void abort_export() { 2023-10-30T13:52:20.329 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_TEMPEXPORTING); 2023-10-30T13:52:20.329 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.329 INFO:tasks.workunit.client.0.smithi148.stdout:- void decode_import(bufferlist::iterator& blp, LogSegment *ls) { 2023-10-30T13:52:20.329 INFO:tasks.workunit.client.0.smithi148.stdout:- int nstate; 2023-10-30T13:52:20.330 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(nstate, blp); 2023-10-30T13:52:20.330 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(version, blp); 2023-10-30T13:52:20.330 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(projected_version, blp); 2023-10-30T13:52:20.330 INFO:tasks.workunit.client.0.smithi148.stdout:- lock._decode(blp); 2023-10-30T13:52:20.330 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(replica_map, blp); 2023-10-30T13:52:20.330 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.331 INFO:tasks.workunit.client.0.smithi148.stdout:- // twiddle 2023-10-30T13:52:20.331 INFO:tasks.workunit.client.0.smithi148.stdout:- state = 0; 2023-10-30T13:52:20.331 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(CDentry::STATE_AUTH); 2023-10-30T13:52:20.331 INFO:tasks.workunit.client.0.smithi148.stdout:- if (nstate & STATE_DIRTY) 2023-10-30T13:52:20.331 INFO:tasks.workunit.client.0.smithi148.stdout:- _mark_dirty(ls); 2023-10-30T13:52:20.332 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!replica_map.empty()) 2023-10-30T13:52:20.332 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_REPLICATED); 2023-10-30T13:52:20.332 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.332 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.332 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- locking -- 2023-10-30T13:52:20.333 INFO:tasks.workunit.client.0.smithi148.stdout:- SimpleLock* get_lock(int type) { 2023-10-30T13:52:20.333 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(type == LOCK_OTYPE_DN); 2023-10-30T13:52:20.333 INFO:tasks.workunit.client.0.smithi148.stdout:- return &lock; 2023-10-30T13:52:20.333 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.333 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_object_info(MDSCacheObjectInfo &info); 2023-10-30T13:52:20.334 INFO:tasks.workunit.client.0.smithi148.stdout:- void encode_lock_state(int type, bufferlist& bl); 2023-10-30T13:52:20.334 INFO:tasks.workunit.client.0.smithi148.stdout:- void decode_lock_state(int type, bufferlist& bl); 2023-10-30T13:52:20.334 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.334 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.334 INFO:tasks.workunit.client.0.smithi148.stdout:- ostream& print_db_line_prefix(ostream& out); 2023-10-30T13:52:20.335 INFO:tasks.workunit.client.0.smithi148.stdout:- void print(ostream& out); 2023-10-30T13:52:20.335 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.335 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class CDir; 2023-10-30T13:52:20.335 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:20.335 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.336 INFO:tasks.workunit.client.0.smithi148.stdout:-ostream& operator<<(ostream& out, CDentry& dn); 2023-10-30T13:52:20.336 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.336 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.336 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.336 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDentryDiscover { 2023-10-30T13:52:20.337 INFO:tasks.workunit.client.0.smithi148.stdout:- string dname; 2023-10-30T13:52:20.337 INFO:tasks.workunit.client.0.smithi148.stdout:- int replica_nonce; 2023-10-30T13:52:20.337 INFO:tasks.workunit.client.0.smithi148.stdout:- int lockstate; 2023-10-30T13:52:20.337 INFO:tasks.workunit.client.0.smithi148.stdout:- off_t dir_offset; 2023-10-30T13:52:20.337 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t remote_ino; 2023-10-30T13:52:20.337 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned char remote_d_type; 2023-10-30T13:52:20.338 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.338 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.338 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentryDiscover() {} 2023-10-30T13:52:20.338 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentryDiscover(CDentry *dn, int nonce) : 2023-10-30T13:52:20.338 INFO:tasks.workunit.client.0.smithi148.stdout:- dname(dn->get_name()), replica_nonce(nonce), 2023-10-30T13:52:20.339 INFO:tasks.workunit.client.0.smithi148.stdout:- lockstate(dn->lock.get_replica_state()), 2023-10-30T13:52:20.339 INFO:tasks.workunit.client.0.smithi148.stdout:- dir_offset(dn->get_dir_offset()), 2023-10-30T13:52:20.339 INFO:tasks.workunit.client.0.smithi148.stdout:- remote_ino(dn->get_remote_ino()), remote_d_type(dn->get_remote_d_type()) { } 2023-10-30T13:52:20.339 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.339 INFO:tasks.workunit.client.0.smithi148.stdout:- string& get_dname() { return dname; } 2023-10-30T13:52:20.339 INFO:tasks.workunit.client.0.smithi148.stdout:- int get_nonce() { return replica_nonce; } 2023-10-30T13:52:20.340 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_remote() { return remote_ino ? true:false; } 2023-10-30T13:52:20.340 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t get_remote_ino() { return remote_ino; } 2023-10-30T13:52:20.340 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned char get_remote_d_type() { return remote_d_type; } 2023-10-30T13:52:20.340 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.340 INFO:tasks.workunit.client.0.smithi148.stdout:- void update_dentry(CDentry *dn) { 2023-10-30T13:52:20.340 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->set_dir_offset(dir_offset); 2023-10-30T13:52:20.341 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->set_replica_nonce(replica_nonce); 2023-10-30T13:52:20.341 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.341 INFO:tasks.workunit.client.0.smithi148.stdout:- void init_dentry_lock(CDentry *dn) { 2023-10-30T13:52:20.341 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->lock.set_state( lockstate ); 2023-10-30T13:52:20.341 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.342 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.342 INFO:tasks.workunit.client.0.smithi148.stdout:- void _encode(bufferlist& bl) { 2023-10-30T13:52:20.342 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(dname, bl); 2023-10-30T13:52:20.342 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(dir_offset, bl); 2023-10-30T13:52:20.342 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(remote_ino, bl); 2023-10-30T13:52:20.342 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(remote_d_type, bl); 2023-10-30T13:52:20.343 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(replica_nonce, bl); 2023-10-30T13:52:20.343 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(lockstate, bl); 2023-10-30T13:52:20.343 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.343 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.343 INFO:tasks.workunit.client.0.smithi148.stdout:- void _decode(bufferlist& bl, int& off) { 2023-10-30T13:52:20.344 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(dname, bl, off); 2023-10-30T13:52:20.344 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(dir_offset, bl, off); 2023-10-30T13:52:20.344 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(remote_ino, bl, off); 2023-10-30T13:52:20.344 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(remote_d_type, bl, off); 2023-10-30T13:52:20.345 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(replica_nonce, bl, off); 2023-10-30T13:52:20.345 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(lockstate, bl, off); 2023-10-30T13:52:20.345 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.346 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.346 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:20.346 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.346 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.346 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.346 INFO:tasks.workunit.client.0.smithi148.stdout:-#endif 2023-10-30T13:52:20.347 INFO:tasks.workunit.client.0.smithi148.stdout:diff --git a/src/mds/CDir.cc b/src/mds/CDir.cc 2023-10-30T13:52:20.347 INFO:tasks.workunit.client.0.smithi148.stdout:index adaf5fa6c0d..e69de29bb2d 100644 2023-10-30T13:52:20.347 INFO:tasks.workunit.client.0.smithi148.stdout:--- a/src/mds/CDir.cc 2023-10-30T13:52:20.347 INFO:tasks.workunit.client.0.smithi148.stdout:+++ b/src/mds/CDir.cc 2023-10-30T13:52:20.347 INFO:tasks.workunit.client.0.smithi148.stdout:@@ -1,1676 +0,0 @@ 2023-10-30T13:52:20.348 INFO:tasks.workunit.client.0.smithi148.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2023-10-30T13:52:20.348 INFO:tasks.workunit.client.0.smithi148.stdout:-// vim: ts=8 sw=2 smarttab 2023-10-30T13:52:20.348 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.348 INFO:tasks.workunit.client.0.smithi148.stdout:- * Ceph - scalable distributed file system 2023-10-30T13:52:20.348 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.348 INFO:tasks.workunit.client.0.smithi148.stdout:- * Copyright (C) 2004-2006 Sage Weil 2023-10-30T13:52:20.349 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.349 INFO:tasks.workunit.client.0.smithi148.stdout:- * This is free software; you can redistribute it and/or 2023-10-30T13:52:20.349 INFO:tasks.workunit.client.0.smithi148.stdout:- * modify it under the terms of the GNU Lesser General Public 2023-10-30T13:52:20.349 INFO:tasks.workunit.client.0.smithi148.stdout:- * License version 2.1, as published by the Free Software 2023-10-30T13:52:20.349 INFO:tasks.workunit.client.0.smithi148.stdout:- * Foundation. See file COPYING. 2023-10-30T13:52:20.350 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.350 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.350 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.350 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.350 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/types.h" 2023-10-30T13:52:20.351 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.351 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CDir.h" 2023-10-30T13:52:20.351 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CDentry.h" 2023-10-30T13:52:20.351 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CInode.h" 2023-10-30T13:52:20.351 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.351 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "MDS.h" 2023-10-30T13:52:20.352 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "MDCache.h" 2023-10-30T13:52:20.352 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "MDSMap.h" 2023-10-30T13:52:20.352 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "LogSegment.h" 2023-10-30T13:52:20.352 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.353 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/Context.h" 2023-10-30T13:52:20.353 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "common/Clock.h" 2023-10-30T13:52:20.353 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.353 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "osdc/Objecter.h" 2023-10-30T13:52:20.354 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.354 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.354 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.354 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "config.h" 2023-10-30T13:52:20.354 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.355 INFO:tasks.workunit.client.0.smithi148.stdout:-#define dout(x) if (x <= g_conf.debug || x <= g_conf.debug_mds) *_dout << dbeginl << g_clock.now() << " mds" << cache->mds->get_nodeid() << ".cache.dir(" << this->dirfrag() << ") " 2023-10-30T13:52:20.355 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.355 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.355 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.356 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.356 INFO:tasks.workunit.client.0.smithi148.stdout:-// PINS 2023-10-30T13:52:20.356 INFO:tasks.workunit.client.0.smithi148.stdout:-//int cdir_pins[CDIR_NUM_PINS] = { 0,0,0,0,0,0,0,0,0,0,0,0,0,0 }; 2023-10-30T13:52:20.356 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.356 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.357 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.357 INFO:tasks.workunit.client.0.smithi148.stdout:-ostream& operator<<(ostream& out, CDir& dir) 2023-10-30T13:52:20.357 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.357 INFO:tasks.workunit.client.0.smithi148.stdout:- filepath path; 2023-10-30T13:52:20.357 INFO:tasks.workunit.client.0.smithi148.stdout:- dir.get_inode()->make_path(path); 2023-10-30T13:52:20.358 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "[dir " << dir.dirfrag() << " " << path << "/"; 2023-10-30T13:52:20.358 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.is_auth()) { 2023-10-30T13:52:20.358 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " auth"; 2023-10-30T13:52:20.358 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.is_replicated()) 2023-10-30T13:52:20.358 INFO:tasks.workunit.client.0.smithi148.stdout:- out << dir.get_replicas(); 2023-10-30T13:52:20.359 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.359 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " pv=" << dir.get_projected_version(); 2023-10-30T13:52:20.359 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " v=" << dir.get_version(); 2023-10-30T13:52:20.359 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " cv=" << dir.get_committing_version(); 2023-10-30T13:52:20.359 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "/" << dir.get_committed_version(); 2023-10-30T13:52:20.359 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "/" << dir.get_committed_version_equivalent(); 2023-10-30T13:52:20.360 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.360 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " rep@" << dir.authority(); 2023-10-30T13:52:20.360 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.get_replica_nonce() > 1) 2023-10-30T13:52:20.360 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "." << dir.get_replica_nonce(); 2023-10-30T13:52:20.360 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.361 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.361 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.is_rep()) out << " REP"; 2023-10-30T13:52:20.361 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.361 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.get_dir_auth() != CDIR_AUTH_DEFAULT) { 2023-10-30T13:52:20.361 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.get_dir_auth().second == CDIR_AUTH_UNKNOWN) 2023-10-30T13:52:20.362 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " dir_auth=" << dir.get_dir_auth().first; 2023-10-30T13:52:20.362 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.362 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " dir_auth=" << dir.get_dir_auth(); 2023-10-30T13:52:20.362 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.362 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.362 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.get_cum_auth_pins()) 2023-10-30T13:52:20.363 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " ap=" << dir.get_auth_pins() << "+" << dir.get_nested_auth_pins(); 2023-10-30T13:52:20.363 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.363 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " state=" << dir.get_state(); 2023-10-30T13:52:20.363 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.state_test(CDir::STATE_COMPLETE)) out << "|complete"; 2023-10-30T13:52:20.363 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.state_test(CDir::STATE_FREEZINGTREE)) out << "|freezingtree"; 2023-10-30T13:52:20.364 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.state_test(CDir::STATE_FROZENTREE)) out << "|frozentree"; 2023-10-30T13:52:20.364 INFO:tasks.workunit.client.0.smithi148.stdout:- //if (dir.state_test(CDir::STATE_FROZENTREELEAF)) out << "|frozentreeleaf"; 2023-10-30T13:52:20.364 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.state_test(CDir::STATE_FROZENDIR)) out << "|frozendir"; 2023-10-30T13:52:20.364 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.state_test(CDir::STATE_FREEZINGDIR)) out << "|freezingdir"; 2023-10-30T13:52:20.364 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.state_test(CDir::STATE_EXPORTBOUND)) out << "|exportbound"; 2023-10-30T13:52:20.364 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.state_test(CDir::STATE_IMPORTBOUND)) out << "|importbound"; 2023-10-30T13:52:20.365 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.365 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " sz=" << dir.get_nitems() << "+" << dir.get_nnull(); 2023-10-30T13:52:20.365 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.get_num_dirty()) 2023-10-30T13:52:20.365 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " dirty=" << dir.get_num_dirty(); 2023-10-30T13:52:20.365 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.365 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.366 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir.get_num_ref()) { 2023-10-30T13:52:20.366 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " |"; 2023-10-30T13:52:20.366 INFO:tasks.workunit.client.0.smithi148.stdout:- dir.print_pin_set(out); 2023-10-30T13:52:20.366 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.366 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.367 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " " << &dir; 2023-10-30T13:52:20.367 INFO:tasks.workunit.client.0.smithi148.stdout:- return out << "]"; 2023-10-30T13:52:20.367 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.367 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.367 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.368 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::print(ostream& out) 2023-10-30T13:52:20.368 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.368 INFO:tasks.workunit.client.0.smithi148.stdout:- out << *this; 2023-10-30T13:52:20.368 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.368 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.368 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.369 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.369 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.369 INFO:tasks.workunit.client.0.smithi148.stdout:-ostream& CDir::print_db_line_prefix(ostream& out) 2023-10-30T13:52:20.369 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.369 INFO:tasks.workunit.client.0.smithi148.stdout:- return out << g_clock.now() << " mds" << cache->mds->get_nodeid() << ".cache.dir(" << this->dirfrag() << ") "; 2023-10-30T13:52:20.370 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.370 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.370 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.370 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.370 INFO:tasks.workunit.client.0.smithi148.stdout:-// ------------------------------------------------------------------- 2023-10-30T13:52:20.371 INFO:tasks.workunit.client.0.smithi148.stdout:-// CDir 2023-10-30T13:52:20.371 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.371 INFO:tasks.workunit.client.0.smithi148.stdout:-CDir::CDir(CInode *in, frag_t fg, MDCache *mdcache, bool auth) : 2023-10-30T13:52:20.371 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty(this) 2023-10-30T13:52:20.371 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.371 INFO:tasks.workunit.client.0.smithi148.stdout:- inode = in; 2023-10-30T13:52:20.372 INFO:tasks.workunit.client.0.smithi148.stdout:- frag = fg; 2023-10-30T13:52:20.372 INFO:tasks.workunit.client.0.smithi148.stdout:- this->cache = mdcache; 2023-10-30T13:52:20.372 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.372 INFO:tasks.workunit.client.0.smithi148.stdout:- nitems = 0; 2023-10-30T13:52:20.372 INFO:tasks.workunit.client.0.smithi148.stdout:- nnull = 0; 2023-10-30T13:52:20.372 INFO:tasks.workunit.client.0.smithi148.stdout:- num_dirty = 0; 2023-10-30T13:52:20.373 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.373 INFO:tasks.workunit.client.0.smithi148.stdout:- state = STATE_INITIAL; 2023-10-30T13:52:20.373 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.373 INFO:tasks.workunit.client.0.smithi148.stdout:- projected_version = version = 0; 2023-10-30T13:52:20.373 INFO:tasks.workunit.client.0.smithi148.stdout:- committing_version = 0; 2023-10-30T13:52:20.374 INFO:tasks.workunit.client.0.smithi148.stdout:- committed_version_equivalent = committed_version = 0; 2023-10-30T13:52:20.374 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.374 INFO:tasks.workunit.client.0.smithi148.stdout:- // dir_auth 2023-10-30T13:52:20.374 INFO:tasks.workunit.client.0.smithi148.stdout:- dir_auth = CDIR_AUTH_DEFAULT; 2023-10-30T13:52:20.374 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.374 INFO:tasks.workunit.client.0.smithi148.stdout:- // auth 2023-10-30T13:52:20.375 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(in->is_dir()); 2023-10-30T13:52:20.375 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth) 2023-10-30T13:52:20.375 INFO:tasks.workunit.client.0.smithi148.stdout:- state |= STATE_AUTH; 2023-10-30T13:52:20.375 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.375 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins = 0; 2023-10-30T13:52:20.376 INFO:tasks.workunit.client.0.smithi148.stdout:- nested_auth_pins = 0; 2023-10-30T13:52:20.376 INFO:tasks.workunit.client.0.smithi148.stdout:- request_pins = 0; 2023-10-30T13:52:20.376 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.376 INFO:tasks.workunit.client.0.smithi148.stdout:- //hack_num_accessed = -1; 2023-10-30T13:52:20.376 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.377 INFO:tasks.workunit.client.0.smithi148.stdout:- dir_rep = REP_NONE; 2023-10-30T13:52:20.377 INFO:tasks.workunit.client.0.smithi148.stdout:- //dir_rep = REP_ALL; // hack: to wring out some bugs! FIXME FIXME 2023-10-30T13:52:20.377 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.377 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.377 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.377 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.378 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.378 INFO:tasks.workunit.client.0.smithi148.stdout:-/*** 2023-10-30T13:52:20.378 INFO:tasks.workunit.client.0.smithi148.stdout:- * linking fun 2023-10-30T13:52:20.378 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.378 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.379 INFO:tasks.workunit.client.0.smithi148.stdout:-CDentry* CDir::add_null_dentry(const string& dname) 2023-10-30T13:52:20.379 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.379 INFO:tasks.workunit.client.0.smithi148.stdout:- // foreign 2023-10-30T13:52:20.379 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(lookup(dname) == 0); 2023-10-30T13:52:20.379 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.379 INFO:tasks.workunit.client.0.smithi148.stdout:- // create dentry 2023-10-30T13:52:20.380 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry* dn = new CDentry(dname, 0); 2023-10-30T13:52:20.380 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth()) 2023-10-30T13:52:20.380 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->state_set(CDentry::STATE_AUTH); 2023-10-30T13:52:20.380 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->lru.lru_insert_mid(dn); 2023-10-30T13:52:20.380 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.381 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->dir = this; 2023-10-30T13:52:20.381 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->version = projected_version; 2023-10-30T13:52:20.381 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.381 INFO:tasks.workunit.client.0.smithi148.stdout:- // add to dir 2023-10-30T13:52:20.381 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(items.count(dn->name) == 0); 2023-10-30T13:52:20.381 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(null_items.count(dn->name) == 0); 2023-10-30T13:52:20.382 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.382 INFO:tasks.workunit.client.0.smithi148.stdout:- items[dn->name] = dn; 2023-10-30T13:52:20.382 INFO:tasks.workunit.client.0.smithi148.stdout:- nnull++; 2023-10-30T13:52:20.382 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.382 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "add_null_dentry " << *dn << dendl; 2023-10-30T13:52:20.383 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.383 INFO:tasks.workunit.client.0.smithi148.stdout:- // pin? 2023-10-30T13:52:20.383 INFO:tasks.workunit.client.0.smithi148.stdout:- if (nnull + nitems == 1) get(PIN_CHILD); 2023-10-30T13:52:20.383 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.383 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nnull + nitems == items.size()); 2023-10-30T13:52:20.384 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(nnull == null_items.size()); 2023-10-30T13:52:20.384 INFO:tasks.workunit.client.0.smithi148.stdout:- return dn; 2023-10-30T13:52:20.384 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.384 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.384 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.385 INFO:tasks.workunit.client.0.smithi148.stdout:-CDentry* CDir::add_primary_dentry(const string& dname, CInode *in) 2023-10-30T13:52:20.385 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.385 INFO:tasks.workunit.client.0.smithi148.stdout:- // primary 2023-10-30T13:52:20.385 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(lookup(dname) == 0); 2023-10-30T13:52:20.385 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.386 INFO:tasks.workunit.client.0.smithi148.stdout:- // create dentry 2023-10-30T13:52:20.386 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry* dn = new CDentry(dname, 0); 2023-10-30T13:52:20.386 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth()) 2023-10-30T13:52:20.386 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->state_set(CDentry::STATE_AUTH); 2023-10-30T13:52:20.386 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->lru.lru_insert_mid(dn); 2023-10-30T13:52:20.387 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.387 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->dir = this; 2023-10-30T13:52:20.387 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->version = projected_version; 2023-10-30T13:52:20.387 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.387 INFO:tasks.workunit.client.0.smithi148.stdout:- // add to dir 2023-10-30T13:52:20.388 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(items.count(dn->name) == 0); 2023-10-30T13:52:20.388 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(null_items.count(dn->name) == 0); 2023-10-30T13:52:20.388 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.388 INFO:tasks.workunit.client.0.smithi148.stdout:- items[dn->name] = dn; 2023-10-30T13:52:20.388 INFO:tasks.workunit.client.0.smithi148.stdout:- link_inode_work( dn, in ); 2023-10-30T13:52:20.388 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.389 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "add_primary_dentry " << *dn << dendl; 2023-10-30T13:52:20.389 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.389 INFO:tasks.workunit.client.0.smithi148.stdout:- // pin? 2023-10-30T13:52:20.389 INFO:tasks.workunit.client.0.smithi148.stdout:- if (nnull + nitems == 1) get(PIN_CHILD); 2023-10-30T13:52:20.390 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.390 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nnull + nitems == items.size()); 2023-10-30T13:52:20.390 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(nnull == null_items.size()); 2023-10-30T13:52:20.390 INFO:tasks.workunit.client.0.smithi148.stdout:- return dn; 2023-10-30T13:52:20.390 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.391 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.391 INFO:tasks.workunit.client.0.smithi148.stdout:-CDentry* CDir::add_remote_dentry(const string& dname, inodeno_t ino, unsigned char d_type) 2023-10-30T13:52:20.391 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.391 INFO:tasks.workunit.client.0.smithi148.stdout:- // foreign 2023-10-30T13:52:20.391 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(lookup(dname) == 0); 2023-10-30T13:52:20.392 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.392 INFO:tasks.workunit.client.0.smithi148.stdout:- // create dentry 2023-10-30T13:52:20.392 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry* dn = new CDentry(dname, ino, d_type); 2023-10-30T13:52:20.392 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth()) 2023-10-30T13:52:20.392 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->state_set(CDentry::STATE_AUTH); 2023-10-30T13:52:20.393 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->lru.lru_insert_mid(dn); 2023-10-30T13:52:20.393 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.393 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->dir = this; 2023-10-30T13:52:20.393 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->version = projected_version; 2023-10-30T13:52:20.393 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.393 INFO:tasks.workunit.client.0.smithi148.stdout:- // add to dir 2023-10-30T13:52:20.394 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(items.count(dn->name) == 0); 2023-10-30T13:52:20.394 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(null_items.count(dn->name) == 0); 2023-10-30T13:52:20.394 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.394 INFO:tasks.workunit.client.0.smithi148.stdout:- items[dn->name] = dn; 2023-10-30T13:52:20.394 INFO:tasks.workunit.client.0.smithi148.stdout:- nitems++; 2023-10-30T13:52:20.395 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.395 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "add_remote_dentry " << *dn << dendl; 2023-10-30T13:52:20.395 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.395 INFO:tasks.workunit.client.0.smithi148.stdout:- // pin? 2023-10-30T13:52:20.395 INFO:tasks.workunit.client.0.smithi148.stdout:- if (nnull + nitems == 1) get(PIN_CHILD); 2023-10-30T13:52:20.396 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.396 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nnull + nitems == items.size()); 2023-10-30T13:52:20.396 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(nnull == null_items.size()); 2023-10-30T13:52:20.396 INFO:tasks.workunit.client.0.smithi148.stdout:- return dn; 2023-10-30T13:52:20.396 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.396 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.397 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.397 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.397 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::remove_dentry(CDentry *dn) 2023-10-30T13:52:20.397 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.397 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "remove_dentry " << *dn << dendl; 2023-10-30T13:52:20.398 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.398 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->inode) { 2023-10-30T13:52:20.398 INFO:tasks.workunit.client.0.smithi148.stdout:- // detach inode and dentry 2023-10-30T13:52:20.398 INFO:tasks.workunit.client.0.smithi148.stdout:- unlink_inode_work(dn); 2023-10-30T13:52:20.398 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.398 INFO:tasks.workunit.client.0.smithi148.stdout:- // remove from null list 2023-10-30T13:52:20.399 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(null_items.count(dn->name) == 1); 2023-10-30T13:52:20.399 INFO:tasks.workunit.client.0.smithi148.stdout:- //null_items.erase(dn->name); 2023-10-30T13:52:20.399 INFO:tasks.workunit.client.0.smithi148.stdout:- nnull--; 2023-10-30T13:52:20.399 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.400 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.400 INFO:tasks.workunit.client.0.smithi148.stdout:- // remove from list 2023-10-30T13:52:20.400 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(items.count(dn->name) == 1); 2023-10-30T13:52:20.400 INFO:tasks.workunit.client.0.smithi148.stdout:- items.erase(dn->name); 2023-10-30T13:52:20.400 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.400 INFO:tasks.workunit.client.0.smithi148.stdout:- // adjust dirty counter? 2023-10-30T13:52:20.401 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->state_test(CDentry::STATE_DIRTY)) 2023-10-30T13:52:20.401 INFO:tasks.workunit.client.0.smithi148.stdout:- num_dirty--; 2023-10-30T13:52:20.401 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.401 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->lru.lru_remove(dn); 2023-10-30T13:52:20.401 INFO:tasks.workunit.client.0.smithi148.stdout:- delete dn; 2023-10-30T13:52:20.401 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.402 INFO:tasks.workunit.client.0.smithi148.stdout:- // unpin? 2023-10-30T13:52:20.402 INFO:tasks.workunit.client.0.smithi148.stdout:- if (nnull + nitems == 0) put(PIN_CHILD); 2023-10-30T13:52:20.402 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.402 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nnull + nitems == items.size()); 2023-10-30T13:52:20.402 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(nnull == null_items.size()); 2023-10-30T13:52:20.402 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.403 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.403 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::link_remote_inode(CDentry *dn, inodeno_t ino, unsigned char d_type) 2023-10-30T13:52:20.403 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.403 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "link_remote_inode " << *dn << " remote " << ino << dendl; 2023-10-30T13:52:20.403 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn->is_null()); 2023-10-30T13:52:20.403 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.404 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->set_remote(ino, d_type); 2023-10-30T13:52:20.404 INFO:tasks.workunit.client.0.smithi148.stdout:- nitems++; 2023-10-30T13:52:20.404 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->clear_dir_offset(); 2023-10-30T13:52:20.404 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.404 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(null_items.count(dn->name) == 1); 2023-10-30T13:52:20.404 INFO:tasks.workunit.client.0.smithi148.stdout:- //null_items.erase(dn->name); 2023-10-30T13:52:20.405 INFO:tasks.workunit.client.0.smithi148.stdout:- nnull--; 2023-10-30T13:52:20.405 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nnull + nitems == items.size()); 2023-10-30T13:52:20.405 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.405 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.405 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::link_primary_inode(CDentry *dn, CInode *in) 2023-10-30T13:52:20.405 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.406 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "link_primary_inode " << *dn << " " << *in << dendl; 2023-10-30T13:52:20.406 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn->is_null()); 2023-10-30T13:52:20.406 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.406 INFO:tasks.workunit.client.0.smithi148.stdout:- link_inode_work(dn,in); 2023-10-30T13:52:20.406 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->clear_dir_offset(); 2023-10-30T13:52:20.407 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.407 INFO:tasks.workunit.client.0.smithi148.stdout:- // remove from null list 2023-10-30T13:52:20.407 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(null_items.count(dn->name) == 1); 2023-10-30T13:52:20.407 INFO:tasks.workunit.client.0.smithi148.stdout:- //null_items.erase(dn->name); 2023-10-30T13:52:20.407 INFO:tasks.workunit.client.0.smithi148.stdout:- nnull--; 2023-10-30T13:52:20.407 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.408 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nnull + nitems == items.size()); 2023-10-30T13:52:20.408 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(nnull == null_items.size()); 2023-10-30T13:52:20.408 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.408 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.409 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::link_inode_work( CDentry *dn, CInode *in) 2023-10-30T13:52:20.409 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.409 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn->inode == 0); 2023-10-30T13:52:20.409 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->inode = in; 2023-10-30T13:52:20.409 INFO:tasks.workunit.client.0.smithi148.stdout:- in->set_primary_parent(dn); 2023-10-30T13:52:20.409 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.410 INFO:tasks.workunit.client.0.smithi148.stdout:- nitems++; // adjust dir size 2023-10-30T13:52:20.410 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.410 INFO:tasks.workunit.client.0.smithi148.stdout:- // set inode version 2023-10-30T13:52:20.410 INFO:tasks.workunit.client.0.smithi148.stdout:- //in->inode.version = dn->get_version(); 2023-10-30T13:52:20.410 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.410 INFO:tasks.workunit.client.0.smithi148.stdout:- // pin dentry? 2023-10-30T13:52:20.411 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in->get_num_ref()) 2023-10-30T13:52:20.411 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->get(CDentry::PIN_INODEPIN); 2023-10-30T13:52:20.411 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.411 INFO:tasks.workunit.client.0.smithi148.stdout:- // adjust auth pin count 2023-10-30T13:52:20.411 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in->auth_pins + in->nested_auth_pins) 2023-10-30T13:52:20.411 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->adjust_nested_auth_pins(in->auth_pins + in->nested_auth_pins); 2023-10-30T13:52:20.412 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.412 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.412 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::unlink_inode( CDentry *dn ) 2023-10-30T13:52:20.412 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.412 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->is_remote()) { 2023-10-30T13:52:20.412 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "unlink_inode " << *dn << dendl; 2023-10-30T13:52:20.413 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.413 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "unlink_inode " << *dn << " " << *dn->inode << dendl; 2023-10-30T13:52:20.413 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.413 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.413 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->clear_dir_offset(); 2023-10-30T13:52:20.413 INFO:tasks.workunit.client.0.smithi148.stdout:- unlink_inode_work(dn); 2023-10-30T13:52:20.414 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.414 INFO:tasks.workunit.client.0.smithi148.stdout:- // add to null list 2023-10-30T13:52:20.414 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(null_items.count(dn->name) == 0); 2023-10-30T13:52:20.414 INFO:tasks.workunit.client.0.smithi148.stdout:- //null_items[dn->name] = dn; 2023-10-30T13:52:20.414 INFO:tasks.workunit.client.0.smithi148.stdout:- nnull++; 2023-10-30T13:52:20.415 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.415 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nnull + nitems == items.size()); 2023-10-30T13:52:20.415 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(nnull == null_items.size()); 2023-10-30T13:52:20.415 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.415 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.415 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::try_remove_unlinked_dn(CDentry *dn) 2023-10-30T13:52:20.416 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.416 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn->dir == this); 2023-10-30T13:52:20.416 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn->is_null()); 2023-10-30T13:52:20.416 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn->is_dirty()); 2023-10-30T13:52:20.416 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.417 INFO:tasks.workunit.client.0.smithi148.stdout:- /* FIXME: there is a bug in this. i think new dentries are properly 2023-10-30T13:52:20.417 INFO:tasks.workunit.client.0.smithi148.stdout:- identified.. e.g. maybe a dentry exists, is committed, is removed, is now 2023-10-30T13:52:20.417 INFO:tasks.workunit.client.0.smithi148.stdout:- dirty+null, then reused and mistakenly considered new.. then it is removed, 2023-10-30T13:52:20.417 INFO:tasks.workunit.client.0.smithi148.stdout:- we remove it here, the dir is fetched, and the dentry exists again. 2023-10-30T13:52:20.417 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.417 INFO:tasks.workunit.client.0.smithi148.stdout:- somethign like that... 2023-10-30T13:52:20.418 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.418 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.418 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.418 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.418 INFO:tasks.workunit.client.0.smithi148.stdout:- // no pins (besides dirty)? 2023-10-30T13:52:20.419 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->get_num_ref() != 1) 2023-10-30T13:52:20.419 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.419 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.419 INFO:tasks.workunit.client.0.smithi148.stdout:- // was the dn new? or is the dir complete (i.e. we don't need negatives)? 2023-10-30T13:52:20.419 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->is_new() || is_complete()) { 2023-10-30T13:52:20.419 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "try_remove_unlinked_dn " << *dn << " in " << *this << dendl; 2023-10-30T13:52:20.420 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->mark_clean(); 2023-10-30T13:52:20.420 INFO:tasks.workunit.client.0.smithi148.stdout:- remove_dentry(dn); 2023-10-30T13:52:20.420 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.420 INFO:tasks.workunit.client.0.smithi148.stdout:- if (version == projected_version && 2023-10-30T13:52:20.420 INFO:tasks.workunit.client.0.smithi148.stdout:- committing_version == committed_version && 2023-10-30T13:52:20.420 INFO:tasks.workunit.client.0.smithi148.stdout:- num_dirty == 0) { 2023-10-30T13:52:20.421 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "try_remove_unlinked_dn committed_equivalent now " << version 2023-10-30T13:52:20.421 INFO:tasks.workunit.client.0.smithi148.stdout:- << " vs committed " << committed_version 2023-10-30T13:52:20.421 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.421 INFO:tasks.workunit.client.0.smithi148.stdout:- committed_version_equivalent = committed_version; 2023-10-30T13:52:20.421 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.421 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.422 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.422 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.422 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.422 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.422 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::unlink_inode_work( CDentry *dn ) 2023-10-30T13:52:20.423 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.423 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *in = dn->inode; 2023-10-30T13:52:20.423 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.423 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->is_remote()) { 2023-10-30T13:52:20.423 INFO:tasks.workunit.client.0.smithi148.stdout:- // remote 2023-10-30T13:52:20.423 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in) 2023-10-30T13:52:20.424 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->unlink_remote(); 2023-10-30T13:52:20.424 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.424 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->set_remote(0, 0); 2023-10-30T13:52:20.424 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.424 INFO:tasks.workunit.client.0.smithi148.stdout:- // primary 2023-10-30T13:52:20.424 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn->is_primary()); 2023-10-30T13:52:20.425 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.425 INFO:tasks.workunit.client.0.smithi148.stdout:- // unpin dentry? 2023-10-30T13:52:20.425 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in->get_num_ref()) 2023-10-30T13:52:20.425 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->put(CDentry::PIN_INODEPIN); 2023-10-30T13:52:20.425 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.425 INFO:tasks.workunit.client.0.smithi148.stdout:- // unlink auth_pin count 2023-10-30T13:52:20.426 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in->auth_pins + in->nested_auth_pins) 2023-10-30T13:52:20.426 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->adjust_nested_auth_pins(0 - (in->auth_pins + in->nested_auth_pins)); 2023-10-30T13:52:20.426 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.426 INFO:tasks.workunit.client.0.smithi148.stdout:- // detach inode 2023-10-30T13:52:20.426 INFO:tasks.workunit.client.0.smithi148.stdout:- in->remove_primary_parent(dn); 2023-10-30T13:52:20.427 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->inode = 0; 2023-10-30T13:52:20.427 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.427 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.427 INFO:tasks.workunit.client.0.smithi148.stdout:- nitems--; // adjust dir size 2023-10-30T13:52:20.427 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.428 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.428 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::remove_null_dentries() { 2023-10-30T13:52:20.428 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "remove_null_dentries " << *this << dendl; 2023-10-30T13:52:20.428 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.428 INFO:tasks.workunit.client.0.smithi148.stdout:- list dns; 2023-10-30T13:52:20.428 INFO:tasks.workunit.client.0.smithi148.stdout:- for (CDir::map_t::iterator it = items.begin(); 2023-10-30T13:52:20.429 INFO:tasks.workunit.client.0.smithi148.stdout:- it != items.end(); 2023-10-30T13:52:20.429 INFO:tasks.workunit.client.0.smithi148.stdout:- it++) { 2023-10-30T13:52:20.429 INFO:tasks.workunit.client.0.smithi148.stdout:- if (it->second->is_null()) 2023-10-30T13:52:20.429 INFO:tasks.workunit.client.0.smithi148.stdout:- dns.push_back(it->second); 2023-10-30T13:52:20.429 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.430 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.430 INFO:tasks.workunit.client.0.smithi148.stdout:- for (list::iterator it = dns.begin(); 2023-10-30T13:52:20.430 INFO:tasks.workunit.client.0.smithi148.stdout:- it != dns.end(); 2023-10-30T13:52:20.430 INFO:tasks.workunit.client.0.smithi148.stdout:- it++) { 2023-10-30T13:52:20.430 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry *dn = *it; 2023-10-30T13:52:20.431 INFO:tasks.workunit.client.0.smithi148.stdout:- remove_dentry(dn); 2023-10-30T13:52:20.431 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.431 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(null_items.empty()); 2023-10-30T13:52:20.431 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nnull == 0); 2023-10-30T13:52:20.431 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nnull + nitems == items.size()); 2023-10-30T13:52:20.432 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.432 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.432 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.432 INFO:tasks.workunit.client.0.smithi148.stdout:-/** 2023-10-30T13:52:20.433 INFO:tasks.workunit.client.0.smithi148.stdout:- * steal_dentry -- semi-violently move a dentry from one CDir to another 2023-10-30T13:52:20.433 INFO:tasks.workunit.client.0.smithi148.stdout:- * (*) violently, in that nitems, most pins, etc. are not correctly maintained 2023-10-30T13:52:20.433 INFO:tasks.workunit.client.0.smithi148.stdout:- * on the old CDir corpse; must call purge_stolen() when finished. 2023-10-30T13:52:20.433 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.433 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::steal_dentry(CDentry *dn) 2023-10-30T13:52:20.433 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.434 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << "steal_dentry " << *dn << dendl; 2023-10-30T13:52:20.434 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.434 INFO:tasks.workunit.client.0.smithi148.stdout:- items[dn->name] = dn; 2023-10-30T13:52:20.434 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.434 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->dir->items.erase(dn->name); 2023-10-30T13:52:20.435 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->dir->items.empty()) 2023-10-30T13:52:20.435 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->dir->put(PIN_CHILD); 2023-10-30T13:52:20.435 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.435 INFO:tasks.workunit.client.0.smithi148.stdout:- if (nnull + nitems == 0) 2023-10-30T13:52:20.436 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_CHILD); 2023-10-30T13:52:20.436 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->is_null()) 2023-10-30T13:52:20.436 INFO:tasks.workunit.client.0.smithi148.stdout:- nnull++; 2023-10-30T13:52:20.436 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.436 INFO:tasks.workunit.client.0.smithi148.stdout:- nitems++; 2023-10-30T13:52:20.436 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.437 INFO:tasks.workunit.client.0.smithi148.stdout:- nested_auth_pins += dn->auth_pins + dn->nested_auth_pins; 2023-10-30T13:52:20.437 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->is_dirty()) 2023-10-30T13:52:20.437 INFO:tasks.workunit.client.0.smithi148.stdout:- num_dirty++; 2023-10-30T13:52:20.437 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.438 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->dir = this; 2023-10-30T13:52:20.438 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.438 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.438 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::purge_stolen(list& waiters) 2023-10-30T13:52:20.438 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.438 INFO:tasks.workunit.client.0.smithi148.stdout:- // take waiters _before_ unfreeze... 2023-10-30T13:52:20.439 INFO:tasks.workunit.client.0.smithi148.stdout:- take_waiting(WAIT_ANY, waiters); 2023-10-30T13:52:20.439 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.439 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth()) { 2023-10-30T13:52:20.439 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_frozen_dir()); 2023-10-30T13:52:20.439 INFO:tasks.workunit.client.0.smithi148.stdout:- unfreeze_dir(); 2023-10-30T13:52:20.440 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.440 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.440 INFO:tasks.workunit.client.0.smithi148.stdout:- nnull = nitems = 0; 2023-10-30T13:52:20.440 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.440 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth()) 2023-10-30T13:52:20.441 INFO:tasks.workunit.client.0.smithi148.stdout:- clear_replica_map(); 2023-10-30T13:52:20.441 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_dirty()) mark_clean(); 2023-10-30T13:52:20.441 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_IMPORTBOUND)) put(PIN_IMPORTBOUND); 2023-10-30T13:52:20.441 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_EXPORTBOUND)) put(PIN_EXPORTBOUND); 2023-10-30T13:52:20.441 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.441 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth_pins > 0) put(PIN_AUTHPIN); 2023-10-30T13:52:20.442 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.442 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(get_num_ref() == (state_test(STATE_STICKY) ? 1:0)); 2023-10-30T13:52:20.442 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.442 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.442 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::init_fragment_pins() 2023-10-30T13:52:20.443 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.443 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!replica_map.empty()) get(PIN_REPLICATED); 2023-10-30T13:52:20.443 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_DIRTY)) get(PIN_DIRTY); 2023-10-30T13:52:20.443 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_EXPORTBOUND)) get(PIN_EXPORTBOUND); 2023-10-30T13:52:20.443 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_IMPORTBOUND)) get(PIN_IMPORTBOUND); 2023-10-30T13:52:20.443 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.444 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.444 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::split(int bits, list& subs, list& waiters) 2023-10-30T13:52:20.444 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.444 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "split by " << bits << " bits on " << *this << dendl; 2023-10-30T13:52:20.444 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.445 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cache->mds->logger) cache->mds->logger->inc("dir_sp"); 2023-10-30T13:52:20.445 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.445 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_complete() || !is_auth()); 2023-10-30T13:52:20.445 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.445 INFO:tasks.workunit.client.0.smithi148.stdout:- list frags; 2023-10-30T13:52:20.446 INFO:tasks.workunit.client.0.smithi148.stdout:- frag.split(bits, frags); 2023-10-30T13:52:20.446 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.446 INFO:tasks.workunit.client.0.smithi148.stdout:- vector subfrags(1 << bits); 2023-10-30T13:52:20.446 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.446 INFO:tasks.workunit.client.0.smithi148.stdout:- double fac = 1.0 / (double)(1 << bits); // for scaling load vecs 2023-10-30T13:52:20.446 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.447 INFO:tasks.workunit.client.0.smithi148.stdout:- // create subfrag dirs 2023-10-30T13:52:20.447 INFO:tasks.workunit.client.0.smithi148.stdout:- int n = 0; 2023-10-30T13:52:20.447 INFO:tasks.workunit.client.0.smithi148.stdout:- for (list::iterator p = frags.begin(); p != frags.end(); ++p) { 2023-10-30T13:52:20.447 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *f = new CDir(inode, *p, cache, is_auth()); 2023-10-30T13:52:20.447 INFO:tasks.workunit.client.0.smithi148.stdout:- f->state_set(state & MASK_STATE_FRAGMENT_KEPT); 2023-10-30T13:52:20.448 INFO:tasks.workunit.client.0.smithi148.stdout:- f->replica_map = replica_map; 2023-10-30T13:52:20.448 INFO:tasks.workunit.client.0.smithi148.stdout:- f->dir_auth = dir_auth; 2023-10-30T13:52:20.448 INFO:tasks.workunit.client.0.smithi148.stdout:- f->init_fragment_pins(); 2023-10-30T13:52:20.448 INFO:tasks.workunit.client.0.smithi148.stdout:- f->version = version; 2023-10-30T13:52:20.448 INFO:tasks.workunit.client.0.smithi148.stdout:- f->projected_version = projected_version; 2023-10-30T13:52:20.448 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.449 INFO:tasks.workunit.client.0.smithi148.stdout:- f->pop_me = pop_me; 2023-10-30T13:52:20.449 INFO:tasks.workunit.client.0.smithi148.stdout:- f->pop_me *= fac; 2023-10-30T13:52:20.449 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.449 INFO:tasks.workunit.client.0.smithi148.stdout:- // FIXME; this is an approximation 2023-10-30T13:52:20.449 INFO:tasks.workunit.client.0.smithi148.stdout:- f->pop_nested = pop_nested; 2023-10-30T13:52:20.450 INFO:tasks.workunit.client.0.smithi148.stdout:- f->pop_nested *= fac; 2023-10-30T13:52:20.450 INFO:tasks.workunit.client.0.smithi148.stdout:- f->pop_auth_subtree = pop_auth_subtree; 2023-10-30T13:52:20.450 INFO:tasks.workunit.client.0.smithi148.stdout:- f->pop_auth_subtree *= fac; 2023-10-30T13:52:20.450 INFO:tasks.workunit.client.0.smithi148.stdout:- f->pop_auth_subtree_nested = pop_auth_subtree_nested; 2023-10-30T13:52:20.450 INFO:tasks.workunit.client.0.smithi148.stdout:- f->pop_auth_subtree_nested *= fac; 2023-10-30T13:52:20.451 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.451 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << " subfrag " << *p << " " << *f << dendl; 2023-10-30T13:52:20.451 INFO:tasks.workunit.client.0.smithi148.stdout:- subfrags[n++] = f; 2023-10-30T13:52:20.451 INFO:tasks.workunit.client.0.smithi148.stdout:- subs.push_back(f); 2023-10-30T13:52:20.451 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->add_dirfrag(f); 2023-10-30T13:52:20.451 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.452 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.452 INFO:tasks.workunit.client.0.smithi148.stdout:- // repartition dentries 2023-10-30T13:52:20.452 INFO:tasks.workunit.client.0.smithi148.stdout:- while (!items.empty()) { 2023-10-30T13:52:20.452 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir::map_t::iterator p = items.begin(); 2023-10-30T13:52:20.452 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.453 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry *dn = p->second; 2023-10-30T13:52:20.453 INFO:tasks.workunit.client.0.smithi148.stdout:- frag_t subfrag = inode->pick_dirfrag(p->first); 2023-10-30T13:52:20.453 INFO:tasks.workunit.client.0.smithi148.stdout:- int n = subfrag.value() >> frag.bits(); 2023-10-30T13:52:20.453 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << " subfrag " << subfrag << " n=" << n << " for " << p->first << dendl; 2023-10-30T13:52:20.453 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *f = subfrags[n]; 2023-10-30T13:52:20.454 INFO:tasks.workunit.client.0.smithi148.stdout:- f->steal_dentry(dn); 2023-10-30T13:52:20.454 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.454 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.454 INFO:tasks.workunit.client.0.smithi148.stdout:- purge_stolen(waiters); 2023-10-30T13:52:20.454 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->close_dirfrag(frag); // selft deletion, watch out. 2023-10-30T13:52:20.455 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.455 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.455 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::merge(int bits, list& waiters) 2023-10-30T13:52:20.455 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.456 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "merge by " << bits << " bits" << dendl; 2023-10-30T13:52:20.456 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.456 INFO:tasks.workunit.client.0.smithi148.stdout:- list frags; 2023-10-30T13:52:20.456 INFO:tasks.workunit.client.0.smithi148.stdout:- frag.split(bits, frags); 2023-10-30T13:52:20.456 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.456 INFO:tasks.workunit.client.0.smithi148.stdout:- for (list::iterator p = frags.begin(); p != frags.end(); ++p) { 2023-10-30T13:52:20.457 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir = inode->get_or_open_dirfrag(cache, *p); 2023-10-30T13:52:20.457 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dir->is_complete()); 2023-10-30T13:52:20.457 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << " subfrag " << *p << " " << *dir << dendl; 2023-10-30T13:52:20.457 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.457 INFO:tasks.workunit.client.0.smithi148.stdout:- // steal dentries 2023-10-30T13:52:20.458 INFO:tasks.workunit.client.0.smithi148.stdout:- while (!dir->items.empty()) 2023-10-30T13:52:20.458 INFO:tasks.workunit.client.0.smithi148.stdout:- steal_dentry(dir->items.begin()->second); 2023-10-30T13:52:20.458 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.458 INFO:tasks.workunit.client.0.smithi148.stdout:- // merge replica map 2023-10-30T13:52:20.458 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator p = dir->replica_map.begin(); 2023-10-30T13:52:20.459 INFO:tasks.workunit.client.0.smithi148.stdout:- p != dir->replica_map.end(); 2023-10-30T13:52:20.459 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.459 INFO:tasks.workunit.client.0.smithi148.stdout:- replica_map[p->first] = MAX(replica_map[p->first], p->second); 2023-10-30T13:52:20.459 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.459 INFO:tasks.workunit.client.0.smithi148.stdout:- // merge state 2023-10-30T13:52:20.459 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(dir->get_state() & MASK_STATE_FRAGMENT_KEPT); 2023-10-30T13:52:20.460 INFO:tasks.workunit.client.0.smithi148.stdout:- dir_auth = dir->dir_auth; 2023-10-30T13:52:20.460 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.460 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->purge_stolen(waiters); 2023-10-30T13:52:20.460 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->close_dirfrag(dir->get_frag()); 2023-10-30T13:52:20.460 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.461 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.461 INFO:tasks.workunit.client.0.smithi148.stdout:- init_fragment_pins(); 2023-10-30T13:52:20.461 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.461 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.461 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.462 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.462 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.462 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.462 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.462 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.463 INFO:tasks.workunit.client.0.smithi148.stdout:-CDirDiscover *CDir::replicate_to(int mds) 2023-10-30T13:52:20.463 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.463 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.463 INFO:tasks.workunit.client.0.smithi148.stdout:- return new CDirDiscover( this, add_replica(mds) ); 2023-10-30T13:52:20.463 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.463 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.464 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.464 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.464 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.464 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.464 INFO:tasks.workunit.client.0.smithi148.stdout:-/**************************************** 2023-10-30T13:52:20.464 INFO:tasks.workunit.client.0.smithi148.stdout:- * WAITING 2023-10-30T13:52:20.465 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.465 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.465 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::add_dentry_waiter(const string& dname, Context *c) 2023-10-30T13:52:20.465 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.465 INFO:tasks.workunit.client.0.smithi148.stdout:- if (waiting_on_dentry.empty()) 2023-10-30T13:52:20.466 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_DNWAITER); 2023-10-30T13:52:20.466 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_on_dentry[dname].push_back(c); 2023-10-30T13:52:20.466 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "add_dentry_waiter dentry " << dname << " " << c << " on " << *this << dendl; 2023-10-30T13:52:20.466 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.466 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.466 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::take_dentry_waiting(const string& dname, list& ls) 2023-10-30T13:52:20.467 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.467 INFO:tasks.workunit.client.0.smithi148.stdout:- if (waiting_on_dentry.empty()) return; 2023-10-30T13:52:20.467 INFO:tasks.workunit.client.0.smithi148.stdout:- if (waiting_on_dentry.count(dname) == 0) return; 2023-10-30T13:52:20.467 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "take_dentry_waiting dentry " << dname 2023-10-30T13:52:20.467 INFO:tasks.workunit.client.0.smithi148.stdout:- << " x " << waiting_on_dentry[dname].size() 2023-10-30T13:52:20.467 INFO:tasks.workunit.client.0.smithi148.stdout:- << " on " << *this << dendl; 2023-10-30T13:52:20.468 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.splice(ls.end(), waiting_on_dentry[dname]); 2023-10-30T13:52:20.468 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_on_dentry.erase(dname); 2023-10-30T13:52:20.468 INFO:tasks.workunit.client.0.smithi148.stdout:- if (waiting_on_dentry.empty()) 2023-10-30T13:52:20.468 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_DNWAITER); 2023-10-30T13:52:20.468 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.469 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.469 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::add_ino_waiter(inodeno_t ino, Context *c) 2023-10-30T13:52:20.469 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.469 INFO:tasks.workunit.client.0.smithi148.stdout:- if (waiting_on_ino.empty()) 2023-10-30T13:52:20.469 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_INOWAITER); 2023-10-30T13:52:20.469 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_on_ino[ino].push_back(c); 2023-10-30T13:52:20.470 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "add_ino_waiter ino " << ino << " " << c << " on " << *this << dendl; 2023-10-30T13:52:20.470 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.470 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.470 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::take_ino_waiting(inodeno_t ino, list& ls) 2023-10-30T13:52:20.470 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.471 INFO:tasks.workunit.client.0.smithi148.stdout:- if (waiting_on_ino.empty()) return; 2023-10-30T13:52:20.471 INFO:tasks.workunit.client.0.smithi148.stdout:- if (waiting_on_ino.count(ino) == 0) return; 2023-10-30T13:52:20.471 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "take_ino_waiting ino " << ino 2023-10-30T13:52:20.471 INFO:tasks.workunit.client.0.smithi148.stdout:- << " x " << waiting_on_ino[ino].size() 2023-10-30T13:52:20.471 INFO:tasks.workunit.client.0.smithi148.stdout:- << " on " << *this << dendl; 2023-10-30T13:52:20.471 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.splice(ls.end(), waiting_on_ino[ino]); 2023-10-30T13:52:20.472 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_on_ino.erase(ino); 2023-10-30T13:52:20.472 INFO:tasks.workunit.client.0.smithi148.stdout:- if (waiting_on_ino.empty()) 2023-10-30T13:52:20.472 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_INOWAITER); 2023-10-30T13:52:20.472 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.472 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.472 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::take_sub_waiting(list& ls) 2023-10-30T13:52:20.473 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.473 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "take_sub_waiting" << dendl; 2023-10-30T13:52:20.473 INFO:tasks.workunit.client.0.smithi148.stdout:- for (hash_map >::iterator p = waiting_on_dentry.begin(); 2023-10-30T13:52:20.473 INFO:tasks.workunit.client.0.smithi148.stdout:- p != waiting_on_dentry.end(); 2023-10-30T13:52:20.473 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.473 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.splice(ls.end(), p->second); 2023-10-30T13:52:20.474 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_on_dentry.clear(); 2023-10-30T13:52:20.474 INFO:tasks.workunit.client.0.smithi148.stdout:- for (hash_map >::iterator p = waiting_on_ino.begin(); 2023-10-30T13:52:20.474 INFO:tasks.workunit.client.0.smithi148.stdout:- p != waiting_on_ino.end(); 2023-10-30T13:52:20.474 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.474 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.splice(ls.end(), p->second); 2023-10-30T13:52:20.474 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_on_ino.clear(); 2023-10-30T13:52:20.475 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.475 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.475 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.475 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.475 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::add_waiter(int tag, Context *c) 2023-10-30T13:52:20.475 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.476 INFO:tasks.workunit.client.0.smithi148.stdout:- // hierarchical? 2023-10-30T13:52:20.476 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.476 INFO:tasks.workunit.client.0.smithi148.stdout:- // at free root? 2023-10-30T13:52:20.476 INFO:tasks.workunit.client.0.smithi148.stdout:- if (tag & WAIT_ATFREEZEROOT) { 2023-10-30T13:52:20.476 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!(is_freezing_tree_root() || is_frozen_tree_root() || 2023-10-30T13:52:20.476 INFO:tasks.workunit.client.0.smithi148.stdout:- is_freezing_dir() || is_frozen_dir())) { 2023-10-30T13:52:20.477 INFO:tasks.workunit.client.0.smithi148.stdout:- // try parent 2023-10-30T13:52:20.477 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "add_waiter " << tag << " " << c << " should be ATFREEZEROOT, " << *this << " is not root, trying parent" << dendl; 2023-10-30T13:52:20.477 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->parent->dir->add_waiter(tag, c); 2023-10-30T13:52:20.477 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.477 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.478 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.478 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.478 INFO:tasks.workunit.client.0.smithi148.stdout:- // at subtree root? 2023-10-30T13:52:20.478 INFO:tasks.workunit.client.0.smithi148.stdout:- if (tag & WAIT_ATSUBTREEROOT) { 2023-10-30T13:52:20.478 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!is_subtree_root()) { 2023-10-30T13:52:20.478 INFO:tasks.workunit.client.0.smithi148.stdout:- // try parent 2023-10-30T13:52:20.479 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "add_waiter " << tag << " " << c << " should be ATSUBTREEROOT, " << *this << " is not root, trying parent" << dendl; 2023-10-30T13:52:20.479 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->parent->dir->add_waiter(tag, c); 2023-10-30T13:52:20.479 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.479 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.479 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.480 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.480 INFO:tasks.workunit.client.0.smithi148.stdout:- MDSCacheObject::add_waiter(tag, c); 2023-10-30T13:52:20.480 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.480 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.480 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.481 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.481 INFO:tasks.workunit.client.0.smithi148.stdout:-/* NOTE: this checks dentry waiters too */ 2023-10-30T13:52:20.481 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::take_waiting(int mask, list& ls) 2023-10-30T13:52:20.481 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.481 INFO:tasks.workunit.client.0.smithi148.stdout:- if (mask & WAIT_DENTRY) { 2023-10-30T13:52:20.481 INFO:tasks.workunit.client.0.smithi148.stdout:- // take each each dentry waiter 2023-10-30T13:52:20.482 INFO:tasks.workunit.client.0.smithi148.stdout:- hash_map >::iterator it = 2023-10-30T13:52:20.482 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_on_dentry.begin(); 2023-10-30T13:52:20.482 INFO:tasks.workunit.client.0.smithi148.stdout:- while (it != waiting_on_dentry.end()) { 2023-10-30T13:52:20.482 INFO:tasks.workunit.client.0.smithi148.stdout:- take_dentry_waiting((it++)->first, ls); // not post-inc 2023-10-30T13:52:20.482 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.483 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.483 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.483 INFO:tasks.workunit.client.0.smithi148.stdout:- // waiting 2023-10-30T13:52:20.483 INFO:tasks.workunit.client.0.smithi148.stdout:- MDSCacheObject::take_waiting(mask, ls); 2023-10-30T13:52:20.483 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.484 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.484 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.484 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::finish_waiting(int mask, int result) 2023-10-30T13:52:20.484 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.484 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(11) << "finish_waiting mask " << hex << mask << dec << " result " << result << " on " << *this << dendl; 2023-10-30T13:52:20.485 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.485 INFO:tasks.workunit.client.0.smithi148.stdout:- list finished; 2023-10-30T13:52:20.485 INFO:tasks.workunit.client.0.smithi148.stdout:- take_waiting(mask, finished); 2023-10-30T13:52:20.485 INFO:tasks.workunit.client.0.smithi148.stdout:- if (result < 0) 2023-10-30T13:52:20.485 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_contexts(finished, result); 2023-10-30T13:52:20.486 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.486 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->mds->queue_waiters(finished); 2023-10-30T13:52:20.486 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.486 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.486 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.487 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.487 INFO:tasks.workunit.client.0.smithi148.stdout:-// dirty/clean 2023-10-30T13:52:20.487 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.487 INFO:tasks.workunit.client.0.smithi148.stdout:-version_t CDir::pre_dirty(version_t min) 2023-10-30T13:52:20.487 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.487 INFO:tasks.workunit.client.0.smithi148.stdout:- if (min > projected_version) 2023-10-30T13:52:20.488 INFO:tasks.workunit.client.0.smithi148.stdout:- projected_version = min; 2023-10-30T13:52:20.488 INFO:tasks.workunit.client.0.smithi148.stdout:- ++projected_version; 2023-10-30T13:52:20.488 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "pre_dirty " << projected_version << dendl; 2023-10-30T13:52:20.488 INFO:tasks.workunit.client.0.smithi148.stdout:- return projected_version; 2023-10-30T13:52:20.489 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.489 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.489 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::_mark_dirty(LogSegment *ls) 2023-10-30T13:52:20.489 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.489 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!state_test(STATE_DIRTY)) { 2023-10-30T13:52:20.489 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_DIRTY); 2023-10-30T13:52:20.490 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "mark_dirty (was clean) " << *this << " version " << version << dendl; 2023-10-30T13:52:20.490 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_DIRTY); 2023-10-30T13:52:20.490 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(ls); 2023-10-30T13:52:20.490 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.490 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "mark_dirty (already dirty) " << *this << " version " << version << dendl; 2023-10-30T13:52:20.491 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.491 INFO:tasks.workunit.client.0.smithi148.stdout:- if (ls) 2023-10-30T13:52:20.491 INFO:tasks.workunit.client.0.smithi148.stdout:- ls->dirty_dirfrags.push_back(&xlist_dirty); 2023-10-30T13:52:20.491 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.491 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.491 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::mark_dirty(version_t pv, LogSegment *ls) 2023-10-30T13:52:20.492 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.492 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(version < pv); 2023-10-30T13:52:20.492 INFO:tasks.workunit.client.0.smithi148.stdout:- version = pv; 2023-10-30T13:52:20.492 INFO:tasks.workunit.client.0.smithi148.stdout:- _mark_dirty(ls); 2023-10-30T13:52:20.492 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.493 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.493 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::mark_clean() 2023-10-30T13:52:20.493 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.493 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "mark_clean " << *this << " version " << version << dendl; 2023-10-30T13:52:20.493 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_DIRTY)) { 2023-10-30T13:52:20.493 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_DIRTY); 2023-10-30T13:52:20.493 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_DIRTY); 2023-10-30T13:52:20.494 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.494 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty.remove_myself(); 2023-10-30T13:52:20.494 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.494 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.494 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.495 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.495 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.495 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.495 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::first_get() 2023-10-30T13:52:20.495 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.495 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->get(CInode::PIN_DIRFRAG); 2023-10-30T13:52:20.496 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.496 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.496 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::last_put() 2023-10-30T13:52:20.496 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.496 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->put(CInode::PIN_DIRFRAG); 2023-10-30T13:52:20.496 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.497 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.497 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.497 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.497 INFO:tasks.workunit.client.0.smithi148.stdout:-/****************************************************************************** 2023-10-30T13:52:20.497 INFO:tasks.workunit.client.0.smithi148.stdout:- * FETCH and COMMIT 2023-10-30T13:52:20.497 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.498 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.498 INFO:tasks.workunit.client.0.smithi148.stdout:-// ----------------------- 2023-10-30T13:52:20.498 INFO:tasks.workunit.client.0.smithi148.stdout:-// FETCH 2023-10-30T13:52:20.498 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.498 INFO:tasks.workunit.client.0.smithi148.stdout:-class C_Dir_Fetch : public Context { 2023-10-30T13:52:20.499 INFO:tasks.workunit.client.0.smithi148.stdout:- protected: 2023-10-30T13:52:20.499 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir; 2023-10-30T13:52:20.499 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.499 INFO:tasks.workunit.client.0.smithi148.stdout:- bufferlist bl; 2023-10-30T13:52:20.499 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.499 INFO:tasks.workunit.client.0.smithi148.stdout:- C_Dir_Fetch(CDir *d) : dir(d) { } 2023-10-30T13:52:20.500 INFO:tasks.workunit.client.0.smithi148.stdout:- void finish(int result) { 2023-10-30T13:52:20.500 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->_fetched(bl); 2023-10-30T13:52:20.500 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.500 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:20.500 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.500 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::fetch(Context *c, bool ignore_authpinnability) 2023-10-30T13:52:20.501 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.501 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "fetch on " << *this << dendl; 2023-10-30T13:52:20.501 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.501 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.501 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!is_complete()); 2023-10-30T13:52:20.501 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.502 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!can_auth_pin() && !ignore_authpinnability) { 2023-10-30T13:52:20.502 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(7) << "fetch waiting for authpinnable" << dendl; 2023-10-30T13:52:20.502 INFO:tasks.workunit.client.0.smithi148.stdout:- add_waiter(WAIT_UNFREEZE, c); 2023-10-30T13:52:20.502 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.502 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.503 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.503 INFO:tasks.workunit.client.0.smithi148.stdout:- if (c) add_waiter(WAIT_COMPLETE, c); 2023-10-30T13:52:20.503 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.503 INFO:tasks.workunit.client.0.smithi148.stdout:- // already fetching? 2023-10-30T13:52:20.503 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(CDir::STATE_FETCHING)) { 2023-10-30T13:52:20.503 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(7) << "already fetching; waiting" << dendl; 2023-10-30T13:52:20.504 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.504 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.504 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.504 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pin(); 2023-10-30T13:52:20.504 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(CDir::STATE_FETCHING); 2023-10-30T13:52:20.504 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.505 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cache->mds->logger) cache->mds->logger->inc("dir_f"); 2023-10-30T13:52:20.505 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.505 INFO:tasks.workunit.client.0.smithi148.stdout:- // start by reading the first hunk of it 2023-10-30T13:52:20.505 INFO:tasks.workunit.client.0.smithi148.stdout:- C_Dir_Fetch *fin = new C_Dir_Fetch(this); 2023-10-30T13:52:20.505 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->mds->objecter->read( get_ondisk_object(), 2023-10-30T13:52:20.506 INFO:tasks.workunit.client.0.smithi148.stdout:- 0, 0, // whole object 2023-10-30T13:52:20.506 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->mds->objecter->osdmap->file_to_object_layout( get_ondisk_object(), 2023-10-30T13:52:20.506 INFO:tasks.workunit.client.0.smithi148.stdout:- g_OSD_MDDirLayout ), 2023-10-30T13:52:20.506 INFO:tasks.workunit.client.0.smithi148.stdout:- &fin->bl, 2023-10-30T13:52:20.507 INFO:tasks.workunit.client.0.smithi148.stdout:- fin ); 2023-10-30T13:52:20.507 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.507 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.507 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::_fetched(bufferlist &bl) 2023-10-30T13:52:20.507 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.507 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "_fetched " << bl.length() 2023-10-30T13:52:20.508 INFO:tasks.workunit.client.0.smithi148.stdout:- << " bytes for " << *this 2023-10-30T13:52:20.508 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.508 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.508 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.508 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!is_frozen()); 2023-10-30T13:52:20.508 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.509 INFO:tasks.workunit.client.0.smithi148.stdout:- // decode. 2023-10-30T13:52:20.509 INFO:tasks.workunit.client.0.smithi148.stdout:- int len = bl.length(); 2023-10-30T13:52:20.509 INFO:tasks.workunit.client.0.smithi148.stdout:- int off = 0; 2023-10-30T13:52:20.509 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t got_version; 2023-10-30T13:52:20.509 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.510 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(got_version, bl, off); 2023-10-30T13:52:20.510 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.510 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "_fetched version " << got_version 2023-10-30T13:52:20.510 INFO:tasks.workunit.client.0.smithi148.stdout:- << ", " << len << " bytes" 2023-10-30T13:52:20.510 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.511 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.511 INFO:tasks.workunit.client.0.smithi148.stdout:- int32_t n; 2023-10-30T13:52:20.511 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(n, bl, off); 2023-10-30T13:52:20.511 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.511 INFO:tasks.workunit.client.0.smithi148.stdout:- //int num_new_inodes_loaded = 0; 2023-10-30T13:52:20.511 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.512 INFO:tasks.workunit.client.0.smithi148.stdout:- for (int i=0; iget_inode() == 0) { 2023-10-30T13:52:20.515 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "_fetched had NEG dentry " << *dn << dendl; 2023-10-30T13:52:20.516 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.516 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "_fetched had dentry " << *dn << dendl; 2023-10-30T13:52:20.516 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.516 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.516 INFO:tasks.workunit.client.0.smithi148.stdout:- // (remote) link 2023-10-30T13:52:20.516 INFO:tasks.workunit.client.0.smithi148.stdout:- dn = add_remote_dentry(dname, ino, d_type); 2023-10-30T13:52:20.517 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.517 INFO:tasks.workunit.client.0.smithi148.stdout:- // link to inode? 2023-10-30T13:52:20.517 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *in = cache->get_inode(ino); // we may or may not have it. 2023-10-30T13:52:20.517 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in) { 2023-10-30T13:52:20.517 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->link_remote(in); 2023-10-30T13:52:20.517 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "_fetched got remote link " << ino << " which we have " << *in << dendl; 2023-10-30T13:52:20.518 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.518 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "_fetched got remote link " << ino << " (dont' have it)" << dendl; 2023-10-30T13:52:20.518 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.518 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.519 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.519 INFO:tasks.workunit.client.0.smithi148.stdout:- else if (type == 'I') { 2023-10-30T13:52:20.519 INFO:tasks.workunit.client.0.smithi148.stdout:- // inode 2023-10-30T13:52:20.519 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.519 INFO:tasks.workunit.client.0.smithi148.stdout:- // parse out inode 2023-10-30T13:52:20.519 INFO:tasks.workunit.client.0.smithi148.stdout:- inode_t inode; 2023-10-30T13:52:20.520 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(inode, bl, off); 2023-10-30T13:52:20.520 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.520 INFO:tasks.workunit.client.0.smithi148.stdout:- string symlink; 2023-10-30T13:52:20.520 INFO:tasks.workunit.client.0.smithi148.stdout:- if (inode.is_symlink()) 2023-10-30T13:52:20.520 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(symlink, bl, off); 2023-10-30T13:52:20.520 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.521 INFO:tasks.workunit.client.0.smithi148.stdout:- fragtree_t fragtree; 2023-10-30T13:52:20.521 INFO:tasks.workunit.client.0.smithi148.stdout:- fragtree._decode(bl, off); 2023-10-30T13:52:20.521 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.521 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn) { 2023-10-30T13:52:20.521 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->get_inode() == 0) { 2023-10-30T13:52:20.521 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "_fetched had NEG dentry " << *dn << dendl; 2023-10-30T13:52:20.522 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.522 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "_fetched had dentry " << *dn << dendl; 2023-10-30T13:52:20.522 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.522 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.522 INFO:tasks.workunit.client.0.smithi148.stdout:- // add inode 2023-10-30T13:52:20.523 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *in = 0; 2023-10-30T13:52:20.523 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cache->have_inode(inode.ino)) { 2023-10-30T13:52:20.523 INFO:tasks.workunit.client.0.smithi148.stdout:- in = cache->get_inode(inode.ino); 2023-10-30T13:52:20.523 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(-12) << "_fetched got (but i already had) " << *in 2023-10-30T13:52:20.523 INFO:tasks.workunit.client.0.smithi148.stdout:- << " mode " << in->inode.mode 2023-10-30T13:52:20.524 INFO:tasks.workunit.client.0.smithi148.stdout:- << " mtime " << in->inode.mtime << dendl; 2023-10-30T13:52:20.524 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(0); // this shouldn't happen!! 2023-10-30T13:52:20.524 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.524 INFO:tasks.workunit.client.0.smithi148.stdout:- // inode 2023-10-30T13:52:20.524 INFO:tasks.workunit.client.0.smithi148.stdout:- in = new CInode(cache); 2023-10-30T13:52:20.525 INFO:tasks.workunit.client.0.smithi148.stdout:- in->inode = inode; 2023-10-30T13:52:20.525 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.525 INFO:tasks.workunit.client.0.smithi148.stdout:- // symlink? 2023-10-30T13:52:20.525 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in->is_symlink()) 2023-10-30T13:52:20.525 INFO:tasks.workunit.client.0.smithi148.stdout:- in->symlink = symlink; 2023-10-30T13:52:20.526 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.526 INFO:tasks.workunit.client.0.smithi148.stdout:- // dirfragtree 2023-10-30T13:52:20.526 INFO:tasks.workunit.client.0.smithi148.stdout:- in->dirfragtree.swap(fragtree); 2023-10-30T13:52:20.526 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.526 INFO:tasks.workunit.client.0.smithi148.stdout:- // add 2023-10-30T13:52:20.527 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->add_inode( in ); 2023-10-30T13:52:20.527 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.527 INFO:tasks.workunit.client.0.smithi148.stdout:- // link 2023-10-30T13:52:20.527 INFO:tasks.workunit.client.0.smithi148.stdout:- dn = add_primary_dentry(dname, in); 2023-10-30T13:52:20.527 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(12) << "_fetched got " << *dn << " " << *in << dendl; 2023-10-30T13:52:20.528 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.528 INFO:tasks.workunit.client.0.smithi148.stdout:- //in->hack_accessed = false; 2023-10-30T13:52:20.528 INFO:tasks.workunit.client.0.smithi148.stdout:- //in->hack_load_stamp = g_clock.now(); 2023-10-30T13:52:20.528 INFO:tasks.workunit.client.0.smithi148.stdout:- //num_new_inodes_loaded++; 2023-10-30T13:52:20.528 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.528 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.529 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.529 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(1) << "corrupt directory, i got tag char '" << type << "' val " << (int)(type) 2023-10-30T13:52:20.529 INFO:tasks.workunit.client.0.smithi148.stdout:- << " at pos " << off << dendl; 2023-10-30T13:52:20.529 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(0); 2023-10-30T13:52:20.529 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.530 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.530 INFO:tasks.workunit.client.0.smithi148.stdout:- // make note of dentry position in the directory 2023-10-30T13:52:20.530 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->dir_offset = dn_offset; 2023-10-30T13:52:20.530 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.531 INFO:tasks.workunit.client.0.smithi148.stdout:- /** clean underwater item? 2023-10-30T13:52:20.531 INFO:tasks.workunit.client.0.smithi148.stdout:- * Underwater item is something that is dirty in our cache from 2023-10-30T13:52:20.531 INFO:tasks.workunit.client.0.smithi148.stdout:- * journal replay, but was previously flushed to disk before the 2023-10-30T13:52:20.531 INFO:tasks.workunit.client.0.smithi148.stdout:- * mds failed. 2023-10-30T13:52:20.531 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.532 INFO:tasks.workunit.client.0.smithi148.stdout:- * We only do this is committed_version == 0. that implies either 2023-10-30T13:52:20.532 INFO:tasks.workunit.client.0.smithi148.stdout:- * - this is a fetch after from a clean/empty CDir is created 2023-10-30T13:52:20.532 INFO:tasks.workunit.client.0.smithi148.stdout:- * (and has no effect, since the dn won't exist); or 2023-10-30T13:52:20.532 INFO:tasks.workunit.client.0.smithi148.stdout:- * - this is a fetch after _recovery_, which is what we're worried 2023-10-30T13:52:20.532 INFO:tasks.workunit.client.0.smithi148.stdout:- * about. Items that are marked dirty from the journal should be 2023-10-30T13:52:20.533 INFO:tasks.workunit.client.0.smithi148.stdout:- * marked clean if they appear on disk. 2023-10-30T13:52:20.533 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.533 INFO:tasks.workunit.client.0.smithi148.stdout:- if (committed_version == 0 && 2023-10-30T13:52:20.533 INFO:tasks.workunit.client.0.smithi148.stdout:- dn && 2023-10-30T13:52:20.533 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->get_version() <= got_version && 2023-10-30T13:52:20.534 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->is_dirty()) { 2023-10-30T13:52:20.534 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "_fetched had underwater dentry " << *dn << ", marking clean" << dendl; 2023-10-30T13:52:20.534 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->mark_clean(); 2023-10-30T13:52:20.534 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.534 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->get_inode()) { 2023-10-30T13:52:20.535 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn->get_inode()->get_version() <= got_version); 2023-10-30T13:52:20.535 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "_fetched had underwater inode " << *dn->get_inode() << ", marking clean" << dendl; 2023-10-30T13:52:20.535 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->get_inode()->mark_clean(); 2023-10-30T13:52:20.535 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.535 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.536 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.536 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(off == len); no, directories may shrink. add this back in when we properly truncate objects on write. 2023-10-30T13:52:20.536 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.536 INFO:tasks.workunit.client.0.smithi148.stdout:- // take the loaded version? 2023-10-30T13:52:20.536 INFO:tasks.workunit.client.0.smithi148.stdout:- // only if we are a fresh CDir* with no prior state. 2023-10-30T13:52:20.537 INFO:tasks.workunit.client.0.smithi148.stdout:- if (version == 0) { 2023-10-30T13:52:20.537 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(projected_version == 0); 2023-10-30T13:52:20.537 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!state_test(STATE_COMMITTING)); 2023-10-30T13:52:20.537 INFO:tasks.workunit.client.0.smithi148.stdout:- projected_version = version = committing_version = committed_version = got_version; 2023-10-30T13:52:20.537 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.538 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.538 INFO:tasks.workunit.client.0.smithi148.stdout:- //cache->mds->logger->inc("newin", num_new_inodes_loaded); 2023-10-30T13:52:20.538 INFO:tasks.workunit.client.0.smithi148.stdout:- //hack_num_accessed = 0; 2023-10-30T13:52:20.538 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.538 INFO:tasks.workunit.client.0.smithi148.stdout:- // mark complete, !fetching 2023-10-30T13:52:20.539 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_COMPLETE); 2023-10-30T13:52:20.539 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FETCHING); 2023-10-30T13:52:20.539 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_unpin(); 2023-10-30T13:52:20.539 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.539 INFO:tasks.workunit.client.0.smithi148.stdout:- // kick waiters 2023-10-30T13:52:20.540 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_COMPLETE, 0); 2023-10-30T13:52:20.540 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.540 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.540 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.540 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.541 INFO:tasks.workunit.client.0.smithi148.stdout:-// ----------------------- 2023-10-30T13:52:20.541 INFO:tasks.workunit.client.0.smithi148.stdout:-// COMMIT 2023-10-30T13:52:20.541 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.541 INFO:tasks.workunit.client.0.smithi148.stdout:-/** 2023-10-30T13:52:20.541 INFO:tasks.workunit.client.0.smithi148.stdout:- * commit 2023-10-30T13:52:20.542 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.542 INFO:tasks.workunit.client.0.smithi148.stdout:- * @param want - min version i want committed 2023-10-30T13:52:20.542 INFO:tasks.workunit.client.0.smithi148.stdout:- * @param c - callback for completion 2023-10-30T13:52:20.542 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.542 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::commit(version_t want, Context *c) 2023-10-30T13:52:20.543 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.543 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "commit want " << want << " on " << *this << dendl; 2023-10-30T13:52:20.543 INFO:tasks.workunit.client.0.smithi148.stdout:- if (want == 0) want = version; 2023-10-30T13:52:20.543 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.543 INFO:tasks.workunit.client.0.smithi148.stdout:- // preconditions 2023-10-30T13:52:20.544 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(want <= version || version == 0); // can't commit the future 2023-10-30T13:52:20.544 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(want > committed_version); // the caller is stupid 2023-10-30T13:52:20.544 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.544 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(can_auth_pin()); 2023-10-30T13:52:20.544 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.545 INFO:tasks.workunit.client.0.smithi148.stdout:- // note: queue up a noop if necessary, so that we always 2023-10-30T13:52:20.545 INFO:tasks.workunit.client.0.smithi148.stdout:- // get an auth_pin. 2023-10-30T13:52:20.545 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!c) 2023-10-30T13:52:20.545 INFO:tasks.workunit.client.0.smithi148.stdout:- c = new C_NoopContext; 2023-10-30T13:52:20.545 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.546 INFO:tasks.workunit.client.0.smithi148.stdout:- // auth_pin on first waiter 2023-10-30T13:52:20.546 INFO:tasks.workunit.client.0.smithi148.stdout:- if (waiting_for_commit.empty()) 2023-10-30T13:52:20.546 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pin(); 2023-10-30T13:52:20.546 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_for_commit[want].push_back(c); 2023-10-30T13:52:20.546 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.547 INFO:tasks.workunit.client.0.smithi148.stdout:- // ok. 2023-10-30T13:52:20.547 INFO:tasks.workunit.client.0.smithi148.stdout:- _commit(want); 2023-10-30T13:52:20.547 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.547 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.547 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.547 INFO:tasks.workunit.client.0.smithi148.stdout:-class C_Dir_RetryCommit : public Context { 2023-10-30T13:52:20.548 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir; 2023-10-30T13:52:20.548 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t want; 2023-10-30T13:52:20.548 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.548 INFO:tasks.workunit.client.0.smithi148.stdout:- C_Dir_RetryCommit(CDir *d, version_t v) : 2023-10-30T13:52:20.548 INFO:tasks.workunit.client.0.smithi148.stdout:- dir(d), want(v) { } 2023-10-30T13:52:20.548 INFO:tasks.workunit.client.0.smithi148.stdout:- void finish(int r) { 2023-10-30T13:52:20.549 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->_commit(want); 2023-10-30T13:52:20.549 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.549 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:20.549 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.549 INFO:tasks.workunit.client.0.smithi148.stdout:-class C_Dir_Committed : public Context { 2023-10-30T13:52:20.550 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir; 2023-10-30T13:52:20.550 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t version; 2023-10-30T13:52:20.550 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.550 INFO:tasks.workunit.client.0.smithi148.stdout:- C_Dir_Committed(CDir *d, version_t v) : dir(d), version(v) { } 2023-10-30T13:52:20.550 INFO:tasks.workunit.client.0.smithi148.stdout:- void finish(int r) { 2023-10-30T13:52:20.551 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->_committed(version); 2023-10-30T13:52:20.551 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.551 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:20.551 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.551 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::_commit(version_t want) 2023-10-30T13:52:20.551 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.552 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "_commit want " << want << " on " << *this << dendl; 2023-10-30T13:52:20.552 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.552 INFO:tasks.workunit.client.0.smithi148.stdout:- // we can't commit things in the future. 2023-10-30T13:52:20.552 INFO:tasks.workunit.client.0.smithi148.stdout:- // (even the projected future.) 2023-10-30T13:52:20.552 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(want <= version || version == 0); 2023-10-30T13:52:20.552 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.553 INFO:tasks.workunit.client.0.smithi148.stdout:- // check pre+postconditions. 2023-10-30T13:52:20.553 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.553 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.553 INFO:tasks.workunit.client.0.smithi148.stdout:- // already committed? 2023-10-30T13:52:20.553 INFO:tasks.workunit.client.0.smithi148.stdout:- if (committed_version >= want) { 2023-10-30T13:52:20.553 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "already committed " << committed_version << " >= " << want << dendl; 2023-10-30T13:52:20.554 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.554 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.554 INFO:tasks.workunit.client.0.smithi148.stdout:- // already committing >= want? 2023-10-30T13:52:20.554 INFO:tasks.workunit.client.0.smithi148.stdout:- if (committing_version >= want) { 2023-10-30T13:52:20.554 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "already committing " << committing_version << " >= " << want << dendl; 2023-10-30T13:52:20.554 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(state_test(STATE_COMMITTING)); 2023-10-30T13:52:20.555 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.555 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.555 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.555 INFO:tasks.workunit.client.0.smithi148.stdout:- // complete? 2023-10-30T13:52:20.555 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!is_complete()) { 2023-10-30T13:52:20.555 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(7) << "commit not complete, fetching first" << dendl; 2023-10-30T13:52:20.556 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cache->mds->logger) cache->mds->logger->inc("dir_ffc"); 2023-10-30T13:52:20.556 INFO:tasks.workunit.client.0.smithi148.stdout:- fetch(new C_Dir_RetryCommit(this, want)); 2023-10-30T13:52:20.556 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.556 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.556 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.556 INFO:tasks.workunit.client.0.smithi148.stdout:- // commit. 2023-10-30T13:52:20.557 INFO:tasks.workunit.client.0.smithi148.stdout:- committing_version = version; 2023-10-30T13:52:20.557 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.557 INFO:tasks.workunit.client.0.smithi148.stdout:- // mark committing (if not already) 2023-10-30T13:52:20.557 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!state_test(STATE_COMMITTING)) { 2023-10-30T13:52:20.557 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "marking committing" << dendl; 2023-10-30T13:52:20.557 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_COMMITTING); 2023-10-30T13:52:20.558 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.558 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.558 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cache->mds->logger) cache->mds->logger->inc("dir_c"); 2023-10-30T13:52:20.558 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.558 INFO:tasks.workunit.client.0.smithi148.stdout:- // encode 2023-10-30T13:52:20.558 INFO:tasks.workunit.client.0.smithi148.stdout:- bufferlist bl; 2023-10-30T13:52:20.559 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.559 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(version, bl); 2023-10-30T13:52:20.559 INFO:tasks.workunit.client.0.smithi148.stdout:- int32_t n = nitems; 2023-10-30T13:52:20.559 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(n, bl); 2023-10-30T13:52:20.559 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.560 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map_t::iterator it = items.begin(); 2023-10-30T13:52:20.560 INFO:tasks.workunit.client.0.smithi148.stdout:- it != items.end(); 2023-10-30T13:52:20.560 INFO:tasks.workunit.client.0.smithi148.stdout:- it++) { 2023-10-30T13:52:20.560 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry *dn = it->second; 2023-10-30T13:52:20.560 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.561 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->is_null()) 2023-10-30T13:52:20.561 INFO:tasks.workunit.client.0.smithi148.stdout:- continue; // skip negative entries 2023-10-30T13:52:20.561 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.561 INFO:tasks.workunit.client.0.smithi148.stdout:- n--; 2023-10-30T13:52:20.561 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.561 INFO:tasks.workunit.client.0.smithi148.stdout:- // primary or remote? 2023-10-30T13:52:20.562 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->is_remote()) { 2023-10-30T13:52:20.562 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t ino = dn->get_remote_ino(); 2023-10-30T13:52:20.562 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned char d_type = dn->get_remote_d_type(); 2023-10-30T13:52:20.562 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(14) << " pos " << bl.length() << " dn '" << it->first << "' remote ino " << ino << dendl; 2023-10-30T13:52:20.563 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.563 INFO:tasks.workunit.client.0.smithi148.stdout:- // marker, name, ino 2023-10-30T13:52:20.563 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append( "L", 1 ); // remote link 2023-10-30T13:52:20.563 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(it->first, bl); 2023-10-30T13:52:20.563 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(ino, bl); 2023-10-30T13:52:20.564 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(d_type, bl); 2023-10-30T13:52:20.564 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.564 INFO:tasks.workunit.client.0.smithi148.stdout:- // primary link 2023-10-30T13:52:20.564 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *in = dn->get_inode(); 2023-10-30T13:52:20.564 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(in); 2023-10-30T13:52:20.565 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.565 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(14) << " pos " << bl.length() << " dn '" << it->first << "' inode " << *in << dendl; 2023-10-30T13:52:20.565 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.565 INFO:tasks.workunit.client.0.smithi148.stdout:- // marker, name, inode, [symlink string] 2023-10-30T13:52:20.566 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append( "I", 1 ); // inode 2023-10-30T13:52:20.566 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(it->first, bl); 2023-10-30T13:52:20.566 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(in->inode, bl); 2023-10-30T13:52:20.566 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.566 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in->is_symlink()) { 2023-10-30T13:52:20.567 INFO:tasks.workunit.client.0.smithi148.stdout:- // include symlink destination! 2023-10-30T13:52:20.567 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(18) << " inlcuding symlink ptr " << in->symlink << dendl; 2023-10-30T13:52:20.567 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(in->symlink, bl); 2023-10-30T13:52:20.567 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.567 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.568 INFO:tasks.workunit.client.0.smithi148.stdout:- in->dirfragtree._encode(bl); 2023-10-30T13:52:20.568 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.568 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.568 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(n == 0); 2023-10-30T13:52:20.568 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.569 INFO:tasks.workunit.client.0.smithi148.stdout:- // write it. 2023-10-30T13:52:20.569 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->mds->objecter->write( get_ondisk_object(), 2023-10-30T13:52:20.569 INFO:tasks.workunit.client.0.smithi148.stdout:- 0, bl.length(), 2023-10-30T13:52:20.569 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->mds->objecter->osdmap->file_to_object_layout( get_ondisk_object(), 2023-10-30T13:52:20.569 INFO:tasks.workunit.client.0.smithi148.stdout:- g_OSD_MDDirLayout ), 2023-10-30T13:52:20.569 INFO:tasks.workunit.client.0.smithi148.stdout:- bl, 2023-10-30T13:52:20.570 INFO:tasks.workunit.client.0.smithi148.stdout:- NULL, new C_Dir_Committed(this, version) ); 2023-10-30T13:52:20.570 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.570 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.570 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.570 INFO:tasks.workunit.client.0.smithi148.stdout:-/** 2023-10-30T13:52:20.571 INFO:tasks.workunit.client.0.smithi148.stdout:- * _committed 2023-10-30T13:52:20.571 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.571 INFO:tasks.workunit.client.0.smithi148.stdout:- * @param v version i just committed 2023-10-30T13:52:20.571 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.571 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::_committed(version_t v) 2023-10-30T13:52:20.571 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.572 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "_committed v " << v << " on " << *this << dendl; 2023-10-30T13:52:20.572 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.572 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.572 INFO:tasks.workunit.client.0.smithi148.stdout:- // take note. 2023-10-30T13:52:20.572 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(v > committed_version); 2023-10-30T13:52:20.572 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(v <= committing_version); 2023-10-30T13:52:20.573 INFO:tasks.workunit.client.0.smithi148.stdout:- committed_version = v; 2023-10-30T13:52:20.573 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.573 INFO:tasks.workunit.client.0.smithi148.stdout:- // _all_ commits done? 2023-10-30T13:52:20.573 INFO:tasks.workunit.client.0.smithi148.stdout:- if (committing_version == committed_version) 2023-10-30T13:52:20.573 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(CDir::STATE_COMMITTING); 2023-10-30T13:52:20.573 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.574 INFO:tasks.workunit.client.0.smithi148.stdout:- // dir clean? 2023-10-30T13:52:20.574 INFO:tasks.workunit.client.0.smithi148.stdout:- if (committed_version == version) 2023-10-30T13:52:20.574 INFO:tasks.workunit.client.0.smithi148.stdout:- mark_clean(); 2023-10-30T13:52:20.574 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.574 INFO:tasks.workunit.client.0.smithi148.stdout:- // dentries clean? 2023-10-30T13:52:20.575 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map_t::iterator it = items.begin(); 2023-10-30T13:52:20.575 INFO:tasks.workunit.client.0.smithi148.stdout:- it != items.end(); ) { 2023-10-30T13:52:20.575 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry *dn = it->second; 2023-10-30T13:52:20.575 INFO:tasks.workunit.client.0.smithi148.stdout:- it++; 2023-10-30T13:52:20.575 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.575 INFO:tasks.workunit.client.0.smithi148.stdout:- // dentry 2023-10-30T13:52:20.576 INFO:tasks.workunit.client.0.smithi148.stdout:- if (committed_version >= dn->get_version()) { 2023-10-30T13:52:20.576 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->is_dirty()) { 2023-10-30T13:52:20.576 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << " dir " << committed_version << " >= dn " << dn->get_version() << " now clean " << *dn << dendl; 2023-10-30T13:52:20.576 INFO:tasks.workunit.client.0.smithi148.stdout:- dn->mark_clean(); 2023-10-30T13:52:20.576 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.576 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.577 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << " dir " << committed_version << " < dn " << dn->get_version() << " still dirty " << *dn << dendl; 2023-10-30T13:52:20.577 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.577 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.577 INFO:tasks.workunit.client.0.smithi148.stdout:- // inode? 2023-10-30T13:52:20.577 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dn->is_primary()) { 2023-10-30T13:52:20.578 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *in = dn->get_inode(); 2023-10-30T13:52:20.578 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(in); 2023-10-30T13:52:20.578 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(in->is_auth()); 2023-10-30T13:52:20.578 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.578 INFO:tasks.workunit.client.0.smithi148.stdout:- if (committed_version >= in->get_version()) { 2023-10-30T13:52:20.579 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in->is_dirty()) { 2023-10-30T13:52:20.579 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << " dir " << committed_version << " >= inode " << in->get_version() << " now clean " << *in << dendl; 2023-10-30T13:52:20.579 INFO:tasks.workunit.client.0.smithi148.stdout:- in->mark_clean(); 2023-10-30T13:52:20.579 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.579 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.579 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << " dir " << committed_version << " < inode " << in->get_version() << " still dirty " << *in << dendl; 2023-10-30T13:52:20.580 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(in->is_dirty()); 2023-10-30T13:52:20.580 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.580 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.580 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.580 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.580 INFO:tasks.workunit.client.0.smithi148.stdout:- // finishers? 2023-10-30T13:52:20.581 INFO:tasks.workunit.client.0.smithi148.stdout:- bool were_waiters = !waiting_for_commit.empty(); 2023-10-30T13:52:20.581 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.581 INFO:tasks.workunit.client.0.smithi148.stdout:- map >::iterator p = waiting_for_commit.begin(); 2023-10-30T13:52:20.581 INFO:tasks.workunit.client.0.smithi148.stdout:- while (p != waiting_for_commit.end()) { 2023-10-30T13:52:20.581 INFO:tasks.workunit.client.0.smithi148.stdout:- map >::iterator n = p; 2023-10-30T13:52:20.581 INFO:tasks.workunit.client.0.smithi148.stdout:- n++; 2023-10-30T13:52:20.582 INFO:tasks.workunit.client.0.smithi148.stdout:- if (p->first > committed_version) break; // haven't committed this far yet. 2023-10-30T13:52:20.582 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->mds->queue_waiters(p->second); 2023-10-30T13:52:20.582 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_for_commit.erase(p); 2023-10-30T13:52:20.582 INFO:tasks.workunit.client.0.smithi148.stdout:- p = n; 2023-10-30T13:52:20.582 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.582 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.583 INFO:tasks.workunit.client.0.smithi148.stdout:- // unpin if we kicked the last waiter. 2023-10-30T13:52:20.583 INFO:tasks.workunit.client.0.smithi148.stdout:- if (were_waiters && 2023-10-30T13:52:20.583 INFO:tasks.workunit.client.0.smithi148.stdout:- waiting_for_commit.empty()) 2023-10-30T13:52:20.583 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_unpin(); 2023-10-30T13:52:20.583 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.584 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.584 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.584 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.584 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.584 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.584 INFO:tasks.workunit.client.0.smithi148.stdout:-// IMPORT/EXPORT 2023-10-30T13:52:20.584 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.585 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::encode_export(bufferlist& bl) 2023-10-30T13:52:20.585 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.585 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(version, bl); 2023-10-30T13:52:20.585 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(committed_version, bl); 2023-10-30T13:52:20.585 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(committed_version_equivalent, bl); 2023-10-30T13:52:20.585 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.586 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(state, bl); 2023-10-30T13:52:20.586 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(dir_rep, bl); 2023-10-30T13:52:20.586 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.586 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(pop_me, bl); 2023-10-30T13:52:20.586 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(pop_auth_subtree, bl); 2023-10-30T13:52:20.587 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.587 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(dir_rep_by, bl); 2023-10-30T13:52:20.587 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(replica_map, bl); 2023-10-30T13:52:20.587 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.587 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_TEMPEXPORTING); 2023-10-30T13:52:20.588 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.588 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.588 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::finish_export(utime_t now) 2023-10-30T13:52:20.588 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.588 INFO:tasks.workunit.client.0.smithi148.stdout:- pop_auth_subtree_nested -= pop_auth_subtree; 2023-10-30T13:52:20.589 INFO:tasks.workunit.client.0.smithi148.stdout:- pop_me.zero(now); 2023-10-30T13:52:20.589 INFO:tasks.workunit.client.0.smithi148.stdout:- pop_auth_subtree.zero(now); 2023-10-30T13:52:20.589 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_TEMPEXPORTING); 2023-10-30T13:52:20.589 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.589 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.589 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::decode_import(bufferlist::iterator& blp) 2023-10-30T13:52:20.589 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.590 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(version, blp); 2023-10-30T13:52:20.590 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(committed_version, blp); 2023-10-30T13:52:20.590 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(committed_version_equivalent, blp); 2023-10-30T13:52:20.590 INFO:tasks.workunit.client.0.smithi148.stdout:- committing_version = committed_version; 2023-10-30T13:52:20.590 INFO:tasks.workunit.client.0.smithi148.stdout:- projected_version = version; 2023-10-30T13:52:20.591 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.591 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned s; 2023-10-30T13:52:20.591 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(s, blp); 2023-10-30T13:52:20.591 INFO:tasks.workunit.client.0.smithi148.stdout:- state &= MASK_STATE_IMPORT_KEPT; 2023-10-30T13:52:20.591 INFO:tasks.workunit.client.0.smithi148.stdout:- state |= (s & MASK_STATE_EXPORTED); 2023-10-30T13:52:20.592 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_dirty()) get(PIN_DIRTY); 2023-10-30T13:52:20.592 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.592 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(dir_rep, blp); 2023-10-30T13:52:20.592 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.592 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(pop_me, blp); 2023-10-30T13:52:20.592 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(pop_auth_subtree, blp); 2023-10-30T13:52:20.593 INFO:tasks.workunit.client.0.smithi148.stdout:- pop_auth_subtree_nested += pop_auth_subtree; 2023-10-30T13:52:20.593 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.593 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(dir_rep_by, blp); 2023-10-30T13:52:20.593 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(replica_map, blp); 2023-10-30T13:52:20.593 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!replica_map.empty()) get(PIN_REPLICATED); 2023-10-30T13:52:20.594 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.594 INFO:tasks.workunit.client.0.smithi148.stdout:- replica_nonce = 0; // no longer defined 2023-10-30T13:52:20.594 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.594 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.594 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.594 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.595 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.595 INFO:tasks.workunit.client.0.smithi148.stdout:-/******************************** 2023-10-30T13:52:20.595 INFO:tasks.workunit.client.0.smithi148.stdout:- * AUTHORITY 2023-10-30T13:52:20.595 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.595 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.596 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.596 INFO:tasks.workunit.client.0.smithi148.stdout:- * if dir_auth.first == parent, auth is same as inode. 2023-10-30T13:52:20.596 INFO:tasks.workunit.client.0.smithi148.stdout:- * unless .second != unknown, in which case that sticks. 2023-10-30T13:52:20.596 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.596 INFO:tasks.workunit.client.0.smithi148.stdout:-pair CDir::authority() 2023-10-30T13:52:20.597 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.597 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_subtree_root()) 2023-10-30T13:52:20.597 INFO:tasks.workunit.client.0.smithi148.stdout:- return dir_auth; 2023-10-30T13:52:20.597 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.597 INFO:tasks.workunit.client.0.smithi148.stdout:- return inode->authority(); 2023-10-30T13:52:20.598 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.598 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.598 INFO:tasks.workunit.client.0.smithi148.stdout:-/** is_subtree_root() 2023-10-30T13:52:20.598 INFO:tasks.workunit.client.0.smithi148.stdout:- * true if this is an auth delegation point. 2023-10-30T13:52:20.598 INFO:tasks.workunit.client.0.smithi148.stdout:- * that is, dir_auth != default (parent,unknown) 2023-10-30T13:52:20.599 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.599 INFO:tasks.workunit.client.0.smithi148.stdout:- * some key observations: 2023-10-30T13:52:20.599 INFO:tasks.workunit.client.0.smithi148.stdout:- * if i am auth: 2023-10-30T13:52:20.599 INFO:tasks.workunit.client.0.smithi148.stdout:- * - any region bound will be an export, or frozen. 2023-10-30T13:52:20.599 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.599 INFO:tasks.workunit.client.0.smithi148.stdout:- * note that this DOES heed dir_auth.pending 2023-10-30T13:52:20.600 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.600 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.600 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CDir::is_subtree_root() 2023-10-30T13:52:20.600 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.600 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir_auth == CDIR_AUTH_DEFAULT) { 2023-10-30T13:52:20.601 INFO:tasks.workunit.client.0.smithi148.stdout:- //dout(10) << "is_subtree_root false " << dir_auth << " != " << CDIR_AUTH_DEFAULT 2023-10-30T13:52:20.601 INFO:tasks.workunit.client.0.smithi148.stdout:- //<< " on " << ino() << dendl; 2023-10-30T13:52:20.601 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.601 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.601 INFO:tasks.workunit.client.0.smithi148.stdout:- //dout(10) << "is_subtree_root true " << dir_auth << " != " << CDIR_AUTH_DEFAULT 2023-10-30T13:52:20.602 INFO:tasks.workunit.client.0.smithi148.stdout:- //<< " on " << ino() << dendl; 2023-10-30T13:52:20.602 INFO:tasks.workunit.client.0.smithi148.stdout:- return true; 2023-10-30T13:52:20.602 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.602 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.602 INFO:tasks.workunit.client.0.smithi148.stdout:-*/ 2023-10-30T13:52:20.602 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.603 INFO:tasks.workunit.client.0.smithi148.stdout:-/** contains(x) 2023-10-30T13:52:20.603 INFO:tasks.workunit.client.0.smithi148.stdout:- * true if we are x, or an ancestor of x 2023-10-30T13:52:20.603 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.603 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CDir::contains(CDir *x) 2023-10-30T13:52:20.603 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.604 INFO:tasks.workunit.client.0.smithi148.stdout:- while (1) { 2023-10-30T13:52:20.604 INFO:tasks.workunit.client.0.smithi148.stdout:- if (x == this) return true; 2023-10-30T13:52:20.604 INFO:tasks.workunit.client.0.smithi148.stdout:- x = x->get_parent_dir(); 2023-10-30T13:52:20.604 INFO:tasks.workunit.client.0.smithi148.stdout:- if (x == 0) return false; 2023-10-30T13:52:20.604 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.605 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.605 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.605 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.605 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.605 INFO:tasks.workunit.client.0.smithi148.stdout:-/** set_dir_auth 2023-10-30T13:52:20.606 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.606 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::set_dir_auth(pair a) 2023-10-30T13:52:20.606 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.606 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "setting dir_auth=" << a 2023-10-30T13:52:20.606 INFO:tasks.workunit.client.0.smithi148.stdout:- << " from " << dir_auth 2023-10-30T13:52:20.607 INFO:tasks.workunit.client.0.smithi148.stdout:- << " on " << *this << dendl; 2023-10-30T13:52:20.607 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.607 INFO:tasks.workunit.client.0.smithi148.stdout:- bool was_subtree = is_subtree_root(); 2023-10-30T13:52:20.607 INFO:tasks.workunit.client.0.smithi148.stdout:- bool was_ambiguous = dir_auth.second >= 0; 2023-10-30T13:52:20.607 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.608 INFO:tasks.workunit.client.0.smithi148.stdout:- // set it. 2023-10-30T13:52:20.608 INFO:tasks.workunit.client.0.smithi148.stdout:- dir_auth = a; 2023-10-30T13:52:20.608 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.608 INFO:tasks.workunit.client.0.smithi148.stdout:- // new subtree root? 2023-10-30T13:52:20.608 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!was_subtree && is_subtree_root()) { 2023-10-30T13:52:20.608 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << " new subtree root, adjusting auth_pins" << dendl; 2023-10-30T13:52:20.609 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.609 INFO:tasks.workunit.client.0.smithi148.stdout:- // adjust nested auth pins 2023-10-30T13:52:20.609 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->adjust_nested_auth_pins(-get_cum_auth_pins()); 2023-10-30T13:52:20.609 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.609 INFO:tasks.workunit.client.0.smithi148.stdout:- // unpin parent of frozen dir/tree? 2023-10-30T13:52:20.610 INFO:tasks.workunit.client.0.smithi148.stdout:- if (inode->is_auth() && (is_frozen_tree_root() || is_frozen_dir())) 2023-10-30T13:52:20.610 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->auth_unpin(); 2023-10-30T13:52:20.610 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.610 INFO:tasks.workunit.client.0.smithi148.stdout:- if (was_subtree && !is_subtree_root()) { 2023-10-30T13:52:20.610 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << " old subtree root, adjusting auth_pins" << dendl; 2023-10-30T13:52:20.611 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.611 INFO:tasks.workunit.client.0.smithi148.stdout:- // adjust nested auth pins 2023-10-30T13:52:20.611 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->adjust_nested_auth_pins(get_cum_auth_pins()); 2023-10-30T13:52:20.611 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.611 INFO:tasks.workunit.client.0.smithi148.stdout:- // pin parent of frozen dir/tree? 2023-10-30T13:52:20.611 INFO:tasks.workunit.client.0.smithi148.stdout:- if (inode->is_auth() && (is_frozen_tree_root() || is_frozen_dir())) 2023-10-30T13:52:20.612 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->auth_pin(); 2023-10-30T13:52:20.612 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.612 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.612 INFO:tasks.workunit.client.0.smithi148.stdout:- // newly single auth? 2023-10-30T13:52:20.612 INFO:tasks.workunit.client.0.smithi148.stdout:- if (was_ambiguous && dir_auth.second == CDIR_AUTH_UNKNOWN) { 2023-10-30T13:52:20.612 INFO:tasks.workunit.client.0.smithi148.stdout:- list ls; 2023-10-30T13:52:20.613 INFO:tasks.workunit.client.0.smithi148.stdout:- take_waiting(WAIT_SINGLEAUTH, ls); 2023-10-30T13:52:20.613 INFO:tasks.workunit.client.0.smithi148.stdout:- cache->mds->queue_waiters(ls); 2023-10-30T13:52:20.613 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.613 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.613 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.613 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.614 INFO:tasks.workunit.client.0.smithi148.stdout:-/***************************************** 2023-10-30T13:52:20.614 INFO:tasks.workunit.client.0.smithi148.stdout:- * AUTH PINS and FREEZING 2023-10-30T13:52:20.614 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.614 INFO:tasks.workunit.client.0.smithi148.stdout:- * the basic plan is that auth_pins only exist in auth regions, and they 2023-10-30T13:52:20.614 INFO:tasks.workunit.client.0.smithi148.stdout:- * prevent a freeze (and subsequent auth change). 2023-10-30T13:52:20.615 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.615 INFO:tasks.workunit.client.0.smithi148.stdout:- * however, we also need to prevent a parent from freezing if a child is frozen. 2023-10-30T13:52:20.615 INFO:tasks.workunit.client.0.smithi148.stdout:- * for that reason, the parent inode of a frozen directory is auth_pinned. 2023-10-30T13:52:20.615 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.615 INFO:tasks.workunit.client.0.smithi148.stdout:- * the oddity is when the frozen directory is a subtree root. if that's the case, 2023-10-30T13:52:20.615 INFO:tasks.workunit.client.0.smithi148.stdout:- * the parent inode isn't frozen. which means that when subtree authority is adjusted 2023-10-30T13:52:20.616 INFO:tasks.workunit.client.0.smithi148.stdout:- * at the bounds, inodes for any frozen bound directories need to get auth_pins at that 2023-10-30T13:52:20.616 INFO:tasks.workunit.client.0.smithi148.stdout:- * time. 2023-10-30T13:52:20.616 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.616 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.616 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.617 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::auth_pin() 2023-10-30T13:52:20.617 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.617 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth_pins == 0) 2023-10-30T13:52:20.617 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_AUTHPIN); 2023-10-30T13:52:20.617 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins++; 2023-10-30T13:52:20.618 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.618 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "auth_pin on " << *this << " count now " << auth_pins << " + " << nested_auth_pins << dendl; 2023-10-30T13:52:20.618 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.618 INFO:tasks.workunit.client.0.smithi148.stdout:- // nest pins? 2023-10-30T13:52:20.618 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_subtree_root()) return; // no. 2023-10-30T13:52:20.619 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(!is_import()); 2023-10-30T13:52:20.619 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.619 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->adjust_nested_auth_pins(1); 2023-10-30T13:52:20.619 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.619 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.619 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::auth_unpin() 2023-10-30T13:52:20.620 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.620 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins--; 2023-10-30T13:52:20.620 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth_pins == 0) 2023-10-30T13:52:20.620 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_AUTHPIN); 2023-10-30T13:52:20.620 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.621 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "auth_unpin on " << *this << " count now " << auth_pins << " + " << nested_auth_pins << dendl; 2023-10-30T13:52:20.621 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(auth_pins >= 0); 2023-10-30T13:52:20.621 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.621 INFO:tasks.workunit.client.0.smithi148.stdout:- maybe_finish_freeze(); // pending freeze? 2023-10-30T13:52:20.621 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.621 INFO:tasks.workunit.client.0.smithi148.stdout:- // nest? 2023-10-30T13:52:20.622 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_subtree_root()) return; // no. 2023-10-30T13:52:20.622 INFO:tasks.workunit.client.0.smithi148.stdout:- //assert(!is_import()); 2023-10-30T13:52:20.622 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.622 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->adjust_nested_auth_pins(-1); 2023-10-30T13:52:20.623 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.623 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.623 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::adjust_nested_auth_pins(int inc) 2023-10-30T13:52:20.623 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.623 INFO:tasks.workunit.client.0.smithi148.stdout:- nested_auth_pins += inc; 2023-10-30T13:52:20.623 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.624 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << "adjust_nested_auth_pins " << inc << " on " << *this 2023-10-30T13:52:20.624 INFO:tasks.workunit.client.0.smithi148.stdout:- << " count now " << auth_pins << " + " << nested_auth_pins << dendl; 2023-10-30T13:52:20.624 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nested_auth_pins >= 0); 2023-10-30T13:52:20.624 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.624 INFO:tasks.workunit.client.0.smithi148.stdout:- maybe_finish_freeze(); // pending freeze? 2023-10-30T13:52:20.624 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.625 INFO:tasks.workunit.client.0.smithi148.stdout:- // adjust my inode? 2023-10-30T13:52:20.625 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_subtree_root()) 2023-10-30T13:52:20.625 INFO:tasks.workunit.client.0.smithi148.stdout:- return; // no, stop. 2023-10-30T13:52:20.625 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.625 INFO:tasks.workunit.client.0.smithi148.stdout:- // yes. 2023-10-30T13:52:20.625 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->adjust_nested_auth_pins(inc); 2023-10-30T13:52:20.626 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.626 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.626 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.626 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.626 INFO:tasks.workunit.client.0.smithi148.stdout:-/***************************************************************************** 2023-10-30T13:52:20.627 INFO:tasks.workunit.client.0.smithi148.stdout:- * FREEZING 2023-10-30T13:52:20.627 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.627 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.627 INFO:tasks.workunit.client.0.smithi148.stdout:-// FREEZE TREE 2023-10-30T13:52:20.627 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.627 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CDir::freeze_tree() 2023-10-30T13:52:20.628 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.628 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!is_frozen()); 2023-10-30T13:52:20.628 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!is_freezing()); 2023-10-30T13:52:20.628 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.628 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pin(); 2023-10-30T13:52:20.629 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_freezeable(true)) { 2023-10-30T13:52:20.629 INFO:tasks.workunit.client.0.smithi148.stdout:- _freeze_tree(); 2023-10-30T13:52:20.629 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_unpin(); 2023-10-30T13:52:20.629 INFO:tasks.workunit.client.0.smithi148.stdout:- return true; 2023-10-30T13:52:20.629 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.629 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_FREEZINGTREE); 2023-10-30T13:52:20.630 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "freeze_tree waiting " << *this << dendl; 2023-10-30T13:52:20.630 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.630 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.630 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.630 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.630 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::_freeze_tree() 2023-10-30T13:52:20.631 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.631 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "_freeze_tree " << *this << dendl; 2023-10-30T13:52:20.631 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_freezeable(true)); 2023-10-30T13:52:20.631 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.631 INFO:tasks.workunit.client.0.smithi148.stdout:- // twiddle state 2023-10-30T13:52:20.631 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FREEZINGTREE); // actually, this may get set again by next context? 2023-10-30T13:52:20.632 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_FROZENTREE); 2023-10-30T13:52:20.632 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_FROZEN); 2023-10-30T13:52:20.632 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.632 INFO:tasks.workunit.client.0.smithi148.stdout:- // auth_pin inode for duration of freeze, if we are not a subtree root. 2023-10-30T13:52:20.632 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth() && !is_subtree_root()) 2023-10-30T13:52:20.632 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->auth_pin(); 2023-10-30T13:52:20.633 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.633 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.633 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::unfreeze_tree() 2023-10-30T13:52:20.633 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.633 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "unfreeze_tree " << *this << dendl; 2023-10-30T13:52:20.633 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.634 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_FROZENTREE)) { 2023-10-30T13:52:20.634 INFO:tasks.workunit.client.0.smithi148.stdout:- // frozen. unfreeze. 2023-10-30T13:52:20.634 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FROZENTREE); 2023-10-30T13:52:20.634 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_FROZEN); 2023-10-30T13:52:20.634 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.634 INFO:tasks.workunit.client.0.smithi148.stdout:- // unpin (may => FREEZEABLE) FIXME: is this order good? 2023-10-30T13:52:20.635 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth() && !is_subtree_root()) 2023-10-30T13:52:20.635 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->auth_unpin(); 2023-10-30T13:52:20.635 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.635 INFO:tasks.workunit.client.0.smithi148.stdout:- // waiters? 2023-10-30T13:52:20.635 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_UNFREEZE); 2023-10-30T13:52:20.636 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.636 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_FROZEN, -1); 2023-10-30T13:52:20.636 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.636 INFO:tasks.workunit.client.0.smithi148.stdout:- // freezing. stop it. 2023-10-30T13:52:20.636 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(state_test(STATE_FREEZINGTREE)); 2023-10-30T13:52:20.637 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FREEZINGTREE); 2023-10-30T13:52:20.637 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_unpin(); 2023-10-30T13:52:20.637 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.637 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_UNFREEZE); 2023-10-30T13:52:20.637 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.638 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.638 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.638 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CDir::is_freezing_tree() 2023-10-30T13:52:20.638 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.638 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir = this; 2023-10-30T13:52:20.638 INFO:tasks.workunit.client.0.smithi148.stdout:- while (1) { 2023-10-30T13:52:20.639 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->is_freezing_tree_root()) return true; 2023-10-30T13:52:20.639 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->is_subtree_root()) return false; 2023-10-30T13:52:20.639 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->inode->parent) 2023-10-30T13:52:20.639 INFO:tasks.workunit.client.0.smithi148.stdout:- dir = dir->inode->parent->dir; 2023-10-30T13:52:20.639 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.640 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; // root on replica 2023-10-30T13:52:20.640 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.640 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.640 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.640 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CDir::is_frozen_tree() 2023-10-30T13:52:20.641 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.641 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir = this; 2023-10-30T13:52:20.641 INFO:tasks.workunit.client.0.smithi148.stdout:- while (1) { 2023-10-30T13:52:20.641 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->is_frozen_tree_root()) return true; 2023-10-30T13:52:20.642 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->is_subtree_root()) return false; 2023-10-30T13:52:20.642 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->inode->parent) 2023-10-30T13:52:20.642 INFO:tasks.workunit.client.0.smithi148.stdout:- dir = dir->inode->parent->dir; 2023-10-30T13:52:20.642 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.642 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; // root on replica 2023-10-30T13:52:20.642 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.643 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.643 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.643 INFO:tasks.workunit.client.0.smithi148.stdout:-CDir *CDir::get_frozen_tree_root() 2023-10-30T13:52:20.643 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.643 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_frozen()); 2023-10-30T13:52:20.643 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir = this; 2023-10-30T13:52:20.644 INFO:tasks.workunit.client.0.smithi148.stdout:- while (1) { 2023-10-30T13:52:20.644 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->is_frozen_tree_root()) 2023-10-30T13:52:20.644 INFO:tasks.workunit.client.0.smithi148.stdout:- return dir; 2023-10-30T13:52:20.644 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->inode->parent) 2023-10-30T13:52:20.644 INFO:tasks.workunit.client.0.smithi148.stdout:- dir = dir->inode->parent->dir; 2023-10-30T13:52:20.644 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.645 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(0); 2023-10-30T13:52:20.645 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.645 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.645 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.645 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.646 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.646 INFO:tasks.workunit.client.0.smithi148.stdout:-// FREEZE DIR 2023-10-30T13:52:20.646 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.646 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CDir::freeze_dir() 2023-10-30T13:52:20.646 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.647 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!is_frozen()); 2023-10-30T13:52:20.647 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!is_freezing()); 2023-10-30T13:52:20.647 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.647 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pin(); 2023-10-30T13:52:20.647 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_freezeable_dir(true)) { 2023-10-30T13:52:20.648 INFO:tasks.workunit.client.0.smithi148.stdout:- _freeze_dir(); 2023-10-30T13:52:20.648 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_unpin(); 2023-10-30T13:52:20.648 INFO:tasks.workunit.client.0.smithi148.stdout:- return true; 2023-10-30T13:52:20.648 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.648 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_FREEZINGDIR); 2023-10-30T13:52:20.649 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "freeze_dir + wait " << *this << dendl; 2023-10-30T13:52:20.649 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.649 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.649 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.649 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.650 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::_freeze_dir() 2023-10-30T13:52:20.650 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.650 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "_freeze_dir " << *this << dendl; 2023-10-30T13:52:20.650 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_freezeable_dir(true)); 2023-10-30T13:52:20.650 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.651 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FREEZINGDIR); 2023-10-30T13:52:20.651 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_FROZENDIR); 2023-10-30T13:52:20.651 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_FROZEN); 2023-10-30T13:52:20.651 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.651 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth() && !is_subtree_root()) 2023-10-30T13:52:20.652 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->auth_pin(); // auth_pin for duration of freeze 2023-10-30T13:52:20.652 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.652 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.652 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.652 INFO:tasks.workunit.client.0.smithi148.stdout:-void CDir::unfreeze_dir() 2023-10-30T13:52:20.653 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.653 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "unfreeze_dir " << *this << dendl; 2023-10-30T13:52:20.653 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.653 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_FROZENDIR)) { 2023-10-30T13:52:20.653 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FROZENDIR); 2023-10-30T13:52:20.654 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_FROZEN); 2023-10-30T13:52:20.654 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.654 INFO:tasks.workunit.client.0.smithi148.stdout:- // unpin (may => FREEZEABLE) FIXME: is this order good? 2023-10-30T13:52:20.654 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth() && !is_subtree_root()) 2023-10-30T13:52:20.654 INFO:tasks.workunit.client.0.smithi148.stdout:- inode->auth_unpin(); 2023-10-30T13:52:20.655 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.655 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_UNFREEZE); 2023-10-30T13:52:20.655 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.655 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_FROZEN, -1); 2023-10-30T13:52:20.655 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.656 INFO:tasks.workunit.client.0.smithi148.stdout:- // still freezing. stop. 2023-10-30T13:52:20.656 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(state_test(STATE_FREEZINGDIR)); 2023-10-30T13:52:20.656 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FREEZINGDIR); 2023-10-30T13:52:20.656 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_unpin(); 2023-10-30T13:52:20.656 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.656 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_UNFREEZE); 2023-10-30T13:52:20.657 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.657 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.657 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.657 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.657 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.657 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.658 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.658 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.658 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.658 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.658 INFO:tasks.workunit.client.0.smithi148.stdout:diff --git a/src/mds/CDir.h b/src/mds/CDir.h 2023-10-30T13:52:20.659 INFO:tasks.workunit.client.0.smithi148.stdout:index 99bad3801e1..e69de29bb2d 100644 2023-10-30T13:52:20.659 INFO:tasks.workunit.client.0.smithi148.stdout:--- a/src/mds/CDir.h 2023-10-30T13:52:20.659 INFO:tasks.workunit.client.0.smithi148.stdout:+++ b/src/mds/CDir.h 2023-10-30T13:52:20.659 INFO:tasks.workunit.client.0.smithi148.stdout:@@ -1,540 +0,0 @@ 2023-10-30T13:52:20.659 INFO:tasks.workunit.client.0.smithi148.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2023-10-30T13:52:20.659 INFO:tasks.workunit.client.0.smithi148.stdout:-// vim: ts=8 sw=2 smarttab 2023-10-30T13:52:20.660 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.660 INFO:tasks.workunit.client.0.smithi148.stdout:- * Ceph - scalable distributed file system 2023-10-30T13:52:20.660 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.660 INFO:tasks.workunit.client.0.smithi148.stdout:- * Copyright (C) 2004-2006 Sage Weil 2023-10-30T13:52:20.660 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.661 INFO:tasks.workunit.client.0.smithi148.stdout:- * This is free software; you can redistribute it and/or 2023-10-30T13:52:20.661 INFO:tasks.workunit.client.0.smithi148.stdout:- * modify it under the terms of the GNU Lesser General Public 2023-10-30T13:52:20.661 INFO:tasks.workunit.client.0.smithi148.stdout:- * License version 2.1, as published by the Free Software 2023-10-30T13:52:20.661 INFO:tasks.workunit.client.0.smithi148.stdout:- * Foundation. See file COPYING. 2023-10-30T13:52:20.661 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.661 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.662 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.662 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.662 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.662 INFO:tasks.workunit.client.0.smithi148.stdout:-#ifndef __CDIR_H 2023-10-30T13:52:20.662 INFO:tasks.workunit.client.0.smithi148.stdout:-#define __CDIR_H 2023-10-30T13:52:20.663 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.663 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/types.h" 2023-10-30T13:52:20.663 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/buffer.h" 2023-10-30T13:52:20.663 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "mdstypes.h" 2023-10-30T13:52:20.663 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "config.h" 2023-10-30T13:52:20.664 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "common/DecayCounter.h" 2023-10-30T13:52:20.664 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.664 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.664 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.664 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.664 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.665 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.665 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.665 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.665 INFO:tasks.workunit.client.0.smithi148.stdout:-using namespace std; 2023-10-30T13:52:20.665 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.666 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.666 INFO:tasks.workunit.client.0.smithi148.stdout:-using __gnu_cxx::hash_map; 2023-10-30T13:52:20.666 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.666 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.666 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CInode.h" 2023-10-30T13:52:20.667 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.667 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDentry; 2023-10-30T13:52:20.667 INFO:tasks.workunit.client.0.smithi148.stdout:-class MDCache; 2023-10-30T13:52:20.667 INFO:tasks.workunit.client.0.smithi148.stdout:-class MDCluster; 2023-10-30T13:52:20.667 INFO:tasks.workunit.client.0.smithi148.stdout:-class Context; 2023-10-30T13:52:20.667 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDirDiscover; 2023-10-30T13:52:20.668 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.668 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.668 INFO:tasks.workunit.client.0.smithi148.stdout:-ostream& operator<<(ostream& out, class CDir& dir); 2023-10-30T13:52:20.668 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.668 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.669 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDir : public MDSCacheObject { 2023-10-30T13:52:20.669 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.669 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- pins -- 2023-10-30T13:52:20.669 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_DNWAITER = 1; 2023-10-30T13:52:20.670 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_INOWAITER = 2; 2023-10-30T13:52:20.670 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_CHILD = 3; 2023-10-30T13:52:20.670 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_FROZEN = 4; 2023-10-30T13:52:20.670 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_SUBTREE = 5; 2023-10-30T13:52:20.670 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_IMPORTING = 7; 2023-10-30T13:52:20.671 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_IMPORTBOUND = 9; 2023-10-30T13:52:20.671 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_EXPORTBOUND = 10; 2023-10-30T13:52:20.671 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_STICKY = 11; 2023-10-30T13:52:20.671 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_SUBTREETEMP = 12; // used by MDCache::trim_non_auth() 2023-10-30T13:52:20.671 INFO:tasks.workunit.client.0.smithi148.stdout:- const char *pin_name(int p) { 2023-10-30T13:52:20.672 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (p) { 2023-10-30T13:52:20.672 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_DNWAITER: return "dnwaiter"; 2023-10-30T13:52:20.672 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_INOWAITER: return "inowaiter"; 2023-10-30T13:52:20.672 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_CHILD: return "child"; 2023-10-30T13:52:20.672 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_FROZEN: return "frozen"; 2023-10-30T13:52:20.672 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_SUBTREE: return "subtree"; 2023-10-30T13:52:20.673 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_IMPORTING: return "importing"; 2023-10-30T13:52:20.673 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_IMPORTBOUND: return "importbound"; 2023-10-30T13:52:20.673 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_EXPORTBOUND: return "exportbound"; 2023-10-30T13:52:20.673 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_STICKY: return "sticky"; 2023-10-30T13:52:20.673 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_SUBTREETEMP: return "subtreetemp"; 2023-10-30T13:52:20.673 INFO:tasks.workunit.client.0.smithi148.stdout:- default: return generic_pin_name(p); 2023-10-30T13:52:20.674 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.674 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.674 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.674 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- state -- 2023-10-30T13:52:20.674 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_COMPLETE = (1<< 1); // the complete contents are in cache 2023-10-30T13:52:20.674 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_FROZENTREE = (1<< 2); // root of tree (bounded by exports) 2023-10-30T13:52:20.675 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_FREEZINGTREE = (1<< 3); // in process of freezing 2023-10-30T13:52:20.675 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_FROZENDIR = (1<< 4); 2023-10-30T13:52:20.675 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_FREEZINGDIR = (1<< 5); 2023-10-30T13:52:20.675 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_COMMITTING = (1<< 6); // mid-commit 2023-10-30T13:52:20.675 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_FETCHING = (1<< 7); // currenting fetching 2023-10-30T13:52:20.675 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_IMPORTBOUND = (1<<10); 2023-10-30T13:52:20.676 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_EXPORTBOUND = (1<<11); 2023-10-30T13:52:20.676 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_EXPORTING = (1<<12); 2023-10-30T13:52:20.676 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_IMPORTING = (1<<13); 2023-10-30T13:52:20.676 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_FRAGMENTING = (1<<14); 2023-10-30T13:52:20.676 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_STICKY = (1<<15); // sticky pin due to inode stickydirs 2023-10-30T13:52:20.676 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_DNPINNEDFRAG = (1<<16); // dir is refragmenting 2023-10-30T13:52:20.677 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.677 INFO:tasks.workunit.client.0.smithi148.stdout:- // common states 2023-10-30T13:52:20.677 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_CLEAN = 0; 2023-10-30T13:52:20.677 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned STATE_INITIAL = 0; 2023-10-30T13:52:20.677 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.677 INFO:tasks.workunit.client.0.smithi148.stdout:- // these state bits are preserved by an import/export 2023-10-30T13:52:20.677 INFO:tasks.workunit.client.0.smithi148.stdout:- // ...except if the directory is hashed, in which case none of them are! 2023-10-30T13:52:20.678 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned MASK_STATE_EXPORTED = 2023-10-30T13:52:20.678 INFO:tasks.workunit.client.0.smithi148.stdout:- (STATE_COMPLETE|STATE_DIRTY); 2023-10-30T13:52:20.678 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned MASK_STATE_IMPORT_KEPT = 2023-10-30T13:52:20.678 INFO:tasks.workunit.client.0.smithi148.stdout:- ( 2023-10-30T13:52:20.678 INFO:tasks.workunit.client.0.smithi148.stdout:- STATE_IMPORTING 2023-10-30T13:52:20.679 INFO:tasks.workunit.client.0.smithi148.stdout:- |STATE_IMPORTBOUND|STATE_EXPORTBOUND 2023-10-30T13:52:20.679 INFO:tasks.workunit.client.0.smithi148.stdout:- |STATE_FROZENTREE 2023-10-30T13:52:20.679 INFO:tasks.workunit.client.0.smithi148.stdout:- |STATE_STICKY); 2023-10-30T13:52:20.679 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned MASK_STATE_EXPORT_KEPT = 2023-10-30T13:52:20.679 INFO:tasks.workunit.client.0.smithi148.stdout:- (STATE_EXPORTING 2023-10-30T13:52:20.680 INFO:tasks.workunit.client.0.smithi148.stdout:- |STATE_IMPORTBOUND|STATE_EXPORTBOUND 2023-10-30T13:52:20.680 INFO:tasks.workunit.client.0.smithi148.stdout:- |STATE_FROZENTREE 2023-10-30T13:52:20.680 INFO:tasks.workunit.client.0.smithi148.stdout:- |STATE_FROZENDIR 2023-10-30T13:52:20.680 INFO:tasks.workunit.client.0.smithi148.stdout:- |STATE_STICKY); 2023-10-30T13:52:20.680 INFO:tasks.workunit.client.0.smithi148.stdout:- static const unsigned MASK_STATE_FRAGMENT_KEPT = 2023-10-30T13:52:20.680 INFO:tasks.workunit.client.0.smithi148.stdout:- (STATE_DIRTY | 2023-10-30T13:52:20.681 INFO:tasks.workunit.client.0.smithi148.stdout:- STATE_COMPLETE | 2023-10-30T13:52:20.681 INFO:tasks.workunit.client.0.smithi148.stdout:- STATE_EXPORTBOUND | 2023-10-30T13:52:20.681 INFO:tasks.workunit.client.0.smithi148.stdout:- STATE_IMPORTBOUND); 2023-10-30T13:52:20.681 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.681 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- rep spec -- 2023-10-30T13:52:20.681 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int REP_NONE = 0; 2023-10-30T13:52:20.682 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int REP_ALL = 1; 2023-10-30T13:52:20.682 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int REP_LIST = 2; 2023-10-30T13:52:20.682 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.682 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.682 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int NONCE_EXPORT = 1; 2023-10-30T13:52:20.682 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.683 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.683 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- wait masks -- 2023-10-30T13:52:20.683 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_DENTRY = (1<<0); // wait for item to be in cache 2023-10-30T13:52:20.683 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_COMPLETE = (1<<1); // wait for complete dir contents 2023-10-30T13:52:20.683 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_FROZEN = (1<<2); // auth pins removed 2023-10-30T13:52:20.683 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.684 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_DNLOCK_OFFSET = 4; 2023-10-30T13:52:20.684 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.684 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_ANY = (0xffffffff); 2023-10-30T13:52:20.684 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_ATFREEZEROOT = (WAIT_UNFREEZE); 2023-10-30T13:52:20.684 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_ATSUBTREEROOT = (WAIT_SINGLEAUTH); 2023-10-30T13:52:20.684 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.685 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.685 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.685 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.685 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.685 INFO:tasks.workunit.client.0.smithi148.stdout:- // context 2023-10-30T13:52:20.685 INFO:tasks.workunit.client.0.smithi148.stdout:- MDCache *cache; 2023-10-30T13:52:20.686 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.686 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *inode; // my inode 2023-10-30T13:52:20.686 INFO:tasks.workunit.client.0.smithi148.stdout:- frag_t frag; // my frag 2023-10-30T13:52:20.686 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.686 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_lt(const MDSCacheObject *r) const { 2023-10-30T13:52:20.687 INFO:tasks.workunit.client.0.smithi148.stdout:- return dirfrag() < ((const CDir*)r)->dirfrag(); 2023-10-30T13:52:20.687 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.687 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.687 INFO:tasks.workunit.client.0.smithi148.stdout:- //int hack_num_accessed; 2023-10-30T13:52:20.687 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.687 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.688 INFO:tasks.workunit.client.0.smithi148.stdout:- //typedef hash_map map_t; // there is a bug somewhere, valgrind me. 2023-10-30T13:52:20.688 INFO:tasks.workunit.client.0.smithi148.stdout:- typedef map map_t; 2023-10-30T13:52:20.688 INFO:tasks.workunit.client.0.smithi148.stdout:-protected: 2023-10-30T13:52:20.688 INFO:tasks.workunit.client.0.smithi148.stdout:- // contents 2023-10-30T13:52:20.688 INFO:tasks.workunit.client.0.smithi148.stdout:- map_t items; // non-null AND null 2023-10-30T13:52:20.689 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned nitems; // # non-null 2023-10-30T13:52:20.689 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned nnull; // # null 2023-10-30T13:52:20.689 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.689 INFO:tasks.workunit.client.0.smithi148.stdout:- int num_dirty; 2023-10-30T13:52:20.689 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.689 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.690 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.690 INFO:tasks.workunit.client.0.smithi148.stdout:- // state 2023-10-30T13:52:20.690 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t version; 2023-10-30T13:52:20.690 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t committing_version; 2023-10-30T13:52:20.690 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t committed_version; 2023-10-30T13:52:20.690 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t committed_version_equivalent; // in case of, e.g., temporary file 2023-10-30T13:52:20.691 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t projected_version; 2023-10-30T13:52:20.691 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.691 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist::item xlist_dirty; 2023-10-30T13:52:20.691 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.691 INFO:tasks.workunit.client.0.smithi148.stdout:- // lock nesting, freeze 2023-10-30T13:52:20.691 INFO:tasks.workunit.client.0.smithi148.stdout:- int auth_pins; 2023-10-30T13:52:20.692 INFO:tasks.workunit.client.0.smithi148.stdout:- int nested_auth_pins; 2023-10-30T13:52:20.692 INFO:tasks.workunit.client.0.smithi148.stdout:- int request_pins; 2023-10-30T13:52:20.692 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.692 INFO:tasks.workunit.client.0.smithi148.stdout:- // cache control (defined for authority; hints for replicas) 2023-10-30T13:52:20.692 INFO:tasks.workunit.client.0.smithi148.stdout:- int dir_rep; 2023-10-30T13:52:20.692 INFO:tasks.workunit.client.0.smithi148.stdout:- set dir_rep_by; // if dir_rep == REP_LIST 2023-10-30T13:52:20.693 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.693 INFO:tasks.workunit.client.0.smithi148.stdout:- // popularity 2023-10-30T13:52:20.693 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrag_load_vec_t pop_me; 2023-10-30T13:52:20.693 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrag_load_vec_t pop_nested; 2023-10-30T13:52:20.693 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrag_load_vec_t pop_auth_subtree; 2023-10-30T13:52:20.693 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrag_load_vec_t pop_auth_subtree_nested; 2023-10-30T13:52:20.694 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.694 INFO:tasks.workunit.client.0.smithi148.stdout:- utime_t last_popularity_sample; 2023-10-30T13:52:20.694 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.694 INFO:tasks.workunit.client.0.smithi148.stdout:- load_spread_t pop_spread; 2023-10-30T13:52:20.694 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.694 INFO:tasks.workunit.client.0.smithi148.stdout:- // and to provide density 2023-10-30T13:52:20.695 INFO:tasks.workunit.client.0.smithi148.stdout:- int num_dentries_nested; 2023-10-30T13:52:20.695 INFO:tasks.workunit.client.0.smithi148.stdout:- int num_dentries_auth_subtree; 2023-10-30T13:52:20.695 INFO:tasks.workunit.client.0.smithi148.stdout:- int num_dentries_auth_subtree_nested; 2023-10-30T13:52:20.695 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.695 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.695 INFO:tasks.workunit.client.0.smithi148.stdout:- // friends 2023-10-30T13:52:20.696 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class Migrator; 2023-10-30T13:52:20.696 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class CInode; 2023-10-30T13:52:20.696 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class MDCache; 2023-10-30T13:52:20.696 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class MDiscover; 2023-10-30T13:52:20.696 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class MDBalancer; 2023-10-30T13:52:20.696 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.697 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class CDirDiscover; 2023-10-30T13:52:20.697 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class CDirExport; 2023-10-30T13:52:20.697 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.697 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.697 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir(CInode *in, frag_t fg, MDCache *mdcache, bool auth); 2023-10-30T13:52:20.697 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.698 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.698 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.698 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- accessors -- 2023-10-30T13:52:20.698 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t ino() const { return inode->ino(); } // deprecate me? 2023-10-30T13:52:20.698 INFO:tasks.workunit.client.0.smithi148.stdout:- frag_t get_frag() const { return frag; } 2023-10-30T13:52:20.699 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrag_t dirfrag() const { return dirfrag_t(inode->ino(), frag); } 2023-10-30T13:52:20.699 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.699 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *get_inode() { return inode; } 2023-10-30T13:52:20.699 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *get_parent_dir() { return inode->get_parent_dir(); } 2023-10-30T13:52:20.699 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.699 INFO:tasks.workunit.client.0.smithi148.stdout:- map_t::iterator begin() { return items.begin(); } 2023-10-30T13:52:20.700 INFO:tasks.workunit.client.0.smithi148.stdout:- map_t::iterator end() { return items.end(); } 2023-10-30T13:52:20.700 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned get_size() { 2023-10-30T13:52:20.700 INFO:tasks.workunit.client.0.smithi148.stdout:- return nitems; 2023-10-30T13:52:20.700 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.700 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned get_nitems() { return nitems; } 2023-10-30T13:52:20.700 INFO:tasks.workunit.client.0.smithi148.stdout:- unsigned get_nnull() { return nnull; } 2023-10-30T13:52:20.701 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.701 INFO:tasks.workunit.client.0.smithi148.stdout:- void inc_num_dirty() { num_dirty++; } 2023-10-30T13:52:20.701 INFO:tasks.workunit.client.0.smithi148.stdout:- void dec_num_dirty() { 2023-10-30T13:52:20.701 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(num_dirty > 0); 2023-10-30T13:52:20.701 INFO:tasks.workunit.client.0.smithi148.stdout:- num_dirty--; 2023-10-30T13:52:20.701 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.702 INFO:tasks.workunit.client.0.smithi148.stdout:- int get_num_dirty() { 2023-10-30T13:52:20.702 INFO:tasks.workunit.client.0.smithi148.stdout:- return num_dirty; 2023-10-30T13:52:20.702 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.702 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.702 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.702 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- dentries and inodes -- 2023-10-30T13:52:20.703 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.703 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry* lookup(const string& n) { 2023-10-30T13:52:20.703 INFO:tasks.workunit.client.0.smithi148.stdout:- map_t::iterator iter = items.find(n); 2023-10-30T13:52:20.703 INFO:tasks.workunit.client.0.smithi148.stdout:- if (iter == items.end()) 2023-10-30T13:52:20.703 INFO:tasks.workunit.client.0.smithi148.stdout:- return 0; 2023-10-30T13:52:20.703 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.704 INFO:tasks.workunit.client.0.smithi148.stdout:- return iter->second; 2023-10-30T13:52:20.704 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.704 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.704 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry* add_null_dentry(const string& dname); 2023-10-30T13:52:20.704 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry* add_primary_dentry(const string& dname, CInode *in); 2023-10-30T13:52:20.704 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry* add_remote_dentry(const string& dname, inodeno_t ino, unsigned char d_type); 2023-10-30T13:52:20.705 INFO:tasks.workunit.client.0.smithi148.stdout:- void remove_dentry( CDentry *dn ); // delete dentry 2023-10-30T13:52:20.705 INFO:tasks.workunit.client.0.smithi148.stdout:- void link_remote_inode( CDentry *dn, inodeno_t ino, unsigned char d_type); 2023-10-30T13:52:20.705 INFO:tasks.workunit.client.0.smithi148.stdout:- void link_primary_inode( CDentry *dn, CInode *in ); 2023-10-30T13:52:20.705 INFO:tasks.workunit.client.0.smithi148.stdout:- void unlink_inode( CDentry *dn ); 2023-10-30T13:52:20.705 INFO:tasks.workunit.client.0.smithi148.stdout:- void try_remove_unlinked_dn(CDentry *dn); 2023-10-30T13:52:20.705 INFO:tasks.workunit.client.0.smithi148.stdout:-private: 2023-10-30T13:52:20.706 INFO:tasks.workunit.client.0.smithi148.stdout:- void link_inode_work( CDentry *dn, CInode *in ); 2023-10-30T13:52:20.706 INFO:tasks.workunit.client.0.smithi148.stdout:- void unlink_inode_work( CDentry *dn ); 2023-10-30T13:52:20.706 INFO:tasks.workunit.client.0.smithi148.stdout:- void remove_null_dentries(); 2023-10-30T13:52:20.706 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.706 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.707 INFO:tasks.workunit.client.0.smithi148.stdout:- void split(int bits, list& subs, list& waiters); 2023-10-30T13:52:20.707 INFO:tasks.workunit.client.0.smithi148.stdout:- void merge(int bits, list& waiters); 2023-10-30T13:52:20.707 INFO:tasks.workunit.client.0.smithi148.stdout:-private: 2023-10-30T13:52:20.707 INFO:tasks.workunit.client.0.smithi148.stdout:- void steal_dentry(CDentry *dn); // from another dir. used by merge/split. 2023-10-30T13:52:20.707 INFO:tasks.workunit.client.0.smithi148.stdout:- void purge_stolen(list& waiters); 2023-10-30T13:52:20.707 INFO:tasks.workunit.client.0.smithi148.stdout:- void init_fragment_pins(); 2023-10-30T13:52:20.708 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.708 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.708 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- authority -- 2023-10-30T13:52:20.708 INFO:tasks.workunit.client.0.smithi148.stdout:- /* 2023-10-30T13:52:20.708 INFO:tasks.workunit.client.0.smithi148.stdout:- * normal: !subtree_root 2023-10-30T13:52:20.709 INFO:tasks.workunit.client.0.smithi148.stdout:- * delegation: subtree_root 2023-10-30T13:52:20.709 INFO:tasks.workunit.client.0.smithi148.stdout:- * ambiguous: subtree_root 2023-10-30T13:52:20.709 INFO:tasks.workunit.client.0.smithi148.stdout:- * subtree_root 2023-10-30T13:52:20.709 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.709 INFO:tasks.workunit.client.0.smithi148.stdout:- pair dir_auth; 2023-10-30T13:52:20.709 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.710 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.710 INFO:tasks.workunit.client.0.smithi148.stdout:- pair authority(); 2023-10-30T13:52:20.710 INFO:tasks.workunit.client.0.smithi148.stdout:- pair get_dir_auth() { return dir_auth; } 2023-10-30T13:52:20.710 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_dir_auth(pair a); 2023-10-30T13:52:20.710 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_dir_auth(int a) { set_dir_auth(pair(a, CDIR_AUTH_UNKNOWN)); } 2023-10-30T13:52:20.711 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_ambiguous_dir_auth() { 2023-10-30T13:52:20.711 INFO:tasks.workunit.client.0.smithi148.stdout:- return dir_auth.second != CDIR_AUTH_UNKNOWN; 2023-10-30T13:52:20.711 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.711 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_full_dir_auth() { 2023-10-30T13:52:20.711 INFO:tasks.workunit.client.0.smithi148.stdout:- return is_auth() && !is_ambiguous_dir_auth(); 2023-10-30T13:52:20.711 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.712 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_full_dir_nonauth() { 2023-10-30T13:52:20.712 INFO:tasks.workunit.client.0.smithi148.stdout:- return !is_auth() && !is_ambiguous_dir_auth(); 2023-10-30T13:52:20.712 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.712 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.712 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_subtree_root() { 2023-10-30T13:52:20.713 INFO:tasks.workunit.client.0.smithi148.stdout:- return dir_auth != CDIR_AUTH_DEFAULT; 2023-10-30T13:52:20.713 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.713 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.713 INFO:tasks.workunit.client.0.smithi148.stdout:- bool contains(CDir *x); // true if we are x or an ancestor of x 2023-10-30T13:52:20.713 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.714 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.714 INFO:tasks.workunit.client.0.smithi148.stdout:- // for giving to clients 2023-10-30T13:52:20.714 INFO:tasks.workunit.client.0.smithi148.stdout:- void get_dist_spec(set& ls, int auth) { 2023-10-30T13:52:20.714 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_rep()) { 2023-10-30T13:52:20.714 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator p = replicas_begin(); 2023-10-30T13:52:20.715 INFO:tasks.workunit.client.0.smithi148.stdout:- p != replicas_end(); 2023-10-30T13:52:20.715 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.715 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.insert(p->first); 2023-10-30T13:52:20.715 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!ls.empty()) 2023-10-30T13:52:20.716 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.insert(auth); 2023-10-30T13:52:20.716 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.716 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.716 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.716 INFO:tasks.workunit.client.0.smithi148.stdout:- CDirDiscover *replicate_to(int mds); 2023-10-30T13:52:20.717 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.717 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.717 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- state -- 2023-10-30T13:52:20.717 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_complete() { return state & STATE_COMPLETE; } 2023-10-30T13:52:20.717 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_exporting() { return state & STATE_EXPORTING; } 2023-10-30T13:52:20.718 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_importing() { return state & STATE_IMPORTING; } 2023-10-30T13:52:20.718 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.718 INFO:tasks.workunit.client.0.smithi148.stdout:- int get_dir_rep() { return dir_rep; } 2023-10-30T13:52:20.718 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_rep() { 2023-10-30T13:52:20.718 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir_rep == REP_NONE) return false; 2023-10-30T13:52:20.719 INFO:tasks.workunit.client.0.smithi148.stdout:- return true; 2023-10-30T13:52:20.719 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.719 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.719 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- fetch -- 2023-10-30T13:52:20.719 INFO:tasks.workunit.client.0.smithi148.stdout:- object_t get_ondisk_object() { return object_t(ino(), frag); } 2023-10-30T13:52:20.720 INFO:tasks.workunit.client.0.smithi148.stdout:- void fetch(Context *c, bool ignore_authpinnability=false); 2023-10-30T13:52:20.720 INFO:tasks.workunit.client.0.smithi148.stdout:- void _fetched(bufferlist &bl); 2023-10-30T13:52:20.720 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.720 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- commit -- 2023-10-30T13:52:20.720 INFO:tasks.workunit.client.0.smithi148.stdout:- map > waiting_for_commit; 2023-10-30T13:52:20.721 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.721 INFO:tasks.workunit.client.0.smithi148.stdout:- void commit_to(version_t want); 2023-10-30T13:52:20.721 INFO:tasks.workunit.client.0.smithi148.stdout:- void commit(version_t want, Context *c); 2023-10-30T13:52:20.721 INFO:tasks.workunit.client.0.smithi148.stdout:- void _commit(version_t want); 2023-10-30T13:52:20.721 INFO:tasks.workunit.client.0.smithi148.stdout:- void _committed(version_t v); 2023-10-30T13:52:20.722 INFO:tasks.workunit.client.0.smithi148.stdout:- void wait_for_commit(Context *c, version_t v=0); 2023-10-30T13:52:20.722 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.722 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- dirtyness -- 2023-10-30T13:52:20.722 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t get_version() { return version; } 2023-10-30T13:52:20.722 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_version(version_t v) { projected_version = version = v; } 2023-10-30T13:52:20.723 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t get_projected_version() { return projected_version; } 2023-10-30T13:52:20.723 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t get_committing_version() { return committing_version; } 2023-10-30T13:52:20.723 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t get_committed_version() { return committed_version; } 2023-10-30T13:52:20.723 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t get_committed_version_equivalent() { return committed_version_equivalent; } 2023-10-30T13:52:20.723 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_committed_version(version_t v) { committed_version = v; } 2023-10-30T13:52:20.723 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.724 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t pre_dirty(version_t min=0); 2023-10-30T13:52:20.724 INFO:tasks.workunit.client.0.smithi148.stdout:- void _mark_dirty(LogSegment *ls); 2023-10-30T13:52:20.724 INFO:tasks.workunit.client.0.smithi148.stdout:- void mark_dirty(version_t pv, LogSegment *ls); 2023-10-30T13:52:20.724 INFO:tasks.workunit.client.0.smithi148.stdout:- void mark_clean(); 2023-10-30T13:52:20.724 INFO:tasks.workunit.client.0.smithi148.stdout:- void mark_complete() { state_set(STATE_COMPLETE); } 2023-10-30T13:52:20.725 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.725 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.725 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- reference counting -- 2023-10-30T13:52:20.725 INFO:tasks.workunit.client.0.smithi148.stdout:- void first_get(); 2023-10-30T13:52:20.725 INFO:tasks.workunit.client.0.smithi148.stdout:- void last_put(); 2023-10-30T13:52:20.726 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.726 INFO:tasks.workunit.client.0.smithi148.stdout:- void request_pin_get() { 2023-10-30T13:52:20.726 INFO:tasks.workunit.client.0.smithi148.stdout:- if (request_pins == 0) get(PIN_REQUEST); 2023-10-30T13:52:20.726 INFO:tasks.workunit.client.0.smithi148.stdout:- request_pins++; 2023-10-30T13:52:20.726 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.726 INFO:tasks.workunit.client.0.smithi148.stdout:- void request_pin_put() { 2023-10-30T13:52:20.727 INFO:tasks.workunit.client.0.smithi148.stdout:- request_pins--; 2023-10-30T13:52:20.727 INFO:tasks.workunit.client.0.smithi148.stdout:- if (request_pins == 0) put(PIN_REQUEST); 2023-10-30T13:52:20.727 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.727 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.728 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.728 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- waiters -- 2023-10-30T13:52:20.728 INFO:tasks.workunit.client.0.smithi148.stdout:-protected: 2023-10-30T13:52:20.728 INFO:tasks.workunit.client.0.smithi148.stdout:- hash_map< string, list > waiting_on_dentry; 2023-10-30T13:52:20.728 INFO:tasks.workunit.client.0.smithi148.stdout:- hash_map< inodeno_t, list > waiting_on_ino; 2023-10-30T13:52:20.729 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.729 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.729 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_waiting_for_dentry(const string& dn) { 2023-10-30T13:52:20.729 INFO:tasks.workunit.client.0.smithi148.stdout:- return waiting_on_dentry.count(dn); 2023-10-30T13:52:20.729 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.730 INFO:tasks.workunit.client.0.smithi148.stdout:- void add_dentry_waiter(const string& dentry, Context *c); 2023-10-30T13:52:20.730 INFO:tasks.workunit.client.0.smithi148.stdout:- void take_dentry_waiting(const string& dentry, list& ls); 2023-10-30T13:52:20.730 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.730 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_waiting_for_ino(inodeno_t ino) { 2023-10-30T13:52:20.730 INFO:tasks.workunit.client.0.smithi148.stdout:- return waiting_on_ino.count(ino); 2023-10-30T13:52:20.730 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.731 INFO:tasks.workunit.client.0.smithi148.stdout:- void add_ino_waiter(inodeno_t ino, Context *c); 2023-10-30T13:52:20.731 INFO:tasks.workunit.client.0.smithi148.stdout:- void take_ino_waiting(inodeno_t ino, list& ls); 2023-10-30T13:52:20.731 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.731 INFO:tasks.workunit.client.0.smithi148.stdout:- void take_sub_waiting(list& ls); // dentry or ino 2023-10-30T13:52:20.731 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.732 INFO:tasks.workunit.client.0.smithi148.stdout:- void add_waiter(int mask, Context *c); 2023-10-30T13:52:20.732 INFO:tasks.workunit.client.0.smithi148.stdout:- void take_waiting(int mask, list& ls); // may include dentry waiters 2023-10-30T13:52:20.732 INFO:tasks.workunit.client.0.smithi148.stdout:- void finish_waiting(int mask, int result = 0); // ditto 2023-10-30T13:52:20.732 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.732 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.733 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- import/export -- 2023-10-30T13:52:20.733 INFO:tasks.workunit.client.0.smithi148.stdout:- void encode_export(bufferlist& bl); 2023-10-30T13:52:20.733 INFO:tasks.workunit.client.0.smithi148.stdout:- void finish_export(utime_t now); 2023-10-30T13:52:20.733 INFO:tasks.workunit.client.0.smithi148.stdout:- void abort_export() { 2023-10-30T13:52:20.733 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_TEMPEXPORTING); 2023-10-30T13:52:20.734 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.734 INFO:tasks.workunit.client.0.smithi148.stdout:- void decode_import(bufferlist::iterator& blp); 2023-10-30T13:52:20.734 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.734 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.734 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- auth pins -- 2023-10-30T13:52:20.734 INFO:tasks.workunit.client.0.smithi148.stdout:- bool can_auth_pin() { return is_auth() && !(is_frozen() || is_freezing()); } 2023-10-30T13:52:20.735 INFO:tasks.workunit.client.0.smithi148.stdout:- int is_auth_pinned() { return auth_pins; } 2023-10-30T13:52:20.735 INFO:tasks.workunit.client.0.smithi148.stdout:- int get_cum_auth_pins() { return auth_pins + nested_auth_pins; } 2023-10-30T13:52:20.735 INFO:tasks.workunit.client.0.smithi148.stdout:- int get_auth_pins() { return auth_pins; } 2023-10-30T13:52:20.735 INFO:tasks.workunit.client.0.smithi148.stdout:- int get_nested_auth_pins() { return nested_auth_pins; } 2023-10-30T13:52:20.736 INFO:tasks.workunit.client.0.smithi148.stdout:- void auth_pin(); 2023-10-30T13:52:20.736 INFO:tasks.workunit.client.0.smithi148.stdout:- void auth_unpin(); 2023-10-30T13:52:20.736 INFO:tasks.workunit.client.0.smithi148.stdout:- void adjust_nested_auth_pins(int inc); 2023-10-30T13:52:20.736 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.736 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- freezing -- 2023-10-30T13:52:20.737 INFO:tasks.workunit.client.0.smithi148.stdout:- bool freeze_tree(); 2023-10-30T13:52:20.737 INFO:tasks.workunit.client.0.smithi148.stdout:- void _freeze_tree(); 2023-10-30T13:52:20.737 INFO:tasks.workunit.client.0.smithi148.stdout:- void unfreeze_tree(); 2023-10-30T13:52:20.737 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.737 INFO:tasks.workunit.client.0.smithi148.stdout:- bool freeze_dir(); 2023-10-30T13:52:20.737 INFO:tasks.workunit.client.0.smithi148.stdout:- void _freeze_dir(); 2023-10-30T13:52:20.738 INFO:tasks.workunit.client.0.smithi148.stdout:- void unfreeze_dir(); 2023-10-30T13:52:20.738 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.738 INFO:tasks.workunit.client.0.smithi148.stdout:- void maybe_finish_freeze() { 2023-10-30T13:52:20.738 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth_pins != 1 || nested_auth_pins != 0) 2023-10-30T13:52:20.738 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.739 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_FREEZINGTREE)) { 2023-10-30T13:52:20.739 INFO:tasks.workunit.client.0.smithi148.stdout:- _freeze_tree(); 2023-10-30T13:52:20.739 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_unpin(); 2023-10-30T13:52:20.739 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_FROZEN); 2023-10-30T13:52:20.739 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.740 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_FREEZINGDIR)) { 2023-10-30T13:52:20.740 INFO:tasks.workunit.client.0.smithi148.stdout:- _freeze_dir(); 2023-10-30T13:52:20.740 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_unpin(); 2023-10-30T13:52:20.740 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_FROZEN); 2023-10-30T13:52:20.740 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.740 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.741 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.741 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_freezing() { return is_freezing_tree() || is_freezing_dir(); } 2023-10-30T13:52:20.741 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_freezing_tree(); 2023-10-30T13:52:20.741 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_freezing_tree_root() { return state & STATE_FREEZINGTREE; } 2023-10-30T13:52:20.741 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_freezing_dir() { return state & STATE_FREEZINGDIR; } 2023-10-30T13:52:20.742 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.742 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_frozen() { return is_frozen_dir() || is_frozen_tree(); } 2023-10-30T13:52:20.742 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_frozen_tree(); 2023-10-30T13:52:20.742 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_frozen_tree_root() { return state & STATE_FROZENTREE; } 2023-10-30T13:52:20.742 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_frozen_dir() { return state & STATE_FROZENDIR; } 2023-10-30T13:52:20.742 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.743 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_freezeable(bool freezing=false) { 2023-10-30T13:52:20.743 INFO:tasks.workunit.client.0.smithi148.stdout:- // no nested auth pins. 2023-10-30T13:52:20.743 INFO:tasks.workunit.client.0.smithi148.stdout:- if ((auth_pins-freezing) > 0 || nested_auth_pins > 0) 2023-10-30T13:52:20.743 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.743 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.744 INFO:tasks.workunit.client.0.smithi148.stdout:- // inode must not be frozen. 2023-10-30T13:52:20.744 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!is_subtree_root() && inode->is_frozen()) 2023-10-30T13:52:20.744 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.744 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.744 INFO:tasks.workunit.client.0.smithi148.stdout:- return true; 2023-10-30T13:52:20.744 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.745 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_freezeable_dir(bool freezing=false) { 2023-10-30T13:52:20.745 INFO:tasks.workunit.client.0.smithi148.stdout:- if ((auth_pins-freezing) > 0) 2023-10-30T13:52:20.745 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.745 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.745 INFO:tasks.workunit.client.0.smithi148.stdout:- // if not subtree root, inode must not be frozen (tree--frozen_dir is okay). 2023-10-30T13:52:20.746 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!is_subtree_root() && inode->is_frozen() && !inode->is_frozen_dir()) 2023-10-30T13:52:20.746 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.746 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.746 INFO:tasks.workunit.client.0.smithi148.stdout:- return true; 2023-10-30T13:52:20.746 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.747 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.747 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *get_frozen_tree_root(); 2023-10-30T13:52:20.747 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.747 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.747 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.747 INFO:tasks.workunit.client.0.smithi148.stdout:- ostream& print_db_line_prefix(ostream& out); 2023-10-30T13:52:20.748 INFO:tasks.workunit.client.0.smithi148.stdout:- void print(ostream& out); 2023-10-30T13:52:20.748 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:20.748 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.748 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.748 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.749 INFO:tasks.workunit.client.0.smithi148.stdout:-// -- encoded state -- 2023-10-30T13:52:20.749 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.749 INFO:tasks.workunit.client.0.smithi148.stdout:-// discover 2023-10-30T13:52:20.749 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.749 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDirDiscover { 2023-10-30T13:52:20.750 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrag_t dirfrag; 2023-10-30T13:52:20.750 INFO:tasks.workunit.client.0.smithi148.stdout:- int nonce; 2023-10-30T13:52:20.750 INFO:tasks.workunit.client.0.smithi148.stdout:- int dir_rep; 2023-10-30T13:52:20.750 INFO:tasks.workunit.client.0.smithi148.stdout:- set rep_by; 2023-10-30T13:52:20.750 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.751 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.751 INFO:tasks.workunit.client.0.smithi148.stdout:- CDirDiscover() {} 2023-10-30T13:52:20.751 INFO:tasks.workunit.client.0.smithi148.stdout:- CDirDiscover(CDir *dir, int nonce) { 2023-10-30T13:52:20.751 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrag = dir->dirfrag(); 2023-10-30T13:52:20.751 INFO:tasks.workunit.client.0.smithi148.stdout:- this->nonce = nonce; 2023-10-30T13:52:20.752 INFO:tasks.workunit.client.0.smithi148.stdout:- dir_rep = dir->dir_rep; 2023-10-30T13:52:20.752 INFO:tasks.workunit.client.0.smithi148.stdout:- rep_by = dir->dir_rep_by; 2023-10-30T13:52:20.752 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.752 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.752 INFO:tasks.workunit.client.0.smithi148.stdout:- void update_dir(CDir *dir) { 2023-10-30T13:52:20.752 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dir->dirfrag() == dirfrag); 2023-10-30T13:52:20.753 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!dir->is_auth()); 2023-10-30T13:52:20.753 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.753 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->replica_nonce = nonce; 2023-10-30T13:52:20.753 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->dir_rep = dir_rep; 2023-10-30T13:52:20.753 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->dir_rep_by = rep_by; 2023-10-30T13:52:20.754 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.754 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.754 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrag_t get_dirfrag() { return dirfrag; } 2023-10-30T13:52:20.754 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.754 INFO:tasks.workunit.client.0.smithi148.stdout:- void _encode(bufferlist& bl) { 2023-10-30T13:52:20.755 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append((char*)&dirfrag, sizeof(dirfrag)); 2023-10-30T13:52:20.755 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append((char*)&nonce, sizeof(nonce)); 2023-10-30T13:52:20.755 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append((char*)&dir_rep, sizeof(dir_rep)); 2023-10-30T13:52:20.755 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(rep_by, bl); 2023-10-30T13:52:20.755 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.755 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.756 INFO:tasks.workunit.client.0.smithi148.stdout:- void _decode(bufferlist& bl, int& off) { 2023-10-30T13:52:20.756 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.copy(off, sizeof(dirfrag), (char*)&dirfrag); 2023-10-30T13:52:20.756 INFO:tasks.workunit.client.0.smithi148.stdout:- off += sizeof(dirfrag); 2023-10-30T13:52:20.756 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.copy(off, sizeof(nonce), (char*)&nonce); 2023-10-30T13:52:20.756 INFO:tasks.workunit.client.0.smithi148.stdout:- off += sizeof(nonce); 2023-10-30T13:52:20.757 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.copy(off, sizeof(dir_rep), (char*)&dir_rep); 2023-10-30T13:52:20.757 INFO:tasks.workunit.client.0.smithi148.stdout:- off += sizeof(dir_rep); 2023-10-30T13:52:20.757 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(rep_by, bl, off); 2023-10-30T13:52:20.757 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.757 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.758 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:20.758 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.758 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.758 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.758 INFO:tasks.workunit.client.0.smithi148.stdout:-#endif 2023-10-30T13:52:20.758 INFO:tasks.workunit.client.0.smithi148.stdout:diff --git a/src/mds/CInode.cc b/src/mds/CInode.cc 2023-10-30T13:52:20.759 INFO:tasks.workunit.client.0.smithi148.stdout:index d65a0a2c219..e69de29bb2d 100644 2023-10-30T13:52:20.759 INFO:tasks.workunit.client.0.smithi148.stdout:--- a/src/mds/CInode.cc 2023-10-30T13:52:20.759 INFO:tasks.workunit.client.0.smithi148.stdout:+++ b/src/mds/CInode.cc 2023-10-30T13:52:20.759 INFO:tasks.workunit.client.0.smithi148.stdout:@@ -1,844 +0,0 @@ 2023-10-30T13:52:20.759 INFO:tasks.workunit.client.0.smithi148.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2023-10-30T13:52:20.760 INFO:tasks.workunit.client.0.smithi148.stdout:-// vim: ts=8 sw=2 smarttab 2023-10-30T13:52:20.760 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.760 INFO:tasks.workunit.client.0.smithi148.stdout:- * Ceph - scalable distributed file system 2023-10-30T13:52:20.760 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.760 INFO:tasks.workunit.client.0.smithi148.stdout:- * Copyright (C) 2004-2006 Sage Weil 2023-10-30T13:52:20.761 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.761 INFO:tasks.workunit.client.0.smithi148.stdout:- * This is free software; you can redistribute it and/or 2023-10-30T13:52:20.761 INFO:tasks.workunit.client.0.smithi148.stdout:- * modify it under the terms of the GNU Lesser General Public 2023-10-30T13:52:20.761 INFO:tasks.workunit.client.0.smithi148.stdout:- * License version 2.1, as published by the Free Software 2023-10-30T13:52:20.761 INFO:tasks.workunit.client.0.smithi148.stdout:- * Foundation. See file COPYING. 2023-10-30T13:52:20.762 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.762 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.762 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.762 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.762 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.763 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CInode.h" 2023-10-30T13:52:20.763 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CDir.h" 2023-10-30T13:52:20.763 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CDentry.h" 2023-10-30T13:52:20.763 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.763 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "MDS.h" 2023-10-30T13:52:20.764 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "MDCache.h" 2023-10-30T13:52:20.764 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "AnchorTable.h" 2023-10-30T13:52:20.764 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.764 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "LogSegment.h" 2023-10-30T13:52:20.764 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.764 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "common/Clock.h" 2023-10-30T13:52:20.765 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.765 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "messages/MLock.h" 2023-10-30T13:52:20.765 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.765 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.765 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.766 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.766 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "config.h" 2023-10-30T13:52:20.766 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.766 INFO:tasks.workunit.client.0.smithi148.stdout:-#define dout(x) if (x <= g_conf.debug || x <= g_conf.debug_mds) *_dout << dbeginl << g_clock.now() << " mds" << mdcache->mds->get_nodeid() << ".cache.ino(" << inode.ino << ") " 2023-10-30T13:52:20.766 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.767 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.767 INFO:tasks.workunit.client.0.smithi148.stdout:-//int cinode_pins[CINODE_NUM_PINS]; // counts 2023-10-30T13:52:20.767 INFO:tasks.workunit.client.0.smithi148.stdout:-ostream& CInode::print_db_line_prefix(ostream& out) 2023-10-30T13:52:20.767 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.767 INFO:tasks.workunit.client.0.smithi148.stdout:- return out << g_clock.now() << " mds" << mdcache->mds->get_nodeid() << ".cache.ino(" << inode.ino << ") "; 2023-10-30T13:52:20.768 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.768 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.768 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.768 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.768 INFO:tasks.workunit.client.0.smithi148.stdout:-ostream& operator<<(ostream& out, CInode& in) 2023-10-30T13:52:20.769 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.769 INFO:tasks.workunit.client.0.smithi148.stdout:- filepath path; 2023-10-30T13:52:20.769 INFO:tasks.workunit.client.0.smithi148.stdout:- in.make_path(path); 2023-10-30T13:52:20.769 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "[inode " << in.inode.ino << " " << path << (in.is_dir() ? "/ ":" "); 2023-10-30T13:52:20.769 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in.is_auth()) { 2023-10-30T13:52:20.770 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "auth"; 2023-10-30T13:52:20.770 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in.is_replicated()) 2023-10-30T13:52:20.770 INFO:tasks.workunit.client.0.smithi148.stdout:- out << in.get_replicas(); 2023-10-30T13:52:20.770 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.770 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "rep@" << in.authority(); 2023-10-30T13:52:20.771 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "." << in.get_replica_nonce(); 2023-10-30T13:52:20.771 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(in.get_replica_nonce() >= 0); 2023-10-30T13:52:20.771 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.771 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.771 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in.is_symlink()) out << " symlink='" << in.symlink << "'"; 2023-10-30T13:52:20.772 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in.is_dir() && !in.dirfragtree.empty()) out << " " << in.dirfragtree; 2023-10-30T13:52:20.772 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.772 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " v" << in.get_version(); 2023-10-30T13:52:20.772 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.772 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in.state_test(CInode::STATE_AMBIGUOUSAUTH)) out << " AMBIGAUTH"; 2023-10-30T13:52:20.773 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in.is_freezing_inode()) out << " FREEZING=" << in.auth_pin_freeze_allowance; 2023-10-30T13:52:20.773 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in.is_frozen_inode()) out << " FROZEN"; 2023-10-30T13:52:20.773 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.773 INFO:tasks.workunit.client.0.smithi148.stdout:- // locks 2023-10-30T13:52:20.773 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " " << in.authlock; 2023-10-30T13:52:20.774 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " " << in.linklock; 2023-10-30T13:52:20.774 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " " << in.dirfragtreelock; 2023-10-30T13:52:20.774 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " " << in.filelock; 2023-10-30T13:52:20.774 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " " << in.dirlock; 2023-10-30T13:52:20.774 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.774 INFO:tasks.workunit.client.0.smithi148.stdout:- // hack: spit out crap on which clients have caps 2023-10-30T13:52:20.775 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!in.get_client_caps().empty()) { 2023-10-30T13:52:20.775 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " caps={"; 2023-10-30T13:52:20.775 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator it = in.get_client_caps().begin(); 2023-10-30T13:52:20.775 INFO:tasks.workunit.client.0.smithi148.stdout:- it != in.get_client_caps().end(); 2023-10-30T13:52:20.775 INFO:tasks.workunit.client.0.smithi148.stdout:- it++) { 2023-10-30T13:52:20.776 INFO:tasks.workunit.client.0.smithi148.stdout:- if (it != in.get_client_caps().begin()) out << ","; 2023-10-30T13:52:20.776 INFO:tasks.workunit.client.0.smithi148.stdout:- out << it->first << "=" << it->second->issued(); 2023-10-30T13:52:20.776 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.776 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "}"; 2023-10-30T13:52:20.776 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.777 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.777 INFO:tasks.workunit.client.0.smithi148.stdout:- if (in.get_num_ref()) { 2023-10-30T13:52:20.777 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " |"; 2023-10-30T13:52:20.777 INFO:tasks.workunit.client.0.smithi148.stdout:- in.print_pin_set(out); 2023-10-30T13:52:20.777 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.777 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.778 INFO:tasks.workunit.client.0.smithi148.stdout:- out << " " << ∈ 2023-10-30T13:52:20.778 INFO:tasks.workunit.client.0.smithi148.stdout:- out << "]"; 2023-10-30T13:52:20.778 INFO:tasks.workunit.client.0.smithi148.stdout:- return out; 2023-10-30T13:52:20.778 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.778 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.779 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.779 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::print(ostream& out) 2023-10-30T13:52:20.779 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.779 INFO:tasks.workunit.client.0.smithi148.stdout:- out << *this; 2023-10-30T13:52:20.779 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.779 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.780 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.780 INFO:tasks.workunit.client.0.smithi148.stdout:-inode_t *CInode::project_inode() 2023-10-30T13:52:20.780 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.780 INFO:tasks.workunit.client.0.smithi148.stdout:- if (projected_inode.empty()) { 2023-10-30T13:52:20.780 INFO:tasks.workunit.client.0.smithi148.stdout:- projected_inode.push_back(new inode_t(inode)); 2023-10-30T13:52:20.781 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.781 INFO:tasks.workunit.client.0.smithi148.stdout:- projected_inode.push_back(new inode_t(*projected_inode.back())); 2023-10-30T13:52:20.781 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.781 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << "project_inode " << projected_inode.back() << dendl; 2023-10-30T13:52:20.781 INFO:tasks.workunit.client.0.smithi148.stdout:- return projected_inode.back(); 2023-10-30T13:52:20.781 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.782 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.782 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::pop_and_dirty_projected_inode(LogSegment *ls) 2023-10-30T13:52:20.782 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.782 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!projected_inode.empty()); 2023-10-30T13:52:20.783 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << "pop_and_dirty_projected_inode " << projected_inode.front() 2023-10-30T13:52:20.783 INFO:tasks.workunit.client.0.smithi148.stdout:- << " v" << projected_inode.front()->version << dendl; 2023-10-30T13:52:20.783 INFO:tasks.workunit.client.0.smithi148.stdout:- mark_dirty(projected_inode.front()->version, ls); 2023-10-30T13:52:20.783 INFO:tasks.workunit.client.0.smithi148.stdout:- inode = *projected_inode.front(); 2023-10-30T13:52:20.783 INFO:tasks.workunit.client.0.smithi148.stdout:- delete projected_inode.front(); 2023-10-30T13:52:20.783 INFO:tasks.workunit.client.0.smithi148.stdout:- projected_inode.pop_front(); 2023-10-30T13:52:20.784 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.784 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.784 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.784 INFO:tasks.workunit.client.0.smithi148.stdout:-// ====== CInode ======= 2023-10-30T13:52:20.784 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.785 INFO:tasks.workunit.client.0.smithi148.stdout:-// dirfrags 2023-10-30T13:52:20.785 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.785 INFO:tasks.workunit.client.0.smithi148.stdout:-frag_t CInode::pick_dirfrag(const string& dn) 2023-10-30T13:52:20.785 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.785 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dirfragtree.empty()) 2023-10-30T13:52:20.785 INFO:tasks.workunit.client.0.smithi148.stdout:- return frag_t(); // avoid the string hash if we can. 2023-10-30T13:52:20.786 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.786 INFO:tasks.workunit.client.0.smithi148.stdout:- static hash H; 2023-10-30T13:52:20.786 INFO:tasks.workunit.client.0.smithi148.stdout:- return dirfragtree[H(dn)]; 2023-10-30T13:52:20.786 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.786 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.787 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::get_dirfrags_under(frag_t fg, list& ls) 2023-10-30T13:52:20.787 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.787 INFO:tasks.workunit.client.0.smithi148.stdout:- list fglist; 2023-10-30T13:52:20.787 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtree.get_leaves_under(fg, fglist); 2023-10-30T13:52:20.787 INFO:tasks.workunit.client.0.smithi148.stdout:- for (list::iterator p = fglist.begin(); 2023-10-30T13:52:20.788 INFO:tasks.workunit.client.0.smithi148.stdout:- p != fglist.end(); 2023-10-30T13:52:20.788 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.788 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dirfrags.count(*p)) 2023-10-30T13:52:20.788 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.push_back(dirfrags[*p]); 2023-10-30T13:52:20.788 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.789 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.789 INFO:tasks.workunit.client.0.smithi148.stdout:-CDir *CInode::get_approx_dirfrag(frag_t fg) 2023-10-30T13:52:20.789 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.789 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir = get_dirfrag(fg); 2023-10-30T13:52:20.789 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir) return dir; 2023-10-30T13:52:20.789 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.790 INFO:tasks.workunit.client.0.smithi148.stdout:- // find a child? 2023-10-30T13:52:20.790 INFO:tasks.workunit.client.0.smithi148.stdout:- list ls; 2023-10-30T13:52:20.790 INFO:tasks.workunit.client.0.smithi148.stdout:- get_dirfrags_under(fg, ls); 2023-10-30T13:52:20.790 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!ls.empty()) 2023-10-30T13:52:20.790 INFO:tasks.workunit.client.0.smithi148.stdout:- return ls.front(); 2023-10-30T13:52:20.790 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.791 INFO:tasks.workunit.client.0.smithi148.stdout:- // try parents? 2023-10-30T13:52:20.791 INFO:tasks.workunit.client.0.smithi148.stdout:- while (1) { 2023-10-30T13:52:20.791 INFO:tasks.workunit.client.0.smithi148.stdout:- fg = fg.parent(); 2023-10-30T13:52:20.791 INFO:tasks.workunit.client.0.smithi148.stdout:- dir = get_dirfrag(fg); 2023-10-30T13:52:20.791 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir) return dir; 2023-10-30T13:52:20.792 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.792 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.792 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.792 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::get_dirfrags(list& ls) 2023-10-30T13:52:20.792 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.792 INFO:tasks.workunit.client.0.smithi148.stdout:- // all dirfrags 2023-10-30T13:52:20.793 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator p = dirfrags.begin(); 2023-10-30T13:52:20.793 INFO:tasks.workunit.client.0.smithi148.stdout:- p != dirfrags.end(); 2023-10-30T13:52:20.793 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.793 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.push_back(p->second); 2023-10-30T13:52:20.793 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.794 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::get_nested_dirfrags(list& ls) 2023-10-30T13:52:20.794 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.794 INFO:tasks.workunit.client.0.smithi148.stdout:- // dirfrags in same subtree 2023-10-30T13:52:20.794 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator p = dirfrags.begin(); 2023-10-30T13:52:20.794 INFO:tasks.workunit.client.0.smithi148.stdout:- p != dirfrags.end(); 2023-10-30T13:52:20.794 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.795 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!p->second->is_subtree_root()) 2023-10-30T13:52:20.795 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.push_back(p->second); 2023-10-30T13:52:20.795 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.795 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::get_subtree_dirfrags(list& ls) 2023-10-30T13:52:20.795 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.795 INFO:tasks.workunit.client.0.smithi148.stdout:- // dirfrags that are roots of new subtrees 2023-10-30T13:52:20.796 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator p = dirfrags.begin(); 2023-10-30T13:52:20.796 INFO:tasks.workunit.client.0.smithi148.stdout:- p != dirfrags.end(); 2023-10-30T13:52:20.796 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.796 INFO:tasks.workunit.client.0.smithi148.stdout:- if (p->second->is_subtree_root()) 2023-10-30T13:52:20.796 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.push_back(p->second); 2023-10-30T13:52:20.797 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.797 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.797 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.797 INFO:tasks.workunit.client.0.smithi148.stdout:-CDir *CInode::get_or_open_dirfrag(MDCache *mdcache, frag_t fg) 2023-10-30T13:52:20.797 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.797 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_dir()); 2023-10-30T13:52:20.798 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.798 INFO:tasks.workunit.client.0.smithi148.stdout:- // have it? 2023-10-30T13:52:20.798 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir = get_dirfrag(fg); 2023-10-30T13:52:20.798 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!dir) { 2023-10-30T13:52:20.798 INFO:tasks.workunit.client.0.smithi148.stdout:- // create it. 2023-10-30T13:52:20.799 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.799 INFO:tasks.workunit.client.0.smithi148.stdout:- dir = new CDir(this, fg, mdcache, true); 2023-10-30T13:52:20.799 INFO:tasks.workunit.client.0.smithi148.stdout:- add_dirfrag(dir); 2023-10-30T13:52:20.799 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.799 INFO:tasks.workunit.client.0.smithi148.stdout:- return dir; 2023-10-30T13:52:20.800 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.800 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.800 INFO:tasks.workunit.client.0.smithi148.stdout:-CDir *CInode::add_dirfrag(CDir *dir) 2023-10-30T13:52:20.800 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.800 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dirfrags.count(dir->dirfrag().frag) == 0); 2023-10-30T13:52:20.800 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrags[dir->dirfrag().frag] = dir; 2023-10-30T13:52:20.801 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.801 INFO:tasks.workunit.client.0.smithi148.stdout:- if (stickydir_ref > 0) { 2023-10-30T13:52:20.801 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->state_set(CDir::STATE_STICKY); 2023-10-30T13:52:20.801 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->get(CDir::PIN_STICKY); 2023-10-30T13:52:20.801 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.801 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.802 INFO:tasks.workunit.client.0.smithi148.stdout:- return dir; 2023-10-30T13:52:20.802 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.802 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.802 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::close_dirfrag(frag_t fg) 2023-10-30T13:52:20.802 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.803 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(14) << "close_dirfrag " << fg << dendl; 2023-10-30T13:52:20.803 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dirfrags.count(fg)); 2023-10-30T13:52:20.803 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.803 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *dir = dirfrags[fg]; 2023-10-30T13:52:20.803 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->remove_null_dentries(); 2023-10-30T13:52:20.803 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.804 INFO:tasks.workunit.client.0.smithi148.stdout:- // clear dirty flag 2023-10-30T13:52:20.804 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dir->is_dirty()) 2023-10-30T13:52:20.804 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->mark_clean(); 2023-10-30T13:52:20.804 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.804 INFO:tasks.workunit.client.0.smithi148.stdout:- if (stickydir_ref > 0) { 2023-10-30T13:52:20.804 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->state_clear(CDir::STATE_STICKY); 2023-10-30T13:52:20.805 INFO:tasks.workunit.client.0.smithi148.stdout:- dir->put(CDir::PIN_STICKY); 2023-10-30T13:52:20.805 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.805 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.805 INFO:tasks.workunit.client.0.smithi148.stdout:- // dump any remaining dentries, for debugging purposes 2023-10-30T13:52:20.805 INFO:tasks.workunit.client.0.smithi148.stdout:- for (CDir::map_t::iterator p = dir->items.begin(); 2023-10-30T13:52:20.806 INFO:tasks.workunit.client.0.smithi148.stdout:- p != dir->items.end(); 2023-10-30T13:52:20.806 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.806 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(14) << "close_dirfrag LEFTOVER dn " << *p->second << dendl; 2023-10-30T13:52:20.806 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.806 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dir->get_num_ref() == 0); 2023-10-30T13:52:20.807 INFO:tasks.workunit.client.0.smithi148.stdout:- delete dir; 2023-10-30T13:52:20.807 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfrags.erase(fg); 2023-10-30T13:52:20.807 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.807 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.807 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::close_dirfrags() 2023-10-30T13:52:20.808 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.808 INFO:tasks.workunit.client.0.smithi148.stdout:- while (!dirfrags.empty()) 2023-10-30T13:52:20.808 INFO:tasks.workunit.client.0.smithi148.stdout:- close_dirfrag(dirfrags.begin()->first); 2023-10-30T13:52:20.808 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.808 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.809 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CInode::has_subtree_root_dirfrag() 2023-10-30T13:52:20.809 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.809 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator p = dirfrags.begin(); 2023-10-30T13:52:20.809 INFO:tasks.workunit.client.0.smithi148.stdout:- p != dirfrags.end(); 2023-10-30T13:52:20.809 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.809 INFO:tasks.workunit.client.0.smithi148.stdout:- if (p->second->is_subtree_root()) 2023-10-30T13:52:20.810 INFO:tasks.workunit.client.0.smithi148.stdout:- return true; 2023-10-30T13:52:20.810 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.810 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.810 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.810 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.811 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::get_stickydirs() 2023-10-30T13:52:20.811 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.811 INFO:tasks.workunit.client.0.smithi148.stdout:- if (stickydir_ref == 0) { 2023-10-30T13:52:20.811 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_STICKYDIRS); 2023-10-30T13:52:20.811 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator p = dirfrags.begin(); 2023-10-30T13:52:20.811 INFO:tasks.workunit.client.0.smithi148.stdout:- p != dirfrags.end(); 2023-10-30T13:52:20.812 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) { 2023-10-30T13:52:20.812 INFO:tasks.workunit.client.0.smithi148.stdout:- p->second->state_set(CDir::STATE_STICKY); 2023-10-30T13:52:20.812 INFO:tasks.workunit.client.0.smithi148.stdout:- p->second->get(CDir::PIN_STICKY); 2023-10-30T13:52:20.812 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.812 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.812 INFO:tasks.workunit.client.0.smithi148.stdout:- stickydir_ref++; 2023-10-30T13:52:20.813 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.813 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.813 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::put_stickydirs() 2023-10-30T13:52:20.813 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.813 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(stickydir_ref > 0); 2023-10-30T13:52:20.813 INFO:tasks.workunit.client.0.smithi148.stdout:- stickydir_ref--; 2023-10-30T13:52:20.814 INFO:tasks.workunit.client.0.smithi148.stdout:- if (stickydir_ref == 0) { 2023-10-30T13:52:20.814 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_STICKYDIRS); 2023-10-30T13:52:20.814 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator p = dirfrags.begin(); 2023-10-30T13:52:20.814 INFO:tasks.workunit.client.0.smithi148.stdout:- p != dirfrags.end(); 2023-10-30T13:52:20.814 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) { 2023-10-30T13:52:20.815 INFO:tasks.workunit.client.0.smithi148.stdout:- p->second->state_clear(CDir::STATE_STICKY); 2023-10-30T13:52:20.815 INFO:tasks.workunit.client.0.smithi148.stdout:- p->second->put(CDir::PIN_STICKY); 2023-10-30T13:52:20.815 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.815 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.815 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.815 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.816 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.816 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.816 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.816 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.816 INFO:tasks.workunit.client.0.smithi148.stdout:-// pins 2023-10-30T13:52:20.816 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.817 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::first_get() 2023-10-30T13:52:20.817 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.817 INFO:tasks.workunit.client.0.smithi148.stdout:- // pin my dentry? 2023-10-30T13:52:20.817 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) 2023-10-30T13:52:20.817 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->get(CDentry::PIN_INODEPIN); 2023-10-30T13:52:20.817 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.818 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.818 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::last_put() 2023-10-30T13:52:20.818 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.818 INFO:tasks.workunit.client.0.smithi148.stdout:- // unpin my dentry? 2023-10-30T13:52:20.818 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) 2023-10-30T13:52:20.819 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->put(CDentry::PIN_INODEPIN); 2023-10-30T13:52:20.819 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.819 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.819 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::add_remote_parent(CDentry *p) 2023-10-30T13:52:20.819 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.819 INFO:tasks.workunit.client.0.smithi148.stdout:- if (remote_parents.empty()) 2023-10-30T13:52:20.820 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_REMOTEPARENT); 2023-10-30T13:52:20.820 INFO:tasks.workunit.client.0.smithi148.stdout:- remote_parents.insert(p); 2023-10-30T13:52:20.820 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.820 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::remove_remote_parent(CDentry *p) 2023-10-30T13:52:20.820 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.820 INFO:tasks.workunit.client.0.smithi148.stdout:- remote_parents.erase(p); 2023-10-30T13:52:20.821 INFO:tasks.workunit.client.0.smithi148.stdout:- if (remote_parents.empty()) 2023-10-30T13:52:20.821 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_REMOTEPARENT); 2023-10-30T13:52:20.821 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.821 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.821 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.821 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.822 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.822 INFO:tasks.workunit.client.0.smithi148.stdout:-CDir *CInode::get_parent_dir() 2023-10-30T13:52:20.822 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.822 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) 2023-10-30T13:52:20.822 INFO:tasks.workunit.client.0.smithi148.stdout:- return parent->dir; 2023-10-30T13:52:20.822 INFO:tasks.workunit.client.0.smithi148.stdout:- return NULL; 2023-10-30T13:52:20.823 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.823 INFO:tasks.workunit.client.0.smithi148.stdout:-CInode *CInode::get_parent_inode() 2023-10-30T13:52:20.823 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.823 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) 2023-10-30T13:52:20.823 INFO:tasks.workunit.client.0.smithi148.stdout:- return parent->dir->inode; 2023-10-30T13:52:20.823 INFO:tasks.workunit.client.0.smithi148.stdout:- return NULL; 2023-10-30T13:52:20.824 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.824 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.824 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.824 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.824 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::make_path_string(string& s) 2023-10-30T13:52:20.824 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.825 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) { 2023-10-30T13:52:20.825 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->make_path_string(s); 2023-10-30T13:52:20.825 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.825 INFO:tasks.workunit.client.0.smithi148.stdout:- else if (is_root()) { 2023-10-30T13:52:20.825 INFO:tasks.workunit.client.0.smithi148.stdout:- s = ""; // root 2023-10-30T13:52:20.825 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.826 INFO:tasks.workunit.client.0.smithi148.stdout:- else if (is_stray()) { 2023-10-30T13:52:20.826 INFO:tasks.workunit.client.0.smithi148.stdout:- s = "~stray"; 2023-10-30T13:52:20.826 INFO:tasks.workunit.client.0.smithi148.stdout:- char n[10]; 2023-10-30T13:52:20.826 INFO:tasks.workunit.client.0.smithi148.stdout:- sprintf(n, "%d", (int)(ino()-MDS_INO_STRAY_OFFSET)); 2023-10-30T13:52:20.826 INFO:tasks.workunit.client.0.smithi148.stdout:- s += n; 2023-10-30T13:52:20.826 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.827 INFO:tasks.workunit.client.0.smithi148.stdout:- else { 2023-10-30T13:52:20.827 INFO:tasks.workunit.client.0.smithi148.stdout:- s = "(dangling)"; // dangling 2023-10-30T13:52:20.827 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.827 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.827 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.827 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::make_path(filepath& fp) 2023-10-30T13:52:20.828 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.828 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) 2023-10-30T13:52:20.828 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->make_path(fp); 2023-10-30T13:52:20.828 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.828 INFO:tasks.workunit.client.0.smithi148.stdout:- fp.set_ino(ino()); 2023-10-30T13:52:20.828 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.829 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.829 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::make_anchor_trace(vector& trace) 2023-10-30T13:52:20.829 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.829 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) { 2023-10-30T13:52:20.829 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->dir->inode->make_anchor_trace(trace); 2023-10-30T13:52:20.830 INFO:tasks.workunit.client.0.smithi148.stdout:- trace.push_back(Anchor(ino(), parent->dir->dirfrag())); 2023-10-30T13:52:20.830 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "make_anchor_trace added " << trace.back() << dendl; 2023-10-30T13:52:20.830 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.830 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.830 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_root() || is_stray()); 2023-10-30T13:52:20.830 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.831 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.831 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::name_stray_dentry(string& dname) 2023-10-30T13:52:20.831 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.831 INFO:tasks.workunit.client.0.smithi148.stdout:- char s[20]; 2023-10-30T13:52:20.831 INFO:tasks.workunit.client.0.smithi148.stdout:-#ifdef __LP64__ 2023-10-30T13:52:20.831 INFO:tasks.workunit.client.0.smithi148.stdout:- sprintf(s, "%lx", inode.ino.val); 2023-10-30T13:52:20.832 INFO:tasks.workunit.client.0.smithi148.stdout:-#else 2023-10-30T13:52:20.832 INFO:tasks.workunit.client.0.smithi148.stdout:- sprintf(s, "%llx", inode.ino.val); 2023-10-30T13:52:20.832 INFO:tasks.workunit.client.0.smithi148.stdout:-#endif 2023-10-30T13:52:20.832 INFO:tasks.workunit.client.0.smithi148.stdout:- dname = s; 2023-10-30T13:52:20.832 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.832 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.833 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.833 INFO:tasks.workunit.client.0.smithi148.stdout:-version_t CInode::pre_dirty() 2023-10-30T13:52:20.833 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.833 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(parent); 2023-10-30T13:52:20.833 INFO:tasks.workunit.client.0.smithi148.stdout:- return parent->pre_dirty(); 2023-10-30T13:52:20.833 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.834 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.834 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::_mark_dirty(LogSegment *ls) 2023-10-30T13:52:20.834 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.834 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!state_test(STATE_DIRTY)) { 2023-10-30T13:52:20.834 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_DIRTY); 2023-10-30T13:52:20.834 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_DIRTY); 2023-10-30T13:52:20.835 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(ls); 2023-10-30T13:52:20.835 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.835 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.835 INFO:tasks.workunit.client.0.smithi148.stdout:- // move myself to this segment's dirty list 2023-10-30T13:52:20.835 INFO:tasks.workunit.client.0.smithi148.stdout:- if (ls) 2023-10-30T13:52:20.835 INFO:tasks.workunit.client.0.smithi148.stdout:- ls->dirty_inodes.push_back(&xlist_dirty); 2023-10-30T13:52:20.836 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.836 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.836 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::mark_dirty(version_t pv, LogSegment *ls) { 2023-10-30T13:52:20.836 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.836 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "mark_dirty " << *this << dendl; 2023-10-30T13:52:20.837 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.837 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(parent); 2023-10-30T13:52:20.837 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.837 INFO:tasks.workunit.client.0.smithi148.stdout:- /* 2023-10-30T13:52:20.837 INFO:tasks.workunit.client.0.smithi148.stdout:- NOTE: I may already be dirty, but this fn _still_ needs to be called so that 2023-10-30T13:52:20.837 INFO:tasks.workunit.client.0.smithi148.stdout:- the directory is (perhaps newly) dirtied, and so that parent_dir_version is 2023-10-30T13:52:20.838 INFO:tasks.workunit.client.0.smithi148.stdout:- updated below. 2023-10-30T13:52:20.838 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.838 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.838 INFO:tasks.workunit.client.0.smithi148.stdout:- // only auth can get dirty. "dirty" async data in replicas is relative to 2023-10-30T13:52:20.838 INFO:tasks.workunit.client.0.smithi148.stdout:- // filelock state, not the dirty flag. 2023-10-30T13:52:20.839 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.839 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.839 INFO:tasks.workunit.client.0.smithi148.stdout:- // touch my private version 2023-10-30T13:52:20.839 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(inode.version < pv); 2023-10-30T13:52:20.839 INFO:tasks.workunit.client.0.smithi148.stdout:- inode.version = pv; 2023-10-30T13:52:20.840 INFO:tasks.workunit.client.0.smithi148.stdout:- _mark_dirty(ls); 2023-10-30T13:52:20.840 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.840 INFO:tasks.workunit.client.0.smithi148.stdout:- // mark dentry too 2023-10-30T13:52:20.840 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->mark_dirty(pv, ls); 2023-10-30T13:52:20.840 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.840 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.841 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.841 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::mark_clean() 2023-10-30T13:52:20.841 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.841 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << " mark_clean " << *this << dendl; 2023-10-30T13:52:20.841 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_DIRTY)) { 2023-10-30T13:52:20.842 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_DIRTY); 2023-10-30T13:52:20.842 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_DIRTY); 2023-10-30T13:52:20.842 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.842 INFO:tasks.workunit.client.0.smithi148.stdout:- // remove myself from ls dirty list 2023-10-30T13:52:20.842 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty.remove_myself(); 2023-10-30T13:52:20.842 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.843 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.843 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.843 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.843 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.843 INFO:tasks.workunit.client.0.smithi148.stdout:-// ------------------ 2023-10-30T13:52:20.843 INFO:tasks.workunit.client.0.smithi148.stdout:-// locking 2023-10-30T13:52:20.844 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.844 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::set_object_info(MDSCacheObjectInfo &info) 2023-10-30T13:52:20.844 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.844 INFO:tasks.workunit.client.0.smithi148.stdout:- info.ino = ino(); 2023-10-30T13:52:20.844 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.844 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.845 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::encode_lock_state(int type, bufferlist& bl) 2023-10-30T13:52:20.845 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.845 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (type) { 2023-10-30T13:52:20.845 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IAUTH: 2023-10-30T13:52:20.845 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.ctime, bl); 2023-10-30T13:52:20.845 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.mode, bl); 2023-10-30T13:52:20.846 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.uid, bl); 2023-10-30T13:52:20.846 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.gid, bl); 2023-10-30T13:52:20.846 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.846 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.846 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_ILINK: 2023-10-30T13:52:20.847 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.ctime, bl); 2023-10-30T13:52:20.847 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.nlink, bl); 2023-10-30T13:52:20.847 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.anchored, bl); 2023-10-30T13:52:20.847 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.847 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.847 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IDIRFRAGTREE: 2023-10-30T13:52:20.848 INFO:tasks.workunit.client.0.smithi148.stdout:- { 2023-10-30T13:52:20.848 INFO:tasks.workunit.client.0.smithi148.stdout:- // encode the raw tree 2023-10-30T13:52:20.848 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtree._encode(bl); 2023-10-30T13:52:20.848 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.848 INFO:tasks.workunit.client.0.smithi148.stdout:- // also specify which frags are mine 2023-10-30T13:52:20.848 INFO:tasks.workunit.client.0.smithi148.stdout:- set myfrags; 2023-10-30T13:52:20.849 INFO:tasks.workunit.client.0.smithi148.stdout:- list dfls; 2023-10-30T13:52:20.849 INFO:tasks.workunit.client.0.smithi148.stdout:- get_dirfrags(dfls); 2023-10-30T13:52:20.849 INFO:tasks.workunit.client.0.smithi148.stdout:- for (list::iterator p = dfls.begin(); p != dfls.end(); ++p) 2023-10-30T13:52:20.849 INFO:tasks.workunit.client.0.smithi148.stdout:- if ((*p)->is_auth()) { 2023-10-30T13:52:20.849 INFO:tasks.workunit.client.0.smithi148.stdout:- frag_t fg = (*p)->get_frag(); 2023-10-30T13:52:20.849 INFO:tasks.workunit.client.0.smithi148.stdout:- myfrags.insert(fg); 2023-10-30T13:52:20.850 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.850 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(myfrags, bl); 2023-10-30T13:52:20.850 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.850 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.850 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.851 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IFILE: 2023-10-30T13:52:20.851 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.size, bl); 2023-10-30T13:52:20.851 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.mtime, bl); 2023-10-30T13:52:20.851 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.atime, bl); 2023-10-30T13:52:20.851 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.851 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.852 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IDIR: 2023-10-30T13:52:20.852 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(inode.mtime, bl); 2023-10-30T13:52:20.852 INFO:tasks.workunit.client.0.smithi148.stdout:- if (0) { 2023-10-30T13:52:20.852 INFO:tasks.workunit.client.0.smithi148.stdout:- map frag_sizes; 2023-10-30T13:52:20.852 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator p = dirfrags.begin(); 2023-10-30T13:52:20.853 INFO:tasks.workunit.client.0.smithi148.stdout:- p != dirfrags.end(); 2023-10-30T13:52:20.853 INFO:tasks.workunit.client.0.smithi148.stdout:- ++p) 2023-10-30T13:52:20.853 INFO:tasks.workunit.client.0.smithi148.stdout:- if (p->second->is_auth()) { 2023-10-30T13:52:20.853 INFO:tasks.workunit.client.0.smithi148.stdout:- //frag_t fg = (*p)->get_frag(); 2023-10-30T13:52:20.853 INFO:tasks.workunit.client.0.smithi148.stdout:- //frag_sizes[f] = dirfrag_size[fg]; 2023-10-30T13:52:20.854 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.854 INFO:tasks.workunit.client.0.smithi148.stdout:- _encode(frag_sizes, bl); 2023-10-30T13:52:20.854 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.854 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.854 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.854 INFO:tasks.workunit.client.0.smithi148.stdout:- default: 2023-10-30T13:52:20.855 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(0); 2023-10-30T13:52:20.855 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.855 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.855 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.855 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::decode_lock_state(int type, bufferlist& bl) 2023-10-30T13:52:20.856 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.856 INFO:tasks.workunit.client.0.smithi148.stdout:- int off = 0; 2023-10-30T13:52:20.856 INFO:tasks.workunit.client.0.smithi148.stdout:- utime_t tm; 2023-10-30T13:52:20.856 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.856 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (type) { 2023-10-30T13:52:20.856 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IAUTH: 2023-10-30T13:52:20.857 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(tm, bl, off); 2023-10-30T13:52:20.857 INFO:tasks.workunit.client.0.smithi148.stdout:- if (inode.ctime < tm) inode.ctime = tm; 2023-10-30T13:52:20.857 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(inode.mode, bl, off); 2023-10-30T13:52:20.857 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(inode.uid, bl, off); 2023-10-30T13:52:20.857 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(inode.gid, bl, off); 2023-10-30T13:52:20.857 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.858 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.858 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_ILINK: 2023-10-30T13:52:20.858 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(tm, bl, off); 2023-10-30T13:52:20.858 INFO:tasks.workunit.client.0.smithi148.stdout:- if (inode.ctime < tm) inode.ctime = tm; 2023-10-30T13:52:20.858 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(inode.nlink, bl, off); 2023-10-30T13:52:20.859 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(inode.anchored, bl, off); 2023-10-30T13:52:20.859 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.859 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.859 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IDIRFRAGTREE: 2023-10-30T13:52:20.859 INFO:tasks.workunit.client.0.smithi148.stdout:- { 2023-10-30T13:52:20.859 INFO:tasks.workunit.client.0.smithi148.stdout:- fragtree_t temp; 2023-10-30T13:52:20.860 INFO:tasks.workunit.client.0.smithi148.stdout:- temp._decode(bl, off); 2023-10-30T13:52:20.860 INFO:tasks.workunit.client.0.smithi148.stdout:- set authfrags; 2023-10-30T13:52:20.860 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(authfrags, bl, off); 2023-10-30T13:52:20.860 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth()) { 2023-10-30T13:52:20.860 INFO:tasks.workunit.client.0.smithi148.stdout:- // auth. believe replica's auth frags only. 2023-10-30T13:52:20.860 INFO:tasks.workunit.client.0.smithi148.stdout:- for (set::iterator p = authfrags.begin(); p != authfrags.end(); ++p) 2023-10-30T13:52:20.861 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtree.force_to_leaf(*p); 2023-10-30T13:52:20.861 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.861 INFO:tasks.workunit.client.0.smithi148.stdout:- // replica. just take the tree. 2023-10-30T13:52:20.861 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtree.swap(temp); 2023-10-30T13:52:20.861 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.861 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.862 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.862 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.862 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IFILE: 2023-10-30T13:52:20.862 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(inode.size, bl, off); 2023-10-30T13:52:20.862 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(inode.mtime, bl, off); 2023-10-30T13:52:20.863 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(inode.atime, bl, off); 2023-10-30T13:52:20.863 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.863 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.863 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IDIR: 2023-10-30T13:52:20.863 INFO:tasks.workunit.client.0.smithi148.stdout:- //::_decode(inode.size, bl, off); 2023-10-30T13:52:20.863 INFO:tasks.workunit.client.0.smithi148.stdout:- _decode(tm, bl, off); 2023-10-30T13:52:20.864 INFO:tasks.workunit.client.0.smithi148.stdout:- if (inode.mtime < tm) { 2023-10-30T13:52:20.864 INFO:tasks.workunit.client.0.smithi148.stdout:- inode.mtime = tm; 2023-10-30T13:52:20.864 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth()) { 2023-10-30T13:52:20.864 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "decode_lock_state auth got mtime " << tm << " > my " << inode.mtime 2023-10-30T13:52:20.864 INFO:tasks.workunit.client.0.smithi148.stdout:- << ", setting dirlock updated flag on " << *this 2023-10-30T13:52:20.865 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.865 INFO:tasks.workunit.client.0.smithi148.stdout:- dirlock.set_updated(); 2023-10-30T13:52:20.865 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.865 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.865 INFO:tasks.workunit.client.0.smithi148.stdout:- if (0) { 2023-10-30T13:52:20.865 INFO:tasks.workunit.client.0.smithi148.stdout:- map dfsz; 2023-10-30T13:52:20.866 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(dfsz, bl, off); 2023-10-30T13:52:20.866 INFO:tasks.workunit.client.0.smithi148.stdout:- // hmm which to keep? 2023-10-30T13:52:20.866 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.866 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.866 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.867 INFO:tasks.workunit.client.0.smithi148.stdout:- default: 2023-10-30T13:52:20.867 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(0); 2023-10-30T13:52:20.867 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.867 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.867 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.868 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::clear_dirty_scattered(int type) 2023-10-30T13:52:20.868 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.868 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "clear_dirty_scattered " << type << " on " << *this << dendl; 2023-10-30T13:52:20.868 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (type) { 2023-10-30T13:52:20.868 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IDIR: 2023-10-30T13:52:20.869 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty_inode_mtime.remove_myself(); 2023-10-30T13:52:20.869 INFO:tasks.workunit.client.0.smithi148.stdout:- break; 2023-10-30T13:52:20.869 INFO:tasks.workunit.client.0.smithi148.stdout:- default: 2023-10-30T13:52:20.869 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(0); 2023-10-30T13:52:20.869 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.870 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.870 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.870 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.870 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.870 INFO:tasks.workunit.client.0.smithi148.stdout:-// waiting 2023-10-30T13:52:20.871 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.871 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CInode::is_frozen() 2023-10-30T13:52:20.871 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.871 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_frozen_inode()) return true; 2023-10-30T13:52:20.871 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent && parent->dir->is_frozen()) return true; 2023-10-30T13:52:20.871 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.872 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.872 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.872 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CInode::is_frozen_dir() 2023-10-30T13:52:20.872 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.872 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent && parent->dir->is_frozen_dir()) return true; 2023-10-30T13:52:20.873 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.873 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.873 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.873 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CInode::is_freezing() 2023-10-30T13:52:20.873 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.874 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_freezing_inode()) return true; 2023-10-30T13:52:20.874 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent && parent->dir->is_freezing()) return true; 2023-10-30T13:52:20.874 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.874 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.874 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.875 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::add_waiter(int tag, Context *c) 2023-10-30T13:52:20.875 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.875 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "add_waiter tag " << tag 2023-10-30T13:52:20.875 INFO:tasks.workunit.client.0.smithi148.stdout:- << " !ambig " << !state_test(STATE_AMBIGUOUSAUTH) 2023-10-30T13:52:20.875 INFO:tasks.workunit.client.0.smithi148.stdout:- << " !frozen " << !is_frozen_inode() 2023-10-30T13:52:20.875 INFO:tasks.workunit.client.0.smithi148.stdout:- << " !freezing " << !is_freezing_inode() 2023-10-30T13:52:20.876 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.876 INFO:tasks.workunit.client.0.smithi148.stdout:- // wait on the directory? 2023-10-30T13:52:20.876 INFO:tasks.workunit.client.0.smithi148.stdout:- // make sure its not the inode that is explicitly ambiguous|freezing|frozen 2023-10-30T13:52:20.876 INFO:tasks.workunit.client.0.smithi148.stdout:- if (((tag & WAIT_SINGLEAUTH) && !state_test(STATE_AMBIGUOUSAUTH)) || 2023-10-30T13:52:20.876 INFO:tasks.workunit.client.0.smithi148.stdout:- ((tag & WAIT_UNFREEZE) && !is_frozen_inode() && !is_freezing_inode())) { 2023-10-30T13:52:20.876 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->dir->add_waiter(tag, c); 2023-10-30T13:52:20.877 INFO:tasks.workunit.client.0.smithi148.stdout:- return; 2023-10-30T13:52:20.877 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.877 INFO:tasks.workunit.client.0.smithi148.stdout:- MDSCacheObject::add_waiter(tag, c); 2023-10-30T13:52:20.877 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.877 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.878 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CInode::freeze_inode(int auth_pin_allowance) 2023-10-30T13:52:20.878 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.878 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(auth_pin_allowance > 0); // otherwise we need to adjust parent's nested_auth_pins 2023-10-30T13:52:20.878 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(auth_pins >= auth_pin_allowance); 2023-10-30T13:52:20.878 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth_pins > auth_pin_allowance) { 2023-10-30T13:52:20.879 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "freeze_inode - waiting for auth_pins to drop to " << auth_pin_allowance << dendl; 2023-10-30T13:52:20.879 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pin_freeze_allowance = auth_pin_allowance; 2023-10-30T13:52:20.879 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_FREEZING); 2023-10-30T13:52:20.879 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_FREEZING); 2023-10-30T13:52:20.879 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.879 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.880 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.880 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "freeze_inode - frozen" << dendl; 2023-10-30T13:52:20.880 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(auth_pins == auth_pin_allowance); 2023-10-30T13:52:20.880 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_FROZEN); 2023-10-30T13:52:20.880 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_FROZEN); 2023-10-30T13:52:20.880 INFO:tasks.workunit.client.0.smithi148.stdout:- return true; 2023-10-30T13:52:20.881 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.881 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.881 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::unfreeze_inode(list& finished) 2023-10-30T13:52:20.881 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.881 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "unfreeze_inode" << dendl; 2023-10-30T13:52:20.882 INFO:tasks.workunit.client.0.smithi148.stdout:- if (state_test(STATE_FREEZING)) { 2023-10-30T13:52:20.882 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FREEZING); 2023-10-30T13:52:20.882 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_FREEZING); 2023-10-30T13:52:20.882 INFO:tasks.workunit.client.0.smithi148.stdout:- } else if (state_test(STATE_FROZEN)) { 2023-10-30T13:52:20.882 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FROZEN); 2023-10-30T13:52:20.883 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_FROZEN); 2023-10-30T13:52:20.883 INFO:tasks.workunit.client.0.smithi148.stdout:- } else 2023-10-30T13:52:20.883 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(0); 2023-10-30T13:52:20.883 INFO:tasks.workunit.client.0.smithi148.stdout:- take_waiting(WAIT_UNFREEZE, finished); 2023-10-30T13:52:20.883 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.883 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.884 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.884 INFO:tasks.workunit.client.0.smithi148.stdout:-// auth_pins 2023-10-30T13:52:20.884 INFO:tasks.workunit.client.0.smithi148.stdout:-bool CInode::can_auth_pin() { 2023-10-30T13:52:20.884 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_freezing_inode() || is_frozen_inode()) return false; 2023-10-30T13:52:20.884 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) 2023-10-30T13:52:20.885 INFO:tasks.workunit.client.0.smithi148.stdout:- return parent->can_auth_pin(); 2023-10-30T13:52:20.885 INFO:tasks.workunit.client.0.smithi148.stdout:- return true; 2023-10-30T13:52:20.885 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.885 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.885 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::auth_pin() 2023-10-30T13:52:20.885 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.886 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth_pins == 0) 2023-10-30T13:52:20.886 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_AUTHPIN); 2023-10-30T13:52:20.886 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins++; 2023-10-30T13:52:20.886 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.886 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "auth_pin on " << *this 2023-10-30T13:52:20.886 INFO:tasks.workunit.client.0.smithi148.stdout:- << " now " << auth_pins << "+" << nested_auth_pins 2023-10-30T13:52:20.887 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.887 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.887 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) 2023-10-30T13:52:20.887 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->adjust_nested_auth_pins( 1 ); 2023-10-30T13:52:20.888 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.888 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.888 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::auth_unpin() 2023-10-30T13:52:20.888 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.888 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins--; 2023-10-30T13:52:20.889 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth_pins == 0) 2023-10-30T13:52:20.889 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_AUTHPIN); 2023-10-30T13:52:20.889 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.889 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "auth_unpin on " << *this 2023-10-30T13:52:20.889 INFO:tasks.workunit.client.0.smithi148.stdout:- << " now " << auth_pins << "+" << nested_auth_pins 2023-10-30T13:52:20.890 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.890 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.890 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(auth_pins >= 0); 2023-10-30T13:52:20.890 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.890 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) 2023-10-30T13:52:20.891 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->adjust_nested_auth_pins( -1 ); 2023-10-30T13:52:20.891 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.891 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_freezing_inode() && 2023-10-30T13:52:20.891 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins == auth_pin_freeze_allowance) { 2023-10-30T13:52:20.891 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(10) << "auth_unpin freezing!" << dendl; 2023-10-30T13:52:20.891 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_FROZEN); 2023-10-30T13:52:20.892 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_FREEZING); 2023-10-30T13:52:20.892 INFO:tasks.workunit.client.0.smithi148.stdout:- state_clear(STATE_FREEZING); 2023-10-30T13:52:20.892 INFO:tasks.workunit.client.0.smithi148.stdout:- state_set(STATE_FROZEN); 2023-10-30T13:52:20.892 INFO:tasks.workunit.client.0.smithi148.stdout:- finish_waiting(WAIT_FROZEN); 2023-10-30T13:52:20.892 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.893 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.893 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.893 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::adjust_nested_auth_pins(int a) 2023-10-30T13:52:20.893 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.893 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!parent) return; 2023-10-30T13:52:20.893 INFO:tasks.workunit.client.0.smithi148.stdout:- nested_auth_pins += a; 2023-10-30T13:52:20.894 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.894 INFO:tasks.workunit.client.0.smithi148.stdout:- dout(15) << "adjust_nested_auth_pins by " << a 2023-10-30T13:52:20.894 INFO:tasks.workunit.client.0.smithi148.stdout:- << " now " << auth_pins << "+" << nested_auth_pins 2023-10-30T13:52:20.894 INFO:tasks.workunit.client.0.smithi148.stdout:- << dendl; 2023-10-30T13:52:20.894 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(nested_auth_pins >= 0); 2023-10-30T13:52:20.894 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.895 INFO:tasks.workunit.client.0.smithi148.stdout:- parent->adjust_nested_auth_pins(a); 2023-10-30T13:52:20.895 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.895 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.895 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.895 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.895 INFO:tasks.workunit.client.0.smithi148.stdout:-// authority 2023-10-30T13:52:20.896 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.896 INFO:tasks.workunit.client.0.smithi148.stdout:-pair CInode::authority() 2023-10-30T13:52:20.896 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.896 INFO:tasks.workunit.client.0.smithi148.stdout:- if (inode_auth.first >= 0) 2023-10-30T13:52:20.896 INFO:tasks.workunit.client.0.smithi148.stdout:- return inode_auth; 2023-10-30T13:52:20.897 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.897 INFO:tasks.workunit.client.0.smithi148.stdout:- if (parent) 2023-10-30T13:52:20.897 INFO:tasks.workunit.client.0.smithi148.stdout:- return parent->dir->authority(); 2023-10-30T13:52:20.897 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.897 INFO:tasks.workunit.client.0.smithi148.stdout:- return CDIR_AUTH_UNDEF; 2023-10-30T13:52:20.897 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.898 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.898 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.898 INFO:tasks.workunit.client.0.smithi148.stdout:-CInodeDiscover* CInode::replicate_to( int rep ) 2023-10-30T13:52:20.898 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.898 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.899 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.899 INFO:tasks.workunit.client.0.smithi148.stdout:- // relax locks? 2023-10-30T13:52:20.899 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!is_replicated()) 2023-10-30T13:52:20.899 INFO:tasks.workunit.client.0.smithi148.stdout:- replicate_relax_locks(); 2023-10-30T13:52:20.899 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.899 INFO:tasks.workunit.client.0.smithi148.stdout:- // return the thinger 2023-10-30T13:52:20.900 INFO:tasks.workunit.client.0.smithi148.stdout:- int nonce = add_replica( rep ); 2023-10-30T13:52:20.900 INFO:tasks.workunit.client.0.smithi148.stdout:- return new CInodeDiscover( this, nonce ); 2023-10-30T13:52:20.900 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.900 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.900 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.901 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.901 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.901 INFO:tasks.workunit.client.0.smithi148.stdout:-// IMPORT/EXPORT 2023-10-30T13:52:20.901 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.901 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::encode_export(bufferlist& bl) 2023-10-30T13:52:20.902 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.902 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(inode, bl); 2023-10-30T13:52:20.902 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(symlink, bl); 2023-10-30T13:52:20.902 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtree._encode(bl); 2023-10-30T13:52:20.902 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.903 INFO:tasks.workunit.client.0.smithi148.stdout:- bool dirty = is_dirty(); 2023-10-30T13:52:20.903 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(dirty, bl); 2023-10-30T13:52:20.903 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.903 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(pop, bl); 2023-10-30T13:52:20.903 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.903 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode_simple(replica_map, bl); 2023-10-30T13:52:20.904 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.904 INFO:tasks.workunit.client.0.smithi148.stdout:- authlock._encode(bl); 2023-10-30T13:52:20.904 INFO:tasks.workunit.client.0.smithi148.stdout:- linklock._encode(bl); 2023-10-30T13:52:20.904 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtreelock._encode(bl); 2023-10-30T13:52:20.904 INFO:tasks.workunit.client.0.smithi148.stdout:- filelock._encode(bl); 2023-10-30T13:52:20.905 INFO:tasks.workunit.client.0.smithi148.stdout:- dirlock._encode(bl); 2023-10-30T13:52:20.905 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.905 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_TEMPEXPORTING); 2023-10-30T13:52:20.905 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.905 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.905 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::finish_export(utime_t now) 2023-10-30T13:52:20.906 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.906 INFO:tasks.workunit.client.0.smithi148.stdout:- pop.zero(now); 2023-10-30T13:52:20.906 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.906 INFO:tasks.workunit.client.0.smithi148.stdout:- // just in case! 2023-10-30T13:52:20.906 INFO:tasks.workunit.client.0.smithi148.stdout:- dirlock.clear_updated(); 2023-10-30T13:52:20.907 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.907 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_TEMPEXPORTING); 2023-10-30T13:52:20.907 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.907 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.907 INFO:tasks.workunit.client.0.smithi148.stdout:-void CInode::decode_import(bufferlist::iterator& p, 2023-10-30T13:52:20.907 INFO:tasks.workunit.client.0.smithi148.stdout:- LogSegment *ls) 2023-10-30T13:52:20.908 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:20.908 INFO:tasks.workunit.client.0.smithi148.stdout:- utime_t old_mtime = inode.mtime; 2023-10-30T13:52:20.908 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(inode, p); 2023-10-30T13:52:20.908 INFO:tasks.workunit.client.0.smithi148.stdout:- if (old_mtime > inode.mtime) { 2023-10-30T13:52:20.908 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dirlock.is_updated()); 2023-10-30T13:52:20.909 INFO:tasks.workunit.client.0.smithi148.stdout:- inode.mtime = old_mtime; // preserve our mtime, if it is larger 2023-10-30T13:52:20.909 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.909 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.909 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(symlink, p); 2023-10-30T13:52:20.909 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtree._decode(p); 2023-10-30T13:52:20.909 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.910 INFO:tasks.workunit.client.0.smithi148.stdout:- bool dirty; 2023-10-30T13:52:20.910 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(dirty, p); 2023-10-30T13:52:20.910 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dirty) 2023-10-30T13:52:20.910 INFO:tasks.workunit.client.0.smithi148.stdout:- _mark_dirty(ls); 2023-10-30T13:52:20.910 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.911 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(pop, p); 2023-10-30T13:52:20.911 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.911 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode_simple(replica_map, p); 2023-10-30T13:52:20.911 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!replica_map.empty()) get(PIN_REPLICATED); 2023-10-30T13:52:20.911 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.912 INFO:tasks.workunit.client.0.smithi148.stdout:- authlock._decode(p); 2023-10-30T13:52:20.912 INFO:tasks.workunit.client.0.smithi148.stdout:- linklock._decode(p); 2023-10-30T13:52:20.912 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtreelock._decode(p); 2023-10-30T13:52:20.912 INFO:tasks.workunit.client.0.smithi148.stdout:- filelock._decode(p); 2023-10-30T13:52:20.912 INFO:tasks.workunit.client.0.smithi148.stdout:- dirlock._decode(p); 2023-10-30T13:52:20.912 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:20.913 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.913 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.913 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.913 INFO:tasks.workunit.client.0.smithi148.stdout:diff --git a/src/mds/CInode.h b/src/mds/CInode.h 2023-10-30T13:52:20.913 INFO:tasks.workunit.client.0.smithi148.stdout:index 8181be69c66..e69de29bb2d 100644 2023-10-30T13:52:20.914 INFO:tasks.workunit.client.0.smithi148.stdout:--- a/src/mds/CInode.h 2023-10-30T13:52:20.914 INFO:tasks.workunit.client.0.smithi148.stdout:+++ b/src/mds/CInode.h 2023-10-30T13:52:20.914 INFO:tasks.workunit.client.0.smithi148.stdout:@@ -1,607 +0,0 @@ 2023-10-30T13:52:20.914 INFO:tasks.workunit.client.0.smithi148.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2023-10-30T13:52:20.914 INFO:tasks.workunit.client.0.smithi148.stdout:-// vim: ts=8 sw=2 smarttab 2023-10-30T13:52:20.915 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:20.915 INFO:tasks.workunit.client.0.smithi148.stdout:- * Ceph - scalable distributed file system 2023-10-30T13:52:20.915 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.915 INFO:tasks.workunit.client.0.smithi148.stdout:- * Copyright (C) 2004-2006 Sage Weil 2023-10-30T13:52:20.916 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.916 INFO:tasks.workunit.client.0.smithi148.stdout:- * This is free software; you can redistribute it and/or 2023-10-30T13:52:20.916 INFO:tasks.workunit.client.0.smithi148.stdout:- * modify it under the terms of the GNU Lesser General Public 2023-10-30T13:52:20.916 INFO:tasks.workunit.client.0.smithi148.stdout:- * License version 2.1, as published by the Free Software 2023-10-30T13:52:20.916 INFO:tasks.workunit.client.0.smithi148.stdout:- * Foundation. See file COPYING. 2023-10-30T13:52:20.916 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:20.917 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:20.917 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.917 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.917 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.917 INFO:tasks.workunit.client.0.smithi148.stdout:-#ifndef __CINODE_H 2023-10-30T13:52:20.918 INFO:tasks.workunit.client.0.smithi148.stdout:-#define __CINODE_H 2023-10-30T13:52:20.918 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.918 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "config.h" 2023-10-30T13:52:20.918 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/types.h" 2023-10-30T13:52:20.918 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/lru.h" 2023-10-30T13:52:20.919 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.919 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "mdstypes.h" 2023-10-30T13:52:20.919 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.919 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "CDentry.h" 2023-10-30T13:52:20.919 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "SimpleLock.h" 2023-10-30T13:52:20.920 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "FileLock.h" 2023-10-30T13:52:20.920 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "ScatterLock.h" 2023-10-30T13:52:20.920 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "LocalLock.h" 2023-10-30T13:52:20.920 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "Capability.h" 2023-10-30T13:52:20.920 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.921 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.921 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.921 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.921 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.921 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.922 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.922 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:20.922 INFO:tasks.workunit.client.0.smithi148.stdout:-using namespace std; 2023-10-30T13:52:20.922 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.922 INFO:tasks.workunit.client.0.smithi148.stdout:-class Context; 2023-10-30T13:52:20.923 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDentry; 2023-10-30T13:52:20.923 INFO:tasks.workunit.client.0.smithi148.stdout:-class CDir; 2023-10-30T13:52:20.923 INFO:tasks.workunit.client.0.smithi148.stdout:-class Message; 2023-10-30T13:52:20.923 INFO:tasks.workunit.client.0.smithi148.stdout:-class CInode; 2023-10-30T13:52:20.923 INFO:tasks.workunit.client.0.smithi148.stdout:-class CInodeDiscover; 2023-10-30T13:52:20.924 INFO:tasks.workunit.client.0.smithi148.stdout:-class MDCache; 2023-10-30T13:52:20.924 INFO:tasks.workunit.client.0.smithi148.stdout:-class LogSegment; 2023-10-30T13:52:20.924 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.924 INFO:tasks.workunit.client.0.smithi148.stdout:-ostream& operator<<(ostream& out, CInode& in); 2023-10-30T13:52:20.924 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.925 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.925 INFO:tasks.workunit.client.0.smithi148.stdout:-// cached inode wrapper 2023-10-30T13:52:20.925 INFO:tasks.workunit.client.0.smithi148.stdout:-class CInode : public MDSCacheObject { 2023-10-30T13:52:20.925 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.925 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- pins -- 2023-10-30T13:52:20.926 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_DIRFRAG = -1; 2023-10-30T13:52:20.926 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_CAPS = 2; // client caps 2023-10-30T13:52:20.926 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_IMPORTING = -4; // importing 2023-10-30T13:52:20.926 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_ANCHORING = 5; 2023-10-30T13:52:20.926 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_UNANCHORING = 6; 2023-10-30T13:52:20.927 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_OPENINGDIR = 7; 2023-10-30T13:52:20.927 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_REMOTEPARENT = 8; 2023-10-30T13:52:20.927 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_BATCHOPENJOURNAL = 9; 2023-10-30T13:52:20.927 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_SCATTERED = 10; 2023-10-30T13:52:20.927 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_STICKYDIRS = 11; 2023-10-30T13:52:20.928 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_PURGING = -12; 2023-10-30T13:52:20.928 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_FREEZING = 13; 2023-10-30T13:52:20.928 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_FROZEN = 14; 2023-10-30T13:52:20.928 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int PIN_IMPORTINGCAPS = 15; 2023-10-30T13:52:20.928 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.928 INFO:tasks.workunit.client.0.smithi148.stdout:- const char *pin_name(int p) { 2023-10-30T13:52:20.929 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (p) { 2023-10-30T13:52:20.929 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_DIRFRAG: return "dirfrag"; 2023-10-30T13:52:20.929 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_CAPS: return "caps"; 2023-10-30T13:52:20.929 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_IMPORTING: return "importing"; 2023-10-30T13:52:20.929 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_ANCHORING: return "anchoring"; 2023-10-30T13:52:20.930 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_UNANCHORING: return "unanchoring"; 2023-10-30T13:52:20.930 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_OPENINGDIR: return "openingdir"; 2023-10-30T13:52:20.930 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_REMOTEPARENT: return "remoteparent"; 2023-10-30T13:52:20.930 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_BATCHOPENJOURNAL: return "batchopenjournal"; 2023-10-30T13:52:20.930 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_SCATTERED: return "scattered"; 2023-10-30T13:52:20.930 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_STICKYDIRS: return "stickydirs"; 2023-10-30T13:52:20.931 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_PURGING: return "purging"; 2023-10-30T13:52:20.931 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_FREEZING: return "freezing"; 2023-10-30T13:52:20.931 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_FROZEN: return "frozen"; 2023-10-30T13:52:20.931 INFO:tasks.workunit.client.0.smithi148.stdout:- case PIN_IMPORTINGCAPS: return "importingcaps"; 2023-10-30T13:52:20.931 INFO:tasks.workunit.client.0.smithi148.stdout:- default: return generic_pin_name(p); 2023-10-30T13:52:20.931 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.932 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.932 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.932 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- state -- 2023-10-30T13:52:20.932 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_EXPORTING = (1<<2); // on nonauth bystander. 2023-10-30T13:52:20.932 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_ANCHORING = (1<<3); 2023-10-30T13:52:20.932 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_UNANCHORING = (1<<4); 2023-10-30T13:52:20.933 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_OPENINGDIR = (1<<5); 2023-10-30T13:52:20.933 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_REJOINUNDEF = (1<<6); // inode contents undefined. 2023-10-30T13:52:20.933 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_FREEZING = (1<<7); 2023-10-30T13:52:20.933 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_FROZEN = (1<<8); 2023-10-30T13:52:20.933 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_AMBIGUOUSAUTH = (1<<9); 2023-10-30T13:52:20.933 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int STATE_EXPORTINGCAPS = (1<<10); 2023-10-30T13:52:20.934 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.934 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- waiters -- 2023-10-30T13:52:20.934 INFO:tasks.workunit.client.0.smithi148.stdout:- //static const int WAIT_SLAVEAGREE = (1<<0); 2023-10-30T13:52:20.934 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_DIR = (1<<1); 2023-10-30T13:52:20.934 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_ANCHORED = (1<<2); 2023-10-30T13:52:20.934 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_UNANCHORED = (1<<3); 2023-10-30T13:52:20.935 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_CAPS = (1<<4); 2023-10-30T13:52:20.935 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_FROZEN = (1<<5); 2023-10-30T13:52:20.935 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.935 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_AUTHLOCK_OFFSET = 5; 2023-10-30T13:52:20.935 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_LINKLOCK_OFFSET = 5 + SimpleLock::WAIT_BITS; 2023-10-30T13:52:20.935 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_DIRFRAGTREELOCK_OFFSET = 5 + 2*SimpleLock::WAIT_BITS; 2023-10-30T13:52:20.936 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_FILELOCK_OFFSET = 5 + 3*SimpleLock::WAIT_BITS; 2023-10-30T13:52:20.936 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_DIRLOCK_OFFSET = 5 + 4*SimpleLock::WAIT_BITS; 2023-10-30T13:52:20.936 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_VERSIONLOCK_OFFSET = 5 + 5*SimpleLock::WAIT_BITS; 2023-10-30T13:52:20.936 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.936 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int WAIT_ANY = 0xffffffff; 2023-10-30T13:52:20.936 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.937 INFO:tasks.workunit.client.0.smithi148.stdout:- // misc 2023-10-30T13:52:20.937 INFO:tasks.workunit.client.0.smithi148.stdout:- static const int EXPORT_NONCE = 1; // nonce given to replicas created by export 2023-10-30T13:52:20.937 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.937 INFO:tasks.workunit.client.0.smithi148.stdout:- ostream& print_db_line_prefix(ostream& out); 2023-10-30T13:52:20.937 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.938 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.938 INFO:tasks.workunit.client.0.smithi148.stdout:- MDCache *mdcache; 2023-10-30T13:52:20.938 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.938 INFO:tasks.workunit.client.0.smithi148.stdout:- // inode contents proper 2023-10-30T13:52:20.938 INFO:tasks.workunit.client.0.smithi148.stdout:- inode_t inode; // the inode itself 2023-10-30T13:52:20.939 INFO:tasks.workunit.client.0.smithi148.stdout:- string symlink; // symlink dest, if symlink 2023-10-30T13:52:20.939 INFO:tasks.workunit.client.0.smithi148.stdout:- fragtree_t dirfragtree; // dir frag tree, if any. always consistent with our dirfrag map. 2023-10-30T13:52:20.939 INFO:tasks.workunit.client.0.smithi148.stdout:- map dirfrag_size; // size of each dirfrag 2023-10-30T13:52:20.939 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.939 INFO:tasks.workunit.client.0.smithi148.stdout:- off_t last_journaled; // log offset for the last time i was journaled 2023-10-30T13:52:20.940 INFO:tasks.workunit.client.0.smithi148.stdout:- off_t last_open_journaled; // log offset for the last journaled EOpen 2023-10-30T13:52:20.940 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.940 INFO:tasks.workunit.client.0.smithi148.stdout:- //bool hack_accessed; 2023-10-30T13:52:20.940 INFO:tasks.workunit.client.0.smithi148.stdout:- //utime_t hack_load_stamp; 2023-10-30T13:52:20.940 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.941 INFO:tasks.workunit.client.0.smithi148.stdout:- // projected values (only defined while dirty) 2023-10-30T13:52:20.941 INFO:tasks.workunit.client.0.smithi148.stdout:- list projected_inode; 2023-10-30T13:52:20.941 INFO:tasks.workunit.client.0.smithi148.stdout:- list projected_dirfragtree; 2023-10-30T13:52:20.941 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.941 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t get_projected_version() { 2023-10-30T13:52:20.941 INFO:tasks.workunit.client.0.smithi148.stdout:- if (projected_inode.empty()) 2023-10-30T13:52:20.942 INFO:tasks.workunit.client.0.smithi148.stdout:- return inode.version; 2023-10-30T13:52:20.942 INFO:tasks.workunit.client.0.smithi148.stdout:- else 2023-10-30T13:52:20.942 INFO:tasks.workunit.client.0.smithi148.stdout:- return projected_inode.back()->version; 2023-10-30T13:52:20.942 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.942 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.943 INFO:tasks.workunit.client.0.smithi148.stdout:- inode_t *project_inode(); 2023-10-30T13:52:20.943 INFO:tasks.workunit.client.0.smithi148.stdout:- void pop_and_dirty_projected_inode(LogSegment *ls); 2023-10-30T13:52:20.943 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.943 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- cache infrastructure -- 2023-10-30T13:52:20.943 INFO:tasks.workunit.client.0.smithi148.stdout:-private: 2023-10-30T13:52:20.943 INFO:tasks.workunit.client.0.smithi148.stdout:- map dirfrags; // cached dir fragments 2023-10-30T13:52:20.944 INFO:tasks.workunit.client.0.smithi148.stdout:- int stickydir_ref; 2023-10-30T13:52:20.944 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.944 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.944 INFO:tasks.workunit.client.0.smithi148.stdout:- frag_t pick_dirfrag(const string &dn); 2023-10-30T13:52:20.944 INFO:tasks.workunit.client.0.smithi148.stdout:- bool has_dirfrags() { return !dirfrags.empty(); } 2023-10-30T13:52:20.944 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir* get_dirfrag(frag_t fg) { 2023-10-30T13:52:20.945 INFO:tasks.workunit.client.0.smithi148.stdout:- if (dirfrags.count(fg)) { 2023-10-30T13:52:20.945 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(g_conf.debug_mds < 2 || dirfragtree.is_leaf(fg)); // performance hack FIXME 2023-10-30T13:52:20.945 INFO:tasks.workunit.client.0.smithi148.stdout:- return dirfrags[fg]; 2023-10-30T13:52:20.945 INFO:tasks.workunit.client.0.smithi148.stdout:- } else 2023-10-30T13:52:20.945 INFO:tasks.workunit.client.0.smithi148.stdout:- return 0; 2023-10-30T13:52:20.945 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.946 INFO:tasks.workunit.client.0.smithi148.stdout:- void get_dirfrags_under(frag_t fg, list& ls); 2023-10-30T13:52:20.946 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir* get_approx_dirfrag(frag_t fg); 2023-10-30T13:52:20.946 INFO:tasks.workunit.client.0.smithi148.stdout:- void get_dirfrags(list& ls); 2023-10-30T13:52:20.946 INFO:tasks.workunit.client.0.smithi148.stdout:- void get_nested_dirfrags(list& ls); 2023-10-30T13:52:20.946 INFO:tasks.workunit.client.0.smithi148.stdout:- void get_subtree_dirfrags(list& ls); 2023-10-30T13:52:20.947 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *get_or_open_dirfrag(MDCache *mdcache, frag_t fg); 2023-10-30T13:52:20.947 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *add_dirfrag(CDir *dir); 2023-10-30T13:52:20.947 INFO:tasks.workunit.client.0.smithi148.stdout:- void close_dirfrag(frag_t fg); 2023-10-30T13:52:20.947 INFO:tasks.workunit.client.0.smithi148.stdout:- void close_dirfrags(); 2023-10-30T13:52:20.947 INFO:tasks.workunit.client.0.smithi148.stdout:- bool has_subtree_root_dirfrag(); 2023-10-30T13:52:20.948 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.948 INFO:tasks.workunit.client.0.smithi148.stdout:- void get_stickydirs(); 2023-10-30T13:52:20.948 INFO:tasks.workunit.client.0.smithi148.stdout:- void put_stickydirs(); 2023-10-30T13:52:20.948 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.948 INFO:tasks.workunit.client.0.smithi148.stdout:- protected: 2023-10-30T13:52:20.948 INFO:tasks.workunit.client.0.smithi148.stdout:- // parent dentries in cache 2023-10-30T13:52:20.949 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry *parent; // primary link 2023-10-30T13:52:20.949 INFO:tasks.workunit.client.0.smithi148.stdout:- set remote_parents; // if hard linked 2023-10-30T13:52:20.949 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.949 INFO:tasks.workunit.client.0.smithi148.stdout:- pair inode_auth; 2023-10-30T13:52:20.949 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.949 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- distributed state -- 2023-10-30T13:52:20.950 INFO:tasks.workunit.client.0.smithi148.stdout:-protected: 2023-10-30T13:52:20.950 INFO:tasks.workunit.client.0.smithi148.stdout:- // file capabilities 2023-10-30T13:52:20.950 INFO:tasks.workunit.client.0.smithi148.stdout:- map client_caps; // client -> caps 2023-10-30T13:52:20.950 INFO:tasks.workunit.client.0.smithi148.stdout:- map mds_caps_wanted; // [auth] mds -> caps wanted 2023-10-30T13:52:20.950 INFO:tasks.workunit.client.0.smithi148.stdout:- int replica_caps_wanted; // [replica] what i've requested from auth 2023-10-30T13:52:20.950 INFO:tasks.workunit.client.0.smithi148.stdout:- utime_t replica_caps_wanted_keep_until; 2023-10-30T13:52:20.951 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.951 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.951 INFO:tasks.workunit.client.0.smithi148.stdout:- // LogSegment xlists i (may) belong to 2023-10-30T13:52:20.951 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist::item xlist_dirty; 2023-10-30T13:52:20.951 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.952 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist::item xlist_open_file; 2023-10-30T13:52:20.952 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist::item xlist_dirty_inode_mtime; 2023-10-30T13:52:20.952 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist::item xlist_purging_inode; 2023-10-30T13:52:20.952 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.952 INFO:tasks.workunit.client.0.smithi148.stdout:-private: 2023-10-30T13:52:20.952 INFO:tasks.workunit.client.0.smithi148.stdout:- // auth pin 2023-10-30T13:52:20.953 INFO:tasks.workunit.client.0.smithi148.stdout:- int auth_pins; 2023-10-30T13:52:20.953 INFO:tasks.workunit.client.0.smithi148.stdout:- int nested_auth_pins; 2023-10-30T13:52:20.953 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.953 INFO:tasks.workunit.client.0.smithi148.stdout:- int auth_pin_freeze_allowance; 2023-10-30T13:52:20.953 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.953 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.954 INFO:tasks.workunit.client.0.smithi148.stdout:- inode_load_vec_t pop; 2023-10-30T13:52:20.954 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.954 INFO:tasks.workunit.client.0.smithi148.stdout:- // friends 2023-10-30T13:52:20.954 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class Server; 2023-10-30T13:52:20.954 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class Locker; 2023-10-30T13:52:20.955 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class Migrator; 2023-10-30T13:52:20.955 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class MDCache; 2023-10-30T13:52:20.955 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class CDir; 2023-10-30T13:52:20.955 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class CInodeExport; 2023-10-30T13:52:20.955 INFO:tasks.workunit.client.0.smithi148.stdout:- friend class CInodeDiscover; 2023-10-30T13:52:20.955 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.956 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:20.956 INFO:tasks.workunit.client.0.smithi148.stdout:- // --------------------------- 2023-10-30T13:52:20.956 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode(MDCache *c, bool auth=true) : 2023-10-30T13:52:20.956 INFO:tasks.workunit.client.0.smithi148.stdout:- mdcache(c), 2023-10-30T13:52:20.956 INFO:tasks.workunit.client.0.smithi148.stdout:- last_journaled(0), last_open_journaled(0), 2023-10-30T13:52:20.956 INFO:tasks.workunit.client.0.smithi148.stdout:- //hack_accessed(true), 2023-10-30T13:52:20.957 INFO:tasks.workunit.client.0.smithi148.stdout:- stickydir_ref(0), 2023-10-30T13:52:20.957 INFO:tasks.workunit.client.0.smithi148.stdout:- parent(0), inode_auth(CDIR_AUTH_DEFAULT), 2023-10-30T13:52:20.957 INFO:tasks.workunit.client.0.smithi148.stdout:- replica_caps_wanted(0), 2023-10-30T13:52:20.957 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty(this), xlist_open_file(this), 2023-10-30T13:52:20.957 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist_dirty_inode_mtime(this), xlist_purging_inode(this), 2023-10-30T13:52:20.957 INFO:tasks.workunit.client.0.smithi148.stdout:- auth_pins(0), nested_auth_pins(0), 2023-10-30T13:52:20.958 INFO:tasks.workunit.client.0.smithi148.stdout:- versionlock(this, LOCK_OTYPE_IVERSION, WAIT_VERSIONLOCK_OFFSET), 2023-10-30T13:52:20.958 INFO:tasks.workunit.client.0.smithi148.stdout:- authlock(this, LOCK_OTYPE_IAUTH, WAIT_AUTHLOCK_OFFSET), 2023-10-30T13:52:20.958 INFO:tasks.workunit.client.0.smithi148.stdout:- linklock(this, LOCK_OTYPE_ILINK, WAIT_LINKLOCK_OFFSET), 2023-10-30T13:52:20.958 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtreelock(this, LOCK_OTYPE_IDIRFRAGTREE, WAIT_DIRFRAGTREELOCK_OFFSET), 2023-10-30T13:52:20.958 INFO:tasks.workunit.client.0.smithi148.stdout:- filelock(this, LOCK_OTYPE_IFILE, WAIT_FILELOCK_OFFSET), 2023-10-30T13:52:20.959 INFO:tasks.workunit.client.0.smithi148.stdout:- dirlock(this, LOCK_OTYPE_IDIR, WAIT_DIRLOCK_OFFSET) 2023-10-30T13:52:20.959 INFO:tasks.workunit.client.0.smithi148.stdout:- { 2023-10-30T13:52:20.959 INFO:tasks.workunit.client.0.smithi148.stdout:- state = 0; 2023-10-30T13:52:20.959 INFO:tasks.workunit.client.0.smithi148.stdout:- if (auth) state_set(STATE_AUTH); 2023-10-30T13:52:20.959 INFO:tasks.workunit.client.0.smithi148.stdout:- }; 2023-10-30T13:52:20.959 INFO:tasks.workunit.client.0.smithi148.stdout:- ~CInode() { 2023-10-30T13:52:20.960 INFO:tasks.workunit.client.0.smithi148.stdout:- close_dirfrags(); 2023-10-30T13:52:20.960 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.960 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.960 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.961 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- accessors -- 2023-10-30T13:52:20.961 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_file() { return inode.is_file(); } 2023-10-30T13:52:20.961 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_symlink() { return inode.is_symlink(); } 2023-10-30T13:52:20.961 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_dir() { return inode.is_dir(); } 2023-10-30T13:52:20.961 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.962 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_anchored() { return inode.anchored; } 2023-10-30T13:52:20.962 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_anchoring() { return state_test(STATE_ANCHORING); } 2023-10-30T13:52:20.962 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_unanchoring() { return state_test(STATE_UNANCHORING); } 2023-10-30T13:52:20.962 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.962 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_root() { return inode.ino == MDS_INO_ROOT; } 2023-10-30T13:52:20.963 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_stray() { return MDS_INO_IS_STRAY(inode.ino); } 2023-10-30T13:52:20.963 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_base() { return inode.ino < MDS_INO_BASE; } 2023-10-30T13:52:20.963 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.963 INFO:tasks.workunit.client.0.smithi148.stdout:- // note: this overloads MDSCacheObject 2023-10-30T13:52:20.963 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_ambiguous_auth() { 2023-10-30T13:52:20.963 INFO:tasks.workunit.client.0.smithi148.stdout:- return state_test(STATE_AMBIGUOUSAUTH) || 2023-10-30T13:52:20.964 INFO:tasks.workunit.client.0.smithi148.stdout:- MDSCacheObject::is_ambiguous_auth(); 2023-10-30T13:52:20.964 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.964 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.964 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.965 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t ino() const { return inode.ino; } 2023-10-30T13:52:20.965 INFO:tasks.workunit.client.0.smithi148.stdout:- inode_t& get_inode() { return inode; } 2023-10-30T13:52:20.965 INFO:tasks.workunit.client.0.smithi148.stdout:- CDentry* get_parent_dn() { return parent; } 2023-10-30T13:52:20.965 INFO:tasks.workunit.client.0.smithi148.stdout:- CDir *get_parent_dir(); 2023-10-30T13:52:20.965 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *get_parent_inode(); 2023-10-30T13:52:20.965 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.966 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_lt(const MDSCacheObject *r) const { 2023-10-30T13:52:20.966 INFO:tasks.workunit.client.0.smithi148.stdout:- return ino() < ((CInode*)r)->ino(); 2023-10-30T13:52:20.966 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.966 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.966 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- misc -- 2023-10-30T13:52:20.967 INFO:tasks.workunit.client.0.smithi148.stdout:- void make_path_string(string& s); 2023-10-30T13:52:20.967 INFO:tasks.workunit.client.0.smithi148.stdout:- void make_path(filepath& s); 2023-10-30T13:52:20.967 INFO:tasks.workunit.client.0.smithi148.stdout:- void make_anchor_trace(vector& trace); 2023-10-30T13:52:20.967 INFO:tasks.workunit.client.0.smithi148.stdout:- void name_stray_dentry(string& dname); 2023-10-30T13:52:20.968 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.968 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.968 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.968 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- dirtyness -- 2023-10-30T13:52:20.968 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t get_version() { return inode.version; } 2023-10-30T13:52:20.969 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.969 INFO:tasks.workunit.client.0.smithi148.stdout:- version_t pre_dirty(); 2023-10-30T13:52:20.969 INFO:tasks.workunit.client.0.smithi148.stdout:- void _mark_dirty(LogSegment *ls); 2023-10-30T13:52:20.969 INFO:tasks.workunit.client.0.smithi148.stdout:- void mark_dirty(version_t projected_dirv, LogSegment *ls); 2023-10-30T13:52:20.969 INFO:tasks.workunit.client.0.smithi148.stdout:- void mark_clean(); 2023-10-30T13:52:20.970 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.970 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.970 INFO:tasks.workunit.client.0.smithi148.stdout:- CInodeDiscover* replicate_to(int rep); 2023-10-30T13:52:20.970 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.970 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.971 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- waiting -- 2023-10-30T13:52:20.971 INFO:tasks.workunit.client.0.smithi148.stdout:- void add_waiter(int tag, Context *c); 2023-10-30T13:52:20.971 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.971 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.971 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- import/export -- 2023-10-30T13:52:20.971 INFO:tasks.workunit.client.0.smithi148.stdout:- void encode_export(bufferlist& bl); 2023-10-30T13:52:20.972 INFO:tasks.workunit.client.0.smithi148.stdout:- void finish_export(utime_t now); 2023-10-30T13:52:20.972 INFO:tasks.workunit.client.0.smithi148.stdout:- void abort_export() { 2023-10-30T13:52:20.972 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_TEMPEXPORTING); 2023-10-30T13:52:20.972 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.972 INFO:tasks.workunit.client.0.smithi148.stdout:- void decode_import(bufferlist::iterator& p, LogSegment *ls); 2023-10-30T13:52:20.973 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.973 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.973 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- locks -- 2023-10-30T13:52:20.973 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:20.973 INFO:tasks.workunit.client.0.smithi148.stdout:- LocalLock versionlock; 2023-10-30T13:52:20.973 INFO:tasks.workunit.client.0.smithi148.stdout:- SimpleLock authlock; 2023-10-30T13:52:20.974 INFO:tasks.workunit.client.0.smithi148.stdout:- SimpleLock linklock; 2023-10-30T13:52:20.974 INFO:tasks.workunit.client.0.smithi148.stdout:- ScatterLock dirfragtreelock; 2023-10-30T13:52:20.974 INFO:tasks.workunit.client.0.smithi148.stdout:- FileLock filelock; 2023-10-30T13:52:20.974 INFO:tasks.workunit.client.0.smithi148.stdout:- ScatterLock dirlock; 2023-10-30T13:52:20.974 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.975 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.975 INFO:tasks.workunit.client.0.smithi148.stdout:- SimpleLock* get_lock(int type) { 2023-10-30T13:52:20.975 INFO:tasks.workunit.client.0.smithi148.stdout:- switch (type) { 2023-10-30T13:52:20.975 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IFILE: return &filelock; 2023-10-30T13:52:20.975 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IAUTH: return &authlock; 2023-10-30T13:52:20.975 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_ILINK: return &linklock; 2023-10-30T13:52:20.976 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IDIRFRAGTREE: return &dirfragtreelock; 2023-10-30T13:52:20.976 INFO:tasks.workunit.client.0.smithi148.stdout:- case LOCK_OTYPE_IDIR: return &dirlock; 2023-10-30T13:52:20.976 INFO:tasks.workunit.client.0.smithi148.stdout:- default: assert(0); return 0; 2023-10-30T13:52:20.976 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.976 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.977 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_object_info(MDSCacheObjectInfo &info); 2023-10-30T13:52:20.977 INFO:tasks.workunit.client.0.smithi148.stdout:- void encode_lock_state(int type, bufferlist& bl); 2023-10-30T13:52:20.977 INFO:tasks.workunit.client.0.smithi148.stdout:- void decode_lock_state(int type, bufferlist& bl); 2023-10-30T13:52:20.977 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.977 INFO:tasks.workunit.client.0.smithi148.stdout:- void clear_dirty_scattered(int type); 2023-10-30T13:52:20.978 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.978 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- caps -- (new) 2023-10-30T13:52:20.978 INFO:tasks.workunit.client.0.smithi148.stdout:- // client caps 2023-10-30T13:52:20.978 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_any_caps() { return !client_caps.empty(); } 2023-10-30T13:52:20.978 INFO:tasks.workunit.client.0.smithi148.stdout:- map& get_client_caps() { return client_caps; } 2023-10-30T13:52:20.978 INFO:tasks.workunit.client.0.smithi148.stdout:- Capability *get_client_cap(int client) { 2023-10-30T13:52:20.979 INFO:tasks.workunit.client.0.smithi148.stdout:- if (client_caps.count(client)) 2023-10-30T13:52:20.979 INFO:tasks.workunit.client.0.smithi148.stdout:- return client_caps[client]; 2023-10-30T13:52:20.979 INFO:tasks.workunit.client.0.smithi148.stdout:- return 0; 2023-10-30T13:52:20.979 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.979 INFO:tasks.workunit.client.0.smithi148.stdout:- Capability *add_client_cap(int client, CInode *in, xlist& cls) { 2023-10-30T13:52:20.979 INFO:tasks.workunit.client.0.smithi148.stdout:- if (client_caps.empty()) 2023-10-30T13:52:20.980 INFO:tasks.workunit.client.0.smithi148.stdout:- get(PIN_CAPS); 2023-10-30T13:52:20.980 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(client_caps.count(client) == 0); 2023-10-30T13:52:20.980 INFO:tasks.workunit.client.0.smithi148.stdout:- Capability *cap = client_caps[client] = new Capability; 2023-10-30T13:52:20.980 INFO:tasks.workunit.client.0.smithi148.stdout:- cap->set_inode(in); 2023-10-30T13:52:20.980 INFO:tasks.workunit.client.0.smithi148.stdout:- cap->add_to_cap_list(cls); 2023-10-30T13:52:20.980 INFO:tasks.workunit.client.0.smithi148.stdout:- return cap; 2023-10-30T13:52:20.981 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.981 INFO:tasks.workunit.client.0.smithi148.stdout:- void remove_client_cap(int client) { 2023-10-30T13:52:20.981 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(client_caps.count(client) == 1); 2023-10-30T13:52:20.981 INFO:tasks.workunit.client.0.smithi148.stdout:- delete client_caps[client]; 2023-10-30T13:52:20.981 INFO:tasks.workunit.client.0.smithi148.stdout:- client_caps.erase(client); 2023-10-30T13:52:20.981 INFO:tasks.workunit.client.0.smithi148.stdout:- if (client_caps.empty()) 2023-10-30T13:52:20.982 INFO:tasks.workunit.client.0.smithi148.stdout:- put(PIN_CAPS); 2023-10-30T13:52:20.982 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.982 INFO:tasks.workunit.client.0.smithi148.stdout:- void reconnect_cap(int client, inode_caps_reconnect_t& icr, xlist& cls) { 2023-10-30T13:52:20.982 INFO:tasks.workunit.client.0.smithi148.stdout:- Capability *cap = get_client_cap(client); 2023-10-30T13:52:20.982 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap) { 2023-10-30T13:52:20.982 INFO:tasks.workunit.client.0.smithi148.stdout:- cap->merge(icr.wanted, icr.issued); 2023-10-30T13:52:20.983 INFO:tasks.workunit.client.0.smithi148.stdout:- } else { 2023-10-30T13:52:20.983 INFO:tasks.workunit.client.0.smithi148.stdout:- cap = add_client_cap(client, this, cls); 2023-10-30T13:52:20.983 INFO:tasks.workunit.client.0.smithi148.stdout:- cap->set_wanted(icr.wanted); 2023-10-30T13:52:20.983 INFO:tasks.workunit.client.0.smithi148.stdout:- cap->issue(icr.issued); 2023-10-30T13:52:20.983 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.984 INFO:tasks.workunit.client.0.smithi148.stdout:- inode.size = MAX(inode.size, icr.size); 2023-10-30T13:52:20.984 INFO:tasks.workunit.client.0.smithi148.stdout:- inode.mtime = MAX(inode.mtime, icr.mtime); 2023-10-30T13:52:20.984 INFO:tasks.workunit.client.0.smithi148.stdout:- inode.atime = MAX(inode.atime, icr.atime); 2023-10-30T13:52:20.984 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.984 INFO:tasks.workunit.client.0.smithi148.stdout:- void clear_client_caps() { 2023-10-30T13:52:20.984 INFO:tasks.workunit.client.0.smithi148.stdout:- while (!client_caps.empty()) 2023-10-30T13:52:20.985 INFO:tasks.workunit.client.0.smithi148.stdout:- remove_client_cap(client_caps.begin()->first); 2023-10-30T13:52:20.985 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.985 INFO:tasks.workunit.client.0.smithi148.stdout:- void export_client_caps(map& cl) { 2023-10-30T13:52:20.985 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator it = client_caps.begin(); 2023-10-30T13:52:20.985 INFO:tasks.workunit.client.0.smithi148.stdout:- it != client_caps.end(); 2023-10-30T13:52:20.986 INFO:tasks.workunit.client.0.smithi148.stdout:- it++) { 2023-10-30T13:52:20.986 INFO:tasks.workunit.client.0.smithi148.stdout:- cl[it->first] = it->second->make_export(); 2023-10-30T13:52:20.986 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.986 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.986 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.986 INFO:tasks.workunit.client.0.smithi148.stdout:- // caps issued, wanted 2023-10-30T13:52:20.987 INFO:tasks.workunit.client.0.smithi148.stdout:- int get_caps_issued() { 2023-10-30T13:52:20.987 INFO:tasks.workunit.client.0.smithi148.stdout:- int c = 0; 2023-10-30T13:52:20.987 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator it = client_caps.begin(); 2023-10-30T13:52:20.987 INFO:tasks.workunit.client.0.smithi148.stdout:- it != client_caps.end(); 2023-10-30T13:52:20.987 INFO:tasks.workunit.client.0.smithi148.stdout:- it++) 2023-10-30T13:52:20.988 INFO:tasks.workunit.client.0.smithi148.stdout:- c |= it->second->issued(); 2023-10-30T13:52:20.988 INFO:tasks.workunit.client.0.smithi148.stdout:- return c; 2023-10-30T13:52:20.988 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.988 INFO:tasks.workunit.client.0.smithi148.stdout:- int get_caps_wanted() { 2023-10-30T13:52:20.988 INFO:tasks.workunit.client.0.smithi148.stdout:- int w = 0; 2023-10-30T13:52:20.988 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator it = client_caps.begin(); 2023-10-30T13:52:20.989 INFO:tasks.workunit.client.0.smithi148.stdout:- it != client_caps.end(); 2023-10-30T13:52:20.989 INFO:tasks.workunit.client.0.smithi148.stdout:- it++) { 2023-10-30T13:52:20.989 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!it->second->is_stale()) 2023-10-30T13:52:20.989 INFO:tasks.workunit.client.0.smithi148.stdout:- w |= it->second->wanted(); 2023-10-30T13:52:20.989 INFO:tasks.workunit.client.0.smithi148.stdout:- //cout << " get_caps_wanted client " << it->first << " " << cap_string(it->second.wanted()) << endl; 2023-10-30T13:52:20.990 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.990 INFO:tasks.workunit.client.0.smithi148.stdout:- if (is_auth()) 2023-10-30T13:52:20.990 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator it = mds_caps_wanted.begin(); 2023-10-30T13:52:20.990 INFO:tasks.workunit.client.0.smithi148.stdout:- it != mds_caps_wanted.end(); 2023-10-30T13:52:20.990 INFO:tasks.workunit.client.0.smithi148.stdout:- it++) { 2023-10-30T13:52:20.991 INFO:tasks.workunit.client.0.smithi148.stdout:- w |= it->second; 2023-10-30T13:52:20.991 INFO:tasks.workunit.client.0.smithi148.stdout:- //cout << " get_caps_wanted mds " << it->first << " " << cap_string(it->second) << endl; 2023-10-30T13:52:20.991 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.991 INFO:tasks.workunit.client.0.smithi148.stdout:- return w; 2023-10-30T13:52:20.991 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.991 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_loner_cap() { 2023-10-30T13:52:20.992 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!mds_caps_wanted.empty()) 2023-10-30T13:52:20.992 INFO:tasks.workunit.client.0.smithi148.stdout:- return false; 2023-10-30T13:52:20.992 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.992 INFO:tasks.workunit.client.0.smithi148.stdout:- int n = 0; 2023-10-30T13:52:20.992 INFO:tasks.workunit.client.0.smithi148.stdout:- for (map::iterator it = client_caps.begin(); 2023-10-30T13:52:20.993 INFO:tasks.workunit.client.0.smithi148.stdout:- it != client_caps.end(); 2023-10-30T13:52:20.993 INFO:tasks.workunit.client.0.smithi148.stdout:- it++) 2023-10-30T13:52:20.993 INFO:tasks.workunit.client.0.smithi148.stdout:- if (!it->second->is_stale()) { 2023-10-30T13:52:20.993 INFO:tasks.workunit.client.0.smithi148.stdout:- if (n) return false; 2023-10-30T13:52:20.993 INFO:tasks.workunit.client.0.smithi148.stdout:- n++; 2023-10-30T13:52:20.993 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.994 INFO:tasks.workunit.client.0.smithi148.stdout:- return (n == 1); 2023-10-30T13:52:20.994 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.994 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.994 INFO:tasks.workunit.client.0.smithi148.stdout:- void replicate_relax_locks() { 2023-10-30T13:52:20.994 INFO:tasks.workunit.client.0.smithi148.stdout:- //dout(10) << " relaxing locks on " << *this << dendl; 2023-10-30T13:52:20.995 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(is_auth()); 2023-10-30T13:52:20.995 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(!is_replicated()); 2023-10-30T13:52:20.995 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.995 INFO:tasks.workunit.client.0.smithi148.stdout:- authlock.replicate_relax(); 2023-10-30T13:52:20.995 INFO:tasks.workunit.client.0.smithi148.stdout:- linklock.replicate_relax(); 2023-10-30T13:52:20.996 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtreelock.replicate_relax(); 2023-10-30T13:52:20.996 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.996 INFO:tasks.workunit.client.0.smithi148.stdout:- if ((get_caps_issued() & (CEPH_CAP_WR|CEPH_CAP_WRBUFFER)) == 0) 2023-10-30T13:52:20.996 INFO:tasks.workunit.client.0.smithi148.stdout:- filelock.replicate_relax(); 2023-10-30T13:52:20.996 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.997 INFO:tasks.workunit.client.0.smithi148.stdout:- dirlock.replicate_relax(); 2023-10-30T13:52:20.997 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.997 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.997 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.997 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- authority -- 2023-10-30T13:52:20.997 INFO:tasks.workunit.client.0.smithi148.stdout:- pair authority(); 2023-10-30T13:52:20.998 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.998 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:20.998 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- auth pins -- 2023-10-30T13:52:20.998 INFO:tasks.workunit.client.0.smithi148.stdout:- int is_auth_pinned() { 2023-10-30T13:52:20.998 INFO:tasks.workunit.client.0.smithi148.stdout:- return auth_pins; 2023-10-30T13:52:20.999 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:20.999 INFO:tasks.workunit.client.0.smithi148.stdout:- void adjust_nested_auth_pins(int a); 2023-10-30T13:52:20.999 INFO:tasks.workunit.client.0.smithi148.stdout:- bool can_auth_pin(); 2023-10-30T13:52:20.999 INFO:tasks.workunit.client.0.smithi148.stdout:- void auth_pin(); 2023-10-30T13:52:20.999 INFO:tasks.workunit.client.0.smithi148.stdout:- void auth_unpin(); 2023-10-30T13:52:21.000 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.000 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.000 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- freeze -- 2023-10-30T13:52:21.000 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_freezing_inode() { return state_test(STATE_FREEZING); } 2023-10-30T13:52:21.001 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_frozen_inode() { return state_test(STATE_FROZEN); } 2023-10-30T13:52:21.001 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_frozen(); 2023-10-30T13:52:21.001 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_frozen_dir(); 2023-10-30T13:52:21.001 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_freezing(); 2023-10-30T13:52:21.001 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.001 INFO:tasks.workunit.client.0.smithi148.stdout:- bool freeze_inode(int auth_pin_allowance=0); 2023-10-30T13:52:21.002 INFO:tasks.workunit.client.0.smithi148.stdout:- void unfreeze_inode(list& finished); 2023-10-30T13:52:21.002 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.002 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.002 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- reference counting -- 2023-10-30T13:52:21.002 INFO:tasks.workunit.client.0.smithi148.stdout:- void bad_put(int by) { 2023-10-30T13:52:21.003 INFO:tasks.workunit.client.0.smithi148.stdout:- generic_dout(7) << " bad put " << *this << " by " << by << " " << pin_name(by) << " was " << ref << " (" << ref_set << ")" << dendl; 2023-10-30T13:52:21.003 INFO:tasks.workunit.client.0.smithi148.stdout:-#ifdef MDS_REF_SET 2023-10-30T13:52:21.003 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(ref_set.count(by) == 1); 2023-10-30T13:52:21.003 INFO:tasks.workunit.client.0.smithi148.stdout:-#endif 2023-10-30T13:52:21.003 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(ref > 0); 2023-10-30T13:52:21.003 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.004 INFO:tasks.workunit.client.0.smithi148.stdout:- void bad_get(int by) { 2023-10-30T13:52:21.004 INFO:tasks.workunit.client.0.smithi148.stdout:- generic_dout(7) << " bad get " << *this << " by " << by << " " << pin_name(by) << " was " << ref << " (" << ref_set << ")" << dendl; 2023-10-30T13:52:21.004 INFO:tasks.workunit.client.0.smithi148.stdout:-#ifdef MDS_REF_SET 2023-10-30T13:52:21.004 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(ref_set.count(by) == 0); 2023-10-30T13:52:21.004 INFO:tasks.workunit.client.0.smithi148.stdout:-#endif 2023-10-30T13:52:21.005 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.005 INFO:tasks.workunit.client.0.smithi148.stdout:- void first_get(); 2023-10-30T13:52:21.005 INFO:tasks.workunit.client.0.smithi148.stdout:- void last_put(); 2023-10-30T13:52:21.005 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.005 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.005 INFO:tasks.workunit.client.0.smithi148.stdout:- // -- hierarchy stuff -- 2023-10-30T13:52:21.006 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:21.006 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_primary_parent(CDentry *p) { 2023-10-30T13:52:21.006 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(parent == 0); 2023-10-30T13:52:21.006 INFO:tasks.workunit.client.0.smithi148.stdout:- parent = p; 2023-10-30T13:52:21.006 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.007 INFO:tasks.workunit.client.0.smithi148.stdout:- void remove_primary_parent(CDentry *dn) { 2023-10-30T13:52:21.007 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(dn == parent); 2023-10-30T13:52:21.007 INFO:tasks.workunit.client.0.smithi148.stdout:- parent = 0; 2023-10-30T13:52:21.007 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.007 INFO:tasks.workunit.client.0.smithi148.stdout:- void add_remote_parent(CDentry *p); 2023-10-30T13:52:21.008 INFO:tasks.workunit.client.0.smithi148.stdout:- void remove_remote_parent(CDentry *p); 2023-10-30T13:52:21.008 INFO:tasks.workunit.client.0.smithi148.stdout:- int num_remote_parents() { 2023-10-30T13:52:21.008 INFO:tasks.workunit.client.0.smithi148.stdout:- return remote_parents.size(); 2023-10-30T13:52:21.008 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.009 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.009 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.009 INFO:tasks.workunit.client.0.smithi148.stdout:- /* 2023-10-30T13:52:21.009 INFO:tasks.workunit.client.0.smithi148.stdout:- // for giving to clients 2023-10-30T13:52:21.009 INFO:tasks.workunit.client.0.smithi148.stdout:- void get_dist_spec(set& ls, int auth, timepair_t& now) { 2023-10-30T13:52:21.009 INFO:tasks.workunit.client.0.smithi148.stdout:- if (( is_dir() && popularity[MDS_POP_CURDOM].get(now) > g_conf.mds_bal_replicate_threshold) || 2023-10-30T13:52:21.010 INFO:tasks.workunit.client.0.smithi148.stdout:- (!is_dir() && popularity[MDS_POP_JUSTME].get(now) > g_conf.mds_bal_replicate_threshold)) { 2023-10-30T13:52:21.010 INFO:tasks.workunit.client.0.smithi148.stdout:- //if (!cached_by.empty() && inode.ino > 1) dout(1) << "distributed spec for " << *this << dendl; 2023-10-30T13:52:21.010 INFO:tasks.workunit.client.0.smithi148.stdout:- ls = cached_by; 2023-10-30T13:52:21.010 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.010 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.011 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:21.011 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.011 INFO:tasks.workunit.client.0.smithi148.stdout:- void print(ostream& out); 2023-10-30T13:52:21.011 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.011 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:21.011 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.012 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.012 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.012 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.012 INFO:tasks.workunit.client.0.smithi148.stdout:-// -- encoded state 2023-10-30T13:52:21.012 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.013 INFO:tasks.workunit.client.0.smithi148.stdout:-// discover 2023-10-30T13:52:21.013 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.013 INFO:tasks.workunit.client.0.smithi148.stdout:-class CInodeDiscover { 2023-10-30T13:52:21.013 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.013 INFO:tasks.workunit.client.0.smithi148.stdout:- inode_t inode; 2023-10-30T13:52:21.013 INFO:tasks.workunit.client.0.smithi148.stdout:- string symlink; 2023-10-30T13:52:21.014 INFO:tasks.workunit.client.0.smithi148.stdout:- fragtree_t dirfragtree; 2023-10-30T13:52:21.014 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.014 INFO:tasks.workunit.client.0.smithi148.stdout:- int replica_nonce; 2023-10-30T13:52:21.014 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.014 INFO:tasks.workunit.client.0.smithi148.stdout:- int authlock_state; 2023-10-30T13:52:21.014 INFO:tasks.workunit.client.0.smithi148.stdout:- int linklock_state; 2023-10-30T13:52:21.015 INFO:tasks.workunit.client.0.smithi148.stdout:- int dirfragtreelock_state; 2023-10-30T13:52:21.015 INFO:tasks.workunit.client.0.smithi148.stdout:- int filelock_state; 2023-10-30T13:52:21.015 INFO:tasks.workunit.client.0.smithi148.stdout:- int dirlock_state; 2023-10-30T13:52:21.015 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.015 INFO:tasks.workunit.client.0.smithi148.stdout:- public: 2023-10-30T13:52:21.015 INFO:tasks.workunit.client.0.smithi148.stdout:- CInodeDiscover() {} 2023-10-30T13:52:21.016 INFO:tasks.workunit.client.0.smithi148.stdout:- CInodeDiscover(CInode *in, int nonce) { 2023-10-30T13:52:21.016 INFO:tasks.workunit.client.0.smithi148.stdout:- inode = in->inode; 2023-10-30T13:52:21.016 INFO:tasks.workunit.client.0.smithi148.stdout:- symlink = in->symlink; 2023-10-30T13:52:21.016 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtree = in->dirfragtree; 2023-10-30T13:52:21.016 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.016 INFO:tasks.workunit.client.0.smithi148.stdout:- replica_nonce = nonce; 2023-10-30T13:52:21.017 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.017 INFO:tasks.workunit.client.0.smithi148.stdout:- authlock_state = in->authlock.get_replica_state(); 2023-10-30T13:52:21.017 INFO:tasks.workunit.client.0.smithi148.stdout:- linklock_state = in->linklock.get_replica_state(); 2023-10-30T13:52:21.017 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtreelock_state = in->dirfragtreelock.get_replica_state(); 2023-10-30T13:52:21.017 INFO:tasks.workunit.client.0.smithi148.stdout:- filelock_state = in->filelock.get_replica_state(); 2023-10-30T13:52:21.017 INFO:tasks.workunit.client.0.smithi148.stdout:- dirlock_state = in->dirlock.get_replica_state(); 2023-10-30T13:52:21.018 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.018 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.018 INFO:tasks.workunit.client.0.smithi148.stdout:- inodeno_t get_ino() { return inode.ino; } 2023-10-30T13:52:21.018 INFO:tasks.workunit.client.0.smithi148.stdout:- int get_replica_nonce() { return replica_nonce; } 2023-10-30T13:52:21.018 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.019 INFO:tasks.workunit.client.0.smithi148.stdout:- void update_inode(CInode *in) { 2023-10-30T13:52:21.019 INFO:tasks.workunit.client.0.smithi148.stdout:- in->inode = inode; 2023-10-30T13:52:21.019 INFO:tasks.workunit.client.0.smithi148.stdout:- in->symlink = symlink; 2023-10-30T13:52:21.019 INFO:tasks.workunit.client.0.smithi148.stdout:- in->dirfragtree = dirfragtree; 2023-10-30T13:52:21.019 INFO:tasks.workunit.client.0.smithi148.stdout:- in->replica_nonce = replica_nonce; 2023-10-30T13:52:21.019 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.020 INFO:tasks.workunit.client.0.smithi148.stdout:- void init_inode_locks(CInode *in) { 2023-10-30T13:52:21.020 INFO:tasks.workunit.client.0.smithi148.stdout:- in->authlock.set_state(authlock_state); 2023-10-30T13:52:21.020 INFO:tasks.workunit.client.0.smithi148.stdout:- in->linklock.set_state(linklock_state); 2023-10-30T13:52:21.020 INFO:tasks.workunit.client.0.smithi148.stdout:- in->dirfragtreelock.set_state(dirfragtreelock_state); 2023-10-30T13:52:21.020 INFO:tasks.workunit.client.0.smithi148.stdout:- in->filelock.set_state(filelock_state); 2023-10-30T13:52:21.021 INFO:tasks.workunit.client.0.smithi148.stdout:- in->dirlock.set_state(dirlock_state); 2023-10-30T13:52:21.021 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.021 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.021 INFO:tasks.workunit.client.0.smithi148.stdout:- void _encode(bufferlist& bl) { 2023-10-30T13:52:21.021 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(inode, bl); 2023-10-30T13:52:21.021 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(symlink, bl); 2023-10-30T13:52:21.022 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtree._encode(bl); 2023-10-30T13:52:21.022 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(replica_nonce, bl); 2023-10-30T13:52:21.022 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(authlock_state, bl); 2023-10-30T13:52:21.022 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(linklock_state, bl); 2023-10-30T13:52:21.022 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(dirfragtreelock_state, bl); 2023-10-30T13:52:21.022 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(filelock_state, bl); 2023-10-30T13:52:21.023 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(dirlock_state, bl); 2023-10-30T13:52:21.023 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.023 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.023 INFO:tasks.workunit.client.0.smithi148.stdout:- void _decode(bufferlist& bl, int& off) { 2023-10-30T13:52:21.023 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(inode, bl, off); 2023-10-30T13:52:21.024 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(symlink, bl, off); 2023-10-30T13:52:21.024 INFO:tasks.workunit.client.0.smithi148.stdout:- dirfragtree._decode(bl, off); 2023-10-30T13:52:21.024 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(replica_nonce, bl, off); 2023-10-30T13:52:21.024 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(authlock_state, bl, off); 2023-10-30T13:52:21.025 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(linklock_state, bl, off); 2023-10-30T13:52:21.025 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(dirfragtreelock_state, bl, off); 2023-10-30T13:52:21.025 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(filelock_state, bl, off); 2023-10-30T13:52:21.025 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(dirlock_state, bl, off); 2023-10-30T13:52:21.025 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.025 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.026 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:21.026 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.026 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.026 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.026 INFO:tasks.workunit.client.0.smithi148.stdout:-#endif 2023-10-30T13:52:21.026 INFO:tasks.workunit.client.0.smithi148.stdout:diff --git a/src/mds/Capability.h b/src/mds/Capability.h 2023-10-30T13:52:21.027 INFO:tasks.workunit.client.0.smithi148.stdout:index 999a999b3cb..e69de29bb2d 100644 2023-10-30T13:52:21.027 INFO:tasks.workunit.client.0.smithi148.stdout:--- a/src/mds/Capability.h 2023-10-30T13:52:21.027 INFO:tasks.workunit.client.0.smithi148.stdout:+++ b/src/mds/Capability.h 2023-10-30T13:52:21.027 INFO:tasks.workunit.client.0.smithi148.stdout:@@ -1,246 +0,0 @@ 2023-10-30T13:52:21.027 INFO:tasks.workunit.client.0.smithi148.stdout:-// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 2023-10-30T13:52:21.027 INFO:tasks.workunit.client.0.smithi148.stdout:-// vim: ts=8 sw=2 smarttab 2023-10-30T13:52:21.028 INFO:tasks.workunit.client.0.smithi148.stdout:-/* 2023-10-30T13:52:21.028 INFO:tasks.workunit.client.0.smithi148.stdout:- * Ceph - scalable distributed file system 2023-10-30T13:52:21.028 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:21.028 INFO:tasks.workunit.client.0.smithi148.stdout:- * Copyright (C) 2004-2006 Sage Weil 2023-10-30T13:52:21.028 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:21.029 INFO:tasks.workunit.client.0.smithi148.stdout:- * This is free software; you can redistribute it and/or 2023-10-30T13:52:21.029 INFO:tasks.workunit.client.0.smithi148.stdout:- * modify it under the terms of the GNU Lesser General Public 2023-10-30T13:52:21.029 INFO:tasks.workunit.client.0.smithi148.stdout:- * License version 2.1, as published by the Free Software 2023-10-30T13:52:21.029 INFO:tasks.workunit.client.0.smithi148.stdout:- * Foundation. See file COPYING. 2023-10-30T13:52:21.029 INFO:tasks.workunit.client.0.smithi148.stdout:- * 2023-10-30T13:52:21.029 INFO:tasks.workunit.client.0.smithi148.stdout:- */ 2023-10-30T13:52:21.030 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.030 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.030 INFO:tasks.workunit.client.0.smithi148.stdout:-#ifndef __CAPABILITY_H 2023-10-30T13:52:21.030 INFO:tasks.workunit.client.0.smithi148.stdout:-#define __CAPABILITY_H 2023-10-30T13:52:21.030 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.030 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/buffer.h" 2023-10-30T13:52:21.031 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "include/xlist.h" 2023-10-30T13:52:21.031 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.031 INFO:tasks.workunit.client.0.smithi148.stdout:-#include 2023-10-30T13:52:21.031 INFO:tasks.workunit.client.0.smithi148.stdout:-using namespace std; 2023-10-30T13:52:21.031 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.031 INFO:tasks.workunit.client.0.smithi148.stdout:-#include "config.h" 2023-10-30T13:52:21.032 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.032 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.032 INFO:tasks.workunit.client.0.smithi148.stdout:-// heuristics 2023-10-30T13:52:21.032 INFO:tasks.workunit.client.0.smithi148.stdout:-//#define CEPH_CAP_DELAYFLUSH 32 2023-10-30T13:52:21.032 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.032 INFO:tasks.workunit.client.0.smithi148.stdout:-inline string cap_string(int cap) 2023-10-30T13:52:21.033 INFO:tasks.workunit.client.0.smithi148.stdout:-{ 2023-10-30T13:52:21.033 INFO:tasks.workunit.client.0.smithi148.stdout:- string s; 2023-10-30T13:52:21.033 INFO:tasks.workunit.client.0.smithi148.stdout:- s = "["; 2023-10-30T13:52:21.033 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap & CEPH_CAP_PIN) s += " pin"; 2023-10-30T13:52:21.033 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap & CEPH_CAP_RDCACHE) s += " rdcache"; 2023-10-30T13:52:21.033 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap & CEPH_CAP_RD) s += " rd"; 2023-10-30T13:52:21.034 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap & CEPH_CAP_WR) s += " wr"; 2023-10-30T13:52:21.034 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap & CEPH_CAP_WRBUFFER) s += " wrbuffer"; 2023-10-30T13:52:21.034 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap & CEPH_CAP_WRBUFFER) s += " wrextend"; 2023-10-30T13:52:21.034 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap & CEPH_CAP_LAZYIO) s += " lazyio"; 2023-10-30T13:52:21.034 INFO:tasks.workunit.client.0.smithi148.stdout:- s += " ]"; 2023-10-30T13:52:21.034 INFO:tasks.workunit.client.0.smithi148.stdout:- return s; 2023-10-30T13:52:21.035 INFO:tasks.workunit.client.0.smithi148.stdout:-} 2023-10-30T13:52:21.035 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.035 INFO:tasks.workunit.client.0.smithi148.stdout:-typedef uint32_t capseq_t; 2023-10-30T13:52:21.035 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.035 INFO:tasks.workunit.client.0.smithi148.stdout:-class CInode; 2023-10-30T13:52:21.035 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.036 INFO:tasks.workunit.client.0.smithi148.stdout:-class Capability { 2023-10-30T13:52:21.036 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:21.036 INFO:tasks.workunit.client.0.smithi148.stdout:- struct Export { 2023-10-30T13:52:21.036 INFO:tasks.workunit.client.0.smithi148.stdout:- int wanted; 2023-10-30T13:52:21.036 INFO:tasks.workunit.client.0.smithi148.stdout:- int issued; 2023-10-30T13:52:21.036 INFO:tasks.workunit.client.0.smithi148.stdout:- int pending; 2023-10-30T13:52:21.037 INFO:tasks.workunit.client.0.smithi148.stdout:- Export() {} 2023-10-30T13:52:21.037 INFO:tasks.workunit.client.0.smithi148.stdout:- Export(int w, int i, int p) : wanted(w), issued(i), pending(p) {} 2023-10-30T13:52:21.037 INFO:tasks.workunit.client.0.smithi148.stdout:- }; 2023-10-30T13:52:21.037 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.037 INFO:tasks.workunit.client.0.smithi148.stdout:-private: 2023-10-30T13:52:21.037 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *inode; 2023-10-30T13:52:21.038 INFO:tasks.workunit.client.0.smithi148.stdout:- int wanted_caps; // what the client wants (ideally) 2023-10-30T13:52:21.038 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.038 INFO:tasks.workunit.client.0.smithi148.stdout:- map cap_history; // seq -> cap 2023-10-30T13:52:21.038 INFO:tasks.workunit.client.0.smithi148.stdout:- capseq_t last_sent, last_recv; 2023-10-30T13:52:21.038 INFO:tasks.workunit.client.0.smithi148.stdout:- capseq_t last_open; 2023-10-30T13:52:21.039 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.039 INFO:tasks.workunit.client.0.smithi148.stdout:- bool suppress; 2023-10-30T13:52:21.039 INFO:tasks.workunit.client.0.smithi148.stdout:- bool stale; 2023-10-30T13:52:21.039 INFO:tasks.workunit.client.0.smithi148.stdout:-public: 2023-10-30T13:52:21.039 INFO:tasks.workunit.client.0.smithi148.stdout:- xlist::item session_caps_item; 2023-10-30T13:52:21.040 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.040 INFO:tasks.workunit.client.0.smithi148.stdout:- Capability(CInode *i=0, int want=0, capseq_t s=0) : 2023-10-30T13:52:21.040 INFO:tasks.workunit.client.0.smithi148.stdout:- inode(i), 2023-10-30T13:52:21.040 INFO:tasks.workunit.client.0.smithi148.stdout:- wanted_caps(want), 2023-10-30T13:52:21.040 INFO:tasks.workunit.client.0.smithi148.stdout:- last_sent(s), 2023-10-30T13:52:21.040 INFO:tasks.workunit.client.0.smithi148.stdout:- last_recv(s), 2023-10-30T13:52:21.041 INFO:tasks.workunit.client.0.smithi148.stdout:- last_open(0), 2023-10-30T13:52:21.041 INFO:tasks.workunit.client.0.smithi148.stdout:- suppress(false), stale(false), 2023-10-30T13:52:21.041 INFO:tasks.workunit.client.0.smithi148.stdout:- session_caps_item(this) { 2023-10-30T13:52:21.041 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.041 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.041 INFO:tasks.workunit.client.0.smithi148.stdout:- capseq_t get_last_open() { return last_open; } 2023-10-30T13:52:21.042 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_last_open() { last_open = last_sent; } 2023-10-30T13:52:21.042 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.042 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_suppress() { return suppress; } 2023-10-30T13:52:21.042 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_suppress(bool b) { suppress = b; } 2023-10-30T13:52:21.042 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.042 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_stale() { return stale; } 2023-10-30T13:52:21.043 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_stale(bool b) { stale = b; } 2023-10-30T13:52:21.043 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.043 INFO:tasks.workunit.client.0.smithi148.stdout:- CInode *get_inode() { return inode; } 2023-10-30T13:52:21.043 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_inode(CInode *i) { inode = i; } 2023-10-30T13:52:21.043 INFO:tasks.workunit.client.0.smithi148.stdout:- void add_to_cap_list(xlist& ls) { 2023-10-30T13:52:21.043 INFO:tasks.workunit.client.0.smithi148.stdout:- ls.push_back(&session_caps_item); 2023-10-30T13:52:21.044 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.044 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.044 INFO:tasks.workunit.client.0.smithi148.stdout:- bool is_null() { return cap_history.empty() && wanted_caps == 0; } 2023-10-30T13:52:21.044 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.044 INFO:tasks.workunit.client.0.smithi148.stdout:- // most recently issued caps. 2023-10-30T13:52:21.044 INFO:tasks.workunit.client.0.smithi148.stdout:- int pending() { 2023-10-30T13:52:21.045 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap_history.count(last_sent)) 2023-10-30T13:52:21.045 INFO:tasks.workunit.client.0.smithi148.stdout:- return cap_history[ last_sent ]; 2023-10-30T13:52:21.045 INFO:tasks.workunit.client.0.smithi148.stdout:- return 0; 2023-10-30T13:52:21.045 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.045 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.046 INFO:tasks.workunit.client.0.smithi148.stdout:- // caps client has confirmed receipt of 2023-10-30T13:52:21.046 INFO:tasks.workunit.client.0.smithi148.stdout:- int confirmed() { 2023-10-30T13:52:21.046 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap_history.count(last_recv)) 2023-10-30T13:52:21.046 INFO:tasks.workunit.client.0.smithi148.stdout:- return cap_history[ last_recv ]; 2023-10-30T13:52:21.046 INFO:tasks.workunit.client.0.smithi148.stdout:- return 0; 2023-10-30T13:52:21.046 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.047 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.047 INFO:tasks.workunit.client.0.smithi148.stdout:- // caps potentially issued 2023-10-30T13:52:21.047 INFO:tasks.workunit.client.0.smithi148.stdout:- int issued() { 2023-10-30T13:52:21.047 INFO:tasks.workunit.client.0.smithi148.stdout:- int c = 0; 2023-10-30T13:52:21.047 INFO:tasks.workunit.client.0.smithi148.stdout:- for (capseq_t seq = last_recv; seq <= last_sent; seq++) { 2023-10-30T13:52:21.048 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap_history.count(seq)) { 2023-10-30T13:52:21.048 INFO:tasks.workunit.client.0.smithi148.stdout:- c |= cap_history[seq]; 2023-10-30T13:52:21.048 INFO:tasks.workunit.client.0.smithi148.stdout:- generic_dout(10) << " cap issued: seq " << seq << " " << cap_string(cap_history[seq]) << " -> " << cap_string(c) << dendl; 2023-10-30T13:52:21.048 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.048 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.048 INFO:tasks.workunit.client.0.smithi148.stdout:- return c; 2023-10-30T13:52:21.049 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.049 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.049 INFO:tasks.workunit.client.0.smithi148.stdout:- // caps this client wants to hold 2023-10-30T13:52:21.049 INFO:tasks.workunit.client.0.smithi148.stdout:- int wanted() { return wanted_caps; } 2023-10-30T13:52:21.049 INFO:tasks.workunit.client.0.smithi148.stdout:- void set_wanted(int w) { 2023-10-30T13:52:21.049 INFO:tasks.workunit.client.0.smithi148.stdout:- wanted_caps = w; 2023-10-30T13:52:21.050 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.050 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.050 INFO:tasks.workunit.client.0.smithi148.stdout:- // needed 2023-10-30T13:52:21.050 INFO:tasks.workunit.client.0.smithi148.stdout:- static int needed(int from) { 2023-10-30T13:52:21.050 INFO:tasks.workunit.client.0.smithi148.stdout:- // strip out wrbuffer, rdcache 2023-10-30T13:52:21.050 INFO:tasks.workunit.client.0.smithi148.stdout:- return from & (CEPH_CAP_WR|CEPH_CAP_RD); 2023-10-30T13:52:21.051 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.051 INFO:tasks.workunit.client.0.smithi148.stdout:- int needed() { return needed(wanted_caps); } 2023-10-30T13:52:21.051 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.051 INFO:tasks.workunit.client.0.smithi148.stdout:- // conflicts 2023-10-30T13:52:21.051 INFO:tasks.workunit.client.0.smithi148.stdout:- static int conflicts(int from) { 2023-10-30T13:52:21.051 INFO:tasks.workunit.client.0.smithi148.stdout:- int c = 0; 2023-10-30T13:52:21.052 INFO:tasks.workunit.client.0.smithi148.stdout:- if (from & CEPH_CAP_WRBUFFER) c |= CEPH_CAP_RDCACHE|CEPH_CAP_RD; 2023-10-30T13:52:21.052 INFO:tasks.workunit.client.0.smithi148.stdout:- if (from & CEPH_CAP_WR) c |= CEPH_CAP_RDCACHE; 2023-10-30T13:52:21.052 INFO:tasks.workunit.client.0.smithi148.stdout:- if (from & CEPH_CAP_RD) c |= CEPH_CAP_WRBUFFER; 2023-10-30T13:52:21.052 INFO:tasks.workunit.client.0.smithi148.stdout:- if (from & CEPH_CAP_RDCACHE) c |= CEPH_CAP_WRBUFFER|CEPH_CAP_WR; 2023-10-30T13:52:21.052 INFO:tasks.workunit.client.0.smithi148.stdout:- return c; 2023-10-30T13:52:21.052 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.053 INFO:tasks.workunit.client.0.smithi148.stdout:- int wanted_conflicts() { return conflicts(wanted()); } 2023-10-30T13:52:21.053 INFO:tasks.workunit.client.0.smithi148.stdout:- int needed_conflicts() { return conflicts(needed()); } 2023-10-30T13:52:21.053 INFO:tasks.workunit.client.0.smithi148.stdout:- int issued_conflicts() { return conflicts(issued()); } 2023-10-30T13:52:21.053 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.053 INFO:tasks.workunit.client.0.smithi148.stdout:- // issue caps; return seq number. 2023-10-30T13:52:21.053 INFO:tasks.workunit.client.0.smithi148.stdout:- capseq_t issue(int c) { 2023-10-30T13:52:21.054 INFO:tasks.workunit.client.0.smithi148.stdout:- ++last_sent; 2023-10-30T13:52:21.054 INFO:tasks.workunit.client.0.smithi148.stdout:- cap_history[last_sent] = c; 2023-10-30T13:52:21.054 INFO:tasks.workunit.client.0.smithi148.stdout:- return last_sent; 2023-10-30T13:52:21.054 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.054 INFO:tasks.workunit.client.0.smithi148.stdout:- capseq_t get_last_seq() { return last_sent; } 2023-10-30T13:52:21.055 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.055 INFO:tasks.workunit.client.0.smithi148.stdout:- Export make_export() { 2023-10-30T13:52:21.055 INFO:tasks.workunit.client.0.smithi148.stdout:- return Export(wanted_caps, issued(), pending()); 2023-10-30T13:52:21.055 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.055 INFO:tasks.workunit.client.0.smithi148.stdout:- void merge(Export& other) { 2023-10-30T13:52:21.055 INFO:tasks.workunit.client.0.smithi148.stdout:- // issued + pending 2023-10-30T13:52:21.056 INFO:tasks.workunit.client.0.smithi148.stdout:- int newpending = other.pending | pending(); 2023-10-30T13:52:21.056 INFO:tasks.workunit.client.0.smithi148.stdout:- if (other.issued & ~newpending) 2023-10-30T13:52:21.056 INFO:tasks.workunit.client.0.smithi148.stdout:- issue(other.issued | newpending); 2023-10-30T13:52:21.056 INFO:tasks.workunit.client.0.smithi148.stdout:- issue(newpending); 2023-10-30T13:52:21.056 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.056 INFO:tasks.workunit.client.0.smithi148.stdout:- // wanted 2023-10-30T13:52:21.057 INFO:tasks.workunit.client.0.smithi148.stdout:- wanted_caps = wanted_caps | other.wanted; 2023-10-30T13:52:21.057 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.057 INFO:tasks.workunit.client.0.smithi148.stdout:- void merge(int otherwanted, int otherissued) { 2023-10-30T13:52:21.057 INFO:tasks.workunit.client.0.smithi148.stdout:- // issued + pending 2023-10-30T13:52:21.057 INFO:tasks.workunit.client.0.smithi148.stdout:- int newpending = pending(); 2023-10-30T13:52:21.057 INFO:tasks.workunit.client.0.smithi148.stdout:- if (otherissued & ~newpending) 2023-10-30T13:52:21.058 INFO:tasks.workunit.client.0.smithi148.stdout:- issue(otherissued | newpending); 2023-10-30T13:52:21.058 INFO:tasks.workunit.client.0.smithi148.stdout:- issue(newpending); 2023-10-30T13:52:21.058 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.058 INFO:tasks.workunit.client.0.smithi148.stdout:- // wanted 2023-10-30T13:52:21.058 INFO:tasks.workunit.client.0.smithi148.stdout:- wanted_caps = wanted_caps | otherwanted; 2023-10-30T13:52:21.058 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.059 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.059 INFO:tasks.workunit.client.0.smithi148.stdout:- // confirm receipt of a previous sent/issued seq. 2023-10-30T13:52:21.059 INFO:tasks.workunit.client.0.smithi148.stdout:- int confirm_receipt(capseq_t seq, int caps) { 2023-10-30T13:52:21.059 INFO:tasks.workunit.client.0.smithi148.stdout:- int r = 0; 2023-10-30T13:52:21.059 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.059 INFO:tasks.workunit.client.0.smithi148.stdout:- // old seqs 2023-10-30T13:52:21.060 INFO:tasks.workunit.client.0.smithi148.stdout:- while (last_recv < seq) { 2023-10-30T13:52:21.060 INFO:tasks.workunit.client.0.smithi148.stdout:- generic_dout(10) << " cap.confirm_receipt forgetting seq " << last_recv << " " << cap_string(cap_history[last_recv]) << dendl; 2023-10-30T13:52:21.060 INFO:tasks.workunit.client.0.smithi148.stdout:- r |= cap_history[last_recv]; 2023-10-30T13:52:21.060 INFO:tasks.workunit.client.0.smithi148.stdout:- cap_history.erase(last_recv); 2023-10-30T13:52:21.060 INFO:tasks.workunit.client.0.smithi148.stdout:- ++last_recv; 2023-10-30T13:52:21.060 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.061 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.061 INFO:tasks.workunit.client.0.smithi148.stdout:- // release current? 2023-10-30T13:52:21.061 INFO:tasks.workunit.client.0.smithi148.stdout:- if (cap_history.count(seq) && 2023-10-30T13:52:21.061 INFO:tasks.workunit.client.0.smithi148.stdout:- cap_history[seq] != caps) { 2023-10-30T13:52:21.061 INFO:tasks.workunit.client.0.smithi148.stdout:- generic_dout(10) << " cap.confirm_receipt revising seq " << seq << " " << cap_string(cap_history[seq]) << " -> " << cap_string(caps) << dendl; 2023-10-30T13:52:21.061 INFO:tasks.workunit.client.0.smithi148.stdout:- // note what we're releasing.. 2023-10-30T13:52:21.062 INFO:tasks.workunit.client.0.smithi148.stdout:- assert(cap_history[seq] & ~caps); 2023-10-30T13:52:21.062 INFO:tasks.workunit.client.0.smithi148.stdout:- r |= cap_history[seq] & ~caps; 2023-10-30T13:52:21.062 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.062 INFO:tasks.workunit.client.0.smithi148.stdout:- cap_history[seq] = caps; // confirmed() now less than before.. 2023-10-30T13:52:21.062 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.062 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.063 INFO:tasks.workunit.client.0.smithi148.stdout:- // null? 2023-10-30T13:52:21.063 INFO:tasks.workunit.client.0.smithi148.stdout:- if (caps == 0 && 2023-10-30T13:52:21.063 INFO:tasks.workunit.client.0.smithi148.stdout:- cap_history.size() == 1 && 2023-10-30T13:52:21.063 INFO:tasks.workunit.client.0.smithi148.stdout:- cap_history.count(seq)) { 2023-10-30T13:52:21.063 INFO:tasks.workunit.client.0.smithi148.stdout:- cap_history.clear(); // viola, null! 2023-10-30T13:52:21.064 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.064 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.064 INFO:tasks.workunit.client.0.smithi148.stdout:- return r; 2023-10-30T13:52:21.064 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.064 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.064 INFO:tasks.workunit.client.0.smithi148.stdout:- void revoke() { 2023-10-30T13:52:21.065 INFO:tasks.workunit.client.0.smithi148.stdout:- if (pending()) 2023-10-30T13:52:21.065 INFO:tasks.workunit.client.0.smithi148.stdout:- issue(0); 2023-10-30T13:52:21.065 INFO:tasks.workunit.client.0.smithi148.stdout:- confirm_receipt(last_sent, 0); 2023-10-30T13:52:21.065 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.065 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.065 INFO:tasks.workunit.client.0.smithi148.stdout:- // serializers 2023-10-30T13:52:21.066 INFO:tasks.workunit.client.0.smithi148.stdout:- void _encode(bufferlist& bl) { 2023-10-30T13:52:21.066 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append((char*)&wanted_caps, sizeof(wanted_caps)); 2023-10-30T13:52:21.066 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append((char*)&last_sent, sizeof(last_sent)); 2023-10-30T13:52:21.066 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.append((char*)&last_recv, sizeof(last_recv)); 2023-10-30T13:52:21.066 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_encode(cap_history, bl); 2023-10-30T13:52:21.066 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.067 INFO:tasks.workunit.client.0.smithi148.stdout:- void _decode(bufferlist& bl, int& off) { 2023-10-30T13:52:21.067 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.copy(off, sizeof(wanted_caps), (char*)&wanted_caps); 2023-10-30T13:52:21.067 INFO:tasks.workunit.client.0.smithi148.stdout:- off += sizeof(wanted_caps); 2023-10-30T13:52:21.067 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.copy(off, sizeof(last_sent), (char*)&last_sent); 2023-10-30T13:52:21.067 INFO:tasks.workunit.client.0.smithi148.stdout:- off += sizeof(last_sent); 2023-10-30T13:52:21.068 INFO:tasks.workunit.client.0.smithi148.stdout:- bl.copy(off, sizeof(last_recv), (char*)&last_recv); 2023-10-30T13:52:21.068 INFO:tasks.workunit.client.0.smithi148.stdout:- off += sizeof(last_recv); 2023-10-30T13:52:21.068 INFO:tasks.workunit.client.0.smithi148.stdout:- ::_decode(cap_history, bl, off); 2023-10-30T13:52:21.068 INFO:tasks.workunit.client.0.smithi148.stdout:- } 2023-10-30T13:52:21.068 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.069 INFO:tasks.workunit.client.0.smithi148.stdout:-}; 2023-10-30T13:52:21.069 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.069 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.069 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.069 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.069 INFO:tasks.workunit.client.0.smithi148.stdout:- 2023-10-30T13:52:21.070 INFO:tasks.workunit.client.0.smithi148.stdout:-#endif 2023-10-30T13:52:21.070 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-10-30T13:52:21.070 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T13:52:21.900 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T13:52:21.963 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a8ef11cfd88a4a36879e10fdab92daed Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T13:52:21.966 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-10-30T13:52:21.975 INFO:tasks.mds_thrash:joining mds_thrasher 2023-10-30T13:52:21.978 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 2 2023-10-30T13:52:21.978 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-10-30T13:52:21.978 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 22 2023-10-30T13:52:21.979 INFO:tasks.mds_thrash:done joining 2023-10-30T13:52:21.979 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-30T13:52:21.993 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-30T13:52:21.993 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:52:21.993 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T13:52:22.005 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-30T13:52:22.006 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T13:52:22.058 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:52:22.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f243e6c9640 1 Processor -- start 2023-10-30T13:52:22.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f243e6c9640 1 -- start start 2023-10-30T13:52:22.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f243e6c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f24380c6bd0 0x7f243806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f243e6c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2438066cf0 con 0x7f2438071110 2023-10-30T13:52:22.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f243e6c9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24380c70d0 con 0x7f243806f7a0 2023-10-30T13:52:22.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f243e6c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24380700c0 con 0x7f24380c6bd0 2023-10-30T13:52:22.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f243cc66640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2438071110 0x7f2438158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41650/0 (socket says 172.21.15.148:41650) 2023-10-30T13:52:22.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f243cc66640 1 -- 172.21.15.148:0/3404448870 learned_addr learned my addr 172.21.15.148:0/3404448870 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:22.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f24377fe640 1 --2- 172.21.15.148:0/3404448870 >> v2:172.21.15.150:3300/0 conn(0x7f24380c6bd0 0x7f243806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.234+0000 7f2436ffd640 1 -- 172.21.15.148:0/3404448870 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652834534 0 0) 0x7f2438066cf0 con 0x7f2438071110 2023-10-30T13:52:22.242 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.238+0000 7f2436ffd640 1 -- 172.21.15.148:0/3404448870 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2420003530 con 0x7f2438071110 2023-10-30T13:52:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.238+0000 7f24377fe640 1 -- 172.21.15.148:0/3404448870 >> v1:172.21.15.150:6789/0 conn(0x7f243806f7a0 legacy=0x7f2438069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.238+0000 7f24377fe640 1 -- 172.21.15.148:0/3404448870 >> v1:172.21.15.148:6789/0 conn(0x7f2438071110 legacy=0x7f2438158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.238+0000 7f2436ffd640 1 -- 172.21.15.148:0/3404448870 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275522329 0 0) 0x7f2420003530 con 0x7f2438071110 2023-10-30T13:52:22.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.238+0000 7f24377fe640 1 -- 172.21.15.148:0/3404448870 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24380c7390 con 0x7f24380c6bd0 2023-10-30T13:52:22.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.238+0000 7f24377fe640 1 --2- 172.21.15.148:0/3404448870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24380c6bd0 0x7f243806fb80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f242c0097c0 tx=0x7f242c01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e023b23d0e8a74ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:22.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.238+0000 7f2436ffd640 1 -- 172.21.15.148:0/3404448870 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2430001a00 con 0x7f24380c6bd0 2023-10-30T13:52:22.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.238+0000 7f2436ffd640 1 -- 172.21.15.148:0/3404448870 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c0a8610 con 0x7f24380c6bd0 2023-10-30T13:52:22.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.238+0000 7f2436ffd640 1 -- 172.21.15.148:0/3404448870 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c0a89a0 con 0x7f24380c6bd0 2023-10-30T13:52:22.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 -- 172.21.15.148:0/3404448870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24380c6bd0 msgr2=0x7f243806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 --2- 172.21.15.148:0/3404448870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24380c6bd0 0x7f243806fb80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f242c0097c0 tx=0x7f242c01eb90 comp rx=0 tx=0).stop 2023-10-30T13:52:22.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 -- 172.21.15.148:0/3404448870 shutdown_connections 2023-10-30T13:52:22.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 --2- 172.21.15.148:0/3404448870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24380c6bd0 0x7f243806fb80 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 -- 172.21.15.148:0/3404448870 >> 172.21.15.148:0/3404448870 conn(0x7f24380c7910 msgr2=0x7f24380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:22.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 -- 172.21.15.148:0/3404448870 shutdown_connections 2023-10-30T13:52:22.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 -- 172.21.15.148:0/3404448870 wait complete. 2023-10-30T13:52:22.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 Processor -- start 2023-10-30T13:52:22.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 -- start start 2023-10-30T13:52:22.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f243806f7a0 0x7f2438152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2438071110 0x7f24380af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24380c6bd0 0x7f2438153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24381531e0 con 0x7f2438071110 2023-10-30T13:52:22.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2438152e60 con 0x7f243806f7a0 2023-10-30T13:52:22.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243e6c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24381e6480 con 0x7f24380c6bd0 2023-10-30T13:52:22.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f24377fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2438071110 0x7f24380af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f24377fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2438071110 0x7f24380af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56540/0 (socket says 172.21.15.148:56540) 2023-10-30T13:52:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f24377fe640 1 -- 172.21.15.148:0/2101928994 learned_addr learned my addr 172.21.15.148:0/2101928994 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f2437fff640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f243806f7a0 0x7f2438152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f243cc66640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24380c6bd0 0x7f2438153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f24377fe640 1 -- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24380c6bd0 msgr2=0x7f2438153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.252 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f24377fe640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24380c6bd0 0x7f2438153630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f24377fe640 1 -- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f243806f7a0 msgr2=0x7f2438152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f24377fe640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f243806f7a0 0x7f2438152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f24377fe640 1 -- 172.21.15.148:0/2101928994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c009420 con 0x7f2438071110 2023-10-30T13:52:22.253 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.242+0000 7f2437fff640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f243806f7a0 0x7f2438152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:22.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f24377fe640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2438071110 0x7f24380af6c0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f242c0b3040 tx=0x7f242c01b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:22.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f2434ff9640 1 -- 172.21.15.148:0/2101928994 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c0b0850 con 0x7f2438071110 2023-10-30T13:52:22.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f2434ff9640 1 -- 172.21.15.148:0/2101928994 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c0b09b0 con 0x7f2438071110 2023-10-30T13:52:22.254 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f2434ff9640 1 -- 172.21.15.148:0/2101928994 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c0b0d20 con 0x7f2438071110 2023-10-30T13:52:22.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2438158000 con 0x7f2438071110 2023-10-30T13:52:22.255 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24381585c0 con 0x7f2438071110 2023-10-30T13:52:22.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2438070610 con 0x7f2438071110 2023-10-30T13:52:22.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f2434ff9640 1 -- 172.21.15.148:0/2101928994 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f242c0b7040 con 0x7f2438071110 2023-10-30T13:52:22.256 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f2434ff9640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24240664a0 0x7f2424068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f2434ff9640 1 -- 172.21.15.148:0/2101928994 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f242c121470 con 0x7f2438071110 2023-10-30T13:52:22.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.246+0000 7f2437fff640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24240664a0 0x7f2424068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.250+0000 7f2437fff640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24240664a0 0x7f2424068960 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2428000f70 tx=0x7f2428009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:22.257 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.254+0000 7f2434ff9640 1 -- 172.21.15.148:0/2101928994 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2438070610 con 0x7f2438071110 2023-10-30T13:52:22.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.402+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2438070610 con 0x7f2438071110 2023-10-30T13:52:22.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f2434ff9640 1 -- 172.21.15.148:0/2101928994 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+19569 (secure 0 0 0) 0x7f2438070610 con 0x7f2438071110 2023-10-30T13:52:22.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:52:22.410 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":67,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:50:55.057709+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.148:6837/1530445724":"2023-10-31T13:50:55.057603+0000","172.21.15.148:6835/1228827025":"2023-10-31T13:49:20.023064+0000","172.21.15.150:6834/116502200":"2023-10-31T13:48:15.005684+0000","172.21.15.148:6833/1228827025":"2023-10-31T13:49:20.023064+0000","172.21.15.148:6837/3703944646":"2023-10-31T13:45:54.968896+0000","172.21.15.148:6835/3829165476":"2023-10-31T13:44:34.947959+0000","172.21.15.148:6833/3829165476":"2023-10-31T13:44:34.947959+0000","172.21.15.148:6836/500606168":"2023-10-31T13:42:29.894087+0000","172.21.15.148:6835/1079809213":"2023-10-31T13:39:54.856162+0000","172.21.15.148:6833/3194894584":"2023-10-31T13:24:29.547356+0000","172.21.15.148:6837/3232618434":"2023-10-31T13:33:54.741392+0000","172.21.15.148:6835/3194894584":"2023-10-31T13:24:29.547356+0000","172.21.15.150:6834/953984429":"2023-10-31T13:22:39.518196+0000","172.21.15.148:6835/3749586444":"2023-10-31T13:15:19.379857+0000","172.21.15.148:6836/3703944646":"2023-10-31T13:45:54.968896+0000","172.21.15.148:6834/4162935019":"2023-10-31T13:18:39.446214+0000","172.21.15.148:6832/4162935019":"2023-10-31T13:18:39.446214+0000","172.21.15.148:6837/500606168":"2023-10-31T13:42:29.894087+0000","172.21.15.148:6833/3749586444":"2023-10-31T13:15:19.379857+0000","172.21.15.150:6836/116502200":"2023-10-31T13:48:15.005684+0000","172.21.15.148:6837/3781833565":"2023-10-31T13:23:29.531899+0000","172.21.15.148:6833/1079809213":"2023-10-31T13:39:54.856162+0000","172.21.15.148:6836/1530445724":"2023-10-31T13:50:55.057603+0000","172.21.15.148:6837/1844602519":"2023-10-31T13:14:19.364086+0000","172.21.15.148:6836/1559416001":"2023-10-31T13:26:44.595428+0000","172.21.15.148:6836/3781833565":"2023-10-31T13:23:29.531899+0000","172.21.15.150:6834/1620877178":"2023-10-31T13:16:49.403499+0000","172.21.15.148:6837/2740075781":"2023-10-31T13:29:49.670604+0000","172.21.15.150:6836/953984429":"2023-10-31T13:22:39.518196+0000","172.21.15.148:6833/1386276781":"2023-10-31T13:35:44.780019+0000","172.21.15.150:6836/1620877178":"2023-10-31T13:16:49.403499+0000","172.21.15.148:6837/1559416001":"2023-10-31T13:26:44.595428+0000","172.21.15.148:6836/2740075781":"2023-10-31T13:29:49.670604+0000","172.21.15.148:6833/2733100982":"2023-10-31T13:28:14.633420+0000","172.21.15.148:6835/2733100982":"2023-10-31T13:28:14.633420+0000","172.21.15.148:6836/1844602519":"2023-10-31T13:14:19.364086+0000","172.21.15.148:6833/172087710":"2023-10-31T13:31:59.711325+0000","172.21.15.148:6835/172087710":"2023-10-31T13:31:59.711325+0000","172.21.15.148:6836/3232618434":"2023-10-31T13:33:54.741392+0000","172.21.15.148:6835/1386276781":"2023-10-31T13:35:44.780019+0000","172.21.15.148:6836/3144126860":"2023-10-31T13:37:34.820655+0000","172.21.15.148:6837/3144126860":"2023-10-31T13:37:34.820655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:52:22.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24240664a0 msgr2=0x7f2424068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24240664a0 0x7f2424068960 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2428000f70 tx=0x7f2428009040 comp rx=0 tx=0).stop 2023-10-30T13:52:22.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2438071110 msgr2=0x7f24380af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2438071110 0x7f24380af6c0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f242c0b3040 tx=0x7f242c01b930 comp rx=0 tx=0).stop 2023-10-30T13:52:22.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 shutdown_connections 2023-10-30T13:52:22.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f24240664a0 0x7f2424068960 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2438071110 0x7f24380af6c0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f24380c6bd0 0x7f2438153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 --2- 172.21.15.148:0/2101928994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f243806f7a0 0x7f2438152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 >> 172.21.15.148:0/2101928994 conn(0x7f24380c7910 msgr2=0x7f24380ca680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:22.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 shutdown_connections 2023-10-30T13:52:22.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.406+0000 7f243e6c9640 1 -- 172.21.15.148:0/2101928994 wait complete. 2023-10-30T13:52:22.431 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T13:10:22.349322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T13:10:32.611151+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T13:10:33.044931+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:52:22.432 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:52:22.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.566+0000 7f855cf65640 1 Processor -- start 2023-10-30T13:52:22.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.566+0000 7f855cf65640 1 -- start start 2023-10-30T13:52:22.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f855cf65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f855806f880 0x7f855806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f855cf65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f855806a7b0 0x7f8558161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f855cf65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85580c6cb0 con 0x7f85580c5810 2023-10-30T13:52:22.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f855cf65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8558071d90 con 0x7f855806a7b0 2023-10-30T13:52:22.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f855cf65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85581585d0 con 0x7f855806f880 2023-10-30T13:52:22.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f855659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f85580c5810 0x7f85580c5bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41662/0 (socket says 172.21.15.148:41662) 2023-10-30T13:52:22.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f855659c640 1 -- 172.21.15.148:0/942994112 learned_addr learned my addr 172.21.15.148:0/942994112 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:22.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8556d9d640 1 --2- 172.21.15.148:0/942994112 >> v2:172.21.15.150:3300/0 conn(0x7f855806a7b0 0x7f8558161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f854ffff640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f855806f880 0x7f855806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888278723 0 0) 0x7f85580c6cb0 con 0x7f85580c5810 2023-10-30T13:52:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8544003530 con 0x7f85580c5810 2023-10-30T13:52:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1133547448 0 0) 0x7f8544003530 con 0x7f85580c5810 2023-10-30T13:52:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f855806f880 msgr2=0x7f855806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:22.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f855806f880 0x7f855806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 >> v2:172.21.15.150:3300/0 conn(0x7f855806a7b0 msgr2=0x7f8558161f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:22.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 --2- 172.21.15.148:0/942994112 >> v2:172.21.15.150:3300/0 conn(0x7f855806a7b0 0x7f8558161f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580c6620 con 0x7f85580c5810 2023-10-30T13:52:22.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f8548001800 con 0x7f85580c5810 2023-10-30T13:52:22.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 >> v1:172.21.15.148:6789/0 conn(0x7f85580c5810 legacy=0x7f85580c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f855806f880 0x7f85440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f85440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544004d10 0x7f8544005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85581585d0 con 0x7f8544003f30 2023-10-30T13:52:22.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854400ba40 con 0x7f8544004d10 2023-10-30T13:52:22.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854400bc00 con 0x7f855806f880 2023-10-30T13:52:22.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f854ffff640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f85440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f855659c640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f855806f880 0x7f85440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8556d9d640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544004d10 0x7f8544005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f854ffff640 1 -- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f855806f880 msgr2=0x7f85440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:22.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f854ffff640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f855806f880 0x7f85440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f854ffff640 1 -- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544004d10 msgr2=0x7f8544005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:22.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f854ffff640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544004d10 0x7f8544005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f854ffff640 1 -- 172.21.15.148:0/942994112 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8544003530 con 0x7f8544003f30 2023-10-30T13:52:22.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.570+0000 7f8556d9d640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8544004d10 0x7f8544005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:22.585 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f854ffff640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f85440047d0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f8540009ae0 tx=0x7f854001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f8357a8b1ed09f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:22.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400a6430 con 0x7f8544003f30 2023-10-30T13:52:22.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85400a6590 con 0x7f8544003f30 2023-10-30T13:52:22.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f8555d9b640 1 -- 172.21.15.148:0/942994112 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400a6940 con 0x7f8544003f30 2023-10-30T13:52:22.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f855cf65640 1 -- 172.21.15.148:0/942994112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 msgr2=0x7f85440047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f855cf65640 1 --2- 172.21.15.148:0/942994112 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f85440047d0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f8540009ae0 tx=0x7f854001ca50 comp rx=0 tx=0).stop 2023-10-30T13:52:22.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f8556d9d640 1 -- 172.21.15.148:0/942994112 reap_dead start 2023-10-30T13:52:22.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f855cf65640 1 -- 172.21.15.148:0/942994112 shutdown_connections 2023-10-30T13:52:22.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f855cf65640 1 -- 172.21.15.148:0/942994112 >> 172.21.15.148:0/942994112 conn(0x7f85580c7910 msgr2=0x7f85580c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:22.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f855cf65640 1 -- 172.21.15.148:0/942994112 shutdown_connections 2023-10-30T13:52:22.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f855cf65640 1 -- 172.21.15.148:0/942994112 wait complete. 2023-10-30T13:52:22.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f855cf65640 1 Processor -- start 2023-10-30T13:52:22.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.574+0000 7f855cf65640 1 -- start start 2023-10-30T13:52:22.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855cf65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f8558161db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855cf65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85581612c0 0x7f85581f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855cf65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8558161700 0x7f855814d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855cf65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85580c30b0 con 0x7f8544003f30 2023-10-30T13:52:22.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855cf65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85580c3390 con 0x7f8558161700 2023-10-30T13:52:22.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855cf65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85580c3670 con 0x7f85581612c0 2023-10-30T13:52:22.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f8558161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f8558161db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56562/0 (socket says 172.21.15.148:56562) 2023-10-30T13:52:22.590 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855659c640 1 -- 172.21.15.148:0/102548018 learned_addr learned my addr 172.21.15.148:0/102548018 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:22.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f8556d9d640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8558161700 0x7f855814d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f854ffff640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85581612c0 0x7f85581f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855659c640 1 -- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85581612c0 msgr2=0x7f85581f2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.591 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855659c640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85581612c0 0x7f85581f2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855659c640 1 -- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8558161700 msgr2=0x7f855814d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855659c640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8558161700 0x7f855814d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855659c640 1 -- 172.21.15.148:0/102548018 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580c3950 con 0x7f8544003f30 2023-10-30T13:52:22.592 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f8556d9d640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8558161700 0x7f855814d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:52:22.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855659c640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f8558161db0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f854800f840 tx=0x7f854800fd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:22.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f854effd640 1 -- 172.21.15.148:0/102548018 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85480072e0 con 0x7f8544003f30 2023-10-30T13:52:22.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f854effd640 1 -- 172.21.15.148:0/102548018 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8548007440 con 0x7f8544003f30 2023-10-30T13:52:22.593 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85581f2700 con 0x7f8544003f30 2023-10-30T13:52:22.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f854effd640 1 -- 172.21.15.148:0/102548018 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85480057b0 con 0x7f8544003f30 2023-10-30T13:52:22.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8558155270 con 0x7f8544003f30 2023-10-30T13:52:22.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85580c75d0 con 0x7f8544003f30 2023-10-30T13:52:22.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.578+0000 7f854effd640 1 -- 172.21.15.148:0/102548018 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8548005910 con 0x7f8544003f30 2023-10-30T13:52:22.594 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.582+0000 7f854effd640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f85380664a0 0x7f8538068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.582+0000 7f854effd640 1 -- 172.21.15.148:0/102548018 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f8548084810 con 0x7f8544003f30 2023-10-30T13:52:22.595 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.582+0000 7f854ffff640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f85380664a0 0x7f8538068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.582+0000 7f854ffff640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f85380664a0 0x7f8538068960 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f85400ad000 tx=0x7f85400a9040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:22.596 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.586+0000 7f854effd640 1 -- 172.21.15.148:0/102548018 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85480513c0 con 0x7f8544003f30 2023-10-30T13:52:22.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f855804e170 con 0x7f8544003f30 2023-10-30T13:52:22.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f854effd640 1 -- 172.21.15.148:0/102548018 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7f85480549e0 con 0x7f8544003f30 2023-10-30T13:52:22.743 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2023-10-30T13:52:22.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f85380664a0 msgr2=0x7f8538068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f85380664a0 0x7f8538068960 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f85400ad000 tx=0x7f85400a9040 comp rx=0 tx=0).stop 2023-10-30T13:52:22.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 msgr2=0x7f8558161db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f8558161db0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f854800f840 tx=0x7f854800fd00 comp rx=0 tx=0).stop 2023-10-30T13:52:22.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 shutdown_connections 2023-10-30T13:52:22.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f85380664a0 0x7f8538068960 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8558161700 0x7f855814d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f85581612c0 0x7f85581f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 --2- 172.21.15.148:0/102548018 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8544003f30 0x7f8558161db0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 >> 172.21.15.148:0/102548018 conn(0x7f85580c7910 msgr2=0x7f855814e950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 shutdown_connections 2023-10-30T13:52:22.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.738+0000 7f855cf65640 1 -- 172.21.15.148:0/102548018 wait complete. 2023-10-30T13:52:22.764 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:52:22.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.898+0000 7f42d1165640 1 Processor -- start 2023-10-30T13:52:22.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.898+0000 7f42d1165640 1 -- start start 2023-10-30T13:52:22.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42d1165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c5810 0x7f42cc0c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42d1165640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f42cc0c6130 0x7f42cc06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42d1165640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc0c6cb0 con 0x7f42cc06f880 2023-10-30T13:52:22.906 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42d1165640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42cc071d90 con 0x7f42cc0c6130 2023-10-30T13:52:22.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42d1165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc1585d0 con 0x7f42cc0c5810 2023-10-30T13:52:22.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42c3fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f42cc0c6130 0x7f42cc06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42c3fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f42cc0c6130 0x7f42cc06a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56578/0 (socket says 172.21.15.148:56578) 2023-10-30T13:52:22.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42c3fff640 1 -- 172.21.15.148:0/457015770 learned_addr learned my addr 172.21.15.148:0/457015770 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:22.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42cad9d640 1 --2- 172.21.15.148:0/457015770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c5810 0x7f42cc0c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42c3fff640 1 -- 172.21.15.148:0/457015770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c5810 msgr2=0x7f42cc0c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:22.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42c3fff640 1 --2- 172.21.15.148:0/457015770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c5810 0x7f42cc0c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42c3fff640 1 -- 172.21.15.148:0/457015770 >> v1:172.21.15.150:6789/0 conn(0x7f42cc06f880 legacy=0x7f42cc15ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.902+0000 7f42c3fff640 1 -- 172.21.15.148:0/457015770 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc0c6620 con 0x7f42cc0c6130 2023-10-30T13:52:22.910 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42c3fff640 1 --2- 172.21.15.148:0/457015770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c6130 0x7f42cc06a270 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f42b40098a0 tx=0x7f42b401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2b83c4e90451ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:22.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42ca59c640 1 -- 172.21.15.148:0/457015770 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40ac640 con 0x7f42cc0c6130 2023-10-30T13:52:22.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42ca59c640 1 -- 172.21.15.148:0/457015770 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b40ac7a0 con 0x7f42cc0c6130 2023-10-30T13:52:22.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42ca59c640 1 -- 172.21.15.148:0/457015770 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40acb30 con 0x7f42cc0c6130 2023-10-30T13:52:22.911 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 -- 172.21.15.148:0/457015770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c6130 msgr2=0x7f42cc06a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 --2- 172.21.15.148:0/457015770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c6130 0x7f42cc06a270 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f42b40098a0 tx=0x7f42b401c400 comp rx=0 tx=0).stop 2023-10-30T13:52:22.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 -- 172.21.15.148:0/457015770 shutdown_connections 2023-10-30T13:52:22.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 --2- 172.21.15.148:0/457015770 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c6130 0x7f42cc06a270 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 --2- 172.21.15.148:0/457015770 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c5810 0x7f42cc0c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.912 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 -- 172.21.15.148:0/457015770 >> 172.21.15.148:0/457015770 conn(0x7f42cc0c7910 msgr2=0x7f42cc0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:22.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 -- 172.21.15.148:0/457015770 shutdown_connections 2023-10-30T13:52:22.913 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 -- 172.21.15.148:0/457015770 wait complete. 2023-10-30T13:52:22.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 Processor -- start 2023-10-30T13:52:22.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.906+0000 7f42d1165640 1 -- start start 2023-10-30T13:52:22.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42d1165640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42cc06f880 0x7f42cc158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.915 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42d1165640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c5810 0x7f42cc1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42d1165640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c6130 0x7f42cc14e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42d1165640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc1f2c60 con 0x7f42cc0c5810 2023-10-30T13:52:22.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42d1165640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc1f2f00 con 0x7f42cc06f880 2023-10-30T13:52:22.916 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42d1165640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc0c8df0 con 0x7f42cc0c6130 2023-10-30T13:52:22.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c5810 0x7f42cc1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c5810 0x7f42cc1591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56580/0 (socket says 172.21.15.148:56580) 2023-10-30T13:52:22.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c3fff640 1 -- 172.21.15.148:0/1300149452 learned_addr learned my addr 172.21.15.148:0/1300149452 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:22.917 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42cad9d640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42cc06f880 0x7f42cc158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42cb59e640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c6130 0x7f42cc14e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c3fff640 1 -- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c6130 msgr2=0x7f42cc14e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c3fff640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c6130 0x7f42cc14e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.918 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c3fff640 1 -- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42cc06f880 msgr2=0x7f42cc158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:22.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c3fff640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42cc06f880 0x7f42cc158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:22.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c3fff640 1 -- 172.21.15.148:0/1300149452 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b40094d0 con 0x7f42cc0c5810 2023-10-30T13:52:22.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c3fff640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c5810 0x7f42cc1591b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f42b40af040 tx=0x7f42b40191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:22.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c37fe640 1 -- 172.21.15.148:0/1300149452 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40b9070 con 0x7f42cc0c5810 2023-10-30T13:52:22.919 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c37fe640 1 -- 172.21.15.148:0/1300149452 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b40ac600 con 0x7f42cc0c5810 2023-10-30T13:52:22.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c37fe640 1 -- 172.21.15.148:0/1300149452 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b40b47f0 con 0x7f42cc0c5810 2023-10-30T13:52:22.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42cc0c90d0 con 0x7f42cc0c5810 2023-10-30T13:52:22.920 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42cc158750 con 0x7f42cc0c5810 2023-10-30T13:52:22.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42cc158940 con 0x7f42cc0c5810 2023-10-30T13:52:22.921 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.910+0000 7f42c37fe640 1 -- 172.21.15.148:0/1300149452 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42b40b3070 con 0x7f42cc0c5810 2023-10-30T13:52:22.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.914+0000 7f42c37fe640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42ac0664a0 0x7f42ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:22.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.914+0000 7f42c37fe640 1 -- 172.21.15.148:0/1300149452 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f42b411db40 con 0x7f42cc0c5810 2023-10-30T13:52:22.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.914+0000 7f42cad9d640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42ac0664a0 0x7f42ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:22.922 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.914+0000 7f42cad9d640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42ac0664a0 0x7f42ac068960 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f42bc006fd0 tx=0x7f42bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:22.923 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:22.918+0000 7f42c37fe640 1 -- 172.21.15.148:0/1300149452 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42b40ea9d0 con 0x7f42cc0c5810 2023-10-30T13:52:23.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.062+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42cc0c75d0 con 0x7f42cc0c5810 2023-10-30T13:52:23.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42c37fe640 1 -- 172.21.15.148:0/1300149452 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) v1 ==== 104+0+11 (secure 0 0 0) 0x7f42b40edff0 con 0x7f42cc0c5810 2023-10-30T13:52:23.069 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 32 2023-10-30T13:52:23.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42ac0664a0 msgr2=0x7f42ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42ac0664a0 0x7f42ac068960 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f42bc006fd0 tx=0x7f42bc008040 comp rx=0 tx=0).stop 2023-10-30T13:52:23.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c5810 msgr2=0x7f42cc1591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c5810 0x7f42cc1591b0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f42b40af040 tx=0x7f42b40191a0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 shutdown_connections 2023-10-30T13:52:23.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f42ac0664a0 0x7f42ac068960 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42cc0c5810 0x7f42cc1591b0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f42cc0c6130 0x7f42cc14e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 --2- 172.21.15.148:0/1300149452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42cc06f880 0x7f42cc158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 >> 172.21.15.148:0/1300149452 conn(0x7f42cc0c7910 msgr2=0x7f42cc0c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:23.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 shutdown_connections 2023-10-30T13:52:23.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.066+0000 7f42d1165640 1 -- 172.21.15.148:0/1300149452 wait complete. 2023-10-30T13:52:23.090 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:52:23.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.226+0000 7fc2566ad640 1 Processor -- start 2023-10-30T13:52:23.232 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.226+0000 7fc2566ad640 1 -- start start 2023-10-30T13:52:23.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc2566ad640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc250153840 0x7fc25015a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc2566ad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153c20 0x7fc2500c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc2566ad640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25015a6b0 con 0x7fc2500cb4c0 2023-10-30T13:52:23.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc2566ad640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc25015dbc0 con 0x7fc250153840 2023-10-30T13:52:23.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc2566ad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25015d730 con 0x7fc250153c20 2023-10-30T13:52:23.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc254eaa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153c20 0x7fc2500c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc2556ab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc250153840 0x7fc25015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc254eaa640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153c20 0x7fc2500c3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:56608/0 (socket says 172.21.15.148:56608) 2023-10-30T13:52:23.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc254eaa640 1 -- 172.21.15.148:0/2726258072 learned_addr learned my addr 172.21.15.148:0/2726258072 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:23.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc254eaa640 1 -- 172.21.15.148:0/2726258072 >> v2:172.21.15.150:3300/0 conn(0x7fc250153840 msgr2=0x7fc25015a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:23.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc254eaa640 1 --2- 172.21.15.148:0/2726258072 >> v2:172.21.15.150:3300/0 conn(0x7fc250153840 0x7fc25015a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.236 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc254eaa640 1 -- 172.21.15.148:0/2726258072 >> v1:172.21.15.150:6789/0 conn(0x7fc2500cb4c0 legacy=0x7fc2500cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.230+0000 7fc254eaa640 1 -- 172.21.15.148:0/2726258072 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2500ca8f0 con 0x7fc250153c20 2023-10-30T13:52:23.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc254eaa640 1 --2- 172.21.15.148:0/2726258072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153c20 0x7fc2500c3eb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc240009870 tx=0x7fc24001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee8e718dd955c0e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:23.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc247fff640 1 -- 172.21.15.148:0/2726258072 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2400a8a30 con 0x7fc250153c20 2023-10-30T13:52:23.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc247fff640 1 -- 172.21.15.148:0/2726258072 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2400a8b90 con 0x7fc250153c20 2023-10-30T13:52:23.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc247fff640 1 -- 172.21.15.148:0/2726258072 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2400b0440 con 0x7fc250153c20 2023-10-30T13:52:23.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 -- 172.21.15.148:0/2726258072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153c20 msgr2=0x7fc2500c3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 --2- 172.21.15.148:0/2726258072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153c20 0x7fc2500c3eb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc240009870 tx=0x7fc24001ed30 comp rx=0 tx=0).stop 2023-10-30T13:52:23.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 -- 172.21.15.148:0/2726258072 shutdown_connections 2023-10-30T13:52:23.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 --2- 172.21.15.148:0/2726258072 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153c20 0x7fc2500c3eb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 --2- 172.21.15.148:0/2726258072 >> v2:172.21.15.150:3300/0 conn(0x7fc250153840 0x7fc25015a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 -- 172.21.15.148:0/2726258072 >> 172.21.15.148:0/2726258072 conn(0x7fc25006be40 msgr2=0x7fc25006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:23.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 -- 172.21.15.148:0/2726258072 shutdown_connections 2023-10-30T13:52:23.241 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 -- 172.21.15.148:0/2726258072 wait complete. 2023-10-30T13:52:23.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 Processor -- start 2023-10-30T13:52:23.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.234+0000 7fc2566ad640 1 -- start start 2023-10-30T13:52:23.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2566ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2500cb4c0 0x7fc250159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.243 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2566ad640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153840 0x7fc25014ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2566ad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc250153c20 0x7fc25014cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2566ad640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2501590a0 con 0x7fc250153c20 2023-10-30T13:52:23.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2566ad640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2501585b0 con 0x7fc2500cb4c0 2023-10-30T13:52:23.244 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2566ad640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2501ef5e0 con 0x7fc250153840 2023-10-30T13:52:23.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc255eac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc250153c20 0x7fc25014cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc255eac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc250153c20 0x7fc25014cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56596/0 (socket says 172.21.15.148:56596) 2023-10-30T13:52:23.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc255eac640 1 -- 172.21.15.148:0/696465186 learned_addr learned my addr 172.21.15.148:0/696465186 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:23.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2556ab640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2500cb4c0 0x7fc250159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.245 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc254eaa640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153840 0x7fc25014ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc255eac640 1 -- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153840 msgr2=0x7fc25014ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc255eac640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153840 0x7fc25014ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc255eac640 1 -- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2500cb4c0 msgr2=0x7fc250159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.246 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc255eac640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2500cb4c0 0x7fc250159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc255eac640 1 -- 172.21.15.148:0/696465186 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2400094d0 con 0x7fc250153c20 2023-10-30T13:52:23.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2556ab640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2500cb4c0 0x7fc250159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:23.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc255eac640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc250153c20 0x7fc25014cf90 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fc24800c7a0 tx=0x7fc24800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:23.247 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc245ffb640 1 -- 172.21.15.148:0/696465186 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc248003b80 con 0x7fc250153c20 2023-10-30T13:52:23.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc245ffb640 1 -- 172.21.15.148:0/696465186 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc248003ce0 con 0x7fc250153c20 2023-10-30T13:52:23.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc250158800 con 0x7fc250153c20 2023-10-30T13:52:23.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc245ffb640 1 -- 172.21.15.148:0/696465186 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24800aa90 con 0x7fc250153c20 2023-10-30T13:52:23.248 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc250158dc0 con 0x7fc250153c20 2023-10-30T13:52:23.249 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25004e9c0 con 0x7fc250153c20 2023-10-30T13:52:23.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.238+0000 7fc245ffb640 1 -- 172.21.15.148:0/696465186 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc24800abf0 con 0x7fc250153c20 2023-10-30T13:52:23.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.242+0000 7fc245ffb640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc2340664d0 0x7fc234068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.242+0000 7fc245ffb640 1 -- 172.21.15.148:0/696465186 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fc248084360 con 0x7fc250153c20 2023-10-30T13:52:23.250 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.242+0000 7fc2556ab640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc2340664d0 0x7fc234068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.242+0000 7fc2556ab640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc2340664d0 0x7fc234068990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009bb0 tx=0x7fc24c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:23.251 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.246+0000 7fc245ffb640 1 -- 172.21.15.148:0/696465186 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc248050f10 con 0x7fc250153c20 2023-10-30T13:52:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.394+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc25015a6b0 con 0x7fc250153c20 2023-10-30T13:52:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.394+0000 7fc245ffb640 1 -- 172.21.15.148:0/696465186 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc25015a6b0 con 0x7fc250153c20 2023-10-30T13:52:23.400 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 64 2023-10-30T13:52:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc2340664d0 msgr2=0x7fc234068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc2340664d0 0x7fc234068990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009bb0 tx=0x7fc24c008040 comp rx=0 tx=0).stop 2023-10-30T13:52:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc250153c20 msgr2=0x7fc25014cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc250153c20 0x7fc25014cf90 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fc24800c7a0 tx=0x7fc24800cc60 comp rx=0 tx=0).stop 2023-10-30T13:52:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 shutdown_connections 2023-10-30T13:52:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fc2340664d0 0x7fc234068990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc250153c20 0x7fc25014cf90 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fc250153840 0x7fc25014ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 --2- 172.21.15.148:0/696465186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2500cb4c0 0x7fc250159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 >> 172.21.15.148:0/696465186 conn(0x7fc25006be40 msgr2=0x7fc2500b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 shutdown_connections 2023-10-30T13:52:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.398+0000 7fc2566ad640 1 -- 172.21.15.148:0/696465186 wait complete. 2023-10-30T13:52:23.421 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-30T13:52:23.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f653550f640 1 Processor -- start 2023-10-30T13:52:23.561 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f653550f640 1 -- start start 2023-10-30T13:52:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f653550f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65300c74e0 0x7f6530155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f653550f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f653006f540 0x7f6530156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.563 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f653550f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65300c0060 con 0x7f653014fc00 2023-10-30T13:52:23.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f653550f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65301566c0 con 0x7f653006f540 2023-10-30T13:52:23.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f653550f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65300c0340 con 0x7f65300c74e0 2023-10-30T13:52:23.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f652f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f653006f540 0x7f6530156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f652f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f653006f540 0x7f6530156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56612/0 (socket says 172.21.15.148:56612) 2023-10-30T13:52:23.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f652f7fe640 1 -- 172.21.15.148:0/1003338455 learned_addr learned my addr 172.21.15.148:0/1003338455 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:23.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f6527fff640 1 --2- 172.21.15.148:0/1003338455 >> v2:172.21.15.150:3300/0 conn(0x7f65300c74e0 0x7f6530155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f652f7fe640 1 -- 172.21.15.148:0/1003338455 >> v2:172.21.15.150:3300/0 conn(0x7f65300c74e0 msgr2=0x7f6530155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:23.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f652f7fe640 1 --2- 172.21.15.148:0/1003338455 >> v2:172.21.15.150:3300/0 conn(0x7f65300c74e0 0x7f6530155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.565 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f652f7fe640 1 -- 172.21.15.148:0/1003338455 >> v1:172.21.15.150:6789/0 conn(0x7f653014fc00 legacy=0x7f653006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:23.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f652f7fe640 1 -- 172.21.15.148:0/1003338455 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6530071090 con 0x7f653006f540 2023-10-30T13:52:23.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.558+0000 7f6527fff640 1 --2- 172.21.15.148:0/1003338455 >> v2:172.21.15.150:3300/0 conn(0x7f65300c74e0 0x7f6530155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:23.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f652f7fe640 1 --2- 172.21.15.148:0/1003338455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 0x7f6530156180 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f65300c8660 tx=0x7f6528020290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f327fbd33f6fd131 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:23.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f652e7fc640 1 -- 172.21.15.148:0/1003338455 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65280b0400 con 0x7f653006f540 2023-10-30T13:52:23.567 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f652e7fc640 1 -- 172.21.15.148:0/1003338455 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65280ac050 con 0x7f653006f540 2023-10-30T13:52:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f652e7fc640 1 -- 172.21.15.148:0/1003338455 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65280b0780 con 0x7f653006f540 2023-10-30T13:52:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 -- 172.21.15.148:0/1003338455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 msgr2=0x7f6530156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 --2- 172.21.15.148:0/1003338455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 0x7f6530156180 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f65300c8660 tx=0x7f6528020290 comp rx=0 tx=0).stop 2023-10-30T13:52:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 -- 172.21.15.148:0/1003338455 shutdown_connections 2023-10-30T13:52:23.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 --2- 172.21.15.148:0/1003338455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 0x7f6530156180 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 --2- 172.21.15.148:0/1003338455 >> v2:172.21.15.150:3300/0 conn(0x7f65300c74e0 0x7f6530155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 -- 172.21.15.148:0/1003338455 >> 172.21.15.148:0/1003338455 conn(0x7f65300c7910 msgr2=0x7f65300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:23.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 -- 172.21.15.148:0/1003338455 shutdown_connections 2023-10-30T13:52:23.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 -- 172.21.15.148:0/1003338455 wait complete. 2023-10-30T13:52:23.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 Processor -- start 2023-10-30T13:52:23.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.562+0000 7f653550f640 1 -- start start 2023-10-30T13:52:23.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f653550f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 0x7f65300ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.572 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f653550f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65300c74e0 0x7f65301e43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f653550f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f653014fc00 0x7f65301e48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f653550f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6530154490 con 0x7f653006f540 2023-10-30T13:52:23.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f653550f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6530154fa0 con 0x7f653014fc00 2023-10-30T13:52:23.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f653550f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6530155bc0 con 0x7f65300c74e0 2023-10-30T13:52:23.573 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 0x7f65300ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 0x7f65300ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56624/0 (socket says 172.21.15.148:56624) 2023-10-30T13:52:23.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652effd640 1 -- 172.21.15.148:0/2206131250 learned_addr learned my addr 172.21.15.148:0/2206131250 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:23.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652f7fe640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f653014fc00 0x7f65301e48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.574 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f6527fff640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65300c74e0 0x7f65301e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652effd640 1 -- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65300c74e0 msgr2=0x7f65301e43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652effd640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65300c74e0 0x7f65301e43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652effd640 1 -- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f653014fc00 msgr2=0x7f65301e48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.575 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652effd640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f653014fc00 0x7f65301e48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652effd640 1 -- 172.21.15.148:0/2206131250 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6528009b30 con 0x7f653006f540 2023-10-30T13:52:23.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652f7fe640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f653014fc00 0x7f65301e48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:23.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f652effd640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 0x7f65300ca410 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f652000b410 tx=0x7f652000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:23.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f65277fe640 1 -- 172.21.15.148:0/2206131250 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6520004020 con 0x7f653006f540 2023-10-30T13:52:23.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f65277fe640 1 -- 172.21.15.148:0/2206131250 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6520004180 con 0x7f653006f540 2023-10-30T13:52:23.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6530154710 con 0x7f653006f540 2023-10-30T13:52:23.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f65277fe640 1 -- 172.21.15.148:0/2206131250 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6520011630 con 0x7f653006f540 2023-10-30T13:52:23.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653014db80 con 0x7f653006f540 2023-10-30T13:52:23.578 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.566+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65300708c0 con 0x7f653006f540 2023-10-30T13:52:23.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.570+0000 7f65277fe640 1 -- 172.21.15.148:0/2206131250 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6520011790 con 0x7f653006f540 2023-10-30T13:52:23.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.570+0000 7f65277fe640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65100664a0 0x7f6510068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:23.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.570+0000 7f65277fe640 1 -- 172.21.15.148:0/2206131250 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f6520084430 con 0x7f653006f540 2023-10-30T13:52:23.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.570+0000 7f6527fff640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65100664a0 0x7f6510068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:23.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.570+0000 7f6527fff640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65100664a0 0x7f6510068960 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f653014cc10 tx=0x7f6518008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:23.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.574+0000 7f65277fe640 1 -- 172.21.15.148:0/2206131250 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6520051060 con 0x7f653006f540 2023-10-30T13:52:23.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6530154c30 con 0x7f653006f540 2023-10-30T13:52:23.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f65277fe640 1 -- 172.21.15.148:0/2206131250 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 42 entries v67) v1 ==== 81+0+2596 (secure 0 0 0) 0x7f6520015a40 con 0x7f653006f540 2023-10-30T13:52:23.737 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6837/1530445724 2023-10-31T13:50:55.057603+0000 2023-10-30T13:52:23.738 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6835/1228827025 2023-10-31T13:49:20.023064+0000 2023-10-30T13:52:23.738 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.150:6834/116502200 2023-10-31T13:48:15.005684+0000 2023-10-30T13:52:23.738 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6833/1228827025 2023-10-31T13:49:20.023064+0000 2023-10-30T13:52:23.738 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6837/3703944646 2023-10-31T13:45:54.968896+0000 2023-10-30T13:52:23.739 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6835/3829165476 2023-10-31T13:44:34.947959+0000 2023-10-30T13:52:23.739 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6833/3829165476 2023-10-31T13:44:34.947959+0000 2023-10-30T13:52:23.739 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6836/500606168 2023-10-31T13:42:29.894087+0000 2023-10-30T13:52:23.739 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6835/1079809213 2023-10-31T13:39:54.856162+0000 2023-10-30T13:52:23.740 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6833/3194894584 2023-10-31T13:24:29.547356+0000 2023-10-30T13:52:23.740 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6837/3232618434 2023-10-31T13:33:54.741392+0000 2023-10-30T13:52:23.740 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6835/3194894584 2023-10-31T13:24:29.547356+0000 2023-10-30T13:52:23.740 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.150:6834/953984429 2023-10-31T13:22:39.518196+0000 2023-10-30T13:52:23.741 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6835/3749586444 2023-10-31T13:15:19.379857+0000 2023-10-30T13:52:23.741 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6836/3703944646 2023-10-31T13:45:54.968896+0000 2023-10-30T13:52:23.741 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6834/4162935019 2023-10-31T13:18:39.446214+0000 2023-10-30T13:52:23.741 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6832/4162935019 2023-10-31T13:18:39.446214+0000 2023-10-30T13:52:23.742 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6837/500606168 2023-10-31T13:42:29.894087+0000 2023-10-30T13:52:23.742 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6833/3749586444 2023-10-31T13:15:19.379857+0000 2023-10-30T13:52:23.742 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.150:6836/116502200 2023-10-31T13:48:15.005684+0000 2023-10-30T13:52:23.742 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6837/3781833565 2023-10-31T13:23:29.531899+0000 2023-10-30T13:52:23.742 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6833/1079809213 2023-10-31T13:39:54.856162+0000 2023-10-30T13:52:23.743 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6836/1530445724 2023-10-31T13:50:55.057603+0000 2023-10-30T13:52:23.743 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6837/1844602519 2023-10-31T13:14:19.364086+0000 2023-10-30T13:52:23.743 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6836/1559416001 2023-10-31T13:26:44.595428+0000 2023-10-30T13:52:23.743 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6836/3781833565 2023-10-31T13:23:29.531899+0000 2023-10-30T13:52:23.744 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.150:6834/1620877178 2023-10-31T13:16:49.403499+0000 2023-10-30T13:52:23.744 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6837/2740075781 2023-10-31T13:29:49.670604+0000 2023-10-30T13:52:23.744 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.150:6836/953984429 2023-10-31T13:22:39.518196+0000 2023-10-30T13:52:23.744 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6833/1386276781 2023-10-31T13:35:44.780019+0000 2023-10-30T13:52:23.745 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.150:6836/1620877178 2023-10-31T13:16:49.403499+0000 2023-10-30T13:52:23.745 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6837/1559416001 2023-10-31T13:26:44.595428+0000 2023-10-30T13:52:23.745 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6836/2740075781 2023-10-31T13:29:49.670604+0000 2023-10-30T13:52:23.745 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6833/2733100982 2023-10-31T13:28:14.633420+0000 2023-10-30T13:52:23.746 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6835/2733100982 2023-10-31T13:28:14.633420+0000 2023-10-30T13:52:23.746 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6836/1844602519 2023-10-31T13:14:19.364086+0000 2023-10-30T13:52:23.746 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6833/172087710 2023-10-31T13:31:59.711325+0000 2023-10-30T13:52:23.746 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6835/172087710 2023-10-31T13:31:59.711325+0000 2023-10-30T13:52:23.746 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6836/3232618434 2023-10-31T13:33:54.741392+0000 2023-10-30T13:52:23.747 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6835/1386276781 2023-10-31T13:35:44.780019+0000 2023-10-30T13:52:23.749 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6836/3144126860 2023-10-31T13:37:34.820655+0000 2023-10-30T13:52:23.749 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148:6837/3144126860 2023-10-31T13:37:34.820655+0000 2023-10-30T13:52:23.750 INFO:teuthology.orchestra.run.smithi148.stderr:listed 42 entries 2023-10-30T13:52:23.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65100664a0 msgr2=0x7f6510068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65100664a0 0x7f6510068960 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f653014cc10 tx=0x7f6518008040 comp rx=0 tx=0).stop 2023-10-30T13:52:23.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 msgr2=0x7f65300ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:23.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 0x7f65300ca410 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f652000b410 tx=0x7f652000b8d0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 shutdown_connections 2023-10-30T13:52:23.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f65100664a0 0x7f6510068960 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f653014fc00 0x7f65301e48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f65300c74e0 0x7f65301e43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 --2- 172.21.15.148:0/2206131250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653006f540 0x7f65300ca410 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:23.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 >> 172.21.15.148:0/2206131250 conn(0x7f65300c7910 msgr2=0x7f653014eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:23.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 shutdown_connections 2023-10-30T13:52:23.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:23.734+0000 7f653550f640 1 -- 172.21.15.148:0/2206131250 wait complete. 2023-10-30T13:52:23.758 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-30T13:52:23.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:52:23.759 DEBUG:teuthology.orchestra.run.smithi148:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-30T13:52:29.593 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:52:29.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:52:29.594 DEBUG:teuthology.orchestra.run.smithi148:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-30T13:52:29.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:52:29.604 DEBUG:teuthology.orchestra.run.smithi148:> ip netns list 2023-10-30T13:52:29.653 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-30T13:52:29.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:52:29.654 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T13:52:29.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:52:29.718 DEBUG:teuthology.orchestra.run.smithi148:> sudo ip link delete ceph-brx 2023-10-30T13:52:29.952 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T13:52:29.963 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T13:52:29.963 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:52:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc6e1d3640 1 Processor -- start 2023-10-30T13:52:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc6e1d3640 1 -- start start 2023-10-30T13:52:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc6e1d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdc6815d6f0 0x7fdc6815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc6e1d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 0x7fdc680cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc6e1d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc680cb800 con 0x7fdc681538a0 2023-10-30T13:52:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc6e1d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc680cb330 con 0x7fdc6815d6f0 2023-10-30T13:52:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc6e1d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680cad70 con 0x7fdc680cf160 2023-10-30T13:52:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc66ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdc6815d6f0 0x7fdc6815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc66ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fdc6815d6f0 0x7fdc6815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56636/0 (socket says 172.21.15.148:56636) 2023-10-30T13:52:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc66ffd640 1 -- 172.21.15.148:0/3141912992 learned_addr learned my addr 172.21.15.148:0/3141912992 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc67fff640 1 --2- 172.21.15.148:0/3141912992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 0x7fdc680cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc66ffd640 1 -- 172.21.15.148:0/3141912992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 msgr2=0x7fdc680cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc66ffd640 1 --2- 172.21.15.148:0/3141912992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 0x7fdc680cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc66ffd640 1 -- 172.21.15.148:0/3141912992 >> v1:172.21.15.150:6789/0 conn(0x7fdc681538a0 legacy=0x7fdc680ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.098+0000 7fdc66ffd640 1 -- 172.21.15.148:0/3141912992 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc680cb050 con 0x7fdc6815d6f0 2023-10-30T13:52:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc66ffd640 1 --2- 172.21.15.148:0/3141912992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 0x7fdc6815a090 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009870 tx=0x7fdc5c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62cbc94ed7179b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc667fc640 1 -- 172.21.15.148:0/3141912992 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0a8a30 con 0x7fdc6815d6f0 2023-10-30T13:52:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc667fc640 1 -- 172.21.15.148:0/3141912992 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5c0a8b90 con 0x7fdc6815d6f0 2023-10-30T13:52:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc667fc640 1 -- 172.21.15.148:0/3141912992 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0b0440 con 0x7fdc6815d6f0 2023-10-30T13:52:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/3141912992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 msgr2=0x7fdc6815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc6e1d3640 1 --2- 172.21.15.148:0/3141912992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 0x7fdc6815a090 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009870 tx=0x7fdc5c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:52:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/3141912992 shutdown_connections 2023-10-30T13:52:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc6e1d3640 1 --2- 172.21.15.148:0/3141912992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 0x7fdc680cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc6e1d3640 1 --2- 172.21.15.148:0/3141912992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 0x7fdc6815a090 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/3141912992 >> 172.21.15.148:0/3141912992 conn(0x7fdc6806bfe0 msgr2=0x7fdc6806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/3141912992 shutdown_connections 2023-10-30T13:52:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.102+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/3141912992 wait complete. 2023-10-30T13:52:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 Processor -- start 2023-10-30T13:52:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 -- start start 2023-10-30T13:52:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 0x7fdc68159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc681538a0 0x7fdc680c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 0x7fdc681e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc681e3bf0 con 0x7fdc6815d6f0 2023-10-30T13:52:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc681580f0 con 0x7fdc681538a0 2023-10-30T13:52:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680c59a0 con 0x7fdc680cf160 2023-10-30T13:52:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc67fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 0x7fdc681e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc67fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 0x7fdc681e2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56644/0 (socket says 172.21.15.148:56644) 2023-10-30T13:52:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc67fff640 1 -- 172.21.15.148:0/2828799284 learned_addr learned my addr 172.21.15.148:0/2828799284 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc66ffd640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc681538a0 0x7fdc680c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc677fe640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 0x7fdc68159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc67fff640 1 -- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 msgr2=0x7fdc68159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc67fff640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 0x7fdc68159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc67fff640 1 -- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc681538a0 msgr2=0x7fdc680c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc67fff640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc681538a0 0x7fdc680c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc67fff640 1 -- 172.21.15.148:0/2828799284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c0094d0 con 0x7fdc6815d6f0 2023-10-30T13:52:30.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc66ffd640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc681538a0 0x7fdc680c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:30.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc67fff640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 0x7fdc681e2a40 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fdc60007d90 tx=0x7fdc6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:30.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc4ffff640 1 -- 172.21.15.148:0/2828799284 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60014420 con 0x7fdc6815d6f0 2023-10-30T13:52:30.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc4ffff640 1 -- 172.21.15.148:0/2828799284 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc600145a0 con 0x7fdc6815d6f0 2023-10-30T13:52:30.119 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc4ffff640 1 -- 172.21.15.148:0/2828799284 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60014950 con 0x7fdc6815d6f0 2023-10-30T13:52:30.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc681584d0 con 0x7fdc6815d6f0 2023-10-30T13:52:30.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc680c4f60 con 0x7fdc6815d6f0 2023-10-30T13:52:30.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.106+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc680b2a50 con 0x7fdc6815d6f0 2023-10-30T13:52:30.120 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.110+0000 7fdc4ffff640 1 -- 172.21.15.148:0/2828799284 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdc60014ab0 con 0x7fdc6815d6f0 2023-10-30T13:52:30.121 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.110+0000 7fdc4ffff640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc540664d0 0x7fdc54068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.110+0000 7fdc4ffff640 1 -- 172.21.15.148:0/2828799284 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fdc60084fe0 con 0x7fdc6815d6f0 2023-10-30T13:52:30.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.110+0000 7fdc677fe640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc540664d0 0x7fdc54068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.110+0000 7fdc677fe640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc540664d0 0x7fdc54068990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdc58000f70 tx=0x7fdc58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:30.122 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.114+0000 7fdc4ffff640 1 -- 172.21.15.148:0/2828799284 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdc60051bf0 con 0x7fdc6815d6f0 2023-10-30T13:52:30.267 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.262+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdc68159b70 con 0x7fdc540664d0 2023-10-30T13:52:30.291 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.286+0000 7fdc4ffff640 1 -- 172.21.15.148:0/2828799284 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293908 (secure 0 0 0) 0x7fdc58001370 con 0x7fdc540664d0 2023-10-30T13:52:30.294 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:52:30.296 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:52:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc540664d0 msgr2=0x7fdc54068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc540664d0 0x7fdc54068990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdc58000f70 tx=0x7fdc58009040 comp rx=0 tx=0).stop 2023-10-30T13:52:30.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 msgr2=0x7fdc681e2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 0x7fdc681e2a40 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fdc60007d90 tx=0x7fdc6000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 shutdown_connections 2023-10-30T13:52:30.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fdc540664d0 0x7fdc54068990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc6815d6f0 0x7fdc681e2a40 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc681538a0 0x7fdc680c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 --2- 172.21.15.148:0/2828799284 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fdc680cf160 0x7fdc68159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 >> 172.21.15.148:0/2828799284 conn(0x7fdc6806bfe0 msgr2=0x7fdc6814ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:30.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 shutdown_connections 2023-10-30T13:52:30.300 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.290+0000 7fdc6e1d3640 1 -- 172.21.15.148:0/2828799284 wait complete. 2023-10-30T13:52:30.314 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1323,"stamp":"2023-10-30T13:52:29.202282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068955283,"num_objects":40989,"num_object_clones":18317,"num_object_copies":81978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40989,"num_whiteouts":14355,"num_read":18168,"num_read_kb":6131546,"num_write":117561,"num_write_kb":3530621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1396,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128326,"ondisk_log_size":128326,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5284484,"kb_used_data":4138692,"kb_used_omap":12,"kb_used_meta":1145715,"kb_avail":749690236,"statfs":{"total":773094113280,"available":767682801664,"internally_reserved":0,"allocated":4238020608,"data_stored":4143231163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1173212728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":100,"apply_latency_ms":100,"commit_latency_ns":100000000,"apply_latency_ns":100000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":56824291,"num_objects":745,"num_object_clones":253,"num_object_copies":1490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":745,"num_whiteouts":242,"num_read":298,"num_read_kb":66939,"num_write":972,"num_write_kb":56191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004500"},"pg_stats":[{"pgid":"3.3f","version":"67'1458","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.951630+0000","last_change":"2023-10-30T13:21:40.584822+0000","last_active":"2023-10-30T13:52:23.951630+0000","last_peered":"2023-10-30T13:52:23.951630+0000","last_clean":"2023-10-30T13:52:23.951630+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:52:23.951630+0000","last_undegraded":"2023-10-30T13:52:23.951630+0000","last_fullsized":"2023-10-30T13:52:23.951630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:13.082992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014806453000000001,"stat_sum":{"num_bytes":11269044,"num_objects":569,"num_object_clones":260,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":205,"num_read":107,"num_read_kb":121,"num_write":1253,"num_write_kb":17103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1464","reported_seq":3387,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.234533+0000","last_change":"2023-10-30T13:21:40.572202+0000","last_active":"2023-10-30T13:52:25.234533+0000","last_peered":"2023-10-30T13:52:25.234533+0000","last_clean":"2023-10-30T13:52:25.234533+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:52:25.234533+0000","last_undegraded":"2023-10-30T13:52:25.234533+0000","last_fullsized":"2023-10-30T13:52:25.234533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:58.516621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059691099999999997,"stat_sum":{"num_bytes":30975930,"num_objects":597,"num_object_clones":279,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":225,"num_read":82,"num_read_kb":30,"num_write":1226,"num_write_kb":32475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1518","reported_seq":3465,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.166473+0000","last_change":"2023-10-30T13:21:40.528480+0000","last_active":"2023-10-30T13:52:24.166473+0000","last_peered":"2023-10-30T13:52:24.166473+0000","last_clean":"2023-10-30T13:52:24.166473+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:52:24.166473+0000","last_undegraded":"2023-10-30T13:52:24.166473+0000","last_fullsized":"2023-10-30T13:52:24.166473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:03.253931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025608710000000002,"stat_sum":{"num_bytes":18704388,"num_objects":617,"num_object_clones":279,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":225,"num_read":106,"num_read_kb":8300,"num_write":1300,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1566","reported_seq":3502,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.918237+0000","last_change":"2023-10-30T13:21:40.567224+0000","last_active":"2023-10-30T13:52:23.918237+0000","last_peered":"2023-10-30T13:52:23.918237+0000","last_clean":"2023-10-30T13:52:23.918237+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:52:23.918237+0000","last_undegraded":"2023-10-30T13:52:23.918237+0000","last_fullsized":"2023-10-30T13:52:23.918237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1566,"log_dups_size":0,"ondisk_log_size":1566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:34.794112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021564890000000001,"stat_sum":{"num_bytes":23329933,"num_objects":630,"num_object_clones":293,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":227,"num_read":112,"num_read_kb":4378,"num_write":1354,"num_write_kb":25006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1582","reported_seq":3502,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.930705+0000","last_change":"2023-10-30T13:21:40.579191+0000","last_active":"2023-10-30T13:52:23.930705+0000","last_peered":"2023-10-30T13:52:23.930705+0000","last_clean":"2023-10-30T13:52:23.930705+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:52:23.930705+0000","last_undegraded":"2023-10-30T13:52:23.930705+0000","last_fullsized":"2023-10-30T13:52:23.930705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1582,"log_dups_size":0,"ondisk_log_size":1582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:10.057619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011808077,"stat_sum":{"num_bytes":16184828,"num_objects":654,"num_object_clones":305,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":251,"num_read":109,"num_read_kb":152,"num_write":1339,"num_write_kb":20879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1451","reported_seq":3398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932299+0000","last_change":"2023-10-30T13:21:40.539757+0000","last_active":"2023-10-30T13:52:23.932299+0000","last_peered":"2023-10-30T13:52:23.932299+0000","last_clean":"2023-10-30T13:52:23.932299+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:52:23.932299+0000","last_undegraded":"2023-10-30T13:52:23.932299+0000","last_fullsized":"2023-10-30T13:52:23.932299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1451,"log_dups_size":0,"ondisk_log_size":1451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:59.387758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01102786,"stat_sum":{"num_bytes":18372801,"num_objects":593,"num_object_clones":277,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":222,"num_read":108,"num_read_kb":4137,"num_write":1217,"num_write_kb":19358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1501","reported_seq":3438,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.927654+0000","last_change":"2023-10-30T13:21:40.580121+0000","last_active":"2023-10-30T13:52:23.927654+0000","last_peered":"2023-10-30T13:52:23.927654+0000","last_clean":"2023-10-30T13:52:23.927654+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:52:23.927654+0000","last_undegraded":"2023-10-30T13:52:23.927654+0000","last_fullsized":"2023-10-30T13:52:23.927654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:42.759583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012047337,"stat_sum":{"num_bytes":9859762,"num_objects":598,"num_object_clones":275,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":211,"num_read":96,"num_read_kb":35,"num_write":1295,"num_write_kb":11866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1450","reported_seq":3406,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.168248+0000","last_change":"2023-10-30T13:21:40.575154+0000","last_active":"2023-10-30T13:52:24.168248+0000","last_peered":"2023-10-30T13:52:24.168248+0000","last_clean":"2023-10-30T13:52:24.168248+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:52:24.168248+0000","last_undegraded":"2023-10-30T13:52:24.168248+0000","last_fullsized":"2023-10-30T13:52:24.168248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1450,"log_dups_size":0,"ondisk_log_size":1450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:28.268146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063254060000000004,"stat_sum":{"num_bytes":13694423,"num_objects":577,"num_object_clones":268,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":211,"num_read":108,"num_read_kb":104,"num_write":1228,"num_write_kb":23250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1510","reported_seq":3452,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.243551+0000","last_change":"2023-10-30T13:21:40.584916+0000","last_active":"2023-10-30T13:52:25.243551+0000","last_peered":"2023-10-30T13:52:25.243551+0000","last_clean":"2023-10-30T13:52:25.243551+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:52:25.243551+0000","last_undegraded":"2023-10-30T13:52:25.243551+0000","last_fullsized":"2023-10-30T13:52:25.243551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1510,"log_dups_size":0,"ondisk_log_size":1510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:08:14.163088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014360304000000001,"stat_sum":{"num_bytes":19134308,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":234,"num_read":109,"num_read_kb":4185,"num_write":1275,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1494","reported_seq":3432,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.931509+0000","last_change":"2023-10-30T13:21:40.580946+0000","last_active":"2023-10-30T13:52:23.931509+0000","last_peered":"2023-10-30T13:52:23.931509+0000","last_clean":"2023-10-30T13:52:23.931509+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:52:23.931509+0000","last_undegraded":"2023-10-30T13:52:23.931509+0000","last_fullsized":"2023-10-30T13:52:23.931509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:29:47.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010277041000000001,"stat_sum":{"num_bytes":19693407,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":208,"num_read":105,"num_read_kb":2976,"num_write":1294,"num_write_kb":29513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1525","reported_seq":3452,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.917846+0000","last_change":"2023-10-30T13:21:40.579672+0000","last_active":"2023-10-30T13:52:23.917846+0000","last_peered":"2023-10-30T13:52:23.917846+0000","last_clean":"2023-10-30T13:52:23.917846+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:52:23.917846+0000","last_undegraded":"2023-10-30T13:52:23.917846+0000","last_fullsized":"2023-10-30T13:52:23.917846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1525,"log_dups_size":0,"ondisk_log_size":1525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:22:11.088793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01048763,"stat_sum":{"num_bytes":11727836,"num_objects":636,"num_object_clones":292,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":231,"num_read":95,"num_read_kb":27,"num_write":1290,"num_write_kb":13174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1518","reported_seq":3443,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.939348+0000","last_change":"2023-10-30T13:21:40.537641+0000","last_active":"2023-10-30T13:52:23.939348+0000","last_peered":"2023-10-30T13:52:23.939348+0000","last_clean":"2023-10-30T13:52:23.939348+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:52:23.939348+0000","last_undegraded":"2023-10-30T13:52:23.939348+0000","last_fullsized":"2023-10-30T13:52:23.939348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:55.899096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085295030000000008,"stat_sum":{"num_bytes":15445695,"num_objects":634,"num_object_clones":293,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":231,"num_read":101,"num_read_kb":102,"num_write":1282,"num_write_kb":21892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1473","reported_seq":3415,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.907291+0000","last_change":"2023-10-30T13:21:40.573274+0000","last_active":"2023-10-30T13:52:23.907291+0000","last_peered":"2023-10-30T13:52:23.907291+0000","last_clean":"2023-10-30T13:52:23.907291+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:52:23.907291+0000","last_undegraded":"2023-10-30T13:52:23.907291+0000","last_fullsized":"2023-10-30T13:52:23.907291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:23:55.087821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004933977,"stat_sum":{"num_bytes":6328355,"num_objects":596,"num_object_clones":272,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":210,"num_read":97,"num_read_kb":4194,"num_write":1284,"num_write_kb":15784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1551","reported_seq":3498,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.231249+0000","last_change":"2023-10-30T13:21:40.528789+0000","last_active":"2023-10-30T13:52:25.231249+0000","last_peered":"2023-10-30T13:52:25.231249+0000","last_clean":"2023-10-30T13:52:25.231249+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:52:25.231249+0000","last_undegraded":"2023-10-30T13:52:25.231249+0000","last_fullsized":"2023-10-30T13:52:25.231249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:55.527282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029909680000000001,"stat_sum":{"num_bytes":31670070,"num_objects":610,"num_object_clones":280,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":227,"num_read":110,"num_read_kb":16437,"num_write":1314,"num_write_kb":33491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1555","reported_seq":3494,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.906435+0000","last_change":"2023-10-30T13:21:40.571731+0000","last_active":"2023-10-30T13:52:23.906435+0000","last_peered":"2023-10-30T13:52:23.906435+0000","last_clean":"2023-10-30T13:52:23.906435+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:52:23.906435+0000","last_undegraded":"2023-10-30T13:52:23.906435+0000","last_fullsized":"2023-10-30T13:52:23.906435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:47.649131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058939090000000001,"stat_sum":{"num_bytes":14342306,"num_objects":613,"num_object_clones":284,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":231,"num_read":106,"num_read_kb":2953,"num_write":1333,"num_write_kb":15925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1748","reported_seq":3606,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923510+0000","last_change":"2023-10-30T13:21:40.536269+0000","last_active":"2023-10-30T13:52:23.923510+0000","last_peered":"2023-10-30T13:52:23.923510+0000","last_clean":"2023-10-30T13:52:23.923510+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:52:23.923510+0000","last_undegraded":"2023-10-30T13:52:23.923510+0000","last_fullsized":"2023-10-30T13:52:23.923510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1748,"log_dups_size":0,"ondisk_log_size":1748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:06.829080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093243890000000006,"stat_sum":{"num_bytes":10965780,"num_objects":703,"num_object_clones":374,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":231,"num_read":115,"num_read_kb":1694,"num_write":1434,"num_write_kb":17364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1579","reported_seq":3505,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.946432+0000","last_change":"2023-10-30T13:21:40.539642+0000","last_active":"2023-10-30T13:52:23.946432+0000","last_peered":"2023-10-30T13:52:23.946432+0000","last_clean":"2023-10-30T13:52:23.946432+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:52:23.946432+0000","last_undegraded":"2023-10-30T13:52:23.946432+0000","last_fullsized":"2023-10-30T13:52:23.946432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1579,"log_dups_size":0,"ondisk_log_size":1579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:12.694457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011284880000000001,"stat_sum":{"num_bytes":18442682,"num_objects":632,"num_object_clones":291,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":228,"num_read":100,"num_read_kb":8212,"num_write":1340,"num_write_kb":24815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1459","reported_seq":3397,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932387+0000","last_change":"2023-10-30T13:21:40.567529+0000","last_active":"2023-10-30T13:52:23.932387+0000","last_peered":"2023-10-30T13:52:23.932387+0000","last_clean":"2023-10-30T13:52:23.932387+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:52:23.932387+0000","last_undegraded":"2023-10-30T13:52:23.932387+0000","last_fullsized":"2023-10-30T13:52:23.932387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:45.983709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002295727,"stat_sum":{"num_bytes":22547199,"num_objects":575,"num_object_clones":267,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":207,"num_read":89,"num_read_kb":4127,"num_write":1240,"num_write_kb":24211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1620","reported_seq":3538,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.922596+0000","last_change":"2023-10-30T13:21:40.572640+0000","last_active":"2023-10-30T13:52:23.922596+0000","last_peered":"2023-10-30T13:52:23.922596+0000","last_clean":"2023-10-30T13:52:23.922596+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:52:23.922596+0000","last_undegraded":"2023-10-30T13:52:23.922596+0000","last_fullsized":"2023-10-30T13:52:23.922596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:31.079099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061689539999999999,"stat_sum":{"num_bytes":11703479,"num_objects":649,"num_object_clones":303,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":237,"num_read":106,"num_read_kb":7307,"num_write":1383,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1553","reported_seq":3491,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.915574+0000","last_change":"2023-10-30T13:21:40.580324+0000","last_active":"2023-10-30T13:52:23.915574+0000","last_peered":"2023-10-30T13:52:23.915574+0000","last_clean":"2023-10-30T13:52:23.915574+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:52:23.915574+0000","last_undegraded":"2023-10-30T13:52:23.915574+0000","last_fullsized":"2023-10-30T13:52:23.915574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1553,"log_dups_size":0,"ondisk_log_size":1553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:52.961020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094394720000000008,"stat_sum":{"num_bytes":31771338,"num_objects":621,"num_object_clones":283,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":229,"num_read":103,"num_read_kb":4126,"num_write":1338,"num_write_kb":42793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1482","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.933727+0000","last_change":"2023-10-30T13:21:40.528854+0000","last_active":"2023-10-30T13:52:23.933727+0000","last_peered":"2023-10-30T13:52:23.933727+0000","last_clean":"2023-10-30T13:52:23.933727+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:52:23.933727+0000","last_undegraded":"2023-10-30T13:52:23.933727+0000","last_fullsized":"2023-10-30T13:52:23.933727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1482,"log_dups_size":0,"ondisk_log_size":1482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:48:55.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027436349999999999,"stat_sum":{"num_bytes":6210074,"num_objects":595,"num_object_clones":273,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":228,"num_read":98,"num_read_kb":33,"num_write":1259,"num_write_kb":10653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1636","reported_seq":3545,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.244114+0000","last_change":"2023-10-30T13:21:40.535360+0000","last_active":"2023-10-30T13:52:25.244114+0000","last_peered":"2023-10-30T13:52:25.244114+0000","last_clean":"2023-10-30T13:52:25.244114+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:52:25.244114+0000","last_undegraded":"2023-10-30T13:52:25.244114+0000","last_fullsized":"2023-10-30T13:52:25.244114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1636,"log_dups_size":0,"ondisk_log_size":1636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:13.413880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082328149999999992,"stat_sum":{"num_bytes":28398973,"num_objects":670,"num_object_clones":311,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":249,"num_read":102,"num_read_kb":120,"num_write":1394,"num_write_kb":38562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1530","reported_seq":3467,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.908084+0000","last_change":"2023-10-30T13:21:40.571762+0000","last_active":"2023-10-30T13:52:23.908084+0000","last_peered":"2023-10-30T13:52:23.908084+0000","last_clean":"2023-10-30T13:52:23.908084+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:52:23.908084+0000","last_undegraded":"2023-10-30T13:52:23.908084+0000","last_fullsized":"2023-10-30T13:52:23.908084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:29:37.541276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037447470000000001,"stat_sum":{"num_bytes":19407388,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":225,"num_read":103,"num_read_kb":4138,"num_write":1298,"num_write_kb":24714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1570","reported_seq":3510,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923610+0000","last_change":"2023-10-30T13:21:40.536309+0000","last_active":"2023-10-30T13:52:23.923610+0000","last_peered":"2023-10-30T13:52:23.923610+0000","last_clean":"2023-10-30T13:52:23.923610+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:52:23.923610+0000","last_undegraded":"2023-10-30T13:52:23.923610+0000","last_fullsized":"2023-10-30T13:52:23.923610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:04:17.809358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091633449999999998,"stat_sum":{"num_bytes":10552296,"num_objects":616,"num_object_clones":294,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":228,"num_read":115,"num_read_kb":24,"num_write":1340,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1410","reported_seq":3369,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.678938+0000","last_change":"2023-10-30T13:21:40.571783+0000","last_active":"2023-10-30T13:52:25.678938+0000","last_peered":"2023-10-30T13:52:25.678938+0000","last_clean":"2023-10-30T13:52:25.678938+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:52:25.678938+0000","last_undegraded":"2023-10-30T13:52:25.678938+0000","last_fullsized":"2023-10-30T13:52:25.678938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1410,"log_dups_size":0,"ondisk_log_size":1410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:41.732560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059368210000000001,"stat_sum":{"num_bytes":15196336,"num_objects":556,"num_object_clones":256,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":202,"num_read":99,"num_read_kb":1380,"num_write":1201,"num_write_kb":16993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'1569","reported_seq":3517,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.166551+0000","last_change":"2023-10-30T13:21:40.581115+0000","last_active":"2023-10-30T13:52:24.166551+0000","last_peered":"2023-10-30T13:52:24.166551+0000","last_clean":"2023-10-30T13:52:24.166551+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:52:24.166551+0000","last_undegraded":"2023-10-30T13:52:24.166551+0000","last_fullsized":"2023-10-30T13:52:24.166551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:31.199977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099065669999999998,"stat_sum":{"num_bytes":9983325,"num_objects":619,"num_object_clones":286,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":221,"num_read":115,"num_read_kb":68,"num_write":1328,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1534","reported_seq":3477,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.910285+0000","last_change":"2023-10-30T13:21:40.572631+0000","last_active":"2023-10-30T13:52:23.910285+0000","last_peered":"2023-10-30T13:52:23.910285+0000","last_clean":"2023-10-30T13:52:23.910285+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:52:23.910285+0000","last_undegraded":"2023-10-30T13:52:23.910285+0000","last_fullsized":"2023-10-30T13:52:23.910285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:53.178210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059382619999999997,"stat_sum":{"num_bytes":23775876,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":236,"num_read":113,"num_read_kb":4145,"num_write":1299,"num_write_kb":26615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1462","reported_seq":3390,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.907620+0000","last_change":"2023-10-30T13:21:40.575712+0000","last_active":"2023-10-30T13:52:23.907620+0000","last_peered":"2023-10-30T13:52:23.907620+0000","last_clean":"2023-10-30T13:52:23.907620+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:52:23.907620+0000","last_undegraded":"2023-10-30T13:52:23.907620+0000","last_fullsized":"2023-10-30T13:52:23.907620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:09.905979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053787089999999997,"stat_sum":{"num_bytes":10833053,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":213,"num_read":88,"num_read_kb":3300,"num_write":1254,"num_write_kb":16238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1646","reported_seq":3545,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.233660+0000","last_change":"2023-10-30T13:21:40.529120+0000","last_active":"2023-10-30T13:52:25.233660+0000","last_peered":"2023-10-30T13:52:25.233660+0000","last_clean":"2023-10-30T13:52:25.233660+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:52:25.233660+0000","last_undegraded":"2023-10-30T13:52:25.233660+0000","last_fullsized":"2023-10-30T13:52:25.233660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1646,"log_dups_size":0,"ondisk_log_size":1646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:56.055314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028849399999999999,"stat_sum":{"num_bytes":11995550,"num_objects":688,"num_object_clones":321,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":244,"num_read":100,"num_read_kb":8267,"num_write":1414,"num_write_kb":14306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1445","reported_seq":3397,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932007+0000","last_change":"2023-10-30T13:21:40.537362+0000","last_active":"2023-10-30T13:52:23.932007+0000","last_peered":"2023-10-30T13:52:23.932007+0000","last_clean":"2023-10-30T13:52:23.932007+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:52:23.932007+0000","last_undegraded":"2023-10-30T13:52:23.932007+0000","last_fullsized":"2023-10-30T13:52:23.932007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:49:05.792581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096547909999999994,"stat_sum":{"num_bytes":15074071,"num_objects":558,"num_object_clones":259,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":221,"num_read":91,"num_read_kb":32,"num_write":1202,"num_write_kb":16754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1663","reported_seq":3603,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.926321+0000","last_change":"2023-10-30T13:21:40.567248+0000","last_active":"2023-10-30T13:52:23.926321+0000","last_peered":"2023-10-30T13:52:23.926321+0000","last_clean":"2023-10-30T13:52:23.926321+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:52:23.926321+0000","last_undegraded":"2023-10-30T13:52:23.926321+0000","last_fullsized":"2023-10-30T13:52:23.926321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:07.337543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018845820000000001,"stat_sum":{"num_bytes":10798567,"num_objects":649,"num_object_clones":300,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":241,"num_read":120,"num_read_kb":27,"num_write":1443,"num_write_kb":17235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1499","reported_seq":3469,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923689+0000","last_change":"2023-10-30T13:21:40.575268+0000","last_active":"2023-10-30T13:52:23.923689+0000","last_peered":"2023-10-30T13:52:23.923689+0000","last_clean":"2023-10-30T13:52:23.923689+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:52:23.923689+0000","last_undegraded":"2023-10-30T13:52:23.923689+0000","last_fullsized":"2023-10-30T13:52:23.923689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1499,"log_dups_size":0,"ondisk_log_size":1499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:50:11.469301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075239010000000004,"stat_sum":{"num_bytes":15423893,"num_objects":576,"num_object_clones":259,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":203,"num_read":109,"num_read_kb":9552,"num_write":1306,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1548","reported_seq":3492,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.920280+0000","last_change":"2023-10-30T13:21:40.537743+0000","last_active":"2023-10-30T13:52:23.920280+0000","last_peered":"2023-10-30T13:52:23.920280+0000","last_clean":"2023-10-30T13:52:23.920280+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:52:23.920280+0000","last_undegraded":"2023-10-30T13:52:23.920280+0000","last_fullsized":"2023-10-30T13:52:23.920280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:01:00.836625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007356301,"stat_sum":{"num_bytes":40089180,"num_objects":622,"num_object_clones":287,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":230,"num_read":115,"num_read_kb":33,"num_write":1312,"num_write_kb":42997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'604","reported_seq":832,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.233111+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:52:19.233111+0000","last_peered":"2023-10-30T13:52:19.233111+0000","last_clean":"2023-10-30T13:52:19.233111+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:52:19.233111+0000","last_undegraded":"2023-10-30T13:52:19.233111+0000","last_fullsized":"2023-10-30T13:52:19.233111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":252,"num_read_kb":249243,"num_write":662,"num_write_kb":74846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1552","reported_seq":3479,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.929541+0000","last_change":"2023-10-30T13:21:40.529443+0000","last_active":"2023-10-30T13:52:23.929541+0000","last_peered":"2023-10-30T13:52:23.929541+0000","last_clean":"2023-10-30T13:52:23.929541+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:52:23.929541+0000","last_undegraded":"2023-10-30T13:52:23.929541+0000","last_fullsized":"2023-10-30T13:52:23.929541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:39.924469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372290000000001,"stat_sum":{"num_bytes":18713903,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":229,"num_read":108,"num_read_kb":4667,"num_write":1329,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1126","reported_seq":1449,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:15.211697+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:52:15.211697+0000","last_peered":"2023-10-30T13:52:15.211697+0000","last_clean":"2023-10-30T13:52:15.211697+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:52:15.211697+0000","last_undegraded":"2023-10-30T13:52:15.211697+0000","last_fullsized":"2023-10-30T13:52:15.211697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1126,"log_dups_size":0,"ondisk_log_size":1126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25293488,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":404,"num_read_kb":309242,"num_write":1229,"num_write_kb":84643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'745","reported_seq":1123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.654334+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:52:25.654334+0000","last_peered":"2023-10-30T13:52:25.654334+0000","last_clean":"2023-10-30T13:52:25.654334+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:52:25.654334+0000","last_undegraded":"2023-10-30T13:52:25.654334+0000","last_fullsized":"2023-10-30T13:52:25.654334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397151,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":403,"num_read_kb":153383,"num_write":818,"num_write_kb":55113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1778","reported_seq":3695,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.668056+0000","last_change":"2023-10-30T13:21:40.575255+0000","last_active":"2023-10-30T13:52:25.668056+0000","last_peered":"2023-10-30T13:52:25.668056+0000","last_clean":"2023-10-30T13:52:25.668056+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:52:25.668056+0000","last_undegraded":"2023-10-30T13:52:25.668056+0000","last_fullsized":"2023-10-30T13:52:25.668056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1778,"log_dups_size":0,"ondisk_log_size":1778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:20.571707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064075410000000001,"stat_sum":{"num_bytes":11666119,"num_objects":729,"num_object_clones":341,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":261,"num_read":141,"num_read_kb":4175,"num_write":1518,"num_write_kb":22588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1399","reported_seq":1668,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.266715+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:52:24.266715+0000","last_peered":"2023-10-30T13:52:24.266715+0000","last_clean":"2023-10-30T13:52:24.266715+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:52:24.266715+0000","last_undegraded":"2023-10-30T13:52:24.266715+0000","last_fullsized":"2023-10-30T13:52:24.266715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1399,"log_dups_size":0,"ondisk_log_size":1399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":354,"num_read_kb":270680,"num_write":1485,"num_write_kb":75457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1501","reported_seq":3434,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.893719+0000","last_change":"2023-10-30T13:21:40.571518+0000","last_active":"2023-10-30T13:52:23.893719+0000","last_peered":"2023-10-30T13:52:23.893719+0000","last_clean":"2023-10-30T13:52:23.893719+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:52:23.893719+0000","last_undegraded":"2023-10-30T13:52:23.893719+0000","last_fullsized":"2023-10-30T13:52:23.893719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:19:07.205330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058964639999999997,"stat_sum":{"num_bytes":19507019,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":240,"num_read":105,"num_read_kb":5362,"num_write":1250,"num_write_kb":24752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'953","reported_seq":1189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.509429+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:52:24.509429+0000","last_peered":"2023-10-30T13:52:24.509429+0000","last_clean":"2023-10-30T13:52:24.509429+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:52:24.509429+0000","last_undegraded":"2023-10-30T13:52:24.509429+0000","last_fullsized":"2023-10-30T13:52:24.509429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":313,"num_read_kb":141404,"num_write":1072,"num_write_kb":63726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1479","reported_seq":3432,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.935516+0000","last_change":"2023-10-30T13:21:40.567796+0000","last_active":"2023-10-30T13:52:23.935516+0000","last_peered":"2023-10-30T13:52:23.935516+0000","last_clean":"2023-10-30T13:52:23.935516+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:52:23.935516+0000","last_undegraded":"2023-10-30T13:52:23.935516+0000","last_fullsized":"2023-10-30T13:52:23.935516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:08:51.743716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025817129999999998,"stat_sum":{"num_bytes":23539910,"num_objects":588,"num_object_clones":267,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":208,"num_read":106,"num_read_kb":8313,"num_write":1264,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"67'1122","reported_seq":1340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:17.672748+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:52:17.672748+0000","last_peered":"2023-10-30T13:52:17.672748+0000","last_clean":"2023-10-30T13:52:17.672748+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:52:17.672748+0000","last_undegraded":"2023-10-30T13:52:17.672748+0000","last_fullsized":"2023-10-30T13:52:17.672748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1122,"log_dups_size":0,"ondisk_log_size":1122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":240,"num_read_kb":201173,"num_write":1196,"num_write_kb":66729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1545","reported_seq":3456,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.938692+0000","last_change":"2023-10-30T13:21:40.528173+0000","last_active":"2023-10-30T13:52:23.938692+0000","last_peered":"2023-10-30T13:52:23.938692+0000","last_clean":"2023-10-30T13:52:23.938692+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:52:23.938692+0000","last_undegraded":"2023-10-30T13:52:23.938692+0000","last_fullsized":"2023-10-30T13:52:23.938692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1545,"log_dups_size":0,"ondisk_log_size":1545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:50:03.879621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022874710000000001,"stat_sum":{"num_bytes":16137402,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":233,"num_read":91,"num_read_kb":34,"num_write":1322,"num_write_kb":18013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1315","reported_seq":1556,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.212373+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:52:24.212373+0000","last_peered":"2023-10-30T13:52:24.212373+0000","last_clean":"2023-10-30T13:52:24.212373+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:52:24.212373+0000","last_undegraded":"2023-10-30T13:52:24.212373+0000","last_fullsized":"2023-10-30T13:52:24.212373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20972192,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":307,"num_read_kb":165804,"num_write":1420,"num_write_kb":50966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1560","reported_seq":3492,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.937626+0000","last_change":"2023-10-30T13:21:40.538439+0000","last_active":"2023-10-30T13:52:23.937626+0000","last_peered":"2023-10-30T13:52:23.937626+0000","last_clean":"2023-10-30T13:52:23.937626+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:52:23.937626+0000","last_undegraded":"2023-10-30T13:52:23.937626+0000","last_fullsized":"2023-10-30T13:52:23.937626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:36:06.776976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087798259999999993,"stat_sum":{"num_bytes":32677996,"num_objects":638,"num_object_clones":293,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":232,"num_read":108,"num_read_kb":10951,"num_write":1340,"num_write_kb":38819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'935","reported_seq":1253,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.960735+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:52:23.960735+0000","last_peered":"2023-10-30T13:52:23.960735+0000","last_clean":"2023-10-30T13:52:23.960735+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:52:23.960735+0000","last_undegraded":"2023-10-30T13:52:23.960735+0000","last_fullsized":"2023-10-30T13:52:23.960735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":413,"num_read_kb":281405,"num_write":1056,"num_write_kb":80196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1490","reported_seq":3405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.927330+0000","last_change":"2023-10-30T13:21:40.583356+0000","last_active":"2023-10-30T13:52:23.927330+0000","last_peered":"2023-10-30T13:52:23.927330+0000","last_clean":"2023-10-30T13:52:23.927330+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:52:23.927330+0000","last_undegraded":"2023-10-30T13:52:23.927330+0000","last_fullsized":"2023-10-30T13:52:23.927330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:46.232350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014787140000000001,"stat_sum":{"num_bytes":19072740,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":238,"num_read":87,"num_read_kb":102,"num_write":1252,"num_write_kb":21862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'1084","reported_seq":1410,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.582275+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:52:16.582275+0000","last_peered":"2023-10-30T13:52:16.582275+0000","last_clean":"2023-10-30T13:52:16.582275+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:52:16.582275+0000","last_undegraded":"2023-10-30T13:52:16.582275+0000","last_fullsized":"2023-10-30T13:52:16.582275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748770,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":476,"num_read_kb":203988,"num_write":1202,"num_write_kb":67245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1496","reported_seq":3446,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.929377+0000","last_change":"2023-10-30T13:21:40.528316+0000","last_active":"2023-10-30T13:52:23.929377+0000","last_peered":"2023-10-30T13:52:23.929377+0000","last_clean":"2023-10-30T13:52:23.929377+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:52:23.929377+0000","last_undegraded":"2023-10-30T13:52:23.929377+0000","last_fullsized":"2023-10-30T13:52:23.929377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:08:42.998206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024076599999999998,"stat_sum":{"num_bytes":14342228,"num_objects":584,"num_object_clones":268,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":211,"num_read":101,"num_read_kb":4159,"num_write":1284,"num_write_kb":24386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'574","reported_seq":755,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.233164+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:52:19.233164+0000","last_peered":"2023-10-30T13:52:19.233164+0000","last_clean":"2023-10-30T13:52:19.233164+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:52:19.233164+0000","last_undegraded":"2023-10-30T13:52:19.233164+0000","last_fullsized":"2023-10-30T13:52:19.233164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":574,"log_dups_size":0,"ondisk_log_size":574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":203,"num_read_kb":83137,"num_write":662,"num_write_kb":45991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1588","reported_seq":3563,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.559855+0000","last_change":"2023-10-30T13:21:40.579925+0000","last_active":"2023-10-30T13:52:18.559855+0000","last_peered":"2023-10-30T13:52:18.559855+0000","last_clean":"2023-10-30T13:52:18.559855+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:52:18.559855+0000","last_undegraded":"2023-10-30T13:52:18.559855+0000","last_fullsized":"2023-10-30T13:52:18.559855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1588,"log_dups_size":0,"ondisk_log_size":1588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:39.495586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011943199999999999,"stat_sum":{"num_bytes":18918293,"num_objects":634,"num_object_clones":295,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":241,"num_read":154,"num_read_kb":114,"num_write":1356,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'499","reported_seq":731,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.240235+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:52:19.240235+0000","last_peered":"2023-10-30T13:52:19.240235+0000","last_clean":"2023-10-30T13:52:19.240235+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:52:19.240235+0000","last_undegraded":"2023-10-30T13:52:19.240235+0000","last_fullsized":"2023-10-30T13:52:19.240235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":289,"num_read_kb":121284,"num_write":554,"num_write_kb":58687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1629","reported_seq":3510,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.242217+0000","last_change":"2023-10-30T13:21:40.576918+0000","last_active":"2023-10-30T13:52:25.242217+0000","last_peered":"2023-10-30T13:52:25.242217+0000","last_clean":"2023-10-30T13:52:25.242217+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:52:25.242217+0000","last_undegraded":"2023-10-30T13:52:25.242217+0000","last_fullsized":"2023-10-30T13:52:25.242217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1629,"log_dups_size":0,"ondisk_log_size":1629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:33.447713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086123929999999994,"stat_sum":{"num_bytes":19792070,"num_objects":655,"num_object_clones":349,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":204,"num_read":113,"num_read_kb":142,"num_write":1344,"num_write_kb":24080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'960","reported_seq":1239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:15.167943+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:52:15.167943+0000","last_peered":"2023-10-30T13:52:15.167943+0000","last_clean":"2023-10-30T13:52:15.167943+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:52:15.167943+0000","last_undegraded":"2023-10-30T13:52:15.167943+0000","last_fullsized":"2023-10-30T13:52:15.167943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36678727,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":465,"num_read_kb":133831,"num_write":1107,"num_write_kb":54499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1554","reported_seq":3475,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.928034+0000","last_change":"2023-10-30T13:21:40.583214+0000","last_active":"2023-10-30T13:52:23.928034+0000","last_peered":"2023-10-30T13:52:23.928034+0000","last_clean":"2023-10-30T13:52:23.928034+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:52:23.928034+0000","last_undegraded":"2023-10-30T13:52:23.928034+0000","last_fullsized":"2023-10-30T13:52:23.928034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1554,"log_dups_size":0,"ondisk_log_size":1554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:22.371097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015028952999999999,"stat_sum":{"num_bytes":21253392,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":236,"num_read":95,"num_read_kb":12339,"num_write":1304,"num_write_kb":30718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'638","reported_seq":863,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.241252+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:52:19.241252+0000","last_peered":"2023-10-30T13:52:19.241252+0000","last_clean":"2023-10-30T13:52:19.241252+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:52:19.241252+0000","last_undegraded":"2023-10-30T13:52:19.241252+0000","last_fullsized":"2023-10-30T13:52:19.241252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165846,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":249,"num_read_kb":148701,"num_write":777,"num_write_kb":54421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1551","reported_seq":3481,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932444+0000","last_change":"2023-10-30T13:21:40.579365+0000","last_active":"2023-10-30T13:52:23.932444+0000","last_peered":"2023-10-30T13:52:23.932444+0000","last_clean":"2023-10-30T13:52:23.932444+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:52:23.932444+0000","last_undegraded":"2023-10-30T13:52:23.932444+0000","last_fullsized":"2023-10-30T13:52:23.932444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:51:50.149626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011682140000000001,"stat_sum":{"num_bytes":26941199,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":237,"num_read":103,"num_read_kb":4114,"num_write":1331,"num_write_kb":32609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'1051","reported_seq":1327,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.149100+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:52:16.149100+0000","last_peered":"2023-10-30T13:52:16.149100+0000","last_clean":"2023-10-30T13:52:16.149100+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:52:16.149100+0000","last_undegraded":"2023-10-30T13:52:16.149100+0000","last_fullsized":"2023-10-30T13:52:16.149100+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1051,"log_dups_size":0,"ondisk_log_size":1051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":381,"num_read_kb":250673,"num_write":1155,"num_write_kb":87740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1528","reported_seq":3470,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.951203+0000","last_change":"2023-10-30T13:21:40.585506+0000","last_active":"2023-10-30T13:52:23.951203+0000","last_peered":"2023-10-30T13:52:23.951203+0000","last_clean":"2023-10-30T13:52:23.951203+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:52:23.951203+0000","last_undegraded":"2023-10-30T13:52:23.951203+0000","last_fullsized":"2023-10-30T13:52:23.951203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:06.635146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014794043,"stat_sum":{"num_bytes":18888621,"num_objects":614,"num_object_clones":283,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":224,"num_read":107,"num_read_kb":4188,"num_write":1306,"num_write_kb":24240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'1227","reported_seq":1478,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.806783+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:52:23.806783+0000","last_peered":"2023-10-30T13:52:23.806783+0000","last_clean":"2023-10-30T13:52:23.806783+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:52:23.806783+0000","last_undegraded":"2023-10-30T13:52:23.806783+0000","last_fullsized":"2023-10-30T13:52:23.806783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1227,"log_dups_size":0,"ondisk_log_size":1227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364039,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":288,"num_read_kb":200539,"num_write":1325,"num_write_kb":71054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1570","reported_seq":3523,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923756+0000","last_change":"2023-10-30T13:21:40.538748+0000","last_active":"2023-10-30T13:52:23.923756+0000","last_peered":"2023-10-30T13:52:23.923756+0000","last_clean":"2023-10-30T13:52:23.923756+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:52:23.923756+0000","last_undegraded":"2023-10-30T13:52:23.923756+0000","last_fullsized":"2023-10-30T13:52:23.923756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:15.511609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010788236,"stat_sum":{"num_bytes":10484856,"num_objects":600,"num_object_clones":281,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":220,"num_read":116,"num_read_kb":28,"num_write":1335,"num_write_kb":13295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'776","reported_seq":1023,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.577979+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:52:16.577979+0000","last_peered":"2023-10-30T13:52:16.577979+0000","last_clean":"2023-10-30T13:52:16.577979+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:52:16.577979+0000","last_undegraded":"2023-10-30T13:52:16.577979+0000","last_fullsized":"2023-10-30T13:52:16.577979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":322,"num_read_kb":174067,"num_write":894,"num_write_kb":62874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1536","reported_seq":3467,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.231736+0000","last_change":"2023-10-30T13:21:40.583977+0000","last_active":"2023-10-30T13:52:25.231736+0000","last_peered":"2023-10-30T13:52:25.231736+0000","last_clean":"2023-10-30T13:52:25.231736+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:52:25.231736+0000","last_undegraded":"2023-10-30T13:52:25.231736+0000","last_fullsized":"2023-10-30T13:52:25.231736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1536,"log_dups_size":0,"ondisk_log_size":1536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:18.653967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015268978000000001,"stat_sum":{"num_bytes":11043591,"num_objects":616,"num_object_clones":291,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":221,"num_read":104,"num_read_kb":3707,"num_write":1322,"num_write_kb":16509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":232,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T13:50:37.450196+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:50:37.450196+0000","last_peered":"2023-10-30T13:50:37.450196+0000","last_clean":"2023-10-30T13:50:37.450196+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:50:37.450196+0000","last_undegraded":"2023-10-30T13:50:37.450196+0000","last_fullsized":"2023-10-30T13:50:37.450196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'617","reported_seq":825,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.795368+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:52:23.795368+0000","last_peered":"2023-10-30T13:52:23.795368+0000","last_clean":"2023-10-30T13:52:23.795368+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:52:23.795368+0000","last_undegraded":"2023-10-30T13:52:23.795368+0000","last_fullsized":"2023-10-30T13:52:23.795368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360178,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":232,"num_read_kb":148904,"num_write":693,"num_write_kb":62690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1532","reported_seq":3445,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.667425+0000","last_change":"2023-10-30T13:21:40.574205+0000","last_active":"2023-10-30T13:52:25.667425+0000","last_peered":"2023-10-30T13:52:25.667425+0000","last_clean":"2023-10-30T13:52:25.667425+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:52:25.667425+0000","last_undegraded":"2023-10-30T13:52:25.667425+0000","last_fullsized":"2023-10-30T13:52:25.667425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1532,"log_dups_size":0,"ondisk_log_size":1532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:45:44.062637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057314540000000004,"stat_sum":{"num_bytes":22780711,"num_objects":633,"num_object_clones":299,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":243,"num_read":92,"num_read_kb":12,"num_write":1291,"num_write_kb":28237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"67'705","reported_seq":936,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.241719+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:52:19.241719+0000","last_peered":"2023-10-30T13:52:19.241719+0000","last_clean":"2023-10-30T13:52:19.241719+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:52:19.241719+0000","last_undegraded":"2023-10-30T13:52:19.241719+0000","last_fullsized":"2023-10-30T13:52:19.241719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":256,"num_read_kb":247261,"num_write":738,"num_write_kb":70569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1453","reported_seq":3390,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.925643+0000","last_change":"2023-10-30T13:21:40.528051+0000","last_active":"2023-10-30T13:52:23.925643+0000","last_peered":"2023-10-30T13:52:23.925643+0000","last_clean":"2023-10-30T13:52:23.925643+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:52:23.925643+0000","last_undegraded":"2023-10-30T13:52:23.925643+0000","last_fullsized":"2023-10-30T13:52:23.925643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1453,"log_dups_size":0,"ondisk_log_size":1453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:08:49.848069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002406114,"stat_sum":{"num_bytes":14184993,"num_objects":568,"num_object_clones":263,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":210,"num_read":80,"num_read_kb":26,"num_write":1239,"num_write_kb":16314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'2040","reported_seq":3405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.224696+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:52:25.224696+0000","last_peered":"2023-10-30T13:52:25.224696+0000","last_clean":"2023-10-30T13:52:25.224696+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:52:25.224696+0000","last_undegraded":"2023-10-30T13:52:25.224696+0000","last_fullsized":"2023-10-30T13:52:25.224696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41670866,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":1534,"num_read_kb":188566,"num_write":2090,"num_write_kb":83865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1512","reported_seq":3453,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.919153+0000","last_change":"2023-10-30T13:21:40.568287+0000","last_active":"2023-10-30T13:52:23.919153+0000","last_peered":"2023-10-30T13:52:23.919153+0000","last_clean":"2023-10-30T13:52:23.919153+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:52:23.919153+0000","last_undegraded":"2023-10-30T13:52:23.919153+0000","last_fullsized":"2023-10-30T13:52:23.919153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:16.792299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002908281,"stat_sum":{"num_bytes":22417951,"num_objects":612,"num_object_clones":284,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":221,"num_read":105,"num_read_kb":44,"num_write":1296,"num_write_kb":23585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'947","reported_seq":1222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.180646+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:52:24.180646+0000","last_peered":"2023-10-30T13:52:24.180646+0000","last_clean":"2023-10-30T13:52:24.180646+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:52:24.180646+0000","last_undegraded":"2023-10-30T13:52:24.180646+0000","last_fullsized":"2023-10-30T13:52:24.180646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":355,"num_read_kb":252215,"num_write":1073,"num_write_kb":75653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1520","reported_seq":3474,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.663361+0000","last_change":"2023-10-30T13:21:40.536723+0000","last_active":"2023-10-30T13:52:25.663361+0000","last_peered":"2023-10-30T13:52:25.663361+0000","last_clean":"2023-10-30T13:52:25.663361+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:52:25.663361+0000","last_undegraded":"2023-10-30T13:52:25.663361+0000","last_fullsized":"2023-10-30T13:52:25.663361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:33:25.077240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007710705,"stat_sum":{"num_bytes":23751577,"num_objects":575,"num_object_clones":266,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":199,"num_read":100,"num_read_kb":489,"num_write":1325,"num_write_kb":27587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'943","reported_seq":1183,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.547312+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:52:18.547312+0000","last_peered":"2023-10-30T13:52:18.547312+0000","last_clean":"2023-10-30T13:52:18.547312+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:52:18.547312+0000","last_undegraded":"2023-10-30T13:52:18.547312+0000","last_fullsized":"2023-10-30T13:52:18.547312+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":294,"num_read_kb":227180,"num_write":1037,"num_write_kb":87579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1589","reported_seq":3499,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923728+0000","last_change":"2023-10-30T13:21:40.539081+0000","last_active":"2023-10-30T13:52:23.923728+0000","last_peered":"2023-10-30T13:52:23.923728+0000","last_clean":"2023-10-30T13:52:23.923728+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:52:23.923728+0000","last_undegraded":"2023-10-30T13:52:23.923728+0000","last_fullsized":"2023-10-30T13:52:23.923728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1589,"log_dups_size":0,"ondisk_log_size":1589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:36.291322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010719574000000001,"stat_sum":{"num_bytes":35538242,"num_objects":650,"num_object_clones":301,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":240,"num_read":91,"num_read_kb":31,"num_write":1345,"num_write_kb":41412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'785","reported_seq":1010,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.206722+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:52:24.206722+0000","last_peered":"2023-10-30T13:52:24.206722+0000","last_clean":"2023-10-30T13:52:24.206722+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:52:24.206722+0000","last_undegraded":"2023-10-30T13:52:24.206722+0000","last_fullsized":"2023-10-30T13:52:24.206722+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":258,"num_read_kb":177173,"num_write":842,"num_write_kb":62518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1541","reported_seq":3467,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.919237+0000","last_change":"2023-10-30T13:21:40.572129+0000","last_active":"2023-10-30T13:52:23.919237+0000","last_peered":"2023-10-30T13:52:23.919237+0000","last_clean":"2023-10-30T13:52:23.919237+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:52:23.919237+0000","last_undegraded":"2023-10-30T13:52:23.919237+0000","last_fullsized":"2023-10-30T13:52:23.919237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1541,"log_dups_size":0,"ondisk_log_size":1541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:23.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059276069999999997,"stat_sum":{"num_bytes":23389062,"num_objects":631,"num_object_clones":294,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":232,"num_read":100,"num_read_kb":25,"num_write":1313,"num_write_kb":24315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'914","reported_seq":1153,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.107011+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:52:19.107011+0000","last_peered":"2023-10-30T13:52:19.107011+0000","last_clean":"2023-10-30T13:52:19.107011+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:52:19.107011+0000","last_undegraded":"2023-10-30T13:52:19.107011+0000","last_fullsized":"2023-10-30T13:52:19.107011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":310,"num_read_kb":109156,"num_write":1491,"num_write_kb":61820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1404","reported_seq":3356,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.931293+0000","last_change":"2023-10-30T13:21:40.576235+0000","last_active":"2023-10-30T13:52:23.931293+0000","last_peered":"2023-10-30T13:52:23.931293+0000","last_clean":"2023-10-30T13:52:23.931293+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:52:23.931293+0000","last_undegraded":"2023-10-30T13:52:23.931293+0000","last_fullsized":"2023-10-30T13:52:23.931293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1404,"log_dups_size":0,"ondisk_log_size":1404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:30.114376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056940879999999999,"stat_sum":{"num_bytes":23042560,"num_objects":571,"num_object_clones":259,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":201,"num_read":92,"num_read_kb":75,"num_write":1201,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'1095","reported_seq":1338,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.531235+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:52:16.531235+0000","last_peered":"2023-10-30T13:52:16.531235+0000","last_clean":"2023-10-30T13:52:16.531235+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:52:16.531235+0000","last_undegraded":"2023-10-30T13:52:16.531235+0000","last_fullsized":"2023-10-30T13:52:16.531235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":342,"num_read_kb":120967,"num_write":1229,"num_write_kb":51134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1436","reported_seq":3397,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.918796+0000","last_change":"2023-10-30T13:21:40.529108+0000","last_active":"2023-10-30T13:52:23.918796+0000","last_peered":"2023-10-30T13:52:23.918796+0000","last_clean":"2023-10-30T13:52:23.918796+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:52:23.918796+0000","last_undegraded":"2023-10-30T13:52:23.918796+0000","last_fullsized":"2023-10-30T13:52:23.918796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1436,"log_dups_size":0,"ondisk_log_size":1436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:16:51.856467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029926589999999999,"stat_sum":{"num_bytes":30813966,"num_objects":572,"num_object_clones":263,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":205,"num_read":105,"num_read_kb":87,"num_write":1219,"num_write_kb":33525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'421","reported_seq":607,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.800787+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:52:23.800787+0000","last_peered":"2023-10-30T13:52:23.800787+0000","last_clean":"2023-10-30T13:52:23.800787+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:52:23.800787+0000","last_undegraded":"2023-10-30T13:52:23.800787+0000","last_fullsized":"2023-10-30T13:52:23.800787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":209,"num_read_kb":100741,"num_write":482,"num_write_kb":29419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1533","reported_seq":3487,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.926144+0000","last_change":"2023-10-30T13:21:40.533380+0000","last_active":"2023-10-30T13:52:23.926144+0000","last_peered":"2023-10-30T13:52:23.926144+0000","last_clean":"2023-10-30T13:52:23.926144+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:52:23.926144+0000","last_undegraded":"2023-10-30T13:52:23.926144+0000","last_fullsized":"2023-10-30T13:52:23.926144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1533,"log_dups_size":0,"ondisk_log_size":1533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:46.427030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057673200000000003,"stat_sum":{"num_bytes":13994934,"num_objects":595,"num_object_clones":279,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":215,"num_read":114,"num_read_kb":29,"num_write":1327,"num_write_kb":19694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'1204","reported_seq":1418,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.263849+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:52:24.263849+0000","last_peered":"2023-10-30T13:52:24.263849+0000","last_clean":"2023-10-30T13:52:24.263849+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:52:24.263849+0000","last_undegraded":"2023-10-30T13:52:24.263849+0000","last_fullsized":"2023-10-30T13:52:24.263849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21903607,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":251,"num_read_kb":99282,"num_write":1273,"num_write_kb":39377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1544","reported_seq":3497,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.892716+0000","last_change":"2023-10-30T13:21:40.575380+0000","last_active":"2023-10-30T13:52:23.892716+0000","last_peered":"2023-10-30T13:52:23.892716+0000","last_clean":"2023-10-30T13:52:23.892716+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:52:23.892716+0000","last_undegraded":"2023-10-30T13:52:23.892716+0000","last_fullsized":"2023-10-30T13:52:23.892716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1544,"log_dups_size":0,"ondisk_log_size":1544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:12.098739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063570620000000001,"stat_sum":{"num_bytes":16293763,"num_objects":612,"num_object_clones":286,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":233,"num_read":120,"num_read_kb":127,"num_write":1302,"num_write_kb":21658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'900","reported_seq":1126,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.166933+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:52:24.166933+0000","last_peered":"2023-10-30T13:52:24.166933+0000","last_clean":"2023-10-30T13:52:24.166933+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:52:24.166933+0000","last_undegraded":"2023-10-30T13:52:24.166933+0000","last_fullsized":"2023-10-30T13:52:24.166933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":272,"num_read_kb":200216,"num_write":1131,"num_write_kb":59038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1424","reported_seq":3373,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.237106+0000","last_change":"2023-10-30T13:21:40.575515+0000","last_active":"2023-10-30T13:52:25.237106+0000","last_peered":"2023-10-30T13:52:25.237106+0000","last_clean":"2023-10-30T13:52:25.237106+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:52:25.237106+0000","last_undegraded":"2023-10-30T13:52:25.237106+0000","last_fullsized":"2023-10-30T13:52:25.237106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1424,"log_dups_size":0,"ondisk_log_size":1424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:20.385719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060201619999999999,"stat_sum":{"num_bytes":10033117,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":215,"num_read":92,"num_read_kb":24,"num_write":1202,"num_write_kb":13175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'967","reported_seq":1222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:21.614332+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:52:21.614332+0000","last_peered":"2023-10-30T13:52:21.614332+0000","last_clean":"2023-10-30T13:52:21.614332+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:52:21.614332+0000","last_undegraded":"2023-10-30T13:52:21.614332+0000","last_fullsized":"2023-10-30T13:52:21.614332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":318,"num_read_kb":202944,"num_write":1058,"num_write_kb":67603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1452","reported_seq":3400,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.930308+0000","last_change":"2023-10-30T13:21:40.575093+0000","last_active":"2023-10-30T13:52:23.930308+0000","last_peered":"2023-10-30T13:52:23.930308+0000","last_clean":"2023-10-30T13:52:23.930308+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:52:23.930308+0000","last_undegraded":"2023-10-30T13:52:23.930308+0000","last_fullsized":"2023-10-30T13:52:23.930308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1452,"log_dups_size":0,"ondisk_log_size":1452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:55:44.047461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079253199999999996,"stat_sum":{"num_bytes":6591050,"num_objects":558,"num_object_clones":254,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":201,"num_read":83,"num_read_kb":63,"num_write":1251,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1486","reported_seq":3430,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.929689+0000","last_change":"2023-10-30T13:21:40.538321+0000","last_active":"2023-10-30T13:52:23.929689+0000","last_peered":"2023-10-30T13:52:23.929689+0000","last_clean":"2023-10-30T13:52:23.929689+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:52:23.929689+0000","last_undegraded":"2023-10-30T13:52:23.929689+0000","last_fullsized":"2023-10-30T13:52:23.929689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:04.667732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010603414,"stat_sum":{"num_bytes":9104816,"num_objects":585,"num_object_clones":276,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":204,"num_read":104,"num_read_kb":44,"num_write":1283,"num_write_kb":14447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'1000","reported_seq":1254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.409685+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:52:18.409685+0000","last_peered":"2023-10-30T13:52:18.409685+0000","last_clean":"2023-10-30T13:52:18.409685+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:52:18.409685+0000","last_undegraded":"2023-10-30T13:52:18.409685+0000","last_fullsized":"2023-10-30T13:52:18.409685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":337,"num_read_kb":187117,"num_write":1194,"num_write_kb":47550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1610","reported_seq":3524,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.937747+0000","last_change":"2023-10-30T13:21:40.537202+0000","last_active":"2023-10-30T13:52:23.937747+0000","last_peered":"2023-10-30T13:52:23.937747+0000","last_clean":"2023-10-30T13:52:23.937747+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:52:23.937747+0000","last_undegraded":"2023-10-30T13:52:23.937747+0000","last_fullsized":"2023-10-30T13:52:23.937747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1610,"log_dups_size":0,"ondisk_log_size":1610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:39:17.448171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097894100000000001,"stat_sum":{"num_bytes":19085997,"num_objects":650,"num_object_clones":307,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":230,"num_read":105,"num_read_kb":202,"num_write":1384,"num_write_kb":24607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1264","reported_seq":1566,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.267745+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:52:24.267745+0000","last_peered":"2023-10-30T13:52:24.267745+0000","last_clean":"2023-10-30T13:52:24.267745+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:52:24.267745+0000","last_undegraded":"2023-10-30T13:52:24.267745+0000","last_fullsized":"2023-10-30T13:52:24.267745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1264,"log_dups_size":0,"ondisk_log_size":1264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34811142,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":418,"num_read_kb":304041,"num_write":1365,"num_write_kb":89722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1502","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.656168+0000","last_change":"2023-10-30T13:21:40.528576+0000","last_active":"2023-10-30T13:52:24.656168+0000","last_peered":"2023-10-30T13:52:24.656168+0000","last_clean":"2023-10-30T13:52:24.656168+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:52:24.656168+0000","last_undegraded":"2023-10-30T13:52:24.656168+0000","last_fullsized":"2023-10-30T13:52:24.656168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1502,"log_dups_size":0,"ondisk_log_size":1502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:58.014283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026802879999999999,"stat_sum":{"num_bytes":22843366,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":237,"num_read":96,"num_read_kb":23,"num_write":1281,"num_write_kb":25082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'773","reported_seq":1005,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.811859+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:52:23.811859+0000","last_peered":"2023-10-30T13:52:23.811859+0000","last_clean":"2023-10-30T13:52:23.811859+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:52:23.811859+0000","last_undegraded":"2023-10-30T13:52:23.811859+0000","last_fullsized":"2023-10-30T13:52:23.811859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":280,"num_read_kb":212075,"num_write":833,"num_write_kb":74728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1454","reported_seq":3408,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.933645+0000","last_change":"2023-10-30T13:21:40.576062+0000","last_active":"2023-10-30T13:52:23.933645+0000","last_peered":"2023-10-30T13:52:23.933645+0000","last_clean":"2023-10-30T13:52:23.933645+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:52:23.933645+0000","last_undegraded":"2023-10-30T13:52:23.933645+0000","last_fullsized":"2023-10-30T13:52:23.933645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1454,"log_dups_size":0,"ondisk_log_size":1454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:28.354972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055040289999999997,"stat_sum":{"num_bytes":14435337,"num_objects":589,"num_object_clones":269,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":212,"num_read":103,"num_read_kb":22,"num_write":1235,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'708","reported_seq":971,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.234153+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:52:19.234153+0000","last_peered":"2023-10-30T13:52:19.234153+0000","last_clean":"2023-10-30T13:52:19.234153+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:52:19.234153+0000","last_undegraded":"2023-10-30T13:52:19.234153+0000","last_fullsized":"2023-10-30T13:52:19.234153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749318,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":374,"num_read_kb":96620,"num_write":839,"num_write_kb":51020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165182927,"num_objects":39302,"num_object_clones":18317,"num_object_copies":78604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39302,"num_whiteouts":14355,"num_read":6653,"num_read_kb":168434,"num_write":83420,"num_write_kb":1459733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424070144,"data_stored":2332381086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97918,"ondisk_log_size":97918,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903181988,"num_objects":1685,"num_object_clones":0,"num_object_copies":3370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":11399,"num_read_kb":5963012,"num_write":33982,"num_write_kb":2068472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1396,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806508032,"data_stored":1806363976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30291,"ondisk_log_size":30291,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575358,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724736,"kb_used_data":571384,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93647104,"statfs":{"total":96636764160,"available":95894634496,"internally_reserved":0,"allocated":585097216,"data_stored":571472003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":157022667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.95099999999999996},"min":{"1min":0.47999999999999998,"5min":0.41199999999999998,"15min":0.40899999999999997},"max":{"1min":1.288,"5min":1.321,"15min":15.381},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81899999999999995,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.315},"max":{"1min":1.212,"5min":1.8400000000000001,"15min":2.9510000000000001},"last":0.77300000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.83799999999999997,"15min":0.92800000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3240000000000001,"5min":1.79,"15min":15.271000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89100000000000001,"15min":0.96199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.42999999999999999},"max":{"1min":1.2769999999999999,"5min":1.6000000000000001,"15min":15.228999999999999},"last":0.84299999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92100000000000004,"15min":0.91300000000000003},"min":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.36299999999999999},"max":{"1min":1.3100000000000001,"5min":1.345,"15min":2.2429999999999999},"last":1.056},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.89000000000000001,"15min":0.877},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.242,"5min":1.929,"15min":2.4540000000000002},"last":0.96899999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95599999999999996,"15min":1.0800000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.256,"5min":2.0249999999999999,"15min":15.324},"last":0.624},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.96999999999999997,"15min":0.98599999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2629999999999999,"5min":1.323,"15min":1.637},"last":1.012}]},{"osd":4,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.091,"15min":1.1479999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.40899999999999997},"max":{"1min":1.3080000000000001,"5min":2.7869999999999999,"15min":15.172000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.093,"5min":1.0600000000000001,"15min":1.089},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.466,"5min":3.032,"15min":4.0209999999999999},"last":1.1259999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1120000000000001,"15min":1.1060000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.454,"5min":1.4770000000000001,"15min":1.7050000000000001},"last":1.163},{"interface":"front","average":{"1min":1.089,"5min":1.0600000000000001,"15min":1.0169999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.54500000000000004},"max":{"1min":1.4159999999999999,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":1.105}]},{"osd":6,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.1819999999999999,"15min":1.2509999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.46500000000000002},"max":{"1min":1.7829999999999999,"5min":3.0350000000000001,"15min":15.427},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.228,"15min":1.331},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.42199999999999999},"max":{"1min":1.841,"5min":3.5169999999999999,"15min":32.976999999999997},"last":1.2869999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575357,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593340,"kb_used_data":449780,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93778500,"statfs":{"total":96636764160,"available":96029184000,"internally_reserved":0,"allocated":460574720,"data_stored":451977574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":146995655},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.55000000000000004,"5min":0.45000000000000001,"15min":0.40999999999999998},"max":{"1min":1.2190000000000001,"5min":1.3999999999999999,"15min":5.9880000000000004},"last":1.087},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.86899999999999999,"15min":0.89200000000000002},"min":{"1min":0.58999999999999997,"5min":0.48599999999999999,"15min":0.35999999999999999},"max":{"1min":1.0620000000000001,"5min":1.3160000000000001,"15min":6.0730000000000004},"last":0.90300000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.93600000000000005,"15min":0.86499999999999999},"min":{"1min":0.58199999999999996,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.585,"5min":6.9930000000000003,"15min":6.9930000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.86399999999999999,"15min":0.86099999999999999},"min":{"1min":0.69299999999999995,"5min":0.52500000000000002,"15min":0.47499999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.9810000000000001},"last":0.79200000000000004}]},{"osd":2,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":0.89400000000000002},"min":{"1min":0.59899999999999998,"5min":0.54400000000000004,"15min":0.47699999999999998},"max":{"1min":1.044,"5min":6.8550000000000004,"15min":6.8550000000000004},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.97999999999999998,"15min":0.96399999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.454,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.83399999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88600000000000001,"15min":0.89400000000000002},"min":{"1min":0.64800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.2130000000000001,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.96899999999999997,"15min":0.98699999999999999},"min":{"1min":0.69899999999999995,"5min":0.64400000000000002,"15min":0.56299999999999994},"max":{"1min":1.536,"5min":5.3179999999999996,"15min":5.3179999999999996},"last":0.72699999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.99099999999999999,"15min":1.046},"min":{"1min":0.59199999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.321,"5min":1.4970000000000001,"15min":15.372},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0629999999999999,"15min":1.05},"min":{"1min":0.79000000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"max":{"1min":1.391,"5min":5.2729999999999997,"15min":6.1580000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0329999999999999,"15min":1.042},"min":{"1min":0.749,"5min":0.60499999999999998,"15min":0.60199999999999998},"max":{"1min":1.2649999999999999,"5min":1.421,"15min":1.8080000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.95199999999999996,"5min":0.73899999999999999,"15min":0.67700000000000005},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.913},"last":0.93999999999999995}]},{"osd":7,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0860000000000001,"15min":1.0960000000000001},"min":{"1min":0.72099999999999997,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2290000000000001,"5min":1.577,"15min":2.75},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.212,"15min":1.1899999999999999},"min":{"1min":1.006,"5min":0.69599999999999995,"15min":0.42499999999999999},"max":{"1min":1.627,"5min":1.627,"15min":3.0590000000000002},"last":0.99299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575358,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672628,"kb_used_data":519276,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93699212,"statfs":{"total":96636764160,"available":95947993088,"internally_reserved":0,"allocated":531738624,"data_stored":518585685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":157022662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88900000000000001,"15min":0.87},"min":{"1min":0.51300000000000001,"5min":0.38400000000000001,"15min":0.32700000000000001},"max":{"1min":1.3300000000000001,"5min":1.4379999999999999,"15min":6.4729999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.88800000000000001,"15min":0.86899999999999999},"min":{"1min":0.56999999999999995,"5min":0.46800000000000003,"15min":0.39100000000000001},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":8.7560000000000002},"last":0.745}]},{"osd":1,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81499999999999995,"15min":0.88200000000000001},"min":{"1min":0.51400000000000001,"5min":0.439,"15min":0.42799999999999999},"max":{"1min":1.2,"5min":1.671,"15min":15.257999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.86099999999999999,"15min":0.94899999999999995},"min":{"1min":0.62,"5min":0.48699999999999999,"15min":0.47399999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":15.175000000000001},"last":0.82399999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.90100000000000002,"15min":0.91100000000000003},"min":{"1min":0.57499999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.228,"5min":1.458,"15min":6.5410000000000004},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.93700000000000006,"15min":0.98699999999999999},"min":{"1min":0.57599999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.52,"15min":6.7080000000000002},"last":1.042}]},{"osd":3,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.0880000000000001,"15min":1.0469999999999999},"min":{"1min":0.69399999999999995,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.3839999999999999,"5min":5.4249999999999998,"15min":6.7699999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.175,"15min":1.1659999999999999},"min":{"1min":0.88600000000000001,"5min":0.58699999999999997,"15min":0.40100000000000002},"max":{"1min":1.4790000000000001,"5min":5.5570000000000004,"15min":15.364000000000001},"last":1.1000000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:51:44 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.92500000000000004,"15min":0.97999999999999998},"min":{"1min":0.63600000000000001,"5min":0.376,"15min":0.36499999999999999},"max":{"1min":2.5830000000000002,"5min":4.3529999999999998,"15min":5.9720000000000004},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.27,"5min":1.246,"15min":1.1459999999999999},"min":{"1min":0.92900000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.5190000000000001,"5min":5.6609999999999996,"15min":14.314},"last":1.1970000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.0129999999999999,"15min":1.123},"min":{"1min":0.69299999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.4239999999999999,"5min":4.2290000000000001,"15min":15.06},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.097,"15min":1.1799999999999999},"min":{"1min":0.83699999999999997,"5min":0.65400000000000003,"15min":0.50900000000000001},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":7.6630000000000003},"last":1.0149999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.0609999999999999,"15min":1.1279999999999999},"min":{"1min":0.745,"5min":0.44,"15min":0.44},"max":{"1min":4.0599999999999996,"5min":5.0419999999999998,"15min":6.8259999999999996},"last":0.875},{"interface":"front","average":{"1min":1.429,"5min":1.2130000000000001,"15min":1.266},"min":{"1min":0.89700000000000002,"5min":0.53500000000000003,"15min":0.46200000000000002},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":19.321999999999999},"last":1.151}]}]},{"osd":4,"up_from":13,"seq":55834575359,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717128,"kb_used_data":569408,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93654712,"statfs":{"total":96636764160,"available":95902425088,"internally_reserved":0,"allocated":583073792,"data_stored":568869284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.51800000000000002,"5min":0.48599999999999999,"15min":0.28100000000000003},"max":{"1min":1.087,"5min":2.202,"15min":9.6690000000000005},"last":1.1839999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85199999999999998,"15min":0.84799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47399999999999998,"15min":0.377},"max":{"1min":1.2010000000000001,"5min":2.0760000000000001,"15min":2.786},"last":0.73299999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.873,"15min":0.88200000000000001},"min":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.41399999999999998},"max":{"1min":1.2170000000000001,"5min":1.8160000000000001,"15min":5.5119999999999996},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.89500000000000002,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.51500000000000001,"15min":0.44700000000000001},"max":{"1min":1.202,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":1.2110000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:51:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.89200000000000002,"15min":0.878},"min":{"1min":0.63200000000000001,"5min":0.53700000000000003,"15min":0.45100000000000001},"max":{"1min":1.212,"5min":2.1549999999999998,"15min":9.0099999999999998},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.95099999999999996,"15min":0.95199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0569999999999999,"5min":7.7359999999999998,"15min":7.7359999999999998},"last":0.97499999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.94499999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.52500000000000002},"max":{"1min":1.1899999999999999,"5min":1.6599999999999999,"15min":4.7080000000000002},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.026,"5min":1.0580000000000001,"15min":1.0209999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.48699999999999999},"max":{"1min":1.2649999999999999,"5min":1.7509999999999999,"15min":3.2749999999999999},"last":1.153}]},{"osd":5,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0249999999999999,"15min":1.0489999999999999},"min":{"1min":0.56299999999999994,"5min":0.52300000000000002,"15min":0.51500000000000001},"max":{"1min":1.3799999999999999,"5min":1.621,"15min":7.8529999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.214,"15min":1.161},"min":{"1min":0.86899999999999999,"5min":0.627,"15min":0.40999999999999998},"max":{"1min":1.538,"5min":7.694,"15min":8.2070000000000007},"last":1.266}]},{"osd":6,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1379999999999999,"15min":1.081},"min":{"1min":0.745,"5min":0.68999999999999995,"15min":0.52600000000000002},"max":{"1min":1.458,"5min":7.7880000000000003,"15min":7.7880000000000003},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.038,"5min":1.1659999999999999,"15min":1.155},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.2929999999999999,"5min":3.7320000000000002,"15min":5.6130000000000004},"last":1.081}]},{"osd":7,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.1950000000000001,"15min":1.1739999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.52000000000000002},"max":{"1min":1.6120000000000001,"5min":7.8380000000000001,"15min":10.715},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0720000000000001,"15min":1.157},"min":{"1min":0.627,"5min":0.627,"15min":0.47199999999999998},"max":{"1min":1.1579999999999999,"5min":2.2450000000000001,"15min":4.3230000000000004},"last":1.119}]}]},{"osd":3,"up_from":13,"seq":55834575357,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707492,"kb_used_data":550044,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93664348,"statfs":{"total":96636764160,"available":95912292352,"internally_reserved":0,"allocated":563245056,"data_stored":549099638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":161216968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.69799999999999995},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64900000000000002,"5min":0.76700000000000002,"15min":1.5429999999999999},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.51000000000000001,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":0.755,"5min":0.76200000000000001,"15min":0.94099999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.28599999999999998,"15min":0.246},"max":{"1min":0.78900000000000003,"5min":0.92000000000000004,"15min":1.018},"last":0.373}]},{"osd":2,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.65100000000000002,"5min":1.056,"15min":1.4690000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.86499999999999999,"15min":1.6799999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.79600000000000004,"15min":0.83599999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.49099999999999999},"max":{"1min":0.96499999999999997,"5min":1.1739999999999999,"15min":1.5900000000000001},"last":0.747},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.84299999999999997,"15min":0.85299999999999998},"min":{"1min":0.66100000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0129999999999999,"5min":1.1419999999999999,"15min":1.3999999999999999},"last":0.76100000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.60799999999999998,"5min":0.57399999999999995,"15min":0.51800000000000002},"max":{"1min":1.095,"5min":1.1879999999999999,"15min":1.5069999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.83599999999999997,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.58599999999999997,"15min":0.5},"max":{"1min":1.232,"5min":1.232,"15min":1.365},"last":0.621}]},{"osd":6,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.88700000000000001,"15min":0.82799999999999996},"min":{"1min":0.66900000000000004,"5min":0.52800000000000002,"15min":0.45400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.88500000000000001,"15min":0.82699999999999996},"min":{"1min":0.69499999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":2.3410000000000002},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.88800000000000001,"15min":0.88200000000000001},"min":{"1min":0.68300000000000005,"5min":0.57799999999999996,"15min":0.51700000000000002},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":2.9449999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.872,"15min":0.89300000000000002},"min":{"1min":0.64300000000000002,"5min":0.54400000000000004,"15min":0.47099999999999997},"max":{"1min":1.343,"5min":1.343,"15min":2.9729999999999999},"last":0.77300000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575360,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484724,"kb_used_data":377260,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93887116,"statfs":{"total":96636764160,"available":96140406784,"internally_reserved":0,"allocated":386314240,"data_stored":377764906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.47999999999999998,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.37},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.50800000000000001,"15min":0.48199999999999998},"min":{"1min":0.41799999999999998,"5min":0.28299999999999997,"15min":0.25700000000000001},"max":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.45900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.58399999999999996,"15min":0.54000000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.0060000000000002,"5min":7.0060000000000002,"15min":7.0060000000000002},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.36499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":1.637},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.622,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.58399999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.78000000000000003,"15min":0.80300000000000005},"min":{"1min":0.59699999999999998,"5min":0.442,"15min":0.442},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.80500000000000005,"15min":0.82699999999999996},"min":{"1min":0.61399999999999999,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":4.1479999999999997,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.88500000000000001,"15min":0.877},"min":{"1min":0.754,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5169999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.80900000000000005,"15min":0.82499999999999996},"min":{"1min":0.58599999999999997,"5min":0.45300000000000001,"15min":0.41999999999999998},"max":{"1min":1.171,"5min":2.762,"15min":2.762},"last":0.81200000000000006}]},{"osd":6,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77400000000000002,"15min":0.79000000000000004},"min":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.47699999999999998},"max":{"1min":0.98699999999999999,"5min":1.6419999999999999,"15min":3.1659999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.84399999999999997,"15min":0.81699999999999995},"min":{"1min":0.65500000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.8210000000000002,"5min":3.8210000000000002,"15min":3.8210000000000002},"last":0.68400000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":0.90800000000000003,"15min":0.85699999999999998},"min":{"1min":0.69599999999999995,"5min":0.55000000000000004,"15min":0.503},"max":{"1min":6.9059999999999997,"5min":6.9059999999999997,"15min":6.9059999999999997},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.0009999999999999,"15min":0.95499999999999996},"min":{"1min":0.81299999999999994,"5min":0.63100000000000001,"15min":0.55900000000000005},"max":{"1min":6.9509999999999996,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.88200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575357,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671940,"kb_used_data":532604,"kb_used_omap":1,"kb_used_meta":139326,"kb_avail":93699900,"statfs":{"total":96636764160,"available":95948697600,"internally_reserved":0,"allocated":545386496,"data_stored":536307441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142670278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30399999999999999,"5min":0.24299999999999999,"15min":0.20799999999999999},"max":{"1min":0.58799999999999997,"5min":0.94999999999999996,"15min":1.1359999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.28299999999999997,"15min":0.251},"max":{"1min":0.64900000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49099999999999999,"15min":0.501},"min":{"1min":0.318,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66800000000000004,"5min":1.002,"15min":2.8660000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.501,"15min":0.48599999999999999},"min":{"1min":0.36899999999999999,"5min":0.27700000000000002,"15min":0.249},"max":{"1min":0.69099999999999995,"5min":1.0589999999999999,"15min":2.6739999999999999},"last":0.625}]},{"osd":3,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.53000000000000003},"min":{"1min":0.373,"5min":0.35099999999999998,"15min":0.314},"max":{"1min":0.73899999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.318},"max":{"1min":0.755,"5min":1.306,"15min":2.71},"last":0.66800000000000004}]},{"osd":4,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82399999999999995,"15min":0.85599999999999998},"min":{"1min":0.622,"5min":0.52400000000000002,"15min":0.40200000000000002},"max":{"1min":1.0489999999999999,"5min":1.2210000000000001,"15min":6.7789999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.92200000000000004},"min":{"1min":0.76500000000000001,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":1.093,"5min":1.2629999999999999,"15min":6.8259999999999996},"last":0.88700000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":0.76200000000000001},"min":{"1min":0.65800000000000003,"5min":0.53800000000000003,"15min":0.47199999999999998},"max":{"1min":0.96499999999999997,"5min":2.133,"15min":2.5750000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.627,"5min":0.53800000000000003,"15min":0.51000000000000001},"max":{"1min":1.0229999999999999,"5min":1.3859999999999999,"15min":2.6179999999999999},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Oct 30 13:51:46 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91800000000000004,"15min":0.89500000000000002},"min":{"1min":0.80000000000000004,"5min":0.61199999999999999,"15min":0.501},"max":{"1min":1.1459999999999999,"5min":1.3999999999999999,"15min":6.0289999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.96599999999999997,"15min":0.95499999999999996},"min":{"1min":0.65500000000000003,"5min":0.625,"15min":0.60299999999999998},"max":{"1min":1.218,"5min":1.4399999999999999,"15min":6.0739999999999998},"last":0.95699999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79500000000000004,"15min":0.78300000000000003},"min":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.47899999999999998},"max":{"1min":1.0369999999999999,"5min":1.161,"15min":2.1579999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.879,"15min":0.88500000000000001},"min":{"1min":0.70899999999999996,"5min":0.55200000000000005,"15min":0.54800000000000004},"max":{"1min":1.1339999999999999,"5min":1.3440000000000001,"15min":6.0519999999999996},"last":0.998}]}]},{"osd":0,"up_from":13,"seq":55834575359,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712496,"kb_used_data":568936,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93659344,"statfs":{"total":96636764160,"available":95907168256,"internally_reserved":0,"allocated":582590464,"data_stored":569154632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.48399999999999999},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.214},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":16.184000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.50800000000000001,"15min":0.59199999999999997},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.28899999999999998},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":16.228000000000002},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.44900000000000001,"15min":0.64100000000000001},"min":{"1min":0.33300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":2.7999999999999998,"5min":2.7999999999999998,"15min":17.454000000000001},"last":0.496},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.65300000000000002},"min":{"1min":0.318,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":1.391,"5min":1.391,"15min":23.212},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.37,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":16.103999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.53100000000000003,"15min":0.73099999999999998},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.32000000000000001},"max":{"1min":2.8340000000000001,"5min":2.8340000000000001,"15min":23.16},"last":0.57799999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:51:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.83299999999999996,"15min":0.92700000000000005},"min":{"1min":0.48499999999999999,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":7.6900000000000004,"5min":7.6900000000000004,"15min":16.309000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.85799999999999998,"15min":0.97499999999999998},"min":{"1min":0.61199999999999999,"5min":0.53800000000000003,"15min":0.46000000000000002},"max":{"1min":2.5939999999999999,"5min":3.4009999999999998,"15min":16.343},"last":0.97199999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85299999999999998,"15min":1.006},"min":{"1min":0.63,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.02,"5min":2.02,"15min":23.266999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.86299999999999999,"15min":1.0329999999999999},"min":{"1min":0.755,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.7240000000000002,"5min":2.7240000000000002,"15min":16.268000000000001},"last":0.90700000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.80500000000000005,"15min":1.1120000000000001},"min":{"1min":0.61099999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":29.774999999999999},"last":0.754},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81499999999999995,"15min":1.0229999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.50700000000000001},"max":{"1min":2.75,"5min":2.75,"15min":17.5},"last":0.92700000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.81999999999999995,"15min":1.202},"min":{"1min":0.66700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.694,"5min":2.694,"15min":47.534999999999997},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.85999999999999999,"15min":1.155},"min":{"1min":0.64100000000000001,"5min":0.60399999999999998,"15min":0.54700000000000004},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":32.476999999999997},"last":0.83199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230703104,"data_stored":230687380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255741952,"data_stored":255719205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179036160,"data_stored":179013521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230858752,"data_stored":230840931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":197275648,"data_stored":197263919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":205541376,"data_stored":205522290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266067968,"data_stored":266051919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":241283072,"data_stored":241264811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350515200,"data_stored":337464605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288862208,"data_stored":280174938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":206495744,"data_stored":198338087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331603968,"data_stored":317845409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":385015808,"data_stored":371192067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":325414912,"data_stored":312650097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193724416,"data_stored":185512357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342437888,"data_stored":329203526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:52:30.321 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:52:30.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689ebdd640 1 Processor -- start 2023-10-30T13:52:30.494 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689ebdd640 1 -- start start 2023-10-30T13:52:30.495 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689ebdd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68981538f0 0x7f6898159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689ebdd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68980cefa0 0x7f68980cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689ebdd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68980ca6a0 con 0x7f689815cd50 2023-10-30T13:52:30.496 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689ebdd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689815d300 con 0x7f68980cefa0 2023-10-30T13:52:30.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689ebdd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689815d5e0 con 0x7f68981538f0 2023-10-30T13:52:30.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689d3da640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f689815cd50 0x7f68980c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41676/0 (socket says 172.21.15.148:41676) 2023-10-30T13:52:30.497 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689d3da640 1 -- 172.21.15.148:0/666350865 learned_addr learned my addr 172.21.15.148:0/666350865 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:30.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689e3dc640 1 --2- 172.21.15.148:0/666350865 >> v2:172.21.15.150:3300/0 conn(0x7f68980cefa0 0x7f68980cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689dbdb640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68981538f0 0x7f6898159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.499 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268479090 0 0) 0x7f68980ca6a0 con 0x7f689815cd50 2023-10-30T13:52:30.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6884003530 con 0x7f689815cd50 2023-10-30T13:52:30.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635626169 0 0) 0x7f6884003530 con 0x7f689815cd50 2023-10-30T13:52:30.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68981538f0 msgr2=0x7f6898159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:30.500 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689cbd9640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68981538f0 0x7f6898159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 >> v2:172.21.15.150:3300/0 conn(0x7f68980cefa0 msgr2=0x7f68980cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:30.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689cbd9640 1 --2- 172.21.15.148:0/666350865 >> v2:172.21.15.150:3300/0 conn(0x7f68980cefa0 0x7f68980cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689815d8c0 con 0x7f689815cd50 2023-10-30T13:52:30.501 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.490+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7f6888001680 con 0x7f689815cd50 2023-10-30T13:52:30.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 >> v1:172.21.15.148:6789/0 conn(0x7f689815cd50 legacy=0x7f68980c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689cbd9640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6884003980 0x7f6884003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.502 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689cbd9640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6884004370 0x7f6884004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689cbd9640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f688400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689815d300 con 0x7f6884004ec0 2023-10-30T13:52:30.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689815d5e0 con 0x7f6884004370 2023-10-30T13:52:30.503 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688400bed0 con 0x7f6884003980 2023-10-30T13:52:30.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689e3dc640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f688400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689e3dc640 1 -- 172.21.15.148:0/666350865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6884003980 msgr2=0x7f6884003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:30.504 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689e3dc640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6884003980 0x7f6884003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689e3dc640 1 -- 172.21.15.148:0/666350865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6884004370 msgr2=0x7f6884004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:30.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689e3dc640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6884004370 0x7f6884004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.505 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689e3dc640 1 -- 172.21.15.148:0/666350865 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6884003530 con 0x7f6884004ec0 2023-10-30T13:52:30.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689e3dc640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f688400b990 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f689000e170 tx=0x7f689001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c3aa68a0c51fd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:30.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68900a5c90 con 0x7f6884004ec0 2023-10-30T13:52:30.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.494+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689000f040 con 0x7f6884004ec0 2023-10-30T13:52:30.509 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689cbd9640 1 -- 172.21.15.148:0/666350865 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68900ad440 con 0x7f6884004ec0 2023-10-30T13:52:30.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 -- 172.21.15.148:0/666350865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 msgr2=0x7f688400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 --2- 172.21.15.148:0/666350865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f688400b990 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f689000e170 tx=0x7f689001cc80 comp rx=0 tx=0).stop 2023-10-30T13:52:30.510 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689e3dc640 1 -- 172.21.15.148:0/666350865 reap_dead start 2023-10-30T13:52:30.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 -- 172.21.15.148:0/666350865 shutdown_connections 2023-10-30T13:52:30.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 -- 172.21.15.148:0/666350865 >> 172.21.15.148:0/666350865 conn(0x7f689806a640 msgr2=0x7f6898065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:30.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 -- 172.21.15.148:0/666350865 shutdown_connections 2023-10-30T13:52:30.511 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 -- 172.21.15.148:0/666350865 wait complete. 2023-10-30T13:52:30.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 Processor -- start 2023-10-30T13:52:30.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 -- start start 2023-10-30T13:52:30.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f6898144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.512 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6898152d00 0x7f68980b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68980b3b50 0x7f68981f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68980b3180 con 0x7f6884004ec0 2023-10-30T13:52:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68980b3fd0 con 0x7f6898152d00 2023-10-30T13:52:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689ebdd640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6898146100 con 0x7f68980b3b50 2023-10-30T13:52:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689dbdb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f6898144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689dbdb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f6898144490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56666/0 (socket says 172.21.15.148:56666) 2023-10-30T13:52:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689dbdb640 1 -- 172.21.15.148:0/1094426736 learned_addr learned my addr 172.21.15.148:0/1094426736 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689d3da640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6898152d00 0x7f68980b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.498+0000 7f689e3dc640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68980b3b50 0x7f68981f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689dbdb640 1 -- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68980b3b50 msgr2=0x7f68981f2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689dbdb640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68980b3b50 0x7f68981f2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.515 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689dbdb640 1 -- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6898152d00 msgr2=0x7f68980b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689dbdb640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6898152d00 0x7f68980b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689dbdb640 1 -- 172.21.15.148:0/1094426736 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6898144e40 con 0x7f6884004ec0 2023-10-30T13:52:30.516 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689dbdb640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f6898144490 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f689400e7a0 tx=0x7f689400ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:30.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f688e7fc640 1 -- 172.21.15.148:0/1094426736 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689400c6b0 con 0x7f6884004ec0 2023-10-30T13:52:30.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f688e7fc640 1 -- 172.21.15.148:0/1094426736 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689400c830 con 0x7f6884004ec0 2023-10-30T13:52:30.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f688e7fc640 1 -- 172.21.15.148:0/1094426736 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689400cbe0 con 0x7f6884004ec0 2023-10-30T13:52:30.517 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6898145120 con 0x7f6884004ec0 2023-10-30T13:52:30.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689815b580 con 0x7f6884004ec0 2023-10-30T13:52:30.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689804e170 con 0x7f6884004ec0 2023-10-30T13:52:30.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f688e7fc640 1 -- 172.21.15.148:0/1094426736 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f689400cd40 con 0x7f6884004ec0 2023-10-30T13:52:30.518 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f688e7fc640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f687c0664a0 0x7f687c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f688e7fc640 1 -- 172.21.15.148:0/1094426736 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f68940846e0 con 0x7f6884004ec0 2023-10-30T13:52:30.519 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.502+0000 7f689d3da640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f687c0664a0 0x7f687c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.506+0000 7f689d3da640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f687c0664a0 0x7f687c068960 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f68880017b0 tx=0x7f6888001660 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:30.520 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.506+0000 7f688e7fc640 1 -- 172.21.15.148:0/1094426736 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6894051290 con 0x7f6884004ec0 2023-10-30T13:52:30.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.654+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6898159a90 con 0x7f687c0664a0 2023-10-30T13:52:30.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.674+0000 7f688e7fc640 1 -- 172.21.15.148:0/1094426736 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293908 (secure 0 0 0) 0x7f688800e870 con 0x7f687c0664a0 2023-10-30T13:52:30.682 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:52:30.684 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:52:30.685 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f687c0664a0 msgr2=0x7f687c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f687c0664a0 0x7f687c068960 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f68880017b0 tx=0x7f6888001660 comp rx=0 tx=0).stop 2023-10-30T13:52:30.686 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 msgr2=0x7f6898144490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f6898144490 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f689400e7a0 tx=0x7f689400ec60 comp rx=0 tx=0).stop 2023-10-30T13:52:30.689 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 shutdown_connections 2023-10-30T13:52:30.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f687c0664a0 0x7f687c068960 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f68980b3b50 0x7f68981f2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.690 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6898152d00 0x7f68980b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 --2- 172.21.15.148:0/1094426736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6884004ec0 0x7f6898144490 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 >> 172.21.15.148:0/1094426736 conn(0x7f689806a640 msgr2=0x7f68980c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:30.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 shutdown_connections 2023-10-30T13:52:30.691 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.678+0000 7f689ebdd640 1 -- 172.21.15.148:0/1094426736 wait complete. 2023-10-30T13:52:30.703 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1323,"stamp":"2023-10-30T13:52:29.202282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068955283,"num_objects":40989,"num_object_clones":18317,"num_object_copies":81978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40989,"num_whiteouts":14355,"num_read":18168,"num_read_kb":6131546,"num_write":117561,"num_write_kb":3530621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1396,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128326,"ondisk_log_size":128326,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5284484,"kb_used_data":4138692,"kb_used_omap":12,"kb_used_meta":1145715,"kb_avail":749690236,"statfs":{"total":773094113280,"available":767682801664,"internally_reserved":0,"allocated":4238020608,"data_stored":4143231163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1173212728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":100,"apply_latency_ms":100,"commit_latency_ns":100000000,"apply_latency_ns":100000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":56824291,"num_objects":745,"num_object_clones":253,"num_object_copies":1490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":745,"num_whiteouts":242,"num_read":298,"num_read_kb":66939,"num_write":972,"num_write_kb":56191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004500"},"pg_stats":[{"pgid":"3.3f","version":"67'1458","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.951630+0000","last_change":"2023-10-30T13:21:40.584822+0000","last_active":"2023-10-30T13:52:23.951630+0000","last_peered":"2023-10-30T13:52:23.951630+0000","last_clean":"2023-10-30T13:52:23.951630+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:52:23.951630+0000","last_undegraded":"2023-10-30T13:52:23.951630+0000","last_fullsized":"2023-10-30T13:52:23.951630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:13.082992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014806453000000001,"stat_sum":{"num_bytes":11269044,"num_objects":569,"num_object_clones":260,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":205,"num_read":107,"num_read_kb":121,"num_write":1253,"num_write_kb":17103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1464","reported_seq":3387,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.234533+0000","last_change":"2023-10-30T13:21:40.572202+0000","last_active":"2023-10-30T13:52:25.234533+0000","last_peered":"2023-10-30T13:52:25.234533+0000","last_clean":"2023-10-30T13:52:25.234533+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:52:25.234533+0000","last_undegraded":"2023-10-30T13:52:25.234533+0000","last_fullsized":"2023-10-30T13:52:25.234533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:58.516621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059691099999999997,"stat_sum":{"num_bytes":30975930,"num_objects":597,"num_object_clones":279,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":225,"num_read":82,"num_read_kb":30,"num_write":1226,"num_write_kb":32475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1518","reported_seq":3465,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.166473+0000","last_change":"2023-10-30T13:21:40.528480+0000","last_active":"2023-10-30T13:52:24.166473+0000","last_peered":"2023-10-30T13:52:24.166473+0000","last_clean":"2023-10-30T13:52:24.166473+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:52:24.166473+0000","last_undegraded":"2023-10-30T13:52:24.166473+0000","last_fullsized":"2023-10-30T13:52:24.166473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:03.253931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025608710000000002,"stat_sum":{"num_bytes":18704388,"num_objects":617,"num_object_clones":279,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":225,"num_read":106,"num_read_kb":8300,"num_write":1300,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1566","reported_seq":3502,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.918237+0000","last_change":"2023-10-30T13:21:40.567224+0000","last_active":"2023-10-30T13:52:23.918237+0000","last_peered":"2023-10-30T13:52:23.918237+0000","last_clean":"2023-10-30T13:52:23.918237+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:52:23.918237+0000","last_undegraded":"2023-10-30T13:52:23.918237+0000","last_fullsized":"2023-10-30T13:52:23.918237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1566,"log_dups_size":0,"ondisk_log_size":1566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:34.794112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021564890000000001,"stat_sum":{"num_bytes":23329933,"num_objects":630,"num_object_clones":293,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":227,"num_read":112,"num_read_kb":4378,"num_write":1354,"num_write_kb":25006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1582","reported_seq":3502,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.930705+0000","last_change":"2023-10-30T13:21:40.579191+0000","last_active":"2023-10-30T13:52:23.930705+0000","last_peered":"2023-10-30T13:52:23.930705+0000","last_clean":"2023-10-30T13:52:23.930705+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:52:23.930705+0000","last_undegraded":"2023-10-30T13:52:23.930705+0000","last_fullsized":"2023-10-30T13:52:23.930705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1582,"log_dups_size":0,"ondisk_log_size":1582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:10.057619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011808077,"stat_sum":{"num_bytes":16184828,"num_objects":654,"num_object_clones":305,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":251,"num_read":109,"num_read_kb":152,"num_write":1339,"num_write_kb":20879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1451","reported_seq":3398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932299+0000","last_change":"2023-10-30T13:21:40.539757+0000","last_active":"2023-10-30T13:52:23.932299+0000","last_peered":"2023-10-30T13:52:23.932299+0000","last_clean":"2023-10-30T13:52:23.932299+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:52:23.932299+0000","last_undegraded":"2023-10-30T13:52:23.932299+0000","last_fullsized":"2023-10-30T13:52:23.932299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1451,"log_dups_size":0,"ondisk_log_size":1451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:59.387758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01102786,"stat_sum":{"num_bytes":18372801,"num_objects":593,"num_object_clones":277,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":222,"num_read":108,"num_read_kb":4137,"num_write":1217,"num_write_kb":19358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1501","reported_seq":3438,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.927654+0000","last_change":"2023-10-30T13:21:40.580121+0000","last_active":"2023-10-30T13:52:23.927654+0000","last_peered":"2023-10-30T13:52:23.927654+0000","last_clean":"2023-10-30T13:52:23.927654+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:52:23.927654+0000","last_undegraded":"2023-10-30T13:52:23.927654+0000","last_fullsized":"2023-10-30T13:52:23.927654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:42.759583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012047337,"stat_sum":{"num_bytes":9859762,"num_objects":598,"num_object_clones":275,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":211,"num_read":96,"num_read_kb":35,"num_write":1295,"num_write_kb":11866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1450","reported_seq":3406,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.168248+0000","last_change":"2023-10-30T13:21:40.575154+0000","last_active":"2023-10-30T13:52:24.168248+0000","last_peered":"2023-10-30T13:52:24.168248+0000","last_clean":"2023-10-30T13:52:24.168248+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:52:24.168248+0000","last_undegraded":"2023-10-30T13:52:24.168248+0000","last_fullsized":"2023-10-30T13:52:24.168248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1450,"log_dups_size":0,"ondisk_log_size":1450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:28.268146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063254060000000004,"stat_sum":{"num_bytes":13694423,"num_objects":577,"num_object_clones":268,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":211,"num_read":108,"num_read_kb":104,"num_write":1228,"num_write_kb":23250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1510","reported_seq":3452,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.243551+0000","last_change":"2023-10-30T13:21:40.584916+0000","last_active":"2023-10-30T13:52:25.243551+0000","last_peered":"2023-10-30T13:52:25.243551+0000","last_clean":"2023-10-30T13:52:25.243551+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:52:25.243551+0000","last_undegraded":"2023-10-30T13:52:25.243551+0000","last_fullsized":"2023-10-30T13:52:25.243551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1510,"log_dups_size":0,"ondisk_log_size":1510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:08:14.163088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014360304000000001,"stat_sum":{"num_bytes":19134308,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":234,"num_read":109,"num_read_kb":4185,"num_write":1275,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1494","reported_seq":3432,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.931509+0000","last_change":"2023-10-30T13:21:40.580946+0000","last_active":"2023-10-30T13:52:23.931509+0000","last_peered":"2023-10-30T13:52:23.931509+0000","last_clean":"2023-10-30T13:52:23.931509+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:52:23.931509+0000","last_undegraded":"2023-10-30T13:52:23.931509+0000","last_fullsized":"2023-10-30T13:52:23.931509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:29:47.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010277041000000001,"stat_sum":{"num_bytes":19693407,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":208,"num_read":105,"num_read_kb":2976,"num_write":1294,"num_write_kb":29513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1525","reported_seq":3452,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.917846+0000","last_change":"2023-10-30T13:21:40.579672+0000","last_active":"2023-10-30T13:52:23.917846+0000","last_peered":"2023-10-30T13:52:23.917846+0000","last_clean":"2023-10-30T13:52:23.917846+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:52:23.917846+0000","last_undegraded":"2023-10-30T13:52:23.917846+0000","last_fullsized":"2023-10-30T13:52:23.917846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1525,"log_dups_size":0,"ondisk_log_size":1525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:22:11.088793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01048763,"stat_sum":{"num_bytes":11727836,"num_objects":636,"num_object_clones":292,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":231,"num_read":95,"num_read_kb":27,"num_write":1290,"num_write_kb":13174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1518","reported_seq":3443,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.939348+0000","last_change":"2023-10-30T13:21:40.537641+0000","last_active":"2023-10-30T13:52:23.939348+0000","last_peered":"2023-10-30T13:52:23.939348+0000","last_clean":"2023-10-30T13:52:23.939348+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:52:23.939348+0000","last_undegraded":"2023-10-30T13:52:23.939348+0000","last_fullsized":"2023-10-30T13:52:23.939348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:55.899096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085295030000000008,"stat_sum":{"num_bytes":15445695,"num_objects":634,"num_object_clones":293,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":231,"num_read":101,"num_read_kb":102,"num_write":1282,"num_write_kb":21892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1473","reported_seq":3415,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.907291+0000","last_change":"2023-10-30T13:21:40.573274+0000","last_active":"2023-10-30T13:52:23.907291+0000","last_peered":"2023-10-30T13:52:23.907291+0000","last_clean":"2023-10-30T13:52:23.907291+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:52:23.907291+0000","last_undegraded":"2023-10-30T13:52:23.907291+0000","last_fullsized":"2023-10-30T13:52:23.907291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:23:55.087821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004933977,"stat_sum":{"num_bytes":6328355,"num_objects":596,"num_object_clones":272,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":210,"num_read":97,"num_read_kb":4194,"num_write":1284,"num_write_kb":15784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1551","reported_seq":3498,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.231249+0000","last_change":"2023-10-30T13:21:40.528789+0000","last_active":"2023-10-30T13:52:25.231249+0000","last_peered":"2023-10-30T13:52:25.231249+0000","last_clean":"2023-10-30T13:52:25.231249+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:52:25.231249+0000","last_undegraded":"2023-10-30T13:52:25.231249+0000","last_fullsized":"2023-10-30T13:52:25.231249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:55.527282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029909680000000001,"stat_sum":{"num_bytes":31670070,"num_objects":610,"num_object_clones":280,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":227,"num_read":110,"num_read_kb":16437,"num_write":1314,"num_write_kb":33491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1555","reported_seq":3494,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.906435+0000","last_change":"2023-10-30T13:21:40.571731+0000","last_active":"2023-10-30T13:52:23.906435+0000","last_peered":"2023-10-30T13:52:23.906435+0000","last_clean":"2023-10-30T13:52:23.906435+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:52:23.906435+0000","last_undegraded":"2023-10-30T13:52:23.906435+0000","last_fullsized":"2023-10-30T13:52:23.906435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:47.649131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058939090000000001,"stat_sum":{"num_bytes":14342306,"num_objects":613,"num_object_clones":284,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":231,"num_read":106,"num_read_kb":2953,"num_write":1333,"num_write_kb":15925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1748","reported_seq":3606,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923510+0000","last_change":"2023-10-30T13:21:40.536269+0000","last_active":"2023-10-30T13:52:23.923510+0000","last_peered":"2023-10-30T13:52:23.923510+0000","last_clean":"2023-10-30T13:52:23.923510+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:52:23.923510+0000","last_undegraded":"2023-10-30T13:52:23.923510+0000","last_fullsized":"2023-10-30T13:52:23.923510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1748,"log_dups_size":0,"ondisk_log_size":1748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:06.829080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093243890000000006,"stat_sum":{"num_bytes":10965780,"num_objects":703,"num_object_clones":374,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":231,"num_read":115,"num_read_kb":1694,"num_write":1434,"num_write_kb":17364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1579","reported_seq":3505,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.946432+0000","last_change":"2023-10-30T13:21:40.539642+0000","last_active":"2023-10-30T13:52:23.946432+0000","last_peered":"2023-10-30T13:52:23.946432+0000","last_clean":"2023-10-30T13:52:23.946432+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:52:23.946432+0000","last_undegraded":"2023-10-30T13:52:23.946432+0000","last_fullsized":"2023-10-30T13:52:23.946432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1579,"log_dups_size":0,"ondisk_log_size":1579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:12.694457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011284880000000001,"stat_sum":{"num_bytes":18442682,"num_objects":632,"num_object_clones":291,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":228,"num_read":100,"num_read_kb":8212,"num_write":1340,"num_write_kb":24815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1459","reported_seq":3397,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932387+0000","last_change":"2023-10-30T13:21:40.567529+0000","last_active":"2023-10-30T13:52:23.932387+0000","last_peered":"2023-10-30T13:52:23.932387+0000","last_clean":"2023-10-30T13:52:23.932387+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:52:23.932387+0000","last_undegraded":"2023-10-30T13:52:23.932387+0000","last_fullsized":"2023-10-30T13:52:23.932387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:45.983709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002295727,"stat_sum":{"num_bytes":22547199,"num_objects":575,"num_object_clones":267,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":207,"num_read":89,"num_read_kb":4127,"num_write":1240,"num_write_kb":24211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1620","reported_seq":3538,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.922596+0000","last_change":"2023-10-30T13:21:40.572640+0000","last_active":"2023-10-30T13:52:23.922596+0000","last_peered":"2023-10-30T13:52:23.922596+0000","last_clean":"2023-10-30T13:52:23.922596+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:52:23.922596+0000","last_undegraded":"2023-10-30T13:52:23.922596+0000","last_fullsized":"2023-10-30T13:52:23.922596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:31.079099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061689539999999999,"stat_sum":{"num_bytes":11703479,"num_objects":649,"num_object_clones":303,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":237,"num_read":106,"num_read_kb":7307,"num_write":1383,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1553","reported_seq":3491,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.915574+0000","last_change":"2023-10-30T13:21:40.580324+0000","last_active":"2023-10-30T13:52:23.915574+0000","last_peered":"2023-10-30T13:52:23.915574+0000","last_clean":"2023-10-30T13:52:23.915574+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:52:23.915574+0000","last_undegraded":"2023-10-30T13:52:23.915574+0000","last_fullsized":"2023-10-30T13:52:23.915574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1553,"log_dups_size":0,"ondisk_log_size":1553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:52.961020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094394720000000008,"stat_sum":{"num_bytes":31771338,"num_objects":621,"num_object_clones":283,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":229,"num_read":103,"num_read_kb":4126,"num_write":1338,"num_write_kb":42793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1482","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.933727+0000","last_change":"2023-10-30T13:21:40.528854+0000","last_active":"2023-10-30T13:52:23.933727+0000","last_peered":"2023-10-30T13:52:23.933727+0000","last_clean":"2023-10-30T13:52:23.933727+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:52:23.933727+0000","last_undegraded":"2023-10-30T13:52:23.933727+0000","last_fullsized":"2023-10-30T13:52:23.933727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1482,"log_dups_size":0,"ondisk_log_size":1482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:48:55.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027436349999999999,"stat_sum":{"num_bytes":6210074,"num_objects":595,"num_object_clones":273,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":228,"num_read":98,"num_read_kb":33,"num_write":1259,"num_write_kb":10653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1636","reported_seq":3545,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.244114+0000","last_change":"2023-10-30T13:21:40.535360+0000","last_active":"2023-10-30T13:52:25.244114+0000","last_peered":"2023-10-30T13:52:25.244114+0000","last_clean":"2023-10-30T13:52:25.244114+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:52:25.244114+0000","last_undegraded":"2023-10-30T13:52:25.244114+0000","last_fullsized":"2023-10-30T13:52:25.244114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1636,"log_dups_size":0,"ondisk_log_size":1636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:13.413880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082328149999999992,"stat_sum":{"num_bytes":28398973,"num_objects":670,"num_object_clones":311,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":249,"num_read":102,"num_read_kb":120,"num_write":1394,"num_write_kb":38562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1530","reported_seq":3467,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.908084+0000","last_change":"2023-10-30T13:21:40.571762+0000","last_active":"2023-10-30T13:52:23.908084+0000","last_peered":"2023-10-30T13:52:23.908084+0000","last_clean":"2023-10-30T13:52:23.908084+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:52:23.908084+0000","last_undegraded":"2023-10-30T13:52:23.908084+0000","last_fullsized":"2023-10-30T13:52:23.908084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:29:37.541276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037447470000000001,"stat_sum":{"num_bytes":19407388,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":225,"num_read":103,"num_read_kb":4138,"num_write":1298,"num_write_kb":24714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1570","reported_seq":3510,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923610+0000","last_change":"2023-10-30T13:21:40.536309+0000","last_active":"2023-10-30T13:52:23.923610+0000","last_peered":"2023-10-30T13:52:23.923610+0000","last_clean":"2023-10-30T13:52:23.923610+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:52:23.923610+0000","last_undegraded":"2023-10-30T13:52:23.923610+0000","last_fullsized":"2023-10-30T13:52:23.923610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:04:17.809358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091633449999999998,"stat_sum":{"num_bytes":10552296,"num_objects":616,"num_object_clones":294,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":228,"num_read":115,"num_read_kb":24,"num_write":1340,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1410","reported_seq":3369,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.678938+0000","last_change":"2023-10-30T13:21:40.571783+0000","last_active":"2023-10-30T13:52:25.678938+0000","last_peered":"2023-10-30T13:52:25.678938+0000","last_clean":"2023-10-30T13:52:25.678938+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:52:25.678938+0000","last_undegraded":"2023-10-30T13:52:25.678938+0000","last_fullsized":"2023-10-30T13:52:25.678938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1410,"log_dups_size":0,"ondisk_log_size":1410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:41.732560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059368210000000001,"stat_sum":{"num_bytes":15196336,"num_objects":556,"num_object_clones":256,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":202,"num_read":99,"num_read_kb":1380,"num_write":1201,"num_write_kb":16993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'1569","reported_seq":3517,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.166551+0000","last_change":"2023-10-30T13:21:40.581115+0000","last_active":"2023-10-30T13:52:24.166551+0000","last_peered":"2023-10-30T13:52:24.166551+0000","last_clean":"2023-10-30T13:52:24.166551+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:52:24.166551+0000","last_undegraded":"2023-10-30T13:52:24.166551+0000","last_fullsized":"2023-10-30T13:52:24.166551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:31.199977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099065669999999998,"stat_sum":{"num_bytes":9983325,"num_objects":619,"num_object_clones":286,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":221,"num_read":115,"num_read_kb":68,"num_write":1328,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1534","reported_seq":3477,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.910285+0000","last_change":"2023-10-30T13:21:40.572631+0000","last_active":"2023-10-30T13:52:23.910285+0000","last_peered":"2023-10-30T13:52:23.910285+0000","last_clean":"2023-10-30T13:52:23.910285+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:52:23.910285+0000","last_undegraded":"2023-10-30T13:52:23.910285+0000","last_fullsized":"2023-10-30T13:52:23.910285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:53.178210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059382619999999997,"stat_sum":{"num_bytes":23775876,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":236,"num_read":113,"num_read_kb":4145,"num_write":1299,"num_write_kb":26615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1462","reported_seq":3390,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.907620+0000","last_change":"2023-10-30T13:21:40.575712+0000","last_active":"2023-10-30T13:52:23.907620+0000","last_peered":"2023-10-30T13:52:23.907620+0000","last_clean":"2023-10-30T13:52:23.907620+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:52:23.907620+0000","last_undegraded":"2023-10-30T13:52:23.907620+0000","last_fullsized":"2023-10-30T13:52:23.907620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:09.905979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053787089999999997,"stat_sum":{"num_bytes":10833053,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":213,"num_read":88,"num_read_kb":3300,"num_write":1254,"num_write_kb":16238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1646","reported_seq":3545,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.233660+0000","last_change":"2023-10-30T13:21:40.529120+0000","last_active":"2023-10-30T13:52:25.233660+0000","last_peered":"2023-10-30T13:52:25.233660+0000","last_clean":"2023-10-30T13:52:25.233660+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:52:25.233660+0000","last_undegraded":"2023-10-30T13:52:25.233660+0000","last_fullsized":"2023-10-30T13:52:25.233660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1646,"log_dups_size":0,"ondisk_log_size":1646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:56.055314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028849399999999999,"stat_sum":{"num_bytes":11995550,"num_objects":688,"num_object_clones":321,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":244,"num_read":100,"num_read_kb":8267,"num_write":1414,"num_write_kb":14306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1445","reported_seq":3397,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932007+0000","last_change":"2023-10-30T13:21:40.537362+0000","last_active":"2023-10-30T13:52:23.932007+0000","last_peered":"2023-10-30T13:52:23.932007+0000","last_clean":"2023-10-30T13:52:23.932007+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:52:23.932007+0000","last_undegraded":"2023-10-30T13:52:23.932007+0000","last_fullsized":"2023-10-30T13:52:23.932007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:49:05.792581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096547909999999994,"stat_sum":{"num_bytes":15074071,"num_objects":558,"num_object_clones":259,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":221,"num_read":91,"num_read_kb":32,"num_write":1202,"num_write_kb":16754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1663","reported_seq":3603,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.926321+0000","last_change":"2023-10-30T13:21:40.567248+0000","last_active":"2023-10-30T13:52:23.926321+0000","last_peered":"2023-10-30T13:52:23.926321+0000","last_clean":"2023-10-30T13:52:23.926321+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:52:23.926321+0000","last_undegraded":"2023-10-30T13:52:23.926321+0000","last_fullsized":"2023-10-30T13:52:23.926321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:07.337543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018845820000000001,"stat_sum":{"num_bytes":10798567,"num_objects":649,"num_object_clones":300,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":241,"num_read":120,"num_read_kb":27,"num_write":1443,"num_write_kb":17235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1499","reported_seq":3469,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923689+0000","last_change":"2023-10-30T13:21:40.575268+0000","last_active":"2023-10-30T13:52:23.923689+0000","last_peered":"2023-10-30T13:52:23.923689+0000","last_clean":"2023-10-30T13:52:23.923689+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:52:23.923689+0000","last_undegraded":"2023-10-30T13:52:23.923689+0000","last_fullsized":"2023-10-30T13:52:23.923689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1499,"log_dups_size":0,"ondisk_log_size":1499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:50:11.469301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075239010000000004,"stat_sum":{"num_bytes":15423893,"num_objects":576,"num_object_clones":259,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":203,"num_read":109,"num_read_kb":9552,"num_write":1306,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1548","reported_seq":3492,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.920280+0000","last_change":"2023-10-30T13:21:40.537743+0000","last_active":"2023-10-30T13:52:23.920280+0000","last_peered":"2023-10-30T13:52:23.920280+0000","last_clean":"2023-10-30T13:52:23.920280+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:52:23.920280+0000","last_undegraded":"2023-10-30T13:52:23.920280+0000","last_fullsized":"2023-10-30T13:52:23.920280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:01:00.836625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007356301,"stat_sum":{"num_bytes":40089180,"num_objects":622,"num_object_clones":287,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":230,"num_read":115,"num_read_kb":33,"num_write":1312,"num_write_kb":42997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'604","reported_seq":832,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.233111+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:52:19.233111+0000","last_peered":"2023-10-30T13:52:19.233111+0000","last_clean":"2023-10-30T13:52:19.233111+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:52:19.233111+0000","last_undegraded":"2023-10-30T13:52:19.233111+0000","last_fullsized":"2023-10-30T13:52:19.233111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":252,"num_read_kb":249243,"num_write":662,"num_write_kb":74846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1552","reported_seq":3479,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.929541+0000","last_change":"2023-10-30T13:21:40.529443+0000","last_active":"2023-10-30T13:52:23.929541+0000","last_peered":"2023-10-30T13:52:23.929541+0000","last_clean":"2023-10-30T13:52:23.929541+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:52:23.929541+0000","last_undegraded":"2023-10-30T13:52:23.929541+0000","last_fullsized":"2023-10-30T13:52:23.929541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:39.924469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372290000000001,"stat_sum":{"num_bytes":18713903,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":229,"num_read":108,"num_read_kb":4667,"num_write":1329,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1126","reported_seq":1449,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:15.211697+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:52:15.211697+0000","last_peered":"2023-10-30T13:52:15.211697+0000","last_clean":"2023-10-30T13:52:15.211697+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:52:15.211697+0000","last_undegraded":"2023-10-30T13:52:15.211697+0000","last_fullsized":"2023-10-30T13:52:15.211697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1126,"log_dups_size":0,"ondisk_log_size":1126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25293488,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":404,"num_read_kb":309242,"num_write":1229,"num_write_kb":84643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'745","reported_seq":1123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.654334+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:52:25.654334+0000","last_peered":"2023-10-30T13:52:25.654334+0000","last_clean":"2023-10-30T13:52:25.654334+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:52:25.654334+0000","last_undegraded":"2023-10-30T13:52:25.654334+0000","last_fullsized":"2023-10-30T13:52:25.654334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397151,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":403,"num_read_kb":153383,"num_write":818,"num_write_kb":55113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1778","reported_seq":3695,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.668056+0000","last_change":"2023-10-30T13:21:40.575255+0000","last_active":"2023-10-30T13:52:25.668056+0000","last_peered":"2023-10-30T13:52:25.668056+0000","last_clean":"2023-10-30T13:52:25.668056+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:52:25.668056+0000","last_undegraded":"2023-10-30T13:52:25.668056+0000","last_fullsized":"2023-10-30T13:52:25.668056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1778,"log_dups_size":0,"ondisk_log_size":1778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:20.571707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064075410000000001,"stat_sum":{"num_bytes":11666119,"num_objects":729,"num_object_clones":341,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":261,"num_read":141,"num_read_kb":4175,"num_write":1518,"num_write_kb":22588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1399","reported_seq":1668,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.266715+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:52:24.266715+0000","last_peered":"2023-10-30T13:52:24.266715+0000","last_clean":"2023-10-30T13:52:24.266715+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:52:24.266715+0000","last_undegraded":"2023-10-30T13:52:24.266715+0000","last_fullsized":"2023-10-30T13:52:24.266715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1399,"log_dups_size":0,"ondisk_log_size":1399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":354,"num_read_kb":270680,"num_write":1485,"num_write_kb":75457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1501","reported_seq":3434,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.893719+0000","last_change":"2023-10-30T13:21:40.571518+0000","last_active":"2023-10-30T13:52:23.893719+0000","last_peered":"2023-10-30T13:52:23.893719+0000","last_clean":"2023-10-30T13:52:23.893719+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:52:23.893719+0000","last_undegraded":"2023-10-30T13:52:23.893719+0000","last_fullsized":"2023-10-30T13:52:23.893719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:19:07.205330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058964639999999997,"stat_sum":{"num_bytes":19507019,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":240,"num_read":105,"num_read_kb":5362,"num_write":1250,"num_write_kb":24752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'953","reported_seq":1189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.509429+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:52:24.509429+0000","last_peered":"2023-10-30T13:52:24.509429+0000","last_clean":"2023-10-30T13:52:24.509429+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:52:24.509429+0000","last_undegraded":"2023-10-30T13:52:24.509429+0000","last_fullsized":"2023-10-30T13:52:24.509429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":313,"num_read_kb":141404,"num_write":1072,"num_write_kb":63726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1479","reported_seq":3432,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.935516+0000","last_change":"2023-10-30T13:21:40.567796+0000","last_active":"2023-10-30T13:52:23.935516+0000","last_peered":"2023-10-30T13:52:23.935516+0000","last_clean":"2023-10-30T13:52:23.935516+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:52:23.935516+0000","last_undegraded":"2023-10-30T13:52:23.935516+0000","last_fullsized":"2023-10-30T13:52:23.935516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:08:51.743716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025817129999999998,"stat_sum":{"num_bytes":23539910,"num_objects":588,"num_object_clones":267,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":208,"num_read":106,"num_read_kb":8313,"num_write":1264,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"67'1122","reported_seq":1340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:17.672748+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:52:17.672748+0000","last_peered":"2023-10-30T13:52:17.672748+0000","last_clean":"2023-10-30T13:52:17.672748+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:52:17.672748+0000","last_undegraded":"2023-10-30T13:52:17.672748+0000","last_fullsized":"2023-10-30T13:52:17.672748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1122,"log_dups_size":0,"ondisk_log_size":1122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":240,"num_read_kb":201173,"num_write":1196,"num_write_kb":66729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1545","reported_seq":3456,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.938692+0000","last_change":"2023-10-30T13:21:40.528173+0000","last_active":"2023-10-30T13:52:23.938692+0000","last_peered":"2023-10-30T13:52:23.938692+0000","last_clean":"2023-10-30T13:52:23.938692+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:52:23.938692+0000","last_undegraded":"2023-10-30T13:52:23.938692+0000","last_fullsized":"2023-10-30T13:52:23.938692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1545,"log_dups_size":0,"ondisk_log_size":1545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:50:03.879621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022874710000000001,"stat_sum":{"num_bytes":16137402,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":233,"num_read":91,"num_read_kb":34,"num_write":1322,"num_write_kb":18013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1315","reported_seq":1556,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.212373+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:52:24.212373+0000","last_peered":"2023-10-30T13:52:24.212373+0000","last_clean":"2023-10-30T13:52:24.212373+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:52:24.212373+0000","last_undegraded":"2023-10-30T13:52:24.212373+0000","last_fullsized":"2023-10-30T13:52:24.212373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20972192,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":307,"num_read_kb":165804,"num_write":1420,"num_write_kb":50966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1560","reported_seq":3492,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.937626+0000","last_change":"2023-10-30T13:21:40.538439+0000","last_active":"2023-10-30T13:52:23.937626+0000","last_peered":"2023-10-30T13:52:23.937626+0000","last_clean":"2023-10-30T13:52:23.937626+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:52:23.937626+0000","last_undegraded":"2023-10-30T13:52:23.937626+0000","last_fullsized":"2023-10-30T13:52:23.937626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:36:06.776976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087798259999999993,"stat_sum":{"num_bytes":32677996,"num_objects":638,"num_object_clones":293,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":232,"num_read":108,"num_read_kb":10951,"num_write":1340,"num_write_kb":38819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'935","reported_seq":1253,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.960735+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:52:23.960735+0000","last_peered":"2023-10-30T13:52:23.960735+0000","last_clean":"2023-10-30T13:52:23.960735+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:52:23.960735+0000","last_undegraded":"2023-10-30T13:52:23.960735+0000","last_fullsized":"2023-10-30T13:52:23.960735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":413,"num_read_kb":281405,"num_write":1056,"num_write_kb":80196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1490","reported_seq":3405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.927330+0000","last_change":"2023-10-30T13:21:40.583356+0000","last_active":"2023-10-30T13:52:23.927330+0000","last_peered":"2023-10-30T13:52:23.927330+0000","last_clean":"2023-10-30T13:52:23.927330+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:52:23.927330+0000","last_undegraded":"2023-10-30T13:52:23.927330+0000","last_fullsized":"2023-10-30T13:52:23.927330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:46.232350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014787140000000001,"stat_sum":{"num_bytes":19072740,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":238,"num_read":87,"num_read_kb":102,"num_write":1252,"num_write_kb":21862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'1084","reported_seq":1410,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.582275+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:52:16.582275+0000","last_peered":"2023-10-30T13:52:16.582275+0000","last_clean":"2023-10-30T13:52:16.582275+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:52:16.582275+0000","last_undegraded":"2023-10-30T13:52:16.582275+0000","last_fullsized":"2023-10-30T13:52:16.582275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748770,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":476,"num_read_kb":203988,"num_write":1202,"num_write_kb":67245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1496","reported_seq":3446,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.929377+0000","last_change":"2023-10-30T13:21:40.528316+0000","last_active":"2023-10-30T13:52:23.929377+0000","last_peered":"2023-10-30T13:52:23.929377+0000","last_clean":"2023-10-30T13:52:23.929377+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:52:23.929377+0000","last_undegraded":"2023-10-30T13:52:23.929377+0000","last_fullsized":"2023-10-30T13:52:23.929377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:08:42.998206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024076599999999998,"stat_sum":{"num_bytes":14342228,"num_objects":584,"num_object_clones":268,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":211,"num_read":101,"num_read_kb":4159,"num_write":1284,"num_write_kb":24386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'574","reported_seq":755,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.233164+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:52:19.233164+0000","last_peered":"2023-10-30T13:52:19.233164+0000","last_clean":"2023-10-30T13:52:19.233164+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:52:19.233164+0000","last_undegraded":"2023-10-30T13:52:19.233164+0000","last_fullsized":"2023-10-30T13:52:19.233164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":574,"log_dups_size":0,"ondisk_log_size":574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":203,"num_read_kb":83137,"num_write":662,"num_write_kb":45991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1588","reported_seq":3563,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.559855+0000","last_change":"2023-10-30T13:21:40.579925+0000","last_active":"2023-10-30T13:52:18.559855+0000","last_peered":"2023-10-30T13:52:18.559855+0000","last_clean":"2023-10-30T13:52:18.559855+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:52:18.559855+0000","last_undegraded":"2023-10-30T13:52:18.559855+0000","last_fullsized":"2023-10-30T13:52:18.559855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1588,"log_dups_size":0,"ondisk_log_size":1588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:39.495586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011943199999999999,"stat_sum":{"num_bytes":18918293,"num_objects":634,"num_object_clones":295,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":241,"num_read":154,"num_read_kb":114,"num_write":1356,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'499","reported_seq":731,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.240235+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:52:19.240235+0000","last_peered":"2023-10-30T13:52:19.240235+0000","last_clean":"2023-10-30T13:52:19.240235+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:52:19.240235+0000","last_undegraded":"2023-10-30T13:52:19.240235+0000","last_fullsized":"2023-10-30T13:52:19.240235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":289,"num_read_kb":121284,"num_write":554,"num_write_kb":58687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1629","reported_seq":3510,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.242217+0000","last_change":"2023-10-30T13:21:40.576918+0000","last_active":"2023-10-30T13:52:25.242217+0000","last_peered":"2023-10-30T13:52:25.242217+0000","last_clean":"2023-10-30T13:52:25.242217+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:52:25.242217+0000","last_undegraded":"2023-10-30T13:52:25.242217+0000","last_fullsized":"2023-10-30T13:52:25.242217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1629,"log_dups_size":0,"ondisk_log_size":1629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:33.447713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086123929999999994,"stat_sum":{"num_bytes":19792070,"num_objects":655,"num_object_clones":349,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":204,"num_read":113,"num_read_kb":142,"num_write":1344,"num_write_kb":24080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'960","reported_seq":1239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:15.167943+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:52:15.167943+0000","last_peered":"2023-10-30T13:52:15.167943+0000","last_clean":"2023-10-30T13:52:15.167943+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:52:15.167943+0000","last_undegraded":"2023-10-30T13:52:15.167943+0000","last_fullsized":"2023-10-30T13:52:15.167943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36678727,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":465,"num_read_kb":133831,"num_write":1107,"num_write_kb":54499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1554","reported_seq":3475,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.928034+0000","last_change":"2023-10-30T13:21:40.583214+0000","last_active":"2023-10-30T13:52:23.928034+0000","last_peered":"2023-10-30T13:52:23.928034+0000","last_clean":"2023-10-30T13:52:23.928034+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:52:23.928034+0000","last_undegraded":"2023-10-30T13:52:23.928034+0000","last_fullsized":"2023-10-30T13:52:23.928034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1554,"log_dups_size":0,"ondisk_log_size":1554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:22.371097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015028952999999999,"stat_sum":{"num_bytes":21253392,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":236,"num_read":95,"num_read_kb":12339,"num_write":1304,"num_write_kb":30718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'638","reported_seq":863,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.241252+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:52:19.241252+0000","last_peered":"2023-10-30T13:52:19.241252+0000","last_clean":"2023-10-30T13:52:19.241252+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:52:19.241252+0000","last_undegraded":"2023-10-30T13:52:19.241252+0000","last_fullsized":"2023-10-30T13:52:19.241252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165846,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":249,"num_read_kb":148701,"num_write":777,"num_write_kb":54421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1551","reported_seq":3481,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932444+0000","last_change":"2023-10-30T13:21:40.579365+0000","last_active":"2023-10-30T13:52:23.932444+0000","last_peered":"2023-10-30T13:52:23.932444+0000","last_clean":"2023-10-30T13:52:23.932444+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:52:23.932444+0000","last_undegraded":"2023-10-30T13:52:23.932444+0000","last_fullsized":"2023-10-30T13:52:23.932444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:51:50.149626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011682140000000001,"stat_sum":{"num_bytes":26941199,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":237,"num_read":103,"num_read_kb":4114,"num_write":1331,"num_write_kb":32609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'1051","reported_seq":1327,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.149100+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:52:16.149100+0000","last_peered":"2023-10-30T13:52:16.149100+0000","last_clean":"2023-10-30T13:52:16.149100+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:52:16.149100+0000","last_undegraded":"2023-10-30T13:52:16.149100+0000","last_fullsized":"2023-10-30T13:52:16.149100+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1051,"log_dups_size":0,"ondisk_log_size":1051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":381,"num_read_kb":250673,"num_write":1155,"num_write_kb":87740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1528","reported_seq":3470,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.951203+0000","last_change":"2023-10-30T13:21:40.585506+0000","last_active":"2023-10-30T13:52:23.951203+0000","last_peered":"2023-10-30T13:52:23.951203+0000","last_clean":"2023-10-30T13:52:23.951203+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:52:23.951203+0000","last_undegraded":"2023-10-30T13:52:23.951203+0000","last_fullsized":"2023-10-30T13:52:23.951203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:06.635146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014794043,"stat_sum":{"num_bytes":18888621,"num_objects":614,"num_object_clones":283,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":224,"num_read":107,"num_read_kb":4188,"num_write":1306,"num_write_kb":24240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'1227","reported_seq":1478,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.806783+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:52:23.806783+0000","last_peered":"2023-10-30T13:52:23.806783+0000","last_clean":"2023-10-30T13:52:23.806783+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:52:23.806783+0000","last_undegraded":"2023-10-30T13:52:23.806783+0000","last_fullsized":"2023-10-30T13:52:23.806783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1227,"log_dups_size":0,"ondisk_log_size":1227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364039,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":288,"num_read_kb":200539,"num_write":1325,"num_write_kb":71054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1570","reported_seq":3523,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923756+0000","last_change":"2023-10-30T13:21:40.538748+0000","last_active":"2023-10-30T13:52:23.923756+0000","last_peered":"2023-10-30T13:52:23.923756+0000","last_clean":"2023-10-30T13:52:23.923756+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:52:23.923756+0000","last_undegraded":"2023-10-30T13:52:23.923756+0000","last_fullsized":"2023-10-30T13:52:23.923756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:15.511609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010788236,"stat_sum":{"num_bytes":10484856,"num_objects":600,"num_object_clones":281,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":220,"num_read":116,"num_read_kb":28,"num_write":1335,"num_write_kb":13295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'776","reported_seq":1023,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.577979+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:52:16.577979+0000","last_peered":"2023-10-30T13:52:16.577979+0000","last_clean":"2023-10-30T13:52:16.577979+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:52:16.577979+0000","last_undegraded":"2023-10-30T13:52:16.577979+0000","last_fullsized":"2023-10-30T13:52:16.577979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":322,"num_read_kb":174067,"num_write":894,"num_write_kb":62874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1536","reported_seq":3467,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.231736+0000","last_change":"2023-10-30T13:21:40.583977+0000","last_active":"2023-10-30T13:52:25.231736+0000","last_peered":"2023-10-30T13:52:25.231736+0000","last_clean":"2023-10-30T13:52:25.231736+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:52:25.231736+0000","last_undegraded":"2023-10-30T13:52:25.231736+0000","last_fullsized":"2023-10-30T13:52:25.231736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1536,"log_dups_size":0,"ondisk_log_size":1536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:18.653967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015268978000000001,"stat_sum":{"num_bytes":11043591,"num_objects":616,"num_object_clones":291,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":221,"num_read":104,"num_read_kb":3707,"num_write":1322,"num_write_kb":16509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":232,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T13:50:37.450196+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:50:37.450196+0000","last_peered":"2023-10-30T13:50:37.450196+0000","last_clean":"2023-10-30T13:50:37.450196+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:50:37.450196+0000","last_undegraded":"2023-10-30T13:50:37.450196+0000","last_fullsized":"2023-10-30T13:50:37.450196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'617","reported_seq":825,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.795368+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:52:23.795368+0000","last_peered":"2023-10-30T13:52:23.795368+0000","last_clean":"2023-10-30T13:52:23.795368+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:52:23.795368+0000","last_undegraded":"2023-10-30T13:52:23.795368+0000","last_fullsized":"2023-10-30T13:52:23.795368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360178,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":232,"num_read_kb":148904,"num_write":693,"num_write_kb":62690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1532","reported_seq":3445,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.667425+0000","last_change":"2023-10-30T13:21:40.574205+0000","last_active":"2023-10-30T13:52:25.667425+0000","last_peered":"2023-10-30T13:52:25.667425+0000","last_clean":"2023-10-30T13:52:25.667425+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:52:25.667425+0000","last_undegraded":"2023-10-30T13:52:25.667425+0000","last_fullsized":"2023-10-30T13:52:25.667425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1532,"log_dups_size":0,"ondisk_log_size":1532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:45:44.062637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057314540000000004,"stat_sum":{"num_bytes":22780711,"num_objects":633,"num_object_clones":299,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":243,"num_read":92,"num_read_kb":12,"num_write":1291,"num_write_kb":28237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"67'705","reported_seq":936,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.241719+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:52:19.241719+0000","last_peered":"2023-10-30T13:52:19.241719+0000","last_clean":"2023-10-30T13:52:19.241719+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:52:19.241719+0000","last_undegraded":"2023-10-30T13:52:19.241719+0000","last_fullsized":"2023-10-30T13:52:19.241719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":256,"num_read_kb":247261,"num_write":738,"num_write_kb":70569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1453","reported_seq":3390,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.925643+0000","last_change":"2023-10-30T13:21:40.528051+0000","last_active":"2023-10-30T13:52:23.925643+0000","last_peered":"2023-10-30T13:52:23.925643+0000","last_clean":"2023-10-30T13:52:23.925643+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:52:23.925643+0000","last_undegraded":"2023-10-30T13:52:23.925643+0000","last_fullsized":"2023-10-30T13:52:23.925643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1453,"log_dups_size":0,"ondisk_log_size":1453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:08:49.848069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002406114,"stat_sum":{"num_bytes":14184993,"num_objects":568,"num_object_clones":263,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":210,"num_read":80,"num_read_kb":26,"num_write":1239,"num_write_kb":16314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'2040","reported_seq":3405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.224696+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:52:25.224696+0000","last_peered":"2023-10-30T13:52:25.224696+0000","last_clean":"2023-10-30T13:52:25.224696+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:52:25.224696+0000","last_undegraded":"2023-10-30T13:52:25.224696+0000","last_fullsized":"2023-10-30T13:52:25.224696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41670866,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":1534,"num_read_kb":188566,"num_write":2090,"num_write_kb":83865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1512","reported_seq":3453,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.919153+0000","last_change":"2023-10-30T13:21:40.568287+0000","last_active":"2023-10-30T13:52:23.919153+0000","last_peered":"2023-10-30T13:52:23.919153+0000","last_clean":"2023-10-30T13:52:23.919153+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:52:23.919153+0000","last_undegraded":"2023-10-30T13:52:23.919153+0000","last_fullsized":"2023-10-30T13:52:23.919153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:16.792299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002908281,"stat_sum":{"num_bytes":22417951,"num_objects":612,"num_object_clones":284,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":221,"num_read":105,"num_read_kb":44,"num_write":1296,"num_write_kb":23585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'947","reported_seq":1222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.180646+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:52:24.180646+0000","last_peered":"2023-10-30T13:52:24.180646+0000","last_clean":"2023-10-30T13:52:24.180646+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:52:24.180646+0000","last_undegraded":"2023-10-30T13:52:24.180646+0000","last_fullsized":"2023-10-30T13:52:24.180646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":355,"num_read_kb":252215,"num_write":1073,"num_write_kb":75653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1520","reported_seq":3474,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.663361+0000","last_change":"2023-10-30T13:21:40.536723+0000","last_active":"2023-10-30T13:52:25.663361+0000","last_peered":"2023-10-30T13:52:25.663361+0000","last_clean":"2023-10-30T13:52:25.663361+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:52:25.663361+0000","last_undegraded":"2023-10-30T13:52:25.663361+0000","last_fullsized":"2023-10-30T13:52:25.663361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:33:25.077240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007710705,"stat_sum":{"num_bytes":23751577,"num_objects":575,"num_object_clones":266,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":199,"num_read":100,"num_read_kb":489,"num_write":1325,"num_write_kb":27587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'943","reported_seq":1183,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.547312+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:52:18.547312+0000","last_peered":"2023-10-30T13:52:18.547312+0000","last_clean":"2023-10-30T13:52:18.547312+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:52:18.547312+0000","last_undegraded":"2023-10-30T13:52:18.547312+0000","last_fullsized":"2023-10-30T13:52:18.547312+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":294,"num_read_kb":227180,"num_write":1037,"num_write_kb":87579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1589","reported_seq":3499,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923728+0000","last_change":"2023-10-30T13:21:40.539081+0000","last_active":"2023-10-30T13:52:23.923728+0000","last_peered":"2023-10-30T13:52:23.923728+0000","last_clean":"2023-10-30T13:52:23.923728+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:52:23.923728+0000","last_undegraded":"2023-10-30T13:52:23.923728+0000","last_fullsized":"2023-10-30T13:52:23.923728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1589,"log_dups_size":0,"ondisk_log_size":1589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:36.291322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010719574000000001,"stat_sum":{"num_bytes":35538242,"num_objects":650,"num_object_clones":301,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":240,"num_read":91,"num_read_kb":31,"num_write":1345,"num_write_kb":41412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'785","reported_seq":1010,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.206722+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:52:24.206722+0000","last_peered":"2023-10-30T13:52:24.206722+0000","last_clean":"2023-10-30T13:52:24.206722+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:52:24.206722+0000","last_undegraded":"2023-10-30T13:52:24.206722+0000","last_fullsized":"2023-10-30T13:52:24.206722+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":258,"num_read_kb":177173,"num_write":842,"num_write_kb":62518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1541","reported_seq":3467,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.919237+0000","last_change":"2023-10-30T13:21:40.572129+0000","last_active":"2023-10-30T13:52:23.919237+0000","last_peered":"2023-10-30T13:52:23.919237+0000","last_clean":"2023-10-30T13:52:23.919237+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:52:23.919237+0000","last_undegraded":"2023-10-30T13:52:23.919237+0000","last_fullsized":"2023-10-30T13:52:23.919237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1541,"log_dups_size":0,"ondisk_log_size":1541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:23.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059276069999999997,"stat_sum":{"num_bytes":23389062,"num_objects":631,"num_object_clones":294,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":232,"num_read":100,"num_read_kb":25,"num_write":1313,"num_write_kb":24315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'914","reported_seq":1153,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.107011+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:52:19.107011+0000","last_peered":"2023-10-30T13:52:19.107011+0000","last_clean":"2023-10-30T13:52:19.107011+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:52:19.107011+0000","last_undegraded":"2023-10-30T13:52:19.107011+0000","last_fullsized":"2023-10-30T13:52:19.107011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":310,"num_read_kb":109156,"num_write":1491,"num_write_kb":61820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1404","reported_seq":3356,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.931293+0000","last_change":"2023-10-30T13:21:40.576235+0000","last_active":"2023-10-30T13:52:23.931293+0000","last_peered":"2023-10-30T13:52:23.931293+0000","last_clean":"2023-10-30T13:52:23.931293+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:52:23.931293+0000","last_undegraded":"2023-10-30T13:52:23.931293+0000","last_fullsized":"2023-10-30T13:52:23.931293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1404,"log_dups_size":0,"ondisk_log_size":1404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:30.114376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056940879999999999,"stat_sum":{"num_bytes":23042560,"num_objects":571,"num_object_clones":259,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":201,"num_read":92,"num_read_kb":75,"num_write":1201,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'1095","reported_seq":1338,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.531235+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:52:16.531235+0000","last_peered":"2023-10-30T13:52:16.531235+0000","last_clean":"2023-10-30T13:52:16.531235+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:52:16.531235+0000","last_undegraded":"2023-10-30T13:52:16.531235+0000","last_fullsized":"2023-10-30T13:52:16.531235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":342,"num_read_kb":120967,"num_write":1229,"num_write_kb":51134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1436","reported_seq":3397,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.918796+0000","last_change":"2023-10-30T13:21:40.529108+0000","last_active":"2023-10-30T13:52:23.918796+0000","last_peered":"2023-10-30T13:52:23.918796+0000","last_clean":"2023-10-30T13:52:23.918796+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:52:23.918796+0000","last_undegraded":"2023-10-30T13:52:23.918796+0000","last_fullsized":"2023-10-30T13:52:23.918796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1436,"log_dups_size":0,"ondisk_log_size":1436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:16:51.856467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029926589999999999,"stat_sum":{"num_bytes":30813966,"num_objects":572,"num_object_clones":263,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":205,"num_read":105,"num_read_kb":87,"num_write":1219,"num_write_kb":33525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'421","reported_seq":607,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.800787+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:52:23.800787+0000","last_peered":"2023-10-30T13:52:23.800787+0000","last_clean":"2023-10-30T13:52:23.800787+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:52:23.800787+0000","last_undegraded":"2023-10-30T13:52:23.800787+0000","last_fullsized":"2023-10-30T13:52:23.800787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":209,"num_read_kb":100741,"num_write":482,"num_write_kb":29419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1533","reported_seq":3487,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.926144+0000","last_change":"2023-10-30T13:21:40.533380+0000","last_active":"2023-10-30T13:52:23.926144+0000","last_peered":"2023-10-30T13:52:23.926144+0000","last_clean":"2023-10-30T13:52:23.926144+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:52:23.926144+0000","last_undegraded":"2023-10-30T13:52:23.926144+0000","last_fullsized":"2023-10-30T13:52:23.926144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1533,"log_dups_size":0,"ondisk_log_size":1533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:46.427030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057673200000000003,"stat_sum":{"num_bytes":13994934,"num_objects":595,"num_object_clones":279,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":215,"num_read":114,"num_read_kb":29,"num_write":1327,"num_write_kb":19694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'1204","reported_seq":1418,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.263849+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:52:24.263849+0000","last_peered":"2023-10-30T13:52:24.263849+0000","last_clean":"2023-10-30T13:52:24.263849+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:52:24.263849+0000","last_undegraded":"2023-10-30T13:52:24.263849+0000","last_fullsized":"2023-10-30T13:52:24.263849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21903607,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":251,"num_read_kb":99282,"num_write":1273,"num_write_kb":39377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1544","reported_seq":3497,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.892716+0000","last_change":"2023-10-30T13:21:40.575380+0000","last_active":"2023-10-30T13:52:23.892716+0000","last_peered":"2023-10-30T13:52:23.892716+0000","last_clean":"2023-10-30T13:52:23.892716+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:52:23.892716+0000","last_undegraded":"2023-10-30T13:52:23.892716+0000","last_fullsized":"2023-10-30T13:52:23.892716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1544,"log_dups_size":0,"ondisk_log_size":1544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:12.098739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063570620000000001,"stat_sum":{"num_bytes":16293763,"num_objects":612,"num_object_clones":286,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":233,"num_read":120,"num_read_kb":127,"num_write":1302,"num_write_kb":21658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'900","reported_seq":1126,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.166933+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:52:24.166933+0000","last_peered":"2023-10-30T13:52:24.166933+0000","last_clean":"2023-10-30T13:52:24.166933+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:52:24.166933+0000","last_undegraded":"2023-10-30T13:52:24.166933+0000","last_fullsized":"2023-10-30T13:52:24.166933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":272,"num_read_kb":200216,"num_write":1131,"num_write_kb":59038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1424","reported_seq":3373,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.237106+0000","last_change":"2023-10-30T13:21:40.575515+0000","last_active":"2023-10-30T13:52:25.237106+0000","last_peered":"2023-10-30T13:52:25.237106+0000","last_clean":"2023-10-30T13:52:25.237106+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:52:25.237106+0000","last_undegraded":"2023-10-30T13:52:25.237106+0000","last_fullsized":"2023-10-30T13:52:25.237106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1424,"log_dups_size":0,"ondisk_log_size":1424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:20.385719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060201619999999999,"stat_sum":{"num_bytes":10033117,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":215,"num_read":92,"num_read_kb":24,"num_write":1202,"num_write_kb":13175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'967","reported_seq":1222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:21.614332+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:52:21.614332+0000","last_peered":"2023-10-30T13:52:21.614332+0000","last_clean":"2023-10-30T13:52:21.614332+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:52:21.614332+0000","last_undegraded":"2023-10-30T13:52:21.614332+0000","last_fullsized":"2023-10-30T13:52:21.614332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":318,"num_read_kb":202944,"num_write":1058,"num_write_kb":67603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1452","reported_seq":3400,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.930308+0000","last_change":"2023-10-30T13:21:40.575093+0000","last_active":"2023-10-30T13:52:23.930308+0000","last_peered":"2023-10-30T13:52:23.930308+0000","last_clean":"2023-10-30T13:52:23.930308+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:52:23.930308+0000","last_undegraded":"2023-10-30T13:52:23.930308+0000","last_fullsized":"2023-10-30T13:52:23.930308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1452,"log_dups_size":0,"ondisk_log_size":1452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:55:44.047461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079253199999999996,"stat_sum":{"num_bytes":6591050,"num_objects":558,"num_object_clones":254,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":201,"num_read":83,"num_read_kb":63,"num_write":1251,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1486","reported_seq":3430,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.929689+0000","last_change":"2023-10-30T13:21:40.538321+0000","last_active":"2023-10-30T13:52:23.929689+0000","last_peered":"2023-10-30T13:52:23.929689+0000","last_clean":"2023-10-30T13:52:23.929689+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:52:23.929689+0000","last_undegraded":"2023-10-30T13:52:23.929689+0000","last_fullsized":"2023-10-30T13:52:23.929689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:04.667732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010603414,"stat_sum":{"num_bytes":9104816,"num_objects":585,"num_object_clones":276,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":204,"num_read":104,"num_read_kb":44,"num_write":1283,"num_write_kb":14447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'1000","reported_seq":1254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.409685+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:52:18.409685+0000","last_peered":"2023-10-30T13:52:18.409685+0000","last_clean":"2023-10-30T13:52:18.409685+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:52:18.409685+0000","last_undegraded":"2023-10-30T13:52:18.409685+0000","last_fullsized":"2023-10-30T13:52:18.409685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":337,"num_read_kb":187117,"num_write":1194,"num_write_kb":47550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1610","reported_seq":3524,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.937747+0000","last_change":"2023-10-30T13:21:40.537202+0000","last_active":"2023-10-30T13:52:23.937747+0000","last_peered":"2023-10-30T13:52:23.937747+0000","last_clean":"2023-10-30T13:52:23.937747+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:52:23.937747+0000","last_undegraded":"2023-10-30T13:52:23.937747+0000","last_fullsized":"2023-10-30T13:52:23.937747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1610,"log_dups_size":0,"ondisk_log_size":1610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:39:17.448171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097894100000000001,"stat_sum":{"num_bytes":19085997,"num_objects":650,"num_object_clones":307,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":230,"num_read":105,"num_read_kb":202,"num_write":1384,"num_write_kb":24607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1264","reported_seq":1566,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.267745+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:52:24.267745+0000","last_peered":"2023-10-30T13:52:24.267745+0000","last_clean":"2023-10-30T13:52:24.267745+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:52:24.267745+0000","last_undegraded":"2023-10-30T13:52:24.267745+0000","last_fullsized":"2023-10-30T13:52:24.267745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1264,"log_dups_size":0,"ondisk_log_size":1264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34811142,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":418,"num_read_kb":304041,"num_write":1365,"num_write_kb":89722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1502","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.656168+0000","last_change":"2023-10-30T13:21:40.528576+0000","last_active":"2023-10-30T13:52:24.656168+0000","last_peered":"2023-10-30T13:52:24.656168+0000","last_clean":"2023-10-30T13:52:24.656168+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:52:24.656168+0000","last_undegraded":"2023-10-30T13:52:24.656168+0000","last_fullsized":"2023-10-30T13:52:24.656168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1502,"log_dups_size":0,"ondisk_log_size":1502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:58.014283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026802879999999999,"stat_sum":{"num_bytes":22843366,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":237,"num_read":96,"num_read_kb":23,"num_write":1281,"num_write_kb":25082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'773","reported_seq":1005,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.811859+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:52:23.811859+0000","last_peered":"2023-10-30T13:52:23.811859+0000","last_clean":"2023-10-30T13:52:23.811859+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:52:23.811859+0000","last_undegraded":"2023-10-30T13:52:23.811859+0000","last_fullsized":"2023-10-30T13:52:23.811859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":280,"num_read_kb":212075,"num_write":833,"num_write_kb":74728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1454","reported_seq":3408,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.933645+0000","last_change":"2023-10-30T13:21:40.576062+0000","last_active":"2023-10-30T13:52:23.933645+0000","last_peered":"2023-10-30T13:52:23.933645+0000","last_clean":"2023-10-30T13:52:23.933645+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:52:23.933645+0000","last_undegraded":"2023-10-30T13:52:23.933645+0000","last_fullsized":"2023-10-30T13:52:23.933645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1454,"log_dups_size":0,"ondisk_log_size":1454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:28.354972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055040289999999997,"stat_sum":{"num_bytes":14435337,"num_objects":589,"num_object_clones":269,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":212,"num_read":103,"num_read_kb":22,"num_write":1235,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'708","reported_seq":971,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.234153+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:52:19.234153+0000","last_peered":"2023-10-30T13:52:19.234153+0000","last_clean":"2023-10-30T13:52:19.234153+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:52:19.234153+0000","last_undegraded":"2023-10-30T13:52:19.234153+0000","last_fullsized":"2023-10-30T13:52:19.234153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749318,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":374,"num_read_kb":96620,"num_write":839,"num_write_kb":51020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165182927,"num_objects":39302,"num_object_clones":18317,"num_object_copies":78604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39302,"num_whiteouts":14355,"num_read":6653,"num_read_kb":168434,"num_write":83420,"num_write_kb":1459733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424070144,"data_stored":2332381086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97918,"ondisk_log_size":97918,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903181988,"num_objects":1685,"num_object_clones":0,"num_object_copies":3370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":11399,"num_read_kb":5963012,"num_write":33982,"num_write_kb":2068472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1396,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806508032,"data_stored":1806363976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30291,"ondisk_log_size":30291,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575358,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724736,"kb_used_data":571384,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93647104,"statfs":{"total":96636764160,"available":95894634496,"internally_reserved":0,"allocated":585097216,"data_stored":571472003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":157022667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.95099999999999996},"min":{"1min":0.47999999999999998,"5min":0.41199999999999998,"15min":0.40899999999999997},"max":{"1min":1.288,"5min":1.321,"15min":15.381},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81899999999999995,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.315},"max":{"1min":1.212,"5min":1.8400000000000001,"15min":2.9510000000000001},"last":0.77300000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.83799999999999997,"15min":0.92800000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3240000000000001,"5min":1.79,"15min":15.271000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89100000000000001,"15min":0.96199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.42999999999999999},"max":{"1min":1.2769999999999999,"5min":1.6000000000000001,"15min":15.228999999999999},"last":0.84299999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92100000000000004,"15min":0.91300000000000003},"min":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.36299999999999999},"max":{"1min":1.3100000000000001,"5min":1.345,"15min":2.2429999999999999},"last":1.056},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.89000000000000001,"15min":0.877},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.242,"5min":1.929,"15min":2.4540000000000002},"last":0.96899999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95599999999999996,"15min":1.0800000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.256,"5min":2.0249999999999999,"15min":15.324},"last":0.624},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.96999999999999997,"15min":0.98599999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2629999999999999,"5min":1.323,"15min":1.637},"last":1.012}]},{"osd":4,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.091,"15min":1.1479999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.40899999999999997},"max":{"1min":1.3080000000000001,"5min":2.7869999999999999,"15min":15.172000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.093,"5min":1.0600000000000001,"15min":1.089},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.466,"5min":3.032,"15min":4.0209999999999999},"last":1.1259999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1120000000000001,"15min":1.1060000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.454,"5min":1.4770000000000001,"15min":1.7050000000000001},"last":1.163},{"interface":"front","average":{"1min":1.089,"5min":1.0600000000000001,"15min":1.0169999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.54500000000000004},"max":{"1min":1.4159999999999999,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":1.105}]},{"osd":6,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.1819999999999999,"15min":1.2509999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.46500000000000002},"max":{"1min":1.7829999999999999,"5min":3.0350000000000001,"15min":15.427},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.228,"15min":1.331},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.42199999999999999},"max":{"1min":1.841,"5min":3.5169999999999999,"15min":32.976999999999997},"last":1.2869999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575357,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593340,"kb_used_data":449780,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93778500,"statfs":{"total":96636764160,"available":96029184000,"internally_reserved":0,"allocated":460574720,"data_stored":451977574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":146995655},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.55000000000000004,"5min":0.45000000000000001,"15min":0.40999999999999998},"max":{"1min":1.2190000000000001,"5min":1.3999999999999999,"15min":5.9880000000000004},"last":1.087},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.86899999999999999,"15min":0.89200000000000002},"min":{"1min":0.58999999999999997,"5min":0.48599999999999999,"15min":0.35999999999999999},"max":{"1min":1.0620000000000001,"5min":1.3160000000000001,"15min":6.0730000000000004},"last":0.90300000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.93600000000000005,"15min":0.86499999999999999},"min":{"1min":0.58199999999999996,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.585,"5min":6.9930000000000003,"15min":6.9930000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.86399999999999999,"15min":0.86099999999999999},"min":{"1min":0.69299999999999995,"5min":0.52500000000000002,"15min":0.47499999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.9810000000000001},"last":0.79200000000000004}]},{"osd":2,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":0.89400000000000002},"min":{"1min":0.59899999999999998,"5min":0.54400000000000004,"15min":0.47699999999999998},"max":{"1min":1.044,"5min":6.8550000000000004,"15min":6.8550000000000004},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.97999999999999998,"15min":0.96399999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.454,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.83399999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88600000000000001,"15min":0.89400000000000002},"min":{"1min":0.64800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.2130000000000001,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.96899999999999997,"15min":0.98699999999999999},"min":{"1min":0.69899999999999995,"5min":0.64400000000000002,"15min":0.56299999999999994},"max":{"1min":1.536,"5min":5.3179999999999996,"15min":5.3179999999999996},"last":0.72699999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.99099999999999999,"15min":1.046},"min":{"1min":0.59199999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.321,"5min":1.4970000000000001,"15min":15.372},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0629999999999999,"15min":1.05},"min":{"1min":0.79000000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"max":{"1min":1.391,"5min":5.2729999999999997,"15min":6.1580000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0329999999999999,"15min":1.042},"min":{"1min":0.749,"5min":0.60499999999999998,"15min":0.60199999999999998},"max":{"1min":1.2649999999999999,"5min":1.421,"15min":1.8080000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.95199999999999996,"5min":0.73899999999999999,"15min":0.67700000000000005},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.913},"last":0.93999999999999995}]},{"osd":7,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0860000000000001,"15min":1.0960000000000001},"min":{"1min":0.72099999999999997,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2290000000000001,"5min":1.577,"15min":2.75},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.212,"15min":1.1899999999999999},"min":{"1min":1.006,"5min":0.69599999999999995,"15min":0.42499999999999999},"max":{"1min":1.627,"5min":1.627,"15min":3.0590000000000002},"last":0.99299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575358,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672628,"kb_used_data":519276,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93699212,"statfs":{"total":96636764160,"available":95947993088,"internally_reserved":0,"allocated":531738624,"data_stored":518585685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":157022662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88900000000000001,"15min":0.87},"min":{"1min":0.51300000000000001,"5min":0.38400000000000001,"15min":0.32700000000000001},"max":{"1min":1.3300000000000001,"5min":1.4379999999999999,"15min":6.4729999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.88800000000000001,"15min":0.86899999999999999},"min":{"1min":0.56999999999999995,"5min":0.46800000000000003,"15min":0.39100000000000001},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":8.7560000000000002},"last":0.745}]},{"osd":1,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81499999999999995,"15min":0.88200000000000001},"min":{"1min":0.51400000000000001,"5min":0.439,"15min":0.42799999999999999},"max":{"1min":1.2,"5min":1.671,"15min":15.257999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.86099999999999999,"15min":0.94899999999999995},"min":{"1min":0.62,"5min":0.48699999999999999,"15min":0.47399999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":15.175000000000001},"last":0.82399999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.90100000000000002,"15min":0.91100000000000003},"min":{"1min":0.57499999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.228,"5min":1.458,"15min":6.5410000000000004},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.93700000000000006,"15min":0.98699999999999999},"min":{"1min":0.57599999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.52,"15min":6.7080000000000002},"last":1.042}]},{"osd":3,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.0880000000000001,"15min":1.0469999999999999},"min":{"1min":0.69399999999999995,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.3839999999999999,"5min":5.4249999999999998,"15min":6.7699999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.175,"15min":1.1659999999999999},"min":{"1min":0.88600000000000001,"5min":0.58699999999999997,"15min":0.40100000000000002},"max":{"1min":1.4790000000000001,"5min":5.5570000000000004,"15min":15.364000000000001},"last":1.1000000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:51:44 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.92500000000000004,"15min":0.97999999999999998},"min":{"1min":0.63600000000000001,"5min":0.376,"15min":0.36499999999999999},"max":{"1min":2.5830000000000002,"5min":4.3529999999999998,"15min":5.9720000000000004},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.27,"5min":1.246,"15min":1.1459999999999999},"min":{"1min":0.92900000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.5190000000000001,"5min":5.6609999999999996,"15min":14.314},"last":1.1970000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.0129999999999999,"15min":1.123},"min":{"1min":0.69299999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.4239999999999999,"5min":4.2290000000000001,"15min":15.06},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.097,"15min":1.1799999999999999},"min":{"1min":0.83699999999999997,"5min":0.65400000000000003,"15min":0.50900000000000001},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":7.6630000000000003},"last":1.0149999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.0609999999999999,"15min":1.1279999999999999},"min":{"1min":0.745,"5min":0.44,"15min":0.44},"max":{"1min":4.0599999999999996,"5min":5.0419999999999998,"15min":6.8259999999999996},"last":0.875},{"interface":"front","average":{"1min":1.429,"5min":1.2130000000000001,"15min":1.266},"min":{"1min":0.89700000000000002,"5min":0.53500000000000003,"15min":0.46200000000000002},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":19.321999999999999},"last":1.151}]}]},{"osd":4,"up_from":13,"seq":55834575359,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717128,"kb_used_data":569408,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93654712,"statfs":{"total":96636764160,"available":95902425088,"internally_reserved":0,"allocated":583073792,"data_stored":568869284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.51800000000000002,"5min":0.48599999999999999,"15min":0.28100000000000003},"max":{"1min":1.087,"5min":2.202,"15min":9.6690000000000005},"last":1.1839999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85199999999999998,"15min":0.84799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47399999999999998,"15min":0.377},"max":{"1min":1.2010000000000001,"5min":2.0760000000000001,"15min":2.786},"last":0.73299999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.873,"15min":0.88200000000000001},"min":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.41399999999999998},"max":{"1min":1.2170000000000001,"5min":1.8160000000000001,"15min":5.5119999999999996},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.89500000000000002,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.51500000000000001,"15min":0.44700000000000001},"max":{"1min":1.202,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":1.2110000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:51:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.89200000000000002,"15min":0.878},"min":{"1min":0.63200000000000001,"5min":0.53700000000000003,"15min":0.45100000000000001},"max":{"1min":1.212,"5min":2.1549999999999998,"15min":9.0099999999999998},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.95099999999999996,"15min":0.95199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0569999999999999,"5min":7.7359999999999998,"15min":7.7359999999999998},"last":0.97499999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.94499999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.52500000000000002},"max":{"1min":1.1899999999999999,"5min":1.6599999999999999,"15min":4.7080000000000002},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.026,"5min":1.0580000000000001,"15min":1.0209999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.48699999999999999},"max":{"1min":1.2649999999999999,"5min":1.7509999999999999,"15min":3.2749999999999999},"last":1.153}]},{"osd":5,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0249999999999999,"15min":1.0489999999999999},"min":{"1min":0.56299999999999994,"5min":0.52300000000000002,"15min":0.51500000000000001},"max":{"1min":1.3799999999999999,"5min":1.621,"15min":7.8529999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.214,"15min":1.161},"min":{"1min":0.86899999999999999,"5min":0.627,"15min":0.40999999999999998},"max":{"1min":1.538,"5min":7.694,"15min":8.2070000000000007},"last":1.266}]},{"osd":6,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1379999999999999,"15min":1.081},"min":{"1min":0.745,"5min":0.68999999999999995,"15min":0.52600000000000002},"max":{"1min":1.458,"5min":7.7880000000000003,"15min":7.7880000000000003},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.038,"5min":1.1659999999999999,"15min":1.155},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.2929999999999999,"5min":3.7320000000000002,"15min":5.6130000000000004},"last":1.081}]},{"osd":7,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.1950000000000001,"15min":1.1739999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.52000000000000002},"max":{"1min":1.6120000000000001,"5min":7.8380000000000001,"15min":10.715},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0720000000000001,"15min":1.157},"min":{"1min":0.627,"5min":0.627,"15min":0.47199999999999998},"max":{"1min":1.1579999999999999,"5min":2.2450000000000001,"15min":4.3230000000000004},"last":1.119}]}]},{"osd":3,"up_from":13,"seq":55834575357,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707492,"kb_used_data":550044,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93664348,"statfs":{"total":96636764160,"available":95912292352,"internally_reserved":0,"allocated":563245056,"data_stored":549099638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":161216968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.69799999999999995},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64900000000000002,"5min":0.76700000000000002,"15min":1.5429999999999999},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.51000000000000001,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":0.755,"5min":0.76200000000000001,"15min":0.94099999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.28599999999999998,"15min":0.246},"max":{"1min":0.78900000000000003,"5min":0.92000000000000004,"15min":1.018},"last":0.373}]},{"osd":2,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.65100000000000002,"5min":1.056,"15min":1.4690000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.86499999999999999,"15min":1.6799999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.79600000000000004,"15min":0.83599999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.49099999999999999},"max":{"1min":0.96499999999999997,"5min":1.1739999999999999,"15min":1.5900000000000001},"last":0.747},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.84299999999999997,"15min":0.85299999999999998},"min":{"1min":0.66100000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0129999999999999,"5min":1.1419999999999999,"15min":1.3999999999999999},"last":0.76100000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.60799999999999998,"5min":0.57399999999999995,"15min":0.51800000000000002},"max":{"1min":1.095,"5min":1.1879999999999999,"15min":1.5069999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.83599999999999997,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.58599999999999997,"15min":0.5},"max":{"1min":1.232,"5min":1.232,"15min":1.365},"last":0.621}]},{"osd":6,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.88700000000000001,"15min":0.82799999999999996},"min":{"1min":0.66900000000000004,"5min":0.52800000000000002,"15min":0.45400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.88500000000000001,"15min":0.82699999999999996},"min":{"1min":0.69499999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":2.3410000000000002},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.88800000000000001,"15min":0.88200000000000001},"min":{"1min":0.68300000000000005,"5min":0.57799999999999996,"15min":0.51700000000000002},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":2.9449999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.872,"15min":0.89300000000000002},"min":{"1min":0.64300000000000002,"5min":0.54400000000000004,"15min":0.47099999999999997},"max":{"1min":1.343,"5min":1.343,"15min":2.9729999999999999},"last":0.77300000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575360,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484724,"kb_used_data":377260,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93887116,"statfs":{"total":96636764160,"available":96140406784,"internally_reserved":0,"allocated":386314240,"data_stored":377764906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.47999999999999998,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.37},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.50800000000000001,"15min":0.48199999999999998},"min":{"1min":0.41799999999999998,"5min":0.28299999999999997,"15min":0.25700000000000001},"max":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.45900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.58399999999999996,"15min":0.54000000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.0060000000000002,"5min":7.0060000000000002,"15min":7.0060000000000002},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.36499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":1.637},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.622,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.58399999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.78000000000000003,"15min":0.80300000000000005},"min":{"1min":0.59699999999999998,"5min":0.442,"15min":0.442},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.80500000000000005,"15min":0.82699999999999996},"min":{"1min":0.61399999999999999,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":4.1479999999999997,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.88500000000000001,"15min":0.877},"min":{"1min":0.754,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5169999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.80900000000000005,"15min":0.82499999999999996},"min":{"1min":0.58599999999999997,"5min":0.45300000000000001,"15min":0.41999999999999998},"max":{"1min":1.171,"5min":2.762,"15min":2.762},"last":0.81200000000000006}]},{"osd":6,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77400000000000002,"15min":0.79000000000000004},"min":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.47699999999999998},"max":{"1min":0.98699999999999999,"5min":1.6419999999999999,"15min":3.1659999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.84399999999999997,"15min":0.81699999999999995},"min":{"1min":0.65500000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.8210000000000002,"5min":3.8210000000000002,"15min":3.8210000000000002},"last":0.68400000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":0.90800000000000003,"15min":0.85699999999999998},"min":{"1min":0.69599999999999995,"5min":0.55000000000000004,"15min":0.503},"max":{"1min":6.9059999999999997,"5min":6.9059999999999997,"15min":6.9059999999999997},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.0009999999999999,"15min":0.95499999999999996},"min":{"1min":0.81299999999999994,"5min":0.63100000000000001,"15min":0.55900000000000005},"max":{"1min":6.9509999999999996,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.88200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575357,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671940,"kb_used_data":532604,"kb_used_omap":1,"kb_used_meta":139326,"kb_avail":93699900,"statfs":{"total":96636764160,"available":95948697600,"internally_reserved":0,"allocated":545386496,"data_stored":536307441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142670278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30399999999999999,"5min":0.24299999999999999,"15min":0.20799999999999999},"max":{"1min":0.58799999999999997,"5min":0.94999999999999996,"15min":1.1359999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.28299999999999997,"15min":0.251},"max":{"1min":0.64900000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49099999999999999,"15min":0.501},"min":{"1min":0.318,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66800000000000004,"5min":1.002,"15min":2.8660000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.501,"15min":0.48599999999999999},"min":{"1min":0.36899999999999999,"5min":0.27700000000000002,"15min":0.249},"max":{"1min":0.69099999999999995,"5min":1.0589999999999999,"15min":2.6739999999999999},"last":0.625}]},{"osd":3,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.53000000000000003},"min":{"1min":0.373,"5min":0.35099999999999998,"15min":0.314},"max":{"1min":0.73899999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.318},"max":{"1min":0.755,"5min":1.306,"15min":2.71},"last":0.66800000000000004}]},{"osd":4,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82399999999999995,"15min":0.85599999999999998},"min":{"1min":0.622,"5min":0.52400000000000002,"15min":0.40200000000000002},"max":{"1min":1.0489999999999999,"5min":1.2210000000000001,"15min":6.7789999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.92200000000000004},"min":{"1min":0.76500000000000001,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":1.093,"5min":1.2629999999999999,"15min":6.8259999999999996},"last":0.88700000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":0.76200000000000001},"min":{"1min":0.65800000000000003,"5min":0.53800000000000003,"15min":0.47199999999999998},"max":{"1min":0.96499999999999997,"5min":2.133,"15min":2.5750000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.627,"5min":0.53800000000000003,"15min":0.51000000000000001},"max":{"1min":1.0229999999999999,"5min":1.3859999999999999,"15min":2.6179999999999999},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Oct 30 13:51:46 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91800000000000004,"15min":0.89500000000000002},"min":{"1min":0.80000000000000004,"5min":0.61199999999999999,"15min":0.501},"max":{"1min":1.1459999999999999,"5min":1.3999999999999999,"15min":6.0289999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.96599999999999997,"15min":0.95499999999999996},"min":{"1min":0.65500000000000003,"5min":0.625,"15min":0.60299999999999998},"max":{"1min":1.218,"5min":1.4399999999999999,"15min":6.0739999999999998},"last":0.95699999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79500000000000004,"15min":0.78300000000000003},"min":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.47899999999999998},"max":{"1min":1.0369999999999999,"5min":1.161,"15min":2.1579999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.879,"15min":0.88500000000000001},"min":{"1min":0.70899999999999996,"5min":0.55200000000000005,"15min":0.54800000000000004},"max":{"1min":1.1339999999999999,"5min":1.3440000000000001,"15min":6.0519999999999996},"last":0.998}]}]},{"osd":0,"up_from":13,"seq":55834575359,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712496,"kb_used_data":568936,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93659344,"statfs":{"total":96636764160,"available":95907168256,"internally_reserved":0,"allocated":582590464,"data_stored":569154632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.48399999999999999},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.214},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":16.184000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.50800000000000001,"15min":0.59199999999999997},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.28899999999999998},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":16.228000000000002},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.44900000000000001,"15min":0.64100000000000001},"min":{"1min":0.33300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":2.7999999999999998,"5min":2.7999999999999998,"15min":17.454000000000001},"last":0.496},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.65300000000000002},"min":{"1min":0.318,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":1.391,"5min":1.391,"15min":23.212},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.37,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":16.103999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.53100000000000003,"15min":0.73099999999999998},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.32000000000000001},"max":{"1min":2.8340000000000001,"5min":2.8340000000000001,"15min":23.16},"last":0.57799999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:51:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.83299999999999996,"15min":0.92700000000000005},"min":{"1min":0.48499999999999999,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":7.6900000000000004,"5min":7.6900000000000004,"15min":16.309000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.85799999999999998,"15min":0.97499999999999998},"min":{"1min":0.61199999999999999,"5min":0.53800000000000003,"15min":0.46000000000000002},"max":{"1min":2.5939999999999999,"5min":3.4009999999999998,"15min":16.343},"last":0.97199999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85299999999999998,"15min":1.006},"min":{"1min":0.63,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.02,"5min":2.02,"15min":23.266999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.86299999999999999,"15min":1.0329999999999999},"min":{"1min":0.755,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.7240000000000002,"5min":2.7240000000000002,"15min":16.268000000000001},"last":0.90700000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.80500000000000005,"15min":1.1120000000000001},"min":{"1min":0.61099999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":29.774999999999999},"last":0.754},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81499999999999995,"15min":1.0229999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.50700000000000001},"max":{"1min":2.75,"5min":2.75,"15min":17.5},"last":0.92700000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.81999999999999995,"15min":1.202},"min":{"1min":0.66700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.694,"5min":2.694,"15min":47.534999999999997},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.85999999999999999,"15min":1.155},"min":{"1min":0.64100000000000001,"5min":0.60399999999999998,"15min":0.54700000000000004},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":32.476999999999997},"last":0.83199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230703104,"data_stored":230687380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255741952,"data_stored":255719205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179036160,"data_stored":179013521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230858752,"data_stored":230840931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":197275648,"data_stored":197263919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":205541376,"data_stored":205522290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266067968,"data_stored":266051919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":241283072,"data_stored":241264811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350515200,"data_stored":337464605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288862208,"data_stored":280174938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":206495744,"data_stored":198338087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331603968,"data_stored":317845409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":385015808,"data_stored":371192067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":325414912,"data_stored":312650097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193724416,"data_stored":185512357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342437888,"data_stored":329203526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:52:30.709 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T13:52:30.710 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:52:30.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.874+0000 7f2e73b1d640 1 Processor -- start 2023-10-30T13:52:30.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.874+0000 7f2e73b1d640 1 -- start start 2023-10-30T13:52:30.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e73b1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 0x7f2e6c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e73b1d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2e6c0c6f60 0x7f2e6c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e73b1d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c0c0a50 con 0x7f2e6c0c8430 2023-10-30T13:52:30.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e73b1d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6c06f6e0 con 0x7f2e6c0c6f60 2023-10-30T13:52:30.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e73b1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c06f9c0 con 0x7f2e6c0c6b80 2023-10-30T13:52:30.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e710b8640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2e6c0c6f60 0x7f2e6c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e710b8640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2e6c0c6f60 0x7f2e6c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56676/0 (socket says 172.21.15.148:56676) 2023-10-30T13:52:30.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e710b8640 1 -- 172.21.15.148:0/3622909377 learned_addr learned my addr 172.21.15.148:0/3622909377 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:30.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e718b9640 1 --2- 172.21.15.148:0/3622909377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 0x7f2e6c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e710b8640 1 -- 172.21.15.148:0/3622909377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 msgr2=0x7f2e6c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:30.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e710b8640 1 --2- 172.21.15.148:0/3622909377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 0x7f2e6c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e710b8640 1 -- 172.21.15.148:0/3622909377 >> v1:172.21.15.150:6789/0 conn(0x7f2e6c0c8430 legacy=0x7f2e6c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.878+0000 7f2e710b8640 1 -- 172.21.15.148:0/3622909377 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c06fca0 con 0x7f2e6c0c6f60 2023-10-30T13:52:30.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e710b8640 1 --2- 172.21.15.148:0/3622909377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c6f60 0x7f2e6c0c0400 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c009870 tx=0x7f2e5c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618463aaef48450 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:30.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e708b7640 1 -- 172.21.15.148:0/3622909377 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c0ab650 con 0x7f2e6c0c6f60 2023-10-30T13:52:30.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e708b7640 1 -- 172.21.15.148:0/3622909377 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e5c0ab7b0 con 0x7f2e6c0c6f60 2023-10-30T13:52:30.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e708b7640 1 -- 172.21.15.148:0/3622909377 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c0abb40 con 0x7f2e6c0c6f60 2023-10-30T13:52:30.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 -- 172.21.15.148:0/3622909377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c6f60 msgr2=0x7f2e6c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 --2- 172.21.15.148:0/3622909377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c6f60 0x7f2e6c0c0400 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c009870 tx=0x7f2e5c01c540 comp rx=0 tx=0).stop 2023-10-30T13:52:30.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 -- 172.21.15.148:0/3622909377 shutdown_connections 2023-10-30T13:52:30.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 --2- 172.21.15.148:0/3622909377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c6f60 0x7f2e6c0c0400 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 --2- 172.21.15.148:0/3622909377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 0x7f2e6c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 -- 172.21.15.148:0/3622909377 >> 172.21.15.148:0/3622909377 conn(0x7f2e6c0c7910 msgr2=0x7f2e6c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:30.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 -- 172.21.15.148:0/3622909377 shutdown_connections 2023-10-30T13:52:30.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 -- 172.21.15.148:0/3622909377 wait complete. 2023-10-30T13:52:30.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 Processor -- start 2023-10-30T13:52:30.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.882+0000 7f2e73b1d640 1 -- start start 2023-10-30T13:52:30.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e73b1d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 0x7f2e6c1f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e73b1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e6c0c6f60 0x7f2e6c13dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e73b1d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c8430 0x7f2e6c14f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e73b1d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c1f3a80 con 0x7f2e6c0c8430 2023-10-30T13:52:30.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e73b1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c13b250 con 0x7f2e6c0c6f60 2023-10-30T13:52:30.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e73b1d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c13ac90 con 0x7f2e6c0c6b80 2023-10-30T13:52:30.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e720ba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c8430 0x7f2e6c14f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e720ba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c8430 0x7f2e6c14f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56688/0 (socket says 172.21.15.148:56688) 2023-10-30T13:52:30.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e720ba640 1 -- 172.21.15.148:0/100179100 learned_addr learned my addr 172.21.15.148:0/100179100 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:30.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e718b9640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 0x7f2e6c1f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e710b8640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e6c0c6f60 0x7f2e6c13dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e720ba640 1 -- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 msgr2=0x7f2e6c1f3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e720ba640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 0x7f2e6c1f3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e720ba640 1 -- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e6c0c6f60 msgr2=0x7f2e6c13dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:30.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e720ba640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e6c0c6f60 0x7f2e6c13dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:30.896 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e720ba640 1 -- 172.21.15.148:0/100179100 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c0094d0 con 0x7f2e6c0c8430 2023-10-30T13:52:30.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e710b8640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e6c0c6f60 0x7f2e6c13dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:30.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e718b9640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 0x7f2e6c1f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:30.897 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e720ba640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c8430 0x7f2e6c14f400 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400c730 tx=0x7f2e6400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:30.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e627fc640 1 -- 172.21.15.148:0/100179100 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64003b80 con 0x7f2e6c0c8430 2023-10-30T13:52:30.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e627fc640 1 -- 172.21.15.148:0/100179100 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e64003ce0 con 0x7f2e6c0c8430 2023-10-30T13:52:30.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e627fc640 1 -- 172.21.15.148:0/100179100 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400aa30 con 0x7f2e6c0c8430 2023-10-30T13:52:30.898 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c13af70 con 0x7f2e6c0c8430 2023-10-30T13:52:30.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6c06a800 con 0x7f2e6c0c8430 2023-10-30T13:52:30.899 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6c1f3bf0 con 0x7f2e6c0c8430 2023-10-30T13:52:30.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.886+0000 7f2e627fc640 1 -- 172.21.15.148:0/100179100 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e6400ab90 con 0x7f2e6c0c8430 2023-10-30T13:52:30.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.890+0000 7f2e627fc640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e500664d0 0x7f2e50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:30.900 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.890+0000 7f2e627fc640 1 -- 172.21.15.148:0/100179100 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f2e64084460 con 0x7f2e6c0c8430 2023-10-30T13:52:30.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.890+0000 7f2e718b9640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e500664d0 0x7f2e50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:30.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.890+0000 7f2e718b9640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e500664d0 0x7f2e50068990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009bb0 tx=0x7f2e68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:30.901 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:30.894+0000 7f2e627fc640 1 -- 172.21.15.148:0/100179100 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e640510a0 con 0x7f2e6c0c8430 2023-10-30T13:52:31.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.038+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e6c1f38e0 con 0x7f2e500664d0 2023-10-30T13:52:31.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.062+0000 7f2e627fc640 1 -- 172.21.15.148:0/100179100 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293908 (secure 0 0 0) 0x7f2e68011a70 con 0x7f2e500664d0 2023-10-30T13:52:31.067 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:52:31.069 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:52:31.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.062+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e500664d0 msgr2=0x7f2e50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:31.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.062+0000 7f2e73b1d640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e500664d0 0x7f2e50068990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009bb0 tx=0x7f2e68008040 comp rx=0 tx=0).stop 2023-10-30T13:52:31.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.062+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c8430 msgr2=0x7f2e6c14f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:31.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.062+0000 7f2e73b1d640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c8430 0x7f2e6c14f400 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400c730 tx=0x7f2e6400cbf0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.066+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 shutdown_connections 2023-10-30T13:52:31.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.066+0000 7f2e73b1d640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2e500664d0 0x7f2e50068990 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.066+0000 7f2e73b1d640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e6c0c8430 0x7f2e6c14f400 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.066+0000 7f2e73b1d640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e6c0c6f60 0x7f2e6c13dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.066+0000 7f2e73b1d640 1 --2- 172.21.15.148:0/100179100 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2e6c0c6b80 0x7f2e6c1f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.066+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 >> 172.21.15.148:0/100179100 conn(0x7f2e6c0c7910 msgr2=0x7f2e6c136570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.066+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 shutdown_connections 2023-10-30T13:52:31.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.066+0000 7f2e73b1d640 1 -- 172.21.15.148:0/100179100 wait complete. 2023-10-30T13:52:31.088 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1323,"stamp":"2023-10-30T13:52:29.202282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068955283,"num_objects":40989,"num_object_clones":18317,"num_object_copies":81978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40989,"num_whiteouts":14355,"num_read":18168,"num_read_kb":6131546,"num_write":117561,"num_write_kb":3530621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1396,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128326,"ondisk_log_size":128326,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5284484,"kb_used_data":4138692,"kb_used_omap":12,"kb_used_meta":1145715,"kb_avail":749690236,"statfs":{"total":773094113280,"available":767682801664,"internally_reserved":0,"allocated":4238020608,"data_stored":4143231163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1173212728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":100,"apply_latency_ms":100,"commit_latency_ns":100000000,"apply_latency_ns":100000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":56824291,"num_objects":745,"num_object_clones":253,"num_object_copies":1490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":745,"num_whiteouts":242,"num_read":298,"num_read_kb":66939,"num_write":972,"num_write_kb":56191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004500"},"pg_stats":[{"pgid":"3.3f","version":"67'1458","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.951630+0000","last_change":"2023-10-30T13:21:40.584822+0000","last_active":"2023-10-30T13:52:23.951630+0000","last_peered":"2023-10-30T13:52:23.951630+0000","last_clean":"2023-10-30T13:52:23.951630+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:52:23.951630+0000","last_undegraded":"2023-10-30T13:52:23.951630+0000","last_fullsized":"2023-10-30T13:52:23.951630+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:13.082992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014806453000000001,"stat_sum":{"num_bytes":11269044,"num_objects":569,"num_object_clones":260,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":205,"num_read":107,"num_read_kb":121,"num_write":1253,"num_write_kb":17103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1464","reported_seq":3387,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.234533+0000","last_change":"2023-10-30T13:21:40.572202+0000","last_active":"2023-10-30T13:52:25.234533+0000","last_peered":"2023-10-30T13:52:25.234533+0000","last_clean":"2023-10-30T13:52:25.234533+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:52:25.234533+0000","last_undegraded":"2023-10-30T13:52:25.234533+0000","last_fullsized":"2023-10-30T13:52:25.234533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:58.516621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059691099999999997,"stat_sum":{"num_bytes":30975930,"num_objects":597,"num_object_clones":279,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":225,"num_read":82,"num_read_kb":30,"num_write":1226,"num_write_kb":32475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1518","reported_seq":3465,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.166473+0000","last_change":"2023-10-30T13:21:40.528480+0000","last_active":"2023-10-30T13:52:24.166473+0000","last_peered":"2023-10-30T13:52:24.166473+0000","last_clean":"2023-10-30T13:52:24.166473+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:52:24.166473+0000","last_undegraded":"2023-10-30T13:52:24.166473+0000","last_fullsized":"2023-10-30T13:52:24.166473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:03.253931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025608710000000002,"stat_sum":{"num_bytes":18704388,"num_objects":617,"num_object_clones":279,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":225,"num_read":106,"num_read_kb":8300,"num_write":1300,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1566","reported_seq":3502,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.918237+0000","last_change":"2023-10-30T13:21:40.567224+0000","last_active":"2023-10-30T13:52:23.918237+0000","last_peered":"2023-10-30T13:52:23.918237+0000","last_clean":"2023-10-30T13:52:23.918237+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:52:23.918237+0000","last_undegraded":"2023-10-30T13:52:23.918237+0000","last_fullsized":"2023-10-30T13:52:23.918237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1566,"log_dups_size":0,"ondisk_log_size":1566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:34.794112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021564890000000001,"stat_sum":{"num_bytes":23329933,"num_objects":630,"num_object_clones":293,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":227,"num_read":112,"num_read_kb":4378,"num_write":1354,"num_write_kb":25006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1582","reported_seq":3502,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.930705+0000","last_change":"2023-10-30T13:21:40.579191+0000","last_active":"2023-10-30T13:52:23.930705+0000","last_peered":"2023-10-30T13:52:23.930705+0000","last_clean":"2023-10-30T13:52:23.930705+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:52:23.930705+0000","last_undegraded":"2023-10-30T13:52:23.930705+0000","last_fullsized":"2023-10-30T13:52:23.930705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1582,"log_dups_size":0,"ondisk_log_size":1582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:10.057619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011808077,"stat_sum":{"num_bytes":16184828,"num_objects":654,"num_object_clones":305,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":251,"num_read":109,"num_read_kb":152,"num_write":1339,"num_write_kb":20879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1451","reported_seq":3398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932299+0000","last_change":"2023-10-30T13:21:40.539757+0000","last_active":"2023-10-30T13:52:23.932299+0000","last_peered":"2023-10-30T13:52:23.932299+0000","last_clean":"2023-10-30T13:52:23.932299+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:52:23.932299+0000","last_undegraded":"2023-10-30T13:52:23.932299+0000","last_fullsized":"2023-10-30T13:52:23.932299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1451,"log_dups_size":0,"ondisk_log_size":1451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:59.387758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01102786,"stat_sum":{"num_bytes":18372801,"num_objects":593,"num_object_clones":277,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":222,"num_read":108,"num_read_kb":4137,"num_write":1217,"num_write_kb":19358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1501","reported_seq":3438,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.927654+0000","last_change":"2023-10-30T13:21:40.580121+0000","last_active":"2023-10-30T13:52:23.927654+0000","last_peered":"2023-10-30T13:52:23.927654+0000","last_clean":"2023-10-30T13:52:23.927654+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:52:23.927654+0000","last_undegraded":"2023-10-30T13:52:23.927654+0000","last_fullsized":"2023-10-30T13:52:23.927654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:42.759583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012047337,"stat_sum":{"num_bytes":9859762,"num_objects":598,"num_object_clones":275,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":211,"num_read":96,"num_read_kb":35,"num_write":1295,"num_write_kb":11866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1450","reported_seq":3406,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.168248+0000","last_change":"2023-10-30T13:21:40.575154+0000","last_active":"2023-10-30T13:52:24.168248+0000","last_peered":"2023-10-30T13:52:24.168248+0000","last_clean":"2023-10-30T13:52:24.168248+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:52:24.168248+0000","last_undegraded":"2023-10-30T13:52:24.168248+0000","last_fullsized":"2023-10-30T13:52:24.168248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1450,"log_dups_size":0,"ondisk_log_size":1450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:28.268146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063254060000000004,"stat_sum":{"num_bytes":13694423,"num_objects":577,"num_object_clones":268,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":211,"num_read":108,"num_read_kb":104,"num_write":1228,"num_write_kb":23250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1510","reported_seq":3452,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.243551+0000","last_change":"2023-10-30T13:21:40.584916+0000","last_active":"2023-10-30T13:52:25.243551+0000","last_peered":"2023-10-30T13:52:25.243551+0000","last_clean":"2023-10-30T13:52:25.243551+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:52:25.243551+0000","last_undegraded":"2023-10-30T13:52:25.243551+0000","last_fullsized":"2023-10-30T13:52:25.243551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1510,"log_dups_size":0,"ondisk_log_size":1510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:08:14.163088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014360304000000001,"stat_sum":{"num_bytes":19134308,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":234,"num_read":109,"num_read_kb":4185,"num_write":1275,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1494","reported_seq":3432,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.931509+0000","last_change":"2023-10-30T13:21:40.580946+0000","last_active":"2023-10-30T13:52:23.931509+0000","last_peered":"2023-10-30T13:52:23.931509+0000","last_clean":"2023-10-30T13:52:23.931509+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:52:23.931509+0000","last_undegraded":"2023-10-30T13:52:23.931509+0000","last_fullsized":"2023-10-30T13:52:23.931509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:29:47.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010277041000000001,"stat_sum":{"num_bytes":19693407,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":208,"num_read":105,"num_read_kb":2976,"num_write":1294,"num_write_kb":29513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1525","reported_seq":3452,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.917846+0000","last_change":"2023-10-30T13:21:40.579672+0000","last_active":"2023-10-30T13:52:23.917846+0000","last_peered":"2023-10-30T13:52:23.917846+0000","last_clean":"2023-10-30T13:52:23.917846+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:52:23.917846+0000","last_undegraded":"2023-10-30T13:52:23.917846+0000","last_fullsized":"2023-10-30T13:52:23.917846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1525,"log_dups_size":0,"ondisk_log_size":1525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:22:11.088793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01048763,"stat_sum":{"num_bytes":11727836,"num_objects":636,"num_object_clones":292,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":231,"num_read":95,"num_read_kb":27,"num_write":1290,"num_write_kb":13174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1518","reported_seq":3443,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.939348+0000","last_change":"2023-10-30T13:21:40.537641+0000","last_active":"2023-10-30T13:52:23.939348+0000","last_peered":"2023-10-30T13:52:23.939348+0000","last_clean":"2023-10-30T13:52:23.939348+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:52:23.939348+0000","last_undegraded":"2023-10-30T13:52:23.939348+0000","last_fullsized":"2023-10-30T13:52:23.939348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:55.899096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085295030000000008,"stat_sum":{"num_bytes":15445695,"num_objects":634,"num_object_clones":293,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":231,"num_read":101,"num_read_kb":102,"num_write":1282,"num_write_kb":21892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1473","reported_seq":3415,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.907291+0000","last_change":"2023-10-30T13:21:40.573274+0000","last_active":"2023-10-30T13:52:23.907291+0000","last_peered":"2023-10-30T13:52:23.907291+0000","last_clean":"2023-10-30T13:52:23.907291+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:52:23.907291+0000","last_undegraded":"2023-10-30T13:52:23.907291+0000","last_fullsized":"2023-10-30T13:52:23.907291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:23:55.087821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004933977,"stat_sum":{"num_bytes":6328355,"num_objects":596,"num_object_clones":272,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":210,"num_read":97,"num_read_kb":4194,"num_write":1284,"num_write_kb":15784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1551","reported_seq":3498,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.231249+0000","last_change":"2023-10-30T13:21:40.528789+0000","last_active":"2023-10-30T13:52:25.231249+0000","last_peered":"2023-10-30T13:52:25.231249+0000","last_clean":"2023-10-30T13:52:25.231249+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:52:25.231249+0000","last_undegraded":"2023-10-30T13:52:25.231249+0000","last_fullsized":"2023-10-30T13:52:25.231249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:55.527282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029909680000000001,"stat_sum":{"num_bytes":31670070,"num_objects":610,"num_object_clones":280,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":227,"num_read":110,"num_read_kb":16437,"num_write":1314,"num_write_kb":33491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1555","reported_seq":3494,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.906435+0000","last_change":"2023-10-30T13:21:40.571731+0000","last_active":"2023-10-30T13:52:23.906435+0000","last_peered":"2023-10-30T13:52:23.906435+0000","last_clean":"2023-10-30T13:52:23.906435+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:52:23.906435+0000","last_undegraded":"2023-10-30T13:52:23.906435+0000","last_fullsized":"2023-10-30T13:52:23.906435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:47.649131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058939090000000001,"stat_sum":{"num_bytes":14342306,"num_objects":613,"num_object_clones":284,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":231,"num_read":106,"num_read_kb":2953,"num_write":1333,"num_write_kb":15925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1748","reported_seq":3606,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923510+0000","last_change":"2023-10-30T13:21:40.536269+0000","last_active":"2023-10-30T13:52:23.923510+0000","last_peered":"2023-10-30T13:52:23.923510+0000","last_clean":"2023-10-30T13:52:23.923510+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:52:23.923510+0000","last_undegraded":"2023-10-30T13:52:23.923510+0000","last_fullsized":"2023-10-30T13:52:23.923510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1748,"log_dups_size":0,"ondisk_log_size":1748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:06.829080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093243890000000006,"stat_sum":{"num_bytes":10965780,"num_objects":703,"num_object_clones":374,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":231,"num_read":115,"num_read_kb":1694,"num_write":1434,"num_write_kb":17364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1579","reported_seq":3505,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.946432+0000","last_change":"2023-10-30T13:21:40.539642+0000","last_active":"2023-10-30T13:52:23.946432+0000","last_peered":"2023-10-30T13:52:23.946432+0000","last_clean":"2023-10-30T13:52:23.946432+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:52:23.946432+0000","last_undegraded":"2023-10-30T13:52:23.946432+0000","last_fullsized":"2023-10-30T13:52:23.946432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1579,"log_dups_size":0,"ondisk_log_size":1579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:12.694457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011284880000000001,"stat_sum":{"num_bytes":18442682,"num_objects":632,"num_object_clones":291,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":228,"num_read":100,"num_read_kb":8212,"num_write":1340,"num_write_kb":24815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1459","reported_seq":3397,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932387+0000","last_change":"2023-10-30T13:21:40.567529+0000","last_active":"2023-10-30T13:52:23.932387+0000","last_peered":"2023-10-30T13:52:23.932387+0000","last_clean":"2023-10-30T13:52:23.932387+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:52:23.932387+0000","last_undegraded":"2023-10-30T13:52:23.932387+0000","last_fullsized":"2023-10-30T13:52:23.932387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:45.983709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002295727,"stat_sum":{"num_bytes":22547199,"num_objects":575,"num_object_clones":267,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":207,"num_read":89,"num_read_kb":4127,"num_write":1240,"num_write_kb":24211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1620","reported_seq":3538,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.922596+0000","last_change":"2023-10-30T13:21:40.572640+0000","last_active":"2023-10-30T13:52:23.922596+0000","last_peered":"2023-10-30T13:52:23.922596+0000","last_clean":"2023-10-30T13:52:23.922596+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:52:23.922596+0000","last_undegraded":"2023-10-30T13:52:23.922596+0000","last_fullsized":"2023-10-30T13:52:23.922596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:31.079099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061689539999999999,"stat_sum":{"num_bytes":11703479,"num_objects":649,"num_object_clones":303,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":237,"num_read":106,"num_read_kb":7307,"num_write":1383,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1553","reported_seq":3491,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.915574+0000","last_change":"2023-10-30T13:21:40.580324+0000","last_active":"2023-10-30T13:52:23.915574+0000","last_peered":"2023-10-30T13:52:23.915574+0000","last_clean":"2023-10-30T13:52:23.915574+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:52:23.915574+0000","last_undegraded":"2023-10-30T13:52:23.915574+0000","last_fullsized":"2023-10-30T13:52:23.915574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1553,"log_dups_size":0,"ondisk_log_size":1553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:52.961020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094394720000000008,"stat_sum":{"num_bytes":31771338,"num_objects":621,"num_object_clones":283,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":229,"num_read":103,"num_read_kb":4126,"num_write":1338,"num_write_kb":42793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1482","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.933727+0000","last_change":"2023-10-30T13:21:40.528854+0000","last_active":"2023-10-30T13:52:23.933727+0000","last_peered":"2023-10-30T13:52:23.933727+0000","last_clean":"2023-10-30T13:52:23.933727+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:52:23.933727+0000","last_undegraded":"2023-10-30T13:52:23.933727+0000","last_fullsized":"2023-10-30T13:52:23.933727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1482,"log_dups_size":0,"ondisk_log_size":1482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:48:55.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027436349999999999,"stat_sum":{"num_bytes":6210074,"num_objects":595,"num_object_clones":273,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":228,"num_read":98,"num_read_kb":33,"num_write":1259,"num_write_kb":10653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1636","reported_seq":3545,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.244114+0000","last_change":"2023-10-30T13:21:40.535360+0000","last_active":"2023-10-30T13:52:25.244114+0000","last_peered":"2023-10-30T13:52:25.244114+0000","last_clean":"2023-10-30T13:52:25.244114+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:52:25.244114+0000","last_undegraded":"2023-10-30T13:52:25.244114+0000","last_fullsized":"2023-10-30T13:52:25.244114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1636,"log_dups_size":0,"ondisk_log_size":1636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:13.413880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082328149999999992,"stat_sum":{"num_bytes":28398973,"num_objects":670,"num_object_clones":311,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":249,"num_read":102,"num_read_kb":120,"num_write":1394,"num_write_kb":38562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1530","reported_seq":3467,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.908084+0000","last_change":"2023-10-30T13:21:40.571762+0000","last_active":"2023-10-30T13:52:23.908084+0000","last_peered":"2023-10-30T13:52:23.908084+0000","last_clean":"2023-10-30T13:52:23.908084+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:52:23.908084+0000","last_undegraded":"2023-10-30T13:52:23.908084+0000","last_fullsized":"2023-10-30T13:52:23.908084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:29:37.541276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037447470000000001,"stat_sum":{"num_bytes":19407388,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":225,"num_read":103,"num_read_kb":4138,"num_write":1298,"num_write_kb":24714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1570","reported_seq":3510,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923610+0000","last_change":"2023-10-30T13:21:40.536309+0000","last_active":"2023-10-30T13:52:23.923610+0000","last_peered":"2023-10-30T13:52:23.923610+0000","last_clean":"2023-10-30T13:52:23.923610+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:52:23.923610+0000","last_undegraded":"2023-10-30T13:52:23.923610+0000","last_fullsized":"2023-10-30T13:52:23.923610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:04:17.809358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091633449999999998,"stat_sum":{"num_bytes":10552296,"num_objects":616,"num_object_clones":294,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":228,"num_read":115,"num_read_kb":24,"num_write":1340,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1410","reported_seq":3369,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.678938+0000","last_change":"2023-10-30T13:21:40.571783+0000","last_active":"2023-10-30T13:52:25.678938+0000","last_peered":"2023-10-30T13:52:25.678938+0000","last_clean":"2023-10-30T13:52:25.678938+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:52:25.678938+0000","last_undegraded":"2023-10-30T13:52:25.678938+0000","last_fullsized":"2023-10-30T13:52:25.678938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1410,"log_dups_size":0,"ondisk_log_size":1410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:41.732560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059368210000000001,"stat_sum":{"num_bytes":15196336,"num_objects":556,"num_object_clones":256,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":202,"num_read":99,"num_read_kb":1380,"num_write":1201,"num_write_kb":16993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'1569","reported_seq":3517,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.166551+0000","last_change":"2023-10-30T13:21:40.581115+0000","last_active":"2023-10-30T13:52:24.166551+0000","last_peered":"2023-10-30T13:52:24.166551+0000","last_clean":"2023-10-30T13:52:24.166551+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:52:24.166551+0000","last_undegraded":"2023-10-30T13:52:24.166551+0000","last_fullsized":"2023-10-30T13:52:24.166551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:31.199977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099065669999999998,"stat_sum":{"num_bytes":9983325,"num_objects":619,"num_object_clones":286,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":221,"num_read":115,"num_read_kb":68,"num_write":1328,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1534","reported_seq":3477,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.910285+0000","last_change":"2023-10-30T13:21:40.572631+0000","last_active":"2023-10-30T13:52:23.910285+0000","last_peered":"2023-10-30T13:52:23.910285+0000","last_clean":"2023-10-30T13:52:23.910285+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:52:23.910285+0000","last_undegraded":"2023-10-30T13:52:23.910285+0000","last_fullsized":"2023-10-30T13:52:23.910285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1534,"log_dups_size":0,"ondisk_log_size":1534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:53.178210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059382619999999997,"stat_sum":{"num_bytes":23775876,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":236,"num_read":113,"num_read_kb":4145,"num_write":1299,"num_write_kb":26615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1462","reported_seq":3390,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.907620+0000","last_change":"2023-10-30T13:21:40.575712+0000","last_active":"2023-10-30T13:52:23.907620+0000","last_peered":"2023-10-30T13:52:23.907620+0000","last_clean":"2023-10-30T13:52:23.907620+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:52:23.907620+0000","last_undegraded":"2023-10-30T13:52:23.907620+0000","last_fullsized":"2023-10-30T13:52:23.907620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:09.905979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053787089999999997,"stat_sum":{"num_bytes":10833053,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":213,"num_read":88,"num_read_kb":3300,"num_write":1254,"num_write_kb":16238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1646","reported_seq":3545,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.233660+0000","last_change":"2023-10-30T13:21:40.529120+0000","last_active":"2023-10-30T13:52:25.233660+0000","last_peered":"2023-10-30T13:52:25.233660+0000","last_clean":"2023-10-30T13:52:25.233660+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:52:25.233660+0000","last_undegraded":"2023-10-30T13:52:25.233660+0000","last_fullsized":"2023-10-30T13:52:25.233660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1646,"log_dups_size":0,"ondisk_log_size":1646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:56.055314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028849399999999999,"stat_sum":{"num_bytes":11995550,"num_objects":688,"num_object_clones":321,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":244,"num_read":100,"num_read_kb":8267,"num_write":1414,"num_write_kb":14306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1445","reported_seq":3397,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932007+0000","last_change":"2023-10-30T13:21:40.537362+0000","last_active":"2023-10-30T13:52:23.932007+0000","last_peered":"2023-10-30T13:52:23.932007+0000","last_clean":"2023-10-30T13:52:23.932007+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:52:23.932007+0000","last_undegraded":"2023-10-30T13:52:23.932007+0000","last_fullsized":"2023-10-30T13:52:23.932007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:49:05.792581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096547909999999994,"stat_sum":{"num_bytes":15074071,"num_objects":558,"num_object_clones":259,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":221,"num_read":91,"num_read_kb":32,"num_write":1202,"num_write_kb":16754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1663","reported_seq":3603,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.926321+0000","last_change":"2023-10-30T13:21:40.567248+0000","last_active":"2023-10-30T13:52:23.926321+0000","last_peered":"2023-10-30T13:52:23.926321+0000","last_clean":"2023-10-30T13:52:23.926321+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:52:23.926321+0000","last_undegraded":"2023-10-30T13:52:23.926321+0000","last_fullsized":"2023-10-30T13:52:23.926321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:07.337543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018845820000000001,"stat_sum":{"num_bytes":10798567,"num_objects":649,"num_object_clones":300,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":241,"num_read":120,"num_read_kb":27,"num_write":1443,"num_write_kb":17235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1499","reported_seq":3469,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923689+0000","last_change":"2023-10-30T13:21:40.575268+0000","last_active":"2023-10-30T13:52:23.923689+0000","last_peered":"2023-10-30T13:52:23.923689+0000","last_clean":"2023-10-30T13:52:23.923689+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:52:23.923689+0000","last_undegraded":"2023-10-30T13:52:23.923689+0000","last_fullsized":"2023-10-30T13:52:23.923689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1499,"log_dups_size":0,"ondisk_log_size":1499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:50:11.469301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075239010000000004,"stat_sum":{"num_bytes":15423893,"num_objects":576,"num_object_clones":259,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":203,"num_read":109,"num_read_kb":9552,"num_write":1306,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1548","reported_seq":3492,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.920280+0000","last_change":"2023-10-30T13:21:40.537743+0000","last_active":"2023-10-30T13:52:23.920280+0000","last_peered":"2023-10-30T13:52:23.920280+0000","last_clean":"2023-10-30T13:52:23.920280+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:52:23.920280+0000","last_undegraded":"2023-10-30T13:52:23.920280+0000","last_fullsized":"2023-10-30T13:52:23.920280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:01:00.836625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007356301,"stat_sum":{"num_bytes":40089180,"num_objects":622,"num_object_clones":287,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":230,"num_read":115,"num_read_kb":33,"num_write":1312,"num_write_kb":42997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'604","reported_seq":832,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.233111+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:52:19.233111+0000","last_peered":"2023-10-30T13:52:19.233111+0000","last_clean":"2023-10-30T13:52:19.233111+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:52:19.233111+0000","last_undegraded":"2023-10-30T13:52:19.233111+0000","last_fullsized":"2023-10-30T13:52:19.233111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":252,"num_read_kb":249243,"num_write":662,"num_write_kb":74846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1552","reported_seq":3479,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.929541+0000","last_change":"2023-10-30T13:21:40.529443+0000","last_active":"2023-10-30T13:52:23.929541+0000","last_peered":"2023-10-30T13:52:23.929541+0000","last_clean":"2023-10-30T13:52:23.929541+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:52:23.929541+0000","last_undegraded":"2023-10-30T13:52:23.929541+0000","last_fullsized":"2023-10-30T13:52:23.929541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:39.924469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372290000000001,"stat_sum":{"num_bytes":18713903,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":229,"num_read":108,"num_read_kb":4667,"num_write":1329,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1126","reported_seq":1449,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:15.211697+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:52:15.211697+0000","last_peered":"2023-10-30T13:52:15.211697+0000","last_clean":"2023-10-30T13:52:15.211697+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:52:15.211697+0000","last_undegraded":"2023-10-30T13:52:15.211697+0000","last_fullsized":"2023-10-30T13:52:15.211697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1126,"log_dups_size":0,"ondisk_log_size":1126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25293488,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":404,"num_read_kb":309242,"num_write":1229,"num_write_kb":84643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'745","reported_seq":1123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.654334+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:52:25.654334+0000","last_peered":"2023-10-30T13:52:25.654334+0000","last_clean":"2023-10-30T13:52:25.654334+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:52:25.654334+0000","last_undegraded":"2023-10-30T13:52:25.654334+0000","last_fullsized":"2023-10-30T13:52:25.654334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397151,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":403,"num_read_kb":153383,"num_write":818,"num_write_kb":55113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1778","reported_seq":3695,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.668056+0000","last_change":"2023-10-30T13:21:40.575255+0000","last_active":"2023-10-30T13:52:25.668056+0000","last_peered":"2023-10-30T13:52:25.668056+0000","last_clean":"2023-10-30T13:52:25.668056+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:52:25.668056+0000","last_undegraded":"2023-10-30T13:52:25.668056+0000","last_fullsized":"2023-10-30T13:52:25.668056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1778,"log_dups_size":0,"ondisk_log_size":1778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:20.571707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064075410000000001,"stat_sum":{"num_bytes":11666119,"num_objects":729,"num_object_clones":341,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":261,"num_read":141,"num_read_kb":4175,"num_write":1518,"num_write_kb":22588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1399","reported_seq":1668,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.266715+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:52:24.266715+0000","last_peered":"2023-10-30T13:52:24.266715+0000","last_clean":"2023-10-30T13:52:24.266715+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:52:24.266715+0000","last_undegraded":"2023-10-30T13:52:24.266715+0000","last_fullsized":"2023-10-30T13:52:24.266715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1399,"log_dups_size":0,"ondisk_log_size":1399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":354,"num_read_kb":270680,"num_write":1485,"num_write_kb":75457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1501","reported_seq":3434,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.893719+0000","last_change":"2023-10-30T13:21:40.571518+0000","last_active":"2023-10-30T13:52:23.893719+0000","last_peered":"2023-10-30T13:52:23.893719+0000","last_clean":"2023-10-30T13:52:23.893719+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:52:23.893719+0000","last_undegraded":"2023-10-30T13:52:23.893719+0000","last_fullsized":"2023-10-30T13:52:23.893719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:19:07.205330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058964639999999997,"stat_sum":{"num_bytes":19507019,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":240,"num_read":105,"num_read_kb":5362,"num_write":1250,"num_write_kb":24752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'953","reported_seq":1189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.509429+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:52:24.509429+0000","last_peered":"2023-10-30T13:52:24.509429+0000","last_clean":"2023-10-30T13:52:24.509429+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:52:24.509429+0000","last_undegraded":"2023-10-30T13:52:24.509429+0000","last_fullsized":"2023-10-30T13:52:24.509429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":313,"num_read_kb":141404,"num_write":1072,"num_write_kb":63726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1479","reported_seq":3432,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.935516+0000","last_change":"2023-10-30T13:21:40.567796+0000","last_active":"2023-10-30T13:52:23.935516+0000","last_peered":"2023-10-30T13:52:23.935516+0000","last_clean":"2023-10-30T13:52:23.935516+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:52:23.935516+0000","last_undegraded":"2023-10-30T13:52:23.935516+0000","last_fullsized":"2023-10-30T13:52:23.935516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:08:51.743716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025817129999999998,"stat_sum":{"num_bytes":23539910,"num_objects":588,"num_object_clones":267,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":208,"num_read":106,"num_read_kb":8313,"num_write":1264,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"67'1122","reported_seq":1340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:17.672748+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:52:17.672748+0000","last_peered":"2023-10-30T13:52:17.672748+0000","last_clean":"2023-10-30T13:52:17.672748+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:52:17.672748+0000","last_undegraded":"2023-10-30T13:52:17.672748+0000","last_fullsized":"2023-10-30T13:52:17.672748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1122,"log_dups_size":0,"ondisk_log_size":1122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":240,"num_read_kb":201173,"num_write":1196,"num_write_kb":66729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1545","reported_seq":3456,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.938692+0000","last_change":"2023-10-30T13:21:40.528173+0000","last_active":"2023-10-30T13:52:23.938692+0000","last_peered":"2023-10-30T13:52:23.938692+0000","last_clean":"2023-10-30T13:52:23.938692+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:52:23.938692+0000","last_undegraded":"2023-10-30T13:52:23.938692+0000","last_fullsized":"2023-10-30T13:52:23.938692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1545,"log_dups_size":0,"ondisk_log_size":1545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:50:03.879621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022874710000000001,"stat_sum":{"num_bytes":16137402,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":233,"num_read":91,"num_read_kb":34,"num_write":1322,"num_write_kb":18013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1315","reported_seq":1556,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.212373+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:52:24.212373+0000","last_peered":"2023-10-30T13:52:24.212373+0000","last_clean":"2023-10-30T13:52:24.212373+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:52:24.212373+0000","last_undegraded":"2023-10-30T13:52:24.212373+0000","last_fullsized":"2023-10-30T13:52:24.212373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20972192,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":307,"num_read_kb":165804,"num_write":1420,"num_write_kb":50966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1560","reported_seq":3492,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.937626+0000","last_change":"2023-10-30T13:21:40.538439+0000","last_active":"2023-10-30T13:52:23.937626+0000","last_peered":"2023-10-30T13:52:23.937626+0000","last_clean":"2023-10-30T13:52:23.937626+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:52:23.937626+0000","last_undegraded":"2023-10-30T13:52:23.937626+0000","last_fullsized":"2023-10-30T13:52:23.937626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:36:06.776976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087798259999999993,"stat_sum":{"num_bytes":32677996,"num_objects":638,"num_object_clones":293,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":232,"num_read":108,"num_read_kb":10951,"num_write":1340,"num_write_kb":38819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'935","reported_seq":1253,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.960735+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:52:23.960735+0000","last_peered":"2023-10-30T13:52:23.960735+0000","last_clean":"2023-10-30T13:52:23.960735+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:52:23.960735+0000","last_undegraded":"2023-10-30T13:52:23.960735+0000","last_fullsized":"2023-10-30T13:52:23.960735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":413,"num_read_kb":281405,"num_write":1056,"num_write_kb":80196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1490","reported_seq":3405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.927330+0000","last_change":"2023-10-30T13:21:40.583356+0000","last_active":"2023-10-30T13:52:23.927330+0000","last_peered":"2023-10-30T13:52:23.927330+0000","last_clean":"2023-10-30T13:52:23.927330+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:52:23.927330+0000","last_undegraded":"2023-10-30T13:52:23.927330+0000","last_fullsized":"2023-10-30T13:52:23.927330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:46.232350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014787140000000001,"stat_sum":{"num_bytes":19072740,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":238,"num_read":87,"num_read_kb":102,"num_write":1252,"num_write_kb":21862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'1084","reported_seq":1410,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.582275+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:52:16.582275+0000","last_peered":"2023-10-30T13:52:16.582275+0000","last_clean":"2023-10-30T13:52:16.582275+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:52:16.582275+0000","last_undegraded":"2023-10-30T13:52:16.582275+0000","last_fullsized":"2023-10-30T13:52:16.582275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748770,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":476,"num_read_kb":203988,"num_write":1202,"num_write_kb":67245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1496","reported_seq":3446,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.929377+0000","last_change":"2023-10-30T13:21:40.528316+0000","last_active":"2023-10-30T13:52:23.929377+0000","last_peered":"2023-10-30T13:52:23.929377+0000","last_clean":"2023-10-30T13:52:23.929377+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:52:23.929377+0000","last_undegraded":"2023-10-30T13:52:23.929377+0000","last_fullsized":"2023-10-30T13:52:23.929377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:08:42.998206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024076599999999998,"stat_sum":{"num_bytes":14342228,"num_objects":584,"num_object_clones":268,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":211,"num_read":101,"num_read_kb":4159,"num_write":1284,"num_write_kb":24386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'574","reported_seq":755,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.233164+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:52:19.233164+0000","last_peered":"2023-10-30T13:52:19.233164+0000","last_clean":"2023-10-30T13:52:19.233164+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:52:19.233164+0000","last_undegraded":"2023-10-30T13:52:19.233164+0000","last_fullsized":"2023-10-30T13:52:19.233164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":574,"log_dups_size":0,"ondisk_log_size":574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":203,"num_read_kb":83137,"num_write":662,"num_write_kb":45991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1588","reported_seq":3563,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.559855+0000","last_change":"2023-10-30T13:21:40.579925+0000","last_active":"2023-10-30T13:52:18.559855+0000","last_peered":"2023-10-30T13:52:18.559855+0000","last_clean":"2023-10-30T13:52:18.559855+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:52:18.559855+0000","last_undegraded":"2023-10-30T13:52:18.559855+0000","last_fullsized":"2023-10-30T13:52:18.559855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1588,"log_dups_size":0,"ondisk_log_size":1588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:39.495586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011943199999999999,"stat_sum":{"num_bytes":18918293,"num_objects":634,"num_object_clones":295,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":241,"num_read":154,"num_read_kb":114,"num_write":1356,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'499","reported_seq":731,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.240235+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:52:19.240235+0000","last_peered":"2023-10-30T13:52:19.240235+0000","last_clean":"2023-10-30T13:52:19.240235+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:52:19.240235+0000","last_undegraded":"2023-10-30T13:52:19.240235+0000","last_fullsized":"2023-10-30T13:52:19.240235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":289,"num_read_kb":121284,"num_write":554,"num_write_kb":58687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1629","reported_seq":3510,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.242217+0000","last_change":"2023-10-30T13:21:40.576918+0000","last_active":"2023-10-30T13:52:25.242217+0000","last_peered":"2023-10-30T13:52:25.242217+0000","last_clean":"2023-10-30T13:52:25.242217+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:52:25.242217+0000","last_undegraded":"2023-10-30T13:52:25.242217+0000","last_fullsized":"2023-10-30T13:52:25.242217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1629,"log_dups_size":0,"ondisk_log_size":1629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:33.447713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086123929999999994,"stat_sum":{"num_bytes":19792070,"num_objects":655,"num_object_clones":349,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":204,"num_read":113,"num_read_kb":142,"num_write":1344,"num_write_kb":24080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'960","reported_seq":1239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:15.167943+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:52:15.167943+0000","last_peered":"2023-10-30T13:52:15.167943+0000","last_clean":"2023-10-30T13:52:15.167943+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:52:15.167943+0000","last_undegraded":"2023-10-30T13:52:15.167943+0000","last_fullsized":"2023-10-30T13:52:15.167943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36678727,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":465,"num_read_kb":133831,"num_write":1107,"num_write_kb":54499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1554","reported_seq":3475,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.928034+0000","last_change":"2023-10-30T13:21:40.583214+0000","last_active":"2023-10-30T13:52:23.928034+0000","last_peered":"2023-10-30T13:52:23.928034+0000","last_clean":"2023-10-30T13:52:23.928034+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:52:23.928034+0000","last_undegraded":"2023-10-30T13:52:23.928034+0000","last_fullsized":"2023-10-30T13:52:23.928034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1554,"log_dups_size":0,"ondisk_log_size":1554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:22.371097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015028952999999999,"stat_sum":{"num_bytes":21253392,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":236,"num_read":95,"num_read_kb":12339,"num_write":1304,"num_write_kb":30718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'638","reported_seq":863,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.241252+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:52:19.241252+0000","last_peered":"2023-10-30T13:52:19.241252+0000","last_clean":"2023-10-30T13:52:19.241252+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:52:19.241252+0000","last_undegraded":"2023-10-30T13:52:19.241252+0000","last_fullsized":"2023-10-30T13:52:19.241252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165846,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":249,"num_read_kb":148701,"num_write":777,"num_write_kb":54421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1551","reported_seq":3481,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.932444+0000","last_change":"2023-10-30T13:21:40.579365+0000","last_active":"2023-10-30T13:52:23.932444+0000","last_peered":"2023-10-30T13:52:23.932444+0000","last_clean":"2023-10-30T13:52:23.932444+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:52:23.932444+0000","last_undegraded":"2023-10-30T13:52:23.932444+0000","last_fullsized":"2023-10-30T13:52:23.932444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:51:50.149626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011682140000000001,"stat_sum":{"num_bytes":26941199,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":237,"num_read":103,"num_read_kb":4114,"num_write":1331,"num_write_kb":32609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'1051","reported_seq":1327,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.149100+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:52:16.149100+0000","last_peered":"2023-10-30T13:52:16.149100+0000","last_clean":"2023-10-30T13:52:16.149100+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:52:16.149100+0000","last_undegraded":"2023-10-30T13:52:16.149100+0000","last_fullsized":"2023-10-30T13:52:16.149100+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1051,"log_dups_size":0,"ondisk_log_size":1051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":381,"num_read_kb":250673,"num_write":1155,"num_write_kb":87740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1528","reported_seq":3470,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.951203+0000","last_change":"2023-10-30T13:21:40.585506+0000","last_active":"2023-10-30T13:52:23.951203+0000","last_peered":"2023-10-30T13:52:23.951203+0000","last_clean":"2023-10-30T13:52:23.951203+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:52:23.951203+0000","last_undegraded":"2023-10-30T13:52:23.951203+0000","last_fullsized":"2023-10-30T13:52:23.951203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:06.635146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014794043,"stat_sum":{"num_bytes":18888621,"num_objects":614,"num_object_clones":283,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":224,"num_read":107,"num_read_kb":4188,"num_write":1306,"num_write_kb":24240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'1227","reported_seq":1478,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.806783+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:52:23.806783+0000","last_peered":"2023-10-30T13:52:23.806783+0000","last_clean":"2023-10-30T13:52:23.806783+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:52:23.806783+0000","last_undegraded":"2023-10-30T13:52:23.806783+0000","last_fullsized":"2023-10-30T13:52:23.806783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1227,"log_dups_size":0,"ondisk_log_size":1227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364039,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":288,"num_read_kb":200539,"num_write":1325,"num_write_kb":71054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1570","reported_seq":3523,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923756+0000","last_change":"2023-10-30T13:21:40.538748+0000","last_active":"2023-10-30T13:52:23.923756+0000","last_peered":"2023-10-30T13:52:23.923756+0000","last_clean":"2023-10-30T13:52:23.923756+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:52:23.923756+0000","last_undegraded":"2023-10-30T13:52:23.923756+0000","last_fullsized":"2023-10-30T13:52:23.923756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:15.511609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010788236,"stat_sum":{"num_bytes":10484856,"num_objects":600,"num_object_clones":281,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":220,"num_read":116,"num_read_kb":28,"num_write":1335,"num_write_kb":13295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'776","reported_seq":1023,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.577979+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:52:16.577979+0000","last_peered":"2023-10-30T13:52:16.577979+0000","last_clean":"2023-10-30T13:52:16.577979+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:52:16.577979+0000","last_undegraded":"2023-10-30T13:52:16.577979+0000","last_fullsized":"2023-10-30T13:52:16.577979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":322,"num_read_kb":174067,"num_write":894,"num_write_kb":62874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1536","reported_seq":3467,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.231736+0000","last_change":"2023-10-30T13:21:40.583977+0000","last_active":"2023-10-30T13:52:25.231736+0000","last_peered":"2023-10-30T13:52:25.231736+0000","last_clean":"2023-10-30T13:52:25.231736+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:52:25.231736+0000","last_undegraded":"2023-10-30T13:52:25.231736+0000","last_fullsized":"2023-10-30T13:52:25.231736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1536,"log_dups_size":0,"ondisk_log_size":1536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:18.653967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015268978000000001,"stat_sum":{"num_bytes":11043591,"num_objects":616,"num_object_clones":291,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":221,"num_read":104,"num_read_kb":3707,"num_write":1322,"num_write_kb":16509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":232,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T13:50:37.450196+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:50:37.450196+0000","last_peered":"2023-10-30T13:50:37.450196+0000","last_clean":"2023-10-30T13:50:37.450196+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:50:37.450196+0000","last_undegraded":"2023-10-30T13:50:37.450196+0000","last_fullsized":"2023-10-30T13:50:37.450196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'617","reported_seq":825,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.795368+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:52:23.795368+0000","last_peered":"2023-10-30T13:52:23.795368+0000","last_clean":"2023-10-30T13:52:23.795368+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:52:23.795368+0000","last_undegraded":"2023-10-30T13:52:23.795368+0000","last_fullsized":"2023-10-30T13:52:23.795368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360178,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":232,"num_read_kb":148904,"num_write":693,"num_write_kb":62690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1532","reported_seq":3445,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.667425+0000","last_change":"2023-10-30T13:21:40.574205+0000","last_active":"2023-10-30T13:52:25.667425+0000","last_peered":"2023-10-30T13:52:25.667425+0000","last_clean":"2023-10-30T13:52:25.667425+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:52:25.667425+0000","last_undegraded":"2023-10-30T13:52:25.667425+0000","last_fullsized":"2023-10-30T13:52:25.667425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1532,"log_dups_size":0,"ondisk_log_size":1532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:45:44.062637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057314540000000004,"stat_sum":{"num_bytes":22780711,"num_objects":633,"num_object_clones":299,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":243,"num_read":92,"num_read_kb":12,"num_write":1291,"num_write_kb":28237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"67'705","reported_seq":936,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.241719+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:52:19.241719+0000","last_peered":"2023-10-30T13:52:19.241719+0000","last_clean":"2023-10-30T13:52:19.241719+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:52:19.241719+0000","last_undegraded":"2023-10-30T13:52:19.241719+0000","last_fullsized":"2023-10-30T13:52:19.241719+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":256,"num_read_kb":247261,"num_write":738,"num_write_kb":70569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1453","reported_seq":3390,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.925643+0000","last_change":"2023-10-30T13:21:40.528051+0000","last_active":"2023-10-30T13:52:23.925643+0000","last_peered":"2023-10-30T13:52:23.925643+0000","last_clean":"2023-10-30T13:52:23.925643+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:52:23.925643+0000","last_undegraded":"2023-10-30T13:52:23.925643+0000","last_fullsized":"2023-10-30T13:52:23.925643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1453,"log_dups_size":0,"ondisk_log_size":1453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:08:49.848069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002406114,"stat_sum":{"num_bytes":14184993,"num_objects":568,"num_object_clones":263,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":210,"num_read":80,"num_read_kb":26,"num_write":1239,"num_write_kb":16314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'2040","reported_seq":3405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.224696+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:52:25.224696+0000","last_peered":"2023-10-30T13:52:25.224696+0000","last_clean":"2023-10-30T13:52:25.224696+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:52:25.224696+0000","last_undegraded":"2023-10-30T13:52:25.224696+0000","last_fullsized":"2023-10-30T13:52:25.224696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41670866,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":1534,"num_read_kb":188566,"num_write":2090,"num_write_kb":83865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1512","reported_seq":3453,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.919153+0000","last_change":"2023-10-30T13:21:40.568287+0000","last_active":"2023-10-30T13:52:23.919153+0000","last_peered":"2023-10-30T13:52:23.919153+0000","last_clean":"2023-10-30T13:52:23.919153+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:52:23.919153+0000","last_undegraded":"2023-10-30T13:52:23.919153+0000","last_fullsized":"2023-10-30T13:52:23.919153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:16.792299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002908281,"stat_sum":{"num_bytes":22417951,"num_objects":612,"num_object_clones":284,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":221,"num_read":105,"num_read_kb":44,"num_write":1296,"num_write_kb":23585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'947","reported_seq":1222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.180646+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:52:24.180646+0000","last_peered":"2023-10-30T13:52:24.180646+0000","last_clean":"2023-10-30T13:52:24.180646+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:52:24.180646+0000","last_undegraded":"2023-10-30T13:52:24.180646+0000","last_fullsized":"2023-10-30T13:52:24.180646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":355,"num_read_kb":252215,"num_write":1073,"num_write_kb":75653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1520","reported_seq":3474,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.663361+0000","last_change":"2023-10-30T13:21:40.536723+0000","last_active":"2023-10-30T13:52:25.663361+0000","last_peered":"2023-10-30T13:52:25.663361+0000","last_clean":"2023-10-30T13:52:25.663361+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:52:25.663361+0000","last_undegraded":"2023-10-30T13:52:25.663361+0000","last_fullsized":"2023-10-30T13:52:25.663361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:33:25.077240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007710705,"stat_sum":{"num_bytes":23751577,"num_objects":575,"num_object_clones":266,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":199,"num_read":100,"num_read_kb":489,"num_write":1325,"num_write_kb":27587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'943","reported_seq":1183,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.547312+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:52:18.547312+0000","last_peered":"2023-10-30T13:52:18.547312+0000","last_clean":"2023-10-30T13:52:18.547312+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:52:18.547312+0000","last_undegraded":"2023-10-30T13:52:18.547312+0000","last_fullsized":"2023-10-30T13:52:18.547312+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":294,"num_read_kb":227180,"num_write":1037,"num_write_kb":87579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1589","reported_seq":3499,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.923728+0000","last_change":"2023-10-30T13:21:40.539081+0000","last_active":"2023-10-30T13:52:23.923728+0000","last_peered":"2023-10-30T13:52:23.923728+0000","last_clean":"2023-10-30T13:52:23.923728+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:52:23.923728+0000","last_undegraded":"2023-10-30T13:52:23.923728+0000","last_fullsized":"2023-10-30T13:52:23.923728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1589,"log_dups_size":0,"ondisk_log_size":1589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:36.291322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010719574000000001,"stat_sum":{"num_bytes":35538242,"num_objects":650,"num_object_clones":301,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":240,"num_read":91,"num_read_kb":31,"num_write":1345,"num_write_kb":41412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'785","reported_seq":1010,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.206722+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:52:24.206722+0000","last_peered":"2023-10-30T13:52:24.206722+0000","last_clean":"2023-10-30T13:52:24.206722+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:52:24.206722+0000","last_undegraded":"2023-10-30T13:52:24.206722+0000","last_fullsized":"2023-10-30T13:52:24.206722+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":258,"num_read_kb":177173,"num_write":842,"num_write_kb":62518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1541","reported_seq":3467,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.919237+0000","last_change":"2023-10-30T13:21:40.572129+0000","last_active":"2023-10-30T13:52:23.919237+0000","last_peered":"2023-10-30T13:52:23.919237+0000","last_clean":"2023-10-30T13:52:23.919237+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:52:23.919237+0000","last_undegraded":"2023-10-30T13:52:23.919237+0000","last_fullsized":"2023-10-30T13:52:23.919237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1541,"log_dups_size":0,"ondisk_log_size":1541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:23.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059276069999999997,"stat_sum":{"num_bytes":23389062,"num_objects":631,"num_object_clones":294,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":232,"num_read":100,"num_read_kb":25,"num_write":1313,"num_write_kb":24315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'914","reported_seq":1153,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.107011+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:52:19.107011+0000","last_peered":"2023-10-30T13:52:19.107011+0000","last_clean":"2023-10-30T13:52:19.107011+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:52:19.107011+0000","last_undegraded":"2023-10-30T13:52:19.107011+0000","last_fullsized":"2023-10-30T13:52:19.107011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":310,"num_read_kb":109156,"num_write":1491,"num_write_kb":61820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1404","reported_seq":3356,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.931293+0000","last_change":"2023-10-30T13:21:40.576235+0000","last_active":"2023-10-30T13:52:23.931293+0000","last_peered":"2023-10-30T13:52:23.931293+0000","last_clean":"2023-10-30T13:52:23.931293+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:52:23.931293+0000","last_undegraded":"2023-10-30T13:52:23.931293+0000","last_fullsized":"2023-10-30T13:52:23.931293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1404,"log_dups_size":0,"ondisk_log_size":1404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:30.114376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056940879999999999,"stat_sum":{"num_bytes":23042560,"num_objects":571,"num_object_clones":259,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":201,"num_read":92,"num_read_kb":75,"num_write":1201,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'1095","reported_seq":1338,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.531235+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:52:16.531235+0000","last_peered":"2023-10-30T13:52:16.531235+0000","last_clean":"2023-10-30T13:52:16.531235+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:52:16.531235+0000","last_undegraded":"2023-10-30T13:52:16.531235+0000","last_fullsized":"2023-10-30T13:52:16.531235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":342,"num_read_kb":120967,"num_write":1229,"num_write_kb":51134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1436","reported_seq":3397,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.918796+0000","last_change":"2023-10-30T13:21:40.529108+0000","last_active":"2023-10-30T13:52:23.918796+0000","last_peered":"2023-10-30T13:52:23.918796+0000","last_clean":"2023-10-30T13:52:23.918796+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:52:23.918796+0000","last_undegraded":"2023-10-30T13:52:23.918796+0000","last_fullsized":"2023-10-30T13:52:23.918796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1436,"log_dups_size":0,"ondisk_log_size":1436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:16:51.856467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029926589999999999,"stat_sum":{"num_bytes":30813966,"num_objects":572,"num_object_clones":263,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":205,"num_read":105,"num_read_kb":87,"num_write":1219,"num_write_kb":33525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'421","reported_seq":607,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.800787+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:52:23.800787+0000","last_peered":"2023-10-30T13:52:23.800787+0000","last_clean":"2023-10-30T13:52:23.800787+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:52:23.800787+0000","last_undegraded":"2023-10-30T13:52:23.800787+0000","last_fullsized":"2023-10-30T13:52:23.800787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":209,"num_read_kb":100741,"num_write":482,"num_write_kb":29419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1533","reported_seq":3487,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.926144+0000","last_change":"2023-10-30T13:21:40.533380+0000","last_active":"2023-10-30T13:52:23.926144+0000","last_peered":"2023-10-30T13:52:23.926144+0000","last_clean":"2023-10-30T13:52:23.926144+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:52:23.926144+0000","last_undegraded":"2023-10-30T13:52:23.926144+0000","last_fullsized":"2023-10-30T13:52:23.926144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1533,"log_dups_size":0,"ondisk_log_size":1533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:46.427030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057673200000000003,"stat_sum":{"num_bytes":13994934,"num_objects":595,"num_object_clones":279,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":215,"num_read":114,"num_read_kb":29,"num_write":1327,"num_write_kb":19694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'1204","reported_seq":1418,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.263849+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:52:24.263849+0000","last_peered":"2023-10-30T13:52:24.263849+0000","last_clean":"2023-10-30T13:52:24.263849+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:52:24.263849+0000","last_undegraded":"2023-10-30T13:52:24.263849+0000","last_fullsized":"2023-10-30T13:52:24.263849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21903607,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":251,"num_read_kb":99282,"num_write":1273,"num_write_kb":39377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1544","reported_seq":3497,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.892716+0000","last_change":"2023-10-30T13:21:40.575380+0000","last_active":"2023-10-30T13:52:23.892716+0000","last_peered":"2023-10-30T13:52:23.892716+0000","last_clean":"2023-10-30T13:52:23.892716+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:52:23.892716+0000","last_undegraded":"2023-10-30T13:52:23.892716+0000","last_fullsized":"2023-10-30T13:52:23.892716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1544,"log_dups_size":0,"ondisk_log_size":1544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:12.098739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063570620000000001,"stat_sum":{"num_bytes":16293763,"num_objects":612,"num_object_clones":286,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":233,"num_read":120,"num_read_kb":127,"num_write":1302,"num_write_kb":21658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'900","reported_seq":1126,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.166933+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:52:24.166933+0000","last_peered":"2023-10-30T13:52:24.166933+0000","last_clean":"2023-10-30T13:52:24.166933+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:52:24.166933+0000","last_undegraded":"2023-10-30T13:52:24.166933+0000","last_fullsized":"2023-10-30T13:52:24.166933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":272,"num_read_kb":200216,"num_write":1131,"num_write_kb":59038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1424","reported_seq":3373,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.237106+0000","last_change":"2023-10-30T13:21:40.575515+0000","last_active":"2023-10-30T13:52:25.237106+0000","last_peered":"2023-10-30T13:52:25.237106+0000","last_clean":"2023-10-30T13:52:25.237106+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:52:25.237106+0000","last_undegraded":"2023-10-30T13:52:25.237106+0000","last_fullsized":"2023-10-30T13:52:25.237106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1424,"log_dups_size":0,"ondisk_log_size":1424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:20.385719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060201619999999999,"stat_sum":{"num_bytes":10033117,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":215,"num_read":92,"num_read_kb":24,"num_write":1202,"num_write_kb":13175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'967","reported_seq":1222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:21.614332+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:52:21.614332+0000","last_peered":"2023-10-30T13:52:21.614332+0000","last_clean":"2023-10-30T13:52:21.614332+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:52:21.614332+0000","last_undegraded":"2023-10-30T13:52:21.614332+0000","last_fullsized":"2023-10-30T13:52:21.614332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":318,"num_read_kb":202944,"num_write":1058,"num_write_kb":67603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1452","reported_seq":3400,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.930308+0000","last_change":"2023-10-30T13:21:40.575093+0000","last_active":"2023-10-30T13:52:23.930308+0000","last_peered":"2023-10-30T13:52:23.930308+0000","last_clean":"2023-10-30T13:52:23.930308+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:52:23.930308+0000","last_undegraded":"2023-10-30T13:52:23.930308+0000","last_fullsized":"2023-10-30T13:52:23.930308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1452,"log_dups_size":0,"ondisk_log_size":1452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:55:44.047461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079253199999999996,"stat_sum":{"num_bytes":6591050,"num_objects":558,"num_object_clones":254,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":201,"num_read":83,"num_read_kb":63,"num_write":1251,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1486","reported_seq":3430,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.929689+0000","last_change":"2023-10-30T13:21:40.538321+0000","last_active":"2023-10-30T13:52:23.929689+0000","last_peered":"2023-10-30T13:52:23.929689+0000","last_clean":"2023-10-30T13:52:23.929689+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:52:23.929689+0000","last_undegraded":"2023-10-30T13:52:23.929689+0000","last_fullsized":"2023-10-30T13:52:23.929689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:04.667732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010603414,"stat_sum":{"num_bytes":9104816,"num_objects":585,"num_object_clones":276,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":204,"num_read":104,"num_read_kb":44,"num_write":1283,"num_write_kb":14447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'1000","reported_seq":1254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.409685+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:52:18.409685+0000","last_peered":"2023-10-30T13:52:18.409685+0000","last_clean":"2023-10-30T13:52:18.409685+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:52:18.409685+0000","last_undegraded":"2023-10-30T13:52:18.409685+0000","last_fullsized":"2023-10-30T13:52:18.409685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":337,"num_read_kb":187117,"num_write":1194,"num_write_kb":47550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1610","reported_seq":3524,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.937747+0000","last_change":"2023-10-30T13:21:40.537202+0000","last_active":"2023-10-30T13:52:23.937747+0000","last_peered":"2023-10-30T13:52:23.937747+0000","last_clean":"2023-10-30T13:52:23.937747+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:52:23.937747+0000","last_undegraded":"2023-10-30T13:52:23.937747+0000","last_fullsized":"2023-10-30T13:52:23.937747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1610,"log_dups_size":0,"ondisk_log_size":1610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:39:17.448171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097894100000000001,"stat_sum":{"num_bytes":19085997,"num_objects":650,"num_object_clones":307,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":230,"num_read":105,"num_read_kb":202,"num_write":1384,"num_write_kb":24607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1264","reported_seq":1566,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.267745+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:52:24.267745+0000","last_peered":"2023-10-30T13:52:24.267745+0000","last_clean":"2023-10-30T13:52:24.267745+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:52:24.267745+0000","last_undegraded":"2023-10-30T13:52:24.267745+0000","last_fullsized":"2023-10-30T13:52:24.267745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1264,"log_dups_size":0,"ondisk_log_size":1264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34811142,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":418,"num_read_kb":304041,"num_write":1365,"num_write_kb":89722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1502","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.656168+0000","last_change":"2023-10-30T13:21:40.528576+0000","last_active":"2023-10-30T13:52:24.656168+0000","last_peered":"2023-10-30T13:52:24.656168+0000","last_clean":"2023-10-30T13:52:24.656168+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:52:24.656168+0000","last_undegraded":"2023-10-30T13:52:24.656168+0000","last_fullsized":"2023-10-30T13:52:24.656168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1502,"log_dups_size":0,"ondisk_log_size":1502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:58.014283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026802879999999999,"stat_sum":{"num_bytes":22843366,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":237,"num_read":96,"num_read_kb":23,"num_write":1281,"num_write_kb":25082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'773","reported_seq":1005,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.811859+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:52:23.811859+0000","last_peered":"2023-10-30T13:52:23.811859+0000","last_clean":"2023-10-30T13:52:23.811859+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:52:23.811859+0000","last_undegraded":"2023-10-30T13:52:23.811859+0000","last_fullsized":"2023-10-30T13:52:23.811859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":280,"num_read_kb":212075,"num_write":833,"num_write_kb":74728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1454","reported_seq":3408,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.933645+0000","last_change":"2023-10-30T13:21:40.576062+0000","last_active":"2023-10-30T13:52:23.933645+0000","last_peered":"2023-10-30T13:52:23.933645+0000","last_clean":"2023-10-30T13:52:23.933645+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:52:23.933645+0000","last_undegraded":"2023-10-30T13:52:23.933645+0000","last_fullsized":"2023-10-30T13:52:23.933645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1454,"log_dups_size":0,"ondisk_log_size":1454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:28.354972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055040289999999997,"stat_sum":{"num_bytes":14435337,"num_objects":589,"num_object_clones":269,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":212,"num_read":103,"num_read_kb":22,"num_write":1235,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'708","reported_seq":971,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.234153+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:52:19.234153+0000","last_peered":"2023-10-30T13:52:19.234153+0000","last_clean":"2023-10-30T13:52:19.234153+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:52:19.234153+0000","last_undegraded":"2023-10-30T13:52:19.234153+0000","last_fullsized":"2023-10-30T13:52:19.234153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749318,"num_objects":65,"num_object_clones":0,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":374,"num_read_kb":96620,"num_write":839,"num_write_kb":51020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165182927,"num_objects":39302,"num_object_clones":18317,"num_object_copies":78604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39302,"num_whiteouts":14355,"num_read":6653,"num_read_kb":168434,"num_write":83420,"num_write_kb":1459733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424070144,"data_stored":2332381086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97918,"ondisk_log_size":97918,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903181988,"num_objects":1685,"num_object_clones":0,"num_object_copies":3370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":11399,"num_read_kb":5963012,"num_write":33982,"num_write_kb":2068472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1396,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806508032,"data_stored":1806363976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30291,"ondisk_log_size":30291,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575358,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724736,"kb_used_data":571384,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93647104,"statfs":{"total":96636764160,"available":95894634496,"internally_reserved":0,"allocated":585097216,"data_stored":571472003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":157022667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.95099999999999996},"min":{"1min":0.47999999999999998,"5min":0.41199999999999998,"15min":0.40899999999999997},"max":{"1min":1.288,"5min":1.321,"15min":15.381},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81899999999999995,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.315},"max":{"1min":1.212,"5min":1.8400000000000001,"15min":2.9510000000000001},"last":0.77300000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.83799999999999997,"15min":0.92800000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3240000000000001,"5min":1.79,"15min":15.271000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89100000000000001,"15min":0.96199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.42999999999999999},"max":{"1min":1.2769999999999999,"5min":1.6000000000000001,"15min":15.228999999999999},"last":0.84299999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92100000000000004,"15min":0.91300000000000003},"min":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.36299999999999999},"max":{"1min":1.3100000000000001,"5min":1.345,"15min":2.2429999999999999},"last":1.056},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.89000000000000001,"15min":0.877},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.242,"5min":1.929,"15min":2.4540000000000002},"last":0.96899999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95599999999999996,"15min":1.0800000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.256,"5min":2.0249999999999999,"15min":15.324},"last":0.624},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.96999999999999997,"15min":0.98599999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2629999999999999,"5min":1.323,"15min":1.637},"last":1.012}]},{"osd":4,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.091,"15min":1.1479999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.40899999999999997},"max":{"1min":1.3080000000000001,"5min":2.7869999999999999,"15min":15.172000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.093,"5min":1.0600000000000001,"15min":1.089},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.466,"5min":3.032,"15min":4.0209999999999999},"last":1.1259999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1120000000000001,"15min":1.1060000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.454,"5min":1.4770000000000001,"15min":1.7050000000000001},"last":1.163},{"interface":"front","average":{"1min":1.089,"5min":1.0600000000000001,"15min":1.0169999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.54500000000000004},"max":{"1min":1.4159999999999999,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":1.105}]},{"osd":6,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.1819999999999999,"15min":1.2509999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.46500000000000002},"max":{"1min":1.7829999999999999,"5min":3.0350000000000001,"15min":15.427},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.228,"15min":1.331},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.42199999999999999},"max":{"1min":1.841,"5min":3.5169999999999999,"15min":32.976999999999997},"last":1.2869999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575357,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593340,"kb_used_data":449780,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93778500,"statfs":{"total":96636764160,"available":96029184000,"internally_reserved":0,"allocated":460574720,"data_stored":451977574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":146995655},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.55000000000000004,"5min":0.45000000000000001,"15min":0.40999999999999998},"max":{"1min":1.2190000000000001,"5min":1.3999999999999999,"15min":5.9880000000000004},"last":1.087},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.86899999999999999,"15min":0.89200000000000002},"min":{"1min":0.58999999999999997,"5min":0.48599999999999999,"15min":0.35999999999999999},"max":{"1min":1.0620000000000001,"5min":1.3160000000000001,"15min":6.0730000000000004},"last":0.90300000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.93600000000000005,"15min":0.86499999999999999},"min":{"1min":0.58199999999999996,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.585,"5min":6.9930000000000003,"15min":6.9930000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.86399999999999999,"15min":0.86099999999999999},"min":{"1min":0.69299999999999995,"5min":0.52500000000000002,"15min":0.47499999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.9810000000000001},"last":0.79200000000000004}]},{"osd":2,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":0.89400000000000002},"min":{"1min":0.59899999999999998,"5min":0.54400000000000004,"15min":0.47699999999999998},"max":{"1min":1.044,"5min":6.8550000000000004,"15min":6.8550000000000004},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.97999999999999998,"15min":0.96399999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.454,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.83399999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88600000000000001,"15min":0.89400000000000002},"min":{"1min":0.64800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.2130000000000001,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.96899999999999997,"15min":0.98699999999999999},"min":{"1min":0.69899999999999995,"5min":0.64400000000000002,"15min":0.56299999999999994},"max":{"1min":1.536,"5min":5.3179999999999996,"15min":5.3179999999999996},"last":0.72699999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.99099999999999999,"15min":1.046},"min":{"1min":0.59199999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.321,"5min":1.4970000000000001,"15min":15.372},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0629999999999999,"15min":1.05},"min":{"1min":0.79000000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"max":{"1min":1.391,"5min":5.2729999999999997,"15min":6.1580000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0329999999999999,"15min":1.042},"min":{"1min":0.749,"5min":0.60499999999999998,"15min":0.60199999999999998},"max":{"1min":1.2649999999999999,"5min":1.421,"15min":1.8080000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.95199999999999996,"5min":0.73899999999999999,"15min":0.67700000000000005},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.913},"last":0.93999999999999995}]},{"osd":7,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0860000000000001,"15min":1.0960000000000001},"min":{"1min":0.72099999999999997,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2290000000000001,"5min":1.577,"15min":2.75},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.212,"15min":1.1899999999999999},"min":{"1min":1.006,"5min":0.69599999999999995,"15min":0.42499999999999999},"max":{"1min":1.627,"5min":1.627,"15min":3.0590000000000002},"last":0.99299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575358,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672628,"kb_used_data":519276,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93699212,"statfs":{"total":96636764160,"available":95947993088,"internally_reserved":0,"allocated":531738624,"data_stored":518585685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":157022662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88900000000000001,"15min":0.87},"min":{"1min":0.51300000000000001,"5min":0.38400000000000001,"15min":0.32700000000000001},"max":{"1min":1.3300000000000001,"5min":1.4379999999999999,"15min":6.4729999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.88800000000000001,"15min":0.86899999999999999},"min":{"1min":0.56999999999999995,"5min":0.46800000000000003,"15min":0.39100000000000001},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":8.7560000000000002},"last":0.745}]},{"osd":1,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81499999999999995,"15min":0.88200000000000001},"min":{"1min":0.51400000000000001,"5min":0.439,"15min":0.42799999999999999},"max":{"1min":1.2,"5min":1.671,"15min":15.257999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.86099999999999999,"15min":0.94899999999999995},"min":{"1min":0.62,"5min":0.48699999999999999,"15min":0.47399999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":15.175000000000001},"last":0.82399999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.90100000000000002,"15min":0.91100000000000003},"min":{"1min":0.57499999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.228,"5min":1.458,"15min":6.5410000000000004},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.93700000000000006,"15min":0.98699999999999999},"min":{"1min":0.57599999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.52,"15min":6.7080000000000002},"last":1.042}]},{"osd":3,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.0880000000000001,"15min":1.0469999999999999},"min":{"1min":0.69399999999999995,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.3839999999999999,"5min":5.4249999999999998,"15min":6.7699999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.175,"15min":1.1659999999999999},"min":{"1min":0.88600000000000001,"5min":0.58699999999999997,"15min":0.40100000000000002},"max":{"1min":1.4790000000000001,"5min":5.5570000000000004,"15min":15.364000000000001},"last":1.1000000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:51:44 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.92500000000000004,"15min":0.97999999999999998},"min":{"1min":0.63600000000000001,"5min":0.376,"15min":0.36499999999999999},"max":{"1min":2.5830000000000002,"5min":4.3529999999999998,"15min":5.9720000000000004},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.27,"5min":1.246,"15min":1.1459999999999999},"min":{"1min":0.92900000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.5190000000000001,"5min":5.6609999999999996,"15min":14.314},"last":1.1970000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.0129999999999999,"15min":1.123},"min":{"1min":0.69299999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.4239999999999999,"5min":4.2290000000000001,"15min":15.06},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.097,"15min":1.1799999999999999},"min":{"1min":0.83699999999999997,"5min":0.65400000000000003,"15min":0.50900000000000001},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":7.6630000000000003},"last":1.0149999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.0609999999999999,"15min":1.1279999999999999},"min":{"1min":0.745,"5min":0.44,"15min":0.44},"max":{"1min":4.0599999999999996,"5min":5.0419999999999998,"15min":6.8259999999999996},"last":0.875},{"interface":"front","average":{"1min":1.429,"5min":1.2130000000000001,"15min":1.266},"min":{"1min":0.89700000000000002,"5min":0.53500000000000003,"15min":0.46200000000000002},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":19.321999999999999},"last":1.151}]}]},{"osd":4,"up_from":13,"seq":55834575359,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717128,"kb_used_data":569408,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93654712,"statfs":{"total":96636764160,"available":95902425088,"internally_reserved":0,"allocated":583073792,"data_stored":568869284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.51800000000000002,"5min":0.48599999999999999,"15min":0.28100000000000003},"max":{"1min":1.087,"5min":2.202,"15min":9.6690000000000005},"last":1.1839999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85199999999999998,"15min":0.84799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47399999999999998,"15min":0.377},"max":{"1min":1.2010000000000001,"5min":2.0760000000000001,"15min":2.786},"last":0.73299999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.873,"15min":0.88200000000000001},"min":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.41399999999999998},"max":{"1min":1.2170000000000001,"5min":1.8160000000000001,"15min":5.5119999999999996},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.89500000000000002,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.51500000000000001,"15min":0.44700000000000001},"max":{"1min":1.202,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":1.2110000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:51:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.89200000000000002,"15min":0.878},"min":{"1min":0.63200000000000001,"5min":0.53700000000000003,"15min":0.45100000000000001},"max":{"1min":1.212,"5min":2.1549999999999998,"15min":9.0099999999999998},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.95099999999999996,"15min":0.95199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0569999999999999,"5min":7.7359999999999998,"15min":7.7359999999999998},"last":0.97499999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.94499999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.52500000000000002},"max":{"1min":1.1899999999999999,"5min":1.6599999999999999,"15min":4.7080000000000002},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.026,"5min":1.0580000000000001,"15min":1.0209999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.48699999999999999},"max":{"1min":1.2649999999999999,"5min":1.7509999999999999,"15min":3.2749999999999999},"last":1.153}]},{"osd":5,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0249999999999999,"15min":1.0489999999999999},"min":{"1min":0.56299999999999994,"5min":0.52300000000000002,"15min":0.51500000000000001},"max":{"1min":1.3799999999999999,"5min":1.621,"15min":7.8529999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.214,"15min":1.161},"min":{"1min":0.86899999999999999,"5min":0.627,"15min":0.40999999999999998},"max":{"1min":1.538,"5min":7.694,"15min":8.2070000000000007},"last":1.266}]},{"osd":6,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1379999999999999,"15min":1.081},"min":{"1min":0.745,"5min":0.68999999999999995,"15min":0.52600000000000002},"max":{"1min":1.458,"5min":7.7880000000000003,"15min":7.7880000000000003},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.038,"5min":1.1659999999999999,"15min":1.155},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.2929999999999999,"5min":3.7320000000000002,"15min":5.6130000000000004},"last":1.081}]},{"osd":7,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.1950000000000001,"15min":1.1739999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.52000000000000002},"max":{"1min":1.6120000000000001,"5min":7.8380000000000001,"15min":10.715},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0720000000000001,"15min":1.157},"min":{"1min":0.627,"5min":0.627,"15min":0.47199999999999998},"max":{"1min":1.1579999999999999,"5min":2.2450000000000001,"15min":4.3230000000000004},"last":1.119}]}]},{"osd":3,"up_from":13,"seq":55834575357,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707492,"kb_used_data":550044,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93664348,"statfs":{"total":96636764160,"available":95912292352,"internally_reserved":0,"allocated":563245056,"data_stored":549099638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":161216968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.69799999999999995},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64900000000000002,"5min":0.76700000000000002,"15min":1.5429999999999999},"last":0.39500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.51000000000000001,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":0.755,"5min":0.76200000000000001,"15min":0.94099999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.28599999999999998,"15min":0.246},"max":{"1min":0.78900000000000003,"5min":0.92000000000000004,"15min":1.018},"last":0.373}]},{"osd":2,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.65100000000000002,"5min":1.056,"15min":1.4690000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.86499999999999999,"15min":1.6799999999999999},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.79600000000000004,"15min":0.83599999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.49099999999999999},"max":{"1min":0.96499999999999997,"5min":1.1739999999999999,"15min":1.5900000000000001},"last":0.747},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.84299999999999997,"15min":0.85299999999999998},"min":{"1min":0.66100000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0129999999999999,"5min":1.1419999999999999,"15min":1.3999999999999999},"last":0.76100000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.60799999999999998,"5min":0.57399999999999995,"15min":0.51800000000000002},"max":{"1min":1.095,"5min":1.1879999999999999,"15min":1.5069999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.83599999999999997,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.58599999999999997,"15min":0.5},"max":{"1min":1.232,"5min":1.232,"15min":1.365},"last":0.621}]},{"osd":6,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.88700000000000001,"15min":0.82799999999999996},"min":{"1min":0.66900000000000004,"5min":0.52800000000000002,"15min":0.45400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.88500000000000001,"15min":0.82699999999999996},"min":{"1min":0.69499999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":2.3410000000000002},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.88800000000000001,"15min":0.88200000000000001},"min":{"1min":0.68300000000000005,"5min":0.57799999999999996,"15min":0.51700000000000002},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":2.9449999999999998},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.872,"15min":0.89300000000000002},"min":{"1min":0.64300000000000002,"5min":0.54400000000000004,"15min":0.47099999999999997},"max":{"1min":1.343,"5min":1.343,"15min":2.9729999999999999},"last":0.77300000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575360,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484724,"kb_used_data":377260,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93887116,"statfs":{"total":96636764160,"available":96140406784,"internally_reserved":0,"allocated":386314240,"data_stored":377764906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.47999999999999998,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.37},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.50800000000000001,"15min":0.48199999999999998},"min":{"1min":0.41799999999999998,"5min":0.28299999999999997,"15min":0.25700000000000001},"max":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"last":0.50600000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.45900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.58399999999999996,"15min":0.54000000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.0060000000000002,"5min":7.0060000000000002,"15min":7.0060000000000002},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.36499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":1.637},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.622,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.58399999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.78000000000000003,"15min":0.80300000000000005},"min":{"1min":0.59699999999999998,"5min":0.442,"15min":0.442},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.80500000000000005,"15min":0.82699999999999996},"min":{"1min":0.61399999999999999,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":4.1479999999999997,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.88500000000000001,"15min":0.877},"min":{"1min":0.754,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5169999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.80900000000000005,"15min":0.82499999999999996},"min":{"1min":0.58599999999999997,"5min":0.45300000000000001,"15min":0.41999999999999998},"max":{"1min":1.171,"5min":2.762,"15min":2.762},"last":0.81200000000000006}]},{"osd":6,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77400000000000002,"15min":0.79000000000000004},"min":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.47699999999999998},"max":{"1min":0.98699999999999999,"5min":1.6419999999999999,"15min":3.1659999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.84399999999999997,"15min":0.81699999999999995},"min":{"1min":0.65500000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.8210000000000002,"5min":3.8210000000000002,"15min":3.8210000000000002},"last":0.68400000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":0.90800000000000003,"15min":0.85699999999999998},"min":{"1min":0.69599999999999995,"5min":0.55000000000000004,"15min":0.503},"max":{"1min":6.9059999999999997,"5min":6.9059999999999997,"15min":6.9059999999999997},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.0009999999999999,"15min":0.95499999999999996},"min":{"1min":0.81299999999999994,"5min":0.63100000000000001,"15min":0.55900000000000005},"max":{"1min":6.9509999999999996,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.88200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575357,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671940,"kb_used_data":532604,"kb_used_omap":1,"kb_used_meta":139326,"kb_avail":93699900,"statfs":{"total":96636764160,"available":95948697600,"internally_reserved":0,"allocated":545386496,"data_stored":536307441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142670278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30399999999999999,"5min":0.24299999999999999,"15min":0.20799999999999999},"max":{"1min":0.58799999999999997,"5min":0.94999999999999996,"15min":1.1359999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.28299999999999997,"15min":0.251},"max":{"1min":0.64900000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.42599999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49099999999999999,"15min":0.501},"min":{"1min":0.318,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66800000000000004,"5min":1.002,"15min":2.8660000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.501,"15min":0.48599999999999999},"min":{"1min":0.36899999999999999,"5min":0.27700000000000002,"15min":0.249},"max":{"1min":0.69099999999999995,"5min":1.0589999999999999,"15min":2.6739999999999999},"last":0.625}]},{"osd":3,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.53000000000000003},"min":{"1min":0.373,"5min":0.35099999999999998,"15min":0.314},"max":{"1min":0.73899999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.318},"max":{"1min":0.755,"5min":1.306,"15min":2.71},"last":0.66800000000000004}]},{"osd":4,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82399999999999995,"15min":0.85599999999999998},"min":{"1min":0.622,"5min":0.52400000000000002,"15min":0.40200000000000002},"max":{"1min":1.0489999999999999,"5min":1.2210000000000001,"15min":6.7789999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.92200000000000004},"min":{"1min":0.76500000000000001,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":1.093,"5min":1.2629999999999999,"15min":6.8259999999999996},"last":0.88700000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":0.76200000000000001},"min":{"1min":0.65800000000000003,"5min":0.53800000000000003,"15min":0.47199999999999998},"max":{"1min":0.96499999999999997,"5min":2.133,"15min":2.5750000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.627,"5min":0.53800000000000003,"15min":0.51000000000000001},"max":{"1min":1.0229999999999999,"5min":1.3859999999999999,"15min":2.6179999999999999},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Oct 30 13:51:46 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91800000000000004,"15min":0.89500000000000002},"min":{"1min":0.80000000000000004,"5min":0.61199999999999999,"15min":0.501},"max":{"1min":1.1459999999999999,"5min":1.3999999999999999,"15min":6.0289999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.96599999999999997,"15min":0.95499999999999996},"min":{"1min":0.65500000000000003,"5min":0.625,"15min":0.60299999999999998},"max":{"1min":1.218,"5min":1.4399999999999999,"15min":6.0739999999999998},"last":0.95699999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79500000000000004,"15min":0.78300000000000003},"min":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.47899999999999998},"max":{"1min":1.0369999999999999,"5min":1.161,"15min":2.1579999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.879,"15min":0.88500000000000001},"min":{"1min":0.70899999999999996,"5min":0.55200000000000005,"15min":0.54800000000000004},"max":{"1min":1.1339999999999999,"5min":1.3440000000000001,"15min":6.0519999999999996},"last":0.998}]}]},{"osd":0,"up_from":13,"seq":55834575359,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712496,"kb_used_data":568936,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93659344,"statfs":{"total":96636764160,"available":95907168256,"internally_reserved":0,"allocated":582590464,"data_stored":569154632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.48399999999999999},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.214},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":16.184000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.50800000000000001,"15min":0.59199999999999997},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.28899999999999998},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":16.228000000000002},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.44900000000000001,"15min":0.64100000000000001},"min":{"1min":0.33300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":2.7999999999999998,"5min":2.7999999999999998,"15min":17.454000000000001},"last":0.496},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.65300000000000002},"min":{"1min":0.318,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":1.391,"5min":1.391,"15min":23.212},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.37,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":16.103999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.53100000000000003,"15min":0.73099999999999998},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.32000000000000001},"max":{"1min":2.8340000000000001,"5min":2.8340000000000001,"15min":23.16},"last":0.57799999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:51:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.83299999999999996,"15min":0.92700000000000005},"min":{"1min":0.48499999999999999,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":7.6900000000000004,"5min":7.6900000000000004,"15min":16.309000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.85799999999999998,"15min":0.97499999999999998},"min":{"1min":0.61199999999999999,"5min":0.53800000000000003,"15min":0.46000000000000002},"max":{"1min":2.5939999999999999,"5min":3.4009999999999998,"15min":16.343},"last":0.97199999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85299999999999998,"15min":1.006},"min":{"1min":0.63,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.02,"5min":2.02,"15min":23.266999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.86299999999999999,"15min":1.0329999999999999},"min":{"1min":0.755,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.7240000000000002,"5min":2.7240000000000002,"15min":16.268000000000001},"last":0.90700000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.80500000000000005,"15min":1.1120000000000001},"min":{"1min":0.61099999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":29.774999999999999},"last":0.754},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81499999999999995,"15min":1.0229999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.50700000000000001},"max":{"1min":2.75,"5min":2.75,"15min":17.5},"last":0.92700000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.81999999999999995,"15min":1.202},"min":{"1min":0.66700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.694,"5min":2.694,"15min":47.534999999999997},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.85999999999999999,"15min":1.155},"min":{"1min":0.64100000000000001,"5min":0.60399999999999998,"15min":0.54700000000000004},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":32.476999999999997},"last":0.83199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230703104,"data_stored":230687380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255741952,"data_stored":255719205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179036160,"data_stored":179013521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230858752,"data_stored":230840931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":197275648,"data_stored":197263919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":205541376,"data_stored":205522290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266067968,"data_stored":266051919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":241283072,"data_stored":241264811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350515200,"data_stored":337464605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288862208,"data_stored":280174938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":206495744,"data_stored":198338087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331603968,"data_stored":317845409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":385015808,"data_stored":371192067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":325414912,"data_stored":312650097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193724416,"data_stored":185512357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342437888,"data_stored":329203526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:52:31.094 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:52:31.269 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f036d2c9640 1 Processor -- start 2023-10-30T13:52:31.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f036d2c9640 1 -- start start 2023-10-30T13:52:31.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f036d2c9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03680c6bd0 0x7f036806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:31.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f036d2c9640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0368071110 0x7f0368158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:31.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f036d2c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0368066cf0 con 0x7f036806f7a0 2023-10-30T13:52:31.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f036d2c9640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03680700c0 con 0x7f0368071110 2023-10-30T13:52:31.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f036d2c9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03680c70d0 con 0x7f03680c6bd0 2023-10-30T13:52:31.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f03677fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0368071110 0x7f0368158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:31.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f0366ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f036806f7a0 0x7f0368069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41684/0 (socket says 172.21.15.148:41684) 2023-10-30T13:52:31.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f0366ffd640 1 -- 172.21.15.148:0/3449430098 learned_addr learned my addr 172.21.15.148:0/3449430098 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:31.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f03677fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0368071110 0x7f0368158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56690/0 (socket says 172.21.15.148:56690) 2023-10-30T13:52:31.274 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f035e7fc640 1 --2- 172.21.15.148:0/3449430098 >> v2:172.21.15.150:3300/0 conn(0x7f03680c6bd0 0x7f036806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:31.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f03677fe640 1 -- 172.21.15.148:0/3449430098 >> v2:172.21.15.150:3300/0 conn(0x7f03680c6bd0 msgr2=0x7f036806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:31.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f03677fe640 1 --2- 172.21.15.148:0/3449430098 >> v2:172.21.15.150:3300/0 conn(0x7f03680c6bd0 0x7f036806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.275 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f03677fe640 1 -- 172.21.15.148:0/3449430098 >> v1:172.21.15.148:6789/0 conn(0x7f036806f7a0 legacy=0x7f0368069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:31.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f03677fe640 1 -- 172.21.15.148:0/3449430098 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680c7390 con 0x7f0368071110 2023-10-30T13:52:31.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.266+0000 7f03667fc640 1 -- 172.21.15.148:0/3449430098 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964139993 0 0) 0x7f0368066cf0 con 0x7f036806f7a0 2023-10-30T13:52:31.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f03677fe640 1 --2- 172.21.15.148:0/3449430098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0368071110 0x7f0368158110 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f036806f1c0 tx=0x7f036001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f0ad622c4f10af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:31.276 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f03667fc640 1 -- 172.21.15.148:0/3449430098 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03600b0440 con 0x7f0368071110 2023-10-30T13:52:31.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f03667fc640 1 -- 172.21.15.148:0/3449430098 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03600b05a0 con 0x7f0368071110 2023-10-30T13:52:31.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f03667fc640 1 -- 172.21.15.148:0/3449430098 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03600b0930 con 0x7f0368071110 2023-10-30T13:52:31.277 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f036d2c9640 1 -- 172.21.15.148:0/3449430098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0368071110 msgr2=0x7f0368158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:31.278 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f036d2c9640 1 --2- 172.21.15.148:0/3449430098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0368071110 0x7f0368158110 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f036806f1c0 tx=0x7f036001f0f0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f036d2c9640 1 -- 172.21.15.148:0/3449430098 shutdown_connections 2023-10-30T13:52:31.280 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f036d2c9640 1 --2- 172.21.15.148:0/3449430098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0368071110 0x7f0368158110 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f036d2c9640 1 --2- 172.21.15.148:0/3449430098 >> v2:172.21.15.150:3300/0 conn(0x7f03680c6bd0 0x7f036806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f036d2c9640 1 -- 172.21.15.148:0/3449430098 >> 172.21.15.148:0/3449430098 conn(0x7f03680c7910 msgr2=0x7f03680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:31.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f036d2c9640 1 -- 172.21.15.148:0/3449430098 shutdown_connections 2023-10-30T13:52:31.281 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.270+0000 7f036d2c9640 1 -- 172.21.15.148:0/3449430098 wait complete. 2023-10-30T13:52:31.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 Processor -- start 2023-10-30T13:52:31.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 -- start start 2023-10-30T13:52:31.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f036806f7a0 0x7f03680ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:31.282 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0368071110 0x7f0368152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:31.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03680c6bd0 0x7f036814ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:31.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03681545a0 con 0x7f03680c6bd0 2023-10-30T13:52:31.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0368152b50 con 0x7f0368071110 2023-10-30T13:52:31.283 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0368152520 con 0x7f036806f7a0 2023-10-30T13:52:31.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f03677fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03680c6bd0 0x7f036814ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:31.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f03677fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03680c6bd0 0x7f036814ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:56702/0 (socket says 172.21.15.148:56702) 2023-10-30T13:52:31.284 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f03677fe640 1 -- 172.21.15.148:0/1167570677 learned_addr learned my addr 172.21.15.148:0/1167570677 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:31.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f035e7fc640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0368071110 0x7f0368152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:31.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f0366ffd640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f036806f7a0 0x7f03680ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:31.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f03677fe640 1 -- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f036806f7a0 msgr2=0x7f03680ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:31.285 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f03677fe640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f036806f7a0 0x7f03680ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f03677fe640 1 -- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0368071110 msgr2=0x7f0368152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:31.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f03677fe640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0368071110 0x7f0368152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f03677fe640 1 -- 172.21.15.148:0/1167570677 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360009a70 con 0x7f03680c6bd0 2023-10-30T13:52:31.286 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f03677fe640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03680c6bd0 0x7f036814ef90 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f036001bd00 tx=0x7f036001be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:31.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f035ffff640 1 -- 172.21.15.148:0/1167570677 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03600b0440 con 0x7f03680c6bd0 2023-10-30T13:52:31.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f035ffff640 1 -- 172.21.15.148:0/1167570677 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03600bc860 con 0x7f03680c6bd0 2023-10-30T13:52:31.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03681e6270 con 0x7f03680c6bd0 2023-10-30T13:52:31.287 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f035ffff640 1 -- 172.21.15.148:0/1167570677 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03600bcc10 con 0x7f03680c6bd0 2023-10-30T13:52:31.288 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.274+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03681527e0 con 0x7f03680c6bd0 2023-10-30T13:52:31.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.278+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0368152cc0 con 0x7f03680c6bd0 2023-10-30T13:52:31.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.278+0000 7f035ffff640 1 -- 172.21.15.148:0/1167570677 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03600c7460 con 0x7f03680c6bd0 2023-10-30T13:52:31.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.278+0000 7f035ffff640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03480664a0 0x7f0348068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:31.289 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.278+0000 7f035ffff640 1 -- 172.21.15.148:0/1167570677 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f03601231e0 con 0x7f03680c6bd0 2023-10-30T13:52:31.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.278+0000 7f0366ffd640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03480664a0 0x7f0348068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:31.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.278+0000 7f0366ffd640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03480664a0 0x7f0348068960 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0350000f70 tx=0x7f0350007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:31.290 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.282+0000 7f035ffff640 1 -- 172.21.15.148:0/1167570677 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03600efe40 con 0x7f03680c6bd0 2023-10-30T13:52:31.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.430+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f036804e170 con 0x7f03680c6bd0 2023-10-30T13:52:31.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f035ffff640 1 -- 172.21.15.148:0/1167570677 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+19569 (secure 0 0 0) 0x7f03600b5090 con 0x7f03680c6bd0 2023-10-30T13:52:31.437 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:52:31.438 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":67,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:50:55.057709+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.148:6837/1530445724":"2023-10-31T13:50:55.057603+0000","172.21.15.148:6835/1228827025":"2023-10-31T13:49:20.023064+0000","172.21.15.150:6834/116502200":"2023-10-31T13:48:15.005684+0000","172.21.15.148:6833/1228827025":"2023-10-31T13:49:20.023064+0000","172.21.15.148:6837/3703944646":"2023-10-31T13:45:54.968896+0000","172.21.15.148:6835/3829165476":"2023-10-31T13:44:34.947959+0000","172.21.15.148:6833/3829165476":"2023-10-31T13:44:34.947959+0000","172.21.15.148:6836/500606168":"2023-10-31T13:42:29.894087+0000","172.21.15.148:6835/1079809213":"2023-10-31T13:39:54.856162+0000","172.21.15.148:6833/3194894584":"2023-10-31T13:24:29.547356+0000","172.21.15.148:6837/3232618434":"2023-10-31T13:33:54.741392+0000","172.21.15.148:6835/3194894584":"2023-10-31T13:24:29.547356+0000","172.21.15.150:6834/953984429":"2023-10-31T13:22:39.518196+0000","172.21.15.148:6835/3749586444":"2023-10-31T13:15:19.379857+0000","172.21.15.148:6836/3703944646":"2023-10-31T13:45:54.968896+0000","172.21.15.148:6834/4162935019":"2023-10-31T13:18:39.446214+0000","172.21.15.148:6832/4162935019":"2023-10-31T13:18:39.446214+0000","172.21.15.148:6837/500606168":"2023-10-31T13:42:29.894087+0000","172.21.15.148:6833/3749586444":"2023-10-31T13:15:19.379857+0000","172.21.15.150:6836/116502200":"2023-10-31T13:48:15.005684+0000","172.21.15.148:6837/3781833565":"2023-10-31T13:23:29.531899+0000","172.21.15.148:6833/1079809213":"2023-10-31T13:39:54.856162+0000","172.21.15.148:6836/1530445724":"2023-10-31T13:50:55.057603+0000","172.21.15.148:6837/1844602519":"2023-10-31T13:14:19.364086+0000","172.21.15.148:6836/1559416001":"2023-10-31T13:26:44.595428+0000","172.21.15.148:6836/3781833565":"2023-10-31T13:23:29.531899+0000","172.21.15.150:6834/1620877178":"2023-10-31T13:16:49.403499+0000","172.21.15.148:6837/2740075781":"2023-10-31T13:29:49.670604+0000","172.21.15.150:6836/953984429":"2023-10-31T13:22:39.518196+0000","172.21.15.148:6833/1386276781":"2023-10-31T13:35:44.780019+0000","172.21.15.150:6836/1620877178":"2023-10-31T13:16:49.403499+0000","172.21.15.148:6837/1559416001":"2023-10-31T13:26:44.595428+0000","172.21.15.148:6836/2740075781":"2023-10-31T13:29:49.670604+0000","172.21.15.148:6833/2733100982":"2023-10-31T13:28:14.633420+0000","172.21.15.148:6835/2733100982":"2023-10-31T13:28:14.633420+0000","172.21.15.148:6836/1844602519":"2023-10-31T13:14:19.364086+0000","172.21.15.148:6833/172087710":"2023-10-31T13:31:59.711325+0000","172.21.15.148:6835/172087710":"2023-10-31T13:31:59.711325+0000","172.21.15.148:6836/3232618434":"2023-10-31T13:33:54.741392+0000","172.21.15.148:6835/1386276781":"2023-10-31T13:35:44.780019+0000","172.21.15.148:6836/3144126860":"2023-10-31T13:37:34.820655+0000","172.21.15.148:6837/3144126860":"2023-10-31T13:37:34.820655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:52:31.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03480664a0 msgr2=0x7f0348068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:31.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03480664a0 0x7f0348068960 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0350000f70 tx=0x7f0350007df0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03680c6bd0 msgr2=0x7f036814ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:31.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03680c6bd0 0x7f036814ef90 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f036001bd00 tx=0x7f036001be90 comp rx=0 tx=0).stop 2023-10-30T13:52:31.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 shutdown_connections 2023-10-30T13:52:31.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f03480664a0 0x7f0348068960 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03680c6bd0 0x7f036814ef90 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0368071110 0x7f0368152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 --2- 172.21.15.148:0/1167570677 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f036806f7a0 0x7f03680ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:31.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 >> 172.21.15.148:0/1167570677 conn(0x7f03680c7910 msgr2=0x7f03680b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:31.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 shutdown_connections 2023-10-30T13:52:31.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:31.434+0000 7f036d2c9640 1 -- 172.21.15.148:0/1167570677 wait complete. 2023-10-30T13:52:32.460 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T13:52:32.460 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:52:32.613 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66bc73e640 1 Processor -- start 2023-10-30T13:52:32.614 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66bc73e640 1 -- start start 2023-10-30T13:52:32.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66bc73e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f66b4156ff0 0x7f66b40c0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66bc73e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66b40c0550 0x7f66b406bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66bc73e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b40c6540 con 0x7f66b406c7f0 2023-10-30T13:52:32.616 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66bc73e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b415aaf0 con 0x7f66b4156ff0 2023-10-30T13:52:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66bc73e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b40c69d0 con 0x7f66b40c0550 2023-10-30T13:52:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66ba4da640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f66b4156ff0 0x7f66b40c0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66ba4da640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f66b4156ff0 0x7f66b40c0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60856/0 (socket says 172.21.15.148:60856) 2023-10-30T13:52:32.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66ba4da640 1 -- 172.21.15.148:0/296986762 learned_addr learned my addr 172.21.15.148:0/296986762 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:32.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66b9cd9640 1 --2- 172.21.15.148:0/296986762 >> v2:172.21.15.150:3300/0 conn(0x7f66b40c0550 0x7f66b406bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66ba4da640 1 -- 172.21.15.148:0/296986762 >> v2:172.21.15.150:3300/0 conn(0x7f66b40c0550 msgr2=0x7f66b406bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:32.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66ba4da640 1 --2- 172.21.15.148:0/296986762 >> v2:172.21.15.150:3300/0 conn(0x7f66b40c0550 0x7f66b406bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66ba4da640 1 -- 172.21.15.148:0/296986762 >> v1:172.21.15.150:6789/0 conn(0x7f66b406c7f0 legacy=0x7f66b4159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:32.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.610+0000 7f66ba4da640 1 -- 172.21.15.148:0/296986762 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b415a810 con 0x7f66b4156ff0 2023-10-30T13:52:32.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66ba4da640 1 --2- 172.21.15.148:0/296986762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b4156ff0 0x7f66b40c0010 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009870 tx=0x7f66b001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65711860322bdc0c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:32.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66b94d8640 1 -- 172.21.15.148:0/296986762 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00a89b0 con 0x7f66b4156ff0 2023-10-30T13:52:32.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66b94d8640 1 -- 172.21.15.148:0/296986762 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b00a8b10 con 0x7f66b4156ff0 2023-10-30T13:52:32.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66b94d8640 1 -- 172.21.15.148:0/296986762 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00a8ea0 con 0x7f66b4156ff0 2023-10-30T13:52:32.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66bc73e640 1 -- 172.21.15.148:0/296986762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b4156ff0 msgr2=0x7f66b40c0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66bc73e640 1 --2- 172.21.15.148:0/296986762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b4156ff0 0x7f66b40c0010 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009870 tx=0x7f66b001ecb0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66bc73e640 1 -- 172.21.15.148:0/296986762 shutdown_connections 2023-10-30T13:52:32.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66bc73e640 1 --2- 172.21.15.148:0/296986762 >> v2:172.21.15.150:3300/0 conn(0x7f66b40c0550 0x7f66b406bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66bc73e640 1 --2- 172.21.15.148:0/296986762 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b4156ff0 0x7f66b40c0010 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66bc73e640 1 -- 172.21.15.148:0/296986762 >> 172.21.15.148:0/296986762 conn(0x7f66b40cb0f0 msgr2=0x7f66b40cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:32.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66bc73e640 1 -- 172.21.15.148:0/296986762 shutdown_connections 2023-10-30T13:52:32.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.614+0000 7f66bc73e640 1 -- 172.21.15.148:0/296986762 wait complete. 2023-10-30T13:52:32.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 Processor -- start 2023-10-30T13:52:32.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 -- start start 2023-10-30T13:52:32.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66b406c7f0 0x7f66b40707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b40c0550 0x7f66b40c2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b4156ff0 0x7f66b40715c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4070320 con 0x7f66b40c0550 2023-10-30T13:52:32.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b40afe10 con 0x7f66b4156ff0 2023-10-30T13:52:32.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4070cf0 con 0x7f66b406c7f0 2023-10-30T13:52:32.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66b9cd9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b40c0550 0x7f66b40c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66b9cd9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b40c0550 0x7f66b40c2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60872/0 (socket says 172.21.15.148:60872) 2023-10-30T13:52:32.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66b9cd9640 1 -- 172.21.15.148:0/1025335418 learned_addr learned my addr 172.21.15.148:0/1025335418 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:32.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bacdb640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b4156ff0 0x7f66b40715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66ba4da640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66b406c7f0 0x7f66b40707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66b9cd9640 1 -- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66b406c7f0 msgr2=0x7f66b40707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66b9cd9640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66b406c7f0 0x7f66b40707b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66b9cd9640 1 -- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b4156ff0 msgr2=0x7f66b40715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66b9cd9640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b4156ff0 0x7f66b40715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66b9cd9640 1 -- 172.21.15.148:0/1025335418 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b00094d0 con 0x7f66b40c0550 2023-10-30T13:52:32.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66ba4da640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f66b406c7f0 0x7f66b40707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:32.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66b9cd9640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b40c0550 0x7f66b40c2520 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f66a400d800 tx=0x7f66a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:32.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66aaffd640 1 -- 172.21.15.148:0/1025335418 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a400b6c0 con 0x7f66b40c0550 2023-10-30T13:52:32.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66aaffd640 1 -- 172.21.15.148:0/1025335418 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a400b840 con 0x7f66b40c0550 2023-10-30T13:52:32.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66aaffd640 1 -- 172.21.15.148:0/1025335418 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a400bbf0 con 0x7f66b40c0550 2023-10-30T13:52:32.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b4151ea0 con 0x7f66b40c0550 2023-10-30T13:52:32.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b406fcd0 con 0x7f66b40c0550 2023-10-30T13:52:32.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.618+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f66b40c5ef0 con 0x7f66b40c0550 2023-10-30T13:52:32.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66aaffd640 1 -- 172.21.15.148:0/1025335418 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f66a400bd50 con 0x7f66b40c0550 2023-10-30T13:52:32.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66aaffd640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66980664a0 0x7f6698068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66aaffd640 1 -- 172.21.15.148:0/1025335418 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f66a4084c60 con 0x7f66b40c0550 2023-10-30T13:52:32.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66aaffd640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f669806f880 0x7f6698071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66aaffd640 1 -- 172.21.15.148:0/1025335418 --> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66980723f0 con 0x7f669806f880 2023-10-30T13:52:32.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66aaffd640 1 -- 172.21.15.148:0/1025335418 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66a40892a0 con 0x7f66b40c0550 2023-10-30T13:52:32.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66ba4da640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66980664a0 0x7f6698068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66bacdb640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f669806f880 0x7f6698071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66bacdb640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f669806f880 0x7f6698071d20 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:32.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66ba4da640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66980664a0 0x7f6698068960 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f66b00b2000 tx=0x7f66b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:32.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.622+0000 7f66aaffd640 1 -- 172.21.15.148:0/1025335418 <== osd.0 v2:172.21.15.148:6800/1454599971 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f66980723f0 con 0x7f669806f880 2023-10-30T13:52:32.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.634+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 --> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f66b40c5c10 con 0x7f669806f880 2023-10-30T13:52:32.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66aaffd640 1 -- 172.21.15.148:0/1025335418 <== osd.0 v2:172.21.15.148:6800/1454599971 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f66ac005730 con 0x7f669806f880 2023-10-30T13:52:32.641 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:52:32.641 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:52:32.642 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:52:32.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f669806f880 msgr2=0x7f6698071d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.148:6800/1454599971,v1:172.21.15.148:6801/1454599971] conn(0x7f669806f880 0x7f6698071d20 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.643 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66980664a0 msgr2=0x7f6698068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f66980664a0 0x7f6698068960 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f66b00b2000 tx=0x7f66b0008040 comp rx=0 tx=0).stop 2023-10-30T13:52:32.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b40c0550 msgr2=0x7f66b40c2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 --2- 172.21.15.148:0/1025335418 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66b40c0550 0x7f66b40c2520 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f66a400d800 tx=0x7f66a400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.644 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bacdb640 1 -- 172.21.15.148:0/1025335418 reap_dead start 2023-10-30T13:52:32.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 shutdown_connections 2023-10-30T13:52:32.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 >> 172.21.15.148:0/1025335418 conn(0x7f66b40cb0f0 msgr2=0x7f66b4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:32.645 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 shutdown_connections 2023-10-30T13:52:32.646 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.638+0000 7f66bc73e640 1 -- 172.21.15.148:0/1025335418 wait complete. 2023-10-30T13:52:32.659 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T13:52:32.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb94d6640 1 Processor -- start 2023-10-30T13:52:32.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb94d6640 1 -- start start 2023-10-30T13:52:32.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb94d6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6eb4153c60 0x7f6eb40ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb94d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb4159830 0x7f6eb4159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb94d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb40cb780 con 0x7f6eb40c4430 2023-10-30T13:52:32.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb94d6640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb4154040 con 0x7f6eb4153c60 2023-10-30T13:52:32.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb94d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb415a2e0 con 0x7f6eb4159830 2023-10-30T13:52:32.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb2ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6eb4153c60 0x7f6eb40ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb2ffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f6eb4153c60 0x7f6eb40ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60876/0 (socket says 172.21.15.148:60876) 2023-10-30T13:52:32.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb2ffd640 1 -- 172.21.15.148:0/2538771898 learned_addr learned my addr 172.21.15.148:0/2538771898 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:32.802 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.794+0000 7f6eb37fe640 1 --2- 172.21.15.148:0/2538771898 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb4159830 0x7f6eb4159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb2ffd640 1 -- 172.21.15.148:0/2538771898 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb4159830 msgr2=0x7f6eb4159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:32.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb2ffd640 1 --2- 172.21.15.148:0/2538771898 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb4159830 0x7f6eb4159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb2ffd640 1 -- 172.21.15.148:0/2538771898 >> v1:172.21.15.150:6789/0 conn(0x7f6eb40c4430 legacy=0x7f6eb40cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb2ffd640 1 -- 172.21.15.148:0/2538771898 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb415a5c0 con 0x7f6eb4153c60 2023-10-30T13:52:32.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb2ffd640 1 --2- 172.21.15.148:0/2538771898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 0x7f6eb40ca360 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009e50 tx=0x7f6ea401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beafec2024b1b3af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:32.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb27fc640 1 -- 172.21.15.148:0/2538771898 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40a57d0 con 0x7f6eb4153c60 2023-10-30T13:52:32.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb27fc640 1 -- 172.21.15.148:0/2538771898 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea40a5930 con 0x7f6eb4153c60 2023-10-30T13:52:32.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb27fc640 1 -- 172.21.15.148:0/2538771898 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40a5cc0 con 0x7f6eb4153c60 2023-10-30T13:52:32.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb94d6640 1 -- 172.21.15.148:0/2538771898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 msgr2=0x7f6eb40ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb94d6640 1 --2- 172.21.15.148:0/2538771898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 0x7f6eb40ca360 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009e50 tx=0x7f6ea401ca10 comp rx=0 tx=0).stop 2023-10-30T13:52:32.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb94d6640 1 -- 172.21.15.148:0/2538771898 shutdown_connections 2023-10-30T13:52:32.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb94d6640 1 --2- 172.21.15.148:0/2538771898 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb4159830 0x7f6eb4159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb94d6640 1 --2- 172.21.15.148:0/2538771898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 0x7f6eb40ca360 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb94d6640 1 -- 172.21.15.148:0/2538771898 >> 172.21.15.148:0/2538771898 conn(0x7f6eb406bfe0 msgr2=0x7f6eb406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:32.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.798+0000 7f6eb94d6640 1 -- 172.21.15.148:0/2538771898 shutdown_connections 2023-10-30T13:52:32.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb94d6640 1 -- 172.21.15.148:0/2538771898 wait complete. 2023-10-30T13:52:32.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb94d6640 1 Processor -- start 2023-10-30T13:52:32.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb94d6640 1 -- start start 2023-10-30T13:52:32.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb94d6640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb40c4430 0x7f6eb40cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb94d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 0x7f6eb40b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb94d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6eb4159830 0x7f6eb40b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb94d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40cd710 con 0x7f6eb4153c60 2023-10-30T13:52:32.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb94d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb41432f0 con 0x7f6eb4159830 2023-10-30T13:52:32.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb94d6640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb414d7d0 con 0x7f6eb40c4430 2023-10-30T13:52:32.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eaa7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 0x7f6eb40b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eaa7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 0x7f6eb40b42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60886/0 (socket says 172.21.15.148:60886) 2023-10-30T13:52:32.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eaa7fc640 1 -- 172.21.15.148:0/4033620433 learned_addr learned my addr 172.21.15.148:0/4033620433 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:32.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb2ffd640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb40c4430 0x7f6eb40cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb37fe640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6eb4159830 0x7f6eb40b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eaa7fc640 1 -- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb40c4430 msgr2=0x7f6eb40cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eaa7fc640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb40c4430 0x7f6eb40cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eaa7fc640 1 -- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6eb4159830 msgr2=0x7f6eb40b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eaa7fc640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6eb4159830 0x7f6eb40b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eaa7fc640 1 -- 172.21.15.148:0/4033620433 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea4009b30 con 0x7f6eb4153c60 2023-10-30T13:52:32.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb2ffd640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb40c4430 0x7f6eb40cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:32.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eb37fe640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6eb4159830 0x7f6eb40b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:32.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eaa7fc640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 0x7f6eb40b42b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00c3c0 tx=0x7f6e9c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:32.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.802+0000 7f6eabfff640 1 -- 172.21.15.148:0/4033620433 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0052d0 con 0x7f6eb4153c60 2023-10-30T13:52:32.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.806+0000 7f6eabfff640 1 -- 172.21.15.148:0/4033620433 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c005430 con 0x7f6eb4153c60 2023-10-30T13:52:32.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.806+0000 7f6eabfff640 1 -- 172.21.15.148:0/4033620433 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c003820 con 0x7f6eb4153c60 2023-10-30T13:52:32.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.806+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb40b47f0 con 0x7f6eb4153c60 2023-10-30T13:52:32.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.806+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb40cd430 con 0x7f6eb4153c60 2023-10-30T13:52:32.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.806+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb404e170 con 0x7f6eb4153c60 2023-10-30T13:52:32.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.806+0000 7f6eabfff640 1 -- 172.21.15.148:0/4033620433 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e9c003980 con 0x7f6eb4153c60 2023-10-30T13:52:32.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.806+0000 7f6eabfff640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e940664d0 0x7f6e94068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:32.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.806+0000 7f6eabfff640 1 -- 172.21.15.148:0/4033620433 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f6e9c083cd0 con 0x7f6eb4153c60 2023-10-30T13:52:32.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.806+0000 7f6eb2ffd640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e940664d0 0x7f6e94068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:32.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.810+0000 7f6eb2ffd640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e940664d0 0x7f6e94068990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40ae000 tx=0x7f6ea40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:32.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.810+0000 7f6eabfff640 1 -- 172.21.15.148:0/4033620433 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e9c050800 con 0x7f6eb4153c60 2023-10-30T13:52:32.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.958+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f6eb40cbf90 con 0x7f6e940664d0 2023-10-30T13:52:32.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eabfff640 1 -- 172.21.15.148:0/4033620433 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6ea40bc090 con 0x7f6e940664d0 2023-10-30T13:52:32.966 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T13:52:32.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e940664d0 msgr2=0x7f6e94068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e940664d0 0x7f6e94068990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40ae000 tx=0x7f6ea40a7070 comp rx=0 tx=0).stop 2023-10-30T13:52:32.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 msgr2=0x7f6eb40b42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:32.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 0x7f6eb40b42b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00c3c0 tx=0x7f6e9c00c880 comp rx=0 tx=0).stop 2023-10-30T13:52:32.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 shutdown_connections 2023-10-30T13:52:32.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f6e940664d0 0x7f6e94068990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6eb4153c60 0x7f6eb40b42b0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6eb4159830 0x7f6eb40b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 --2- 172.21.15.148:0/4033620433 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f6eb40c4430 0x7f6eb40cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:32.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 >> 172.21.15.148:0/4033620433 conn(0x7f6eb406bfe0 msgr2=0x7f6eb4152f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:32.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 shutdown_connections 2023-10-30T13:52:32.970 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:32.962+0000 7f6eb94d6640 1 -- 172.21.15.148:0/4033620433 wait complete. 2023-10-30T13:52:32.987 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T13:52:32.987 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:52:33.130 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f61433d3640 1 Processor -- start 2023-10-30T13:52:33.131 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f61433d3640 1 -- start start 2023-10-30T13:52:33.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f61433d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f613c153930 0x7f613c0c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f61433d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f613c153d10 0x7f613c15a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.133 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f61433d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c15a7b0 con 0x7f613c069e90 2023-10-30T13:52:33.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f61433d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f613c0c5ec0 con 0x7f613c153d10 2023-10-30T13:52:33.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f61433d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f613c0c61a0 con 0x7f613c153930 2023-10-30T13:52:33.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f614096e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f613c153d10 0x7f613c15a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.134 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f614096e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f613c153d10 0x7f613c15a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60898/0 (socket says 172.21.15.148:60898) 2023-10-30T13:52:33.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.126+0000 7f614096e640 1 -- 172.21.15.148:0/2908435460 learned_addr learned my addr 172.21.15.148:0/2908435460 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:33.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.130+0000 7f614116f640 1 --2- 172.21.15.148:0/2908435460 >> v2:172.21.15.150:3300/0 conn(0x7f613c153930 0x7f613c0c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.130+0000 7f614096e640 1 -- 172.21.15.148:0/2908435460 >> v2:172.21.15.150:3300/0 conn(0x7f613c153930 msgr2=0x7f613c0c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:33.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.130+0000 7f614096e640 1 --2- 172.21.15.148:0/2908435460 >> v2:172.21.15.150:3300/0 conn(0x7f613c153930 0x7f613c0c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.135 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.130+0000 7f614096e640 1 -- 172.21.15.148:0/2908435460 >> v1:172.21.15.150:6789/0 conn(0x7f613c069e90 legacy=0x7f613c070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.136 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.130+0000 7f614096e640 1 -- 172.21.15.148:0/2908435460 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c0c6480 con 0x7f613c153d10 2023-10-30T13:52:33.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.130+0000 7f614096e640 1 --2- 172.21.15.148:0/2908435460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c153d10 0x7f613c15a270 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f613c0cbd70 tx=0x7f612c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b0c2342e79ba06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.130+0000 7f6133fff640 1 -- 172.21.15.148:0/2908435460 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c0a89f0 con 0x7f613c153d10 2023-10-30T13:52:33.137 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.130+0000 7f6133fff640 1 -- 172.21.15.148:0/2908435460 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612c0a8b50 con 0x7f613c153d10 2023-10-30T13:52:33.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.130+0000 7f6133fff640 1 -- 172.21.15.148:0/2908435460 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c0b0440 con 0x7f613c153d10 2023-10-30T13:52:33.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 -- 172.21.15.148:0/2908435460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c153d10 msgr2=0x7f613c15a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 --2- 172.21.15.148:0/2908435460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c153d10 0x7f613c15a270 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f613c0cbd70 tx=0x7f612c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.138 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 -- 172.21.15.148:0/2908435460 shutdown_connections 2023-10-30T13:52:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 --2- 172.21.15.148:0/2908435460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c153d10 0x7f613c15a270 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 --2- 172.21.15.148:0/2908435460 >> v2:172.21.15.150:3300/0 conn(0x7f613c153930 0x7f613c0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 -- 172.21.15.148:0/2908435460 >> 172.21.15.148:0/2908435460 conn(0x7f613c0c7910 msgr2=0x7f613c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 -- 172.21.15.148:0/2908435460 shutdown_connections 2023-10-30T13:52:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 -- 172.21.15.148:0/2908435460 wait complete. 2023-10-30T13:52:33.141 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 Processor -- start 2023-10-30T13:52:33.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 -- start start 2023-10-30T13:52:33.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c069e90 0x7f613c0b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c153930 0x7f613c06bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.142 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f613c153d10 0x7f613c0c24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c0c4af0 con 0x7f613c069e90 2023-10-30T13:52:33.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c0c5690 con 0x7f613c153930 2023-10-30T13:52:33.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f61433d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613c0c4d60 con 0x7f613c153d10 2023-10-30T13:52:33.143 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f614116f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c069e90 0x7f613c0b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f614116f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c069e90 0x7f613c0b0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60902/0 (socket says 172.21.15.148:60902) 2023-10-30T13:52:33.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f614116f640 1 -- 172.21.15.148:0/3226980393 learned_addr learned my addr 172.21.15.148:0/3226980393 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:33.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f614096e640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c153930 0x7f613c06bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f6141970640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f613c153d10 0x7f613c0c24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.144 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f614116f640 1 -- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f613c153d10 msgr2=0x7f613c0c24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f614116f640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f613c153d10 0x7f613c0c24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f614116f640 1 -- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c153930 msgr2=0x7f613c06bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f614116f640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c153930 0x7f613c06bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f614116f640 1 -- 172.21.15.148:0/3226980393 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c0094d0 con 0x7f613c069e90 2023-10-30T13:52:33.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.134+0000 7f6141970640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f613c153d10 0x7f613c0c24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:33.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f614116f640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c069e90 0x7f613c0b0920 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f613800d830 tx=0x7f613800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6131ffb640 1 -- 172.21.15.148:0/3226980393 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613800b6c0 con 0x7f613c069e90 2023-10-30T13:52:33.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6131ffb640 1 -- 172.21.15.148:0/3226980393 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613800b820 con 0x7f613c069e90 2023-10-30T13:52:33.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6131ffb640 1 -- 172.21.15.148:0/3226980393 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613800bb90 con 0x7f613c069e90 2023-10-30T13:52:33.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c0c5040 con 0x7f613c069e90 2023-10-30T13:52:33.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c0b04b0 con 0x7f613c069e90 2023-10-30T13:52:33.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f613c0c54f0 con 0x7f613c069e90 2023-10-30T13:52:33.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6131ffb640 1 -- 172.21.15.148:0/3226980393 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f613801e460 con 0x7f613c069e90 2023-10-30T13:52:33.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6131ffb640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61200664a0 0x7f6120068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6131ffb640 1 -- 172.21.15.148:0/3226980393 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f6138084d70 con 0x7f613c069e90 2023-10-30T13:52:33.149 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6131ffb640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7f612006f880 0x7f6120071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6131ffb640 1 -- 172.21.15.148:0/3226980393 --> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61200723f0 con 0x7f612006f880 2023-10-30T13:52:33.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6131ffb640 1 -- 172.21.15.148:0/3226980393 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6138087350 con 0x7f613c069e90 2023-10-30T13:52:33.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f614096e640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61200664a0 0x7f6120068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.150 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.138+0000 7f6141970640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7f612006f880 0x7f6120071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.142+0000 7f6141970640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7f612006f880 0x7f6120071d20 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.142+0000 7f614096e640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61200664a0 0x7f6120068960 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f612c0b2000 tx=0x7f612c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.151 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.142+0000 7f6131ffb640 1 -- 172.21.15.148:0/3226980393 <== osd.1 v2:172.21.15.148:6815/1053568651 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f61200723f0 con 0x7f612006f880 2023-10-30T13:52:33.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 --> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f613c0c6a40 con 0x7f612006f880 2023-10-30T13:52:33.158 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f6131ffb640 1 -- 172.21.15.148:0/3226980393 <== osd.1 v2:172.21.15.148:6815/1053568651 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6134001f40 con 0x7f612006f880 2023-10-30T13:52:33.159 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:52:33.159 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:52:33.159 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:52:33.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7f612006f880 msgr2=0x7f6120071d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.160 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.148:6815/1053568651,v1:172.21.15.148:6817/1053568651] conn(0x7f612006f880 0x7f6120071d20 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61200664a0 msgr2=0x7f6120068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f61200664a0 0x7f6120068960 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f612c0b2000 tx=0x7f612c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:52:33.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c069e90 msgr2=0x7f613c0b0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.161 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 --2- 172.21.15.148:0/3226980393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f613c069e90 0x7f613c0b0920 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f613800d830 tx=0x7f613800dcf0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f6141970640 1 -- 172.21.15.148:0/3226980393 reap_dead start 2023-10-30T13:52:33.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 shutdown_connections 2023-10-30T13:52:33.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 >> 172.21.15.148:0/3226980393 conn(0x7f613c0c7910 msgr2=0x7f613c157830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:33.162 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 shutdown_connections 2023-10-30T13:52:33.163 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.154+0000 7f61433d3640 1 -- 172.21.15.148:0/3226980393 wait complete. 2023-10-30T13:52:33.177 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T13:52:33.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a7f72640 1 Processor -- start 2023-10-30T13:52:33.317 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a7f72640 1 -- start start 2023-10-30T13:52:33.318 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a7f72640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f72a00c6b40 0x7f72a0161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a7f72640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a00c0ed0 0x7f72a006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a7f72640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a00c72b0 con 0x7f72a0158090 2023-10-30T13:52:33.319 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a7f72640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a0162120 con 0x7f72a00c6b40 2023-10-30T13:52:33.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a7f72640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00c6fd0 con 0x7f72a00c0ed0 2023-10-30T13:52:33.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a550d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f72a00c6b40 0x7f72a0161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a550d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f72a00c6b40 0x7f72a0161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60908/0 (socket says 172.21.15.148:60908) 2023-10-30T13:52:33.320 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a550d640 1 -- 172.21.15.148:0/2337360901 learned_addr learned my addr 172.21.15.148:0/2337360901 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:33.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a650f640 1 --2- 172.21.15.148:0/2337360901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a00c0ed0 0x7f72a006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.321 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a550d640 1 -- 172.21.15.148:0/2337360901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a00c0ed0 msgr2=0x7f72a006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:33.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a550d640 1 --2- 172.21.15.148:0/2337360901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a00c0ed0 0x7f72a006a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a550d640 1 -- 172.21.15.148:0/2337360901 >> v1:172.21.15.150:6789/0 conn(0x7f72a0158090 legacy=0x7f72a0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a550d640 1 -- 172.21.15.148:0/2337360901 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a015dd30 con 0x7f72a00c6b40 2023-10-30T13:52:33.322 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.314+0000 7f72a650f640 1 --2- 172.21.15.148:0/2337360901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a00c0ed0 0x7f72a006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:33.323 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a550d640 1 --2- 172.21.15.148:0/2337360901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c6b40 0x7f72a0161be0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f7290009870 tx=0x7f729001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9adeed1357c3ec8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a4d0c640 1 -- 172.21.15.148:0/2337360901 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72900a8a30 con 0x7f72a00c6b40 2023-10-30T13:52:33.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a4d0c640 1 -- 172.21.15.148:0/2337360901 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72900a8b90 con 0x7f72a00c6b40 2023-10-30T13:52:33.324 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a4d0c640 1 -- 172.21.15.148:0/2337360901 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72900b0440 con 0x7f72a00c6b40 2023-10-30T13:52:33.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a7f72640 1 -- 172.21.15.148:0/2337360901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c6b40 msgr2=0x7f72a0161be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a7f72640 1 --2- 172.21.15.148:0/2337360901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c6b40 0x7f72a0161be0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f7290009870 tx=0x7f729001ed30 comp rx=0 tx=0).stop 2023-10-30T13:52:33.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a7f72640 1 -- 172.21.15.148:0/2337360901 shutdown_connections 2023-10-30T13:52:33.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a7f72640 1 --2- 172.21.15.148:0/2337360901 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a00c0ed0 0x7f72a006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.325 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a7f72640 1 --2- 172.21.15.148:0/2337360901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c6b40 0x7f72a0161be0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a7f72640 1 -- 172.21.15.148:0/2337360901 >> 172.21.15.148:0/2337360901 conn(0x7f72a00c7910 msgr2=0x7f72a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:33.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a7f72640 1 -- 172.21.15.148:0/2337360901 shutdown_connections 2023-10-30T13:52:33.326 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.318+0000 7f72a7f72640 1 -- 172.21.15.148:0/2337360901 wait complete. 2023-10-30T13:52:33.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 Processor -- start 2023-10-30T13:52:33.328 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 -- start start 2023-10-30T13:52:33.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c0ed0 0x7f72a0153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a0158090 0x7f72a01eaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72a014b1d0 0x7f72a0149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.329 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a01f69a0 con 0x7f72a00c0ed0 2023-10-30T13:52:33.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a014cc10 con 0x7f72a014b1d0 2023-10-30T13:52:33.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a014b9e0 con 0x7f72a0158090 2023-10-30T13:52:33.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a5d0e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c0ed0 0x7f72a0153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.330 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a5d0e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c0ed0 0x7f72a0153f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60920/0 (socket says 172.21.15.148:60920) 2023-10-30T13:52:33.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a5d0e640 1 -- 172.21.15.148:0/852424111 learned_addr learned my addr 172.21.15.148:0/852424111 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:33.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a650f640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72a014b1d0 0x7f72a0149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a550d640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a0158090 0x7f72a01eaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a5d0e640 1 -- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a0158090 msgr2=0x7f72a01eaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.331 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a5d0e640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a0158090 0x7f72a01eaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a5d0e640 1 -- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72a014b1d0 msgr2=0x7f72a0149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a5d0e640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72a014b1d0 0x7f72a0149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a5d0e640 1 -- 172.21.15.148:0/852424111 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72900094d0 con 0x7f72a00c0ed0 2023-10-30T13:52:33.332 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a5d0e640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c0ed0 0x7f72a0153f40 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c3c0 tx=0x7f729c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72967fc640 1 -- 172.21.15.148:0/852424111 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c0052d0 con 0x7f72a00c0ed0 2023-10-30T13:52:33.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72967fc640 1 -- 172.21.15.148:0/852424111 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729c005430 con 0x7f72a00c0ed0 2023-10-30T13:52:33.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72967fc640 1 -- 172.21.15.148:0/852424111 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c003840 con 0x7f72a00c0ed0 2023-10-30T13:52:33.333 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a014bcc0 con 0x7f72a00c0ed0 2023-10-30T13:52:33.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a014c280 con 0x7f72a00c0ed0 2023-10-30T13:52:33.334 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.322+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a01f6ae0 con 0x7f72a00c0ed0 2023-10-30T13:52:33.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.326+0000 7f72967fc640 1 -- 172.21.15.148:0/852424111 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f729c0039a0 con 0x7f72a00c0ed0 2023-10-30T13:52:33.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.326+0000 7f72967fc640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72840664a0 0x7f7284068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.335 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.326+0000 7f72967fc640 1 -- 172.21.15.148:0/852424111 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f729c083c10 con 0x7f72a00c0ed0 2023-10-30T13:52:33.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.326+0000 7f72a550d640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72840664a0 0x7f7284068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.326+0000 7f72a550d640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72840664a0 0x7f7284068960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f72900b2000 tx=0x7f7290008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.336 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.330+0000 7f72967fc640 1 -- 172.21.15.148:0/852424111 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f729c050850 con 0x7f72a00c0ed0 2023-10-30T13:52:33.484 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f72a015e2f0 con 0x7f72840664a0 2023-10-30T13:52:33.485 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72967fc640 1 -- 172.21.15.148:0/852424111 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f72900b0440 con 0x7f72840664a0 2023-10-30T13:52:33.486 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T13:52:33.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72840664a0 msgr2=0x7f7284068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72840664a0 0x7f7284068960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f72900b2000 tx=0x7f7290008040 comp rx=0 tx=0).stop 2023-10-30T13:52:33.488 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c0ed0 msgr2=0x7f72a0153f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c0ed0 0x7f72a0153f40 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c3c0 tx=0x7f729c00c880 comp rx=0 tx=0).stop 2023-10-30T13:52:33.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 shutdown_connections 2023-10-30T13:52:33.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72840664a0 0x7f7284068960 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72a014b1d0 0x7f72a0149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.489 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f72a0158090 0x7f72a01eaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 --2- 172.21.15.148:0/852424111 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72a00c0ed0 0x7f72a0153f40 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 >> 172.21.15.148:0/852424111 conn(0x7f72a00c7910 msgr2=0x7f72a01560d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:33.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 shutdown_connections 2023-10-30T13:52:33.490 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.482+0000 7f72a7f72640 1 -- 172.21.15.148:0/852424111 wait complete. 2023-10-30T13:52:33.507 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T13:52:33.508 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:52:33.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.642+0000 7fd2ab901640 1 Processor -- start 2023-10-30T13:52:33.648 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.642+0000 7fd2ab901640 1 -- start start 2023-10-30T13:52:33.649 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2ab901640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2a40c6f10 0x7fd2a415a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2ab901640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a40c73e0 con 0x7fd2a415d5e0 2023-10-30T13:52:33.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2ab901640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a40c76c0 con 0x7fd2a415d9c0 2023-10-30T13:52:33.650 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2ab901640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a40c79a0 con 0x7fd2a40c6f10 2023-10-30T13:52:33.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a969d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd2a415d5e0 0x7fd2a40be700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45824/0 (socket says 172.21.15.148:45824) 2023-10-30T13:52:33.651 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a969d640 1 -- 172.21.15.148:0/3175684178 learned_addr learned my addr 172.21.15.148:0/3175684178 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:33.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a8e9c640 1 --2- 172.21.15.148:0/3175684178 >> v2:172.21.15.150:3300/0 conn(0x7fd2a40c6f10 0x7fd2a415a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107890114 0 0) 0x7fd2a40c73e0 con 0x7fd2a415d5e0 2023-10-30T13:52:33.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd290003530 con 0x7fd2a415d5e0 2023-10-30T13:52:33.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321984772 0 0) 0x7fd290003530 con 0x7fd2a415d5e0 2023-10-30T13:52:33.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 >> v2:172.21.15.150:3300/0 conn(0x7fd2a40c6f10 msgr2=0x7fd2a415a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:33.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 --2- 172.21.15.148:0/3175684178 >> v2:172.21.15.150:3300/0 conn(0x7fd2a40c6f10 0x7fd2a415a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 >> v1:172.21.15.150:6789/0 conn(0x7fd2a415d9c0 legacy=0x7fd2a4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a40c7c80 con 0x7fd2a415d5e0 2023-10-30T13:52:33.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fd2a0001680 con 0x7fd2a415d5e0 2023-10-30T13:52:33.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 >> v1:172.21.15.148:6789/0 conn(0x7fd2a415d5e0 legacy=0x7fd2a40be700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd290003980 0x7fd290003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2900047d0 0x7fd290004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd290005190 0x7fd29000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40c79a0 con 0x7fd290005190 2023-10-30T13:52:33.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29000c040 con 0x7fd2900047d0 2023-10-30T13:52:33.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29000c200 con 0x7fd290003980 2023-10-30T13:52:33.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a9e9e640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd290005190 0x7fd29000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a969d640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd290003980 0x7fd290003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a8e9c640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2900047d0 0x7fd290004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a9e9e640 1 -- 172.21.15.148:0/3175684178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd290003980 msgr2=0x7fd290003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:33.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a9e9e640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd290003980 0x7fd290003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a9e9e640 1 -- 172.21.15.148:0/3175684178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2900047d0 msgr2=0x7fd290004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:33.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a9e9e640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2900047d0 0x7fd290004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.646+0000 7fd2a9e9e640 1 -- 172.21.15.148:0/3175684178 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd290003530 con 0x7fd290005190 2023-10-30T13:52:33.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd2a9e9e640 1 -- 172.21.15.148:0/3175684178 reap_dead start 2023-10-30T13:52:33.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd2a9e9e640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd290005190 0x7fd29000b990 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fd29c001e40 tx=0x7fd29c01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbf998204f24285 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c003b80 con 0x7fd290005190 2023-10-30T13:52:33.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c003ce0 con 0x7fd290005190 2023-10-30T13:52:33.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd29bfff640 1 -- 172.21.15.148:0/3175684178 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c005240 con 0x7fd290005190 2023-10-30T13:52:33.663 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd2ab901640 1 -- 172.21.15.148:0/3175684178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd290005190 msgr2=0x7fd29000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd2ab901640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd290005190 0x7fd29000b990 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fd29c001e40 tx=0x7fd29c01c740 comp rx=0 tx=0).stop 2023-10-30T13:52:33.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd2ab901640 1 -- 172.21.15.148:0/3175684178 shutdown_connections 2023-10-30T13:52:33.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd2ab901640 1 --2- 172.21.15.148:0/3175684178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd290005190 0x7fd29000b990 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd2ab901640 1 -- 172.21.15.148:0/3175684178 >> 172.21.15.148:0/3175684178 conn(0x7fd2a40cb490 msgr2=0x7fd2a40cb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:33.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd2ab901640 1 -- 172.21.15.148:0/3175684178 shutdown_connections 2023-10-30T13:52:33.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.650+0000 7fd2ab901640 1 -- 172.21.15.148:0/3175684178 wait complete. 2023-10-30T13:52:33.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 Processor -- start 2023-10-30T13:52:33.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 -- start start 2023-10-30T13:52:33.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd290005190 0x7fd2a41588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2a41ee370 0x7fd2a4154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2a4155260 0x7fd2a41e2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41575b0 con 0x7fd2a41ee370 2023-10-30T13:52:33.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a4158de0 con 0x7fd290005190 2023-10-30T13:52:33.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a4157800 con 0x7fd2a4155260 2023-10-30T13:52:33.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a8e9c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2a41ee370 0x7fd2a4154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a8e9c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2a41ee370 0x7fd2a4154850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60950/0 (socket says 172.21.15.148:60950) 2023-10-30T13:52:33.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a8e9c640 1 -- 172.21.15.148:0/2457703896 learned_addr learned my addr 172.21.15.148:0/2457703896 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:33.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a969d640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd290005190 0x7fd2a41588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a9e9e640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2a4155260 0x7fd2a41e2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a8e9c640 1 -- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2a4155260 msgr2=0x7fd2a41e2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a8e9c640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd2a4155260 0x7fd2a41e2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a8e9c640 1 -- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd290005190 msgr2=0x7fd2a41588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a8e9c640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd290005190 0x7fd2a41588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a8e9c640 1 -- 172.21.15.148:0/2457703896 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a4157ae0 con 0x7fd2a41ee370 2023-10-30T13:52:33.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2a8e9c640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2a41ee370 0x7fd2a4154850 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fd294011960 tx=0x7fd294011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd299ffb640 1 -- 172.21.15.148:0/2457703896 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294011e90 con 0x7fd2a41ee370 2023-10-30T13:52:33.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd299ffb640 1 -- 172.21.15.148:0/2457703896 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd294004240 con 0x7fd2a41ee370 2023-10-30T13:52:33.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd299ffb640 1 -- 172.21.15.148:0/2457703896 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400f820 con 0x7fd2a41ee370 2023-10-30T13:52:33.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a4157dc0 con 0x7fd2a41ee370 2023-10-30T13:52:33.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a4158380 con 0x7fd2a41ee370 2023-10-30T13:52:33.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.654+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2a4159b30 con 0x7fd2a41ee370 2023-10-30T13:52:33.671 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd299ffb640 1 -- 172.21.15.148:0/2457703896 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd29400f980 con 0x7fd2a41ee370 2023-10-30T13:52:33.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd299ffb640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2880664a0 0x7fd288068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd299ffb640 1 -- 172.21.15.148:0/2457703896 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fd294084b90 con 0x7fd2a41ee370 2023-10-30T13:52:33.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd299ffb640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7fd28806f880 0x7fd288071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd299ffb640 1 -- 172.21.15.148:0/2457703896 --> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2880723f0 con 0x7fd28806f880 2023-10-30T13:52:33.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd299ffb640 1 -- 172.21.15.148:0/2457703896 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2940892a0 con 0x7fd2a41ee370 2023-10-30T13:52:33.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd2a969d640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2880664a0 0x7fd288068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.674 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd2a9e9e640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7fd28806f880 0x7fd288071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd2a9e9e640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7fd28806f880 0x7fd288071d20 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd2a969d640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2880664a0 0x7fd288068960 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0007d30 tx=0x7fd2a0000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.675 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.658+0000 7fd299ffb640 1 -- 172.21.15.148:0/2457703896 <== osd.2 v2:172.21.15.148:6802/2893462459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fd2880723f0 con 0x7fd28806f880 2023-10-30T13:52:33.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 --> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd2a4159850 con 0x7fd28806f880 2023-10-30T13:52:33.677 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd299ffb640 1 -- 172.21.15.148:0/2457703896 <== osd.2 v2:172.21.15.148:6802/2893462459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd29c0ad040 con 0x7fd28806f880 2023-10-30T13:52:33.677 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:52:33.678 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:52:33.678 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:52:33.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7fd28806f880 msgr2=0x7fd288071d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.679 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.148:6802/2893462459,v1:172.21.15.148:6804/2893462459] conn(0x7fd28806f880 0x7fd288071d20 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2880664a0 msgr2=0x7fd288068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd2880664a0 0x7fd288068960 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0007d30 tx=0x7fd2a0000ee0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2a41ee370 msgr2=0x7fd2a4154850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.680 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 --2- 172.21.15.148:0/2457703896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2a41ee370 0x7fd2a4154850 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fd294011960 tx=0x7fd294011d60 comp rx=0 tx=0).stop 2023-10-30T13:52:33.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2a9e9e640 1 -- 172.21.15.148:0/2457703896 reap_dead start 2023-10-30T13:52:33.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 shutdown_connections 2023-10-30T13:52:33.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 >> 172.21.15.148:0/2457703896 conn(0x7fd2a40cb490 msgr2=0x7fd2a41530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:33.681 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 shutdown_connections 2023-10-30T13:52:33.682 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.674+0000 7fd2ab901640 1 -- 172.21.15.148:0/2457703896 wait complete. 2023-10-30T13:52:33.695 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T13:52:33.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.830+0000 7fa93e31d640 1 Processor -- start 2023-10-30T13:52:33.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.830+0000 7fa93e31d640 1 -- start start 2023-10-30T13:52:33.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93e31d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 0x7fa93815a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93e31d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa9380c2c70 0x7fa938164190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93e31d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9380c1770 con 0x7fa93815a560 2023-10-30T13:52:33.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93e31d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa938069e90 con 0x7fa9380c2c70 2023-10-30T13:52:33.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93e31d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380c1b10 con 0x7fa938156ff0 2023-10-30T13:52:33.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93c8ba640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa9380c2c70 0x7fa938164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa9377fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa93815a560 0x7fa93806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45840/0 (socket says 172.21.15.148:45840) 2023-10-30T13:52:33.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa9377fe640 1 -- 172.21.15.148:0/3071215040 learned_addr learned my addr 172.21.15.148:0/3071215040 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:33.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa937fff640 1 --2- 172.21.15.148:0/3071215040 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 0x7fa93815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93c8ba640 1 -- 172.21.15.148:0/3071215040 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 msgr2=0x7fa93815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:33.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93c8ba640 1 --2- 172.21.15.148:0/3071215040 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 0x7fa93815a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93c8ba640 1 -- 172.21.15.148:0/3071215040 >> v1:172.21.15.148:6789/0 conn(0x7fa93815a560 legacy=0x7fa93806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93c8ba640 1 -- 172.21.15.148:0/3071215040 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938070a90 con 0x7fa9380c2c70 2023-10-30T13:52:33.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa936ffd640 1 -- 172.21.15.148:0/3071215040 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400148197 0 0) 0x7fa92c001120 con 0x7fa93815a560 2023-10-30T13:52:33.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.834+0000 7fa93c8ba640 1 --2- 172.21.15.148:0/3071215040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 0x7fa938164190 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fa930009d90 tx=0x7fa93001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6965f640247da2db server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa936ffd640 1 -- 172.21.15.148:0/3071215040 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9300af600 con 0x7fa9380c2c70 2023-10-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa936ffd640 1 -- 172.21.15.148:0/3071215040 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9300ab050 con 0x7fa9380c2c70 2023-10-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa936ffd640 1 -- 172.21.15.148:0/3071215040 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9300af980 con 0x7fa9380c2c70 2023-10-30T13:52:33.843 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 -- 172.21.15.148:0/3071215040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 msgr2=0x7fa938164190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 --2- 172.21.15.148:0/3071215040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 0x7fa938164190 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fa930009d90 tx=0x7fa93001f130 comp rx=0 tx=0).stop 2023-10-30T13:52:33.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 -- 172.21.15.148:0/3071215040 shutdown_connections 2023-10-30T13:52:33.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 --2- 172.21.15.148:0/3071215040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 0x7fa938164190 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 --2- 172.21.15.148:0/3071215040 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 0x7fa93815a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 -- 172.21.15.148:0/3071215040 >> 172.21.15.148:0/3071215040 conn(0x7fa9380cbeb0 msgr2=0x7fa9380cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 -- 172.21.15.148:0/3071215040 shutdown_connections 2023-10-30T13:52:33.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 -- 172.21.15.148:0/3071215040 wait complete. 2023-10-30T13:52:33.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 Processor -- start 2023-10-30T13:52:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 -- start start 2023-10-30T13:52:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 0x7fa9380b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 0x7fa938152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93815a560 0x7fa9381f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380b0690 con 0x7fa9380c2c70 2023-10-30T13:52:33.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380b0a20 con 0x7fa93815a560 2023-10-30T13:52:33.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa93e31d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa938152d50 con 0x7fa938156ff0 2023-10-30T13:52:33.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa937fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 0x7fa9380b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.838+0000 7fa937fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 0x7fa9380b0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60976/0 (socket says 172.21.15.148:60976) 2023-10-30T13:52:33.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa937fff640 1 -- 172.21.15.148:0/2686410455 learned_addr learned my addr 172.21.15.148:0/2686410455 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:33.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa93c8ba640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93815a560 0x7fa9381f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa9377fe640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 0x7fa938152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa937fff640 1 -- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 msgr2=0x7fa938152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa937fff640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 0x7fa938152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa937fff640 1 -- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93815a560 msgr2=0x7fa9381f2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:33.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa937fff640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93815a560 0x7fa9381f2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:33.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa937fff640 1 -- 172.21.15.148:0/2686410455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa930009a70 con 0x7fa9380c2c70 2023-10-30T13:52:33.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa937fff640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 0x7fa9380b0150 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fa92800e930 tx=0x7fa92800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa934ff9640 1 -- 172.21.15.148:0/2686410455 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928009690 con 0x7fa9380c2c70 2023-10-30T13:52:33.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa934ff9640 1 -- 172.21.15.148:0/2686410455 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92800ca00 con 0x7fa9380c2c70 2023-10-30T13:52:33.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa934ff9640 1 -- 172.21.15.148:0/2686410455 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92800cd70 con 0x7fa9380c2c70 2023-10-30T13:52:33.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9381f0070 con 0x7fa9380c2c70 2023-10-30T13:52:33.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9381f2f80 con 0x7fa9380c2c70 2023-10-30T13:52:33.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93804e170 con 0x7fa9380c2c70 2023-10-30T13:52:33.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.842+0000 7fa934ff9640 1 -- 172.21.15.148:0/2686410455 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa928013470 con 0x7fa9380c2c70 2023-10-30T13:52:33.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.846+0000 7fa934ff9640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9240664a0 0x7fa924068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:33.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.846+0000 7fa934ff9640 1 -- 172.21.15.148:0/2686410455 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fa928084880 con 0x7fa9380c2c70 2023-10-30T13:52:33.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.846+0000 7fa9377fe640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9240664a0 0x7fa924068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:33.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.846+0000 7fa9377fe640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9240664a0 0x7fa924068960 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa9380afc30 tx=0x7fa92c007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:33.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.846+0000 7fa934ff9640 1 -- 172.21.15.148:0/2686410455 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa928051610 con 0x7fa9380c2c70 2023-10-30T13:52:34.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.994+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9380c0ed0 con 0x7fa9240664a0 2023-10-30T13:52:34.000 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa934ff9640 1 -- 172.21.15.148:0/2686410455 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa92c007da0 con 0x7fa9240664a0 2023-10-30T13:52:34.001 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T13:52:34.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9240664a0 msgr2=0x7fa924068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9240664a0 0x7fa924068960 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa9380afc30 tx=0x7fa92c007c10 comp rx=0 tx=0).stop 2023-10-30T13:52:34.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 msgr2=0x7fa9380b0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.003 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 0x7fa9380b0150 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fa92800e930 tx=0x7fa92800edf0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 shutdown_connections 2023-10-30T13:52:34.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa9240664a0 0x7fa924068960 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa93815a560 0x7fa9381f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa938156ff0 0x7fa938152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.004 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 --2- 172.21.15.148:0/2686410455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9380c2c70 0x7fa9380b0150 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 >> 172.21.15.148:0/2686410455 conn(0x7fa9380cbeb0 msgr2=0x7fa938070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:34.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 shutdown_connections 2023-10-30T13:52:34.005 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:33.998+0000 7fa93e31d640 1 -- 172.21.15.148:0/2686410455 wait complete. 2023-10-30T13:52:34.022 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T13:52:34.022 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:52:34.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc8ff5640 1 Processor -- start 2023-10-30T13:52:34.201 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc8ff5640 1 -- start start 2023-10-30T13:52:34.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc8ff5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc40cb3c0 0x7ffbc40c4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc8ff5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc40c4540 con 0x7ffbc41536d0 2023-10-30T13:52:34.203 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc8ff5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc40cb810 con 0x7ffbc415d9d0 2023-10-30T13:52:34.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc8ff5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc415a210 con 0x7ffbc40cb3c0 2023-10-30T13:52:34.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc259c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ffbc41536d0 0x7ffbc4159760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45856/0 (socket says 172.21.15.148:45856) 2023-10-30T13:52:34.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc259c640 1 -- 172.21.15.148:0/4074848850 learned_addr learned my addr 172.21.15.148:0/4074848850 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:34.204 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc2d9d640 1 --2- 172.21.15.148:0/4074848850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc40cb3c0 0x7ffbc40c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc159a640 1 -- 172.21.15.148:0/4074848850 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895638441 0 0) 0x7ffbc40c4540 con 0x7ffbc41536d0 2023-10-30T13:52:34.205 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc159a640 1 -- 172.21.15.148:0/4074848850 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbac003530 con 0x7ffbc41536d0 2023-10-30T13:52:34.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc2d9d640 1 -- 172.21.15.148:0/4074848850 >> v1:172.21.15.150:6789/0 conn(0x7ffbc415d9d0 legacy=0x7ffbc40cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc159a640 1 -- 172.21.15.148:0/4074848850 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900071995 0 0) 0x7ffbc40cb810 con 0x7ffbc415d9d0 2023-10-30T13:52:34.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc2d9d640 1 -- 172.21.15.148:0/4074848850 >> v1:172.21.15.148:6789/0 conn(0x7ffbc41536d0 legacy=0x7ffbc4159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.206 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.198+0000 7ffbc2d9d640 1 -- 172.21.15.148:0/4074848850 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc415a4f0 con 0x7ffbc40cb3c0 2023-10-30T13:52:34.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc2d9d640 1 --2- 172.21.15.148:0/4074848850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc40cb3c0 0x7ffbc40c4000 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009dc0 tx=0x7ffbbc01c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0ef841bb5f4f8d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc159a640 1 -- 172.21.15.148:0/4074848850 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc0a5a00 con 0x7ffbc40cb3c0 2023-10-30T13:52:34.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc159a640 1 -- 172.21.15.148:0/4074848850 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc0a5b60 con 0x7ffbc40cb3c0 2023-10-30T13:52:34.208 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc159a640 1 -- 172.21.15.148:0/4074848850 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc0ad440 con 0x7ffbc40cb3c0 2023-10-30T13:52:34.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4074848850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc40cb3c0 msgr2=0x7ffbc40c4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc8ff5640 1 --2- 172.21.15.148:0/4074848850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc40cb3c0 0x7ffbc40c4000 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009dc0 tx=0x7ffbbc01c940 comp rx=0 tx=0).stop 2023-10-30T13:52:34.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4074848850 shutdown_connections 2023-10-30T13:52:34.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc8ff5640 1 --2- 172.21.15.148:0/4074848850 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc40cb3c0 0x7ffbc40c4000 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.209 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4074848850 >> 172.21.15.148:0/4074848850 conn(0x7ffbc406bf70 msgr2=0x7ffbc406c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:34.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4074848850 shutdown_connections 2023-10-30T13:52:34.210 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.202+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4074848850 wait complete. 2023-10-30T13:52:34.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 Processor -- start 2023-10-30T13:52:34.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 -- start start 2023-10-30T13:52:34.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbc40cb3c0 0x7ffbc41442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbc41536d0 0x7ffbc4143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc415d9d0 0x7ffbc40ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40b33d0 con 0x7ffbc41536d0 2023-10-30T13:52:34.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40b4bb0 con 0x7ffbc40cb3c0 2023-10-30T13:52:34.213 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4145500 con 0x7ffbc415d9d0 2023-10-30T13:52:34.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc1d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbc41536d0 0x7ffbc4143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbc40cb3c0 0x7ffbc41442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc1d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbc41536d0 0x7ffbc4143a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:60988/0 (socket says 172.21.15.148:60988) 2023-10-30T13:52:34.214 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc1d9b640 1 -- 172.21.15.148:0/4103998805 learned_addr learned my addr 172.21.15.148:0/4103998805 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:34.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc2d9d640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc415d9d0 0x7ffbc40ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc1d9b640 1 -- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc415d9d0 msgr2=0x7ffbc40ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc1d9b640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7ffbc415d9d0 0x7ffbc40ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.215 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc1d9b640 1 -- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbc40cb3c0 msgr2=0x7ffbc41442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc1d9b640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffbc40cb3c0 0x7ffbc41442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc1d9b640 1 -- 172.21.15.148:0/4103998805 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc009a70 con 0x7ffbc41536d0 2023-10-30T13:52:34.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc1d9b640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbc41536d0 0x7ffbc4143a60 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8001ec0 tx=0x7ffbb800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbb2ffd640 1 -- 172.21.15.148:0/4103998805 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80054e0 con 0x7ffbc41536d0 2023-10-30T13:52:34.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbb2ffd640 1 -- 172.21.15.148:0/4103998805 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb8003600 con 0x7ffbc41536d0 2023-10-30T13:52:34.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbb2ffd640 1 -- 172.21.15.148:0/4103998805 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80039b0 con 0x7ffbc41536d0 2023-10-30T13:52:34.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc41457c0 con 0x7ffbc41536d0 2023-10-30T13:52:34.217 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.206+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc40b45f0 con 0x7ffbc41536d0 2023-10-30T13:52:34.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffbc415d4b0 con 0x7ffbc41536d0 2023-10-30T13:52:34.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbb2ffd640 1 -- 172.21.15.148:0/4103998805 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffbb8003b10 con 0x7ffbc41536d0 2023-10-30T13:52:34.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbb2ffd640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffba40664a0 0x7ffba4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.219 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbb2ffd640 1 -- 172.21.15.148:0/4103998805 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7ffbb8083ca0 con 0x7ffbc41536d0 2023-10-30T13:52:34.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbb2ffd640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ffba406f880 0x7ffba4071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbb2ffd640 1 -- 172.21.15.148:0/4103998805 --> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffba40723f0 con 0x7ffba406f880 2023-10-30T13:52:34.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbb2ffd640 1 -- 172.21.15.148:0/4103998805 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbb80882a0 con 0x7ffbc41536d0 2023-10-30T13:52:34.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbc259c640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffba40664a0 0x7ffba4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbc2d9d640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ffba406f880 0x7ffba4071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbc2d9d640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ffba406f880 0x7ffba4071d20 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.210+0000 7ffbc259c640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffba40664a0 0x7ffba4068960 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4007fd0 tx=0x7ffbb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.214+0000 7ffbb2ffd640 1 -- 172.21.15.148:0/4103998805 <== osd.3 v2:172.21.15.148:6824/1150735205 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7ffba40723f0 con 0x7ffba406f880 2023-10-30T13:52:34.230 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 --> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ffbc40b31a0 con 0x7ffba406f880 2023-10-30T13:52:34.231 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbb2ffd640 1 -- 172.21.15.148:0/4103998805 <== osd.3 v2:172.21.15.148:6824/1150735205 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ffbc40b31a0 con 0x7ffba406f880 2023-10-30T13:52:34.231 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:52:34.232 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:52:34.232 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:52:34.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ffba406f880 msgr2=0x7ffba4071d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.148:6824/1150735205,v1:172.21.15.148:6825/1150735205] conn(0x7ffba406f880 0x7ffba4071d20 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.233 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffba40664a0 msgr2=0x7ffba4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7ffba40664a0 0x7ffba4068960 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4007fd0 tx=0x7ffbb4009040 comp rx=0 tx=0).stop 2023-10-30T13:52:34.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbc41536d0 msgr2=0x7ffbc4143a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 --2- 172.21.15.148:0/4103998805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbc41536d0 0x7ffbc4143a60 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8001ec0 tx=0x7ffbb800c8e0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc2d9d640 1 -- 172.21.15.148:0/4103998805 reap_dead start 2023-10-30T13:52:34.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 shutdown_connections 2023-10-30T13:52:34.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 >> 172.21.15.148:0/4103998805 conn(0x7ffbc406bf70 msgr2=0x7ffbc40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:34.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 shutdown_connections 2023-10-30T13:52:34.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.226+0000 7ffbc8ff5640 1 -- 172.21.15.148:0/4103998805 wait complete. 2023-10-30T13:52:34.251 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T13:52:34.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.386+0000 7efc46d65640 1 Processor -- start 2023-10-30T13:52:34.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.386+0000 7efc46d65640 1 -- start start 2023-10-30T13:52:34.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc46d65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc400c6b80 0x7efc40071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc46d65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc40070010 con 0x7efc4013fd10 2023-10-30T13:52:34.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc46d65640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc40072260 con 0x7efc400c6f60 2023-10-30T13:52:34.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc46d65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4006f6e0 con 0x7efc400c6b80 2023-10-30T13:52:34.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc45302640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efc4013fd10 0x7efc401565a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45872/0 (socket says 172.21.15.148:45872) 2023-10-30T13:52:34.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc45302640 1 -- 172.21.15.148:0/147847007 learned_addr learned my addr 172.21.15.148:0/147847007 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:34.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc44b01640 1 --2- 172.21.15.148:0/147847007 >> v2:172.21.15.150:3300/0 conn(0x7efc400c6b80 0x7efc40071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389136463 0 0) 0x7efc40070010 con 0x7efc4013fd10 2023-10-30T13:52:34.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc28003530 con 0x7efc4013fd10 2023-10-30T13:52:34.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2052499094 0 0) 0x7efc28003530 con 0x7efc4013fd10 2023-10-30T13:52:34.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 >> v2:172.21.15.150:3300/0 conn(0x7efc400c6b80 msgr2=0x7efc40071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:34.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 --2- 172.21.15.148:0/147847007 >> v2:172.21.15.150:3300/0 conn(0x7efc400c6b80 0x7efc40071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 >> v1:172.21.15.150:6789/0 conn(0x7efc400c6f60 legacy=0x7efc4013f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4006f9c0 con 0x7efc4013fd10 2023-10-30T13:52:34.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7efc38001a00 con 0x7efc4013fd10 2023-10-30T13:52:34.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 >> v1:172.21.15.148:6789/0 conn(0x7efc4013fd10 legacy=0x7efc401565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc28003980 0x7efc28003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc280047d0 0x7efc28004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc2800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4006f6e0 con 0x7efc28005190 2023-10-30T13:52:34.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2800e240 con 0x7efc280047d0 2023-10-30T13:52:34.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2800e400 con 0x7efc28003980 2023-10-30T13:52:34.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc45302640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc2800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc44b01640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc28003980 0x7efc28003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc37fff640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc280047d0 0x7efc28004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc45302640 1 -- 172.21.15.148:0/147847007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc28003980 msgr2=0x7efc28003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:34.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc45302640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc28003980 0x7efc28003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc45302640 1 -- 172.21.15.148:0/147847007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc280047d0 msgr2=0x7efc28004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:34.401 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc45302640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc280047d0 0x7efc28004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.402 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.390+0000 7efc45302640 1 -- 172.21.15.148:0/147847007 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc28003530 con 0x7efc28005190 2023-10-30T13:52:34.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc45302640 1 -- 172.21.15.148:0/147847007 reap_dead start 2023-10-30T13:52:34.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc45302640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc2800dba0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7efc3800a800 tx=0x7efc3801d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbeccc991f28d0e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc380072d0 con 0x7efc28005190 2023-10-30T13:52:34.405 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc380a8070 con 0x7efc28005190 2023-10-30T13:52:34.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc377fe640 1 -- 172.21.15.148:0/147847007 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc38007670 con 0x7efc28005190 2023-10-30T13:52:34.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc46d65640 1 -- 172.21.15.148:0/147847007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 msgr2=0x7efc2800dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc46d65640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc2800dba0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7efc3800a800 tx=0x7efc3801d300 comp rx=0 tx=0).stop 2023-10-30T13:52:34.406 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc46d65640 1 -- 172.21.15.148:0/147847007 shutdown_connections 2023-10-30T13:52:34.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc46d65640 1 --2- 172.21.15.148:0/147847007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc2800dba0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc46d65640 1 -- 172.21.15.148:0/147847007 >> 172.21.15.148:0/147847007 conn(0x7efc400c7910 msgr2=0x7efc400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:34.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc46d65640 1 -- 172.21.15.148:0/147847007 shutdown_connections 2023-10-30T13:52:34.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc46d65640 1 -- 172.21.15.148:0/147847007 wait complete. 2023-10-30T13:52:34.407 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc46d65640 1 Processor -- start 2023-10-30T13:52:34.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.394+0000 7efc46d65640 1 -- start start 2023-10-30T13:52:34.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc46d65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc400aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc46d65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2d90 0x7efc400ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc46d65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c1190 0x7efc401ee860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc46d65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc400b0010 con 0x7efc28005190 2023-10-30T13:52:34.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc46d65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc400c2900 con 0x7efc400c2d90 2023-10-30T13:52:34.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc46d65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc400c31f0 con 0x7efc400c1190 2023-10-30T13:52:34.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc44b01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc400aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.409 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc44b01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc400aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32774/0 (socket says 172.21.15.148:32774) 2023-10-30T13:52:34.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc44b01640 1 -- 172.21.15.148:0/3348696738 learned_addr learned my addr 172.21.15.148:0/3348696738 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:34.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc37fff640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2d90 0x7efc400ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.410 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc45302640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c1190 0x7efc401ee860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc44b01640 1 -- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c1190 msgr2=0x7efc401ee860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc44b01640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c1190 0x7efc401ee860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc44b01640 1 -- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2d90 msgr2=0x7efc400ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc44b01640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2d90 0x7efc400ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc44b01640 1 -- 172.21.15.148:0/3348696738 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc401ef1d0 con 0x7efc28005190 2023-10-30T13:52:34.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc44b01640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc400aeb00 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7efc3c011590 tx=0x7efc3c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc357fa640 1 -- 172.21.15.148:0/3348696738 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c015870 con 0x7efc28005190 2023-10-30T13:52:34.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc357fa640 1 -- 172.21.15.148:0/3348696738 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c0159d0 con 0x7efc28005190 2023-10-30T13:52:34.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc401ef4b0 con 0x7efc28005190 2023-10-30T13:52:34.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc357fa640 1 -- 172.21.15.148:0/3348696738 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c015d40 con 0x7efc28005190 2023-10-30T13:52:34.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc400c22b0 con 0x7efc28005190 2023-10-30T13:52:34.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.398+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4004e170 con 0x7efc28005190 2023-10-30T13:52:34.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.402+0000 7efc357fa640 1 -- 172.21.15.148:0/3348696738 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efc3c013400 con 0x7efc28005190 2023-10-30T13:52:34.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.402+0000 7efc357fa640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc2c0664a0 0x7efc2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.402+0000 7efc357fa640 1 -- 172.21.15.148:0/3348696738 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7efc3c084c20 con 0x7efc28005190 2023-10-30T13:52:34.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.402+0000 7efc37fff640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc2c0664a0 0x7efc2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.402+0000 7efc37fff640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc2c0664a0 0x7efc2c068960 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7efc40159670 tx=0x7efc30009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.406+0000 7efc357fa640 1 -- 172.21.15.148:0/3348696738 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efc3c0519b0 con 0x7efc28005190 2023-10-30T13:52:34.564 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.558+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7efc400723f0 con 0x7efc2c0664a0 2023-10-30T13:52:34.566 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc357fa640 1 -- 172.21.15.148:0/3348696738 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7efc300115a0 con 0x7efc2c0664a0 2023-10-30T13:52:34.566 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T13:52:34.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc2c0664a0 msgr2=0x7efc2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.568 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc2c0664a0 0x7efc2c068960 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7efc40159670 tx=0x7efc30009040 comp rx=0 tx=0).stop 2023-10-30T13:52:34.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 msgr2=0x7efc400aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc400aeb00 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7efc3c011590 tx=0x7efc3c011a50 comp rx=0 tx=0).stop 2023-10-30T13:52:34.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 shutdown_connections 2023-10-30T13:52:34.569 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7efc2c0664a0 0x7efc2c068960 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7efc400c1190 0x7efc401ee860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc400c2d90 0x7efc400ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 --2- 172.21.15.148:0/3348696738 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc28005190 0x7efc400aeb00 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.570 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 >> 172.21.15.148:0/3348696738 conn(0x7efc400c7910 msgr2=0x7efc400c53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:34.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 shutdown_connections 2023-10-30T13:52:34.571 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.562+0000 7efc46d65640 1 -- 172.21.15.148:0/3348696738 wait complete. 2023-10-30T13:52:34.588 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T13:52:34.588 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:52:34.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5baa5d3640 1 Processor -- start 2023-10-30T13:52:34.730 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5baa5d3640 1 -- start start 2023-10-30T13:52:34.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5baa5d3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5ba41517f0 0x7f5ba4157620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.731 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5baa5d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5ba4158b10 0x7f5ba40c6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5baa5d3640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba406c810 con 0x7f5ba415ac10 2023-10-30T13:52:34.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5baa5d3640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba415bbf0 con 0x7f5ba41517f0 2023-10-30T13:52:34.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5baa5d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba4071ed0 con 0x7f5ba4158b10 2023-10-30T13:52:34.732 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5ba3fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5ba41517f0 0x7f5ba4157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5ba3fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f5ba41517f0 0x7f5ba4157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32784/0 (socket says 172.21.15.148:32784) 2023-10-30T13:52:34.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5ba3fff640 1 -- 172.21.15.148:0/1117805320 learned_addr learned my addr 172.21.15.148:0/1117805320 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:34.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5ba37fe640 1 --2- 172.21.15.148:0/1117805320 >> v2:172.21.15.150:3300/0 conn(0x7f5ba4158b10 0x7f5ba40c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.733 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5ba3fff640 1 -- 172.21.15.148:0/1117805320 >> v2:172.21.15.150:3300/0 conn(0x7f5ba4158b10 msgr2=0x7f5ba40c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:34.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5ba3fff640 1 --2- 172.21.15.148:0/1117805320 >> v2:172.21.15.150:3300/0 conn(0x7f5ba4158b10 0x7f5ba40c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5ba3fff640 1 -- 172.21.15.148:0/1117805320 >> v1:172.21.15.148:6789/0 conn(0x7f5ba415ac10 legacy=0x7f5ba40c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.734 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.726+0000 7f5ba3fff640 1 -- 172.21.15.148:0/1117805320 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba415b630 con 0x7f5ba41517f0 2023-10-30T13:52:34.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5ba3fff640 1 --2- 172.21.15.148:0/1117805320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba41517f0 0x7f5ba4157620 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009870 tx=0x7f5b9401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19c5eb4bbad773b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5ba2ffd640 1 -- 172.21.15.148:0/1117805320 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b940ab600 con 0x7f5ba41517f0 2023-10-30T13:52:34.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5ba2ffd640 1 -- 172.21.15.148:0/1117805320 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b940a7050 con 0x7f5ba41517f0 2023-10-30T13:52:34.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5ba2ffd640 1 -- 172.21.15.148:0/1117805320 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b940ab980 con 0x7f5ba41517f0 2023-10-30T13:52:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5baa5d3640 1 -- 172.21.15.148:0/1117805320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba41517f0 msgr2=0x7f5ba4157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5baa5d3640 1 --2- 172.21.15.148:0/1117805320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba41517f0 0x7f5ba4157620 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009870 tx=0x7f5b9401c500 comp rx=0 tx=0).stop 2023-10-30T13:52:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5baa5d3640 1 -- 172.21.15.148:0/1117805320 shutdown_connections 2023-10-30T13:52:34.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5baa5d3640 1 --2- 172.21.15.148:0/1117805320 >> v2:172.21.15.150:3300/0 conn(0x7f5ba4158b10 0x7f5ba40c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5baa5d3640 1 --2- 172.21.15.148:0/1117805320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba41517f0 0x7f5ba4157620 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5baa5d3640 1 -- 172.21.15.148:0/1117805320 >> 172.21.15.148:0/1117805320 conn(0x7f5ba40c7910 msgr2=0x7f5ba40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:34.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5baa5d3640 1 -- 172.21.15.148:0/1117805320 shutdown_connections 2023-10-30T13:52:34.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.730+0000 7f5baa5d3640 1 -- 172.21.15.148:0/1117805320 wait complete. 2023-10-30T13:52:34.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 Processor -- start 2023-10-30T13:52:34.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 -- start start 2023-10-30T13:52:34.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ba41517f0 0x7f5ba41598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba4158b10 0x7f5ba41491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ba415ac10 0x7f5ba414c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba41496e0 con 0x7f5ba4158b10 2023-10-30T13:52:34.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba414acc0 con 0x7f5ba41517f0 2023-10-30T13:52:34.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba41f58e0 con 0x7f5ba415ac10 2023-10-30T13:52:34.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba4158b10 0x7f5ba41491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba4158b10 0x7f5ba41491a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32790/0 (socket says 172.21.15.148:32790) 2023-10-30T13:52:34.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba37fe640 1 -- 172.21.15.148:0/3512944784 learned_addr learned my addr 172.21.15.148:0/3512944784 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:34.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba3fff640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ba41517f0 0x7f5ba41598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba8b70640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ba415ac10 0x7f5ba414c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba37fe640 1 -- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ba415ac10 msgr2=0x7f5ba414c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba37fe640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f5ba415ac10 0x7f5ba414c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba37fe640 1 -- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ba41517f0 msgr2=0x7f5ba41598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba37fe640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ba41517f0 0x7f5ba41598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba37fe640 1 -- 172.21.15.148:0/3512944784 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b940094d0 con 0x7f5ba4158b10 2023-10-30T13:52:34.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba37fe640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba4158b10 0x7f5ba41491a0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800e930 tx=0x7f5b9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba0ff9640 1 -- 172.21.15.148:0/3512944784 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b98009690 con 0x7f5ba4158b10 2023-10-30T13:52:34.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba0ff9640 1 -- 172.21.15.148:0/3512944784 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9800c9a0 con 0x7f5ba4158b10 2023-10-30T13:52:34.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5ba0ff9640 1 -- 172.21.15.148:0/3512944784 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9800cd10 con 0x7f5ba4158b10 2023-10-30T13:52:34.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba41f5bc0 con 0x7f5ba4158b10 2023-10-30T13:52:34.746 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.734+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba41f6180 con 0x7f5ba4158b10 2023-10-30T13:52:34.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.738+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5ba40c5f70 con 0x7f5ba4158b10 2023-10-30T13:52:34.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.738+0000 7f5ba0ff9640 1 -- 172.21.15.148:0/3512944784 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b98013470 con 0x7f5ba4158b10 2023-10-30T13:52:34.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.738+0000 7f5ba0ff9640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b900664a0 0x7f5b90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.738+0000 7f5ba0ff9640 1 -- 172.21.15.148:0/3512944784 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f5b98084b60 con 0x7f5ba4158b10 2023-10-30T13:52:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.738+0000 7f5ba0ff9640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7f5b9006f880 0x7f5b90071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.738+0000 7f5ba0ff9640 1 -- 172.21.15.148:0/3512944784 --> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b900723f0 con 0x7f5b9006f880 2023-10-30T13:52:34.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.738+0000 7f5ba0ff9640 1 -- 172.21.15.148:0/3512944784 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b98084ed0 con 0x7f5ba4158b10 2023-10-30T13:52:34.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.738+0000 7f5ba3fff640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b900664a0 0x7f5b90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.742+0000 7f5ba3fff640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b900664a0 0x7f5b90068960 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5b940ad000 tx=0x7f5b94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.742+0000 7f5ba8b70640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7f5b9006f880 0x7f5b90071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.742+0000 7f5ba8b70640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7f5b9006f880 0x7f5b90071d20 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.746+0000 7f5ba0ff9640 1 -- 172.21.15.148:0/3512944784 <== osd.4 v2:172.21.15.150:6800/381432752 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f5b900723f0 con 0x7f5b9006f880 2023-10-30T13:52:34.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 --> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5ba40c5c90 con 0x7f5b9006f880 2023-10-30T13:52:34.765 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5ba0ff9640 1 -- 172.21.15.148:0/3512944784 <== osd.4 v2:172.21.15.150:6800/381432752 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5b9c001e80 con 0x7f5b9006f880 2023-10-30T13:52:34.766 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:52:34.766 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:52:34.766 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:52:34.767 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7f5b9006f880 msgr2=0x7f5b90071d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6800/381432752,v1:172.21.15.150:6801/381432752] conn(0x7f5b9006f880 0x7f5b90071d20 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b900664a0 msgr2=0x7f5b90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f5b900664a0 0x7f5b90068960 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f5b940ad000 tx=0x7f5b94008040 comp rx=0 tx=0).stop 2023-10-30T13:52:34.768 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba4158b10 msgr2=0x7f5ba41491a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 --2- 172.21.15.148:0/3512944784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5ba4158b10 0x7f5ba41491a0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800e930 tx=0x7f5b9800edf0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5ba8b70640 1 -- 172.21.15.148:0/3512944784 reap_dead start 2023-10-30T13:52:34.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 shutdown_connections 2023-10-30T13:52:34.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 >> 172.21.15.148:0/3512944784 conn(0x7f5ba40c7910 msgr2=0x7f5ba414f790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:34.769 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 shutdown_connections 2023-10-30T13:52:34.770 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.762+0000 7f5baa5d3640 1 -- 172.21.15.148:0/3512944784 wait complete. 2023-10-30T13:52:34.786 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T13:52:34.932 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.926+0000 7f2bc759e640 1 Processor -- start 2023-10-30T13:52:34.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.926+0000 7f2bc759e640 1 -- start start 2023-10-30T13:52:34.934 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc759e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80c5550 0x7f2bc80cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc759e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2bc80cab00 0x7f2bc80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc759e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc80cba30 con 0x7f2bc80cb050 2023-10-30T13:52:34.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc759e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc80ca290 con 0x7f2bc80cab00 2023-10-30T13:52:34.935 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc759e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80c9cf0 con 0x7f2bc80c5550 2023-10-30T13:52:34.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc6d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2bc80cab00 0x7f2bc80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc6d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2bc80cab00 0x7f2bc80cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32802/0 (socket says 172.21.15.148:32802) 2023-10-30T13:52:34.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc6d9d640 1 -- 172.21.15.148:0/4011417538 learned_addr learned my addr 172.21.15.148:0/4011417538 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:34.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc659c640 1 --2- 172.21.15.148:0/4011417538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80c5550 0x7f2bc80cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc6d9d640 1 -- 172.21.15.148:0/4011417538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80c5550 msgr2=0x7f2bc80cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:34.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc6d9d640 1 --2- 172.21.15.148:0/4011417538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80c5550 0x7f2bc80cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc6d9d640 1 -- 172.21.15.148:0/4011417538 >> v1:172.21.15.150:6789/0 conn(0x7f2bc80cb050 legacy=0x7f2bc80cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.930+0000 7f2bc6d9d640 1 -- 172.21.15.148:0/4011417538 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc815d990 con 0x7f2bc80cab00 2023-10-30T13:52:34.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc6d9d640 1 --2- 172.21.15.148:0/4011417538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80cab00 0x7f2bc80cf440 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009e50 tx=0x7f2bc001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09e48c5cd01c1b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc559a640 1 -- 172.21.15.148:0/4011417538 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc00a57d0 con 0x7f2bc80cab00 2023-10-30T13:52:34.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc559a640 1 -- 172.21.15.148:0/4011417538 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc00a5930 con 0x7f2bc80cab00 2023-10-30T13:52:34.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc559a640 1 -- 172.21.15.148:0/4011417538 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc00a5cc0 con 0x7f2bc80cab00 2023-10-30T13:52:34.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc759e640 1 -- 172.21.15.148:0/4011417538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80cab00 msgr2=0x7f2bc80cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc759e640 1 --2- 172.21.15.148:0/4011417538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80cab00 0x7f2bc80cf440 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009e50 tx=0x7f2bc001ca10 comp rx=0 tx=0).stop 2023-10-30T13:52:34.940 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc759e640 1 -- 172.21.15.148:0/4011417538 shutdown_connections 2023-10-30T13:52:34.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc759e640 1 --2- 172.21.15.148:0/4011417538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80cab00 0x7f2bc80cf440 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc759e640 1 --2- 172.21.15.148:0/4011417538 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80c5550 0x7f2bc80cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.941 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc759e640 1 -- 172.21.15.148:0/4011417538 >> 172.21.15.148:0/4011417538 conn(0x7f2bc806bed0 msgr2=0x7f2bc806c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:34.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc759e640 1 -- 172.21.15.148:0/4011417538 shutdown_connections 2023-10-30T13:52:34.942 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.934+0000 7f2bc759e640 1 -- 172.21.15.148:0/4011417538 wait complete. 2023-10-30T13:52:34.944 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 Processor -- start 2023-10-30T13:52:34.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 -- start start 2023-10-30T13:52:34.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80c5550 0x7f2bc8150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80cab00 0x7f2bc814eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.945 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc80cb050 0x7f2bc80ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc81f4b10 con 0x7f2bc80c5550 2023-10-30T13:52:34.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc81f53b0 con 0x7f2bc80cb050 2023-10-30T13:52:34.946 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc81f2080 con 0x7f2bc80cab00 2023-10-30T13:52:34.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80c5550 0x7f2bc8150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80c5550 0x7f2bc8150530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32816/0 (socket says 172.21.15.148:32816) 2023-10-30T13:52:34.947 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc659c640 1 -- 172.21.15.148:0/2638578017 learned_addr learned my addr 172.21.15.148:0/2638578017 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:34.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc6d9d640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc80cb050 0x7f2bc80ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc5d9b640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80cab00 0x7f2bc814eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc659c640 1 -- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80cab00 msgr2=0x7f2bc814eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc659c640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80cab00 0x7f2bc814eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc659c640 1 -- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc80cb050 msgr2=0x7f2bc80ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:34.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc659c640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc80cb050 0x7f2bc80ce120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:34.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc659c640 1 -- 172.21.15.148:0/2638578017 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc0009b30 con 0x7f2bc80c5550 2023-10-30T13:52:34.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc5d9b640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80cab00 0x7f2bc814eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:52:34.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc659c640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80c5550 0x7f2bc8150530 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800e8d0 tx=0x7f2bb800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bb6ffd640 1 -- 172.21.15.148:0/2638578017 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8009510 con 0x7f2bc80c5550 2023-10-30T13:52:34.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bb6ffd640 1 -- 172.21.15.148:0/2638578017 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb800c940 con 0x7f2bc80c5550 2023-10-30T13:52:34.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc81f22f0 con 0x7f2bc80c5550 2023-10-30T13:52:34.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bb6ffd640 1 -- 172.21.15.148:0/2638578017 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb800ccb0 con 0x7f2bc80c5550 2023-10-30T13:52:34.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc81f5040 con 0x7f2bc80c5550 2023-10-30T13:52:34.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.938+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc80c8610 con 0x7f2bc80c5550 2023-10-30T13:52:34.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.942+0000 7f2bb6ffd640 1 -- 172.21.15.148:0/2638578017 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2bb8013470 con 0x7f2bc80c5550 2023-10-30T13:52:34.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.942+0000 7f2bb6ffd640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2ba80664d0 0x7f2ba8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:34.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.942+0000 7f2bb6ffd640 1 -- 172.21.15.148:0/2638578017 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f2bb80849b0 con 0x7f2bc80c5550 2023-10-30T13:52:34.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.942+0000 7f2bc5d9b640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2ba80664d0 0x7f2ba8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:34.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.942+0000 7f2bc5d9b640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2ba80664d0 0x7f2ba8068990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8153a80 tx=0x7f2bbc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:34.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:34.946+0000 7f2bb6ffd640 1 -- 172.21.15.148:0/2638578017 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2bb8051730 con 0x7f2bc80c5550 2023-10-30T13:52:35.093 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:35.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:52:35.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.106+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bc815d570 con 0x7f2ba80664d0 2023-10-30T13:52:35.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.106+0000 7f2bb6ffd640 1 -- 172.21.15.148:0/2638578017 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2bbc001440 con 0x7f2ba80664d0 2023-10-30T13:52:35.113 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T13:52:35.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2ba80664d0 msgr2=0x7f2ba8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2ba80664d0 0x7f2ba8068990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8153a80 tx=0x7f2bbc009040 comp rx=0 tx=0).stop 2023-10-30T13:52:35.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80c5550 msgr2=0x7f2bc8150530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.115 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80c5550 0x7f2bc8150530 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800e8d0 tx=0x7f2bb800ecd0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 shutdown_connections 2023-10-30T13:52:35.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2ba80664d0 0x7f2ba8068990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bc80cb050 0x7f2bc80ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.116 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2bc80cab00 0x7f2bc814eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 --2- 172.21.15.148:0/2638578017 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc80c5550 0x7f2bc8150530 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 >> 172.21.15.148:0/2638578017 conn(0x7f2bc806bed0 msgr2=0x7f2bc80ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:35.117 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 shutdown_connections 2023-10-30T13:52:35.118 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.110+0000 7f2bc759e640 1 -- 172.21.15.148:0/2638578017 wait complete. 2023-10-30T13:52:35.135 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T13:52:35.135 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:52:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45df50f640 1 Processor -- start 2023-10-30T13:52:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45df50f640 1 -- start start 2023-10-30T13:52:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45df50f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f45d814cf40 0x7f45d8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45df50f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d815a270 con 0x7f45d815a700 2023-10-30T13:52:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45df50f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d806c7c0 con 0x7f45d80c8430 2023-10-30T13:52:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45df50f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d806a3d0 con 0x7f45d814cf40 2023-10-30T13:52:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45dd2ab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f45d814cf40 0x7f45d8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45dcaaa640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f45d815a700 0x7f45d8156490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45890/0 (socket says 172.21.15.148:45890) 2023-10-30T13:52:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45dd2ab640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f45d814cf40 0x7f45d8069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32828/0 (socket says 172.21.15.148:32828) 2023-10-30T13:52:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45dcaaa640 1 -- 172.21.15.148:0/1250850584 learned_addr learned my addr 172.21.15.148:0/1250850584 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45dd2ab640 1 -- 172.21.15.148:0/1250850584 >> v1:172.21.15.150:6789/0 conn(0x7f45d80c8430 legacy=0x7f45d80c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45dd2ab640 1 -- 172.21.15.148:0/1250850584 >> v1:172.21.15.148:6789/0 conn(0x7f45d815a700 legacy=0x7f45d8156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45dd2ab640 1 -- 172.21.15.148:0/1250850584 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d8156ed0 con 0x7f45d814cf40 2023-10-30T13:52:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.334+0000 7f45cffff640 1 -- 172.21.15.148:0/1250850584 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393237678 0 0) 0x7f45d815a270 con 0x7f45d815a700 2023-10-30T13:52:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45dd2ab640 1 --2- 172.21.15.148:0/1250850584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d814cf40 0x7f45d8069e90 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f45d40097c0 tx=0x7f45d401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfae4a7038a5c4f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45cffff640 1 -- 172.21.15.148:0/1250850584 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d40ab600 con 0x7f45d814cf40 2023-10-30T13:52:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45cffff640 1 -- 172.21.15.148:0/1250850584 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d40a7050 con 0x7f45d814cf40 2023-10-30T13:52:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45cffff640 1 -- 172.21.15.148:0/1250850584 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d40ab980 con 0x7f45d814cf40 2023-10-30T13:52:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45df50f640 1 -- 172.21.15.148:0/1250850584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d814cf40 msgr2=0x7f45d8069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45df50f640 1 --2- 172.21.15.148:0/1250850584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d814cf40 0x7f45d8069e90 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f45d40097c0 tx=0x7f45d401c480 comp rx=0 tx=0).stop 2023-10-30T13:52:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45df50f640 1 -- 172.21.15.148:0/1250850584 shutdown_connections 2023-10-30T13:52:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45df50f640 1 --2- 172.21.15.148:0/1250850584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d814cf40 0x7f45d8069e90 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45df50f640 1 -- 172.21.15.148:0/1250850584 >> 172.21.15.148:0/1250850584 conn(0x7f45d80c7910 msgr2=0x7f45d80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45df50f640 1 -- 172.21.15.148:0/1250850584 shutdown_connections 2023-10-30T13:52:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.338+0000 7f45df50f640 1 -- 172.21.15.148:0/1250850584 wait complete. 2023-10-30T13:52:35.349 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 Processor -- start 2023-10-30T13:52:35.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 -- start start 2023-10-30T13:52:35.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d80c8430 0x7f45d814a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.350 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45d814cf40 0x7f45d81f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d815a700 0x7f45d8146590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d8149d20 con 0x7f45d815a700 2023-10-30T13:52:35.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d81f1fd0 con 0x7f45d80c8430 2023-10-30T13:52:35.351 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d81f2460 con 0x7f45d814cf40 2023-10-30T13:52:35.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45ddaac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d815a700 0x7f45d8146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45ddaac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d815a700 0x7f45d8146590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32836/0 (socket says 172.21.15.148:32836) 2023-10-30T13:52:35.352 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45ddaac640 1 -- 172.21.15.148:0/589161084 learned_addr learned my addr 172.21.15.148:0/589161084 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:35.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45dd2ab640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d80c8430 0x7f45d814a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45dcaaa640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45d814cf40 0x7f45d81f4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45ddaac640 1 -- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45d814cf40 msgr2=0x7f45d81f4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.353 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45ddaac640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45d814cf40 0x7f45d81f4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45ddaac640 1 -- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d80c8430 msgr2=0x7f45d814a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45ddaac640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d80c8430 0x7f45d814a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45ddaac640 1 -- 172.21.15.148:0/589161084 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d4009420 con 0x7f45d815a700 2023-10-30T13:52:35.354 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45dcaaa640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f45d814cf40 0x7f45d81f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:35.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45ddaac640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d815a700 0x7f45d8146590 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f45d000c7a0 tx=0x7f45d000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45cdffb640 1 -- 172.21.15.148:0/589161084 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0003b80 con 0x7f45d815a700 2023-10-30T13:52:35.355 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45cdffb640 1 -- 172.21.15.148:0/589161084 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d0003ce0 con 0x7f45d815a700 2023-10-30T13:52:35.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45cdffb640 1 -- 172.21.15.148:0/589161084 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d000aa30 con 0x7f45d815a700 2023-10-30T13:52:35.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d814a210 con 0x7f45d815a700 2023-10-30T13:52:35.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.342+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d8149f50 con 0x7f45d815a700 2023-10-30T13:52:35.356 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f45d814d320 con 0x7f45d815a700 2023-10-30T13:52:35.357 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45cdffb640 1 -- 172.21.15.148:0/589161084 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f45d000ab90 con 0x7f45d815a700 2023-10-30T13:52:35.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45cdffb640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45bc0664a0 0x7f45bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.358 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45cdffb640 1 -- 172.21.15.148:0/589161084 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f45d0084460 con 0x7f45d815a700 2023-10-30T13:52:35.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45cdffb640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f45bc06f880 0x7f45bc071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45cdffb640 1 -- 172.21.15.148:0/589161084 --> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45bc0723f0 con 0x7f45bc06f880 2023-10-30T13:52:35.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45cdffb640 1 -- 172.21.15.148:0/589161084 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45d0086a40 con 0x7f45d815a700 2023-10-30T13:52:35.359 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45dd2ab640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45bc0664a0 0x7f45bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45dcaaa640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f45bc06f880 0x7f45bc071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45dcaaa640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f45bc06f880 0x7f45bc071d20 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.346+0000 7f45dd2ab640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45bc0664a0 0x7f45bc068960 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f45d40ad000 tx=0x7f45d40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.360 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.350+0000 7f45cdffb640 1 -- 172.21.15.148:0/589161084 <== osd.5 v2:172.21.15.150:6802/2046109052 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f45bc0723f0 con 0x7f45bc06f880 2023-10-30T13:52:35.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 --> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f45d8159f00 con 0x7f45bc06f880 2023-10-30T13:52:35.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45cdffb640 1 -- 172.21.15.148:0/589161084 <== osd.5 v2:172.21.15.150:6802/2046109052 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f45c80070b0 con 0x7f45bc06f880 2023-10-30T13:52:35.367 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:52:35.368 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:52:35.368 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:52:35.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f45bc06f880 msgr2=0x7f45bc071d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6802/2046109052,v1:172.21.15.150:6805/2046109052] conn(0x7f45bc06f880 0x7f45bc071d20 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45bc0664a0 msgr2=0x7f45bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f45bc0664a0 0x7f45bc068960 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f45d40ad000 tx=0x7f45d40a6000 comp rx=0 tx=0).stop 2023-10-30T13:52:35.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d815a700 msgr2=0x7f45d8146590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 --2- 172.21.15.148:0/589161084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45d815a700 0x7f45d8146590 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f45d000c7a0 tx=0x7f45d000cc60 comp rx=0 tx=0).stop 2023-10-30T13:52:35.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45ddaac640 1 -- 172.21.15.148:0/589161084 reap_dead start 2023-10-30T13:52:35.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 shutdown_connections 2023-10-30T13:52:35.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 >> 172.21.15.148:0/589161084 conn(0x7f45d80c7910 msgr2=0x7f45d81561d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:35.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 shutdown_connections 2023-10-30T13:52:35.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.362+0000 7f45df50f640 1 -- 172.21.15.148:0/589161084 wait complete. 2023-10-30T13:52:35.386 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T13:52:35.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a426b640 1 Processor -- start 2023-10-30T13:52:35.526 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a426b640 1 -- start start 2023-10-30T13:52:35.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a426b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c153c60 0x7fb89c0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.527 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a426b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb89c159830 0x7fb89c159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a426b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c0cb780 con 0x7fb89c0c4430 2023-10-30T13:52:35.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a426b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89c154040 con 0x7fb89c159830 2023-10-30T13:52:35.528 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a426b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c15a2e0 con 0x7fb89c153c60 2023-10-30T13:52:35.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a2808640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb89c159830 0x7fb89c159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.529 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a2007640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c153c60 0x7fb89c0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a1806640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb89c0c4430 0x7fb89c0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:60936/0 (socket says 172.21.15.148:60936) 2023-10-30T13:52:35.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a1806640 1 -- 172.21.15.148:0/522180651 learned_addr learned my addr 172.21.15.148:0/522180651 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:35.530 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a2007640 1 -- 172.21.15.148:0/522180651 >> v2:172.21.15.150:3300/0 conn(0x7fb89c159830 msgr2=0x7fb89c159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:35.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a2007640 1 --2- 172.21.15.148:0/522180651 >> v2:172.21.15.150:3300/0 conn(0x7fb89c159830 0x7fb89c159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a2007640 1 -- 172.21.15.148:0/522180651 >> v1:172.21.15.150:6789/0 conn(0x7fb89c0c4430 legacy=0x7fb89c0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a2007640 1 -- 172.21.15.148:0/522180651 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c15a5c0 con 0x7fb89c153c60 2023-10-30T13:52:35.531 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.522+0000 7fb8a2808640 1 --2- 172.21.15.148:0/522180651 >> v2:172.21.15.150:3300/0 conn(0x7fb89c159830 0x7fb89c159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:52:35.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.526+0000 7fb8a2007640 1 --2- 172.21.15.148:0/522180651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c153c60 0x7fb89c0ca360 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb8980097c0 tx=0x7fb89801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=496fdb14f77264ff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.526+0000 7fb8a1005640 1 -- 172.21.15.148:0/522180651 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8980ab600 con 0x7fb89c153c60 2023-10-30T13:52:35.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.526+0000 7fb8a1005640 1 -- 172.21.15.148:0/522180651 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8980a7050 con 0x7fb89c153c60 2023-10-30T13:52:35.533 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.526+0000 7fb8a1005640 1 -- 172.21.15.148:0/522180651 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8980ab980 con 0x7fb89c153c60 2023-10-30T13:52:35.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.526+0000 7fb8a426b640 1 -- 172.21.15.148:0/522180651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c153c60 msgr2=0x7fb89c0ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.526+0000 7fb8a426b640 1 --2- 172.21.15.148:0/522180651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c153c60 0x7fb89c0ca360 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb8980097c0 tx=0x7fb89801c480 comp rx=0 tx=0).stop 2023-10-30T13:52:35.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.526+0000 7fb8a426b640 1 -- 172.21.15.148:0/522180651 shutdown_connections 2023-10-30T13:52:35.534 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.526+0000 7fb8a426b640 1 --2- 172.21.15.148:0/522180651 >> v2:172.21.15.150:3300/0 conn(0x7fb89c159830 0x7fb89c159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.526+0000 7fb8a426b640 1 --2- 172.21.15.148:0/522180651 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c153c60 0x7fb89c0ca360 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 -- 172.21.15.148:0/522180651 >> 172.21.15.148:0/522180651 conn(0x7fb89c06bfe0 msgr2=0x7fb89c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:35.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 -- 172.21.15.148:0/522180651 shutdown_connections 2023-10-30T13:52:35.535 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 -- 172.21.15.148:0/522180651 wait complete. 2023-10-30T13:52:35.537 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 Processor -- start 2023-10-30T13:52:35.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 -- start start 2023-10-30T13:52:35.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0c4430 0x7fb89c1511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.538 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c153c60 0x7fb89c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c159830 0x7fb89c1f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c0b5470 con 0x7fb89c153c60 2023-10-30T13:52:35.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c1f19e0 con 0x7fb89c0c4430 2023-10-30T13:52:35.539 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a426b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c0c7180 con 0x7fb89c159830 2023-10-30T13:52:35.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a1806640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c153c60 0x7fb89c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a2007640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0c4430 0x7fb89c1511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a1806640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c153c60 0x7fb89c0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32840/0 (socket says 172.21.15.148:32840) 2023-10-30T13:52:35.540 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a1806640 1 -- 172.21.15.148:0/2668075140 learned_addr learned my addr 172.21.15.148:0/2668075140 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:35.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a2808640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c159830 0x7fb89c1f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a1806640 1 -- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c159830 msgr2=0x7fb89c1f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.541 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a1806640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c159830 0x7fb89c1f1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a1806640 1 -- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0c4430 msgr2=0x7fb89c1511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a1806640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0c4430 0x7fb89c1511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.530+0000 7fb8a1806640 1 -- 172.21.15.148:0/2668075140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894009a70 con 0x7fb89c153c60 2023-10-30T13:52:35.542 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb8a1806640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c153c60 0x7fb89c0cdc20 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00c500 tx=0x7fb88c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb892ffd640 1 -- 172.21.15.148:0/2668075140 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c005cf0 con 0x7fb89c153c60 2023-10-30T13:52:35.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb892ffd640 1 -- 172.21.15.148:0/2668075140 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c005e50 con 0x7fb89c153c60 2023-10-30T13:52:35.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb898009420 con 0x7fb89c153c60 2023-10-30T13:52:35.543 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb892ffd640 1 -- 172.21.15.148:0/2668075140 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0072d0 con 0x7fb89c153c60 2023-10-30T13:52:35.544 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89c0b4bd0 con 0x7fb89c153c60 2023-10-30T13:52:35.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89c1f1b20 con 0x7fb89c153c60 2023-10-30T13:52:35.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb892ffd640 1 -- 172.21.15.148:0/2668075140 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb88c007430 con 0x7fb89c153c60 2023-10-30T13:52:35.545 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb892ffd640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb892ffd640 1 -- 172.21.15.148:0/2668075140 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fb88c083ac0 con 0x7fb89c153c60 2023-10-30T13:52:35.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.534+0000 7fb8a2007640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.546 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.538+0000 7fb8a2007640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb8980ad000 tx=0x7fb898008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.547 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.538+0000 7fb892ffd640 1 -- 172.21.15.148:0/2668075140 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb89c1f1b20 con 0x7fb89c153c60 2023-10-30T13:52:35.692 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.686+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fb89c0cbf90 con 0x7fb8800664a0 2023-10-30T13:52:35.693 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb892ffd640 1 -- 172.21.15.148:0/2668075140 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb8980bc090 con 0x7fb8800664a0 2023-10-30T13:52:35.694 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T13:52:35.695 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 msgr2=0x7fb880068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb8980ad000 tx=0x7fb898008040 comp rx=0 tx=0).stop 2023-10-30T13:52:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c153c60 msgr2=0x7fb89c0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c153c60 0x7fb89c0cdc20 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00c500 tx=0x7fb88c00c900 comp rx=0 tx=0).stop 2023-10-30T13:52:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 shutdown_connections 2023-10-30T13:52:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb8800664a0 0x7fb880068960 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb89c153c60 0x7fb89c0cdc20 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb89c159830 0x7fb89c1f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 --2- 172.21.15.148:0/2668075140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb89c0c4430 0x7fb89c1511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 >> 172.21.15.148:0/2668075140 conn(0x7fb89c06bfe0 msgr2=0x7fb89c0cd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 shutdown_connections 2023-10-30T13:52:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.690+0000 7fb8a426b640 1 -- 172.21.15.148:0/2668075140 wait complete. 2023-10-30T13:52:35.715 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T13:52:35.715 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:52:35.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.862+0000 7fd95899c640 1 Processor -- start 2023-10-30T13:52:35.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd95899c640 1 -- start start 2023-10-30T13:52:35.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd95899c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 0x7fd95415a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd95899c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95415d1b0 con 0x7fd9540cb1c0 2023-10-30T13:52:35.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd95899c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95415a740 con 0x7fd9540c4100 2023-10-30T13:52:35.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd95899c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9540cb770 con 0x7fd954153a80 2023-10-30T13:52:35.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd95359e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd9540cb1c0 0x7fd9540ceeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45904/0 (socket says 172.21.15.148:45904) 2023-10-30T13:52:35.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd95359e640 1 -- 172.21.15.148:0/2304990504 learned_addr learned my addr 172.21.15.148:0/2304990504 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:35.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd952d9d640 1 --2- 172.21.15.148:0/2304990504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 0x7fd95415a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd951d9b640 1 -- 172.21.15.148:0/2304990504 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866079366 0 0) 0x7fd95415d1b0 con 0x7fd9540cb1c0 2023-10-30T13:52:35.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd951d9b640 1 -- 172.21.15.148:0/2304990504 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd93c003530 con 0x7fd9540cb1c0 2023-10-30T13:52:35.873 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd951d9b640 1 -- 172.21.15.148:0/2304990504 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939123605 0 0) 0x7fd95415a740 con 0x7fd9540c4100 2023-10-30T13:52:35.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd951d9b640 1 -- 172.21.15.148:0/2304990504 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95415d1b0 con 0x7fd9540c4100 2023-10-30T13:52:35.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd952d9d640 1 -- 172.21.15.148:0/2304990504 >> v1:172.21.15.150:6789/0 conn(0x7fd9540c4100 legacy=0x7fd95415d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd952d9d640 1 -- 172.21.15.148:0/2304990504 >> v1:172.21.15.148:6789/0 conn(0x7fd9540cb1c0 legacy=0x7fd9540ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.874 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.866+0000 7fd952d9d640 1 -- 172.21.15.148:0/2304990504 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9540ca6e0 con 0x7fd954153a80 2023-10-30T13:52:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd952d9d640 1 --2- 172.21.15.148:0/2304990504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 0x7fd95415a200 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd9440097f0 tx=0x7fd94401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f1ed23e64807bcb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd951d9b640 1 -- 172.21.15.148:0/2304990504 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9440a89b0 con 0x7fd954153a80 2023-10-30T13:52:35.877 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd951d9b640 1 -- 172.21.15.148:0/2304990504 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9440a8b10 con 0x7fd954153a80 2023-10-30T13:52:35.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd951d9b640 1 -- 172.21.15.148:0/2304990504 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9440a8ea0 con 0x7fd954153a80 2023-10-30T13:52:35.878 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd95899c640 1 -- 172.21.15.148:0/2304990504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 msgr2=0x7fd95415a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd95899c640 1 --2- 172.21.15.148:0/2304990504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 0x7fd95415a200 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd9440097f0 tx=0x7fd94401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd95899c640 1 -- 172.21.15.148:0/2304990504 shutdown_connections 2023-10-30T13:52:35.879 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd95899c640 1 --2- 172.21.15.148:0/2304990504 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 0x7fd95415a200 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd95899c640 1 -- 172.21.15.148:0/2304990504 >> 172.21.15.148:0/2304990504 conn(0x7fd9540719c0 msgr2=0x7fd95406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:35.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd95899c640 1 -- 172.21.15.148:0/2304990504 shutdown_connections 2023-10-30T13:52:35.880 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.874+0000 7fd95899c640 1 -- 172.21.15.148:0/2304990504 wait complete. 2023-10-30T13:52:35.881 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95899c640 1 Processor -- start 2023-10-30T13:52:35.882 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95899c640 1 -- start start 2023-10-30T13:52:35.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95899c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9540c4100 0x7fd954154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.883 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95899c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9540cb1c0 0x7fd9540cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95899c640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 0x7fd9541f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95899c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd954153360 con 0x7fd9540cb1c0 2023-10-30T13:52:35.884 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95899c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd954154950 con 0x7fd9540c4100 2023-10-30T13:52:35.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95899c640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9541e5200 con 0x7fd954153a80 2023-10-30T13:52:35.885 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9540cb1c0 0x7fd9540cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.886 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9540cb1c0 0x7fd9540cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32854/0 (socket says 172.21.15.148:32854) 2023-10-30T13:52:35.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95259c640 1 -- 172.21.15.148:0/209019300 learned_addr learned my addr 172.21.15.148:0/209019300 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:35.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd952d9d640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9540c4100 0x7fd954154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95359e640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 0x7fd9541f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.887 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95259c640 1 -- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 msgr2=0x7fd9541f6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95259c640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd954153a80 0x7fd9541f6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95259c640 1 -- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9540c4100 msgr2=0x7fd954154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.888 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95259c640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9540c4100 0x7fd954154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.878+0000 7fd95259c640 1 -- 172.21.15.148:0/209019300 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd944009420 con 0x7fd9540cb1c0 2023-10-30T13:52:35.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.882+0000 7fd95259c640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9540cb1c0 0x7fd9540cf1d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fd94800c500 tx=0x7fd94800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.882+0000 7fd93b7fe640 1 -- 172.21.15.148:0/209019300 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9480052d0 con 0x7fd9540cb1c0 2023-10-30T13:52:35.889 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.882+0000 7fd93b7fe640 1 -- 172.21.15.148:0/209019300 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd948005430 con 0x7fd9540cb1c0 2023-10-30T13:52:35.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.882+0000 7fd93b7fe640 1 -- 172.21.15.148:0/209019300 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948012870 con 0x7fd9540cb1c0 2023-10-30T13:52:35.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.882+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9541e4160 con 0x7fd9540cb1c0 2023-10-30T13:52:35.890 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.882+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9541e4720 con 0x7fd9540cb1c0 2023-10-30T13:52:35.891 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.882+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd9540ca400 con 0x7fd9540cb1c0 2023-10-30T13:52:35.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.882+0000 7fd93b7fe640 1 -- 172.21.15.148:0/209019300 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd9480129d0 con 0x7fd9540cb1c0 2023-10-30T13:52:35.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.882+0000 7fd93b7fe640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9400664d0 0x7fd940068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.892 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.886+0000 7fd93b7fe640 1 -- 172.21.15.148:0/209019300 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fd948084a90 con 0x7fd9540cb1c0 2023-10-30T13:52:35.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.886+0000 7fd93b7fe640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fd94006f8b0 0x7fd940071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:35.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.886+0000 7fd93b7fe640 1 -- 172.21.15.148:0/209019300 --> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd940072420 con 0x7fd94006f8b0 2023-10-30T13:52:35.893 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.886+0000 7fd93b7fe640 1 -- 172.21.15.148:0/209019300 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd948084e00 con 0x7fd9540cb1c0 2023-10-30T13:52:35.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.886+0000 7fd952d9d640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9400664d0 0x7fd940068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.886+0000 7fd95359e640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fd94006f8b0 0x7fd940071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:35.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.886+0000 7fd95359e640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fd94006f8b0 0x7fd940071d50 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.894 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.886+0000 7fd952d9d640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9400664d0 0x7fd940068990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd9440b2000 tx=0x7fd9440aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:35.895 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.886+0000 7fd93b7fe640 1 -- 172.21.15.148:0/209019300 <== osd.6 v2:172.21.15.150:6804/397365804 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fd940072420 con 0x7fd94006f8b0 2023-10-30T13:52:35.904 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.898+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 --> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd9540ca160 con 0x7fd94006f8b0 2023-10-30T13:52:35.905 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd93b7fe640 1 -- 172.21.15.148:0/209019300 <== osd.6 v2:172.21.15.150:6804/397365804 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd94c001fb0 con 0x7fd94006f8b0 2023-10-30T13:52:35.905 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:52:35.906 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:52:35.906 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:52:35.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fd94006f8b0 msgr2=0x7fd940071d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6804/397365804,v1:172.21.15.150:6806/397365804] conn(0x7fd94006f8b0 0x7fd940071d50 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9400664d0 msgr2=0x7fd940068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.907 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd9400664d0 0x7fd940068990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd9440b2000 tx=0x7fd9440aa070 comp rx=0 tx=0).stop 2023-10-30T13:52:35.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9540cb1c0 msgr2=0x7fd9540cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:35.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 --2- 172.21.15.148:0/209019300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9540cb1c0 0x7fd9540cf1d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fd94800c500 tx=0x7fd94800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:52:35.908 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95359e640 1 -- 172.21.15.148:0/209019300 reap_dead start 2023-10-30T13:52:35.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 shutdown_connections 2023-10-30T13:52:35.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 >> 172.21.15.148:0/209019300 conn(0x7fd9540719c0 msgr2=0x7fd95414dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:35.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 shutdown_connections 2023-10-30T13:52:35.909 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:35.902+0000 7fd95899c640 1 -- 172.21.15.148:0/209019300 wait complete. 2023-10-30T13:52:35.924 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T13:52:36.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.058+0000 7f7316f0f640 1 Processor -- start 2023-10-30T13:52:36.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.058+0000 7f7316f0f640 1 -- start start 2023-10-30T13:52:36.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.058+0000 7f7316f0f640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f731006ade0 0x7f73100c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.058+0000 7f7316f0f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731006a4e0 con 0x7f7310151910 2023-10-30T13:52:36.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.058+0000 7f7316f0f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73100c6c50 con 0x7f731006a970 2023-10-30T13:52:36.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.058+0000 7f7316f0f640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73100c6f30 con 0x7f731006ade0 2023-10-30T13:52:36.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.058+0000 7f7314cab640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7310151910 0x7f7310158120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45916/0 (socket says 172.21.15.148:45916) 2023-10-30T13:52:36.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.058+0000 7f7314cab640 1 -- 172.21.15.148:0/3567421395 learned_addr learned my addr 172.21.15.148:0/3567421395 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:36.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.058+0000 7f73154ac640 1 --2- 172.21.15.148:0/3567421395 >> v2:172.21.15.148:3300/0 conn(0x7f731006ade0 0x7f73100c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f73077fe640 1 -- 172.21.15.148:0/3567421395 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487107784 0 0) 0x7f731006a4e0 con 0x7f7310151910 2023-10-30T13:52:36.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f73077fe640 1 -- 172.21.15.148:0/3567421395 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f8003530 con 0x7f7310151910 2023-10-30T13:52:36.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f73154ac640 1 -- 172.21.15.148:0/3567421395 >> v1:172.21.15.150:6789/0 conn(0x7f731006a970 legacy=0x7f73100c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.067 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f73154ac640 1 -- 172.21.15.148:0/3567421395 >> v1:172.21.15.148:6789/0 conn(0x7f7310151910 legacy=0x7f7310158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.068 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f73154ac640 1 -- 172.21.15.148:0/3567421395 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73100c7210 con 0x7f731006ade0 2023-10-30T13:52:36.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f73154ac640 1 --2- 172.21.15.148:0/3567421395 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f731006ade0 0x7f73100c5f20 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f7308009d90 tx=0x7f730801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7dcb5db76dfa3eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.069 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f73077fe640 1 -- 172.21.15.148:0/3567421395 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73080af600 con 0x7f731006ade0 2023-10-30T13:52:36.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f73077fe640 1 -- 172.21.15.148:0/3567421395 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73080ab050 con 0x7f731006ade0 2023-10-30T13:52:36.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f73077fe640 1 -- 172.21.15.148:0/3567421395 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73080af980 con 0x7f731006ade0 2023-10-30T13:52:36.070 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f7316f0f640 1 -- 172.21.15.148:0/3567421395 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f731006ade0 msgr2=0x7f73100c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f7316f0f640 1 --2- 172.21.15.148:0/3567421395 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f731006ade0 0x7f73100c5f20 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f7308009d90 tx=0x7f730801f130 comp rx=0 tx=0).stop 2023-10-30T13:52:36.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f7316f0f640 1 -- 172.21.15.148:0/3567421395 shutdown_connections 2023-10-30T13:52:36.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f7316f0f640 1 --2- 172.21.15.148:0/3567421395 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f731006ade0 0x7f73100c5f20 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.071 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f7316f0f640 1 -- 172.21.15.148:0/3567421395 >> 172.21.15.148:0/3567421395 conn(0x7f73100c7910 msgr2=0x7f73100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:36.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.062+0000 7f7316f0f640 1 -- 172.21.15.148:0/3567421395 shutdown_connections 2023-10-30T13:52:36.072 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 -- 172.21.15.148:0/3567421395 wait complete. 2023-10-30T13:52:36.074 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 Processor -- start 2023-10-30T13:52:36.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 -- start start 2023-10-30T13:52:36.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f731006a970 0x7f73100ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.075 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7310151910 0x7f73100ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73100c0e50 0x7f73100c12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100ae7d0 con 0x7f73100c0e50 2023-10-30T13:52:36.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100ca9a0 con 0x7f7310151910 2023-10-30T13:52:36.076 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73100c09c0 con 0x7f731006a970 2023-10-30T13:52:36.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73154ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73100c0e50 0x7f73100c12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73154ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73100c0e50 0x7f73100c12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32874/0 (socket says 172.21.15.148:32874) 2023-10-30T13:52:36.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73154ac640 1 -- 172.21.15.148:0/2808079540 learned_addr learned my addr 172.21.15.148:0/2808079540 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:36.077 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7307fff640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7310151910 0x7f73100ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7314cab640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f731006a970 0x7f73100ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73154ac640 1 -- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f731006a970 msgr2=0x7f73100ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.078 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73154ac640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f731006a970 0x7f73100ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73154ac640 1 -- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7310151910 msgr2=0x7f73100ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73154ac640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7310151910 0x7f73100ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73154ac640 1 -- 172.21.15.148:0/2808079540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7308009a70 con 0x7f73100c0e50 2023-10-30T13:52:36.079 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7307fff640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7310151910 0x7f73100ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:36.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73154ac640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73100c0e50 0x7f73100c12e0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f7308009d90 tx=0x7f730801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73057fa640 1 -- 172.21.15.148:0/2808079540 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308009450 con 0x7f73100c0e50 2023-10-30T13:52:36.080 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73057fa640 1 -- 172.21.15.148:0/2808079540 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73080b9410 con 0x7f73100c0e50 2023-10-30T13:52:36.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f73057fa640 1 -- 172.21.15.148:0/2808079540 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73080b97c0 con 0x7f73100c0e50 2023-10-30T13:52:36.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.066+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731014c7e0 con 0x7f73100c0e50 2023-10-30T13:52:36.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.070+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731014c540 con 0x7f73100c0e50 2023-10-30T13:52:36.081 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.070+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731004e170 con 0x7f73100c0e50 2023-10-30T13:52:36.082 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.070+0000 7f73057fa640 1 -- 172.21.15.148:0/2808079540 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f73080b6050 con 0x7f73100c0e50 2023-10-30T13:52:36.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.070+0000 7f73057fa640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72fc0664d0 0x7f72fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.070+0000 7f73057fa640 1 -- 172.21.15.148:0/2808079540 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f7308122c00 con 0x7f73100c0e50 2023-10-30T13:52:36.083 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.070+0000 7f7314cab640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72fc0664d0 0x7f72fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.074+0000 7f7314cab640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72fc0664d0 0x7f72fc068990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f730c0011f0 tx=0x7f730c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.084 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.074+0000 7f73057fa640 1 -- 172.21.15.148:0/2808079540 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73080ef7b0 con 0x7f73100c0e50 2023-10-30T13:52:36.234 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.230+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f731006a170 con 0x7f72fc0664d0 2023-10-30T13:52:36.235 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.230+0000 7f73057fa640 1 -- 172.21.15.148:0/2808079540 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f730c0013f0 con 0x7f72fc0664d0 2023-10-30T13:52:36.235 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T13:52:36.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72fc0664d0 msgr2=0x7f72fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.237 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72fc0664d0 0x7f72fc068990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f730c0011f0 tx=0x7f730c009040 comp rx=0 tx=0).stop 2023-10-30T13:52:36.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73100c0e50 msgr2=0x7f73100c12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73100c0e50 0x7f73100c12e0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f7308009d90 tx=0x7f730801bed0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.238 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 shutdown_connections 2023-10-30T13:52:36.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f72fc0664d0 0x7f72fc068990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73100c0e50 0x7f73100c12e0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.239 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7310151910 0x7f73100ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 --2- 172.21.15.148:0/2808079540 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f731006a970 0x7f73100ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 >> 172.21.15.148:0/2808079540 conn(0x7f73100c7910 msgr2=0x7f731014f770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:36.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 shutdown_connections 2023-10-30T13:52:36.240 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.234+0000 7f7316f0f640 1 -- 172.21.15.148:0/2808079540 wait complete. 2023-10-30T13:52:36.256 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T13:52:36.257 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:52:36.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.402+0000 7fa0d830f640 1 Processor -- start 2023-10-30T13:52:36.408 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.402+0000 7fa0d830f640 1 -- start start 2023-10-30T13:52:36.411 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d830f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0d00cb3c0 0x7fa0d00c4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d830f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d00c4540 con 0x7fa0d015d9d0 2023-10-30T13:52:36.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d830f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d00cb810 con 0x7fa0d01536d0 2023-10-30T13:52:36.412 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d830f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d015a210 con 0x7fa0d00cb3c0 2023-10-30T13:52:36.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d58aa640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa0d015d9d0 0x7fa0d00cf090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45926/0 (socket says 172.21.15.148:45926) 2023-10-30T13:52:36.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d58aa640 1 -- 172.21.15.148:0/2383441615 learned_addr learned my addr 172.21.15.148:0/2383441615 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:36.413 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d68ac640 1 --2- 172.21.15.148:0/2383441615 >> v2:172.21.15.150:3300/0 conn(0x7fa0d00cb3c0 0x7fa0d00c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844488628 0 0) 0x7fa0d00c4540 con 0x7fa0d015d9d0 2023-10-30T13:52:36.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0bc003530 con 0x7fa0d015d9d0 2023-10-30T13:52:36.414 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 193327172 0 0) 0x7fa0bc003530 con 0x7fa0d015d9d0 2023-10-30T13:52:36.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 >> v2:172.21.15.150:3300/0 conn(0x7fa0d00cb3c0 msgr2=0x7fa0d00c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:36.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 --2- 172.21.15.148:0/2383441615 >> v2:172.21.15.150:3300/0 conn(0x7fa0d00cb3c0 0x7fa0d00c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 >> v1:172.21.15.150:6789/0 conn(0x7fa0d01536d0 legacy=0x7fa0d0159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.415 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d015a4f0 con 0x7fa0d015d9d0 2023-10-30T13:52:36.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fa0c0001680 con 0x7fa0d015d9d0 2023-10-30T13:52:36.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 >> v1:172.21.15.148:6789/0 conn(0x7fa0d015d9d0 legacy=0x7fa0d00cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.416 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0bc003980 0x7fa0bc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0bc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc005190 0x7fa0bc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d015a210 con 0x7fa0bc0047d0 2023-10-30T13:52:36.417 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc00c030 con 0x7fa0bc005190 2023-10-30T13:52:36.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc00c1f0 con 0x7fa0bc003980 2023-10-30T13:52:36.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d58aa640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0bc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.418 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d68ac640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc005190 0x7fa0bc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d60ab640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0bc003980 0x7fa0bc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d58aa640 1 -- 172.21.15.148:0/2383441615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0bc003980 msgr2=0x7fa0bc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:36.419 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d58aa640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0bc003980 0x7fa0bc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d58aa640 1 -- 172.21.15.148:0/2383441615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc005190 msgr2=0x7fa0bc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:36.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d58aa640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0bc005190 0x7fa0bc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d58aa640 1 -- 172.21.15.148:0/2383441615 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc003530 con 0x7fa0bc0047d0 2023-10-30T13:52:36.420 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d68ac640 1 -- 172.21.15.148:0/2383441615 reap_dead start 2023-10-30T13:52:36.421 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.406+0000 7fa0d60ab640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0bc003980 0x7fa0bc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:52:36.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d58aa640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0bc004c50 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c240 tx=0x7fa0c001f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca53e2b669464c48 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.424 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0005ae0 con 0x7fa0bc0047d0 2023-10-30T13:52:36.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c00ab070 con 0x7fa0bc0047d0 2023-10-30T13:52:36.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d50a9640 1 -- 172.21.15.148:0/2383441615 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0007070 con 0x7fa0bc0047d0 2023-10-30T13:52:36.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d830f640 1 -- 172.21.15.148:0/2383441615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 msgr2=0x7fa0bc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.425 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d830f640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0bc004c50 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c240 tx=0x7fa0c001f370 comp rx=0 tx=0).stop 2023-10-30T13:52:36.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d830f640 1 -- 172.21.15.148:0/2383441615 shutdown_connections 2023-10-30T13:52:36.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d830f640 1 --2- 172.21.15.148:0/2383441615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0bc004c50 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.426 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d830f640 1 -- 172.21.15.148:0/2383441615 >> 172.21.15.148:0/2383441615 conn(0x7fa0d006bf70 msgr2=0x7fa0d006c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:36.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d830f640 1 -- 172.21.15.148:0/2383441615 shutdown_connections 2023-10-30T13:52:36.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d830f640 1 -- 172.21.15.148:0/2383441615 wait complete. 2023-10-30T13:52:36.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.410+0000 7fa0d830f640 1 Processor -- start 2023-10-30T13:52:36.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 -- start start 2023-10-30T13:52:36.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0d00cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d00c90d0 0x7fa0d01ef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d015c380 0x7fa0d00cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d00c67b0 con 0x7fa0bc0047d0 2023-10-30T13:52:36.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d01553b0 con 0x7fa0d00c90d0 2023-10-30T13:52:36.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d0154b10 con 0x7fa0d015c380 2023-10-30T13:52:36.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d60ab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0d00cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d60ab640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0d00cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32898/0 (socket says 172.21.15.148:32898) 2023-10-30T13:52:36.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d60ab640 1 -- 172.21.15.148:0/361441633 learned_addr learned my addr 172.21.15.148:0/361441633 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:36.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d58aa640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d00c90d0 0x7fa0d01ef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d68ac640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d015c380 0x7fa0d00cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d60ab640 1 -- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d015c380 msgr2=0x7fa0d00cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d60ab640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa0d015c380 0x7fa0d00cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d60ab640 1 -- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d00c90d0 msgr2=0x7fa0d01ef6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d60ab640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d00c90d0 0x7fa0d01ef6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d60ab640 1 -- 172.21.15.148:0/361441633 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d0154df0 con 0x7fa0bc0047d0 2023-10-30T13:52:36.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d58aa640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0d00c90d0 0x7fa0d01ef6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:36.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d60ab640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0d00cf1b0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00d4d0 tx=0x7fa0cc00d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0c6ffd640 1 -- 172.21.15.148:0/361441633 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc005480 con 0x7fa0bc0047d0 2023-10-30T13:52:36.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0c6ffd640 1 -- 172.21.15.148:0/361441633 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc007030 con 0x7fa0bc0047d0 2023-10-30T13:52:36.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d01550d0 con 0x7fa0bc0047d0 2023-10-30T13:52:36.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0c6ffd640 1 -- 172.21.15.148:0/361441633 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc0073e0 con 0x7fa0bc0047d0 2023-10-30T13:52:36.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d01ef1d0 con 0x7fa0bc0047d0 2023-10-30T13:52:36.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.414+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa0d015d1d0 con 0x7fa0bc0047d0 2023-10-30T13:52:36.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0c6ffd640 1 -- 172.21.15.148:0/361441633 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0cc007540 con 0x7fa0bc0047d0 2023-10-30T13:52:36.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0c6ffd640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 0x7fa0b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0c6ffd640 1 -- 172.21.15.148:0/361441633 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fa0cc083ec0 con 0x7fa0bc0047d0 2023-10-30T13:52:36.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0c6ffd640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fa0b406f8b0 0x7fa0b4071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0c6ffd640 1 -- 172.21.15.148:0/361441633 --> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0b4072420 con 0x7fa0b406f8b0 2023-10-30T13:52:36.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0c6ffd640 1 -- 172.21.15.148:0/361441633 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0cc0864a0 con 0x7fa0bc0047d0 2023-10-30T13:52:36.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0d58aa640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 0x7fa0b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0d68ac640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fa0b406f8b0 0x7fa0b4071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0d58aa640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 0x7fa0b4068990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00af000 tx=0x7fa0c00aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.418+0000 7fa0d68ac640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fa0b406f8b0 0x7fa0b4071d50 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.437 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.422+0000 7fa0c6ffd640 1 -- 172.21.15.148:0/361441633 <== osd.7 v2:172.21.15.150:6824/1265253901 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa0b4072420 con 0x7fa0b406f8b0 2023-10-30T13:52:36.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 --> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa0d015d4b0 con 0x7fa0b406f8b0 2023-10-30T13:52:36.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0c6ffd640 1 -- 172.21.15.148:0/361441633 <== osd.7 v2:172.21.15.150:6824/1265253901 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa0c800f040 con 0x7fa0b406f8b0 2023-10-30T13:52:36.439 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2023-10-30T13:52:36.440 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:52:36.440 INFO:teuthology.orchestra.run.smithi148.stdout:} 2023-10-30T13:52:36.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fa0b406f8b0 msgr2=0x7fa0b4071d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6824/1265253901,v1:172.21.15.150:6825/1265253901] conn(0x7fa0b406f8b0 0x7fa0b4071d50 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 msgr2=0x7fa0b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa0b40664d0 0x7fa0b4068990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa0c00af000 tx=0x7fa0c00aa000 comp rx=0 tx=0).stop 2023-10-30T13:52:36.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 msgr2=0x7fa0d00cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 --2- 172.21.15.148:0/361441633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa0bc0047d0 0x7fa0d00cf1b0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00d4d0 tx=0x7fa0cc00d990 comp rx=0 tx=0).stop 2023-10-30T13:52:36.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d68ac640 1 -- 172.21.15.148:0/361441633 reap_dead start 2023-10-30T13:52:36.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 shutdown_connections 2023-10-30T13:52:36.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 >> 172.21.15.148:0/361441633 conn(0x7fa0d006bf70 msgr2=0x7fa0d01518f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:36.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 shutdown_connections 2023-10-30T13:52:36.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.434+0000 7fa0d830f640 1 -- 172.21.15.148:0/361441633 wait complete. 2023-10-30T13:52:36.458 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T13:52:36.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.610+0000 7f1b393a1640 1 Processor -- start 2023-10-30T13:52:36.615 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.610+0000 7f1b393a1640 1 -- start start 2023-10-30T13:52:36.617 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b393a1640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1b34153c60 0x7f1b340ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b393a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b340c4430 0x7f1b340cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b393a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b340cb780 con 0x7f1b34159830 2023-10-30T13:52:36.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b393a1640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b34154040 con 0x7f1b34153c60 2023-10-30T13:52:36.618 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b393a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340cc9f0 con 0x7f1b340c4430 2023-10-30T13:52:36.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b327fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b340c4430 0x7f1b340cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b327fc640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b340c4430 0x7f1b340cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3301/0 says I am v2:172.21.15.148:55258/0 (socket says 172.21.15.148:55258) 2023-10-30T13:52:36.619 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b327fc640 1 -- 172.21.15.148:0/1021493377 learned_addr learned my addr 172.21.15.148:0/1021493377 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:36.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b32ffd640 1 --2- 172.21.15.148:0/1021493377 >> v2:172.21.15.148:3300/0 conn(0x7f1b34153c60 0x7f1b340ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b32ffd640 1 -- 172.21.15.148:0/1021493377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b340c4430 msgr2=0x7f1b340cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:36.620 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b32ffd640 1 --2- 172.21.15.148:0/1021493377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b340c4430 0x7f1b340cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b32ffd640 1 -- 172.21.15.148:0/1021493377 >> v1:172.21.15.150:6789/0 conn(0x7f1b34159830 legacy=0x7f1b340cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.621 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.614+0000 7f1b32ffd640 1 -- 172.21.15.148:0/1021493377 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3415a2e0 con 0x7f1b34153c60 2023-10-30T13:52:36.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b32ffd640 1 --2- 172.21.15.148:0/1021493377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b34153c60 0x7f1b340ca360 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f1b240097c0 tx=0x7f1b2401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deee43245d3e8b36 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.622 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b31ffb640 1 -- 172.21.15.148:0/1021493377 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240ab600 con 0x7f1b34153c60 2023-10-30T13:52:36.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b31ffb640 1 -- 172.21.15.148:0/1021493377 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b240a7050 con 0x7f1b34153c60 2023-10-30T13:52:36.623 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b31ffb640 1 -- 172.21.15.148:0/1021493377 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240ab980 con 0x7f1b34153c60 2023-10-30T13:52:36.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b393a1640 1 -- 172.21.15.148:0/1021493377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b34153c60 msgr2=0x7f1b340ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b393a1640 1 --2- 172.21.15.148:0/1021493377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b34153c60 0x7f1b340ca360 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f1b240097c0 tx=0x7f1b2401c480 comp rx=0 tx=0).stop 2023-10-30T13:52:36.624 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b393a1640 1 -- 172.21.15.148:0/1021493377 shutdown_connections 2023-10-30T13:52:36.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b393a1640 1 --2- 172.21.15.148:0/1021493377 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b340c4430 0x7f1b340cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b393a1640 1 --2- 172.21.15.148:0/1021493377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b34153c60 0x7f1b340ca360 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b393a1640 1 -- 172.21.15.148:0/1021493377 >> 172.21.15.148:0/1021493377 conn(0x7f1b3406bfe0 msgr2=0x7f1b3406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:36.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b393a1640 1 -- 172.21.15.148:0/1021493377 shutdown_connections 2023-10-30T13:52:36.626 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.618+0000 7f1b393a1640 1 -- 172.21.15.148:0/1021493377 wait complete. 2023-10-30T13:52:36.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b393a1640 1 Processor -- start 2023-10-30T13:52:36.627 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b393a1640 1 -- start start 2023-10-30T13:52:36.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b393a1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b340c4430 0x7f1b340c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b393a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b34153c60 0x7f1b340ce720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.628 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b393a1640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b34159830 0x7f1b34144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b393a1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340b31b0 con 0x7f1b340c4430 2023-10-30T13:52:36.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b393a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340c68f0 con 0x7f1b34153c60 2023-10-30T13:52:36.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b393a1640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340b3450 con 0x7f1b34159830 2023-10-30T13:52:36.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b32ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b340c4430 0x7f1b340c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b32ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b340c4430 0x7f1b340c7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32910/0 (socket says 172.21.15.148:32910) 2023-10-30T13:52:36.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b32ffd640 1 -- 172.21.15.148:0/1582624367 learned_addr learned my addr 172.21.15.148:0/1582624367 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:36.630 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b327fc640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b34153c60 0x7f1b340ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b337fe640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b34159830 0x7f1b34144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b32ffd640 1 -- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b34159830 msgr2=0x7f1b34144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.632 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b32ffd640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b34159830 0x7f1b34144c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b32ffd640 1 -- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b34153c60 msgr2=0x7f1b340ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b32ffd640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b34153c60 0x7f1b340ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b32ffd640 1 -- 172.21.15.148:0/1582624367 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b24009420 con 0x7f1b340c4430 2023-10-30T13:52:36.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.622+0000 7f1b32ffd640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b340c4430 0x7f1b340c7e90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f1b2401a0b0 tx=0x7f1b24019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b1b7fe640 1 -- 172.21.15.148:0/1582624367 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240b4410 con 0x7f1b340c4430 2023-10-30T13:52:36.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b1b7fe640 1 -- 172.21.15.148:0/1582624367 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b240b4570 con 0x7f1b340c4430 2023-10-30T13:52:36.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b340b3730 con 0x7f1b340c4430 2023-10-30T13:52:36.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b1b7fe640 1 -- 172.21.15.148:0/1582624367 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240b4920 con 0x7f1b340c4430 2023-10-30T13:52:36.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b341435c0 con 0x7f1b340c4430 2023-10-30T13:52:36.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3404e170 con 0x7f1b340c4430 2023-10-30T13:52:36.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b1b7fe640 1 -- 172.21.15.148:0/1582624367 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1b240b2040 con 0x7f1b340c4430 2023-10-30T13:52:36.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b1b7fe640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b200664a0 0x7f1b20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b1b7fe640 1 -- 172.21.15.148:0/1582624367 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f1b2411e490 con 0x7f1b340c4430 2023-10-30T13:52:36.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.626+0000 7f1b327fc640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b200664a0 0x7f1b20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.630+0000 7f1b327fc640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b200664a0 0x7f1b20068960 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1b34158ba0 tx=0x7f1b28009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.630+0000 7f1b1b7fe640 1 -- 172.21.15.148:0/1582624367 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1b240eb180 con 0x7f1b340c4430 2023-10-30T13:52:36.788 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.782+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b340b35c0 con 0x7f1b200664a0 2023-10-30T13:52:36.789 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b1b7fe640 1 -- 172.21.15.148:0/1582624367 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1b340b35c0 con 0x7f1b200664a0 2023-10-30T13:52:36.790 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T13:52:36.791 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b200664a0 msgr2=0x7f1b20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b200664a0 0x7f1b20068960 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1b34158ba0 tx=0x7f1b28009240 comp rx=0 tx=0).stop 2023-10-30T13:52:36.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b340c4430 msgr2=0x7f1b340c7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.792 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b340c4430 0x7f1b340c7e90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f1b2401a0b0 tx=0x7f1b24019220 comp rx=0 tx=0).stop 2023-10-30T13:52:36.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 shutdown_connections 2023-10-30T13:52:36.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1b200664a0 0x7f1b20068960 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1b34159830 0x7f1b34144c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.793 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b34153c60 0x7f1b340ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 --2- 172.21.15.148:0/1582624367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b340c4430 0x7f1b340c7e90 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 >> 172.21.15.148:0/1582624367 conn(0x7f1b3406bfe0 msgr2=0x7f1b34151d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:36.794 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 shutdown_connections 2023-10-30T13:52:36.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.786+0000 7f1b393a1640 1 -- 172.21.15.148:0/1582624367 wait complete. 2023-10-30T13:52:36.811 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:52:36.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.942+0000 7f589c91d640 1 Processor -- start 2023-10-30T13:52:36.948 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.942+0000 7f589c91d640 1 -- start start 2023-10-30T13:52:36.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f589c91d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f589806e470 0x7f589806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.949 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f589c91d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589806afa0 con 0x7f58980c8490 2023-10-30T13:52:36.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f589c91d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5898154150 con 0x7f5898153d70 2023-10-30T13:52:36.950 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f589c91d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f589806d780 con 0x7f589806e470 2023-10-30T13:52:36.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f5896d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f58980c8490 0x7f58980c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45928/0 (socket says 172.21.15.148:45928) 2023-10-30T13:52:36.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f5896d9d640 1 -- 172.21.15.148:0/529394843 learned_addr learned my addr 172.21.15.148:0/529394843 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:36.951 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f588dd9b640 1 --2- 172.21.15.148:0/529394843 >> v2:172.21.15.150:3300/0 conn(0x7f589806e470 0x7f589806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f5895d9b640 1 -- 172.21.15.148:0/529394843 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403514329 0 0) 0x7f589806afa0 con 0x7f58980c8490 2023-10-30T13:52:36.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f5895d9b640 1 -- 172.21.15.148:0/529394843 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5884003530 con 0x7f58980c8490 2023-10-30T13:52:36.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f588dd9b640 1 -- 172.21.15.148:0/529394843 >> v1:172.21.15.150:6789/0 conn(0x7f5898153d70 legacy=0x7f58980cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f588dd9b640 1 -- 172.21.15.148:0/529394843 >> v1:172.21.15.148:6789/0 conn(0x7f58980c8490 legacy=0x7f58980c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.952 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f5895d9b640 1 -- 172.21.15.148:0/529394843 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3680885129 0 0) 0x7f5884003530 con 0x7f58980c8490 2023-10-30T13:52:36.953 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.946+0000 7f588dd9b640 1 -- 172.21.15.148:0/529394843 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589806e030 con 0x7f589806e470 2023-10-30T13:52:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f588dd9b640 1 --2- 172.21.15.148:0/529394843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 0x7f589806ca80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f58800097c0 tx=0x7f588001c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1086795696f6a91b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f5895d9b640 1 -- 172.21.15.148:0/529394843 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5890001a00 con 0x7f589806e470 2023-10-30T13:52:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f5895d9b640 1 -- 172.21.15.148:0/529394843 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58800a7050 con 0x7f589806e470 2023-10-30T13:52:36.954 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f5895d9b640 1 -- 172.21.15.148:0/529394843 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58800ab600 con 0x7f589806e470 2023-10-30T13:52:36.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f589c91d640 1 -- 172.21.15.148:0/529394843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 msgr2=0x7f589806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.955 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f589c91d640 1 --2- 172.21.15.148:0/529394843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 0x7f589806ca80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f58800097c0 tx=0x7f588001c370 comp rx=0 tx=0).stop 2023-10-30T13:52:36.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f589c91d640 1 -- 172.21.15.148:0/529394843 shutdown_connections 2023-10-30T13:52:36.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f589c91d640 1 --2- 172.21.15.148:0/529394843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 0x7f589806ca80 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f589c91d640 1 -- 172.21.15.148:0/529394843 >> 172.21.15.148:0/529394843 conn(0x7f5898071ea0 msgr2=0x7f5898065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:36.956 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f589c91d640 1 -- 172.21.15.148:0/529394843 shutdown_connections 2023-10-30T13:52:36.957 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f589c91d640 1 -- 172.21.15.148:0/529394843 wait complete. 2023-10-30T13:52:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.950+0000 7f589c91d640 1 Processor -- start 2023-10-30T13:52:36.959 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 -- start start 2023-10-30T13:52:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 0x7f5898143f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58980c8490 0x7f5898155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5898153d70 0x7f58981556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.960 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5898154d70 con 0x7f5898153d70 2023-10-30T13:52:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5898156f80 con 0x7f589806e470 2023-10-30T13:52:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58980cc1c0 con 0x7f58980c8490 2023-10-30T13:52:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f5896d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5898153d70 0x7f58981556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.961 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f5896d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5898153d70 0x7f58981556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:32918/0 (socket says 172.21.15.148:32918) 2023-10-30T13:52:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f5896d9d640 1 -- 172.21.15.148:0/2880898423 learned_addr learned my addr 172.21.15.148:0/2880898423 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f588dd9b640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58980c8490 0x7f5898155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589659c640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 0x7f5898143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.962 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f5896d9d640 1 -- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58980c8490 msgr2=0x7f5898155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f5896d9d640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58980c8490 0x7f5898155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f5896d9d640 1 -- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 msgr2=0x7f5898143f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:36.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f5896d9d640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 0x7f5898143f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:36.963 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f5896d9d640 1 -- 172.21.15.148:0/2880898423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5880009420 con 0x7f5898153d70 2023-10-30T13:52:36.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589659c640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 0x7f5898143f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:36.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f5896d9d640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5898153d70 0x7f58981556d0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f5890001840 tx=0x7f589000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f588f7fe640 1 -- 172.21.15.148:0/2880898423 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5890012400 con 0x7f5898153d70 2023-10-30T13:52:36.964 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f588f7fe640 1 -- 172.21.15.148:0/2880898423 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5890012560 con 0x7f5898153d70 2023-10-30T13:52:36.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f588f7fe640 1 -- 172.21.15.148:0/2880898423 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5890012910 con 0x7f5898153d70 2023-10-30T13:52:36.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58981543d0 con 0x7f5898153d70 2023-10-30T13:52:36.965 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5898154990 con 0x7f5898153d70 2023-10-30T13:52:36.966 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.954+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5898154eb0 con 0x7f5898153d70 2023-10-30T13:52:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.958+0000 7f588f7fe640 1 -- 172.21.15.148:0/2880898423 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5890012a70 con 0x7f5898153d70 2023-10-30T13:52:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.958+0000 7f588f7fe640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58780664a0 0x7f5878068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.958+0000 7f588f7fe640 1 -- 172.21.15.148:0/2880898423 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f5890084e30 con 0x7f5898153d70 2023-10-30T13:52:36.967 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.958+0000 7f589659c640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58780664a0 0x7f5878068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.958+0000 7f589659c640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58780664a0 0x7f5878068960 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5888000f70 tx=0x7f5888009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:36.968 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:36.962+0000 7f588f7fe640 1 -- 172.21.15.148:0/2880898423 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58900519e0 con 0x7f5898153d70 2023-10-30T13:52:37.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.106+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f589806b4d0 con 0x7f58780664a0 2023-10-30T13:52:37.139 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.134+0000 7f588f7fe640 1 -- 172.21.15.148:0/2880898423 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293778 (secure 0 0 0) 0x7f5888001ca0 con 0x7f58780664a0 2023-10-30T13:52:37.142 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:52:37.144 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:52:37.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58780664a0 msgr2=0x7f5878068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:37.145 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58780664a0 0x7f5878068960 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5888000f70 tx=0x7f5888009040 comp rx=0 tx=0).stop 2023-10-30T13:52:37.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5898153d70 msgr2=0x7f58981556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:37.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5898153d70 0x7f58981556d0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f5890001840 tx=0x7f589000a430 comp rx=0 tx=0).stop 2023-10-30T13:52:37.146 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 shutdown_connections 2023-10-30T13:52:37.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f58780664a0 0x7f5878068960 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:37.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5898153d70 0x7f58981556d0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:37.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f58980c8490 0x7f5898155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:37.147 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 --2- 172.21.15.148:0/2880898423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f589806e470 0x7f5898143f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:37.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 >> 172.21.15.148:0/2880898423 conn(0x7f5898071ea0 msgr2=0x7f5898073650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:37.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 shutdown_connections 2023-10-30T13:52:37.148 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:37.138+0000 7f589c91d640 1 -- 172.21.15.148:0/2880898423 wait complete. 2023-10-30T13:52:37.168 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1326,"stamp":"2023-10-30T13:52:35.204956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2072078858,"num_objects":41001,"num_object_clones":18317,"num_object_copies":82002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41001,"num_whiteouts":14350,"num_read":18168,"num_read_kb":6131546,"num_write":117708,"num_write_kb":3533838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1407,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129228,"ondisk_log_size":129228,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5294748,"kb_used_data":4144796,"kb_used_omap":12,"kb_used_meta":1149875,"kb_avail":749679972,"statfs":{"total":773094113280,"available":767672291328,"internally_reserved":0,"allocated":4244271104,"data_stored":4149478313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1177472568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":89,"apply_latency_ms":89,"commit_latency_ns":89000000,"apply_latency_ns":89000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33183904,"num_objects":400,"num_object_clones":53,"num_object_copies":800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":39,"num_read":4,"num_read_kb":8,"num_write":618,"num_write_kb":32899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004771"},"pg_stats":[{"pgid":"3.3f","version":"67'1473","reported_seq":3440,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.511286+0000","last_change":"2023-10-30T13:21:40.584822+0000","last_active":"2023-10-30T13:52:29.511286+0000","last_peered":"2023-10-30T13:52:29.511286+0000","last_clean":"2023-10-30T13:52:29.511286+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:52:29.511286+0000","last_undegraded":"2023-10-30T13:52:29.511286+0000","last_fullsized":"2023-10-30T13:52:29.511286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:13.082992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014806453000000001,"stat_sum":{"num_bytes":11269044,"num_objects":569,"num_object_clones":260,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":205,"num_read":107,"num_read_kb":121,"num_write":1253,"num_write_kb":17103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1479","reported_seq":3402,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.436731+0000","last_change":"2023-10-30T13:21:40.572202+0000","last_active":"2023-10-30T13:52:29.436731+0000","last_peered":"2023-10-30T13:52:29.436731+0000","last_clean":"2023-10-30T13:52:29.436731+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:52:29.436731+0000","last_undegraded":"2023-10-30T13:52:29.436731+0000","last_fullsized":"2023-10-30T13:52:29.436731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:58.516621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059691099999999997,"stat_sum":{"num_bytes":30975930,"num_objects":597,"num_object_clones":279,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":225,"num_read":82,"num_read_kb":30,"num_write":1226,"num_write_kb":32475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1530","reported_seq":3477,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.437508+0000","last_change":"2023-10-30T13:21:40.528480+0000","last_active":"2023-10-30T13:52:29.437508+0000","last_peered":"2023-10-30T13:52:29.437508+0000","last_clean":"2023-10-30T13:52:29.437508+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:52:29.437508+0000","last_undegraded":"2023-10-30T13:52:29.437508+0000","last_fullsized":"2023-10-30T13:52:29.437508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:12:03.253931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025608710000000002,"stat_sum":{"num_bytes":18704388,"num_objects":617,"num_object_clones":279,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":225,"num_read":106,"num_read_kb":8300,"num_write":1300,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1584","reported_seq":3520,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.433376+0000","last_change":"2023-10-30T13:21:40.567224+0000","last_active":"2023-10-30T13:52:29.433376+0000","last_peered":"2023-10-30T13:52:29.433376+0000","last_clean":"2023-10-30T13:52:29.433376+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:52:29.433376+0000","last_undegraded":"2023-10-30T13:52:29.433376+0000","last_fullsized":"2023-10-30T13:52:29.433376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:32:34.794112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021564890000000001,"stat_sum":{"num_bytes":23329933,"num_objects":630,"num_object_clones":293,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":227,"num_read":112,"num_read_kb":4378,"num_write":1354,"num_write_kb":25006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'1592","reported_seq":3512,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.481064+0000","last_change":"2023-10-30T13:21:40.579191+0000","last_active":"2023-10-30T13:52:29.481064+0000","last_peered":"2023-10-30T13:52:29.481064+0000","last_clean":"2023-10-30T13:52:29.481064+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:52:29.481064+0000","last_undegraded":"2023-10-30T13:52:29.481064+0000","last_fullsized":"2023-10-30T13:52:29.481064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1592,"log_dups_size":0,"ondisk_log_size":1592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:55:10.057619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011808077,"stat_sum":{"num_bytes":16184828,"num_objects":654,"num_object_clones":305,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":251,"num_read":109,"num_read_kb":152,"num_write":1339,"num_write_kb":20879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1459","reported_seq":3406,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.486338+0000","last_change":"2023-10-30T13:21:40.539757+0000","last_active":"2023-10-30T13:52:29.486338+0000","last_peered":"2023-10-30T13:52:29.486338+0000","last_clean":"2023-10-30T13:52:29.486338+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:52:29.486338+0000","last_undegraded":"2023-10-30T13:52:29.486338+0000","last_fullsized":"2023-10-30T13:52:29.486338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:59.387758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01102786,"stat_sum":{"num_bytes":18372801,"num_objects":593,"num_object_clones":277,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":222,"num_read":108,"num_read_kb":4137,"num_write":1217,"num_write_kb":19358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1515","reported_seq":3452,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.469168+0000","last_change":"2023-10-30T13:21:40.580121+0000","last_active":"2023-10-30T13:52:29.469168+0000","last_peered":"2023-10-30T13:52:29.469168+0000","last_clean":"2023-10-30T13:52:29.469168+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:52:29.469168+0000","last_undegraded":"2023-10-30T13:52:29.469168+0000","last_fullsized":"2023-10-30T13:52:29.469168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:42.759583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012047337,"stat_sum":{"num_bytes":9859762,"num_objects":598,"num_object_clones":275,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":211,"num_read":96,"num_read_kb":35,"num_write":1295,"num_write_kb":11866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'1464","reported_seq":3420,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.461917+0000","last_change":"2023-10-30T13:21:40.575154+0000","last_active":"2023-10-30T13:52:29.461917+0000","last_peered":"2023-10-30T13:52:29.461917+0000","last_clean":"2023-10-30T13:52:29.461917+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:52:29.461917+0000","last_undegraded":"2023-10-30T13:52:29.461917+0000","last_fullsized":"2023-10-30T13:52:29.461917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:28.268146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063254060000000004,"stat_sum":{"num_bytes":13694423,"num_objects":577,"num_object_clones":268,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":211,"num_read":108,"num_read_kb":104,"num_write":1228,"num_write_kb":23250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1528","reported_seq":3470,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.522277+0000","last_change":"2023-10-30T13:21:40.584916+0000","last_active":"2023-10-30T13:52:29.522277+0000","last_peered":"2023-10-30T13:52:29.522277+0000","last_clean":"2023-10-30T13:52:29.522277+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:52:29.522277+0000","last_undegraded":"2023-10-30T13:52:29.522277+0000","last_fullsized":"2023-10-30T13:52:29.522277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:08:14.163088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014360304000000001,"stat_sum":{"num_bytes":19134308,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":234,"num_read":109,"num_read_kb":4185,"num_write":1275,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1501","reported_seq":3439,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.604334+0000","last_change":"2023-10-30T13:21:40.580946+0000","last_active":"2023-10-30T13:52:29.604334+0000","last_peered":"2023-10-30T13:52:29.604334+0000","last_clean":"2023-10-30T13:52:29.604334+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:52:29.604334+0000","last_undegraded":"2023-10-30T13:52:29.604334+0000","last_fullsized":"2023-10-30T13:52:29.604334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:29:47.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010277041000000001,"stat_sum":{"num_bytes":19693407,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":208,"num_read":105,"num_read_kb":2976,"num_write":1294,"num_write_kb":29513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1536","reported_seq":3463,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.437626+0000","last_change":"2023-10-30T13:21:40.579672+0000","last_active":"2023-10-30T13:52:29.437626+0000","last_peered":"2023-10-30T13:52:29.437626+0000","last_clean":"2023-10-30T13:52:29.437626+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:52:29.437626+0000","last_undegraded":"2023-10-30T13:52:29.437626+0000","last_fullsized":"2023-10-30T13:52:29.437626+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1536,"log_dups_size":0,"ondisk_log_size":1536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:22:11.088793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01048763,"stat_sum":{"num_bytes":11727836,"num_objects":636,"num_object_clones":292,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":230,"num_read":95,"num_read_kb":27,"num_write":1294,"num_write_kb":13174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1536","reported_seq":3461,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.506181+0000","last_change":"2023-10-30T13:21:40.537641+0000","last_active":"2023-10-30T13:52:29.506181+0000","last_peered":"2023-10-30T13:52:29.506181+0000","last_clean":"2023-10-30T13:52:29.506181+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:52:29.506181+0000","last_undegraded":"2023-10-30T13:52:29.506181+0000","last_fullsized":"2023-10-30T13:52:29.506181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1536,"log_dups_size":0,"ondisk_log_size":1536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:55.899096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085295030000000008,"stat_sum":{"num_bytes":15445695,"num_objects":634,"num_object_clones":293,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":231,"num_read":101,"num_read_kb":102,"num_write":1284,"num_write_kb":21892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1485","reported_seq":3427,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.433015+0000","last_change":"2023-10-30T13:21:40.573274+0000","last_active":"2023-10-30T13:52:29.433015+0000","last_peered":"2023-10-30T13:52:29.433015+0000","last_clean":"2023-10-30T13:52:29.433015+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:52:29.433015+0000","last_undegraded":"2023-10-30T13:52:29.433015+0000","last_fullsized":"2023-10-30T13:52:29.433015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:23:55.087821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004933977,"stat_sum":{"num_bytes":6328355,"num_objects":596,"num_object_clones":272,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":210,"num_read":97,"num_read_kb":4194,"num_write":1284,"num_write_kb":15784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'1559","reported_seq":3506,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.434992+0000","last_change":"2023-10-30T13:21:40.528789+0000","last_active":"2023-10-30T13:52:29.434992+0000","last_peered":"2023-10-30T13:52:29.434992+0000","last_clean":"2023-10-30T13:52:29.434992+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:52:29.434992+0000","last_undegraded":"2023-10-30T13:52:29.434992+0000","last_fullsized":"2023-10-30T13:52:29.434992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1559,"log_dups_size":0,"ondisk_log_size":1559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:55.527282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029909680000000001,"stat_sum":{"num_bytes":31670070,"num_objects":610,"num_object_clones":280,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":227,"num_read":110,"num_read_kb":16437,"num_write":1314,"num_write_kb":33491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1570","reported_seq":3509,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.449057+0000","last_change":"2023-10-30T13:21:40.571731+0000","last_active":"2023-10-30T13:52:29.449057+0000","last_peered":"2023-10-30T13:52:29.449057+0000","last_clean":"2023-10-30T13:52:29.449057+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:52:29.449057+0000","last_undegraded":"2023-10-30T13:52:29.449057+0000","last_fullsized":"2023-10-30T13:52:29.449057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:27:47.649131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058939090000000001,"stat_sum":{"num_bytes":14342306,"num_objects":613,"num_object_clones":284,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":231,"num_read":106,"num_read_kb":2953,"num_write":1333,"num_write_kb":15925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'1763","reported_seq":3621,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.456709+0000","last_change":"2023-10-30T13:21:40.536269+0000","last_active":"2023-10-30T13:52:29.456709+0000","last_peered":"2023-10-30T13:52:29.456709+0000","last_clean":"2023-10-30T13:52:29.456709+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:52:29.456709+0000","last_undegraded":"2023-10-30T13:52:29.456709+0000","last_fullsized":"2023-10-30T13:52:29.456709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1763,"log_dups_size":0,"ondisk_log_size":1763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:06.829080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093243890000000006,"stat_sum":{"num_bytes":10965780,"num_objects":703,"num_object_clones":374,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":231,"num_read":115,"num_read_kb":1694,"num_write":1434,"num_write_kb":17364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'1598","reported_seq":3524,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.494137+0000","last_change":"2023-10-30T13:21:40.539642+0000","last_active":"2023-10-30T13:52:29.494137+0000","last_peered":"2023-10-30T13:52:29.494137+0000","last_clean":"2023-10-30T13:52:29.494137+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:52:29.494137+0000","last_undegraded":"2023-10-30T13:52:29.494137+0000","last_fullsized":"2023-10-30T13:52:29.494137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1598,"log_dups_size":0,"ondisk_log_size":1598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:12:12.694457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011284880000000001,"stat_sum":{"num_bytes":18442682,"num_objects":632,"num_object_clones":291,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":228,"num_read":100,"num_read_kb":8212,"num_write":1340,"num_write_kb":24815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1476","reported_seq":3414,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.431731+0000","last_change":"2023-10-30T13:21:40.567529+0000","last_active":"2023-10-30T13:52:29.431731+0000","last_peered":"2023-10-30T13:52:29.431731+0000","last_clean":"2023-10-30T13:52:29.431731+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:52:29.431731+0000","last_undegraded":"2023-10-30T13:52:29.431731+0000","last_fullsized":"2023-10-30T13:52:29.431731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:45.983709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002295727,"stat_sum":{"num_bytes":22547199,"num_objects":575,"num_object_clones":267,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":207,"num_read":89,"num_read_kb":4127,"num_write":1240,"num_write_kb":24211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1630","reported_seq":3548,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.437005+0000","last_change":"2023-10-30T13:21:40.572640+0000","last_active":"2023-10-30T13:52:29.437005+0000","last_peered":"2023-10-30T13:52:29.437005+0000","last_clean":"2023-10-30T13:52:29.437005+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:52:29.437005+0000","last_undegraded":"2023-10-30T13:52:29.437005+0000","last_fullsized":"2023-10-30T13:52:29.437005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1630,"log_dups_size":0,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:31.079099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061689539999999999,"stat_sum":{"num_bytes":11703479,"num_objects":649,"num_object_clones":303,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":237,"num_read":106,"num_read_kb":7307,"num_write":1383,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1568","reported_seq":3506,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.423306+0000","last_change":"2023-10-30T13:21:40.580324+0000","last_active":"2023-10-30T13:52:29.423306+0000","last_peered":"2023-10-30T13:52:29.423306+0000","last_clean":"2023-10-30T13:52:29.423306+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:52:29.423306+0000","last_undegraded":"2023-10-30T13:52:29.423306+0000","last_fullsized":"2023-10-30T13:52:29.423306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:52.961020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094394720000000008,"stat_sum":{"num_bytes":31771338,"num_objects":621,"num_object_clones":283,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":229,"num_read":103,"num_read_kb":4126,"num_write":1340,"num_write_kb":42793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'1495","reported_seq":3438,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.427169+0000","last_change":"2023-10-30T13:21:40.528854+0000","last_active":"2023-10-30T13:52:29.427169+0000","last_peered":"2023-10-30T13:52:29.427169+0000","last_clean":"2023-10-30T13:52:29.427169+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:52:29.427169+0000","last_undegraded":"2023-10-30T13:52:29.427169+0000","last_fullsized":"2023-10-30T13:52:29.427169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1495,"log_dups_size":0,"ondisk_log_size":1495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:48:55.267335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027436349999999999,"stat_sum":{"num_bytes":6210074,"num_objects":595,"num_object_clones":273,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":228,"num_read":98,"num_read_kb":33,"num_write":1261,"num_write_kb":10653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'1648","reported_seq":3557,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.494800+0000","last_change":"2023-10-30T13:21:40.535360+0000","last_active":"2023-10-30T13:52:29.494800+0000","last_peered":"2023-10-30T13:52:29.494800+0000","last_clean":"2023-10-30T13:52:29.494800+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:52:29.494800+0000","last_undegraded":"2023-10-30T13:52:29.494800+0000","last_fullsized":"2023-10-30T13:52:29.494800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1648,"log_dups_size":0,"ondisk_log_size":1648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:42:13.413880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082328149999999992,"stat_sum":{"num_bytes":28398973,"num_objects":670,"num_object_clones":311,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":248,"num_read":102,"num_read_kb":120,"num_write":1396,"num_write_kb":38562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1544","reported_seq":3481,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.410420+0000","last_change":"2023-10-30T13:21:40.571762+0000","last_active":"2023-10-30T13:52:29.410420+0000","last_peered":"2023-10-30T13:52:29.410420+0000","last_clean":"2023-10-30T13:52:29.410420+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:52:29.410420+0000","last_undegraded":"2023-10-30T13:52:29.410420+0000","last_fullsized":"2023-10-30T13:52:29.410420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1544,"log_dups_size":0,"ondisk_log_size":1544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:29:37.541276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037447470000000001,"stat_sum":{"num_bytes":19407388,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":225,"num_read":103,"num_read_kb":4138,"num_write":1300,"num_write_kb":24714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1586","reported_seq":3526,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.456805+0000","last_change":"2023-10-30T13:21:40.536309+0000","last_active":"2023-10-30T13:52:29.456805+0000","last_peered":"2023-10-30T13:52:29.456805+0000","last_clean":"2023-10-30T13:52:29.456805+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:52:29.456805+0000","last_undegraded":"2023-10-30T13:52:29.456805+0000","last_fullsized":"2023-10-30T13:52:29.456805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1586,"log_dups_size":0,"ondisk_log_size":1586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:04:17.809358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091633449999999998,"stat_sum":{"num_bytes":10552296,"num_objects":616,"num_object_clones":294,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":228,"num_read":115,"num_read_kb":24,"num_write":1342,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1421","reported_seq":3380,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.368873+0000","last_change":"2023-10-30T13:21:40.571783+0000","last_active":"2023-10-30T13:52:29.368873+0000","last_peered":"2023-10-30T13:52:29.368873+0000","last_clean":"2023-10-30T13:52:29.368873+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:52:29.368873+0000","last_undegraded":"2023-10-30T13:52:29.368873+0000","last_fullsized":"2023-10-30T13:52:29.368873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:41.732560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059368210000000001,"stat_sum":{"num_bytes":15196336,"num_objects":556,"num_object_clones":256,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":202,"num_read":99,"num_read_kb":1380,"num_write":1203,"num_write_kb":16993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'1585","reported_seq":3533,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.601776+0000","last_change":"2023-10-30T13:21:40.581115+0000","last_active":"2023-10-30T13:52:29.601776+0000","last_peered":"2023-10-30T13:52:29.601776+0000","last_clean":"2023-10-30T13:52:29.601776+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:52:29.601776+0000","last_undegraded":"2023-10-30T13:52:29.601776+0000","last_fullsized":"2023-10-30T13:52:29.601776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1585,"log_dups_size":0,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:31.199977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099065669999999998,"stat_sum":{"num_bytes":9983325,"num_objects":619,"num_object_clones":286,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":221,"num_read":115,"num_read_kb":68,"num_write":1330,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1543","reported_seq":3486,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.423772+0000","last_change":"2023-10-30T13:21:40.572631+0000","last_active":"2023-10-30T13:52:29.423772+0000","last_peered":"2023-10-30T13:52:29.423772+0000","last_clean":"2023-10-30T13:52:29.423772+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:52:29.423772+0000","last_undegraded":"2023-10-30T13:52:29.423772+0000","last_fullsized":"2023-10-30T13:52:29.423772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1543,"log_dups_size":0,"ondisk_log_size":1543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:53.178210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059382619999999997,"stat_sum":{"num_bytes":23775876,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":236,"num_read":113,"num_read_kb":4145,"num_write":1301,"num_write_kb":26615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1478","reported_seq":3406,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.462288+0000","last_change":"2023-10-30T13:21:40.575712+0000","last_active":"2023-10-30T13:52:29.462288+0000","last_peered":"2023-10-30T13:52:29.462288+0000","last_clean":"2023-10-30T13:52:29.462288+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:52:29.462288+0000","last_undegraded":"2023-10-30T13:52:29.462288+0000","last_fullsized":"2023-10-30T13:52:29.462288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1478,"log_dups_size":0,"ondisk_log_size":1478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:09.905979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053787089999999997,"stat_sum":{"num_bytes":10833053,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":213,"num_read":88,"num_read_kb":3300,"num_write":1256,"num_write_kb":16238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1651","reported_seq":3550,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.443935+0000","last_change":"2023-10-30T13:21:40.529120+0000","last_active":"2023-10-30T13:52:29.443935+0000","last_peered":"2023-10-30T13:52:29.443935+0000","last_clean":"2023-10-30T13:52:29.443935+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:52:29.443935+0000","last_undegraded":"2023-10-30T13:52:29.443935+0000","last_fullsized":"2023-10-30T13:52:29.443935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1651,"log_dups_size":0,"ondisk_log_size":1651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:56.055314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028849399999999999,"stat_sum":{"num_bytes":11995550,"num_objects":688,"num_object_clones":321,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":244,"num_read":100,"num_read_kb":8267,"num_write":1414,"num_write_kb":14306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'1460","reported_seq":3412,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.594950+0000","last_change":"2023-10-30T13:21:40.537362+0000","last_active":"2023-10-30T13:52:29.594950+0000","last_peered":"2023-10-30T13:52:29.594950+0000","last_clean":"2023-10-30T13:52:29.594950+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:52:29.594950+0000","last_undegraded":"2023-10-30T13:52:29.594950+0000","last_fullsized":"2023-10-30T13:52:29.594950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1460,"log_dups_size":0,"ondisk_log_size":1460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:49:05.792581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096547909999999994,"stat_sum":{"num_bytes":15074071,"num_objects":558,"num_object_clones":259,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":221,"num_read":91,"num_read_kb":32,"num_write":1204,"num_write_kb":16754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1673","reported_seq":3613,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.374116+0000","last_change":"2023-10-30T13:21:40.567248+0000","last_active":"2023-10-30T13:52:29.374116+0000","last_peered":"2023-10-30T13:52:29.374116+0000","last_clean":"2023-10-30T13:52:29.374116+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:52:29.374116+0000","last_undegraded":"2023-10-30T13:52:29.374116+0000","last_fullsized":"2023-10-30T13:52:29.374116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:07.337543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018845820000000001,"stat_sum":{"num_bytes":10798567,"num_objects":649,"num_object_clones":300,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":241,"num_read":120,"num_read_kb":27,"num_write":1443,"num_write_kb":17235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1515","reported_seq":3485,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.428657+0000","last_change":"2023-10-30T13:21:40.575268+0000","last_active":"2023-10-30T13:52:29.428657+0000","last_peered":"2023-10-30T13:52:29.428657+0000","last_clean":"2023-10-30T13:52:29.428657+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:52:29.428657+0000","last_undegraded":"2023-10-30T13:52:29.428657+0000","last_fullsized":"2023-10-30T13:52:29.428657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:50:11.469301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075239010000000004,"stat_sum":{"num_bytes":15423893,"num_objects":576,"num_object_clones":259,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":203,"num_read":109,"num_read_kb":9552,"num_write":1308,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'1559","reported_seq":3503,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.425336+0000","last_change":"2023-10-30T13:21:40.537743+0000","last_active":"2023-10-30T13:52:29.425336+0000","last_peered":"2023-10-30T13:52:29.425336+0000","last_clean":"2023-10-30T13:52:29.425336+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:52:29.425336+0000","last_undegraded":"2023-10-30T13:52:29.425336+0000","last_fullsized":"2023-10-30T13:52:29.425336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1559,"log_dups_size":0,"ondisk_log_size":1559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:01:00.836625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007356301,"stat_sum":{"num_bytes":40089180,"num_objects":622,"num_object_clones":287,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":230,"num_read":115,"num_read_kb":33,"num_write":1312,"num_write_kb":42997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'604","reported_seq":832,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.233111+0000","last_change":"2023-10-30T13:10:33.714515+0000","last_active":"2023-10-30T13:52:19.233111+0000","last_peered":"2023-10-30T13:52:19.233111+0000","last_clean":"2023-10-30T13:52:19.233111+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:52:19.233111+0000","last_undegraded":"2023-10-30T13:52:19.233111+0000","last_fullsized":"2023-10-30T13:52:19.233111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.438300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":252,"num_read_kb":249243,"num_write":662,"num_write_kb":74846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1560","reported_seq":3487,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.433209+0000","last_change":"2023-10-30T13:21:40.529443+0000","last_active":"2023-10-30T13:52:29.433209+0000","last_peered":"2023-10-30T13:52:29.433209+0000","last_clean":"2023-10-30T13:52:29.433209+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:52:29.433209+0000","last_undegraded":"2023-10-30T13:52:29.433209+0000","last_fullsized":"2023-10-30T13:52:29.433209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:39.924469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372290000000001,"stat_sum":{"num_bytes":18713903,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":229,"num_read":108,"num_read_kb":4667,"num_write":1329,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1126","reported_seq":1449,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:15.211697+0000","last_change":"2023-10-30T13:10:33.715547+0000","last_active":"2023-10-30T13:52:15.211697+0000","last_peered":"2023-10-30T13:52:15.211697+0000","last_clean":"2023-10-30T13:52:15.211697+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:52:15.211697+0000","last_undegraded":"2023-10-30T13:52:15.211697+0000","last_fullsized":"2023-10-30T13:52:15.211697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1126,"log_dups_size":0,"ondisk_log_size":1126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:08.065512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25293488,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":404,"num_read_kb":309242,"num_write":1229,"num_write_kb":84643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'746","reported_seq":1124,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.654761+0000","last_change":"2023-10-30T13:10:33.713387+0000","last_active":"2023-10-30T13:52:25.654761+0000","last_peered":"2023-10-30T13:52:25.654761+0000","last_clean":"2023-10-30T13:52:25.654761+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:52:25.654761+0000","last_undegraded":"2023-10-30T13:52:25.654761+0000","last_fullsized":"2023-10-30T13:52:25.654761+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":746,"log_dups_size":0,"ondisk_log_size":746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397151,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":403,"num_read_kb":153383,"num_write":819,"num_write_kb":55113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'1791","reported_seq":3708,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.447675+0000","last_change":"2023-10-30T13:21:40.575255+0000","last_active":"2023-10-30T13:52:29.447675+0000","last_peered":"2023-10-30T13:52:29.447675+0000","last_clean":"2023-10-30T13:52:29.447675+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:52:29.447675+0000","last_undegraded":"2023-10-30T13:52:29.447675+0000","last_fullsized":"2023-10-30T13:52:29.447675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1791,"log_dups_size":0,"ondisk_log_size":1791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:20.571707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064075410000000001,"stat_sum":{"num_bytes":11666119,"num_objects":729,"num_object_clones":341,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":261,"num_read":141,"num_read_kb":4175,"num_write":1520,"num_write_kb":22588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1399","reported_seq":1668,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.266715+0000","last_change":"2023-10-30T13:10:33.711567+0000","last_active":"2023-10-30T13:52:24.266715+0000","last_peered":"2023-10-30T13:52:24.266715+0000","last_clean":"2023-10-30T13:52:24.266715+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:52:24.266715+0000","last_undegraded":"2023-10-30T13:52:24.266715+0000","last_fullsized":"2023-10-30T13:52:24.266715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1399,"log_dups_size":0,"ondisk_log_size":1399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:46.879933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":354,"num_read_kb":270680,"num_write":1485,"num_write_kb":75457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1514","reported_seq":3447,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.440687+0000","last_change":"2023-10-30T13:21:40.571518+0000","last_active":"2023-10-30T13:52:29.440687+0000","last_peered":"2023-10-30T13:52:29.440687+0000","last_clean":"2023-10-30T13:52:29.440687+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:52:29.440687+0000","last_undegraded":"2023-10-30T13:52:29.440687+0000","last_fullsized":"2023-10-30T13:52:29.440687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1514,"log_dups_size":0,"ondisk_log_size":1514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:19:07.205330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058964639999999997,"stat_sum":{"num_bytes":19507019,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":240,"num_read":105,"num_read_kb":5362,"num_write":1250,"num_write_kb":24752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'955","reported_seq":1191,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.152943+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:52:29.152943+0000","last_peered":"2023-10-30T13:52:29.152943+0000","last_clean":"2023-10-30T13:52:29.152943+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:52:29.152943+0000","last_undegraded":"2023-10-30T13:52:29.152943+0000","last_fullsized":"2023-10-30T13:52:29.152943+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:22.749899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":313,"num_read_kb":141404,"num_write":1076,"num_write_kb":63727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1493","reported_seq":3446,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.402962+0000","last_change":"2023-10-30T13:21:40.567796+0000","last_active":"2023-10-30T13:52:29.402962+0000","last_peered":"2023-10-30T13:52:29.402962+0000","last_clean":"2023-10-30T13:52:29.402962+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:52:29.402962+0000","last_undegraded":"2023-10-30T13:52:29.402962+0000","last_fullsized":"2023-10-30T13:52:29.402962+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1493,"log_dups_size":0,"ondisk_log_size":1493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:08:51.743716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025817129999999998,"stat_sum":{"num_bytes":23539910,"num_objects":588,"num_object_clones":267,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":208,"num_read":106,"num_read_kb":8313,"num_write":1266,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"67'1122","reported_seq":1340,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:17.672748+0000","last_change":"2023-10-30T13:10:33.717679+0000","last_active":"2023-10-30T13:52:17.672748+0000","last_peered":"2023-10-30T13:52:17.672748+0000","last_clean":"2023-10-30T13:52:17.672748+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:52:17.672748+0000","last_undegraded":"2023-10-30T13:52:17.672748+0000","last_fullsized":"2023-10-30T13:52:17.672748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1122,"log_dups_size":0,"ondisk_log_size":1122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:02.841118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":240,"num_read_kb":201173,"num_write":1196,"num_write_kb":66729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1557","reported_seq":3468,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.485273+0000","last_change":"2023-10-30T13:21:40.528173+0000","last_active":"2023-10-30T13:52:29.485273+0000","last_peered":"2023-10-30T13:52:29.485273+0000","last_clean":"2023-10-30T13:52:29.485273+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:52:29.485273+0000","last_undegraded":"2023-10-30T13:52:29.485273+0000","last_fullsized":"2023-10-30T13:52:29.485273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:50:03.879621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022874710000000001,"stat_sum":{"num_bytes":16137402,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":233,"num_read":91,"num_read_kb":34,"num_write":1322,"num_write_kb":18013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1315","reported_seq":1556,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.212373+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:52:24.212373+0000","last_peered":"2023-10-30T13:52:24.212373+0000","last_clean":"2023-10-30T13:52:24.212373+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:52:24.212373+0000","last_undegraded":"2023-10-30T13:52:24.212373+0000","last_fullsized":"2023-10-30T13:52:24.212373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:24:36.150105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20972192,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":307,"num_read_kb":165804,"num_write":1420,"num_write_kb":50966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1574","reported_seq":3506,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.498368+0000","last_change":"2023-10-30T13:21:40.538439+0000","last_active":"2023-10-30T13:52:29.498368+0000","last_peered":"2023-10-30T13:52:29.498368+0000","last_clean":"2023-10-30T13:52:29.498368+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:52:29.498368+0000","last_undegraded":"2023-10-30T13:52:29.498368+0000","last_fullsized":"2023-10-30T13:52:29.498368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:36:06.776976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087798259999999993,"stat_sum":{"num_bytes":32677996,"num_objects":638,"num_object_clones":293,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":232,"num_read":108,"num_read_kb":10951,"num_write":1342,"num_write_kb":38819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'936","reported_seq":1254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.110194+0000","last_change":"2023-10-30T13:10:33.724357+0000","last_active":"2023-10-30T13:52:24.110194+0000","last_peered":"2023-10-30T13:52:24.110194+0000","last_clean":"2023-10-30T13:52:24.110194+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:52:24.110194+0000","last_undegraded":"2023-10-30T13:52:24.110194+0000","last_fullsized":"2023-10-30T13:52:24.110194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:04.620177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":413,"num_read_kb":281405,"num_write":1058,"num_write_kb":80201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1512","reported_seq":3427,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.485596+0000","last_change":"2023-10-30T13:21:40.583356+0000","last_active":"2023-10-30T13:52:29.485596+0000","last_peered":"2023-10-30T13:52:29.485596+0000","last_clean":"2023-10-30T13:52:29.485596+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:52:29.485596+0000","last_undegraded":"2023-10-30T13:52:29.485596+0000","last_fullsized":"2023-10-30T13:52:29.485596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:46.232350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014787140000000001,"stat_sum":{"num_bytes":19072740,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":238,"num_read":87,"num_read_kb":102,"num_write":1252,"num_write_kb":21862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'1084","reported_seq":1410,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:16.582275+0000","last_change":"2023-10-30T13:10:33.695702+0000","last_active":"2023-10-30T13:52:16.582275+0000","last_peered":"2023-10-30T13:52:16.582275+0000","last_clean":"2023-10-30T13:52:16.582275+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:52:16.582275+0000","last_undegraded":"2023-10-30T13:52:16.582275+0000","last_fullsized":"2023-10-30T13:52:16.582275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:25:59.172251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748770,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":476,"num_read_kb":203988,"num_write":1202,"num_write_kb":67245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1509","reported_seq":3459,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.429316+0000","last_change":"2023-10-30T13:21:40.528316+0000","last_active":"2023-10-30T13:52:29.429316+0000","last_peered":"2023-10-30T13:52:29.429316+0000","last_clean":"2023-10-30T13:52:29.429316+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:52:29.429316+0000","last_undegraded":"2023-10-30T13:52:29.429316+0000","last_fullsized":"2023-10-30T13:52:29.429316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1509,"log_dups_size":0,"ondisk_log_size":1509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:08:42.998206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024076599999999998,"stat_sum":{"num_bytes":14342228,"num_objects":584,"num_object_clones":268,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":211,"num_read":101,"num_read_kb":4159,"num_write":1284,"num_write_kb":24386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'575","reported_seq":756,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:21.604458+0000","last_change":"2023-10-30T13:10:33.708304+0000","last_active":"2023-10-30T13:52:21.604458+0000","last_peered":"2023-10-30T13:52:21.604458+0000","last_clean":"2023-10-30T13:52:21.604458+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:52:21.604458+0000","last_undegraded":"2023-10-30T13:52:21.604458+0000","last_fullsized":"2023-10-30T13:52:21.604458+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:57:39.196241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":203,"num_read_kb":83137,"num_write":663,"num_write_kb":45991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1604","reported_seq":3579,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.475270+0000","last_change":"2023-10-30T13:21:40.579925+0000","last_active":"2023-10-30T13:52:29.475270+0000","last_peered":"2023-10-30T13:52:29.475270+0000","last_clean":"2023-10-30T13:52:29.475270+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:52:29.475270+0000","last_undegraded":"2023-10-30T13:52:29.475270+0000","last_fullsized":"2023-10-30T13:52:29.475270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1604,"log_dups_size":0,"ondisk_log_size":1604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:39.495586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011943199999999999,"stat_sum":{"num_bytes":18918293,"num_objects":634,"num_object_clones":295,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":241,"num_read":154,"num_read_kb":114,"num_write":1356,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'500","reported_seq":732,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.241593+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:52:19.241593+0000","last_peered":"2023-10-30T13:52:19.241593+0000","last_clean":"2023-10-30T13:52:19.241593+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:52:19.241593+0000","last_undegraded":"2023-10-30T13:52:19.241593+0000","last_fullsized":"2023-10-30T13:52:19.241593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:31:01.723213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":289,"num_read_kb":121284,"num_write":555,"num_write_kb":58687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1640","reported_seq":3521,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.428452+0000","last_change":"2023-10-30T13:21:40.576918+0000","last_active":"2023-10-30T13:52:29.428452+0000","last_peered":"2023-10-30T13:52:29.428452+0000","last_clean":"2023-10-30T13:52:29.428452+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:52:29.428452+0000","last_undegraded":"2023-10-30T13:52:29.428452+0000","last_fullsized":"2023-10-30T13:52:29.428452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1640,"log_dups_size":0,"ondisk_log_size":1640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:33.447713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086123929999999994,"stat_sum":{"num_bytes":19792070,"num_objects":655,"num_object_clones":349,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":204,"num_read":113,"num_read_kb":142,"num_write":1348,"num_write_kb":24080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'961","reported_seq":1240,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:15.360504+0000","last_change":"2023-10-30T13:10:33.698029+0000","last_active":"2023-10-30T13:52:15.360504+0000","last_peered":"2023-10-30T13:52:15.360504+0000","last_clean":"2023-10-30T13:52:15.360504+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:52:15.360504+0000","last_undegraded":"2023-10-30T13:52:15.360504+0000","last_fullsized":"2023-10-30T13:52:15.360504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.865231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36678727,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":465,"num_read_kb":133831,"num_write":1108,"num_write_kb":54499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1563","reported_seq":3484,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.477290+0000","last_change":"2023-10-30T13:21:40.583214+0000","last_active":"2023-10-30T13:52:29.477290+0000","last_peered":"2023-10-30T13:52:29.477290+0000","last_clean":"2023-10-30T13:52:29.477290+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:52:29.477290+0000","last_undegraded":"2023-10-30T13:52:29.477290+0000","last_fullsized":"2023-10-30T13:52:29.477290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:27:22.371097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015028952999999999,"stat_sum":{"num_bytes":21253392,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":236,"num_read":95,"num_read_kb":12339,"num_write":1306,"num_write_kb":30718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'639","reported_seq":864,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.467326+0000","last_change":"2023-10-30T13:10:33.698270+0000","last_active":"2023-10-30T13:52:23.467326+0000","last_peered":"2023-10-30T13:52:23.467326+0000","last_clean":"2023-10-30T13:52:23.467326+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:52:23.467326+0000","last_undegraded":"2023-10-30T13:52:23.467326+0000","last_fullsized":"2023-10-30T13:52:23.467326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:28:12.335582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165846,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":249,"num_read_kb":148701,"num_write":778,"num_write_kb":54421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1567","reported_seq":3497,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.484126+0000","last_change":"2023-10-30T13:21:40.579365+0000","last_active":"2023-10-30T13:52:29.484126+0000","last_peered":"2023-10-30T13:52:29.484126+0000","last_clean":"2023-10-30T13:52:29.484126+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:52:29.484126+0000","last_undegraded":"2023-10-30T13:52:29.484126+0000","last_fullsized":"2023-10-30T13:52:29.484126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:51:50.149626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011682140000000001,"stat_sum":{"num_bytes":26941199,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":237,"num_read":103,"num_read_kb":4114,"num_write":1331,"num_write_kb":32609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'1052","reported_seq":1328,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.508023+0000","last_change":"2023-10-30T13:10:33.694762+0000","last_active":"2023-10-30T13:52:24.508023+0000","last_peered":"2023-10-30T13:52:24.508023+0000","last_clean":"2023-10-30T13:52:24.508023+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:52:24.508023+0000","last_undegraded":"2023-10-30T13:52:24.508023+0000","last_fullsized":"2023-10-30T13:52:24.508023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1052,"log_dups_size":0,"ondisk_log_size":1052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:37.609259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":381,"num_read_kb":250673,"num_write":1157,"num_write_kb":87754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1540","reported_seq":3482,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.519115+0000","last_change":"2023-10-30T13:21:40.585506+0000","last_active":"2023-10-30T13:52:29.519115+0000","last_peered":"2023-10-30T13:52:29.519115+0000","last_clean":"2023-10-30T13:52:29.519115+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:52:29.519115+0000","last_undegraded":"2023-10-30T13:52:29.519115+0000","last_fullsized":"2023-10-30T13:52:29.519115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:06.635146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014794043,"stat_sum":{"num_bytes":18888621,"num_objects":614,"num_object_clones":283,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":224,"num_read":107,"num_read_kb":4188,"num_write":1306,"num_write_kb":24240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'1228","reported_seq":1479,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.167394+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:52:24.167394+0000","last_peered":"2023-10-30T13:52:24.167394+0000","last_clean":"2023-10-30T13:52:24.167394+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:52:24.167394+0000","last_undegraded":"2023-10-30T13:52:24.167394+0000","last_fullsized":"2023-10-30T13:52:24.167394+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1228,"log_dups_size":0,"ondisk_log_size":1228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:06.633719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364039,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":288,"num_read_kb":200539,"num_write":1326,"num_write_kb":71054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'1581","reported_seq":3534,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.476807+0000","last_change":"2023-10-30T13:21:40.538748+0000","last_active":"2023-10-30T13:52:29.476807+0000","last_peered":"2023-10-30T13:52:29.476807+0000","last_clean":"2023-10-30T13:52:29.476807+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:52:29.476807+0000","last_undegraded":"2023-10-30T13:52:29.476807+0000","last_fullsized":"2023-10-30T13:52:29.476807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:16:15.511609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010788236,"stat_sum":{"num_bytes":10484856,"num_objects":600,"num_object_clones":281,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":220,"num_read":116,"num_read_kb":28,"num_write":1335,"num_write_kb":13295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'778","reported_seq":1025,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.156747+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:52:29.156747+0000","last_peered":"2023-10-30T13:52:29.156747+0000","last_clean":"2023-10-30T13:52:29.156747+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:52:29.156747+0000","last_undegraded":"2023-10-30T13:52:29.156747+0000","last_fullsized":"2023-10-30T13:52:29.156747+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:45:51.434837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":322,"num_read_kb":174067,"num_write":896,"num_write_kb":62874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1542","reported_seq":3473,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.483465+0000","last_change":"2023-10-30T13:21:40.583977+0000","last_active":"2023-10-30T13:52:29.483465+0000","last_peered":"2023-10-30T13:52:29.483465+0000","last_clean":"2023-10-30T13:52:29.483465+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:52:29.483465+0000","last_undegraded":"2023-10-30T13:52:29.483465+0000","last_fullsized":"2023-10-30T13:52:29.483465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:18.653967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015268978000000001,"stat_sum":{"num_bytes":11043591,"num_objects":616,"num_object_clones":291,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":221,"num_read":104,"num_read_kb":3707,"num_write":1322,"num_write_kb":16509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":232,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T13:50:37.450196+0000","last_change":"2023-10-30T13:10:24.437540+0000","last_active":"2023-10-30T13:50:37.450196+0000","last_peered":"2023-10-30T13:50:37.450196+0000","last_clean":"2023-10-30T13:50:37.450196+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:50:37.450196+0000","last_undegraded":"2023-10-30T13:50:37.450196+0000","last_fullsized":"2023-10-30T13:50:37.450196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:22.508530+0000","last_clean_scrub_stamp":"2023-10-30T13:10:22.508530+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:17.842767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'621","reported_seq":829,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.128052+0000","last_change":"2023-10-30T13:10:33.708305+0000","last_active":"2023-10-30T13:52:29.128052+0000","last_peered":"2023-10-30T13:52:29.128052+0000","last_clean":"2023-10-30T13:52:29.128052+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:52:29.128052+0000","last_undegraded":"2023-10-30T13:52:29.128052+0000","last_fullsized":"2023-10-30T13:52:29.128052+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:13.346585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360178,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":232,"num_read_kb":148904,"num_write":701,"num_write_kb":62706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1545","reported_seq":3458,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.465643+0000","last_change":"2023-10-30T13:21:40.574205+0000","last_active":"2023-10-30T13:52:29.465643+0000","last_peered":"2023-10-30T13:52:29.465643+0000","last_clean":"2023-10-30T13:52:29.465643+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:52:29.465643+0000","last_undegraded":"2023-10-30T13:52:29.465643+0000","last_fullsized":"2023-10-30T13:52:29.465643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1545,"log_dups_size":0,"ondisk_log_size":1545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:45:44.062637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057314540000000004,"stat_sum":{"num_bytes":22780711,"num_objects":633,"num_object_clones":299,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":242,"num_read":92,"num_read_kb":12,"num_write":1293,"num_write_kb":28237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"67'707","reported_seq":938,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.141156+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:52:29.141156+0000","last_peered":"2023-10-30T13:52:29.141156+0000","last_clean":"2023-10-30T13:52:29.141156+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:52:29.141156+0000","last_undegraded":"2023-10-30T13:52:29.141156+0000","last_fullsized":"2023-10-30T13:52:29.141156+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":707,"log_dups_size":0,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:07.486866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":256,"num_read_kb":247261,"num_write":740,"num_write_kb":70569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'1468","reported_seq":3405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.435146+0000","last_change":"2023-10-30T13:21:40.528051+0000","last_active":"2023-10-30T13:52:29.435146+0000","last_peered":"2023-10-30T13:52:29.435146+0000","last_clean":"2023-10-30T13:52:29.435146+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:52:29.435146+0000","last_undegraded":"2023-10-30T13:52:29.435146+0000","last_fullsized":"2023-10-30T13:52:29.435146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:08:49.848069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002406114,"stat_sum":{"num_bytes":14184993,"num_objects":568,"num_object_clones":263,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":210,"num_read":80,"num_read_kb":26,"num_write":1239,"num_write_kb":16314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'2040","reported_seq":3405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.224696+0000","last_change":"2023-10-30T13:10:33.695393+0000","last_active":"2023-10-30T13:52:25.224696+0000","last_peered":"2023-10-30T13:52:25.224696+0000","last_clean":"2023-10-30T13:52:25.224696+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:52:25.224696+0000","last_undegraded":"2023-10-30T13:52:25.224696+0000","last_fullsized":"2023-10-30T13:52:25.224696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:26.543038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41670866,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":1534,"num_read_kb":188566,"num_write":2090,"num_write_kb":83865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1527","reported_seq":3468,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.433857+0000","last_change":"2023-10-30T13:21:40.568287+0000","last_active":"2023-10-30T13:52:29.433857+0000","last_peered":"2023-10-30T13:52:29.433857+0000","last_clean":"2023-10-30T13:52:29.433857+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:52:29.433857+0000","last_undegraded":"2023-10-30T13:52:29.433857+0000","last_fullsized":"2023-10-30T13:52:29.433857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:16.792299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002908281,"stat_sum":{"num_bytes":22417951,"num_objects":612,"num_object_clones":284,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":221,"num_read":105,"num_read_kb":44,"num_write":1298,"num_write_kb":23585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'949","reported_seq":1224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.064958+0000","last_change":"2023-10-30T13:10:33.708406+0000","last_active":"2023-10-30T13:52:29.064958+0000","last_peered":"2023-10-30T13:52:29.064958+0000","last_clean":"2023-10-30T13:52:29.064958+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:52:29.064958+0000","last_undegraded":"2023-10-30T13:52:29.064958+0000","last_fullsized":"2023-10-30T13:52:29.064958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:25.796779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":355,"num_read_kb":252215,"num_write":1076,"num_write_kb":75665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'1536","reported_seq":3490,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.425902+0000","last_change":"2023-10-30T13:21:40.536723+0000","last_active":"2023-10-30T13:52:29.425902+0000","last_peered":"2023-10-30T13:52:29.425902+0000","last_clean":"2023-10-30T13:52:29.425902+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:52:29.425902+0000","last_undegraded":"2023-10-30T13:52:29.425902+0000","last_fullsized":"2023-10-30T13:52:29.425902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1536,"log_dups_size":0,"ondisk_log_size":1536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:33:25.077240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007710705,"stat_sum":{"num_bytes":23751577,"num_objects":575,"num_object_clones":266,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":199,"num_read":100,"num_read_kb":489,"num_write":1325,"num_write_kb":27587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'943","reported_seq":1183,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:18.547312+0000","last_change":"2023-10-30T13:10:33.688299+0000","last_active":"2023-10-30T13:52:18.547312+0000","last_peered":"2023-10-30T13:52:18.547312+0000","last_clean":"2023-10-30T13:52:18.547312+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:52:18.547312+0000","last_undegraded":"2023-10-30T13:52:18.547312+0000","last_fullsized":"2023-10-30T13:52:18.547312+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:17.341528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":294,"num_read_kb":227180,"num_write":1037,"num_write_kb":87579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1598","reported_seq":3508,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.462060+0000","last_change":"2023-10-30T13:21:40.539081+0000","last_active":"2023-10-30T13:52:29.462060+0000","last_peered":"2023-10-30T13:52:29.462060+0000","last_clean":"2023-10-30T13:52:29.462060+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:52:29.462060+0000","last_undegraded":"2023-10-30T13:52:29.462060+0000","last_fullsized":"2023-10-30T13:52:29.462060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1598,"log_dups_size":0,"ondisk_log_size":1598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:36.291322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010719574000000001,"stat_sum":{"num_bytes":35538242,"num_objects":650,"num_object_clones":301,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":239,"num_read":91,"num_read_kb":31,"num_write":1347,"num_write_kb":41412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'787","reported_seq":1012,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.066091+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:52:29.066091+0000","last_peered":"2023-10-30T13:52:29.066091+0000","last_clean":"2023-10-30T13:52:29.066091+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:52:29.066091+0000","last_undegraded":"2023-10-30T13:52:29.066091+0000","last_fullsized":"2023-10-30T13:52:29.066091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:48.762636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":258,"num_read_kb":177173,"num_write":845,"num_write_kb":62523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1551","reported_seq":3477,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.395612+0000","last_change":"2023-10-30T13:21:40.572129+0000","last_active":"2023-10-30T13:52:29.395612+0000","last_peered":"2023-10-30T13:52:29.395612+0000","last_clean":"2023-10-30T13:52:29.395612+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:52:29.395612+0000","last_undegraded":"2023-10-30T13:52:29.395612+0000","last_fullsized":"2023-10-30T13:52:29.395612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:23.250683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059276069999999997,"stat_sum":{"num_bytes":23389062,"num_objects":631,"num_object_clones":294,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":232,"num_read":100,"num_read_kb":25,"num_write":1313,"num_write_kb":24315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'915","reported_seq":1154,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.122770+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:52:24.122770+0000","last_peered":"2023-10-30T13:52:24.122770+0000","last_clean":"2023-10-30T13:52:24.122770+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:52:24.122770+0000","last_undegraded":"2023-10-30T13:52:24.122770+0000","last_fullsized":"2023-10-30T13:52:24.122770+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":915,"log_dups_size":0,"ondisk_log_size":915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:32:11.419519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":310,"num_read_kb":109156,"num_write":1494,"num_write_kb":61821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1421","reported_seq":3373,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.481362+0000","last_change":"2023-10-30T13:21:40.576235+0000","last_active":"2023-10-30T13:52:29.481362+0000","last_peered":"2023-10-30T13:52:29.481362+0000","last_clean":"2023-10-30T13:52:29.481362+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:52:29.481362+0000","last_undegraded":"2023-10-30T13:52:29.481362+0000","last_fullsized":"2023-10-30T13:52:29.481362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:30.114376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056940879999999999,"stat_sum":{"num_bytes":23042560,"num_objects":571,"num_object_clones":259,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":201,"num_read":92,"num_read_kb":75,"num_write":1201,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'1096","reported_seq":1339,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:17.691407+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:52:17.691407+0000","last_peered":"2023-10-30T13:52:17.691407+0000","last_clean":"2023-10-30T13:52:17.691407+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:52:17.691407+0000","last_undegraded":"2023-10-30T13:52:17.691407+0000","last_fullsized":"2023-10-30T13:52:17.691407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1096,"log_dups_size":0,"ondisk_log_size":1096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:30:09.988865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":342,"num_read_kb":120967,"num_write":1231,"num_write_kb":51147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1445","reported_seq":3406,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.435924+0000","last_change":"2023-10-30T13:21:40.529108+0000","last_active":"2023-10-30T13:52:29.435924+0000","last_peered":"2023-10-30T13:52:29.435924+0000","last_clean":"2023-10-30T13:52:29.435924+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:52:29.435924+0000","last_undegraded":"2023-10-30T13:52:29.435924+0000","last_fullsized":"2023-10-30T13:52:29.435924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:16:51.856467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029926589999999999,"stat_sum":{"num_bytes":30813966,"num_objects":572,"num_object_clones":263,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":205,"num_read":105,"num_read_kb":87,"num_write":1219,"num_write_kb":33525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'421","reported_seq":607,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:23.800787+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:52:23.800787+0000","last_peered":"2023-10-30T13:52:23.800787+0000","last_clean":"2023-10-30T13:52:23.800787+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:52:23.800787+0000","last_undegraded":"2023-10-30T13:52:23.800787+0000","last_fullsized":"2023-10-30T13:52:23.800787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:58.789193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":209,"num_read_kb":100741,"num_write":482,"num_write_kb":29419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1546","reported_seq":3500,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.446635+0000","last_change":"2023-10-30T13:21:40.533380+0000","last_active":"2023-10-30T13:52:29.446635+0000","last_peered":"2023-10-30T13:52:29.446635+0000","last_clean":"2023-10-30T13:52:29.446635+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:52:29.446635+0000","last_undegraded":"2023-10-30T13:52:29.446635+0000","last_fullsized":"2023-10-30T13:52:29.446635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:46.427030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057673200000000003,"stat_sum":{"num_bytes":13994934,"num_objects":595,"num_object_clones":279,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":215,"num_read":114,"num_read_kb":29,"num_write":1329,"num_write_kb":19694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'1215","reported_seq":1429,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.310060+0000","last_change":"2023-10-30T13:10:33.699461+0000","last_active":"2023-10-30T13:52:29.310060+0000","last_peered":"2023-10-30T13:52:29.310060+0000","last_clean":"2023-10-30T13:52:29.310060+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:52:29.310060+0000","last_undegraded":"2023-10-30T13:52:29.310060+0000","last_fullsized":"2023-10-30T13:52:29.310060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:18.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23082160,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":251,"num_read_kb":99282,"num_write":1284,"num_write_kb":40532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1556","reported_seq":3509,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.445212+0000","last_change":"2023-10-30T13:21:40.575380+0000","last_active":"2023-10-30T13:52:29.445212+0000","last_peered":"2023-10-30T13:52:29.445212+0000","last_clean":"2023-10-30T13:52:29.445212+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:52:29.445212+0000","last_undegraded":"2023-10-30T13:52:29.445212+0000","last_fullsized":"2023-10-30T13:52:29.445212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1556,"log_dups_size":0,"ondisk_log_size":1556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:12.098739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063570620000000001,"stat_sum":{"num_bytes":16293763,"num_objects":612,"num_object_clones":286,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":233,"num_read":120,"num_read_kb":127,"num_write":1302,"num_write_kb":21658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'901","reported_seq":1127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.175742+0000","last_change":"2023-10-30T13:10:33.692122+0000","last_active":"2023-10-30T13:52:24.175742+0000","last_peered":"2023-10-30T13:52:24.175742+0000","last_clean":"2023-10-30T13:52:24.175742+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:52:24.175742+0000","last_undegraded":"2023-10-30T13:52:24.175742+0000","last_fullsized":"2023-10-30T13:52:24.175742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:50.127248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":272,"num_read_kb":200216,"num_write":1134,"num_write_kb":59039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1437","reported_seq":3386,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.437202+0000","last_change":"2023-10-30T13:21:40.575515+0000","last_active":"2023-10-30T13:52:29.437202+0000","last_peered":"2023-10-30T13:52:29.437202+0000","last_clean":"2023-10-30T13:52:29.437202+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:52:29.437202+0000","last_undegraded":"2023-10-30T13:52:29.437202+0000","last_fullsized":"2023-10-30T13:52:29.437202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1437,"log_dups_size":0,"ondisk_log_size":1437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:10:20.385719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060201619999999999,"stat_sum":{"num_bytes":10033117,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":215,"num_read":92,"num_read_kb":24,"num_write":1204,"num_write_kb":13175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'971","reported_seq":1226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.131441+0000","last_change":"2023-10-30T13:10:33.721412+0000","last_active":"2023-10-30T13:52:29.131441+0000","last_peered":"2023-10-30T13:52:29.131441+0000","last_clean":"2023-10-30T13:52:29.131441+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:52:29.131441+0000","last_undegraded":"2023-10-30T13:52:29.131441+0000","last_fullsized":"2023-10-30T13:52:29.131441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:55.127023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":318,"num_read_kb":202944,"num_write":1064,"num_write_kb":67614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1468","reported_seq":3416,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.420345+0000","last_change":"2023-10-30T13:21:40.575093+0000","last_active":"2023-10-30T13:52:29.420345+0000","last_peered":"2023-10-30T13:52:29.420345+0000","last_clean":"2023-10-30T13:52:29.420345+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:52:29.420345+0000","last_undegraded":"2023-10-30T13:52:29.420345+0000","last_fullsized":"2023-10-30T13:52:29.420345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:55:44.047461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079253199999999996,"stat_sum":{"num_bytes":6591050,"num_objects":558,"num_object_clones":254,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":200,"num_read":83,"num_read_kb":63,"num_write":1255,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1498","reported_seq":3442,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.463196+0000","last_change":"2023-10-30T13:21:40.538321+0000","last_active":"2023-10-30T13:52:29.463196+0000","last_peered":"2023-10-30T13:52:29.463196+0000","last_clean":"2023-10-30T13:52:29.463196+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:52:29.463196+0000","last_undegraded":"2023-10-30T13:52:29.463196+0000","last_fullsized":"2023-10-30T13:52:29.463196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:04.667732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010603414,"stat_sum":{"num_bytes":9104816,"num_objects":585,"num_object_clones":276,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":204,"num_read":104,"num_read_kb":44,"num_write":1285,"num_write_kb":14447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'1003","reported_seq":1257,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.128560+0000","last_change":"2023-10-30T13:10:33.721856+0000","last_active":"2023-10-30T13:52:29.128560+0000","last_peered":"2023-10-30T13:52:29.128560+0000","last_clean":"2023-10-30T13:52:29.128560+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:52:29.128560+0000","last_undegraded":"2023-10-30T13:52:29.128560+0000","last_fullsized":"2023-10-30T13:52:29.128560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1003,"log_dups_size":0,"ondisk_log_size":1003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:08.805834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":337,"num_read_kb":187117,"num_write":1200,"num_write_kb":47577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1627","reported_seq":3541,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.488786+0000","last_change":"2023-10-30T13:21:40.537202+0000","last_active":"2023-10-30T13:52:29.488786+0000","last_peered":"2023-10-30T13:52:29.488786+0000","last_clean":"2023-10-30T13:52:29.488786+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:52:29.488786+0000","last_undegraded":"2023-10-30T13:52:29.488786+0000","last_fullsized":"2023-10-30T13:52:29.488786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1627,"log_dups_size":0,"ondisk_log_size":1627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:39:17.448171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097894100000000001,"stat_sum":{"num_bytes":19085997,"num_objects":650,"num_object_clones":307,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":230,"num_read":105,"num_read_kb":202,"num_write":1384,"num_write_kb":24607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1288","reported_seq":1590,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.306924+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:52:29.306924+0000","last_peered":"2023-10-30T13:52:29.306924+0000","last_clean":"2023-10-30T13:52:29.306924+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:52:29.306924+0000","last_undegraded":"2023-10-30T13:52:29.306924+0000","last_fullsized":"2023-10-30T13:52:29.306924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:14:07.621621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36756164,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":418,"num_read_kb":304041,"num_write":1390,"num_write_kb":91637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1511","reported_seq":3434,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.482072+0000","last_change":"2023-10-30T13:21:40.528576+0000","last_active":"2023-10-30T13:52:29.482072+0000","last_peered":"2023-10-30T13:52:29.482072+0000","last_clean":"2023-10-30T13:52:29.482072+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:52:29.482072+0000","last_undegraded":"2023-10-30T13:52:29.482072+0000","last_fullsized":"2023-10-30T13:52:29.482072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:22:58.014283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026802879999999999,"stat_sum":{"num_bytes":22843366,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":237,"num_read":96,"num_read_kb":23,"num_write":1281,"num_write_kb":25082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'774","reported_seq":1006,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.165918+0000","last_change":"2023-10-30T13:10:33.689662+0000","last_active":"2023-10-30T13:52:24.165918+0000","last_peered":"2023-10-30T13:52:24.165918+0000","last_clean":"2023-10-30T13:52:24.165918+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:52:24.165918+0000","last_undegraded":"2023-10-30T13:52:24.165918+0000","last_fullsized":"2023-10-30T13:52:24.165918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:12.841467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":280,"num_read_kb":212075,"num_write":835,"num_write_kb":74769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1466","reported_seq":3420,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.432479+0000","last_change":"2023-10-30T13:21:40.576062+0000","last_active":"2023-10-30T13:52:29.432479+0000","last_peered":"2023-10-30T13:52:29.432479+0000","last_clean":"2023-10-30T13:52:29.432479+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:52:29.432479+0000","last_undegraded":"2023-10-30T13:52:29.432479+0000","last_fullsized":"2023-10-30T13:52:29.432479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:28.354972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055040289999999997,"stat_sum":{"num_bytes":14435337,"num_objects":589,"num_object_clones":269,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":212,"num_read":103,"num_read_kb":22,"num_write":1235,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'709","reported_seq":972,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:19.234865+0000","last_change":"2023-10-30T13:10:33.715080+0000","last_active":"2023-10-30T13:52:19.234865+0000","last_peered":"2023-10-30T13:52:19.234865+0000","last_clean":"2023-10-30T13:52:19.234865+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:52:19.234865+0000","last_undegraded":"2023-10-30T13:52:19.234865+0000","last_fullsized":"2023-10-30T13:52:19.234865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:44:33.369423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749318,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":374,"num_read_kb":96620,"num_write":840,"num_write_kb":51020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165182927,"num_objects":39302,"num_object_clones":18317,"num_object_copies":78604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39302,"num_whiteouts":14350,"num_read":6653,"num_read_kb":168434,"num_write":83476,"num_write_kb":1459733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424070144,"data_stored":2332381086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98751,"ondisk_log_size":98751,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":906305563,"num_objects":1697,"num_object_clones":0,"num_object_copies":3394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":11399,"num_read_kb":5963012,"num_write":34073,"num_write_kb":2071689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1407,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1812758528,"data_stored":1812611126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30360,"ondisk_log_size":30360,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575359,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727788,"kb_used_data":574436,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93644052,"statfs":{"total":96636764160,"available":95891509248,"internally_reserved":0,"allocated":588222464,"data_stored":574595578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":157022667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.95099999999999996},"min":{"1min":0.47999999999999998,"5min":0.41199999999999998,"15min":0.40899999999999997},"max":{"1min":1.288,"5min":1.321,"15min":15.381},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81899999999999995,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.315},"max":{"1min":1.212,"5min":1.8400000000000001,"15min":2.9510000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.83799999999999997,"15min":0.92800000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3240000000000001,"5min":1.79,"15min":15.271000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89100000000000001,"15min":0.96199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.42999999999999999},"max":{"1min":1.2769999999999999,"5min":1.6000000000000001,"15min":15.228999999999999},"last":0.70299999999999996}]},{"osd":2,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92100000000000004,"15min":0.91300000000000003},"min":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.36299999999999999},"max":{"1min":1.3100000000000001,"5min":1.345,"15min":2.2429999999999999},"last":1.05},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.89000000000000001,"15min":0.877},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.242,"5min":1.929,"15min":2.4540000000000002},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95599999999999996,"15min":1.0800000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.256,"5min":2.0249999999999999,"15min":15.324},"last":0.996},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.96999999999999997,"15min":0.98599999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2629999999999999,"5min":1.323,"15min":1.637},"last":0.95099999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.091,"15min":1.1479999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.40899999999999997},"max":{"1min":1.3080000000000001,"5min":2.7869999999999999,"15min":15.172000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.093,"5min":1.0600000000000001,"15min":1.089},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.466,"5min":3.032,"15min":4.0209999999999999},"last":1.0840000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1120000000000001,"15min":1.1060000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.454,"5min":1.4770000000000001,"15min":1.7050000000000001},"last":1.175},{"interface":"front","average":{"1min":1.089,"5min":1.0600000000000001,"15min":1.0169999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.54500000000000004},"max":{"1min":1.4159999999999999,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":0.745}]},{"osd":6,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.1819999999999999,"15min":1.2509999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.46500000000000002},"max":{"1min":1.7829999999999999,"5min":3.0350000000000001,"15min":15.427},"last":1.204},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.228,"15min":1.331},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.42199999999999999},"max":{"1min":1.841,"5min":3.5169999999999999,"15min":32.976999999999997},"last":1.26}]}]},{"osd":6,"up_from":13,"seq":55834575358,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595240,"kb_used_data":451680,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93776600,"statfs":{"total":96636764160,"available":96027238400,"internally_reserved":0,"allocated":462520320,"data_stored":453922596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":146995655},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.82099999999999995,"15min":0.84599999999999997},"min":{"1min":0.55000000000000004,"5min":0.45000000000000001,"15min":0.40999999999999998},"max":{"1min":1.2190000000000001,"5min":1.3999999999999999,"15min":5.9880000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.86899999999999999,"15min":0.89200000000000002},"min":{"1min":0.58999999999999997,"5min":0.48599999999999999,"15min":0.35999999999999999},"max":{"1min":1.0620000000000001,"5min":1.3160000000000001,"15min":6.0730000000000004},"last":0.86199999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.93600000000000005,"15min":0.86499999999999999},"min":{"1min":0.58199999999999996,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.585,"5min":6.9930000000000003,"15min":6.9930000000000003},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.86399999999999999,"15min":0.86099999999999999},"min":{"1min":0.69299999999999995,"5min":0.52500000000000002,"15min":0.47499999999999998},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.9810000000000001},"last":0.76800000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.91800000000000004,"15min":0.89400000000000002},"min":{"1min":0.59899999999999998,"5min":0.54400000000000004,"15min":0.47699999999999998},"max":{"1min":1.044,"5min":6.8550000000000004,"15min":6.8550000000000004},"last":1.05},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.97999999999999998,"15min":0.96399999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.454,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.69199999999999995}]},{"osd":3,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88600000000000001,"15min":0.89400000000000002},"min":{"1min":0.64800000000000002,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.2130000000000001,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.96899999999999997,"15min":0.98699999999999999},"min":{"1min":0.69899999999999995,"5min":0.64400000000000002,"15min":0.56299999999999994},"max":{"1min":1.536,"5min":5.3179999999999996,"15min":5.3179999999999996},"last":0.95499999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.99099999999999999,"15min":1.046},"min":{"1min":0.59199999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.321,"5min":1.4970000000000001,"15min":15.372},"last":1.016},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0629999999999999,"15min":1.05},"min":{"1min":0.79000000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"max":{"1min":1.391,"5min":5.2729999999999997,"15min":6.1580000000000004},"last":1.1200000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0329999999999999,"15min":1.042},"min":{"1min":0.749,"5min":0.60499999999999998,"15min":0.60199999999999998},"max":{"1min":1.2649999999999999,"5min":1.421,"15min":1.8080000000000001},"last":1.157},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.202,"15min":1.1759999999999999},"min":{"1min":0.95199999999999996,"5min":0.73899999999999999,"15min":0.67700000000000005},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.913},"last":1.2430000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:51:37 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0860000000000001,"15min":1.0960000000000001},"min":{"1min":0.72099999999999997,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2290000000000001,"5min":1.577,"15min":2.75},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.212,"15min":1.1899999999999999},"min":{"1min":1.006,"5min":0.69599999999999995,"15min":0.42499999999999999},"max":{"1min":1.627,"5min":1.627,"15min":3.0590000000000002},"last":1.302}]}]},{"osd":5,"up_from":13,"seq":55834575359,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676788,"kb_used_data":519276,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93695052,"statfs":{"total":96636764160,"available":95943733248,"internally_reserved":0,"allocated":531738624,"data_stored":518585685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":161282502},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88900000000000001,"15min":0.87},"min":{"1min":0.51300000000000001,"5min":0.38400000000000001,"15min":0.32700000000000001},"max":{"1min":1.3300000000000001,"5min":1.4379999999999999,"15min":6.4729999999999999},"last":0.871},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.88800000000000001,"15min":0.86899999999999999},"min":{"1min":0.56999999999999995,"5min":0.46800000000000003,"15min":0.39100000000000001},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":8.7560000000000002},"last":1.135}]},{"osd":1,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.81499999999999995,"15min":0.88200000000000001},"min":{"1min":0.51400000000000001,"5min":0.439,"15min":0.42799999999999999},"max":{"1min":1.2,"5min":1.671,"15min":15.257999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.86099999999999999,"15min":0.94899999999999995},"min":{"1min":0.62,"5min":0.48699999999999999,"15min":0.47399999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":15.175000000000001},"last":0.82399999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.90100000000000002,"15min":0.91100000000000003},"min":{"1min":0.57499999999999996,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.228,"5min":1.458,"15min":6.5410000000000004},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.93700000000000006,"15min":0.98699999999999999},"min":{"1min":0.57599999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.238,"5min":1.52,"15min":6.7080000000000002},"last":0.92900000000000005}]},{"osd":3,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.0880000000000001,"15min":1.0469999999999999},"min":{"1min":0.69399999999999995,"5min":0.58399999999999996,"15min":0.54900000000000004},"max":{"1min":1.3839999999999999,"5min":5.4249999999999998,"15min":6.7699999999999996},"last":1.238},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.175,"15min":1.1659999999999999},"min":{"1min":0.88600000000000001,"5min":0.58699999999999997,"15min":0.40100000000000002},"max":{"1min":1.4790000000000001,"5min":5.5570000000000004,"15min":15.364000000000001},"last":1.292}]},{"osd":4,"last update":"Mon Oct 30 13:51:44 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.92500000000000004,"15min":0.97999999999999998},"min":{"1min":0.63600000000000001,"5min":0.376,"15min":0.36499999999999999},"max":{"1min":2.5830000000000002,"5min":4.3529999999999998,"15min":5.9720000000000004},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.27,"5min":1.246,"15min":1.1459999999999999},"min":{"1min":0.92900000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.5190000000000001,"5min":5.6609999999999996,"15min":14.314},"last":1.1779999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:51:33 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.0129999999999999,"15min":1.123},"min":{"1min":0.69299999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.4239999999999999,"5min":4.2290000000000001,"15min":15.06},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.097,"15min":1.1799999999999999},"min":{"1min":0.83699999999999997,"5min":0.65400000000000003,"15min":0.50900000000000001},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":7.6630000000000003},"last":1.208}]},{"osd":7,"last update":"Mon Oct 30 13:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.0609999999999999,"15min":1.1279999999999999},"min":{"1min":0.745,"5min":0.44,"15min":0.44},"max":{"1min":4.0599999999999996,"5min":5.0419999999999998,"15min":6.8259999999999996},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.429,"5min":1.2130000000000001,"15min":1.266},"min":{"1min":0.89700000000000002,"5min":0.53500000000000003,"15min":0.46200000000000002},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":19.321999999999999},"last":1.494}]}]},{"osd":4,"up_from":13,"seq":55834575360,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717128,"kb_used_data":569408,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93654712,"statfs":{"total":96636764160,"available":95902425088,"internally_reserved":0,"allocated":583073792,"data_stored":568869284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.77700000000000002,"15min":0.80900000000000005},"min":{"1min":0.51800000000000002,"5min":0.48599999999999999,"15min":0.28100000000000003},"max":{"1min":1.087,"5min":2.202,"15min":9.6690000000000005},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85199999999999998,"15min":0.84799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47399999999999998,"15min":0.377},"max":{"1min":1.2010000000000001,"5min":2.0760000000000001,"15min":2.786},"last":0.64100000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.873,"15min":0.88200000000000001},"min":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.41399999999999998},"max":{"1min":1.2170000000000001,"5min":1.8160000000000001,"15min":5.5119999999999996},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.89500000000000002,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.51500000000000001,"15min":0.44700000000000001},"max":{"1min":1.202,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.749}]},{"osd":2,"last update":"Mon Oct 30 13:51:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.89200000000000002,"15min":0.878},"min":{"1min":0.63200000000000001,"5min":0.53700000000000003,"15min":0.45100000000000001},"max":{"1min":1.212,"5min":2.1549999999999998,"15min":9.0099999999999998},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.95099999999999996,"15min":0.95199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0569999999999999,"5min":7.7359999999999998,"15min":7.7359999999999998},"last":0.95999999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.94499999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.52500000000000002},"max":{"1min":1.1899999999999999,"5min":1.6599999999999999,"15min":4.7080000000000002},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.0580000000000001,"15min":1.0209999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.48699999999999999},"max":{"1min":1.2649999999999999,"5min":1.7509999999999999,"15min":3.2749999999999999},"last":1.101}]},{"osd":5,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0249999999999999,"15min":1.0489999999999999},"min":{"1min":0.56299999999999994,"5min":0.52300000000000002,"15min":0.51500000000000001},"max":{"1min":1.3799999999999999,"5min":1.621,"15min":7.8529999999999998},"last":1.052},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.214,"15min":1.161},"min":{"1min":0.86899999999999999,"5min":0.627,"15min":0.40999999999999998},"max":{"1min":1.538,"5min":7.694,"15min":8.2070000000000007},"last":1.1819999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1379999999999999,"15min":1.081},"min":{"1min":0.745,"5min":0.68999999999999995,"15min":0.52600000000000002},"max":{"1min":1.458,"5min":7.7880000000000003,"15min":7.7880000000000003},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.038,"5min":1.1659999999999999,"15min":1.155},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.2929999999999999,"5min":3.7320000000000002,"15min":5.6130000000000004},"last":1.2250000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:51:34 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.1950000000000001,"15min":1.1739999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.52000000000000002},"max":{"1min":1.6120000000000001,"5min":7.8380000000000001,"15min":10.715},"last":1.3109999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0720000000000001,"15min":1.157},"min":{"1min":0.627,"5min":0.627,"15min":0.47199999999999998},"max":{"1min":1.1579999999999999,"5min":2.2450000000000001,"15min":4.3230000000000004},"last":1.141}]}]},{"osd":3,"up_from":13,"seq":55834575358,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707492,"kb_used_data":550044,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93664348,"statfs":{"total":96636764160,"available":95912292352,"internally_reserved":0,"allocated":563245056,"data_stored":549099638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":161216968},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.69799999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64900000000000002,"5min":0.76700000000000002,"15min":1.5429999999999999},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.51000000000000001,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":0.755,"5min":0.76200000000000001,"15min":0.94099999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.28599999999999998,"15min":0.246},"max":{"1min":0.78900000000000003,"5min":0.92000000000000004,"15min":1.018},"last":0.49199999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.65100000000000002,"5min":1.056,"15min":1.4690000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.86499999999999999,"15min":1.6799999999999999},"last":0.434}]},{"osd":4,"last update":"Mon Oct 30 13:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.79600000000000004,"15min":0.83599999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.49099999999999999},"max":{"1min":0.96499999999999997,"5min":1.1739999999999999,"15min":1.5900000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.84299999999999997,"15min":0.85299999999999998},"min":{"1min":0.66100000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0129999999999999,"5min":1.1419999999999999,"15min":1.3999999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.60799999999999998,"5min":0.57399999999999995,"15min":0.51800000000000002},"max":{"1min":1.095,"5min":1.1879999999999999,"15min":1.5069999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.83599999999999997,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.58599999999999997,"15min":0.5},"max":{"1min":1.232,"5min":1.232,"15min":1.365},"last":0.81000000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.88700000000000001,"15min":0.82799999999999996},"min":{"1min":0.66900000000000004,"5min":0.52800000000000002,"15min":0.45400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.88500000000000001,"15min":0.82699999999999996},"min":{"1min":0.69499999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":2.3410000000000002},"last":0.84599999999999997}]},{"osd":7,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.88800000000000001,"15min":0.88200000000000001},"min":{"1min":0.68300000000000005,"5min":0.57799999999999996,"15min":0.51700000000000002},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":2.9449999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.872,"15min":0.89300000000000002},"min":{"1min":0.64300000000000002,"5min":0.54400000000000004,"15min":0.47099999999999997},"max":{"1min":1.343,"5min":1.343,"15min":2.9729999999999999},"last":0.752}]}]},{"osd":2,"up_from":13,"seq":55834575361,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484724,"kb_used_data":377260,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93887116,"statfs":{"total":96636764160,"available":96140406784,"internally_reserved":0,"allocated":386314240,"data_stored":377764906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.47999999999999998,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.50800000000000001,"15min":0.48199999999999998},"min":{"1min":0.41799999999999998,"5min":0.28299999999999997,"15min":0.25700000000000001},"max":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"last":0.42499999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.45900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.496},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.58399999999999996,"15min":0.54000000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.0060000000000002,"5min":7.0060000000000002,"15min":7.0060000000000002},"last":0.40100000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.36499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":1.637},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.622,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.48099999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.78000000000000003,"15min":0.80300000000000005},"min":{"1min":0.59699999999999998,"5min":0.442,"15min":0.442},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.626},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.80500000000000005,"15min":0.82699999999999996},"min":{"1min":0.61399999999999999,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":4.1479999999999997,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.748}]},{"osd":5,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.88500000000000001,"15min":0.877},"min":{"1min":0.754,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5169999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.80900000000000005,"15min":0.82499999999999996},"min":{"1min":0.58599999999999997,"5min":0.45300000000000001,"15min":0.41999999999999998},"max":{"1min":1.171,"5min":2.762,"15min":2.762},"last":0.76400000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77400000000000002,"15min":0.79000000000000004},"min":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.47699999999999998},"max":{"1min":0.98699999999999999,"5min":1.6419999999999999,"15min":3.1659999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.84399999999999997,"15min":0.81699999999999995},"min":{"1min":0.65500000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.8210000000000002,"5min":3.8210000000000002,"15min":3.8210000000000002},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":0.90800000000000003,"15min":0.85699999999999998},"min":{"1min":0.69599999999999995,"5min":0.55000000000000004,"15min":0.503},"max":{"1min":6.9059999999999997,"5min":6.9059999999999997,"15min":6.9059999999999997},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.0009999999999999,"15min":0.95499999999999996},"min":{"1min":0.81299999999999994,"5min":0.63100000000000001,"15min":0.55900000000000005},"max":{"1min":6.9509999999999996,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.93100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575358,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673092,"kb_used_data":533756,"kb_used_omap":1,"kb_used_meta":139326,"kb_avail":93698748,"statfs":{"total":96636764160,"available":95947517952,"internally_reserved":0,"allocated":546566144,"data_stored":537485994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142670278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30399999999999999,"5min":0.24299999999999999,"15min":0.20799999999999999},"max":{"1min":0.58799999999999997,"5min":0.94999999999999996,"15min":1.1359999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.28299999999999997,"15min":0.251},"max":{"1min":0.64900000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49099999999999999,"15min":0.501},"min":{"1min":0.318,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.66800000000000004,"5min":1.002,"15min":2.8660000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.498,"5min":0.501,"15min":0.48599999999999999},"min":{"1min":0.36899999999999999,"5min":0.27700000000000002,"15min":0.249},"max":{"1min":0.69099999999999995,"5min":1.0589999999999999,"15min":2.6739999999999999},"last":0.40400000000000003}]},{"osd":3,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.53000000000000003},"min":{"1min":0.373,"5min":0.35099999999999998,"15min":0.314},"max":{"1min":0.73899999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.497},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.372,"15min":0.318},"max":{"1min":0.755,"5min":1.306,"15min":2.71},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82399999999999995,"15min":0.85599999999999998},"min":{"1min":0.622,"5min":0.52400000000000002,"15min":0.40200000000000002},"max":{"1min":1.0489999999999999,"5min":1.2210000000000001,"15min":6.7789999999999999},"last":0.872},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.89600000000000002,"15min":0.92200000000000004},"min":{"1min":0.76500000000000001,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":1.093,"5min":1.2629999999999999,"15min":6.8259999999999996},"last":0.91100000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":0.76200000000000001},"min":{"1min":0.65800000000000003,"5min":0.53800000000000003,"15min":0.47199999999999998},"max":{"1min":0.96499999999999997,"5min":2.133,"15min":2.5750000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.627,"5min":0.53800000000000003,"15min":0.51000000000000001},"max":{"1min":1.0229999999999999,"5min":1.3859999999999999,"15min":2.6179999999999999},"last":0.81699999999999995}]},{"osd":6,"last update":"Mon Oct 30 13:51:46 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91800000000000004,"15min":0.89500000000000002},"min":{"1min":0.80000000000000004,"5min":0.61199999999999999,"15min":0.501},"max":{"1min":1.1459999999999999,"5min":1.3999999999999999,"15min":6.0289999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.96599999999999997,"15min":0.95499999999999996},"min":{"1min":0.65500000000000003,"5min":0.625,"15min":0.60299999999999998},"max":{"1min":1.218,"5min":1.4399999999999999,"15min":6.0739999999999998},"last":0.98599999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:51:40 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79500000000000004,"15min":0.78300000000000003},"min":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.47899999999999998},"max":{"1min":1.0369999999999999,"5min":1.161,"15min":2.1579999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.879,"15min":0.88500000000000001},"min":{"1min":0.70899999999999996,"5min":0.55200000000000005,"15min":0.54800000000000004},"max":{"1min":1.1339999999999999,"5min":1.3440000000000001,"15min":6.0519999999999996},"last":0.83299999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575360,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712496,"kb_used_data":568936,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93659344,"statfs":{"total":96636764160,"available":95907168256,"internally_reserved":0,"allocated":582590464,"data_stored":569154632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.40100000000000002,"15min":0.48399999999999999},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.214},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":16.184000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.50800000000000001,"15min":0.59199999999999997},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.28899999999999998},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":16.228000000000002},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.44900000000000001,"15min":0.64100000000000001},"min":{"1min":0.33300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":2.7999999999999998,"5min":2.7999999999999998,"15min":17.454000000000001},"last":0.374},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.65300000000000002},"min":{"1min":0.318,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":1.391,"5min":1.391,"15min":23.212},"last":0.443}]},{"osd":3,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.37,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":16.103999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.53100000000000003,"15min":0.73099999999999998},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.32000000000000001},"max":{"1min":2.8340000000000001,"5min":2.8340000000000001,"15min":23.16},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:51:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.83299999999999996,"15min":0.92700000000000005},"min":{"1min":0.48499999999999999,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":7.6900000000000004,"5min":7.6900000000000004,"15min":16.309000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.85799999999999998,"15min":0.97499999999999998},"min":{"1min":0.61199999999999999,"5min":0.53800000000000003,"15min":0.46000000000000002},"max":{"1min":2.5939999999999999,"5min":3.4009999999999998,"15min":16.343},"last":0.623}]},{"osd":5,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85299999999999998,"15min":1.006},"min":{"1min":0.63,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.02,"5min":2.02,"15min":23.266999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.86299999999999999,"15min":1.0329999999999999},"min":{"1min":0.755,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.7240000000000002,"5min":2.7240000000000002,"15min":16.268000000000001},"last":0.83999999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.80500000000000005,"15min":1.1120000000000001},"min":{"1min":0.61099999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":29.774999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.81499999999999995,"15min":1.0229999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.50700000000000001},"max":{"1min":2.75,"5min":2.75,"15min":17.5},"last":0.82499999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.81999999999999995,"15min":1.202},"min":{"1min":0.66700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.694,"5min":2.694,"15min":47.534999999999997},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.85999999999999999,"15min":1.155},"min":{"1min":0.64100000000000001,"5min":0.60399999999999998,"15min":0.54700000000000004},"max":{"1min":2.8690000000000002,"5min":2.8690000000000002,"15min":32.476999999999997},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230703104,"data_stored":230687380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256921600,"data_stored":256897758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179036160,"data_stored":179013521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230858752,"data_stored":230840931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":197275648,"data_stored":197263919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":205541376,"data_stored":205522290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":268013568,"data_stored":267996941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":244408320,"data_stored":244388386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350515200,"data_stored":337464605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288862208,"data_stored":280174938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":206495744,"data_stored":198338087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331603968,"data_stored":317845409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":385015808,"data_stored":371192067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":325414912,"data_stored":312650097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193724416,"data_stored":185512357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342437888,"data_stored":329203526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:52:37.176 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.177 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.177 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.177 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.178 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.178 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.178 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.179 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.179 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.179 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.180 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.180 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.180 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.181 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.181 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.181 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.182 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.182 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.182 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.183 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.183 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.183 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.184 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.184 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.185 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.185 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.185 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.186 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.186 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.186 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.187 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.187 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.187 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.188 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.188 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.188 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.189 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.189 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.189 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.190 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.190 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.191 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.191 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.191 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.192 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.192 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.192 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.192 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.193 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.193 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.194 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.194 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.194 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.195 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.195 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.195 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.196 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.196 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.197 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.197 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.197 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.198 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.198 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.198 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.199 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T13:10:22.508530+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=22, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.199 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.200 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.200 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.200 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.201 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.201 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.201 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.202 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.202 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.203 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.203 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.203 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.204 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.204 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.205 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.205 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.205 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.206 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.206 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.207 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.207 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.207 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.208 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.208 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.208 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.209 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.209 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.210 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.210 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.210 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.211 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.211 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:37.211 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:52:57.212 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:52:57.366 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce916ea640 1 Processor -- start 2023-10-30T13:52:57.367 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce916ea640 1 -- start start 2023-10-30T13:52:57.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce916ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c15d6f0 0x7fce8c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.368 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce916ea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce8c0cf160 0x7fce8c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce916ea640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0cb800 con 0x7fce8c1538a0 2023-10-30T13:52:57.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce916ea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8c0cb330 con 0x7fce8c0cf160 2023-10-30T13:52:57.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce916ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c0cad70 con 0x7fce8c15d6f0 2023-10-30T13:52:57.369 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce8affd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fce8c1538a0 0x7fce8c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:33850/0 (socket says 172.21.15.148:33850) 2023-10-30T13:52:57.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce8affd640 1 -- 172.21.15.148:0/2281520818 learned_addr learned my addr 172.21.15.148:0/2281520818 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:52:57.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce8b7fe640 1 --2- 172.21.15.148:0/2281520818 >> v2:172.21.15.150:3300/0 conn(0x7fce8c0cf160 0x7fce8c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.370 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.361+0000 7fce83fff640 1 --2- 172.21.15.148:0/2281520818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c15d6f0 0x7fce8c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.365+0000 7fce8a7fc640 1 -- 172.21.15.148:0/2281520818 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871097278 0 0) 0x7fce8c0cb800 con 0x7fce8c1538a0 2023-10-30T13:52:57.371 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.365+0000 7fce8a7fc640 1 -- 172.21.15.148:0/2281520818 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce78003530 con 0x7fce8c1538a0 2023-10-30T13:52:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.365+0000 7fce8a7fc640 1 -- 172.21.15.148:0/2281520818 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3092766386 0 0) 0x7fce78003530 con 0x7fce8c1538a0 2023-10-30T13:52:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.365+0000 7fce8b7fe640 1 -- 172.21.15.148:0/2281520818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c15d6f0 msgr2=0x7fce8c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:52:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.365+0000 7fce8b7fe640 1 --2- 172.21.15.148:0/2281520818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c15d6f0 0x7fce8c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.372 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.365+0000 7fce8b7fe640 1 -- 172.21.15.148:0/2281520818 >> v1:172.21.15.148:6789/0 conn(0x7fce8c1538a0 legacy=0x7fce8c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.373 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.365+0000 7fce8b7fe640 1 -- 172.21.15.148:0/2281520818 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c0cb050 con 0x7fce8c0cf160 2023-10-30T13:52:57.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.365+0000 7fce8b7fe640 1 --2- 172.21.15.148:0/2281520818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce8c0cf160 0x7fce8c0cf590 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fce84009d90 tx=0x7fce8401f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13af2e76d56aa93e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:57.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce8a7fc640 1 -- 172.21.15.148:0/2281520818 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c001680 con 0x7fce8c0cf160 2023-10-30T13:52:57.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce8a7fc640 1 -- 172.21.15.148:0/2281520818 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce840a8bb0 con 0x7fce8c0cf160 2023-10-30T13:52:57.374 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce8a7fc640 1 -- 172.21.15.148:0/2281520818 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce840b0440 con 0x7fce8c0cf160 2023-10-30T13:52:57.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 -- 172.21.15.148:0/2281520818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce8c0cf160 msgr2=0x7fce8c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 --2- 172.21.15.148:0/2281520818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce8c0cf160 0x7fce8c0cf590 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fce84009d90 tx=0x7fce8401f060 comp rx=0 tx=0).stop 2023-10-30T13:52:57.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 -- 172.21.15.148:0/2281520818 shutdown_connections 2023-10-30T13:52:57.375 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 --2- 172.21.15.148:0/2281520818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce8c0cf160 0x7fce8c0cf590 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 --2- 172.21.15.148:0/2281520818 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c15d6f0 0x7fce8c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 -- 172.21.15.148:0/2281520818 >> 172.21.15.148:0/2281520818 conn(0x7fce8c06bfe0 msgr2=0x7fce8c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:57.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 -- 172.21.15.148:0/2281520818 shutdown_connections 2023-10-30T13:52:57.376 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 -- 172.21.15.148:0/2281520818 wait complete. 2023-10-30T13:52:57.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 Processor -- start 2023-10-30T13:52:57.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 -- start start 2023-10-30T13:52:57.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8c0cf160 0x7fce8c1504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c1538a0 0x7fce8c065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.379 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce8c15d6f0 0x7fce8c1ef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c14fe60 con 0x7fce8c0cf160 2023-10-30T13:52:57.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c1f32f0 con 0x7fce8c15d6f0 2023-10-30T13:52:57.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.369+0000 7fce916ea640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c0c5710 con 0x7fce8c1538a0 2023-10-30T13:52:57.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8c0cf160 0x7fce8c1504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8c0cf160 0x7fce8c1504f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35836/0 (socket says 172.21.15.148:35836) 2023-10-30T13:52:57.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8affd640 1 -- 172.21.15.148:0/2675622537 learned_addr learned my addr 172.21.15.148:0/2675622537 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:52:57.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce83fff640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c1538a0 0x7fce8c065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8b7fe640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce8c15d6f0 0x7fce8c1ef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8affd640 1 -- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c1538a0 msgr2=0x7fce8c065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8affd640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c1538a0 0x7fce8c065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8affd640 1 -- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce8c15d6f0 msgr2=0x7fce8c1ef1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8affd640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce8c15d6f0 0x7fce8c1ef1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8affd640 1 -- 172.21.15.148:0/2675622537 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce84009a70 con 0x7fce8c0cf160 2023-10-30T13:52:57.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce83fff640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c1538a0 0x7fce8c065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:52:57.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce8affd640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8c0cf160 0x7fce8c1504f0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c750 tx=0x7fce7c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:57.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce837fe640 1 -- 172.21.15.148:0/2675622537 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c005ae0 con 0x7fce8c0cf160 2023-10-30T13:52:57.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce837fe640 1 -- 172.21.15.148:0/2675622537 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c005c40 con 0x7fce8c0cf160 2023-10-30T13:52:57.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8c150b30 con 0x7fce8c0cf160 2023-10-30T13:52:57.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce837fe640 1 -- 172.21.15.148:0/2675622537 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c007070 con 0x7fce8c0cf160 2023-10-30T13:52:57.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8c1f29c0 con 0x7fce8c0cf160 2023-10-30T13:52:57.385 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8c0b2a50 con 0x7fce8c0cf160 2023-10-30T13:52:57.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.373+0000 7fce837fe640 1 -- 172.21.15.148:0/2675622537 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fce7c0071d0 con 0x7fce8c0cf160 2023-10-30T13:52:57.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.377+0000 7fce837fe640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce6c0664d0 0x7fce6c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:52:57.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.377+0000 7fce837fe640 1 -- 172.21.15.148:0/2675622537 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fce7c0844d0 con 0x7fce8c0cf160 2023-10-30T13:52:57.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.377+0000 7fce83fff640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce6c0664d0 0x7fce6c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:52:57.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.377+0000 7fce83fff640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce6c0664d0 0x7fce6c068990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fce8c15ba90 tx=0x7fce74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:52:57.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.381+0000 7fce837fe640 1 -- 172.21.15.148:0/2675622537 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fce7c051120 con 0x7fce8c0cf160 2023-10-30T13:52:57.557 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.553+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce8c159b70 con 0x7fce6c0664d0 2023-10-30T13:52:57.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.573+0000 7fce837fe640 1 -- 172.21.15.148:0/2675622537 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292602 (secure 0 0 0) 0x7fce74011a10 con 0x7fce6c0664d0 2023-10-30T13:52:57.583 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:52:57.585 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:52:57.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce6c0664d0 msgr2=0x7fce6c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce6c0664d0 0x7fce6c068990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fce8c15ba90 tx=0x7fce74008040 comp rx=0 tx=0).stop 2023-10-30T13:52:57.586 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8c0cf160 msgr2=0x7fce8c1504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:52:57.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8c0cf160 0x7fce8c1504f0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c750 tx=0x7fce7c00cc10 comp rx=0 tx=0).stop 2023-10-30T13:52:57.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 shutdown_connections 2023-10-30T13:52:57.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fce6c0664d0 0x7fce6c068990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.587 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce8c15d6f0 0x7fce8c1ef1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fce8c1538a0 0x7fce8c065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 --2- 172.21.15.148:0/2675622537 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8c0cf160 0x7fce8c1504f0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:52:57.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 >> 172.21.15.148:0/2675622537 conn(0x7fce8c06bfe0 msgr2=0x7fce8c150ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:52:57.588 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 shutdown_connections 2023-10-30T13:52:57.589 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:52:57.581+0000 7fce916ea640 1 -- 172.21.15.148:0/2675622537 wait complete. 2023-10-30T13:52:57.609 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1337,"stamp":"2023-10-30T13:52:57.212751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2074440002,"num_objects":41005,"num_object_clones":18317,"num_object_copies":82010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41005,"num_whiteouts":14350,"num_read":18168,"num_read_kb":6131546,"num_write":117776,"num_write_kb":3536253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1411,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11942485,"num_omap_keys":19988,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138430,"ondisk_log_size":138430,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5338204,"kb_used_data":4149404,"kb_used_omap":12,"kb_used_meta":1188723,"kb_avail":749636516,"statfs":{"total":773094113280,"available":767627792384,"internally_reserved":0,"allocated":4248989696,"data_stored":4154200601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1217252920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,9,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":669948,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6551743,"num_omap_keys":10676,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003548"},"pg_stats":[{"pgid":"3.3f","version":"67'1473","reported_seq":3441,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.519050+0000","last_change":"2023-10-30T13:21:40.584822+0000","last_active":"2023-10-30T13:52:29.519050+0000","last_peered":"2023-10-30T13:52:29.519050+0000","last_clean":"2023-10-30T13:52:29.519050+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:52:29.519050+0000","last_undegraded":"2023-10-30T13:52:29.519050+0000","last_fullsized":"2023-10-30T13:52:29.519050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014806453000000001,"stat_sum":{"num_bytes":11269044,"num_objects":569,"num_object_clones":260,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":205,"num_read":107,"num_read_kb":121,"num_write":1253,"num_write_kb":17103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1479","reported_seq":3403,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.436814+0000","last_change":"2023-10-30T13:21:40.572202+0000","last_active":"2023-10-30T13:52:29.436814+0000","last_peered":"2023-10-30T13:52:29.436814+0000","last_clean":"2023-10-30T13:52:29.436814+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:52:29.436814+0000","last_undegraded":"2023-10-30T13:52:29.436814+0000","last_fullsized":"2023-10-30T13:52:29.436814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059691099999999997,"stat_sum":{"num_bytes":30975930,"num_objects":597,"num_object_clones":279,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":225,"num_read":82,"num_read_kb":30,"num_write":1226,"num_write_kb":32475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'1530","reported_seq":3478,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.437623+0000","last_change":"2023-10-30T13:21:40.528480+0000","last_active":"2023-10-30T13:52:29.437623+0000","last_peered":"2023-10-30T13:52:29.437623+0000","last_clean":"2023-10-30T13:52:29.437623+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:52:29.437623+0000","last_undegraded":"2023-10-30T13:52:29.437623+0000","last_fullsized":"2023-10-30T13:52:29.437623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025608710000000002,"stat_sum":{"num_bytes":18704388,"num_objects":617,"num_object_clones":279,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":225,"num_read":106,"num_read_kb":8300,"num_write":1300,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'1584","reported_seq":3521,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.433474+0000","last_change":"2023-10-30T13:21:40.567224+0000","last_active":"2023-10-30T13:52:29.433474+0000","last_peered":"2023-10-30T13:52:29.433474+0000","last_clean":"2023-10-30T13:52:29.433474+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:52:29.433474+0000","last_undegraded":"2023-10-30T13:52:29.433474+0000","last_fullsized":"2023-10-30T13:52:29.433474+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021564890000000001,"stat_sum":{"num_bytes":23329933,"num_objects":630,"num_object_clones":293,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":227,"num_read":112,"num_read_kb":4378,"num_write":1354,"num_write_kb":25006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'2231","reported_seq":4187,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:38.545335+0000","last_change":"2023-10-30T13:52:38.545426+0000","last_active":"2023-10-30T13:52:38.545335+0000","last_peered":"2023-10-30T13:52:38.545335+0000","last_clean":"2023-10-30T13:52:38.545335+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:52:38.545335+0000","last_undegraded":"2023-10-30T13:52:38.545335+0000","last_fullsized":"2023-10-30T13:52:38.545335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2231","last_scrub_stamp":"2023-10-30T13:52:38.545206+0000","last_deep_scrub":"67'2231","last_deep_scrub_stamp":"2023-10-30T13:52:38.545206+0000","last_clean_scrub_stamp":"2023-10-30T13:52:38.545206+0000","objects_scrubbed":654,"log_size":2231,"log_dups_size":0,"ondisk_log_size":2231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:21:20.355710+0000","scrub_duration":1.430729712,"objects_trimmed":0,"snaptrim_duration":0.011808077,"stat_sum":{"num_bytes":16184828,"num_objects":654,"num_object_clones":305,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":251,"num_read":109,"num_read_kb":152,"num_write":1339,"num_write_kb":20879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'1459","reported_seq":3407,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.494992+0000","last_change":"2023-10-30T13:21:40.539757+0000","last_active":"2023-10-30T13:52:29.494992+0000","last_peered":"2023-10-30T13:52:29.494992+0000","last_clean":"2023-10-30T13:52:29.494992+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:52:29.494992+0000","last_undegraded":"2023-10-30T13:52:29.494992+0000","last_fullsized":"2023-10-30T13:52:29.494992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01102786,"stat_sum":{"num_bytes":18372801,"num_objects":593,"num_object_clones":277,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":222,"num_read":108,"num_read_kb":4137,"num_write":1217,"num_write_kb":19358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1515","reported_seq":3453,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.469312+0000","last_change":"2023-10-30T13:21:40.580121+0000","last_active":"2023-10-30T13:52:29.469312+0000","last_peered":"2023-10-30T13:52:29.469312+0000","last_clean":"2023-10-30T13:52:29.469312+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:52:29.469312+0000","last_undegraded":"2023-10-30T13:52:29.469312+0000","last_fullsized":"2023-10-30T13:52:29.469312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012047337,"stat_sum":{"num_bytes":9859762,"num_objects":598,"num_object_clones":275,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":211,"num_read":96,"num_read_kb":35,"num_write":1295,"num_write_kb":11866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'2037","reported_seq":4025,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.982110+0000","last_change":"2023-10-30T13:52:51.982168+0000","last_active":"2023-10-30T13:52:51.982110+0000","last_peered":"2023-10-30T13:52:51.982110+0000","last_clean":"2023-10-30T13:52:51.982110+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:52:51.982110+0000","last_undegraded":"2023-10-30T13:52:51.982110+0000","last_fullsized":"2023-10-30T13:52:51.982110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2037","last_scrub_stamp":"2023-10-30T13:52:51.982035+0000","last_deep_scrub":"67'2037","last_deep_scrub_stamp":"2023-10-30T13:52:51.982035+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.982035+0000","objects_scrubbed":577,"log_size":2037,"log_dups_size":0,"ondisk_log_size":2037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:47.358388+0000","scrub_duration":1.116769473,"objects_trimmed":0,"snaptrim_duration":0.0063254060000000004,"stat_sum":{"num_bytes":13694423,"num_objects":577,"num_object_clones":268,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":211,"num_read":108,"num_read_kb":104,"num_write":1228,"num_write_kb":23250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1528","reported_seq":3471,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.522374+0000","last_change":"2023-10-30T13:21:40.584916+0000","last_active":"2023-10-30T13:52:29.522374+0000","last_peered":"2023-10-30T13:52:29.522374+0000","last_clean":"2023-10-30T13:52:29.522374+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:52:29.522374+0000","last_undegraded":"2023-10-30T13:52:29.522374+0000","last_fullsized":"2023-10-30T13:52:29.522374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014360304000000001,"stat_sum":{"num_bytes":19134308,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":234,"num_read":109,"num_read_kb":4185,"num_write":1275,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1501","reported_seq":3440,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.604439+0000","last_change":"2023-10-30T13:21:40.580946+0000","last_active":"2023-10-30T13:52:29.604439+0000","last_peered":"2023-10-30T13:52:29.604439+0000","last_clean":"2023-10-30T13:52:29.604439+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:52:29.604439+0000","last_undegraded":"2023-10-30T13:52:29.604439+0000","last_fullsized":"2023-10-30T13:52:29.604439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010277041000000001,"stat_sum":{"num_bytes":19693407,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":208,"num_read":105,"num_read_kb":2976,"num_write":1294,"num_write_kb":29513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1536","reported_seq":3464,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.440713+0000","last_change":"2023-10-30T13:21:40.579672+0000","last_active":"2023-10-30T13:52:29.440713+0000","last_peered":"2023-10-30T13:52:29.440713+0000","last_clean":"2023-10-30T13:52:29.440713+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:52:29.440713+0000","last_undegraded":"2023-10-30T13:52:29.440713+0000","last_fullsized":"2023-10-30T13:52:29.440713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1536,"log_dups_size":0,"ondisk_log_size":1536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01048763,"stat_sum":{"num_bytes":11727836,"num_objects":636,"num_object_clones":292,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":230,"num_read":95,"num_read_kb":27,"num_write":1294,"num_write_kb":13174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'1536","reported_seq":3462,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.507282+0000","last_change":"2023-10-30T13:21:40.537641+0000","last_active":"2023-10-30T13:52:29.507282+0000","last_peered":"2023-10-30T13:52:29.507282+0000","last_clean":"2023-10-30T13:52:29.507282+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:52:29.507282+0000","last_undegraded":"2023-10-30T13:52:29.507282+0000","last_fullsized":"2023-10-30T13:52:29.507282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1536,"log_dups_size":0,"ondisk_log_size":1536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085295030000000008,"stat_sum":{"num_bytes":15445695,"num_objects":634,"num_object_clones":293,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":231,"num_read":101,"num_read_kb":102,"num_write":1284,"num_write_kb":21892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'1485","reported_seq":3428,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.435605+0000","last_change":"2023-10-30T13:21:40.573274+0000","last_active":"2023-10-30T13:52:29.435605+0000","last_peered":"2023-10-30T13:52:29.435605+0000","last_clean":"2023-10-30T13:52:29.435605+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:52:29.435605+0000","last_undegraded":"2023-10-30T13:52:29.435605+0000","last_fullsized":"2023-10-30T13:52:29.435605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004933977,"stat_sum":{"num_bytes":6328355,"num_objects":596,"num_object_clones":272,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":210,"num_read":97,"num_read_kb":4194,"num_write":1284,"num_write_kb":15784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'2163","reported_seq":4144,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:46.206549+0000","last_change":"2023-10-30T13:52:46.206609+0000","last_active":"2023-10-30T13:52:46.206549+0000","last_peered":"2023-10-30T13:52:46.206549+0000","last_clean":"2023-10-30T13:52:46.206549+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:52:46.206549+0000","last_undegraded":"2023-10-30T13:52:46.206549+0000","last_fullsized":"2023-10-30T13:52:46.206549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2163","last_scrub_stamp":"2023-10-30T13:52:46.206474+0000","last_deep_scrub":"67'2163","last_deep_scrub_stamp":"2023-10-30T13:52:46.206474+0000","last_clean_scrub_stamp":"2023-10-30T13:52:46.206474+0000","objects_scrubbed":610,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:31.417387+0000","scrub_duration":1.043425775,"objects_trimmed":0,"snaptrim_duration":0.0029909680000000001,"stat_sum":{"num_bytes":31670070,"num_objects":610,"num_object_clones":280,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":227,"num_read":110,"num_read_kb":16437,"num_write":1314,"num_write_kb":33491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'1570","reported_seq":3511,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:33.487307+0000","last_change":"2023-10-30T13:21:40.571731+0000","last_active":"2023-10-30T13:52:33.487307+0000","last_peered":"2023-10-30T13:52:33.487307+0000","last_clean":"2023-10-30T13:52:33.487307+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:52:33.487307+0000","last_undegraded":"2023-10-30T13:52:33.487307+0000","last_fullsized":"2023-10-30T13:52:33.487307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058939090000000001,"stat_sum":{"num_bytes":14342306,"num_objects":613,"num_object_clones":284,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":231,"num_read":106,"num_read_kb":2953,"num_write":1333,"num_write_kb":15925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'2459","reported_seq":4355,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:46.948880+0000","last_change":"2023-10-30T13:52:46.948942+0000","last_active":"2023-10-30T13:52:46.948880+0000","last_peered":"2023-10-30T13:52:46.948880+0000","last_clean":"2023-10-30T13:52:46.948880+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:52:46.948880+0000","last_undegraded":"2023-10-30T13:52:46.948880+0000","last_fullsized":"2023-10-30T13:52:46.948880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2459","last_scrub_stamp":"2023-10-30T13:52:46.948803+0000","last_deep_scrub":"67'2459","last_deep_scrub_stamp":"2023-10-30T13:52:46.948803+0000","last_clean_scrub_stamp":"2023-10-30T13:52:46.948803+0000","objects_scrubbed":703,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:36:20.176163+0000","scrub_duration":1.765876443,"objects_trimmed":0,"snaptrim_duration":0.0093243890000000006,"stat_sum":{"num_bytes":10965780,"num_objects":703,"num_object_clones":374,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":231,"num_read":115,"num_read_kb":1694,"num_write":1434,"num_write_kb":17364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'2223","reported_seq":4184,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.397041+0000","last_change":"2023-10-30T13:52:51.397109+0000","last_active":"2023-10-30T13:52:51.397041+0000","last_peered":"2023-10-30T13:52:51.397041+0000","last_clean":"2023-10-30T13:52:51.397041+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:52:51.397041+0000","last_undegraded":"2023-10-30T13:52:51.397041+0000","last_fullsized":"2023-10-30T13:52:51.397041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2223","last_scrub_stamp":"2023-10-30T13:52:51.396947+0000","last_deep_scrub":"67'2223","last_deep_scrub_stamp":"2023-10-30T13:52:51.396947+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.396947+0000","objects_scrubbed":632,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:49.530995+0000","scrub_duration":1.1860778729999999,"objects_trimmed":0,"snaptrim_duration":0.011284880000000001,"stat_sum":{"num_bytes":18442682,"num_objects":632,"num_object_clones":291,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":228,"num_read":100,"num_read_kb":8212,"num_write":1340,"num_write_kb":24815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'1476","reported_seq":3416,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:34.002389+0000","last_change":"2023-10-30T13:21:40.567529+0000","last_active":"2023-10-30T13:52:34.002389+0000","last_peered":"2023-10-30T13:52:34.002389+0000","last_clean":"2023-10-30T13:52:34.002389+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:52:34.002389+0000","last_undegraded":"2023-10-30T13:52:34.002389+0000","last_fullsized":"2023-10-30T13:52:34.002389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002295727,"stat_sum":{"num_bytes":22547199,"num_objects":575,"num_object_clones":267,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":207,"num_read":89,"num_read_kb":4127,"num_write":1240,"num_write_kb":24211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1630","reported_seq":3549,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.437160+0000","last_change":"2023-10-30T13:21:40.572640+0000","last_active":"2023-10-30T13:52:29.437160+0000","last_peered":"2023-10-30T13:52:29.437160+0000","last_clean":"2023-10-30T13:52:29.437160+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:52:29.437160+0000","last_undegraded":"2023-10-30T13:52:29.437160+0000","last_fullsized":"2023-10-30T13:52:29.437160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1630,"log_dups_size":0,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061689539999999999,"stat_sum":{"num_bytes":11703479,"num_objects":649,"num_object_clones":303,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":237,"num_read":106,"num_read_kb":7307,"num_write":1383,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"67'1568","reported_seq":3507,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.435729+0000","last_change":"2023-10-30T13:21:40.580324+0000","last_active":"2023-10-30T13:52:29.435729+0000","last_peered":"2023-10-30T13:52:29.435729+0000","last_clean":"2023-10-30T13:52:29.435729+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:52:29.435729+0000","last_undegraded":"2023-10-30T13:52:29.435729+0000","last_fullsized":"2023-10-30T13:52:29.435729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094394720000000008,"stat_sum":{"num_bytes":31771338,"num_objects":621,"num_object_clones":283,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":229,"num_read":103,"num_read_kb":4126,"num_write":1340,"num_write_kb":42793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'2081","reported_seq":4057,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.440606+0000","last_change":"2023-10-30T13:52:49.440653+0000","last_active":"2023-10-30T13:52:49.440606+0000","last_peered":"2023-10-30T13:52:49.440606+0000","last_clean":"2023-10-30T13:52:49.440606+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:52:49.440606+0000","last_undegraded":"2023-10-30T13:52:49.440606+0000","last_fullsized":"2023-10-30T13:52:49.440606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:52:49.440530+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:52:49.440530+0000","last_clean_scrub_stamp":"2023-10-30T13:52:49.440530+0000","objects_scrubbed":595,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:04.396223+0000","scrub_duration":1.3029579039999999,"objects_trimmed":0,"snaptrim_duration":0.0027436349999999999,"stat_sum":{"num_bytes":6210074,"num_objects":595,"num_object_clones":273,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":228,"num_read":98,"num_read_kb":33,"num_write":1261,"num_write_kb":10653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'2305","reported_seq":4250,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.423134+0000","last_change":"2023-10-30T13:52:45.423196+0000","last_active":"2023-10-30T13:52:45.423134+0000","last_peered":"2023-10-30T13:52:45.423134+0000","last_clean":"2023-10-30T13:52:45.423134+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:52:45.423134+0000","last_undegraded":"2023-10-30T13:52:45.423134+0000","last_fullsized":"2023-10-30T13:52:45.423134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2305","last_scrub_stamp":"2023-10-30T13:52:45.423033+0000","last_deep_scrub":"67'2305","last_deep_scrub_stamp":"2023-10-30T13:52:45.423033+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.423033+0000","objects_scrubbed":670,"log_size":2305,"log_dups_size":0,"ondisk_log_size":2305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:15.141666+0000","scrub_duration":1.276721341,"objects_trimmed":0,"snaptrim_duration":0.0082328149999999992,"stat_sum":{"num_bytes":28398973,"num_objects":670,"num_object_clones":311,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":248,"num_read":102,"num_read_kb":120,"num_write":1396,"num_write_kb":38562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'1544","reported_seq":3482,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.416495+0000","last_change":"2023-10-30T13:21:40.571762+0000","last_active":"2023-10-30T13:52:29.416495+0000","last_peered":"2023-10-30T13:52:29.416495+0000","last_clean":"2023-10-30T13:52:29.416495+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:52:29.416495+0000","last_undegraded":"2023-10-30T13:52:29.416495+0000","last_fullsized":"2023-10-30T13:52:29.416495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1544,"log_dups_size":0,"ondisk_log_size":1544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037447470000000001,"stat_sum":{"num_bytes":19407388,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":225,"num_read":103,"num_read_kb":4138,"num_write":1300,"num_write_kb":24714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'1586","reported_seq":3527,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.458699+0000","last_change":"2023-10-30T13:21:40.536309+0000","last_active":"2023-10-30T13:52:29.458699+0000","last_peered":"2023-10-30T13:52:29.458699+0000","last_clean":"2023-10-30T13:52:29.458699+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:52:29.458699+0000","last_undegraded":"2023-10-30T13:52:29.458699+0000","last_fullsized":"2023-10-30T13:52:29.458699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1586,"log_dups_size":0,"ondisk_log_size":1586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091633449999999998,"stat_sum":{"num_bytes":10552296,"num_objects":616,"num_object_clones":294,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":228,"num_read":115,"num_read_kb":24,"num_write":1342,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1421","reported_seq":3381,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.594953+0000","last_change":"2023-10-30T13:21:40.571783+0000","last_active":"2023-10-30T13:52:29.594953+0000","last_peered":"2023-10-30T13:52:29.594953+0000","last_clean":"2023-10-30T13:52:29.594953+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:52:29.594953+0000","last_undegraded":"2023-10-30T13:52:29.594953+0000","last_fullsized":"2023-10-30T13:52:29.594953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059368210000000001,"stat_sum":{"num_bytes":15196336,"num_objects":556,"num_object_clones":256,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":202,"num_read":99,"num_read_kb":1380,"num_write":1203,"num_write_kb":16993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'1585","reported_seq":3534,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.601898+0000","last_change":"2023-10-30T13:21:40.581115+0000","last_active":"2023-10-30T13:52:29.601898+0000","last_peered":"2023-10-30T13:52:29.601898+0000","last_clean":"2023-10-30T13:52:29.601898+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:52:29.601898+0000","last_undegraded":"2023-10-30T13:52:29.601898+0000","last_fullsized":"2023-10-30T13:52:29.601898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1585,"log_dups_size":0,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099065669999999998,"stat_sum":{"num_bytes":9983325,"num_objects":619,"num_object_clones":286,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":221,"num_read":115,"num_read_kb":68,"num_write":1330,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1543","reported_seq":3487,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.436117+0000","last_change":"2023-10-30T13:21:40.572631+0000","last_active":"2023-10-30T13:52:29.436117+0000","last_peered":"2023-10-30T13:52:29.436117+0000","last_clean":"2023-10-30T13:52:29.436117+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:52:29.436117+0000","last_undegraded":"2023-10-30T13:52:29.436117+0000","last_fullsized":"2023-10-30T13:52:29.436117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1543,"log_dups_size":0,"ondisk_log_size":1543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059382619999999997,"stat_sum":{"num_bytes":23775876,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":236,"num_read":113,"num_read_kb":4145,"num_write":1301,"num_write_kb":26615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'1478","reported_seq":3407,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:30.126045+0000","last_change":"2023-10-30T13:21:40.575712+0000","last_active":"2023-10-30T13:52:30.126045+0000","last_peered":"2023-10-30T13:52:30.126045+0000","last_clean":"2023-10-30T13:52:30.126045+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:52:30.126045+0000","last_undegraded":"2023-10-30T13:52:30.126045+0000","last_fullsized":"2023-10-30T13:52:30.126045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1478,"log_dups_size":0,"ondisk_log_size":1478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053787089999999997,"stat_sum":{"num_bytes":10833053,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":213,"num_read":88,"num_read_kb":3300,"num_write":1256,"num_write_kb":16238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'1651","reported_seq":3551,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.444055+0000","last_change":"2023-10-30T13:21:40.529120+0000","last_active":"2023-10-30T13:52:29.444055+0000","last_peered":"2023-10-30T13:52:29.444055+0000","last_clean":"2023-10-30T13:52:29.444055+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:52:29.444055+0000","last_undegraded":"2023-10-30T13:52:29.444055+0000","last_fullsized":"2023-10-30T13:52:29.444055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1651,"log_dups_size":0,"ondisk_log_size":1651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028849399999999999,"stat_sum":{"num_bytes":11995550,"num_objects":688,"num_object_clones":321,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":244,"num_read":100,"num_read_kb":8267,"num_write":1414,"num_write_kb":14306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'2013","reported_seq":3996,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:52.418032+0000","last_change":"2023-10-30T13:52:52.418138+0000","last_active":"2023-10-30T13:52:52.418032+0000","last_peered":"2023-10-30T13:52:52.418032+0000","last_clean":"2023-10-30T13:52:52.418032+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:52:52.418032+0000","last_undegraded":"2023-10-30T13:52:52.418032+0000","last_fullsized":"2023-10-30T13:52:52.418032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2013","last_scrub_stamp":"2023-10-30T13:52:52.417929+0000","last_deep_scrub":"67'2013","last_deep_scrub_stamp":"2023-10-30T13:52:52.417929+0000","last_clean_scrub_stamp":"2023-10-30T13:52:52.417929+0000","objects_scrubbed":558,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:39:32.467346+0000","scrub_duration":1.1806941150000001,"objects_trimmed":0,"snaptrim_duration":0.0096547909999999994,"stat_sum":{"num_bytes":15074071,"num_objects":558,"num_object_clones":259,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":221,"num_read":91,"num_read_kb":32,"num_write":1204,"num_write_kb":16754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'1673","reported_seq":3614,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.374204+0000","last_change":"2023-10-30T13:21:40.567248+0000","last_active":"2023-10-30T13:52:29.374204+0000","last_peered":"2023-10-30T13:52:29.374204+0000","last_clean":"2023-10-30T13:52:29.374204+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:52:29.374204+0000","last_undegraded":"2023-10-30T13:52:29.374204+0000","last_fullsized":"2023-10-30T13:52:29.374204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018845820000000001,"stat_sum":{"num_bytes":10798567,"num_objects":649,"num_object_clones":300,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":241,"num_read":120,"num_read_kb":27,"num_write":1443,"num_write_kb":17235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1515","reported_seq":3486,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.432421+0000","last_change":"2023-10-30T13:21:40.575268+0000","last_active":"2023-10-30T13:52:29.432421+0000","last_peered":"2023-10-30T13:52:29.432421+0000","last_clean":"2023-10-30T13:52:29.432421+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:52:29.432421+0000","last_undegraded":"2023-10-30T13:52:29.432421+0000","last_fullsized":"2023-10-30T13:52:29.432421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075239010000000004,"stat_sum":{"num_bytes":15423893,"num_objects":576,"num_object_clones":259,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":203,"num_read":109,"num_read_kb":9552,"num_write":1308,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'2174","reported_seq":4153,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:47.980889+0000","last_change":"2023-10-30T13:52:47.980946+0000","last_active":"2023-10-30T13:52:47.980889+0000","last_peered":"2023-10-30T13:52:47.980889+0000","last_clean":"2023-10-30T13:52:47.980889+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:52:47.980889+0000","last_undegraded":"2023-10-30T13:52:47.980889+0000","last_fullsized":"2023-10-30T13:52:47.980889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2174","last_scrub_stamp":"2023-10-30T13:52:47.980802+0000","last_deep_scrub":"67'2174","last_deep_scrub_stamp":"2023-10-30T13:52:47.980802+0000","last_clean_scrub_stamp":"2023-10-30T13:52:47.980802+0000","objects_scrubbed":622,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:54:24.281648+0000","scrub_duration":1.0857265140000001,"objects_trimmed":0,"snaptrim_duration":0.007356301,"stat_sum":{"num_bytes":40089180,"num_objects":622,"num_object_clones":287,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":230,"num_read":115,"num_read_kb":33,"num_write":1312,"num_write_kb":42997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'644","reported_seq":882,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.019678+0000","last_change":"2023-10-30T13:52:45.019766+0000","last_active":"2023-10-30T13:52:45.019678+0000","last_peered":"2023-10-30T13:52:45.019678+0000","last_clean":"2023-10-30T13:52:45.019678+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:52:45.019678+0000","last_undegraded":"2023-10-30T13:52:45.019678+0000","last_fullsized":"2023-10-30T13:52:45.019678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'644","last_scrub_stamp":"2023-10-30T13:52:45.019575+0000","last_deep_scrub":"67'644","last_deep_scrub_stamp":"2023-10-30T13:52:45.019575+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.019575+0000","objects_scrubbed":42,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:11:59.207979+0000","scrub_duration":0.065999268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":252,"num_read_kb":249243,"num_write":662,"num_write_kb":74846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":549256,"num_omap_keys":974,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1560","reported_seq":3488,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.433305+0000","last_change":"2023-10-30T13:21:40.529443+0000","last_active":"2023-10-30T13:52:29.433305+0000","last_peered":"2023-10-30T13:52:29.433305+0000","last_clean":"2023-10-30T13:52:29.433305+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:52:29.433305+0000","last_undegraded":"2023-10-30T13:52:29.433305+0000","last_fullsized":"2023-10-30T13:52:29.433305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372290000000001,"stat_sum":{"num_bytes":18713903,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":229,"num_read":108,"num_read_kb":4667,"num_write":1329,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1176","reported_seq":1510,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.939028+0000","last_change":"2023-10-30T13:52:43.939106+0000","last_active":"2023-10-30T13:52:43.939028+0000","last_peered":"2023-10-30T13:52:43.939028+0000","last_clean":"2023-10-30T13:52:43.939028+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:52:43.939028+0000","last_undegraded":"2023-10-30T13:52:43.939028+0000","last_fullsized":"2023-10-30T13:52:43.939028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1176","last_scrub_stamp":"2023-10-30T13:52:43.938938+0000","last_deep_scrub":"67'1176","last_deep_scrub_stamp":"2023-10-30T13:52:43.938938+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.938938+0000","objects_scrubbed":51,"log_size":1176,"log_dups_size":0,"ondisk_log_size":1176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:18.962998+0000","scrub_duration":0.19282650300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25293488,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":404,"num_read_kb":309242,"num_write":1231,"num_write_kb":84663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":763115,"num_omap_keys":1100,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'794","reported_seq":1183,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:35.070009+0000","last_change":"2023-10-30T13:52:35.070059+0000","last_active":"2023-10-30T13:52:35.070009+0000","last_peered":"2023-10-30T13:52:35.070009+0000","last_clean":"2023-10-30T13:52:35.070009+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:52:35.070009+0000","last_undegraded":"2023-10-30T13:52:35.070009+0000","last_fullsized":"2023-10-30T13:52:35.070009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'794","last_scrub_stamp":"2023-10-30T13:52:35.069918+0000","last_deep_scrub":"67'794","last_deep_scrub_stamp":"2023-10-30T13:52:35.069918+0000","last_clean_scrub_stamp":"2023-10-30T13:52:35.069918+0000","objects_scrubbed":54,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:21.898394+0000","scrub_duration":0.14371538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397151,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":403,"num_read_kb":153383,"num_write":820,"num_write_kb":55113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":519482,"num_omap_keys":897,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'2508","reported_seq":4463,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:50.977937+0000","last_change":"2023-10-30T13:52:50.978041+0000","last_active":"2023-10-30T13:52:50.977937+0000","last_peered":"2023-10-30T13:52:50.977937+0000","last_clean":"2023-10-30T13:52:50.977937+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:52:50.977937+0000","last_undegraded":"2023-10-30T13:52:50.977937+0000","last_fullsized":"2023-10-30T13:52:50.977937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2508","last_scrub_stamp":"2023-10-30T13:52:50.977836+0000","last_deep_scrub":"67'2508","last_deep_scrub_stamp":"2023-10-30T13:52:50.977836+0000","last_clean_scrub_stamp":"2023-10-30T13:52:50.977836+0000","objects_scrubbed":729,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:26.785649+0000","scrub_duration":1.0834898019999999,"objects_trimmed":0,"snaptrim_duration":0.0064075410000000001,"stat_sum":{"num_bytes":11666119,"num_objects":729,"num_object_clones":341,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":261,"num_read":141,"num_read_kb":4175,"num_write":1520,"num_write_kb":22588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1448","reported_seq":1728,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.318938+0000","last_change":"2023-10-30T13:52:43.318938+0000","last_active":"2023-10-30T13:52:43.318938+0000","last_peered":"2023-10-30T13:52:43.318938+0000","last_clean":"2023-10-30T13:52:43.318938+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:52:43.318938+0000","last_undegraded":"2023-10-30T13:52:43.318938+0000","last_fullsized":"2023-10-30T13:52:43.318938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1447","last_scrub_stamp":"2023-10-30T13:52:43.318712+0000","last_deep_scrub":"67'1447","last_deep_scrub_stamp":"2023-10-30T13:52:43.318712+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.318712+0000","objects_scrubbed":54,"log_size":1448,"log_dups_size":0,"ondisk_log_size":1448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:38.717163+0000","scrub_duration":0.18293932199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":354,"num_read_kb":270680,"num_write":1488,"num_write_kb":75460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593681,"num_omap_keys":1037,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1514","reported_seq":3448,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.440807+0000","last_change":"2023-10-30T13:21:40.571518+0000","last_active":"2023-10-30T13:52:29.440807+0000","last_peered":"2023-10-30T13:52:29.440807+0000","last_clean":"2023-10-30T13:52:29.440807+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:52:29.440807+0000","last_undegraded":"2023-10-30T13:52:29.440807+0000","last_fullsized":"2023-10-30T13:52:29.440807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1514,"log_dups_size":0,"ondisk_log_size":1514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058964639999999997,"stat_sum":{"num_bytes":19507019,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":240,"num_read":105,"num_read_kb":5362,"num_write":1250,"num_write_kb":24752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'957","reported_seq":1194,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.234381+0000","last_change":"2023-10-30T13:10:33.725176+0000","last_active":"2023-10-30T13:52:36.234381+0000","last_peered":"2023-10-30T13:52:36.234381+0000","last_clean":"2023-10-30T13:52:36.234381+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:52:36.234381+0000","last_undegraded":"2023-10-30T13:52:36.234381+0000","last_fullsized":"2023-10-30T13:52:36.234381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":957,"log_dups_size":0,"ondisk_log_size":957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":313,"num_read_kb":141404,"num_write":1078,"num_write_kb":63728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'1493","reported_seq":3447,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.403063+0000","last_change":"2023-10-30T13:21:40.567796+0000","last_active":"2023-10-30T13:52:29.403063+0000","last_peered":"2023-10-30T13:52:29.403063+0000","last_clean":"2023-10-30T13:52:29.403063+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:52:29.403063+0000","last_undegraded":"2023-10-30T13:52:29.403063+0000","last_fullsized":"2023-10-30T13:52:29.403063+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1493,"log_dups_size":0,"ondisk_log_size":1493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025817129999999998,"stat_sum":{"num_bytes":23539910,"num_objects":588,"num_object_clones":267,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":208,"num_read":106,"num_read_kb":8313,"num_write":1266,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"67'1170","reported_seq":1398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:39.386385+0000","last_change":"2023-10-30T13:52:39.386488+0000","last_active":"2023-10-30T13:52:39.386385+0000","last_peered":"2023-10-30T13:52:39.386385+0000","last_clean":"2023-10-30T13:52:39.386385+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:52:39.386385+0000","last_undegraded":"2023-10-30T13:52:39.386385+0000","last_fullsized":"2023-10-30T13:52:39.386385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1170","last_scrub_stamp":"2023-10-30T13:52:39.386286+0000","last_deep_scrub":"67'1170","last_deep_scrub_stamp":"2023-10-30T13:52:39.386286+0000","last_clean_scrub_stamp":"2023-10-30T13:52:39.386286+0000","objects_scrubbed":49,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:00.965375+0000","scrub_duration":0.15933681099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":240,"num_read_kb":201173,"num_write":1196,"num_write_kb":66729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":467606,"num_omap_keys":916,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'1557","reported_seq":3469,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.485414+0000","last_change":"2023-10-30T13:21:40.528173+0000","last_active":"2023-10-30T13:52:29.485414+0000","last_peered":"2023-10-30T13:52:29.485414+0000","last_clean":"2023-10-30T13:52:29.485414+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:52:29.485414+0000","last_undegraded":"2023-10-30T13:52:29.485414+0000","last_fullsized":"2023-10-30T13:52:29.485414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1557,"log_dups_size":0,"ondisk_log_size":1557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022874710000000001,"stat_sum":{"num_bytes":16137402,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":233,"num_read":91,"num_read_kb":34,"num_write":1322,"num_write_kb":18013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1315","reported_seq":1557,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:25.968871+0000","last_change":"2023-10-30T13:10:33.921704+0000","last_active":"2023-10-30T13:52:25.968871+0000","last_peered":"2023-10-30T13:52:25.968871+0000","last_clean":"2023-10-30T13:52:25.968871+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:52:25.968871+0000","last_undegraded":"2023-10-30T13:52:25.968871+0000","last_fullsized":"2023-10-30T13:52:25.968871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20972192,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":307,"num_read_kb":165804,"num_write":1420,"num_write_kb":50966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1574","reported_seq":3508,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:35.112140+0000","last_change":"2023-10-30T13:21:40.538439+0000","last_active":"2023-10-30T13:52:35.112140+0000","last_peered":"2023-10-30T13:52:35.112140+0000","last_clean":"2023-10-30T13:52:35.112140+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:52:35.112140+0000","last_undegraded":"2023-10-30T13:52:35.112140+0000","last_fullsized":"2023-10-30T13:52:35.112140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087798259999999993,"stat_sum":{"num_bytes":32677996,"num_objects":638,"num_object_clones":293,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":232,"num_read":108,"num_read_kb":10951,"num_write":1342,"num_write_kb":38819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'985","reported_seq":1315,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.944257+0000","last_change":"2023-10-30T13:52:51.944325+0000","last_active":"2023-10-30T13:52:51.944257+0000","last_peered":"2023-10-30T13:52:51.944257+0000","last_clean":"2023-10-30T13:52:51.944257+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:52:51.944257+0000","last_undegraded":"2023-10-30T13:52:51.944257+0000","last_fullsized":"2023-10-30T13:52:51.944257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'985","last_scrub_stamp":"2023-10-30T13:52:51.944137+0000","last_deep_scrub":"67'985","last_deep_scrub_stamp":"2023-10-30T13:52:51.944137+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.944137+0000","objects_scrubbed":50,"log_size":985,"log_dups_size":0,"ondisk_log_size":985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:12.000749+0000","scrub_duration":0.18964769300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":413,"num_read_kb":281405,"num_write":1059,"num_write_kb":80202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":643541,"num_omap_keys":878,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1512","reported_seq":3428,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.577780+0000","last_change":"2023-10-30T13:21:40.583356+0000","last_active":"2023-10-30T13:52:29.577780+0000","last_peered":"2023-10-30T13:52:29.577780+0000","last_clean":"2023-10-30T13:52:29.577780+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:52:29.577780+0000","last_undegraded":"2023-10-30T13:52:29.577780+0000","last_fullsized":"2023-10-30T13:52:29.577780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014787140000000001,"stat_sum":{"num_bytes":19072740,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":238,"num_read":87,"num_read_kb":102,"num_write":1252,"num_write_kb":21862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'1144","reported_seq":1482,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:37.361974+0000","last_change":"2023-10-30T13:52:37.361974+0000","last_active":"2023-10-30T13:52:37.361974+0000","last_peered":"2023-10-30T13:52:37.361974+0000","last_clean":"2023-10-30T13:52:37.361974+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:52:37.361974+0000","last_undegraded":"2023-10-30T13:52:37.361974+0000","last_fullsized":"2023-10-30T13:52:37.361974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1144","last_scrub_stamp":"2023-10-30T13:52:37.361775+0000","last_deep_scrub":"67'1144","last_deep_scrub_stamp":"2023-10-30T13:52:37.361775+0000","last_clean_scrub_stamp":"2023-10-30T13:52:37.361775+0000","objects_scrubbed":60,"log_size":1144,"log_dups_size":0,"ondisk_log_size":1144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:58.622750+0000","scrub_duration":0.22738265299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748770,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":476,"num_read_kb":203988,"num_write":1202,"num_write_kb":67245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":644581,"num_omap_keys":1115,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'1607","reported_seq":3556,"reported_epoch":67,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T13:52:52.406224+0000","last_change":"2023-10-30T13:52:52.212460+0000","last_active":"2023-10-30T13:52:52.406224+0000","last_peered":"2023-10-30T13:52:52.406224+0000","last_clean":"2023-10-30T13:52:52.406224+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:52:52.406224+0000","last_undegraded":"2023-10-30T13:52:52.406224+0000","last_fullsized":"2023-10-30T13:52:52.406224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":98,"log_size":1607,"log_dups_size":0,"ondisk_log_size":1607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024076599999999998,"stat_sum":{"num_bytes":14342228,"num_objects":584,"num_object_clones":268,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":211,"num_read":101,"num_read_kb":4159,"num_write":1284,"num_write_kb":24386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'616","reported_seq":808,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.351480+0000","last_change":"2023-10-30T13:52:40.351638+0000","last_active":"2023-10-30T13:52:40.351480+0000","last_peered":"2023-10-30T13:52:40.351480+0000","last_clean":"2023-10-30T13:52:40.351480+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:52:40.351480+0000","last_undegraded":"2023-10-30T13:52:40.351480+0000","last_fullsized":"2023-10-30T13:52:40.351480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'616","last_scrub_stamp":"2023-10-30T13:52:40.351385+0000","last_deep_scrub":"67'616","last_deep_scrub_stamp":"2023-10-30T13:52:40.351385+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.351385+0000","objects_scrubbed":45,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:25.211671+0000","scrub_duration":0.16031920099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":203,"num_read_kb":83137,"num_write":663,"num_write_kb":45991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":437141,"num_omap_keys":850,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1604","reported_seq":3581,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.789128+0000","last_change":"2023-10-30T13:21:40.579925+0000","last_active":"2023-10-30T13:52:36.789128+0000","last_peered":"2023-10-30T13:52:36.789128+0000","last_clean":"2023-10-30T13:52:36.789128+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:52:36.789128+0000","last_undegraded":"2023-10-30T13:52:36.789128+0000","last_fullsized":"2023-10-30T13:52:36.789128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1604,"log_dups_size":0,"ondisk_log_size":1604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011943199999999999,"stat_sum":{"num_bytes":18918293,"num_objects":634,"num_object_clones":295,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":241,"num_read":154,"num_read_kb":114,"num_write":1356,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'501","reported_seq":735,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.234298+0000","last_change":"2023-10-30T13:10:33.724173+0000","last_active":"2023-10-30T13:52:36.234298+0000","last_peered":"2023-10-30T13:52:36.234298+0000","last_clean":"2023-10-30T13:52:36.234298+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:52:36.234298+0000","last_undegraded":"2023-10-30T13:52:36.234298+0000","last_fullsized":"2023-10-30T13:52:36.234298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":289,"num_read_kb":121284,"num_write":557,"num_write_kb":58696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1640","reported_seq":3523,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.234545+0000","last_change":"2023-10-30T13:21:40.576918+0000","last_active":"2023-10-30T13:52:36.234545+0000","last_peered":"2023-10-30T13:52:36.234545+0000","last_clean":"2023-10-30T13:52:36.234545+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:52:36.234545+0000","last_undegraded":"2023-10-30T13:52:36.234545+0000","last_fullsized":"2023-10-30T13:52:36.234545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1640,"log_dups_size":0,"ondisk_log_size":1640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086123929999999994,"stat_sum":{"num_bytes":19792070,"num_objects":655,"num_object_clones":349,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":204,"num_read":113,"num_read_kb":142,"num_write":1348,"num_write_kb":24080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1016","reported_seq":1307,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.650095+0000","last_change":"2023-10-30T13:52:49.650144+0000","last_active":"2023-10-30T13:52:49.650095+0000","last_peered":"2023-10-30T13:52:49.650095+0000","last_clean":"2023-10-30T13:52:49.650095+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:52:49.650095+0000","last_undegraded":"2023-10-30T13:52:49.650095+0000","last_fullsized":"2023-10-30T13:52:49.650095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1016","last_scrub_stamp":"2023-10-30T13:52:49.650007+0000","last_deep_scrub":"67'1016","last_deep_scrub_stamp":"2023-10-30T13:52:49.650007+0000","last_clean_scrub_stamp":"2023-10-30T13:52:49.650007+0000","objects_scrubbed":55,"log_size":1016,"log_dups_size":0,"ondisk_log_size":1016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:38.507861+0000","scrub_duration":0.17591847999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36678727,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":465,"num_read_kb":133831,"num_write":1108,"num_write_kb":54499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":702779,"num_omap_keys":1002,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1563","reported_seq":3486,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.789042+0000","last_change":"2023-10-30T13:21:40.583214+0000","last_active":"2023-10-30T13:52:36.789042+0000","last_peered":"2023-10-30T13:52:36.789042+0000","last_clean":"2023-10-30T13:52:36.789042+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:52:36.789042+0000","last_undegraded":"2023-10-30T13:52:36.789042+0000","last_fullsized":"2023-10-30T13:52:36.789042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015028952999999999,"stat_sum":{"num_bytes":21253392,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":236,"num_read":95,"num_read_kb":12339,"num_write":1306,"num_write_kb":30718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'688","reported_seq":924,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.045822+0000","last_change":"2023-10-30T13:52:45.045919+0000","last_active":"2023-10-30T13:52:45.045822+0000","last_peered":"2023-10-30T13:52:45.045822+0000","last_clean":"2023-10-30T13:52:45.045822+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:52:45.045822+0000","last_undegraded":"2023-10-30T13:52:45.045822+0000","last_fullsized":"2023-10-30T13:52:45.045822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'688","last_scrub_stamp":"2023-10-30T13:52:45.045739+0000","last_deep_scrub":"67'688","last_deep_scrub_stamp":"2023-10-30T13:52:45.045739+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.045739+0000","objects_scrubbed":51,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:31:53.871616+0000","scrub_duration":0.14959883700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165846,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":249,"num_read_kb":148701,"num_write":778,"num_write_kb":54421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":420719,"num_omap_keys":809,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1567","reported_seq":3498,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.484160+0000","last_change":"2023-10-30T13:21:40.579365+0000","last_active":"2023-10-30T13:52:29.484160+0000","last_peered":"2023-10-30T13:52:29.484160+0000","last_clean":"2023-10-30T13:52:29.484160+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:52:29.484160+0000","last_undegraded":"2023-10-30T13:52:29.484160+0000","last_fullsized":"2023-10-30T13:52:29.484160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011682140000000001,"stat_sum":{"num_bytes":26941199,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":237,"num_read":103,"num_read_kb":4114,"num_write":1331,"num_write_kb":32609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'1113","reported_seq":1400,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:48.645056+0000","last_change":"2023-10-30T13:52:48.645108+0000","last_active":"2023-10-30T13:52:48.645056+0000","last_peered":"2023-10-30T13:52:48.645056+0000","last_clean":"2023-10-30T13:52:48.645056+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:52:48.645056+0000","last_undegraded":"2023-10-30T13:52:48.645056+0000","last_fullsized":"2023-10-30T13:52:48.645056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1113","last_scrub_stamp":"2023-10-30T13:52:48.644981+0000","last_deep_scrub":"67'1113","last_deep_scrub_stamp":"2023-10-30T13:52:48.644981+0000","last_clean_scrub_stamp":"2023-10-30T13:52:48.644981+0000","objects_scrubbed":63,"log_size":1113,"log_dups_size":0,"ondisk_log_size":1113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:56.952822+0000","scrub_duration":0.19793386600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":381,"num_read_kb":250673,"num_write":1157,"num_write_kb":87754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":620304,"num_omap_keys":1053,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1540","reported_seq":3483,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.519212+0000","last_change":"2023-10-30T13:21:40.585506+0000","last_active":"2023-10-30T13:52:29.519212+0000","last_peered":"2023-10-30T13:52:29.519212+0000","last_clean":"2023-10-30T13:52:29.519212+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:52:29.519212+0000","last_undegraded":"2023-10-30T13:52:29.519212+0000","last_fullsized":"2023-10-30T13:52:29.519212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014794043,"stat_sum":{"num_bytes":18888621,"num_objects":614,"num_object_clones":283,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":224,"num_read":107,"num_read_kb":4188,"num_write":1306,"num_write_kb":24240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'1229","reported_seq":1481,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:33.487158+0000","last_change":"2023-10-30T13:10:33.695078+0000","last_active":"2023-10-30T13:52:33.487158+0000","last_peered":"2023-10-30T13:52:33.487158+0000","last_clean":"2023-10-30T13:52:33.487158+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:52:33.487158+0000","last_undegraded":"2023-10-30T13:52:33.487158+0000","last_fullsized":"2023-10-30T13:52:33.487158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1229,"log_dups_size":0,"ondisk_log_size":1229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364039,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":288,"num_read_kb":200539,"num_write":1327,"num_write_kb":71054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'2176","reported_seq":4163,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:48.181004+0000","last_change":"2023-10-30T13:52:48.181151+0000","last_active":"2023-10-30T13:52:48.181004+0000","last_peered":"2023-10-30T13:52:48.181004+0000","last_clean":"2023-10-30T13:52:48.181004+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:52:48.181004+0000","last_undegraded":"2023-10-30T13:52:48.181004+0000","last_fullsized":"2023-10-30T13:52:48.181004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2176","last_scrub_stamp":"2023-10-30T13:52:48.180852+0000","last_deep_scrub":"67'2176","last_deep_scrub_stamp":"2023-10-30T13:52:48.180852+0000","last_clean_scrub_stamp":"2023-10-30T13:52:48.180852+0000","objects_scrubbed":600,"log_size":2176,"log_dups_size":0,"ondisk_log_size":2176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:11.947771+0000","scrub_duration":1.0044801649999999,"objects_trimmed":0,"snaptrim_duration":0.010788236,"stat_sum":{"num_bytes":10484856,"num_objects":600,"num_object_clones":281,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":220,"num_read":116,"num_read_kb":28,"num_write":1335,"num_write_kb":13295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'779","reported_seq":1027,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:34.002273+0000","last_change":"2023-10-30T13:10:33.716900+0000","last_active":"2023-10-30T13:52:34.002273+0000","last_peered":"2023-10-30T13:52:34.002273+0000","last_clean":"2023-10-30T13:52:34.002273+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:52:34.002273+0000","last_undegraded":"2023-10-30T13:52:34.002273+0000","last_fullsized":"2023-10-30T13:52:34.002273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":174067,"num_write":897,"num_write_kb":62874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1542","reported_seq":3474,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.487490+0000","last_change":"2023-10-30T13:21:40.583977+0000","last_active":"2023-10-30T13:52:29.487490+0000","last_peered":"2023-10-30T13:52:29.487490+0000","last_clean":"2023-10-30T13:52:29.487490+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:52:29.487490+0000","last_undegraded":"2023-10-30T13:52:29.487490+0000","last_fullsized":"2023-10-30T13:52:29.487490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015268978000000001,"stat_sum":{"num_bytes":11043591,"num_objects":616,"num_object_clones":291,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":221,"num_read":104,"num_read_kb":3707,"num_write":1322,"num_write_kb":16509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":244,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.159767+0000","last_change":"2023-10-30T13:52:40.159977+0000","last_active":"2023-10-30T13:52:40.159767+0000","last_peered":"2023-10-30T13:52:40.159767+0000","last_clean":"2023-10-30T13:52:40.159767+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:52:40.159767+0000","last_undegraded":"2023-10-30T13:52:40.159767+0000","last_fullsized":"2023-10-30T13:52:40.159767+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-30T13:52:40.159494+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-30T13:52:40.159494+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.159494+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:37:13.673701+0000","scrub_duration":0.017799540999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'670","reported_seq":889,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:42.315206+0000","last_change":"2023-10-30T13:52:42.315332+0000","last_active":"2023-10-30T13:52:42.315206+0000","last_peered":"2023-10-30T13:52:42.315206+0000","last_clean":"2023-10-30T13:52:42.315206+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:52:42.315206+0000","last_undegraded":"2023-10-30T13:52:42.315206+0000","last_fullsized":"2023-10-30T13:52:42.315206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'670","last_scrub_stamp":"2023-10-30T13:52:42.314975+0000","last_deep_scrub":"67'670","last_deep_scrub_stamp":"2023-10-30T13:52:42.314975+0000","last_clean_scrub_stamp":"2023-10-30T13:52:42.314975+0000","objects_scrubbed":53,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:16.143637+0000","scrub_duration":0.170007133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360178,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":232,"num_read_kb":148904,"num_write":701,"num_write_kb":62706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":385690,"num_omap_keys":753,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'1545","reported_seq":3459,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.465787+0000","last_change":"2023-10-30T13:21:40.574205+0000","last_active":"2023-10-30T13:52:29.465787+0000","last_peered":"2023-10-30T13:52:29.465787+0000","last_clean":"2023-10-30T13:52:29.465787+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:52:29.465787+0000","last_undegraded":"2023-10-30T13:52:29.465787+0000","last_fullsized":"2023-10-30T13:52:29.465787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1545,"log_dups_size":0,"ondisk_log_size":1545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057314540000000004,"stat_sum":{"num_bytes":22780711,"num_objects":633,"num_object_clones":299,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":242,"num_read":92,"num_read_kb":12,"num_write":1293,"num_write_kb":28237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"67'707","reported_seq":940,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.788801+0000","last_change":"2023-10-30T13:10:33.721788+0000","last_active":"2023-10-30T13:52:36.788801+0000","last_peered":"2023-10-30T13:52:36.788801+0000","last_clean":"2023-10-30T13:52:36.788801+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:52:36.788801+0000","last_undegraded":"2023-10-30T13:52:36.788801+0000","last_fullsized":"2023-10-30T13:52:36.788801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":707,"log_dups_size":0,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":256,"num_read_kb":247261,"num_write":740,"num_write_kb":70569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'2028","reported_seq":3997,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.341768+0000","last_change":"2023-10-30T13:52:40.341843+0000","last_active":"2023-10-30T13:52:40.341768+0000","last_peered":"2023-10-30T13:52:40.341768+0000","last_clean":"2023-10-30T13:52:40.341768+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:52:40.341768+0000","last_undegraded":"2023-10-30T13:52:40.341768+0000","last_fullsized":"2023-10-30T13:52:40.341768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2028","last_scrub_stamp":"2023-10-30T13:52:40.341675+0000","last_deep_scrub":"67'2028","last_deep_scrub_stamp":"2023-10-30T13:52:40.341675+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.341675+0000","objects_scrubbed":568,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:22:08.372774+0000","scrub_duration":1.158419098,"objects_trimmed":0,"snaptrim_duration":0.002406114,"stat_sum":{"num_bytes":14184993,"num_objects":568,"num_object_clones":263,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":210,"num_read":80,"num_read_kb":26,"num_write":1239,"num_write_kb":16314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'2090","reported_seq":3466,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.634617+0000","last_change":"2023-10-30T13:52:36.634675+0000","last_active":"2023-10-30T13:52:36.634617+0000","last_peered":"2023-10-30T13:52:36.634617+0000","last_clean":"2023-10-30T13:52:36.634617+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:52:36.634617+0000","last_undegraded":"2023-10-30T13:52:36.634617+0000","last_fullsized":"2023-10-30T13:52:36.634617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2090","last_scrub_stamp":"2023-10-30T13:52:36.634514+0000","last_deep_scrub":"67'2090","last_deep_scrub_stamp":"2023-10-30T13:52:36.634514+0000","last_clean_scrub_stamp":"2023-10-30T13:52:36.634514+0000","objects_scrubbed":53,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:20.160935+0000","scrub_duration":0.14709519600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41670866,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":1534,"num_read_kb":188566,"num_write":2090,"num_write_kb":83865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":477756,"num_omap_keys":784,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'1527","reported_seq":3470,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:33.487242+0000","last_change":"2023-10-30T13:21:40.568287+0000","last_active":"2023-10-30T13:52:33.487242+0000","last_peered":"2023-10-30T13:52:33.487242+0000","last_clean":"2023-10-30T13:52:33.487242+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:52:33.487242+0000","last_undegraded":"2023-10-30T13:52:33.487242+0000","last_fullsized":"2023-10-30T13:52:33.487242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002908281,"stat_sum":{"num_bytes":22417951,"num_objects":612,"num_object_clones":284,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":221,"num_read":105,"num_read_kb":44,"num_write":1298,"num_write_kb":23585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'1007","reported_seq":1293,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.427338+0000","last_change":"2023-10-30T13:52:36.427454+0000","last_active":"2023-10-30T13:52:36.427338+0000","last_peered":"2023-10-30T13:52:36.427338+0000","last_clean":"2023-10-30T13:52:36.427338+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:52:36.427338+0000","last_undegraded":"2023-10-30T13:52:36.427338+0000","last_fullsized":"2023-10-30T13:52:36.427338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1007","last_scrub_stamp":"2023-10-30T13:52:36.427140+0000","last_deep_scrub":"67'1007","last_deep_scrub_stamp":"2023-10-30T13:52:36.427140+0000","last_clean_scrub_stamp":"2023-10-30T13:52:36.427140+0000","objects_scrubbed":60,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:11.996639+0000","scrub_duration":0.17205899599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":355,"num_read_kb":252215,"num_write":1079,"num_write_kb":75669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":633823,"num_omap_keys":985,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'2106","reported_seq":4092,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.008269+0000","last_change":"2023-10-30T13:52:51.008354+0000","last_active":"2023-10-30T13:52:51.008269+0000","last_peered":"2023-10-30T13:52:51.008269+0000","last_clean":"2023-10-30T13:52:51.008269+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:52:51.008269+0000","last_undegraded":"2023-10-30T13:52:51.008269+0000","last_fullsized":"2023-10-30T13:52:51.008269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2106","last_scrub_stamp":"2023-10-30T13:52:51.008186+0000","last_deep_scrub":"67'2106","last_deep_scrub_stamp":"2023-10-30T13:52:51.008186+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.008186+0000","objects_scrubbed":575,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:38.209876+0000","scrub_duration":1.1485167089999999,"objects_trimmed":0,"snaptrim_duration":0.007710705,"stat_sum":{"num_bytes":23751577,"num_objects":575,"num_object_clones":266,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":199,"num_read":100,"num_read_kb":489,"num_write":1325,"num_write_kb":27587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'998","reported_seq":1249,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.917921+0000","last_change":"2023-10-30T13:52:43.917980+0000","last_active":"2023-10-30T13:52:43.917921+0000","last_peered":"2023-10-30T13:52:43.917921+0000","last_clean":"2023-10-30T13:52:43.917921+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:52:43.917921+0000","last_undegraded":"2023-10-30T13:52:43.917921+0000","last_fullsized":"2023-10-30T13:52:43.917921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'998","last_scrub_stamp":"2023-10-30T13:52:43.917841+0000","last_deep_scrub":"67'998","last_deep_scrub_stamp":"2023-10-30T13:52:43.917841+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.917841+0000","objects_scrubbed":56,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:41:43.976830+0000","scrub_duration":0.14471989900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":294,"num_read_kb":227180,"num_write":1039,"num_write_kb":87590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":525988,"num_omap_keys":913,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'1598","reported_seq":3509,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.473742+0000","last_change":"2023-10-30T13:21:40.539081+0000","last_active":"2023-10-30T13:52:29.473742+0000","last_peered":"2023-10-30T13:52:29.473742+0000","last_clean":"2023-10-30T13:52:29.473742+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:52:29.473742+0000","last_undegraded":"2023-10-30T13:52:29.473742+0000","last_fullsized":"2023-10-30T13:52:29.473742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1598,"log_dups_size":0,"ondisk_log_size":1598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010719574000000001,"stat_sum":{"num_bytes":35538242,"num_objects":650,"num_object_clones":301,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":239,"num_read":91,"num_read_kb":31,"num_write":1347,"num_write_kb":41412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'787","reported_seq":1013,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.142911+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:52:29.142911+0000","last_peered":"2023-10-30T13:52:29.142911+0000","last_clean":"2023-10-30T13:52:29.142911+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:52:29.142911+0000","last_undegraded":"2023-10-30T13:52:29.142911+0000","last_fullsized":"2023-10-30T13:52:29.142911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":258,"num_read_kb":177173,"num_write":845,"num_write_kb":62523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1551","reported_seq":3478,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.395707+0000","last_change":"2023-10-30T13:21:40.572129+0000","last_active":"2023-10-30T13:52:29.395707+0000","last_peered":"2023-10-30T13:52:29.395707+0000","last_clean":"2023-10-30T13:52:29.395707+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:52:29.395707+0000","last_undegraded":"2023-10-30T13:52:29.395707+0000","last_fullsized":"2023-10-30T13:52:29.395707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059276069999999997,"stat_sum":{"num_bytes":23389062,"num_objects":631,"num_object_clones":294,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":232,"num_read":100,"num_read_kb":25,"num_write":1313,"num_write_kb":24315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'916","reported_seq":1156,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:33.487117+0000","last_change":"2023-10-30T13:10:33.726198+0000","last_active":"2023-10-30T13:52:33.487117+0000","last_peered":"2023-10-30T13:52:33.487117+0000","last_clean":"2023-10-30T13:52:33.487117+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:52:33.487117+0000","last_undegraded":"2023-10-30T13:52:33.487117+0000","last_fullsized":"2023-10-30T13:52:33.487117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":916,"log_dups_size":0,"ondisk_log_size":916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":310,"num_read_kb":109156,"num_write":1496,"num_write_kb":61821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1421","reported_seq":3374,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.481476+0000","last_change":"2023-10-30T13:21:40.576235+0000","last_active":"2023-10-30T13:52:29.481476+0000","last_peered":"2023-10-30T13:52:29.481476+0000","last_clean":"2023-10-30T13:52:29.481476+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:52:29.481476+0000","last_undegraded":"2023-10-30T13:52:29.481476+0000","last_fullsized":"2023-10-30T13:52:29.481476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056940879999999999,"stat_sum":{"num_bytes":23042560,"num_objects":571,"num_object_clones":259,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":201,"num_read":92,"num_read_kb":75,"num_write":1201,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'1097","reported_seq":1341,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:34.750332+0000","last_change":"2023-10-30T13:10:33.725486+0000","last_active":"2023-10-30T13:52:34.750332+0000","last_peered":"2023-10-30T13:52:34.750332+0000","last_clean":"2023-10-30T13:52:34.750332+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:52:34.750332+0000","last_undegraded":"2023-10-30T13:52:34.750332+0000","last_fullsized":"2023-10-30T13:52:34.750332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":342,"num_read_kb":120967,"num_write":1233,"num_write_kb":51153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'1445","reported_seq":3407,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.940035+0000","last_change":"2023-10-30T13:21:40.529108+0000","last_active":"2023-10-30T13:52:29.940035+0000","last_peered":"2023-10-30T13:52:29.940035+0000","last_clean":"2023-10-30T13:52:29.940035+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:52:29.940035+0000","last_undegraded":"2023-10-30T13:52:29.940035+0000","last_fullsized":"2023-10-30T13:52:29.940035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029926589999999999,"stat_sum":{"num_bytes":30813966,"num_objects":572,"num_object_clones":263,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":205,"num_read":105,"num_read_kb":87,"num_write":1219,"num_write_kb":33525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'421","reported_seq":608,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:24.060242+0000","last_change":"2023-10-30T13:10:33.722848+0000","last_active":"2023-10-30T13:52:24.060242+0000","last_peered":"2023-10-30T13:52:24.060242+0000","last_clean":"2023-10-30T13:52:24.060242+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:52:24.060242+0000","last_undegraded":"2023-10-30T13:52:24.060242+0000","last_fullsized":"2023-10-30T13:52:24.060242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":209,"num_read_kb":100741,"num_write":482,"num_write_kb":29419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'1546","reported_seq":3501,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.447018+0000","last_change":"2023-10-30T13:21:40.533380+0000","last_active":"2023-10-30T13:52:29.447018+0000","last_peered":"2023-10-30T13:52:29.447018+0000","last_clean":"2023-10-30T13:52:29.447018+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:52:29.447018+0000","last_undegraded":"2023-10-30T13:52:29.447018+0000","last_fullsized":"2023-10-30T13:52:29.447018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057673200000000003,"stat_sum":{"num_bytes":13994934,"num_objects":595,"num_object_clones":279,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":215,"num_read":114,"num_read_kb":29,"num_write":1329,"num_write_kb":19694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'1274","reported_seq":1499,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.113426+0000","last_change":"2023-10-30T13:52:34.896255+0000","last_active":"2023-10-30T13:52:49.113426+0000","last_peered":"2023-10-30T13:52:49.113426+0000","last_clean":"2023-10-30T13:52:49.113426+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:52:49.113426+0000","last_undegraded":"2023-10-30T13:52:49.113426+0000","last_fullsized":"2023-10-30T13:52:49.113426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1266","last_scrub_stamp":"2023-10-30T13:52:34.896121+0000","last_deep_scrub":"67'1266","last_deep_scrub_stamp":"2023-10-30T13:52:34.896121+0000","last_clean_scrub_stamp":"2023-10-30T13:52:34.896121+0000","objects_scrubbed":51,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:43.920989+0000","scrub_duration":0.15482325699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24584186,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":251,"num_read_kb":99282,"num_write":1299,"num_write_kb":42036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":678730,"num_omap_keys":1151,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'1556","reported_seq":3510,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.450089+0000","last_change":"2023-10-30T13:21:40.575380+0000","last_active":"2023-10-30T13:52:29.450089+0000","last_peered":"2023-10-30T13:52:29.450089+0000","last_clean":"2023-10-30T13:52:29.450089+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:52:29.450089+0000","last_undegraded":"2023-10-30T13:52:29.450089+0000","last_fullsized":"2023-10-30T13:52:29.450089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1556,"log_dups_size":0,"ondisk_log_size":1556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0063570620000000001,"stat_sum":{"num_bytes":16293763,"num_objects":612,"num_object_clones":286,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":233,"num_read":120,"num_read_kb":127,"num_write":1302,"num_write_kb":21658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'946","reported_seq":1182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:35.887235+0000","last_change":"2023-10-30T13:52:35.887314+0000","last_active":"2023-10-30T13:52:35.887235+0000","last_peered":"2023-10-30T13:52:35.887235+0000","last_clean":"2023-10-30T13:52:35.887235+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:52:35.887235+0000","last_undegraded":"2023-10-30T13:52:35.887235+0000","last_fullsized":"2023-10-30T13:52:35.887235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'946","last_scrub_stamp":"2023-10-30T13:52:35.887142+0000","last_deep_scrub":"67'946","last_deep_scrub_stamp":"2023-10-30T13:52:35.887142+0000","last_clean_scrub_stamp":"2023-10-30T13:52:35.887142+0000","objects_scrubbed":46,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:06:00.273772+0000","scrub_duration":0.098447004000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":272,"num_read_kb":200216,"num_write":1137,"num_write_kb":59039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":497965,"num_omap_keys":763,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1437","reported_seq":3387,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.438795+0000","last_change":"2023-10-30T13:21:40.575515+0000","last_active":"2023-10-30T13:52:29.438795+0000","last_peered":"2023-10-30T13:52:29.438795+0000","last_clean":"2023-10-30T13:52:29.438795+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:52:29.438795+0000","last_undegraded":"2023-10-30T13:52:29.438795+0000","last_fullsized":"2023-10-30T13:52:29.438795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1437,"log_dups_size":0,"ondisk_log_size":1437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060201619999999999,"stat_sum":{"num_bytes":10033117,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":215,"num_read":92,"num_read_kb":24,"num_write":1204,"num_write_kb":13175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'1025","reported_seq":1291,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:44.395023+0000","last_change":"2023-10-30T13:52:44.395104+0000","last_active":"2023-10-30T13:52:44.395023+0000","last_peered":"2023-10-30T13:52:44.395023+0000","last_clean":"2023-10-30T13:52:44.395023+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:52:44.395023+0000","last_undegraded":"2023-10-30T13:52:44.395023+0000","last_fullsized":"2023-10-30T13:52:44.395023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1025","last_scrub_stamp":"2023-10-30T13:52:44.394896+0000","last_deep_scrub":"67'1025","last_deep_scrub_stamp":"2023-10-30T13:52:44.394896+0000","last_clean_scrub_stamp":"2023-10-30T13:52:44.394896+0000","objects_scrubbed":56,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:31.686901+0000","scrub_duration":0.17278753699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":318,"num_read_kb":202944,"num_write":1065,"num_write_kb":67614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":660981,"num_omap_keys":1062,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1468","reported_seq":3417,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.712417+0000","last_change":"2023-10-30T13:21:40.575093+0000","last_active":"2023-10-30T13:52:29.712417+0000","last_peered":"2023-10-30T13:52:29.712417+0000","last_clean":"2023-10-30T13:52:29.712417+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:52:29.712417+0000","last_undegraded":"2023-10-30T13:52:29.712417+0000","last_fullsized":"2023-10-30T13:52:29.712417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079253199999999996,"stat_sum":{"num_bytes":6591050,"num_objects":558,"num_object_clones":254,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":200,"num_read":83,"num_read_kb":63,"num_write":1255,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1498","reported_seq":3443,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.483360+0000","last_change":"2023-10-30T13:21:40.538321+0000","last_active":"2023-10-30T13:52:29.483360+0000","last_peered":"2023-10-30T13:52:29.483360+0000","last_clean":"2023-10-30T13:52:29.483360+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:52:29.483360+0000","last_undegraded":"2023-10-30T13:52:29.483360+0000","last_fullsized":"2023-10-30T13:52:29.483360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010603414,"stat_sum":{"num_bytes":9104816,"num_objects":585,"num_object_clones":276,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":204,"num_read":104,"num_read_kb":44,"num_write":1285,"num_write_kb":14447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'1055","reported_seq":1320,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.345047+0000","last_change":"2023-10-30T13:52:43.345144+0000","last_active":"2023-10-30T13:52:43.345047+0000","last_peered":"2023-10-30T13:52:43.345047+0000","last_clean":"2023-10-30T13:52:43.345047+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:52:43.345047+0000","last_undegraded":"2023-10-30T13:52:43.345047+0000","last_fullsized":"2023-10-30T13:52:43.345047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1055","last_scrub_stamp":"2023-10-30T13:52:43.344949+0000","last_deep_scrub":"67'1055","last_deep_scrub_stamp":"2023-10-30T13:52:43.344949+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.344949+0000","objects_scrubbed":55,"log_size":1055,"log_dups_size":0,"ondisk_log_size":1055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:59:10.108303+0000","scrub_duration":0.154960875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":337,"num_read_kb":187117,"num_write":1200,"num_write_kb":47577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706060,"num_omap_keys":1147,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'1627","reported_seq":3542,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.488911+0000","last_change":"2023-10-30T13:21:40.537202+0000","last_active":"2023-10-30T13:52:29.488911+0000","last_peered":"2023-10-30T13:52:29.488911+0000","last_clean":"2023-10-30T13:52:29.488911+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:52:29.488911+0000","last_undegraded":"2023-10-30T13:52:29.488911+0000","last_fullsized":"2023-10-30T13:52:29.488911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1627,"log_dups_size":0,"ondisk_log_size":1627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097894100000000001,"stat_sum":{"num_bytes":19085997,"num_objects":650,"num_object_clones":307,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":230,"num_read":105,"num_read_kb":202,"num_write":1384,"num_write_kb":24607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1314","reported_seq":1617,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.181718+0000","last_change":"2023-10-30T13:10:33.728769+0000","last_active":"2023-10-30T13:52:49.181718+0000","last_peered":"2023-10-30T13:52:49.181718+0000","last_clean":"2023-10-30T13:52:49.181718+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:52:49.181718+0000","last_undegraded":"2023-10-30T13:52:49.181718+0000","last_fullsized":"2023-10-30T13:52:49.181718+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":1314,"log_dups_size":0,"ondisk_log_size":1314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37615282,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":418,"num_read_kb":304041,"num_write":1417,"num_write_kb":92493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1511","reported_seq":3435,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.485244+0000","last_change":"2023-10-30T13:21:40.528576+0000","last_active":"2023-10-30T13:52:29.485244+0000","last_peered":"2023-10-30T13:52:29.485244+0000","last_clean":"2023-10-30T13:52:29.485244+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:52:29.485244+0000","last_undegraded":"2023-10-30T13:52:29.485244+0000","last_fullsized":"2023-10-30T13:52:29.485244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026802879999999999,"stat_sum":{"num_bytes":22843366,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":237,"num_read":96,"num_read_kb":23,"num_write":1281,"num_write_kb":25082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'822","reported_seq":1066,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:44.024796+0000","last_change":"2023-10-30T13:52:44.024796+0000","last_active":"2023-10-30T13:52:44.024796+0000","last_peered":"2023-10-30T13:52:44.024796+0000","last_clean":"2023-10-30T13:52:44.024796+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:52:44.024796+0000","last_undegraded":"2023-10-30T13:52:44.024796+0000","last_fullsized":"2023-10-30T13:52:44.024796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'822","last_scrub_stamp":"2023-10-30T13:52:44.024664+0000","last_deep_scrub":"67'822","last_deep_scrub_stamp":"2023-10-30T13:52:44.024664+0000","last_clean_scrub_stamp":"2023-10-30T13:52:44.024664+0000","objects_scrubbed":50,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:54.864042+0000","scrub_duration":0.11718745699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":280,"num_read_kb":212075,"num_write":835,"num_write_kb":74769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":365319,"num_omap_keys":701,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'1466","reported_seq":3422,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:34.567347+0000","last_change":"2023-10-30T13:21:40.576062+0000","last_active":"2023-10-30T13:52:34.567347+0000","last_peered":"2023-10-30T13:52:34.567347+0000","last_clean":"2023-10-30T13:52:34.567347+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:52:34.567347+0000","last_undegraded":"2023-10-30T13:52:34.567347+0000","last_fullsized":"2023-10-30T13:52:34.567347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055040289999999997,"stat_sum":{"num_bytes":14435337,"num_objects":589,"num_object_clones":269,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":212,"num_read":103,"num_read_kb":22,"num_write":1235,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'768","reported_seq":1042,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:34.271316+0000","last_change":"2023-10-30T13:52:34.271376+0000","last_active":"2023-10-30T13:52:34.271316+0000","last_peered":"2023-10-30T13:52:34.271316+0000","last_clean":"2023-10-30T13:52:34.271316+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:52:34.271316+0000","last_undegraded":"2023-10-30T13:52:34.271316+0000","last_fullsized":"2023-10-30T13:52:34.271316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'768","last_scrub_stamp":"2023-10-30T13:52:34.271229+0000","last_deep_scrub":"67'768","last_deep_scrub_stamp":"2023-10-30T13:52:34.271229+0000","last_clean_scrub_stamp":"2023-10-30T13:52:34.271229+0000","objects_scrubbed":64,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:00.506995+0000","scrub_duration":0.20155240999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749318,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":374,"num_read_kb":96620,"num_write":840,"num_write_kb":51020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":647968,"num_omap_keys":1098,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165182927,"num_objects":39302,"num_object_clones":18317,"num_object_copies":78604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39302,"num_whiteouts":14350,"num_read":6653,"num_read_kb":168434,"num_write":83476,"num_write_kb":1459733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424070144,"data_stored":2332381086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106839,"ondisk_log_size":106839,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":908666707,"num_objects":1701,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":11399,"num_read_kb":5963012,"num_write":34141,"num_write_kb":2074104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1411,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11942485,"num_omap_keys":19988,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1817477120,"data_stored":1817333414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31472,"ondisk_log_size":31472,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575363,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738412,"kb_used_data":576740,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93633428,"statfs":{"total":96636764160,"available":95880630272,"internally_reserved":0,"allocated":590581760,"data_stored":576956722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":165542347},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84499999999999997,"15min":0.95099999999999996},"min":{"1min":0.47999999999999998,"5min":0.41199999999999998,"15min":0.40899999999999997},"max":{"1min":1.288,"5min":1.321,"15min":15.381},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81899999999999995,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.315},"max":{"1min":1.212,"5min":1.8400000000000001,"15min":2.9510000000000001},"last":0.33700000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.83799999999999997,"15min":0.92800000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3240000000000001,"5min":1.79,"15min":15.271000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.89100000000000001,"15min":0.96199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.42999999999999999},"max":{"1min":1.2769999999999999,"5min":1.6000000000000001,"15min":15.228999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.92100000000000004,"15min":0.91300000000000003},"min":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.36299999999999999},"max":{"1min":1.3100000000000001,"5min":1.345,"15min":2.2429999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.89000000000000001,"15min":0.877},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.242,"5min":1.929,"15min":2.4540000000000002},"last":0.378}]},{"osd":3,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95599999999999996,"15min":1.0800000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.256,"5min":2.0249999999999999,"15min":15.324},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.96999999999999997,"15min":0.98599999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2629999999999999,"5min":1.323,"15min":1.637},"last":0.76300000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.091,"15min":1.1479999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.40899999999999997},"max":{"1min":1.3080000000000001,"5min":2.7869999999999999,"15min":15.172000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.093,"5min":1.0600000000000001,"15min":1.089},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.466,"5min":3.032,"15min":4.0209999999999999},"last":0.58899999999999997}]},{"osd":5,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1120000000000001,"15min":1.1060000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.454,"5min":1.4770000000000001,"15min":1.7050000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.089,"5min":1.0600000000000001,"15min":1.0169999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.54500000000000004},"max":{"1min":1.4159999999999999,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":0.623}]},{"osd":6,"last update":"Mon Oct 30 13:51:57 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.1819999999999999,"15min":1.2509999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.46500000000000002},"max":{"1min":1.7829999999999999,"5min":3.0350000000000001,"15min":15.427},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.228,"15min":1.331},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.42199999999999999},"max":{"1min":1.841,"5min":3.5169999999999999,"15min":32.976999999999997},"last":0.72999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575362,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596080,"kb_used_data":452520,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93775760,"statfs":{"total":96636764160,"available":96026378240,"internally_reserved":0,"allocated":463380480,"data_stored":454781714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":146995655},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.82599999999999996,"15min":0.83999999999999997},"min":{"1min":0.53400000000000003,"5min":0.45000000000000001,"15min":0.40999999999999998},"max":{"1min":1.1890000000000001,"5min":1.3999999999999999,"15min":5.9880000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.877,"15min":0.88600000000000001},"min":{"1min":0.63500000000000001,"5min":0.48599999999999999,"15min":0.35999999999999999},"max":{"1min":1.1519999999999999,"5min":1.3160000000000001,"15min":6.0730000000000004},"last":0.53000000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.90900000000000003,"15min":0.86899999999999999},"min":{"1min":0.54000000000000004,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.153,"5min":6.9930000000000003,"15min":6.9930000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.85199999999999998,"15min":0.85499999999999998},"min":{"1min":0.67500000000000004,"5min":0.52500000000000002,"15min":0.47499999999999998},"max":{"1min":1.0620000000000001,"5min":1.2410000000000001,"15min":1.9810000000000001},"last":0.83899999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.91500000000000004,"15min":0.89700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.47699999999999998},"max":{"1min":1.141,"5min":6.8550000000000004,"15min":6.8550000000000004},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.95599999999999996,"15min":0.96199999999999997},"min":{"1min":0.63400000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2110000000000001,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.879}]},{"osd":3,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.56000000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.4830000000000001,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.98099999999999998,"15min":0.98399999999999999},"min":{"1min":0.72699999999999998,"5min":0.64400000000000002,"15min":0.629},"max":{"1min":1.3100000000000001,"5min":5.3179999999999996,"15min":5.3179999999999996},"last":0.68700000000000006}]},{"osd":4,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.98199999999999998,"15min":1.0369999999999999},"min":{"1min":0.752,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3129999999999999,"5min":1.4970000000000001,"15min":15.372},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.089,"15min":1.048},"min":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.58299999999999996},"max":{"1min":1.5820000000000001,"5min":5.2729999999999997,"15min":6.1580000000000004},"last":0.91700000000000004}]},{"osd":5,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.036,"15min":1.032},"min":{"1min":0.69399999999999995,"5min":0.60499999999999998,"15min":0.60199999999999998},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.8080000000000001},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.234,"5min":1.198,"15min":1.1839999999999999},"min":{"1min":0.93999999999999995,"5min":0.73899999999999999,"15min":0.67700000000000005},"max":{"1min":1.6519999999999999,"5min":1.6639999999999999,"15min":1.913},"last":1.0029999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.1080000000000001,"15min":1.0980000000000001},"min":{"1min":0.86899999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":2.202},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.22,"15min":1.2},"min":{"1min":0.99299999999999999,"5min":0.69599999999999995,"15min":0.42499999999999999},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":3.0590000000000002},"last":1.0349999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575363,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685108,"kb_used_data":519276,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93686732,"statfs":{"total":96636764160,"available":95935213568,"internally_reserved":0,"allocated":531738624,"data_stored":518585685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.88900000000000001,"15min":0.87},"min":{"1min":0.374,"5min":0.374,"15min":0.32700000000000001},"max":{"1min":1.2869999999999999,"5min":1.4379999999999999,"15min":6.4729999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":0.95099999999999996,"15min":0.88900000000000001},"min":{"1min":0.55400000000000005,"5min":0.53100000000000003,"15min":0.39100000000000001},"max":{"1min":3.585,"5min":3.585,"15min":8.7560000000000002},"last":0.32000000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.874,"15min":0.89100000000000001},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.42799999999999999},"max":{"1min":3.6360000000000001,"5min":3.6360000000000001,"15min":15.257999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.151,"5min":0.91900000000000004,"15min":0.96299999999999997},"min":{"1min":0.54200000000000004,"5min":0.52200000000000002,"15min":0.47399999999999998},"max":{"1min":4.0439999999999996,"5min":4.0439999999999996,"15min":15.175000000000001},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.90000000000000002,"15min":0.91700000000000004},"min":{"1min":0.503,"5min":0.503,"15min":0.42999999999999999},"max":{"1min":3.0179999999999998,"5min":3.0179999999999998,"15min":6.5410000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.997,"5min":0.94099999999999995,"15min":0.98899999999999999},"min":{"1min":0.56499999999999995,"5min":0.49099999999999999,"15min":0.46000000000000002},"max":{"1min":2.8879999999999999,"5min":2.8879999999999999,"15min":6.7080000000000002},"last":0.34399999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.087,"15min":1.048},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.54900000000000004},"max":{"1min":2.9340000000000002,"5min":5.4249999999999998,"15min":6.7699999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.181,"5min":1.171,"15min":1.1719999999999999},"min":{"1min":0.74399999999999999,"5min":0.58699999999999997,"15min":0.40100000000000002},"max":{"1min":2.9750000000000001,"5min":5.5570000000000004,"15min":15.364000000000001},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:52:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.94299999999999995,"15min":0.98499999999999999},"min":{"1min":0.621,"5min":0.376,"15min":0.36499999999999999},"max":{"1min":1.24,"5min":4.3529999999999998,"15min":5.9720000000000004},"last":0.503},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2350000000000001,"15min":1.157},"min":{"1min":0.65900000000000003,"5min":0.56699999999999995,"15min":0.53000000000000003},"max":{"1min":1.395,"5min":5.6609999999999996,"15min":14.314},"last":0.59499999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.0720000000000001,"15min":1.129},"min":{"1min":0.68100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":3.4279999999999999,"5min":4.2290000000000001,"15min":15.06},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.363,"5min":1.157,"15min":1.1899999999999999},"min":{"1min":0.79200000000000004,"5min":0.65700000000000003,"15min":0.50900000000000001},"max":{"1min":3.508,"5min":3.508,"15min":7.6630000000000003},"last":0.57599999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:51:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.0609999999999999,"15min":1.1279999999999999},"min":{"1min":0.745,"5min":0.44,"15min":0.44},"max":{"1min":4.0599999999999996,"5min":5.0419999999999998,"15min":6.8259999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.429,"5min":1.2130000000000001,"15min":1.266},"min":{"1min":0.89700000000000002,"5min":0.53500000000000003,"15min":0.46200000000000002},"max":{"1min":2.8250000000000002,"5min":2.8250000000000002,"15min":19.321999999999999},"last":0.65900000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575364,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726856,"kb_used_data":569408,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93644984,"statfs":{"total":96636764160,"available":95892463616,"internally_reserved":0,"allocated":583073792,"data_stored":568869284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":161216966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.751,"15min":0.79700000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.1839999999999999,"5min":2.202,"15min":9.6690000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.48499999999999999,"5min":0.47699999999999998,"15min":0.377},"max":{"1min":1.206,"5min":2.0760000000000001,"15min":2.786},"last":0.94299999999999995}]},{"osd":1,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.84899999999999998,"15min":0.872},"min":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.41399999999999998},"max":{"1min":1.278,"5min":1.8160000000000001,"15min":5.5119999999999996},"last":1.2230000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.88100000000000001,"15min":0.85899999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.44700000000000001},"max":{"1min":1.2110000000000001,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":1.034}]},{"osd":2,"last update":"Mon Oct 30 13:51:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.89200000000000002,"15min":0.878},"min":{"1min":0.63200000000000001,"5min":0.53700000000000003,"15min":0.45100000000000001},"max":{"1min":1.212,"5min":2.1549999999999998,"15min":9.0099999999999998},"last":1.3440000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.95099999999999996,"15min":0.95199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0569999999999999,"5min":7.7359999999999998,"15min":7.7359999999999998},"last":0.89100000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.93799999999999994,"15min":0.94699999999999995},"min":{"1min":0.63700000000000001,"5min":0.56200000000000006,"15min":0.52500000000000002},"max":{"1min":1.248,"5min":1.6000000000000001,"15min":4.7080000000000002},"last":1.3180000000000001},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.034,"15min":1.018},"min":{"1min":0.67900000000000005,"5min":0.66700000000000004,"15min":0.48699999999999999},"max":{"1min":1.153,"5min":1.7509999999999999,"15min":3.2749999999999999},"last":1.3959999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.0029999999999999,"15min":1.0409999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.29,"5min":1.3799999999999999,"15min":7.8529999999999998},"last":1.425},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.212,"15min":1.1759999999999999},"min":{"1min":0.68200000000000005,"5min":0.65400000000000003,"15min":0.496},"max":{"1min":1.4350000000000001,"5min":7.694,"15min":8.2070000000000007},"last":1.7829999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.125,"15min":1.083},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.52600000000000002},"max":{"1min":1.4510000000000001,"5min":7.7880000000000003,"15min":7.7880000000000003},"last":1.5820000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.1459999999999999,"15min":1.151},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.395,"5min":3.7320000000000002,"15min":5.6130000000000004},"last":1.472}]},{"osd":7,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.194,"15min":1.1759999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.59299999999999997},"max":{"1min":1.5109999999999999,"5min":7.8380000000000001,"15min":10.715},"last":1.6870000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.0369999999999999,"15min":1.147},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.47199999999999998},"max":{"1min":1.321,"5min":2.2450000000000001,"15min":4.3230000000000004},"last":1.085}]}]},{"osd":3,"up_from":13,"seq":55834575362,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715812,"kb_used_data":550044,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93656028,"statfs":{"total":96636764160,"available":95903772672,"internally_reserved":0,"allocated":563245056,"data_stored":549099638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":169736648},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.69799999999999995},"last":0.441},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.47299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64900000000000002,"5min":0.76700000000000002,"15min":1.5429999999999999},"last":0.46899999999999997}]},{"osd":1,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.51000000000000001,"15min":0.46500000000000002},"min":{"1min":0.373,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":0.755,"5min":0.76200000000000001,"15min":0.94099999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.28599999999999998,"15min":0.246},"max":{"1min":0.78900000000000003,"5min":0.92000000000000004,"15min":1.018},"last":0.42699999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.65100000000000002,"5min":1.056,"15min":1.4690000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.41299999999999998,"5min":0.311,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.86499999999999999,"15min":1.6799999999999999},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:52:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.79600000000000004,"15min":0.83599999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.49099999999999999},"max":{"1min":0.96499999999999997,"5min":1.1739999999999999,"15min":1.5900000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.84299999999999997,"15min":0.85299999999999998},"min":{"1min":0.66100000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0129999999999999,"5min":1.1419999999999999,"15min":1.3999999999999999},"last":0.78600000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.60799999999999998,"5min":0.57399999999999995,"15min":0.51800000000000002},"max":{"1min":1.095,"5min":1.1879999999999999,"15min":1.5069999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.83599999999999997,"15min":0.81999999999999995},"min":{"1min":0.621,"5min":0.58599999999999997,"15min":0.5},"max":{"1min":1.232,"5min":1.232,"15min":1.365},"last":0.81599999999999995}]},{"osd":6,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.88700000000000001,"15min":0.82799999999999996},"min":{"1min":0.66900000000000004,"5min":0.52800000000000002,"15min":0.45400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.88500000000000001,"15min":0.82699999999999996},"min":{"1min":0.69499999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":2.3410000000000002},"last":0.77400000000000002}]},{"osd":7,"last update":"Mon Oct 30 13:51:55 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.88800000000000001,"15min":0.88200000000000001},"min":{"1min":0.68300000000000005,"5min":0.57799999999999996,"15min":0.51700000000000002},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":2.9449999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.872,"15min":0.89300000000000002},"min":{"1min":0.64300000000000002,"5min":0.54400000000000004,"15min":0.47099999999999997},"max":{"1min":1.343,"5min":1.343,"15min":2.9729999999999999},"last":0.84399999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575365,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488884,"kb_used_data":377260,"kb_used_omap":1,"kb_used_meta":111614,"kb_avail":93882956,"statfs":{"total":96636764160,"available":96136146944,"internally_reserved":0,"allocated":386314240,"data_stored":377764906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114293190},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.47999999999999998,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.50800000000000001,"15min":0.48199999999999998},"min":{"1min":0.41799999999999998,"5min":0.28299999999999997,"15min":0.25700000000000001},"max":{"1min":4.2910000000000004,"5min":4.2910000000000004,"15min":4.2910000000000004},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.45900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.024,"5min":4.024,"15min":4.024},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.58399999999999996,"15min":0.54000000000000004},"min":{"1min":0.32500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.0060000000000002,"5min":7.0060000000000002,"15min":7.0060000000000002},"last":0.56599999999999995}]},{"osd":3,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.36499999999999999,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":1.637},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.622,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.38800000000000001,"15min":0.33700000000000002},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.52000000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.78000000000000003,"15min":0.80300000000000005},"min":{"1min":0.59699999999999998,"5min":0.442,"15min":0.442},"max":{"1min":4.1970000000000001,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.80500000000000005,"15min":0.82699999999999996},"min":{"1min":0.61399999999999999,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":4.1479999999999997,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.76700000000000002}]},{"osd":5,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.88500000000000001,"15min":0.877},"min":{"1min":0.754,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.5169999999999999,"5min":2.7989999999999999,"15min":2.7989999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.80900000000000005,"15min":0.82499999999999996},"min":{"1min":0.58599999999999997,"5min":0.45300000000000001,"15min":0.41999999999999998},"max":{"1min":1.171,"5min":2.762,"15min":2.762},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.77400000000000002,"15min":0.79000000000000004},"min":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.47699999999999998},"max":{"1min":0.98699999999999999,"5min":1.6419999999999999,"15min":3.1659999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.84399999999999997,"15min":0.81699999999999995},"min":{"1min":0.65500000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.8210000000000002,"5min":3.8210000000000002,"15min":3.8210000000000002},"last":0.80600000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:51:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":0.90800000000000003,"15min":0.85699999999999998},"min":{"1min":0.69599999999999995,"5min":0.55000000000000004,"15min":0.503},"max":{"1min":6.9059999999999997,"5min":6.9059999999999997,"15min":6.9059999999999997},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.0009999999999999,"15min":0.95499999999999996},"min":{"1min":0.81299999999999994,"5min":0.63100000000000001,"15min":0.55900000000000005},"max":{"1min":6.9509999999999996,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.81999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575362,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678716,"kb_used_data":535220,"kb_used_omap":1,"kb_used_meta":143486,"kb_avail":93693124,"statfs":{"total":96636764160,"available":95941758976,"internally_reserved":0,"allocated":548065280,"data_stored":538988020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146930118},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24299999999999999,"15min":0.22800000000000001},"max":{"1min":0.629,"5min":0.94999999999999996,"15min":1.1359999999999999},"last":0.309},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.28799999999999998,"5min":0.28299999999999997,"15min":0.251},"max":{"1min":0.72599999999999998,"5min":0.84099999999999997,"15min":0.96699999999999997},"last":0.41599999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47999999999999998,"15min":0.499},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.627,"5min":0.78000000000000003,"15min":2.8660000000000001},"last":0.437},{"interface":"front","average":{"1min":0.504,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.32800000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67900000000000005,"5min":1.0589999999999999,"15min":2.6739999999999999},"last":0.625}]},{"osd":3,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.314},"max":{"1min":0.70599999999999996,"5min":0.76900000000000002,"15min":1.2569999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57399999999999995},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":0.749,"5min":0.81399999999999995,"15min":2.71},"last":0.501}]},{"osd":4,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.81699999999999995,"15min":0.85199999999999998},"min":{"1min":0.52700000000000002,"5min":0.52400000000000002,"15min":0.40200000000000002},"max":{"1min":1.0389999999999999,"5min":1.2210000000000001,"15min":6.7789999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.88800000000000001,"15min":0.92100000000000004},"min":{"1min":0.59299999999999997,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":1.0900000000000001,"5min":1.2629999999999999,"15min":6.8259999999999996},"last":0.83799999999999997}]},{"osd":5,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.76400000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":2.133,"15min":2.5750000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.80700000000000005,"15min":0.80200000000000005},"min":{"1min":0.621,"5min":0.53800000000000003,"15min":0.51000000000000001},"max":{"1min":1.462,"5min":1.462,"15min":2.6179999999999999},"last":0.61099999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:52:46 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.91500000000000004,"15min":0.89700000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.501},"max":{"1min":1.357,"5min":1.3879999999999999,"15min":6.0289999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.95899999999999996,"15min":0.95399999999999996},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":6.0739999999999998},"last":1.077}]},{"osd":7,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.80800000000000005,"15min":0.78100000000000003},"min":{"1min":0.70299999999999996,"5min":0.55000000000000004,"15min":0.47899999999999998},"max":{"1min":1.1319999999999999,"5min":1.161,"15min":1.163},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.86699999999999999,"15min":0.877},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.55200000000000005},"max":{"1min":1.153,"5min":1.3440000000000001,"15min":6.0519999999999996},"last":0.71799999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575364,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708336,"kb_used_data":568936,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93663504,"statfs":{"total":96636764160,"available":95911428096,"internally_reserved":0,"allocated":582590464,"data_stored":569154632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142735814},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.40100000000000002,"15min":0.47599999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":0.54400000000000004,"5min":2.0630000000000002,"15min":16.184000000000001},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.50900000000000001,"15min":0.59099999999999997},"min":{"1min":0.36199999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.629,"5min":2.0979999999999999,"15min":16.228000000000002},"last":0.36199999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45300000000000001,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.70499999999999996,"5min":2.7999999999999998,"15min":17.454000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.46800000000000003,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":0.61799999999999999,"5min":1.391,"15min":23.212},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51100000000000001,"15min":0.62},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":1.4690000000000001,"15min":16.103999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52700000000000002,"15min":0.72399999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60199999999999998,"5min":2.8340000000000001,"15min":23.16},"last":0.30299999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:51:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.83299999999999996,"15min":0.92700000000000005},"min":{"1min":0.48499999999999999,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":7.6900000000000004,"5min":7.6900000000000004,"15min":16.309000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.85799999999999998,"15min":0.97499999999999998},"min":{"1min":0.61199999999999999,"5min":0.53800000000000003,"15min":0.46000000000000002},"max":{"1min":2.5939999999999999,"5min":3.4009999999999998,"15min":16.343},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.81599999999999995,"15min":0.995},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94299999999999995,"5min":2.02,"15min":23.266999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.86599999999999999,"15min":1.0329999999999999},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.999,"5min":2.7240000000000002,"15min":16.268000000000001},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.79400000000000004,"15min":1.091},"min":{"1min":0.52500000000000002,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":1.0269999999999999,"5min":2.6560000000000001,"15min":29.774999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.80600000000000005,"15min":1.0149999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":2.75,"15min":17.5},"last":0.997}]},{"osd":7,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.81699999999999995,"15min":1.1919999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.51300000000000001},"max":{"1min":1.0229999999999999,"5min":2.694,"15min":47.534999999999997},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":1.147},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.54700000000000004},"max":{"1min":1.272,"5min":2.8690000000000002,"15min":32.476999999999997},"last":1.272}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230703104,"data_stored":230687380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258420736,"data_stored":258399784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179036160,"data_stored":179013521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230858752,"data_stored":230840931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":197275648,"data_stored":197263919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":205541376,"data_stored":205522290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":268873728,"data_stored":268856059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":246767616,"data_stored":246749530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350515200,"data_stored":337464605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288862208,"data_stored":280174938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":206495744,"data_stored":198338087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331603968,"data_stored":317845409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":385015808,"data_stored":371192067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":325414912,"data_stored":312650097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193724416,"data_stored":185512357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342437888,"data_stored":329203526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:52:57.647 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.647 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.647 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.648 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.648 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.648 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.649 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.649 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.650 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.650 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.650 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.651 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.651 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.651 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.652 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.652 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.652 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.653 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.653 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.654 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.654 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.654 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.655 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.655 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.655 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.656 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.656 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.656 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.657 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.657 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.657 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.658 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.658 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.659 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.659 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.659 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.660 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.660 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.660 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.661 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.661 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.661 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.662 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.662 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.663 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.663 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.663 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.664 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.664 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.664 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.665 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.665 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.665 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.666 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.666 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.667 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.667 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.667 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.668 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.668 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.668 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.669 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:52:57.669 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:53:05.165 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:05.169 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:17.670 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:53:17.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e7794640 1 Processor -- start 2023-10-30T13:53:17.830 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e7794640 1 -- start start 2023-10-30T13:53:17.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e7794640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa7e015d6f0 0x7fa7e015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:17.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e7794640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e00cf160 0x7fa7e00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:17.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e7794640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e00cb800 con 0x7fa7e01538a0 2023-10-30T13:53:17.832 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e7794640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e00cb330 con 0x7fa7e015d6f0 2023-10-30T13:53:17.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e7794640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00cad70 con 0x7fa7e00cf160 2023-10-30T13:53:17.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e5530640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa7e01538a0 0x7fa7e00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:38096/0 (socket says 172.21.15.148:38096) 2023-10-30T13:53:17.833 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e5530640 1 -- 172.21.15.148:0/3661360333 learned_addr learned my addr 172.21.15.148:0/3661360333 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:53:17.835 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e4d2f640 1 --2- 172.21.15.148:0/3661360333 >> v2:172.21.15.150:3300/0 conn(0x7fa7e015d6f0 0x7fa7e015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:17.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7e5d31640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e00cf160 0x7fa7e00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:17.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638093088 0 0) 0x7fa7e00cb800 con 0x7fa7e01538a0 2023-10-30T13:53:17.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.829+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7cc003530 con 0x7fa7e01538a0 2023-10-30T13:53:17.836 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3181338929 0 0) 0x7fa7cc003530 con 0x7fa7e01538a0 2023-10-30T13:53:17.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e00cf160 msgr2=0x7fa7e00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:17.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e00cf160 0x7fa7e00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.837 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 >> v2:172.21.15.150:3300/0 conn(0x7fa7e015d6f0 msgr2=0x7fa7e015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:17.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 --2- 172.21.15.148:0/3661360333 >> v2:172.21.15.150:3300/0 conn(0x7fa7e015d6f0 0x7fa7e015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00cb050 con 0x7fa7e01538a0 2023-10-30T13:53:17.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1091396426 0 0) 0x7fa7dc001680 con 0x7fa7e01538a0 2023-10-30T13:53:17.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 >> v1:172.21.15.148:6789/0 conn(0x7fa7e01538a0 legacy=0x7fa7e00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:17.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7cc003980 0x7fa7cc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:17.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7cc0047d0 0x7fa7cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:17.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e00cf160 0x7fa7cc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:17.839 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00cb330 con 0x7fa7cc0047d0 2023-10-30T13:53:17.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00cad70 con 0x7fa7cc003980 2023-10-30T13:53:17.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc00b990 con 0x7fa7e00cf160 2023-10-30T13:53:17.840 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7e4d2f640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7cc0047d0 0x7fa7cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:17.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7e5d31640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e00cf160 0x7fa7cc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:17.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7e5530640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7cc003980 0x7fa7cc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:17.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7e4d2f640 1 -- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e00cf160 msgr2=0x7fa7cc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:17.841 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7e4d2f640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e00cf160 0x7fa7cc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7e4d2f640 1 -- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7cc003980 msgr2=0x7fa7cc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:17.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7e4d2f640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7cc003980 0x7fa7cc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.842 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.833+0000 7fa7e4d2f640 1 -- 172.21.15.148:0/3661360333 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc003530 con 0x7fa7cc0047d0 2023-10-30T13:53:17.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7e4d2f640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7cc0047d0 0x7fa7cc004c50 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009ae0 tx=0x7fa7d001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d659e313f78be8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:17.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d00a55a0 con 0x7fa7cc0047d0 2023-10-30T13:53:17.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d00a5700 con 0x7fa7cc0047d0 2023-10-30T13:53:17.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7d7fff640 1 -- 172.21.15.148:0/3661360333 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d00a5af0 con 0x7fa7cc0047d0 2023-10-30T13:53:17.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7e7794640 1 -- 172.21.15.148:0/3661360333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7cc0047d0 msgr2=0x7fa7cc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:17.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7e7794640 1 --2- 172.21.15.148:0/3661360333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7cc0047d0 0x7fa7cc004c50 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009ae0 tx=0x7fa7d001ca90 comp rx=0 tx=0).stop 2023-10-30T13:53:17.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7e5d31640 1 -- 172.21.15.148:0/3661360333 reap_dead start 2023-10-30T13:53:17.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7e7794640 1 -- 172.21.15.148:0/3661360333 shutdown_connections 2023-10-30T13:53:17.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7e7794640 1 -- 172.21.15.148:0/3661360333 >> 172.21.15.148:0/3661360333 conn(0x7fa7e006bfe0 msgr2=0x7fa7e006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:17.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7e7794640 1 -- 172.21.15.148:0/3661360333 shutdown_connections 2023-10-30T13:53:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.837+0000 7fa7e7794640 1 -- 172.21.15.148:0/3661360333 wait complete. 2023-10-30T13:53:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 Processor -- start 2023-10-30T13:53:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 -- start start 2023-10-30T13:53:17.848 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e015c290 0x7fa7e01f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:17.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7e015c670 0x7fa7e01efed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:17.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e01e60f0 0x7fa7e01e6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:17.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e0150c50 con 0x7fa7e015c670 2023-10-30T13:53:17.849 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e01f39f0 con 0x7fa7e015c290 2023-10-30T13:53:17.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e015b7b0 con 0x7fa7e01e60f0 2023-10-30T13:53:17.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e4d2f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7e015c670 0x7fa7e01efed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:17.850 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e4d2f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7e015c670 0x7fa7e01efed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:35976/0 (socket says 172.21.15.148:35976) 2023-10-30T13:53:17.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e4d2f640 1 -- 172.21.15.148:0/1980989993 learned_addr learned my addr 172.21.15.148:0/1980989993 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:53:17.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e5530640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e015c290 0x7fa7e01f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:17.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e5d31640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e01e60f0 0x7fa7e01e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:17.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e4d2f640 1 -- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e01e60f0 msgr2=0x7fa7e01e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:17.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e4d2f640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e01e60f0 0x7fa7e01e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e4d2f640 1 -- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e015c290 msgr2=0x7fa7e01f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:17.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e4d2f640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e015c290 0x7fa7e01f34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:17.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e4d2f640 1 -- 172.21.15.148:0/1980989993 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e015af50 con 0x7fa7e015c670 2023-10-30T13:53:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e4d2f640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7e015c670 0x7fa7e01efed0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009ae0 tx=0x7fa7cc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7d5ffb640 1 -- 172.21.15.148:0/1980989993 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d00ad640 con 0x7fa7e015c670 2023-10-30T13:53:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7d5ffb640 1 -- 172.21.15.148:0/1980989993 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d00ad7a0 con 0x7fa7e015c670 2023-10-30T13:53:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e015b1f0 con 0x7fa7e015c670 2023-10-30T13:53:17.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7d5ffb640 1 -- 172.21.15.148:0/1980989993 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d00adb30 con 0x7fa7e015c670 2023-10-30T13:53:17.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.841+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e01ef6d0 con 0x7fa7e015c670 2023-10-30T13:53:17.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.845+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e01ef530 con 0x7fa7e015c670 2023-10-30T13:53:17.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.845+0000 7fa7d5ffb640 1 -- 172.21.15.148:0/1980989993 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7d00b4040 con 0x7fa7e015c670 2023-10-30T13:53:17.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.845+0000 7fa7d5ffb640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7c40664d0 0x7fa7c4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.845+0000 7fa7d5ffb640 1 -- 172.21.15.148:0/1980989993 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fa7d011e660 con 0x7fa7e015c670 2023-10-30T13:53:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.849+0000 7fa7e5530640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7c40664d0 0x7fa7c4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:17.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.849+0000 7fa7e5530640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7c40664d0 0x7fa7c4068990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc004850 tx=0x7fa7dc007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:17.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:17.853+0000 7fa7d5ffb640 1 -- 172.21.15.148:0/1980989993 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7e01ef530 con 0x7fa7e015c670 2023-10-30T13:53:18.073 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.073+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7e0159b70 con 0x7fa7c40664d0 2023-10-30T13:53:18.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.097+0000 7fa7d5ffb640 1 -- 172.21.15.148:0/1980989993 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294910 (secure 0 0 0) 0x7fa7dc011ad0 con 0x7fa7c40664d0 2023-10-30T13:53:18.100 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:53:18.102 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:53:18.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7c40664d0 msgr2=0x7fa7c4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:18.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7c40664d0 0x7fa7c4068990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc004850 tx=0x7fa7dc007040 comp rx=0 tx=0).stop 2023-10-30T13:53:18.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7e015c670 msgr2=0x7fa7e01efed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:18.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7e015c670 0x7fa7e01efed0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009ae0 tx=0x7fa7cc005800 comp rx=0 tx=0).stop 2023-10-30T13:53:18.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 shutdown_connections 2023-10-30T13:53:18.104 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa7c40664d0 0x7fa7c4068990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:18.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa7e015c670 0x7fa7e01efed0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:18.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa7e01e60f0 0x7fa7e01e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:18.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 --2- 172.21.15.148:0/1980989993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e015c290 0x7fa7e01f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:18.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 >> 172.21.15.148:0/1980989993 conn(0x7fa7e006bfe0 msgr2=0x7fa7e01523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:18.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 shutdown_connections 2023-10-30T13:53:18.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:18.101+0000 7fa7e7794640 1 -- 172.21.15.148:0/1980989993 wait complete. 2023-10-30T13:53:18.128 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1347,"stamp":"2023-10-30T13:53:17.219264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2074908826,"num_objects":41005,"num_object_clones":18317,"num_object_copies":82010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41005,"num_whiteouts":14350,"num_read":18168,"num_read_kb":6131546,"num_write":117782,"num_write_kb":3536715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1411,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17412914,"num_omap_keys":29642,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157427,"ondisk_log_size":157427,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5332148,"kb_used_data":4150324,"kb_used_omap":12,"kb_used_meta":1181747,"kb_avail":749642572,"statfs":{"total":773094113280,"available":767633993728,"internally_reserved":0,"allocated":4249931776,"data_stored":4155138249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1210109496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2870,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1888576,"num_omap_keys":3309,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003501"},"pg_stats":[{"pgid":"3.3f","version":"67'1473","reported_seq":3441,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.519050+0000","last_change":"2023-10-30T13:21:40.584822+0000","last_active":"2023-10-30T13:52:29.519050+0000","last_peered":"2023-10-30T13:52:29.519050+0000","last_clean":"2023-10-30T13:52:29.519050+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:52:29.519050+0000","last_undegraded":"2023-10-30T13:52:29.519050+0000","last_fullsized":"2023-10-30T13:52:29.519050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014806453000000001,"stat_sum":{"num_bytes":11269044,"num_objects":569,"num_object_clones":260,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":205,"num_read":107,"num_read_kb":121,"num_write":1253,"num_write_kb":17103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'1479","reported_seq":3403,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.436814+0000","last_change":"2023-10-30T13:21:40.572202+0000","last_active":"2023-10-30T13:52:29.436814+0000","last_peered":"2023-10-30T13:52:29.436814+0000","last_clean":"2023-10-30T13:52:29.436814+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:52:29.436814+0000","last_undegraded":"2023-10-30T13:52:29.436814+0000","last_fullsized":"2023-10-30T13:52:29.436814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059691099999999997,"stat_sum":{"num_bytes":30975930,"num_objects":597,"num_object_clones":279,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":225,"num_read":82,"num_read_kb":30,"num_write":1226,"num_write_kb":32475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'2143","reported_seq":4123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.190609+0000","last_change":"2023-10-30T13:52:54.190676+0000","last_active":"2023-10-30T13:52:54.190609+0000","last_peered":"2023-10-30T13:52:54.190609+0000","last_clean":"2023-10-30T13:52:54.190609+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:52:54.190609+0000","last_undegraded":"2023-10-30T13:52:54.190609+0000","last_fullsized":"2023-10-30T13:52:54.190609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2143","last_scrub_stamp":"2023-10-30T13:52:54.190532+0000","last_deep_scrub":"67'2143","last_deep_scrub_stamp":"2023-10-30T13:52:54.190532+0000","last_clean_scrub_stamp":"2023-10-30T13:52:54.190532+0000","objects_scrubbed":617,"log_size":2143,"log_dups_size":0,"ondisk_log_size":2143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:27.035524+0000","scrub_duration":0.94181986699999998,"objects_trimmed":0,"snaptrim_duration":0.0025608710000000002,"stat_sum":{"num_bytes":18704388,"num_objects":617,"num_object_clones":279,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":225,"num_read":106,"num_read_kb":8300,"num_write":1300,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'2214","reported_seq":4184,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.681924+0000","last_change":"2023-10-30T13:52:59.682002+0000","last_active":"2023-10-30T13:52:59.681924+0000","last_peered":"2023-10-30T13:52:59.681924+0000","last_clean":"2023-10-30T13:52:59.681924+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:52:59.681924+0000","last_undegraded":"2023-10-30T13:52:59.681924+0000","last_fullsized":"2023-10-30T13:52:59.681924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2214","last_scrub_stamp":"2023-10-30T13:52:59.681815+0000","last_deep_scrub":"67'2214","last_deep_scrub_stamp":"2023-10-30T13:52:59.681815+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.681815+0000","objects_scrubbed":630,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:49.252489+0000","scrub_duration":1.2152727269999999,"objects_trimmed":0,"snaptrim_duration":0.0021564890000000001,"stat_sum":{"num_bytes":23329933,"num_objects":630,"num_object_clones":293,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":227,"num_read":112,"num_read_kb":4378,"num_write":1354,"num_write_kb":25006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'2231","reported_seq":4187,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:38.545335+0000","last_change":"2023-10-30T13:52:38.545426+0000","last_active":"2023-10-30T13:52:38.545335+0000","last_peered":"2023-10-30T13:52:38.545335+0000","last_clean":"2023-10-30T13:52:38.545335+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:52:38.545335+0000","last_undegraded":"2023-10-30T13:52:38.545335+0000","last_fullsized":"2023-10-30T13:52:38.545335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2231","last_scrub_stamp":"2023-10-30T13:52:38.545206+0000","last_deep_scrub":"67'2231","last_deep_scrub_stamp":"2023-10-30T13:52:38.545206+0000","last_clean_scrub_stamp":"2023-10-30T13:52:38.545206+0000","objects_scrubbed":654,"log_size":2231,"log_dups_size":0,"ondisk_log_size":2231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:21:20.355710+0000","scrub_duration":1.430729712,"objects_trimmed":0,"snaptrim_duration":0.011808077,"stat_sum":{"num_bytes":16184828,"num_objects":654,"num_object_clones":305,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":251,"num_read":109,"num_read_kb":152,"num_write":1339,"num_write_kb":20879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'2052","reported_seq":4033,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:56.370945+0000","last_change":"2023-10-30T13:52:56.370945+0000","last_active":"2023-10-30T13:52:56.370945+0000","last_peered":"2023-10-30T13:52:56.370945+0000","last_clean":"2023-10-30T13:52:56.370945+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:52:56.370945+0000","last_undegraded":"2023-10-30T13:52:56.370945+0000","last_fullsized":"2023-10-30T13:52:56.370945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2052","last_scrub_stamp":"2023-10-30T13:52:56.370794+0000","last_deep_scrub":"67'2052","last_deep_scrub_stamp":"2023-10-30T13:52:56.370794+0000","last_clean_scrub_stamp":"2023-10-30T13:52:56.370794+0000","objects_scrubbed":593,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:21.000907+0000","scrub_duration":1.065003666,"objects_trimmed":0,"snaptrim_duration":0.01102786,"stat_sum":{"num_bytes":18372801,"num_objects":593,"num_object_clones":277,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":222,"num_read":108,"num_read_kb":4137,"num_write":1217,"num_write_kb":19358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'1515","reported_seq":3454,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.789089+0000","last_change":"2023-10-30T13:21:40.580121+0000","last_active":"2023-10-30T13:52:36.789089+0000","last_peered":"2023-10-30T13:52:36.789089+0000","last_clean":"2023-10-30T13:52:36.789089+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:52:36.789089+0000","last_undegraded":"2023-10-30T13:52:36.789089+0000","last_fullsized":"2023-10-30T13:52:36.789089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012047337,"stat_sum":{"num_bytes":9859762,"num_objects":598,"num_object_clones":275,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":211,"num_read":96,"num_read_kb":35,"num_write":1295,"num_write_kb":11866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'2037","reported_seq":4026,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.982168+0000","last_change":"2023-10-30T13:52:51.982168+0000","last_active":"2023-10-30T13:52:51.982168+0000","last_peered":"2023-10-30T13:52:51.982168+0000","last_clean":"2023-10-30T13:52:51.982168+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:52:51.982168+0000","last_undegraded":"2023-10-30T13:52:51.982168+0000","last_fullsized":"2023-10-30T13:52:51.982168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2037","last_scrub_stamp":"2023-10-30T13:52:51.982035+0000","last_deep_scrub":"67'2037","last_deep_scrub_stamp":"2023-10-30T13:52:51.982035+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.982035+0000","objects_scrubbed":577,"log_size":2037,"log_dups_size":0,"ondisk_log_size":2037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:47.358388+0000","scrub_duration":1.116769473,"objects_trimmed":0,"snaptrim_duration":0.0063254060000000004,"stat_sum":{"num_bytes":13694423,"num_objects":577,"num_object_clones":268,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":211,"num_read":108,"num_read_kb":104,"num_write":1228,"num_write_kb":23250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1528","reported_seq":3471,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.522374+0000","last_change":"2023-10-30T13:21:40.584916+0000","last_active":"2023-10-30T13:52:29.522374+0000","last_peered":"2023-10-30T13:52:29.522374+0000","last_clean":"2023-10-30T13:52:29.522374+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:52:29.522374+0000","last_undegraded":"2023-10-30T13:52:29.522374+0000","last_fullsized":"2023-10-30T13:52:29.522374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014360304000000001,"stat_sum":{"num_bytes":19134308,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":234,"num_read":109,"num_read_kb":4185,"num_write":1275,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'1501","reported_seq":3440,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.604439+0000","last_change":"2023-10-30T13:21:40.580946+0000","last_active":"2023-10-30T13:52:29.604439+0000","last_peered":"2023-10-30T13:52:29.604439+0000","last_clean":"2023-10-30T13:52:29.604439+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:52:29.604439+0000","last_undegraded":"2023-10-30T13:52:29.604439+0000","last_fullsized":"2023-10-30T13:52:29.604439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010277041000000001,"stat_sum":{"num_bytes":19693407,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":208,"num_read":105,"num_read_kb":2976,"num_write":1294,"num_write_kb":29513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'1536","reported_seq":3464,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.440713+0000","last_change":"2023-10-30T13:21:40.579672+0000","last_active":"2023-10-30T13:52:29.440713+0000","last_peered":"2023-10-30T13:52:29.440713+0000","last_clean":"2023-10-30T13:52:29.440713+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:52:29.440713+0000","last_undegraded":"2023-10-30T13:52:29.440713+0000","last_fullsized":"2023-10-30T13:52:29.440713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1536,"log_dups_size":0,"ondisk_log_size":1536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01048763,"stat_sum":{"num_bytes":11727836,"num_objects":636,"num_object_clones":292,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":230,"num_read":95,"num_read_kb":27,"num_write":1294,"num_write_kb":13174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'2168","reported_seq":4127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:55.137033+0000","last_change":"2023-10-30T13:52:55.137145+0000","last_active":"2023-10-30T13:52:55.137033+0000","last_peered":"2023-10-30T13:52:55.137033+0000","last_clean":"2023-10-30T13:52:55.137033+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:52:55.137033+0000","last_undegraded":"2023-10-30T13:52:55.137033+0000","last_fullsized":"2023-10-30T13:52:55.137033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2168","last_scrub_stamp":"2023-10-30T13:52:55.136932+0000","last_deep_scrub":"67'2168","last_deep_scrub_stamp":"2023-10-30T13:52:55.136932+0000","last_clean_scrub_stamp":"2023-10-30T13:52:55.136932+0000","objects_scrubbed":634,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:10.926178+0000","scrub_duration":1.2701765250000001,"objects_trimmed":0,"snaptrim_duration":0.0085295030000000008,"stat_sum":{"num_bytes":15445695,"num_objects":634,"num_object_clones":293,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":231,"num_read":101,"num_read_kb":102,"num_write":1284,"num_write_kb":21892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'2081","reported_seq":4056,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:11.778193+0000","last_change":"2023-10-30T13:53:11.778257+0000","last_active":"2023-10-30T13:53:11.778193+0000","last_peered":"2023-10-30T13:53:11.778193+0000","last_clean":"2023-10-30T13:53:11.778193+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:53:11.778193+0000","last_undegraded":"2023-10-30T13:53:11.778193+0000","last_fullsized":"2023-10-30T13:53:11.778193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:53:11.778101+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:53:11.778101+0000","last_clean_scrub_stamp":"2023-10-30T13:53:11.778101+0000","objects_scrubbed":596,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:38.043584+0000","scrub_duration":0.84397403100000001,"objects_trimmed":0,"snaptrim_duration":0.004933977,"stat_sum":{"num_bytes":6328355,"num_objects":596,"num_object_clones":272,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":210,"num_read":97,"num_read_kb":4194,"num_write":1284,"num_write_kb":15784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'2163","reported_seq":4144,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:46.206549+0000","last_change":"2023-10-30T13:52:46.206609+0000","last_active":"2023-10-30T13:52:46.206549+0000","last_peered":"2023-10-30T13:52:46.206549+0000","last_clean":"2023-10-30T13:52:46.206549+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:52:46.206549+0000","last_undegraded":"2023-10-30T13:52:46.206549+0000","last_fullsized":"2023-10-30T13:52:46.206549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2163","last_scrub_stamp":"2023-10-30T13:52:46.206474+0000","last_deep_scrub":"67'2163","last_deep_scrub_stamp":"2023-10-30T13:52:46.206474+0000","last_clean_scrub_stamp":"2023-10-30T13:52:46.206474+0000","objects_scrubbed":610,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:31.417387+0000","scrub_duration":1.043425775,"objects_trimmed":0,"snaptrim_duration":0.0029909680000000001,"stat_sum":{"num_bytes":31670070,"num_objects":610,"num_object_clones":280,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":227,"num_read":110,"num_read_kb":16437,"num_write":1314,"num_write_kb":33491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'2183","reported_seq":4156,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:05.806848+0000","last_change":"2023-10-30T13:53:05.806946+0000","last_active":"2023-10-30T13:53:05.806848+0000","last_peered":"2023-10-30T13:53:05.806848+0000","last_clean":"2023-10-30T13:53:05.806848+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:53:05.806848+0000","last_undegraded":"2023-10-30T13:53:05.806848+0000","last_fullsized":"2023-10-30T13:53:05.806848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2183","last_scrub_stamp":"2023-10-30T13:53:05.806777+0000","last_deep_scrub":"67'2183","last_deep_scrub_stamp":"2023-10-30T13:53:05.806777+0000","last_clean_scrub_stamp":"2023-10-30T13:53:05.806777+0000","objects_scrubbed":613,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:17.084832+0000","scrub_duration":1.1056300050000001,"objects_trimmed":0,"snaptrim_duration":0.0058939090000000001,"stat_sum":{"num_bytes":14342306,"num_objects":613,"num_object_clones":284,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":231,"num_read":106,"num_read_kb":2953,"num_write":1333,"num_write_kb":15925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'2459","reported_seq":4355,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:46.948880+0000","last_change":"2023-10-30T13:52:46.948942+0000","last_active":"2023-10-30T13:52:46.948880+0000","last_peered":"2023-10-30T13:52:46.948880+0000","last_clean":"2023-10-30T13:52:46.948880+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:52:46.948880+0000","last_undegraded":"2023-10-30T13:52:46.948880+0000","last_fullsized":"2023-10-30T13:52:46.948880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2459","last_scrub_stamp":"2023-10-30T13:52:46.948803+0000","last_deep_scrub":"67'2459","last_deep_scrub_stamp":"2023-10-30T13:52:46.948803+0000","last_clean_scrub_stamp":"2023-10-30T13:52:46.948803+0000","objects_scrubbed":703,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:36:20.176163+0000","scrub_duration":1.765876443,"objects_trimmed":0,"snaptrim_duration":0.0093243890000000006,"stat_sum":{"num_bytes":10965780,"num_objects":703,"num_object_clones":374,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":231,"num_read":115,"num_read_kb":1694,"num_write":1434,"num_write_kb":17364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'2223","reported_seq":4184,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.397041+0000","last_change":"2023-10-30T13:52:51.397109+0000","last_active":"2023-10-30T13:52:51.397041+0000","last_peered":"2023-10-30T13:52:51.397041+0000","last_clean":"2023-10-30T13:52:51.397041+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:52:51.397041+0000","last_undegraded":"2023-10-30T13:52:51.397041+0000","last_fullsized":"2023-10-30T13:52:51.397041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2223","last_scrub_stamp":"2023-10-30T13:52:51.396947+0000","last_deep_scrub":"67'2223","last_deep_scrub_stamp":"2023-10-30T13:52:51.396947+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.396947+0000","objects_scrubbed":632,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:49.530995+0000","scrub_duration":1.1860778729999999,"objects_trimmed":0,"snaptrim_duration":0.011284880000000001,"stat_sum":{"num_bytes":18442682,"num_objects":632,"num_object_clones":291,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":228,"num_read":100,"num_read_kb":8212,"num_write":1340,"num_write_kb":24815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'2051","reported_seq":4022,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:58.672924+0000","last_change":"2023-10-30T13:52:58.673009+0000","last_active":"2023-10-30T13:52:58.672924+0000","last_peered":"2023-10-30T13:52:58.672924+0000","last_clean":"2023-10-30T13:52:58.672924+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:52:58.672924+0000","last_undegraded":"2023-10-30T13:52:58.672924+0000","last_fullsized":"2023-10-30T13:52:58.672924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2051","last_scrub_stamp":"2023-10-30T13:52:58.672817+0000","last_deep_scrub":"67'2051","last_deep_scrub_stamp":"2023-10-30T13:52:58.672817+0000","last_clean_scrub_stamp":"2023-10-30T13:52:58.672817+0000","objects_scrubbed":575,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:13:12.046899+0000","scrub_duration":1.1758492060000001,"objects_trimmed":0,"snaptrim_duration":0.002295727,"stat_sum":{"num_bytes":22547199,"num_objects":575,"num_object_clones":267,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":207,"num_read":89,"num_read_kb":4127,"num_write":1240,"num_write_kb":24211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'1630","reported_seq":3549,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.437160+0000","last_change":"2023-10-30T13:21:40.572640+0000","last_active":"2023-10-30T13:52:29.437160+0000","last_peered":"2023-10-30T13:52:29.437160+0000","last_clean":"2023-10-30T13:52:29.437160+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:52:29.437160+0000","last_undegraded":"2023-10-30T13:52:29.437160+0000","last_fullsized":"2023-10-30T13:52:29.437160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1630,"log_dups_size":0,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061689539999999999,"stat_sum":{"num_bytes":11703479,"num_objects":649,"num_object_clones":303,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":237,"num_read":106,"num_read_kb":7307,"num_write":1383,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"67'2189","reported_seq":4161,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:04.864343+0000","last_change":"2023-10-30T13:53:04.864418+0000","last_active":"2023-10-30T13:53:04.864343+0000","last_peered":"2023-10-30T13:53:04.864343+0000","last_clean":"2023-10-30T13:53:04.864343+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:53:04.864343+0000","last_undegraded":"2023-10-30T13:53:04.864343+0000","last_fullsized":"2023-10-30T13:53:04.864343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2189","last_scrub_stamp":"2023-10-30T13:53:04.864251+0000","last_deep_scrub":"67'2189","last_deep_scrub_stamp":"2023-10-30T13:53:04.864251+0000","last_clean_scrub_stamp":"2023-10-30T13:53:04.864251+0000","objects_scrubbed":621,"log_size":2189,"log_dups_size":0,"ondisk_log_size":2189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:46.599510+0000","scrub_duration":1.113390213,"objects_trimmed":0,"snaptrim_duration":0.0094394720000000008,"stat_sum":{"num_bytes":31771338,"num_objects":621,"num_object_clones":283,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":229,"num_read":103,"num_read_kb":4126,"num_write":1340,"num_write_kb":42793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'2081","reported_seq":4057,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.440606+0000","last_change":"2023-10-30T13:52:49.440653+0000","last_active":"2023-10-30T13:52:49.440606+0000","last_peered":"2023-10-30T13:52:49.440606+0000","last_clean":"2023-10-30T13:52:49.440606+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:52:49.440606+0000","last_undegraded":"2023-10-30T13:52:49.440606+0000","last_fullsized":"2023-10-30T13:52:49.440606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:52:49.440530+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:52:49.440530+0000","last_clean_scrub_stamp":"2023-10-30T13:52:49.440530+0000","objects_scrubbed":595,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:04.396223+0000","scrub_duration":1.3029579039999999,"objects_trimmed":0,"snaptrim_duration":0.0027436349999999999,"stat_sum":{"num_bytes":6210074,"num_objects":595,"num_object_clones":273,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":228,"num_read":98,"num_read_kb":33,"num_write":1261,"num_write_kb":10653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'2305","reported_seq":4250,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.423134+0000","last_change":"2023-10-30T13:52:45.423196+0000","last_active":"2023-10-30T13:52:45.423134+0000","last_peered":"2023-10-30T13:52:45.423134+0000","last_clean":"2023-10-30T13:52:45.423134+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:52:45.423134+0000","last_undegraded":"2023-10-30T13:52:45.423134+0000","last_fullsized":"2023-10-30T13:52:45.423134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2305","last_scrub_stamp":"2023-10-30T13:52:45.423033+0000","last_deep_scrub":"67'2305","last_deep_scrub_stamp":"2023-10-30T13:52:45.423033+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.423033+0000","objects_scrubbed":670,"log_size":2305,"log_dups_size":0,"ondisk_log_size":2305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:15.141666+0000","scrub_duration":1.276721341,"objects_trimmed":0,"snaptrim_duration":0.0082328149999999992,"stat_sum":{"num_bytes":28398973,"num_objects":670,"num_object_clones":311,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":248,"num_read":102,"num_read_kb":120,"num_write":1396,"num_write_kb":38562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'2154","reported_seq":4126,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:08.009726+0000","last_change":"2023-10-30T13:53:08.009776+0000","last_active":"2023-10-30T13:53:08.009726+0000","last_peered":"2023-10-30T13:53:08.009726+0000","last_clean":"2023-10-30T13:53:08.009726+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:53:08.009726+0000","last_undegraded":"2023-10-30T13:53:08.009726+0000","last_fullsized":"2023-10-30T13:53:08.009726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2154","last_scrub_stamp":"2023-10-30T13:53:08.009651+0000","last_deep_scrub":"67'2154","last_deep_scrub_stamp":"2023-10-30T13:53:08.009651+0000","last_clean_scrub_stamp":"2023-10-30T13:53:08.009651+0000","objects_scrubbed":610,"log_size":2154,"log_dups_size":0,"ondisk_log_size":2154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:41:21.317083+0000","scrub_duration":1.0714371359999999,"objects_trimmed":0,"snaptrim_duration":0.0037447470000000001,"stat_sum":{"num_bytes":19407388,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":225,"num_read":103,"num_read_kb":4138,"num_write":1300,"num_write_kb":24714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'2202","reported_seq":4177,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:08.361856+0000","last_change":"2023-10-30T13:53:08.361983+0000","last_active":"2023-10-30T13:53:08.361856+0000","last_peered":"2023-10-30T13:53:08.361856+0000","last_clean":"2023-10-30T13:53:08.361856+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:53:08.361856+0000","last_undegraded":"2023-10-30T13:53:08.361856+0000","last_fullsized":"2023-10-30T13:53:08.361856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2202","last_scrub_stamp":"2023-10-30T13:53:08.361660+0000","last_deep_scrub":"67'2202","last_deep_scrub_stamp":"2023-10-30T13:53:08.361660+0000","last_clean_scrub_stamp":"2023-10-30T13:53:08.361660+0000","objects_scrubbed":616,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:03.069890+0000","scrub_duration":1.1435550349999999,"objects_trimmed":0,"snaptrim_duration":0.0091633449999999998,"stat_sum":{"num_bytes":10552296,"num_objects":616,"num_object_clones":294,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":228,"num_read":115,"num_read_kb":24,"num_write":1342,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1421","reported_seq":3381,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.594953+0000","last_change":"2023-10-30T13:21:40.571783+0000","last_active":"2023-10-30T13:52:29.594953+0000","last_peered":"2023-10-30T13:52:29.594953+0000","last_clean":"2023-10-30T13:52:29.594953+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:52:29.594953+0000","last_undegraded":"2023-10-30T13:52:29.594953+0000","last_fullsized":"2023-10-30T13:52:29.594953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059368210000000001,"stat_sum":{"num_bytes":15196336,"num_objects":556,"num_object_clones":256,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":202,"num_read":99,"num_read_kb":1380,"num_write":1203,"num_write_kb":16993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'1585","reported_seq":3534,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.601898+0000","last_change":"2023-10-30T13:21:40.581115+0000","last_active":"2023-10-30T13:52:29.601898+0000","last_peered":"2023-10-30T13:52:29.601898+0000","last_clean":"2023-10-30T13:52:29.601898+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:52:29.601898+0000","last_undegraded":"2023-10-30T13:52:29.601898+0000","last_fullsized":"2023-10-30T13:52:29.601898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1585,"log_dups_size":0,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099065669999999998,"stat_sum":{"num_bytes":9983325,"num_objects":619,"num_object_clones":286,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":221,"num_read":115,"num_read_kb":68,"num_write":1330,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'1543","reported_seq":3487,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.436117+0000","last_change":"2023-10-30T13:21:40.572631+0000","last_active":"2023-10-30T13:52:29.436117+0000","last_peered":"2023-10-30T13:52:29.436117+0000","last_clean":"2023-10-30T13:52:29.436117+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:52:29.436117+0000","last_undegraded":"2023-10-30T13:52:29.436117+0000","last_fullsized":"2023-10-30T13:52:29.436117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1543,"log_dups_size":0,"ondisk_log_size":1543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059382619999999997,"stat_sum":{"num_bytes":23775876,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":236,"num_read":113,"num_read_kb":4145,"num_write":1301,"num_write_kb":26615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'2078","reported_seq":4040,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:03.067704+0000","last_change":"2023-10-30T13:53:03.067770+0000","last_active":"2023-10-30T13:53:03.067704+0000","last_peered":"2023-10-30T13:53:03.067704+0000","last_clean":"2023-10-30T13:53:03.067704+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:53:03.067704+0000","last_undegraded":"2023-10-30T13:53:03.067704+0000","last_fullsized":"2023-10-30T13:53:03.067704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2078","last_scrub_stamp":"2023-10-30T13:53:03.067613+0000","last_deep_scrub":"67'2078","last_deep_scrub_stamp":"2023-10-30T13:53:03.067613+0000","last_clean_scrub_stamp":"2023-10-30T13:53:03.067613+0000","objects_scrubbed":600,"log_size":2078,"log_dups_size":0,"ondisk_log_size":2078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:58.122248+0000","scrub_duration":1.2008048499999999,"objects_trimmed":0,"snaptrim_duration":0.0053787089999999997,"stat_sum":{"num_bytes":10833053,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":213,"num_read":88,"num_read_kb":3300,"num_write":1256,"num_write_kb":16238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'2337","reported_seq":4272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:55.080264+0000","last_change":"2023-10-30T13:52:55.080329+0000","last_active":"2023-10-30T13:52:55.080264+0000","last_peered":"2023-10-30T13:52:55.080264+0000","last_clean":"2023-10-30T13:52:55.080264+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:52:55.080264+0000","last_undegraded":"2023-10-30T13:52:55.080264+0000","last_fullsized":"2023-10-30T13:52:55.080264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2337","last_scrub_stamp":"2023-10-30T13:52:55.080176+0000","last_deep_scrub":"67'2337","last_deep_scrub_stamp":"2023-10-30T13:52:55.080176+0000","last_clean_scrub_stamp":"2023-10-30T13:52:55.080176+0000","objects_scrubbed":688,"log_size":2337,"log_dups_size":0,"ondisk_log_size":2337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:15:48.148329+0000","scrub_duration":0.84244230799999997,"objects_trimmed":0,"snaptrim_duration":0.0028849399999999999,"stat_sum":{"num_bytes":11995550,"num_objects":688,"num_object_clones":321,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":244,"num_read":100,"num_read_kb":8267,"num_write":1414,"num_write_kb":14306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'2013","reported_seq":3996,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:52.418032+0000","last_change":"2023-10-30T13:52:52.418138+0000","last_active":"2023-10-30T13:52:52.418032+0000","last_peered":"2023-10-30T13:52:52.418032+0000","last_clean":"2023-10-30T13:52:52.418032+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:52:52.418032+0000","last_undegraded":"2023-10-30T13:52:52.418032+0000","last_fullsized":"2023-10-30T13:52:52.418032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2013","last_scrub_stamp":"2023-10-30T13:52:52.417929+0000","last_deep_scrub":"67'2013","last_deep_scrub_stamp":"2023-10-30T13:52:52.417929+0000","last_clean_scrub_stamp":"2023-10-30T13:52:52.417929+0000","objects_scrubbed":558,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:39:32.467346+0000","scrub_duration":1.1806941150000001,"objects_trimmed":0,"snaptrim_duration":0.0096547909999999994,"stat_sum":{"num_bytes":15074071,"num_objects":558,"num_object_clones":259,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":221,"num_read":91,"num_read_kb":32,"num_write":1204,"num_write_kb":16754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'2322","reported_seq":4298,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:57.949237+0000","last_change":"2023-10-30T13:52:57.949288+0000","last_active":"2023-10-30T13:52:57.949237+0000","last_peered":"2023-10-30T13:52:57.949237+0000","last_clean":"2023-10-30T13:52:57.949237+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:52:57.949237+0000","last_undegraded":"2023-10-30T13:52:57.949237+0000","last_fullsized":"2023-10-30T13:52:57.949237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2322","last_scrub_stamp":"2023-10-30T13:52:57.949159+0000","last_deep_scrub":"67'2322","last_deep_scrub_stamp":"2023-10-30T13:52:57.949159+0000","last_clean_scrub_stamp":"2023-10-30T13:52:57.949159+0000","objects_scrubbed":649,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:26.563664+0000","scrub_duration":1.454754868,"objects_trimmed":0,"snaptrim_duration":0.0018845820000000001,"stat_sum":{"num_bytes":10798567,"num_objects":649,"num_object_clones":300,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":241,"num_read":120,"num_read_kb":27,"num_write":1443,"num_write_kb":17235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'1833","reported_seq":3800,"reported_epoch":67,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T13:53:12.495247+0000","last_change":"2023-10-30T13:53:11.855084+0000","last_active":"2023-10-30T13:53:12.495247+0000","last_peered":"2023-10-30T13:53:12.495247+0000","last_clean":"2023-10-30T13:53:12.495247+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:53:12.495247+0000","last_undegraded":"2023-10-30T13:53:12.495247+0000","last_fullsized":"2023-10-30T13:53:12.495247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":318,"log_size":1833,"log_dups_size":0,"ondisk_log_size":1833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075239010000000004,"stat_sum":{"num_bytes":15423893,"num_objects":576,"num_object_clones":259,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":203,"num_read":109,"num_read_kb":9552,"num_write":1308,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'2174","reported_seq":4153,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:47.980889+0000","last_change":"2023-10-30T13:52:47.980946+0000","last_active":"2023-10-30T13:52:47.980889+0000","last_peered":"2023-10-30T13:52:47.980889+0000","last_clean":"2023-10-30T13:52:47.980889+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:52:47.980889+0000","last_undegraded":"2023-10-30T13:52:47.980889+0000","last_fullsized":"2023-10-30T13:52:47.980889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2174","last_scrub_stamp":"2023-10-30T13:52:47.980802+0000","last_deep_scrub":"67'2174","last_deep_scrub_stamp":"2023-10-30T13:52:47.980802+0000","last_clean_scrub_stamp":"2023-10-30T13:52:47.980802+0000","objects_scrubbed":622,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:54:24.281648+0000","scrub_duration":1.0857265140000001,"objects_trimmed":0,"snaptrim_duration":0.007356301,"stat_sum":{"num_bytes":40089180,"num_objects":622,"num_object_clones":287,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":230,"num_read":115,"num_read_kb":33,"num_write":1312,"num_write_kb":42997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'644","reported_seq":883,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.019766+0000","last_change":"2023-10-30T13:52:45.019766+0000","last_active":"2023-10-30T13:52:45.019766+0000","last_peered":"2023-10-30T13:52:45.019766+0000","last_clean":"2023-10-30T13:52:45.019766+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:52:45.019766+0000","last_undegraded":"2023-10-30T13:52:45.019766+0000","last_fullsized":"2023-10-30T13:52:45.019766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'644","last_scrub_stamp":"2023-10-30T13:52:45.019575+0000","last_deep_scrub":"67'644","last_deep_scrub_stamp":"2023-10-30T13:52:45.019575+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.019575+0000","objects_scrubbed":42,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:11:59.207979+0000","scrub_duration":0.065999268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":252,"num_read_kb":249243,"num_write":662,"num_write_kb":74846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":549256,"num_omap_keys":974,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'1657","reported_seq":3570,"reported_epoch":67,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T13:53:12.424551+0000","last_change":"2023-10-30T13:53:12.250245+0000","last_active":"2023-10-30T13:53:12.424551+0000","last_peered":"2023-10-30T13:53:12.424551+0000","last_clean":"2023-10-30T13:53:12.424551+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:53:12.424551+0000","last_undegraded":"2023-10-30T13:53:12.424551+0000","last_fullsized":"2023-10-30T13:53:12.424551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":97,"log_size":1657,"log_dups_size":0,"ondisk_log_size":1657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031372290000000001,"stat_sum":{"num_bytes":18713903,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":229,"num_read":108,"num_read_kb":4667,"num_write":1329,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1176","reported_seq":1511,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.939106+0000","last_change":"2023-10-30T13:52:43.939106+0000","last_active":"2023-10-30T13:52:43.939106+0000","last_peered":"2023-10-30T13:52:43.939106+0000","last_clean":"2023-10-30T13:52:43.939106+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:52:43.939106+0000","last_undegraded":"2023-10-30T13:52:43.939106+0000","last_fullsized":"2023-10-30T13:52:43.939106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1176","last_scrub_stamp":"2023-10-30T13:52:43.938938+0000","last_deep_scrub":"67'1176","last_deep_scrub_stamp":"2023-10-30T13:52:43.938938+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.938938+0000","objects_scrubbed":51,"log_size":1176,"log_dups_size":0,"ondisk_log_size":1176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:18.962998+0000","scrub_duration":0.19282650300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25293488,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":404,"num_read_kb":309242,"num_write":1231,"num_write_kb":84663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":763115,"num_omap_keys":1100,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'794","reported_seq":1183,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:35.070009+0000","last_change":"2023-10-30T13:52:35.070059+0000","last_active":"2023-10-30T13:52:35.070009+0000","last_peered":"2023-10-30T13:52:35.070009+0000","last_clean":"2023-10-30T13:52:35.070009+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:52:35.070009+0000","last_undegraded":"2023-10-30T13:52:35.070009+0000","last_fullsized":"2023-10-30T13:52:35.070009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'794","last_scrub_stamp":"2023-10-30T13:52:35.069918+0000","last_deep_scrub":"67'794","last_deep_scrub_stamp":"2023-10-30T13:52:35.069918+0000","last_clean_scrub_stamp":"2023-10-30T13:52:35.069918+0000","objects_scrubbed":54,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:21.898394+0000","scrub_duration":0.14371538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397151,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":403,"num_read_kb":153383,"num_write":820,"num_write_kb":55113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":519482,"num_omap_keys":897,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'2508","reported_seq":4463,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:50.977937+0000","last_change":"2023-10-30T13:52:50.978041+0000","last_active":"2023-10-30T13:52:50.977937+0000","last_peered":"2023-10-30T13:52:50.977937+0000","last_clean":"2023-10-30T13:52:50.977937+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:52:50.977937+0000","last_undegraded":"2023-10-30T13:52:50.977937+0000","last_fullsized":"2023-10-30T13:52:50.977937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2508","last_scrub_stamp":"2023-10-30T13:52:50.977836+0000","last_deep_scrub":"67'2508","last_deep_scrub_stamp":"2023-10-30T13:52:50.977836+0000","last_clean_scrub_stamp":"2023-10-30T13:52:50.977836+0000","objects_scrubbed":729,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:26.785649+0000","scrub_duration":1.0834898019999999,"objects_trimmed":0,"snaptrim_duration":0.0064075410000000001,"stat_sum":{"num_bytes":11666119,"num_objects":729,"num_object_clones":341,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":261,"num_read":141,"num_read_kb":4175,"num_write":1520,"num_write_kb":22588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1448","reported_seq":1728,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.318938+0000","last_change":"2023-10-30T13:52:43.318938+0000","last_active":"2023-10-30T13:52:43.318938+0000","last_peered":"2023-10-30T13:52:43.318938+0000","last_clean":"2023-10-30T13:52:43.318938+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:52:43.318938+0000","last_undegraded":"2023-10-30T13:52:43.318938+0000","last_fullsized":"2023-10-30T13:52:43.318938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1447","last_scrub_stamp":"2023-10-30T13:52:43.318712+0000","last_deep_scrub":"67'1447","last_deep_scrub_stamp":"2023-10-30T13:52:43.318712+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.318712+0000","objects_scrubbed":54,"log_size":1448,"log_dups_size":0,"ondisk_log_size":1448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:38.717163+0000","scrub_duration":0.18293932199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":354,"num_read_kb":270680,"num_write":1488,"num_write_kb":75460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593681,"num_omap_keys":1037,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'1932","reported_seq":3887,"reported_epoch":67,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T13:53:12.378635+0000","last_change":"2023-10-30T13:53:11.655262+0000","last_active":"2023-10-30T13:53:12.378635+0000","last_peered":"2023-10-30T13:53:12.378635+0000","last_clean":"2023-10-30T13:53:12.378635+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:53:12.378635+0000","last_undegraded":"2023-10-30T13:53:12.378635+0000","last_fullsized":"2023-10-30T13:53:12.378635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":418,"log_size":1932,"log_dups_size":0,"ondisk_log_size":1932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058964639999999997,"stat_sum":{"num_bytes":19507019,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":240,"num_read":105,"num_read_kb":5362,"num_write":1250,"num_write_kb":24752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'1012","reported_seq":1259,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:02.942744+0000","last_change":"2023-10-30T13:53:02.942802+0000","last_active":"2023-10-30T13:53:02.942744+0000","last_peered":"2023-10-30T13:53:02.942744+0000","last_clean":"2023-10-30T13:53:02.942744+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:53:02.942744+0000","last_undegraded":"2023-10-30T13:53:02.942744+0000","last_fullsized":"2023-10-30T13:53:02.942744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1012","last_scrub_stamp":"2023-10-30T13:53:02.942656+0000","last_deep_scrub":"67'1012","last_deep_scrub_stamp":"2023-10-30T13:53:02.942656+0000","last_clean_scrub_stamp":"2023-10-30T13:53:02.942656+0000","objects_scrubbed":56,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:31.384045+0000","scrub_duration":0.195907044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":313,"num_read_kb":141404,"num_write":1079,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":564647,"num_omap_keys":1018,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'2081","reported_seq":4067,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:00.832604+0000","last_change":"2023-10-30T13:53:00.832674+0000","last_active":"2023-10-30T13:53:00.832604+0000","last_peered":"2023-10-30T13:53:00.832604+0000","last_clean":"2023-10-30T13:53:00.832604+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:53:00.832604+0000","last_undegraded":"2023-10-30T13:53:00.832604+0000","last_fullsized":"2023-10-30T13:53:00.832604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:53:00.832511+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:53:00.832511+0000","last_clean_scrub_stamp":"2023-10-30T13:53:00.832511+0000","objects_scrubbed":588,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:07:11.916144+0000","scrub_duration":1.374556924,"objects_trimmed":0,"snaptrim_duration":0.0025817129999999998,"stat_sum":{"num_bytes":23539910,"num_objects":588,"num_object_clones":267,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":208,"num_read":106,"num_read_kb":8313,"num_write":1266,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"67'1170","reported_seq":1398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:39.386385+0000","last_change":"2023-10-30T13:52:39.386488+0000","last_active":"2023-10-30T13:52:39.386385+0000","last_peered":"2023-10-30T13:52:39.386385+0000","last_clean":"2023-10-30T13:52:39.386385+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:52:39.386385+0000","last_undegraded":"2023-10-30T13:52:39.386385+0000","last_fullsized":"2023-10-30T13:52:39.386385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1170","last_scrub_stamp":"2023-10-30T13:52:39.386286+0000","last_deep_scrub":"67'1170","last_deep_scrub_stamp":"2023-10-30T13:52:39.386286+0000","last_clean_scrub_stamp":"2023-10-30T13:52:39.386286+0000","objects_scrubbed":49,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:00.965375+0000","scrub_duration":0.15933681099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":240,"num_read_kb":201173,"num_write":1196,"num_write_kb":66729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":467606,"num_omap_keys":916,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'2185","reported_seq":4130,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:12.335263+0000","last_change":"2023-10-30T13:53:12.335354+0000","last_active":"2023-10-30T13:53:12.335263+0000","last_peered":"2023-10-30T13:53:12.335263+0000","last_clean":"2023-10-30T13:53:12.335263+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:53:12.335263+0000","last_undegraded":"2023-10-30T13:53:12.335263+0000","last_fullsized":"2023-10-30T13:53:12.335263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2185","last_scrub_stamp":"2023-10-30T13:53:12.335127+0000","last_deep_scrub":"67'2185","last_deep_scrub_stamp":"2023-10-30T13:53:12.335127+0000","last_clean_scrub_stamp":"2023-10-30T13:53:12.335127+0000","objects_scrubbed":628,"log_size":2185,"log_dups_size":0,"ondisk_log_size":2185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:20:26.571198+0000","scrub_duration":1.1178727850000001,"objects_trimmed":0,"snaptrim_duration":0.0022874710000000001,"stat_sum":{"num_bytes":16137402,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":233,"num_read":91,"num_read_kb":34,"num_write":1322,"num_write_kb":18013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1358","reported_seq":1609,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.996716+0000","last_change":"2023-10-30T13:52:53.996816+0000","last_active":"2023-10-30T13:52:53.996716+0000","last_peered":"2023-10-30T13:52:53.996716+0000","last_clean":"2023-10-30T13:52:53.996716+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:52:53.996716+0000","last_undegraded":"2023-10-30T13:52:53.996716+0000","last_fullsized":"2023-10-30T13:52:53.996716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1358","last_scrub_stamp":"2023-10-30T13:52:53.996599+0000","last_deep_scrub":"67'1358","last_deep_scrub_stamp":"2023-10-30T13:52:53.996599+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.996599+0000","objects_scrubbed":45,"log_size":1358,"log_dups_size":0,"ondisk_log_size":1358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:58.575120+0000","scrub_duration":0.17104654599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20972192,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":307,"num_read_kb":165804,"num_write":1420,"num_write_kb":50966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":509116,"num_omap_keys":950,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'2207","reported_seq":4174,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.120386+0000","last_change":"2023-10-30T13:52:54.120454+0000","last_active":"2023-10-30T13:52:54.120386+0000","last_peered":"2023-10-30T13:52:54.120386+0000","last_clean":"2023-10-30T13:52:54.120386+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:52:54.120386+0000","last_undegraded":"2023-10-30T13:52:54.120386+0000","last_fullsized":"2023-10-30T13:52:54.120386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2207","last_scrub_stamp":"2023-10-30T13:52:54.120300+0000","last_deep_scrub":"67'2207","last_deep_scrub_stamp":"2023-10-30T13:52:54.120300+0000","last_clean_scrub_stamp":"2023-10-30T13:52:54.120300+0000","objects_scrubbed":638,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:39:00.511615+0000","scrub_duration":1.2561105379999999,"objects_trimmed":0,"snaptrim_duration":0.0087798259999999993,"stat_sum":{"num_bytes":32677996,"num_objects":638,"num_object_clones":293,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":232,"num_read":108,"num_read_kb":10951,"num_write":1342,"num_write_kb":38819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'986","reported_seq":1316,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.944325+0000","last_change":"2023-10-30T13:52:51.944325+0000","last_active":"2023-10-30T13:52:51.944325+0000","last_peered":"2023-10-30T13:52:51.944325+0000","last_clean":"2023-10-30T13:52:51.944325+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:52:51.944325+0000","last_undegraded":"2023-10-30T13:52:51.944325+0000","last_fullsized":"2023-10-30T13:52:51.944325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'985","last_scrub_stamp":"2023-10-30T13:52:51.944137+0000","last_deep_scrub":"67'985","last_deep_scrub_stamp":"2023-10-30T13:52:51.944137+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.944137+0000","objects_scrubbed":50,"log_size":986,"log_dups_size":0,"ondisk_log_size":986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:12.000749+0000","scrub_duration":0.18964769300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":413,"num_read_kb":281405,"num_write":1060,"num_write_kb":80203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":643541,"num_omap_keys":878,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'1512","reported_seq":3428,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.577780+0000","last_change":"2023-10-30T13:21:40.583356+0000","last_active":"2023-10-30T13:52:29.577780+0000","last_peered":"2023-10-30T13:52:29.577780+0000","last_clean":"2023-10-30T13:52:29.577780+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:52:29.577780+0000","last_undegraded":"2023-10-30T13:52:29.577780+0000","last_fullsized":"2023-10-30T13:52:29.577780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014787140000000001,"stat_sum":{"num_bytes":19072740,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":238,"num_read":87,"num_read_kb":102,"num_write":1252,"num_write_kb":21862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'1144","reported_seq":1482,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:37.361974+0000","last_change":"2023-10-30T13:52:37.361974+0000","last_active":"2023-10-30T13:52:37.361974+0000","last_peered":"2023-10-30T13:52:37.361974+0000","last_clean":"2023-10-30T13:52:37.361974+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:52:37.361974+0000","last_undegraded":"2023-10-30T13:52:37.361974+0000","last_fullsized":"2023-10-30T13:52:37.361974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1144","last_scrub_stamp":"2023-10-30T13:52:37.361775+0000","last_deep_scrub":"67'1144","last_deep_scrub_stamp":"2023-10-30T13:52:37.361775+0000","last_clean_scrub_stamp":"2023-10-30T13:52:37.361775+0000","objects_scrubbed":60,"log_size":1144,"log_dups_size":0,"ondisk_log_size":1144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:58.622750+0000","scrub_duration":0.22738265299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748770,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":476,"num_read_kb":203988,"num_write":1202,"num_write_kb":67245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":644581,"num_omap_keys":1115,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'2090","reported_seq":4074,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.327640+0000","last_change":"2023-10-30T13:52:53.327719+0000","last_active":"2023-10-30T13:52:53.327640+0000","last_peered":"2023-10-30T13:52:53.327640+0000","last_clean":"2023-10-30T13:52:53.327640+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:52:53.327640+0000","last_undegraded":"2023-10-30T13:52:53.327640+0000","last_fullsized":"2023-10-30T13:52:53.327640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2090","last_scrub_stamp":"2023-10-30T13:52:53.327572+0000","last_deep_scrub":"67'2090","last_deep_scrub_stamp":"2023-10-30T13:52:53.327572+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.327572+0000","objects_scrubbed":584,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:03.787789+0000","scrub_duration":1.1150200729999999,"objects_trimmed":0,"snaptrim_duration":0.0024076599999999998,"stat_sum":{"num_bytes":14342228,"num_objects":584,"num_object_clones":268,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":211,"num_read":101,"num_read_kb":4159,"num_write":1284,"num_write_kb":24386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'616","reported_seq":808,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.351480+0000","last_change":"2023-10-30T13:52:40.351638+0000","last_active":"2023-10-30T13:52:40.351480+0000","last_peered":"2023-10-30T13:52:40.351480+0000","last_clean":"2023-10-30T13:52:40.351480+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:52:40.351480+0000","last_undegraded":"2023-10-30T13:52:40.351480+0000","last_fullsized":"2023-10-30T13:52:40.351480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'616","last_scrub_stamp":"2023-10-30T13:52:40.351385+0000","last_deep_scrub":"67'616","last_deep_scrub_stamp":"2023-10-30T13:52:40.351385+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.351385+0000","objects_scrubbed":45,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:25.211671+0000","scrub_duration":0.16031920099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":203,"num_read_kb":83137,"num_write":663,"num_write_kb":45991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":437141,"num_omap_keys":850,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1604","reported_seq":3581,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.789128+0000","last_change":"2023-10-30T13:21:40.579925+0000","last_active":"2023-10-30T13:52:36.789128+0000","last_peered":"2023-10-30T13:52:36.789128+0000","last_clean":"2023-10-30T13:52:36.789128+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:52:36.789128+0000","last_undegraded":"2023-10-30T13:52:36.789128+0000","last_fullsized":"2023-10-30T13:52:36.789128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1604,"log_dups_size":0,"ondisk_log_size":1604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011943199999999999,"stat_sum":{"num_bytes":18918293,"num_objects":634,"num_object_clones":295,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":241,"num_read":154,"num_read_kb":114,"num_write":1356,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'557","reported_seq":801,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:57.004659+0000","last_change":"2023-10-30T13:52:57.004722+0000","last_active":"2023-10-30T13:52:57.004659+0000","last_peered":"2023-10-30T13:52:57.004659+0000","last_clean":"2023-10-30T13:52:57.004659+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:52:57.004659+0000","last_undegraded":"2023-10-30T13:52:57.004659+0000","last_fullsized":"2023-10-30T13:52:57.004659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'557","last_scrub_stamp":"2023-10-30T13:52:57.004551+0000","last_deep_scrub":"67'557","last_deep_scrub_stamp":"2023-10-30T13:52:57.004551+0000","last_clean_scrub_stamp":"2023-10-30T13:52:57.004551+0000","objects_scrubbed":57,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:29:17.022207+0000","scrub_duration":0.20230519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":289,"num_read_kb":121284,"num_write":557,"num_write_kb":58696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":599967,"num_omap_keys":1105,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'1640","reported_seq":3523,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.234545+0000","last_change":"2023-10-30T13:21:40.576918+0000","last_active":"2023-10-30T13:52:36.234545+0000","last_peered":"2023-10-30T13:52:36.234545+0000","last_clean":"2023-10-30T13:52:36.234545+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:52:36.234545+0000","last_undegraded":"2023-10-30T13:52:36.234545+0000","last_fullsized":"2023-10-30T13:52:36.234545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1640,"log_dups_size":0,"ondisk_log_size":1640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086123929999999994,"stat_sum":{"num_bytes":19792070,"num_objects":655,"num_object_clones":349,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":204,"num_read":113,"num_read_kb":142,"num_write":1348,"num_write_kb":24080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1016","reported_seq":1307,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.650095+0000","last_change":"2023-10-30T13:52:49.650144+0000","last_active":"2023-10-30T13:52:49.650095+0000","last_peered":"2023-10-30T13:52:49.650095+0000","last_clean":"2023-10-30T13:52:49.650095+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:52:49.650095+0000","last_undegraded":"2023-10-30T13:52:49.650095+0000","last_fullsized":"2023-10-30T13:52:49.650095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1016","last_scrub_stamp":"2023-10-30T13:52:49.650007+0000","last_deep_scrub":"67'1016","last_deep_scrub_stamp":"2023-10-30T13:52:49.650007+0000","last_clean_scrub_stamp":"2023-10-30T13:52:49.650007+0000","objects_scrubbed":55,"log_size":1016,"log_dups_size":0,"ondisk_log_size":1016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:38.507861+0000","scrub_duration":0.17591847999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36678727,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":465,"num_read_kb":133831,"num_write":1108,"num_write_kb":54499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":702779,"num_omap_keys":1002,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'1563","reported_seq":3486,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.789042+0000","last_change":"2023-10-30T13:21:40.583214+0000","last_active":"2023-10-30T13:52:36.789042+0000","last_peered":"2023-10-30T13:52:36.789042+0000","last_clean":"2023-10-30T13:52:36.789042+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:52:36.789042+0000","last_undegraded":"2023-10-30T13:52:36.789042+0000","last_fullsized":"2023-10-30T13:52:36.789042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015028952999999999,"stat_sum":{"num_bytes":21253392,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":236,"num_read":95,"num_read_kb":12339,"num_write":1306,"num_write_kb":30718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'688","reported_seq":924,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.045822+0000","last_change":"2023-10-30T13:52:45.045919+0000","last_active":"2023-10-30T13:52:45.045822+0000","last_peered":"2023-10-30T13:52:45.045822+0000","last_clean":"2023-10-30T13:52:45.045822+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:52:45.045822+0000","last_undegraded":"2023-10-30T13:52:45.045822+0000","last_fullsized":"2023-10-30T13:52:45.045822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'688","last_scrub_stamp":"2023-10-30T13:52:45.045739+0000","last_deep_scrub":"67'688","last_deep_scrub_stamp":"2023-10-30T13:52:45.045739+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.045739+0000","objects_scrubbed":51,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:31:53.871616+0000","scrub_duration":0.14959883700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165846,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":249,"num_read_kb":148701,"num_write":778,"num_write_kb":54421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":420719,"num_omap_keys":809,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'1567","reported_seq":3498,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.484160+0000","last_change":"2023-10-30T13:21:40.579365+0000","last_active":"2023-10-30T13:52:29.484160+0000","last_peered":"2023-10-30T13:52:29.484160+0000","last_clean":"2023-10-30T13:52:29.484160+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:52:29.484160+0000","last_undegraded":"2023-10-30T13:52:29.484160+0000","last_fullsized":"2023-10-30T13:52:29.484160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011682140000000001,"stat_sum":{"num_bytes":26941199,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":237,"num_read":103,"num_read_kb":4114,"num_write":1331,"num_write_kb":32609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'1113","reported_seq":1400,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:48.645056+0000","last_change":"2023-10-30T13:52:48.645108+0000","last_active":"2023-10-30T13:52:48.645056+0000","last_peered":"2023-10-30T13:52:48.645056+0000","last_clean":"2023-10-30T13:52:48.645056+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:52:48.645056+0000","last_undegraded":"2023-10-30T13:52:48.645056+0000","last_fullsized":"2023-10-30T13:52:48.645056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1113","last_scrub_stamp":"2023-10-30T13:52:48.644981+0000","last_deep_scrub":"67'1113","last_deep_scrub_stamp":"2023-10-30T13:52:48.644981+0000","last_clean_scrub_stamp":"2023-10-30T13:52:48.644981+0000","objects_scrubbed":63,"log_size":1113,"log_dups_size":0,"ondisk_log_size":1113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:56.952822+0000","scrub_duration":0.19793386600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":381,"num_read_kb":250673,"num_write":1157,"num_write_kb":87754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":620304,"num_omap_keys":1053,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1540","reported_seq":3483,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.519212+0000","last_change":"2023-10-30T13:21:40.585506+0000","last_active":"2023-10-30T13:52:29.519212+0000","last_peered":"2023-10-30T13:52:29.519212+0000","last_clean":"2023-10-30T13:52:29.519212+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:52:29.519212+0000","last_undegraded":"2023-10-30T13:52:29.519212+0000","last_fullsized":"2023-10-30T13:52:29.519212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014794043,"stat_sum":{"num_bytes":18888621,"num_objects":614,"num_object_clones":283,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":224,"num_read":107,"num_read_kb":4188,"num_write":1306,"num_write_kb":24240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'1274","reported_seq":1535,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.034148+0000","last_change":"2023-10-30T13:52:53.034230+0000","last_active":"2023-10-30T13:52:53.034148+0000","last_peered":"2023-10-30T13:52:53.034148+0000","last_clean":"2023-10-30T13:52:53.034148+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:52:53.034148+0000","last_undegraded":"2023-10-30T13:52:53.034148+0000","last_fullsized":"2023-10-30T13:52:53.034148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1274","last_scrub_stamp":"2023-10-30T13:52:53.034072+0000","last_deep_scrub":"67'1274","last_deep_scrub_stamp":"2023-10-30T13:52:53.034072+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.034072+0000","objects_scrubbed":46,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:05:34.464424+0000","scrub_duration":0.19364663100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364039,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":288,"num_read_kb":200539,"num_write":1327,"num_write_kb":71054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":407575,"num_omap_keys":755,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'2176","reported_seq":4163,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:48.181004+0000","last_change":"2023-10-30T13:52:48.181151+0000","last_active":"2023-10-30T13:52:48.181004+0000","last_peered":"2023-10-30T13:52:48.181004+0000","last_clean":"2023-10-30T13:52:48.181004+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:52:48.181004+0000","last_undegraded":"2023-10-30T13:52:48.181004+0000","last_fullsized":"2023-10-30T13:52:48.181004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2176","last_scrub_stamp":"2023-10-30T13:52:48.180852+0000","last_deep_scrub":"67'2176","last_deep_scrub_stamp":"2023-10-30T13:52:48.180852+0000","last_clean_scrub_stamp":"2023-10-30T13:52:48.180852+0000","objects_scrubbed":600,"log_size":2176,"log_dups_size":0,"ondisk_log_size":2176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:11.947771+0000","scrub_duration":1.0044801649999999,"objects_trimmed":0,"snaptrim_duration":0.010788236,"stat_sum":{"num_bytes":10484856,"num_objects":600,"num_object_clones":281,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":220,"num_read":116,"num_read_kb":28,"num_write":1335,"num_write_kb":13295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'832","reported_seq":1090,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.593477+0000","last_change":"2023-10-30T13:52:53.593522+0000","last_active":"2023-10-30T13:52:53.593477+0000","last_peered":"2023-10-30T13:52:53.593477+0000","last_clean":"2023-10-30T13:52:53.593477+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:52:53.593477+0000","last_undegraded":"2023-10-30T13:52:53.593477+0000","last_fullsized":"2023-10-30T13:52:53.593477+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'832","last_scrub_stamp":"2023-10-30T13:52:53.593416+0000","last_deep_scrub":"67'832","last_deep_scrub_stamp":"2023-10-30T13:52:53.593416+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.593416+0000","objects_scrubbed":54,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:41.436296+0000","scrub_duration":0.145814324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":174067,"num_write":897,"num_write_kb":62874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":516505,"num_omap_keys":962,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1542","reported_seq":3474,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.487490+0000","last_change":"2023-10-30T13:21:40.583977+0000","last_active":"2023-10-30T13:52:29.487490+0000","last_peered":"2023-10-30T13:52:29.487490+0000","last_clean":"2023-10-30T13:52:29.487490+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:52:29.487490+0000","last_undegraded":"2023-10-30T13:52:29.487490+0000","last_fullsized":"2023-10-30T13:52:29.487490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015268978000000001,"stat_sum":{"num_bytes":11043591,"num_objects":616,"num_object_clones":291,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":221,"num_read":104,"num_read_kb":3707,"num_write":1322,"num_write_kb":16509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":244,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.159767+0000","last_change":"2023-10-30T13:52:40.159977+0000","last_active":"2023-10-30T13:52:40.159767+0000","last_peered":"2023-10-30T13:52:40.159767+0000","last_clean":"2023-10-30T13:52:40.159767+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:52:40.159767+0000","last_undegraded":"2023-10-30T13:52:40.159767+0000","last_fullsized":"2023-10-30T13:52:40.159767+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-30T13:52:40.159494+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-30T13:52:40.159494+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.159494+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:37:13.673701+0000","scrub_duration":0.017799540999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'670","reported_seq":889,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:42.315206+0000","last_change":"2023-10-30T13:52:42.315332+0000","last_active":"2023-10-30T13:52:42.315206+0000","last_peered":"2023-10-30T13:52:42.315206+0000","last_clean":"2023-10-30T13:52:42.315206+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:52:42.315206+0000","last_undegraded":"2023-10-30T13:52:42.315206+0000","last_fullsized":"2023-10-30T13:52:42.315206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'670","last_scrub_stamp":"2023-10-30T13:52:42.314975+0000","last_deep_scrub":"67'670","last_deep_scrub_stamp":"2023-10-30T13:52:42.314975+0000","last_clean_scrub_stamp":"2023-10-30T13:52:42.314975+0000","objects_scrubbed":53,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:16.143637+0000","scrub_duration":0.170007133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360178,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":232,"num_read_kb":148904,"num_write":701,"num_write_kb":62706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":385690,"num_omap_keys":753,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'2178","reported_seq":4125,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:11.189453+0000","last_change":"2023-10-30T13:53:11.189527+0000","last_active":"2023-10-30T13:53:11.189453+0000","last_peered":"2023-10-30T13:53:11.189453+0000","last_clean":"2023-10-30T13:53:11.189453+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:53:11.189453+0000","last_undegraded":"2023-10-30T13:53:11.189453+0000","last_fullsized":"2023-10-30T13:53:11.189453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2178","last_scrub_stamp":"2023-10-30T13:53:11.189379+0000","last_deep_scrub":"67'2178","last_deep_scrub_stamp":"2023-10-30T13:53:11.189379+0000","last_clean_scrub_stamp":"2023-10-30T13:53:11.189379+0000","objects_scrubbed":633,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:47.706046+0000","scrub_duration":1.230169694,"objects_trimmed":0,"snaptrim_duration":0.0057314540000000004,"stat_sum":{"num_bytes":22780711,"num_objects":633,"num_object_clones":299,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":242,"num_read":92,"num_read_kb":12,"num_write":1293,"num_write_kb":28237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"67'756","reported_seq":998,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:04.119917+0000","last_change":"2023-10-30T13:53:04.120074+0000","last_active":"2023-10-30T13:53:04.119917+0000","last_peered":"2023-10-30T13:53:04.119917+0000","last_clean":"2023-10-30T13:53:04.119917+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:53:04.119917+0000","last_undegraded":"2023-10-30T13:53:04.119917+0000","last_fullsized":"2023-10-30T13:53:04.119917+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'756","last_scrub_stamp":"2023-10-30T13:53:04.119811+0000","last_deep_scrub":"67'756","last_deep_scrub_stamp":"2023-10-30T13:53:04.119811+0000","last_clean_scrub_stamp":"2023-10-30T13:53:04.119811+0000","objects_scrubbed":49,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:34.012792+0000","scrub_duration":0.144148941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":256,"num_read_kb":247261,"num_write":740,"num_write_kb":70569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":420741,"num_omap_keys":813,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'2028","reported_seq":3997,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.341768+0000","last_change":"2023-10-30T13:52:40.341843+0000","last_active":"2023-10-30T13:52:40.341768+0000","last_peered":"2023-10-30T13:52:40.341768+0000","last_clean":"2023-10-30T13:52:40.341768+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:52:40.341768+0000","last_undegraded":"2023-10-30T13:52:40.341768+0000","last_fullsized":"2023-10-30T13:52:40.341768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2028","last_scrub_stamp":"2023-10-30T13:52:40.341675+0000","last_deep_scrub":"67'2028","last_deep_scrub_stamp":"2023-10-30T13:52:40.341675+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.341675+0000","objects_scrubbed":568,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:22:08.372774+0000","scrub_duration":1.158419098,"objects_trimmed":0,"snaptrim_duration":0.002406114,"stat_sum":{"num_bytes":14184993,"num_objects":568,"num_object_clones":263,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":210,"num_read":80,"num_read_kb":26,"num_write":1239,"num_write_kb":16314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'2090","reported_seq":3466,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.634617+0000","last_change":"2023-10-30T13:52:36.634675+0000","last_active":"2023-10-30T13:52:36.634617+0000","last_peered":"2023-10-30T13:52:36.634617+0000","last_clean":"2023-10-30T13:52:36.634617+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:52:36.634617+0000","last_undegraded":"2023-10-30T13:52:36.634617+0000","last_fullsized":"2023-10-30T13:52:36.634617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2090","last_scrub_stamp":"2023-10-30T13:52:36.634514+0000","last_deep_scrub":"67'2090","last_deep_scrub_stamp":"2023-10-30T13:52:36.634514+0000","last_clean_scrub_stamp":"2023-10-30T13:52:36.634514+0000","objects_scrubbed":53,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:20.160935+0000","scrub_duration":0.14709519600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41670866,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":1534,"num_read_kb":188566,"num_write":2090,"num_write_kb":83865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":477756,"num_omap_keys":784,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'2139","reported_seq":4115,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:09.536900+0000","last_change":"2023-10-30T13:53:09.536942+0000","last_active":"2023-10-30T13:53:09.536900+0000","last_peered":"2023-10-30T13:53:09.536900+0000","last_clean":"2023-10-30T13:53:09.536900+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:53:09.536900+0000","last_undegraded":"2023-10-30T13:53:09.536900+0000","last_fullsized":"2023-10-30T13:53:09.536900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2139","last_scrub_stamp":"2023-10-30T13:53:09.536825+0000","last_deep_scrub":"67'2139","last_deep_scrub_stamp":"2023-10-30T13:53:09.536825+0000","last_clean_scrub_stamp":"2023-10-30T13:53:09.536825+0000","objects_scrubbed":612,"log_size":2139,"log_dups_size":0,"ondisk_log_size":2139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:54:03.730208+0000","scrub_duration":0.83069248900000003,"objects_trimmed":0,"snaptrim_duration":0.002908281,"stat_sum":{"num_bytes":22417951,"num_objects":612,"num_object_clones":284,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":221,"num_read":105,"num_read_kb":44,"num_write":1298,"num_write_kb":23585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'1007","reported_seq":1293,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.427338+0000","last_change":"2023-10-30T13:52:36.427454+0000","last_active":"2023-10-30T13:52:36.427338+0000","last_peered":"2023-10-30T13:52:36.427338+0000","last_clean":"2023-10-30T13:52:36.427338+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:52:36.427338+0000","last_undegraded":"2023-10-30T13:52:36.427338+0000","last_fullsized":"2023-10-30T13:52:36.427338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1007","last_scrub_stamp":"2023-10-30T13:52:36.427140+0000","last_deep_scrub":"67'1007","last_deep_scrub_stamp":"2023-10-30T13:52:36.427140+0000","last_clean_scrub_stamp":"2023-10-30T13:52:36.427140+0000","objects_scrubbed":60,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:11.996639+0000","scrub_duration":0.17205899599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":355,"num_read_kb":252215,"num_write":1079,"num_write_kb":75669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":633823,"num_omap_keys":985,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'2106","reported_seq":4092,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.008269+0000","last_change":"2023-10-30T13:52:51.008354+0000","last_active":"2023-10-30T13:52:51.008269+0000","last_peered":"2023-10-30T13:52:51.008269+0000","last_clean":"2023-10-30T13:52:51.008269+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:52:51.008269+0000","last_undegraded":"2023-10-30T13:52:51.008269+0000","last_fullsized":"2023-10-30T13:52:51.008269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2106","last_scrub_stamp":"2023-10-30T13:52:51.008186+0000","last_deep_scrub":"67'2106","last_deep_scrub_stamp":"2023-10-30T13:52:51.008186+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.008186+0000","objects_scrubbed":575,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:38.209876+0000","scrub_duration":1.1485167089999999,"objects_trimmed":0,"snaptrim_duration":0.007710705,"stat_sum":{"num_bytes":23751577,"num_objects":575,"num_object_clones":266,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":199,"num_read":100,"num_read_kb":489,"num_write":1325,"num_write_kb":27587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'998","reported_seq":1249,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.917921+0000","last_change":"2023-10-30T13:52:43.917980+0000","last_active":"2023-10-30T13:52:43.917921+0000","last_peered":"2023-10-30T13:52:43.917921+0000","last_clean":"2023-10-30T13:52:43.917921+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:52:43.917921+0000","last_undegraded":"2023-10-30T13:52:43.917921+0000","last_fullsized":"2023-10-30T13:52:43.917921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'998","last_scrub_stamp":"2023-10-30T13:52:43.917841+0000","last_deep_scrub":"67'998","last_deep_scrub_stamp":"2023-10-30T13:52:43.917841+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.917841+0000","objects_scrubbed":56,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:41:43.976830+0000","scrub_duration":0.14471989900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":294,"num_read_kb":227180,"num_write":1039,"num_write_kb":87590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":525988,"num_omap_keys":913,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2248","reported_seq":4193,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:01.680696+0000","last_change":"2023-10-30T13:53:01.680795+0000","last_active":"2023-10-30T13:53:01.680696+0000","last_peered":"2023-10-30T13:53:01.680696+0000","last_clean":"2023-10-30T13:53:01.680696+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:53:01.680696+0000","last_undegraded":"2023-10-30T13:53:01.680696+0000","last_fullsized":"2023-10-30T13:53:01.680696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2248","last_scrub_stamp":"2023-10-30T13:53:01.680522+0000","last_deep_scrub":"67'2248","last_deep_scrub_stamp":"2023-10-30T13:53:01.680522+0000","last_clean_scrub_stamp":"2023-10-30T13:53:01.680522+0000","objects_scrubbed":650,"log_size":2248,"log_dups_size":0,"ondisk_log_size":2248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:34.935002+0000","scrub_duration":1.35250176,"objects_trimmed":0,"snaptrim_duration":0.010719574000000001,"stat_sum":{"num_bytes":35538242,"num_objects":650,"num_object_clones":301,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":239,"num_read":91,"num_read_kb":31,"num_write":1347,"num_write_kb":41412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'787","reported_seq":1013,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.142911+0000","last_change":"2023-10-30T13:10:33.721559+0000","last_active":"2023-10-30T13:52:29.142911+0000","last_peered":"2023-10-30T13:52:29.142911+0000","last_clean":"2023-10-30T13:52:29.142911+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:52:29.142911+0000","last_undegraded":"2023-10-30T13:52:29.142911+0000","last_fullsized":"2023-10-30T13:52:29.142911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:32.661389+0000","last_clean_scrub_stamp":"2023-10-30T13:10:32.661389+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":258,"num_read_kb":177173,"num_write":845,"num_write_kb":62523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'1551","reported_seq":3478,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.395707+0000","last_change":"2023-10-30T13:21:40.572129+0000","last_active":"2023-10-30T13:52:29.395707+0000","last_peered":"2023-10-30T13:52:29.395707+0000","last_clean":"2023-10-30T13:52:29.395707+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:52:29.395707+0000","last_undegraded":"2023-10-30T13:52:29.395707+0000","last_fullsized":"2023-10-30T13:52:29.395707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059276069999999997,"stat_sum":{"num_bytes":23389062,"num_objects":631,"num_object_clones":294,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":232,"num_read":100,"num_read_kb":25,"num_write":1313,"num_write_kb":24315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'969","reported_seq":1219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:00.092676+0000","last_change":"2023-10-30T13:53:00.092749+0000","last_active":"2023-10-30T13:53:00.092676+0000","last_peered":"2023-10-30T13:53:00.092676+0000","last_clean":"2023-10-30T13:53:00.092676+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:53:00.092676+0000","last_undegraded":"2023-10-30T13:53:00.092676+0000","last_fullsized":"2023-10-30T13:53:00.092676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'969","last_scrub_stamp":"2023-10-30T13:53:00.092550+0000","last_deep_scrub":"67'969","last_deep_scrub_stamp":"2023-10-30T13:53:00.092550+0000","last_clean_scrub_stamp":"2023-10-30T13:53:00.092550+0000","objects_scrubbed":54,"log_size":969,"log_dups_size":0,"ondisk_log_size":969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:09.851840+0000","scrub_duration":0.31438043799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":310,"num_read_kb":109156,"num_write":1496,"num_write_kb":61821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":485487,"num_omap_keys":848,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1992","reported_seq":3976,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:56.936214+0000","last_change":"2023-10-30T13:52:56.936303+0000","last_active":"2023-10-30T13:52:56.936214+0000","last_peered":"2023-10-30T13:52:56.936214+0000","last_clean":"2023-10-30T13:52:56.936214+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:52:56.936214+0000","last_undegraded":"2023-10-30T13:52:56.936214+0000","last_fullsized":"2023-10-30T13:52:56.936214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1992","last_scrub_stamp":"2023-10-30T13:52:56.936120+0000","last_deep_scrub":"67'1992","last_deep_scrub_stamp":"2023-10-30T13:52:56.936120+0000","last_clean_scrub_stamp":"2023-10-30T13:52:56.936120+0000","objects_scrubbed":571,"log_size":1992,"log_dups_size":0,"ondisk_log_size":1992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:21.220503+0000","scrub_duration":1.0834315910000001,"objects_trimmed":0,"snaptrim_duration":0.0056940879999999999,"stat_sum":{"num_bytes":23042560,"num_objects":571,"num_object_clones":259,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":201,"num_read":92,"num_read_kb":75,"num_write":1201,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'1157","reported_seq":1411,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.055552+0000","last_change":"2023-10-30T13:52:54.055652+0000","last_active":"2023-10-30T13:52:54.055552+0000","last_peered":"2023-10-30T13:52:54.055552+0000","last_clean":"2023-10-30T13:52:54.055552+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:52:54.055552+0000","last_undegraded":"2023-10-30T13:52:54.055552+0000","last_fullsized":"2023-10-30T13:52:54.055552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1157","last_scrub_stamp":"2023-10-30T13:52:54.055439+0000","last_deep_scrub":"67'1157","last_deep_scrub_stamp":"2023-10-30T13:52:54.055439+0000","last_clean_scrub_stamp":"2023-10-30T13:52:54.055439+0000","objects_scrubbed":62,"log_size":1157,"log_dups_size":0,"ondisk_log_size":1157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:52.120399+0000","scrub_duration":0.23085536300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":342,"num_read_kb":120967,"num_write":1233,"num_write_kb":51153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":723056,"num_omap_keys":1120,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'2017","reported_seq":4010,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.248739+0000","last_change":"2023-10-30T13:52:59.248901+0000","last_active":"2023-10-30T13:52:59.248739+0000","last_peered":"2023-10-30T13:52:59.248739+0000","last_clean":"2023-10-30T13:52:59.248739+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:52:59.248739+0000","last_undegraded":"2023-10-30T13:52:59.248739+0000","last_fullsized":"2023-10-30T13:52:59.248739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2017","last_scrub_stamp":"2023-10-30T13:52:59.248637+0000","last_deep_scrub":"67'2017","last_deep_scrub_stamp":"2023-10-30T13:52:59.248637+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.248637+0000","objects_scrubbed":572,"log_size":2017,"log_dups_size":0,"ondisk_log_size":2017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:42.043697+0000","scrub_duration":1.079787013,"objects_trimmed":0,"snaptrim_duration":0.0029926589999999999,"stat_sum":{"num_bytes":30813966,"num_objects":572,"num_object_clones":263,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":205,"num_read":105,"num_read_kb":87,"num_write":1219,"num_write_kb":33525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'468","reported_seq":664,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.111720+0000","last_change":"2023-10-30T13:52:59.111898+0000","last_active":"2023-10-30T13:52:59.111720+0000","last_peered":"2023-10-30T13:52:59.111720+0000","last_clean":"2023-10-30T13:52:59.111720+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:52:59.111720+0000","last_undegraded":"2023-10-30T13:52:59.111720+0000","last_fullsized":"2023-10-30T13:52:59.111720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'468","last_scrub_stamp":"2023-10-30T13:52:59.111509+0000","last_deep_scrub":"67'468","last_deep_scrub_stamp":"2023-10-30T13:52:59.111509+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.111509+0000","objects_scrubbed":47,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:39.063865+0000","scrub_duration":0.1401828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":209,"num_read_kb":100741,"num_write":482,"num_write_kb":29419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":340147,"num_omap_keys":605,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'2140","reported_seq":4127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:57.118260+0000","last_change":"2023-10-30T13:52:57.118430+0000","last_active":"2023-10-30T13:52:57.118260+0000","last_peered":"2023-10-30T13:52:57.118260+0000","last_clean":"2023-10-30T13:52:57.118260+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:52:57.118260+0000","last_undegraded":"2023-10-30T13:52:57.118260+0000","last_fullsized":"2023-10-30T13:52:57.118260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2140","last_scrub_stamp":"2023-10-30T13:52:57.118012+0000","last_deep_scrub":"67'2140","last_deep_scrub_stamp":"2023-10-30T13:52:57.118012+0000","last_clean_scrub_stamp":"2023-10-30T13:52:57.118012+0000","objects_scrubbed":595,"log_size":2140,"log_dups_size":0,"ondisk_log_size":2140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:16:30.111801+0000","scrub_duration":1.2581118739999999,"objects_trimmed":0,"snaptrim_duration":0.0057673200000000003,"stat_sum":{"num_bytes":13994934,"num_objects":595,"num_object_clones":279,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":215,"num_read":114,"num_read_kb":29,"num_write":1329,"num_write_kb":19694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'1276","reported_seq":1501,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.113521+0000","last_change":"2023-10-30T13:52:34.896255+0000","last_active":"2023-10-30T13:52:54.113521+0000","last_peered":"2023-10-30T13:52:54.113521+0000","last_clean":"2023-10-30T13:52:54.113521+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:52:54.113521+0000","last_undegraded":"2023-10-30T13:52:54.113521+0000","last_fullsized":"2023-10-30T13:52:54.113521+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1266","last_scrub_stamp":"2023-10-30T13:52:34.896121+0000","last_deep_scrub":"67'1266","last_deep_scrub_stamp":"2023-10-30T13:52:34.896121+0000","last_clean_scrub_stamp":"2023-10-30T13:52:34.896121+0000","objects_scrubbed":51,"log_size":1276,"log_dups_size":0,"ondisk_log_size":1276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:43.920989+0000","scrub_duration":0.15482325699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25006847,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":251,"num_read_kb":99282,"num_write":1301,"num_write_kb":42450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":678730,"num_omap_keys":1151,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'2168","reported_seq":4155,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:06.891046+0000","last_change":"2023-10-30T13:53:06.891119+0000","last_active":"2023-10-30T13:53:06.891046+0000","last_peered":"2023-10-30T13:53:06.891046+0000","last_clean":"2023-10-30T13:53:06.891046+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:53:06.891046+0000","last_undegraded":"2023-10-30T13:53:06.891046+0000","last_fullsized":"2023-10-30T13:53:06.891046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2168","last_scrub_stamp":"2023-10-30T13:53:06.890960+0000","last_deep_scrub":"67'2168","last_deep_scrub_stamp":"2023-10-30T13:53:06.890960+0000","last_clean_scrub_stamp":"2023-10-30T13:53:06.890960+0000","objects_scrubbed":612,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:45:46.242387+0000","scrub_duration":0.94320374900000004,"objects_trimmed":0,"snaptrim_duration":0.0063570620000000001,"stat_sum":{"num_bytes":16293763,"num_objects":612,"num_object_clones":286,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":233,"num_read":120,"num_read_kb":127,"num_write":1302,"num_write_kb":21658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'946","reported_seq":1182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:35.887235+0000","last_change":"2023-10-30T13:52:35.887314+0000","last_active":"2023-10-30T13:52:35.887235+0000","last_peered":"2023-10-30T13:52:35.887235+0000","last_clean":"2023-10-30T13:52:35.887235+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:52:35.887235+0000","last_undegraded":"2023-10-30T13:52:35.887235+0000","last_fullsized":"2023-10-30T13:52:35.887235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'946","last_scrub_stamp":"2023-10-30T13:52:35.887142+0000","last_deep_scrub":"67'946","last_deep_scrub_stamp":"2023-10-30T13:52:35.887142+0000","last_clean_scrub_stamp":"2023-10-30T13:52:35.887142+0000","objects_scrubbed":46,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:06:00.273772+0000","scrub_duration":0.098447004000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":272,"num_read_kb":200216,"num_write":1137,"num_write_kb":59039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":497965,"num_omap_keys":763,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1999","reported_seq":3979,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.786787+0000","last_change":"2023-10-30T13:52:59.786866+0000","last_active":"2023-10-30T13:52:59.786787+0000","last_peered":"2023-10-30T13:52:59.786787+0000","last_clean":"2023-10-30T13:52:59.786787+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:52:59.786787+0000","last_undegraded":"2023-10-30T13:52:59.786787+0000","last_fullsized":"2023-10-30T13:52:59.786787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1999","last_scrub_stamp":"2023-10-30T13:52:59.786686+0000","last_deep_scrub":"67'1999","last_deep_scrub_stamp":"2023-10-30T13:52:59.786686+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.786686+0000","objects_scrubbed":563,"log_size":1999,"log_dups_size":0,"ondisk_log_size":1999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:35.870896+0000","scrub_duration":1.004166471,"objects_trimmed":0,"snaptrim_duration":0.0060201619999999999,"stat_sum":{"num_bytes":10033117,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":215,"num_read":92,"num_read_kb":24,"num_write":1204,"num_write_kb":13175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'1025","reported_seq":1291,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:44.395023+0000","last_change":"2023-10-30T13:52:44.395104+0000","last_active":"2023-10-30T13:52:44.395023+0000","last_peered":"2023-10-30T13:52:44.395023+0000","last_clean":"2023-10-30T13:52:44.395023+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:52:44.395023+0000","last_undegraded":"2023-10-30T13:52:44.395023+0000","last_fullsized":"2023-10-30T13:52:44.395023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1025","last_scrub_stamp":"2023-10-30T13:52:44.394896+0000","last_deep_scrub":"67'1025","last_deep_scrub_stamp":"2023-10-30T13:52:44.394896+0000","last_clean_scrub_stamp":"2023-10-30T13:52:44.394896+0000","objects_scrubbed":56,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:31.686901+0000","scrub_duration":0.17278753699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":318,"num_read_kb":202944,"num_write":1065,"num_write_kb":67614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":660981,"num_omap_keys":1062,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'1468","reported_seq":3417,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.712417+0000","last_change":"2023-10-30T13:21:40.575093+0000","last_active":"2023-10-30T13:52:29.712417+0000","last_peered":"2023-10-30T13:52:29.712417+0000","last_clean":"2023-10-30T13:52:29.712417+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:52:29.712417+0000","last_undegraded":"2023-10-30T13:52:29.712417+0000","last_fullsized":"2023-10-30T13:52:29.712417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079253199999999996,"stat_sum":{"num_bytes":6591050,"num_objects":558,"num_object_clones":254,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":200,"num_read":83,"num_read_kb":63,"num_write":1255,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'2083","reported_seq":4060,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:04.412236+0000","last_change":"2023-10-30T13:53:04.412236+0000","last_active":"2023-10-30T13:53:04.412236+0000","last_peered":"2023-10-30T13:53:04.412236+0000","last_clean":"2023-10-30T13:53:04.412236+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:53:04.412236+0000","last_undegraded":"2023-10-30T13:53:04.412236+0000","last_fullsized":"2023-10-30T13:53:04.412236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2083","last_scrub_stamp":"2023-10-30T13:53:04.411986+0000","last_deep_scrub":"67'2083","last_deep_scrub_stamp":"2023-10-30T13:53:04.411986+0000","last_clean_scrub_stamp":"2023-10-30T13:53:04.411986+0000","objects_scrubbed":585,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:37.812935+0000","scrub_duration":1.1529056559999999,"objects_trimmed":0,"snaptrim_duration":0.010603414,"stat_sum":{"num_bytes":9104816,"num_objects":585,"num_object_clones":276,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":204,"num_read":104,"num_read_kb":44,"num_write":1285,"num_write_kb":14447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'1055","reported_seq":1321,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.345144+0000","last_change":"2023-10-30T13:52:43.345144+0000","last_active":"2023-10-30T13:52:43.345144+0000","last_peered":"2023-10-30T13:52:43.345144+0000","last_clean":"2023-10-30T13:52:43.345144+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:52:43.345144+0000","last_undegraded":"2023-10-30T13:52:43.345144+0000","last_fullsized":"2023-10-30T13:52:43.345144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1055","last_scrub_stamp":"2023-10-30T13:52:43.344949+0000","last_deep_scrub":"67'1055","last_deep_scrub_stamp":"2023-10-30T13:52:43.344949+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.344949+0000","objects_scrubbed":55,"log_size":1055,"log_dups_size":0,"ondisk_log_size":1055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:59:10.108303+0000","scrub_duration":0.154960875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":337,"num_read_kb":187117,"num_write":1200,"num_write_kb":47577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706060,"num_omap_keys":1147,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'2277","reported_seq":4227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:55.617708+0000","last_change":"2023-10-30T13:52:55.617708+0000","last_active":"2023-10-30T13:52:55.617708+0000","last_peered":"2023-10-30T13:52:55.617708+0000","last_clean":"2023-10-30T13:52:55.617708+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:52:55.617708+0000","last_undegraded":"2023-10-30T13:52:55.617708+0000","last_fullsized":"2023-10-30T13:52:55.617708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2277","last_scrub_stamp":"2023-10-30T13:52:55.617575+0000","last_deep_scrub":"67'2277","last_deep_scrub_stamp":"2023-10-30T13:52:55.617575+0000","last_clean_scrub_stamp":"2023-10-30T13:52:55.617575+0000","objects_scrubbed":650,"log_size":2277,"log_dups_size":0,"ondisk_log_size":2277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:16:39.666383+0000","scrub_duration":1.3596852369999999,"objects_trimmed":0,"snaptrim_duration":0.0097894100000000001,"stat_sum":{"num_bytes":19085997,"num_objects":650,"num_object_clones":307,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":230,"num_read":105,"num_read_kb":202,"num_write":1384,"num_write_kb":24607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1382","reported_seq":1696,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:05.200352+0000","last_change":"2023-10-30T13:53:05.200352+0000","last_active":"2023-10-30T13:53:05.200352+0000","last_peered":"2023-10-30T13:53:05.200352+0000","last_clean":"2023-10-30T13:53:05.200352+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:53:05.200352+0000","last_undegraded":"2023-10-30T13:53:05.200352+0000","last_fullsized":"2023-10-30T13:53:05.200352+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1382","last_scrub_stamp":"2023-10-30T13:53:05.200180+0000","last_deep_scrub":"67'1382","last_deep_scrub_stamp":"2023-10-30T13:53:05.200180+0000","last_clean_scrub_stamp":"2023-10-30T13:53:05.200180+0000","objects_scrubbed":68,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:00.361266+0000","scrub_duration":0.25105026499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37661445,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":418,"num_read_kb":304041,"num_write":1419,"num_write_kb":92539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903188,"num_omap_keys":1478,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'2147","reported_seq":4104,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:06.340082+0000","last_change":"2023-10-30T13:53:06.340161+0000","last_active":"2023-10-30T13:53:06.340082+0000","last_peered":"2023-10-30T13:53:06.340082+0000","last_clean":"2023-10-30T13:53:06.340082+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:53:06.340082+0000","last_undegraded":"2023-10-30T13:53:06.340082+0000","last_fullsized":"2023-10-30T13:53:06.340082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2147","last_scrub_stamp":"2023-10-30T13:53:06.340001+0000","last_deep_scrub":"67'2147","last_deep_scrub_stamp":"2023-10-30T13:53:06.340001+0000","last_clean_scrub_stamp":"2023-10-30T13:53:06.340001+0000","objects_scrubbed":636,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:44.975735+0000","scrub_duration":1.142281364,"objects_trimmed":0,"snaptrim_duration":0.0026802879999999999,"stat_sum":{"num_bytes":22843366,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":237,"num_read":96,"num_read_kb":23,"num_write":1281,"num_write_kb":25082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'822","reported_seq":1066,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:44.024796+0000","last_change":"2023-10-30T13:52:44.024796+0000","last_active":"2023-10-30T13:52:44.024796+0000","last_peered":"2023-10-30T13:52:44.024796+0000","last_clean":"2023-10-30T13:52:44.024796+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:52:44.024796+0000","last_undegraded":"2023-10-30T13:52:44.024796+0000","last_fullsized":"2023-10-30T13:52:44.024796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'822","last_scrub_stamp":"2023-10-30T13:52:44.024664+0000","last_deep_scrub":"67'822","last_deep_scrub_stamp":"2023-10-30T13:52:44.024664+0000","last_clean_scrub_stamp":"2023-10-30T13:52:44.024664+0000","objects_scrubbed":50,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:54.864042+0000","scrub_duration":0.11718745699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":280,"num_read_kb":212075,"num_write":835,"num_write_kb":74769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":365319,"num_omap_keys":701,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'2055","reported_seq":4042,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:05.863446+0000","last_change":"2023-10-30T13:53:05.863560+0000","last_active":"2023-10-30T13:53:05.863446+0000","last_peered":"2023-10-30T13:53:05.863446+0000","last_clean":"2023-10-30T13:53:05.863446+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:53:05.863446+0000","last_undegraded":"2023-10-30T13:53:05.863446+0000","last_fullsized":"2023-10-30T13:53:05.863446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2055","last_scrub_stamp":"2023-10-30T13:53:05.863371+0000","last_deep_scrub":"67'2055","last_deep_scrub_stamp":"2023-10-30T13:53:05.863371+0000","last_clean_scrub_stamp":"2023-10-30T13:53:05.863371+0000","objects_scrubbed":589,"log_size":2055,"log_dups_size":0,"ondisk_log_size":2055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:54:20.796229+0000","scrub_duration":0.95279065399999996,"objects_trimmed":0,"snaptrim_duration":0.0055040289999999997,"stat_sum":{"num_bytes":14435337,"num_objects":589,"num_object_clones":269,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":212,"num_read":103,"num_read_kb":22,"num_write":1235,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'768","reported_seq":1042,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:34.271316+0000","last_change":"2023-10-30T13:52:34.271376+0000","last_active":"2023-10-30T13:52:34.271316+0000","last_peered":"2023-10-30T13:52:34.271316+0000","last_clean":"2023-10-30T13:52:34.271316+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:52:34.271316+0000","last_undegraded":"2023-10-30T13:52:34.271316+0000","last_fullsized":"2023-10-30T13:52:34.271316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'768","last_scrub_stamp":"2023-10-30T13:52:34.271229+0000","last_deep_scrub":"67'768","last_deep_scrub_stamp":"2023-10-30T13:52:34.271229+0000","last_clean_scrub_stamp":"2023-10-30T13:52:34.271229+0000","objects_scrubbed":64,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:00.506995+0000","scrub_duration":0.20155240999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749318,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":374,"num_read_kb":96620,"num_write":840,"num_write_kb":51020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":647968,"num_omap_keys":1098,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165182927,"num_objects":39302,"num_object_clones":18317,"num_object_copies":78604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39302,"num_whiteouts":14350,"num_read":6653,"num_read_kb":168434,"num_write":83476,"num_write_kb":1459733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424070144,"data_stored":2332381086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125304,"ondisk_log_size":125304,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":909135531,"num_objects":1701,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":11399,"num_read_kb":5963012,"num_write":34147,"num_write_kb":2074566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1411,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17412914,"num_omap_keys":29642,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1818419200,"data_stored":1818271062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32004,"ondisk_log_size":32004,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575367,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730552,"kb_used_data":577200,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93641288,"statfs":{"total":96636764160,"available":95888678912,"internally_reserved":0,"allocated":591052800,"data_stored":577425546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":157022667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.81000000000000005,"15min":0.91300000000000003},"min":{"1min":0.45100000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3080000000000001,"15min":15.381},"last":0.627},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.78100000000000003,"15min":0.84599999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.315},"max":{"1min":1.05,"5min":1.8400000000000001,"15min":2.8079999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.80700000000000005,"15min":0.90600000000000003},"min":{"1min":0.51300000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1839999999999999,"5min":1.79,"15min":15.271000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.87,"15min":0.95199999999999996},"min":{"1min":0.58199999999999996,"5min":0.46000000000000002,"15min":0.42999999999999999},"max":{"1min":1.2769999999999999,"5min":1.6000000000000001,"15min":15.228999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"min":{"1min":0.65000000000000002,"5min":0.45600000000000002,"15min":0.36299999999999999},"max":{"1min":1.123,"5min":1.345,"15min":2.2429999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.86699999999999999,"15min":0.86599999999999999},"min":{"1min":0.378,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4610000000000001,"5min":1.929,"15min":1.929},"last":0.77300000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.91800000000000004,"15min":1.056},"min":{"1min":0.54100000000000004,"5min":0.372,"15min":0.372},"max":{"1min":1.159,"5min":2.0249999999999999,"15min":15.324},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.99399999999999999,"15min":0.98999999999999999},"min":{"1min":0.65400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.379,"5min":1.379,"15min":1.637},"last":1.0780000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.05,"15min":1.115},"min":{"1min":0.47999999999999998,"5min":0.433,"15min":0.40899999999999997},"max":{"1min":1.2929999999999999,"5min":2.7869999999999999,"15min":15.172000000000001},"last":1.034},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0740000000000001,"15min":1.077},"min":{"1min":0.39700000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.6379999999999999,"5min":3.032,"15min":3.032},"last":1.1830000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.091,"15min":1.1000000000000001},"min":{"1min":0.57999999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.409,"5min":1.4770000000000001,"15min":1.7050000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.046,"15min":1.0109999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.554,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":0.998}]},{"osd":6,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1850000000000001,"15min":1.2470000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.8340000000000001,"5min":3.0350000000000001,"15min":15.427},"last":1.27},{"interface":"front","average":{"1min":1.266,"5min":1.24,"15min":1.3200000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.42199999999999999},"max":{"1min":1.948,"5min":3.5169999999999999,"15min":32.976999999999997},"last":1.319}]}]},{"osd":6,"up_from":13,"seq":55834575366,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591964,"kb_used_data":452564,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93779876,"statfs":{"total":96636764160,"available":96030593024,"internally_reserved":0,"allocated":463425536,"data_stored":454827877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142735815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.82599999999999996,"15min":0.83999999999999997},"min":{"1min":0.53400000000000003,"5min":0.45000000000000001,"15min":0.40999999999999998},"max":{"1min":1.1890000000000001,"5min":1.3999999999999999,"15min":5.9880000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.877,"15min":0.88600000000000001},"min":{"1min":0.63500000000000001,"5min":0.48599999999999999,"15min":0.35999999999999999},"max":{"1min":1.1519999999999999,"5min":1.3160000000000001,"15min":6.0730000000000004},"last":0.66700000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.90900000000000003,"15min":0.86899999999999999},"min":{"1min":0.54000000000000004,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.153,"5min":6.9930000000000003,"15min":6.9930000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.85199999999999998,"15min":0.85499999999999998},"min":{"1min":0.67500000000000004,"5min":0.52500000000000002,"15min":0.47499999999999998},"max":{"1min":1.0620000000000001,"5min":1.2410000000000001,"15min":1.9810000000000001},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.91500000000000004,"15min":0.89700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.47699999999999998},"max":{"1min":1.141,"5min":6.8550000000000004,"15min":6.8550000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.95599999999999996,"15min":0.96199999999999997},"min":{"1min":0.63400000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2110000000000001,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.65100000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.56000000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.4830000000000001,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.98099999999999998,"15min":0.98399999999999999},"min":{"1min":0.72699999999999998,"5min":0.64400000000000002,"15min":0.629},"max":{"1min":1.3100000000000001,"5min":5.3179999999999996,"15min":5.3179999999999996},"last":0.94799999999999995}]},{"osd":4,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.98199999999999998,"15min":1.0369999999999999},"min":{"1min":0.752,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3129999999999999,"5min":1.4970000000000001,"15min":15.372},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.089,"15min":1.048},"min":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.58299999999999996},"max":{"1min":1.5820000000000001,"5min":5.2729999999999997,"15min":6.1580000000000004},"last":1.012}]},{"osd":5,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.036,"15min":1.032},"min":{"1min":0.69399999999999995,"5min":0.60499999999999998,"15min":0.60199999999999998},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.8080000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.234,"5min":1.198,"15min":1.1839999999999999},"min":{"1min":0.93999999999999995,"5min":0.73899999999999999,"15min":0.67700000000000005},"max":{"1min":1.6519999999999999,"5min":1.6639999999999999,"15min":1.913},"last":0.79900000000000004}]},{"osd":7,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.1080000000000001,"15min":1.0980000000000001},"min":{"1min":0.86899999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":2.202},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.22,"15min":1.2},"min":{"1min":0.99299999999999999,"5min":0.69599999999999995,"15min":0.42499999999999999},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":3.0590000000000002},"last":1.1240000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575367,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672628,"kb_used_data":519276,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93699212,"statfs":{"total":96636764160,"available":95947993088,"internally_reserved":0,"allocated":531738624,"data_stored":518585685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":157022662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.88900000000000001,"15min":0.87},"min":{"1min":0.374,"5min":0.374,"15min":0.32700000000000001},"max":{"1min":1.2869999999999999,"5min":1.4379999999999999,"15min":6.4729999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.1140000000000001,"5min":0.95099999999999996,"15min":0.88900000000000001},"min":{"1min":0.55400000000000005,"5min":0.53100000000000003,"15min":0.39100000000000001},"max":{"1min":3.585,"5min":3.585,"15min":8.7560000000000002},"last":0.83899999999999997}]},{"osd":1,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.874,"15min":0.89100000000000001},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.42799999999999999},"max":{"1min":3.6360000000000001,"5min":3.6360000000000001,"15min":15.257999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.151,"5min":0.91900000000000004,"15min":0.96299999999999997},"min":{"1min":0.54200000000000004,"5min":0.52200000000000002,"15min":0.47399999999999998},"max":{"1min":4.0439999999999996,"5min":4.0439999999999996,"15min":15.175000000000001},"last":0.48999999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.90000000000000002,"15min":0.91700000000000004},"min":{"1min":0.503,"5min":0.503,"15min":0.42999999999999999},"max":{"1min":3.0179999999999998,"5min":3.0179999999999998,"15min":6.5410000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.997,"5min":0.94099999999999995,"15min":0.98899999999999999},"min":{"1min":0.56499999999999995,"5min":0.49099999999999999,"15min":0.46000000000000002},"max":{"1min":2.8879999999999999,"5min":2.8879999999999999,"15min":6.7080000000000002},"last":0.88100000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.087,"15min":1.048},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.54900000000000004},"max":{"1min":2.9340000000000002,"5min":5.4249999999999998,"15min":6.7699999999999996},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.181,"5min":1.171,"15min":1.1719999999999999},"min":{"1min":0.74399999999999999,"5min":0.58699999999999997,"15min":0.40100000000000002},"max":{"1min":2.9750000000000001,"5min":5.5570000000000004,"15min":15.364000000000001},"last":1.0429999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:52:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.94299999999999995,"15min":0.98499999999999999},"min":{"1min":0.621,"5min":0.376,"15min":0.36499999999999999},"max":{"1min":1.24,"5min":4.3529999999999998,"15min":5.9720000000000004},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2350000000000001,"15min":1.157},"min":{"1min":0.65900000000000003,"5min":0.56699999999999995,"15min":0.53000000000000003},"max":{"1min":1.395,"5min":5.6609999999999996,"15min":14.314},"last":1.131}]},{"osd":6,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.0720000000000001,"15min":1.129},"min":{"1min":0.68100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":3.4279999999999999,"5min":4.2290000000000001,"15min":15.06},"last":1.012},{"interface":"front","average":{"1min":1.363,"5min":1.157,"15min":1.1899999999999999},"min":{"1min":0.79200000000000004,"5min":0.65700000000000003,"15min":0.50900000000000001},"max":{"1min":3.508,"5min":3.508,"15min":7.6630000000000003},"last":1.095}]},{"osd":7,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1.0289999999999999,"15min":1.105},"min":{"1min":0.46300000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.98099999999999998,"5min":5.0419999999999998,"15min":6.8259999999999996},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.208,"15min":1.2729999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.494,"5min":2.8250000000000002,"15min":19.321999999999999},"last":1.194}]}]},{"osd":4,"up_from":13,"seq":55834575368,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726856,"kb_used_data":569408,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93644984,"statfs":{"total":96636764160,"available":95892463616,"internally_reserved":0,"allocated":583073792,"data_stored":568869284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":161216966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.751,"15min":0.79700000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.1839999999999999,"5min":2.202,"15min":9.6690000000000005},"last":0.624},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.48499999999999999,"5min":0.47699999999999998,"15min":0.377},"max":{"1min":1.206,"5min":2.0760000000000001,"15min":2.786},"last":0.876}]},{"osd":1,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.84899999999999998,"15min":0.872},"min":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.41399999999999998},"max":{"1min":1.278,"5min":1.8160000000000001,"15min":5.5119999999999996},"last":1.036},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.88100000000000001,"15min":0.85899999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.44700000000000001},"max":{"1min":1.2110000000000001,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.96899999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.89900000000000002,"15min":0.88400000000000001},"min":{"1min":0.58099999999999996,"5min":0.53700000000000003,"15min":0.48299999999999998},"max":{"1min":1.508,"5min":2.1549999999999998,"15min":9.0099999999999998},"last":1.476},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.93300000000000005,"15min":0.94199999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.0939999999999999,"5min":7.7359999999999998,"15min":7.7359999999999998},"last":0.69599999999999995}]},{"osd":3,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.93799999999999994,"15min":0.94699999999999995},"min":{"1min":0.63700000000000001,"5min":0.56200000000000006,"15min":0.52500000000000002},"max":{"1min":1.248,"5min":1.6000000000000001,"15min":4.7080000000000002},"last":1.107},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.034,"15min":1.018},"min":{"1min":0.67900000000000005,"5min":0.66700000000000004,"15min":0.48699999999999999},"max":{"1min":1.153,"5min":1.7509999999999999,"15min":3.2749999999999999},"last":1.1679999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.0029999999999999,"15min":1.0409999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.29,"5min":1.3799999999999999,"15min":7.8529999999999998},"last":1.272},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.212,"15min":1.1759999999999999},"min":{"1min":0.68200000000000005,"5min":0.65400000000000003,"15min":0.496},"max":{"1min":1.4350000000000001,"5min":7.694,"15min":8.2070000000000007},"last":1.407}]},{"osd":6,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.125,"15min":1.083},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.52600000000000002},"max":{"1min":1.4510000000000001,"5min":7.7880000000000003,"15min":7.7880000000000003},"last":1.544},{"interface":"front","average":{"1min":1.087,"5min":1.1459999999999999,"15min":1.151},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.395,"5min":3.7320000000000002,"15min":5.6130000000000004},"last":1.2310000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.194,"15min":1.1759999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.59299999999999997},"max":{"1min":1.5109999999999999,"5min":7.8380000000000001,"15min":10.715},"last":1.617},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.0369999999999999,"15min":1.147},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.47199999999999998},"max":{"1min":1.321,"5min":2.2450000000000001,"15min":4.3230000000000004},"last":0.81399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575366,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729636,"kb_used_data":550044,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":93642204,"statfs":{"total":96636764160,"available":95889616896,"internally_reserved":0,"allocated":563245056,"data_stored":549099638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":183892424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42699999999999999,"15min":0.40899999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.23000000000000001},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.45200000000000001,"15min":0.46800000000000003},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70899999999999996,"5min":0.76700000000000002,"15min":1.5429999999999999},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.504,"15min":0.46700000000000003},"min":{"1min":0.32400000000000001,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":0.65600000000000003,"5min":0.76200000000000001,"15min":0.94099999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.246},"max":{"1min":0.83099999999999996,"5min":0.92000000000000004,"15min":1.018},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.503,"15min":0.51000000000000001},"min":{"1min":0.35099999999999998,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.81599999999999995,"5min":1.056,"15min":1.4690000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86499999999999999,"15min":1.6799999999999999},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.78200000000000003,"15min":0.82499999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.97699999999999998,"5min":1.1739999999999999,"15min":1.5900000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83199999999999996,"15min":0.84599999999999997},"min":{"1min":0.56100000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1,"5min":1.1419999999999999,"15min":1.3999999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.76700000000000002,"15min":0.79600000000000004},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.51800000000000002},"max":{"1min":0.86299999999999999,"5min":1.1879999999999999,"15min":1.5069999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.81000000000000005,"15min":0.81899999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.5},"max":{"1min":0.95999999999999996,"5min":1.232,"15min":1.365},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.85199999999999998,"15min":0.82599999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45400000000000001},"max":{"1min":1.05,"5min":1.256,"15min":1.256},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.87,"15min":0.82899999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.498},"max":{"1min":1.0760000000000001,"5min":1.3069999999999999,"15min":2.3410000000000002},"last":0.90200000000000002}]},{"osd":7,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.86799999999999999,"15min":0.876},"min":{"1min":0.61599999999999999,"5min":0.57799999999999996,"15min":0.51700000000000002},"max":{"1min":1.0569999999999999,"5min":1.3320000000000001,"15min":2.9449999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.86099999999999999,"15min":0.88200000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":1.0780000000000001,"5min":1.343,"15min":2.9729999999999999},"last":0.78400000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575369,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484724,"kb_used_data":377260,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93887116,"statfs":{"total":96636764160,"available":96140406784,"internally_reserved":0,"allocated":386314240,"data_stored":377764906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.54400000000000004,"15min":0.47599999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":9.8249999999999993,"5min":9.8249999999999993,"15min":9.8249999999999993},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53600000000000003,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.32100000000000001,"15min":0.25700000000000001},"max":{"1min":2.7200000000000002,"5min":4.2910000000000004,"15min":4.2910000000000004},"last":0.318}]},{"osd":1,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.64800000000000002,"15min":0.58799999999999997},"min":{"1min":0.38700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":9.7420000000000009,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.435},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58699999999999997,"15min":0.54200000000000004},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.2839999999999998,"5min":7.0060000000000002,"15min":7.0060000000000002},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.52900000000000003,"15min":0.504},"min":{"1min":0.371,"5min":0.35099999999999998,"15min":0.28299999999999997},"max":{"1min":2.8119999999999998,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.64500000000000002,"15min":0.56999999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.33700000000000002},"max":{"1min":2.843,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.48999999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79800000000000004,"15min":0.80300000000000005},"min":{"1min":0.56200000000000006,"5min":0.442,"15min":0.442},"max":{"1min":2.4129999999999998,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.37},"max":{"1min":2.681,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.66600000000000004}]},{"osd":5,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.89300000000000002,"15min":0.88200000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.50900000000000001},"max":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7989999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.79500000000000004,"15min":0.82499999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"max":{"1min":2.4750000000000001,"5min":2.4750000000000001,"15min":2.762},"last":0.61499999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.84499999999999997,"15min":0.81599999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.47699999999999998},"max":{"1min":9.7210000000000001,"5min":9.7210000000000001,"15min":9.7210000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.2050000000000001,"5min":0.91200000000000003,"15min":0.84599999999999997},"min":{"1min":0.622,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":9.7889999999999997,"5min":9.7889999999999997,"15min":9.7889999999999997},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":0.97299999999999998,"15min":0.875},"min":{"1min":0.629,"5min":0.58999999999999997,"15min":0.503},"max":{"1min":6.8529999999999998,"5min":6.9059999999999997,"15min":6.9059999999999997},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.0589999999999999,"15min":0.97099999999999997},"min":{"1min":0.64000000000000001,"5min":0.63700000000000001,"15min":0.55900000000000005},"max":{"1min":6.8929999999999998,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.65600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575366,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687452,"kb_used_data":535636,"kb_used_omap":1,"kb_used_meta":151806,"kb_avail":93684388,"statfs":{"total":96636764160,"available":95932813312,"internally_reserved":0,"allocated":548491264,"data_stored":539410681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":155449798},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24299999999999999,"15min":0.22800000000000001},"max":{"1min":0.629,"5min":0.94999999999999996,"15min":1.1359999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.28799999999999998,"5min":0.28299999999999997,"15min":0.251},"max":{"1min":0.72599999999999998,"5min":0.84099999999999997,"15min":0.96699999999999997},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47999999999999998,"15min":0.499},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.627,"5min":0.78000000000000003,"15min":2.8660000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.32800000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67900000000000005,"5min":1.0589999999999999,"15min":2.6739999999999999},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.314},"max":{"1min":0.70599999999999996,"5min":0.76900000000000002,"15min":1.2569999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57399999999999995},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":0.749,"5min":0.81399999999999995,"15min":2.71},"last":0.66300000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.81699999999999995,"15min":0.85199999999999998},"min":{"1min":0.52700000000000002,"5min":0.52400000000000002,"15min":0.40200000000000002},"max":{"1min":1.0389999999999999,"5min":1.2210000000000001,"15min":6.7789999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.88800000000000001,"15min":0.92100000000000004},"min":{"1min":0.59299999999999997,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":1.0900000000000001,"5min":1.2629999999999999,"15min":6.8259999999999996},"last":0.88600000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.76400000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":2.133,"15min":2.5750000000000002},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.80700000000000005,"15min":0.80200000000000005},"min":{"1min":0.621,"5min":0.53800000000000003,"15min":0.51000000000000001},"max":{"1min":1.462,"5min":1.462,"15min":2.6179999999999999},"last":0.84399999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:52:46 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.91500000000000004,"15min":0.89700000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.501},"max":{"1min":1.357,"5min":1.3879999999999999,"15min":6.0289999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.95899999999999996,"15min":0.95399999999999996},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":6.0739999999999998},"last":0.95499999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.80800000000000005,"15min":0.78100000000000003},"min":{"1min":0.70299999999999996,"5min":0.55000000000000004,"15min":0.47899999999999998},"max":{"1min":1.1319999999999999,"5min":1.161,"15min":1.163},"last":0.746},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.86699999999999999,"15min":0.877},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.55200000000000005},"max":{"1min":1.153,"5min":1.3440000000000001,"15min":6.0519999999999996},"last":0.91100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575368,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708336,"kb_used_data":568936,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93663504,"statfs":{"total":96636764160,"available":95911428096,"internally_reserved":0,"allocated":582590464,"data_stored":569154632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142735814},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.40100000000000002,"15min":0.47599999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":0.54400000000000004,"5min":2.0630000000000002,"15min":16.184000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.50900000000000001,"15min":0.59099999999999997},"min":{"1min":0.36199999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.629,"5min":2.0979999999999999,"15min":16.228000000000002},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45300000000000001,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.70499999999999996,"5min":2.7999999999999998,"15min":17.454000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.46800000000000003,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":0.61799999999999999,"5min":1.391,"15min":23.212},"last":0.27000000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51100000000000001,"15min":0.62},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":1.4690000000000001,"15min":16.103999999999999},"last":0.315},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52700000000000002,"15min":0.72399999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60199999999999998,"5min":2.8340000000000001,"15min":23.16},"last":0.41599999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:52:53 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.81899999999999995,"15min":0.91800000000000004},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":0.96399999999999997,"5min":7.6900000000000004,"15min":16.309000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.82099999999999995,"15min":0.95999999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97399999999999998,"5min":3.4009999999999998,"15min":16.343},"last":0.78100000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.81599999999999995,"15min":0.995},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94299999999999995,"5min":2.02,"15min":23.266999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.86599999999999999,"15min":1.0329999999999999},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.999,"5min":2.7240000000000002,"15min":16.268000000000001},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.79400000000000004,"15min":1.091},"min":{"1min":0.52500000000000002,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":1.0269999999999999,"5min":2.6560000000000001,"15min":29.774999999999999},"last":0.755},{"interface":"front","average":{"1min":0.752,"5min":0.80600000000000005,"15min":1.0149999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":2.75,"15min":17.5},"last":0.629}]},{"osd":7,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.81699999999999995,"15min":1.1919999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.51300000000000001},"max":{"1min":1.0229999999999999,"5min":2.694,"15min":47.534999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":1.147},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.54700000000000004},"max":{"1min":1.272,"5min":2.8690000000000002,"15min":32.476999999999997},"last":0.82599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230703104,"data_stored":230687380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258846720,"data_stored":258822445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179036160,"data_stored":179013521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230858752,"data_stored":230840931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":197275648,"data_stored":197263919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":205541376,"data_stored":205522290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":268918784,"data_stored":268902222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":247238656,"data_stored":247218354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350515200,"data_stored":337464605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288862208,"data_stored":280174938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":206495744,"data_stored":198338087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331603968,"data_stored":317845409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":385015808,"data_stored":371192067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":325414912,"data_stored":312650097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193724416,"data_stored":185512357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342437888,"data_stored":329203526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:53:18.135 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.135 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.136 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.136 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.136 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.137 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.137 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.138 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.138 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.138 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.139 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.139 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.140 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.140 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.141 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.141 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.142 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.142 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.142 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.143 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.143 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T13:10:32.661389+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.144 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.144 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:18.145 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:53:35.227 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:35.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:53:38.146 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:53:38.293 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c16db8640 1 Processor -- start 2023-10-30T13:53:38.294 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c16db8640 1 -- start start 2023-10-30T13:53:38.295 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c16db8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c10157010 0x7f2c100715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c16db8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c100c8460 0x7f2c10070650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c16db8640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2c10159810 0x7f2c10071bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.296 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c16db8640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c100c21b0 con 0x7f2c10159810 2023-10-30T13:53:38.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c16db8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c100c1530 con 0x7f2c10157010 2023-10-30T13:53:38.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c16db8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c100c0f70 con 0x7f2c100c8460 2023-10-30T13:53:38.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c15355640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2c10159810 0x7f2c10071bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.297 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c15355640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2c10159810 0x7f2c10071bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:48946/0 (socket says 172.21.15.148:48946) 2023-10-30T13:53:38.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c14b54640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c10157010 0x7f2c100715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c15355640 1 -- 172.21.15.148:0/397838062 learned_addr learned my addr 172.21.15.148:0/397838062 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:53:38.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c07fff640 1 --2- 172.21.15.148:0/397838062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c100c8460 0x7f2c10070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.298 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c15355640 1 -- 172.21.15.148:0/397838062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c100c8460 msgr2=0x7f2c10070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:38.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c15355640 1 --2- 172.21.15.148:0/397838062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c100c8460 0x7f2c10070650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c15355640 1 -- 172.21.15.148:0/397838062 >> v2:172.21.15.150:3300/0 conn(0x7f2c10157010 msgr2=0x7f2c100715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:38.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c15355640 1 --2- 172.21.15.148:0/397838062 >> v2:172.21.15.150:3300/0 conn(0x7f2c10157010 0x7f2c100715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.299 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.292+0000 7f2c15355640 1 -- 172.21.15.148:0/397838062 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c100c1250 con 0x7f2c10159810 2023-10-30T13:53:38.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c15355640 1 --2- 172.21.15.148:0/397838062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 0x7f2c10071bd0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f2c1006d550 tx=0x7f2c0801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48716be97b02fee7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:38.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c077fe640 1 -- 172.21.15.148:0/397838062 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c080af400 con 0x7f2c10159810 2023-10-30T13:53:38.301 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c077fe640 1 -- 172.21.15.148:0/397838062 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c080ab050 con 0x7f2c10159810 2023-10-30T13:53:38.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c077fe640 1 -- 172.21.15.148:0/397838062 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c080af780 con 0x7f2c10159810 2023-10-30T13:53:38.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c16db8640 1 -- 172.21.15.148:0/397838062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 msgr2=0x7f2c10071bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.302 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c16db8640 1 --2- 172.21.15.148:0/397838062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 0x7f2c10071bd0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f2c1006d550 tx=0x7f2c0801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c16db8640 1 -- 172.21.15.148:0/397838062 shutdown_connections 2023-10-30T13:53:38.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c16db8640 1 --2- 172.21.15.148:0/397838062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 0x7f2c10071bd0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c16db8640 1 --2- 172.21.15.148:0/397838062 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c100c8460 0x7f2c10070650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.303 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c16db8640 1 --2- 172.21.15.148:0/397838062 >> v2:172.21.15.150:3300/0 conn(0x7f2c10157010 0x7f2c100715c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c16db8640 1 -- 172.21.15.148:0/397838062 >> 172.21.15.148:0/397838062 conn(0x7f2c100cbeb0 msgr2=0x7f2c100cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:38.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c16db8640 1 -- 172.21.15.148:0/397838062 shutdown_connections 2023-10-30T13:53:38.304 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.296+0000 7f2c16db8640 1 -- 172.21.15.148:0/397838062 wait complete. 2023-10-30T13:53:38.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 Processor -- start 2023-10-30T13:53:38.305 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 -- start start 2023-10-30T13:53:38.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c100c8460 0x7f2c100cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.307 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c10157010 0x7f2c101f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 0x7f2c1006b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c101f1e80 con 0x7f2c10159810 2023-10-30T13:53:38.308 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1006aed0 con 0x7f2c100c8460 2023-10-30T13:53:38.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c10069e90 con 0x7f2c10157010 2023-10-30T13:53:38.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c14b54640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c100c8460 0x7f2c100cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c14b54640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c100c8460 0x7f2c100cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.148:56974/0 (socket says 172.21.15.148:56974) 2023-10-30T13:53:38.309 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c14b54640 1 -- 172.21.15.148:0/2664879024 learned_addr learned my addr 172.21.15.148:0/2664879024 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:53:38.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c15355640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 0x7f2c1006b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c07fff640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c10157010 0x7f2c101f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c15355640 1 -- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c10157010 msgr2=0x7f2c101f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.310 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c15355640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c10157010 0x7f2c101f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c15355640 1 -- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c100c8460 msgr2=0x7f2c100cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c15355640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c100c8460 0x7f2c100cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c15355640 1 -- 172.21.15.148:0/2664879024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c08009b30 con 0x7f2c10159810 2023-10-30T13:53:38.311 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c15355640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 0x7f2c1006b620 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f2c1006c590 tx=0x7f2c0801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:38.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c057fa640 1 -- 172.21.15.148:0/2664879024 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c080c0070 con 0x7f2c10159810 2023-10-30T13:53:38.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c057fa640 1 -- 172.21.15.148:0/2664879024 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c08009510 con 0x7f2c10159810 2023-10-30T13:53:38.312 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1006a130 con 0x7f2c10159810 2023-10-30T13:53:38.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c057fa640 1 -- 172.21.15.148:0/2664879024 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c080afd20 con 0x7f2c10159810 2023-10-30T13:53:38.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1006a6f0 con 0x7f2c10159810 2023-10-30T13:53:38.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.300+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1004e170 con 0x7f2c10159810 2023-10-30T13:53:38.313 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.304+0000 7f2c057fa640 1 -- 172.21.15.148:0/2664879024 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c080b6040 con 0x7f2c10159810 2023-10-30T13:53:38.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.304+0000 7f2c057fa640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2bfc0664a0 0x7f2bfc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:38.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.304+0000 7f2c057fa640 1 -- 172.21.15.148:0/2664879024 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f2c08121ac0 con 0x7f2c10159810 2023-10-30T13:53:38.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.304+0000 7f2c14b54640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2bfc0664a0 0x7f2bfc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:38.314 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.304+0000 7f2c14b54640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2bfc0664a0 0x7f2bfc068960 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009c20 tx=0x7f2c0c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:38.315 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.308+0000 7f2c057fa640 1 -- 172.21.15.148:0/2664879024 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c080ee870 con 0x7f2c10159810 2023-10-30T13:53:38.461 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.460+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c100710a0 con 0x7f2bfc0664a0 2023-10-30T13:53:38.470 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.468+0000 7f2c057fa640 1 -- 172.21.15.148:0/2664879024 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+296033 (secure 0 0 0) 0x7f2c0c009950 con 0x7f2bfc0664a0 2023-10-30T13:53:38.473 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:53:38.475 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:53:38.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2bfc0664a0 msgr2=0x7f2bfc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2bfc0664a0 0x7f2bfc068960 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009c20 tx=0x7f2c0c009190 comp rx=0 tx=0).stop 2023-10-30T13:53:38.476 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 msgr2=0x7f2c1006b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:38.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 0x7f2c1006b620 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f2c1006c590 tx=0x7f2c0801bf60 comp rx=0 tx=0).stop 2023-10-30T13:53:38.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 shutdown_connections 2023-10-30T13:53:38.477 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f2bfc0664a0 0x7f2bfc068960 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c10159810 0x7f2c1006b620 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f2c10157010 0x7f2c101f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 --2- 172.21.15.148:0/2664879024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c100c8460 0x7f2c100cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:38.478 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 >> 172.21.15.148:0/2664879024 conn(0x7f2c100cbeb0 msgr2=0x7f2c10151cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:38.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 shutdown_connections 2023-10-30T13:53:38.479 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:38.472+0000 7f2c16db8640 1 -- 172.21.15.148:0/2664879024 wait complete. 2023-10-30T13:53:38.493 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1357,"stamp":"2023-10-30T13:53:37.225143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2074908826,"num_objects":41005,"num_object_clones":18317,"num_object_copies":82010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41005,"num_whiteouts":14350,"num_read":18168,"num_read_kb":6131546,"num_write":117782,"num_write_kb":3536715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1411,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17841559,"num_omap_keys":30407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167884,"ondisk_log_size":167884,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5391732,"kb_used_data":4150324,"kb_used_omap":12,"kb_used_meta":1241331,"kb_avail":749582988,"statfs":{"total":773094113280,"available":767572979712,"internally_reserved":0,"allocated":4249931776,"data_stored":4155138249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1271123512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"3.3f","version":"67'1473","reported_seq":3441,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.519050+0000","last_change":"2023-10-30T13:21:40.584822+0000","last_active":"2023-10-30T13:52:29.519050+0000","last_peered":"2023-10-30T13:52:29.519050+0000","last_clean":"2023-10-30T13:52:29.519050+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:52:29.519050+0000","last_undegraded":"2023-10-30T13:52:29.519050+0000","last_fullsized":"2023-10-30T13:52:29.519050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014806453000000001,"stat_sum":{"num_bytes":11269044,"num_objects":569,"num_object_clones":260,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":205,"num_read":107,"num_read_kb":121,"num_write":1253,"num_write_kb":17103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'2076","reported_seq":4032,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:18.542413+0000","last_change":"2023-10-30T13:53:18.542467+0000","last_active":"2023-10-30T13:53:18.542413+0000","last_peered":"2023-10-30T13:53:18.542413+0000","last_clean":"2023-10-30T13:53:18.542413+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:53:18.542413+0000","last_undegraded":"2023-10-30T13:53:18.542413+0000","last_fullsized":"2023-10-30T13:53:18.542413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2076","last_scrub_stamp":"2023-10-30T13:53:18.542317+0000","last_deep_scrub":"67'2076","last_deep_scrub_stamp":"2023-10-30T13:53:18.542317+0000","last_clean_scrub_stamp":"2023-10-30T13:53:18.542317+0000","objects_scrubbed":597,"log_size":2076,"log_dups_size":0,"ondisk_log_size":2076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:28:37.756101+0000","scrub_duration":0.79951518899999996,"objects_trimmed":0,"snaptrim_duration":0.0059691099999999997,"stat_sum":{"num_bytes":30975930,"num_objects":597,"num_object_clones":279,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":225,"num_read":82,"num_read_kb":30,"num_write":1226,"num_write_kb":32475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'2143","reported_seq":4123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.190609+0000","last_change":"2023-10-30T13:52:54.190676+0000","last_active":"2023-10-30T13:52:54.190609+0000","last_peered":"2023-10-30T13:52:54.190609+0000","last_clean":"2023-10-30T13:52:54.190609+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:52:54.190609+0000","last_undegraded":"2023-10-30T13:52:54.190609+0000","last_fullsized":"2023-10-30T13:52:54.190609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2143","last_scrub_stamp":"2023-10-30T13:52:54.190532+0000","last_deep_scrub":"67'2143","last_deep_scrub_stamp":"2023-10-30T13:52:54.190532+0000","last_clean_scrub_stamp":"2023-10-30T13:52:54.190532+0000","objects_scrubbed":617,"log_size":2143,"log_dups_size":0,"ondisk_log_size":2143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:27.035524+0000","scrub_duration":0.94181986699999998,"objects_trimmed":0,"snaptrim_duration":0.0025608710000000002,"stat_sum":{"num_bytes":18704388,"num_objects":617,"num_object_clones":279,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":225,"num_read":106,"num_read_kb":8300,"num_write":1300,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'2214","reported_seq":4185,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.682002+0000","last_change":"2023-10-30T13:52:59.682002+0000","last_active":"2023-10-30T13:52:59.682002+0000","last_peered":"2023-10-30T13:52:59.682002+0000","last_clean":"2023-10-30T13:52:59.682002+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:52:59.682002+0000","last_undegraded":"2023-10-30T13:52:59.682002+0000","last_fullsized":"2023-10-30T13:52:59.682002+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2214","last_scrub_stamp":"2023-10-30T13:52:59.681815+0000","last_deep_scrub":"67'2214","last_deep_scrub_stamp":"2023-10-30T13:52:59.681815+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.681815+0000","objects_scrubbed":630,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:49.252489+0000","scrub_duration":1.2152727269999999,"objects_trimmed":0,"snaptrim_duration":0.0021564890000000001,"stat_sum":{"num_bytes":23329933,"num_objects":630,"num_object_clones":293,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":227,"num_read":112,"num_read_kb":4378,"num_write":1354,"num_write_kb":25006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'2231","reported_seq":4187,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:38.545335+0000","last_change":"2023-10-30T13:52:38.545426+0000","last_active":"2023-10-30T13:52:38.545335+0000","last_peered":"2023-10-30T13:52:38.545335+0000","last_clean":"2023-10-30T13:52:38.545335+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:52:38.545335+0000","last_undegraded":"2023-10-30T13:52:38.545335+0000","last_fullsized":"2023-10-30T13:52:38.545335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2231","last_scrub_stamp":"2023-10-30T13:52:38.545206+0000","last_deep_scrub":"67'2231","last_deep_scrub_stamp":"2023-10-30T13:52:38.545206+0000","last_clean_scrub_stamp":"2023-10-30T13:52:38.545206+0000","objects_scrubbed":654,"log_size":2231,"log_dups_size":0,"ondisk_log_size":2231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:21:20.355710+0000","scrub_duration":1.430729712,"objects_trimmed":0,"snaptrim_duration":0.011808077,"stat_sum":{"num_bytes":16184828,"num_objects":654,"num_object_clones":305,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":251,"num_read":109,"num_read_kb":152,"num_write":1339,"num_write_kb":20879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'2052","reported_seq":4033,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:56.370945+0000","last_change":"2023-10-30T13:52:56.370945+0000","last_active":"2023-10-30T13:52:56.370945+0000","last_peered":"2023-10-30T13:52:56.370945+0000","last_clean":"2023-10-30T13:52:56.370945+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:52:56.370945+0000","last_undegraded":"2023-10-30T13:52:56.370945+0000","last_fullsized":"2023-10-30T13:52:56.370945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2052","last_scrub_stamp":"2023-10-30T13:52:56.370794+0000","last_deep_scrub":"67'2052","last_deep_scrub_stamp":"2023-10-30T13:52:56.370794+0000","last_clean_scrub_stamp":"2023-10-30T13:52:56.370794+0000","objects_scrubbed":593,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:21.000907+0000","scrub_duration":1.065003666,"objects_trimmed":0,"snaptrim_duration":0.01102786,"stat_sum":{"num_bytes":18372801,"num_objects":593,"num_object_clones":277,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":222,"num_read":108,"num_read_kb":4137,"num_write":1217,"num_write_kb":19358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'2113","reported_seq":4084,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:24.134174+0000","last_change":"2023-10-30T13:53:24.134280+0000","last_active":"2023-10-30T13:53:24.134174+0000","last_peered":"2023-10-30T13:53:24.134174+0000","last_clean":"2023-10-30T13:53:24.134174+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:53:24.134174+0000","last_undegraded":"2023-10-30T13:53:24.134174+0000","last_fullsized":"2023-10-30T13:53:24.134174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2113","last_scrub_stamp":"2023-10-30T13:53:24.134014+0000","last_deep_scrub":"67'2113","last_deep_scrub_stamp":"2023-10-30T13:53:24.134014+0000","last_clean_scrub_stamp":"2023-10-30T13:53:24.134014+0000","objects_scrubbed":598,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:24:00.966228+0000","scrub_duration":1.0382753499999999,"objects_trimmed":0,"snaptrim_duration":0.012047337,"stat_sum":{"num_bytes":9859762,"num_objects":598,"num_object_clones":275,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":211,"num_read":96,"num_read_kb":35,"num_write":1295,"num_write_kb":11866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'2037","reported_seq":4026,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.982168+0000","last_change":"2023-10-30T13:52:51.982168+0000","last_active":"2023-10-30T13:52:51.982168+0000","last_peered":"2023-10-30T13:52:51.982168+0000","last_clean":"2023-10-30T13:52:51.982168+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:52:51.982168+0000","last_undegraded":"2023-10-30T13:52:51.982168+0000","last_fullsized":"2023-10-30T13:52:51.982168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2037","last_scrub_stamp":"2023-10-30T13:52:51.982035+0000","last_deep_scrub":"67'2037","last_deep_scrub_stamp":"2023-10-30T13:52:51.982035+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.982035+0000","objects_scrubbed":577,"log_size":2037,"log_dups_size":0,"ondisk_log_size":2037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:47.358388+0000","scrub_duration":1.116769473,"objects_trimmed":0,"snaptrim_duration":0.0063254060000000004,"stat_sum":{"num_bytes":13694423,"num_objects":577,"num_object_clones":268,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":211,"num_read":108,"num_read_kb":104,"num_write":1228,"num_write_kb":23250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'1528","reported_seq":3471,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:29.522374+0000","last_change":"2023-10-30T13:21:40.584916+0000","last_active":"2023-10-30T13:52:29.522374+0000","last_peered":"2023-10-30T13:52:29.522374+0000","last_clean":"2023-10-30T13:52:29.522374+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:52:29.522374+0000","last_undegraded":"2023-10-30T13:52:29.522374+0000","last_fullsized":"2023-10-30T13:52:29.522374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014360304000000001,"stat_sum":{"num_bytes":19134308,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":234,"num_read":109,"num_read_kb":4185,"num_write":1275,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'2111","reported_seq":4082,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:24.064933+0000","last_change":"2023-10-30T13:53:24.065003+0000","last_active":"2023-10-30T13:53:24.064933+0000","last_peered":"2023-10-30T13:53:24.064933+0000","last_clean":"2023-10-30T13:53:24.064933+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:53:24.064933+0000","last_undegraded":"2023-10-30T13:53:24.064933+0000","last_fullsized":"2023-10-30T13:53:24.064933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2111","last_scrub_stamp":"2023-10-30T13:53:24.064851+0000","last_deep_scrub":"67'2111","last_deep_scrub_stamp":"2023-10-30T13:53:24.064851+0000","last_clean_scrub_stamp":"2023-10-30T13:53:24.064851+0000","objects_scrubbed":610,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:12.208328+0000","scrub_duration":1.1952141599999999,"objects_trimmed":0,"snaptrim_duration":0.010277041000000001,"stat_sum":{"num_bytes":19693407,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":208,"num_read":105,"num_read_kb":2976,"num_write":1294,"num_write_kb":29513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'2172","reported_seq":4133,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:19.900333+0000","last_change":"2023-10-30T13:53:19.900470+0000","last_active":"2023-10-30T13:53:19.900333+0000","last_peered":"2023-10-30T13:53:19.900333+0000","last_clean":"2023-10-30T13:53:19.900333+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:53:19.900333+0000","last_undegraded":"2023-10-30T13:53:19.900333+0000","last_fullsized":"2023-10-30T13:53:19.900333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2172","last_scrub_stamp":"2023-10-30T13:53:19.900196+0000","last_deep_scrub":"67'2172","last_deep_scrub_stamp":"2023-10-30T13:53:19.900196+0000","last_clean_scrub_stamp":"2023-10-30T13:53:19.900196+0000","objects_scrubbed":636,"log_size":2172,"log_dups_size":0,"ondisk_log_size":2172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:36.711893+0000","scrub_duration":0.95872091000000004,"objects_trimmed":0,"snaptrim_duration":0.01048763,"stat_sum":{"num_bytes":11727836,"num_objects":636,"num_object_clones":292,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":230,"num_read":95,"num_read_kb":27,"num_write":1294,"num_write_kb":13174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'2168","reported_seq":4127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:55.137033+0000","last_change":"2023-10-30T13:52:55.137145+0000","last_active":"2023-10-30T13:52:55.137033+0000","last_peered":"2023-10-30T13:52:55.137033+0000","last_clean":"2023-10-30T13:52:55.137033+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:52:55.137033+0000","last_undegraded":"2023-10-30T13:52:55.137033+0000","last_fullsized":"2023-10-30T13:52:55.137033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2168","last_scrub_stamp":"2023-10-30T13:52:55.136932+0000","last_deep_scrub":"67'2168","last_deep_scrub_stamp":"2023-10-30T13:52:55.136932+0000","last_clean_scrub_stamp":"2023-10-30T13:52:55.136932+0000","objects_scrubbed":634,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:10.926178+0000","scrub_duration":1.2701765250000001,"objects_trimmed":0,"snaptrim_duration":0.0085295030000000008,"stat_sum":{"num_bytes":15445695,"num_objects":634,"num_object_clones":293,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":231,"num_read":101,"num_read_kb":102,"num_write":1284,"num_write_kb":21892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'2081","reported_seq":4056,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:11.778193+0000","last_change":"2023-10-30T13:53:11.778257+0000","last_active":"2023-10-30T13:53:11.778193+0000","last_peered":"2023-10-30T13:53:11.778193+0000","last_clean":"2023-10-30T13:53:11.778193+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:53:11.778193+0000","last_undegraded":"2023-10-30T13:53:11.778193+0000","last_fullsized":"2023-10-30T13:53:11.778193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:53:11.778101+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:53:11.778101+0000","last_clean_scrub_stamp":"2023-10-30T13:53:11.778101+0000","objects_scrubbed":596,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:38.043584+0000","scrub_duration":0.84397403100000001,"objects_trimmed":0,"snaptrim_duration":0.004933977,"stat_sum":{"num_bytes":6328355,"num_objects":596,"num_object_clones":272,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":210,"num_read":97,"num_read_kb":4194,"num_write":1284,"num_write_kb":15784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'2163","reported_seq":4144,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:46.206549+0000","last_change":"2023-10-30T13:52:46.206609+0000","last_active":"2023-10-30T13:52:46.206549+0000","last_peered":"2023-10-30T13:52:46.206549+0000","last_clean":"2023-10-30T13:52:46.206549+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:52:46.206549+0000","last_undegraded":"2023-10-30T13:52:46.206549+0000","last_fullsized":"2023-10-30T13:52:46.206549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2163","last_scrub_stamp":"2023-10-30T13:52:46.206474+0000","last_deep_scrub":"67'2163","last_deep_scrub_stamp":"2023-10-30T13:52:46.206474+0000","last_clean_scrub_stamp":"2023-10-30T13:52:46.206474+0000","objects_scrubbed":610,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:31.417387+0000","scrub_duration":1.043425775,"objects_trimmed":0,"snaptrim_duration":0.0029909680000000001,"stat_sum":{"num_bytes":31670070,"num_objects":610,"num_object_clones":280,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":227,"num_read":110,"num_read_kb":16437,"num_write":1314,"num_write_kb":33491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'2183","reported_seq":4156,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:05.806848+0000","last_change":"2023-10-30T13:53:05.806946+0000","last_active":"2023-10-30T13:53:05.806848+0000","last_peered":"2023-10-30T13:53:05.806848+0000","last_clean":"2023-10-30T13:53:05.806848+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:53:05.806848+0000","last_undegraded":"2023-10-30T13:53:05.806848+0000","last_fullsized":"2023-10-30T13:53:05.806848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2183","last_scrub_stamp":"2023-10-30T13:53:05.806777+0000","last_deep_scrub":"67'2183","last_deep_scrub_stamp":"2023-10-30T13:53:05.806777+0000","last_clean_scrub_stamp":"2023-10-30T13:53:05.806777+0000","objects_scrubbed":613,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:17.084832+0000","scrub_duration":1.1056300050000001,"objects_trimmed":0,"snaptrim_duration":0.0058939090000000001,"stat_sum":{"num_bytes":14342306,"num_objects":613,"num_object_clones":284,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":231,"num_read":106,"num_read_kb":2953,"num_write":1333,"num_write_kb":15925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'2459","reported_seq":4355,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:46.948880+0000","last_change":"2023-10-30T13:52:46.948942+0000","last_active":"2023-10-30T13:52:46.948880+0000","last_peered":"2023-10-30T13:52:46.948880+0000","last_clean":"2023-10-30T13:52:46.948880+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:52:46.948880+0000","last_undegraded":"2023-10-30T13:52:46.948880+0000","last_fullsized":"2023-10-30T13:52:46.948880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2459","last_scrub_stamp":"2023-10-30T13:52:46.948803+0000","last_deep_scrub":"67'2459","last_deep_scrub_stamp":"2023-10-30T13:52:46.948803+0000","last_clean_scrub_stamp":"2023-10-30T13:52:46.948803+0000","objects_scrubbed":703,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:36:20.176163+0000","scrub_duration":1.765876443,"objects_trimmed":0,"snaptrim_duration":0.0093243890000000006,"stat_sum":{"num_bytes":10965780,"num_objects":703,"num_object_clones":374,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":231,"num_read":115,"num_read_kb":1694,"num_write":1434,"num_write_kb":17364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'2223","reported_seq":4184,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.397041+0000","last_change":"2023-10-30T13:52:51.397109+0000","last_active":"2023-10-30T13:52:51.397041+0000","last_peered":"2023-10-30T13:52:51.397041+0000","last_clean":"2023-10-30T13:52:51.397041+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:52:51.397041+0000","last_undegraded":"2023-10-30T13:52:51.397041+0000","last_fullsized":"2023-10-30T13:52:51.397041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2223","last_scrub_stamp":"2023-10-30T13:52:51.396947+0000","last_deep_scrub":"67'2223","last_deep_scrub_stamp":"2023-10-30T13:52:51.396947+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.396947+0000","objects_scrubbed":632,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:49.530995+0000","scrub_duration":1.1860778729999999,"objects_trimmed":0,"snaptrim_duration":0.011284880000000001,"stat_sum":{"num_bytes":18442682,"num_objects":632,"num_object_clones":291,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":228,"num_read":100,"num_read_kb":8212,"num_write":1340,"num_write_kb":24815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'2051","reported_seq":4022,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:58.672924+0000","last_change":"2023-10-30T13:52:58.673009+0000","last_active":"2023-10-30T13:52:58.672924+0000","last_peered":"2023-10-30T13:52:58.672924+0000","last_clean":"2023-10-30T13:52:58.672924+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:52:58.672924+0000","last_undegraded":"2023-10-30T13:52:58.672924+0000","last_fullsized":"2023-10-30T13:52:58.672924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2051","last_scrub_stamp":"2023-10-30T13:52:58.672817+0000","last_deep_scrub":"67'2051","last_deep_scrub_stamp":"2023-10-30T13:52:58.672817+0000","last_clean_scrub_stamp":"2023-10-30T13:52:58.672817+0000","objects_scrubbed":575,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:13:12.046899+0000","scrub_duration":1.1758492060000001,"objects_trimmed":0,"snaptrim_duration":0.002295727,"stat_sum":{"num_bytes":22547199,"num_objects":575,"num_object_clones":267,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":207,"num_read":89,"num_read_kb":4127,"num_write":1240,"num_write_kb":24211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'2279","reported_seq":4233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:22.456694+0000","last_change":"2023-10-30T13:53:22.456770+0000","last_active":"2023-10-30T13:53:22.456694+0000","last_peered":"2023-10-30T13:53:22.456694+0000","last_clean":"2023-10-30T13:53:22.456694+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:53:22.456694+0000","last_undegraded":"2023-10-30T13:53:22.456694+0000","last_fullsized":"2023-10-30T13:53:22.456694+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2279","last_scrub_stamp":"2023-10-30T13:53:22.456585+0000","last_deep_scrub":"67'2279","last_deep_scrub_stamp":"2023-10-30T13:53:22.456585+0000","last_clean_scrub_stamp":"2023-10-30T13:53:22.456585+0000","objects_scrubbed":649,"log_size":2279,"log_dups_size":0,"ondisk_log_size":2279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:13.665995+0000","scrub_duration":0.75823801199999996,"objects_trimmed":0,"snaptrim_duration":0.0061689539999999999,"stat_sum":{"num_bytes":11703479,"num_objects":649,"num_object_clones":303,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":237,"num_read":106,"num_read_kb":7307,"num_write":1383,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"67'2189","reported_seq":4162,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:04.864418+0000","last_change":"2023-10-30T13:53:04.864418+0000","last_active":"2023-10-30T13:53:04.864418+0000","last_peered":"2023-10-30T13:53:04.864418+0000","last_clean":"2023-10-30T13:53:04.864418+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:53:04.864418+0000","last_undegraded":"2023-10-30T13:53:04.864418+0000","last_fullsized":"2023-10-30T13:53:04.864418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2189","last_scrub_stamp":"2023-10-30T13:53:04.864251+0000","last_deep_scrub":"67'2189","last_deep_scrub_stamp":"2023-10-30T13:53:04.864251+0000","last_clean_scrub_stamp":"2023-10-30T13:53:04.864251+0000","objects_scrubbed":621,"log_size":2189,"log_dups_size":0,"ondisk_log_size":2189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:46.599510+0000","scrub_duration":1.113390213,"objects_trimmed":0,"snaptrim_duration":0.0094394720000000008,"stat_sum":{"num_bytes":31771338,"num_objects":621,"num_object_clones":283,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":229,"num_read":103,"num_read_kb":4126,"num_write":1340,"num_write_kb":42793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'2081","reported_seq":4058,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.440653+0000","last_change":"2023-10-30T13:52:49.440653+0000","last_active":"2023-10-30T13:52:49.440653+0000","last_peered":"2023-10-30T13:52:49.440653+0000","last_clean":"2023-10-30T13:52:49.440653+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:52:49.440653+0000","last_undegraded":"2023-10-30T13:52:49.440653+0000","last_fullsized":"2023-10-30T13:52:49.440653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:52:49.440530+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:52:49.440530+0000","last_clean_scrub_stamp":"2023-10-30T13:52:49.440530+0000","objects_scrubbed":595,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:04.396223+0000","scrub_duration":1.3029579039999999,"objects_trimmed":0,"snaptrim_duration":0.0027436349999999999,"stat_sum":{"num_bytes":6210074,"num_objects":595,"num_object_clones":273,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":228,"num_read":98,"num_read_kb":33,"num_write":1261,"num_write_kb":10653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'2305","reported_seq":4251,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.423196+0000","last_change":"2023-10-30T13:52:45.423196+0000","last_active":"2023-10-30T13:52:45.423196+0000","last_peered":"2023-10-30T13:52:45.423196+0000","last_clean":"2023-10-30T13:52:45.423196+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:52:45.423196+0000","last_undegraded":"2023-10-30T13:52:45.423196+0000","last_fullsized":"2023-10-30T13:52:45.423196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2305","last_scrub_stamp":"2023-10-30T13:52:45.423033+0000","last_deep_scrub":"67'2305","last_deep_scrub_stamp":"2023-10-30T13:52:45.423033+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.423033+0000","objects_scrubbed":670,"log_size":2305,"log_dups_size":0,"ondisk_log_size":2305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:15.141666+0000","scrub_duration":1.276721341,"objects_trimmed":0,"snaptrim_duration":0.0082328149999999992,"stat_sum":{"num_bytes":28398973,"num_objects":670,"num_object_clones":311,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":248,"num_read":102,"num_read_kb":120,"num_write":1396,"num_write_kb":38562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'2154","reported_seq":4126,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:08.009726+0000","last_change":"2023-10-30T13:53:08.009776+0000","last_active":"2023-10-30T13:53:08.009726+0000","last_peered":"2023-10-30T13:53:08.009726+0000","last_clean":"2023-10-30T13:53:08.009726+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:53:08.009726+0000","last_undegraded":"2023-10-30T13:53:08.009726+0000","last_fullsized":"2023-10-30T13:53:08.009726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2154","last_scrub_stamp":"2023-10-30T13:53:08.009651+0000","last_deep_scrub":"67'2154","last_deep_scrub_stamp":"2023-10-30T13:53:08.009651+0000","last_clean_scrub_stamp":"2023-10-30T13:53:08.009651+0000","objects_scrubbed":610,"log_size":2154,"log_dups_size":0,"ondisk_log_size":2154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:41:21.317083+0000","scrub_duration":1.0714371359999999,"objects_trimmed":0,"snaptrim_duration":0.0037447470000000001,"stat_sum":{"num_bytes":19407388,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":225,"num_read":103,"num_read_kb":4138,"num_write":1300,"num_write_kb":24714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'2202","reported_seq":4177,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:08.361856+0000","last_change":"2023-10-30T13:53:08.361983+0000","last_active":"2023-10-30T13:53:08.361856+0000","last_peered":"2023-10-30T13:53:08.361856+0000","last_clean":"2023-10-30T13:53:08.361856+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:53:08.361856+0000","last_undegraded":"2023-10-30T13:53:08.361856+0000","last_fullsized":"2023-10-30T13:53:08.361856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2202","last_scrub_stamp":"2023-10-30T13:53:08.361660+0000","last_deep_scrub":"67'2202","last_deep_scrub_stamp":"2023-10-30T13:53:08.361660+0000","last_clean_scrub_stamp":"2023-10-30T13:53:08.361660+0000","objects_scrubbed":616,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:03.069890+0000","scrub_duration":1.1435550349999999,"objects_trimmed":0,"snaptrim_duration":0.0091633449999999998,"stat_sum":{"num_bytes":10552296,"num_objects":616,"num_object_clones":294,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":228,"num_read":115,"num_read_kb":24,"num_write":1342,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1977","reported_seq":3967,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:20.526287+0000","last_change":"2023-10-30T13:53:20.526336+0000","last_active":"2023-10-30T13:53:20.526287+0000","last_peered":"2023-10-30T13:53:20.526287+0000","last_clean":"2023-10-30T13:53:20.526287+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:53:20.526287+0000","last_undegraded":"2023-10-30T13:53:20.526287+0000","last_fullsized":"2023-10-30T13:53:20.526287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1977","last_scrub_stamp":"2023-10-30T13:53:20.526216+0000","last_deep_scrub":"67'1977","last_deep_scrub_stamp":"2023-10-30T13:53:20.526216+0000","last_clean_scrub_stamp":"2023-10-30T13:53:20.526216+0000","objects_scrubbed":556,"log_size":1977,"log_dups_size":0,"ondisk_log_size":1977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:49.058832+0000","scrub_duration":0.847919223,"objects_trimmed":0,"snaptrim_duration":0.0059368210000000001,"stat_sum":{"num_bytes":15196336,"num_objects":556,"num_object_clones":256,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":202,"num_read":99,"num_read_kb":1380,"num_write":1203,"num_write_kb":16993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'2204","reported_seq":4186,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:25.736195+0000","last_change":"2023-10-30T13:53:25.736280+0000","last_active":"2023-10-30T13:53:25.736195+0000","last_peered":"2023-10-30T13:53:25.736195+0000","last_clean":"2023-10-30T13:53:25.736195+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:53:25.736195+0000","last_undegraded":"2023-10-30T13:53:25.736195+0000","last_fullsized":"2023-10-30T13:53:25.736195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2204","last_scrub_stamp":"2023-10-30T13:53:25.736081+0000","last_deep_scrub":"67'2204","last_deep_scrub_stamp":"2023-10-30T13:53:25.736081+0000","last_clean_scrub_stamp":"2023-10-30T13:53:25.736081+0000","objects_scrubbed":619,"log_size":2204,"log_dups_size":0,"ondisk_log_size":2204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:14:55.998934+0000","scrub_duration":0.77972822399999997,"objects_trimmed":0,"snaptrim_duration":0.0099065669999999998,"stat_sum":{"num_bytes":9983325,"num_objects":619,"num_object_clones":286,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":221,"num_read":115,"num_read_kb":68,"num_write":1330,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'2157","reported_seq":4133,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:16.541747+0000","last_change":"2023-10-30T13:53:16.541808+0000","last_active":"2023-10-30T13:53:16.541747+0000","last_peered":"2023-10-30T13:53:16.541747+0000","last_clean":"2023-10-30T13:53:16.541747+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:53:16.541747+0000","last_undegraded":"2023-10-30T13:53:16.541747+0000","last_fullsized":"2023-10-30T13:53:16.541747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2157","last_scrub_stamp":"2023-10-30T13:53:16.541670+0000","last_deep_scrub":"67'2157","last_deep_scrub_stamp":"2023-10-30T13:53:16.541670+0000","last_clean_scrub_stamp":"2023-10-30T13:53:16.541670+0000","objects_scrubbed":614,"log_size":2157,"log_dups_size":0,"ondisk_log_size":2157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:07.326667+0000","scrub_duration":0.80191293600000002,"objects_trimmed":0,"snaptrim_duration":0.0059382619999999997,"stat_sum":{"num_bytes":23775876,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":236,"num_read":113,"num_read_kb":4145,"num_write":1301,"num_write_kb":26615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'2078","reported_seq":4040,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:03.067704+0000","last_change":"2023-10-30T13:53:03.067770+0000","last_active":"2023-10-30T13:53:03.067704+0000","last_peered":"2023-10-30T13:53:03.067704+0000","last_clean":"2023-10-30T13:53:03.067704+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:53:03.067704+0000","last_undegraded":"2023-10-30T13:53:03.067704+0000","last_fullsized":"2023-10-30T13:53:03.067704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2078","last_scrub_stamp":"2023-10-30T13:53:03.067613+0000","last_deep_scrub":"67'2078","last_deep_scrub_stamp":"2023-10-30T13:53:03.067613+0000","last_clean_scrub_stamp":"2023-10-30T13:53:03.067613+0000","objects_scrubbed":600,"log_size":2078,"log_dups_size":0,"ondisk_log_size":2078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:58.122248+0000","scrub_duration":1.2008048499999999,"objects_trimmed":0,"snaptrim_duration":0.0053787089999999997,"stat_sum":{"num_bytes":10833053,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":213,"num_read":88,"num_read_kb":3300,"num_write":1256,"num_write_kb":16238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'2337","reported_seq":4272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:55.080264+0000","last_change":"2023-10-30T13:52:55.080329+0000","last_active":"2023-10-30T13:52:55.080264+0000","last_peered":"2023-10-30T13:52:55.080264+0000","last_clean":"2023-10-30T13:52:55.080264+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:52:55.080264+0000","last_undegraded":"2023-10-30T13:52:55.080264+0000","last_fullsized":"2023-10-30T13:52:55.080264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2337","last_scrub_stamp":"2023-10-30T13:52:55.080176+0000","last_deep_scrub":"67'2337","last_deep_scrub_stamp":"2023-10-30T13:52:55.080176+0000","last_clean_scrub_stamp":"2023-10-30T13:52:55.080176+0000","objects_scrubbed":688,"log_size":2337,"log_dups_size":0,"ondisk_log_size":2337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:15:48.148329+0000","scrub_duration":0.84244230799999997,"objects_trimmed":0,"snaptrim_duration":0.0028849399999999999,"stat_sum":{"num_bytes":11995550,"num_objects":688,"num_object_clones":321,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":244,"num_read":100,"num_read_kb":8267,"num_write":1414,"num_write_kb":14306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'2013","reported_seq":3996,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:52.418032+0000","last_change":"2023-10-30T13:52:52.418138+0000","last_active":"2023-10-30T13:52:52.418032+0000","last_peered":"2023-10-30T13:52:52.418032+0000","last_clean":"2023-10-30T13:52:52.418032+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:52:52.418032+0000","last_undegraded":"2023-10-30T13:52:52.418032+0000","last_fullsized":"2023-10-30T13:52:52.418032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2013","last_scrub_stamp":"2023-10-30T13:52:52.417929+0000","last_deep_scrub":"67'2013","last_deep_scrub_stamp":"2023-10-30T13:52:52.417929+0000","last_clean_scrub_stamp":"2023-10-30T13:52:52.417929+0000","objects_scrubbed":558,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:39:32.467346+0000","scrub_duration":1.1806941150000001,"objects_trimmed":0,"snaptrim_duration":0.0096547909999999994,"stat_sum":{"num_bytes":15074071,"num_objects":558,"num_object_clones":259,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":221,"num_read":91,"num_read_kb":32,"num_write":1204,"num_write_kb":16754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'2322","reported_seq":4298,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:57.949237+0000","last_change":"2023-10-30T13:52:57.949288+0000","last_active":"2023-10-30T13:52:57.949237+0000","last_peered":"2023-10-30T13:52:57.949237+0000","last_clean":"2023-10-30T13:52:57.949237+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:52:57.949237+0000","last_undegraded":"2023-10-30T13:52:57.949237+0000","last_fullsized":"2023-10-30T13:52:57.949237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2322","last_scrub_stamp":"2023-10-30T13:52:57.949159+0000","last_deep_scrub":"67'2322","last_deep_scrub_stamp":"2023-10-30T13:52:57.949159+0000","last_clean_scrub_stamp":"2023-10-30T13:52:57.949159+0000","objects_scrubbed":649,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:26.563664+0000","scrub_duration":1.454754868,"objects_trimmed":0,"snaptrim_duration":0.0018845820000000001,"stat_sum":{"num_bytes":10798567,"num_objects":649,"num_object_clones":300,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":241,"num_read":120,"num_read_kb":27,"num_write":1443,"num_write_kb":17235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'2091","reported_seq":4094,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:12.954329+0000","last_change":"2023-10-30T13:53:12.954423+0000","last_active":"2023-10-30T13:53:12.954329+0000","last_peered":"2023-10-30T13:53:12.954329+0000","last_clean":"2023-10-30T13:53:12.954329+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:53:12.954329+0000","last_undegraded":"2023-10-30T13:53:12.954329+0000","last_fullsized":"2023-10-30T13:53:12.954329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2091","last_scrub_stamp":"2023-10-30T13:53:12.954208+0000","last_deep_scrub":"67'2091","last_deep_scrub_stamp":"2023-10-30T13:53:12.954208+0000","last_clean_scrub_stamp":"2023-10-30T13:53:12.954208+0000","objects_scrubbed":576,"log_size":2091,"log_dups_size":0,"ondisk_log_size":2091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:45.742650+0000","scrub_duration":1.0988574289999999,"objects_trimmed":0,"snaptrim_duration":0.0075239010000000004,"stat_sum":{"num_bytes":15423893,"num_objects":576,"num_object_clones":259,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":203,"num_read":109,"num_read_kb":9552,"num_write":1308,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'2174","reported_seq":4153,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:47.980889+0000","last_change":"2023-10-30T13:52:47.980946+0000","last_active":"2023-10-30T13:52:47.980889+0000","last_peered":"2023-10-30T13:52:47.980889+0000","last_clean":"2023-10-30T13:52:47.980889+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:52:47.980889+0000","last_undegraded":"2023-10-30T13:52:47.980889+0000","last_fullsized":"2023-10-30T13:52:47.980889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2174","last_scrub_stamp":"2023-10-30T13:52:47.980802+0000","last_deep_scrub":"67'2174","last_deep_scrub_stamp":"2023-10-30T13:52:47.980802+0000","last_clean_scrub_stamp":"2023-10-30T13:52:47.980802+0000","objects_scrubbed":622,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:54:24.281648+0000","scrub_duration":1.0857265140000001,"objects_trimmed":0,"snaptrim_duration":0.007356301,"stat_sum":{"num_bytes":40089180,"num_objects":622,"num_object_clones":287,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":230,"num_read":115,"num_read_kb":33,"num_write":1312,"num_write_kb":42997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'644","reported_seq":883,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.019766+0000","last_change":"2023-10-30T13:52:45.019766+0000","last_active":"2023-10-30T13:52:45.019766+0000","last_peered":"2023-10-30T13:52:45.019766+0000","last_clean":"2023-10-30T13:52:45.019766+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:52:45.019766+0000","last_undegraded":"2023-10-30T13:52:45.019766+0000","last_fullsized":"2023-10-30T13:52:45.019766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'644","last_scrub_stamp":"2023-10-30T13:52:45.019575+0000","last_deep_scrub":"67'644","last_deep_scrub_stamp":"2023-10-30T13:52:45.019575+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.019575+0000","objects_scrubbed":42,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:11:59.207979+0000","scrub_duration":0.065999268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":252,"num_read_kb":249243,"num_write":662,"num_write_kb":74846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":549256,"num_omap_keys":974,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'2188","reported_seq":4150,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:13.306061+0000","last_change":"2023-10-30T13:53:13.306155+0000","last_active":"2023-10-30T13:53:13.306061+0000","last_peered":"2023-10-30T13:53:13.306061+0000","last_clean":"2023-10-30T13:53:13.306061+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:53:13.306061+0000","last_undegraded":"2023-10-30T13:53:13.306061+0000","last_fullsized":"2023-10-30T13:53:13.306061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2188","last_scrub_stamp":"2023-10-30T13:53:13.305925+0000","last_deep_scrub":"67'2188","last_deep_scrub_stamp":"2023-10-30T13:53:13.305925+0000","last_clean_scrub_stamp":"2023-10-30T13:53:13.305925+0000","objects_scrubbed":628,"log_size":2188,"log_dups_size":0,"ondisk_log_size":2188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:47.133988+0000","scrub_duration":1.0555437249999999,"objects_trimmed":0,"snaptrim_duration":0.0031372290000000001,"stat_sum":{"num_bytes":18713903,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":229,"num_read":108,"num_read_kb":4667,"num_write":1329,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1176","reported_seq":1511,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.939106+0000","last_change":"2023-10-30T13:52:43.939106+0000","last_active":"2023-10-30T13:52:43.939106+0000","last_peered":"2023-10-30T13:52:43.939106+0000","last_clean":"2023-10-30T13:52:43.939106+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:52:43.939106+0000","last_undegraded":"2023-10-30T13:52:43.939106+0000","last_fullsized":"2023-10-30T13:52:43.939106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1176","last_scrub_stamp":"2023-10-30T13:52:43.938938+0000","last_deep_scrub":"67'1176","last_deep_scrub_stamp":"2023-10-30T13:52:43.938938+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.938938+0000","objects_scrubbed":51,"log_size":1176,"log_dups_size":0,"ondisk_log_size":1176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:18.962998+0000","scrub_duration":0.19282650300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25293488,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":404,"num_read_kb":309242,"num_write":1231,"num_write_kb":84663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":763115,"num_omap_keys":1100,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'794","reported_seq":1183,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:35.070009+0000","last_change":"2023-10-30T13:52:35.070059+0000","last_active":"2023-10-30T13:52:35.070009+0000","last_peered":"2023-10-30T13:52:35.070009+0000","last_clean":"2023-10-30T13:52:35.070009+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:52:35.070009+0000","last_undegraded":"2023-10-30T13:52:35.070009+0000","last_fullsized":"2023-10-30T13:52:35.070009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'794","last_scrub_stamp":"2023-10-30T13:52:35.069918+0000","last_deep_scrub":"67'794","last_deep_scrub_stamp":"2023-10-30T13:52:35.069918+0000","last_clean_scrub_stamp":"2023-10-30T13:52:35.069918+0000","objects_scrubbed":54,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:21.898394+0000","scrub_duration":0.14371538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397151,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":403,"num_read_kb":153383,"num_write":820,"num_write_kb":55113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":519482,"num_omap_keys":897,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'2508","reported_seq":4463,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:50.977937+0000","last_change":"2023-10-30T13:52:50.978041+0000","last_active":"2023-10-30T13:52:50.977937+0000","last_peered":"2023-10-30T13:52:50.977937+0000","last_clean":"2023-10-30T13:52:50.977937+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:52:50.977937+0000","last_undegraded":"2023-10-30T13:52:50.977937+0000","last_fullsized":"2023-10-30T13:52:50.977937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2508","last_scrub_stamp":"2023-10-30T13:52:50.977836+0000","last_deep_scrub":"67'2508","last_deep_scrub_stamp":"2023-10-30T13:52:50.977836+0000","last_clean_scrub_stamp":"2023-10-30T13:52:50.977836+0000","objects_scrubbed":729,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:26.785649+0000","scrub_duration":1.0834898019999999,"objects_trimmed":0,"snaptrim_duration":0.0064075410000000001,"stat_sum":{"num_bytes":11666119,"num_objects":729,"num_object_clones":341,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":261,"num_read":141,"num_read_kb":4175,"num_write":1520,"num_write_kb":22588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1448","reported_seq":1728,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.318938+0000","last_change":"2023-10-30T13:52:43.318938+0000","last_active":"2023-10-30T13:52:43.318938+0000","last_peered":"2023-10-30T13:52:43.318938+0000","last_clean":"2023-10-30T13:52:43.318938+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:52:43.318938+0000","last_undegraded":"2023-10-30T13:52:43.318938+0000","last_fullsized":"2023-10-30T13:52:43.318938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1447","last_scrub_stamp":"2023-10-30T13:52:43.318712+0000","last_deep_scrub":"67'1447","last_deep_scrub_stamp":"2023-10-30T13:52:43.318712+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.318712+0000","objects_scrubbed":54,"log_size":1448,"log_dups_size":0,"ondisk_log_size":1448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:38.717163+0000","scrub_duration":0.18293932199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":354,"num_read_kb":270680,"num_write":1488,"num_write_kb":75460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593681,"num_omap_keys":1037,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'2128","reported_seq":4095,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:12.713494+0000","last_change":"2023-10-30T13:53:12.713584+0000","last_active":"2023-10-30T13:53:12.713494+0000","last_peered":"2023-10-30T13:53:12.713494+0000","last_clean":"2023-10-30T13:53:12.713494+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:53:12.713494+0000","last_undegraded":"2023-10-30T13:53:12.713494+0000","last_fullsized":"2023-10-30T13:53:12.713494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2128","last_scrub_stamp":"2023-10-30T13:53:12.713409+0000","last_deep_scrub":"67'2128","last_deep_scrub_stamp":"2023-10-30T13:53:12.713409+0000","last_clean_scrub_stamp":"2023-10-30T13:53:12.713409+0000","objects_scrubbed":614,"log_size":2128,"log_dups_size":0,"ondisk_log_size":2128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:10.476233+0000","scrub_duration":1.058017481,"objects_trimmed":0,"snaptrim_duration":0.0058964639999999997,"stat_sum":{"num_bytes":19507019,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":240,"num_read":105,"num_read_kb":5362,"num_write":1250,"num_write_kb":24752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'1012","reported_seq":1259,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:02.942744+0000","last_change":"2023-10-30T13:53:02.942802+0000","last_active":"2023-10-30T13:53:02.942744+0000","last_peered":"2023-10-30T13:53:02.942744+0000","last_clean":"2023-10-30T13:53:02.942744+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:53:02.942744+0000","last_undegraded":"2023-10-30T13:53:02.942744+0000","last_fullsized":"2023-10-30T13:53:02.942744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1012","last_scrub_stamp":"2023-10-30T13:53:02.942656+0000","last_deep_scrub":"67'1012","last_deep_scrub_stamp":"2023-10-30T13:53:02.942656+0000","last_clean_scrub_stamp":"2023-10-30T13:53:02.942656+0000","objects_scrubbed":56,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:31.384045+0000","scrub_duration":0.195907044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":313,"num_read_kb":141404,"num_write":1079,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":564647,"num_omap_keys":1018,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'2081","reported_seq":4067,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:00.832604+0000","last_change":"2023-10-30T13:53:00.832674+0000","last_active":"2023-10-30T13:53:00.832604+0000","last_peered":"2023-10-30T13:53:00.832604+0000","last_clean":"2023-10-30T13:53:00.832604+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:53:00.832604+0000","last_undegraded":"2023-10-30T13:53:00.832604+0000","last_fullsized":"2023-10-30T13:53:00.832604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:53:00.832511+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:53:00.832511+0000","last_clean_scrub_stamp":"2023-10-30T13:53:00.832511+0000","objects_scrubbed":588,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:07:11.916144+0000","scrub_duration":1.374556924,"objects_trimmed":0,"snaptrim_duration":0.0025817129999999998,"stat_sum":{"num_bytes":23539910,"num_objects":588,"num_object_clones":267,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":208,"num_read":106,"num_read_kb":8313,"num_write":1266,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"67'1170","reported_seq":1398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:39.386385+0000","last_change":"2023-10-30T13:52:39.386488+0000","last_active":"2023-10-30T13:52:39.386385+0000","last_peered":"2023-10-30T13:52:39.386385+0000","last_clean":"2023-10-30T13:52:39.386385+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:52:39.386385+0000","last_undegraded":"2023-10-30T13:52:39.386385+0000","last_fullsized":"2023-10-30T13:52:39.386385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1170","last_scrub_stamp":"2023-10-30T13:52:39.386286+0000","last_deep_scrub":"67'1170","last_deep_scrub_stamp":"2023-10-30T13:52:39.386286+0000","last_clean_scrub_stamp":"2023-10-30T13:52:39.386286+0000","objects_scrubbed":49,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:00.965375+0000","scrub_duration":0.15933681099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":240,"num_read_kb":201173,"num_write":1196,"num_write_kb":66729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":467606,"num_omap_keys":916,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'2185","reported_seq":4130,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:12.335263+0000","last_change":"2023-10-30T13:53:12.335354+0000","last_active":"2023-10-30T13:53:12.335263+0000","last_peered":"2023-10-30T13:53:12.335263+0000","last_clean":"2023-10-30T13:53:12.335263+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:53:12.335263+0000","last_undegraded":"2023-10-30T13:53:12.335263+0000","last_fullsized":"2023-10-30T13:53:12.335263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2185","last_scrub_stamp":"2023-10-30T13:53:12.335127+0000","last_deep_scrub":"67'2185","last_deep_scrub_stamp":"2023-10-30T13:53:12.335127+0000","last_clean_scrub_stamp":"2023-10-30T13:53:12.335127+0000","objects_scrubbed":628,"log_size":2185,"log_dups_size":0,"ondisk_log_size":2185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:20:26.571198+0000","scrub_duration":1.1178727850000001,"objects_trimmed":0,"snaptrim_duration":0.0022874710000000001,"stat_sum":{"num_bytes":16137402,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":233,"num_read":91,"num_read_kb":34,"num_write":1322,"num_write_kb":18013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1358","reported_seq":1609,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.996716+0000","last_change":"2023-10-30T13:52:53.996816+0000","last_active":"2023-10-30T13:52:53.996716+0000","last_peered":"2023-10-30T13:52:53.996716+0000","last_clean":"2023-10-30T13:52:53.996716+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:52:53.996716+0000","last_undegraded":"2023-10-30T13:52:53.996716+0000","last_fullsized":"2023-10-30T13:52:53.996716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1358","last_scrub_stamp":"2023-10-30T13:52:53.996599+0000","last_deep_scrub":"67'1358","last_deep_scrub_stamp":"2023-10-30T13:52:53.996599+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.996599+0000","objects_scrubbed":45,"log_size":1358,"log_dups_size":0,"ondisk_log_size":1358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:58.575120+0000","scrub_duration":0.17104654599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20972192,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":307,"num_read_kb":165804,"num_write":1420,"num_write_kb":50966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":509116,"num_omap_keys":950,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'2207","reported_seq":4174,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.120386+0000","last_change":"2023-10-30T13:52:54.120454+0000","last_active":"2023-10-30T13:52:54.120386+0000","last_peered":"2023-10-30T13:52:54.120386+0000","last_clean":"2023-10-30T13:52:54.120386+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:52:54.120386+0000","last_undegraded":"2023-10-30T13:52:54.120386+0000","last_fullsized":"2023-10-30T13:52:54.120386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2207","last_scrub_stamp":"2023-10-30T13:52:54.120300+0000","last_deep_scrub":"67'2207","last_deep_scrub_stamp":"2023-10-30T13:52:54.120300+0000","last_clean_scrub_stamp":"2023-10-30T13:52:54.120300+0000","objects_scrubbed":638,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:39:00.511615+0000","scrub_duration":1.2561105379999999,"objects_trimmed":0,"snaptrim_duration":0.0087798259999999993,"stat_sum":{"num_bytes":32677996,"num_objects":638,"num_object_clones":293,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":232,"num_read":108,"num_read_kb":10951,"num_write":1342,"num_write_kb":38819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'986","reported_seq":1316,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.944325+0000","last_change":"2023-10-30T13:52:51.944325+0000","last_active":"2023-10-30T13:52:51.944325+0000","last_peered":"2023-10-30T13:52:51.944325+0000","last_clean":"2023-10-30T13:52:51.944325+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:52:51.944325+0000","last_undegraded":"2023-10-30T13:52:51.944325+0000","last_fullsized":"2023-10-30T13:52:51.944325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'985","last_scrub_stamp":"2023-10-30T13:52:51.944137+0000","last_deep_scrub":"67'985","last_deep_scrub_stamp":"2023-10-30T13:52:51.944137+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.944137+0000","objects_scrubbed":50,"log_size":986,"log_dups_size":0,"ondisk_log_size":986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:12.000749+0000","scrub_duration":0.18964769300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":413,"num_read_kb":281405,"num_write":1060,"num_write_kb":80203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":643541,"num_omap_keys":878,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'2126","reported_seq":4076,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:28.206398+0000","last_change":"2023-10-30T13:53:28.206516+0000","last_active":"2023-10-30T13:53:28.206398+0000","last_peered":"2023-10-30T13:53:28.206398+0000","last_clean":"2023-10-30T13:53:28.206398+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:53:28.206398+0000","last_undegraded":"2023-10-30T13:53:28.206398+0000","last_fullsized":"2023-10-30T13:53:28.206398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2126","last_scrub_stamp":"2023-10-30T13:53:28.206307+0000","last_deep_scrub":"67'2126","last_deep_scrub_stamp":"2023-10-30T13:53:28.206307+0000","last_clean_scrub_stamp":"2023-10-30T13:53:28.206307+0000","objects_scrubbed":614,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:13:17.846078+0000","scrub_duration":1.132694131,"objects_trimmed":0,"snaptrim_duration":0.014787140000000001,"stat_sum":{"num_bytes":19072740,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":238,"num_read":87,"num_read_kb":102,"num_write":1252,"num_write_kb":21862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'1144","reported_seq":1482,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:37.361974+0000","last_change":"2023-10-30T13:52:37.361974+0000","last_active":"2023-10-30T13:52:37.361974+0000","last_peered":"2023-10-30T13:52:37.361974+0000","last_clean":"2023-10-30T13:52:37.361974+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:52:37.361974+0000","last_undegraded":"2023-10-30T13:52:37.361974+0000","last_fullsized":"2023-10-30T13:52:37.361974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1144","last_scrub_stamp":"2023-10-30T13:52:37.361775+0000","last_deep_scrub":"67'1144","last_deep_scrub_stamp":"2023-10-30T13:52:37.361775+0000","last_clean_scrub_stamp":"2023-10-30T13:52:37.361775+0000","objects_scrubbed":60,"log_size":1144,"log_dups_size":0,"ondisk_log_size":1144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:58.622750+0000","scrub_duration":0.22738265299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748770,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":476,"num_read_kb":203988,"num_write":1202,"num_write_kb":67245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":644581,"num_omap_keys":1115,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'2090","reported_seq":4074,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.327640+0000","last_change":"2023-10-30T13:52:53.327719+0000","last_active":"2023-10-30T13:52:53.327640+0000","last_peered":"2023-10-30T13:52:53.327640+0000","last_clean":"2023-10-30T13:52:53.327640+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:52:53.327640+0000","last_undegraded":"2023-10-30T13:52:53.327640+0000","last_fullsized":"2023-10-30T13:52:53.327640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2090","last_scrub_stamp":"2023-10-30T13:52:53.327572+0000","last_deep_scrub":"67'2090","last_deep_scrub_stamp":"2023-10-30T13:52:53.327572+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.327572+0000","objects_scrubbed":584,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:03.787789+0000","scrub_duration":1.1150200729999999,"objects_trimmed":0,"snaptrim_duration":0.0024076599999999998,"stat_sum":{"num_bytes":14342228,"num_objects":584,"num_object_clones":268,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":211,"num_read":101,"num_read_kb":4159,"num_write":1284,"num_write_kb":24386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'616","reported_seq":808,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.351480+0000","last_change":"2023-10-30T13:52:40.351638+0000","last_active":"2023-10-30T13:52:40.351480+0000","last_peered":"2023-10-30T13:52:40.351480+0000","last_clean":"2023-10-30T13:52:40.351480+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:52:40.351480+0000","last_undegraded":"2023-10-30T13:52:40.351480+0000","last_fullsized":"2023-10-30T13:52:40.351480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'616","last_scrub_stamp":"2023-10-30T13:52:40.351385+0000","last_deep_scrub":"67'616","last_deep_scrub_stamp":"2023-10-30T13:52:40.351385+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.351385+0000","objects_scrubbed":45,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:25.211671+0000","scrub_duration":0.16031920099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":203,"num_read_kb":83137,"num_write":663,"num_write_kb":45991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":437141,"num_omap_keys":850,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'1825","reported_seq":3800,"reported_epoch":67,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T13:53:32.666548+0000","last_change":"2023-10-30T13:53:32.178905+0000","last_active":"2023-10-30T13:53:32.666548+0000","last_peered":"2023-10-30T13:53:32.666548+0000","last_clean":"2023-10-30T13:53:32.666548+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:53:32.666548+0000","last_undegraded":"2023-10-30T13:53:32.666548+0000","last_fullsized":"2023-10-30T13:53:32.666548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":221,"log_size":1825,"log_dups_size":0,"ondisk_log_size":1825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011943199999999999,"stat_sum":{"num_bytes":18918293,"num_objects":634,"num_object_clones":295,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":241,"num_read":154,"num_read_kb":114,"num_write":1356,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'557","reported_seq":801,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:57.004659+0000","last_change":"2023-10-30T13:52:57.004722+0000","last_active":"2023-10-30T13:52:57.004659+0000","last_peered":"2023-10-30T13:52:57.004659+0000","last_clean":"2023-10-30T13:52:57.004659+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:52:57.004659+0000","last_undegraded":"2023-10-30T13:52:57.004659+0000","last_fullsized":"2023-10-30T13:52:57.004659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'557","last_scrub_stamp":"2023-10-30T13:52:57.004551+0000","last_deep_scrub":"67'557","last_deep_scrub_stamp":"2023-10-30T13:52:57.004551+0000","last_clean_scrub_stamp":"2023-10-30T13:52:57.004551+0000","objects_scrubbed":57,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:29:17.022207+0000","scrub_duration":0.20230519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":289,"num_read_kb":121284,"num_write":557,"num_write_kb":58696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":599967,"num_omap_keys":1105,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'2295","reported_seq":4212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:19.344683+0000","last_change":"2023-10-30T13:53:19.344746+0000","last_active":"2023-10-30T13:53:19.344683+0000","last_peered":"2023-10-30T13:53:19.344683+0000","last_clean":"2023-10-30T13:53:19.344683+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:53:19.344683+0000","last_undegraded":"2023-10-30T13:53:19.344683+0000","last_fullsized":"2023-10-30T13:53:19.344683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2295","last_scrub_stamp":"2023-10-30T13:53:19.344588+0000","last_deep_scrub":"67'2295","last_deep_scrub_stamp":"2023-10-30T13:53:19.344588+0000","last_clean_scrub_stamp":"2023-10-30T13:53:19.344588+0000","objects_scrubbed":655,"log_size":2295,"log_dups_size":0,"ondisk_log_size":2295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:21:59.259100+0000","scrub_duration":1.411775386,"objects_trimmed":0,"snaptrim_duration":0.0086123929999999994,"stat_sum":{"num_bytes":19792070,"num_objects":655,"num_object_clones":349,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":204,"num_read":113,"num_read_kb":142,"num_write":1348,"num_write_kb":24080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1016","reported_seq":1307,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.650095+0000","last_change":"2023-10-30T13:52:49.650144+0000","last_active":"2023-10-30T13:52:49.650095+0000","last_peered":"2023-10-30T13:52:49.650095+0000","last_clean":"2023-10-30T13:52:49.650095+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:52:49.650095+0000","last_undegraded":"2023-10-30T13:52:49.650095+0000","last_fullsized":"2023-10-30T13:52:49.650095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1016","last_scrub_stamp":"2023-10-30T13:52:49.650007+0000","last_deep_scrub":"67'1016","last_deep_scrub_stamp":"2023-10-30T13:52:49.650007+0000","last_clean_scrub_stamp":"2023-10-30T13:52:49.650007+0000","objects_scrubbed":55,"log_size":1016,"log_dups_size":0,"ondisk_log_size":1016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:38.507861+0000","scrub_duration":0.17591847999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36678727,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":465,"num_read_kb":133831,"num_write":1108,"num_write_kb":54499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":702779,"num_omap_keys":1002,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'2188","reported_seq":4144,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:25.049855+0000","last_change":"2023-10-30T13:53:25.049916+0000","last_active":"2023-10-30T13:53:25.049855+0000","last_peered":"2023-10-30T13:53:25.049855+0000","last_clean":"2023-10-30T13:53:25.049855+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:53:25.049855+0000","last_undegraded":"2023-10-30T13:53:25.049855+0000","last_fullsized":"2023-10-30T13:53:25.049855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2188","last_scrub_stamp":"2023-10-30T13:53:25.049775+0000","last_deep_scrub":"67'2188","last_deep_scrub_stamp":"2023-10-30T13:53:25.049775+0000","last_clean_scrub_stamp":"2023-10-30T13:53:25.049775+0000","objects_scrubbed":625,"log_size":2188,"log_dups_size":0,"ondisk_log_size":2188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:45.692320+0000","scrub_duration":0.916050107,"objects_trimmed":0,"snaptrim_duration":0.015028952999999999,"stat_sum":{"num_bytes":21253392,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":236,"num_read":95,"num_read_kb":12339,"num_write":1306,"num_write_kb":30718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'688","reported_seq":925,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.045919+0000","last_change":"2023-10-30T13:52:45.045919+0000","last_active":"2023-10-30T13:52:45.045919+0000","last_peered":"2023-10-30T13:52:45.045919+0000","last_clean":"2023-10-30T13:52:45.045919+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:52:45.045919+0000","last_undegraded":"2023-10-30T13:52:45.045919+0000","last_fullsized":"2023-10-30T13:52:45.045919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'688","last_scrub_stamp":"2023-10-30T13:52:45.045739+0000","last_deep_scrub":"67'688","last_deep_scrub_stamp":"2023-10-30T13:52:45.045739+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.045739+0000","objects_scrubbed":51,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:31:53.871616+0000","scrub_duration":0.14959883700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165846,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":249,"num_read_kb":148701,"num_write":778,"num_write_kb":54421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":420719,"num_omap_keys":809,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'2193","reported_seq":4158,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:28.930438+0000","last_change":"2023-10-30T13:53:28.930491+0000","last_active":"2023-10-30T13:53:28.930438+0000","last_peered":"2023-10-30T13:53:28.930438+0000","last_clean":"2023-10-30T13:53:28.930438+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:53:28.930438+0000","last_undegraded":"2023-10-30T13:53:28.930438+0000","last_fullsized":"2023-10-30T13:53:28.930438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2193","last_scrub_stamp":"2023-10-30T13:53:28.930356+0000","last_deep_scrub":"67'2193","last_deep_scrub_stamp":"2023-10-30T13:53:28.930356+0000","last_clean_scrub_stamp":"2023-10-30T13:53:28.930356+0000","objects_scrubbed":626,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:45.236130+0000","scrub_duration":0.82498087200000003,"objects_trimmed":0,"snaptrim_duration":0.011682140000000001,"stat_sum":{"num_bytes":26941199,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":237,"num_read":103,"num_read_kb":4114,"num_write":1331,"num_write_kb":32609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'1113","reported_seq":1401,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:48.645108+0000","last_change":"2023-10-30T13:52:48.645108+0000","last_active":"2023-10-30T13:52:48.645108+0000","last_peered":"2023-10-30T13:52:48.645108+0000","last_clean":"2023-10-30T13:52:48.645108+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:52:48.645108+0000","last_undegraded":"2023-10-30T13:52:48.645108+0000","last_fullsized":"2023-10-30T13:52:48.645108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1113","last_scrub_stamp":"2023-10-30T13:52:48.644981+0000","last_deep_scrub":"67'1113","last_deep_scrub_stamp":"2023-10-30T13:52:48.644981+0000","last_clean_scrub_stamp":"2023-10-30T13:52:48.644981+0000","objects_scrubbed":63,"log_size":1113,"log_dups_size":0,"ondisk_log_size":1113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:56.952822+0000","scrub_duration":0.19793386600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":381,"num_read_kb":250673,"num_write":1157,"num_write_kb":87754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":620304,"num_omap_keys":1053,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'1540","reported_seq":3484,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.789120+0000","last_change":"2023-10-30T13:21:40.585506+0000","last_active":"2023-10-30T13:52:36.789120+0000","last_peered":"2023-10-30T13:52:36.789120+0000","last_clean":"2023-10-30T13:52:36.789120+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:52:36.789120+0000","last_undegraded":"2023-10-30T13:52:36.789120+0000","last_fullsized":"2023-10-30T13:52:36.789120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T13:10:33.664409+0000","last_clean_scrub_stamp":"2023-10-30T13:10:33.664409+0000","objects_scrubbed":0,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014794043,"stat_sum":{"num_bytes":18888621,"num_objects":614,"num_object_clones":283,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":224,"num_read":107,"num_read_kb":4188,"num_write":1306,"num_write_kb":24240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'1274","reported_seq":1536,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.034230+0000","last_change":"2023-10-30T13:52:53.034230+0000","last_active":"2023-10-30T13:52:53.034230+0000","last_peered":"2023-10-30T13:52:53.034230+0000","last_clean":"2023-10-30T13:52:53.034230+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:52:53.034230+0000","last_undegraded":"2023-10-30T13:52:53.034230+0000","last_fullsized":"2023-10-30T13:52:53.034230+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1274","last_scrub_stamp":"2023-10-30T13:52:53.034072+0000","last_deep_scrub":"67'1274","last_deep_scrub_stamp":"2023-10-30T13:52:53.034072+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.034072+0000","objects_scrubbed":46,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:05:34.464424+0000","scrub_duration":0.19364663100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364039,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":288,"num_read_kb":200539,"num_write":1327,"num_write_kb":71054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":407575,"num_omap_keys":755,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'2176","reported_seq":4163,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:48.181004+0000","last_change":"2023-10-30T13:52:48.181151+0000","last_active":"2023-10-30T13:52:48.181004+0000","last_peered":"2023-10-30T13:52:48.181004+0000","last_clean":"2023-10-30T13:52:48.181004+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:52:48.181004+0000","last_undegraded":"2023-10-30T13:52:48.181004+0000","last_fullsized":"2023-10-30T13:52:48.181004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2176","last_scrub_stamp":"2023-10-30T13:52:48.180852+0000","last_deep_scrub":"67'2176","last_deep_scrub_stamp":"2023-10-30T13:52:48.180852+0000","last_clean_scrub_stamp":"2023-10-30T13:52:48.180852+0000","objects_scrubbed":600,"log_size":2176,"log_dups_size":0,"ondisk_log_size":2176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:11.947771+0000","scrub_duration":1.0044801649999999,"objects_trimmed":0,"snaptrim_duration":0.010788236,"stat_sum":{"num_bytes":10484856,"num_objects":600,"num_object_clones":281,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":220,"num_read":116,"num_read_kb":28,"num_write":1335,"num_write_kb":13295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'832","reported_seq":1091,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.593522+0000","last_change":"2023-10-30T13:52:53.593522+0000","last_active":"2023-10-30T13:52:53.593522+0000","last_peered":"2023-10-30T13:52:53.593522+0000","last_clean":"2023-10-30T13:52:53.593522+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:52:53.593522+0000","last_undegraded":"2023-10-30T13:52:53.593522+0000","last_fullsized":"2023-10-30T13:52:53.593522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'832","last_scrub_stamp":"2023-10-30T13:52:53.593416+0000","last_deep_scrub":"67'832","last_deep_scrub_stamp":"2023-10-30T13:52:53.593416+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.593416+0000","objects_scrubbed":54,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:41.436296+0000","scrub_duration":0.145814324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":174067,"num_write":897,"num_write_kb":62874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":516505,"num_omap_keys":962,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'2158","reported_seq":4124,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:30.220672+0000","last_change":"2023-10-30T13:53:30.220750+0000","last_active":"2023-10-30T13:53:30.220672+0000","last_peered":"2023-10-30T13:53:30.220672+0000","last_clean":"2023-10-30T13:53:30.220672+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:53:30.220672+0000","last_undegraded":"2023-10-30T13:53:30.220672+0000","last_fullsized":"2023-10-30T13:53:30.220672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2158","last_scrub_stamp":"2023-10-30T13:53:30.220578+0000","last_deep_scrub":"67'2158","last_deep_scrub_stamp":"2023-10-30T13:53:30.220578+0000","last_clean_scrub_stamp":"2023-10-30T13:53:30.220578+0000","objects_scrubbed":616,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:58.809379+0000","scrub_duration":1.1187861859999999,"objects_trimmed":0,"snaptrim_duration":0.015268978000000001,"stat_sum":{"num_bytes":11043591,"num_objects":616,"num_object_clones":291,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":221,"num_read":104,"num_read_kb":3707,"num_write":1322,"num_write_kb":16509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":244,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.159767+0000","last_change":"2023-10-30T13:52:40.159977+0000","last_active":"2023-10-30T13:52:40.159767+0000","last_peered":"2023-10-30T13:52:40.159767+0000","last_clean":"2023-10-30T13:52:40.159767+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:52:40.159767+0000","last_undegraded":"2023-10-30T13:52:40.159767+0000","last_fullsized":"2023-10-30T13:52:40.159767+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-30T13:52:40.159494+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-30T13:52:40.159494+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.159494+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:37:13.673701+0000","scrub_duration":0.017799540999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'670","reported_seq":889,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:42.315206+0000","last_change":"2023-10-30T13:52:42.315332+0000","last_active":"2023-10-30T13:52:42.315206+0000","last_peered":"2023-10-30T13:52:42.315206+0000","last_clean":"2023-10-30T13:52:42.315206+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:52:42.315206+0000","last_undegraded":"2023-10-30T13:52:42.315206+0000","last_fullsized":"2023-10-30T13:52:42.315206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'670","last_scrub_stamp":"2023-10-30T13:52:42.314975+0000","last_deep_scrub":"67'670","last_deep_scrub_stamp":"2023-10-30T13:52:42.314975+0000","last_clean_scrub_stamp":"2023-10-30T13:52:42.314975+0000","objects_scrubbed":53,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:16.143637+0000","scrub_duration":0.170007133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360178,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":232,"num_read_kb":148904,"num_write":701,"num_write_kb":62706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":385690,"num_omap_keys":753,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'2178","reported_seq":4125,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:11.189453+0000","last_change":"2023-10-30T13:53:11.189527+0000","last_active":"2023-10-30T13:53:11.189453+0000","last_peered":"2023-10-30T13:53:11.189453+0000","last_clean":"2023-10-30T13:53:11.189453+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:53:11.189453+0000","last_undegraded":"2023-10-30T13:53:11.189453+0000","last_fullsized":"2023-10-30T13:53:11.189453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2178","last_scrub_stamp":"2023-10-30T13:53:11.189379+0000","last_deep_scrub":"67'2178","last_deep_scrub_stamp":"2023-10-30T13:53:11.189379+0000","last_clean_scrub_stamp":"2023-10-30T13:53:11.189379+0000","objects_scrubbed":633,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:47.706046+0000","scrub_duration":1.230169694,"objects_trimmed":0,"snaptrim_duration":0.0057314540000000004,"stat_sum":{"num_bytes":22780711,"num_objects":633,"num_object_clones":299,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":242,"num_read":92,"num_read_kb":12,"num_write":1293,"num_write_kb":28237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"67'756","reported_seq":998,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:04.119917+0000","last_change":"2023-10-30T13:53:04.120074+0000","last_active":"2023-10-30T13:53:04.119917+0000","last_peered":"2023-10-30T13:53:04.119917+0000","last_clean":"2023-10-30T13:53:04.119917+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:53:04.119917+0000","last_undegraded":"2023-10-30T13:53:04.119917+0000","last_fullsized":"2023-10-30T13:53:04.119917+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'756","last_scrub_stamp":"2023-10-30T13:53:04.119811+0000","last_deep_scrub":"67'756","last_deep_scrub_stamp":"2023-10-30T13:53:04.119811+0000","last_clean_scrub_stamp":"2023-10-30T13:53:04.119811+0000","objects_scrubbed":49,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:34.012792+0000","scrub_duration":0.144148941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":256,"num_read_kb":247261,"num_write":740,"num_write_kb":70569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":420741,"num_omap_keys":813,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'2028","reported_seq":3997,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.341768+0000","last_change":"2023-10-30T13:52:40.341843+0000","last_active":"2023-10-30T13:52:40.341768+0000","last_peered":"2023-10-30T13:52:40.341768+0000","last_clean":"2023-10-30T13:52:40.341768+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:52:40.341768+0000","last_undegraded":"2023-10-30T13:52:40.341768+0000","last_fullsized":"2023-10-30T13:52:40.341768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2028","last_scrub_stamp":"2023-10-30T13:52:40.341675+0000","last_deep_scrub":"67'2028","last_deep_scrub_stamp":"2023-10-30T13:52:40.341675+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.341675+0000","objects_scrubbed":568,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:22:08.372774+0000","scrub_duration":1.158419098,"objects_trimmed":0,"snaptrim_duration":0.002406114,"stat_sum":{"num_bytes":14184993,"num_objects":568,"num_object_clones":263,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":210,"num_read":80,"num_read_kb":26,"num_write":1239,"num_write_kb":16314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'2090","reported_seq":3466,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.634617+0000","last_change":"2023-10-30T13:52:36.634675+0000","last_active":"2023-10-30T13:52:36.634617+0000","last_peered":"2023-10-30T13:52:36.634617+0000","last_clean":"2023-10-30T13:52:36.634617+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:52:36.634617+0000","last_undegraded":"2023-10-30T13:52:36.634617+0000","last_fullsized":"2023-10-30T13:52:36.634617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2090","last_scrub_stamp":"2023-10-30T13:52:36.634514+0000","last_deep_scrub":"67'2090","last_deep_scrub_stamp":"2023-10-30T13:52:36.634514+0000","last_clean_scrub_stamp":"2023-10-30T13:52:36.634514+0000","objects_scrubbed":53,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:20.160935+0000","scrub_duration":0.14709519600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41670866,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":1534,"num_read_kb":188566,"num_write":2090,"num_write_kb":83865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":477756,"num_omap_keys":784,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'2139","reported_seq":4115,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:09.536900+0000","last_change":"2023-10-30T13:53:09.536942+0000","last_active":"2023-10-30T13:53:09.536900+0000","last_peered":"2023-10-30T13:53:09.536900+0000","last_clean":"2023-10-30T13:53:09.536900+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:53:09.536900+0000","last_undegraded":"2023-10-30T13:53:09.536900+0000","last_fullsized":"2023-10-30T13:53:09.536900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2139","last_scrub_stamp":"2023-10-30T13:53:09.536825+0000","last_deep_scrub":"67'2139","last_deep_scrub_stamp":"2023-10-30T13:53:09.536825+0000","last_clean_scrub_stamp":"2023-10-30T13:53:09.536825+0000","objects_scrubbed":612,"log_size":2139,"log_dups_size":0,"ondisk_log_size":2139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:54:03.730208+0000","scrub_duration":0.83069248900000003,"objects_trimmed":0,"snaptrim_duration":0.002908281,"stat_sum":{"num_bytes":22417951,"num_objects":612,"num_object_clones":284,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":221,"num_read":105,"num_read_kb":44,"num_write":1298,"num_write_kb":23585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'1007","reported_seq":1293,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.427338+0000","last_change":"2023-10-30T13:52:36.427454+0000","last_active":"2023-10-30T13:52:36.427338+0000","last_peered":"2023-10-30T13:52:36.427338+0000","last_clean":"2023-10-30T13:52:36.427338+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:52:36.427338+0000","last_undegraded":"2023-10-30T13:52:36.427338+0000","last_fullsized":"2023-10-30T13:52:36.427338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1007","last_scrub_stamp":"2023-10-30T13:52:36.427140+0000","last_deep_scrub":"67'1007","last_deep_scrub_stamp":"2023-10-30T13:52:36.427140+0000","last_clean_scrub_stamp":"2023-10-30T13:52:36.427140+0000","objects_scrubbed":60,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:11.996639+0000","scrub_duration":0.17205899599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":355,"num_read_kb":252215,"num_write":1079,"num_write_kb":75669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":633823,"num_omap_keys":985,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'2106","reported_seq":4092,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.008269+0000","last_change":"2023-10-30T13:52:51.008354+0000","last_active":"2023-10-30T13:52:51.008269+0000","last_peered":"2023-10-30T13:52:51.008269+0000","last_clean":"2023-10-30T13:52:51.008269+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:52:51.008269+0000","last_undegraded":"2023-10-30T13:52:51.008269+0000","last_fullsized":"2023-10-30T13:52:51.008269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2106","last_scrub_stamp":"2023-10-30T13:52:51.008186+0000","last_deep_scrub":"67'2106","last_deep_scrub_stamp":"2023-10-30T13:52:51.008186+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.008186+0000","objects_scrubbed":575,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:38.209876+0000","scrub_duration":1.1485167089999999,"objects_trimmed":0,"snaptrim_duration":0.007710705,"stat_sum":{"num_bytes":23751577,"num_objects":575,"num_object_clones":266,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":199,"num_read":100,"num_read_kb":489,"num_write":1325,"num_write_kb":27587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'998","reported_seq":1249,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.917921+0000","last_change":"2023-10-30T13:52:43.917980+0000","last_active":"2023-10-30T13:52:43.917921+0000","last_peered":"2023-10-30T13:52:43.917921+0000","last_clean":"2023-10-30T13:52:43.917921+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:52:43.917921+0000","last_undegraded":"2023-10-30T13:52:43.917921+0000","last_fullsized":"2023-10-30T13:52:43.917921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'998","last_scrub_stamp":"2023-10-30T13:52:43.917841+0000","last_deep_scrub":"67'998","last_deep_scrub_stamp":"2023-10-30T13:52:43.917841+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.917841+0000","objects_scrubbed":56,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:41:43.976830+0000","scrub_duration":0.14471989900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":294,"num_read_kb":227180,"num_write":1039,"num_write_kb":87590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":525988,"num_omap_keys":913,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2248","reported_seq":4193,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:01.680696+0000","last_change":"2023-10-30T13:53:01.680795+0000","last_active":"2023-10-30T13:53:01.680696+0000","last_peered":"2023-10-30T13:53:01.680696+0000","last_clean":"2023-10-30T13:53:01.680696+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:53:01.680696+0000","last_undegraded":"2023-10-30T13:53:01.680696+0000","last_fullsized":"2023-10-30T13:53:01.680696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2248","last_scrub_stamp":"2023-10-30T13:53:01.680522+0000","last_deep_scrub":"67'2248","last_deep_scrub_stamp":"2023-10-30T13:53:01.680522+0000","last_clean_scrub_stamp":"2023-10-30T13:53:01.680522+0000","objects_scrubbed":650,"log_size":2248,"log_dups_size":0,"ondisk_log_size":2248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:34.935002+0000","scrub_duration":1.35250176,"objects_trimmed":0,"snaptrim_duration":0.010719574000000001,"stat_sum":{"num_bytes":35538242,"num_objects":650,"num_object_clones":301,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":239,"num_read":91,"num_read_kb":31,"num_write":1347,"num_write_kb":41412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'834","reported_seq":1069,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:17.238395+0000","last_change":"2023-10-30T13:53:17.238539+0000","last_active":"2023-10-30T13:53:17.238395+0000","last_peered":"2023-10-30T13:53:17.238395+0000","last_clean":"2023-10-30T13:53:17.238395+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:53:17.238395+0000","last_undegraded":"2023-10-30T13:53:17.238395+0000","last_fullsized":"2023-10-30T13:53:17.238395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'834","last_scrub_stamp":"2023-10-30T13:53:17.238196+0000","last_deep_scrub":"67'834","last_deep_scrub_stamp":"2023-10-30T13:53:17.238196+0000","last_clean_scrub_stamp":"2023-10-30T13:53:17.238196+0000","objects_scrubbed":47,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:25:51.753551+0000","scrub_duration":0.16901470499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":258,"num_read_kb":177173,"num_write":845,"num_write_kb":62523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":428645,"num_omap_keys":765,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'2182","reported_seq":4142,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:13.698099+0000","last_change":"2023-10-30T13:53:13.698196+0000","last_active":"2023-10-30T13:53:13.698099+0000","last_peered":"2023-10-30T13:53:13.698099+0000","last_clean":"2023-10-30T13:53:13.698099+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:53:13.698099+0000","last_undegraded":"2023-10-30T13:53:13.698099+0000","last_fullsized":"2023-10-30T13:53:13.698099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2182","last_scrub_stamp":"2023-10-30T13:53:13.697999+0000","last_deep_scrub":"67'2182","last_deep_scrub_stamp":"2023-10-30T13:53:13.697999+0000","last_clean_scrub_stamp":"2023-10-30T13:53:13.697999+0000","objects_scrubbed":631,"log_size":2182,"log_dups_size":0,"ondisk_log_size":2182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:54.907462+0000","scrub_duration":0.98641216499999995,"objects_trimmed":0,"snaptrim_duration":0.0059276069999999997,"stat_sum":{"num_bytes":23389062,"num_objects":631,"num_object_clones":294,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":232,"num_read":100,"num_read_kb":25,"num_write":1313,"num_write_kb":24315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'969","reported_seq":1219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:00.092676+0000","last_change":"2023-10-30T13:53:00.092749+0000","last_active":"2023-10-30T13:53:00.092676+0000","last_peered":"2023-10-30T13:53:00.092676+0000","last_clean":"2023-10-30T13:53:00.092676+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:53:00.092676+0000","last_undegraded":"2023-10-30T13:53:00.092676+0000","last_fullsized":"2023-10-30T13:53:00.092676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'969","last_scrub_stamp":"2023-10-30T13:53:00.092550+0000","last_deep_scrub":"67'969","last_deep_scrub_stamp":"2023-10-30T13:53:00.092550+0000","last_clean_scrub_stamp":"2023-10-30T13:53:00.092550+0000","objects_scrubbed":54,"log_size":969,"log_dups_size":0,"ondisk_log_size":969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:09.851840+0000","scrub_duration":0.31438043799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":310,"num_read_kb":109156,"num_write":1496,"num_write_kb":61821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":485487,"num_omap_keys":848,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1992","reported_seq":3976,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:56.936214+0000","last_change":"2023-10-30T13:52:56.936303+0000","last_active":"2023-10-30T13:52:56.936214+0000","last_peered":"2023-10-30T13:52:56.936214+0000","last_clean":"2023-10-30T13:52:56.936214+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:52:56.936214+0000","last_undegraded":"2023-10-30T13:52:56.936214+0000","last_fullsized":"2023-10-30T13:52:56.936214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1992","last_scrub_stamp":"2023-10-30T13:52:56.936120+0000","last_deep_scrub":"67'1992","last_deep_scrub_stamp":"2023-10-30T13:52:56.936120+0000","last_clean_scrub_stamp":"2023-10-30T13:52:56.936120+0000","objects_scrubbed":571,"log_size":1992,"log_dups_size":0,"ondisk_log_size":1992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:21.220503+0000","scrub_duration":1.0834315910000001,"objects_trimmed":0,"snaptrim_duration":0.0056940879999999999,"stat_sum":{"num_bytes":23042560,"num_objects":571,"num_object_clones":259,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":201,"num_read":92,"num_read_kb":75,"num_write":1201,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'1157","reported_seq":1411,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.055552+0000","last_change":"2023-10-30T13:52:54.055652+0000","last_active":"2023-10-30T13:52:54.055552+0000","last_peered":"2023-10-30T13:52:54.055552+0000","last_clean":"2023-10-30T13:52:54.055552+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:52:54.055552+0000","last_undegraded":"2023-10-30T13:52:54.055552+0000","last_fullsized":"2023-10-30T13:52:54.055552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1157","last_scrub_stamp":"2023-10-30T13:52:54.055439+0000","last_deep_scrub":"67'1157","last_deep_scrub_stamp":"2023-10-30T13:52:54.055439+0000","last_clean_scrub_stamp":"2023-10-30T13:52:54.055439+0000","objects_scrubbed":62,"log_size":1157,"log_dups_size":0,"ondisk_log_size":1157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:52.120399+0000","scrub_duration":0.23085536300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":342,"num_read_kb":120967,"num_write":1233,"num_write_kb":51153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":723056,"num_omap_keys":1120,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'2017","reported_seq":4010,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.248739+0000","last_change":"2023-10-30T13:52:59.248901+0000","last_active":"2023-10-30T13:52:59.248739+0000","last_peered":"2023-10-30T13:52:59.248739+0000","last_clean":"2023-10-30T13:52:59.248739+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:52:59.248739+0000","last_undegraded":"2023-10-30T13:52:59.248739+0000","last_fullsized":"2023-10-30T13:52:59.248739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2017","last_scrub_stamp":"2023-10-30T13:52:59.248637+0000","last_deep_scrub":"67'2017","last_deep_scrub_stamp":"2023-10-30T13:52:59.248637+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.248637+0000","objects_scrubbed":572,"log_size":2017,"log_dups_size":0,"ondisk_log_size":2017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:42.043697+0000","scrub_duration":1.079787013,"objects_trimmed":0,"snaptrim_duration":0.0029926589999999999,"stat_sum":{"num_bytes":30813966,"num_objects":572,"num_object_clones":263,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":205,"num_read":105,"num_read_kb":87,"num_write":1219,"num_write_kb":33525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'468","reported_seq":664,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.111720+0000","last_change":"2023-10-30T13:52:59.111898+0000","last_active":"2023-10-30T13:52:59.111720+0000","last_peered":"2023-10-30T13:52:59.111720+0000","last_clean":"2023-10-30T13:52:59.111720+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:52:59.111720+0000","last_undegraded":"2023-10-30T13:52:59.111720+0000","last_fullsized":"2023-10-30T13:52:59.111720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'468","last_scrub_stamp":"2023-10-30T13:52:59.111509+0000","last_deep_scrub":"67'468","last_deep_scrub_stamp":"2023-10-30T13:52:59.111509+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.111509+0000","objects_scrubbed":47,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:39.063865+0000","scrub_duration":0.1401828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":209,"num_read_kb":100741,"num_write":482,"num_write_kb":29419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":340147,"num_omap_keys":605,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'2140","reported_seq":4127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:57.118260+0000","last_change":"2023-10-30T13:52:57.118430+0000","last_active":"2023-10-30T13:52:57.118260+0000","last_peered":"2023-10-30T13:52:57.118260+0000","last_clean":"2023-10-30T13:52:57.118260+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:52:57.118260+0000","last_undegraded":"2023-10-30T13:52:57.118260+0000","last_fullsized":"2023-10-30T13:52:57.118260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2140","last_scrub_stamp":"2023-10-30T13:52:57.118012+0000","last_deep_scrub":"67'2140","last_deep_scrub_stamp":"2023-10-30T13:52:57.118012+0000","last_clean_scrub_stamp":"2023-10-30T13:52:57.118012+0000","objects_scrubbed":595,"log_size":2140,"log_dups_size":0,"ondisk_log_size":2140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:16:30.111801+0000","scrub_duration":1.2581118739999999,"objects_trimmed":0,"snaptrim_duration":0.0057673200000000003,"stat_sum":{"num_bytes":13994934,"num_objects":595,"num_object_clones":279,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":215,"num_read":114,"num_read_kb":29,"num_write":1329,"num_write_kb":19694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'1276","reported_seq":1501,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.113521+0000","last_change":"2023-10-30T13:52:34.896255+0000","last_active":"2023-10-30T13:52:54.113521+0000","last_peered":"2023-10-30T13:52:54.113521+0000","last_clean":"2023-10-30T13:52:54.113521+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:52:54.113521+0000","last_undegraded":"2023-10-30T13:52:54.113521+0000","last_fullsized":"2023-10-30T13:52:54.113521+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1266","last_scrub_stamp":"2023-10-30T13:52:34.896121+0000","last_deep_scrub":"67'1266","last_deep_scrub_stamp":"2023-10-30T13:52:34.896121+0000","last_clean_scrub_stamp":"2023-10-30T13:52:34.896121+0000","objects_scrubbed":51,"log_size":1276,"log_dups_size":0,"ondisk_log_size":1276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:43.920989+0000","scrub_duration":0.15482325699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25006847,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":251,"num_read_kb":99282,"num_write":1301,"num_write_kb":42450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":678730,"num_omap_keys":1151,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'2168","reported_seq":4155,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:06.891046+0000","last_change":"2023-10-30T13:53:06.891119+0000","last_active":"2023-10-30T13:53:06.891046+0000","last_peered":"2023-10-30T13:53:06.891046+0000","last_clean":"2023-10-30T13:53:06.891046+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:53:06.891046+0000","last_undegraded":"2023-10-30T13:53:06.891046+0000","last_fullsized":"2023-10-30T13:53:06.891046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2168","last_scrub_stamp":"2023-10-30T13:53:06.890960+0000","last_deep_scrub":"67'2168","last_deep_scrub_stamp":"2023-10-30T13:53:06.890960+0000","last_clean_scrub_stamp":"2023-10-30T13:53:06.890960+0000","objects_scrubbed":612,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:45:46.242387+0000","scrub_duration":0.94320374900000004,"objects_trimmed":0,"snaptrim_duration":0.0063570620000000001,"stat_sum":{"num_bytes":16293763,"num_objects":612,"num_object_clones":286,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":233,"num_read":120,"num_read_kb":127,"num_write":1302,"num_write_kb":21658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'946","reported_seq":1182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:35.887235+0000","last_change":"2023-10-30T13:52:35.887314+0000","last_active":"2023-10-30T13:52:35.887235+0000","last_peered":"2023-10-30T13:52:35.887235+0000","last_clean":"2023-10-30T13:52:35.887235+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:52:35.887235+0000","last_undegraded":"2023-10-30T13:52:35.887235+0000","last_fullsized":"2023-10-30T13:52:35.887235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'946","last_scrub_stamp":"2023-10-30T13:52:35.887142+0000","last_deep_scrub":"67'946","last_deep_scrub_stamp":"2023-10-30T13:52:35.887142+0000","last_clean_scrub_stamp":"2023-10-30T13:52:35.887142+0000","objects_scrubbed":46,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:06:00.273772+0000","scrub_duration":0.098447004000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":272,"num_read_kb":200216,"num_write":1137,"num_write_kb":59039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":497965,"num_omap_keys":763,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1999","reported_seq":3980,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.786866+0000","last_change":"2023-10-30T13:52:59.786866+0000","last_active":"2023-10-30T13:52:59.786866+0000","last_peered":"2023-10-30T13:52:59.786866+0000","last_clean":"2023-10-30T13:52:59.786866+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:52:59.786866+0000","last_undegraded":"2023-10-30T13:52:59.786866+0000","last_fullsized":"2023-10-30T13:52:59.786866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1999","last_scrub_stamp":"2023-10-30T13:52:59.786686+0000","last_deep_scrub":"67'1999","last_deep_scrub_stamp":"2023-10-30T13:52:59.786686+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.786686+0000","objects_scrubbed":563,"log_size":1999,"log_dups_size":0,"ondisk_log_size":1999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:35.870896+0000","scrub_duration":1.004166471,"objects_trimmed":0,"snaptrim_duration":0.0060201619999999999,"stat_sum":{"num_bytes":10033117,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":215,"num_read":92,"num_read_kb":24,"num_write":1204,"num_write_kb":13175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'1025","reported_seq":1292,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:44.395104+0000","last_change":"2023-10-30T13:52:44.395104+0000","last_active":"2023-10-30T13:52:44.395104+0000","last_peered":"2023-10-30T13:52:44.395104+0000","last_clean":"2023-10-30T13:52:44.395104+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:52:44.395104+0000","last_undegraded":"2023-10-30T13:52:44.395104+0000","last_fullsized":"2023-10-30T13:52:44.395104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1025","last_scrub_stamp":"2023-10-30T13:52:44.394896+0000","last_deep_scrub":"67'1025","last_deep_scrub_stamp":"2023-10-30T13:52:44.394896+0000","last_clean_scrub_stamp":"2023-10-30T13:52:44.394896+0000","objects_scrubbed":56,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:31.686901+0000","scrub_duration":0.17278753699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":318,"num_read_kb":202944,"num_write":1065,"num_write_kb":67614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":660981,"num_omap_keys":1062,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'2026","reported_seq":4005,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:13.613831+0000","last_change":"2023-10-30T13:53:13.614016+0000","last_active":"2023-10-30T13:53:13.613831+0000","last_peered":"2023-10-30T13:53:13.613831+0000","last_clean":"2023-10-30T13:53:13.613831+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:53:13.613831+0000","last_undegraded":"2023-10-30T13:53:13.613831+0000","last_fullsized":"2023-10-30T13:53:13.613831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2026","last_scrub_stamp":"2023-10-30T13:53:13.613723+0000","last_deep_scrub":"67'2026","last_deep_scrub_stamp":"2023-10-30T13:53:13.613723+0000","last_clean_scrub_stamp":"2023-10-30T13:53:13.613723+0000","objects_scrubbed":558,"log_size":2026,"log_dups_size":0,"ondisk_log_size":2026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:19.365469+0000","scrub_duration":0.69858885699999995,"objects_trimmed":0,"snaptrim_duration":0.0079253199999999996,"stat_sum":{"num_bytes":6591050,"num_objects":558,"num_object_clones":254,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":200,"num_read":83,"num_read_kb":63,"num_write":1255,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'2083","reported_seq":4060,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:04.412236+0000","last_change":"2023-10-30T13:53:04.412236+0000","last_active":"2023-10-30T13:53:04.412236+0000","last_peered":"2023-10-30T13:53:04.412236+0000","last_clean":"2023-10-30T13:53:04.412236+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:53:04.412236+0000","last_undegraded":"2023-10-30T13:53:04.412236+0000","last_fullsized":"2023-10-30T13:53:04.412236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2083","last_scrub_stamp":"2023-10-30T13:53:04.411986+0000","last_deep_scrub":"67'2083","last_deep_scrub_stamp":"2023-10-30T13:53:04.411986+0000","last_clean_scrub_stamp":"2023-10-30T13:53:04.411986+0000","objects_scrubbed":585,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:37.812935+0000","scrub_duration":1.1529056559999999,"objects_trimmed":0,"snaptrim_duration":0.010603414,"stat_sum":{"num_bytes":9104816,"num_objects":585,"num_object_clones":276,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":204,"num_read":104,"num_read_kb":44,"num_write":1285,"num_write_kb":14447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'1055","reported_seq":1321,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.345144+0000","last_change":"2023-10-30T13:52:43.345144+0000","last_active":"2023-10-30T13:52:43.345144+0000","last_peered":"2023-10-30T13:52:43.345144+0000","last_clean":"2023-10-30T13:52:43.345144+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:52:43.345144+0000","last_undegraded":"2023-10-30T13:52:43.345144+0000","last_fullsized":"2023-10-30T13:52:43.345144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1055","last_scrub_stamp":"2023-10-30T13:52:43.344949+0000","last_deep_scrub":"67'1055","last_deep_scrub_stamp":"2023-10-30T13:52:43.344949+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.344949+0000","objects_scrubbed":55,"log_size":1055,"log_dups_size":0,"ondisk_log_size":1055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:59:10.108303+0000","scrub_duration":0.154960875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":337,"num_read_kb":187117,"num_write":1200,"num_write_kb":47577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706060,"num_omap_keys":1147,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'2277","reported_seq":4227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:55.617708+0000","last_change":"2023-10-30T13:52:55.617708+0000","last_active":"2023-10-30T13:52:55.617708+0000","last_peered":"2023-10-30T13:52:55.617708+0000","last_clean":"2023-10-30T13:52:55.617708+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:52:55.617708+0000","last_undegraded":"2023-10-30T13:52:55.617708+0000","last_fullsized":"2023-10-30T13:52:55.617708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2277","last_scrub_stamp":"2023-10-30T13:52:55.617575+0000","last_deep_scrub":"67'2277","last_deep_scrub_stamp":"2023-10-30T13:52:55.617575+0000","last_clean_scrub_stamp":"2023-10-30T13:52:55.617575+0000","objects_scrubbed":650,"log_size":2277,"log_dups_size":0,"ondisk_log_size":2277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:16:39.666383+0000","scrub_duration":1.3596852369999999,"objects_trimmed":0,"snaptrim_duration":0.0097894100000000001,"stat_sum":{"num_bytes":19085997,"num_objects":650,"num_object_clones":307,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":230,"num_read":105,"num_read_kb":202,"num_write":1384,"num_write_kb":24607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1382","reported_seq":1696,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:05.200352+0000","last_change":"2023-10-30T13:53:05.200352+0000","last_active":"2023-10-30T13:53:05.200352+0000","last_peered":"2023-10-30T13:53:05.200352+0000","last_clean":"2023-10-30T13:53:05.200352+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:53:05.200352+0000","last_undegraded":"2023-10-30T13:53:05.200352+0000","last_fullsized":"2023-10-30T13:53:05.200352+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1382","last_scrub_stamp":"2023-10-30T13:53:05.200180+0000","last_deep_scrub":"67'1382","last_deep_scrub_stamp":"2023-10-30T13:53:05.200180+0000","last_clean_scrub_stamp":"2023-10-30T13:53:05.200180+0000","objects_scrubbed":68,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:00.361266+0000","scrub_duration":0.25105026499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37661445,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":418,"num_read_kb":304041,"num_write":1419,"num_write_kb":92539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903188,"num_omap_keys":1478,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'2147","reported_seq":4104,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:06.340082+0000","last_change":"2023-10-30T13:53:06.340161+0000","last_active":"2023-10-30T13:53:06.340082+0000","last_peered":"2023-10-30T13:53:06.340082+0000","last_clean":"2023-10-30T13:53:06.340082+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:53:06.340082+0000","last_undegraded":"2023-10-30T13:53:06.340082+0000","last_fullsized":"2023-10-30T13:53:06.340082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2147","last_scrub_stamp":"2023-10-30T13:53:06.340001+0000","last_deep_scrub":"67'2147","last_deep_scrub_stamp":"2023-10-30T13:53:06.340001+0000","last_clean_scrub_stamp":"2023-10-30T13:53:06.340001+0000","objects_scrubbed":636,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:44.975735+0000","scrub_duration":1.142281364,"objects_trimmed":0,"snaptrim_duration":0.0026802879999999999,"stat_sum":{"num_bytes":22843366,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":237,"num_read":96,"num_read_kb":23,"num_write":1281,"num_write_kb":25082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'822","reported_seq":1066,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:44.024796+0000","last_change":"2023-10-30T13:52:44.024796+0000","last_active":"2023-10-30T13:52:44.024796+0000","last_peered":"2023-10-30T13:52:44.024796+0000","last_clean":"2023-10-30T13:52:44.024796+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:52:44.024796+0000","last_undegraded":"2023-10-30T13:52:44.024796+0000","last_fullsized":"2023-10-30T13:52:44.024796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'822","last_scrub_stamp":"2023-10-30T13:52:44.024664+0000","last_deep_scrub":"67'822","last_deep_scrub_stamp":"2023-10-30T13:52:44.024664+0000","last_clean_scrub_stamp":"2023-10-30T13:52:44.024664+0000","objects_scrubbed":50,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:54.864042+0000","scrub_duration":0.11718745699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":280,"num_read_kb":212075,"num_write":835,"num_write_kb":74769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":365319,"num_omap_keys":701,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'2055","reported_seq":4042,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:05.863446+0000","last_change":"2023-10-30T13:53:05.863560+0000","last_active":"2023-10-30T13:53:05.863446+0000","last_peered":"2023-10-30T13:53:05.863446+0000","last_clean":"2023-10-30T13:53:05.863446+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:53:05.863446+0000","last_undegraded":"2023-10-30T13:53:05.863446+0000","last_fullsized":"2023-10-30T13:53:05.863446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2055","last_scrub_stamp":"2023-10-30T13:53:05.863371+0000","last_deep_scrub":"67'2055","last_deep_scrub_stamp":"2023-10-30T13:53:05.863371+0000","last_clean_scrub_stamp":"2023-10-30T13:53:05.863371+0000","objects_scrubbed":589,"log_size":2055,"log_dups_size":0,"ondisk_log_size":2055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:54:20.796229+0000","scrub_duration":0.95279065399999996,"objects_trimmed":0,"snaptrim_duration":0.0055040289999999997,"stat_sum":{"num_bytes":14435337,"num_objects":589,"num_object_clones":269,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":212,"num_read":103,"num_read_kb":22,"num_write":1235,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'768","reported_seq":1042,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:34.271316+0000","last_change":"2023-10-30T13:52:34.271376+0000","last_active":"2023-10-30T13:52:34.271316+0000","last_peered":"2023-10-30T13:52:34.271316+0000","last_clean":"2023-10-30T13:52:34.271316+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:52:34.271316+0000","last_undegraded":"2023-10-30T13:52:34.271316+0000","last_fullsized":"2023-10-30T13:52:34.271316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'768","last_scrub_stamp":"2023-10-30T13:52:34.271229+0000","last_deep_scrub":"67'768","last_deep_scrub_stamp":"2023-10-30T13:52:34.271229+0000","last_clean_scrub_stamp":"2023-10-30T13:52:34.271229+0000","objects_scrubbed":64,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:00.506995+0000","scrub_duration":0.20155240999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749318,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":374,"num_read_kb":96620,"num_write":840,"num_write_kb":51020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":647968,"num_omap_keys":1098,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165182927,"num_objects":39302,"num_object_clones":18317,"num_object_copies":78604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39302,"num_whiteouts":14350,"num_read":6653,"num_read_kb":168434,"num_write":83476,"num_write_kb":1459733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424070144,"data_stored":2332381086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135714,"ondisk_log_size":135714,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":909135531,"num_objects":1701,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":11399,"num_read_kb":5963012,"num_write":34147,"num_write_kb":2074566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1411,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17841559,"num_omap_keys":30407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1818419200,"data_stored":1818271062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32051,"ondisk_log_size":32051,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575371,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743032,"kb_used_data":577200,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93628808,"statfs":{"total":96636764160,"available":95875899392,"internally_reserved":0,"allocated":591052800,"data_stored":577425546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":169802187},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.81000000000000005,"15min":0.91300000000000003},"min":{"1min":0.45100000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3080000000000001,"15min":15.381},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.78100000000000003,"15min":0.84599999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.315},"max":{"1min":1.05,"5min":1.8400000000000001,"15min":2.8079999999999998},"last":0.91100000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.80700000000000005,"15min":0.90600000000000003},"min":{"1min":0.51300000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1839999999999999,"5min":1.79,"15min":15.271000000000001},"last":0.877},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.87,"15min":0.95199999999999996},"min":{"1min":0.58199999999999996,"5min":0.46000000000000002,"15min":0.42999999999999999},"max":{"1min":1.2769999999999999,"5min":1.6000000000000001,"15min":15.228999999999999},"last":0.97399999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"min":{"1min":0.65000000000000002,"5min":0.45600000000000002,"15min":0.36299999999999999},"max":{"1min":1.123,"5min":1.345,"15min":2.2429999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.86699999999999999,"15min":0.86599999999999999},"min":{"1min":0.378,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4610000000000001,"5min":1.929,"15min":1.929},"last":0.82899999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.91800000000000004,"15min":1.056},"min":{"1min":0.54100000000000004,"5min":0.372,"15min":0.372},"max":{"1min":1.159,"5min":2.0249999999999999,"15min":15.324},"last":1.042},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.99399999999999999,"15min":0.98999999999999999},"min":{"1min":0.65400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.379,"5min":1.379,"15min":1.637},"last":1.075}]},{"osd":4,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.05,"15min":1.115},"min":{"1min":0.47999999999999998,"5min":0.433,"15min":0.40899999999999997},"max":{"1min":1.2929999999999999,"5min":2.7869999999999999,"15min":15.172000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0740000000000001,"15min":1.077},"min":{"1min":0.39700000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.6379999999999999,"5min":3.032,"15min":3.032},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.091,"15min":1.1000000000000001},"min":{"1min":0.57999999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.409,"5min":1.4770000000000001,"15min":1.7050000000000001},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.046,"15min":1.0109999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.554,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":1.012}]},{"osd":6,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1850000000000001,"15min":1.2470000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.8340000000000001,"5min":3.0350000000000001,"15min":15.427},"last":1.165},{"interface":"front","average":{"1min":1.266,"5min":1.24,"15min":1.3200000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.42199999999999999},"max":{"1min":1.948,"5min":3.5169999999999999,"15min":32.976999999999997},"last":1.24}]}]},{"osd":6,"up_from":13,"seq":55834575370,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587804,"kb_used_data":452564,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93784036,"statfs":{"total":96636764160,"available":96034852864,"internally_reserved":0,"allocated":463425536,"data_stored":454827877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138475975},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.82599999999999996,"15min":0.83999999999999997},"min":{"1min":0.53400000000000003,"5min":0.45000000000000001,"15min":0.40999999999999998},"max":{"1min":1.1890000000000001,"5min":1.3999999999999999,"15min":5.9880000000000004},"last":0.754},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.877,"15min":0.88600000000000001},"min":{"1min":0.63500000000000001,"5min":0.48599999999999999,"15min":0.35999999999999999},"max":{"1min":1.1519999999999999,"5min":1.3160000000000001,"15min":6.0730000000000004},"last":1.153}]},{"osd":1,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.90900000000000003,"15min":0.86899999999999999},"min":{"1min":0.54000000000000004,"5min":0.42199999999999999,"15min":0.41999999999999998},"max":{"1min":1.153,"5min":6.9930000000000003,"15min":6.9930000000000003},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.85199999999999998,"15min":0.85499999999999998},"min":{"1min":0.67500000000000004,"5min":0.52500000000000002,"15min":0.47499999999999998},"max":{"1min":1.0620000000000001,"5min":1.2410000000000001,"15min":1.9810000000000001},"last":0.92400000000000004}]},{"osd":2,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.91500000000000004,"15min":0.89700000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.47699999999999998},"max":{"1min":1.141,"5min":6.8550000000000004,"15min":6.8550000000000004},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.95599999999999996,"15min":0.96199999999999997},"min":{"1min":0.63400000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2110000000000001,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.84099999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.56000000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.4830000000000001,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":1.3},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.98099999999999998,"15min":0.98399999999999999},"min":{"1min":0.72699999999999998,"5min":0.64400000000000002,"15min":0.629},"max":{"1min":1.3100000000000001,"5min":5.3179999999999996,"15min":5.3179999999999996},"last":1.254}]},{"osd":4,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.98199999999999998,"15min":1.0369999999999999},"min":{"1min":0.752,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3129999999999999,"5min":1.4970000000000001,"15min":15.372},"last":1.075},{"interface":"front","average":{"1min":1.123,"5min":1.089,"15min":1.048},"min":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.58299999999999996},"max":{"1min":1.5820000000000001,"5min":5.2729999999999997,"15min":6.1580000000000004},"last":1.022}]},{"osd":5,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.036,"15min":1.032},"min":{"1min":0.69399999999999995,"5min":0.60499999999999998,"15min":0.60199999999999998},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.8080000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.234,"5min":1.198,"15min":1.1839999999999999},"min":{"1min":0.93999999999999995,"5min":0.73899999999999999,"15min":0.67700000000000005},"max":{"1min":1.6519999999999999,"5min":1.6639999999999999,"15min":1.913},"last":1.1259999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:52:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.1080000000000001,"15min":1.0980000000000001},"min":{"1min":0.86899999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":2.202},"last":0.873},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.22,"15min":1.2},"min":{"1min":0.99299999999999999,"5min":0.69599999999999995,"15min":0.42499999999999999},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":3.0590000000000002},"last":1.1879999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575371,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676788,"kb_used_data":519276,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93695052,"statfs":{"total":96636764160,"available":95943733248,"internally_reserved":0,"allocated":531738624,"data_stored":518585685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":161282502},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.88900000000000001,"15min":0.87},"min":{"1min":0.374,"5min":0.374,"15min":0.32700000000000001},"max":{"1min":1.2869999999999999,"5min":1.4379999999999999,"15min":6.4729999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":0.95099999999999996,"15min":0.88900000000000001},"min":{"1min":0.55400000000000005,"5min":0.53100000000000003,"15min":0.39100000000000001},"max":{"1min":3.585,"5min":3.585,"15min":8.7560000000000002},"last":0.97199999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.874,"15min":0.89100000000000001},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.42799999999999999},"max":{"1min":3.6360000000000001,"5min":3.6360000000000001,"15min":15.257999999999999},"last":1.056},{"interface":"front","average":{"1min":1.151,"5min":0.91900000000000004,"15min":0.96299999999999997},"min":{"1min":0.54200000000000004,"5min":0.52200000000000002,"15min":0.47399999999999998},"max":{"1min":4.0439999999999996,"5min":4.0439999999999996,"15min":15.175000000000001},"last":1.3300000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.90000000000000002,"15min":0.91700000000000004},"min":{"1min":0.503,"5min":0.503,"15min":0.42999999999999999},"max":{"1min":3.0179999999999998,"5min":3.0179999999999998,"15min":6.5410000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.997,"5min":0.94099999999999995,"15min":0.98899999999999999},"min":{"1min":0.56499999999999995,"5min":0.49099999999999999,"15min":0.46000000000000002},"max":{"1min":2.8879999999999999,"5min":2.8879999999999999,"15min":6.7080000000000002},"last":1.089}]},{"osd":3,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.087,"15min":1.048},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.54900000000000004},"max":{"1min":2.9340000000000002,"5min":5.4249999999999998,"15min":6.7699999999999996},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.171,"15min":1.1719999999999999},"min":{"1min":0.74399999999999999,"5min":0.58699999999999997,"15min":0.40100000000000002},"max":{"1min":2.9750000000000001,"5min":5.5570000000000004,"15min":15.364000000000001},"last":1.234}]},{"osd":4,"last update":"Mon Oct 30 13:52:49 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.94299999999999995,"15min":0.98499999999999999},"min":{"1min":0.621,"5min":0.376,"15min":0.36499999999999999},"max":{"1min":1.24,"5min":4.3529999999999998,"15min":5.9720000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2350000000000001,"15min":1.157},"min":{"1min":0.65900000000000003,"5min":0.56699999999999995,"15min":0.53000000000000003},"max":{"1min":1.395,"5min":5.6609999999999996,"15min":14.314},"last":0.94299999999999995}]},{"osd":6,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.0720000000000001,"15min":1.129},"min":{"1min":0.68100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":3.4279999999999999,"5min":4.2290000000000001,"15min":15.06},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.363,"5min":1.157,"15min":1.1899999999999999},"min":{"1min":0.79200000000000004,"5min":0.65700000000000003,"15min":0.50900000000000001},"max":{"1min":3.508,"5min":3.508,"15min":7.6630000000000003},"last":1.1359999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1.0289999999999999,"15min":1.105},"min":{"1min":0.46300000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.98099999999999998,"5min":5.0419999999999998,"15min":6.8259999999999996},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.208,"15min":1.2729999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.494,"5min":2.8250000000000002,"15min":19.321999999999999},"last":1.282}]}]},{"osd":4,"up_from":13,"seq":55834575372,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735176,"kb_used_data":569408,"kb_used_omap":1,"kb_used_meta":165758,"kb_avail":93636664,"statfs":{"total":96636764160,"available":95883943936,"internally_reserved":0,"allocated":583073792,"data_stored":568869284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169736646},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.751,"15min":0.79700000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.1839999999999999,"5min":2.202,"15min":9.6690000000000005},"last":0.874},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.81899999999999995,"15min":0.83399999999999996},"min":{"1min":0.48499999999999999,"5min":0.47699999999999998,"15min":0.377},"max":{"1min":1.206,"5min":2.0760000000000001,"15min":2.786},"last":0.82399999999999995}]},{"osd":1,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.84899999999999998,"15min":0.872},"min":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.41399999999999998},"max":{"1min":1.278,"5min":1.8160000000000001,"15min":5.5119999999999996},"last":0.748},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.88100000000000001,"15min":0.85899999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.44700000000000001},"max":{"1min":1.2110000000000001,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.89900000000000002,"15min":0.88400000000000001},"min":{"1min":0.58099999999999996,"5min":0.53700000000000003,"15min":0.48299999999999998},"max":{"1min":1.508,"5min":2.1549999999999998,"15min":9.0099999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.93300000000000005,"15min":0.94199999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.0939999999999999,"5min":7.7359999999999998,"15min":7.7359999999999998},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.93799999999999994,"15min":0.94699999999999995},"min":{"1min":0.63700000000000001,"5min":0.56200000000000006,"15min":0.52500000000000002},"max":{"1min":1.248,"5min":1.6000000000000001,"15min":4.7080000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.034,"15min":1.018},"min":{"1min":0.67900000000000005,"5min":0.66700000000000004,"15min":0.48699999999999999},"max":{"1min":1.153,"5min":1.7509999999999999,"15min":3.2749999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.0029999999999999,"15min":1.0409999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.29,"5min":1.3799999999999999,"15min":7.8529999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.212,"15min":1.1759999999999999},"min":{"1min":0.68200000000000005,"5min":0.65400000000000003,"15min":0.496},"max":{"1min":1.4350000000000001,"5min":7.694,"15min":8.2070000000000007},"last":0.77300000000000002}]},{"osd":6,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.125,"15min":1.083},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.52600000000000002},"max":{"1min":1.4510000000000001,"5min":7.7880000000000003,"15min":7.7880000000000003},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.087,"5min":1.1459999999999999,"15min":1.151},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.395,"5min":3.7320000000000002,"15min":5.6130000000000004},"last":0.97599999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:52:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.194,"15min":1.1759999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.59299999999999997},"max":{"1min":1.5109999999999999,"5min":7.8380000000000001,"15min":10.715},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.0369999999999999,"15min":1.147},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.47199999999999998},"max":{"1min":1.321,"5min":2.2450000000000001,"15min":4.3230000000000004},"last":0.72099999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575370,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737956,"kb_used_data":550044,"kb_used_omap":1,"kb_used_meta":187902,"kb_avail":93633884,"statfs":{"total":96636764160,"available":95881097216,"internally_reserved":0,"allocated":563245056,"data_stored":549099638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":192412104},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42699999999999999,"15min":0.40899999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.23000000000000001},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.45200000000000001,"15min":0.46800000000000003},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70899999999999996,"5min":0.76700000000000002,"15min":1.5429999999999999},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.504,"15min":0.46700000000000003},"min":{"1min":0.32400000000000001,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":0.65600000000000003,"5min":0.76200000000000001,"15min":0.94099999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.246},"max":{"1min":0.83099999999999996,"5min":0.92000000000000004,"15min":1.018},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.503,"15min":0.51000000000000001},"min":{"1min":0.35099999999999998,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.81599999999999995,"5min":1.056,"15min":1.4690000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86499999999999999,"15min":1.6799999999999999},"last":0.59399999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.78200000000000003,"15min":0.82499999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.97699999999999998,"5min":1.1739999999999999,"15min":1.5900000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83199999999999996,"15min":0.84599999999999997},"min":{"1min":0.56100000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1,"5min":1.1419999999999999,"15min":1.3999999999999999},"last":0.85499999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.76700000000000002,"15min":0.79600000000000004},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.51800000000000002},"max":{"1min":0.86299999999999999,"5min":1.1879999999999999,"15min":1.5069999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.81000000000000005,"15min":0.81899999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.5},"max":{"1min":0.95999999999999996,"5min":1.232,"15min":1.365},"last":0.93100000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.85199999999999998,"15min":0.82599999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45400000000000001},"max":{"1min":1.05,"5min":1.256,"15min":1.256},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.87,"15min":0.82899999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.498},"max":{"1min":1.0760000000000001,"5min":1.3069999999999999,"15min":2.3410000000000002},"last":0.95499999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.86799999999999999,"15min":0.876},"min":{"1min":0.61599999999999999,"5min":0.57799999999999996,"15min":0.51700000000000002},"max":{"1min":1.0569999999999999,"5min":1.3320000000000001,"15min":2.9449999999999998},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.86099999999999999,"15min":0.88200000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":1.0780000000000001,"5min":1.343,"15min":2.9729999999999999},"last":0.72499999999999998}]}]},{"osd":2,"up_from":13,"seq":55834575373,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511028,"kb_used_data":377260,"kb_used_omap":1,"kb_used_meta":133758,"kb_avail":93860812,"statfs":{"total":96636764160,"available":96113471488,"internally_reserved":0,"allocated":386314240,"data_stored":377764906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":136968646},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.54400000000000004,"15min":0.47599999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":9.8249999999999993,"5min":9.8249999999999993,"15min":9.8249999999999993},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53600000000000003,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.32100000000000001,"15min":0.25700000000000001},"max":{"1min":2.7200000000000002,"5min":4.2910000000000004,"15min":4.2910000000000004},"last":0.80000000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.64800000000000002,"15min":0.58799999999999997},"min":{"1min":0.38700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":9.7420000000000009,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58699999999999997,"15min":0.54200000000000004},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.2839999999999998,"5min":7.0060000000000002,"15min":7.0060000000000002},"last":0.629}]},{"osd":3,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.52900000000000003,"15min":0.504},"min":{"1min":0.371,"5min":0.35099999999999998,"15min":0.28299999999999997},"max":{"1min":2.8119999999999998,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.64500000000000002,"15min":0.56999999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.33700000000000002},"max":{"1min":2.843,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.72399999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79800000000000004,"15min":0.80300000000000005},"min":{"1min":0.56200000000000006,"5min":0.442,"15min":0.442},"max":{"1min":2.4129999999999998,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.37},"max":{"1min":2.681,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.66900000000000004}]},{"osd":5,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.89300000000000002,"15min":0.88200000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.50900000000000001},"max":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7989999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.79500000000000004,"15min":0.82499999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"max":{"1min":2.4750000000000001,"5min":2.4750000000000001,"15min":2.762},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.84499999999999997,"15min":0.81599999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.47699999999999998},"max":{"1min":9.7210000000000001,"5min":9.7210000000000001,"15min":9.7210000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.2050000000000001,"5min":0.91200000000000003,"15min":0.84599999999999997},"min":{"1min":0.622,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":9.7889999999999997,"5min":9.7889999999999997,"15min":9.7889999999999997},"last":0.749}]},{"osd":7,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":0.97299999999999998,"15min":0.875},"min":{"1min":0.629,"5min":0.58999999999999997,"15min":0.503},"max":{"1min":6.8529999999999998,"5min":6.9059999999999997,"15min":6.9059999999999997},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.0589999999999999,"15min":0.97099999999999997},"min":{"1min":0.64000000000000001,"5min":0.63700000000000001,"15min":0.55900000000000005},"max":{"1min":6.8929999999999998,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.82999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575370,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683292,"kb_used_data":535636,"kb_used_omap":1,"kb_used_meta":147646,"kb_avail":93688548,"statfs":{"total":96636764160,"available":95937073152,"internally_reserved":0,"allocated":548491264,"data_stored":539410681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151189958},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41899999999999998,"15min":0.42499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24299999999999999,"15min":0.22800000000000001},"max":{"1min":0.629,"5min":0.94999999999999996,"15min":1.1359999999999999},"last":0.496},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.28799999999999998,"5min":0.28299999999999997,"15min":0.251},"max":{"1min":0.72599999999999998,"5min":0.84099999999999997,"15min":0.96699999999999997},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47999999999999998,"15min":0.499},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.627,"5min":0.78000000000000003,"15min":2.8660000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.32800000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67900000000000005,"5min":1.0589999999999999,"15min":2.6739999999999999},"last":0.61199999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.314},"max":{"1min":0.70599999999999996,"5min":0.76900000000000002,"15min":1.2569999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57399999999999995},"min":{"1min":0.439,"5min":0.372,"15min":0.372},"max":{"1min":0.749,"5min":0.81399999999999995,"15min":2.71},"last":0.65800000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.81699999999999995,"15min":0.85199999999999998},"min":{"1min":0.52700000000000002,"5min":0.52400000000000002,"15min":0.40200000000000002},"max":{"1min":1.0389999999999999,"5min":1.2210000000000001,"15min":6.7789999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.88800000000000001,"15min":0.92100000000000004},"min":{"1min":0.59299999999999997,"5min":0.55200000000000005,"15min":0.53300000000000003},"max":{"1min":1.0900000000000001,"5min":1.2629999999999999,"15min":6.8259999999999996},"last":0.89200000000000002}]},{"osd":5,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.76400000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":2.133,"15min":2.5750000000000002},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.80700000000000005,"15min":0.80200000000000005},"min":{"1min":0.621,"5min":0.53800000000000003,"15min":0.51000000000000001},"max":{"1min":1.462,"5min":1.462,"15min":2.6179999999999999},"last":0.76000000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:52:46 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.91500000000000004,"15min":0.89700000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.501},"max":{"1min":1.357,"5min":1.3879999999999999,"15min":6.0289999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.95899999999999996,"15min":0.95399999999999996},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":6.0739999999999998},"last":0.93999999999999995}]},{"osd":7,"last update":"Mon Oct 30 13:52:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.80800000000000005,"15min":0.78100000000000003},"min":{"1min":0.70299999999999996,"5min":0.55000000000000004,"15min":0.47899999999999998},"max":{"1min":1.1319999999999999,"5min":1.161,"15min":1.163},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.86699999999999999,"15min":0.877},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.55200000000000005},"max":{"1min":1.153,"5min":1.3440000000000001,"15min":6.0519999999999996},"last":1.006}]}]},{"osd":0,"up_from":13,"seq":55834575372,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716656,"kb_used_data":568936,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93655184,"statfs":{"total":96636764160,"available":95902908416,"internally_reserved":0,"allocated":582590464,"data_stored":569154632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.40100000000000002,"15min":0.47599999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":0.54400000000000004,"5min":2.0630000000000002,"15min":16.184000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.50900000000000001,"15min":0.59099999999999997},"min":{"1min":0.36199999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.629,"5min":2.0979999999999999,"15min":16.228000000000002},"last":0.748}]},{"osd":2,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45300000000000001,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.70499999999999996,"5min":2.7999999999999998,"15min":17.454000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.46800000000000003,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":0.61799999999999999,"5min":1.391,"15min":23.212},"last":0.83899999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51100000000000001,"15min":0.62},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":1.4690000000000001,"15min":16.103999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52700000000000002,"15min":0.72399999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60199999999999998,"5min":2.8340000000000001,"15min":23.16},"last":0.88500000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:52:53 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.81899999999999995,"15min":0.91800000000000004},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":0.96399999999999997,"5min":7.6900000000000004,"15min":16.309000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.82099999999999995,"15min":0.95999999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97399999999999998,"5min":3.4009999999999998,"15min":16.343},"last":0.61699999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.81599999999999995,"15min":0.995},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94299999999999995,"5min":2.02,"15min":23.266999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.86599999999999999,"15min":1.0329999999999999},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.999,"5min":2.7240000000000002,"15min":16.268000000000001},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.79400000000000004,"15min":1.091},"min":{"1min":0.52500000000000002,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":1.0269999999999999,"5min":2.6560000000000001,"15min":29.774999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.80600000000000005,"15min":1.0149999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":2.75,"15min":17.5},"last":0.92000000000000004}]},{"osd":7,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.81699999999999995,"15min":1.1919999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.51300000000000001},"max":{"1min":1.0229999999999999,"5min":2.694,"15min":47.534999999999997},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":1.147},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.54700000000000004},"max":{"1min":1.272,"5min":2.8690000000000002,"15min":32.476999999999997},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230703104,"data_stored":230687380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258846720,"data_stored":258822445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179036160,"data_stored":179013521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230858752,"data_stored":230840931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":197275648,"data_stored":197263919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":205541376,"data_stored":205522290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":268918784,"data_stored":268902222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":247238656,"data_stored":247218354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350515200,"data_stored":337464605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288862208,"data_stored":280174938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":206495744,"data_stored":198338087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331603968,"data_stored":317845409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":385015808,"data_stored":371192067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":325414912,"data_stored":312650097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193724416,"data_stored":185512357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342437888,"data_stored":329203526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:53:38.500 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:38.500 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:38.501 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:38.501 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T13:10:33.664409+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=10, tm_sec=33, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=52, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:53:38.502 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:53:58.503 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:53:58.652 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.648+0000 7f169546b640 1 Processor -- start 2023-10-30T13:53:58.653 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.648+0000 7f169546b640 1 -- start start 2023-10-30T13:53:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f169546b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f169006f7a0 0x7f169006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f169546b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1690071110 0x7f1690158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f169546b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1690066cf0 con 0x7f16900c6bd0 2023-10-30T13:53:58.654 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f169546b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16900700c0 con 0x7f169006f7a0 2023-10-30T13:53:58.655 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f169546b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c70d0 con 0x7f1690071110 2023-10-30T13:53:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f168effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f169006f7a0 0x7f169006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f168e7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f16900c6bd0 0x7f1690069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.148:60414/0 (socket says 172.21.15.148:60414) 2023-10-30T13:53:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f168e7fc640 1 -- 172.21.15.148:0/3941813682 learned_addr learned my addr 172.21.15.148:0/3941813682 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:53:58.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f168f7fe640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1690071110 0x7f1690158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f168f7fe640 1 -- 172.21.15.148:0/3941813682 >> v2:172.21.15.150:3300/0 conn(0x7f169006f7a0 msgr2=0x7f169006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:58.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f168f7fe640 1 --2- 172.21.15.148:0/3941813682 >> v2:172.21.15.150:3300/0 conn(0x7f169006f7a0 0x7f169006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f168f7fe640 1 -- 172.21.15.148:0/3941813682 >> v1:172.21.15.150:6789/0 conn(0x7f16900c6bd0 legacy=0x7f1690069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.657 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f168f7fe640 1 -- 172.21.15.148:0/3941813682 --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900c7390 con 0x7f1690071110 2023-10-30T13:53:58.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.652+0000 7f168effd640 1 --2- 172.21.15.148:0/3941813682 >> v2:172.21.15.150:3300/0 conn(0x7f169006f7a0 0x7f169006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:53:58.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f168f7fe640 1 --2- 172.21.15.148:0/3941813682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1690071110 0x7f1690158110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1688009d90 tx=0x7f168801f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7af58aa6b0ef4fa5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:58.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f168dffb640 1 -- 172.21.15.148:0/3941813682 <== mon.2 v2:172.21.15.150:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16880af600 con 0x7f1690071110 2023-10-30T13:53:58.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f168dffb640 1 -- 172.21.15.148:0/3941813682 <== mon.2 v2:172.21.15.150:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16880ab050 con 0x7f1690071110 2023-10-30T13:53:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f168dffb640 1 -- 172.21.15.148:0/3941813682 <== mon.2 v2:172.21.15.150:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16880af980 con 0x7f1690071110 2023-10-30T13:53:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f169546b640 1 -- 172.21.15.148:0/3941813682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1690071110 msgr2=0x7f1690158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f169546b640 1 --2- 172.21.15.148:0/3941813682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1690071110 0x7f1690158110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1688009d90 tx=0x7f168801f130 comp rx=0 tx=0).stop 2023-10-30T13:53:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f169546b640 1 -- 172.21.15.148:0/3941813682 shutdown_connections 2023-10-30T13:53:58.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f169546b640 1 --2- 172.21.15.148:0/3941813682 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1690071110 0x7f1690158110 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f169546b640 1 --2- 172.21.15.148:0/3941813682 >> v2:172.21.15.150:3300/0 conn(0x7f169006f7a0 0x7f169006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f169546b640 1 -- 172.21.15.148:0/3941813682 >> 172.21.15.148:0/3941813682 conn(0x7f16900c7910 msgr2=0x7f16900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:58.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f169546b640 1 -- 172.21.15.148:0/3941813682 shutdown_connections 2023-10-30T13:53:58.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.656+0000 7f169546b640 1 -- 172.21.15.148:0/3941813682 wait complete. 2023-10-30T13:53:58.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 Processor -- start 2023-10-30T13:53:58.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 -- start start 2023-10-30T13:53:58.664 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f169006f7a0 0x7f16901de6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1690071110 0x7f1690157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16900c6bd0 0x7f16901f0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169014d780 con 0x7f1690071110 2023-10-30T13:53:58.665 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16901f0d80 con 0x7f16900c6bd0 2023-10-30T13:53:58.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16901e3330 con 0x7f169006f7a0 2023-10-30T13:53:58.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1690071110 0x7f1690157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1690071110 0x7f1690157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38988/0 (socket says 172.21.15.148:38988) 2023-10-30T13:53:58.666 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168e7fc640 1 -- 172.21.15.148:0/2428192177 learned_addr learned my addr 172.21.15.148:0/2428192177 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:53:58.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168f7fe640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16900c6bd0 0x7f16901f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168effd640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f169006f7a0 0x7f16901de6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.667 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168e7fc640 1 -- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f169006f7a0 msgr2=0x7f16901de6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168e7fc640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f169006f7a0 0x7f16901de6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168e7fc640 1 -- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16900c6bd0 msgr2=0x7f16901f0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168e7fc640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16900c6bd0 0x7f16901f0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.668 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168e7fc640 1 -- 172.21.15.148:0/2428192177 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680009420 con 0x7f1690071110 2023-10-30T13:53:58.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168f7fe640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16900c6bd0 0x7f16901f0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:58.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168effd640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f169006f7a0 0x7f16901de6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:58.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f168e7fc640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1690071110 0x7f1690157320 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f168400c4d0 tx=0x7f168400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:58.669 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f16777fe640 1 -- 172.21.15.148:0/2428192177 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684005cf0 con 0x7f1690071110 2023-10-30T13:53:58.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f16777fe640 1 -- 172.21.15.148:0/2428192177 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684005e50 con 0x7f1690071110 2023-10-30T13:53:58.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f16777fe640 1 -- 172.21.15.148:0/2428192177 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684007270 con 0x7f1690071110 2023-10-30T13:53:58.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1688009a70 con 0x7f1690071110 2023-10-30T13:53:58.670 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16900c4470 con 0x7f1690071110 2023-10-30T13:53:58.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.660+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169004e170 con 0x7f1690071110 2023-10-30T13:53:58.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.664+0000 7f16777fe640 1 -- 172.21.15.148:0/2428192177 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f16840073d0 con 0x7f1690071110 2023-10-30T13:53:58.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.664+0000 7f16777fe640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f167c0664a0 0x7f167c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:58.672 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.664+0000 7f16777fe640 1 -- 172.21.15.148:0/2428192177 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f1684083a20 con 0x7f1690071110 2023-10-30T13:53:58.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.664+0000 7f168effd640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f167c0664a0 0x7f167c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:58.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.664+0000 7f168effd640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f167c0664a0 0x7f167c068960 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f1680009ca0 tx=0x7f1680008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:58.673 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.668+0000 7f16777fe640 1 -- 172.21.15.148:0/2428192177 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f16840507b0 con 0x7f1690071110 2023-10-30T13:53:58.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.812+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 --> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16901e34a0 con 0x7f167c0664a0 2023-10-30T13:53:58.838 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.836+0000 7f16777fe640 1 -- 172.21.15.148:0/2428192177 <== mgr.4105 v2:172.21.15.150:6832/1299332652 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+296416 (secure 0 0 0) 0x7f16901e34a0 con 0x7f167c0664a0 2023-10-30T13:53:58.841 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:53:58.843 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2023-10-30T13:53:58.844 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f167c0664a0 msgr2=0x7f167c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f167c0664a0 0x7f167c068960 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f1680009ca0 tx=0x7f1680008040 comp rx=0 tx=0).stop 2023-10-30T13:53:58.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1690071110 msgr2=0x7f1690157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:58.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1690071110 0x7f1690157320 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f168400c4d0 tx=0x7f168400c990 comp rx=0 tx=0).stop 2023-10-30T13:53:58.845 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 shutdown_connections 2023-10-30T13:53:58.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f167c0664a0 0x7f167c068960 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1690071110 0x7f1690157320 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16900c6bd0 0x7f16901f0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.846 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 --2- 172.21.15.148:0/2428192177 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f169006f7a0 0x7f16901de6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:58.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 >> 172.21.15.148:0/2428192177 conn(0x7f16900c7910 msgr2=0x7f16900c8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:58.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 shutdown_connections 2023-10-30T13:53:58.847 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:58.840+0000 7f169546b640 1 -- 172.21.15.148:0/2428192177 wait complete. 2023-10-30T13:53:58.861 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":1367,"stamp":"2023-10-30T13:53:57.231597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2074908826,"num_objects":41005,"num_object_clones":18317,"num_object_copies":82010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41005,"num_whiteouts":14350,"num_read":18168,"num_read_kb":6131546,"num_write":117782,"num_write_kb":3536715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1411,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17841559,"num_omap_keys":30407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170102,"ondisk_log_size":170102,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5419380,"kb_used_data":4150324,"kb_used_omap":12,"kb_used_meta":1268979,"kb_avail":749555340,"statfs":{"total":773094113280,"available":767544668160,"internally_reserved":0,"allocated":4249931776,"data_stored":4155138249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":1299435064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004071"},"pg_stats":[{"pgid":"3.3f","version":"67'2042","reported_seq":4042,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:38.384345+0000","last_change":"2023-10-30T13:53:38.384417+0000","last_active":"2023-10-30T13:53:38.384345+0000","last_peered":"2023-10-30T13:53:38.384345+0000","last_clean":"2023-10-30T13:53:38.384345+0000","last_became_active":"2023-10-30T13:10:34.377531+0000","last_became_peered":"2023-10-30T13:10:34.377531+0000","last_unstale":"2023-10-30T13:53:38.384345+0000","last_undegraded":"2023-10-30T13:53:38.384345+0000","last_fullsized":"2023-10-30T13:53:38.384345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2042","last_scrub_stamp":"2023-10-30T13:53:38.384225+0000","last_deep_scrub":"67'2042","last_deep_scrub_stamp":"2023-10-30T13:53:38.384225+0000","last_clean_scrub_stamp":"2023-10-30T13:53:38.384225+0000","objects_scrubbed":569,"log_size":2042,"log_dups_size":0,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:37:16.749760+0000","scrub_duration":1.1132751320000001,"objects_trimmed":0,"snaptrim_duration":0.014806453000000001,"stat_sum":{"num_bytes":11269044,"num_objects":569,"num_object_clones":260,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":205,"num_read":107,"num_read_kb":121,"num_write":1253,"num_write_kb":17103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"67'2076","reported_seq":4032,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:18.542413+0000","last_change":"2023-10-30T13:53:18.542467+0000","last_active":"2023-10-30T13:53:18.542413+0000","last_peered":"2023-10-30T13:53:18.542413+0000","last_clean":"2023-10-30T13:53:18.542413+0000","last_became_active":"2023-10-30T13:10:34.323254+0000","last_became_peered":"2023-10-30T13:10:34.323254+0000","last_unstale":"2023-10-30T13:53:18.542413+0000","last_undegraded":"2023-10-30T13:53:18.542413+0000","last_fullsized":"2023-10-30T13:53:18.542413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2076","last_scrub_stamp":"2023-10-30T13:53:18.542317+0000","last_deep_scrub":"67'2076","last_deep_scrub_stamp":"2023-10-30T13:53:18.542317+0000","last_clean_scrub_stamp":"2023-10-30T13:53:18.542317+0000","objects_scrubbed":597,"log_size":2076,"log_dups_size":0,"ondisk_log_size":2076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:28:37.756101+0000","scrub_duration":0.79951518899999996,"objects_trimmed":0,"snaptrim_duration":0.0059691099999999997,"stat_sum":{"num_bytes":30975930,"num_objects":597,"num_object_clones":279,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":225,"num_read":82,"num_read_kb":30,"num_write":1226,"num_write_kb":32475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"67'2143","reported_seq":4123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.190609+0000","last_change":"2023-10-30T13:52:54.190676+0000","last_active":"2023-10-30T13:52:54.190609+0000","last_peered":"2023-10-30T13:52:54.190609+0000","last_clean":"2023-10-30T13:52:54.190609+0000","last_became_active":"2023-10-30T13:10:34.331857+0000","last_became_peered":"2023-10-30T13:10:34.331857+0000","last_unstale":"2023-10-30T13:52:54.190609+0000","last_undegraded":"2023-10-30T13:52:54.190609+0000","last_fullsized":"2023-10-30T13:52:54.190609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2143","last_scrub_stamp":"2023-10-30T13:52:54.190532+0000","last_deep_scrub":"67'2143","last_deep_scrub_stamp":"2023-10-30T13:52:54.190532+0000","last_clean_scrub_stamp":"2023-10-30T13:52:54.190532+0000","objects_scrubbed":617,"log_size":2143,"log_dups_size":0,"ondisk_log_size":2143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:27.035524+0000","scrub_duration":0.94181986699999998,"objects_trimmed":0,"snaptrim_duration":0.0025608710000000002,"stat_sum":{"num_bytes":18704388,"num_objects":617,"num_object_clones":279,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":225,"num_read":106,"num_read_kb":8300,"num_write":1300,"num_write_kb":27896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"67'2214","reported_seq":4185,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.682002+0000","last_change":"2023-10-30T13:52:59.682002+0000","last_active":"2023-10-30T13:52:59.682002+0000","last_peered":"2023-10-30T13:52:59.682002+0000","last_clean":"2023-10-30T13:52:59.682002+0000","last_became_active":"2023-10-30T13:11:15.435552+0000","last_became_peered":"2023-10-30T13:11:15.435552+0000","last_unstale":"2023-10-30T13:52:59.682002+0000","last_undegraded":"2023-10-30T13:52:59.682002+0000","last_fullsized":"2023-10-30T13:52:59.682002+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2214","last_scrub_stamp":"2023-10-30T13:52:59.681815+0000","last_deep_scrub":"67'2214","last_deep_scrub_stamp":"2023-10-30T13:52:59.681815+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.681815+0000","objects_scrubbed":630,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:52:49.252489+0000","scrub_duration":1.2152727269999999,"objects_trimmed":0,"snaptrim_duration":0.0021564890000000001,"stat_sum":{"num_bytes":23329933,"num_objects":630,"num_object_clones":293,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":227,"num_read":112,"num_read_kb":4378,"num_write":1354,"num_write_kb":25006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"67'2231","reported_seq":4187,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:38.545335+0000","last_change":"2023-10-30T13:52:38.545426+0000","last_active":"2023-10-30T13:52:38.545335+0000","last_peered":"2023-10-30T13:52:38.545335+0000","last_clean":"2023-10-30T13:52:38.545335+0000","last_became_active":"2023-10-30T13:10:34.370743+0000","last_became_peered":"2023-10-30T13:10:34.370743+0000","last_unstale":"2023-10-30T13:52:38.545335+0000","last_undegraded":"2023-10-30T13:52:38.545335+0000","last_fullsized":"2023-10-30T13:52:38.545335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2231","last_scrub_stamp":"2023-10-30T13:52:38.545206+0000","last_deep_scrub":"67'2231","last_deep_scrub_stamp":"2023-10-30T13:52:38.545206+0000","last_clean_scrub_stamp":"2023-10-30T13:52:38.545206+0000","objects_scrubbed":654,"log_size":2231,"log_dups_size":0,"ondisk_log_size":2231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:21:20.355710+0000","scrub_duration":1.430729712,"objects_trimmed":0,"snaptrim_duration":0.011808077,"stat_sum":{"num_bytes":16184828,"num_objects":654,"num_object_clones":305,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":251,"num_read":109,"num_read_kb":152,"num_write":1339,"num_write_kb":20879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'2052","reported_seq":4033,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:56.370945+0000","last_change":"2023-10-30T13:52:56.370945+0000","last_active":"2023-10-30T13:52:56.370945+0000","last_peered":"2023-10-30T13:52:56.370945+0000","last_clean":"2023-10-30T13:52:56.370945+0000","last_became_active":"2023-10-30T13:10:34.334841+0000","last_became_peered":"2023-10-30T13:10:34.334841+0000","last_unstale":"2023-10-30T13:52:56.370945+0000","last_undegraded":"2023-10-30T13:52:56.370945+0000","last_fullsized":"2023-10-30T13:52:56.370945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2052","last_scrub_stamp":"2023-10-30T13:52:56.370794+0000","last_deep_scrub":"67'2052","last_deep_scrub_stamp":"2023-10-30T13:52:56.370794+0000","last_clean_scrub_stamp":"2023-10-30T13:52:56.370794+0000","objects_scrubbed":593,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:21.000907+0000","scrub_duration":1.065003666,"objects_trimmed":0,"snaptrim_duration":0.01102786,"stat_sum":{"num_bytes":18372801,"num_objects":593,"num_object_clones":277,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":222,"num_read":108,"num_read_kb":4137,"num_write":1217,"num_write_kb":19358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"67'2113","reported_seq":4084,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:24.134174+0000","last_change":"2023-10-30T13:53:24.134280+0000","last_active":"2023-10-30T13:53:24.134174+0000","last_peered":"2023-10-30T13:53:24.134174+0000","last_clean":"2023-10-30T13:53:24.134174+0000","last_became_active":"2023-10-30T13:10:34.371007+0000","last_became_peered":"2023-10-30T13:10:34.371007+0000","last_unstale":"2023-10-30T13:53:24.134174+0000","last_undegraded":"2023-10-30T13:53:24.134174+0000","last_fullsized":"2023-10-30T13:53:24.134174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2113","last_scrub_stamp":"2023-10-30T13:53:24.134014+0000","last_deep_scrub":"67'2113","last_deep_scrub_stamp":"2023-10-30T13:53:24.134014+0000","last_clean_scrub_stamp":"2023-10-30T13:53:24.134014+0000","objects_scrubbed":598,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:24:00.966228+0000","scrub_duration":1.0382753499999999,"objects_trimmed":0,"snaptrim_duration":0.012047337,"stat_sum":{"num_bytes":9859762,"num_objects":598,"num_object_clones":275,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":211,"num_read":96,"num_read_kb":35,"num_write":1295,"num_write_kb":11866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'2037","reported_seq":4026,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.982168+0000","last_change":"2023-10-30T13:52:51.982168+0000","last_active":"2023-10-30T13:52:51.982168+0000","last_peered":"2023-10-30T13:52:51.982168+0000","last_clean":"2023-10-30T13:52:51.982168+0000","last_became_active":"2023-10-30T13:10:34.347867+0000","last_became_peered":"2023-10-30T13:10:34.347867+0000","last_unstale":"2023-10-30T13:52:51.982168+0000","last_undegraded":"2023-10-30T13:52:51.982168+0000","last_fullsized":"2023-10-30T13:52:51.982168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2037","last_scrub_stamp":"2023-10-30T13:52:51.982035+0000","last_deep_scrub":"67'2037","last_deep_scrub_stamp":"2023-10-30T13:52:51.982035+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.982035+0000","objects_scrubbed":577,"log_size":2037,"log_dups_size":0,"ondisk_log_size":2037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:07:47.358388+0000","scrub_duration":1.116769473,"objects_trimmed":0,"snaptrim_duration":0.0063254060000000004,"stat_sum":{"num_bytes":13694423,"num_objects":577,"num_object_clones":268,"num_object_copies":1154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":211,"num_read":108,"num_read_kb":104,"num_write":1228,"num_write_kb":23250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"67'2150","reported_seq":4126,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:39.363990+0000","last_change":"2023-10-30T13:53:39.364044+0000","last_active":"2023-10-30T13:53:39.363990+0000","last_peered":"2023-10-30T13:53:39.363990+0000","last_clean":"2023-10-30T13:53:39.363990+0000","last_became_active":"2023-10-30T13:10:34.378294+0000","last_became_peered":"2023-10-30T13:10:34.378294+0000","last_unstale":"2023-10-30T13:53:39.363990+0000","last_undegraded":"2023-10-30T13:53:39.363990+0000","last_fullsized":"2023-10-30T13:53:39.363990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2150","last_scrub_stamp":"2023-10-30T13:53:39.363893+0000","last_deep_scrub":"67'2150","last_deep_scrub_stamp":"2023-10-30T13:53:39.363893+0000","last_clean_scrub_stamp":"2023-10-30T13:53:39.363893+0000","objects_scrubbed":622,"log_size":2150,"log_dups_size":0,"ondisk_log_size":2150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:58:17.544333+0000","scrub_duration":1.088209669,"objects_trimmed":0,"snaptrim_duration":0.014360304000000001,"stat_sum":{"num_bytes":19134308,"num_objects":622,"num_object_clones":284,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":234,"num_read":109,"num_read_kb":4185,"num_write":1275,"num_write_kb":24930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"67'2111","reported_seq":4082,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:24.064933+0000","last_change":"2023-10-30T13:53:24.065003+0000","last_active":"2023-10-30T13:53:24.064933+0000","last_peered":"2023-10-30T13:53:24.064933+0000","last_clean":"2023-10-30T13:53:24.064933+0000","last_became_active":"2023-10-30T13:10:34.356010+0000","last_became_peered":"2023-10-30T13:10:34.356010+0000","last_unstale":"2023-10-30T13:53:24.064933+0000","last_undegraded":"2023-10-30T13:53:24.064933+0000","last_fullsized":"2023-10-30T13:53:24.064933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2111","last_scrub_stamp":"2023-10-30T13:53:24.064851+0000","last_deep_scrub":"67'2111","last_deep_scrub_stamp":"2023-10-30T13:53:24.064851+0000","last_clean_scrub_stamp":"2023-10-30T13:53:24.064851+0000","objects_scrubbed":610,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:13:12.208328+0000","scrub_duration":1.1952141599999999,"objects_trimmed":0,"snaptrim_duration":0.010277041000000001,"stat_sum":{"num_bytes":19693407,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":208,"num_read":105,"num_read_kb":2976,"num_write":1294,"num_write_kb":29513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"67'2172","reported_seq":4133,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:19.900333+0000","last_change":"2023-10-30T13:53:19.900470+0000","last_active":"2023-10-30T13:53:19.900333+0000","last_peered":"2023-10-30T13:53:19.900333+0000","last_clean":"2023-10-30T13:53:19.900333+0000","last_became_active":"2023-10-30T13:11:15.446386+0000","last_became_peered":"2023-10-30T13:11:15.446386+0000","last_unstale":"2023-10-30T13:53:19.900333+0000","last_undegraded":"2023-10-30T13:53:19.900333+0000","last_fullsized":"2023-10-30T13:53:19.900333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2172","last_scrub_stamp":"2023-10-30T13:53:19.900196+0000","last_deep_scrub":"67'2172","last_deep_scrub_stamp":"2023-10-30T13:53:19.900196+0000","last_clean_scrub_stamp":"2023-10-30T13:53:19.900196+0000","objects_scrubbed":636,"log_size":2172,"log_dups_size":0,"ondisk_log_size":2172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:36.711893+0000","scrub_duration":0.95872091000000004,"objects_trimmed":0,"snaptrim_duration":0.01048763,"stat_sum":{"num_bytes":11727836,"num_objects":636,"num_object_clones":292,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":230,"num_read":95,"num_read_kb":27,"num_write":1294,"num_write_kb":13174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"67'2168","reported_seq":4127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:55.137033+0000","last_change":"2023-10-30T13:52:55.137145+0000","last_active":"2023-10-30T13:52:55.137033+0000","last_peered":"2023-10-30T13:52:55.137033+0000","last_clean":"2023-10-30T13:52:55.137033+0000","last_became_active":"2023-10-30T13:10:34.364293+0000","last_became_peered":"2023-10-30T13:10:34.364293+0000","last_unstale":"2023-10-30T13:52:55.137033+0000","last_undegraded":"2023-10-30T13:52:55.137033+0000","last_fullsized":"2023-10-30T13:52:55.137033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2168","last_scrub_stamp":"2023-10-30T13:52:55.136932+0000","last_deep_scrub":"67'2168","last_deep_scrub_stamp":"2023-10-30T13:52:55.136932+0000","last_clean_scrub_stamp":"2023-10-30T13:52:55.136932+0000","objects_scrubbed":634,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:10.926178+0000","scrub_duration":1.2701765250000001,"objects_trimmed":0,"snaptrim_duration":0.0085295030000000008,"stat_sum":{"num_bytes":15445695,"num_objects":634,"num_object_clones":293,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":231,"num_read":101,"num_read_kb":102,"num_write":1284,"num_write_kb":21892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"67'2081","reported_seq":4056,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:11.778193+0000","last_change":"2023-10-30T13:53:11.778257+0000","last_active":"2023-10-30T13:53:11.778193+0000","last_peered":"2023-10-30T13:53:11.778193+0000","last_clean":"2023-10-30T13:53:11.778193+0000","last_became_active":"2023-10-30T13:10:34.351695+0000","last_became_peered":"2023-10-30T13:10:34.351695+0000","last_unstale":"2023-10-30T13:53:11.778193+0000","last_undegraded":"2023-10-30T13:53:11.778193+0000","last_fullsized":"2023-10-30T13:53:11.778193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:53:11.778101+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:53:11.778101+0000","last_clean_scrub_stamp":"2023-10-30T13:53:11.778101+0000","objects_scrubbed":596,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:53:38.043584+0000","scrub_duration":0.84397403100000001,"objects_trimmed":0,"snaptrim_duration":0.004933977,"stat_sum":{"num_bytes":6328355,"num_objects":596,"num_object_clones":272,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":210,"num_read":97,"num_read_kb":4194,"num_write":1284,"num_write_kb":15784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"67'2163","reported_seq":4144,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:46.206549+0000","last_change":"2023-10-30T13:52:46.206609+0000","last_active":"2023-10-30T13:52:46.206549+0000","last_peered":"2023-10-30T13:52:46.206549+0000","last_clean":"2023-10-30T13:52:46.206549+0000","last_became_active":"2023-10-30T13:10:34.323102+0000","last_became_peered":"2023-10-30T13:10:34.323102+0000","last_unstale":"2023-10-30T13:52:46.206549+0000","last_undegraded":"2023-10-30T13:52:46.206549+0000","last_fullsized":"2023-10-30T13:52:46.206549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2163","last_scrub_stamp":"2023-10-30T13:52:46.206474+0000","last_deep_scrub":"67'2163","last_deep_scrub_stamp":"2023-10-30T13:52:46.206474+0000","last_clean_scrub_stamp":"2023-10-30T13:52:46.206474+0000","objects_scrubbed":610,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:31.417387+0000","scrub_duration":1.043425775,"objects_trimmed":0,"snaptrim_duration":0.0029909680000000001,"stat_sum":{"num_bytes":31670070,"num_objects":610,"num_object_clones":280,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":227,"num_read":110,"num_read_kb":16437,"num_write":1314,"num_write_kb":33491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"67'2183","reported_seq":4156,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:05.806848+0000","last_change":"2023-10-30T13:53:05.806946+0000","last_active":"2023-10-30T13:53:05.806848+0000","last_peered":"2023-10-30T13:53:05.806848+0000","last_clean":"2023-10-30T13:53:05.806848+0000","last_became_active":"2023-10-30T13:10:34.359775+0000","last_became_peered":"2023-10-30T13:10:34.359775+0000","last_unstale":"2023-10-30T13:53:05.806848+0000","last_undegraded":"2023-10-30T13:53:05.806848+0000","last_fullsized":"2023-10-30T13:53:05.806848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2183","last_scrub_stamp":"2023-10-30T13:53:05.806777+0000","last_deep_scrub":"67'2183","last_deep_scrub_stamp":"2023-10-30T13:53:05.806777+0000","last_clean_scrub_stamp":"2023-10-30T13:53:05.806777+0000","objects_scrubbed":613,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:17.084832+0000","scrub_duration":1.1056300050000001,"objects_trimmed":0,"snaptrim_duration":0.0058939090000000001,"stat_sum":{"num_bytes":14342306,"num_objects":613,"num_object_clones":284,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":231,"num_read":106,"num_read_kb":2953,"num_write":1333,"num_write_kb":15925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"67'2459","reported_seq":4355,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:46.948880+0000","last_change":"2023-10-30T13:52:46.948942+0000","last_active":"2023-10-30T13:52:46.948880+0000","last_peered":"2023-10-30T13:52:46.948880+0000","last_clean":"2023-10-30T13:52:46.948880+0000","last_became_active":"2023-10-30T13:10:34.332904+0000","last_became_peered":"2023-10-30T13:10:34.332904+0000","last_unstale":"2023-10-30T13:52:46.948880+0000","last_undegraded":"2023-10-30T13:52:46.948880+0000","last_fullsized":"2023-10-30T13:52:46.948880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2459","last_scrub_stamp":"2023-10-30T13:52:46.948803+0000","last_deep_scrub":"67'2459","last_deep_scrub_stamp":"2023-10-30T13:52:46.948803+0000","last_clean_scrub_stamp":"2023-10-30T13:52:46.948803+0000","objects_scrubbed":703,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:36:20.176163+0000","scrub_duration":1.765876443,"objects_trimmed":0,"snaptrim_duration":0.0093243890000000006,"stat_sum":{"num_bytes":10965780,"num_objects":703,"num_object_clones":374,"num_object_copies":1406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":231,"num_read":115,"num_read_kb":1694,"num_write":1434,"num_write_kb":17364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"67'2223","reported_seq":4184,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.397041+0000","last_change":"2023-10-30T13:52:51.397109+0000","last_active":"2023-10-30T13:52:51.397041+0000","last_peered":"2023-10-30T13:52:51.397041+0000","last_clean":"2023-10-30T13:52:51.397041+0000","last_became_active":"2023-10-30T13:10:34.371355+0000","last_became_peered":"2023-10-30T13:10:34.371355+0000","last_unstale":"2023-10-30T13:52:51.397041+0000","last_undegraded":"2023-10-30T13:52:51.397041+0000","last_fullsized":"2023-10-30T13:52:51.397041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2223","last_scrub_stamp":"2023-10-30T13:52:51.396947+0000","last_deep_scrub":"67'2223","last_deep_scrub_stamp":"2023-10-30T13:52:51.396947+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.396947+0000","objects_scrubbed":632,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:49.530995+0000","scrub_duration":1.1860778729999999,"objects_trimmed":0,"snaptrim_duration":0.011284880000000001,"stat_sum":{"num_bytes":18442682,"num_objects":632,"num_object_clones":291,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":228,"num_read":100,"num_read_kb":8212,"num_write":1340,"num_write_kb":24815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"67'2051","reported_seq":4022,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:58.672924+0000","last_change":"2023-10-30T13:52:58.673009+0000","last_active":"2023-10-30T13:52:58.672924+0000","last_peered":"2023-10-30T13:52:58.672924+0000","last_clean":"2023-10-30T13:52:58.672924+0000","last_became_active":"2023-10-30T13:10:34.352485+0000","last_became_peered":"2023-10-30T13:10:34.352485+0000","last_unstale":"2023-10-30T13:52:58.672924+0000","last_undegraded":"2023-10-30T13:52:58.672924+0000","last_fullsized":"2023-10-30T13:52:58.672924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2051","last_scrub_stamp":"2023-10-30T13:52:58.672817+0000","last_deep_scrub":"67'2051","last_deep_scrub_stamp":"2023-10-30T13:52:58.672817+0000","last_clean_scrub_stamp":"2023-10-30T13:52:58.672817+0000","objects_scrubbed":575,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:13:12.046899+0000","scrub_duration":1.1758492060000001,"objects_trimmed":0,"snaptrim_duration":0.002295727,"stat_sum":{"num_bytes":22547199,"num_objects":575,"num_object_clones":267,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":207,"num_read":89,"num_read_kb":4127,"num_write":1240,"num_write_kb":24211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"67'2279","reported_seq":4233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:22.456694+0000","last_change":"2023-10-30T13:53:22.456770+0000","last_active":"2023-10-30T13:53:22.456694+0000","last_peered":"2023-10-30T13:53:22.456694+0000","last_clean":"2023-10-30T13:53:22.456694+0000","last_became_active":"2023-10-30T13:11:15.435745+0000","last_became_peered":"2023-10-30T13:11:15.435745+0000","last_unstale":"2023-10-30T13:53:22.456694+0000","last_undegraded":"2023-10-30T13:53:22.456694+0000","last_fullsized":"2023-10-30T13:53:22.456694+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2279","last_scrub_stamp":"2023-10-30T13:53:22.456585+0000","last_deep_scrub":"67'2279","last_deep_scrub_stamp":"2023-10-30T13:53:22.456585+0000","last_clean_scrub_stamp":"2023-10-30T13:53:22.456585+0000","objects_scrubbed":649,"log_size":2279,"log_dups_size":0,"ondisk_log_size":2279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:44:13.665995+0000","scrub_duration":0.75823801199999996,"objects_trimmed":0,"snaptrim_duration":0.0061689539999999999,"stat_sum":{"num_bytes":11703479,"num_objects":649,"num_object_clones":303,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":237,"num_read":106,"num_read_kb":7307,"num_write":1383,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2c","version":"67'2189","reported_seq":4162,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:04.864418+0000","last_change":"2023-10-30T13:53:04.864418+0000","last_active":"2023-10-30T13:53:04.864418+0000","last_peered":"2023-10-30T13:53:04.864418+0000","last_clean":"2023-10-30T13:53:04.864418+0000","last_became_active":"2023-10-30T13:10:34.365764+0000","last_became_peered":"2023-10-30T13:10:34.365764+0000","last_unstale":"2023-10-30T13:53:04.864418+0000","last_undegraded":"2023-10-30T13:53:04.864418+0000","last_fullsized":"2023-10-30T13:53:04.864418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2189","last_scrub_stamp":"2023-10-30T13:53:04.864251+0000","last_deep_scrub":"67'2189","last_deep_scrub_stamp":"2023-10-30T13:53:04.864251+0000","last_clean_scrub_stamp":"2023-10-30T13:53:04.864251+0000","objects_scrubbed":621,"log_size":2189,"log_dups_size":0,"ondisk_log_size":2189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:46.599510+0000","scrub_duration":1.113390213,"objects_trimmed":0,"snaptrim_duration":0.0094394720000000008,"stat_sum":{"num_bytes":31771338,"num_objects":621,"num_object_clones":283,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":229,"num_read":103,"num_read_kb":4126,"num_write":1340,"num_write_kb":42793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"67'2081","reported_seq":4058,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.440653+0000","last_change":"2023-10-30T13:52:49.440653+0000","last_active":"2023-10-30T13:52:49.440653+0000","last_peered":"2023-10-30T13:52:49.440653+0000","last_clean":"2023-10-30T13:52:49.440653+0000","last_became_active":"2023-10-30T13:10:34.368494+0000","last_became_peered":"2023-10-30T13:10:34.368494+0000","last_unstale":"2023-10-30T13:52:49.440653+0000","last_undegraded":"2023-10-30T13:52:49.440653+0000","last_fullsized":"2023-10-30T13:52:49.440653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:52:49.440530+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:52:49.440530+0000","last_clean_scrub_stamp":"2023-10-30T13:52:49.440530+0000","objects_scrubbed":595,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:04.396223+0000","scrub_duration":1.3029579039999999,"objects_trimmed":0,"snaptrim_duration":0.0027436349999999999,"stat_sum":{"num_bytes":6210074,"num_objects":595,"num_object_clones":273,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":228,"num_read":98,"num_read_kb":33,"num_write":1261,"num_write_kb":10653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"67'2305","reported_seq":4251,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.423196+0000","last_change":"2023-10-30T13:52:45.423196+0000","last_active":"2023-10-30T13:52:45.423196+0000","last_peered":"2023-10-30T13:52:45.423196+0000","last_clean":"2023-10-30T13:52:45.423196+0000","last_became_active":"2023-10-30T13:10:34.372144+0000","last_became_peered":"2023-10-30T13:10:34.372144+0000","last_unstale":"2023-10-30T13:52:45.423196+0000","last_undegraded":"2023-10-30T13:52:45.423196+0000","last_fullsized":"2023-10-30T13:52:45.423196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2305","last_scrub_stamp":"2023-10-30T13:52:45.423033+0000","last_deep_scrub":"67'2305","last_deep_scrub_stamp":"2023-10-30T13:52:45.423033+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.423033+0000","objects_scrubbed":670,"log_size":2305,"log_dups_size":0,"ondisk_log_size":2305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:55:15.141666+0000","scrub_duration":1.276721341,"objects_trimmed":0,"snaptrim_duration":0.0082328149999999992,"stat_sum":{"num_bytes":28398973,"num_objects":670,"num_object_clones":311,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":670,"num_whiteouts":248,"num_read":102,"num_read_kb":120,"num_write":1396,"num_write_kb":38562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'2154","reported_seq":4126,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:08.009726+0000","last_change":"2023-10-30T13:53:08.009776+0000","last_active":"2023-10-30T13:53:08.009726+0000","last_peered":"2023-10-30T13:53:08.009726+0000","last_clean":"2023-10-30T13:53:08.009726+0000","last_became_active":"2023-10-30T13:10:34.359046+0000","last_became_peered":"2023-10-30T13:10:34.359046+0000","last_unstale":"2023-10-30T13:53:08.009726+0000","last_undegraded":"2023-10-30T13:53:08.009726+0000","last_fullsized":"2023-10-30T13:53:08.009726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2154","last_scrub_stamp":"2023-10-30T13:53:08.009651+0000","last_deep_scrub":"67'2154","last_deep_scrub_stamp":"2023-10-30T13:53:08.009651+0000","last_clean_scrub_stamp":"2023-10-30T13:53:08.009651+0000","objects_scrubbed":610,"log_size":2154,"log_dups_size":0,"ondisk_log_size":2154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:41:21.317083+0000","scrub_duration":1.0714371359999999,"objects_trimmed":0,"snaptrim_duration":0.0037447470000000001,"stat_sum":{"num_bytes":19407388,"num_objects":610,"num_object_clones":284,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":225,"num_read":103,"num_read_kb":4138,"num_write":1300,"num_write_kb":24714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'2202","reported_seq":4177,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:08.361856+0000","last_change":"2023-10-30T13:53:08.361983+0000","last_active":"2023-10-30T13:53:08.361856+0000","last_peered":"2023-10-30T13:53:08.361856+0000","last_clean":"2023-10-30T13:53:08.361856+0000","last_became_active":"2023-10-30T13:10:34.346635+0000","last_became_peered":"2023-10-30T13:10:34.346635+0000","last_unstale":"2023-10-30T13:53:08.361856+0000","last_undegraded":"2023-10-30T13:53:08.361856+0000","last_fullsized":"2023-10-30T13:53:08.361856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2202","last_scrub_stamp":"2023-10-30T13:53:08.361660+0000","last_deep_scrub":"67'2202","last_deep_scrub_stamp":"2023-10-30T13:53:08.361660+0000","last_clean_scrub_stamp":"2023-10-30T13:53:08.361660+0000","objects_scrubbed":616,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:03.069890+0000","scrub_duration":1.1435550349999999,"objects_trimmed":0,"snaptrim_duration":0.0091633449999999998,"stat_sum":{"num_bytes":10552296,"num_objects":616,"num_object_clones":294,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":228,"num_read":115,"num_read_kb":24,"num_write":1342,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"67'1977","reported_seq":3967,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:20.526287+0000","last_change":"2023-10-30T13:53:20.526336+0000","last_active":"2023-10-30T13:53:20.526287+0000","last_peered":"2023-10-30T13:53:20.526287+0000","last_clean":"2023-10-30T13:53:20.526287+0000","last_became_active":"2023-10-30T13:11:15.440207+0000","last_became_peered":"2023-10-30T13:11:15.440207+0000","last_unstale":"2023-10-30T13:53:20.526287+0000","last_undegraded":"2023-10-30T13:53:20.526287+0000","last_fullsized":"2023-10-30T13:53:20.526287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1977","last_scrub_stamp":"2023-10-30T13:53:20.526216+0000","last_deep_scrub":"67'1977","last_deep_scrub_stamp":"2023-10-30T13:53:20.526216+0000","last_clean_scrub_stamp":"2023-10-30T13:53:20.526216+0000","objects_scrubbed":556,"log_size":1977,"log_dups_size":0,"ondisk_log_size":1977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:33:49.058832+0000","scrub_duration":0.847919223,"objects_trimmed":0,"snaptrim_duration":0.0059368210000000001,"stat_sum":{"num_bytes":15196336,"num_objects":556,"num_object_clones":256,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":202,"num_read":99,"num_read_kb":1380,"num_write":1203,"num_write_kb":16993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'2204","reported_seq":4186,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:25.736195+0000","last_change":"2023-10-30T13:53:25.736280+0000","last_active":"2023-10-30T13:53:25.736195+0000","last_peered":"2023-10-30T13:53:25.736195+0000","last_clean":"2023-10-30T13:53:25.736195+0000","last_became_active":"2023-10-30T13:10:34.379374+0000","last_became_peered":"2023-10-30T13:10:34.379374+0000","last_unstale":"2023-10-30T13:53:25.736195+0000","last_undegraded":"2023-10-30T13:53:25.736195+0000","last_fullsized":"2023-10-30T13:53:25.736195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2204","last_scrub_stamp":"2023-10-30T13:53:25.736081+0000","last_deep_scrub":"67'2204","last_deep_scrub_stamp":"2023-10-30T13:53:25.736081+0000","last_clean_scrub_stamp":"2023-10-30T13:53:25.736081+0000","objects_scrubbed":619,"log_size":2204,"log_dups_size":0,"ondisk_log_size":2204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:14:55.998934+0000","scrub_duration":0.77972822399999997,"objects_trimmed":0,"snaptrim_duration":0.0099065669999999998,"stat_sum":{"num_bytes":9983325,"num_objects":619,"num_object_clones":286,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":221,"num_read":115,"num_read_kb":68,"num_write":1330,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"67'2157","reported_seq":4133,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:16.541747+0000","last_change":"2023-10-30T13:53:16.541808+0000","last_active":"2023-10-30T13:53:16.541747+0000","last_peered":"2023-10-30T13:53:16.541747+0000","last_clean":"2023-10-30T13:53:16.541747+0000","last_became_active":"2023-10-30T13:10:34.331746+0000","last_became_peered":"2023-10-30T13:10:34.331746+0000","last_unstale":"2023-10-30T13:53:16.541747+0000","last_undegraded":"2023-10-30T13:53:16.541747+0000","last_fullsized":"2023-10-30T13:53:16.541747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2157","last_scrub_stamp":"2023-10-30T13:53:16.541670+0000","last_deep_scrub":"67'2157","last_deep_scrub_stamp":"2023-10-30T13:53:16.541670+0000","last_clean_scrub_stamp":"2023-10-30T13:53:16.541670+0000","objects_scrubbed":614,"log_size":2157,"log_dups_size":0,"ondisk_log_size":2157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:40:07.326667+0000","scrub_duration":0.80191293600000002,"objects_trimmed":0,"snaptrim_duration":0.0059382619999999997,"stat_sum":{"num_bytes":23775876,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":236,"num_read":113,"num_read_kb":4145,"num_write":1301,"num_write_kb":26615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"67'2078","reported_seq":4040,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:03.067704+0000","last_change":"2023-10-30T13:53:03.067770+0000","last_active":"2023-10-30T13:53:03.067704+0000","last_peered":"2023-10-30T13:53:03.067704+0000","last_clean":"2023-10-30T13:53:03.067704+0000","last_became_active":"2023-10-30T13:10:34.362858+0000","last_became_peered":"2023-10-30T13:10:34.362858+0000","last_unstale":"2023-10-30T13:53:03.067704+0000","last_undegraded":"2023-10-30T13:53:03.067704+0000","last_fullsized":"2023-10-30T13:53:03.067704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2078","last_scrub_stamp":"2023-10-30T13:53:03.067613+0000","last_deep_scrub":"67'2078","last_deep_scrub_stamp":"2023-10-30T13:53:03.067613+0000","last_clean_scrub_stamp":"2023-10-30T13:53:03.067613+0000","objects_scrubbed":600,"log_size":2078,"log_dups_size":0,"ondisk_log_size":2078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:58.122248+0000","scrub_duration":1.2008048499999999,"objects_trimmed":0,"snaptrim_duration":0.0053787089999999997,"stat_sum":{"num_bytes":10833053,"num_objects":600,"num_object_clones":279,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":213,"num_read":88,"num_read_kb":3300,"num_write":1256,"num_write_kb":16238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"67'2337","reported_seq":4272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:55.080264+0000","last_change":"2023-10-30T13:52:55.080329+0000","last_active":"2023-10-30T13:52:55.080264+0000","last_peered":"2023-10-30T13:52:55.080264+0000","last_clean":"2023-10-30T13:52:55.080264+0000","last_became_active":"2023-10-30T13:10:34.331882+0000","last_became_peered":"2023-10-30T13:10:34.331882+0000","last_unstale":"2023-10-30T13:52:55.080264+0000","last_undegraded":"2023-10-30T13:52:55.080264+0000","last_fullsized":"2023-10-30T13:52:55.080264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2337","last_scrub_stamp":"2023-10-30T13:52:55.080176+0000","last_deep_scrub":"67'2337","last_deep_scrub_stamp":"2023-10-30T13:52:55.080176+0000","last_clean_scrub_stamp":"2023-10-30T13:52:55.080176+0000","objects_scrubbed":688,"log_size":2337,"log_dups_size":0,"ondisk_log_size":2337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:15:48.148329+0000","scrub_duration":0.84244230799999997,"objects_trimmed":0,"snaptrim_duration":0.0028849399999999999,"stat_sum":{"num_bytes":11995550,"num_objects":688,"num_object_clones":321,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":244,"num_read":100,"num_read_kb":8267,"num_write":1414,"num_write_kb":14306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"67'2013","reported_seq":3996,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:52.418032+0000","last_change":"2023-10-30T13:52:52.418138+0000","last_active":"2023-10-30T13:52:52.418032+0000","last_peered":"2023-10-30T13:52:52.418032+0000","last_clean":"2023-10-30T13:52:52.418032+0000","last_became_active":"2023-10-30T13:10:34.336823+0000","last_became_peered":"2023-10-30T13:10:34.336823+0000","last_unstale":"2023-10-30T13:52:52.418032+0000","last_undegraded":"2023-10-30T13:52:52.418032+0000","last_fullsized":"2023-10-30T13:52:52.418032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2013","last_scrub_stamp":"2023-10-30T13:52:52.417929+0000","last_deep_scrub":"67'2013","last_deep_scrub_stamp":"2023-10-30T13:52:52.417929+0000","last_clean_scrub_stamp":"2023-10-30T13:52:52.417929+0000","objects_scrubbed":558,"log_size":2013,"log_dups_size":0,"ondisk_log_size":2013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:39:32.467346+0000","scrub_duration":1.1806941150000001,"objects_trimmed":0,"snaptrim_duration":0.0096547909999999994,"stat_sum":{"num_bytes":15074071,"num_objects":558,"num_object_clones":259,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":221,"num_read":91,"num_read_kb":32,"num_write":1204,"num_write_kb":16754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"67'2322","reported_seq":4298,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:57.949237+0000","last_change":"2023-10-30T13:52:57.949288+0000","last_active":"2023-10-30T13:52:57.949237+0000","last_peered":"2023-10-30T13:52:57.949237+0000","last_clean":"2023-10-30T13:52:57.949237+0000","last_became_active":"2023-10-30T13:10:34.359119+0000","last_became_peered":"2023-10-30T13:10:34.359119+0000","last_unstale":"2023-10-30T13:52:57.949237+0000","last_undegraded":"2023-10-30T13:52:57.949237+0000","last_fullsized":"2023-10-30T13:52:57.949237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2322","last_scrub_stamp":"2023-10-30T13:52:57.949159+0000","last_deep_scrub":"67'2322","last_deep_scrub_stamp":"2023-10-30T13:52:57.949159+0000","last_clean_scrub_stamp":"2023-10-30T13:52:57.949159+0000","objects_scrubbed":649,"log_size":2322,"log_dups_size":0,"ondisk_log_size":2322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:27:26.563664+0000","scrub_duration":1.454754868,"objects_trimmed":0,"snaptrim_duration":0.0018845820000000001,"stat_sum":{"num_bytes":10798567,"num_objects":649,"num_object_clones":300,"num_object_copies":1298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":241,"num_read":120,"num_read_kb":27,"num_write":1443,"num_write_kb":17235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"67'2091","reported_seq":4094,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:12.954329+0000","last_change":"2023-10-30T13:53:12.954423+0000","last_active":"2023-10-30T13:53:12.954329+0000","last_peered":"2023-10-30T13:53:12.954329+0000","last_clean":"2023-10-30T13:53:12.954329+0000","last_became_active":"2023-10-30T13:10:34.343918+0000","last_became_peered":"2023-10-30T13:10:34.343918+0000","last_unstale":"2023-10-30T13:53:12.954329+0000","last_undegraded":"2023-10-30T13:53:12.954329+0000","last_fullsized":"2023-10-30T13:53:12.954329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2091","last_scrub_stamp":"2023-10-30T13:53:12.954208+0000","last_deep_scrub":"67'2091","last_deep_scrub_stamp":"2023-10-30T13:53:12.954208+0000","last_clean_scrub_stamp":"2023-10-30T13:53:12.954208+0000","objects_scrubbed":576,"log_size":2091,"log_dups_size":0,"ondisk_log_size":2091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:45.742650+0000","scrub_duration":1.0988574289999999,"objects_trimmed":0,"snaptrim_duration":0.0075239010000000004,"stat_sum":{"num_bytes":15423893,"num_objects":576,"num_object_clones":259,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":203,"num_read":109,"num_read_kb":9552,"num_write":1308,"num_write_kb":16780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"67'2174","reported_seq":4153,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:47.980889+0000","last_change":"2023-10-30T13:52:47.980946+0000","last_active":"2023-10-30T13:52:47.980889+0000","last_peered":"2023-10-30T13:52:47.980889+0000","last_clean":"2023-10-30T13:52:47.980889+0000","last_became_active":"2023-10-30T13:10:34.361891+0000","last_became_peered":"2023-10-30T13:10:34.361891+0000","last_unstale":"2023-10-30T13:52:47.980889+0000","last_undegraded":"2023-10-30T13:52:47.980889+0000","last_fullsized":"2023-10-30T13:52:47.980889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2174","last_scrub_stamp":"2023-10-30T13:52:47.980802+0000","last_deep_scrub":"67'2174","last_deep_scrub_stamp":"2023-10-30T13:52:47.980802+0000","last_clean_scrub_stamp":"2023-10-30T13:52:47.980802+0000","objects_scrubbed":622,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:54:24.281648+0000","scrub_duration":1.0857265140000001,"objects_trimmed":0,"snaptrim_duration":0.007356301,"stat_sum":{"num_bytes":40089180,"num_objects":622,"num_object_clones":287,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":230,"num_read":115,"num_read_kb":33,"num_write":1312,"num_write_kb":42997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"67'644","reported_seq":883,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.019766+0000","last_change":"2023-10-30T13:52:45.019766+0000","last_active":"2023-10-30T13:52:45.019766+0000","last_peered":"2023-10-30T13:52:45.019766+0000","last_clean":"2023-10-30T13:52:45.019766+0000","last_became_active":"2023-10-30T13:10:33.714227+0000","last_became_peered":"2023-10-30T13:10:33.714227+0000","last_unstale":"2023-10-30T13:52:45.019766+0000","last_undegraded":"2023-10-30T13:52:45.019766+0000","last_fullsized":"2023-10-30T13:52:45.019766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'644","last_scrub_stamp":"2023-10-30T13:52:45.019575+0000","last_deep_scrub":"67'644","last_deep_scrub_stamp":"2023-10-30T13:52:45.019575+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.019575+0000","objects_scrubbed":42,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:11:59.207979+0000","scrub_duration":0.065999268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":252,"num_read_kb":249243,"num_write":662,"num_write_kb":74846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":549256,"num_omap_keys":974,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"67'2188","reported_seq":4150,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:13.306061+0000","last_change":"2023-10-30T13:53:13.306155+0000","last_active":"2023-10-30T13:53:13.306061+0000","last_peered":"2023-10-30T13:53:13.306061+0000","last_clean":"2023-10-30T13:53:13.306061+0000","last_became_active":"2023-10-30T13:10:34.360680+0000","last_became_peered":"2023-10-30T13:10:34.360680+0000","last_unstale":"2023-10-30T13:53:13.306061+0000","last_undegraded":"2023-10-30T13:53:13.306061+0000","last_fullsized":"2023-10-30T13:53:13.306061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2188","last_scrub_stamp":"2023-10-30T13:53:13.305925+0000","last_deep_scrub":"67'2188","last_deep_scrub_stamp":"2023-10-30T13:53:13.305925+0000","last_clean_scrub_stamp":"2023-10-30T13:53:13.305925+0000","objects_scrubbed":628,"log_size":2188,"log_dups_size":0,"ondisk_log_size":2188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:47.133988+0000","scrub_duration":1.0555437249999999,"objects_trimmed":0,"snaptrim_duration":0.0031372290000000001,"stat_sum":{"num_bytes":18713903,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":229,"num_read":108,"num_read_kb":4667,"num_write":1329,"num_write_kb":20689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"67'1176","reported_seq":1511,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.939106+0000","last_change":"2023-10-30T13:52:43.939106+0000","last_active":"2023-10-30T13:52:43.939106+0000","last_peered":"2023-10-30T13:52:43.939106+0000","last_clean":"2023-10-30T13:52:43.939106+0000","last_became_active":"2023-10-30T13:10:33.714836+0000","last_became_peered":"2023-10-30T13:10:33.714836+0000","last_unstale":"2023-10-30T13:52:43.939106+0000","last_undegraded":"2023-10-30T13:52:43.939106+0000","last_fullsized":"2023-10-30T13:52:43.939106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1176","last_scrub_stamp":"2023-10-30T13:52:43.938938+0000","last_deep_scrub":"67'1176","last_deep_scrub_stamp":"2023-10-30T13:52:43.938938+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.938938+0000","objects_scrubbed":51,"log_size":1176,"log_dups_size":0,"ondisk_log_size":1176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:18.962998+0000","scrub_duration":0.19282650300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25293488,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":404,"num_read_kb":309242,"num_write":1231,"num_write_kb":84663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":763115,"num_omap_keys":1100,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"67'794","reported_seq":1183,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:35.070009+0000","last_change":"2023-10-30T13:52:35.070059+0000","last_active":"2023-10-30T13:52:35.070009+0000","last_peered":"2023-10-30T13:52:35.070009+0000","last_clean":"2023-10-30T13:52:35.070009+0000","last_became_active":"2023-10-30T13:10:33.713070+0000","last_became_peered":"2023-10-30T13:10:33.713070+0000","last_unstale":"2023-10-30T13:52:35.070009+0000","last_undegraded":"2023-10-30T13:52:35.070009+0000","last_fullsized":"2023-10-30T13:52:35.070009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'794","last_scrub_stamp":"2023-10-30T13:52:35.069918+0000","last_deep_scrub":"67'794","last_deep_scrub_stamp":"2023-10-30T13:52:35.069918+0000","last_clean_scrub_stamp":"2023-10-30T13:52:35.069918+0000","objects_scrubbed":54,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:21.898394+0000","scrub_duration":0.14371538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397151,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":403,"num_read_kb":153383,"num_write":820,"num_write_kb":55113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":519482,"num_omap_keys":897,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"67'2508","reported_seq":4463,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:50.977937+0000","last_change":"2023-10-30T13:52:50.978041+0000","last_active":"2023-10-30T13:52:50.977937+0000","last_peered":"2023-10-30T13:52:50.977937+0000","last_clean":"2023-10-30T13:52:50.977937+0000","last_became_active":"2023-10-30T13:10:34.329461+0000","last_became_peered":"2023-10-30T13:10:34.329461+0000","last_unstale":"2023-10-30T13:52:50.977937+0000","last_undegraded":"2023-10-30T13:52:50.977937+0000","last_fullsized":"2023-10-30T13:52:50.977937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2508","last_scrub_stamp":"2023-10-30T13:52:50.977836+0000","last_deep_scrub":"67'2508","last_deep_scrub_stamp":"2023-10-30T13:52:50.977836+0000","last_clean_scrub_stamp":"2023-10-30T13:52:50.977836+0000","objects_scrubbed":729,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:22:26.785649+0000","scrub_duration":1.0834898019999999,"objects_trimmed":0,"snaptrim_duration":0.0064075410000000001,"stat_sum":{"num_bytes":11666119,"num_objects":729,"num_object_clones":341,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":261,"num_read":141,"num_read_kb":4175,"num_write":1520,"num_write_kb":22588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"67'1448","reported_seq":1728,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.318938+0000","last_change":"2023-10-30T13:52:43.318938+0000","last_active":"2023-10-30T13:52:43.318938+0000","last_peered":"2023-10-30T13:52:43.318938+0000","last_clean":"2023-10-30T13:52:43.318938+0000","last_became_active":"2023-10-30T13:10:33.710778+0000","last_became_peered":"2023-10-30T13:10:33.710778+0000","last_unstale":"2023-10-30T13:52:43.318938+0000","last_undegraded":"2023-10-30T13:52:43.318938+0000","last_fullsized":"2023-10-30T13:52:43.318938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1447","last_scrub_stamp":"2023-10-30T13:52:43.318712+0000","last_deep_scrub":"67'1447","last_deep_scrub_stamp":"2023-10-30T13:52:43.318712+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.318712+0000","objects_scrubbed":54,"log_size":1448,"log_dups_size":0,"ondisk_log_size":1448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:38.717163+0000","scrub_duration":0.18293932199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":354,"num_read_kb":270680,"num_write":1488,"num_write_kb":75460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593681,"num_omap_keys":1037,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"67'2128","reported_seq":4095,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:12.713494+0000","last_change":"2023-10-30T13:53:12.713584+0000","last_active":"2023-10-30T13:53:12.713494+0000","last_peered":"2023-10-30T13:53:12.713494+0000","last_clean":"2023-10-30T13:53:12.713494+0000","last_became_active":"2023-10-30T13:10:34.331119+0000","last_became_peered":"2023-10-30T13:10:34.331119+0000","last_unstale":"2023-10-30T13:53:12.713494+0000","last_undegraded":"2023-10-30T13:53:12.713494+0000","last_fullsized":"2023-10-30T13:53:12.713494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2128","last_scrub_stamp":"2023-10-30T13:53:12.713409+0000","last_deep_scrub":"67'2128","last_deep_scrub_stamp":"2023-10-30T13:53:12.713409+0000","last_clean_scrub_stamp":"2023-10-30T13:53:12.713409+0000","objects_scrubbed":614,"log_size":2128,"log_dups_size":0,"ondisk_log_size":2128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:10.476233+0000","scrub_duration":1.058017481,"objects_trimmed":0,"snaptrim_duration":0.0058964639999999997,"stat_sum":{"num_bytes":19507019,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":240,"num_read":105,"num_read_kb":5362,"num_write":1250,"num_write_kb":24752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"67'1012","reported_seq":1259,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:02.942744+0000","last_change":"2023-10-30T13:53:02.942802+0000","last_active":"2023-10-30T13:53:02.942744+0000","last_peered":"2023-10-30T13:53:02.942744+0000","last_clean":"2023-10-30T13:53:02.942744+0000","last_became_active":"2023-10-30T13:10:33.724348+0000","last_became_peered":"2023-10-30T13:10:33.724348+0000","last_unstale":"2023-10-30T13:53:02.942744+0000","last_undegraded":"2023-10-30T13:53:02.942744+0000","last_fullsized":"2023-10-30T13:53:02.942744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1012","last_scrub_stamp":"2023-10-30T13:53:02.942656+0000","last_deep_scrub":"67'1012","last_deep_scrub_stamp":"2023-10-30T13:53:02.942656+0000","last_clean_scrub_stamp":"2023-10-30T13:53:02.942656+0000","objects_scrubbed":56,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:31.384045+0000","scrub_duration":0.195907044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360218,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":313,"num_read_kb":141404,"num_write":1079,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":564647,"num_omap_keys":1018,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"67'2081","reported_seq":4067,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:00.832604+0000","last_change":"2023-10-30T13:53:00.832674+0000","last_active":"2023-10-30T13:53:00.832604+0000","last_peered":"2023-10-30T13:53:00.832604+0000","last_clean":"2023-10-30T13:53:00.832604+0000","last_became_active":"2023-10-30T13:11:15.529842+0000","last_became_peered":"2023-10-30T13:11:15.529842+0000","last_unstale":"2023-10-30T13:53:00.832604+0000","last_undegraded":"2023-10-30T13:53:00.832604+0000","last_fullsized":"2023-10-30T13:53:00.832604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2081","last_scrub_stamp":"2023-10-30T13:53:00.832511+0000","last_deep_scrub":"67'2081","last_deep_scrub_stamp":"2023-10-30T13:53:00.832511+0000","last_clean_scrub_stamp":"2023-10-30T13:53:00.832511+0000","objects_scrubbed":588,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:07:11.916144+0000","scrub_duration":1.374556924,"objects_trimmed":0,"snaptrim_duration":0.0025817129999999998,"stat_sum":{"num_bytes":23539910,"num_objects":588,"num_object_clones":267,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":208,"num_read":106,"num_read_kb":8313,"num_write":1266,"num_write_kb":25326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"67'1170","reported_seq":1398,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:39.386385+0000","last_change":"2023-10-30T13:52:39.386488+0000","last_active":"2023-10-30T13:52:39.386385+0000","last_peered":"2023-10-30T13:52:39.386385+0000","last_clean":"2023-10-30T13:52:39.386385+0000","last_became_active":"2023-10-30T13:10:33.716834+0000","last_became_peered":"2023-10-30T13:10:33.716834+0000","last_unstale":"2023-10-30T13:52:39.386385+0000","last_undegraded":"2023-10-30T13:52:39.386385+0000","last_fullsized":"2023-10-30T13:52:39.386385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1170","last_scrub_stamp":"2023-10-30T13:52:39.386286+0000","last_deep_scrub":"67'1170","last_deep_scrub_stamp":"2023-10-30T13:52:39.386286+0000","last_clean_scrub_stamp":"2023-10-30T13:52:39.386286+0000","objects_scrubbed":49,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:00.965375+0000","scrub_duration":0.15933681099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":240,"num_read_kb":201173,"num_write":1196,"num_write_kb":66729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":467606,"num_omap_keys":916,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"67'2185","reported_seq":4130,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:12.335263+0000","last_change":"2023-10-30T13:53:12.335354+0000","last_active":"2023-10-30T13:53:12.335263+0000","last_peered":"2023-10-30T13:53:12.335263+0000","last_clean":"2023-10-30T13:53:12.335263+0000","last_became_active":"2023-10-30T13:10:34.334948+0000","last_became_peered":"2023-10-30T13:10:34.334948+0000","last_unstale":"2023-10-30T13:53:12.335263+0000","last_undegraded":"2023-10-30T13:53:12.335263+0000","last_fullsized":"2023-10-30T13:53:12.335263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2185","last_scrub_stamp":"2023-10-30T13:53:12.335127+0000","last_deep_scrub":"67'2185","last_deep_scrub_stamp":"2023-10-30T13:53:12.335127+0000","last_clean_scrub_stamp":"2023-10-30T13:53:12.335127+0000","objects_scrubbed":628,"log_size":2185,"log_dups_size":0,"ondisk_log_size":2185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:20:26.571198+0000","scrub_duration":1.1178727850000001,"objects_trimmed":0,"snaptrim_duration":0.0022874710000000001,"stat_sum":{"num_bytes":16137402,"num_objects":628,"num_object_clones":297,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":233,"num_read":91,"num_read_kb":34,"num_write":1322,"num_write_kb":18013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"67'1358","reported_seq":1609,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.996716+0000","last_change":"2023-10-30T13:52:53.996816+0000","last_active":"2023-10-30T13:52:53.996716+0000","last_peered":"2023-10-30T13:52:53.996716+0000","last_clean":"2023-10-30T13:52:53.996716+0000","last_became_active":"2023-10-30T13:10:33.921435+0000","last_became_peered":"2023-10-30T13:10:33.921435+0000","last_unstale":"2023-10-30T13:52:53.996716+0000","last_undegraded":"2023-10-30T13:52:53.996716+0000","last_fullsized":"2023-10-30T13:52:53.996716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1358","last_scrub_stamp":"2023-10-30T13:52:53.996599+0000","last_deep_scrub":"67'1358","last_deep_scrub_stamp":"2023-10-30T13:52:53.996599+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.996599+0000","objects_scrubbed":45,"log_size":1358,"log_dups_size":0,"ondisk_log_size":1358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:02:58.575120+0000","scrub_duration":0.17104654599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20972192,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":307,"num_read_kb":165804,"num_write":1420,"num_write_kb":50966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":509116,"num_omap_keys":950,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'2207","reported_seq":4174,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.120386+0000","last_change":"2023-10-30T13:52:54.120454+0000","last_active":"2023-10-30T13:52:54.120386+0000","last_peered":"2023-10-30T13:52:54.120386+0000","last_clean":"2023-10-30T13:52:54.120386+0000","last_became_active":"2023-10-30T13:10:34.380267+0000","last_became_peered":"2023-10-30T13:10:34.380267+0000","last_unstale":"2023-10-30T13:52:54.120386+0000","last_undegraded":"2023-10-30T13:52:54.120386+0000","last_fullsized":"2023-10-30T13:52:54.120386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2207","last_scrub_stamp":"2023-10-30T13:52:54.120300+0000","last_deep_scrub":"67'2207","last_deep_scrub_stamp":"2023-10-30T13:52:54.120300+0000","last_clean_scrub_stamp":"2023-10-30T13:52:54.120300+0000","objects_scrubbed":638,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:39:00.511615+0000","scrub_duration":1.2561105379999999,"objects_trimmed":0,"snaptrim_duration":0.0087798259999999993,"stat_sum":{"num_bytes":32677996,"num_objects":638,"num_object_clones":293,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":232,"num_read":108,"num_read_kb":10951,"num_write":1342,"num_write_kb":38819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"67'986","reported_seq":1316,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.944325+0000","last_change":"2023-10-30T13:52:51.944325+0000","last_active":"2023-10-30T13:52:51.944325+0000","last_peered":"2023-10-30T13:52:51.944325+0000","last_clean":"2023-10-30T13:52:51.944325+0000","last_became_active":"2023-10-30T13:10:33.716839+0000","last_became_peered":"2023-10-30T13:10:33.716839+0000","last_unstale":"2023-10-30T13:52:51.944325+0000","last_undegraded":"2023-10-30T13:52:51.944325+0000","last_fullsized":"2023-10-30T13:52:51.944325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'985","last_scrub_stamp":"2023-10-30T13:52:51.944137+0000","last_deep_scrub":"67'985","last_deep_scrub_stamp":"2023-10-30T13:52:51.944137+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.944137+0000","objects_scrubbed":50,"log_size":986,"log_dups_size":0,"ondisk_log_size":986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:09:12.000749+0000","scrub_duration":0.18964769300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":413,"num_read_kb":281405,"num_write":1060,"num_write_kb":80203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":643541,"num_omap_keys":878,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"67'2126","reported_seq":4076,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:28.206398+0000","last_change":"2023-10-30T13:53:28.206516+0000","last_active":"2023-10-30T13:53:28.206398+0000","last_peered":"2023-10-30T13:53:28.206398+0000","last_clean":"2023-10-30T13:53:28.206398+0000","last_became_active":"2023-10-30T13:10:34.370840+0000","last_became_peered":"2023-10-30T13:10:34.370840+0000","last_unstale":"2023-10-30T13:53:28.206398+0000","last_undegraded":"2023-10-30T13:53:28.206398+0000","last_fullsized":"2023-10-30T13:53:28.206398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2126","last_scrub_stamp":"2023-10-30T13:53:28.206307+0000","last_deep_scrub":"67'2126","last_deep_scrub_stamp":"2023-10-30T13:53:28.206307+0000","last_clean_scrub_stamp":"2023-10-30T13:53:28.206307+0000","objects_scrubbed":614,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:13:17.846078+0000","scrub_duration":1.132694131,"objects_trimmed":0,"snaptrim_duration":0.014787140000000001,"stat_sum":{"num_bytes":19072740,"num_objects":614,"num_object_clones":289,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":238,"num_read":87,"num_read_kb":102,"num_write":1252,"num_write_kb":21862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"67'1144","reported_seq":1482,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:37.361974+0000","last_change":"2023-10-30T13:52:37.361974+0000","last_active":"2023-10-30T13:52:37.361974+0000","last_peered":"2023-10-30T13:52:37.361974+0000","last_clean":"2023-10-30T13:52:37.361974+0000","last_became_active":"2023-10-30T13:10:33.694737+0000","last_became_peered":"2023-10-30T13:10:33.694737+0000","last_unstale":"2023-10-30T13:52:37.361974+0000","last_undegraded":"2023-10-30T13:52:37.361974+0000","last_fullsized":"2023-10-30T13:52:37.361974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1144","last_scrub_stamp":"2023-10-30T13:52:37.361775+0000","last_deep_scrub":"67'1144","last_deep_scrub_stamp":"2023-10-30T13:52:37.361775+0000","last_clean_scrub_stamp":"2023-10-30T13:52:37.361775+0000","objects_scrubbed":60,"log_size":1144,"log_dups_size":0,"ondisk_log_size":1144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:47:58.622750+0000","scrub_duration":0.22738265299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748770,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":476,"num_read_kb":203988,"num_write":1202,"num_write_kb":67245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":644581,"num_omap_keys":1115,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"67'2090","reported_seq":4074,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.327640+0000","last_change":"2023-10-30T13:52:53.327719+0000","last_active":"2023-10-30T13:52:53.327640+0000","last_peered":"2023-10-30T13:52:53.327640+0000","last_clean":"2023-10-30T13:52:53.327640+0000","last_became_active":"2023-10-30T13:10:34.368495+0000","last_became_peered":"2023-10-30T13:10:34.368495+0000","last_unstale":"2023-10-30T13:52:53.327640+0000","last_undegraded":"2023-10-30T13:52:53.327640+0000","last_fullsized":"2023-10-30T13:52:53.327640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2090","last_scrub_stamp":"2023-10-30T13:52:53.327572+0000","last_deep_scrub":"67'2090","last_deep_scrub_stamp":"2023-10-30T13:52:53.327572+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.327572+0000","objects_scrubbed":584,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:03.787789+0000","scrub_duration":1.1150200729999999,"objects_trimmed":0,"snaptrim_duration":0.0024076599999999998,"stat_sum":{"num_bytes":14342228,"num_objects":584,"num_object_clones":268,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":211,"num_read":101,"num_read_kb":4159,"num_write":1284,"num_write_kb":24386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"67'616","reported_seq":808,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.351480+0000","last_change":"2023-10-30T13:52:40.351638+0000","last_active":"2023-10-30T13:52:40.351480+0000","last_peered":"2023-10-30T13:52:40.351480+0000","last_clean":"2023-10-30T13:52:40.351480+0000","last_became_active":"2023-10-30T13:10:33.707542+0000","last_became_peered":"2023-10-30T13:10:33.707542+0000","last_unstale":"2023-10-30T13:52:40.351480+0000","last_undegraded":"2023-10-30T13:52:40.351480+0000","last_fullsized":"2023-10-30T13:52:40.351480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'616","last_scrub_stamp":"2023-10-30T13:52:40.351385+0000","last_deep_scrub":"67'616","last_deep_scrub_stamp":"2023-10-30T13:52:40.351385+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.351385+0000","objects_scrubbed":45,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:05:25.211671+0000","scrub_duration":0.16031920099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":203,"num_read_kb":83137,"num_write":663,"num_write_kb":45991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":437141,"num_omap_keys":850,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"67'2238","reported_seq":4249,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:33.395905+0000","last_change":"2023-10-30T13:53:33.395971+0000","last_active":"2023-10-30T13:53:33.395905+0000","last_peered":"2023-10-30T13:53:33.395905+0000","last_clean":"2023-10-30T13:53:33.395905+0000","last_became_active":"2023-10-30T13:10:34.372308+0000","last_became_peered":"2023-10-30T13:10:34.372308+0000","last_unstale":"2023-10-30T13:53:33.395905+0000","last_undegraded":"2023-10-30T13:53:33.395905+0000","last_fullsized":"2023-10-30T13:53:33.395905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2238","last_scrub_stamp":"2023-10-30T13:53:33.395821+0000","last_deep_scrub":"67'2238","last_deep_scrub_stamp":"2023-10-30T13:53:33.395821+0000","last_clean_scrub_stamp":"2023-10-30T13:53:33.395821+0000","objects_scrubbed":634,"log_size":2238,"log_dups_size":0,"ondisk_log_size":2238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:18:57.009369+0000","scrub_duration":1.2166736549999999,"objects_trimmed":0,"snaptrim_duration":0.011943199999999999,"stat_sum":{"num_bytes":18918293,"num_objects":634,"num_object_clones":295,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":241,"num_read":154,"num_read_kb":114,"num_write":1356,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"67'557","reported_seq":801,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:57.004659+0000","last_change":"2023-10-30T13:52:57.004722+0000","last_active":"2023-10-30T13:52:57.004659+0000","last_peered":"2023-10-30T13:52:57.004659+0000","last_clean":"2023-10-30T13:52:57.004659+0000","last_became_active":"2023-10-30T13:10:33.723541+0000","last_became_peered":"2023-10-30T13:10:33.723541+0000","last_unstale":"2023-10-30T13:52:57.004659+0000","last_undegraded":"2023-10-30T13:52:57.004659+0000","last_fullsized":"2023-10-30T13:52:57.004659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'557","last_scrub_stamp":"2023-10-30T13:52:57.004551+0000","last_deep_scrub":"67'557","last_deep_scrub_stamp":"2023-10-30T13:52:57.004551+0000","last_clean_scrub_stamp":"2023-10-30T13:52:57.004551+0000","objects_scrubbed":57,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:29:17.022207+0000","scrub_duration":0.20230519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":289,"num_read_kb":121284,"num_write":557,"num_write_kb":58696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":599967,"num_omap_keys":1105,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"67'2295","reported_seq":4212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:19.344683+0000","last_change":"2023-10-30T13:53:19.344746+0000","last_active":"2023-10-30T13:53:19.344683+0000","last_peered":"2023-10-30T13:53:19.344683+0000","last_clean":"2023-10-30T13:53:19.344683+0000","last_became_active":"2023-10-30T13:10:34.341813+0000","last_became_peered":"2023-10-30T13:10:34.341813+0000","last_unstale":"2023-10-30T13:53:19.344683+0000","last_undegraded":"2023-10-30T13:53:19.344683+0000","last_fullsized":"2023-10-30T13:53:19.344683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2295","last_scrub_stamp":"2023-10-30T13:53:19.344588+0000","last_deep_scrub":"67'2295","last_deep_scrub_stamp":"2023-10-30T13:53:19.344588+0000","last_clean_scrub_stamp":"2023-10-30T13:53:19.344588+0000","objects_scrubbed":655,"log_size":2295,"log_dups_size":0,"ondisk_log_size":2295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:21:59.259100+0000","scrub_duration":1.411775386,"objects_trimmed":0,"snaptrim_duration":0.0086123929999999994,"stat_sum":{"num_bytes":19792070,"num_objects":655,"num_object_clones":349,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":204,"num_read":113,"num_read_kb":142,"num_write":1348,"num_write_kb":24080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"67'1016","reported_seq":1307,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:49.650095+0000","last_change":"2023-10-30T13:52:49.650144+0000","last_active":"2023-10-30T13:52:49.650095+0000","last_peered":"2023-10-30T13:52:49.650095+0000","last_clean":"2023-10-30T13:52:49.650095+0000","last_became_active":"2023-10-30T13:10:33.697691+0000","last_became_peered":"2023-10-30T13:10:33.697691+0000","last_unstale":"2023-10-30T13:52:49.650095+0000","last_undegraded":"2023-10-30T13:52:49.650095+0000","last_fullsized":"2023-10-30T13:52:49.650095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1016","last_scrub_stamp":"2023-10-30T13:52:49.650007+0000","last_deep_scrub":"67'1016","last_deep_scrub_stamp":"2023-10-30T13:52:49.650007+0000","last_clean_scrub_stamp":"2023-10-30T13:52:49.650007+0000","objects_scrubbed":55,"log_size":1016,"log_dups_size":0,"ondisk_log_size":1016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:38.507861+0000","scrub_duration":0.17591847999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36678727,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":465,"num_read_kb":133831,"num_write":1108,"num_write_kb":54499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":702779,"num_omap_keys":1002,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"67'2188","reported_seq":4144,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:25.049855+0000","last_change":"2023-10-30T13:53:25.049916+0000","last_active":"2023-10-30T13:53:25.049855+0000","last_peered":"2023-10-30T13:53:25.049855+0000","last_clean":"2023-10-30T13:53:25.049855+0000","last_became_active":"2023-10-30T13:10:34.371000+0000","last_became_peered":"2023-10-30T13:10:34.371000+0000","last_unstale":"2023-10-30T13:53:25.049855+0000","last_undegraded":"2023-10-30T13:53:25.049855+0000","last_fullsized":"2023-10-30T13:53:25.049855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2188","last_scrub_stamp":"2023-10-30T13:53:25.049775+0000","last_deep_scrub":"67'2188","last_deep_scrub_stamp":"2023-10-30T13:53:25.049775+0000","last_clean_scrub_stamp":"2023-10-30T13:53:25.049775+0000","objects_scrubbed":625,"log_size":2188,"log_dups_size":0,"ondisk_log_size":2188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:45.692320+0000","scrub_duration":0.916050107,"objects_trimmed":0,"snaptrim_duration":0.015028952999999999,"stat_sum":{"num_bytes":21253392,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":236,"num_read":95,"num_read_kb":12339,"num_write":1306,"num_write_kb":30718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'688","reported_seq":925,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:45.045919+0000","last_change":"2023-10-30T13:52:45.045919+0000","last_active":"2023-10-30T13:52:45.045919+0000","last_peered":"2023-10-30T13:52:45.045919+0000","last_clean":"2023-10-30T13:52:45.045919+0000","last_became_active":"2023-10-30T13:10:33.697616+0000","last_became_peered":"2023-10-30T13:10:33.697616+0000","last_unstale":"2023-10-30T13:52:45.045919+0000","last_undegraded":"2023-10-30T13:52:45.045919+0000","last_fullsized":"2023-10-30T13:52:45.045919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'688","last_scrub_stamp":"2023-10-30T13:52:45.045739+0000","last_deep_scrub":"67'688","last_deep_scrub_stamp":"2023-10-30T13:52:45.045739+0000","last_clean_scrub_stamp":"2023-10-30T13:52:45.045739+0000","objects_scrubbed":51,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:31:53.871616+0000","scrub_duration":0.14959883700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165846,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":249,"num_read_kb":148701,"num_write":778,"num_write_kb":54421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":420719,"num_omap_keys":809,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"67'2193","reported_seq":4158,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:28.930438+0000","last_change":"2023-10-30T13:53:28.930491+0000","last_active":"2023-10-30T13:53:28.930438+0000","last_peered":"2023-10-30T13:53:28.930438+0000","last_clean":"2023-10-30T13:53:28.930438+0000","last_became_active":"2023-10-30T13:10:34.374443+0000","last_became_peered":"2023-10-30T13:10:34.374443+0000","last_unstale":"2023-10-30T13:53:28.930438+0000","last_undegraded":"2023-10-30T13:53:28.930438+0000","last_fullsized":"2023-10-30T13:53:28.930438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2193","last_scrub_stamp":"2023-10-30T13:53:28.930356+0000","last_deep_scrub":"67'2193","last_deep_scrub_stamp":"2023-10-30T13:53:28.930356+0000","last_clean_scrub_stamp":"2023-10-30T13:53:28.930356+0000","objects_scrubbed":626,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:45.236130+0000","scrub_duration":0.82498087200000003,"objects_trimmed":0,"snaptrim_duration":0.011682140000000001,"stat_sum":{"num_bytes":26941199,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":237,"num_read":103,"num_read_kb":4114,"num_write":1331,"num_write_kb":32609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'1113","reported_seq":1401,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:48.645108+0000","last_change":"2023-10-30T13:52:48.645108+0000","last_active":"2023-10-30T13:52:48.645108+0000","last_peered":"2023-10-30T13:52:48.645108+0000","last_clean":"2023-10-30T13:52:48.645108+0000","last_became_active":"2023-10-30T13:10:33.694195+0000","last_became_peered":"2023-10-30T13:10:33.694195+0000","last_unstale":"2023-10-30T13:52:48.645108+0000","last_undegraded":"2023-10-30T13:52:48.645108+0000","last_fullsized":"2023-10-30T13:52:48.645108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1113","last_scrub_stamp":"2023-10-30T13:52:48.644981+0000","last_deep_scrub":"67'1113","last_deep_scrub_stamp":"2023-10-30T13:52:48.644981+0000","last_clean_scrub_stamp":"2023-10-30T13:52:48.644981+0000","objects_scrubbed":63,"log_size":1113,"log_dups_size":0,"ondisk_log_size":1113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:56.952822+0000","scrub_duration":0.19793386600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":63,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":381,"num_read_kb":250673,"num_write":1157,"num_write_kb":87754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":620304,"num_omap_keys":1053,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"67'2154","reported_seq":4131,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:41.209492+0000","last_change":"2023-10-30T13:53:41.209567+0000","last_active":"2023-10-30T13:53:41.209492+0000","last_peered":"2023-10-30T13:53:41.209492+0000","last_clean":"2023-10-30T13:53:41.209492+0000","last_became_active":"2023-10-30T13:10:34.382117+0000","last_became_peered":"2023-10-30T13:10:34.382117+0000","last_unstale":"2023-10-30T13:53:41.209492+0000","last_undegraded":"2023-10-30T13:53:41.209492+0000","last_fullsized":"2023-10-30T13:53:41.209492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2154","last_scrub_stamp":"2023-10-30T13:53:41.209404+0000","last_deep_scrub":"67'2154","last_deep_scrub_stamp":"2023-10-30T13:53:41.209404+0000","last_clean_scrub_stamp":"2023-10-30T13:53:41.209404+0000","objects_scrubbed":614,"log_size":2154,"log_dups_size":0,"ondisk_log_size":2154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:01:13.221180+0000","scrub_duration":0.93015355799999999,"objects_trimmed":0,"snaptrim_duration":0.014794043,"stat_sum":{"num_bytes":18888621,"num_objects":614,"num_object_clones":283,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":224,"num_read":107,"num_read_kb":4188,"num_write":1306,"num_write_kb":24240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"67'1274","reported_seq":1536,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.034230+0000","last_change":"2023-10-30T13:52:53.034230+0000","last_active":"2023-10-30T13:52:53.034230+0000","last_peered":"2023-10-30T13:52:53.034230+0000","last_clean":"2023-10-30T13:52:53.034230+0000","last_became_active":"2023-10-30T13:10:33.694753+0000","last_became_peered":"2023-10-30T13:10:33.694753+0000","last_unstale":"2023-10-30T13:52:53.034230+0000","last_undegraded":"2023-10-30T13:52:53.034230+0000","last_fullsized":"2023-10-30T13:52:53.034230+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1274","last_scrub_stamp":"2023-10-30T13:52:53.034072+0000","last_deep_scrub":"67'1274","last_deep_scrub_stamp":"2023-10-30T13:52:53.034072+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.034072+0000","objects_scrubbed":46,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:05:34.464424+0000","scrub_duration":0.19364663100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364039,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":288,"num_read_kb":200539,"num_write":1327,"num_write_kb":71054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":407575,"num_omap_keys":755,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"67'2176","reported_seq":4163,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:48.181004+0000","last_change":"2023-10-30T13:52:48.181151+0000","last_active":"2023-10-30T13:52:48.181004+0000","last_peered":"2023-10-30T13:52:48.181004+0000","last_clean":"2023-10-30T13:52:48.181004+0000","last_became_active":"2023-10-30T13:10:34.342142+0000","last_became_peered":"2023-10-30T13:10:34.342142+0000","last_unstale":"2023-10-30T13:52:48.181004+0000","last_undegraded":"2023-10-30T13:52:48.181004+0000","last_fullsized":"2023-10-30T13:52:48.181004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2176","last_scrub_stamp":"2023-10-30T13:52:48.180852+0000","last_deep_scrub":"67'2176","last_deep_scrub_stamp":"2023-10-30T13:52:48.180852+0000","last_clean_scrub_stamp":"2023-10-30T13:52:48.180852+0000","objects_scrubbed":600,"log_size":2176,"log_dups_size":0,"ondisk_log_size":2176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:07:11.947771+0000","scrub_duration":1.0044801649999999,"objects_trimmed":0,"snaptrim_duration":0.010788236,"stat_sum":{"num_bytes":10484856,"num_objects":600,"num_object_clones":281,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":220,"num_read":116,"num_read_kb":28,"num_write":1335,"num_write_kb":13295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"67'832","reported_seq":1091,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:53.593522+0000","last_change":"2023-10-30T13:52:53.593522+0000","last_active":"2023-10-30T13:52:53.593522+0000","last_peered":"2023-10-30T13:52:53.593522+0000","last_clean":"2023-10-30T13:52:53.593522+0000","last_became_active":"2023-10-30T13:10:33.716578+0000","last_became_peered":"2023-10-30T13:10:33.716578+0000","last_unstale":"2023-10-30T13:52:53.593522+0000","last_undegraded":"2023-10-30T13:52:53.593522+0000","last_fullsized":"2023-10-30T13:52:53.593522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'832","last_scrub_stamp":"2023-10-30T13:52:53.593416+0000","last_deep_scrub":"67'832","last_deep_scrub_stamp":"2023-10-30T13:52:53.593416+0000","last_clean_scrub_stamp":"2023-10-30T13:52:53.593416+0000","objects_scrubbed":54,"log_size":832,"log_dups_size":0,"ondisk_log_size":832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:14:41.436296+0000","scrub_duration":0.145814324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166406,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":174067,"num_write":897,"num_write_kb":62874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":516505,"num_omap_keys":962,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'2158","reported_seq":4124,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:30.220672+0000","last_change":"2023-10-30T13:53:30.220750+0000","last_active":"2023-10-30T13:53:30.220672+0000","last_peered":"2023-10-30T13:53:30.220672+0000","last_clean":"2023-10-30T13:53:30.220672+0000","last_became_active":"2023-10-30T13:10:34.371578+0000","last_became_peered":"2023-10-30T13:10:34.371578+0000","last_unstale":"2023-10-30T13:53:30.220672+0000","last_undegraded":"2023-10-30T13:53:30.220672+0000","last_fullsized":"2023-10-30T13:53:30.220672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2158","last_scrub_stamp":"2023-10-30T13:53:30.220578+0000","last_deep_scrub":"67'2158","last_deep_scrub_stamp":"2023-10-30T13:53:30.220578+0000","last_clean_scrub_stamp":"2023-10-30T13:53:30.220578+0000","objects_scrubbed":616,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:03:58.809379+0000","scrub_duration":1.1187861859999999,"objects_trimmed":0,"snaptrim_duration":0.015268978000000001,"stat_sum":{"num_bytes":11043591,"num_objects":616,"num_object_clones":291,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":221,"num_read":104,"num_read_kb":3707,"num_write":1322,"num_write_kb":16509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":244,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.159767+0000","last_change":"2023-10-30T13:52:40.159977+0000","last_active":"2023-10-30T13:52:40.159767+0000","last_peered":"2023-10-30T13:52:40.159767+0000","last_clean":"2023-10-30T13:52:40.159767+0000","last_became_active":"2023-10-30T13:10:24.435401+0000","last_became_peered":"2023-10-30T13:10:24.435401+0000","last_unstale":"2023-10-30T13:52:40.159767+0000","last_undegraded":"2023-10-30T13:52:40.159767+0000","last_fullsized":"2023-10-30T13:52:40.159767+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-30T13:52:40.159494+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-30T13:52:40.159494+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.159494+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:37:13.673701+0000","scrub_duration":0.017799540999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'670","reported_seq":889,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:42.315206+0000","last_change":"2023-10-30T13:52:42.315332+0000","last_active":"2023-10-30T13:52:42.315206+0000","last_peered":"2023-10-30T13:52:42.315206+0000","last_clean":"2023-10-30T13:52:42.315206+0000","last_became_active":"2023-10-30T13:10:33.707687+0000","last_became_peered":"2023-10-30T13:10:33.707687+0000","last_unstale":"2023-10-30T13:52:42.315206+0000","last_undegraded":"2023-10-30T13:52:42.315206+0000","last_fullsized":"2023-10-30T13:52:42.315206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'670","last_scrub_stamp":"2023-10-30T13:52:42.314975+0000","last_deep_scrub":"67'670","last_deep_scrub_stamp":"2023-10-30T13:52:42.314975+0000","last_clean_scrub_stamp":"2023-10-30T13:52:42.314975+0000","objects_scrubbed":53,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:16.143637+0000","scrub_duration":0.170007133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360178,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":232,"num_read_kb":148904,"num_write":701,"num_write_kb":62706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":385690,"num_omap_keys":753,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'2178","reported_seq":4125,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:11.189453+0000","last_change":"2023-10-30T13:53:11.189527+0000","last_active":"2023-10-30T13:53:11.189453+0000","last_peered":"2023-10-30T13:53:11.189453+0000","last_clean":"2023-10-30T13:53:11.189453+0000","last_became_active":"2023-10-30T13:10:34.348152+0000","last_became_peered":"2023-10-30T13:10:34.348152+0000","last_unstale":"2023-10-30T13:53:11.189453+0000","last_undegraded":"2023-10-30T13:53:11.189453+0000","last_fullsized":"2023-10-30T13:53:11.189453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2178","last_scrub_stamp":"2023-10-30T13:53:11.189379+0000","last_deep_scrub":"67'2178","last_deep_scrub_stamp":"2023-10-30T13:53:11.189379+0000","last_clean_scrub_stamp":"2023-10-30T13:53:11.189379+0000","objects_scrubbed":633,"log_size":2178,"log_dups_size":0,"ondisk_log_size":2178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:37:47.706046+0000","scrub_duration":1.230169694,"objects_trimmed":0,"snaptrim_duration":0.0057314540000000004,"stat_sum":{"num_bytes":22780711,"num_objects":633,"num_object_clones":299,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":242,"num_read":92,"num_read_kb":12,"num_write":1293,"num_write_kb":28237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"67'756","reported_seq":998,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:04.119917+0000","last_change":"2023-10-30T13:53:04.120074+0000","last_active":"2023-10-30T13:53:04.119917+0000","last_peered":"2023-10-30T13:53:04.119917+0000","last_clean":"2023-10-30T13:53:04.119917+0000","last_became_active":"2023-10-30T13:10:33.720587+0000","last_became_peered":"2023-10-30T13:10:33.720587+0000","last_unstale":"2023-10-30T13:53:04.119917+0000","last_undegraded":"2023-10-30T13:53:04.119917+0000","last_fullsized":"2023-10-30T13:53:04.119917+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'756","last_scrub_stamp":"2023-10-30T13:53:04.119811+0000","last_deep_scrub":"67'756","last_deep_scrub_stamp":"2023-10-30T13:53:04.119811+0000","last_clean_scrub_stamp":"2023-10-30T13:53:04.119811+0000","objects_scrubbed":49,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:34.012792+0000","scrub_duration":0.144148941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":256,"num_read_kb":247261,"num_write":740,"num_write_kb":70569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":420741,"num_omap_keys":813,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"67'2028","reported_seq":3997,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:40.341768+0000","last_change":"2023-10-30T13:52:40.341843+0000","last_active":"2023-10-30T13:52:40.341768+0000","last_peered":"2023-10-30T13:52:40.341768+0000","last_clean":"2023-10-30T13:52:40.341768+0000","last_became_active":"2023-10-30T13:10:34.359315+0000","last_became_peered":"2023-10-30T13:10:34.359315+0000","last_unstale":"2023-10-30T13:52:40.341768+0000","last_undegraded":"2023-10-30T13:52:40.341768+0000","last_fullsized":"2023-10-30T13:52:40.341768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2028","last_scrub_stamp":"2023-10-30T13:52:40.341675+0000","last_deep_scrub":"67'2028","last_deep_scrub_stamp":"2023-10-30T13:52:40.341675+0000","last_clean_scrub_stamp":"2023-10-30T13:52:40.341675+0000","objects_scrubbed":568,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:22:08.372774+0000","scrub_duration":1.158419098,"objects_trimmed":0,"snaptrim_duration":0.002406114,"stat_sum":{"num_bytes":14184993,"num_objects":568,"num_object_clones":263,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":210,"num_read":80,"num_read_kb":26,"num_write":1239,"num_write_kb":16314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"67'2090","reported_seq":3466,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.634617+0000","last_change":"2023-10-30T13:52:36.634675+0000","last_active":"2023-10-30T13:52:36.634617+0000","last_peered":"2023-10-30T13:52:36.634617+0000","last_clean":"2023-10-30T13:52:36.634617+0000","last_became_active":"2023-10-30T13:10:33.695104+0000","last_became_peered":"2023-10-30T13:10:33.695104+0000","last_unstale":"2023-10-30T13:52:36.634617+0000","last_undegraded":"2023-10-30T13:52:36.634617+0000","last_fullsized":"2023-10-30T13:52:36.634617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2090","last_scrub_stamp":"2023-10-30T13:52:36.634514+0000","last_deep_scrub":"67'2090","last_deep_scrub_stamp":"2023-10-30T13:52:36.634514+0000","last_clean_scrub_stamp":"2023-10-30T13:52:36.634514+0000","objects_scrubbed":53,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:23:20.160935+0000","scrub_duration":0.14709519600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41670866,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":1534,"num_read_kb":188566,"num_write":2090,"num_write_kb":83865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":477756,"num_omap_keys":784,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"67'2139","reported_seq":4115,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:09.536900+0000","last_change":"2023-10-30T13:53:09.536942+0000","last_active":"2023-10-30T13:53:09.536900+0000","last_peered":"2023-10-30T13:53:09.536900+0000","last_clean":"2023-10-30T13:53:09.536900+0000","last_became_active":"2023-10-30T13:10:34.332619+0000","last_became_peered":"2023-10-30T13:10:34.332619+0000","last_unstale":"2023-10-30T13:53:09.536900+0000","last_undegraded":"2023-10-30T13:53:09.536900+0000","last_fullsized":"2023-10-30T13:53:09.536900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2139","last_scrub_stamp":"2023-10-30T13:53:09.536825+0000","last_deep_scrub":"67'2139","last_deep_scrub_stamp":"2023-10-30T13:53:09.536825+0000","last_clean_scrub_stamp":"2023-10-30T13:53:09.536825+0000","objects_scrubbed":612,"log_size":2139,"log_dups_size":0,"ondisk_log_size":2139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:54:03.730208+0000","scrub_duration":0.83069248900000003,"objects_trimmed":0,"snaptrim_duration":0.002908281,"stat_sum":{"num_bytes":22417951,"num_objects":612,"num_object_clones":284,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":221,"num_read":105,"num_read_kb":44,"num_write":1298,"num_write_kb":23585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"67'1007","reported_seq":1293,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:36.427338+0000","last_change":"2023-10-30T13:52:36.427454+0000","last_active":"2023-10-30T13:52:36.427338+0000","last_peered":"2023-10-30T13:52:36.427338+0000","last_clean":"2023-10-30T13:52:36.427338+0000","last_became_active":"2023-10-30T13:10:33.707614+0000","last_became_peered":"2023-10-30T13:10:33.707614+0000","last_unstale":"2023-10-30T13:52:36.427338+0000","last_undegraded":"2023-10-30T13:52:36.427338+0000","last_fullsized":"2023-10-30T13:52:36.427338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1007","last_scrub_stamp":"2023-10-30T13:52:36.427140+0000","last_deep_scrub":"67'1007","last_deep_scrub_stamp":"2023-10-30T13:52:36.427140+0000","last_clean_scrub_stamp":"2023-10-30T13:52:36.427140+0000","objects_scrubbed":60,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:11.996639+0000","scrub_duration":0.17205899599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":355,"num_read_kb":252215,"num_write":1079,"num_write_kb":75669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":53,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":633823,"num_omap_keys":985,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'2106","reported_seq":4092,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:51.008269+0000","last_change":"2023-10-30T13:52:51.008354+0000","last_active":"2023-10-30T13:52:51.008269+0000","last_peered":"2023-10-30T13:52:51.008269+0000","last_clean":"2023-10-30T13:52:51.008269+0000","last_became_active":"2023-10-30T13:10:34.361314+0000","last_became_peered":"2023-10-30T13:10:34.361314+0000","last_unstale":"2023-10-30T13:52:51.008269+0000","last_undegraded":"2023-10-30T13:52:51.008269+0000","last_fullsized":"2023-10-30T13:52:51.008269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2106","last_scrub_stamp":"2023-10-30T13:52:51.008186+0000","last_deep_scrub":"67'2106","last_deep_scrub_stamp":"2023-10-30T13:52:51.008186+0000","last_clean_scrub_stamp":"2023-10-30T13:52:51.008186+0000","objects_scrubbed":575,"log_size":2106,"log_dups_size":0,"ondisk_log_size":2106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:03:38.209876+0000","scrub_duration":1.1485167089999999,"objects_trimmed":0,"snaptrim_duration":0.007710705,"stat_sum":{"num_bytes":23751577,"num_objects":575,"num_object_clones":266,"num_object_copies":1150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":199,"num_read":100,"num_read_kb":489,"num_write":1325,"num_write_kb":27587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"67'998","reported_seq":1249,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.917921+0000","last_change":"2023-10-30T13:52:43.917980+0000","last_active":"2023-10-30T13:52:43.917921+0000","last_peered":"2023-10-30T13:52:43.917921+0000","last_clean":"2023-10-30T13:52:43.917921+0000","last_became_active":"2023-10-30T13:10:33.687827+0000","last_became_peered":"2023-10-30T13:10:33.687827+0000","last_unstale":"2023-10-30T13:52:43.917921+0000","last_undegraded":"2023-10-30T13:52:43.917921+0000","last_fullsized":"2023-10-30T13:52:43.917921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'998","last_scrub_stamp":"2023-10-30T13:52:43.917841+0000","last_deep_scrub":"67'998","last_deep_scrub_stamp":"2023-10-30T13:52:43.917841+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.917841+0000","objects_scrubbed":56,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:41:43.976830+0000","scrub_duration":0.14471989900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":294,"num_read_kb":227180,"num_write":1039,"num_write_kb":87590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":525988,"num_omap_keys":913,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2248","reported_seq":4193,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:01.680696+0000","last_change":"2023-10-30T13:53:01.680795+0000","last_active":"2023-10-30T13:53:01.680696+0000","last_peered":"2023-10-30T13:53:01.680696+0000","last_clean":"2023-10-30T13:53:01.680696+0000","last_became_active":"2023-10-30T13:10:34.347855+0000","last_became_peered":"2023-10-30T13:10:34.347855+0000","last_unstale":"2023-10-30T13:53:01.680696+0000","last_undegraded":"2023-10-30T13:53:01.680696+0000","last_fullsized":"2023-10-30T13:53:01.680696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2248","last_scrub_stamp":"2023-10-30T13:53:01.680522+0000","last_deep_scrub":"67'2248","last_deep_scrub_stamp":"2023-10-30T13:53:01.680522+0000","last_clean_scrub_stamp":"2023-10-30T13:53:01.680522+0000","objects_scrubbed":650,"log_size":2248,"log_dups_size":0,"ondisk_log_size":2248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:34.935002+0000","scrub_duration":1.35250176,"objects_trimmed":0,"snaptrim_duration":0.010719574000000001,"stat_sum":{"num_bytes":35538242,"num_objects":650,"num_object_clones":301,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":239,"num_read":91,"num_read_kb":31,"num_write":1347,"num_write_kb":41412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"67'834","reported_seq":1069,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:17.238395+0000","last_change":"2023-10-30T13:53:17.238539+0000","last_active":"2023-10-30T13:53:17.238395+0000","last_peered":"2023-10-30T13:53:17.238395+0000","last_clean":"2023-10-30T13:53:17.238395+0000","last_became_active":"2023-10-30T13:10:33.720637+0000","last_became_peered":"2023-10-30T13:10:33.720637+0000","last_unstale":"2023-10-30T13:53:17.238395+0000","last_undegraded":"2023-10-30T13:53:17.238395+0000","last_fullsized":"2023-10-30T13:53:17.238395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'834","last_scrub_stamp":"2023-10-30T13:53:17.238196+0000","last_deep_scrub":"67'834","last_deep_scrub_stamp":"2023-10-30T13:53:17.238196+0000","last_clean_scrub_stamp":"2023-10-30T13:53:17.238196+0000","objects_scrubbed":47,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:25:51.753551+0000","scrub_duration":0.16901470499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":258,"num_read_kb":177173,"num_write":845,"num_write_kb":62523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":428645,"num_omap_keys":765,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'2182","reported_seq":4142,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:13.698099+0000","last_change":"2023-10-30T13:53:13.698196+0000","last_active":"2023-10-30T13:53:13.698099+0000","last_peered":"2023-10-30T13:53:13.698099+0000","last_clean":"2023-10-30T13:53:13.698099+0000","last_became_active":"2023-10-30T13:10:34.331761+0000","last_became_peered":"2023-10-30T13:10:34.331761+0000","last_unstale":"2023-10-30T13:53:13.698099+0000","last_undegraded":"2023-10-30T13:53:13.698099+0000","last_fullsized":"2023-10-30T13:53:13.698099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2182","last_scrub_stamp":"2023-10-30T13:53:13.697999+0000","last_deep_scrub":"67'2182","last_deep_scrub_stamp":"2023-10-30T13:53:13.697999+0000","last_clean_scrub_stamp":"2023-10-30T13:53:13.697999+0000","objects_scrubbed":631,"log_size":2182,"log_dups_size":0,"ondisk_log_size":2182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:20:54.907462+0000","scrub_duration":0.98641216499999995,"objects_trimmed":0,"snaptrim_duration":0.0059276069999999997,"stat_sum":{"num_bytes":23389062,"num_objects":631,"num_object_clones":294,"num_object_copies":1262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":232,"num_read":100,"num_read_kb":25,"num_write":1313,"num_write_kb":24315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"67'969","reported_seq":1219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:00.092676+0000","last_change":"2023-10-30T13:53:00.092749+0000","last_active":"2023-10-30T13:53:00.092676+0000","last_peered":"2023-10-30T13:53:00.092676+0000","last_clean":"2023-10-30T13:53:00.092676+0000","last_became_active":"2023-10-30T13:10:33.725901+0000","last_became_peered":"2023-10-30T13:10:33.725901+0000","last_unstale":"2023-10-30T13:53:00.092676+0000","last_undegraded":"2023-10-30T13:53:00.092676+0000","last_fullsized":"2023-10-30T13:53:00.092676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'969","last_scrub_stamp":"2023-10-30T13:53:00.092550+0000","last_deep_scrub":"67'969","last_deep_scrub_stamp":"2023-10-30T13:53:00.092550+0000","last_clean_scrub_stamp":"2023-10-30T13:53:00.092550+0000","objects_scrubbed":54,"log_size":969,"log_dups_size":0,"ondisk_log_size":969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:16:09.851840+0000","scrub_duration":0.31438043799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":310,"num_read_kb":109156,"num_write":1496,"num_write_kb":61821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":485487,"num_omap_keys":848,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'1992","reported_seq":3976,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:56.936214+0000","last_change":"2023-10-30T13:52:56.936303+0000","last_active":"2023-10-30T13:52:56.936214+0000","last_peered":"2023-10-30T13:52:56.936214+0000","last_clean":"2023-10-30T13:52:56.936214+0000","last_became_active":"2023-10-30T13:10:34.369500+0000","last_became_peered":"2023-10-30T13:10:34.369500+0000","last_unstale":"2023-10-30T13:52:56.936214+0000","last_undegraded":"2023-10-30T13:52:56.936214+0000","last_fullsized":"2023-10-30T13:52:56.936214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1992","last_scrub_stamp":"2023-10-30T13:52:56.936120+0000","last_deep_scrub":"67'1992","last_deep_scrub_stamp":"2023-10-30T13:52:56.936120+0000","last_clean_scrub_stamp":"2023-10-30T13:52:56.936120+0000","objects_scrubbed":571,"log_size":1992,"log_dups_size":0,"ondisk_log_size":1992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:21.220503+0000","scrub_duration":1.0834315910000001,"objects_trimmed":0,"snaptrim_duration":0.0056940879999999999,"stat_sum":{"num_bytes":23042560,"num_objects":571,"num_object_clones":259,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":201,"num_read":92,"num_read_kb":75,"num_write":1201,"num_write_kb":27814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"67'1157","reported_seq":1411,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.055552+0000","last_change":"2023-10-30T13:52:54.055652+0000","last_active":"2023-10-30T13:52:54.055552+0000","last_peered":"2023-10-30T13:52:54.055552+0000","last_clean":"2023-10-30T13:52:54.055552+0000","last_became_active":"2023-10-30T13:10:33.724895+0000","last_became_peered":"2023-10-30T13:10:33.724895+0000","last_unstale":"2023-10-30T13:52:54.055552+0000","last_undegraded":"2023-10-30T13:52:54.055552+0000","last_fullsized":"2023-10-30T13:52:54.055552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1157","last_scrub_stamp":"2023-10-30T13:52:54.055439+0000","last_deep_scrub":"67'1157","last_deep_scrub_stamp":"2023-10-30T13:52:54.055439+0000","last_clean_scrub_stamp":"2023-10-30T13:52:54.055439+0000","objects_scrubbed":62,"log_size":1157,"log_dups_size":0,"ondisk_log_size":1157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:52.120399+0000","scrub_duration":0.23085536300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":342,"num_read_kb":120967,"num_write":1233,"num_write_kb":51153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":723056,"num_omap_keys":1120,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"67'2017","reported_seq":4010,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.248739+0000","last_change":"2023-10-30T13:52:59.248901+0000","last_active":"2023-10-30T13:52:59.248739+0000","last_peered":"2023-10-30T13:52:59.248739+0000","last_clean":"2023-10-30T13:52:59.248739+0000","last_became_active":"2023-10-30T13:10:34.323051+0000","last_became_peered":"2023-10-30T13:10:34.323051+0000","last_unstale":"2023-10-30T13:52:59.248739+0000","last_undegraded":"2023-10-30T13:52:59.248739+0000","last_fullsized":"2023-10-30T13:52:59.248739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2017","last_scrub_stamp":"2023-10-30T13:52:59.248637+0000","last_deep_scrub":"67'2017","last_deep_scrub_stamp":"2023-10-30T13:52:59.248637+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.248637+0000","objects_scrubbed":572,"log_size":2017,"log_dups_size":0,"ondisk_log_size":2017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:42.043697+0000","scrub_duration":1.079787013,"objects_trimmed":0,"snaptrim_duration":0.0029926589999999999,"stat_sum":{"num_bytes":30813966,"num_objects":572,"num_object_clones":263,"num_object_copies":1144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":205,"num_read":105,"num_read_kb":87,"num_write":1219,"num_write_kb":33525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"67'468","reported_seq":664,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.111720+0000","last_change":"2023-10-30T13:52:59.111898+0000","last_active":"2023-10-30T13:52:59.111720+0000","last_peered":"2023-10-30T13:52:59.111720+0000","last_clean":"2023-10-30T13:52:59.111720+0000","last_became_active":"2023-10-30T13:10:33.722232+0000","last_became_peered":"2023-10-30T13:10:33.722232+0000","last_unstale":"2023-10-30T13:52:59.111720+0000","last_undegraded":"2023-10-30T13:52:59.111720+0000","last_fullsized":"2023-10-30T13:52:59.111720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'468","last_scrub_stamp":"2023-10-30T13:52:59.111509+0000","last_deep_scrub":"67'468","last_deep_scrub_stamp":"2023-10-30T13:52:59.111509+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.111509+0000","objects_scrubbed":47,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:39.063865+0000","scrub_duration":0.1401828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":209,"num_read_kb":100741,"num_write":482,"num_write_kb":29419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":340147,"num_omap_keys":605,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"67'2140","reported_seq":4127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:57.118260+0000","last_change":"2023-10-30T13:52:57.118430+0000","last_active":"2023-10-30T13:52:57.118260+0000","last_peered":"2023-10-30T13:52:57.118260+0000","last_clean":"2023-10-30T13:52:57.118260+0000","last_became_active":"2023-10-30T13:10:34.361446+0000","last_became_peered":"2023-10-30T13:10:34.361446+0000","last_unstale":"2023-10-30T13:52:57.118260+0000","last_undegraded":"2023-10-30T13:52:57.118260+0000","last_fullsized":"2023-10-30T13:52:57.118260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2140","last_scrub_stamp":"2023-10-30T13:52:57.118012+0000","last_deep_scrub":"67'2140","last_deep_scrub_stamp":"2023-10-30T13:52:57.118012+0000","last_clean_scrub_stamp":"2023-10-30T13:52:57.118012+0000","objects_scrubbed":595,"log_size":2140,"log_dups_size":0,"ondisk_log_size":2140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:16:30.111801+0000","scrub_duration":1.2581118739999999,"objects_trimmed":0,"snaptrim_duration":0.0057673200000000003,"stat_sum":{"num_bytes":13994934,"num_objects":595,"num_object_clones":279,"num_object_copies":1190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":215,"num_read":114,"num_read_kb":29,"num_write":1329,"num_write_kb":19694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"67'1276","reported_seq":1501,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:54.113521+0000","last_change":"2023-10-30T13:52:34.896255+0000","last_active":"2023-10-30T13:52:54.113521+0000","last_peered":"2023-10-30T13:52:54.113521+0000","last_clean":"2023-10-30T13:52:54.113521+0000","last_became_active":"2023-10-30T13:10:33.699095+0000","last_became_peered":"2023-10-30T13:10:33.699095+0000","last_unstale":"2023-10-30T13:52:54.113521+0000","last_undegraded":"2023-10-30T13:52:54.113521+0000","last_fullsized":"2023-10-30T13:52:54.113521+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1266","last_scrub_stamp":"2023-10-30T13:52:34.896121+0000","last_deep_scrub":"67'1266","last_deep_scrub_stamp":"2023-10-30T13:52:34.896121+0000","last_clean_scrub_stamp":"2023-10-30T13:52:34.896121+0000","objects_scrubbed":51,"log_size":1276,"log_dups_size":0,"ondisk_log_size":1276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:43.920989+0000","scrub_duration":0.15482325699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25006847,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":251,"num_read_kb":99282,"num_write":1301,"num_write_kb":42450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":678730,"num_omap_keys":1151,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"67'2168","reported_seq":4155,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:06.891046+0000","last_change":"2023-10-30T13:53:06.891119+0000","last_active":"2023-10-30T13:53:06.891046+0000","last_peered":"2023-10-30T13:53:06.891046+0000","last_clean":"2023-10-30T13:53:06.891046+0000","last_became_active":"2023-10-30T13:10:34.330408+0000","last_became_peered":"2023-10-30T13:10:34.330408+0000","last_unstale":"2023-10-30T13:53:06.891046+0000","last_undegraded":"2023-10-30T13:53:06.891046+0000","last_fullsized":"2023-10-30T13:53:06.891046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2168","last_scrub_stamp":"2023-10-30T13:53:06.890960+0000","last_deep_scrub":"67'2168","last_deep_scrub_stamp":"2023-10-30T13:53:06.890960+0000","last_clean_scrub_stamp":"2023-10-30T13:53:06.890960+0000","objects_scrubbed":612,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:45:46.242387+0000","scrub_duration":0.94320374900000004,"objects_trimmed":0,"snaptrim_duration":0.0063570620000000001,"stat_sum":{"num_bytes":16293763,"num_objects":612,"num_object_clones":286,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":233,"num_read":120,"num_read_kb":127,"num_write":1302,"num_write_kb":21658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'946","reported_seq":1182,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:35.887235+0000","last_change":"2023-10-30T13:52:35.887314+0000","last_active":"2023-10-30T13:52:35.887235+0000","last_peered":"2023-10-30T13:52:35.887235+0000","last_clean":"2023-10-30T13:52:35.887235+0000","last_became_active":"2023-10-30T13:10:33.691118+0000","last_became_peered":"2023-10-30T13:10:33.691118+0000","last_unstale":"2023-10-30T13:52:35.887235+0000","last_undegraded":"2023-10-30T13:52:35.887235+0000","last_fullsized":"2023-10-30T13:52:35.887235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'946","last_scrub_stamp":"2023-10-30T13:52:35.887142+0000","last_deep_scrub":"67'946","last_deep_scrub_stamp":"2023-10-30T13:52:35.887142+0000","last_clean_scrub_stamp":"2023-10-30T13:52:35.887142+0000","objects_scrubbed":46,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:06:00.273772+0000","scrub_duration":0.098447004000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":272,"num_read_kb":200216,"num_write":1137,"num_write_kb":59039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":497965,"num_omap_keys":763,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"67'1999","reported_seq":3980,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:59.786866+0000","last_change":"2023-10-30T13:52:59.786866+0000","last_active":"2023-10-30T13:52:59.786866+0000","last_peered":"2023-10-30T13:52:59.786866+0000","last_clean":"2023-10-30T13:52:59.786866+0000","last_became_active":"2023-10-30T13:10:34.330779+0000","last_became_peered":"2023-10-30T13:10:34.330779+0000","last_unstale":"2023-10-30T13:52:59.786866+0000","last_undegraded":"2023-10-30T13:52:59.786866+0000","last_fullsized":"2023-10-30T13:52:59.786866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1999","last_scrub_stamp":"2023-10-30T13:52:59.786686+0000","last_deep_scrub":"67'1999","last_deep_scrub_stamp":"2023-10-30T13:52:59.786686+0000","last_clean_scrub_stamp":"2023-10-30T13:52:59.786686+0000","objects_scrubbed":563,"log_size":1999,"log_dups_size":0,"ondisk_log_size":1999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:22:35.870896+0000","scrub_duration":1.004166471,"objects_trimmed":0,"snaptrim_duration":0.0060201619999999999,"stat_sum":{"num_bytes":10033117,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":215,"num_read":92,"num_read_kb":24,"num_write":1204,"num_write_kb":13175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"67'1025","reported_seq":1292,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:44.395104+0000","last_change":"2023-10-30T13:52:44.395104+0000","last_active":"2023-10-30T13:52:44.395104+0000","last_peered":"2023-10-30T13:52:44.395104+0000","last_clean":"2023-10-30T13:52:44.395104+0000","last_became_active":"2023-10-30T13:10:33.720494+0000","last_became_peered":"2023-10-30T13:10:33.720494+0000","last_unstale":"2023-10-30T13:52:44.395104+0000","last_undegraded":"2023-10-30T13:52:44.395104+0000","last_fullsized":"2023-10-30T13:52:44.395104+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1025","last_scrub_stamp":"2023-10-30T13:52:44.394896+0000","last_deep_scrub":"67'1025","last_deep_scrub_stamp":"2023-10-30T13:52:44.394896+0000","last_clean_scrub_stamp":"2023-10-30T13:52:44.394896+0000","objects_scrubbed":56,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:31.686901+0000","scrub_duration":0.17278753699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":318,"num_read_kb":202944,"num_write":1065,"num_write_kb":67614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":660981,"num_omap_keys":1062,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"67'2026","reported_seq":4005,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:13.613831+0000","last_change":"2023-10-30T13:53:13.614016+0000","last_active":"2023-10-30T13:53:13.613831+0000","last_peered":"2023-10-30T13:53:13.613831+0000","last_clean":"2023-10-30T13:53:13.613831+0000","last_became_active":"2023-10-30T13:10:34.364513+0000","last_became_peered":"2023-10-30T13:10:34.364513+0000","last_unstale":"2023-10-30T13:53:13.613831+0000","last_undegraded":"2023-10-30T13:53:13.613831+0000","last_fullsized":"2023-10-30T13:53:13.613831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2026","last_scrub_stamp":"2023-10-30T13:53:13.613723+0000","last_deep_scrub":"67'2026","last_deep_scrub_stamp":"2023-10-30T13:53:13.613723+0000","last_clean_scrub_stamp":"2023-10-30T13:53:13.613723+0000","objects_scrubbed":558,"log_size":2026,"log_dups_size":0,"ondisk_log_size":2026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:38:19.365469+0000","scrub_duration":0.69858885699999995,"objects_trimmed":0,"snaptrim_duration":0.0079253199999999996,"stat_sum":{"num_bytes":6591050,"num_objects":558,"num_object_clones":254,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":200,"num_read":83,"num_read_kb":63,"num_write":1255,"num_write_kb":11544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"67'2083","reported_seq":4060,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:04.412236+0000","last_change":"2023-10-30T13:53:04.412236+0000","last_active":"2023-10-30T13:53:04.412236+0000","last_peered":"2023-10-30T13:53:04.412236+0000","last_clean":"2023-10-30T13:53:04.412236+0000","last_became_active":"2023-10-30T13:10:34.364796+0000","last_became_peered":"2023-10-30T13:10:34.364796+0000","last_unstale":"2023-10-30T13:53:04.412236+0000","last_undegraded":"2023-10-30T13:53:04.412236+0000","last_fullsized":"2023-10-30T13:53:04.412236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2083","last_scrub_stamp":"2023-10-30T13:53:04.411986+0000","last_deep_scrub":"67'2083","last_deep_scrub_stamp":"2023-10-30T13:53:04.411986+0000","last_clean_scrub_stamp":"2023-10-30T13:53:04.411986+0000","objects_scrubbed":585,"log_size":2083,"log_dups_size":0,"ondisk_log_size":2083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:36:37.812935+0000","scrub_duration":1.1529056559999999,"objects_trimmed":0,"snaptrim_duration":0.010603414,"stat_sum":{"num_bytes":9104816,"num_objects":585,"num_object_clones":276,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":204,"num_read":104,"num_read_kb":44,"num_write":1285,"num_write_kb":14447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"67'1055","reported_seq":1321,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:43.345144+0000","last_change":"2023-10-30T13:52:43.345144+0000","last_active":"2023-10-30T13:52:43.345144+0000","last_peered":"2023-10-30T13:52:43.345144+0000","last_clean":"2023-10-30T13:52:43.345144+0000","last_became_active":"2023-10-30T13:10:33.720960+0000","last_became_peered":"2023-10-30T13:10:33.720960+0000","last_unstale":"2023-10-30T13:52:43.345144+0000","last_undegraded":"2023-10-30T13:52:43.345144+0000","last_fullsized":"2023-10-30T13:52:43.345144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1055","last_scrub_stamp":"2023-10-30T13:52:43.344949+0000","last_deep_scrub":"67'1055","last_deep_scrub_stamp":"2023-10-30T13:52:43.344949+0000","last_clean_scrub_stamp":"2023-10-30T13:52:43.344949+0000","objects_scrubbed":55,"log_size":1055,"log_dups_size":0,"ondisk_log_size":1055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:59:10.108303+0000","scrub_duration":0.154960875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":337,"num_read_kb":187117,"num_write":1200,"num_write_kb":47577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":706060,"num_omap_keys":1147,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'2277","reported_seq":4227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:55.617708+0000","last_change":"2023-10-30T13:52:55.617708+0000","last_active":"2023-10-30T13:52:55.617708+0000","last_peered":"2023-10-30T13:52:55.617708+0000","last_clean":"2023-10-30T13:52:55.617708+0000","last_became_active":"2023-10-30T13:10:34.369113+0000","last_became_peered":"2023-10-30T13:10:34.369113+0000","last_unstale":"2023-10-30T13:52:55.617708+0000","last_undegraded":"2023-10-30T13:52:55.617708+0000","last_fullsized":"2023-10-30T13:52:55.617708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2277","last_scrub_stamp":"2023-10-30T13:52:55.617575+0000","last_deep_scrub":"67'2277","last_deep_scrub_stamp":"2023-10-30T13:52:55.617575+0000","last_clean_scrub_stamp":"2023-10-30T13:52:55.617575+0000","objects_scrubbed":650,"log_size":2277,"log_dups_size":0,"ondisk_log_size":2277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:16:39.666383+0000","scrub_duration":1.3596852369999999,"objects_trimmed":0,"snaptrim_duration":0.0097894100000000001,"stat_sum":{"num_bytes":19085997,"num_objects":650,"num_object_clones":307,"num_object_copies":1300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":230,"num_read":105,"num_read_kb":202,"num_write":1384,"num_write_kb":24607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"67'1382","reported_seq":1696,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:05.200352+0000","last_change":"2023-10-30T13:53:05.200352+0000","last_active":"2023-10-30T13:53:05.200352+0000","last_peered":"2023-10-30T13:53:05.200352+0000","last_clean":"2023-10-30T13:53:05.200352+0000","last_became_active":"2023-10-30T13:10:33.728174+0000","last_became_peered":"2023-10-30T13:10:33.728174+0000","last_unstale":"2023-10-30T13:53:05.200352+0000","last_undegraded":"2023-10-30T13:53:05.200352+0000","last_fullsized":"2023-10-30T13:53:05.200352+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1382","last_scrub_stamp":"2023-10-30T13:53:05.200180+0000","last_deep_scrub":"67'1382","last_deep_scrub_stamp":"2023-10-30T13:53:05.200180+0000","last_clean_scrub_stamp":"2023-10-30T13:53:05.200180+0000","objects_scrubbed":68,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:15:00.361266+0000","scrub_duration":0.25105026499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37661445,"num_objects":68,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":418,"num_read_kb":304041,"num_write":1419,"num_write_kb":92539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":903188,"num_omap_keys":1478,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"67'2147","reported_seq":4104,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:06.340082+0000","last_change":"2023-10-30T13:53:06.340161+0000","last_active":"2023-10-30T13:53:06.340082+0000","last_peered":"2023-10-30T13:53:06.340082+0000","last_clean":"2023-10-30T13:53:06.340082+0000","last_became_active":"2023-10-30T13:10:34.344537+0000","last_became_peered":"2023-10-30T13:10:34.344537+0000","last_unstale":"2023-10-30T13:53:06.340082+0000","last_undegraded":"2023-10-30T13:53:06.340082+0000","last_fullsized":"2023-10-30T13:53:06.340082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2147","last_scrub_stamp":"2023-10-30T13:53:06.340001+0000","last_deep_scrub":"67'2147","last_deep_scrub_stamp":"2023-10-30T13:53:06.340001+0000","last_clean_scrub_stamp":"2023-10-30T13:53:06.340001+0000","objects_scrubbed":636,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:44.975735+0000","scrub_duration":1.142281364,"objects_trimmed":0,"snaptrim_duration":0.0026802879999999999,"stat_sum":{"num_bytes":22843366,"num_objects":636,"num_object_clones":293,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":237,"num_read":96,"num_read_kb":23,"num_write":1281,"num_write_kb":25082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"67'822","reported_seq":1066,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:44.024796+0000","last_change":"2023-10-30T13:52:44.024796+0000","last_active":"2023-10-30T13:52:44.024796+0000","last_peered":"2023-10-30T13:52:44.024796+0000","last_clean":"2023-10-30T13:52:44.024796+0000","last_became_active":"2023-10-30T13:10:33.689325+0000","last_became_peered":"2023-10-30T13:10:33.689325+0000","last_unstale":"2023-10-30T13:52:44.024796+0000","last_undegraded":"2023-10-30T13:52:44.024796+0000","last_fullsized":"2023-10-30T13:52:44.024796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'822","last_scrub_stamp":"2023-10-30T13:52:44.024664+0000","last_deep_scrub":"67'822","last_deep_scrub_stamp":"2023-10-30T13:52:44.024664+0000","last_clean_scrub_stamp":"2023-10-30T13:52:44.024664+0000","objects_scrubbed":50,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:54.864042+0000","scrub_duration":0.11718745699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":280,"num_read_kb":212075,"num_write":835,"num_write_kb":74769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":365319,"num_omap_keys":701,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"67'2055","reported_seq":4042,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:53:05.863446+0000","last_change":"2023-10-30T13:53:05.863560+0000","last_active":"2023-10-30T13:53:05.863446+0000","last_peered":"2023-10-30T13:53:05.863446+0000","last_clean":"2023-10-30T13:53:05.863446+0000","last_became_active":"2023-10-30T13:10:34.354086+0000","last_became_peered":"2023-10-30T13:10:34.354086+0000","last_unstale":"2023-10-30T13:53:05.863446+0000","last_undegraded":"2023-10-30T13:53:05.863446+0000","last_fullsized":"2023-10-30T13:53:05.863446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2055","last_scrub_stamp":"2023-10-30T13:53:05.863371+0000","last_deep_scrub":"67'2055","last_deep_scrub_stamp":"2023-10-30T13:53:05.863371+0000","last_clean_scrub_stamp":"2023-10-30T13:53:05.863371+0000","objects_scrubbed":589,"log_size":2055,"log_dups_size":0,"ondisk_log_size":2055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:54:20.796229+0000","scrub_duration":0.95279065399999996,"objects_trimmed":0,"snaptrim_duration":0.0055040289999999997,"stat_sum":{"num_bytes":14435337,"num_objects":589,"num_object_clones":269,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":212,"num_read":103,"num_read_kb":22,"num_write":1235,"num_write_kb":16517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"67'768","reported_seq":1042,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-30T13:52:34.271316+0000","last_change":"2023-10-30T13:52:34.271376+0000","last_active":"2023-10-30T13:52:34.271316+0000","last_peered":"2023-10-30T13:52:34.271316+0000","last_clean":"2023-10-30T13:52:34.271316+0000","last_became_active":"2023-10-30T13:10:33.714821+0000","last_became_peered":"2023-10-30T13:10:33.714821+0000","last_unstale":"2023-10-30T13:52:34.271316+0000","last_undegraded":"2023-10-30T13:52:34.271316+0000","last_fullsized":"2023-10-30T13:52:34.271316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'768","last_scrub_stamp":"2023-10-30T13:52:34.271229+0000","last_deep_scrub":"67'768","last_deep_scrub_stamp":"2023-10-30T13:52:34.271229+0000","last_clean_scrub_stamp":"2023-10-30T13:52:34.271229+0000","objects_scrubbed":64,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:49:00.506995+0000","scrub_duration":0.20155240999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749318,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":374,"num_read_kb":96620,"num_write":840,"num_write_kb":51020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":647968,"num_omap_keys":1098,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165182927,"num_objects":39302,"num_object_clones":18317,"num_object_copies":78604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39302,"num_whiteouts":14350,"num_read":6653,"num_read_kb":168434,"num_write":83476,"num_write_kb":1459733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424070144,"data_stored":2332381086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137932,"ondisk_log_size":137932,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":909135531,"num_objects":1701,"num_object_clones":0,"num_object_copies":3402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":11399,"num_read_kb":5963012,"num_write":34147,"num_write_kb":2074566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1411,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17841559,"num_omap_keys":30407,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1818419200,"data_stored":1818271062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32051,"ondisk_log_size":32051,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575375,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756856,"kb_used_data":577200,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93614984,"statfs":{"total":96636764160,"available":95861743616,"internally_reserved":0,"allocated":591052800,"data_stored":577425546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":183957963},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.81000000000000005,"15min":0.91300000000000003},"min":{"1min":0.45100000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3080000000000001,"15min":15.381},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.78100000000000003,"15min":0.84599999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.315},"max":{"1min":1.05,"5min":1.8400000000000001,"15min":2.8079999999999998},"last":0.879}]},{"osd":1,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.80700000000000005,"15min":0.90600000000000003},"min":{"1min":0.51300000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1839999999999999,"5min":1.79,"15min":15.271000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.87,"15min":0.95199999999999996},"min":{"1min":0.58199999999999996,"5min":0.46000000000000002,"15min":0.42999999999999999},"max":{"1min":1.2769999999999999,"5min":1.6000000000000001,"15min":15.228999999999999},"last":0.91300000000000003}]},{"osd":2,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.89900000000000002,"15min":0.90600000000000003},"min":{"1min":0.65000000000000002,"5min":0.45600000000000002,"15min":0.36299999999999999},"max":{"1min":1.123,"5min":1.345,"15min":2.2429999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.86699999999999999,"15min":0.86599999999999999},"min":{"1min":0.378,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4610000000000001,"5min":1.929,"15min":1.929},"last":1.002}]},{"osd":3,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.91800000000000004,"15min":1.056},"min":{"1min":0.54100000000000004,"5min":0.372,"15min":0.372},"max":{"1min":1.159,"5min":2.0249999999999999,"15min":15.324},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.99399999999999999,"15min":0.98999999999999999},"min":{"1min":0.65400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.379,"5min":1.379,"15min":1.637},"last":0.84599999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.05,"15min":1.115},"min":{"1min":0.47999999999999998,"5min":0.433,"15min":0.40899999999999997},"max":{"1min":1.2929999999999999,"5min":2.7869999999999999,"15min":15.172000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0740000000000001,"15min":1.077},"min":{"1min":0.39700000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.6379999999999999,"5min":3.032,"15min":3.032},"last":0.97699999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.091,"15min":1.1000000000000001},"min":{"1min":0.57999999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.409,"5min":1.4770000000000001,"15min":1.7050000000000001},"last":1.052},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.046,"15min":1.0109999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.554,"5min":4.8899999999999997,"15min":4.8899999999999997},"last":1.097}]},{"osd":6,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1850000000000001,"15min":1.2470000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.8340000000000001,"5min":3.0350000000000001,"15min":15.427},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.24,"15min":1.3200000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.42199999999999999},"max":{"1min":1.948,"5min":3.5169999999999999,"15min":32.976999999999997},"last":1.0760000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575374,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591964,"kb_used_data":452564,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93779876,"statfs":{"total":96636764160,"available":96030593024,"internally_reserved":0,"allocated":463425536,"data_stored":454827877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142735815},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:53:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.79500000000000004,"15min":0.82999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3300000000000001,"5min":1.3999999999999999,"15min":5.9880000000000004},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.85799999999999998,"15min":0.88300000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.1839999999999999,"5min":1.3160000000000001,"15min":6.0730000000000004},"last":1.0289999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:53:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.90700000000000003,"15min":0.871},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.41999999999999998},"max":{"1min":1.2709999999999999,"5min":6.9930000000000003,"15min":6.9930000000000003},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85399999999999998,"15min":0.85299999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0940000000000001,"5min":1.2410000000000001,"15min":1.9810000000000001},"last":0.70999999999999996}]},{"osd":2,"last update":"Mon Oct 30 13:53:40 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.91900000000000004,"15min":0.89000000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.4339999999999999,"5min":6.8550000000000004,"15min":6.8550000000000004},"last":1},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.96099999999999997,"15min":0.95899999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.5169999999999999,"5min":5.2030000000000003,"15min":5.2030000000000003},"last":0.754}]},{"osd":3,"last update":"Mon Oct 30 13:53:40 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.88700000000000001,"15min":0.879},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.377,"5min":5.1740000000000004,"15min":5.1740000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.96599999999999997,"15min":0.96799999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.254,"5min":5.3179999999999996,"15min":5.3179999999999996},"last":0.96299999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:53:40 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94699999999999995,"15min":1.0309999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":15.372},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.0580000000000001,"15min":1.0329999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.468,"5min":5.2729999999999997,"15min":6.1580000000000004},"last":0.81699999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:53:40 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":1.004,"15min":1.0149999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.8080000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.165,"15min":1.177},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.794,"5min":1.794,"15min":1.913},"last":1.153}]},{"osd":7,"last update":"Mon Oct 30 13:53:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.0600000000000001,"15min":1.083},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.157,"5min":1.6659999999999999,"15min":2.202},"last":1.071},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.1839999999999999,"15min":1.1899999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":3.0590000000000002},"last":1.119}]}]},{"osd":5,"up_from":13,"seq":55834575375,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694772,"kb_used_data":519276,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":93677068,"statfs":{"total":96636764160,"available":95925317632,"internally_reserved":0,"allocated":531738624,"data_stored":518585685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179698118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.85199999999999998,"15min":0.86099999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.0940000000000001,"5min":1.4379999999999999,"15min":6.4729999999999999},"last":1.085},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.92000000000000004,"15min":0.88200000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.129,"5min":3.585,"15min":8.7560000000000002},"last":0.97099999999999997}]},{"osd":1,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.85699999999999998,"15min":0.88600000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.175,"5min":3.6360000000000001,"15min":15.257999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.747,"5min":0.88600000000000001,"15min":0.95599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.423,"5min":4.0439999999999996,"15min":15.175000000000001},"last":0.93500000000000005}]},{"osd":2,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.86799999999999999,"15min":0.90300000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.42999999999999999},"max":{"1min":1.1799999999999999,"5min":3.0179999999999998,"15min":6.5410000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.90400000000000003,"15min":0.97699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.2190000000000001,"5min":2.8879999999999999,"15min":6.7080000000000002},"last":0.85599999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.071,"15min":1.04},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.357,"5min":5.4249999999999998,"15min":6.7699999999999996},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.1359999999999999,"15min":1.1699999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.40100000000000002},"max":{"1min":1.4730000000000001,"5min":5.5570000000000004,"15min":15.364000000000001},"last":1.012}]},{"osd":4,"last update":"Mon Oct 30 13:53:51 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.92200000000000004,"15min":0.96199999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.6859999999999999,"5min":4.3529999999999998,"15min":5.9720000000000004},"last":1.117},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.1859999999999999,"15min":1.1419999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.4670000000000001,"5min":5.6609999999999996,"15min":14.314},"last":1.2270000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":1.0329999999999999,"15min":1.113},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.583,"5min":4.2290000000000001,"15min":15.06},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.996,"5min":1.1359999999999999,"15min":1.177},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.744,"5min":3.508,"15min":7.6630000000000003},"last":1.1679999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":1.0289999999999999,"15min":1.105},"min":{"1min":0.46300000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.98099999999999998,"5min":5.0419999999999998,"15min":6.8259999999999996},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.123,"5min":1.208,"15min":1.2729999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.494,"5min":2.8250000000000002,"15min":19.321999999999999},"last":1.3100000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575376,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726856,"kb_used_data":569408,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":93644984,"statfs":{"total":96636764160,"available":95892463616,"internally_reserved":0,"allocated":583073792,"data_stored":568869284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":161216966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.755,"15min":0.79700000000000004},"min":{"1min":0.32800000000000001,"5min":0.308,"15min":0.308},"max":{"1min":1.2130000000000001,"5min":2.202,"15min":9.6690000000000005},"last":1.083},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.84299999999999997,"15min":0.84499999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3069999999999999,"5min":2.0760000000000001,"15min":2.786},"last":1.0109999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86899999999999999,"15min":0.873},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.2609999999999999,"5min":1.8160000000000001,"15min":5.5119999999999996},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.90300000000000002,"15min":0.86399999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.2809999999999999,"5min":7.6120000000000001,"15min":7.6120000000000001},"last":0.94899999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:53:00 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.89900000000000002,"15min":0.88400000000000001},"min":{"1min":0.58099999999999996,"5min":0.53700000000000003,"15min":0.48299999999999998},"max":{"1min":1.508,"5min":2.1549999999999998,"15min":9.0099999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.93300000000000005,"15min":0.94199999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.0939999999999999,"5min":7.7359999999999998,"15min":7.7359999999999998},"last":0.77900000000000003}]},{"osd":3,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.93799999999999994,"15min":0.94699999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.4139999999999999,"5min":1.6000000000000001,"15min":4.7080000000000002},"last":1.179},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0369999999999999,"15min":1.0249999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3959999999999999,"5min":1.7509999999999999,"15min":3.2749999999999999},"last":0.84399999999999997}]},{"osd":5,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.97499999999999998,"15min":1.03},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":7.8529999999999998},"last":1.153},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1839999999999999,"15min":1.1759999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.9079999999999999,"5min":7.694,"15min":8.2070000000000007},"last":1.2030000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.119,"15min":1.0800000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.732,"5min":7.7880000000000003,"15min":7.7880000000000003},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.147,"15min":1.1539999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.796,"5min":3.7320000000000002,"15min":5.6130000000000004},"last":1.236}]},{"osd":7,"last update":"Mon Oct 30 13:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.1819999999999999,"15min":1.1719999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":2.347,"5min":7.8380000000000001,"15min":10.715},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.99199999999999999,"15min":1.1240000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.1960000000000002,"5min":2.2450000000000001,"15min":4.3230000000000004},"last":1.117}]}]},{"osd":3,"up_from":13,"seq":55834575374,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737956,"kb_used_data":550044,"kb_used_omap":1,"kb_used_meta":187902,"kb_avail":93633884,"statfs":{"total":96636764160,"available":95881097216,"internally_reserved":0,"allocated":563245056,"data_stored":549099638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":192412104},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42699999999999999,"15min":0.40899999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.23000000000000001},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.625},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.45200000000000001,"15min":0.46800000000000003},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70899999999999996,"5min":0.76700000000000002,"15min":1.5429999999999999},"last":0.57599999999999996}]},{"osd":1,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.504,"15min":0.46700000000000003},"min":{"1min":0.32400000000000001,"5min":0.314,"15min":0.27900000000000003},"max":{"1min":0.65600000000000003,"5min":0.76200000000000001,"15min":0.94099999999999995},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.246},"max":{"1min":0.83099999999999996,"5min":0.92000000000000004,"15min":1.018},"last":0.66500000000000004}]},{"osd":2,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.503,"15min":0.51000000000000001},"min":{"1min":0.35099999999999998,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.81599999999999995,"5min":1.056,"15min":1.4690000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.39500000000000002,"5min":0.33400000000000002,"15min":0.27500000000000002},"max":{"1min":0.77100000000000002,"5min":0.86499999999999999,"15min":1.6799999999999999},"last":0.78900000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:53:07 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.78200000000000003,"15min":0.82499999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.97699999999999998,"5min":1.1739999999999999,"15min":1.5900000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.83199999999999996,"15min":0.84599999999999997},"min":{"1min":0.56100000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1,"5min":1.1419999999999999,"15min":1.3999999999999999},"last":1.0409999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.76700000000000002,"15min":0.79600000000000004},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.51800000000000002},"max":{"1min":0.86299999999999999,"5min":1.1879999999999999,"15min":1.5069999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.81000000000000005,"15min":0.81899999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.5},"max":{"1min":0.95999999999999996,"5min":1.232,"15min":1.365},"last":1.075}]},{"osd":6,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.85199999999999998,"15min":0.82599999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.45400000000000001},"max":{"1min":1.05,"5min":1.256,"15min":1.256},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.87,"15min":0.82899999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.498},"max":{"1min":1.0760000000000001,"5min":1.3069999999999999,"15min":2.3410000000000002},"last":1.0089999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:52:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.86799999999999999,"15min":0.876},"min":{"1min":0.61599999999999999,"5min":0.57799999999999996,"15min":0.51700000000000002},"max":{"1min":1.0569999999999999,"5min":1.3320000000000001,"15min":2.9449999999999998},"last":1.105},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.86099999999999999,"15min":0.88200000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":1.0780000000000001,"5min":1.343,"15min":2.9729999999999999},"last":1.0549999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575377,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511028,"kb_used_data":377260,"kb_used_omap":1,"kb_used_meta":133758,"kb_avail":93860812,"statfs":{"total":96636764160,"available":96113471488,"internally_reserved":0,"allocated":386314240,"data_stored":377764906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":136968646},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.54400000000000004,"15min":0.47599999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.23300000000000001},"max":{"1min":9.8249999999999993,"5min":9.8249999999999993,"15min":9.8249999999999993},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53600000000000003,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.32100000000000001,"15min":0.25700000000000001},"max":{"1min":2.7200000000000002,"5min":4.2910000000000004,"15min":4.2910000000000004},"last":0.45300000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.64800000000000002,"15min":0.58799999999999997},"min":{"1min":0.38700000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":9.7420000000000009,"5min":9.7420000000000009,"15min":9.7420000000000009},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58699999999999997,"15min":0.54200000000000004},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.2839999999999998,"5min":7.0060000000000002,"15min":7.0060000000000002},"last":0.56599999999999995}]},{"osd":3,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.52900000000000003,"15min":0.504},"min":{"1min":0.371,"5min":0.35099999999999998,"15min":0.28299999999999997},"max":{"1min":2.8119999999999998,"5min":2.8119999999999998,"15min":2.8119999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.64500000000000002,"15min":0.56999999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.33700000000000002},"max":{"1min":2.843,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79800000000000004,"15min":0.80300000000000005},"min":{"1min":0.56200000000000006,"5min":0.442,"15min":0.442},"max":{"1min":2.4129999999999998,"5min":4.1970000000000001,"15min":4.1970000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.37},"max":{"1min":2.681,"5min":4.1479999999999997,"15min":4.1479999999999997},"last":0.89600000000000002}]},{"osd":5,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.89300000000000002,"15min":0.88200000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.50900000000000001},"max":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7989999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.79500000000000004,"15min":0.82499999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"max":{"1min":2.4750000000000001,"5min":2.4750000000000001,"15min":2.762},"last":0.91900000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":0.84499999999999997,"15min":0.81599999999999995},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.47699999999999998},"max":{"1min":9.7210000000000001,"5min":9.7210000000000001,"15min":9.7210000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.2050000000000001,"5min":0.91200000000000003,"15min":0.84599999999999997},"min":{"1min":0.622,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":9.7889999999999997,"5min":9.7889999999999997,"15min":9.7889999999999997},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":0.97299999999999998,"15min":0.875},"min":{"1min":0.629,"5min":0.58999999999999997,"15min":0.503},"max":{"1min":6.8529999999999998,"5min":6.9059999999999997,"15min":6.9059999999999997},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.0589999999999999,"15min":0.97099999999999997},"min":{"1min":0.64000000000000001,"5min":0.63700000000000001,"15min":0.55900000000000005},"max":{"1min":6.8929999999999998,"5min":6.9509999999999996,"15min":6.9509999999999996},"last":0.83699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575374,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683292,"kb_used_data":535636,"kb_used_omap":1,"kb_used_meta":147646,"kb_avail":93688548,"statfs":{"total":96636764160,"available":95937073152,"internally_reserved":0,"allocated":548491264,"data_stored":539410681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151189958},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:53:46 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.22800000000000001},"max":{"1min":0.63400000000000001,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.251},"max":{"1min":0.77100000000000002,"5min":0.84099999999999997,"15min":0.96699999999999997},"last":0.501}]},{"osd":2,"last update":"Mon Oct 30 13:53:46 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47599999999999998,"15min":0.48999999999999999},"min":{"1min":0.30099999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":1.002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.505,"15min":0.48399999999999999},"min":{"1min":0.34899999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68100000000000005,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.68500000000000005}]},{"osd":3,"last update":"Mon Oct 30 13:53:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.35999999999999999,"5min":0.34999999999999998,"15min":0.314},"max":{"1min":1.004,"5min":1.004,"15min":1.2569999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.306},"last":0.66500000000000004}]},{"osd":4,"last update":"Mon Oct 30 13:53:46 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.80400000000000005,"15min":0.82899999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.40200000000000002},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":5.96},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88400000000000001,"15min":0.89900000000000002},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.53300000000000003},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":5.9909999999999997},"last":1.032}]},{"osd":5,"last update":"Mon Oct 30 13:53:46 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77000000000000002,"15min":0.76300000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":2.133},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79300000000000004,"15min":0.79400000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51000000000000001},"max":{"1min":1.2350000000000001,"5min":1.462,"15min":1.462},"last":0.997}]},{"osd":6,"last update":"Mon Oct 30 13:53:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90900000000000003,"15min":0.89500000000000002},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.501},"max":{"1min":1.403,"5min":1.403,"15min":6.0289999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.95399999999999996,"15min":0.94999999999999996},"min":{"1min":0.59799999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":6.0739999999999998},"last":1.1160000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:53:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80100000000000005,"15min":0.77900000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86699999999999999,"15min":0.873},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":6.0519999999999996},"last":1.1579999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575376,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716656,"kb_used_data":568936,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93655184,"statfs":{"total":96636764160,"available":95902908416,"internally_reserved":0,"allocated":582590464,"data_stored":569154632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151255494},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.40100000000000002,"15min":0.47599999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":0.54400000000000004,"5min":2.0630000000000002,"15min":16.184000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.50900000000000001,"15min":0.59099999999999997},"min":{"1min":0.36199999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.629,"5min":2.0979999999999999,"15min":16.228000000000002},"last":0.61699999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.45300000000000001,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.70499999999999996,"5min":2.7999999999999998,"15min":17.454000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.46800000000000003,"15min":0.64700000000000002},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":0.61799999999999999,"5min":1.391,"15min":23.212},"last":0.45600000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51100000000000001,"15min":0.62},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":1.4690000000000001,"15min":16.103999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.52700000000000002,"15min":0.72399999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60199999999999998,"5min":2.8340000000000001,"15min":23.16},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon Oct 30 13:52:53 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.81899999999999995,"15min":0.91800000000000004},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.39200000000000002},"max":{"1min":0.96399999999999997,"5min":7.6900000000000004,"15min":16.309000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.82099999999999995,"15min":0.95999999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97399999999999998,"5min":3.4009999999999998,"15min":16.343},"last":0.79400000000000004}]},{"osd":5,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.81599999999999995,"15min":0.995},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94299999999999995,"5min":2.02,"15min":23.266999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.86599999999999999,"15min":1.0329999999999999},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.999,"5min":2.7240000000000002,"15min":16.268000000000001},"last":0.93999999999999995}]},{"osd":6,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.79400000000000004,"15min":1.091},"min":{"1min":0.52500000000000002,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":1.0269999999999999,"5min":2.6560000000000001,"15min":29.774999999999999},"last":0.874},{"interface":"front","average":{"1min":0.752,"5min":0.80600000000000005,"15min":1.0149999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":2.75,"15min":17.5},"last":0.83399999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:52:52 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.81699999999999995,"15min":1.1919999999999999},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.51300000000000001},"max":{"1min":1.0229999999999999,"5min":2.694,"15min":47.534999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":1.147},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.54700000000000004},"max":{"1min":1.272,"5min":2.8690000000000002,"15min":32.476999999999997},"last":0.92400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":230703104,"data_stored":230687380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258846720,"data_stored":258822445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179036160,"data_stored":179013521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230858752,"data_stored":230840931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":197275648,"data_stored":197263919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":205541376,"data_stored":205522290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":268918784,"data_stored":268902222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":247238656,"data_stored":247218354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350515200,"data_stored":337464605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288862208,"data_stored":280174938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":206495744,"data_stored":198338087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331603968,"data_stored":317845409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":385015808,"data_stored":371192067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":325414912,"data_stored":312650097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193724416,"data_stored":185512357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342437888,"data_stored":329203526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:53:58.869 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T13:53:59.043 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.040+0000 7f1ebcb65640 1 Processor -- start 2023-10-30T13:53:59.044 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.040+0000 7f1ebcb65640 1 -- start start 2023-10-30T13:53:59.045 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1ebcb65640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1eb81508c0 0x7f1eb80c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1ebcb65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 0x7f1eb80c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1ebcb65640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1eb80bfec0 0x7f1eb80c0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.046 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1ebcb65640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb80c85b0 con 0x7f1eb81508c0 2023-10-30T13:53:59.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1ebcb65640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb80c0a30 con 0x7f1eb80bfec0 2023-10-30T13:53:59.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1ebcb65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb806f650 con 0x7f1eb80722a0 2023-10-30T13:53:59.047 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1eb81508c0 0x7f1eb80c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb659c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1eb81508c0 0x7f1eb80c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39000/0 (socket says 172.21.15.148:39000) 2023-10-30T13:53:59.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb659c640 1 -- 172.21.15.148:0/1929816887 learned_addr learned my addr 172.21.15.148:0/1929816887 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:53:59.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb6d9d640 1 --2- 172.21.15.148:0/1929816887 >> v2:172.21.15.150:3300/0 conn(0x7f1eb80bfec0 0x7f1eb80c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.048 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eadd9b640 1 --2- 172.21.15.148:0/1929816887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 0x7f1eb80c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb659c640 1 -- 172.21.15.148:0/1929816887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 msgr2=0x7f1eb80c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:59.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb659c640 1 --2- 172.21.15.148:0/1929816887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 0x7f1eb80c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.049 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb659c640 1 -- 172.21.15.148:0/1929816887 >> v2:172.21.15.150:3300/0 conn(0x7f1eb80bfec0 msgr2=0x7f1eb80c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:59.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb659c640 1 --2- 172.21.15.148:0/1929816887 >> v2:172.21.15.150:3300/0 conn(0x7f1eb80bfec0 0x7f1eb80c0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb659c640 1 -- 172.21.15.148:0/1929816887 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb806f930 con 0x7f1eb81508c0 2023-10-30T13:53:59.050 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb6d9d640 1 --2- 172.21.15.148:0/1929816887 >> v2:172.21.15.150:3300/0 conn(0x7f1eb80bfec0 0x7f1eb80c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:53:59.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.044+0000 7f1eb659c640 1 --2- 172.21.15.148:0/1929816887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 0x7f1eb80c6920 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1eb806f1b0 tx=0x7f1ea801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf2eed24470d9cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:59.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1eb5d9b640 1 -- 172.21.15.148:0/1929816887 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea80ab600 con 0x7f1eb81508c0 2023-10-30T13:53:59.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1eb5d9b640 1 -- 172.21.15.148:0/1929816887 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea80a7050 con 0x7f1eb81508c0 2023-10-30T13:53:59.052 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1eb5d9b640 1 -- 172.21.15.148:0/1929816887 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea80ab980 con 0x7f1eb81508c0 2023-10-30T13:53:59.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 -- 172.21.15.148:0/1929816887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 msgr2=0x7f1eb80c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/1929816887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 0x7f1eb80c6920 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1eb806f1b0 tx=0x7f1ea801c470 comp rx=0 tx=0).stop 2023-10-30T13:53:59.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 -- 172.21.15.148:0/1929816887 shutdown_connections 2023-10-30T13:53:59.053 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/1929816887 >> v2:172.21.15.150:3300/0 conn(0x7f1eb80bfec0 0x7f1eb80c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/1929816887 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 0x7f1eb80c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/1929816887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 0x7f1eb80c6920 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.054 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 -- 172.21.15.148:0/1929816887 >> 172.21.15.148:0/1929816887 conn(0x7f1eb80c7910 msgr2=0x7f1eb80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:59.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 -- 172.21.15.148:0/1929816887 shutdown_connections 2023-10-30T13:53:59.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 -- 172.21.15.148:0/1929816887 wait complete. 2023-10-30T13:53:59.055 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 Processor -- start 2023-10-30T13:53:59.056 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 -- start start 2023-10-30T13:53:59.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 0x7f1eb80ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.058 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eb80bfec0 0x7f1eb80c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 0x7f1eb80ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8154f60 con 0x7f1eb81508c0 2023-10-30T13:53:59.059 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80b0840 con 0x7f1eb80bfec0 2023-10-30T13:53:59.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.048+0000 7f1ebcb65640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80adfc0 con 0x7f1eb80722a0 2023-10-30T13:53:59.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 0x7f1eb80ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 0x7f1eb80ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39012/0 (socket says 172.21.15.148:39012) 2023-10-30T13:53:59.060 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb6d9d640 1 -- 172.21.15.148:0/2459768789 learned_addr learned my addr 172.21.15.148:0/2459768789 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:53:59.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eadd9b640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eb80bfec0 0x7f1eb80c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb659c640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 0x7f1eb80ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.061 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb6d9d640 1 -- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 msgr2=0x7f1eb80ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb6d9d640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 0x7f1eb80ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb6d9d640 1 -- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eb80bfec0 msgr2=0x7f1eb80c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb6d9d640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eb80bfec0 0x7f1eb80c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.062 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb6d9d640 1 -- 172.21.15.148:0/2459768789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8009480 con 0x7f1eb81508c0 2023-10-30T13:53:59.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eb6d9d640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 0x7f1eb80ca530 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000e810 tx=0x7f1eb000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:59.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eaf7fe640 1 -- 172.21.15.148:0/2459768789 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb000c6d0 con 0x7f1eb81508c0 2023-10-30T13:53:59.063 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eaf7fe640 1 -- 172.21.15.148:0/2459768789 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb000c830 con 0x7f1eb81508c0 2023-10-30T13:53:59.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb8154520 con 0x7f1eb81508c0 2023-10-30T13:53:59.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eaf7fe640 1 -- 172.21.15.148:0/2459768789 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb000cbe0 con 0x7f1eb81508c0 2023-10-30T13:53:59.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb80b0540 con 0x7f1eb81508c0 2023-10-30T13:53:59.064 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb8154760 con 0x7f1eb81508c0 2023-10-30T13:53:59.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.052+0000 7f1eaf7fe640 1 -- 172.21.15.148:0/2459768789 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1eb000cd40 con 0x7f1eb81508c0 2023-10-30T13:53:59.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.056+0000 7f1eaf7fe640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e980664a0 0x7f1e98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.065 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.056+0000 7f1eaf7fe640 1 -- 172.21.15.148:0/2459768789 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f1eb0084430 con 0x7f1eb81508c0 2023-10-30T13:53:59.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.056+0000 7f1eb659c640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e980664a0 0x7f1e98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.056+0000 7f1eb659c640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e980664a0 0x7f1e98068960 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80ad000 tx=0x7f1ea80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:59.066 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.060+0000 7f1eaf7fe640 1 -- 172.21.15.148:0/2459768789 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1eb8154760 con 0x7f1eb81508c0 2023-10-30T13:53:59.212 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.208+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f1eb8154760 con 0x7f1eb81508c0 2023-10-30T13:53:59.216 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.212+0000 7f1eaf7fe640 1 -- 172.21.15.148:0/2459768789 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7f1eb8154760 con 0x7f1eb81508c0 2023-10-30T13:53:59.218 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1eaf7fe640 1 -- 172.21.15.148:0/2459768789 <== mon.0 v2:172.21.15.148:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1eb0009400 con 0x7f1eb81508c0 2023-10-30T13:53:59.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e980664a0 msgr2=0x7f1e98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e980664a0 0x7f1e98068960 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80ad000 tx=0x7f1ea80a6000 comp rx=0 tx=0).stop 2023-10-30T13:53:59.220 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 msgr2=0x7f1eb80ca530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 0x7f1eb80ca530 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000e810 tx=0x7f1eb000ecd0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 shutdown_connections 2023-10-30T13:53:59.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f1e980664a0 0x7f1e98068960 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.221 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1eb81508c0 0x7f1eb80ca530 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1eb80bfec0 0x7f1eb80c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 --2- 172.21.15.148:0/2459768789 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f1eb80722a0 0x7f1eb80ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.222 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 >> 172.21.15.148:0/2459768789 conn(0x7f1eb80c7910 msgr2=0x7f1eb80c1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:59.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 shutdown_connections 2023-10-30T13:53:59.223 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.216+0000 7f1ebcb65640 1 -- 172.21.15.148:0/2459768789 wait complete. 2023-10-30T13:53:59.238 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-30T13:53:59.239 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:59.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb43c06b640 1 Processor -- start 2023-10-30T13:53:59.378 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb43c06b640 1 -- start start 2023-10-30T13:53:59.380 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb43c06b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fb43415d6f0 0x7fb43415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb43c06b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4340cf160 0x7fb4340cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb43c06b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4340cb800 con 0x7fb4341538a0 2023-10-30T13:53:59.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb43c06b640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4340cb330 con 0x7fb43415d6f0 2023-10-30T13:53:59.381 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb43c06b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340cad70 con 0x7fb4340cf160 2023-10-30T13:53:59.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb439e07640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb4341538a0 0x7fb4340ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49798/0 (socket says 172.21.15.148:49798) 2023-10-30T13:53:59.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb439e07640 1 -- 172.21.15.148:0/1225799437 learned_addr learned my addr 172.21.15.148:0/1225799437 (peer_addr_for_me v1:172.21.15.148:0/0) 2023-10-30T13:53:59.382 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb439606640 1 --2- 172.21.15.148:0/1225799437 >> v2:172.21.15.148:3300/0 conn(0x7fb43415d6f0 0x7fb43415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb43a608640 1 --2- 172.21.15.148:0/1225799437 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4340cf160 0x7fb4340cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb438e05640 1 -- 172.21.15.148:0/1225799437 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627153732 0 0) 0x7fb4340cb800 con 0x7fb4341538a0 2023-10-30T13:53:59.383 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb439606640 1 -- 172.21.15.148:0/1225799437 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4340cf160 msgr2=0x7fb4340cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:59.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb439606640 1 --2- 172.21.15.148:0/1225799437 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4340cf160 0x7fb4340cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb439606640 1 -- 172.21.15.148:0/1225799437 >> v1:172.21.15.148:6789/0 conn(0x7fb4341538a0 legacy=0x7fb4340ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.384 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.376+0000 7fb439606640 1 -- 172.21.15.148:0/1225799437 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4340cb050 con 0x7fb43415d6f0 2023-10-30T13:53:59.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb439606640 1 --2- 172.21.15.148:0/1225799437 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb43415d6f0 0x7fb43415a090 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fb4240097c0 tx=0x7fb42401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322f38f32bcb8604 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:59.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb438e05640 1 -- 172.21.15.148:0/1225799437 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4240a8970 con 0x7fb43415d6f0 2023-10-30T13:53:59.386 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb438e05640 1 -- 172.21.15.148:0/1225799437 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb4240a8ad0 con 0x7fb43415d6f0 2023-10-30T13:53:59.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb438e05640 1 -- 172.21.15.148:0/1225799437 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4240a8e80 con 0x7fb43415d6f0 2023-10-30T13:53:59.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb43c06b640 1 -- 172.21.15.148:0/1225799437 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb43415d6f0 msgr2=0x7fb43415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.387 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb43c06b640 1 --2- 172.21.15.148:0/1225799437 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb43415d6f0 0x7fb43415a090 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fb4240097c0 tx=0x7fb42401ec70 comp rx=0 tx=0).stop 2023-10-30T13:53:59.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb43c06b640 1 -- 172.21.15.148:0/1225799437 shutdown_connections 2023-10-30T13:53:59.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb43c06b640 1 --2- 172.21.15.148:0/1225799437 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb43415d6f0 0x7fb43415a090 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb43c06b640 1 --2- 172.21.15.148:0/1225799437 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb4340cf160 0x7fb4340cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.388 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb43c06b640 1 -- 172.21.15.148:0/1225799437 >> 172.21.15.148:0/1225799437 conn(0x7fb43406bfe0 msgr2=0x7fb43406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:59.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb43c06b640 1 -- 172.21.15.148:0/1225799437 shutdown_connections 2023-10-30T13:53:59.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.380+0000 7fb43c06b640 1 -- 172.21.15.148:0/1225799437 wait complete. 2023-10-30T13:53:59.389 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 Processor -- start 2023-10-30T13:53:59.390 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 -- start start 2023-10-30T13:53:59.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4340cf160 0x7fb4341e5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.392 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4341538a0 0x7fb43415c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb43415d6f0 0x7fb4341f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340c7cb0 con 0x7fb4340cf160 2023-10-30T13:53:59.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4341e57c0 con 0x7fb4341538a0 2023-10-30T13:53:59.393 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4341e4740 con 0x7fb43415d6f0 2023-10-30T13:53:59.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439e07640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4340cf160 0x7fb4341e5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439e07640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4340cf160 0x7fb4341e5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39020/0 (socket says 172.21.15.148:39020) 2023-10-30T13:53:59.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439e07640 1 -- 172.21.15.148:0/2886668986 learned_addr learned my addr 172.21.15.148:0/2886668986 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:53:59.394 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439606640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4341538a0 0x7fb43415c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43a608640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb43415d6f0 0x7fb4341f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439e07640 1 -- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb43415d6f0 msgr2=0x7fb4341f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.395 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439e07640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb43415d6f0 0x7fb4341f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439e07640 1 -- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4341538a0 msgr2=0x7fb43415c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439e07640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4341538a0 0x7fb43415c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439e07640 1 -- 172.21.15.148:0/2886668986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb424009420 con 0x7fb4340cf160 2023-10-30T13:53:59.396 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb439e07640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4340cf160 0x7fb4341e5280 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fb4300011f0 tx=0x7fb43000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:59.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb42a7fc640 1 -- 172.21.15.148:0/2886668986 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4300074e0 con 0x7fb4340cf160 2023-10-30T13:53:59.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb42a7fc640 1 -- 172.21.15.148:0/2886668986 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb430007640 con 0x7fb4340cf160 2023-10-30T13:53:59.397 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb42a7fc640 1 -- 172.21.15.148:0/2886668986 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430004a30 con 0x7fb4340cf160 2023-10-30T13:53:59.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4340c8950 con 0x7fb4340cf160 2023-10-30T13:53:59.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.384+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4341e4c30 con 0x7fb4340cf160 2023-10-30T13:53:59.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.388+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4340b2a50 con 0x7fb4340cf160 2023-10-30T13:53:59.398 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.388+0000 7fb42a7fc640 1 -- 172.21.15.148:0/2886668986 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb430004b90 con 0x7fb4340cf160 2023-10-30T13:53:59.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.388+0000 7fb42a7fc640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4180664a0 0x7fb418068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.388+0000 7fb42a7fc640 1 -- 172.21.15.148:0/2886668986 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fb4300837e0 con 0x7fb4340cf160 2023-10-30T13:53:59.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.388+0000 7fb439606640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4180664a0 0x7fb418068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.399 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.388+0000 7fb439606640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4180664a0 0x7fb418068960 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb4240b2000 tx=0x7fb424008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:59.400 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.392+0000 7fb42a7fc640 1 -- 172.21.15.148:0/2886668986 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4300504d0 con 0x7fb4340cf160 2023-10-30T13:53:59.576 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.572+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb43404e170 con 0x7fb4340cf160 2023-10-30T13:53:59.577 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb42a7fc640 1 -- 172.21.15.148:0/2886668986 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6534 (secure 0 0 0) 0x7fb430053af0 con 0x7fb4340cf160 2023-10-30T13:53:59.578 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:53:59.578 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2057257086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2057257086},{"type":"v1","addr":"172.21.15.148:6837","nonce":2057257086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:39.103468+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:59.578 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 166 2023-10-30T13:53:59.579 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4180664a0 msgr2=0x7fb418068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4180664a0 0x7fb418068960 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb4240b2000 tx=0x7fb424008040 comp rx=0 tx=0).stop 2023-10-30T13:53:59.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4340cf160 msgr2=0x7fb4341e5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4340cf160 0x7fb4341e5280 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fb4300011f0 tx=0x7fb43000c9b0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.580 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 shutdown_connections 2023-10-30T13:53:59.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fb4180664a0 0x7fb418068960 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fb43415d6f0 0x7fb4341f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.581 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4341538a0 0x7fb43415c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 --2- 172.21.15.148:0/2886668986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb4340cf160 0x7fb4341e5280 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 >> 172.21.15.148:0/2886668986 conn(0x7fb43406bfe0 msgr2=0x7fb43406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:59.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 shutdown_connections 2023-10-30T13:53:59.582 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.576+0000 7fb43c06b640 1 -- 172.21.15.148:0/2886668986 wait complete. 2023-10-30T13:53:59.598 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:53:59.735 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.732+0000 7f17a8f3e640 1 Processor -- start 2023-10-30T13:53:59.736 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.732+0000 7f17a8f3e640 1 -- start start 2023-10-30T13:53:59.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.732+0000 7f17a8f3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a40c6b80 0x7f17a40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.732+0000 7f17a8f3e640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f17a413f4a0 0x7f17a40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.732+0000 7f17a8f3e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a40c0a50 con 0x7f17a40c6f60 2023-10-30T13:53:59.737 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.732+0000 7f17a8f3e640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a406f6e0 con 0x7f17a413f4a0 2023-10-30T13:53:59.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.732+0000 7f17a8f3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a406f9c0 con 0x7f17a40c6b80 2023-10-30T13:53:59.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a2d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f17a413f4a0 0x7f17a40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.738 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a2d9d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f17a413f4a0 0x7f17a40c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39022/0 (socket says 172.21.15.148:39022) 2023-10-30T13:53:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a2d9d640 1 -- 172.21.15.148:0/3165797516 learned_addr learned my addr 172.21.15.148:0/3165797516 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:53:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a259c640 1 --2- 172.21.15.148:0/3165797516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a40c6b80 0x7f17a40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a2d9d640 1 -- 172.21.15.148:0/3165797516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a40c6b80 msgr2=0x7f17a40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:53:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a2d9d640 1 --2- 172.21.15.148:0/3165797516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a40c6b80 0x7f17a40bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a2d9d640 1 -- 172.21.15.148:0/3165797516 >> v1:172.21.15.150:6789/0 conn(0x7f17a40c6f60 legacy=0x7f17a413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a2d9d640 1 -- 172.21.15.148:0/3165797516 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a406fca0 con 0x7f17a413f4a0 2023-10-30T13:53:59.741 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a2d9d640 1 --2- 172.21.15.148:0/3165797516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a413f4a0 0x7f17a40c0400 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f179c00d150 tx=0x7f179c01ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bcabb194683d6a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:59.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a1d9b640 1 -- 172.21.15.148:0/3165797516 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c00b2d0 con 0x7f17a413f4a0 2023-10-30T13:53:59.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a1d9b640 1 -- 172.21.15.148:0/3165797516 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f179c00b430 con 0x7f17a413f4a0 2023-10-30T13:53:59.742 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.736+0000 7f17a1d9b640 1 -- 172.21.15.148:0/3165797516 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c0a7c30 con 0x7f17a413f4a0 2023-10-30T13:53:59.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3165797516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a413f4a0 msgr2=0x7f17a40c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 --2- 172.21.15.148:0/3165797516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a413f4a0 0x7f17a40c0400 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f179c00d150 tx=0x7f179c01ec10 comp rx=0 tx=0).stop 2023-10-30T13:53:59.743 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3165797516 shutdown_connections 2023-10-30T13:53:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 --2- 172.21.15.148:0/3165797516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a413f4a0 0x7f17a40c0400 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 --2- 172.21.15.148:0/3165797516 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a40c6b80 0x7f17a40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3165797516 >> 172.21.15.148:0/3165797516 conn(0x7f17a40c7910 msgr2=0x7f17a40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:59.744 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3165797516 shutdown_connections 2023-10-30T13:53:59.745 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3165797516 wait complete. 2023-10-30T13:53:59.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 Processor -- start 2023-10-30T13:53:59.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 -- start start 2023-10-30T13:53:59.747 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17a40c6b80 0x7f17a4135180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a40c6f60 0x7f17a41edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a413f4a0 0x7f17a41ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a41f1240 con 0x7f17a40c6f60 2023-10-30T13:53:59.748 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a41f1d80 con 0x7f17a40c6b80 2023-10-30T13:53:59.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a8f3e640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a413a6e0 con 0x7f17a413f4a0 2023-10-30T13:53:59.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f179bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a40c6f60 0x7f17a41edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f179bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a40c6f60 0x7f17a41edcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39030/0 (socket says 172.21.15.148:39030) 2023-10-30T13:53:59.749 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f179bfff640 1 -- 172.21.15.148:0/3895790957 learned_addr learned my addr 172.21.15.148:0/3895790957 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:53:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a259c640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17a40c6b80 0x7f17a4135180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.740+0000 7f17a2d9d640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a413f4a0 0x7f17a41ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179bfff640 1 -- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a413f4a0 msgr2=0x7f17a41ee210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179bfff640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a413f4a0 0x7f17a41ee210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179bfff640 1 -- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17a40c6b80 msgr2=0x7f17a4135180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179bfff640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17a40c6b80 0x7f17a4135180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179bfff640 1 -- 172.21.15.148:0/3895790957 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c00bd40 con 0x7f17a40c6f60 2023-10-30T13:53:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179bfff640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a40c6f60 0x7f17a41edcd0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f178c00c4d0 tx=0x7f178c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179affd640 1 -- 172.21.15.148:0/3895790957 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c0052d0 con 0x7f17a40c6f60 2023-10-30T13:53:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179affd640 1 -- 172.21.15.148:0/3895790957 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f178c005430 con 0x7f17a40c6f60 2023-10-30T13:53:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179affd640 1 -- 172.21.15.148:0/3895790957 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c012850 con 0x7f17a40c6f60 2023-10-30T13:53:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a4154df0 con 0x7f17a40c6f60 2023-10-30T13:53:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a41f1450 con 0x7f17a40c6f60 2023-10-30T13:53:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a404e170 con 0x7f17a40c6f60 2023-10-30T13:53:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179affd640 1 -- 172.21.15.148:0/3895790957 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f178c0129b0 con 0x7f17a40c6f60 2023-10-30T13:53:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.744+0000 7f179affd640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17840664a0 0x7f1784068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:53:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.748+0000 7f179affd640 1 -- 172.21.15.148:0/3895790957 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f178c084100 con 0x7f17a40c6f60 2023-10-30T13:53:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.748+0000 7f17a259c640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17840664a0 0x7f1784068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:53:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.748+0000 7f17a259c640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17840664a0 0x7f1784068960 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f1794009b60 tx=0x7f1794009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:53:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.752+0000 7f179affd640 1 -- 172.21.15.148:0/3895790957 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f178c050d70 con 0x7f17a40c6f60 2023-10-30T13:53:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17a406c5c0 con 0x7f17a40c6f60 2023-10-30T13:53:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f179affd640 1 -- 172.21.15.148:0/3895790957 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6534 (secure 0 0 0) 0x7f178c052ca0 con 0x7f17a40c6f60 2023-10-30T13:53:59.934 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:53:59.934 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1180930508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1180930508},{"type":"v1","addr":"172.21.15.150:6837","nonce":1180930508}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3684521078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3684521078},{"type":"v1","addr":"172.21.15.150:6839","nonce":3684521078}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5400,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6834/650045128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":650045128},{"type":"v1","addr":"172.21.15.148:6834","nonce":650045128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":8313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2057257086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2057257086},{"type":"v1","addr":"172.21.15.148:6837","nonce":2057257086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T13:10:34.058265+0000","modified":"2023-10-30T13:51:39.103468+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8145,"mds_1":8041},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8041":{"gid":8041,"name":"d","rank":1,"incarnation":152,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6836/1947781173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1947781173},{"type":"v1","addr":"172.21.15.150:6836","nonce":1947781173}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8145":{"gid":8145,"name":"a","rank":0,"incarnation":159,"state":"up:active","state_seq":28,"addr":"172.21.15.148:6835/1526234342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6833","nonce":1526234342},{"type":"v1","addr":"172.21.15.148:6835","nonce":1526234342}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:53:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:dumped fsmap epoch 166 2023-10-30T13:53:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17840664a0 msgr2=0x7f1784068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17840664a0 0x7f1784068960 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f1794009b60 tx=0x7f1794009190 comp rx=0 tx=0).stop 2023-10-30T13:53:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a40c6f60 msgr2=0x7f17a41edcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:53:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a40c6f60 0x7f17a41edcd0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f178c00c4d0 tx=0x7f178c00c990 comp rx=0 tx=0).stop 2023-10-30T13:53:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 shutdown_connections 2023-10-30T13:53:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f17840664a0 0x7f1784068960 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f17a40c6f60 0x7f17a41edcd0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f17a413f4a0 0x7f17a41ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 --2- 172.21.15.148:0/3895790957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17a40c6b80 0x7f17a4135180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:53:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 >> 172.21.15.148:0/3895790957 conn(0x7f17a40c7910 msgr2=0x7f17a4131400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:53:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 shutdown_connections 2023-10-30T13:53:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:53:59.932+0000 7f17a8f3e640 1 -- 172.21.15.148:0/3895790957 wait complete. 2023-10-30T13:53:59.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:54:00.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c23e4640 1 Processor -- start 2023-10-30T13:54:00.093 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c23e4640 1 -- start start 2023-10-30T13:54:00.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c23e4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f13bc1538f0 0x7f13bc159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c23e4640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f13bc0cefa0 0x7f13bc0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.095 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c23e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0ca6a0 con 0x7f13bc15cd50 2023-10-30T13:54:00.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c23e4640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc0cf8f0 con 0x7f13bc0cefa0 2023-10-30T13:54:00.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c23e4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc15d300 con 0x7f13bc1538f0 2023-10-30T13:54:00.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c1be3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f13bc0cefa0 0x7f13bc0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.096 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c1be3640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f13bc0cefa0 0x7f13bc0cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39036/0 (socket says 172.21.15.148:39036) 2023-10-30T13:54:00.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c0be1640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f13bc15cd50 0x7f13bc0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49802/0 (socket says 172.21.15.148:49802) 2023-10-30T13:54:00.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c1be3640 1 -- 172.21.15.148:0/88513870 learned_addr learned my addr 172.21.15.148:0/88513870 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:00.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c13e2640 1 --2- 172.21.15.148:0/88513870 >> v2:172.21.15.150:3300/0 conn(0x7f13bc1538f0 0x7f13bc159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.097 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c1be3640 1 -- 172.21.15.148:0/88513870 >> v2:172.21.15.150:3300/0 conn(0x7f13bc1538f0 msgr2=0x7f13bc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:00.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c1be3640 1 --2- 172.21.15.148:0/88513870 >> v2:172.21.15.150:3300/0 conn(0x7f13bc1538f0 0x7f13bc159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c1be3640 1 -- 172.21.15.148:0/88513870 >> v1:172.21.15.148:6789/0 conn(0x7f13bc15cd50 legacy=0x7f13bc0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.098 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13b3fff640 1 -- 172.21.15.148:0/88513870 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017692962 0 0) 0x7f13bc0ca6a0 con 0x7f13bc15cd50 2023-10-30T13:54:00.099 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.092+0000 7f13c1be3640 1 -- 172.21.15.148:0/88513870 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc15d5e0 con 0x7f13bc0cefa0 2023-10-30T13:54:00.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13c1be3640 1 --2- 172.21.15.148:0/88513870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc0cefa0 0x7f13bc0cf3b0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0c9cb0 tx=0x7f13b401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82af7d7493ef32c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13b3fff640 1 -- 172.21.15.148:0/88513870 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b40af400 con 0x7f13bc0cefa0 2023-10-30T13:54:00.100 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13b3fff640 1 -- 172.21.15.148:0/88513870 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f13b40ab050 con 0x7f13bc0cefa0 2023-10-30T13:54:00.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13b3fff640 1 -- 172.21.15.148:0/88513870 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b40af7a0 con 0x7f13bc0cefa0 2023-10-30T13:54:00.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13c23e4640 1 -- 172.21.15.148:0/88513870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc0cefa0 msgr2=0x7f13bc0cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13c23e4640 1 --2- 172.21.15.148:0/88513870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc0cefa0 0x7f13bc0cf3b0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0c9cb0 tx=0x7f13b401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.101 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13c23e4640 1 -- 172.21.15.148:0/88513870 shutdown_connections 2023-10-30T13:54:00.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13c23e4640 1 --2- 172.21.15.148:0/88513870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc0cefa0 0x7f13bc0cf3b0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13c23e4640 1 --2- 172.21.15.148:0/88513870 >> v2:172.21.15.150:3300/0 conn(0x7f13bc1538f0 0x7f13bc159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13c23e4640 1 -- 172.21.15.148:0/88513870 >> 172.21.15.148:0/88513870 conn(0x7f13bc06a640 msgr2=0x7f13bc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:00.102 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13c23e4640 1 -- 172.21.15.148:0/88513870 shutdown_connections 2023-10-30T13:54:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.096+0000 7f13c23e4640 1 -- 172.21.15.148:0/88513870 wait complete. 2023-10-30T13:54:00.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 Processor -- start 2023-10-30T13:54:00.105 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 -- start start 2023-10-30T13:54:00.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13bc0cefa0 0x7f13bc157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc1538f0 0x7f13bc13bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13bc15cd50 0x7f13bc13c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.106 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc1510a0 con 0x7f13bc1538f0 2023-10-30T13:54:00.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc142780 con 0x7f13bc15cd50 2023-10-30T13:54:00.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc158460 con 0x7f13bc0cefa0 2023-10-30T13:54:00.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c0be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc1538f0 0x7f13bc13bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.107 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c0be1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc1538f0 0x7f13bc13bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39042/0 (socket says 172.21.15.148:39042) 2023-10-30T13:54:00.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c0be1640 1 -- 172.21.15.148:0/938901543 learned_addr learned my addr 172.21.15.148:0/938901543 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:00.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c1be3640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13bc15cd50 0x7f13bc13c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.108 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c13e2640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13bc0cefa0 0x7f13bc157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c0be1640 1 -- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13bc0cefa0 msgr2=0x7f13bc157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c0be1640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13bc0cefa0 0x7f13bc157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c0be1640 1 -- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13bc15cd50 msgr2=0x7f13bc13c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.109 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c0be1640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13bc15cd50 0x7f13bc13c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c0be1640 1 -- 172.21.15.148:0/938901543 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4009b30 con 0x7f13bc1538f0 2023-10-30T13:54:00.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c1be3640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13bc15cd50 0x7f13bc13c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:00.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c0be1640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc1538f0 0x7f13bc13bed0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00c4b0 tx=0x7f13ac00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.110 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13b1ffb640 1 -- 172.21.15.148:0/938901543 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac0074e0 con 0x7f13bc1538f0 2023-10-30T13:54:00.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13b1ffb640 1 -- 172.21.15.148:0/938901543 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f13ac007660 con 0x7f13bc1538f0 2023-10-30T13:54:00.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc159400 con 0x7f13bc1538f0 2023-10-30T13:54:00.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13b1ffb640 1 -- 172.21.15.148:0/938901543 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac004a50 con 0x7f13bc1538f0 2023-10-30T13:54:00.111 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc157440 con 0x7f13bc1538f0 2023-10-30T13:54:00.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.100+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc04e170 con 0x7f13bc1538f0 2023-10-30T13:54:00.112 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.104+0000 7f13b1ffb640 1 -- 172.21.15.148:0/938901543 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13ac004bb0 con 0x7f13bc1538f0 2023-10-30T13:54:00.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.104+0000 7f13b1ffb640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13a00664a0 0x7f13a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.113 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.104+0000 7f13b1ffb640 1 -- 172.21.15.148:0/938901543 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7f13ac083720 con 0x7f13bc1538f0 2023-10-30T13:54:00.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.104+0000 7f13c13e2640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13a00664a0 0x7f13a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.104+0000 7f13c13e2640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13a00664a0 0x7f13a0068960 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009af0 tx=0x7f13b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.114 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.108+0000 7f13b1ffb640 1 -- 172.21.15.148:0/938901543 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13ac053f20 con 0x7f13bc1538f0 2023-10-30T13:54:00.266 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.264+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13bc146220 con 0x7f13bc1538f0 2023-10-30T13:54:00.268 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.264+0000 7f13b1ffb640 1 -- 172.21.15.148:0/938901543 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+19569 (secure 0 0 0) 0x7f13ac004e60 con 0x7f13bc1538f0 2023-10-30T13:54:00.268 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-30T13:54:00.268 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":67,"fsid":"23a3d51e-f2ab-4ffa-93fe-42135bef502c","created":"2023-10-30T13:10:09.559995+0000","modified":"2023-10-30T13:50:55.057709+0000","last_up_change":"2023-10-30T13:10:21.491635+0000","last_in_change":"2023-10-30T13:10:11.955160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T13:10:22.349322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T13:10:32.611151+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T13:10:33.044931+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dff19510-6825-4999-b159-238bfb5ff95f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6801","nonce":1454599971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6803","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6805","nonce":1454599971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6811","nonce":1454599971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1454599971},{"type":"v1","addr":"172.21.15.148:6808","nonce":1454599971}]},"public_addr":"172.21.15.148:6801/1454599971","cluster_addr":"172.21.15.148:6805/1454599971","heartbeat_back_addr":"172.21.15.148:6811/1454599971","heartbeat_front_addr":"172.21.15.148:6808/1454599971","state":["exists","up"]},{"osd":1,"uuid":"4fc28859-6374-44ef-96b4-9f7c0c4ddcb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6815","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6817","nonce":1053568651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6819","nonce":1053568651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6823","nonce":1053568651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":1053568651},{"type":"v1","addr":"172.21.15.148:6821","nonce":1053568651}]},"public_addr":"172.21.15.148:6817/1053568651","cluster_addr":"172.21.15.148:6819/1053568651","heartbeat_back_addr":"172.21.15.148:6823/1053568651","heartbeat_front_addr":"172.21.15.148:6821/1053568651","state":["exists","up"]},{"osd":2,"uuid":"21c3986b-480c-4bdd-a85b-3d51710c7822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6804","nonce":2893462459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6807","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6809","nonce":2893462459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6816","nonce":2893462459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2893462459},{"type":"v1","addr":"172.21.15.148:6813","nonce":2893462459}]},"public_addr":"172.21.15.148:6804/2893462459","cluster_addr":"172.21.15.148:6809/2893462459","heartbeat_back_addr":"172.21.15.148:6816/2893462459","heartbeat_front_addr":"172.21.15.148:6813/2893462459","state":["exists","up"]},{"osd":3,"uuid":"d9e7369f-9c53-41cd-a15f-9a1cfcdd33ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6825","nonce":1150735205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6827","nonce":1150735205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6831","nonce":1150735205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":1150735205},{"type":"v1","addr":"172.21.15.148:6829","nonce":1150735205}]},"public_addr":"172.21.15.148:6825/1150735205","cluster_addr":"172.21.15.148:6827/1150735205","heartbeat_back_addr":"172.21.15.148:6831/1150735205","heartbeat_front_addr":"172.21.15.148:6829/1150735205","state":["exists","up"]},{"osd":4,"uuid":"8929676f-29ab-4520-acb5-cfe61a0a3063","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6801","nonce":381432752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6807","nonce":381432752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6819","nonce":381432752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":381432752},{"type":"v1","addr":"172.21.15.150:6813","nonce":381432752}]},"public_addr":"172.21.15.150:6801/381432752","cluster_addr":"172.21.15.150:6807/381432752","heartbeat_back_addr":"172.21.15.150:6819/381432752","heartbeat_front_addr":"172.21.15.150:6813/381432752","state":["exists","up"]},{"osd":5,"uuid":"468a4500-4bc3-4397-a579-372ced468db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6805","nonce":2046109052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6812","nonce":2046109052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6822","nonce":2046109052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6815","nonce":2046109052},{"type":"v1","addr":"172.21.15.150:6818","nonce":2046109052}]},"public_addr":"172.21.15.150:6805/2046109052","cluster_addr":"172.21.15.150:6812/2046109052","heartbeat_back_addr":"172.21.15.150:6822/2046109052","heartbeat_front_addr":"172.21.15.150:6818/2046109052","state":["exists","up"]},{"osd":6,"uuid":"752c1053-247f-4e90-8054-5ace15b90f2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6806","nonce":397365804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6811","nonce":397365804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6823","nonce":397365804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":397365804},{"type":"v1","addr":"172.21.15.150:6817","nonce":397365804}]},"public_addr":"172.21.15.150:6806/397365804","cluster_addr":"172.21.15.150:6811/397365804","heartbeat_back_addr":"172.21.15.150:6823/397365804","heartbeat_front_addr":"172.21.15.150:6817/397365804","state":["exists","up"]},{"osd":7,"uuid":"1151f789-6221-4eae-9398-c4a4ac83e170","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6825","nonce":1265253901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6827","nonce":1265253901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6831","nonce":1265253901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":1265253901},{"type":"v1","addr":"172.21.15.150:6829","nonce":1265253901}]},"public_addr":"172.21.15.150:6825/1265253901","cluster_addr":"172.21.15.150:6827/1265253901","heartbeat_back_addr":"172.21.15.150:6831/1265253901","heartbeat_front_addr":"172.21.15.150:6829/1265253901","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.554932+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.653814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.508335+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.684295+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.569052+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.618691+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.472213+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T13:10:17.607515+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.148:6837/1530445724":"2023-10-31T13:50:55.057603+0000","172.21.15.148:6835/1228827025":"2023-10-31T13:49:20.023064+0000","172.21.15.150:6834/116502200":"2023-10-31T13:48:15.005684+0000","172.21.15.148:6833/1228827025":"2023-10-31T13:49:20.023064+0000","172.21.15.148:6837/3703944646":"2023-10-31T13:45:54.968896+0000","172.21.15.148:6835/3829165476":"2023-10-31T13:44:34.947959+0000","172.21.15.148:6833/3829165476":"2023-10-31T13:44:34.947959+0000","172.21.15.148:6836/500606168":"2023-10-31T13:42:29.894087+0000","172.21.15.148:6835/1079809213":"2023-10-31T13:39:54.856162+0000","172.21.15.148:6833/3194894584":"2023-10-31T13:24:29.547356+0000","172.21.15.148:6837/3232618434":"2023-10-31T13:33:54.741392+0000","172.21.15.148:6835/3194894584":"2023-10-31T13:24:29.547356+0000","172.21.15.150:6834/953984429":"2023-10-31T13:22:39.518196+0000","172.21.15.148:6835/3749586444":"2023-10-31T13:15:19.379857+0000","172.21.15.148:6836/3703944646":"2023-10-31T13:45:54.968896+0000","172.21.15.148:6834/4162935019":"2023-10-31T13:18:39.446214+0000","172.21.15.148:6832/4162935019":"2023-10-31T13:18:39.446214+0000","172.21.15.148:6837/500606168":"2023-10-31T13:42:29.894087+0000","172.21.15.148:6833/3749586444":"2023-10-31T13:15:19.379857+0000","172.21.15.150:6836/116502200":"2023-10-31T13:48:15.005684+0000","172.21.15.148:6837/3781833565":"2023-10-31T13:23:29.531899+0000","172.21.15.148:6833/1079809213":"2023-10-31T13:39:54.856162+0000","172.21.15.148:6836/1530445724":"2023-10-31T13:50:55.057603+0000","172.21.15.148:6837/1844602519":"2023-10-31T13:14:19.364086+0000","172.21.15.148:6836/1559416001":"2023-10-31T13:26:44.595428+0000","172.21.15.148:6836/3781833565":"2023-10-31T13:23:29.531899+0000","172.21.15.150:6834/1620877178":"2023-10-31T13:16:49.403499+0000","172.21.15.148:6837/2740075781":"2023-10-31T13:29:49.670604+0000","172.21.15.150:6836/953984429":"2023-10-31T13:22:39.518196+0000","172.21.15.148:6833/1386276781":"2023-10-31T13:35:44.780019+0000","172.21.15.150:6836/1620877178":"2023-10-31T13:16:49.403499+0000","172.21.15.148:6837/1559416001":"2023-10-31T13:26:44.595428+0000","172.21.15.148:6836/2740075781":"2023-10-31T13:29:49.670604+0000","172.21.15.148:6833/2733100982":"2023-10-31T13:28:14.633420+0000","172.21.15.148:6835/2733100982":"2023-10-31T13:28:14.633420+0000","172.21.15.148:6836/1844602519":"2023-10-31T13:14:19.364086+0000","172.21.15.148:6833/172087710":"2023-10-31T13:31:59.711325+0000","172.21.15.148:6835/172087710":"2023-10-31T13:31:59.711325+0000","172.21.15.148:6836/3232618434":"2023-10-31T13:33:54.741392+0000","172.21.15.148:6835/1386276781":"2023-10-31T13:35:44.780019+0000","172.21.15.148:6836/3144126860":"2023-10-31T13:37:34.820655+0000","172.21.15.148:6837/3144126860":"2023-10-31T13:37:34.820655+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:54:00.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13a00664a0 msgr2=0x7f13a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.270 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13a00664a0 0x7f13a0068960 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009af0 tx=0x7f13b8008040 comp rx=0 tx=0).stop 2023-10-30T13:54:00.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc1538f0 msgr2=0x7f13bc13bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc1538f0 0x7f13bc13bed0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00c4b0 tx=0x7f13ac00c970 comp rx=0 tx=0).stop 2023-10-30T13:54:00.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 shutdown_connections 2023-10-30T13:54:00.271 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f13a00664a0 0x7f13a0068960 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13bc1538f0 0x7f13bc13bed0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13bc15cd50 0x7f13bc13c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 --2- 172.21.15.148:0/938901543 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f13bc0cefa0 0x7f13bc157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.272 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 >> 172.21.15.148:0/938901543 conn(0x7f13bc06a640 msgr2=0x7f13bc06bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:00.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 shutdown_connections 2023-10-30T13:54:00.273 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.268+0000 7f13c23e4640 1 -- 172.21.15.148:0/938901543 wait complete. 2023-10-30T13:54:00.289 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-30T13:54:00.427 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.424+0000 7fa3d599d640 1 Processor -- start 2023-10-30T13:54:00.428 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.424+0000 7fa3d599d640 1 -- start start 2023-10-30T13:54:00.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3d599d640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa3d0153c60 0x7fa3d00ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.429 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3d599d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 0x7fa3d0159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3d599d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d00cb780 con 0x7fa3d00c4430 2023-10-30T13:54:00.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3d599d640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d0154040 con 0x7fa3d0153c60 2023-10-30T13:54:00.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3d599d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d015a2e0 con 0x7fa3d0159830 2023-10-30T13:54:00.430 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3ceffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa3d0153c60 0x7fa3d00ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3ceffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa3d0153c60 0x7fa3d00ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39046/0 (socket says 172.21.15.148:39046) 2023-10-30T13:54:00.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3ceffd640 1 -- 172.21.15.148:0/3601579202 learned_addr learned my addr 172.21.15.148:0/3601579202 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:00.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3cf7fe640 1 --2- 172.21.15.148:0/3601579202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 0x7fa3d0159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.431 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3ceffd640 1 -- 172.21.15.148:0/3601579202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 msgr2=0x7fa3d0159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3ceffd640 1 --2- 172.21.15.148:0/3601579202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 0x7fa3d0159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3ceffd640 1 -- 172.21.15.148:0/3601579202 >> v1:172.21.15.148:6789/0 conn(0x7fa3d00c4430 legacy=0x7fa3d00cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3ceffd640 1 -- 172.21.15.148:0/3601579202 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d015a5c0 con 0x7fa3d0153c60 2023-10-30T13:54:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.428+0000 7fa3ceffd640 1 --2- 172.21.15.148:0/3601579202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 0x7fa3d00ca360 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009870 tx=0x7fa3c001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e575f81fb21825a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3cdffb640 1 -- 172.21.15.148:0/3601579202 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00a7070 con 0x7fa3d0153c60 2023-10-30T13:54:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3cdffb640 1 -- 172.21.15.148:0/3601579202 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa3c00ab430 con 0x7fa3d0153c60 2023-10-30T13:54:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3cdffb640 1 -- 172.21.15.148:0/3601579202 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00ab7e0 con 0x7fa3d0153c60 2023-10-30T13:54:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 -- 172.21.15.148:0/3601579202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 msgr2=0x7fa3d00ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 --2- 172.21.15.148:0/3601579202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 0x7fa3d00ca360 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009870 tx=0x7fa3c001c540 comp rx=0 tx=0).stop 2023-10-30T13:54:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 -- 172.21.15.148:0/3601579202 shutdown_connections 2023-10-30T13:54:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 --2- 172.21.15.148:0/3601579202 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 0x7fa3d0159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 --2- 172.21.15.148:0/3601579202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 0x7fa3d00ca360 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 -- 172.21.15.148:0/3601579202 >> 172.21.15.148:0/3601579202 conn(0x7fa3d006bfe0 msgr2=0x7fa3d006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 -- 172.21.15.148:0/3601579202 shutdown_connections 2023-10-30T13:54:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 -- 172.21.15.148:0/3601579202 wait complete. 2023-10-30T13:54:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 Processor -- start 2023-10-30T13:54:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 -- start start 2023-10-30T13:54:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d00c4430 0x7fa3d0154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 0x7fa3d014d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 0x7fa3d01569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d01546c0 con 0x7fa3d0153c60 2023-10-30T13:54:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d014cc60 con 0x7fa3d00c4430 2023-10-30T13:54:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3d599d640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d0155330 con 0x7fa3d0159830 2023-10-30T13:54:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3ce7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 0x7fa3d014d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3ce7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 0x7fa3d014d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39050/0 (socket says 172.21.15.148:39050) 2023-10-30T13:54:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.432+0000 7fa3ce7fc640 1 -- 172.21.15.148:0/494084384 learned_addr learned my addr 172.21.15.148:0/494084384 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3ceffd640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d00c4430 0x7fa3d0154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3cf7fe640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 0x7fa3d01569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3ce7fc640 1 -- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 msgr2=0x7fa3d01569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3ce7fc640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 0x7fa3d01569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3ce7fc640 1 -- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d00c4430 msgr2=0x7fa3d0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3ce7fc640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d00c4430 0x7fa3d0154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3ce7fc640 1 -- 172.21.15.148:0/494084384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c00094d0 con 0x7fa3d0153c60 2023-10-30T13:54:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3ceffd640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d00c4430 0x7fa3d0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3ce7fc640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 0x7fa3d014d610 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c4d0 tx=0x7fa3c400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3b7fff640 1 -- 172.21.15.148:0/494084384 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40052d0 con 0x7fa3d0153c60 2023-10-30T13:54:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3b7fff640 1 -- 172.21.15.148:0/494084384 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa3c4005430 con 0x7fa3d0153c60 2023-10-30T13:54:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3b7fff640 1 -- 172.21.15.148:0/494084384 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4012850 con 0x7fa3d0153c60 2023-10-30T13:54:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d0155610 con 0x7fa3d0153c60 2023-10-30T13:54:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d0155bd0 con 0x7fa3d0153c60 2023-10-30T13:54:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d004e170 con 0x7fa3d0153c60 2023-10-30T13:54:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3b7fff640 1 -- 172.21.15.148:0/494084384 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3c40129b0 con 0x7fa3d0153c60 2023-10-30T13:54:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3b7fff640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3bc0664d0 0x7fa3bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.436+0000 7fa3b7fff640 1 -- 172.21.15.148:0/494084384 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7660+0+0 (secure 0 0 0) 0x7fa3c4084200 con 0x7fa3d0153c60 2023-10-30T13:54:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.440+0000 7fa3ceffd640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3bc0664d0 0x7fa3bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.440+0000 7fa3ceffd640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3bc0664d0 0x7fa3bc068990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00af000 tx=0x7fa3c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.448 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.444+0000 7fa3b7fff640 1 -- 172.21.15.148:0/494084384 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3c4050db0 con 0x7fa3d0153c60 2023-10-30T13:54:00.625 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.624+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa3d014a520 con 0x7fa3d0153c60 2023-10-30T13:54:00.629 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.628+0000 7fa3b7fff640 1 -- 172.21.15.148:0/494084384 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v167) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa3c40543d0 con 0x7fa3d0153c60 2023-10-30T13:54:00.629 INFO:teuthology.orchestra.run.smithi148.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-30T13:54:00.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.628+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3bc0664d0 msgr2=0x7fa3bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.633 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.628+0000 7fa3d599d640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3bc0664d0 0x7fa3bc068990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00af000 tx=0x7fa3c0008040 comp rx=0 tx=0).stop 2023-10-30T13:54:00.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.628+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 msgr2=0x7fa3d014d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.628+0000 7fa3d599d640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 0x7fa3d014d610 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400c4d0 tx=0x7fa3c400c990 comp rx=0 tx=0).stop 2023-10-30T13:54:00.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.632+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 shutdown_connections 2023-10-30T13:54:00.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.632+0000 7fa3d599d640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fa3bc0664d0 0x7fa3bc068990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.632+0000 7fa3d599d640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3d0153c60 0x7fa3d014d610 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.632+0000 7fa3d599d640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fa3d0159830 0x7fa3d01569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.632+0000 7fa3d599d640 1 --2- 172.21.15.148:0/494084384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3d00c4430 0x7fa3d0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.632+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 >> 172.21.15.148:0/494084384 conn(0x7fa3d006bfe0 msgr2=0x7fa3d00cd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:00.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.632+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 shutdown_connections 2023-10-30T13:54:00.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.632+0000 7fa3d599d640 1 -- 172.21.15.148:0/494084384 wait complete. 2023-10-30T13:54:00.652 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-30T13:54:00.710 INFO:tasks.ceph.mds.a.smithi148.stdout:starting mds.a at 2023-10-30T13:54:00.731 INFO:tasks.ceph.mds.d.smithi150.stdout:starting mds.d at 2023-10-30T13:54:00.795 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.792+0000 7f8a871c5640 1 Processor -- start 2023-10-30T13:54:00.796 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.792+0000 7f8a871c5640 1 -- start start 2023-10-30T13:54:00.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a871c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 0x7f8a800c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.797 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a871c5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8a800c6e60 0x7f8a80156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a871c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a800c0a30 con 0x7f8a800722a0 2023-10-30T13:54:00.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a871c5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a800c85b0 con 0x7f8a800c6e60 2023-10-30T13:54:00.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a871c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8006f650 con 0x7f8a801508c0 2023-10-30T13:54:00.798 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a85762640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8a800c6e60 0x7f8a80156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a85762640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f8a800c6e60 0x7f8a80156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39064/0 (socket says 172.21.15.148:39064) 2023-10-30T13:54:00.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a85762640 1 -- 172.21.15.148:0/3267143719 learned_addr learned my addr 172.21.15.148:0/3267143719 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:00.799 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a84f61640 1 --2- 172.21.15.148:0/3267143719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 0x7f8a800c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a85762640 1 -- 172.21.15.148:0/3267143719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 msgr2=0x7f8a800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:00.800 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a85762640 1 --2- 172.21.15.148:0/3267143719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 0x7f8a800c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a85762640 1 -- 172.21.15.148:0/3267143719 >> v1:172.21.15.150:6789/0 conn(0x7f8a800722a0 legacy=0x7f8a800bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a85762640 1 -- 172.21.15.148:0/3267143719 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8006f930 con 0x7f8a800c6e60 2023-10-30T13:54:00.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a84f61640 1 --2- 172.21.15.148:0/3267143719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 0x7f8a800c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:54:00.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a85762640 1 --2- 172.21.15.148:0/3267143719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800c6e60 0x7f8a80156930 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009e50 tx=0x7f8a7801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9266808ac5a7f4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.796+0000 7f8a777fe640 1 -- 172.21.15.148:0/3267143719 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a780a57d0 con 0x7f8a800c6e60 2023-10-30T13:54:00.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a777fe640 1 -- 172.21.15.148:0/3267143719 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f8a780a5930 con 0x7f8a800c6e60 2023-10-30T13:54:00.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a777fe640 1 -- 172.21.15.148:0/3267143719 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a780a5ce0 con 0x7f8a800c6e60 2023-10-30T13:54:00.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 -- 172.21.15.148:0/3267143719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800c6e60 msgr2=0x7f8a80156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 --2- 172.21.15.148:0/3267143719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800c6e60 0x7f8a80156930 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009e50 tx=0x7f8a7801ca10 comp rx=0 tx=0).stop 2023-10-30T13:54:00.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 -- 172.21.15.148:0/3267143719 shutdown_connections 2023-10-30T13:54:00.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 --2- 172.21.15.148:0/3267143719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800c6e60 0x7f8a80156930 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 --2- 172.21.15.148:0/3267143719 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 0x7f8a800c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 -- 172.21.15.148:0/3267143719 >> 172.21.15.148:0/3267143719 conn(0x7f8a800c7910 msgr2=0x7f8a800c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:00.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 -- 172.21.15.148:0/3267143719 shutdown_connections 2023-10-30T13:54:00.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 -- 172.21.15.148:0/3267143719 wait complete. 2023-10-30T13:54:00.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 Processor -- start 2023-10-30T13:54:00.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 -- start start 2023-10-30T13:54:00.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800722a0 0x7f8a801403b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a800c6e60 0x7f8a80144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 0x7f8a801f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a801f1d50 con 0x7f8a800722a0 2023-10-30T13:54:00.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a801652d0 con 0x7f8a800c6e60 2023-10-30T13:54:00.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a871c5640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80165740 con 0x7f8a801508c0 2023-10-30T13:54:00.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a84f61640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800722a0 0x7f8a801403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a84f61640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800722a0 0x7f8a801403b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39078/0 (socket says 172.21.15.148:39078) 2023-10-30T13:54:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.800+0000 7f8a84f61640 1 -- 172.21.15.148:0/2021347895 learned_addr learned my addr 172.21.15.148:0/2021347895 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a85762640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 0x7f8a801f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.811 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a77fff640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a800c6e60 0x7f8a80144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a84f61640 1 -- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 msgr2=0x7f8a801f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a84f61640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 0x7f8a801f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a84f61640 1 -- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a800c6e60 msgr2=0x7f8a80144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:00.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a84f61640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a800c6e60 0x7f8a80144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:00.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a84f61640 1 -- 172.21.15.148:0/2021347895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78009b30 con 0x7f8a800722a0 2023-10-30T13:54:00.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a84f61640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800722a0 0x7f8a801403b0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00d800 tx=0x7f8a7c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a757fa640 1 -- 172.21.15.148:0/2021347895 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c0059a0 con 0x7f8a800722a0 2023-10-30T13:54:00.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a757fa640 1 -- 172.21.15.148:0/2021347895 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f8a7c005b00 con 0x7f8a800722a0 2023-10-30T13:54:00.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a757fa640 1 -- 172.21.15.148:0/2021347895 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c004030 con 0x7f8a800722a0 2023-10-30T13:54:00.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a800c8cb0 con 0x7f8a800722a0 2023-10-30T13:54:00.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a800c9250 con 0x7f8a800722a0 2023-10-30T13:54:00.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8004e170 con 0x7f8a800722a0 2023-10-30T13:54:00.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a757fa640 1 -- 172.21.15.148:0/2021347895 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a7c004190 con 0x7f8a800722a0 2023-10-30T13:54:00.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a757fa640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8a6c0664a0 0x7f8a6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:00.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a757fa640 1 -- 172.21.15.148:0/2021347895 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f8a7c084800 con 0x7f8a800722a0 2023-10-30T13:54:00.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.804+0000 7f8a77fff640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8a6c0664a0 0x7f8a6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:00.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.808+0000 7f8a77fff640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8a6c0664a0 0x7f8a6c068960 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8a80072250 tx=0x7f8a70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:00.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.808+0000 7f8a757fa640 1 -- 172.21.15.148:0/2021347895 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a7c051300 con 0x7f8a800722a0 2023-10-30T13:54:00.991 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:00.988+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8a8014b760 con 0x7f8a800722a0 2023-10-30T13:54:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.628+0000 7f8a757fa640 1 -- 172.21.15.148:0/2021347895 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v168) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8a7c004440 con 0x7f8a800722a0 2023-10-30T13:54:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8a6c0664a0 msgr2=0x7f8a6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8a6c0664a0 0x7f8a6c068960 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8a80072250 tx=0x7f8a70009040 comp rx=0 tx=0).stop 2023-10-30T13:54:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800722a0 msgr2=0x7f8a801403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800722a0 0x7f8a801403b0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00d800 tx=0x7f8a7c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 shutdown_connections 2023-10-30T13:54:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f8a6c0664a0 0x7f8a6c068960 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f8a801508c0 0x7f8a801f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a800c6e60 0x7f8a80144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 --2- 172.21.15.148:0/2021347895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a800722a0 0x7f8a801403b0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 >> 172.21.15.148:0/2021347895 conn(0x7f8a800c7910 msgr2=0x7f8a80146c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 shutdown_connections 2023-10-30T13:54:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.632+0000 7f8a871c5640 1 -- 172.21.15.148:0/2021347895 wait complete. 2023-10-30T13:54:01.655 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-30T13:54:01.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ecac09640 1 Processor -- start 2023-10-30T13:54:01.801 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ecac09640 1 -- start start 2023-10-30T13:54:01.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ecac09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ec4156ed0 0x7f3ec40c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:01.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ecac09640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3ec40c6eb0 0x7f3ec40c2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:01.803 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ecac09640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec40c6a50 con 0x7f3ec40c2ca0 2023-10-30T13:54:01.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ecac09640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40c6630 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ecac09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40c4900 con 0x7f3ec4156ed0 2023-10-30T13:54:01.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ec3fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3ec40c6eb0 0x7f3ec40c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:01.804 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ec3fff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3ec40c6eb0 0x7f3ec40c2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39080/0 (socket says 172.21.15.148:39080) 2023-10-30T13:54:01.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ec3fff640 1 -- 172.21.15.148:0/4195573837 learned_addr learned my addr 172.21.15.148:0/4195573837 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:01.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ec89a5640 1 --2- 172.21.15.148:0/4195573837 >> v2:172.21.15.150:3300/0 conn(0x7f3ec4156ed0 0x7f3ec40c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:01.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ec3fff640 1 -- 172.21.15.148:0/4195573837 >> v2:172.21.15.150:3300/0 conn(0x7f3ec4156ed0 msgr2=0x7f3ec40c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:01.805 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ec3fff640 1 --2- 172.21.15.148:0/4195573837 >> v2:172.21.15.150:3300/0 conn(0x7f3ec4156ed0 0x7f3ec40c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ec3fff640 1 -- 172.21.15.148:0/4195573837 >> v1:172.21.15.150:6789/0 conn(0x7f3ec40c2ca0 legacy=0x7f3ec406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:01.806 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.800+0000 7f3ec3fff640 1 -- 172.21.15.148:0/4195573837 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec40c3750 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.807 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ec3fff640 1 --2- 172.21.15.148:0/4195573837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 0x7f3ec40c2760 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f3ec406fcc0 tx=0x7f3eb001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd470a3950c7d496 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:01.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ec37fe640 1 -- 172.21.15.148:0/4195573837 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00a6040 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ec37fe640 1 -- 172.21.15.148:0/4195573837 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3eb00aa600 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ec37fe640 1 -- 172.21.15.148:0/4195573837 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00aa9b0 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.808 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 -- 172.21.15.148:0/4195573837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 msgr2=0x7f3ec40c2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:01.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 --2- 172.21.15.148:0/4195573837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 0x7f3ec40c2760 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f3ec406fcc0 tx=0x7f3eb001c4a0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 -- 172.21.15.148:0/4195573837 shutdown_connections 2023-10-30T13:54:01.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 --2- 172.21.15.148:0/4195573837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 0x7f3ec40c2760 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.809 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 --2- 172.21.15.148:0/4195573837 >> v2:172.21.15.150:3300/0 conn(0x7f3ec4156ed0 0x7f3ec40c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 -- 172.21.15.148:0/4195573837 >> 172.21.15.148:0/4195573837 conn(0x7f3ec40ccf80 msgr2=0x7f3ec40cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:01.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 -- 172.21.15.148:0/4195573837 shutdown_connections 2023-10-30T13:54:01.810 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 -- 172.21.15.148:0/4195573837 wait complete. 2023-10-30T13:54:01.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 Processor -- start 2023-10-30T13:54:01.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.804+0000 7f3ecac09640 1 -- start start 2023-10-30T13:54:01.812 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ecac09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec40c2ca0 0x7f3ec40c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:01.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ecac09640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 0x7f3ec414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:01.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ecac09640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec4156ed0 0x7f3ec414dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:01.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ecac09640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec414bf40 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.813 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ecac09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40c86b0 con 0x7f3ec40c2ca0 2023-10-30T13:54:01.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ecac09640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40c8950 con 0x7f3ec4156ed0 2023-10-30T13:54:01.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 0x7f3ec414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:01.814 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 0x7f3ec414ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39094/0 (socket says 172.21.15.148:39094) 2023-10-30T13:54:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec3fff640 1 -- 172.21.15.148:0/1883376524 learned_addr learned my addr 172.21.15.148:0/1883376524 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec89a5640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec40c2ca0 0x7f3ec40c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec91a6640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec4156ed0 0x7f3ec414dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec3fff640 1 -- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec4156ed0 msgr2=0x7f3ec414dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:01.815 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec3fff640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec4156ed0 0x7f3ec414dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec3fff640 1 -- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec40c2ca0 msgr2=0x7f3ec40c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:01.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec3fff640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec40c2ca0 0x7f3ec40c8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:01.816 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec3fff640 1 -- 172.21.15.148:0/1883376524 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb00094d0 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec3fff640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 0x7f3ec414ba00 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f3eb001a0d0 tx=0x7f3eb001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:01.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec17fa640 1 -- 172.21.15.148:0/1883376524 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00b3800 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec17fa640 1 -- 172.21.15.148:0/1883376524 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3eb00b3960 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ec17fa640 1 -- 172.21.15.148:0/1883376524 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00b3cd0 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.817 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec414c3a0 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec414b200 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.818 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.808+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec414c200 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.812+0000 7f3ec17fa640 1 -- 172.21.15.148:0/1883376524 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3eb00c6430 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.812+0000 7f3ec17fa640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3ea80664a0 0x7f3ea8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:01.819 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.812+0000 7f3ec17fa640 1 -- 172.21.15.148:0/1883376524 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7832+0+0 (secure 0 0 0) 0x7f3eb011f3c0 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.812+0000 7f3ec89a5640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3ea80664a0 0x7f3ea8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:01.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.812+0000 7f3ec89a5640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3ea80664a0 0x7f3ea8068960 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800b930 tx=0x7f3eb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:01.820 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.816+0000 7f3ec17fa640 1 -- 172.21.15.148:0/1883376524 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3ec414c200 con 0x7f3ec40c6eb0 2023-10-30T13:54:01.969 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:01.968+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3ec414c200 con 0x7f3ec40c6eb0 2023-10-30T13:54:02.635 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.632+0000 7f3ec17fa640 1 -- 172.21.15.148:0/1883376524 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v69) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3ec414c200 con 0x7f3ec40c6eb0 2023-10-30T13:54:02.635 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'cephfs_metadata' removed 2023-10-30T13:54:02.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3ea80664a0 msgr2=0x7f3ea8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:02.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3ea80664a0 0x7f3ea8068960 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800b930 tx=0x7f3eb8008040 comp rx=0 tx=0).stop 2023-10-30T13:54:02.638 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 msgr2=0x7f3ec414ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:02.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 0x7f3ec414ba00 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f3eb001a0d0 tx=0x7f3eb001ace0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.639 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 shutdown_connections 2023-10-30T13:54:02.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7f3ea80664a0 0x7f3ea8068960 secure :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800b930 tx=0x7f3eb8008040 comp rx=0 tx=0).stop 2023-10-30T13:54:02.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3ec40c6eb0 0x7f3ec414ba00 secure :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f3eb001a0d0 tx=0x7f3eb001ace0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7f3ec4156ed0 0x7f3ec414dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.640 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 --2- 172.21.15.148:0/1883376524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ec40c2ca0 0x7f3ec40c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 >> 172.21.15.148:0/1883376524 conn(0x7f3ec40ccf80 msgr2=0x7f3ec41544d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:02.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 shutdown_connections 2023-10-30T13:54:02.641 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.636+0000 7f3ecac09640 1 -- 172.21.15.148:0/1883376524 wait complete. 2023-10-30T13:54:02.669 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-30T13:54:02.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd3a6101640 1 Processor -- start 2023-10-30T13:54:02.851 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd3a6101640 1 -- start start 2023-10-30T13:54:02.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd3a6101640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd3a01536d0 0x7fd3a00cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.852 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd3a6101640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a015d9d0 0x7fd3a00c4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd3a6101640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a00c4540 con 0x7fd3a0159760 2023-10-30T13:54:02.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd3a6101640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a015a210 con 0x7fd3a01536d0 2023-10-30T13:54:02.853 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd3a6101640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a015a4f0 con 0x7fd3a015d9d0 2023-10-30T13:54:02.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd39f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd3a01536d0 0x7fd3a00cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd39f7fe640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fd3a01536d0 0x7fd3a00cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46312/0 (socket says 172.21.15.148:46312) 2023-10-30T13:54:02.854 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.848+0000 7fd39f7fe640 1 -- 172.21.15.148:0/2702712992 learned_addr learned my addr 172.21.15.148:0/2702712992 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:02.855 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.852+0000 7fd39effd640 1 --2- 172.21.15.148:0/2702712992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a015d9d0 0x7fd3a00c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.852+0000 7fd39f7fe640 1 -- 172.21.15.148:0/2702712992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a015d9d0 msgr2=0x7fd3a00c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:54:02.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.852+0000 7fd39f7fe640 1 --2- 172.21.15.148:0/2702712992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a015d9d0 0x7fd3a00c4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.856 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.852+0000 7fd39f7fe640 1 -- 172.21.15.148:0/2702712992 >> v1:172.21.15.148:6789/0 conn(0x7fd3a0159760 legacy=0x7fd3a00cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:02.857 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.852+0000 7fd39f7fe640 1 -- 172.21.15.148:0/2702712992 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a015cef0 con 0x7fd3a01536d0 2023-10-30T13:54:02.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.852+0000 7fd39f7fe640 1 --2- 172.21.15.148:0/2702712992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a01536d0 0x7fd3a00cb3c0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd390009870 tx=0x7fd39001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c988970b2884ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:02.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.852+0000 7fd39e7fc640 1 -- 172.21.15.148:0/2702712992 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3900ab600 con 0x7fd3a01536d0 2023-10-30T13:54:02.858 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.852+0000 7fd39e7fc640 1 -- 172.21.15.148:0/2702712992 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd3900a7050 con 0x7fd3a01536d0 2023-10-30T13:54:02.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.852+0000 7fd39e7fc640 1 -- 172.21.15.148:0/2702712992 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3900ab9a0 con 0x7fd3a01536d0 2023-10-30T13:54:02.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 -- 172.21.15.148:0/2702712992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a01536d0 msgr2=0x7fd3a00cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:02.859 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 --2- 172.21.15.148:0/2702712992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a01536d0 0x7fd3a00cb3c0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd390009870 tx=0x7fd39001c540 comp rx=0 tx=0).stop 2023-10-30T13:54:02.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 -- 172.21.15.148:0/2702712992 shutdown_connections 2023-10-30T13:54:02.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 --2- 172.21.15.148:0/2702712992 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a015d9d0 0x7fd3a00c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 --2- 172.21.15.148:0/2702712992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a01536d0 0x7fd3a00cb3c0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.860 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 -- 172.21.15.148:0/2702712992 >> 172.21.15.148:0/2702712992 conn(0x7fd3a006bf70 msgr2=0x7fd3a006c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:02.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 -- 172.21.15.148:0/2702712992 shutdown_connections 2023-10-30T13:54:02.861 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 -- 172.21.15.148:0/2702712992 wait complete. 2023-10-30T13:54:02.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 Processor -- start 2023-10-30T13:54:02.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 -- start start 2023-10-30T13:54:02.863 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3a01536d0 0x7fd3a015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a0159760 0x7fd3a00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a015d9d0 0x7fd3a00cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a015c050 con 0x7fd3a015d9d0 2023-10-30T13:54:02.864 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a00ce260 con 0x7fd3a01536d0 2023-10-30T13:54:02.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd3a6101640 1 -- --> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a00b3970 con 0x7fd3a0159760 2023-10-30T13:54:02.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd39ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a015d9d0 0x7fd3a00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd39ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a015d9d0 0x7fd3a00cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:46322/0 (socket says 172.21.15.148:46322) 2023-10-30T13:54:02.865 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd39effd640 1 --2- >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a0159760 0x7fd3a00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.856+0000 7fd39ffff640 1 -- 172.21.15.148:0/3001619328 learned_addr learned my addr 172.21.15.148:0/3001619328 (peer_addr_for_me v2:172.21.15.148:0/0) 2023-10-30T13:54:02.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd39f7fe640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3a01536d0 0x7fd3a015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.866 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd39ffff640 1 -- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a0159760 msgr2=0x7fd3a00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:02.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd39ffff640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a0159760 0x7fd3a00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd39ffff640 1 -- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3a01536d0 msgr2=0x7fd3a015c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:02.867 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd39ffff640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3a01536d0 0x7fd3a015c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:02.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd39ffff640 1 -- 172.21.15.148:0/3001619328 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3900094d0 con 0x7fd3a015d9d0 2023-10-30T13:54:02.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd39ffff640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a015d9d0 0x7fd3a00cdd20 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd39800c730 tx=0x7fd39800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:02.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd387fff640 1 -- 172.21.15.148:0/3001619328 <== mon.0 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398003b80 con 0x7fd3a015d9d0 2023-10-30T13:54:02.868 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd387fff640 1 -- 172.21.15.148:0/3001619328 <== mon.0 v2:172.21.15.148:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fd398003ce0 con 0x7fd3a015d9d0 2023-10-30T13:54:02.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd387fff640 1 -- 172.21.15.148:0/3001619328 <== mon.0 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39800aa50 con 0x7fd3a015d9d0 2023-10-30T13:54:02.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a00b3c50 con 0x7fd3a015d9d0 2023-10-30T13:54:02.869 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a015bc70 con 0x7fd3a015d9d0 2023-10-30T13:54:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a015c190 con 0x7fd3a015d9d0 2023-10-30T13:54:02.870 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.860+0000 7fd387fff640 1 -- 172.21.15.148:0/3001619328 <== mon.0 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd39800abb0 con 0x7fd3a015d9d0 2023-10-30T13:54:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.864+0000 7fd387fff640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd38c0664a0 0x7fd38c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:54:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.864+0000 7fd387fff640 1 -- 172.21.15.148:0/3001619328 <== mon.0 v2:172.21.15.148:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7394+0+0 (secure 0 0 0) 0x7fd398084110 con 0x7fd3a015d9d0 2023-10-30T13:54:02.871 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.864+0000 7fd39f7fe640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd38c0664a0 0x7fd38c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:54:02.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.864+0000 7fd39f7fe640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd38c0664a0 0x7fd38c068960 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0072e20 tx=0x7fd390008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:54:02.872 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:02.868+0000 7fd387fff640 1 -- 172.21.15.148:0/3001619328 <== mon.0 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3a015c190 con 0x7fd3a015d9d0 2023-10-30T13:54:03.029 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.028+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd3a015c190 con 0x7fd3a015d9d0 2023-10-30T13:54:03.656 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.652+0000 7fd387fff640 1 -- 172.21.15.148:0/3001619328 <== mon.0 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v70) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd3a015c190 con 0x7fd3a015d9d0 2023-10-30T13:54:03.657 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'cephfs_data' removed 2023-10-30T13:54:03.658 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd38c0664a0 msgr2=0x7fd38c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:03.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd38c0664a0 0x7fd38c068960 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0072e20 tx=0x7fd390008040 comp rx=0 tx=0).stop 2023-10-30T13:54:03.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a015d9d0 msgr2=0x7fd3a00cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:54:03.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a015d9d0 0x7fd3a00cdd20 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd39800c730 tx=0x7fd39800cbf0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.659 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 shutdown_connections 2023-10-30T13:54:03.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:6832/1299332652,v1:172.21.15.150:6833/1299332652] conn(0x7fd38c0664a0 0x7fd38c068960 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3a015d9d0 0x7fd3a00cdd20 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:3301/0,v1:172.21.15.150:6790/0] conn(0x7fd3a0159760 0x7fd3a00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.660 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 --2- 172.21.15.148:0/3001619328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3a01536d0 0x7fd3a015c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:54:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 >> 172.21.15.148:0/3001619328 conn(0x7fd3a006bf70 msgr2=0x7fd3a00c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:54:03.661 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 shutdown_connections 2023-10-30T13:54:03.662 INFO:teuthology.orchestra.run.smithi148.stderr:2023-10-30T13:54:03.656+0000 7fd3a6101640 1 -- 172.21.15.148:0/3001619328 wait complete. 2023-10-30T13:54:03.701 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T13:54:03.701 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:54:03.702 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:03.719 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:54:03.720 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:54:03.720 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:03.779 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:54:03.780 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:54:03.780 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:03.839 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:54:03.840 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T13:54:03.840 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:03.853 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T13:54:03.853 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:54:03.854 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:03.908 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:54:03.908 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T13:54:03.908 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:03.965 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T13:54:03.966 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T13:54:03.966 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T13:54:03.967 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:05.483 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:05.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:06.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-30T13:54:10.069 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T13:54:10.070 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T13:54:10.070 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:10.152 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T13:54:10.153 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T13:54:10.153 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:12.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-30T13:54:12.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-30T13:54:16.255 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T13:54:16.256 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T13:54:16.256 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:16.348 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T13:54:16.349 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T13:54:16.349 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:16.439 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T13:54:16.439 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T13:54:16.440 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:18.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-10-30T13:54:18.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-30T13:54:22.542 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T13:54:22.542 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T13:54:22.543 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:22.612 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T13:54:22.612 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T13:54:22.613 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:22.684 INFO:tasks.ceph.osd.7:Stopped 2023-10-30T13:54:22.684 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T13:54:22.685 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T13:54:22.685 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:22.737 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T13:54:22.738 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T13:54:22.738 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:22.767 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T13:54:22.768 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T13:54:22.768 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T13:54:22.769 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:22.791 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T13:54:22.792 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T13:54:22.792 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:22.830 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T13:54:22.831 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T13:54:22.831 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:54:22.895 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T13:54:22.896 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T13:54:22.896 DEBUG:teuthology.orchestra.run.smithi148:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | head -n 1 2023-10-30T13:54:22.937 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:54:22.937 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T13:54:23.169 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:54:23.169 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T13:54:23.382 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:54:23.383 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T13:54:23.602 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:54:23.603 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T13:54:23.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-10-30T13:54:23.835 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:54:23.835 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T13:54:24.548 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:54:24.548 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T13:54:24.792 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:54:24.793 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T13:54:24.966 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:54:24.966 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T13:54:25.199 INFO:tasks.ceph:Archiving mon data... 2023-10-30T13:54:25.201 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/data/mon.a.tgz 2023-10-30T13:54:25.201 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2023-10-30T13:54:25.211 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.6HSFiN9amb 2023-10-30T13:54:25.211 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6HSFiN9amb 2023-10-30T13:54:25.343 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.6HSFiN9amb 2023-10-30T13:54:25.417 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.6HSFiN9amb is 367KB 2023-10-30T13:54:25.487 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.6HSFiN9amb 2023-10-30T13:54:25.495 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/data/mon.b.tgz 2023-10-30T13:54:25.496 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2023-10-30T13:54:25.502 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.NgzdGvCDYx 2023-10-30T13:54:25.503 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.NgzdGvCDYx 2023-10-30T13:54:25.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.NgzdGvCDYx 2023-10-30T13:54:25.691 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.NgzdGvCDYx is 369KB 2023-10-30T13:54:25.753 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.NgzdGvCDYx 2023-10-30T13:54:25.758 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/data/mon.c.tgz 2023-10-30T13:54:25.758 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2023-10-30T13:54:25.803 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.Djucm1Uoos 2023-10-30T13:54:25.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Djucm1Uoos 2023-10-30T13:54:25.926 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.Djucm1Uoos 2023-10-30T13:54:25.992 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.Djucm1Uoos is 369KB 2023-10-30T13:54:26.009 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.Djucm1Uoos 2023-10-30T13:54:26.014 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T13:54:26.015 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T13:54:26.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T13:54:26.071 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T13:54:26.071 DEBUG:teuthology.orchestra.run.smithi148:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T13:54:26.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T13:54:26.119 INFO:teuthology.orchestra.run.smithi148.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T13:54:26.147 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T13:54:26.151 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T13:54:26.153 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/remote/smithi148/crash 2023-10-30T13:54:26.154 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T13:54:26.188 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/remote/smithi150/crash 2023-10-30T13:54:26.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T13:54:26.204 INFO:tasks.ceph:Shutting down logrotate 2023-10-30T13:54:26.204 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:26.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:26.287 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:26.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T13:54:26.337 INFO:tasks.ceph:Compressing logs... 2023-10-30T13:54:26.337 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T13:54:26.355 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T13:54:26.372 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59386.log 2023-10-30T13:54:26.374 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60650.log 2023-10-30T13:54:26.375 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47595.log 2023-10-30T13:54:26.375 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59386.log.gz 2023-10-30T13:54:26.376 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49963.log 2023-10-30T13:54:26.376 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60650.log.gz 2023-10-30T13:54:26.377 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44026.log 2023-10-30T13:54:26.377 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47595.log.gz 2023-10-30T13:54:26.378 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.49963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62984.log 2023-10-30T13:54:26.378 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49963.log.gz 2023-10-30T13:54:26.379 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61654.log 2023-10-30T13:54:26.380 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44026.log.gz 2023-10-30T13:54:26.380 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58238.log 2023-10-30T13:54:26.380 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62984.log.gz 2023-10-30T13:54:26.382 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62191.log 2023-10-30T13:54:26.382 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61654.log.gz 2023-10-30T13:54:26.383 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57884.log 2023-10-30T13:54:26.383 INFO:teuthology.orchestra.run.smithi148.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58238.log.gz 2023-10-30T13:54:26.384 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57478.log 2023-10-30T13:54:26.384 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62191.log.gz 2023-10-30T13:54:26.384 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45507.log 2023-10-30T13:54:26.384 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57884.log.gz 2023-10-30T13:54:26.385 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58157.log 2023-10-30T13:54:26.386 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57478.log.gz 2023-10-30T13:54:26.386 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57990.log 2023-10-30T13:54:26.386 INFO:teuthology.orchestra.run.smithi148.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.45507.log.gz 2023-10-30T13:54:26.387 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49909.log 2023-10-30T13:54:26.388 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58157.log.gz 2023-10-30T13:54:26.388 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45192.log 2023-10-30T13:54:26.388 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57990.log.gz 2023-10-30T13:54:26.389 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.49909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51134.log 2023-10-30T13:54:26.390 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49909.log.gz 2023-10-30T13:54:26.390 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55835.log 2023-10-30T13:54:26.390 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45192.log.gz 2023-10-30T13:54:26.391 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51108.log 2023-10-30T13:54:26.391 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51134.log.gz 2023-10-30T13:54:26.392 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47702.log 2023-10-30T13:54:26.392 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55835.log.gz 2023-10-30T13:54:26.393 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44376.log 2023-10-30T13:54:26.393 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51108.log.gz 2023-10-30T13:54:26.394 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47702.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T13:54:26.394 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47702.log.gz 2023-10-30T13:54:26.395 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54655.log 2023-10-30T13:54:26.395 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44376.log.gz 2023-10-30T13:54:26.395 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46050.log 2023-10-30T13:54:26.396 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51241.log 2023-10-30T13:54:26.397 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54655.log.gz 2023-10-30T13:54:26.397 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46050.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46050.log.gzgzip 2023-10-30T13:54:26.397 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64493.log 2023-10-30T13:54:26.398 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47916.log 2023-10-30T13:54:26.399 INFO:teuthology.orchestra.run.smithi148.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51241.log.gz 2023-10-30T13:54:26.399 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64493.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64493.log.gz 2023-10-30T13:54:26.400 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43748.log 2023-10-30T13:54:26.401 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46383.log 2023-10-30T13:54:26.401 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47916.log.gz 2023-10-30T13:54:26.401 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43748.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.43748.log.gz 2023-10-30T13:54:26.402 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46102.log 2023-10-30T13:54:26.403 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56186.log 2023-10-30T13:54:26.403 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46383.log.gz 2023-10-30T13:54:26.404 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46102.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46102.log.gz 2023-10-30T13:54:26.404 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61522.log 2023-10-30T13:54:26.405 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56186.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56186.log.gz 2023-10-30T13:54:26.405 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60265.log 2023-10-30T13:54:26.406 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61522.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61522.log.gz 2023-10-30T13:54:26.406 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54785.log 2023-10-30T13:54:26.407 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60265.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.60265.log.gz 2023-10-30T13:54:26.407 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54064.log 2023-10-30T13:54:26.408 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57101.log 2023-10-30T13:54:26.408 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54785.log.gz 2023-10-30T13:54:26.409 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44000.log 2023-10-30T13:54:26.410 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54064.log.gz 2023-10-30T13:54:26.410 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46435.log 2023-10-30T13:54:26.411 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57101.log.gz 2023-10-30T13:54:26.413 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63412.log 2023-10-30T13:54:26.413 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44000.log.gz 2023-10-30T13:54:26.413 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48583.log 2023-10-30T13:54:26.414 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46435.log.gz 2023-10-30T13:54:26.414 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53514.log 2023-10-30T13:54:26.414 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63412.log.gz 2023-10-30T13:54:26.416 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38762.log 2023-10-30T13:54:26.417 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T13:54:26.417 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38762.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T13:54:26.417 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38762.log.gz 2023-10-30T13:54:26.418 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T13:54:26.418 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38786.log 2023-10-30T13:54:26.418 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38642.log 2023-10-30T13:54:26.418 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38786.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38786.log.gz 2023-10-30T13:54:26.419 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T13:54:26.419 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-10-30T13:54:26.419 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T13:54:26.419 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38642.log: gzip -5 --verbose/var/log/ceph/ceph-mds.f.log: -- /var/log/ceph/ceph-client.admin.38714.log 2023-10-30T13:54:26.420 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38642.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38690.log 2023-10-30T13:54:26.420 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T13:54:26.420 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38714.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T13:54:26.420 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38690.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38714.log.gz 2023-10-30T13:54:26.422 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T13:54:26.423 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38690.log.gz 2023-10-30T13:54:26.423 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-30T13:54:26.427 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54309.log 2023-10-30T13:54:26.427 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48583.log.gz 2023-10-30T13:54:26.428 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43784.log 2023-10-30T13:54:26.428 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53514.log.gz 2023-10-30T13:54:26.428 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54309.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54309.log.gz 2023-10-30T13:54:26.428 INFO:teuthology.orchestra.run.smithi148.stderr: 95.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T13:54:26.429 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61183.log 2023-10-30T13:54:26.429 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54467.log 2023-10-30T13:54:26.429 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43784.log.gz 2023-10-30T13:54:26.429 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60508.log 2023-10-30T13:54:26.430 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61183.log.gz 2023-10-30T13:54:26.430 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55726.log 2023-10-30T13:54:26.430 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54467.log.gz 2023-10-30T13:54:26.431 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60508.log: gzip 89.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.60508.log.gz -- 2023-10-30T13:54:26.431 INFO:teuthology.orchestra.run.smithi148.stderr: /var/log/ceph/ceph-client.admin.54012.log 2023-10-30T13:54:26.431 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61576.log 2023-10-30T13:54:26.431 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55726.log.gz 2023-10-30T13:54:26.432 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59332.log 2023-10-30T13:54:26.432 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54012.log.gz 2023-10-30T13:54:26.432 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59140.log 2023-10-30T13:54:26.432 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61576.log.gz 2023-10-30T13:54:26.433 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59332.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T13:54:26.433 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59332.log.gz 2023-10-30T13:54:26.433 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64073.log 2023-10-30T13:54:26.433 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59140.log.gz 2023-10-30T13:54:26.434 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52824.log 2023-10-30T13:54:26.434 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47647.log 2023-10-30T13:54:26.434 INFO:teuthology.orchestra.run.smithi148.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64073.log.gz 2023-10-30T13:54:26.436 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.audit.log: 94.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T13:54:26.436 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-30T13:54:26.436 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mds.d.log: 94.4% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-10-30T13:54:26.437 INFO:teuthology.orchestra.run.smithi150.stderr: 93.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T13:54:26.437 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T13:54:26.437 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38810.log 2023-10-30T13:54:26.440 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.b.log: 94.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-30T13:54:26.440 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T13:54:26.440 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38810.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38810.log.gz 2023-10-30T13:54:26.440 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38738.log 2023-10-30T13:54:26.445 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63360.log 2023-10-30T13:54:26.445 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52824.log.gz 2023-10-30T13:54:26.446 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50172.log 2023-10-30T13:54:26.446 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47647.log.gz 2023-10-30T13:54:26.446 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63360.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63360.log.gz 2023-10-30T13:54:26.446 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64177.log 2023-10-30T13:54:26.447 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43893.log 2023-10-30T13:54:26.447 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50172.log.gz 2023-10-30T13:54:26.447 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46024.log 2023-10-30T13:54:26.448 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64177.log.gz 2023-10-30T13:54:26.448 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51161.log 2023-10-30T13:54:26.448 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43893.log.gz 2023-10-30T13:54:26.449 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46024.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46024.log.gz 2023-10-30T13:54:26.449 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56997.log 2023-10-30T13:54:26.449 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59055.log 2023-10-30T13:54:26.449 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51161.log.gz 2023-10-30T13:54:26.450 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64339.log 2023-10-30T13:54:26.450 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56997.log.gz 2023-10-30T13:54:26.450 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62905.log 2023-10-30T13:54:26.450 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59055.log.gz 2023-10-30T13:54:26.451 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64339.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64339.log.gz 2023-10-30T13:54:26.451 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64312.log 2023-10-30T13:54:26.451 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46751.log 2023-10-30T13:54:26.452 INFO:teuthology.orchestra.run.smithi148.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62905.log.gz 2023-10-30T13:54:26.452 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44678.log 2023-10-30T13:54:26.452 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64312.log.gz 2023-10-30T13:54:26.453 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55479.log 2023-10-30T13:54:26.453 INFO:teuthology.orchestra.run.smithi148.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46751.log.gz 2023-10-30T13:54:26.453 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44678.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44678.log.gz 2023-10-30T13:54:26.453 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50513.log 2023-10-30T13:54:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58625.log 2023-10-30T13:54:26.454 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55479.log.gz 2023-10-30T13:54:26.454 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58873.log 2023-10-30T13:54:26.454 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50513.log.gz 2023-10-30T13:54:26.455 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58625.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58625.log.gz 2023-10-30T13:54:26.455 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58072.log 2023-10-30T13:54:26.456 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38666.log 2023-10-30T13:54:26.457 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38738.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38738.log.gz 2023-10-30T13:54:26.462 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59765.log 2023-10-30T13:54:26.462 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58873.log.gz 2023-10-30T13:54:26.462 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51213.log 88.9% 2023-10-30T13:54:26.462 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58072.log.gz 2023-10-30T13:54:26.463 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57341.log 2023-10-30T13:54:26.463 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59765.log.gz 2023-10-30T13:54:26.463 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51213.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51213.log.gz 2023-10-30T13:54:26.464 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53601.log 2023-10-30T13:54:26.464 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57341.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57341.log.gz 2023-10-30T13:54:26.464 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63558.log 2023-10-30T13:54:26.464 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53601.log: gzip -5 --verbose 89.4% -- -- replaced with /var/log/ceph/ceph-client.admin.53601.log.gz 2023-10-30T13:54:26.465 INFO:teuthology.orchestra.run.smithi148.stderr: /var/log/ceph/ceph-client.admin.60291.log 2023-10-30T13:54:26.465 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63558.log: gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.59528.log -- replaced with /var/log/ceph/ceph-client.admin.63558.log.gz 2023-10-30T13:54:26.465 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T13:54:26.466 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60291.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.48662.log -- replaced with /var/log/ceph/ceph-client.admin.60291.log.gz 2023-10-30T13:54:26.466 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T13:54:26.466 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59528.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59528.log.gz 2023-10-30T13:54:26.466 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56371.log 2023-10-30T13:54:26.467 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48662.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.48662.log.gz 2023-10-30T13:54:26.467 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53436.log 2023-10-30T13:54:26.467 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55591.log 2023-10-30T13:54:26.468 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56371.log.gz 2023-10-30T13:54:26.468 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53436.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53436.log.gz 2023-10-30T13:54:26.468 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45596.log 2023-10-30T13:54:26.468 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53174.log 2023-10-30T13:54:26.469 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55591.log.gz 2023-10-30T13:54:26.469 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46668.log 2023-10-30T13:54:26.469 INFO:teuthology.orchestra.run.smithi148.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.45596.log.gz 2023-10-30T13:54:26.469 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61053.log 2023-10-30T13:54:26.470 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53174.log.gz 2023-10-30T13:54:26.470 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45920.log 2023-10-30T13:54:26.470 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46668.log.gz 2023-10-30T13:54:26.472 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38666.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38666.log.gz 2023-10-30T13:54:26.476 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61053.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61053.log.gz 2023-10-30T13:54:26.476 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56215.log 2023-10-30T13:54:26.476 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58977.log 2023-10-30T13:54:26.477 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45920.log.gz 2023-10-30T13:54:26.477 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54227.log 2023-10-30T13:54:26.477 INFO:teuthology.orchestra.run.smithi148.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56215.log.gz 2023-10-30T13:54:26.478 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61287.log 2023-10-30T13:54:26.478 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58977.log.gz 2023-10-30T13:54:26.478 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54227.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54227.log.gz 2023-10-30T13:54:26.478 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62879.log 2023-10-30T13:54:26.479 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56862.log 2023-10-30T13:54:26.479 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61287.log.gz 2023-10-30T13:54:26.479 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46409.log 2023-10-30T13:54:26.480 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62879.log.gz 2023-10-30T13:54:26.480 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54870.log 2023-10-30T13:54:26.480 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56862.log.gz 2023-10-30T13:54:26.480 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46409.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46409.log.gz 2023-10-30T13:54:26.481 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53627.log 2023-10-30T13:54:26.481 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64441.log 2023-10-30T13:54:26.481 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54870.log.gz 2023-10-30T13:54:26.481 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55375.log 2023-10-30T13:54:26.482 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53627.log.gz 2023-10-30T13:54:26.482 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60842.log 2023-10-30T13:54:26.482 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64441.log.gz 2023-10-30T13:54:26.482 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55375.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55375.log.gz 2023-10-30T13:54:26.483 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55700.log 2023-10-30T13:54:26.483 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44890.log 2023-10-30T13:54:26.483 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60842.log.gz 2023-10-30T13:54:26.487 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61496.log 2023-10-30T13:54:26.488 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55700.log.gz 2023-10-30T13:54:26.488 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44298.log 2023-10-30T13:54:26.488 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44890.log.gz 2023-10-30T13:54:26.489 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61496.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61496.log.gz 2023-10-30T13:54:26.489 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59254.log 2023-10-30T13:54:26.489 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57586.log 2023-10-30T13:54:26.489 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44298.log.gz 2023-10-30T13:54:26.490 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45132.log 2023-10-30T13:54:26.490 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59254.log.gz 2023-10-30T13:54:26.490 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57586.log: gzip -5 --verbose 88.3% -- -- replaced with /var/log/ceph/ceph-client.admin.57586.log.gz /var/log/ceph/ceph-client.admin.64415.log 2023-10-30T13:54:26.491 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T13:54:26.491 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45132.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45132.log.gz 2023-10-30T13:54:26.491 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63230.log 2023-10-30T13:54:26.491 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43867.log 2023-10-30T13:54:26.492 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64415.log.gz 2023-10-30T13:54:26.492 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50120.log 2023-10-30T13:54:26.492 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63230.log.gz 2023-10-30T13:54:26.492 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43867.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43867.log.gz 2023-10-30T13:54:26.493 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50827.log 2023-10-30T13:54:26.493 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50120.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50120.log.gz 2023-10-30T13:54:26.493 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51678.log 2023-10-30T13:54:26.497 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50827.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50827.log.gz 2023-10-30T13:54:26.497 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49060.log 2023-10-30T13:54:26.498 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51678.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51678.log.gz 2023-10-30T13:54:26.498 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57638.log 2023-10-30T13:54:26.498 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.49060.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49060.log.gz 2023-10-30T13:54:26.498 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53973.log 2023-10-30T13:54:26.499 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57638.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57638.log.gz 2023-10-30T13:54:26.499 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48609.log 2023-10-30T13:54:26.499 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53973.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53973.log.gz 2023-10-30T13:54:26.500 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45696.log 2023-10-30T13:54:26.500 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48609.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48609.log.gz 2023-10-30T13:54:26.500 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62791.log 2023-10-30T13:54:26.500 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50801.log 2023-10-30T13:54:26.501 INFO:teuthology.orchestra.run.smithi148.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.45696.log.gz 2023-10-30T13:54:26.501 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47512.log 89.4% 2023-10-30T13:54:26.501 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62791.log.gz 2023-10-30T13:54:26.502 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50801.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.50801.log.gz 2023-10-30T13:54:26.502 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46609.log 2023-10-30T13:54:26.505 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47512.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47512.log.gz 2023-10-30T13:54:26.505 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61872.log 2023-10-30T13:54:26.506 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46609.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46609.log.gz 2023-10-30T13:54:26.506 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61339.log 2023-10-30T13:54:26.506 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61872.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61872.log.gz 2023-10-30T13:54:26.506 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60184.log 2023-10-30T13:54:26.507 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63915.log 2023-10-30T13:54:26.507 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61339.log.gz 2023-10-30T13:54:26.507 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60184.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60184.log.gz 2023-10-30T13:54:26.508 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64564.log 2023-10-30T13:54:26.508 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63915.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63915.log.gz 2023-10-30T13:54:26.508 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-30T13:54:26.508 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64564.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64564.log.gz 2023-10-30T13:54:26.509 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46829.log 2023-10-30T13:54:26.511 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60921.log 2023-10-30T13:54:26.511 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46829.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46829.log.gz 2023-10-30T13:54:26.511 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55109.log 2023-10-30T13:54:26.512 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60921.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60921.log.gz 2023-10-30T13:54:26.512 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54415.log 2023-10-30T13:54:26.512 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55109.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55109.log.gz 2023-10-30T13:54:26.512 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64019.log 2023-10-30T13:54:26.513 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54415.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54415.log.gz 2023-10-30T13:54:26.513 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56971.log 2023-10-30T13:54:26.515 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64019.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64019.log.gz 2023-10-30T13:54:26.515 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58599.log 2023-10-30T13:54:26.515 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56971.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56971.log.gz 2023-10-30T13:54:26.515 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55539.log 2023-10-30T13:54:26.516 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58599.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.58599.log.gz --verbose 2023-10-30T13:54:26.516 INFO:teuthology.orchestra.run.smithi148.stderr: -- /var/log/ceph/ceph-client.admin.60737.log 2023-10-30T13:54:26.516 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55539.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55539.log.gz 2023-10-30T13:54:26.517 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47621.log 2023-10-30T13:54:26.518 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60737.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60737.log.gz 2023-10-30T13:54:26.519 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60973.log 2023-10-30T13:54:26.519 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47621.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47621.log.gz 2023-10-30T13:54:26.519 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46232.log 2023-10-30T13:54:26.519 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60973.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60973.log.gz 2023-10-30T13:54:26.520 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48747.log 2023-10-30T13:54:26.521 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46232.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46232.log.gz 2023-10-30T13:54:26.521 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61079.log 2023-10-30T13:54:26.521 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51840.log 2023-10-30T13:54:26.522 INFO:teuthology.orchestra.run.smithi148.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48747.log.gz 2023-10-30T13:54:26.523 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61079.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61079.log.gz 2023-10-30T13:54:26.523 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54837.log 2023-10-30T13:54:26.523 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51840.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51840.log.gz 2023-10-30T13:54:26.524 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62745.log 2023-10-30T13:54:26.525 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54837.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54837.log.gz 2023-10-30T13:54:26.525 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57049.log 2023-10-30T13:54:26.525 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62745.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62745.log.gz 2023-10-30T13:54:26.525 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53015.log 2023-10-30T13:54:26.526 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57049.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57049.log.gz 2023-10-30T13:54:26.527 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45429.log 2023-10-30T13:54:26.527 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53015.log: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.53015.log.gz 2023-10-30T13:54:26.528 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60999.log 2023-10-30T13:54:26.528 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45429.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45429.log.gz 2023-10-30T13:54:26.529 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55916.log 2023-10-30T13:54:26.529 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60999.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60999.log.gz 2023-10-30T13:54:26.530 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62323.log 2023-10-30T13:54:26.530 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55916.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55916.log.gz 2023-10-30T13:54:26.531 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50303.log 2023-10-30T13:54:26.532 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62323.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62323.log.gz 2023-10-30T13:54:26.532 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T13:54:26.533 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50303.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50303.log.gz 2023-10-30T13:54:26.533 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54977.log 2023-10-30T13:54:26.534 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60396.log 2023-10-30T13:54:26.534 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54977.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54977.log.gz 2023-10-30T13:54:26.535 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63610.log 2023-10-30T13:54:26.536 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60396.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60396.log.gz 2023-10-30T13:54:26.536 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59580.log 2023-10-30T13:54:26.537 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50723.log 2023-10-30T13:54:26.538 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63610.log.gz 2023-10-30T13:54:26.539 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59580.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59580.log.gz 2023-10-30T13:54:26.539 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55804.log 2023-10-30T13:54:26.540 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50723.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50723.log.gz 2023-10-30T13:54:26.541 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50749.log 2023-10-30T13:54:26.542 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53891.log 2023-10-30T13:54:26.543 INFO:teuthology.orchestra.run.smithi148.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55804.log.gz 2023-10-30T13:54:26.544 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50749.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50749.log.gz 2023-10-30T13:54:26.544 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58704.log 2023-10-30T13:54:26.544 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53891.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53891.log.gz 2023-10-30T13:54:26.545 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55031.log 2023-10-30T13:54:26.546 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58573.log 2023-10-30T13:54:26.547 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58704.log.gz 2023-10-30T13:54:26.548 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55031.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55031.log.gz 2023-10-30T13:54:26.548 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55401.log 2023-10-30T13:54:26.549 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58573.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58573.log.gz 2023-10-30T13:54:26.550 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53839.log 2023-10-30T13:54:26.550 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55401.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55401.log.gz 2023-10-30T13:54:26.551 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58343.log 2023-10-30T13:54:26.552 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54194.log 2023-10-30T13:54:26.553 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53839.log.gz 2023-10-30T13:54:26.553 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58343.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58343.log.gz 2023-10-30T13:54:26.553 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47349.log 2023-10-30T13:54:26.554 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50907.log 2023-10-30T13:54:26.555 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54194.log.gz 2023-10-30T13:54:26.556 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47349.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47349.log.gz 2023-10-30T13:54:26.556 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44481.log 2023-10-30T13:54:26.557 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50907.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50907.log.gz 2023-10-30T13:54:26.557 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64538.log 2023-10-30T13:54:26.558 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56697.log 2023-10-30T13:54:26.559 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44481.log.gz 2023-10-30T13:54:26.560 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64538.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64538.log.gz 2023-10-30T13:54:26.560 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56888.log 2023-10-30T13:54:26.562 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56697.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56697.log.gz 2023-10-30T13:54:26.562 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51494.log 2023-10-30T13:54:26.563 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56888.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56888.log.gz 2023-10-30T13:54:26.563 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59499.log 2023-10-30T13:54:26.564 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51494.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51494.log.gz 2023-10-30T13:54:26.565 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60045.log 2023-10-30T13:54:26.566 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54441.log 2023-10-30T13:54:26.567 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59499.log.gz 2023-10-30T13:54:26.567 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60045.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60045.log.gz 2023-10-30T13:54:26.568 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53384.log 2023-10-30T13:54:26.568 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57910.log 2023-10-30T13:54:26.569 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54441.log.gz 2023-10-30T13:54:26.569 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53384.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53384.log.gz 2023-10-30T13:54:26.570 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45874.log 2023-10-30T13:54:26.571 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59872.log 2023-10-30T13:54:26.572 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57910.log.gz 2023-10-30T13:54:26.572 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.0.45874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45874.log.gz 2023-10-30T13:54:26.573 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54038.log 2023-10-30T13:54:26.573 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60099.log 2023-10-30T13:54:26.574 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59872.log.gz 2023-10-30T13:54:26.575 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54038.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54038.log.gz 2023-10-30T13:54:26.575 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61846.log 2023-10-30T13:54:26.576 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45104.log 2023-10-30T13:54:26.577 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5%/var/log/ceph/ceph-client.admin.61846.log: -- replaced with /var/log/ceph/ceph-client.admin.60099.log.gz 2023-10-30T13:54:26.577 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61846.log.gz 2023-10-30T13:54:26.578 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46206.log 2023-10-30T13:54:26.578 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51345.log 2023-10-30T13:54:26.579 INFO:teuthology.orchestra.run.smithi148.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45104.log.gz 2023-10-30T13:54:26.580 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46206.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46206.log.gz 2023-10-30T13:54:26.580 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43974.log 2023-10-30T13:54:26.581 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60239.log 2023-10-30T13:54:26.581 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51345.log.gz 2023-10-30T13:54:26.582 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43974.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43974.log.gz 2023-10-30T13:54:26.583 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47731.log 2023-10-30T13:54:26.583 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60239.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60239.log.gz 2023-10-30T13:54:26.584 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61550.log 2023-10-30T13:54:26.585 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51371.log 2023-10-30T13:54:26.586 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47731.log.gz 2023-10-30T13:54:26.587 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61550.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61550.log.gz 2023-10-30T13:54:26.587 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56099.log 2023-10-30T13:54:26.588 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51371.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51371.log.gz 2023-10-30T13:54:26.588 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52665.log 2023-10-30T13:54:26.589 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62083.log 2023-10-30T13:54:26.590 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56099.log.gz 2023-10-30T13:54:26.590 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52665.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52665.log.gz 2023-10-30T13:54:26.591 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61628.log 2023-10-30T13:54:26.592 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62083.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62083.log.gz 2023-10-30T13:54:26.592 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46855.log 2023-10-30T13:54:26.593 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61602.log 2023-10-30T13:54:26.594 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61628.log.gz 2023-10-30T13:54:26.595 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46855.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46855.log.gz 2023-10-30T13:54:26.595 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54759.log 2023-10-30T13:54:26.596 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61602.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61602.log.gz 2023-10-30T13:54:26.597 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51520.log 2023-10-30T13:54:26.597 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45349.log 2023-10-30T13:54:26.598 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54759.log.gz 2023-10-30T13:54:26.599 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51520.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51520.log.gz 2023-10-30T13:54:26.599 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54733.log 2023-10-30T13:54:26.600 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45349.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45349.log.gz 2023-10-30T13:54:26.601 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51423.log 2023-10-30T13:54:26.602 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54733.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54733.log.gz 2023-10-30T13:54:26.602 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47861.log 2023-10-30T13:54:26.604 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57718.log 2023-10-30T13:54:26.604 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51423.log.gz 2023-10-30T13:54:26.605 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44219.log 2023-10-30T13:54:26.606 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47861.log.gz 2023-10-30T13:54:26.606 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57718.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57718.log.gz 2023-10-30T13:54:26.607 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59606.log 2023-10-30T13:54:26.608 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53927.log 2023-10-30T13:54:26.608 INFO:teuthology.orchestra.run.smithi148.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44219.log.gz 2023-10-30T13:54:26.609 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59606.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59606.log.gz 2023-10-30T13:54:26.610 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59081.log 2023-10-30T13:54:26.611 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53927.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53927.log.gz 2023-10-30T13:54:26.611 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57692.log 2023-10-30T13:54:26.612 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59081.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59081.log.gz 2023-10-30T13:54:26.613 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63282.log 2023-10-30T13:54:26.614 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57692.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57692.log.gz 2023-10-30T13:54:26.614 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53488.log 2023-10-30T13:54:26.615 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63282.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63282.log.gz 2023-10-30T13:54:26.616 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56134.log 2023-10-30T13:54:26.617 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57075.log 2023-10-30T13:54:26.617 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56134.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53488.log.gz 2023-10-30T13:54:26.618 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56134.log.gz 2023-10-30T13:54:26.619 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50224.log 2023-10-30T13:54:26.619 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57075.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57075.log.gz 2023-10-30T13:54:26.620 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50697.log 2023-10-30T13:54:26.620 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50224.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50224.log.gz 2023-10-30T13:54:26.621 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54950.log 2023-10-30T13:54:26.622 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50567.log 2023-10-30T13:54:26.623 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50697.log.gz 2023-10-30T13:54:26.623 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54950.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54950.log.gz 2023-10-30T13:54:26.624 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51397.log 2023-10-30T13:54:26.625 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50567.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50567.log.gz 2023-10-30T13:54:26.625 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59738.log 2023-10-30T13:54:26.626 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59712.log 2023-10-30T13:54:26.627 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51397.log.gz 2023-10-30T13:54:26.628 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59738.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59738.log.gz 2023-10-30T13:54:26.628 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54389.log 2023-10-30T13:54:26.629 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59712.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59712.log.gz 2023-10-30T13:54:26.630 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46487.log 2023-10-30T13:54:26.630 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59554.log 2023-10-30T13:54:26.631 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54389.log.gz 2023-10-30T13:54:26.632 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46487.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46487.log.gz 2023-10-30T13:54:26.632 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51652.log 2023-10-30T13:54:26.634 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59554.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59554.log.gz 2023-10-30T13:54:26.634 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50331.log 2023-10-30T13:54:26.635 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51652.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51652.log.gz 2023-10-30T13:54:26.635 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47323.log 2023-10-30T13:54:26.636 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47401.log 2023-10-30T13:54:26.637 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47323.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50331.log.gz 2023-10-30T13:54:26.637 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47323.log.gz 2023-10-30T13:54:26.638 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51572.log 2023-10-30T13:54:26.639 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38221.log 2023-10-30T13:54:26.639 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47401.log.gz 2023-10-30T13:54:26.640 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51572.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51572.log.gz 2023-10-30T13:54:26.640 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.33262.log 2023-10-30T13:54:26.642 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62557.log 2023-10-30T13:54:26.642 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38221.log.gz 2023-10-30T13:54:26.643 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.tmp-client.admin.33262.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.33262.log.gz 2023-10-30T13:54:26.643 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60479.log 2023-10-30T13:54:26.644 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54922.log 2023-10-30T13:54:26.645 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62557.log.gz 2023-10-30T13:54:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60479.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60479.log.gz 2023-10-30T13:54:26.646 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58737.log 2023-10-30T13:54:26.647 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54922.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54922.log.gz 2023-10-30T13:54:26.648 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57393.log 2023-10-30T13:54:26.649 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58737.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58737.log.gz 2023-10-30T13:54:26.649 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63941.log 2023-10-30T13:54:26.650 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57393.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T13:54:26.651 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57393.log.gz 2023-10-30T13:54:26.652 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63967.log 2023-10-30T13:54:26.652 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63941.log.gz 2023-10-30T13:54:26.653 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57445.log 2023-10-30T13:54:26.655 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63967.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.0.45865.log 2023-10-30T13:54:26.655 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63967.log.gz 2023-10-30T13:54:26.656 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50593.log 2023-10-30T13:54:26.657 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57445.log.gz 2023-10-30T13:54:26.657 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.0.45865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45865.log.gz 2023-10-30T13:54:26.658 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59660.log 2023-10-30T13:54:26.659 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63043.log 2023-10-30T13:54:26.659 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50593.log.gz 2023-10-30T13:54:26.661 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56397.log 2023-10-30T13:54:26.661 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59660.log.gz 2023-10-30T13:54:26.662 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53462.log 2023-10-30T13:54:26.662 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63043.log.gz 2023-10-30T13:54:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54361.log 2023-10-30T13:54:26.664 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56397.log.gz 2023-10-30T13:54:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53462.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53462.log.gz 2023-10-30T13:54:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63069.log 2023-10-30T13:54:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54361.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54361.log.gz -5 2023-10-30T13:54:26.667 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53279.log 2023-10-30T13:54:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59791.log 2023-10-30T13:54:26.669 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63069.log.gz 2023-10-30T13:54:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63386.log 2023-10-30T13:54:26.670 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53279.log.gz 2023-10-30T13:54:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56319.log 2023-10-30T13:54:26.671 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59791.log.gz 2023-10-30T13:54:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45323.log 2023-10-30T13:54:26.673 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63386.log.gz 2023-10-30T13:54:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63662.log 88.1% 2023-10-30T13:54:26.674 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56319.log.gz 2023-10-30T13:54:26.675 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63993.log 2023-10-30T13:54:26.676 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45323.log.gz 2023-10-30T13:54:26.677 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44674.log 2023-10-30T13:54:26.677 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63662.log.gz 2023-10-30T13:54:26.678 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48804.log 2023-10-30T13:54:26.679 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63993.log.gz 2023-10-30T13:54:26.680 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55453.log 2023-10-30T13:54:26.680 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44674.log.gz 2023-10-30T13:54:26.681 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52503.log 2023-10-30T13:54:26.682 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48804.log.gz 2023-10-30T13:54:26.683 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52850.log 2023-10-30T13:54:26.683 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55453.log.gz 2023-10-30T13:54:26.684 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52503.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52503.log.gz 2023-10-30T13:54:26.684 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48721.log 2023-10-30T13:54:26.686 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50959.log 2023-10-30T13:54:26.686 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52850.log.gz 2023-10-30T13:54:26.687 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44896.log 2023-10-30T13:54:26.688 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48721.log.gz 2023-10-30T13:54:26.689 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58395.log 2023-10-30T13:54:26.689 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50959.log.gz 2023-10-30T13:54:26.690 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44896.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44896.log.gz -5 2023-10-30T13:54:26.690 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56345.log 2023-10-30T13:54:26.692 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56425.log 2023-10-30T13:54:26.692 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58395.log.gz 2023-10-30T13:54:26.693 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63759.log 2023-10-30T13:54:26.694 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56345.log.gz 2023-10-30T13:54:26.695 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61470.log 2023-10-30T13:54:26.695 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56425.log.gz 2023-10-30T13:54:26.696 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62505.log 2023-10-30T13:54:26.696 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63759.log.gz 2023-10-30T13:54:26.698 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62427.log 2023-10-30T13:54:26.698 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61470.log.gz 2023-10-30T13:54:26.699 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55161.log 2023-10-30T13:54:26.699 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62505.log.gz 2023-10-30T13:54:26.700 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62427.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62427.log.gz 2023-10-30T13:54:26.701 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51004.log 2023-10-30T13:54:26.702 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55942.log 2023-10-30T13:54:26.702 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55161.log.gz 2023-10-30T13:54:26.703 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52902.log 2023-10-30T13:54:26.704 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51004.log.gz 2023-10-30T13:54:26.705 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55942.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55942.log.gz 2023-10-30T13:54:26.705 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44873.log 2023-10-30T13:54:26.707 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46554.log 2023-10-30T13:54:26.707 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52902.log.gz 2023-10-30T13:54:26.708 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44873.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44873.log.gz 2023-10-30T13:54:26.708 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54681.log 2023-10-30T13:54:26.710 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62957.log 2023-10-30T13:54:26.710 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46554.log.gz 2023-10-30T13:54:26.711 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44245.log 2023-10-30T13:54:26.712 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54681.log.gz 2023-10-30T13:54:26.712 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58183.log 2023-10-30T13:54:26.713 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62957.log.gz 2023-10-30T13:54:26.714 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58369.log 2023-10-30T13:54:26.714 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44245.log.gz 2023-10-30T13:54:26.716 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48503.log 2023-10-30T13:54:26.716 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58183.log.gz 2023-10-30T13:54:26.717 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45751.log 2023-10-30T13:54:26.717 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58369.log.gz 2023-10-30T13:54:26.718 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64099.log 2023-10-30T13:54:26.719 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48503.log.gz 2023-10-30T13:54:26.720 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58652.log 2023-10-30T13:54:26.720 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45751.log.gz 2023-10-30T13:54:26.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64099.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64099.log.gz 2023-10-30T13:54:26.722 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58429.log 2023-10-30T13:54:26.723 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62931.log 2023-10-30T13:54:26.723 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58652.log.gz 2023-10-30T13:54:26.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58429.log: gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.58429.log.gz 2023-10-30T13:54:26.724 INFO:teuthology.orchestra.run.smithi148.stderr: /var/log/ceph/ceph-client.admin.64125.log 2023-10-30T13:54:26.726 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62931.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62931.log.gz 2023-10-30T13:54:26.726 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44690.log 2023-10-30T13:54:26.727 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58131.log 2023-10-30T13:54:26.728 INFO:teuthology.orchestra.run.smithi148.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64125.log.gz 2023-10-30T13:54:26.729 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58951.log 2023-10-30T13:54:26.729 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44690.log.gz 2023-10-30T13:54:26.730 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58131.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T13:54:26.730 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58131.log.gz 2023-10-30T13:54:26.732 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57127.log 2023-10-30T13:54:26.732 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58951.log.gz 2023-10-30T13:54:26.734 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63584.log 2023-10-30T13:54:26.735 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55349.log 2023-10-30T13:54:26.736 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57127.log.gz 2023-10-30T13:54:26.737 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63584.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63584.log.gz 2023-10-30T13:54:26.738 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55349.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55349.log.gz 2023-10-30T13:54:26.739 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53865.log 2023-10-30T13:54:26.740 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52340.log 2023-10-30T13:54:26.742 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57936.log 2023-10-30T13:54:26.742 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53865.log.gz 2023-10-30T13:54:26.743 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52340.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52340.log.gz 2023-10-30T13:54:26.744 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57936.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57936.log.gz 2023-10-30T13:54:26.745 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59447.log 2023-10-30T13:54:26.746 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56241.log 2023-10-30T13:54:26.748 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59447.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59447.log.gz -5 2023-10-30T13:54:26.748 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45570.log 2023-10-30T13:54:26.749 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57286.log 2023-10-30T13:54:26.750 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56241.log.gz 2023-10-30T13:54:26.751 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57234.log 2023-10-30T13:54:26.751 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45570.log.gz 2023-10-30T13:54:26.752 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53148.log 2023-10-30T13:54:26.752 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57286.log.gz 2023-10-30T13:54:26.754 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46258.log 2023-10-30T13:54:26.754 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57234.log.gz 2023-10-30T13:54:26.755 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53148.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53148.log.gz 2023-10-30T13:54:26.756 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46258.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46258.log.gz 2023-10-30T13:54:26.757 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45078.log 2023-10-30T13:54:26.758 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62637.log 2023-10-30T13:54:26.760 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45078.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45078.log.gz -5 2023-10-30T13:54:26.760 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46803.log 2023-10-30T13:54:26.761 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62637.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62637.log.gz 2023-10-30T13:54:26.762 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59280.log 2023-10-30T13:54:26.763 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52585.log 2023-10-30T13:54:26.763 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46803.log.gz 2023-10-30T13:54:26.765 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44082.log 87.8% 2023-10-30T13:54:26.765 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59280.log.gz 2023-10-30T13:54:26.766 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52585.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52585.log.gz 2023-10-30T13:54:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53122.log 2023-10-30T13:54:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44082.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44082.log.gz 2023-10-30T13:54:26.768 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56505.log 2023-10-30T13:54:26.769 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53122.log: gzip 87.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.53122.log.gz --verbose 2023-10-30T13:54:26.769 INFO:teuthology.orchestra.run.smithi148.stderr: -- /var/log/ceph/ceph-client.admin.58789.log 2023-10-30T13:54:26.771 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52956.log 2023-10-30T13:54:26.771 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56505.log.gz 2023-10-30T13:54:26.772 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55135.log 2023-10-30T13:54:26.773 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58789.log.gz 2023-10-30T13:54:26.774 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63438.log 2023-10-30T13:54:26.774 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52956.log.gz 2023-10-30T13:54:26.775 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60816.log 2023-10-30T13:54:26.775 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55135.log.gz 2023-10-30T13:54:26.777 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63438.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63438.log.gz -5 2023-10-30T13:54:26.777 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55778.log 2023-10-30T13:54:26.778 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60816.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60816.log.gz 2023-10-30T13:54:26.779 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58925.log 2023-10-30T13:54:26.780 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48635.log 2023-10-30T13:54:26.780 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55778.log.gz 2023-10-30T13:54:26.781 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58925.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58925.log.gz 2023-10-30T13:54:26.782 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48635.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48635.log.gz 2023-10-30T13:54:26.783 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43690.log 2023-10-30T13:54:26.784 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52798.log 2023-10-30T13:54:26.786 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63733.log 2023-10-30T13:54:26.786 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43690.log.gz 2023-10-30T13:54:26.787 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2023-10-30T13:54:26.788 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52798.log.gz 2023-10-30T13:54:26.788 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63733.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63733.log.gz 2023-10-30T13:54:26.789 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64151.log 2023-10-30T13:54:26.791 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44115.log 2023-10-30T13:54:26.791 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2023-10-30T13:54:26.792 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64151.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64151.log.gz 2023-10-30T13:54:26.793 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44193.log 2023-10-30T13:54:26.794 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44115.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44115.log.gz 2023-10-30T13:54:26.795 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63334.log 2023-10-30T13:54:26.796 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44193.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44193.log.gz 2023-10-30T13:54:26.796 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57744.log 2023-10-30T13:54:26.797 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64363.log 2023-10-30T13:54:26.798 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63334.log.gz 2023-10-30T13:54:26.798 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57744.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57744.log.gz 2023-10-30T13:54:26.800 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64363.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.64363.log.gz 2023-10-30T13:54:26.801 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44324.log 2023-10-30T13:54:26.802 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45296.log 2023-10-30T13:54:26.804 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56451.log 2023-10-30T13:54:26.804 INFO:teuthology.orchestra.run.smithi148.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44324.log.gz 2023-10-30T13:54:26.805 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45296.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45296.log.gz 2023-10-30T13:54:26.805 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53096.log 2023-10-30T13:54:26.807 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54142.log 2023-10-30T13:54:26.807 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56451.log.gz 2023-10-30T13:54:26.808 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61235.log 2023-10-30T13:54:26.808 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53096.log.gz 2023-10-30T13:54:26.809 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54142.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54142.log.gz 2023-10-30T13:54:26.810 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46076.log 2023-10-30T13:54:26.811 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61235.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61235.log.gz 2023-10-30T13:54:26.811 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48885.log 2023-10-30T13:54:26.813 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53760.log 2023-10-30T13:54:26.813 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46076.log.gz 2023-10-30T13:54:26.814 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48885.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48885.log.gz 2023-10-30T13:54:26.814 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52418.log 2023-10-30T13:54:26.816 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58044.log 2023-10-30T13:54:26.816 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53760.log.gz 2023-10-30T13:54:26.817 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47676.log 2023-10-30T13:54:26.818 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52418.log.gz 2023-10-30T13:54:26.818 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58044.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58044.log.gz 2023-10-30T13:54:26.819 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46720.log 2023-10-30T13:54:26.820 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60422.log 2023-10-30T13:54:26.820 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47676.log.gz 2023-10-30T13:54:26.821 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53568.log 2023-10-30T13:54:26.822 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46720.log.gz 2023-10-30T13:54:26.823 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60422.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60422.log.gz 2023-10-30T13:54:26.823 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61687.log 2023-10-30T13:54:26.824 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53568.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53568.log.gz 2023-10-30T13:54:26.825 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55187.log 2023-10-30T13:54:26.826 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63863.log 2023-10-30T13:54:26.827 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61687.log.gz 2023-10-30T13:54:26.827 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55187.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55187.log.gz 2023-10-30T13:54:26.828 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46284.log 2023-10-30T13:54:26.829 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63863.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63863.log.gz 2023-10-30T13:54:26.830 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46284.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46284.log.gz 2023-10-30T13:54:26.831 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47564.log 2023-10-30T13:54:26.832 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51598.log 2023-10-30T13:54:26.834 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55291.log 2023-10-30T13:54:26.834 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47564.log.gz 2023-10-30T13:54:26.835 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56267.log 2023-10-30T13:54:26.836 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51598.log.gz 2023-10-30T13:54:26.837 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55291.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.55291.log.gz -5 2023-10-30T13:54:26.837 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63308.log 2023-10-30T13:54:26.838 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44898.log 2023-10-30T13:54:26.839 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56267.log.gz 2023-10-30T13:54:26.840 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52637.log 2023-10-30T13:54:26.840 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63308.log.gz 2023-10-30T13:54:26.841 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44898.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44898.log.gz 2023-10-30T13:54:26.841 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62479.log 2023-10-30T13:54:26.843 INFO:teuthology.orchestra.run.smithi150.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T13:54:26.844 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59188.log 2023-10-30T13:54:26.844 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52637.log.gz 2023-10-30T13:54:26.845 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62479.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62479.log.gz 2023-10-30T13:54:26.845 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58291.log 2023-10-30T13:54:26.846 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54168.log 2023-10-30T13:54:26.846 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59188.log.gz 2023-10-30T13:54:26.847 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44428.log 2023-10-30T13:54:26.848 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58291.log.gz 2023-10-30T13:54:26.848 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47427.log 2023-10-30T13:54:26.849 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54168.log.gz 2023-10-30T13:54:26.850 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44428.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44428.log.gz 2023-10-30T13:54:26.850 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45375.log 2023-10-30T13:54:26.852 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54576.log 2023-10-30T13:54:26.852 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47427.log.gz 2023-10-30T13:54:26.852 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45375.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45375.log.gz 2023-10-30T13:54:26.853 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56160.log 2023-10-30T13:54:26.855 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52928.log 2023-10-30T13:54:26.855 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54576.log.gz 2023-10-30T13:54:26.856 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56160.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56160.log.gz 2023-10-30T13:54:26.857 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52928.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52928.log.gz 2023-10-30T13:54:26.858 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T13:54:26.860 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58018.log 2023-10-30T13:54:26.861 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56919.log 2023-10-30T13:54:26.862 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58018.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58018.log.gz 2023-10-30T13:54:26.863 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56919.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56919.log.gz 2023-10-30T13:54:26.864 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57964.log 2023-10-30T13:54:26.866 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61898.log 2023-10-30T13:54:26.867 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57964.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57964.log.gz 2023-10-30T13:54:26.868 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T13:54:26.868 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61898.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61898.log.gz 2023-10-30T13:54:26.869 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55003.log 2023-10-30T13:54:26.870 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44683.log 2023-10-30T13:54:26.872 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48778.log 2023-10-30T13:54:26.872 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55003.log.gz 2023-10-30T13:54:26.873 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44683.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44683.log.gz 2023-10-30T13:54:26.874 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48778.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48778.log.gz 2023-10-30T13:54:26.875 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44874.log 2023-10-30T13:54:26.876 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57666.log 2023-10-30T13:54:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44874.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44874.log.gz 2023-10-30T13:54:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57666.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57666.log.gz 2023-10-30T13:54:26.880 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52314.log 2023-10-30T13:54:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61416.log 2023-10-30T13:54:26.883 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56645.log 2023-10-30T13:54:26.883 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52314.log.gz 2023-10-30T13:54:26.884 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60869.log 2023-10-30T13:54:26.884 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61416.log.gz 2023-10-30T13:54:26.885 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56645.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56645.log.gz 2023-10-30T13:54:26.885 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46154.log 2023-10-30T13:54:26.887 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60318.log 2023-10-30T13:54:26.887 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60869.log.gz 2023-10-30T13:54:26.888 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46154.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46154.log.gz 2023-10-30T13:54:26.888 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52746.log 2023-10-30T13:54:26.890 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60613.log 2023-10-30T13:54:26.890 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60318.log.gz 2023-10-30T13:54:26.891 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52746.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52746.log.gz 2023-10-30T13:54:26.892 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59358.log 2023-10-30T13:54:26.893 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60613.log.gz 2023-10-30T13:54:26.894 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63204.log 2023-10-30T13:54:26.896 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62245.log 2023-10-30T13:54:26.896 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59358.log.gz 2023-10-30T13:54:26.897 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63204.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63204.log.gz 2023-10-30T13:54:26.898 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62245.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62245.log.gz 2023-10-30T13:54:26.899 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61131.log 2023-10-30T13:54:26.900 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61978.log 2023-10-30T13:54:26.902 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52558.log 2023-10-30T13:54:26.902 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61131.log.gz 2023-10-30T13:54:26.903 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61978.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61978.log.gz 2023-10-30T13:54:26.905 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53356.log 2023-10-30T13:54:26.905 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52558.log.gz 2023-10-30T13:54:26.906 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53708.log 2023-10-30T13:54:26.908 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53356.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53356.log.gz 2023-10-30T13:54:26.908 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58547.log 2023-10-30T13:54:26.910 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45946.log 2023-10-30T13:54:26.910 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53708.log.gz 2023-10-30T13:54:26.910 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58547.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58547.log.gz 2023-10-30T13:54:26.912 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45946.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45946.log.gz 2023-10-30T13:54:26.913 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59029.log 2023-10-30T13:54:26.914 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45646.log 2023-10-30T13:54:26.915 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59029.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.59029.log.gz 2023-10-30T13:54:26.916 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50251.log 2023-10-30T13:54:26.917 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63636.log 2023-10-30T13:54:26.917 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50251.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.45646.log.gz 2023-10-30T13:54:26.918 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50251.log.gz 2023-10-30T13:54:26.920 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63636.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63636.log.gz 2023-10-30T13:54:26.920 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62056.log 2023-10-30T13:54:26.922 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46635.log 2023-10-30T13:54:26.923 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62056.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62056.log.gz 2023-10-30T13:54:26.923 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64467.log 2023-10-30T13:54:26.924 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46635.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46635.log.gz 2023-10-30T13:54:26.925 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63256.log 2023-10-30T13:54:26.926 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64467.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64467.log.gz 2023-10-30T13:54:26.927 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48477.log 2023-10-30T13:54:26.928 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63471.log 2023-10-30T13:54:26.929 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63256.log.gz 2023-10-30T13:54:26.929 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48477.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48477.log.gz 2023-10-30T13:54:26.931 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63471.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63471.log.gz 2023-10-30T13:54:26.931 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62349.log 2023-10-30T13:54:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50068.log 2023-10-30T13:54:26.934 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62349.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62349.log.gz 2023-10-30T13:54:26.935 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45218.log 2023-10-30T13:54:26.936 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57531.log 2023-10-30T13:54:26.936 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50068.log.gz 2023-10-30T13:54:26.937 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56073.log 2023-10-30T13:54:26.938 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45218.log.gz 2023-10-30T13:54:26.939 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57531.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.57531.log.gz --verbose 2023-10-30T13:54:26.939 INFO:teuthology.orchestra.run.smithi148.stderr: -- /var/log/ceph/ceph-client.admin.44670.log 2023-10-30T13:54:26.940 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45455.log 2023-10-30T13:54:26.941 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56073.log.gz 2023-10-30T13:54:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44670.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44670.log.gz 2023-10-30T13:54:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51056.log 2023-10-30T13:54:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52392.log 2023-10-30T13:54:26.944 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45455.log.gz 2023-10-30T13:54:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44479.log 2023-10-30T13:54:26.945 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51056.log.gz 2023-10-30T13:54:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52392.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52392.log.gz 2023-10-30T13:54:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44479.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44479.log.gz 2023-10-30T13:54:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44402.log 2023-10-30T13:54:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46310.log 2023-10-30T13:54:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45972.log 2023-10-30T13:54:26.951 INFO:teuthology.orchestra.run.smithi148.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44402.log.gz 2023-10-30T13:54:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46310.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46310.log.gz 2023-10-30T13:54:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57612.log 2023-10-30T13:54:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45972.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.45972.log.gz 2023-10-30T13:54:26.954 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54545.log 2023-10-30T13:54:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56836.log 2023-10-30T13:54:26.956 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57612.log.gz 2023-10-30T13:54:26.956 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54545.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54545.log.gz 2023-10-30T13:54:26.957 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60680.log 2023-10-30T13:54:26.958 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52989.log 2023-10-30T13:54:26.959 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56836.log.gz 2023-10-30T13:54:26.960 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51467.log 2023-10-30T13:54:26.960 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60680.log.gz 2023-10-30T13:54:26.961 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52989.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.52989.log.gz -5 2023-10-30T13:54:26.961 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53786.log 2023-10-30T13:54:26.963 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51293.log 2023-10-30T13:54:26.963 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51467.log.gz 2023-10-30T13:54:26.964 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53542.log 2023-10-30T13:54:26.964 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53786.log.gz 2023-10-30T13:54:26.965 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51293.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.55995.log -- replaced with /var/log/ceph/ceph-client.admin.51293.log.gz 2023-10-30T13:54:26.966 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T13:54:26.967 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64229.log 2023-10-30T13:54:26.967 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53542.log.gz 2023-10-30T13:54:26.968 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55995.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.55995.log.gz -5 2023-10-30T13:54:26.968 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44167.log 2023-10-30T13:54:26.970 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45244.log 2023-10-30T13:54:26.970 INFO:teuthology.orchestra.run.smithi148.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64229.log.gz 2023-10-30T13:54:26.971 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63124.log 2023-10-30T13:54:26.972 INFO:teuthology.orchestra.run.smithi148.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44167.log.gz 2023-10-30T13:54:26.973 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62689.log 2023-10-30T13:54:26.973 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45244.log.gz 2023-10-30T13:54:26.974 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63124.log: gzip 89.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.63124.log.gz -- 2023-10-30T13:54:26.975 INFO:teuthology.orchestra.run.smithi148.stderr: /var/log/ceph/ceph-client.admin.55242.log 2023-10-30T13:54:26.976 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54493.log 2023-10-30T13:54:26.976 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62689.log.gz 2023-10-30T13:54:26.977 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55242.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55242.log.gz -5 2023-10-30T13:54:26.977 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51626.log 2023-10-30T13:54:26.979 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55881.log 2023-10-30T13:54:26.979 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54493.log.gz 2023-10-30T13:54:26.980 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51626.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51626.log.gz 2023-10-30T13:54:26.980 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50645.log 2023-10-30T13:54:26.981 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55881.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55881.log.gz 2023-10-30T13:54:26.982 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61313.log 2023-10-30T13:54:26.983 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50645.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50645.log.gz 2023-10-30T13:54:26.984 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61368.log 2023-10-30T13:54:26.985 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56777.log 2023-10-30T13:54:26.985 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61313.log.gz 2023-10-30T13:54:26.986 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61368.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.61368.log.gz -5 2023-10-30T13:54:26.986 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49034.log 2023-10-30T13:54:26.988 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50041.log 2023-10-30T13:54:26.988 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56777.log.gz 2023-10-30T13:54:26.989 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.49034.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49034.log.gz 2023-10-30T13:54:26.989 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50146.log 2023-10-30T13:54:26.991 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64284.log 2023-10-30T13:54:26.991 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50041.log.gz 2023-10-30T13:54:26.992 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43948.log 87.8% 2023-10-30T13:54:26.993 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50146.log.gz 2023-10-30T13:54:26.994 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46128.log 2023-10-30T13:54:26.994 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64284.log.gz 2023-10-30T13:54:26.995 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43948.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43948.log.gz 2023-10-30T13:54:26.996 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46128.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46128.log.gz 2023-10-30T13:54:26.997 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59993.log 2023-10-30T13:54:26.998 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63010.log 2023-10-30T13:54:27.000 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55646.log 2023-10-30T13:54:27.000 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59993.log.gz 2023-10-30T13:54:27.001 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50357.log 2023-10-30T13:54:27.002 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63010.log.gz 2023-10-30T13:54:27.002 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55646.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55646.log.gz 2023-10-30T13:54:27.004 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50357.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50357.log.gz 2023-10-30T13:54:27.004 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52366.log 2023-10-30T13:54:27.006 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63178.log 2023-10-30T13:54:27.007 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55427.log 2023-10-30T13:54:27.007 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52366.log.gz 2023-10-30T13:54:27.009 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44272.log 2023-10-30T13:54:27.009 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63178.log.gz 2023-10-30T13:54:27.010 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59686.log 2023-10-30T13:54:27.010 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55427.log.gz 2023-10-30T13:54:27.011 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44272.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44272.log.gz 2023-10-30T13:54:27.013 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59686.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59686.log.gz 2023-10-30T13:54:27.013 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53653.log 2023-10-30T13:54:27.015 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56047.log 2023-10-30T13:54:27.016 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46336.log 2023-10-30T13:54:27.017 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53653.log.gz 2023-10-30T13:54:27.017 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56047.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56047.log.gz 2023-10-30T13:54:27.019 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46336.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46336.log.gz 2023-10-30T13:54:27.019 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58477.log 2023-10-30T13:54:27.020 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46936.log 2023-10-30T13:54:27.022 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52691.log 2023-10-30T13:54:27.023 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58477.log.gz 2023-10-30T13:54:27.023 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46936.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46936.log.gz 2023-10-30T13:54:27.025 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52691.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52691.log.gz 2023-10-30T13:54:27.026 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63150.log 2023-10-30T13:54:27.027 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60789.log 2023-10-30T13:54:27.028 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59939.log 2023-10-30T13:54:27.029 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63150.log.gz 2023-10-30T13:54:27.030 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55620.log 2023-10-30T13:54:27.030 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60789.log.gz 2023-10-30T13:54:27.031 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59939.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59939.log.gz 2023-10-30T13:54:27.032 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55620.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55620.log.gz 2023-10-30T13:54:27.033 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44141.log 2023-10-30T13:54:27.034 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45998.log 2023-10-30T13:54:27.036 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62715.log 2023-10-30T13:54:27.036 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44141.log.gz 2023-10-30T13:54:27.037 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45998.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45998.log.gz 2023-10-30T13:54:27.038 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62715.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62715.log.gz 2023-10-30T13:54:27.039 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62135.log 2023-10-30T13:54:27.040 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62109.log 2023-10-30T13:54:27.042 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52444.log 2023-10-30T13:54:27.042 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62135.log.gz 2023-10-30T13:54:27.043 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62109.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62109.log.gz 2023-10-30T13:54:27.045 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50541.log 2023-10-30T13:54:27.045 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52444.log.gz 2023-10-30T13:54:27.046 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64045.log 2023-10-30T13:54:27.048 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45166.log 2023-10-30T13:54:27.048 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50541.log.gz 2023-10-30T13:54:27.049 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64045.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64045.log.gz 2023-10-30T13:54:27.050 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45166.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45166.log.gz 2023-10-30T13:54:27.051 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50409.log 2023-10-30T13:54:27.052 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55323.log 2023-10-30T13:54:27.054 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54335.log 2023-10-30T13:54:27.054 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50409.log.gz 2023-10-30T13:54:27.055 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55323.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55323.log.gz 2023-10-30T13:54:27.057 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54335.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54335.log.gz 2023-10-30T13:54:27.057 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58678.log 2023-10-30T13:54:27.058 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60370.log 2023-10-30T13:54:27.060 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58678.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.58678.log.gz 2023-10-30T13:54:27.060 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48531.log 2023-10-30T13:54:27.061 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60370.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60370.log.gz 2023-10-30T13:54:27.062 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59824.log 2023-10-30T13:54:27.063 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48531.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.48531.log.gz 2023-10-30T13:54:27.063 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64389.log 2023-10-30T13:54:27.064 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50619.log 2023-10-30T13:54:27.065 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59824.log.gz 2023-10-30T13:54:27.065 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64389.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64389.log.gz 2023-10-30T13:54:27.066 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61952.log 2023-10-30T13:54:27.067 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54280.log 2023-10-30T13:54:27.067 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50619.log.gz 2023-10-30T13:54:27.069 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61952.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61952.log.gz 2023-10-30T13:54:27.070 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54280.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54280.log.gz 2023-10-30T13:54:27.071 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59473.log 2023-10-30T13:54:27.072 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56671.log 2023-10-30T13:54:27.074 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59473.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59473.log.gz 2023-10-30T13:54:27.074 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53682.log 2023-10-30T13:54:27.075 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52529.log 2023-10-30T13:54:27.075 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56671.log.gz 2023-10-30T13:54:27.076 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53682.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53682.log.gz 2023-10-30T13:54:27.078 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52529.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52529.log.gz 2023-10-30T13:54:27.078 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59107.log 2023-10-30T13:54:27.079 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53410.log 2023-10-30T13:54:27.081 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49989.log 2023-10-30T13:54:27.082 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59107.log.gz 2023-10-30T13:54:27.082 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53410.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53410.log.gz 2023-10-30T13:54:27.084 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.49989.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49989.log.gz 2023-10-30T13:54:27.084 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47538.log 2023-10-30T13:54:27.086 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53813.log 2023-10-30T13:54:27.087 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47538.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.47538.log.gz 2023-10-30T13:54:27.088 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57156.log 2023-10-30T13:54:27.089 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53813.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T13:54:27.089 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53813.log.gz 2023-10-30T13:54:27.090 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57156.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57156.log.gz 2023-10-30T13:54:27.093 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44054.log 2023-10-30T13:54:27.094 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48557.log 2023-10-30T13:54:27.096 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44054.log: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T13:54:27.096 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-30T13:54:27.096 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44054.log.gz 2023-10-30T13:54:27.097 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61713.log 2023-10-30T13:54:27.097 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48557.log.gz 2023-10-30T13:54:27.100 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.61713.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61713.log.gz 2023-10-30T13:54:27.100 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50094.log 2023-10-30T13:54:27.102 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52876.log 2023-10-30T13:54:27.104 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50094.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-30T13:54:27.104 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50094.log.gz 2023-10-30T13:54:27.105 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52876.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52876.log.gz 2023-10-30T13:54:27.105 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44884.log 2023-10-30T13:54:27.108 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47453.log 2023-10-30T13:54:27.108 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44884.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44884.log.gz 2023-10-30T13:54:27.111 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47453.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47453.log.gz 2023-10-30T13:54:27.112 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62663.log 2023-10-30T13:54:27.116 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62297.log 2023-10-30T13:54:27.116 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57023.log 2023-10-30T13:54:27.116 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62663.log.gz 2023-10-30T13:54:27.118 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.62297.log: -- /var/log/ceph/ceph-client.admin.56021.log 2023-10-30T13:54:27.118 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62297.log.gz 2023-10-30T13:54:27.125 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45401.log 2023-10-30T13:54:27.125 INFO:teuthology.orchestra.run.smithi148.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57023.log.gz 2023-10-30T13:54:27.125 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45401.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45401.log.gz 2023-10-30T13:54:27.125 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56021.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56021.log.gz 2023-10-30T13:54:27.126 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49935.log 2023-10-30T13:54:27.127 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.49935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48025.log 2023-10-30T13:54:27.127 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49935.log.gz 2023-10-30T13:54:27.129 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55752.log 2023-10-30T13:54:27.133 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48025.log: /var/log/ceph/ceph-client.admin.55752.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56293.log 2023-10-30T13:54:27.134 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48025.log.gz 2023-10-30T13:54:27.135 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55752.log.gz 2023-10-30T13:54:27.138 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.56293.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56293.log.gz 2023-10-30T13:54:27.139 INFO:teuthology.orchestra.run.smithi148.stderr: -- /var/log/ceph/ceph-client.admin.60587.log 2023-10-30T13:54:27.141 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46461.log 2023-10-30T13:54:27.143 INFO:teuthology.orchestra.run.smithi148.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60587.log.gz 2023-10-30T13:54:27.145 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50383.log 2023-10-30T13:54:27.149 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46461.log: /var/log/ceph/ceph-client.admin.50383.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50383.log.gz 2023-10-30T13:54:27.153 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52719.log 2023-10-30T13:54:27.153 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52719.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52719.log.gz 2023-10-30T13:54:27.154 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -- replaced with /var/log/ceph/ceph-client.admin.46461.log.gz -5 2023-10-30T13:54:27.154 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47757.log 2023-10-30T13:54:27.157 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47757.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47757.log.gz 2023-10-30T13:54:27.157 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57504.log 2023-10-30T13:54:27.159 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60132.log 2023-10-30T13:54:27.164 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60132.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60132.log.gz 2023-10-30T13:54:27.165 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57504.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57504.log.gz 2023-10-30T13:54:27.165 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45481.log 2023-10-30T13:54:27.168 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45481.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45481.log.gz 2023-10-30T13:54:27.168 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60554.log 2023-10-30T13:54:27.171 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60554.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60554.log.gz 2023-10-30T13:54:27.172 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63785.log 2023-10-30T13:54:27.175 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63785.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63785.log.gz 2023-10-30T13:54:27.175 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51030.log 2023-10-30T13:54:27.178 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51030.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51030.log.gz 2023-10-30T13:54:27.178 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51546.log 2023-10-30T13:54:27.181 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51546.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51546.log.gz 2023-10-30T13:54:27.182 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48963.log 2023-10-30T13:54:27.185 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48963.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48963.log.gz 2023-10-30T13:54:27.185 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54090.log 2023-10-30T13:54:27.188 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54090.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54090.log.gz 2023-10-30T13:54:27.189 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52772.log 2023-10-30T13:54:27.191 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52772.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52772.log.gz 2023-10-30T13:54:27.192 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46777.log 2023-10-30T13:54:27.195 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46777.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46777.log.gz 2023-10-30T13:54:27.195 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61261.log 2023-10-30T13:54:27.198 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61261.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61261.log.gz 2023-10-30T13:54:27.199 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53253.log 2023-10-30T13:54:27.202 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53253.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53253.log.gz 2023-10-30T13:54:27.202 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48937.log 2023-10-30T13:54:27.205 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48937.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48937.log.gz 2023-10-30T13:54:27.206 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60709.log 2023-10-30T13:54:27.209 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60709.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60709.log.gz 2023-10-30T13:54:27.209 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46180.log 2023-10-30T13:54:27.212 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46180.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46180.log.gz 2023-10-30T13:54:27.212 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61739.log 2023-10-30T13:54:27.215 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61739.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61739.log.gz 2023-10-30T13:54:27.216 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60158.log 2023-10-30T13:54:27.219 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60158.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60158.log.gz 2023-10-30T13:54:27.219 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54622.log 2023-10-30T13:54:27.222 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54622.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54622.log.gz 2023-10-30T13:54:27.222 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62585.log 2023-10-30T13:54:27.225 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59967.log 2023-10-30T13:54:27.226 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62585.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62585.log.gz 2023-10-30T13:54:27.229 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59967.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59967.log.gz 2023-10-30T13:54:27.229 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58899.log 2023-10-30T13:54:27.232 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58899.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58899.log.gz 2023-10-30T13:54:27.232 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60344.log 2023-10-30T13:54:27.235 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60344.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60344.log.gz 2023-10-30T13:54:27.236 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53200.log 2023-10-30T13:54:27.239 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53200.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53200.log.gz 2023-10-30T13:54:27.239 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54896.log 2023-10-30T13:54:27.242 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54896.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54896.log.gz 2023-10-30T13:54:27.242 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45270.log 2023-10-30T13:54:27.246 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45270.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45270.log.gz 2023-10-30T13:54:27.246 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58847.log 2023-10-30T13:54:27.249 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58847.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58847.log.gz 2023-10-30T13:54:27.249 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62004.log 2023-10-30T13:54:27.250 INFO:teuthology.orchestra.run.smithi150.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T13:54:27.252 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62004.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62004.log.gz 2023-10-30T13:54:27.253 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55968.log 2023-10-30T13:54:27.256 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55968.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55968.log.gz 2023-10-30T13:54:27.256 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48447.log 2023-10-30T13:54:27.259 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48447.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48447.log.gz 2023-10-30T13:54:27.260 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53734.log 2023-10-30T13:54:27.263 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53734.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53734.log.gz 2023-10-30T13:54:27.263 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53041.log 2023-10-30T13:54:27.266 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53041.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53041.log.gz 2023-10-30T13:54:27.267 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55057.log 2023-10-30T13:54:27.270 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55057.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55057.log.gz 2023-10-30T13:54:27.270 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61105.log 2023-10-30T13:54:27.273 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61105.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61105.log.gz 2023-10-30T13:54:27.273 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48830.log 2023-10-30T13:54:27.276 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48830.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48830.log.gz 2023-10-30T13:54:27.277 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57260.log 2023-10-30T13:54:27.280 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57260.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57260.log.gz 2023-10-30T13:54:27.280 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38346.log 2023-10-30T13:54:27.283 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38346.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38346.log.gz 2023-10-30T13:54:27.283 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50438.log 2023-10-30T13:54:27.287 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50438.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50438.log.gz 2023-10-30T13:54:27.287 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44350.log 2023-10-30T13:54:27.290 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44350.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44350.log.gz 2023-10-30T13:54:27.290 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60947.log 2023-10-30T13:54:27.294 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44470.log 2023-10-30T13:54:27.294 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60947.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60947.log.gz 2023-10-30T13:54:27.295 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56750.log 2023-10-30T13:54:27.295 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44470.log.gz 2023-10-30T13:54:27.298 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56750.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56750.log.gz 2023-10-30T13:54:27.298 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44689.log 2023-10-30T13:54:27.301 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54253.log 2023-10-30T13:54:27.301 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44689.log.gz 2023-10-30T13:54:27.303 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54707.log 2023-10-30T13:54:27.304 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54253.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54253.log.gz 2023-10-30T13:54:27.307 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54707.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54707.log.gz 2023-10-30T13:54:27.308 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57367.log 2023-10-30T13:54:27.311 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57367.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57367.log.gz 2023-10-30T13:54:27.312 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54811.log 2023-10-30T13:54:27.315 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54811.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54811.log.gz 2023-10-30T13:54:27.315 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50277.log 2023-10-30T13:54:27.318 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50277.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50277.log.gz 2023-10-30T13:54:27.318 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57858.log 2023-10-30T13:54:27.321 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63707.log 2023-10-30T13:54:27.322 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57858.log.gz 2023-10-30T13:54:27.323 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47213.log 2023-10-30T13:54:27.326 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63707.log: gzip 88.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.43922.log 2023-10-30T13:54:27.326 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47213.log: -- replaced with /var/log/ceph/ceph-client.admin.63707.log.gz 2023-10-30T13:54:27.327 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47213.log.gz 2023-10-30T13:54:27.330 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43922.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43922.log.gz 2023-10-30T13:54:27.330 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50879.log 2023-10-30T13:54:27.333 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50879.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50879.log.gz 2023-10-30T13:54:27.333 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60895.log 2023-10-30T13:54:27.340 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60895.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60895.log.gz 2023-10-30T13:54:27.341 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44897.log 2023-10-30T13:54:27.341 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44897.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44897.log.gz 2023-10-30T13:54:27.341 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53308.log 2023-10-30T13:54:27.344 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53308.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53308.log.gz 2023-10-30T13:54:27.344 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56538.log 2023-10-30T13:54:27.346 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56538.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56538.log.gz 2023-10-30T13:54:27.347 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48911.log 2023-10-30T13:54:27.350 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48911.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48911.log.gz 2023-10-30T13:54:27.351 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38111.log 2023-10-30T13:54:27.354 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38111.log: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.38111.log.gz 2023-10-30T13:54:27.355 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44684.log 2023-10-30T13:54:27.357 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58264.log 2023-10-30T13:54:27.357 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44684.log.gz 2023-10-30T13:54:27.358 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50198.log 2023-10-30T13:54:27.360 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58264.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58264.log.gz 2023-10-30T13:54:27.363 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50198.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50198.log.gz 2023-10-30T13:54:27.364 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59412.log 2023-10-30T13:54:27.368 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59412.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59412.log.gz 2023-10-30T13:54:27.368 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50933.log 2023-10-30T13:54:27.371 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50933.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50933.log.gz 2023-10-30T13:54:27.371 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48859.log 2023-10-30T13:54:27.374 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48859.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48859.log.gz 2023-10-30T13:54:27.375 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51267.log 2023-10-30T13:54:27.378 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51267.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51267.log.gz 2023-10-30T13:54:27.378 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47036.log 2023-10-30T13:54:27.381 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47036.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47036.log.gz 2023-10-30T13:54:27.381 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52477.log 2023-10-30T13:54:27.385 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52477.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52477.log.gz 2023-10-30T13:54:27.385 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61157.log 2023-10-30T13:54:27.388 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61157.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61157.log.gz 2023-10-30T13:54:27.388 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44463.log 2023-10-30T13:54:27.391 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48695.log 2023-10-30T13:54:27.392 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44463.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44463.log.gz 2023-10-30T13:54:27.395 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.48695.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48695.log.gz 2023-10-30T13:54:27.395 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61926.log 2023-10-30T13:54:27.398 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61926.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61926.log.gz 2023-10-30T13:54:27.398 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56723.log 2023-10-30T13:54:27.401 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56723.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56723.log.gz 2023-10-30T13:54:27.402 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57314.log 2023-10-30T13:54:27.405 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57314.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57314.log.gz 2023-10-30T13:54:27.405 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56479.log 2023-10-30T13:54:27.408 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56479.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56479.log.gz 2023-10-30T13:54:27.409 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52258.log 2023-10-30T13:54:27.412 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52258.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52258.log.gz 2023-10-30T13:54:27.412 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44478.log 2023-10-30T13:54:27.415 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44478.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44478.log.gz 2023-10-30T13:54:27.415 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44682.log 2023-10-30T13:54:27.419 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44682.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44682.log.gz 2023-10-30T13:54:27.419 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52286.log 2023-10-30T13:54:27.422 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52286.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52286.log.gz 2023-10-30T13:54:27.422 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44893.log 2023-10-30T13:54:27.425 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44893.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44893.log.gz 2023-10-30T13:54:27.426 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62217.log 2023-10-30T13:54:27.429 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62217.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62217.log.gz 2023-10-30T13:54:27.429 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59306.log 2023-10-30T13:54:27.432 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59306.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59306.log.gz 2023-10-30T13:54:27.432 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63837.log 2023-10-30T13:54:27.435 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63837.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63837.log.gz 2023-10-30T13:54:27.436 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47479.log 2023-10-30T13:54:27.439 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47479.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47479.log.gz 2023-10-30T13:54:27.439 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63095.log 2023-10-30T13:54:27.443 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63095.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63095.log.gz 2023-10-30T13:54:27.443 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46881.log 2023-10-30T13:54:27.446 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46881.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46881.log.gz 2023-10-30T13:54:27.446 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58212.log 2023-10-30T13:54:27.449 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58212.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58212.log.gz 2023-10-30T13:54:27.450 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44465.log 2023-10-30T13:54:27.453 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44465.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44465.log.gz 2023-10-30T13:54:27.453 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62531.log 2023-10-30T13:54:27.456 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62531.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62531.log.gz 2023-10-30T13:54:27.456 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57560.log 2023-10-30T13:54:27.459 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57560.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57560.log.gz 2023-10-30T13:54:27.460 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51082.log 2023-10-30T13:54:27.463 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51082.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51082.log.gz 2023-10-30T13:54:27.463 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62453.log 2023-10-30T13:54:27.466 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62453.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62453.log.gz 2023-10-30T13:54:27.466 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45894.log 2023-10-30T13:54:27.470 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.45894.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45894.log.gz 2023-10-30T13:54:27.470 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44482.log 2023-10-30T13:54:27.471 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62030.log 2023-10-30T13:54:27.474 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.44482.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44482.log.gz 2023-10-30T13:54:27.474 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60763.log 2023-10-30T13:54:27.475 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62030.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62030.log.gz 2023-10-30T13:54:27.478 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60763.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60763.log.gz 2023-10-30T13:54:27.478 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59633.log 2023-10-30T13:54:27.481 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59633.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59633.log.gz 2023-10-30T13:54:27.482 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52611.log 2023-10-30T13:54:27.485 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.52611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47184.log 2023-10-30T13:54:27.485 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52611.log.gz 2023-10-30T13:54:27.486 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58098.log 2023-10-30T13:54:27.488 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47184.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47184.log.gz 2023-10-30T13:54:27.491 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58098.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58098.log.gz 2023-10-30T13:54:27.492 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61768.log 2023-10-30T13:54:27.495 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61768.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61768.log.gz 2023-10-30T13:54:27.496 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55083.log 2023-10-30T13:54:27.499 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55083.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55083.log.gz 2023-10-30T13:54:27.499 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50671.log 2023-10-30T13:54:27.502 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50671.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50671.log.gz 2023-10-30T13:54:27.503 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61209.log 2023-10-30T13:54:27.507 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55565.log 2023-10-30T13:54:27.507 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61209.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61209.log.gz 2023-10-30T13:54:27.507 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64203.log 2023-10-30T13:54:27.507 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55565.log.gz 2023-10-30T13:54:27.510 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64203.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64203.log.gz 2023-10-30T13:54:27.510 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63889.log 2023-10-30T13:54:27.514 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63889.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63889.log.gz 2023-10-30T13:54:27.514 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50775.log 2023-10-30T13:54:27.517 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50775.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50775.log.gz 2023-10-30T13:54:27.517 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55506.log 2023-10-30T13:54:27.520 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55506.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55506.log.gz 2023-10-30T13:54:27.521 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58818.log 2023-10-30T13:54:27.523 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58818.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58818.log.gz 2023-10-30T13:54:27.524 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61820.log 2023-10-30T13:54:27.526 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61820.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61820.log.gz 2023-10-30T13:54:27.527 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57775.log 2023-10-30T13:54:27.530 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57775.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57775.log.gz 2023-10-30T13:54:27.531 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51319.log 2023-10-30T13:54:27.533 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51319.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51319.log.gz 2023-10-30T13:54:27.534 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-10-30T13:54:27.540 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64256.log 2023-10-30T13:54:27.546 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60019.log 2023-10-30T13:54:27.547 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.64256.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64256.log.gz 2023-10-30T13:54:27.549 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55213.log 2023-10-30T13:54:27.551 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60019.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60019.log.gz 2023-10-30T13:54:27.554 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63811.log 2023-10-30T13:54:27.557 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55213.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55213.log.gz 2023-10-30T13:54:27.558 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54116.log 2023-10-30T13:54:27.562 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63811.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62163.log 2023-10-30T13:54:27.565 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54116.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63811.log.gz 2023-10-30T13:54:27.567 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62163.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62163.log.gz 2023-10-30T13:54:27.568 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54116.log.gz 2023-10-30T13:54:27.569 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50853.log 2023-10-30T13:54:27.574 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54519.log 2023-10-30T13:54:27.575 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50853.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50853.log.gz 2023-10-30T13:54:27.578 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47375.log 2023-10-30T13:54:27.579 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.54519.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54519.log.gz 2023-10-30T13:54:27.581 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61025.log 2023-10-30T13:54:27.582 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.47375.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47375.log.gz 2023-10-30T13:54:27.585 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62611.log 2023-10-30T13:54:27.587 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61025.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61025.log.gz 2023-10-30T13:54:27.588 INFO:teuthology.orchestra.run.smithi150.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T13:54:27.588 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60451.log 2023-10-30T13:54:27.590 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62611.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62611.log.gz 2023-10-30T13:54:27.595 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60451.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60451.log.gz 2023-10-30T13:54:27.597 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46977.log 2023-10-30T13:54:27.602 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56945.log 2023-10-30T13:54:27.603 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46977.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46977.log.gz 2023-10-30T13:54:27.605 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56803.log 2023-10-30T13:54:27.607 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56945.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56945.log.gz 2023-10-30T13:54:27.608 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50015.log 2023-10-30T13:54:27.610 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56803.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56803.log.gz 2023-10-30T13:54:27.613 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55674.log 2023-10-30T13:54:27.615 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50015.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50015.log.gz 2023-10-30T13:54:27.618 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.55674.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55674.log.gz 2023-10-30T13:54:27.621 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62271.log 2023-10-30T13:54:27.622 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46694.log 2023-10-30T13:54:27.623 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62271.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62271.log.gz 2023-10-30T13:54:27.626 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57419.log 2023-10-30T13:54:27.627 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46694.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46694.log.gz 2023-10-30T13:54:27.633 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59003.log 2023-10-30T13:54:27.634 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.57419.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57419.log.gz 2023-10-30T13:54:27.638 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58763.log 2023-10-30T13:54:27.639 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.59003.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59003.log.gz 2023-10-30T13:54:27.642 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50484.log 2023-10-30T13:54:27.643 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58763.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58763.log.gz 2023-10-30T13:54:27.649 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.50484.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53227.log 2023-10-30T13:54:27.649 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50484.log.gz 2023-10-30T13:54:27.654 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60071.log 2023-10-30T13:54:27.655 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53227.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53227.log.gz 2023-10-30T13:54:27.657 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53070.log 2023-10-30T13:54:27.659 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60071.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60071.log.gz 2023-10-30T13:54:27.665 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62375.log 2023-10-30T13:54:27.666 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.53070.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53070.log.gz 2023-10-30T13:54:27.671 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62375.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62375.log.gz 2023-10-30T13:54:27.672 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51187.log 2023-10-30T13:54:27.677 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61794.log 2023-10-30T13:54:27.679 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.51187.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51187.log.gz 2023-10-30T13:54:27.681 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62853.log 2023-10-30T13:54:27.683 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.61794.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.61794.log.gz 2023-10-30T13:54:27.689 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46907.log 2023-10-30T13:54:27.690 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62853.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62853.log.gz 2023-10-30T13:54:27.694 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62401.log 2023-10-30T13:54:27.695 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.46907.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46907.log.gz 2023-10-30T13:54:27.701 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.62401.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63517.log 2023-10-30T13:54:27.705 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62401.log.gz 2023-10-30T13:54:27.705 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56619.log 2023-10-30T13:54:27.707 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.63517.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63517.log.gz 2023-10-30T13:54:27.710 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60213.log 2023-10-30T13:54:27.711 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.56619.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56619.log.gz 2023-10-30T13:54:27.714 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58317.log 2023-10-30T13:54:27.715 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.60213.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60213.log.gz 2023-10-30T13:54:27.718 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.58317.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58317.log.gz 2023-10-30T13:54:28.932 INFO:teuthology.orchestra.run.smithi148.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T13:54:32.869 INFO:teuthology.orchestra.run.smithi150.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T13:54:33.070 INFO:teuthology.orchestra.run.smithi148.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-10-30T13:54:34.533 INFO:teuthology.orchestra.run.smithi148.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T13:54:35.051 INFO:teuthology.orchestra.run.smithi150.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T13:54:35.123 INFO:teuthology.orchestra.run.smithi150.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T13:54:35.321 INFO:teuthology.orchestra.run.smithi150.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T13:54:36.358 INFO:teuthology.orchestra.run.smithi148.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T13:54:36.687 INFO:teuthology.orchestra.run.smithi148.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T13:54:38.395 INFO:teuthology.orchestra.run.smithi148.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T13:54:54.711 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-30T13:54:54.712 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-10-30T13:54:54.713 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m28.330s 2023-10-30T13:54:54.713 INFO:teuthology.orchestra.run.smithi150.stderr:user 1m1.268s 2023-10-30T13:54:54.713 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m2.174s 2023-10-30T13:55:52.199 INFO:teuthology.orchestra.run.smithi148.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-30T13:55:53.492 INFO:teuthology.orchestra.run.smithi148.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-30T13:55:53.494 INFO:teuthology.orchestra.run.smithi148.stderr: 2023-10-30T13:55:53.494 INFO:teuthology.orchestra.run.smithi148.stderr:real 1m27.135s 2023-10-30T13:55:53.495 INFO:teuthology.orchestra.run.smithi148.stderr:user 3m24.007s 2023-10-30T13:55:53.495 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m16.726s 2023-10-30T13:55:53.496 INFO:tasks.ceph:Archiving logs... 2023-10-30T13:55:53.497 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/remote/smithi148/log 2023-10-30T13:55:53.498 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T13:56:16.470 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/remote/smithi150/log 2023-10-30T13:56:16.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T13:56:23.070 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T13:56:23.079 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T13:56:23.079 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T13:56:23.084 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T13:56:23.177 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T13:56:23.185 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T13:56:23.186 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:56:23.190 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:56:23.203 INFO:teuthology.orchestra.run.smithi148.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:56:23.203 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================== 2023-10-30T13:56:23.204 INFO:teuthology.orchestra.run.smithi148.stdout:*hv01.front.sepi 44.4.53.2 2 u 46 64 377 0.071 -5.389 1.141 2023-10-30T13:56:23.204 INFO:teuthology.orchestra.run.smithi148.stdout:+hv02.front.sepi 63.231.80.2 3 u 31 64 377 0.099 -0.670 0.785 2023-10-30T13:56:23.204 INFO:teuthology.orchestra.run.smithi148.stdout:+hv03.front.sepi 74.6.168.72 3 u 49 64 377 0.089 -3.424 1.151 2023-10-30T13:56:23.204 INFO:teuthology.orchestra.run.smithi148.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T13:56:23.206 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:56:23.207 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2023-10-30T13:56:23.207 INFO:teuthology.orchestra.run.smithi150.stdout:*hv01.front.sepi 44.4.53.2 2 u 49 64 377 0.086 -3.355 1.175 2023-10-30T13:56:23.207 INFO:teuthology.orchestra.run.smithi150.stdout:+hv02.front.sepi 63.231.80.2 3 u 40 64 377 0.098 +0.929 0.478 2023-10-30T13:56:23.207 INFO:teuthology.orchestra.run.smithi150.stdout:+hv03.front.sepi 74.6.168.72 3 u 41 64 377 0.113 -0.822 0.698 2023-10-30T13:56:23.208 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T13:56:23.208 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T13:56:23.217 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T13:56:23.218 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T13:56:23.227 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T13:56:23.235 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T13:56:23.244 INFO:teuthology.task.internal:Duration was 3513.444367 seconds 2023-10-30T13:56:23.244 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T13:56:23.254 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T13:56:23.254 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T13:56:23.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T13:56:23.350 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T13:56:23.351 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2023-10-30T13:56:23.351 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T13:56:23.373 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2023-10-30T13:56:23.374 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T13:56:23.384 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T13:56:23.385 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T13:56:23.423 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T13:56:23.459 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T13:56:23.460 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T13:56:23.503 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T13:56:23.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T13:56:23.597 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T13:56:23.597 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T13:56:23.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T13:56:23.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T13:56:23.639 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T13:56:23.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T13:56:23.677 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2023-10-30T13:56:23.683 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2023-10-30T13:56:23.702 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T13:56:23.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:56:23.751 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T13:56:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:56:23.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T13:56:23.767 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T13:56:23.767 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/remote/smithi148 2023-10-30T13:56:23.768 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T13:56:23.817 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041/remote/smithi150 2023-10-30T13:56:23.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T13:56:23.837 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T13:56:23.837 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T13:56:23.859 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T13:56:23.883 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T13:56:23.894 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T13:56:23.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T13:56:23.911 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T13:56:23.911 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T13:56:23.915 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T13:56:23.920 INFO:teuthology.orchestra.run.smithi148.stdout: 657967 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 13:56 /home/ubuntu/cephtest 2023-10-30T13:56:23.931 INFO:teuthology.orchestra.run.smithi150.stdout: 524291 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 13:56 /home/ubuntu/cephtest 2023-10-30T13:56:23.933 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T13:56:23.943 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T13:56:23.963 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T13:56:23.990 DEBUG:teuthology.nuke:shortname: smithi148 2023-10-30T13:56:23.991 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T13:56:24.019 DEBUG:teuthology.nuke:shortname: smithi150 2023-10-30T13:56:24.019 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T13:56:24.029 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:48:11.395517', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKeRIfRxzODgVbK6MlfsIgDCiYV1EZSosY4UyltWr7mHCPO7pEuEDj8D6YWgeqTEN/fv5kOWCQIbkChn6/v4dvg='} 2023-10-30T13:56:24.039 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:48:11.393968', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPEFw8CnH+x/8AQAfMpXjhYNhZQb3piNcsHdDTMDzCoqz04lQJ3zW4OeCWF+KuNDgMaIbs2PAfuzKUqbzbZ66m0='} 2023-10-30T13:56:24.077 INFO:teuthology.orchestra.console.smithi148:Power off 2023-10-30T13:56:24.078 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T13:56:24.093 INFO:teuthology.orchestra.console.smithi150:Power off 2023-10-30T13:56:24.094 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T13:56:24.102 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2023-10-30T13:56:24.103 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:56:24.115 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2023-10-30T13:56:24.116 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:56:24.126 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-10-30T13:56:24.138 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-30T13:56:28.127 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:56:28.139 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:56:28.238 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-10-30T13:56:28.249 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-30T13:56:32.243 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:56:32.251 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:56:32.353 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-10-30T13:56:32.360 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2023-10-30T13:56:32.461 INFO:teuthology.orchestra.console.smithi150:Power off completed 2023-10-30T13:56:36.357 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:56:36.467 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-10-30T13:56:36.568 INFO:teuthology.orchestra.console.smithi148:Power off completed 2023-10-30T13:56:36.674 INFO:teuthology.run:Summary data: client.0-kernel-sha1: eacf00cc51e204f401b4d5d4f84825a68f373c74 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 3513.4443674087524 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi148 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a8ef11cfd88a4a36879e10fdab92daed status: fail success: false 2023-10-30T13:56:36.675 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T13:56:36.747 INFO:teuthology.run:FAIL